@ross135 If you’re using Netlify’s Prerendering, turn it off.
See some recent related threads here:
https://answers.netlify.com/t/google-search-console-can-no-longer-index-site/147118
https://answers.netlify.com/t/web-pages-returning-504-gateway-timeout-or-502-bad-gateway-in-crawlers-like-google-console-ahrefs-and-semrush/147387
https://answers.netlify.com/t/crawlers-seeing-502-504-errors-when-netlify-prerendering-is-enabled/147626
https://answers.netlify.com/t/500-errors-when-being-crawled-how-to-verify/149488
If you have your own serverless functions, you can check the Function logs