Screaming Frog SEO Crawler receiving 429 Too Many Requests

Hi, we are receiving 429 response codes when using Screaming Frog to crawl our site. They don’t appear to be coming from our serverless function, so were wondering if Netlify can confirm if they come from their own layer? An example request ID is 01HY0GCJM2R7W0MXQTP2HWD9BS. Thanks.

@b42tom Netlify will be able to confirm, (I don’t work for them), but from a quick check of past forum topics it does look like 429 is a status they return when rate limiting for anomalous requests.

Checking Screaming Frog’s own documentation for a 429 status they suggest lowering the Crawl speed:

https://www.screamingfrog.co.uk/seo-spider/user-guide/configuration/#speed

From our logs, the 429 (at least for that request) came from your Function and not Netlify.