Blocking AI Scraping Bots

I’m looking to figure out if there is a way to block access to my websites from different AI scraping bots. There are some suggestions here on ways to approach this but I’m not sure how to implement them on netlify. In particular, I’d like to block access from specific user-agents. I see that it is possible to do rate limiting on user-agents but that is all I can find (and this requires core-enterprise).

Any help would be appreciated, thank you.

robots.txt is the best way (as long as the bot respects that file). You can otherwise use Edge Functions: Home | Edge Functions on Netlify (edge-functions-examples.netlify.app)

1 Like