Site bandwidth usage grew 4x with no increase in unique users

Netlify site: http://murderbridge.netlify.app/
On Sept 8th, starting ~12pm EST the bandwidth usage for my site increased to ~4x the usual amount

This would be expected if the site grew in popularity but the number of unique users stayed constant in the same time period. I also had no changes made to the site for the past 8 - 9 months.

Google analytics had roughly the same findings (with the pageviews also staying the same). At the rate I am consuming bandwidth, I will use up all my bandwidth within a week or two.

I suspected someone was hotlinking my assets or site content, and set up a Netlify edge function to block requests that aren’t from an allow list. So far this hasn’t had much effect.

I paid for a Netlify analytics upgrade, to see if I can tell what exactly is being exploited, but was a bit disappointed to find only generic level metrics that only gave me a marginally better window into when the issues began.

Are there any suggestions on where I can look or try to get to the bottom of this?

Ok I think the issue is that bots are spam scraping my site

We have an extension that will help you by-default block certain user agents. This extension will help you block null UA’s, as well as some other popular bots/scrapers here:
https://app.netlify.com/extensions/user-agent-blocker

You can also refer to our blog post about using a robots.txt file for blocking crawlers:

https://developers.netlify.com/guides/blocking-ai-bots-and-controlling-crawlers/

Thank you so much Sid! Updating robots.txt and adding _redirects fixed my issue.

1 Like