45GB of daily bandwidth usage

Support
,
1

Hello,
I’m having a problem with excessive bandwidth usage of https://www.ingrid.com/. Netlify analytics
shows 45GB of daily bandwidth usage. I’m using gatsby & netlify-cms for this site. Any ideas what
can cause such high usage?

Thanks,
Krzysztof

2

hi @kzuraw - that is a lot of bandwidth, indeed!

there are couple of ways we an approach this - does analytics give you any information about which pages are the most visited ones?

you might also download your deploy to see if you maybe are serving some assets that are unexpectedly large:

try those things first, and then we can keep troubleshooting!

3

Thanks for response :wave:t2:

  1. This is from analytics:

    image
    image610×878 31.8 KB

  2. From deploy:

du -h | sort -rh
110M	.
 40M	./admin
 37M	./static
8.6M	./img

It seems that admin & static are the largest - but will they count as 45GB daily? :thinking:

5

Hi, @kzuraw. I see the index.html file itself is about 1.2 MB uncompressed. It is only 167 kB compressed but I’m seeing a great number of requests coming in with a user-agent which contains “Go-http-client” and that client isn’t requesting the compressed files so the CDN sends them uncompressed.

(To be clear, no web server is permitted to send Gzip or Brotli compress content unless the client sends the headers to allow it. This client doesn’t send the headers so our CDN nodes are not allowed to send the content compressed.)

Even if it were compressed the 665594 page views would be 103 GB for just index.html. That doesn’t include the images, CSS, javascript and other assets.

In short, for the site above I’m showing 1297 GB of bandwidth used in the last 30 days. What isn’t clear is why there are so many requests coming from the Go HTTP client.

If there are other questions about this, please let us know.

6

Thanks for reply - it is possible to get dump of logs from one random hour? Thanks to that I can
check on my side what is happening.

7

nope, we can only send you results of query output against our log system with aggregated data. The logs themselves are protected PII.

So, what we can help with is questions like:

“Can you tell me how many different IP’s were involved in requesting index.html yesterday with the User-Agent of Go-http-client” and “Can you show me the pattern of requests for that file as a bar chart, over the past week?” and “What are my top referrers for /path.html?”. Not limited to those, but they are good examples to work from :slight_smile:

8

Thanks for answer - it turns out that it was our internal probing that caused such high usage. After
this change was reverted usage is normal - thank you for help :smile: