Help: I'm being billed a lot for bandwidth

[antoinerousseau]

Hi,
With some friends we created a little free quiz for the lockdown, but it got viral with 1M visitors yesterday alone.
The URL is https://tuvascraquer.fr
When I got a first extra bandwidth billing notification yesterday morning, I decided to move the DNS to CloudFlare to use their cache and reduce my bills, and it seemed to work, saving about 240 GB, leaving 65 GB to Netlify (although I don’t understand why it doesn’t save more).

Problem is, I just received an email notifying me of an extra $60 package added to my bill, even though yesterday afternoon I was only at 24 GB of my previous package in the billing section of my account! And now I get:

Breakdown of service for the current billing period (subscription fees will be prorated appropriately in your receipt):

• Mar 11 to Apr 11: Bandwidth ($0.00/month)
• Mar 31: Bandwidth extra usage package ($20.00)
• 9:00 AM: 3 Bandwidth extra usage package ($60.00)

I’m happy to pay $20 a day maximum but I can’t afford more and it seems to be growing
Do you know how I could reduce the costs since it’s a non-profit thing? Or should I just shut the website down?

Thanks for the help,

Antoine

[luke]

Hi, @antoinerousseau. I suppose it is time for both “congratulations on” and also “so sorry to hear about” the popularity of your site.

The following two screenshots show the bandwidth for this site for the last seven days and the last two days respectively. The time window for each segment (the x-axis unit) is one hour and the y-axis is bytes of bandwidth used (bytes not bits).

Seven days:

Two days:

It does appear that the changes you made today greatly reduced the bandwidth being used but it is still significant.

The times in the screenshots are UTC-0700 (minus seven hours). If the new peak usage is a max of 500 MB per hour, that is 12 GB a day. This would be 360 GB in next 30 days which would be another $60 (3 * $20 for three bandwidth add-ons). That is $60 more total for the thirty day projection and not $60 a day.

However, if that “bubble” at the start of the new day is the new normal behavior it would be higher. The last twenty-four hours of bandwidth for this site was just over 34 GB. With rounding, comes out to 1021 GB for thirty days which would be ten bandwidth add-ons for a total of $200 (10 * $20) over the next thirty days.

The only way to avoid the bandwidth charges is to not use the bandwidth. If you are okay with the new bandwidth usage rate, we can check again to make sure the projected $60 to $200 in the next thirty days is accurate.

We are happy to check on this in again in a day or two to see which of the new bandwidth projections is the more likely or if some new projection makes more sense.

However, if this bandwidth use and related charges are not acceptable, it will be up to you to decide how to best address that. This could mean completely taking the site down or moving it to another hosting service.

​Please let us know if there are other question and/or if you’d like another update about this in a day or two.

[antoinerousseau]

Hi @luke,

Thank you very much for this fast and detailed answer! It helps.
That would be great to have this chart right in the Netlify billing section but I guess it’s not that simple.
I added more aggressive caching rules on CloudFlare and am now getting 99%+ cache hits so it should be better.

Another issue I had this morning when waking up is that the JS assets where doing “too many redirects” errors… after some digging it appeared that CloudFlare was trying to access these over http but Netlify was redirecting to the https version, not knowing it was server-to-server maybe?
It could be because the domain was initially configured in Netlify and a certificate was provisioned so it has http to https redirects in place? Here is what I was getting, with (“hit”) or without (“dynamic”) CloudFlare cache:

curl -I https://tuvascraquer.fr/webpack-runtime-1e73942cf63b57021d45.js
HTTP/1.1 301 Moved Permanently
Date: Thu, 02 Apr 2020 07:42:09 GMT
Content-Type: text/plain
Content-Length: 79
Connection: keep-alive
Set-Cookie: __cfduid=dac5f784fd410bbc9e619ecbba6ac0d001585813329; expires=Sat, 02-May-20 07:42:09 GMT; path=/; domain=.tuvascraquer.fr; HttpOnly; SameSite=Lax; Secure
Location: https://tuvascraquer.fr/webpack-runtime-1e73942cf63b57021d45.js
Referrer-Policy: same-origin
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block
Age: 1498
Cache-Control: public,max-age=31536000,immutable
X-NF-Request-ID: be76d635-9508-40a8-9dca-03f3e9fbc73f-2383821
CF-Cache-Status: DYNAMIC
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Server: cloudflare
CF-RAY: 57d8e95a6bb6b769-CDG

I changed CloudFlare SSL settings from “flexible” to “full” (and cleared the cache), and it seems to be resolved, but I’m not sure what happened, and I’m not sure if Netlify will always accept https requests from CloudFlare, given that Netlify doesn’t serve the certificate to the public…
Any idea? Maybe an option to disable redirecting from http to https on Netlify?

Anyway, thanks again!

[luke]

Hi, @antoinerousseau, Netlify will always serve HTTPS for any site hosted with our service.

We can only provisions automatic SSL certificates from Let’s Encrypt for domains if their DNS records point directly to Netlify using one of our two supported methods: Netlify DNS or external DNS.

There is more about this here:

[Support Guide] What problems could occur when using Cloudflare in front of Netlify?

Last reviewed by Netlify Support - August, 2023 Note: We recommend not using both Cloudflare’s CDN (“Accelerate and Protect”, the orange cloud in their UI) and Netlify for the same site at the same time. Why? Read on! Netlify’s web service is not designed to work optimally with another CDN “in front of” our CDN. Proxying to our service is in general not supported and we advise you not to do it for best hosting results. Cloudflare has made similar statements that it doesn’t recommend this usa…

Is it possible for Cloudflare to make the HTTPS requests to the subdomain of netlify.com for this site? We will serve our wildcard SSL certificate (*.netlify.com) for this domain name.

So, for example, the asset above would be at a URL similar to this:

https://example-subdomain-here.netlify.com/webpack-runtime-1e73942cf63b57021d45.js

If this isn’t possible, another solution is to upload a custom SSL certificate to Netlify and continue to update that manually each time it it going to expire.

If there are other questions about this, we’re happy to answer.

[antoinerousseau]

I’m not sure I can choose how CloudFlare requests assets on Netlify, I only added some CNAME records and it worked:

This works well with the “full” SSL setting in CloudFlare, but not the “flexible” one:

It appears to be because CloudFlare requests Netlify over HTTP, not HTTPS:
https://support.cloudflare.com/hc/en-us/articles/115000219871#h_dfa85774-c19f-4f49-b11b-bf9bacf6e831

Which makes sense to always use the “full” setting with Netlify then.

The weird thing is that it didn’t cause any issue with another similar setup at whenwillyoucrack.com.
The only difference is that I didn’t first use the Netlify DNS.

Anyway, thanks for all the help! All good now

[noreff]

Hi @antoinerousseau, thanks for sharing this! I have similar case and it helped me a lot. You mention that you added more aggressive caching rules on CloudFlare, don’t you mind sharing those?

[antoinerousseau]

Hi @noreff, I added a CloudFlare Page Rule for mywebsite.com/* stating:

Cache Level: Cache Everything, Edge Cache TTL: 12 hours

And I manually purge the cache after each deploy using CloudFlare’s API.

[noreff]

@antoinerousseau thanks a lot!

[hillary]

Thanks for sharing, @antoinerousseau! @noreff, let us know if you have other questions related to this.