Gzip file serving

Hi, @deepbluezen, we are always happy to research unusual behavior and thank you for letting us know those files are served gzipped for you.

There are edge cases to be aware of for compress file serving (which is otherwise automatic - you don’t need to manually gzip files before deploying).

For example, we don’t gzip things that are bigger when gzipped. It might sound odd for gzipping to make files larger but this often happens for PNGs. This is because they are already highly compressed so the gzip headers only make them bigger.

If you do see anything you think is unusual, please always feel free to make new posts and/or topics here. We will be happy to take a look. The same goes for questions too. It doesn’t have to be only for bugs. We love answering any questions there are about our service. :+1:

1 Like

@luke I’m indeed seeing js files that are over 100KB which are not being compressed. Here’s an example site:
https://the-expert-on-call.netlify.app/

Hi, @teamgi. All the Netlify hosted javascript files load compressed for me. What makes you believe they are not sent compressed?

If you still think the responses are uncompressed, we need to be able to track the HTTP responses with this issue. The simplest way to do this is to send us the x-nf-request-id header which we send with every HTTP response.

There more information about this header here:

If that header isn’t available for any reason, please send the information it replaces (or as many of these details as possible). Those details are:

  • the complete URL requested
  • the IP address for the system making the request
  • the IP address for the CDN node that responded
  • the day of the request
  • the time of the request
  • the timezone the time is in

@luke thanks for looking into this. It must have been a temporary issue. We ran a test yesterday on webpagetest.org and our js files were all coming up as uncompressed. Just ran it again and everything looks good now. Thanks again for the quick response.

1 Like

Hi,
@luke, @fool I have also problem with gzipping, I can’t find any file (which is bigger than 1kB), to be gzipped. Please let me know if I need to add something to _headers file, maybe I have somehow turned it off? Or maybe it’s because I use CloudFlare as my DNS provider? It happens when I use CloudFlare’s URL and direct Netlify’s link.

Hiya @degregar!

No need to set anything custom for this feature to work, and your DNS provider is irrelevant.

Your browser announces support for Brotli compression (as well as gzip/deflate) in this request header from your screenshot:

image

Our service responds with a Brotli compressed asset:

image

So I think things are working well. Let me know if you still disagree!

Hi @fool,

Sorry to pile in on this thread, but I just want to sanity check my observations in terms of asset compression, as I feel like I’m going a bit mad.

I have a two site setup where content is built and available on site 1 (wizardly-payne-3612b7) and served to a frontend on site 2 (clever-murdock-f2e04c). Site 1 is served via a proxy route on site 2 to mitigate CORS issues (some of the assets are downloaded via JS).

Here’s a request for a jpg image, which is considerably over the 1kb size mentioned above, and can be compressed to a considerably smaller size with gzip. I don’t believe this file is being compressed for transfer:

I tested this in Chrome, Safari and FireFox and saw the same behaviour, but I considered that there might be something weird in my local client setup causing Netlify to avoid compressing assets. However, I observed the same results for the same asset request using a remote machine on browserling.com:

I finally considered that the proxying, custom headers or some other technical implementation detail in my setup could be causing the issue, so I checked the site @teamgi had posted in August. I also found no compression was applied to this png despite it being over the 1kb minimum and despite gziping seemingly producing a smaller file in my tests.

Would you be able to help me understand if I’m doing anything wrong that’s causing Netlify to avoid compression of these assets please? I know that I can probably further optimise the size of the original assets in many cases (as well as using some of the modern image formats like webp and avif), but gzip or brotli encoding feels like a baseline performance gain that could improve the speed of download for these assets.

Hi, @andrewbridge, the JPEG itself is already an incredibly highly compressed file format.

There is a thought experiment in data compression theory which asks, “Can all files be compressed?”

The only logical answer is “no”. Taking the thought experiment to the extreme, if the answer was “yes” then all results of a file compression can themselves be compressed again. Again, if this was true (and it is not) we could keep compressing any file over and over until it was only a single bit in size. This is obviously not possible.

Clearly a single 0 or 1 in binary cannot represent all possible files. There is a limit to how much any one file can be compressed.

Why do I ramble about data compression theory?

To summarize, if you gzip this image with the best gzip compression - it gets bigger not smaller:

-rw-r--r--  1 me  people  28806 Nov 18 18:32 kings-disease-nas-xsmall-square.jpg
-rw-r--r--  1 me  people  28838 Nov 18 18:32 kings-disease-nas-xsmall-square.jpg.gz

Gzipping adds 32 bytes to the file size in my tests (which were done using Apple gzip 287.100.2 and the --best option). We won’t compress a file if there is no gain. I’m not sure how you are gzipping if you are getting a different result. Again, JPEGs are very hard to compress precisely because that is what a JPEG is; it is already a highly compressed file.

It will be very common for already compressed files not to be further compressed by brotli or gzip for this reason.

If there are other questions about this, please let us know.

Hi @luke,

I have large CSV files that I want my web app to request on demand. Since they’re large, I’ve manually gzipped them (~25 MB each → ~2 MB each). Testing the web app locally, they load fine, as browsers can automatically “unzip” gzip files without any libraries.

However, when deploying to Netlify, I get back gobbledygook data for these files. My guess was that Netlify is applying another layer of compression onto the already-gzipped files, and this seems to be confirmed by this thread? And furthermore, if I understand correctly, there is no way to tell Netlify not to compress an already compressed file?

I’d really like to avoid having to install a package like pako and have an additional in-JavaScript decompression step (probably slower than browser built-in decompression). Furthermore, I’m also deploying to other places which may not automatically compress files like this, so I’d have to change the data loading logic depending on where the app is being deployed.

Edit It looks like GitHub Pages, for example, does the same thing. Automatic compression. I guess this is standard practice for (free) hosting services, so not a big deal.

Hi @vincerubinetti welcome back to the forums! You’re correct that Netlify automatically compress files to optimize bandwidth and improve performance. This compression is typically applied to regular text-based files like HTML, CSS, and JavaScript files.

It might be best to serve the uncompressed CSV files directly to Netlify. This way, you don’t have to worry about handling compression and can ensure consistent behavior across platforms. While the file size will be larger, it can be a simpler.

If I know that a certain format supports compression, how do I tell netlify that I want it to do so?
(DXT, ETC, etc compressed textures)

Screen of hosted .dds files

Can I add my own formats to auto compression?

No, but you can compress those files yourself and setup an Edge Function to serve those files instead.