Netlify corrupting served data files by compressing already-compressed files

In my app we have some large (10-20 MB) wasm and data files that we pre-compress using Brotli in-order to minimize bandwidth. For example, this file https://62b142977e4d505a425e9438--adoring-tesla-fc69b1.netlify.app/spatial-webgl/Spatial_6.25.0_250c0f59_dev/Build/WebGL.wasm.br.

It appears that Netlify is Brotli compressing the file again when serving it. At runtime, I’m getting WebAssembly compilation issues like this:

CompileError: WebAssembly.instantiate(): expected magic word 00 61 73 6d, found 6b 8d 00 55 @+0

When serving the exact same source file that I uploaded to Netlify as part of the build and serve it locally or through an S3 bucket (https://spatial-public-assets.s3.amazonaws.com/spatial-webgl/Spatial_6.25.0_250c0f59_dev/Build/WebGL.wasm.br) there’s no problem.

I’ve seen some other threads about disabling Brotli compression (like Serving pre-compressed Brotli files - #5 by universe), but I didn’t quite understand the suggested workaround.

Are there any other suggestions? Could Netlify just skip compressing files that already end in .br, or give us some way of configuring that behavior?

1 Like

hi there, have you tried turning asset optimization off in the UI?

Just tried disabling asset optimization entirely in the site’s settings and deployed a new version. It did not fix the problem, still having the same issue.

Hi @Lou,

I don’t think compression of files is the issue here. Even if we serve it by compressing using Brotli further, in the end, it should be decoded as original by your browser.

Also, there’s a huge difference in the two files you’ve shared. The one uploaded to Netlify seems 12 MB while one on AWS seems 50+ MB. This is not some compression applied by Netlify, but two entirely different files.

I’d advise you to check the file that’s being uploaded and compare those two.

Hi @hrishikesh,

The details you’ve described are the exact problem. Netlify is brotli compressing an already compressed file, which has a very small effect on the size. When you click the link to download the file, your browser automatically decompresses as part of the download process. So when downloading the version served from Netlify, the browser decompresses it once, decompressing the little bit that the second compression pass accomplished. The 12.6MB file you end up with is still compressed — hence why it’s so much smaller.

When downloading the version from S3, the browser decompresses the original, highly compressed file that was uploaded, hence why the downloaded file is over 4x larger.

This is a little easier to see when analyzing text files instead of binary files. Here’s a JS file that we’ve been pre-compressing and uploading to S3, and are trying to move to same-origin hosting through Netlify

Note that when viewing the S3 version, it loads as expected and you can read the text, while the Netlify one has strange characters throughout. This is because it’s still compressed — the browser’s converting what is still binary data to text and ending up with that garbage.

You can further verify this by manually decompressing the JS file hosted on Netlify. Using the official brotli utility and running brotli -d WebGL.framework.js.br and then inspecting the output file you can see that it is the expected, fully decompressed JS. And if you compare it against the file hosted on S3 you’ll see that it is the exact same file, byte for byte.

I can’t reproduce this behaviour. I downloaded the file from your S3 link, published it to Netlify here:

https://cool-fox-2a357c.netlify.app/WebGL.framework.js.br

I get a correct file served (at least doesn’t show the mangled characters like you showed)

(I had to set content-type to text/javascript or else it was being served as `application/octet-stream, thus downloading the file instead of displaying it in browser)

Note, to verify, I disabled Brotli in my browser (and even GZip to be sure), and tried to load the file from your Netlify link. Since browser was now not sending the accept-encoding header, Netlify would not serve Brotli and I was still getting the weird file - which makes me think something else has happened during the file upload.

I can’t reproduce this behaviour. I downloaded the file from your S3 link, published it to Netlify here:

https://cool-fox-2a357c.netlify.app/WebGL.framework.js.br

I get a correct file served (at least doesn’t show the mangled characters like you showed)

Did you upload the compressed version or uncompressed version? When you download it from S3 through the browser it’s automatically decompressed. Did you download it with curl?

I disabled Brotli in my browser (and even GZip to be sure), and tried to load the file from your Netlify link. Since browser was now not sending the accept-encoding header, Netlify would not serve Brotli and I was still getting the weird file

Right, Netlify does not recompress the file in the absence of the accept-encoding header, but the browser also therefore does not decompress the response. And since the file was already compressed before uploading it, it’s served as is — compressed, without a response header instructing the browser to decompress it — showing as all the garbled content.

Hi, @Lou. Netlify isn’t corrupting anything. We send exactly the file you uploaded. As you uploaded it compressed, we send it in that format.

Netlify will never auto-detect the pre-compressed files as this is not an available feature at this time. Our support team can enter a feature request to support pre-compressing your files but that isn’t possible now. What you are doing is not supported currently.

All compression on the wire at Netlify happens automatically. In your case, you are uploading an already compressed file. Netlify is then sending that file as is and also brotli compressing it again. The browser doesn’t know there are two layers of compression. This is why the decompressed file is still compressed (because uploading pre-compressed files is not supported).

The solution here is to not pre-compress the files and to let our service do it automatically. If you want us to enter that feature request to support manually pre-compressing files, please let us know.

Hi @luke

Thanks for your response and for understanding the situation. Yes, please enter the feature request to support manually pre-compressing files.

It’s probably not as simple as this, but a flag/setting/checkbox for something along the lines of "don’t compress files ending in .br, but still serve them as brotli compressed with a Content-Encoding header with a value of br " would solve this problem for us.

The automatic compression that Netlify is doing isn’t at a high-enough setting for this use case. We’re compressing these at a quality level of 11, and it takes quite a bit of time; it’s not practical to do at serve-time.