Serving pre-compressed Brotli files

This is a follow-up to this comment:
https://answers.netlify.com/t/gzip-file-serving/3445/15

In relation to this site:
https://hugo-stork.netlify.app

As part of the build process, a 6.0 MB binary file (stork.index.json) is created in the root of site, consumed by a Wasm file for embedded site search. By giving the file a json extension[1], Netlify compresses with Brotli to produce a 2.1 MB file. This is excellent.

However, with aggressive Brotli compression ( -q 11) we can reduce the file size by another 28% to 1.5 MB.

We can improve performance and decrease cost by serving pre-compressed assets when present. For example:

public/
├── file.ext
├── file.ext.br
└── file.ext.gz

netlify-precompress-assets

Please let me know if there’s a way to accomplish this with what we have today.


  1. Yes, we’re lying about the file type to trigger compression. ↩︎

2 Likes

Hey @jmooring,

Glad to have you here. Thanks for all the help on Hugo forums!

About the following:

There’s no direct way at the moment. There’s a feature in works that might enable this kind of a functionality in future, but it’s still weeks or months away from general availability.

What you’re trying to do, might be somewhat doable with Netlify Functions. However, depending on your use case this might or might not be the best way.

Inside the function, you can try something like:

import Axios from 'axios'
export async function handler(event) {
  const url = `https://${event.headers.host}/${decodeURIComponent(event.queryStringParameters.target)}/?checked=true`
  if (event.headers['accept-encoding'].toLowerCase().includes('br')) {
    Axios({
      method: 'head',
      url
    }).then(({status}) => {
      if (status === 200) {
        return {
          headers: {
            location: url
          },
          statusCode: 301
        }
      }
    }).catch(error => {
      return {
        body: error,
        statusCode: 500
      }
    })
  } else if (event.headers['accept-encoding'].toLowerCase().includes('gzip')) {
    // same as above
  } else {
    // redirect to the normal file
  }
}

You can setup the redirect for the files like:

[[redirects]]
  force = true
  from = "/compressed/*"
  status = 301
  to = "/.netlify/functions/compress/?target=:splat"
  [redirects.conditions]
    check = ":check"

You need to place all your “compressed” files in the compressed folder and make requests to those URLs. Furthermore, it’s important to use the query parameter check when requesting these files to make sure these requests get redirected. Similarly, it’s important to “clear” the query parameters from the redirect made using the function by sending a different parameter or it will end in a redirect loop.

Does this work for you?

Thank you for your response. Although the reduction in file size is desirable, I am not comfortable recommending this approach due to its perceived complexity; too many moving parts, and too many ways to get it wrong. While I don’t particularly enjoy pandering to the lowest common denominator, sometimes it’s the better choice.

Thanks again.
Joe

1 Like

There’s no direct way at the moment. There’s a feature in works that might enable this kind of a functionality in future, but it’s still weeks or months away from general availability.

Did this feature get released? I’m also interested in serving precompressed assets with brotli -q 11, as the difference in file size in large. An extra ~30% saving is a lot!

Yeah, we’ve got Edge Functions now, which should allow you to redirect to the file you wish to.

Hi

I agree with @jmooring that the original suggestion on Mar 19 has too many moving parts.

Also, as far as I’m aware the Edge Functions solution also has many moving parts and is not straightforward to configure.

What we really need is the standard functionality to be as @jmooring describes in his original post. Or, maybe, that the functionality he describes to be enabled via a tick box or simple true/false in configuration.

Is there a technical reason why this is not an appropriate suggestion? I have to admit I’m puzzled why it’s so difficult to do something that I would consider a standard approach - i.e. serve the pre-compressed files if they exist (with brotli preferred).

I find having to jump through so many hoops to achieve 30% better compression to be quite odd. It’s not a small difference.

Hi there and thanks for the feedback!

We are a SaaS (or PaaS) service depending on how you define those terms; Regardless, our business can scale because we build only the most important features. Put another way, we do not intend to build every feature that might be useful to a subset of our customer base. We instead attempt to build the ones that will be useful to the most customers, or otherwise best serve our business interests. Customers have rarely asked for this feature, and since our business goal is not trying to sell “web hosting” so much as “a collaboration platform that includes web hosting among many other features and much configurability”, this request likely falls even further in priority since it does not help folks collaborate at all.

This is an existing feature request, and as you point out, it would certainly be possible to implement. I have passed your suggestion on to our Product team, who plans our roadmap and chooses what we’ll build in the future. In the end, they’ll choose what to build, but I would not expect them to build this. Why’s that? Read on to find out!

We have thousands of feature requests open (counting just the ones I’ve seen in my past 6 years here), so we clearly have to prioritize among them. This is not a frequently requested feature (twice before that I am aware of), and since our CDN already automatically compresses your text assets using brotli or gzip (not, I understand, quite as thoroughly as you can yourself), it’s very likely not going to become a high priority for us anytime soon.

If you want that level of control over the details of your site service, you do have some options.

We’ve already suggested a few ways to approach this on our platform:

  • If you want to stay on our service and “in the cloud”, you’ll want to do something like use an Edge Function or our redirects. Or similar features on other platforms exist as well, of course.
  • If you want total control over how your files are served, you’ll probably want to run a server.

Too complicated for you? That’s a decision you will have to make, but it will be our suggestion for these less-requested behaviors:

  • you can build it on our platform if you want to and we’ll guide you (but, we will offer you advice and you choose to implement it or not);
  • you can of course apply your standards and say it’s not something you want to support in production, and implement it on another service
  • you could also say that it isn’t that important to you, doesn’t justify the effort expense of finding another service or solving this yourself, and then you don’t need to do anything.

But…you’ll be best suited trying to do things at Netlify, if you work within our available featureset and possible extensions, which this has been pointed out is possible in more than one way for those who are motivated :slight_smile: