Confusing headers behaviour

I’m seeing some really weird behaviour with headers.

Here’s my headers file:

/*
  X-Frame-Options: DENY

/*.c.*
  Cache-Control: max-age=31536000
  x-test-header: foo

I thought this would set the Cache-Control and x-test-header headers on URLs that contain .c.. But it seems to be missing from URLs I’d expect, and applied to URLs I wouldn’t expect.

Any idea what’s going on?

Project: focused-booth-abbed1

Huh, I just saw _headers for newbies which suggests that * doesn’t really work as documented.

And Custom headers for Large Media - #4 by luke which says that headers can’t be applied to large media.

Could Custom headers | Netlify Docs be updated to prevent others being tripped up by the headers issue? It doesn’t really mention the limitations, and I’m still not really sure what they are (can * only be at the end of a rule?). Also, my deploy says “All header rules deployed without errors.” - it seems like this would be a good place to point out if a header rule isn’t supported.

Could Large Media requirements and limitations | Netlify Docs be updated to prevent others being tripped up by the large media + headers issue?

Also, I’d like to disable large media on my site, and go back to plain git. Large Media requirements and limitations | Netlify Docs says I need your help with that.

Hi, @jakearchibald, there is an open feature request for the docs change update about Large Media limitations with custom headers based on the other topic you linked to. We’ll post an update there (and here) when the docs are updated.

About the headers documentation it does say the following:

Paths can contain * or :placeholders . A :placeholder matches anything except / , while a * matches anything.

There are no examples which show that this can be used with a file extension. What wording would make it more clear that a file extension cannot be used? Would this be better?

Paths can contain * or :placeholders . A :placeholder matches anything except / , while a * matches anything.

You cannot use * to make rules like /*.html or /*/more/path/segments/. The * will match everything to the end of the line and .html and /more/path/segments/ would be ignored.

Regarding the deploy saying “All header rules deployed” despite the rules not working, would you please send us a link to that deploy?

Regarding removing Large Media would you please look over the following Support Guide?

That covers the steps involved. If you would then follow-up here with the identifier of the site, we’ll follow-up to let you know when the Large Media add-on is removed.

Excellent, cheers!

I get that there isn’t an example of it being used to match file extensions, but I don’t think the absence of an example suggests it doesn’t work. Otherwise, that would suggest that :placeholder doesn’t work at all, since there isn’t a single example of it on that page. I don’t think that’s the case, and I don’t think that’s how folks interpret documentation.

I think it’s better to explain how it works, rather than give an incomplete description, then try and fill in the gaps with exceptions. If I’ve understood how it works correctly, I think this describes it:

You can use * at the end of a pattern to match anything that appears afterwards. This makes the pattern a “starts with” match. So /foo/* will match any URL that starts with /foo/, including /foo/ itself, but also /foo/bar/ and /foo/hello/world/yo.html.

Warning: Although * can only be used at the end of a pattern, Netlify does not validate this. Instead, any pattern characters after the first * are simply discarded. Netlify will silently accept a rule like /posts/*.jpg, but it will not behave as you intend. It will be interpreted as /posts/*, so the rule will apply for /posts/cat.jpg, but it will also apply for /posts/data.json.

You need a strong warning here, because no one would include a rule like /posts/*.jpg expecting the .jpg to be ignored.

You wouldn’t need the strong warning if Netlify caught the issue at build time, and failed the build with a message like “Header rule error: * can only appear at the end of a pattern, see [url] for more details”.
This is my preference, as I would have discovered the issue at build time, rather than in production.

Netlify App - in this deploy I had a header rule like /*.c.*, which is being reinterpreted as /* without warning.

Cheers! I’ll put all that in a separate reply.

2 Likes

Consideration #1: Yeah, I’ll deploy to a test branch initially. I’ve also made an additional clone of the repo, including large files, just in case everything goes wrong.

Consideration #2: Yeah, I’ll continue to use Git LFS. I’ve set GIT_LFS_ENABLED=true.

Is there a way to set GIT_LFS_FETCH_INCLUDE to match everything?

Fwiw, [Support Guide] Uninstalling Large Media refers to the environment variables as “headers” which might confuse folks.

I think I’m ready for “Large Media add-on” to be removed.

Is there anything else you need from me before “Large Media add-on” is disabled?

Hi, @jakearchibald, the Large Media add-on has been removed now. For the GIT_LFS_FETCH_INCLUDE questions, I do believe GIT_LFS_FETCH_INCLUDE=* does just this.

I fixed the reference to “headers” in the support guide also. The feature requests and docs issues still need filing and I will follow-up again to confirm when that is complete as well.

Excellent. I’ve done the additional steps and redeployed. Everything looks good now. Many thanks for the the help!

The header documentation on the Netlify site is still unclear. The alternative given in Confusing headers behaviour - #4 by jakearchibald would be better

1 Like

Valuable feedback, @jakearchibald. Thanks! We’ve got a docs enhancement feature request raised internally so we can put this limitation and considerations on the team’s radar. We’ll loop back if and when any changes are considered!