Our site is having cache issues. When I deploy a new build, the cache on the site is rendering old content or missing content on the page from updates. The updates show when a user refreshes the page, but I need to set up a recache policy for returning users that want to see new content on a page. Should I remove all of my header settings and follow this article, Better Living Through Caching, with “max-age=0, must-revalidate, public”?
To note, I am moving our site from gatsby-source-graphql to gatsby-source-wordpress-experimental and attached below is the headers in my netlify.toml file.
[[headers]]
for = "*.js" # js files should be set this way
[headers.values]
Cache-Control = "public, max-age=604800"
X-Frame-Options = "DENY"
X-XSS-Protection = "1; mode=block"
[[headers]]
for = "*.css" # css files too
[headers.values]
Cache-Control = "public, max-age=604800"
X-Frame-Options = "DENY"
X-XSS-Protection = "1; mode=block"
Hi, @shill, if you override our caching this can mean that a local browser (not our CDN) will cache and load content that is no longer up to date. Note, if that is done, you control those changes not us.
If you do see out of date content coming from our CDN, we are happy to troubleshoot that. In those cases, we need a way to identify the request that was incorrect. To do this, we most often use the x-nf-request-id response header.
There is more information about this header in the support guide below:
If that header isn’t available for any reason, please send the information it replaces (or as many of these details as possible). Those details are:
the complete URL requested
the IP address for the system making the request
the IP address for the CDN node that responded
the day of the request
the time of the request
the timezone the time is in
To summarize, if you changed the headers, the local browser might be caching out of date content as a result. If so, if you stop using those cache-control headers and use our etag header instead for cache invalidation (which is automatic) it should resolve the issue. If you send us the x-nf-request-id for an out of date response, we can tell you more about why it happened.
If there are other questions, please include them at any time.
Hi @luke@gregraven , attached is our headers from the main page url before and after I removed all of our overrides in the netlify.toml file. We pushed an upgrade of our tech stack (with the overrides in the toml file) and users with local cache were getting white screens with the error message attached. I want to deploy again with the removed cache rules in the toml file, but wanted to get the green light from you that it should behave as expected with the CDN.
Let me know otherwise and a solution to removing the local cache from users that visit our site. Thank you!!
@shill Regarding the user-supplied error list, you should make the .map files available if your JS files include references to them, but their absence shouldn’t be causing any serious problems. I’d focus on those top three errors for now.
Apologies for not being clear. Those missing 404 files in the errors are from the old deploys and the cache is searching for them. I need to remove the cache and these issues should go away since there will be new CSS and JS files with the refreshed cache.
Let me know if there is something in this that could be causing the cache not revalidating for returning users. Thank you!
@gregraven, as an update, I updated the gatsby browser to refresh and still had errors.
@luke, here is the x-nf-request-id --> 06221951-8ec4-421e-9dce-3564f4e25ea9-9654443 that appears to be out of date, since it is showing and I am still having a white screen appear on the site.
Hi, @shill. The x-nf-request-id above was for a 304 response to a deploy preview URL on 2020-10-08 at 15:20:10.511 PDT.
Would you please send us the x-nf-request-id for the 404 itself?
I looked for 404s for the IP address that made the request for the ``x-nf-request-id` above around this same time. The only 404 I found was for a URL with this path:
This 404 was on 2020-10-08 at 14:25:25.030 PDT. This 404 however is correctly a 404.
The file above was served from 2020-10-08 12:59:10.832 to 2020-10-08 13:04:05.357.
At that time a new deploy was published and this new deploy no longer contains the file above. The request for this file at 14:25:25.030 PDT would be a 404 because it stopped being part of the deployed site at 13:04:05.357 PDT, over one hour before.
The deploy which created the file is: # 5f7f67e7588394012116ed56
The very next deploy deletes the file: # 5f7f6c0509553e012f6ecb2b
Again, if you have x-nf-request-id headers for other 404s, we would be happy to research this further.
netlify.tom file →
[[headers]]
for = “.js" # js files should be set this way
[headers.values]
Access-Control-Allow-Origin = "”
[[headers]]
for = “.css" # css files too
[headers.values]
Access-Control-Allow-Origin = "”
[[headers]]
for = “.json" # json files too
[headers.values]
Access-Control-Allow-Origin = "”
Hi, @shill. I don’t think we ever discovered the root cause for this issue.
Please note when troubleshooting using x-nf-request-id headers, I need the header for the problem response. The x-nf-request-id headers sent in this topic are all for successful responses (with 200 or 304 statuses). For example, the x-nf-request-id header sent today is for a 304 response (for x-nf-request-id: ba139ab1-1b2b-4a0c-b88c-d17c9dd9d24c-74894273). To troubleshoot, I need to know the x-nf-request-id for the 404s and I didn’t get any of those headers.
My hypothesis as to what was happening is this:
a cache-control header was used to extend the time an asset is cached
a user revisits the site and the cache-control header caused the cached content to be used even though our normal caching rules would have invalidated it
the cached content is out of date and references files no longer in the deploy resulting 404s for those assets
Again, this is only a best guess because I was never able to reproduce the issue. It also sounds like when the cache-control headers were removed the issue stopped happening. This is supporting evidence for my hypothesis but I have no hard proof (with correlation not being causation and all that jazz).
If you do see this issue again, please send an x-nf-request-id for a 404 and we will be happy to take another look.
Hi @luke@gregraven
right now am facing the above mentioned issue.
i use “gatsby-plugin-netlify” but didn’t mentioned any custom headers.
i use “gatsby-plugin-offline” but never enabled them on “gatsby-config.js”
the site was fine until i remove netlify.toml file(since i mentioned only build command, publish dir and function dir i wanted to take them to UI so i removed)
once removed the toml file and the very next build bring the above error. previouds build version css file sent 404 error.
p.s(i have changed www to non www which brings the site but same error when next build happens. so i would like to fix it properly) share your thoughts please.
Hey @shill can you able to deploy your changes after you set non-www, to www? in my case once i switch them to www to non-www, can able to trigger deploy without any errors… but it brings same error.
Hi, @visuvelu. I’m seeing that the request id above was made at 2020-11-30 08:25:27 UTC. The deploy which was active that that time was deploy 5fc4a21a330a580966ac4adf.
The file in question however, isn’t part of that deploy so that HTTP request returned a 404. I checked the other HTTP requests around that time for the same requesting IP address and they all receiving the same deploy id.
My questions for you, “Is why did your system request that file?” It wasn’t part of that deploy and I don’t see it referenced in there referrer header URL either.
In other words, this browser did make a request for a file which wasn’t in that deploy. So, the 404 is correct. The mystery to solve now is why did the browser request that particular file. However, the answer to that is something you will need to tell me because I don’t have that information.
For example, a HAR file recording of the issue would show us more about what happened in the surrounding requests. HAR files also normally record the exact response so if an incorrect version was served this could be verified in the HAR file.
Would you be willing to make a recording of that happening and send it to us? I can also move this troubleshooting to a private support ticket so we can talk openly about real URLs without sharing those details publicly.
Hi @luke
thank you for your information, yes i would like share real URLs in private support ticket and i can explain how the browser seeking previous build URL and will send you HAR file in that ticket itself.
can you please move this too private support ticket?
