Since at least yesterday our customers, primarily in California, have been getting incorrect assets served to them. Our JS and CSS files live under /assets and we have observed responses sporadically being return that are content-type “text/plain” with a body of “hello” instead of the correct Javascript or CSS files. This is happening to a subset of requests and not every asset in the page load is effected in the same way each time.
Initially we thought this might be Cloudflare (which sits in-front of the app) but after disabling it temporarily and removing all caching we still saw these strange responses from Netlify.
We have not had reports of this happening outside of California.
We are experiencing an issue with some of our javascript and CSS files, this issue is location related, it only happens to members of our team in San Francisco, or if
we use a VPN, when we open the website, we are getting a blank page, after checking the assets, we noticed that the content of some JS and CSS is being replaced with the text: “hello”, we contacted Cloudflare, and they asked us to open a ticket here, Can someone please help?
I have moved your post into an existing thread by @john2 which is related to streamline things. I have also escalated this to ensure Netlify Support Engineers see this.
Cloudflare support are telling me it’s on the Netlify side.
Netlify support are telling me it’s on the Cloudflare side.
Both services say multiple customers have reported it.
Our very rough and early investigation leads us to believe the hello file is most likely being served somewhere from the Netlify stack before it hits the deployed assets. Cloudflare caching just increases the chance of more users seeing it.
We have used Cloudflare + Netlify for years without a problem like this.
I’ve raised this with our security team and as far as we can tell it is specific to cloudflare’s cache, and while I can’t promise it doesn’t affect anyone else, it does not seem to originate on our service.
As Coel has helpfully pointed out, we do not provide tech support for people who proxy to our service from Cloudflare nor anywhere else, at least not below the enterprise account level.
One customer who reported this yesterday resolved it by turning off caching at Cloudflare, so you are welcome to try that out as well if you will keep using the unsupported configuration.
In the end, we still recommend you not proxy to us, which will also ensure that no problems in Cloudflare’s handling of Netlify requests or in its cache can affect your site’s service.
Our networking team will continue the investigation for us tomorrow, but since this is entirely in cloudflare’s cache as far as we can tell, they may not find anything to help.
Please do let us know in this thread if you discover anything else in the meantime that could be relevant.
We’ve likely found the cause for this and rolled back the code that would have caused this.
Please clear you Cloudflare caches if you’re still seeing this.
Thank you for the patience while we investigated this.
We’ll follow up with a more detailed explanation here soon.
Hey everyone, this was definitely an issue on our side. We were testing an internal change to our routing mechanism and returned a testing message for some of the requests. Unfortunately, in a subset of CDN locations, for customers on our regular network (not HP Edge) that have a proxy in front of Netlify, those test responses were routed through their proxy. That meant that these test responses were cached there and served instead of the corresponding asset.
We reverted the change as soon as it was reported. Sorry for the inconvenience this caused.
