It seems that we are affected by this as well, it seems that last Friday everything was working well, but today i.e. 18th of October we experienced security issue - users are randomly logged as other users, we believe it is connected with cached query params. Can you suggest any workarounds or ideally provide a quick fix for us.
@fool@perry - this issue is still ongoing for us us across multiple production sites in our account. Can you please confirm that the fix has propagated across the full network? And/or have the team purge the CDN cache for our account?
Thanks for fast response. I’m testing it right now, but it seems I still can see our sensitive redirects are served from your cache i.e. I see the 301 Http Status for my calls. So I don’t know what your fix was and what we should expect. But even though the response is cached - it contains correct data … at least for know, I’ll keep testing.
PS. What was the fix on your side? Should we expect those 301 at all ?
@delpiersos While we fixed this going forward, reports like yours help us ensure the fix has reached your sites, which I’ve just done. Let me know in case you see any wrong data being returned starting about 3 minutes ago, please!
Hi folks, just wanted to close the loop on the cause of and resolution to this situation.
Between approximately 1600 UTC on 11 Oct and 23:06 UTC on 18 Oct, a feature on our CDN that normalizes URL’s (from http://site.com/path to http://site.com/path/) in some situations cached Location HTTP response headers and then later served them to incorrect customers, causing wrong 301 redirects that caused some of your visitors to see incorrect content intended for prior visitors.
The bug was caused by a server side misconfiguration, and was fixed by reconfiguring. To prevent recurrence of the same class of problem, we have already reviewed all current configuration and started work on automating prevention of that type of configuration problem, but also:
Streamlining our Severity 1 Incident response runbooks and communication workflows for customers on impactful issues like these,
Reviewing code around our interacting systems that handle redirects,
and contributing to increase our debug information to help us troubleshoot complicated issues like these.
I posted yesterday about this issue recurring again - for such a serious issue that may impact every single Netlify customer I would have expected you may respond reasonably promptly. It’s been 18 hours and no advice on this.
Site IDs we’ve been witnessing this reoccur on, include:
There are no error messages being provided by Netlify, it’s simply caching query parameters and serving them to other people. The errors are all occurring in our backend application when someone (unknowingly) attempts to perform an action with a query param that’s not theirs.