So sorry to hear about the trouble! We’ve unlisted this thread so that you can tell us a bit more about it so we can research:
what’s the sitename?
what’s an example URL?
when was this behavior observed, and if only recently, how long had it been since your site layout/mechanisms for delivering these redirects had changed before then?
I’ve also cc:ed our security team on the investigation since I do recognize that it is a major issue for you.
Thanks so muchf
Our team has rolled out a likely fix and I’ve purged our CDN cache for your account. Could you let me know if you can still reproduce the problem please, @Halimb ?
Hi,
It seems that we are affected by this as well, it seems that last Friday everything was working well, but today i.e. 18th of October we experienced security issue - users are randomly logged as other users, we believe it is connected with cached query params. Can you suggest any workarounds or ideally provide a quick fix for us.
The behavior was observed since Wednesday 13th, we had never experienced it before, and there were no changes on our part to the site layout / mechanisms involved in the bug since over 4 months.
@fool@perry - this issue is still ongoing for us us across multiple production sites in our account. Can you please confirm that the fix has propagated across the full network? And/or have the team purge the CDN cache for our account?
Thanks for fast response. I’m testing it right now, but it seems I still can see our sensitive redirects are served from your cache i.e. I see the 301 Http Status for my calls. So I don’t know what your fix was and what we should expect. But even though the response is cached - it contains correct data … at least for know, I’ll keep testing.
PS. What was the fix on your side? Should we expect those 301 at all ?
Hiya @kwalczak in general, 301 redirects are not necessarily problems; we do intentionally send some 301 redirects in many situations. We are only worried about incorrect redirects in this thread, so do let me know if you have an example of “visit URL with query params x and get a 301 redirect that I didn’t configure, to query params y which would have been for another user”, please send an x-nf-request-id (cf [Support Guide] Netlify Support asked for the 'x-nf-request-id' header? What is it and how do I find it?) so we can best investigate.
@delpiersos While we fixed this going forward, reports like yours help us ensure the fix has reached your sites, which I’ve just done. Let me know in case you see any wrong data being returned starting about 3 minutes ago, please!
Hi folks, just wanted to close the loop on the cause of and resolution to this situation.
Between approximately 1600 UTC on 11 Oct and 23:06 UTC on 18 Oct, a feature on our CDN that normalizes URL’s (from http://site.com/path to Login | HSTS Redirection Community) in some situations cached Location HTTP response headers and then later served them to incorrect customers, causing wrong 301 redirects that caused some of your visitors to see incorrect content intended for prior visitors.
The bug was caused by a server side misconfiguration, and was fixed by reconfiguring. To prevent recurrence of the same class of problem, we have already reviewed all current configuration and started work on automating prevention of that type of configuration problem, but also:
Streamlining our Severity 1 Incident response runbooks and communication workflows for customers on impactful issues like these,
Reviewing code around our interacting systems that handle redirects,
and contributing to increase our debug information to help us troubleshoot complicated issues like these.
@fool@perry
I posted yesterday about this issue recurring again - for such a serious issue that may impact every single Netlify customer I would have expected you may respond reasonably promptly. It’s been 18 hours and no advice on this.
Thanks for reaching out and thanks for your patience. We did have an incident on February 21st impacting redirects, but all fixes have been deployed. We do not believe that your regression is related to that, but you can read more here: Netlify Status - Missing redirects and headers in some deploys.
If you are still encountering obstacles please respond with your site IDs if they are not the same as the ones you have previously shared and any error messages you have received.
Site IDs we’ve been witnessing this reoccur on, include:
4a9a9e2b-3f58-4b06-83db-02e3e9a08941
dc443c16-ea76-4a9d-92e4-6fe844c46cfb
There are no error messages being provided by Netlify, it’s simply caching query parameters and serving them to other people. The errors are all occurring in our backend application when someone (unknowingly) attempts to perform an action with a query param that’s not theirs.