Query string cached when redirecting

Support
27

So sorry to hear about the trouble! We’ve unlisted this thread so that you can tell us a bit more about it so we can research:

  1. what’s the sitename?
  2. what’s an example URL?
  3. when was this behavior observed, and if only recently, how long had it been since your site layout/mechanisms for delivering these redirects had changed before then?

I’ve also cc:ed our security team on the investigation since I do recognize that it is a major issue for you.
Thanks so muchf

1 Like
29

Our team has rolled out a likely fix and I’ve purged our CDN cache for your account. Could you let me know if you can still reproduce the problem please, @Halimb ?

30

Hi,
It seems that we are affected by this as well, it seems that last Friday everything was working well, but today i.e. 18th of October we experienced security issue - users are randomly logged as other users, we believe it is connected with cached query params. Can you suggest any workarounds or ideally provide a quick fix for us.

Regrds

31

hi @kwalczak , can you tell me your site name please? or API ID? thank you!

32

Hi,
Please find the API ID here 3cc0f2e1-bb9e-4d09-bd72-b18fc2ec16e8

Regards

33

thank you, @kwalczak - we have rolled out a fix, are you still experiencing this issue?

34

If you are still experiencing it, @kwalczak - please give us a URL that shows the problem so we can investigate further.

35

Hi @fool

Thanks for the quick reply.

  1. Here are the site details:

Name: evercam-dashboard
API ID: eb79402c-ee56-4a53-bde4-153f83e09868

  1. Example URL
    from : dash.evercam.io/v2/cameras
    301 to: dash.evercam.io/v2/cameras/?api_id=6eeb77cc&api_key=<API_KEY>

  2. The behavior was observed since Wednesday 13th, we had never experienced it before, and there were no changes on our part to the site layout / mechanisms involved in the bug since over 4 months.

Thanks

36

Thanks for those details! However, I was attempting to say in my last post that “I think we’ve fixed it - can you verify our fix?” @Halimb ?

37

@fool @perry - this issue is still ongoing for us us across multiple production sites in our account. Can you please confirm that the fix has propagated across the full network? And/or have the team purge the CDN cache for our account?

app_id: cd7f8ab6-eaec-4f37-a683-75c30770609d
app_id: bc8eaa6a-b99b-4f64-823e-39dcfb138c9d
app_id: 44ea52cd-6bb6-466c-b794-28fa475f846f
app_id: a684fdf2-622f-40fe-bb53-3192346d2dd1
app_id: c8d1382a-e93f-4cc9-be5c-33a3f294501b
app_id: 931dfd32-8af3-4de1-8443-4b57297841f8
app_id: 9793f105-f3a3-4237-8976-e4f0808d95db
app_id: 2ce95c18-6aee-4587-be66-fd49d7cde804
app_id: 1a930c9d-fe27-4fa4-84f7-e3bfb7559bd8
app_id: 2ccded0a-b365-483c-af26-b4ca11d613af
app_id: a0b0749d-e9cb-49f2-a7f6-7de694069d86
app_id: a07129eb-ee9e-4922-bd3c-d51f35e243e7
app_id: 79021407-57d5-4ca4-a8d8-28ba84499013
app_id: a18f7ec6-6b6d-451a-aa42-3d7e2d595f5d
app_id: 5167e6b7-6138-44f5-b293-506ba8049540
app_id: 88fba77e-c1e6-489d-a70f-5c3b92ce2229
app_id: dabae28a-d81a-4c90-adf4-52580f2e227d
app_id: 01fa6524-68cc-41fc-9776-4c53b1069f03
app_id: bcebf05e-1f7c-44c3-b00f-256cedbdbc17

38

Hi @fool , @perry,

Thanks for fast response. I’m testing it right now, but it seems I still can see our sensitive redirects are served from your cache i.e. I see the 301 Http Status for my calls. So I don’t know what your fix was and what we should expect. But even though the response is cached - it contains correct data … at least for know, I’ll keep testing.

PS. What was the fix on your side? Should we expect those 301 at all ?

Regards
Karol W.

39

Hiya @kwalczak in general, 301 redirects are not necessarily problems; we do intentionally send some 301 redirects in many situations. We are only worried about incorrect redirects in this thread, so do let me know if you have an example of “visit URL with query params x and get a 301 redirect that I didn’t configure, to query params y which would have been for another user”, please send an x-nf-request-id (cf [Support Guide] Netlify Support asked for the 'x-nf-request-id' header? What is it and how do I find it?) so we can best investigate.

@delpiersos While we fixed this going forward, reports like yours help us ensure the fix has reached your sites, which I’ve just done. Let me know in case you see any wrong data being returned starting about 3 minutes ago, please!

40

Hi folks, just wanted to close the loop on the cause of and resolution to this situation.

Between approximately 1600 UTC on 11 Oct and 23:06 UTC on 18 Oct, a feature on our CDN that normalizes URL’s (from http://site.com/path to Login | HSTS Redirection Community) in some situations cached Location HTTP response headers and then later served them to incorrect customers, causing wrong 301 redirects that caused some of your visitors to see incorrect content intended for prior visitors.

The bug was caused by a server side misconfiguration, and was fixed by reconfiguring. To prevent recurrence of the same class of problem, we have already reviewed all current configuration and started work on automating prevention of that type of configuration problem, but also:

  • Streamlining our Severity 1 Incident response runbooks and communication workflows for customers on impactful issues like these,
  • Reviewing code around our interacting systems that handle redirects,
  • and contributing to increase our debug information to help us troubleshoot complicated issues like these.

Let me know if you have any questions!

41

Hi @fool ,
Unfortunately this appears to be occurring again today.
Has there been a regression?

42

@fool @perry
I posted yesterday about this issue recurring again - for such a serious issue that may impact every single Netlify customer I would have expected you may respond reasonably promptly. It’s been 18 hours and no advice on this.

43

Hey there, @delpiersos :wave:

Thanks for reaching out and thanks for your patience. We did have an incident on February 21st impacting redirects, but all fixes have been deployed. We do not believe that your regression is related to that, but you can read more here: Netlify Status - Missing redirects and headers in some deploys.

If you are still encountering obstacles please respond with your site IDs if they are not the same as the ones you have previously shared and any error messages you have received.

44

Hi Hillary,

Site IDs we’ve been witnessing this reoccur on, include:

  • 4a9a9e2b-3f58-4b06-83db-02e3e9a08941
  • dc443c16-ea76-4a9d-92e4-6fe844c46cfb

There are no error messages being provided by Netlify, it’s simply caching query parameters and serving them to other people. The errors are all occurring in our backend application when someone (unknowingly) attempts to perform an action with a query param that’s not theirs.

45

hi there delpiersos, sorry for the delay. are you still experiencing the errors?