Since this morning we noticed that query params in redirects are suddenly beeing cached. Without any code changes nor environment changes or Netlify settings. This looks like a major issue since we pass personal tokens in the query params.
?tracecode=5678&zipcode=5678 are query params we used before.
netlify.toml:
[[redirects]]
from = “/path1/path2”
to = “/path3/”
status = 302
force = true # COMMENT: ensure that we always redirect
headers = {X-From = “Netlify”}
We noticed the same problem since today where url with query string are globally cached. On our side that means that users where able to access the content of other users because of the redirection.
Attached you can find an example of such a redirect that happened (the redirect change the parameter, especially in this case individualId and documentId
[[headers]]
# Define which paths this specific [[headers]] block will cover.
for = "/static/registry.json"
[headers.values]
Access-Control-Allow-Origin = "*"
[[headers]]
for = "/?q*"
[headers.values]
cache-control = '''
max-age=0,
no-cache,
no-store,
must-revalidate'''
We added the cache invalidation on /?q to try to fix the issue, unsuccessfully. We have no redirect rules
We are experiencing same issue, with a large user base receiving personalized data via individual user tokens. Because of forced redirects, user-data was breached this AM, with all system users being directed to other individuals’ dashboards, which are served by API to an SSG using a token. Help me understand the rationale in such an aggressive caching/redirecting behaviour being released into stable environments.
Netlify, what have you done? ! The most basic promise of the internet is that I type in a link and you send me to that link.By failing to keep that promise, you have broken systems which rely on individual user tokens. You have leaked many peoples’ data. Today, I type in a link with my token, and I end up seeing someone else’s data. It feels like a betrayal, to be honest. You’ve broken a system for me which many people worked very hard on for a very long time. With no warning, for no good reason. @marcus
hey @f3rg! thanks for sharing that. We appreciate it - we are aware of some of the issues you are describing and currently working on it. Not the experience we want you to have, for sure. Stay tuned, we’ll update here as soon as we have news.
@perry Perry, Imagine a scenario where a hypothetical technology company rolled out a hypothetical COVID19 screening tool to a hypothetical large factory where hypothetical employees had to be screened daily or were refused access to the factory. Your reply is helpful and conveys good intentions, but I want to ensure you understand the magnitude of the problem.
Here’s what we are doing:
We are actively looking in to this internally - i see some of our most knowledgeable folks on the case - and we will present options to you as soon as we have them.
In the meantime, I have some information from one of our devs that could be a short term mitigation. is there any way you shift to not using the apex domain, but the www subdomain for sensitive query params so they will not be redirected?
thanks for your patience - again, this isn’t the experience we are hoping for and we are working to mitigate.
a quick update that we are making some progress, but are not in the all clear yet. I or someone from my team will circle back here when we have something specific to share.
All of our activation emails have something like this “https://app.ourdomain.com/?access_token=abcdefg”
Our users are reporting when they click this link that they are logging in as other users. I assume this is the same issue reported above. We likely need to bring our entire company site down until this is fixed. Is there any update?
Yup, that does sound like the same thing, @Enalmada. Check out my next update in this thread for current status and next steps in case you are still affected.
I understand this was a highly impactful issue to the folks in this thread and a few others we had heard from. I am personally deeply sorry for this impact, and I do speak for Netlify on this as well.
Our engineers have found the cause and repaired it in our codebase and deployed that fix, and have started a process to repair our CDN’s cache for all impacted sites. We are working on sites from paid customers first, and then working our way to the free ones. We expect all paid sites to be fixed by tomorrow, and the free ones will likely not complete until sometime tomorrow or even next week.
However, you can self-serve on the fix already: redeploying your affected site(s) will clear its cache on the CDN, so as soon as you do so, the problem should no longer occur.
We’d love to hear from you as to whether the fix has worked for you (or perhaps more actionably, if it hasn’t, and at what example URL or x-nf-request-id in case your URL’s are sensitive or unshareable publicly (we can find the URL from that, but an attacker could not).
@f3rg I do hear you loud and clear that this was a breach of trust. We will be doing an internal retrospective call about the causes and how to prevent them from happening in the future, and will report back here. I don’t believe anything we say can rebuild trust, but I do believe that things we do - improving testing, discovering the full nature of the root cause of both the problem and missing it during our (extensive!) pre-testing of the feature - are one step in that process, as is the transparency we’ll provide here. You of course have to make the call about whether any of this is enough for you, but as your advocate to the business, my promise is that I am doing all that I can to see those actions implemented and reporting back here as to the progress.
Just wanted to report back to this thread that our cache repair for all free customers without a custom domain who have not deployed in the past 6 days (the last group to complete, millions of sites) was just completed.
So, nobody should see any leakage anymore, but I think everyone in this thread had probably redeployed already and was no longer affected.
We are a paid Netlify customer and we experienced a major security issue during the past week where users were randomly and intermittently logged in as other users.
After thorough investigation, we were able to pinpoint the issue to Netlify’s cached URL parameters for redirects.
We don’t have any custom redirects set up in netlify.toml.
The issue was 301 redirects for routes like:
/route-name?token=<token>
301 redirect to: /route-name/?token=<another_token_cached_by_netlify>
Notice the extra trailing slash and the wrong parameter value.
This redirect was made automatically by Netlify and contained cached sensitive data in the URL params, resulting in users accessing resources they should never have accessed.
We first saw this happen on Wednesday 13th, having not seen it previously (over 4 months). Then it began to happen to several users and we were able to replicate it both in browsers and in basic http requests in command line scripts.
To mitigate this, we disabled the Pretty URLs option in Site settings > Build & deploy > Post processing > Asset optimization as described here Redirect options | Netlify Docs
Also, as a precautionary measure, to address the security risk, we disabled the login feature that uses URL params in our own code.
Because of this, certain features of our app are turned off as we’re reluctant to enable it again and run into the initial problem (users logging in as other users)
Could you please clarify:
Whether you’re aware of this issue, and how different it is from the scenario described above, as we don’t have any custom redirects setup?
With the Pretty URLs option disabled, are we guaranteed the issue wont happen again?
