Redirect rules from months no longer parsed the same way?

[fivetwelve]

[Updated the post to be more brief]

We’re experiencing some redirect issues at our site, https://www.almex.com, both it and our dev branch deployments. The redirect rules were consistently the same and working since last year and issues showed up around mid-January.

I’d appreciate any comments about my fix as well as any insight to why the rules are behaving differently.

The problem:
Our redirect URL is based on the visitor’s geography. Visiting almex.com/ would take you to almex.com/northamerica/en, example.

When the site tries to access any path starting at root, those requests are being rewritten as well, resulting understandably in 404’s. e.g. For reference our tech stack uses Gatsby for the site build. So attempts to load /page-data/northamerica/en/[someproduct]/page-data.json, redirect to /northamerica/en/page-data/northamerica/en/[someproduct]/page-data.json. This happens very often but not 100% of the time.

Pre-existing rules:

/*  /northamerica/es/:splat  302 Country=ca,us Language=es
/*  /northamerica/en/:splat  302 Country=ca,us

# Redirect customers for South Africa
/*  /southafrica/en/:splat  302 Country=ao,bw,cm,cg,cd,ke,mg,mw,mu,mz,na,za,tz,ug,zm,zw 

# Redirect customers for Asia Pacific
/*  /australia/en/:splat  302 Country=au,kh,ck,fj,pf,jp,kp,kr,la,my,mn,mm,nr,nc,nz,pg,ph,sg,sb,tw,th,vn

[similar country rules removed for brevity, last rule follows]

# Fallback redirect failing all of the above, countries not listed
/*  /northamerica/en/:splat  301

My interim fix:
I removed the wildcard/splat. A redirect still happens correctly at the root domain-only and prevents rewrites of root-based references. I have a temp deployment at https://604793130b8a77e853cfe7ac--almex-group.netlify.app/ with these changes:

/  /northamerica/es/  302 Country=ca,us Language=es
/  /northamerica/en/  302 Country=ca,us
[etc...]
/  /northamerica/en/  301

We lose the ability to have additional helpful redirects but at least the errors are prevented. I think my boss might be okay with that.

So I can understand why a wildcard could be causing those unexpected redirects but shouldn’t the rules only apply to external HTTP requests and not those originating from the same domain?

Were my rules set up incorrectly to begin with? Has there been a change in parsing behind the scenes? A combination of the two?

[luke]

Hi, @fivetwelve. We need some way to see what you are seeing. For example, the HAR file or a known URL and deploy where this can be seen.

Would it be possible to make a HAR file recording of the issue happening?

Likewise, you mentioned this:

@fivetwelve

For reference our tech stack uses Gatsby for the site build. So attempts to load /page-data/northamerica/en/[someproduct]/page-data.json, redirect to /northamerica/en/page-data/northamerica/en/[someproduct]/page-data.json. This happens very often but not 100% of the time.

Do you happen to have URLs where that is known to happen? Is it happening for the production deploy? If not, do you have the deploy id for the deploy where it does happen (and the path that trigger it)?

If we can see the real redirects occurring we can tell you which rules are causing it and how to change the rules to prevent it.

[fivetwelve]

Thanks for the reply, Luke. I have not yet propagated my fix to the live site so, yes, it can be seen at the moment at https://www.almex.com/northamerica/en. Update: we may have to update prod sooner than later but here’s a past preview build (Feb 17th) that exhibits the issue: https://602d72a49cf0b2452f1b10fc--almex-group.netlify.app/

Network inspector shows examples of unexpected/incorrect redirects.

Here’s an example:

I guess it could subscribe to the /* → /northamerica/en/:splat rule but this goes back to what I was wondering about this being an internal request [Gatsby] vs a visitor’s and the biggest question is why has this started happening recently?

The error is a result of the rule but I’m not sure exactly whether the rule is at fault or the parsing engine behind them. I appreciate any insight!

—Victor

[luke]

Hi, @fivetwelve. I can see that the 301 redirect on that URL did happen. I can even see which deploy way published at the time. I tried reproducing the issue with the exact same path (/page-data/northamerica/en/bat-jt1-002/page-data.json) and the same deploy for the x-nf-request-id id in the screenshot (deploy 60461fe1e178c200d4ca275e and x-nf-request-id: 3fde6010-7600-4c16-8a66-4d665a6a496b-29069972).

However, I cannot trigger the same issue when I test:

$ curl --compressed  -svo /dev/null https://60461fe1e178c200d4ca275e--almex-group.netlify.app/page-data/northamerica/en/bat-jt1-002/page-data.json  2>&1 | egrep '^<'
< HTTP/2 200
< accept-ranges: bytes
< cache-control: public, max-age=0, must-revalidate
< content-type: application/json
< date: Fri, 12 Mar 2021 10:10:21 GMT
< etag: "1fb70103b96fac170cf18aaa524aba15-ssl-df"
< referrer-policy: same-origin
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-robots-tag: noindex
< x-xss-protection: 1; mode=block
< content-encoding: gzip
< age: 0
< server: Netlify
< vary: Accept-Encoding
< x-nf-request-id: bef3b44a-0b8a-40fa-8c54-edc45a0b55a2-4323792
<

Again, though, our logs show the same as your screenshot. It definitely did happen at the time.

I’m also not seeing the issue with the deploy you sent:

$ curl -svo /dev/null https://602d72a49cf0b2452f1b10fc--almex-group.netlify.app/page-data/northamerica/en/bat-jt1-002/page-data.json  2>&1 | egrep '^<'
< HTTP/2 200
< accept-ranges: bytes
< cache-control: public, max-age=0, must-revalidate
< content-length: 22412
< content-type: application/json
< date: Fri, 12 Mar 2021 10:17:04 GMT
< etag: "1fb70103b96fac170cf18aaa524aba15-ssl"
< referrer-policy: same-origin
< strict-transport-security: max-age=31536000; includeSubDomains; preload
< x-content-type-options: nosniff
< x-frame-options: DENY
< x-robots-tag: noindex
< x-xss-protection: 1; mode=block
< age: 0
< server: Netlify
< x-nf-request-id: e7f90e0e-df53-4e4d-8186-d7efee5b01a7-16987235
<

So, I know it did happen but not why and I cannot reproduce it now. Are you still seeing this behavior?

[fivetwelve]

@luke Thanks for the follow-up. Yes, our prod deployment is working only because I used the stripped-down version of the _redirects without splats. The issue remains to some degree.

I’ve tested the Feb17th deployment today and I don’t see the same errors as I reported. However, I tested a different build https://603e71dd98d7e61af1d0ff36--almex-group.netlify.app/ that still shows the issue. I see similar 404s appearing in both Chrome and Firefox (MacOS) browsers. The _redirects rules are a little different because my colleague added some rules trying to resolve/bypass the page-data rewrites & others. Nonetheless, they didn’t appear to have any impact.

Ideally we’d like for the original splat rules to work properly. We’d like to be able to advertise a single URL for example, www.almex.com/oem-services and have int’l customers get redirected to their designated path.

Thanks,
Victor

Branch Deploy: dev@HEAD. Mar 2 at 12:11 PM (should actually be 10:11AM EST)

_redirect contents:

# REDIRECT RULES FOR LIVE & DEV DEPLOYMENT
# ref https://docs.netlify.com/routing/redirects/redirect-options#redirect-by-country-or-language
# discussion https://community.netlify.com/t/redirect-not-fully-working-in-spite-of-following-redirects-documentation/16730

#images
/img/* /img/:splat 200!

#static
/static/* /static/:splat 200!

# Redirect customers for North America
/page-data/northamerica/es/* /page-data/northamerica/es/:splat 200!
/page-data/northamerica/en/* /page-data/northamerica/en/:splat 200!
/* /northamerica/es/:splat 302 Country=ca,us Language=es
/* /northamerica/en/:splat 302 Country=ca,us

# Redirect customers for South Africa
/page-data/southafrica/en/* /page-data/southafrica/en/:splat 200!
/*  /southafrica/en/:splat 302 Country=ao,bw,cm,cg,cd,ke,mg,mw,mu,mz,na,za,tz,ug,zm,zw 

# Redirect customers for Shaw Almex Pacific
/page-data/australia/en/* /page-data/australia/en/:splat 200!
/*  /australia/en/:splat 302 Country=au,kh,ck,fj,pf,jp,kp,kr,la,my,mn,mm,nr,nc,nz,pg,ph,sg,sb,tw,th,vn

# Redirect customers for Europe
/page-data/europe/de/* /page-data/europe/de/:splat 200!
/page-data/europe/en/* /page-data/europe/en/:splat 200!
/*  /europe/de/:splat 302 Country=af,dz,am,at,bh,be,ba,bg,bf,es,hr,cy,cz,dk,eg,gb,er,ee,et,fo,fi,fr,ga,ge,de,gh,gr,gl,gg,gn,hu,is,ir,iq,ie,im,il,it,ci,je,jo,kz,kw,kg,lv,lb,lr,ly,li,lt,lu,mk,ml,mt,mr,md,me,ma,np,nl,ng,gb,no,om,pk,pl,pt,qa,ro,ru,sa,gb,sn,rs,sl,sk,si,es,sd,se,ch,sy,tj,tn,tr,tm,ua,ae,gb,uz,gb,ye,mc Language=de
/*  /europe/en/:splat 302 Country=af,dz,am,at,bh,be,ba,bg,bf,es,hr,cy,cz,dk,eg,gb,er,ee,et,fo,fi,fr,ga,ge,de,gh,gr,gl,gg,gn,hu,is,ir,iq,ie,im,il,it,ci,je,jo,kz,kw,kg,lv,lb,lr,ly,li,lt,lu,mk,ml,mt,mr,md,me,ma,np,nl,ng,gb,no,om,pk,pl,pt,qa,ro,ru,sa,gb,sn,rs,sl,sk,si,es,sd,se,ch,sy,tj,tn,tr,tm,ua,ae,gb,uz,gb,ye,mc

# Redirect customers for Chile
/page-data/chile/es/* /page-data/chile/es/:splat 200!
/page-data/chile/en/* /page-data/chile/en/:splat 200!
/*  /chile/es/:splat 302 Country=ar,cl,py,uy Language=es
/*  /chile/en/:splat 302 Country=ar,cl,py,uy

# Redirect customers for China
/page-data/china/en/* /page-data/china/en/:splat 200!
/*  /china/en/:splat 302 Country=cn,hk

# Redirect customers for Asia Pacific
/page-data/asiapacific/en/* /page-data/asiapacific/en/:splat 200!
/*  /asiapacific/en/:splat 302 Country=kp,kr,kh,jp,la,my,mn,mm,nr,nz,pg,ph,sh,tw,th,vn,kh

# Redirect customers for Peru
/page-data/peru/en/* /page-data/peru/en/:splat 200!
/*  /peru/en/:splat 302 Country=pe

# Redirect customers for Brazil
/page-data/brasil/en/* /page-data/brasil/en/:splat 200!
/*  /brasil/en/:splat 302 Country=br

# Redirect customers for India
/page-data/india/en/* /page-data/india/en/:splat 200!
/*  /india/en/:splat 302 Country=in

# Redirect customers for Indonesia
/page-data/indonesia/en/* /page-data/indonesia/en/:splat 200!
/*  /indonesia/en/:splat 302 Country=id,mo

# Redirect customers for Mexico
/page-data/mexico/en/* /page-data/mexico/en/:splat 200!
/*  /mexico/en/:splat 302 Country=mx


# Fallback redirect failing all of the above
/*  /northamerica/en/:splat  301

[Scott]

By any chance have you recently moved to a version of Gatsby greater than v2.32.0?

No changes on our end but I know that there have been several issues with redirects since upgrading beyond this version, e.g. createRedirect broke gatsby-plugin-netlify redirects on 2.32.7 · Issue #29779 · gatsbyjs/gatsby · GitHub

If not, I’d like some clarification on all of the above. There’s a lot of information but all I need is:

• a deploy ID where you’re seeing the misbehaving redirect
• an example of a URL which is misbehaving
• what you expect the redirect to do, if behaving

Cheers!

[fivetwelve]

Thanks for the reply, @Scott, and I appreciate you digging further into the issue.

At the time of the deploy mentioned further down,
"gatsby": "^2.25.3"
"gatsby-plugin-netlify": "^2.3.5"

• deploy ID, if I understand your request, is 603e71dd98d7e61af1d0ff36. That’s also the prefix to the preview URL?
• the home page https://603e71dd98d7e61af1d0ff36--almex-group.netlify.app is incorrectly redirecting root-level files from within the site/application. Gatsby should be able to load things like site.webmanifest and favicon files without them getting redirected. Similarly, if I click on the text link in the carousel’s first tile, it takes me to https://603e71dd98d7e61af1d0ff36--almex-group.netlify.app/northamerica/en/bat-jt1-002 where it is supposed to load an associated file (I’m guessing chunked code) 8c351d9deedd55ab6e896526f2898f8de0bbf623-344c9f566f5c84b88444.js also from root but instead prepends the /northamerica/en redirect to it.

Also, in the link you mentioned, a fix was just merged in. Unfortunately, I think it’s a non-starter because the resulting version is not compatible with Gatsby versions < 3.x.

[Scott]

I understand. Let’s not focus on the Gatsby side of things because this isn’t related to that

So I can understand why a wildcard could be causing those unexpected redirects but shouldn’t the rules only apply to external HTTP requests and not those originating from the same domain?

In layman’s terms, a 301 is a redirect as opposed to a rewrite so a new request is made. It’s the last rule in your list which is causing the static asset route to be prefixed with /northamerica/en/.

Were my rules set up incorrectly to begin with? Has there been a change in parsing behind the scenes? A combination of the two?

No changes in this area behind-the-scenes for a long while though I wouldn’t say the rules are incorrect.

Three solutions:

• Sacrifice the 301 catch-all in favour of a 404 (/* /404 404)
• Move your assets to an /assets directory, and create a redirect rule above your catch-all which ensures that they are preserved (and don’t hit the catch-all – /assets/* /assets/* 200!)
• Duplicate your assets in to the /northamerica/en directory

[fivetwelve]

In this deployment, https://6054cf449656e0000835d9c7--almex-group.netlify.app/,

• I’ve updated gatsby and gatsby-plugin-netlify (that apparently has the applied patch mentioned in an aforementioned link) to their latest minor versions.
• I’ve cleared the build cache and retried a deployment
• I’ve changed the 301 catch-all to your 404 suggestion (no other changes to rules from my original)

I didn’t move or duplicate assets because the show-stoppers aren’t the favicon or site.webmanifest, it’s the chunked code that Gatsby is trying to load from root. Because we have 12 regions and more forthcoming, not just /northamerica/ it’s not a tenable solution to duplicate folders to every region and language permutation.

As before, Gatsby’s attempts to load data at root appear to be affected due to the redirect. The unusual thing I see in the network inspector is that the 404s aren’t occurring for every request. They vary between 200, 302 and 404. Not sure if Gatsby’s pre-fetching are running into issues with the redirects.

This is just an aside but to reiterate earlier, this was never an issue last year with the aforementioned rules we had in place. Because we use the country detection rules to redirect our different visitors appropriately our catch-all was to redirect users to /northamerica/en as our default for countries we fail/neglect to check against. The 404 causes us to lose this ‘feature’. My fix in prod (which is not in this deployment) would further causes us to lose splat redirects.

I think I’m a bit stuck as to how to continue to diagnose the root cause. The mix of 200/302/404 responses are leading me to wonder if Gatsby’s pre-fetch having issues with redirects or with the gatsby-plugin-netlify itself. I’m not great at figuring out middleware but I’m a bit at a loss as to where to look to.

As always, feedback and suggestions welcome.

Addendum: here’s the config I’ve been using with gatsby-plugin-netlify (hasn’t changed in 2 years):

{
  resolve: `gatsby-plugin-netlify`, // Always last in the plugins array
  options: {
    mergeSecurityHeaders: true, // boolean to turn off the default security headers
    mergeLinkHeaders: false, // boolean to turn off the default gatsby js headers (disabled by default, until gzip is fixed for server push)
    mergeCachingHeaders: true, // boolean to turn off the default caching headers
  },
},

[Scott]

Hey there!

I’ve looked at this several times now with fresh eyes and nothing seems out of the ordinary.

You’re requesting an asset at /component---src-templates-landing-template-js-a494bf53910e65708b72.js but you have country redirects in place so this will always become /[country/lang path]/component---src-templates-landing-template-js-a494bf53910e65708b72.js.

Best thing to do would be to push assets to an /assets directory, like /assets/component---src-templates-landing-template-js-a494bf53910e65708b72.js and then perform a proxy on it, on the first line of your redirects file:

/assets/* /assets/:splat 200!

[fivetwelve]

Thanks, Scott. I understand where you’re going with that suggestion but I’m not sure I’m yet up to reconfigure Gatsby’s webpack defaults. Not dismissing it completely though. Before I go down that road, I wanted to mention our team has an older build which has been running gatsby 2.25.3 (vs 2.32.9) and gatsby-plugin-netlify 2.3.5 (vs 2.11.1) over at https://content--almex-group.netlify.app/northamerica/en.

This build does not exhibit the weird redirect behaviours we see in the more recent & prod builds. It uses the same gatsby-plugin-netlify options as above and the same original redirect config [it’s saved as a different filename and copied as /public/_redirects at build time]. Inspector shows good 200 responses of JS files loading from root even with the redirect fallback present.

Any other avenues you might suggest I explore? I’m tempted to roll versions back but it’s not the best foot forward. Our plan was to take advantage of Gatsby caching in the more recent 2.x versions.

[Scott]

Interesting discovery! Regrettably, I don’t know a whole heap about the Gatsby/Netlify plugin or what options are available. It sounds like you were relying on client-side routes but there was a breaking change of sorts! This is where I’d be looking