Split-tested site intermittent loading issues

Hey all -

Throwing this out there in case anyone has thoughts on how this intermittent issue we’re experiencing.

We’re running a split test on our site, mixmax.com, and we’ve heard reports of loading issues. I managed to reproduce the issue myself and take this screenshot:
unnamed

(Unfortunately I didn’t grab more information than that screenshot. I wish I grabbed headers or a HAR file - sorry!)

What appears to be happening is that my browser is not respecting the Netlify A/B test cookie (“nf_ab”) on first page load, then loading page assets which are being returned 50/50 from the split test sites. So a random 50% of them were failing.

I can reproduce this issue 100% of the time by turning off cookies in the my browser. Though I’m certain I had cookies enabled when I took the screenshot of the issue. Our users that reported the same issue likely also had cookies enabled.

Has anyone seen an issue like this? This could either be a bug in Netlify’s split test implementation - in that it doesn’t always return the nf_ab cookie - or it could just be a Chrome bug causing it to ignore cookies.

A question for the Netlify team: is there a way I can make our split-test robust to browsers without cookies enabled? Shouldn’t the absence of a cookie in a request for anything other than a page route always return the asset?

Hi, @bradvogel, I also replied to you about this outside of the community site.

The HAR file would be helpful and tell us more. This site send preload headers like so:

link: </webpack-runtime-8953c9ceddded8545c76.js>; rel=preload; as=script, </framework-00aa9e2d91dcd9b04ab1.js>; rel=preload; as=script, </532a2f07-e9b47ba81f9833c05b06.js>; rel=preload; as=script, </app-89868972a156a87a7386.js>; rel=preload; as=script, </b094a35c0f5ce1c2657e2bdae0126f160705e04a-424e0cd32bda7de8d013.js>; rel=preload; as=script, </e8fa3c24b67d719095e66040e1889a76bbbd3251-e73600c803196b886db5.js>; rel=preload; as=script, </2300d3db9ed66f982fe9bbee5bab459ce3cfff0d-8d7176af8a7e0e81b116.js>; rel=preload; as=script, </component---src-templates-index-page-js-6827f22ab90b5f6286b5.js>; rel=preload; as=script, </page-data/app-data.json>; rel=preload; as=fetch; crossorigin, </page-data/index/page-data.json>; rel=preload; as=fetch; crossorigin

The 404s in the screenshot match those headers.

If this is reproducible with split testing enabled it would help us to enable the split testing again so we can reproduce this and find the root cause.

Hi!

We seem to be experiencing similar issues on our split tested website. I did some investigation and all individual branch deployments are working fine. However, some assets are 404’ing on the split test production URL. Some times the website does load but the issue happens more often than not. At least 3 of our developers have verified this, even after several attempts of completely cleaning up website data + cache + reloading.

We are experimenting with 8 split test branches, which we added through the API. These are in the order they show up in the UI and I’m including their % and what I assume to be the nf_ab range:

  • master - 12% (0 to 0.12)
  • st-test-1 - 13% (0.12 to 0.25)
  • st-test-2 - 13% (0.25 to 0.38)
  • st-test-1__st-test-2 - 13% (0.38 to 0.51)
  • st-test-3 - 13% (0.51 to 0.64)
  • st-test-1__st-test-3 - 12% (0.64 to 0.76)
  • st-test-2__st-test-3 - 12% (0.76 to 0.88)
  • st-test-1__st-test-2__st-test-3 - 12% (0.88 to 1)

Currently, nf_ab for my requests is 0.749593, so I’m expecting the st-test-1__st-test-3 version. However, I’m getting index.html from st-test-2__st-test-3. Maybe the considered order of the branches under the hood is different from the one in the UI?

The issue, though, is that the link /_next/static/QY9KoEpC8IH7nGrZMM1io/_ssgManifest.js found in the index.html file from st-test-2__st-test-3 404s. The actual link that seems to work is /_next/static/kdWoPsInizKhoaMDihJss/_ssgManifest.js, which is from branch st-test-1__st-test-2__st-test-3. There are some more JS assets that 404 too.

Looks like even though the nf_ab remains the same for both requests, assets from different branches are being served. Has anyone else noticed this happening before? Could I be missing something important that is leading to this?

Hi, @nunomluz, I did confirm that the split testing isn’t in a predictable order. The precentages are correct but they are not broken down into the segments above (the order is different). (It looks like the production branch is always last instead of first.)

I also see there is a support ticket open for this with more details and I replied with the current split test splits there.

If we get a general solution in the support ticket (something we can share here without sharing private information), we will follow-up and post that solution here.

Please feel free to reply either place but I recommend replying to the support ticket for now.