Issue with opt-in split testing

Support
1

Problem

I am trying to implement opt-in split testing for a soft launch of new gatsby + shopify site between two branches: “production” and “old-site-redirect”, but it’s not working very consistently. There are times I’m able to switch between branches back-to-back, but there are times where I get locked to one branch. Sometimes I get varying results between browsers, and I made sure to test in incognito/private mode. Also sometimes when the branch does switch, the subpaths will redirect back to other branch.

Details

The domain is sixthreezero-web.netlify.app.
The “production” branch points to the new gatsby site, and “old-site-redirect” points to a branch with a redirect that proxies back to the original Shopify site:

[[redirects]]
   from = "/*"
   to = "https://www.sixthreezero.com/:splat"
   status = 200
   force = true

Note: The to property is going to point to the original shopify domain at launch.

The split test is set to production = 1%, old-site-redirect = 99%.
I would set it 0 / 100 but the UI removes the branch if it is set to 0%.

I referenced this article to implement opt-in:

Testing Process
I run the following code in the console to test switching branches:
To production:

document.cookie = "nf_ab=production; expires=Thu, 06 May 2021 00:00:00 GMT; path=/; SameSite=Strict";
window.location.reload(true);

To old-site-redirect:

document.cookie = "nf_ab=old-site-redirect; expires=Thu, 06 May 2021 00:00:00 GMT; path=/; SameSite=Strict";
window.location.reload(true);

I initially tested without the extra cookie options but ended up adding more and more to see what sticks.

2

I wouldn’t expect that to work well. The feature was designed around testing between two branches of a single netlify site, not between a branch and a wholesale proxy to another service.

This is one of the reasons the feature is marked as beta - we haven’t put the final polish on it, and don’t have docs on the potential failure modes, but proxying away from our service is definitely a major likelihood for misbehavior like you’re experiencing. (versus a rewrite within the site - e.g. /articles /blog 200! or even potentially between netlify sites would not be a problem; put another way, external proxying relies on a caching layer that is not ab-test aware).

I’d suggest not using our split testing for this workflow.

2 Likes
3

Thanks I’ll look into other solutions

1 Like