I’m trying to use the A/B split testing feature to serve two different versions of my site’s homepage (Gatsby), each one with a different header.
To do that I’ve created two branch deploys and enabled split testing, it seemed to work correctly, the only problem was that netlify always serves the same version of the site to a specific user, but I don’t need that behaviour for my use case.
I saw that this works through a “nf_ab” cookie returned from netlify on the first call to the CDN, so i thought i could override this behaviour by just deleting the cookie on each page load.
Just to take a test i tried to delete the cookie manually and refresh the page, it works almost all the time, but sometimes it returns a page with mixed elements from the two deploys.
I’ve made some debugging, and I think the problem is that at least sometimes the “nf_ab” cookie is not returned from the first network call, the one that returns the html document, but from one of the following, so this happens:
User visits the page with no “nf_ab” cookie.
The html document is requested, the request is handled by one of the two sites deploys, let’s call it “A”, but no “nf_ab” cookie is returned with it.
The next requests are made, page data, javascript, etc, there is still no cookie so the request is again handled randomly by one of the two deploys, this time by the version “B”.
With the last request a cookie is returned with the value corresponding to the “B” version, so the following requests are made correctly.
This leads to having problems on the site, like having the header background from the “A” version and the text from the “B” one, etc.
Sorry to hear you are encountering some difficulties, and thanks for reaching out. Can you please share your site name and your repo with us? It would be helpful for us in order to look into this further.
I’m removing the “nf_ab” cookie on the window.onbeforeunload event, and as I was saying on the first post, it work most of the times, but sometimes it mixes the contents from the two sites.
For example in this case there is the version B title with the version A background image.
I suspect the reason is the one i explained in the first post, there is no “nf_ab” cookie in the response for the first request (Html document), so it’s getting data from two different deploys, but i am not sure about that, it’s just my guess.
Not sure if this is what you needed, let me know if it’s the correct file.
In this case the returned HTML is the one from the version B, but site data is from the version A.
I also see some 404 error while trying to get the JS files, not sure if they are related to this problem.
Yes, it’s related. It’s happening because your HTML file is loading from a deploy that has those files, but the JS files are returning from the deploy that doesn’t have those files.
About the actual issue, looking at that HAR file it appears that the content is mixed between the 2 deploys. We just recently had a very long discussion regarding the exact same issue with another user. While we’ve seen such an issue in the past, it’s really hard for us to reproduce or see the error. Even in your case, 4 of us tested and we all got the correct content served. This is probably a bug in Split Testing.
But there’s a workaround for this issue that seems to have fixed the issue for the other user. You need to:
Set up two sites, a site with Branch A and another one with Branch B.
On the first page load of Branch A, set a cookie with a (pseudo-)random probability. For example, split_test with any value like true .
After that, set up cookie-based redirects: Redirect options | Netlify Docs which would redirect all users with that cookie to the Branch B-based site.
We are currently having the same issue on our website and found a way to reproduce it.
Would it be possible to arrange a call to show you? It would be much easier.
Don’t hesitate to send me an email
Hi, @LionelPaulus. We have sent you an email (to the same email address use for this support forum).
If you receive that email, please reply there to share the reproduction steps with us. However, if you don’t see that email, please do reply here to let us know.
