Split test loads stale Deployment of branch

Support
1

Hi, at the moment I’m attempting to test two branches using a split test, and I’m always seeing an OUTDATED version of the production branch without being able to ever see the B version, despite seeing each respective branch’s updated versions when I deploy and publish individually with the split test disabled. This happened once before and resolved itself overnight somehow when I disabled it and re-enabled it the next day. I’m experiencing it again today so I’m raising the flag here.

Site: https://ecp-qa.netlify.app/

If I enable the split test, with prod branch “featureTest_develop” set to 100%, with “featureTest_deveop-b” set to 0%, I always see an outdated version of it from at least a few days ago.

This happened once before, and I disabled the split test and re-enabled it the next day, and then I got my updated branch.

My troubleshooting has included:

  1. clearing cache and deploying each updated version of the branch.
  2. swapping prod branch and other branch
  3. changing both branches to published, confirming the new versions are indeed the most recent versions.
  4. removing nf_ab manually from browser before enabling the split test
  5. manually setting nf_ab to either branch name (which has worked for the previous split test, no longer worked here)

At this point I’m at a loss of what to do besides give it time. Not sure if there’s something in the background that eventually enables split test to update to the most recent branch and adjust the nf_ab’s cookie value, but this is a blocker on immediately getting a test up and running.

UPDATE:
did nothing and tried to duplicate the steps above at the end of the work day yesterday, and it continued to not work.

Like I said, something happened overnight, and now when I enable the split test, it works as intended. There’s nothing in the documentation that says this is expected behavior.

Something’s either expiring or updating in that time. Not sure what.

Another update:
Without changing anything, we’re again seeing an old version of the prod branch with no a/b test functionality.

2

Hi @basjmontech,

It looks like split testing is currently off. We would need to see it in the broken state to test. Could you turn it back on, or create another site for us to test?

3

hi @hrishikesh, I attempted to create a new site (https://ecp-ab-test.netlify.app) but even after deploying the two separate branches I’m interested in AB testing, only one of them displays in available branches dropdown for the B test. I’m still only seeing old branches deployed. Maybe if I wait again over night it’ll fix itself?

image
image1256×446 22.7 KB

image
image1279×246 14.6 KB

image

As a side note, I did notice in the Netlify docs that split testing on netlify has a warning that it won’t work correctly with functions. I’m using nextjs out of the box w/ the netlify plugin, and it generates 3 serverless functions. Would this break split testing functionality?

4

hiya are you still experiencing this ?

5

Hi @SamO, yes it is. I believe the AB test is still active.

6

Based on the screenshots you’ve shared, one of them is your production branch and you just have 1 other branch deploy. So, this looks expected to me.

I see, you’ve deployed some additional branches now and they can be added to the split testing.

Since you’re using Next.js, this can indeed be a problem s the docs say:

This can be a concern for sites that use server-side rendering (SSR) or incremental static regeneration (ISR).