Split testing Under the Hood?

At a high level, we have X copies of your site (1 per branch in the split test) served from our CDN. Choice as to which version to serve, as well as the mapping of “been here before, should serve the same version” happens at the CDN edge, at browse time, without any “round trip” to the backing store to decide which version to serve.

How we choose which version to serve is:

  • for new visitors (which means “no nf_ab cookie is set for this site” - so for instance curl is always a new visitor unless you set the cookie to a valid value explicitly), we use the percentages you’ve configured to offer a statistically random version of the site. As we serve them a version, we set that nf_ab cookie to a real number between 0 and 1 to indicate what the random selection was, and it maps to those percentages in the test (so a value of .1 would map to the 0-10% portion of the test)
  • for repeat visitors, we see the nf_ab cookie in the HTTP request, and serve them the appropriate version of the site (which will be the one they saw before, unless you’ve changed your settings in the meantime).

There are additionally some major caveats I wanted to call out:

  • if you proxy to us, this will typically break the branch affinity and will lead to trouble. More details in this article
  • if you make API calls via methods like XmlHTTPRequest, or from 3rd party services, make sure you forward and respect the nf_ab cookie when you do this - if a request arrives here without it set, we’ll pick a branch as described above - even if it was an API call for a specific branch test that the user is already viewing, we can’t tell that unless the cookie is set at REQUEST time for EVERY request.

Not sure if that’s what you were looking for or not, but happy to go deeper into whatever follow-up questions you might have.