Suggestion for dealing with complex split tests

Hey y’all :wave:

So, Netlify’s split testing is perfect for code changes to provide behaviors such as feature flags and beta access (such as demonstrated by Phil Hawksworth), but when it comes to content changes, things start getting messy and quite limited

Let’s say we want to allow editors of a headless CMS that powers a Netlify site to create variations of each page, individually. The homepage might have 3 variations, the about page 2, and so on. Currently, to pull this off without relying on client-side techniques - which, as we know, hurt performance, UX and SEO -, you’d have to automatically create repository branches for each combination of variations, and then hit the Netlify API to register the split tests. In the example above, here’s what we’d have:

  • 3 variations for home (H1, H2, H3)
  • times 2 variations for the about page (A1, A2)
  • equals 6 total branches, 5 new (H1A1, H1A2, H2A1…)
  • plus, there’s the whole issue of figuring out how to split between these variations: if H1 should represent 40% for the homepage, H1A1 + H1A2 should equal 40% of the traffic, but the distribution between them is dependent between A1 and A2. It’s easy to see how the complexity scales when we’re dealing with 3+ sets of pages.
  • finally, orchestrating all of the API calls to the git provider and to Netlify would be a lot of work, and managing so many branches in the repository would be any developer’s nightmare.

Overall, relying on the split testing functionality for this use case doesn’t seem to be an option, unless I’m missing something, of course (any feedback appreciated!). My second thought was to take a look into the capabilities of redirects and rewrites. As powerful as those are (thank you for that, by the way!), there’s no way to run any logic that would allow us to attribute variations. So, here’s my suggestion for a redirects feature that would make Netlify that much more powerful for ambitious, performance-critical JAMstack websites:

# Variations for the homepage
/  40%  /       200!
/  60%  /home-2 200!

# Variations for about
/about  10% /about-2 200!
/about  60% /about-3 200!
/about  30% /about   200!

# As we already have key=id for denoting query strings,
#  {number.decimal}% seems like a good option for variations

Let me know what you think of this suggestion :smile: This is based on my personal preferences and experience, without paying much attention to technical details, such as backward compatibility, so I’m 100% open for feedback and adjustments.

Netlify is an amazing product and this particular facet of a website is the last remaining puzzle piece for web projects at our agency, thank you for being awesome and keep up the good work :tada:

Hi, @hcavalieri, we are happy to enter a feature request for this. I want to be sure I understand the feature request before filing it though.

You want a way of doing redirects based on percentages in the redirects file. This would allow split testing inside of a single deploy. Is that correct?

There is one potential downside that I see from this. This may not apply for your use case though so please let me know if it doesn’t and if we should still file the feature request or not.

The downside goes something like this. Imagine the site is selling widgets and you are split testing two promotions: one for 10% off widgets and one for %20 off.

With our current method for split tests, we use a cookie to keep the browser always using a single deploy so all pages would be consistent. If you visit the landing page and get the 20% off split test, the cookie will make sure you also get the 20% off version on the product detail page.

If the redirect is happening with percentages in the _redirects file, there is no concept of a specific deploy. There is no way make sure the person that sees %20 off on the home page also sees %20 off on the product detail page.

Now, if each page is completely independent, then it doesn’t matter which version the user sees of a new page so the random percentage solution does work. However, if consistency between versions is required as a user moves from page to page the percentage solution won’t do this.

Again, our solution for the consistency issue is to make sure each visitor only sees a single version (deploy id) for a site no matter which page it is. This is only possible when the split testing is done by deploy.

Now that this limitation has been pointed out, do you still want the feature request for percentage based redirects in netlify.toml?