What are the SPECIFIC limitations for split testing with proxies

Hello, your docs say that split testing does “not work properly” with proxies. We’ve been trying your split testing and it seems to work fine for our use case, but we don’t want to rely on it without understanding what the specific limitations are.

We use netlify force redirects to proxy requests to certain paths to our api servers. I can see that you’re not rewriting the domain set-cookie header but that works fine for us.

I wrote in to paid support and got back an answer saying that we should not use the split testing feature but instead use edge functions like in this example: Set up an AB Test | Edge Functions on Netlify

But that example doesn’t show ANYTHING at all about how to do what we want to do with split testing, namely: we want to serve ONE branch to one set of user and ANOTHER branch to serve a second set of users. That example shows how to write a cookie using an edge function (which is TRIVIAL).

I requested more information but now they’re ghosting me so I’m trying this support channel instead.

So can someone answer one of the following questions:

  • What are the SPECIFIC limitations for split testing & proxies?
  • How can one achieve split testing with edge functions as recommended by your support team?

Thanks

That’s an odd way to say you got a response in 19 hours on average. About 10 hours for the first message, 48 hours for the next and 40 minutes for the last one before this message that I am writing which is about 19 hours after your last message in the helpdesk. I can understand our response times might not be the fastest, but please be mindful and appreciative of our efforts. :slight_smile:

Regarding the specific question, it largely includes problems outlined here: [Support Guide] Why not proxy to Netlify?, specifically:

and

Furthermore, it would also depend on the proxy’s configuration. For example the way the proxy itself might be itterfering with caching or working with cookies, etc.

About how to achieve it with Edge Functions, it would be something similar to:

import type {Context} from '@netlify/edge-functions'
export default async (request : Request, context : Context) => {
  const variantFromCookie = context.cookies.get('variant')
  function sendRewrite(variant : number) {
    const url = new URL(request.url)
    if (variant >= 0.5) {
      return new URL(url.pathname,'https://main-branch--subdomain.netlify.app/')
    } else {
      return new URL(url.pathname,'https://other-branch--subdomain.netlify.app/')
    }
  }
  if (variantFromCookie) {
    return sendRewrite(variantFromCookie)
  } else {
    const variantRandom = Math.random()
    context.cookies.set({
      name: 'variant',
      value: variantRandom.toString()
    })
    return sendRewrite(variantRandom)
  }
}
export const config : Config = {
  path: '/*'
}

Note that, this was something you should have attempted by reading the documentation and trying out some examples. We do not write or provide support with custom code, but considering this might be useful to more users who would want to know this, I wrote an example. This is untested, I wrote it directly on the forums, but this should give you an idea. This would also need some fine-tuning based on each site and use-case, but this is how a very primitive split testing can be setup using Edge Functions + branch deploys.

OK, let’s get specific about timestamps since you’ve got your nose out of joint.

It is true that 11 hours after my initial support request, you replied. The reply was pretty vacuous… you didn’t explain what the limitations of the proxies are, nor did you give a helpful workaround. Instead you shared a link to a trivial example of an edge function that writes a cookie but does not actually PROVIDE split testing capabilities.

2.5 hours later, I responded to this. Then I waited 2 days before I got frustrated.

I don’t consider a useless answer to count towards the average. Moreover, the SECOND reply wasn’t helpful EITHER… we FINALLY got to a marginally helpful response after the THIRD response.

It’s only marginally helpful because you’re now sending me links that don’t seem to apply: you’re sending links about “Why not proxy to Netlify?” but I don’t intend to proxy to netlify. Does writing an edge function mean that to you? It doesn’t to me… All requests from the browser will go directly to Netlify, I am not propsing Proxying TO netlify.

So, I STILL don’t know what the limitations of your split testing plus proxies are (except for some vagueness about “might interfere”). It seems like what you’re telling me to implement is PROBABLY what your split testing code implements internally, in which case it seems like it would be odd to re-implement it myself.

Regarding your comment about “Note that, this was something you should have attempted by reading the documentation and trying out some examples” I DID read the documentation and I DID try some examples. I also SUCCESSFULLY wired up split testing and it appears to be working perfectly. That’s why I reached out in the first place, to try to understand why NOT to use split testing for this-- I was looking for the hidden gotcha that might bite us. If those docs existed you certainly haven’t managed to send them to me.

It seems like your split testing employs some kind of logic that is NOT tied to a specific branch 's code becasue you can configure which branch is being used for the split test without modifying the code for that branch itself. That would be handy but given that I’m never going to be allowed the privilege of the specifics of what the limitations actually ARE (e.g. “we don’t rewrite cookies” or “we don’t dynamically change origin headers” etc) I agree it’s better that we roll our own solution, even though the Netlify provided one might be much better for us.

While I do understand that you may be frustrated by the response times, there is no guaranteed response time for non-Enterprise support tickets.

Also, regardless of how long you waited, it is never appropriate behavior to be insulting or to shout. Writing in all upper-case letters is widely considered rude with regards to internet etiquette. You have written in all upper-case letters over a dozen times above. Besides the shouting, you are using insulting adjectives to describe the work our Support team does. For example, these are words you have chosen to describe our work:

  • vacuous
  • useless

I realize those adjectives were used to describe responses and not people. However, the emotional impact of those words on the people that work hard to provide answers will likely be quite similar. Saying “your reply is useless” will have the same emotional impact as saying “you are useless” for many people. Comments like “since you’ve got your nose out of joint” are also rude. Even if you did not intend to offend, it feels hard to consider that this was not a conscious effort. In the past, we had subtly hinted you to be a bit softer:

… but since this problem continues, I felt the need to make an extra effort to address this before it becomes a bigger problem for any further communication here. To promote a healthy discussion instead of an heated argument, we’d only continue to communicate further provided you decide to calm down.


With all that out of the way, I apologise on my part for the confusion. When talking about proxies, the more common use-case that causes an issue is proxying to Netlify. Thus, I assumed you’re asking about the same. In that context, the answer I provided was relevant. Upon further review, it seems you were asking about proxying from Netlify. With regards to the proxying not playing well with our split tests, there are two specific edge cases where the split test feature doesn’t work well with regards to proxying:

  • when the site in question has proxy redirect rules that differ between two branches (because the third-party proxy targets might set cache-control headers that break the split test for the proxied URLs)
  • when you proxy within Netlify (from one Netlify site to another) and the proxy target site (the site being proxied to) has split testing enabled

As you do not appear to be using different redirect rules in the two split tests and you are not proxying within Netlify’s infrastructure to a site with split testing enabled, neither of the caveats above will apply here. I do think you can use the in beta split testing feature here without running into any of the unhandled edge cases above.