What are the SPECIFIC limitations for split testing with proxies

Hello, your docs say that split testing does “not work properly” with proxies. We’ve been trying your split testing and it seems to work fine for our use case, but we don’t want to rely on it without understanding what the specific limitations are.

We use netlify force redirects to proxy requests to certain paths to our api servers. I can see that you’re not rewriting the domain set-cookie header but that works fine for us.

I wrote in to paid support and got back an answer saying that we should not use the split testing feature but instead use edge functions like in this example: Set up an AB Test | Edge Functions on Netlify

But that example doesn’t show ANYTHING at all about how to do what we want to do with split testing, namely: we want to serve ONE branch to one set of user and ANOTHER branch to serve a second set of users. That example shows how to write a cookie using an edge function (which is TRIVIAL).

I requested more information but now they’re ghosting me so I’m trying this support channel instead.

So can someone answer one of the following questions:

  • What are the SPECIFIC limitations for split testing & proxies?
  • How can one achieve split testing with edge functions as recommended by your support team?

Thanks

That’s an odd way to say you got a response in 19 hours on average. About 10 hours for the first message, 48 hours for the next and 40 minutes for the last one before this message that I am writing which is about 19 hours after your last message in the helpdesk. I can understand our response times might not be the fastest, but please be mindful and appreciative of our efforts. :slight_smile:

Regarding the specific question, it largely includes problems outlined here: [Support Guide] Why not proxy to Netlify?, specifically:

and

Furthermore, it would also depend on the proxy’s configuration. For example the way the proxy itself might be itterfering with caching or working with cookies, etc.

About how to achieve it with Edge Functions, it would be something similar to:

import type {Context} from '@netlify/edge-functions'
export default async (request : Request, context : Context) => {
  const variantFromCookie = context.cookies.get('variant')
  function sendRewrite(variant : number) {
    const url = new URL(request.url)
    if (variant >= 0.5) {
      return new URL(url.pathname,'https://main-branch--subdomain.netlify.app/')
    } else {
      return new URL(url.pathname,'https://other-branch--subdomain.netlify.app/')
    }
  }
  if (variantFromCookie) {
    return sendRewrite(variantFromCookie)
  } else {
    const variantRandom = Math.random()
    context.cookies.set({
      name: 'variant',
      value: variantRandom.toString()
    })
    return sendRewrite(variantRandom)
  }
}
export const config : Config = {
  path: '/*'
}

Note that, this was something you should have attempted by reading the documentation and trying out some examples. We do not write or provide support with custom code, but considering this might be useful to more users who would want to know this, I wrote an example. This is untested, I wrote it directly on the forums, but this should give you an idea. This would also need some fine-tuning based on each site and use-case, but this is how a very primitive split testing can be setup using Edge Functions + branch deploys.

OK, let’s get specific about timestamps since you’ve got your nose out of joint.

It is true that 11 hours after my initial support request, you replied. The reply was pretty vacuous… you didn’t explain what the limitations of the proxies are, nor did you give a helpful workaround. Instead you shared a link to a trivial example of an edge function that writes a cookie but does not actually PROVIDE split testing capabilities.

2.5 hours later, I responded to this. Then I waited 2 days before I got frustrated.

I don’t consider a useless answer to count towards the average. Moreover, the SECOND reply wasn’t helpful EITHER… we FINALLY got to a marginally helpful response after the THIRD response.

It’s only marginally helpful because you’re now sending me links that don’t seem to apply: you’re sending links about “Why not proxy to Netlify?” but I don’t intend to proxy to netlify. Does writing an edge function mean that to you? It doesn’t to me… All requests from the browser will go directly to Netlify, I am not propsing Proxying TO netlify.

So, I STILL don’t know what the limitations of your split testing plus proxies are (except for some vagueness about “might interfere”). It seems like what you’re telling me to implement is PROBABLY what your split testing code implements internally, in which case it seems like it would be odd to re-implement it myself.

Regarding your comment about “Note that, this was something you should have attempted by reading the documentation and trying out some examples” I DID read the documentation and I DID try some examples. I also SUCCESSFULLY wired up split testing and it appears to be working perfectly. That’s why I reached out in the first place, to try to understand why NOT to use split testing for this-- I was looking for the hidden gotcha that might bite us. If those docs existed you certainly haven’t managed to send them to me.

It seems like your split testing employs some kind of logic that is NOT tied to a specific branch 's code becasue you can configure which branch is being used for the split test without modifying the code for that branch itself. That would be handy but given that I’m never going to be allowed the privilege of the specifics of what the limitations actually ARE (e.g. “we don’t rewrite cookies” or “we don’t dynamically change origin headers” etc) I agree it’s better that we roll our own solution, even though the Netlify provided one might be much better for us.

While I do understand that you may be frustrated by the response times, there is no guaranteed response time for non-Enterprise support tickets.

Also, regardless of how long you waited, it is never appropriate behavior to be insulting or to shout. Writing in all upper-case letters is widely considered rude with regards to internet etiquette. You have written in all upper-case letters over a dozen times above. Besides the shouting, you are using insulting adjectives to describe the work our Support team does. For example, these are words you have chosen to describe our work:

  • vacuous
  • useless

I realize those adjectives were used to describe responses and not people. However, the emotional impact of those words on the people that work hard to provide answers will likely be quite similar. Saying “your reply is useless” will have the same emotional impact as saying “you are useless” for many people. Comments like “since you’ve got your nose out of joint” are also rude. Even if you did not intend to offend, it feels hard to consider that this was not a conscious effort. In the past, we had subtly hinted you to be a bit softer:

… but since this problem continues, I felt the need to make an extra effort to address this before it becomes a bigger problem for any further communication here. To promote a healthy discussion instead of an heated argument, we’d only continue to communicate further provided you decide to calm down.


With all that out of the way, I apologise on my part for the confusion. When talking about proxies, the more common use-case that causes an issue is proxying to Netlify. Thus, I assumed you’re asking about the same. In that context, the answer I provided was relevant. Upon further review, it seems you were asking about proxying from Netlify. With regards to the proxying not playing well with our split tests, there are two specific edge cases where the split test feature doesn’t work well with regards to proxying:

  • when the site in question has proxy redirect rules that differ between two branches (because the third-party proxy targets might set cache-control headers that break the split test for the proxied URLs)
  • when you proxy within Netlify (from one Netlify site to another) and the proxy target site (the site being proxied to) has split testing enabled

As you do not appear to be using different redirect rules in the two split tests and you are not proxying within Netlify’s infrastructure to a site with split testing enabled, neither of the caveats above will apply here. I do think you can use the in beta split testing feature here without running into any of the unhandled edge cases above.

Hello,

I have read the documentation several times since you said “You really should” etc…

I tried the thing you recommended with edge functions, redirecting to a branch deployment on the same site and I get the following error:

“Error: Rewrite to ‘https://–.netlify.app/’ is not allowed: edge functions can only rewrite requests to the same base URL”

i am sorry you think I am unkind. I’m incredibly frustrated because the documentation is ambiguous and when I get answers from support it seems that they do not work. Perhaps I am not reading things correctly?

On my end it feels like you are being condescending-- you’re implying I’m not reading the documentation and that it’s disrespectful to expect that getting a solution resolved might take weeks and weeks of back and forth.

I have diligently read all the instructions and docs and tried for many hours and hours to reproduce a working solution. I have found (and previously reported) bugs in the split site testing beta and thus do not feel comfortable using it. I cannot accomplish what you’re suggesting with edge functions (per the error message I’m getting described above).

With all respect for your efforts to help me, I will say that this has not been a productive effort. I wish I had never tried to achieve this because we’ve spent a lot of time internally on something that doesn’t seem to work well enough to use in the real world.

Thank you for your efforts to help me. I hope you can pass on my dissatisfaction with both the features of the product and the accuracy and round-trip-latency of questions to the appropriate recipients.

For future readers:

I realised the problem with my pseudo-code after I posted it, so I was working on an alternative solution. However, I ran into a bug with that, so was waiting on devs to fix that before posting a solution here. This should finally be resolved and here’s a working solution (tested).

Step 1: Create Site 1 and add the following Edge Function to it:

import type {Context} from '@netlify/edge-functions'
export default async (request : Request, context : Context) => {
  const url = new URL(request.url)
  if (url.searchParams.get('split')) {
    return
  }
  const variantFromCookie = context.cookies.get('variant')
  function sendRewrite(variant : number | string) {
    if (parseFloat(variant) >= 0.5) {
      return new URL(`/branch-2/${url.pathname}?split=true`, url.origin)
    } else {
      return new URL(`/branch-3/${url.pathname}?split=true`, url.origin)
    }
  }
  if (variantFromCookie) {
    return sendRewrite(variantFromCookie)
  } else {
    const variantRandom = Math.random()
    context.cookies.set({
      name: 'variant',
      value: variantRandom.toString()
    })
    return sendRewrite(variantRandom)
  }
}
export const config : Config = {
  path: '/*'
}

This is a very basic example consisting of just two branches and you can expand it as needed for more variants. In the same site, add the following rewrites (in netlify.toml):

[[redirects]]
  force = true
  from = "/branch-2/*"
  status = 200
  to = "https://branch-2--site-2.netlify.app/:splat"
[[redirects]]
  force = true
  from = "/branch-3/*"
  status = 200
  to = "https://branch-3--site-2.netlify.app/:splat"

Step 2: Create Site 2 and create branch-2 and branch-3 for that site. Use that URL in the above rewrites in the netlify.toml.

This should solve it. As for an explanation on how this works:

  • The Edge Function first checks if the URL has a particular query parameter. This is important to add as a circuit breaker to prevent a loop. If the query param exists, the request skips the Edge Function and proceeds normally.
  • If the params doesn’t exist, it then checks for the cookie. Based on the value of the cookie, it rewrites internally to either /branch-2/ or /branch-3/ along with a query param (not visible in browser) required to end the request chain as explained in the previous point. This /branch-2/ or /branch-3/ URL now matches the Netlify Rewrites added in the netlify.toml and rewrites to the other site.
  • If a cookie doesn’t exist either, we create a random value, set that as the cookie and send the rewrite based on this value.

In the browser, this makes it seem as if the first site is doing all of this as the URL remains the same.


With that being said, we’re also working on a new Rewrite class for Edge Funtions that would make site-aware rewrites much simpler. That would allow you to use something like my previous code without having to create 2 sites for this functionality.