Getting 504 GatewayTimeout response

Charles from Snipcart here. We recently got reports of customers having issues with Snipcart with their sites hosted on Netlify.

We do have an order validation process internally, when a customer checks out, we do a request to the page where the product is located. This request is made by our servers with a User-Agent set to Snipcart/1.0. This is a simple GET request, then we parse the HTML returned by the website to do our validation stuff.

Since a couple of days, we are getting reports of customers having issues with this process. After investigating our logs, we figured out that Netlify returns a 504 error.

Here’s an example of what we can see in the logs:

Sent crawling request to https://www.smyssly.com/cs/products/82/intenzivni-serum-kyseliny-hyaluronove-4-percent//. Response status code is 504 (GatewayTimeout)

Is it possible that we’re getting rate limited or something like this? We do have lots of customers running on Netlify, and most of our crawling requests will come from the same IP addreses.

1 Like

Hey there, @snipcart :wave:

Thank you so much for reaching out. I have looped in our Support Engineers, and will we follow up on this Forums post when we have more information. Please do share any updates or changes should things come up in the interim.

Again, we appreciate you letting us know!

1 Like

@snipcart Also just posted something that may be similar for my Snipcart implementation - check out my thread. I actually moved to using your JSON crawler because I seemed to have problems with Snipcart crawling the normal HTML data-item-url of my orders.

1 Like

Thanks @djben , I just noticed that .NET automatically sends this header, but we can change the behavior. I’ll run some tests to see if we can safely remove it. That’s pretty weird though because we started having this issue quite recently and we haven’t change anything on our end for quite a while. I think it should be handled properly by Netlify, but it might be easier for us to change this.

2 Likes

Hey team! So, what I can see is that the 504s occur when the site takes >10s to respond. The Snipcart User-Agent is denoted by us as a prerendering UA and, as such, we will timeout requests which take >10s. Obviously, this isn’t every request hence their sporadic nature. These don’t seem like prerender requests based on your original post.

In the meantime, those configuring the cart could (should?) adhere to pre-rendering best practices (window.prerenderReady configuration, in this case). But I know that this is easier said than done.

Feel free to reach out to us – either here or over in the ticket – with your thoughts, and we can look & see whether there’s anything we can do to distinguish these requests and serve them more reliably.

1 Like

Thanks @Pie. I’d like to hear your thoughts about the Expect header mentioned by @djben. We’re about to roll out a fix that will remove this header from our end.

The window.prerenderReady configuration would require changes for all of our customers using Snipcart + Netlify. Our crawler does a very simple GET request and parse the HTML returned. We don’t run any JavaScript on our end therefore I don’t think our UA should be denoted as a prerendering UA.

It’s quite possible that this isn’t helping – our proxy would be looking for a 200 and will time out (for non-function requests) after just short of 30 seconds. 100s are unhandled for the best part.

On a semi-related note: functions, by default, time out after 10 seconds. Customers with function invocations which overrun this would ideally push these to background functions or, for functions which can’t be run async, a Pro account and an outreach to support to bump these up to 26 seconds should help.

I can confirm that we still get the issue even without the Expect header. We’ll suggest to the customers having issues to implement the prerenderReady configuration, or to use our JSON crawler instead.

But, if there’s anything else you can think of, that’d be appreciated. Netlify is the only provider that we have this issue with, and it’s also one of the most used among our user base. It’d be great if we could figure out something.

Interesting. We’re not seeing any 504s since midnight BST (13 hours ago) when the Snipcart* UA is used:

Do you have a date/time or x-nf-request-id for a request since this time? Do you know then the Expect header was revoked from requests?

Just as a side note, I am observing that customers who were experiencing 504 errors all have pre-rendering enabled in their Netlify UI.

Their issues are not isolated to Snipcart. In fact, any prerender service will 504 for them. Impacted customers seem to have pre-rendering configured sub-optimally and ought to familiarise with our guide, most notably caveat 2, which I touched on before.

Search___netlify-us-production

Alternatively, disabling pre-rendering should serve as an adequate workaround :+1:.

1 Like

Expect header was removed yesterday at around ~13h30 EST. We got an occurrence of a 504 at: Jun 07, 2021 at 19:04:05.199. We don’t log the request ID unfortunately.

Thanks for the info about prerendering, we’ll let the customer know about this for sure. If Snipcart UA wasn’t flagged as a pre-rendering UA, would we still have this issue? I’m afraid this will increase the support load on our end, many customers won’t realize that this might be caused by this setting and will contact us directly to figure this out

Seems like a double-edged sword. Without probing our traffic team, I’m not sure why the Snipcart UA is defined as a prerender UA. I’m sure there’s good reason!

However, assuming we were able to make the Snipcart UA non-prerender, it doesn’t actually fix the root problem that these customer haven’t configured pre-rendering properly and, as such, will still have issues.

1 Like

Yeah I get it, and after reading your article, I believe we should be considered as a pre-rendered UA. We’re a crawler and we need the whole page content, so I guess that makes sense.

We’re about to launch a support forum soon and we’ll most likely write an article about this in there.

Thanks a lot for the help @Pie!

3 Likes