Timeout when posting large file data to external API using a proxy (rewrite)

Support
,
1

Hello

First time poster here hoping to get a hand!

I’ve have an CRA site that gets data and posts data to a REST API outside of netlify. Some of the data being posted to the API can be pretty big (sometime over 1gb). I’ve used the rewrites file to create a proxy, so the site can connect to the API. This works great, but if a post request isn’t completed within about 30 seconds, it fails and I get a net::ERR_HTTP2_PROTOCOL_ERROR in the console.

I’m guessing there’s a timeout limit somewhere, but could anyone point me in the right direction as to where I can update it. Or is this something that can’t be done?

I know there is some limitations around large media and timeouts on netlify functions, but as the data is only being proxied through netlify, I’m hoping there’s a way to change this limit.

File uploads work without restrictions (well, not any that I’ve hit) when using the app locally, so I think the issue is related to netilfy.

Any help/guidance would be awesome

Thanks

Tom

2

Welcome to the forums @Tom_Woodley

If I read/understand correctly, you have a proxy such as

/api    https://example.com/api

which you are using to upload a file to from your site. Might I assume you are using a form, possibly a serverless function, to handle this? Function have a 10 second execution limit and 15min for background functions. If you are using a form, there is a 30 second timeout on file uploads which correlates with what you are seeing.

3

Thanks @coelmay that’s exactly right. I’m using a form input to allow users to add files, which get’s uploaded on submit via an ajax api route. Normally 30 seconds would be ample, but as we’re working with large file data, it can often exceed this.

Is there any way to circumvent this?

Many thanks for your help!

Tom

4

The only circumvention I can think of is not using a proxy. I don’t believe the system was designed for such uses.

Others may have more to add than that.

5

Hi @Tom_Woodley,

You’re right. There’s a 30 sec connection limit for a proxy and the connection should close within that limit. There’s no way to increase this unfortunately. So, what you’re trying might not be possible.

1 Like
6

I’m having the same issue with my Netlify-hosted frontend. Requests to my backend are proxied using a Netlify rewrite rule in the style described here. This is great as I don’t have to deal with CORS and cookies just work with “SameSite: strict”.

I have a simple file upload form that allow users to upload images. The upload is performed in a graphql multipart request.

The files concerned are usually only 5-10MB (I have a 10MB limit per upload) in size but on slower connections this can easily hit the 26s timeout. Is there no way at all to raise it? Even upping it to ~1 minute would allow a much greater share of file uploads to succeed.

7

Hi @harrygr,

Sorry to bear bad news, but that’s one of the hard limits that can’t/won’t change at least for the time being.

I think the best bet for you would be to add a client-side timer to let the know they have only this much time to upload and it will fail - this is far from ideal, I agree but it’s either this or no proxy at all.

8

That’s a shame. The proxy is a really useful feature for apis. I think the answer might have to be to stop using it.