POST file stream from serverless netlify function

Support
1

Hi all,
We’re running apollo/gql server via netlify functions with apollo-server-lambda.
All has been working fine so far.

We are creating a very simple file upload functionality.
We upload the file from a react client, then proxy this to a third party backend service via post.

This works fine in local dev with ts-node server, but fails when we either try to run on netflify servers, or via netlify dev.

we get timeout errors

    Error during invocation:  {
  errorMessage: 'Task timed out after 10.00 seconds',
  errorType: 'TimeoutError',
  stackTrace: [
    'new TimeoutError ([PATH]/node/v12.18.2/lib/node_modules/netlify-cli/node_modules/lambda-local/build/lib/utils.js:110:28)',
    'Context.<anonymous> ([PATH]/node/v12.18.2/lib/node_modules/netlify-cli/node_modules/lambda-local/build/lib/context.js:110:19)',
    'listOnTimeout (internal/timers.js:549:17)',
    'processTimers (internal/timers.js:492:7)'
  ],
  level: 'error',
  [Symbol(level)]: 'error',
  [Symbol(message)]: '{"errorMessage":"Task timed out after 10.00 seconds","errorType":"TimeoutError","stackTrace":["new TimeoutError ([PATH]/node/v12.18.2/lib/node_modules/netlify-cli/node_modules/lambda-local/build/lib/utils.js:110:28)","Context.<anonymous> ([PATH]/node/v12.18.2/lib/node_modules/netlify-cli/node_modules/lambda-local/build/lib/context.js:110:19)","listOnTimeout (internal/timers.js:549:17)","processTimers (internal/timers.js:492:7)"],"level":"error"}'
}

essentially we upload file to netlify and save it to /tmp.
we verify file has been written with

fs.statSync(filePath) // check passes

then we try to post to external service via apollo RESTdatasources which is using node-fetch under the hood.
Other POSTS to same server work fine with standard JSON text payloads.

We’ve additionally verified this external service works fine with postman etc, the response is quick, a few 100 ms. So it’s not just a slow service causing timeouts.

 var stats = fs.statSync(filePath)

  if(fs.existsSync(filePath)){
    console.log('can read') // we see this in the log
  }

  await this.post(
    `${EndPoints.MY_COOL_ENDPOINT}`,
   fs.createReadStream(filePath)
 
  )

is this a non-starter? I.e. is there some limitation in the serverless/netlify architecture which would prevent passing a file stream as body to a POST request?
we’re not getting any other useful errors other than the timeout.

Is there an alternate way to post files from netlify server side?
These files don’t stay in the netlify environment, i.e. our GQL server is really just a proxy, and S3 bucket is not an option.

2

Hey @lbox,
I don’t believe we currently support streaming from functions (see a related thread here: Stream content to function response body - #9). Can you send the file all at once, i.e. use fs.readFile or fs.readFileSync?

3

hi jen, thanks for response. but that didn’t work either for us.
we ended up abandoning this approach and were able to get going directly to bucket via AWS SDK approved. that seems to work fine.

1 Like
4

Makes sense, glad to hear you found something that worked