Netlify function function not handling "Expect: 100-continue" header from HTTP POST request

Support
,
1

I’ve got a Netlify function used within my Gatsby app that is built to handle callbacks from Snipcart so I can update a database when a user places an order.

Snipcart is sending a “Expect: 100-continue” header with their webhook, which bombs out my function. The request appears to time out, and eventually I see this in my app console log:

BadRequestError: request aborted
    at IncomingMessage.onAborted (/usr/local/lib/node_modules/netlify-cli/node_modules/body-parser/node_modules/raw-body/index.js:231:10)
    at IncomingMessage.emit (events.js:314:20)
    at abortIncoming (_http_server.js:533:9)
    at socketOnClose (_http_server.js:526:3)
    at Socket.emit (events.js:326:22)
    at TCP.<anonymous> (net.js:675:12)

I can reproduce this easily with a curl request:

curl -d ‘{“somefield”:“somevalue”}’ -H ‘Content-Type: application/json’ -H ‘Expect: 100-continue’ http://localhost:8888/.netlify/functions/syncSnipcart

I doubt the Snipcart team will be able to update their Webhook processing to remove this header for my hooks, so what can I do (if anything) to get this fixed?

When checking my Ngrok logs, I do see my POST request appears to be coming in with the body intact, so I think everything should work fine once I can get this sorted out.

And if it makes any difference, I’m using netlify-lambda in my local dev environment to test out my functions (so far working great once I got it working smoothly with my local build process).

2

From reading 100 Continue - HTTP | MDN this is telling your script (the client) that it can continue with the request and send the body.

3

Understood. I’m not quite sure why Snipcart is sending the header they are, but any time Netlify’s function API receives this header in a POST request, it bombs out.

4

I can find nothing in Snipcart’s docs about this. I would suggest contacting Snipcart support about why you are receiving the header.

Netlify may bomb out as it is not expecting (no pun intended) the ‘Expect’ header, or your function isn’t expecting it.

5

I did submit a message to Snipcart’s Intercom support chat about their usage of the header. They also responded in another thread here for what may or may not be a related issue that they’ll be looking into why they send this header with their POST call. The problem isn’t my function, it’s the core HTTP handler itself. My function isn’t even getting the event object and I’m not getting any exceptions caught when wrapping the whole thing in a try/catch.

I took a look at the netlify-lambda Github repo and I honestly don’t see anything for them to fix here - the issue seems to lie in it’s dependencies - either express, body-parser, or NodeJS itself (netlify-lambda uses node 10x).

6

I spent all night hacking around my dev environment (dumb product manager here who moonlights as a Javascript coder lol) and found if I can modify the request headers early enough, I can simply strip out the offending header and things work out OK.

Modifying /usr/local/lib/node_modules/netlify-cli/src/utils/proxy.js, see my additional function inserted on line 284 for removing any headers that are set to ‘removeHeader’ in my header rules from _headers or netlify.toml:

  // Custom function that reads Netlify's header rules, and strips out any headers that are set to "removeHeader".
  proxy.before('web', 'stream', (req) => {
    const requestURL = new URL(req.url, `http://${req.headers.host || 'localhost'}`)
    const pathHeaderRules = headersForPath(headerRules, requestURL.pathname)
    if (!isEmpty(pathHeaderRules)) {
      Object.entries(pathHeaderRules).forEach(([key, val]) => {
        if (val && val[0] === 'removeHeader' && req.headers[key]) {
          delete req.headers[key];
        }
      })
    }
  });

I would have forked the package but considering the change to support stripping headers would ultimately have to be made by the Netlify team within their infrastructure for my production app, it’s kind of moot.

It does seem like it wouldn’t be a bad idea to support stripping out headers in this fashion if Netlify supports adding them?

The ‘expect’ header also must be stripped out in the proxy.before function as any later in the stack still fouls up the processing.

I’m not sure what the harm would be in moving all header modification to this proxy.before function so I just kept things simple and focused only on supporting the removal of unwanted headers.

1 Like
7

:rofl: :rofl: :rofl: :rofl: :rofl: :rofl:

A Hundred Million Internets for the effort.
You could always submit it on GitHub as a feature request.

1 Like
8

I’m totally wilding out before starting my day (back to back sprint reviews with my two SCRUM teams up next!) and I created a PR :rofl:

9

@djben we made changes on our end so that this header won’t be sent anymore. Hopefully this will fix the issues we were having since a few weeks! Let us know if you happen to test it and still get issues.

2 Likes
10

@snipcart Thanks a bunch for the rapid changes! Netlify also updated their netlify dev environment to strip out the expect headers after I submitted a PR with a fix.

I’ve got your cart integrated on a netlify-hosted Gatsby-built site that I built from the ground up as my very first React project for my friend’s youth hockey team:

https://floridabulldogshockey.com/shop

The shopping cart data comes from static file content – my friend who runs the hockey team can easily just edit files and drop in pictures on his laptop, then just use Github Desktop to check in the changes to an asset-only repository to see the changes go live on the site.

I built in order tracking for registered users (Firebase integration), which of course is where the webhooks come in for updating user orders :slight_smile: