Robot User-Agent header causes Netlify functions to improperly interpret POST request as GET

UPDATE: Solved :heart_eyes:

See my reply to this post, below.

Original post

Hello, I’ve been experiencing strange behavior from my Netlify function from my deploy-preview. I’m totally stumped and would appreciate any help.

tl;dr

When a Netlify function receives a POST request from the Slack Event API with the following header, Netlify interprets the request as a GET request. Is there any way to correct or circumvent this problem?

'User-Agent: Slackbot 1.0 (+https://api.slack.com/robots)'

The full story:

My function below handles the Slack challenge without fail when I run it locally using an ngrok tunnel.

exports.handler = async (event, context) => {
  console.log('EVENT', { event })

  const { body } = event
  const data = body ? JSON.parse(body) : { challenge: 'none' }

  console.log('DATA', { data })

  return {
    statusCode: 200,
    headers: {
      'Content-type': 'text/plain',
    },
    body: data.challenge,
  }
}

However my function fails when run from the deploy, in preview or production. Slack returns the following message:

Our Request:

    POST
    "body": { 
    	 "type": "url_verification",
    	 "token": "eiX__obscured_by_me__916",
    	 "challenge": "MMoUApArQ6VWQO7TElAhp2nEGXxztQrGpcUeVdfEW9neGEbbLpX1"
    }

Your Response:

    "code": 200
    "error": "challenge_failed"
    "body": {
     <html><head></head><body><pre style="word-wrap: break-word; white-space: pre-wrap;">none</pre></body></html> 
    }

Looking at the body from my function’s response, it appears as though it received as a GET. I confirmed this by looking at the function’s log from the deploy:

8:07:05 PM: 2021-03-05T02:07:05.717Z	ba94b36d-54b4-41ca-83d8-a59c69d6195f	INFO	EVENT {
  event: {
    path: '/.netlify/functions/tester-text',
    httpMethod: 'GET',
    headers: {
      accept: '*/*',
      'accept-encoding': 'gzip',
      'client-ip': '2001:470:12b:3:0:2ce:fc90:6e5',
      cookie: '__cfduid=d210a36bef68471de0c744646845d3cbd1614191895; experimentation_subject_id=eyJfcmFpbHMiOnsibWVzc2FnZSI6IklqRTFNREkyTnpZMUxUUTRNalF0TkRFMFl5MWhPVEUzTFRnM04yWTRNemxtTlRnd01TST0iLCJleHAiOm51bGwsInB1ciI6ImNvb2tpZS5leHBlcmltZW50YXRpb25fc3ViamVjdF9pZCJ9fQ%3D%3D--3db436929a6121bc7f6f9213cc34aa1a98b2aad9',
      host: 'deploy-preview-758--seeds.netlify.app',
      'user-agent': 'Discourse Forum Onebox v2.7.0.beta4',
      via: 'https/1.1 Netlify[14218673-7aa6-4b45-9c99-daa848251ad4] (Netlify Edge Server)',
      'x-bb-ab': '0.976536',
      'x-bb-client-request-uuid': '14218673-7aa6-4b45-9c99-daa848251ad4-53456025',
      'x-bb-ip': '2001:470:12b:3:0:2ce:fc90:6e5',
      'x-bb-loop': '1',
      'x-country': 'US',
      'x-forwarded-for': '2001:470:12b:3:0:2ce:fc90:6e5',
      'x-forwarded-proto': 'https',
      'x-nf-client-connection-ip': '2001:470:12b:3:0:2ce:fc90:6e5',
      'x-nf-request-id': '14218673-7aa6-4b45-9c99-daa848251ad4-53456025'
    },
    multiValueHeaders: {
      Accept: [Array],
      'Accept-Encoding': [Array],
      'Client-Ip': [Array],
      Cookie: [Array],
      'User-Agent': [Array],
      Via: [Array],
      'X-Bb-Ab': [Array],
      'X-Bb-Client-Request-Uuid': [Array],
      'X-Bb-Ip': [Array],
      'X-Bb-Loop': [Array],
      'X-Country': [Array],
      'X-Forwarded-For': [Array],
      'X-Forwarded-Proto': [Array],
      'X-Nf-Client-Connection-Ip': [Array],
      'X-Nf-Request-Id': [Array],
      host: [Array]
    },
    queryStringParameters: {},
    multiValueQueryStringParameters: {},
    body: '',
    isBase64Encoded: true
  }
}
8:07:05 PM: 2021-03-05T02:07:05.717Z	ba94b36d-54b4-41ca-83d8-a59c69d6195f	INFO	DATA { data: { challenge: 'none' } }

At Slack support’s suggestion, I used a listener hook at https://dev.oscato.com/ to generate an HTTP inspector to look at the request composition from Slack’s challenge request:

Method
[POST]
Headers

Accept: */*
Accept-Encoding: gzip,deflate
Connection: close
Content-Length: 129
Content-Type: application/json
User-Agent: Slackbot 1.0 (+https://api.slack.com/robots)
X-Slack-Request-Timestamp: 1614870550
X-Slack-Signature: v0=045d5e7aad57b99767ae25377627ed2264b353a658a2212f44e92964f4afe3a7

Body

{
  "token": "eiX__obscured_by_me__916",
  "challenge": "6O5sHgTpfsMYP3lEYzFkIeTvR0Gmzv2CyhnUKDQgVHu8qjjf9JJt",
  "type": "url_verification"
}

Experimenting with curl, I discovered that the User-Agent header is causing the false GET interpretation by Netlify.

curl -X POST \
'https://deploy-preview-758--seeds.netlify.app/.netlify/functions/tester-text' \
-H 'User-Agent: Slackbot 1.0 (+https://api.slack.com/robots)' \
-d '{"token":"eiX__obscured_by_me__916","challenge":"Mbg6GreKIIMXL0Y5Yr64gCpvs2VKj9Ly88265qkSBy9U6jdZqJKE","type":"url_verification"}'

When this header is removed or set to another value, the request is properly interpreted as a POST.

I would greatly appreciate any help in resolving this problem.

Thanks!

Maurice

I read up on user-agents identified as bots, and then search here in the Netlify forums for “user-agent” and found this post, “User-agent blocking?”, where they mentioned pre-rendering. In that post, they state:

Prerendered content is only displayed in certain situations. Specifically it:
• will only be used for a Netlify website once prerendering is enabled on your site’s build & deploy settings page
• will only be returned in response to a web request when there is a User-Agent 32 sent with the request that needs this special content, or a special URL is used (see below for more details around manual testing and the special URL).

Reading up on Netlify’s prerendering, I then went and deselected the site’s prerendering option, leading to return of the expected behavior!

Once the Slack Event api verifies my endpoint, I can then re-enable prerendering :relaxed:

thank you so much for sharing this! this so helpful for others dealing with the same problems :netliconfetti:

1 Like