Getting 502 errors from URL rewrite

Support
, ,
#1

I have been getting 502 errors for a rewrite to my API which is hosted on Heroku.

My Netlify site name is furnish-app.netlify.app. My _redirects file looks like this:

/api/* https://furnish-app.herokuapp.com/api/:splat 200
/* /index.html 200
https://furnish-app.netlify.app/* https://www.furnish.app/:splat 301!

When I make an API request directly to my API I get the expected response; e.g. https://furnish-app.herokuapp.com/api/stylequiz/1?productCategory=displaycabinet works correctly.

However when I do the same request via my Netlify URL which I get a 502 response; e.g. https://www.furnish.app/api/stylequiz/1?productCategory=displaycabinet should be getting rewritten to same URL as above but it in fact fails with a 502 error.

When I make requests via the latter URL I can see in my Heroku logs that the requests are coming through and they all have 200 statuses, so it looks like the request gets successfully redirected to the Heroku API but then something goes wrong on the way back from Heroku to Netlify. This is causing my app to be broken so any help would be really appreciated. Thanks!

#2

hi there, is this related at all?

#3

Hi, thanks for replying.

I don’t think so. I’m not using functions or lambdas, I’m just using a _redirects file so I don’t think there’s anything I could have done wrong.

#4

Hey @aron :wave:t2:

I’m currently looking at this. I wonder if there’s an issue with the splat redirect no passing through query parameters. When you hit the Heroku endpoint without a query param it renders a 400… so it’s possible that Netlify’s middle-man rewriting is spitting back a 502 to you if Heroku is sending a 400 on the other end. I know that functionality is documented here but I haven’t seen much calling out the positive or negative use cases of query parameters :thinking:

That’s my first thought, anyway. Going to keep poking around and see what I can find :slight_smile:

–
Jon

EDIT: actually, upon hitting the API pass-through from CLI, I’m noting the 502 being:

HTTP/1.1 502 Invalid HTTP Response

which smells a lot like the underlying service responded with a bad HTTP code. Still not certain the query parameters are for sure a factor, but definitely feels like Heroku isn’t sending back a 200! :thinking:

#5

Oh. Well, terminal digging helps…

Are you stringifying your JSON before you return it from your API? Hitting https://furnish-app.herokuapp.com/api/stylequiz/1?productCategory=displaycabinet from my browser renders fine but hitting it from the CLI gives me

HTTP/1.1 200 OK
Connection: keep-alive
Content-Type: application/json; charset=utf-8
Date: Wed, 10 Jun 2020 14:49:27 GMT
Server: Cowboy



+-----------------------------------------+
| NOTE: binary data not shown in terminal |
+-----------------------------------------+

which makes me think perhaps the API isn’t actually sending back stringified JSON (which is what application/json MIME type should imply) but actual binary data representing maybe the javascript object itself from your API.

Could you share some of that code? This could definitely be causing your issue.

–
Jon

#6

Hi @jonsully thanks for looking into this!

Hm that is very odd. I am most definitely stringifying my JSON, I’m not trying to send unstringified JS objects over the wire - I’m not even using a Node app, it’s an F# app that requires me to stringify all responses.

What CLI are you using to make that request that returns binary data? Making a request with cURL just returns the regular stringified JSON just like I get in the browser.

I could share my code but it’s a pretty enormous web app so not sure which parts would be relevant. It’s certainly not a Node app putting an un-stringified JavaScript object in the response if that’s what you’re thinking. But also I haven’t changed anything about how the response is sent or serialised. This just started happening suddenly without any meaningful changes in my code.

#7

Actually I wonder if the problem could be that the response is gzip encoded… That could explain why you’re seeing only binary data in the CLI you’re using, if that CLI isn’t aware of gzip encoded responses, unlike the browser and cURL.

If that is the cause though then:

  1. Why is Netlify trying to decode my API response instead of just passing it through as is?
  2. What changed to make this suddenly break when it was working perfectly before?
#9

I don’t know that we are trying to decode the response - I see a connection error (from our CDN node to heroku) in our internal logs. While there was an outage at heroku today on this topic: https://status.heroku.com/incidents/2045 - your logs are older and seem unrelated (though the symptoms were the same at the time).

I’ve put the details about this into the meeting agenda for a pairing our Support team will have with our traffic & delivery team next week to get their input, since there was a clear switchover after the 4th of June in our internal logs from working to not.

We’ll follow up with you after we get a chance to talk with them!

1 Like
#11

Unfortunately beyond getting a bug filed our team did not get to the bottom of it. Sorry I don’t have better news for you today, but I will follow up here in case we do get a fix for that bug.

#12

Hey @aron

It seems like our reverse-proxy does not like the fact that one of the response headers is malformed:

$ curl -v "https://furnish-app.herokuapp.com/api/stylequiz/1?productCategory=displaycabinet" -o /dev/null
...
< Access-Control-Allow-Methods : GET, POST, PUT, DELETE, OPTIONS

Note the space before the colon

Could you try to fix this and see if it still fails to proxy?

Aside: This might still be a regression on our side with a recent deploy, that made this case suddenly fail. We are looking into that.

2 Likes
#13

That fixed it! Excellent find, thank you so much :tada::pray:

This link now works https://www.furnish.app/api/stylequiz/1?productCategory=displaycabinet

Odd that it changed all of a sudden though, looks like the reverse proxy was more forgiving of malformed response headers before for some reason.