All of a sudden receiving “forbidden” error on reverse proxy

Support
,
1

Hey all, so I have two pages in Netlify sitting behind proxies that are set up on another server. They have been working all along but all of a sudden today they are throwing the following error:

{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/starting-drone-business/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

It’s Black Friday so I’m not sure if traffic has anything to do with it. This seems like an issue on Netlify and support with the other host confirmed thinks the same thing. Really kind of desperate here any ideas?

2

Hi, @kmcaloon, can you give us an example URL which is returning the 403 response? Also, the x-nf-request-id header for the HTTP response with the 403 would also be helpful. Would you please share that with us as well?

Finally, I must point out that we don’t support using a proxy in front of our CDN as documented here:

Quoting that page:

Required best practices

Netlify’s CDN works best when it is used directly without firewalls, proxies, or other CDNs in front of it. Your sites should have DNS configured as described in our documentation. We may require you to reconfigure your sites and DNS per our recommendations before we can assist with networking issues.

3

Hi @luke, thanks for getting back to me.

So we temporarily disabled the proxy when it started giving the error. I just reenabled and now the page is actually showing up: Start Your Drone Business with DARTdrones - DARTdrones.

Don’t know if this helps at this point, but here is the header value you asked for: 57485e40-64cb-4978-ad12-1ae90ac52ddd-1490649

If we did need to serve the content behind a proxy what options might we have?

4

Hi, @kmcaloon, I’m showing the request for the x-nf-request-id posted was returned a 200 HTTP status and which resulted in us sending 51445 bytes of data.

I checked the URL in question (which was https://condescending-booth-644542.netlify.com/starting-drone-business/) and there were no 403 responses from Netlify for that URL in the last seven days.

Regarding using a proxy in front of Netlify, you may do so if you choose but this can make providing technical support more difficult and we may be limited in our ability to troubleshoot. In this case, you appear to have direct access to the proxy itself (which hopefully includes logs from the proxy) so this limitation will likely not play a large factor when troubleshooting (which it can be when the proxy is controlled by a third-party and there is no access to its logs).

If you can get the x-nf-request-id headers from the proxy’s logs, we will likely still be able to troubleshoot.

If there are other questions about this, please let us know.

5

Ok thanks for this @luke. So I don’t have direct access to the proxy but I reached out and the host told me that they are “not reporting the 403” because it is on Netlify’s end. It’s strange that the URL had nothing unsual in the logs in the last seven days. I wish I had taken a screenshot of the headers, that would have been smart huh :roll_eyes:? Anyway for some other reasons we are actually going in another direction and won’t need the proxys right now, but for future reference could you theorize into what might cause these issues?

6

I cannot think of any reason why Netlify would return a 403 for an otherwise valid HTTP request. I’ve seen issues which might generate a 404 but never a 403. Also, if we serve a 403, there is a record of the 403 in the request logs. I see no 403 requests for this site during the times mentioned.

If there is no proxy, Netlify can see all steps in serving the content and you can see all the information at the browser/request end. We have logs for the entire process and can troubleshoot it in detail.

When a proxy is introduced, now there is a hidden layer between our systems and the browsers or other software making the HTTP requests. There is a layer we cannot troubleshoot without data from the proxy.

What often occurs is this:

  • Netlify says we sent the data correctly and the proxy is at fault.
  • The proxy says they are sending the data correctly and Netlify is at fault.

If the proxy service can provide the x-nf-request-id headers for the 403s, we can troubleshoot the issue in more detail. Again, though, I see no 403s at the reported time for that site.

If the 403s were from Netlify, those responses will have an x-nf-request-id header. That header would help us to find the specific HTTP request/response which generated the 403 and we could find out why it occurred.

If we don’t have the x-nf-request-id header, then we will need additional details to research. For example:

  • the date/time/timezone for the HTTP request
  • the URL requested
  • the IP address making the request
  • the IP address that responded

Finally, if we cannot get troubleshooting details from the proxy, the only remaining option is to remove the proxy. If the issue stops when the proxy is removed, then this becomes evidence to support the hypothesis that the it is the source of the issue.

To summarize, with no proxy we can troubleshoot the request from end to end. With a proxy, however, we now need information from the proxy to troubleshoot and, without it, the only option left is to remove the proxy itself for testing.