Seeing inconsistency with `client-ip:` headers in HTTP request proxied from Netlify

Support
1

One of our common customers has been complaining about increased error rates and we’ve tracked the cause to the fact that some of your proxy nodes are sending out conflicting information.

It’s… acceptable… if the only supported header is x-nf-client-connection-ip, but can you fix your setup to not send garbage in the other headers?

Looking back into application error logs, there was a period between Mar 16 and Mar 23 when we would occasionally see 127.0.0.1 in client-ip, an instance on Apr 14 when we saw two different globally routable IPs:

ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="80.7.…" HTTP_X_FORWARDED_FOR="85.115.…")

(note that in these errors, x-forwarded-for comes from x-nf-client-connection-ip; we normalise it elsewhere)

But as of Jun 25 we’ve been seeing internally reserved IPs show up very frequently:

ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.114" HTTP_X_FORWARDED_FOR="2600:1700:…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.114" HTTP_X_FORWARDED_FOR="199.59.…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.17" HTTP_X_FORWARDED_FOR="54.221.…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.106" HTTP_X_FORWARDED_FOR="3.238.…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.17" HTTP_X_FORWARDED_FOR="50.229.…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.124" HTTP_X_FORWARDED_FOR="2003:e9:…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.106" HTTP_X_FORWARDED_FOR="68.132.…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.88" HTTP_X_FORWARDED_FOR="2603:7000:…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.17" HTTP_X_FORWARDED_FOR="107.150.…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.17" HTTP_X_FORWARDED_FOR="107.150.…")
ActionDispatch::RemoteIp::IpSpoofAttackError (IP spoofing attack?! HTTP_CLIENT_IP="100.64.0.114" HTTP_X_FORWARDED_FOR="24.108.…")

And the customer’s users have started to complain about increased error rates as a result.

3

Hi, @Supermathie. I’ve split this post into a new topic as I want to be able to keep the previous thread focused on the discussion of if the client-ip: header is supported or not.

If I understand correctly, you have a more specific issue here which is that this header’s content (supported or not) has been changing over time for HTTP requests which are proxied to your API from Netlify. If my understanding is incorrect, please do feel free to correct me.

I believe our common customer has opened a support ticket with us about this and I’m getting permission to CC you on that ticket so we can share more details in a private communication. (Our privacy policy forbids sharing IP addresses for example and I personally really appreciate that you took the time to obscure the last two bytes of any public IPs above.)

So, we should be adding you to that support ticket shortly. I’ll likely be filing an issue and escalating this to our developers to resolve. If we find a solution or workaround for this in the ticket, I’ll loop back here to include that information here.

2 Likes
4

Hi, @Supermathie. I received permission and you should receive an email about the ticket (# 62021).

If you do not receive the email or if you would prefer to communicate here, please do feel free to reply here anytime.

1 Like
5

Sure! Also wanted to point out that other posts have identified this as well: "Client-Ip" contains an unroutable "100.64.x.x" address

1 Like