In that case, I don’t think your service is being banned. The IP bans that we add are temporary and work on IP - domain pairs (not the path). So if some IP tried to access any page on your site more than a certain threshold, it’d be banned for some time.
I’d be happy to check further, though I’m not next to my system right now (I’m replying from phone) and there’s no one from the Support team working at this hour. I’m replying as I happened to be on-call this week and able to check as much as I could from my phone, but any further checking would require my laptop which I don’t currently have and won’t have for the next few hours. My actual shift starts even later, but I’m okay making an exception here.
Yes, I’m aware the endpoint isn’t banned per se, but the IPs used by the service hammering that endpoint.
I’ve recently made changes to our infrastructure, thus consolidating the webhook calls to originate from fewer IPs instead of a wide range. I’m relieved to finally understand the root cause since this has caused thousands of $ of damage the past month.
Could we assess if there’s a good chance, perhaps over 50%, of resolving this today? I’ve started transitioning to another service as a precaution, but I’m keen to resolve this swiftly. I appreciate your support in navigating this urgent matter.
This other IP was making requests to /.netlify/functions/getImageData (which is again not the endpoint you’re complaining about).
All this to say, while there are bans, I think the problem you’re reporting and the one I’m talking about are 2 different issues. The bans here seem to be legit as we’re banning your users and not the AWS-based service that’s contacting your endpoint.
Thanks for the extensive analysis. So the IPs banned are not related to our webhook calling service? Well that changes everything then. And those IPs you have banned are legit bots that need to be banned, no worries there.
Are you now certain no IPs are banned due calling the webhook endpoint? Because I already started coding up a proxy service which changes the IP where I’m contacting Netlify’s endpoint, but please let me know if this is redundant work.
Also, maybe helpful to you: I was mistaken that the calls to the webhook are coming from AWS IP, they don’t
I checked further and now filtered the top IP addresses in the past 7 days that are hitting the endpoint you mentioned. I don’t see at least the top 3 IPs to be banned (so chances are less that the ones after those could be).
Furthermore, I reduced the scope to checking for the past hour. I picked the top IPs from the past hour to check how many requests they are making. The highest number of requests seem to be 3 per minute:
As I mentioned before, the bans are not for paths. If a particular IP will call any path on your website over a certain threshold, it would be blocked. If an IP will call the same path on your website for a reasonable number of times, it won’t be affected. The point being made here is that, the path is irrelevant, the IP ↔ domain relation is more important.
Regarding your issue, so far I have not found any evidence of Netlify causing any type of issue for the mentioned endpoint. Based on the logs, the endpoint seems to be consistently active and responding with 200s all around. The top IPs that requested that endpoint don’t seem to be blocked.
If you’re still running into issues, I’m happy to check further if you can provide a specific set of IPs to check for, or a more robust reproduction of the issue.
I understand it’s not the endpoint that’s being blocked, but the IPs—I was just not clear earlier in my reply.
My main concern is to make sure that no IPs accessing that endpoint are banned. You checked the top IPs - does that include a large percentage of the IPs calling the endpoint? So I can now be definitely sure it’s not Netlify that’s causing the blocking?
That’s the part I cannot say with 100% certainty. There are several IPs accessing your endpoint and manually checking each IP for a ban is not easy. I checked the top few and they didn’t appear to be banned, so I think it could be safe to assume that if the top IPs didn’t get banned, others should not.
From my end, I tried checking what I could with the provided information. I checked the list of IPs that were banned for your domain and found out that they never tried to access the endpoint. I also checked the top IPs that accessed your endpoint that checked that they were not banned. In both these scenarios, I think your concern is addressed that Netlify is not causing the issue you’re seeing.
The only way to be 100% certain would be when you would share us some example IPs that you think were banned, so we can check our logs for those specific IPs. That’s a much easier search to conduct.
Note that, currently all your requests are marked as coming from Cloudflare (as you’ve configured). So we’re filtering and matching the requests based on the IP sent by Cloudflare in x-forwarded-for header. We use that as the IP to identify the users as the actual IP of the requests is Cloudflare.
