That is weird indeed. As for the analytics, you should be able to see a 24 hours view by changing at the top. Other than the regex being insufficient, I’m not sure what may be going on.
Ah, I see that, and when I set it to last 24hrs and watched for a few minutes, I see specific page 404 counts increasing, so they are not being handled by the edge function even though when I request such pages, I see the 404 response that is returned by the edge function.
In case I wasn’t being very clear, if I request /blog/page/2, I get the 404 returned by the edge function:
And the pattern matching is correct, because if I request /blog/page/abc, I get the 404 for my site from nextjs:
And yet I see the 404 count for /blog/page/2 in the “Top resources not found” counter.
Oh, now that I think about it, I don’t think the edge function will have any impact in the 404s you see in the analytics, because those include the 404s returned by the edge function.
So you should be seeing the reduction in the 404s of the serverless function (which you are not for some reason), but you shouldn’t see any reduction in the analytics…
The question now is how you’ll find which 404 addresses are being requested that are bypassing your edge function and hitting the serverless function. Maybe you could try logging those addresses somehow inside Next (haven’t tested this)?
@hrishikesh @SamO is there anyway to get the full analytics, i.e. all the addresses being requested, so that @jfrank14 can see which other 404s he needs to block? I would expect paying for analytics would give us more than just a top 15 of requests…
@fvieira @jfrank14 Thanks so much for raising this concern regarding 404 requests causing function invocations. We’ve escalated it to our developers for greater visibility on this issue affecting you two as well as other customers on Netlify. While I don’t have an immediate workaround outside of what you two have been discussing here, we will be looking at potential solutions going forward.
Regarding full analytics for @jfrank14 to better understand what needs to be blocked, we’re looking into what information we can provide that would be helpful. I do see a ticket already exists in our helpdesk, #297858. We will be following up there when we have more information to share.
If y’all have additional questions, feel free to raise them here and we’ll do our best to address them!
@sid.m I’m no longer sure that having the full list of 404’s would help, since it’s clear that even the 404’s that are blocked by my edge function still show up in this list. Having the rest of that list would possibly let me block other patterns, but that won’t help me because blocking them doesn’t seem to help.
This function invocation situation is a serious problem, because it means that sites are charged for something that they can’t control and that isn’t organic usage. According to the site analytics (and more or less corroborated by Google Analytics), my site is serving around 300 pages a day. This is a low volume site that’s important to me but doesn’t bring in any revenue, so putting it on a free (for low usage) provider makes sense. But if spambots asking for pages that don’t even exist can force me to spend $25/month on this, it’s no longer cost competitive with GoDaddy or some other hosting provider that doesn’t charge in this way (and which also provides 24/7 phone tech support).
@jfrank14 I completely understand your concerns here and genuinely appreciate the feedback regarding your experience on the platform. I’ve communicated that feedback to our development team for additional context on this issue.
Is there anything we can do in the meantime to help you have a better experience? You stated the full list of 404s wouldn’t be helpful, is there anything we could provide that would be helpful?
Can you comment on whether blocking a url via edge function should make it not use a function invocation? @fvieira seems to have had success doing this, but when I did the same thing, it didn’t change anything. I’m not clear from the docs what does or does not invoke a function, and if blocking via edge function means that it does NOT invoke a server function. Certainly, I AM blocking 404’s in this way, and yet they are getting through as invocations anyway.
Hi @jfrank14 - we’ve followed up on your ticket in our helpdesk to get more details about how you’re implementing that edge function. Please take a look and follow up there! Thanks so much.
I have responded to that ticket. The advice isn’t really helpful because it doesn’t answer the basic question about why url patterns that are clearly being handled and rejected by the edge function are still showing up in “Top resources not found”. I also can’t get a clear answer about whether such pages count as function invocations.
It is now only 1/3rd of the way through the month and I have again used 75% of my function invocations, which is going to result in another upcharge, and I am getting very frustrated here.
@jfrank14 we’ve followed up in your ticket