Why is my site invoking hundreds of Next.js Server Handler functions even with low usage?

[jfrank14]

I deployed a nextjs site to Netlify on 8/31/2024 using a free plan, and have watched it use around 110,000 of these function invocations in that time, about 15,000 per day. I don’t know exactly what these calls are doing, but I thought this was at least a sign of huge web traffic, at most one function per page view. But I finally thought to install google analytics and it seems to be showing 500 invocations in the time that there were only 2 actual page views.

I suspect google is accurate here and now I really need to understand what these function invocations are doing, since I’m soon going to run out and have to start paying for them. The advantage of Netlify was to be free for low volume, but if the volume of a limited resource (function invocations) is going to be huge anyway, I need to reconsider this.

Update: I just viewed the logs and they look like this:

Sep 7, 04:23:23 PM: a2f1678e Duration: 7.51 ms	Memory Usage: 189 MB	
Sep 7, 04:23:25 PM: 3d25ff97 Duration: 34.11 ms	Memory Usage: 189 MB	
Sep 7, 04:23:38 PM: 5c632b38 Duration: 6.95 ms	Memory Usage: 189 MB	
Sep 7, 04:23:40 PM: 7bd32c37 Duration: 21.16 ms	Memory Usage: 189 MB	
Sep 7, 04:23:48 PM: e0fe5e27 Duration: 4.33 ms	Memory Usage: 189 MB	
Sep 7, 04:23:50 PM: b21b47f9 Duration: 37.39 ms	Memory Usage: 189 MB	
Sep 7, 04:23:52 PM: 0fc2b609 Duration: 27.04 ms	Memory Usage: 189 MB	

So one of these is running every few seconds all day long, and I don’t think that corresponds to site activity, so what is going on here?

[hrishikesh]

What site is this about?

[jfrank14]

The site in question is https://jennymilchman.com.

[fvieira]

Hi @hrishikesh, I’m having the exact same problem, a website (https://avaocean.no/) that was running fine under the free plan for months and suddenly it got upgraded to functions Level 1 because of a ton of requests, which apparently all return 404 as you can see on this image.

Any direction to help debug what is going on?

[fvieira]

Hi @hrishikesh, can you give any guidance on this issue, even if it’s talking to someone else?

This was my first time trying Netlify for a Next project, and since I have no idea how to debug this huge amount of requests, my only potential solution for my client is to migrate back to Vercel, which I’m really loath to do as I use Netlify for everything else.

[jfrank14]

Agreed. It’s very concerning to me that my utterly simple, mostly static site, could have 80,000 function invocations a day, and you have to pay for these, and there’s no way to know why or how to fix it.

[SamO]

Hi @fvieira thanks for reaching out! I escalated your query to our helpdesk. Our team will follow up with more insights soon via email.

Thank you for your patience!

[fvieira]

Hi @SamO, thanks for your help, I’m still waiting to be contacted, but meanwhile I think I’ve managed to fix our requests problems.
I activated Netlify Analytics, and in it I found something interesting which you can see below, which is that we have thousands of requests to php files which we don’t have.

This is clearly what is causing all the 404 errors and is probably caused by some spam bots. Since blocking these requests is not trivial, what I did was use an edge function that runs for every *.php request that quickly returns a very small 404 page instead of the normal one, saving bandwidth and preventing the request from hitting our Next function.

It’s still early to say whether it fixed everything, but it definitely the invocations to the Next function, as you can see from the drop in 404 invocation in the image below (arrow marks the time when I deployed the solution).

@jfrank14 I’m not sure if your problem is the same, but you can either pay for Netlify Analytics to find out, or test our solution to see if it solves it for you, it should be safe as long as you don’t have legitimate *.php files in your project.

The solution is simply to install @netlify/edge-functions, and add this file to the netlify/edge-functions folder.

import type { Config } from '@netlify/edge-functions'

export default async function () {
  const html404 =
    '<!DOCTYPE html><html><head><title>404 Not Found</title></head><body><h1>404 Not Found</h1></body></html>'

  return new Response(html404, {
    status: 404,
    headers: {
      'Content-Type': 'text/html',
      'netlify-cdn-cache-control':
        'durable, immutable, max-age=31536000, public',
    },
  })
}

export const config: Config = {
  cache: 'manual',
  pattern: '^.*\\.[Pp][Hh][Pp]$',
}

[jfrank14]

Excellent work finding this!

I can try this, but it bothers me that the basic analytics to discover such a thing are an upcharge, and that the responsibility to block 404’s is the responsibility of every single site, rather than something that netlify provides out of the box.

And what if the spambots stop asking for php pages and just start asking for something else, in particular something that looks just like real pages? The only way I can imagine defending against this is an explicit list of every valid page, which would be pretty tedious to maintain.

[fvieira]

Yes, I don’t think this is a perfect solution by far, both understanding what was happening and fixing it was a subpar experience.
I do appreciate the power/flexibility of edge functions, and that for what is essentially a free service, we already get a lot of value and functionality from it, but it’s still a bit worrisome that I couldn’t see a better path forward even as a paying customer other than maybe going to the Enterprise plan.

[jfrank14]

@fvieira So I did the upgrade to get the analytics and I can see 404’s, but that explains only a small part of the problem. In the past 7 days, I have 14,453 page requests plus 404’s, but 76,467 function invocations.

Now maybe that’s because the analytics only show “Top” resources, which is the top 15 of them, and there’s a long tail of 404 hits that is contributing tens of thousands more 404’s, but you can’t see them.

So now I’m even more annoyed that I’m paying for analytics that don’t really let me analyze the situation.

Also, the 404’s are mostly robots.txt and /blog/page/N, for various values of N. So the blocking rule is reasonably simple, but without more data I don’t know if it’s at all complete.

@hrishikesh @SamO Is there a way to download the FULL usage analytics, so I can explain this discrepancy? Could there be anything other than page hits that would cause a function invocation? Could there ever be more than one function invocation per page hit (remembering that these pages are all just static HTML)?

[fvieira]

My guess (and that’s all it is as I also only had partial information), is that the discrepancy is just due to it only showing the Top results, and likely blocking those will be enough as the rest that you can’t see probably follow the same pattern.

[jfrank14]

@fvieira I will give it a try, and thanks so much for discovering it.

I’m puzzled why this actually solves the problem? Why does a 404 trigger a function invocation, but rejecting it in this way (i.e. with a function) NOT count as a function invocation?

[fvieira]

Oh, it does count as a function invocation, but it’s a Edge Function invocation, not a Serverless function, and Edge functions have higher limits and way cheaper prices.

[jfrank14]

You have been far too generous with your help already, so I hate to ask, but what’s the difference between these two kinds of functions and why would one be SO much cheaper than the other?

[fvieira]

I’m going to rely on GPT to help me answer this one as it probably can explain better than me, but I’ll just add that while Serverless functions only run if you call their addresses, and are often used as lightweight APIs, Edge functions, as their name implies, run on the edge of the CDN, and they don’t have addresses, instead they intercept requests to whatever files you want (e.g. HTML, CSS, JS) before the normal processing (e.g. allowing you to block certain requests), and/or intercept responses afterwards (e.g. you could use it to inject the country of the visitor into the returned HTML).

GPT answer

Netlify’s serverless functions and edge functions both serve to run backend code without managing server infrastructure, but they operate in different environments and have different use cases:

Serverless Functions

1. Execution Location: Serverless functions run in centralized data centers.
2. Use Case: Ideal for handling standard backend tasks like CRUD operations, third-party API integrations, or custom business logic.
3. Cost: They are generally priced based on the number of executions and the execution time, which can get costly with high traffic.
4. Scalability: Automatically scales with the number of requests.
5. Latency: Higher latency compared to edge functions, especially if the user is geographically distant from the data center.

Edge Functions

1. Execution Location: Edge functions run at the edge of the network, closer to the user, utilizing a distributed network of nodes around the globe.
2. Use Case: Best for tasks that benefit from low latency, such as personalization, A/B testing, and handling user authentication at the edge.
3. Cost: Generally cheaper because they are designed to run lightweight and low-latency tasks. They are optimized for high-frequency, short-duration executions.
4. Scalability: Instantly scales, and because they are distributed, they can handle more requests efficiently.
5. Latency: Lower latency due to proximity to the user.

Why Edge Functions Can Be Cheaper

• Efficiency: Edge functions are optimized for quick executions, which use fewer computational resources and result in lower costs.
• Reduced Data Transfer: Since data doesn’t have to travel to central servers, there’s less data transfer involved, which can reduce costs.
• Caching and Distribution: Edge functions can make effective use of caching and distribute load across various nodes, minimizing the need for costly data center resources.

Edge functions are particularly cost-effective for high-volume or globally distributed applications where reducing latency and data transfer costs are critical.

[jfrank14]

Well that was very helpful, thanks. Your human touch still matters though

[jfrank14]

@fvieira So yesterday I implemented this 404 handler exactly as you showed and I tested it and did receive the 404 page in return. But today I still see no reduction in function calls!

You can see just about as many today as in the previous days.

[fvieira]

Are you sure you are not seeing the “Last 24 hours” view instead of the “7 days”? Because I see in your screenshot 12:00 AM which only shows when you are seeing the “Last 24 hours” view.

Also check on the All Statuses if you see any drop in 404 errors like I do.

[jfrank14]

I think this is the view you’re talking about:

The vertical red line is around the time I committed the code with the 404 handler, and the 404’s do not go away, as in your graph. And the “Top resources not found” analytic only lets me view the whole month, so I can’t even filter by the past 24 hours to see if, perhaps, there are just as many 404’s, but following some long tail pattern that I missed in my regex.

@hrishikesh Are you following this? Can netlify provide any official help here?