Sporadic errors: "This edge function has crashed - The deployment failed while serving the request"

Sitename: bls-io

Hi there,

We have recently added a single, simple edge function that adds a cookie to tag users with their geo. Here’s the code:

// netlify/edge-functions/injectGeoCookie.ts
import type { Context } from 'netlify:edge';

export default (_: Request, ctx: Context): void => {
  const geo: Context['geo'] = ctx.geo || {};
  ctx.cookies.set({
    name: 'bls.country',
    value: geo?.country?.code || 'UNKNOWN',
  });
};
// netlify.toml
[[edge_functions]]
  path = "/s/*" 
  function = "injectGeoCookie"
[[edge_functions]]
  path = "/z/*" 
  function = "injectGeoCookie"

(the /s/ and /z/ routes are where our app lives).

We deployed this to production on June 23rd 2022 and have since had sporadic complaints from users that they cannot use our app because when visiting it they don’t get shown our app but just a Netlify error message.
One such user complaint came in on June 29th at around 3pm CEST.

I tried to reproduce but it worked for me at first.
I am based in Berlin, the user was based in Brazil. So I VPN’d into Brazil, and sure enough I was able to reproduce:

I immediately checked the Edge Function Log but couldn’t see anything:

I then triggered a redeploy via Deploys → Trigger Deploy → Deploy Site in the Netlify console. That fixed it!

So it looks like there the issue might have been caused by a corrupt Edge Function deploy in the Brazilian Point of Presence.

I understand that Edge Functions are still Beta and that we might have been to eager with using them in production already.

But, before we move away from Edge functions again, my questions:

  1. Is this a known issue?
  2. Is there anything we can do from our end to prevent this?

Thanks,
Yanick

1 Like

Hey Yanick, greetings from Potsdam!

We had a small incident related to Edge Functions on 22nd of June, which would’ve resulted in the same behaviour that you’re seeing from Brazil. But judging by the time of your Deploy, it doesn’t look like your site should’ve been subject to this.

I looked at the logs for the Request ID you sent, but didn’t find anything. The error message “The deployment failed” can occur for a lot of different error causes. We’re currently working with Deno to improve our observability into these kinds of errors. Once that’s in place, we should be able to debug this better.

How’s your experience with Edge Functions otherwise? Are you happy with the feature / is there anything missing that we should add? (apart from fixing these bugs, obviously ^^)

Hey Simon, greetings from Kreuzberg and thanks for the swift reply!

So we had another incident today where a user from Asia reported outages.
It worked for me in Europe, then I VPN’d into Asia and sure enough:

I tried a couple times, it failed each time with status code 502 and Response No resource with given identifier found.
Here are some request IDs:
01G79MD1P63NDA26HP4NY2PG2H
01G79MGN8646WXZ2BFHVNZD8J1
01G79MKHVE1NH3GEAR2NM1E42Z
01G79MMWE1MA24045N35EW00WN

10 minutes later everything worked again, the issue resolved itself.

Was this incident also related to edge functions, despite the error message here not talking about Edge functions? We think yes, because it occurred exclusively on the route where we use the edge function. We have another route that runs the same code but without the edge function, and that route worked.
Also, the issue resolved itself after 10 minutes without us taking any action.
So our current hypothesis that this incident was also caused by the edge function.

For now, we will move away from Netlify Edge functions and use something else for our localization - the current stability risk is too high for us. Once it’s stable we would be happy to try again.

Kind regards,
Yanick

I looked at these request IDs, and couldn’t find anything again. I reached out to the Deno people to find out more about this, and will ping back once we know more :+1:

Is there a way to use https://deno.land/x/sentry@deno or any other packages to log the errors occurred in edge functions?

First of all: The Problem that Yanick is facing probably lives somewhere in Netlify’s or Deno’s infrastructure. So Sentry would probably not catch these errors.

If you want to use Sentry though, your best bet is probably the @sentry/browser package with a HTTP/HTTPS transport. You’d need to write your own wrapper to adapt it to Netlify Edge Functions.

I’m not sure if the Sentry package you linked is an official one (the repository seems to be private), so take this with a grain of salt.

2 Likes

Thanks for the detailed explanation, I’m new to Deno and not entirely sure what to expect from edge functions.
I stumbled upon GitHub - robertcepa/toucan-js: Cloudflare Workers client for Sentry and it seemed to be designed for Cloudflare workers only. I wonder if this will be a better choice?
If so, what should I do to ensure it works with edge functions? Just by testing if it can run properly in my local deno installation? Sorry if the answer is obvious as I’m new to Deno.
I fully understand even if this works it will catch user code errors instead of problems in Netlify infrastructure.

not entirely sure what to expect from edge functions

That’s good feedback! Could you share what’s missing, is there any documentation / content we could write to make this easier and more obvious? We’d love for Edge Functions to be easy to use + understand, and every bit of feedback is very much appreciated :slight_smile:

I stumbled upon toucan-js, is this a better choice?

I looked at the source code of that repository, and it seems to be custom-built for Cloudflare workers. I’d probably try my luck with the @sentry/browser approach I listed above, and see how good it works :slight_smile:
(btw, I checked back with the Sentry team, and can confirm that the repository is indeed not official.)

Hi,
Thanks for the promptly reply.
I guess I would like to hear the answers to the following questions the most:
How should I setup the local environment for edge functions development?
I’ve installed deno but obviously there is still some gaps between running deno programs locally vs. edge functions. For example I cannot test routing with local deno installation. As of now the developer experience leaves a lot to desire. Hopefully there are something in your plan.

Have you checked out:

1 Like

I’m getting this error a few seconds after deploying. I see the error once or twice and then I don’t see it anymore.

Some ids to check:

  • 01G8NFZQDFVXFX08H6YYY7NVEA
  • 01G8NK8PKQ5NXHTGNP78QPKK60

That is fascinating, I digged into https://cli.netlify.com/ and Get started with Edge Functions | Netlify Docs however it was not mentioned there.

This was in the Deno newsletter today: GitHub - GJZwiers/sentry_deno: Unofficial port of the Sentry SDK for JavaScript to Deno. It’s an unofficial port of the Sentry SDK to Deno. Thought of you when I read it, would love to hear wether it works good for you!

1 Like

I just tried it out today, it works however we used to put build version (git commit hash) and DSN in environment variables, it seemed Netlify edge function is not offering such ability so we have to go an extra mile to inject them at build time…

== Update ==
Nope, speaking too quick, Sentry for Deno cannot work properly in edge function:


Because it is trying to access location?

environment variables can be accessed via Deno.env: Environment variables | Edge Functions on Netlify

Can you elaborate on why this doesn’t work? The screenshot looks like it’s from Sentry to me, which looks like a success :smiley:

Hi,
Deno.env.get("MY_IMPORTANT_VARIABLE");
Works, I’ll see if I can automate the variable setting with:
netlify env:set MY_ENV_VAR "some value"
It seemed most documents indicates environment variable set in UI is used for build time only…

Regarding Sentry integration, it seemed the package will try to access location global variable by default(since it assume it is by default browser environment?) thus throwing the error.
The original error have nothing to do with location at all.

I don’t believe so. Almost all of the Netlify-official document that I’ve reviewed or read, says that variables in netlify.toml are build-time only. Variables in UI are available during runtime. If you indeed saw us say otherwise somewhere, let us know and we can change that.

I think the developers of Sentry for Deno would be super happy about receiving a bug report about this! This doesn’t sound like intended behaviour then.

Hi, I am also facing an issue with edge functions sporadically crashing. I am not using any build process, and I wrote the functions in JS, not TS. It seems that certain deploys of my website have no problem with my one edge function, while other deploys always fail, even if I manually empty cache and rebuild from the same git commit. This happens even if I don’t modify the file for the edge function. I hope that I’m not derailing the conversation; please let me know if I am!

import { Context } from "netlify:edge";

export default async (request, context) => {
	context.log("icon.js");
	// get SVG content
	const response = await context.next();
	const page = await response.text();

	let color = new URL(request.url).searchParams.get("color") || "#000000";
	if (color != "transparent" && color.match(/^[0-9A-F]+$/i)) {
		switch (color.length) {
			case 3:
			case 4:
			case 6:
			case 8:
				color = `#${color}`;
		}
	}
	const updatedSVG = page.replaceAll("{{color}}", color);
	return new Response(updatedSVG, response);
};

This is just a simple text replacement script.

If you did that, you do not need this line:

That line is just importing types for context. It’s irrelevant in a JS file.

In any case, that’s not the actual issue. If you can share your site name or some request IDs like the people above have shared, we can check the issue. The code seems to be otherwise correct.