Netlify Environment variable resolving to undefined on some deployments

We have an application with multiple tenants deployed via Netlify. In the last few weeks many of the tenants are failing certain tasks because the key we deploy as environment variable resolved to undefined. I have checked all of the problematic deploys and the env variable is set according to the UI, but adding logging code shows that variable as undefined when running in a Netlify function.

This code has not changed in many months, and was working fine. I do not believe it would be even possible to invalidate an environment variable via code, but I have scanned the repository and this env var is accessed only once. All other variables appear to be resolving correctly.

This is an Astro site using Astro js version 2.10, with the @astrojs/netlify adapter versioned as ^2.6.0.

Some example site IDs where we are seeing this issue:

65ed38ba-bdca-4b23-8441-c09367d23772

a2e7f390-2a7f-493b-8c4f-c4562221c2b6

40990203-bca3-44e6-8ec5-6632142b2dc7

The code accesses the variable as follows:

const IIIF_KEY = import.meta.env.IIIF_KEY;

export const post: APIRoute = async ({ request, cookies }) => {
  // Verify if the user is logged in
--<snip>--

Logging added to the problematic sites show IIIF_KEY as undefined. Yet the value in the UI is valid.

Is there a way to reproduce the issue for us?

Well I could show you the logs where it is failing. As I said, it should work and has for awhile.

Hey Lorin,

Thanks for submitting a support ticket and apologies again for the hassle. We’ve followed up there!