Hi, I’m having some issues deploying Netlify functions through the CI. Here is the error I’m getting:
Failed to upload file: &{Name:health Sum:c3f2203387ca2e24ec3d20d8c27359ee1b8e49e2787bba658975267c978ae874 Runtime:js Size:<nil> Path: Buffer:0xc0001f6330}
Full end of log:
12:34:14 AM: Function Dir: /opt/build/repo/functions/dist
12:34:14 AM: TempDir: /tmp/zisi-857404437
12:34:15 AM: Prepping functions with zip-it-and-ship-it 0.3.1
12:34:34 AM: [ { path: '/tmp/zisi-857404437/health.zip', runtime: 'js' },
12:34:34 AM: { path: '/tmp/zisi-857404437/process3dModel.zip',
12:34:34 AM: runtime: 'js' } ]
12:34:34 AM: Prepping functions complete
[...]
12:34:34 AM: Build script success
12:34:34 AM: Starting to deploy site from 'studio/build'
12:34:34 AM: Creating deploy tree
12:34:34 AM: 0 new files to upload
12:34:34 AM: 2 new functions to upload
12:36:56 AM: Failed to upload file: &{Name:process3dModel Sum:15bec3b5d5780a2927fdb3d879651765dd97f9adcaab59915caf184e81eadc27 Runtime:js Size:<nil> Path: Buffer:0xc000934000}
12:37:04 AM: Failed to upload file: &{Name:health Sum:c3f2203387ca2e24ec3d20d8c27359ee1b8e49e2787bba658975267c978ae874 Runtime:js Size:<nil> Path: Buffer:0xc0001f6330}
12:37:04 AM: failed during stage 'deploying site': Failed to execute deploy: Upload cancelled: health
12:37:04 AM: Failing build: Failed to deploy site
12:37:05 AM: Finished processing build request in 6m21.830165011s
This gets built by babel to transpile with a node@8 target, then deployed using zip-it-and-ship-it. I have tried keeping only the simplest function (“/health” reply “OK”) but it still fails.
I see in your deploy log you are deploying firebase functions. Can you clarify that? Also is your repo public? or can you share the code for your package.json and the function that’s failing? Also, have you tried renaming your environment variables? maybe try prefixing their current names.
I’m having both Netlify and Firebase functions (well, trying to) as firebase allows for longer running functions, but netlify functions come with CI and versioning managed. These are completely separate though, different packages in a monorepo.
I’m able to reproduce now, it seems to be related to environment variables as you suggested, but I’m not able to pinpoint exactly what. With my envs it fails, if I prefix them it fails, if I remove them it works, if I replace their value by “test” it works (prefixed or not). Really can’t think of any rational reason . I’ve made sure to clear cache each time I redeploy.
So, I’ve tested each environment variable independently, brought it down to a specific one, “FIREBASE_SERVICE_ACCOUNT”. Of course changing its name didn’t change anything since prefixing didn’t help. Looked in the content and found nothing suspicious. It’s a firebase / gcloud service account key, a stringified JSON object with a few values including urls, emails, strings, and a ssh-like private key as one field.
Once I remove that value, say replace the value by “test”, it works. Once I add it it fails. I have another service account as another env there, it looks exactly the same, so I tried copying the value (which works fine) and then it fails.
Is there something like a maximum length of environment variables at play here? Only thing I can think of.
Note that this code doesn’t need the environment variables, they are just copied from the failing repo to reproduce the error.
It’s 2290 characters . I know that’s a lot but I can’t find a better way to handle google cloud / firebase service account keys than via environment variables. Note that I have 2 such environment variables, so if the upper limit is on the total of all environment variables, it could be the reason too.
If that’s not the case, any idea from the reproduction repo I’ve put together?
As a test, can you BASE64 encode the environment variable? It may not be an issue with size. or if you can provide us with a test string of that environment variable so we can also test, and just change the values so they aren’t sharing any actual sensitive information (but the values are the same length).
I have tested BASE64 encoding and it did not help. I’ve created a new service account key and deactivated it immediately so I can pass the value to you. I’ve double checked and deploy fails with this value but works if i replace it by “test”. Note that there are 2 such values in my envs (in case it’s a total length thing).
Thanks @tibotiber . Instead of encoding the entire serialized object, can you encode the individual secret values and save them to separate en vars and then try using them that way? You may still want to base64 encode them to preserve formatting.
and creating a string, which must be <4096 characters including commas, possibly after encoding. So a very long string like that will likely cause that kind of breakage, considering that all build variables are taken together and our own variables that are set (such as AWS_SECRET_KEY and the like to enable the integration and potentially some others like these that we set in build: Build configuration overview | Netlify Docs) are non-optional).
Wow, thanks @fool, pretty sure that’s the wall I’m hitting.
I honestly find Google Cloud a bit annoying for not having first-class support for simple secrets like everyone else. I’ll look into this for a solution and post back here what I come up with so it can help others. Worst case, I’m currently using 2 service accounts, one for CI deploy, and one for my app. I could use a single one and get around the issue, but separation of concerns on the service accounts is not gonna be great. Anyone having an idea, feel free to comment
Small update on this (better late than never ). There was no way around using 2 service accounts and I can’t possibly change the 4096 characters limit, so I’ve moved the 2 buckets on a single firebase project so I can do CI and app uploads using a single service account. Not ideal but fixed!
Thanks again for all the help on this @futuregerald and @fool. Really appreciate it.
Another small update regarding this topic - we’ve updated our Functions docs to include a warning about AWS’s environment property limits to provide more guidance to folks working with Functions based on the troubleshooting discoveries in this thread. Thanks everyone for this conversation that led to a docs improvement!
. I’ve made sure to clear cache each time I redeploy.
. I know that’s a lot but I can’t find a better way to handle google cloud / firebase service account keys than via environment variables. Note that I have 2 such environment variables, so if the upper limit is on the total of all environment variables, it could be the reason too.
). There was no way around using 2 service accounts and I can’t possibly change the 4096 characters limit, so I’ve moved the 2 buckets on a single firebase project so I can do CI and app uploads using a single service account. Not ideal but fixed!