Functions - Failed to upload file

[tibotiber]

Hi, I’m having some issues deploying Netlify functions through the CI. Here is the error I’m getting:

Failed to upload file: &{Name:health Sum:c3f2203387ca2e24ec3d20d8c27359ee1b8e49e2787bba658975267c978ae874 Runtime:js Size:<nil> Path: Buffer:0xc0001f6330}

Full end of log:

12:34:14 AM: Function Dir: /opt/build/repo/functions/dist
12:34:14 AM: TempDir: /tmp/zisi-857404437
12:34:15 AM: Prepping functions with zip-it-and-ship-it 0.3.1
12:34:34 AM: [ { path: '/tmp/zisi-857404437/health.zip', runtime: 'js' },
12:34:34 AM:   { path: '/tmp/zisi-857404437/process3dModel.zip',
12:34:34 AM:     runtime: 'js' } ]
12:34:34 AM: Prepping functions complete
[...]
12:34:34 AM: Build script success
12:34:34 AM: Starting to deploy site from 'studio/build'
12:34:34 AM: Creating deploy tree 
12:34:34 AM: 0 new files to upload
12:34:34 AM: 2 new functions to upload
12:36:56 AM: Failed to upload file: &{Name:process3dModel Sum:15bec3b5d5780a2927fdb3d879651765dd97f9adcaab59915caf184e81eadc27 Runtime:js Size:<nil> Path: Buffer:0xc000934000}
12:37:04 AM: Failed to upload file: &{Name:health Sum:c3f2203387ca2e24ec3d20d8c27359ee1b8e49e2787bba658975267c978ae874 Runtime:js Size:<nil> Path: Buffer:0xc0001f6330}
12:37:04 AM: failed during stage 'deploying site': Failed to execute deploy: Upload cancelled: health
12:37:04 AM: Failing build: Failed to deploy site
12:37:05 AM: Finished processing build request in 6m21.830165011s

This gets built by babel to transpile with a node@8 target, then deployed using zip-it-and-ship-it. I have tried keeping only the simplest function ("/health" reply “OK”) but it still fails.

I’m not too sure what can cause this. I’ve searched the docs and forum but didn’t find anything else than https://answers.netlify.com/t/failed-during-stage-deploying-site/1499 which doesn’t provide path to resolve the issue. No clue in the zip-it-and-ship-it repo either.

I originally thought this was due to yarn workspaces, but I don’t think so since I’ve handled modules hoisting so zip-it-and-ship-it can work. See the related issue I opened at https://github.com/netlify/zip-it-and-ship-it/issues/37.

The Netlify site is called “smplrspace”. I have now disabled functions until this gets resolved.

Appreciate any help or pointers, thanks folks.

[futuregerald]

Hi @tibotiber,

I see in your deploy log you are deploying firebase functions. Can you clarify that? Also is your repo public? or can you share the code for your package.json and the function that’s failing? Also, have you tried renaming your environment variables? maybe try prefixing their current names.

[tibotiber]

Hi @futuregerald,

Thanks a lot for the answer.

I’m having both Netlify and Firebase functions (well, trying to) as firebase allows for longer running functions, but netlify functions come with CI and versioning managed. These are completely separate though, different packages in a monorepo.

My repo is private, but I have created a minimal reproduction repo that’s public. You can check https://github.com/tibotiber/netlify-functions-example, and the associated netlify site is called smplr-fail-example.

I’m able to reproduce now, it seems to be related to environment variables as you suggested, but I’m not able to pinpoint exactly what. With my envs it fails, if I prefix them it fails, if I remove them it works, if I replace their value by “test” it works (prefixed or not). Really can’t think of any rational reason . I’ve made sure to clear cache each time I redeploy.

Any idea?

Thanks,
Thibaut

[tibotiber]

So, I’ve tested each environment variable independently, brought it down to a specific one, “FIREBASE_SERVICE_ACCOUNT”. Of course changing its name didn’t change anything since prefixing didn’t help. Looked in the content and found nothing suspicious. It’s a firebase / gcloud service account key, a stringified JSON object with a few values including urls, emails, strings, and a ssh-like private key as one field.

Once I remove that value, say replace the value by “test”, it works. Once I add it it fails. I have another service account as another env there, it looks exactly the same, so I tried copying the value (which works fine) and then it fails.

Is there something like a maximum length of environment variables at play here? Only thing I can think of.

Note that this code doesn’t need the environment variables, they are just copied from the failing repo to reproduce the error.

[futuregerald]

Hi @tibotiber,

There is an upper limit but that shouldn’t be a problem here. How big is stringified object that you’re saving as the value?

[tibotiber]

It’s 2290 characters . I know that’s a lot but I can’t find a better way to handle google cloud / firebase service account keys than via environment variables. Note that I have 2 such environment variables, so if the upper limit is on the total of all environment variables, it could be the reason too.

If that’s not the case, any idea from the reproduction repo I’ve put together?

Thanks @futuregerald!

[futuregerald]

Hi,

As a test, can you BASE64 encode the environment variable? It may not be an issue with size. or if you can provide us with a test string of that environment variable so we can also test, and just change the values so they aren’t sharing any actual sensitive information (but the values are the same length).

[tibotiber]

Thanks Gerald,

I have tested BASE64 encoding and it did not help. I’ve created a new service account key and deactivated it immediately so I can pass the value to you. I’ve double checked and deploy fails with this value but works if i replace it by “test”. Note that there are 2 such values in my envs (in case it’s a total length thing).

Here you go:

{ "type": "service_account", "project_id": "example", "private_key_id": "0977112ade55284d19b4fafd393302681eee14e2", "private_key": "-----BEGIN PRIVATE KEY-----\nMIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQCyOIxwFpU5ilWE\nmXRwzk65giGGH/JbixNNs8LRn8VSu+fKUK6CaPXja6Klj2H2PQBDDPfHNMLqgbIP\nceqNabXKgZtN/0sjVzOA0H573K1TSDFx0R74XwJHTdmx3XFq+cMklHBY7DtAsoWU\n/jFhiY5i56qlsJ4bSIYTd1ZpYUYw51IPJ2R0CQXkfNmRbJTh37L/9GJVqmHJfkvc\nQfp++7ghtu0qmV8i18I8plbX7cmPIRPa30/rpWJk+qAzmgl9VWNxTHCata27ambi\nh3YlHVMzgOiyrhATE1wHi+csiYI7zYbcIzvdKGEd8hJqsy6IZFvonZ+LHp0AJ8cU\nYX7nUIDHAgMBAAECggEAEsjCCJhgvxFVBSZVbwRXqNbN6MCaP0rPzItLV+PSnO9A\nNYM+eXFNpVw1ZuTPavAwBGEsBnuJpcTouxcDJUeEiSUS9OwZA88Plx6ijjSKsjH0\nQC5N6Ni+uTw72zLfXuqXRlY85ypy7lVCnhsSoliowMTxKAiPCYoh05Bq8610g/fY\njnwPIK4oaz5UOa1g+ne03v51URTe5/yBf8UG/iNT/lbMRewdjPTHppvl7GTQxbTx\nO04JLU40LZvenhVosg/JIrWZjY1+7bX8FElzsQtpP0IDko6coLWxxJY37Ls4vueI\n1Gv+fIjNMbQt3wmTZ68BZiMEZSDJaTQps6l3UIPTyQKBgQDecUpwIM2r2fT6EAGe\nFqKzAKXsnY6kcOa+rEBQTwUneSRgCVW3+G3RJUZaOQVp6OLpu9wtidS+RzWuA7OT\nwdJoRfktx+30UTYYV2K2iUtVib6WSGxywr1QfJAjn8LxYMikNDkbqi5bERwcOrzr\nLuH4MMNdcrce3pJJAuHPdoDSDwKBgQDNG2wLG97TKSJTnjEkkfQR0PfDI3L2qpyS\n5zxlovh4BnAGT9E1iJLyK5JLz0OAOa7G9HAFS4IzpIDf9GqEWqvoap3ytcJs23wo\nXEdPDdXMH+Z/+qiGyYhf/rBczLghoVwsnPNzNw+tawzKdzbJ6LxLZplc51jaIskk\nBCaJGgI9yQKBgFE+D3HkiTm64T5zmiOjIMk/81n4MaDdm2kIgHTUZ42DMUXiIuyO\nT42Tj55kNRX5eOblNgVr0cetOm3T7EKLB84NwHR4EPCquJcrU2JXwqv7IyDAUtuX\nRI2g8QQR1aFTN/TBHhp8jXG1Tg8BZP6AoQeEx9XZkAg2QjdoVhDvtx0BAoGAK9Vu\nc7lTF6G02lX2frBKjvwv1x5eVWUE7UhJ2hbILy6BBnhMZ7p2XRb+vwDeliq9tXtL\n7XXDr7G6cSJVGVAfaR5P/yzlDkqh8CY4fyafyG4Q3sz880FiydEuc8a0m0tW3Zn1\nVWwxB4jXRPXRWblJdHvKztYKYSQKBI52hdpvHDECgYEA1ad0rmal5QqU3NEzRt12\n9V8rEyi98g8hUJFYINVDM2jZ8VbSNgSr59tkOkAHGCpZF5LGz4L8IxMIL074JHmr\n1KojrTJ9HMcpH5dd9Zbly7bKJZGbOd1QAXY9y6h3pGLN4KOUjCULJzCTkzalyKkT\nxOspRYw6q7VkUvf3c8TevrU=\n-----END PRIVATE KEY-----\n", "client_email": "example@appspot.gserviceaccount.com", "client_id": "114011228090578176469", "auth_uri": "https://accounts.google.com/o/oauth2/auth", "token_uri": "https://oauth2.googleapis.com/token", "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs", "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/example%40appspot.gserviceaccount.com" }

Hope this can help and thanks again. Really appreciate the help.

[futuregerald]

Thanks @tibotiber . Instead of encoding the entire serialized object, can you encode the individual secret values and save them to separate en vars and then try using them that way? You may still want to base64 encode them to preserve formatting.

[tibotiber]

Thanks again @futuregerald. I’ve tried splitting the object, the failure to upload comes down to the private key field.

• if it’s there: it fails
• if I encode it: it fails
• if I replace it by a pure string, i.e. [a-z], of the same length: it fails
• if I replace it by “test”: it succeeds

It looks like a length issue don’t you think? What are the actual limits in place? Single variable length as well as total length of all variables?

Thanks,
Thibaut

[fool]

I believe the limitation in question is an AWS limitation (https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/environments-cfg-softwaresettings.html search for “4,096”) - around all environment variables, when first concatenated like this:

VARNAME1=val1,VARNAME2=val2

and creating a string, which must be <4096 characters including commas, possibly after encoding. So a very long string like that will likely cause that kind of breakage, considering that all build variables are taken together and our own variables that are set (such as AWS_SECRET_KEY and the like to enable the integration and potentially some others like these that we set in build: https://www.netlify.com/docs/continuous-deployment/#environment-variables) are non-optional).

[tibotiber]

Wow, thanks @fool, pretty sure that’s the wall I’m hitting.

I honestly find Google Cloud a bit annoying for not having first-class support for simple secrets like everyone else. I’ll look into this for a solution and post back here what I come up with so it can help others. Worst case, I’m currently using 2 service accounts, one for CI deploy, and one for my app. I could use a single one and get around the issue, but separation of concerns on the service accounts is not gonna be great. Anyone having an idea, feel free to comment

[tibotiber]

Small update on this (better late than never ). There was no way around using 2 service accounts and I can’t possibly change the 4096 characters limit, so I’ve moved the 2 buckets on a single firebase project so I can do CI and app uploads using a single service account. Not ideal but fixed!

Thanks again for all the help on this @futuregerald and @fool. Really appreciate it.

[futuregerald]

Thanks for the update @tibotiber! Glad you found a solution.

[Kristen]

Another small update regarding this topic - we’ve updated our Functions docs to include a warning about AWS’s environment property limits to provide more guidance to folks working with Functions based on the troubleshooting discoveries in this thread. Thanks everyone for this conversation that led to a docs improvement!

[tibotiber]

That’s great news Kristen. Awesome job on the new docs :).