Functions not uploading; deployment stuck at upload stage

I am trying to deploy my site and it is stuck on the “Netlify’s robots are busy building and deploying your site to our CDN” final step. I tried waiting for hours but it did not deploy.


Cancelling and clearing cache has not helped.

Seems like part of the problem might be that functions aren’t uploading?

I managed to deploy another site, so it seems like there’s something specific to this site. It can’t deploy an existing successful deploy, so it’s not about the code changing. It seems like it has to be something funky at Netlify/AWS.

@fool, any insights here? It’s failing in the upload stage, which is Netlify’s code that I have no ability to debug.

I made a branch without any functions, and it deployed. I tried stripping the debug symbols from my function to see if I went over some unwritten size limit for the function, but it still didn’t upload. Seems pretty squarely in the Netlify camp to resolve this one.

I tried having it deploy a function from another repo, and that worked, but it refuses to upload the function from this repo, even though the code is unchanged from what’s deployed on master.

9:34:23 PM: Starting to deploy site from 'dist'
9:34:25 PM: Calculating files to upload
9:34:25 PM: 0 new files to upload
9:34:25 PM: 1 new functions to upload
9:35:39 PM: Failed to upload file: gateway
9:35:39 PM: Section completed: deploying
9:35:39 PM: Site deploy was successfully initiated

The logs say “Site deploy was successfully initiated” even after it fails to upload, so that’s definitely a bug in the Netlify deploy process because it should really declare failure here, even if there is something (what, since I’m deploying what’s on master?) funky with my function.

The known good repo also refuses to upload my function.

9:46:49 PM: Starting to deploy site from 'public'
9:46:50 PM: Calculating files to upload
9:46:50 PM: 0 new files to upload
9:46:50 PM: 1 new functions to upload
9:48:04 PM: Failed to upload file: almanack-api
9:48:04 PM: Section completed: deploying
9:48:04 PM: Site deploy was successfully initiated

I tried commenting out almost everything in my function, and it still will not upload.

A stripped down version of the function with virtually nothing in it still won’t upload. It’s 12MB, the same as the code that does upload, so it’s not hitting an invisible file size limit.

I deleted the embeded templates in my function, and it uploaded. My current hypothesis is that the problem is your server is falsely detecting the binary as being a webpage because it contains <html> in it, and then the upload fails and everything else breaks.

@Melvin ? @luke ? @hillary ?

Removing the code for //go:embed is sufficient to make the function upload succeed. See Test deploying without background API by carlmjohnson · Pull Request #308 · spotlightpa/almanack · GitHub. This code has been in my project for almost two years at this point. It has to be on your end.

@laura? Sorry to keep being a pest, but no one from Netlify has replied to this thread to let me know they’re working on it, and I’m completely blocked by it.

Here is a minimal reproduction. Take a working repo. Add an embed.FS package variable and reference it during initialization. Upload will fail. My best guess is you have some WAF that think it’s a virus or something, but who knows.

The problem seems to have resolved itself. Did you change something on the backend?

Hi @carlmjohnson,

I was checking: Deploy details | Deploys | spotlightpa-almanack | Netlify, and it turns out, some changes to our abuse detection have got tipped off, which were preventing the functions from being uploaded.

At this stage, you do seem to be unblocked, so I’ll share your issue with the devs to filter abuse better.

Thanks for the update.