500 Internal Server Error on "CDN Diffing Files"

Got the go-ahead so we’ve applied that to fs5 as well!

1 Like

Hi Jen!

We’re currently seeing a TimeoutError on both instances:

Deploy path:        /home/circleci/project/public
Functions path:     /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to live site URL...
- Hashing files...
✔ Finished hashing 2057 files and 2 functions
- CDN diffing files...
✔ CDN requesting 6 files and 1 functions
- Uploading 7 files
✔ Finished uploading 7 assets
- Waiting for deploy to go live...
 ›   Warning: 
 ›   {
 ›     "name": "TimeoutError"
 ›   }
 ›
    TimeoutError: Promise timed out after 490000 milliseconds

Exited with code exit status 2
CircleCI received exit code 2

Thanks!

Hey @thomas.milewski,
One of our backend engineers dug into this and found that it seems to be a different issue than the CDN diffing timeout. Our logs show that during the upload, some files changed locally and the SHAs didn’t match what was initially sent when the deploy was created. Let me know if that lines up or if you’re still running into this when you deploy!

Hi @jen!

I appreciate your team looking into this. I’ll try and look a bit more on our end, but nothing should be changing these files between the hashing and uploads of the files. Additionally, this whole deployment process hasn’t changed for upwards of eight months. So, I’m a little :man_shrugging:t2: at this point, haha.

Right now, though, I’m seeing the following:

Deploy path:        /home/circleci/project/public
Functions path:     /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to live site URL...
- Hashing files...
✔ Finished hashing 2471 files and 2 functions
- CDN diffing files...
CircleCI received exit code 2

Thanks again for your help!

OK, so when you do something like:

do-your-build ; find /deploy -exec sha {} \; ; netlify deploy ; find /deploy -exec sha {} \;

…can you confirm that there is no difference except between the first and last run of the sha listings? I understand you haven’t changed anything explicitly, but that doesn’t mean it hasn’t been changed for you by something like a sneaky node_module or other process that you didn’t intend :slight_smile:

Thanks, @fool / @jen!

I was having some issues with sha so I went with sha256sum. Hopefully, that works.

**circleci@593e8f781018** : **~/project** $ find ./public -type f -exec sha256sum "{}" \; > before.txt ; netlify deploy --auth [AUTH] --dir public --message 'staging deploy from CircleCI' --site [SITE] --timeout 120 ; find ./public -type f -exec sha256sum "{}" \; > after.txt

Deploy path:  /home/circleci/project/public

Functions path: /home/circleci/project/functions

Configuration path: /home/circleci/project/netlify.toml

Deploying to draft URL...

- Hashing files...

✔ Finished hashing 2467 files and 2 functions

- CDN diffing files...

✔ CDN requesting 7 files and 0 functions

- Uploading 7 files

✔ Finished uploading 7 assets

 - Waiting for deploy to go live...

Warning:
› {
› “name”: “TimeoutError”
› }

TimeoutError: Promise timed out after 120000 milliseconds

**circleci@593e8f781018** : **~/project** $ diff before.txt after.txt 

**circleci@593e8f781018** : **~/project** $

Nothing changed during the deployment process.

For reference:

**circleci@593e8f781018** : **~/project** $ cat before.txt 

48e800396955a6169c73a85b11e76b228426fe48b467da30299115c6d6416af6 ./public/static/sprite-a64ab3ed3f68ff6570dd1214cdabea1d.svg

dcd87eb2d4218727e377032fa022b28e7aae070c8249235240351d95aa31fccb ./public/404/index.html

c51e70024000a60e4b188ef24587e5724bec75e12ba4dbe5a4385d58fd639912 ./public/accessibility/index.html

...

Could you let us know your CLI version, @thomas.milewski? If you aren’t already at v2.51.2 or greater, could you upgrade for me and let me know if things get any more reliable?

Hello @fool I’m now seeing this today too. Just running a manual deploy with netlify deploy

✔ CDN requesting 0 files and 1 functions
⠄ (0/1) Uploading graphql... ›   Warning: JSONHTTPError: undefined 500
 ›   Warning:
 ›   {
 ›     "name": "JSONHTTPError",
 ›     "status": 500,
 ›     "json": {
 ›       "status": 500,
 ›       "error": "Internal Server Error"
 ›     }
 ›   }
 ›

CLI version: netlify-cli/2.64.1 darwin-x64 node-v14.0.0
Although I’ve tried with an earlier version (not sure which exactly, same issue)

The site is just a test project, with very little config.

Thanks for confirming!

Unfortunately, I can’t really tell what happened in this case without more details. Would you mind re-running with the environment variable $DEBUG set to * ? That will generate a debug log that lets us correlate this with internal logs and troubleshoot, and you can attach it to a response here (or paste it in a gist and link to it).

Please please be careful when you share those logs - they WILL contain your authentication token (looks like a 64 character hexadecimal string) which you need to guard like a password, so please elide it from your output before posting!

Hey guys,

having the same problem when trying to deploy a simple redwood starter with netlify-cli/2.67.4 darwin-x64 node-v14.15.0:

Here is the debug output:

redwood-eval (master*) » DEBUG=* netlify deploy                                                                                    ~/Coding/test/redwood-eval  
  @oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/package.json +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/oclif.manifest.json +4ms
  @oclif/config:netlify-cli using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/oclif.manifest.json +0ms
  @oclif/config reading user plugins pjson /Users/username/.local/share/netlify-cli/package.json +0ms
  @oclif/config loadJSON /Users/username/.local/share/netlify-cli/package.json +1ms
  @oclif/config loading plugins [
  '@oclif/plugin-not-found',
  '@oclif/plugin-plugins',
  '@oclif/plugin-help'
] +1ms
  @oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/package.json +5ms
  @oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/package.json +0ms
  @oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/package.json +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/oclif.manifest.json +1ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/oclif.manifest.json +1ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/oclif.manifest.json +1ms
  @oclif/config:@oclif/plugin-not-found using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/oclif.manifest.json +0ms
  @oclif/config:@oclif/plugin-plugins using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/oclif.manifest.json +0ms
  @oclif/config:@oclif/plugin-help using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/oclif.manifest.json +0ms
  @oclif/config config done +8ms
  @oclif/config start init hook +0ms
  netlify:netlify-cli:hooks:init start /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/hooks/init +0ms
  netlify:netlify-cli:hooks:init done +60ms
  @oclif/config init hook done +206ms
  netlify init version: @oclif/command@1.6.1 argv: [ 'deploy' ] +0ms
  @oclif/config runCommand deploy [] +5ms
  @oclif/config:netlify-cli require /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js +223ms
  @oclif/config start prerun hook +2s
  @oclif/config prerun hook done +1ms
  @oclif/config start analytics hook +485ms
  netlify:netlify-cli:hooks:analytics start /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/hooks/analytics +0ms
  netlify:netlify-cli:hooks:analytics done +1ms
  @oclif/config analytics hook done +2ms
Deploy path:        /Users/username/Coding/test/redwood-eval/web/dist
Functions path:     /Users/username/Coding/test/redwood-eval/api/dist/functions
Configuration path: /Users/username/Coding/test/redwood-eval/netlify.toml
Deploying to draft URL...
.   Hashing files...  precinct paperwork: invoking precinct +0ms
  precinct options given:  { includeCore: false } +0ms
  precinct we assume this is JS +0ms
  precinct parsed the file content into an ast +13ms
  precinct module type:  commonjs +0ms
  precinct paperwork: invoking precinct +5ms
  precinct options given:  { includeCore: false } +0ms
  precinct we assume this is JS +0ms
  precinct parsed the file content into an ast +1ms
  precinct module type:  commonjs +0ms
✔ Finished hashing 15 files and 1 functions
✔ CDN requesting 0 files and 1 functions
⠓ (0/1) Uploading graphql...Warning: JSONHTTPError: undefined 500
    at Object.warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/errors/lib/index.js:33:15)
    at warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:57:16)
    at runDeploy (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:238:9)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async DeployCommand.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:422:21)
    at async DeployCommand._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
    at async Config.runCommand (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/config/lib/config.js:172:9)
    at async Main.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/main.js:22:9)
    at async Main._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
Warning: 
{
  "name": "JSONHTTPError",
  "status": 500,
  "json": {
    "status": 500,
    "error": "Internal Server Error"
  }
}

    at Object.warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/errors/lib/index.js:33:15)
    at warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:57:16)
    at runDeploy (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:239:9)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async DeployCommand.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:422:21)
    at async DeployCommand._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
    at async Config.runCommand (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/config/lib/config.js:172:9)
    at async Main.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/main.js:22:9)
    at async Main._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
⠋ (0/1) Uploading graphql...JSONHTTPError: Internal Server Error
    at parseResponse (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:12:11)
    at runMicrotasks (<anonymous>)
    at async callMethod (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:36:26)

regards

hey there, sorry to be slow to get back to you - are you still seeing this problem when you deploy? if yes, please let us know right away.

Hey @perry, no problem. I just checked it with the latest netlify-cli/2.68.3 darwin-x64 node-v14.15.0 but the problem still exists:

...
Deploying to draft URL...
✔ Finished hashing 23 files and 1 functions
✔ CDN requesting 10 files and 1 functions
╫ (10/11) Uploading graphql... ›   Warning: JSONHTTPError: undefined 500
 ›   Warning: 
 ›   {
 ›      "name": "JSONHTTPError",
 ›      "status": 500,
 ›      "json": {
 ›        "status": 500,
 ›        "error": "Internal Server Error"
 ›      }
 ›   }
 ›
╪ (10/11) Uploading graphql...JSONHTTPError: Internal Server Error
    at parseResponse (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:12:11)
    at runMicrotasks (<anonymous>)
    at async callMethod (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:36:26)

Before that it is uploading the graphql function for round about 4 minutes.

regards

Hi, @toxsick. This stands out to me as a likely red flag hinting at the root cause:

Before that it is uploading the graphql function for round about 4 minutes.

Netlify Functions are built on-top of AWS Lambda. The fact that the upload of the function takes 4 minutes implies that it is a large bundle. Would you please confirm the size of the zip file bundle for that function?

If it is over 50 MB as a compress archive file (a zip file), then it is exceeding the limits of the underlying technology - AWS Lambda. The limits for Lambda are documented here:

Quoting that page:

| Deployment package | size 50 MB (zipped, for direct upload) |

If the zip file is smaller than that please let us know. If it is larger, something will need to be changed for it to meet the requirements for the zip file size.

Hey @luke Luke, the file .netlify/functions/graphql.zip resulting from the local build has a size of 41.3 MB. Quite big, but within the limits I guess. Also when I deploy the same code from Github the Netlify build (and function deployment) works without a problem.

regards

Hi, @toxsick. There is a 26 second timeout and, if the upload takes longer than that, it will fail with a 500 response.

We are looking into extending that timeout and I do believe this is the root cause. Can you time the upload of that function to see it is taking longer than 26 seconds?

Hey @luke, it definately takes longer than 28 secs:

» time netlify deploy                                                                                                         ~/Coding/fuf/redwood-eval  
Deploy path:        /Users/<username>/Coding/redwood-eval/web/dist
Functions path:     /Users/<username>/Coding/redwood-eval/api/dist/functions
Configuration path: /Users/<username>/Coding/redwood-eval/netlify.toml
Deploying to draft URL...
✔ Finished hashing 13 files and 1 functions
✔ CDN requesting 5 files and 1 functions
🙉  (5/6) Uploading graphql... ›   Warning: JSONHTTPError: undefined 500
 ›   Warning: 
 ›   {
 ›      "name": "JSONHTTPError",
 ›      "status": 500,
 ›      "json": {
 ›        "status": 500,
 ›        "error": "Internal Server Error"
 ›      }
 ›   }
 ›
JSONHTTPError: Internal Server Error
    at parseResponse (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:12:11)
    at runMicrotasks (<anonymous>)
    at async callMethod (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:36:26)
netlify deploy  33.10s user 6.61s system 14% cpu 4:35.52 total

So around 4 and a half minutes before it failes.

regards

That deploy time seems quite unexpected! To confirm, you’re saying that this step:

✔ CDN requesting 5 files and 1 functions
🙉  (5/6) Uploading graphql... ›   Warning: JSONHTTPError: undefined 500
 ›   Warning: 
[...]

takes 4.5 minutes on its own? Seems like we are uploading very little so I am pretty confused by that. Or maybe you’re saying the build takes a long time (since that time is included from your time ... command - but without timestamps, I can’t tell which part is fast and which slow).

If the function upload part takes >30s that could definitely be a problem, just wanted to confirm that before adding you to the open report our developers are looking into :slight_smile:

Hey @fool, just gave it a fresh try with the latest cli version and YES it spends only 20s hashing and then around 4m uploading the graphql function before it failes:

» netlify -v
netlify-cli/2.69.0 darwin-x64 node-v14.15.0

» time netlify deploy
Deploy path:        /Users/username/Coding/fuf/redwood-eval/web/dist
Functions path:     /Users/username/Coding/fuf/redwood-eval/api/dist/functions
Configuration path: /Users/username/Coding/fuf/redwood-eval/netlify.toml
Deploying to draft URL...
✔ Finished hashing 13 files and 1 functions

-> hashing takes ~20s

✔ CDN requesting 0 files and 1 functions
    (0/1) Uploading graphql... ›   Warning: JSONHTTPError:  500
 ›   Warning: 
 ›   {
 ›     "name": "JSONHTTPError",
 ›     "status": 500,
 ›     "json": {
 ›       "status": 500,
 ›       "error": "Internal Server Error"
 ›     }
 ›   }
 ›
،    (0/1) Uploading graphql...    JSONHTTPError: Internal Server Error
netlify deploy  33.26s user 6.80s system 14% cpu 4:32.15 total

The graphql function it tries to upload is ~40MB in size:

» ls -lh .netlify/functions
-rw-r--r--  1 username  staff    39M Dec  1 10:30 graphql.zip

regards

Can I jump in here and ask how is the function bundled?
If you’re running netlify build and then netlify deploy (or netlify deploy --build) it will be bundled by zip it and ship it which tree shakes user code but not dependencies code.
This is a very safe way to bundle functions and works almost every time, but can lead to a big bundle.
The reason behind that behavior is that many node packages are not meant to be bundled and/or tree shaked like packages targeted for browsers.

A potential workaround for this is to use netlify-lambda to “pre-bundle” functions. netlify-lambda uses webpack under the hood and you can provide a custom webpack config to workaround specific issues.

Can you try configuring netlify-lambda and tell us if it works? If it doesn’t work can you share an example repo with the issue?

Hey @erez,

thanks for the context. Your remarks got me thinking why my relatively basic redwood app graphql function is already 40mb in size. I cleaned up a few dependencies (mainly babel and eslint stuff). This brought down the size of the graphql.zip created by netlify deploy --build to ~30mb and TADA the deployment worked :partying_face:

I also tried, as you suggested, to use netlify-lambda, but to be honest I didn’t understand / figure out which webpack config I should use.

For me the problem is resolved for now, I guess it could come back biting me when my function grows, but I will have an eye on that.

On the netlify side I think the error messages could be improved, in case you run into size limits with your functions, but I understand that my scenario is probably some kind of edge case in regard to function size and upload timings.

Thanks everybody!

2 Likes