500 Internal Server Error on "CDN Diffing Files"

Site: fs5-staging.netlify.app (though it’s happening for multiple sites)

All of a sudden, as of yesterday afternoon, we’ve been receiving 500 errors on “CDN diffing files”:

Deploy path:        /home/circleci/project/public
Functions path:     /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to live site URL...
- Hashing files...
✔ Finished hashing 2845 files and 2 functions
- CDN diffing files...
 ›   Warning: JSONHTTPError: undefined 500
 ›   Warning: 
 ›   {
 ›     "name": "JSONHTTPError",
 ›     "status": 500,
 ›     "json": {
 ›       "status": 500,
 ›       "error": "Internal Server Error"
 ›     }
 ›   }
 ›
    JSONHTTPError: Internal Server Error

Exited with code exit status 2
CircleCI received exit code 2

hi there, sorry to be slow to reply. Are you still seeing this error?

I just noticed this reply. Yes, off-and-on. We’re currently experiencing it now, it was fine about an hour ago.

hey thomas, we think we might know what is going on, but we want to do a little more investigation first. We’ll have a chance to speak to the developer who will know for sure tomorrow, and we’ll get back to you when we can. thank you for your patience.

1 Like

Hey @thomas.milewski,
Our backend team is working on optimizing deploy performance for CLI deploys, and gave us the go-ahead to enable some of those improvements for your fs5-staging site before they roll out across the board in the next few weeks. We’d love to hear if you notice any changes (hopefully positive ones) to your deploys. And please let us know if you’d like us to enable this on any of your other sites!

1 Like

So far so good! Would you mind applying this to fs5 as well. Thank you!

Hey @thomas.milewski, we haven’t forgotten about you! Just need confirmation from our backend team before applying this since it’s not available to everyone. We’ll update here as soon as we can.

Got the go-ahead so we’ve applied that to fs5 as well!

1 Like

Hi Jen!

We’re currently seeing a TimeoutError on both instances:

Deploy path:        /home/circleci/project/public
Functions path:     /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to live site URL...
- Hashing files...
✔ Finished hashing 2057 files and 2 functions
- CDN diffing files...
✔ CDN requesting 6 files and 1 functions
- Uploading 7 files
✔ Finished uploading 7 assets
- Waiting for deploy to go live...
 ›   Warning: 
 ›   {
 ›     "name": "TimeoutError"
 ›   }
 ›
    TimeoutError: Promise timed out after 490000 milliseconds

Exited with code exit status 2
CircleCI received exit code 2

Thanks!

Hey @thomas.milewski,
One of our backend engineers dug into this and found that it seems to be a different issue than the CDN diffing timeout. Our logs show that during the upload, some files changed locally and the SHAs didn’t match what was initially sent when the deploy was created. Let me know if that lines up or if you’re still running into this when you deploy!

Hi @jen!

I appreciate your team looking into this. I’ll try and look a bit more on our end, but nothing should be changing these files between the hashing and uploads of the files. Additionally, this whole deployment process hasn’t changed for upwards of eight months. So, I’m a little :man_shrugging:t2: at this point, haha.

Right now, though, I’m seeing the following:

Deploy path:        /home/circleci/project/public
Functions path:     /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to live site URL...
- Hashing files...
✔ Finished hashing 2471 files and 2 functions
- CDN diffing files...
CircleCI received exit code 2

Thanks again for your help!

OK, so when you do something like:

do-your-build ; find /deploy -exec sha {} \; ; netlify deploy ; find /deploy -exec sha {} \;

…can you confirm that there is no difference except between the first and last run of the sha listings? I understand you haven’t changed anything explicitly, but that doesn’t mean it hasn’t been changed for you by something like a sneaky node_module or other process that you didn’t intend :slight_smile:

Thanks, @fool / @jen!

I was having some issues with sha so I went with sha256sum. Hopefully, that works.

**circleci@593e8f781018** : **~/project** $ find ./public -type f -exec sha256sum "{}" \; > before.txt ; netlify deploy --auth [AUTH] --dir public --message 'staging deploy from CircleCI' --site [SITE] --timeout 120 ; find ./public -type f -exec sha256sum "{}" \; > after.txt

Deploy path:  /home/circleci/project/public

Functions path: /home/circleci/project/functions

Configuration path: /home/circleci/project/netlify.toml

Deploying to draft URL...

- Hashing files...

✔ Finished hashing 2467 files and 2 functions

- CDN diffing files...

✔ CDN requesting 7 files and 0 functions

- Uploading 7 files

✔ Finished uploading 7 assets

 - Waiting for deploy to go live...

Warning:
› {
› “name”: “TimeoutError”
› }

TimeoutError: Promise timed out after 120000 milliseconds

**circleci@593e8f781018** : **~/project** $ diff before.txt after.txt 

**circleci@593e8f781018** : **~/project** $

Nothing changed during the deployment process.

For reference:

**circleci@593e8f781018** : **~/project** $ cat before.txt 

48e800396955a6169c73a85b11e76b228426fe48b467da30299115c6d6416af6 ./public/static/sprite-a64ab3ed3f68ff6570dd1214cdabea1d.svg

dcd87eb2d4218727e377032fa022b28e7aae070c8249235240351d95aa31fccb ./public/404/index.html

c51e70024000a60e4b188ef24587e5724bec75e12ba4dbe5a4385d58fd639912 ./public/accessibility/index.html

...

Could you let us know your CLI version, @thomas.milewski? If you aren’t already at v2.51.2 or greater, could you upgrade for me and let me know if things get any more reliable?

Hello @fool I’m now seeing this today too. Just running a manual deploy with netlify deploy

✔ CDN requesting 0 files and 1 functions
⠄ (0/1) Uploading graphql... ›   Warning: JSONHTTPError: undefined 500
 ›   Warning:
 ›   {
 ›     "name": "JSONHTTPError",
 ›     "status": 500,
 ›     "json": {
 ›       "status": 500,
 ›       "error": "Internal Server Error"
 ›     }
 ›   }
 ›

CLI version: netlify-cli/2.64.1 darwin-x64 node-v14.0.0
Although I’ve tried with an earlier version (not sure which exactly, same issue)

The site is just a test project, with very little config.

Thanks for confirming!

Unfortunately, I can’t really tell what happened in this case without more details. Would you mind re-running with the environment variable $DEBUG set to * ? That will generate a debug log that lets us correlate this with internal logs and troubleshoot, and you can attach it to a response here (or paste it in a gist and link to it).

Please please be careful when you share those logs - they WILL contain your authentication token (looks like a 64 character hexadecimal string) which you need to guard like a password, so please elide it from your output before posting!

Hey guys,

having the same problem when trying to deploy a simple redwood starter with netlify-cli/2.67.4 darwin-x64 node-v14.15.0:

Here is the debug output:

redwood-eval (master*) » DEBUG=* netlify deploy                                                                                    ~/Coding/test/redwood-eval  
  @oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/package.json +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/oclif.manifest.json +4ms
  @oclif/config:netlify-cli using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/oclif.manifest.json +0ms
  @oclif/config reading user plugins pjson /Users/username/.local/share/netlify-cli/package.json +0ms
  @oclif/config loadJSON /Users/username/.local/share/netlify-cli/package.json +1ms
  @oclif/config loading plugins [
  '@oclif/plugin-not-found',
  '@oclif/plugin-plugins',
  '@oclif/plugin-help'
] +1ms
  @oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/package.json +5ms
  @oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/package.json +0ms
  @oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/package.json +0ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/oclif.manifest.json +1ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/oclif.manifest.json +1ms
  @oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/oclif.manifest.json +1ms
  @oclif/config:@oclif/plugin-not-found using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/oclif.manifest.json +0ms
  @oclif/config:@oclif/plugin-plugins using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/oclif.manifest.json +0ms
  @oclif/config:@oclif/plugin-help using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/oclif.manifest.json +0ms
  @oclif/config config done +8ms
  @oclif/config start init hook +0ms
  netlify:netlify-cli:hooks:init start /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/hooks/init +0ms
  netlify:netlify-cli:hooks:init done +60ms
  @oclif/config init hook done +206ms
  netlify init version: @oclif/command@1.6.1 argv: [ 'deploy' ] +0ms
  @oclif/config runCommand deploy [] +5ms
  @oclif/config:netlify-cli require /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js +223ms
  @oclif/config start prerun hook +2s
  @oclif/config prerun hook done +1ms
  @oclif/config start analytics hook +485ms
  netlify:netlify-cli:hooks:analytics start /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/hooks/analytics +0ms
  netlify:netlify-cli:hooks:analytics done +1ms
  @oclif/config analytics hook done +2ms
Deploy path:        /Users/username/Coding/test/redwood-eval/web/dist
Functions path:     /Users/username/Coding/test/redwood-eval/api/dist/functions
Configuration path: /Users/username/Coding/test/redwood-eval/netlify.toml
Deploying to draft URL...
.   Hashing files...  precinct paperwork: invoking precinct +0ms
  precinct options given:  { includeCore: false } +0ms
  precinct we assume this is JS +0ms
  precinct parsed the file content into an ast +13ms
  precinct module type:  commonjs +0ms
  precinct paperwork: invoking precinct +5ms
  precinct options given:  { includeCore: false } +0ms
  precinct we assume this is JS +0ms
  precinct parsed the file content into an ast +1ms
  precinct module type:  commonjs +0ms
✔ Finished hashing 15 files and 1 functions
✔ CDN requesting 0 files and 1 functions
⠓ (0/1) Uploading graphql...Warning: JSONHTTPError: undefined 500
    at Object.warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/errors/lib/index.js:33:15)
    at warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:57:16)
    at runDeploy (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:238:9)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async DeployCommand.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:422:21)
    at async DeployCommand._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
    at async Config.runCommand (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/config/lib/config.js:172:9)
    at async Main.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/main.js:22:9)
    at async Main._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
Warning: 
{
  "name": "JSONHTTPError",
  "status": 500,
  "json": {
    "status": 500,
    "error": "Internal Server Error"
  }
}

    at Object.warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/errors/lib/index.js:33:15)
    at warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:57:16)
    at runDeploy (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:239:9)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async DeployCommand.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:422:21)
    at async DeployCommand._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
    at async Config.runCommand (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/config/lib/config.js:172:9)
    at async Main.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/main.js:22:9)
    at async Main._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
⠋ (0/1) Uploading graphql...JSONHTTPError: Internal Server Error
    at parseResponse (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:12:11)
    at runMicrotasks (<anonymous>)
    at async callMethod (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:36:26)

regards

hey there, sorry to be slow to get back to you - are you still seeing this problem when you deploy? if yes, please let us know right away.

Hey @perry, no problem. I just checked it with the latest netlify-cli/2.68.3 darwin-x64 node-v14.15.0 but the problem still exists:

...
Deploying to draft URL...
✔ Finished hashing 23 files and 1 functions
✔ CDN requesting 10 files and 1 functions
╫ (10/11) Uploading graphql... ›   Warning: JSONHTTPError: undefined 500
 ›   Warning: 
 ›   {
 ›      "name": "JSONHTTPError",
 ›      "status": 500,
 ›      "json": {
 ›        "status": 500,
 ›        "error": "Internal Server Error"
 ›      }
 ›   }
 ›
╪ (10/11) Uploading graphql...JSONHTTPError: Internal Server Error
    at parseResponse (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:12:11)
    at runMicrotasks (<anonymous>)
    at async callMethod (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:36:26)

Before that it is uploading the graphql function for round about 4 minutes.

regards

Hi, @toxsick. This stands out to me as a likely red flag hinting at the root cause:

Before that it is uploading the graphql function for round about 4 minutes.

Netlify Functions are built on-top of AWS Lambda. The fact that the upload of the function takes 4 minutes implies that it is a large bundle. Would you please confirm the size of the zip file bundle for that function?

If it is over 50 MB as a compress archive file (a zip file), then it is exceeding the limits of the underlying technology - AWS Lambda. The limits for Lambda are documented here:

Quoting that page:

| Deployment package | size 50 MB (zipped, for direct upload) |

If the zip file is smaller than that please let us know. If it is larger, something will need to be changed for it to meet the requirements for the zip file size.