Site: fs5-staging.netlify.app (though it’s happening for multiple sites)
All of a sudden, as of yesterday afternoon, we’ve been receiving 500 errors on “CDN diffing files”:
Deploy path: /home/circleci/project/public
Functions path: /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to live site URL...
- Hashing files...
✔ Finished hashing 2845 files and 2 functions
- CDN diffing files...
› Warning: JSONHTTPError: undefined 500
› Warning:
› {
› "name": "JSONHTTPError",
› "status": 500,
› "json": {
› "status": 500,
› "error": "Internal Server Error"
› }
› }
›
JSONHTTPError: Internal Server Error
Exited with code exit status 2
CircleCI received exit code 2
perry
August 17, 2020, 8:16pm
2
hi there, sorry to be slow to reply. Are you still seeing this error?
I just noticed this reply. Yes, off-and-on. We’re currently experiencing it now, it was fine about an hour ago.
perry
August 19, 2020, 9:50pm
4
hey thomas, we think we might know what is going on, but we want to do a little more investigation first. We’ll have a chance to speak to the developer who will know for sure tomorrow, and we’ll get back to you when we can. thank you for your patience.
1 Like
jen
August 20, 2020, 4:02pm
5
Hey @thomas.milewski ,
Our backend team is working on optimizing deploy performance for CLI deploys, and gave us the go-ahead to enable some of those improvements for your fs5-staging
site before they roll out across the board in the next few weeks. We’d love to hear if you notice any changes (hopefully positive ones) to your deploys. And please let us know if you’d like us to enable this on any of your other sites!
1 Like
So far so good! Would you mind applying this to fs5
as well. Thank you!
jen
August 25, 2020, 7:12pm
8
Hey @thomas.milewski , we haven’t forgotten about you! Just need confirmation from our backend team before applying this since it’s not available to everyone. We’ll update here as soon as we can.
jen
August 25, 2020, 8:45pm
10
Got the go-ahead so we’ve applied that to fs5
as well!
1 Like
Hi Jen!
We’re currently seeing a TimeoutError on both instances:
Deploy path: /home/circleci/project/public
Functions path: /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to live site URL...
- Hashing files...
✔ Finished hashing 2057 files and 2 functions
- CDN diffing files...
✔ CDN requesting 6 files and 1 functions
- Uploading 7 files
✔ Finished uploading 7 assets
- Waiting for deploy to go live...
› Warning:
› {
› "name": "TimeoutError"
› }
›
TimeoutError: Promise timed out after 490000 milliseconds
Exited with code exit status 2
CircleCI received exit code 2
Thanks!
jen
September 8, 2020, 6:56pm
12
Hey @thomas.milewski ,
One of our backend engineers dug into this and found that it seems to be a different issue than the CDN diffing timeout. Our logs show that during the upload, some files changed locally and the SHAs didn’t match what was initially sent when the deploy was created. Let me know if that lines up or if you’re still running into this when you deploy!
Hi @jen !
I appreciate your team looking into this. I’ll try and look a bit more on our end, but nothing should be changing these files between the hashing and uploads of the files. Additionally, this whole deployment process hasn’t changed for upwards of eight months. So, I’m a little at this point, haha.
Right now, though, I’m seeing the following:
Deploy path: /home/circleci/project/public
Functions path: /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to live site URL...
- Hashing files...
✔ Finished hashing 2471 files and 2 functions
- CDN diffing files...
CircleCI received exit code 2
Thanks again for your help!
fool
September 9, 2020, 8:07pm
15
OK, so when you do something like:
do-your-build ; find /deploy -exec sha {} \; ; netlify deploy ; find /deploy -exec sha {} \;
…can you confirm that there is no difference except between the first and last run of the sha listings? I understand you haven’t changed anything explicitly, but that doesn’t mean it hasn’t been changed for you by something like a sneaky node_module or other process that you didn’t intend
Thanks, @fool / @jen !
I was having some issues with sha
so I went with sha256sum
. Hopefully, that works.
**circleci@593e8f781018** : **~/project** $ find ./public -type f -exec sha256sum "{}" \; > before.txt ; netlify deploy --auth [AUTH] --dir public --message 'staging deploy from CircleCI' --site [SITE] --timeout 120 ; find ./public -type f -exec sha256sum "{}" \; > after.txt
Deploy path: /home/circleci/project/public
Functions path: /home/circleci/project/functions
Configuration path: /home/circleci/project/netlify.toml
Deploying to draft URL...
- Hashing files...
✔ Finished hashing 2467 files and 2 functions
- CDN diffing files...
✔ CDN requesting 7 files and 0 functions
- Uploading 7 files
✔ Finished uploading 7 assets
- Waiting for deploy to go live...
Warning:
› {
› “name”: “TimeoutError”
› }
›
TimeoutError: Promise timed out after 120000 milliseconds
**circleci@593e8f781018** : **~/project** $ diff before.txt after.txt
**circleci@593e8f781018** : **~/project** $
Nothing changed during the deployment process.
For reference:
**circleci@593e8f781018** : **~/project** $ cat before.txt
48e800396955a6169c73a85b11e76b228426fe48b467da30299115c6d6416af6 ./public/static/sprite-a64ab3ed3f68ff6570dd1214cdabea1d.svg
dcd87eb2d4218727e377032fa022b28e7aae070c8249235240351d95aa31fccb ./public/404/index.html
c51e70024000a60e4b188ef24587e5724bec75e12ba4dbe5a4385d58fd639912 ./public/accessibility/index.html
...
fool
September 14, 2020, 3:06am
17
Could you let us know your CLI version, @thomas.milewski ? If you aren’t already at v2.51.2 or greater, could you upgrade for me and let me know if things get any more reliable?
edamame
September 29, 2020, 11:03pm
18
Hello @fool I’m now seeing this today too. Just running a manual deploy with netlify deploy
✔ CDN requesting 0 files and 1 functions
⠄ (0/1) Uploading graphql... › Warning: JSONHTTPError: undefined 500
› Warning:
› {
› "name": "JSONHTTPError",
› "status": 500,
› "json": {
› "status": 500,
› "error": "Internal Server Error"
› }
› }
›
CLI version: netlify-cli/2.64.1 darwin-x64 node-v14.0.0
Although I’ve tried with an earlier version (not sure which exactly, same issue)
The site is just a test project, with very little config.
fool
September 30, 2020, 7:55pm
19
Thanks for confirming!
Unfortunately, I can’t really tell what happened in this case without more details. Would you mind re-running with the environment variable $DEBUG
set to *
? That will generate a debug log that lets us correlate this with internal logs and troubleshoot, and you can attach it to a response here (or paste it in a gist and link to it).
Please please be careful when you share those logs - they WILL contain your authentication token (looks like a 64 character hexadecimal string) which you need to guard like a password, so please elide it from your output before posting!
toxsick
November 5, 2020, 7:45am
20
Hey guys,
having the same problem when trying to deploy a simple redwood starter with netlify-cli/2.67.4 darwin-x64 node-v14.15.0
:
Here is the debug output:
redwood-eval (master*) » DEBUG=* netlify deploy ~/Coding/test/redwood-eval
@oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli +0ms
@oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/package.json +0ms
@oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/oclif.manifest.json +4ms
@oclif/config:netlify-cli using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/oclif.manifest.json +0ms
@oclif/config reading user plugins pjson /Users/username/.local/share/netlify-cli/package.json +0ms
@oclif/config loadJSON /Users/username/.local/share/netlify-cli/package.json +1ms
@oclif/config loading plugins [
'@oclif/plugin-not-found',
'@oclif/plugin-plugins',
'@oclif/plugin-help'
] +1ms
@oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found +0ms
@oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/package.json +5ms
@oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins +0ms
@oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/package.json +0ms
@oclif/config reading core plugin /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help +0ms
@oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/package.json +0ms
@oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/oclif.manifest.json +1ms
@oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/oclif.manifest.json +1ms
@oclif/config loadJSON /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/oclif.manifest.json +1ms
@oclif/config:@oclif/plugin-not-found using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-not-found/oclif.manifest.json +0ms
@oclif/config:@oclif/plugin-plugins using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-plugins/oclif.manifest.json +0ms
@oclif/config:@oclif/plugin-help using manifest from /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/plugin-help/oclif.manifest.json +0ms
@oclif/config config done +8ms
@oclif/config start init hook +0ms
netlify:netlify-cli:hooks:init start /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/hooks/init +0ms
netlify:netlify-cli:hooks:init done +60ms
@oclif/config init hook done +206ms
netlify init version: @oclif/command@1.6.1 argv: [ 'deploy' ] +0ms
@oclif/config runCommand deploy [] +5ms
@oclif/config:netlify-cli require /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js +223ms
@oclif/config start prerun hook +2s
@oclif/config prerun hook done +1ms
@oclif/config start analytics hook +485ms
netlify:netlify-cli:hooks:analytics start /Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/hooks/analytics +0ms
netlify:netlify-cli:hooks:analytics done +1ms
@oclif/config analytics hook done +2ms
Deploy path: /Users/username/Coding/test/redwood-eval/web/dist
Functions path: /Users/username/Coding/test/redwood-eval/api/dist/functions
Configuration path: /Users/username/Coding/test/redwood-eval/netlify.toml
Deploying to draft URL...
. Hashing files... precinct paperwork: invoking precinct +0ms
precinct options given: { includeCore: false } +0ms
precinct we assume this is JS +0ms
precinct parsed the file content into an ast +13ms
precinct module type: commonjs +0ms
precinct paperwork: invoking precinct +5ms
precinct options given: { includeCore: false } +0ms
precinct we assume this is JS +0ms
precinct parsed the file content into an ast +1ms
precinct module type: commonjs +0ms
✔ Finished hashing 15 files and 1 functions
✔ CDN requesting 0 files and 1 functions
⠓ (0/1) Uploading graphql...Warning: JSONHTTPError: undefined 500
at Object.warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/errors/lib/index.js:33:15)
at warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:57:16)
at runDeploy (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:238:9)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async DeployCommand.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:422:21)
at async DeployCommand._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
at async Config.runCommand (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/config/lib/config.js:172:9)
at async Main.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/main.js:22:9)
at async Main._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
Warning:
{
"name": "JSONHTTPError",
"status": 500,
"json": {
"status": 500,
"error": "Internal Server Error"
}
}
at Object.warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/errors/lib/index.js:33:15)
at warn (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:57:16)
at runDeploy (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:239:9)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5)
at async DeployCommand.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/src/commands/deploy.js:422:21)
at async DeployCommand._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
at async Config.runCommand (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/config/lib/config.js:172:9)
at async Main.run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/main.js:22:9)
at async Main._run (/Users/username/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/@oclif/command/lib/command.js:43:20)
⠋ (0/1) Uploading graphql...JSONHTTPError: Internal Server Error
at parseResponse (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:12:11)
at runMicrotasks (<anonymous>)
at async callMethod (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:36:26)
regards
perry
November 10, 2020, 9:01pm
21
hey there, sorry to be slow to get back to you - are you still seeing this problem when you deploy? if yes, please let us know right away.
toxsick
November 11, 2020, 9:00am
22
Hey @perry , no problem. I just checked it with the latest netlify-cli/2.68.3 darwin-x64 node-v14.15.0
but the problem still exists:
...
Deploying to draft URL...
✔ Finished hashing 23 files and 1 functions
✔ CDN requesting 10 files and 1 functions
╫ (10/11) Uploading graphql... › Warning: JSONHTTPError: undefined 500
› Warning:
› {
› "name": "JSONHTTPError",
› "status": 500,
› "json": {
› "status": 500,
› "error": "Internal Server Error"
› }
› }
›
╪ (10/11) Uploading graphql...JSONHTTPError: Internal Server Error
at parseResponse (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/response.js:12:11)
at runMicrotasks (<anonymous>)
at async callMethod (~/.nvm/versions/node/v14.15.0/lib/node_modules/netlify-cli/node_modules/netlify/src/methods/index.js:36:26)
Before that it is uploading the graphql function for round about 4 minutes.
regards
luke
November 12, 2020, 3:22am
23
Hi, @toxsick . This stands out to me as a likely red flag hinting at the root cause:
Before that it is uploading the graphql function for round about 4 minutes.
Netlify Functions are built on-top of AWS Lambda. The fact that the upload of the function takes 4 minutes implies that it is a large bundle. Would you please confirm the size of the zip file bundle for that function?
If it is over 50 MB as a compress archive file (a zip file), then it is exceeding the limits of the underlying technology - AWS Lambda. The limits for Lambda are documented here:
Quoting that page:
| Deployment package | size 50 MB (zipped, for direct upload) |
If the zip file is smaller than that please let us know. If it is larger, something will need to be changed for it to meet the requirements for the zip file size.