[Support Guide] Deploy logs truncated or showing “Shutting down logging, [number] messages pending”

Last checked April 2022

UPDATE: This issue was largely fixed in Apr 2020 - see this post for details.


Often, with a failed deploy (or even a successful one), you will see a log line similar to this:

1:23:45 PM: Shutting down logging, 67 messages pending

Other times, your build logs may just cut off right in the middle of a very long log quietly, right in the middle of a 70% finished webpack run, with no such message. But if the logs cut off mid-build without one of these two messages, you’re probably seeing only partial logs.

7:26:40 AM: Failing build: Failed to build site
7:26:40 AM: Finished processing build request in XX.YYYYYYYs

or

8:16:36 AM: Site is live
8:16:54 AM: Finished processing build request in XX.YYYYYYYs

What does this mean? Well, it means that the build process completed before all log messages were uploaded to our logging service. This is not something you need to worry about or work to address unless it is impacting your ability to debug a failing build.

When debugging a build, this tends to be a big problem because troubleshooting a failed deploy begins with examining those logs. If the error itself is missing, that is quite an obstacle. :angry:

But wait! There is a workaround for this. :smile: And we’re working on a longer term solution of upgrading the log system, so the workaround is only required to help you troubleshoot for now.

Before we begin, though, a disclaimer:

  • Note: These are temporary settings for diagnostics only. You must remove them after getting full logs as they force all builds to fail ! (Even if the build would otherwise succeed.)

Okay, so moving on to that workaround. It is done by using two commands:

  • sleep
  • false

The sleep command causes the system to wait for a specified number of seconds. But there is a problem: that sleep will also return an exit code of zero (0).

Why is a zero exit code a problem? Well, the exit code is what tells the build system if the build was a success or not. A zero exit code from the last command run is an indication of a success.

So, if we add a sleep command to wait for the logs to finish uploading we have a new problem. The build system will then attempt to deploy the failing build (since it looks successful - because it ended with a zero exit code).

Now the false command comes to our rescue. It will return a non-zero exit code (normally 1). This will force the build to fail.

So for example, if your build command is this:

npm run build

Then the new command would be:

npm run build ; sleep 120 ; false

In the example above sleep 120 means wait 120 seconds (two minutes). This is usually enough time to see the full deploy logs; if you’re missing many hundreds of lines (>500) you may want to extend it to a higher number, maybe 300 or more. The semi-colon (;) is used to separate the commands so each will run independent of the exit code from the prior command, which is ONLY desirable in this one case. Usually you should use && to chain build commands so that only if EACH command succeeds, will the next one run, ensuring that any failure will cause your build to be marked as failed.

As a reminder, these TEMPORARY settings will always cause every build to fail . So, please remove the settings once you have a full log to debug because all future builds will fail until these settings are removed.

Edit:

As @cjs noted, the following makes the delay only occur when the previous command has a non-zero exit code and is a much better solution because it won’t force otherwise successful builds to fail:

<build command here> || { sleep 120; false; }

For example:

npm run build || { sleep 120; false; }
2 Likes

If you want to sleep only on failed builds, returning immediately with success on builds that pass, you could try a command like:

npm run build || { sleep 120; false; }

(Note that second semicolon before the closing brace; that’s essential.)

1 Like

Where do I run this command?

hi blattinum,

in your build settings:

In the past, you may be seen the message "Shutting down logging, ..." in your build logs and it made it very difficult to debug builds that failed in this way since the actual error that shows the issue is truncated out of the logs. Yesterday, we released a fix that should greatly improve this behavior. Our buildbot will now wait up to 15 seconds if your build has pending logs. Also, speed of streaming a log line has also been improved. Between these two improvements, you should no longer run into this message. But if you do come across this messaging after April 23, 2020, please link us to your deploy below so we can investigate.

1 Like