Build stuck at "start saving pip cache"

Hi,

The build of my website is stuck at the end, when Netlify tries to save the pip cache.

My site is Gatsby/Novela (no Python), and open-source. It did work before. I tried to clean the cache but still the same error.

Just tried to disable the build plugin beta, and was able to publish without troubles, so it may be related!

Reenabling the build beta demo did break the build again, so it looks related

Hey @slorber, thanks for the detailed report! I’ll share this internally. Just so you know, the failure here was because of how long your site took to build with Netlify Build enabled and doesn’t necessarily have to do with the pip cache specifically:

3:31:42 AM: failed during stage 'building site': Command did not finish within the time limit
3:31:43 AM: Finished processing build request in 21m24.129226506s

Each build must complete in 15 minutes or less, and it looks like this step took almost 5 minutes (!):

3:13:54 AM: ❯ Installing plugins
3:13:54 AM:    - netlify-plugin-gatsby-cache
3:18:23 AM: ​
3:18:23 AM: ❯ Loading plugins
3:18:23 AM:    - netlify-plugin-gatsby-cache@0.2.1

And then this step was a little less than 5 minutes as well (about the same amount of time as your successful build took):

3:23:11 AM: success Generating image thumbnails - 281.976s - 229/229 0.81/s

Anyway, thanks again and I’ve added your report to our issue about this.

Hi @jen

My site is just a Gatsby blog with … 4 MDX articles. It builds in 1min on my computer with fresh gatsby caches.

Without netlify build plugins, it builds in 5 minutes (which is already significantly slower than my own computer, but it has to yarn install + restore cache + other steps… still a lot slower).

With netlify build plugins, it times out. I clearly don’t think this is a problem on my side. As the logs show, on the deployment you linked, the “saving pip dependencies” step takes like 10 minutes, before the 15min timeout is reached.

At you can see in this screenshot, my site build command, and the postbuild plugin has already run 10 minutes before the timeout is reached.

This is clearly not a problem on my side.

My site did build in the regular 5 minutes in the deployment you linked.
cf the gatsby build log: 12:23:11 PM: info Done building in 287.455754854 sec

You can see that from 12:23 to 12:31, the site is stuck during 8 minutes at the “started saving pip cache” step. This IS the cause of the build timeout, my site is not the cause.

I didn’t mean to suggest the failure was your problem—Build Plugins was clearly the issue—I just didn’t think pip cache was the specific Build Plugins failure. But you were right, and I misspoke! Pip cache does seem to be the culprit. The issue is here if you want to follow: Automatic installation of plugins makes build much slower · Issue #1137 · netlify/build · GitHub

Thanks, Wil check this issue

The weird thing is that I have other, more complex Gatsby sites using build plugins without this issue

It is weird! I didn’t see other sites in your team with the Build Plugins beta enabled, but if you wanted to share them here or chime in on that issue, we’d definitely appreciate the additional context.

Hi @jen, the other sites are in other teams
The team you look at is my own personal account for my blog and experiments, not something I use for production deployments.

I can’t publish my employer’s name publicly but here’s a deployment ID: 5e988da23d23ab765d5d1ea3

Thanks, I’ll take a look! And glad you chimed in on that issue :slight_smile:

Hey @slorber, I wanted to follow up on this: it seems like pip cache actually includes much more than the pip cache. See this issue:

So anything with pip cache could have to do with anything that might go inside ~/.cache.

There’s an open PR to at least fix the naming issue here and clarify logging a bit here:

Hi @slorber,

We’ve just released some fixes, could you see if the problem is still appearing?
If it is, could you please do a du -hs ~/.cache/* in your build.command to inspect that cache directory? There is probably something there taking more space than it should.