Build cache not working. Size limits?

Support
,
1

Hi! I’m trying to optimize a complex build by storing things in /opt/build/cache, but things don’t seem to be in the cache in a second build. I’m still in the process of debugging, but I wanted to first ask: is there a limit to the size of what will be persisted in the build cache? This site can result in about 2GB of stuff in there.

Thank you.

2

There’s no size limit on what you store; 2Gb should work fine.

2 Likes
3

Thank you!

Is there anything you know of that could result in failure to persist that directory across builds?

4

Nothing systemic, as long as they are on the same branch - we only intend to use cache on same branch or PR as it started on. So, you do of course need to be resilient to it not being present, e.g. on a new PR.

You could try listing the directory before your build completes, and at restore time (before your build starts) to ensure that it does/doesn’t save and restore what you are expecting, and you can see what we try to do in the code, here:

5

I’m still unable to get this to work. :worried:

This is an example deployment: Netlify App

Look for the lines that say /opt/build/cache contents. I’m trying to cache the /opt/build/cache/xdg directory. It doesn’t exist at the beginning, and it’s been created by the time it finishes. But a following build doesn’t seem to pick it up again.

6

I don’t see that we’d archive that path since it’s not one of the ones we create…are you using one of those paths mentioned in my selected lines from the github script?

You might also consider other implementations such as cache-me-outside - npm which are known to work, in inspiring your design.

8

It turned out that I had accidentally made the cache contents way larger than I thought. It may have been like 6GB at some point. This happened because my program was putting stuff in ~/.cache, a.k.a. $XDG_CACHE_HOME, and Netlify was caching it without my knowledge as the “pip cache”. I think it would be better to either cache only the pip directory or improve the logs to make it clear that it is potentially more than just the pip cache.

What’s weird is that my efforts of clearing the cache and rebuilding didn’t seem to have any effect. It felt like the cache was corrupt in some other way.

I started a new Netlify site (with the same sources) with a fresh cache and was able to get it all working.

Thanks for the help @fool! I hope this will help someone else as well.