Netlify cache util not working as expected

[AverageNetlifyBoi]

Hi, I am trying to cache files in between builds but with no success. I am trying to cache ./dist folder and its contents, that is - static and my-project-name folders (I am using Angular with Scully).

module.exports = {
  async onPreBuild({utils}) {
    await utils.cache.restore('./dist');
  },
  async onPostBuild({utils}) {
    await utils.cache.save('./dist');
  },
};

I did tens of console.logs, directory is correct, paths seem ok, files to cache are there, but when it caches it only shows ./dist/static/sitemap.xml as the only cached file. Interestingly, I no longer use sitemap plugin and it still shows that one file out of nowhere. I did clear Netlify cache, and still, same behaviour.

I also tried netlify-plugin-cache but behaviour is exactly the same. One happy thing is that locally with Netlify CLI (ntl build) it does work with zero issues, everything caches properly. Am I missing something? Thanks!

[luke]

Hi, @AverageNetlifyBoi. One thing that jumps out is that you are saving before the build and then restoring after the build. Shouldn’t it be the other way around? Meaning, shouldn’t you restore before building and then save after the build is complete?

[AverageNetlifyBoi]

Sorry, I made a mistake writing this, post fixed (I do have correct implementation in my project).

[luke]

Hi, @AverageNetlifyBoi. Many static site generators (SSGs) delete the contents of the “publish directory” (whatever that is for that particular SSG). I believe for Scully that is ./dist.

Are you sure it isn’t Scully itself deleting the contents of that directory?

[AverageNetlifyBoi]

I did console logs of all existing files and everything seems okay. Important thing to notice is that locally Netlify Dev does cache properly with zero problems. Issue only occurs during deployments.

[hrishikesh]

Happy to check is. Is there a repo we could try?

[AverageNetlifyBoi]

There you go: GitHub - filipwiniarski/netlify-cache-test

This is a minimial setup of an Angular app with Scully connected to a Netlify site. I added logs to the plugin. It always shows cached files on post build properly, but in pre build it shows none. I can also confirm that in build scripts in my private project those “cached” files are not available during/after restoration. It says it caches over 700 paths and then restores only 3 paths - funny enough, one of those is a very old sitemap.xml file which has been removed from the repo long time ago (scripts, and plugins are not generating this file). There is something weird happening with those cache utils

Locally, with Netlify CLI, everything works like a charm. Maybe I’m missing some point…

[hrishikesh]

Hey @AverageNetlifyBoi,

I tried your repo and got this:

I simply forked it and deployed it as it is. Do I need to do something special to trigger the behaviour you mentioned?

[AverageNetlifyBoi]

Yes, that is correct. When you run next build you’ll see no cached files on pre build hook. List method from Netlify Cache Utils shows paths of files before the save or after the restoration, which I did, but the output shows no files restored.

You can run npm run jam script in order to generate static folder as well - I don’t know how cache utils work exactly. It generates two folders inside /opt/build/repo/dist folder, more files to cache.

[hrishikesh]

I inspected the cache (we could download it from our end) for the test website I created and I found the following:

So it appears that the cache is all going inside dist/netlify-cache-test folder, but all the files do exist that the log mentioned above. I don’t see any sitemap.xml too.

I was also able to see the files in the new deploy:

So it appears to be working. Would you be able to create a new site with the same repo?

[AverageNetlifyBoi]

I am running npm run jam with public folder set to dist/static, here is my log of repetitive deploy:

Sitemap.xml is the case in my private repository which I am not allowed to share any info about.

Netlify site: https://netlify-cache-test.netlify.app/

[hrishikesh]

Hey @AverageNetlifyBoi,

I am not able to reproduce the error even with your build command. The cache appears to work fine, check this one: Netlify App

The problem however seems to be with the overall cache for your site. For example, check this: Netlify App and Netlify App.

Looks like the cache for your website never got stored. I checked our logs and the cache is not being restored due to 404. This usually happens when the cache didn’t make it to our storage in the previous deploys. Would you be able to try this on a fresh site? Since the setup you gave me appears to work just fine, maybe trying it on a fresh site would help.

[AverageNetlifyBoi]

This is a fresh site I deployed yesterday. I have to implement incremental builds for a few really big clients where I am not able to create new sites. I am looking for a solution to fix caching issue without migrating the sites

[AverageNetlifyBoi]

Here are the logs from one of the clients sites where cache is being extracted (no failure) and yet only 3 paths are extracted instead of 700. Also, sitemap.xml does not exists in current repo/build output

[hrishikesh]

Could you share this site’s ID or subdomain?

[hrishikesh]

Hi @AverageNetlifyBoi,

I checked your deploys and it looks like the files you’re trying to cache are not actually making it in the cache. Your logs state they’re, but that’s not what’s happening.

I’d suggest to try saving each file, then check if that file has got saved using has: build/README.md at main · netlify/build · GitHub and see if it adds any additional insights?

[AverageNetlifyBoi]

Same story, in post build has returns true for all paths listed by list method. On pre build it only returns true for files listed in list, but when I check desired files it returns false.

[hrishikesh]

This doesn’t happen in the test repo that you sent earlier, so maybe something is going wrong with the repo you’re actually using.

Would it be possible to share that?

[AverageNetlifyBoi]

I am not allowed to share you access to our repo. However I do believe this is the same issue since behaviour is the same - both repos build, save cache and during restoration cached files are not available.

[hrishikesh]

@AverageNetlifyBoi

do believe this is the same issue since behaviour is the same

That’s exactly the problem. The repo you shared has the cache working fine as mentioned in a few responses above, which is the only reason why I thought debugging your actual repo would make sense.