Gatsby Incremental Build with custom (in-house) CMS

[cjoejoe1]

Hi,

I have already used incremental builds with Gatsby <> Netlify <> Contentful. I followed the Netlify tutorial and it works great!

I now want to use my in-house CMS (based on firebase) to tell Netlify to deploye the site when a change is made in the data in firebase (not the files).

I have a version working that does a “clean cache and deploye” when data is modified, which redeployes and updates the data on the site. It works, but it’s a full redeploye which is slow.

How can I launch an incremental deploye that only updates the pages that have had a data change?

Thanks for your help,
JP/

[elden]

Hi @cjoejoe1

I believe you’re almost there. The only real issue is that you are using a command to clean cache. What that’s doing is clearing the build cache which means that every time you build, the build has to download all of your code and dependencies from scratch. With the cache, we only have to grab what has changed so it’s much faster.

Perhaps you already know that or perhaps you were thinking it was doing cache busting? You shouldn’t need to do any cache busting since part of the build process is to invalidate the cache on our edge network. This article might help explain better than I’m doing.

[Support Guide] How does Netlify’s CDN handle caching files?

Last reviewed by Netlify Support on August 2023 While the topic of caching effectively is complex, Netlify’s solution is designed to be automatic and worry-free. It shouldn’t require any special configuration or build tools like cache-busting or service workers to deliver an optimal experience. What can go wrong? Some cache-busting techniques can make your site build and load more slowly. Scarier still, modifying Netlify’s default caching settings can break atomic rollbacks and deploys, which…

I’d try pulling out the command that is clearing the cache and see if that improves your build performance and see how your CMS handles it.

[cjoejoe1]

Hi Elden, thanks for your help!

If I don’t clean the cache on build, netlify only seems to recognise changes to files, and not the data that changes in firebase.

However, when I do this with contentful as a datasource I don’t need to clean the cache. Netlify seems to see a difference in data somehow and incremental works.

So I know the incremental works, just maybe there is an extra datasource check done at build time that i am not adding ?

Thanks again Elden, I’m quite stuck …

[luke]

Hi, @cjoejoe1.

tl;dr - This appears to be a Gatsby liimitation and not a Netlify limitation. I’ll explain in more detail below.

Netlify wouldn’t treat the two data sources differently but it is possible the site’s build code (your code) does. As I’m not familiar with the build code, I’m only able to make guesses about the root cause. I also did some research to help my guesses to be (slightly more) educated guesses.

The cache clearing behavior seems to work because when there is no cached data, the build code always fetches everything again which guarantees the latest versions are used.

It also sounds like that with Contentful, the site build code was able to detect when the local cached version was out of date and would therefore download a more recent version.

What isn’t clear is the exact details about how the site build code detected the differences with Contentful and why this code doesn’t do the same for Firebase.

When researching this I found this Contentful blog post below:

Gatsby incremental build with Contentful

Quoting that page:

Not every content management framework is a candidate for incremental builds. To build incrementally, Gatsby requires tight integration with a CMS to be able to detect changes.

Based on this, it is possible that Gatsby itself doesn’t support incremental builds with Firebase.

I also found this blog post but it is over two years old:

What is Gatsby Incremental Builds? | Gatsby

This is talking about Gatsby Cloud specifically but I do think this information applies to incremental builds anywhere. There is says:

To experience the fastest build times, update your site with content from Gatsby Cloud’s first-class CMS integrations. These include Contentful, Sanity, DatoCMS and Cosmic. WordPress and Drupal support are in alpha.

I notice that Firebase isn’t listed there but that is also a more than two year old post so this might be out of date.

Long story short, I cannot find any documentation anywhere about how to get Gatsby incremental builds working with Firebase - working at Netlify or anywhere else. Based on this, it does appear that the explanation here may be that this isn’t supported yet by Gatsby itself.

​Please let us know if there are other questions about this.