Unknown Gatsby Build Error -- while trying to enable incremental builds

Hey Netlify,

While working on a client’s site I am getting this error

12:18:18 PM: npm ERR! code ELIFECYCLE
12:18:18 PM: npm ERR! errno 1
12:18:18 PM: npm ERR! @repo/web@1.0.1 build: `cross-env GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --verbose --log-pages`
12:18:18 PM: npm ERR! Exit status 1
12:18:18 PM: npm ERR!
12:18:18 PM: npm ERR! Failed at the @repo/web@1.0.1 build script.
12:18:18 PM: npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
12:18:18 PM: npm ERR! A complete log of this run can be found in:
12:18:18 PM: npm ERR!     /opt/buildhome/.npm/_logs/2022-11-22T09_18_18_371Z-debug.log
12:18:18 PM: ​
12:18:18 PM: ────────────────────────────────────────────────────────────────
12:18:18 PM:   "build.command" failed                                        
12:18:18 PM: ────────────────────────────────────────────────────────────────
12:18:18 PM: ​
12:18:18 PM:   Error message
12:18:18 PM:   Command failed with exit code 1: npm run build (https://ntl.fyi/exit-code-1)
12:18:18 PM: ​
12:18:18 PM:   Error location
12:18:18 PM:   In build.command from netlify.toml:
12:18:18 PM:   npm run build
12:18:18 PM: ​

As you can imagine I can’t make heads or tails of this. I have even tried to add a .npmrc file to no avail. I can confirm the build is working well locally, so I can’t make sense of what is really going on. I’m trying to add incremental builds that will reduce the amount of build time needed, so I am using cross-event as prescribed here. I am using gatsby v4.24, so I am not sure if the aforementioned solution is still relevant.

Link to build error here

Hey there @cmshaki :wave:

Thanks for your patience here. Looking closer at this, it appears as though you are having a memory issue.

Can you try using Docker image (here is a link explaining how to install it) to see how much memory your site uses? Our CI/CD build system uses this docker image to build, so we are suggesting you use the same conditions.

Once you understand how much memory is being used, you can work to optimize your memory usage.

Hey thank you Hillary, hopefully I can find out what could be causing the issue, will test out the container,

regards

Okay, I have emulated the memory error, now, I was interested in knowing, is it possible to have two netlify deployments of the same domain e.g www.domain.com and www.domain.com/ecommerce. Also, can I get the exact specs of the docker container pertaining to CPU and memory as used online.

regards,

It is, but it’s not as easy. There are 2 ways:

  1. Run a weird build command:

Something like

cd folder1 && npm run build && cd folder2 && npm run build && cp publish-dir-of-site-1 final && cp publish-dir-of-site-2 final

In other words, you need to cd to each of your site’s folder, build that, and copy that to a single directory that you can publish.

  1. Use Netlify Rewrites:

Simpler to implement, but can also lead to some issues depending on your setup. Just create 2 different sites, and use Netlify Rewrites to make them point to each other.

Here you go: [Support Guide] Making sure your builds use appropriate resources for Netlify's build system

Okay thank you hrishikesh,

I have a question is there a way to configure gatsby netlify to stop updating the cache, or does that happen when we lock auto publish?

Which cache are you referring to? The build cache, or the CDN cache? We only upload what we need on the CDN so you wouldn’t want to prevent that :slight_smile:

If you mean the build cache, certainly you can choose what gets stored; it’s just a directory you can access during build:

We can’t really provide tech support once you start messing with that though, since while it is possible for you to change it, we won’t be willing to help you do it or debug what happens when you do, beyond pointing out that it is possible and that this plugin exists to help you review what’s inside of it: GitHub - netlify-labs/netlify-plugin-debug-cache: Debug & verify the contents of your Netlify build cache

Hey,

Hey thanks for your reply I’m still having problems accessing the cache folder on netlify


On local build there is a .netlify/cache folder which has the cache folder, and I’m able to access it, however on netlify, I am unable to access it. I have tried accessing /opt/buildhome/cache and ./.netlify/cache to no success

Here is the shell script I’m using

#!/bin/bash

# Starting time
start=`date +%s`

# Run build
npm run build || exit 0

# Find cache folder of house-prices pages 
GATSBY_NETLIFY_CACHE_PATH=$(find /opt/buildhome/cache/root -type d -name "house-prices" -print -quit || 0)
GATSBY_NETLIFY_TEST_CACHE_PATH=$(find /opt/buildhome/cache/root -type d -name "nsw" -print -quit || 0)

# Check if house-prices sub-dirs don't exist
# on cache folder
if [ -d $GATSBY_NETLIFY_TEST_CACHE_PATH ]
then
  ls /opt/buildhome
  echo "Copying house-prices pages from cache"
  rsync --info=progress2 -auvz $GATSBY_NETLIFY_CACHE_PATH ./public
fi

end=`date +%s`
runtime=$((end-start))
echo "It took $runtime seconds for everything to run"

Regards,

Hey @cmshaki , it doesn’t look like the plugin that @fool mentioned is installed on your site – have you tried installing that plugin to see if you’re able to inspect the cache? I’ve included the link to the plugin again here for your reference: GitHub - netlify-labs/netlify-plugin-debug-cache: Debug & verify the contents of your Netlify build cache

Please let us know and we can work from there!

Thanks @amelia and @fool will work with the plugin, however is there a way I can acccess the cache without it, what I’m trying to do is make sure some pages persist, where I copy certain pages from the cache, as building them over again reaches the limits of the current docker containers. Also since this plugin is for debugging purposes, will it be suitable for production? Just to explain a bit on my process, I make a local build of the site, with all pages using netlify build, and after I stop auto publishing and lock in on that deploy. Also, on the repo I have commented out pages I don’t want rebuilt, so triggering the build on netlify runs the above script, this builds less the pages I do not want rebuilt, and copys from cache before any changes are made to it, ensuring peristence, maybe I should go about it another way, I’m open to ideas.

Okay I have noted that the build cache isn’t that of gatsby public folder but rather that of it’s build tools and plugins. Okay, so I guess what I’m after is the CDN cache, where I can for example ensure some pages persist and that new subsequent builds don’t take effect to these pages, or maybe I am making a feature request :slightly_smiling_face:

I’d recommend creating a build plugin to add/remove files from Netlify’s cache:

And no, you still seem to be talking about the build cache and not the CDN cache.

Hey,

This totally makes sense, thanks @hrishikesh, and @amelia I had enabled it and then proceeded to disable it. But I understand now that for gatsby you have to save the public pages on its cache via a plugin, so I guess that’s why I was only getting build tools and plugins, again would like to thank the whole support team, I wouldn’t have been able to solve this without you guys.

Regards