Incremental Gatsby Build time Increases

sam.r · May 15, 2020, 8:20am

Hi, I’ve followed the blog post.

I’m not sure if Automatic installation of plugins makes build much slower · Issue #1137 · netlify/build · GitHub is the reason I’m slower on builds.

I’m also not sure why a relatively simple site takes so long to build. It’s not huge by any means. 100 pages or so. (We’re writing content for it now, but don’t anticipate anything more than 300).

Here are logs for DEPLOYMENT ID: 5ebd966644ab6900079a5cbe

8:05:10 PM: Build ready to start
8:05:12 PM: build-image version: 6dfe19d15f524c85d6f9bf7df9fb30b0a9f0a61a
8:05:12 PM: build-image tag: v3.3.10
8:05:12 PM: buildbot version: 0d19834c19ebfe54208907d729718495339fe085
8:05:12 PM: Fetching cached dependencies
8:05:12 PM: Starting to download cache of 1001.3MB
8:05:33 PM: Finished downloading cache in 20.929889286s
8:05:33 PM: Starting to extract cache
8:06:19 PM: Finished extracting cache in 45.87051409s
8:06:19 PM: Finished fetching cache in 1m7.108810136s
8:06:19 PM: Starting to prepare the repo for build
8:06:20 PM: Preparing Git Reference refs/heads/master
8:06:23 PM: Starting build script
8:06:23 PM: Installing dependencies
8:06:23 PM: Started restoring cached node version
8:06:28 PM: Finished restoring cached node version
8:06:29 PM: v10.20.1 is already installed.
8:06:31 PM: Now using node v10.20.1 (npm v6.14.4)
8:06:31 PM: Attempting ruby version 2.6.2, read from environment
8:06:33 PM: Using ruby version 2.6.2
8:06:33 PM: Using PHP version 5.6
8:06:33 PM: 5.2 is already installed.
8:06:33 PM: Using Swift version 5.2
8:06:33 PM: Started restoring cached node modules
8:06:33 PM: Finished restoring cached node modules
8:06:33 PM: Started restoring cached yarn cache
8:06:33 PM: Finished restoring cached yarn cache
8:06:34 PM: Installing NPM modules using Yarn version 1.17.0
8:06:35 PM: yarn install v1.17.0
8:06:35 PM: [1/4] Resolving packages...
8:06:37 PM: [2/4] Fetching packages...
8:07:59 PM: info fsevents@2.1.2: The platform "linux" is incompatible with this module.
8:07:59 PM: info "fsevents@2.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
8:07:59 PM: info fsevents@1.2.11: The platform "linux" is incompatible with this module.
8:07:59 PM: info "fsevents@1.2.11" is an optional dependency and failed compatibility check. Excluding it from installation.
8:07:59 PM: info fsevents@2.0.6: The platform "linux" is incompatible with this module.
8:07:59 PM: info "fsevents@2.0.6" is an optional dependency and failed compatibility check. Excluding it from installation.
8:07:59 PM: [3/4] Linking dependencies...
8:07:59 PM: warning " > @crello/react-lottie@0.0.7" has incorrect peer dependency "react@~16.9.0".
8:07:59 PM: warning " > @crello/react-lottie@0.0.7" has incorrect peer dependency "react-dom@~16.9.0".
8:07:59 PM: warning " > babel-loader@8.0.6" has unmet peer dependency "webpack@>=2".
8:07:59 PM: warning "gatsby > @pmmmwh/react-refresh-webpack-plugin@0.3.1" has incorrect peer dependency "react-refresh@^0.8.2".
8:07:59 PM: warning "gatsby > @typescript-eslint/eslint-plugin > tsutils@3.17.1" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta".
8:07:59 PM: warning " > gatsby-link@2.2.27" has unmet peer dependency "@reach/router@^1.1.1".
8:07:59 PM: warning "gatsby-plugin-favicon > favicons-webpack-plugin@0.0.9" has unmet peer dependency "webpack@^1.13.0 || ^2.0.0 || ^3.0.0 || ^4.0.0".
8:07:59 PM: warning "gatsby-plugin-sass > sass-loader@7.3.1" has unmet peer dependency "webpack@^3.0.0 || ^4.0.0".
8:07:59 PM: warning " > gatsby-plugin-styled-components@3.1.16" has unmet peer dependency "babel-plugin-styled-components@>1.5.0".
8:07:59 PM: warning "gatsby-transformer-inline-svg > gatsby > react-hot-loader@4.12.18" has unmet peer dependency "@types/react@^15.0.0 || ^16.0.0".
8:07:59 PM: warning "gatsby-transformer-inline-svg > gatsby > gatsby-cli > ink@2.6.0" has unmet peer dependency "@types/react@>=16.8.0".
8:07:59 PM: warning " > lodash-webpack-plugin@0.11.5" has unmet peer dependency "webpack@^2.0.0 || ^3.0.0 || ^4.0.0".
8:07:59 PM: warning " > react-dates@20.3.0" has unmet peer dependency "moment@^2.18.1".
8:07:59 PM: warning " > react-dates@20.3.0" has unmet peer dependency "react-with-direction@^1.3.0".
8:07:59 PM: warning "react-dates > react-moment-proptypes@1.7.0" has unmet peer dependency "moment@>=1.6.0".
8:07:59 PM: warning "react-stripe-menu > react-scripts > @typescript-eslint/eslint-plugin@1.6.0" has unmet peer dependency "typescript@*".
8:07:59 PM: warning "react-stripe-menu > react-scripts > @typescript-eslint/parser@1.6.0" has unmet peer dependency "typescript@*".
8:07:59 PM: warning "react-stripe-menu > react-scripts > @typescript-eslint/eslint-plugin > @typescript-eslint/typescript-estree@1.6.0" has unmet peer dependency "typescript@*".
8:07:59 PM: warning " > eslint-config-airbnb@18.0.1" has unmet peer dependency "eslint-plugin-react-hooks@^1.7.0".
8:09:58 PM: [4/4] Building fresh packages...
8:10:32 PM: Done in 237.08s.
8:10:32 PM: NPM modules installed using Yarn
8:10:32 PM: Started restoring cached go cache
8:10:32 PM: Finished restoring cached go cache
8:10:32 PM: go version go1.12 linux/amd64
8:10:32 PM: go version go1.12 linux/amd64
8:10:32 PM: Installing missing commands
8:10:32 PM: Verify run directory
8:10:35 PM: 
8:10:35 PM: ┌─────────────────────────────┐
8:10:35 PM: │        Netlify Build        │
8:10:35 PM: └─────────────────────────────┘
8:10:35 PM: 
8:10:35 PM: ❯ Version
8:10:35 PM:   @netlify/build 0.4.26
8:10:35 PM: 
8:10:35 PM: ❯ Flags
8:10:35 PM:   mode: buildbot
8:10:35 PM: 
8:10:35 PM: ❯ Current directory
8:10:35 PM:   /opt/build/repo
8:10:35 PM: 
8:10:35 PM: ❯ Config file
8:10:35 PM:   /opt/build/repo/netlify.toml
8:10:35 PM: 
8:10:35 PM: ❯ Context
8:10:35 PM:   production
8:10:35 PM: 
8:10:35 PM: ❯ Installing plugins
8:10:35 PM:    - netlify-plugin-gatsby-cache
8:14:32 PM: 
8:14:32 PM: ❯ Loading plugins
8:14:32 PM:    - netlify-plugin-gatsby-cache@0.2.1
8:14:32 PM: 
8:14:32 PM: ┌────────────────────────────────────────────────────────┐
8:14:32 PM: │ 1. onPreBuild command from netlify-plugin-gatsby-cache │
8:14:32 PM: └────────────────────────────────────────────────────────┘
8:14:32 PM: 
8:14:34 PM: Found a Gatsby cache. We’re about to go FAST. ⚡️
8:14:34 PM: 
8:14:34 PM: (netlify-plugin-gatsby-cache onPreBuild completed in 1.5s)
8:14:34 PM: 
8:14:34 PM: ┌────────────────────────────────────┐
8:14:34 PM: │ 2. build.command from netlify.toml │
8:14:34 PM: └────────────────────────────────────┘
8:14:34 PM: 
8:14:34 PM: $ GATSBY_EXPERIMENTAL_PAGE_BUILD_ON_DATA_CHANGES=true gatsby build --log-pages
8:14:37 PM: 
8:14:37 PM: 
8:14:37 PM: success open and validate gatsby-configs - 0.175s
8:14:38 PM: 
8:14:38 PM: success load plugins - 1.450s
8:14:38 PM: 
8:14:38 PM: success onPreInit - 0.018s
8:14:38 PM: 
8:14:38 PM: success initialize cache - 0.033s
8:14:39 PM: success copy gatsby files - 0.137s
8:14:39 PM: success Optional chaining available
8:14:39 PM: 
8:14:39 PM: success Optional chaining available
8:14:39 PM: 
8:14:39 PM: success Optional chaining available
8:14:39 PM: success Optional chaining available
8:14:39 PM: 
8:14:39 PM: success onPreBootstrap - 0.055s
8:14:39 PM: success createSchemaCustomization - 0.100s
8:14:39 PM: 
8:14:39 PM: Starting to fetch data from Contentful
8:14:39 PM: 
8:14:39 PM: Fetching default locale
8:14:39 PM: default locale is : en-US
8:14:40 PM: contentTypes fetched 16
8:14:40 PM: Updated entries  1
8:14:40 PM: Deleted entries  0
8:14:40 PM: 
8:14:40 PM: Updated assets  0
8:14:40 PM: 
8:14:40 PM: Deleted assets  0
8:14:40 PM: Fetch Contentful data: 675.972ms
8:14:41 PM: 
8:14:41 PM: success source and transform nodes - 1.910s
8:14:47 PM: success building schema - 5.956s
8:14:47 PM: success createPages - 0.527s
8:14:47 PM: success createPagesStatefully - 0.317s
8:14:47 PM: success onPreExtractQueries - 0.001s
8:14:48 PM: success update schema - 0.141s
8:14:51 PM: success extract queries from components - 3.140s
8:14:51 PM: success write out requires - 0.006s
8:14:51 PM: success write out redirect data - 0.001s
8:14:51 PM: success Build manifest and related icons - 0.002s
8:14:51 PM: success onPostBootstrap - 0.003s
8:14:51 PM: 
8:14:51 PM: ⠀
8:14:51 PM: info bootstrap finished - 16.914s
8:14:51 PM: 
8:14:51 PM: ⠀
8:16:21 PM: 
8:16:21 PM: warning Browserslist: caniuse-lite is outdated. Please run next command `yarn upgrade`
8:16:30 PM: 
8:16:30 PM: success Building production JavaScript and CSS bundles - 98.625s
8:16:31 PM: 
8:16:31 PM: 
8:16:31 PM: success run queries - 100.665s - 46/46 0.46/s
8:17:36 PM: success Building static HTML for pages - 64.081s - 1/1 0.02/s
8:17:36 PM: 
8:17:36 PM: success Delete previous page data - 0.001s
8:17:36 PM: success onPostBuild - 0.056s
8:17:36 PM: info Done building in 181.865 sec
8:17:36 PM: info Built pages:
8:17:36 PM: Updated page: /manage
8:17:36 PM: 
8:17:36 PM: (build.command completed in 3m 2.3s)
8:17:36 PM: 
8:17:36 PM: ┌─────────────────────────────────────────────────────────┐
8:17:36 PM: │ 3. onPostBuild command from netlify-plugin-gatsby-cache │
8:17:36 PM: └─────────────────────────────────────────────────────────┘
8:17:36 PM: 
8:17:38 PM: Stored the Gatsby cache to speed up future builds.
8:17:38 PM: 
8:17:38 PM: (netlify-plugin-gatsby-cache onPostBuild completed in 1.6s)
8:17:38 PM: 
8:17:38 PM: ┌─────────────────────────────┐
8:17:38 PM: │   Netlify Build Complete    │
8:17:38 PM: └─────────────────────────────┘
8:17:38 PM: 
8:17:38 PM: (Netlify Build completed in 7m 3.1s)
8:17:38 PM: Caching artifacts
8:17:38 PM: Started saving node modules
8:17:38 PM: Finished saving node modules
8:17:38 PM: Started saving yarn cache
8:17:38 PM: Finished saving yarn cache
8:17:38 PM: Started saving pip cache
8:20:22 PM: Finished saving pip cache
8:20:22 PM: Started saving emacs cask dependencies
8:20:23 PM: Finished saving emacs cask dependencies
8:20:23 PM: Started saving maven dependencies
8:20:23 PM: Finished saving maven dependencies
8:20:23 PM: Started saving boot dependencies
8:20:23 PM: Finished saving boot dependencies
8:20:23 PM: Started saving go dependencies
8:20:23 PM: Finished saving go dependencies
8:20:23 PM: Build script success
8:20:23 PM: Starting to deploy site from 'public'
8:20:23 PM: Creating deploy tree asynchronously
8:20:23 PM: Creating deploy upload records
8:20:25 PM: 2 new files to upload
8:20:25 PM: 0 new functions to upload
8:20:27 PM: Starting post processing
8:20:30 PM: Post processing done
8:20:30 PM: Site is live
8:23:05 PM: Finished processing build request in 17m52.864352614s

Is there anything obvious that can be done to use incremental builds and not have 20minute build times on a simple site? Would love to know if you can provide pointers for something perhaps not set up correctly.

Also, i’ve disabled the beta now.

PS. Does incremental work with Contentful Webhooks? Is there something we have to do. What the blog post doesn’t explain is if pretty “Trigger Build” from the Netlify UI always forces a full build. I spent 90 minutes clicking it back to back without any changes and it always was x2 or more slower.

Scott · May 18, 2020, 2:38pm

Hey @sam.r, did you check out this publication from our blog?

sam.r · May 19, 2020, 9:53am

Yes. That’s what I meant when saying following the blog post, but yeah, that was how I followed.
The incremental build clearly is attempting to do sttuff, else the build times wouldn’t be x3 times slower than the normal build, but obviously that’s not the desired behaviour.

expertsleepcenters · May 20, 2020, 8:07pm

I’m experiencing the same issue but, using Dato CMS build hook instead. Builds went from 2 minutes to almost 6 minutes. The only change I made from the blog post was adding this to the netlify.toml file.

[build]
   publish = "public"

per the GetHub Repo doc’s for this plug-in.

jlengstorf · May 20, 2020, 9:32pm

Hey there!

It looks like there are two significant bugs in the beta that are adding to the build times:

currently, Node modules installed with Yarn are reinstalled on every build — this adds a whole bunch of extra time: Yarn install runs even when a node_modules cache is found · Issue #433 · netlify/build-image · GitHub
installing build plugins automatically adds extra build time: Automatic installation of plugins makes build much slower · Issue #1137 · netlify/build · GitHub

I can’t say for sure without reviewing the logs, but my guess is that the actual Gatsby builds are faster, but the startup time for Netlify is slower due to these bugs (please correct me if I’m wrong).

These bugs will both end up getting fixed, so in time this will resolve itself. However, if you want to fix it on your own, you can make two changes and it should resolve this issue:

Switch to npm for plugin installation. Delete the yarn.lock in your project and use npm install to generate a package-lock.json. Swap any uses of yarn out for npm (e.g. npm run build instead of yarn build) to make sure things work as expected.

This will remove the double-install of packages, which in @sam.r’s case from the logs above would remove 4 minutes from the build.
Install the build plugins manually. We automatically install build plugins as a convenience, but it’s adding additional time. You can avoid this added time if you add plugins as dependencies to your project:
```
npm install netlify-plugin-gatsby-cache
```

These two fixes should remove the slowdowns and let you get the benefits of Gatsby incremental builds. Please report back if you try it so we can verify that there aren’t other things happening!

Thanks!