Hello
I have a website with a lot of pages, 70k right now and increasing, and my build time is naturally also increasing. I’ve been testing how to optimize my build, added the netlify-cache plugin, but I was not able to keep it under 15 minutes limit. Is there any way to overcome this problem?
Site name: onde-assistir-online
And here is the log:
https://app.netlify.com/sites/onde-assistir-online/deploys/5f05d97cdafb83bd0c11f89a
Thanks!
hi there, i am unable to see that log - did you change your site name, or maybe you removed it?
We can extend the build time if we have a credit card on file for you, so that we can automatically bill you of build minutes should you need them.
hey @perry, thanks for the anwser.
so here is the full log:
11:34:37 AM: Build ready to start
11:34:38 AM: build-image version: ca811f47d4c1cbd1812d1eb6ecb0c977e86d1a1d
11:34:38 AM: build-image tag: v3.3.20
11:34:38 AM: buildbot version: 78b5536ab4f742c26705d3e953381b9cde6e22ef
11:34:39 AM: Fetching cached dependencies
11:34:39 AM: Starting to download cache of 1.1GB
11:34:44 AM: Finished downloading cache in 5.127654662s
11:34:44 AM: Starting to extract cache
11:35:18 AM: Finished extracting cache in 34.576229077s
11:35:19 AM: Finished fetching cache in 39.958121602s
11:35:19 AM: Starting to prepare the repo for build
11:35:19 AM: Preparing Git Reference refs/heads/master
11:35:21 AM: Different publish path detected, going to use the one specified in the Netlify configuration file: 'public' versus 'public/' in the Netlify UI
11:35:21 AM: Starting build script
11:35:21 AM: Installing dependencies
11:35:21 AM: Python version set to 2.7
11:35:21 AM: Started restoring cached node version
11:35:25 AM: Finished restoring cached node version
11:35:25 AM: v10.21.0 is already installed.
11:35:26 AM: Now using node v10.21.0 (npm v6.14.4)
11:35:26 AM: Started restoring cached build plugins
11:35:26 AM: Finished restoring cached build plugins
11:35:26 AM: Attempting ruby version 2.6.2, read from environment
11:35:28 AM: Using ruby version 2.6.2
11:35:28 AM: Using PHP version 5.6
11:35:28 AM: 5.2 is already installed.
11:35:28 AM: Using Swift version 5.2
11:35:28 AM: Started restoring cached node modules
11:35:28 AM: Finished restoring cached node modules
11:35:28 AM: Started restoring cached yarn cache
11:35:28 AM: Finished restoring cached yarn cache
11:35:29 AM: Installing NPM modules using Yarn version 1.17.0
11:35:30 AM: yarn install v1.17.0
11:35:30 AM: [1/4] Resolving packages...
11:35:31 AM: [2/4] Fetching packages...
11:36:28 AM: info fsevents@2.1.3: The platform "linux" is incompatible with this module.
11:36:28 AM: info "fsevents@2.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
11:36:28 AM: info fsevents@1.2.13: The platform "linux" is incompatible with this module.
11:36:28 AM: info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
11:36:28 AM: [3/4] Linking dependencies...
11:36:28 AM: warning "gatsby > @pmmmwh/react-refresh-webpack-plugin@0.3.3" has incorrect peer dependency "react-refresh@^0.8.2".
11:36:28 AM: warning "gatsby > @typescript-eslint/eslint-plugin > tsutils@3.17.1" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta".
11:36:28 AM: warning " > gatsby-plugin-styled-components@3.3.9" has unmet peer dependency "babel-plugin-styled-components@>1.5.0".
11:36:28 AM: warning "react-instantsearch-dom > algoliasearch-helper@3.1.2" has unmet peer dependency "algoliasearch@>= 3.1 < 5".
11:36:28 AM: warning " > slick-carousel@1.8.1" has unmet peer dependency "jquery@>=1.8.0".
11:36:28 AM: warning " > styled-components@5.1.1" has unmet peer dependency "react-is@>= 16.8.0".
11:37:15 AM: [4/4] Building fresh packages...
11:37:19 AM: success Saved lockfile.
11:37:19 AM: Done in 109.20s.
11:37:19 AM: NPM modules installed using Yarn
11:37:19 AM: Started restoring cached go cache
11:37:21 AM: Finished restoring cached go cache
11:37:21 AM: Installing Go version 1.12
11:37:21 AM: unset GOOS;
11:37:21 AM: unset GOARCH;
11:37:21 AM: export GOROOT='/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64';
11:37:21 AM: export PATH="/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64/bin:${PATH}";
11:37:21 AM: go version >&2;
11:37:21 AM: export GIMME_ENV='/opt/buildhome/.gimme_cache/env/go1.12.linux.amd64.env';
11:37:21 AM: go version go1.12 linux/amd64
11:37:21 AM: Installing missing commands
11:37:21 AM: Verify run directory
11:37:22 AM:
11:37:22 AM: ┌─────────────────────────────┐
11:37:22 AM: │ Netlify Build │
11:37:22 AM: └─────────────────────────────┘
11:37:22 AM:
11:37:22 AM: ❯ Version
11:37:22 AM: @netlify/build 2.0.27
11:37:22 AM:
11:37:22 AM: ❯ Flags
11:37:22 AM: deployId: 5f05d97cdafb83bd0c11f89a
11:37:22 AM: mode: buildbot
11:37:22 AM:
11:37:22 AM: ❯ Current directory
11:37:22 AM: /opt/build/repo
11:37:22 AM:
11:37:22 AM: ❯ Config file
11:37:22 AM: /opt/build/repo/netlify.toml
11:37:22 AM:
11:37:22 AM: ❯ Context
11:37:22 AM: production
11:37:23 AM:
11:37:23 AM: ❯ Loading plugins
11:37:23 AM: - netlify-plugin-gatsby-cache@0.3.0 from netlify.toml
11:37:23 AM:
11:37:23 AM: ┌────────────────────────────────────────────────────────┐
11:37:23 AM: │ 1. onPreBuild command from netlify-plugin-gatsby-cache │
11:37:23 AM: └────────────────────────────────────────────────────────┘
11:37:23 AM:
11:37:43 AM: Found a Gatsby cache. We’re about to go FAST. ⚡️
11:37:43 AM:
11:37:43 AM: (netlify-plugin-gatsby-cache onPreBuild completed in 19.8s)
11:37:43 AM:
11:37:43 AM: ┌───────────────────────────────────┐
11:37:43 AM: │ 2. Build command from Netlify app │
11:37:43 AM: └───────────────────────────────────┘
11:37:43 AM:
11:37:43 AM: $ gatsby build
11:37:50 AM: success open and validate gatsby-configs - 0.072s
11:37:51 AM: success load plugins - 0.508s
11:37:51 AM: success onPreInit - 0.011s
11:37:54 AM: success delete html and css files from previous builds - 2.739s
11:37:54 AM: info One or more of your plugins have changed since the last time you ran Gatsby. As
11:37:54 AM: a precaution, we're deleting your site's cache to ensure there's no stale data.success initialize cache - 0.057s
11:37:54 AM: success copy gatsby files - 0.038s
11:37:54 AM: success onPreBootstrap - 0.014s
11:37:54 AM: success createSchemaCustomization - 0.005s
11:38:30 AM: success source and transform nodes - 36.116s
11:38:32 AM: success building schema - 1.467s
11:40:07 AM: success createPages - 95.571s
11:40:07 AM: success createPagesStatefully - 0.108s
11:40:10 AM: success updating schema - 2.188s
11:40:10 AM: success onPreExtractQueries - 0.000s
11:40:10 AM: success extract queries from components - 0.353s
11:40:10 AM: success write out redirect data - 0.003s
11:40:10 AM: warning The icon(src/assets/images/icon.png) you provided to 'gatsby-plugin-manifest' is not square.
11:40:10 AM: The icons we generate will be square and for the best results we recommend you provide a square icon.
11:40:11 AM: success Build manifest and related icons - 0.794s
11:40:11 AM: success onPostBootstrap - 0.797s
11:40:11 AM: info bootstrap finished - 148.071s
11:40:13 AM: success run static queries - 1.372s - 3/3 2.19/s
11:40:46 AM: success run page queries - 33.282s - 78413/78413 2355.99/s
11:40:47 AM: success write out requires - 0.511s
11:41:50 AM: success Building production JavaScript and CSS bundles - 63.473s
11:41:51 AM: success Rewriting compilation hashes - 0.002s
11:48:57 AM: success Building static HTML for pages - 354.148s - 78413/78413 221.41/s
11:49:17 AM: success onPostBuild - 20.036s
11:49:25 AM: info Done building in 702.755 sec
11:49:27 AM:
11:49:27 AM: (build.command completed in 11m 44.6s)
11:49:27 AM:
11:49:27 AM: ┌─────────────────────────────────────────────────────────┐
11:49:27 AM: │ 3. onPostBuild command from netlify-plugin-gatsby-cache │
11:49:27 AM: └─────────────────────────────────────────────────────────┘
11:49:27 AM:
11:51:51 AM: Stored the Gatsby cache to speed up future builds.
11:51:51 AM:
11:51:51 AM: (netlify-plugin-gatsby-cache onPostBuild completed in 2m 23.5s)
11:51:51 AM:
11:51:51 AM: ┌─────────────────────────────┐
11:51:51 AM: │ Netlify Build Complete │
11:51:51 AM: └─────────────────────────────┘
11:51:51 AM:
11::51 AM: (Netlify Build completed in 14m 28.4s)
11:51:51 AM: Execution timed out after 15m0s
11:51:51 AM: Error running command: Command did not finish within the time limit
11:51:51 AM: Failing build: Failed to build site
11:51:51 AM: Failed during stage 'building site': Command did not finish within the time limit
11:51:51 AM: Finished processing build request in 17m12.567064531s
I have a credit card associated at my billing settings. It would be great to try again with more build minutes 
hi there, we extended the build time for you. let us know if this works better!
thanks @perry
I just tried again but unfortunely I had a different error at uploading, here is the log:
4:16:51 PM: Build ready to start
4:16:57 PM: build-image version: ca811f47d4c1cbd1812d1eb6ecb0c977e86d1a1d
4:16:57 PM: build-image tag: v3.3.20
4:16:57 PM: buildbot version: 78b5536ab4f742c26705d3e953381b9cde6e22ef
4:16:57 PM: Fetching cached dependencies
4:16:57 PM: Starting to download cache of 1.1GB
4:17:03 PM: Finished downloading cache in 6.520275344s
4:17:03 PM: Starting to extract cache
4:17:36 PM: Finished extracting cache in 32.783734144s
4:17:36 PM: Finished fetching cache in 39.556006608s
4:17:36 PM: Starting to prepare the repo for build
4:17:37 PM: Preparing Git Reference refs/heads/master
4:17:39 PM: Starting build script
4:17:39 PM: Installing dependencies
4:17:39 PM: Python version set to 2.7
4:17:39 PM: Started restoring cached node version
4:17:42 PM: Finished restoring cached node version
4:17:43 PM: v10.21.0 is already installed.
4:17:43 PM: Now using node v10.21.0 (npm v6.14.4)
4:17:44 PM: Started restoring cached build plugins
4:17:44 PM: Finished restoring cached build plugins
4:17:44 PM: Attempting ruby version 2.6.2, read from environment
4:17:45 PM: Using ruby version 2.6.2
4:17:45 PM: Using PHP version 5.6
4:17:45 PM: 5.2 is already installed.
4:17:45 PM: Using Swift version 5.2
4:17:45 PM: Started restoring cached node modules
4:17:45 PM: Finished restoring cached node modules
4:17:45 PM: Started restoring cached yarn cache
4:17:45 PM: Finished restoring cached yarn cache
4:17:46 PM: Installing NPM modules using Yarn version 1.17.0
4:17:47 PM: yarn install v1.17.0
4:17:47 PM: [1/4] Resolving packages...
4:17:48 PM: [2/4] Fetching packages...
4:18:38 PM: [3/4] Linking dependencies...
4:19:22 PM: [4/4] Building fresh packages...
4:19:26 PM: success Saved lockfile.
4:19:26 PM: Done in 99.50s.
4:19:26 PM: NPM modules installed using Yarn
4:19:27 PM: Started restoring cached go cache
4:19:28 PM: Finished restoring cached go cache
4:19:28 PM: Installing Go version 1.12
4:19:28 PM: unset GOOS;
4:19:28 PM: unset GOARCH;
4:19:28 PM: export GOROOT='/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64';
4:19:28 PM: export PATH="/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64/bin:${PATH}";
4:19:28 PM: go version >&2;
4:19:28 PM: export GIMME_ENV='/opt/buildhome/.gimme_cache/env/go1.12.linux.amd64.env';
4:19:28 PM: go version go1.12 linux/amd64
4:19:28 PM: Installing missing commands
4:19:28 PM: Verify run directory
4:19:29 PM:
4:19:29 PM: ┌─────────────────────────────┐
4:19:29 PM: │ Netlify Build │
4:19:29 PM: └─────────────────────────────┘
4:19:29 PM:
4:19:29 PM: ❯ Version
4:19:29 PM: @netlify/build 2.0.27
4:19:29 PM:
4:19:29 PM: ❯ Flags
4:19:29 PM: deployId: 5f061ba3b503adc4fd51edd6
4:19:29 PM: mode: buildbot
4:19:29 PM:
4:19:29 PM: ❯ Current directory
4:19:29 PM: /opt/build/repo
4:19:29 PM:
4:19:29 PM: ❯ Config file
4:19:29 PM: /opt/build/repo/netlify.toml
4:19:29 PM:
4:19:29 PM: ❯ Context
4:19:29 PM: production
4:19:30 PM:
4:19:30 PM: ❯ Loading plugins
4:19:30 PM: - netlify-plugin-gatsby-cache@0.3.0 from netlify.toml
4:19:30 PM:
4:19:30 PM: ┌────────────────────────────────────────────────────────┐
4:19:30 PM: │ 1. onPreBuild command from netlify-plugin-gatsby-cache │
4:19:30 PM: └────────────────────────────────────────────────────────┘
4:19:30 PM:
4:19:49 PM: Found a Gatsby cache. We’re about to go FAST. ⚡️
4:19:49 PM:
4:19:49 PM: (netlify-plugin-gatsby-cache onPreBuild completed in 19.8s)
4:19:49 PM:
4:19:49 PM: ┌───────────────────────────────────┐
4:19:49 PM: │ 2. Build command from Netlify app │
4:19:49 PM: └───────────────────────────────────┘
4:19:49 PM:
4:19:49 PM: $ gatsby build
4:19:57 PM: success open and validate gatsby-configs - 0.079s
4:19:57 PM: success load plugins - 0.516s
4:19:57 PM: success onPreInit - 0.010s
4:20:00 PM: success delete html and css files from previous builds - 2.566s
success initialize cache - 0.053s
4:20:00 PM: success copy gatsby files - 0.042s
4:20:00 PM: success onPreBootstrap - 0.010s
4:20:00 PM: success createSchemaCustomization - 0.003s
4:20:33 PM: success source and transform nodes - 32.522s
4:20:34 PM: success building schema - 1.521s
4:22:18 PM: success createPages - 104.111s
4:22:18 PM: success createPagesStatefully - 0.113s
4:22:20 PM: success updating schema - 1.945s
4:22:20 PM: success onPreExtractQueries - 0.000s
4:22:21 PM: success extract queries from components - 0.246s
4:22:21 PM: success write out redirect data - 0.001s
4:22:21 PM: success Build manifest and related icons - 0.172s
4:22:21 PM: success onPostBootstrap - 0.173s
4:22:21 PM: info bootstrap finished - 151.229s
4:22:22 PM: success run static queries - 1.069s - 3/3 2.81/s
4:22:51 PM: success run page queries - 28.993s - 78413/78413 2704.50/s
4:22:52 PM: success write out requires - 0.477s
4:23:51 PM: success Building production JavaScript and CSS bundles - 59.576s
4:23:52 PM: success Rewriting compilation hashes - 0.002s
4:30:34 PM: success Building static HTML for pages - 331.818s - 78413/78413 236.31/s
4:30:53 PM: success onPostBuild - 18.702s
4:31:02 PM: info Done building in 672.004 sec
4:31:03 PM:
4:31:03 PM: (build.command completed in 11m 13.9s)
4:31:03 PM:
4:31:03 PM: ┌─────────────────────────────────────────────────────────┐
4:31:03 PM: │ 3. onPostBuild command from netlify-plugin-gatsby-cache │
4:31:03 PM: └─────────────────────────────────────────────────────────┘
4:31:03 PM:
4:33:26 PM: Stored the Gatsby cache to speed up future builds.
4:33:26 PM:
4:33:26 PM: (netlify-plugin-gatsby-cache onPostBuild completed in 2m 23s)
4:33:27 PM:
4:33:27 PM: ┌─────────────────────────────┐
4:33:27 PM: │ Netlify Build Complete │
4:33:27 PM: └─────────────────────────────┘
4:33:27 PM:
4:33:27 PM: (Netlify Build completed in 13m 57.1s)
4:33:27 PM: Caching artifacts
4:33:27 PM: Started saving node modules
4:33:27 PM: Finished saving node modules
4:33:27 PM: Started saving build plugins
4:33:27 PM: Finished saving build plugins
4:33:27 PM: Started saving yarn cache
4:33:27 PM: Finished saving yarn cache
4:33:27 PM: Started saving pip cache
4:33:27 PM: Finished saving pip cache
4:33:27 PM: Started saving emacs cask dependencies
4:33:27 PM: Finished saving emacs cask dependencies
4:33:27 PM: Started saving maven dependencies
4:33:27 PM: Finished saving maven dependencies
4:33:27 PM: Started saving boot dependencies
4:33:27 PM: Finished saving boot dependencies
4:33:27 PM: Started saving go dependencies
4:33:34 PM: Finished saving go dependencies
4:33:34 PM: Build script success
4:33:34 PM: Starting to deploy site from 'public'
4:34:19 PM: Creating deploy tree asynchronously
4:34:25 PM: Creating deploy upload records
4:37:03 PM: 156845 new files to upload
4:37:03 PM: 0 new functions to upload
5:01:52 PM: Build exceeded maximum allowed runtime
5:03:44 PM: Failed to upload file: filme/o-jogo-da-forca/index.html
5:03:46 PM: Failed to upload file: filme/o-jogo-da-guerra/index.html
5:03:47 PM: Failed to upload file: filme/o-jogo-da-trapaca/index.html
5:03:47 PM: Failed to upload file: filme/o-jik-geu-dae-man/index.html
5:03:48 PM: Failed to upload file: filme/o-jogo-com-o-fogo/index.html
5:03:51 PM: Failed to upload file: filme/o-jardim/index.html
5:03:52 PM: Failed to upload file: filme/o-jogo-da-verdade/index.html
5:03:57 PM: Failed to upload file: filme/o-irmao-que-veio-de-outro-planeta/index.html
5:04:09 PM: Failed to upload file: filme/o-jantar/index.html
5:04:10 PM: Failed to upload file: filme/o-invisivel/index.html
5:04:10 PM: Failing build: Failed to deploy site
5:04:40 PM: Failed to inform the API about a failed build, please retry the build or contact support
5:04:40 PM: Finished processing build request in 47m43.737511642s
Even with a large amount of pages, the amount of time to build (47min) seems too much. What I could be doing wrong?
seems like you are using gatsby, correct? Maybe this would be helpful:
there may also be some build plugins that could be helpful?
we can push the build time higher, if you would like us to, just let us know.
hey @perry, thanks for the anwser.
I had better results by enabling the gatsby incremental build feature!
However, since the website has a huge amount of pages and will increase, I think that I’d like to have more build time, just to be safe.
Hey @urielscola,
For us to go ahead and up your time limit to 30 minutes, could you ensure that you have a credit card on record with us and also clarify whether you’d like the new time limit to take place on just this site or across your whole team?
Hey just to clarify - @Pie - I already upgraded @urielscola up to 30 mins and they have a credit card on file. We are talking about going up to an hour here, is that correct?