Hello
I have a website with a lot of pages, 70k right now and increasing, and my build time is naturally also increasing. I’ve been testing how to optimize my build, added the netlify-cache plugin, but I was not able to keep it under 15 minutes limit. Is there any way to overcome this problem?
Site name: onde-assistir-online
And here is the log:
Thanks!
perry
July 8, 2020, 4:11pm
2
hi there, i am unable to see that log - did you change your site name, or maybe you removed it?
We can extend the build time if we have a credit card on file for you, so that we can automatically bill you of build minutes should you need them.
hey @perry , thanks for the anwser.
so here is the full log:
11:34:37 AM: Build ready to start
11:34:38 AM: build-image version: ca811f47d4c1cbd1812d1eb6ecb0c977e86d1a1d
11:34:38 AM: build-image tag: v3.3.20
11:34:38 AM: buildbot version: 78b5536ab4f742c26705d3e953381b9cde6e22ef
11:34:39 AM: Fetching cached dependencies
11:34:39 AM: Starting to download cache of 1.1GB
11:34:44 AM: Finished downloading cache in 5.127654662s
11:34:44 AM: Starting to extract cache
11:35:18 AM: Finished extracting cache in 34.576229077s
11:35:19 AM: Finished fetching cache in 39.958121602s
11:35:19 AM: Starting to prepare the repo for build
11:35:19 AM: Preparing Git Reference refs/heads/master
11:35:21 AM: Different publish path detected, going to use the one specified in the Netlify configuration file: 'public' versus 'public/' in the Netlify UI
11:35:21 AM: Starting build script
11:35:21 AM: Installing dependencies
11:35:21 AM: Python version set to 2.7
11:35:21 AM: Started restoring cached node version
11:35:25 AM: Finished restoring cached node version
11:35:25 AM: v10.21.0 is already installed.
11:35:26 AM: Now using node v10.21.0 (npm v6.14.4)
11:35:26 AM: Started restoring cached build plugins
11:35:26 AM: Finished restoring cached build plugins
11:35:26 AM: Attempting ruby version 2.6.2, read from environment
11:35:28 AM: Using ruby version 2.6.2
11:35:28 AM: Using PHP version 5.6
11:35:28 AM: 5.2 is already installed.
11:35:28 AM: Using Swift version 5.2
11:35:28 AM: Started restoring cached node modules
11:35:28 AM: Finished restoring cached node modules
11:35:28 AM: Started restoring cached yarn cache
11:35:28 AM: Finished restoring cached yarn cache
11:35:29 AM: Installing NPM modules using Yarn version 1.17.0
11:35:30 AM: yarn install v1.17.0
11:35:30 AM: [1/4] Resolving packages...
11:35:31 AM: [2/4] Fetching packages...
11:36:28 AM: info fsevents@2.1.3: The platform "linux" is incompatible with this module.
11:36:28 AM: info "fsevents@2.1.3" is an optional dependency and failed compatibility check. Excluding it from installation.
11:36:28 AM: info fsevents@1.2.13: The platform "linux" is incompatible with this module.
11:36:28 AM: info "fsevents@1.2.13" is an optional dependency and failed compatibility check. Excluding it from installation.
11:36:28 AM: [3/4] Linking dependencies...
11:36:28 AM: warning "gatsby > @pmmmwh/react-refresh-webpack-plugin@0.3.3" has incorrect peer dependency "react-refresh@^0.8.2".
11:36:28 AM: warning "gatsby > @typescript-eslint/eslint-plugin > tsutils@3.17.1" has unmet peer dependency "typescript@>=2.8.0 || >= 3.2.0-dev || >= 3.3.0-dev || >= 3.4.0-dev || >= 3.5.0-dev || >= 3.6.0-dev || >= 3.6.0-beta || >= 3.7.0-dev || >= 3.7.0-beta".
11:36:28 AM: warning " > gatsby-plugin-styled-components@3.3.9" has unmet peer dependency "babel-plugin-styled-components@>1.5.0".
11:36:28 AM: warning "react-instantsearch-dom > algoliasearch-helper@3.1.2" has unmet peer dependency "algoliasearch@>= 3.1 < 5".
11:36:28 AM: warning " > slick-carousel@1.8.1" has unmet peer dependency "jquery@>=1.8.0".
11:36:28 AM: warning " > styled-components@5.1.1" has unmet peer dependency "react-is@>= 16.8.0".
11:37:15 AM: [4/4] Building fresh packages...
11:37:19 AM: success Saved lockfile.
11:37:19 AM: Done in 109.20s.
11:37:19 AM: NPM modules installed using Yarn
11:37:19 AM: Started restoring cached go cache
11:37:21 AM: Finished restoring cached go cache
11:37:21 AM: Installing Go version 1.12
11:37:21 AM: unset GOOS;
11:37:21 AM: unset GOARCH;
11:37:21 AM: export GOROOT='/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64';
11:37:21 AM: export PATH="/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64/bin:${PATH}";
11:37:21 AM: go version >&2;
11:37:21 AM: export GIMME_ENV='/opt/buildhome/.gimme_cache/env/go1.12.linux.amd64.env';
11:37:21 AM: go version go1.12 linux/amd64
11:37:21 AM: Installing missing commands
11:37:21 AM: Verify run directory
11:37:22 AM:
11:37:22 AM: ┌─────────────────────────────┐
11:37:22 AM: │ Netlify Build │
11:37:22 AM: └─────────────────────────────┘
11:37:22 AM:
11:37:22 AM: ❯ Version
11:37:22 AM: @netlify/build 2.0.27
11:37:22 AM:
11:37:22 AM: ❯ Flags
11:37:22 AM: deployId: 5f05d97cdafb83bd0c11f89a
11:37:22 AM: mode: buildbot
11:37:22 AM:
11:37:22 AM: ❯ Current directory
11:37:22 AM: /opt/build/repo
11:37:22 AM:
11:37:22 AM: ❯ Config file
11:37:22 AM: /opt/build/repo/netlify.toml
11:37:22 AM:
11:37:22 AM: ❯ Context
11:37:22 AM: production
11:37:23 AM:
11:37:23 AM: ❯ Loading plugins
11:37:23 AM: - netlify-plugin-gatsby-cache@0.3.0 from netlify.toml
11:37:23 AM:
11:37:23 AM: ┌────────────────────────────────────────────────────────┐
11:37:23 AM: │ 1. onPreBuild command from netlify-plugin-gatsby-cache │
11:37:23 AM: └────────────────────────────────────────────────────────┘
11:37:23 AM:
11:37:43 AM: Found a Gatsby cache. We’re about to go FAST. ⚡️
11:37:43 AM:
11:37:43 AM: (netlify-plugin-gatsby-cache onPreBuild completed in 19.8s)
11:37:43 AM:
11:37:43 AM: ┌───────────────────────────────────┐
11:37:43 AM: │ 2. Build command from Netlify app │
11:37:43 AM: └───────────────────────────────────┘
11:37:43 AM:
11:37:43 AM: $ gatsby build
11:37:50 AM: success open and validate gatsby-configs - 0.072s
11:37:51 AM: success load plugins - 0.508s
11:37:51 AM: success onPreInit - 0.011s
11:37:54 AM: success delete html and css files from previous builds - 2.739s
11:37:54 AM: info One or more of your plugins have changed since the last time you ran Gatsby. As
11:37:54 AM: a precaution, we're deleting your site's cache to ensure there's no stale data.success initialize cache - 0.057s
11:37:54 AM: success copy gatsby files - 0.038s
11:37:54 AM: success onPreBootstrap - 0.014s
11:37:54 AM: success createSchemaCustomization - 0.005s
11:38:30 AM: success source and transform nodes - 36.116s
11:38:32 AM: success building schema - 1.467s
11:40:07 AM: success createPages - 95.571s
11:40:07 AM: success createPagesStatefully - 0.108s
11:40:10 AM: success updating schema - 2.188s
11:40:10 AM: success onPreExtractQueries - 0.000s
11:40:10 AM: success extract queries from components - 0.353s
11:40:10 AM: success write out redirect data - 0.003s
11:40:10 AM: warning The icon(src/assets/images/icon.png) you provided to 'gatsby-plugin-manifest' is not square.
11:40:10 AM: The icons we generate will be square and for the best results we recommend you provide a square icon.
11:40:11 AM: success Build manifest and related icons - 0.794s
11:40:11 AM: success onPostBootstrap - 0.797s
11:40:11 AM: info bootstrap finished - 148.071s
11:40:13 AM: success run static queries - 1.372s - 3/3 2.19/s
11:40:46 AM: success run page queries - 33.282s - 78413/78413 2355.99/s
11:40:47 AM: success write out requires - 0.511s
11:41:50 AM: success Building production JavaScript and CSS bundles - 63.473s
11:41:51 AM: success Rewriting compilation hashes - 0.002s
11:48:57 AM: success Building static HTML for pages - 354.148s - 78413/78413 221.41/s
11:49:17 AM: success onPostBuild - 20.036s
11:49:25 AM: info Done building in 702.755 sec
11:49:27 AM:
11:49:27 AM: (build.command completed in 11m 44.6s)
11:49:27 AM:
11:49:27 AM: ┌─────────────────────────────────────────────────────────┐
11:49:27 AM: │ 3. onPostBuild command from netlify-plugin-gatsby-cache │
11:49:27 AM: └─────────────────────────────────────────────────────────┘
11:49:27 AM:
11:51:51 AM: Stored the Gatsby cache to speed up future builds.
11:51:51 AM:
11:51:51 AM: (netlify-plugin-gatsby-cache onPostBuild completed in 2m 23.5s)
11:51:51 AM:
11:51:51 AM: ┌─────────────────────────────┐
11:51:51 AM: │ Netlify Build Complete │
11:51:51 AM: └─────────────────────────────┘
11:51:51 AM:
11::51 AM: (Netlify Build completed in 14m 28.4s)
11:51:51 AM: Execution timed out after 15m0s
11:51:51 AM: Error running command: Command did not finish within the time limit
11:51:51 AM: Failing build: Failed to build site
11:51:51 AM: Failed during stage 'building site': Command did not finish within the time limit
11:51:51 AM: Finished processing build request in 17m12.567064531s
I have a credit card associated at my billing settings. It would be great to try again with more build minutes
perry
July 8, 2020, 7:15pm
4
hi there, we extended the build time for you. let us know if this works better!
thanks @perry
I just tried again but unfortunely I had a different error at uploading, here is the log:
4:16:51 PM: Build ready to start
4:16:57 PM: build-image version: ca811f47d4c1cbd1812d1eb6ecb0c977e86d1a1d
4:16:57 PM: build-image tag: v3.3.20
4:16:57 PM: buildbot version: 78b5536ab4f742c26705d3e953381b9cde6e22ef
4:16:57 PM: Fetching cached dependencies
4:16:57 PM: Starting to download cache of 1.1GB
4:17:03 PM: Finished downloading cache in 6.520275344s
4:17:03 PM: Starting to extract cache
4:17:36 PM: Finished extracting cache in 32.783734144s
4:17:36 PM: Finished fetching cache in 39.556006608s
4:17:36 PM: Starting to prepare the repo for build
4:17:37 PM: Preparing Git Reference refs/heads/master
4:17:39 PM: Starting build script
4:17:39 PM: Installing dependencies
4:17:39 PM: Python version set to 2.7
4:17:39 PM: Started restoring cached node version
4:17:42 PM: Finished restoring cached node version
4:17:43 PM: v10.21.0 is already installed.
4:17:43 PM: Now using node v10.21.0 (npm v6.14.4)
4:17:44 PM: Started restoring cached build plugins
4:17:44 PM: Finished restoring cached build plugins
4:17:44 PM: Attempting ruby version 2.6.2, read from environment
4:17:45 PM: Using ruby version 2.6.2
4:17:45 PM: Using PHP version 5.6
4:17:45 PM: 5.2 is already installed.
4:17:45 PM: Using Swift version 5.2
4:17:45 PM: Started restoring cached node modules
4:17:45 PM: Finished restoring cached node modules
4:17:45 PM: Started restoring cached yarn cache
4:17:45 PM: Finished restoring cached yarn cache
4:17:46 PM: Installing NPM modules using Yarn version 1.17.0
4:17:47 PM: yarn install v1.17.0
4:17:47 PM: [1/4] Resolving packages...
4:17:48 PM: [2/4] Fetching packages...
4:18:38 PM: [3/4] Linking dependencies...
4:19:22 PM: [4/4] Building fresh packages...
4:19:26 PM: success Saved lockfile.
4:19:26 PM: Done in 99.50s.
4:19:26 PM: NPM modules installed using Yarn
4:19:27 PM: Started restoring cached go cache
4:19:28 PM: Finished restoring cached go cache
4:19:28 PM: Installing Go version 1.12
4:19:28 PM: unset GOOS;
4:19:28 PM: unset GOARCH;
4:19:28 PM: export GOROOT='/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64';
4:19:28 PM: export PATH="/opt/buildhome/.gimme_cache/versions/go1.12.linux.amd64/bin:${PATH}";
4:19:28 PM: go version >&2;
4:19:28 PM: export GIMME_ENV='/opt/buildhome/.gimme_cache/env/go1.12.linux.amd64.env';
4:19:28 PM: go version go1.12 linux/amd64
4:19:28 PM: Installing missing commands
4:19:28 PM: Verify run directory
4:19:29 PM:
4:19:29 PM: ┌─────────────────────────────┐
4:19:29 PM: │ Netlify Build │
4:19:29 PM: └─────────────────────────────┘
4:19:29 PM:
4:19:29 PM: ❯ Version
4:19:29 PM: @netlify/build 2.0.27
4:19:29 PM:
4:19:29 PM: ❯ Flags
4:19:29 PM: deployId: 5f061ba3b503adc4fd51edd6
4:19:29 PM: mode: buildbot
4:19:29 PM:
4:19:29 PM: ❯ Current directory
4:19:29 PM: /opt/build/repo
4:19:29 PM:
4:19:29 PM: ❯ Config file
4:19:29 PM: /opt/build/repo/netlify.toml
4:19:29 PM:
4:19:29 PM: ❯ Context
4:19:29 PM: production
4:19:30 PM:
4:19:30 PM: ❯ Loading plugins
4:19:30 PM: - netlify-plugin-gatsby-cache@0.3.0 from netlify.toml
4:19:30 PM:
4:19:30 PM: ┌────────────────────────────────────────────────────────┐
4:19:30 PM: │ 1. onPreBuild command from netlify-plugin-gatsby-cache │
4:19:30 PM: └────────────────────────────────────────────────────────┘
4:19:30 PM:
4:19:49 PM: Found a Gatsby cache. We’re about to go FAST. ⚡️
4:19:49 PM:
4:19:49 PM: (netlify-plugin-gatsby-cache onPreBuild completed in 19.8s)
4:19:49 PM:
4:19:49 PM: ┌───────────────────────────────────┐
4:19:49 PM: │ 2. Build command from Netlify app │
4:19:49 PM: └───────────────────────────────────┘
4:19:49 PM:
4:19:49 PM: $ gatsby build
4:19:57 PM: success open and validate gatsby-configs - 0.079s
4:19:57 PM: success load plugins - 0.516s
4:19:57 PM: success onPreInit - 0.010s
4:20:00 PM: success delete html and css files from previous builds - 2.566s
success initialize cache - 0.053s
4:20:00 PM: success copy gatsby files - 0.042s
4:20:00 PM: success onPreBootstrap - 0.010s
4:20:00 PM: success createSchemaCustomization - 0.003s
4:20:33 PM: success source and transform nodes - 32.522s
4:20:34 PM: success building schema - 1.521s
4:22:18 PM: success createPages - 104.111s
4:22:18 PM: success createPagesStatefully - 0.113s
4:22:20 PM: success updating schema - 1.945s
4:22:20 PM: success onPreExtractQueries - 0.000s
4:22:21 PM: success extract queries from components - 0.246s
4:22:21 PM: success write out redirect data - 0.001s
4:22:21 PM: success Build manifest and related icons - 0.172s
4:22:21 PM: success onPostBootstrap - 0.173s
4:22:21 PM: info bootstrap finished - 151.229s
4:22:22 PM: success run static queries - 1.069s - 3/3 2.81/s
4:22:51 PM: success run page queries - 28.993s - 78413/78413 2704.50/s
4:22:52 PM: success write out requires - 0.477s
4:23:51 PM: success Building production JavaScript and CSS bundles - 59.576s
4:23:52 PM: success Rewriting compilation hashes - 0.002s
4:30:34 PM: success Building static HTML for pages - 331.818s - 78413/78413 236.31/s
4:30:53 PM: success onPostBuild - 18.702s
4:31:02 PM: info Done building in 672.004 sec
4:31:03 PM:
4:31:03 PM: (build.command completed in 11m 13.9s)
4:31:03 PM:
4:31:03 PM: ┌─────────────────────────────────────────────────────────┐
4:31:03 PM: │ 3. onPostBuild command from netlify-plugin-gatsby-cache │
4:31:03 PM: └─────────────────────────────────────────────────────────┘
4:31:03 PM:
4:33:26 PM: Stored the Gatsby cache to speed up future builds.
4:33:26 PM:
4:33:26 PM: (netlify-plugin-gatsby-cache onPostBuild completed in 2m 23s)
4:33:27 PM:
4:33:27 PM: ┌─────────────────────────────┐
4:33:27 PM: │ Netlify Build Complete │
4:33:27 PM: └─────────────────────────────┘
4:33:27 PM:
4:33:27 PM: (Netlify Build completed in 13m 57.1s)
4:33:27 PM: Caching artifacts
4:33:27 PM: Started saving node modules
4:33:27 PM: Finished saving node modules
4:33:27 PM: Started saving build plugins
4:33:27 PM: Finished saving build plugins
4:33:27 PM: Started saving yarn cache
4:33:27 PM: Finished saving yarn cache
4:33:27 PM: Started saving pip cache
4:33:27 PM: Finished saving pip cache
4:33:27 PM: Started saving emacs cask dependencies
4:33:27 PM: Finished saving emacs cask dependencies
4:33:27 PM: Started saving maven dependencies
4:33:27 PM: Finished saving maven dependencies
4:33:27 PM: Started saving boot dependencies
4:33:27 PM: Finished saving boot dependencies
4:33:27 PM: Started saving go dependencies
4:33:34 PM: Finished saving go dependencies
4:33:34 PM: Build script success
4:33:34 PM: Starting to deploy site from 'public'
4:34:19 PM: Creating deploy tree asynchronously
4:34:25 PM: Creating deploy upload records
4:37:03 PM: 156845 new files to upload
4:37:03 PM: 0 new functions to upload
5:01:52 PM: Build exceeded maximum allowed runtime
5:03:44 PM: Failed to upload file: filme/o-jogo-da-forca/index.html
5:03:46 PM: Failed to upload file: filme/o-jogo-da-guerra/index.html
5:03:47 PM: Failed to upload file: filme/o-jogo-da-trapaca/index.html
5:03:47 PM: Failed to upload file: filme/o-jik-geu-dae-man/index.html
5:03:48 PM: Failed to upload file: filme/o-jogo-com-o-fogo/index.html
5:03:51 PM: Failed to upload file: filme/o-jardim/index.html
5:03:52 PM: Failed to upload file: filme/o-jogo-da-verdade/index.html
5:03:57 PM: Failed to upload file: filme/o-irmao-que-veio-de-outro-planeta/index.html
5:04:09 PM: Failed to upload file: filme/o-jantar/index.html
5:04:10 PM: Failed to upload file: filme/o-invisivel/index.html
5:04:10 PM: Failing build: Failed to deploy site
5:04:40 PM: Failed to inform the API about a failed build, please retry the build or contact support
5:04:40 PM: Finished processing build request in 47m43.737511642s
Even with a large amount of pages, the amount of time to build (47min) seems too much. What I could be doing wrong?
perry
July 8, 2020, 8:15pm
6
seems like you are using gatsby, correct? Maybe this would be helpful:
there may also be some build plugins that could be helpful?
perry
July 8, 2020, 8:16pm
7
we can push the build time higher, if you would like us to, just let us know.
hey @perry , thanks for the anwser.
I had better results by enabling the gatsby incremental build feature!
However, since the website has a huge amount of pages and will increase, I think that I’d like to have more build time, just to be safe.
Scott
July 9, 2020, 1:50pm
9
urielscola:
onde-assistir-online
Hey @urielscola ,
For us to go ahead and up your time limit to 30 minutes, could you ensure that you have a credit card on record with us and also clarify whether you’d like the new time limit to take place on just this site or across your whole team ?
perry
July 9, 2020, 8:42pm
10
Hey just to clarify - @Scott - I already upgraded @urielscola up to 30 mins and they have a credit card on file. We are talking about going up to an hour here, is that correct?