[Support Guide] How can I optimize my Netlify build time?

It looks like our team is bumping up against the threshold allotted for Open Source projects. We build and run e2e tests using CircleCI before accepting a pull request; is it possible, using the Netlify API, to publish a “Deploy Preview” from Circle CI?

The hope being that there’s no reason to build our application multiple times on different platforms and that this could save us some build minutes.

If not, CircleCI has a “Manual Approval” workflow that we use to gate expensive tasks. Is there a mechanism for kicking off a deploy preview manually from a GitHub status check?

@fool @perry Yikes, this 502 error is happening a lot, I don’t see it in the logs before Feb 10. Is there any reason that they could be connected?

Doesn’t seem like 502’s would be related, no, @salmonax. 502 usually means that you have a very large site that isn’t being synced fast enough. You could try adding a --timeout 900 flag with your CLI usage to see if it helps.

heya @dannyrb , sure - you can publish from Circle but the most obvious path would be to use our cli to publish it directly, sending the already-built files from Circle. The default netlify deploy publishes in “draft” mode which is what some call a deploy preview (but the URL will be something like https://hash--yoursitename.netlify.com rather than the “usual” deploy-preview-xyz--sitename.netlify.com") - regardless, it is not published at your production URL automatically which is what I think you are after :slight_smile: You could also choose to trigger a build only via Circle - but it’d be for HEAD on whatever branch, rather than the specific commit that Circle built. You’d use that via an incoming build hook which we can configure to build even when the “usual” automatic commit-based deploys are turned off.

I’d use the second workflow to “gate” those deploys - only build on certain conditions, which you can check for at Circle.

Let me know what you think, and I can help reconfigure (either removing the repo link entirely so you MUST publish with the CLI, or deconfiguring automatic builds in favor of webhooks you configure).

1 Like

Sorry to bother you again. Is there a way to manually trigger the build + deploy from Netlify’s UI? I want to build and deploy only at specific times, while disabling all automated build + deploy actions. Thanks!

Yes, but what you had me do is totally unlink Netlify from your git provider, so you’ll have to relink the repo now to do that :upside_down_face: . Please review the very long post that started this topic as to all the different ways you can control your builds - it will take a combination of setting appropriate build branches in the deploy settings, as well as using features like our ignore builds script (specific example here: [Support Guide] How can I optimize my Netlify build time? - #10 by marcus), or maybe using [skip ci] or [skip netlify] (documented here: Manage deploys | Netlify Docs).

1 Like

Thank you, I went with [skip ci].

There is now Stop Builds functionality, so you can turn it on and off at will :slight_smile:

If that doesn’t suit your needs, could you let us know more details about why it doesn’t? We can still turn off here but hoping we won’t need to anymore since that feature is released.

1 Like

It’s close to what I was looking for!
Thank you so much!
I’m going to use that feature :smiley:

Hey @fool thanks for the suggestion! While I hit the same issue as @nicolasrouanne, using a script works.

I have the same issue as @nicolasrouanne, the command works locally (macOS) but not when run on Netlify in the build step.

Hey @kmiyashiro,

What error are you getting? Have you tried the .sh script as advised? What did this show/tell?

Hi there. Reading this thread with interest. My use case is that I have active editors (people) on a CMS like Forestry.io. Imagine that they log in and edit ten documents in series, over 10 minutes. In this case I wish to avoid building the site ten times in ten minutes.

Is there a way to ask Netlify to not build a commit until X minutes have passed since the last commit?

The way this would work would be like a tape-delay on Live TV. Something like “wait five minutes to build this, and ignore if a new commit comes first?”

I’d enable a “build delay” variable on every site, immediately. I bet it would drastically reduce CPU for builds on many sites.

That is not a feature we have today, though it’s not a bad suggestion so I will get a feature request filed, though I do not expect it to be implemented soon since you are the first to ask for it.

If your CMS commits to git, we will skip intermediate builds while some are queued - but for most build hook triggered builds (I think you maybe use this feature to trigger a build on update from forestry? Build hooks | Netlify Docs).

So I guess you have a few options based on what’s available today:

  • reconfigure forestry not to send so many updates. Not sure if that is possible, but it is the best advice I know of. Some other headless CMS’s (contentful for instance) have a setting about how often they notify webhooks about changes - e.g. “on intentional save, not on autosave”). Maybe forestry has something similar?
  • don’t autopublish - do so manually. Use a build hook such as I linked BUT NOT FROM FORESTRY - instead, from something like a cron server, or a scheduled zap to build every hour during the editing day, or once a day. Or, only when someone pushes a button or runs an API call manually.
  • send the updates from forestry to an intermediary system like zapier which then triggers our builds selectively: Zapier can notice “I’ll only forward this if I haven’t just forwarded one in the past few minutes” or “I’ll notify about every 5th one” and other similar logic.

Except for reconfiguring forestry which I understand may not be possible or desirable for other reasons, I don’t love giving out workarounds that require other systems, but to accomplish your goal, that’s the best I have for you today.

That’s so cool, thanks for the reply.

Often, in a first-party application built by a tech company for a tech company, building from Circle or elsewhere would provide ample control, but in this particular use case, we are trying to cater to small business “mom and pop” users who are minimally tech-savvy (hence Forestry). If Forestry has such a feature I would use it. Building on cron would likely not result in a net savings due to many days where no edits would happen at all. Zapier’s not a bad suggestion. All that said, if there could be a deploy-delay setting in Netlify, I’d think it would REALLY reduce overhead, likely more for Netlify than for me :wink:

Thanks again for filing the feature request

I get the same exact error as nicolasrouanne:

I tried creating a bash script as you suggested, I still get this message:

4:41:26 PM: Detected ignore command in Netlify configuration file. Proceeding with the specified command: './netlify-ignore.sh'
4:41:26 PM: Attempt to ignore dependabot deploys in script
4:41:26 PM: User-specified ignore command returned exit code 0. Returning early from build.
4:41:26 PM: Failed during stage 'checking build content for changes': Canceled build due to no content change
4:41:26 PM: Finished processing build request in 18.713326072s

Keep in mind that the command itself (git log... grep dependabot) works perfectly fine when run locally, it just does not work in netlify’s build process.

Edit: I just realized I have “dependabot” in my test commit, let me try removing it and seeing if the script works…

Edit2: It works! Thanks! I would suggest changing the original blog post to put the command in a script.

1 Like

Hi @luke, I want to deploy my site that is a Jekyll blog at one path segment (site.com/blog) and a React SPA at another (site.com/app). Thus I use jekyll build and npm run build respectively during the build process.

Since it’s not a standard config, Netlify doesn’t cache the gems nor npm packages.

Is there a way for me to specify what custom directories I want to be cached / files to check for updates?

I’ve looked around the docs and blog posts but couldn’t find anything about this.

Thank you!

Hi, @brycewray, you might workaround this by manually backing up the directories in question to /opt/build/cache and then manually restoring them again as part of the site build.

I don’t have “prior art” for doing this outside of a very simple shell script. However, there is a build plugin which allows for saving a directory listing of all cache contents for debugging:

Using this plugin might be helpful if you do decide to write custom code to handle this.

The code might work like this:

  • check to see if the cached files exist
  • if so move those files from the cache to the required location
  • proceed with the usual build
  • if the build is successful, copy the required files back to the cache directory
  • the build system will automatically back up the /opt/build/cache` directory after the successful build is complete

If this is done, the files will be able to be copied from the cache (using the same process above) at the start of the next build.

​Please let us know if there are other questions about this.

1 Like

Actually was @brianjychan who asked that one, but autocorrect probably got you on the “br” front. :wink: Thanks, though.