Build fails with Focal image due to unwanted old Ruby

Support
1

My site (site name: skeema) is built using Hugo. Today I switched it from build image Trusty to Focal. (I skipped over Xenial, never used it.)

My builds immediately started failing due to inability to install an old Ruby version (2.3.6). This doesn’t make sense to me, since my builds don’t directly use Ruby at all and I’m not requesting an old Ruby version anywhere.

Reviewing the build logs, besides Ruby I also see old versions of other software: Go 1.10, Node 8.17.0. I don’t specify any of these in my env vars – literally the only thing I set was HUGO_VERSION.

Why is my build using all these old versions by default, rather than the Focal defaults?
It feels like Netlify somehow cached dependency versions from Trusty but isn’t updating them for Focal defaults?

If I explicitly set RUBY_VERSION to 2.6.2, my build starts working again. So I’m leaving that in place for now. But since I don’t directly use Ruby at all, I’d prefer not to keep this in my config.

Here’s the log of the failing build (prior to explicitly setting RUBY_VERSION) even when doing a cache-clearing build:

11:59:44 AM: Build ready to start
11:59:46 AM: build-image version: 3802cb5fb68688f5cbd80f2e312aa0ce78813b3a
11:59:46 AM: build-image tag: v4.0.4
11:59:46 AM: buildbot version: 487f6d01b44481b4d57253f3d6071f198697110f
11:59:46 AM: Fetching cached dependencies
11:59:46 AM: Failed to fetch cache, continuing with build
11:59:46 AM: Starting to prepare the repo for build
11:59:47 AM: No cached dependencies found. Cloning fresh repo
11:59:47 AM: git clone https://github.com/ [REDACTED private repo]
11:59:48 AM: Preparing Git Reference refs/heads/master
11:59:48 AM: Parsing package.json dependencies
11:59:49 AM: Starting build script
11:59:49 AM: Installing dependencies
11:59:49 AM: Python version set to 2.7
11:59:51 AM: Downloading and installing node v8.17.0…
11:59:51 AM: Downloading https://nodejs.org/dist/v8.17.0/node-v8.17.0-linux-x64.tar.xz…
11:59:51 AM: Computing checksum with sha256sum
11:59:51 AM: Checksums matched!
11:59:53 AM: Now using node v8.17.0 (npm v6.13.4)
11:59:54 AM: Started restoring cached build plugins
11:59:54 AM: Finished restoring cached build plugins
11:59:54 AM: Attempting ruby version 2.3.6, read from environment
11:59:55 AM: Required ruby-2.3.6 is not installed - installing.
11:59:55 AM: Searching for binary rubies, this might take some time.
11:59:55 AM: Found remote file https://rvm_io.global.ssl.fastly.net/binaries/ubuntu/20.04/x86_64/ruby-2.3.6.tar.bz2
11:59:55 AM: Checking requirements for ubuntu.
11:59:56 AM: Missing required packages: libssl1.0-dev
11:59:56 AM: RVM autolibs is now configured with mode ‘2’ =>
11:59:56 AM: ‘Allow RVM to use package manager if found, fail if dependencies are missing. This is default.’,
11:59:56 AM: please run rvm autolibs enable to let RVM do its job or run and read rvm autolibs [help]
11:59:56 AM: or visit RVM: Ruby Version Manager - for more information.
11:59:56 AM: Found undesired packages: libssl-dev
11:59:56 AM: RVM autolibs is now configured with mode ‘2’ =>
11:59:56 AM: ‘Allow RVM to use package manager if found, fail if dependencies are missing. This is default.’,
11:59:56 AM: please run rvm autolibs enable to let RVM do its job or run and read rvm autolibs [help]
11:59:56 AM: or visit RVM: Ruby Version Manager - for more information.
11:59:56 AM: Requirements installation failed with status: 1.
11:59:56 AM: ruby-2.3.6 - #gemset created /opt/buildhome/.rvm/gems/ruby-2.3.6
11:59:56 AM: Required ruby-2.3.6 is not installed - installing.
11:59:57 AM: Searching for binary rubies, this might take some time.
11:59:57 AM: Found remote file https://rvm_io.global.ssl.fastly.net/binaries/ubuntu/20.04/x86_64/ruby-2.3.6.tar.bz2
11:59:57 AM: Checking requirements for ubuntu.
11:59:58 AM: Missing required packages: libssl1.0-dev
11:59:58 AM: RVM autolibs is now configured with mode ‘2’ =>
11:59:58 AM: ‘Allow RVM to use package manager if found, fail if dependencies are missing. This is default.’,
11:59:58 AM: please run rvm autolibs enable to let RVM do its job or run and read rvm autolibs [help]
11:59:58 AM: or visit RVM: Ruby Version Manager - for more information.
11:59:58 AM: Found undesired packages: libssl-dev
11:59:58 AM: RVM autolibs is now configured with mode ‘2’ =>
11:59:58 AM: ‘Allow RVM to use package manager if found, fail if dependencies are missing. This is default.’,
11:59:58 AM: please run rvm autolibs enable to let RVM do its job or run and read rvm autolibs [help]
11:59:58 AM: or visit RVM: Ruby Version Manager - for more information.
11:59:58 AM: Requirements installation failed with status: 1.
11:59:58 AM: ruby-2.3.6 - #importing gemsetfile /opt/buildhome/.rvm/gemsets/default.gems evaluated to empty gem list
11:59:58 AM: ruby-2.3.6 - #generating default wrappers…
11:59:58 AM: Error running ‘run_gem_wrappers regenerate’,
11:59:58 AM: please read /opt/buildhome/.rvm/log/1625846398_ruby-2.3.6/gemset.wrappers.default.log
11:59:58 AM: Using /opt/buildhome/.rvm/gems/ruby-2.3.6
11:59:58 AM: Warning! Executable ‘ruby’ missing, something went wrong with this ruby installation!
11:59:58 AM: Warning! Executable ‘gem’ missing, something went wrong with this ruby installation!
11:59:58 AM: Warning! Executable ‘irb’ missing, something went wrong with this ruby installation!
11:59:58 AM: Using ruby version 2.3.6
11:59:58 AM: /opt/buildhome/.rvm/scripts/override_gem: line 19: gem: command not found
11:59:58 AM: Error installing bundler
11:59:58 AM: Build was terminated: Build script returned non-zero exit code: 1
11:59:58 AM: Creating deploy upload records
11:59:58 AM: Failing build: Failed to build site
11:59:58 AM: Failed during stage ‘building site’: Build script returned non-zero exit code: 1
11:59:58 AM: Finished processing build request in 12.086976143s

3 Likes
4

Hi there, @evanelias :wave:

Welcome to the Netlify Forums :netliconfetti: this is a great question.

When folks create their first site on Netlify and we run the first build, we set a version for all of the major languages that match the default version installed on the build image at the time. This is then stored on the site’s repo object.

I believe that if you re-connect your repo, it will reset the versions for you. In order to try this, you can follow these steps:

For your selected site, go to Site settings > Build & deploy > Build settings , select Link to a different repository , and then select the same repository to re-link your site to it.

Let me know if this works!

10 Likes
5

Awesome, thank you, that fixed it!

Out of curiosity, is this default-version-pinning behavior documented anywhere or exposed in the UI anywhere? It seems counter-intuitive to pin versions of languages that my build isn’t even directly using… and as far as I can see, the Trusty build image migration guide didn’t mention this either.

That said, I’m relieved the fix was easy and quick. Thank you again for the excellent and prompt support @hillary , I greatly appreciate it!

1 Like
6

Hey there, @evanelias :wave:

So glad the fix was quick! To your point about documentation, thanks for raising that! I have shared your feedback with both our team that designed the build plugin as well as our Docs team.

1 Like
7

thanks for your question my issue is fixed too.

8

Good points! We’re working on adding some more helpful information to the deploy logs so that it’s clearer how the different language versions are selected. I’ve also added some extra notes about default version pinning in the final section of the migration guide.

In the docs, we mention pinning to the default image version in the dependency management doc, at the top of each of the Node, Ruby, and Python sections.

I understand that it seems strange that we would set the version for a language you’re not even using. However, it’s not always possible to know for sure that a language isn’t being used, and our priority is to keep builds running as they always have, even if we change a default behind the scenes.

2 Likes
9

Thank you @verythorough! The extra note in the migration guide is helpful and more clear, and the reasoning for the pinning makes better sense now.

Regarding the default-version pinning already being mentioned in the dependency management doc, thanks, I did indeed miss that. For me, there are still 2 points of confusion there:

  1. Unless I missed something, that doc doesn’t address what happens if I change the selected build image. It says the pinning is done to preserve working builds “even if the build image’s defaults change”, but that’s not the same scenario as changing the selected build image entirely. My assumption had been that changing the build image would also change the default versions, since other CI systems I’ve used work that way.

  2. Because my build isn’t using Ruby, Node, Python, etc, I didn’t even read these sections of the docs in the first place. So any mention of pinning there would naturally be missed anyway :slight_smile:

Longer-term I think an ideal fix may be to prompt the user upon changing build image: do you want to keep all default dependency versions from your old image, or move to the defaults of your new image? And/or maybe have a UI page that shows what versions are being used for each dependency, along with the configuration source (env var, netlify.toml, build image default, old pinned default, etc).

That said, as a dev tools engineer I totally understand there’s a very tricky balance between “simple and uncluttered UI without too many configurables” and “enough flexibility for power users and experts”. But eventually Xenial will hit EOL too some day, and I’d imagine a much greater number of users will encounter this situation at that point, which would be painful with the current functionality.

11

Hi @evanelias,

Those are some really great suggestions and we cans how it could have been confusing for the users. We would definitely keep this in mind moving forward. The team is already working to improving a lots of things including the docs, and feedback like yours would help us make the most appropriate changes.

12

The tl;dr of this solution is set a RUBY_VERSION environment variable, or add a .ruby-version file to the site’s base directory.

13

@brianzelip sorry but that TL;DR is not accurate… the solution here was to un-link and then re-link the git repo, in order to reset the pinned dependency versions.

If you instead set an arbitrary Ruby version, you’ll have slower builds and it may eventually break again some day if that Ruby version goes missing. I tried this originally and it worked (see my original post) but my question was about how to not do this.

3 Likes
14

That’s right @evanelias, glad you followed up. This thread helped me solve my different problem of wanting to use ruby and having my new Focal build image break, and I was really directing that comment at my future self in case I experienced another build break. I’ll leave my earlier comment here for provenance.

1 Like
15

Hi, @evanelias. I realize this thread has been quiet since July but came across this and wanted to point out a small detail (small but maybe important to someone out there). I’m following up about this comment in particular:

Technically speaking, behind the scenes an “arbitrary” version (for certain definitions of arbitrary) is being set even with the repo re-linking because again the language versions all get pinned again at that time.

So, whether or not you set RUBY_VERSION=2.7.2 manually (either via the Netlify web UI or using netlify.toml) or if you relink the repo, both solutions still pin the Ruby version.

Also, 2.7.2 is the default for Focal and if you are not using Ruby manually setting it to that (or re-linking as you recommend) will be faster as there will be no need to install a new Ruby version when building.

So, back to my comment about “for certain definitions of arbitrary”. People often use words in ways that differ from their dictionary definitions so I want to be sure. So, if “arbitrary” means “random” or “determined by impulse rather than reason” then yes, a randomly chosen Ruby version is bad and slower unless you luckily guess the default.

By comparison, your suggestion of using the relinking will definitely guarantee the default is pinned and is therefore preferable. :+1:

I did just want to point out that if “arbitrary version” was being used (connotatively) to mean “fixed” instead of “automatic” then I wanted to clarify that both solution do created a “fixed” Ruby version. It was a small detail but I did want to point it out.

16

By “arbitrary” I do in fact mean “random or determined by impulse rather than reason”.

In other words: if your build doesn’t need a specific Ruby version or doesn’t even use Ruby at all, but you’re setting RUBY_VERSION anyway just to work around this issue, you are choosing a Ruby version arbitrarily by definition.

I do understand that a version is pinned either way, as @hillary explained very clearly. fwiw personally I think this silently-pinned-at-repo-linking-time behavior is extremely counterintuitive and runs contrary to every other CI/build system I’ve encountered, but I suppose that’s a separate topic…

17

To give a little context, the practice of pinning dependency versions was started in the early days of Netlify, with the goal of making sites “just work” out f the box, and continue working even if default versions on the build image change. The pinning happens for Ruby, Node, Yarn, and Go, and happens for all sites, because it’s difficult to clearly identify that a site isn’t using a language.

However, as we started introducing new build image versions, and now are deprecating old ones, this pinning system isn’t serving folks’ needs as well as it used to. As a result, we’re working on changing how we handle dependency versioning, keeping in mind that we’re still optimizing for builds that work the first time and every time thereafter, with minimum configuration required.

We’ll post here and in the Feature Updates category when we do!

1 Like
18

Thanks so much for this explanation

1 Like
19

Glad you found this useful, @ubuskaspeed :netliconfetti:

20

Update: To mitigate this issue for folks going forward, we are now automatically setting the RUBY_VERSION to 2.7.2 when you upgrade to Focal and we detect that an old version of Ruby is pinned (lower than 2.4). Thanks for your patience and feedback, everyone!

1 Like
21

thanks for the awesome information.

1 Like