Ruby detection fails if Gemfile is a symlink

slim · December 17, 2020, 3:37pm

in order to share my gem config between several repos I made the Gemfile and Gemfile.lock symlinks into a submodule.

While the submodule is updated by your git clone command and so the linked file exists, the ruby setup and gem caching do not work.

Replacing the symblink with the file makes everything works again.

Any ideas? I can put everything in the build command in the netlify.toml but then you lose gem caching.

jonsully · December 17, 2020, 4:03pm

Hey @slim

Does this work locally? I wonder if this is actually something having to do with Bundler and the way it opens the file - I don’t know off hand but “follow if it’s a symlink” sounds like a flag you’d pass to the File.open method within Bundler, not necessarily something having to do with the build environment. Nifty issue though

–
Jon

slim · December 17, 2020, 4:21pm

Hey @jonsully

What doesn’t work is the Deployment build installing bundler (getting version from gemfile.lock) and fetching gems from cache. this doesn’t happen locally with netlify build.

My guess is the file stat for Gemfile.lock fails if a link

I can always do the following - but we lose the gem caching between deploy builds…

[build]
command = "git submodule update --init --remote && gem install bundler && bundle install && bundle exec jekyll build --config '_config.yml,_config_staging.yml'"

I just realise the netlify build’s clone does’t update submodules to latest commit either (ie no --remote), just that recorded in the superproject (i guess it’s using --recurse-submodules) We have no control over that.

So that’s 2 reasons this won’t work

fool · December 23, 2020, 12:34am

Yup, our scripts specifically say this must be a file:

github.com

netlify/build-image/blob/xenial/run-build-functions.sh#L308


      
          fi
          
          
if [ -n "$NPM_TOKEN" ]
          then
            if [ ! -f .npmrc ]
            then
              echo "//registry.npmjs.org/:_authToken=${NPM_TOKEN}" > .npmrc
            fi
          fi
          
          
# Automatically installed Build plugins
          if [ ! -d "$PWD/.netlify" ]
          then
            mkdir "$PWD/.netlify"
          fi
          restore_cwd_cache ".netlify/plugins" "build plugins"
          
          
# Ruby version
          local tmprv="${RUBY_VERSION:=$defaultRubyVersion}"
          source $HOME/.rvm/scripts/rvm
          # rvm will overwrite RUBY_VERSION, so we must control it

So what you’re seeing is it working as designed, though I understand, not as you’d wish.

You can of course run bundle install yourself in this case, before building, and that may work for you, depending on your site layout.

Re: how we fetch submodules, it looks like this, and indeed, you cannot change the behavior, but if it is suboptimal, do please feel empowered to file an issue (here), about what you’d like to see work differently and explain your use case a bit, please!

git clone <project> ; cd project ; git submodule update -f --init

slim · December 23, 2020, 10:33am

@fool

Thanks Chris

So what you’re seeing is it working as designed, though I understand, not as you’d wish.

Ah, I missed that in the scripts, even though I had just looked at them I think I incorrectly assumed that behaviour was tied up with cache fetching before the scripts.

You can of course run bundle install yourself in this case, before building, and that may work for you, depending on your site layout.

When I tried that I found I lost the the caching of Gem modules and some with platform builds are taking a very long time to install. In the end I changed tack and created a Gem with just the shared dependencies in and gave each module a simple Gemfile that depends on that.

By the way would it be possible to share the cache more between different builds? eg a preview and published build both usually use the same Gemfile dependencies so would speed builds up if unchanged.

Re: how we fetch submodules, it looks like this, and indeed, you cannot change the behavior, but if it is suboptimal, do please feel empowered to file an issue (here), about what you’d like to see work differently and explain your use case a bit, please!

git clone <project> ; cd project ; git submodule update -f --init

So I’m picking up maintaining a site and the use of git modules. I’m not convinced they are the best way of managing deps here but it is what we have. Interesting to see what git commands you use I was expecting some variant of git clone --recurse-submodules.

Anyway, this hopefully explains our use case. But in summary, we need the latest commit of each submodule when we deploy. So I prefix all builds with a git submodule update --init --remote before all builds (local using Netllify CLI, Netlify and GitHub workflow. I don’t need --recurse in this case as the git submodules used in sub sections of the site is only for their preview builds. The cost of the separate iteration through all submodules with two git submodule commands is probably not worth worrying about

Thanks again and wishing all at Netlify a happy Christmas

perry · December 23, 2020, 9:49pm

thanks for the well wishes, @slim! we’ll have a skeleton crew on deck over the hols but we’ll be around in the week between xmas and new years a bit more, and then back to normal after NYE.

fool · January 1, 2021, 12:07am

By the way would it be possible to share the cache more between different builds? eg a preview and published build both usually use the same Gemfile dependencies so would speed builds up if unchanged.

We intentionally DON’T do this since many people (us included) test dependency or node version upgrades in a branch or PR. It’s not a bad idea but I have no idea how we’d implement it. I’ll ask our build team to take a look at this thread to see what they think though

slim · January 1, 2021, 10:13am

Yeah, I thought after I posted it would probably need to be opt in.

Happy new year!