Constant and frustrating build issues with LFS

Support
, , , ,
1

Hello,

I’ve lost my sanity trying to deal with LFS in Netlify and need help. Frequently encountered problems during builds #8 told me to reach out if I was still having issues. I think I might be in that 1%.

I’m building a Gatsby site from GitLab using their LFS implementation for images and fonts. When developing locally I’m able to work normally. My files are pulled from LFS and the file pointers are replaced with the real files. My local production builds work just fine.

When I started using Netlify to deploy my site this week I noticed that webpack was inlining my fonts (as base64 url blobs) into the site, but when decoded those blobs only contained the content of the LFS file pointers. After a lot of troubleshooting and trial and error I discovered that Netlify handles LFS strangely.

GIT_LFS_ENABLED=true is required to fix build cache errors (which I experienced), but that just uses git lfs clone instead of git clone. git lfs clone is deprecated and LFS is handled internally by git clone.

❯ git lfs clone git@gitlab.com:johnrichter/personal-website test-lfs-clone
WARNING: 'git lfs clone' is deprecated and will not be updated with new flags from 'git clone'

'git clone' has been updated in upstream Git to have comparable speeds to 'git lfs clone'.

My local git versions are

  • git version 2.24.0
  • git-lfs/2.8.0 (GitHub; darwin amd64; go 1.12.7)

I was able to figure out that only the font files (woff2, woff, eot, and ttf) would not clone successfully with LFS.

$ git lfs fetch --all; git lfs checkout; cat /opt/build/repo/.git/lfs/logs/*
fetch: 100 object(s) found, done
fetch: Fetching all references...
batch request: missing protocol: ""
error: failed to fetch some objects from ''
...
Skipped checkout for "src/assets/fonts/Exo_300.eot", content not local. Use fetch to download.
Skipped checkout for "src/assets/fonts/Exo_300.ttf", content not local. Use fetch to download.
Skipped checkout for "src/assets/fonts/Exo_300.woff", content not local. Use fetch to download.
Skipped checkout for "src/assets/fonts/Exo_300.woff2", content not local. Use fetch to download.

The first issue I’ve found is with Git. batch request: missing protocol: "" and error: failed to fetch some objects from '' resolve to issues with git lfs not having SSH support and the dependence that LFS has on the remote being origin. I was able to add origin by running git remote add origin https://gitlab.com/johnrichter/personal-website.git before gatsby build, but that alone didn’t fix issues since it happened after the initial clone. After adding git lfs fetch --all && git lfs checkout I was able to finally able to get LFS to load all files correctly.

My final build command was git remote add origin https://gitlab.com/johnrichter/personal-website.git && git lfs fetch --all && git lfs checkout && gatsby build.

I thought I was home free, but now I’m running into issues with subsequent builds that are saying that remote origin already exists. This does not happen if I manually build without the cache.

Git LFS enabled
Preparing Git Reference merge-requests/15/head
Error adding remote: fatal: remote origin already exists.
Failing build: Failed to prepare repo

This support question hinted at writing a script that adds origin if it doesn’t exist before the build and then removes origin after the build to avoid the remote being configured for the next build from the Netlify cache, but its not working. I’ve tried to run this script directly as the build command and as a yarn preinstall command, but nothing works. Using preinstall did show me that my build script environment needs to be set carefully. When running the script as a normal build script I tried using set -ex bash ..., but that didn’t work. Perhaps I need to do that within the script.

At this point I’m contemplating on doing my builds and deployments within GitLab CI via Netlify CLI because I think it’d be easier. I’d rather use Netlify CD to do this bercause of the ease of automatic deploy previews though!

The repo is here. My Netlify site is jrichter-io.netlify.com. Links to the builds for troubleshooting are in my follow up comment because I’m limited to 6 links for new topics in this forum.

Thanks for reading to the end. I’d really really appreciate help. I’m not sure what to do. Maybe I’ll pull down the build image, read its source, and try to rapidly test solutions locally in Docker.

2

Links to build logs:

3

More links to build logs. This 6 link limit is really annoying.

4

Good news everyone!

I found build-image#175 which described my exact issue with fonts not being loaded from LFS. It suggested using GIT_LFS_FETCH_INCLUDE to override the lfs.fetchinclude filter set within the build image.

I had no idea this config existed and I’m going to guess that lfs.fetchinclude is being used for large file protection. I thought this was a Git env variable at first, but after some digging in the git lfs repo it seems that it’s actually a Netlify-specific env variable. Why this isn’t documented somewhere has been beyond incredibly frustrating. I’ve spent over 24 hours and over 200 build minutes trying to figure that all I needed was to override this config setting.

My build command is now simply gatsby build with GIT_LFS_ENABLED=TRUE and GIT_LFS_FETCH_INCLUDE=* in the UI.

18

I’m still having issues with the cache though. All builds using the cache fail with Error adding remote: fatal: remote origin already exists.. I’m not doing any git repo manipulation anymore and I haven’t figured out how or found the source code for the build system’s repo restoration from cache. It is clearly trying to update the remote.

1 Like
5

An example build

31
31696×378 22.7 KB

6

By the way, these “new user limits” are incredibly frustrating. I just want to give as much detail to the problem as possible to make troubleshooting easier for y’all and I’m forced to do that with many comments on my own thread.

21

Oh, the irony.

7

@johnrichter, thank you for the detailed information and welcome to our Netlify community site. Sharing all of those discoveries here will (hopefully) be quite beneficial to other people searching community for similar issues.

Back to the issue you are asking about! The build linked to most recently failed because an attempt to update the repo was happening using a cached repo clone which was now out of date.

I retried that same deploy preview with “Retry deploy” button on the build page itself and used the option “Clear cache and deploy site” when confirming. This new deploy was successful which can be checked here:

Would you please confirm that that version of the deployed site has all required assets (like the embedded fonts) being served correctly now? It looks correct to me but I could be missing something. I can confirm the font warnings in the javascript console do not occur on the deploy above (but did see in the non-working deploys).

Unfortunately, I do believe this workaround of clearing the build cache will be needed for any deploys with a cached version of the repo (meaning any deploys for a branch which has been deployed at Netlify before the repo origin change was made). However, this should only be required once per branch and they should continue to work correctly after this.

Would you please try clearing the cache and redeploying for any other branches still getting the deploy error about the origin? If this doesn’t work and/or if there are other questions, please let us know with follow-up post here.

Finally, I also agree the[new user limits on links in posts (which are a build in part of the software our community site uses) are often frustrating. They are unfortunate requirement because of the spam/bot traffic which target sites with a comments/posts feature. (In other words, spammers are why we can’t have nice things. :imp: )

Again, though, we do appreciate you sharing all of this hard work and we’re happy to troubleshoot further if issues persist.

1 Like
9

@johnrichter, I also skipped you ahead to “trust level 3” since I’m sure your not a spammer. :slight_smile:

1 Like
10

You’re welcome and thanks for the bump to TL3! I’ve heard of Discourse, but never actually used it before so I appreciate the link to their docs. Its a shame that some people have to ruin it for the rest of us :frowning:

Your deploy looks good :+1: All the fonts are loading correctly and the rest of the site is good. What’s interesting is that I’ve been running the “Clear cache and deploy site” option in the UI to retry builds and achieve successful deployments while this has been happening.

I just ran some tests with a new branch and merged that branch into one with an existing one (develop/v2) just to check all cases since I did the cache clearing yesterday evening. The new branch added new font files in LFS and uses those new fonts throughout the site. Everything built correctly and the deploy previews look good!

All with no git issues. As a bonus, now that the Netlify cache is working correctly with my repo, my Gatsby caching plugin can now cache optimized images and assets between builds. They went from 5-7m down to 1-3m! I’ll have to keep an eye on my cache size over time and reset it if it gets too large, but it’s a huge win in my book. When I first started this on early this week my builds were hitting the 15m timeout… Pre-optimizing all of my images was also a big win.

Thanks for the help @luke!

11

If you do notice issues with the cache growing over time, please feel free to follow-up here (or in a new topic if you prefer). Also, thank you for follow-ing up to let us know you got this working and what your results are so far.

1 Like
12

Hello :wave:

I have had a lot of issues the past hour to make LFS with GitLab to works.

I was using the following netlify.toml build configuration:

[build.environment]
  HUGO_VERSION = "0.68.3"
  HUGO_ENV = "production"
  HUGO_ENABLEGITINFO = "true"
  GIT_LFS_ENABLED = "true"
  GIT_LFS_SMUDGE_FILTER = "*.mp3"
  GIT_LFS_FETCH_INCLUDE = "*"

But the netlify CI was basically ignoring all the git lfs environment variable.

If I set the exact same variable in the Web UI it works.

I understand that to read the netlify.toml file one has to already fetch the repository, ideally this case should be handled either by a warning, a failed build, or an additional LFS fetch after the usual git clone.

Hope this feedback is useful :slightly_smiling_face:

13

HI, @Mayeu, and welcome to the Netlify community site. :smiley:

Yes, all git specific environment variables must be set in the UI as the git clone must occur before the netlify.toml file can be read so setting the environment variables there won’t work.

I’d be happy to enter a feature request for this. Would you please confirm what would be your preferred solution for this feature request?

Is your preferred solution to do one of the following? If so, which one?

  • do a second git clone
  • print a warning but build
  • fail the build with an error

If there is some other preferred solution, please let us know.

14

Hi!

I’m using Netlify Large Media for images. During the build, I’d like to optimise some images and read their dimensions. For that I need the files to be fetched in the build process, instead of having just the pointer files present.

However, I was unable to disable the GIT_LFS_SKIP_SMUDGE=1 flag or fetch the files in any other way. Running git lfs fetch --all in the build command fails due to missing credentials. Setting GIT_LFS_ENABLED=true leads to failing builds as well (“Configured environment variable GIT_LFS_ENABLED=true conflicts with Netlify Large Media.”)

Screen Shot 2020-07-19 at 11.36.57
Screen Shot 2020-07-19 at 11.36.572146×910 238 KB

Any help would be greatly appreciated!

15

This is covered in the “Limitations” section of the Large Media documentation here:

Quoting:

  • Files tracked with Large Media are uploaded directly to the Netlify Large Media storage service on push, completely bypassing the site build. This saves build time, but also means that the files are not available to tools that process asset files during the build, such as Hugo’s image processing or the gatsby-image plugin. Depending on your needs, you may be able to replace this functionality with Netlify’s image transformation service.

Is not possible to do build time transforms with Large Media.

If you want to uninstall Large Media for this reason, would you please confirm you have read the following Support Guide?

If you have read that, please confirm you have backups of all files and let us know which site to remove Large Media from. To identify the site, you can send us the site subdomain, API ID (the id not the key), or a link to the site (whichever you prefer).

Once we know which site, we’ll reply soon to confirm the add-on has been uninstalled.

​Please let us know if there are any questions and, if not, please tell us the site to do this for. If you want to leave Large Media active for this site, we’ll take no action unless you tell us to do so.

16

Hi @luke — thank you for the detailed response. I’ll do my best to check all the docs first next time!

In case I decide to disable Large Media, I’ll open a topic in #admin.