Bitbucket based git-lfs build just hangs forever

Support
,
1

we reached a size limit on our bitbucket git repo and have extracted the files into git-lfs (hosted by bitbucket). The files show on bitbucket lfs list, we are able to clone the repo locally and see them, but our netlify builds seem to hang now.

The site name is energycap-help-center
we have currently disabled all deployments as the existing site is running in production and we wish to ensure we do not affect it until we can validate the changes. Changes are being made on the develop branch. We’ve read the forums around uninstalling the netlify LFS provider but do not have a .lfsconfig file (and never used the service). We have set the environment variables for GIT_LFS_ENABLED and the output indicates that it sees that. It just can no longer clone the repo.

Before we set the environment variable to GIT_LFS_ENABLED it tried to a build without it and failed to complete the git clone step (because the LFS files were not all pushed to bitbucket yet). That’s the only indication we’ve seen that it COULD work now that we have the files on bitbucket.

Can you help us figure out why the build is getting stuck?

Here’s the build output:

3:22:24 PM: build-image version: 3d3c7e8b4321e2c1a54a2c4584fb46ba742b1630 (focal)
3:22:24 PM: buildbot version: 3b67b7d618a27c5b3d9a768872060db0ddff3d98
3:22:24 PM: Fetching cached dependencies
3:22:24 PM: Starting to download cache of 5.9GB
3:23:12 PM: Finished downloading cache in 48.431s
3:23:12 PM: Starting to extract cache
3:23:24 PM: Finished extracting cache in 11.418s
3:23:24 PM: Finished fetching cache in 1m0.172s
3:23:24 PM: Starting to prepare the repo for build
3:23:24 PM: Git LFS enabled
3:23:25 PM: Preparing Git Reference refs/heads/develop
(then no more output for more than 10 minutes)
2

The build did eventually complete. It hung for 33 minutes with no output, but then finished. Prior builds were ~2 minutes, and subsequent builds have also been ~2 minutes. I don’t know much about the netlify build process, but it could be that the rewrite of the repo to remove all the files to LFS caused some kind of major cache invalidation that made the build take 15x as long. If it were producing output during that time we likely would have allowed it to continue - that might be a useful improvement for the netlify team.

For other people facing something like this, hopefully the post helps. We didn’t need to do anything to remove the netlify LFS module (we never installed it, and it appears despite what some other forum posts suggest it isn’t there by default). It’s possible the slowness was not caused by the LFS integration at all so much as modifying all the hashes in the repo.

For reference, the repo is 8057 files and takes up 2.1GB on disk with a fresh clone (including all LFS files). It takes me about 2 minutes to do a fresh clone on my development machine. The time it took on netlify is not outrageous, just confusing since there was no output. This can now be resolved