Getting build failure: "Failed to prepare repo"

Support
,
1

Hi there,

We’re seeing consistent build failures across all our deployments for these two sites (which both use the same repository):

The build log tells us something went wrong while ‘preparing’ the repository:

11:27:02 AM: Build ready to start
11:27:04 AM: build-image version: 30f629161c0736b1a3ecd8b418e5eeffab5c0faf
11:27:04 AM: build-image tag: v3.3.14
11:27:04 AM: buildbot version: a383ed3390159974f55f285b04457b44cb49efb0
11:27:04 AM: Fetching cached dependencies
11:27:05 AM: Starting to download cache of 533.9MB
11:27:18 AM: Finished downloading cache in 13.331919866s
11:27:18 AM: Starting to extract cache
11:27:45 AM: Finished extracting cache in 27.281222134s
11:27:45 AM: Finished fetching cache in 40.749238123s
11:27:45 AM: Starting to prepare the repo for build
11:27:46 AM: Preparing Git Reference pull/1143/head
11:27:48 AM: Failing build: Failed to prepare repo
11:27:48 AM: Failed during stage 'preparing repo': exit status 1
11:27:48 AM: Finished processing build request in 43.562272405s

It’s likely a problem with the cache, but we can’t really debug what’s going on. Typically:

It’s a monorepo, so a possibility is that artefacts of our internal dependencies (which are written to the root node_modules as a symlink) end up in the cache and cause problems when checked out in the next build.

We’ve tried proposed solutions in these posts, without success:

These are our current build settings:

image
image942×463 19.9 KB

It’s important to note our setup (using a yarn workspaces managed monorepo) worked perfectly fine with Netlify for a long time and started failing on June 3 with this build: 5ed75ae1bb0bed00070e0791. Since then, the only way for us to get a successful deploy is by retrying without the cache.

1 Like
2

Thanks for very precise debugging details - this is the best report I’ve seen all day, Kudos!

I can see from our internal logs that we are failing to correctly handle your cached repo. Clearing the cache at build time as you have discovered is the only workaround I’m aware of that works reliably, though sometimes it will enable future builds to work as well.

I have linked this thread to the bug report so I can follow up with you when we find a fix!

1 Like
4

@fool any chance for getting a ballpark ETA on this? Is there anything we can do to further debug/help resolve the problem?

We rely on Netlify for our PR reviews and might need to plan for alternatives if this continues to be a problem for us.

5

Sorry but no timeline at present. I can tell you the team is actively investigating, but we haven’t quite tracked down the root cause yet so it would be presumptuous to say there will be a fix or what the timeline might be.

I promise that I have communicated that this is a problem affecting a growing number of customers to the team.

As of yestrerday, we put additional logging in place to try to track down root cause, so I don’t think we need specific reproduction cases or research, since we now see every time it happens in our internal logs.

1 Like
6

Bump. This is introducing significant friction in our review process.

7

There are many reasons for that error; only one is covered in this thread and we have fixed the primary cause for this bug as it affected most customer, (sorry @hongaar I forgot to follow up!)

Could you tell us your deploy ID so we can confirm it is this, and not some other, bug?

8

hi @fool!

Not trying to hijack this older thread but we are running into this issue consistently now.
Our current repo set-up was working fine with the netlify CI/CD, and then on what seems to be october 30th we, started getting this behaviour.

This behaviour is consistent: the build always fails. And then if we clear cache and re-deploy it always succeeds (assuming no other unrelated errors)
We’ve tried setting up new deployer, on new account with different keys, and the issue persists. I’ve read a few different community threads about this issue, but not clear to me exactly how twe can troubleshoot further from our end.
Can you give us some pointers? what information from our end would help you look at this internally?

we are using gitmodules, which seem to be one of the common issue in other related threads, but the behaviour is the same whether the git submodule has changed or not in a given deployment.

build ID: 5fe0a877165aa4000744d08az

8:51:51 AM: Build ready to start

8:51:54 AM: build-image version: 53b83b6bede2920f236b25b6f5a95334320dc849
8:51:54 AM: build-image tag: v3.6.0
8:51:54 AM: buildbot version: 8ae026ef4905d9174e416775c6b64aa19950569b
8:51:54 AM: Fetching cached dependencies
8:51:54 AM: Starting to download cache of 177.6MB
8:51:56 AM: Finished downloading cache in 1.577144966s
8:51:56 AM: Starting to extract cache
8:52:03 AM: Finished extracting cache in 6.887758431s
8:52:03 AM: Finished fetching cache in 8.508896795s
8:52:03 AM: Starting to prepare the repo for build
8:52:03 AM: Preparing Git Reference refs/heads/staging
8:52:06 AM: Error fetching branch: https://github.com/[company name]/[repo] refs/heads/staging
8:52:06 AM: Failing build: Failed to prepare repo
8:52:06 AM: Failed during stage ‘preparing repo’: exit status 1
8:52:06 AM: Finished processing build request in 11.840423003s

9

Sure, here’s what our internal logs show in this build:

Error fetching branch: https://github.com/ASTD/atd-api refs/heads/staging: From https://github.com/ASTD/atd-api
 * branch              staging    -> FETCH_HEAD
Fetching submodule src/lib
From github.com:ASTD/web-lib
   24bd47a..4900819  master     -> origin/master
fatal: remote error: upload-pack: not our ref 7cc46f08d4bbd231d375725155ee0318464bf248
Errors during submodule fetch:
        src/lib

I’d guess that you have some bad reference in your repo somewhere that you need to remove.

10

thanks a lot for the reply!
but this project clones/checks out and builds fine locally

also this error occurs every single time, regardless of whether we’ve made changes and bumped the sha tracked version of a given submodule

would it be better to create a new issue (or write support directly?)

thanks again for taking the time to respond.

11

Hiya @lbox - nothing else for us to do on our side, unfortunately; it is in your codebase somewhere and you’ll need to ferret it out.

This thread has helped the few other customers who ended up in this situation: