Cached builds keep failing on github fetch

Support
1

This problem is happening again - every time I have to clear the cache and rebuild! :frowning:

please see https://answers.netlify.com/t/build-keeps-failing-when-fetching-from-github/

I can only assume it is a timeout as the fetch is slow. Or the submodules use is clashing with the cash?

2

can you provide a deploy log please?

3

@perry sure. It’s always the same, with zero visibility. In the previous thread someone dug into the logs.

Site is Netlify App

5:02:00 PM: Build ready to start
5:02:01 PM: build-image version: d84c79427e8f83c1ba17bcdd7b3fe38059376b68
5:02:01 PM: build-image tag: v3.6.1
5:02:01 PM: buildbot version: f8648819b2de8637c0fbd87ac67f86c2856f6ad3
5:02:01 PM: Fetching cached dependencies
5:02:01 PM: Starting to download cache of 192.8MB
5:02:03 PM: Finished downloading cache in 1.368847495s
5:02:03 PM: Starting to extract cache
5:02:07 PM: Finished extracting cache in 4.711865337s
5:02:07 PM: Finished fetching cache in 6.1274091s
5:02:07 PM: Starting to prepare the repo for build
5:02:08 PM: Preparing Git Reference refs/heads/master
5:02:11 PM: Error fetching branch: https://github.com/w3c/wai-website refs/heads/master
5:02:11 PM: Failing build: Failed to prepare repo
5:02:11 PM: Failed during stage 'preparing repo': exit status 1
5:02:11 PM: Finished processing build request in 10.002894282s
4

Thanks for your patience while I tried to figure out what is happening here, @slim ! In the good news department, we are seeing a git error when we pull. In the bad news department, it is something you’re going to have to fix, but hopefully my sharing it here will help!

When we have build cache, this is what we run:

  1. fetch and unpack cache
  2. change to cached copy of your repo and run:
git fetch -f -u remote refs/head/master
git checkout -f FETCH_HEAD
git submodule update -f --init

And that leads to this error, which you’ll want to correct:

Error fetching branch: https://github.com/w3c/wai-website refs/heads/master: From https://github.com/w3c/wai-website
 * branch            master     -> FETCH_HEAD
Fetching submodule _external/resources/wai-accessibility-principles
From https://github.com/w3c/wai-accessibility-principles
   3ec7def..e0de0c9  master     -> origin/master
Fetching submodule _external/resources/wai-intro-wcag
From https://github.com/w3c/wai-intro-wcag
   9b60a55..88b9d36  master     -> origin/master
Fetching submodule _external/resources/wai-perspective-videos
From https://github.com/w3c/wai-perspective-videos
 * [new branch]      add-Videos-br     -> origin/add-Videos-br
 * [new branch]      add-changelog     -> origin/add-changelog
   1553bd5..cdeded1  master            -> origin/master
 * [new branch]      slhenry-in-parent -> origin/slhenry-in-parent
 * [new branch]      slhenry-testing   -> origin/slhenry-testing
   dc242f0..3a3b656  test-pr-preview   -> origin/test-pr-preview
Could not access submodule '_external/data'
Errors during submodule fetch:
        _external/resources/wai-perspective-videos

Seems like there are some incorrect or inaccessible submodules configured in the main repo?

1 Like
5

Thanks @perry. I agree it looks like git fetch errors. This is strange as

  1. it never fails using uncached
  2. it nearly always fails with cache
  3. we have a git hub action that does similar and never fails. Might be different git version.
  4. the repository flaged as errors are all good an working normally.

So it seems to be cache related. Or cache plus specific git submodules use. We don’t use -f. Is that the same git command for uncached?

BTW we actually want --remote to update all the submodule refs in our current usage so end up with 2 gitmodules iterations. Ideally we could modify your initial command.

I’ll see if I can ever reproduce the error with an uncached build by experimenting with various. Options. I’ll also check those specific refs.

Is it possible github are throwing spurious errors?

Is it a problem with the way submodules are cached followed by an init. Perhaps if the submodules have had checking?

On a wild guess, I wonder if submodule --sync is relevant. I never full grokked what it does.

6

You cannot modify the command we use; if you want to use our CI it must work as specified in my post. (I understand you wish it worked differently - and there is an open feature request for recursive submodule updates which I think would help you, so I have added your voice to it so we can report back here if the behavior changes).

Based on past experience, I do not believe this is a spurious error; it’s one you have to fix in your codebase. We do run different commands when using a non-cached repo, (git clone instead of the fetch/checkout pair), so you should be able to reproduce in your own freshly-cloned copy of the repo locally (so it has no submodules cloned before you start, or other local changes), when you run those commands.

1 Like
7

except it is intermittent when using cached builds. Currently it is not failing again.

8

@fool on further reflection, I was assuming the intermittent nature was due to some system performance thing like speed during busy times. However i could also be changes to any of our repos.

Anyway, if you think we would get visibility into the problem with a local build, could you not surface that information in your build logs? Just saying ‘error’ is not much help after all. :slight_smile:

Thanks

1 Like