I’m having an issue at the beginning of the build process, during the stage ‘preparing repo’, where the connection to our self-managed GitLab instance times out sometimes.
I insist on the “sometimes” because some builds do pass, but I cannot tell what’s different, so that sometimes it works and sometimes it doesn’t.
Netlify site name:
Log of a build where the issue happens:
1:09:46 PM: build-image version: a2d22d22e4555d1ef0a972ed14a0a4b366ad20c4 (focal) 1:09:46 PM: build-image tag: v4.16.3 1:09:46 PM: buildbot version: 9a6b4d0d37eb2a90e2c482e1d6cfe9a0793e6262 1:09:46 PM: Fetching cached dependencies 1:09:46 PM: Starting to download cache of 115.6MB 1:09:47 PM: Finished downloading cache in 1.063823507s 1:09:47 PM: Starting to extract cache 1:09:49 PM: Finished extracting cache in 1.674892752s 1:09:49 PM: Finished fetching cache in 2.794568427s 1:09:49 PM: Starting to prepare the repo for build 1:12:00 PM: User git error while checking for ref refs/heads/feat/label-range-search 1:12:00 PM: Failing build: Failed to prepare repo 1:12:00 PM: Failed during stage 'preparing repo': error checking for ref: : ssh: connect to host gitlab.liip.ch port 22: Connection timed out fatal: Could not read from remote repository. Please make sure you have the correct access rights and the repository exists. : exit status 128 1:12:00 PM: Finished processing build request in 2m13.939729416s
What we already tried
Our first guess was that the issue comes from our self-managed GitLab instance. We asked our service provider to look into it, but they didn’t find any failed attempts in the logs.
Here’s their analysis:
We investigated the issue and it seems that connections to the gitlab server weren’t established.
IP tables should not block the connections via SSH:
1287K 80M ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 22 /* 010 ssh accept tcp/v4 */
The connection attempt wasn’t logged on the gitlab server - at least on the 04.01 were you reported the ticket, assuming that you attempted to connection on that day. Failed attempts should be visible in the gitlab-shell logs or sshd logs, which also didn’t show any results.
This can point out that the connection from Netlify to the gitlab server was not established properly, but only judging the issue without any hints like IP address or date/time from attempts.
As you mentioned it is possible that Netlify didn’t accept the Gitlab host key, but it’s hard to tell without logs that provides such information.
We also checked the topic Frequently encountered problems during builds.
Enabling the debug mode didn’t provide additional information during the ‘preparing repo’ stage. Only when it goes beyond this step, we can see that there are more logs.
And as the build does sometimes pass, I understand that the permission to access the repository is set correctly. This rules out the most common root cause for the error 128.
Please let me know if there’s any additional information that might be useful to figure this out.
Thank you in advance for your help.