Hi, @rsooo, @Markus, and @tjenkinson. It seems you all have some unanswered questions here.
The permissions that the build image uses to access the repo are immediately dropped by the build image after the initial repository clone or fetch. This means you will need to configure additional permissions to access the repo if you want to work with it after the initial clone or fetch.
Hypothetically, the instructions for accessing other repositories can be extended to access the initial repository after the build images drops the default permissions. Those instructions can be found here:
Using those instructions you should be able to work with the repo after the clone or fetch. If there are questions about configuring this access, please let us know.