Builds failing: same image with different python versions

Support
1

site: sharp-shooter-malcolm-60451 ( https://allcells.willbe.blue)

This problem happens occasionally and sometimes it can take weeks to resume building successfully.

The build started to fail at the 21st of March and kept failing until the 24th. At that time I triggered a manual deploy(with clear cache) and the build was successfully until it started failing again in the 28th of March and kept failing until today.

I haven’t change a thing for some time now. Last time this happened it ‘fixed’ itself after a couple of days. The reason for failing is obvious: the python version is not the correct one. It is not obvious is why the same build image has two different python versions. Here’s a list of links of the relevant builds:

20th of March (Published)

4:00:26 AM: Build ready to start
4:00:28 AM: build-image version: 2eaac833881accc211f920f6c5a9f4121fdb15ab (focal)
4:00:28 AM: buildbot version: 2eaac833881accc211f920f6c5a9f4121fdb15ab
4:00:28 AM: Fetching cached dependencies
...
4:00:32 AM: Python version set to 2.7

21st of March (Failed)

4:00:25 AM: Build ready to start
4:00:27 AM: build-image version: 8d7bce6c4bf50ad5a72d21731c81d47e9b46a4a1 (focal)
4:00:27 AM: buildbot version: 8d7bce6c4bf50ad5a72d21731c81d47e9b46a4a1
4:00:27 AM: Fetching cached dependencies
...
4:00:33 AM: Python version set to 3.8

24th of March (Published) - Manually triggered with clean cache

6:39:13 PM: Build ready to start
6:39:15 PM: build-image version: 099264affa13d33ffbaea4ccf963ddbead423c30 (focal)
6:39:15 PM: buildbot version: 099264affa13d33ffbaea4ccf963ddbead423c30
6:39:15 PM: Building without cache
...
6:39:19 PM: Python version set to 2.7

28th of March (Failed)

4:00:25 AM: Build ready to start
4:00:31 AM: build-image version: 5e2052dbf11f1636a7bac1839aff8991d186d620 (focal)
4:00:31 AM: buildbot version: 5e2052dbf11f1636a7bac1839aff8991d186d620
4:00:31 AM: Fetching cached dependencies
...
4:00:43 AM: Python version set to 3.8

Buiild Settings:
image

Before someones suggest to change the python scripts to 3 realize that, the next time the Build image has python 2.7 it will fail again. So this is not a solution.

3

Hey @Rivotti,

This has been raised with the devs and we would follow up as we have more info.

4

@Rivotti we upgraded the Python version in our build system to 3.8 but you can use 2.7 by creating a runtime.txt and specifiying the version ie 2.7. Then clear cache and build. Let me know if this helps.

5

@Japheth_Obala yes, this fixed the problem, thank you.

6

Thanks for confirming!