[FEATURE] Retry for CI Builds

[lupyuen]

Is your feature request related to a problem? Please describe.

We did a Quantitative Analysis of our Most Expensive PR (in terms of GitHub Runner Usage). We discovered that our CI Jobs are failing too often due to the Failed Downloading of Build Dependencies...

• Apache NuttX RTOS: Analysing the Most Expensive PR
• Spreadsheet for Most Expensive PR

Which causes Developer Frustration and Wasted GitHub Runners. We propose to implement Retry for CI Builds, so that the Build Dependencies will be successfully downloaded when we retry a build. This means that a NuttX Target like pinephone:nsh will be rebuilt when it fails.

Describe the solution you'd like

@simbit18 Would you be able to implement this solution? It's not so urgent, but we need more time to test it thoroughly. Please lemme know thanks :-)

• Explained here:
  Apache NuttX RTOS: Retrying a CI Build
  Apache NuttX RTOS: Testing of Retry for CI Builds
• We retry dotest 5 times in case of failure. We could implement a new wrapper function retrytest that will retry dotest 5 times. (dotest builds a Single NuttX Target like pinephone:nsh)
• For Simplicity: We will retry on Any Failure: Download Failed, Compile Error, ...
• Each Retry will have a Random Exponential Backoff. Retry 1: Wait randomly 0 to 60 seconds. Retry 2: Wait randomly 0 to 120 seconds. Retry 3: Wait randomly 0 to 240 seconds. Etc
• We will test at NuttX Mirror Repo, though I think the Download Failures don't happen so often at that repo
• We will also test at NuttX Build Farm, which has Download Failures much more often

Retry Logic looks like this, it refers to a Single NuttX Target like pinephone:nsh:

dotest is called here, which we can rename as retrytest:

nuttx/tools/testbuild.sh

Lines 582 to 597 in 76fe46a

	# Perform the build test for each entry in the test list file
	for line in $testlist; do
	firstch=${line:0:1}
	if [ "X$firstch" == "X/" ]; then
	dir=`echo $line | cut -d',' -f1`
	list=`find boards$dir -name defconfig | cut -d'/' -f4,6`
	for i in ${list}; do
	dotest $i${line/"$dir"/}
	done
	else
	dotest $line
	fi
	done

Describe alternatives you've considered

We considered the alternatives explained in this video:

• Apache NuttX RTOS: Retrying a CI Build

Verification

• I have verified before submitting the report.

[simbit18]

@lupyuen I’ve been looking for a solution to the problem of failed package downloads for a while now.

Here are some of my thoughts, the result of wracking my remaining brain cells :)

Simple solution

We use curl as our download tool, and it already offers a --retry option.
https://everything.curl.dev/usingcurl/downloads/retry.html

So we could use this method to retry X times.

Most comprehensive solution

A solution similar to the one used with espressif (esp-hal-3rdparty with git), but here with curl

add to the file https://github.com/apache/nuttx/blob/master/tools/Config.mk

# CURL - Download file. cURL command to download a file in the
#        folder on your local machine.
#        The fourth argument is an storage path. The third argument is used
#        if it is not provided or is empty.
# Example: $(call CURL,$(URL_BASE),$(FOO_ARCHIVE),foo.out,foo_store.out)

define CURL
	$(ECHO_BEGIN)"Downloading: $2"
	$(Q) $(if $4, \
       $(call CURLSTORE,$1/$2,$4/$2,$(if $(3),$(3)/$(2),$(2))), \
       curl -L $(if $(V),,-Ss) $(1)/$(2) -o $(if $(3),$(3)/$(2),$(2)))
	$(ECHO_END)
endef

# CURLSTORE - Download file is an storage path.
#             Internal a CURL

define CURLSTORE
	$(ECHO_BEGIN)"Search in the storage path -> $2"
	$(Q) $(if $(wildcard $2),, \
       curl -L $(if $(V),,-Ss) $(1) -o $(2))
	$(Q) cp -f $2 $3
	$(ECHO_END)
endef

When the same packages are downloaded repeatedly in a job, you can use the nxtmpdir folder
to avoid performing a second download.

Package storage is enabled as follows -S on [nuttx/.github/workflows/build.yml]

             ./cibuild.sh -c -A -N -R -S testlist/${{matrix.boards}}.dat 

nuttx/tools/configure.sh

Line 35 in 201406b

-S adds the nxtmpdir folder for third-party packages.

You can use this system + a simple solution to try to reduce the likelihood of errors

Of course, here it is necessary to standardize all the logic for downloading packages.
(which is the right thing to do for proper maintenance)

I think the second is more comprehensive, but, we need the community to help!!!

We could launch an “adopt a .zip or .tar” campaign!!! :)

Your approach could also be another solution, but it has a different impact

on the time required for changes and testing.

Of course, if it’s a GitHub issue, no solution will fix the problem. :)

[lupyuen]

Thanks @simbit18! Yeah we might have a problem getting NuttX Community to help with the curl fixes, which are highly intrusive and script-specific. And ultimately you and I will have to standby and make sure that every single fix works correctly :-)

Do you think we should go ahead to do the Build Retry our own (less intrusive) way, so we won't depend too much on NuttX Community? I'm just worried that the next Peak Period might come again soon, and we'll get into serious trouble with ASF Infra Team, if we use too many NuttX Runners :-(

[lupyuen]

Sorry: Can curl do Random Exponential Backoff? I suspect a simple retry might not be sufficient, GitHub might still block our downloads, due to suspected spamming :-(

[simbit18]

Your solution is a good one, but it takes a lot of time, which I don't have right now.

[lupyuen]

Yeah I fully understand :-) Let's reopen this feature later when we're ready. It's really difficult to push any changes to NuttX CI because...

1. Only You and I understand how NuttX CI works. Very hard to get anyone else to help us.
2. NuttX CI is Mission Critical. We can't afford to break it at any time. (And yes, you and I will surely get blamed)
3. NuttX CI is extremely complex, with lots of legacy quirks. Too much possibility of breaking things, especially when we introduce intrusive changes.

The next time NuttX runs out of GitHub Runners: Maybe you and I won't be around any more? Let's wait and see :-)

[simbit18]

@lupyuen Making changes is a very delicate matter.

Perhaps we need to consider a completely new system for CI, but that takes time.

Anyway, when is the next marathon? :)

[lupyuen]

Wonder if this will work for retrying CI Builds? I'll test on NuttX Build Farm, since it fails Dependency Download quite often...

lupyuen13/nuttx@master...lupyuen13:nuttx:retry-build

[simbit18]

@lupyuen That sounds like a great solution!
It should work.

[lupyuen]

Hmmm interesting: Our Retry Build reuses the Corrupted Zip File (from the previous download). So the Retry Build will always fail. Lemme try to clean up, delete all Zip Files, and discard any leftover files:

git clean -fd
git status

pushd ../apps
git clean -fd
git status
popd

Here's the earlier log: https://gitlab.com/lupyuen/nuttx-build-log/-/snippets/5971712#L1122

Build Attempt 1
====================================================================================
Configuration/Tool: raspberrypi-4b/lvgl
  Building NuttX...
[v9.2.1.zip]
  End-of-central-directory signature not found.
  ...
Wait 55 seconds (60 backoff)
Build Attempt 2
====================================================================================
Configuration/Tool: raspberrypi-4b/lvgl
Ignored files:
  (use "git add -f <file>..." to include in what will be committed)
	graphics/lvgl/v9.2.1.zip
  ...
  Building NuttX...
[v9.2.1.zip]
  End-of-central-directory signature not found.