kevin/meshtastic_firmware
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

ci: warn and skip the pio check job on transient network failures (#11145)

Browse Source 
* ci: warn and skip the pio check job on transient network failures

The `check` matrix job intermittently fails before it analyses anything:
`pio check` has to fetch the platform package first, and that download gets
reset mid-flight by the CDN often enough to be a nuisance. Three of the last
~200 CI runs died this way, all identical and all ~1s into the run:

    Checking station-g3 > cppcheck (board: station-g3; platform:
    https://github.com/pioarduino/platform-espressif32/.../platform-espressif32.zip)
    requests.exceptions.ConnectionError: ('Connection aborted.',
        ConnectionResetError(104, 'Connection reset by peer'))

cppcheck never ran, so there is no signal at all - just a red X someone has to
re-run by hand.

`pio check` exits 1 for both a dead socket and a real defect, so only the output
can tell them apart. check-all.sh now tees the run and classifies the failure:

  * cppcheck reported something (summary table or a `file:line: [severity]`
    defect line) -> real, propagate the exit status. This is checked first, so
    no amount of network noise elsewhere in the log can mask a genuine defect.
  * transport-level error and nothing else -> log a ::warning:: annotation
    naming the board and the underlying error, and exit 0 under CI (exit 2
    locally, so a local run never quietly reports itself clean).
  * anything else -> propagate the exit status.

404 / 403 / UnknownPackageError are deliberately not treated as transient: a
missing or forbidden package is a reproducible break from a bad pin in a .ini,
not weather.

No workflow change is needed - gh-action-firmware's entrypoint runs
/workspace/bin/check-all.sh for MT_TARGET=check, so the whole fix lives in this
repo and applies to local runs too.

Verified by replaying the captured CI logs through the script with a stubbed
`pio`: the station-g3 connection-reset log skips with a warning under CI and
exits 2 locally, the rak11200 `Total 1 0 0` defect log still fails, that same
defect log with the reset traceback appended still fails, and a 404 still fails.

* ci: pass board names to pio as an argument array

Addresses CodeRabbit review feedback on the previous commit (and the SC2086 the
line has carried since it was written): BOARDS was a space-joined string and
$CHECK was expanded unquoted, so a board override containing whitespace or a
glob character could turn into extra pio arguments.

BOARDS and CHECK are arrays now, and pio is invoked with "${CHECK[@]}", so board
names reach it as literal arguments. The two skip messages use "${BOARDS[*]}" -
a bare "${BOARDS}" would have quietly named only the first board.

shellcheck -x is clean on the file. Re-verified with a stubbed pio that records
its argv: 'tb*' and 'tbeam extra' each arrive as one literal argument, the
default list still expands to 15 -e pairs, a two-board skip names both boards,
and all six classification cases (network/CI, network/local, real defects,
defects with a reset traceback appended, 404, clean pass) are unchanged.
This commit is contained in:
Ben Meadors
2026-07-23 06:25:18 -05:00
committed by GitHub
co-authored by GitHub
parent ef1aedd569
commit d9150e8dc7
1 changed files with 66 additions and 10 deletions
Download Patch File Download Diff File Expand all files Collapse all files
bin/check-all.sh
+66 -10
View File
@@ -1,26 +1,82 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Note: This is a prototype for how we could add static code analysis to the CI. # Run cppcheck static analysis over one or more PlatformIO environments.
#
# Why this is more than a bare `pio check`: the CI `check` matrix job in main_matrix.yml runs this
# script inside meshtastic/gh-action-firmware, and `pio check` must download the platform package
# before it can analyse anything. That download is regularly reset mid-flight by the CDN, e.g.
#
# Checking station-g3 > cppcheck (board: station-g3; platform: .../platform-espressif32.zip)
# requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, ...))
#
# The analysis never starts, so there is no signal at all - just a red X someone has to re-run by
# hand. `pio check` exits 1 for that AND for real defects, so the exit code cannot tell them apart;
# only the output can. This script classifies the failure:
#
# * cppcheck produced results (summary table or defect lines) -> real, propagate the exit status.
# * transport-level network error and nothing else -> warn and skip.
# * anything else -> propagate the exit status.
#
# A missing or forbidden package (404 / UnknownPackageError / 403) is deliberately NOT treated as
# transient: that is a reproducible break from a bad pin in a .ini, not weather.
#
# Exit codes: 0 = clean (or skipped under CI), 1 = defects/error, 2 = skipped locally.
set -e set -uo pipefail
VERSION=$(bin/buildinfo.py long) VERSION=$(bin/buildinfo.py long) || exit 1
# The shell vars the build tool expects to find # The shell vars the build tool expects to find
export APP_VERSION=$VERSION export APP_VERSION=$VERSION
if [[ $# -gt 0 ]]; then if [[ $# -gt 0 ]]; then
# can override which environment by passing arg # can override which environment by passing arg
BOARDS="$@" BOARDS=("$@")
else else
BOARDS="tlora-v2 tlora-v1 tlora_v1_3 tlora-v2-1-1.6 tbeam heltec-v2.0 heltec-v2.1 tbeam0.7 meshtastic-diy-v1 rak4631 rak4631_eink rak11200 t-echo canaryone pca10059_diy_eink" BOARDS=(tlora-v2 tlora-v1 tlora_v1_3 tlora-v2-1-1.6 tbeam heltec-v2.0 heltec-v2.1 tbeam0.7 meshtastic-diy-v1 rak4631 rak4631_eink rak11200 t-echo canaryone pca10059_diy_eink)
fi fi
echo "BOARDS:${BOARDS}" echo "BOARDS:${BOARDS[*]}"
CHECK="" # Array, not a string: board names reach pio as literal arguments, so an override containing
for BOARD in $BOARDS; do # whitespace or a glob character cannot turn into extra pio arguments.
CHECK="${CHECK} -e ${BOARD}" CHECK=()
for BOARD in "${BOARDS[@]}"; do
CHECK+=(-e "${BOARD}")
done done
pio check --flags "-DAPP_VERSION=${APP_VERSION} --suppressions-list=suppressions.txt --inline-suppr" $CHECK --skip-packages --pattern="src/" --fail-on-defect=low --fail-on-defect=medium --fail-on-defect=high LOG=$(mktemp)
trap 'rm -f "$LOG"' EXIT
# Keep streaming to the console so the CI log reads exactly as it did before; tee a copy for the
# post-mortem classification below.
pio check --flags "-DAPP_VERSION=${APP_VERSION} --suppressions-list=suppressions.txt --inline-suppr" "${CHECK[@]}" --skip-packages --pattern="src/" --fail-on-defect=low --fail-on-defect=medium --fail-on-defect=high 2>&1 | tee "$LOG"
STATUS=${PIPESTATUS[0]}
if [[ $STATUS -eq 0 ]]; then
exit 0
fi
# Fail closed: if cppcheck got far enough to report anything, the run is real no matter what other
# noise is in the log.
if grep -Eqi '^Component[[:space:]]+.*HIGH|^Total[[:space:]]+[0-9]|^[^[:space:]]+:[0-9]+: \[(high|medium|low)' "$LOG"; then
exit "$STATUS"
fi
# Transport-level failures only. No 403/404/UnknownPackageError here, on purpose.
TRANSIENT='requests\.exceptions\.(ConnectionError|Timeout|ReadTimeout|ConnectTimeout)|ConnectionResetError|urllib3\.exceptions\.(ProtocolError|MaxRetryError|NameResolutionError|ReadTimeoutError)|Read timed out|Connection aborted|Connection refused|Temporary failure in name resolution|Could not resolve host|(429|500|502|503|504) (Server|Client) Error|Too Many Requests'
if ! grep -Eq "$TRANSIENT" "$LOG"; then
exit "$STATUS"
fi
REASON=$(grep -Em1 "$TRANSIENT" "$LOG" | tr -d '\r' | cut -c1-200)
echo "RESULT: SKIPPED ${BOARDS[*]} - transient network failure: ${REASON}"
if [[ ${GITHUB_ACTIONS:-false} == "true" ]]; then
echo "::warning title=Static analysis skipped (${BOARDS[*]})::pio check could not download its packages: ${REASON}"
exit 0
fi
exit 2