From d9150e8dc76d87d57e375ac998c83b9cba22073e Mon Sep 17 00:00:00 2001 From: Ben Meadors Date: Thu, 23 Jul 2026 06:25:18 -0500 Subject: [PATCH] ci: warn and skip the pio check job on transient network failures (#11145) * ci: warn and skip the pio check job on transient network failures The `check` matrix job intermittently fails before it analyses anything: `pio check` has to fetch the platform package first, and that download gets reset mid-flight by the CDN often enough to be a nuisance. Three of the last ~200 CI runs died this way, all identical and all ~1s into the run: Checking station-g3 > cppcheck (board: station-g3; platform: https://github.com/pioarduino/platform-espressif32/.../platform-espressif32.zip) requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) cppcheck never ran, so there is no signal at all - just a red X someone has to re-run by hand. `pio check` exits 1 for both a dead socket and a real defect, so only the output can tell them apart. check-all.sh now tees the run and classifies the failure: * cppcheck reported something (summary table or a `file:line: [severity]` defect line) -> real, propagate the exit status. This is checked first, so no amount of network noise elsewhere in the log can mask a genuine defect. * transport-level error and nothing else -> log a ::warning:: annotation naming the board and the underlying error, and exit 0 under CI (exit 2 locally, so a local run never quietly reports itself clean). * anything else -> propagate the exit status. 404 / 403 / UnknownPackageError are deliberately not treated as transient: a missing or forbidden package is a reproducible break from a bad pin in a .ini, not weather. No workflow change is needed - gh-action-firmware's entrypoint runs /workspace/bin/check-all.sh for MT_TARGET=check, so the whole fix lives in this repo and applies to local runs too. Verified by replaying the captured CI logs through the script with a stubbed `pio`: the station-g3 connection-reset log skips with a warning under CI and exits 2 locally, the rak11200 `Total 1 0 0` defect log still fails, that same defect log with the reset traceback appended still fails, and a 404 still fails. * ci: pass board names to pio as an argument array Addresses CodeRabbit review feedback on the previous commit (and the SC2086 the line has carried since it was written): BOARDS was a space-joined string and $CHECK was expanded unquoted, so a board override containing whitespace or a glob character could turn into extra pio arguments. BOARDS and CHECK are arrays now, and pio is invoked with "${CHECK[@]}", so board names reach it as literal arguments. The two skip messages use "${BOARDS[*]}" - a bare "${BOARDS}" would have quietly named only the first board. shellcheck -x is clean on the file. Re-verified with a stubbed pio that records its argv: 'tb*' and 'tbeam extra' each arrive as one literal argument, the default list still expands to 15 -e pairs, a two-board skip names both boards, and all six classification cases (network/CI, network/local, real defects, defects with a reset traceback appended, 404, clean pass) are unchanged. --- bin/check-all.sh | 76 +++++++++++++++++++++++++++++++++++++++++------- 1 file changed, 66 insertions(+), 10 deletions(-) diff --git a/bin/check-all.sh b/bin/check-all.sh index 9c7fc694d..3186899aa 100755 --- a/bin/check-all.sh +++ b/bin/check-all.sh @@ -1,26 +1,82 @@ #!/usr/bin/env bash -# Note: This is a prototype for how we could add static code analysis to the CI. +# Run cppcheck static analysis over one or more PlatformIO environments. +# +# Why this is more than a bare `pio check`: the CI `check` matrix job in main_matrix.yml runs this +# script inside meshtastic/gh-action-firmware, and `pio check` must download the platform package +# before it can analyse anything. That download is regularly reset mid-flight by the CDN, e.g. +# +# Checking station-g3 > cppcheck (board: station-g3; platform: .../platform-espressif32.zip) +# requests.exceptions.ConnectionError: ('Connection aborted.', ConnectionResetError(104, ...)) +# +# The analysis never starts, so there is no signal at all - just a red X someone has to re-run by +# hand. `pio check` exits 1 for that AND for real defects, so the exit code cannot tell them apart; +# only the output can. This script classifies the failure: +# +# * cppcheck produced results (summary table or defect lines) -> real, propagate the exit status. +# * transport-level network error and nothing else -> warn and skip. +# * anything else -> propagate the exit status. +# +# A missing or forbidden package (404 / UnknownPackageError / 403) is deliberately NOT treated as +# transient: that is a reproducible break from a bad pin in a .ini, not weather. +# +# Exit codes: 0 = clean (or skipped under CI), 1 = defects/error, 2 = skipped locally. -set -e +set -uo pipefail -VERSION=$(bin/buildinfo.py long) +VERSION=$(bin/buildinfo.py long) || exit 1 # The shell vars the build tool expects to find export APP_VERSION=$VERSION if [[ $# -gt 0 ]]; then # can override which environment by passing arg - BOARDS="$@" + BOARDS=("$@") else - BOARDS="tlora-v2 tlora-v1 tlora_v1_3 tlora-v2-1-1.6 tbeam heltec-v2.0 heltec-v2.1 tbeam0.7 meshtastic-diy-v1 rak4631 rak4631_eink rak11200 t-echo canaryone pca10059_diy_eink" + BOARDS=(tlora-v2 tlora-v1 tlora_v1_3 tlora-v2-1-1.6 tbeam heltec-v2.0 heltec-v2.1 tbeam0.7 meshtastic-diy-v1 rak4631 rak4631_eink rak11200 t-echo canaryone pca10059_diy_eink) fi -echo "BOARDS:${BOARDS}" +echo "BOARDS:${BOARDS[*]}" -CHECK="" -for BOARD in $BOARDS; do - CHECK="${CHECK} -e ${BOARD}" +# Array, not a string: board names reach pio as literal arguments, so an override containing +# whitespace or a glob character cannot turn into extra pio arguments. +CHECK=() +for BOARD in "${BOARDS[@]}"; do + CHECK+=(-e "${BOARD}") done -pio check --flags "-DAPP_VERSION=${APP_VERSION} --suppressions-list=suppressions.txt --inline-suppr" $CHECK --skip-packages --pattern="src/" --fail-on-defect=low --fail-on-defect=medium --fail-on-defect=high +LOG=$(mktemp) +trap 'rm -f "$LOG"' EXIT + +# Keep streaming to the console so the CI log reads exactly as it did before; tee a copy for the +# post-mortem classification below. +pio check --flags "-DAPP_VERSION=${APP_VERSION} --suppressions-list=suppressions.txt --inline-suppr" "${CHECK[@]}" --skip-packages --pattern="src/" --fail-on-defect=low --fail-on-defect=medium --fail-on-defect=high 2>&1 | tee "$LOG" +STATUS=${PIPESTATUS[0]} + +if [[ $STATUS -eq 0 ]]; then + exit 0 +fi + +# Fail closed: if cppcheck got far enough to report anything, the run is real no matter what other +# noise is in the log. +if grep -Eqi '^Component[[:space:]]+.*HIGH|^Total[[:space:]]+[0-9]|^[^[:space:]]+:[0-9]+: \[(high|medium|low)' "$LOG"; then + exit "$STATUS" +fi + +# Transport-level failures only. No 403/404/UnknownPackageError here, on purpose. +TRANSIENT='requests\.exceptions\.(ConnectionError|Timeout|ReadTimeout|ConnectTimeout)|ConnectionResetError|urllib3\.exceptions\.(ProtocolError|MaxRetryError|NameResolutionError|ReadTimeoutError)|Read timed out|Connection aborted|Connection refused|Temporary failure in name resolution|Could not resolve host|(429|500|502|503|504) (Server|Client) Error|Too Many Requests' + +if ! grep -Eq "$TRANSIENT" "$LOG"; then + exit "$STATUS" +fi + +REASON=$(grep -Em1 "$TRANSIENT" "$LOG" | tr -d '\r' | cut -c1-200) + +echo "RESULT: SKIPPED ${BOARDS[*]} - transient network failure: ${REASON}" + +if [[ ${GITHUB_ACTIONS:-false} == "true" ]]; then + echo "::warning title=Static analysis skipped (${BOARDS[*]})::pio check could not download its packages: ${REASON}" + exit 0 +fi + +exit 2