meshtastic-firmware/.claude/commands/repro.md

description, argument-hint

description	argument-hint
Re-run a specific test N times in isolation to triage flakes, diff firmware logs between passes and failures	<test-node-id> [count=5]

/repro — flakiness triage for one test

Re-run a single pytest node ID N times in isolation, track pass rate, and surface what's different in the firmware logs between the passing attempts and the failing ones. Turns "it's flaky, I guess" into "it fails when X, passes when Y."

What to do

1. Parse $ARGUMENTS: first token is the pytest node id (e.g. tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[nrf52->esp32s3]); second token is an integer count (default 5, cap at 20). If the first token doesn't look like a test path (no :: and no tests/ prefix), treat the whole $ARGUMENTS as a -k filter instead.

2. Sanity-check the hub first (so we're not measuring "nothing plugged in" N times): call mcp__meshtastic__list_devices. If the test name contains nrf52 or esp32s3 and the matching VID isn't present, stop and report — re-running won't help.

3. Loop N times. For each iteration:

   ./mcp-server/run-tests.sh <test-id> --tb=short -p no:cacheprovider
   

   Capture: exit code, duration, and (on failure) the Meshtastic debug firmware log section from mcp-server/tests/report.html. -p no:cacheprovider suppresses pytest's .pytest_cache writes so iterations don't influence each other.

4. Track a small structured tally:

   attempt 1: PASS (42s)
   attempt 2: FAIL (128s)  ← firmware log 200-line tail captured
   attempt 3: PASS (39s)
   attempt 4: FAIL (121s)
   attempt 5: PASS (41s)
   --------------------------------------
   pass rate: 3/5 (60%)   |   mean duration: 74s
   

5. On mixed outcomes: diff the firmware log tails between a representative passing attempt and a representative failing attempt. Focus on:

   • Error-level lines only present in failures (PKI_UNKNOWN_PUBKEY, Alloc an err=, Skip send, No suitable channel)
   • Timing around the assertion event — did a broadcast go out, was there an ACK, did NAK fire?
   • Device state fields that changed (nodesByNum entries, region/preset, channel_num)

   Surface the top 3 differences as a "passes when / fails when" table. Don't dump full logs — pull specific lines with uptime timestamps.

6. Classify the flake into one of:

   • LoRa airtime collision → pass rate improves with fewer concurrent transmitters; propose a time.sleep gap or retry bump in the test body.
   • PKI key staleness → fails on first attempt, passes after self-heal; existing retry loop in test_direct_with_ack.py handles this.
   • NodeInfo cooldown → Skip send NodeInfo since we sent it <600s ago in fail-only logs; needs broadcast_nodeinfo_ping() warmup.
   • Hardware-specific (one direction fails, other passes; one device's firmware is older; driver wedged) → specific recovery pointer. For a device that's wedged past touch_1200bps, the next escalation is uhubctl_cycle(role=..., confirm=True) to hard-power-cycle its hub port (requires uhubctl installed).
   • Device went dark mid-run → fails from some attempt onward, never recovers, firmware log stops arriving. Almost always hardware: a Guru crash + frozen CDC. Hard-power-cycle via uhubctl_cycle(role=..., confirm=True) before the next iteration; if that also fails, escalate to replug.
   • Genuinely unknown → say so; don't invent a root cause.

7. Report back with:

   • Pass rate and mean duration.
   • Classification + evidence (the specific log lines that support it).
   • A suggested next step (re-run with specific args, open /diagnose, edit a specific test file, nothing).

Examples

• /repro tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[esp32s3->nrf52] 10 — runs 10 times, diffs firmware logs.
• /repro broadcast_delivers — no ::, no tests/, so interpreted as -k broadcast_delivers; runs every matching test the default 5 times.
• /repro tests/telemetry/test_device_telemetry_broadcast.py 3 — shorter run for a slow test.

Constraints

• Don't exceed count=20 per invocation — airtime and USB wear add up. If the user asks for 50, negotiate down.
• Don't rebuild firmware as part of triage; flakes that only reproduce under different firmware belong in a separate session.
• If the FIRST attempt fails AND the rest all pass, that's a classic "state leak from a prior test" → say so and suggest running with --force-bake or starting from a clean state rather than chasing the first failure.