Add MCP server for interacting with meshtastic devices and testing framework / TUI (#10194)

* Start of MCP server and test suite * Add MCP server for interacting with meshtastic devices and testing framework / TUI * Update mcp-server/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix mcp-server review feedback from thread Agent-Logs-Url: https://github.com/meshtastic/firmware/sessions/91dc128a-ed50-4d07-8bb2-3dc6623a05f7 Co-authored-by: thebentern <9000580+thebentern@users.noreply.github.com> * Enhance StreamAPI and PhoneAPI for improved log record handling and concurrency control * Semgrep fixes * Trunk and semgrep fixes * optimize pio streaming tee file writes Agent-Logs-Url: https://github.com/meshtastic/firmware/sessions/04e26c6b-6a2b-45be-bbeb-79ae4d0be633 Co-authored-by: thebentern <9000580+thebentern@users.noreply.github.com> * chore: remove redundant log handle assignment Agent-Logs-Url: https://github.com/meshtastic/firmware/sessions/04e26c6b-6a2b-45be-bbeb-79ae4d0be633 Co-authored-by: thebentern <9000580+thebentern@users.noreply.github.com> * Consolidate type imports and remove placeholder test files * Add tests for config persistence and more exchange messages * Refactor position test to validate on-demand request/reply behavior * Remove position request/reply test and update README for telemetry behavior * Fix transmit history file to get removed on factory reset --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
2026-04-18 11:29:02 -05:00
parent 8fd0a7f283
commit 6b15571e14
77 changed files with 10701 additions and 13 deletions
@@ -0,0 +1,65 @@
+---
+description: Re-run a specific test N times in isolation to triage flakes, diff firmware logs between passes and failures
+argument-hint: <test-node-id> [count=5]
+---
+
+# `/repro` — flakiness triage for one test
+
+Re-run a single pytest node ID N times in isolation, track pass rate, and surface what's _different_ in the firmware logs between the passing attempts and the failing ones. Turns "it's flaky, I guess" into "it fails when X, passes when Y."
+
+## What to do
+
+1. **Parse `$ARGUMENTS`**: first token is the pytest node id (e.g. `tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[nrf52->esp32s3]`); second token is an integer count (default `5`, cap at `20`). If the first token doesn't look like a test path (no `::` and no `tests/` prefix), treat the whole `$ARGUMENTS` as a `-k` filter instead.
+
+2. **Sanity-check the hub first** (so we're not measuring "nothing plugged in" N times): call `mcp__meshtastic__list_devices`. If the test name contains `nrf52` or `esp32s3` and the matching VID isn't present, stop and report — re-running won't help.
+
+3. **Loop N times**. For each iteration:
+
+   ```bash
+   ./mcp-server/run-tests.sh <test-id> --tb=short -p no:cacheprovider
+   ```
+
+   Capture: exit code, duration, and (on failure) the `Meshtastic debug` firmware log section from `mcp-server/tests/report.html`. `-p no:cacheprovider` suppresses pytest's `.pytest_cache` writes so iterations don't influence each other.
+
+4. **Track a small structured tally**:
+
+   ```text
+   attempt 1: PASS (42s)
+   attempt 2: FAIL (128s)  ← firmware log 200-line tail captured
+   attempt 3: PASS (39s)
+   attempt 4: FAIL (121s)
+   attempt 5: PASS (41s)
+   --------------------------------------
+   pass rate: 3/5 (60%)   |   mean duration: 74s
+   ```
+
+5. **On mixed outcomes**: diff the firmware log tails between a representative passing attempt and a representative failing attempt. Focus on:
+   - Error-level lines only present in failures (`PKI_UNKNOWN_PUBKEY`, `Alloc an err=`, `Skip send`, `No suitable channel`)
+   - Timing around the assertion event — did a broadcast go out, was there an ACK, did NAK fire?
+   - Device state fields that changed (nodesByNum entries, region/preset, channel_num)
+
+   Surface the top 3 differences as a "passes when / fails when" table. Don't dump full logs — pull specific lines with uptime timestamps.
+
+6. **Classify the flake** into one of:
+   - **LoRa airtime collision** → pass rate improves with fewer concurrent transmitters; propose a `time.sleep` gap or retry bump in the test body.
+   - **PKI key staleness** → fails on first attempt, passes after self-heal; existing retry loop in `test_direct_with_ack.py` handles this.
+   - **NodeInfo cooldown** → `Skip send NodeInfo since we sent it <600s ago` in fail-only logs; needs `broadcast_nodeinfo_ping()` warmup.
+   - **Hardware-specific** (one direction fails, other passes; one device's firmware is older; driver wedged) → specific recovery pointer.
+   - **Genuinely unknown** → say so; don't invent a root cause.
+
+7. **Report back** with:
+   - Pass rate and mean duration.
+   - Classification + evidence (the specific log lines that support it).
+   - A suggested next step (re-run with specific args, open `/diagnose`, edit a specific test file, nothing).
+
+## Examples
+
+- `/repro tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[esp32s3->nrf52] 10` — runs 10 times, diffs firmware logs.
+- `/repro broadcast_delivers` — no `::`, no `tests/`, so interpreted as `-k broadcast_delivers`; runs every matching test the default 5 times.
+- `/repro tests/telemetry/test_device_telemetry_broadcast.py 3` — shorter run for a slow test.
+
+## Constraints
+
+- Don't exceed `count=20` per invocation — airtime and USB wear add up. If the user asks for 50, negotiate down.
+- Don't rebuild firmware as part of triage; flakes that only reproduce under different firmware belong in a separate session.
+- If the FIRST attempt fails AND the rest all pass, that's a classic "state leak from a prior test" → say so and suggest running with `--force-bake` or starting from a clean state rather than chasing the first failure.