emdashes begone (#10847)
This commit is contained in:
Tom
GitHub
@@ -15,11 +15,11 @@ The Claude Code commands and Copilot prompts cover the same three workflows but
|
||||
- **Claude Code** (`/test`) uses `$ARGUMENTS` for pass-through, has direct access to Bash + all MCP tools registered in the user's settings, and runs in the terminal context.
|
||||
- **Copilot** (`/mcp-test`) runs in VS Code's agent mode; it has terminal + MCP access too but typically asks the operator to confirm inputs interactively.
|
||||
|
||||
A contributor using either IDE gets equivalent assistance. Keep the two in sync when behavior changes — the diff of intent should be minimal.
|
||||
A contributor using either IDE gets equivalent assistance. Keep the two in sync when behavior changes - the diff of intent should be minimal.
|
||||
|
||||
## House rules
|
||||
|
||||
- **No destructive writes without explicit operator approval.** Skills that could reflash, factory-reset, or reboot a device must describe the action and stop — the operator authorizes.
|
||||
- **No destructive writes without explicit operator approval.** Skills that could reflash, factory-reset, or reboot a device must describe the action and stop - the operator authorizes.
|
||||
- **Interpret failures, don't just echo them.** The skill body should pull firmware log lines from `mcp-server/tests/report.html` (the `Meshtastic debug` section, attached by `tests/conftest.py::pytest_runtest_makereport`) and classify the failure.
|
||||
- **Keep MCP tool calls sequential per port.** SerialInterface holds an exclusive port lock; two parallel tool calls on the same port deadlock.
|
||||
- **Never speculate about root cause.** If the evidence doesn't support a classification, say "unknown" and list what you'd need to disambiguate.
|
||||
|
||||
@@ -3,7 +3,7 @@ description: Produce a device health report using the meshtastic MCP tools (devi
|
||||
argument-hint: [role=all|nrf52|esp32s3|<port>]
|
||||
---
|
||||
|
||||
# `/diagnose` — device health report
|
||||
# `/diagnose` - device health report
|
||||
|
||||
Call the meshtastic MCP tool bundle and format a structured health report for one or all detected devices. Zero guesswork for the operator.
|
||||
|
||||
@@ -18,13 +18,13 @@ Call the meshtastic MCP tool bundle and format a structured health report for on
|
||||
- A `/dev/cu.*` path → only that one port.
|
||||
- Anything else → treat as a substring match against the `port` string.
|
||||
|
||||
3. **For each selected device, in sequence (NOT parallel — SerialInterface holds an exclusive port lock):**
|
||||
- `mcp__meshtastic__device_info(port=<p>)` — captures `my_node_num`, `long_name`, `short_name`, `firmware_version`, `hw_model`, `region`, `num_nodes`, `primary_channel`.
|
||||
- `mcp__meshtastic__list_nodes(port=<p>)` — count of peers, which ones have `publicKey` set, SNR/RSSI distribution.
|
||||
- `mcp__meshtastic__get_config(section="lora", port=<p>)` — region, preset, channel_num, tx_power, hop_limit.
|
||||
3. **For each selected device, in sequence (NOT parallel - SerialInterface holds an exclusive port lock):**
|
||||
- `mcp__meshtastic__device_info(port=<p>)` - captures `my_node_num`, `long_name`, `short_name`, `firmware_version`, `hw_model`, `region`, `num_nodes`, `primary_channel`.
|
||||
- `mcp__meshtastic__list_nodes(port=<p>)` - count of peers, which ones have `publicKey` set, SNR/RSSI distribution.
|
||||
- `mcp__meshtastic__get_config(section="lora", port=<p>)` - region, preset, channel_num, tx_power, hop_limit.
|
||||
- Optionally, if the device seems unhappy (fails to connect, `num_nodes==1` when ≥2 are plugged in, missing firmware*version), open a short firmware log window: `mcp__meshtastic__serial_open(port=<p>, env=<inferred-env>)`, wait 3s, `serial_read(session_id=<s>, max_lines=100)`, `serial_close(session_id=<s>)`. The env should be inferred from the VID map in `mcp-server/run-tests.sh` (nrf52 → rak4631, esp32s3 → heltec-v3) unless `MESHTASTIC_MCP_ENV*<ROLE>` is set.
|
||||
|
||||
4. **Hub health** (call once, not per-device): `mcp__meshtastic__uhubctl_list()` — enumerates every USB hub the host can see. Note which hubs advertise `ppps=true` and which hub hosts each Meshtastic device (cross-reference by VID). Flag it in the report if:
|
||||
4. **Hub health** (call once, not per-device): `mcp__meshtastic__uhubctl_list()` - enumerates every USB hub the host can see. Note which hubs advertise `ppps=true` and which hub hosts each Meshtastic device (cross-reference by VID). Flag it in the report if:
|
||||
- No hub advertises PPPS → `tests/recovery/` can't run on this setup; hard-recovery via `uhubctl_cycle` isn't available.
|
||||
- A Meshtastic device is on a non-PPPS hub → note it; operator may want to move the device to a PPPS hub to unlock auto-recovery.
|
||||
- `uhubctl_list` raises `ConfigError: uhubctl not found` → just say `uhubctl not installed` in the report; don't treat as a fault.
|
||||
@@ -45,17 +45,17 @@ Call the meshtastic MCP tool bundle and format a structured health report for on
|
||||
Keep it scannable. If a field is missing or abnormal (no pubkey for a known peer, region=UNSET, num_nodes inconsistent with the hub, device on non-PPPS hub), flag it inline with a short `⚠︎ <one-line reason>`.
|
||||
|
||||
6. **Cross-device correlation** (only when >1 device is inspected):
|
||||
- Do both sides see each other in `nodesByNum`? If one does and the other doesn't, that's asymmetric NodeInfo — flag it.
|
||||
- Do both sides see each other in `nodesByNum`? If one does and the other doesn't, that's asymmetric NodeInfo - flag it.
|
||||
- Do the LoRa configs match? (region, channel_num, modem_preset should all agree; mismatch = no mesh)
|
||||
- Do the primary channel NAMES match? Mismatch = different PSK = no decode.
|
||||
|
||||
7. **Recorder slice (cheap, always available).** The mcp-server runs an autouse log recorder that's been collecting from every connected device. Pull two short slices to surface anything weird that's already happened:
|
||||
- `mcp__meshtastic__logs_window(start="-2m", level="WARN|ERROR|CRIT", max_lines=20)` — recent firmware errors. If empty, say "no recent errors"; don't manufacture concern.
|
||||
- `mcp__meshtastic__telemetry_timeline(window="1h", field="free_heap", max_points=60)` — heap trend. If `slope_per_min < -50`, flag it and recommend `/leakhunt window=6h` for a deeper read; otherwise just note the current free heap.
|
||||
- If `recorder_status` shows `running:false` or `files.telemetry.last_ts` is null, note "recorder has no telemetry yet — enable `set_debug_log_api(True)` to populate" and skip this step gracefully.
|
||||
- `mcp__meshtastic__logs_window(start="-2m", level="WARN|ERROR|CRIT", max_lines=20)` - recent firmware errors. If empty, say "no recent errors"; don't manufacture concern.
|
||||
- `mcp__meshtastic__telemetry_timeline(window="1h", field="free_heap", max_points=60)` - heap trend. If `slope_per_min < -50`, flag it and recommend `/leakhunt window=6h` for a deeper read; otherwise just note the current free heap.
|
||||
- If `recorder_status` shows `running:false` or `files.telemetry.last_ts` is null, note "recorder has no telemetry yet - enable `set_debug_log_api(True)` to populate" and skip this step gracefully.
|
||||
|
||||
8. **Suggest next actions only for specific, recognisable failure modes**:
|
||||
- Stale PKI pubkey one-way → "run `/test tests/mesh/test_direct_with_ack.py` — the retry + nodeinfo-ping heals this in the test path."
|
||||
- Stale PKI pubkey one-way → "run `/test tests/mesh/test_direct_with_ack.py` - the retry + nodeinfo-ping heals this in the test path."
|
||||
- Region mismatch → "re-bake one side via `./mcp-server/run-tests.sh --force-bake`."
|
||||
- Device unreachable, reachable via DFU → `touch_1200bps(port=...)` + `pio_flash`. If not even DFU responds AND the device is on a PPPS hub, escalate to `uhubctl_cycle(role=..., confirm=True)`.
|
||||
- CP2102-wedged-driver on macOS → see the note in `run-tests.sh`.
|
||||
@@ -63,6 +63,6 @@ Call the meshtastic MCP tool bundle and format a structured health report for on
|
||||
|
||||
## What NOT to do
|
||||
|
||||
- No writes. No `set_config`, no `reboot`, no `factory_reset`. This is a read-only diagnostic skill — if the operator wants to change state, they'll ask explicitly.
|
||||
- No writes. No `set_config`, no `reboot`, no `factory_reset`. This is a read-only diagnostic skill - if the operator wants to change state, they'll ask explicitly.
|
||||
- No `flash` / `erase_and_flash`. Those are separate escalations.
|
||||
- No holding SerialInterface across tool calls — open, query, close; next device. The port lock is exclusive.
|
||||
- No holding SerialInterface across tool calls - open, query, close; next device. The port lock is exclusive.
|
||||
|
||||
@@ -5,40 +5,40 @@ argument-hint: [window=1h] [field=free_heap] [variant=local]
|
||||
|
||||
<!-- markdownlint-disable MD029 -->
|
||||
|
||||
# `/leakhunt` — read the recorder, classify a memory leak
|
||||
# `/leakhunt` - read the recorder, classify a memory leak
|
||||
|
||||
Use the always-on recorder (`mcp-server/.mtlog/`) to read a heap timeline plus the matching log slice and produce a one-page verdict: **steady / slow leak / fragmentation / OOM-imminent**. No firmware changes, no special build flags — the LocalStats telemetry packet that the firmware already broadcasts every ~60 s carries `heap_free_bytes` and `heap_total_bytes`.
|
||||
Use the always-on recorder (`mcp-server/.mtlog/`) to read a heap timeline plus the matching log slice and produce a one-page verdict: **steady / slow leak / fragmentation / OOM-imminent**. No firmware changes, no special build flags - the LocalStats telemetry packet that the firmware already broadcasts every ~60 s carries `heap_free_bytes` and `heap_total_bytes`.
|
||||
|
||||
## Two signal paths — pick the right one
|
||||
## Two signal paths - pick the right one
|
||||
|
||||
| Path | Build flag | Cadence | Per-thread attribution | Cost |
|
||||
| --------------------- | ---------------- | -------------- | ---------------------- | ------------------------- |
|
||||
| LocalStats packet | (default) | ~60 s | No | Free — always on |
|
||||
| LocalStats packet | (default) | ~60 s | No | Free - always on |
|
||||
| `[heap N]` log prefix | `-DDEBUG_HEAP=1` | every log line | Yes (Thread X leaked) | Bigger flash + log volume |
|
||||
|
||||
Both feed the same `telemetry_timeline(field="free_heap")` query — when DEBUG_HEAP is on, the recorder synthesizes telemetry rows from log prefixes (tagged `source: debug_heap`), so a single timeline call gets whichever signal is available. **For a slow leak diagnosis, the default path is plenty** (60 s cadence over 6 h = 360 points; linear regression over that nails sub-100-byte/min slopes). **DEBUG_HEAP is for attribution** — when the slope is real and you need to know which thread is leaking.
|
||||
Both feed the same `telemetry_timeline(field="free_heap")` query - when DEBUG_HEAP is on, the recorder synthesizes telemetry rows from log prefixes (tagged `source: debug_heap`), so a single timeline call gets whichever signal is available. **For a slow leak diagnosis, the default path is plenty** (60 s cadence over 6 h = 360 points; linear regression over that nails sub-100-byte/min slopes). **DEBUG_HEAP is for attribution** - when the slope is real and you need to know which thread is leaking.
|
||||
|
||||
## What to do
|
||||
|
||||
1. **Parse `$ARGUMENTS`**: optional `window` (default `1h`, accepts `30m`/`6h`/`-3d`/etc.), optional `field` (default `free_heap`; alternates: `total_heap`, `battery_level`, anything in the LocalStats variant), optional `variant` (default `local`; alternates: `device`, `environment`, `power`, `airQuality`, `health`).
|
||||
|
||||
2. **Verify the recorder is alive** — call `mcp__meshtastic__recorder_status`. Check:
|
||||
2. **Verify the recorder is alive** - call `mcp__meshtastic__recorder_status`. Check:
|
||||
- `running == True`
|
||||
- `files.telemetry.lines > 0` (at least one telemetry packet recorded — if zero, the device hasn't broadcast LocalStats yet OR `set_debug_log_api` has never been on; tell the operator to run `mcp__meshtastic__set_debug_log_api(enabled=True)` and wait one device-update interval)
|
||||
- `files.telemetry.last_ts` within the last 5 minutes (if older, the device is silent — log that, not "leak detected")
|
||||
- `files.telemetry.lines > 0` (at least one telemetry packet recorded - if zero, the device hasn't broadcast LocalStats yet OR `set_debug_log_api` has never been on; tell the operator to run `mcp__meshtastic__set_debug_log_api(enabled=True)` and wait one device-update interval)
|
||||
- `files.telemetry.last_ts` within the last 5 minutes (if older, the device is silent - log that, not "leak detected")
|
||||
|
||||
3. **Detect whether DEBUG_HEAP is active** — `mcp__meshtastic__logs_window(start="-2m", grep=r"\\[heap \\d+\\]", max_lines=3)`. If any line matches, the firmware has the prefix → DEBUG_HEAP is on, expect higher-cadence data and `heap_event` rows. If zero matches over the last 2 minutes, you're on the LocalStats-only path.
|
||||
3. **Detect whether DEBUG_HEAP is active** - `mcp__meshtastic__logs_window(start="-2m", grep=r"\\[heap \\d+\\]", max_lines=3)`. If any line matches, the firmware has the prefix → DEBUG_HEAP is on, expect higher-cadence data and `heap_event` rows. If zero matches over the last 2 minutes, you're on the LocalStats-only path.
|
||||
|
||||
4. **Pull the timeline** — `mcp__meshtastic__telemetry_timeline(window=$window, variant=$variant, field=$field, max_points=200)`. Read:
|
||||
- `samples` — how many raw points contributed
|
||||
- `min`, `max` — total swing
|
||||
- `slope_per_min` — units per minute (linear regression over the whole window)
|
||||
4. **Pull the timeline** - `mcp__meshtastic__telemetry_timeline(window=$window, variant=$variant, field=$field, max_points=200)`. Read:
|
||||
- `samples` - how many raw points contributed
|
||||
- `min`, `max` - total swing
|
||||
- `slope_per_min` - units per minute (linear regression over the whole window)
|
||||
|
||||
5. **Pull the log context for the same window** — `mcp__meshtastic__logs_window(start="-${window}", grep="Heap status|leaked heap|freed heap|out of memory|Alloc an err|panic|abort", max_lines=200)`. These are the strings the firmware emits when something memory-related happens (`DEBUG_HEAP` builds emit `"Heap status:"` and `"leaked heap"` lines; production builds emit `"Alloc an err"` on failure and `"out of memory"` on OOM).
|
||||
5. **Pull the log context for the same window** - `mcp__meshtastic__logs_window(start="-${window}", grep="Heap status|leaked heap|freed heap|out of memory|Alloc an err|panic|abort", max_lines=200)`. These are the strings the firmware emits when something memory-related happens (`DEBUG_HEAP` builds emit `"Heap status:"` and `"leaked heap"` lines; production builds emit `"Alloc an err"` on failure and `"out of memory"` on OOM).
|
||||
|
||||
6. **Pull marker events** so we know if the operator labeled phases — `mcp__meshtastic__events_window(start="-${window}", kind="mark|connection_lost|connection_established")`. If a `connection_lost` overlaps a sharp drop, that's not a leak; that's a reboot.
|
||||
6. **Pull marker events** so we know if the operator labeled phases - `mcp__meshtastic__events_window(start="-${window}", kind="mark|connection_lost|connection_established")`. If a `connection_lost` overlaps a sharp drop, that's not a leak; that's a reboot.
|
||||
|
||||
6a. **(DEBUG_HEAP only) Per-thread attribution** — `mcp__meshtastic__logs_window(start="-${window}", grep="leaked heap", max_lines=200)`. Each row has a structured `heap_event` field with `{kind, thread, before, after, delta}`. Aggregate by thread: sum the `delta` over the window per thread name. The thread with the largest cumulative negative delta is your suspect. Note the count too — a thread with 50× small leaks is different from 1× big leak.
|
||||
6a. **(DEBUG_HEAP only) Per-thread attribution** - `mcp__meshtastic__logs_window(start="-${window}", grep="leaked heap", max_lines=200)`. Each row has a structured `heap_event` field with `{kind, thread, before, after, delta}`. Aggregate by thread: sum the `delta` over the window per thread name. The thread with the largest cumulative negative delta is your suspect. Note the count too - a thread with 50× small leaks is different from 1× big leak.
|
||||
|
||||
7. **Classify** based on what the data says, NOT on what you wish it said. Use these rules in order:
|
||||
- **Insufficient data** (< 5 samples): say so. Suggest a longer window or longer wait. Stop.
|
||||
@@ -47,7 +47,7 @@ Both feed the same `telemetry_timeline(field="free_heap")` query — when DEBUG_
|
||||
- **Slow leak**: `slope_per_min < -50` AND `max - min > 1000` AND no reboot. The heap is monotonically (or near-monotonically) declining. Estimate time-to-zero: `min / -slope_per_min` minutes. Surface it.
|
||||
- **Fragmentation suspect**: `slope_per_min` close to zero (|x| < 50) BUT min trends down across the window AND the log slice shows `Alloc an err` warnings WITHOUT total OOM. Means free total is OK but largest contiguous block is shrinking. Recommend a `DEBUG_HEAP` build to confirm.
|
||||
- **Steady**: |slope_per_min| < 50, no error lines. Heap is fine.
|
||||
- **Recovery curve**: slope is POSITIVE — heap recovered. Either a workload completed or GC fired. Note it; not a leak.
|
||||
- **Recovery curve**: slope is POSITIVE - heap recovered. Either a workload completed or GC fired. Note it; not a leak.
|
||||
|
||||
8. **Report**:
|
||||
|
||||
@@ -58,14 +58,14 @@ Both feed the same `telemetry_timeline(field="free_heap")` query — when DEBUG_
|
||||
build : DEBUG_HEAP=ON (per-line prefix detected)
|
||||
samples : 14,200 over 6h (cadence ~1.5s, log-line synth)
|
||||
free_heap : min 92,344 / max 124,008 / range 31,664
|
||||
slope : -82 bytes/min (negative — heap declining)
|
||||
slope : -82 bytes/min (negative - heap declining)
|
||||
reboots : none in window
|
||||
OOM events : none
|
||||
error lines : 3× "Alloc an err=ESP_ERR_NO_MEM" at +4h12m, +5h08m, +5h44m
|
||||
thread leaks : (DEBUG_HEAP) MeshPacket -3,124 B over 18 events
|
||||
Router -1,408 B over 4 events
|
||||
others -240 B
|
||||
verdict : SLOW LEAK — primary suspect MeshPacket thread
|
||||
verdict : SLOW LEAK - primary suspect MeshPacket thread
|
||||
est. time-to-OOM: ~1,127 min (~18.8 h) at current slope
|
||||
evidence : (3 log line citations with uptimes)
|
||||
```
|
||||
@@ -86,7 +86,7 @@ Both feed the same `telemetry_timeline(field="free_heap")` query — when DEBUG_
|
||||
|
||||
- Don't assume a leak from a single dip. LocalStats fires every ~60 s and the firmware naturally allocates+frees on each broadcast cycle; one packet sees the trough. Look at the slope, not the deltas.
|
||||
- Don't recommend code changes. This skill diagnoses; the operator decides what to fix.
|
||||
- Don't enable `set_debug_log_api` automatically — if it's off, telemetry isn't reaching pubsub anyway, and the recorder will be empty. Tell the operator to flip it on and wait, then re-run.
|
||||
- Don't enable `set_debug_log_api` automatically - if it's off, telemetry isn't reaching pubsub anyway, and the recorder will be empty. Tell the operator to flip it on and wait, then re-run.
|
||||
- Don't run heavy workloads to "trigger the leak." The recorder is passive; we read what's there.
|
||||
|
||||
## Companion: `mark_event` for stress runs
|
||||
|
||||
@@ -5,7 +5,7 @@ argument-hint: <test-node-id> [count=5]
|
||||
|
||||
<!-- markdownlint-disable MD029 -->
|
||||
|
||||
# `/repro` — flakiness triage for one test
|
||||
# `/repro` - flakiness triage for one test
|
||||
|
||||
Re-run a single pytest node ID N times in isolation, track pass rate, and surface what's _different_ in the firmware logs between the passing attempts and the failing ones. Turns "it's flaky, I guess" into "it fails when X, passes when Y."
|
||||
|
||||
@@ -13,7 +13,7 @@ Re-run a single pytest node ID N times in isolation, track pass rate, and surfac
|
||||
|
||||
1. **Parse `$ARGUMENTS`**: first token is the pytest node id (e.g. `tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[nrf52->esp32s3]`); second token is an integer count (default `5`, cap at `20`). If the first token doesn't look like a test path (no `::` and no `tests/` prefix), treat the whole `$ARGUMENTS` as a `-k` filter instead.
|
||||
|
||||
2. **Sanity-check the hub first** (so we're not measuring "nothing plugged in" N times): call `mcp__meshtastic__list_devices`. If the test name contains `nrf52` or `esp32s3` and the matching VID isn't present, stop and report — re-running won't help.
|
||||
2. **Sanity-check the hub first** (so we're not measuring "nothing plugged in" N times): call `mcp__meshtastic__list_devices`. If the test name contains `nrf52` or `esp32s3` and the matching VID isn't present, stop and report - re-running won't help.
|
||||
|
||||
3. **Loop N times**. For each iteration:
|
||||
|
||||
@@ -37,10 +37,10 @@ Re-run a single pytest node ID N times in isolation, track pass rate, and surfac
|
||||
|
||||
5. **On mixed outcomes**: diff the firmware log tails between a representative passing attempt and a representative failing attempt. Focus on:
|
||||
- Error-level lines only present in failures (`PKI_UNKNOWN_PUBKEY`, `Alloc an err=`, `Skip send`, `No suitable channel`)
|
||||
- Timing around the assertion event — did a broadcast go out, was there an ACK, did NAK fire?
|
||||
- Timing around the assertion event - did a broadcast go out, was there an ACK, did NAK fire?
|
||||
- Device state fields that changed (nodesByNum entries, region/preset, channel_num)
|
||||
|
||||
Surface the top 3 differences as a "passes when / fails when" table. Don't dump full logs — pull specific lines with uptime timestamps.
|
||||
Surface the top 3 differences as a "passes when / fails when" table. Don't dump full logs - pull specific lines with uptime timestamps.
|
||||
|
||||
5a. **Archive recorder slices per attempt** (no extra device interaction; the recorder runs autouse). Right after each attempt finishes, capture its `(start_ts, end_ts)` and call `mcp__meshtastic__recorder_export(start=<start>, end=<end>, dest_dir="mcp-server/tests/repro_artifacts/<safe-test-id>/attempt_<n>/")`. This drops a `logs.jsonl`, `telemetry.jsonl`, `packets.jsonl`, and `events.jsonl` snapshot scoped to the attempt window. Use these for cross-attempt diffs in step 5: `jq '.line' logs.jsonl` is faster than re-running the test, and the telemetry slice lets you compare heap behavior across attempts.
|
||||
|
||||
@@ -59,12 +59,12 @@ Re-run a single pytest node ID N times in isolation, track pass rate, and surfac
|
||||
|
||||
## Examples
|
||||
|
||||
- `/repro tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[esp32s3->nrf52] 10` — runs 10 times, diffs firmware logs.
|
||||
- `/repro broadcast_delivers` — no `::`, no `tests/`, so interpreted as `-k broadcast_delivers`; runs every matching test the default 5 times.
|
||||
- `/repro tests/telemetry/test_device_telemetry_broadcast.py 3` — shorter run for a slow test.
|
||||
- `/repro tests/mesh/test_direct_with_ack.py::test_direct_with_ack_roundtrip[esp32s3->nrf52] 10` - runs 10 times, diffs firmware logs.
|
||||
- `/repro broadcast_delivers` - no `::`, no `tests/`, so interpreted as `-k broadcast_delivers`; runs every matching test the default 5 times.
|
||||
- `/repro tests/telemetry/test_device_telemetry_broadcast.py 3` - shorter run for a slow test.
|
||||
|
||||
## Constraints
|
||||
|
||||
- Don't exceed `count=20` per invocation — airtime and USB wear add up. If the user asks for 50, negotiate down.
|
||||
- Don't exceed `count=20` per invocation - airtime and USB wear add up. If the user asks for 50, negotiate down.
|
||||
- Don't rebuild firmware as part of triage; flakes that only reproduce under different firmware belong in a separate session.
|
||||
- If the FIRST attempt fails AND the rest all pass, that's a classic "state leak from a prior test" → say so and suggest running with `--force-bake` or starting from a clean state rather than chasing the first failure.
|
||||
|
||||
@@ -3,7 +3,7 @@ description: Run the mcp-server test suite (auto-detects devices) and interpret
|
||||
argument-hint: [pytest-args]
|
||||
---
|
||||
|
||||
# `/test` — mcp-server test runner with interpretation
|
||||
# `/test` - mcp-server test runner with interpretation
|
||||
|
||||
Run `mcp-server/run-tests.sh` and make sense of the output so the operator doesn't have to.
|
||||
|
||||
@@ -15,25 +15,25 @@ Run `mcp-server/run-tests.sh` and make sense of the output so the operator doesn
|
||||
./mcp-server/run-tests.sh $ARGUMENTS
|
||||
```
|
||||
|
||||
The wrapper auto-detects connected Meshtastic devices, maps each to its PlatformIO env, exports the required `MESHTASTIC_MCP_ENV_*` env vars, and invokes pytest. If the user passed no arguments, the wrapper supplies a sensible default set (`tests/ --html=tests/report.html --self-contained-html --junitxml=tests/junit.xml -v --tb=short`). A `--report-log=tests/reportlog.jsonl` arg is always appended (unless the operator passed their own). `--assume-baked` is deliberately NOT in the defaults — `test_00_bake.py` has its own skip-if-already-baked check and runs the ~8 s verification by default. Operators can opt into the fast path with `--assume-baked`, or force a reflash with `--force-bake`.
|
||||
The wrapper auto-detects connected Meshtastic devices, maps each to its PlatformIO env, exports the required `MESHTASTIC_MCP_ENV_*` env vars, and invokes pytest. If the user passed no arguments, the wrapper supplies a sensible default set (`tests/ --html=tests/report.html --self-contained-html --junitxml=tests/junit.xml -v --tb=short`). A `--report-log=tests/reportlog.jsonl` arg is always appended (unless the operator passed their own). `--assume-baked` is deliberately NOT in the defaults - `test_00_bake.py` has its own skip-if-already-baked check and runs the ~8 s verification by default. Operators can opt into the fast path with `--assume-baked`, or force a reflash with `--force-bake`.
|
||||
|
||||
2. **Read the pre-flight header.** First ~6 lines print the detected hub (role → port → env). If that line reads `detected hub : (none)`, the wrapper will narrow to `tests/unit` only — say so explicitly in your summary so the operator knows hardware tiers were skipped.
|
||||
2. **Read the pre-flight header.** First ~6 lines print the detected hub (role → port → env). If that line reads `detected hub : (none)`, the wrapper will narrow to `tests/unit` only - say so explicitly in your summary so the operator knows hardware tiers were skipped.
|
||||
|
||||
3. **On pass**: one-line summary of the form `N passed, M skipped in <duration>`. Don't enumerate the test names — the user can read those. Do mention any SKIPPED tests and name the cause:
|
||||
3. **On pass**: one-line summary of the form `N passed, M skipped in <duration>`. Don't enumerate the test names - the user can read those. Do mention any SKIPPED tests and name the cause:
|
||||
- `"role not present on hub"` → device unplugged; operator knows to reconnect.
|
||||
- `"firmware not baked with USERPREFS_UI_TEST_LOG"` → tests/ui skipped because the macro isn't in firmware yet; suggest `--force-bake`.
|
||||
- `"uhubctl not installed"` → tests/recovery + peer-offline skipped; suggest `brew install uhubctl` / `apt install uhubctl`.
|
||||
- `"no PPPS-capable hubs detected"` → tests/recovery skipped because the hub doesn't support per-port power; the tier will never run on that setup.
|
||||
- `"opencv-python-headless is not installed"` → tests/ui auto-deselected by run-tests.sh; suggest `pip install -e 'mcp-server/.[ui]'`.
|
||||
|
||||
4. **On failure**: for every FAILED test, open `mcp-server/tests/report.html` and extract the `Meshtastic debug` section for that test. pytest-html embeds the firmware log stream + device state dump there; the 200-line firmware log tail is usually enough to explain the failure. Summarise: which test, one-line assertion message, the firmware log lines that matter (things like `PKI_UNKNOWN_PUBKEY`, `Skip send NodeInfo`, `Error=`, `Guru Meditation`, `assertion failed`). For UI-tier failures also glance at `mcp-server/tests/ui_captures/<session>/<test>/transcript.md` — it records each step's frame + OCR.
|
||||
4. **On failure**: for every FAILED test, open `mcp-server/tests/report.html` and extract the `Meshtastic debug` section for that test. pytest-html embeds the firmware log stream + device state dump there; the 200-line firmware log tail is usually enough to explain the failure. Summarise: which test, one-line assertion message, the firmware log lines that matter (things like `PKI_UNKNOWN_PUBKEY`, `Skip send NodeInfo`, `Error=`, `Guru Meditation`, `assertion failed`). For UI-tier failures also glance at `mcp-server/tests/ui_captures/<session>/<test>/transcript.md` - it records each step's frame + OCR.
|
||||
|
||||
5. **Classify the failure** as one of:
|
||||
- **Transient/flake**: LoRa collision, timing-sensitive assertion, first-attempt NAK + successful retry pattern. Propose `/repro <test_node_id>` to confirm.
|
||||
- **Environmental**: device unreachable, port busy, CP2102 driver wedged. Suggest the specific recovery in escalation order: (a) replug USB, (b) `touch_1200bps(port=...)` + `pio_flash` for nRF52 DFU, (c) `uhubctl_cycle(role="nrf52", confirm=True)` when a device is fully wedged past DFU (needs `uhubctl` installed — `baked_single`'s auto-recovery hook does this once automatically). Also check `git status userPrefs.jsonc`.
|
||||
- **Environmental**: device unreachable, port busy, CP2102 driver wedged. Suggest the specific recovery in escalation order: (a) replug USB, (b) `touch_1200bps(port=...)` + `pio_flash` for nRF52 DFU, (c) `uhubctl_cycle(role="nrf52", confirm=True)` when a device is fully wedged past DFU (needs `uhubctl` installed - `baked_single`'s auto-recovery hook does this once automatically). Also check `git status userPrefs.jsonc`.
|
||||
- **Regression**: same assertion fails repeatedly, firmware log shows a new/unusual error. Surface the diff between expected and observed, identify the module likely responsible.
|
||||
|
||||
6. **Never run destructive recovery automatically.** If a failure looks like it needs a reflash, factory*reset, `uhubctl_cycle`, or USB replug, \_describe what to do* — don't execute. The operator decides.
|
||||
6. **Never run destructive recovery automatically.** If a failure looks like it needs a reflash, factory*reset, `uhubctl_cycle`, or USB replug, \_describe what to do* - don't execute. The operator decides.
|
||||
|
||||
## Arguments handling
|
||||
|
||||
@@ -43,5 +43,5 @@ Run `mcp-server/run-tests.sh` and make sense of the output so the operator doesn
|
||||
|
||||
## Side-effects to mention in summary
|
||||
|
||||
- The session fixture snapshots `userPrefs.jsonc` at session start and restores at teardown (plus on `atexit`). After a clean run, `git status userPrefs.jsonc` should be empty. If the wrapper's pre-flight printed a warning about a stale sidecar, call that out — means a prior session crashed.
|
||||
- The session fixture snapshots `userPrefs.jsonc` at session start and restores at teardown (plus on `atexit`). After a clean run, `git status userPrefs.jsonc` should be empty. If the wrapper's pre-flight printed a warning about a stale sidecar, call that out - means a prior session crashed.
|
||||
- `mcp-server/tests/report.html` and `junit.xml` are regenerated on every run; the HTML is self-contained (shareable).
|
||||
|
||||
Reference in New Issue
Block a user