Add MCP server for interacting with meshtastic devices and testing framework / TUI (#10194)
* Start of MCP server and test suite * Add MCP server for interacting with meshtastic devices and testing framework / TUI * Update mcp-server/README.md Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * fix mcp-server review feedback from thread Agent-Logs-Url: https://github.com/meshtastic/firmware/sessions/91dc128a-ed50-4d07-8bb2-3dc6623a05f7 Co-authored-by: thebentern <9000580+thebentern@users.noreply.github.com> * Enhance StreamAPI and PhoneAPI for improved log record handling and concurrency control * Semgrep fixes * Trunk and semgrep fixes * optimize pio streaming tee file writes Agent-Logs-Url: https://github.com/meshtastic/firmware/sessions/04e26c6b-6a2b-45be-bbeb-79ae4d0be633 Co-authored-by: thebentern <9000580+thebentern@users.noreply.github.com> * chore: remove redundant log handle assignment Agent-Logs-Url: https://github.com/meshtastic/firmware/sessions/04e26c6b-6a2b-45be-bbeb-79ae4d0be633 Co-authored-by: thebentern <9000580+thebentern@users.noreply.github.com> * Consolidate type imports and remove placeholder test files * Add tests for config persistence and more exchange messages * Refactor position test to validate on-demand request/reply behavior * Remove position request/reply test and update README for telemetry behavior * Fix transmit history file to get removed on factory reset --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
This commit is contained in:
Ben Meadors
co-authored by
GitHub
Copilot
copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
GitHub
Copilot
copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
parent
aab4cd086f
commit
c8dac10348
@@ -0,0 +1,116 @@
|
||||
# Meshtastic MCP Server — Test Harness
|
||||
|
||||
Automated test suite for the MCP server, organized around real operator
|
||||
concerns rather than generic "unit vs hardware".
|
||||
|
||||
## Tiers
|
||||
|
||||
| Dir | Hardware | Question this tier answers |
|
||||
| --------------- | ----------------------- | --------------------------------------------------------------------- |
|
||||
| `unit/` | none | Do the parsing / filtering / profile-generation primitives work? |
|
||||
| `provisioning/` | 1 device, per-test bake | Did my pre-bake recipe stick? Does it survive a factory reset? |
|
||||
| `admin/` | 1 device, shared bake | Do my daily admin ops (owner, channel URL, config writes) round-trip? |
|
||||
| `mesh/` | 2 devices, shared bake | Do my devices actually form a mesh? Send + receive? ACKs? |
|
||||
| `telemetry/` | 2 devices, shared bake | Is telemetry reporting? Is position broadcast correct? |
|
||||
| `monitor/` | 1 device, shared bake | Is the boot log clean (no panics)? |
|
||||
| `fleet/` | varies | Are my CI runs isolated from each other? Are reflashes idempotent? |
|
||||
|
||||
## Quick start
|
||||
|
||||
```bash
|
||||
cd mcp-server
|
||||
pip install -e ".[test]"
|
||||
|
||||
# No hardware — 33 unit tests, ~3 seconds
|
||||
pytest tests/unit -v
|
||||
|
||||
# Hub attached (nRF52840 + ESP32-S3) — first run bakes, then exercises everything
|
||||
pytest tests/ --html=report.html
|
||||
|
||||
# Hub already baked with session profile (dev loop) — skip bake
|
||||
pytest tests/ --assume-baked --html=report.html
|
||||
|
||||
# Force a rebake (new firmware, new seed, etc.)
|
||||
pytest tests/ --force-bake --html=report.html
|
||||
```
|
||||
|
||||
## CLI flags
|
||||
|
||||
- `--force-bake` — always reflash both roles at session start, even if the
|
||||
current state matches the session profile.
|
||||
- `--assume-baked` — skip `test_00_bake.py` entirely. Use when you know the
|
||||
devices are already baked and want a fast dev loop.
|
||||
- `--hub-profile=<yaml>` — point at a YAML file for non-default hub hardware.
|
||||
Default targets VID `0x239a` (nRF52) and `0x303a`/`0x10c4` (ESP32-S3).
|
||||
- `--no-teardown-rebake` — skip the session-end rebake that `provisioning/`
|
||||
and `fleet/` tests perform. Useful in rapid iteration.
|
||||
|
||||
## Environment variables
|
||||
|
||||
- `MESHTASTIC_FIRMWARE_ROOT` — firmware repo path (defaults to `../` from tests/)
|
||||
- `MESHTASTIC_MCP_ENV_NRF52` — PlatformIO env for the nRF52 role (default
|
||||
`rak4631`)
|
||||
- `MESHTASTIC_MCP_ENV_ESP32S3` — PlatformIO env for the ESP32-S3 role (default
|
||||
`heltec-v3`)
|
||||
- `MESHTASTIC_MCP_SEED` — override the session PSK seed (default:
|
||||
`pytest-<unix-ts>`). Set this to reproduce a specific failing run.
|
||||
|
||||
## Fixtures you'll use when adding tests
|
||||
|
||||
All defined in `conftest.py`:
|
||||
|
||||
- **`hub_devices`** → `{"nrf52": "/dev/cu.X", "esp32s3": "/dev/cu.Y"}`. Auto-
|
||||
skips the test if a required role isn't present.
|
||||
- **`test_profile`** → USERPREFS dict for the session (`build_testing_profile`).
|
||||
- **`no_region_profile`** → variant without `USERPREFS_CONFIG_LORA_REGION`.
|
||||
- **`baked_mesh`** → verifies both devices are baked with the session profile
|
||||
(does NOT reflash — that's `test_00_bake.py`'s job).
|
||||
- **`baked_single`** → single verified baked device; parametrize `request.param`
|
||||
to pick role.
|
||||
- **`serial_capture`** → factory; `cap = serial_capture("esp32s3")` starts a
|
||||
pio device monitor session, drains into a per-test buffer, attaches the
|
||||
buffer to the pytest-html report on failure.
|
||||
- **`wait_until`** → exponential-backoff polling helper; `wait_until(lambda:
|
||||
predicate(), timeout=60)` replaces flaky `time.sleep()` patterns.
|
||||
|
||||
## Reports
|
||||
|
||||
`pytest --html=report.html` produces a self-contained HTML with:
|
||||
|
||||
- Per-test pass/fail/skip with timings
|
||||
- On failure: serial log capture from any `serial_capture` fixture used
|
||||
- On failure: `device_info` + lora config JSON for every role on the hub
|
||||
- Session seed and session start time (for reproducibility)
|
||||
|
||||
`pytest --junitxml=junit.xml` produces CI-integration XML.
|
||||
|
||||
`tool_coverage.json` is emitted at session end in the tests directory — shows
|
||||
which of the 38 MCP tools the run exercised. Useful for closing test gaps.
|
||||
|
||||
## Adding a new test
|
||||
|
||||
1. Pick the category that matches the operator concern (not the technical
|
||||
surface). "Does my fleet's owner name persist" is `admin/`, not `unit/`.
|
||||
2. If you need both devices, depend on `baked_mesh`. If you need one, depend
|
||||
on `baked_single`. If you need to mutate hardware state, put it in
|
||||
`provisioning/` or `fleet/` and add a `try/finally` teardown that re-bakes
|
||||
the session profile.
|
||||
3. Use `wait_until` for anything involving LoRa timing — fixed `sleep()`
|
||||
produces flakes.
|
||||
4. Use `serial_capture` when you need to observe firmware log output (e.g.
|
||||
"did the packet get decoded?").
|
||||
5. Add a `@pytest.mark.timeout(N)` — mesh tests routinely hit LoRa-airtime
|
||||
waits; default pytest timeout is infinite.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **All hardware tests SKIP** → hub not detected. Plug in the USB hub, verify
|
||||
with `pytest tests/ --collect-only` or `python -c "from meshtastic_mcp import
|
||||
devices; print(devices.list_devices())"`.
|
||||
- **`baked_mesh` fails with "devices not baked"** → run `pytest
|
||||
tests/test_00_bake.py` first, or pass `--force-bake` on the full run.
|
||||
- **Mesh formation tests time out** → check that both devices are on the same
|
||||
session profile (`--force-bake` forces both to the current seed).
|
||||
- **Provisioning tests leave device in bad state** → teardowns re-bake, but
|
||||
if a test crashes between "bake broken state" and "bake good state", run
|
||||
`pytest tests/test_00_bake.py --force-bake` to recover.
|
||||
Reference in New Issue
Block a user