docs: update test instructions to prefer bin/run-tests.sh; fix suite count to 19; add Copilot interface caveat
This commit is contained in:
nomdetom
@@ -635,27 +635,47 @@ Most workflows can be triggered manually via `workflow_dispatch` for testing.
|
||||
|
||||
### Native unit tests (C++)
|
||||
|
||||
Unit tests in `test/` directory with 17 test suites:
|
||||
Unit tests in `test/` directory with 19 test suites:
|
||||
|
||||
- `test_admin_radio/` - LoRa region/config validation and AdminModule dispatch
|
||||
- `test_atak/` - ATAK integration
|
||||
- `test_crypto/` - Cryptography
|
||||
- `test_default/` - Default configuration
|
||||
- `test_hop_scaling/` - Hop scaling histogram and required-hop logic
|
||||
- `test_http_content_handler/` - HTTP handling
|
||||
- `test_mac_from_string/` - MAC address parsing
|
||||
- `test_mesh_module/` - Module framework
|
||||
- `test_meshpacket_serializer/` - Packet serialization
|
||||
- `test_mqtt/` - MQTT integration
|
||||
- `test_packet_history/` - Packet history tracking
|
||||
- `test_packet_signing/` - Packet signing
|
||||
- `test_position_precision/` - Position precision helpers
|
||||
- `test_radio/` - Radio interface
|
||||
- `test_serial/` - Serial communication
|
||||
- `test_traffic_management/` - Traffic management
|
||||
- `test_traffic_management/` - Traffic management (dedup, rate-limit, hop-trim, role exceptions)
|
||||
- `test_transmit_history/` - Retransmission tracking
|
||||
- `test_type_conversions/` - NodeDB v25 type conversion (bitfield round-trips, NodeInfoLite)
|
||||
- `test_utf8/` - UTF-8 utilities
|
||||
|
||||
Run command (preferred — avoids pipe-buffering and the Ubuntu externally-managed-environment error):
|
||||
**Preferred run command — `bin/run-tests.sh`** (uses the `coverage` env with ASan/LSan sanitizers, emits an unambiguous RED/AMBER/GREEN verdict, and catches missing suites as AMBER):
|
||||
|
||||
```bash
|
||||
./bin/run-tests.sh # all suites
|
||||
./bin/run-tests.sh -f test_traffic_management # single suite
|
||||
./bin/run-tests.sh -f test_traffic_management > /tmp/test_out.txt 2>&1; tail -5 /tmp/test_out.txt
|
||||
```
|
||||
|
||||
Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER. The final line is machine-readable:
|
||||
|
||||
```
|
||||
RESULT: GREEN 19/19 suites passed
|
||||
RESULT: RED test_traffic_management: 1 failed
|
||||
RESULT: AMBER 18/19 suites ran (missing: test_radio) — all that ran passed
|
||||
```
|
||||
|
||||
> **Copilot interface note:** When running tests via the Copilot chat interface, edits made through the chat may not be reflected in the on-disk files that the test binary reads. If tests pass in chat but fail locally (or vice versa), verify the files on disk match what you expect before trusting the result. Always confirm with a local terminal run.
|
||||
|
||||
Raw `pio test` (no sanitizers, no verdict logic) — use only when you need to override the env:
|
||||
|
||||
```bash
|
||||
~/.platformio/penv/bin/python -m platformio test -e native -f test_your_suite > /tmp/test_out.txt 2>&1
|
||||
@@ -663,7 +683,7 @@ grep -E 'error:|PASS|FAIL|succeeded|failed' /tmp/test_out.txt
|
||||
tail -15 /tmp/test_out.txt
|
||||
```
|
||||
|
||||
Do **not** use `pio test … | tail -N` — it discards build errors and shows stale cached results. Do **not** use `pio test … | grep` — line-buffering makes the terminal appear hung until the process exits. Redirect to a file first, then grep.
|
||||
Do **not** pipe `pio test` — line-buffering makes the terminal appear hung and hides build errors.
|
||||
|
||||
Simulation testing: `bin/test-simulator.sh`
|
||||
|
||||
|
||||
@@ -10,18 +10,18 @@ This file (`AGENTS.md`) is a short pointer + quick reference for agents that don
|
||||
|
||||
## Quick command reference
|
||||
|
||||
| Action | Command |
|
||||
| -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Build a firmware variant | `pio run -e <env>` (e.g. `pio run -e rak4631`, `pio run -e heltec-v3`) |
|
||||
| Build native macOS host binary | `pio run -e native-macos` (Homebrew prereqs + CH341 LoRa setup in `variants/native/portduino/platformio.ini`) |
|
||||
| Clean + rebuild | `pio run -e <env> -t clean && pio run -e <env>` |
|
||||
| Flash a device | `pio run -e <env> -t upload --upload-port <port>` (or use the `pio_flash` MCP tool) |
|
||||
| Run firmware unit tests (native) | `~/.platformio/penv/bin/python -m platformio test -e native > /tmp/test_out.txt 2>&1` then `grep -E 'error:\|PASS\|FAIL\|succeeded\|failed' /tmp/test_out.txt` (redirect first — piping causes line-buffering) |
|
||||
| Run MCP hardware tests | `./mcp-server/run-tests.sh` |
|
||||
| Live TUI test runner | `mcp-server/.venv/bin/meshtastic-mcp-test-tui` |
|
||||
| Format before commit | `trunk fmt` |
|
||||
| Regenerate protobuf bindings | `bin/regen-protos.sh` |
|
||||
| Generate CI matrix | `./bin/generate_ci_matrix.py all [--level pr]` |
|
||||
| Action | Command |
|
||||
| -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||||
| Build a firmware variant | `pio run -e <env>` (e.g. `pio run -e rak4631`, `pio run -e heltec-v3`) |
|
||||
| Build native macOS host binary | `pio run -e native-macos` (Homebrew prereqs + CH341 LoRa setup in `variants/native/portduino/platformio.ini`) |
|
||||
| Clean + rebuild | `pio run -e <env> -t clean && pio run -e <env>` |
|
||||
| Flash a device | `pio run -e <env> -t upload --upload-port <port>` (or use the `pio_flash` MCP tool) |
|
||||
| Run firmware unit tests (native) | `./bin/run-tests.sh` (preferred — ASan/LSan + RED/AMBER/GREEN verdict); or raw: `~/.platformio/penv/bin/python -m platformio test -e native > /tmp/test_out.txt 2>&1` |
|
||||
| Run MCP hardware tests | `./mcp-server/run-tests.sh` |
|
||||
| Live TUI test runner | `mcp-server/.venv/bin/meshtastic-mcp-test-tui` |
|
||||
| Format before commit | `trunk fmt` |
|
||||
| Regenerate protobuf bindings | `bin/regen-protos.sh` |
|
||||
| Generate CI matrix | `./bin/generate_ci_matrix.py all [--level pr]` |
|
||||
|
||||
## MCP server (device + test automation)
|
||||
|
||||
@@ -108,7 +108,7 @@ Sequence these; don't parallelize on the same port.
|
||||
| `src/modules/` | Feature modules; `Telemetry/Sensor/` has 50+ I2C sensor drivers |
|
||||
| `variants/` | 200+ hardware variant definitions (`variant.h` + `platformio.ini` per board) |
|
||||
| `protobufs/` | `.proto` definitions; regenerate with `bin/regen-protos.sh` |
|
||||
| `test/` | Firmware unit tests (17 suites; `pio test -e native`) |
|
||||
| `test/` | Firmware unit tests (19 suites; `./bin/run-tests.sh` preferred, falls back to `pio test -e native`) |
|
||||
| `mcp-server/` | Python MCP server + pytest hardware integration tests |
|
||||
| `mcp-server/tests/` | Tiered pytest suite: `unit/`, `mesh/`, `telemetry/`, `monitor/`, `recovery/`, `ui/`, `fleet/`, `admin/`, `provisioning/` |
|
||||
| `.claude/commands/` | Claude Code slash command bodies |
|
||||
|
||||
+15
-1
@@ -4,6 +4,20 @@ This directory contains C++ unit tests that run on the host machine via Platform
|
||||
|
||||
## Running Tests
|
||||
|
||||
**Preferred: use `bin/run-tests.sh`** — it runs the `coverage` env (ASan/LSan sanitizers), cross-checks the number of suites that actually ran, and emits an unambiguous RED/AMBER/GREEN verdict:
|
||||
|
||||
```bash
|
||||
./bin/run-tests.sh # all suites
|
||||
./bin/run-tests.sh -f test_traffic_management # single suite
|
||||
./bin/run-tests.sh -f test_traffic_management > /tmp/test_out.txt 2>&1; tail -5 /tmp/test_out.txt
|
||||
```
|
||||
|
||||
Exit codes: 0 = GREEN, 1 = RED, 2 = AMBER.
|
||||
|
||||
> **Copilot interface note:** When running tests via the Copilot chat interface, edits made through the chat may not be reflected in the on-disk files that the test binary reads. If tests pass in chat but fail locally (or vice versa), verify the files on disk match what you expect before trusting the result. Always confirm with a local terminal run.
|
||||
|
||||
**Raw `pio test` (no sanitizers, no verdict logic)** — use when you need to override the env or inspect verbose Unity output:
|
||||
|
||||
```bash
|
||||
# All test suites
|
||||
pio test -e native
|
||||
@@ -17,7 +31,7 @@ pio test -e native -f test_your_module -vvv
|
||||
|
||||
**Never pipe through `| tail -N` to shorten output.** PlatformIO prints build errors at the top of output and test results at the bottom; `tail` will show stale cached results from a prior successful build while hiding the compile error that caused the current run to fail.
|
||||
|
||||
**Preferred pattern — redirect to file, then grep:**
|
||||
**Preferred pattern for raw pio — redirect to file, then grep:**
|
||||
|
||||
```bash
|
||||
# Redirect all output to a file; grep for errors and results after it exits
|
||||
|
||||
Reference in New Issue
Block a user