Commit Graph
54 Commits
Author SHA1 Message Date
22072c5f4b Pr1.5 tmm nexthop (#10745)
* TrafficManagement: flat unified cache + persistent next-hop overflow store

Reworks the TrafficManagementModule cache layer (policing behaviour unchanged
from upstream) and adds a routing-hint overflow store:

- Flatten the ring: replace the cuckoo-hashed unified cache and the bucketed
  PSRAM NodeInfo index with plain flat arrays + linear scan (same idiom as
  WarmNodeStore). At LoRa packet rates an O(n) scan of the cache is negligible,
  and it removes a large amount of hashing/displacement complexity. The cache
  entry is 11 B; timestamps use a uniform +1 presence-offset so a 0 byte always
  means "empty" across every sub-store. Adds rebaseEpoch() so cached state
  survives the ~19 h relative-timestamp horizon instead of being flushed.

- Next-hop overflow cache: setNextHop/getNextHopHint store a confirmed last-byte
  relay for a destination, written only from NextHopRouter's ACK-confirmed
  decision (and mirrored from TraceRoute). NextHopRouter::getNextHop falls back
  to this cache when the hot NodeDB has no hint, so DMs/relays to long-tail
  nodes keep routing after the node ages out of NodeInfoLite.

- Persistence: preloadNextHopsFromNodeDB warm-starts the cache from persisted
  NodeInfoLite hints on first maintenance pass; next_hop entries are kept alive
  across the maintenance sweep (no TTL) and never clobbered by a stale preload.

All packet-policing logic (rate limit, position dedup, unknown-packet drop,
NodeInfo direct response, hop exhaustion) is the existing upstream behaviour,
untouched. HAS_TRAFFIC_MANAGEMENT defaults on so the module is compiled in. (see note).

Tests: upstream policing suite now actually runs (adds the MeshTypes.h include
that gates HAS_TRAFFIC_MANAGEMENT) plus 4 next-hop tests. Role-aware throttles,
politeness, precision clamp, port-interval and mesh-radius gating — and the
rate-limit >255 saturation fix — are deferred to the advanced-TMM branch.

Note: default dedup movement grid moves to ~91m, which also means 1.5km required to end up with the same signature position - coarser and therefore further than before.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* TrafficManagement: fix cppcheck constVariablePointer warning

`node` in preloadNextHopsFromNodeDB() is never written through — mark
it const to satisfy cppcheck's constVariablePointer check in CI.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add multi-hop NextHop recovery tests and unit tests for routing reliability

- Introduced a new test suite for multi-hop NextHop directed-message delivery and relay recovery in `test_nexthop_multihop_recovery.py`. This includes tests for end-to-end delivery and recovery after relay drop.
- Implemented unit tests in `test_main.cpp` for NextHop routing reliability mitigations, covering:
  - M1: Ambiguity-aware last-byte resolution.
  - M2: NextHopRouter's strict-neighbor gate and hop limit checks.
  - M3: Route-health freshness and failure decay.
- Enhanced mock classes to facilitate controlled testing of node behaviors and routing logic.

* grafting fixed

* Address Copilot review for PR #10735 (NextHop improvements)

- docs/nexthop-routing-reliability.md: update status from "no code
  changes yet" to reflect that mitigations and tests are implemented

RAM pressure and MIGRATION_VERBOSE concerns addressed upstream in
PR2.5 (per-platform TRAFFIC_MANAGEMENT_CACHE_SIZE) and PR2 (verbose
default=0) respectively; (0,0) sentinel fixed in PR2.5.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* CI: fix cppcheck constVariablePointer and test include path

- NextHopRouter.cpp: qualify two RouteHealth *h locals as const — only
  read for stale-route checks, never mutated through the pointer
- Router.cpp: qualify meshtastic_NodeInfoLite *node as const in
  shouldDecrementHopLimit — only read for favorite/role predicate
- test_position_module/test_main.cpp: change bare PositionModule.h to
  modules/PositionModule.h — build_flags sets -Isrc, not -Isrc/modules,
  so the bare form fails to resolve in the native PlatformIO test env

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* WarmStore: cache device role + protected category in last_heard low bits

Steal the low 6 bits of WarmNodeEntry.last_heard to carry an evicted node's
device role (4 bits) and a protected category (2 bits) for the hop-trim path,
at zero record-size cost (entry stays 40 B; no RAM/flash growth). The high bits
remain a real unix-seconds timestamp, quantised to 64 s — ample for warm LRU
ordering of long-tail nodes.

- absorb() packs role/protectedCat; place()/ring replay store the raw word so
  metadata round-trips through flash. LRU compares masked time (warmTimeOf).
- take() rehydration masks the metadata bits and restores the cached role so a
  re-admitted node isn't stuck at CLIENT until its next NodeInfo.
- NodeDB classifies the category (favorite/ignored/verified -> Flag;
  tracker/sensor/tak_tracker -> Role) at each eviction site.
- WarmNodeStore::lookupMeta() exposes role/category to consumers.
- Bump WARM_RING_MAGIC (WRNG->WRN2): old rings read as erased and rebuild;
  warm data is a non-critical evictee cache, so discard-on-upgrade is safe.

Tests: test_warm_store 11/11 (new meta round-trip + quantisation-aware ordering);
NodeDB compiles (test_nodedb_blocked 4/4).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WarmStore: migrate v1 rings/files by discarding last_heard, not the data

Previously the WRNG->WRN2 magic bump treated old rings as erased, discarding all
warm entries — including the PKI public keys that let evicted nodes keep
decrypting DMs. Instead, read v1 (WRNG / WRM1) records and keep each node's
identity + public key, discarding only last_heard (its low bits would otherwise
be misread as the new role/protected metadata). Records re-rank and re-learn
their role on next contact.

- Ring backend (nRF52840): ringReadHeader accepts both magics and reports v1 via
  an out-param; replay zeroes last_heard for v1 records. If the active head page
  is v1, force a rotation so new v2 records never land in a v1-headered page
  (which would discard their freshly-set role on the next load). Legacy pages
  convert to v2 as the ring rotates.
- File backend (warm.dat): bump WARM_STORE_MAGIC WRM1->WRM2; accept WRM1, verify
  CRC against the stored bytes, then discard last_heard and mark dirty so the
  next save rewrites as v2.

Tests: test_warm_store 12/12 (adds test_ws_v1_migration_discardsLastHeard:
key survives, role/protected reset).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* WarmStore: guard role bit-width + test eviction carries role/protected

- static_assert that the device role enum still fits the 4-bit warm metadata
  field (WARM_ROLE_MASK); fails the build loudly if a new role is added past 15
  rather than silently truncating role on eviction. (Max role today = 12.)
- Add test_migration_carriesRoleAndProtectedIntoWarm: a demoted TRACKER lands in
  the warm tier with its key, role=TRACKER and protected category=Role; a demoted
  CLIENT carries role=CLIENT/None. Exercises the NodeDB eviction path +
  warmProtectedCategory classification (the warm-store unit tests only cover
  absorb() directly).

Tests: test_nodedb_blocked 5/5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* fix copilot comments

* fix(test): restore #if HAS_TRAFFIC_MANAGEMENT guard in TMM test

The rebase onto PR1.5 lost the top-level HAS_TRAFFIC_MANAGEMENT guard
that PR1.5 introduced, leaving the #else/#endif tail orphaned and
causing compile errors on non-TMM builds.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-06-19 19:52:58 -05:00
TomGitHubClaude Opus 4.8github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>Ben Meadors
79a7dcc46c Pr1 nodedb warmstore (#10705)
* NodeDB: 3-tier node store with persistent warm tier (long-tail identity retention)

Introduces a tiered NodeDB so the device retains identity (public key,
last_heard) for far more nodes than fit in the full-record hot store,
without growing heap or the persisted nodes.proto unboundedly.

- Hot store: full NodeInfoLite, MAX_NUM_NODES (120 on nRF52).
- Satellite maps: position/telemetry/environment/status capped at
  MAX_SATELLITE_NODES (40 freshest); eviction via enforceSatelliteCaps /
  evictSatelliteOverCap.
- Warm tier (WarmNodeStore): 40 B {num,last_heard,public_key} records for
  evicted nodes so DMs to/from long-tail nodes keep encrypting/decrypting.
  Persisted to /prefs/warm.dat, or on nRF52840 a dedicated 12 KB raw-flash
  record-ring below LittleFS (3x4 KB pages; see linker scripts + the
  nrf52_warm_region.py post-link guard).

NodeDB::getOrCreateMeshNode now demotes evicted nodes into the warm tier and
re-admits them (restoring key/last_heard). Router PKI decrypt/encode resolve
the peer key via NodeDB::copyPublicKey (hot store, then warm tier).

NodeInfoLite gains snr_q4 (sint32, Q4-encoded dB); the float snr is zeroed on
disk. NodeInfoLite grows 105 -> 112 B; backup 2432 -> 2468 B.

Note: the snr_q4 .proto change still needs to land in the protobufs submodule
(generated header is updated here; submodule pointer left at upstream).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* NodeDB: robust receive + retention for blocked (ignored) nodes

Hardens how ignored/favourite nodes are received over admin and retained,
closing paths where a block could be lost or accidentally cleared.

- Blocking keeps the node's public key (admin set_ignored_node and
  addFromContact no longer zero it / drop the warm-tier key), so a blocked
  peer stays a verifiable identity.
- set_ignored_node creates the node if absent, so a block by node ID sticks
  even for a node we've never heard from (e.g. pushed by a remote admin) with
  no NodeInfo or key.
- Eviction protection (favourite/ignored/manually-verified) now also applies to
  the load-time hot-store migration and is never undone by cleanupMeshDB, which
  previously purged ignored nodes that lacked user info.
- The hot-store migration leaves our own node (index 0) in place and prefers to
  demote non-protected nodes, like the runtime eviction scan.

Caps the protected set (favourite + ignored + verified) at MAX_NUM_NODES-2 via
NodeDB::setProtectedFlag(), so at least two evictable slots always remain and
getOrCreateMeshNode can always make room — replacing the previous unconditional
append that could run off the end of the node vector when every node was
protected. A locally-set favourite/ignore that hits the cap reports back to the
phone via a ClientNotification.

Adds test_nodedb_blocked covering the migration, favourite/ignored eviction
protection, ignored-survives-cleanup, and the protected-node cap. The
maintenance methods stay private in production; the test reaches them through a
PIO_UNIT_TESTING-guarded friend shim.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

# Conflicts:
#	src/mesh/NodeDB.h

* fix copilot comments

* once again

* WarmNodeStore: fix cppcheck warnings (uninitvar, constVariablePointer)

Zero-initialise `stranded[]` and `seqs[]/order[]` VLAs so cppcheck can
verify there are no unguarded reads of uninitialised memory (the guards
exist but are not visible to static analysis). Mark two local pointers
`const` where the pointed-to entry is never mutated after assignment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* self-care added to assist 2.7 and 2.8 nodedb migration

* Tidy warm-store/self-care: comments, guards, log + flash cleanup

Style/cleanup pass over the branch (no behavior change except the noted
preprocessor simplifications, which are semantically identical):

- Comments: move function descriptions to the headers, cap in-function
  comments at ~3-4 lines, drop leading-number step markers, label stacked
  #endif blocks, de-decorate banner comments.
- dumpToLog: fully gate decl + definition + AdminModule call site behind
  MESHTASTIC_NODEDB_MIGRATION_VERBOSE so it compiles out when disabled
  (~1.2 KB when off).
- mesh-pb-constants: drop the dead nRF52832 WARM_NODE_COUNT branch and trim
  the macro docs.
- WarmNodeStore: simplify the redundant `ARCH_NRF52 && NRF52840_XXAA` guards
  to `NRF52840_XXAA`, add a kNoPage sentinel for the ring page state.
- Shorten the always-on LOG_WARN strings (~120 B flash).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* more tidying up, aligning with docs and undoing other-arch regressions

* Update protobufs (#19)

Co-authored-by: NomDeTom <116762865+NomDeTom@users.noreply.github.com>

* made the migration pathway cleareer

* address copilot review

* fixed a copilot review on a downstream PR.

* Address Copilot review comments for PR #10705 (warmstore/nodedb)

- WarmNodeStore.h: default MIGRATION_VERBOSE to 0 (suppress info-level
  chatter on production builds; opt in with =1)
- WarmNodeStore.cpp load(): move memset to top of function so all
  failure paths (header-read fail, invalid header) leave entries clear
- WarmNodeStore.cpp save(): replace manual spiLock lock/unlock around
  mkdir with LockGuard covering the full SafeFile sequence, matching
  the lock discipline in load()
- Router.cpp: memcpy(&p->public_key.bytes, ...) -> memcpy(p->public_key.bytes,
  ...) — pass decayed uint8_t* rather than pointer-to-array
- AdminModule.cpp: check setProtectedFlag return for PKC auto-favorite;
  log cap-refusal warning instead of unconditional "auto-favoriting"
- nrf52_warm_region.py: error message references both v6.ld and v7.ld

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* NodeDB: formatting cleanup (blank lines after preprocessor blocks)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Lukewarm store

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-06-18 08:41:34 +01:00
e4f4d1f9e7 Ci test report filter (#10722)
* ci(test report): drop no-status testsuites from the PlatformIO report

PlatformIO emits a self-closing <testsuite tests="0"/> row for every test_*
dir x every hardware variant it can't run on native (~4900 rows). They carry
no pass/fail/skip status and bury the suites that actually ran in the dorny
Test Report. Strip them (in generate-reports, before the reporter) so the
report shows only real results. The uploaded artifact keeps the full XML.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* Add bin/run-tests.sh: standardised local test verdict

Self-contained RED/AMBER/GREEN runner: matches all pass/fail spellings and
cross-checks the suite count against test/test_*/ so a missing suite shows AMBER.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* run-tests.sh: detect sanitizer faults + live build/test progress

Two usability improvements to the local verdict script:

1. Sanitizer-fault detection. A sanitizer-instrumented (coverage) build aborts
   non-zero at EXIT on an ASan/LSan/UBSan/TSan fault — most often a LeakSanitizer
   leak — AFTER every test has printed [PASSED], so pio reports it as
   [ERRORED]/SIGHUP with no :FAIL: anywhere and it masquerades as a phantom
   failure. verdict_red now recognises the documented fault signatures (ERROR:/
   WARNING: <San>:, SUMMARY: <San>:, Direct/Indirect leak of, heap-use-after-free,
   runtime error:, etc. — not the benign "failed to intercept" startup noise) and
   reports e.g. "RESULT: RED sanitizer fault — SUMMARY: AddressSanitizer: ...",
   plus the "run the binary bare (gdb hides it via ptrace)" recipe. Also flags the
   "all tests passed but aborted at exit" shape when the report was swallowed.

2. Progress trail. Long rebuilds were silent for minutes. A background heartbeat
   now appends a status line every few seconds to .pio/build/<env>/.runtests-progress
   (always — tail -f it to check a backgrounded/piped run) and live-updates the tty
   for interactive --quiet runs: build = objects recompiled / cached total + ETA;
   test = suites done / expected. The object-count baseline is cached per env in the
   gitignored build dir. Writes never touch the parsed verdict log; tty writes are
   guarded so a no-tty run emits no redirect-open error.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-15 13:53:17 -05:00
b938b63e8a fix a long-running CI bug that overran a lot (#10707)
* fix the fix

* Address Copilot review: add EXIT trap and clarify PKC comment

Add `trap` to kill meshtasticd on any early exit (python harness
failure, socket timeout) so CI never leaks a background process.

Reword the ARCH_PORTDUINO comment to make explicit that pki_encrypted=true
causes the from==0 plain-admin branch to be skipped, routing into the
PKC key-check — the underlying logic was correct all along.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Update PORTDUINO comment to reflect from==0 auth fix

The from==0 branch no longer requires !pki_encrypted (fixed upstream
in this branch), so update the simulator comment to reflect the actual
remaining reason for the early intercept: is_managed could still block
exit_simulator even for local packets.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-06-13 18:46:29 -05:00
TomandGitHub b76e5e6ba4 removes NRF52832 from codebase - it is vestigal at best (#10709) 2026-06-13 18:46:08 -05:00
8bb5364d8c tunk (#10684)
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-06-11 07:57:06 -05:00
TomandGitHub ab882c5619 EU regions merge (#10675)
* stronger together

* validate 2.4ghz regions

* less noise

* you're right, and that shapens the analysis significantly

* sassy rejoinder
2026-06-10 18:37:14 +01:00
TomandGitHub 1410f170f9 makes clod format as it goes (#10645) 2026-06-07 08:09:19 -05:00
TomGitHubcopilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>Austin
5c1b6b2a23 Size change reporting (#10488)
* feat: add firmware size reporting and comparison scripts from #9860

* feat: add silence output feature to size_report and implement tests for size reporting scripts

* rm shame

* Fix baseline artifact paths in size report workflow

* feat: add firmware size reporting and comparison scripts from #9860

* feat: add silence output feature to size_report and implement tests for size reporting scripts

* rm shame

* Fix baseline artifact paths in size report workflow

* fix write permissions

* tunk

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: Austin <vidplace7@gmail.com>
2026-06-04 06:31:27 -05:00
de345939af Automatic variable hop limits based on mesh activity and size estimation (#10176)
* asdf

* Implement SphereOfInfluenceModule for traffic management and eviction tracking

* Implement Sphere of Influence module for dynamic hop limit adjustment and role-based floor

* Update SAMPLING_DENOMINATOR to improve mesh size estimation accuracy

* Add debug logging for scale factor estimation and per-hop node counts in SphereOfInfluenceModule

* Enable variable hop limits and role-based hop floors in Sphere of Influence module

* Respond to copilot review

* Disable variable hop limits and role-based hop floors in Sphere of Influence module

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Implement adaptive sampling for unique node ID tracking in Sphere of Influence module

* Add state persistence for Sphere of Influence module

* Enhance Sphere of Influence module with state management and adaptive sampling adjustments

* Refactor hop scaling functionality: remove SphereOfInfluenceModule and introduce HopScalingModule

- Deleted SphereOfInfluenceModule.h, consolidating its responsibilities into the new HopScalingModule.
- Added HopScalingModule.h and HopScalingModule.cpp to manage hop scaling logic, including eviction tracking and sampling-based mesh size estimation.
- Implemented methods for recording evictions and packet senders, estimating scale factors, and computing required hops based on node activity.
- Introduced state persistence for hop scaling parameters to maintain continuity across reboots.
- Enhanced thread safety and modularity by utilizing concurrency features.

* Guard out STM32. Sowwy.

* Refactor HopScalingModule: enhance sampling logic and improve state management

* Add unit tests for HopScalingModule: implement mock database and various test scenarios

* Refactor test output in HopScalingModule tests: replace printf with TEST_MESSAGE for better integration with Unity

* Refactor HopScalingModule logging: replace lastStatusMode with descriptive mode names for improved readability

* Refactor test_main.cpp: change NodeNum variable to static and improve comments for clarity

* Remove unnecessary delay in setup function for improved test performance

* Add missing include for MeshTypes in test_main.cpp

* Refactor HopScalingModule tests: enhance mesh topology scenarios and improve test clarity

* Update HopScalingModule tests: flesh out node scenarios and improve clarity for dense and sparse mesh cases

* Fix politeness factor calculations in HopScalingModule and update related test scenarios for clarity. Remove outdated design doc.

* Enhance HopScalingModule: add sampled estimate for scaling decisions and refactor initial run state management

* Add sample traffic injection for HopScaling tests to enhance sampledEst visibility

* Enhance HopScalingModule: adjust windowFraction calculation for early triggers and improve test output formatting

* Enhance HopScalingModule: add jitter functionality to sampling denominator and update tests for consistent behavior

* Enhance HopScalingModule: implement adaptive sampling denominator adjustment and add reset functionality for tests

* Enhance HopScalingTestShim: add test-only clock and window helpers, update injectSampleTraffic for adaptive sampling, and improve scenario summary output

* Enhance HopScalingModule: add detailed documentation for functions, improve clarity of jitter and sampling logic, and reset functionality in tests

* Enhance HopScalingModule: add evictionEstimate parameter to estimateScaleFactor and update related logging for improved mesh size estimation

* Enhance HopScalingModule: adjust effective rolls calculation for improved accuracy, add eviction estimate logic, responding to all copilot review points

* Implement CompactHistogram for parallel hop scaling sampling

- Added CompactHistogram class to track node hop distances with bitwise sampling.
- Integrated CompactHistogram into HopScalingModule for independent packet sampling.
- Updated NodeDB to feed both the hop scaling module and the new histogram sampler.
- Enhanced HopScalingModule with methods to sample packets for the histogram and retrieve hop distribution statistics.
- Implemented tests for CompactHistogram functionality, including sampling, window rolling, and adaptive denominator scaling.
- Updated existing tests to validate the integration of the new histogram sampling mechanism.

* Enhance CompactHistogram and HopScalingModule: add per-hop distribution functionality, improve time handling for unit tests, and refine test setup for deterministic behavior

* CompactHistogram: add mesh size estimation, improve entry replacement logic, and update logging for per-hop distribution

* Refactor HopScalingModule and CompactHistogram integration

- Removed the suggestedHopFromCompactHistogram function to streamline hop suggestion logic.
- Updated HopScalingModule to directly utilize CompactHistogram's internal methods for hop suggestions and sampling.
- Enhanced logging in HopScalingModule to provide detailed histogram statistics.
- Modified test cases to ensure comprehensive coverage of new histogram behaviors and sampling logic.
- Improved node ID distribution in tests to better exercise sampling mechanisms.
- Ensured that filtering denominators are held for 12 hours before dropping, enhancing stability in sampling.

* Refactor CompactHistogram to support 13-hour activity tracking and introduce politeness regimes

- Updated the bitfield structure to accommodate 13-hour seen tracking.
- Changed the logic in rollHour() to analyze activity over the last 0-2 hours vs. 1-3 hours for politeness factor calculation.
- Introduced three politeness levels: GENEROUS, DEFAULT, and STRICT based on recent activity ratios.
- Adjusted filtering and sampling logic to reflect the new 13-hour tracking period.
- Updated unit tests to validate new behavior and ensure proper functionality of politeness regimes.

* Enhance CompactHistogram and HopScalingModule for improved sampling and decision-making

- Introduced a session-specific hash seed in CompactHistogram to reduce bias in node ID sampling.
- Updated sampling logic to use hashed node IDs instead of raw IDs for filtering and entry management.
- Added histogram rollover tracking in HopScalingModule to ensure proper decision-making after initial data collection.
- Adjusted logging to reflect the active state of the histogram and its comparison with NodeDB advisory hops.
- Enhanced unit tests to validate new sampling logic and memory layout changes.

* Expose hashNodeId for testing in CompactHistogram

* Add mesh trend statistics to CompactHistogram for enhanced activity tracking

* Implement histogram state persistence in CompactHistogram with save and load functions

* Refactor CompactHistogram to improve entry management and enhance rollHour logging

* feat: add HopScalingModule for adaptive hop limit recommendations

Introduces HopScalingModule, a sampled hop-distance histogram that
recommends the minimum hop limit needed to reach ~40 nodes, and
automatically reducing the hops as the mesh grows.

Key design:
- 512-byte packed histogram (128 × 4-byte Record entries) embedded
   in a new HopScalingModule.
- Each Record: 16-bit node hash, 3-bit hop distance, 13-bit seen bitmap
- Sampling filter: only nodes where (hash & (denom-1)) == 0 are kept;
  denominator doubles on overflow and halves when utilisation is low
- Hourly rollHour(): tallies per-hop counts, walks scaled buckets to
  find the minimum hop satisfying TARGET_AFFECTED_NODES (40), applies a
  politeness extension based on recent/older activity ratio, shifts all
  seen bitmaps, and persists state to /prefs/hopScalingState.bin
- Hop recommendation gated by bootstrap (requires >=1 rollHour before
  overriding HOP_MAX)
- NodeDB calls samplePacketForHistogram() on every non-MQTT rx packet
- Module also estimates total mesh size and logs useful information about
  mesh characteristics.

Changes:
- src/modules/HopScalingModule.h/.cpp: new module
- src/mesh/NodeDB.cpp: wire up samplePacketForHistogram
- src/mesh/Router.cpp: consume getLastRequiredHop()
- test/test_hop_scaling/: 12-test suite covering all mesh topologies and
  anticipated operational requirements

* test: increase run iterations in sparse to dense transition test

* feat: refactor HopScalingModule to use RUNS_PER_HOUR constant and improve logging

* feat: enhance HopScalingModule with filtering denominator management and add tests for state transitions

* refactor: remove CompactHistogram module and related files

* address copilot review comments

* Tweak: packet sampling only lora

* ove role-based hop floor logic and related definitions into the module - keep it in one place.

* Refactor MockNodeDB to use nodeInfoLiteSetBit for MQTT flag setting

* Refactor hop scaling parameters and logic to integer maths and put default values in defaults.h. Small flash size reduction, no functional impact.

* Update unit test preprocessor directives to PIO_UNIT_TESTING for consistency

* refactor: improve test output organization and clarity in hop scaling tests

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-06-04 05:59:49 -05:00
TomandGitHub fe23dcfa3a Merge pull request #10619 from NomDeTom/test_fixes
Some fixes and tidies for testing both online and in unit_tests
2026-06-03 13:43:54 +01:00
TomandGitHub ee441dd7b2 Merge pull request #10560 from meshtastic/migrate-overrideslot
Move overrideSlot from RegionProfile to RegionInfo (override per-region)
2026-05-30 22:04:37 +01:00
5e69bc6c3f Enable Narrow and Lite regions for EU (#10120)
* Enable Lite and Narrow regions and introduce getEffectiveDutyCycle for Lite profiles

* Add TrafficType enum and extend getConfiguredOrDefaultMsScaled to manage based on regionProfile settings

* Refactor telemetry modules to include TrafficType in getConfiguredOrDefaultMsScaled calls

* Update submodule protobufs to latest commit

* Add support for new region presets and modem presets in menu options

* Add new LoRa region codes and modem presets for EU bands

* boof

* Add modem presets for LITE and NARROW configurations

* Update subproject commit reference in protobufs

* Update protobufs

* Refactor modem preset definitions to use macro for consistency and clarity

* Refactor modem preset cases to use PRESET macro for consistency

* fix: update LoRa region code for EU 868 narrowband configuration

Co-authored-by: Copilot <copilot@github.com>

* Fix test suite failure

Co-authored-by: Copilot <copilot@github.com>

* Add override slot override - for when one override isn't enough.

Co-authored-by: Copilot <copilot@github.com>

* address copilot comments

---------

Co-authored-by: Copilot <copilot@github.com>
2026-05-21 10:20:09 -05:00
5e2ca8aed4 LR2021 radio on NRF_Promicro (#10401)
* LR2021 radio on NRF_Promicro

Co-authored-by: Copilot <copilot@github.com>

* Refactor LR2021 interface includes and conditional compilation for improved clarity

Co-authored-by: Copilot <copilot@github.com>

* Refactor LR20x0 interface: remove unused includes and update comments for clarity

* Fix LR2021 max power definitions and add radio type detection tests

* remove potato radio type detection tests

* Include placeholder for DCDC - currently requires godmode

* Added godmode features - not enabled by default

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-05-07 20:24:39 -05:00
TomandGitHub b11d29ff31 fix(mesh): update reconfigure methods to return true instead of RADIOLIB_ERR_NONE (#10407) 2026-05-07 05:57:34 -05:00
fcef46f4b0 dependency swap - INA3221Sensor (#10379)
* dependency swap - INA3221Sensor
update INA3221 initialization and measurement methods for compatibility with Rob Tillaart's library

Co-authored-by: Copilot <copilot@github.com>

* Addresses copilot review

Co-authored-by: Copilot <copilot@github.com>

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* Refine comments on USB detection and INA3221

Updated comments regarding USB detection and INA3221 usage.

* Fix static_assert conditions for INA3221 channel definitions

Co-authored-by: Copilot <copilot@github.com>

* moved macro defines earlier to allow better use.

Co-authored-by: Copilot <copilot@github.com>

* Add raw register read methods for bus voltage and shunt current in INA3221Sensor

Co-authored-by: Copilot <copilot@github.com>

---------

Co-authored-by: Copilot <copilot@github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-05-05 06:52:07 -05:00
TomandGitHub db9fdd6794 Fix: filter out SKIPPED tests in PlatformIO output to improve log clarity (#10214) 2026-04-21 16:43:35 -05:00
TomandBen Meadors 76dea77929 Add authoring guide for native unit tests in README.md (#10201)
* Add authoring guide for native unit tests in README.md

* Enhance documentation for agent tooling and native unit tests in README and related files

---------

Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-04-21 10:49:08 -05:00
f396200d38 Add authoring guide for native unit tests in README.md (#10201)
* Add authoring guide for native unit tests in README.md

* Enhance documentation for agent tooling and native unit tests in README and related files

---------

Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-04-19 13:30:50 -05:00
TomandGitHub b2c8cbb78d Enhance traffic management by adjusting position update interval and refining hop exhaustion logic based on channel congestion (#9921) 2026-04-10 10:53:04 -05:00
TomandGitHub a3b49b9028 Add powerlimits to reconfigured radio settings as well as init settings. (#10025)
* Add powerlimits to reconfigured radio settings as well as init settings.

* Refactor preamble length handling for wide LoRa configurations

* Moved the preamble setting to the main class and made the references static
2026-04-09 16:18:05 -05:00
TomandGitHub efd2613bd7 feat: add new configuration files for LR11xx variants (#9761)
* feat: add new configuration files for LR11xx variants

* style: reformat RF switch mode table for improved readability
2026-04-01 10:46:27 +11:00
f88bc732cc Improved manual build flow to make it easier (#8839)
* Improved flow to make easier

The emojis are intentional! I had minimal LLM input!!!

* try and fix input variable sanitisation

* and again

* again

* Copilot fixed it for me

* copilot didn't fix it for me

* copilot might have fixed it and I broke it by copypasting

---------

Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-03-31 07:53:59 -05:00
TomandBen Meadors 9e61c44629 fix(routing): prevent licensed users from rebroadcasting packets to or from unlicensed users (#9958)
* fix(routing): prevent licensed users from rebroadcasting packets from unlicensed or unknown users

* fix(routing): prevent licensed users from rebroadcasting packets to or from unlicensed users

---------

Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-03-31 07:36:04 -05:00
TomandBen Meadors 2955c12d2a Configure NFC pins as GPIO for older bootloaders (#10016)
* Configure NFC pins as GPIO for older bootloaders

Should hopefully solve the issue in https://github.com/meshtastic/firmware/issues/9986

* Fix formatting of CONFIG_NFCT_PINS_AS_GPIOS flag in platformio.ini

---------

Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-03-30 09:06:09 -05:00
283ccb83d1 Configure NFC pins as GPIO for older bootloaders (#10016)
* Configure NFC pins as GPIO for older bootloaders

Should hopefully solve the issue in https://github.com/meshtastic/firmware/issues/9986

* Fix formatting of CONFIG_NFCT_PINS_AS_GPIOS flag in platformio.ini

---------

Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-03-30 09:03:10 -05:00
bfaf6c6b20 fix(routing): prevent licensed users from rebroadcasting packets to or from unlicensed users (#9958)
* fix(routing): prevent licensed users from rebroadcasting packets from unlicensed or unknown users

* fix(routing): prevent licensed users from rebroadcasting packets to or from unlicensed users

---------

Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-03-22 13:47:52 -05:00
22d63fa69c Lora settings expansion and validation logic improvement (#9878)
* Enhance LoRa configuration with modem presets and validation logic

* Rename bootstrapLoRaConfigFromPreset tests to validateModemConfig for clarity and consistency

* additional tidy-ups to the validateModemConfig - still fundamentally broken at this point

* Enhance region validation by adding numPresets to RegionInfo and implementing validateRegionConfig in RadioInterface

* Add validation for modem configuration in applyModemConfig

* Fix region unset handling and improve modem config validation in handleSetConfig

* Refactor LoRa configuration validation methods and introduce clamping method for invalid settings

* Update handleSetConfig to use fromOthers parameter to either correct or reject invalid settings

* Fix some of the copilot review comments for LoRa configuration validation and clamping methods; add tests for region and preset handling

* Redid the slot default checking and calculation. Should resolve the outstanding comments.

* Add bandwidth calculation for LoRa modem preset fallback in clampConfigLora

* Remove unused preset name variable in validateConfigLora and fix default frequency slot check in applyModemConfig

* update tests for region handling

* Got the synthetic colleague to add mock service for testing

* Flash savings... hopefully

* Refactor modem preset handling to use sentinel values and improve default preset retrieval

* Refactor region handling to use profile structures for modem presets and channel calculations

* added comments for clarity on parameters

* Add shadow table tests and validateConfigLora enhancements for region presets

* Add isFromUs tests for preset validation in AdminModule

* Respond to copilot github review

* address copilot comments

* address null poointers

* fix build errors

* Fix the fix, undo the silly suggestions from synthetic reviewer.

* we all float here

* Fix include path for AdminModule in test_main.cpp

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

* More suggestion fixes

* admin module merge conflicts

* admin module fixes from merge hell

* fix: initialize default frequency slot and custom channel name; update LNA mode handling

* save some bytes...

* fix: simplify error logging for bandwidth checks in LoRa configuration

* Update src/mesh/MeshRadio.h

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
2026-03-20 07:43:47 -05:00
e1d238b75f Remove early return during scan of BME address for BMP sensors (#9935)
* Enable pre-hop drop handling by default

* Remove early break if BME/DPS sensors are not detected at the BME address

* revert sneaky change

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-03-19 07:13:14 -05:00
c88b802e32 Remove early return during scan of BME address for BMP sensors (#9935)
* Enable pre-hop drop handling by default

* Remove early break if BME/DPS sensors are not detected at the BME address

* revert sneaky change

* Potential fix for pull request finding

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

---------

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
2026-03-19 07:12:50 -05:00
TomandBen Meadors 58fee80b30 Add spoof detection for UDP packets in UdpMulticastHandler (#9905)
* Add spoof detection for UDP packets in UdpMulticastHandler

* Implement isFromUs function for packet origin validation

* ampersand
2026-03-19 07:09:54 -05:00
TomandGitHub 4890f7084f Add spoof detection for UDP packets in UdpMulticastHandler (#9905)
* Add spoof detection for UDP packets in UdpMulticastHandler

* Implement isFromUs function for packet origin validation

* ampersand
2026-03-14 19:34:19 -05:00
b7bf251798 Scaling tweaks (#9653)
* refactor: update throttling factor calculation and add unit tests for scaling behavior

* refactor: adjust throttling factor calculation for improved accuracy in different configurations

* Update src/mesh/Default.h

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update src/mesh/Default.h

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* refactor: enhance throttling factor calculation and introduce pow_of_2 utility function

* refactor: improve expected ms calculation in unit tests for Default::getConfiguredOrDefaultMsScaled

* refactor: improve scaling logic for routers and sensors in computeExpectedMs function

---------

Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2026-03-04 05:59:42 -06:00
c3321771ef Xiao NRF - define suitable i2c pins for the sub-variants (#8866)
Co-authored-by: Christian Walther <cwalther@gmx.ch>
2026-02-12 20:20:54 +01:00
TomandGitHub a092f6bb22 Refactor logging in ProtobufModule to ensure message details are logged after successful decoding (#9536) 2026-02-09 19:56:48 -06:00
52fd362720 Fix gps pin defs for various NRF variants. (#9034)
* fix on nrf52_promicro

* try fix for GPS issue

* fix GPS pin assignment in variant.h

* cleared up some comments and confirmed pinouts from schematics

---------

Co-authored-by: macvenez <macvenez@gmail.com>
2025-12-27 06:50:07 -06:00
e5c3eda2a2 Fix gps pin defs for various NRF variants. (#9034)
* fix on nrf52_promicro

* try fix for GPS issue

* fix GPS pin assignment in variant.h

* cleared up some comments and confirmed pinouts from schematics

---------

Co-authored-by: macvenez <macvenez@gmail.com>
2025-12-24 12:04:28 -06:00
TomandGitHub 8060134224 promicro doesn't need these. (#8873) 2025-12-06 13:25:57 +11:00
TomandGitHub aa85fbbcc4 Promicro documentation update (#8864)
* Delete variants/nrf52840/diy/nrf52_promicro_diy_tcxo/Schematic_Pro-Micro_Pinouts.pdf

remove old file

* Add updated schematic

* Update GPS TX and RX pin definitions after swap

* Update GPS pin definitions and schematic link

Updated the schematic link to reflect GPS pin definition changes.
2025-12-04 13:35:50 -06:00
TomandGitHub 486fa74549 Actions: Remove native from build_one (#8685)
* Remove native from the build, and remove the required permissions

* Delete .github/workflows/build_one_arch.yml

Its borken and not really needed. one_target is the goal.
2025-11-25 16:18:55 -05:00
85ea22ac38 Update to Pro-micro variants (#8600)
* Update to Pro-micro variants

Schematic updated
Xtal variant removed
Extra module added to list
Extra explanation added to readme.

* Fix markdown formatting in readme.md

* Fix formatting in readme.md for RF switch section

---------

Co-authored-by: Tom <116762865+Nestpebble@users.noreply.github.com>
2025-11-18 06:31:41 -06:00
36c2178570 Update to Pro-micro variants (#8600)
* Update to Pro-micro variants

Schematic updated
Xtal variant removed
Extra module added to list
Extra explanation added to readme.

* Fix markdown formatting in readme.md

* Fix formatting in readme.md for RF switch section

---------

Co-authored-by: Tom <116762865+Nestpebble@users.noreply.github.com>
2025-11-09 09:24:03 -06:00
TomandGitHub b8bfed2810 return to 45 days and put a closure message. 2025-10-14 12:37:40 +01:00
TomandGitHub 910fe911f8 Update stale_bot.yml 2025-10-13 20:12:45 +01:00
TomandGitHub 9ab9650248 Update stale_bot.yml
Extend stale period to 60 days, and added a message on stale marking.
2025-10-13 20:04:10 +01:00
TomandBen Meadors d1fd102952 Add another seeed_xiao_nrf52840_kit build environment for I2C pinout (#8036)
* Update platformio.ini

* Remove some more extraneous lines
2025-09-21 06:28:28 -05:00
TomandGitHub 5701755608 Add another seeed_xiao_nrf52840_kit build environment for I2C pinout (#8036)
* Update platformio.ini

* Remove some more extraneous lines
2025-09-21 06:27:39 -05:00
72b9a02f3e (resubmission) Manual GitHub actions to allow building one target or arch (#7997)
* Reset the modified files

* Fix some changes

* Fix some changes

* Trunk. That is all.

---------

Co-authored-by: Tom <116762865+Nestpebble@users.noreply.github.com>
2025-09-19 10:16:46 -05:00
c73fe85ec8 (resubmission) Manual GitHub actions to allow building one target or arch (#7997)
* Reset the modified files

* Fix some changes

* Fix some changes

* Trunk. That is all.

---------

Co-authored-by: Tom <116762865+Nestpebble@users.noreply.github.com>
2025-09-18 19:51:14 -05:00
22fcd102a0 (resubmission) Manual GitHub actions to allow building one target or arch (#7997)
* Reset the modified files

* Fix some changes

* Fix some changes

* Trunk. That is all.

---------

Co-authored-by: Tom <116762865+Nestpebble@users.noreply.github.com>
2025-09-16 06:41:22 -05:00
TomandGitHub 15d2ae17f8 Add note to hydra to note that the button pin has no pull-up (#6979)
Add note to hydra to note that the button pin has no pull-up. Use an external resistor or remove the `#define`.
2025-06-07 06:55:58 -05:00
TomandGitHub 62e1974d09 Add clarifying note about AHT20 also being included with AHT10 library (#6787)
* Update AHT10.h

Add clarifying note about AHT20 also being included

* Update AHT10.cpp

Add clarifying note about AHT20 also being included
2025-05-11 17:17:37 +08:00
TomandGitHub ca9bf6b31a Update XIAO_NRF_KIT RXEN Pin definition (#6717)
The XIAO NRF kit comes with a hat which has a slightly different pinout than the one supplied with the XIAO S3. At the last update, the RXEN pin was not moved with the rest.
2025-05-07 21:20:15 -05:00
64def246ee Corrected some misinformation (#5995)
Change the module text too soon , before it had chance to reach a final conclusion.

Co-authored-by: Tom <116762865+Nestpebble@users.noreply.github.com>
2025-02-06 11:36:04 +08:00