Automatic variable hop limits based on mesh activity and size estimation (#10176)

* asdf

* Implement SphereOfInfluenceModule for traffic management and eviction tracking

* Implement Sphere of Influence module for dynamic hop limit adjustment and role-based floor

* Update SAMPLING_DENOMINATOR to improve mesh size estimation accuracy

* Add debug logging for scale factor estimation and per-hop node counts in SphereOfInfluenceModule

* Enable variable hop limits and role-based hop floors in Sphere of Influence module

* Respond to copilot review

* Disable variable hop limits and role-based hop floors in Sphere of Influence module

* Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Implement adaptive sampling for unique node ID tracking in Sphere of Influence module

* Add state persistence for Sphere of Influence module

* Enhance Sphere of Influence module with state management and adaptive sampling adjustments

* Refactor hop scaling functionality: remove SphereOfInfluenceModule and introduce HopScalingModule

- Deleted SphereOfInfluenceModule.h, consolidating its responsibilities into the new HopScalingModule.
- Added HopScalingModule.h and HopScalingModule.cpp to manage hop scaling logic, including eviction tracking and sampling-based mesh size estimation.
- Implemented methods for recording evictions and packet senders, estimating scale factors, and computing required hops based on node activity.
- Introduced state persistence for hop scaling parameters to maintain continuity across reboots.
- Enhanced thread safety and modularity by utilizing concurrency features.

* Guard out STM32. Sowwy.

* Refactor HopScalingModule: enhance sampling logic and improve state management

* Add unit tests for HopScalingModule: implement mock database and various test scenarios

* Refactor test output in HopScalingModule tests: replace printf with TEST_MESSAGE for better integration with Unity

* Refactor HopScalingModule logging: replace lastStatusMode with descriptive mode names for improved readability

* Refactor test_main.cpp: change NodeNum variable to static and improve comments for clarity

* Remove unnecessary delay in setup function for improved test performance

* Add missing include for MeshTypes in test_main.cpp

* Refactor HopScalingModule tests: enhance mesh topology scenarios and improve test clarity

* Update HopScalingModule tests: flesh out node scenarios and improve clarity for dense and sparse mesh cases

* Fix politeness factor calculations in HopScalingModule and update related test scenarios for clarity. Remove outdated design doc.

* Enhance HopScalingModule: add sampled estimate for scaling decisions and refactor initial run state management

* Add sample traffic injection for HopScaling tests to enhance sampledEst visibility

* Enhance HopScalingModule: adjust windowFraction calculation for early triggers and improve test output formatting

* Enhance HopScalingModule: add jitter functionality to sampling denominator and update tests for consistent behavior

* Enhance HopScalingModule: implement adaptive sampling denominator adjustment and add reset functionality for tests

* Enhance HopScalingTestShim: add test-only clock and window helpers, update injectSampleTraffic for adaptive sampling, and improve scenario summary output

* Enhance HopScalingModule: add detailed documentation for functions, improve clarity of jitter and sampling logic, and reset functionality in tests

* Enhance HopScalingModule: add evictionEstimate parameter to estimateScaleFactor and update related logging for improved mesh size estimation

* Enhance HopScalingModule: adjust effective rolls calculation for improved accuracy, add eviction estimate logic, responding to all copilot review points

* Implement CompactHistogram for parallel hop scaling sampling

- Added CompactHistogram class to track node hop distances with bitwise sampling.
- Integrated CompactHistogram into HopScalingModule for independent packet sampling.
- Updated NodeDB to feed both the hop scaling module and the new histogram sampler.
- Enhanced HopScalingModule with methods to sample packets for the histogram and retrieve hop distribution statistics.
- Implemented tests for CompactHistogram functionality, including sampling, window rolling, and adaptive denominator scaling.
- Updated existing tests to validate the integration of the new histogram sampling mechanism.

* Enhance CompactHistogram and HopScalingModule: add per-hop distribution functionality, improve time handling for unit tests, and refine test setup for deterministic behavior

* CompactHistogram: add mesh size estimation, improve entry replacement logic, and update logging for per-hop distribution

* Refactor HopScalingModule and CompactHistogram integration

- Removed the suggestedHopFromCompactHistogram function to streamline hop suggestion logic.
- Updated HopScalingModule to directly utilize CompactHistogram's internal methods for hop suggestions and sampling.
- Enhanced logging in HopScalingModule to provide detailed histogram statistics.
- Modified test cases to ensure comprehensive coverage of new histogram behaviors and sampling logic.
- Improved node ID distribution in tests to better exercise sampling mechanisms.
- Ensured that filtering denominators are held for 12 hours before dropping, enhancing stability in sampling.

* Refactor CompactHistogram to support 13-hour activity tracking and introduce politeness regimes

- Updated the bitfield structure to accommodate 13-hour seen tracking.
- Changed the logic in rollHour() to analyze activity over the last 0-2 hours vs. 1-3 hours for politeness factor calculation.
- Introduced three politeness levels: GENEROUS, DEFAULT, and STRICT based on recent activity ratios.
- Adjusted filtering and sampling logic to reflect the new 13-hour tracking period.
- Updated unit tests to validate new behavior and ensure proper functionality of politeness regimes.

* Enhance CompactHistogram and HopScalingModule for improved sampling and decision-making

- Introduced a session-specific hash seed in CompactHistogram to reduce bias in node ID sampling.
- Updated sampling logic to use hashed node IDs instead of raw IDs for filtering and entry management.
- Added histogram rollover tracking in HopScalingModule to ensure proper decision-making after initial data collection.
- Adjusted logging to reflect the active state of the histogram and its comparison with NodeDB advisory hops.
- Enhanced unit tests to validate new sampling logic and memory layout changes.

* Expose hashNodeId for testing in CompactHistogram

* Add mesh trend statistics to CompactHistogram for enhanced activity tracking

* Implement histogram state persistence in CompactHistogram with save and load functions

* Refactor CompactHistogram to improve entry management and enhance rollHour logging

* feat: add HopScalingModule for adaptive hop limit recommendations

Introduces HopScalingModule, a sampled hop-distance histogram that
recommends the minimum hop limit needed to reach ~40 nodes, and
automatically reducing the hops as the mesh grows.

Key design:
- 512-byte packed histogram (128 × 4-byte Record entries) embedded
   in a new HopScalingModule.
- Each Record: 16-bit node hash, 3-bit hop distance, 13-bit seen bitmap
- Sampling filter: only nodes where (hash & (denom-1)) == 0 are kept;
  denominator doubles on overflow and halves when utilisation is low
- Hourly rollHour(): tallies per-hop counts, walks scaled buckets to
  find the minimum hop satisfying TARGET_AFFECTED_NODES (40), applies a
  politeness extension based on recent/older activity ratio, shifts all
  seen bitmaps, and persists state to /prefs/hopScalingState.bin
- Hop recommendation gated by bootstrap (requires >=1 rollHour before
  overriding HOP_MAX)
- NodeDB calls samplePacketForHistogram() on every non-MQTT rx packet
- Module also estimates total mesh size and logs useful information about
  mesh characteristics.

Changes:
- src/modules/HopScalingModule.h/.cpp: new module
- src/mesh/NodeDB.cpp: wire up samplePacketForHistogram
- src/mesh/Router.cpp: consume getLastRequiredHop()
- test/test_hop_scaling/: 12-test suite covering all mesh topologies and
  anticipated operational requirements

* test: increase run iterations in sparse to dense transition test

* feat: refactor HopScalingModule to use RUNS_PER_HOUR constant and improve logging

* feat: enhance HopScalingModule with filtering denominator management and add tests for state transitions

* refactor: remove CompactHistogram module and related files

* address copilot review comments

* Tweak: packet sampling only lora

* ove role-based hop floor logic and related definitions into the module - keep it in one place.

* Refactor MockNodeDB to use nodeInfoLiteSetBit for MQTT flag setting

* Refactor hop scaling parameters and logic to integer maths and put default values in defaults.h. Small flash size reduction, no functional impact.

* Update unit test preprocessor directives to PIO_UNIT_TESTING for consistency

* refactor: improve test output organization and clarity in hop scaling tests

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

mcp-server/run-tests.sh

@@ -191,10 +191,12 @@ echo
 # PASS/FAIL — every hardware test would SKIP with "role not present". We
 # narrow to tests/unit explicitly so the summary reads as "no hardware,
 # unit suite only" instead of "big skip count looks suspicious".
+# Keep terminal output condensed (`-q -r fE`) so skip-heavy runs do not print
+# each skipped test in full; skip counts still appear in pytest's summary.
 if [[ -z $DETECTED && $# -eq 0 ]]; then
 	echo "[pre-flight] no supported devices detected; running unit tier only."
 	echo
-	exec "$VENV_PY" -m pytest tests/unit -v --report-log=tests/reportlog.jsonl
+	exec "$VENV_PY" -m pytest tests/unit -q -r fE --report-log=tests/reportlog.jsonl
 fi
 
 # Default pytest args when the user passed none. Power users can invoke
@@ -210,11 +212,13 @@ fi
 # skipping half the hardware tests with "not baked with session profile"
 # errors. Power users who know their hardware is current and want to shave
 # those seconds can pass `--assume-baked` explicitly.
+# Defaults also use condensed reporting (`-q -r fE`) to avoid listing every
+# skipped test verbatim while still surfacing failures/errors and summary data.
 if [[ $# -eq 0 ]]; then
 	set -- tests/ \
 		--html=tests/report.html --self-contained-html \
 		--junitxml=tests/junit.xml \
-		-v --tb=short
+		-q -r fE --tb=short
 fi
 
 # UI tier requires opencv-python-headless (and ideally easyocr). If it's