From de345939aff93e145c373ea07e182194f54ac866 Mon Sep 17 00:00:00 2001 From: Tom <116762865+NomDeTom@users.noreply.github.com> Date: Thu, 4 Jun 2026 11:59:49 +0100 Subject: [PATCH] Automatic variable hop limits based on mesh activity and size estimation (#10176) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * asdf * Implement SphereOfInfluenceModule for traffic management and eviction tracking * Implement Sphere of Influence module for dynamic hop limit adjustment and role-based floor * Update SAMPLING_DENOMINATOR to improve mesh size estimation accuracy * Add debug logging for scale factor estimation and per-hop node counts in SphereOfInfluenceModule * Enable variable hop limits and role-based hop floors in Sphere of Influence module * Respond to copilot review * Disable variable hop limits and role-based hop floors in Sphere of Influence module * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Implement adaptive sampling for unique node ID tracking in Sphere of Influence module * Add state persistence for Sphere of Influence module * Enhance Sphere of Influence module with state management and adaptive sampling adjustments * Refactor hop scaling functionality: remove SphereOfInfluenceModule and introduce HopScalingModule - Deleted SphereOfInfluenceModule.h, consolidating its responsibilities into the new HopScalingModule. - Added HopScalingModule.h and HopScalingModule.cpp to manage hop scaling logic, including eviction tracking and sampling-based mesh size estimation. - Implemented methods for recording evictions and packet senders, estimating scale factors, and computing required hops based on node activity. - Introduced state persistence for hop scaling parameters to maintain continuity across reboots. - Enhanced thread safety and modularity by utilizing concurrency features. * Guard out STM32. Sowwy. * Refactor HopScalingModule: enhance sampling logic and improve state management * Add unit tests for HopScalingModule: implement mock database and various test scenarios * Refactor test output in HopScalingModule tests: replace printf with TEST_MESSAGE for better integration with Unity * Refactor HopScalingModule logging: replace lastStatusMode with descriptive mode names for improved readability * Refactor test_main.cpp: change NodeNum variable to static and improve comments for clarity * Remove unnecessary delay in setup function for improved test performance * Add missing include for MeshTypes in test_main.cpp * Refactor HopScalingModule tests: enhance mesh topology scenarios and improve test clarity * Update HopScalingModule tests: flesh out node scenarios and improve clarity for dense and sparse mesh cases * Fix politeness factor calculations in HopScalingModule and update related test scenarios for clarity. Remove outdated design doc. * Enhance HopScalingModule: add sampled estimate for scaling decisions and refactor initial run state management * Add sample traffic injection for HopScaling tests to enhance sampledEst visibility * Enhance HopScalingModule: adjust windowFraction calculation for early triggers and improve test output formatting * Enhance HopScalingModule: add jitter functionality to sampling denominator and update tests for consistent behavior * Enhance HopScalingModule: implement adaptive sampling denominator adjustment and add reset functionality for tests * Enhance HopScalingTestShim: add test-only clock and window helpers, update injectSampleTraffic for adaptive sampling, and improve scenario summary output * Enhance HopScalingModule: add detailed documentation for functions, improve clarity of jitter and sampling logic, and reset functionality in tests * Enhance HopScalingModule: add evictionEstimate parameter to estimateScaleFactor and update related logging for improved mesh size estimation * Enhance HopScalingModule: adjust effective rolls calculation for improved accuracy, add eviction estimate logic, responding to all copilot review points * Implement CompactHistogram for parallel hop scaling sampling - Added CompactHistogram class to track node hop distances with bitwise sampling. - Integrated CompactHistogram into HopScalingModule for independent packet sampling. - Updated NodeDB to feed both the hop scaling module and the new histogram sampler. - Enhanced HopScalingModule with methods to sample packets for the histogram and retrieve hop distribution statistics. - Implemented tests for CompactHistogram functionality, including sampling, window rolling, and adaptive denominator scaling. - Updated existing tests to validate the integration of the new histogram sampling mechanism. * Enhance CompactHistogram and HopScalingModule: add per-hop distribution functionality, improve time handling for unit tests, and refine test setup for deterministic behavior * CompactHistogram: add mesh size estimation, improve entry replacement logic, and update logging for per-hop distribution * Refactor HopScalingModule and CompactHistogram integration - Removed the suggestedHopFromCompactHistogram function to streamline hop suggestion logic. - Updated HopScalingModule to directly utilize CompactHistogram's internal methods for hop suggestions and sampling. - Enhanced logging in HopScalingModule to provide detailed histogram statistics. - Modified test cases to ensure comprehensive coverage of new histogram behaviors and sampling logic. - Improved node ID distribution in tests to better exercise sampling mechanisms. - Ensured that filtering denominators are held for 12 hours before dropping, enhancing stability in sampling. * Refactor CompactHistogram to support 13-hour activity tracking and introduce politeness regimes - Updated the bitfield structure to accommodate 13-hour seen tracking. - Changed the logic in rollHour() to analyze activity over the last 0-2 hours vs. 1-3 hours for politeness factor calculation. - Introduced three politeness levels: GENEROUS, DEFAULT, and STRICT based on recent activity ratios. - Adjusted filtering and sampling logic to reflect the new 13-hour tracking period. - Updated unit tests to validate new behavior and ensure proper functionality of politeness regimes. * Enhance CompactHistogram and HopScalingModule for improved sampling and decision-making - Introduced a session-specific hash seed in CompactHistogram to reduce bias in node ID sampling. - Updated sampling logic to use hashed node IDs instead of raw IDs for filtering and entry management. - Added histogram rollover tracking in HopScalingModule to ensure proper decision-making after initial data collection. - Adjusted logging to reflect the active state of the histogram and its comparison with NodeDB advisory hops. - Enhanced unit tests to validate new sampling logic and memory layout changes. * Expose hashNodeId for testing in CompactHistogram * Add mesh trend statistics to CompactHistogram for enhanced activity tracking * Implement histogram state persistence in CompactHistogram with save and load functions * Refactor CompactHistogram to improve entry management and enhance rollHour logging * feat: add HopScalingModule for adaptive hop limit recommendations Introduces HopScalingModule, a sampled hop-distance histogram that recommends the minimum hop limit needed to reach ~40 nodes, and automatically reducing the hops as the mesh grows. Key design: - 512-byte packed histogram (128 × 4-byte Record entries) embedded in a new HopScalingModule. - Each Record: 16-bit node hash, 3-bit hop distance, 13-bit seen bitmap - Sampling filter: only nodes where (hash & (denom-1)) == 0 are kept; denominator doubles on overflow and halves when utilisation is low - Hourly rollHour(): tallies per-hop counts, walks scaled buckets to find the minimum hop satisfying TARGET_AFFECTED_NODES (40), applies a politeness extension based on recent/older activity ratio, shifts all seen bitmaps, and persists state to /prefs/hopScalingState.bin - Hop recommendation gated by bootstrap (requires >=1 rollHour before overriding HOP_MAX) - NodeDB calls samplePacketForHistogram() on every non-MQTT rx packet - Module also estimates total mesh size and logs useful information about mesh characteristics. Changes: - src/modules/HopScalingModule.h/.cpp: new module - src/mesh/NodeDB.cpp: wire up samplePacketForHistogram - src/mesh/Router.cpp: consume getLastRequiredHop() - test/test_hop_scaling/: 12-test suite covering all mesh topologies and anticipated operational requirements * test: increase run iterations in sparse to dense transition test * feat: refactor HopScalingModule to use RUNS_PER_HOUR constant and improve logging * feat: enhance HopScalingModule with filtering denominator management and add tests for state transitions * refactor: remove CompactHistogram module and related files * address copilot review comments * Tweak: packet sampling only lora * ove role-based hop floor logic and related definitions into the module - keep it in one place. * Refactor MockNodeDB to use nodeInfoLiteSetBit for MQTT flag setting * Refactor hop scaling parameters and logic to integer maths and put default values in defaults.h. Small flash size reduction, no functional impact. * Update unit test preprocessor directives to PIO_UNIT_TESTING for consistency * refactor: improve test output organization and clarity in hop scaling tests --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> --- mcp-server/run-tests.sh | 8 +- src/mesh/Default.h | 6 + src/mesh/NodeDB.cpp | 11 + src/mesh/Router.cpp | 27 + src/mesh/mesh-pb-constants.h | 9 + src/meshUtils.h | 6 + src/modules/HopScalingModule.cpp | 493 +++++++++++++++++++ src/modules/HopScalingModule.h | 351 +++++++++++++ src/modules/Modules.cpp | 7 + test/README.md | 23 +- test/test_hop_scaling/test_main.cpp | 738 ++++++++++++++++++++++++++++ variants/native/portduino/variant.h | 3 + 12 files changed, 1672 insertions(+), 10 deletions(-) create mode 100644 src/modules/HopScalingModule.cpp create mode 100644 src/modules/HopScalingModule.h create mode 100644 test/test_hop_scaling/test_main.cpp diff --git a/mcp-server/run-tests.sh b/mcp-server/run-tests.sh index c84a8f751..95640bffd 100755 --- a/mcp-server/run-tests.sh +++ b/mcp-server/run-tests.sh @@ -191,10 +191,12 @@ echo # PASS/FAIL — every hardware test would SKIP with "role not present". We # narrow to tests/unit explicitly so the summary reads as "no hardware, # unit suite only" instead of "big skip count looks suspicious". +# Keep terminal output condensed (`-q -r fE`) so skip-heavy runs do not print +# each skipped test in full; skip counts still appear in pytest's summary. if [[ -z $DETECTED && $# -eq 0 ]]; then echo "[pre-flight] no supported devices detected; running unit tier only." echo - exec "$VENV_PY" -m pytest tests/unit -v --report-log=tests/reportlog.jsonl + exec "$VENV_PY" -m pytest tests/unit -q -r fE --report-log=tests/reportlog.jsonl fi # Default pytest args when the user passed none. Power users can invoke @@ -210,11 +212,13 @@ fi # skipping half the hardware tests with "not baked with session profile" # errors. Power users who know their hardware is current and want to shave # those seconds can pass `--assume-baked` explicitly. +# Defaults also use condensed reporting (`-q -r fE`) to avoid listing every +# skipped test verbatim while still surfacing failures/errors and summary data. if [[ $# -eq 0 ]]; then set -- tests/ \ --html=tests/report.html --self-contained-html \ --junitxml=tests/junit.xml \ - -v --tb=short + -q -r fE --tb=short fi # UI tier requires opencv-python-headless (and ideally easyocr). If it's diff --git a/src/mesh/Default.h b/src/mesh/Default.h index b1ebf5f2f..97f87fc7b 100644 --- a/src/mesh/Default.h +++ b/src/mesh/Default.h @@ -37,6 +37,12 @@ enum class TrafficType { POSITION, TELEMETRY }; #define default_traffic_mgmt_position_precision_bits 24 // ~10m grid cells #define default_traffic_mgmt_position_min_interval_secs (ONE_DAY / 2) // 12 hours between identical positions +// Hop scaling defaults +#define default_hop_scaling_min_target_nodes 40 // walk threshold: first hop reaching this cumulative count +#define default_hop_scaling_max_target_nodes 80 // generous extension ceiling (2 × min) +#define default_hop_scaling_min_target_nodes_floor 5 // minimum allowed min_target_nodes +#define default_hop_scaling_max_target_nodes_ceiling 512 // maximum allowed max_target_nodes + #ifdef USERPREFS_RINGTONE_NAG_SECS #define default_ringtone_nag_secs USERPREFS_RINGTONE_NAG_SECS #else diff --git a/src/mesh/NodeDB.cpp b/src/mesh/NodeDB.cpp index 9c1b85dad..ac724e5e0 100644 --- a/src/mesh/NodeDB.cpp +++ b/src/mesh/NodeDB.cpp @@ -25,6 +25,9 @@ #include "mesh/generated/meshtastic/deviceonly_legacy.pb.h" #include "meshUtils.h" #include "modules/NeighborInfoModule.h" +#if HAS_VARIABLE_HOPS +#include "modules/HopScalingModule.h" +#endif #include "xmodem.h" #include #include @@ -2538,6 +2541,14 @@ void NodeDB::updateFrom(const meshtastic_MeshPacket &mp) nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_VIA_MQTT_MASK, mp.via_mqtt); // Store if we received this packet via MQTT +#if HAS_VARIABLE_HOPS + // Only sample packets that arrived over LoRa. + if (mp.transport_mechanism == meshtastic_MeshPacket_TransportMechanism_TRANSPORT_LORA && hopScalingModule) { + uint8_t hopCount = std::max(int8_t(0), getHopsAway(mp)); + hopScalingModule->samplePacketForHistogram(mp.from, hopCount); + } +#endif + // If hopStart was set and there wasn't someone messing with the limit in the middle, add hopsAway const int8_t hopsAway = getHopsAway(mp); if (hopsAway >= 0) { diff --git a/src/mesh/Router.cpp b/src/mesh/Router.cpp index 778d4c4c6..300dcb0f6 100644 --- a/src/mesh/Router.cpp +++ b/src/mesh/Router.cpp @@ -15,6 +15,9 @@ #if HAS_TRAFFIC_MANAGEMENT #include "modules/TrafficManagementModule.h" #endif +#if HAS_VARIABLE_HOPS +#include "modules/HopScalingModule.h" +#endif #if !MESHTASTIC_EXCLUDE_MQTT #include "mqtt/MQTT.h" #endif @@ -355,6 +358,30 @@ ErrorCode Router::send(meshtastic_MeshPacket *p) p->from = getFrom(p); p->relay_node = nodeDB->getLastByteOfNodeNum(getNodeNum()); // set the relayer to us + +#if HAS_VARIABLE_HOPS + // Apply HopScaling hop recommendation to routine outgoing broadcasts + if (isFromUs(p) && isBroadcast(p->to) && hopScalingModule && p->which_payload_variant == meshtastic_MeshPacket_decoded_tag) { + switch (p->decoded.portnum) { + case meshtastic_PortNum_POSITION_APP: + case meshtastic_PortNum_TELEMETRY_APP: + case meshtastic_PortNum_NODEINFO_APP: + case meshtastic_PortNum_NEIGHBORINFO_APP: { + uint8_t variableHopLimit = hopScalingModule->getLastRequiredHop(); + + // Never exceed user-configured hop_limit + if (variableHopLimit < p->hop_limit) { + LOG_DEBUG("[HOPSCALE] hop_limit %u -> %u for portnum %u", p->hop_limit, variableHopLimit, p->decoded.portnum); + p->hop_limit = variableHopLimit; + } + break; + } + default: + break; + } + } +#endif + // If we are the original transmitter, set the hop limit with which we start if (isFromUs(p)) p->hop_start = p->hop_limit; diff --git a/src/mesh/mesh-pb-constants.h b/src/mesh/mesh-pb-constants.h index 30cea9383..b4d999011 100644 --- a/src/mesh/mesh-pb-constants.h +++ b/src/mesh/mesh-pb-constants.h @@ -109,6 +109,15 @@ static inline int get_max_num_nodes() #define HAS_TRAFFIC_MANAGEMENT 0 #endif +// HopScalingModule - variable hop module: dynamically adjusts broadcast hop_limit based on mesh density +// Enable per-variant by defining HAS_VARIABLE_HOPS=1 in variant.h +#ifdef ARCH_STM32WL +#define HAS_VARIABLE_HOPS 0 +#endif +#ifndef HAS_VARIABLE_HOPS +#define HAS_VARIABLE_HOPS 1 +#endif + // Cache size for traffic management (number of nodes to track) // Can be overridden per-variant based on available memory #ifndef TRAFFIC_MANAGEMENT_CACHE_SIZE diff --git a/src/meshUtils.h b/src/meshUtils.h index 4c450b3c4..dd591f333 100644 --- a/src/meshUtils.h +++ b/src/meshUtils.h @@ -66,4 +66,10 @@ inline uint32_t pow_of_2(uint32_t n) return 1 << n; } +/// Returns true if n is a power of two (n >= 1). +template constexpr bool is_pow_of_2(T n) +{ + return n >= T(1) && (n & (n - T(1))) == T(0); +} + #define IS_ONE_OF(item, ...) isOneOf(item, sizeof((int[]){__VA_ARGS__}) / sizeof(int), __VA_ARGS__) diff --git a/src/modules/HopScalingModule.cpp b/src/modules/HopScalingModule.cpp new file mode 100644 index 000000000..f50bb92f9 --- /dev/null +++ b/src/modules/HopScalingModule.cpp @@ -0,0 +1,493 @@ +#include "HopScalingModule.h" +#include "SafeFile.h" +#include "meshUtils.h" + +#if HAS_VARIABLE_HOPS + +#include "FSCommon.h" +#include "NodeDB.h" +#include "SPILock.h" +#include "concurrency/LockGuard.h" +#include "mesh-pb-constants.h" +#include +#include +#include + +namespace +{ +// Module scheduling +constexpr uint32_t INITIAL_DELAY_MS = 30 * 1000UL; // Startup grace period before first run +constexpr uint32_t RUN_INTERVAL_MS = 5 * 60 * 1000UL; // Emit micro-summary every 5 minutes +// RUNS_PER_HOUR is a public class constant in HopScalingModule.h + +// Persistence +// Note: this only needs incrementing if the published arrangement changes. For testing purposes, or prior to widespread release, +// it can stay the same even if the internal layout changes. +constexpr uint32_t HISTOGRAM_STATE_MAGIC = 0x48535432; // 'HST2' — layout v2 +constexpr uint8_t HISTOGRAM_STATE_VERSION = 1; +constexpr const char *HISTOGRAM_STATE_FILE = "/prefs/hopScalingState.bin"; + +#pragma pack(push, 1) +struct PersistedHistogram { + uint32_t magic; + uint8_t version; + uint8_t samplingDenominator; + uint8_t filteringDenominator; + uint8_t filterDenomHoldRollsRemaining; // rollHour() calls remaining in the hold; 0 when expired/not active + uint16_t hashSeed; + Record entries[HopScalingModule::CAPACITY]; // full 512-byte array; count derived on load +}; +#pragma pack(pop) + +} // namespace + +HopScalingModule *hopScalingModule; + +// --------------------------------------------------------------------------- +// Lifecycle +// --------------------------------------------------------------------------- + +HopScalingModule::HopScalingModule() : concurrency::OSThread("HopScaling") +{ + clear(); + loadFromDisk(); + setIntervalFromNow(INITIAL_DELAY_MS); +} + +void HopScalingModule::clear() +{ + memset(entries, 0, sizeof(entries)); + count = 0; + samplingDenominator = DENOM_MIN; + filteringDenominator = DENOM_MIN; + filteringDenomHoldRollsRemaining = 0; + lastPerHopCounts = {}; + lastSuggestedHop = MAX_HOP; + lastPoliteNumer = POLITENESS_DEFAULT; + lastTrendStats = {}; + memset(denominatorHistory, DENOM_MIN, sizeof(denominatorHistory)); +#ifndef PIO_UNIT_TESTING + hashSeed = static_cast(random()); +#else + hashSeed = 0; // deterministic in unit tests +#endif +} + +// --------------------------------------------------------------------------- +// Persistence +// --------------------------------------------------------------------------- + +void HopScalingModule::saveToDisk() const +{ +#ifdef FSCom + FSCom.mkdir("/prefs"); + PersistedHistogram state{}; + state.magic = HISTOGRAM_STATE_MAGIC; + state.version = HISTOGRAM_STATE_VERSION; + state.samplingDenominator = samplingDenominator; + state.filteringDenominator = filteringDenominator; + state.filterDenomHoldRollsRemaining = filteringDenomHoldRollsRemaining; + state.hashSeed = hashSeed; + // Save all CAPACITY slots; count is reconstructed on load by scanning seenHoursAgo. + memcpy(state.entries, entries, sizeof(state.entries)); + auto file = SafeFile(HISTOGRAM_STATE_FILE, true); + const size_t written = file.write(reinterpret_cast(&state), sizeof(state)); + if (file.close() && written == sizeof(state)) { + LOG_DEBUG("[HOPSCALE] Saved: count=%u samp=1/%u filt=1/%u holdRollsRemaining=%u", count, samplingDenominator, + filteringDenominator, state.filterDenomHoldRollsRemaining); + } else { + LOG_WARN("[HOPSCALE] Failed to write %s (%u of %u bytes)", HISTOGRAM_STATE_FILE, static_cast(written), + static_cast(sizeof(state))); + } +#endif +} + +void HopScalingModule::loadFromDisk() +{ +#ifdef FSCom + concurrency::LockGuard g(spiLock); + auto file = FSCom.open(HISTOGRAM_STATE_FILE, FILE_O_READ); + if (!file) + return; + PersistedHistogram state{}; + const bool readOk = (file.read(reinterpret_cast(&state), sizeof(state)) == sizeof(state)); + file.close(); + // Validate magic, version, denom range, denom power-of-two invariant, and hold counter. + if (!readOk || state.magic != HISTOGRAM_STATE_MAGIC || state.version != HISTOGRAM_STATE_VERSION || + state.samplingDenominator < DENOM_MIN || state.samplingDenominator > DENOM_MAX || + state.filteringDenominator < state.samplingDenominator || state.filteringDenominator > DENOM_MAX || + !is_pow_of_2(state.samplingDenominator) || !is_pow_of_2(state.filteringDenominator) || + state.filterDenomHoldRollsRemaining > FILTER_DENOM_HOLD_ROLLS) { + LOG_DEBUG("[HOPSCALE] No valid persisted state (magic=%08x ver=%u samp=%u filt=%u hold=%u), starting fresh", state.magic, + state.version, state.samplingDenominator, state.filteringDenominator, state.filterDenomHoldRollsRemaining); + return; + } + // Derive count by scanning: active entries have seenHoursAgo != 0; pack them to the front. + + uint8_t restored = 0; + for (uint8_t i = 0; i < CAPACITY && restored < CAPACITY; i++) { + if (state.entries[i].seenHoursAgo != 0u) { + entries[restored++] = state.entries[i]; + } + } + count = restored; + samplingDenominator = state.samplingDenominator; + filteringDenominator = state.filteringDenominator; + filteringDenomHoldRollsRemaining = state.filterDenomHoldRollsRemaining; + // denominatorHistory can't be recovered; initialise all slots to filteringDenominator so + // the first few post-reboot scaledPerHour values use a safe (slightly conservative) multiplier. + memset(denominatorHistory, filteringDenominator, sizeof(denominatorHistory)); + hashSeed = state.hashSeed; + LOG_INFO("[HOPSCALE] Restored: count=%u samp=1/%u filt=1/%u holdRollsRemaining=%u", count, samplingDenominator, + filteringDenominator, state.filterDenomHoldRollsRemaining); +#endif +} + +// --------------------------------------------------------------------------- +// Core API +// --------------------------------------------------------------------------- + +void HopScalingModule::samplePacketForHistogram(uint32_t nodeId, uint8_t hopCount) +{ + const uint16_t hash = hashNodeId(nodeId); + + if (!passesFilter(hash, samplingDenominator)) + return; + + hopCount = std::min(hopCount, MAX_HOP); + + // Update an existing entry + Record *entry = nullptr; + for (uint8_t i = 0; i < count; i++) { + if (entries[i].nodeHash == hash) { + entry = &entries[i]; + break; + } + } + if (entry) { + entry->hops_away = hopCount; + markCurrentHour(*entry); + return; + } + + // New node: trim if necessary before allocating a slot + if (getFillPercentage() >= FILL_HIGH_PCT) { + trimIfNeeded(); + } + + if (count < CAPACITY) { + entries[count].nodeHash = hash; + entries[count].hops_away = hopCount; + entries[count].seenHoursAgo = 1u; // mark current hour + count++; + } else { + LOG_WARN("[HOPSCALE] Histogram full at samp=1/%u (DENOM_MAX=%u); dropping node hash=0x%04x; hop recommendation may be " + "skewed!!!", + samplingDenominator, DENOM_MAX, hash); + } +} + +void HopScalingModule::rollHour() +{ + // Advance denominatorHistory before the tally so each slot h holds the filteringDenominator + // that was active when seenHoursAgo bit h was set. hourlyRaw[h] is then gated per-slot by + // denominatorHistory[h], giving a correct population estimate for each historical hour even + // when filteringDenominator changes between rolls. Scale-up backfills the entire array so + // the invariant holds retroactively (see trimIfNeeded()). + for (uint8_t h = 12; h > 0; h--) + denominatorHistory[h] = denominatorHistory[h - 1]; + denominatorHistory[0] = filteringDenominator; + + // 1. Tally per-hop counts and per-slot hourly activity in one pass. + // hourlyRaw[h]: gated per-slot by denominatorHistory[h] so the raw count and its + // multiplier are always consistent, even across filteringDenominator transitions. + // counts.*: gated uniformly by the current filteringDenominator for a consistent + // population estimate used by the hop-walk recommendation (step 2). + PerHopCounts counts{}; + uint16_t hourlyRaw[13] = {}; + uint16_t trendNewThisHour = 0; + uint16_t trendReturning = 0; + uint16_t trendLapsed = 0; + uint16_t trendOlderThan4h = 0; + uint16_t trendAgingOut = 0; + for (uint8_t i = 0; i < count; i++) { + const uint16_t hash = entries[i].nodeHash; + const uint32_t seen = entries[i].seenHoursAgo; + + // Per-slot hourly activity: gate each slot by its own denominator. + for (uint8_t h = 0; h < 13; h++) { + if ((seen & (1u << h)) && passesFilter(hash, denominatorHistory[h])) + hourlyRaw[h]++; + } + + // Hop counts and trend stats: uniform current-denominator gate. + if (!passesFilter(hash, filteringDenominator)) + continue; + + if (seenInLast13h(entries[i])) { + counts.perHop[entries[i].hops_away]++; + counts.total++; + } + const bool heardThisHour = (seen & 1u) != 0u; + const bool heardLastHour = (seen & 2u) != 0u; + const bool hasOlderHistory = (seen >> 1u) != 0u; + const bool recentlySilent = (seen & 0xFu) == 0u; + if (heardThisHour && !hasOlderHistory) + trendNewThisHour++; + else if (heardThisHour && hasOlderHistory) + trendReturning++; + if (!heardThisHour && heardLastHour) + trendLapsed++; + if (recentlySilent && (seen & 0x1FF0u) != 0u) + trendOlderThan4h++; + if (seen == (1u << 12u)) + trendAgingOut++; + } + lastPerHopCounts = counts; + + // 1b. Compute politeness factor from the 0-2 h vs 1-3 h activity ratio. + { + const uint32_t recent = static_cast(hourlyRaw[0]) + hourlyRaw[1]; + const uint32_t older = static_cast(hourlyRaw[1]) + hourlyRaw[2]; + if (older > 1 && recent > 1) { + const uint32_t r = static_cast(recent) * ACTIVITY_WEIGHT_SCALE; + const uint32_t o = static_cast(older); + if (r < o * ACTIVITY_WEIGHT_GENEROUS_MAX_NUMER) + lastPoliteNumer = POLITENESS_GENEROUS; + else if (r > o * ACTIVITY_WEIGHT_STRICT_MIN_NUMER) + lastPoliteNumer = POLITENESS_STRICT; + else + lastPoliteNumer = POLITENESS_DEFAULT; + } else { + lastPoliteNumer = POLITENESS_DEFAULT; + } + } + + // 1c. Scale and cache trend stats (denominatorHistory already advanced above). + { + MeshTrendStats t{}; + for (uint8_t h = 0; h < 13; h++) { + const uint32_t s = static_cast(hourlyRaw[h]) * denominatorHistory[h]; + t.scaledPerHour[h] = static_cast(std::min(s, UINT16_MAX)); + } + auto scale = [&](uint16_t raw) -> uint16_t { + return static_cast(std::min(static_cast(raw) * filteringDenominator, UINT16_MAX)); + }; + t.newThisHour = scale(trendNewThisHour); + t.returningThisHour = scale(trendReturning); + t.lapsedSinceLastHour = scale(trendLapsed); + t.olderThan4h = scale(trendOlderThan4h); + t.agingOut = scale(trendAgingOut); + lastTrendStats = t; + } + + // 2. Walk scaled hop buckets to produce a hop-limit recommendation. + // effectiveMin: walk threshold — first hop whose cumulative count reaches this. + // effectiveMax: ceiling on the one-hop extension check with GENEROUS politeness. + const uint16_t effectiveMin = TARGET_AFFECTED_NODES; + const uint16_t effectiveMax = MAX_TARGET_NODES; + uint8_t suggested = MAX_HOP; + if (counts.total > 0) { + uint32_t cumulative = 0; + for (uint8_t hop = 0; hop <= MAX_HOP; hop++) { + cumulative += static_cast(counts.perHop[hop]) * filteringDenominator; + if (cumulative >= effectiveMin) { + suggested = hop; + break; + } + } + if (suggested < MAX_HOP) { + const uint32_t atNext = static_cast(counts.perHop[suggested + 1]) * filteringDenominator; + // politeLimit = effectiveMin + gap * politeNumer / POLITENESS_DENOM + // Multiply both sides by POLITENESS_DENOM to stay in integers. + const uint32_t gap = static_cast(effectiveMax) - static_cast(effectiveMin); + if ((cumulative + atNext) * POLITENESS_DENOM <= + static_cast(effectiveMin) * POLITENESS_DENOM + gap * lastPoliteNumer) { + suggested++; + } + } + } + lastSuggestedHop = suggested; + + // 3. Log scaled per-hop counts and recommendation. + { + uint16_t scaled[MAX_HOP + 1]; + for (uint8_t h = 0; h <= MAX_HOP; h++) { + const uint32_t s = static_cast(counts.perHop[h]) * filteringDenominator; + scaled[h] = static_cast(std::min(s, UINT16_MAX)); + } + const uint32_t scaledTotal = static_cast(counts.total) * filteringDenominator; + memcpy(lastScaledPerHop, scaled, sizeof(lastScaledPerHop)); + LOG_INFO("[HOPSCALE] rollHour: entries=%u/128 samp=1/%u filt=1/%u counted=%u est=%u suggestedHop=%u polite=%u/4", count, + samplingDenominator, filteringDenominator, counts.total, static_cast(scaledTotal), suggested, + lastPoliteNumer); + + const auto &ts = lastTrendStats; + LOG_INFO("[HOPSCALE] scaledSeenPerHour (h0=now): [%u %u %u %u %u %u %u %u %u %u %u %u %u]", ts.scaledPerHour[0], + ts.scaledPerHour[1], ts.scaledPerHour[2], ts.scaledPerHour[3], ts.scaledPerHour[4], ts.scaledPerHour[5], + ts.scaledPerHour[6], ts.scaledPerHour[7], ts.scaledPerHour[8], ts.scaledPerHour[9], ts.scaledPerHour[10], + ts.scaledPerHour[11], ts.scaledPerHour[12]); + LOG_INFO("[HOPSCALE] trend: new=%u returning=%u lapsed=%u olderThan4h=%u agingOut=%u", ts.newThisHour, + ts.returningThisHour, ts.lapsedSinceLastHour, ts.olderThan4h, ts.agingOut); + } + + // 4. Scale-down check: if fewer than FILL_LOW_PCT% of capacity pass the filteringDenominator + // gate and are active, halve samplingDenominator to admit more nodes. + // Note: during a filteringDenominator hold period, lowering samplingDenominator does not + // immediately improve counts.total (new admissions don't pass the elevated + // filteringDenominator). On a genuinely quieting mesh this check can therefore fire on + // consecutive hours, cascading samplingDenominator toward DENOM_MIN. This is intentional: + // rapid re-admission allows quick recovery if the mesh returns. The hop recommendation + // stays conservative (MAX_HOP) throughout because filteringDenominator remains elevated; + // step 5 below re-synchronises the denominators once the hold expires. + if (counts.total * 100u < static_cast(CAPACITY) * FILL_LOW_PCT) { + if (samplingDenominator > DENOM_MIN) { + samplingDenominator = static_cast(samplingDenominator / 2u); + LOG_INFO("[HOPSCALE] Scale-down: sampling denom halved to %u (filter denom=%u)", samplingDenominator, + filteringDenominator); + } + } + + // 5. Tick down the hold counter; once it reaches zero, halve filteringDenominator toward + // samplingDenominator once per rollHour() (= once per hour) rather than a single jump: + // avoids a sudden large change in the hop-walk count when samplingDenominator cascaded + // down significantly during the hold period. No new hold is placed on each step — the + // 13-roll hold already guaranteed that re-admitted nodes have full seenHoursAgo history; + // further pacing is provided naturally by the 1-step-per-hour rate. denominatorHistory + // is updated automatically by the shift at the top of rollHour(), so no backfill here. + if (filteringDenominator > samplingDenominator) { + if (filteringDenomHoldRollsRemaining > 0) + filteringDenomHoldRollsRemaining--; + if (filteringDenomHoldRollsRemaining == 0) { + const uint8_t stepped = static_cast(filteringDenominator / 2u); + filteringDenominator = (stepped > samplingDenominator) ? stepped : samplingDenominator; + LOG_INFO("[HOPSCALE] Filter denom stepped to %u (samp=1/%u)", filteringDenominator, samplingDenominator); + } + } + + // 6. Shift all seen bitmaps left by one slot (opens a fresh slot for the new hour). + for (uint8_t i = 0; i < count; i++) { + rollSeenBits(entries[i]); + } + + if (histogramRollCount < 255) + histogramRollCount++; + + saveToDisk(); +} + +// --------------------------------------------------------------------------- +// Internal helpers +// --------------------------------------------------------------------------- + +void HopScalingModule::trimIfNeeded() +{ + // Step 1: evict stale entries (not seen in any of the past 13 hours). + uint8_t newCount = 0; + for (uint8_t i = 0; i < count; i++) { + if (seenInLast13h(entries[i])) { + if (i != newCount) { + entries[newCount] = entries[i]; + } + newCount++; + } + } + count = newCount; + + // Step 2: if still too full, double the sampling denominator and remove non-matching entries. + if (getFillPercentage() >= FILL_HIGH_PCT && samplingDenominator < DENOM_MAX) { + samplingDenominator = static_cast( + std::min(static_cast(samplingDenominator) * 2u, static_cast(DENOM_MAX))); + filteringDenominator = std::max(filteringDenominator, samplingDenominator); + filteringDenomHoldRollsRemaining = FILTER_DENOM_HOLD_ROLLS; + // Raise any denominatorHistory slot that is below the new filteringDenominator. + // Slots already above it (recorded during a prior scale-up that hasn't fully stepped + // down yet) are left untouched: eviction at samplingDenominator retains exactly those + // entries, so the old higher gate remains accurate for those historical hours. + // Slots below the new value must be raised because the eviction removed entries that + // had been admitted at the looser old gate — the remaining entries represent a 1/N + // subsample where N is the new filteringDenominator, not the old smaller value. + for (uint8_t h = 0; h < 13; h++) + denominatorHistory[h] = std::max(denominatorHistory[h], filteringDenominator); + LOG_INFO("[HOPSCALE] Scale-up: samp denom doubled to %u (filt=%u)", samplingDenominator, filteringDenominator); + + newCount = 0; + for (uint8_t i = 0; i < count; i++) { + if (passesFilter(entries[i].nodeHash, samplingDenominator)) { + if (i != newCount) { + entries[newCount] = entries[i]; + } + newCount++; + } + } + count = newCount; + } +} + +void HopScalingModule::logStatusReport(bool didHourlyUpdate) const +{ + const bool histActive = (histogramRollCount > 0 && count > 0); + const auto &histCounts = lastPerHopCounts; + const uint8_t runsRemaining = didHourlyUpdate ? RUNS_PER_HOUR : (RUNS_PER_HOUR - runsSinceLastHourlyUpdate); + const uint8_t minsUntilRollover = runsRemaining * (RUN_INTERVAL_MS / (60 * 1000UL)); + + LOG_INFO("[HOPSCALE] hop=%u histActive=%u fill=%u%% samp=1/%u filt=1/%u entries=%u lastCounted=%u polite=%u/4 " + "nextRoll=%umin", + lastRequiredHop, histActive ? 1u : 0u, getFillPercentage(), samplingDenominator, filteringDenominator, count, + histCounts.total, lastPoliteNumer, minsUntilRollover); + + LOG_INFO("[HOPSCALE] nodes perHop: [%u %u %u %u %u %u %u %u]", histCounts.perHop[0], histCounts.perHop[1], + histCounts.perHop[2], histCounts.perHop[3], histCounts.perHop[4], histCounts.perHop[5], histCounts.perHop[6], + histCounts.perHop[7]); + LOG_INFO("[HOPSCALE] last scaled perHop: [%u %u %u %u %u %u %u %u]", lastScaledPerHop[0], lastScaledPerHop[1], + lastScaledPerHop[2], lastScaledPerHop[3], lastScaledPerHop[4], lastScaledPerHop[5], lastScaledPerHop[6], + lastScaledPerHop[7]); +} + +int32_t HopScalingModule::runOnce() +{ + const bool isFirstRun = !hasCompletedInitialRun; + bool didHourlyUpdate = false; + + if (isFirstRun) { + hasCompletedInitialRun = true; + runsSinceLastHourlyUpdate = 0; + didHourlyUpdate = true; + } else { + runsSinceLastHourlyUpdate++; + if (runsSinceLastHourlyUpdate >= RUNS_PER_HOUR) { + runsSinceLastHourlyUpdate = 0; + didHourlyUpdate = true; + } + } + + if (didHourlyUpdate && !isFirstRun) { + rollHour(); + } + + if (didHourlyUpdate) { + uint8_t suggested = (histogramRollCount > 0 && count > 0) ? lastSuggestedHop : HOP_MAX; + // Role-based hop floor: TRACKER/TAK_TRACKER always reach at least 2 hops, + // SENSOR reaches at least 1, so these reporting roles remain reachable even + // on a dense mesh where the histogram recommends a lower hop count. + uint8_t roleFloor = 0; + switch (config.device.role) { + case meshtastic_Config_DeviceConfig_Role_TRACKER: + case meshtastic_Config_DeviceConfig_Role_TAK_TRACKER: + roleFloor = 2; + break; + case meshtastic_Config_DeviceConfig_Role_SENSOR: + roleFloor = 1; + break; + default: + break; + } + lastRequiredHop = std::max(suggested, roleFloor); + } + + logStatusReport(didHourlyUpdate); + + return RUN_INTERVAL_MS; +} + +#endif diff --git a/src/modules/HopScalingModule.h b/src/modules/HopScalingModule.h new file mode 100644 index 000000000..ebb939ce3 --- /dev/null +++ b/src/modules/HopScalingModule.h @@ -0,0 +1,351 @@ +#pragma once + +#include "MeshTypes.h" +#include "concurrency/OSThread.h" +#include "configuration.h" +#include "mesh/Default.h" +#include "mesh/mesh-pb-constants.h" +#include +#include +#include + +#if HAS_VARIABLE_HOPS + +/** + * HopScalingModule: Sampled hop-distance histogram for mesh-aware hop limit recommendations. + * + * Memory layout: 512 bytes total (128 entries × 4 bytes/entry, no padding) + * - 16-bit XOR-fold hash of node ID + * - 3-bit hops away (0–7) + * - 13-bit hourly seen bitmap + * All three fields are packed into a single 32-bit Record; sizeof(Record) == 4. + * + * Sampling: + * - A node is added only when passesFilter(hashNodeId(nodeId), samplingDenominator), + * i.e. (hash16(nodeId) & (samplingDenominator – 1)) == 0 (hash-space subsample, not raw ID) + * - samplingDenominator starts at 1 (sample all), doubles when the list exceeds FILL_HIGH_PCT + * - filteringDenominator tracks samplingDenominator upward immediately but does not drop back + * down until FILTER_DENOM_HOLD_MS (13 h) have elapsed since the last scale-up + * + * Hourly rollover (rollHour()): + * - Summarises per-hop node counts for entries matching filteringDenominator and seen in the + * last 13 hours + * - Scales each hop bucket by filteringDenominator and walks the buckets to recommend a hop + * limit, matching the same algorithm used in HopScalingModule + * - Shifts the 13-bit seen bitmap left by one slot to open a fresh slot for the new hour; + * nodes not seen in 13 consecutive hours have all seen bits cleared (stale) + * - Checks for scale-down: if fewer than FILL_LOW_PCT of capacity pass filteringDenominator, + * samplingDenominator is halved (filteringDenominator is held until the 13-h lock expires) + * + * Thread-safety: all access is single-threaded via the main loop cooperative scheduler. + */ + +struct Record { + uint32_t nodeHash : 16; + uint32_t hops_away : 3; + uint32_t seenHoursAgo : 13; +}; +static_assert(sizeof(Record) == 4); + +class HopScalingModule : private concurrency::OSThread +{ + public: + // ----------------------------------------------------------------------- + // Capacity and memory layout + // ----------------------------------------------------------------------- + static constexpr size_t CAPACITY = 128; + static constexpr size_t ENTRY_BYTES = sizeof(Record); + static constexpr size_t TOTAL_BYTES = CAPACITY * ENTRY_BYTES; + + // Denominator limits (must be powers of 2) + static constexpr uint8_t DENOM_MIN = 1; + static constexpr uint8_t DENOM_MAX = 128; + + static constexpr uint8_t MAX_HOP = 7; + + // Fill-level thresholds (percent of CAPACITY) + static constexpr uint8_t FILL_HIGH_PCT = 80; + static constexpr uint8_t FILL_LOW_PCT = 20; + + // How long filteringDenominator is held at an elevated level before it may drop. + // + // This value is deliberately equal to the seenHoursAgo window (13 hours / 13 bits). + // Invariant: every entry that existed when a scale-up fired had seenHoursAgo != 0 at + // that moment (trimIfNeeded() evicts stale entries before doubling the denominator), + // so it remains seenInLast13h for at most 13 more rollHour() calls — exactly the + // hold duration. That means entries from the scale-up event keep counts.total above + // the scale-down threshold for the entire hold period under normal (active) mesh + // conditions. On a genuinely quieting mesh the scale-down CAN fire before the hold + // expires — each firing halves samplingDenominator but filteringDenominator stays + // elevated, so the hop recommendation correctly stays conservative (MAX_HOP) while + // the cascade runs. The cascade is bounded at DENOM_MIN (7 halvings from DENOM_MAX); + // when the hold finally expires, step 5 of rollHour() halves filteringDenominator + // once per hour (rather than jumping directly to samplingDenominator) until the two + // converge, giving the hop-walk a gradual, 1-step-per-hour descent. + static constexpr uint32_t FILTER_DENOM_HOLD_MS = 13UL * 60UL * 60UL * 1000UL; // 13 h (documentation only) + // Number of rollHour() calls the hold spans — equals the seenHoursAgo window width. + // filteringDenomHoldRollsRemaining is initialised to this value on scale-up and + // decremented once per rollHour(); step-down begins when it reaches zero. + static constexpr uint8_t FILTER_DENOM_HOLD_ROLLS = 13u; + + // Hop-walk: target cumulative affected-node count when choosing a hop limit + static constexpr uint16_t TARGET_AFFECTED_NODES = default_hop_scaling_min_target_nodes; + + // Clamp bounds enforced on min_target_nodes / max_target_nodes + static constexpr uint16_t MIN_TARGET_NODES_FLOOR = default_hop_scaling_min_target_nodes_floor; + static constexpr uint16_t MAX_TARGET_NODES_CEILING = default_hop_scaling_max_target_nodes_ceiling; + static constexpr uint16_t MAX_TARGET_NODES = default_hop_scaling_max_target_nodes; + + // Politeness factors for the one-hop extension check in the hop walk. + // Stored as integer numerators over POLITENESS_DENOM (4): + // politeLimit = min + gap * politeNumer / POLITENESS_DENOM + // STRICT → min + 25% of gap; DEFAULT → midpoint; GENEROUS → max + static constexpr uint8_t POLITENESS_DENOM = 4u; + static constexpr uint8_t POLITENESS_GENEROUS = 4u; // 4/4 = 1.00 + static constexpr uint8_t POLITENESS_DEFAULT = 2u; // 2/4 = 0.50 + static constexpr uint8_t POLITENESS_STRICT = 1u; // 1/4 = 0.25 + + // Activity weight thresholds (ratio of 0-2 h window vs 1-3 h window). + // Cross-multiply form: recent * ACTIVITY_WEIGHT_SCALE vs older * threshold_numer. + // GENEROUS if recent*10 < older*9 (ratio < 0.9); STRICT if recent*10 > older*12 (ratio > 1.2) + static constexpr uint8_t ACTIVITY_WEIGHT_SCALE = 10u; + static constexpr uint8_t ACTIVITY_WEIGHT_GENEROUS_MAX_NUMER = 9u; + static constexpr uint8_t ACTIVITY_WEIGHT_STRICT_MIN_NUMER = 12u; + + // Scheduling: number of 5-minute runOnce() ticks that make up one hourly rollover + static constexpr uint8_t RUNS_PER_HOUR = 12; + + // ----------------------------------------------------------------------- + // Types + // ----------------------------------------------------------------------- + + /// Per-hop node counts produced at each hourly rollover. + struct PerHopCounts { + uint16_t perHop[MAX_HOP + 1] = {}; + uint16_t total = 0; + }; + + /// Mesh activity trend stats produced at each hourly rollover. + /// All counts are scaled by filteringDenominator (i.e. estimated full-mesh population). + /// + /// Bitmap interpretation (before the hourly shift): bit 0 = just-completed hour, bit 12 = 12 h ago. + struct MeshTrendStats { + /// Estimated node count per hour slot (h=0 is the just-completed hour, h=12 is 12 h ago). + uint16_t scaledPerHour[13] = {}; + /// Nodes heard only this hour with no prior bitmap history — indicates new arrivals. + uint16_t newThisHour = 0; + /// Nodes heard this hour that also appeared in at least one older hour — stable regulars. + uint16_t returningThisHour = 0; + /// Nodes heard last hour but silent this hour — potential departures. + uint16_t lapsedSinceLastHour = 0; + /// Nodes absent from the last 4 hours but still present in some older hour (5–13 h) — quieting down. + uint16_t olderThan4h = 0; + /// Nodes whose only remaining history is the 13th hour (bit 12 only) — about to age out entirely. + uint16_t agingOut = 0; + }; + + // ----------------------------------------------------------------------- + // Lifecycle + // ----------------------------------------------------------------------- + + HopScalingModule(); + ~HopScalingModule() = default; + + /// Reset all entries and state. + void clear(); + + // ----------------------------------------------------------------------- + // Core API + // ----------------------------------------------------------------------- + + /// Record a received packet. + /// Adds or updates an entry when passesFilter(hashNodeId(nodeId), samplingDenominator), + /// i.e. when the 16-bit XOR-fold hash of the node ID falls in the 1/samplingDenominator + /// subsample of the hash space. This is NOT a raw nodeId modulo check. + /// Marks the current hour as seen and updates the stored hop count to the last observed value. + /// Triggers a trim pass if the list exceeds FILL_HIGH_PCT after the insertion. + void samplePacketForHistogram(uint32_t nodeId, uint8_t hopCount); + + // ----------------------------------------------------------------------- + // Accessors + // ----------------------------------------------------------------------- + + uint8_t getLastRequiredHop() const { return lastRequiredHop; } + uint8_t getEntryCount() const { return count; } + uint8_t getFillPercentage() const { return static_cast((static_cast(count) * 100u) / CAPACITY); } + uint8_t getSamplingDenominator() const { return samplingDenominator; } + uint8_t getFilteringDenominator() const { return filteringDenominator; } + float getPoliteness() const { return lastPoliteNumer / static_cast(POLITENESS_DENOM); } + const PerHopCounts &getLastPerHopCounts() const { return lastPerHopCounts; } + uint8_t getLastSuggestedHop() const { return lastSuggestedHop; } + const MeshTrendStats &getLastTrendStats() const { return lastTrendStats; } + + // Compatibility accessors used by tests + uint8_t getCompactHistogramEntryCount() const { return getEntryCount(); } + uint8_t getCompactHistogramDenominator() const { return getSamplingDenominator(); } + uint8_t getCompactHistogramFilterDenominator() const { return getFilteringDenominator(); } + uint8_t getCompactHistogramSuggestedHop() const { return getLastSuggestedHop(); } + size_t getCompactHistogramAllSampleCount() const { return getEntryCount(); } + + /// Force both sampling and filtering denominators to a specific value. + /// Intended for unit tests that need a deterministic starting denominator. + void setSamplingDenominator(uint8_t d) + { + samplingDenominator = (d < DENOM_MIN) ? DENOM_MIN : (d > DENOM_MAX ? DENOM_MAX : d); + filteringDenominator = samplingDenominator; + filteringDenomHoldRollsRemaining = 0; + } + +#ifdef PIO_UNIT_TESTING + // Writable from tests as HopScalingModule::s_testNowMs; drives nowMs() in PIO_UNIT_TESTING builds. + inline static uint32_t s_testNowMs = 0; + /// Override the per-session hash seed. Use in tests that need a specific sampling distribution. + void setHashSeed(uint16_t seed) { hashSeed = seed; } + uint16_t getHashSeed() const { return hashSeed; } + /// Expose hashNodeId for tests that need to compute which node IDs pass a given denominator. + uint16_t hashNodeIdPublic(uint32_t nodeId) const { return hashNodeId(nodeId); } +#endif + + protected: + int32_t runOnce() override; + + private: +#ifdef PIO_UNIT_TESTING + friend class HopScalingTestShim; +#endif + + /// Perform hourly rollover. + /// 1. Tallies per-hop counts for entries matching filteringDenominator and seen in 13 h. + /// 2. Walks the scaled hop buckets and returns the recommended hop limit. + /// 3. Logs scaled per-hop counts and recommendation. + /// 4. Checks for scale-down (< FILL_LOW_PCT of capacity pass filteringDenominator). + /// 5. Decrements filteringDenomHoldRollsRemaining (if > 0); once it reaches zero, halves + /// filteringDenominator once toward samplingDenominator per rollHour() call. + /// 6. Shifts all seen bitmaps left by one hour slot. + void rollHour(); + // ----------------------------------------------------------------------- + // Persistence + // ----------------------------------------------------------------------- + + /// Persist the histogram state (entries, denominators, hold-timer) to flash. + /// No-op on platforms without a filesystem. Performs a full delete-and-rewrite of + /// the state file on each call; avoid calling more frequently than once per rollHour(). + void saveToDisk() const; + + /// Restore histogram state from flash. Safe to call even when no file exists. + /// Call once after construction, before the first rollHour(), to warm-start the + /// histogram across reboots without waiting 13 hours for data to re-accumulate. + /// The restored entries are available immediately for sampling, but the first + /// rollHour() (triggered by the second runOnce() tick) is needed before a warm-start + /// recommendation replaces the HOP_MAX boot default. + void loadFromDisk(); + + /// Remove stale entries (seen-bits all zero) and, if the list is still crowded, + /// double samplingDenominator and filteringDenominator and remove non-matching entries. + void trimIfNeeded(); + + void logStatusReport(bool didHourlyUpdate) const; + + // ----------------------------------------------------------------------- + // Histogram storage + // ----------------------------------------------------------------------- + Record entries[CAPACITY] = {}; + uint8_t count = 0; + + // ----------------------------------------------------------------------- + // Denominator state + // + // Two separate denominators control two distinct gates: + // + // samplingDenominator — admission gate. A node is added/updated only when + // passesFilter(hash, samplingDenominator). Lower value = more permissive = + // more nodes enter = represents recent mesh state. + // + // filteringDenominator — counting gate. The hop-walk tally in rollHour() only + // counts entries that pass passesFilter(hash, filteringDenominator). It moves + // up with samplingDenominator immediately (scale-up) but is held at the + // elevated value for FILTER_DENOM_HOLD_MS (13 h) after any scale-up before it + // may drop back down (scale-down). + // + // Why the estimate is invariant: passesFilter uses a hash-based uniform subsample. + // For any two powers-of-two denominators D ≤ F, the fraction of D-sampled entries + // that also pass F is exactly D/F. Therefore: + // raw_count × F = (total × D/F) × F = total × D + // The population estimate is the same whether we count with D or with F. + // The hold period is not about accuracy — it is about stability: it prevents the + // hop recommendation from reacting to recently-admitted nodes that have not yet + // accumulated enough seenHoursAgo history to be statistically reliable. + // + // denominatorHistory[h] — the filteringDenominator used to both gate and scale + // hourlyRaw[h]. Invariant: denominatorHistory[h] always equals the + // filteringDenominator that was active when seenHoursAgo bit h was set. + // rollHour() advances the array at the very start (before the tally loop), then + // gates hourlyRaw[h] per-slot by denominatorHistory[h] — each slot's raw count + // and multiplier are therefore always consistent, even when filteringDenominator + // changes between rolls (e.g. hold expiry). On scale-up (trimIfNeeded()), the + // entire array is backfilled uniformly with the new filteringDenominator to + // preserve the invariant retroactively for all 13 slots. Initialised to + // DENOM_MIN (1); scaledPerHour slots that draw from a 1 entry are unscaled — + // correct for a fresh instance with no prior history. + // ----------------------------------------------------------------------- + uint8_t samplingDenominator = DENOM_MIN; + uint8_t filteringDenominator = DENOM_MIN; + uint8_t filteringDenomHoldRollsRemaining = 0; // counts down from FILTER_DENOM_HOLD_ROLLS to 0; step-down fires at 0 + uint8_t denominatorHistory[13] = {}; + uint16_t hashSeed = 0; + + // ----------------------------------------------------------------------- + // Cached hourly results + // ----------------------------------------------------------------------- + PerHopCounts lastPerHopCounts = {}; + uint16_t lastScaledPerHop[MAX_HOP + 1] = {}; + uint8_t lastSuggestedHop = MAX_HOP; + uint8_t lastPoliteNumer = POLITENESS_DEFAULT; + MeshTrendStats lastTrendStats = {}; + + // ----------------------------------------------------------------------- + // Hop recommendation state + // ----------------------------------------------------------------------- + uint8_t lastRequiredHop = HOP_MAX; + uint8_t histogramRollCount = 0; + + // ----------------------------------------------------------------------- + // Scheduler state + // ----------------------------------------------------------------------- + bool hasCompletedInitialRun = false; + uint8_t runsSinceLastHourlyUpdate = 0; + + // ----------------------------------------------------------------------- + // Inline record helpers + // ----------------------------------------------------------------------- + + // Record field semantics: + // nodeHash → XOR-fold of full 32-bit node ID to 16 bits + // hops_away → hop distance (0–7) + // seenHoursAgo → 13-bit per-hour seen bitmap + // bit 0 = seen in the current / most-recent hour + // bit 12 = seen 12 hours ago + // Shifts left on each rollHour(); 0 means not seen in 13 h. + + /// XOR-fold + golden-ratio hash of a 32-bit node ID to 16 bits, mixed with the session seed. + /// Multiplying by floor(2^32 / φ) gives uniform avalanche; XORing the seed ensures different + /// devices (or the same device after a clear()) sample a different subset of node IDs. + /// For seed=0 the function is deterministic, which is used in PIO_UNIT_TESTING builds. + uint16_t hashNodeId(uint32_t nodeId) const { return static_cast((nodeId * 2654435761u) >> 16) ^ hashSeed; } + static bool seenInLast13h(const Record &r) { return r.seenHoursAgo != 0u; } + static void markCurrentHour(Record &r) { r.seenHoursAgo |= 1u; } + static void rollSeenBits(Record &r) { r.seenHoursAgo = (r.seenHoursAgo << 1u) & 0x1FFFu; } + static bool passesFilter(uint16_t nodeHash, uint8_t denom) { return (nodeHash & static_cast(denom - 1u)) == 0u; } + + public: + // Clock — public so tests can share the same timebase via HopScalingModule::s_testNowMs +#ifdef PIO_UNIT_TESTING + static uint32_t nowMs() { return s_testNowMs; } +#else + static uint32_t nowMs() { return millis(); } +#endif +}; + +extern HopScalingModule *hopScalingModule; + +#endif diff --git a/src/modules/Modules.cpp b/src/modules/Modules.cpp index d3ab9076d..5cc94a68f 100644 --- a/src/modules/Modules.cpp +++ b/src/modules/Modules.cpp @@ -41,6 +41,9 @@ #if HAS_TRAFFIC_MANAGEMENT && !MESHTASTIC_EXCLUDE_TRAFFIC_MANAGEMENT #include "modules/TrafficManagementModule.h" #endif +#if HAS_VARIABLE_HOPS +#include "modules/HopScalingModule.h" +#endif #include "modules/TextMessageModule.h" #if !MESHTASTIC_EXCLUDE_TRACEROUTE #include "modules/TraceRouteModule.h" @@ -131,6 +134,10 @@ void setupModules() } #endif +#if HAS_VARIABLE_HOPS + hopScalingModule = new HopScalingModule(); +#endif + #if !MESHTASTIC_EXCLUDE_ADMIN adminModule = new AdminModule(); #endif diff --git a/test/README.md b/test/README.md index 4a49b1b54..05d6878f5 100644 --- a/test/README.md +++ b/test/README.md @@ -200,7 +200,7 @@ class MockNodeDB : public NodeDB node.num = num; node.has_hops_away = hasHops; node.hops_away = hopsAway; - node.via_mqtt = viaMqtt; + nodeInfoLiteSetBit(&node, NODEINFO_BITFIELD_VIA_MQTT_MASK, viaMqtt); node.last_heard = getTime() - ageSecs; testNodes.push_back(node); meshNodes = &testNodes; @@ -358,6 +358,7 @@ PlatformIO defines `PIO_UNIT_TESTING` during `pio test` builds. Several producti - [ ] Set `nodeDB = mockNodeDB` - [ ] Delete persisted state files (`FSCom.remove(...)`) - [ ] Reset file-scope mutable globals +- [ ] Reset mock clock to a safe base value (e.g. `mockTime = ONE_HOUR_MS`) — prevents unsigned subtraction underflow in time-dependent logic - [ ] Disable randomness/jitter flags - [ ] In `tearDown`: null the global singleton pointer, restore flags @@ -375,15 +376,21 @@ A well-structured test suite follows this pattern: | Suite | Module Under Test | | ---------------------------- | ----------------------------- | +| `test_admin_radio` | Admin + LoRa region config | +| `test_atak` | ATAK integration | | `test_crypto` | CryptoEngine | -| `test_mqtt` | MQTT integration | -| `test_radio` | Radio interface | +| `test_default` | Default configuration helpers | +| `test_hop_scaling` | Hop scaling algorithm | +| `test_http_content_handler` | HTTP handling | +| `test_mac_from_string` | MAC address parsing | | `test_mesh_module` | Module framework | | `test_meshpacket_serializer` | Packet serialization | -| `test_transmit_history` | Retransmission tracking | -| `test_atak` | ATAK integration | -| `test_default` | Default configuration helpers | -| `test_http_content_handler` | HTTP handling | +| `test_mqtt` | MQTT integration | +| `test_packet_history` | Packet history tracking | +| `test_position_precision` | Position precision helpers | +| `test_radio` | Radio interface | | `test_serial` | Serial communication | -| `test_hop_scaling` | Hop scaling algorithm | | `test_traffic_management` | Traffic management | +| `test_transmit_history` | Retransmission tracking | +| `test_type_conversions` | NodeDB v25 type conversions | +| `test_utf8` | UTF-8 utilities | diff --git a/test/test_hop_scaling/test_main.cpp b/test/test_hop_scaling/test_main.cpp new file mode 100644 index 000000000..5aa135be1 --- /dev/null +++ b/test/test_hop_scaling/test_main.cpp @@ -0,0 +1,738 @@ +#include "MeshTypes.h" +#include "TestUtil.h" +#include + +#if HAS_VARIABLE_HOPS + +#include "FSCommon.h" +#include "gps/RTC.h" +#include "mesh/NodeDB.h" +#include "modules/HopScalingModule.h" +#include +#include +#include + +// Unity only shows TEST_MESSAGE output. printf goes to stdout which the runner swallows. +#define MSG_BUF_LEN 200 +#define TEST_MSG_FMT(fmt, ...) \ + do { \ + char _buf[MSG_BUF_LEN]; \ + snprintf(_buf, sizeof(_buf), fmt, __VA_ARGS__); \ + TEST_MESSAGE(_buf); \ + } while (0) + +static constexpr NodeNum kLocalNode = 0x11111111; + +// Shared mock clock — drives HopScalingModule::nowMs() +static uint32_t &mockTime = HopScalingModule::s_testNowMs; +static constexpr uint32_t ONE_HOUR_MS = 3600UL * 1000UL; + +// --------------------------------------------------------------------------- +// MockNodeDB — not used for hop decisions any more, kept for completeness +// --------------------------------------------------------------------------- +class MockNodeDB : public NodeDB +{ + public: + void clearTestNodes() + { + testNodes.clear(); + numMeshNodes = 0; + } + + void addTestNode(NodeNum num, uint8_t hopsAway, bool hasHops, uint32_t ageSecs, bool viaMqtt = false) + { + meshtastic_NodeInfoLite node = meshtastic_NodeInfoLite_init_zero; + node.num = num; + node.has_hops_away = hasHops; + node.hops_away = hopsAway; + nodeInfoLiteSetBit(&node, NODEINFO_BITFIELD_VIA_MQTT_MASK, viaMqtt); + node.last_heard = getTime() - ageSecs; + testNodes.push_back(node); + meshNodes = &testNodes; + numMeshNodes = testNodes.size(); + } + + std::vector testNodes; +}; + +// --------------------------------------------------------------------------- +// Test shim — expose protected/private members for direct invocation +// --------------------------------------------------------------------------- +class HopScalingTestShim : public HopScalingModule +{ + public: + using HopScalingModule::runOnce; + using HopScalingModule::samplePacketForHistogram; + + using HopScalingModule::getLastRequiredHop; + + // Test-only helpers (require UNIT_TEST friend access) + void rollHourTest() { rollHour(); } + void setHistogramDenominator(uint8_t d) { setSamplingDenominator(d); } + + /// Directly set denominator state, bypassing any scale-up/down logic. + /// Used by tests that need a specific pre-condition without triggering trim. + void forceFilterDenomState(uint8_t samp, uint8_t filt, uint8_t holdRolls) + { + samplingDenominator = samp; + filteringDenominator = filt; + filteringDenomHoldRollsRemaining = holdRolls; + } + uint8_t getFilteringDenomHoldRollsRemaining() const { return filteringDenomHoldRollsRemaining; } + + /// Insert an entry with an explicit hash, bypassing the sampling filter. + /// Used to fill the histogram to a known state without depending on hashNodeId distribution. + void forceInsertEntry(uint16_t hash, uint8_t hops) + { + if (count < CAPACITY) { + entries[count].nodeHash = hash; + entries[count].hops_away = hops; + entries[count].seenHoursAgo = 1u; + count++; + } + } + + // Size introspection for test_memory_layout + static constexpr size_t sizeofSelf() { return sizeof(HopScalingModule); } +}; + +static MockNodeDB *mockNodeDB = nullptr; + +// Create deterministic IDs that produce a broad spread of 16-bit hashes. +// HopScalingModule admission uses passesFilter(hashNodeId(nodeId), denom), NOT a raw nodeId +// modulo check — do not assume (nodeId & (denom-1)) == 0 determines whether a node is admitted. +static uint32_t makeDistributedNodeId(uint32_t baseId, uint32_t ordinal, uint32_t salt = 0) +{ + return baseId + salt + (ordinal * 33u); +} + +// --------------------------------------------------------------------------- +// Helpers — mesh topology builders +// --------------------------------------------------------------------------- + +// Helper: add N nodes at a given hop with ages spread across a time range. +static void addNodesAtHop(uint32_t baseId, uint8_t hop, uint32_t count, uint32_t ageSecs, uint32_t stride = 10) +{ + for (uint32_t i = 0; i < count; i++) { + const uint32_t nodeId = makeDistributedNodeId(baseId, i, static_cast(hop) << 8); + mockNodeDB->addTestNode(nodeId, hop, true, ageSecs + i * stride); + } +} + +// Feed sampled traffic into the histogram. +// Advances mock clock by one hour per roll and calls rollHour() so each roll produces data. +static void injectSampleTraffic(HopScalingTestShim &shim, uint32_t baseId, const uint16_t hopDist[HOP_MAX + 1], + uint8_t numRolls = 16) +{ + shim.setHistogramDenominator(HopScalingModule::DENOM_MIN); + + for (uint8_t roll = 0; roll < numRolls; ++roll) { + mockTime += ONE_HOUR_MS; + + uint16_t ordinal = 0; + for (uint8_t hop = 0; hop <= HOP_MAX; ++hop) { + for (uint16_t n = 0; n < hopDist[hop]; ++n) { + const uint32_t nodeId = makeDistributedNodeId(baseId, ordinal); + shim.samplePacketForHistogram(nodeId, hop); + ++ordinal; + } + } + shim.rollHourTest(); + } +} + +static void assertCompactHistogramActive(HopScalingTestShim &shim) +{ + TEST_ASSERT_GREATER_THAN_UINT8(0, shim.getCompactHistogramEntryCount()); + TEST_ASSERT_TRUE(shim.getCompactHistogramAllSampleCount() > 0); +} + +// --------------------------------------------------------------------------- +// Topology builders +// --------------------------------------------------------------------------- + +// Scenario A: Dense local mesh — 110 nodes, heavy at hops 0–2. +static void buildDenseLocalMesh() +{ + mockNodeDB->clearTestNodes(); + addNodesAtHop(0x1000, 0, 25, 120); + addNodesAtHop(0x2000, 1, 30, 300); + addNodesAtHop(0x3000, 2, 15, 600); + addNodesAtHop(0x4000, 3, 5, 1200); + addNodesAtHop(0x5000, 4, 10, 1800); + addNodesAtHop(0x6000, 5, 15, 2400); + addNodesAtHop(0x7000, 6, 10, 3000); +} + +// Scenario B: Spread sparse mesh — 76 nodes across hops 0–7. +static void buildSpreadSparseMesh() +{ + mockNodeDB->clearTestNodes(); + addNodesAtHop(0x1000, 0, 5, 120); + addNodesAtHop(0x2000, 1, 8, 300); + addNodesAtHop(0x3000, 2, 12, 600); + addNodesAtHop(0x4000, 3, 15, 900); + addNodesAtHop(0x5000, 4, 10, 1200); + addNodesAtHop(0x6000, 5, 6, 1800); + addNodesAtHop(0x7000, 6, 10, 3000); + addNodesAtHop(0x8000, 7, 10, 3600); +} + +// Scenario C: Deep linear chain — 22 thin nodes, never reaches 40. +static void buildDeepLinearChain() +{ + mockNodeDB->clearTestNodes(); + addNodesAtHop(0x1000, 0, 2, 120); + addNodesAtHop(0x2000, 1, 3, 300); + addNodesAtHop(0x3000, 2, 3, 600); + addNodesAtHop(0x4000, 3, 4, 900); + addNodesAtHop(0x5000, 4, 3, 1200); + addNodesAtHop(0x6000, 5, 2, 1800); + addNodesAtHop(0x7000, 6, 2, 2400); + addNodesAtHop(0x8000, 7, 3, 3600); +} + +// Scenario D: Router cluster — 71 nodes, 45 at hop 2. +static void buildRouterCluster() +{ + mockNodeDB->clearTestNodes(); + addNodesAtHop(0x1000, 0, 3, 120); + addNodesAtHop(0x2000, 1, 5, 300); + addNodesAtHop(0x3000, 2, 45, 600); + addNodesAtHop(0x4000, 3, 8, 1200); + addNodesAtHop(0x5000, 4, 3, 1200); + addNodesAtHop(0x6000, 5, 2, 1800); + addNodesAtHop(0x7000, 6, 2, 2400); + addNodesAtHop(0x8000, 7, 3, 3600); +} + +// Scenario E: Megamesh — 199 nodes (DB near capacity). +static void buildMegamesh() +{ + mockNodeDB->clearTestNodes(); + addNodesAtHop(0x01000, 0, 30, 120); + addNodesAtHop(0x02000, 1, 40, 300); + addNodesAtHop(0x03000, 2, 35, 600); + addNodesAtHop(0x04000, 3, 30, 900); + addNodesAtHop(0x05000, 4, 20, 1200); + addNodesAtHop(0x06000, 5, 15, 1800); + addNodesAtHop(0x07000, 6, 14, 2400); + addNodesAtHop(0x08000, 7, 15, 3600); +} + +// --------------------------------------------------------------------------- +// Tests — Topology-driven hop reduction scenarios +// --------------------------------------------------------------------------- + +void test_dense_local_telemetry() +{ + TEST_MESSAGE("=== Dense local mesh: telemetry broadcast ==="); + TEST_MESSAGE("Topology: 110 nodes with 25/30/15 nodes at hops 0/1/2 and a thinner tail to hop 6."); + TEST_MESSAGE("Expectation: cumulative reaches 55 nodes by hop 1, result stays tightly constrained."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + buildDenseLocalMesh(); + const uint16_t distA[HOP_MAX + 1] = {25, 30, 15, 5, 10, 15, 10, 0}; + injectSampleTraffic(*shim, 0x91000000, distA); + shim->runOnce(); + + TEST_MSG_FMT("Dense local: hop=%u", shim->getLastRequiredHop()); + TEST_ASSERT_TRUE(shim->getLastRequiredHop() <= 3); + TEST_ASSERT_TRUE(shim->getLastRequiredHop() >= 1); + assertCompactHistogramActive(*shim); + + hopScalingModule = nullptr; +} + +void test_spread_sparse_position() +{ + TEST_MESSAGE("=== Spread sparse mesh: position broadcast ==="); + TEST_MESSAGE("Topology: 76 nodes spread across all hops, reaching 40 nodes only when hop 3 is included."); + TEST_MESSAGE("Expectation: hop settles in the 3-5 range."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + buildSpreadSparseMesh(); + const uint16_t distB[HOP_MAX + 1] = {5, 8, 12, 15, 10, 6, 10, 10}; + injectSampleTraffic(*shim, 0x92000000, distB); + shim->runOnce(); + + TEST_MSG_FMT("Spread sparse: hop=%u", shim->getLastRequiredHop()); + TEST_ASSERT_TRUE(shim->getLastRequiredHop() >= 3); + TEST_ASSERT_TRUE(shim->getLastRequiredHop() <= 5); + assertCompactHistogramActive(*shim); + + hopScalingModule = nullptr; +} + +void test_deep_chain_position() +{ + TEST_MESSAGE("=== Deep linear chain: position broadcast ==="); + TEST_MESSAGE("Topology: 22 nodes spread thinly across hops 0-7, never reaching the 40-node floor."); + TEST_MESSAGE("Expectation: module must keep HOP_MAX."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + buildDeepLinearChain(); + const uint16_t distC[HOP_MAX + 1] = {2, 3, 3, 4, 3, 2, 2, 3}; + injectSampleTraffic(*shim, 0x93000000, distC); + shim->runOnce(); + + TEST_MSG_FMT("Deep chain: hop=%u", shim->getLastRequiredHop()); + TEST_ASSERT_EQUAL_UINT8(HOP_MAX, shim->getLastRequiredHop()); + assertCompactHistogramActive(*shim); + + hopScalingModule = nullptr; +} + +void test_router_cluster_telemetry() +{ + TEST_MESSAGE("=== Router cluster: telemetry broadcast ==="); + TEST_MESSAGE("Topology: 71 nodes with a concentrated 45-node cluster at hop 2."); + TEST_MESSAGE("Expectation: result stays in the 2-4 range."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + buildRouterCluster(); + const uint16_t distD[HOP_MAX + 1] = {3, 5, 45, 8, 3, 2, 2, 3}; + injectSampleTraffic(*shim, 0x94000000, distD); + shim->runOnce(); + + TEST_MSG_FMT("Router cluster: hop=%u", shim->getLastRequiredHop()); + TEST_ASSERT_TRUE(shim->getLastRequiredHop() >= 2); + TEST_ASSERT_TRUE(shim->getLastRequiredHop() <= 4); + assertCompactHistogramActive(*shim); + + hopScalingModule = nullptr; +} + +void test_megamesh_eviction_scaling() +{ + TEST_MESSAGE("=== Megamesh with eviction scaling ==="); + TEST_MESSAGE("Topology: NodeDB at capacity (199 nodes), ~2000-node mesh with sustained eviction pressure."); + TEST_MESSAGE("Expectation: sustained evictions tracked in rolling average, hop stays well below HOP_MAX."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + buildMegamesh(); + + const uint16_t distE[HOP_MAX + 1] = {301, 402, 352, 301, 201, 151, 141, 151}; + injectSampleTraffic(*shim, 0x9B000000, distE); + + shim->runOnce(); + uint8_t hopBefore = shim->getLastRequiredHop(); + TEST_MSG_FMT("Megamesh initial: hop=%u", hopBefore); + + for (int hour = 0; hour < 3; hour++) { + mockTime += ONE_HOUR_MS; + { + const uint16_t megaDist[HOP_MAX + 1] = {301, 402, 352, 301, 201, 151, 141, 151}; + uint16_t ordinal = 0; + for (uint8_t hop = 0; hop <= HOP_MAX; ++hop) { + for (uint16_t n = 0; n < megaDist[hop]; ++n) { + const uint32_t nodeId = makeDistributedNodeId(0x9C000000u, ordinal, static_cast(hour) * 0x10000u); + shim->samplePacketForHistogram(nodeId, hop); + ++ordinal; + } + } + } + + for (int run = 0; run < 7; run++) + shim->runOnce(); + + TEST_MSG_FMT("Megamesh hour %d: hop=%u", hour + 1, shim->getLastRequiredHop()); + } + + TEST_MESSAGE("Assertion: hop stays well below HOP_MAX on a large-distribution mesh."); + TEST_ASSERT_TRUE(shim->getLastRequiredHop() <= 3); + assertCompactHistogramActive(*shim); + + hopScalingModule = nullptr; +} + +void test_sparse_to_dense_transition() +{ + TEST_MESSAGE("=== Sparse-to-dense transition ==="); + TEST_MESSAGE("Topology change: start with a 22-node deep chain, then inject 50 new neighbors at hops 0-1."); + TEST_MESSAGE("Expectation: hop drops sharply once the local neighborhood becomes dense."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + + buildDeepLinearChain(); + const uint16_t distC2[HOP_MAX + 1] = {2, 3, 3, 4, 3, 2, 2, 3}; + injectSampleTraffic(*shim, 0x95000000, distC2); + shim->runOnce(); + uint8_t hopSparse = shim->getLastRequiredHop(); + TEST_MSG_FMT("Phase 1 sparse: hop=%u (expect %u)", hopSparse, HOP_MAX); + TEST_ASSERT_EQUAL_UINT8(HOP_MAX, hopSparse); + + addNodesAtHop(0xA000, 0, 25, 120); + addNodesAtHop(0xB000, 1, 25, 300); + + for (uint32_t i = 0; i < 25; ++i) + shim->samplePacketForHistogram(makeDistributedNodeId(0xA000, i, static_cast(0) << 8), 0); + for (uint32_t i = 0; i < 25; ++i) + shim->samplePacketForHistogram(makeDistributedNodeId(0xB000, i, static_cast(1) << 8), 1); + + for (int run = 0; run < HopScalingModule::RUNS_PER_HOUR; run++) + shim->runOnce(); + + uint8_t hopDense = shim->getLastRequiredHop(); + TEST_MSG_FMT("Phase 2 dense: hop=%u (expect <= 3)", hopDense); + + TEST_ASSERT_TRUE(hopDense < hopSparse); + TEST_ASSERT_TRUE(hopDense <= 3); + assertCompactHistogramActive(*shim); + + hopScalingModule = nullptr; +} + +void test_state_persistence() +{ + TEST_MESSAGE("=== State persistence across restart ==="); + TEST_MESSAGE("Expectation: histogram entries survive instance teardown and reload."); + + { + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + + const uint16_t dist[HOP_MAX + 1] = {5, 8, 12, 10, 5, 3, 2, 1}; + injectSampleTraffic(*shim, 0x9D000000, dist, 2); + + TEST_MSG_FMT("Phase 1: entries=%u hop=%u", shim->getEntryCount(), shim->getLastRequiredHop()); + TEST_ASSERT_GREATER_THAN_UINT8(0, shim->getEntryCount()); + hopScalingModule = nullptr; + } + + { + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + shim->runOnce(); + + TEST_MSG_FMT("Phase 2 restored: entries=%u hop=%u", shim->getEntryCount(), shim->getLastRequiredHop()); + TEST_ASSERT_GREATER_THAN_UINT8(0, shim->getEntryCount()); + hopScalingModule = nullptr; + } +} + +void test_hourly_roll() +{ + TEST_MESSAGE("=== Hourly roll cycle ==="); + TEST_MESSAGE("Expectation: histogram accumulates data and provides valid hop recommendation after multiple rolls."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + buildSpreadSparseMesh(); + shim->setHistogramDenominator(HopScalingModule::DENOM_MIN); + + for (uint32_t i = 1; i <= 30; i++) { + const uint32_t nodeId = makeDistributedNodeId(0x97000000, i, 0xAAu); + shim->samplePacketForHistogram(nodeId, static_cast(i % (HOP_MAX + 1))); + } + + for (int run = 0; run < 13; run++) { + int32_t interval = shim->runOnce(); + TEST_ASSERT_GREATER_THAN(0, interval); + } + + TEST_MSG_FMT("Hourly roll: hop=%u entries=%u", shim->getLastRequiredHop(), shim->getEntryCount()); + assertCompactHistogramActive(*shim); + + hopScalingModule = nullptr; +} + +void test_intermediate_status() +{ + TEST_MESSAGE("=== Intermediate status (no recomputation) ==="); + TEST_MESSAGE("Expectation: runs between hourly updates leave hop unchanged."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + buildRouterCluster(); + const uint16_t distD[HOP_MAX + 1] = {3, 5, 45, 8, 3, 2, 2, 3}; + injectSampleTraffic(*shim, 0x98000000, distD); + + shim->runOnce(); + uint8_t hopAfterInitial = shim->getLastRequiredHop(); + TEST_MSG_FMT("Initial: hop=%u", hopAfterInitial); + + for (int run = 0; run < 3; run++) { + shim->runOnce(); + TEST_ASSERT_EQUAL_UINT8(hopAfterInitial, shim->getLastRequiredHop()); + } + + TEST_MSG_FMT("After 3 intermediate runs: hop=%u (unchanged)", shim->getLastRequiredHop()); + hopScalingModule = nullptr; +} + +void test_startup_blank_state() +{ + TEST_MESSAGE("=== Startup with blank state ==="); + TEST_MESSAGE("Expectation: fresh instance starts with zeroed rolling averages and a valid hop result."); + +#ifdef FSCom + FSCom.remove("/prefs/hopScalingState.bin"); +#endif + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + buildDeepLinearChain(); + + int32_t interval = shim->runOnce(); + + TEST_ASSERT_GREATER_THAN(0, interval); + TEST_ASSERT_TRUE(shim->getLastRequiredHop() <= HOP_MAX); + + TEST_MSG_FMT("Startup blank: hop=%u", shim->getLastRequiredHop()); + + hopScalingModule = nullptr; +} + +// --------------------------------------------------------------------------- +// Tests — Denominator state machine +// --------------------------------------------------------------------------- + +void test_denominator_rises_on_overflow() +{ + TEST_MESSAGE("=== samplingDenominator doubles when histogram overflows ==="); + TEST_MESSAGE("Fill to > FILL_HIGH_PCT with forceInsertEntry, then trigger via samplePacketForHistogram."); + TEST_MESSAGE("Expectation: samp/filt both double to 2, hold set to FILTER_DENOM_HOLD_ROLLS."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + + // Insert 103 entries with hashes 1..103 (all distinct, no sampling-filter skew). + // 103 / 128 = 80.4% fill, which meets FILL_HIGH_PCT=80. + // Odd hashes (1,3,...,103) will be evicted when denom doubles to 2; even ones survive. + static constexpr uint8_t FILL_COUNT = 103u; + for (uint8_t i = 1; i <= FILL_COUNT; i++) + shim->forceInsertEntry(i, 2u); + + TEST_ASSERT_EQUAL_UINT8(HopScalingModule::DENOM_MIN, shim->getSamplingDenominator()); + TEST_ASSERT_EQUAL_UINT8(HopScalingModule::DENOM_MIN, shim->getFilteringDenominator()); + TEST_ASSERT_EQUAL_UINT8(0u, shim->getFilteringDenomHoldRollsRemaining()); + TEST_ASSERT_EQUAL_UINT8(FILL_COUNT, shim->getEntryCount()); + + // A new node passes the denom=1 admission gate; fill ≥ 80% triggers trimIfNeeded → doubling. + shim->samplePacketForHistogram(0xB0000000u, 1u); + + TEST_MSG_FMT("After scale-up: samp=1/%u filt=1/%u holdRolls=%u entries=%u", shim->getSamplingDenominator(), + shim->getFilteringDenominator(), shim->getFilteringDenomHoldRollsRemaining(), shim->getEntryCount()); + + TEST_ASSERT_EQUAL_UINT8(2u, shim->getSamplingDenominator()); + TEST_ASSERT_EQUAL_UINT8(2u, shim->getFilteringDenominator()); + TEST_ASSERT_EQUAL_UINT8(HopScalingModule::FILTER_DENOM_HOLD_ROLLS, shim->getFilteringDenomHoldRollsRemaining()); + // After evicting entries with (hash & 1) != 0, roughly half the entries remain. + TEST_ASSERT_LESS_THAN_UINT8(FILL_COUNT, shim->getEntryCount()); + + hopScalingModule = nullptr; +} + +void test_filtering_denom_hold_counts_down() +{ + TEST_MESSAGE("=== filteringDenominator held while hold counter > 0 ==="); + TEST_MESSAGE("Force filt=4 samp=1 hold=3; verify no step for 2 rolls, then step fires on roll 3."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + + // samp=DENOM_MIN so scale-down in step 4 can't go lower; hold=3 for a short, fast test. + shim->forceFilterDenomState(HopScalingModule::DENOM_MIN, 4u, 3u); + + shim->rollHourTest(); // hold 3→2, no step + TEST_ASSERT_EQUAL_UINT8(4u, shim->getFilteringDenominator()); + TEST_ASSERT_EQUAL_UINT8(2u, shim->getFilteringDenomHoldRollsRemaining()); + + shim->rollHourTest(); // hold 2→1, no step + TEST_ASSERT_EQUAL_UINT8(4u, shim->getFilteringDenominator()); + TEST_ASSERT_EQUAL_UINT8(1u, shim->getFilteringDenomHoldRollsRemaining()); + + // Roll 3: hold 1→0, step fires — filteringDenominator halves to max(2, samp=1) = 2. + shim->rollHourTest(); + TEST_MSG_FMT("After hold expires: filt=1/%u samp=1/%u holdRolls=%u", shim->getFilteringDenominator(), + shim->getSamplingDenominator(), shim->getFilteringDenomHoldRollsRemaining()); + TEST_ASSERT_EQUAL_UINT8(2u, shim->getFilteringDenominator()); + TEST_ASSERT_EQUAL_UINT8(0u, shim->getFilteringDenomHoldRollsRemaining()); + + hopScalingModule = nullptr; +} + +void test_filtering_denom_steps_down_gradually() +{ + TEST_MESSAGE("=== filteringDenominator descends one halving per rollHour() after hold expires ==="); + TEST_MESSAGE("Force filt=8 samp=1 hold=1; expect 8→4→2→1 over 3 rolls, then stable."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + + shim->forceFilterDenomState(HopScalingModule::DENOM_MIN, 8u, 1u); + + shim->rollHourTest(); // hold 1→0, step: 8/2=4 > 1, filt=4 + TEST_ASSERT_EQUAL_UINT8(4u, shim->getFilteringDenominator()); + + shim->rollHourTest(); // hold=0 (no decrement), step: 4/2=2 > 1, filt=2 + TEST_ASSERT_EQUAL_UINT8(2u, shim->getFilteringDenominator()); + + shim->rollHourTest(); // step: 2/2=1, not > samp=1, filt=samp=1 — converged + TEST_ASSERT_EQUAL_UINT8(1u, shim->getFilteringDenominator()); + + shim->rollHourTest(); // filt==samp, outer if is false — no further change + TEST_ASSERT_EQUAL_UINT8(1u, shim->getFilteringDenominator()); + TEST_ASSERT_EQUAL_UINT8(HopScalingModule::DENOM_MIN, shim->getSamplingDenominator()); + + hopScalingModule = nullptr; +} + +void test_full_at_denom_max_drops_entry() +{ + TEST_MESSAGE("=== Full histogram at DENOM_MAX drops new entries ==="); + TEST_MESSAGE("Fill CAPACITY entries, force samp=DENOM_MAX, sample admissible node."); + TEST_MESSAGE("Expectation: entry count stays at CAPACITY (LOG_WARN fires; visible in test output)."); + + auto shim = std::unique_ptr(new HopScalingTestShim()); + hopScalingModule = shim.get(); + shim->setHashSeed(0); // deterministic hash for admissible-ID search + + shim->forceFilterDenomState(HopScalingModule::DENOM_MAX, HopScalingModule::DENOM_MAX, 0u); + + // Fill with odd hashes 1,3,5,...,(2*CAPACITY-1). None are multiples of 128, so none + // collide with the admissible node's hash (which must be a multiple of 128). + for (uint16_t i = 0; i < HopScalingModule::CAPACITY; i++) + shim->forceInsertEntry(static_cast(2u * i + 1u), 1u); + + TEST_ASSERT_EQUAL_UINT8(HopScalingModule::CAPACITY, shim->getEntryCount()); + + // Find a node ID whose hash passes DENOM_MAX, i.e. (hash & 127) == 0. + uint32_t admissibleId = 0; + for (uint32_t id = 1u; id < 0x10000u; id++) { + if ((shim->hashNodeIdPublic(id) & (HopScalingModule::DENOM_MAX - 1u)) == 0u) { + admissibleId = id; + break; + } + } + TEST_ASSERT_NOT_EQUAL(0u, admissibleId); // sanity: the hash space is dense enough to find one quickly + + shim->samplePacketForHistogram(admissibleId, 3u); + + TEST_MSG_FMT("After drop attempt: entries=%u CAPACITY=%u admissibleId=0x%08x hash=0x%04x", shim->getEntryCount(), + static_cast(HopScalingModule::CAPACITY), admissibleId, + static_cast(shim->hashNodeIdPublic(admissibleId))); + TEST_ASSERT_EQUAL_UINT8(HopScalingModule::CAPACITY, shim->getEntryCount()); + + hopScalingModule = nullptr; +} + +void test_scenario_summary_output() +{ + TEST_MESSAGE("=== Scenario summary ==="); + TEST_MESSAGE("Scenario | Nodes | Distribution | Hop | Why"); + TEST_MESSAGE("A: Dense local | 110 | 25/30/15/5/10/15/10 h0-6 | 1-2 | 55 nodes at h1 >> 40"); + TEST_MESSAGE("B: Spread | 76 | 5/8/12/15/10/6/10/10 h0-7 | 3-4 | Need h3 to reach 40"); + TEST_MESSAGE("C: Deep chain | 22 | 2/3/3/4/3/2/2/3 h0-7 | 7 | Never reaches 40"); + TEST_MESSAGE("D: Router | 71 | 3/5/45/8/3/2/2/3 h0-7 | 2-3 | 45-node hop-2 cluster"); + TEST_MESSAGE("E: Megamesh | 199 | 30/40/35/30/20/15/14/15 h0-7 | 0-1 | Dense low-hop histogram"); + TEST_MESSAGE("F: Transition | 22->72 | Chain -> dense local | 7-><=3 | Adapts to new neighbors"); + TEST_MESSAGE("G: Persistence | -- | -- | -- | Eviction avg survives reboot"); + TEST_MESSAGE(""); + TEST_MESSAGE("=== Denominator state machine summary ==="); + TEST_MESSAGE("Test | Pre-condition | Expectation"); + TEST_MESSAGE("H: Rises on overflow | 103 entries forced, denom=1 | samp/filt→2, holdRolls=13"); + TEST_MESSAGE( + "I: Hold counts down | filt=4 samp=1 hold=3 | no step for 2 rolls, step on roll 3: filt→2"); + TEST_MESSAGE("J: Steps down gradually | filt=8 samp=1 hold=1 | 8→4→2→1 over 3 rolls, stable on 4th"); + TEST_MESSAGE("K: Full at DENOM_MAX drops entry | 128 entries, samp=filt=128 | count stays 128, LOG_WARN emitted"); +} + +static void test_memory_layout() +{ + TEST_MSG_FMT("%-35s %6s %s", "Type", "bytes", "Notes"); + TEST_MSG_FMT("%-35s %6zu %s", "Record", sizeof(Record), "nodeHash:16 + hops:3 + seen:13 (32-bit packed)"); + TEST_MSG_FMT("%-35s %6zu %s", "HopScalingModule::PerHopCounts", sizeof(HopScalingModule::PerHopCounts), + "perHop[8](16) + total(2)"); + TEST_MSG_FMT("%-35s %6zu %s", "HopScalingModule (instance)", HopScalingTestShim::sizeofSelf(), + "entries[128](512) + denom state + cached results + OSThread overhead"); + + TEST_PASS(); +} + +// --------------------------------------------------------------------------- +// Unity setup / teardown / main +// --------------------------------------------------------------------------- + +void setUp(void) +{ + if (!mockNodeDB) + mockNodeDB = new MockNodeDB(); + mockNodeDB->clearTestNodes(); + + config = meshtastic_LocalConfig_init_zero; + moduleConfig = meshtastic_LocalModuleConfig_init_zero; + myNodeInfo.my_node_num = kLocalNode; + nodeDB = mockNodeDB; + +#ifdef FSCom + FSCom.remove("/prefs/hopScalingState.bin"); +#endif + + // Reset mock clock to a known base (1 hour in so subtraction never underflows) + mockTime = ONE_HOUR_MS; +} + +void tearDown(void) +{ + hopScalingModule = nullptr; +} + +void setup() +{ + initializeTestEnvironment(); + nodeDB = mockNodeDB; + + UNITY_BEGIN(); + + printf("\n=== Topology-driven hop reduction ===\n"); + RUN_TEST(test_dense_local_telemetry); + RUN_TEST(test_spread_sparse_position); + RUN_TEST(test_deep_chain_position); + RUN_TEST(test_router_cluster_telemetry); + RUN_TEST(test_megamesh_eviction_scaling); + RUN_TEST(test_sparse_to_dense_transition); + + printf("\n=== Lifecycle ===\n"); + RUN_TEST(test_state_persistence); + RUN_TEST(test_hourly_roll); + RUN_TEST(test_intermediate_status); + RUN_TEST(test_startup_blank_state); + + printf("\n=== Denominator state machine ===\n"); + RUN_TEST(test_denominator_rises_on_overflow); + RUN_TEST(test_filtering_denom_hold_counts_down); + RUN_TEST(test_filtering_denom_steps_down_gradually); + RUN_TEST(test_full_at_denom_max_drops_entry); + + printf("\n=== Summary ===\n"); + RUN_TEST(test_memory_layout); + RUN_TEST(test_scenario_summary_output); + + exit(UNITY_END()); +} + +void loop() {} + +#else // !HAS_VARIABLE_HOPS + +void setUp(void) {} +void tearDown(void) {} + +void setup() +{ + initializeTestEnvironment(); + UNITY_BEGIN(); + exit(UNITY_END()); +} + +void loop() {} + +#endif \ No newline at end of file diff --git a/variants/native/portduino/variant.h b/variants/native/portduino/variant.h index c23d17b8d..fb36e2588 100644 --- a/variants/native/portduino/variant.h +++ b/variants/native/portduino/variant.h @@ -13,6 +13,9 @@ #ifndef HAS_TRAFFIC_MANAGEMENT #define HAS_TRAFFIC_MANAGEMENT 1 #endif +#ifndef HAS_VARIABLE_HOPS +#define HAS_VARIABLE_HOPS 1 +#endif #ifndef TRAFFIC_MANAGEMENT_CACHE_SIZE #define TRAFFIC_MANAGEMENT_CACHE_SIZE 2048 #endif