Pr1.5 tmm nexthop (#10745)
* TrafficManagement: flat unified cache + persistent next-hop overflow store Reworks the TrafficManagementModule cache layer (policing behaviour unchanged from upstream) and adds a routing-hint overflow store: - Flatten the ring: replace the cuckoo-hashed unified cache and the bucketed PSRAM NodeInfo index with plain flat arrays + linear scan (same idiom as WarmNodeStore). At LoRa packet rates an O(n) scan of the cache is negligible, and it removes a large amount of hashing/displacement complexity. The cache entry is 11 B; timestamps use a uniform +1 presence-offset so a 0 byte always means "empty" across every sub-store. Adds rebaseEpoch() so cached state survives the ~19 h relative-timestamp horizon instead of being flushed. - Next-hop overflow cache: setNextHop/getNextHopHint store a confirmed last-byte relay for a destination, written only from NextHopRouter's ACK-confirmed decision (and mirrored from TraceRoute). NextHopRouter::getNextHop falls back to this cache when the hot NodeDB has no hint, so DMs/relays to long-tail nodes keep routing after the node ages out of NodeInfoLite. - Persistence: preloadNextHopsFromNodeDB warm-starts the cache from persisted NodeInfoLite hints on first maintenance pass; next_hop entries are kept alive across the maintenance sweep (no TTL) and never clobbered by a stale preload. All packet-policing logic (rate limit, position dedup, unknown-packet drop, NodeInfo direct response, hop exhaustion) is the existing upstream behaviour, untouched. HAS_TRAFFIC_MANAGEMENT defaults on so the module is compiled in. (see note). Tests: upstream policing suite now actually runs (adds the MeshTypes.h include that gates HAS_TRAFFIC_MANAGEMENT) plus 4 next-hop tests. Role-aware throttles, politeness, precision clamp, port-interval and mesh-radius gating — and the rate-limit >255 saturation fix — are deferred to the advanced-TMM branch. Note: default dedup movement grid moves to ~91m, which also means 1.5km required to end up with the same signature position - coarser and therefore further than before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * TrafficManagement: fix cppcheck constVariablePointer warning `node` in preloadNextHopsFromNodeDB() is never written through — mark it const to satisfy cppcheck's constVariablePointer check in CI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add multi-hop NextHop recovery tests and unit tests for routing reliability - Introduced a new test suite for multi-hop NextHop directed-message delivery and relay recovery in `test_nexthop_multihop_recovery.py`. This includes tests for end-to-end delivery and recovery after relay drop. - Implemented unit tests in `test_main.cpp` for NextHop routing reliability mitigations, covering: - M1: Ambiguity-aware last-byte resolution. - M2: NextHopRouter's strict-neighbor gate and hop limit checks. - M3: Route-health freshness and failure decay. - Enhanced mock classes to facilitate controlled testing of node behaviors and routing logic. * grafting fixed * Address Copilot review for PR #10735 (NextHop improvements) - docs/nexthop-routing-reliability.md: update status from "no code changes yet" to reflect that mitigations and tests are implemented RAM pressure and MIGRATION_VERBOSE concerns addressed upstream in PR2.5 (per-platform TRAFFIC_MANAGEMENT_CACHE_SIZE) and PR2 (verbose default=0) respectively; (0,0) sentinel fixed in PR2.5. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * CI: fix cppcheck constVariablePointer and test include path - NextHopRouter.cpp: qualify two RouteHealth *h locals as const — only read for stale-route checks, never mutated through the pointer - Router.cpp: qualify meshtastic_NodeInfoLite *node as const in shouldDecrementHopLimit — only read for favorite/role predicate - test_position_module/test_main.cpp: change bare PositionModule.h to modules/PositionModule.h — build_flags sets -Isrc, not -Isrc/modules, so the bare form fails to resolve in the native PlatformIO test env Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * WarmStore: cache device role + protected category in last_heard low bits Steal the low 6 bits of WarmNodeEntry.last_heard to carry an evicted node's device role (4 bits) and a protected category (2 bits) for the hop-trim path, at zero record-size cost (entry stays 40 B; no RAM/flash growth). The high bits remain a real unix-seconds timestamp, quantised to 64 s — ample for warm LRU ordering of long-tail nodes. - absorb() packs role/protectedCat; place()/ring replay store the raw word so metadata round-trips through flash. LRU compares masked time (warmTimeOf). - take() rehydration masks the metadata bits and restores the cached role so a re-admitted node isn't stuck at CLIENT until its next NodeInfo. - NodeDB classifies the category (favorite/ignored/verified -> Flag; tracker/sensor/tak_tracker -> Role) at each eviction site. - WarmNodeStore::lookupMeta() exposes role/category to consumers. - Bump WARM_RING_MAGIC (WRNG->WRN2): old rings read as erased and rebuild; warm data is a non-critical evictee cache, so discard-on-upgrade is safe. Tests: test_warm_store 11/11 (new meta round-trip + quantisation-aware ordering); NodeDB compiles (test_nodedb_blocked 4/4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WarmStore: migrate v1 rings/files by discarding last_heard, not the data Previously the WRNG->WRN2 magic bump treated old rings as erased, discarding all warm entries — including the PKI public keys that let evicted nodes keep decrypting DMs. Instead, read v1 (WRNG / WRM1) records and keep each node's identity + public key, discarding only last_heard (its low bits would otherwise be misread as the new role/protected metadata). Records re-rank and re-learn their role on next contact. - Ring backend (nRF52840): ringReadHeader accepts both magics and reports v1 via an out-param; replay zeroes last_heard for v1 records. If the active head page is v1, force a rotation so new v2 records never land in a v1-headered page (which would discard their freshly-set role on the next load). Legacy pages convert to v2 as the ring rotates. - File backend (warm.dat): bump WARM_STORE_MAGIC WRM1->WRM2; accept WRM1, verify CRC against the stored bytes, then discard last_heard and mark dirty so the next save rewrites as v2. Tests: test_warm_store 12/12 (adds test_ws_v1_migration_discardsLastHeard: key survives, role/protected reset). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WarmStore: guard role bit-width + test eviction carries role/protected - static_assert that the device role enum still fits the 4-bit warm metadata field (WARM_ROLE_MASK); fails the build loudly if a new role is added past 15 rather than silently truncating role on eviction. (Max role today = 12.) - Add test_migration_carriesRoleAndProtectedIntoWarm: a demoted TRACKER lands in the warm tier with its key, role=TRACKER and protected category=Role; a demoted CLIENT carries role=CLIENT/None. Exercises the NodeDB eviction path + warmProtectedCategory classification (the warm-store unit tests only cover absorb() directly). Tests: test_nodedb_blocked 5/5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix copilot comments * fix(test): restore #if HAS_TRAFFIC_MANAGEMENT guard in TMM test The rebase onto PR1.5 lost the top-level HAS_TRAFFIC_MANAGEMENT guard that PR1.5 introduced, leaving the #else/#endif tail orphaned and causing compile errors on non-TMM builds. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
This commit is contained in:
Tom
co-authored by
GitHub
Claude Opus 4.8
Ben Meadors
GitHub
Claude Opus 4.8
Ben Meadors
parent
ca7d82629d
commit
22072c5f4b
+121
-5
@@ -1145,6 +1145,20 @@ void NodeDB::initConfigIntervals()
|
||||
#endif
|
||||
}
|
||||
|
||||
// Always-on traffic management defaults. Only booleans are written; every
|
||||
// numeric field stays 0 and resolves to its default_traffic_mgmt_* macro at
|
||||
// use (e.g. position dedup precision/interval), so fork-wide tuning changes
|
||||
// take effect without another migration. Rate limiting and the features that
|
||||
// exhaust or reshape relayed traffic (exhaust_hop_*, drop_unknown_enabled,
|
||||
// nodeinfo_direct_response) stay opt-in.
|
||||
static void installTrafficManagementDefaults(meshtastic_LocalModuleConfig &mc)
|
||||
{
|
||||
mc.has_traffic_management = true;
|
||||
mc.traffic_management = meshtastic_ModuleConfig_TrafficManagementConfig_init_zero;
|
||||
mc.traffic_management.enabled = true;
|
||||
mc.traffic_management.position_dedup_enabled = true;
|
||||
}
|
||||
|
||||
void NodeDB::installDefaultModuleConfig()
|
||||
{
|
||||
LOG_INFO("Install default ModuleConfig");
|
||||
@@ -1262,6 +1276,8 @@ void NodeDB::installDefaultModuleConfig()
|
||||
moduleConfig.has_neighbor_info = true;
|
||||
moduleConfig.neighbor_info.enabled = false;
|
||||
|
||||
installTrafficManagementDefaults(moduleConfig);
|
||||
|
||||
moduleConfig.has_detection_sensor = true;
|
||||
moduleConfig.detection_sensor.enabled = false;
|
||||
moduleConfig.detection_sensor.detection_trigger_type = meshtastic_ModuleConfig_DetectionSensorConfig_TriggerType_LOGIC_HIGH;
|
||||
@@ -1613,6 +1629,25 @@ bool NodeDB::enforceSatelliteCaps()
|
||||
return trimmedAny;
|
||||
}
|
||||
|
||||
// Classify an evicted node's hop-protected category for the warm tier. Favorite/ignored/
|
||||
// verified are local flags (rarely reach warm — they're eviction-protected — but classify
|
||||
// them if they do); otherwise tracker/sensor/tak_tracker are role-protected.
|
||||
static uint8_t warmProtectedCategory(const meshtastic_NodeInfoLite &n)
|
||||
{
|
||||
if (n.bitfield & (NODEINFO_BITFIELD_IS_FAVORITE_MASK | NODEINFO_BITFIELD_IS_IGNORED_MASK |
|
||||
NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK))
|
||||
return static_cast<uint8_t>(WarmProtected::Flag);
|
||||
if (IS_ONE_OF(n.role, meshtastic_Config_DeviceConfig_Role_TRACKER, meshtastic_Config_DeviceConfig_Role_SENSOR,
|
||||
meshtastic_Config_DeviceConfig_Role_TAK_TRACKER))
|
||||
return static_cast<uint8_t>(WarmProtected::Role);
|
||||
return static_cast<uint8_t>(WarmProtected::None);
|
||||
}
|
||||
|
||||
// The warm tier packs the device role into a 4-bit field (WARM_ROLE_MASK). Fail the build
|
||||
// loudly if a new role outgrows it, rather than silently truncating role on eviction.
|
||||
static_assert(_meshtastic_Config_DeviceConfig_Role_MAX <= WARM_ROLE_MASK,
|
||||
"device role no longer fits the 4-bit warm metadata field");
|
||||
|
||||
void NodeDB::cleanupMeshDB()
|
||||
{
|
||||
int newPos = 0, removed = 0;
|
||||
@@ -1639,7 +1674,7 @@ void NodeDB::cleanupMeshDB()
|
||||
// Keep any key we learned (e.g. via a DM before the NodeInfo
|
||||
// exchange completed) rather than losing it with the purge.
|
||||
if (n.public_key.size == 32)
|
||||
warmStore.absorb(gone, n.last_heard, n.public_key.bytes);
|
||||
warmStore.absorb(gone, n.last_heard, n.public_key.bytes, n.role, warmProtectedCategory(n));
|
||||
#endif
|
||||
|
||||
eraseNodeSatellites(gone);
|
||||
@@ -1822,7 +1857,8 @@ void NodeDB::demoteOldestHotNodesToWarm()
|
||||
continue;
|
||||
// Keep the public key if we have one (40 B warm record); keyless nodes
|
||||
// still get a placeholder so re-admission restores last_heard.
|
||||
warmStore.absorb(n.num, n.last_heard, n.public_key.size > 0 ? n.public_key.bytes : nullptr);
|
||||
warmStore.absorb(n.num, n.last_heard, n.public_key.size > 0 ? n.public_key.bytes : nullptr, n.role,
|
||||
warmProtectedCategory(n));
|
||||
// Demotion drops the node from the header table, so drop its satellites
|
||||
// too (the eviction chokepoint) — they'd otherwise orphan until the next
|
||||
// enforceSatelliteCaps pass.
|
||||
@@ -2226,6 +2262,16 @@ void NodeDB::loadFromDisk()
|
||||
}
|
||||
}
|
||||
|
||||
// Always-on traffic management: a device that has NEVER configured TMM
|
||||
// (has_traffic_management false — AdminModule always sets the has_ flag on
|
||||
// write, even when disabling) gets the fork defaults. Explicitly configured
|
||||
// devices keep their exact settings.
|
||||
if (!moduleConfig.has_traffic_management) {
|
||||
LOG_INFO("Traffic management never configured, installing always-on defaults");
|
||||
installTrafficManagementDefaults(moduleConfig);
|
||||
saveToDisk(SEGMENT_MODULECONFIG);
|
||||
}
|
||||
|
||||
state = loadProto(channelFileName, meshtastic_ChannelFile_size, sizeof(meshtastic_ChannelFile), &meshtastic_ChannelFile_msg,
|
||||
&channelFile);
|
||||
if (state != LoadFileResult::LOAD_SUCCESS) {
|
||||
@@ -3295,6 +3341,73 @@ meshtastic_NodeInfoLite *NodeDB::getMeshNode(NodeNum n)
|
||||
return NULL;
|
||||
}
|
||||
|
||||
ResolvedNode NodeDB::resolveLastByte(uint8_t lastByte, bool requireDirectNeighbor)
|
||||
{
|
||||
ResolvedNode result; // defaults to {None, 0}
|
||||
|
||||
// 0 is the NO_RELAY_NODE / NO_NEXT_HOP_PREFERENCE sentinel (also what MQTT-sourced packets carry
|
||||
// when hop_start==0). getLastByteOfNodeNum() never yields 0, so nothing can legitimately match.
|
||||
if (lastByte == 0)
|
||||
return result;
|
||||
|
||||
const NodeNum self = getNodeNum();
|
||||
NodeNum firstMatch = 0;
|
||||
uint8_t matches = 0;
|
||||
|
||||
for (size_t i = 0; i < numMeshNodes; i++) {
|
||||
const meshtastic_NodeInfoLite *node = &meshNodes->at(i);
|
||||
|
||||
// Candidate gate: never resolve to ourselves, the sentinels, or an ignored node.
|
||||
if (node->num == self || node->num == 0 || node->num == NODENUM_BROADCAST)
|
||||
continue;
|
||||
if (nodeInfoLiteIsIgnored(node))
|
||||
continue;
|
||||
if (getLastByteOfNodeNum(node->num) != lastByte) // cheapest discriminator last
|
||||
continue;
|
||||
|
||||
// Relevance gate: is this node a plausible relay for the requested scope?
|
||||
bool relevant;
|
||||
if (requireDirectNeighbor) {
|
||||
relevant = node->has_hops_away && node->hops_away == 0 && sinceLastSeen(node) < NEXTHOP_NEIGHBOR_FRESH_SECS;
|
||||
} else {
|
||||
const bool directNeighbor = node->has_hops_away && node->hops_away == 0;
|
||||
const bool routerRole =
|
||||
IS_ONE_OF(node->role, meshtastic_Config_DeviceConfig_Role_ROUTER, meshtastic_Config_DeviceConfig_Role_ROUTER_LATE,
|
||||
meshtastic_Config_DeviceConfig_Role_CLIENT_BASE);
|
||||
relevant = directNeighbor || nodeInfoLiteIsFavorite(node) || routerRole;
|
||||
}
|
||||
if (!relevant)
|
||||
continue;
|
||||
|
||||
if (++matches == 1) {
|
||||
firstMatch = node->num;
|
||||
} else {
|
||||
// A second relevant candidate shares this byte: ambiguous. No further scanning can
|
||||
// change that, so stop early and report the collision.
|
||||
result.status = LastByteResolution::Ambiguous;
|
||||
result.num = 0;
|
||||
return result;
|
||||
}
|
||||
}
|
||||
|
||||
if (matches == 1) {
|
||||
result.status = LastByteResolution::Unique;
|
||||
result.num = firstMatch;
|
||||
}
|
||||
return result;
|
||||
}
|
||||
|
||||
bool NodeDB::resolveUniqueLastByte(uint8_t lastByte, bool requireDirectNeighbor, NodeNum *outNum)
|
||||
{
|
||||
ResolvedNode r = resolveLastByte(lastByte, requireDirectNeighbor);
|
||||
if (r.status == LastByteResolution::Unique) {
|
||||
if (outNum)
|
||||
*outNum = r.num;
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
// returns true if the maximum number of nodes is reached or we are running low on memory
|
||||
bool NodeDB::isFull()
|
||||
{
|
||||
@@ -3365,8 +3478,8 @@ meshtastic_NodeInfoLite *NodeDB::getOrCreateMeshNode(NodeNum n)
|
||||
#if WARM_NODE_COUNT > 0
|
||||
// Demote to the warm tier so the identity (and crucially the
|
||||
// PKI key) outlives the hot-store slot.
|
||||
warmStore.absorb(evicted.num, evicted.last_heard,
|
||||
evicted.public_key.size == 32 ? evicted.public_key.bytes : NULL);
|
||||
warmStore.absorb(evicted.num, evicted.last_heard, evicted.public_key.size == 32 ? evicted.public_key.bytes : NULL,
|
||||
evicted.role, warmProtectedCategory(evicted));
|
||||
#endif
|
||||
eraseNodeSatellites(evicted.num);
|
||||
// Shove the remaining nodes down the chain
|
||||
@@ -3395,7 +3508,10 @@ meshtastic_NodeInfoLite *NodeDB::getOrCreateMeshNode(NodeNum n)
|
||||
// Re-admission: restore what the warm tier kept for this node
|
||||
WarmNodeEntry warm;
|
||||
if (warmStore.take(n, warm)) {
|
||||
lite->last_heard = warm.last_heard;
|
||||
lite->last_heard = warmTimeOf(warm); // mask off the stolen role/protected metadata bits
|
||||
// Restore the role the warm tier cached, so re-admission isn't stuck at CLIENT
|
||||
// until the next NodeInfo arrives.
|
||||
lite->role = static_cast<meshtastic_Config_DeviceConfig_Role>(warmRoleOf(warm));
|
||||
if (!memfll(warm.public_key, 0, sizeof(warm.public_key))) {
|
||||
lite->public_key.size = 32;
|
||||
memcpy(lite->public_key.bytes, warm.public_key, 32);
|
||||
|
||||
Reference in New Issue
Block a user