* TrafficManagement: flat unified cache + persistent next-hop overflow store Reworks the TrafficManagementModule cache layer (policing behaviour unchanged from upstream) and adds a routing-hint overflow store: - Flatten the ring: replace the cuckoo-hashed unified cache and the bucketed PSRAM NodeInfo index with plain flat arrays + linear scan (same idiom as WarmNodeStore). At LoRa packet rates an O(n) scan of the cache is negligible, and it removes a large amount of hashing/displacement complexity. The cache entry is 11 B; timestamps use a uniform +1 presence-offset so a 0 byte always means "empty" across every sub-store. Adds rebaseEpoch() so cached state survives the ~19 h relative-timestamp horizon instead of being flushed. - Next-hop overflow cache: setNextHop/getNextHopHint store a confirmed last-byte relay for a destination, written only from NextHopRouter's ACK-confirmed decision (and mirrored from TraceRoute). NextHopRouter::getNextHop falls back to this cache when the hot NodeDB has no hint, so DMs/relays to long-tail nodes keep routing after the node ages out of NodeInfoLite. - Persistence: preloadNextHopsFromNodeDB warm-starts the cache from persisted NodeInfoLite hints on first maintenance pass; next_hop entries are kept alive across the maintenance sweep (no TTL) and never clobbered by a stale preload. All packet-policing logic (rate limit, position dedup, unknown-packet drop, NodeInfo direct response, hop exhaustion) is the existing upstream behaviour, untouched. HAS_TRAFFIC_MANAGEMENT defaults on so the module is compiled in. (see note). Tests: upstream policing suite now actually runs (adds the MeshTypes.h include that gates HAS_TRAFFIC_MANAGEMENT) plus 4 next-hop tests. Role-aware throttles, politeness, precision clamp, port-interval and mesh-radius gating — and the rate-limit >255 saturation fix — are deferred to the advanced-TMM branch. Note: default dedup movement grid moves to ~91m, which also means 1.5km required to end up with the same signature position - coarser and therefore further than before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * TrafficManagement: fix cppcheck constVariablePointer warning `node` in preloadNextHopsFromNodeDB() is never written through — mark it const to satisfy cppcheck's constVariablePointer check in CI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add multi-hop NextHop recovery tests and unit tests for routing reliability - Introduced a new test suite for multi-hop NextHop directed-message delivery and relay recovery in `test_nexthop_multihop_recovery.py`. This includes tests for end-to-end delivery and recovery after relay drop. - Implemented unit tests in `test_main.cpp` for NextHop routing reliability mitigations, covering: - M1: Ambiguity-aware last-byte resolution. - M2: NextHopRouter's strict-neighbor gate and hop limit checks. - M3: Route-health freshness and failure decay. - Enhanced mock classes to facilitate controlled testing of node behaviors and routing logic. * grafting fixed * Address Copilot review for PR #10735 (NextHop improvements) - docs/nexthop-routing-reliability.md: update status from "no code changes yet" to reflect that mitigations and tests are implemented RAM pressure and MIGRATION_VERBOSE concerns addressed upstream in PR2.5 (per-platform TRAFFIC_MANAGEMENT_CACHE_SIZE) and PR2 (verbose default=0) respectively; (0,0) sentinel fixed in PR2.5. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * CI: fix cppcheck constVariablePointer and test include path - NextHopRouter.cpp: qualify two RouteHealth *h locals as const — only read for stale-route checks, never mutated through the pointer - Router.cpp: qualify meshtastic_NodeInfoLite *node as const in shouldDecrementHopLimit — only read for favorite/role predicate - test_position_module/test_main.cpp: change bare PositionModule.h to modules/PositionModule.h — build_flags sets -Isrc, not -Isrc/modules, so the bare form fails to resolve in the native PlatformIO test env Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * WarmStore: cache device role + protected category in last_heard low bits Steal the low 6 bits of WarmNodeEntry.last_heard to carry an evicted node's device role (4 bits) and a protected category (2 bits) for the hop-trim path, at zero record-size cost (entry stays 40 B; no RAM/flash growth). The high bits remain a real unix-seconds timestamp, quantised to 64 s — ample for warm LRU ordering of long-tail nodes. - absorb() packs role/protectedCat; place()/ring replay store the raw word so metadata round-trips through flash. LRU compares masked time (warmTimeOf). - take() rehydration masks the metadata bits and restores the cached role so a re-admitted node isn't stuck at CLIENT until its next NodeInfo. - NodeDB classifies the category (favorite/ignored/verified -> Flag; tracker/sensor/tak_tracker -> Role) at each eviction site. - WarmNodeStore::lookupMeta() exposes role/category to consumers. - Bump WARM_RING_MAGIC (WRNG->WRN2): old rings read as erased and rebuild; warm data is a non-critical evictee cache, so discard-on-upgrade is safe. Tests: test_warm_store 11/11 (new meta round-trip + quantisation-aware ordering); NodeDB compiles (test_nodedb_blocked 4/4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WarmStore: migrate v1 rings/files by discarding last_heard, not the data Previously the WRNG->WRN2 magic bump treated old rings as erased, discarding all warm entries — including the PKI public keys that let evicted nodes keep decrypting DMs. Instead, read v1 (WRNG / WRM1) records and keep each node's identity + public key, discarding only last_heard (its low bits would otherwise be misread as the new role/protected metadata). Records re-rank and re-learn their role on next contact. - Ring backend (nRF52840): ringReadHeader accepts both magics and reports v1 via an out-param; replay zeroes last_heard for v1 records. If the active head page is v1, force a rotation so new v2 records never land in a v1-headered page (which would discard their freshly-set role on the next load). Legacy pages convert to v2 as the ring rotates. - File backend (warm.dat): bump WARM_STORE_MAGIC WRM1->WRM2; accept WRM1, verify CRC against the stored bytes, then discard last_heard and mark dirty so the next save rewrites as v2. Tests: test_warm_store 12/12 (adds test_ws_v1_migration_discardsLastHeard: key survives, role/protected reset). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WarmStore: guard role bit-width + test eviction carries role/protected - static_assert that the device role enum still fits the 4-bit warm metadata field (WARM_ROLE_MASK); fails the build loudly if a new role is added past 15 rather than silently truncating role on eviction. (Max role today = 12.) - Add test_migration_carriesRoleAndProtectedIntoWarm: a demoted TRACKER lands in the warm tier with its key, role=TRACKER and protected category=Role; a demoted CLIENT carries role=CLIENT/None. Exercises the NodeDB eviction path + warmProtectedCategory classification (the warm-store unit tests only cover absorb() directly). Tests: test_nodedb_blocked 5/5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix copilot comments * fix(test): restore #if HAS_TRAFFIC_MANAGEMENT guard in TMM test The rebase onto PR1.5 lost the top-level HAS_TRAFFIC_MANAGEMENT guard that PR1.5 introduced, leaving the #else/#endif tail orphaned and causing compile errors on non-TMM builds. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
296 lines
9.1 KiB
C++
296 lines
9.1 KiB
C++
// Unit tests for the warm ("long-tail") node tier — src/mesh/WarmNodeStore.cpp.
|
|
// Covers admission/eviction policy (keyed entries outrank keyless), take()
|
|
// rehydration semantics, and a tolerant persistence round trip.
|
|
#include "MeshTypes.h" // BEFORE TestUtil.h — provides WARM_NODE_COUNT via mesh-pb-constants.h
|
|
#include "TestUtil.h"
|
|
#include <unity.h>
|
|
|
|
#if defined(ARCH_PORTDUINO)
|
|
#define WS_TEST_ENTRY extern "C"
|
|
#else
|
|
#define WS_TEST_ENTRY
|
|
#endif
|
|
|
|
#if WARM_NODE_COUNT > 0
|
|
|
|
#include "FSCommon.h"
|
|
#include "mesh/WarmNodeStore.h"
|
|
#include <cstring>
|
|
#include <vector>
|
|
|
|
namespace
|
|
{
|
|
|
|
void makeKey(uint8_t out[32], uint8_t seed)
|
|
{
|
|
memset(out, 0, 32);
|
|
out[0] = seed;
|
|
out[31] = seed ^ 0xA5;
|
|
}
|
|
|
|
} // namespace
|
|
|
|
void setUp(void) {}
|
|
void tearDown(void) {}
|
|
|
|
void test_ws_absorb_and_copyKey_roundTrip()
|
|
{
|
|
WarmNodeStore ws;
|
|
uint8_t key[32], got[32];
|
|
makeKey(key, 7);
|
|
TEST_ASSERT_TRUE(ws.absorb(0x100, 1000, key));
|
|
TEST_ASSERT_TRUE(ws.contains(0x100));
|
|
TEST_ASSERT_TRUE(ws.copyKey(0x100, got));
|
|
TEST_ASSERT_EQUAL_MEMORY(key, got, 32);
|
|
TEST_ASSERT_EQUAL(1, ws.count());
|
|
}
|
|
|
|
void test_ws_keylessEntry_hasNoKey()
|
|
{
|
|
WarmNodeStore ws;
|
|
uint8_t got[32];
|
|
TEST_ASSERT_TRUE(ws.absorb(0x200, 1000, NULL));
|
|
TEST_ASSERT_TRUE(ws.contains(0x200));
|
|
TEST_ASSERT_FALSE(ws.copyKey(0x200, got));
|
|
}
|
|
|
|
void test_ws_absorb_rejectsNodeNumZero()
|
|
{
|
|
WarmNodeStore ws;
|
|
TEST_ASSERT_FALSE(ws.absorb(0, 1000, NULL));
|
|
TEST_ASSERT_EQUAL(0, ws.count());
|
|
}
|
|
|
|
void test_ws_absorb_updatesExistingEntry()
|
|
{
|
|
WarmNodeStore ws;
|
|
uint8_t key[32], got[32];
|
|
makeKey(key, 9);
|
|
TEST_ASSERT_TRUE(ws.absorb(0x300, 1000, NULL));
|
|
TEST_ASSERT_TRUE(ws.absorb(0x300, 2000, key)); // later eviction learned a key
|
|
TEST_ASSERT_EQUAL(1, ws.count());
|
|
TEST_ASSERT_TRUE(ws.copyKey(0x300, got));
|
|
TEST_ASSERT_EQUAL_MEMORY(key, got, 32);
|
|
}
|
|
|
|
void test_ws_take_removesEntry()
|
|
{
|
|
WarmNodeStore ws;
|
|
uint8_t key[32];
|
|
makeKey(key, 3);
|
|
ws.absorb(0x400, 1234, key, 5 /* TRACKER */, (uint8_t)WarmProtected::Role);
|
|
|
|
WarmNodeEntry e;
|
|
TEST_ASSERT_TRUE(ws.take(0x400, e));
|
|
TEST_ASSERT_EQUAL(0x400, e.num);
|
|
// last_heard is quantised to the metadata quantum; role/protected ride the low bits.
|
|
TEST_ASSERT_EQUAL(1234u & WARM_TIME_MASK, warmTimeOf(e));
|
|
TEST_ASSERT_EQUAL(5, warmRoleOf(e));
|
|
TEST_ASSERT_EQUAL((uint8_t)WarmProtected::Role, warmProtOf(e));
|
|
TEST_ASSERT_EQUAL_MEMORY(key, e.public_key, 32);
|
|
TEST_ASSERT_FALSE(ws.contains(0x400));
|
|
TEST_ASSERT_FALSE(ws.take(0x400, e));
|
|
TEST_ASSERT_EQUAL(0, ws.count());
|
|
}
|
|
|
|
void test_ws_keylessCandidate_neverEvictsKeyedEntries()
|
|
{
|
|
WarmNodeStore ws;
|
|
uint8_t key[32];
|
|
// Fill the store entirely with keyed entries
|
|
for (size_t i = 0; i < ws.capacity(); i++) {
|
|
makeKey(key, (uint8_t)i);
|
|
TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, 100 + i, key));
|
|
}
|
|
TEST_ASSERT_EQUAL(ws.capacity(), ws.count());
|
|
// A keyless candidate (even a fresh one) must be rejected
|
|
TEST_ASSERT_FALSE(ws.absorb(0x9999, 999999, NULL));
|
|
TEST_ASSERT_FALSE(ws.contains(0x9999));
|
|
}
|
|
|
|
void test_ws_keyedCandidate_evictsOldestKeylessFirst()
|
|
{
|
|
WarmNodeStore ws;
|
|
uint8_t key[32];
|
|
makeKey(key, 0x42);
|
|
// Fill with keyed entries except two keyless ones in the middle
|
|
for (size_t i = 0; i < ws.capacity(); i++) {
|
|
const bool keyless = (i == 5 || i == 10);
|
|
// Timestamps spaced by the 64 s warm metadata quantum (<<6) so LRU order survives
|
|
// quantisation: keyless i=10 is oldest (50), i=5 next (60), keyed all older (10).
|
|
TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, (keyless ? (i == 10 ? 50u : 60u) : 10u) << 6, keyless ? NULL : key));
|
|
}
|
|
// Keyed candidate must displace the OLDEST KEYLESS entry (0x100A, ts=50),
|
|
// even though every keyed entry is older (ts=10)
|
|
uint8_t k2[32];
|
|
makeKey(k2, 0x43);
|
|
TEST_ASSERT_TRUE(ws.absorb(0x8888, 70u << 6, k2));
|
|
TEST_ASSERT_FALSE(ws.contains(0x1000 + 10));
|
|
TEST_ASSERT_TRUE(ws.contains(0x1000 + 5));
|
|
TEST_ASSERT_TRUE(ws.contains(0x8888));
|
|
}
|
|
|
|
void test_ws_keyedCandidate_evictsOldestKeyedWhenNoKeyless()
|
|
{
|
|
WarmNodeStore ws;
|
|
uint8_t key[32];
|
|
for (size_t i = 0; i < ws.capacity(); i++) {
|
|
makeKey(key, (uint8_t)i);
|
|
TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, 1000 + i, key)); // 0x1000 is the oldest
|
|
}
|
|
uint8_t k2[32];
|
|
makeKey(k2, 0x44);
|
|
TEST_ASSERT_TRUE(ws.absorb(0x7777, 999999, k2));
|
|
TEST_ASSERT_TRUE(ws.contains(0x7777));
|
|
TEST_ASSERT_FALSE(ws.contains(0x1000)); // oldest keyed evicted
|
|
TEST_ASSERT_EQUAL(ws.capacity(), ws.count());
|
|
}
|
|
|
|
void test_ws_meta_roundTrip()
|
|
{
|
|
WarmNodeStore ws;
|
|
uint8_t key[32];
|
|
makeKey(key, 0x77);
|
|
// Keyed TRACKER(5)/Role-protected and keyless SENSOR(6)/unprotected.
|
|
TEST_ASSERT_TRUE(ws.absorb(0x700, 1234, key, 5 /* TRACKER */, (uint8_t)WarmProtected::Role));
|
|
TEST_ASSERT_TRUE(ws.absorb(0x701, 5678, NULL, 6 /* SENSOR */, (uint8_t)WarmProtected::None));
|
|
|
|
uint8_t role = 0xFF, prot = 0xFF;
|
|
TEST_ASSERT_TRUE(ws.lookupMeta(0x700, role, prot));
|
|
TEST_ASSERT_EQUAL(5, role);
|
|
TEST_ASSERT_EQUAL((uint8_t)WarmProtected::Role, prot);
|
|
TEST_ASSERT_TRUE(ws.lookupMeta(0x701, role, prot));
|
|
TEST_ASSERT_EQUAL(6, role);
|
|
TEST_ASSERT_EQUAL((uint8_t)WarmProtected::None, prot);
|
|
// Absent node yields false and leaves outputs untouched-by-contract (just check return).
|
|
TEST_ASSERT_FALSE(ws.lookupMeta(0x999, role, prot));
|
|
// Default args still compile (role/protected = 0 = CLIENT/None).
|
|
TEST_ASSERT_TRUE(ws.absorb(0x702, 9999, NULL));
|
|
TEST_ASSERT_TRUE(ws.lookupMeta(0x702, role, prot));
|
|
TEST_ASSERT_EQUAL(0, role);
|
|
TEST_ASSERT_EQUAL((uint8_t)WarmProtected::None, prot);
|
|
}
|
|
|
|
void test_ws_remove_and_clear()
|
|
{
|
|
WarmNodeStore ws;
|
|
ws.absorb(0x500, 1, NULL);
|
|
ws.absorb(0x501, 2, NULL);
|
|
ws.remove(0x500);
|
|
TEST_ASSERT_FALSE(ws.contains(0x500));
|
|
TEST_ASSERT_EQUAL(1, ws.count());
|
|
ws.clear();
|
|
TEST_ASSERT_EQUAL(0, ws.count());
|
|
}
|
|
|
|
void test_ws_persistence_roundTrip()
|
|
{
|
|
WarmNodeStore a;
|
|
uint8_t key[32], got[32];
|
|
makeKey(key, 0x55);
|
|
a.absorb(0x600, 4242, key);
|
|
a.absorb(0x601, 4243, NULL);
|
|
if (!a.saveIfDirty()) {
|
|
TEST_IGNORE_MESSAGE("Filesystem not available in this test environment");
|
|
return;
|
|
}
|
|
|
|
WarmNodeStore b;
|
|
b.load();
|
|
TEST_ASSERT_TRUE(b.contains(0x600));
|
|
TEST_ASSERT_TRUE(b.contains(0x601));
|
|
TEST_ASSERT_TRUE(b.copyKey(0x600, got));
|
|
TEST_ASSERT_EQUAL_MEMORY(key, got, 32);
|
|
|
|
// Cleanup so reruns start fresh
|
|
b.clear();
|
|
b.saveIfDirty();
|
|
}
|
|
|
|
// Migration: a v1 (WRM1) warm.dat must keep identity + key but discard last_heard
|
|
// (so its low bits aren't misread as role/protected). File backend only.
|
|
void test_ws_v1_migration_discardsLastHeard()
|
|
{
|
|
WarmNodeStore a;
|
|
uint8_t key[32], got[32];
|
|
makeKey(key, 0x66);
|
|
a.absorb(0x900, 123456, key, 5 /* TRACKER */, (uint8_t)WarmProtected::Role);
|
|
if (!a.saveIfDirty()) {
|
|
TEST_IGNORE_MESSAGE("Filesystem not available in this test environment");
|
|
return;
|
|
}
|
|
|
|
// Read the whole v2 file, flip the 4-byte header magic to v1 ("WRM1"), write it back.
|
|
// (CRC covers only the entry bytes, so patching the header magic keeps it valid.)
|
|
std::vector<uint8_t> buf;
|
|
{
|
|
auto f = FSCom.open("/prefs/warm.dat", FILE_O_READ);
|
|
if (!f) {
|
|
TEST_IGNORE_MESSAGE("warm.dat not readable in this environment");
|
|
return;
|
|
}
|
|
buf.resize(f.size());
|
|
f.read(buf.data(), buf.size());
|
|
f.close();
|
|
}
|
|
TEST_ASSERT_TRUE(buf.size() >= 4);
|
|
const uint32_t v1magic = 0x314D5257u; // "WRM1"
|
|
memcpy(buf.data(), &v1magic, sizeof(v1magic));
|
|
{
|
|
auto f = FSCom.open("/prefs/warm.dat", FILE_O_WRITE);
|
|
TEST_ASSERT_TRUE((bool)f);
|
|
f.write(buf.data(), buf.size());
|
|
f.close();
|
|
}
|
|
|
|
WarmNodeStore b;
|
|
b.load();
|
|
TEST_ASSERT_TRUE(b.contains(0x900)); // identity survived migration
|
|
TEST_ASSERT_TRUE(b.copyKey(0x900, got)); // public key survived
|
|
TEST_ASSERT_EQUAL_MEMORY(key, got, 32);
|
|
uint8_t role = 0xFF, prot = 0xFF;
|
|
TEST_ASSERT_TRUE(b.lookupMeta(0x900, role, prot));
|
|
TEST_ASSERT_EQUAL(0, role); // last_heard discarded → role/protected reset
|
|
TEST_ASSERT_EQUAL((uint8_t)WarmProtected::None, prot);
|
|
|
|
b.clear();
|
|
b.saveIfDirty();
|
|
}
|
|
|
|
WS_TEST_ENTRY void setup()
|
|
{
|
|
initializeTestEnvironment();
|
|
UNITY_BEGIN();
|
|
RUN_TEST(test_ws_absorb_and_copyKey_roundTrip);
|
|
RUN_TEST(test_ws_keylessEntry_hasNoKey);
|
|
RUN_TEST(test_ws_absorb_rejectsNodeNumZero);
|
|
RUN_TEST(test_ws_absorb_updatesExistingEntry);
|
|
RUN_TEST(test_ws_take_removesEntry);
|
|
RUN_TEST(test_ws_keylessCandidate_neverEvictsKeyedEntries);
|
|
RUN_TEST(test_ws_keyedCandidate_evictsOldestKeylessFirst);
|
|
RUN_TEST(test_ws_keyedCandidate_evictsOldestKeyedWhenNoKeyless);
|
|
RUN_TEST(test_ws_meta_roundTrip);
|
|
RUN_TEST(test_ws_remove_and_clear);
|
|
RUN_TEST(test_ws_persistence_roundTrip);
|
|
RUN_TEST(test_ws_v1_migration_discardsLastHeard);
|
|
exit(UNITY_END());
|
|
}
|
|
|
|
WS_TEST_ENTRY void loop() {}
|
|
|
|
#else
|
|
|
|
void setUp(void) {}
|
|
void tearDown(void) {}
|
|
|
|
WS_TEST_ENTRY void setup()
|
|
{
|
|
initializeTestEnvironment();
|
|
UNITY_BEGIN();
|
|
exit(UNITY_END());
|
|
}
|
|
|
|
WS_TEST_ENTRY void loop() {}
|
|
|
|
#endif
|