* TrafficManagement: flat unified cache + persistent next-hop overflow store Reworks the TrafficManagementModule cache layer (policing behaviour unchanged from upstream) and adds a routing-hint overflow store: - Flatten the ring: replace the cuckoo-hashed unified cache and the bucketed PSRAM NodeInfo index with plain flat arrays + linear scan (same idiom as WarmNodeStore). At LoRa packet rates an O(n) scan of the cache is negligible, and it removes a large amount of hashing/displacement complexity. The cache entry is 11 B; timestamps use a uniform +1 presence-offset so a 0 byte always means "empty" across every sub-store. Adds rebaseEpoch() so cached state survives the ~19 h relative-timestamp horizon instead of being flushed. - Next-hop overflow cache: setNextHop/getNextHopHint store a confirmed last-byte relay for a destination, written only from NextHopRouter's ACK-confirmed decision (and mirrored from TraceRoute). NextHopRouter::getNextHop falls back to this cache when the hot NodeDB has no hint, so DMs/relays to long-tail nodes keep routing after the node ages out of NodeInfoLite. - Persistence: preloadNextHopsFromNodeDB warm-starts the cache from persisted NodeInfoLite hints on first maintenance pass; next_hop entries are kept alive across the maintenance sweep (no TTL) and never clobbered by a stale preload. All packet-policing logic (rate limit, position dedup, unknown-packet drop, NodeInfo direct response, hop exhaustion) is the existing upstream behaviour, untouched. HAS_TRAFFIC_MANAGEMENT defaults on so the module is compiled in. (see note). Tests: upstream policing suite now actually runs (adds the MeshTypes.h include that gates HAS_TRAFFIC_MANAGEMENT) plus 4 next-hop tests. Role-aware throttles, politeness, precision clamp, port-interval and mesh-radius gating — and the rate-limit >255 saturation fix — are deferred to the advanced-TMM branch. Note: default dedup movement grid moves to ~91m, which also means 1.5km required to end up with the same signature position - coarser and therefore further than before. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * TrafficManagement: fix cppcheck constVariablePointer warning `node` in preloadNextHopsFromNodeDB() is never written through — mark it const to satisfy cppcheck's constVariablePointer check in CI. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Add multi-hop NextHop recovery tests and unit tests for routing reliability - Introduced a new test suite for multi-hop NextHop directed-message delivery and relay recovery in `test_nexthop_multihop_recovery.py`. This includes tests for end-to-end delivery and recovery after relay drop. - Implemented unit tests in `test_main.cpp` for NextHop routing reliability mitigations, covering: - M1: Ambiguity-aware last-byte resolution. - M2: NextHopRouter's strict-neighbor gate and hop limit checks. - M3: Route-health freshness and failure decay. - Enhanced mock classes to facilitate controlled testing of node behaviors and routing logic. * grafting fixed * Address Copilot review for PR #10735 (NextHop improvements) - docs/nexthop-routing-reliability.md: update status from "no code changes yet" to reflect that mitigations and tests are implemented RAM pressure and MIGRATION_VERBOSE concerns addressed upstream in PR2.5 (per-platform TRAFFIC_MANAGEMENT_CACHE_SIZE) and PR2 (verbose default=0) respectively; (0,0) sentinel fixed in PR2.5. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * CI: fix cppcheck constVariablePointer and test include path - NextHopRouter.cpp: qualify two RouteHealth *h locals as const — only read for stale-route checks, never mutated through the pointer - Router.cpp: qualify meshtastic_NodeInfoLite *node as const in shouldDecrementHopLimit — only read for favorite/role predicate - test_position_module/test_main.cpp: change bare PositionModule.h to modules/PositionModule.h — build_flags sets -Isrc, not -Isrc/modules, so the bare form fails to resolve in the native PlatformIO test env Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * WarmStore: cache device role + protected category in last_heard low bits Steal the low 6 bits of WarmNodeEntry.last_heard to carry an evicted node's device role (4 bits) and a protected category (2 bits) for the hop-trim path, at zero record-size cost (entry stays 40 B; no RAM/flash growth). The high bits remain a real unix-seconds timestamp, quantised to 64 s — ample for warm LRU ordering of long-tail nodes. - absorb() packs role/protectedCat; place()/ring replay store the raw word so metadata round-trips through flash. LRU compares masked time (warmTimeOf). - take() rehydration masks the metadata bits and restores the cached role so a re-admitted node isn't stuck at CLIENT until its next NodeInfo. - NodeDB classifies the category (favorite/ignored/verified -> Flag; tracker/sensor/tak_tracker -> Role) at each eviction site. - WarmNodeStore::lookupMeta() exposes role/category to consumers. - Bump WARM_RING_MAGIC (WRNG->WRN2): old rings read as erased and rebuild; warm data is a non-critical evictee cache, so discard-on-upgrade is safe. Tests: test_warm_store 11/11 (new meta round-trip + quantisation-aware ordering); NodeDB compiles (test_nodedb_blocked 4/4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WarmStore: migrate v1 rings/files by discarding last_heard, not the data Previously the WRNG->WRN2 magic bump treated old rings as erased, discarding all warm entries — including the PKI public keys that let evicted nodes keep decrypting DMs. Instead, read v1 (WRNG / WRM1) records and keep each node's identity + public key, discarding only last_heard (its low bits would otherwise be misread as the new role/protected metadata). Records re-rank and re-learn their role on next contact. - Ring backend (nRF52840): ringReadHeader accepts both magics and reports v1 via an out-param; replay zeroes last_heard for v1 records. If the active head page is v1, force a rotation so new v2 records never land in a v1-headered page (which would discard their freshly-set role on the next load). Legacy pages convert to v2 as the ring rotates. - File backend (warm.dat): bump WARM_STORE_MAGIC WRM1->WRM2; accept WRM1, verify CRC against the stored bytes, then discard last_heard and mark dirty so the next save rewrites as v2. Tests: test_warm_store 12/12 (adds test_ws_v1_migration_discardsLastHeard: key survives, role/protected reset). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * WarmStore: guard role bit-width + test eviction carries role/protected - static_assert that the device role enum still fits the 4-bit warm metadata field (WARM_ROLE_MASK); fails the build loudly if a new role is added past 15 rather than silently truncating role on eviction. (Max role today = 12.) - Add test_migration_carriesRoleAndProtectedIntoWarm: a demoted TRACKER lands in the warm tier with its key, role=TRACKER and protected category=Role; a demoted CLIENT carries role=CLIENT/None. Exercises the NodeDB eviction path + warmProtectedCategory classification (the warm-store unit tests only cover absorb() directly). Tests: test_nodedb_blocked 5/5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * fix copilot comments * fix(test): restore #if HAS_TRAFFIC_MANAGEMENT guard in TMM test The rebase onto PR1.5 lost the top-level HAS_TRAFFIC_MANAGEMENT guard that PR1.5 introduced, leaving the #else/#endif tail orphaned and causing compile errors on non-TMM builds. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
228 lines
9.1 KiB
C++
228 lines
9.1 KiB
C++
// Tests for the NodeDB hot-store migration and favourite/ignored (blocked)
|
|
// retention paths — src/mesh/NodeDB.cpp.
|
|
#include "MeshTypes.h" // BEFORE TestUtil.h — provides WARM_NODE_COUNT / MAX_NUM_NODES via mesh-pb-constants.h
|
|
#include "TestUtil.h"
|
|
#include <unity.h>
|
|
|
|
#if defined(ARCH_PORTDUINO)
|
|
#define NDB_TEST_ENTRY extern "C"
|
|
#else
|
|
#define NDB_TEST_ENTRY
|
|
#endif
|
|
|
|
// The migration demotes overflow into the warm tier, so these tests need it.
|
|
#if WARM_NODE_COUNT > 0
|
|
|
|
#include "mesh/NodeDB.h"
|
|
#include <cstring>
|
|
|
|
// Subclass shim: exposes the private maintenance paths (via the friend
|
|
// declaration in NodeDB.h) and lets a test own the hot store directly
|
|
// (meshNodes/numMeshNodes are public). Declared at global scope so it matches
|
|
// `friend class NodeDBTestShim` — an anonymous-namespace class would not.
|
|
class NodeDBTestShim : public NodeDB
|
|
{
|
|
public:
|
|
void runDemote() { demoteOldestHotNodesToWarm(); }
|
|
void runCleanup() { cleanupMeshDB(); }
|
|
|
|
// Read back the role + protected category the warm tier cached for a node.
|
|
bool warmMeta(NodeNum n, uint8_t &role, uint8_t &prot) { return warmStore.lookupMeta(n, role, prot); }
|
|
|
|
void clearHot()
|
|
{
|
|
meshNodes->clear();
|
|
numMeshNodes = 0;
|
|
}
|
|
|
|
void push(NodeNum num, uint32_t lastHeard, bool favorite, bool ignored, bool withUser, bool withKey,
|
|
meshtastic_Config_DeviceConfig_Role role = meshtastic_Config_DeviceConfig_Role_CLIENT)
|
|
{
|
|
meshtastic_NodeInfoLite n = meshtastic_NodeInfoLite_init_zero;
|
|
n.num = num;
|
|
n.last_heard = lastHeard;
|
|
n.role = role;
|
|
if (favorite)
|
|
nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true);
|
|
if (ignored)
|
|
nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_IS_IGNORED_MASK, true);
|
|
if (withUser)
|
|
nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_HAS_USER_MASK, true);
|
|
if (withKey) {
|
|
n.public_key.size = 32;
|
|
memset(n.public_key.bytes, static_cast<uint8_t>(num & 0xff), 32);
|
|
n.public_key.bytes[0] = 0x01; // ensure non-zero (all-zero == "no key")
|
|
}
|
|
meshNodes->push_back(n);
|
|
numMeshNodes = meshNodes->size();
|
|
}
|
|
|
|
// Index 0 is our own node; the eviction/migration scans treat it as self.
|
|
void seedSelf() { push(0x0BADF00D, 0xFFFFFFFFu, false, false, /*withUser=*/true, /*withKey=*/false); }
|
|
};
|
|
|
|
namespace
|
|
{
|
|
|
|
NodeDBTestShim *db = nullptr;
|
|
|
|
bool warmHasKey(NodeNum n)
|
|
{
|
|
meshtastic_NodeInfoLite_public_key_t k = {0, {0}};
|
|
return db->copyPublicKey(n, k) && k.size == 32;
|
|
}
|
|
|
|
} // namespace
|
|
|
|
void setUp(void)
|
|
{
|
|
db->clearHot();
|
|
}
|
|
void tearDown(void) {}
|
|
|
|
// Migration: a database from a larger-cap build trims to MAX_NUM_NODES; the
|
|
// oldest non-protected nodes are demoted into the warm tier (keys preserved),
|
|
// while self, favourites and ignored survive even when they are the oldest.
|
|
static void test_migration_demotesOldestKeepsKeepersAndSelf(void)
|
|
{
|
|
db->seedSelf();
|
|
const int extra = MAX_NUM_NODES + 30; // overflow well past the MAX-2 cap
|
|
for (int i = 1; i <= extra; i++) {
|
|
const bool fav = (i == 1); // oldest, but a favourite
|
|
const bool ign = (i == 2); // 2nd-oldest, but blocked
|
|
db->push(2000 + i, /*last_heard=*/i, fav, ign, /*withUser=*/true, /*withKey=*/true);
|
|
}
|
|
|
|
db->runDemote();
|
|
|
|
TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES, (int)db->getNumMeshNodes());
|
|
TEST_ASSERT_NOT_NULL(db->getMeshNode(0x0BADF00D)); // self retained
|
|
TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + 1)); // oldest favourite retained
|
|
TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + 2)); // oldest ignored retained
|
|
TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + extra)); // freshest retained
|
|
TEST_ASSERT_NULL(db->getMeshNode(2000 + 3)); // oldest non-protected demoted out of hot
|
|
TEST_ASSERT_TRUE(warmHasKey(2000 + 3)); // ...but its key kept in the warm tier
|
|
}
|
|
|
|
// Eviction carries the device role + protected category into the warm tier. A TRACKER is
|
|
// hop-protected but NOT eviction-protected, so it gets demoted with its key; the warm
|
|
// record must report role=TRACKER / category=Role. A plain CLIENT carries role=CLIENT/None.
|
|
static void test_migration_carriesRoleAndProtectedIntoWarm(void)
|
|
{
|
|
db->seedSelf();
|
|
const int extra = MAX_NUM_NODES + 30; // overflow so the oldest non-protected are demoted
|
|
for (int i = 1; i <= extra; i++) {
|
|
const auto role = (i == 3) ? meshtastic_Config_DeviceConfig_Role_TRACKER : meshtastic_Config_DeviceConfig_Role_CLIENT;
|
|
db->push(2000 + i, /*last_heard=*/i, /*favorite=*/false, /*ignored=*/false, /*withUser=*/true,
|
|
/*withKey=*/true, role);
|
|
}
|
|
|
|
db->runDemote();
|
|
|
|
uint8_t role = 0xFF, prot = 0xFF;
|
|
// TRACKER (i=3): demoted out of hot, key kept, role + protected carried into warm.
|
|
TEST_ASSERT_NULL(db->getMeshNode(2000 + 3));
|
|
TEST_ASSERT_TRUE(warmHasKey(2000 + 3));
|
|
TEST_ASSERT_TRUE(db->warmMeta(2000 + 3, role, prot));
|
|
TEST_ASSERT_EQUAL(meshtastic_Config_DeviceConfig_Role_TRACKER, role);
|
|
TEST_ASSERT_EQUAL((uint8_t)WarmProtected::Role, prot);
|
|
// CLIENT (i=4): also demoted, carries role=CLIENT / category=None.
|
|
TEST_ASSERT_TRUE(db->warmMeta(2000 + 4, role, prot));
|
|
TEST_ASSERT_EQUAL(meshtastic_Config_DeviceConfig_Role_CLIENT, role);
|
|
TEST_ASSERT_EQUAL((uint8_t)WarmProtected::None, prot);
|
|
}
|
|
|
|
// Favourite handling: a favourite is never the eviction victim, even when it is
|
|
// the oldest node in a full hot store.
|
|
static void test_eviction_preservesFavorite(void)
|
|
{
|
|
db->seedSelf();
|
|
for (int i = 1; i < MAX_NUM_NODES; i++) { // fill to MAX_NUM_NODES total (incl. self)
|
|
const bool fav = (i == 1); // oldest non-self, favourite
|
|
db->push(3000 + i, /*last_heard=*/i, fav, false, /*withUser=*/true, /*withKey=*/true);
|
|
}
|
|
TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES, (int)db->getNumMeshNodes()); // full
|
|
|
|
TEST_ASSERT_NOT_NULL(db->getOrCreateMeshNode(0x99990000)); // forces an eviction
|
|
|
|
TEST_ASSERT_NOT_NULL(db->getMeshNode(3000 + 1)); // favourite survived despite being oldest
|
|
TEST_ASSERT_NULL(db->getMeshNode(3000 + 2)); // oldest non-favourite evicted
|
|
TEST_ASSERT_NOT_NULL(db->getMeshNode(0x99990000));
|
|
}
|
|
|
|
// Ignored handling: an ignored node survives eviction (like a favourite), and is
|
|
// never purged by cleanupMeshDB even with no user info (a block set by bare ID).
|
|
static void test_ignored_survivesEvictionAndCleanup(void)
|
|
{
|
|
// (a) eviction protection
|
|
db->clearHot();
|
|
db->seedSelf();
|
|
for (int i = 1; i < MAX_NUM_NODES; i++) {
|
|
const bool ign = (i == 1); // oldest non-self, blocked
|
|
db->push(4000 + i, /*last_heard=*/i, false, ign, /*withUser=*/true, /*withKey=*/true);
|
|
}
|
|
TEST_ASSERT_NOT_NULL(db->getOrCreateMeshNode(0x88880000));
|
|
TEST_ASSERT_NOT_NULL(db->getMeshNode(4000 + 1)); // blocked node survived
|
|
TEST_ASSERT_NULL(db->getMeshNode(4000 + 2)); // oldest non-blocked evicted
|
|
|
|
// (b) cleanup protection — ignored kept without user info, plain no-user purged
|
|
db->clearHot();
|
|
db->seedSelf();
|
|
db->push(5000, 100, false, /*ignored=*/true, /*withUser=*/false, false);
|
|
db->push(5001, 100, false, false, /*withUser=*/false, false);
|
|
db->runCleanup();
|
|
TEST_ASSERT_NOT_NULL(db->getMeshNode(5000)); // blocked-by-ID kept despite no user info
|
|
TEST_ASSERT_NULL(db->getMeshNode(5001)); // ordinary no-user node purged
|
|
}
|
|
|
|
// Protected-node cap: at most MAX_NUM_NODES-2 nodes may be protected, so >=2
|
|
// evictable slots always remain. setProtectedFlag refuses once the cap is hit.
|
|
static void test_protectedCap_refusesBeyondLimit(void)
|
|
{
|
|
db->seedSelf();
|
|
for (int i = 0; i < MAX_NUM_NODES - 2; i++)
|
|
db->push(6000 + i, 100, /*favorite=*/true, false, /*withUser=*/true, false);
|
|
TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES - 2, db->numProtectedNodes());
|
|
|
|
db->push(7000, 100, false, false, /*withUser=*/true, false);
|
|
meshtastic_NodeInfoLite *fresh = db->getMeshNode(7000);
|
|
TEST_ASSERT_NOT_NULL(fresh);
|
|
TEST_ASSERT_FALSE(db->setProtectedFlag(fresh, NODEINFO_BITFIELD_IS_IGNORED_MASK, true)); // refused at cap
|
|
TEST_ASSERT_FALSE(nodeInfoLiteIsIgnored(fresh)); // unchanged
|
|
TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES - 2, db->numProtectedNodes());
|
|
|
|
// Adding another flag to an already-protected node doesn't grow the set, so
|
|
// it's still allowed at the cap.
|
|
meshtastic_NodeInfoLite *already = db->getMeshNode(6000);
|
|
TEST_ASSERT_TRUE(db->setProtectedFlag(already, NODEINFO_BITFIELD_IS_IGNORED_MASK, true));
|
|
}
|
|
|
|
NDB_TEST_ENTRY void setup()
|
|
{
|
|
initializeTestEnvironment();
|
|
db = new NodeDBTestShim();
|
|
nodeDB = db;
|
|
|
|
UNITY_BEGIN();
|
|
RUN_TEST(test_migration_demotesOldestKeepsKeepersAndSelf);
|
|
RUN_TEST(test_migration_carriesRoleAndProtectedIntoWarm);
|
|
RUN_TEST(test_eviction_preservesFavorite);
|
|
RUN_TEST(test_ignored_survivesEvictionAndCleanup);
|
|
RUN_TEST(test_protectedCap_refusesBeyondLimit);
|
|
exit(UNITY_END());
|
|
}
|
|
NDB_TEST_ENTRY void loop() {}
|
|
|
|
#else // WARM_NODE_COUNT == 0 — nothing to exercise here
|
|
|
|
void setUp(void) {}
|
|
void tearDown(void) {}
|
|
NDB_TEST_ENTRY void setup()
|
|
{
|
|
UNITY_BEGIN();
|
|
exit(UNITY_END());
|
|
}
|
|
NDB_TEST_ENTRY void loop() {}
|
|
|
|
#endif
|