kevin/meshtastic-firmware
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

Pr1 nodedb warmstore (#10705)

Browse Source 
* NodeDB: 3-tier node store with persistent warm tier (long-tail identity retention)

Introduces a tiered NodeDB so the device retains identity (public key,
last_heard) for far more nodes than fit in the full-record hot store,
without growing heap or the persisted nodes.proto unboundedly.

- Hot store: full NodeInfoLite, MAX_NUM_NODES (120 on nRF52).
- Satellite maps: position/telemetry/environment/status capped at
  MAX_SATELLITE_NODES (40 freshest); eviction via enforceSatelliteCaps /
  evictSatelliteOverCap.
- Warm tier (WarmNodeStore): 40 B {num,last_heard,public_key} records for
  evicted nodes so DMs to/from long-tail nodes keep encrypting/decrypting.
  Persisted to /prefs/warm.dat, or on nRF52840 a dedicated 12 KB raw-flash
  record-ring below LittleFS (3x4 KB pages; see linker scripts + the
  nrf52_warm_region.py post-link guard).

NodeDB::getOrCreateMeshNode now demotes evicted nodes into the warm tier and
re-admits them (restoring key/last_heard). Router PKI decrypt/encode resolve
the peer key via NodeDB::copyPublicKey (hot store, then warm tier).

NodeInfoLite gains snr_q4 (sint32, Q4-encoded dB); the float snr is zeroed on
disk. NodeInfoLite grows 105 -> 112 B; backup 2432 -> 2468 B.

Note: the snr_q4 .proto change still needs to land in the protobufs submodule
(generated header is updated here; submodule pointer left at upstream).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* NodeDB: robust receive + retention for blocked (ignored) nodes

Hardens how ignored/favourite nodes are received over admin and retained,
closing paths where a block could be lost or accidentally cleared.

- Blocking keeps the node's public key (admin set_ignored_node and
  addFromContact no longer zero it / drop the warm-tier key), so a blocked
  peer stays a verifiable identity.
- set_ignored_node creates the node if absent, so a block by node ID sticks
  even for a node we've never heard from (e.g. pushed by a remote admin) with
  no NodeInfo or key.
- Eviction protection (favourite/ignored/manually-verified) now also applies to
  the load-time hot-store migration and is never undone by cleanupMeshDB, which
  previously purged ignored nodes that lacked user info.
- The hot-store migration leaves our own node (index 0) in place and prefers to
  demote non-protected nodes, like the runtime eviction scan.

Caps the protected set (favourite + ignored + verified) at MAX_NUM_NODES-2 via
NodeDB::setProtectedFlag(), so at least two evictable slots always remain and
getOrCreateMeshNode can always make room — replacing the previous unconditional
append that could run off the end of the node vector when every node was
protected. A locally-set favourite/ignore that hits the cap reports back to the
phone via a ClientNotification.

Adds test_nodedb_blocked covering the migration, favourite/ignored eviction
protection, ignored-survives-cleanup, and the protected-node cap. The
maintenance methods stay private in production; the test reaches them through a
PIO_UNIT_TESTING-guarded friend shim.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

# Conflicts:
#	src/mesh/NodeDB.h

* fix copilot comments

* once again

* WarmNodeStore: fix cppcheck warnings (uninitvar, constVariablePointer)

Zero-initialise `stranded[]` and `seqs[]/order[]` VLAs so cppcheck can
verify there are no unguarded reads of uninitialised memory (the guards
exist but are not visible to static analysis). Mark two local pointers
`const` where the pointed-to entry is never mutated after assignment.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* self-care added to assist 2.7 and 2.8 nodedb migration

* Tidy warm-store/self-care: comments, guards, log + flash cleanup

Style/cleanup pass over the branch (no behavior change except the noted
preprocessor simplifications, which are semantically identical):

- Comments: move function descriptions to the headers, cap in-function
  comments at ~3-4 lines, drop leading-number step markers, label stacked
  #endif blocks, de-decorate banner comments.
- dumpToLog: fully gate decl + definition + AdminModule call site behind
  MESHTASTIC_NODEDB_MIGRATION_VERBOSE so it compiles out when disabled
  (~1.2 KB when off).
- mesh-pb-constants: drop the dead nRF52832 WARM_NODE_COUNT branch and trim
  the macro docs.
- WarmNodeStore: simplify the redundant `ARCH_NRF52 && NRF52840_XXAA` guards
  to `NRF52840_XXAA`, add a kNoPage sentinel for the ring page state.
- Shorten the always-on LOG_WARN strings (~120 B flash).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* more tidying up, aligning with docs and undoing other-arch regressions

* Update protobufs (#19)

Co-authored-by: NomDeTom <116762865+NomDeTom@users.noreply.github.com>

* made the migration pathway cleareer

* address copilot review

* fixed a copilot review on a downstream PR.

* Address Copilot review comments for PR #10705 (warmstore/nodedb)

- WarmNodeStore.h: default MIGRATION_VERBOSE to 0 (suppress info-level
  chatter on production builds; opt in with =1)
- WarmNodeStore.cpp load(): move memset to top of function so all
  failure paths (header-read fail, invalid header) leave entries clear
- WarmNodeStore.cpp save(): replace manual spiLock lock/unlock around
  mkdir with LockGuard covering the full SafeFile sequence, matching
  the lock discipline in load()
- Router.cpp: memcpy(&p->public_key.bytes, ...) -> memcpy(p->public_key.bytes,
  ...) — pass decayed uint8_t* rather than pointer-to-array
- AdminModule.cpp: check setProtectedFlag return for PKC auto-favorite;
  log cap-refusal warning instead of unconditional "auto-favoriting"
- nrf52_warm_region.py: error message references both v6.ld and v7.ld

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* NodeDB: formatting cleanup (blank lines after preprocessor blocks)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Lukewarm store

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Ben Meadors <benmmeadors@gmail.com>
This commit is contained in:
Tom
2026-06-18 08:41:34 +01:00
committed by GitHub
co-authored by GitHub Claude Opus 4.8 github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Ben Meadors
parent b311e01ce0
commit 79a7dcc46c
17 changed files with 1848 additions and 87 deletions
Download Patch File Download Diff File Expand all files Collapse all files
test/test_nodedb_blocked/test_main.cpp
+193
View File
@@ -0,0 +1,193 @@
// Tests for the NodeDB hot-store migration and favourite/ignored (blocked)
// retention paths — src/mesh/NodeDB.cpp.
#include "MeshTypes.h" // BEFORE TestUtil.h — provides WARM_NODE_COUNT / MAX_NUM_NODES via mesh-pb-constants.h
#include "TestUtil.h"
#include <unity.h>
#if defined(ARCH_PORTDUINO)
#define NDB_TEST_ENTRY extern "C"
#else
#define NDB_TEST_ENTRY
#endif
// The migration demotes overflow into the warm tier, so these tests need it.
#if WARM_NODE_COUNT > 0
#include "mesh/NodeDB.h"
#include <cstring>
// Subclass shim: exposes the private maintenance paths (via the friend
// declaration in NodeDB.h) and lets a test own the hot store directly
// (meshNodes/numMeshNodes are public). Declared at global scope so it matches
// `friend class NodeDBTestShim` — an anonymous-namespace class would not.
class NodeDBTestShim : public NodeDB
{
public:
void runDemote() { demoteOldestHotNodesToWarm(); }
void runCleanup() { cleanupMeshDB(); }
void clearHot()
{
meshNodes->clear();
numMeshNodes = 0;
}
void push(NodeNum num, uint32_t lastHeard, bool favorite, bool ignored, bool withUser, bool withKey)
{
meshtastic_NodeInfoLite n = meshtastic_NodeInfoLite_init_zero;
n.num = num;
n.last_heard = lastHeard;
if (favorite)
nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true);
if (ignored)
nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_IS_IGNORED_MASK, true);
if (withUser)
nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_HAS_USER_MASK, true);
if (withKey) {
n.public_key.size = 32;
memset(n.public_key.bytes, static_cast<uint8_t>(num & 0xff), 32);
n.public_key.bytes[0] = 0x01; // ensure non-zero (all-zero == "no key")
}
meshNodes->push_back(n);
numMeshNodes = meshNodes->size();
}
// Index 0 is our own node; the eviction/migration scans treat it as self.
void seedSelf() { push(0x0BADF00D, 0xFFFFFFFFu, false, false, /*withUser=*/true, /*withKey=*/false); }
};
namespace
{
NodeDBTestShim *db = nullptr;
bool warmHasKey(NodeNum n)
{
meshtastic_NodeInfoLite_public_key_t k = {0, {0}};
return db->copyPublicKey(n, k) && k.size == 32;
}
} // namespace
void setUp(void)
{
db->clearHot();
}
void tearDown(void) {}
// Migration: a database from a larger-cap build trims to MAX_NUM_NODES; the
// oldest non-protected nodes are demoted into the warm tier (keys preserved),
// while self, favourites and ignored survive even when they are the oldest.
static void test_migration_demotesOldestKeepsKeepersAndSelf(void)
{
db->seedSelf();
const int extra = MAX_NUM_NODES + 30; // overflow well past the MAX-2 cap
for (int i = 1; i <= extra; i++) {
const bool fav = (i == 1); // oldest, but a favourite
const bool ign = (i == 2); // 2nd-oldest, but blocked
db->push(2000 + i, /*last_heard=*/i, fav, ign, /*withUser=*/true, /*withKey=*/true);
}
db->runDemote();
TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES, (int)db->getNumMeshNodes());
TEST_ASSERT_NOT_NULL(db->getMeshNode(0x0BADF00D)); // self retained
TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + 1)); // oldest favourite retained
TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + 2)); // oldest ignored retained
TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + extra)); // freshest retained
TEST_ASSERT_NULL(db->getMeshNode(2000 + 3)); // oldest non-protected demoted out of hot
TEST_ASSERT_TRUE(warmHasKey(2000 + 3)); // ...but its key kept in the warm tier
}
// Favourite handling: a favourite is never the eviction victim, even when it is
// the oldest node in a full hot store.
static void test_eviction_preservesFavorite(void)
{
db->seedSelf();
for (int i = 1; i < MAX_NUM_NODES; i++) { // fill to MAX_NUM_NODES total (incl. self)
const bool fav = (i == 1); // oldest non-self, favourite
db->push(3000 + i, /*last_heard=*/i, fav, false, /*withUser=*/true, /*withKey=*/true);
}
TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES, (int)db->getNumMeshNodes()); // full
TEST_ASSERT_NOT_NULL(db->getOrCreateMeshNode(0x99990000)); // forces an eviction
TEST_ASSERT_NOT_NULL(db->getMeshNode(3000 + 1)); // favourite survived despite being oldest
TEST_ASSERT_NULL(db->getMeshNode(3000 + 2)); // oldest non-favourite evicted
TEST_ASSERT_NOT_NULL(db->getMeshNode(0x99990000));
}
// Ignored handling: an ignored node survives eviction (like a favourite), and is
// never purged by cleanupMeshDB even with no user info (a block set by bare ID).
static void test_ignored_survivesEvictionAndCleanup(void)
{
// (a) eviction protection
db->clearHot();
db->seedSelf();
for (int i = 1; i < MAX_NUM_NODES; i++) {
const bool ign = (i == 1); // oldest non-self, blocked
db->push(4000 + i, /*last_heard=*/i, false, ign, /*withUser=*/true, /*withKey=*/true);
}
TEST_ASSERT_NOT_NULL(db->getOrCreateMeshNode(0x88880000));
TEST_ASSERT_NOT_NULL(db->getMeshNode(4000 + 1)); // blocked node survived
TEST_ASSERT_NULL(db->getMeshNode(4000 + 2)); // oldest non-blocked evicted
// (b) cleanup protection — ignored kept without user info, plain no-user purged
db->clearHot();
db->seedSelf();
db->push(5000, 100, false, /*ignored=*/true, /*withUser=*/false, false);
db->push(5001, 100, false, false, /*withUser=*/false, false);
db->runCleanup();
TEST_ASSERT_NOT_NULL(db->getMeshNode(5000)); // blocked-by-ID kept despite no user info
TEST_ASSERT_NULL(db->getMeshNode(5001)); // ordinary no-user node purged
}
// Protected-node cap: at most MAX_NUM_NODES-2 nodes may be protected, so >=2
// evictable slots always remain. setProtectedFlag refuses once the cap is hit.
static void test_protectedCap_refusesBeyondLimit(void)
{
db->seedSelf();
for (int i = 0; i < MAX_NUM_NODES - 2; i++)
db->push(6000 + i, 100, /*favorite=*/true, false, /*withUser=*/true, false);
TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES - 2, db->numProtectedNodes());
db->push(7000, 100, false, false, /*withUser=*/true, false);
meshtastic_NodeInfoLite *fresh = db->getMeshNode(7000);
TEST_ASSERT_NOT_NULL(fresh);
TEST_ASSERT_FALSE(db->setProtectedFlag(fresh, NODEINFO_BITFIELD_IS_IGNORED_MASK, true)); // refused at cap
TEST_ASSERT_FALSE(nodeInfoLiteIsIgnored(fresh)); // unchanged
TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES - 2, db->numProtectedNodes());
// Adding another flag to an already-protected node doesn't grow the set, so
// it's still allowed at the cap.
meshtastic_NodeInfoLite *already = db->getMeshNode(6000);
TEST_ASSERT_TRUE(db->setProtectedFlag(already, NODEINFO_BITFIELD_IS_IGNORED_MASK, true));
}
NDB_TEST_ENTRY void setup()
{
initializeTestEnvironment();
db = new NodeDBTestShim();
nodeDB = db;
UNITY_BEGIN();
RUN_TEST(test_migration_demotesOldestKeepsKeepersAndSelf);
RUN_TEST(test_eviction_preservesFavorite);
RUN_TEST(test_ignored_survivesEvictionAndCleanup);
RUN_TEST(test_protectedCap_refusesBeyondLimit);
exit(UNITY_END());
}
NDB_TEST_ENTRY void loop() {}
#else // WARM_NODE_COUNT == 0 — nothing to exercise here
void setUp(void) {}
void tearDown(void) {}
NDB_TEST_ENTRY void setup()
{
UNITY_BEGIN();
exit(UNITY_END());
}
NDB_TEST_ENTRY void loop() {}
#endif
test/test_warm_store/test_main.cpp
+211
View File
@@ -0,0 +1,211 @@
// Unit tests for the warm ("long-tail") node tier — src/mesh/WarmNodeStore.cpp.
// Covers admission/eviction policy (keyed entries outrank keyless), take()
// rehydration semantics, and a tolerant persistence round trip.
#include "MeshTypes.h" // BEFORE TestUtil.h — provides WARM_NODE_COUNT via mesh-pb-constants.h
#include "TestUtil.h"
#include <unity.h>
#if defined(ARCH_PORTDUINO)
#define WS_TEST_ENTRY extern "C"
#else
#define WS_TEST_ENTRY
#endif
#if WARM_NODE_COUNT > 0
#include "mesh/WarmNodeStore.h"
#include <cstring>
namespace
{
void makeKey(uint8_t out[32], uint8_t seed)
{
memset(out, 0, 32);
out[0] = seed;
out[31] = seed ^ 0xA5;
}
} // namespace
void setUp(void) {}
void tearDown(void) {}
void test_ws_absorb_and_copyKey_roundTrip()
{
WarmNodeStore ws;
uint8_t key[32], got[32];
makeKey(key, 7);
TEST_ASSERT_TRUE(ws.absorb(0x100, 1000, key));
TEST_ASSERT_TRUE(ws.contains(0x100));
TEST_ASSERT_TRUE(ws.copyKey(0x100, got));
TEST_ASSERT_EQUAL_MEMORY(key, got, 32);
TEST_ASSERT_EQUAL(1, ws.count());
}
void test_ws_keylessEntry_hasNoKey()
{
WarmNodeStore ws;
uint8_t got[32];
TEST_ASSERT_TRUE(ws.absorb(0x200, 1000, NULL));
TEST_ASSERT_TRUE(ws.contains(0x200));
TEST_ASSERT_FALSE(ws.copyKey(0x200, got));
}
void test_ws_absorb_rejectsNodeNumZero()
{
WarmNodeStore ws;
TEST_ASSERT_FALSE(ws.absorb(0, 1000, NULL));
TEST_ASSERT_EQUAL(0, ws.count());
}
void test_ws_absorb_updatesExistingEntry()
{
WarmNodeStore ws;
uint8_t key[32], got[32];
makeKey(key, 9);
TEST_ASSERT_TRUE(ws.absorb(0x300, 1000, NULL));
TEST_ASSERT_TRUE(ws.absorb(0x300, 2000, key)); // later eviction learned a key
TEST_ASSERT_EQUAL(1, ws.count());
TEST_ASSERT_TRUE(ws.copyKey(0x300, got));
TEST_ASSERT_EQUAL_MEMORY(key, got, 32);
}
void test_ws_take_removesEntry()
{
WarmNodeStore ws;
uint8_t key[32];
makeKey(key, 3);
ws.absorb(0x400, 1234, key);
WarmNodeEntry e;
TEST_ASSERT_TRUE(ws.take(0x400, e));
TEST_ASSERT_EQUAL(0x400, e.num);
TEST_ASSERT_EQUAL(1234, e.last_heard);
TEST_ASSERT_EQUAL_MEMORY(key, e.public_key, 32);
TEST_ASSERT_FALSE(ws.contains(0x400));
TEST_ASSERT_FALSE(ws.take(0x400, e));
TEST_ASSERT_EQUAL(0, ws.count());
}
void test_ws_keylessCandidate_neverEvictsKeyedEntries()
{
WarmNodeStore ws;
uint8_t key[32];
// Fill the store entirely with keyed entries
for (size_t i = 0; i < ws.capacity(); i++) {
makeKey(key, (uint8_t)i);
TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, 100 + i, key));
}
TEST_ASSERT_EQUAL(ws.capacity(), ws.count());
// A keyless candidate (even a fresh one) must be rejected
TEST_ASSERT_FALSE(ws.absorb(0x9999, 999999, NULL));
TEST_ASSERT_FALSE(ws.contains(0x9999));
}
void test_ws_keyedCandidate_evictsOldestKeylessFirst()
{
WarmNodeStore ws;
uint8_t key[32];
makeKey(key, 0x42);
// Fill with keyed entries except two keyless ones in the middle
for (size_t i = 0; i < ws.capacity(); i++) {
const bool keyless = (i == 5 || i == 10);
TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, keyless ? (i == 10 ? 50 : 60) : 10, keyless ? NULL : key));
}
// Keyed candidate must displace the OLDEST KEYLESS entry (0x100A, ts=50),
// even though every keyed entry is older (ts=10)
uint8_t k2[32];
makeKey(k2, 0x43);
TEST_ASSERT_TRUE(ws.absorb(0x8888, 70, k2));
TEST_ASSERT_FALSE(ws.contains(0x1000 + 10));
TEST_ASSERT_TRUE(ws.contains(0x1000 + 5));
TEST_ASSERT_TRUE(ws.contains(0x8888));
}
void test_ws_keyedCandidate_evictsOldestKeyedWhenNoKeyless()
{
WarmNodeStore ws;
uint8_t key[32];
for (size_t i = 0; i < ws.capacity(); i++) {
makeKey(key, (uint8_t)i);
TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, 1000 + i, key)); // 0x1000 is the oldest
}
uint8_t k2[32];
makeKey(k2, 0x44);
TEST_ASSERT_TRUE(ws.absorb(0x7777, 999999, k2));
TEST_ASSERT_TRUE(ws.contains(0x7777));
TEST_ASSERT_FALSE(ws.contains(0x1000)); // oldest keyed evicted
TEST_ASSERT_EQUAL(ws.capacity(), ws.count());
}
void test_ws_remove_and_clear()
{
WarmNodeStore ws;
ws.absorb(0x500, 1, NULL);
ws.absorb(0x501, 2, NULL);
ws.remove(0x500);
TEST_ASSERT_FALSE(ws.contains(0x500));
TEST_ASSERT_EQUAL(1, ws.count());
ws.clear();
TEST_ASSERT_EQUAL(0, ws.count());
}
void test_ws_persistence_roundTrip()
{
WarmNodeStore a;
uint8_t key[32], got[32];
makeKey(key, 0x55);
a.absorb(0x600, 4242, key);
a.absorb(0x601, 4243, NULL);
if (!a.saveIfDirty()) {
TEST_IGNORE_MESSAGE("Filesystem not available in this test environment");
return;
}
WarmNodeStore b;
b.load();
TEST_ASSERT_TRUE(b.contains(0x600));
TEST_ASSERT_TRUE(b.contains(0x601));
TEST_ASSERT_TRUE(b.copyKey(0x600, got));
TEST_ASSERT_EQUAL_MEMORY(key, got, 32);
// Cleanup so reruns start fresh
b.clear();
b.saveIfDirty();
}
WS_TEST_ENTRY void setup()
{
initializeTestEnvironment();
UNITY_BEGIN();
RUN_TEST(test_ws_absorb_and_copyKey_roundTrip);
RUN_TEST(test_ws_keylessEntry_hasNoKey);
RUN_TEST(test_ws_absorb_rejectsNodeNumZero);
RUN_TEST(test_ws_absorb_updatesExistingEntry);
RUN_TEST(test_ws_take_removesEntry);
RUN_TEST(test_ws_keylessCandidate_neverEvictsKeyedEntries);
RUN_TEST(test_ws_keyedCandidate_evictsOldestKeylessFirst);
RUN_TEST(test_ws_keyedCandidate_evictsOldestKeyedWhenNoKeyless);
RUN_TEST(test_ws_remove_and_clear);
RUN_TEST(test_ws_persistence_roundTrip);
exit(UNITY_END());
}
WS_TEST_ENTRY void loop() {}
#else
void setUp(void) {}
void tearDown(void) {}
WS_TEST_ENTRY void setup()
{
initializeTestEnvironment();
UNITY_BEGIN();
exit(UNITY_END());
}
WS_TEST_ENTRY void loop() {}
#endif