From 79a7dcc46c63d507b551ddc2ac13a16067731b23 Mon Sep 17 00:00:00 2001 From: Tom <116762865+NomDeTom@users.noreply.github.com> Date: Thu, 18 Jun 2026 08:41:34 +0100 Subject: [PATCH] Pr1 nodedb warmstore (#10705) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit * NodeDB: 3-tier node store with persistent warm tier (long-tail identity retention) Introduces a tiered NodeDB so the device retains identity (public key, last_heard) for far more nodes than fit in the full-record hot store, without growing heap or the persisted nodes.proto unboundedly. - Hot store: full NodeInfoLite, MAX_NUM_NODES (120 on nRF52). - Satellite maps: position/telemetry/environment/status capped at MAX_SATELLITE_NODES (40 freshest); eviction via enforceSatelliteCaps / evictSatelliteOverCap. - Warm tier (WarmNodeStore): 40 B {num,last_heard,public_key} records for evicted nodes so DMs to/from long-tail nodes keep encrypting/decrypting. Persisted to /prefs/warm.dat, or on nRF52840 a dedicated 12 KB raw-flash record-ring below LittleFS (3x4 KB pages; see linker scripts + the nrf52_warm_region.py post-link guard). NodeDB::getOrCreateMeshNode now demotes evicted nodes into the warm tier and re-admits them (restoring key/last_heard). Router PKI decrypt/encode resolve the peer key via NodeDB::copyPublicKey (hot store, then warm tier). NodeInfoLite gains snr_q4 (sint32, Q4-encoded dB); the float snr is zeroed on disk. NodeInfoLite grows 105 -> 112 B; backup 2432 -> 2468 B. Note: the snr_q4 .proto change still needs to land in the protobufs submodule (generated header is updated here; submodule pointer left at upstream). Co-Authored-By: Claude Opus 4.8 (1M context) * NodeDB: robust receive + retention for blocked (ignored) nodes Hardens how ignored/favourite nodes are received over admin and retained, closing paths where a block could be lost or accidentally cleared. - Blocking keeps the node's public key (admin set_ignored_node and addFromContact no longer zero it / drop the warm-tier key), so a blocked peer stays a verifiable identity. - set_ignored_node creates the node if absent, so a block by node ID sticks even for a node we've never heard from (e.g. pushed by a remote admin) with no NodeInfo or key. - Eviction protection (favourite/ignored/manually-verified) now also applies to the load-time hot-store migration and is never undone by cleanupMeshDB, which previously purged ignored nodes that lacked user info. - The hot-store migration leaves our own node (index 0) in place and prefers to demote non-protected nodes, like the runtime eviction scan. Caps the protected set (favourite + ignored + verified) at MAX_NUM_NODES-2 via NodeDB::setProtectedFlag(), so at least two evictable slots always remain and getOrCreateMeshNode can always make room — replacing the previous unconditional append that could run off the end of the node vector when every node was protected. A locally-set favourite/ignore that hits the cap reports back to the phone via a ClientNotification. Adds test_nodedb_blocked covering the migration, favourite/ignored eviction protection, ignored-survives-cleanup, and the protected-node cap. The maintenance methods stay private in production; the test reaches them through a PIO_UNIT_TESTING-guarded friend shim. Co-Authored-By: Claude Opus 4.8 (1M context) # Conflicts: # src/mesh/NodeDB.h * fix copilot comments * once again * WarmNodeStore: fix cppcheck warnings (uninitvar, constVariablePointer) Zero-initialise `stranded[]` and `seqs[]/order[]` VLAs so cppcheck can verify there are no unguarded reads of uninitialised memory (the guards exist but are not visible to static analysis). Mark two local pointers `const` where the pointed-to entry is never mutated after assignment. Co-Authored-By: Claude Sonnet 4.6 * self-care added to assist 2.7 and 2.8 nodedb migration * Tidy warm-store/self-care: comments, guards, log + flash cleanup Style/cleanup pass over the branch (no behavior change except the noted preprocessor simplifications, which are semantically identical): - Comments: move function descriptions to the headers, cap in-function comments at ~3-4 lines, drop leading-number step markers, label stacked #endif blocks, de-decorate banner comments. - dumpToLog: fully gate decl + definition + AdminModule call site behind MESHTASTIC_NODEDB_MIGRATION_VERBOSE so it compiles out when disabled (~1.2 KB when off). - mesh-pb-constants: drop the dead nRF52832 WARM_NODE_COUNT branch and trim the macro docs. - WarmNodeStore: simplify the redundant `ARCH_NRF52 && NRF52840_XXAA` guards to `NRF52840_XXAA`, add a kNoPage sentinel for the ring page state. - Shorten the always-on LOG_WARN strings (~120 B flash). Co-Authored-By: Claude Opus 4.8 (1M context) * more tidying up, aligning with docs and undoing other-arch regressions * Update protobufs (#19) Co-authored-by: NomDeTom <116762865+NomDeTom@users.noreply.github.com> * made the migration pathway cleareer * address copilot review * fixed a copilot review on a downstream PR. * Address Copilot review comments for PR #10705 (warmstore/nodedb) - WarmNodeStore.h: default MIGRATION_VERBOSE to 0 (suppress info-level chatter on production builds; opt in with =1) - WarmNodeStore.cpp load(): move memset to top of function so all failure paths (header-read fail, invalid header) leave entries clear - WarmNodeStore.cpp save(): replace manual spiLock lock/unlock around mkdir with LockGuard covering the full SafeFile sequence, matching the lock discipline in load() - Router.cpp: memcpy(&p->public_key.bytes, ...) -> memcpy(p->public_key.bytes, ...) — pass decayed uint8_t* rather than pointer-to-array - AdminModule.cpp: check setProtectedFlag return for PKC auto-favorite; log cap-refusal warning instead of unconditional "auto-favoriting" - nrf52_warm_region.py: error message references both v6.ld and v7.ld Co-Authored-By: Claude Sonnet 4.6 * NodeDB: formatting cleanup (blank lines after preprocessor blocks) Co-Authored-By: Claude Sonnet 4.6 * Lukewarm store --------- Co-authored-by: Claude Opus 4.8 (1M context) Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Ben Meadors --- .github/copilot-instructions.md | 19 +- extra_scripts/nrf52_warm_region.py | 70 ++++ src/graphics/draw/MenuHandler.cpp | 17 +- src/mesh/NodeDB.cpp | 463 +++++++++++++++++++-- src/mesh/NodeDB.h | 86 +++- src/mesh/NodeDBLegacyMigration.cpp | 3 + src/mesh/Router.cpp | 30 +- src/mesh/WarmNodeStore.cpp | 548 +++++++++++++++++++++++++ src/mesh/WarmNodeStore.h | 130 ++++++ src/mesh/mesh-pb-constants.h | 66 ++- src/modules/AdminModule.cpp | 41 +- src/platform/nrf52/nrf52840_s140_v6.ld | 46 +++ src/platform/nrf52/nrf52840_s140_v7.ld | 7 +- test/test_nodedb_blocked/test_main.cpp | 193 +++++++++ test/test_warm_store/test_main.cpp | 211 ++++++++++ variants/nrf52840/nrf52.ini | 1 + variants/nrf52840/nrf52840.ini | 4 + 17 files changed, 1848 insertions(+), 87 deletions(-) create mode 100644 extra_scripts/nrf52_warm_region.py create mode 100644 src/mesh/WarmNodeStore.cpp create mode 100644 src/mesh/WarmNodeStore.h create mode 100644 src/platform/nrf52/nrf52840_s140_v6.ld create mode 100644 test/test_nodedb_blocked/test_main.cpp create mode 100644 test/test_warm_store/test_main.cpp diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 0e2f179ca..404eb021c 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -191,7 +191,24 @@ Writers go through `setNodeStatus`, `updatePosition`, `updateTelemetry` (which d ### Eviction -Every code path that drops a node from the header table must also evict the satellites. The single chokepoint is `eraseNodeSatellites(NodeNum)`; it's already called from `getOrCreateMeshNode`'s oldest-boring eviction, `removeNodeByNum`, both branches of `resetNodes`, `cleanupMeshDB`, `addFromContact`'s ignored-branch, and `AdminModule`'s `set_ignored_node`. Add new eviction sites here, not by calling `.erase()` directly. +Every code path that drops a node from the header table must also evict the satellites. The single chokepoint is `eraseNodeSatellites(NodeNum)`; it's already called from `getOrCreateMeshNode`'s oldest-boring eviction, `demoteOldestHotNodesToWarm` (the over-cap warm-tier migration), `removeNodeByNum`, both branches of `resetNodes`, `cleanupMeshDB`, `addFromContact`'s ignored-branch, and `AdminModule`'s `set_ignored_node`. Add new eviction sites here, not by calling `.erase()` directly. (Note: `enforceSatelliteCaps`/`evictSatelliteOverCap` call `.erase()` directly on purpose — that's a satellite-only cap trim where the node _stays_ in the header, a different operation from this chokepoint.) + +### Warm tier (long-tail identity) + +On every arch except STM32WL and bare nRF52832 (`WARM_NODE_COUNT > 0`), a node evicted from the header table is not forgotten outright: `WarmNodeStore` (`src/mesh/WarmNodeStore.{h,cpp}`) keeps a 40 B `{num, last_heard, public_key}` record per evicted node — primarily so PKI DMs to/from a long-tail node keep decrypting without re-running a NodeInfo exchange (the rest of `NodeInfoLite` rebuilds from traffic in seconds). + +- **Write:** `getOrCreateMeshNode`'s eviction and `demoteOldestHotNodesToWarm` (the over-cap boot migration) call `warmStore.absorb(num, last_heard, key)` _before_ the node leaves the header. +- **Read-back:** `getOrCreateMeshNode` calls `warmStore.take()` to rehydrate `last_heard` + key when a warm node is re-admitted; `copyPublicKey()` falls back to the warm tier so the PKI send path finds keys for evicted peers. +- **Persistence:** nRF52840 uses a 12 KB raw-flash record-ring at `0xEA000` (below LittleFS; append + replay + compact-on-rotate, link-guarded by `nrf52840_s140_v7.ld` and `extra_scripts/nrf52_warm_region.py`). Everywhere else: a `/prefs/warm.dat` snapshot flushed by `saveIfDirty()` on the node-DB save cadence. +- **Tunables** (`mesh-pb-constants.h`): `WARM_NODE_COUNT` (per-arch; `0` disables the tier) and `MAX_NUM_NODES` (hot cap — 120 on nRF52840/generic ESP32 to fit the 28 KB LittleFS; ESP32-S3 keeps its flash-scaled 100/200/250, portduino 250). Verbose migration/self-care tracing routes through `LOG_MIGRATION`, gated by `MESHTASTIC_NODEDB_MIGRATION_VERBOSE`. + +### Satellite caps + +Only the freshest `MAX_SATELLITE_NODES` nodes keep satellite payloads; the rest of the header table carries just the `NodeInfoLite`. The cap is **per-platform**: 40 on RAM-constrained parts (nRF52840, generic ESP32) since the four maps live in internal SRAM (not PSRAM, ~408 B/node across the four), and 250 on flash-rich hosts (ESP32-S3, portduino) so every hot node can carry rich data as before the cap existed. `enforceSatelliteCaps()` trims each map to the cap on load (returns whether it trimmed); `evictSatelliteOverCap()` trims before each insert. Eviction is by the owning node's hot `last_heard` (stalest first, demoted/absent nodes rank as `last_heard==0`); self is never trimmed. + +### On-boot self-care + +`NodeDB::nodeDBSelfCare()` runs once identity is established (the constructor after key (re)gen, and `reloadFromDisk()` — _not_ inside `loadFromDisk`, where `getNodeNum()` is still 0). It confirms self is present (warns if a non-empty DB is missing us — a foreign/over-cap file), pins self to index 0, demotes/trims only **non-self** overflow into the warm tier, then rewrites `nodes.proto` **once** and only if it healed something — and never while encrypted storage is locked (it would persist placeholder defaults). `loadFromDisk` deliberately leaves the loaded store untrimmed for this pass. ### Sync flow: thin NodeInfo + post-COMPLETE_ID replay (no opt-in) diff --git a/extra_scripts/nrf52_warm_region.py b/extra_scripts/nrf52_warm_region.py new file mode 100644 index 000000000..e1a942bef --- /dev/null +++ b/extra_scripts/nrf52_warm_region.py @@ -0,0 +1,70 @@ +#!/usr/bin/env python3 +# trunk-ignore-all(ruff/F821) +# trunk-ignore-all(flake8/F821): For SConstruct imports +# +# Post-link guard for the warm-node-store raw-flash region on nRF52840. +# +# The 3 app-region pages below LittleFS (0xEA000-0xED000, reclaimed by whole-image +# LTO) are reserved for the WarmNodeStore record-ring (see WarmNodeStore.h). Our +# linker scripts (nrf52840_s140_v6.ld and nrf52840_s140_v7.ld) cap the image at +# 0xEA000, but boards on the framework-default script (FLASH ending at 0xED000) could +# silently place code in those pages — the first warm-store save would then brick the +# device. This turns that into a build failure. +# +# Image flash end = __etext + sizeof(.data) (loaded at LMA __etext); symbols from +# the framework's nrf52_common.ld. +import os + +Import("env") + +WARM_REGION_BASE = 0xEA000 # keep in sync with WARM_FLASH_REGION_BASE in WarmNodeStore.h (3 x 4 KB record-ring) + +_tc = env.PioPlatform().get_package_dir("toolchain-gccarmnoneeabi") or "" +_NM = os.path.join(_tc, "bin", "arm-none-eabi-nm") +if not os.path.isfile(_NM): + _NM = "arm-none-eabi-nm" # fall back to PATH + + +def _assert_warm_region_clear(source, target, env): + import subprocess + import sys + + try: + elf = env.subst("$BUILD_DIR/${PROGNAME}.elf") + out = subprocess.check_output([_NM, elf], universal_newlines=True) + except Exception as exc: # tooling hiccup: warn loudly, don't wedge the build + print("nrf52_warm_region: WARNING - guard skipped (nm failed: %s)" % exc) + return + + syms = {} + for line in out.split("\n"): + f = line.split() + if len(f) >= 3 and f[-1] in ("__etext", "__data_start__", "__data_end__"): + syms[f[-1]] = int(f[0], 16) + if len(syms) != 3: + print("nrf52_warm_region: WARNING - guard skipped (linker symbols not found)") + return + + flash_end = syms["__etext"] + (syms["__data_end__"] - syms["__data_start__"]) + if flash_end > WARM_REGION_BASE: + sys.stderr.write( + "\n*** nrf52 warm-region guard: image ends at 0x%X, past the reserved " + "warm-store region at 0x%X ***\n" + "The 12 KB region at 0xEA000 holds the WarmNodeStore record-ring; a warm-store\n" + "save would overwrite this firmware's tail. Shrink the image, or shrink/move\n" + "the region (WARM_FLASH_REGION_BASE in src/mesh/WarmNodeStore.h, the FLASH\n" + "LENGTH in src/platform/nrf52/nrf52840_s140_v6.ld and _v7.ld, and this guard).\n\n" + % (flash_end, WARM_REGION_BASE) + ) + from SCons.Script import Exit + + Exit(1) + print( + "nrf52_warm_region: guard OK -- image ends at 0x%X, %d KB clear of the warm region" + % (flash_end, (WARM_REGION_BASE - flash_end) // 1024) + ) + + +# Attach to the phony "buildprog" alias (not the .elf node) so the guard runs +# on incremental relinks too -- same reasoning as nrf52_lto.py's guard. +env.AddPostAction("buildprog", _assert_warm_region_clear) diff --git a/src/graphics/draw/MenuHandler.cpp b/src/graphics/draw/MenuHandler.cpp index 365739112..0f4d8d713 100644 --- a/src/graphics/draw/MenuHandler.cpp +++ b/src/graphics/draw/MenuHandler.cpp @@ -1572,6 +1572,7 @@ void menuHandler::manageNodeMenu() nodeDB->set_favorite(false, menuHandler::pickedNodeNum); } else { LOG_INFO("Adding node %08X to favorites", menuHandler::pickedNodeNum); + // set_favorite() already logs PROTECTED_CAP_WARN_FMT on a cap refusal; don't double-log here. nodeDB->set_favorite(true, menuHandler::pickedNodeNum); } screen->setFrames(graphics::Screen::FOCUS_PRESERVE); @@ -1615,15 +1616,23 @@ void menuHandler::manageNodeMenu() return; } + bool changed = false; if (nodeInfoLiteIsIgnored(n)) { nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_IGNORED_MASK, false); LOG_INFO("Unignoring node %08X", menuHandler::pickedNodeNum); - } else { - nodeInfoLiteSetBit(n, NODEINFO_BITFIELD_IS_IGNORED_MASK, true); + changed = true; + } else if (nodeDB->setProtectedFlag(n, NODEINFO_BITFIELD_IS_IGNORED_MASK, true)) { LOG_INFO("Ignoring node %08X", menuHandler::pickedNodeNum); + changed = true; + } else { + LOG_WARN(NodeDB::PROTECTED_CAP_WARN_FMT, "ignore", menuHandler::pickedNodeNum, MAX_NUM_NODES - 2); + } + // Only persist/notify when the ignore bit actually moved; a cap + // refusal changed nothing and shouldn't trigger a prefs save. + if (changed) { + nodeDB->notifyObservers(true); + nodeDB->saveToDisk(); } - nodeDB->notifyObservers(true); - nodeDB->saveToDisk(); screen->setFrames(graphics::Screen::FOCUS_PRESERVE); return; } diff --git a/src/mesh/NodeDB.cpp b/src/mesh/NodeDB.cpp index 90eec34f8..38025ad10 100644 --- a/src/mesh/NodeDB.cpp +++ b/src/mesh/NodeDB.cpp @@ -197,6 +197,8 @@ bool meshtastic_NodeDatabase_callback(pb_istream_t *istream, pb_ostream_t *ostre if (ostream) { const auto *vec = static_cast *>(iter->pData); for (auto item : *vec) { + item.snr_q4 = (int32_t)(item.snr * 4.0f); + item.snr = 0.0f; if (!pb_encode_tag_for_field(ostream, iter)) return false; if (!pb_encode_submessage(ostream, meshtastic_NodeInfoLite_fields, &item)) @@ -206,8 +208,12 @@ bool meshtastic_NodeDatabase_callback(pb_istream_t *istream, pb_ostream_t *ostre if (istream && istream->bytes_left) { meshtastic_NodeInfoLite node = meshtastic_NodeInfoLite_init_zero; auto *vec = static_cast *>(iter->pData); - if (pb_decode(istream, meshtastic_NodeInfoLite_fields, &node)) + if (pb_decode(istream, meshtastic_NodeInfoLite_fields, &node)) { + if (node.snr_q4) + node.snr = node.snr_q4 / 4.0f; + node.snr_q4 = 0; vec->push_back(node); + } } return true; } @@ -482,9 +488,17 @@ NodeDB::NodeDB() myNodeInfo.my_node_num = crc32Buffer(config.security.public_key.bytes, config.security.public_key.size); } #endif - // Include our owner in the node db under our nodenum - meshtastic_NodeInfoLite *info = getOrCreateMeshNode(getNodeNum()); - TypeConversions::CopyUserToNodeInfoLite(info, owner); + // Identity is now established, so run the self-care pass on the store + // loadFromDisk() deliberately left untrimmed: confirm self, trim/demote only + // non-self overflow, pin self to index 0, rewrite once if healed. + nodeDBSelfCare(); + + // If we migrated from legacy during loadFromDisk(), persist the migrated DB + // only after identity and self-care are established. + if (migrationSavePending) { + saveNodeDatabaseToDisk(); + migrationSavePending = false; + } // If node database has not been saved for the first time, save it now #ifdef FSCom @@ -689,6 +703,39 @@ template std::vector snapshotSatelliteNodeNums(const Map } return result; } + +// Drop the stalest entry of `map` (staleness proxied via the owner's +// last_heard; 0 = owner evicted, i.e. an orphan — first out). Never evicts our +// own node's entry. Caller holds satelliteMutex. Returns false if nothing +// could be evicted. +template bool evictStalestSatellite(NodeDB &db, Map &map) +{ + auto victim = map.end(); + uint32_t victimTs = UINT32_MAX; + for (auto it = map.begin(); it != map.end(); ++it) { + if (it->first == db.getNodeNum()) + continue; + uint32_t ts = db.hotNodeLastHeard(it->first); + if (ts < victimTs) { + victimTs = ts; + victim = it; + } + } + if (victim == map.end()) + return false; + map.erase(victim); + return true; +} + +// Keep `map` within MAX_SATELLITE_NODES ahead of inserting `incoming` (the +// tier-1/tier-2 split: only the freshest MAX_SATELLITE_NODES nodes carry +// satellite payloads). Caller holds satelliteMutex. +template void evictSatelliteOverCap(NodeDB &db, Map &map, NodeNum incoming) +{ + if (map.size() < MAX_SATELLITE_NODES || map.count(incoming)) + return; + evictStalestSatellite(db, map); +} } // namespace void NodeDB::resetRadioConfig(bool is_fresh_install) @@ -721,6 +768,7 @@ bool NodeDB::factoryReset(bool eraseBleBonds) LOG_ERROR("Could not remove rangetest.csv file"); } #endif + spiLock->unlock(); // rmDir above nuked the .dat file, but TransmitHistory's in-memory @@ -731,6 +779,14 @@ bool NodeDB::factoryReset(bool eraseBleBonds) #if HAS_SCREEN messageStore.clearAllMessages(); #endif + +#if WARM_NODE_COUNT > 0 + // On nRF52840 the warm tier lives in raw flash outside /prefs, so rmDir + // didn't touch it; clear it and persist the empty store. + warmStore.clear(); + warmStore.saveIfDirty(); +#endif + // second, install default state (this will deal with the duplicate mac address issue) installDefaultNodeDatabase(); installDefaultDeviceState(); @@ -745,6 +801,7 @@ bool NodeDB::factoryReset(bool eraseBleBonds) // This will erase what's in NVS including ssl keys, persistent variables and ble pairing nvs_flash_erase(); #endif + #ifdef ARCH_NRF52 LOG_INFO("Clear bluetooth bonds!"); bond_print_list(BLE_GAP_ROLE_PERIPH); @@ -767,12 +824,15 @@ void NodeDB::installDefaultNodeDatabase() #if !MESHTASTIC_EXCLUDE_POSITIONDB nodePositions.clear(); #endif + #if !MESHTASTIC_EXCLUDE_TELEMETRYDB nodeTelemetry.clear(); #endif + #if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB nodeEnvironment.clear(); #endif + #if !MESHTASTIC_EXCLUDE_STATUSDB nodeStatus.clear(); #endif @@ -835,34 +895,42 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) #else config.lora.region = meshtastic_Config_LoRaConfig_RegionCode_UNSET; #endif + #ifdef USERPREFS_LORACONFIG_MODEM_PRESET config.lora.modem_preset = USERPREFS_LORACONFIG_MODEM_PRESET; #else config.lora.modem_preset = meshtastic_Config_LoRaConfig_ModemPreset_LONG_FAST; #endif + #ifdef USERPREFS_LORACONFIG_USE_PRESET config.lora.use_preset = USERPREFS_LORACONFIG_USE_PRESET; #else config.lora.use_preset = true; #endif + #ifdef USERPREFS_LORACONFIG_BANDWIDTH config.lora.bandwidth = USERPREFS_LORACONFIG_BANDWIDTH; #endif + #ifdef USERPREFS_LORACONFIG_SPREAD_FACTOR config.lora.spread_factor = USERPREFS_LORACONFIG_SPREAD_FACTOR; #endif + #ifdef USERPREFS_LORACONFIG_CODING_RATE config.lora.coding_rate = USERPREFS_LORACONFIG_CODING_RATE; #endif + #ifdef USERPREFS_LORACONFIG_OVERRIDE_FREQUENCY config.lora.override_frequency = USERPREFS_LORACONFIG_OVERRIDE_FREQUENCY; #endif + config.lora.hop_limit = HOP_RELIABLE; #ifdef USERPREFS_CONFIG_LORA_IGNORE_MQTT config.lora.ignore_mqtt = USERPREFS_CONFIG_LORA_IGNORE_MQTT; #else config.lora.ignore_mqtt = false; #endif + // Initialize admin_key_count to zero byte numAdminKeys = 0; @@ -892,6 +960,7 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) numAdminKeys++; } #endif + config.security.admin_key_count = numAdminKeys; if (shouldPreserveKey) { @@ -906,6 +975,7 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) #ifdef PIN_GPS_EN config.position.gps_en_gpio = PIN_GPS_EN; #endif + #if defined(USERPREFS_CONFIG_GPS_MODE) config.position.gps_mode = USERPREFS_CONFIG_GPS_MODE; #elif !HAS_GPS || GPS_DEFAULT_NOT_PRESENT @@ -918,11 +988,13 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) #else config.position.gps_mode = meshtastic_Config_PositionConfig_GpsMode_ENABLED; #endif + #ifdef USERPREFS_CONFIG_SMART_POSITION_ENABLED config.position.position_broadcast_smart_enabled = USERPREFS_CONFIG_SMART_POSITION_ENABLED; #else config.position.position_broadcast_smart_enabled = true; #endif + config.position.broadcast_smart_minimum_distance = 100; config.position.broadcast_smart_minimum_interval_secs = default_broadcast_smart_minimum_interval_secs; if (config.device.role != meshtastic_Config_DeviceConfig_Role_ROUTER && @@ -942,6 +1014,7 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) // default to bluetooth capability of platform as default config.bluetooth.enabled = true; #endif + config.bluetooth.fixed_pin = defaultBLEPin; #if defined(ST7735_CS) || defined(USE_EINK) || defined(ILI9341_DRIVER) || defined(ILI9342_DRIVER) || defined(ST7789_CS) || \ @@ -953,7 +1026,7 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) if (st7789_id == 0xFFFFFF) { hasScreen = false; } -#endif +#endif // HELTEC_MESH_NODE_T114 #elif ARCH_PORTDUINO bool hasScreen = false; if (portduino_config.displayPanel) @@ -973,6 +1046,7 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) config.bluetooth.mode = hasScreen ? meshtastic_Config_BluetoothConfig_PairingMode_RANDOM_PIN : meshtastic_Config_BluetoothConfig_PairingMode_FIXED_PIN; #endif + // for backward compat, default position flags are ALT+MSL config.position.position_flags = (meshtastic_Config_PositionConfig_PositionFlags_ALTITUDE | meshtastic_Config_PositionConfig_PositionFlags_ALTITUDE_MSL | @@ -985,8 +1059,8 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) config.network.enabled_protocols = USERPREFS_NETWORK_ENABLED_PROTOCOLS; #else config.network.enabled_protocols = 0; -#endif -#endif +#endif // Network enabled protocols +#endif // UDP Multicast #ifdef USERPREFS_NETWORK_WIFI_ENABLED config.network.wifi_enabled = USERPREFS_NETWORK_WIFI_ENABLED; @@ -1013,16 +1087,20 @@ void NodeDB::installDefaultConfig(bool preserveKey = false) #ifdef DISPLAY_FLIP_SCREEN config.display.flip_screen = true; #endif + #ifdef RAK4630 config.display.wake_on_tap_or_motion = true; #endif + #if defined(T_WATCH_S3) || defined(SENSECAP_INDICATOR) config.display.screen_on_secs = 30; config.display.wake_on_tap_or_motion = true; #endif + #ifdef COMPASS_ORIENTATION config.display.compass_orientation = COMPASS_ORIENTATION; #endif + #if defined(ARCH_ESP32) && !MESHTASTIC_EXCLUDE_WIFI if (MeshtasticOTA::isUpdated()) { MeshtasticOTA::recoverConfig(&config.network); @@ -1046,6 +1124,7 @@ void NodeDB::initConfigIntervals() #else config.position.gps_update_interval = default_gps_update_interval; #endif + #ifdef USERPREFS_CONFIG_POSITION_BROADCAST_INTERVAL config.position.position_broadcast_secs = USERPREFS_CONFIG_POSITION_BROADCAST_INTERVAL; #else @@ -1082,22 +1161,26 @@ void NodeDB::installDefaultModuleConfig() defined(NEOPIXEL_STATUS_NOTIFICATION_PIN) moduleConfig.external_notification.enabled = true; #endif + #if defined(PIN_BUZZER) moduleConfig.external_notification.output_buzzer = PIN_BUZZER; moduleConfig.external_notification.use_pwm = true; moduleConfig.external_notification.alert_message_buzzer = true; #endif + #if defined(PIN_VIBRATION) moduleConfig.external_notification.output_vibra = PIN_VIBRATION; moduleConfig.external_notification.alert_message_vibra = true; moduleConfig.external_notification.output_ms = 500; #endif + #if defined(LED_NOTIFICATION) moduleConfig.external_notification.output = LED_NOTIFICATION; moduleConfig.external_notification.active = LED_STATE_ON; moduleConfig.external_notification.alert_message = true; moduleConfig.external_notification.output_ms = 1000; #endif + #if defined(PIN_VIBRATION) moduleConfig.external_notification.nag_timeout = 2; #elif defined(PIN_BUZZER) || defined(LED_NOTIFICATION) || defined(NEOPIXEL_STATUS_NOTIFICATION_PIN) @@ -1114,14 +1197,16 @@ void NodeDB::installDefaultModuleConfig() moduleConfig.external_notification.nag_timeout = 0; #else moduleConfig.external_notification.nag_timeout = default_ringtone_nag_secs; -#endif -#endif +#endif // HAS_TFT +#endif // HAS_I2S + #ifdef NANO_G2_ULTRA moduleConfig.external_notification.enabled = true; moduleConfig.external_notification.alert_message = true; moduleConfig.external_notification.output_ms = 100; moduleConfig.external_notification.active = true; -#endif +#endif // NANO_G2_ULTRA + #ifdef T_LORA_PAGER moduleConfig.canned_message.updown1_enabled = true; moduleConfig.canned_message.inputbroker_pin_a = ROTARY_A; @@ -1131,35 +1216,43 @@ void NodeDB::installDefaultModuleConfig() moduleConfig.canned_message.inputbroker_event_ccw = meshtastic_ModuleConfig_CannedMessageConfig_InputEventChar(29); moduleConfig.canned_message.inputbroker_event_press = meshtastic_ModuleConfig_CannedMessageConfig_InputEventChar_SELECT; #endif + moduleConfig.has_canned_message = true; + #if USERPREFS_MQTT_ENABLED && !MESHTASTIC_EXCLUDE_MQTT moduleConfig.mqtt.enabled = true; #endif + #ifdef USERPREFS_MQTT_ADDRESS strncpy(moduleConfig.mqtt.address, USERPREFS_MQTT_ADDRESS, sizeof(moduleConfig.mqtt.address)); #else strncpy(moduleConfig.mqtt.address, default_mqtt_address, sizeof(moduleConfig.mqtt.address)); #endif + #ifdef USERPREFS_MQTT_USERNAME strncpy(moduleConfig.mqtt.username, USERPREFS_MQTT_USERNAME, sizeof(moduleConfig.mqtt.username)); #else strncpy(moduleConfig.mqtt.username, default_mqtt_username, sizeof(moduleConfig.mqtt.username)); #endif + #ifdef USERPREFS_MQTT_PASSWORD strncpy(moduleConfig.mqtt.password, USERPREFS_MQTT_PASSWORD, sizeof(moduleConfig.mqtt.password)); #else strncpy(moduleConfig.mqtt.password, default_mqtt_password, sizeof(moduleConfig.mqtt.password)); #endif + #ifdef USERPREFS_MQTT_ROOT_TOPIC strncpy(moduleConfig.mqtt.root, USERPREFS_MQTT_ROOT_TOPIC, sizeof(moduleConfig.mqtt.root)); #else strncpy(moduleConfig.mqtt.root, default_mqtt_root, sizeof(moduleConfig.mqtt.root)); #endif + #ifdef USERPREFS_MQTT_ENCRYPTION_ENABLED moduleConfig.mqtt.encryption_enabled = USERPREFS_MQTT_ENCRYPTION_ENABLED; #else moduleConfig.mqtt.encryption_enabled = default_mqtt_encryption_enabled; #endif + #ifdef USERPREFS_MQTT_TLS_ENABLED moduleConfig.mqtt.tls_enabled = USERPREFS_MQTT_TLS_ENABLED; #else @@ -1253,11 +1346,13 @@ void NodeDB::initModuleConfigIntervals() #else moduleConfig.telemetry.device_update_interval = MAX_INTERVAL; #endif + #ifdef USERPREFS_CONFIG_ENV_TELEM_UPDATE_INTERVAL moduleConfig.telemetry.environment_update_interval = USERPREFS_CONFIG_ENV_TELEM_UPDATE_INTERVAL; #else moduleConfig.telemetry.environment_update_interval = 0; #endif + #ifdef USERPREFS_CONFIG_ENVIRONMENT_MEASUREMENT_ENABLED moduleConfig.telemetry.environment_measurement_enabled = USERPREFS_CONFIG_ENVIRONMENT_MEASUREMENT_ENABLED; #endif @@ -1302,6 +1397,10 @@ void NodeDB::resetNodes(bool keepFavorites) std::fill(nodeDatabase.nodes.begin() + 1, nodeDatabase.nodes.end(), meshtastic_NodeInfoLite()); } (void)ourNum; +#if WARM_NODE_COUNT > 0 + warmStore.clear(); // warm entries are never favorites; a DB reset clears them too +#endif + devicestate.has_rx_waypoint = false; saveNodeDatabaseToDisk(); saveDeviceStateToDisk(); @@ -1323,6 +1422,11 @@ void NodeDB::removeNodeByNum(NodeNum nodeNum) meshtastic_NodeInfoLite()); if (removed) eraseNodeSatellites(nodeNum); +#if WARM_NODE_COUNT > 0 + // Explicit user removal: don't let the warm tier resurrect the node + warmStore.remove(nodeNum); +#endif + LOG_DEBUG("NodeDB::removeNodeByNum purged %d entries. Save changes", removed); saveNodeDatabaseToDisk(); } @@ -1436,6 +1540,7 @@ void NodeDB::setNodeStatus(NodeNum n, const meshtastic_StatusMessage &status) (void)status; #else concurrency::LockGuard guard(&satelliteMutex); + evictSatelliteOverCap(*this, nodeStatus, n); nodeStatus[n] = status; #endif } @@ -1447,6 +1552,7 @@ void NodeDB::touchNodePositionTime(NodeNum n, uint32_t time) (void)time; #else concurrency::LockGuard guard(&satelliteMutex); + evictSatelliteOverCap(*this, nodePositions, n); nodePositions[n].time = time; #endif } @@ -1457,23 +1563,65 @@ void NodeDB::eraseNodeSatellites(NodeNum n) #if !MESHTASTIC_EXCLUDE_POSITIONDB nodePositions.erase(n); #endif + #if !MESHTASTIC_EXCLUDE_TELEMETRYDB nodeTelemetry.erase(n); #endif + #if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB nodeEnvironment.erase(n); #endif + #if !MESHTASTIC_EXCLUDE_STATUSDB nodeStatus.erase(n); #endif } +bool NodeDB::enforceSatelliteCaps() +{ + concurrency::LockGuard guard(&satelliteMutex); + bool trimmedAny = false; + auto trim = [this, &trimmedAny](auto &map, const char *name) { + const size_t before = map.size(); + while (map.size() > MAX_SATELLITE_NODES) { + if (!evictStalestSatellite(*this, map)) + break; + } + if (map.size() != before) { + trimmedAny = true; + LOG_MIGRATION("Trimmed %s satellites %u -> %u (cap %d)", name, (unsigned)before, (unsigned)map.size(), + MAX_SATELLITE_NODES); + } + }; +#if !MESHTASTIC_EXCLUDE_POSITIONDB + trim(nodePositions, "position"); +#endif + +#if !MESHTASTIC_EXCLUDE_TELEMETRYDB + trim(nodeTelemetry, "telemetry"); +#endif + +#if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB + trim(nodeEnvironment, "environment"); +#endif + +#if !MESHTASTIC_EXCLUDE_STATUSDB + trim(nodeStatus, "status"); +#endif + + (void)trim; // all four maps may be compiled out + return trimmedAny; +} + void NodeDB::cleanupMeshDB() { int newPos = 0, removed = 0; for (int i = 0; i < numMeshNodes; i++) { meshtastic_NodeInfoLite &n = meshNodes->at(i); - if (nodeInfoLiteHasUser(&n)) { + // Keep ignored (blocked) nodes even without user info: a block set by + // bare node ID has no NodeInfo and would otherwise be purged here, + // silently dropping the block. + if (nodeInfoLiteHasUser(&n) || nodeInfoLiteIsIgnored(&n)) { if (n.public_key.size > 0) { if (memfll(n.public_key.bytes, 0, n.public_key.size)) { n.public_key.size = 0; @@ -1486,8 +1634,16 @@ void NodeDB::cleanupMeshDB() } else { // No user info - drop this node and its satellites const NodeNum gone = n.num; - if (gone) + if (gone) { +#if WARM_NODE_COUNT > 0 + // Keep any key we learned (e.g. via a DM before the NodeInfo + // exchange completed) rather than losing it with the purge. + if (n.public_key.size == 32) + warmStore.absorb(gone, n.last_heard, n.public_key.bytes); +#endif + eraseNodeSatellites(gone); + } removed++; } } @@ -1518,12 +1674,15 @@ void NodeDB::installDefaultDeviceState() #else snprintf(owner.long_name, sizeof(owner.long_name), "Meshtastic %04x", getNodeNum() & 0x0ffff); #endif + clampLongName(owner.long_name); // vendor userprefs may exceed the local cap + #ifdef USERPREFS_CONFIG_OWNER_SHORT_NAME snprintf(owner.short_name, sizeof(owner.short_name), (const char *)USERPREFS_CONFIG_OWNER_SHORT_NAME); #else snprintf(owner.short_name, sizeof(owner.short_name), "%04x", getNodeNum() & 0x0ffff); #endif + snprintf(owner.id, sizeof(owner.id), "!%08x", getNodeNum()); // Default node ID now based on nodenum memcpy(owner.macaddr, ourMacAddr, sizeof(owner.macaddr)); owner.has_is_unmessagable = true; @@ -1637,6 +1796,114 @@ LoadFileResult NodeDB::loadProto(const char *filename, size_t protoSize, size_t return state; } +#if WARM_NODE_COUNT > 0 +void NodeDB::demoteOldestHotNodesToWarm() +{ + const int keep = MAX_NUM_NODES; + if (numMeshNodes <= keep) + return; + + // Protected nodes (favorite/ignored/verified) outrank recency and are demoted + // only when the store is full of them; within a class, most-recently-heard + // wins. Index 0 is self and stays put (sort from +1), as in runtime eviction. + std::sort(meshNodes->begin() + 1, meshNodes->begin() + numMeshNodes, + [](const meshtastic_NodeInfoLite &a, const meshtastic_NodeInfoLite &b) { + const bool ka = nodeInfoLiteIsProtected(&a); + const bool kb = nodeInfoLiteIsProtected(&b); + if (ka != kb) + return ka; + return a.last_heard > b.last_heard; + }); + + int demoted = 0; + for (int i = keep; i < numMeshNodes; i++) { + const meshtastic_NodeInfoLite &n = (*meshNodes)[i]; + if (n.num == 0) + continue; + // Keep the public key if we have one (40 B warm record); keyless nodes + // still get a placeholder so re-admission restores last_heard. + warmStore.absorb(n.num, n.last_heard, n.public_key.size > 0 ? n.public_key.bytes : nullptr); + // Demotion drops the node from the header table, so drop its satellites + // too (the eviction chokepoint) — they'd otherwise orphan until the next + // enforceSatelliteCaps pass. + eraseNodeSatellites(n.num); + demoted++; + } + numMeshNodes = keep; // the resize() in loadFromDisk reclaims the demoted tail + LOG_MIGRATION("NodeDB migration: demoted %d node(s) over %d into the warm tier (keepers preferred)", demoted, keep); +} +#endif + +void NodeDB::nodeDBSelfCare() +{ + if (!meshNodes) + return; + + const NodeNum self = getNodeNum(); + const bool nodesOverCap = numMeshNodes > MAX_NUM_NODES; + + // Confirm self is present and its key matches what we just (re)derived. A + // non-empty DB that doesn't contain us means a foreign/over-cap or corrupt + // nodes.proto was loaded; an empty DB is just a fresh device (no warning). + meshtastic_NodeInfoLite *selfNode = getMeshNode(self); + if (!selfNode && numMeshNodes > 0) { + LOG_WARN("NodeDB self-care: self 0x%08x absent from DB, re-adding", (unsigned)self); + } else if (selfNode && owner.public_key.size == 32 && selfNode->public_key.size == 32 && + memcmp(selfNode->public_key.bytes, owner.public_key.bytes, 32) != 0) { + LOG_WARN("NodeDB self-care: self 0x%08x key mismatch, refreshing", (unsigned)self); + } + + // Maintenance that must never touch self. Pin self to index 0 first so + // the positional demote/eviction scans (which skip index 0) provably exclude + // us, wherever the loaded file happened to place our row. + if (selfNode && numMeshNodes > 0 && selfNode != &meshNodes->at(0)) { + std::swap(meshNodes->at(0), *selfNode); + } + +#if WARM_NODE_COUNT > 0 + if (nodesOverCap) + demoteOldestHotNodesToWarm(); // demotes oldest NON-self overflow; index 0 (us) left in place +#endif + + if (numMeshNodes > MAX_NUM_NODES) { + LOG_WARN("NodeDB self-care: %d over cap %d, truncating", numMeshNodes, MAX_NUM_NODES); + numMeshNodes = MAX_NUM_NODES; + } + // Normalise the backing store to the hot cap so getOrCreateMeshNode always + // has spare slots to append into (it indexes meshNodes->at(numMeshNodes++)). + meshNodes->resize(MAX_NUM_NODES); + + const bool satsTrimmed = enforceSatelliteCaps(); + + // Ensure self exists, sits at index 0, and carries current owner info — after + // any demotion has freed a slot. Covers the foreign/fixture case where the + // loaded file did not contain us at all. + meshtastic_NodeInfoLite *info = getOrCreateMeshNode(self); + if (info) { + TypeConversions::CopyUserToNodeInfoLite(info, owner); + if (info != &meshNodes->at(0)) + std::swap(meshNodes->at(0), *info); + } + + // One-shot rewrite: only when we healed something, and never while storage + // is locked — a locked boot loads placeholder defaults that must not be written + // over the encrypted store; reloadFromDisk() re-runs self-care once unlocked. +#ifdef MESHTASTIC_ENCRYPTED_STORAGE + const bool storageLocked = EncryptedStorage::isLockdownActive() && !EncryptedStorage::isUnlocked(); +#else + const bool storageLocked = false; +#endif + + if ((nodesOverCap || satsTrimmed) && !storageLocked) { + LOG_MIGRATION("NodeDB self-care: healed store (nodes-over-cap:%s sats-trimmed:%s); rewriting nodes.proto once", + nodesOverCap ? "yes" : "no", satsTrimmed ? "yes" : "no"); + saveNodeDatabaseToDisk(); +#if WARM_NODE_COUNT > 0 + warmStore.saveIfDirty(); +#endif + } +} + void NodeDB::loadFromDisk() { // Mark the current device state as completely unusable, so that if we fail reading the entire file from @@ -1650,6 +1917,8 @@ void NodeDB::loadFromDisk() storageCorruptThisLoad = false; #endif + migrationSavePending = false; + meshtastic_Config_SecurityConfig backupSecurity = meshtastic_Config_SecurityConfig_init_zero; #ifdef ARCH_ESP32 @@ -1659,6 +1928,7 @@ void NodeDB::loadFromDisk() rmDir("/static/static"); // Remove bad static web files bundle from initial 2.5.13 release spiLock->unlock(); #endif + #ifdef FSCom #if defined(FACTORY_INSTALL) && !defined(ARCH_PORTDUINO) spiLock->lock(); @@ -1673,7 +1943,7 @@ void NodeDB::loadFromDisk() } } spiLock->unlock(); -#endif +#endif // FACTORY_INSTALL, not PORTDUINO spiLock->lock(); if (FSCom.exists(legacyPrefFileName)) { spiLock->unlock(); @@ -1691,7 +1961,8 @@ void NodeDB::loadFromDisk() spiLock->unlock(); } -#endif +#endif // FSCom + #ifdef MESHTASTIC_ENCRYPTED_STORAGE // Only take the locked-boot defaults path when lockdown is ACTIVE (the // device is provisioned) AND storage is still locked. A lockdown-capable @@ -1745,7 +2016,7 @@ void NodeDB::loadFromDisk() installDefaultNodeDatabase(); } else if (nodeDatabase.version < DEVICESTATE_CUR_VER) { if (migrateLegacyNodeDatabase()) - saveNodeDatabaseToDisk(); + migrationSavePending = true; else installDefaultNodeDatabase(); } else { @@ -1758,33 +2029,40 @@ void NodeDB::loadFromDisk() #else 0u; #endif + const unsigned telCount = #if !MESHTASTIC_EXCLUDE_TELEMETRYDB (unsigned)nodeTelemetry.size(); #else 0u; #endif + const unsigned envCount = #if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB (unsigned)nodeEnvironment.size(); #else 0u; #endif + const unsigned statusCount = #if !MESHTASTIC_EXCLUDE_STATUSDB (unsigned)nodeStatus.size(); #else 0u; #endif + LOG_INFO("Loaded saved nodedatabase v%d: %d nodes, %u pos, %u tel, %u env, %u status", nodeDatabase.version, nodeDatabase.nodes.size(), posCount, telCount, envCount, statusCount); } - if (numMeshNodes > MAX_NUM_NODES) { - LOG_WARN("Node count %d exceeds MAX_NUM_NODES %d, truncating", numMeshNodes, MAX_NUM_NODES); - numMeshNodes = MAX_NUM_NODES; - } - meshNodes->resize(MAX_NUM_NODES); + // Left UNTRIMMED on purpose: trim/demote/satellite-cap/self-pin/rewrite all + // run in nodeDBSelfCare() once getNodeNum() is valid (still 0 here on a cold + // boot, so we could only assume index 0 == self — the very bug being fixed). +#if WARM_NODE_COUNT > 0 + // Load the warm tier so its on-disk snapshot is available before the node DB + // is exercised (and before nodeDBSelfCare() demotes any overflow into it). + warmStore.load(); +#endif // static DeviceState scratch; We no longer read into a tempbuf because this structure is 15KB of valuable RAM state = loadProto(deviceStateFileName, meshtastic_DeviceState_size, sizeof(meshtastic_DeviceState), @@ -1856,18 +2134,23 @@ void NodeDB::loadFromDisk() #ifdef USERPREFS_CONFIG_LORA_REGION config.lora.region = USERPREFS_CONFIG_LORA_REGION; #endif + #ifdef USERPREFS_LORACONFIG_USE_PRESET config.lora.use_preset = USERPREFS_LORACONFIG_USE_PRESET; #endif + #ifdef USERPREFS_LORACONFIG_BANDWIDTH config.lora.bandwidth = USERPREFS_LORACONFIG_BANDWIDTH; #endif + #ifdef USERPREFS_LORACONFIG_SPREAD_FACTOR config.lora.spread_factor = USERPREFS_LORACONFIG_SPREAD_FACTOR; #endif + #ifdef USERPREFS_LORACONFIG_CODING_RATE config.lora.coding_rate = USERPREFS_LORACONFIG_CODING_RATE; #endif + #ifdef USERPREFS_LORACONFIG_OVERRIDE_FREQUENCY config.lora.override_frequency = USERPREFS_LORACONFIG_OVERRIDE_FREQUENCY; #endif @@ -1884,6 +2167,7 @@ void NodeDB::loadFromDisk() #if defined(USERPREFS_USE_ADMIN_KEY_0) || defined(USERPREFS_USE_ADMIN_KEY_1) || defined(USERPREFS_USE_ADMIN_KEY_2) uint16_t sum = 0; #endif + #ifdef USERPREFS_USE_ADMIN_KEY_0 for (uint8_t b = 0; b < 32; b++) { @@ -2076,6 +2360,17 @@ bool NodeDB::reloadFromDisk() return false; } + // loadFromDisk() leaves the store untrimmed; run self-care now (getNodeNum() + // is valid at runtime) to trim/demote non-self overflow, pin self to index 0 + // and normalise the backing store before the node DB is exercised again. + nodeDBSelfCare(); + + // Preserve constructor ordering: persist any migration only after self-care. + if (migrationSavePending) { + saveNodeDatabaseToDisk(); + migrationSavePending = false; + } + // Push the now-real config to the radio. if (rIface) { channels.onConfigChanged(); @@ -2196,6 +2491,7 @@ bool NodeDB::saveChannelsToDisk() FSCom.mkdir("/prefs"); spiLock->unlock(); #endif + return saveProto(channelFileName, meshtastic_ChannelFile_size, &meshtastic_ChannelFile_msg, &channelFile); } @@ -2251,6 +2547,7 @@ bool NodeDB::saveNodeDatabaseToDisk() FSCom.mkdir("/prefs"); spiLock->unlock(); #endif + // Project the maps into the on-disk vectors just before encoding; cleared // again on the way out so we don't carry duplicate state. concurrency::LockGuard guard(&satelliteMutex); @@ -2267,6 +2564,7 @@ bool NodeDB::saveNodeDatabaseToDisk() #else nodeDatabase.positions.clear(); #endif + #if !MESHTASTIC_EXCLUDE_TELEMETRYDB nodeDatabase.telemetry.clear(); nodeDatabase.telemetry.reserve(nodeTelemetry.size()); @@ -2280,6 +2578,7 @@ bool NodeDB::saveNodeDatabaseToDisk() #else nodeDatabase.telemetry.clear(); #endif + #if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB nodeDatabase.environment.clear(); nodeDatabase.environment.reserve(nodeEnvironment.size()); @@ -2293,6 +2592,7 @@ bool NodeDB::saveNodeDatabaseToDisk() #else nodeDatabase.environment.clear(); #endif + #if !MESHTASTIC_EXCLUDE_STATUSDB nodeDatabase.status.clear(); nodeDatabase.status.reserve(nodeStatus.size()); @@ -2319,6 +2619,11 @@ bool NodeDB::saveNodeDatabaseToDisk() nodeDatabase.environment.shrink_to_fit(); nodeDatabase.status.clear(); nodeDatabase.status.shrink_to_fit(); +#if WARM_NODE_COUNT > 0 + // Same cadence as the node DB; failure is logged but must not propagate — + // a false return from here would trigger saveToDisk()'s fsFormat() path. + warmStore.saveIfDirty(); +#endif return ok; } @@ -2354,6 +2659,7 @@ bool NodeDB::saveToDiskNoRetry(int saveWhat) FSCom.mkdir("/prefs"); spiLock->unlock(); #endif + if (saveWhat & SEGMENT_CONFIG) { config.has_device = true; config.has_display = true; @@ -2555,6 +2861,7 @@ void NodeDB::updatePosition(uint32_t nodeId, const meshtastic_Position &p, RxSou #else { concurrency::LockGuard guard(&satelliteMutex); + evictSatelliteOverCap(*this, nodePositions, nodeId); meshtastic_PositionLite &slot = nodePositions[nodeId]; // creates default-zero entry if missing if (src == RX_SRC_LOCAL) { @@ -2612,8 +2919,10 @@ void NodeDB::updateTelemetry(uint32_t nodeId, const meshtastic_Telemetry &t, RxS } #if !MESHTASTIC_EXCLUDE_TELEMETRYDB concurrency::LockGuard guard(&satelliteMutex); + evictSatelliteOverCap(*this, nodeTelemetry, nodeId); nodeTelemetry[nodeId] = t.variant.device_metrics; #endif + } else if (t.which_variant == meshtastic_Telemetry_environment_metrics_tag) { if (src == RX_SRC_LOCAL) { LOG_DEBUG("updateTelemetry LOCAL env"); @@ -2622,8 +2931,10 @@ void NodeDB::updateTelemetry(uint32_t nodeId, const meshtastic_Telemetry &t, RxS } #if !MESHTASTIC_EXCLUDE_ENVIRONMENTDB concurrency::LockGuard guard(&satelliteMutex); + evictSatelliteOverCap(*this, nodeEnvironment, nodeId); nodeEnvironment[nodeId] = t.variant.environment_metrics; #endif + } else { return; // air_quality / power / local_stats / health: not stored per-node } @@ -2652,13 +2963,13 @@ void NodeDB::addFromContact(meshtastic_SharedContact contact) info->num = contact.node_num; TypeConversions::CopyUserToNodeInfoLite(info, contact.user); if (contact.should_ignore) { - // If should_ignore is set, - // we need to clear the public key and other cruft, in addition to setting the node as ignored - nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_IS_IGNORED_MASK, true); + // Block the contact and drop its rich satellite data, but keep the + // public key copied above — an ignored peer keeps a usable identity + // (a verifiable target) rather than a bare node number. + if (!setProtectedFlag(info, NODEINFO_BITFIELD_IS_IGNORED_MASK, true)) + LOG_WARN(PROTECTED_CAP_WARN_FMT, "ignore", contact.node_num, MAX_NUM_NODES - 2); nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_IS_FAVORITE_MASK, false); eraseNodeSatellites(contact.node_num); - info->public_key.size = 0; - memset(info->public_key.bytes, 0, sizeof(info->public_key.bytes)); } else { /* Clients are sending add_contact before every text message DM (because clients may hold a larger node database with * public keys than the radio holds). However, we don't want to update last_heard just because we sent someone a DM! @@ -2677,12 +2988,16 @@ void NodeDB::addFromContact(meshtastic_SharedContact contact) } else { // Normal case: set is_favorite to prevent expiration. // last_heard will remain as-is (or remain 0 if this entry wasn't in the nodeDB). - nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); + // If the protected cap refuses the favorite, fall back to stamping last_heard so the + // contact still isn't the first eviction victim. + if (!setProtectedFlag(info, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true)) + info->last_heard = getTime(); } // As the clients will begin sending the contact with DMs, we want to strictly check if the node is manually verified if (contact.manually_verified) { - nodeInfoLiteSetBit(info, NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK, true); + if (!setProtectedFlag(info, NODEINFO_BITFIELD_IS_KEY_MANUALLY_VERIFIED_MASK, true)) + LOG_WARN(PROTECTED_CAP_WARN_FMT, "verify", contact.node_num, MAX_NUM_NODES - 2); } // Mark the node's key as manually verified to indicate trustworthiness. updateGUIforNode = info; @@ -2815,14 +3130,48 @@ void NodeDB::updateFrom(const meshtastic_MeshPacket &mp) } } -void NodeDB::set_favorite(bool is_favorite, uint32_t nodeId) +int NodeDB::numProtectedNodes() const +{ + int count = 0; + for (int i = 0; i < numMeshNodes; i++) + if (nodeInfoLiteIsProtected(&meshNodes->at(i))) + count++; + return count; +} + +bool NodeDB::setProtectedFlag(meshtastic_NodeInfoLite *node, uint32_t mask, bool on) +{ + if (!node) + return false; + if (!on) { + nodeInfoLiteSetBit(node, mask, false); + return true; + } + // Adding a flag to a node that is already protected doesn't grow the + // protected set, so it's always allowed. A newly-protected node is refused + // once the protected set has reached MAX_NUM_NODES-2, leaving two evictable + // slots so getOrCreateMeshNode can always make room. + if (nodeInfoLiteIsProtected(node) || numProtectedNodes() < MAX_NUM_NODES - 2) { + nodeInfoLiteSetBit(node, mask, true); + return true; + } + return false; +} + +bool NodeDB::set_favorite(bool is_favorite, uint32_t nodeId) { meshtastic_NodeInfoLite *lite = getMeshNode(nodeId); - if (lite && nodeInfoLiteIsFavorite(lite) != is_favorite) { - nodeInfoLiteSetBit(lite, NODEINFO_BITFIELD_IS_FAVORITE_MASK, is_favorite); + if (!lite) + return false; + if (nodeInfoLiteIsFavorite(lite) == is_favorite) + return true; // already in the requested state + if (setProtectedFlag(lite, NODEINFO_BITFIELD_IS_FAVORITE_MASK, is_favorite)) { sortMeshDB(); saveNodeDatabaseToDisk(); + return true; } + LOG_WARN(PROTECTED_CAP_WARN_FMT, "favorite", nodeId, MAX_NUM_NODES - 2); + return false; } bool NodeDB::isFavorite(uint32_t nodeId) @@ -2952,6 +3301,30 @@ bool NodeDB::isFull() return (numMeshNodes >= MAX_NUM_NODES) || (memGet.getFreeHeap() < MINIMUM_SAFE_FREE_HEAP); } +uint32_t NodeDB::hotNodeLastHeard(NodeNum n) const +{ + for (int i = 0; i < numMeshNodes; i++) + if (meshNodes->at(i).num == n) + return meshNodes->at(i).last_heard; + return 0; +} + +bool NodeDB::copyPublicKey(NodeNum n, meshtastic_NodeInfoLite_public_key_t &out) +{ + const meshtastic_NodeInfoLite *info = getMeshNode(n); + if (info && info->public_key.size == 32) { + out = info->public_key; + return true; + } +#if WARM_NODE_COUNT > 0 + if (warmStore.copyKey(n, out.bytes)) { + out.size = 32; + return true; + } +#endif + return false; +} + /// Find a node in our DB, create an empty NodeInfo if missing meshtastic_NodeInfoLite *NodeDB::getOrCreateMeshNode(NodeNum n) { @@ -2988,7 +3361,14 @@ meshtastic_NodeInfoLite *NodeDB::getOrCreateMeshNode(NodeNum n) } if (oldestIndex != -1) { - eraseNodeSatellites(meshNodes->at(oldestIndex).num); + const meshtastic_NodeInfoLite &evicted = meshNodes->at(oldestIndex); +#if WARM_NODE_COUNT > 0 + // Demote to the warm tier so the identity (and crucially the + // PKI key) outlives the hot-store slot. + warmStore.absorb(evicted.num, evicted.last_heard, + evicted.public_key.size == 32 ? evicted.public_key.bytes : NULL); +#endif + eraseNodeSatellites(evicted.num); // Shove the remaining nodes down the chain for (int i = oldestIndex; i < numMeshNodes - 1; i++) { meshNodes->at(i) = meshNodes->at(i + 1); @@ -2996,12 +3376,33 @@ meshtastic_NodeInfoLite *NodeDB::getOrCreateMeshNode(NodeNum n) (numMeshNodes)--; } } + // Don't append past the end of the vector. The protected-node cap + // (numProtectedNodes() <= MAX_NUM_NODES-2) means the eviction above frees + // a slot in normal operation; this guards the legacy case of a pre-cap + // database that is full of protected nodes — refuse rather than overrun. + if (numMeshNodes >= MAX_NUM_NODES) + return NULL; + // Pre-size before append when run before nodeDBSelfCare() (boot keygen); else at() aborts on nRF52. + if (static_cast(numMeshNodes) >= meshNodes->size()) + meshNodes->resize(numMeshNodes + 1); // add the node at the end lite = &meshNodes->at((numMeshNodes)++); // everything is missing except the nodenum memset(lite, 0, sizeof(*lite)); lite->num = n; +#if WARM_NODE_COUNT > 0 + // Re-admission: restore what the warm tier kept for this node + WarmNodeEntry warm; + if (warmStore.take(n, warm)) { + lite->last_heard = warm.last_heard; + if (!memfll(warm.public_key, 0, sizeof(warm.public_key))) { + lite->public_key.size = 32; + memcpy(lite->public_key.bytes, warm.public_key, 32); + } + LOG_MIGRATION("Rehydrated node 0x%x from warm tier (key=%d)", n, lite->public_key.size == 32); + } +#endif LOG_INFO("Adding node to database with %i nodes and %u bytes free!", numMeshNodes, memGet.getFreeHeap()); } diff --git a/src/mesh/NodeDB.h b/src/mesh/NodeDB.h index 3fe244294..f2df5ef1d 100644 --- a/src/mesh/NodeDB.h +++ b/src/mesh/NodeDB.h @@ -11,6 +11,7 @@ #include "MeshTypes.h" #include "NodeStatus.h" +#include "WarmNodeStore.h" #include "concurrency/Lock.h" #include "configuration.h" #include "mesh-pb-constants.h" @@ -226,9 +227,25 @@ class NodeDB bool updateUser(uint32_t nodeId, meshtastic_User &p, uint8_t channelIndex = 0); /* - * Sets a node either favorite or unfavorite + * Sets a node either favorite or unfavorite. Returns true if the node ends + * up in the requested state; false if the node is unknown or favouriting + * was refused by the protected-node cap (MAX_NUM_NODES - 2). */ - void set_favorite(bool is_favorite, uint32_t nodeId); + bool set_favorite(bool is_favorite, uint32_t nodeId); + + /// Count of eviction-protected (favourite/ignored/manually-verified) nodes. + int numProtectedNodes() const; + + /// printf-style warning emitted when setProtectedFlag() refuses a node at + /// the cap. %s = verb (favorite/ignore), 0x%08x = node, %d = cap. Shared by + /// LOG_WARN here and AdminModule::sendWarning so the wording stays in sync. + static constexpr const char *PROTECTED_CAP_WARN_FMT = "Can't %s 0x%08x: protected-node limit (%d) reached"; + + /// Turn an eviction-protection flag (favourite/ignored/verified) on/off. Off + /// always succeeds; on returns false (no change) once the protected set hits + /// the cap (MAX_NUM_NODES-2), keeping >=2 always-evictable slots. Callers + /// surface the refusal to the user. + bool setProtectedFlag(meshtastic_NodeInfoLite *node, uint32_t mask, bool on); /* * Returns true if the node is in the NodeDB and marked as favorite @@ -295,6 +312,24 @@ class NodeDB virtual meshtastic_NodeInfoLite *getMeshNode(NodeNum n); size_t getNumMeshNodes() { return numMeshNodes; } + /// Find a node in our DB, create an empty NodeInfoLite if missing (evicting + /// the oldest non-protected node when full). Public so admin handlers can + /// register a node we have not heard from yet (e.g. to block it by ID). + meshtastic_NodeInfoLite *getOrCreateMeshNode(NodeNum n); + +#if WARM_NODE_COUNT > 0 + // Warm ("long-tail") tier: minimal {num, last_heard, public_key} records + // for nodes evicted from the hot store. See WarmNodeStore.h. + WarmNodeStore warmStore; +#endif + + /// Copy the 32-byte public key for node n — hot store first, then the warm + /// tier. Returns false if we don't know a key for n. + bool copyPublicKey(NodeNum n, meshtastic_NodeInfoLite_public_key_t &out); + + /// last_heard of a hot-store node, or 0 if absent. Plain scan of meshNodes + /// with no allocation side effects (unlike getOrCreateMeshNode). + uint32_t hotNodeLastHeard(NodeNum n) const; // Thread-safe satellite-map accessors. Return false if absent or the // corresponding DB is compiled out. @@ -341,11 +376,15 @@ class NodeDB emptyNodeDatabase.version = DEVICESTATE_CUR_VER; size_t nodeDatabaseSize; pb_get_encoded_size(&nodeDatabaseSize, meshtastic_NodeDatabase_fields, &emptyNodeDatabase); - // Always include satellite slots so backups from higher-cap peers - // decode without truncation, even when our build excludes the DBs. - return nodeDatabaseSize + (MAX_NUM_NODES * meshtastic_NodeInfoLite_size) + - (MAX_NUM_NODES * meshtastic_NodePositionEntry_size) + (MAX_NUM_NODES * meshtastic_NodeTelemetryEntry_size) + - (MAX_NUM_NODES * meshtastic_NodeEnvironmentEntry_size) + (MAX_NUM_NODES * meshtastic_NodeStatusEntry_size); + // Decode-stream size ceiling only — no buffer this big is allocated (load + // streams from the file). Sized for the largest file any prior firmware + // could write (250-node ESP32-S3, satellites uncapped) so capacity + // downgrades / peer backups still decode; excess is trimmed after load. + // (not constexpr: portduino resolves MAX_NUM_NODES from runtime config) + const size_t loadCeiling = ((size_t)MAX_NUM_NODES > 250) ? (size_t)MAX_NUM_NODES : 250; + return nodeDatabaseSize + (loadCeiling * meshtastic_NodeInfoLite_size) + + (loadCeiling * meshtastic_NodePositionEntry_size) + (loadCeiling * meshtastic_NodeTelemetryEntry_size) + + (loadCeiling * meshtastic_NodeEnvironmentEntry_size) + (loadCeiling * meshtastic_NodeStatusEntry_size); } // returns true if the maximum number of nodes is reached or we are running low on memory @@ -430,11 +469,10 @@ class NodeDB mutable concurrency::Lock satelliteMutex; bool duplicateWarned = false; bool localPositionUpdatedSinceBoot = false; + bool migrationSavePending = false; uint32_t lastNodeDbSave = 0; // when we last saved our db to flash uint32_t lastBackupAttempt = 0; // when we last tried a backup automatically or manually uint32_t lastSort = 0; // When last sorted the nodeDB - /// Find a node in our DB, create an empty NodeInfoLite if missing - meshtastic_NodeInfoLite *getOrCreateMeshNode(NodeNum n); /* * Internal boolean to track sorting paused @@ -447,9 +485,33 @@ class NodeDB /// read our db from flash void loadFromDisk(); +#ifdef PIO_UNIT_TESTING + // Grant the unit-test shim access to the private maintenance paths below + // (migration / cleanup / eviction) without relaxing production access. + friend class NodeDBTestShim; +#endif + /// purge db entries without user info void cleanupMeshDB(); + /// Trim each satellite map down to MAX_SATELLITE_NODES, dropping the + /// stalest entries (used after loading files written before the cap, or by + /// a build with a larger cap). Returns true iff anything was trimmed. + bool enforceSatelliteCaps(); + + /// Node-DB self-care; call only once identity is established (getNodeNum() + /// valid). Confirms self is present, trims/demotes only NON-self overflow, and + /// rewrites the store once when something changed (never while storage locked). + void nodeDBSelfCare(); + +#if WARM_NODE_COUNT > 0 + /// A database from a larger-cap build (e.g. the pre-fork 150-node nRF52 store) + /// can exceed MAX_NUM_NODES on load. Rank the hot store, demote the oldest + /// overflow into the warm tier preserving {num, last_heard, public_key} so PKI + /// DMs survive instead of dropping on truncation. + void demoteOldestHotNodesToWarm(); +#endif + /// Reinit device state from scratch (not loading from disk) void installDefaultDeviceState(), installDefaultNodeDatabase(), installDefaultChannels(), installDefaultConfig(bool preserveKey), installDefaultModuleConfig(); @@ -570,6 +632,12 @@ inline bool nodeInfoLiteHasXeddsaSigned(const meshtastic_NodeInfoLite *n) { return n && (n->bitfield & NODEINFO_BITFIELD_HAS_XEDDSA_SIGNED_MASK); } +/// A node that the eviction/migration paths must not drop: a favourite, an +/// ignored (blocked) node, or a manually-verified key. +inline bool nodeInfoLiteIsProtected(const meshtastic_NodeInfoLite *n) +{ + return nodeInfoLiteIsFavorite(n) || nodeInfoLiteIsIgnored(n) || nodeInfoLiteIsKeyManuallyVerified(n); +} inline void nodeInfoLiteSetBit(meshtastic_NodeInfoLite *n, uint32_t mask, bool value) { diff --git a/src/mesh/NodeDBLegacyMigration.cpp b/src/mesh/NodeDBLegacyMigration.cpp index 18d6daa4d..107a4e32d 100644 --- a/src/mesh/NodeDBLegacyMigration.cpp +++ b/src/mesh/NodeDBLegacyMigration.cpp @@ -14,6 +14,7 @@ #include "configuration.h" #include "mesh-pb-constants.h" #include "mesh/generated/meshtastic/deviceonly_legacy.pb.h" +#include "meshUtils.h" #include #include @@ -88,8 +89,10 @@ bool NodeDB::migrateLegacyNodeDatabase() slim.bitfield |= NODEINFO_BITFIELD_HAS_USER_MASK; strncpy(slim.long_name, legacy.user.long_name, sizeof(slim.long_name)); slim.long_name[sizeof(slim.long_name) - 1] = '\0'; + sanitizeUtf8(slim.long_name, sizeof(slim.long_name)); // replace bad bytes so nanopb encode never fails strncpy(slim.short_name, legacy.user.short_name, sizeof(slim.short_name)); slim.short_name[sizeof(slim.short_name) - 1] = '\0'; + sanitizeUtf8(slim.short_name, sizeof(slim.short_name)); // same — v24 names may contain non-UTF-8 bytes slim.hw_model = legacy.user.hw_model; slim.role = legacy.user.role; if (legacy.user.is_licensed) diff --git a/src/mesh/Router.cpp b/src/mesh/Router.cpp index f176f60e6..9b3aa4676 100644 --- a/src/mesh/Router.cpp +++ b/src/mesh/Router.cpp @@ -485,14 +485,16 @@ DecodeState perhapsDecode(meshtastic_MeshPacket *p) bool decrypted = false; ChannelIndex chIndex = 0; #if !(MESHTASTIC_EXCLUDE_PKI) - // Attempt PKI decryption first - if (p->channel == 0 && isToUs(p) && p->to > 0 && !isBroadcast(p->to) && nodeDB->getMeshNode(p->from) != nullptr && - nodeDB->getMeshNode(p->from)->public_key.size > 0 && nodeDB->getMeshNode(p->to) != nullptr && - nodeDB->getMeshNode(p->to)->public_key.size > 0 && rawSize > MESHTASTIC_PKC_OVERHEAD) { + // Attempt PKI decryption first. The sender's key may come from the hot + // store or the warm tier (nodes evicted from the hot store keep their key + // there), so DMs from long-tail nodes still decrypt. + meshtastic_NodeInfoLite_public_key_t fromKey = {0, {0}}; + if (p->channel == 0 && isToUs(p) && p->to > 0 && !isBroadcast(p->to) && nodeDB->copyPublicKey(p->from, fromKey) && + nodeDB->getMeshNode(p->to) != nullptr && nodeDB->getMeshNode(p->to)->public_key.size > 0 && + rawSize > MESHTASTIC_PKC_OVERHEAD) { LOG_DEBUG("Attempt PKI decryption"); - if (crypto->decryptCurve25519(p->from, nodeDB->getMeshNode(p->from)->public_key, p->id, rawSize, p->encrypted.bytes, - bytes)) { + if (crypto->decryptCurve25519(p->from, fromKey, p->id, rawSize, p->encrypted.bytes, bytes)) { LOG_INFO("PKI Decryption worked!"); meshtastic_Data decodedtmp; @@ -503,7 +505,7 @@ DecodeState perhapsDecode(meshtastic_MeshPacket *p) decrypted = true; LOG_INFO("Packet decrypted using PKI!"); p->pki_encrypted = true; - memcpy(&p->public_key.bytes, nodeDB->getMeshNode(p->from)->public_key.bytes, 32); + memcpy(p->public_key.bytes, fromKey.bytes, 32); p->public_key.size = 32; p->decoded = decodedtmp; p->which_payload_variant = meshtastic_MeshPacket_decoded_tag; // change type to decoded @@ -720,7 +722,10 @@ meshtastic_Routing_Error perhapsEncode(meshtastic_MeshPacket *p) ChannelIndex chIndex = p->channel; // keep as a local because we are about to change it #if !(MESHTASTIC_EXCLUDE_PKI) - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(p->to); + // Destination key from the hot store or the warm tier (evicted + // long-tail nodes keep their key there) + meshtastic_NodeInfoLite_public_key_t destKey = {0, {0}}; + bool haveDestKey = nodeDB->copyPublicKey(p->to, destKey); // We may want to retool things so we can send a PKC packet when the client specifies a key and nodenum, even if the node // is not in the local nodedb // First, only PKC encrypt packets we are originating @@ -743,18 +748,17 @@ meshtastic_Routing_Error perhapsEncode(meshtastic_MeshPacket *p) if (numbytes + MESHTASTIC_HEADER_LENGTH + MESHTASTIC_PKC_OVERHEAD > MAX_LORA_PAYLOAD_LEN) return meshtastic_Routing_Error_TOO_LARGE; // Check for a known public key for the destination - if (node == nullptr || node->public_key.size != 32) { + if (!haveDestKey) { LOG_WARN("Unknown public key for destination node 0x%08x (portnum %d), refusing to send legacy DM", p->to, p->decoded.portnum); return meshtastic_Routing_Error_PKI_SEND_FAIL_PUBLIC_KEY; } - if (p->pki_encrypted && !memfll(p->public_key.bytes, 0, 32) && - memcmp(p->public_key.bytes, node->public_key.bytes, 32) != 0) { + if (p->pki_encrypted && !memfll(p->public_key.bytes, 0, 32) && memcmp(p->public_key.bytes, destKey.bytes, 32) != 0) { LOG_WARN("Client public key differs from requested: 0x%02x, stored key begins 0x%02x", *p->public_key.bytes, - *node->public_key.bytes); + *destKey.bytes); return meshtastic_Routing_Error_PKI_FAILED; } - crypto->encryptCurve25519(p->to, getFrom(p), node->public_key, p->id, numbytes, bytes, p->encrypted.bytes); + crypto->encryptCurve25519(p->to, getFrom(p), destKey, p->id, numbytes, bytes, p->encrypted.bytes); numbytes += MESHTASTIC_PKC_OVERHEAD; p->channel = 0; p->pki_encrypted = true; diff --git a/src/mesh/WarmNodeStore.cpp b/src/mesh/WarmNodeStore.cpp new file mode 100644 index 000000000..8617f505c --- /dev/null +++ b/src/mesh/WarmNodeStore.cpp @@ -0,0 +1,548 @@ +#include "WarmNodeStore.h" + +#if WARM_NODE_COUNT > 0 + +#include "FSCommon.h" +#include "SPILock.h" +#include "SafeFile.h" +#include "configuration.h" +#include "power/PowerHAL.h" +#include +#include + +#if defined(NRF52840_XXAA) +#include "flash/flash_nrf5x.h" +#define WARM_RING_MAGIC 0x474E5257u // "WRNG" +// A tombstone is an entry record whose last_heard is all-ones — getTime() +// (unix seconds) cannot reach 0xFFFFFFFF until 2106, and erased flash is +// detected via num == 0xFFFFFFFF before last_heard is ever inspected. +#define WARM_RING_TOMBSTONE 0xFFFFFFFFu +#else +// warm.dat layout: this header followed by count packed WarmNodeEntry records. +struct WarmStoreHeader { + uint32_t magic; // WARM_STORE_MAGIC + uint32_t reserved; // 0; kept so the header stays 16 B + uint16_t count; // entries persisted + uint16_t entrySize; // sizeof(WarmNodeEntry), format guard + uint32_t crc; // crc32 over count * entrySize bytes +}; +static_assert(sizeof(WarmStoreHeader) == 16, "header layout is part of the persistence format"); + +#define WARM_STORE_MAGIC 0x314D5257u // "WRM1" + +#ifdef FSCom +static const char *warmFileName = "/prefs/warm.dat"; +#endif +#endif // NRF52840_XXAA + +static inline bool keyIsSet(const uint8_t key[32]) +{ + for (int i = 0; i < 32; i++) + if (key[i]) + return true; + return false; +} + +WarmNodeStore::WarmNodeStore() +{ +#if defined(ARCH_ESP32) && defined(BOARD_HAS_PSRAM) + entries = static_cast(ps_calloc(WARM_NODE_COUNT, sizeof(WarmNodeEntry))); + if (!entries) { + LOG_WARN("WarmStore: PSRAM alloc failed, using heap"); + entries = static_cast(calloc(WARM_NODE_COUNT, sizeof(WarmNodeEntry))); + } +#else + entries = static_cast(calloc(WARM_NODE_COUNT, sizeof(WarmNodeEntry))); +#endif +#if defined(NRF52840_XXAA) + memset(pageOf, kNoPage, sizeof(pageOf)); +#endif +} + +WarmNodeStore::~WarmNodeStore() +{ + free(entries); // always malloc-family (calloc / ps_calloc) + entries = nullptr; +} + +WarmNodeEntry *WarmNodeStore::find(NodeNum num) const +{ + if (!entries || !num) + return nullptr; + for (size_t i = 0; i < WARM_NODE_COUNT; i++) + if (entries[i].num == num) + return &entries[i]; + return nullptr; +} + +// Slot placement with the keyed-first admission policy. Shared by absorb() +// and the ring replay, so the policy is applied identically in both paths. +WarmNodeEntry *WarmNodeStore::place(NodeNum num, uint32_t lastHeard, const uint8_t *key32) +{ + if (!entries || !num) + return nullptr; + + const bool candidateKeyed = key32 && keyIsSet(key32); + + WarmNodeEntry *slot = find(num); + const bool sameNode = slot != nullptr; + if (!slot) { + // Pick a victim: any empty slot, else the oldest keyless entry, else + // (only for keyed candidates) the oldest keyed entry. + WarmNodeEntry *oldestKeyless = nullptr, *oldestKeyed = nullptr; + for (size_t i = 0; i < WARM_NODE_COUNT; i++) { + WarmNodeEntry &e = entries[i]; + if (!e.num) { + slot = &e; + break; + } + if (keyIsSet(e.public_key)) { + if (!oldestKeyed || e.last_heard < oldestKeyed->last_heard) + oldestKeyed = &e; + } else { + if (!oldestKeyless || e.last_heard < oldestKeyless->last_heard) + oldestKeyless = &e; + } + } + if (!slot) + slot = oldestKeyless ? oldestKeyless : (candidateKeyed ? oldestKeyed : nullptr); + if (!slot) + return nullptr; // store full of keyed entries and the candidate has no key + } + + slot->num = num; + slot->last_heard = lastHeard; + if (candidateKeyed) + memcpy(slot->public_key, key32, 32); + else if (!sameNode) + // Repurposing a victim slot for a different node: clear its stale key. + // A keyless refresh of a node already here keeps the key we learned. + memset(slot->public_key, 0, 32); + return slot; +} + +bool WarmNodeStore::absorb(NodeNum num, uint32_t lastHeard, const uint8_t *key32) +{ + const WarmNodeEntry *slot = place(num, lastHeard, key32); + if (!slot) + return false; + persistEntry(*slot); + LOG_MIGRATION("WarmStore absorb 0x%08x key=%d last_heard=%u (now %u/%u)", (unsigned)num, keyIsSet(slot->public_key) ? 1 : 0, + (unsigned)lastHeard, (unsigned)count(), (unsigned)capacity()); + return true; +} + +bool WarmNodeStore::take(NodeNum num, WarmNodeEntry &out) +{ + WarmNodeEntry *e = find(num); + if (!e) + return false; + out = *e; + const int idx = static_cast(e - entries); + memset(e, 0, sizeof(*e)); + persistRemove(num, idx); + LOG_MIGRATION("WarmStore take(rehydrate) 0x%08x key=%d (now %u/%u)", (unsigned)num, keyIsSet(out.public_key) ? 1 : 0, + (unsigned)count(), (unsigned)capacity()); + return true; +} + +#if MESHTASTIC_NODEDB_MIGRATION_VERBOSE +void WarmNodeStore::dumpToLog(const char *reason) const +{ + if (!entries) { + LOG_MIGRATION("WarmStore dump (%s): backend not allocated", reason); + return; + } + LOG_MIGRATION("WarmStore dump (%s): %u live / %u cap ==>", reason, (unsigned)count(), (unsigned)capacity()); + unsigned shown = 0; + for (size_t i = 0; i < WARM_NODE_COUNT; i++) { + const WarmNodeEntry &e = entries[i]; + if (e.num == 0) + continue; + LOG_MIGRATION(" warm[%3u] 0x%08x last_heard=%u key=%d", (unsigned)i, (unsigned)e.num, (unsigned)e.last_heard, + keyIsSet(e.public_key) ? 1 : 0); + shown++; + } + LOG_MIGRATION("WarmStore dump (%s): <== end (%u entries)", reason, shown); +} +#endif // MESHTASTIC_NODEDB_MIGRATION_VERBOSE + +bool WarmNodeStore::copyKey(NodeNum num, uint8_t out[32]) const +{ + const WarmNodeEntry *e = find(num); + if (!e || !keyIsSet(e->public_key)) + return false; + memcpy(out, e->public_key, 32); + return true; +} + +bool WarmNodeStore::contains(NodeNum num) const +{ + return find(num) != nullptr; +} + +void WarmNodeStore::remove(NodeNum num) +{ + WarmNodeEntry *e = find(num); + if (e) { + const int idx = static_cast(e - entries); + memset(e, 0, sizeof(*e)); + persistRemove(num, idx); + } +} + +void WarmNodeStore::clear() +{ + if (!entries) + return; + memset(entries, 0, WARM_NODE_COUNT * sizeof(WarmNodeEntry)); +#if defined(NRF52840_XXAA) + memset(pageOf, kNoPage, sizeof(pageOf)); +#endif + persistClear(); +} + +size_t WarmNodeStore::count() const +{ + size_t n = 0; + if (entries) + for (size_t i = 0; i < WARM_NODE_COUNT; i++) + if (entries[i].num) + n++; + return n; +} + +bool WarmNodeStore::saveIfDirty() +{ + if (!dirty) + return true; + bool ok = save(); + if (ok) + dirty = false; + return ok; +} + +#if defined(NRF52840_XXAA) + +// Raw-flash record-ring backend (nRF52840). +// 3 × 4 KB pages below LittleFS. Mutations append 40 B records (entry snapshot, +// or tombstone with last_heard == 0xFFFFFFFF) via the shared flash_nrf5x page +// cache; saveIfDirty() is the durability point. A full page reclaims the oldest +// (stranded live entries re-appended, then erased). Flash access holds spiLock — +// the page cache is shared with InternalFS/LittleFS. + +bool WarmNodeStore::ringReadHeader(uint8_t page, WarmPageHeader &h) const +{ + flash_nrf5x_read(&h, WARM_FLASH_PAGE_ADDR(page), sizeof(h)); + return h.magic == WARM_RING_MAGIC && h.seq != 0xFFFFFFFFu; +} + +// Caller holds spiLock. +void WarmNodeStore::ringOpenPage(uint8_t page) +{ + // Drop any cached state for the page before the real erase, so a later + // cache flush can't resurrect stale bytes. + flash_nrf5x_flush(); + flash_nrf5x_erase(WARM_FLASH_PAGE_ADDR(page)); + WarmPageHeader h; + h.magic = WARM_RING_MAGIC; + h.seq = nextSeq++; + flash_nrf5x_write(WARM_FLASH_PAGE_ADDR(page), &h, sizeof(h)); + activePage = page; + writeSlot = 0; +} + +// Caller holds spiLock. May recurse once via ringAppend if the stranded set +// fills the fresh page exactly — bounded by WARM_NODE_COUNT <= 2*kRecordsPerPage. +void WarmNodeStore::ringRotate() +{ + uint8_t target = 0; + if (activePage != kNoPage) { + // Lowest-seq valid page, preferring erased pages; never the active one + uint32_t bestSeq = 0; + bool found = false; + for (uint8_t p = 0; p < WARM_FLASH_PAGES; p++) { + if (p == activePage) + continue; + WarmPageHeader h; + if (!ringReadHeader(p, h)) { + target = p; // erased/invalid page: free real estate, take it + found = true; + break; + } + if (!found || static_cast(h.seq - bestSeq) < 0) { + target = p; + bestSeq = h.seq; + found = true; + } + } + } + + // Capture live entries stranded in the page we're about to erase + int stranded[WARM_NODE_COUNT] = {}; + int nStranded = 0; + for (size_t i = 0; i < WARM_NODE_COUNT; i++) { + if (entries[i].num && pageOf[i] == target) + stranded[nStranded++] = static_cast(i); + if (pageOf[i] == target) + pageOf[i] = kNoPage; + } + + ringOpenPage(target); + + for (int k = 0; k < nStranded; k++) + ringAppend(entries[stranded[k]], stranded[k]); +} + +// Caller holds spiLock. +void WarmNodeStore::ringAppend(const WarmNodeEntry &rec, int storeSlot) +{ + if (activePage == kNoPage || writeSlot >= kRecordsPerPage) + ringRotate(); + const uint32_t addr = + WARM_FLASH_PAGE_ADDR(activePage) + sizeof(WarmPageHeader) + static_cast(writeSlot) * sizeof(WarmNodeEntry); + flash_nrf5x_write(addr, &rec, sizeof(rec)); + writeSlot++; + if (storeSlot >= 0) + pageOf[storeSlot] = activePage; + dirty = true; +} + +void WarmNodeStore::persistEntry(const WarmNodeEntry &e) +{ + concurrency::LockGuard g(spiLock); + ringAppend(e, static_cast(&e - entries)); +} + +void WarmNodeStore::persistRemove(NodeNum num, int storeSlot) +{ + if (storeSlot >= 0 && storeSlot < static_cast(WARM_NODE_COUNT)) + pageOf[storeSlot] = 0xFF; + WarmNodeEntry tomb; + memset(&tomb, 0, sizeof(tomb)); + tomb.num = num; + tomb.last_heard = WARM_RING_TOMBSTONE; + concurrency::LockGuard g(spiLock); + ringAppend(tomb, -1); +} + +void WarmNodeStore::persistClear() +{ + concurrency::LockGuard g(spiLock); + flash_nrf5x_flush(); + for (uint8_t p = 0; p < WARM_FLASH_PAGES; p++) + flash_nrf5x_erase(WARM_FLASH_PAGE_ADDR(p)); + activePage = 0xFF; + writeSlot = 0; + nextSeq = 1; + dirty = false; // the erased ring already reflects the empty store +} + +void WarmNodeStore::load() +{ + if (!entries) + return; + concurrency::LockGuard g(spiLock); + + // Order valid pages by ascending seq so replay applies oldest first + uint8_t order[WARM_FLASH_PAGES] = {}; + uint32_t seqs[WARM_FLASH_PAGES] = {}; + uint8_t nValid = 0; + uint8_t nCorrupt = 0; + for (uint8_t p = 0; p < WARM_FLASH_PAGES; p++) { + WarmPageHeader h; + if (!ringReadHeader(p, h)) { + // An erased page reads back all-ones; any other magic is a + // partially-written or bit-rotted header we're dropping, so flag it + // rather than silently treating the loss as a clean empty ring. + if (h.magic != 0xFFFFFFFFu) + nCorrupt++; + continue; + } + uint8_t pos = nValid; + while (pos > 0 && static_cast(h.seq - seqs[pos - 1]) < 0) { + order[pos] = order[pos - 1]; + seqs[pos] = seqs[pos - 1]; + pos--; + } + order[pos] = p; + seqs[pos] = h.seq; + nValid++; + } + + if (nValid == 0) { + activePage = 0xFF; + writeSlot = 0; + nextSeq = 1; + if (nCorrupt) + LOG_WARN("WarmStore: ring unreadable (%u bad page(s)), empty", nCorrupt); + else + LOG_INFO("WarmStore: ring empty, starting fresh"); + return; + } + + uint32_t replayed = 0; + for (uint8_t k = 0; k < nValid; k++) { + const uint8_t p = order[k]; + uint16_t slot = 0; + for (; slot < kRecordsPerPage; slot++) { + WarmNodeEntry rec; + flash_nrf5x_read(&rec, WARM_FLASH_PAGE_ADDR(p) + sizeof(WarmPageHeader) + (uint32_t)slot * sizeof(rec), sizeof(rec)); + if (rec.num == 0xFFFFFFFFu) + break; // erased space: end of this page's records (append-only) + if (rec.num == 0) + continue; // unexpected; skip defensively + replayed++; + if (rec.last_heard == WARM_RING_TOMBSTONE) { + WarmNodeEntry *e = find(rec.num); + if (e) { + pageOf[e - entries] = 0xFF; + memset(e, 0, sizeof(*e)); + } + } else { + const WarmNodeEntry *e = place(rec.num, rec.last_heard, rec.public_key); + if (e) + pageOf[e - entries] = p; + } + } + if (k == nValid - 1) { // newest page becomes the active head + activePage = p; + writeSlot = slot; + nextSeq = seqs[k] + 1; + } + } + if (nCorrupt) + LOG_WARN("WarmStore: dropped %u corrupt ring page(s), some nodes lost", nCorrupt); + LOG_INFO("WarmStore: replayed %u ring records -> %u live nodes (page %u, slot %u)", (unsigned)replayed, (unsigned)count(), + activePage, writeSlot); +} + +bool WarmNodeStore::save() +{ + if (!powerHAL_isPowerLevelSafe()) { + LOG_ERROR("Error: trying to save WarmStore on unsafe device power level."); + return false; + } + concurrency::LockGuard g(spiLock); + flash_nrf5x_flush(); + return true; +} + +#else // !NRF52840_XXAA -------------------- + +void WarmNodeStore::persistEntry(const WarmNodeEntry &e) +{ + (void)e; + dirty = true; +} + +void WarmNodeStore::persistRemove(NodeNum num, int storeSlot) +{ + (void)num; + (void)storeSlot; + dirty = true; +} + +void WarmNodeStore::persistClear() +{ + dirty = true; +} + +#ifdef FSCom + +// ---- File persistence: /prefs/warm.dat snapshots ---------------------------- + +// Compact occupied slots to the front of `dst`; returns the count. +static uint16_t packEntries(const WarmNodeEntry *src, WarmNodeEntry *dst) +{ + uint16_t n = 0; + for (size_t i = 0; i < WARM_NODE_COUNT; i++) + if (src[i].num) + dst[n++] = src[i]; + return n; +} + +void WarmNodeStore::load() +{ + if (!entries) + return; + // Clear first — all failure paths below then correctly represent "empty", + // even if load() is called on an already-used instance. + memset(entries, 0, WARM_NODE_COUNT * sizeof(WarmNodeEntry)); + concurrency::LockGuard g(spiLock); + auto f = FSCom.open(warmFileName, FILE_O_READ); + if (!f) + return; + WarmStoreHeader h; + if ((size_t)f.read((uint8_t *)&h, sizeof(h)) != sizeof(h)) { + f.close(); + LOG_WARN("WarmStore: %s header read failed, starting empty", warmFileName); + return; + } + if (h.magic != WARM_STORE_MAGIC || h.entrySize != sizeof(WarmNodeEntry) || h.count > WARM_NODE_COUNT) { + f.close(); + LOG_WARN("WarmStore: %s header invalid (magic=0x%08x entrySize=%u count=%u), starting empty", warmFileName, h.magic, + h.entrySize, h.count); + return; + } + if (h.count) { + const size_t len = (size_t)h.count * sizeof(WarmNodeEntry); + const bool readOk = (size_t)f.read((uint8_t *)entries, len) == len; + f.close(); + if (!readOk) { + LOG_WARN("WarmStore: %s entries read failed, starting empty", warmFileName); + return; + } + if (crc32Buffer(entries, len) != h.crc) { + LOG_WARN("WarmStore: %s CRC mismatch, starting empty", warmFileName); + return; + } + } else { + f.close(); + } + LOG_INFO("WarmStore: loaded %u warm nodes from %s", h.count, warmFileName); +} + +bool WarmNodeStore::save() +{ + if (!entries) + return false; + if (!powerHAL_isPowerLevelSafe()) { + LOG_ERROR("Error: trying to save WarmStore on unsafe device power level."); + return false; + } + + std::vector packed(WARM_NODE_COUNT); + WarmStoreHeader h; + h.magic = WARM_STORE_MAGIC; + h.reserved = 0; + h.count = packEntries(entries, packed.data()); + h.entrySize = sizeof(WarmNodeEntry); + h.crc = crc32Buffer(packed.data(), h.count * sizeof(WarmNodeEntry)); + + concurrency::LockGuard g(spiLock); + FSCom.mkdir("/prefs"); + + auto f = SafeFile(warmFileName, false); + f.write((const uint8_t *)&h, sizeof(h)); + f.write((const uint8_t *)packed.data(), h.count * sizeof(WarmNodeEntry)); + bool ok = f.close(); + if (!ok) + LOG_ERROR("WarmStore: can't write %s", warmFileName); + else + LOG_DEBUG("WarmStore: saved %u warm nodes to %s", h.count, warmFileName); + return ok; +} + +#else + +void WarmNodeStore::load() {} +bool WarmNodeStore::save() +{ + return true; +} + +#endif // FSCom +#endif // NRF52840_XXAA + +#endif // WARM_NODE_COUNT > 0 diff --git a/src/mesh/WarmNodeStore.h b/src/mesh/WarmNodeStore.h new file mode 100644 index 000000000..c5320317b --- /dev/null +++ b/src/mesh/WarmNodeStore.h @@ -0,0 +1,130 @@ +#pragma once + +#include "MeshTypes.h" +#include "mesh-pb-constants.h" +#include +#include + +// Verbose tracing for the warm-store migration + NodeDB self-care. Per-event / +// per-boot chatter routes through this so it can be silenced in one place (set +// to 0) once they're proven; genuine LOG_WARN anomalies stay unconditional. +#ifndef MESHTASTIC_NODEDB_MIGRATION_VERBOSE +#define MESHTASTIC_NODEDB_MIGRATION_VERBOSE 0 +#endif +#if MESHTASTIC_NODEDB_MIGRATION_VERBOSE +#define LOG_MIGRATION(...) LOG_INFO(__VA_ARGS__) +#else +#define LOG_MIGRATION(...) ((void)0) +#endif + +#if WARM_NODE_COUNT > 0 + +/** + * Warm ("long-tail") node tier. + * + * Minimal identity record (NodeNum, last_heard, Curve25519 public key) for nodes + * evicted from the hot NodeInfoLite store, so DMs to/from them keep encrypting — + * the key is expensive to re-learn, the rest rebuilds from traffic in seconds. + * Flat fixed array, linear scan (only on hot-store misses), LRU by last_heard + * with keyed entries outranking keyless. + * + * Persistence: nRF52840 uses a 12 KB raw-flash record-ring below LittleFS + * (append + replay + compact-on-rotate — see the backend in WarmNodeStore.cpp, + * link-guarded by nrf52840_s140_v7.ld). Everywhere else: /prefs/warm.dat. + */ +struct WarmNodeEntry { + NodeNum num; // 0 = empty slot + uint32_t last_heard; // recency for LRU ordering + uint8_t public_key[32]; // all-zero = no key (a real key is never all-zero) +}; +static_assert(sizeof(WarmNodeEntry) == 40, "WarmNodeEntry must stay 40 B — persistence format depends on it"); + +// Gated on NRF52840_XXAA: the ring sits at 0xEA000 +// valid only on the 1 MB-flash nRF52840. +#if defined(NRF52840_XXAA) +#define WARM_FLASH_PAGE_SIZE 4096u +#define WARM_FLASH_PAGES 3u +#define WARM_FLASH_REGION_BASE (0xED000u - WARM_FLASH_PAGES * WARM_FLASH_PAGE_SIZE) // 0xEA000 +#define WARM_FLASH_PAGE_ADDR(i) (WARM_FLASH_REGION_BASE + (i)*WARM_FLASH_PAGE_SIZE) +#endif + +class WarmNodeStore +{ + public: + WarmNodeStore(); + ~WarmNodeStore(); + WarmNodeStore(const WarmNodeStore &) = delete; + WarmNodeStore &operator=(const WarmNodeStore &) = delete; + + /// Remember an evicted hot node. Keyless candidates never displace keyed + /// entries; otherwise the oldest (keyless-first) entry is replaced. + /// @return true if the node was stored or updated + bool absorb(NodeNum num, uint32_t lastHeard, const uint8_t *key32 /* may be NULL */); + + /// Find and remove an entry (used when the node is re-admitted to the hot store). + bool take(NodeNum num, WarmNodeEntry &out); + + /// Copy the 32-byte public key for a node, if we have one. + bool copyKey(NodeNum num, uint8_t out[32]) const; + + bool contains(NodeNum num) const; + void remove(NodeNum num); + void clear(); + size_t count() const; + size_t capacity() const { return entries ? WARM_NODE_COUNT : 0; } + +#if MESHTASTIC_NODEDB_MIGRATION_VERBOSE + /// Debug: dump every live warm entry (num / last_heard / has-key) to the + /// console. Compiled out unless MESHTASTIC_NODEDB_MIGRATION_VERBOSE. + void dumpToLog(const char *reason = "dump") const; +#endif + + /// Load persisted entries (called once at boot, after the node DB loads). + void load(); + /// Durability point, piggybacked on the node-database save cadence. On the + /// ring backend this flushes the shared flash page cache; on the file + /// backend it writes the warm.dat snapshot. + bool saveIfDirty(); + + private: + WarmNodeEntry *entries = nullptr; // WARM_NODE_COUNT slots; PSRAM on ESP32 when available + bool dirty = false; + + WarmNodeEntry *find(NodeNum num) const; + // Internal slot-placement shared by absorb() and ring replay: applies the + // keyed-first admission policy without touching persistence. + WarmNodeEntry *place(NodeNum num, uint32_t lastHeard, const uint8_t *key32); + + // Persistence hooks called from the mutation paths. File backend: mark + // dirty. Ring backend: append an upsert/tombstone record (+ mark dirty). + void persistEntry(const WarmNodeEntry &e); // e must point into entries[] + void persistRemove(NodeNum num, int storeSlot); + void persistClear(); + +#if defined(NRF52840_XXAA) + // nRF52840 raw-flash record-ring state. + struct WarmPageHeader { + uint32_t magic; // WARM_RING_MAGIC + uint32_t seq; // page generation; 0xFFFFFFFF = erased/unused + }; + static_assert(sizeof(WarmPageHeader) == 8, "page header is part of the flash format"); + static constexpr uint16_t kRecordsPerPage = (WARM_FLASH_PAGE_SIZE - sizeof(WarmPageHeader)) / sizeof(WarmNodeEntry); // 102 + static_assert(WARM_NODE_COUNT <= 2 * ((WARM_FLASH_PAGE_SIZE - 8) / 40), "live set must fit the ring with one page reclaimed"); + + static constexpr uint8_t kNoPage = 0xFF; // "no page" sentinel for activePage / pageOf[] + + uint8_t activePage = kNoPage; // no page opened yet (fresh/erased ring) + uint16_t writeSlot = 0; // next free record slot in the active page + uint32_t nextSeq = 1; // seq for the next page opened + uint8_t pageOf[WARM_NODE_COUNT]; // flash page holding each RAM slot's newest record; kNoPage = none + + void ringAppend(const WarmNodeEntry &rec, int storeSlot /* -1 for tombstones */); + void ringRotate(); // reclaim oldest page, compacting stranded live entries + void ringOpenPage(uint8_t page); // erase + write header (seq = nextSeq++) + bool ringReadHeader(uint8_t page, WarmPageHeader &h) const; +#endif + + bool save(); +}; + +#endif // WARM_NODE_COUNT > 0 diff --git a/src/mesh/mesh-pb-constants.h b/src/mesh/mesh-pb-constants.h index b4d999011..c7edb98a3 100644 --- a/src/mesh/mesh-pb-constants.h +++ b/src/mesh/mesh-pb-constants.h @@ -48,39 +48,40 @@ static_assert(sizeof(meshtastic_NodeInfoLite) <= 130, "NodeInfoLite size increas #define MESHTASTIC_EXCLUDE_POSITIONDB 1 #else #define MESHTASTIC_EXCLUDE_POSITIONDB 0 -#endif -#endif +#endif // STM32WL +#endif // MESHTASTIC_EXCLUDE_POSITIONDB #ifndef MESHTASTIC_EXCLUDE_TELEMETRYDB #if defined(ARCH_STM32WL) #define MESHTASTIC_EXCLUDE_TELEMETRYDB 1 #else #define MESHTASTIC_EXCLUDE_TELEMETRYDB 0 -#endif -#endif +#endif // STM32WL +#endif // MESHTASTIC_EXCLUDE_TELEMETRYDB #ifndef MESHTASTIC_EXCLUDE_ENVIRONMENTDB #if defined(ARCH_STM32WL) #define MESHTASTIC_EXCLUDE_ENVIRONMENTDB 1 #else #define MESHTASTIC_EXCLUDE_ENVIRONMENTDB 0 -#endif -#endif +#endif // STM32WL +#endif // MESHTASTIC_EXCLUDE_ENVIRONMENTDB #ifndef MESHTASTIC_EXCLUDE_STATUSDB #if defined(ARCH_STM32WL) || defined(MESHTASTIC_EXCLUDE_STATUS) #define MESHTASTIC_EXCLUDE_STATUSDB 1 #else #define MESHTASTIC_EXCLUDE_STATUSDB 0 -#endif -#endif +#endif // STM32WL +#endif // MESHTASTIC_EXCLUDE_STATUSDB -/// max number of nodes allowed in the nodeDB +/// Max nodes in the hot store (full NodeInfoLite). Evicted nodes' identities +/// live in the warm tier (WARM_NODE_COUNT). nRF52840 caps at 120 to keep +/// nodes.proto inside the stock 28 KB LittleFS; flash-rich platforms (ESP32-S3, +/// portduino) keep their larger hot store and lean on warm only for the tail. #ifndef MAX_NUM_NODES #if defined(ARCH_STM32WL) #define MAX_NUM_NODES 10 -#elif defined(ARCH_NRF52) -#define MAX_NUM_NODES 150 #elif defined(CONFIG_IDF_TARGET_ESP32S3) #include "Esp.h" static inline int get_max_num_nodes() @@ -95,10 +96,43 @@ static inline int get_max_num_nodes() } } #define MAX_NUM_NODES get_max_num_nodes() +#elif defined(ARCH_PORTDUINO) +#define MAX_NUM_NODES 250 // native host: no flash/RAM constraint; match the ESP32-S3 top tier #else -#define MAX_NUM_NODES 100 -#endif -#endif +#define MAX_NUM_NODES 120 // nRF52840 (28 KB LittleFS) and generic ESP32 +#endif // platform +#endif // MAX_NUM_NODES + +/// Per-map cap (position/telemetry/environment/status): only the freshest +/// MAX_SATELLITE_NODES nodes keep satellite payloads, the rest just the +/// NodeInfoLite header. RAM-bound (the maps are internal-SRAM, not PSRAM), so +/// flash-rich hosts get a cap >= their hot store (satellites for every node, as +/// before the cap existed) while constrained parts stay at 40. +#ifndef MAX_SATELLITE_NODES +#if defined(CONFIG_IDF_TARGET_ESP32S3) || defined(ARCH_PORTDUINO) +#define MAX_SATELLITE_NODES 250 +#else +#define MAX_SATELLITE_NODES 40 // nRF52840 (28 KB LittleFS) and generic ESP32 +#endif // platform +#endif // MAX_SATELLITE_NODES + +/// Warm tier: 40 B {num, last_heard, public_key} records kept for evicted nodes +/// so DMs to/from them keep decrypting. 0 disables it; size is per-platform +/// below, persisted to /prefs/warm.dat (or the nRF52840 raw-flash ring). +#ifndef WARM_NODE_COUNT +#if defined(ARCH_STM32WL) +#define WARM_NODE_COUNT 0 +#elif defined(NRF52840_XXAA) +// Keyed on the NRF52840_XXAA build flag, not ARCH_NRF52: the latter (from +// architecture.h via configuration.h) isn't defined this early in every include +// chain. Backed by the raw-flash ring below LittleFS — see WarmNodeStore.h. +#define WARM_NODE_COUNT 200 +#elif defined(CONFIG_IDF_TARGET_ESP32S3) +#define WARM_NODE_COUNT 2000 // PSRAM-backed when available; warm.dat ~80 KB +#else +#define WARM_NODE_COUNT 320 +#endif // platform +#endif // WARM_NODE_COUNT /// Max number of channels allowed #define MAX_NUM_CHANNELS (member_size(meshtastic_ChannelFile, channels) / member_size(meshtastic_ChannelFile, channels[0])) @@ -125,8 +159,8 @@ static inline int get_max_num_nodes() #define TRAFFIC_MANAGEMENT_CACHE_SIZE 1000 #else #define TRAFFIC_MANAGEMENT_CACHE_SIZE 0 -#endif -#endif +#endif // HAS_TRAFFIC_MANAGEMENT +#endif // TRAFFIC_MANAGEMENT_CACHE_SIZE /// helper function for encoding a record as a protobuf, any failures to encode are fatal and we will panic /// returns the encoded packet size diff --git a/src/modules/AdminModule.cpp b/src/modules/AdminModule.cpp index 8715e202d..af4e3f969 100644 --- a/src/modules/AdminModule.cpp +++ b/src/modules/AdminModule.cpp @@ -181,8 +181,11 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta // without the user doing so deliberately. LOG_INFO("PKC admin valid, but not auto-favoriting node %x because role==CLIENT_BASE", mp.from); } else { - LOG_INFO("PKC admin valid. Auto-favoriting node %x", mp.from); - nodeInfoLiteSetBit(remoteNode, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); + if (nodeDB->setProtectedFlag(remoteNode, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true)) { + LOG_INFO("PKC admin valid. Auto-favoriting node %x", mp.from); + } else { + LOG_WARN("PKC admin valid, but auto-favorite refused for node %x (protected-node cap)", mp.from); + } } } } else { @@ -472,10 +475,13 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta LOG_INFO("Client received set_favorite_node command"); meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(r->set_favorite_node); if (node != NULL) { - nodeInfoLiteSetBit(node, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); - saveChanges(SEGMENT_NODEDATABASE, false); - if (screen) - screen->setFrames(graphics::Screen::FOCUS_PRESERVE); // <-- Rebuild screens + if (nodeDB->setProtectedFlag(node, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true)) { + saveChanges(SEGMENT_NODEDATABASE, false); + if (screen) + screen->setFrames(graphics::Screen::FOCUS_PRESERVE); // <-- Rebuild screens + } else if (mp.from == 0) { // local request from the phone — tell the user why it didn't take + sendWarning(NodeDB::PROTECTED_CAP_WARN_FMT, "favorite", r->set_favorite_node, MAX_NUM_NODES - 2); + } } break; } @@ -492,13 +498,17 @@ bool AdminModule::handleReceivedProtobuf(const meshtastic_MeshPacket &mp, meshta } case meshtastic_AdminMessage_set_ignored_node_tag: { LOG_INFO("Client received set_ignored_node command"); - meshtastic_NodeInfoLite *node = nodeDB->getMeshNode(r->set_ignored_node); + // Unlike the sibling node-targeted admin commands, create the entry if + // it's absent so the block sticks for a node we've not heard from yet + // (e.g. one a remote admin asks us to block) with no NodeInfo or key. + meshtastic_NodeInfoLite *node = nodeDB->getOrCreateMeshNode(r->set_ignored_node); if (node != NULL) { - nodeInfoLiteSetBit(node, NODEINFO_BITFIELD_IS_IGNORED_MASK, true); - nodeDB->eraseNodeSatellites(node->num); - node->public_key.size = 0; - memset(node->public_key.bytes, 0, sizeof(node->public_key.bytes)); - saveChanges(SEGMENT_NODEDATABASE, false); + if (nodeDB->setProtectedFlag(node, NODEINFO_BITFIELD_IS_IGNORED_MASK, true)) { + nodeDB->eraseNodeSatellites(node->num); + saveChanges(SEGMENT_NODEDATABASE, false); + } else if (mp.from == 0) { // local request from the phone — tell the user why it didn't take + sendWarning(NodeDB::PROTECTED_CAP_WARN_FMT, "ignore", r->set_ignored_node, MAX_NUM_NODES - 2); + } } break; } @@ -1405,6 +1415,13 @@ void AdminModule::handleGetNodeRemoteHardwarePins(const meshtastic_MeshPacket &r void AdminModule::handleGetDeviceMetadata(const meshtastic_MeshPacket &req) { +#if WARM_NODE_COUNT > 0 && MESHTASTIC_NODEDB_MIGRATION_VERBOSE + // Debug aid: dump the warm tier to the console on a local metadata request + // (e.g. `meshtastic --info` over USB/BLE). Gated to req.from == 0 so remote + // or admin polling can't spam the console. + if (nodeDB && req.from == 0) + nodeDB->warmStore.dumpToLog("admin get_metadata"); +#endif meshtastic_AdminMessage r = meshtastic_AdminMessage_init_default; r.get_device_metadata_response = getDeviceMetadata(); r.which_payload_variant = meshtastic_AdminMessage_get_device_metadata_response_tag; diff --git a/src/platform/nrf52/nrf52840_s140_v6.ld b/src/platform/nrf52/nrf52840_s140_v6.ld new file mode 100644 index 000000000..c8dac4e55 --- /dev/null +++ b/src/platform/nrf52/nrf52840_s140_v6.ld @@ -0,0 +1,46 @@ +/* Linker script to configure memory regions. */ + +SEARCH_DIR(.) +GROUP(-lgcc -lc -lnosys) + +MEMORY +{ + /* App region ends at 0xEA000, not 0xED000: the 12 KB warm-node-store + * record-ring (3 x 4 KB pages, src/mesh/WarmNodeStore.h) occupies + * 0xEA000-0xED000, directly below the stock LittleFS partition. Boards on + * the framework-default linker script are covered by the post-link guard + * in extra_scripts/nrf52_warm_region.py instead. + * + * S140 v6.x app region starts at 0x26000 (152 KB SoftDevice); v7.x uses + * 0x27000. All other boundaries are identical — see nrf52840_s140_v7.ld. */ + FLASH (rx) : ORIGIN = 0x26000, LENGTH = 0xEA000 - 0x26000 + + /* SRAM required by Softdevice depend on + * - Attribute Table Size (Number of Services and Characteristics) + * - Vendor UUID count + * - Max ATT MTU + * - Concurrent connection peripheral + central + secure links + * - Event Len, HVN queue, Write CMD queue + */ + RAM (rwx) : ORIGIN = 0x20006000, LENGTH = 0x20040000 - 0x20006000 +} + +SECTIONS +{ + . = ALIGN(4); + .svc_data : + { + PROVIDE(__start_svc_data = .); + KEEP(*(.svc_data)) + PROVIDE(__stop_svc_data = .); + } > RAM + + .fs_data : + { + PROVIDE(__start_fs_data = .); + KEEP(*(.fs_data)) + PROVIDE(__stop_fs_data = .); + } > RAM +} INSERT AFTER .data; + +INCLUDE "nrf52_common.ld" diff --git a/src/platform/nrf52/nrf52840_s140_v7.ld b/src/platform/nrf52/nrf52840_s140_v7.ld index 6aaeb4034..4546b4a7a 100644 --- a/src/platform/nrf52/nrf52840_s140_v7.ld +++ b/src/platform/nrf52/nrf52840_s140_v7.ld @@ -5,7 +5,12 @@ GROUP(-lgcc -lc -lnosys) MEMORY { - FLASH (rx) : ORIGIN = 0x27000, LENGTH = 0xED000 - 0x27000 + /* App region ends at 0xEA000, not 0xED000: the 12 KB warm-node-store + * record-ring (3 x 4 KB pages, src/mesh/WarmNodeStore.h) occupies + * 0xEA000-0xED000, directly below the stock LittleFS partition. Boards on + * the framework-default linker script are covered by the post-link guard + * in extra_scripts/nrf52_warm_region.py instead. */ + FLASH (rx) : ORIGIN = 0x27000, LENGTH = 0xEA000 - 0x27000 /* SRAM required by Softdevice depend on * - Attribute Table Size (Number of Services and Characteristics) diff --git a/test/test_nodedb_blocked/test_main.cpp b/test/test_nodedb_blocked/test_main.cpp new file mode 100644 index 000000000..2bbe9fb6b --- /dev/null +++ b/test/test_nodedb_blocked/test_main.cpp @@ -0,0 +1,193 @@ +// Tests for the NodeDB hot-store migration and favourite/ignored (blocked) +// retention paths — src/mesh/NodeDB.cpp. +#include "MeshTypes.h" // BEFORE TestUtil.h — provides WARM_NODE_COUNT / MAX_NUM_NODES via mesh-pb-constants.h +#include "TestUtil.h" +#include + +#if defined(ARCH_PORTDUINO) +#define NDB_TEST_ENTRY extern "C" +#else +#define NDB_TEST_ENTRY +#endif + +// The migration demotes overflow into the warm tier, so these tests need it. +#if WARM_NODE_COUNT > 0 + +#include "mesh/NodeDB.h" +#include + +// Subclass shim: exposes the private maintenance paths (via the friend +// declaration in NodeDB.h) and lets a test own the hot store directly +// (meshNodes/numMeshNodes are public). Declared at global scope so it matches +// `friend class NodeDBTestShim` — an anonymous-namespace class would not. +class NodeDBTestShim : public NodeDB +{ + public: + void runDemote() { demoteOldestHotNodesToWarm(); } + void runCleanup() { cleanupMeshDB(); } + + void clearHot() + { + meshNodes->clear(); + numMeshNodes = 0; + } + + void push(NodeNum num, uint32_t lastHeard, bool favorite, bool ignored, bool withUser, bool withKey) + { + meshtastic_NodeInfoLite n = meshtastic_NodeInfoLite_init_zero; + n.num = num; + n.last_heard = lastHeard; + if (favorite) + nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_IS_FAVORITE_MASK, true); + if (ignored) + nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_IS_IGNORED_MASK, true); + if (withUser) + nodeInfoLiteSetBit(&n, NODEINFO_BITFIELD_HAS_USER_MASK, true); + if (withKey) { + n.public_key.size = 32; + memset(n.public_key.bytes, static_cast(num & 0xff), 32); + n.public_key.bytes[0] = 0x01; // ensure non-zero (all-zero == "no key") + } + meshNodes->push_back(n); + numMeshNodes = meshNodes->size(); + } + + // Index 0 is our own node; the eviction/migration scans treat it as self. + void seedSelf() { push(0x0BADF00D, 0xFFFFFFFFu, false, false, /*withUser=*/true, /*withKey=*/false); } +}; + +namespace +{ + +NodeDBTestShim *db = nullptr; + +bool warmHasKey(NodeNum n) +{ + meshtastic_NodeInfoLite_public_key_t k = {0, {0}}; + return db->copyPublicKey(n, k) && k.size == 32; +} + +} // namespace + +void setUp(void) +{ + db->clearHot(); +} +void tearDown(void) {} + +// Migration: a database from a larger-cap build trims to MAX_NUM_NODES; the +// oldest non-protected nodes are demoted into the warm tier (keys preserved), +// while self, favourites and ignored survive even when they are the oldest. +static void test_migration_demotesOldestKeepsKeepersAndSelf(void) +{ + db->seedSelf(); + const int extra = MAX_NUM_NODES + 30; // overflow well past the MAX-2 cap + for (int i = 1; i <= extra; i++) { + const bool fav = (i == 1); // oldest, but a favourite + const bool ign = (i == 2); // 2nd-oldest, but blocked + db->push(2000 + i, /*last_heard=*/i, fav, ign, /*withUser=*/true, /*withKey=*/true); + } + + db->runDemote(); + + TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES, (int)db->getNumMeshNodes()); + TEST_ASSERT_NOT_NULL(db->getMeshNode(0x0BADF00D)); // self retained + TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + 1)); // oldest favourite retained + TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + 2)); // oldest ignored retained + TEST_ASSERT_NOT_NULL(db->getMeshNode(2000 + extra)); // freshest retained + TEST_ASSERT_NULL(db->getMeshNode(2000 + 3)); // oldest non-protected demoted out of hot + TEST_ASSERT_TRUE(warmHasKey(2000 + 3)); // ...but its key kept in the warm tier +} + +// Favourite handling: a favourite is never the eviction victim, even when it is +// the oldest node in a full hot store. +static void test_eviction_preservesFavorite(void) +{ + db->seedSelf(); + for (int i = 1; i < MAX_NUM_NODES; i++) { // fill to MAX_NUM_NODES total (incl. self) + const bool fav = (i == 1); // oldest non-self, favourite + db->push(3000 + i, /*last_heard=*/i, fav, false, /*withUser=*/true, /*withKey=*/true); + } + TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES, (int)db->getNumMeshNodes()); // full + + TEST_ASSERT_NOT_NULL(db->getOrCreateMeshNode(0x99990000)); // forces an eviction + + TEST_ASSERT_NOT_NULL(db->getMeshNode(3000 + 1)); // favourite survived despite being oldest + TEST_ASSERT_NULL(db->getMeshNode(3000 + 2)); // oldest non-favourite evicted + TEST_ASSERT_NOT_NULL(db->getMeshNode(0x99990000)); +} + +// Ignored handling: an ignored node survives eviction (like a favourite), and is +// never purged by cleanupMeshDB even with no user info (a block set by bare ID). +static void test_ignored_survivesEvictionAndCleanup(void) +{ + // (a) eviction protection + db->clearHot(); + db->seedSelf(); + for (int i = 1; i < MAX_NUM_NODES; i++) { + const bool ign = (i == 1); // oldest non-self, blocked + db->push(4000 + i, /*last_heard=*/i, false, ign, /*withUser=*/true, /*withKey=*/true); + } + TEST_ASSERT_NOT_NULL(db->getOrCreateMeshNode(0x88880000)); + TEST_ASSERT_NOT_NULL(db->getMeshNode(4000 + 1)); // blocked node survived + TEST_ASSERT_NULL(db->getMeshNode(4000 + 2)); // oldest non-blocked evicted + + // (b) cleanup protection — ignored kept without user info, plain no-user purged + db->clearHot(); + db->seedSelf(); + db->push(5000, 100, false, /*ignored=*/true, /*withUser=*/false, false); + db->push(5001, 100, false, false, /*withUser=*/false, false); + db->runCleanup(); + TEST_ASSERT_NOT_NULL(db->getMeshNode(5000)); // blocked-by-ID kept despite no user info + TEST_ASSERT_NULL(db->getMeshNode(5001)); // ordinary no-user node purged +} + +// Protected-node cap: at most MAX_NUM_NODES-2 nodes may be protected, so >=2 +// evictable slots always remain. setProtectedFlag refuses once the cap is hit. +static void test_protectedCap_refusesBeyondLimit(void) +{ + db->seedSelf(); + for (int i = 0; i < MAX_NUM_NODES - 2; i++) + db->push(6000 + i, 100, /*favorite=*/true, false, /*withUser=*/true, false); + TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES - 2, db->numProtectedNodes()); + + db->push(7000, 100, false, false, /*withUser=*/true, false); + meshtastic_NodeInfoLite *fresh = db->getMeshNode(7000); + TEST_ASSERT_NOT_NULL(fresh); + TEST_ASSERT_FALSE(db->setProtectedFlag(fresh, NODEINFO_BITFIELD_IS_IGNORED_MASK, true)); // refused at cap + TEST_ASSERT_FALSE(nodeInfoLiteIsIgnored(fresh)); // unchanged + TEST_ASSERT_EQUAL_INT(MAX_NUM_NODES - 2, db->numProtectedNodes()); + + // Adding another flag to an already-protected node doesn't grow the set, so + // it's still allowed at the cap. + meshtastic_NodeInfoLite *already = db->getMeshNode(6000); + TEST_ASSERT_TRUE(db->setProtectedFlag(already, NODEINFO_BITFIELD_IS_IGNORED_MASK, true)); +} + +NDB_TEST_ENTRY void setup() +{ + initializeTestEnvironment(); + db = new NodeDBTestShim(); + nodeDB = db; + + UNITY_BEGIN(); + RUN_TEST(test_migration_demotesOldestKeepsKeepersAndSelf); + RUN_TEST(test_eviction_preservesFavorite); + RUN_TEST(test_ignored_survivesEvictionAndCleanup); + RUN_TEST(test_protectedCap_refusesBeyondLimit); + exit(UNITY_END()); +} +NDB_TEST_ENTRY void loop() {} + +#else // WARM_NODE_COUNT == 0 — nothing to exercise here + +void setUp(void) {} +void tearDown(void) {} +NDB_TEST_ENTRY void setup() +{ + UNITY_BEGIN(); + exit(UNITY_END()); +} +NDB_TEST_ENTRY void loop() {} + +#endif diff --git a/test/test_warm_store/test_main.cpp b/test/test_warm_store/test_main.cpp new file mode 100644 index 000000000..3c8455883 --- /dev/null +++ b/test/test_warm_store/test_main.cpp @@ -0,0 +1,211 @@ +// Unit tests for the warm ("long-tail") node tier — src/mesh/WarmNodeStore.cpp. +// Covers admission/eviction policy (keyed entries outrank keyless), take() +// rehydration semantics, and a tolerant persistence round trip. +#include "MeshTypes.h" // BEFORE TestUtil.h — provides WARM_NODE_COUNT via mesh-pb-constants.h +#include "TestUtil.h" +#include + +#if defined(ARCH_PORTDUINO) +#define WS_TEST_ENTRY extern "C" +#else +#define WS_TEST_ENTRY +#endif + +#if WARM_NODE_COUNT > 0 + +#include "mesh/WarmNodeStore.h" +#include + +namespace +{ + +void makeKey(uint8_t out[32], uint8_t seed) +{ + memset(out, 0, 32); + out[0] = seed; + out[31] = seed ^ 0xA5; +} + +} // namespace + +void setUp(void) {} +void tearDown(void) {} + +void test_ws_absorb_and_copyKey_roundTrip() +{ + WarmNodeStore ws; + uint8_t key[32], got[32]; + makeKey(key, 7); + TEST_ASSERT_TRUE(ws.absorb(0x100, 1000, key)); + TEST_ASSERT_TRUE(ws.contains(0x100)); + TEST_ASSERT_TRUE(ws.copyKey(0x100, got)); + TEST_ASSERT_EQUAL_MEMORY(key, got, 32); + TEST_ASSERT_EQUAL(1, ws.count()); +} + +void test_ws_keylessEntry_hasNoKey() +{ + WarmNodeStore ws; + uint8_t got[32]; + TEST_ASSERT_TRUE(ws.absorb(0x200, 1000, NULL)); + TEST_ASSERT_TRUE(ws.contains(0x200)); + TEST_ASSERT_FALSE(ws.copyKey(0x200, got)); +} + +void test_ws_absorb_rejectsNodeNumZero() +{ + WarmNodeStore ws; + TEST_ASSERT_FALSE(ws.absorb(0, 1000, NULL)); + TEST_ASSERT_EQUAL(0, ws.count()); +} + +void test_ws_absorb_updatesExistingEntry() +{ + WarmNodeStore ws; + uint8_t key[32], got[32]; + makeKey(key, 9); + TEST_ASSERT_TRUE(ws.absorb(0x300, 1000, NULL)); + TEST_ASSERT_TRUE(ws.absorb(0x300, 2000, key)); // later eviction learned a key + TEST_ASSERT_EQUAL(1, ws.count()); + TEST_ASSERT_TRUE(ws.copyKey(0x300, got)); + TEST_ASSERT_EQUAL_MEMORY(key, got, 32); +} + +void test_ws_take_removesEntry() +{ + WarmNodeStore ws; + uint8_t key[32]; + makeKey(key, 3); + ws.absorb(0x400, 1234, key); + + WarmNodeEntry e; + TEST_ASSERT_TRUE(ws.take(0x400, e)); + TEST_ASSERT_EQUAL(0x400, e.num); + TEST_ASSERT_EQUAL(1234, e.last_heard); + TEST_ASSERT_EQUAL_MEMORY(key, e.public_key, 32); + TEST_ASSERT_FALSE(ws.contains(0x400)); + TEST_ASSERT_FALSE(ws.take(0x400, e)); + TEST_ASSERT_EQUAL(0, ws.count()); +} + +void test_ws_keylessCandidate_neverEvictsKeyedEntries() +{ + WarmNodeStore ws; + uint8_t key[32]; + // Fill the store entirely with keyed entries + for (size_t i = 0; i < ws.capacity(); i++) { + makeKey(key, (uint8_t)i); + TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, 100 + i, key)); + } + TEST_ASSERT_EQUAL(ws.capacity(), ws.count()); + // A keyless candidate (even a fresh one) must be rejected + TEST_ASSERT_FALSE(ws.absorb(0x9999, 999999, NULL)); + TEST_ASSERT_FALSE(ws.contains(0x9999)); +} + +void test_ws_keyedCandidate_evictsOldestKeylessFirst() +{ + WarmNodeStore ws; + uint8_t key[32]; + makeKey(key, 0x42); + // Fill with keyed entries except two keyless ones in the middle + for (size_t i = 0; i < ws.capacity(); i++) { + const bool keyless = (i == 5 || i == 10); + TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, keyless ? (i == 10 ? 50 : 60) : 10, keyless ? NULL : key)); + } + // Keyed candidate must displace the OLDEST KEYLESS entry (0x100A, ts=50), + // even though every keyed entry is older (ts=10) + uint8_t k2[32]; + makeKey(k2, 0x43); + TEST_ASSERT_TRUE(ws.absorb(0x8888, 70, k2)); + TEST_ASSERT_FALSE(ws.contains(0x1000 + 10)); + TEST_ASSERT_TRUE(ws.contains(0x1000 + 5)); + TEST_ASSERT_TRUE(ws.contains(0x8888)); +} + +void test_ws_keyedCandidate_evictsOldestKeyedWhenNoKeyless() +{ + WarmNodeStore ws; + uint8_t key[32]; + for (size_t i = 0; i < ws.capacity(); i++) { + makeKey(key, (uint8_t)i); + TEST_ASSERT_TRUE(ws.absorb(0x1000 + i, 1000 + i, key)); // 0x1000 is the oldest + } + uint8_t k2[32]; + makeKey(k2, 0x44); + TEST_ASSERT_TRUE(ws.absorb(0x7777, 999999, k2)); + TEST_ASSERT_TRUE(ws.contains(0x7777)); + TEST_ASSERT_FALSE(ws.contains(0x1000)); // oldest keyed evicted + TEST_ASSERT_EQUAL(ws.capacity(), ws.count()); +} + +void test_ws_remove_and_clear() +{ + WarmNodeStore ws; + ws.absorb(0x500, 1, NULL); + ws.absorb(0x501, 2, NULL); + ws.remove(0x500); + TEST_ASSERT_FALSE(ws.contains(0x500)); + TEST_ASSERT_EQUAL(1, ws.count()); + ws.clear(); + TEST_ASSERT_EQUAL(0, ws.count()); +} + +void test_ws_persistence_roundTrip() +{ + WarmNodeStore a; + uint8_t key[32], got[32]; + makeKey(key, 0x55); + a.absorb(0x600, 4242, key); + a.absorb(0x601, 4243, NULL); + if (!a.saveIfDirty()) { + TEST_IGNORE_MESSAGE("Filesystem not available in this test environment"); + return; + } + + WarmNodeStore b; + b.load(); + TEST_ASSERT_TRUE(b.contains(0x600)); + TEST_ASSERT_TRUE(b.contains(0x601)); + TEST_ASSERT_TRUE(b.copyKey(0x600, got)); + TEST_ASSERT_EQUAL_MEMORY(key, got, 32); + + // Cleanup so reruns start fresh + b.clear(); + b.saveIfDirty(); +} + +WS_TEST_ENTRY void setup() +{ + initializeTestEnvironment(); + UNITY_BEGIN(); + RUN_TEST(test_ws_absorb_and_copyKey_roundTrip); + RUN_TEST(test_ws_keylessEntry_hasNoKey); + RUN_TEST(test_ws_absorb_rejectsNodeNumZero); + RUN_TEST(test_ws_absorb_updatesExistingEntry); + RUN_TEST(test_ws_take_removesEntry); + RUN_TEST(test_ws_keylessCandidate_neverEvictsKeyedEntries); + RUN_TEST(test_ws_keyedCandidate_evictsOldestKeylessFirst); + RUN_TEST(test_ws_keyedCandidate_evictsOldestKeyedWhenNoKeyless); + RUN_TEST(test_ws_remove_and_clear); + RUN_TEST(test_ws_persistence_roundTrip); + exit(UNITY_END()); +} + +WS_TEST_ENTRY void loop() {} + +#else + +void setUp(void) {} +void tearDown(void) {} + +WS_TEST_ENTRY void setup() +{ + initializeTestEnvironment(); + UNITY_BEGIN(); + exit(UNITY_END()); +} + +WS_TEST_ENTRY void loop() {} + +#endif diff --git a/variants/nrf52840/nrf52.ini b/variants/nrf52840/nrf52.ini index f7f3d0885..56ab06ef1 100644 --- a/variants/nrf52840/nrf52.ini +++ b/variants/nrf52840/nrf52.ini @@ -15,6 +15,7 @@ extra_scripts = ${env.extra_scripts} extra_scripts/nrf52_extra.py pre:extra_scripts/nrf52_lto.py + extra_scripts/nrf52_warm_region.py ; post-link guard: image must end below the 12 KB warm-store raw-flash region at 0xEA000-0xED000 build_type = release build_flags = diff --git a/variants/nrf52840/nrf52840.ini b/variants/nrf52840/nrf52840.ini index c5590cbc3..610b5b530 100644 --- a/variants/nrf52840/nrf52840.ini +++ b/variants/nrf52840/nrf52840.ini @@ -1,6 +1,10 @@ [nrf52840_base] extends = nrf52_base +; Cap the app image at 0xEA000 (below the WarmNodeStore raw-flash region). +; Boards that have upgraded to S140 v7 override this in their own platformio.ini. +board_build.ldscript = src/platform/nrf52/nrf52840_s140_v6.ld + build_flags = ${nrf52_base.build_flags} -DSERIAL_BUFFER_SIZE=4096