develop
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  
|
d846780a9b |
More fuzz tests and small fixes for the findings (#10864)
* first pass tests * more tests * Fix two crafted-admin-packet crashes found by the E5 fuzzer Both are reachable from an authorized admin (local from==0, admin channel, or PKC) - remote DoS: 1. SIGFPE in LoRa config validation. A set_config LoRaConfig with use_preset=false and bandwidth=0 makes freqSlotWidth 0, so numFreqSlots is 0 and `hash(name) % numFreqSlots` (RadioInterface.cpp) divides by zero. Guard the modulo; the existing channel_num check then rejects/ clamps the config. 2. Stack overflow in Channels::getKey. A SECONDARY channel at the primary slot with an empty PSK recursed into getKey(primaryIndex) forever. Skip the primary-key borrow when chIndex == primaryIndex. Re-enable the E5 admin fuzzer to hit both triggers again (use_preset both ways incl. bandwidth 0, plus the set_channel tag) as regression guards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Correct fuzz-test invariants after the crash fixes - E5 admin fuzz: node eviction under a filling NodeDB is legitimate, so assert only the bounded-count invariant, not that a specific seed node survives 6000 mutating ops. - TMM blitz: scope off the nodeinfo direct-response send path (it needs a fully-wired MeshService/phone queue the fixture doesn't provide; the deterministic directResponse tests cover it). The crafted-nodenum rate/unknown/position cache stress is unchanged. clod helped too * realistic tests * test: dedup fuzz RNG into shared test/support/DeterministicRng.h The four in-tree fuzz suites (test_fuzz_decode, test_fuzz_packets, test_hop_scaling, test_traffic_management) each carried a byte-identical copy of the seeded 64-bit LCG (rngSeed/rngNext/rngByte/rngRange). Hoist it into one shared header so there is a single generator to reason about and no risk of the copies drifting. static inline keeps per-suite state per translation unit and avoids -Wunused-function for suites that don't use every helper. Also corrects a stale comment in test_traffic_management (the blitz's nodeinfo direct-response path is intentionally left off). No behavioral change: same constants, same per-suite seeds. clod helped too * test: fuzz uncovered ProtobufModule handlers and the MQTT downlink ingress Extend the in-tree fuzz coverage to packet sources that previously had none: - test_fuzz_packets E8/E9/E10: drive PositionModule, DeviceTelemetryModule and NeighborInfoModule at handleReceivedProtobuf directly (via using-shims, bypassing the ProtobufModule reply/send path so no router is needed). The fixture already stands up nodeDB/service/channels, and nodeStatus/powerStatus are auto-initialized in main.cpp, so no new globals are required. Adds a shared fuzzRxHeader() helper for crafting adversarial RX packet headers. - test_fuzz_decode: add meshtastic_KeyVerification to the decode table. The KeyVerification and StoreForward handler paths are documented as decode-level only, with the concrete reason each is intrinsic (private-state gating / PSRAM + self-pointer wiring), not a fixture gap. - test_mqtt: test_receiveFuzzServiceEnvelope blitzes the non-RF broker-push ingress (onReceiveProto) two ways - raw garbage bytes that must fail envelope decode cleanly, and a well-formed ServiceEnvelope wrapping a crafted inner MeshPacket over crafted channel_id/gateway_id - exercising the channel match, isFromUs, XEdDSA receive policy and perhapsDecode chain. Adds a deliverRaw() passthrough to MQTTUnitTest. All under the coverage env (ASan/LSan). No firmware/src changes. Full sweep GREEN 27/27, 544 cases. clod helped too * Harden LoRa/channel config against crafted admin messages; consolidate test helpers Production (review findings on the hot-fuzz crash fixes): - Clamp bandwidth at the source (clampBandwidthKHz) in checkOrClampConfigLora and applyModemConfig so numFreqSlots can never be 0 for any consumer; a bandwidth-0 set_config previously passed validation and re-armed the SIGFPE on the next applyModemConfig. - Guard applyModemConfig's hash % numFreqSlots (the validator's sibling modulo was fixed earlier but this one was still unguarded). - Enforce the primary-channel invariant in Channels::onConfigChanged: a config demoting every slot now re-promotes the stale SECONDARY slot (keeping its key) or restores the default channel if the slot is DISABLED, instead of leaving every getPrimaryIndex() reader on a non-primary slot. The getKey recursion guard stays as defense-in-depth. Tests: - New test/support/MockMeshService.h and AdminModuleTestShim.h replace four byte-identical mocks and three divergent admin shims (test_mqtt's capturing mock is genuinely different and stays). - DeterministicRng.h: add rngFill() (replaces 14 hand-rolled fill loops) and rngEdgeNodeNum() (unifies the three NodeNum boundary pools). - Extract fuzzChannelSettings() shared by the set_channel case and fuzzBeacon. - fuzzBeacon: the un-terminated branch now fills the whole buffer with non-NUL bytes so the strnlen bound is actually stressed (~50% of iterations, not ~4%). - E6 beacon fuzz: replace the TEST_ASSERT_TRUE(true) tautology with real invariants (handler never consumes; offers land in lastReceivedOffer keyed to the sender). - Trim the seven over-long comment blocks flagged against the 1-2 line rule; the FINDINGS trailer moves to this commit message (see production notes). Full native suite GREEN 27/27 under the coverage (ASan/LSan) env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Clamp UTF-8 char length in the emote walkers; add test_fuzz_emotes A TEXT_MESSAGE payload is opaque protobuf bytes, so PB_VALIDATE_UTF8 never screens it - invalid UTF-8 and truncated multi-byte lead bytes reach the emote/width render path verbatim. EmoteRenderer's walkers advanced by utf8CharLen(lead) without clamping to the bytes actually remaining, so a truncated lead (e.g. a lone 0xF0, which claims 4 bytes) near the end of the buffer made getUtf8ChunkWidth's memcpy read past the string. ASan confirms a heap-buffer-overflow READ from measureStringWithEmotes. Add utf8CharLenClamped() and use it at every walk site (width measure, truncation cut-loop, and the draw-path text-run/chunk builders); the one already-guarded site (matchAtIgnoringModifiers) is unchanged. New test/test_fuzz_emotes drives measureStringWithEmotes and truncateToWidth over adversarial byte strings (biased to embed/end in truncated multi-byte leads) in exact-sized heap buffers so any over-read is a hard ASan fault. Its headless display uses a synthetic font (firstChar 0, fontData centered in a large buffer) so the stock OLEDDisplay::getStringWidth - which indexes the font jump table with a signed char and over-reads for any byte >= 0x80 - does not mask the finding. native-suite-count bumped 27 -> 28. Full native suite GREEN 28/28 under the coverage (ASan/LSan) env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * Keep emote width measurement in-bounds for non-ASCII bytes OLEDDisplay::getStringWidth (the utf8=false path EmoteRenderer uses on default builds) indexes the font jump table by (c - firstChar) with a signed char and no bounds check, so any byte outside printable ASCII - high bytes from UTF-8 text, but also a stray control byte like 0x0A - reads outside the font array. On-device this reads adjacent flash and returns a garbage width; under ASan the test_fuzz_emotes fuzzer flags it as a global-buffer-overflow, and it made the non-ASCII width measurement meaningless either way. The OLED driver is a pinned upstream dependency, so guard it firmware-side in EmoteRenderer's getStringWidth helper: measure a sanitized copy where any byte outside [0x20, 0x7E] counts as a '?' placeholder. Printable ASCII is unchanged and the UA/RU lookup path is untouched. test_fuzz_emotes now drives a real ArialMT font instead of the synthetic in-bounds font it needed before this fix, so the suite exercises the true production width path (utf8CharLen clamp + this sanitizer) end to end. The same fuzzer tripped the global-buffer-overflow before this change. Full native suite GREEN 28/28 under the coverage (ASan/LSan) env. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
TomandGitHub
|
3becaf2d95 | emdashes begone (#10847) | ||
|
de345939af |
Automatic variable hop limits based on mesh activity and size estimation (#10176)
* asdf * Implement SphereOfInfluenceModule for traffic management and eviction tracking * Implement Sphere of Influence module for dynamic hop limit adjustment and role-based floor * Update SAMPLING_DENOMINATOR to improve mesh size estimation accuracy * Add debug logging for scale factor estimation and per-hop node counts in SphereOfInfluenceModule * Enable variable hop limits and role-based hop floors in Sphere of Influence module * Respond to copilot review * Disable variable hop limits and role-based hop floors in Sphere of Influence module * Apply suggestions from code review Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> * Implement adaptive sampling for unique node ID tracking in Sphere of Influence module * Add state persistence for Sphere of Influence module * Enhance Sphere of Influence module with state management and adaptive sampling adjustments * Refactor hop scaling functionality: remove SphereOfInfluenceModule and introduce HopScalingModule - Deleted SphereOfInfluenceModule.h, consolidating its responsibilities into the new HopScalingModule. - Added HopScalingModule.h and HopScalingModule.cpp to manage hop scaling logic, including eviction tracking and sampling-based mesh size estimation. - Implemented methods for recording evictions and packet senders, estimating scale factors, and computing required hops based on node activity. - Introduced state persistence for hop scaling parameters to maintain continuity across reboots. - Enhanced thread safety and modularity by utilizing concurrency features. * Guard out STM32. Sowwy. * Refactor HopScalingModule: enhance sampling logic and improve state management * Add unit tests for HopScalingModule: implement mock database and various test scenarios * Refactor test output in HopScalingModule tests: replace printf with TEST_MESSAGE for better integration with Unity * Refactor HopScalingModule logging: replace lastStatusMode with descriptive mode names for improved readability * Refactor test_main.cpp: change NodeNum variable to static and improve comments for clarity * Remove unnecessary delay in setup function for improved test performance * Add missing include for MeshTypes in test_main.cpp * Refactor HopScalingModule tests: enhance mesh topology scenarios and improve test clarity * Update HopScalingModule tests: flesh out node scenarios and improve clarity for dense and sparse mesh cases * Fix politeness factor calculations in HopScalingModule and update related test scenarios for clarity. Remove outdated design doc. * Enhance HopScalingModule: add sampled estimate for scaling decisions and refactor initial run state management * Add sample traffic injection for HopScaling tests to enhance sampledEst visibility * Enhance HopScalingModule: adjust windowFraction calculation for early triggers and improve test output formatting * Enhance HopScalingModule: add jitter functionality to sampling denominator and update tests for consistent behavior * Enhance HopScalingModule: implement adaptive sampling denominator adjustment and add reset functionality for tests * Enhance HopScalingTestShim: add test-only clock and window helpers, update injectSampleTraffic for adaptive sampling, and improve scenario summary output * Enhance HopScalingModule: add detailed documentation for functions, improve clarity of jitter and sampling logic, and reset functionality in tests * Enhance HopScalingModule: add evictionEstimate parameter to estimateScaleFactor and update related logging for improved mesh size estimation * Enhance HopScalingModule: adjust effective rolls calculation for improved accuracy, add eviction estimate logic, responding to all copilot review points * Implement CompactHistogram for parallel hop scaling sampling - Added CompactHistogram class to track node hop distances with bitwise sampling. - Integrated CompactHistogram into HopScalingModule for independent packet sampling. - Updated NodeDB to feed both the hop scaling module and the new histogram sampler. - Enhanced HopScalingModule with methods to sample packets for the histogram and retrieve hop distribution statistics. - Implemented tests for CompactHistogram functionality, including sampling, window rolling, and adaptive denominator scaling. - Updated existing tests to validate the integration of the new histogram sampling mechanism. * Enhance CompactHistogram and HopScalingModule: add per-hop distribution functionality, improve time handling for unit tests, and refine test setup for deterministic behavior * CompactHistogram: add mesh size estimation, improve entry replacement logic, and update logging for per-hop distribution * Refactor HopScalingModule and CompactHistogram integration - Removed the suggestedHopFromCompactHistogram function to streamline hop suggestion logic. - Updated HopScalingModule to directly utilize CompactHistogram's internal methods for hop suggestions and sampling. - Enhanced logging in HopScalingModule to provide detailed histogram statistics. - Modified test cases to ensure comprehensive coverage of new histogram behaviors and sampling logic. - Improved node ID distribution in tests to better exercise sampling mechanisms. - Ensured that filtering denominators are held for 12 hours before dropping, enhancing stability in sampling. * Refactor CompactHistogram to support 13-hour activity tracking and introduce politeness regimes - Updated the bitfield structure to accommodate 13-hour seen tracking. - Changed the logic in rollHour() to analyze activity over the last 0-2 hours vs. 1-3 hours for politeness factor calculation. - Introduced three politeness levels: GENEROUS, DEFAULT, and STRICT based on recent activity ratios. - Adjusted filtering and sampling logic to reflect the new 13-hour tracking period. - Updated unit tests to validate new behavior and ensure proper functionality of politeness regimes. * Enhance CompactHistogram and HopScalingModule for improved sampling and decision-making - Introduced a session-specific hash seed in CompactHistogram to reduce bias in node ID sampling. - Updated sampling logic to use hashed node IDs instead of raw IDs for filtering and entry management. - Added histogram rollover tracking in HopScalingModule to ensure proper decision-making after initial data collection. - Adjusted logging to reflect the active state of the histogram and its comparison with NodeDB advisory hops. - Enhanced unit tests to validate new sampling logic and memory layout changes. * Expose hashNodeId for testing in CompactHistogram * Add mesh trend statistics to CompactHistogram for enhanced activity tracking * Implement histogram state persistence in CompactHistogram with save and load functions * Refactor CompactHistogram to improve entry management and enhance rollHour logging * feat: add HopScalingModule for adaptive hop limit recommendations Introduces HopScalingModule, a sampled hop-distance histogram that recommends the minimum hop limit needed to reach ~40 nodes, and automatically reducing the hops as the mesh grows. Key design: - 512-byte packed histogram (128 × 4-byte Record entries) embedded in a new HopScalingModule. - Each Record: 16-bit node hash, 3-bit hop distance, 13-bit seen bitmap - Sampling filter: only nodes where (hash & (denom-1)) == 0 are kept; denominator doubles on overflow and halves when utilisation is low - Hourly rollHour(): tallies per-hop counts, walks scaled buckets to find the minimum hop satisfying TARGET_AFFECTED_NODES (40), applies a politeness extension based on recent/older activity ratio, shifts all seen bitmaps, and persists state to /prefs/hopScalingState.bin - Hop recommendation gated by bootstrap (requires >=1 rollHour before overriding HOP_MAX) - NodeDB calls samplePacketForHistogram() on every non-MQTT rx packet - Module also estimates total mesh size and logs useful information about mesh characteristics. Changes: - src/modules/HopScalingModule.h/.cpp: new module - src/mesh/NodeDB.cpp: wire up samplePacketForHistogram - src/mesh/Router.cpp: consume getLastRequiredHop() - test/test_hop_scaling/: 12-test suite covering all mesh topologies and anticipated operational requirements * test: increase run iterations in sparse to dense transition test * feat: refactor HopScalingModule to use RUNS_PER_HOUR constant and improve logging * feat: enhance HopScalingModule with filtering denominator management and add tests for state transitions * refactor: remove CompactHistogram module and related files * address copilot review comments * Tweak: packet sampling only lora * ove role-based hop floor logic and related definitions into the module - keep it in one place. * Refactor MockNodeDB to use nodeInfoLiteSetBit for MQTT flag setting * Refactor hop scaling parameters and logic to integer maths and put default values in defaults.h. Small flash size reduction, no functional impact. * Update unit test preprocessor directives to PIO_UNIT_TESTING for consistency * refactor: improve test output organization and clarity in hop scaling tests --------- Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> |