Traffic Management Module for packet forwarding logic (#9358)

* Add ESP32 Power Management lessons learned document

Documents our experimentation with ESP-IDF DFS and why it doesn't
work well for Meshtastic (RTOS locks, BLE locks, USB issues).

Proposes simpler alternative: manual setCpuFrequencyMhz() control
with explicit triggers for when to go fast vs slow.

* Addition of traffic management module

* Fixing compile issues, but may still need to update protobufs.

* Fixing log2Floor in cuckoo hash function

* Adding support for traffic management in PhoneAPI.

* Making router_preserve_hops work without checking if the previous hop was a router. Also works for CLIENT_BASE.

* Adding station-g2 and portduino varients to be able to use this module.

* Spoofing from address for nodeinfo cache

* Changing name and behavior for zero_hop_telemetry / zero_hop_position

* Name change for exhausting telemetry packets and setting hop_limit to 1 so it will be 0 when sent.

* Updated hop logic, including exhaustRequested flag to bypass some checks later in the code.

* Reducing memory on nrf52 nodes further to 12 bytes per entry, 12KB total using 8 bit hashes with 0.4% collision. Probably ok. Adding portduino to the platforms that don't need to worry about memory as much.

* Fixing hopsAway for nodeinfo responses.

* traffic_management.nodeinfo_direct_response_min_hops -> traffic_management.nodeinfo_direct_response_max_hops

* Removing dry run mode

* Updates to UnifiedCacheEntry to use a common cache, created defaults for some values, reduced a couple bytes per entry by using a resolution-scale time selection based on configuration value.

* Enhance traffic management logging and configuration. Updated log messages in NextHopRouter and Router to include more context. Adjusted traffic management configuration checks in AdminModule and improved cache handling in TrafficManagementModule. Ensured consistent enabling of traffic management across various variants.

* Implement destructor for TrafficManagementModule and improve cache allocation handling. The destructor ensures proper deallocation of cache memory based on its allocation source (PSRAM or heap). Additionally, updated cache allocation logic to log warnings only when PSRAM allocation fails.

* Update TrafficManagementModule with enhanced comments for clarity and improve cache handling logic. Update protobuf submodule to latest commit.

* Creating consistent log messages

* Remove docs/ESP32_Power_Management.md from traffic_module

* Add unit tests for Traffic Management Module functionality

* Fixing compile issues, but may still need to update protobufs.

* Adding support for traffic management in PhoneAPI.

* Making router_preserve_hops work without checking if the previous hop was a router. Also works for CLIENT_BASE.

* Enhance traffic management logging and configuration. Updated log messages in NextHopRouter and Router to include more context. Adjusted traffic management configuration checks in AdminModule and improved cache handling in TrafficManagementModule. Ensured consistent enabling of traffic management across various variants.

* Implement destructor for TrafficManagementModule and improve cache allocation handling. The destructor ensures proper deallocation of cache memory based on its allocation source (PSRAM or heap). Additionally, updated cache allocation logic to log warnings only when PSRAM allocation fails.

* Update TrafficManagementModule with enhanced comments for clarity and improve cache handling logic. Update protobuf submodule to latest commit.

* Add mock classes and unit tests for Traffic Management Module functionality.

* Refactor setup and loop functions in test_main.cpp to include extern "C" linkage

* Update comment to include reduced  memory requirements

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Re-arranging comments for programmers with the attention span of less than 5 lines of code.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* Update comments in TrafficManagementModule to reflect changes in timestamp epoch handling and memory optimization details.

* bug: Use node-wide config_ok_to_mqtt setting for cached NodeInfo replies.

* Better way to handle clearing the ok_to_mqtt bit

* Add bucketing to cuckoo hashing, allowing for 95% occupied rate before major eviction problems.

* Extend nodeinfo cache for psram devices.

* Refactor traffic management to make hop exhaustion packet-scoped. Nice catch.

* Implement better position precision sanitization in TrafficManagementModule.

* Added logic in TrafficManagementModule to invalidate stale traffic state. Also, added some tests to avoid future me from creating a regression here.

* Fixing tests for native

* Enhance TrafficManagementModule to improve NodeInfo response handling and position deduplication logic. Added tests to ensure local packets bypass transit filters and that NodeInfo requests correctly update the requester information in the cache. Updated deduplication checks to prevent dropping valid position packets under certain conditions.

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

src/mesh/Default.h

@@ -30,6 +30,11 @@
 #define min_node_info_broadcast_secs 60 * 60 // No regular broadcasts of more than once an hour
 #define min_neighbor_info_broadcast_secs 4 * 60 * 60
 #define default_map_publish_interval_secs 60 * 60
+
+// Traffic management defaults
+#define default_traffic_mgmt_position_precision_bits 24         // ~10m grid cells
+#define default_traffic_mgmt_position_min_interval_secs ONE_DAY // 1 day between identical positions
+
 #ifdef USERPREFS_RINGTONE_NAG_SECS
 #define default_ringtone_nag_secs USERPREFS_RINGTONE_NAG_SECS
 #else

src/mesh/NextHopRouter.cpp

@@ -4,6 +4,9 @@
 #if !MESHTASTIC_EXCLUDE_TRACEROUTE
 #include "modules/TraceRouteModule.h"
 #endif
+#if HAS_TRAFFIC_MANAGEMENT
+#include "modules/TrafficManagementModule.h"
+#endif
 #include "NodeDB.h"
 
 NextHopRouter::NextHopRouter() {}
@@ -126,15 +129,28 @@ void NextHopRouter::sniffReceived(const meshtastic_MeshPacket *p, const meshtast
 /* Check if we should be rebroadcasting this packet if so, do so. */
 bool NextHopRouter::perhapsRebroadcast(const meshtastic_MeshPacket *p)
 {
-    if (!isToUs(p) && !isFromUs(p) && p->hop_limit > 0) {
+    // Check if traffic management wants to exhaust this packet's hops
+    bool exhaustHops = false;
+#if HAS_TRAFFIC_MANAGEMENT
+    if (trafficManagementModule && trafficManagementModule->shouldExhaustHops(*p)) {
+        exhaustHops = true;
+    }
+#endif
+
+    // Allow rebroadcast if hop_limit > 0 OR if we're exhausting hops (which sets hop_limit = 0 but still needs one relay)
+    if (!isToUs(p) && !isFromUs(p) && (p->hop_limit > 0 || exhaustHops)) {
         if (p->id != 0) {
             if (isRebroadcaster()) {
                 if (p->next_hop == NO_NEXT_HOP_PREFERENCE || p->next_hop == nodeDB->getLastByteOfNodeNum(getNodeNum())) {
                     meshtastic_MeshPacket *tosend = packetPool.allocCopy(*p); // keep a copy because we will be sending it
                     LOG_INFO("Rebroadcast received message coming from %x", p->relay_node);
 
-                    // Use shared logic to determine if hop_limit should be decremented
-                    if (shouldDecrementHopLimit(p)) {
+                    // If exhausting hops, force hop_limit = 0 regardless of other logic
+                    if (exhaustHops) {
+                        tosend->hop_limit = 0;
+                        LOG_INFO("Traffic management: exhausting hops for 0x%08x, setting hop_limit=0", getFrom(p));
+                    } else if (shouldDecrementHopLimit(p)) {
+                        // Use shared logic to determine if hop_limit should be decremented
                         tosend->hop_limit--; // bump down the hop count
                     } else {
                         LOG_INFO("favorite-ROUTER/CLIENT_BASE-to-ROUTER/CLIENT_BASE rebroadcast: preserving hop_limit");

src/mesh/PhoneAPI.cpp

@@ -449,6 +449,11 @@ size_t PhoneAPI::getFromRadio(uint8_t *buf)
             fromRadioScratch.moduleConfig.which_payload_variant = meshtastic_ModuleConfig_paxcounter_tag;
             fromRadioScratch.moduleConfig.payload_variant.paxcounter = moduleConfig.paxcounter;
             break;
+        case meshtastic_ModuleConfig_traffic_management_tag:
+            LOG_DEBUG("Send module config: traffic management");
+            fromRadioScratch.moduleConfig.which_payload_variant = meshtastic_ModuleConfig_traffic_management_tag;
+            fromRadioScratch.moduleConfig.payload_variant.traffic_management = moduleConfig.traffic_management;
+            break;
         default:
             LOG_ERROR("Unknown module config type %d", config_state);
         }

src/mesh/Router.cpp

@@ -11,6 +11,9 @@
 #include "mesh-pb-constants.h"
 #include "meshUtils.h"
 #include "modules/RoutingModule.h"
+#if HAS_TRAFFIC_MANAGEMENT
+#include "modules/TrafficManagementModule.h"
+#endif
 #if !MESHTASTIC_EXCLUDE_MQTT
 #include "mqtt/MQTT.h"
 #endif
@@ -95,6 +98,20 @@ bool Router::shouldDecrementHopLimit(const meshtastic_MeshPacket *p)
         return true;
     }
 
+#if HAS_TRAFFIC_MANAGEMENT
+    // When router_preserve_hops is enabled, preserve hops for decoded packets that are not
+    // position or telemetry (those have their own exhaust_hop controls).
+    if (moduleConfig.has_traffic_management && moduleConfig.traffic_management.enabled &&
+        moduleConfig.traffic_management.router_preserve_hops && p->which_payload_variant == meshtastic_MeshPacket_decoded_tag &&
+        p->decoded.portnum != meshtastic_PortNum_POSITION_APP && p->decoded.portnum != meshtastic_PortNum_TELEMETRY_APP) {
+        LOG_DEBUG("Router hop preserved: port=%d from=0x%08x (traffic_management)", p->decoded.portnum, getFrom(p));
+        if (trafficManagementModule) {
+            trafficManagementModule->recordRouterHopPreserved();
+        }
+        return false;
+    }
+#endif
+
     // For subsequent hops, check if previous relay is a favorite router
     // Optimized search for favorite routers with matching last byte
     // Check ordering optimized for IoT devices (cheapest checks first)

src/mesh/mesh-pb-constants.h

@@ -69,6 +69,22 @@ static inline int get_max_num_nodes()
 /// Max number of channels allowed
 #define MAX_NUM_CHANNELS (member_size(meshtastic_ChannelFile, channels) / member_size(meshtastic_ChannelFile, channels[0]))
 
+// Traffic Management module configuration
+// Enable per-variant by defining HAS_TRAFFIC_MANAGEMENT=1 in variant.h
+#ifndef HAS_TRAFFIC_MANAGEMENT
+#define HAS_TRAFFIC_MANAGEMENT 0
+#endif
+
+// Cache size for traffic management (number of nodes to track)
+// Can be overridden per-variant based on available memory
+#ifndef TRAFFIC_MANAGEMENT_CACHE_SIZE
+#if HAS_TRAFFIC_MANAGEMENT
+#define TRAFFIC_MANAGEMENT_CACHE_SIZE 1000
+#else
+#define TRAFFIC_MANAGEMENT_CACHE_SIZE 0
+#endif
+#endif
+
 /// helper function for encoding a record as a protobuf, any failures to encode are fatal and we will panic
 /// returns the encoded packet size
 size_t pb_encode_to_bytes(uint8_t *destbuf, size_t destbufsize, const pb_msgdesc_t *fields, const void *src_struct);
@@ -90,4 +106,4 @@ bool writecb(pb_ostream_t *stream, const uint8_t *buf, size_t count);
  */
 bool is_in_helper(uint32_t n, const uint32_t *array, pb_size_t count);
 
-#define is_in_repeated(name, n) is_in_helper(n, name, name##_count)
+#define is_in_repeated(name, n) is_in_helper(n, name, name##_count)