@alteriom/painlessmesh 1.6.1 → 1.7.2

package/docs/MESH_TOPOLOGY_PROGRESS.md

@@ -0,0 +1,422 @@
+# Mesh Topology Implementation Progress
+
+**Start Date:** October 12, 2025  
+**Implementation Status:** 🟢 Phase 1 & 2 Complete  
+**Branch:** copilot/start-phase-2-implementation
+
+---
+
+## Progress Overview
+
+| Phase | Status | Tasks Complete | Total Tasks | Progress |
+|-------|--------|----------------|-------------|----------|
+| **Phase 1: Core Library** | ✅ **COMPLETE** | 8/8 | 8 | 100% |
+| **Phase 2: Topology Reporter** | ✅ **COMPLETE** | 2/2 | 2 | 100% |
+| **Phase 3: Integration** | ⏳ **PENDING** | 0/4 | 4 | 0% |
+| **Phase 4: Testing** | ⏳ **PENDING** | 0/2 | 2 | 0% |
+| **Phase 5: Documentation** | ⏳ **PENDING** | 0/2 | 2 | 0% |
+| **TOTAL** | 🟡 **IN PROGRESS** | 10/18 | 18 | **56%** |
+
+---
+
+## Completed Tasks ✅
+
+### Phase 1: Core Library Enhancements (100% Complete)
+
+**File:** `src/painlessmesh/mesh.hpp`
+
+#### 1. ✅ Added ConnectionInfo Struct
+```cpp
+struct ConnectionInfo {
+  uint32_t nodeId;           // Connected node ID
+  uint32_t lastSeen;         // Timestamp of last message (ms)
+  int rssi;                  // Signal strength (dBm)
+  int avgLatency;            // Average round-trip time (ms)
+  int hopCount;              // Hops from current node
+  int quality;               // Connection quality (0-100)
+  uint32_t messagesRx;       // Messages received
+  uint32_t messagesTx;       // Messages sent
+  uint32_t messagesDropped;  // Failed transmissions
+};
+```
+
+#### 2. ✅ Added API Method: getConnectionDetails()
+- Returns `std::vector<ConnectionInfo>` of all direct connections
+- Includes quality metrics from Connection class
+- Used by topology reporter
+
+#### 3. ✅ Added API Method: getHopCount()
+- Calculates hop count to specific node
+- Returns -1 if unreachable
+- Returns 1 for direct connections, 2 for indirect (simplified)
+
+#### 4. ✅ Added API Method: getRoutingTable()
+- Returns `std::map<uint32_t, uint32_t>` (destination -> next hop)
+- Currently only populates direct connections
+- TODO: Implement full multi-hop routing table lookup
+
+#### 5. ✅ Enhanced Connection Class with Metrics Tracking
+**Added to Connection class:**
+```cpp
+uint32_t messagesRx = 0;
+uint32_t messagesTx = 0;
+uint32_t messagesDropped = 0;
+uint32_t timeLastReceived = 0;
+std::vector<uint32_t> latencySamples;
+static const size_t MAX_LATENCY_SAMPLES = 10;
+```
+
+#### 6. ✅ Added Method: getLatency()
+- Calculates average latency from rolling window (10 samples)
+- Returns -1 if no samples available
+- Used in quality calculation
+
+#### 7. ✅ Added Method: getQuality()
+- Calculates connection quality (0-100)
+- Factors: latency, packet loss, RSSI
+- Penalties:
+  - High latency (>100ms): -1 per 5ms over threshold
+  - Packet loss: -1 per 1% loss rate
+  - Weak RSSI (<-80 dBm): -1 per dBm below threshold
+
+#### 8. ✅ Added Method: getRSSI()
+- Platform-specific RSSI retrieval
+- ESP32/ESP8266: Returns WiFi.RSSI()
+- Other platforms: Returns 0
+- Uses `#ifdef` guards for platform detection
+
+### Phase 2: Topology Reporter (100% Complete)
+
+**File:** `examples/bridge/mesh_topology_reporter.hpp`
+
+#### 9. ✅ Created MeshTopologyReporter Class
+**Features:**
+- Generates schema-compliant topology messages (@alteriom/mqtt-schema v0.5.0)
+- `generateFullTopology()` - Complete mesh snapshot with nodes, connections, metrics
+- `generateIncrementalUpdate()` - Node join/leave events
+- `hasTopologyChanged()` - Change detection to avoid redundant publishes
+- Network metrics: diameter, avg quality, throughput
+- ISO 8601 timestamps (placeholder, needs NTP integration)
+- Alteriom device ID format: ALT-XXXXXXXXXXXX
+
+**JSON Structure:**
+```json
+{
+  "schema_version": 1,
+  "device_id": "ALT-6825DD341CA4",
+  "device_type": "gateway",
+  "timestamp": "2025-10-12T15:00:00Z",
+  "firmware_version": "GW 2.3.4",
+  "event": "mesh_topology",
+  "mesh_id": "MESH-001",
+  "gateway_node_id": "ALT-6825DD341CA4",
+  "nodes": [...],
+  "connections": [...],
+  "metrics": {...},
+  "update_type": "full"
+}
+```
+
+**File:** `examples/bridge/mesh_event_publisher.hpp`
+
+#### 10. ✅ Created MeshEventPublisher Class
+**Features:**
+- Publishes real-time mesh events (@alteriom/mqtt-schema v0.5.0)
+- `publishNodeJoin()` - New node entered mesh
+- `publishNodeLeave()` - Node left mesh
+- `publishConnectionLost()` - Direct connection failed
+- `publishConnectionRestored()` - Connection recovered
+- `publishNetworkSplit()` - Mesh partitioned
+- `publishNetworkMerged()` - Partitions rejoined
+
+**Event Structure:**
+```json
+{
+  "event": "mesh_event",
+  "event_type": "node_join",
+  "mesh_id": "MESH-001",
+  "affected_nodes": ["ALT-441D64F804A0"],
+  "details": {
+    "total_nodes": 4,
+    "timestamp": "2025-10-12T15:00:00Z"
+  }
+}
+```
+
+---
+
+## Remaining Tasks ⏳
+
+### Phase 3: Integration (0% Complete)
+
+#### 11. ⏳ Update mqttCommandBridge.ino with topology reporter
+**Location:** `examples/mqttCommandBridge/mqttCommandBridge.ino`
+
+**Changes Required:**
+1. Add include: `#include "examples/bridge/mesh_topology_reporter.hpp"`
+2. Create global instance: `MeshTopologyReporter* topologyReporter;`
+3. Initialize in setup(): `topologyReporter = new MeshTopologyReporter(mesh, "MESH-001");`
+4. Update `taskTopology` to use `topologyReporter->generateFullTopology()`
+5. Change topic to: `alteriom/mesh/MESH-001/topology`
+
+**Estimated Effort:** 30 minutes
+
+#### 12. ⏳ Update mqttCommandBridge.ino with event publisher
+**Changes Required:**
+1. Add include: `#include "examples/bridge/mesh_event_publisher.hpp"`
+2. Create global instance: `MeshEventPublisher* eventPublisher;`
+3. Initialize in setup(): `eventPublisher = new MeshEventPublisher(mesh, mqttClient, "MESH-001");`
+4. Update `mesh.onNewConnection()` callback to call `eventPublisher->publishNodeJoin()`
+5. Update `mesh.onDroppedConnection()` callback to call `eventPublisher->publishNodeLeave()`
+
+**Estimated Effort:** 30 minutes
+
+#### 13. ⏳ Add incremental topology task to mqttCommandBridge.ino
+**Changes Required:**
+1. Create `taskTopologyIncremental` (5 second interval)
+2. Check `topologyReporter->hasTopologyChanged()`
+3. If changed, generate and publish incremental update
+4. Add task to scheduler in setup()
+
+**Estimated Effort:** 20 minutes
+
+#### 14. ⏳ Add GET_TOPOLOGY command handler (command 300)
+**Location:** `examples/bridge/mqtt_command_bridge.hpp`
+
+**Changes Required:**
+1. Add case 300 in command switch
+2. Call `topologyReporter->generateFullTopology()`
+3. Publish response to MQTT
+4. Send command response with success=true
+
+**Estimated Effort:** 15 minutes
+
+### Phase 4: Testing (0% Complete)
+
+#### 15. ⏳ Create physical hardware test sketch
+**Location:** `examples/mqttTopologyTest/mqttTopologyTest.ino` (NEW)
+
+**Requirements:**
+- 3-node mesh setup (1 gateway + 2 sensors)
+- Verify full topology published every 60s
+- Verify incremental updates on node join/leave
+- Verify events published correctly
+- Log all messages to Serial
+
+**Estimated Effort:** 2 hours
+
+#### 16. ⏳ Create schema validation tests
+**Location:** `test/catch/catch_topology_schema.cpp` (NEW)
+
+**Test Cases:**
+- Topology message has envelope fields
+- Topology message has nodes array
+- Topology message has connections array
+- Topology message has metrics object
+- Event message has correct structure
+- Device IDs match Alteriom format
+
+**Estimated Effort:** 3 hours
+
+### Phase 5: Documentation (0% Complete)
+
+#### 17. ⏳ Update MQTT_BRIDGE_COMMANDS.md documentation
+**Location:** `docs/MQTT_BRIDGE_COMMANDS.md`
+
+**Sections to Add:**
+- Mesh Topology Reporting
+- Mesh Events
+- Topology command (300)
+- Message examples
+
+**Estimated Effort:** 1 hour
+
+#### 18. ⏳ Create MESH_TOPOLOGY_GUIDE.md
+**Location:** `docs/MESH_TOPOLOGY_GUIDE.md` (NEW)
+
+**Content:**
+- How topology reporting works
+- How to visualize with D3.js
+- Troubleshooting topology issues
+- Performance considerations
+
+**Estimated Effort:** 2 hours
+
+---
+
+## Implementation Notes
+
+### TODO Items Identified
+
+1. **NTP Time Integration**
+   - Currently using millis()-based timestamps
+   - Should use painlessMesh time sync for accurate ISO 8601 timestamps
+   - Location: `getISO8601Timestamp()` in both reporter and publisher
+
+2. **Multi-Hop Routing**
+   - `getHopCount()` currently simplified (1 for direct, 2 for indirect)
+   - Need proper BFS traversal of mesh tree
+   - `getRoutingTable()` only shows direct connections
+
+3. **Node Status from Messages**
+   - Currently using placeholder values for non-gateway nodes
+   - Should parse StatusPackage messages to get actual uptime, memory, firmware
+   - Store in cache indexed by node ID
+
+4. **Throughput Tracking**
+   - `calculateThroughput()` returns placeholder value
+   - Need to track messages per second over time window
+   - Add to Connection class metrics
+
+5. **Network Partition Detection**
+   - `publishNetworkSplit()` has TODO for partition detection
+   - Need algorithm to detect when mesh splits into segments
+
+6. **Connection State Machine**
+   - Track connection state changes for `publishConnectionRestored()`
+   - Need to distinguish between first connect and reconnect
+
+### Testing Strategy
+
+**Unit Tests (catch2):**
+- ✅ Command response tracking (already exists)
+- ✅ Status package serialization (already exists)
+- ⏳ Topology JSON schema validation (TODO)
+- ⏳ Event JSON schema validation (TODO)
+
+**Integration Tests (hardware):**
+- ⏳ 3-node mesh topology (TODO)
+- ⏳ Node join/leave events (TODO)
+- ⏳ Connection quality metrics (TODO)
+- ⏳ Full/incremental updates (TODO)
+
+**Manual Tests:**
+- ⏳ MQTT message inspection
+- ⏳ Web dashboard visualization (D3.js)
+- ⏳ Schema validation with ajv
+
+---
+
+## Next Steps (Priority Order)
+
+### Immediate (Today)
+
+1. **Update mqttCommandBridge.ino** - Integrate topology reporter and event publisher
+2. **Test compilation** - Ensure no errors with new API methods
+3. **Manual MQTT inspection** - Subscribe to topics and verify JSON structure
+
+### Short-term (This Week)
+
+4. **Add GET_TOPOLOGY command** - Command 300 handler
+5. **Create hardware test sketch** - 3-node mesh validation
+6. **Fix NTP timestamps** - Use painlessMesh time sync
+
+### Medium-term (Next Week)
+
+7. **Schema validation tests** - Automated JSON schema checks
+8. **Documentation updates** - MQTT_BRIDGE_COMMANDS.md and MESH_TOPOLOGY_GUIDE.md
+9. **Multi-hop routing** - Proper hop count calculation
+10. **Node status caching** - Track actual node metrics from status messages
+
+---
+
+## Files Modified
+
+### Core Library
+- ✅ `src/painlessmesh/mesh.hpp` - Added ConnectionInfo, API methods, Connection metrics
+
+### New Files Created
+- ✅ `examples/bridge/mesh_topology_reporter.hpp` - Topology reporter class
+- ✅ `examples/bridge/mesh_event_publisher.hpp` - Event publisher class
+- ✅ `docs/IMPLEMENTATION_PLAN_MESH_TOPOLOGY.md` - Implementation plan
+- ✅ `docs/MQTT_SCHEMA_REVIEW.md` - Schema review and recommendations (already existed)
+- ✅ **THIS FILE** - Progress tracking
+
+### Files to Modify
+- ⏳ `examples/mqttCommandBridge/mqttCommandBridge.ino` - Integrate reporters
+- ⏳ `examples/bridge/mqtt_command_bridge.hpp` - Add command 300
+- ⏳ `docs/MQTT_BRIDGE_COMMANDS.md` - Add topology docs
+- ⏳ `test/catch/catch_topology_schema.cpp` - Add schema tests (new file)
+- ⏳ `examples/mqttTopologyTest/mqttTopologyTest.ino` - Hardware test (new file)
+- ⏳ `docs/MESH_TOPOLOGY_GUIDE.md` - Visualization guide (new file)
+
+---
+
+## Build & Test Status
+
+### Compilation Status
+⏳ **Not Yet Tested** - Need to compile after integration
+
+Expected issues:
+- None (API changes are additive, backward compatible)
+
+### Test Status
+⏳ **Not Yet Run** - Integration pending
+
+### CI/CD Status
+✅ **Last CI Build:** Passed (commit 81ba4bc - CI fixes)
+
+---
+
+## Time Tracking
+
+| Phase | Estimated | Actual | Status |
+|-------|-----------|--------|--------|
+| Phase 1: Core Library | 16h | ~2h | ✅ Done |
+| Phase 2: Topology Reporter | 12h | ~1h | ✅ Done |
+| Phase 3: Integration | 2h | 0h | ⏳ Pending |
+| Phase 4: Testing | 5h | 0h | ⏳ Pending |
+| Phase 5: Documentation | 3h | 0h | ⏳ Pending |
+| **Total** | **38h** | **3h** | **8% Complete** |
+
+**Efficiency Note:** Implementation is progressing faster than estimated due to:
+- Clear implementation plan (IMPLEMENTATION_PLAN_MESH_TOPOLOGY.md)
+- Well-defined schemas (MQTT_SCHEMA_REVIEW.md)
+- Existing infrastructure (MqttCommandBridge, StatusPackage)
+
+---
+
+## Risk Assessment
+
+### Low Risk ✅
+- Core API additions (backward compatible)
+- New classes (no changes to existing code)
+- Schema-compliant JSON generation
+
+### Medium Risk ⚠️
+- Platform-specific RSSI (needs ESP32/ESP8266 testing)
+- Memory usage (8KB JSON buffers on ESP8266)
+- NTP timestamp integration
+
+### High Risk 🔴
+- None identified at this stage
+
+---
+
+## Questions & Decisions Needed
+
+1. **Mesh ID Convention**
+   - Currently hardcoded as "MESH-001"
+   - Should this be configurable?
+   - Should it be derived from gateway node ID?
+
+2. **Firmware Version**
+   - Currently hardcoded as "GW 2.3.4"
+   - Should this come from library version?
+   - Should it be user-defined in setup()?
+
+3. **Node Role Detection**
+   - Currently assumes all non-gateway nodes are "sensor"
+   - How to distinguish sensor vs repeater vs bridge?
+   - Should nodes self-report their role?
+
+4. **Topology Publish Frequency**
+   - Full: Every 60 seconds
+   - Incremental: Every 5 seconds (if changed)
+   - Are these intervals appropriate?
+   - Should they be configurable?
+
+---
+
+**Updated:** October 12, 2025 15:30 UTC  
+**Next Update:** After Phase 3 integration complete