@alteriom/painlessmesh 1.8.14 → 1.9.0

package/docs/archive/PHASE2_IMPLEMENTATION.md

@@ -1,567 +0,0 @@
-# Phase 2 Implementation Details
-
-## Overview
-
-This document provides technical details on the Phase 2 implementation of broadcast OTA and MQTT status bridge features for painlessMesh.
-
-**Phase 2 Features:**
-1. **Broadcast OTA (Option 1A)** - True mesh-wide firmware distribution
-2. **MQTT Status Bridge (Option 2E)** - Professional monitoring integration
-
----
-
-## Feature 1: Broadcast OTA Implementation
-
-### Architecture
-
-The broadcast OTA feature extends the existing OTA plugin to support true mesh-wide broadcast distribution. Instead of each node requesting chunks individually (unicast), the root node broadcasts chunks once to all nodes simultaneously.
-
-**Message Flow:**
-```
-Unicast Mode (Phase 1):
-Root → Node1: Announce
-Node1 → Root: DataRequest(chunk 0)
-Root → Node1: Data(chunk 0)
-Node1 → Root: DataRequest(chunk 1)
-... repeated for each node and each chunk
-
-Broadcast Mode (Phase 2):
-Root → All: Broadcast Announce
-Root → All: Broadcast Data(chunk 0)
-Root → All: Broadcast Data(chunk 1)
-... all nodes receive simultaneously
-```
-
-### Code Changes
-
-#### 1. Data Routing Enhancement
-
-**File:** `src/painlessmesh/ota.hpp`
-
-**Change:** Modified `Data::replyTo()` to automatically set broadcast routing based on the `broadcasted` flag.
-
-```cpp
-static Data replyTo(const DataRequest& req, TSTRING data, size_t partNo) {
-  Data d;
-  // ... existing field initialization ...
-  
-  // Phase 2: Set routing to BROADCAST for true broadcast mode
-  if (req.broadcasted) {
-    d.routing = router::BROADCAST;
-  }
-  return d;
-}
-```
-
-**Rationale:**
-- The `Data` class inherits from `DataRequest`, which sets routing to `router::SINGLE` by default
-- When `broadcasted=true`, we override routing to `router::BROADCAST`
-- This ensures data chunks are broadcast to all nodes instead of unicast to requester
-- Backward compatible: defaults to SINGLE routing when `broadcasted=false`
-
-#### 2. Sender Callback Enhancement
-
-**File:** `src/painlessmesh/ota.hpp`
-
-**Change:** Updated sender callback to log broadcast operations.
-
-```cpp
-mesh.sendPackage(&reply);
-if (pkg.broadcasted) {
-  Log(DEBUG, "OTA: Broadcasting chunk %d/%d\n", pkg.partNo, pkg.noPart);
-}
-```
-
-**Rationale:**
-- Routing is now handled automatically by `Data::replyTo()`
-- Added debug logging for broadcast mode visibility
-- Single code path for both unicast and broadcast modes
-
-### How It Works
-
-#### Sender Side (Root Node)
-
-1. **Announce Phase:**
-   - Root node calls `mesh.offerOTA(..., broadcasted=true)`
-   - Creates periodic task to broadcast `Announce` message
-   - Announce includes `broadcasted=true` flag
-
-2. **Data Distribution Phase:**
-   - Root node receives `DataRequest` from any node (typically root itself)
-   - Loads firmware chunk from storage via callback
-   - Creates `Data` message with chunk content
-   - `Data::replyTo()` automatically sets routing to BROADCAST
-   - Broadcasts chunk to all nodes simultaneously
-
-3. **Completion:**
-   - All nodes receive all chunks
-   - No individual acknowledgments required
-   - Root continues until all chunks sent
-
-#### Receiver Side (All Nodes)
-
-1. **Announce Reception:**
-   - Receives broadcast `Announce` message
-   - Checks if firmware matches role/hardware
-   - Checks if MD5 is different from current firmware
-   - If `broadcasted=true` and node is root: starts requesting chunks
-   - If `broadcasted=true` and node is not root: listens passively
-
-2. **Data Reception:**
-   - Receives broadcast `Data` chunks
-   - Assembles chunks in sequence
-   - Handles out-of-order delivery automatically
-   - Writes to flash progressively
-   - Reboots when all chunks received
-
-3. **Out-of-Sequence Handling:**
-   - If node misses chunks or receives out of order
-   - Existing code falls back to unicast mode
-   - Requests missing chunks directly from root
-   - Maintains reliability despite broadcast limitations
-
-### Performance Analysis
-
-**Network Traffic Comparison:**
-
-For a mesh with N nodes and F firmware chunks:
-
-| Mode | Transmissions | Example (50 nodes, 150 chunks) |
-|------|--------------|--------------------------------|
-| Unicast | N × F | 50 × 150 = 7,500 |
-| Broadcast | F | 150 |
-| Reduction | (N-1) / N × 100% | 98% |
-
-**Memory Usage:**
-- Per node: +2-5KB for chunk bitmap and assembly buffer
-- Root node: No additional memory (reuses existing OTA buffers)
-
-**Update Time:**
-- Unicast: O(N × F) - Sequential per node
-- Broadcast: O(F) - Parallel to all nodes
-- Speedup: ~N times faster for large meshes
-
-### Testing
-
-**Test Coverage:**
-- Existing OTA tests continue to pass
-- Backward compatibility verified (unicast mode still works)
-- No new test failures introduced
-
-**Manual Testing Checklist:**
-- [ ] Broadcast OTA to 2-5 node test mesh
-- [ ] Broadcast OTA to 10+ node mesh
-- [ ] Mixed mode: Some nodes broadcast, some unicast
-- [ ] Out-of-sequence chunk handling
-- [ ] Network congestion handling
-- [ ] Failure recovery (node reboot during OTA)
-
----
-
-## Feature 2: MQTT Status Bridge Implementation
-
-### Architecture
-
-The MQTT Status Bridge is a helper class that collects mesh status and publishes it to MQTT topics at configurable intervals.
-
-**Design Pattern:**
-- Composition pattern: Bridge wraps mesh and MQTT client
-- Periodic task pattern: Uses mesh scheduler for timed publishing
-- Observer pattern: Reacts to mesh state changes
-
-**Component Diagram:**
-```
-┌─────────────────────┐
-│  painlessMesh       │
-│  - Node list        │
-│  - Topology         │
-│  - Metrics          │
-└──────┬──────────────┘
-       │
-       ↓ (reads)
-┌──────────────────────┐
-│ MqttStatusBridge     │
-│ - Collect status     │
-│ - Format JSON        │
-│ - Schedule publish   │
-└──────┬───────────────┘
-       │
-       ↓ (publishes)
-┌──────────────────────┐
-│  MQTT Broker         │
-│  - mesh/status/*     │
-└──────────────────────┘
-```
-
-### Code Implementation
-
-#### Class Structure
-
-**File:** `examples/bridge/mqtt_status_bridge.hpp`
-
-```cpp
-class MqttStatusBridge {
-private:
-  painlessMesh& mesh;
-  PubSubClient& mqttClient;
-  uint32_t publishInterval;
-  bool enableTopologyPublish;
-  bool enableMetricsPublish;
-  bool enableAlertsPublish;
-  bool enablePerNodePublish;
-  String topicPrefix;
-  Task* publishTask;
-  
-public:
-  MqttStatusBridge(painlessMesh& mesh, PubSubClient& mqttClient);
-  
-  // Configuration
-  void setPublishInterval(uint32_t interval);
-  void setTopicPrefix(const String& prefix);
-  void enableTopology(bool enable);
-  void enableMetrics(bool enable);
-  void enableAlerts(bool enable);
-  void enablePerNode(bool enable);
-  
-  // Control
-  void begin();
-  void stop();
-  void publishNow();
-  
-private:
-  void publishStatus();
-  void publishNodeList();
-  void publishTopology();
-  void publishMetrics();
-  void publishAlerts();
-  void publishPerNodeStatus();
-};
-```
-
-#### Key Methods
-
-**1. begin() - Start Publishing**
-```cpp
-void begin() {
-  publishTask = &mesh.addTask(
-    TASK_MILLISECOND * publishInterval,
-    TASK_FOREVER,
-    [this]() { this->publishStatus(); }
-  );
-  publishTask->enable();
-}
-```
-
-**2. publishStatus() - Main Publishing Logic**
-```cpp
-void publishStatus() {
-  if (!mqttClient.connected()) return;
-  
-  publishNodeList();
-  if (enableTopologyPublish) publishTopology();
-  if (enableMetricsPublish) publishMetrics();
-  if (enableAlertsPublish) publishAlerts();
-  if (enablePerNodePublish) publishPerNodeStatus();
-}
-```
-
-**3. publishNodeList() - Node List JSON**
-```cpp
-void publishNodeList() {
-  auto nodes = mesh.getNodeList(true);
-  
-  String payload = "{\"nodes\":[";
-  for (size_t i = 0; i < nodes.size(); i++) {
-    if (i > 0) payload += ",";
-    payload += String(nodes[i]);
-  }
-  payload += "],\"count\":";
-  payload += String(nodes.size());
-  payload += ",\"timestamp\":";
-  payload += String(millis());
-  payload += "}";
-  
-  mqttClient.publish((topicPrefix + "nodes").c_str(), payload.c_str());
-}
-```
-
-**4. publishTopology() - Mesh Structure**
-```cpp
-void publishTopology() {
-  String topology = mesh.subConnectionJson(false);
-  mqttClient.publish((topicPrefix + "topology").c_str(), topology.c_str());
-}
-```
-
-**5. publishMetrics() - Performance Stats**
-```cpp
-void publishMetrics() {
-  String payload = "{";
-  payload += "\"nodeCount\":" + String(mesh.getNodeList(true).size());
-  payload += ",\"rootNodeId\":" + String(mesh.getNodeId());
-  payload += ",\"uptime\":" + String(millis() / 1000);
-  payload += ",\"freeHeap\":" + String(ESP.getFreeHeap());
-  payload += ",\"timestamp\":" + String(millis());
-  payload += "}";
-  
-  mqttClient.publish((topicPrefix + "metrics").c_str(), payload.c_str());
-}
-```
-
-**6. publishAlerts() - Active Alerts**
-```cpp
-void publishAlerts() {
-  String payload = "{\"alerts\":[";
-  bool hasAlerts = false;
-  
-  // Example: Low memory alert
-  if (ESP.getFreeHeap() < 10000) {
-    payload += "{\"type\":\"LOW_MEMORY\",\"severity\":\"critical\"}";
-    hasAlerts = true;
-  }
-  
-  payload += "],\"timestamp\":" + String(millis()) + "}";
-  mqttClient.publish((topicPrefix + "alerts").c_str(), payload.c_str());
-}
-```
-
-### MQTT Topic Schema
-
-#### 1. mesh/status/nodes
-```json
-{
-  "nodes": [123456, 789012, 345678],
-  "count": 3,
-  "timestamp": 1234567890
-}
-```
-
-#### 2. mesh/status/topology
-```json
-{
-  "nodeId": 123456,
-  "subs": [
-    {"nodeId": 789012, "subs": []},
-    {"nodeId": 345678, "subs": []}
-  ]
-}
-```
-
-#### 3. mesh/status/metrics
-```json
-{
-  "nodeCount": 3,
-  "rootNodeId": 123456,
-  "uptime": 3600,
-  "freeHeap": 45000,
-  "freeHeapKB": 43,
-  "timestamp": 1234567890
-}
-```
-
-#### 4. mesh/status/alerts
-```json
-{
-  "alerts": [
-    {
-      "type": "LOW_MEMORY",
-      "severity": "critical",
-      "message": "Free heap below 10KB"
-    }
-  ],
-  "timestamp": 1234567890
-}
-```
-
-#### 5. mesh/status/node/{nodeId}
-```json
-{
-  "nodeId": 123456,
-  "connected": true,
-  "freeHeap": 45000,
-  "timestamp": 1234567890
-}
-```
-
-### Performance Considerations
-
-**Memory Usage:**
-- Bridge object: ~200 bytes
-- JSON formatting buffers: ~2-5KB temporary
-- Total overhead: +5-8KB on root node
-
-**MQTT Traffic:**
-| Feature | Size/Publish | Recommended Interval |
-|---------|-------------|---------------------|
-| Node List | ~200 bytes | 30-60s |
-| Topology | 1-5KB | 60-120s |
-| Metrics | ~300 bytes | 30-60s |
-| Alerts | ~400 bytes | 30-60s |
-| Per-node (50 nodes) | ~7.5KB | 120-300s |
-
-**Scalability:**
-- Small mesh (1-10 nodes): All features enabled, 30s interval
-- Medium mesh (10-50 nodes): Disable per-node, 60s interval
-- Large mesh (50+ nodes): Metrics/alerts only, 120s interval
-
-### Integration Points
-
-**Grafana:**
-- Use MQTT datasource plugin
-- Query topics for time-series data
-- Create dashboards for node count, memory, topology
-
-**InfluxDB:**
-- Use Telegraf MQTT consumer
-- Parse JSON payloads
-- Store time-series data
-
-**Prometheus:**
-- Use MQTT exporter
-- Convert MQTT messages to Prometheus metrics
-- Scrape metrics endpoint
-
-**Home Assistant:**
-- Use MQTT sensor integration
-- Create sensors for each metric
-- Build automations based on alerts
-
----
-
-## Backward Compatibility
-
-### Breaking Changes
-**None.** Phase 2 is fully backward compatible.
-
-### Compatibility Matrix
-
-| Feature | Phase 1 | Phase 2 | Compatible? |
-|---------|---------|---------|-------------|
-| Unicast OTA | ✅ | ✅ | ✅ Yes |
-| Compressed OTA | ✅ | ✅ | ✅ Yes |
-| Broadcast OTA | ❌ | ✅ | ✅ Yes (optional) |
-| Enhanced Status | ✅ | ✅ | ✅ Yes |
-| MQTT Bridge | ❌ | ✅ | ✅ Yes (optional) |
-
-### Migration Path
-
-**No code changes required** to maintain Phase 1 behavior:
-```cpp
-// This continues to work exactly as in Phase 1
-mesh.offerOTA(role, hardware, md5, parts, false, false, true);
-```
-
-**Opt-in to Phase 2 features:**
-```cpp
-// Enable broadcast by adding one parameter
-mesh.offerOTA(role, hardware, md5, parts, false, true, true);
-//                                                    ^^^^
-
-// Enable MQTT monitoring by including header
-#include "examples/bridge/mqtt_status_bridge.hpp"
-MqttStatusBridge bridge(mesh, mqttClient);
-bridge.begin();
-```
-
----
-
-## Testing Strategy
-
-### Unit Tests
-- [x] Existing tests continue to pass (80 assertions)
-- [ ] TODO: Add specific broadcast OTA tests
-- [ ] TODO: Add MQTT bridge unit tests
-
-### Integration Tests
-- [ ] Test broadcast OTA with 2 nodes
-- [ ] Test broadcast OTA with 10+ nodes
-- [ ] Test MQTT publishing to real broker
-- [ ] Test MQTT with Grafana integration
-- [ ] Test mixed mode (broadcast + unicast nodes)
-
-### Performance Tests
-- [ ] Measure network traffic reduction
-- [ ] Measure update time improvement
-- [ ] Measure memory usage
-- [ ] Measure MQTT traffic volume
-- [ ] Test with 50+ node mesh
-
----
-
-## Known Limitations
-
-### Broadcast OTA
-
-1. **No per-node targeting:** All nodes receive all chunks. Cannot target specific nodes.
-   - **Workaround:** Use role/hardware filtering in Announce
-   
-2. **Network reliability:** Broadcast packets may be dropped in congested networks.
-   - **Mitigation:** Out-of-sequence handler falls back to unicast
-
-3. **Memory overhead:** Each node needs buffer for chunk assembly (+2-5KB)
-   - **Impact:** May be significant for ESP8266 with limited RAM
-
-### MQTT Status Bridge
-
-1. **Single point of failure:** Bridge node must remain online
-   - **Mitigation:** Use reliable root node hardware
-
-2. **External network required:** Needs WiFi connection to MQTT broker
-   - **Impact:** Not suitable for pure mesh-only deployments
-
-3. **MQTT broker dependency:** Requires external MQTT infrastructure
-   - **Mitigation:** Use public MQTT brokers for testing
-
----
-
-## Future Enhancements
-
-### Phase 3 Candidates
-
-1. **Progressive Rollout OTA (Option 1B)**
-   - Phased firmware distribution
-   - Health monitoring between phases
-   - Automatic rollback on failures
-
-2. **Real-time Telemetry Streams (Option 2C)**
-   - Continuous metrics streaming
-   - Anomaly detection
-   - Predictive alerting
-
-3. **Advanced Alert System**
-   - Custom alert rules
-   - Alert escalation
-   - Integration with notification services
-
-### Potential Improvements
-
-1. **Chunk Bitmap Tracking**
-   - Track received chunks explicitly
-   - Request specific missing chunks
-   - Improve reliability in lossy networks
-
-2. **Rate Limiting**
-   - Adaptive broadcast rate based on congestion
-   - Prevent mesh saturation
-   - QoS-aware transmission
-
-3. **MQTT Bridge Enhancements**
-   - Bidirectional command handling
-   - OTA trigger via MQTT
-   - Remote configuration updates
-
----
-
-## References
-
-- [PHASE2_GUIDE.md](../PHASE2_GUIDE.md) - User documentation
-- [FEATURE_PROPOSALS.md](FEATURE_PROPOSALS.md) - Original feature proposals
-- [phase2_features.ino](../../examples/alteriom/phase2_features.ino) - Example code
-- [mqtt_status_bridge.hpp](../../examples/bridge/mqtt_status_bridge.hpp) - Bridge implementation
-
----
-
-**Document Version:** 1.0  
-**Last Updated:** December 2024  
-**Authors:** Alteriom Development Team  
-**Status:** ✅ Implementation Complete