sds-library 0.0.1a6__tar.gz

sds_library-0.0.1a6/MANIFEST.in

@@ -0,0 +1,11 @@
+# Include C source files for building from source
+recursive-include ../src *.c
+recursive-include ../include *.h
+recursive-include ../platform *.c *.cpp
+
+# Include the CFFI definitions
+include sds/_cdefs.h
+
+# Include documentation
+include README.md
+include ../LICENSE

sds_library-0.0.1a6/PKG-INFO

@@ -0,0 +1,559 @@
+Metadata-Version: 2.4
+Name: sds-library
+Version: 0.0.1a6
+Summary: Python bindings for the SDS (Synchronized Data Structures) library
+Author: SDS Team
+License: MIT
+Project-URL: Homepage, https://github.com/EdgeKVM-Inc/sds-library
+Project-URL: Documentation, https://github.com/EdgeKVM-Inc/sds-library#readme
+Project-URL: Repository, https://github.com/EdgeKVM-Inc/sds-library
+Keywords: iot,mqtt,embedded,synchronization,dds
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Operating System :: MacOS :: MacOS X
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: System :: Networking
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: cffi>=1.0.0
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: mypy; extra == "dev"
+Requires-Dist: black; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+
+# SDS Library - Python Bindings
+
+Python bindings for the SDS (Synchronized Data Structures) library, providing
+a Pythonic interface for IoT state synchronization over MQTT.
+
+## Installation
+
+```bash
+pip install sds-library
+```
+
+For development installation:
+
+```bash
+cd python
+pip install -e ".[dev]"
+```
+
+## Quick Start
+
+### Generate Types from Schema
+
+The code generator creates Python types from your `.sds` schema file:
+
+```bash
+# Generate Python types (and optionally C types)
+python tools/sds_codegen.py schema.sds --python -o python/
+```
+
+This creates `sds_types.py` with dataclasses matching your C structs.
+
+### Device Node (Using Generated Types)
+
+```python
+from sds import SdsNode, Role
+from sds_types import SensorData  # Generated from schema.sds
+
+with SdsNode("py_sensor_01", "localhost") as node:
+    # Register with schema bundle - no manual dataclass needed!
+    table = node.register_table("SensorData", Role.DEVICE, schema=SensorData)
+    
+    @node.on_config("SensorData")
+    def handle_config(table_type):
+        print(f"Config: threshold={table.config.threshold}")
+    
+    while True:
+        # C-like attribute access
+        table.state.temperature = 23.5
+        table.state.humidity = 65.0
+        table.status.error_code = 0
+        
+        node.poll(timeout_ms=1000)
+```
+
+### Device Node (Manual Schema Definition)
+
+If you prefer not to use the code generator:
+
+```python
+from dataclasses import dataclass
+from sds import SdsNode, Role, Field
+
+@dataclass
+class SensorState:
+    temperature: float = Field(float32=True)
+    humidity: float = Field(float32=True)
+
+with SdsNode("py_sensor_01", "localhost") as node:
+    table = node.register_table(
+        "SensorData", Role.DEVICE,
+        state_schema=SensorState,
+    )
+    table.state.temperature = 23.5
+```
+
+### Owner Node (C-like Syntax)
+
+```python
+from sds import SdsNode, Role
+
+with SdsNode("py_owner_01", "localhost") as node:
+    table = node.register_table(
+        "SensorData",
+        Role.OWNER,
+        status_schema=SensorStatus,  # For reading device status
+    )
+    
+    # Set config
+    table.config.threshold = 25.0
+    
+    @node.on_status("SensorData")
+    def handle_status(table_type, from_node):
+        device = table.get_device(from_node)
+        if device and device.online:
+            print(f"{from_node}: battery={device.status.battery_percent}%")
+    
+    while True:
+        node.poll(timeout_ms=1000)
+        
+        # Iterate all devices
+        for node_id, device in table.iter_devices():
+            print(f"{node_id}: online={device.online}")
+```
+
+## Features
+
+- **C-like Syntax**: Access table data directly (`table.state.temperature = 23.5`)
+- **Zero Protocol Drift**: Python uses the exact same C implementation
+- **Pythonic API**: Context managers, decorators, and keyword arguments
+- **Thread-Safe**: All operations protected by locks for multi-threaded use
+- **Cross-Platform**: Linux (x86_64, ARM64) and macOS
+- **Type Hints**: Full type annotations for IDE support
+- **Device Eviction**: Automatic cleanup of offline device slots
+
+## Device Eviction
+
+When devices disconnect unexpectedly, SDS receives MQTT LWT (Last Will and Testament) 
+messages and can automatically evict them from status slots after a grace period.
+This prevents slots from being permanently consumed by devices that never reconnect.
+
+```python
+# Enable eviction with 60-second grace period
+with SdsNode("owner", "localhost", eviction_grace_ms=60000) as node:
+    table = node.register_table("SensorData", Role.OWNER, schema=SensorData)
+    
+    @node.on_device_evicted()
+    def handle_eviction(table_type: str, node_id: str):
+        print(f"Device {node_id} was evicted from {table_type}")
+    
+    while True:
+        node.poll(timeout_ms=1000)
+        
+        # Check if a device has eviction pending
+        device = table.get_device("sensor_01")
+        if device and device.eviction_pending:
+            print("Device will be evicted soon if it doesn't reconnect")
+```
+
+## Thread Safety
+
+SdsNode is fully thread-safe. Multiple threads can safely:
+- Call `poll()` concurrently
+- Access table data (`table.state.temperature`)
+- Register tables and callbacks
+
+```python
+import threading
+from sds import SdsNode, Role
+
+with SdsNode("my_node", "localhost") as node:
+    table = node.register_table("SensorData", Role.DEVICE, schema=SensorData)
+    
+    def polling_thread():
+        while True:
+            node.poll(timeout_ms=100)
+    
+    def update_thread():
+        while True:
+            table.state.temperature = read_sensor()
+            time.sleep(1)
+    
+    # Both threads can safely access the node and table
+    t1 = threading.Thread(target=polling_thread)
+    t2 = threading.Thread(target=update_thread)
+    t1.start()
+    t2.start()
+```
+
+**Note:** Callbacks are executed while holding the lock. Avoid blocking
+operations in callbacks to prevent deadlocks.
+
+## Logging Configuration
+
+SDS uses Python's standard `logging` module. By default, log messages go
+to a `NullHandler` (silent). To enable logging:
+
+```python
+from sds import configure_logging
+import logging
+
+# Quick setup - logs to stderr at INFO level
+configure_logging(level=logging.INFO)
+
+# Or configure manually
+logging.getLogger("sds").setLevel(logging.DEBUG)
+logging.getLogger("sds").addHandler(logging.StreamHandler())
+```
+
+## Connection Resilience
+
+SdsNode includes built-in connection retry with exponential backoff:
+
+```python
+from sds import SdsNode
+
+node = SdsNode(
+    "my_node",
+    "mqtt.example.com",
+    retry_count=5,           # Try 5 times (default: 3)
+    retry_delay_ms=2000,     # Start with 2 second delay (default: 1000)
+    connect_timeout_ms=10000 # 10 second timeout (default: 5000)
+)
+```
+
+On connection failure, retries are attempted with exponential backoff
+(delay doubles each retry). Non-connection errors are not retried.
+
+## Error Handling Best Practices
+
+```python
+from sds import (
+    SdsNode, SdsMqttError, SdsTableError, 
+    SdsValidationError, SdsError
+)
+
+try:
+    with SdsNode("my_node", "localhost") as node:
+        table = node.register_table("SensorData", Role.DEVICE)
+        
+        while True:
+            try:
+                node.poll()
+            except SdsMqttError:
+                # MQTT connection lost - will auto-reconnect
+                logging.warning("MQTT disconnected, waiting...")
+                time.sleep(1)
+                
+except SdsValidationError as e:
+    # Invalid node_id or configuration
+    logging.error(f"Configuration error: {e}")
+except SdsMqttError as e:
+    # Failed to connect after all retries
+    logging.error(f"Could not connect to MQTT: {e}")
+except SdsError as e:
+    # Other SDS errors
+    logging.error(f"SDS error: {e}")
+```
+
+## Requirements
+
+- Python 3.8+
+- MQTT broker (e.g., Mosquitto)
+- libpaho-mqtt development headers (for building from source)
+
+## Configuration Options
+
+### Delta Sync (v0.5.0+)
+
+Enable delta sync to only send changed fields, reducing bandwidth:
+
+```python
+with SdsNode(
+    "sensor_01", 
+    "localhost",
+    enable_delta_sync=True,           # Only send changed fields
+    delta_float_tolerance=0.01        # Ignore tiny float changes
+) as node:
+    # ...
+```
+
+**Benefits:**
+- Reduced bandwidth (only changed fields transmitted)
+- Lower power consumption on battery devices
+- Works automatically with codegen-generated tables
+
+**Limitations:**
+- Config messages are always full (retained on broker)
+- Status heartbeats are always full (liveness detection)
+- Manual schema definitions use full sync
+
+### Eviction Grace Period
+
+Configure how long to wait before evicting offline devices:
+
+```python
+with SdsNode(
+    "owner", 
+    "localhost",
+    eviction_grace_ms=30000  # 30 seconds before eviction
+) as node:
+    @node.on_device_evicted()
+    def handle_eviction(table_type, node_id):
+        print(f"Device {node_id} evicted from {table_type}")
+```
+
+### Raw MQTT Publish/Subscribe (v0.5.0+)
+
+Send and receive custom MQTT messages through the SDS-managed connection. Useful for
+logging, diagnostics, or application-specific messages that don't fit the table model.
+
+**Publishing:**
+```python
+with SdsNode("sensor_01", "localhost") as node:
+    # Check connection status
+    if node.is_connected():
+        # Publish a log message
+        node.publish_raw(
+            f"log/{node.node_id}",
+            '{"level": "info", "msg": "Sensor started"}',
+            qos=0,
+            retained=False
+        )
+        
+        # Publish binary data
+        node.publish_raw("sensor/raw_data", b'\x00\x01\x02\x03')
+```
+
+**Subscribing:**
+```python
+def on_log(topic: str, payload: bytes):
+    print(f"Log from {topic}: {payload.decode()}")
+
+with SdsNode("controller", "localhost") as node:
+    # Subscribe to all logs (+ matches any single level)
+    node.subscribe_raw("log/+", on_log)
+    
+    # Or use # for multi-level wildcard
+    node.subscribe_raw("sensors/#", on_sensor_data)
+    
+    # Main loop
+    while True:
+        node.poll()
+        time.sleep(0.1)
+    
+    # Unsubscribe when done
+    node.unsubscribe_raw("log/+")
+```
+
+**Methods:**
+
+| Method | Description |
+|--------|-------------|
+| `is_connected()` | Returns `True` if connected to MQTT broker |
+| `publish_raw(topic, payload, qos=0, retained=False)` | Publish arbitrary MQTT message |
+| `subscribe_raw(topic, callback, qos=0)` | Subscribe to topic pattern with callback |
+| `unsubscribe_raw(topic)` | Unsubscribe from a topic pattern |
+
+**Notes:**
+- Topics starting with `sds/` are reserved and will raise `ValueError`
+- `payload` for publish can be `str` (UTF-8 encoded) or `bytes`
+- Callback receives `(topic: str, payload: bytes)`
+- Maximum 8 concurrent raw subscriptions
+- Wildcard subscriptions (`log/+`) count as 1 subscription regardless of matching topics
+
+## API Reference
+
+### SdsNode
+
+The main class for interacting with SDS.
+
+```python
+class SdsNode:
+    def __init__(self, node_id: str, broker_host: str, port: int = 1883,
+                 username: str | None = None, password: str | None = None,
+                 eviction_grace_ms: int = 0):
+        """
+        Initialize SDS node and connect to MQTT broker.
+        
+        Args:
+            eviction_grace_ms: Grace period before evicting offline devices (0 = disabled).
+                              For OWNER roles, when a device disconnects (LWT), an eviction
+                              timer starts. If the device doesn't reconnect within this period,
+                              it's evicted from status slots, freeing the slot for new devices.
+        """
+    
+    def register_table(self, table_type: str, role: Role,
+                       sync_interval_ms: int | None = None,
+                       config_schema: Type | None = None,
+                       state_schema: Type | None = None,
+                       status_schema: Type | None = None) -> SdsTable:
+        """Register a table and return SdsTable for C-like access."""
+    
+    def get_table(self, table_type: str) -> SdsTable:
+        """Get a previously registered table."""
+    
+    def unregister_table(self, table_type: str) -> None:
+        """Unregister a table."""
+    
+    def poll(self, timeout_ms: int = 0) -> None:
+        """Process MQTT messages and sync changes."""
+    
+    def is_ready(self) -> bool:
+        """Check if connected to MQTT broker."""
+    
+    def on_error(self, callback) -> None:
+        """Register error callback."""
+    
+    def on_version_mismatch(self, callback) -> None:
+        """Register version mismatch callback."""
+```
+
+### SdsTable
+
+Provides C-like attribute access to table sections.
+
+```python
+class SdsTable:
+    table_type: str          # The table type name
+    role: Role               # OWNER or DEVICE
+    
+    # Device role properties
+    state: SectionProxy      # Read/write state section
+    status: SectionProxy     # Read/write status section
+    config: SectionProxy     # Read-only config (from owner)
+    
+    # Owner role properties
+    config: SectionProxy     # Read/write config section
+    device_count: int        # Number of known devices
+    
+    def get_device(self, node_id: str) -> DeviceView | None:
+        """Get a device's data (owner role only)."""
+    
+    def iter_devices(self) -> Iterator[tuple[str, DeviceView]]:
+        """Iterate over all known devices (owner role only)."""
+```
+
+### Role Enum
+
+```python
+class Role(Enum):
+    OWNER = 0   # Publishes config, receives state/status
+    DEVICE = 1  # Receives config, publishes state/status
+```
+
+### LogLevel Enum
+
+```python
+class LogLevel(Enum):
+    NONE = 0    # Disable all logging
+    ERROR = 1   # Error conditions only
+    WARN = 2    # Warnings and errors
+    INFO = 3    # Informational messages
+    DEBUG = 4   # All messages
+
+# Functions
+set_log_level(level: LogLevel) -> None
+get_log_level() -> LogLevel
+```
+
+### Field Helper
+
+Define field types for schema dataclasses:
+
+```python
+from sds import Field, FieldType
+
+@dataclass
+class MyConfig:
+    command: int = Field(uint8=True)
+    value: float = Field(float32=True)
+    name: str = Field(string_len=32)
+```
+
+### SectionProxy Convenience Methods
+
+```python
+# Convert section to dictionary
+data = table.state.to_dict()
+# {'temperature': 23.5, 'humidity': 65.0}
+
+# Set multiple fields from dictionary
+table.state.from_dict({'temperature': 24.0, 'humidity': 60.0})
+```
+
+### Exceptions
+
+```python
+class SdsError(Exception):
+    """Base exception for SDS errors."""
+    
+class SdsNotInitializedError(SdsError):
+    """Raised when SDS is not initialized."""
+    
+class SdsMqttError(SdsError):
+    """Raised on MQTT connection errors."""
+    
+class SdsTableError(SdsError):
+    """Raised on table registration errors."""
+
+class SdsValidationError(SdsError):
+    """Raised when input validation fails (e.g., invalid node_id)."""
+```
+
+## Building from Source
+
+### Prerequisites
+
+**Linux (Debian/Ubuntu):**
+```bash
+sudo apt-get install libpaho-mqtt-dev python3-dev
+```
+
+**Linux (RHEL/CentOS):**
+```bash
+sudo yum install paho-c-devel python3-devel
+```
+
+**macOS:**
+```bash
+brew install libpaho-mqtt
+```
+
+### Build
+
+```bash
+cd python
+pip install build
+python -m build
+```
+
+## Testing
+
+```bash
+cd python
+pip install -e ".[dev]"
+pytest
+```
+
+For integration tests (requires MQTT broker):
+```bash
+pytest -m requires_mqtt
+```
+
+## License
+
+MIT License - see LICENSE file for details.