reliomq 0.1.0__tar.gz

reliomq-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 pitpiboon
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

reliomq-0.1.0/PKG-INFO

@@ -0,0 +1,438 @@
+Metadata-Version: 2.4
+Name: reliomq
+Version: 0.1.0
+Summary: Durable, application-acknowledged MQTT publishing and forwarding
+License: MIT
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: paho-mqtt<3.0,>=2.0
+Dynamic: license-file
+
+# reliomq
+
+**Durable, end-to-end confirmed MQTT delivery for Python.**
+
+`reliomq` sits on top of [`paho-mqtt`](https://pypi.org/project/paho-mqtt/)
+and adds the part QoS 1 doesn't give you: a message survives a crash, an
+outage, or a broker restart, and is retried automatically — in order, under
+its original ID — until an application confirms it actually arrived.
+
+```python
+from reliomq import ReliabilityConfig, ReliablePublisher
+
+publisher = ReliablePublisher(ReliabilityConfig(
+    host="localhost",
+    queue_path="pending.jsonl",
+    data_topic="reliable/ingress",
+    ack_topic="reliable/acks",
+))
+publisher.start()
+publisher.publish(topic="factory/machine1/data", payload={"temperature": 25.2})
+publisher.stop()
+```
+
+That call durably queues the message before it ever touches the network.
+Everything else — reconnects, retries, ordering, and knowing when it's
+actually safe to forget the message — is handled for you.
+
+Requires **Python 3.11+** and **Paho MQTT 2.x**.
+
+## Why not just `qos=1`?
+
+QoS 1 only proves the broker your process is directly connected to accepted
+one publish. It proves nothing about:
+
+- whether your process crashes or loses power before that publish happens;
+- whether the broker forwards it any further (a bridge, another hop);
+- whether anything ever confirms, at the application level, that the message
+  did its job.
+
+`reliomq` closes that gap with a durable outbox plus an **application-level
+ACK**, so "the network layer said OK" is never mistaken for "the message is
+handled."
+
+## Features
+
+- **Durable-before-network writes** — every `publish()` is `fsync`'d to disk
+  before the first network attempt, so a crash immediately after `publish()`
+  returns still can't lose the message.
+- **QoS 1 MQTT delivery**, always — reliability isn't opt-in per call.
+- **Application-level end-to-end ACK**, correlated by a stable `event_id`,
+  in addition to the MQTT PUBACK.
+- **Automatic, unique event IDs** — generated for you, or you can supply
+  your own; never regenerated on retry.
+- **Strict FIFO recovery** — the oldest pending message is always retried
+  first; a new live message can never overtake it.
+- **Automatic restart recovery** — a fresh process just opens the same
+  queue file and continues exactly where the last one left off.
+- **Automatic retry** on broker outage, network failure, publish errors,
+  publish-confirmation timeout, and ACK timeout — nothing is deleted on
+  any of these.
+- **Optional bridge component** (`ReliableMqttBridge`) that relays between
+  two brokers and only ACKs the source *after* the destination publish is
+  confirmed — never before.
+- **Fail-closed forwarding** — any bridge failure (offline, malformed
+  message, full queue, timeout, shutdown) sends no ACK, so the source keeps
+  retrying instead of silently dropping the message.
+- **Safe ACK handling** — stale, late, duplicate, wrong-ID, and malformed
+  ACKs are all detected and ignored rather than treated as success.
+- **Automatic reconnect/backoff** via Paho, with explicit connection-state
+  tracking so the library never publishes while it knows it's disconnected.
+- **Clean shutdown** — stopping interrupts an in-progress ACK wait without
+  ever deleting the durable record for the message being waited on.
+- **Corruption-safe storage** — a damaged queue line is logged and skipped,
+  never silently deleted; a crash mid-write can't destroy the next record.
+- **Pluggable client construction** — inject your own `client_factory` for
+  TLS, auth, or any other Paho client customization.
+- **Strict JSON envelope validation** — rejects `NaN`/`Infinity`, bytes,
+  tuples, non-string keys, and other values with no exact JSON form.
+- **Thread-safe by design** — internal locks are never held during a
+  network wait, so an ACK can never be missed to a race.
+
+## Reliability guarantee
+
+`reliomq` provides **at-least-once delivery to the destination broker**
+when the publisher and bridge are used together. It favors *never silently
+losing a message* over *never duplicating one*.
+
+```text
+application
+    |  fsync durable outbox
+    v
+ReliablePublisher -- QoS 1 --> source broker
+                                   |
+                                   v
+                            ReliableMqttBridge
+                                   |
+                                   | QoS 1 + PUBACK
+                                   v
+                           destination broker
+                                   |
+                                   | correlated application ACK
+                                   v
+                         publisher removes outbox head
+```
+
+A record is deleted from the durable outbox **only** after a valid ACK
+carrying its exact `event_id` arrives back on the configured ACK topic.
+Broker acceptance alone is not proof that a final subscriber processed the
+message — if you need that stronger boundary, use a persistent MQTT
+subscription or extend the protocol with a consumer ACK of your own.
+
+Duplicates remain possible: the destination publish can succeed and the
+source ACK can then be lost, in which case the publisher correctly retries
+the same stable `event_id`. **Consumers should store processed event IDs
+and make handling idempotent** — see `examples/consumer_dedup.py`.
+
+## Install
+
+```bash
+# From GitHub, pinned to a release (recommended)
+pip install "git+https://github.com/pitpib23/reliomq.git@v0.1.0"
+
+# Or track the latest commit on main
+pip install "git+https://github.com/pitpib23/reliomq.git"
+
+# Or from a local clone
+pip install -e /path/to/reliomq
+```
+
+The only runtime dependency is `paho-mqtt>=2,<3`. There is no GPIO or other
+hardware dependency of any kind.
+
+## Publisher integration
+
+```python
+from reliomq import ReliabilityConfig, ReliablePublisher
+
+config = ReliabilityConfig(
+    host="localhost",
+    port=1883,
+    queue_path="mqtt_pending.jsonl",
+    data_topic="reliable/ingress",
+    ack_topic="reliable/acks",
+    ack_timeout=3.0,
+    publish_timeout=2.0,
+    retry_interval=10.0,
+)
+
+publisher = ReliablePublisher(config)
+publisher.start()
+
+event_id = publisher.publish(
+    topic="factory/machine1/data",
+    payload={"temperature": 25.2, "pressure": 4.1},
+)
+
+# Optional: block application code until this process observes delivery.
+delivered = publisher.wait_for_delivery(event_id, timeout=10.0)
+
+publisher.stop()
+```
+
+`ReliablePublisher` also works as a context manager: `with
+ReliablePublisher(config) as publisher: ...` calls `start()`/`stop()` for
+you.
+
+`payload` may be any strict JSON value: an object with string keys, an
+array, a string, a finite number, a boolean, or `null`. `NaN`, infinity,
+bytes, tuples, custom objects, and mappings with non-string keys are
+rejected. An ID is generated automatically unless `event_id=` is supplied.
+Explicit IDs must be globally unique and must not be reused for different
+content.
+
+`publish()` means "durably accepted," not "already delivered." If the
+outbox cannot be written safely, it raises a `StoreError` instead of
+pretending to have accepted the message.
+
+### Replacing a normal Paho publish
+
+Before:
+
+```python
+client.publish("factory/machine1/data", payload=json_text, qos=1)
+```
+
+After:
+
+```python
+event_id = publisher.publish(
+    topic="factory/machine1/data",
+    payload={"temperature": 25.2},
+)
+```
+
+The caller no longer owns reconnect loops, ACK races, durable retry, or FIFO
+recovery.
+
+## Bridge integration
+
+Use `ReliableMqttBridge` only if you need to relay messages from one broker
+to another. Run it as its own service or process; its source `data_topic`
+and `ack_topic` must match the publisher's configuration.
+
+```python
+from reliomq import BridgeConfig, ReliableMqttBridge
+
+bridge = ReliableMqttBridge(
+    BridgeConfig(
+        source_host="localhost",
+        source_port=1883,
+        destination_host="mqtt.example.net",
+        destination_port=1883,
+        data_topic="reliable/ingress",
+        ack_topic="reliable/acks",
+        destination_publish_timeout=2.0,
+        source_ack_publish_timeout=0.5,
+    )
+)
+
+bridge.start()
+...
+bridge.stop()
+```
+
+The bridge forwards to the destination topic stored in each message. The
+destination payload retains the deduplication key:
+
+```json
+{
+  "version": 1,
+  "event_id": "47913ac65ac84213a9361b393b845708",
+  "payload": {"temperature": 25.2}
+}
+```
+
+The bridge's handoff queue is intentionally memory-only. A malformed
+message, full queue, outage, publish error, timeout, or shutdown produces no
+success ACK; the publisher still owns the durable record and retries it.
+Deploy only one ordinary bridge subscriber per route unless duplicate
+forwarding is intended.
+
+## Wire protocol
+
+Publisher to bridge, on `data_topic` with QoS 1 and `retain=False`:
+
+```json
+{
+  "version": 1,
+  "event_id": "47913ac65ac84213a9361b393b845708",
+  "topic": "factory/machine1/data",
+  "payload": {"temperature": 25.2}
+}
+```
+
+Bridge to publisher, on `ack_topic` with QoS 1 and `retain=False`:
+
+```json
+{"version": 1, "event_id": "47913ac65ac84213a9361b393b845708"}
+```
+
+Protocol objects are strict and versioned. Unknown/missing fields, malformed
+UTF-8/JSON, invalid IDs, and ACKs for any ID other than the one currently in
+flight are ignored. Correlation uses the ID alone rather than payload
+equality, so it works with any payload shape.
+
+## Persistence and FIFO recovery
+
+The outbox is a lightweight JSONL file suitable for edge devices:
+
+- one complete message per line;
+- append, flush, and file `fsync` before `publish()` returns;
+- one in-process lock around reads and writes;
+- same-directory temporary file, `fsync`, and atomic replace when removing
+  the confirmed head;
+- best-effort parent-directory `fsync` on platforms that support it;
+- stable event ID and destination topic across restart and every retry.
+
+Corrupt physical lines are logged, preserved byte-for-byte, and skipped when
+finding the FIFO order among valid messages — never silently deleted. I/O
+errors are raised instead of being mistaken for an empty queue. If a crash
+leaves a torn final line, the next append begins on a new line so it does
+not destroy the next valid record.
+
+One process must own a queue path. The store is thread-safe but does not
+attempt cross-process file locking. JSONL removal rewrites the file, so a
+database-backed store may be more appropriate for extremely large queues or
+sustained high write rates.
+
+## Connection, retry, and shutdown behavior
+
+Both components use Paho's asynchronous network loop and reconnect backoff.
+The publisher does not send until the broker connection and ACK subscription
+are ready. A disconnect interrupts the current ACK wait without removing its
+record. Reconnect wakes recovery immediately; other failures retry after
+`retry_interval`.
+
+`ReliablePublisher.stop()` interrupts an ACK wait and joins the worker.
+Because the in-flight message was already durable, it remains for the next
+process. `ReliableMqttBridge.stop()` stops accepting new input, lets the
+bounded in-flight publish/ACK sequence finish, and leaves queued items
+unacknowledged so their publishers recover them. Start/stop are idempotent;
+a stopped instance is not restartable, so create a new instance to restart a
+service.
+
+## Configuration reference
+
+QoS is fixed at 1 on both configs. Topics are validated as publish topics
+and cannot contain MQTT wildcards. Authentication/TLS is applied by
+supplying a configured `client_factory` when constructing a component (see
+`examples/tls_auth_client.py`) — the config objects intentionally carry no
+credentials.
+
+`ReliabilityConfig`:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `host` | required | Broker hostname |
+| `queue_path` | required | Durable outbox file path |
+| `port` | `1883` | Broker port |
+| `client_id` | auto-generated | MQTT client ID |
+| `data_topic` | `reliomq/messages` | Topic the publisher sends on |
+| `ack_topic` | `reliomq/acks` | Topic the publisher listens on for ACKs |
+| `qos` | `1` | Fixed at 1 |
+| `ack_timeout` | `3.0`s | How long to wait for the application ACK |
+| `publish_timeout` | `2.0`s | How long to wait for MQTT publish confirmation |
+| `retry_interval` | `10.0`s | Delay between retries after a failure |
+| `keepalive` | `60`s | MQTT keepalive |
+| `reconnect_min_delay` / `reconnect_max_delay` | `1.0`s / `60.0`s | Paho reconnect backoff range |
+
+`BridgeConfig`:
+
+| Field | Default | Meaning |
+|---|---|---|
+| `source_host` / `destination_host` | required | The two brokers being bridged |
+| `source_port` / `destination_port` | `1883` | Ports for each broker |
+| `source_client_id` / `destination_client_id` | auto-generated | MQTT client IDs for each side |
+| `data_topic` / `ack_topic` | `reliomq/messages` / `reliomq/acks` | Must match the publisher |
+| `qos` | `1` | Fixed at 1 |
+| `keepalive` | `60`s | MQTT keepalive |
+| `destination_publish_timeout` | `2.0`s | Confirmation wait on the destination publish |
+| `source_ack_publish_timeout` | `0.5`s | Confirmation wait on the source ACK publish |
+| `retry_interval` | `10.0`s | Subscription retry delay |
+| `reconnect_min_delay` / `reconnect_max_delay` | `1.0`s / `60.0`s | Paho reconnect backoff range |
+| `max_queue_size` | `1000` | Bound on the bridge's in-memory handoff queue |
+
+## Logging
+
+`reliomq` uses the standard library `logging` module and configures
+nothing on its own — no handlers, no forced level, no `basicConfig()` call.
+Nothing prints until your application configures logging.
+
+| Logger | Used by |
+|---|---|
+| `reliomq.publisher` | `ReliablePublisher`, and the `DurableMessageStore` it creates internally |
+| `reliomq.bridge` | `ReliableMqttBridge` (override with `bridge_logger=`) |
+| `reliomq.store` | a `DurableMessageStore` you construct directly without passing `logger=` |
+| `reliomq.mqtt` | the `confirmed_publish()` helper |
+
+Turn logs on with one line, since propagation is never disabled:
+
+```python
+import logging
+logging.basicConfig(level=logging.INFO)               # everything, INFO and up
+logging.getLogger("reliomq").setLevel(logging.DEBUG)   # or scope it to just this library
+```
+
+**Levels, by meaning:**
+
+- **DEBUG** — routine traffic: message durably queued, ACK subscription
+  ready, delivery confirmed.
+- **INFO** — lifecycle: publisher/bridge started/stopped, MQTT connected.
+- **WARNING** — recoverable trouble that's expected during normal outage
+  handling: disconnects, rejected subscriptions, late/malformed/wrong-ID
+  ACKs ignored, forward failures, a full bridge queue. Nothing is lost when
+  you see these.
+- **ERROR** (some via `logger.exception()`, with a traceback) — things that
+  should not happen: the worker failing to stop promptly on shutdown, an ACK
+  matching a message that turned out not to be the durable oldest, or an
+  unexpected exception in the delivery/forward loop.
+
+## Examples
+
+All scripts live in `examples/` and are runnable directly (`python
+examples/<name>.py`) against a real broker unless noted otherwise:
+
+- `basic.py` — minimal one-shot publish and `wait_for_delivery`.
+- `sensor_loop.py` — a long-running periodic publisher with graceful
+  SIGINT/SIGTERM shutdown and a pending-backlog warning; the shape most
+  edge/IoT integrations actually use.
+- `bridge.py` — minimal standalone forwarder service.
+- `consumer_dedup.py` — a plain Paho subscriber (not part of this package)
+  showing the recommended `event_id` deduplication pattern for a final
+  consumer of bridged messages.
+- `local_end_to_end.py` — publisher + bridge + consumer wired together
+  against one local Mosquitto instance, so you can watch real PUBACKs,
+  reconnects, and the on-disk queue file; kill and restart the broker
+  mid-run to see `pending_count()` rise and drain.
+- `tls_auth_client.py` — injecting TLS and username/password auth through a
+  custom `client_factory` without adding security config to the library.
+
+## Tests
+
+Run the deterministic suite from this directory:
+
+```bash
+python -m unittest discover -s tests -v
+```
+
+`test_protocol.py`, `test_store.py`, `test_ack.py`, `test_mqtt.py`, and
+`test_config.py` each test one module in isolation: envelope/ACK wire
+encoding, durable FIFO storage, the single-waiter ACK correlator, the small
+Paho helper functions, and configuration validation, respectively.
+`test_publisher.py` and `test_bridge.py` drive each component through a fake
+Paho client to exercise success, broker outage, return-code failure,
+publish-confirmation timeout, ACK timeout, restart, FIFO recovery,
+wrong/late/malformed/duplicate ACKs, reconnect, bridge failure/success, and
+shutdown state transitions. `test_pipeline.py` goes a level higher: it wires
+a real `ReliablePublisher` to a real `ReliableMqttBridge` through two linked
+fake clients that relay `publish()` calls the way a broker would, so the two
+components run on their own real background threads and exchange genuine
+envelope/ACK traffic — catching integration regressions that per-component
+unit tests with directly injected ACKs cannot see. An optional Mosquitto
+integration test is skipped when the broker executable is unavailable.
+
+## License
+
+MIT — see [LICENSE](LICENSE).