webhookd-sdk 0.2.0__tar.gz → 0.3.0__tar.gz

{webhookd_sdk-0.2.0 → webhookd_sdk-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: webhookd-sdk
-Version: 0.2.0
+Version: 0.3.0
 Summary: Official Python SDK for NimbusNexus Webhooks — publish events + verify webhook signatures.
 Project-URL: Homepage, https://github.com/NimbusNexus/Webhooks-sdks/tree/develop/python#readme
 Project-URL: Repository, https://github.com/NimbusNexus/Webhooks-sdks
@@ -24,6 +24,10 @@ Provides-Extra: dev
 Requires-Dist: mypy; extra == 'dev'
 Requires-Dist: pytest; extra == 'dev'
 Requires-Dist: ruff; extra == 'dev'
+Provides-Extra: postgres
+Requires-Dist: psycopg[binary]>=3; extra == 'postgres'
+Provides-Extra: redis
+Requires-Dist: redis>=5; extra == 'redis'
 Description-Content-Type: text/markdown
 
 # webhookd-sdk (Python)
@@ -75,6 +79,54 @@ with Client("https://webhooks.example.com", api_key="whsk_…") as wh:
 Transient failures (connection errors, `429`, `5xx`) are retried with backoff (a `429` honours
 `Retry-After`); other `4xx` raise `WebhookdAPIError`.
 
+## Outbox / durable buffering (producers)
+
+`publish()` calls webhookd synchronously — if webhookd is unreachable it raises and the event is
+lost. The **write-first outbox** decouples the two: `enqueue()` durably persists the event to a
+pluggable `Store` and returns IMMEDIATELY (no network); `drain()` (or a background drainer) ships the
+buffered events later. Every send carries `Idempotency-Key = record.id`, so a re-drain after a crash
+or a lost response never double-publishes — webhookd dedupes. Delivery is **at-least-once**: nothing
+is lost while webhookd is down.
+
+```python
+from webhookd_sdk import Client, SQLiteStore
+
+# 1. Configure a durable store (survives process restarts).
+store = SQLiteStore("outbox.db")
+
+with Client("https://webhooks.example.com", api_key="whsk_…", store=store) as wh:
+    # 2. enqueue() instead of publish() — writes to the store and returns at once, NO network call.
+    record_id = wh.enqueue("order.created", {"order_id": "ord_123", "total": 4200})
+
+    # 3a. Drain on demand (returns {"sent", "failed", "remaining"}):
+    wh.drain()
+
+    # 3b. …or run a background drainer that calls drain() every 5s until the client closes.
+    wh.start_drainer(interval_seconds=5)
+    # ... your app keeps enqueuing; the drainer ships in the background ...
+    wh.stop_drainer()   # also called automatically by Client.close()/__exit__
+```
+
+**Idempotency guarantee.** `record_id` is the `idempotency_key` you pass (or a generated UUID v4) and
+becomes the `Idempotency-Key` header on every delivery attempt for that record. If the process
+crashes after a send but before the response is recorded, the next `drain()` re-sends with the *same*
+key and webhookd returns the original event without re-fanning-out. A record that keeps failing is
+retried with capped exponential backoff up to `max_attempts` (default 10), then flagged **dead**
+(never retried again) and handed to the optional `on_dead` callback.
+
+**Built-in stores** — pick one for `Client(..., store=...)`:
+
+| Store | Durable? | Extra needed |
+| --- | --- | --- |
+| `MemoryStore` | No (in-process) | — (stdlib) |
+| `FileStore(dir)` | Yes (per-record JSON files) | — (stdlib) |
+| `SQLiteStore(path)` | Yes (transactional) | — (stdlib `sqlite3`) |
+| `RedisStore(url)` | Yes | `pip install 'webhookd-sdk[redis]'` |
+| `PostgresStore(dsn)` | Yes | `pip install 'webhookd-sdk[postgres]'` |
+
+The core SDK stays zero-dependency; `RedisStore` / `PostgresStore` lazily import their driver only
+when you construct them.
+
 ## Manage endpoints, keys & deliveries (operators)
 
 The same `Client` wraps the control-plane API — register receivers, mint keys, and drain the

{webhookd_sdk-0.2.0 → webhookd_sdk-0.3.0}/README.md

@@ -47,6 +47,54 @@ with Client("https://webhooks.example.com", api_key="whsk_…") as wh:
 Transient failures (connection errors, `429`, `5xx`) are retried with backoff (a `429` honours
 `Retry-After`); other `4xx` raise `WebhookdAPIError`.
 
+## Outbox / durable buffering (producers)
+
+`publish()` calls webhookd synchronously — if webhookd is unreachable it raises and the event is
+lost. The **write-first outbox** decouples the two: `enqueue()` durably persists the event to a
+pluggable `Store` and returns IMMEDIATELY (no network); `drain()` (or a background drainer) ships the
+buffered events later. Every send carries `Idempotency-Key = record.id`, so a re-drain after a crash
+or a lost response never double-publishes — webhookd dedupes. Delivery is **at-least-once**: nothing
+is lost while webhookd is down.
+
+```python
+from webhookd_sdk import Client, SQLiteStore
+
+# 1. Configure a durable store (survives process restarts).
+store = SQLiteStore("outbox.db")
+
+with Client("https://webhooks.example.com", api_key="whsk_…", store=store) as wh:
+    # 2. enqueue() instead of publish() — writes to the store and returns at once, NO network call.
+    record_id = wh.enqueue("order.created", {"order_id": "ord_123", "total": 4200})
+
+    # 3a. Drain on demand (returns {"sent", "failed", "remaining"}):
+    wh.drain()
+
+    # 3b. …or run a background drainer that calls drain() every 5s until the client closes.
+    wh.start_drainer(interval_seconds=5)
+    # ... your app keeps enqueuing; the drainer ships in the background ...
+    wh.stop_drainer()   # also called automatically by Client.close()/__exit__
+```
+
+**Idempotency guarantee.** `record_id` is the `idempotency_key` you pass (or a generated UUID v4) and
+becomes the `Idempotency-Key` header on every delivery attempt for that record. If the process
+crashes after a send but before the response is recorded, the next `drain()` re-sends with the *same*
+key and webhookd returns the original event without re-fanning-out. A record that keeps failing is
+retried with capped exponential backoff up to `max_attempts` (default 10), then flagged **dead**
+(never retried again) and handed to the optional `on_dead` callback.
+
+**Built-in stores** — pick one for `Client(..., store=...)`:
+
+| Store | Durable? | Extra needed |
+| --- | --- | --- |
+| `MemoryStore` | No (in-process) | — (stdlib) |
+| `FileStore(dir)` | Yes (per-record JSON files) | — (stdlib) |
+| `SQLiteStore(path)` | Yes (transactional) | — (stdlib `sqlite3`) |
+| `RedisStore(url)` | Yes | `pip install 'webhookd-sdk[redis]'` |
+| `PostgresStore(dsn)` | Yes | `pip install 'webhookd-sdk[postgres]'` |
+
+The core SDK stays zero-dependency; `RedisStore` / `PostgresStore` lazily import their driver only
+when you construct them.
+
 ## Manage endpoints, keys & deliveries (operators)
 
 The same `Client` wraps the control-plane API — register receivers, mint keys, and drain the

{webhookd_sdk-0.2.0 → webhookd_sdk-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "webhookd-sdk"
-version = "0.2.0"
+version = "0.3.0"
 description = "Official Python SDK for NimbusNexus Webhooks — publish events + verify webhook signatures."
 readme = "README.md"
 requires-python = ">=3.11"
@@ -30,6 +30,8 @@ Repository = "https://github.com/NimbusNexus/Webhooks-sdks"
 Issues = "https://github.com/NimbusNexus/Webhooks-sdks/issues"
 
 [project.optional-dependencies]
+redis = ["redis>=5"]
+postgres = ["psycopg[binary]>=3"]
 dev = ["pytest", "ruff", "mypy"]
 
 [tool.hatch.build.targets.wheel]

webhookd_sdk-0.3.0/tests/test_outbox.py

@@ -0,0 +1,293 @@
+"""Write-first async outbox — a SHARED store-contract test parametrized over every Store, plus the
+client enqueue/drain flow driven through an httpx MockTransport (no real webhookd).
+
+Memory / File / SQLite run fully and locally (no server). Redis and Postgres run the SAME contract
+only when ``WEBHOOKD_TEST_REDIS_URL`` / ``WEBHOOKD_TEST_PG_URL`` point at a reachable server; they
+skip cleanly otherwise.
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import time
+import uuid
+from collections.abc import Iterator
+from typing import Any
+
+import httpx
+import pytest
+
+from webhookd_sdk import (
+    Client,
+    FileStore,
+    MemoryStore,
+    PostgresStore,
+    Record,
+    RedisStore,
+    SQLiteStore,
+    Store,
+)
+from webhookd_sdk import client as client_mod
+
+_OK = {
+    "id": "evt_db",
+    "event_uid": "u1",
+    "event_type": "order.created",
+    "application": "default",
+    "environment": "prod",
+    "deliveries_created": 1,
+    "source": None,
+}
+
+
+def _record(rid: str, *, created_at: float, next_attempt_at: float | None = None) -> Record:
+    return Record(
+        id=rid,
+        event_type="order.created",
+        payload={"id": rid},
+        created_at=created_at,
+        next_attempt_at=created_at if next_attempt_at is None else next_attempt_at,
+    )
+
+
+# -- store factories: memory/file/sqlite always run; redis/pg skip unless a server is configured ---
+def _make_memory(tmp_path: Any) -> Store:
+    return MemoryStore()
+
+
+def _make_file(tmp_path: Any) -> Store:
+    return FileStore(tmp_path / "outbox")
+
+
+def _make_sqlite(tmp_path: Any) -> Store:
+    return SQLiteStore(str(tmp_path / "outbox.db"))
+
+
+def _make_redis(tmp_path: Any) -> Store:
+    url = os.environ.get("WEBHOOKD_TEST_REDIS_URL")
+    if not url:
+        pytest.skip("set WEBHOOKD_TEST_REDIS_URL to run the Redis store contract")
+    try:
+        store = RedisStore(url, namespace=f"test:{uuid.uuid4()}")
+        store._redis.ping()  # type: ignore[attr-defined]
+    except Exception as exc:  # noqa: BLE001 — unreachable server => skip, never fail
+        pytest.skip(f"Redis unreachable: {exc}")
+    return store
+
+
+def _make_pg(tmp_path: Any) -> Store:
+    dsn = os.environ.get("WEBHOOKD_TEST_PG_URL")
+    if not dsn:
+        pytest.skip("set WEBHOOKD_TEST_PG_URL to run the Postgres store contract")
+    try:
+        table = f"webhookd_outbox_test_{uuid.uuid4().hex}"
+        store = PostgresStore(dsn, table=table)
+    except Exception as exc:  # noqa: BLE001 — unreachable server => skip, never fail
+        pytest.skip(f"Postgres unreachable: {exc}")
+    return store
+
+
+@pytest.fixture(
+    params=[_make_memory, _make_file, _make_sqlite, _make_redis, _make_pg],
+    ids=["memory", "file", "sqlite", "redis", "postgres"],
+)
+def store(request: pytest.FixtureRequest, tmp_path: Any) -> Iterator[Store]:
+    s = request.param(tmp_path)
+    try:
+        yield s
+    finally:
+        s.close()
+
+
+# -- shared store contract ----------------------------------------------------
+def test_save_and_size(store: Store) -> None:
+    assert store.size() == 0
+    store.save(_record("a", created_at=1.0))
+    store.save(_record("b", created_at=2.0))
+    assert store.size() == 2
+
+
+def test_save_is_upsert_by_id(store: Store) -> None:
+    store.save(_record("a", created_at=1.0))
+    updated = _record("a", created_at=1.0)
+    updated.payload = {"changed": True}
+    store.save(updated)
+    assert store.size() == 1  # same id overwrote, not appended
+    rows = store.list_pending(10)
+    assert len(rows) == 1 and rows[0].payload == {"changed": True}
+
+
+def test_list_pending_orders_oldest_first(store: Store) -> None:
+    store.save(_record("newer", created_at=200.0))
+    store.save(_record("older", created_at=100.0))
+    ids = [r.id for r in store.list_pending(10)]
+    assert ids == ["older", "newer"]
+
+
+def test_list_pending_filters_not_yet_due(store: Store) -> None:
+    now = time.time()
+    store.save(_record("due", created_at=now - 10, next_attempt_at=now - 5))
+    store.save(_record("future", created_at=now - 10, next_attempt_at=now + 3600))
+    ids = [r.id for r in store.list_pending(10)]
+    assert ids == ["due"]  # the future record is withheld until its next_attempt_at
+
+
+def test_list_pending_respects_limit(store: Store) -> None:
+    for i in range(5):
+        store.save(_record(f"r{i}", created_at=float(i)))
+    assert len(store.list_pending(3)) == 3
+
+
+def test_mark_sent_removes(store: Store) -> None:
+    store.save(_record("a", created_at=1.0))
+    store.mark_sent("a")
+    assert store.size() == 0
+    assert store.list_pending(10) == []
+
+
+def test_mark_failed_reschedules_and_bumps_attempts(store: Store) -> None:
+    now = time.time()
+    store.save(_record("a", created_at=now - 10, next_attempt_at=now - 5))
+    # reschedule into the future -> no longer due, but still pending (not lost)
+    store.mark_failed("a", "boom", 3, now + 3600)
+    assert store.size() == 1
+    assert store.list_pending(10) == []
+    # bring it back due and confirm the failure fields persisted
+    store.mark_failed("a", "boom-again", 4, now - 1)
+    rows = store.list_pending(10)
+    assert len(rows) == 1
+    assert rows[0].attempts == 4 and rows[0].last_error == "boom-again"
+
+
+# -- durability (survives a fresh store over the same backing) ----------------
+def test_file_store_survives_restart(tmp_path: Any) -> None:
+    path = tmp_path / "durable"
+    s1 = FileStore(path)
+    s1.save(_record("a", created_at=1.0))
+    s1.close()
+    s2 = FileStore(path)
+    assert s2.size() == 1 and s2.list_pending(10)[0].id == "a"
+    s2.close()
+
+
+def test_sqlite_store_survives_restart(tmp_path: Any) -> None:
+    path = str(tmp_path / "durable.db")
+    s1 = SQLiteStore(path)
+    s1.save(_record("a", created_at=1.0))
+    s1.close()
+    s2 = SQLiteStore(path)
+    assert s2.size() == 1 and s2.list_pending(10)[0].id == "a"
+    s2.close()
+
+
+# -- client enqueue/drain flow (mock transport, no real webhookd) -------------
+def _client(handler: Any, *, store: Store, **kw: Any) -> Client:
+    return Client(
+        "https://wh.example.com",
+        "whsk_x",
+        transport=httpx.MockTransport(handler),
+        store=store,
+        **kw,
+    )
+
+
+def test_enqueue_writes_no_network_then_drain_sends_all_with_idem_headers() -> None:
+    seen: list[dict[str, Any]] = []
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        seen.append(
+            {
+                "idem": request.headers.get("Idempotency-Key"),
+                "body": json.loads(request.content),
+            }
+        )
+        return httpx.Response(201, json=_OK)
+
+    store = MemoryStore()
+    with _client(handler, store=store) as c:
+        ids = [c.enqueue("order.created", {"n": i}) for i in range(3)]
+        assert store.size() == 3 and seen == []  # enqueue made no network call
+        result = c.drain()
+
+    assert result == {"sent": 3, "failed": 0, "remaining": 0}
+    assert [s["idem"] for s in seen] == ids  # each send carried its record.id as the key
+    assert all(s["body"]["environment"] == "prod" for s in seen)
+
+
+def test_supplied_idempotency_key_is_used_as_id_and_header() -> None:
+    seen: dict[str, str | None] = {}
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        seen["idem"] = request.headers.get("Idempotency-Key")
+        return httpx.Response(201, json=_OK)
+
+    store = MemoryStore()
+    with _client(handler, store=store) as c:
+        rid = c.enqueue("order.created", {"id": 1}, idempotency_key="order-123")
+        assert rid == "order-123"
+        c.drain()
+    assert seen["idem"] == "order-123"
+
+
+def test_failing_transport_marks_failed_not_sent_then_retries_same_id(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    # zero the backoff so the rescheduled record is immediately due on the next drain
+    monkeypatch.setattr(client_mod, "_backoff", lambda attempt: 0.0)
+    keys: list[str | None] = []
+    calls = {"n": 0}
+
+    def handler(request: httpx.Request) -> httpx.Response:
+        keys.append(request.headers.get("Idempotency-Key"))
+        calls["n"] += 1
+        if calls["n"] == 1:
+            return httpx.Response(503, json={"error": {"code": "unavailable", "message": "down"}})
+        return httpx.Response(201, json=_OK)
+
+    store = MemoryStore()
+    with _client(handler, store=store, max_retries=0) as c:
+        rid = c.enqueue("order.created", {"id": 1})
+        first = c.drain()  # webhookd down -> failed, kept, attempts bumped
+        assert first == {"sent": 0, "failed": 1, "remaining": 1}
+        pending = store.list_pending(10)
+        assert len(pending) == 1 and pending[0].attempts == 1 and not pending[0].sent
+        second = c.drain()  # webhookd back -> sent
+        assert second == {"sent": 1, "failed": 0, "remaining": 0}
+
+    assert keys == [rid, rid]  # the SAME record.id is reused on retry (idempotent)
+
+
+def test_max_attempts_ceiling_flags_dead_and_fires_hook(
+    monkeypatch: pytest.MonkeyPatch,
+) -> None:
+    monkeypatch.setattr(client_mod, "_backoff", lambda attempt: 0.0)
+
+    def handler(_request: httpx.Request) -> httpx.Response:
+        return httpx.Response(500, json={"error": {"code": "boom", "message": "always"}})
+
+    dead: list[Record] = []
+    store = MemoryStore()
+    with _client(handler, store=store, max_retries=0, on_dead=dead.append) as c:
+        c.enqueue("order.created", {"id": 1})
+        result = c.drain(max_attempts=1)  # 1 failure == the ceiling -> dead, not retried
+
+    assert result == {"sent": 0, "failed": 1, "remaining": 0}  # dead is excluded from remaining
+    assert store.list_pending(10) == []  # never retried again
+    assert len(dead) == 1 and dead[0].dead is True
+
+
+def test_background_drainer_starts_and_stops_cleanly() -> None:
+    def handler(_request: httpx.Request) -> httpx.Response:
+        return httpx.Response(201, json=_OK)
+
+    store = MemoryStore()
+    with _client(handler, store=store) as c:
+        c.enqueue("order.created", {"id": 1})
+        c.start_drainer(0.01)
+        deadline = time.time() + 2.0
+        while store.size() > 0 and time.time() < deadline:
+            time.sleep(0.01)
+        c.stop_drainer()
+    assert store.size() == 0
+    assert c._drainer is None  # joined cleanly

{webhookd_sdk-0.2.0 → webhookd_sdk-0.3.0}/webhookd_sdk/__init__.py

@@ -8,16 +8,32 @@ from __future__ import annotations
 
 from webhookd_sdk.client import Client, Event
 from webhookd_sdk.errors import WebhookdAPIError, WebhookdError
+from webhookd_sdk.outbox import (
+    FileStore,
+    MemoryStore,
+    PostgresStore,
+    Record,
+    RedisStore,
+    SQLiteStore,
+    Store,
+)
 from webhookd_sdk.signature import DEFAULT_TOLERANCE_SECONDS, sign, verify
 
 # Keep in lockstep with pyproject.toml + the release tag (see publish.yml's bump checklist).
-__version__ = "0.2.0"
+__version__ = "0.3.0"
 
 __all__ = [
     "Client",
     "Event",
     "WebhookdAPIError",
     "WebhookdError",
+    "Store",
+    "Record",
+    "MemoryStore",
+    "FileStore",
+    "SQLiteStore",
+    "RedisStore",
+    "PostgresStore",
     "DEFAULT_TOLERANCE_SECONDS",
     "sign",
     "verify",

{webhookd_sdk-0.2.0 → webhookd_sdk-0.3.0}/webhookd_sdk/client.py

@@ -2,13 +2,17 @@
 
 from __future__ import annotations
 
+import threading
 import time
+import uuid
+from collections.abc import Callable
 from dataclasses import dataclass
 from typing import Any
 
 import httpx
 
 from webhookd_sdk.errors import WebhookdAPIError, WebhookdError
+from webhookd_sdk.outbox import Record, Store
 
 _RETRY_STATUSES = frozenset({429, 500, 502, 503, 504})
 
@@ -59,11 +63,17 @@ class Client:
         timeout: float = 10.0,
         max_retries: int = 2,
         transport: httpx.BaseTransport | None = None,
+        store: Store | None = None,
+        on_dead: Callable[[Record], None] | None = None,
     ):
         self._base = base_url.rstrip("/")
         self._key = api_key
         self._max_retries = max_retries
         self._client = httpx.Client(timeout=timeout, transport=transport)
+        self._store = store
+        self._on_dead = on_dead
+        self._drainer: threading.Thread | None = None
+        self._drainer_stop = threading.Event()
 
     def publish(
         self,
@@ -91,6 +101,115 @@ class Client:
         resp = self._request("POST", "/v1/events", json=body, extra_headers=extra_headers)
         return Event._from_json(resp.json())
 
+    # -- write-first async outbox ----------------------------------------------
+    def enqueue(
+        self,
+        event_type: str,
+        payload: dict[str, Any],
+        *,
+        idempotency_key: str | None = None,
+        environment: str = "prod",
+        application: str = "default",
+        source: str | None = None,
+    ) -> str:
+        """Durably buffer one event and return its id WITHOUT any network call.
+
+        The id is ``idempotency_key`` (if given) or a fresh UUID v4; it becomes the
+        ``Idempotency-Key`` on every later :meth:`drain` send, so re-draining after a crash never
+        double-publishes. Requires a ``store`` to be configured on the client.
+        """
+        if self._store is None:
+            raise WebhookdError("enqueue requires a store — construct Client(..., store=...)")
+        now = time.time()
+        record = Record(
+            id=idempotency_key or str(uuid.uuid4()),
+            event_type=event_type,
+            payload=payload,
+            environment=environment,
+            application=application,
+            source=source,
+            created_at=now,
+            next_attempt_at=now,
+        )
+        self._store.save(record)
+        return record.id
+
+    def drain(self, *, limit: int = 100, max_attempts: int = 10) -> dict[str, int]:
+        """Send buffered records to webhookd. Returns ``{"sent", "failed", "remaining"}``.
+
+        For each due record it POSTs ``/v1/events`` with header ``Idempotency-Key = record.id``.
+        On 2xx the record is removed (``mark_sent``). On failure ``attempts`` is bumped, the next
+        retry is scheduled with capped exponential backoff, and the record is kept; once
+        ``attempts`` reaches ``max_attempts`` the record is flagged dead (not retried again) and
+        handed to the ``on_dead`` callback if configured. ``remaining`` is the pending count.
+        """
+        if self._store is None:
+            raise WebhookdError("drain requires a store — construct Client(..., store=...)")
+        sent = 0
+        failed = 0
+        for record in self._store.list_pending(limit):
+            body: dict[str, Any] = {
+                "event_type": record.event_type,
+                "payload": record.payload,
+                "environment": record.environment,
+                "application": record.application,
+            }
+            if record.source is not None:
+                body["source"] = record.source
+            try:
+                self._request(
+                    "POST", "/v1/events", json=body, extra_headers={"Idempotency-Key": record.id}
+                )
+            except WebhookdError as exc:
+                record.attempts += 1
+                record.last_error = str(exc)
+                if record.attempts >= max_attempts:
+                    record.dead = True
+                    self._store.save(record)
+                    if self._on_dead is not None:
+                        self._on_dead(record)
+                else:
+                    record.next_attempt_at = time.time() + _backoff(record.attempts)
+                    self._store.mark_failed(
+                        record.id, record.last_error, record.attempts, record.next_attempt_at
+                    )
+                failed += 1
+            else:
+                self._store.mark_sent(record.id)
+                sent += 1
+        return {"sent": sent, "failed": failed, "remaining": self._store.size()}
+
+    def start_drainer(
+        self, interval_seconds: float, *, limit: int = 100, max_attempts: int = 10
+    ) -> None:
+        """Start a daemon thread that calls :meth:`drain` every ``interval_seconds`` until
+        :meth:`stop_drainer` (or :meth:`close`). A per-drain exception is swallowed so a transient
+        outage never kills the loop. Idempotent — a second call while one is running is a no-op."""
+        if self._store is None:
+            raise WebhookdError("start_drainer requires a store — construct Client(..., store=...)")
+        if self._drainer is not None and self._drainer.is_alive():
+            return
+        self._drainer_stop.clear()
+
+        def _loop() -> None:
+            while not self._drainer_stop.is_set():
+                try:
+                    self.drain(limit=limit, max_attempts=max_attempts)
+                except Exception:  # noqa: BLE001 — keep the loop alive across transient failures
+                    pass
+                self._drainer_stop.wait(interval_seconds)
+
+        self._drainer = threading.Thread(target=_loop, name="webhookd-drainer", daemon=True)
+        self._drainer.start()
+
+    def stop_drainer(self, *, timeout: float | None = 5.0) -> None:
+        """Signal the background drainer to stop and join it. Safe to call when none is running."""
+        self._drainer_stop.set()
+        drainer = self._drainer
+        if drainer is not None:
+            drainer.join(timeout)
+            self._drainer = None
+
     # -- endpoints -------------------------------------------------------------
     def create_endpoint(
         self,
@@ -229,6 +348,7 @@ class Client:
         return self._request("POST", f"/v1/deliveries/{delivery_id}/redeliver").json()
 
     def close(self) -> None:
+        self.stop_drainer()
         self._client.close()
 
     def __enter__(self) -> Client:

webhookd_sdk-0.3.0/webhookd_sdk/outbox.py

@@ -0,0 +1,546 @@
+"""Write-first async outbox — a durable buffer for publishing events.
+
+The producer calls :meth:`Client.enqueue` which writes an event :class:`Record` to a pluggable
+:class:`Store` and returns IMMEDIATELY (no network). A later :meth:`Client.drain` (or the background
+drainer started via :meth:`Client.start_drainer`) POSTs the buffered records to webhookd. Every send
+carries ``Idempotency-Key = record.id`` so a re-drain after a crash or a lost response never
+double-publishes (webhookd dedupes). Delivery is at-least-once: nothing is lost while webhookd is
+down.
+
+Five stores ship in the box: :class:`MemoryStore` (in-process, non-durable — the default/tests),
+:class:`FileStore` (a directory of per-record JSON files with atomic writes), :class:`SQLiteStore`
+(stdlib ``sqlite3``, transactional) — all runnable with no server — plus :class:`RedisStore` and
+:class:`PostgresStore`, which lazily import their optional drivers (``pip install
+'webhookd-sdk[redis]'`` / ``'[postgres]'``).
+"""
+
+from __future__ import annotations
+
+import json
+import os
+import sqlite3
+import threading
+import time
+from abc import ABC, abstractmethod
+from dataclasses import asdict, dataclass, field
+from typing import Any
+from urllib.parse import quote
+
+
+@dataclass
+class Record:
+    """One buffered event. ``id`` is the ``Idempotency-Key`` (caller-supplied or a UUID v4)."""
+
+    id: str
+    event_type: str
+    payload: dict[str, Any]
+    environment: str = "prod"
+    application: str = "default"
+    source: str | None = None
+    created_at: float = field(default_factory=time.time)
+    attempts: int = 0
+    last_error: str | None = None
+    next_attempt_at: float = field(default_factory=time.time)
+    sent: bool = False
+    dead: bool = False
+
+    def to_dict(self) -> dict[str, Any]:
+        return asdict(self)
+
+    @classmethod
+    def from_dict(cls, d: dict[str, Any]) -> Record:
+        return cls(
+            id=d["id"],
+            event_type=d["event_type"],
+            payload=d["payload"],
+            environment=d.get("environment", "prod"),
+            application=d.get("application", "default"),
+            source=d.get("source"),
+            created_at=d.get("created_at", 0.0),
+            attempts=d.get("attempts", 0),
+            last_error=d.get("last_error"),
+            next_attempt_at=d.get("next_attempt_at", 0.0),
+            sent=bool(d.get("sent", False)),
+            dead=bool(d.get("dead", False)),
+        )
+
+
+def _clone(record: Record) -> Record:
+    """A deep-ish copy so a store never hands out an object the caller can mutate in place."""
+    return Record.from_dict(json.loads(json.dumps(record.to_dict())))
+
+
+class Store(ABC):
+    """Durable buffer of pending :class:`Record`s. All methods are safe to call from the drainer
+    thread and the caller thread concurrently."""
+
+    @abstractmethod
+    def save(self, record: Record) -> None:
+        """Insert-or-UPDATE by ``record.id`` — the same id overwrites, so enqueue is idempotent."""
+
+    @abstractmethod
+    def list_pending(self, limit: int) -> list[Record]:
+        """Records not yet sent (and not dead) whose ``next_attempt_at <= now``, oldest first (by
+        ``created_at``), capped at ``limit``."""
+
+    @abstractmethod
+    def mark_sent(self, id: str) -> None:
+        """Remove a record after a 2xx."""
+
+    @abstractmethod
+    def mark_failed(
+        self, id: str, error: str | None, attempts: int, next_attempt_at: float
+    ) -> None:
+        """Persist the failure and schedule the next retry."""
+
+    @abstractmethod
+    def size(self) -> int:
+        """Count of pending (not sent, not dead) records."""
+
+    def close(self) -> None:  # noqa: B027 — optional hook; concrete stores override when needed
+        """Release any resources. The default is a no-op."""
+
+
+# -- MemoryStore --------------------------------------------------------------
+class MemoryStore(Store):
+    """In-process, non-durable store. The default — ideal for tests and for callers that only want
+    the enqueue/drain shape without persistence."""
+
+    def __init__(self) -> None:
+        self._rows: dict[str, Record] = {}
+        self._lock = threading.Lock()
+
+    def save(self, record: Record) -> None:
+        with self._lock:
+            self._rows[record.id] = _clone(record)
+
+    def list_pending(self, limit: int) -> list[Record]:
+        now = time.time()
+        with self._lock:
+            due = [
+                r
+                for r in self._rows.values()
+                if not r.sent and not r.dead and r.next_attempt_at <= now
+            ]
+        due.sort(key=lambda r: r.created_at)
+        return [_clone(r) for r in due[:limit]]
+
+    def mark_sent(self, id: str) -> None:
+        with self._lock:
+            self._rows.pop(id, None)
+
+    def mark_failed(
+        self, id: str, error: str | None, attempts: int, next_attempt_at: float
+    ) -> None:
+        with self._lock:
+            r = self._rows.get(id)
+            if r is not None:
+                r.last_error = error
+                r.attempts = attempts
+                r.next_attempt_at = next_attempt_at
+
+    def size(self) -> int:
+        with self._lock:
+            return sum(1 for r in self._rows.values() if not r.sent and not r.dead)
+
+
+# -- FileStore ----------------------------------------------------------------
+class FileStore(Store):
+    """Durable on-disk store: one JSON file per record in ``directory``, written atomically
+    (temp file + ``os.replace``) so a crash mid-write never corrupts a record. Survives process
+    restart — a fresh :class:`FileStore` over the same directory sees every un-sent record."""
+
+    def __init__(self, directory: str | os.PathLike[str]) -> None:
+        self._dir = os.fspath(directory)
+        os.makedirs(self._dir, exist_ok=True)
+        self._lock = threading.Lock()
+
+    def _path(self, id: str) -> str:
+        return os.path.join(self._dir, quote(id, safe="") + ".json")
+
+    def _write(self, record: Record) -> None:
+        path = self._path(record.id)
+        tmp = f"{path}.{os.getpid()}.{threading.get_ident()}.tmp"
+        with open(tmp, "w", encoding="utf-8") as fh:
+            json.dump(record.to_dict(), fh)
+        os.replace(tmp, path)
+
+    def _read(self, path: str) -> Record | None:
+        try:
+            with open(path, encoding="utf-8") as fh:
+                return Record.from_dict(json.load(fh))
+        except (FileNotFoundError, json.JSONDecodeError):
+            return None
+
+    def save(self, record: Record) -> None:
+        with self._lock:
+            self._write(record)
+
+    def _all(self) -> list[Record]:
+        rows: list[Record] = []
+        for name in os.listdir(self._dir):
+            if not name.endswith(".json"):
+                continue
+            r = self._read(os.path.join(self._dir, name))
+            if r is not None:
+                rows.append(r)
+        return rows
+
+    def list_pending(self, limit: int) -> list[Record]:
+        now = time.time()
+        with self._lock:
+            due = [r for r in self._all() if not r.sent and not r.dead and r.next_attempt_at <= now]
+        due.sort(key=lambda r: r.created_at)
+        return due[:limit]
+
+    def mark_sent(self, id: str) -> None:
+        with self._lock:
+            try:
+                os.remove(self._path(id))
+            except FileNotFoundError:
+                pass
+
+    def mark_failed(
+        self, id: str, error: str | None, attempts: int, next_attempt_at: float
+    ) -> None:
+        with self._lock:
+            r = self._read(self._path(id))
+            if r is not None:
+                r.last_error = error
+                r.attempts = attempts
+                r.next_attempt_at = next_attempt_at
+                self._write(r)
+
+    def size(self) -> int:
+        with self._lock:
+            return sum(1 for r in self._all() if not r.sent and not r.dead)
+
+
+# -- SQLiteStore --------------------------------------------------------------
+_SQLITE_DDL = """
+CREATE TABLE IF NOT EXISTS webhookd_outbox (
+    id              TEXT PRIMARY KEY,
+    event_type      TEXT NOT NULL,
+    payload         TEXT NOT NULL,
+    environment     TEXT NOT NULL,
+    application     TEXT NOT NULL,
+    source          TEXT,
+    created_at      REAL NOT NULL,
+    attempts        INTEGER NOT NULL,
+    last_error      TEXT,
+    next_attempt_at REAL NOT NULL,
+    sent            INTEGER NOT NULL DEFAULT 0,
+    dead            INTEGER NOT NULL DEFAULT 0
+)
+"""
+
+
+class SQLiteStore(Store):
+    """Durable, transactional store on stdlib ``sqlite3`` — no native dependency. Point it at a file
+    path (``":memory:"`` is non-durable). Upserts on ``id``; safe across the drainer thread via a
+    lock plus ``check_same_thread=False``."""
+
+    def __init__(self, path: str | os.PathLike[str]) -> None:
+        self._conn = sqlite3.connect(os.fspath(path), check_same_thread=False)
+        self._lock = threading.Lock()
+        with self._lock:
+            self._conn.execute(_SQLITE_DDL)
+            self._conn.commit()
+
+    @staticmethod
+    def _row_to_record(row: tuple[Any, ...]) -> Record:
+        return Record(
+            id=row[0],
+            event_type=row[1],
+            payload=json.loads(row[2]),
+            environment=row[3],
+            application=row[4],
+            source=row[5],
+            created_at=row[6],
+            attempts=row[7],
+            last_error=row[8],
+            next_attempt_at=row[9],
+            sent=bool(row[10]),
+            dead=bool(row[11]),
+        )
+
+    def save(self, record: Record) -> None:
+        with self._lock:
+            self._conn.execute(
+                """
+                INSERT INTO webhookd_outbox
+                    (id, event_type, payload, environment, application, source,
+                     created_at, attempts, last_error, next_attempt_at, sent, dead)
+                VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                ON CONFLICT(id) DO UPDATE SET
+                    event_type=excluded.event_type,
+                    payload=excluded.payload,
+                    environment=excluded.environment,
+                    application=excluded.application,
+                    source=excluded.source,
+                    created_at=excluded.created_at,
+                    attempts=excluded.attempts,
+                    last_error=excluded.last_error,
+                    next_attempt_at=excluded.next_attempt_at,
+                    sent=excluded.sent,
+                    dead=excluded.dead
+                """,
+                (
+                    record.id,
+                    record.event_type,
+                    json.dumps(record.payload),
+                    record.environment,
+                    record.application,
+                    record.source,
+                    record.created_at,
+                    record.attempts,
+                    record.last_error,
+                    record.next_attempt_at,
+                    int(record.sent),
+                    int(record.dead),
+                ),
+            )
+            self._conn.commit()
+
+    def list_pending(self, limit: int) -> list[Record]:
+        now = time.time()
+        with self._lock:
+            cur = self._conn.execute(
+                """
+                SELECT id, event_type, payload, environment, application, source,
+                       created_at, attempts, last_error, next_attempt_at, sent, dead
+                FROM webhookd_outbox
+                WHERE sent = 0 AND dead = 0 AND next_attempt_at <= ?
+                ORDER BY created_at ASC
+                LIMIT ?
+                """,
+                (now, limit),
+            )
+            rows = cur.fetchall()
+        return [self._row_to_record(r) for r in rows]
+
+    def mark_sent(self, id: str) -> None:
+        with self._lock:
+            self._conn.execute("DELETE FROM webhookd_outbox WHERE id = ?", (id,))
+            self._conn.commit()
+
+    def mark_failed(
+        self, id: str, error: str | None, attempts: int, next_attempt_at: float
+    ) -> None:
+        with self._lock:
+            self._conn.execute(
+                """
+                UPDATE webhookd_outbox
+                SET last_error = ?, attempts = ?, next_attempt_at = ?
+                WHERE id = ?
+                """,
+                (error, attempts, next_attempt_at, id),
+            )
+            self._conn.commit()
+
+    def size(self) -> int:
+        with self._lock:
+            cur = self._conn.execute(
+                "SELECT COUNT(*) FROM webhookd_outbox WHERE sent = 0 AND dead = 0"
+            )
+            return int(cur.fetchone()[0])
+
+    def close(self) -> None:
+        with self._lock:
+            self._conn.close()
+
+
+# -- RedisStore ---------------------------------------------------------------
+class RedisStore(Store):
+    """Durable store on Redis: a sorted set keyed on ``next_attempt_at`` holds the due-ordering of
+    pending ids, and a hash holds the record bodies. Sent/dead records are removed from the sorted
+    set so they are never retried. Requires the optional ``redis`` driver (lazily imported)."""
+
+    def __init__(self, url: str, *, namespace: str = "webhookd:outbox") -> None:
+        import redis  # lazy: optional extra [redis]
+
+        # Typed as Any: the driver is an optional dependency, so pin nothing to its concrete types.
+        self._redis: Any = redis.Redis.from_url(url, decode_responses=True)
+        self._zkey = f"{namespace}:pending"
+        self._hkey = f"{namespace}:records"
+
+    def save(self, record: Record) -> None:
+        pipe = self._redis.pipeline()
+        pipe.hset(self._hkey, record.id, json.dumps(record.to_dict()))
+        if record.sent or record.dead:
+            pipe.zrem(self._zkey, record.id)
+        else:
+            pipe.zadd(self._zkey, {record.id: record.next_attempt_at})
+        pipe.execute()
+
+    def list_pending(self, limit: int) -> list[Record]:
+        now = time.time()
+        ids = list(self._redis.zrangebyscore(self._zkey, "-inf", now))
+        if not ids:
+            return []
+        bodies = self._redis.hmget(self._hkey, ids)
+        rows = [Record.from_dict(json.loads(b)) for b in bodies if b is not None]
+        rows.sort(key=lambda r: r.created_at)
+        return rows[:limit]
+
+    def mark_sent(self, id: str) -> None:
+        pipe = self._redis.pipeline()
+        pipe.zrem(self._zkey, id)
+        pipe.hdel(self._hkey, id)
+        pipe.execute()
+
+    def mark_failed(
+        self, id: str, error: str | None, attempts: int, next_attempt_at: float
+    ) -> None:
+        body = self._redis.hget(self._hkey, id)
+        if body is None:
+            return
+        r = Record.from_dict(json.loads(body))
+        r.last_error = error
+        r.attempts = attempts
+        r.next_attempt_at = next_attempt_at
+        pipe = self._redis.pipeline()
+        pipe.hset(self._hkey, id, json.dumps(r.to_dict()))
+        pipe.zadd(self._zkey, {id: next_attempt_at})
+        pipe.execute()
+
+    def size(self) -> int:
+        return int(self._redis.zcard(self._zkey))
+
+    def close(self) -> None:
+        self._redis.close()
+
+
+# -- PostgresStore ------------------------------------------------------------
+_PG_DDL = """
+CREATE TABLE IF NOT EXISTS webhookd_outbox (
+    id              TEXT PRIMARY KEY,
+    event_type      TEXT NOT NULL,
+    payload         JSONB NOT NULL,
+    environment     TEXT NOT NULL,
+    application     TEXT NOT NULL,
+    source          TEXT,
+    created_at      DOUBLE PRECISION NOT NULL,
+    attempts        INTEGER NOT NULL,
+    last_error      TEXT,
+    next_attempt_at DOUBLE PRECISION NOT NULL,
+    sent            BOOLEAN NOT NULL DEFAULT FALSE,
+    dead            BOOLEAN NOT NULL DEFAULT FALSE
+)
+"""
+
+
+class PostgresStore(Store):
+    """Durable, transactional store on PostgreSQL. Upserts on ``id``; ``list_pending`` selects
+    ``WHERE NOT sent AND NOT dead AND next_attempt_at <= now ORDER BY created_at``. Requires the
+    optional ``psycopg`` (v3) driver (lazily imported)."""
+
+    def __init__(self, dsn: str, *, table: str = "webhookd_outbox") -> None:
+        import psycopg  # lazy: optional extra [postgres]
+
+        self._conn = psycopg.connect(dsn, autocommit=True)
+        self._lock = threading.Lock()
+        # `table` is a fixed identifier chosen by the caller, not user input; EVERY statement below
+        # (CREATE + all CRUD) is templated with it, so a custom table name works end to end.
+        self._table = table
+        with self._lock, self._conn.cursor() as cur:
+            cur.execute(
+                _PG_DDL if table == "webhookd_outbox" else _PG_DDL.replace("webhookd_outbox", table)
+            )
+
+    def save(self, record: Record) -> None:
+        with self._lock, self._conn.cursor() as cur:
+            cur.execute(
+                f"""
+                INSERT INTO {self._table}
+                    (id, event_type, payload, environment, application, source,
+                     created_at, attempts, last_error, next_attempt_at, sent, dead)
+                VALUES (%s, %s, %s::jsonb, %s, %s, %s, %s, %s, %s, %s, %s, %s)
+                ON CONFLICT (id) DO UPDATE SET
+                    event_type=EXCLUDED.event_type,
+                    payload=EXCLUDED.payload,
+                    environment=EXCLUDED.environment,
+                    application=EXCLUDED.application,
+                    source=EXCLUDED.source,
+                    created_at=EXCLUDED.created_at,
+                    attempts=EXCLUDED.attempts,
+                    last_error=EXCLUDED.last_error,
+                    next_attempt_at=EXCLUDED.next_attempt_at,
+                    sent=EXCLUDED.sent,
+                    dead=EXCLUDED.dead
+                """,
+                (
+                    record.id,
+                    record.event_type,
+                    json.dumps(record.payload),
+                    record.environment,
+                    record.application,
+                    record.source,
+                    record.created_at,
+                    record.attempts,
+                    record.last_error,
+                    record.next_attempt_at,
+                    record.sent,
+                    record.dead,
+                ),
+            )
+
+    def list_pending(self, limit: int) -> list[Record]:
+        now = time.time()
+        with self._lock, self._conn.cursor() as cur:
+            cur.execute(
+                f"""
+                SELECT id, event_type, payload, environment, application, source,
+                       created_at, attempts, last_error, next_attempt_at, sent, dead
+                FROM {self._table}
+                WHERE NOT sent AND NOT dead AND next_attempt_at <= %s
+                ORDER BY created_at ASC
+                LIMIT %s
+                """,
+                (now, limit),
+            )
+            rows = cur.fetchall()
+        return [
+            Record(
+                id=r[0],
+                event_type=r[1],
+                payload=r[2],
+                environment=r[3],
+                application=r[4],
+                source=r[5],
+                created_at=r[6],
+                attempts=r[7],
+                last_error=r[8],
+                next_attempt_at=r[9],
+                sent=r[10],
+                dead=r[11],
+            )
+            for r in rows
+        ]
+
+    def mark_sent(self, id: str) -> None:
+        with self._lock, self._conn.cursor() as cur:
+            cur.execute(f"DELETE FROM {self._table} WHERE id = %s", (id,))
+
+    def mark_failed(
+        self, id: str, error: str | None, attempts: int, next_attempt_at: float
+    ) -> None:
+        with self._lock, self._conn.cursor() as cur:
+            cur.execute(
+                f"""
+                UPDATE {self._table}
+                SET last_error = %s, attempts = %s, next_attempt_at = %s
+                WHERE id = %s
+                """,
+                (error, attempts, next_attempt_at, id),
+            )
+
+    def size(self) -> int:
+        with self._lock, self._conn.cursor() as cur:
+            cur.execute(f"SELECT COUNT(*) FROM {self._table} WHERE NOT sent AND NOT dead")
+            row = cur.fetchone()
+            return int(row[0]) if row else 0
+
+    def close(self) -> None:
+        with self._lock:
+            self._conn.close()