duckstring 0.3.0__tar.gz → 0.5.0__tar.gz

duckstring-0.5.0/CHANGELOG.md

@@ -0,0 +1,93 @@
+# Changelog
+
+Notable changes per release. Versions before 0.5.0 are recorded in the git history and the `v*` tags.
+
+## 0.5.0 — 2026-08-03
+
+The cloud release: a Catchment can now run its Ducks on AWS, keep its data on S3, and serve that data
+over standard wire protocols. Everything below is opt-in — a Catchment with no cloud configuration
+behaves exactly as it did in 0.4.0, and a `pond.toml` asking for remote compute still runs anywhere.
+
+### Cloud compute
+
+- **Duck launchers**: a Pond's Duck runs on the Catchment's own box (default), on **Fargate** (the
+  default remote backend — serverless containers, task-role IAM), or on **EC2** (the escape hatch, for
+  sizes and GPUs Fargate does not offer). Built-in **S/M/L/XL preset pools** resolve with zero setup.
+- **Duck Pools**: named remote-compute pools (`duckstring duck pool add`), with per-Pond compute
+  declared in `pond.toml` and overridable operationally (`duckstring duck set`).
+- **The Pool agent**: one shared machine per named Pool hosting many co-resident Ducks.
+- **Auto-relay**: a Catchment on a laptop behind NAT can run cloud Ducks with no manual tunnel — the
+  first remote spawn provisions a small always-reachable box and holds a reverse tunnel to it.
+- **Remote failures are observable**: boot output is teed to the console (EC2) / CloudWatch (Fargate)
+  and the tail is attached to the Pond's failure, so a Duck that dies before dialling back still says
+  why. Spawn-failure reasons (missing image, missing AMI, bad IAM) surface on the Pond rather than as a
+  generic crash.
+- Per-provider **startup grace** so a cold machine is not judged by the steady-state silence window.
+
+### Data plane on S3
+
+- A Catchment's data root can be an object store (`duckstring catchment settings --data-root s3://…`),
+  with an S3-compatible **endpoint override** for MinIO/Ceph/R2.
+- **Switch, adopt, or migrate** an existing plane, with a background copy and live progress.
+- **Local-first publish + async Persist**: a run publishes locally and mirrors durably in the
+  background, so compute never waits on the object store; `persisted_f` is the durable watermark and
+  freshness gating is Pool-aware.
+- **S3-resident state**: a merge base is read as a view rather than hydrated, and retention pruning is
+  floor-anchored rather than inferred from absence.
+
+### The Flock — over-envelope compute
+
+A comprehensive `pond.trickle(...)` recompute that exceeds the Duck's memory envelope can be dispatched
+to a serverless engine (**Athena** ships first), while the Duck keeps merge/diff/publish.
+
+**DuckDB is the authority.** Dispatch decides *where* work runs, never *what* is published, enforced
+structurally: an engine-owned allow-list of expressions proven equivalent on both engines (division and
+CAST are excluded — the two engines genuinely disagree), a `conform` step that casts the engine's result
+to DuckDB's own schema and rejects a differing column set, and degradation to local compute on any
+failure. Dispatch counters ride `/metrics`, since a silently degrading Flock is otherwise invisible.
+
+### Data serving
+
+- A sandboxed, warm, read-only query surface over published data: catalog = catchment, schema = pond.
+- **Postgres wire** and **Arrow Flight SQL** adapters, plus `/api/serve` and a CLI.
+- A **Catalog UI** unifying the query surface, with role-governed access.
+
+### Lineage
+
+- Table-level (observed per run), column-level (static per version, with a sqlglot upgrade for SQL
+  outputs), and row-level temporal provenance (`duckstring trace`).
+- **OpenLineage** events emitted on run completion for catalog integration.
+
+### dbt-mode Ponds
+
+Deploy a dbt project as a Pond with no `@ripple` code: each dbt model becomes a Ripple with its own
+freshness, failure and retry tracking, and `ref()` becomes the intra-Pond graph. Opt in with the
+`duckstring[dbt]` extra.
+
+### Correctness and operability
+
+- **Version contract**: a lossless widening of a column type is accepted (a data-dependent branch could
+  legitimately wedge a live Pond permanently); `reset-contract` is the escape hatch for a genuine
+  narrowing; contract failures carry a dedicated sub-reason end-to-end.
+- **Reads reject a stale local publish** the Catchment knows is behind, rather than serving it.
+- Pond Runs the Pond has moved past are closed rather than stranded at `running`.
+- A Pool machine that disappears is detected and relaunched, instead of wedging every Pond on it.
+- The **serving sandbox** no longer spills into the shared system temp — DuckDB exempts its temp
+  directory from the external-access lock, which on Linux exposed everything under `/tmp` to a
+  read-level user.
+- An idle Duck backs off instead of polling ten times a second.
+- Opt-in **re-notify cadence** for alert channels while a failure or freshness episode persists.
+- Incremental object-store egress (`mode=append`) mirrors the published collection per delivery.
+
+### Testing
+
+- The object-store data plane runs against a **real S3 API** in CI (MinIO, which rejects unsigned
+  requests — the class of bug moto cannot catch).
+- Real-data demo sets: **TPC-DS** and **GHArchive** (the latter offline against a committed fixture).
+- Python matrix, the dbt extra, real-Postgres egress, and frontend tests in CI.
+
+### Documentation
+
+- A full **AWS guide** (`guides/cloud.md`): the cloud-enable gate, the three IAM roles, security-group
+  asymmetry, the AMI Python constraint, the build-your-own-image policy, and the Flock's authority rules.
+- New guides for lineage, dbt-mode, and querying data.

{duckstring-0.3.0 → duckstring-0.5.0}/MANIFEST.in

@@ -1,5 +1,7 @@
 # The sdist ships the installable package only — keep internal design notes and the test suite
 # out of the PyPI source tarball (they're in the git repo for contributors).
+include CHANGELOG.md
+
 prune tests
 prune plans
 prune experiment

{duckstring-0.3.0/src/duckstring.egg-info → duckstring-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: duckstring
-Version: 0.3.0
+Version: 0.5.0
 Summary: Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
 Author-email: Duckstring <dev@duckstring.com>
 License:                                  Apache License
@@ -225,15 +225,29 @@ Requires-Dist: pyiceberg>=0.7
 Requires-Dist: pyarrow>=14
 Requires-Dist: pytz
 Provides-Extra: iceberg
+Provides-Extra: dbt
+Requires-Dist: dbt-duckdb>=1.9; extra == "dbt"
+Provides-Extra: lineage
+Requires-Dist: sqlglot>=23; extra == "lineage"
+Provides-Extra: aws
+Requires-Dist: s3fs>=2023.1; extra == "aws"
+Requires-Dist: boto3>=1.34; extra == "aws"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: moto[server]>=5.0; extra == "dev"
 Requires-Dist: pytest-timeout>=0.5; extra == "dev"
 Requires-Dist: ruff>=0.3; extra == "dev"
 Requires-Dist: mypy>=1.8; extra == "dev"
+Requires-Dist: dbt-duckdb>=1.9; extra == "dev"
+Requires-Dist: sqlglot>=23; extra == "dev"
+Requires-Dist: s3fs>=2023.1; extra == "dev"
+Requires-Dist: boto3>=1.34; extra == "dev"
+Requires-Dist: pg8000>=1.31; extra == "dev"
 Dynamic: license-file
 
 # Duckstring
-*There is no DAG.*
+
+Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
 
 Duckstring treats data transformations as software packages. Upstream dependencies are declared per Pond (unit operation), defining the DAG without the need for its direct management. 
 
@@ -297,6 +311,8 @@ Ponds execute on demand signals sent to an Outlet, in two flavours — **push**
 
 A Tide keeps an Outlet no staler than a bound (`duckstring trigger tide reports 1d`); a Wave keeps it as fresh as the pipeline can supply. See [Triggers](https://docs.duckstring.com/guides/triggers) for the full semantics.
 
+*There is no DAG.*
+
 ## Going further
 
 Full documentation lives at **[docs.duckstring.com](https://docs.duckstring.com)**:

{duckstring-0.3.0 → duckstring-0.5.0}/README.md

@@ -1,5 +1,6 @@
 # Duckstring
-*There is no DAG.*
+
+Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
 
 Duckstring treats data transformations as software packages. Upstream dependencies are declared per Pond (unit operation), defining the DAG without the need for its direct management. 
 
@@ -63,6 +64,8 @@ Ponds execute on demand signals sent to an Outlet, in two flavours — **push**
 
 A Tide keeps an Outlet no staler than a bound (`duckstring trigger tide reports 1d`); a Wave keeps it as fresh as the pipeline can supply. See [Triggers](https://docs.duckstring.com/guides/triggers) for the full semantics.
 
+*There is no DAG.*
+
 ## Going further
 
 Full documentation lives at **[docs.duckstring.com](https://docs.duckstring.com)**:

{duckstring-0.3.0 → duckstring-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "duckstring"
-version = "0.3.0"
+version = "0.5.0"
 description = "Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -35,11 +35,35 @@ dependencies = [
 [project.optional-dependencies]
 # Retained for back-compat: the Iceberg deps are in core now (Iceberg is the default plane).
 iceberg = []
+# dbt-mode Ponds (plans/dbt.md): deploy a dbt project as a Pond, each model a Ripple. Heavyweight and
+# opt-in — only pulled when a team actually runs dbt-mode. dbt-duckdb pulls dbt-core transitively.
+dbt = [
+  "dbt-duckdb>=1.9",
+]
+# Column lineage through .sql() escape hatches (plans/lineage.md Phase 3): a real SQL parser, opt-in --
+# without it those outputs are honestly "opaque", never guessed. Pure python, light.
+lineage = [
+  "sqlglot>=23",
+]
+# AWS / cloud (plans/cloud-config.md): the object-store data plane + egress (s3fs — fsspec's small-file
+# work; DuckDB does the bulk Parquet I/O over httpfs on its own), and the remote-compute launchers
+# (EC2/Fargate/relay) + the Athena Flock engine (boto3). Opt-in — a local/offline Catchment needs none.
+aws = [
+  "s3fs>=2023.1",
+  "boto3>=1.34",
+]
 dev = [
   "pytest>=8.0",
+  # A local S3 API, so the object-store data plane is a normal test rather than a live-AWS session.
+  "moto[server]>=5.0",
   "pytest-timeout>=0.5",
   "ruff>=0.3",
   "mypy>=1.8",
+  "dbt-duckdb>=1.9",
+  "sqlglot>=23",
+  "s3fs>=2023.1",
+  "boto3>=1.34",
+  "pg8000>=1.31",
 ]
 
 [tool.setuptools]

{duckstring-0.3.0 → duckstring-0.5.0}/src/duckstring/__init__.py

@@ -5,7 +5,6 @@ from .core import (
     Catchment,
     Pond,
     Puddle,
-    Ripple,
     puddle,
     ripple,
 )
@@ -19,7 +18,6 @@ __all__ = [
     "Catchment",
     "Pond",
     "Puddle",
-    "Ripple",
     "puddle",
     "ripple",
     "__version__",

duckstring-0.5.0/src/duckstring/alerts/__init__.py

@@ -0,0 +1,21 @@
+"""Alerts — failure & freshness notifications to external channels. See plans/alerts.md.
+
+The observability sibling of a Spout: operational config + a scheme-selected notifier seam +
+``${env:}``/``${secret:}`` credentials + an async delivery worker that never cascades a failure back into
+the engine. Alerting *observes* the state the engine already computes; it adds no orchestration state.
+"""
+
+from __future__ import annotations
+
+from .base import Notifier, NotifierError, get_notifier, parse_notifier_destination
+from .event import KNOWN_EVENTS, AlertEvent, normalise_events
+
+__all__ = [
+    "AlertEvent",
+    "KNOWN_EVENTS",
+    "Notifier",
+    "NotifierError",
+    "get_notifier",
+    "normalise_events",
+    "parse_notifier_destination",
+]

duckstring-0.5.0/src/duckstring/alerts/base.py

@@ -0,0 +1,94 @@
+"""The notifier seam (see plans/alerts.md) — mirrors :mod:`duckstring.egress.base`.
+
+A small, scheme-selected interface the alert worker delivers an :class:`AlertEvent` through. The channel
+destination is a URI whose scheme picks the notifier; ``get_notifier(destination)`` resolves it, exactly
+like ``get_egress``. Credentials travel inside the URI as ``${env:NAME}``/``${secret:NAME}`` references and
+are resolved only at send time (:mod:`duckstring.egress.credentials`) — never persisted or logged resolved.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from typing import Callable, Protocol, runtime_checkable
+from urllib.parse import urlparse
+
+from ..egress import credentials
+from .event import AlertEvent
+
+# Schemes a channel may target. http/https → a webhook (Slack-incoming-webhook compatible); mailto → SMTP.
+WEBHOOK_SCHEMES = {"http", "https"}
+EMAIL_SCHEMES = {"mailto"}
+KNOWN_SCHEMES = WEBHOOK_SCHEMES | EMAIL_SCHEMES
+
+
+class NotifierError(ValueError):
+    """An invalid channel destination, or a delivery/connectivity failure (sanitised — never a credential)."""
+
+
+@dataclass(frozen=True)
+class Destination:
+    scheme: str
+    raw: str  # the original URI, with ${...} references intact (resolved only at send time)
+
+
+def parse_notifier_destination(uri: str) -> Destination:
+    """Validate a channel destination URI: a known scheme + well-formed ``${...}`` references. Does **not**
+    resolve credentials (so a channel can be created before its secrets are present). Raises NotifierError."""
+    if not uri or not uri.strip():
+        raise NotifierError("destination must not be empty")
+    try:
+        credentials.references(uri)  # validates ${env:}/${secret:} syntax
+    except credentials.CredentialError as exc:
+        raise NotifierError(str(exc)) from exc
+    scheme = urlparse(uri).scheme.lower()
+    if not scheme:
+        raise NotifierError(f"destination {uri!r} has no scheme — expected e.g. https://…, mailto:…")
+    if scheme not in KNOWN_SCHEMES:
+        raise NotifierError(
+            f"unsupported alert destination scheme {scheme!r} — supported: {', '.join(sorted(KNOWN_SCHEMES))}"
+        )
+    return Destination(scheme=scheme, raw=uri)
+
+
+@runtime_checkable
+class Notifier(Protocol):
+    def send(self, event: AlertEvent) -> None:
+        """Deliver ``event`` to the destination. Raises :class:`NotifierError` (sanitised) on failure."""
+        ...
+
+    def test(self) -> None:
+        """Probe connectivity/credentials without delivering a real alert (the ``alert test`` command).
+        Returns on success; raises :class:`NotifierError` (sanitised) on failure."""
+        ...
+
+
+_REGISTRY: dict[str, Callable[[Destination], Notifier]] = {}
+
+
+def register(scheme: str, factory: Callable[[Destination], Notifier]) -> None:
+    _REGISTRY[scheme] = factory
+
+
+def get_notifier(destination: str) -> Notifier:
+    """Resolve the notifier for a channel destination by its scheme. Raises :class:`NotifierError` for an
+    unknown scheme, or a known scheme whose notifier is not built."""
+    dest = parse_notifier_destination(destination)
+    factory = _REGISTRY.get(dest.scheme)
+    if factory is None:
+        raise NotifierError(
+            f"notifier for scheme {dest.scheme!r} is not implemented yet (built: "
+            f"{', '.join(sorted(_REGISTRY)) or 'none'})"
+        )
+    return factory(dest)
+
+
+def _register_builtins() -> None:
+    from .email import EmailNotifier
+    from .webhook import WebhookNotifier
+
+    for driver in (WebhookNotifier, EmailNotifier):
+        for scheme in driver.SCHEMES:
+            register(scheme, driver)
+
+
+_register_builtins()

duckstring-0.5.0/src/duckstring/alerts/email.py

@@ -0,0 +1,106 @@
+"""The email notifier (``mailto:``) — the simplest floor.
+
+``mailto:ops@x.com,dev@x.com?smtp=host:587&from=alerts@x.com&user=${env:SMTP_USER}&password=${secret:SMTP_PASS}&tls=1``
+
+SMTP host/port/user/password/from come from the URI query, or the ``DUCKSTRING_SMTP_*`` environment as a
+fallback (so a Catchment can carry one SMTP config for every mailto channel). Credentials are resolved from
+``${env:}``/``${secret:}`` only at send time. Uses the stdlib ``smtplib`` — no new dependency.
+"""
+
+from __future__ import annotations
+
+import os
+from email.message import EmailMessage
+from urllib.parse import parse_qs, unquote, urlparse
+
+from .base import Destination, NotifierError
+from .event import AlertEvent
+
+_TIMEOUT = 20.0
+
+
+class EmailNotifier:
+    SCHEMES = ("mailto",)
+
+    def __init__(self, dest: Destination):
+        parsed = urlparse(dest.raw)
+        self.recipients = [r.strip() for r in unquote(parsed.path).split(",") if r.strip()]
+        if not self.recipients:
+            raise NotifierError("mailto: destination has no recipient — use mailto:you@example.com")
+        q = {k: v[0] for k, v in parse_qs(parsed.query).items()}
+        self._raw = q  # references intact; resolved at send time
+        self.smtp = q.get("smtp") or os.environ.get("DUCKSTRING_SMTP_HOST", "")
+        self.sender = q.get("from") or os.environ.get("DUCKSTRING_SMTP_FROM", "duckstring@localhost")
+        tls = q.get("tls")
+        self.tls = (tls not in ("0", "false", "no")) if tls is not None else \
+            (os.environ.get("DUCKSTRING_SMTP_TLS", "1") not in ("0", "false", "no"))
+        if not self.smtp:
+            raise NotifierError(
+                "mailto: destination needs an SMTP server — add ?smtp=host:port or set DUCKSTRING_SMTP_HOST"
+            )
+
+    def _host_port(self) -> tuple[str, int]:
+        host, _, port = self.smtp.partition(":")
+        return host, int(port) if port else 587
+
+    def _credentials(self) -> tuple[str | None, str | None]:
+        from ..egress import credentials
+
+        user = self._raw.get("user") or os.environ.get("DUCKSTRING_SMTP_USER")
+        password = self._raw.get("password") or os.environ.get("DUCKSTRING_SMTP_PASSWORD")
+        # Resolve ${env:}/${secret:} refs at call time — never persisted/logged resolved.
+        user = credentials.resolve(user) if user else None
+        password = credentials.resolve(password) if password else None
+        return user, password
+
+    def _connect(self):
+        import smtplib
+
+        host, port = self._host_port()
+        server = smtplib.SMTP(host, port, timeout=_TIMEOUT)
+        try:
+            server.ehlo()
+            if self.tls:
+                server.starttls()
+                server.ehlo()
+            user, password = self._credentials()
+            if user and password:
+                server.login(user, password)
+        except Exception:
+            server.close()
+            raise
+        return server
+
+    def send(self, event: AlertEvent) -> None:
+        msg = EmailMessage()
+        msg["Subject"] = event.summary()
+        msg["From"] = self.sender
+        msg["To"] = ", ".join(self.recipients)
+        lines = [event.message]
+        if event.pond:
+            lines.append(f"\nPond: {event.pond}")
+        if event.f:
+            lines.append(f"Freshness: {event.f}")
+        if event.detail:
+            for k, v in event.detail.items():
+                lines.append(f"{k}: {v}")
+        msg.set_content("\n".join(lines))
+        try:
+            server = self._connect()
+            try:
+                server.send_message(msg)
+            finally:
+                server.quit()
+        except NotifierError:
+            raise
+        except Exception as exc:  # noqa: BLE001 — sanitise: type only, never the credential/host detail
+            raise NotifierError(f"email delivery failed: {type(exc).__name__}") from None
+
+    def test(self) -> None:
+        try:
+            server = self._connect()  # connect + STARTTLS + login, deliver nothing
+            server.quit()
+        except NotifierError:
+            raise
+        except Exception as exc:  # noqa: BLE001
+            raise NotifierError(f"SMTP connection failed: {type(exc).__name__}") from None

duckstring-0.5.0/src/duckstring/alerts/event.py

@@ -0,0 +1,108 @@
+"""The alert event model + the event-kind vocabulary (see plans/alerts.md).
+
+An :class:`AlertEvent` is the rendered, **sanitised** payload a notifier delivers. It reuses what
+``/api/runs`` surfaces (error message, freshness, pond) but never a raw traceback — a channel destination
+can be third-party, and a traceback can leak paths/connection strings (the same concern behind the API's
+``_redact_tracebacks``). Keep it JSON-serialisable: it is stored verbatim in the ``alert_delivery`` outbox.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from datetime import datetime, timezone
+
+# The event vocabulary. `freshness` is tick-driven (the SLA sweep); the rest are transition-driven.
+KNOWN_EVENTS = ("failure", "contract", "spout", "recovery", "freshness")
+
+# Machine-consumer kinds: valid subscriptions, but deliberately EXCLUDED from the `all` expansion —
+# an ops Slack channel subscribed to `all` must never receive raw catalog events. `openlineage` carries
+# a standard OpenLineage RunEvent per completed Pond Run (plans/lineage.md Phase 4); the webhook
+# notifier posts it verbatim (no Slack `text` wrapper).
+EXPLICIT_EVENTS = ("openlineage",)
+
+# Default severity per kind (a channel filters by kind, not severity — this is only for the payload/label).
+SEVERITY = {
+    "failure": "error",
+    "contract": "error",
+    "spout": "error",
+    "freshness": "warning",
+    "recovery": "info",
+    "openlineage": "info",
+}
+
+
+def normalise_events(events: str | None) -> tuple[str, ...]:
+    """Parse a channel's ``events`` CSV (or ``all``/empty) into a validated tuple of known kinds.
+
+    ``all`` / empty → every *notification* kind (machine-consumer kinds like ``openlineage`` need an
+    explicit subscription). Raises :class:`ValueError` naming an unknown kind."""
+    if not events or events.strip().lower() == "all":
+        return KNOWN_EVENTS
+    out = []
+    for raw in events.split(","):
+        kind = raw.strip().lower()
+        if not kind:
+            continue
+        if kind not in KNOWN_EVENTS and kind not in EXPLICIT_EVENTS:
+            raise ValueError(
+                f"unknown alert event {kind!r} — choose from "
+                f"{', '.join(KNOWN_EVENTS + EXPLICIT_EVENTS)} (or 'all')"
+            )
+        out.append(kind)
+    if not out:
+        return KNOWN_EVENTS
+    return tuple(dict.fromkeys(out))  # de-duplicated, order preserved
+
+
+@dataclass
+class AlertEvent:
+    """A rendered notification. ``detail`` carries kind-specific extras (e.g. the blocked-downstream blast
+    radius for a failure); never put a traceback in it."""
+
+    kind: str
+    pond: str | None
+    title: str
+    message: str
+    severity: str = ""
+    f: str | None = None
+    catchment: str | None = None
+    ts: str = ""
+    detail: dict = field(default_factory=dict)
+
+    def __post_init__(self) -> None:
+        if not self.severity:
+            self.severity = SEVERITY.get(self.kind, "info")
+        if not self.ts:
+            self.ts = datetime.now(timezone.utc).isoformat()
+
+    def summary(self) -> str:
+        """A one-line human summary (the webhook ``text`` / the email subject line)."""
+        where = f" [{self.catchment}]" if self.catchment else ""
+        return f"{self.severity.upper()}{where}: {self.title}"
+
+    def to_payload(self) -> dict:
+        return {
+            "kind": self.kind,
+            "severity": self.severity,
+            "pond": self.pond,
+            "f": self.f,
+            "title": self.title,
+            "message": self.message,
+            "catchment": self.catchment,
+            "ts": self.ts,
+            "detail": self.detail,
+        }
+
+    @classmethod
+    def from_payload(cls, payload: dict) -> "AlertEvent":
+        return cls(
+            kind=payload.get("kind", ""),
+            pond=payload.get("pond"),
+            title=payload.get("title", ""),
+            message=payload.get("message", ""),
+            severity=payload.get("severity", ""),
+            f=payload.get("f"),
+            catchment=payload.get("catchment"),
+            ts=payload.get("ts", ""),
+            detail=payload.get("detail") or {},
+        )

duckstring-0.5.0/src/duckstring/alerts/webhook.py

@@ -0,0 +1,49 @@
+"""The webhook notifier (``http``/``https``) — the highest-leverage channel.
+
+POSTs a JSON body that is both a plain structured event **and** Slack-incoming-webhook compatible: a
+top-level ``text`` summary (which Slack renders, and a generic receiver can read) plus the full structured
+event. So one driver covers Slack, a generic webhook receiver, and (via a proxy) PagerDuty's Events API.
+Any credential/token in the URL is a ``${env:}``/``${secret:}`` reference, resolved only at send time.
+"""
+
+from __future__ import annotations
+
+from .base import Destination, NotifierError
+from .event import AlertEvent
+
+_TIMEOUT = 15.0
+
+
+class WebhookNotifier:
+    SCHEMES = ("http", "https")
+
+    def __init__(self, dest: Destination):
+        self.dest = dest
+
+    def _post(self, event: AlertEvent) -> None:
+        import httpx
+
+        from ..egress import credentials
+
+        url = credentials.resolve(self.dest.raw)  # resolve any ${env:}/${secret:} token — never logged
+        if event.kind == "openlineage" and event.detail.get("event"):
+            body = event.detail["event"]  # a standard OpenLineage RunEvent — posted verbatim, no wrapper
+        else:
+            body = {"text": event.summary(), **event.to_payload()}  # `text` for Slack; the rest generic
+        try:
+            resp = httpx.post(url, json=body, timeout=_TIMEOUT)
+            resp.raise_for_status()
+        except httpx.HTTPStatusError as exc:
+            # Sanitise: report status + reason, never the URL (it may carry a token in the path/query).
+            raise NotifierError(f"webhook returned {exc.response.status_code}") from None
+        except httpx.HTTPError as exc:
+            raise NotifierError(f"webhook delivery failed: {type(exc).__name__}") from None
+
+    def send(self, event: AlertEvent) -> None:
+        self._post(event)
+
+    def test(self) -> None:
+        self._post(AlertEvent(
+            kind="recovery", pond=None, title="Duckstring alert channel test",
+            message="This is a test notification — your alert channel is configured correctly.",
+        ))

duckstring-0.5.0/src/duckstring/catchment/alert_worker.py

@@ -0,0 +1,58 @@
+"""The alert worker — delivers queued notifications to their channels. See plans/alerts.md.
+
+One async task in the Catchment process (outbound I/O, the exact shape of the egress worker): each pass it
+drains the pending ``alert_delivery`` rows the engine enqueued (:meth:`Driver.take_alert_deliveries`),
+resolves each channel's notifier by scheme, ``send``s it in a threadpool with a per-send timeout, and marks
+the row sent (:meth:`Driver.mark_delivery_sent`) or bumps attempts / parks it failed at the cap
+(:meth:`Driver.mark_delivery_failed`).
+
+**A send failure never propagates into the engine** — it is recorded on the delivery row (auditable via
+``alert log``) and retried on the next tick until it succeeds or hits ``MAX_ATTEMPTS``. A permanently-broken
+channel therefore stops retrying but leaves a visible failed row, never a Pond failure.
+"""
+
+from __future__ import annotations
+
+import asyncio
+
+from fastapi.concurrency import run_in_threadpool
+
+_RECONCILE_INTERVAL = 5.0  # self-healing tick — retries pending deliveries a previous send left behind
+_PER_SEND_TIMEOUT = 30.0   # ceiling on one delivery, so a slow channel can't starve the others
+MAX_ATTEMPTS = 6           # after this many failed sends, park the delivery 'failed' (stop retrying)
+
+
+def _deliver(destination: str, payload: dict) -> None:
+    """Send one notification (blocking — runs in the thread pool). Raises on any failure."""
+    from ..alerts import AlertEvent, get_notifier
+
+    notifier = get_notifier(destination)  # resolves the scheme's driver (+ validates the URI)
+    notifier.send(AlertEvent.from_payload(payload))
+
+
+async def _drain(driver) -> None:
+    for row in driver.take_alert_deliveries():
+        try:
+            await asyncio.wait_for(
+                run_in_threadpool(_deliver, row["destination"], row["payload"]),
+                timeout=_PER_SEND_TIMEOUT,
+            )
+        except Exception as exc:  # noqa: BLE001 — any delivery error is recorded, never raised into the engine
+            driver.mark_delivery_failed(row["id"], f"{type(exc).__name__}: {exc}", MAX_ATTEMPTS)
+        else:
+            driver.mark_delivery_sent(row["id"])
+
+
+async def run_alert_worker(driver, wake: asyncio.Event) -> None:
+    """Drain queued alert deliveries on each wake (a delivery was enqueued) or the reconcile tick.
+    Cancelled on shutdown."""
+    while True:
+        try:
+            await asyncio.wait_for(wake.wait(), timeout=_RECONCILE_INTERVAL)
+        except asyncio.TimeoutError:
+            pass  # periodic reconcile — retry anything still pending
+        wake.clear()
+        try:
+            await _drain(driver)
+        except Exception as exc:  # keep the loop alive
+            print(f"[catchment] alert worker error: {exc}", flush=True)