duckstring 0.4.0__tar.gz → 0.5.0__tar.gz

duckstring-0.5.0/CHANGELOG.md

@@ -0,0 +1,93 @@
+# Changelog
+
+Notable changes per release. Versions before 0.5.0 are recorded in the git history and the `v*` tags.
+
+## 0.5.0 — 2026-08-03
+
+The cloud release: a Catchment can now run its Ducks on AWS, keep its data on S3, and serve that data
+over standard wire protocols. Everything below is opt-in — a Catchment with no cloud configuration
+behaves exactly as it did in 0.4.0, and a `pond.toml` asking for remote compute still runs anywhere.
+
+### Cloud compute
+
+- **Duck launchers**: a Pond's Duck runs on the Catchment's own box (default), on **Fargate** (the
+  default remote backend — serverless containers, task-role IAM), or on **EC2** (the escape hatch, for
+  sizes and GPUs Fargate does not offer). Built-in **S/M/L/XL preset pools** resolve with zero setup.
+- **Duck Pools**: named remote-compute pools (`duckstring duck pool add`), with per-Pond compute
+  declared in `pond.toml` and overridable operationally (`duckstring duck set`).
+- **The Pool agent**: one shared machine per named Pool hosting many co-resident Ducks.
+- **Auto-relay**: a Catchment on a laptop behind NAT can run cloud Ducks with no manual tunnel — the
+  first remote spawn provisions a small always-reachable box and holds a reverse tunnel to it.
+- **Remote failures are observable**: boot output is teed to the console (EC2) / CloudWatch (Fargate)
+  and the tail is attached to the Pond's failure, so a Duck that dies before dialling back still says
+  why. Spawn-failure reasons (missing image, missing AMI, bad IAM) surface on the Pond rather than as a
+  generic crash.
+- Per-provider **startup grace** so a cold machine is not judged by the steady-state silence window.
+
+### Data plane on S3
+
+- A Catchment's data root can be an object store (`duckstring catchment settings --data-root s3://…`),
+  with an S3-compatible **endpoint override** for MinIO/Ceph/R2.
+- **Switch, adopt, or migrate** an existing plane, with a background copy and live progress.
+- **Local-first publish + async Persist**: a run publishes locally and mirrors durably in the
+  background, so compute never waits on the object store; `persisted_f` is the durable watermark and
+  freshness gating is Pool-aware.
+- **S3-resident state**: a merge base is read as a view rather than hydrated, and retention pruning is
+  floor-anchored rather than inferred from absence.
+
+### The Flock — over-envelope compute
+
+A comprehensive `pond.trickle(...)` recompute that exceeds the Duck's memory envelope can be dispatched
+to a serverless engine (**Athena** ships first), while the Duck keeps merge/diff/publish.
+
+**DuckDB is the authority.** Dispatch decides *where* work runs, never *what* is published, enforced
+structurally: an engine-owned allow-list of expressions proven equivalent on both engines (division and
+CAST are excluded — the two engines genuinely disagree), a `conform` step that casts the engine's result
+to DuckDB's own schema and rejects a differing column set, and degradation to local compute on any
+failure. Dispatch counters ride `/metrics`, since a silently degrading Flock is otherwise invisible.
+
+### Data serving
+
+- A sandboxed, warm, read-only query surface over published data: catalog = catchment, schema = pond.
+- **Postgres wire** and **Arrow Flight SQL** adapters, plus `/api/serve` and a CLI.
+- A **Catalog UI** unifying the query surface, with role-governed access.
+
+### Lineage
+
+- Table-level (observed per run), column-level (static per version, with a sqlglot upgrade for SQL
+  outputs), and row-level temporal provenance (`duckstring trace`).
+- **OpenLineage** events emitted on run completion for catalog integration.
+
+### dbt-mode Ponds
+
+Deploy a dbt project as a Pond with no `@ripple` code: each dbt model becomes a Ripple with its own
+freshness, failure and retry tracking, and `ref()` becomes the intra-Pond graph. Opt in with the
+`duckstring[dbt]` extra.
+
+### Correctness and operability
+
+- **Version contract**: a lossless widening of a column type is accepted (a data-dependent branch could
+  legitimately wedge a live Pond permanently); `reset-contract` is the escape hatch for a genuine
+  narrowing; contract failures carry a dedicated sub-reason end-to-end.
+- **Reads reject a stale local publish** the Catchment knows is behind, rather than serving it.
+- Pond Runs the Pond has moved past are closed rather than stranded at `running`.
+- A Pool machine that disappears is detected and relaunched, instead of wedging every Pond on it.
+- The **serving sandbox** no longer spills into the shared system temp — DuckDB exempts its temp
+  directory from the external-access lock, which on Linux exposed everything under `/tmp` to a
+  read-level user.
+- An idle Duck backs off instead of polling ten times a second.
+- Opt-in **re-notify cadence** for alert channels while a failure or freshness episode persists.
+- Incremental object-store egress (`mode=append`) mirrors the published collection per delivery.
+
+### Testing
+
+- The object-store data plane runs against a **real S3 API** in CI (MinIO, which rejects unsigned
+  requests — the class of bug moto cannot catch).
+- Real-data demo sets: **TPC-DS** and **GHArchive** (the latter offline against a committed fixture).
+- Python matrix, the dbt extra, real-Postgres egress, and frontend tests in CI.
+
+### Documentation
+
+- A full **AWS guide** (`guides/cloud.md`): the cloud-enable gate, the three IAM roles, security-group
+  asymmetry, the AMI Python constraint, the build-your-own-image policy, and the Flock's authority rules.
+- New guides for lineage, dbt-mode, and querying data.

{duckstring-0.4.0 → duckstring-0.5.0}/MANIFEST.in

@@ -1,5 +1,7 @@
 # The sdist ships the installable package only — keep internal design notes and the test suite
 # out of the PyPI source tarball (they're in the git repo for contributors).
+include CHANGELOG.md
+
 prune tests
 prune plans
 prune experiment

{duckstring-0.4.0/src/duckstring.egg-info → duckstring-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: duckstring
-Version: 0.4.0
+Version: 0.5.0
 Summary: Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
 Author-email: Duckstring <dev@duckstring.com>
 License:                                  Apache License
@@ -225,15 +225,29 @@ Requires-Dist: pyiceberg>=0.7
 Requires-Dist: pyarrow>=14
 Requires-Dist: pytz
 Provides-Extra: iceberg
+Provides-Extra: dbt
+Requires-Dist: dbt-duckdb>=1.9; extra == "dbt"
+Provides-Extra: lineage
+Requires-Dist: sqlglot>=23; extra == "lineage"
+Provides-Extra: aws
+Requires-Dist: s3fs>=2023.1; extra == "aws"
+Requires-Dist: boto3>=1.34; extra == "aws"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: moto[server]>=5.0; extra == "dev"
 Requires-Dist: pytest-timeout>=0.5; extra == "dev"
 Requires-Dist: ruff>=0.3; extra == "dev"
 Requires-Dist: mypy>=1.8; extra == "dev"
+Requires-Dist: dbt-duckdb>=1.9; extra == "dev"
+Requires-Dist: sqlglot>=23; extra == "dev"
+Requires-Dist: s3fs>=2023.1; extra == "dev"
+Requires-Dist: boto3>=1.34; extra == "dev"
+Requires-Dist: pg8000>=1.31; extra == "dev"
 Dynamic: license-file
 
 # Duckstring
-*There is no DAG.*
+
+Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
 
 Duckstring treats data transformations as software packages. Upstream dependencies are declared per Pond (unit operation), defining the DAG without the need for its direct management. 
 
@@ -297,6 +311,8 @@ Ponds execute on demand signals sent to an Outlet, in two flavours — **push**
 
 A Tide keeps an Outlet no staler than a bound (`duckstring trigger tide reports 1d`); a Wave keeps it as fresh as the pipeline can supply. See [Triggers](https://docs.duckstring.com/guides/triggers) for the full semantics.
 
+*There is no DAG.*
+
 ## Going further
 
 Full documentation lives at **[docs.duckstring.com](https://docs.duckstring.com)**:

{duckstring-0.4.0 → duckstring-0.5.0}/README.md

@@ -1,5 +1,6 @@
 # Duckstring
-*There is no DAG.*
+
+Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
 
 Duckstring treats data transformations as software packages. Upstream dependencies are declared per Pond (unit operation), defining the DAG without the need for its direct management. 
 
@@ -63,6 +64,8 @@ Ponds execute on demand signals sent to an Outlet, in two flavours — **push**
 
 A Tide keeps an Outlet no staler than a bound (`duckstring trigger tide reports 1d`); a Wave keeps it as fresh as the pipeline can supply. See [Triggers](https://docs.duckstring.com/guides/triggers) for the full semantics.
 
+*There is no DAG.*
+
 ## Going further
 
 Full documentation lives at **[docs.duckstring.com](https://docs.duckstring.com)**:

{duckstring-0.4.0 → duckstring-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "duckstring"
-version = "0.4.0"
+version = "0.5.0"
 description = "Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -35,11 +35,35 @@ dependencies = [
 [project.optional-dependencies]
 # Retained for back-compat: the Iceberg deps are in core now (Iceberg is the default plane).
 iceberg = []
+# dbt-mode Ponds (plans/dbt.md): deploy a dbt project as a Pond, each model a Ripple. Heavyweight and
+# opt-in — only pulled when a team actually runs dbt-mode. dbt-duckdb pulls dbt-core transitively.
+dbt = [
+  "dbt-duckdb>=1.9",
+]
+# Column lineage through .sql() escape hatches (plans/lineage.md Phase 3): a real SQL parser, opt-in --
+# without it those outputs are honestly "opaque", never guessed. Pure python, light.
+lineage = [
+  "sqlglot>=23",
+]
+# AWS / cloud (plans/cloud-config.md): the object-store data plane + egress (s3fs — fsspec's small-file
+# work; DuckDB does the bulk Parquet I/O over httpfs on its own), and the remote-compute launchers
+# (EC2/Fargate/relay) + the Athena Flock engine (boto3). Opt-in — a local/offline Catchment needs none.
+aws = [
+  "s3fs>=2023.1",
+  "boto3>=1.34",
+]
 dev = [
   "pytest>=8.0",
+  # A local S3 API, so the object-store data plane is a normal test rather than a live-AWS session.
+  "moto[server]>=5.0",
   "pytest-timeout>=0.5",
   "ruff>=0.3",
   "mypy>=1.8",
+  "dbt-duckdb>=1.9",
+  "sqlglot>=23",
+  "s3fs>=2023.1",
+  "boto3>=1.34",
+  "pg8000>=1.31",
 ]
 
 [tool.setuptools]

{duckstring-0.4.0 → duckstring-0.5.0}/src/duckstring/__init__.py

@@ -5,7 +5,6 @@ from .core import (
     Catchment,
     Pond,
     Puddle,
-    Ripple,
     puddle,
     ripple,
 )
@@ -19,7 +18,6 @@ __all__ = [
     "Catchment",
     "Pond",
     "Puddle",
-    "Ripple",
     "puddle",
     "ripple",
     "__version__",

{duckstring-0.4.0 → duckstring-0.5.0}/src/duckstring/alerts/event.py

@@ -14,6 +14,12 @@ from datetime import datetime, timezone
 # The event vocabulary. `freshness` is tick-driven (the SLA sweep); the rest are transition-driven.
 KNOWN_EVENTS = ("failure", "contract", "spout", "recovery", "freshness")
 
+# Machine-consumer kinds: valid subscriptions, but deliberately EXCLUDED from the `all` expansion —
+# an ops Slack channel subscribed to `all` must never receive raw catalog events. `openlineage` carries
+# a standard OpenLineage RunEvent per completed Pond Run (plans/lineage.md Phase 4); the webhook
+# notifier posts it verbatim (no Slack `text` wrapper).
+EXPLICIT_EVENTS = ("openlineage",)
+
 # Default severity per kind (a channel filters by kind, not severity — this is only for the payload/label).
 SEVERITY = {
     "failure": "error",
@@ -21,13 +27,15 @@ SEVERITY = {
     "spout": "error",
     "freshness": "warning",
     "recovery": "info",
+    "openlineage": "info",
 }
 
 
 def normalise_events(events: str | None) -> tuple[str, ...]:
     """Parse a channel's ``events`` CSV (or ``all``/empty) into a validated tuple of known kinds.
 
-    ``all`` / empty → every kind. Raises :class:`ValueError` naming an unknown kind."""
+    ``all`` / empty → every *notification* kind (machine-consumer kinds like ``openlineage`` need an
+    explicit subscription). Raises :class:`ValueError` naming an unknown kind."""
     if not events or events.strip().lower() == "all":
         return KNOWN_EVENTS
     out = []
@@ -35,8 +43,11 @@ def normalise_events(events: str | None) -> tuple[str, ...]:
         kind = raw.strip().lower()
         if not kind:
             continue
-        if kind not in KNOWN_EVENTS:
-            raise ValueError(f"unknown alert event {kind!r} — choose from {', '.join(KNOWN_EVENTS)} (or 'all')")
+        if kind not in KNOWN_EVENTS and kind not in EXPLICIT_EVENTS:
+            raise ValueError(
+                f"unknown alert event {kind!r} — choose from "
+                f"{', '.join(KNOWN_EVENTS + EXPLICIT_EVENTS)} (or 'all')"
+            )
         out.append(kind)
     if not out:
         return KNOWN_EVENTS

{duckstring-0.4.0 → duckstring-0.5.0}/src/duckstring/alerts/webhook.py

@@ -26,7 +26,10 @@ class WebhookNotifier:
         from ..egress import credentials
 
         url = credentials.resolve(self.dest.raw)  # resolve any ${env:}/${secret:} token — never logged
-        body = {"text": event.summary(), **event.to_payload()}  # `text` for Slack; the rest for generic receivers
+        if event.kind == "openlineage" and event.detail.get("event"):
+            body = event.detail["event"]  # a standard OpenLineage RunEvent — posted verbatim, no wrapper
+        else:
+            body = {"text": event.summary(), **event.to_payload()}  # `text` for Slack; the rest generic
         try:
             resp = httpx.post(url, json=body, timeout=_TIMEOUT)
             resp.raise_for_status()

{duckstring-0.4.0 → duckstring-0.5.0}/src/duckstring/catchment/app.py

@@ -12,7 +12,7 @@ from fastapi.staticfiles import StaticFiles
 from . import auth
 from .db import connect, ensure_identity, migrate
 from .driver import Driver
-from .launcher import NoopLauncher, SubprocessLauncher
+from .launcher import NoopLauncher, SubprocessLauncher, load_launcher_class
 from .routes import router
 
 _STATIC_DIR = Path(__file__).parent / "static"
@@ -41,16 +41,71 @@ async def _lifespan(app: FastAPI):
         # keys, so rotating those never disrupts running Ducks).
         # base_url None = unknown (the platform picked the bind address): the launcher defers spawns
         # until the dial-back middleware learns the address from the first request.
-        launcher = SubprocessLauncher(
-            app.state.root, base_url, token=app.state.duck_token, data_root=app.state.data_root
-        )
+        # DUCKSTRING_DUCK_LAUNCHER=module:Class (prereqs D6) swaps the implementation — same
+        # constructor contract, same interface; the Duck still dials back over the duck channel.
+        if os.environ.get("DUCKSTRING_DUCK_LAUNCHER"):
+            # A fully-custom launcher (D6) takes over entirely — same constructor contract as before.
+            launcher_cls = load_launcher_class(os.environ["DUCKSTRING_DUCK_LAUNCHER"])
+            launcher = launcher_cls(
+                app.state.root, base_url, token=app.state.duck_token, data_root=app.state.data_root
+            )
+        else:
+            # The dispatching launcher: `catchment`-targeted Ponds on this box (subprocess), pool/
+            # dedicated ones on EC2 — the same code whether the Catchment is local or hosted. The EC2
+            # backend is only built when cloud is enabled (remote data root + AWS creds); otherwise
+            # remote targets degrade to local (plans/cloud-config.md).
+            from .cloud_backends import build_remote_backends
+            from .launcher import DispatchingLauncher
+
+            local = SubprocessLauncher(
+                app.state.root, base_url, token=app.state.duck_token, data_root=app.state.data_root
+            )
+            # The remote backends (Fargate/EC2 + the shared dial-back) exist only when cloud is enabled
+            # (remote data root + AWS creds); otherwise remote targets degrade to local. The same builder
+            # re-attaches them live when creds / the data root change at runtime (see cloud_backends).
+            remotes, dialback = build_remote_backends(
+                app.state.root, base_url, app.state.duck_token, app.state.data_root, app.state.secret_store)
+            launcher = DispatchingLauncher(local, remotes, dialback=dialback, default_provider="fargate")
     driver = Driver(app.state.db, app.state.root, base_url, launcher, data_root=app.state.data_root)
     app.state.driver = driver
     app.state.launcher = launcher
 
+    # Boot credential check: the cloud gate is presence-based (deterministic, network-free), so a
+    # present-but-rejected key would show "enabled" while every remote launch fails. Validate the creds
+    # actually authenticate in the background — the result caches on app.state.cloud_creds (surfaced on
+    # /api/status so the UI can persistently warn) and logs a warning. Non-blocking, never fails boot.
+    # Validity is surfaced, not folded into the gate (a transient STS blip must not strand a running setup).
+    from .cloud_backends import refresh_credential_status
+    app.state.cloud_creds = None
+    refresh_credential_status(app.state, force=True)
+
     # Restore: resume any Pond Runs that were in flight when the Catchment last stopped.
     driver.resume_incomplete()
 
+    # Data-serving wire adapters (plans/data-serving.md): the Postgres wire (default) + Arrow Flight SQL,
+    # each on its own port when configured, sharing the sandboxed serving core. Put TLS + a network ACL
+    # in front for a hosted Catchment (the password is an API key).
+    app.state.serve_wires = []
+    pg_port = os.environ.get("DUCKSTRING_SERVE_PG_PORT")
+    if pg_port:
+        from .pg_wire import PgWireServer
+
+        pg = PgWireServer(driver, api_key=app.state.api_key,
+                          host=os.environ.get("DUCKSTRING_SERVE_HOST", "127.0.0.1"), port=int(pg_port))
+        pg.start()
+        app.state.serve_wires.append(pg)
+    flight_port = os.environ.get("DUCKSTRING_SERVE_FLIGHT_PORT")
+    if flight_port:
+        try:
+            from .flight_sql import FlightSqlServer
+
+            fl = FlightSqlServer(driver, api_key=app.state.api_key,
+                                 host=os.environ.get("DUCKSTRING_SERVE_HOST", "127.0.0.1"), port=int(flight_port))
+            fl.start()
+            app.state.serve_wires.append(fl)
+        except ImportError:
+            pass  # Flight needs pyarrow.flight; skip if unavailable
+
     from .alert_worker import run_alert_worker
     from .egress_worker import run_egress_worker
     from .poller import run_poller
@@ -97,6 +152,8 @@ async def _lifespan(app: FastAPI):
             except asyncio.CancelledError:
                 pass
         launcher.shutdown_all()
+        for wire in getattr(app.state, "serve_wires", []):
+            wire.stop()
         if app.state.data_lease is not None:
             from .data_lease import release_lease
 
@@ -134,6 +191,12 @@ def create_app(
     migrate(con)
     ensure_identity(con, name or os.environ.get("DUCKSTRING_CATCHMENT_NAME"))
 
+    # The data root can be a persisted Catchment setting (attached via the API after the Catchment is
+    # made — plans/cloud-config.md), behind an explicit argument and the env for platform hosting.
+    if not data_root:
+        from .cloud import DATA_ROOT_KEY, get_setting
+        data_root = get_setting(con, DATA_ROOT_KEY)
+
     # Writer lease on an external data root — refuse to start if a *different* live Catchment owns it (two
     # Catchments racing one lake's Iceberg catalog would dangle its pointer). A same-id restart reclaims
     # instantly; only engaged for an external DUCKSTRING_DATA_ROOT, so the local default is untouched.
@@ -167,6 +230,10 @@ def create_app(
     from .secrets import SecretStore
     app.state.secret_store = SecretStore(root)
     credentials.set_secret_provider(app.state.secret_store.get)
+    # Give the AWS_* secrets teeth: load them into the environment so botocore's chain uses them (the
+    # enable-by-secret cloud flow). Real env wins.
+    from . import cloud as _cloud
+    _cloud.load_aws_env(app.state.secret_store)
     # The address Ducks dial back to: explicit argument (the CLI passes its bind address), or the
     # environment, or None — unknown, because the host platform picks the bind address (e.g. Posit
     # Connect). When None it is learned from the first request's ASGI scope below.

{duckstring-0.4.0 → duckstring-0.5.0}/src/duckstring/catchment/auth.py

@@ -141,6 +141,23 @@ def get_principal(request: Request) -> Principal:
     return Principal(level=None, is_duck=False)
 
 
+def level_for_key(con: sqlite3.Connection | None, api_key: str | None, supplied: str) -> Level | None:
+    """The access :class:`Level` for a raw key — the reusable core of :func:`get_principal`, for
+    non-HTTP front doors (the pg / Flight wire servers, where the key arrives as a password). Open mode
+    (no keys configured) → FULL; a bad/absent key → None."""
+    if not auth_configured(con, api_key):
+        return Level.FULL
+    if supplied:
+        if con is not None:
+            h = hash_key(supplied)
+            for level, khash in _key_hashes(con).items():
+                if secrets.compare_digest(khash, h):
+                    return NAME_TO_LEVEL[level]
+        if api_key and secrets.compare_digest(supplied, api_key):
+            return Level.FULL
+    return None
+
+
 # ─── Route guards (declared on each route as `dependencies=[...]`) ────────────
 
 

duckstring-0.5.0/src/duckstring/catchment/cloud.py

@@ -0,0 +1,156 @@
+"""Catchment-level cloud config (plans/cloud-config.md, increment 2).
+
+The persisted ``catchment_setting`` key/value store (migration 021) plus the **cloud-enable gate**:
+remote compute needs shared object storage a remote box can read, so the gate is *S3 data root +
+AWS credentials present*. The data-plane target is a **persisted setting** (settable after the
+Catchment is made), but **set-once in practice** — switching it once data exists would strand that
+data, so the setter refuses a populated Catchment (a migration is deferred).
+"""
+
+from __future__ import annotations
+
+import functools
+import os
+import sqlite3
+
+DATA_ROOT_KEY = "data_root"
+_REMOTE_SCHEMES = ("s3", "gs")
+
+
+# ─── the generic key/value setting store ─────────────────────────────────────────
+
+def get_setting(con: sqlite3.Connection, key: str) -> str | None:
+    row = con.execute("SELECT value FROM catchment_setting WHERE key = ?", (key,)).fetchone()
+    return row[0] if row else None
+
+
+def set_setting(con: sqlite3.Connection, key: str, value: str | None) -> None:
+    if value is None:
+        con.execute("DELETE FROM catchment_setting WHERE key = ?", (key,))
+    else:
+        con.execute(
+            "INSERT INTO catchment_setting (key, value) VALUES (?, ?) "
+            "ON CONFLICT(key) DO UPDATE SET value = excluded.value",
+            (key, value),
+        )
+    con.commit()
+
+
+def all_settings(con: sqlite3.Connection) -> dict[str, str]:
+    return dict(con.execute("SELECT key, value FROM catchment_setting").fetchall())
+
+
+def has_published_data(con: sqlite3.Connection) -> bool:
+    """Whether the Catchment has ever published output — a successful Pond Run. The set-once guard
+    keys off this: while it is False the data root is freely settable; once True, switching it would
+    strand the published data, so the setter refuses."""
+    row = con.execute("SELECT 1 FROM pond_run WHERE status = 'success' LIMIT 1").fetchone()
+    return row is not None
+
+
+# ─── the cloud-enable gate ──────────────────────────────────────────────────────
+
+def is_remote(data_root: str | None) -> bool:
+    """A data root a remote box can read — an object store (s3://, gs://). A local path / None means
+    only Catchment Ducks are launchable (no shared storage)."""
+    return bool(data_root) and data_root.split("://", 1)[0].lower() in _REMOTE_SCHEMES
+
+
+def aws_configured(secret_store=None) -> bool:
+    """AWS credentials the control plane can use to launch/submit: env creds/profile/role, or an
+    ``AWS_*`` secret in the write-only store (worker boxes themselves use instance roles — creds here
+    are the control-plane half)."""
+    if any(os.environ.get(k) for k in ("AWS_ACCESS_KEY_ID", "AWS_PROFILE", "AWS_ROLE_ARN",
+                                       "AWS_WEB_IDENTITY_TOKEN_FILE")):
+        return True
+    if secret_store is not None:
+        try:
+            if any(n["name"].startswith("AWS_") for n in secret_store.names()):
+                return True
+        except Exception:
+            pass
+    # Fall back to the botocore credential chain (a `[default]` profile, SSO, an instance role) so a
+    # plain `aws configure` enables cloud — not only explicit env vars. Cached (process-lifetime).
+    return _chain_has_credentials()
+
+
+@functools.lru_cache(maxsize=1)
+def _chain_has_credentials() -> bool:
+    try:
+        import boto3
+        return boto3.Session().get_credentials() is not None
+    except Exception:
+        return False
+
+
+AWS_SECRET_PREFIX = "AWS_"
+
+
+def load_aws_env(secret_store=None) -> None:
+    """Inject any ``AWS_*`` secrets into the process environment so botocore's default credential chain
+    actually picks them up. ``aws_configured`` already *counts* an ``AWS_*`` secret as creds (the gate);
+    this gives that its teeth — every boto3 client is built off the ambient chain, so without this a
+    secret-stored key would flip the gate green but real launches would fail with no ambient creds.
+
+    Real environment variables win (``setdefault``), so an operator's explicit env is never overridden.
+    Called at startup and after a matching secret is set (see routes/secrets)."""
+    if secret_store is None:
+        return
+    try:
+        names = [n["name"] for n in secret_store.names() if n["name"].startswith(AWS_SECRET_PREFIX)]
+    except Exception:
+        return
+    changed = False
+    for name in names:
+        if name not in os.environ:
+            value = secret_store.get(name)
+            if value is not None:
+                os.environ[name] = value
+                changed = True
+    if changed:
+        _chain_has_credentials.cache_clear()
+
+
+def validate_credentials(region: str | None = None) -> str | None:
+    """A *live* credential check: None if the control-plane AWS creds authenticate (STS
+    GetCallerIdentity), else a short error string. Deliberately **separate** from the presence-based
+    ``aws_configured`` gate — the gate must stay deterministic and network-free (a transient STS blip
+    must never flip cloud off and strand a running setup), so validity is surfaced (a boot warning, the
+    Verify button), not folded into the gate. A region is required only to sign the request."""
+    region = region or os.environ.get("AWS_REGION") or os.environ.get("AWS_DEFAULT_REGION") or "us-east-1"
+    try:
+        import boto3
+        boto3.client("sts", region_name=region).get_caller_identity()
+        return None
+    except Exception as exc:
+        return str(exc) or exc.__class__.__name__
+
+
+def cloud_status(data_root: str | None, secret_store=None, cred_status: dict | None = None) -> dict:
+    """The gate + its reasons — surfaced on /api/status and the settings endpoint so the UI can grey
+    out remote-compute options and explain why. ``cred_status`` (the cached live-validation result, see
+    cloud_backends) adds ``creds_valid``/``creds_error`` so the UI can persistently warn when the gate is
+    enabled but the credentials don't authenticate (``creds_valid is None`` = not yet checked / gate off
+    → no warning)."""
+    from . import cloud_deploy
+
+    remote = is_remote(data_root)
+    aws = aws_configured(secret_store)
+    enabled = remote and aws
+    out = {
+        "data_root": data_root,
+        "data_root_remote": remote,
+        "aws_configured": aws,
+        "cloud_enabled": enabled,
+        # Provider readiness — Cloud enabled AND the provider's env deployment config is adequate to launch
+        # WITHOUT per-pool input. Fargate readiness is what gates the S/M/L/XL presets (they rely on the
+        # env); a per-pool/dedicated Duck can still carry its own config independently of these.
+        "fargate_enabled": bool(enabled and not cloud_deploy.missing_fields("fargate", None)),
+        "ec2_enabled": bool(enabled and not cloud_deploy.missing_fields("ec2", None)),
+        "creds_valid": None,
+        "creds_error": None,
+    }
+    if cred_status is not None:
+        out["creds_valid"] = cred_status.get("valid")
+        out["creds_error"] = cred_status.get("error")
+    return out