duckstring 0.6.1__tar.gz → 0.7.0__tar.gz

{duckstring-0.6.1 → duckstring-0.7.0}/CHANGELOG.md

@@ -2,6 +2,186 @@
 
 Notable changes per release. Versions before 0.5.0 are recorded in the git history and the `v*` tags.
 
+## 0.7.0 — 2026-10-22
+
+A single trigger, Refresh, replaces Tap, Wave, Pulse and Tide, and a Tap is now an app's access point to a
+Pond. Also DuckDB 2.0 support and Spouts that deliver to a DuckLake. The old trigger commands and routes are
+removed with no compatibility period, so read each section's Upgrading notes. Needs `duckstring-trickle` 0.2.1.
+
+### One trigger: Refresh
+
+Tap, Wave, Pulse and Tide are replaced by a single trigger, **Refresh**, which asks for a Pond's data to be at
+least as fresh as now. A recurring Refresh repeats it:
+
+- `duckstring refresh add reports --every 1d`: a day after the later of the last request and the last refresh of
+  the data (the old Tide).
+- `duckstring refresh add reports --every 1d --start 06:00 [--on MON,TUE,WED,THU,FRI] [--until T]`: at set times,
+  on a window's grid, whatever else refreshes the Pond. New. A Catchment that was down asks once, for the latest
+  occurrence, when it returns.
+- `duckstring refresh add reports --continuous` (or `--every 0s`): as fresh as the pipeline allows, paced to its
+  slowest step (the old Wave).
+
+`duckstring refresh now reports` refreshes once (the old Pulse). A Pond can have any number of scheduled Refreshes and
+one `--every` or continuous Refresh; `refresh ls` lists them with when each next asks, and `refresh rm` removes
+one. In the web UI, a Pond's trigger buttons became a Refresh section (Refresh Now, and Recurring Refresh for
+Continuous, Every N and schedules), and its recurring Refreshes and Taps are listed in a strip at the bottom of its
+box on the canvas.
+
+Upgrading:
+
+- **Existing Waves become continuous Refreshes and Tides become `--every` Refreshes** (migration `031`).
+- **`POST /api/ponds/{name}/refresh` changed meaning.** It used to mark a Pond for a rebuild; it now refreshes the
+  Pond (at `demand` level). The rebuild flag moved to `POST /api/ponds/{name}/rebuild`, and the command to
+  `duckstring control rebuild` (and the batch operation `refresh` to `rebuild`).
+  Scripts that called `/refresh` for a rebuild need updating.
+- **Pull is gone from the engine.** A Tap's "anything newer" request no longer exists: a Refresh always asks for
+  data as of now. Wake now runs once if the Sources are fresher at that moment, rather than waiting for them.
+- `duckstring trigger window POND add|list|remove` is now `duckstring window add|ls|rm POND`.
+- **The old commands and routes are removed, with no compatibility period**: `duckstring trigger …`, `control
+  refresh`, `do --refresh`, and `POST /tap`, `/pulse`, `/wave`, `/tide` and `/untrigger`. Scripts need the new ones.
+- **Both ends of a duct need this release.** A Draw sends no requests to an upstream Catchment older than Refresh
+  (there, `/refresh` would mark the Pond for a rebuild) and logs that it needs upgrading.
+- **`catchment open --tap-on-get` is removed** (it went with Tap, and fired on every reader, including the web
+  UI's data viewer). An open Pond stays open. Taps, below, replace it.
+- `/api/status` lists each Pond's `refreshes` (with `next_due`), `demand_f` and `rebuild_pending`, in place of
+  `trigger`, `has_pull`, `target_f` and `refresh_pending`.
+- The Orchestration Theory page was removed from the docs; the Scheduling guide and the `refresh` and `window`
+  CLI reference cover what it said that users need.
+
+### Taps: freshness that follows usage
+
+A **Tap** is an access point on a Pond for one app: `duckstring refresh tap add reports --name dashboard --fresh-within
+5m`. It has its own key, shown once, which reads the Pond's exposed tables (as a read-level key does) over HTTP
+(`/api/query`, `/api/serve/query`), the Postgres wire or Arrow Flight, and is refused everywhere else. A query through it sends the Pond a Refresh when the newest request on the Pond, or the
+data, is at least `--fresh-within` old (the `--every` clock, checked per query; `0`, the default, refreshes on every
+query unless a request is already waiting), so the pipeline keeps up while the app is in use and goes quiet when it
+isn't. A query is always answered at once from what is published. Queries with any other key never send demand.
+
+- `refresh tap ls|rm|rotate POND`; the routes `/api/ponds/{name}/taps` (full). The web UI lists a Pond's Taps at the
+  foot of its Refresh section, adds them in a modal, and shows a Tap's details, Rotate and Delete when clicked.
+- `/metrics` gains `duckstring_tap_queries_total` and `duckstring_tap_refreshes_total`; `/api/status` lists each
+  Pond's `taps` (never keys).
+- **`GET /api/ponds/{name}/freshness`** says how fresh a Pond's data is: `freshness` (the point in time it
+  reflects), `changed` (when its output last changed), `refreshing`, and when each table and Object last changed. A
+  Tap's key can call it for its own Pond, and `Catchment.freshness()` wraps it. It replaces a route of the same name
+  that the HTTP reference described as the Pond's freshness but that listed a Trickle table's run freshnesses for
+  the data viewer; that list moved to `/api/ponds/{name}/tables/{table}/runs`.
+- The Python `Catchment` client takes a Tap's key as `api_key`, and then needs no `pond`.
+- The demo `reports` exposes `monthly_summary` (`[serve] tables`), so a Tap on it works out of the box.
+- Migration `032` (`pond_tap`).
+
+### The CLI, regrouped
+
+The top level goes from 21 commands to 13: `status`, `query`, `get`, `refresh`, `window`, `control`, `data`, `pond`,
+`catchment`, `duck`, `spout`, `secret` and `alert`. Tables and Objects are named `POND.NAME` everywhere, as in the
+Python API. No old forms are kept:
+
+| Before | Now |
+|---|---|
+| `query POND [TABLE] [--sql SQL]` | `query POND.TABLE`, or `query SQL [--pond POND]` |
+| `serve query SQL` | `query SQL` (with `--csv`, `--json` and `--parquet` too) |
+| `get POND TABLE [--path]`, `get-object POND NAME [--out]` | `get POND.NAME [--out]` |
+| `objects POND`, `serve status` | `data ls [POND]` (with each table's exposure and last change) |
+| `delete-table POND TABLE`, `delete-object POND NAME` | `data delete POND.NAME` |
+| `serve expose POND TABLE` | `data expose POND.TABLE` |
+| `lineage [POND] [--table T]`, `trace POND.TABLE` | `data lineage [POND[.TABLE]]`, `data trace POND.TABLE` |
+| `serve promote POND` | `pond promote POND` |
+| `puddle ls\|show\|query` | `pond puddle ls\|show\|query` |
+| `catchment open\|close` | `catchment duct open\|close` |
+| `do PONDS --kill …` | `control kill PONDS …`: every `control` verb takes several Ponds and `--all`, `--tree`, `--between`, `--downstream`; `--wipe` is `control wipe`, `--remove` is `pond remove` |
+
+`POST /api/serve/query` takes a `format` (`csv`, `json` or `parquet`) to return a file.
+
+### Orchestration
+
+- **Requests that arrive faster than the pipeline can serve them are paced to its slowest step** (the throttle),
+  always on, with no setting. A step that receives two requests during one run is held there, and one request goes
+  upstream each time it starts a new run.
+- **A Refresh no longer refreshes an optional Source after its first build.** Demand propagates only to a Pond's
+  required Sources: a Pond never waits for an optional Source, so asking it ran it for nothing. An optional Source
+  that has never run still gets one request, so its Sink has output to read, and a Pond whose Sources are all
+  optional still asks every one of them. If an optional Source needs a cadence of its own, give it its own Refresh.
+- **A repeat request for the freshness a Pond is already running is ignored.** It used to start a second
+  Run at the same freshness, which the no-change skip then completed as a pass, ending the Run in flight
+  early. A duct re-sending its downstream's request could cause it.
+- **A join's Sources read together under saturation.** When some of a Pond's Sources are throttled and others
+  aren't, its requests reach the faster Sources only once the throttled ones start their next run, so a join
+  isn't held back by a fast Source that read early.
+- **A Pond already throttled through another Refresh's Source still sends its first request.** It used to hold
+  that request until its own first completion, which the request itself was needed for.
+- **Sleep keeps other Ponds' demand.** Sleeping a Pond (or a path with `--upstream`) clears its own demand, but a
+  pending request from another Pond it was serving is now sent to it again; before, that request stranded.
+- **Demand is no longer lost at a halted Source.** A Pond that stops being blocked (after Clear, a recovery or a
+  redeploy) re-sends its newest unmet request upstream. Before, a request waiting on a Source whose Run had failed
+  or been killed stranded after Clear until something else demanded the path.
+
+### Redeploys and contract failures
+
+- **A redeploy takes effect while a recurring Refresh keeps the Pond busy.** A warm Duck used to keep running the
+  code it started with until it went idle, which a continuous Refresh never let it do, so a fix redeploy changed
+  nothing. A Duck now says which version it runs when it asks for work; a Duck on a replaced version finishes the
+  Runs it holds and exits, and a Duck on the new version picks up the next Run.
+- **A Run in flight across a redeploy is recorded against the version that started it.** Its events were recorded
+  against the new version, which the database refused, and the Duck retried the refused event forever: the Pond
+  showed as running for good, and nothing it did afterwards was recorded.
+- **A contract violation fails the Pond.** The Duck reported the Run's last Ripple as a success before it found the
+  output broke the version's contract, so the Catchment counted the Run done and dropped the failure that followed:
+  the Pond never showed as failed, and its Sinks ran on output that was never published. A Duck from before this
+  release still reports the old way, so redeploy Ducks with the Catchment.
+
+### DuckDB 2.0
+
+Duckstring and Trickle support DuckDB 2.0 as well as 1.5 (`duckdb>=1.5,<3`). DuckDB 2.0 needs Python 3.11 or
+newer, so on Python 3.10 DuckDB stays at 1.5.
+
+Upgrading:
+
+- **Ripple code may need a change.** In DuckDB 2.0 a query run with `execute()` holds its connection until its rows
+  are read to the end, so `pond.con.execute(...).fetchone()` followed by another statement fails, and relations
+  no longer have `fetchone()`. Read with `fetchall()`. A Pond whose lock keeps DuckDB 1.5 stays on 1.5 while the
+  Catchment runs 2.0, so Ponds can move over one at a time.
+- A column that is NULL in every row and has no other type (`SELECT NULL AS note`) is published as `INTEGER`, as
+  DuckDB 1.5 typed it. DuckDB 2.0 would give it its own NULL type, which refuses later values.
+- Tables written with `if_changed=True` republish once after the upgrade, since change fingerprints are tied to
+  the DuckDB version.
+- A Pond database written by DuckDB 2.0 can't be opened by 1.5. A Pond moved back to 1.5 rebuilds its database
+  from what it published, as it would after losing it.
+- On DuckDB 2.0 an integer `//` or `%` by zero raises. A Flock job on StarRocks now fails in the same place and
+  runs locally, rather than publishing NULL.
+- Trickle 0.2.1 is required.
+
+### DuckLake Spouts
+
+A Spout can deliver to a [DuckLake](https://ducklake.select):
+`duckstring spout add reports --to 'ducklake:sqlite:/srv/lake/meta.sqlite?data_path=s3://lakehouse/files/'`. The
+destination is DuckDB's own attach string for the lake (SQLite, Postgres or DuckDB catalogs), with `data_path`,
+`schema` and the data path's credentials as options. Any table can be delivered: a merge Trickle by its primary
+key, an append Trickle's new rows, a plain table replaced whole. A run's tables go in one transaction, so the
+lake gains one snapshot per delivery, authored `duckstring`. Delivery is exactly-once. Duckstring never
+migrates a lake's catalog, and leaves compaction and snapshot expiry to the lake's owner.
+
+Also for Spouts:
+
+- `spout resync` now re-delivers in full to Postgres too. Before, a destination that keeps its own record of
+  what it has received (Postgres) treated a resync as already delivered.
+- A Postgres table first filled by a full load no longer gains a `_duckstring_f` column that later deliveries
+  left empty, and a column a table gains is added to the destination table.
+- A Spout delivering all of a Pond's tables no longer delivers a merge Trickle's changelog as a table of its own.
+
+### Contract checks on remote Ducks
+
+A merge or append Trickle with an integer `SUM` column failed its version contract on the first run of a fresh
+Duck (every run on Fargate, EC2 or a pool, after its idle shutdown). `SUM` of an integer is `HUGEINT`, which
+Parquet stores as `DOUBLE`, so a Duck that rebuilt its database from what it had published held `DOUBLE` where the
+contract had recorded `HUGEINT`. The check now compares types as published. Before this release, contract
+failures found at that point were dropped without a trace, which hid it.
+
+### Documentation and Playground
+
+The Concepts and Guides have screenshots and recordings of the web UI and the Playground, and the landing page
+leads with the Playground. The Playground has a `chain` example (`?example=chain`), the 1 s → 3 s → 1 s chain the
+Orchestration page uses.
+
 ## 0.6.1 — 2026-10-08
 
 ### Performance

{duckstring-0.6.1/src/duckstring.egg-info → duckstring-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: duckstring
-Version: 0.6.1
+Version: 0.7.0
 Summary: An open source data engineering platform for DuckDB: versioned transformations, pull orchestration, incremental processing and a built-in catalog.
 Author-email: Duckstring <dev@duckstring.com>
 License:                                  Apache License
@@ -212,7 +212,8 @@ Project-URL: Playground, https://playground.duckstring.com
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: duckdb<2,>=1.5
+Requires-Dist: duckdb<3,>=1.5
+Requires-Dist: duckstring-trickle<0.3,>=0.2.1
 Requires-Dist: rich>=13.7
 Requires-Dist: typer>=0.12
 Requires-Dist: fastapi>=0.111
@@ -232,6 +233,10 @@ Requires-Dist: sqlglot>=23; extra == "lineage"
 Provides-Extra: aws
 Requires-Dist: s3fs>=2023.1; extra == "aws"
 Requires-Dist: boto3>=1.34; extra == "aws"
+Requires-Dist: sqlglot<31,>=30.21; extra == "aws"
+Provides-Extra: starrocks
+Requires-Dist: pymysql>=1.1; extra == "starrocks"
+Requires-Dist: sqlglot<31,>=30.21; extra == "starrocks"
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0; extra == "dev"
 Requires-Dist: moto[server]>=5.0; extra == "dev"
@@ -239,10 +244,11 @@ Requires-Dist: pytest-timeout>=0.5; extra == "dev"
 Requires-Dist: ruff>=0.3; extra == "dev"
 Requires-Dist: mypy>=1.8; extra == "dev"
 Requires-Dist: dbt-duckdb>=1.9; extra == "dev"
-Requires-Dist: sqlglot>=23; extra == "dev"
+Requires-Dist: sqlglot<31,>=30.21; extra == "dev"
 Requires-Dist: s3fs>=2023.1; extra == "dev"
 Requires-Dist: boto3>=1.34; extra == "dev"
 Requires-Dist: pg8000>=1.31; extra == "dev"
+Requires-Dist: pymysql>=1.1; extra == "dev"
 Dynamic: license-file
 
 # Duckstring
@@ -274,9 +280,13 @@ The pipeline follows from those declarations, so it never has to be drawn by han
 Schedules are set on the Ponds whose output is actually used, at the end of the pipeline. Duckstring works back from there, running each upstream Pond only as often as something downstream needs it, so a path nobody consumes never runs. A daily job is one command:
 
 ```bash
-duckstring trigger tide reports 1d
+duckstring refresh add reports --every 1d
 ```
 
+Data can also stay fresh just because it's being used: a [Tap](https://docs.duckstring.com/guides/serving_apps_with_taps) gives an app its own key, and its queries refresh the Pond when the data is older than you allow, so nothing runs while the app is idle.
+
+The [Playground](https://playground.duckstring.com) runs the demo pipeline in your browser, with a short guided tour of how it's scheduled.
+
 ## Incremental processing
 
 A *Trickle* stores a table's changes as a Z-set: rows weighted +1 when added and -1 when removed. Joins and aggregations written with the Trickle builder recompute only the keys that changed since the last run, and handle updates and deletes correctly across any shape of join:
@@ -325,8 +335,8 @@ This starts a local Catchment, with its web UI at http://127.0.0.1:7474. Leave i
 mkdir demo && cd demo
 duckstring pond demo                       # transactions, products → sales → reports
 duckstring pond deploy --all --yes
-duckstring trigger pulse reports           # run the pipeline once, from the end
-duckstring query reports monthly_summary
+duckstring refresh now reports                 # run the pipeline once, from the end
+duckstring query reports.monthly_summary
 ```
 
 The [Quickstart](https://docs.duckstring.com/quickstart) goes through the same steps with explanations.
@@ -343,7 +353,7 @@ The [Quickstart](https://docs.duckstring.com/quickstart) goes through the same s
 
 ## Documentation
 
-[docs.duckstring.com](https://docs.duckstring.com) has the concepts, task guides, and a reference for the CLI, Python API, `pond.toml` and HTTP API. To try pull orchestration without installing anything, use the [Playground](https://playground.duckstring.com).
+[docs.duckstring.com](https://docs.duckstring.com) has the concepts, task guides, and a reference for the CLI, Python API, `pond.toml` and HTTP API. To try pull orchestration without installing anything, the [Playground](https://playground.duckstring.com) simulates it in the browser, with a guided tour.
 
 ## Status
 

{duckstring-0.6.1 → duckstring-0.7.0}/README.md

@@ -27,9 +27,13 @@ The pipeline follows from those declarations, so it never has to be drawn by han
 Schedules are set on the Ponds whose output is actually used, at the end of the pipeline. Duckstring works back from there, running each upstream Pond only as often as something downstream needs it, so a path nobody consumes never runs. A daily job is one command:
 
 ```bash
-duckstring trigger tide reports 1d
+duckstring refresh add reports --every 1d
 ```
 
+Data can also stay fresh just because it's being used: a [Tap](https://docs.duckstring.com/guides/serving_apps_with_taps) gives an app its own key, and its queries refresh the Pond when the data is older than you allow, so nothing runs while the app is idle.
+
+The [Playground](https://playground.duckstring.com) runs the demo pipeline in your browser, with a short guided tour of how it's scheduled.
+
 ## Incremental processing
 
 A *Trickle* stores a table's changes as a Z-set: rows weighted +1 when added and -1 when removed. Joins and aggregations written with the Trickle builder recompute only the keys that changed since the last run, and handle updates and deletes correctly across any shape of join:
@@ -78,8 +82,8 @@ This starts a local Catchment, with its web UI at http://127.0.0.1:7474. Leave i
 mkdir demo && cd demo
 duckstring pond demo                       # transactions, products → sales → reports
 duckstring pond deploy --all --yes
-duckstring trigger pulse reports           # run the pipeline once, from the end
-duckstring query reports monthly_summary
+duckstring refresh now reports                 # run the pipeline once, from the end
+duckstring query reports.monthly_summary
 ```
 
 The [Quickstart](https://docs.duckstring.com/quickstart) goes through the same steps with explanations.
@@ -96,7 +100,7 @@ The [Quickstart](https://docs.duckstring.com/quickstart) goes through the same s
 
 ## Documentation
 
-[docs.duckstring.com](https://docs.duckstring.com) has the concepts, task guides, and a reference for the CLI, Python API, `pond.toml` and HTTP API. To try pull orchestration without installing anything, use the [Playground](https://playground.duckstring.com).
+[docs.duckstring.com](https://docs.duckstring.com) has the concepts, task guides, and a reference for the CLI, Python API, `pond.toml` and HTTP API. To try pull orchestration without installing anything, the [Playground](https://playground.duckstring.com) simulates it in the browser, with a guided tour.
 
 ## Status
 

{duckstring-0.6.1 → duckstring-0.7.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "duckstring"
-version = "0.6.1"
+version = "0.7.0"
 description = "An open source data engineering platform for DuckDB: versioned transformations, pull orchestration, incremental processing and a built-in catalog."
 readme = "README.md"
 requires-python = ">=3.10"
@@ -10,10 +10,13 @@ authors = [
 ]
 
 dependencies = [
-  # The tested range: CI runs the suite on the floor and the latest release, and against the pre-release
-  # in an allowed-to-fail job. Raise the ceiling once the suite passes on the next major's release
-  # (plans/duckdb-2-ready.md). A Pond environment gets this Duckstring, so these bounds hold there too.
-  "duckdb>=1.5,<2",
+  # The tested range: CI runs the suite on the floor (1.5, an LTS) and the latest release, and against the
+  # pre-release in an allowed-to-fail job. Raise the ceiling once the suite passes on the next major's release
+  # (plans/duckdb-2.md). A Pond environment gets this Duckstring, so these bounds hold there too.
+  "duckdb>=1.5,<3",
+  # The incremental engine (packages/trickle), released separately. A compatible range: while it's 0.x a
+  # minor release can change its API or storage format, so Duckstring moves to one deliberately.
+  "duckstring-trickle>=0.2.1,<0.3",
   "rich>=13.7",
   "typer>=0.12",
   "fastapi>=0.111",
@@ -48,10 +51,19 @@ lineage = [
 ]
 # AWS / cloud (plans/cloud-config.md): the object-store data plane + egress (s3fs — fsspec's small-file
 # work; DuckDB does the bulk Parquet I/O over httpfs on its own), and the remote-compute launchers
-# (EC2/Fargate/relay) + the Athena Flock engine (boto3). Opt-in — a local/offline Catchment needs none.
+# (EC2/Fargate/relay) + the Athena Flock engine (boto3, and sqlglot to translate its jobs). Opt-in: a
+# local/offline Catchment needs none.
 aws = [
   "s3fs>=2023.1",
   "boto3>=1.34",
+  "sqlglot>=30.21,<31",
+]
+# The StarRocks Flock engine (plans/flock-coverage.md): the MySQL-protocol client, and sqlglot to translate
+# jobs. sqlglot is pinned to one major because what an engine is trusted with is measured against its
+# translations: re-run tests/test_flock_starrocks_conformance.py before widening it.
+starrocks = [
+  "pymysql>=1.1",
+  "sqlglot>=30.21,<31",
 ]
 dev = [
   "pytest>=8.0",
@@ -61,10 +73,11 @@ dev = [
   "ruff>=0.3",
   "mypy>=1.8",
   "dbt-duckdb>=1.9",
-  "sqlglot>=23",
+  "sqlglot>=30.21,<31",
   "s3fs>=2023.1",
   "boto3>=1.34",
   "pg8000>=1.31",
+  "pymysql>=1.1",
 ]
 
 [tool.setuptools]
@@ -77,6 +90,12 @@ package-dir = { "" = "src" }
 [tool.setuptools.packages.find]
 where = ["src"]
 
+[tool.uv.workspace]
+members = ["packages/trickle"]
+
+[tool.uv.sources]
+duckstring-trickle = { workspace = true }
+
 [project.urls]
 Homepage = "https://duckstring.com"
 Documentation = "https://docs.duckstring.com"
@@ -88,7 +107,7 @@ duckstring = "duckstring.cli:main"
 ds = "duckstring.cli:main"
 
 [tool.pytest.ini_options]
-testpaths = ["tests"]
+testpaths = ["tests", "packages/trickle/tests"]
 # Some suites share fixtures by importing `tests.<module>`; put the repo root on the path so a bare
 # `pytest` (what CI runs) resolves them the same as `python -m pytest`.
 pythonpath = ["."]
@@ -104,3 +123,8 @@ extend-exclude = ["experiment"]
 [tool.ruff.lint]
 select = ["E", "F", "I", "B"]
 ignore = ["B008"]  # typer.Option/Argument in defaults is intentional
+
+[tool.ruff.lint.isort]
+# Explicit, because ruff versions disagree on modules that only resolve through an alias: newer ones class
+# duckstring.trickle.* (now packages/trickle, aliased) as third-party. `trickle` itself is third-party here.
+known-first-party = ["duckstring"]

{duckstring-0.6.1 → duckstring-0.7.0}/src/duckstring/catchment/app.py

@@ -9,6 +9,7 @@ from pathlib import Path
 from fastapi import FastAPI
 from fastapi.staticfiles import StaticFiles
 
+from ..trickle_io import first_row
 from . import auth
 from .db import connect, ensure_identity, migrate
 from .driver import Driver
@@ -19,7 +20,7 @@ _STATIC_DIR = Path(__file__).parent / "static"
 
 
 async def _scheduler(driver: Driver) -> None:
-    """Drive clock processes (Tide deadlines, window boundaries, Wave-on-idle) at next_wake."""
+    """Drive clock processes (recurring Refreshes, window boundaries) at next_wake."""
     while True:
         nw = driver.next_wake()
         now = datetime.now(timezone.utc)
@@ -205,7 +206,7 @@ def create_app(
         from ..storage import get_storage
         from .data_lease import acquire_lease
 
-        cid = con.execute("SELECT value FROM catchment_meta WHERE key = 'id'").fetchone()
+        cid = first_row(con.execute("SELECT value FROM catchment_meta WHERE key = 'id'"))
         owner_id = cid[0] if cid else "unknown"
         lease_store = get_storage(data_root)
         acquire_lease(lease_store, owner_id)  # raises LeaseConflict → the server does not start

{duckstring-0.6.1 → duckstring-0.7.0}/src/duckstring/catchment/auth.py

@@ -4,9 +4,14 @@ The single-key model is split into a **total-ordered** ladder — ``read ⊂ dem
 authorization check stays one integer comparison:
 
 - **read** — read & query data only (no ducts, no demand).
-- **demand** — read + create demand (tap/wave/pulse/tide) + connect a downstream duct.
+- **demand** — read + create demand (refresh now, recurring Refreshes) + connect a downstream duct.
 - **full** — everything: deploy, the control verbs, windows, ducts, key rotation.
 
+Besides the ladder, a **Tap** key (plans/taps.md) is a principal of its own: it names one Tap on one Pond line,
+can query only that Tap's tables, and its queries may send that Pond a Refresh. It is accepted only on the
+routes guarded by ``read_or_tap`` (the query routes) and refused (403) everywhere else. Tap keys are recognised
+in open mode too, since what they add is the Refresh, not access.
+
 Each route declares a *minimum* level via the ready-made dependencies ``read`` / ``demand`` / ``full``
 (or ``duck`` for the internal worker channel). A request's level is the matched key's level. Routes are
 **fail-closed**: :func:`audit_routes` runs at app construction and raises if any ``/api`` route (bar the
@@ -31,6 +36,7 @@ from enum import IntEnum
 
 from fastapi import Depends, HTTPException, Request
 
+from ..trickle_io import first_row
 from .db import locked
 
 # Routes that need no credential at all (parity with the old open `/api/health`).
@@ -79,7 +85,7 @@ def generate(con: sqlite3.Connection, levels: list[str] | None = None) -> dict[s
 def ensure_duck_token(con: sqlite3.Connection) -> str:
     """Mint the internal Duck dial-back token once (persisted in ``catchment_meta``), then return it.
     Stable across restarts so a Duck that outlived the Catchment keeps authenticating."""
-    row = con.execute("SELECT value FROM catchment_meta WHERE key = 'duck_token'").fetchone()
+    row = first_row(con.execute("SELECT value FROM catchment_meta WHERE key = 'duck_token'"))
     if row is not None:
         return row[0]
     token = secrets.token_urlsafe(32)
@@ -128,10 +134,26 @@ def auth_configured(con: sqlite3.Connection | None, api_key: str | None) -> bool
 # ─── Request-time resolution ─────────────────────────────────────────────────
 
 
+@dataclass(frozen=True)
+class TapScope:
+    """The Tap a key belongs to: its Pond line (a pond key) and its name."""
+    pond: str
+    name: str
+
+
 @dataclass
 class Principal:
     level: Level | None  # the access level of a user credential (None = unauthenticated)
     is_duck: bool        # the internal worker token (only valid on the duck channel)
+    tap: TapScope | None = None  # a Tap's key: queries only that Tap's tables (level is None)
+
+
+def tap_for_key(driver, supplied: str) -> TapScope | None:
+    """The Tap a raw key belongs to, if any (the Driver holds the Taps' key hashes in memory)."""
+    if not supplied or driver is None:
+        return None
+    lookup = getattr(driver, "tap_for_hash", None)
+    return lookup(hash_key(supplied)) if lookup is not None else None
 
 
 def _extract_token(request: Request) -> str:
@@ -152,6 +174,10 @@ def get_principal(request: Request) -> Principal:
     if duck_token and supplied and secrets.compare_digest(supplied, duck_token):
         return Principal(level=None, is_duck=True)
 
+    tap = tap_for_key(getattr(state, "driver", None), supplied)
+    if tap is not None:
+        return Principal(level=None, is_duck=False, tap=tap)
+
     if not auth_configured(con, api_key):
         return Principal(level=Level.FULL, is_duck=False)  # open mode — no credential required
 
@@ -167,6 +193,16 @@ def get_principal(request: Request) -> Principal:
     return Principal(level=None, is_duck=False)
 
 
+def principal_for_key(driver, api_key: str | None, supplied: str) -> Principal | None:
+    """The principal for a raw key on a non-HTTP front door (the pg / Flight wires): a Tap, else the key's
+    access level (open mode → FULL). None for a bad or absent key."""
+    tap = tap_for_key(driver, supplied)
+    if tap is not None:
+        return Principal(level=None, is_duck=False, tap=tap)
+    level = level_for_key(getattr(driver, "db", None), api_key, supplied)
+    return Principal(level=level, is_duck=False) if level is not None else None
+
+
 def level_for_key(con: sqlite3.Connection | None, api_key: str | None, supplied: str) -> Level | None:
     """The access :class:`Level` for a raw key — the reusable core of :func:`get_principal`, for
     non-HTTP front doors (the pg / Flight wire servers, where the key arrives as a password). Open mode
@@ -187,10 +223,14 @@ def level_for_key(con: sqlite3.Connection | None, api_key: str | None, supplied:
 # ─── Route guards (declared on each route as `dependencies=[...]`) ────────────
 
 
-def _level_guard(required: Level):
+def _level_guard(required: Level, allow_tap: bool = False):
     def dep(principal: Principal = Depends(get_principal)) -> Principal:
         if principal.is_duck:  # the worker token is for the duck channel only
             raise HTTPException(status_code=403, detail="endpoint not available to the worker token")
+        if principal.tap is not None:
+            if allow_tap:
+                return principal
+            raise HTTPException(status_code=403, detail="a Tap's key can only query its Tap")
         if principal.level is None:
             raise HTTPException(status_code=401, detail="Invalid or missing API key")
         if principal.level < required:
@@ -219,6 +259,7 @@ def _duck_guard():
 read = _level_guard(Level.READ)
 demand = _level_guard(Level.DEMAND)
 full = _level_guard(Level.FULL)
+read_or_tap = _level_guard(Level.READ, allow_tap=True)  # the query routes: a read key or a Tap's key
 duck = _duck_guard()
 
 

{duckstring-0.6.1 → duckstring-0.7.0}/src/duckstring/catchment/cloud.py

@@ -13,6 +13,8 @@ import functools
 import os
 import sqlite3
 
+from ..trickle_io import first_row
+
 DATA_ROOT_KEY = "data_root"
 _REMOTE_SCHEMES = ("s3", "gs")
 
@@ -20,7 +22,7 @@ _REMOTE_SCHEMES = ("s3", "gs")
 # ─── the generic key/value setting store ─────────────────────────────────────────
 
 def get_setting(con: sqlite3.Connection, key: str) -> str | None:
-    row = con.execute("SELECT value FROM catchment_setting WHERE key = ?", (key,)).fetchone()
+    row = first_row(con.execute("SELECT value FROM catchment_setting WHERE key = ?", (key,)))
     return row[0] if row else None
 
 
@@ -44,7 +46,7 @@ def has_published_data(con: sqlite3.Connection) -> bool:
     """Whether the Catchment has ever published output — a successful Pond Run. The set-once guard
     keys off this: while it is False the data root is freely settable; once True, switching it would
     strand the published data, so the setter refuses."""
-    row = con.execute("SELECT 1 FROM pond_run WHERE status = 'success' LIMIT 1").fetchone()
+    row = first_row(con.execute("SELECT 1 FROM pond_run WHERE status = 'success' LIMIT 1"))
     return row is not None
 
 

{duckstring-0.6.1 → duckstring-0.7.0}/src/duckstring/catchment/db.py

@@ -5,13 +5,15 @@ from contextlib import nullcontext
 from datetime import datetime, timezone
 from pathlib import Path
 
+from ..trickle_io import first_row
+
 _SCHEMA_DIR = Path(__file__).parent / "schema"
 
 
 def ensure_identity(con: sqlite3.Connection, name: str | None = None) -> None:
     """Mint this Catchment's stable id on first start (never changes), and set/refresh its optional
     display name. Call after migrate()."""
-    row = con.execute("SELECT value FROM catchment_meta WHERE key = 'id'").fetchone()
+    row = first_row(con.execute("SELECT value FROM catchment_meta WHERE key = 'id'"))
     if row is None:
         con.execute("INSERT INTO catchment_meta (key, value) VALUES ('id', ?)", (str(uuid.uuid4()),))
     if name: