duckstring 0.5.0__tar.gz → 0.6.0__tar.gz

duckstring-0.6.0/CHANGELOG.md

@@ -0,0 +1,197 @@
+# Changelog
+
+Notable changes per release. Versions before 0.5.0 are recorded in the git history and the `v*` tags.
+
+## 0.6.0 — 2026-10-05
+
+SQL Ripples, clustering, Pond environments and a simpler data plane. The Iceberg plane is gone: every
+Catchment now publishes versioned Parquet, which pins each run's Source reads and costs a third of the
+export time. See "Upgrading" at the end.
+
+### Building Ponds
+
+- **SQL Ripples**: a Ripple can be one `SELECT` in a file, declared in `pond.toml` under
+  `[ripples.NAME]` with its `parents`, the Source tables it `reads`, and how it writes (`overwrite`,
+  `merge` or `append`). Nothing is inferred from the SQL; each query is checked against its declaration
+  at deploy, so a forgotten parent is refused instead of silently reading the previous run's table. SQL
+  and Python Ripples mix freely (`@ripple(parents=...)` now takes names), and column lineage is
+  recorded for SQL Ripples too. `duckstring pond demo --sql` shows the default pipeline in SQL.
+- **Static tables**: files shipped with a Pond's code, declared under `[static.NAME]`, are read-only
+  tables every Ripple can query.
+- **Pond environments**: a Pond with a `pyproject.toml` and `uv.lock` runs in its own environment, built
+  at deploy on the Catchment and by cloud and Pool Ducks on their own machines. A missing or stale lock,
+  or code that won't import, now refuses the deploy with the reason (an import error used to register no
+  Ripples).
+- **`.pondignore`**: chooses which files a deploy uploads; by default `puddles/`, `.env` files, hidden
+  directories and caches stay behind.
+- Names containing dots can be referenced with backticks (``sales.`daily.v2` ``).
+- A merge Trickle's name is a view over its current state, so plain SQL in later Ripples reads what
+  `read_table` returns.
+
+### Performance
+
+- **Clustering**: `cluster_by` on `merge_table`, the builder's `.merge()`, `write_table` and SQL Ripples
+  orders a table for reads that filter on those columns. One column sorts; two or more are interleaved
+  with a rank-Morton order (each column's quantile ranks, bit-interleaved), so every listed column
+  prunes, skewed ones included. A merge table's compacted base is now ordered by its primary key by
+  default instead of by freshness, which no read could use.
+- A merge table's warm tier is read where it's published instead of being copied into a fresh Duck's
+  database, and is no longer also re-published in full on every export.
+
+### Data plane
+
+- **Versioned overwrite tables**: each run publishes a plain table as a new immutable version, and a
+  Sink run reads the version its Source had published when the run started, even if the Source publishes
+  again meanwhile. Superseded versions are kept only while a running Sink still reads them.
+- **The Iceberg plane is removed**, with `pyiceberg`, `DUCKSTRING_DATA_PLANE` and
+  `DUCKSTRING_ICEBERG_KEEP_SNAPSHOTS`. It cost about three times the export time, more than twice the
+  memory and over three times the disk of the Parquet plane, for interop no external engine could use.
+- A registry DuckDB can't open (written by a newer DuckDB, or corrupt) is set aside and rebuilt from
+  published state instead of stopping the Duck.
+
+### Cloud and egress
+
+- Flock settings and engine credentials travel with each run, so Fargate and EC2 Ducks can use the
+  Flock; they received neither before.
+- The **Duck image is published** to GHCR for amd64 and arm64, tagged with the version, so cloud Ducks
+  need no image of your own unless Ponds need system libraries.
+- The S3 Spout now honours `?endpoint=` and `DUCKSTRING_S3_ENDPOINT`, so it works with MinIO, R2 and
+  Ceph; it used to reach only AWS. Tested against MinIO in CI.
+- A failed Spout delivery is retried when its source next publishes, and a Spout or alert destination
+  can be a single `${env:...}` or `${secret:...}` reference.
+- A Pond Draw records its upstream's real version, so a `[sources]` pin above `{major}.0.0` deploys.
+- A Duck that lost its local publish restores it from the durable layer before rebuilding, so Ponds on
+  the same machine never read it incomplete; the durable copy no longer loses warm bands after such a
+  rebuild.
+
+### Correctness and operability
+
+- **DuckDB is bounded** to the tested range, `>=1.5,<2`, and CI tests the floor and the DuckDB 2.0
+  pre-release. A change to or from DuckDB 2.0's `VARIANT` type counts as breaking in the version
+  contract.
+- `catchment init` and `start` warn when an open Catchment binds beyond loopback, since a deploy runs code
+  on the machine.
+- Querying a missing table says the Pond hasn't run yet, or lists the tables it has.
+- A Pool agent stopped with SIGTERM stops its Ducks.
+- A Fargate or EC2 Duck shut down for being idle is replaced on the Pond's next run. Before, the Catchment
+  kept the departed Duck's record, launched nothing, and failed the run as "Lost contact" three minutes
+  later.
+- An EC2 Duck's instance terminates when the Duck exits cleanly, instead of running, and billing, after
+  the Duck has gone. A Duck that crashes leaves its instance up so its console output can be read.
+- SQL Ripples that read a Source work on the DuckDB 2.0 pre-release.
+- A Catchment without API keys could answer a request with 401 under load. The Catchment's database
+  connection is shared by its threads, and the key check read it without the lock the engine holds,
+  so it sometimes saw a key that wasn't there. Every user of the connection now takes that lock, and
+  the key check is cached.
+- A Duck that has heard nothing from its Catchment for an hour while idle exits, so a Duck left behind by
+  a Catchment that was deleted or stopped hard doesn't run, and bill, indefinitely. Set
+  `DUCKSTRING_DUCK_ORPHAN_MINUTES` on the Catchment to change the limit, or `0` to turn it off. Pool
+  machines aren't affected.
+
+### Documentation
+
+The documentation is rewritten: Quickstart, Concepts, task-led Guides and a full Reference (Python API,
+`pond.toml`, every CLI command group, the HTTP API, formats and environment variables), with the README and
+landing page repositioned around Duckstring as a data engineering platform for DuckDB.
+
+### Upgrading
+
+- **Published layout**: plain tables are now written as `{table}__v/{version}.parquet`. Readers still read
+  a Source's old single `{table}.parquet` until it publishes again, so Ponds can be upgraded in any order.
+- **Leftover Iceberg files**: a data directory published by the Iceberg plane keeps a `catalog.json` and a
+  `pond/` directory that nothing reads now. Delete them; on a Catchment using local-first publish with an
+  S3 data root, the Persist mirror would otherwise upload them.
+- **`DUCKSTRING_DATA_PLANE`** is no longer read; remove it from the environment.
+- **DuckDB 2.0** isn't supported yet: installing Duckstring keeps DuckDB below 2, including in Pond
+  environments that lock a newer one.
+- The Catchment database migrates itself on start (one new column).
+
+## 0.5.0 — 2026-08-03
+
+The cloud release: a Catchment can now run its Ducks on AWS, keep its data on S3, and serve that data
+over standard wire protocols. Everything below is opt-in — a Catchment with no cloud configuration
+behaves exactly as it did in 0.4.0, and a `pond.toml` asking for remote compute still runs anywhere.
+
+### Cloud compute
+
+- **Duck launchers**: a Pond's Duck runs on the Catchment's own box (default), on **Fargate** (the
+  default remote backend — serverless containers, task-role IAM), or on **EC2** (the escape hatch, for
+  sizes and GPUs Fargate does not offer). Built-in **S/M/L/XL preset pools** resolve with zero setup.
+- **Duck Pools**: named remote-compute pools (`duckstring duck pool add`), with per-Pond compute
+  declared in `pond.toml` and overridable operationally (`duckstring duck set`).
+- **The Pool agent**: one shared machine per named Pool hosting many co-resident Ducks.
+- **Auto-relay**: a Catchment on a laptop behind NAT can run cloud Ducks with no manual tunnel — the
+  first remote spawn provisions a small always-reachable box and holds a reverse tunnel to it.
+- **Remote failures are observable**: boot output is teed to the console (EC2) / CloudWatch (Fargate)
+  and the tail is attached to the Pond's failure, so a Duck that dies before dialling back still says
+  why. Spawn-failure reasons (missing image, missing AMI, bad IAM) surface on the Pond rather than as a
+  generic crash.
+- Per-provider **startup grace** so a cold machine is not judged by the steady-state silence window.
+
+### Data plane on S3
+
+- A Catchment's data root can be an object store (`duckstring catchment settings --data-root s3://…`),
+  with an S3-compatible **endpoint override** for MinIO/Ceph/R2.
+- **Switch, adopt, or migrate** an existing plane, with a background copy and live progress.
+- **Local-first publish + async Persist**: a run publishes locally and mirrors durably in the
+  background, so compute never waits on the object store; `persisted_f` is the durable watermark and
+  freshness gating is Pool-aware.
+- **S3-resident state**: a merge base is read as a view rather than hydrated, and retention pruning is
+  floor-anchored rather than inferred from absence.
+
+### The Flock — over-envelope compute
+
+A comprehensive `pond.trickle(...)` recompute that exceeds the Duck's memory envelope can be dispatched
+to a serverless engine (**Athena** ships first), while the Duck keeps merge/diff/publish.
+
+**DuckDB is the authority.** Dispatch decides *where* work runs, never *what* is published, enforced
+structurally: an engine-owned allow-list of expressions proven equivalent on both engines (division and
+CAST are excluded — the two engines genuinely disagree), a `conform` step that casts the engine's result
+to DuckDB's own schema and rejects a differing column set, and degradation to local compute on any
+failure. Dispatch counters ride `/metrics`, since a silently degrading Flock is otherwise invisible.
+
+### Data serving
+
+- A sandboxed, warm, read-only query surface over published data: catalog = catchment, schema = pond.
+- **Postgres wire** and **Arrow Flight SQL** adapters, plus `/api/serve` and a CLI.
+- A **Catalog UI** unifying the query surface, with role-governed access.
+
+### Lineage
+
+- Table-level (observed per run), column-level (static per version, with a sqlglot upgrade for SQL
+  outputs), and row-level temporal provenance (`duckstring trace`).
+- **OpenLineage** events emitted on run completion for catalog integration.
+
+### dbt-mode Ponds
+
+Deploy a dbt project as a Pond with no `@ripple` code: each dbt model becomes a Ripple with its own
+freshness, failure and retry tracking, and `ref()` becomes the intra-Pond graph. Opt in with the
+`duckstring[dbt]` extra.
+
+### Correctness and operability
+
+- **Version contract**: a lossless widening of a column type is accepted (a data-dependent branch could
+  legitimately wedge a live Pond permanently); `reset-contract` is the escape hatch for a genuine
+  narrowing; contract failures carry a dedicated sub-reason end-to-end.
+- **Reads reject a stale local publish** the Catchment knows is behind, rather than serving it.
+- Pond Runs the Pond has moved past are closed rather than stranded at `running`.
+- A Pool machine that disappears is detected and relaunched, instead of wedging every Pond on it.
+- The **serving sandbox** no longer spills into the shared system temp — DuckDB exempts its temp
+  directory from the external-access lock, which on Linux exposed everything under `/tmp` to a
+  read-level user.
+- An idle Duck backs off instead of polling ten times a second.
+- Opt-in **re-notify cadence** for alert channels while a failure or freshness episode persists.
+- Incremental object-store egress (`mode=append`) mirrors the published collection per delivery.
+
+### Testing
+
+- The object-store data plane runs against a **real S3 API** in CI (MinIO, which rejects unsigned
+  requests — the class of bug moto cannot catch).
+- Real-data demo sets: **TPC-DS** and **GHArchive** (the latter offline against a committed fixture).
+- Python matrix, the dbt extra, real-Postgres egress, and frontend tests in CI.
+
+### Documentation
+
+- A full **AWS guide** (`guides/cloud.md`): the cloud-enable gate, the three IAM roles, security-group
+  asymmetry, the AMI Python constraint, the build-your-own-image policy, and the Flock's authority rules.
+- New guides for lineage, dbt-mode, and querying data.

{duckstring-0.5.0/src/duckstring.egg-info → duckstring-0.6.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: duckstring
-Version: 0.5.0
-Summary: Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
+Version: 0.6.0
+Summary: An open source data engineering platform for DuckDB: versioned transformations, pull orchestration, incremental processing and a built-in catalog.
 Author-email: Duckstring <dev@duckstring.com>
 License:                                  Apache License
                                    Version 2.0, January 2004
@@ -212,7 +212,7 @@ Project-URL: Playground, https://playground.duckstring.com
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: duckdb>=0.10
+Requires-Dist: duckdb<2,>=1.5
 Requires-Dist: rich>=13.7
 Requires-Dist: typer>=0.12
 Requires-Dist: fastapi>=0.111
@@ -221,10 +221,10 @@ Requires-Dist: uvicorn[standard]>=0.29
 Requires-Dist: httpx>=0.25
 Requires-Dist: tomli>=2.0; python_version < "3.11"
 Requires-Dist: tomli-w>=1.0
-Requires-Dist: pyiceberg>=0.7
 Requires-Dist: pyarrow>=14
 Requires-Dist: pytz
-Provides-Extra: iceberg
+Requires-Dist: pathspec>=0.10
+Requires-Dist: uv>=0.5
 Provides-Extra: dbt
 Requires-Dist: dbt-duckdb>=1.9; extra == "dbt"
 Provides-Extra: lineage
@@ -247,85 +247,111 @@ Dynamic: license-file
 
 # Duckstring
 
-Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
+**Get your ducks in a row.**
 
-Duckstring treats data transformations as software packages. Upstream dependencies are declared per Pond (unit operation), defining the DAG without the need for its direct management. 
+Duckstring is an open source data engineering platform built on DuckDB. Data into the terabytes fits comfortably on a single machine, which is where DuckDB is at its best, and Duckstring provides what a pipeline needs around it: packaging and versioning, orchestration, incremental processing, a catalog and delivery. It runs the same on a laptop, on a single server, or with work spread across cloud machines.
 
-Ponds are upgraded and deployed to Duckstring's pull-based Catchment (orchestrator) atomically - like upgrading a package - with the earlier version continuing to execute until there are no consumers dependent on it. Upstream defines constraints on what it can consume, downstream defines when it's needed, and the Catchment optimally executes the sequence of Ponds supplying it with the best currency and frequency as possible.
+[Documentation](https://docs.duckstring.com) · [Quickstart](https://docs.duckstring.com/quickstart) · [Playground](https://playground.duckstring.com)
 
-You should not need to manage the DAG. You should not need global governance. You should know yourself and your suppliers and trust that you'll get what you need when you need it.
+## Versioned transformations
 
-## Core Concepts
+Each set of transformations is a *Pond*: a Python project whose `pond.toml` names it, gives it a semantic version and lists the Ponds it reads from.
 
-The main elements:
+```toml
+[pond]
+name = "sales"
+version = "1.2.0"
 
-- **Catchment**: Control environment (FastAPI + UI + CLI)
-- **Pond**: Versioned container with declared upstream dependencies
-- **Ripple**: Unit operation within a Pond (e.g. a single transformation producing a table)
-- **Trickle**: DBSP-based incremental engine operating within a Ripple
+[sources]
+transactions = "1.0.0"
+products = "1.1.0"
+```
+
+The pipeline follows from those declarations, so it never has to be drawn by hand. A breaking change ships as a new major version, which runs alongside the old one while downstream Ponds move across at their own pace.
+
+## Pull orchestration
 
-Ponds are typed or referred to in context:
+Schedules are set on the Ponds whose output is actually used, at the end of the pipeline. Duckstring works back from there, running each upstream Pond only as often as something downstream needs it, so a path nobody consumes never runs. A daily job is one command:
+
+```bash
+duckstring trigger tide reports 1d
+```
 
-- **Source**: A parent Pond
-- **Sink**: A child Pond
-- **Inlet**: A Pond with external dependencies and no Sources
-- **Outlet**: A Pond with no Sinks (e.g. outputs final data products)
+## Incremental processing
+
+A *Trickle* stores a table's changes as a Z-set: rows weighted +1 when added and -1 when removed. Joins and aggregations written with the Trickle builder recompute only the keys that changed since the last run, and handle updates and deletes correctly across any shape of join:
+
+```python
+from duckstring import ripple
+
+
+@ripple
+def priced_line(pond):
+    (
+        pond.trickle("orders.order_line")
+        .join(pond.trickle("catalog.product"), on="product_id")
+        .select(
+            "s0.order_id, s0.product_id, s0.quantity, s1.unit_price, "
+            "CAST(round(s0.quantity * s1.unit_price, 2) AS DECIMAL(14,2)) AS revenue"
+        )
+        .merge("priced_line", pk="order_id")
+    )
+```
 
-To see the orchestration model in action without installing anything, try the [Duckstring Playground](https://playground.duckstring.com).
+## Code and data together
+
+The runtime, called the *Catchment*, is also the catalog. Each major version of a Pond is a schema holding its tables, with lineage, run history and the version's schema contract recorded alongside. Query it from the CLI, from Python, or from any Postgres client:
+
+```sql
+SELECT * FROM reports_v1.monthly_summary;
+```
+
+## Python and the command line
+
+Ripples, the steps inside a Pond, are Python functions with a DuckDB connection, so SQL, DuckDB's relation API, pandas and Arrow all work. Everything a Catchment does can be driven from the `duckstring` CLI or its HTTP API, and a web UI shows pipelines as they run.
 
 ## Quickstart
 
+Duckstring needs Python 3.10 or newer.
+
 ```bash
 pip install duckstring
-
-# Start a local Catchment (the runtime + web UI) — leave it running
 duckstring catchment init --name dev
 ```
 
-Then, in another terminal:
+This starts a local Catchment, with its web UI at http://127.0.0.1:7474. Leave it running, and in a second terminal:
 
 ```bash
-# Create the demo pipeline (transactions, products → sales → reports) and deploy it
 mkdir demo && cd demo
-duckstring pond demo
-duckstring pond deploy --all -y
-
-# Run it end to end, once
-duckstring trigger pulse reports
-
-# Look around
-duckstring status                          # live state of every Pond
-duckstring query reports monthly_summary   # peek at an output table
+duckstring pond demo                       # transactions, products → sales → reports
+duckstring pond deploy --all --yes
+duckstring trigger pulse reports           # run the pipeline once, from the end
+duckstring query reports monthly_summary
 ```
 
-The Catchment also serves a live web UI at `http://127.0.0.1:7474` — the Pond graph, freshness, run history, and the full trigger/control surface.
-
-## Execution
+The [Quickstart](https://docs.duckstring.com/quickstart) goes through the same steps with explanations.
 
-Ponds execute on demand signals sent to an Outlet, in two flavours — **push** runs the lineage forward to a target freshness; **pull** propagates demand upstream so every Pond re-runs as its Sources update, naturally throttled to the bottleneck. Each comes as a one-shot or a standing trigger:
+## Also included
 
-| | Once | Continuously |
-|---|---|---|
-| **Push** | Pulse | Tide |
-| **Pull** | Tap | Wave |
+- [Local testing](https://docs.duckstring.com/guides/testing_with_puddles) against small samples of each Source, called Puddles
+- [dbt projects](https://docs.duckstring.com/guides/dbt_projects) deployed as Ponds, one step per model
+- [Delivery](https://docs.duckstring.com/guides/delivering_with_spouts) to Postgres (incremental and exactly once), S3, Google Cloud Storage and local files
+- [Retries, alerts and Prometheus metrics](https://docs.duckstring.com/guides/monitoring_and_failures)
+- [Per-Pond Python environments](https://docs.duckstring.com/guides/writing_ripples#python-dependencies) from `pyproject.toml` and `uv.lock`
+- [Cloud compute](https://docs.duckstring.com/guides/cloud_compute_on_aws) on AWS Fargate or EC2, chosen per Pond, with data on S3
+- [Connected Catchments](https://docs.duckstring.com/guides/connecting_catchments), where a Pond reads from a Pond on another Catchment
 
-A Tide keeps an Outlet no staler than a bound (`duckstring trigger tide reports 1d`); a Wave keeps it as fresh as the pipeline can supply. See [Triggers](https://docs.duckstring.com/guides/triggers) for the full semantics.
+## Documentation
 
-*There is no DAG.*
+[docs.duckstring.com](https://docs.duckstring.com) has the concepts, task guides, and a reference for the CLI, Python API, `pond.toml` and HTTP API. To try pull orchestration without installing anything, use the [Playground](https://playground.duckstring.com).
 
-## Going further
+## Status
 
-Full documentation lives at **[docs.duckstring.com](https://docs.duckstring.com)**:
+Duckstring is pre-1.0, so interfaces may still change between minor versions. Changes are listed in the [changelog](CHANGELOG.md).
 
-- [Quickstart](https://docs.duckstring.com/getting-started/quickstart) — the path above, with explanations
-- [Theory](https://docs.duckstring.com/theory) — the freshness-based orchestration model in depth
-- [Versioning](https://docs.duckstring.com/concepts/versioning) — SemVer on Ponds, concurrent major versions, atomic upgrades
-- [Local testing](https://docs.duckstring.com/guides/local-testing) — Puddles: test a Pond before deploying it
-- [Fault tolerance](https://docs.duckstring.com/guides/fault-tolerance) — retry budgets, failure states, recovery
-- [Running a Catchment](https://docs.duckstring.com/guides/running-a-catchment) — hosting, authentication, platform deployment
-- [CLI](https://docs.duckstring.com/reference/cli) / [HTTP API](https://docs.duckstring.com/reference/http-api) — full references
+## Hosted Catchments
 
-There are future plans for a hosted Catchment service at [duckstring.com](https://duckstring.com). If you're interested, please [get in touch](mailto:dev@duckstring.com).
+A hosted Catchment service is planned at [duckstring.com](https://duckstring.com). If you're interested, get in touch at [dev@duckstring.com](mailto:dev@duckstring.com).
 
 ## License
 

duckstring-0.6.0/README.md

@@ -0,0 +1,111 @@
+# Duckstring
+
+**Get your ducks in a row.**
+
+Duckstring is an open source data engineering platform built on DuckDB. Data into the terabytes fits comfortably on a single machine, which is where DuckDB is at its best, and Duckstring provides what a pipeline needs around it: packaging and versioning, orchestration, incremental processing, a catalog and delivery. It runs the same on a laptop, on a single server, or with work spread across cloud machines.
+
+[Documentation](https://docs.duckstring.com) · [Quickstart](https://docs.duckstring.com/quickstart) · [Playground](https://playground.duckstring.com)
+
+## Versioned transformations
+
+Each set of transformations is a *Pond*: a Python project whose `pond.toml` names it, gives it a semantic version and lists the Ponds it reads from.
+
+```toml
+[pond]
+name = "sales"
+version = "1.2.0"
+
+[sources]
+transactions = "1.0.0"
+products = "1.1.0"
+```
+
+The pipeline follows from those declarations, so it never has to be drawn by hand. A breaking change ships as a new major version, which runs alongside the old one while downstream Ponds move across at their own pace.
+
+## Pull orchestration
+
+Schedules are set on the Ponds whose output is actually used, at the end of the pipeline. Duckstring works back from there, running each upstream Pond only as often as something downstream needs it, so a path nobody consumes never runs. A daily job is one command:
+
+```bash
+duckstring trigger tide reports 1d
+```
+
+## Incremental processing
+
+A *Trickle* stores a table's changes as a Z-set: rows weighted +1 when added and -1 when removed. Joins and aggregations written with the Trickle builder recompute only the keys that changed since the last run, and handle updates and deletes correctly across any shape of join:
+
+```python
+from duckstring import ripple
+
+
+@ripple
+def priced_line(pond):
+    (
+        pond.trickle("orders.order_line")
+        .join(pond.trickle("catalog.product"), on="product_id")
+        .select(
+            "s0.order_id, s0.product_id, s0.quantity, s1.unit_price, "
+            "CAST(round(s0.quantity * s1.unit_price, 2) AS DECIMAL(14,2)) AS revenue"
+        )
+        .merge("priced_line", pk="order_id")
+    )
+```
+
+## Code and data together
+
+The runtime, called the *Catchment*, is also the catalog. Each major version of a Pond is a schema holding its tables, with lineage, run history and the version's schema contract recorded alongside. Query it from the CLI, from Python, or from any Postgres client:
+
+```sql
+SELECT * FROM reports_v1.monthly_summary;
+```
+
+## Python and the command line
+
+Ripples, the steps inside a Pond, are Python functions with a DuckDB connection, so SQL, DuckDB's relation API, pandas and Arrow all work. Everything a Catchment does can be driven from the `duckstring` CLI or its HTTP API, and a web UI shows pipelines as they run.
+
+## Quickstart
+
+Duckstring needs Python 3.10 or newer.
+
+```bash
+pip install duckstring
+duckstring catchment init --name dev
+```
+
+This starts a local Catchment, with its web UI at http://127.0.0.1:7474. Leave it running, and in a second terminal:
+
+```bash
+mkdir demo && cd demo
+duckstring pond demo                       # transactions, products → sales → reports
+duckstring pond deploy --all --yes
+duckstring trigger pulse reports           # run the pipeline once, from the end
+duckstring query reports monthly_summary
+```
+
+The [Quickstart](https://docs.duckstring.com/quickstart) goes through the same steps with explanations.
+
+## Also included
+
+- [Local testing](https://docs.duckstring.com/guides/testing_with_puddles) against small samples of each Source, called Puddles
+- [dbt projects](https://docs.duckstring.com/guides/dbt_projects) deployed as Ponds, one step per model
+- [Delivery](https://docs.duckstring.com/guides/delivering_with_spouts) to Postgres (incremental and exactly once), S3, Google Cloud Storage and local files
+- [Retries, alerts and Prometheus metrics](https://docs.duckstring.com/guides/monitoring_and_failures)
+- [Per-Pond Python environments](https://docs.duckstring.com/guides/writing_ripples#python-dependencies) from `pyproject.toml` and `uv.lock`
+- [Cloud compute](https://docs.duckstring.com/guides/cloud_compute_on_aws) on AWS Fargate or EC2, chosen per Pond, with data on S3
+- [Connected Catchments](https://docs.duckstring.com/guides/connecting_catchments), where a Pond reads from a Pond on another Catchment
+
+## Documentation
+
+[docs.duckstring.com](https://docs.duckstring.com) has the concepts, task guides, and a reference for the CLI, Python API, `pond.toml` and HTTP API. To try pull orchestration without installing anything, use the [Playground](https://playground.duckstring.com).
+
+## Status
+
+Duckstring is pre-1.0, so interfaces may still change between minor versions. Changes are listed in the [changelog](CHANGELOG.md).
+
+## Hosted Catchments
+
+A hosted Catchment service is planned at [duckstring.com](https://duckstring.com). If you're interested, get in touch at [dev@duckstring.com](mailto:dev@duckstring.com).
+
+## License
+
+[Apache 2.0](LICENSE)

{duckstring-0.5.0 → duckstring-0.6.0}/pyproject.toml

@@ -1,7 +1,7 @@
 [project]
 name = "duckstring"
-version = "0.5.0"
-description = "Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically."
+version = "0.6.0"
+description = "An open source data engineering platform for DuckDB: versioned transformations, pull orchestration, incremental processing and a built-in catalog."
 readme = "README.md"
 requires-python = ">=3.10"
 license = { file = "LICENSE" }
@@ -10,7 +10,10 @@ authors = [
 ]
 
 dependencies = [
-  "duckdb>=0.10",
+  # The tested range: CI runs the suite on the floor and the latest release, and against the pre-release
+  # in an allowed-to-fail job. Raise the ceiling once the suite passes on the next major's release
+  # (plans/duckdb-2-ready.md). A Pond environment gets this Duckstring, so these bounds hold there too.
+  "duckdb>=1.5,<2",
   "rich>=13.7",
   "typer>=0.12",
   "fastapi>=0.111",
@@ -19,22 +22,20 @@ dependencies = [
   "httpx>=0.25",
   "tomli>=2.0; python_version < '3.11'",
   "tomli-w>=1.0",
-  # The default data plane: an Apache Iceberg base layer (snapshots + schema metadata over the
-  # Parquet files we already write). Core deps because Iceberg is default-on; set
-  # DUCKSTRING_DATA_PLANE=parquet to opt out (the lighter fallback, also for offline Catchments that
-  # can't fetch DuckDB's iceberg extension). NB: we use our own file-backed catalog
-  # (duckstring.iceberg_catalog) rather than pyiceberg's SqlCatalog, so SQLAlchemy is NOT pulled in.
-  "pyiceberg>=0.7",
+  # Arrow interchange: the Flight SQL server and Arrow fetches from DuckDB.
   "pyarrow>=14",
   # DuckDB requires pytz to convert TIMESTAMPTZ columns to Python datetimes on fetch
   # (our _duckstring_f freshness column is TIMESTAMPTZ, read back in dataplane/trickle export).
   # It is otherwise only present transitively (pandas), so a clean install lacks it.
   "pytz",
+  # .pondignore matching with exact .gitignore semantics (negation, anchoring, **). Pure Python.
+  "pathspec>=0.10",
+  # Builds each Pond's own environment from its pyproject.toml + uv.lock (plans/pond-environments.md).
+  # A self-contained binary wheel; duckstring.environments finds it with uv.find_uv_bin().
+  "uv>=0.5",
 ]
 
 [project.optional-dependencies]
-# Retained for back-compat: the Iceberg deps are in core now (Iceberg is the default plane).
-iceberg = []
 # dbt-mode Ponds (plans/dbt.md): deploy a dbt project as a Pond, each model a Ripple. Heavyweight and
 # opt-in — only pulled when a team actually runs dbt-mode. dbt-duckdb pulls dbt-core transitively.
 dbt = [
@@ -88,6 +89,9 @@ ds = "duckstring.cli:main"
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]
+# Some suites share fixtures by importing `tests.<module>`; put the repo root on the path so a bare
+# `pytest` (what CI runs) resolves them the same as `python -m pytest`.
+pythonpath = ["."]
 # Tight enough to catch an accidental real sleep (sim tests finish in ms), loose enough to absorb
 # cold-runner I/O on CI (SQLite migrate / uvicorn startup can exceed 1s on shared runners).
 timeout = 5

{duckstring-0.5.0 → duckstring-0.6.0}/src/duckstring/alerts/base.py

@@ -27,19 +27,37 @@ class NotifierError(ValueError):
 
 @dataclass(frozen=True)
 class Destination:
-    scheme: str
+    scheme: str | None  # None: the whole destination is a reference, its scheme known only once resolved
     raw: str  # the original URI, with ${...} references intact (resolved only at send time)
 
 
-def parse_notifier_destination(uri: str) -> Destination:
+def parse_notifier_destination(uri: str, *, resolve: bool = False) -> Destination:
     """Validate a channel destination URI: a known scheme + well-formed ``${...}`` references. Does **not**
-    resolve credentials (so a channel can be created before its secrets are present). Raises NotifierError."""
+    resolve credentials (so a channel can be created before its secrets are present). Raises NotifierError.
+
+    A destination that is **entirely** one reference (``${secret:SLACK_WEBHOOK}``) is accepted with
+    ``scheme=None``; ``resolve=True`` (at send/test time) resolves it and checks the resulting scheme. Errors
+    name the reference, never its value."""
     if not uri or not uri.strip():
         raise NotifierError("destination must not be empty")
     try:
         credentials.references(uri)  # validates ${env:}/${secret:} syntax
     except credentials.CredentialError as exc:
         raise NotifierError(str(exc)) from exc
+    if credentials.whole_reference(uri):
+        if not resolve:
+            return Destination(scheme=None, raw=uri)
+        try:
+            target = credentials.resolve(uri)
+        except credentials.CredentialError as exc:
+            raise NotifierError(str(exc)) from exc
+        scheme = urlparse(target).scheme.lower()
+        if scheme not in KNOWN_SCHEMES:
+            raise NotifierError(
+                f"destination {uri.strip()} resolves to a value without a supported scheme — expected "
+                "https://…, http://… or mailto:…"
+            )
+        return Destination(scheme=scheme, raw=uri)
     scheme = urlparse(uri).scheme.lower()
     if not scheme:
         raise NotifierError(f"destination {uri!r} has no scheme — expected e.g. https://…, mailto:…")
@@ -72,7 +90,7 @@ def register(scheme: str, factory: Callable[[Destination], Notifier]) -> None:
 def get_notifier(destination: str) -> Notifier:
     """Resolve the notifier for a channel destination by its scheme. Raises :class:`NotifierError` for an
     unknown scheme, or a known scheme whose notifier is not built."""
-    dest = parse_notifier_destination(destination)
+    dest = parse_notifier_destination(destination, resolve=True)
     factory = _REGISTRY.get(dest.scheme)
     if factory is None:
         raise NotifierError(

{duckstring-0.5.0 → duckstring-0.6.0}/src/duckstring/alerts/email.py

@@ -23,7 +23,15 @@ class EmailNotifier:
     SCHEMES = ("mailto",)
 
     def __init__(self, dest: Destination):
-        parsed = urlparse(dest.raw)
+        from ..egress import credentials
+
+        raw = dest.raw
+        if credentials.whole_reference(raw):  # the whole mailto: URI is held in a variable or secret
+            try:
+                raw = credentials.resolve(raw)
+            except credentials.CredentialError as exc:
+                raise NotifierError(str(exc)) from exc
+        parsed = urlparse(raw)
         self.recipients = [r.strip() for r in unquote(parsed.path).split(",") if r.strip()]
         if not self.recipients:
             raise NotifierError("mailto: destination has no recipient — use mailto:you@example.com")

{duckstring-0.5.0 → duckstring-0.6.0}/src/duckstring/catchment/app.py

@@ -198,7 +198,7 @@ def create_app(
         data_root = get_setting(con, DATA_ROOT_KEY)
 
     # Writer lease on an external data root — refuse to start if a *different* live Catchment owns it (two
-    # Catchments racing one lake's Iceberg catalog would dangle its pointer). A same-id restart reclaims
+    # Catchments publishing and pruning the same lines would break single-writer-per-line). A same-id restart reclaims
     # instantly; only engaged for an external DUCKSTRING_DATA_ROOT, so the local default is untouched.
     data_lease = None
     if data_root: