duckstring 0.1.0__tar.gz → 0.3.0__tar.gz

duckstring-0.3.0/MANIFEST.in

@@ -0,0 +1,8 @@
+# The sdist ships the installable package only — keep internal design notes and the test suite
+# out of the PyPI source tarball (they're in the git repo for contributors).
+prune tests
+prune plans
+prune experiment
+prune docs
+prune frontend
+prune playground

{duckstring-0.1.0 → duckstring-0.3.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: duckstring
-Version: 0.1.0
-Summary: Duckstring is a Python-native data-mesh orchestration framework designed to be as naturally extensible as installing packages.
+Version: 0.3.0
+Summary: Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically.
 Author-email: Duckstring <dev@duckstring.com>
 License:                                  Apache License
                                    Version 2.0, January 2004
@@ -205,6 +205,10 @@ License:                                  Apache License
            See the License for the specific language governing permissions and
            limitations under the License.
         
+Project-URL: Homepage, https://duckstring.com
+Project-URL: Documentation, https://docs.duckstring.com
+Project-URL: Repository, https://github.com/duckstring-dev/duckstring
+Project-URL: Playground, https://playground.duckstring.com
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -212,242 +216,101 @@ Requires-Dist: duckdb>=0.10
 Requires-Dist: rich>=13.7
 Requires-Dist: typer>=0.12
 Requires-Dist: fastapi>=0.111
+Requires-Dist: python-multipart>=0.0.9
 Requires-Dist: uvicorn[standard]>=0.29
+Requires-Dist: httpx>=0.25
+Requires-Dist: tomli>=2.0; python_version < "3.11"
+Requires-Dist: tomli-w>=1.0
+Requires-Dist: pyiceberg>=0.7
+Requires-Dist: pyarrow>=14
+Requires-Dist: pytz
+Provides-Extra: iceberg
 Provides-Extra: dev
 Requires-Dist: pytest>=8.0; extra == "dev"
+Requires-Dist: pytest-timeout>=0.5; extra == "dev"
 Requires-Dist: ruff>=0.3; extra == "dev"
 Requires-Dist: mypy>=1.8; extra == "dev"
 Dynamic: license-file
 
 # Duckstring
+*There is no DAG.*
 
-Duckstring is a data pipeline framework built around modular, versioned nodes called **Ponds**. Each Pond specifies its immediate parents (with version), allowing for the formation of a DAG much like one would install packages. 
+Duckstring treats data transformations as software packages. Upstream dependencies are declared per Pond (unit operation), defining the DAG without the need for its direct management. 
 
-Pond execution is orchestrated within an environment - a **Catchment** - that controls storage and other global settings. It uses a pull-based system modelled after Kanban, with **Outlets**  (terminal Ponds) sending demand upstream. This allows each Pond to be modified and deployed independently, with any paths in the DAG that are not attached to any Outlet automatically skipped. 
+Ponds are upgraded and deployed to Duckstring's pull-based Catchment (orchestrator) atomically - like upgrading a package - with the earlier version continuing to execute until there are no consumers dependent on it. Upstream defines constraints on what it can consume, downstream defines when it's needed, and the Catchment optimally executes the sequence of Ponds supplying it with the best currency and frequency as possible.
 
-Duckstring is built on the philosophy that most data pipelines are not truly "big data" and with good design can execute on a single compute node. It is primarily designed for batch and incremental workloads for tables on the order of tens of millions of rows (e.g. <50M).
-
-The default engine is DuckDB, though this is configurable. Duckstring is however an independent project and is not affiliated with, endorsed by, or maintained by the DuckDB project.
-
--- **Note**: As the project is in development, most of the notes below should be read as *indended functionality*, and most features are not yet implemented.
+You should not need to manage the DAG. You should not need global governance. You should know yourself and your suppliers and trust that you'll get what you need when you need it.
 
 ## Core Concepts
 
-- **Catchment**: Control environment - a FastAPI application
-- **Pond**: Versioned transformation unit with declared upstream dependencies - the main element of version control
-- **Inlet**: Pond with external dependencies and no upstream Ponds
-- **Outlet**: Pond with no downstream Ponds (e.g. outputs final data products)
-- **Ripple**: Unit operation within a Pond (e.g. a single transformation producing a table)
-
-## Installation
-
-```bash
-pip install duckstring
-```
-
-## Quickstart
-
-### 1) Connect to a Catchment
-
-A Catchment is the execution environment, receiving Ponds and managing runs. It runs either as a local daemon or as a remote server, allowing you to start locally and seamlessly upgrade to a hosted/cloud server if you need to later.
-
-#### Start a Catchment Server
-
-To run a Catchment locally, run:
-
-```bash
-duckstring catchment start --name dev --port 5000 --root ~/.duckstring/dev
-```
-
-This will start a server with name 'dev' at port 5000 (the default, if none specified) and store Catchment details at `~/.duckstring/dev` (default is `~/.duckstring/{name}`). If any of these options are omitted you will be prompted on start.
-
-#### Connect to a Remote Server
-
-Alternatively, you can connect to a server running a Catchment:
-
-```bash
-duckstring catchment connect --name dev --path https://path.to.catchment
-```
-
-This will prompt for any necessary auth, and will add the Catchment under the specified name.
-
-#### Connect to *duckstring.com*
-
-There are future plans for a dedicated Catchment service at https://duckstring.com. If you're interested, please [contact me](mailto:dev@duckstring.com).
-
-### 2) Define Pond(s)
-
-#### Demo Ponds
-
-If you want to see an example sequence of Ponds in action immediately, create three project directories and run one of these commands in each:
-
-```bash
-duckstring pond demo inlet
-duckstring pond demo pond
-duckstring pond demo outlet
-```
-
-It's recommended to do this before attempting to make your own so that you can get a feel for the structure.
-
-#### Custom Pond
-
-Create a project directory and run:
-
-```bash
-duckstring pond init example_pond
-```
-
-This will create a duckstring pond structure:
-
-```text
-root/
-|-- src/
-|   |-- pond.py
-|-- pond.toml
-|-- __main__.py
-|-- .gitignore
-|-- README.md
-```
-
-Here `pond.py` contains the code for a single Ripple operation (currently blank), and `pond.toml` specifies the Pond name "example_pond" and version (defaulting to "0.1.0").
-
-### 3) Deploy to Catchment
-
-#### From Local
-
-From a Pond's project root run:
-
-```bash
-duckstring deploy dev
-```
-
-This will read the pond name, version and type (Inlet, Pond, Outlet) from `pond.toml` and deploy the project contents to the Catchment specified by name (here `dev`).
-
-Alternatively, you can import the Pond using the Catchment UI.
-
-#### From Git
-
-If you are using git with a remote, you can deploy with:
+The main elements:
 
-```bash
-duckstring deploy dev --git {branch|commit|tag}
-```
-
-This will use the current branch/commit/tag to define the Pond. Upon each execution the Catchment will clone the repository and run it.
-
-This can also be specified using the Catchment UI.
-
-### 3) Execute
-
-Ponds are executed by sending a Demand signal from an Outlet. This propagates backwards through the DAG until it reaches each upstream Inlet, causing them to execute, with children beginning upon completion of all of their parents.
-
-These examples will use the Pond `outlet`, version `1.0.0`, as the execution reference. All examples may also be alternatively executed using the Catchment UI.
-
-#### Pulse
-
-To initiate a single run:
-
-```bash
-duckstring pulse dev outlet
-```
-
-The `pulse` mode emits a Demand signal from `outlet`, and when it begins execution, sends a Stop signal. This causes it to execute exactly once.
-
-This will automatically run against the maximum version available for that Pond. To use a specific version:
-
-```bash
-duckstring pulse dev outlet --version 1
-```
-
-#### Wave
-
-To continuously run:
-
-```bash
-duckstring wave dev outlet
-```
-
-The `wave` mode emits a Demand signal from `outlet`, and when it begins execution, sends *another* Demand signal. This causes it to execute continuously, as frequently as the DAG allows (i.e. at a period equal to the execution time of the slowest Ripple in any Pond).
-
-#### Tide
-
-To run at a scheduled frequency:
-
-```bash
-duckstring tide dev outlet 15 2 * * * --local
-```
-
-This would run at 2:15am every day local time, using cron syntax. Omitting the `--local` flag defaults to UTC.
-
-### 4) Monitor
-
-To print out a summary of current processes in the Catchment:
-
-```bash
-duckstring status dev
-```
-
-This will print to CLI a summary for each Pond that is either currently executing or has Demand.
-
-To include all Ponds:
+- **Catchment**: Control environment (FastAPI + UI + CLI)
+- **Pond**: Versioned container with declared upstream dependencies
+- **Ripple**: Unit operation within a Pond (e.g. a single transformation producing a table)
+- **Trickle**: DBSP-based incremental engine operating within a Ripple
 
-```bash
-duckstring status dev --all
-```
+Ponds are typed or referred to in context:
 
-### 5) Retrieve Data
+- **Source**: A parent Pond
+- **Sink**: A child Pond
+- **Inlet**: A Pond with external dependencies and no Sources
+- **Outlet**: A Pond with no Sinks (e.g. outputs final data products)
 
-#### Get
+To see the orchestration model in action without installing anything, try the [Duckstring Playground](https://playground.duckstring.com).
 
-The simplest way to retrieve data is to load by the Ripple name. This returns the entire contents of the directory, and does not require that the data be in a tabular format (e.g. SQL-compatible).
+## Quickstart
 
 ```bash
-duckstring get dev outlet daily
-```
-
-This writes a directory `./ponds/outlet/daily` with the 'daily' Ripple's contents. You may also override the default location:
+pip install duckstring
 
-```bash
-duckstring get dev outlet daily --path ./daily_output
+# Start a local Catchment (the runtime + web UI) — leave it running
+duckstring catchment init --name dev
 ```
 
-#### SQL Query
-
-If the target is an SQL-compatible table (e.g. DuckDB or Parquet), an SQL statement may be sent directly, outputting the result to the command line:
+Then, in another terminal:
 
 ```bash
-duckstring query dev outlet --sql "SELECT * FROM daily WHERE id=1;"
-```
+# Create the demo pipeline (transactions, products → sales → reports) and deploy it
+mkdir demo && cd demo
+duckstring pond demo
+duckstring pond deploy --all -y
 
-Alternatively, include a file path:
+# Run it end to end, once
+duckstring trigger pulse reports
 
-```bash
-duckstring query dev outlet --sql @path/to/query.sql
+# Look around
+duckstring status                          # live state of every Pond
+duckstring query reports monthly_summary   # peek at an output table
 ```
 
-Omitting the `--sql` statement queries with a default SELECT * LIMIT 10 on the specified table:
+The Catchment also serves a live web UI at `http://127.0.0.1:7474` — the Pond graph, freshness, run history, and the full trigger/control surface.
 
-```bash
-duckstring query dev outlet daily
-```
+## Execution
 
-##### Write to file
+Ponds execute on demand signals sent to an Outlet, in two flavours — **push** runs the lineage forward to a target freshness; **pull** propagates demand upstream so every Pond re-runs as its Sources update, naturally throttled to the bottleneck. Each comes as a one-shot or a standing trigger:
 
-To output to a file, include a flag for the file format, followed by the file name:
+| | Once | Continuously |
+|---|---|---|
+| **Push** | Pulse | Tide |
+| **Pull** | Tap | Wave |
 
-`--csv`: Comma-separated values
-`--json`: JSON records
-`--parquet`: Parquet file
+A Tide keeps an Outlet no staler than a bound (`duckstring trigger tide reports 1d`); a Wave keeps it as fresh as the pipeline can supply. See [Triggers](https://docs.duckstring.com/guides/triggers) for the full semantics.
 
-This writes by default to `./ponds/outlet/daily/{filename}`. To overrite the default location you may use the `--path` flag.
+## Going further
 
-For example, to execute an sql statement from file `query.sql` and write the result to CSV at the current directory:
+Full documentation lives at **[docs.duckstring.com](https://docs.duckstring.com)**:
 
-```bash
-duckstring query dev outlet --sql @query.sql --csv daily.csv --path .
-```
+- [Quickstart](https://docs.duckstring.com/getting-started/quickstart) — the path above, with explanations
+- [Theory](https://docs.duckstring.com/theory) — the freshness-based orchestration model in depth
+- [Versioning](https://docs.duckstring.com/concepts/versioning) — SemVer on Ponds, concurrent major versions, atomic upgrades
+- [Local testing](https://docs.duckstring.com/guides/local-testing) — Puddles: test a Pond before deploying it
+- [Fault tolerance](https://docs.duckstring.com/guides/fault-tolerance) — retry budgets, failure states, recovery
+- [Running a Catchment](https://docs.duckstring.com/guides/running-a-catchment) — hosting, authentication, platform deployment
+- [CLI](https://docs.duckstring.com/reference/cli) / [HTTP API](https://docs.duckstring.com/reference/http-api) — full references
 
-## Further Reading
+There are future plans for a hosted Catchment service at [duckstring.com](https://duckstring.com). If you're interested, please [get in touch](mailto:dev@duckstring.com).
 
-For more detail on each component, please read the corresponding documentation in `docs/`.
+## License
 
+[Apache 2.0](LICENSE)

duckstring-0.3.0/README.md

@@ -0,0 +1,82 @@
+# Duckstring
+*There is no DAG.*
+
+Duckstring treats data transformations as software packages. Upstream dependencies are declared per Pond (unit operation), defining the DAG without the need for its direct management. 
+
+Ponds are upgraded and deployed to Duckstring's pull-based Catchment (orchestrator) atomically - like upgrading a package - with the earlier version continuing to execute until there are no consumers dependent on it. Upstream defines constraints on what it can consume, downstream defines when it's needed, and the Catchment optimally executes the sequence of Ponds supplying it with the best currency and frequency as possible.
+
+You should not need to manage the DAG. You should not need global governance. You should know yourself and your suppliers and trust that you'll get what you need when you need it.
+
+## Core Concepts
+
+The main elements:
+
+- **Catchment**: Control environment (FastAPI + UI + CLI)
+- **Pond**: Versioned container with declared upstream dependencies
+- **Ripple**: Unit operation within a Pond (e.g. a single transformation producing a table)
+- **Trickle**: DBSP-based incremental engine operating within a Ripple
+
+Ponds are typed or referred to in context:
+
+- **Source**: A parent Pond
+- **Sink**: A child Pond
+- **Inlet**: A Pond with external dependencies and no Sources
+- **Outlet**: A Pond with no Sinks (e.g. outputs final data products)
+
+To see the orchestration model in action without installing anything, try the [Duckstring Playground](https://playground.duckstring.com).
+
+## Quickstart
+
+```bash
+pip install duckstring
+
+# Start a local Catchment (the runtime + web UI) — leave it running
+duckstring catchment init --name dev
+```
+
+Then, in another terminal:
+
+```bash
+# Create the demo pipeline (transactions, products → sales → reports) and deploy it
+mkdir demo && cd demo
+duckstring pond demo
+duckstring pond deploy --all -y
+
+# Run it end to end, once
+duckstring trigger pulse reports
+
+# Look around
+duckstring status                          # live state of every Pond
+duckstring query reports monthly_summary   # peek at an output table
+```
+
+The Catchment also serves a live web UI at `http://127.0.0.1:7474` — the Pond graph, freshness, run history, and the full trigger/control surface.
+
+## Execution
+
+Ponds execute on demand signals sent to an Outlet, in two flavours — **push** runs the lineage forward to a target freshness; **pull** propagates demand upstream so every Pond re-runs as its Sources update, naturally throttled to the bottleneck. Each comes as a one-shot or a standing trigger:
+
+| | Once | Continuously |
+|---|---|---|
+| **Push** | Pulse | Tide |
+| **Pull** | Tap | Wave |
+
+A Tide keeps an Outlet no staler than a bound (`duckstring trigger tide reports 1d`); a Wave keeps it as fresh as the pipeline can supply. See [Triggers](https://docs.duckstring.com/guides/triggers) for the full semantics.
+
+## Going further
+
+Full documentation lives at **[docs.duckstring.com](https://docs.duckstring.com)**:
+
+- [Quickstart](https://docs.duckstring.com/getting-started/quickstart) — the path above, with explanations
+- [Theory](https://docs.duckstring.com/theory) — the freshness-based orchestration model in depth
+- [Versioning](https://docs.duckstring.com/concepts/versioning) — SemVer on Ponds, concurrent major versions, atomic upgrades
+- [Local testing](https://docs.duckstring.com/guides/local-testing) — Puddles: test a Pond before deploying it
+- [Fault tolerance](https://docs.duckstring.com/guides/fault-tolerance) — retry budgets, failure states, recovery
+- [Running a Catchment](https://docs.duckstring.com/guides/running-a-catchment) — hosting, authentication, platform deployment
+- [CLI](https://docs.duckstring.com/reference/cli) / [HTTP API](https://docs.duckstring.com/reference/http-api) — full references
+
+There are future plans for a hosted Catchment service at [duckstring.com](https://duckstring.com). If you're interested, please [get in touch](mailto:dev@duckstring.com).
+
+## License
+
+[Apache 2.0](LICENSE)

duckstring-0.3.0/pyproject.toml

@@ -0,0 +1,78 @@
+[project]
+name = "duckstring"
+version = "0.3.0"
+description = "Build data pipelines the way you build software: version each transform, declare its dependencies, and Duckstring resolves the execution DAG automatically."
+readme = "README.md"
+requires-python = ">=3.10"
+license = { file = "LICENSE" }
+authors = [
+  { name = "Duckstring", email = "dev@duckstring.com" }
+]
+
+dependencies = [
+  "duckdb>=0.10",
+  "rich>=13.7",
+  "typer>=0.12",
+  "fastapi>=0.111",
+  "python-multipart>=0.0.9",
+  "uvicorn[standard]>=0.29",
+  "httpx>=0.25",
+  "tomli>=2.0; python_version < '3.11'",
+  "tomli-w>=1.0",
+  # The default data plane: an Apache Iceberg base layer (snapshots + schema metadata over the
+  # Parquet files we already write). Core deps because Iceberg is default-on; set
+  # DUCKSTRING_DATA_PLANE=parquet to opt out (the lighter fallback, also for offline Catchments that
+  # can't fetch DuckDB's iceberg extension). NB: we use our own file-backed catalog
+  # (duckstring.iceberg_catalog) rather than pyiceberg's SqlCatalog, so SQLAlchemy is NOT pulled in.
+  "pyiceberg>=0.7",
+  "pyarrow>=14",
+  # DuckDB requires pytz to convert TIMESTAMPTZ columns to Python datetimes on fetch
+  # (our _duckstring_f freshness column is TIMESTAMPTZ, read back in dataplane/trickle export).
+  # It is otherwise only present transitively (pandas), so a clean install lacks it.
+  "pytz",
+]
+
+[project.optional-dependencies]
+# Retained for back-compat: the Iceberg deps are in core now (Iceberg is the default plane).
+iceberg = []
+dev = [
+  "pytest>=8.0",
+  "pytest-timeout>=0.5",
+  "ruff>=0.3",
+  "mypy>=1.8",
+]
+
+[tool.setuptools]
+package-dir = { "" = "src" }
+
+[tool.setuptools.package-data]
+"duckstring.catchment" = ["static/**/*", "schema/*.sql"]
+"duckstring" = ["demo/**/*", "demo/**/.gitignore"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[project.urls]
+Homepage = "https://duckstring.com"
+Documentation = "https://docs.duckstring.com"
+Repository = "https://github.com/duckstring-dev/duckstring"
+Playground = "https://playground.duckstring.com"
+
+[project.scripts]
+duckstring = "duckstring.cli:main"
+ds = "duckstring.cli:main"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+# Tight enough to catch an accidental real sleep (sim tests finish in ms), loose enough to absorb
+# cold-runner I/O on CI (SQLite migrate / uvicorn startup can exceed 1s on shared runners).
+timeout = 5
+
+[tool.ruff]
+line-length = 128
+target-version = "py310"
+extend-exclude = ["experiment"]
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "B"]
+ignore = ["B008"]  # typer.Option/Argument in defaults is intentional

duckstring-0.3.0/src/duckstring/__init__.py

@@ -0,0 +1,26 @@
+from importlib.metadata import PackageNotFoundError
+from importlib.metadata import version as _pkg_version
+
+from .core import (
+    Catchment,
+    Pond,
+    Puddle,
+    Ripple,
+    puddle,
+    ripple,
+)
+
+try:
+    __version__ = _pkg_version("duckstring")
+except PackageNotFoundError:  # running from a source tree without an installed dist
+    __version__ = "0.0.0"
+
+__all__ = [
+    "Catchment",
+    "Pond",
+    "Puddle",
+    "Ripple",
+    "puddle",
+    "ripple",
+    "__version__",
+]

duckstring-0.3.0/src/duckstring/acc.py

@@ -0,0 +1,13 @@
+"""Backwards-compatible alias — scan (order-dependent) metric specs live in :mod:`duckstring.trickle.acc`.
+
+Kept so ``from duckstring import acc`` works alongside ``from duckstring import agg``. See
+:mod:`duckstring.trickle_io` for the rationale.
+"""
+
+from __future__ import annotations
+
+from .trickle import acc as _acc
+
+
+def __getattr__(name):  # PEP 562: forward all names to the relocated module
+    return getattr(_acc, name)

duckstring-0.3.0/src/duckstring/agg.py

@@ -0,0 +1,12 @@
+"""Backwards-compatible alias — aggregate metric specs now live in :mod:`duckstring.trickle.agg`.
+
+Kept so ``from duckstring import agg`` continues to work. See :mod:`duckstring.trickle_io` for the rationale.
+"""
+
+from __future__ import annotations
+
+from .trickle import agg as _agg
+
+
+def __getattr__(name):  # PEP 562: forward all names to the relocated module
+    return getattr(_agg, name)

duckstring-0.3.0/src/duckstring/catchment/app.py

@@ -0,0 +1,123 @@
+from __future__ import annotations
+
+import asyncio
+import os
+import secrets
+from contextlib import asynccontextmanager
+from datetime import datetime, timezone
+from pathlib import Path
+
+from fastapi import FastAPI
+from fastapi.responses import JSONResponse
+from fastapi.staticfiles import StaticFiles
+
+from .db import connect, ensure_identity, migrate
+from .driver import Driver
+from .launcher import NoopLauncher, SubprocessLauncher
+from .routes import router
+
+_STATIC_DIR = Path(__file__).parent / "static"
+
+
+async def _scheduler(driver: Driver) -> None:
+    """Drive clock processes (Tide deadlines, window boundaries, Wave-on-idle) at next_wake."""
+    while True:
+        nw = driver.next_wake()
+        now = datetime.now(timezone.utc)
+        delay = (nw - now).total_seconds() if nw else 1.0
+        await asyncio.sleep(max(0.05, min(delay, 5.0)))
+        try:
+            driver.scheduler_tick()
+        except Exception as exc:  # keep the loop alive
+            print(f"[catchment] scheduler error: {exc}", flush=True)
+
+
+@asynccontextmanager
+async def _lifespan(app: FastAPI):
+    base_url = app.state.base_url
+    if os.environ.get("DUCKSTRING_DISABLE_DUCKS"):
+        launcher = NoopLauncher()
+    else:
+        # Ducks dial back over the same authenticated surface — they present the API key as their token.
+        # base_url None = unknown (the platform picked the bind address): the launcher defers spawns
+        # until the dial-back middleware learns the address from the first request.
+        launcher = SubprocessLauncher(app.state.root, base_url, token=app.state.api_key or "")
+    driver = Driver(app.state.db, app.state.root, base_url, launcher)
+    app.state.driver = driver
+    app.state.launcher = launcher
+
+    # Restore: resume any Pond Runs that were in flight when the Catchment last stopped.
+    driver.resume_incomplete()
+
+    from .poller import run_poller
+
+    # The poller wakes immediately when a Draw acquires demand (so it solicits its upstream at once),
+    # instead of waiting for its next cycle. The driver signals across threads via the running loop.
+    wake = asyncio.Event()
+    loop = asyncio.get_running_loop()
+    driver.set_notify(lambda: loop.call_soon_threadsafe(wake.set))
+
+    scheduler = asyncio.create_task(_scheduler(driver))
+    poller = asyncio.create_task(run_poller(driver, app.state.root, wake))
+    try:
+        yield
+    finally:
+        for task in (scheduler, poller):
+            task.cancel()
+            try:
+                await task
+            except asyncio.CancelledError:
+                pass
+        launcher.shutdown_all()
+
+
+def create_app(
+    root: Path, api_key: str | None = None, base_url: str | None = None, name: str | None = None
+) -> FastAPI:
+    root.mkdir(parents=True, exist_ok=True)
+    con = connect(root / "duck.db")
+    migrate(con)
+    ensure_identity(con, name or os.environ.get("DUCKSTRING_CATCHMENT_NAME"))
+
+    app = FastAPI(title="Duckstring Catchment", lifespan=_lifespan)
+    app.state.root = root
+    app.state.db = con
+    # API key: explicit argument, or the environment (useful for containers/remote serving). When
+    # set, every /api request (except /api/health) must present it — Bearer header or X-Duck-Token.
+    app.state.api_key = api_key or os.environ.get("DUCKSTRING_API_KEY") or None
+    # The address Ducks dial back to: explicit argument (the CLI passes its bind address), or the
+    # environment, or None — unknown, because the host platform picks the bind address (e.g. Posit
+    # Connect). When None it is learned from the first request's ASGI scope below.
+    app.state.base_url = base_url or os.environ.get("DUCKSTRING_CATCHMENT_URL") or None
+
+    @app.middleware("http")
+    async def _learn_dialback_address(request, call_next):
+        launcher = getattr(app.state, "launcher", None)
+        if launcher is not None and getattr(launcher, "base_url", "") is None:
+            server = request.scope.get("server")  # the server's bound (host, port) per the ASGI spec
+            if server and server[1]:  # a unix socket has port None — nothing TCP to dial
+                host = "127.0.0.1" if server[0] in ("0.0.0.0", "::") else server[0]
+                url = f"http://{host}:{server[1]}"
+                app.state.base_url = url
+                app.state.driver.base_url = url
+                launcher.set_base_url(url)  # spawns any Ducks that were waiting on the address
+        return await call_next(request)
+
+    @app.middleware("http")
+    async def _require_api_key(request, call_next):
+        key = app.state.api_key
+        path = request.url.path
+        if key and path.startswith("/api") and path != "/api/health":
+            auth = request.headers.get("authorization", "")
+            supplied = auth[7:] if auth.lower().startswith("bearer ") else ""
+            supplied = supplied or request.headers.get("x-duck-token", "")
+            if not secrets.compare_digest(supplied, key):
+                return JSONResponse({"detail": "Invalid or missing API key"}, status_code=401)
+        return await call_next(request)
+
+    app.include_router(router, prefix="/api")
+
+    if _STATIC_DIR.exists():
+        app.mount("/", StaticFiles(directory=_STATIC_DIR, html=True), name="frontend")
+
+    return app