continuo-python-runtime 0.8.0__tar.gz → 0.9.0__tar.gz

continuo_python_runtime-0.9.0/.python-version

@@ -0,0 +1 @@
+3.14.2

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/CHANGELOG.md

@@ -5,6 +5,90 @@ follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+## [0.9.0] - 2026-10-11
+
+### Added
+
+- Contracts may set `mode_type` — `replace` (the default), `append` or
+  `upsert` — and, for `upsert` only, `primary_key`: a non-empty list of output
+  columns, each spelled exactly as in `output_columns` and declared
+  `nullable: false`, names unique case-insensitively, and none a
+  floating-point (`DOUBLE PRECISION`) column — the warehouse collapses `0.0`
+  and `-0.0`, so a float key is rejected at validation rather than failing at
+  write time.
+  `python-csv` nodes allow only `replace`. `continuo-runtime validate`, `merge`
+  and `run` enforce these rules; the harness's write path for the new modes
+  follows in a later entry.
+- The merged wire entry carries `mode_type` when it is not `replace` and
+  `primary_key` (authored order) for `upsert`; both are part of
+  `config_hash`. Entries of nodes that do not set them, and their hashes, are
+  unchanged.
+- `continuo-engine-contract`: opt-in write capabilities on top of
+  `WarehouseAdapter` — `TableIntrospection` (`table_columns`, returning
+  `LiveColumn(name, nullable)`; `LookupError` for a missing table),
+  `AppendCapable` (`append`) and `UpsertCapable` (`upsert`, which runs
+  `ensure_unique_key` and then the engine's `_upsert`). `ensure_table` takes an
+  optional keyword-only `primary_key`. `continuo_engine_contract.keys` holds
+  `ensure_unique_key` and `DuplicateKeyError(ValueError)`.
+- `continuo_engine_contract.testing.conformance`: a pytest conformance suite for the
+  write capabilities (`IntrospectionConformance`, `AppendConformance`,
+  `UpsertConformance`). Adapter integration suites subclass the reusable
+  classes and run them against their real engine, providing `adapter` and
+  `schema` fixtures. Install with `continuo-engine-contract[testing]`.
+- `continuo-postgres-adapter`: `append` (batched inserts in one transaction),
+  `upsert` (`INSERT ... ON CONFLICT (key) DO UPDATE`, `DO NOTHING` when every
+  column is in the key) and `table_columns`; `ensure_table` declares an
+  upsert key as the table's `PRIMARY KEY`. `load` now shares the insert path
+  (no behavior change).
+- `continuo-duckdb-adapter`: `append` (one transaction), `upsert` (`MERGE`
+  from the registered Arrow data, one transaction) and `table_columns`.
+  `ensure_table` accepts and ignores an upsert key: DuckLake has no
+  constraints, so the key is guaranteed by the write path.
+- `continuo-trino-adapter`: `append` and `upsert` through a stage table
+  shaped like the target, then one `INSERT ... SELECT` or one `MERGE` — one
+  Iceberg commit per write; the stage is dropped best-effort afterwards.
+  `table_columns` reads `information_schema`. `ensure_table` accepts an
+  upsert key (no constraint DDL) and rejects it with `format_version: 1`.
+  `load` now shares the stage helpers (no behavior change).
+- The harness writes each node with its `mode_type`: `load` for `replace`,
+  `append`, or `upsert` with the node's `primary_key` (also passed to
+  `ensure_table`). An adapter without the needed capability fails with
+  `LoadError: <Adapter> does not support <mode>` before the script runs; an
+  `upsert` whose rows repeat a key fails with `ConformError` and writes
+  nothing. Before an `append` or `upsert` the live table's columns and
+  nullability are compared with `output_columns`; any difference is a
+  `LoadError` naming each one. See "Write modes" in the README.
+
+### Changed
+
+- The Python version is pinned in one place, `.python-version` (`3.14.2`), the
+  single source of truth: the engine images' `Dockerfile`s pin the same exact
+  patch (`python:3.14.2-slim`), and `requires-python`, mypy `python_version`,
+  the trove classifiers and `setup-python` inputs track its major.minor. A new
+  consistency guard (`tests/test_python_version_consistency.py`) scans every
+  `pyproject.toml`, `Dockerfile*` and workflow and fails on any drift.
+- `continuo-engine-contract` 0.7.3 -> 0.8.0 (the write capabilities above).
+  The runtime and the three adapters pin it exactly, as do the engine images'
+  `image-requirements-*.txt`.
+- **Breaking for newly copied templates.** The `template/` release workflow
+  calls continuo's public release API instead of an unauthenticated webhook:
+  - `RELEASE_ENDPOINT` keeps its name but is now the origin of continuo's
+    `auth.publicUrl` (`scheme://host[:port]`, no path), which is also the OIDC
+    audience.
+  - The repository must be bound to the service in continuo's
+    `ciAuth.bindings`.
+  - The first release of a service is an operator bootstrap; the workflow never
+    sends `bootstrap`.
+
+  The workflow checks the endpoint and does an authenticated read of
+  `/api/v1/current-prod` before building, so a wrong URL, audience or missing
+  binding fails early. It then submits to `<RELEASE_ENDPOINT>/api/v1/releases`
+  with a fresh GitHub Actions OIDC token per call and polls
+  `GET /api/v1/releases/{id}` for up to about 50 minutes (`timeout-minutes: 60`),
+  failing on `rejected` and `superseded`. A newer push no longer cancels a run
+  that is waiting for its release (`cancel-in-progress: false`). Workflows
+  already copied from the template are untouched.
+
 ## [0.8.0] - 2026-10-01
 
 Packages in this release: `continuo-python-runtime` 0.8.0 (no runtime code

continuo_python_runtime-0.9.0/CLAUDE.md

@@ -0,0 +1,67 @@
+# continuo-python-runtime
+
+The validation/execution runtime that continuo's executor runs as a Kubernetes
+Job. Python 3.12, uv workspace: the runtime (`continuo_python_runtime`), the
+engine contract (`contract/`), and the engine adapters (`adapters/postgres`,
+`adapters/trino`, `adapters/duckdb`).
+
+## CHANGELOG is not optional
+
+Every user-facing change keeps `CHANGELOG.md` current. Two steps, both required:
+
+- **On every PR** that changes behavior, the CLI/op surface, a package version,
+  or a container: add a bullet under `## [Unreleased]` (Keep a Changelog style —
+  Added / Changed / Fixed / Removed). No entry, no merge.
+- **On every release**: rename `## [Unreleased]` to `## [X.Y.Z] - YYYY-MM-DD`,
+  add a fresh empty `## [Unreleased]` above it, and tag. `release.yml` reads the
+  section matching the pushed tag for the GitHub Release notes, so a release with
+  no matching section ships with generic auto-notes and the changelog silently
+  loses that version.
+
+If the changelog is missing a shipped version, backfill it from that tag's
+merged PRs before adding anything new.
+
+## Engine adapter conformance
+
+`contract/continuo_engine_contract/testing/conformance.py`
+(`continuo_engine_contract.testing.conformance`) is the shared, published
+conformance suite for engine adapter authors. Its dependencies are installed
+with `continuo-engine-contract[testing]`.
+
+- Every adapter MUST satisfy the engine contract and pass the shared
+  conformance classes for every capability it declares: `IntrospectionConformance`
+  for `TableIntrospection`, `AppendConformance` for `AppendCapable`, and
+  `UpsertConformance` for `UpsertCapable`. Append and upsert adapters must also
+  run the introspection class. An adapter need not declare capabilities its
+  engine cannot support.
+- Each adapter's integration suite MUST subclass the applicable classes,
+  supply connected `adapter` and temporary `schema` fixtures, and run them
+  against its real warehouse in CI. In-memory reference tests alone do not
+  establish an adapter's conformance. Do not skip or weaken shared checks to
+  accommodate an adapter that violates the contract.
+- Every new or changed engine-independent contract requirement, including
+  bug regressions, MUST have coverage in this shared testing module. Extend
+  the applicable conformance class, or add a class for a new capability, so
+  all adapters declaring it are checked by the same cases. Exercise the
+  reusable cases in `contract/tests/test_conformance_suite.py` as well.
+- Engine-specific implementation details, such as SQL generation, driver
+  behavior and authentication, belong in the owning adapter's tests. Add a
+  shared case when the behavior expresses a requirement of the engine
+  contract; unrelated runtime features do not belong in this module.
+- Before completing an adapter or contract change, run `contract/tests` and
+  the affected adapter's unit and real-engine integration suites in Docker.
+
+## Releasing
+
+A release is one `chore(release):` commit that bumps versions and updates the
+changelog, then a `vX.Y.Z` tag on it. Bump the version of every package whose
+source changed since the last tag — `scripts/check_version_bumps.py` fails the
+PyPI publish otherwise. The tag builds and publishes the `-postgres` / `-trino` / `-duckdb`
+images and the PyPI packages. See `CONTRIBUTING.md` for the full steps.
+
+## Conventions
+
+- Standard `logging` for diagnostics, never `print` — except the CLI's
+  sentinel-framed result blocks, which own stdout.
+- In-repo packages are pinned exactly, not with ranges (see `pyproject.toml`).
+- Commits are signed off (`git commit -s`); CI enforces the DCO.

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/Dockerfile.duckdb

@@ -1,4 +1,4 @@
-FROM python:3.14-slim
+FROM python:3.14.2-slim
 # One dual-role image per engine: the python-node runtime harness (default
 # command `run`) and the blue/green validation runner (`validation-op`). It
 # installs the PUBLISHED, versioned libraries, never the repo source:

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/Dockerfile.postgres

@@ -1,4 +1,4 @@
-FROM python:3.14-slim
+FROM python:3.14.2-slim
 # One dual-role image per engine — the python-node runtime harness (default
 # command `run`) and the blue/green validation runner (`validation-op`). It
 # installs the PUBLISHED, versioned libraries, never the repo source:

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/Dockerfile.trino

@@ -1,4 +1,4 @@
-FROM python:3.14-slim
+FROM python:3.14.2-slim
 # One dual-role image per engine — the python-node runtime harness (default
 # command `run`) and the blue/green validation runner (`validation-op`). It
 # installs the PUBLISHED, versioned libraries, never the repo source:

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: continuo-python-runtime
-Version: 0.8.0
+Version: 0.9.0
 Summary: Runtime harness, contract tooling, and CI lint for Continuo python nodes.
 Author: Simone Carolini
 Maintainer: Simone Carolini
@@ -12,7 +12,7 @@ Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3.14
 Requires-Python: >=3.14
 Requires-Dist: boto3==1.43.85
-Requires-Dist: continuo-engine-contract==0.7.3
+Requires-Dist: continuo-engine-contract==0.8.0
 Requires-Dist: pyarrow==25.0.1
 Requires-Dist: pyyaml==6.0.3
 Requires-Dist: sqlglot==30.17.0
@@ -73,7 +73,8 @@ engine adapters, the validation runner, and the node harness. The former
 `continuo-validation` repository was merged in — there is no longer a separate
 validation-side port, adapter class, entry-point group, or image. One
 `WarehouseAdapter` per engine serves both the data plane (`fetch` /
-`ensure_table` / `load`) and validation (`ensure_schema` / `drop_schema` /
+`ensure_table` / `load`, plus the opt-in `table_columns` / `append` / `upsert`
+capabilities) and validation (`ensure_schema` / `drop_schema` /
 `build_empty_from_sql` / `build_empty_from_columns` / `clone_empty_from_prod` /
 `check_binds`), and one image per engine runs both roles.
 
@@ -121,9 +122,13 @@ the Go parser has not been taught is a production outage, not a refactor.
    service name (one service name per domain repo).
 3. Configure repository variables in GitHub (Settings → Secrets and
    variables → Actions): `REGISTRY` (your Docker registry), `BUCKET` (your
-   S3 bucket for contract artifacts), `RELEASE_ENDPOINT` (the release
-   webhook endpoint). `RELEASE_ENDPOINT` is the **base URL** of the Continuo
-   API (no `/releases` suffix) — the workflow appends `/releases` itself.
+   S3 bucket for contract artifacts), `RELEASE_ENDPOINT` (the base
+   URL of your continuo install, `scheme://host[:port]` with no path, the origin of continuo's
+   `auth.publicUrl`). The
+   workflow calls `<RELEASE_ENDPOINT>/api/v1/releases` with its GitHub Actions
+   OIDC token, so the repository must be bound to your service in continuo's
+   `ciAuth.bindings` (see [Releasing from CI](https://github.com/carolsimone/continuo/blob/main/deploy/README.md#releasing-from-ci-github-actions)),
+   and an operator bootstraps the service's first release.
 4. Configure repository secrets: `AWS_ACCESS_KEY_ID` and
    `AWS_SECRET_ACCESS_KEY` for the S3 upload. The template workflow pushes
    the built image to GHCR using the workflow's own `GITHUB_TOKEN` (granted
@@ -191,6 +196,62 @@ produced, never how it is checked or written.
   script reads with `os.environ[...]`. The contract never holds a secret value
   or env var name; the Secret itself is created in the cluster by the operator.
 
+## Write modes
+
+`mode_type:` chooses how the harness writes a node's table on each run:
+
+| `mode_type` | Each run | Rows from earlier runs |
+| --- | --- | --- |
+| `replace` (default) | empties the table, inserts this run's rows | gone |
+| `append` | inserts this run's rows | kept |
+| `upsert` | inserts rows with a new key, overwrites rows whose key exists | kept unless overwritten; never deleted |
+
+```yaml
+- schema: fx
+  table: fx_rates
+  kind: python-api
+  mode_type: upsert
+  primary_key: [rate_date, base, quote]
+  output_columns:
+    - {name: rate_date, type: DATE,       nullable: false}
+    - {name: base,      type: VARCHAR(3), nullable: false}
+    - {name: quote,     type: VARCHAR(3), nullable: false}
+    - {name: rate,      type: DECIMAL(12,6)}
+```
+
+- `python-csv` nodes only `replace`.
+- `primary_key` is required for `upsert` and rejected otherwise: a non-empty
+  list of output columns declared `nullable: false` (a NULL key never
+  matches), each spelled exactly as in `output_columns`, names unique
+  case-insensitively. A Postgres `replace` table that
+  wants uniqueness uses `config.indexes` with `unique: true`.
+- Every write is atomic: all of the run's rows or none. An `append` or
+  `upsert` run with zero rows writes nothing (a `replace` run with zero rows
+  still empties the table).
+- `append` is a plain insert: a run that committed but was reported failed
+  and is retried stores its rows twice. `upsert` is idempotent.
+- An `upsert` run whose own rows repeat a key fails with `ConformError` and
+  writes nothing.
+- Before an `append` or `upsert` the harness compares the live table with
+  `output_columns` — names case-insensitively, nullability exactly; order and
+  types are not compared — and fails with `LoadError` naming every missing
+  column, unexpected column and nullability mismatch. Changing an `append` or
+  `upsert` table (columns, types, nullability, key or mode) needs a new table
+  name (`_v2`) under a new contract; continuo rejects the release otherwise.
+  Adding a nullable column is no exception: the existing rows cannot be
+  backfilled.
+- The installed engine must support the mode; all three shipped adapters do.
+  A bring-your-own adapter declares `AppendCapable` / `UpsertCapable` from
+  `continuo-engine-contract` (and can run its conformance suite); without them
+  an `append`/`upsert` node fails with `LoadError: <Adapter> does not support
+  <mode>` before its script runs.
+- Engines: on Postgres the key is the table's `PRIMARY KEY`, created with the
+  table; DuckLake and Trino/Iceberg have no constraints, so the key is
+  guaranteed by the write path only. On Trino an `upsert` table needs Iceberg
+  `format_version` 2 (the default).
+- `full_refresh` stays refused for every python node: a rebuild would destroy
+  an accumulating table.
+
 ## The script API
 
 A node script is a Python file with exactly one required entry point:
@@ -277,8 +338,8 @@ text.
 | `ContractError` | Contract missing or invalid, node not found for `NODE_ID`, or the declared script is missing/unreachable. | The contract yaml or the `script:` path. |
 | `ReadError` | `ctx.read()` was called with an undeclared name, or a declared read failed at the warehouse. | The `reads:` map, or the upstream query/warehouse access. |
 | `ScriptError` | `run()` raised, has no callable `run`, or returned a value that isn't Arrow-convertible. | The node script. |
-| `ConformError` | Structural mismatch (extra/missing/duplicate columns), a strict-cast failure, a not-null violation, or a VARCHAR/CHAR overflow. | The script's output shape, or the `output_columns` declaration. |
-| `LoadError` | Adapter construction failed, or the DDL/INSERT failed at the warehouse during the write. | Warehouse connectivity/permissions, or the target table. |
+| `ConformError` | Structural mismatch (extra/missing/duplicate columns), a strict-cast failure, a not-null violation, a VARCHAR/CHAR overflow, or an `upsert` run whose rows repeat a primary key. | The script's output shape, or the `output_columns` declaration. |
+| `LoadError` | Adapter construction failed, the installed engine does not support the node's `mode_type`, the live table's shape differs from `output_columns` (append/upsert), or the DDL/INSERT failed at the warehouse during the write. | Warehouse connectivity/permissions, the engine image, or the target table. |
 
 ## Engine selection
 
@@ -286,18 +347,21 @@ A domain repo picks its warehouse engine by which base image it builds
 `FROM`:
 
 ```dockerfile
-FROM ghcr.io/carolsimone/continuo-python-runtime-postgres:v0.8.0
+FROM ghcr.io/carolsimone/continuo-python-runtime-postgres:v0.9.0
+# or
+FROM ghcr.io/carolsimone/continuo-python-runtime-trino:v0.9.0
 # or
-FROM ghcr.io/carolsimone/continuo-python-runtime-trino:v0.8.0
+FROM ghcr.io/carolsimone/continuo-python-runtime-duckdb:v0.9.0
 ```
 
 The engine is part of the image **name**; the tag is the bare version, so
 Continuo's Helm chart can pin an image as `<name>:vX.Y.Z@sha256:<digest>`.
 
 Each image bakes in exactly one `WarehouseAdapter` for that engine — the
-pinned PyPI version of the `continuo-postgres-adapter` or
-`continuo-trino-adapter` package built from this repo's `adapters/postgres/`
-or `adapters/trino/` source (see the table above) — registered under the
+pinned PyPI version of the `continuo-postgres-adapter`,
+`continuo-trino-adapter` or `continuo-duckdb-adapter` package built from this
+repo's `adapters/postgres/`, `adapters/trino/` or `adapters/duckdb/` source
+(see the table above) — registered under the
 `continuo_engine.adapters` entry-point group (entry names `postgres` /
 `trino` / `duckdb`). The runtime discovers it via `discover_adapter()` at run time, so a
 single image serves every node in the service and the release-time

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/README.md

@@ -53,7 +53,8 @@ engine adapters, the validation runner, and the node harness. The former
 `continuo-validation` repository was merged in — there is no longer a separate
 validation-side port, adapter class, entry-point group, or image. One
 `WarehouseAdapter` per engine serves both the data plane (`fetch` /
-`ensure_table` / `load`) and validation (`ensure_schema` / `drop_schema` /
+`ensure_table` / `load`, plus the opt-in `table_columns` / `append` / `upsert`
+capabilities) and validation (`ensure_schema` / `drop_schema` /
 `build_empty_from_sql` / `build_empty_from_columns` / `clone_empty_from_prod` /
 `check_binds`), and one image per engine runs both roles.
 
@@ -101,9 +102,13 @@ the Go parser has not been taught is a production outage, not a refactor.
    service name (one service name per domain repo).
 3. Configure repository variables in GitHub (Settings → Secrets and
    variables → Actions): `REGISTRY` (your Docker registry), `BUCKET` (your
-   S3 bucket for contract artifacts), `RELEASE_ENDPOINT` (the release
-   webhook endpoint). `RELEASE_ENDPOINT` is the **base URL** of the Continuo
-   API (no `/releases` suffix) — the workflow appends `/releases` itself.
+   S3 bucket for contract artifacts), `RELEASE_ENDPOINT` (the base
+   URL of your continuo install, `scheme://host[:port]` with no path, the origin of continuo's
+   `auth.publicUrl`). The
+   workflow calls `<RELEASE_ENDPOINT>/api/v1/releases` with its GitHub Actions
+   OIDC token, so the repository must be bound to your service in continuo's
+   `ciAuth.bindings` (see [Releasing from CI](https://github.com/carolsimone/continuo/blob/main/deploy/README.md#releasing-from-ci-github-actions)),
+   and an operator bootstraps the service's first release.
 4. Configure repository secrets: `AWS_ACCESS_KEY_ID` and
    `AWS_SECRET_ACCESS_KEY` for the S3 upload. The template workflow pushes
    the built image to GHCR using the workflow's own `GITHUB_TOKEN` (granted
@@ -171,6 +176,62 @@ produced, never how it is checked or written.
   script reads with `os.environ[...]`. The contract never holds a secret value
   or env var name; the Secret itself is created in the cluster by the operator.
 
+## Write modes
+
+`mode_type:` chooses how the harness writes a node's table on each run:
+
+| `mode_type` | Each run | Rows from earlier runs |
+| --- | --- | --- |
+| `replace` (default) | empties the table, inserts this run's rows | gone |
+| `append` | inserts this run's rows | kept |
+| `upsert` | inserts rows with a new key, overwrites rows whose key exists | kept unless overwritten; never deleted |
+
+```yaml
+- schema: fx
+  table: fx_rates
+  kind: python-api
+  mode_type: upsert
+  primary_key: [rate_date, base, quote]
+  output_columns:
+    - {name: rate_date, type: DATE,       nullable: false}
+    - {name: base,      type: VARCHAR(3), nullable: false}
+    - {name: quote,     type: VARCHAR(3), nullable: false}
+    - {name: rate,      type: DECIMAL(12,6)}
+```
+
+- `python-csv` nodes only `replace`.
+- `primary_key` is required for `upsert` and rejected otherwise: a non-empty
+  list of output columns declared `nullable: false` (a NULL key never
+  matches), each spelled exactly as in `output_columns`, names unique
+  case-insensitively. A Postgres `replace` table that
+  wants uniqueness uses `config.indexes` with `unique: true`.
+- Every write is atomic: all of the run's rows or none. An `append` or
+  `upsert` run with zero rows writes nothing (a `replace` run with zero rows
+  still empties the table).
+- `append` is a plain insert: a run that committed but was reported failed
+  and is retried stores its rows twice. `upsert` is idempotent.
+- An `upsert` run whose own rows repeat a key fails with `ConformError` and
+  writes nothing.
+- Before an `append` or `upsert` the harness compares the live table with
+  `output_columns` — names case-insensitively, nullability exactly; order and
+  types are not compared — and fails with `LoadError` naming every missing
+  column, unexpected column and nullability mismatch. Changing an `append` or
+  `upsert` table (columns, types, nullability, key or mode) needs a new table
+  name (`_v2`) under a new contract; continuo rejects the release otherwise.
+  Adding a nullable column is no exception: the existing rows cannot be
+  backfilled.
+- The installed engine must support the mode; all three shipped adapters do.
+  A bring-your-own adapter declares `AppendCapable` / `UpsertCapable` from
+  `continuo-engine-contract` (and can run its conformance suite); without them
+  an `append`/`upsert` node fails with `LoadError: <Adapter> does not support
+  <mode>` before its script runs.
+- Engines: on Postgres the key is the table's `PRIMARY KEY`, created with the
+  table; DuckLake and Trino/Iceberg have no constraints, so the key is
+  guaranteed by the write path only. On Trino an `upsert` table needs Iceberg
+  `format_version` 2 (the default).
+- `full_refresh` stays refused for every python node: a rebuild would destroy
+  an accumulating table.
+
 ## The script API
 
 A node script is a Python file with exactly one required entry point:
@@ -257,8 +318,8 @@ text.
 | `ContractError` | Contract missing or invalid, node not found for `NODE_ID`, or the declared script is missing/unreachable. | The contract yaml or the `script:` path. |
 | `ReadError` | `ctx.read()` was called with an undeclared name, or a declared read failed at the warehouse. | The `reads:` map, or the upstream query/warehouse access. |
 | `ScriptError` | `run()` raised, has no callable `run`, or returned a value that isn't Arrow-convertible. | The node script. |
-| `ConformError` | Structural mismatch (extra/missing/duplicate columns), a strict-cast failure, a not-null violation, or a VARCHAR/CHAR overflow. | The script's output shape, or the `output_columns` declaration. |
-| `LoadError` | Adapter construction failed, or the DDL/INSERT failed at the warehouse during the write. | Warehouse connectivity/permissions, or the target table. |
+| `ConformError` | Structural mismatch (extra/missing/duplicate columns), a strict-cast failure, a not-null violation, a VARCHAR/CHAR overflow, or an `upsert` run whose rows repeat a primary key. | The script's output shape, or the `output_columns` declaration. |
+| `LoadError` | Adapter construction failed, the installed engine does not support the node's `mode_type`, the live table's shape differs from `output_columns` (append/upsert), or the DDL/INSERT failed at the warehouse during the write. | Warehouse connectivity/permissions, the engine image, or the target table. |
 
 ## Engine selection
 
@@ -266,18 +327,21 @@ A domain repo picks its warehouse engine by which base image it builds
 `FROM`:
 
 ```dockerfile
-FROM ghcr.io/carolsimone/continuo-python-runtime-postgres:v0.8.0
+FROM ghcr.io/carolsimone/continuo-python-runtime-postgres:v0.9.0
+# or
+FROM ghcr.io/carolsimone/continuo-python-runtime-trino:v0.9.0
 # or
-FROM ghcr.io/carolsimone/continuo-python-runtime-trino:v0.8.0
+FROM ghcr.io/carolsimone/continuo-python-runtime-duckdb:v0.9.0
 ```
 
 The engine is part of the image **name**; the tag is the bare version, so
 Continuo's Helm chart can pin an image as `<name>:vX.Y.Z@sha256:<digest>`.
 
 Each image bakes in exactly one `WarehouseAdapter` for that engine — the
-pinned PyPI version of the `continuo-postgres-adapter` or
-`continuo-trino-adapter` package built from this repo's `adapters/postgres/`
-or `adapters/trino/` source (see the table above) — registered under the
+pinned PyPI version of the `continuo-postgres-adapter`,
+`continuo-trino-adapter` or `continuo-duckdb-adapter` package built from this
+repo's `adapters/postgres/`, `adapters/trino/` or `adapters/duckdb/` source
+(see the table above) — registered under the
 `continuo_engine.adapters` entry-point group (entry names `postgres` /
 `trino` / `duckdb`). The runtime discovers it via `discover_adapter()` at run time, so a
 single image serves every node in the service and the release-time

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/adapters/duckdb/continuo_duckdb_adapter/adapter.py

@@ -6,9 +6,9 @@ port leaves to the concrete engine, ``required_env`` and ``from_env``.
 """
 from __future__ import annotations
 
-from .application.warehouse import LakeWarehouse
-from .infrastructure.session import DuckLakeSession
-from .infrastructure.settings import REQUIRED_ENV, DuckLakeSettings
+from continuo_duckdb_adapter.application.warehouse import LakeWarehouse
+from continuo_duckdb_adapter.infrastructure.session import DuckLakeSession
+from continuo_duckdb_adapter.infrastructure.settings import REQUIRED_ENV, DuckLakeSettings
 
 
 class DuckDBAdapter(LakeWarehouse):

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/adapters/duckdb/continuo_duckdb_adapter/application/ports.py

@@ -6,9 +6,11 @@ from collections.abc import Sequence
 from contextlib import AbstractContextManager
 from typing import TYPE_CHECKING
 
-from ..domain.columns import ColumnDefinition
-from ..domain.identifiers import Identifier, QualifiedTable
-from ..domain.layout import TableLayout
+from continuo_engine_contract.port import LiveColumn  # type: ignore[import-untyped]
+
+from continuo_duckdb_adapter.domain.columns import ColumnDefinition
+from continuo_duckdb_adapter.domain.identifiers import Identifier, QualifiedTable
+from continuo_duckdb_adapter.domain.layout import TableLayout
 
 if TYPE_CHECKING:  # pragma: no cover
     import pyarrow  # type: ignore[import-untyped]
@@ -35,6 +37,10 @@ class LakeGateway(ABC):
     @abstractmethod
     def table_exists(self, table: QualifiedTable) -> bool: ...
 
+    @abstractmethod
+    def columns(self, table: QualifiedTable) -> "list[LiveColumn]":
+        """Name and nullability of every column of *table*, in table order; [] when absent."""
+
     @abstractmethod
     def drop_table_if_exists(self, table: QualifiedTable) -> None: ...
 
@@ -73,5 +79,11 @@ class LakeGateway(ABC):
     def insert_arrow(self, table: QualifiedTable, data: "pyarrow.Table") -> None:
         """Append *data* in its own column order."""
 
+    @abstractmethod
+    def merge_arrow(
+        self, table: QualifiedTable, data: "pyarrow.Table", key: Sequence[str]
+    ) -> None:
+        """Insert *data*'s rows with a new *key*, overwrite rows whose key exists."""
+
     @abstractmethod
     def close(self) -> None: ...

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/adapters/duckdb/continuo_duckdb_adapter/application/warehouse.py

@@ -10,16 +10,21 @@ from __future__ import annotations
 
 import logging
 import time
-from collections.abc import Callable
+from collections.abc import Callable, Sequence
 from typing import TYPE_CHECKING, Any
 
-from continuo_engine_contract.port import WarehouseAdapter  # type: ignore[import-untyped]
+from continuo_engine_contract.port import (  # type: ignore[import-untyped]
+    AppendCapable,
+    LiveColumn,
+    UpsertCapable,
+    WarehouseAdapter,
+)
 from continuo_engine_contract.sql import ensure_single_read  # type: ignore[import-untyped]
 
-from ..domain.columns import ColumnDefinition, column_types
-from ..domain.identifiers import Identifier, QualifiedTable
-from ..domain.layout import TableLayout
-from .ports import LakeConflictError, LakeGateway
+from continuo_duckdb_adapter.domain.columns import ColumnDefinition, column_types
+from continuo_duckdb_adapter.domain.identifiers import Identifier, QualifiedTable
+from continuo_duckdb_adapter.domain.layout import TableLayout
+from continuo_duckdb_adapter.application.ports import LakeConflictError, LakeGateway
 
 if TYPE_CHECKING:  # pragma: no cover
     import pyarrow as pa  # type: ignore[import-untyped]
@@ -42,7 +47,7 @@ def _columns(raw: list[dict]) -> list[ColumnDefinition]:
     return [ColumnDefinition.from_mapping(entry) for entry in raw]
 
 
-class LakeWarehouse(WarehouseAdapter):
+class LakeWarehouse(WarehouseAdapter, AppendCapable, UpsertCapable):
     """WarehouseAdapter implemented over a LakeGateway."""
 
     def __init__(
@@ -159,6 +164,17 @@ class LakeWarehouse(WarehouseAdapter):
         """
         TableLayout.from_config(config, {name: None for name in column_names})
 
+    def table_columns(self, schema: str, table: str) -> list[LiveColumn]:
+        """Columns of ``schema.table`` as the DuckLake catalog reports them.
+
+        Raises:
+            LookupError: If the table does not exist.
+        """
+        columns = self._gateway.columns(QualifiedTable.of(schema, table))
+        if not columns:
+            raise LookupError(f"table {schema}.{table} does not exist")
+        return columns
+
     def ensure_table(
         self,
         schema: str,
@@ -166,6 +182,7 @@ class LakeWarehouse(WarehouseAdapter):
         columns: list[dict[str, Any]],
         *,
         config: dict[str, Any],
+        primary_key: Sequence[str] = (),
     ) -> None:
         """Create the table if absent; layout is applied only when this call creates it.
 
@@ -173,7 +190,12 @@ class LakeWarehouse(WarehouseAdapter):
         existence check and the create share one transaction; a concurrent
         creator makes one side fail with a conflict, and the retry then finds
         the table and returns.
+
+        *primary_key* is accepted and ignored: DuckLake rejects PRIMARY KEY and
+        UNIQUE constraints, so an upsert node's key is guaranteed by ``upsert``
+        (the port's duplicate check, then MERGE), not by the table.
         """
+        del primary_key
         defs = _columns(columns)
         layout = TableLayout.from_config(config, column_types(defs))
         self.ensure_schema(schema)
@@ -200,6 +222,41 @@ class LakeWarehouse(WarehouseAdapter):
                 logger.info("loading %d row(s) into %s.%s", data.num_rows, schema, table)
                 self._gateway.insert_arrow(target, data)
 
+    def append(self, schema: str, table: str, data: "pa.Table") -> None:
+        """Atomically insert every row of *data*; earlier rows stay (one transaction).
+
+        The insert names *data*'s columns. A 0-row *data* opens no transaction,
+        so DuckLake records no snapshot for it.
+        """
+        if not data.num_rows:
+            logger.info("append of 0 rows into %s.%s: nothing to write", schema, table)
+            return
+        target = QualifiedTable.of(schema, table)
+        with self._gateway.transaction():
+            logger.info("appending %d row(s) into %s.%s", data.num_rows, schema, table)
+            self._gateway.insert_arrow(target, data)
+
+    def _upsert(
+        self, schema: str, table: str, data: "pa.Table", primary_key: Sequence[str]
+    ) -> None:
+        """MERGE *data* into ``schema.table`` on *primary_key*, in one transaction.
+
+        DuckLake's MERGE does not detect a key repeated within *data* (it
+        inserts both rows of a new key and keeps the last of an existing one);
+        the port's ``upsert`` has already rejected that. A 0-row *data* opens
+        no transaction.
+        """
+        if not data.num_rows:
+            logger.info("upsert of 0 rows into %s.%s: nothing to write", schema, table)
+            return
+        target = QualifiedTable.of(schema, table)
+        with self._gateway.transaction():
+            logger.info(
+                "upserting %d row(s) into %s.%s on (%s)",
+                data.num_rows, schema, table, ", ".join(primary_key),
+            )
+            self._gateway.merge_arrow(target, data, list(primary_key))
+
     def close(self) -> None:
         """Release the underlying connection."""
         self._gateway.close()

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/adapters/duckdb/continuo_duckdb_adapter/domain/columns.py

@@ -8,7 +8,7 @@ from typing import Any
 
 from continuo_engine_contract.types import validate_column_type  # type: ignore[import-untyped]
 
-from .identifiers import Identifier
+from continuo_duckdb_adapter.domain.identifiers import Identifier
 
 _TEMPORAL = re.compile(r"^(TIMESTAMP|DATE)\Z", re.IGNORECASE | re.ASCII)
 

{continuo_python_runtime-0.8.0 → continuo_python_runtime-0.9.0}/adapters/duckdb/continuo_duckdb_adapter/domain/layout.py

@@ -13,8 +13,8 @@ from typing import Any
 
 from continuo_engine_contract.config import ensure_known_keys  # type: ignore[import-untyped]
 
-from .columns import is_temporal_type
-from .identifiers import Identifier
+from continuo_duckdb_adapter.domain.columns import is_temporal_type
+from continuo_duckdb_adapter.domain.identifiers import Identifier
 
 ENGINE = "duckdb"
 KNOWN_CONFIG_KEYS: tuple[str, ...] = ("partitioned_by", "sorted_by")