continuo-duckdb-adapter 0.1.0__tar.gz

continuo_duckdb_adapter-0.1.0/.gitignore

@@ -0,0 +1,13 @@
+.idea/
+.DS_Store
+__pycache__/
+.venv/
+dist/
+.superpowers/
+docs/superpowers/
+docs/ddd-violations/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+.worktrees/

continuo_duckdb_adapter-0.1.0/PKG-INFO

@@ -0,0 +1,124 @@
+Metadata-Version: 2.5
+Name: continuo-duckdb-adapter
+Version: 0.1.0
+Summary: DuckDB (DuckLake) warehouse adapter for Continuo: validation and python-node runtime.
+Author: Simone Carolini
+Maintainer: Simone Carolini
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.14
+Requires-Dist: continuo-engine-contract==0.7.3
+Requires-Dist: duckdb==1.5.6
+Requires-Dist: pyarrow==25.0.1
+Description-Content-Type: text/markdown
+
+# continuo-duckdb-adapter
+
+DuckDB warehouse adapter for Continuo, running on a **DuckLake**: the catalog
+lives in Postgres and table data is Parquet on S3/MinIO, so every Kubernetes
+Job shares one transactional warehouse. Implements
+`continuo_engine_contract.port.WarehouseAdapter`; registered under the
+`continuo_engine.adapters` entry-point group as `duckdb`.
+
+## Environment
+
+| Variable | Required | Default | Meaning |
+|---|---|---|---|
+| `DUCKDB_CATALOG_HOST` | yes | | Postgres host holding the DuckLake catalog |
+| `DUCKDB_CATALOG_PORT` | no | `5432` | |
+| `DUCKDB_CATALOG_DB` | yes | | |
+| `DUCKDB_CATALOG_USER` | yes | | |
+| `DUCKDB_CATALOG_PASSWORD` | no | empty | |
+| `DUCKDB_DATA_PATH` | yes | | Data location, e.g. `s3://bucket/lake/` (fixed once the catalog exists) |
+| `DUCKDB_S3_ENDPOINT` | no | AWS | `host:port`, for MinIO and other S3-compatible stores |
+| `DUCKDB_S3_ACCESS_KEY_ID` / `DUCKDB_S3_SECRET_ACCESS_KEY` | no | credential chain | Static credentials; omit to use the AWS credential chain |
+| `DUCKDB_S3_REGION` | no | `us-east-1` | |
+| `DUCKDB_S3_URL_STYLE` | no | `path` with an endpoint, else `vhost` | `path` or `vhost` |
+| `DUCKDB_S3_USE_SSL` | no | `true` | |
+| `DUCKDB_EXTENSION_DIRECTORY` | no | DuckDB default | Where `ducklake`, `postgres`, `httpfs` and `aws` (the credential chain) are loaded from (set in the image) |
+| `DUCKDB_TEMP_DIRECTORY` | no | `.tmp` in the working directory | Where DuckDB spills larger-than-memory work; must be writable by the runtime user (the image sets `/tmp/duckdb-tmp`) |
+| `DUCKDB_DATA_INLINING_ROW_LIMIT` | no | DuckLake default | `0` writes every insert as a Parquet file instead of inlining small ones in the catalog |
+
+Settings are parsed strictly and fail fast with the variable named:
+
+- `DUCKDB_S3_USE_SSL` accepts `true`/`false`/`1`/`0`/`yes`/`no` (any case);
+  anything else is an error rather than silently meaning "true".
+- `DUCKDB_S3_ACCESS_KEY_ID` and `DUCKDB_S3_SECRET_ACCESS_KEY` are set together or
+  not at all; one without the other is rejected (it would otherwise fall back to
+  the credential chain without saying so).
+- `DUCKDB_CATALOG_PORT` and `DUCKDB_DATA_INLINING_ROW_LIMIT` take ASCII digits
+  only.
+
+## Credentials and failure modes
+
+The catalog password is handed to libpq through a private (mode 0600) temporary
+passfile, not in the connection string, so it does not appear in DuckDB's error
+text or in `duckdb_databases()`. Every DuckDB error also passes through one
+redaction point that masks the catalog password and the S3 secret (every
+spelling) before the error reaches the result block or the pod logs; host, port
+and database stay visible. The password travels inline instead (still redacted from errors) when it
+contains a line break, when `PGPASSWORD` is set in the environment (libpq would
+prefer it to any passfile), or when no temp file can be created. The `aws`
+extension is only loaded for the AWS credential-chain S3 path.
+
+A first attach to a brand-new catalog from several Jobs at once can race on
+DuckLake's metadata creation; the adapter retries exactly that failure a few
+times, and surfaces every other connection error immediately.
+
+## Physical layout (`config`)
+
+- `partitioned_by`: non-empty list of a column name, or
+  `{column, transform, buckets}` with `transform` in `identity`, `bucket`
+  (`buckets` required), `year`, `month`, `day`, `hour` (time transforms need a
+  `DATE` or `TIMESTAMP` column).
+- `sorted_by`: non-empty list of a column name, or
+  `{column, direction: asc|desc, nulls: first|last}`.
+- Columns must be declared in `output_columns`. Any other key, including
+  postgres's `indexes`, is rejected before any DDL runs.
+- Layout is applied when `ensure_table` creates the table; changing it on an
+  existing table is a no-op. `build_empty_from_columns` (the release gate)
+  always rebuilds.
+
+## Engine behaviour to know
+
+- DuckDB drops the length of `VARCHAR(n)` / `CHAR(n)`: the catalog shows plain
+  `VARCHAR`. Length is enforced by `conform()` for python nodes only, not by the
+  table.
+- DuckLake inlines small inserts into the catalog (see
+  `DUCKDB_DATA_INLINING_ROW_LIMIT`), so partitioning and sorting are applied to
+  Parquet files only after a flush; set the limit to `0` when files must be
+  laid out per write.
+- A read is one single query: top-level `PIVOT` / `UNPIVOT` statements are
+  rejected by the single-read gate. Wrap them: `SELECT * FROM (PIVOT ...)`.
+
+## Parity with the postgres and trino adapters
+
+| Behaviour | postgres | trino | duckdb |
+|---|---|---|---|
+| `drop_schema` | `DROP SCHEMA IF EXISTS ... CASCADE` | same | same (tables and views go too) |
+| `ensure_schema` under concurrency | session advisory lock | `IF NOT EXISTS`, tolerating a concurrent creation | `IF NOT EXISTS`, bounded retry on a DuckLake snapshot conflict |
+| `check_binds` | `EXPLAIN` in `BEGIN READ ONLY` | `EXPLAIN (TYPE VALIDATE)` | `EXPLAIN` in `BEGIN TRANSACTION READ ONLY`, always rolled back |
+| `ensure_table` | creates if absent, layout on create | same | same, in one transaction with the existence check |
+| `load` (replace contents) | one transaction | atomic table swap (no multi-statement transactions) | one transaction: `DELETE` then `INSERT` |
+| Physical layout keys | `indexes` | `partitioning`, `sorted_by`, `format`, `format_version` | `partitioned_by`, `sorted_by` |
+
+Not applicable here: postgres `indexes`; trino `format` and `format_version`
+(DuckLake always writes Parquet). The postgres advisory lock has no DuckLake
+counterpart and is replaced by the bounded conflict retry; the trino table swap
+is replaced by the single `DELETE` + `INSERT` transaction.
+
+## Layout of the code
+
+`domain/` (pure rules) <- `application/` (use cases, `LakeGateway` port) <-
+`infrastructure/` (DuckDB/DuckLake I/O, SQL rendering); `adapter.py` is the
+composition root and the entry-point target.
+
+## Tests
+
+```bash
+uv run pytest adapters/duckdb/tests -m "not integration" -v
+docker compose -f tests/smoke/duckdb-stack/docker-compose.yml up -d --wait
+uv run pytest adapters/duckdb/tests -m integration -v
+docker compose -f tests/smoke/duckdb-stack/docker-compose.yml down -v
+```

continuo_duckdb_adapter-0.1.0/README.md

@@ -0,0 +1,109 @@
+# continuo-duckdb-adapter
+
+DuckDB warehouse adapter for Continuo, running on a **DuckLake**: the catalog
+lives in Postgres and table data is Parquet on S3/MinIO, so every Kubernetes
+Job shares one transactional warehouse. Implements
+`continuo_engine_contract.port.WarehouseAdapter`; registered under the
+`continuo_engine.adapters` entry-point group as `duckdb`.
+
+## Environment
+
+| Variable | Required | Default | Meaning |
+|---|---|---|---|
+| `DUCKDB_CATALOG_HOST` | yes | | Postgres host holding the DuckLake catalog |
+| `DUCKDB_CATALOG_PORT` | no | `5432` | |
+| `DUCKDB_CATALOG_DB` | yes | | |
+| `DUCKDB_CATALOG_USER` | yes | | |
+| `DUCKDB_CATALOG_PASSWORD` | no | empty | |
+| `DUCKDB_DATA_PATH` | yes | | Data location, e.g. `s3://bucket/lake/` (fixed once the catalog exists) |
+| `DUCKDB_S3_ENDPOINT` | no | AWS | `host:port`, for MinIO and other S3-compatible stores |
+| `DUCKDB_S3_ACCESS_KEY_ID` / `DUCKDB_S3_SECRET_ACCESS_KEY` | no | credential chain | Static credentials; omit to use the AWS credential chain |
+| `DUCKDB_S3_REGION` | no | `us-east-1` | |
+| `DUCKDB_S3_URL_STYLE` | no | `path` with an endpoint, else `vhost` | `path` or `vhost` |
+| `DUCKDB_S3_USE_SSL` | no | `true` | |
+| `DUCKDB_EXTENSION_DIRECTORY` | no | DuckDB default | Where `ducklake`, `postgres`, `httpfs` and `aws` (the credential chain) are loaded from (set in the image) |
+| `DUCKDB_TEMP_DIRECTORY` | no | `.tmp` in the working directory | Where DuckDB spills larger-than-memory work; must be writable by the runtime user (the image sets `/tmp/duckdb-tmp`) |
+| `DUCKDB_DATA_INLINING_ROW_LIMIT` | no | DuckLake default | `0` writes every insert as a Parquet file instead of inlining small ones in the catalog |
+
+Settings are parsed strictly and fail fast with the variable named:
+
+- `DUCKDB_S3_USE_SSL` accepts `true`/`false`/`1`/`0`/`yes`/`no` (any case);
+  anything else is an error rather than silently meaning "true".
+- `DUCKDB_S3_ACCESS_KEY_ID` and `DUCKDB_S3_SECRET_ACCESS_KEY` are set together or
+  not at all; one without the other is rejected (it would otherwise fall back to
+  the credential chain without saying so).
+- `DUCKDB_CATALOG_PORT` and `DUCKDB_DATA_INLINING_ROW_LIMIT` take ASCII digits
+  only.
+
+## Credentials and failure modes
+
+The catalog password is handed to libpq through a private (mode 0600) temporary
+passfile, not in the connection string, so it does not appear in DuckDB's error
+text or in `duckdb_databases()`. Every DuckDB error also passes through one
+redaction point that masks the catalog password and the S3 secret (every
+spelling) before the error reaches the result block or the pod logs; host, port
+and database stay visible. The password travels inline instead (still redacted from errors) when it
+contains a line break, when `PGPASSWORD` is set in the environment (libpq would
+prefer it to any passfile), or when no temp file can be created. The `aws`
+extension is only loaded for the AWS credential-chain S3 path.
+
+A first attach to a brand-new catalog from several Jobs at once can race on
+DuckLake's metadata creation; the adapter retries exactly that failure a few
+times, and surfaces every other connection error immediately.
+
+## Physical layout (`config`)
+
+- `partitioned_by`: non-empty list of a column name, or
+  `{column, transform, buckets}` with `transform` in `identity`, `bucket`
+  (`buckets` required), `year`, `month`, `day`, `hour` (time transforms need a
+  `DATE` or `TIMESTAMP` column).
+- `sorted_by`: non-empty list of a column name, or
+  `{column, direction: asc|desc, nulls: first|last}`.
+- Columns must be declared in `output_columns`. Any other key, including
+  postgres's `indexes`, is rejected before any DDL runs.
+- Layout is applied when `ensure_table` creates the table; changing it on an
+  existing table is a no-op. `build_empty_from_columns` (the release gate)
+  always rebuilds.
+
+## Engine behaviour to know
+
+- DuckDB drops the length of `VARCHAR(n)` / `CHAR(n)`: the catalog shows plain
+  `VARCHAR`. Length is enforced by `conform()` for python nodes only, not by the
+  table.
+- DuckLake inlines small inserts into the catalog (see
+  `DUCKDB_DATA_INLINING_ROW_LIMIT`), so partitioning and sorting are applied to
+  Parquet files only after a flush; set the limit to `0` when files must be
+  laid out per write.
+- A read is one single query: top-level `PIVOT` / `UNPIVOT` statements are
+  rejected by the single-read gate. Wrap them: `SELECT * FROM (PIVOT ...)`.
+
+## Parity with the postgres and trino adapters
+
+| Behaviour | postgres | trino | duckdb |
+|---|---|---|---|
+| `drop_schema` | `DROP SCHEMA IF EXISTS ... CASCADE` | same | same (tables and views go too) |
+| `ensure_schema` under concurrency | session advisory lock | `IF NOT EXISTS`, tolerating a concurrent creation | `IF NOT EXISTS`, bounded retry on a DuckLake snapshot conflict |
+| `check_binds` | `EXPLAIN` in `BEGIN READ ONLY` | `EXPLAIN (TYPE VALIDATE)` | `EXPLAIN` in `BEGIN TRANSACTION READ ONLY`, always rolled back |
+| `ensure_table` | creates if absent, layout on create | same | same, in one transaction with the existence check |
+| `load` (replace contents) | one transaction | atomic table swap (no multi-statement transactions) | one transaction: `DELETE` then `INSERT` |
+| Physical layout keys | `indexes` | `partitioning`, `sorted_by`, `format`, `format_version` | `partitioned_by`, `sorted_by` |
+
+Not applicable here: postgres `indexes`; trino `format` and `format_version`
+(DuckLake always writes Parquet). The postgres advisory lock has no DuckLake
+counterpart and is replaced by the bounded conflict retry; the trino table swap
+is replaced by the single `DELETE` + `INSERT` transaction.
+
+## Layout of the code
+
+`domain/` (pure rules) <- `application/` (use cases, `LakeGateway` port) <-
+`infrastructure/` (DuckDB/DuckLake I/O, SQL rendering); `adapter.py` is the
+composition root and the entry-point target.
+
+## Tests
+
+```bash
+uv run pytest adapters/duckdb/tests -m "not integration" -v
+docker compose -f tests/smoke/duckdb-stack/docker-compose.yml up -d --wait
+uv run pytest adapters/duckdb/tests -m integration -v
+docker compose -f tests/smoke/duckdb-stack/docker-compose.yml down -v
+```

continuo_duckdb_adapter-0.1.0/continuo_duckdb_adapter/__init__.py

@@ -0,0 +1 @@
+"""DuckDB (DuckLake) warehouse adapter for Continuo."""

continuo_duckdb_adapter-0.1.0/continuo_duckdb_adapter/adapter.py

@@ -0,0 +1,25 @@
+"""Composition root: wires settings, session and use cases. Entry-point target.
+
+The only module that knows every layer. The use cases live in
+``application.warehouse.LakeWarehouse``; this class supplies the two hooks the
+port leaves to the concrete engine, ``required_env`` and ``from_env``.
+"""
+from __future__ import annotations
+
+from .application.warehouse import LakeWarehouse
+from .infrastructure.session import DuckLakeSession
+from .infrastructure.settings import REQUIRED_ENV, DuckLakeSettings
+
+
+class DuckDBAdapter(LakeWarehouse):
+    """WarehouseAdapter for DuckDB on a DuckLake (Postgres catalog, S3 data)."""
+
+    @classmethod
+    def required_env(cls) -> list[str]:
+        """Vars that must be non-empty before connecting."""
+        return list(REQUIRED_ENV)
+
+    @classmethod
+    def from_env(cls) -> "DuckDBAdapter":
+        """Connect from DUCKDB_* env (see DuckLakeSettings for every variable)."""
+        return cls(DuckLakeSession.connect(DuckLakeSettings.from_env()))

continuo_duckdb_adapter-0.1.0/continuo_duckdb_adapter/application/__init__.py

@@ -0,0 +1 @@
+"""Use cases. Depends on the LakeGateway port, never on infrastructure."""

continuo_duckdb_adapter-0.1.0/continuo_duckdb_adapter/application/ports.py

@@ -0,0 +1,77 @@
+"""The port the use cases need from a DuckLake. Implemented by infrastructure."""
+from __future__ import annotations
+
+from abc import ABC, abstractmethod
+from collections.abc import Sequence
+from contextlib import AbstractContextManager
+from typing import TYPE_CHECKING
+
+from ..domain.columns import ColumnDefinition
+from ..domain.identifiers import Identifier, QualifiedTable
+from ..domain.layout import TableLayout
+
+if TYPE_CHECKING:  # pragma: no cover
+    import pyarrow  # type: ignore[import-untyped]
+
+
+class LakeConflictError(Exception):
+    """A concurrent DuckLake transaction won a snapshot conflict; safe to retry."""
+
+
+class LakeGateway(ABC):
+    """Everything the warehouse use cases ask of one DuckLake connection."""
+
+    @abstractmethod
+    def transaction(self) -> AbstractContextManager[None]:
+        """BEGIN ... COMMIT; ROLLBACK (logged, never masking) if the body raises."""
+
+    @abstractmethod
+    def create_schema_if_not_exists(self, schema: Identifier) -> None: ...
+
+    @abstractmethod
+    def drop_schema_cascade(self, schema: Identifier) -> None:
+        """Drop the schema and everything in it; a no-op when absent."""
+
+    @abstractmethod
+    def table_exists(self, table: QualifiedTable) -> bool: ...
+
+    @abstractmethod
+    def drop_table_if_exists(self, table: QualifiedTable) -> None: ...
+
+    @abstractmethod
+    def create_table(
+        self,
+        table: QualifiedTable,
+        columns: Sequence[ColumnDefinition],
+        *,
+        if_not_exists: bool = False,
+    ) -> None: ...
+
+    @abstractmethod
+    def create_empty_table_as(self, table: QualifiedTable, select_sql: str) -> None:
+        """Create *table* empty, shaped by the SELECT (no terminator, single read)."""
+
+    @abstractmethod
+    def create_empty_clone(self, target: QualifiedTable, source: QualifiedTable) -> None:
+        """Create *target* empty with *source*'s shape."""
+
+    @abstractmethod
+    def apply_layout(self, table: QualifiedTable, layout: TableLayout) -> None:
+        """Apply partitioning and sort order to an existing table."""
+
+    @abstractmethod
+    def explain_read(self, sql: str) -> None:
+        """Bind-check one read without scanning data; raise if it does not bind."""
+
+    @abstractmethod
+    def fetch_arrow(self, sql: str) -> "pyarrow.Table": ...
+
+    @abstractmethod
+    def delete_all(self, table: QualifiedTable) -> None: ...
+
+    @abstractmethod
+    def insert_arrow(self, table: QualifiedTable, data: "pyarrow.Table") -> None:
+        """Append *data* in its own column order."""
+
+    @abstractmethod
+    def close(self) -> None: ...

continuo_duckdb_adapter-0.1.0/continuo_duckdb_adapter/application/warehouse.py

@@ -0,0 +1,205 @@
+"""The warehouse use cases: one class for both roles the port serves.
+
+Validation DDL (this task) and python-node data-plane I/O (next task) share one
+connection, so they are one implementation, as in the postgres and trino
+adapters. The class depends only on the LakeGateway port; ``required_env`` and
+``from_env`` stay abstract and are supplied by the composition root
+(``continuo_duckdb_adapter.adapter.DuckDBAdapter``).
+"""
+from __future__ import annotations
+
+import logging
+import time
+from collections.abc import Callable
+from typing import TYPE_CHECKING, Any
+
+from continuo_engine_contract.port import WarehouseAdapter  # type: ignore[import-untyped]
+from continuo_engine_contract.sql import ensure_single_read  # type: ignore[import-untyped]
+
+from ..domain.columns import ColumnDefinition, column_types
+from ..domain.identifiers import Identifier, QualifiedTable
+from ..domain.layout import TableLayout
+from .ports import LakeConflictError, LakeGateway
+
+if TYPE_CHECKING:  # pragma: no cover
+    import pyarrow as pa  # type: ignore[import-untyped]
+
+logger = logging.getLogger("continuo_duckdb_adapter")
+
+# DuckLake has no advisory lock. Parallel validation nodes race on creation and
+# the loser fails its COMMIT with a snapshot conflict; the retry then finds the
+# object exists. Bounded so a genuine, persistent conflict still surfaces.
+_CONFLICT_ATTEMPTS = 5
+_CONFLICT_BACKOFF_SECONDS = 0.05
+
+
+def _bare_read(sql: str) -> str:
+    """*sql* without surrounding whitespace or one trailing terminator."""
+    return sql.strip().rstrip(";").strip()
+
+
+def _columns(raw: list[dict]) -> list[ColumnDefinition]:
+    return [ColumnDefinition.from_mapping(entry) for entry in raw]
+
+
+class LakeWarehouse(WarehouseAdapter):
+    """WarehouseAdapter implemented over a LakeGateway."""
+
+    def __init__(
+        self, gateway: LakeGateway, *, sleep: Callable[[float], None] = time.sleep
+    ) -> None:
+        self._gateway = gateway
+        self._sleep = sleep
+
+    def _retrying(self, operation: Callable[[], None]) -> None:
+        for attempt in range(1, _CONFLICT_ATTEMPTS + 1):
+            try:
+                operation()
+                return
+            except LakeConflictError:
+                if attempt == _CONFLICT_ATTEMPTS:
+                    raise
+                logger.info("concurrent change conflicted (attempt %d); retrying", attempt)
+                self._sleep(_CONFLICT_BACKOFF_SECONDS * attempt)
+
+    # --- Schema lifecycle ---------------------------------------------------
+
+    def ensure_schema(self, schema: str) -> None:
+        """Idempotently create *schema*; safe under concurrent callers."""
+        target = Identifier(schema)
+        logger.info("ensuring schema %s exists", schema)
+        self._retrying(lambda: self._gateway.create_schema_if_not_exists(target))
+
+    def drop_schema(self, schema: str) -> None:
+        """Idempotently drop *schema* and everything in it; no-op if absent."""
+        logger.info("dropping candidate schema %s", schema)
+        self._gateway.drop_schema_cascade(Identifier(schema))
+
+    # --- Validation builds --------------------------------------------------
+
+    def build_empty_from_sql(self, schema: str, table: str, compiled_sql: str) -> None:
+        """Create ``schema.table`` empty, shaped by the compiled SELECT.
+
+        The parse gate runs first: DuckDB executes every ``;``-separated
+        statement handed to it, so a stacked statement would run for real.
+        """
+        ensure_single_read(compiled_sql, dialect="duckdb")
+        inner = _bare_read(compiled_sql)
+        target = QualifiedTable.of(schema, table)
+        with self._gateway.transaction():
+            self._gateway.drop_table_if_exists(target)
+            self._gateway.create_empty_table_as(target, inner)
+
+    def clone_empty_from_prod(self, candidate_schema: str, prod_schema: str, table: str) -> None:
+        """Create ``candidate_schema.table`` empty, shaped like ``prod_schema.table``."""
+        target = QualifiedTable.of(candidate_schema, table)
+        source = QualifiedTable.of(prod_schema, table)
+        with self._gateway.transaction():
+            self._gateway.drop_table_if_exists(target)
+            self._gateway.create_empty_clone(target, source)
+
+    def build_empty_from_columns(
+        self, schema: str, table: str, columns: list[dict], config: dict
+    ) -> None:
+        """Create ``schema.table`` empty from declared typed columns (drop-then-create).
+
+        Types and layout are validated before the first statement; the rebuild is
+        one transaction, so a failure leaves the prior table in place.
+        """
+        defs = _columns(columns)
+        layout = TableLayout.from_config(config, column_types(defs))
+        target = QualifiedTable.of(schema, table)
+        with self._gateway.transaction():
+            self._gateway.drop_table_if_exists(target)
+            self._gateway.create_table(target, defs)
+            if not layout.is_empty:
+                self._gateway.apply_layout(target, layout)
+
+    def check_binds(self, sql: str) -> None:
+        """Verify *sql* binds against current schema state; scans no data.
+
+        The parse gate runs first: DuckDB executes every ``;``-separated
+        statement handed to it, so a stacked statement would run for real.
+        """
+        ensure_single_read(sql, dialect="duckdb")
+        inner = _bare_read(sql)
+        logger.info("bind-checking read via EXPLAIN")
+        self._gateway.explain_read(inner)
+
+    # --- Python-node data plane ---------------------------------------------
+
+    def fetch(self, sql: str) -> "pa.Table":
+        """Execute one declared read and return the result as an Arrow table.
+
+        Not re-gated here, matching the postgres adapter: the harness loads the
+        contract with ``check_reads=False``, so ``check_binds`` is the gated path
+        (single-read parse gate, then a bind check).
+        """
+        data = self._gateway.fetch_arrow(sql)
+        seen: set[str] = set()
+        duplicates: set[str] = set()
+        for name in data.schema.names:
+            if name in seen:
+                duplicates.add(name)
+            seen.add(name)
+        if duplicates:
+            raise ValueError(f"duplicate column name(s) in SELECT result: {sorted(duplicates)!r}")
+        return data
+
+    @classmethod
+    def validate_config(cls, config: dict[str, Any] | None, column_names: list[str]) -> None:
+        """Validate *config* against this engine's vocabulary, without connecting.
+
+        The harness calls this right after selecting the node so a malformed
+        ``config`` fails in the first second, not after the script has run. It
+        only knows column names, so the DATE/TIMESTAMP check for time transforms
+        is deferred; ``ensure_table`` runs the complete check and remains the
+        enforcement point (the harness skips adapters without this method; see
+        docs/boundary-contract.md section 13.4).
+        """
+        TableLayout.from_config(config, {name: None for name in column_names})
+
+    def ensure_table(
+        self,
+        schema: str,
+        table: str,
+        columns: list[dict[str, Any]],
+        *,
+        config: dict[str, Any],
+    ) -> None:
+        """Create the table if absent; layout is applied only when this call creates it.
+
+        Validation comes first, so a bad config or type emits no statement. The
+        existence check and the create share one transaction; a concurrent
+        creator makes one side fail with a conflict, and the retry then finds
+        the table and returns.
+        """
+        defs = _columns(columns)
+        layout = TableLayout.from_config(config, column_types(defs))
+        self.ensure_schema(schema)
+        target = QualifiedTable.of(schema, table)
+        self._retrying(lambda: self._create_if_absent(target, defs, layout))
+
+    def _create_if_absent(
+        self, target: QualifiedTable, defs: list[ColumnDefinition], layout: TableLayout
+    ) -> None:
+        with self._gateway.transaction():
+            if self._gateway.table_exists(target):
+                return
+            logger.info("creating table %s.%s", target.schema.name, target.table.name)
+            self._gateway.create_table(target, defs)
+            if not layout.is_empty:
+                self._gateway.apply_layout(target, layout)
+
+    def load(self, schema: str, table: str, data: "pa.Table") -> None:
+        """Atomically replace the table's contents with *data* (one transaction)."""
+        target = QualifiedTable.of(schema, table)
+        with self._gateway.transaction():
+            self._gateway.delete_all(target)
+            if data.num_rows:
+                logger.info("loading %d row(s) into %s.%s", data.num_rows, schema, table)
+                self._gateway.insert_arrow(target, data)
+
+    def close(self) -> None:
+        """Release the underlying connection."""
+        self._gateway.close()

continuo_duckdb_adapter-0.1.0/continuo_duckdb_adapter/domain/__init__.py

@@ -0,0 +1 @@
+"""Pure domain rules: no duckdb, psycopg2, pyarrow or boto imports."""

continuo_duckdb_adapter-0.1.0/continuo_duckdb_adapter/domain/columns.py

@@ -0,0 +1,46 @@
+"""Typed column definitions, validated against the contract's SQL type grammar."""
+from __future__ import annotations
+
+import re
+from collections.abc import Mapping, Sequence
+from dataclasses import dataclass
+from typing import Any
+
+from continuo_engine_contract.types import validate_column_type  # type: ignore[import-untyped]
+
+from .identifiers import Identifier
+
+_TEMPORAL = re.compile(r"^(TIMESTAMP|DATE)\Z", re.IGNORECASE | re.ASCII)
+
+
+def is_temporal_type(type_str: str) -> bool:
+    """True for the contract types that support year/month/day/hour partitioning."""
+    return _TEMPORAL.match(type_str) is not None
+
+
+@dataclass(frozen=True)
+class ColumnDefinition:
+    """One declared output column. ``type`` is validated on construction because
+    the text is interpolated into DDL; the name is quoted by the renderer."""
+
+    name: Identifier
+    type: str
+    nullable: bool = True
+
+    def __post_init__(self) -> None:
+        validate_column_type(self.type)
+
+    @property
+    def is_temporal(self) -> bool:
+        return is_temporal_type(self.type)
+
+    @classmethod
+    def from_mapping(cls, raw: Mapping[str, Any]) -> "ColumnDefinition":
+        if not isinstance(raw, Mapping) or "name" not in raw or "type" not in raw:
+            raise ValueError(f"column must be a mapping with 'name' and 'type', got {raw!r}")
+        return cls(Identifier(raw["name"]), raw["type"], bool(raw.get("nullable", True)))
+
+
+def column_types(columns: Sequence[ColumnDefinition]) -> dict[str, str]:
+    """Declared column name -> contract type, for layout validation."""
+    return {column.name.name: column.type for column in columns}

continuo_duckdb_adapter-0.1.0/continuo_duckdb_adapter/domain/identifiers.py

@@ -0,0 +1,33 @@
+"""Identifier value objects.
+
+They validate and carry names. Turning a name into quoted SQL is the job of the
+infrastructure layer's ``DdlRenderer``, so this module stays engine-neutral.
+"""
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+
+@dataclass(frozen=True)
+class Identifier:
+    """A schema, table or column name, validated but never quoted."""
+
+    name: str
+
+    def __post_init__(self) -> None:
+        if not isinstance(self.name, str) or not self.name:
+            raise ValueError(f"identifier must be a non-empty string, got {self.name!r}")
+        if "\x00" in self.name:
+            raise ValueError("identifier must not contain a NUL character")
+
+
+@dataclass(frozen=True)
+class QualifiedTable:
+    """A table addressed by schema and name."""
+
+    schema: Identifier
+    table: Identifier
+
+    @classmethod
+    def of(cls, schema: str, table: str) -> "QualifiedTable":
+        return cls(Identifier(schema), Identifier(table))