interlaced 2.3.0__tar.gz → 2.4.0__tar.gz

{interlaced-2.3.0/src/interlaced.egg-info → interlaced-2.4.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: interlaced
-Version: 2.3.0
-Summary: Python/SQL-first data platform: transformation, built-in orchestration, and durable streaming ingestion
+Version: 2.4.0
+Summary: Python and SQL models in one DAG. Transformation, orchestration and durable streaming in one process, on DuckDB and Postgres.
 Author-email: Mark <mark@interlace.sh>
 License-Expression: MIT
 Project-URL: Homepage, https://interlace.sh
@@ -13,8 +13,10 @@ Keywords: data,pipeline,orchestration,transformation,etl,streaming,dbt,sqlmesh,d
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Topic :: Software Development :: Build Tools
+Classifier: Topic :: Database
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: SQL
+Classifier: Operating System :: POSIX :: Linux
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -64,15 +66,41 @@ Dynamic: license-file
 
 # interlace
 
-**Python/SQL-first data platform: transformation, orchestration, and durable streaming — one process.**
+**Python and SQL models are the same kind of node in one DAG.**
 
-interlace is an independent, MIT-licensed alternative to dbt/SQLMesh that also replaces the
-orchestrator (no Airflow) and the ingestion layer (Cloudflare-Pipelines-style durable streams).
-Models are `.sql` files or Python functions; state is versioned snapshots with virtual
+A `.py` model sits mid-graph with SQL either side, in both directions, with no bridge and no
+separate runtime — running in-process on DuckDB and Postgres, not only on a cloud warehouse.
+The Python model stays a plain function: call it in a test with no warehouse and no session.
+
+How that compares with dbt and SQLMesh, including where they are ahead, is on
+[interlace.sh/why](https://interlace.sh/why).
+
+```sql
+-- models/raw_events.sql                    SQL
+SELECT event_id, user_id, kind, amount, country, ts FROM read_parquet('events/*.parquet')
+```
+```python
+# models/enriched_events.py                 Python, mid-DAG
+@model()                    # the parameter name IS the dependency — no depends_on
+def enriched_events(raw_events):
+    for batch in raw_events.reader():       # Arrow in, Arrow out, bounded memory
+        yield add_revenue(batch)
+```
+```sql
+-- models/event_summary.sql                 SQL again, straight over the Python
+SELECT country, count(*) FILTER (WHERE is_conversion) AS conversions
+FROM enriched_events GROUP BY country
+```
+
+`interlace init` scaffolds exactly this shape, runnable, with no external source.
+
+That is the wedge. The rest is the reveal: interlace is an independent, MIT-licensed alternative
+to dbt/SQLMesh that also replaces the orchestrator (no Airflow) and the ingestion layer
+(Cloudflare-Pipelines-style durable streams). State is versioned snapshots with virtual
 environments and a terraform-style plan/apply; everything runs in a single daemon on
-DuckDB + DuckLake by default.
+DuckDB by default (DuckLake one config line away).
 
-> **Status: 2.0.** Requires Python 3.12+.
+> **2.x — see [releases](https://github.com/interlace-sh/interlace/releases).** Requires Python 3.12+.
 > The package is published to PyPI as **`interlaced`**; the import name and CLI are `interlace`.
 
 ```bash
@@ -80,6 +108,13 @@ pip install 'interlaced[service]'   # the CLI + daemon; core CLI only: pip insta
 # more extras: [adbc] postgres/redshift · [spark] · [polars] · [all]
 ```
 
+> **Platforms.** Developed on Linux; CI runs Linux only. Nothing in the codebase is
+> platform-specific — no `fork`, no signal handling, no POSIX-only calls, no shelling out — and
+> every dependency ships macOS and Windows wheels, so both are expected to work. But
+> **neither is tested**, so treat them as unverified rather than supported. If you run interlace
+> on macOS or Windows, please open an issue either way; that is the fastest route to changing
+> this paragraph.
+
 ## Sixty seconds
 
 ```bash
@@ -210,7 +245,7 @@ prod, so a dev apply never writes to a live external table (opt in with
 
 ## Multi-engine
 
-Models run on **named engines**: DuckDB/DuckLake by default, Postgres natively over ADBC
+Models run on **named engines**: DuckDB by default (DuckLake opt-in), Postgres natively over ADBC
 (`pip install 'interlaced[adbc]'`), Spark (beta, `[spark]` extra), plus alpha adapters for
 MotherDuck, Redshift, Snowflake and BigQuery (wired and dialect-correct, not yet run against a
 live account), with per-model pinning:
@@ -252,7 +287,8 @@ other processes (CLI runs, ad-hoc DuckDB clients) then share it concurrently by
 
 - The IR is a **sqlglot AST**; the wire format is an **Arrow RecordBatchReader**; strategies
   are AST builders and dialect appears only at `transpile()`.
-- Storage defaults to **DuckLake** (Parquet + SQL catalog) opened as DuckDB's primary database.
+- Storage defaults to a plain **DuckDB** file; **DuckLake** (Parquet + SQL catalog, and
+  concurrent writers) is one `database: ducklake:…` line away.
 - Control plane (snapshots, intervals, queue, events, keys) is **SQLite WAL**; Postgres is the
   scale-out swap.
 - Streams live in their own durable log; the materializer commits data + watermark in one
@@ -269,7 +305,7 @@ Toolchain is pinned with [proto](https://moonrepo.dev/proto), tasks run via
 ```bash
 proto install
 moon run interlace:sync      # install deps
-moon run interlace:test      # 350+ tests
+moon run interlace:test      # 500+ tests
 moon run interlace:check     # black + ruff (CI equivalent)
 moon run interlace:typecheck # mypy
 ```

{interlaced-2.3.0 → interlaced-2.4.0}/README.md

@@ -1,14 +1,40 @@
 # interlace
 
-**Python/SQL-first data platform: transformation, orchestration, and durable streaming — one process.**
+**Python and SQL models are the same kind of node in one DAG.**
 
-interlace is an independent, MIT-licensed alternative to dbt/SQLMesh that also replaces the
-orchestrator (no Airflow) and the ingestion layer (Cloudflare-Pipelines-style durable streams).
-Models are `.sql` files or Python functions; state is versioned snapshots with virtual
+A `.py` model sits mid-graph with SQL either side, in both directions, with no bridge and no
+separate runtime — running in-process on DuckDB and Postgres, not only on a cloud warehouse.
+The Python model stays a plain function: call it in a test with no warehouse and no session.
+
+How that compares with dbt and SQLMesh, including where they are ahead, is on
+[interlace.sh/why](https://interlace.sh/why).
+
+```sql
+-- models/raw_events.sql                    SQL
+SELECT event_id, user_id, kind, amount, country, ts FROM read_parquet('events/*.parquet')
+```
+```python
+# models/enriched_events.py                 Python, mid-DAG
+@model()                    # the parameter name IS the dependency — no depends_on
+def enriched_events(raw_events):
+    for batch in raw_events.reader():       # Arrow in, Arrow out, bounded memory
+        yield add_revenue(batch)
+```
+```sql
+-- models/event_summary.sql                 SQL again, straight over the Python
+SELECT country, count(*) FILTER (WHERE is_conversion) AS conversions
+FROM enriched_events GROUP BY country
+```
+
+`interlace init` scaffolds exactly this shape, runnable, with no external source.
+
+That is the wedge. The rest is the reveal: interlace is an independent, MIT-licensed alternative
+to dbt/SQLMesh that also replaces the orchestrator (no Airflow) and the ingestion layer
+(Cloudflare-Pipelines-style durable streams). State is versioned snapshots with virtual
 environments and a terraform-style plan/apply; everything runs in a single daemon on
-DuckDB + DuckLake by default.
+DuckDB by default (DuckLake one config line away).
 
-> **Status: 2.0.** Requires Python 3.12+.
+> **2.x — see [releases](https://github.com/interlace-sh/interlace/releases).** Requires Python 3.12+.
 > The package is published to PyPI as **`interlaced`**; the import name and CLI are `interlace`.
 
 ```bash
@@ -16,6 +42,13 @@ pip install 'interlaced[service]'   # the CLI + daemon; core CLI only: pip insta
 # more extras: [adbc] postgres/redshift · [spark] · [polars] · [all]
 ```
 
+> **Platforms.** Developed on Linux; CI runs Linux only. Nothing in the codebase is
+> platform-specific — no `fork`, no signal handling, no POSIX-only calls, no shelling out — and
+> every dependency ships macOS and Windows wheels, so both are expected to work. But
+> **neither is tested**, so treat them as unverified rather than supported. If you run interlace
+> on macOS or Windows, please open an issue either way; that is the fastest route to changing
+> this paragraph.
+
 ## Sixty seconds
 
 ```bash
@@ -146,7 +179,7 @@ prod, so a dev apply never writes to a live external table (opt in with
 
 ## Multi-engine
 
-Models run on **named engines**: DuckDB/DuckLake by default, Postgres natively over ADBC
+Models run on **named engines**: DuckDB by default (DuckLake opt-in), Postgres natively over ADBC
 (`pip install 'interlaced[adbc]'`), Spark (beta, `[spark]` extra), plus alpha adapters for
 MotherDuck, Redshift, Snowflake and BigQuery (wired and dialect-correct, not yet run against a
 live account), with per-model pinning:
@@ -188,7 +221,8 @@ other processes (CLI runs, ad-hoc DuckDB clients) then share it concurrently by
 
 - The IR is a **sqlglot AST**; the wire format is an **Arrow RecordBatchReader**; strategies
   are AST builders and dialect appears only at `transpile()`.
-- Storage defaults to **DuckLake** (Parquet + SQL catalog) opened as DuckDB's primary database.
+- Storage defaults to a plain **DuckDB** file; **DuckLake** (Parquet + SQL catalog, and
+  concurrent writers) is one `database: ducklake:…` line away.
 - Control plane (snapshots, intervals, queue, events, keys) is **SQLite WAL**; Postgres is the
   scale-out swap.
 - Streams live in their own durable log; the materializer commits data + watermark in one
@@ -205,7 +239,7 @@ Toolchain is pinned with [proto](https://moonrepo.dev/proto), tasks run via
 ```bash
 proto install
 moon run interlace:sync      # install deps
-moon run interlace:test      # 350+ tests
+moon run interlace:test      # 500+ tests
 moon run interlace:check     # black + ruff (CI equivalent)
 moon run interlace:typecheck # mypy
 ```

{interlaced-2.3.0 → interlaced-2.4.0}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "interlaced"
-version = "2.3.0"
-description = "Python/SQL-first data platform: transformation, built-in orchestration, and durable streaming ingestion"
+version = "2.4.0"
+description = "Python and SQL models in one DAG. Transformation, orchestration and durable streaming in one process, on DuckDB and Postgres."
 readme = "README.md"
 requires-python = ">=3.12"
 license = "MIT"
@@ -17,8 +17,12 @@ classifiers = [
     "Development Status :: 5 - Production/Stable",
     "Intended Audience :: Developers",
     "Topic :: Software Development :: Build Tools",
+    "Topic :: Database",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: SQL",
+    # Linux is what CI runs and what this is developed on. Nothing in the codebase
+    # is platform-specific, but macOS and Windows are untested — see the README.
+    "Operating System :: POSIX :: Linux",
 ]
 
 # Core: the IR + engine + state + plan spine (Phase 1). Everything is Arrow- and

{interlaced-2.3.0 → interlaced-2.4.0}/src/interlace/config/config.py

@@ -84,7 +84,7 @@ class EngineConfig(BaseModel):
     declaring them fails at open until an adapter ships.
     """
 
-    type: str = "ducklake"
+    type: str = "duckdb"
     # Path / URI for DuckDB-family engines. Also accepted on the project top level
     # as ``database:`` (synthesised into the ``default`` engine).
     database: str | None = None
@@ -117,13 +117,14 @@ class ProjectConfig(BaseModel):
     default_engine: str = "default"
     engines: dict[str, EngineConfig] = Field(default_factory=dict)
     state_path: str = ".interlace/state.db"  # SQLite control-plane database
-    # The warehouse. Default is DuckLake (Parquet data + SQL catalog) via DuckDB.
-    # Also accepted: a DuckLake catalog hosted in a SQL database
-    # ("ducklake:postgres:dbname=... host=..." — pair with data_path/metadata_schema),
-    # a plain DuckDB file path, ":memory:", or "quack:<host>:<port>" to connect to a
-    # warehouse served by `interlace serve --quack`.
+    # The warehouse. Default is a plain DuckDB file — simplest, single-process.
+    # Also accepted: a DuckLake catalog (``ducklake:.interlace/warehouse.ducklake``, or
+    # hosted in a SQL DB: ``ducklake:postgres:dbname=... host=...`` — pair with
+    # data_path/metadata_schema) which serialises catalog writes so `interlace serve`
+    # and a separate CLI can share the warehouse concurrently; ":memory:"; or
+    # "quack:<host>:<port>" to connect to a warehouse served by `interlace serve --quack`.
     # When ``engines.default`` is not set, these top-level fields synthesise it.
-    database: str = "ducklake:.interlace/warehouse.ducklake"
+    database: str = ".interlace/warehouse.duckdb"
     # The warehouse catalog's ATTACH alias (defaults to ``name``). Set it when a
     # schema inside the warehouse shares the project name — see EngineConfig.alias.
     alias: str | None = None

{interlaced-2.3.0 → interlaced-2.4.0}/src/interlace/dsl/decorators.py

@@ -114,6 +114,18 @@ class ModelDef:
     schedule: dict[str, str] | None = None  # {"cron": "0 * * * *"} or {"every": "5m"} for `interlace serve`
     checks: tuple[CheckSpec, ...] = ()  # data-quality checks; error severity gates promotion
 
+    def __post_init__(self) -> None:
+        # `@model(checks=…)` normalises through parse_checks, but a ModelDef built
+        # directly — the dynamic-model path, which is what generated models and dbt
+        # migrations use — stored the dicts raw and only failed at compile time with
+        # `AttributeError: 'dict' object has no attribute 'type'`. Normalise here too,
+        # so one spelling works on both surfaces and a bad check fails at declaration.
+        # Passed through as-is, not as list(...): parse_checks already handles a bare
+        # CheckSpec and reports a non-list clearly, both of which list() would mangle
+        # (TypeError on a CheckSpec; a dict silently degraded to its keys). Always run,
+        # so `checks=[]` normalises to the declared tuple rather than staying a list.
+        self.checks = parse_checks(self.checks, self.name)
+
     @property
     def is_terminal(self) -> bool:
         """A terminal model delivers into an external destination (table/file):

{interlaced-2.3.0 → interlaced-2.4.0}/src/interlace/engines/duckdb.py

@@ -28,6 +28,7 @@ from __future__ import annotations
 
 import asyncio
 import contextlib
+import re
 import threading
 from collections.abc import Iterator, Sequence
 from uuid import uuid4
@@ -38,6 +39,7 @@ import tenacity
 from sqlglot import exp
 
 from interlace.engines.base import EngineAdapter, EngineCaps, LoadMode
+from interlace.exceptions import ConfigurationError
 from interlace.ir.relation import TableRef
 
 _DUCKDB_CAPS = EngineCaps(
@@ -58,6 +60,39 @@ _commit_retry = tenacity.retry(
 )
 
 
+@contextlib.contextmanager
+def _clean_lock_error(database: str) -> Iterator[None]:
+    """Translate DuckDB's file-lock conflict into one actionable line.
+
+    A DuckDB/DuckLake database is held by a single process. The common way to hit
+    that is running a CLI command (`interlace query`, `plan`, `apply`) while
+    `interlace serve` is up — an obvious thing to do, since serve is the daemon
+    and query is the console's CLI counterpart. Raw, that surfaces as a dozen
+    frames ending in `duckdb.IOException`, which names neither the cause nor the
+    fix. The fix is `--quack`, and it is already documented; the error just never
+    said so.
+    """
+    try:
+        yield
+    except duckdb.IOException as exc:
+        message = str(exc)
+        # "Conflicting lock is held in <exe> (PID n)" is the Linux rendering — DuckDB
+        # names the holder from /proc/locks. Elsewhere the message is the bare "Could
+        # not set lock on file", and a same-process conflict says "already held", so
+        # match all three or macOS/Windows keep the raw traceback.
+        if not any(marker in message for marker in ("Conflicting lock", "Could not set lock", "already held")):
+            raise
+        holder = re.search(r"\(PID (\d+)\)", message)
+        held_by = f" (PID {holder.group(1)})" if holder else ""
+        raise ConfigurationError(
+            f"the warehouse {database!r} is already open in another process{held_by}. "
+            "DuckDB allows one process at a time — stop `interlace serve`, or serve the "
+            "warehouse over the quack protocol (`interlace serve --quack quack:localhost:4213`) "
+            "and point this process at `database: quack:localhost:4213` to share it.",
+            details={"database": database},
+        ) from None
+
+
 def _affected(cur: duckdb.DuckDBPyConnection) -> int:
     """DML/CTAS/COPY return their affected-row count as a one-cell result; DDL returns
     nothing. Never raises — row stats are best-effort decoration, not correctness."""
@@ -108,7 +143,9 @@ class DuckDBAdapter(EngineAdapter):
 
     @classmethod
     def connect(cls, path: str) -> DuckDBAdapter:
-        return cls(duckdb.connect(path), serialise_writes=path.startswith("ducklake:"))
+        with _clean_lock_error(path):
+            conn = duckdb.connect(path)
+        return cls(conn, serialise_writes=path.startswith("ducklake:"))
 
     @classmethod
     def connect_ducklake(
@@ -139,7 +176,8 @@ class DuckDBAdapter(EngineAdapter):
         options_sql = f" ({', '.join(options)})" if options else ""
         escaped = catalog.replace("'", "''")
         alias_sql = exp.to_identifier(alias).sql("duckdb")
-        conn.execute(f"ATTACH IF NOT EXISTS '{escaped}' AS {alias_sql}{options_sql}")
+        with _clean_lock_error(catalog):
+            conn.execute(f"ATTACH IF NOT EXISTS '{escaped}' AS {alias_sql}{options_sql}")
         conn.execute(f"USE {alias_sql}")
         # LOAD, secrets, and ATTACH are all instance-wide — they carry into every
         # cursor and must run ONCE (re-running CREATE OR REPLACE SECRET per cursor
@@ -162,10 +200,23 @@ class DuckDBAdapter(EngineAdapter):
         with contextlib.suppress(Exception):
             self._conn.interrupt()
 
+    def search_files_from(self, directory: str) -> None:
+        """Resolve relative read paths (``read_csv_auto('seeds/x.csv')``) against
+        ``directory`` — the project root — as well as the process CWD.
+
+        Additive: a CWD-relative path still resolves, so this only ever widens what a
+        model can find. GLOBAL scope because a plain ``SET`` is session-scoped and would
+        not reach the per-task cursors that actually run the queries. Reads only —
+        ``COPY`` targets stay CWD-relative, which is why exports resolve their own paths
+        against the root (``plan.apply._resolve_export_path``)."""
+        escaped = directory.replace("'", "''")
+        self._conn.execute(f"SET GLOBAL file_search_path='{escaped}'")
+
     def attach(self, alias: str, uri: str) -> None:
         """ATTACH another database (duckdb/sqlite/postgres/... URI) under ``alias``."""
         escaped = uri.replace("'", "''")
-        self._conn.execute(f"ATTACH IF NOT EXISTS '{escaped}' AS {exp.to_identifier(alias).sql('duckdb')}")
+        with _clean_lock_error(uri):  # attaching a held duckdb/ducklake file conflicts just like opening one
+            self._conn.execute(f"ATTACH IF NOT EXISTS '{escaped}' AS {exp.to_identifier(alias).sql('duckdb')}")
         self._attached.append(alias)
         if uri.startswith("ducklake:"):  # writes may now reach a DuckLake catalog (e.g. table sinks)
             if isinstance(self._write_lock, contextlib.nullcontext):

{interlaced-2.3.0 → interlaced-2.4.0}/src/interlace/plan/apply.py

@@ -14,6 +14,7 @@ from __future__ import annotations
 import asyncio
 import contextlib
 import logging
+import re
 import time
 from collections.abc import Callable, Mapping
 from dataclasses import dataclass, field, replace
@@ -107,9 +108,9 @@ async def _merge_python_output(
     pre_statements: list[exp.Expression] = []
     columns: list[str] | None = None
     if exists:
-        pre_statements, source, columns = await _align_stage_to_target(
-            engine, stage, target, exclude=strategy.managed_columns
-        )
+        alignment = await _align_stage_to_target(engine, stage, target, strategy)
+        pre_statements = alignment.pre_statements
+        source, columns = alignment.for_strategy(strategy)
     elif isinstance(strategy, HashMerge):
         # hash_merge builds its _hash from the column list; on the first build the target
         # doesn't exist yet (so no align pass ran), so take the columns from the staged output
@@ -127,21 +128,54 @@ async def _merge_python_output(
     return written, interval
 
 
+@dataclass(frozen=True)
+class Alignment:
+    """A staged source fitted to an existing target, in both widths.
+
+    ``source``/``columns`` cover the target's full column set — what a strategy that
+    binds positionally or compares whole rows (``INSERT ... SELECT *``, ``EXCEPT``)
+    needs, with columns the model doesn't produce NULL-filled in.
+    ``produced_source``/``produced`` cover only the model's own columns; a strategy
+    that names its columns takes these instead, so the rest of the target is left
+    alone on an update and takes its DEFAULTs on an insert. ``unproduced`` names the
+    difference — target columns this model has no value for."""
+
+    pre_statements: list[exp.Expression]
+    source: exp.Query
+    columns: list[str]
+    produced_source: exp.Query
+    produced: list[str]
+    unproduced: list[str]
+
+    def for_strategy(self, strategy: Strategy) -> tuple[exp.Query, list[str]]:
+        """The (source, column list) pair this strategy should be planned against."""
+        if strategy.writes_named_columns:
+            return self.produced_source, self.produced
+        return self.source, self.columns
+
+
 async def _align_stage_to_target(
-    engine: EngineAdapter, stage: TableRef, target: TableRef, exclude: tuple[str, ...] = ()
-) -> tuple[list[exp.Expression], exp.Query, list[str]]:
+    engine: EngineAdapter, stage: TableRef, target: TableRef, strategy: Strategy
+) -> Alignment:
     """Align a staged source to an EXISTING target: additive ALTERs for new columns,
-    widening type promotions in place, and a projection over the stage matching the
-    target's final column set (NULL-fill vanished columns, cast type drift). Returns
-    (pre-statements, aligned source select, target column order).
+    widening type promotions in place, and projections over the stage fitted to the
+    target's final column set (NULL-fill vanished columns, cast type drift).
 
-    ``exclude`` names strategy-managed bookkeeping columns (e.g. scd2's validity
-    pair): they live on the target but never in the model's output, so they must
-    not be NULL-filled into the aligned source — the strategy owns them."""
+    The strategy's managed bookkeeping columns (e.g. scd's validity pair, hash_merge's
+    ``_hash``) live on the target but never in the model's output, so they are held out
+    of both projections — the strategy owns them, and must find them already there."""
     target_columns = await engine.describe(target)
+    _require_managed_columns(strategy, target, target_columns)
+    exclude = strategy.managed_columns
     for column in exclude:
         target_columns.pop(column, None)
     stage_columns = await engine.describe(stage)
+    if clash := [c for c in stage_columns if c in exclude]:
+        raise PlanError(
+            f"model output column {clash[0]!r} collides with a column the "
+            f"{_strategy_name(strategy)} strategy manages — rename it in the model",
+            details={"target": target.to_expr().sql(), "columns": clash},
+        )
     target_expr = target.to_expr()
     pre_statements: list[exp.Expression] = []
     for column, dtype in stage_columns.items():
@@ -170,16 +204,60 @@ async def _align_stage_to_target(
     # Any remaining type mismatch (e.g. a numeric field arriving as VARCHAR) is cast
     # to the target's type — deterministic, and loudly fails the run on values that
     # genuinely don't convert rather than silently corrupting the column.
-    projection = []
+    projection: list[exp.Expression] = []
+    produced_projection: list[exp.Expression] = []
+    produced: list[str] = []
+    unproduced: list[str] = []
     for column, dtype in target_columns.items():
         if column not in stage_columns:
             projection.append(exp.alias_(exp.Cast(this=exp.Null(), to=exp.DataType.build(dtype)), column))
-        elif stage_columns[column] != dtype:
-            projection.append(exp.alias_(exp.Cast(this=exp.column(column), to=exp.DataType.build(dtype)), column))
+            unproduced.append(column)
+            continue
+        if stage_columns[column] != dtype:
+            fitted: exp.Expression = exp.alias_(exp.Cast(this=exp.column(column), to=exp.DataType.build(dtype)), column)
         else:
-            projection.append(exp.column(column))
-    source = exp.select(*projection).from_(stage.to_expr())
-    return pre_statements, source, list(target_columns)
+            fitted = exp.column(column)
+        projection.append(fitted)
+        produced_projection.append(fitted.copy())
+        produced.append(column)
+    return Alignment(
+        pre_statements=pre_statements,
+        source=exp.select(*projection).from_(stage.to_expr()),
+        columns=list(target_columns),
+        produced_source=exp.select(*produced_projection).from_(stage.to_expr()),
+        produced=produced,
+        unproduced=unproduced,
+    )
+
+
+def _strategy_name(strategy: Strategy) -> str:
+    """The strategy's config keyword (``HashMerge`` -> ``hash_merge``), for messages."""
+    return re.sub(r"(?<!^)(?=[A-Z])", "_", type(strategy).__name__).lower()
+
+
+def _require_managed_columns(strategy: Strategy, target: TableRef, target_columns: Mapping[str, str]) -> None:
+    """A strategy that keeps bookkeeping columns can only take over a table that already
+    carries them. Missing ones would otherwise surface as an engine binder error deep in
+    the strategy's UPDATE — most likely an externally-owned table that interlace did not
+    create, or a strategy swapped under an existing delivery target.
+
+    Nothing to say when the target describes empty: it isn't really there (a snapshot row
+    can outlive its table), so the caller is not taking over anything. Matching is
+    case-insensitive — an engine that folds unquoted identifiers (Snowflake, BigQuery)
+    stores ``_hash`` as ``_HASH`` and would otherwise trip this on every run."""
+    if not target_columns:
+        return
+    present = {c.casefold() for c in target_columns}
+    missing = [c for c in strategy.managed_columns if c.casefold() not in present]
+    if missing:
+        name = _strategy_name(strategy)
+        qualified = target.to_expr().sql()
+        raise PlanError(
+            f"{name} needs its bookkeeping column(s) {', '.join(missing)} on {qualified}, which already "
+            f"exists without them — interlace did not create this table. Use strategy: merge, or add "
+            f"the column(s) to it.",
+            details={"target": qualified, "missing": missing, "strategy": name},
+        )
 
 
 async def _deliver_table(
@@ -194,10 +272,11 @@ async def _deliver_table(
 
     The external target is never dropped (grants and readers survive). When it already
     exists the source is staged in the warehouse and aligned to the target (additive
-    ALTERs, widening, NULL-fill, casts) so a model that grows or reorders columns evolves
-    the destination instead of breaking it or positionally corrupting it. The insert
-    binds positionally against the aligned source, which reproduces the target's column
-    order exactly.
+    ALTERs, widening, casts) so a model that grows or reorders columns evolves the
+    destination instead of breaking it or positionally corrupting it. A strategy that
+    names its columns is then planned against the model's own columns alone, leaving the
+    rest of the target untouched (see :class:`Alignment`); a whole-row one gets the source
+    widened to the target with NULLs, and binds positionally in the target's order.
 
     Two cases skip staging and run the strategy directly against the target: the first
     delivery (the ensure-create matches the source), and any windowed ``incremental``
@@ -213,9 +292,23 @@ async def _deliver_table(
     stage = TableRef(schema=XFER_SCHEMA, name=f"{model.name}__sink_stage")
     await engine.create_schema(stage.schema)
     await engine.execute(exp.Create(this=stage.to_expr(), kind="TABLE", replace=True, expression=resolved.copy()))
-    pre_statements, aligned, columns = await _align_stage_to_target(
-        engine, stage, target, exclude=strategy.managed_columns
-    )
+    alignment = await _align_stage_to_target(engine, stage, target, strategy)
+    pre_statements = alignment.pre_statements
+    aligned, columns = alignment.for_strategy(strategy)
+    if alignment.unproduced and not strategy.writes_named_columns:
+        # A whole-row strategy rewrites rows entire, so any column the model doesn't
+        # produce is reset on every run — and the deleting ones (replace, full_merge)
+        # drop rows it doesn't supply at all. Fine when interlace owns the table (this
+        # is just a widened source); destructive when it shares it with another writer.
+        logger.warning(
+            "%s: strategy %s writes whole rows into %s — it resets columns this model does not "
+            "produce (%s) on every delivery, and (replace, full_merge) deletes rows it does not "
+            "supply. Use merge, hash_merge or append if another writer owns part of this table.",
+            model.name,
+            model.strategy,
+            target.to_expr().sql(),
+            ", ".join(alignment.unproduced),
+        )
     statements = strategy.plan_statements(SqlRelation(ast=aligned), target, engine.caps, interval, columns)
     # One transaction may write only ONE attached database: the delivery batch writes the
     # external target; the stage lives in the warehouse and is dropped separately (a

{interlaced-2.3.0 → interlaced-2.4.0}/src/interlace/project.py

@@ -182,7 +182,7 @@ class Project:
             if not database.startswith("md:"):
                 database = f"md:{database}"
             return DuckDBAdapter.connect(database)
-        database = cfg.database or "ducklake:.interlace/warehouse.ducklake"
+        database = cfg.database or ".interlace/warehouse.duckdb"
         if cfg.type == "quack" or database.startswith("quack:"):
             from interlace.engines.quack import QuackAdapter
 
@@ -209,6 +209,11 @@ class Project:
                 database = str(path)
             engine = DuckDBAdapter.connect(database)
 
+        # A model's relative read path (`read_csv_auto('seeds/x.csv')`) is documented to
+        # resolve against the project root. DuckDB resolves against the process CWD, which
+        # is only the same thing when you happen to run from the root — not under
+        # `--path`, `interlace serve`, or the scheduler.
+        engine.search_files_from(str(self.root))
         for alias, uri in cfg.attach.items():
             target = uri
             if uri.startswith(("postgres:", "postgres://", "postgresql://")):

{interlaced-2.3.0 → interlaced-2.4.0}/src/interlace/scaffold.py

@@ -72,10 +72,27 @@ def scaffold_project(root: Path, name: str | None = None, template: str = DEFAUL
 
     written: list[Path] = []
     for item in sorted(source.rglob("*")):
-        if item.name == _META_FILE or not item.is_file():
+        if item.name == _META_FILE or not item.is_file() or _is_install_artefact(item):
             continue
         target = root / item.relative_to(source)
         target.parent.mkdir(parents=True, exist_ok=True)
-        target.write_text(item.read_text().replace(_NAME_TOKEN, project_name))
+        try:
+            text = item.read_text(encoding="utf-8")
+        except UnicodeDecodeError:
+            # A template may legitimately carry a binary fixture (parquet, sqlite).
+            target.write_bytes(item.read_bytes())
+        else:
+            target.write_text(text.replace(_NAME_TOKEN, project_name), encoding="utf-8")
         written.append(target)
     return written
+
+
+def _is_install_artefact(item: Path) -> bool:
+    """Templates are real projects, so they contain ``.py`` model files — and pip
+    byte-compiles every ``.py`` in the wheel at install time. That leaves
+    ``__pycache__/*.pyc`` sitting inside the *installed* template tree, which is an
+    artefact of the install and never part of the template. Copying one into a new
+    project is wrong, and reading it as text raised ``UnicodeDecodeError`` on the
+    first command a pip user ever ran. (uv does not byte-compile by default, which
+    is why this only reproduced via pip.)"""
+    return "__pycache__" in item.parts or item.suffix in {".pyc", ".pyo"}