interlaced 2.3.0__tar.gz → 2.4.1__tar.gz

{interlaced-2.3.0/src/interlaced.egg-info → interlaced-2.4.1}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: interlaced
-Version: 2.3.0
-Summary: Python/SQL-first data platform: transformation, built-in orchestration, and durable streaming ingestion
+Version: 2.4.1
+Summary: Python and SQL models in one DAG. Transformation, orchestration and durable streaming in one process, on DuckDB and Postgres.
 Author-email: Mark <mark@interlace.sh>
 License-Expression: MIT
 Project-URL: Homepage, https://interlace.sh
@@ -13,8 +13,10 @@ Keywords: data,pipeline,orchestration,transformation,etl,streaming,dbt,sqlmesh,d
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Topic :: Software Development :: Build Tools
+Classifier: Topic :: Database
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: SQL
+Classifier: Operating System :: POSIX :: Linux
 Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
@@ -64,15 +66,41 @@ Dynamic: license-file
 
 # interlace
 
-**Python/SQL-first data platform: transformation, orchestration, and durable streaming — one process.**
+**Python and SQL models are the same kind of node in one DAG.**
 
-interlace is an independent, MIT-licensed alternative to dbt/SQLMesh that also replaces the
-orchestrator (no Airflow) and the ingestion layer (Cloudflare-Pipelines-style durable streams).
-Models are `.sql` files or Python functions; state is versioned snapshots with virtual
+A `.py` model sits mid-graph with SQL either side, in both directions, with no bridge and no
+separate runtime — running in-process on DuckDB and Postgres, not only on a cloud warehouse.
+The Python model stays a plain function: call it in a test with no warehouse and no session.
+
+How that compares with dbt and SQLMesh, including where they are ahead, is on
+[interlace.sh/why](https://interlace.sh/why).
+
+```sql
+-- models/raw_events.sql                    SQL
+SELECT event_id, user_id, kind, amount, country, ts FROM read_parquet('events/*.parquet')
+```
+```python
+# models/enriched_events.py                 Python, mid-DAG
+@model()                    # the parameter name IS the dependency — no depends_on
+def enriched_events(raw_events):
+    for batch in raw_events.reader():       # Arrow in, Arrow out, bounded memory
+        yield add_revenue(batch)
+```
+```sql
+-- models/event_summary.sql                 SQL again, straight over the Python
+SELECT country, count(*) FILTER (WHERE is_conversion) AS conversions
+FROM enriched_events GROUP BY country
+```
+
+`interlace init` scaffolds exactly this shape, runnable, with no external source.
+
+That is the wedge. The rest is the reveal: interlace is an independent, MIT-licensed alternative
+to dbt/SQLMesh that also replaces the orchestrator (no Airflow) and the ingestion layer
+(Cloudflare-Pipelines-style durable streams). State is versioned snapshots with virtual
 environments and a terraform-style plan/apply; everything runs in a single daemon on
-DuckDB + DuckLake by default.
+DuckDB by default (DuckLake one config line away).
 
-> **Status: 2.0.** Requires Python 3.12+.
+> **2.x — see [releases](https://github.com/interlace-sh/interlace/releases).** Requires Python 3.12+.
 > The package is published to PyPI as **`interlaced`**; the import name and CLI are `interlace`.
 
 ```bash
@@ -80,6 +108,13 @@ pip install 'interlaced[service]'   # the CLI + daemon; core CLI only: pip insta
 # more extras: [adbc] postgres/redshift · [spark] · [polars] · [all]
 ```
 
+> **Platforms.** Developed on Linux; CI runs Linux only. Nothing in the codebase is
+> platform-specific — no `fork`, no signal handling, no POSIX-only calls, no shelling out — and
+> every dependency ships macOS and Windows wheels, so both are expected to work. But
+> **neither is tested**, so treat them as unverified rather than supported. If you run interlace
+> on macOS or Windows, please open an issue either way; that is the fastest route to changing
+> this paragraph.
+
 ## Sixty seconds
 
 ```bash
@@ -210,7 +245,7 @@ prod, so a dev apply never writes to a live external table (opt in with
 
 ## Multi-engine
 
-Models run on **named engines**: DuckDB/DuckLake by default, Postgres natively over ADBC
+Models run on **named engines**: DuckDB by default (DuckLake opt-in), Postgres natively over ADBC
 (`pip install 'interlaced[adbc]'`), Spark (beta, `[spark]` extra), plus alpha adapters for
 MotherDuck, Redshift, Snowflake and BigQuery (wired and dialect-correct, not yet run against a
 live account), with per-model pinning:
@@ -252,7 +287,8 @@ other processes (CLI runs, ad-hoc DuckDB clients) then share it concurrently by
 
 - The IR is a **sqlglot AST**; the wire format is an **Arrow RecordBatchReader**; strategies
   are AST builders and dialect appears only at `transpile()`.
-- Storage defaults to **DuckLake** (Parquet + SQL catalog) opened as DuckDB's primary database.
+- Storage defaults to a plain **DuckDB** file; **DuckLake** (Parquet + SQL catalog, and
+  concurrent writers) is one `database: ducklake:…` line away.
 - Control plane (snapshots, intervals, queue, events, keys) is **SQLite WAL**; Postgres is the
   scale-out swap.
 - Streams live in their own durable log; the materializer commits data + watermark in one
@@ -269,7 +305,7 @@ Toolchain is pinned with [proto](https://moonrepo.dev/proto), tasks run via
 ```bash
 proto install
 moon run interlace:sync      # install deps
-moon run interlace:test      # 350+ tests
+moon run interlace:test      # 500+ tests
 moon run interlace:check     # black + ruff (CI equivalent)
 moon run interlace:typecheck # mypy
 ```

{interlaced-2.3.0 → interlaced-2.4.1}/README.md

@@ -1,14 +1,40 @@
 # interlace
 
-**Python/SQL-first data platform: transformation, orchestration, and durable streaming — one process.**
+**Python and SQL models are the same kind of node in one DAG.**
 
-interlace is an independent, MIT-licensed alternative to dbt/SQLMesh that also replaces the
-orchestrator (no Airflow) and the ingestion layer (Cloudflare-Pipelines-style durable streams).
-Models are `.sql` files or Python functions; state is versioned snapshots with virtual
+A `.py` model sits mid-graph with SQL either side, in both directions, with no bridge and no
+separate runtime — running in-process on DuckDB and Postgres, not only on a cloud warehouse.
+The Python model stays a plain function: call it in a test with no warehouse and no session.
+
+How that compares with dbt and SQLMesh, including where they are ahead, is on
+[interlace.sh/why](https://interlace.sh/why).
+
+```sql
+-- models/raw_events.sql                    SQL
+SELECT event_id, user_id, kind, amount, country, ts FROM read_parquet('events/*.parquet')
+```
+```python
+# models/enriched_events.py                 Python, mid-DAG
+@model()                    # the parameter name IS the dependency — no depends_on
+def enriched_events(raw_events):
+    for batch in raw_events.reader():       # Arrow in, Arrow out, bounded memory
+        yield add_revenue(batch)
+```
+```sql
+-- models/event_summary.sql                 SQL again, straight over the Python
+SELECT country, count(*) FILTER (WHERE is_conversion) AS conversions
+FROM enriched_events GROUP BY country
+```
+
+`interlace init` scaffolds exactly this shape, runnable, with no external source.
+
+That is the wedge. The rest is the reveal: interlace is an independent, MIT-licensed alternative
+to dbt/SQLMesh that also replaces the orchestrator (no Airflow) and the ingestion layer
+(Cloudflare-Pipelines-style durable streams). State is versioned snapshots with virtual
 environments and a terraform-style plan/apply; everything runs in a single daemon on
-DuckDB + DuckLake by default.
+DuckDB by default (DuckLake one config line away).
 
-> **Status: 2.0.** Requires Python 3.12+.
+> **2.x — see [releases](https://github.com/interlace-sh/interlace/releases).** Requires Python 3.12+.
 > The package is published to PyPI as **`interlaced`**; the import name and CLI are `interlace`.
 
 ```bash
@@ -16,6 +42,13 @@ pip install 'interlaced[service]'   # the CLI + daemon; core CLI only: pip insta
 # more extras: [adbc] postgres/redshift · [spark] · [polars] · [all]
 ```
 
+> **Platforms.** Developed on Linux; CI runs Linux only. Nothing in the codebase is
+> platform-specific — no `fork`, no signal handling, no POSIX-only calls, no shelling out — and
+> every dependency ships macOS and Windows wheels, so both are expected to work. But
+> **neither is tested**, so treat them as unverified rather than supported. If you run interlace
+> on macOS or Windows, please open an issue either way; that is the fastest route to changing
+> this paragraph.
+
 ## Sixty seconds
 
 ```bash
@@ -146,7 +179,7 @@ prod, so a dev apply never writes to a live external table (opt in with
 
 ## Multi-engine
 
-Models run on **named engines**: DuckDB/DuckLake by default, Postgres natively over ADBC
+Models run on **named engines**: DuckDB by default (DuckLake opt-in), Postgres natively over ADBC
 (`pip install 'interlaced[adbc]'`), Spark (beta, `[spark]` extra), plus alpha adapters for
 MotherDuck, Redshift, Snowflake and BigQuery (wired and dialect-correct, not yet run against a
 live account), with per-model pinning:
@@ -188,7 +221,8 @@ other processes (CLI runs, ad-hoc DuckDB clients) then share it concurrently by
 
 - The IR is a **sqlglot AST**; the wire format is an **Arrow RecordBatchReader**; strategies
   are AST builders and dialect appears only at `transpile()`.
-- Storage defaults to **DuckLake** (Parquet + SQL catalog) opened as DuckDB's primary database.
+- Storage defaults to a plain **DuckDB** file; **DuckLake** (Parquet + SQL catalog, and
+  concurrent writers) is one `database: ducklake:…` line away.
 - Control plane (snapshots, intervals, queue, events, keys) is **SQLite WAL**; Postgres is the
   scale-out swap.
 - Streams live in their own durable log; the materializer commits data + watermark in one
@@ -205,7 +239,7 @@ Toolchain is pinned with [proto](https://moonrepo.dev/proto), tasks run via
 ```bash
 proto install
 moon run interlace:sync      # install deps
-moon run interlace:test      # 350+ tests
+moon run interlace:test      # 500+ tests
 moon run interlace:check     # black + ruff (CI equivalent)
 moon run interlace:typecheck # mypy
 ```

{interlaced-2.3.0 → interlaced-2.4.1}/pyproject.toml

@@ -4,8 +4,8 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "interlaced"
-version = "2.3.0"
-description = "Python/SQL-first data platform: transformation, built-in orchestration, and durable streaming ingestion"
+version = "2.4.1"
+description = "Python and SQL models in one DAG. Transformation, orchestration and durable streaming in one process, on DuckDB and Postgres."
 readme = "README.md"
 requires-python = ">=3.12"
 license = "MIT"
@@ -17,8 +17,12 @@ classifiers = [
     "Development Status :: 5 - Production/Stable",
     "Intended Audience :: Developers",
     "Topic :: Software Development :: Build Tools",
+    "Topic :: Database",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: SQL",
+    # Linux is what CI runs and what this is developed on. Nothing in the codebase
+    # is platform-specific, but macOS and Windows are untested — see the README.
+    "Operating System :: POSIX :: Linux",
 ]
 
 # Core: the IR + engine + state + plan spine (Phase 1). Everything is Arrow- and
@@ -176,6 +180,7 @@ markers = [
     "integration: Integration tests (slower, require database)",
     "slow: Slow tests (may take a long time)",
     "requires_db: Needs a reachable external database (e.g. Postgres)",
+    "requires_network: Needs the public internet (e.g. an example reading a remote CSV)",
 ]
 asyncio_mode = "auto"
 asyncio_default_fixture_loop_scope = "function"

{interlaced-2.3.0 → interlaced-2.4.1}/src/interlace/cli/main.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import asyncio
 import contextlib
+import time
 from datetime import datetime
 from pathlib import Path
 from typing import Any
@@ -467,6 +468,7 @@ async def _execute(
             restate=restate,
         )
         progress = _build_progress(plan_result)
+        started = time.perf_counter()
         try:
             with progress.progress if progress else contextlib.nullcontext():
                 result = await apply_plan(
@@ -487,7 +489,7 @@ async def _execute(
         await _render_empty_incrementals(result, compiled, engines)
         verb = "Restated" if restate else "Ran"
         console.print(
-            f"[green]{verb} {len(set(result.built))} model(s) ({len(result.built)} task(s)); "
+            f"[green]{verb} {len(set(result.built))} model(s) in {time.perf_counter() - started:.2f}s; "
             f"promoted {result.promoted} to '{environment}'.[/green]"
         )
     finally:

{interlaced-2.3.0 → interlaced-2.4.1}/src/interlace/config/config.py

@@ -84,7 +84,7 @@ class EngineConfig(BaseModel):
     declaring them fails at open until an adapter ships.
     """
 
-    type: str = "ducklake"
+    type: str = "duckdb"
     # Path / URI for DuckDB-family engines. Also accepted on the project top level
     # as ``database:`` (synthesised into the ``default`` engine).
     database: str | None = None
@@ -117,13 +117,14 @@ class ProjectConfig(BaseModel):
     default_engine: str = "default"
     engines: dict[str, EngineConfig] = Field(default_factory=dict)
     state_path: str = ".interlace/state.db"  # SQLite control-plane database
-    # The warehouse. Default is DuckLake (Parquet data + SQL catalog) via DuckDB.
-    # Also accepted: a DuckLake catalog hosted in a SQL database
-    # ("ducklake:postgres:dbname=... host=..." — pair with data_path/metadata_schema),
-    # a plain DuckDB file path, ":memory:", or "quack:<host>:<port>" to connect to a
-    # warehouse served by `interlace serve --quack`.
+    # The warehouse. Default is a plain DuckDB file — simplest, single-process.
+    # Also accepted: a DuckLake catalog (``ducklake:.interlace/warehouse.ducklake``, or
+    # hosted in a SQL DB: ``ducklake:postgres:dbname=... host=...`` — pair with
+    # data_path/metadata_schema) which serialises catalog writes so `interlace serve`
+    # and a separate CLI can share the warehouse concurrently; ":memory:"; or
+    # "quack:<host>:<port>" to connect to a warehouse served by `interlace serve --quack`.
     # When ``engines.default`` is not set, these top-level fields synthesise it.
-    database: str = "ducklake:.interlace/warehouse.ducklake"
+    database: str = ".interlace/warehouse.duckdb"
     # The warehouse catalog's ATTACH alias (defaults to ``name``). Set it when a
     # schema inside the warehouse shares the project name — see EngineConfig.alias.
     alias: str | None = None
@@ -144,6 +145,9 @@ class ProjectConfig(BaseModel):
     # This is T0 federation (all SQL still runs in DuckDB) — see docs/architecture/MULTI_ENGINE.md.
     attach: dict[str, str] = Field(default_factory=dict)
     model_paths: list[str] = Field(default_factory=lambda: ["models"])
+    # Directories of CREATE MACRO definitions, expanded into every model at compile
+    # time (see interlace.ir.macros). Missing directories are ignored.
+    macro_paths: list[str] = Field(default_factory=lambda: ["macros"])
     # How many models build concurrently. Lower it to serialise a build end-to-end
     # when diagnosing a crash or a warehouse that dislikes parallel writers; `run`
     # and `apply` also take --parallelism, which wins over this.

{interlaced-2.3.0 → interlaced-2.4.1}/src/interlace/dsl/decorators.py

@@ -114,6 +114,18 @@ class ModelDef:
     schedule: dict[str, str] | None = None  # {"cron": "0 * * * *"} or {"every": "5m"} for `interlace serve`
     checks: tuple[CheckSpec, ...] = ()  # data-quality checks; error severity gates promotion
 
+    def __post_init__(self) -> None:
+        # `@model(checks=…)` normalises through parse_checks, but a ModelDef built
+        # directly — the dynamic-model path, which is what generated models and dbt
+        # migrations use — stored the dicts raw and only failed at compile time with
+        # `AttributeError: 'dict' object has no attribute 'type'`. Normalise here too,
+        # so one spelling works on both surfaces and a bad check fails at declaration.
+        # Passed through as-is, not as list(...): parse_checks already handles a bare
+        # CheckSpec and reports a non-list clearly, both of which list() would mangle
+        # (TypeError on a CheckSpec; a dict silently degraded to its keys). Always run,
+        # so `checks=[]` normalises to the declared tuple rather than staying a list.
+        self.checks = parse_checks(self.checks, self.name)
+
     @property
     def is_terminal(self) -> bool:
         """A terminal model delivers into an external destination (table/file):

{interlaced-2.3.0 → interlaced-2.4.1}/src/interlace/dsl/discovery.py

@@ -9,8 +9,10 @@ cleared first for a clean load.
 
 from __future__ import annotations
 
+import contextlib
 import importlib.util
 import sys
+from collections.abc import Iterator
 from pathlib import Path
 from typing import Any
 
@@ -18,22 +20,61 @@ from interlace.checks.spec import parse_checks
 from interlace.dsl.decorators import REGISTRY, ModelDef, _as_columns, _as_tuple, validate_materialise
 from interlace.dsl.sql_config import extract_sql_config
 from interlace.exceptions import DefinitionError, InterlaceError
+from interlace.ir.macros import Macro, parse_macros
 
 
 def discover_models(root: Path, model_paths: list[str], default_dialect: str) -> list[ModelDef]:
     REGISTRY.clear()
-    for relative in model_paths:
+    imported = set(sys.modules)
+    try:
+        for relative in model_paths:
+            base = root / relative
+            if not base.is_dir():
+                continue
+            for sql_file in sorted(base.rglob("*.sql")):
+                config, sql = extract_sql_config(sql_file.read_text())
+                REGISTRY.register_model(_sql_model(_model_name(base, sql_file), sql, config, default_dialect))
+            for py_file in sorted(base.rglob("*.py")):
+                if py_file.name.startswith("_"):
+                    continue
+                _import_module(base, py_file)
+    finally:
+        _forget_project_modules(root, imported)
+    return list(REGISTRY.models.values())
+
+
+def _forget_project_modules(root: Path, before: set[str]) -> None:
+    """Drop the project's own helper modules from the import cache.
+
+    A helper is imported under whatever plain name it has (``_macros``), so leaving it
+    cached would hand the *next* project — a reload under ``interlace serve``, another
+    project in the same process — the first one's version of a file with the same name.
+    Only modules loaded from inside this project during this pass are dropped."""
+    for name in set(sys.modules) - before:
+        module = sys.modules.get(name)
+        origin = getattr(module, "__file__", None)
+        if origin and Path(origin).is_relative_to(root):
+            del sys.modules[name]
+
+
+def discover_macros(root: Path, macro_paths: list[str], default_dialect: str) -> dict[str, Macro]:
+    """Parse every ``CREATE MACRO`` under the macro paths, keyed by casefolded name."""
+    macros: dict[str, Macro] = {}
+    for relative in macro_paths:
         base = root / relative
         if not base.is_dir():
             continue
-        for sql_file in sorted(base.rglob("*.sql")):
-            config, sql = extract_sql_config(sql_file.read_text())
-            REGISTRY.register_model(_sql_model(_model_name(base, sql_file), sql, config, default_dialect))
-        for py_file in sorted(base.rglob("*.py")):
-            if py_file.name.startswith("_"):
-                continue
-            _import_module(base, py_file)
-    return list(REGISTRY.models.values())
+        for macro_file in sorted(base.rglob("*.sql")):
+            source = str(macro_file.relative_to(root))
+            for macro in parse_macros(macro_file.read_text(), default_dialect, source):
+                key = macro.name.casefold()
+                if key in macros:
+                    raise DefinitionError(
+                        f"macro {macro.name!r} is defined twice: {macros[key].source} and {source}",
+                        details={"macro": macro.name},
+                    )
+                macros[key] = macro
+    return macros
 
 
 def _sql_model(default_name: str, sql: str, config: dict[str, Any], default_dialect: str) -> ModelDef:
@@ -89,6 +130,18 @@ def _relative_to_cwd(file: Path) -> str:
         return str(file)
 
 
+@contextlib.contextmanager
+def _importable(directory: Path) -> Iterator[None]:
+    """Put ``directory`` at the front of ``sys.path``, and take it back off."""
+    entry = str(directory)
+    sys.path.insert(0, entry)
+    try:
+        yield
+    finally:
+        with contextlib.suppress(ValueError):
+            sys.path.remove(entry)
+
+
 def _import_module(base: Path, file: Path) -> None:
     module_name = "interlace_model_" + "_".join(file.relative_to(base).with_suffix("").parts)
     spec = importlib.util.spec_from_file_location(module_name, file)
@@ -97,7 +150,13 @@ def _import_module(base: Path, file: Path) -> None:
     module = importlib.util.module_from_spec(spec)
     sys.modules[module_name] = module
     try:
-        spec.loader.exec_module(module)
+        # The model's own directory goes on the path for the duration of the import, so a
+        # model can `from _shared import ...` a helper sitting next to it — the closest
+        # thing to a macro. Files starting with `_` are already skipped as models, which
+        # is only useful if they can be imported. Removed again straight after: a project's
+        # helpers must not leak into the import path of everything that follows.
+        with _importable(file.parent):
+            spec.loader.exec_module(module)
     except InterlaceError:
         raise  # a bad @model config is already a clean, actionable error — don't rewrap it
     except Exception as exc:

{interlaced-2.3.0 → interlaced-2.4.1}/src/interlace/engines/duckdb.py

@@ -28,6 +28,7 @@ from __future__ import annotations
 
 import asyncio
 import contextlib
+import re
 import threading
 from collections.abc import Iterator, Sequence
 from uuid import uuid4
@@ -38,6 +39,7 @@ import tenacity
 from sqlglot import exp
 
 from interlace.engines.base import EngineAdapter, EngineCaps, LoadMode
+from interlace.exceptions import ConfigurationError
 from interlace.ir.relation import TableRef
 
 _DUCKDB_CAPS = EngineCaps(
@@ -58,6 +60,39 @@ _commit_retry = tenacity.retry(
 )
 
 
+@contextlib.contextmanager
+def _clean_lock_error(database: str) -> Iterator[None]:
+    """Translate DuckDB's file-lock conflict into one actionable line.
+
+    A DuckDB/DuckLake database is held by a single process. The common way to hit
+    that is running a CLI command (`interlace query`, `plan`, `apply`) while
+    `interlace serve` is up — an obvious thing to do, since serve is the daemon
+    and query is the console's CLI counterpart. Raw, that surfaces as a dozen
+    frames ending in `duckdb.IOException`, which names neither the cause nor the
+    fix. The fix is `--quack`, and it is already documented; the error just never
+    said so.
+    """
+    try:
+        yield
+    except duckdb.IOException as exc:
+        message = str(exc)
+        # "Conflicting lock is held in <exe> (PID n)" is the Linux rendering — DuckDB
+        # names the holder from /proc/locks. Elsewhere the message is the bare "Could
+        # not set lock on file", and a same-process conflict says "already held", so
+        # match all three or macOS/Windows keep the raw traceback.
+        if not any(marker in message for marker in ("Conflicting lock", "Could not set lock", "already held")):
+            raise
+        holder = re.search(r"\(PID (\d+)\)", message)
+        held_by = f" (PID {holder.group(1)})" if holder else ""
+        raise ConfigurationError(
+            f"the warehouse {database!r} is already open in another process{held_by}. "
+            "DuckDB allows one process at a time — stop `interlace serve`, or serve the "
+            "warehouse over the quack protocol (`interlace serve --quack quack:localhost:4213`) "
+            "and point this process at `database: quack:localhost:4213` to share it.",
+            details={"database": database},
+        ) from None
+
+
 def _affected(cur: duckdb.DuckDBPyConnection) -> int:
     """DML/CTAS/COPY return their affected-row count as a one-cell result; DDL returns
     nothing. Never raises — row stats are best-effort decoration, not correctness."""
@@ -108,7 +143,9 @@ class DuckDBAdapter(EngineAdapter):
 
     @classmethod
     def connect(cls, path: str) -> DuckDBAdapter:
-        return cls(duckdb.connect(path), serialise_writes=path.startswith("ducklake:"))
+        with _clean_lock_error(path):
+            conn = duckdb.connect(path)
+        return cls(conn, serialise_writes=path.startswith("ducklake:"))
 
     @classmethod
     def connect_ducklake(
@@ -139,7 +176,8 @@ class DuckDBAdapter(EngineAdapter):
         options_sql = f" ({', '.join(options)})" if options else ""
         escaped = catalog.replace("'", "''")
         alias_sql = exp.to_identifier(alias).sql("duckdb")
-        conn.execute(f"ATTACH IF NOT EXISTS '{escaped}' AS {alias_sql}{options_sql}")
+        with _clean_lock_error(catalog):
+            conn.execute(f"ATTACH IF NOT EXISTS '{escaped}' AS {alias_sql}{options_sql}")
         conn.execute(f"USE {alias_sql}")
         # LOAD, secrets, and ATTACH are all instance-wide — they carry into every
         # cursor and must run ONCE (re-running CREATE OR REPLACE SECRET per cursor
@@ -162,10 +200,23 @@ class DuckDBAdapter(EngineAdapter):
         with contextlib.suppress(Exception):
             self._conn.interrupt()
 
+    def search_files_from(self, directory: str) -> None:
+        """Resolve relative read paths (``read_csv_auto('seeds/x.csv')``) against
+        ``directory`` — the project root — as well as the process CWD.
+
+        Additive: a CWD-relative path still resolves, so this only ever widens what a
+        model can find. GLOBAL scope because a plain ``SET`` is session-scoped and would
+        not reach the per-task cursors that actually run the queries. Reads only —
+        ``COPY`` targets stay CWD-relative, which is why exports resolve their own paths
+        against the root (``plan.apply._resolve_export_path``)."""
+        escaped = directory.replace("'", "''")
+        self._conn.execute(f"SET GLOBAL file_search_path='{escaped}'")
+
     def attach(self, alias: str, uri: str) -> None:
         """ATTACH another database (duckdb/sqlite/postgres/... URI) under ``alias``."""
         escaped = uri.replace("'", "''")
-        self._conn.execute(f"ATTACH IF NOT EXISTS '{escaped}' AS {exp.to_identifier(alias).sql('duckdb')}")
+        with _clean_lock_error(uri):  # attaching a held duckdb/ducklake file conflicts just like opening one
+            self._conn.execute(f"ATTACH IF NOT EXISTS '{escaped}' AS {exp.to_identifier(alias).sql('duckdb')}")
         self._attached.append(alias)
         if uri.startswith("ducklake:"):  # writes may now reach a DuckLake catalog (e.g. table sinks)
             if isinstance(self._write_lock, contextlib.nullcontext):

{interlaced-2.3.0 → interlaced-2.4.1}/src/interlace/graph/project.py

@@ -12,7 +12,7 @@ from __future__ import annotations
 
 import inspect
 import textwrap
-from collections.abc import Iterable
+from collections.abc import Iterable, Mapping
 from dataclasses import dataclass, field
 
 from sqlglot import exp
@@ -23,6 +23,7 @@ from interlace.exceptions import CompilationError, DefinitionError
 from interlace.graph.dag import DependencyGraph
 from interlace.ir.canonicalize import parse, table_references
 from interlace.ir.fingerprint import canonical_sql, data_fingerprint, metadata_fingerprint
+from interlace.ir.macros import Macro, expand_macros
 from interlace.ir.relation import TableRef
 
 _PHYSICAL_PREFIX = "interlace__"
@@ -93,7 +94,7 @@ def _physical_table(name: str, fingerprint: str, catalog: str | None) -> TableRe
 
 
 def _resolve_dependencies(
-    model: ModelDef, names: set[str], default_dialect: str
+    model: ModelDef, names: set[str], default_dialect: str, macros: Mapping[str, Macro]
 ) -> tuple[tuple[str, ...], exp.Expression | None, str]:
     dialect = model.dialect or default_dialect
     deps: list[str] = []
@@ -115,6 +116,10 @@ def _resolve_dependencies(
             raise CompilationError(
                 f"model {model.name!r}: {exc.message}", details={**exc.details, "model": model.name}
             ) from exc
+        # Before anything reads the AST: the fingerprint is canonical SQL, so expanding
+        # here is what makes a macro edit rebuild its callers, and what lets a macro body
+        # reference a model and have that count as a dependency.
+        ast = expand_macros(ast, macros, model.name)
         for ref in table_references(ast):
             if ref in names:
                 add(ref)
@@ -149,12 +154,14 @@ def compile_models(
     known_engines: set[str] | None = None,
     catalog: str | None = None,
     checks: Iterable[CheckDef] = (),
+    macros: Mapping[str, Macro] | None = None,
 ) -> CompiledProject:
     """Compile models into a fingerprinted, topologically-ordered project.
 
     ``checks`` are ``@check``-decorated Python functions, attached by model name.
     ``engine_dialects`` maps engine name → sqlglot dialect (used when a model
     omits ``dialect``). ``known_engines`` validates model ``engine`` pins.
+    ``macros`` are expanded into every model's AST before it is fingerprinted.
     """
     definitions = {m.name: m for m in models}
     names = set(definitions)
@@ -178,7 +185,7 @@ def compile_models(
             )
         # Authoring dialect: explicit model dialect, else the engine's, else project default.
         model_default_dialect = dialects_by_engine.get(engine, default_dialect)
-        deps, ast, dialect = _resolve_dependencies(definition, names, model_default_dialect)
+        deps, ast, dialect = _resolve_dependencies(definition, names, model_default_dialect, macros or {})
         resolved[name] = (deps, ast, dialect, engine)
         graph.add_node(name)
         for dep in deps: