duckrun 0.3.17.dev2__tar.gz → 0.3.17.dev4__tar.gz

{duckrun-0.3.17.dev2/duckrun.egg-info → duckrun-0.3.17.dev4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: duckrun
-Version: 0.3.17.dev2
+Version: 0.3.17.dev4
 Summary: A dbt adapter that runs SQL in DuckDB and materializes to Delta Lake (delta_rs).
 Author: mim
 License: MIT
@@ -291,11 +291,15 @@ reclaimed — duckrun favors read-safety over immediate disk savings.
 ## Connection API (notebook)
 
 Besides the dbt adapter, duckrun ships a storage-neutral, PySpark-shaped `duckrun.connect()` for
-interactive/notebook use (local, S3, GCS, ADLS, OneLake). `conn.sql(...)` is **read-only** (including
-time travel — `delta_scan('…', version => N)`); writes go through the Spark surface: a `DataFrame`
-with `.write…saveAsTable()` (modes `overwrite` / `append` / `safeappend` / `ignore`) and a `DeltaTable` handle
-(`conn.delta_table(name)` / `DeltaTable.forName`) with `.merge(...)`, `.delete()`, `.update()`,
-`.replaceWhere()`, `.version()`, plus `conn.read` and `conn.catalog`.
+interactive/notebook use (local, S3, GCS, ADLS, OneLake). `conn.sql(...)` runs reads (including time
+travel — `delta_scan('…', version => N)`) and applies **raw SQL DML** (`create table … as`, `insert`,
+`update`, `delete`, `alter add column`, `drop`) straight to the Delta table via delta_rs — every
+`CREATE TABLE` is Delta-backed, only `CREATE TEMP TABLE`/`CREATE VIEW` stay native DuckDB, and forms
+delta_rs can't express (`MERGE`, `UPDATE … FROM`, multi-statement) are rejected with a pointer to the
+write API. Writes also go through the Spark surface: a `DataFrame` with `.write…saveAsTable()` (modes
+`overwrite` / `append` / `safeappend` / `ignore`) and a `DeltaTable` handle (`conn.delta_table(name)`
+/ `DeltaTable.forName`) with `.merge(...)`, `.delete()`, `.update()`, `.replaceWhere()`, `.version()`,
+plus `conn.read` and `conn.catalog`. See [the DML matrix](docs/connection-api.md#raw-sql-dml-through-connsql).
 
 `merge` is **snapshot-pinned by default** — Spark's single-snapshot MERGE, with no extra arguments:
 the target version is captured and the commit is validated against it, so a concurrent writer fails
@@ -347,7 +351,7 @@ None of this is required to use duckrun — `pip install duckrun` is unaffected.
 
 **Testing.** `tests/integration_tests/aemo/` is a small dbt project built against OneLake, and
 `tests/integration_tests/coffee/` is the connection-API coffee-shop scenario / stress test (CI:
-[`integration.yml`](.github/workflows/integration.yml)); `tests/conformance/`
+[`integration_tests_onelake.yml`](.github/workflows/integration_tests_onelake.yml)); `tests/conformance/`
 runs the official suite (above); `tests/correctness/` proves the concurrency guarantees. The cards
 in those docs are rendered live by CI, so they always reflect the latest `main`.
 

{duckrun-0.3.17.dev2 → duckrun-0.3.17.dev4}/README.md

@@ -262,11 +262,15 @@ reclaimed — duckrun favors read-safety over immediate disk savings.
 ## Connection API (notebook)
 
 Besides the dbt adapter, duckrun ships a storage-neutral, PySpark-shaped `duckrun.connect()` for
-interactive/notebook use (local, S3, GCS, ADLS, OneLake). `conn.sql(...)` is **read-only** (including
-time travel — `delta_scan('…', version => N)`); writes go through the Spark surface: a `DataFrame`
-with `.write…saveAsTable()` (modes `overwrite` / `append` / `safeappend` / `ignore`) and a `DeltaTable` handle
-(`conn.delta_table(name)` / `DeltaTable.forName`) with `.merge(...)`, `.delete()`, `.update()`,
-`.replaceWhere()`, `.version()`, plus `conn.read` and `conn.catalog`.
+interactive/notebook use (local, S3, GCS, ADLS, OneLake). `conn.sql(...)` runs reads (including time
+travel — `delta_scan('…', version => N)`) and applies **raw SQL DML** (`create table … as`, `insert`,
+`update`, `delete`, `alter add column`, `drop`) straight to the Delta table via delta_rs — every
+`CREATE TABLE` is Delta-backed, only `CREATE TEMP TABLE`/`CREATE VIEW` stay native DuckDB, and forms
+delta_rs can't express (`MERGE`, `UPDATE … FROM`, multi-statement) are rejected with a pointer to the
+write API. Writes also go through the Spark surface: a `DataFrame` with `.write…saveAsTable()` (modes
+`overwrite` / `append` / `safeappend` / `ignore`) and a `DeltaTable` handle (`conn.delta_table(name)`
+/ `DeltaTable.forName`) with `.merge(...)`, `.delete()`, `.update()`, `.replaceWhere()`, `.version()`,
+plus `conn.read` and `conn.catalog`. See [the DML matrix](docs/connection-api.md#raw-sql-dml-through-connsql).
 
 `merge` is **snapshot-pinned by default** — Spark's single-snapshot MERGE, with no extra arguments:
 the target version is captured and the commit is validated against it, so a concurrent writer fails
@@ -318,7 +322,7 @@ None of this is required to use duckrun — `pip install duckrun` is unaffected.
 
 **Testing.** `tests/integration_tests/aemo/` is a small dbt project built against OneLake, and
 `tests/integration_tests/coffee/` is the connection-API coffee-shop scenario / stress test (CI:
-[`integration.yml`](.github/workflows/integration.yml)); `tests/conformance/`
+[`integration_tests_onelake.yml`](.github/workflows/integration_tests_onelake.yml)); `tests/conformance/`
 runs the official suite (above); `tests/correctness/` proves the concurrency guarantees. The cards
 in those docs are rendered live by CI, so they always reflect the latest `main`.
 

duckrun-0.3.17.dev4/dbt/adapters/duckrun/__version__.py

@@ -0,0 +1 @@
+version = "0.3.17.dev4"

duckrun-0.3.17.dev4/dbt/adapters/duckrun/delta_dml.py

@@ -0,0 +1,457 @@
+"""Route raw SQL DML against duckrun-managed (Delta-backed) relations to delta_rs.
+
+duckrun intercepts writes at the dbt *materialization* layer (a model/seed/snapshot goes through
+the materialization macros -> store_relation -> delta_rs). But a duckrun relation is surfaced as a
+read-only ``delta_scan`` view, so *raw* DML sent straight to the connection — ``delete from``,
+``update``, ``insert into ... select``, ``alter table ... add column``, ``create table ... as
+select`` — lands on a view and fails ("Can only delete from base table"), or would create a native
+DuckDB table that bypasses Delta entirely.
+
+This module intercepts those statements (at the cursor, see environment.DuckrunCursorWrapper) and
+applies them to the Delta table **via delta_rs only**, then refreshes the ``delta_scan`` view — so
+nothing relies on a native, mutable DuckDB table, and every op works on local AND abfss/OneLake
+stores (delta_rs carries ``storage_options``). ``create table ... as`` writes a new Delta table;
+the mutate forms (delete/update/insert/alter) apply only when a Delta table already exists at the
+target (otherwise the statement passes through — e.g. the test's native ``fact``/``seed``).
+
+``drop table`` unregisters the ``delta_scan`` view AND overwrites the table (via delta_rs) to a
+one-column ``TOMBSTONE_COLUMN`` marker, which discovery recognizes and hides. It does NOT delete
+data: delta_rs has no drop, and removing the Delta files would be a filesystem hack that fails on
+object stores. The directory persists until a human purges it; a later ``create table ... as``
+overwrites the tombstone with real data and the table is live again.
+
+The seed loader's own SQL (``create table <t> (<col defs>)``, ``insert ... values``, ``COPY``) lands
+on a native DuckDB table, not a Delta table: bare ``create table (<col defs>)`` becomes a Delta
+table only when a ``default_schema`` is set (the connection API), and the dbt/cursor path passes
+None — so the seed's table stays native. ``insert ... values`` *does* match a form here, but the
+mutate guard only applies it when a Delta table already exists at the target — the seed's native
+table has none, so it falls through untouched. duckrun's own materializations emit ``create ...
+view`` (not ``table``), so they pass through too.
+
+Supported / unsupported (what reaches delta_rs):
+
+    create [or replace] table x [if not exists] as <query>   Delta overwrite (query: select/with/(…))
+    create table x (<col defs>)                               empty Delta table (connection API only)
+    create temp/temporary table …                            native DuckDB (pass through)  ── invariant:
+    create view …                                            native DuckDB (pass through)  ── only TEMP
+                                                                                            and VIEW are
+                                                                                            native; every
+                                                                                            other CREATE
+                                                                                            TABLE is Delta
+    insert into x [(cols)] select …                          Delta append (projected onto target schema)
+    insert into x [(cols)] values …                          Delta append (projected onto target schema)
+    [with …] insert into x select …                          Delta append (CTE re-attached to the body)
+    delete from x [where …]                                  delta_rs delete
+    update x set … [where …]                                 delta_rs update
+    alter table x add column …                               Delta overwrite (widen schema)
+    drop table x                                             tombstone (no data deleted)
+    merge … / update … from / delete … using / multi-stmt   NOT handled here — the connection API
+                                                             (session.sql) rejects them with a clear
+                                                             error; the dbt path never emits them.
+"""
+import re
+from typing import List, Optional, Tuple
+
+from . import engine
+
+# `drop table` tombstone: a dropped relation is overwritten (via delta_rs) to a table whose ONLY
+# column is this marker, so (a) discovery recognizes it as dropped and hides it, and (b) anyone who
+# opens the files sees an obviously-not-a-real-table schema rather than a plausible empty table. No
+# data is deleted — the directory stays until a human purges it; a later `create table ... as`
+# overwrites the marker schema with real data and the table is live again.
+TOMBSTONE_COLUMN = "__duckrun_deleted__"
+
+
+def _columns_are_tombstone(colnames) -> bool:
+    return [str(c).lower() for c in colnames] == [TOMBSTONE_COLUMN]
+
+
+def is_dropped(con, location: str, storage_options=None) -> bool:
+    """True if the Delta table at ``location`` is a duckrun drop-tombstone (single marker column).
+
+    Used by discovery (dbt + connection API) to hide dropped tables. Best-effort: anything that
+    can't be opened/scanned is treated as 'not a tombstone' (let normal handling deal with it).
+    """
+    loc_sql = str(location).replace("'", "''")
+    try:
+        rel = con.execute(f"select * from delta_scan('{loc_sql}') limit 0")
+        return _columns_are_tombstone([d[0] for d in rel.description])
+    except Exception:
+        return False
+
+# --- statement matchers (leading-anchored, DOTALL so multi-line bodies match) ----------------
+# `create [or replace] table [if not exists] <rel> as <query>`. The body is ANY query text (a bare
+# `select …`, a `with … select …` CTE, or a parenthesised `(select …)`); it's handed to DuckDB
+# verbatim so anything DuckDB accepts after `as` works.
+_CREATE_AS = re.compile(
+    r"\s*create\s+(?P<orrep>or\s+replace\s+)?table\s+(?P<ine>if\s+not\s+exists\s+)?"
+    r"(?P<rel>.+?)\s+as\s+(?P<body>.+)",
+    re.I | re.S,
+)
+# `create [or replace] table [if not exists] <rel> (<col defs>)` — no `as`. Connection-API only
+# (see _create_coldefs): materializes an EMPTY Delta table so `CREATE TABLE` is always Delta-backed.
+_CREATE_COLDEFS = re.compile(
+    r"\s*create\s+(?:or\s+replace\s+)?table\s+(?:if\s+not\s+exists\s+)?"
+    r"(?P<rel>.+?)\s*\((?P<defs>.+)\)\s*;?\s*",
+    re.I | re.S,
+)
+_INSERT_SELECT = re.compile(
+    r"\s*insert\s+into\s+(?P<rel>.+?)\s*(?:\((?P<cols>[^)]*)\))?\s+(?P<body>select\b.*)",
+    re.I | re.S,
+)
+_INSERT_VALUES = re.compile(
+    r"\s*insert\s+into\s+(?P<rel>.+?)\s*(?:\((?P<cols>[^)]*)\))?\s*values\s+(?P<body>\(.+)",
+    re.I | re.S,
+)
+_DELETE = re.compile(
+    r"\s*delete\s+from\s+(?P<rel>.+?)(?:\s+where\s+(?P<where>.+))?\s*;?\s*", re.I | re.S
+)
+_UPDATE = re.compile(
+    r"\s*update\s+(?P<rel>.+?)\s+set\s+(?P<set>.+?)(?:\s+where\s+(?P<where>.+?))?\s*;?\s*",
+    re.I | re.S,
+)
+_ALTER_ADD = re.compile(
+    r"\s*alter\s+table\s+(?P<rel>.+?)\s+add\s+column\s+(?P<col>\S+)\s+(?P<def>.+?)\s*;?\s*",
+    re.I | re.S,
+)
+_DROP = re.compile(
+    r"\s*drop\s+table\s+(?:if\s+exists\s+)?(?P<rel>[^\s;]+)\s*;?\s*", re.I | re.S
+)
+# `create temp/temporary table …` is DuckDB-local scratch by design and must NEVER be captured —
+# checked first in try_handle so it always passes through to native DuckDB (the invariant: only
+# CREATE TEMP TABLE is native; every other CREATE TABLE is Delta-backed).
+_CREATE_TEMP_RE = re.compile(r"\s*create\s+(?:or\s+replace\s+)?(?:temp|temporary)\b", re.I)
+# CTE/whitespace handling: a leading `with …` block followed by a top-level INSERT/UPDATE/DELETE.
+# leading `\b` is load-bearing: _find_top_level tries this at every depth-0 index, so without it the
+# verb would match inside an identifier (e.g. `update` within `last_update`).
+_LEADING_WITH = re.compile(r"\s*with\b", re.I)
+_DRIVING_DML = re.compile(r"\b(?:insert\s+into|update|delete\s+from)\b", re.I)
+
+
+def _strip_leading(query: str) -> str:
+    """Drop leading whitespace and ``--`` / ``/* */`` comments so the first keyword is visible."""
+    s = query
+    while True:
+        t = s.lstrip()
+        if t.startswith("--"):
+            nl = t.find("\n")
+            s = "" if nl == -1 else t[nl + 1:]
+        elif t.startswith("/*"):
+            end = t.find("*/")
+            s = "" if end == -1 else t[end + 2:]
+        else:
+            return t
+
+
+def _find_top_level(s: str, pattern) -> int:
+    """Index of the first ``pattern`` match at paren-depth 0 and outside quotes, else -1.
+
+    Lets us tell a top-level clause (the ``FROM`` of ``UPDATE … FROM``, the verb after a leading
+    ``WITH``) from the same keyword nested in a subquery, without a full SQL parser."""
+    depth, quote, i, n = 0, None, 0, len(s)
+    while i < n:
+        ch = s[i]
+        if quote:
+            if ch == quote:
+                quote = None
+        elif ch in ("'", '"'):
+            quote = ch
+        elif ch in "([":
+            depth += 1
+        elif ch in ")]":
+            depth -= 1
+        elif depth == 0 and pattern.match(s, i):
+            return i
+        i += 1
+    return -1
+
+
+def _split_leading_with(sql: str) -> Tuple[str, str]:
+    """``(with_clause, remainder)`` for ``WITH … <INSERT/UPDATE/DELETE> …``; ``('', sql)`` otherwise.
+
+    So ``WITH c AS (…) INSERT INTO t SELECT … FROM c`` reaches the matchers (which anchor on the
+    verb) and the CTE is preserved when the body is evaluated. A leading ``WITH`` that drives a
+    plain ``SELECT`` (a read) is left untouched."""
+    if not _LEADING_WITH.match(sql):
+        return "", sql
+    idx = _find_top_level(sql, _DRIVING_DML)
+    if idx <= 0:
+        return "", sql
+    return sql[:idx].rstrip(), sql[idx:]
+
+
+def _fullmatch(pattern, sql):
+    return pattern.fullmatch(sql.strip())
+
+
+def _split_relation(rel: str) -> Tuple[Optional[str], Optional[str]]:
+    """`"db"."schema"."tbl"` / `schema.tbl` / `tbl` -> (schema, identifier), quotes stripped."""
+    parts = [p.strip().strip('"') for p in rel.strip().split(".")]
+    if not parts or not parts[-1]:
+        return None, None
+    identifier = parts[-1]
+    schema = parts[-2] if len(parts) >= 2 else None
+    return schema, identifier
+
+
+def _split_top_level_commas(s: str) -> List[str]:
+    """Split on commas that aren't inside parentheses or quotes (so ``left(email, 3)`` stays whole)."""
+    out, depth, start, quote = [], 0, 0, None
+    for i, ch in enumerate(s):
+        if quote:
+            if ch == quote:
+                quote = None
+        elif ch in ("'", '"'):
+            quote = ch
+        elif ch in "([":
+            depth += 1
+        elif ch in ")]":
+            depth -= 1
+        elif ch == "," and depth == 0:
+            out.append(s[start:i])
+            start = i + 1
+    out.append(s[start:])
+    return [p.strip() for p in out if p.strip()]
+
+
+class _DeltaDML:
+    """One attempt to handle a statement; ``run()`` returns True if it was applied to Delta."""
+
+    def __init__(self, cursor, root_path: str, storage_options, default_schema=None):
+        self.cursor = cursor
+        self.root_path = root_path.rstrip("/")
+        self.so = storage_options
+        self.default_schema = default_schema
+        self._with_clause = ""  # a leading `WITH …` preceding an INSERT, prepended to the body
+
+    def _loc(self, schema: str, identifier: str) -> str:
+        return f"{self.root_path}/{schema}/{identifier}"
+
+    def _resolve(self, rel: str):
+        """(schema, identifier, location) for ``rel``, falling back to default_schema for an
+        unqualified name (the connection API relies on a current database). (None, None, None) when
+        no schema can be determined."""
+        schema, identifier = _split_relation(rel)
+        schema = schema or self.default_schema
+        if not schema or not identifier:
+            return None, None, None
+        return schema, identifier, self._loc(schema, identifier)
+
+    def _exists(self, loc: str) -> bool:
+        return engine.table_exists(loc, self.so)
+
+    def _refresh_view(self, rel: str, schema: str, loc: str) -> None:
+        loc_sql = loc.replace("'", "''")
+        self.cursor.execute(f'create schema if not exists "{schema}"')
+        self.cursor.execute(
+            f"create or replace view {rel} as select * from delta_scan('{loc_sql}')"
+        )
+
+    def try_handle(self, sql: str) -> bool:
+        # CREATE TEMP/TEMPORARY TABLE is native DuckDB scratch by design — never capture it.
+        if _CREATE_TEMP_RE.match(sql):
+            return False
+        m = _fullmatch(_CREATE_AS, sql)
+        if m and "__duckrun" not in m.group("rel"):
+            return self._create_as(m)
+        m = _fullmatch(_CREATE_COLDEFS, sql)
+        if m and "__duckrun" not in m.group("rel"):
+            return self._create_coldefs(m)
+        m = _fullmatch(_INSERT_SELECT, sql)
+        if m:
+            return self._mutate(m, self._insert_select)
+        m = _fullmatch(_INSERT_VALUES, sql)
+        if m:
+            return self._mutate(m, self._insert_values)
+        m = _fullmatch(_DELETE, sql)
+        if m:
+            return self._mutate(m, self._delete)
+        m = _fullmatch(_UPDATE, sql)
+        if m:
+            return self._mutate(m, self._update)
+        m = _fullmatch(_ALTER_ADD, sql)
+        if m:
+            return self._mutate(m, self._alter_add)
+        m = _fullmatch(_DROP, sql)
+        if m:
+            return self._drop(m)
+        return False
+
+    # -- create table <rel> as <query>: always materialize as a duckrun Delta table ------------
+    def _create_as(self, m) -> bool:
+        rel = m.group("rel").strip()
+        schema, identifier, loc = self._resolve(rel)
+        if not loc:
+            return False
+        # dbt/cursor path (no default_schema): keep the ORIGINAL narrow interception — only a plain
+        # `create table … as select …` routes to Delta. The wider forms (`or replace`, a CTE or a
+        # parenthesised body) are a connection-API affordance; on the dbt path they must stay native
+        # so dbt keeps owning the relation. dbt-internal CTAS like store_failures' `create table … as
+        # (select …)` is a real TABLE dbt later drops/recreates — turning it into a delta_scan VIEW
+        # breaks that ("Existing object … is of type View, trying to drop type Table").
+        if self.default_schema is None and (
+            m.group("orrep") or not re.match(r"select\b", m.group("body").lstrip(), re.I)
+        ):
+            return False
+        # `if not exists` over a live (non-tombstone) table is a no-op — just (re)surface the view.
+        if m.group("ine") and self._exists(loc) and not is_dropped(self.cursor, loc, self.so):
+            self._refresh_view(rel, schema, loc)
+            return True
+        data = self.cursor.sql(m.group("body"))
+        # overwrite_schema so this replaces a prior table (or a drop-tombstone) wholesale — a live
+        # table is recreated with the real schema, clearing any tombstone marker.
+        engine.write_delta(loc, data, "overwrite", overwrite_schema=True, storage_options=self.so)
+        self._refresh_view(rel, schema, loc)
+        return True
+
+    # -- create table <rel> (<col defs>): an EMPTY Delta table (connection API only) -----------
+    def _create_coldefs(self, m) -> bool:
+        # Only the connection API (which carries a current database) makes a bare `CREATE TABLE
+        # (col defs)` a Delta table — so `CREATE TABLE` is always Delta-backed there. The dbt/cursor
+        # path passes default_schema=None: the seed loader emits this exact form and RELIES on it
+        # landing as a native DuckDB table, so we pass through untouched.
+        if self.default_schema is None:
+            return False
+        rel = m.group("rel").strip()
+        schema, identifier, loc = self._resolve(rel)
+        if not loc:
+            return False
+        # Let DuckDB parse the column defs (types, constraints, nested parens) by building the table
+        # as a TEMP, then take a 0-row typed relation from it and write that as an empty Delta table.
+        tmp = f"__duckrun_empty_{abs(hash((schema, identifier))) & 0xFFFFFFFF}"
+        self.cursor.execute(f'create or replace temp table "{tmp}" ({m.group("defs")})')
+        try:
+            empty = self.cursor.sql(f'select * from "{tmp}" limit 0')
+            engine.write_delta(loc, empty, "overwrite", overwrite_schema=True, storage_options=self.so)
+        finally:
+            self.cursor.execute(f'drop table if exists "{tmp}"')
+        self._refresh_view(rel, schema, loc)
+        return True
+
+    # -- forms that only apply when a Delta table already exists at the target ------------------
+    def _mutate(self, m, op) -> bool:
+        rel = m.group("rel").strip()
+        schema, identifier, loc = self._resolve(rel)
+        if not loc or not self._exists(loc):
+            return False  # native relation (e.g. the test's `fact`/`seed`) -> let DuckDB handle it
+        if self._with_clause and op != self._insert_select:
+            return False  # `WITH … UPDATE/DELETE` can't be expressed through a delta_rs predicate
+        op(m, rel, schema, loc)
+        self._refresh_view(rel, schema, loc)
+        return True
+
+    def _delete(self, m, rel, schema, loc) -> None:
+        where = m.group("where")
+        engine._delta_table(loc, self.so).delete(predicate=where.strip() if where else None)
+
+    def _update(self, m, rel, schema, loc) -> None:
+        updates = {}
+        for assign in _split_top_level_commas(m.group("set")):
+            col, _, expr = assign.partition("=")
+            updates[col.strip().strip('"')] = expr.strip()
+        where = m.group("where")
+        engine._delta_table(loc, self.so).update(
+            updates=updates, predicate=where.strip() if where else None
+        )
+
+    def _insert_select(self, m, rel, schema, loc) -> None:
+        body = m.group("body")
+        if self._with_clause:  # `WITH … INSERT INTO t SELECT …`: re-attach the CTE to the body
+            body = f"{self._with_clause} {body}"
+        cols = m.group("cols")
+        if cols:  # `insert into t (a, b) select …` → project the query onto the named columns
+            self._append_projected(loc, self._provided(cols), f"({body})")
+        else:  # column count/order already matches the target → append as-is
+            engine.write_delta(loc, self.cursor.sql(body), "append", storage_options=self.so)
+
+    def _insert_values(self, m, rel, schema, loc) -> None:
+        # `insert into <rel> [(<cols>)] values (...)`: the literals supply every target column when
+        # no list is given, in order; otherwise the named columns.
+        cols = m.group("cols")
+        provided = self._provided(cols) if cols else None
+        self._append_projected(loc, provided, f"(values {m.group('body')})")
+
+    @staticmethod
+    def _provided(cols: str) -> List[str]:
+        return [c.strip().strip('"') for c in cols.split(",")]
+
+    def _append_projected(self, loc, provided, derived: str) -> None:
+        """Append a ``derived`` table (a ``(values …)`` tuple list or a ``(select …)`` subquery) to
+        the Delta table at ``loc``, projecting its columns onto the FULL target schema: supplied
+        columns come from ``derived`` (positional when ``provided`` is None), any unsupplied target
+        column is a typed NULL, and every projected column is cast to the target column's type so
+        the appended Arrow schema matches the table exactly (what a plain SQL INSERT does, and it
+        stops a literal wider than the column from forcing delta_rs to add a new writer feature on
+        append)."""
+        loc_sql = loc.replace("'", "''")
+        template = self.cursor.sql(f"select * from delta_scan('{loc_sql}') limit 0")
+        target_cols = list(template.columns)
+        target_types = [str(t) for t in template.types]
+        by_lower = {c.lower(): c for c in target_cols}
+
+        if provided is None:  # positional → every target column, in order
+            provided = target_cols
+        else:  # explicit column list → canonicalize to the target's casing
+            provided = [by_lower.get(c.lower(), c) for c in provided]
+        provided_set = set(provided)
+
+        quoted = ", ".join('"' + c + '"' for c in provided)
+        inner = f"{derived} v({quoted})"
+        exprs = [
+            f'cast(v."{col}" as {typ}) as "{col}"' if col in provided_set
+            else f'cast(null as {typ}) as "{col}"'
+            for col, typ in zip(target_cols, target_types)
+        ]
+        data = self.cursor.sql(f"select {', '.join(exprs)} from {inner}")
+        engine.write_delta(loc, data, "append", storage_options=self.so)
+
+    def _alter_add(self, m, rel, schema, loc) -> None:
+        col = m.group("col").strip().strip('"')
+        # Keep only the column type (drop any DEFAULT/NULL clause); add it as an all-null column by
+        # rewriting the table with overwrite_schema so delta_rs accepts the widened schema.
+        coltype = re.split(r"\s+default\b|\s+null\b", m.group("def"), flags=re.I)[0].strip() or "VARCHAR"
+        loc_sql = loc.replace("'", "''")
+        data = self.cursor.sql(
+            f'select *, cast(null as {coltype}) as "{col}" from delta_scan(\'{loc_sql}\')'
+        )
+        engine.write_delta(loc, data, "overwrite", overwrite_schema=True, storage_options=self.so)
+
+    def _drop(self, m) -> bool:
+        # `drop table` on a duckrun relation: unregister the delta_scan view AND, via delta_rs,
+        # overwrite the table to a one-column tombstone (TOMBSTONE_COLUMN) so a later glob discovery
+        # hides it. NO data is deleted — delta_rs has no drop, and removing the Delta files would be
+        # a filesystem hack that fails on object stores. The directory persists until a human purges
+        # it; a later `create table ... as` overwrites the tombstone with real data. If the relation
+        # isn't a duckrun-managed Delta table, fall through and let DuckDB drop the native table.
+        rel = m.group("rel").strip()
+        schema, identifier, loc = self._resolve(rel)
+        if not loc or not self._exists(loc):
+            return False
+        tombstone = self.cursor.sql(f"select true as {TOMBSTONE_COLUMN}")
+        engine.write_delta(loc, tombstone, "overwrite", overwrite_schema=True, storage_options=self.so)
+        self.cursor.execute(f"drop view if exists {rel}")
+        return True
+
+
+def handle(cursor, root_path, storage_options, sql: str, default_schema=None) -> bool:
+    """Apply ``sql`` to Delta if it's a DML form targeting a duckrun-managed relation, using
+    ``cursor`` to evaluate any SELECT body and to (re)create the ``delta_scan`` view.
+
+    Every handled form goes through delta_rs (``engine.write_delta`` / ``DeltaTable.delete`` /
+    ``.update``), which carries ``storage_options`` and so works on local AND abfss/OneLake stores.
+    ``default_schema`` resolves an unqualified table name (the connection API has a current
+    database; the dbt path always renders fully-qualified names so passes None).
+    Returns True if handled (the caller must NOT also run it on DuckDB), False to pass through —
+    anything unrecognized, or (for the mutate forms) a target that isn't a Delta table.
+    """
+    if not root_path:
+        return False
+    sql = _strip_leading(sql)  # so leading comments/whitespace don't hide the verb
+    with_clause, body = _split_leading_with(sql)  # peel a leading `WITH …` off an INSERT/etc.
+    # Cheap pre-filter: only the candidate DML verbs.
+    head = body[:7].lower()
+    if not head.startswith(("delete", "update", "insert", "create", "alter", "drop")):
+        return False
+    dml = _DeltaDML(cursor, root_path, storage_options, default_schema)
+    dml._with_clause = with_clause
+    return dml.try_handle(body)

{duckrun-0.3.17.dev2 → duckrun-0.3.17.dev4}/dbt/adapters/duckrun/delta_plugin.py

@@ -6,6 +6,7 @@ connection (``configure_connection``), and on ``store()`` hands the model relati
 straight to delta_rs. DuckDB relations expose the Arrow C-stream interface, which
 deltalake 1.x consumes directly, so there is no pyarrow dependency.
 """
+import re
 from typing import Any, Optional
 
 from dbt.adapters.duckdb.plugins import BasePlugin
@@ -155,11 +156,22 @@ class Plugin(BasePlugin):
         # Table-like (non-incremental) models always overwrite. Incremental models
         # overwrite on first run / full-refresh, then apply the incremental strategy.
         if not incremental or full_refresh or not exists:
+            # This branch is a CREATE OR REPLACE: a table model, a --full-refresh, or a first run.
+            # When we are REPLACING an existing table (exists), allow delta_rs to replace the schema
+            # wholesale (schema_mode="overwrite") — the model SQL defines the new schema, exactly as
+            # `CREATE OR REPLACE TABLE` does on every other warehouse. Without it, delta_rs's strict
+            # overwrite keeps the OLD schema/protocol and so can't change a column's type or write a
+            # column needing a new writer feature the old table lacks (e.g. retyping to ::timestamp /
+            # timestampNtz). This is scoped to the full-rebuild replace ONLY — NOT append, safeappend,
+            # merge, or microbatch, which must keep their strict, schema-stable writes. A fresh create
+            # (not exists) doesn't need it. A user's explicit merge_schema still wins.
+            overwrite_schema = exists and not merge_schema
             with engine.mem_profile("overwrite", con=cur):
                 engine.write_delta(
                     path, data, "overwrite",
                     partition_by=partition_by,
                     merge_schema=merge_schema,
+                    overwrite_schema=overwrite_schema,
                     storage_options=storage_options,
                     compaction_threshold=self._compaction_threshold,
                 )
@@ -201,7 +213,7 @@ class Plugin(BasePlugin):
                     insert_only=(strategy == "insert"),
                     update_columns=cfg.get("merge_update_columns"),
                     exclude_columns=cfg.get("merge_exclude_columns"),
-                    predicates=self._merge_predicates(cfg),
+                    predicates=self._merge_predicates(cfg, data.columns),
                     update_condition=self._rewrite_merge_aliases(cfg.get("merge_update_condition")),
                     insert_condition=self._rewrite_merge_aliases(cfg.get("merge_insert_condition")),
                     merge_schema=evolve_schema,
@@ -470,16 +482,37 @@ class Plugin(BasePlugin):
             return None
         return str(expr).replace("DBT_INTERNAL_DEST", "target").replace("DBT_INTERNAL_SOURCE", "source")
 
+    @staticmethod
+    def _qualify_predicate(expr, columns):
+        """Prefix bare references to known target columns with ``target.``.
+
+        duckrun folds ``incremental_predicates`` into the merge condition
+        (``target.k = source.k AND <predicate>``). A bare column there (e.g. ``id != 2``) exists
+        on BOTH the source and target, so delta_rs rejects it as an ambiguous reference. dbt's
+        ``incremental_predicates`` constrain the existing/target rows (the delete+insert delete, the
+        merge ON), so we qualify bare column tokens to ``target.``. Only exact column-name tokens
+        that aren't already qualified (preceded by ``.``) or quoted/literal are rewritten — literals
+        and functions (e.g. ``current_date``, which is not a column) are left untouched."""
+        if not expr or not columns:
+            return expr
+        # Longest names first so a column that's a prefix of another isn't partially matched.
+        for col in sorted({str(c) for c in columns}, key=len, reverse=True):
+            # whole-word col, not preceded by '.', a word char, or a quote (already qualified/quoted).
+            pattern = re.compile(r'(?<![.\w"\'])' + re.escape(col) + r'\b', re.I)
+            expr = pattern.sub(lambda m: "target." + m.group(0), expr)
+        return expr
+
     @classmethod
-    def _merge_predicates(cls, cfg: dict):
+    def _merge_predicates(cls, cfg: dict, columns=None):
         """dbt ``incremental_predicates`` (or ``predicates``), with dbt's standard merge
-        aliases rewritten to the ones delta_rs uses here."""
+        aliases rewritten to the ones delta_rs uses here and bare column refs qualified to
+        ``target.`` (see ``_qualify_predicate``)."""
         preds = cfg.get("incremental_predicates") or cfg.get("predicates")
         if not preds:
             return None
         if isinstance(preds, str):
             preds = [preds]
-        return [cls._rewrite_merge_aliases(p) for p in preds]
+        return [cls._qualify_predicate(cls._rewrite_merge_aliases(p), columns) for p in preds]
 
     @staticmethod
     def _resolve_schema_change(on_schema_change, path, data, storage_options) -> bool:

{duckrun-0.3.17.dev2 → duckrun-0.3.17.dev4}/dbt/adapters/duckrun/engine.py

@@ -536,6 +536,37 @@ def table_exists(path: str, storage_options: Optional[Dict[str, str]] = None) ->
         return False
 
 
+def delta_stats(cur, path: str, storage_options: Optional[Dict[str, str]] = None):
+    """Cheap table statistics for ``dbt docs generate``, read from the Delta **log** (no data scan).
+
+    ``DeltaTable.get_add_actions()`` carries per-file ``num_records`` / ``size_bytes`` /
+    ``modification_time``; summing rows+bytes and taking the latest mtime gives the whole table's
+    stats without opening any data file. Aggregation goes through the DuckDB cursor (``cur``) via a
+    replacement scan over the arro3 table — no pyarrow dependency.
+
+    Returns ``{"num_rows", "bytes", "last_modified"}`` (last_modified = epoch milliseconds), or
+    ``None`` on ANY failure (a drop-tombstone, a missing table, an unreachable/credential-less remote
+    store). Best-effort by design: a statless catalog is fine, but a docs build must never break.
+    """
+    try:
+        add_actions = _delta_table(path, storage_options).get_add_actions()  # noqa: F841 (replacement scan)
+        row = cur.sql(
+            "select coalesce(sum(num_records), 0)::bigint, "
+            "coalesce(sum(size_bytes), 0)::bigint, "
+            "max(modification_time)::bigint from add_actions"
+        ).fetchone()
+    except Exception as exc:  # best-effort: docs stats must never fail catalog generation
+        logger.debug(f"duckrun: no Delta stats for {path!r}: {exc}")
+        return None
+    if row is None:
+        return None
+    return {
+        "num_rows": int(row[0]),
+        "bytes": int(row[1]),
+        "last_modified": int(row[2]) if row[2] is not None else None,
+    }
+
+
 # Delta column-metadata key under which we stash a dbt column description, and the dollar-quote
 # label used to embed arbitrary comment text (newlines, quotes, dollar signs) in COMMENT ON SQL.
 _DELTA_COMMENT_KEY = "comment"