duckrun 0.3.17.dev7__tar.gz → 0.3.18__tar.gz

{duckrun-0.3.17.dev7/duckrun.egg-info → duckrun-0.3.18}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: duckrun
-Version: 0.3.17.dev7
+Version: 0.3.18
 Summary: A dbt adapter that runs SQL in DuckDB and materializes to Delta Lake (delta_rs).
 Author: mim
 License: MIT
@@ -12,7 +12,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: dbt-duckdb>=1.8
 Requires-Dist: dbt-core<2.0,>=1.8
-Requires-Dist: duckdb==1.5.4
+Requires-Dist: duckdb<1.6.0,>=1.5.4
 Requires-Dist: deltalake<1.5.1,>=1.5.0
 Requires-Dist: requests
 Provides-Extra: local
@@ -67,6 +67,21 @@ pip install duckrun
 
 That single install pulls in `dbt-duckdb` (and therefore `duckdb`) plus `deltalake`.
 
+### In a Microsoft Fabric Python notebook
+
+duckrun needs `duckdb` ≥ 1.5.4 — the release where `delta_scan` gained its `version => N`
+parameter, which duckrun uses for snapshot-pinned reads. Fabric notebooks ship a **stable**
+`duckdb` release, which trails the newest one, so the `duckdb` already loaded in the kernel may
+predate 1.5.4. Upgrade, then restart the Python kernel so the new version loads.
+
+```python
+!pip install duckrun --upgrade
+notebookutils.session.restartPython()
+```
+
+If you skip the restart, duckrun fails loud at `connect()` (and on `dbt run`) and tells you to
+restart — it won't quietly run on the older `duckdb`/`deltalake` still bound in the kernel.
+
 ## Configure your profile
 
 ```yaml
@@ -86,6 +101,14 @@ my_project:
 Persisted models are written to `<root_path>/<schema>/<model>` (e.g.
 `./warehouse/dbo/orders`), or to an explicit `config(location=...)`.
 
+### OneLake: use GUID paths for now
+
+Address OneLake tables by **workspace GUID + lakehouse GUID**, not friendly names —
+`abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/...`. This
+sidesteps an upstream `duckdb-delta` read bug ("No files in log segment") that is **already fixed
+upstream but still rolling out to production OneLake**. Friendly-name paths will work again once
+the fix finishes deploying.
+
 ### Fabric Lakehouse without a schema
 
 A schema-less Lakehouse (tables straight under `Tables/`, no `Tables/<schema>/` grouping) is
@@ -95,7 +118,7 @@ let the schema fill that slot:
 
 ```yaml
       schema: Tables
-      root_path: "abfss://<ws>@onelake.dfs.fabric.microsoft.com/<lh>.Lakehouse"
+      root_path: "abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>"
 ```
 
 Since models are written to `<root_path>/<schema>/<model>`, this lands them at
@@ -309,7 +332,7 @@ unchanged since the call, else raises `CommitFailedError`.
 
 ```python
 import duckrun
-conn = duckrun.connect("abfss://ws@onelake.dfs.fabric.microsoft.com/lh.Lakehouse/Tables/dbo")
+conn = duckrun.connect("abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/dbo")
 conn.sql("select * from orders").write.mode("overwrite").saveAsTable("orders_copy")
 conn.table("orders_copy").show()
 

{duckrun-0.3.17.dev7 → duckrun-0.3.18}/README.md

@@ -38,6 +38,21 @@ pip install duckrun
 
 That single install pulls in `dbt-duckdb` (and therefore `duckdb`) plus `deltalake`.
 
+### In a Microsoft Fabric Python notebook
+
+duckrun needs `duckdb` ≥ 1.5.4 — the release where `delta_scan` gained its `version => N`
+parameter, which duckrun uses for snapshot-pinned reads. Fabric notebooks ship a **stable**
+`duckdb` release, which trails the newest one, so the `duckdb` already loaded in the kernel may
+predate 1.5.4. Upgrade, then restart the Python kernel so the new version loads.
+
+```python
+!pip install duckrun --upgrade
+notebookutils.session.restartPython()
+```
+
+If you skip the restart, duckrun fails loud at `connect()` (and on `dbt run`) and tells you to
+restart — it won't quietly run on the older `duckdb`/`deltalake` still bound in the kernel.
+
 ## Configure your profile
 
 ```yaml
@@ -57,6 +72,14 @@ my_project:
 Persisted models are written to `<root_path>/<schema>/<model>` (e.g.
 `./warehouse/dbo/orders`), or to an explicit `config(location=...)`.
 
+### OneLake: use GUID paths for now
+
+Address OneLake tables by **workspace GUID + lakehouse GUID**, not friendly names —
+`abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/...`. This
+sidesteps an upstream `duckdb-delta` read bug ("No files in log segment") that is **already fixed
+upstream but still rolling out to production OneLake**. Friendly-name paths will work again once
+the fix finishes deploying.
+
 ### Fabric Lakehouse without a schema
 
 A schema-less Lakehouse (tables straight under `Tables/`, no `Tables/<schema>/` grouping) is
@@ -66,7 +89,7 @@ let the schema fill that slot:
 
 ```yaml
       schema: Tables
-      root_path: "abfss://<ws>@onelake.dfs.fabric.microsoft.com/<lh>.Lakehouse"
+      root_path: "abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>"
 ```
 
 Since models are written to `<root_path>/<schema>/<model>`, this lands them at
@@ -280,7 +303,7 @@ unchanged since the call, else raises `CommitFailedError`.
 
 ```python
 import duckrun
-conn = duckrun.connect("abfss://ws@onelake.dfs.fabric.microsoft.com/lh.Lakehouse/Tables/dbo")
+conn = duckrun.connect("abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/dbo")
 conn.sql("select * from orders").write.mode("overwrite").saveAsTable("orders_copy")
 conn.table("orders_copy").show()
 

duckrun-0.3.18/dbt/adapters/duckrun/__version__.py

@@ -0,0 +1 @@
+version = "0.3.18"

{duckrun-0.3.17.dev7 → duckrun-0.3.18}/dbt/adapters/duckrun/impl.py

@@ -32,6 +32,10 @@ class DuckrunConnectionManager(DuckDBConnectionManager):
 
     @classmethod
     def open(cls, connection):
+        # Fail loud if the kernel still has Fabric's stale duckdb/deltalake loaded (installed an
+        # upgrade but skipped notebookutils.session.restartPython()). Lazy import: same wheel.
+        from duckrun._runtime import check_runtime_versions
+        check_runtime_versions()
         # duckrun runs single-threaded, so it uses ONE DuckDB connection for the whole run
         # (DuckrunEnvironment) instead of dbt-duckdb's per-handle cursors — see environment.py.
         # Pre-seed the base class's singleton _ENV with it for the local case; remote/MotherDuck

duckrun-0.3.18/duckrun/_runtime.py

@@ -0,0 +1,51 @@
+"""Runtime version guardrail.
+
+duckrun needs ``duckdb`` >= 1.5.4 — the release where ``delta_scan`` gained its ``version => N``
+parameter (used for snapshot-pinned reads) — and ``deltalake`` >= 1.5.0 (for the merge
+``max_spill_size`` cap). A Microsoft Fabric Python notebook ships a *stable* ``duckdb`` release,
+which trails the newest one, so the ``duckdb`` already imported in the kernel may predate 1.5.4.
+``pip install duckrun --upgrade`` writes the new wheels to disk, but the already-loaded modules stay
+bound until the kernel restarts — so a user who skips the restart would keep running on the older
+modules, quietly losing snapshot-pinned reads and the spill cap.
+
+This check turns that into a loud, actionable error. It inspects the *loaded* versions (not the
+pin), so it fires exactly on the forgot-to-restart case.
+"""
+from packaging.version import Version
+
+# Floors duckrun needs at *runtime* — keep in sync with the pins in pyproject.toml:
+#   duckdb 1.5.4    -> delta_scan('...', version => N) for snapshot-pinned incremental reads
+#   deltalake 1.5.0 -> max_spill_size on MERGE to cap merge RAM and avoid OOM on large upserts
+_MIN_DUCKDB = "1.5.4"
+_MIN_DELTALAKE = "1.5.0"
+
+_REMEDY = (
+    "In a Fabric Python notebook, upgrade then restart the kernel so the new versions load:\n"
+    "    !pip install duckrun --upgrade\n"
+    "    notebookutils.session.restartPython()\n"
+    "then re-run. (Elsewhere: pip install -U 'duckdb>={duckdb}' 'deltalake>={deltalake}' and "
+    "restart the interpreter.)"
+).format(duckdb=_MIN_DUCKDB, deltalake=_MIN_DELTALAKE)
+
+
+def check_runtime_versions():
+    """Raise ``RuntimeError`` if the *loaded* duckdb/deltalake are older than duckrun requires.
+
+    Catches the notebook "installed but forgot ``restartPython()``" case: the kernel keeps the
+    older duckdb/deltalake bound until restart. Idempotent and cheap; called at each entry point
+    (``duckrun.connect()`` and the dbt connection open).
+    """
+    import duckdb
+    import deltalake
+
+    too_old = []
+    if Version(duckdb.__version__) < Version(_MIN_DUCKDB):
+        too_old.append(f"duckdb {duckdb.__version__} (need >= {_MIN_DUCKDB})")
+    if Version(deltalake.__version__) < Version(_MIN_DELTALAKE):
+        too_old.append(f"deltalake {deltalake.__version__} (need >= {_MIN_DELTALAKE})")
+
+    if too_old:
+        raise RuntimeError(
+            "duckrun needs a newer " + " and ".join(too_old) + " than the kernel has loaded.\n"
+            + _REMEDY
+        )

{duckrun-0.3.17.dev7 → duckrun-0.3.18}/duckrun/session.py

@@ -16,6 +16,7 @@ import duckdb
 
 from dbt.adapters.duckrun import delta_dml, engine, remote, secret
 from . import auth
+from ._runtime import check_runtime_versions
 
 
 # Statements that would WRITE to a table — rejected by the read-only conn.sql() with a pointer to
@@ -105,6 +106,40 @@ def _qlit(text: str) -> str:
     return str(text).replace("'", "''")
 
 
+def _strip_query_context(msg: str) -> str:
+    """DuckDB appends the offending statement to errors as ``\\nLINE N: <sql>\\n   ^``. When that
+    statement is one duckrun generated internally (the ``delta_scan`` view), echoing it back is
+    noise that makes the failure look like it's about the caller's input. Keep the real error
+    text; drop the generated-SQL context."""
+    idx = msg.find("\nLINE ")
+    return msg[:idx].rstrip() if idx != -1 else msg
+
+
+_GUID = re.compile(r"^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$")
+
+
+def _onelake_guid_hint(root_path: str) -> Optional[str]:
+    """Workaround note for the OneLake ``delta_scan`` bug, shown only when a friendly-name
+    ``abfss://`` path is involved. OneLake's delta_scan can fail to enumerate a valid table's
+    ``_delta_log`` when the path uses friendly workspace/lakehouse names (duckdb-delta#307); the
+    GUID form reads fine. Returns ``None`` for non-abfss paths or paths already using GUIDs (no
+    point nagging those)."""
+    if not remote.is_abfss(root_path):
+        return None
+    workspace, _host, path = remote._parse_abfss(root_path)
+    lakehouse = path.split("/", 1)[0] if path else ""
+    if lakehouse.lower().endswith(".lakehouse"):
+        lakehouse = lakehouse[: -len(".Lakehouse")]
+    if _GUID.match(workspace) and _GUID.match(lakehouse):
+        return None
+    return (
+        "OneLake's delta_scan can fail to read a valid table's _delta_log when the abfss path uses "
+        "friendly names — a known upstream issue (duckdb-delta#307). Until it's fixed, use the "
+        "workspace and lakehouse GUIDs, e.g. "
+        "abfss://<workspace-guid>@onelake.dfs.fabric.microsoft.com/<lakehouse-guid>/Tables"
+    )
+
+
 def _split_root_schema(path: str, schema: Optional[str]):
     """Normalize ``path`` into ``(root_path, schema)``.
 
@@ -227,10 +262,22 @@ class DuckSession:
 
     def _register_view(self, schema: str, table: str):
         path = f"{self.root_path.rstrip('/')}/{schema}/{table}"
-        self.con.execute(
-            f"CREATE OR REPLACE VIEW {_qid(schema)}.{_qid(table)} AS "
-            f"SELECT * FROM delta_scan('{_qlit(path)}')"
-        )
+        try:
+            self.con.execute(
+                f"CREATE OR REPLACE VIEW {_qid(schema)}.{_qid(table)} AS "
+                f"SELECT * FROM delta_scan('{_qlit(path)}')"
+            )
+        except Exception as exc:
+            # delta_scan failed reading the table. Keep the real engine error (it's the signal —
+            # e.g. the OneLake "No files in log segment" delta-kernel bug), but drop DuckDB's echo
+            # of the CREATE VIEW statement *we* generated, and say which table/path it was. Suppress
+            # the chained original (`from None`) so the noisy SQL echo doesn't reappear in tracebacks.
+            hint = _onelake_guid_hint(self.root_path)
+            raise RuntimeError(
+                f"duckrun: could not read Delta table {schema}.{table} at '{path}':\n"
+                f"{_strip_query_context(str(exc))}"
+                + (f"\n\n{hint}" if hint else "")
+            ) from None
 
     def _set_search_path(self, schema: str):
         try:
@@ -573,4 +620,5 @@ def connect(path: str, storage_options: Optional[Dict[str, str]] = None,
         >>> conn.sql("SHOW TABLES").show()
         >>> conn.sql("select * from orders").write.mode("overwrite").saveAsTable("orders_copy")
     """
+    check_runtime_versions()  # fail loud if Fabric's stale duckdb/deltalake are still loaded
     return DuckSession(path, storage_options, schema, compaction_threshold)

{duckrun-0.3.17.dev7 → duckrun-0.3.18/duckrun.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: duckrun
-Version: 0.3.17.dev7
+Version: 0.3.18
 Summary: A dbt adapter that runs SQL in DuckDB and materializes to Delta Lake (delta_rs).
 Author: mim
 License: MIT
@@ -12,7 +12,7 @@ Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: dbt-duckdb>=1.8
 Requires-Dist: dbt-core<2.0,>=1.8
-Requires-Dist: duckdb==1.5.4
+Requires-Dist: duckdb<1.6.0,>=1.5.4
 Requires-Dist: deltalake<1.5.1,>=1.5.0
 Requires-Dist: requests
 Provides-Extra: local
@@ -67,6 +67,21 @@ pip install duckrun
 
 That single install pulls in `dbt-duckdb` (and therefore `duckdb`) plus `deltalake`.
 
+### In a Microsoft Fabric Python notebook
+
+duckrun needs `duckdb` ≥ 1.5.4 — the release where `delta_scan` gained its `version => N`
+parameter, which duckrun uses for snapshot-pinned reads. Fabric notebooks ship a **stable**
+`duckdb` release, which trails the newest one, so the `duckdb` already loaded in the kernel may
+predate 1.5.4. Upgrade, then restart the Python kernel so the new version loads.
+
+```python
+!pip install duckrun --upgrade
+notebookutils.session.restartPython()
+```
+
+If you skip the restart, duckrun fails loud at `connect()` (and on `dbt run`) and tells you to
+restart — it won't quietly run on the older `duckdb`/`deltalake` still bound in the kernel.
+
 ## Configure your profile
 
 ```yaml
@@ -86,6 +101,14 @@ my_project:
 Persisted models are written to `<root_path>/<schema>/<model>` (e.g.
 `./warehouse/dbo/orders`), or to an explicit `config(location=...)`.
 
+### OneLake: use GUID paths for now
+
+Address OneLake tables by **workspace GUID + lakehouse GUID**, not friendly names —
+`abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/...`. This
+sidesteps an upstream `duckdb-delta` read bug ("No files in log segment") that is **already fixed
+upstream but still rolling out to production OneLake**. Friendly-name paths will work again once
+the fix finishes deploying.
+
 ### Fabric Lakehouse without a schema
 
 A schema-less Lakehouse (tables straight under `Tables/`, no `Tables/<schema>/` grouping) is
@@ -95,7 +118,7 @@ let the schema fill that slot:
 
 ```yaml
       schema: Tables
-      root_path: "abfss://<ws>@onelake.dfs.fabric.microsoft.com/<lh>.Lakehouse"
+      root_path: "abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>"
 ```
 
 Since models are written to `<root_path>/<schema>/<model>`, this lands them at
@@ -309,7 +332,7 @@ unchanged since the call, else raises `CommitFailedError`.
 
 ```python
 import duckrun
-conn = duckrun.connect("abfss://ws@onelake.dfs.fabric.microsoft.com/lh.Lakehouse/Tables/dbo")
+conn = duckrun.connect("abfss://<workspace_id>@onelake.dfs.fabric.microsoft.com/<lakehouse_id>/Tables/dbo")
 conn.sql("select * from orders").write.mode("overwrite").saveAsTable("orders_copy")
 conn.table("orders_copy").show()
 

{duckrun-0.3.17.dev7 → duckrun-0.3.18}/duckrun.egg-info/SOURCES.txt

@@ -21,6 +21,7 @@ dbt/include/duckrun/macros/materializations/incremental.sql
 dbt/include/duckrun/macros/materializations/snapshot.sql
 dbt/include/duckrun/macros/materializations/table.sql
 duckrun/__init__.py
+duckrun/_runtime.py
 duckrun/auth.py
 duckrun/delta_table.py
 duckrun/session.py

{duckrun-0.3.17.dev7 → duckrun-0.3.18}/duckrun.egg-info/requires.txt

@@ -1,6 +1,6 @@
 dbt-duckdb>=1.8
 dbt-core<2.0,>=1.8
-duckdb==1.5.4
+duckdb<1.6.0,>=1.5.4
 deltalake<1.5.1,>=1.5.0
 requests
 

{duckrun-0.3.17.dev7 → duckrun-0.3.18}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "duckrun"
-version = "0.3.17.dev7"
+version = "0.3.18"
 description = "A dbt adapter that runs SQL in DuckDB and materializes to Delta Lake (delta_rs)."
 readme = "README.md"
 license = {text = "MIT"}
@@ -21,23 +21,22 @@ dependencies = [
     # fails at `from dbt.cli.main import dbtRunner`. Declared directly because we only depend on
     # dbt-core transitively, so the ceiling has to live here to bite.
     "dbt-core>=1.8,<2.0",
-    # Pin exactly 1.5.4 (stable). NOT a bare ">=1.5.4" floor: this project resolves with pip --pre
-    # (to pick up dbt pre-releases), and --pre is global, so an open floor lets pip pull the newest
-    # duckdb *prerelease* — verified: "--pre duckdb>=1.5.4" resolves to 1.6.0.dev12, an unstable,
-    # untested engine. An upper cap (<1.6.0) blocks 1.6.0.dev but a future 1.5.5.devN would still
-    # slip through under --pre, so pin exactly, matching the exact deltalake pin below.
-    # 1.5.4 is the first stable build whose bundled duckdb-delta
-    # extension supports `delta_scan('...', version => N)` (duckdb-delta #312) — the version-pinned
-    # read this project now relies on to make the incremental read and the write commit resolve at
-    # ONE Delta snapshot (Spark single-snapshot MERGE parity; see the staging-read pin in
-    # _delta_core.sql and merge_delta's read_version). The earlier 1.5.2+ "No files in log segment"
-    # read regression is avoided by addressing OneLake tables via GUID (workspace_id/lakehouse_id)
-    # abfss paths, and is fixed upstream.
+    # duckdb floor is 1.5.4 with a <1.6.0 cap (a floor, NOT an exact pin): 1.5.4 is the first
+    # stable build whose bundled duckdb-delta extension supports `delta_scan('...', version => N)`
+    # (duckdb-delta #312) — the version-pinned read this project relies on to make the incremental
+    # read and the write commit resolve at ONE Delta snapshot (Spark single-snapshot MERGE parity;
+    # see the staging-read pin in _delta_core.sql and merge_delta's read_version). Stable 1.5.x
+    # patches above the floor are fine. The <1.6.0 cap matters because this project resolves with
+    # pip --pre (to pick up dbt pre-releases) and --pre is global: an open floor would let pip pull
+    # an unstable duckdb *prerelease* (verified: "--pre duckdb>=1.5.4" resolves to 1.6.0.dev12).
+    # Per PEP 440, "<1.6.0" excludes 1.6.0 AND its prereleases (1.6.0.devN), so no dev build slips
+    # in. The earlier 1.5.2+ "No files in log segment" read regression is avoided by addressing
+    # OneLake tables via GUID (workspace_id/lakehouse_id) abfss paths, and is fixed upstream.
     # deltalake floor stays 1.5.0 (not just a ceiling): 1.5.0 is the first release with MERGE
     # disk-spill config (max_spill_size), which engine.merge_delta relies on to cap the merge's RAM
     # and avoid OOM on large upserts; the matching <1.5.1 ceiling avoids the deltalake delta-log
     # write-side regression, pinning exactly 1.5.0.
-    "duckdb==1.5.4",
+    "duckdb>=1.5.4,<1.6.0",
     "deltalake>=1.5.0,<1.5.1",
     # The top-level connection API (duckrun.connect) discovers OneLake tables via the DFS REST
     # API directly; requests is otherwise only a transitive dbt dependency.

duckrun-0.3.17.dev7/dbt/adapters/duckrun/__version__.py

@@ -1 +0,0 @@
-version = "0.3.17.dev7"