databricks-sql-connector 4.2.7__tar.gz → 4.3.0__tar.gz

{databricks_sql_connector-4.2.7 → databricks_sql_connector-4.3.0}/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Release History
 
+# 4.3.0 (2026-06-12)
+- **New: optional Rust kernel backend (`use_kernel=True`).** Adds an alternative connection path backed by the native [`databricks-sql-kernel`](https://pypi.org/project/databricks-sql-kernel/) client (a Rust core exposed via PyO3), installable with the new `databricks-sql-connector[kernel]` extra. The kernel talks to Databricks over the **SEA (Statement Execution API) HTTP transport** — not Thrift — with CloudFetch and inline-Arrow result fetching, so `use_kernel=True` gives you a modern SEA-native client through the same DB-API surface. Supports PAT, OAuth M2M, and OAuth U2M auth. Requires Python >= 3.10 (the kernel wheel is `cp310-abi3`); on older interpreters the extra is a no-op and `use_kernel=True` raises a clear `ImportError`. The default backend remains Thrift — opt in per connection.
+- Kernel backend behavior is aligned with the Thrift backend so application code works the same either way: consistent cursor-state tracking (`query_id` / `get_query_state`), metadata (catalogs/schemas/tables/columns with JDBC-style filter semantics and case-insensitive `table_types`), DML `rowcount`, server-sourced async execution state, sync `cancel()`, fail-loud staging/volume operations, and structured error context (SQLSTATE, diagnostic info). Kernel logs surface through Python `logging` under the `databricks.sql.kernel` logger (databricks/databricks-sql-python#824, #825, #830, #838, #839 by @vikrantpuppala)
+- Revert the thrift 0.23.0 bump that broke installation on DBR LTS (ES-1960554) (databricks/databricks-sql-python#840 by @vikrantpuppala)
+
 # 4.2.7 (2026-06-02)
 - Extract SPOG org-id from cluster http_path for non-Thrift requests (databricks/databricks-sql-python#817 by @msrathore-db)
 - Remove empty chunks in CloudFetch concatenation (databricks/databricks-sql-python#814 by @jprakash-db)

{databricks_sql_connector-4.2.7 → databricks_sql_connector-4.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: databricks-sql-connector
-Version: 4.2.7
+Version: 4.3.0
 Summary: Databricks SQL Connector for Python
 License: Apache-2.0
 License-File: LICENSE
@@ -16,21 +16,24 @@ Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
+Provides-Extra: kernel
 Provides-Extra: pyarrow
+Requires-Dist: databricks-sql-kernel (>=0.2.0,<0.3.0) ; (python_version >= "3.10") and (extra == "kernel")
 Requires-Dist: lz4 (>=4.0.2,<5.0.0) ; python_version >= "3.8" and python_version < "3.14"
 Requires-Dist: lz4 (>=4.4.5,<5.0.0) ; python_version >= "3.14"
 Requires-Dist: oauthlib (>=3.1.0,<4.0.0)
 Requires-Dist: openpyxl (>=3.0.10,<4.0.0)
 Requires-Dist: pandas (>=1.2.5,<4.0.0) ; python_version >= "3.8" and python_version < "3.13"
 Requires-Dist: pandas (>=2.2.3,<4.0.0) ; python_version >= "3.13"
-Requires-Dist: pyarrow (>=14.0.1) ; (python_version >= "3.8" and python_version < "3.13") and (extra == "pyarrow")
+Requires-Dist: pyarrow (>=14.0.1) ; (python_version >= "3.10" and python_version < "3.13") and (extra == "pyarrow")
+Requires-Dist: pyarrow (>=14.0.1,<23) ; (python_version >= "3.8" and python_version < "3.10") and (extra == "pyarrow")
 Requires-Dist: pyarrow (>=18.0.0) ; (python_version == "3.13") and (extra == "pyarrow")
 Requires-Dist: pyarrow (>=22.0.0) ; (python_version >= "3.14") and (extra == "pyarrow")
 Requires-Dist: pybreaker (>=1.0.0,<2.0.0)
 Requires-Dist: pyjwt (>=2.0.0,<3.0.0)
 Requires-Dist: python-dateutil (>=2.8.0,<3.0.0)
 Requires-Dist: requests (>=2.18.1,<3.0.0)
-Requires-Dist: thrift (>=0.22.0,<0.24.0)
+Requires-Dist: thrift (>=0.22.0,<0.23.0)
 Requires-Dist: urllib3 (>=1.26)
 Project-URL: Bug Tracker, https://github.com/databricks/databricks-sql-python/issues
 Project-URL: Homepage, https://github.com/databricks/databricks-sql-python
@@ -68,6 +71,30 @@ Install using `pip install databricks-sql-connector`
 ### Installing the core library with PyArrow
 Install using `pip install databricks-sql-connector[pyarrow]`
 
+### Installing with the Rust kernel backend (`use_kernel=True`)
+Install using `pip install databricks-sql-connector[kernel]`
+
+This adds the optional [`databricks-sql-kernel`](https://pypi.org/project/databricks-sql-kernel/)
+extension (a native Rust client core, exposed via PyO3). Pass
+`use_kernel=True` to `sql.connect(...)` to route the connection through it
+instead of the default Thrift backend:
+
+```python
+connection = sql.connect(
+  server_hostname=host,
+  http_path=http_path,
+  access_token=token,
+  use_kernel=True,
+)
+```
+
+Notes:
+- Requires **Python >= 3.10** (the kernel wheel is published as
+  `cp310-abi3`). On older interpreters the `[kernel]` extra installs
+  nothing and `use_kernel=True` raises an `ImportError`.
+- The extra also pulls in PyArrow, which the kernel result path requires.
+- Authentication supports PAT (`access_token`), OAuth M2M, and OAuth U2M.
+
 
 ```bash
 export DATABRICKS_HOST=********.databricks.com

{databricks_sql_connector-4.2.7 → databricks_sql_connector-4.3.0}/README.md

@@ -30,6 +30,30 @@ Install using `pip install databricks-sql-connector`
 ### Installing the core library with PyArrow
 Install using `pip install databricks-sql-connector[pyarrow]`
 
+### Installing with the Rust kernel backend (`use_kernel=True`)
+Install using `pip install databricks-sql-connector[kernel]`
+
+This adds the optional [`databricks-sql-kernel`](https://pypi.org/project/databricks-sql-kernel/)
+extension (a native Rust client core, exposed via PyO3). Pass
+`use_kernel=True` to `sql.connect(...)` to route the connection through it
+instead of the default Thrift backend:
+
+```python
+connection = sql.connect(
+  server_hostname=host,
+  http_path=http_path,
+  access_token=token,
+  use_kernel=True,
+)
+```
+
+Notes:
+- Requires **Python >= 3.10** (the kernel wheel is published as
+  `cp310-abi3`). On older interpreters the `[kernel]` extra installs
+  nothing and `use_kernel=True` raises an `ImportError`.
+- The extra also pulls in PyArrow, which the kernel result path requires.
+- Authentication supports PAT (`access_token`), OAuth M2M, and OAuth U2M.
+
 
 ```bash
 export DATABRICKS_HOST=********.databricks.com

databricks_sql_connector-4.3.0/pyproject.toml

@@ -0,0 +1,135 @@
+[tool.poetry]
+name = "databricks-sql-connector"
+version = "4.3.0"
+description = "Databricks SQL Connector for Python"
+authors = ["Databricks <databricks-sql-connector-maintainers@databricks.com>"]
+license = "Apache-2.0"
+readme = "README.md"
+packages = [{ include = "databricks", from = "src" }]
+include = ["CHANGELOG.md"]
+
+[tool.poetry.dependencies]
+python = "^3.8.0"
+thrift = "~=0.22.0"
+pandas = [
+    { version = ">=1.2.5,<4.0.0", python = ">=3.8,<3.13" },
+    { version = ">=2.2.3,<4.0.0", python = ">=3.13" }
+]
+lz4 = [
+    { version = "^4.0.2", python = ">=3.8,<3.14" },
+    { version = "^4.4.5", python = ">=3.14" }
+]
+requests = "^2.18.1"
+oauthlib = "^3.1.0"
+openpyxl = "^3.0.10"
+urllib3 = ">=1.26"
+python-dateutil = "^2.8.0"
+pyarrow = [
+    # The <3.10 band is capped at <23 because pyarrow>=23 dropped
+    # Python 3.9 (it requires >=3.10). Without the cap, poetry tries to
+    # unify this entry with the kernel's transitive pyarrow>=23.0.1,<24
+    # across the 3.8–3.10 slice and `poetry lock` fails ("pyarrow is
+    # forbidden"). The cap removes no installable version — the newest
+    # pyarrow with a 3.9 wheel is 21.x — it just makes that explicit to
+    # the solver so the optional [kernel] extra (python>=3.10) can
+    # coexist. See the kernel dep + [kernel] extra below.
+    { version = ">=14.0.1,<23", python = ">=3.8,<3.10", optional=true },
+    { version = ">=14.0.1", python = ">=3.10,<3.13", optional=true },
+    { version = ">=18.0.0", python = ">=3.13,<3.14", optional=true },
+    { version = ">=22.0.0", python = ">=3.14", optional=true }
+]
+pyjwt = "^2.0.0"
+pybreaker = "^1.0.0"
+requests-kerberos = {version = "^0.15.0", optional = true}
+# Optional Rust kernel backend for ``use_kernel=True`` (PyO3 wheel).
+# Pulled in only via the ``[kernel]`` extra below. The published wheel
+# is ``abi3`` with ``Requires-Python: >=3.10`` (built ``abi3-py310``),
+# so the dependency is gated to Python >= 3.10: on 3.8/3.9 the
+# ``[kernel]`` extra resolves to nothing and ``use_kernel=True`` raises
+# a clear ImportError at runtime (see backend/kernel/_errors.py).
+#
+# Floor is 0.2.0 (``^0.2.0`` == ``>=0.2.0,<0.3.0``). The kernel is
+# pre-1.0, so each 0.x minor may carry breaking changes — the ``<0.3.0``
+# cap means we bump this deliberately when the kernel ships 0.3.0 rather
+# than letting a potentially-breaking minor flow in automatically. 0.2.0
+# keeps the same Requires-Python (>=3.10) and pyarrow (>=23.0.1,<24) pin
+# as 0.1.x, so the gating below is unchanged.
+databricks-sql-kernel = {version = "^0.2.0", optional = true, python = ">=3.10"}
+
+
+[tool.poetry.extras]
+pyarrow = ["pyarrow"]
+# ``pip install databricks-sql-connector[kernel]`` adds the Rust kernel
+# backend so ``use_kernel=True`` works. No-op on Python < 3.10 (the
+# wheel's floor) — those users get a runtime ImportError if they pass
+# ``use_kernel=True``.
+#
+# The kernel result path (``backend/kernel/result_set.py``) needs
+# pyarrow, but it is NOT listed in this extra on purpose: the published
+# kernel wheel declares ``pyarrow>=23.0.1,<24`` as a hard runtime
+# dependency, so ``pip install ...[kernel]`` already pulls a compatible
+# pyarrow transitively. Listing bare ``pyarrow`` here additionally
+# forces poetry to co-resolve an unconstrained pyarrow against the
+# kernel's ``>=23.0.1,<24`` (which itself requires Python >=3.10) across
+# the connector's full 3.8–3.14 support matrix, which is unsatisfiable
+# on 3.8/3.9 and breaks ``poetry lock``. The kernel's own dependency
+# metadata is the single source of truth for the pyarrow floor.
+kernel = ["databricks-sql-kernel"]
+
+[tool.poetry.group.dev.dependencies]
+pytest = "^7.1.2"
+mypy = "^1.10.1"
+pylint = ">=2.12.0"
+black = "^22.3.0"
+pytest-dotenv = "^0.5.2"
+pytest-cov = "^4.0.0"
+pytest-xdist = "^3.0.0"
+numpy = [
+    { version = ">=1.16.6", python = ">=3.8,<3.11" },
+    { version = ">=1.23.4", python = ">=3.11" },
+]
+
+[tool.poetry.urls]
+"Homepage" = "https://github.com/databricks/databricks-sql-python"
+"Bug Tracker" = "https://github.com/databricks/databricks-sql-python/issues"
+
+[build-system]
+requires = ["poetry-core>=1.0.0"]
+build-backend = "poetry.core.masonry.api"
+
+[tool.mypy]
+ignore_missing_imports = "true"
+exclude = ['ttypes\.py$', 'TCLIService\.py$']
+
+[tool.black]
+exclude = '/(\.eggs|\.git|\.hg|\.mypy_cache|\.nox|\.tox|\.venv|\.svn|_build|buck-out|build|dist|thrift_api)/'
+
+[tool.pytest.ini_options]
+markers = [
+    "reviewed: Test case has been reviewed by Databricks",
+    "serial: Tests that must run serially (not parallelized)",
+    "realkernel: Requires the real databricks-sql-kernel wheel and an unpolluted sys.modules (no fake kernel stub); must run in a separate pytest invocation from tests that fake databricks_sql_kernel (deselect with -m 'not realkernel', run alone with -m realkernel).",
+]
+minversion = "6.0"
+log_cli = "false"
+log_cli_level = "INFO"
+testpaths = ["tests"]
+env_files = ["test.env"]
+
+[tool.coverage.run]
+source = ["src"]
+branch = true
+omit = [
+    "*/tests/*",
+    "*/test_*",
+    "*/__pycache__/*",
+    "*/thrift_api/*",  
+]
+
+[tool.coverage.report]
+precision = 2
+show_missing = true
+skip_covered = false
+
+[tool.coverage.xml]
+output = "coverage.xml"

{databricks_sql_connector-4.2.7 → databricks_sql_connector-4.3.0}/src/databricks/sql/__init__.py

@@ -71,7 +71,7 @@ DATETIME = DBAPITypeObject("timestamp")
 DATE = DBAPITypeObject("date")
 ROWID = DBAPITypeObject()
 
-__version__ = "4.2.7"
+__version__ = "4.3.0"
 USER_AGENT_NAME = "PyDatabricksSqlConnector"
 
 # These two functions are pyhive legacy

{databricks_sql_connector-4.2.7 → databricks_sql_connector-4.3.0}/src/databricks/sql/backend/kernel/_errors.py

@@ -38,6 +38,8 @@ flow is visible at the call site, the helper is a pure function
 
 from __future__ import annotations
 
+import logging
+
 from databricks.sql.exc import (
     DatabaseError,
     Error,
@@ -50,13 +52,40 @@ try:
     import databricks_sql_kernel as _kernel  # type: ignore[import-not-found]
 except ImportError as exc:  # pragma: no cover - same hint as client.py
     raise ImportError(
-        "use_kernel=True requires the databricks-sql-kernel extension, which "
-        "is not yet published on PyPI. Build and install it locally from the "
-        "databricks-sql-kernel repo:\n"
-        "  cd databricks-sql-kernel/pyo3 && maturin develop --release\n"
-        "(into the same venv as databricks-sql-connector)."
+        "use_kernel=True requires the optional databricks-sql-kernel "
+        "extension, which is not installed. Install it with:\n"
+        '  pip install "databricks-sql-connector[kernel]"\n'
+        "The kernel wheel requires Python >= 3.10; on older interpreters "
+        "use_kernel is unavailable. For local kernel development you can "
+        "instead build it from the databricks-sql-kernel repo:\n"
+        "  cd databricks-sql-kernel/pyo3 && maturin develop --release"
     ) from exc
 
+# Route the kernel's Rust-side logs into Python's ``logging`` as soon as
+# the extension loads. The kernel emits under the ``databricks.sql.kernel``
+# logger (a child of the connector's ``databricks.sql`` namespace), so a
+# customer who configures ``databricks.sql`` logging gets kernel logs for
+# free with no extra setup.
+#
+# This is a best-effort, non-essential feature: it must never take down
+# ``use_kernel=True`` for a process. ``getattr`` guards against an older
+# kernel wheel that predates the function. The ``try`` guards against the
+# call itself throwing — note ``except BaseException`` is deliberate: a
+# panic raised across the PyO3 boundary surfaces as
+# ``pyo3_runtime.PanicException``, which derives from ``BaseException``
+# (not ``Exception``), so a narrower clause would let it escape module
+# import and fail every kernel-backed connection. The kernel side is
+# idempotent and returns rather than panics on a double install, but we
+# do not rely on that here — the guard holds regardless of the Rust impl.
+_kernel_init_logging = getattr(_kernel, "init_logging", None)
+if _kernel_init_logging is not None:
+    try:
+        _kernel_init_logging()
+    except BaseException as exc:  # noqa: BLE001 - see comment above re: PanicException
+        logging.getLogger(__name__).debug(
+            "kernel log bridge init failed; continuing without it: %r", exc
+        )
+
 
 # Map a kernel `code` slug to the PEP 249 exception class that best
 # captures it. The match isn't a perfect 1:1 — PEP 249 has a
@@ -97,7 +126,23 @@ def reraise_kernel_error(exc: "_kernel.KernelError") -> "Error":
     """
     code = getattr(exc, "code", "Unknown")
     cls = _CODE_TO_EXCEPTION.get(code, DatabaseError)
-    new = cls(getattr(exc, "message", str(exc)))
+
+    # For ServerOperationError, reproduce the Thrift backend's
+    # ``context`` dict so callers that read
+    # ``err.context["diagnostic-info"]`` (the Spark stack trace) /
+    # ``err.context["operation-id"]`` get the same shape on the kernel
+    # path. ``diagnostic_info`` is forwarded from the kernel error (it
+    # now crosses the PyO3 boundary; older wheels return ``None`` via
+    # ``getattr``, so this degrades gracefully). Matches
+    # thrift_backend.py's ServerOperationError construction.
+    context = None
+    if cls is ServerOperationError:
+        context = {
+            "operation-id": getattr(exc, "query_id", None),
+            "diagnostic-info": getattr(exc, "diagnostic_info", None),
+        }
+    new = cls(getattr(exc, "message", str(exc)), context)
+
     for attr in (
         "code",
         "sql_state",
@@ -106,6 +151,12 @@ def reraise_kernel_error(exc: "_kernel.KernelError") -> "Error":
         "http_status",
         "retryable",
         "query_id",
+        # Extended server status now forwarded across the PyO3 boundary
+        # (kernel #121). ``getattr(..., None)`` keeps this forward-safe
+        # against an older wheel that doesn't set these attrs.
+        "display_message",
+        "diagnostic_info",
+        "error_details_json",
     ):
         setattr(new, attr, getattr(exc, attr, None))
     return new