arrowbricks 3.0.4__tar.gz → 3.1.1__tar.gz

{arrowbricks-3.0.4 → arrowbricks-3.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: arrowbricks
-Version: 3.0.4
+Version: 3.1.1
 Requires-Dist: arro3-core>=0.8 ; extra == 'arro3'
 Provides-Extra: arro3
 License-File: LICENSE
@@ -77,9 +77,14 @@ async for item in client.stream_query_json("SELECT * FROM my_catalog.my_schema.b
 
 See [`examples/basic.py`](examples/basic.py) for a runnable version,
 [`examples/cursor_paging.py`](examples/cursor_paging.py) for paging a large
-result with `fetchmany`/`fetchmany_arrow` without buffering it all upfront, or
+result with `fetchmany`/`fetchmany_arrow` without buffering it all upfront,
 [`examples/azure_auth.py`](examples/azure_auth.py) for a caching
-`token_provider` built on Azure AD (`DefaultAzureCredential`).
+`token_provider` built on Azure AD (`DefaultAzureCredential`, needs
+`azure-identity`), or [`examples/oauth_m2m_auth.py`](examples/oauth_m2m_auth.py)
+for the same idea using Databricks' own OAuth machine-to-machine
+client-credentials flow instead -- like `databricks-sql-connector`'s
+`auth_type="databricks-oauth"`, but stdlib-only (`urllib.request`), no
+extra dependency.
 
 ## FastAPI SSE example
 
@@ -135,19 +140,71 @@ arrowbricks has no opinion on *how* you get a token and no cloud-SDK dependency
 conn = connect(host=..., warehouse_id=..., token_provider=my_token_provider)
 ```
 
+## Observability
+
+Pass `on_event` to `connect`/`DatabricksClient` to get a `QueryStats` snapshot once per query, at completion (success, cancelled, timed out, or errored):
+
+```python
+def log_query(stats):
+    print(
+        f"{stats.statement_id}: {stats.outcome} in {stats.fetch_s:.2f}s, {stats.bytes_downloaded} bytes, {stats.retry_count} retries"
+    )
+
+
+conn = connect(host=..., warehouse_id=..., token=..., on_event=log_query)
+```
+
+`on_event` is sync or async, same as `token_provider`, and applies to every query run through that client (not passed per-call). It's strictly fire-and-forget: any exception it raises is caught and swallowed on the Rust side, never surfaced to your query, and dispatching it never blocks or measurably slows the actual fetch -- a slow or broken logging/metrics callback can't make your queries slower or fail them. `QueryStats` fields: `statement_id: str`, `protocol: "thrift" | "sea"`, `warehouse_wait_s: float` (0.0 if the client already knew the warehouse was running), `submit_to_ready_s: float` (statement submit -> ready to fetch), `fetch_s: float` (download + decode), `num_chunks: int`, `bytes_downloaded: int`, `retry_count: int`, `concurrency_used: int`, and `outcome: "success" | "cancelled" | "timeout" | "error"`.
+
+`on_event` fires once the fetch of a given query actually completes (or is abandoned) -- a `Cursor` whose result is never fully drained (e.g. only a partial `fetchmany()`, then abandoned) never fires one, same category as this package's existing documented gap around GC-based cleanup.
+
+## Cancellation
+
+When a chunk download times out (`total_timeout_s`) or your code cancels the surrounding coroutine (`task.cancel()`/`asyncio.wait_for`) while it's in flight, arrowbricks fires a best-effort server-side cancel in the background (Thrift's `CancelOperation`, or SEA's `POST .../cancel`) so Databricks stops running the query instead of finishing it for nobody. This is fire-and-forget: the `QueryTimeout`/cancellation still reaches you immediately, and the cancel call's own result (success or failure) is never surfaced or awaited.
+
+This covers `Cursor.fetchall_streamed()`/`fetchall_arrow_streamed()`, `client.stream_query_json(...)`, and the lower-level `._core.Client`'s own streamed APIs -- everything that wraps the chunk-download phase in the Rust-level heartbeat. Neither `total_timeout_s` nor a bare `task.cancel()` on `Cursor.execute()`/`execute_streamed()` itself (the initial submit/poll wait) triggers a cancel -- by the time a query's chunks are being fetched at all, the statement has typically already finished running server-side, so there's usually nothing left to cancel there in practice.
+
+## Errors
+
+Every exception arrowbricks raises itself is an `ArrowbricksError` -- a `RuntimeError` subclass, so an `except RuntimeError` written before this hierarchy existed keeps catching everything unchanged. Three subclasses split by what actually changes what your code should do next:
+
+- `TransientError` -- a network blip, connection reset, or 5xx that survived every internal retry (`retry_attempts`, exponential backoff up to `retry_max_wait_s` -- see below) before ever reaching you. Back off further and try again later; retrying immediately just repeats what already failed.
+- `AuthError` -- HTTP 401/403, even after every internal retry re-fetched a token from your `token_provider`, *or* your `token_provider` itself raised (e.g. its own OAuth refresh call came back unauthorized). Treat the credential itself as bad/expired, not a transient blip.
+- `StatementError` -- the SQL statement itself failed or was canceled server-side (bad SQL, a permissions error, a warehouse-side query failure). Retrying the identical statement will fail identically; this isn't a transport problem.
+
+Anything else (a generic non-auth/non-statement HTTP error, an internal parse/decode failure) raises the plain `ArrowbricksError` base.
+
+```python
+from arrowbricks import ArrowbricksError, AuthError, StatementError, TransientError
+
+try:
+    await cursor.execute(sql)
+except AuthError:
+    refresh_credentials()
+except StatementError:
+    raise  # bad SQL -- don't retry
+except TransientError:
+    await asyncio.sleep(5)
+    await cursor.execute(sql)  # safe to retry
+except ArrowbricksError:
+    raise
+```
+
+`QueryTimeout` (`total_timeout_s` exceeded -- see Cancellation above) also subclasses `ArrowbricksError`, so `except ArrowbricksError` is a genuine catch-all for every exception this package raises itself.
+
 ## API
 
 - `connect(host, warehouse_id, *, token=None, token_provider=None, ...) -> Connection`
 - `Connection.cursor() -> Cursor`
 - `Connection.client -> DatabricksClient` -- the same client `cursor()` uses, for lower-level access (e.g. `stream_query_json`, `upload_volume_file`).
-- `Cursor.execute(sql, parameters=None, *, row_limit=None, offset=None, catalog=None, schema=None, total_timeout_s=None, prefer_inline=False) -> Cursor` -- submits and waits for the statement, like a real DB-API cursor. `parameters`, if given, is Databricks' own named-parameter format -- `[{"name": ..., "value": ..., "type": ...}]` bound against `:name` markers in `sql`. `prefer_inline=True` tries fetching a small result (well under Databricks' 25 MiB inline cap) in the same round trip as the submission itself, skipping the chunk-fetch entirely -- if the result turns out too big, or has a column type this can't convert (nested ARRAY/MAP/STRUCT, VARIANT), it transparently re-runs the query the normal way, so a caller who sets this without actually expecting a small result pays for the query twice. Leave it off unless you know the result is small.
+- `Cursor.execute(sql, parameters=None, *, row_limit=None, offset=None, catalog=None, schema=None, total_timeout_s=None, prefer_inline=False) -> Cursor` -- submits and waits for the statement, like a real DB-API cursor. `parameters`, if given, is Databricks' own named-parameter format -- `[{"name": ..., "value": ..., "type": ...}]` bound against `:name` markers in `sql`. `prefer_inline=True` tries fetching a small result (well under Databricks' 25 MiB inline cap) in the same round trip as the submission itself, skipping the chunk-fetch entirely. If the result turns out too big, the statement fails server-side before ever really executing, and arrowbricks transparently re-runs it the normal way -- safe to double-submit, since nothing committed the first time; a caller who sets `prefer_inline` without actually expecting a small result just pays for the query twice in this case. If instead the result comes back but has a column type this can't convert (nested ARRAY/MAP/STRUCT, VARIANT), the statement already *succeeded* -- arrowbricks does **not** silently re-run it (that could duplicate a write for non-idempotent SQL); it raises `ArrowbricksError` naming the statement instead, so you know it ran and can decide yourself whether re-running is safe. Leave `prefer_inline` off unless you know the result is small, and unless you know the query is a `SELECT` (or otherwise idempotent) if you want the byte-limit fallback's automatic retry to be safe too.
 - `Cursor.execute_streamed(...)` -- same args, but an async generator yielding `HEARTBEAT` while waiting on a slow cold start, then the ready `Cursor` -- for bridging e.g. an SSE connection. Its timeout/heartbeats stop the moment the statement is ready, *before* any chunk has been downloaded -- see `fetchall_streamed` below for the download phase itself.
 - `Cursor.fetchone() -> tuple | None`, `Cursor.fetchmany(size) -> list[tuple]`, `Cursor.fetchall() -> list[tuple]`, and iterating a `Cursor` directly -- row tuples; needs the `arro3` extra.
 - `Cursor.fetchmany_arrow(size) -> Table`, `Cursor.fetchall_arrow() -> Table` -- an Arrow table (implements `__arrow_c_stream__`, so arro3/pyarrow/DuckDB can all consume it directly, zero-copy).
 - `Cursor.fetchall_streamed(*, total_timeout_s=None)` / `Cursor.fetchall_arrow_streamed(*, total_timeout_s=None)` -- like `fetchall()`/`fetchall_arrow()`, but yield `HEARTBEAT` while pulling chunks instead of blocking silently, then the final rows/Table -- for a caller downloading a large result over SSE who needs heartbeats (and a timeout) through the *download*, not just the initial wait. Compose with `execute_streamed` and a shared deadline if you want one combined budget across both phases (see `examples/fastapi_sse_pivot.py`).
 - `Cursor.description` -- DB-API-style `[(name, type_name, None, None, None, None, None), ...]` after `execute()`.
 - `client.stream_query_json(sql, **kwargs)` (or the equivalent free function `stream_query_json(client, sql, **kwargs)`) -- yields `HEARTBEAT`, then each row as a JSON string, as soon as its chunk arrives. Timestamps come out as full ISO-8601, every column key is always present (`"col":null` for a null value, never an omitted key). JSON has no literal for NaN/Infinity/-Infinity, so those come back as `"col":null` by default -- pass `non_finite_floats="string"` to get `"col":"NaN"`/`"col":"Infinity"`/`"col":"-Infinity"` instead if you need to tell them apart from a real NULL.
-- `DatabricksClient(host, warehouse_id, *, token=None, token_provider=None, protocol="thrift", ...)` -- the lower-level client `Connection` wraps. `client.upload_volume_file(volume_path, data)`/`client.delete_volume_file(volume_path)` for the Files API. Pass `protocol="sea"` to opt into the REST Statement Execution API backend instead of the default Thrift one (see above) -- `prefer_inline` (SEA-only) has no effect under `protocol="thrift"` (silent no-op, not an error), since Thrift's own inline-result mechanism already covers that case.
+- `DatabricksClient(host, warehouse_id, *, token=None, token_provider=None, protocol="thrift", on_event=None, retry_attempts=6, retry_max_wait_s=20.0, ...)` -- the lower-level client `Connection` wraps. `client.upload_volume_file(volume_path, data)`/`client.delete_volume_file(volume_path)` for the Files API. Pass `protocol="sea"` to opt into the REST Statement Execution API backend instead of the default Thrift one (see above) -- `prefer_inline` (SEA-only) has no effect under `protocol="thrift"` (silent no-op, not an error), since Thrift's own inline-result mechanism already covers that case. `on_event` -- see Observability below. `retry_attempts`/`retry_max_wait_s` tune the retry policy behind `TransientError`/`AuthError` (see Errors above) -- total attempts (the first try plus `retry_attempts - 1` retries) and the exponential-backoff ceiling in seconds, for every retryable request this client makes (statement submit/poll, chunk-index resolution, chunk download, volume file ops). `retry_attempts` must be at least 1 (raises `ValueError` otherwise).
 - `write_ipc_stream(table, buf)` -- writes any Arrow-C-Data-Interface-compatible object as an uncompressed Arrow-IPC stream (see below).
 - `ReplayableArrowChunk(data: bytes, chunk_index, declared_row_count=None)` -- wraps raw Arrow-IPC stream bytes (e.g. previously downloaded and stored) so they can be read more than once via `__arrow_c_stream__` (a schema peek, then the actual scan -- DuckDB's registration path does this), and `.to_table()` for a one-shot parse. No extra dependency needed.
 

{arrowbricks-3.0.4 → arrowbricks-3.1.1}/README.md

@@ -65,9 +65,14 @@ async for item in client.stream_query_json("SELECT * FROM my_catalog.my_schema.b
 
 See [`examples/basic.py`](examples/basic.py) for a runnable version,
 [`examples/cursor_paging.py`](examples/cursor_paging.py) for paging a large
-result with `fetchmany`/`fetchmany_arrow` without buffering it all upfront, or
+result with `fetchmany`/`fetchmany_arrow` without buffering it all upfront,
 [`examples/azure_auth.py`](examples/azure_auth.py) for a caching
-`token_provider` built on Azure AD (`DefaultAzureCredential`).
+`token_provider` built on Azure AD (`DefaultAzureCredential`, needs
+`azure-identity`), or [`examples/oauth_m2m_auth.py`](examples/oauth_m2m_auth.py)
+for the same idea using Databricks' own OAuth machine-to-machine
+client-credentials flow instead -- like `databricks-sql-connector`'s
+`auth_type="databricks-oauth"`, but stdlib-only (`urllib.request`), no
+extra dependency.
 
 ## FastAPI SSE example
 
@@ -123,19 +128,71 @@ arrowbricks has no opinion on *how* you get a token and no cloud-SDK dependency
 conn = connect(host=..., warehouse_id=..., token_provider=my_token_provider)
 ```
 
+## Observability
+
+Pass `on_event` to `connect`/`DatabricksClient` to get a `QueryStats` snapshot once per query, at completion (success, cancelled, timed out, or errored):
+
+```python
+def log_query(stats):
+    print(
+        f"{stats.statement_id}: {stats.outcome} in {stats.fetch_s:.2f}s, {stats.bytes_downloaded} bytes, {stats.retry_count} retries"
+    )
+
+
+conn = connect(host=..., warehouse_id=..., token=..., on_event=log_query)
+```
+
+`on_event` is sync or async, same as `token_provider`, and applies to every query run through that client (not passed per-call). It's strictly fire-and-forget: any exception it raises is caught and swallowed on the Rust side, never surfaced to your query, and dispatching it never blocks or measurably slows the actual fetch -- a slow or broken logging/metrics callback can't make your queries slower or fail them. `QueryStats` fields: `statement_id: str`, `protocol: "thrift" | "sea"`, `warehouse_wait_s: float` (0.0 if the client already knew the warehouse was running), `submit_to_ready_s: float` (statement submit -> ready to fetch), `fetch_s: float` (download + decode), `num_chunks: int`, `bytes_downloaded: int`, `retry_count: int`, `concurrency_used: int`, and `outcome: "success" | "cancelled" | "timeout" | "error"`.
+
+`on_event` fires once the fetch of a given query actually completes (or is abandoned) -- a `Cursor` whose result is never fully drained (e.g. only a partial `fetchmany()`, then abandoned) never fires one, same category as this package's existing documented gap around GC-based cleanup.
+
+## Cancellation
+
+When a chunk download times out (`total_timeout_s`) or your code cancels the surrounding coroutine (`task.cancel()`/`asyncio.wait_for`) while it's in flight, arrowbricks fires a best-effort server-side cancel in the background (Thrift's `CancelOperation`, or SEA's `POST .../cancel`) so Databricks stops running the query instead of finishing it for nobody. This is fire-and-forget: the `QueryTimeout`/cancellation still reaches you immediately, and the cancel call's own result (success or failure) is never surfaced or awaited.
+
+This covers `Cursor.fetchall_streamed()`/`fetchall_arrow_streamed()`, `client.stream_query_json(...)`, and the lower-level `._core.Client`'s own streamed APIs -- everything that wraps the chunk-download phase in the Rust-level heartbeat. Neither `total_timeout_s` nor a bare `task.cancel()` on `Cursor.execute()`/`execute_streamed()` itself (the initial submit/poll wait) triggers a cancel -- by the time a query's chunks are being fetched at all, the statement has typically already finished running server-side, so there's usually nothing left to cancel there in practice.
+
+## Errors
+
+Every exception arrowbricks raises itself is an `ArrowbricksError` -- a `RuntimeError` subclass, so an `except RuntimeError` written before this hierarchy existed keeps catching everything unchanged. Three subclasses split by what actually changes what your code should do next:
+
+- `TransientError` -- a network blip, connection reset, or 5xx that survived every internal retry (`retry_attempts`, exponential backoff up to `retry_max_wait_s` -- see below) before ever reaching you. Back off further and try again later; retrying immediately just repeats what already failed.
+- `AuthError` -- HTTP 401/403, even after every internal retry re-fetched a token from your `token_provider`, *or* your `token_provider` itself raised (e.g. its own OAuth refresh call came back unauthorized). Treat the credential itself as bad/expired, not a transient blip.
+- `StatementError` -- the SQL statement itself failed or was canceled server-side (bad SQL, a permissions error, a warehouse-side query failure). Retrying the identical statement will fail identically; this isn't a transport problem.
+
+Anything else (a generic non-auth/non-statement HTTP error, an internal parse/decode failure) raises the plain `ArrowbricksError` base.
+
+```python
+from arrowbricks import ArrowbricksError, AuthError, StatementError, TransientError
+
+try:
+    await cursor.execute(sql)
+except AuthError:
+    refresh_credentials()
+except StatementError:
+    raise  # bad SQL -- don't retry
+except TransientError:
+    await asyncio.sleep(5)
+    await cursor.execute(sql)  # safe to retry
+except ArrowbricksError:
+    raise
+```
+
+`QueryTimeout` (`total_timeout_s` exceeded -- see Cancellation above) also subclasses `ArrowbricksError`, so `except ArrowbricksError` is a genuine catch-all for every exception this package raises itself.
+
 ## API
 
 - `connect(host, warehouse_id, *, token=None, token_provider=None, ...) -> Connection`
 - `Connection.cursor() -> Cursor`
 - `Connection.client -> DatabricksClient` -- the same client `cursor()` uses, for lower-level access (e.g. `stream_query_json`, `upload_volume_file`).
-- `Cursor.execute(sql, parameters=None, *, row_limit=None, offset=None, catalog=None, schema=None, total_timeout_s=None, prefer_inline=False) -> Cursor` -- submits and waits for the statement, like a real DB-API cursor. `parameters`, if given, is Databricks' own named-parameter format -- `[{"name": ..., "value": ..., "type": ...}]` bound against `:name` markers in `sql`. `prefer_inline=True` tries fetching a small result (well under Databricks' 25 MiB inline cap) in the same round trip as the submission itself, skipping the chunk-fetch entirely -- if the result turns out too big, or has a column type this can't convert (nested ARRAY/MAP/STRUCT, VARIANT), it transparently re-runs the query the normal way, so a caller who sets this without actually expecting a small result pays for the query twice. Leave it off unless you know the result is small.
+- `Cursor.execute(sql, parameters=None, *, row_limit=None, offset=None, catalog=None, schema=None, total_timeout_s=None, prefer_inline=False) -> Cursor` -- submits and waits for the statement, like a real DB-API cursor. `parameters`, if given, is Databricks' own named-parameter format -- `[{"name": ..., "value": ..., "type": ...}]` bound against `:name` markers in `sql`. `prefer_inline=True` tries fetching a small result (well under Databricks' 25 MiB inline cap) in the same round trip as the submission itself, skipping the chunk-fetch entirely. If the result turns out too big, the statement fails server-side before ever really executing, and arrowbricks transparently re-runs it the normal way -- safe to double-submit, since nothing committed the first time; a caller who sets `prefer_inline` without actually expecting a small result just pays for the query twice in this case. If instead the result comes back but has a column type this can't convert (nested ARRAY/MAP/STRUCT, VARIANT), the statement already *succeeded* -- arrowbricks does **not** silently re-run it (that could duplicate a write for non-idempotent SQL); it raises `ArrowbricksError` naming the statement instead, so you know it ran and can decide yourself whether re-running is safe. Leave `prefer_inline` off unless you know the result is small, and unless you know the query is a `SELECT` (or otherwise idempotent) if you want the byte-limit fallback's automatic retry to be safe too.
 - `Cursor.execute_streamed(...)` -- same args, but an async generator yielding `HEARTBEAT` while waiting on a slow cold start, then the ready `Cursor` -- for bridging e.g. an SSE connection. Its timeout/heartbeats stop the moment the statement is ready, *before* any chunk has been downloaded -- see `fetchall_streamed` below for the download phase itself.
 - `Cursor.fetchone() -> tuple | None`, `Cursor.fetchmany(size) -> list[tuple]`, `Cursor.fetchall() -> list[tuple]`, and iterating a `Cursor` directly -- row tuples; needs the `arro3` extra.
 - `Cursor.fetchmany_arrow(size) -> Table`, `Cursor.fetchall_arrow() -> Table` -- an Arrow table (implements `__arrow_c_stream__`, so arro3/pyarrow/DuckDB can all consume it directly, zero-copy).
 - `Cursor.fetchall_streamed(*, total_timeout_s=None)` / `Cursor.fetchall_arrow_streamed(*, total_timeout_s=None)` -- like `fetchall()`/`fetchall_arrow()`, but yield `HEARTBEAT` while pulling chunks instead of blocking silently, then the final rows/Table -- for a caller downloading a large result over SSE who needs heartbeats (and a timeout) through the *download*, not just the initial wait. Compose with `execute_streamed` and a shared deadline if you want one combined budget across both phases (see `examples/fastapi_sse_pivot.py`).
 - `Cursor.description` -- DB-API-style `[(name, type_name, None, None, None, None, None), ...]` after `execute()`.
 - `client.stream_query_json(sql, **kwargs)` (or the equivalent free function `stream_query_json(client, sql, **kwargs)`) -- yields `HEARTBEAT`, then each row as a JSON string, as soon as its chunk arrives. Timestamps come out as full ISO-8601, every column key is always present (`"col":null` for a null value, never an omitted key). JSON has no literal for NaN/Infinity/-Infinity, so those come back as `"col":null` by default -- pass `non_finite_floats="string"` to get `"col":"NaN"`/`"col":"Infinity"`/`"col":"-Infinity"` instead if you need to tell them apart from a real NULL.
-- `DatabricksClient(host, warehouse_id, *, token=None, token_provider=None, protocol="thrift", ...)` -- the lower-level client `Connection` wraps. `client.upload_volume_file(volume_path, data)`/`client.delete_volume_file(volume_path)` for the Files API. Pass `protocol="sea"` to opt into the REST Statement Execution API backend instead of the default Thrift one (see above) -- `prefer_inline` (SEA-only) has no effect under `protocol="thrift"` (silent no-op, not an error), since Thrift's own inline-result mechanism already covers that case.
+- `DatabricksClient(host, warehouse_id, *, token=None, token_provider=None, protocol="thrift", on_event=None, retry_attempts=6, retry_max_wait_s=20.0, ...)` -- the lower-level client `Connection` wraps. `client.upload_volume_file(volume_path, data)`/`client.delete_volume_file(volume_path)` for the Files API. Pass `protocol="sea"` to opt into the REST Statement Execution API backend instead of the default Thrift one (see above) -- `prefer_inline` (SEA-only) has no effect under `protocol="thrift"` (silent no-op, not an error), since Thrift's own inline-result mechanism already covers that case. `on_event` -- see Observability below. `retry_attempts`/`retry_max_wait_s` tune the retry policy behind `TransientError`/`AuthError` (see Errors above) -- total attempts (the first try plus `retry_attempts - 1` retries) and the exponential-backoff ceiling in seconds, for every retryable request this client makes (statement submit/poll, chunk-index resolution, chunk download, volume file ops). `retry_attempts` must be at least 1 (raises `ValueError` otherwise).
 - `write_ipc_stream(table, buf)` -- writes any Arrow-C-Data-Interface-compatible object as an uncompressed Arrow-IPC stream (see below).
 - `ReplayableArrowChunk(data: bytes, chunk_index, declared_row_count=None)` -- wraps raw Arrow-IPC stream bytes (e.g. previously downloaded and stored) so they can be read more than once via `__arrow_c_stream__` (a schema peek, then the actual scan -- DuckDB's registration path does this), and `.to_table()` for a one-shot parse. No extra dependency needed.
 

{arrowbricks-3.0.4 → arrowbricks-3.1.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "arrowbricks"
-version = "3.0.4"
+version = "3.1.1"
 description = "Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a DB-API-ish Cursor (fetchone/fetchmany/fetchall/fetchall_arrow) or NDJSON streaming. Rust/PyO3 core throughout -- zero required runtime dependencies."
 readme = "README.md"
 license = "MIT"

{arrowbricks-3.0.4 → arrowbricks-3.1.1}/rust/arrowbricks_core/Cargo.lock

@@ -241,14 +241,16 @@ dependencies = [
 
 [[package]]
 name = "arrowbricks_core"
-version = "3.0.4"
+version = "3.1.1"
 dependencies = [
  "arrow",
  "arrow-json",
+ "base64 0.23.1",
  "bytes",
  "chrono",
  "hyper-rustls",
  "lz4_flex",
+ "proptest",
  "pyo3",
  "pyo3-arrow",
  "pyo3-async-runtimes",
@@ -304,6 +306,21 @@ version = "0.23.1"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "ac07cdecf99051d9a5238b80f35af32cdeba5b336e55d957b318b50137e18da5"
 
+[[package]]
+name = "bit-set"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "08807e080ed7f9d5433fa9b275196cfc35414f66a0c79d864dc51a0d825231a3"
+dependencies = [
+ "bit-vec",
+]
+
+[[package]]
+name = "bit-vec"
+version = "0.8.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "5e764a1d40d510daf35e07be9eb06e75770908c27d411ee6c92109c9840eaaf7"
+
 [[package]]
 name = "bitflags"
 version = "2.13.1"
@@ -458,6 +475,22 @@ version = "1.0.2"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
 
+[[package]]
+name = "errno"
+version = "0.3.14"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "39cab71617ae0d63f51a36d69f866391735b51691dbda63cf6f96d042b63efeb"
+dependencies = [
+ "libc",
+ "windows-sys 0.61.2",
+]
+
+[[package]]
+name = "fastrand"
+version = "2.5.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "da7c62ceae207dd37ea5b845da6a0696c799f85e97da1ab5b7910be3c1c80223"
+
 [[package]]
 name = "find-msvc-tools"
 version = "0.1.10"
@@ -1038,6 +1071,12 @@ version = "0.2.16"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
 
+[[package]]
+name = "linux-raw-sys"
+version = "0.12.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
+
 [[package]]
 name = "litemap"
 version = "0.8.2"
@@ -1232,6 +1271,15 @@ dependencies = [
  "zerovec",
 ]
 
+[[package]]
+name = "ppv-lite86"
+version = "0.2.21"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "85eae3c4ed2f50dcfe72643da4befc30deadb458a9b590d720cde2f2b1e97da9"
+dependencies = [
+ "zerocopy",
+]
+
 [[package]]
 name = "proc-macro2"
 version = "1.0.107"
@@ -1241,6 +1289,25 @@ dependencies = [
  "unicode-ident",
 ]
 
+[[package]]
+name = "proptest"
+version = "1.11.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "4b45fcc2344c680f5025fe57779faef368840d0bd1f42f216291f0dc4ace4744"
+dependencies = [
+ "bit-set",
+ "bit-vec",
+ "bitflags",
+ "num-traits",
+ "rand",
+ "rand_chacha",
+ "rand_xorshift",
+ "regex-syntax",
+ "rusty-fork",
+ "tempfile",
+ "unarray",
+]
+
 [[package]]
 name = "pyo3"
 version = "0.29.2"
@@ -1346,6 +1413,12 @@ dependencies = [
  "serde",
 ]
 
+[[package]]
+name = "quick-error"
+version = "1.2.3"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "a1d01941d82fa2ab50be1e79e6714289dd7cde78eba4c074bc5a4374f650dfe0"
+
 [[package]]
 name = "quote"
 version = "1.0.47"
@@ -1361,6 +1434,44 @@ version = "5.3.0"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "69cdb34c158ceb288df11e18b4bd39de994f6657d83847bdffdbd7f346754b0f"
 
+[[package]]
+name = "rand"
+version = "0.9.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b9ef1d0d795eb7d84685bca4f72f3649f064e6641543d3a8c415898726a57b41"
+dependencies = [
+ "rand_chacha",
+ "rand_core",
+]
+
+[[package]]
+name = "rand_chacha"
+version = "0.9.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "d3022b5f1df60f26e1ffddd6c66e8aa15de382ae63b3a0c1bfc0e4d3e3f325cb"
+dependencies = [
+ "ppv-lite86",
+ "rand_core",
+]
+
+[[package]]
+name = "rand_core"
+version = "0.9.5"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "76afc826de14238e6e8c374ddcc1fa19e374fd8dd986b0d2af0d02377261d83c"
+dependencies = [
+ "getrandom 0.3.4",
+]
+
+[[package]]
+name = "rand_xorshift"
+version = "0.4.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "513962919efc330f829edb2535844d1b912b0fbe2ca165d613e4e8788bb05a5a"
+dependencies = [
+ "rand_core",
+]
+
 [[package]]
 name = "rawpointer"
 version = "0.2.1"
@@ -1462,6 +1573,19 @@ dependencies = [
  "semver",
 ]
 
+[[package]]
+name = "rustix"
+version = "1.1.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
+dependencies = [
+ "bitflags",
+ "errno",
+ "libc",
+ "linux-raw-sys",
+ "windows-sys 0.61.2",
+]
+
 [[package]]
 name = "rustls"
 version = "0.23.43"
@@ -1542,6 +1666,18 @@ version = "1.0.23"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "cf54715a573b99ac80df0bc206da022bcd442c974952c7b9720069370852e21f"
 
+[[package]]
+name = "rusty-fork"
+version = "0.3.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "cc6bf79ff24e648f6da1f8d1f011e9cac26491b619e6b9280f2b47f1774e6ee2"
+dependencies = [
+ "fnv",
+ "quick-error",
+ "tempfile",
+ "wait-timeout",
+]
+
 [[package]]
 name = "ryu"
 version = "1.0.23"
@@ -1760,6 +1896,19 @@ version = "0.13.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "adb6935a6f5c20170eeceb1a3835a49e12e19d792f6dd344ccc76a985ca5a6ca"
 
+[[package]]
+name = "tempfile"
+version = "3.27.0"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
+dependencies = [
+ "fastrand",
+ "getrandom 0.3.4",
+ "once_cell",
+ "rustix",
+ "windows-sys 0.61.2",
+]
+
 [[package]]
 name = "thiserror"
 version = "1.0.69"
@@ -1945,6 +2094,12 @@ version = "2.1.3"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "8464ec13c3691491391d9fce00f6416c9a48e46972f72d7865688be2080192c9"
 
+[[package]]
+name = "unarray"
+version = "0.1.4"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "eaea85b334db583fe3274d12b4cd1880032beab409c0d774be044d4480ab9a94"
+
 [[package]]
 name = "unicode-ident"
 version = "1.0.24"
@@ -1993,6 +2148,15 @@ version = "0.9.5"
 source = "registry+https://github.com/rust-lang/crates.io-index"
 checksum = "0b928f33d975fc6ad9f86c8f283853ad26bdd5b10b7f1542aa2fa15e2289105a"
 
+[[package]]
+name = "wait-timeout"
+version = "0.2.1"
+source = "registry+https://github.com/rust-lang/crates.io-index"
+checksum = "09ac3b126d3914f9849036f826e054cbabdc8519970b8998ddaf3b5bd3c65f11"
+dependencies = [
+ "libc",
+]
+
 [[package]]
 name = "walkdir"
 version = "2.5.0"

{arrowbricks-3.0.4 → arrowbricks-3.1.1}/rust/arrowbricks_core/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "arrowbricks_core"
-version = "3.0.4"
+version = "3.1.1"
 edition = "2024"
 readme = "README.md"
 
@@ -15,6 +15,12 @@ crate-type = ["cdylib", "rlib"]
 arrow = { version = "59.1.0", default-features = false, features = ["ipc"] }
 arrow-json = "59.1.0"
 bytes = "1.12.1"
+# Already an unconditional transitive dependency of arrow-cast (pulled in by
+# pyo3-arrow) at this exact version -- declared directly here too so
+# json_convert.rs can decode BINARY columns without hand-rolling base64. No
+# new code ships that wasn't already linked in (feature unification, same
+# reasoning as chrono/rustls below).
+base64 = "0.23"
 # Already an unconditional transitive dependency of pyo3-arrow (see its own
 # comment below) -- declared directly here too so json_convert.rs can parse
 # DATE/TIMESTAMP/TIMESTAMP_NTZ strings without hand-rolling calendar
@@ -75,6 +81,20 @@ extension-module = ["pyo3/extension-module"]
 [dev-dependencies]
 tokio = { version = "1.53.1", features = ["rt-multi-thread", "macros", "test-util", "net", "io-util"] }
 wiremock = "0.6.5"
+# Property-based fuzzing for the hand-rolled wire-protocol parsers
+# (thrift.rs's Reader, json_convert.rs's STRUCT type_text tokenizer,
+# client.rs's decompress_lz4_frame) -- dev-only, runs inside plain
+# `cargo test` via the `proptest!` macro (`#[test]` under the hood, no
+# separate fuzz-target/corpus infra the way cargo-fuzz/afl need, and no
+# nightly toolchain requirement either). Never touches the shipped wheel's
+# dependency graph -- `[dev-dependencies]` are only ever linked into `cargo
+# test`/`cargo bench` binaries, never into the `cdylib` maturin actually
+# builds (see this crate's own `extension-module` feature/profile split
+# above for how that build works). Default config (256 cases/property) is
+# left alone; per-property `ProptestConfig` below trims a couple of the
+# slower ones (deep nesting, wide byte vectors) so `cargo test` doesn't
+# become noticeably slower overall.
+proptest = "1"
 
 # Currently clean on all three by luck, not enforcement -- pinning them here
 # so a future PR that introduces one fails CI instead of silently landing.

{arrowbricks-3.0.4 → arrowbricks-3.1.1}/rust/arrowbricks_core/README.md

@@ -66,7 +66,7 @@ unlike the table object itself.
 
 ## API
 
-- `Client(host, warehouse_id, *, token=None, token_provider=None, chunk_fetch_concurrency=64, http_timeout=60.0, wait_timeout="30s", warehouse_start_timeout=300.0, warehouse_confirmed_running_ttl_s=30.0, compress_results=True, protocol="thrift")` -- exactly one of `token`/`token_provider`. `token_provider` is a callable (sync or async) returning a token string, called fresh on every request, no caching. `compress_results` requests LZ4-compressed cloud-fetch chunks (see above); set `False` to opt out. `protocol="thrift"` (the default, as of this crate's own real-workspace benchmarking -- see `AGENTS.md`'s design-invariant entry) speaks the same HiveServer2-compatible Thrift-over-HTTPS protocol `databricks-sql-connector` uses by default (`thrift.rs`, a hand-rolled `TBinaryProtocol` reader/writer -- no new Cargo dependency) -- measurably faster for small queries (its `ExecuteStatement` RPC can return a small result inline via `getDirectResults`, in the same call that submits the statement) at the cost of `prefer_inline` becoming a silent no-op (Thrift has no INLINE-disposition equivalent, and doesn't need one); never slower than SEA on any query shape tested. `protocol="sea"` instead talks to the REST Statement Execution API -- still fully supported, opt in explicitly if you have a reason to prefer it. On par with SEA for a large, multi-chunk result too -- `run_thrift_fetch_loop` (`pipeline.rs`) fans its chunk downloads out across a `chunk_fetch_concurrency`-sized worker pool spanning the *whole* result (pipelined with the sequential `FetchResults` discovery calls, not serialized behind them), the same concurrency shape as SEA's own `fetch_chunks_with_backpressure` (see `AGENTS.md`'s own entry on this).
+- `Client(host, warehouse_id, *, token=None, token_provider=None, chunk_fetch_concurrency=64, http_timeout=60.0, wait_timeout="30s", warehouse_start_timeout=300.0, warehouse_confirmed_running_ttl_s=30.0, compress_results=True, protocol="thrift", retry_attempts=6, retry_max_wait_s=20.0)` -- exactly one of `token`/`token_provider`. `retry_attempts`/`retry_max_wait_s` tune the retry policy (total attempts, exponential-backoff ceiling in seconds) behind every retryable request this client makes; `retry_attempts` must be at least 1. `token_provider` is a callable (sync or async) returning a token string, called fresh on every request, no caching. `compress_results` requests LZ4-compressed cloud-fetch chunks (see above); set `False` to opt out. `protocol="thrift"` (the default, as of this crate's own real-workspace benchmarking -- see `AGENTS.md`'s design-invariant entry) speaks the same HiveServer2-compatible Thrift-over-HTTPS protocol `databricks-sql-connector` uses by default (`thrift.rs`, a hand-rolled `TBinaryProtocol` reader/writer -- no new Cargo dependency) -- measurably faster for small queries (its `ExecuteStatement` RPC can return a small result inline via `getDirectResults`, in the same call that submits the statement) at the cost of `prefer_inline` becoming a silent no-op (Thrift has no INLINE-disposition equivalent, and doesn't need one); never slower than SEA on any query shape tested. `protocol="sea"` instead talks to the REST Statement Execution API -- still fully supported, opt in explicitly if you have a reason to prefer it. On par with SEA for a large, multi-chunk result too -- `run_thrift_fetch_loop` (`pipeline.rs`) fans its chunk downloads out across a `chunk_fetch_concurrency`-sized worker pool spanning the *whole* result (pipelined with the sequential `FetchResults` discovery calls, not serialized behind them), the same concurrency shape as SEA's own `fetch_chunks_with_backpressure` (see `AGENTS.md`'s own entry on this).
 - `Client.execute(statement, *, catalog=None, schema=None, parameters=None, prefer_inline=False) -> ResultSet` -- submits and starts background chunk fetching without pulling anything yet. `parameters` is Databricks' own named-parameter format (`[{"name":..., "value":..., "type":...}]`), passed straight through. `prefer_inline=True` submits with `disposition=INLINE, format=JSON_ARRAY` instead, for a caller who expects a small (well under Databricks' 25 MiB inline cap) result and wants to skip the chunk-fetch round trip -- on a result too big for INLINE, or containing a column type `json_convert.rs` doesn't map (STRUCT/ARRAY-of-STRUCT/MAP/VARIANT), it transparently falls back to a second, normal `execute()` and the query runs twice. See `AGENTS.md`'s "Design invariants" section (in the root package) for the full reasoning and real-workspace verification behind this.
   - `ResultSet.fetchmany_arrow(n) -> Table` -- pulls/decodes only as many chunks as needed for `n` rows, buffering the rest; may return fewer than `n` once exhausted.
   - `ResultSet.fetchall_arrow() -> Table` -- drains everything remaining.