arrowbricks 3.1.2__tar.gz → 3.1.4__tar.gz

{arrowbricks-3.1.2 → arrowbricks-3.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: arrowbricks
-Version: 3.1.2
+Version: 3.1.4
 Requires-Dist: arro3-core>=0.8 ; extra == 'arro3'
 Provides-Extra: arro3
 License-File: LICENSE
@@ -271,12 +271,17 @@ The [official driver](https://github.com/databricks/databricks-sql-python) is th
 
 Measured against a real Databricks SQL warehouse (Azure Databricks, `2X-Small` **Pro** serverless warehouse, Photon on, 1-4 auto-scaling clusters -- the smallest/cheapest warehouse tier, deliberately: a bigger warehouse would narrow the gap by making the query itself slower and the client-side overhead this compares proportionally smaller). Query: `SELECT id, id * 2 AS doubled, CAST(id AS STRING) AS label FROM range(200000)` (200k rows, 3 columns), 3 timed runs after 1 discarded warm-up run, one connection reused per library:
 
-| | avg | stdev | range | peak RSS during the query |
-|---|---|---|---|---|
-| `databricks-sql-connector` | 0.88s | 0.06s | 0.83s - 0.95s | 16 MB |
-| arrowbricks | 0.50s | 0.02s | 0.48s - 0.52s | 16 MB |
+| | avg | stdev | range |
+|---|---|---|---|
+| `databricks-sql-connector` | 0.88s | 0.06s | 0.83s - 0.95s |
+| arrowbricks | 0.50s | 0.02s | 0.48s - 0.52s |
 
-**arrowbricks: ~1.8x faster**, same order-of-magnitude peak memory *for this query size* -- at 200k rows the Python interpreter's own baseline footprint dominates over the actual result data for both libraries, so this particular number doesn't show a difference. The real memory/footprint difference is in what gets installed, not what a single small query allocates:
+These historical latency measurements compare the connector's `fetchall()`
+(Python rows) with arrowbricks' `fetchall_arrow()` (Arrow), which do different
+amounts of materialization. The previously published 16 MB memory figures
+were invalid: the script sampled peak RSS before importing the libraries or
+running queries. It now measures after the workload; rerun it for actual
+peak memory on your machine. The installed-footprint measurements were:
 
 | | installed size (package + all required deps) |
 |---|---|
@@ -292,6 +297,35 @@ DATABRICKS_TOKEN=dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
 python examples/benchmark_vs_connector.py
 ```
 
+For comparing arrowbricks versions with identical APIs, use
+[`examples/benchmark_versions.py`](examples/benchmark_versions.py). It keeps
+one connection per version, discards warm-ups, alternates execution order,
+passes concurrency explicitly, and checks row counts. Add `--verify-ipc`
+(requires `arro3-core`) to compare serialized Arrow results in memory after
+timing. Results must have stable values, schema metadata, row order, and batch boundaries;
+verification contributes to peak process memory, and checksums are not printed. See
+[`benchmarks/2026-09-06.md`](benchmarks/2026-09-06.md) for replay/cache measurements
+and [`benchmarks/2026-09-06-downloads.md`](benchmarks/2026-09-06-downloads.md)
+for the subsequent cloud-fetch scheduling measurements and limits.
+
+Follow-up experiments cover [spare request slots](benchmarks/2026-09-06-spare-slots.md),
+[concurrent replay and NDJSON encoding](benchmarks/2026-09-06-lowlevel.md),
+and the rejected [LZ4 capacity](benchmarks/2026-09-06-lz4-capacity.md) and
+[direct-fill buffer](benchmarks/2026-09-06-direct-fill.md) changes.
+
+Cached IPC replay can be measured without a warehouse:
+
+```bash
+uv run python examples/benchmark_replay.py --rows 200000 --columns 16 --repeats 8
+```
+
+Replays now share immutable input bytes across decoded tables, reducing
+repeated copies and memory use. Arrow may still copy misaligned fixed-width
+buffers or decompress IPC-compressed bodies. Keeping a small slice of a
+decoded array can retain the full source allocation until that slice is released.
+Decoding releases the Python interpreter lock, so independent replay calls
+can run concurrently on separate Python threads.
+
 `BENCHMARK_SQL` overrides the query, `BENCHMARK_RUNS` (default 3) controls how many timed runs to average.
 
 ## A note on Arrow IPC compression

{arrowbricks-3.1.2 → arrowbricks-3.1.4}/README.md

@@ -259,12 +259,17 @@ The [official driver](https://github.com/databricks/databricks-sql-python) is th
 
 Measured against a real Databricks SQL warehouse (Azure Databricks, `2X-Small` **Pro** serverless warehouse, Photon on, 1-4 auto-scaling clusters -- the smallest/cheapest warehouse tier, deliberately: a bigger warehouse would narrow the gap by making the query itself slower and the client-side overhead this compares proportionally smaller). Query: `SELECT id, id * 2 AS doubled, CAST(id AS STRING) AS label FROM range(200000)` (200k rows, 3 columns), 3 timed runs after 1 discarded warm-up run, one connection reused per library:
 
-| | avg | stdev | range | peak RSS during the query |
-|---|---|---|---|---|
-| `databricks-sql-connector` | 0.88s | 0.06s | 0.83s - 0.95s | 16 MB |
-| arrowbricks | 0.50s | 0.02s | 0.48s - 0.52s | 16 MB |
-
-**arrowbricks: ~1.8x faster**, same order-of-magnitude peak memory *for this query size* -- at 200k rows the Python interpreter's own baseline footprint dominates over the actual result data for both libraries, so this particular number doesn't show a difference. The real memory/footprint difference is in what gets installed, not what a single small query allocates:
+| | avg | stdev | range |
+|---|---|---|---|
+| `databricks-sql-connector` | 0.88s | 0.06s | 0.83s - 0.95s |
+| arrowbricks | 0.50s | 0.02s | 0.48s - 0.52s |
+
+These historical latency measurements compare the connector's `fetchall()`
+(Python rows) with arrowbricks' `fetchall_arrow()` (Arrow), which do different
+amounts of materialization. The previously published 16 MB memory figures
+were invalid: the script sampled peak RSS before importing the libraries or
+running queries. It now measures after the workload; rerun it for actual
+peak memory on your machine. The installed-footprint measurements were:
 
 | | installed size (package + all required deps) |
 |---|---|
@@ -280,6 +285,35 @@ DATABRICKS_TOKEN=dapiXXXXXXXXXXXXXXXXXXXXXXXXXXXX \
 python examples/benchmark_vs_connector.py
 ```
 
+For comparing arrowbricks versions with identical APIs, use
+[`examples/benchmark_versions.py`](examples/benchmark_versions.py). It keeps
+one connection per version, discards warm-ups, alternates execution order,
+passes concurrency explicitly, and checks row counts. Add `--verify-ipc`
+(requires `arro3-core`) to compare serialized Arrow results in memory after
+timing. Results must have stable values, schema metadata, row order, and batch boundaries;
+verification contributes to peak process memory, and checksums are not printed. See
+[`benchmarks/2026-09-06.md`](benchmarks/2026-09-06.md) for replay/cache measurements
+and [`benchmarks/2026-09-06-downloads.md`](benchmarks/2026-09-06-downloads.md)
+for the subsequent cloud-fetch scheduling measurements and limits.
+
+Follow-up experiments cover [spare request slots](benchmarks/2026-09-06-spare-slots.md),
+[concurrent replay and NDJSON encoding](benchmarks/2026-09-06-lowlevel.md),
+and the rejected [LZ4 capacity](benchmarks/2026-09-06-lz4-capacity.md) and
+[direct-fill buffer](benchmarks/2026-09-06-direct-fill.md) changes.
+
+Cached IPC replay can be measured without a warehouse:
+
+```bash
+uv run python examples/benchmark_replay.py --rows 200000 --columns 16 --repeats 8
+```
+
+Replays now share immutable input bytes across decoded tables, reducing
+repeated copies and memory use. Arrow may still copy misaligned fixed-width
+buffers or decompress IPC-compressed bodies. Keeping a small slice of a
+decoded array can retain the full source allocation until that slice is released.
+Decoding releases the Python interpreter lock, so independent replay calls
+can run concurrently on separate Python threads.
+
 `BENCHMARK_SQL` overrides the query, `BENCHMARK_RUNS` (default 3) controls how many timed runs to average.
 
 ## A note on Arrow IPC compression

{arrowbricks-3.1.2 → arrowbricks-3.1.4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "arrowbricks"
-version = "3.1.2"
+version = "3.1.4"
 description = "Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a DB-API-ish Cursor (fetchone/fetchmany/fetchall/fetchall_arrow) or NDJSON streaming. Rust/PyO3 core throughout -- zero required runtime dependencies."
 readme = "README.md"
 license = "MIT"

{arrowbricks-3.1.2 → arrowbricks-3.1.4}/rust/arrowbricks_core/Cargo.lock

@@ -34,39 +34,6 @@ dependencies = [
  "libc",
 ]
 
-[[package]]
-name = "arrow"
-version = "59.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "61d285d16bce7d0be61912f7928342b673067b6b7d7ef6cc179258ba7de1fecf"
-dependencies = [
- "arrow-arith",
- "arrow-array",
- "arrow-buffer",
- "arrow-cast",
- "arrow-data",
- "arrow-ipc",
- "arrow-ord",
- "arrow-row",
- "arrow-schema",
- "arrow-select",
- "arrow-string",
-]
-
-[[package]]
-name = "arrow-arith"
-version = "59.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "757ef1836251e88222542a7da2623bc1c9cb9e20afefa6db2c41e79991cd91d4"
-dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
- "chrono",
- "num-traits",
-]
-
 [[package]]
 name = "arrow-array"
 version = "59.2.0"
@@ -186,19 +153,6 @@ dependencies = [
  "arrow-select",
 ]
 
-[[package]]
-name = "arrow-row"
-version = "59.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "bbec439386df71ad570e6758a946111322b9e9dc8db83b5527321f0b4c9119c2"
-dependencies = [
- "arrow-array",
- "arrow-buffer",
- "arrow-data",
- "arrow-schema",
- "half",
-]
-
 [[package]]
 name = "arrow-schema"
 version = "59.2.0"
@@ -223,33 +177,20 @@ dependencies = [
 ]
 
 [[package]]
-name = "arrow-string"
-version = "59.2.0"
-source = "registry+https://github.com/rust-lang/crates.io-index"
-checksum = "c838a25bb3691e919e0f617616ac51a4ff8517a952e29ca133cf0c22b2ce65b1"
+name = "arrowbricks_core"
+version = "3.1.4"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
- "arrow-data",
- "arrow-schema",
- "arrow-select",
- "memchr",
- "num-traits",
- "regex",
- "regex-syntax",
-]
-
-[[package]]
-name = "arrowbricks_core"
-version = "3.1.2"
-dependencies = [
- "arrow",
+ "arrow-ipc",
  "arrow-json",
+ "arrow-schema",
  "base64 0.23.1",
  "bytes",
  "chrono",
  "hyper-rustls",
  "lz4_flex",
+ "memchr",
  "proptest",
  "pyo3",
  "pyo3-arrow",

{arrowbricks-3.1.2 → arrowbricks-3.1.4}/rust/arrowbricks_core/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "arrowbricks_core"
-version = "3.1.2"
+version = "3.1.4"
 edition = "2024"
 readme = "README.md"
 
@@ -9,12 +9,16 @@ name = "arrowbricks_core"
 crate-type = ["cdylib", "rlib"]
 
 [dependencies]
-# default-features off: arrow's defaults are csv+ipc+json, and we only ever
-# decode Arrow-IPC chunks -- csv/json parsing (plus their transitive deps)
-# would just be dead weight in the shipped .so.
-arrow = { version = "59.1.0", default-features = false, features = ["ipc"] }
+# Depend on the components we use. The umbrella `arrow` crate also builds
+# arithmetic, string, and row kernels even with default-features disabled.
+arrow-array = "59.1.0"
+arrow-buffer = "59.1.0"
+arrow-ipc = "59.1.0"
+arrow-schema = "59.1.0"
 arrow-json = "59.1.0"
 bytes = "1.12.1"
+# Already in the dependency graph; vectorized newline search in NDJSON output.
+memchr = "2"
 # Already an unconditional transitive dependency of arrow-cast (pulled in by
 # pyo3-arrow) at this exact version -- declared directly here too so
 # json_convert.rs can decode BINARY columns without hand-rolling base64. No
@@ -103,6 +107,11 @@ redundant_clone = "warn"
 large_enum_variant = "warn"
 needless_collect = "warn"
 
+[profile.dev]
+# Keep source locations in backtraces without caching full debug type data
+# for Arrow and the HTTP stack. Incremental compilation remains enabled.
+debug = "line-tables-only"
+
 [profile.release]
 # opt-level="s" over the implicit default (3) or "z" -- a size-vs-speed
 # tradeoff previously rejected here on the reasoning that "3" was what the
@@ -126,3 +135,9 @@ opt-level = "s"
 strip = true
 lto = true
 codegen-units = 1
+
+[profile.release.package.pyo3-arrow]
+# This bridge is dominated by interoperability helpers. Optimizing it for
+# size saves ~32 KiB on macOS arm64; the IPC decoder keeps opt-level="s".
+# Replay and live-query benchmarks are recorded in benchmarks/2026-09-06.md.
+opt-level = "z"

{arrowbricks-3.1.2 → arrowbricks-3.1.4}/rust/arrowbricks_core/README.md

@@ -32,6 +32,12 @@ This builds a mixed Python/Rust project: the compiled extension lands at
 `mypy`/`pyright`/`ty` type-checking -- a compiled PyO3 extension has no type
 info of its own without them.
 
+Development/test builds retain source line information for backtraces but
+omit full debug type data to reduce the Cargo cache. For full debugger
+variable inspection, run with `CARGO_PROFILE_DEV_DEBUG=2`. Incremental
+compilation stays enabled. Release builds optimize the Python/Arrow bridge
+for size while keeping the IPC decoder at its existing optimization level.
+
 ## Quickstart
 
 Importable directly (bypassing the `arrowbricks` Python wrapper) as
@@ -67,7 +73,7 @@ unlike the table object itself.
 ## API
 
 - `Client(host, warehouse_id, *, token=None, token_provider=None, chunk_fetch_concurrency=64, http_timeout=60.0, wait_timeout="30s", warehouse_start_timeout=300.0, warehouse_confirmed_running_ttl_s=30.0, compress_results=True, protocol="thrift", retry_attempts=6, retry_max_wait_s=20.0)` -- exactly one of `token`/`token_provider`. `retry_attempts`/`retry_max_wait_s` tune the retry policy (total attempts, exponential-backoff ceiling in seconds) behind every retryable request this client makes; `retry_attempts` must be at least 1. `token_provider` is a callable (sync or async) returning a token string, called fresh on every request, no caching. `compress_results` requests LZ4-compressed cloud-fetch chunks (see above); set `False` to opt out. `protocol="thrift"` (the default, as of this crate's own real-workspace benchmarking -- see `AGENTS.md`'s design-invariant entry) speaks the same HiveServer2-compatible Thrift-over-HTTPS protocol `databricks-sql-connector` uses by default (`thrift.rs`, a hand-rolled `TBinaryProtocol` reader/writer -- no new Cargo dependency) -- measurably faster for small queries (its `ExecuteStatement` RPC can return a small result inline via `getDirectResults`, in the same call that submits the statement) at the cost of `prefer_inline` becoming a silent no-op (Thrift has no INLINE-disposition equivalent, and doesn't need one); never slower than SEA on any query shape tested. `protocol="sea"` instead talks to the REST Statement Execution API -- still fully supported, opt in explicitly if you have a reason to prefer it. On par with SEA for a large, multi-chunk result too -- `run_thrift_fetch_loop` (`pipeline.rs`) fans its chunk downloads out across a `chunk_fetch_concurrency`-sized worker pool spanning the *whole* result (pipelined with the sequential `FetchResults` discovery calls, not serialized behind them), the same concurrency shape as SEA's own `fetch_chunks_with_backpressure` (see `AGENTS.md`'s own entry on this).
-- `Client.execute(statement, *, catalog=None, schema=None, parameters=None, prefer_inline=False) -> ResultSet` -- submits and starts background chunk fetching without pulling anything yet. `parameters` is Databricks' own named-parameter format (`[{"name":..., "value":..., "type":...}]`), passed straight through. `prefer_inline=True` submits with `disposition=INLINE, format=JSON_ARRAY` instead, for a caller who expects a small (well under Databricks' 25 MiB inline cap) result and wants to skip the chunk-fetch round trip -- on a result too big for INLINE, or containing a column type `json_convert.rs` doesn't map (STRUCT/ARRAY-of-STRUCT/MAP/VARIANT), it transparently falls back to a second, normal `execute()` and the query runs twice. See `AGENTS.md`'s "Design invariants" section (in the root package) for the full reasoning and real-workspace verification behind this.
+- `Client.execute(statement, *, catalog=None, schema=None, parameters=None, prefer_inline=False) -> ResultSet` -- submits and starts background chunk fetching without pulling anything yet. `parameters` is Databricks' own named-parameter format (`[{"name":..., "value":..., "type":...}]`), passed straight through. `prefer_inline=True` submits with `disposition=INLINE, format=JSON_ARRAY` instead, for a caller who expects a small (well under Databricks' 25 MiB inline cap) result and wants to skip the chunk-fetch round trip -- the recognized INLINE byte-limit failure falls back to a second, normal `execute()`. If an already-succeeded result cannot be converted (including unsupported empty STRUCT arrays), it raises `ArrowbricksError` without resubmitting the statement. Use the byte-limit retry only with SQL that is safe to execute again. See `AGENTS.md`'s "Design invariants" section (in the root package) for the full reasoning and real-workspace verification behind this.
   - `ResultSet.fetchmany_arrow(n) -> Table` -- pulls/decodes only as many chunks as needed for `n` rows, buffering the rest; may return fewer than `n` once exhausted.
   - `ResultSet.fetchall_arrow() -> Table` -- drains everything remaining.
   - `ResultSet.fetchall_arrow_streamed(*, total_timeout_s=None)` -- same as `fetchall_arrow()`, but an async iterator yielding the `HEARTBEAT` singleton while pulling chunks instead of blocking silently (bridge e.g. an SSE connection through the download), then a `Table` exactly once. Raises if `total_timeout_s` elapses first.
@@ -76,7 +82,7 @@ unlike the table object itself.
 - `Client.stream_ndjson_lines(statement, *, catalog=None, schema=None, parameters=None, total_timeout_s=None)` -- chunk-at-a-time: an async iterator yielding the `HEARTBEAT` singleton while waiting on the statement or any individual chunk, then a `list[str]` of NDJSON lines (one per row, explicit nulls, ISO-8601 timestamps) per chunk in logical order. Decode and JSON encoding both happen in Rust -- backs `stream_query_json` end to end.
 - `Client.upload_volume_file(volume_path, data: bytes)` / `Client.delete_volume_file(volume_path)` -- Unity Catalog volume files via the Files API. Delete treats a 404 as success (idempotent). Both raise a plain `RuntimeError` (message only) on failure.
 - `write_ipc_stream(stream, buf)` -- free function; writes any object implementing `__arrow_c_stream__` (a `Table` from this crate, arro3, pyarrow, ...) as uncompressed Arrow-IPC stream bytes to a Python file-like object. No dependency needed regardless of the input's origin.
-- `read_ipc_stream(data: bytes) -> Table` -- free function; the exact inverse of `write_ipc_stream`, parsing raw Arrow-IPC stream bytes back into a `Table`. No dependency needed regardless of where the bytes came from -- backs `arrowbricks.ReplayableArrowChunk`, which needs to re-parse the same cached bytes on every `__arrow_c_stream__` call.
+- `read_ipc_stream(data: bytes) -> Table` -- free function; the exact inverse of `write_ipc_stream`, parsing raw Arrow-IPC stream bytes back into a `Table`. No dependency needed regardless of where the bytes came from -- backs `arrowbricks.ReplayableArrowChunk`, which needs to re-parse the same cached bytes on every `__arrow_c_stream__` call. Decoding releases the Python interpreter lock so independent replay calls can run concurrently; decoded arrays retain ownership of the immutable input bytes.
 - `HEARTBEAT` -- module-level singleton; compare with `is`, e.g. `if item is _core.HEARTBEAT: ...`.
 
 ## With DuckDB

arrowbricks-3.1.4/rust/arrowbricks_core/proptest-regressions/json_convert.txt

@@ -0,0 +1,7 @@
+# Seeds for failure cases proptest has generated in the past. It is
+# automatically read and these particular cases re-run before any
+# novel cases are generated.
+#
+# It is recommended to check this file in to source control so that
+# everyone who runs the test benefits from these saved cases.
+cc c23ce2dbe22693eff6644d643de77c6c113dc918c876434391fb93a05438d706 # shrinks to type_name = "STRUCT", precision = None, scale = None, type_text = Some(""), values = []