arrowbricks 0.2.0__tar.gz → 1.0.0__tar.gz

{arrowbricks-0.2.0 → arrowbricks-1.0.0}/PKG-INFO

@@ -1,25 +1,25 @@
 Metadata-Version: 2.4
 Name: arrowbricks
-Version: 0.2.0
-Summary: Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a DB-API-ish Cursor (fetchone/fetchmany/fetchall/fetchall_arrow) or NDJSON streaming. Single Arrow engine (arro3), no DuckDB.
-Project-URL: Repository, https://github.com/bmsuisse/arrowbricks
-License-Expression: MIT
+Version: 1.0.0
+Requires-Dist: arro3-core>=0.8 ; extra == 'arro3'
+Provides-Extra: arro3
 License-File: LICENSE
+Summary: Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a DB-API-ish Cursor (fetchone/fetchmany/fetchall/fetchall_arrow) or NDJSON streaming. Rust/PyO3 core throughout -- zero required runtime dependencies.
+License-Expression: MIT
 Requires-Python: >=3.11
-Requires-Dist: arro3-core>=0.8
-Requires-Dist: arro3-io>=0.8
-Requires-Dist: httpx>=0.27
-Description-Content-Type: text/markdown
+Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
+Project-URL: Repository, https://github.com/bmsuisse/arrowbricks
 
 # arrowbricks
 
-Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a `Cursor` shaped like [`databricks-sql-python`](https://github.com/databricks/databricks-sql-python)'s (`execute`, `fetchone`/`fetchmany`/`fetchall`, `fetchall_arrow`/`fetchmany_arrow`), or `stream_query_json` for streaming NDJSON. One Arrow engine ([arro3](https://github.com/kylebarron/arro3)), no DuckDB, no pandas/pyarrow.
+Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a `Cursor` shaped like [`databricks-sql-python`](https://github.com/databricks/databricks-sql-python)'s (`execute`, `fetchone`/`fetchmany`/`fetchall`, `fetchall_arrow`/`fetchmany_arrow`), or `stream_query_json` for streaming NDJSON.
 
-- Single responsibility: Databricks to Arrow via arro3. No embedded query engine -- that's [duckbricks](https://github.com/bmsuisse/duckbricks), built on top of this.
-- Bring-your-own-auth -- a static token or your own token-refresh callable. No cloud-SDK dependency baked in.
-- Result-order preserved even though chunks can complete out of order over the network.
-- Chunks are fetched lazily as `fetchone`/`fetchmany`/`fetchall` actually need them, not all upfront.
-- Heartbeats between slow chunks (`execute_streamed`/`stream_query_json`), so a caller streaming this over e.g. SSE never goes silent during a cold warehouse start.
+- **Rust core.** Statement submit/poll, bounded-concurrency chunk fetch, the reorder buffer, and Arrow-IPC decode all run in a [PyO3](https://pyo3.rs)/[arrow-rs](https://github.com/apache/arrow-rs) extension bundled in this same package -- 1.6x-2.5x faster than a pure-Python/asyncio client on a multi-chunk result, scaling further with chunk count and concurrency where asyncio+GIL plateaus.
+- **Zero required dependencies.** `pip install arrowbricks` and go.
+- **Bring-your-own-auth** -- a static token or your own token-refresh callable. No cloud-SDK dependency baked in.
+- **Result order preserved** even though chunks can complete out of order over the network.
+- **Lazy fetching** -- chunks are pulled only as `fetchone`/`fetchmany`/`fetchall` actually need them, not all upfront.
+- **Heartbeats** between slow chunks (`execute_streamed`/`stream_query_json`), so a caller streaming this over e.g. SSE never goes silent during a cold warehouse start.
 
 ## Install
 
@@ -27,7 +27,7 @@ Runs SQL against a Databricks SQL warehouse via the Statement Execution API and
 pip install arrowbricks
 ```
 
-Dependencies: `httpx` + `arro3-core` + `arro3-io`. That's the whole tree.
+Ships as precompiled platform wheels (Linux/macOS/Windows) -- no Rust toolchain needed, and nothing else to install for most of the API. Row-tuple fetches (`fetchone`/`fetchmany`/`fetchall`) need one optional extra: `pip install arrowbricks[arro3]` -- see [Arrow vs. row-tuple fetches](#arrow-vs-row-tuple-fetches) below.
 
 ## Quickstart
 
@@ -49,7 +49,7 @@ async def main():
         print(row)
 
     await cursor.execute("SELECT * FROM my_catalog.my_schema.my_table LIMIT 100")
-    table = await cursor.fetchall_arrow()  # an arro3 Table
+    table = await cursor.fetchall_arrow()  # an Arrow table (arro3/pyarrow/DuckDB-compatible)
 
 
 asyncio.run(main())
@@ -78,14 +78,6 @@ combined heartbeat/timeout budget across both the wait and the download, or
 [`examples/azure_auth.py`](examples/azure_auth.py) for a caching
 `token_provider` built on Azure AD (`DefaultAzureCredential`).
 
-## Why not `databricks-sql-connector`?
-
-The [official driver](https://github.com/databricks/databricks-sql-python) is the right choice if you need full DB-API 2.0 compatibility over Databricks' Thrift/ODBC-style protocol. If you just want a query result as Arrow/JSON in your own async app, it drags in a lot for that: `pandas`, `thrift`, `openpyxl`, `pybreaker`, `pyjwt`, `oauthlib`, `lz4`, `requests`, `urllib3` as hard dependencies. arrowbricks talks to the plain REST Statement Execution API instead, and its whole dependency tree is `httpx` + `arro3-core` + `arro3-io`. The `Cursor` API is deliberately shaped like the official driver's so switching between them is mostly a constructor change, but arrowbricks is async throughout (`execute`, `fetchone`, etc. are all coroutines) -- there's no sync escape hatch.
-
-## Why not `duckbricks`?
-
-[duckbricks](https://github.com/bmsuisse/duckbricks) does the same Databricks-to-Arrow work, then goes further: it uses a real embedded DuckDB engine to materialize results into your own DuckDB connection/table (`feed_select_to_duckdb_table`), or push a DuckDB query's result *up* to Databricks (`feed_duckdb_table_to_databricks`). If you need that -- a real local SQL engine sitting on top, not just "run this query, get Arrow/JSON back" -- use duckbricks; it depends on arrowbricks for the Databricks/Arrow half. If you don't need DuckDB at all, arrowbricks alone is the smaller, single-responsibility half.
-
 ## Auth
 
 `connect`/`DatabricksClient` take either:
@@ -103,24 +95,53 @@ conn = connect(host=..., warehouse_id=..., token_provider=my_token_provider)
 
 - `connect(host, warehouse_id, *, token=None, token_provider=None, ...) -> Connection`
 - `Connection.cursor() -> Cursor`
-- `Connection.client -> DatabricksClient` -- the same client `cursor()` uses, for lower-level access (e.g. `stream_query_json`, `execute_json_statement`, `upload_volume_file`).
+- `Connection.client -> DatabricksClient` -- the same client `cursor()` uses, for lower-level access (e.g. `stream_query_json`, `upload_volume_file`).
 - `Cursor.execute(sql, parameters=None, *, row_limit=None, offset=None, catalog=None, schema=None, total_timeout_s=None) -> Cursor` -- submits and waits for the statement, like a real DB-API cursor. `parameters`, if given, is Databricks' own named-parameter format -- `[{"name": ..., "value": ..., "type": ...}]` bound against `:name` markers in `sql`.
 - `Cursor.execute_streamed(...)` -- same args, but an async generator yielding `HEARTBEAT` while waiting on a slow cold start, then the ready `Cursor` -- for bridging e.g. an SSE connection. Its timeout/heartbeats stop the moment the statement is ready, *before* any chunk has been downloaded -- see `fetchall_streamed` below for the download phase itself.
-- `Cursor.fetchone() -> tuple | None`, `Cursor.fetchmany(size) -> list[tuple]`, `Cursor.fetchall() -> list[tuple]`
-- `Cursor.fetchmany_arrow(size) -> arro3.core.Table`, `Cursor.fetchall_arrow() -> arro3.core.Table`
+- `Cursor.fetchone() -> tuple | None`, `Cursor.fetchmany(size) -> list[tuple]`, `Cursor.fetchall() -> list[tuple]`, and iterating a `Cursor` directly -- row tuples; needs the `arro3` extra.
+- `Cursor.fetchmany_arrow(size) -> Table`, `Cursor.fetchall_arrow() -> Table` -- an Arrow table (implements `__arrow_c_stream__`, so arro3/pyarrow/DuckDB can all consume it directly, zero-copy).
 - `Cursor.fetchall_streamed(*, total_timeout_s=None)` / `Cursor.fetchall_arrow_streamed(*, total_timeout_s=None)` -- like `fetchall()`/`fetchall_arrow()`, but yield `HEARTBEAT` while pulling chunks instead of blocking silently, then the final rows/Table -- for a caller downloading a large result over SSE who needs heartbeats (and a timeout) through the *download*, not just the initial wait. Compose with `execute_streamed` and a shared deadline if you want one combined budget across both phases (see `examples/fastapi_sse_pivot.py`).
-- `Cursor` is an async iterator, yielding one row (tuple) at a time.
 - `Cursor.description` -- DB-API-style `[(name, type_name, None, None, None, None, None), ...]` after `execute()`.
 - `stream_query_json(client, sql, **kwargs)` -- yields `HEARTBEAT`, then each row as a JSON string, as soon as its chunk arrives. Timestamps come out as full ISO-8601, every column key is always present (`"col":null` for a null value, never an omitted key).
-- `DatabricksClient(host, warehouse_id, *, token=None, token_provider=None, ...)` -- the lower-level client `Connection` wraps. `client.execute_json_statement(sql, ...)` for plain JSON rows with no Arrow parse at all; `client.upload_volume_file(volume_path, data)`/`client.delete_volume_file(volume_path)` for the Files API.
-- `write_ipc_stream(table_or_chunk, buf)` -- thin wrapper around `arro3.io.write_ipc_stream` that always writes uncompressed bodies (see below).
+- `DatabricksClient(host, warehouse_id, *, token=None, token_provider=None, ...)` -- the lower-level client `Connection` wraps. `client.upload_volume_file(volume_path, data)`/`client.delete_volume_file(volume_path)` for the Files API.
+- `write_ipc_stream(table, buf)` -- writes any Arrow-C-Data-Interface-compatible object as an uncompressed Arrow-IPC stream (see below).
 
 `Cursor.execute`/`execute_streamed`/`stream_query_json` all accept `catalog`, `schema`, `row_limit`, `offset`, and `total_timeout_s`.
 
+## Arrow vs. row-tuple fetches
+
+Everything above works with zero dependencies installed *except* row-tuple fetches. `fetchall_arrow`/`fetchmany_arrow` return an Arrow table straight from the Rust core -- the faster path if your code can consume Arrow directly (DuckDB, pyarrow, polars, a Parquet writer, ...):
+
+```python
+import duckdb
+
+table = await cursor.fetchall_arrow()
+duckdb.sql("SELECT count(*) FROM table").show()  # DuckDB reads it zero-copy
+```
+
+`fetchone`/`fetchmany`/`fetchall` (and iterating a `Cursor` directly) materialize actual Python tuples instead -- `("id", "label")`-style rows you can index into, print, or pass to code that doesn't know about Arrow at all. That conversion needs `arro3-core` (`pip install arrowbricks[arro3]`):
+
+```python
+await cursor.execute("SELECT id, label FROM my_catalog.my_schema.my_table")
+async for row in cursor:  # or: rows = await cursor.fetchall()
+    print(row[0], row[1])
+```
+
+Calling a row-tuple method without `arro3-core` installed raises a `ModuleNotFoundError` naming the exact install command, rather than failing silently or with a confusing traceback.
+
+## Rust core
+
+`rust/arrowbricks_core` is the crate implementing the hot path above, built into this same `arrowbricks` wheel as a compiled submodule -- not a separate PyPI package. See [its own README](rust/arrowbricks_core/README.md) for the crate-level design, plus standalone DuckDB and FastAPI SSE examples against the compiled extension directly.
+
+## Why not `databricks-sql-connector`?
+
+The [official driver](https://github.com/databricks/databricks-sql-python) is the right choice if you need full DB-API 2.0 compatibility over Databricks' Thrift/ODBC-style protocol. If you just want a query result as Arrow/JSON in your own async app, it drags in a lot for that: `pandas`, `thrift`, `openpyxl`, `pybreaker`, `pyjwt`, `oauthlib`, `lz4`, `requests`, `urllib3` as hard dependencies. arrowbricks talks to the plain REST Statement Execution API instead, with a Rust core and zero required dependencies of its own. The `Cursor` API is deliberately shaped like the official driver's so switching between them is mostly a constructor change, but arrowbricks is async throughout (`execute`, `fetchone`, etc. are all coroutines) -- there's no sync escape hatch.
+
 ## A note on Arrow IPC compression
 
-`write_ipc_stream` (and everything in this package that serializes Arrow-IPC bytes) always writes **uncompressed** bodies. arro3's own default (`compression="LZ4"`) is transparently decompressed by DuckDB's Arrow reader, but not necessarily by every other Arrow IPC reader -- notably, `duckdb-wasm`'s browser-side decoder silently fails to parse LZ4-compressed bodies. If you're producing bytes that might be consumed by something other than a Python DuckDB connection, this default matters.
+`write_ipc_stream` (and everything in this package that serializes Arrow-IPC bytes) always writes **uncompressed** bodies. A compressed body (arro3's own default is `compression="LZ4"`) is transparently decompressed by some Arrow readers (e.g. DuckDB's) but not necessarily by every other Arrow IPC reader -- notably, `duckdb-wasm`'s browser-side decoder silently fails to parse LZ4-compressed bodies. Since arrowbricks' bytes might end up read by anything, plain uncompressed is the safe default.
 
 ## License
 
-MIT
+MIT
+

{arrowbricks-0.2.0 → arrowbricks-1.0.0}/README.md

@@ -1,12 +1,13 @@
 # arrowbricks
 
-Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a `Cursor` shaped like [`databricks-sql-python`](https://github.com/databricks/databricks-sql-python)'s (`execute`, `fetchone`/`fetchmany`/`fetchall`, `fetchall_arrow`/`fetchmany_arrow`), or `stream_query_json` for streaming NDJSON. One Arrow engine ([arro3](https://github.com/kylebarron/arro3)), no DuckDB, no pandas/pyarrow.
+Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a `Cursor` shaped like [`databricks-sql-python`](https://github.com/databricks/databricks-sql-python)'s (`execute`, `fetchone`/`fetchmany`/`fetchall`, `fetchall_arrow`/`fetchmany_arrow`), or `stream_query_json` for streaming NDJSON.
 
-- Single responsibility: Databricks to Arrow via arro3. No embedded query engine -- that's [duckbricks](https://github.com/bmsuisse/duckbricks), built on top of this.
-- Bring-your-own-auth -- a static token or your own token-refresh callable. No cloud-SDK dependency baked in.
-- Result-order preserved even though chunks can complete out of order over the network.
-- Chunks are fetched lazily as `fetchone`/`fetchmany`/`fetchall` actually need them, not all upfront.
-- Heartbeats between slow chunks (`execute_streamed`/`stream_query_json`), so a caller streaming this over e.g. SSE never goes silent during a cold warehouse start.
+- **Rust core.** Statement submit/poll, bounded-concurrency chunk fetch, the reorder buffer, and Arrow-IPC decode all run in a [PyO3](https://pyo3.rs)/[arrow-rs](https://github.com/apache/arrow-rs) extension bundled in this same package -- 1.6x-2.5x faster than a pure-Python/asyncio client on a multi-chunk result, scaling further with chunk count and concurrency where asyncio+GIL plateaus.
+- **Zero required dependencies.** `pip install arrowbricks` and go.
+- **Bring-your-own-auth** -- a static token or your own token-refresh callable. No cloud-SDK dependency baked in.
+- **Result order preserved** even though chunks can complete out of order over the network.
+- **Lazy fetching** -- chunks are pulled only as `fetchone`/`fetchmany`/`fetchall` actually need them, not all upfront.
+- **Heartbeats** between slow chunks (`execute_streamed`/`stream_query_json`), so a caller streaming this over e.g. SSE never goes silent during a cold warehouse start.
 
 ## Install
 
@@ -14,7 +15,7 @@ Runs SQL against a Databricks SQL warehouse via the Statement Execution API and
 pip install arrowbricks
 ```
 
-Dependencies: `httpx` + `arro3-core` + `arro3-io`. That's the whole tree.
+Ships as precompiled platform wheels (Linux/macOS/Windows) -- no Rust toolchain needed, and nothing else to install for most of the API. Row-tuple fetches (`fetchone`/`fetchmany`/`fetchall`) need one optional extra: `pip install arrowbricks[arro3]` -- see [Arrow vs. row-tuple fetches](#arrow-vs-row-tuple-fetches) below.
 
 ## Quickstart
 
@@ -36,7 +37,7 @@ async def main():
         print(row)
 
     await cursor.execute("SELECT * FROM my_catalog.my_schema.my_table LIMIT 100")
-    table = await cursor.fetchall_arrow()  # an arro3 Table
+    table = await cursor.fetchall_arrow()  # an Arrow table (arro3/pyarrow/DuckDB-compatible)
 
 
 asyncio.run(main())
@@ -65,14 +66,6 @@ combined heartbeat/timeout budget across both the wait and the download, or
 [`examples/azure_auth.py`](examples/azure_auth.py) for a caching
 `token_provider` built on Azure AD (`DefaultAzureCredential`).
 
-## Why not `databricks-sql-connector`?
-
-The [official driver](https://github.com/databricks/databricks-sql-python) is the right choice if you need full DB-API 2.0 compatibility over Databricks' Thrift/ODBC-style protocol. If you just want a query result as Arrow/JSON in your own async app, it drags in a lot for that: `pandas`, `thrift`, `openpyxl`, `pybreaker`, `pyjwt`, `oauthlib`, `lz4`, `requests`, `urllib3` as hard dependencies. arrowbricks talks to the plain REST Statement Execution API instead, and its whole dependency tree is `httpx` + `arro3-core` + `arro3-io`. The `Cursor` API is deliberately shaped like the official driver's so switching between them is mostly a constructor change, but arrowbricks is async throughout (`execute`, `fetchone`, etc. are all coroutines) -- there's no sync escape hatch.
-
-## Why not `duckbricks`?
-
-[duckbricks](https://github.com/bmsuisse/duckbricks) does the same Databricks-to-Arrow work, then goes further: it uses a real embedded DuckDB engine to materialize results into your own DuckDB connection/table (`feed_select_to_duckdb_table`), or push a DuckDB query's result *up* to Databricks (`feed_duckdb_table_to_databricks`). If you need that -- a real local SQL engine sitting on top, not just "run this query, get Arrow/JSON back" -- use duckbricks; it depends on arrowbricks for the Databricks/Arrow half. If you don't need DuckDB at all, arrowbricks alone is the smaller, single-responsibility half.
-
 ## Auth
 
 `connect`/`DatabricksClient` take either:
@@ -90,24 +83,52 @@ conn = connect(host=..., warehouse_id=..., token_provider=my_token_provider)
 
 - `connect(host, warehouse_id, *, token=None, token_provider=None, ...) -> Connection`
 - `Connection.cursor() -> Cursor`
-- `Connection.client -> DatabricksClient` -- the same client `cursor()` uses, for lower-level access (e.g. `stream_query_json`, `execute_json_statement`, `upload_volume_file`).
+- `Connection.client -> DatabricksClient` -- the same client `cursor()` uses, for lower-level access (e.g. `stream_query_json`, `upload_volume_file`).
 - `Cursor.execute(sql, parameters=None, *, row_limit=None, offset=None, catalog=None, schema=None, total_timeout_s=None) -> Cursor` -- submits and waits for the statement, like a real DB-API cursor. `parameters`, if given, is Databricks' own named-parameter format -- `[{"name": ..., "value": ..., "type": ...}]` bound against `:name` markers in `sql`.
 - `Cursor.execute_streamed(...)` -- same args, but an async generator yielding `HEARTBEAT` while waiting on a slow cold start, then the ready `Cursor` -- for bridging e.g. an SSE connection. Its timeout/heartbeats stop the moment the statement is ready, *before* any chunk has been downloaded -- see `fetchall_streamed` below for the download phase itself.
-- `Cursor.fetchone() -> tuple | None`, `Cursor.fetchmany(size) -> list[tuple]`, `Cursor.fetchall() -> list[tuple]`
-- `Cursor.fetchmany_arrow(size) -> arro3.core.Table`, `Cursor.fetchall_arrow() -> arro3.core.Table`
+- `Cursor.fetchone() -> tuple | None`, `Cursor.fetchmany(size) -> list[tuple]`, `Cursor.fetchall() -> list[tuple]`, and iterating a `Cursor` directly -- row tuples; needs the `arro3` extra.
+- `Cursor.fetchmany_arrow(size) -> Table`, `Cursor.fetchall_arrow() -> Table` -- an Arrow table (implements `__arrow_c_stream__`, so arro3/pyarrow/DuckDB can all consume it directly, zero-copy).
 - `Cursor.fetchall_streamed(*, total_timeout_s=None)` / `Cursor.fetchall_arrow_streamed(*, total_timeout_s=None)` -- like `fetchall()`/`fetchall_arrow()`, but yield `HEARTBEAT` while pulling chunks instead of blocking silently, then the final rows/Table -- for a caller downloading a large result over SSE who needs heartbeats (and a timeout) through the *download*, not just the initial wait. Compose with `execute_streamed` and a shared deadline if you want one combined budget across both phases (see `examples/fastapi_sse_pivot.py`).
-- `Cursor` is an async iterator, yielding one row (tuple) at a time.
 - `Cursor.description` -- DB-API-style `[(name, type_name, None, None, None, None, None), ...]` after `execute()`.
 - `stream_query_json(client, sql, **kwargs)` -- yields `HEARTBEAT`, then each row as a JSON string, as soon as its chunk arrives. Timestamps come out as full ISO-8601, every column key is always present (`"col":null` for a null value, never an omitted key).
-- `DatabricksClient(host, warehouse_id, *, token=None, token_provider=None, ...)` -- the lower-level client `Connection` wraps. `client.execute_json_statement(sql, ...)` for plain JSON rows with no Arrow parse at all; `client.upload_volume_file(volume_path, data)`/`client.delete_volume_file(volume_path)` for the Files API.
-- `write_ipc_stream(table_or_chunk, buf)` -- thin wrapper around `arro3.io.write_ipc_stream` that always writes uncompressed bodies (see below).
+- `DatabricksClient(host, warehouse_id, *, token=None, token_provider=None, ...)` -- the lower-level client `Connection` wraps. `client.upload_volume_file(volume_path, data)`/`client.delete_volume_file(volume_path)` for the Files API.
+- `write_ipc_stream(table, buf)` -- writes any Arrow-C-Data-Interface-compatible object as an uncompressed Arrow-IPC stream (see below).
 
 `Cursor.execute`/`execute_streamed`/`stream_query_json` all accept `catalog`, `schema`, `row_limit`, `offset`, and `total_timeout_s`.
 
+## Arrow vs. row-tuple fetches
+
+Everything above works with zero dependencies installed *except* row-tuple fetches. `fetchall_arrow`/`fetchmany_arrow` return an Arrow table straight from the Rust core -- the faster path if your code can consume Arrow directly (DuckDB, pyarrow, polars, a Parquet writer, ...):
+
+```python
+import duckdb
+
+table = await cursor.fetchall_arrow()
+duckdb.sql("SELECT count(*) FROM table").show()  # DuckDB reads it zero-copy
+```
+
+`fetchone`/`fetchmany`/`fetchall` (and iterating a `Cursor` directly) materialize actual Python tuples instead -- `("id", "label")`-style rows you can index into, print, or pass to code that doesn't know about Arrow at all. That conversion needs `arro3-core` (`pip install arrowbricks[arro3]`):
+
+```python
+await cursor.execute("SELECT id, label FROM my_catalog.my_schema.my_table")
+async for row in cursor:  # or: rows = await cursor.fetchall()
+    print(row[0], row[1])
+```
+
+Calling a row-tuple method without `arro3-core` installed raises a `ModuleNotFoundError` naming the exact install command, rather than failing silently or with a confusing traceback.
+
+## Rust core
+
+`rust/arrowbricks_core` is the crate implementing the hot path above, built into this same `arrowbricks` wheel as a compiled submodule -- not a separate PyPI package. See [its own README](rust/arrowbricks_core/README.md) for the crate-level design, plus standalone DuckDB and FastAPI SSE examples against the compiled extension directly.
+
+## Why not `databricks-sql-connector`?
+
+The [official driver](https://github.com/databricks/databricks-sql-python) is the right choice if you need full DB-API 2.0 compatibility over Databricks' Thrift/ODBC-style protocol. If you just want a query result as Arrow/JSON in your own async app, it drags in a lot for that: `pandas`, `thrift`, `openpyxl`, `pybreaker`, `pyjwt`, `oauthlib`, `lz4`, `requests`, `urllib3` as hard dependencies. arrowbricks talks to the plain REST Statement Execution API instead, with a Rust core and zero required dependencies of its own. The `Cursor` API is deliberately shaped like the official driver's so switching between them is mostly a constructor change, but arrowbricks is async throughout (`execute`, `fetchone`, etc. are all coroutines) -- there's no sync escape hatch.
+
 ## A note on Arrow IPC compression
 
-`write_ipc_stream` (and everything in this package that serializes Arrow-IPC bytes) always writes **uncompressed** bodies. arro3's own default (`compression="LZ4"`) is transparently decompressed by DuckDB's Arrow reader, but not necessarily by every other Arrow IPC reader -- notably, `duckdb-wasm`'s browser-side decoder silently fails to parse LZ4-compressed bodies. If you're producing bytes that might be consumed by something other than a Python DuckDB connection, this default matters.
+`write_ipc_stream` (and everything in this package that serializes Arrow-IPC bytes) always writes **uncompressed** bodies. A compressed body (arro3's own default is `compression="LZ4"`) is transparently decompressed by some Arrow readers (e.g. DuckDB's) but not necessarily by every other Arrow IPC reader -- notably, `duckdb-wasm`'s browser-side decoder silently fails to parse LZ4-compressed bodies. Since arrowbricks' bytes might end up read by anything, plain uncompressed is the safe default.
 
 ## License
 
-MIT
+MIT

arrowbricks-1.0.0/pyproject.toml

@@ -0,0 +1,81 @@
+[project]
+name = "arrowbricks"
+version = "1.0.0"
+description = "Runs SQL against a Databricks SQL warehouse via the Statement Execution API and hands you the result as Arrow -- a DB-API-ish Cursor (fetchone/fetchmany/fetchall/fetchall_arrow) or NDJSON streaming. Rust/PyO3 core throughout -- zero required runtime dependencies."
+readme = "README.md"
+license = "MIT"
+license-files = ["LICENSE"]
+requires-python = ">=3.11"
+# No hard runtime dependencies -- httpx and arro3-io are fully eliminated
+# (see rust/arrowbricks_core's own reqwest client + Arrow-IPC/NDJSON
+# writer). arro3-core is optional (see below): only Cursor.fetchone/
+# fetchmany/fetchall's row materialization needs it -- fetchall_arrow,
+# execute_arrow, stream_query_json, upload_volume_file/delete_volume_file,
+# and Cursor.description all work with zero dependencies installed.
+dependencies = []
+
+[project.optional-dependencies]
+# pyo3-arrow's own Table.column()/schema methods are designed to hand back
+# the caller's *real* arro3.core objects for further arro3-ecosystem
+# interop (not a value pyo3-arrow can materialize itself), so
+# fetchone/fetchmany/fetchall need this installed. `pip install
+# arrowbricks[arro3]` -- or handle rows as Arrow via fetchall_arrow/
+# fetchmany_arrow instead, which don't need it at all.
+arro3 = ["arro3-core>=0.8"]
+
+[project.urls]
+Repository = "https://github.com/bmsuisse/arrowbricks"
+
+[build-system]
+requires = ["maturin>=1.0,<2.0"]
+build-backend = "maturin"
+
+# The compiled hot path (rust/arrowbricks_core) builds into this same wheel as
+# arrowbricks._core -- not a separately published package -- so `pip install
+# arrowbricks` is the only install step. python-source points at the existing
+# src/ layout (arrowbricks/__init__.py, client.py, cursor.py, _streaming.py);
+# manifest-path points back at the Cargo.toml living under rust/, since it
+# isn't in the same directory as this pyproject.toml.
+[tool.maturin]
+python-source = "src"
+module-name = "arrowbricks._core"
+manifest-path = "rust/arrowbricks_core/Cargo.toml"
+# `license-files` above declares LICENSE in metadata, but maturin's sdist
+# build doesn't bundle a repo-root file just because it's referenced there --
+# it needs to be told explicitly, or PyPI rejects the upload with "License-File
+# LICENSE does not exist in distribution file" (a real failure the first time
+# v1.0.0 released: all 5 wheels published fine, only the sdist hit this).
+include = ["LICENSE"]
+
+[dependency-groups]
+# arro3-core/arro3-io are test-only -- tests/conftest.py uses them to build
+# synthetic Arrow-IPC chunk bytes so tests exercise a real Arrow IPC round
+# trip. The package itself has no runtime dependency on them (or anything
+# else) -- see rust/arrowbricks_core's own Arrow/NDJSON encoding.
+dev = [
+    "arro3-core>=0.8",
+    "arro3-io>=0.8",
+    "pytest>=8.0",
+    "pytest-asyncio>=0.24",
+    "ruff>=0.6",
+    "ty>=0.0.65",
+]
+
+[tool.pytest.ini_options]
+asyncio_mode = "auto"
+# Bare `pytest`/`pytest -q` (no explicit path) stays scoped to the Cursor-
+# level suite -- rust/arrowbricks_core/tests_py (the compiled extension's own
+# PyO3-level tests) is invoked explicitly by path where it's needed (CI's
+# build job, or by hand), not auto-discovered every run.
+testpaths = ["tests"]
+
+[tool.ruff]
+line-length = 120
+target-version = "py311"
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "S", "B"]
+
+[tool.ruff.lint.per-file-ignores]
+"tests/*" = ["S101", "S106"]
+"rust/arrowbricks_core/tests_py/*" = ["S101", "S106"]

arrowbricks-1.0.0/rust/arrowbricks_core/.gitignore

@@ -0,0 +1,5 @@
+# Reference skill docs installed locally via `npx skills add` -- dev tooling,
+# not project source.
+.agents/
+.claude/
+skills-lock.json