arrowsql 0.2.0__tar.gz → 0.3.0__tar.gz

{arrowsql-0.2.0 → arrowsql-0.3.0}/CHANGELOG.md

@@ -8,12 +8,76 @@ Version numbers in this project are deliberately three different things:
 
 | what | value | meaning |
 |---|---|---|
-| distribution `arrowsql` | 0.2.0 | this project's release number (`pyproject.toml`) |
+| distribution `arrowsql` | 0.3.0 | this project's release number (`pyproject.toml`) |
 | `arrowsql.__version__` | 1.16.0 | the upstream `mssql-python` release the Python layer is vendored from; kept so code that checks the upstream version keeps working |
 | `arrowsql.ddbc_bindings.__version__` | 1.0.0 | version of the Rust binding contract (`docs/CONTRACT.md`) |
 
 ## [Unreleased]
 
+## [0.3.0] - 2026-11-07
+
+### Added
+
+- `crates/arrowsql-core`: the driver core as a Rust crate without pyo3
+  (connect, execute, fetch, Arrow decoding, pooling), with a public API for
+  other Rust crates: `Connection::connect` / `execute`, `Statement::next_batch`
+  / `into_reader`, and `read_arrow_stream(options, query, Partitioning,
+  ReadOptions)`, a cancellable, `Send` Arrow `RecordBatchReader` over a
+  partitioned parallel read. The Python extension is a pyo3 layer over it; no
+  behaviour change for Python.
+- `arrowsql-core`: the partitions of a read as separate readers or Arrow C
+  streams. `plan_partitions` returns the planned `Slice`s, each opens on its
+  own connection (`Slice::open`, `Slice::open_ffi_stream`), so Spark tasks or
+  pyarrow in another shared library read them zero-copy;
+  `ParallelReader::into_ffi_stream` / `StatementReader::into_ffi_stream`
+  export the merged and single-statement readers the same way.
+- `crates/arrowsql-rowcount`: a pure-Rust example and integration test of the
+  core API (single, merged parallel and per-partition Arrow C stream reads).
+- Process-wide DNS cache for connects (`DnsCacheTTL=<seconds>` keyword, env
+  `ARROWSQL_DNS_CACHE_TTL`, default 300 s, 0 disables). A host that resolved
+  once is not resolved again for every connection (Spark tasks, parallel read
+  slices, pool members, reconnects), like the JDBC driver's long-lived cache.
+  An expired entry is served at once while one background thread refreshes it
+  (up to an hour past its TTL), so a slow or failing resolver ("Temporary
+  failure in name resolution") no longer costs each connection seconds or a
+  failed connect. Concurrent lookups of one host share a single resolver call;
+  each resolver call is bounded (`ARROWSQL_DNS_RESOLVE_TIMEOUT`, default 10 s)
+  and a temporary failure is retried once. Failures are never cached; if no
+  cached address accepts, the entry is dropped and the host is resolved once
+  more. IP literals and connection diagnostics bypass the cache.
+  `ddbc_bindings.dns_cache_stats()` reports its counters. Vendored mssql-tds
+  patch 15.
+
+### Changed
+
+- `read_arrow` / `read_arrow_stream`: slice planning, the MIN/MAX and catalog
+  probes, the worker threads and the merge run in Rust (`arrowsql-core`);
+  `arrow_parallel.py` only checks arguments. Same plans, rows and exception
+  classes as before (parity tests against the previous implementation in
+  `tests/common_issues/test_rs_read_arrow_parity.py`); invalid
+  `partition_bounds` now fail before the first connection is opened. The
+  UTF-8 transfer rewrite (`utf8_strings`) moved to Rust as well.
+- Rust dependents of `arrowsql-core`: the core declares no global allocator
+  (mimalloc is the `ddbc_bindings` default feature `mimalloc`), and both
+  crates depend on `vendor/mssql-tds` by path instead of a `[patch]`, so a git
+  dependency on the core builds the patched TDS crate without a patch section
+  of its own. Workspace-wide `cargo test` / `clippy` take `--exclude mssql-tds`.
+
+### Fixed
+
+- `write_arrow(..., workers > 1)` accepts an iterable of record batches (list, generator, objects
+  with `__arrow_c_array__`) like `workers=1`, without pyarrow (r5 F9).
+- `write_arrow(..., workers > 1)` reads a native `cursor.arrow_stream()` on demand instead of
+  draining it into memory before the workers start; the drain remains for single-connection
+  `bulkcopy_arrow`, where it avoids a deadlock (r5 F15).
+- Async `executemany` with a leading NULL decimal no longer declares the whole
+  batch `decimal(18,10)` (later values lost digits).
+- Async manual-commit `BEGIN` obeys cancellation.
+- After a server-side rollback under XACT_ABORT, writes stay rejected (25000)
+  until `commit()`/`rollback()`: the guard sees through comments, CTEs and
+  `BEGIN TRY` blocks, is no longer consumed by the first rejection and also
+  covers the async API.
+
 ## [0.2.0] - 2026-11-01
 
 ### Added

{arrowsql-0.2.0 → arrowsql-0.3.0}/CONTRIBUTING.md

@@ -20,7 +20,8 @@ anyone else can just set `DB_CONNECTION_STRING`.
 ```bash
 python -m pytest -q --timeout 120 tests/common_issues tests/port
 python scripts/check_upstream_baseline.py     # vendored upstream suite vs tests/upstream_passing_baseline.txt
-cargo test --lib
+cargo test --workspace --exclude mssql-tds --lib
+cargo test -p arrowsql-core --test live && cargo test -p arrowsql-rowcount   # Rust API, needs DB_CONNECTION_STRING
 ```
 
 Scope and rules for the three suites: [docs/TEST_SCOPE.md](docs/TEST_SCOPE.md). A new
@@ -29,7 +30,7 @@ add it to `tests/upstream_excluded.txt` with a one-line reason.
 
 ## Code
 
-- `cargo clippy --all-targets` should not add warnings; run `cargo fmt` on files you touch.
+- `cargo clippy --workspace --exclude mssql-tds --all-targets` should not add warnings; run `cargo fmt` on files you touch.
 - Edits to the vendored Python layer (`python/arrowsql`) and to `vendor/mssql-tds`
   are marked `RUST-PORT` and listed in `docs/PYTHON_LAYER_CHANGES.md` /
   `vendor/mssql-tds/VENDOR.md`. Keep upstream's lines unchanged where you can.

{arrowsql-0.2.0 → arrowsql-0.3.0}/Cargo.lock

@@ -78,6 +78,30 @@ dependencies = [
  "bitflags",
 ]
 
+[[package]]
+name = "arrowsql-core"
+version = "0.3.0"
+dependencies = [
+ "arrow-array",
+ "arrow-buffer",
+ "arrow-schema",
+ "async-trait",
+ "encoding_rs",
+ "log",
+ "mssql-tds",
+ "parking_lot",
+ "tokio",
+ "uuid",
+]
+
+[[package]]
+name = "arrowsql-rowcount"
+version = "0.3.0"
+dependencies = [
+ "arrow-array",
+ "arrowsql-core",
+]
+
 [[package]]
 name = "asn1-rs"
 version = "0.7.2"
@@ -274,13 +298,15 @@ checksum = "4583a4551df46e2792f82ceeac45e850d2e2d5debba0b91f102385cda5b11f06"
 
 [[package]]
 name = "ddbc_bindings"
-version = "0.2.0"
+version = "0.3.0"
 dependencies = [
  "arrow-array",
  "arrow-buffer",
  "arrow-schema",
+ "arrowsql-core",
  "async-trait",
  "encoding_rs",
+ "log",
  "mimalloc",
  "mssql-tds",
  "parking_lot",

{arrowsql-0.2.0 → arrowsql-0.3.0}/Cargo.toml

@@ -1,29 +1,45 @@
 [package]
 name = "ddbc_bindings"
-version = "0.2.0"
+version = "0.3.0"
 edition = "2024"
 license = "MIT"
 description = "Rust (PyO3 + mssql-tds) replacement for mssql-python's ODBC-based ddbc_bindings"
 publish = false
 readme = "README.md"
 
+[workspace]
+members = ["crates/*"]
+# vendor/mssql-tds is a member too (a path dependency under the root), but its
+# tests need dev-dependencies the vendored copy leaves out: workspace-wide
+# cargo test / clippy runs pass `--exclude mssql-tds`.
+resolver = "3"
+
 [lib]
 name = "ddbc_bindings"
 crate-type = ["cdylib", "rlib"]
 
 [features]
+default = ["mimalloc"]
+# mimalloc as the process allocator of the extension module (the core
+# declares none, so a Rust dependent keeps its own).
+mimalloc = ["dep:mimalloc"]
 # Legacy TLS: native-tls (OpenSSL) + OpenSSL Always Encrypted crypto instead of rustls.
 # Adds ~8 MB (libssl, libcrypto, libzstd) to the wheel. Build: maturin build --features native-tls
-native-tls = ["mssql-tds/native-tls-backend"]
+native-tls = ["mssql-tds/native-tls-backend", "arrowsql-core/native-tls"]
 
 [dependencies]
 pyo3 = { version = "0.29.0", features = ["extension-module"] }
-# Microsoft's native TDS implementation (MIT). Pinned: the API is pre-1.0.
-mssql-tds = { git = "https://github.com/microsoft/mssql-rs", rev = "6b6b48ecbbbdad297039d1641e2be4775eed998b", default-features = false, features = ["gssapi", "rustls-tls"] }
+# Connect/execute/fetch, Arrow decoding and the parallel read, without pyo3.
+arrowsql-core = { path = "crates/arrowsql-core" }
+log = "0.4"
+# Microsoft's native TDS implementation (MIT), vendored with local patches
+# (vendor/mssql-tds/VENDOR.md). A path dependency, so crates that depend on
+# arrowsql-core by git get the patched copy without a [patch] of their own.
+mssql-tds = { path = "vendor/mssql-tds", default-features = false, features = ["gssapi", "rustls-tls"] }
 tokio = { version = "1", features = ["rt", "rt-multi-thread", "net", "time", "sync", "macros"] }
 parking_lot = "0.12"
 uuid = "1"
-mimalloc = { version = "0.1", default-features = false }
+mimalloc = { version = "0.1", default-features = false, optional = true }
 # Arrow arrays + the C Data / C Stream interfaces (src/arrow/). Columns are
 # built with our own buffers and only wrapped as arrow-rs arrays for export.
 arrow-array = { version = "60", default-features = false, features = ["ffi"] }
@@ -37,13 +53,13 @@ async-trait = "0.1"
 # Windows: in-tree Schannel TLS (msodbcsql's stack) instead of rustls, which fails the
 # handshake with SQL Server's self-generated certificate on Windows.
 [target.'cfg(windows)'.dependencies]
-mssql-tds = { git = "https://github.com/microsoft/mssql-rs", rev = "6b6b48ecbbbdad297039d1641e2be4775eed998b", default-features = false, features = ["gssapi", "rustls-tls", "tls-schannel-direct-on-windows"] }
+mssql-tds = { path = "vendor/mssql-tds", default-features = false, features = ["gssapi", "rustls-tls", "tls-schannel-direct-on-windows"] }
 
 # musl cannot load a shared library that uses initial-exec TLS through dlopen
 # ("initial-exec TLS resolves to dynamic definition"): mimalloc's thread-local
 # heap pointer has to use the local-dynamic model there.
 [target.'cfg(target_env = "musl")'.dependencies]
-mimalloc = { version = "0.1", default-features = false, features = ["local_dynamic_tls"] }
+mimalloc = { version = "0.1", default-features = false, features = ["local_dynamic_tls"], optional = true }
 
 [build-dependencies]
 pyo3-build-config = "0.29.0"
@@ -65,11 +81,6 @@ panic = "unwind"
 # Stripped symbols: smaller .so, no effect on speed (no runtime symbolication needed).
 strip = true
 
-# Microsoft's mssql-tds at the pinned rev, with one local patch (see
-# vendor/mssql-tds/VENDOR.md). Remove when the fix is upstream.
-[patch."https://github.com/microsoft/mssql-rs"]
-mssql-tds = { path = "vendor/mssql-tds" }
-
 # Hot crates at opt-level 3: the bindings with the inlined mssql-tds, pyo3 and
 # tokio, plus allocator, text decoding, buffers and record-layer crypto.
 [profile.release.package.mimalloc]
@@ -94,6 +105,8 @@ opt-level = 3
 opt-level = 3
 [profile.release.package.ddbc_bindings]
 opt-level = 3
+[profile.release.package.arrowsql-core]
+opt-level = 3
 [profile.release.package.mssql-tds]
 opt-level = 3
 [profile.release.package.pyo3]

{arrowsql-0.2.0 → arrowsql-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: arrowsql
-Version: 0.2.0
+Version: 0.3.0
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Operating System :: POSIX :: Linux
@@ -61,6 +61,8 @@ Project-URL: Upstream, https://github.com/microsoft/mssql-python
 - **2-7x faster than mssql-python**, up to 10x faster than pyodbc on large reads.
 - **Arrow out of the box:** results go straight to DuckDB, polars, pyarrow and Delta, no pyarrow required.
 - **Parallel reads and bulk loads** over several connections.
+- **Rust core crate:** `crates/arrowsql-core` (no pyo3) exposes connect, execute and the parallel Arrow read to other Rust programs.
+- **DNS cache:** resolved hosts are reused and served stale while refreshing (`DnsCacheTTL=<seconds>`, default 300, `0` disables).
 - **No ODBC driver:** native TDS in Rust, zero required dependencies, wheels for Linux, macOS and Windows.
 - **Drop-in:** same DB-API, same connection strings; `import mssql_python` keeps working.
 

{arrowsql-0.2.0 → arrowsql-0.3.0}/README.md

@@ -20,6 +20,8 @@
 - **2-7x faster than mssql-python**, up to 10x faster than pyodbc on large reads.
 - **Arrow out of the box:** results go straight to DuckDB, polars, pyarrow and Delta, no pyarrow required.
 - **Parallel reads and bulk loads** over several connections.
+- **Rust core crate:** `crates/arrowsql-core` (no pyo3) exposes connect, execute and the parallel Arrow read to other Rust programs.
+- **DNS cache:** resolved hosts are reused and served stale while refreshing (`DnsCacheTTL=<seconds>`, default 300, `0` disables).
 - **No ODBC driver:** native TDS in Rust, zero required dependencies, wheels for Linux, macOS and Windows.
 - **Drop-in:** same DB-API, same connection strings; `import mssql_python` keeps working.
 

arrowsql-0.3.0/crates/arrowsql-core/Cargo.toml

@@ -0,0 +1,31 @@
+[package]
+name = "arrowsql-core"
+version = "0.3.0"
+edition = "2024"
+license = "MIT"
+description = "SQL Server over TDS into Arrow record batches: the Python-free core of arrowsql"
+publish = false
+
+[features]
+# Legacy TLS: native-tls (OpenSSL) + OpenSSL Always Encrypted crypto instead of rustls.
+native-tls = ["mssql-tds/native-tls-backend"]
+
+[dependencies]
+# Microsoft's native TDS implementation (MIT), vendored with local patches
+# (vendor/mssql-tds/VENDOR.md): a path dependency, so a git dependent of this
+# crate builds the patched copy without a [patch] section.
+mssql-tds = { path = "../../vendor/mssql-tds", default-features = false, features = ["gssapi", "rustls-tls"] }
+tokio = { version = "1", features = ["rt", "rt-multi-thread", "net", "time", "sync", "macros"] }
+parking_lot = "0.12"
+uuid = "1"
+arrow-array = { version = "60", default-features = false, features = ["ffi"] }
+arrow-buffer = "60"
+arrow-schema = { version = "60", features = ["ffi"] }
+encoding_rs = "0.8"
+async-trait = "0.1"
+log = "0.4"
+
+# Windows: in-tree Schannel TLS (msodbcsql's stack) instead of rustls, which fails the
+# handshake with SQL Server's self-generated certificate on Windows.
+[target.'cfg(windows)'.dependencies]
+mssql-tds = { path = "../../vendor/mssql-tds", default-features = false, features = ["gssapi", "rustls-tls", "tls-schannel-direct-on-windows"] }