snowflake-sql-api 0.1.2__tar.gz → 0.1.4__tar.gz

{snowflake_sql_api-0.1.2 → snowflake_sql_api-0.1.4}/CONTRIBUTING.md

@@ -34,7 +34,7 @@ All four must pass (and `pre-commit run --all-files` is clean). Use `coverage ru
 (`snowflake_sql_api.testing`), and `coverage run` starts tracing before that
 plugin is imported so import-time lines are measured correctly. The coverage gate
 is enforced (`fail_under = 89` in `pyproject.toml`); `coverage report` exits
-non-zero below it. CI runs the same checks across Python 3.9-3.13.
+non-zero below it. CI runs the same checks across Python 3.11-3.15.
 
 Checklist:
 

{snowflake_sql_api-0.1.2 → snowflake_sql_api-0.1.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: snowflake-sql-api
-Version: 0.1.2
+Version: 0.1.4
 Summary: Lightweight, pure-Python client for Snowflake's SQL API v2
 Project-URL: Homepage, https://github.com/hampsterx/snowflake-sql-api
 Project-URL: Repository, https://github.com/hampsterx/snowflake-sql-api
@@ -34,20 +34,20 @@ Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
 Classifier: Programming Language :: Python :: Implementation :: CPython
 Classifier: Topic :: Database
 Classifier: Typing :: Typed
-Requires-Python: >=3.9
+Requires-Python: >=3.11
 Requires-Dist: cryptography>=3.4
 Requires-Dist: httpx>=0.24
 Requires-Dist: pyjwt>=2.4
 Provides-Extra: dev
-Requires-Dist: black<25.10,>=24.0; extra == 'dev'
+Requires-Dist: black<26.6,>=26.5; extra == 'dev'
 Requires-Dist: build>=1.0; extra == 'dev'
 Requires-Dist: coverage[toml]>=7.0; extra == 'dev'
 Requires-Dist: mypy>=1.8; extra == 'dev'
@@ -64,10 +64,16 @@ Description-Content-Type: text/markdown
 
 # snowflake-sql-api
 
+[![CI](https://github.com/hampsterx/snowflake-sql-api/actions/workflows/ci.yml/badge.svg)](https://github.com/hampsterx/snowflake-sql-api/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/snowflake-sql-api)](https://pypi.org/project/snowflake-sql-api/)
+[![Python versions](https://img.shields.io/pypi/pyversions/snowflake-sql-api)](https://pypi.org/project/snowflake-sql-api/)
+[![License: MIT](https://img.shields.io/pypi/l/snowflake-sql-api)](LICENSE)
+[![Typed](https://img.shields.io/badge/typing-typed-blue)](https://peps.python.org/pep-0561/)
+
 A lightweight, pure-Python client for [Snowflake's SQL API v2](https://docs.snowflake.com/en/developer-guide/sql-api/index)
 (the `POST /api/v2/statements` REST endpoint).
 
-> **Status: in active development, not yet released.** The API may change before `v0.1.0`.
+> **Status: pre-1.0.** The API may still change between minor versions.
 
 ## Why
 
@@ -87,6 +93,7 @@ stays small and cold-starts quickly. Core dependencies: `httpx`, `PyJWT`,
 - Keypair (JWT) authentication, including encrypted private keys
 - Result type coercion (NUMBER/Decimal, dates/timestamps, VARIANT, and more)
 - Multi-partition result handling for large result sets
+- Lazy row streams and bounded row batches, with CSV and JSONL exports
 - Query helpers, DML, batch insert, async statement submission
 - A small CLI for ad-hoc queries
 - Optional pandas (`[pandas]`) and typed-row (`[pydantic]`) helpers, kept out of
@@ -132,6 +139,41 @@ CLI:
 snowflake-sql-api query "SELECT current_version()"
 ```
 
+Stream large results or export them without collecting every row:
+
+```python
+with SnowflakeClient.from_env() as client:
+    for batch in client.query_batches("SELECT id, amount FROM sales", batch_size=500):
+        process(batch)
+
+    with open("sales.csv", "w", encoding="utf-8", newline="") as output:
+        count = client.export_csv("SELECT id, amount FROM sales", output)
+```
+
+Streaming fetches completed results one server partition at a time. Memory scales
+with the largest decoded partition plus the application batch. See
+[streaming results and exports](docs/streaming.md) for row streams, async usage,
+serialization, and partial-output behavior.
+
+Insert a record stream in bounded requests:
+
+```python
+with SnowflakeClient.from_env() as client:
+    progress = client.insert_batches(
+        "events", ["id", "label"], ((i, f"event {i}") for i in range(2500)),
+        max_rows=1000, max_bytes=1_000_000,
+    )
+    print(progress.input_rows, progress.rows_affected)
+```
+
+`insert_many()` sends one statement; `insert_batches()` sends sequential statements
+and can partially complete. Its byte budget covers the exact JSON body. On failure,
+`BatchWriteError` reports confirmed progress, consumed input and the failed request's
+identity. Reconstruct replayable input and reconcile unknown outcomes before retrying;
+do not resume with the same advanced iterator. The async equivalent is awaited and
+accepts the same synchronous iterable, whose production can block the event loop.
+See [batch writes](docs/batch-writes.md) for limits and recovery semantics.
+
 ## Prior art and alternatives
 
 `snowflake-sql-api` is not the first REST-based Snowflake client. If it does not
@@ -146,10 +188,10 @@ fit your needs, one of these might:
 | [`rb_snowflake_client`](https://github.com/rinsed-org/rb-snowflake-client) | Ruby | Mature pure-Ruby SQL API client (streaming, connection pooling). |
 | [`rsql`](https://github.com/theseus-rs/rsql) | Rust | Multi-database SQL CLI with a Snowflake SQL API driver. |
 
-**How `snowflake-sql-api` differs:** pure Python over `httpx` with no dependency on
-the official connector (small install, fast cold start), paired sync and async
-clients, and keypair auth, type coercion, partition handling, and a CLI as tested
-first-class behavior.
+**How `snowflake-sql-api` differs:** pure Python over `httpx`, with no
+dependency on the official connector (small install, fast cold start), paired
+sync and async clients, keypair auth, type coercion, partition handling, and a
+CLI as tested first-class behavior.
 
 Note: the similarly named `snowflake-sql-api-async` and `snowflake-sql-api-client`
 are separate, unaffiliated projects.

{snowflake_sql_api-0.1.2 → snowflake_sql_api-0.1.4}/README.md

@@ -1,9 +1,15 @@
 # snowflake-sql-api
 
+[![CI](https://github.com/hampsterx/snowflake-sql-api/actions/workflows/ci.yml/badge.svg)](https://github.com/hampsterx/snowflake-sql-api/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/snowflake-sql-api)](https://pypi.org/project/snowflake-sql-api/)
+[![Python versions](https://img.shields.io/pypi/pyversions/snowflake-sql-api)](https://pypi.org/project/snowflake-sql-api/)
+[![License: MIT](https://img.shields.io/pypi/l/snowflake-sql-api)](LICENSE)
+[![Typed](https://img.shields.io/badge/typing-typed-blue)](https://peps.python.org/pep-0561/)
+
 A lightweight, pure-Python client for [Snowflake's SQL API v2](https://docs.snowflake.com/en/developer-guide/sql-api/index)
 (the `POST /api/v2/statements` REST endpoint).
 
-> **Status: in active development, not yet released.** The API may change before `v0.1.0`.
+> **Status: pre-1.0.** The API may still change between minor versions.
 
 ## Why
 
@@ -23,6 +29,7 @@ stays small and cold-starts quickly. Core dependencies: `httpx`, `PyJWT`,
 - Keypair (JWT) authentication, including encrypted private keys
 - Result type coercion (NUMBER/Decimal, dates/timestamps, VARIANT, and more)
 - Multi-partition result handling for large result sets
+- Lazy row streams and bounded row batches, with CSV and JSONL exports
 - Query helpers, DML, batch insert, async statement submission
 - A small CLI for ad-hoc queries
 - Optional pandas (`[pandas]`) and typed-row (`[pydantic]`) helpers, kept out of
@@ -68,6 +75,41 @@ CLI:
 snowflake-sql-api query "SELECT current_version()"
 ```
 
+Stream large results or export them without collecting every row:
+
+```python
+with SnowflakeClient.from_env() as client:
+    for batch in client.query_batches("SELECT id, amount FROM sales", batch_size=500):
+        process(batch)
+
+    with open("sales.csv", "w", encoding="utf-8", newline="") as output:
+        count = client.export_csv("SELECT id, amount FROM sales", output)
+```
+
+Streaming fetches completed results one server partition at a time. Memory scales
+with the largest decoded partition plus the application batch. See
+[streaming results and exports](docs/streaming.md) for row streams, async usage,
+serialization, and partial-output behavior.
+
+Insert a record stream in bounded requests:
+
+```python
+with SnowflakeClient.from_env() as client:
+    progress = client.insert_batches(
+        "events", ["id", "label"], ((i, f"event {i}") for i in range(2500)),
+        max_rows=1000, max_bytes=1_000_000,
+    )
+    print(progress.input_rows, progress.rows_affected)
+```
+
+`insert_many()` sends one statement; `insert_batches()` sends sequential statements
+and can partially complete. Its byte budget covers the exact JSON body. On failure,
+`BatchWriteError` reports confirmed progress, consumed input and the failed request's
+identity. Reconstruct replayable input and reconcile unknown outcomes before retrying;
+do not resume with the same advanced iterator. The async equivalent is awaited and
+accepts the same synchronous iterable, whose production can block the event loop.
+See [batch writes](docs/batch-writes.md) for limits and recovery semantics.
+
 ## Prior art and alternatives
 
 `snowflake-sql-api` is not the first REST-based Snowflake client. If it does not
@@ -82,10 +124,10 @@ fit your needs, one of these might:
 | [`rb_snowflake_client`](https://github.com/rinsed-org/rb-snowflake-client) | Ruby | Mature pure-Ruby SQL API client (streaming, connection pooling). |
 | [`rsql`](https://github.com/theseus-rs/rsql) | Rust | Multi-database SQL CLI with a Snowflake SQL API driver. |
 
-**How `snowflake-sql-api` differs:** pure Python over `httpx` with no dependency on
-the official connector (small install, fast cold start), paired sync and async
-clients, and keypair auth, type coercion, partition handling, and a CLI as tested
-first-class behavior.
+**How `snowflake-sql-api` differs:** pure Python over `httpx`, with no
+dependency on the official connector (small install, fast cold start), paired
+sync and async clients, keypair auth, type coercion, partition handling, and a
+CLI as tested first-class behavior.
 
 Note: the similarly named `snowflake-sql-api-async` and `snowflake-sql-api-client`
 are separate, unaffiliated projects.

{snowflake_sql_api-0.1.2 → snowflake_sql_api-0.1.4}/pyproject.toml

@@ -7,7 +7,7 @@ name = "snowflake-sql-api"
 dynamic = ["version"]
 description = "Lightweight, pure-Python client for Snowflake's SQL API v2"
 readme = "README.md"
-requires-python = ">=3.9"
+requires-python = ">=3.11"
 license = { file = "LICENSE" }
 authors = [{ name = "hampsterx" }]
 keywords = ["snowflake", "sql", "rest", "api", "httpx", "serverless", "lambda"]
@@ -17,11 +17,11 @@ classifiers = [
     "License :: OSI Approved :: MIT License",
     "Operating System :: OS Independent",
     "Programming Language :: Python :: 3",
-    "Programming Language :: Python :: 3.9",
-    "Programming Language :: Python :: 3.10",
     "Programming Language :: Python :: 3.11",
     "Programming Language :: Python :: 3.12",
     "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+    "Programming Language :: Python :: 3.15",
     "Programming Language :: Python :: Implementation :: CPython",
     "Topic :: Database",
     "Typing :: Typed",
@@ -41,12 +41,10 @@ dev = [
     "pytest-asyncio>=0.21",
     "respx>=0.20",
     "ruff>=0.4",
-    # Capped below 25.10 so CI formats with the same black the pre-commit hook
-    # pins (rev 25.9.0 in .pre-commit-config.yaml). Move the two together by
-    # hand; Dependabot ignores black (.github/dependabot.yml). Keep the cap at
-    # or below 25.12 while Python 3.9 is supported: 25.12.0+ require 3.10, so
-    # a higher cap resolves a different black per interpreter.
-    "black>=24.0,<25.10",
+    # Held to 26.5.x so CI formats with the same black the pre-commit hook
+    # pins (rev 26.5.1 in .pre-commit-config.yaml). Move the two together by
+    # hand; Dependabot ignores black (.github/dependabot.yml).
+    "black>=26.5,<26.6",
     "mypy>=1.8",
     "build>=1.0",
     "pre-commit>=3.5",
@@ -89,13 +87,13 @@ include = [
 
 [tool.ruff]
 line-length = 88
-target-version = "py39"
+target-version = "py311"
 
 [tool.ruff.lint]
 select = ["E", "F", "W", "I", "B", "C4", "SIM"]
-# E501 (line length) is owned by black. `UP` (pyupgrade) is omitted: it would
-# rewrite runtime type aliases like `Optional[Sequence[Any]]` to `X | None`,
-# which raises at import time on Python 3.9 (PEP 604 on typing generics is 3.10+).
+# E501 (line length) is owned by black. `UP` (pyupgrade) is not enabled yet: the
+# code predates the 3.11 floor and still spells `Optional`/`List`, so turning it
+# on is a ~250-site annotation rewrite of its own.
 ignore = ["E501"]
 
 [tool.ruff.lint.per-file-ignores]
@@ -103,14 +101,14 @@ ignore = ["E501"]
 
 [tool.black]
 line-length = 88
-# Pin to py39 grammar: our code uses no newer syntax, and a single floor target
-# keeps black's AST safety check parseable by the interpreter running CI.
-target-version = ["py39"]
+# Pin to the floor's grammar: a single floor target keeps black's AST safety
+# check parseable by the interpreter running CI.
+target-version = ["py311"]
 
 [tool.mypy]
-# Lowest target this mypy accepts; true 3.9 runtime compat is gated by the
-# pytest matrix (3.9-3.13), which imports every module under each interpreter.
-python_version = "3.10"
+# The runtime floor; the pytest matrix (3.11-3.15) imports every module under
+# each supported interpreter.
+python_version = "3.11"
 files = ["snowflake_sql_api"]
 disallow_untyped_defs = true
 disallow_incomplete_defs = true

{snowflake_sql_api-0.1.2 → snowflake_sql_api-0.1.4}/snowflake_sql_api/__init__.py

@@ -27,8 +27,10 @@ except ImportError:  # pragma: no cover - plain source checkout, never built
         __version__ = "0.0.0.dev0"
 
 from .aclient import AsyncQueryHandle, AsyncSnowflakeClient
+from .batches import BatchSummary, BatchWriteResult, FailedBatch
 from .client import QueryHandle, SnowflakeClient
 from .exceptions import (
+    BatchWriteError,
     ResultNotReady,
     SnowflakeAuthError,
     SnowflakeConfigError,
@@ -48,6 +50,10 @@ __all__ = [
     "QueryHandle",
     "AsyncQueryHandle",
     "RetryPolicy",
+    "BatchSummary",
+    "BatchWriteResult",
+    "FailedBatch",
+    "BatchWriteError",
     "SnowflakeError",
     "SnowflakeConfigError",
     "SnowflakeAuthError",

{snowflake_sql_api-0.1.2 → snowflake_sql_api-0.1.4}/snowflake_sql_api/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.1.2'
-__version_tuple__ = version_tuple = (0, 1, 2)
+__version__ = version = '0.1.4'
+__version_tuple__ = version_tuple = (0, 1, 4)
 
 __commit_id__ = commit_id = None

{snowflake_sql_api-0.1.2 → snowflake_sql_api-0.1.4}/snowflake_sql_api/aclient.py

@@ -12,11 +12,23 @@ import os
 import time
 import uuid
 from pathlib import Path
-from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
+from typing import (
+    Any,
+    AsyncGenerator,
+    Callable,
+    Dict,
+    Iterable,
+    List,
+    Optional,
+    Sequence,
+    TextIO,
+    Tuple,
+)
 
 import httpx
 
 from .auth import KeypairAuthenticator, account_hostname
+from .batches import BatchWriteResult, _BatchExecution, _BatchPlanner
 from .bindings import to_bindings
 from .client import DEFAULT_POLL_TIMEOUT, _columns_from, _rows_affected
 from .escaping import quote_identifier, quote_name
@@ -26,7 +38,13 @@ from .exceptions import (
     SnowflakeRequestError,
     SnowflakeTimeoutError,
 )
-from .pagination import fetch_all_partitions_async
+from .exports import (
+    CSVWriter,
+    validate_batch_size,
+    validate_export_columns,
+    write_jsonl,
+)
+from .pagination import fetch_all_partitions_async, iter_result_rows_async
 from .retry import DEFAULT_RETRY_POLICY, RetryPolicy
 from .transport import AsyncTransport, StatementResponse
 from .types import ColumnMeta, Row, coerce_rows
@@ -201,13 +219,13 @@ class AsyncSnowflakeClient:
                 )
             await asyncio.sleep(self._poll_interval)
 
-    async def _collect(
+    async def _prepare_result(
         self,
         response: StatementResponse,
         *,
         timeout: Optional[float],
         poll: bool = True,
-    ) -> Tuple[List[ColumnMeta], List[List[Any]]]:
+    ) -> Tuple[List[ColumnMeta], str, Dict[str, Any]]:
         if response.is_running:
             if not poll:
                 raise ResultNotReady(statement_handle=response.statement_handle)
@@ -220,8 +238,27 @@ class AsyncSnowflakeClient:
         handle = response.statement_handle
         if handle is None:
             raise SnowflakeRequestError("completed statement has no handle")
-        raw_rows = await fetch_all_partitions_async(self._transport, handle, body)
-        return columns, raw_rows
+        return columns, handle, body
+
+    async def _collect(
+        self,
+        response: StatementResponse,
+        *,
+        timeout: Optional[float],
+        poll: bool = True,
+    ) -> Tuple[List[ColumnMeta], List[List[Any]]]:
+        columns, handle, body = await self._prepare_result(
+            response, timeout=timeout, poll=poll
+        )
+        return columns, await fetch_all_partitions_async(self._transport, handle, body)
+
+    async def _stream_result(
+        self, sql: str, params: Params
+    ) -> Tuple[List[ColumnMeta], AsyncGenerator[Row, None]]:
+        columns, handle, body = await self._prepare_result(
+            await self._run(sql, params), timeout=self._statement_timeout
+        )
+        return columns, iter_result_rows_async(self._transport, handle, body, columns)
 
     # -- query surface ----------------------------------------------------
 
@@ -233,6 +270,73 @@ class AsyncSnowflakeClient:
         )
         return coerce_rows(raw_rows, columns)
 
+    async def query_stream(
+        self, sql: str, params: Params = None
+    ) -> AsyncGenerator[Row, None]:
+        """Yield converted rows on demand; call directly with async for.
+
+        Submission starts on first iteration. Keep the client open while
+        consuming; await aclose() on a retained stream when stopping early.
+        """
+        _, rows = await self._stream_result(sql, params)
+        try:
+            async for row in rows:
+                yield row
+        finally:
+            await rows.aclose()
+
+    async def query_batches(
+        self, sql: str, params: Params = None, *, batch_size: int = 1000
+    ) -> AsyncGenerator[List[Row], None]:
+        """Yield batches of at most batch_size rows across partition boundaries."""
+        validate_batch_size(batch_size)
+        rows = self.query_stream(sql, params)
+        try:
+            batch: List[Row] = []
+            async for row in rows:
+                batch.append(row)
+                if len(batch) == batch_size:
+                    yield batch
+                    batch = []
+            if batch:
+                yield batch
+        finally:
+            await rows.aclose()
+
+    async def export_csv(self, sql: str, output: TextIO, params: Params = None) -> int:
+        """Await query I/O and write CSV to a caller-owned synchronous text sink.
+
+        Return the data-row count. Failures may leave partial output.
+        """
+        columns, rows = await self._stream_result(sql, params)
+        try:
+            writer = CSVWriter(output, columns)
+            count = 0
+            async for row in rows:
+                writer.write(row)
+                count += 1
+        finally:
+            await rows.aclose()
+        return count
+
+    async def export_jsonl(
+        self, sql: str, output: TextIO, params: Params = None
+    ) -> int:
+        """Await query I/O and write JSONL to a caller-owned synchronous text sink.
+
+        Return the row count. Failures may leave partial output.
+        """
+        columns, rows = await self._stream_result(sql, params)
+        try:
+            validate_export_columns(columns)
+            count = 0
+            async for row in rows:
+                write_jsonl(output, row)
+                count += 1
+        finally:
+            await rows.aclose()
+        return count
+
     async def query_one(self, sql: str, params: Params = None) -> Optional[Row]:
         """Return the first row, or ``None`` if the result is empty."""
         rows = await self.query(sql, params)
@@ -286,6 +390,52 @@ class AsyncSnowflakeClient:
 
     # -- async submit -----------------------------------------------------
 
+    async def insert_batches(
+        self,
+        table: str,
+        columns: Sequence[str],
+        rows: Iterable[Sequence[Any]],
+        *,
+        max_rows: int = 1000,
+        max_bytes: int = 1_000_000,
+    ) -> BatchWriteResult:
+        """Insert a single-pass source sequentially within row and wire-byte budgets.
+
+        Independent statements can commit before a later failure. BatchWriteError
+        reports confirmed progress and uncertain outcomes; see docs/batch-writes.md.
+        Input is an ordinary iterable, consumed on the calling thread.
+        """
+        planner = _BatchPlanner(
+            table, columns, rows, max_rows, max_bytes, self._build_payload
+        )
+        while (batch := planner.next_batch()) is not None:
+            try:
+                self._notify(batch.sql, batch.params)
+            except Exception as exc:
+                raise planner.error(exc, "validation") from exc
+            execution = _BatchExecution(planner, batch)
+            try:
+                response = await self._transport.submit(
+                    {}, request_id=execution.failed.request_id, content=batch.content
+                )
+                execution.received(
+                    response.statement_handle,
+                    response.submission_attempts,
+                    response.is_running,
+                )
+                if response.is_running:
+                    handle = response.statement_handle
+                    if handle is None:
+                        raise SnowflakeRequestError("running statement has no handle")
+                    response = await self._poll(handle, self._statement_timeout)
+                    execution.succeeded = True
+                affected = _rows_affected(response.body, strict=True)
+            except Exception as exc:
+                raise execution.error(exc) from exc
+            planner.confirm(batch, affected)
+            del batch
+        return planner.result()
+
     async def submit(self, sql: str, params: Params = None) -> AsyncQueryHandle:
         """Submit a long-running statement and return an :class:`AsyncQueryHandle`."""
         response = await self._run(sql, params, async_exec=True)