vgi-python 0.41.0__tar.gz → 0.42.1__tar.gz

{vgi_python-0.41.0 → vgi_python-0.42.1}/CLAUDE.md

@@ -297,6 +297,7 @@ vgi-client --input data.parquet --function sum_all_columns --worker vgi-fixture-
 | `VGI_OAUTH_DEVICE_CODE_CLIENT_ID` | Client ID for device-code flow (optional, URL-safe chars only) |
 | `VGI_OAUTH_DEVICE_CODE_CLIENT_SECRET` | Client secret for device-code flow (optional, URL-safe chars only) |
 | `VGI_OAUTH_USE_ID_TOKEN` | When `1`/`true`/`yes`, clients use OIDC `id_token` as Bearer instead of `access_token` |
+| `VGI_RPC_GRANT_KEYS` | Comma-separated base64 32-byte sealed-grant keys (first mints, all verify); enables grants as HTTP bearers. Also `VGI_RPC_GRANT_AUDIENCE`, `VGI_RPC_GRANT_MAX_TTL_SECONDS` |
 | `VGI_INTROSPECT_PRINCIPALS` | Comma-separated principals permitted to call `vgi_rpc.Identity.v1`'s `introspect_token`. Required — no permissive default — whenever the worker implements `resolve_token()` (see below) |
 | `VGI_WORKER_ACCESS_LOG_SAMPLE` | Fraction of *successful* calls to keep in the access log, `0.0`–`1.0`. Errors are always kept; the decision is per call, so every record of one stream shares a fate |
 | `VGI_WORKER_ACCESS_LOG_ASYNC` | When `1`/`true`/`yes`, emit access-log records from a listener thread. Bounded queue; full means drop, and a crash loses whatever is queued |
@@ -484,6 +485,24 @@ sidecar outage raised as one is read as "not my credential, try the next" and
 ends up a 401 from the end of the chain, restarting every session in the fleet
 over a thirty-second blip.
 
+### Grants and `resolve_token` as HTTP Bearers
+
+With `VGI_RPC_GRANT_KEYS` (or `vgi-serve --grant-key`) set, the HTTP worker
+hosts Identity even without hooks. It mints sealed `vgig1.` grants through
+`issue_grant`, unless the worker overrides `mint_grant`, and accepts them back
+as `Authorization: Bearer` credentials, authenticated as the grant's owner. A
+worker that overrides `resolve_token` also has it consulted for bearers. The
+order is the deployment's authenticator, then grants, then `resolve_token`. A
+bad `vgig1.` token is a 401 and never reaches `resolve_token`. vgi-rpc composes
+this in `make_wsgi_app`. vgi-python resolves the keys in
+`vgi.rpc_server.resolve_grant_keys` and passes them explicitly to `RpcServer`
+(`None` off HTTP, never vgi-rpc's `"env"` default), so grants stay HTTP-only. It
+also puts the bearers *inside* VGI's proxy-proof gate
+(`serve._compose_identity_bearer`), because vgi-rpc refuses to OR them beside
+one. Allowlist rules are unchanged. See `docs/authentication.md` and
+`tests/test_grant_auth.py`, which also runs the shared
+`grant_token_vectors.json` through `create_app`.
+
 ### Hosting Additional Protocols (`hosted_protocols`)
 
 Every transport (stdio, unix/named pipe, TCP, Iroh, HTTP) builds its `RpcServer`

{vgi_python-0.41.0 → vgi_python-0.42.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: vgi-python
-Version: 0.41.0
+Version: 0.42.1
 Summary: Vector Gateway Interface - Connect DuckDB to external programs via Apache Arrow
 Project-URL: Homepage, https://query.farm
 Project-URL: Repository, https://github.com/Query-farm/vgi-python
@@ -162,7 +162,7 @@ Requires-Dist: httpx2>=2.12.0
 Requires-Dist: platformdirs
 Requires-Dist: pyarrow
 Requires-Dist: typer>=0.9
-Requires-Dist: vgi-rpc>=0.48.0
+Requires-Dist: vgi-rpc>=0.49.0
 Provides-Extra: azure
 Requires-Dist: azure-identity>=1.16.0; extra == 'azure'
 Requires-Dist: pymssql>=2.3.0; extra == 'azure'

{vgi_python-0.41.0 → vgi_python-0.42.1}/docs/api/index.md

@@ -32,7 +32,7 @@ added incrementally.
 | [Auth & Secrets](auth.md) | `AuthContext`, `CallContext`, bearer/JWT authenticators, secret protocol | HTTP: `[http]`; JWT: `[oauth]` |
 | [Observability](observability.md) | OpenTelemetry tracing, worker logging configuration | `[otel]` for tracing |
 | [HTTP](http.md) | Worker page, blob storage, request-size middleware | `pip install vgi-python[http]` |
-| [Transactor](transactor.md) | `TransactorClient`, `TransactorProtocol` | `pip install vgi-python[transactor]` |
+| [Transactor](transactor.md) | `TransactorClient`, `TransactorProtocol` | `[transactor]` extra and a [compatible engine build](transactor.md#runtime-requirements) |
 | [Exceptions](exceptions.md) | VGI exception types | — |
 
 ## Import Convention

vgi_python-0.42.1/docs/api/transactor.md

@@ -0,0 +1,32 @@
+# Transactor
+
+!!! note "Advanced — reference only"
+    The transactor is an advanced feature without a dedicated how-to guide yet. This page is the
+    API reference; start from the [tutorial](../tutorial/index.md) and
+    [function patterns](../how-to/function-patterns.md) if you're new to VGI.
+
+The transactor is a long-lived subprocess that gives worker functions transactional access to a
+database, mediated by `TransactorClient` over the `TransactorProtocol`.
+
+## Runtime requirements
+
+The bundled transactor requires both:
+
+- `pip install 'vgi-python[transactor]'` for its Python dependencies.
+- A VGI fork build of `duckdb-python` providing `DuckDBPyConnection.subcursor()`,
+  which lets reads share an open write transaction. This extension has not yet
+  been merged into Haybarn or upstream DuckDB; installing the extra alone is
+  insufficient.
+
+Engine resolution prefers `haybarn` when installed, then falls back to `duckdb`.
+Use an environment where the selected engine provides `subcursor()`; installing
+a compatible `duckdb` build alongside an incompatible `haybarn` installation
+does not override that preference.
+
+These requirements apply to the bundled transactor. Workers can implement
+[table writes](../catalog-interface.md#table-writes-and-indexes) against their own
+backing data sources without using it.
+
+## API
+
+::: vgi.transactor

{vgi_python-0.41.0 → vgi_python-0.42.1}/docs/authentication.md

@@ -24,7 +24,7 @@ the principal in the `AuthContext` available to functions.
 | Variable | Description |
 |----------|-------------|
 | `VGI_BEARER_TOKENS` | Comma-separated `token=principal` pairs for static bearer auth |
-| `VGI_JWT_ISSUER` | JWT issuer URL (requires `vgi[oauth]` extra) |
+| `VGI_JWT_ISSUER` | JWT issuer URL (requires `vgi-python[oauth]` extra) |
 | `VGI_JWT_AUDIENCE` | JWT audience string, comma-separated for multiple audiences (required when `VGI_JWT_ISSUER` is set) |
 | `VGI_JWT_JWKS_URI` | JWKS endpoint URL (auto-discovered from issuer if omitted) |
 | `VGI_OAUTH_RESOURCE` | OAuth resource URL for RFC 9728 metadata |
@@ -39,6 +39,48 @@ When both `VGI_BEARER_TOKENS` and `VGI_JWT_ISSUER` are set, they are
 chained — JWT validation is attempted first, falling back to bearer token
 lookup.
 
+## Grants and resolved tokens as bearer credentials
+
+`vgi_rpc.Identity.v1`'s `issue_grant` mints a standing credential that
+unattended automation later presents as an ordinary bearer. Over HTTP the
+worker accepts two kinds of identity credential after the deployment's own
+authenticator:
+
+1. **Sealed grants** (opt-in). Set `VGI_RPC_GRANT_KEYS` (or
+   `vgi-serve --http --grant-key KEY`, repeatable). `issue_grant` then mints a
+   `vgig1.` grant sealed with the first key, unless the worker overrides
+   `mint_grant`. Every key verifies, so to rotate you add the new key first and
+   drop the old one once its grants have expired. A request carrying
+   `Authorization: Bearer vgig1.…` is authenticated as the grant's owner, with
+   `domain="grant"` and claims `grant_id`, `scopes` and `purpose`. The claims
+   carry no `auth_time`, so a grant cannot mint another grant.
+2. **`resolve_token`**. A worker that overrides
+   [`Worker.resolve_token`][vgi.worker.Worker.resolve_token] has it consulted
+   for any other bearer, with `domain="token"`. `None` means the credential is
+   unknown, which is a 401 unless something else accepts it.
+   `AuthUnavailableError` is a 503 with the hook's `Retry-After`.
+
+The order is the deployment's authenticator (JWT, static bearer), then sealed
+grants, then `resolve_token`. A `vgig1.` token that does not verify is a 401
+that stops the chain: it never reaches `resolve_token`. A JWS-shaped token or
+one over 4096 bytes never reaches the hook either. Your own `authenticate`
+callback must raise `ValueError` for a credential it does not recognise, so the
+chain moves on. One that answers anonymous for every request ends the chain
+before grants are checked. With no authenticator configured, a request with no
+`Authorization` header stays anonymous as before.
+
+| Variable | Description |
+|----------|-------------|
+| `VGI_RPC_GRANT_KEYS` | Comma-separated standard-base64 keys, 32 bytes each. The first mints and all verify. Unset means grants are off. A malformed key stops startup |
+| `VGI_RPC_GRANT_AUDIENCE` | Bound into every grant, so two deployments that share a key still reject each other's grants |
+| `VGI_RPC_GRANT_MAX_TTL_SECONDS` | Ceiling on a grant's lifetime (default 7 days) |
+
+The startup rules are unchanged. Overriding `resolve_token` still requires
+`VGI_INTROSPECT_PRINCIPALS`. Grant keys alone need no allowlist, because
+minting answers only for the caller. Grants and `resolve_token` bearers are
+HTTP-only, like the rest of Identity. When `VGI_PROXY_PROOF_MODE` is on, both
+are accepted only *inside* the proxy-proof gate, never as an alternative to it.
+
 ## Programmatic API
 
 ```python test="skip"

{vgi_python-0.41.0 → vgi_python-0.42.1}/docs/catalog-interface.md

@@ -944,7 +944,7 @@ if __name__ == "__main__":
 
 ---
 
-## API Limitations
+## Catalog capabilities and limitations { #api-limitations }
 
 The current CatalogInterface has the following limitations:
 
@@ -952,8 +952,33 @@ The current CatalogInterface has the following limitations:
 - **Tags**: Cannot be updated after object creation
 - **Schema metadata**: Comments and tags cannot be updated on schemas
 - **Constraints**: Only NOT NULL can be added/dropped (no ALTER for UNIQUE/CHECK)
-- **Indexes**: Not supported
-- **INSERT/UPDATE/DELETE**: Not yet implemented (metadata only)
+
+### Table writes and indexes
+
+`INSERT`, `UPDATE`, and `DELETE` are supported when the catalog supplies a write
+function for the operation. For declarative catalogs, configure the
+[`Table`][vgi.catalog.descriptors.Table] descriptor:
+
+| Operation | Descriptor field | Requirement |
+|-----------|------------------|-------------|
+| `INSERT` | `insert_function` | A `TableInOutGenerator` that accepts rows to insert |
+| `UPDATE` | `update_function` | A write function and a scan `function` that provides row IDs |
+| `DELETE` | `delete_function` | A write function and a scan `function` that provides row IDs |
+
+Each field defaults to `None`, which leaves that operation unsupported. Custom
+catalogs implement `table_insert_function_get()`, `table_update_function_get()`,
+or `table_delete_function_get()` to return the corresponding `ScanFunctionResult`.
+The worker's write functions implement the changes to the backing data source;
+declaring a table alone does not make it writable.
+
+`ReadOnlyCatalogInterface` rejects catalog DDL, but can expose these write functions
+on declarative tables. Table writes do not require the bundled
+[transactor](api/transactor.md); that is an optional database-access service with
+additional engine requirements.
+
+Index metadata can be declared with [`Index`][vgi.catalog.descriptors.Index]. Custom
+catalogs can implement `index_get()`, `index_create()`, and `index_drop()`; support
+depends on the catalog implementation and backing data source.
 
 ---
 

{vgi_python-0.41.0 → vgi_python-0.42.1}/docs/cli.md

@@ -545,7 +545,7 @@ vgi-fixture-worker --log-format json
 | `VGI_QUIET=1` | Suppress the interactive-terminal startup warning (same as `--quiet`) |
 | `VGI_FILTER_DEBUG=1` | Enable filter pushdown debug logging |
 | `VGI_BEARER_TOKENS` | Comma-separated `token=principal` pairs for static bearer auth (HTTP only) |
-| `VGI_JWT_ISSUER` | JWT issuer URL for JWT/JWKS auth (requires `vgi[oauth]` extra) |
+| `VGI_JWT_ISSUER` | JWT issuer URL for JWT/JWKS auth (requires `vgi-python[oauth]` extra) |
 | `VGI_JWT_AUDIENCE` | JWT audience string, comma-separated for multiple audiences (required when `VGI_JWT_ISSUER` is set) |
 | `VGI_JWT_JWKS_URI` | JWKS endpoint URL (auto-discovered if omitted) |
 | `VGI_OAUTH_RESOURCE` | OAuth resource URL for RFC 9728 metadata |

{vgi_python-0.41.0 → vgi_python-0.42.1}/docs/how-to/function-patterns.md

@@ -1,5 +1,5 @@
 ---
-description: "How to write each of the four VGI function patterns: scalar, table, table-in-out, and aggregate — with a complete runnable worker for each."
+description: "How to write each of the five VGI function patterns: scalar, table, table-in-out, aggregate, and buffering — with a complete runnable worker for each."
 ---
 
 # Function patterns
@@ -178,11 +178,14 @@ ATTACH 'aggregates' (TYPE vgi, LOCATION 'uv run sum_worker.py');
 SELECT category, aggregates.vgi_sum(value) FROM t GROUP BY category;
 ```
 
-??? info "State must be an `ArrowSerializableDataclass`"
-    Table generators and aggregates keep state between calls. Because that state is serialized
-    across parallel workers (aggregates) and HTTP round-trips, the framework requires it to extend
-    `ArrowSerializableDataclass`. Annotate fields with `ArrowType(...)` so the wire type is
-    explicit.
+??? info "State must be serializable"
+    Table generators and aggregates keep state between calls. A state type must provide
+    `serialize_to_bytes()` and `deserialize_from_bytes()` so the framework can carry it across
+    HTTP round-trips or persist it between aggregate calls. `ArrowSerializableDataclass`
+    implements these methods for you and is the usual starting point. Custom state types can
+    implement the same [`StreamStateCodec`][vgi.function.StreamStateCodec] contract without
+    inheriting from `ArrowSerializableDataclass`. See
+    [State Class](../aggregate-functions.md#state-class) for both approaches.
 
 ## Buffering
 

{vgi_python-0.41.0 → vgi_python-0.42.1}/docs/how-to/state-storage.md

@@ -22,9 +22,11 @@ function that coordinates partial results across workers.
 
 ## Two kinds of "state" — don't confuse them
 
-- **Generator cursor state** — the small `ArrowSerializableDataclass` a table generator keeps
-  *within one scan* (see [streaming with state](function-patterns.md#streaming-with-state)). It
-  lives in the worker for the duration of the call.
+- **Generator cursor state** — the small serializable object a table generator keeps
+  *within one scan* (see [streaming with state](function-patterns.md#streaming-with-state)).
+  Over HTTP, the framework serializes it between requests. `ArrowSerializableDataclass`
+  supplies the serialization methods; custom types can implement
+  [`StreamStateCodec`][vgi.function.StreamStateCodec] instead.
 - **Shared storage** (this page) — state that must outlive a single call or be shared across
   **separate worker processes**, e.g. combining partial aggregate results. This is backed by a
   pluggable store.

{vgi_python-0.41.0 → vgi_python-0.42.1}/docs/shared-storage.md

@@ -10,7 +10,7 @@ variable.
 | Backend | Value | Use Case | Dependencies |
 |---------|-------|----------|-------------|
 | SQLite | `sqlite` (default) | Local / subprocess transport | None (stdlib) |
-| Azure SQL | `azure-sql` | Azure cloud deployments | `vgi[azure]` |
+| Azure SQL | `azure-sql` | Azure cloud deployments | `vgi-python[azure]` |
 | Cloudflare DO | `cloudflare-do` | Edge / multi-cloud deployments | None (stdlib) |
 
 ## SQLite (Default)
@@ -36,7 +36,7 @@ workers run on separate hosts.
 1. Install the Azure extra:
 
 ```bash
-pip install vgi[azure]
+pip install 'vgi-python[azure]'
 ```
 
 2. Create an Azure SQL Database (Serverless recommended for cost):

{vgi_python-0.41.0 → vgi_python-0.42.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "vgi-python"
-version = "0.41.0"
+version = "0.42.1"
 description = "Vector Gateway Interface - Connect DuckDB to external programs via Apache Arrow"
 readme = "README.md"
 keywords = [
@@ -40,7 +40,7 @@ dependencies = [
     "pyarrow",
     "typer>=0.9",
     "platformdirs",
-    "vgi-rpc>=0.48.0",
+    "vgi-rpc>=0.49.0",
     "httpx2>=2.12.0",
 ]
 

{vgi_python-0.41.0 → vgi_python-0.42.1}/tests/test_bind_request_at_clause.py

@@ -13,10 +13,14 @@ Python protocol contract (round-trip + additive wire compatibility + accessors).
 
 from __future__ import annotations
 
+from types import SimpleNamespace
+
 import pyarrow as pa
 import pyarrow.ipc as ipc
+import pytest
 
 from vgi.arguments import Arguments
+from vgi.client.client import Client
 from vgi.invocation import FunctionType
 from vgi.protocol import BindRequest, InitRequest
 from vgi.table_function import BindParams, ProcessParams, ResolvedSecrets
@@ -108,3 +112,31 @@ def test_process_params_accessor_none_when_no_init_call() -> None:
     )
     assert params.at_unit is None
     assert params.at_value is None
+
+
+def test_client_bind_forwards_at_clause_and_secrets(monkeypatch: pytest.MonkeyPatch) -> None:
+    """``Client.bind`` carries at_unit/at_value and secrets onto its BindRequest, as table_function does."""
+    captured: list[BindRequest] = []
+    client = Client("unused-worker")
+    client._primary = SimpleNamespace(proxy=None)  # type: ignore[assignment]
+
+    def fake_do_bind(_proxy: object, request: BindRequest, _callback: object) -> object:
+        captured.append(request)
+        return SimpleNamespace()
+
+    monkeypatch.setattr(client, "_do_bind", fake_do_bind)
+    client.bind(
+        function_name="sequence",
+        schema_path=["main"],
+        at_unit="VERSION",
+        at_value="3",
+        secrets={"token": "s3cr3t"},
+    )
+    (request,) = captured
+    assert (request.at_unit, request.at_value) == ("VERSION", "3")
+    assert request.secrets is not None
+
+    captured.clear()
+    client.bind(function_name="sequence", schema_path=["main"])
+    (request,) = captured
+    assert (request.at_unit, request.at_value, request.secrets) == (None, None, None)

{vgi_python-0.41.0 → vgi_python-0.42.1}/tests/test_filter_v2.py

@@ -22,6 +22,7 @@ from vgi import (
     FilterVersionError,
     RuntimeFilterAlgorithmCapability,
     deserialize_filters,
+    filter_v2,
 )
 
 _METADATA: dict[bytes, bytes] = {
@@ -767,3 +768,36 @@ def test_standard_v1_evaluator_requires_oracle_engine(
         with pytest.raises(filter_v2.FilterV2Error, match=f"found {engine} {re.escape(version)}"):
             filter_v2._get_evaluation_connection(context)
         assert opened == []
+
+
+class _NoNbytes:
+    """Stands in for a view-typed batch under pyarrow < 19, where ``nbytes`` raises."""
+
+    def __init__(self, total: int | None) -> None:
+        self._total = total
+
+    @property
+    def nbytes(self) -> int:
+        raise pa.ArrowTypeError("not implemented for string_view")
+
+    def get_total_buffer_size(self) -> int:
+        if self._total is None:
+            raise pa.ArrowNotImplementedError("not implemented")
+        return self._total
+
+
+def test_payload_size_falls_back_when_nbytes_is_unavailable() -> None:
+    """A view-typed payload is measured by total buffer size instead of raising ArrowTypeError."""
+    assert filter_v2._payload_nbytes(_NoNbytes(123)) == 123  # type: ignore[arg-type]
+
+
+def test_payload_size_unmeasurable_is_a_filter_error() -> None:
+    """When no measurement works, the caller sees FilterV2Error, not a raw Arrow error."""
+    with pytest.raises(filter_v2.FilterV2Error, match="cannot measure filter payload size"):
+        filter_v2._payload_nbytes(_NoNbytes(None))  # type: ignore[arg-type]
+
+
+def test_payload_size_of_a_view_typed_batch() -> None:
+    """A real string_view payload is measured without error."""
+    batch = pa.record_batch([pa.array(["a", "bc"], type=pa.string_view())], names=["needles"])
+    assert filter_v2._payload_nbytes(batch) > 0