vgi-python 0.40.0__tar.gz → 0.42.0__tar.gz

{vgi_python-0.40.0 → vgi_python-0.42.0}/CLAUDE.md

@@ -297,6 +297,7 @@ vgi-client --input data.parquet --function sum_all_columns --worker vgi-fixture-
 | `VGI_OAUTH_DEVICE_CODE_CLIENT_ID` | Client ID for device-code flow (optional, URL-safe chars only) |
 | `VGI_OAUTH_DEVICE_CODE_CLIENT_SECRET` | Client secret for device-code flow (optional, URL-safe chars only) |
 | `VGI_OAUTH_USE_ID_TOKEN` | When `1`/`true`/`yes`, clients use OIDC `id_token` as Bearer instead of `access_token` |
+| `VGI_RPC_GRANT_KEYS` | Comma-separated base64 32-byte sealed-grant keys (first mints, all verify); enables grants as HTTP bearers. Also `VGI_RPC_GRANT_AUDIENCE`, `VGI_RPC_GRANT_MAX_TTL_SECONDS` |
 | `VGI_INTROSPECT_PRINCIPALS` | Comma-separated principals permitted to call `vgi_rpc.Identity.v1`'s `introspect_token`. Required — no permissive default — whenever the worker implements `resolve_token()` (see below) |
 | `VGI_WORKER_ACCESS_LOG_SAMPLE` | Fraction of *successful* calls to keep in the access log, `0.0`–`1.0`. Errors are always kept; the decision is per call, so every record of one stream shares a fate |
 | `VGI_WORKER_ACCESS_LOG_ASYNC` | When `1`/`true`/`yes`, emit access-log records from a listener thread. Bounded queue; full means drop, and a crash loses whatever is queued |
@@ -484,6 +485,24 @@ sidecar outage raised as one is read as "not my credential, try the next" and
 ends up a 401 from the end of the chain, restarting every session in the fleet
 over a thirty-second blip.
 
+### Grants and `resolve_token` as HTTP Bearers
+
+With `VGI_RPC_GRANT_KEYS` (or `vgi-serve --grant-key`) set, the HTTP worker
+hosts Identity even without hooks. It mints sealed `vgig1.` grants through
+`issue_grant`, unless the worker overrides `mint_grant`, and accepts them back
+as `Authorization: Bearer` credentials, authenticated as the grant's owner. A
+worker that overrides `resolve_token` also has it consulted for bearers. The
+order is the deployment's authenticator, then grants, then `resolve_token`. A
+bad `vgig1.` token is a 401 and never reaches `resolve_token`. vgi-rpc composes
+this in `make_wsgi_app`. vgi-python resolves the keys in
+`vgi.rpc_server.resolve_grant_keys` and passes them explicitly to `RpcServer`
+(`None` off HTTP, never vgi-rpc's `"env"` default), so grants stay HTTP-only. It
+also puts the bearers *inside* VGI's proxy-proof gate
+(`serve._compose_identity_bearer`), because vgi-rpc refuses to OR them beside
+one. Allowlist rules are unchanged. See `docs/authentication.md` and
+`tests/test_grant_auth.py`, which also runs the shared
+`grant_token_vectors.json` through `create_app`.
+
 ### Hosting Additional Protocols (`hosted_protocols`)
 
 Every transport (stdio, unix/named pipe, TCP, Iroh, HTTP) builds its `RpcServer`
@@ -643,7 +662,14 @@ The worker validates paths, orders parents first and enforces the etag rules
 `catalog_contents_attach_independent` (with `catalog_version_frozen`, the worker
 caches the serialized response per catalog version; `ReadOnlyCatalogInterface`
 sets it). Fixture catalogs: `vgi/_test_fixtures/catalog_contents.py`
-(`contents_probe/broken/legacy/memory/reval/hash`). See `docs/catalog-interface.md`.
+(`contents_probe/broken/legacy/memory/reval/hash`). Every SDK's fixture worker
+serves the same six catalogs: the extension's five `catalog_contents*.test` files
+run against all of them (no `VGI_CATALOG_CONTENTS_WORKER` gate any more), so these
+semantics are the cross-SDK contract. Client side, `Client.load_catalog(attach=,
+previous=)` is the whole-catalog enumeration: `catalog_contents` only when
+advertised, per-schema fallback on failure / in a transaction, `if_none_match`
+revalidation via `previous` (`tests/catalog/test_client_load_catalog.py`). See
+`docs/catalog-interface.md`.
 
 ## Additional Documentation
 

{vgi_python-0.40.0 → vgi_python-0.42.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: vgi-python
-Version: 0.40.0
+Version: 0.42.0
 Summary: Vector Gateway Interface - Connect DuckDB to external programs via Apache Arrow
 Project-URL: Homepage, https://query.farm
 Project-URL: Repository, https://github.com/Query-farm/vgi-python
@@ -162,7 +162,7 @@ Requires-Dist: httpx2>=2.12.0
 Requires-Dist: platformdirs
 Requires-Dist: pyarrow
 Requires-Dist: typer>=0.9
-Requires-Dist: vgi-rpc>=0.48.0
+Requires-Dist: vgi-rpc>=0.49.0
 Provides-Extra: azure
 Requires-Dist: azure-identity>=1.16.0; extra == 'azure'
 Requires-Dist: pymssql>=2.3.0; extra == 'azure'

{vgi_python-0.40.0 → vgi_python-0.42.0}/docs/api/index.md

@@ -32,7 +32,7 @@ added incrementally.
 | [Auth & Secrets](auth.md) | `AuthContext`, `CallContext`, bearer/JWT authenticators, secret protocol | HTTP: `[http]`; JWT: `[oauth]` |
 | [Observability](observability.md) | OpenTelemetry tracing, worker logging configuration | `[otel]` for tracing |
 | [HTTP](http.md) | Worker page, blob storage, request-size middleware | `pip install vgi-python[http]` |
-| [Transactor](transactor.md) | `TransactorClient`, `TransactorProtocol` | `pip install vgi-python[transactor]` |
+| [Transactor](transactor.md) | `TransactorClient`, `TransactorProtocol` | `[transactor]` extra and a [compatible engine build](transactor.md#runtime-requirements) |
 | [Exceptions](exceptions.md) | VGI exception types | — |
 
 ## Import Convention

vgi_python-0.42.0/docs/api/transactor.md

@@ -0,0 +1,32 @@
+# Transactor
+
+!!! note "Advanced — reference only"
+    The transactor is an advanced feature without a dedicated how-to guide yet. This page is the
+    API reference; start from the [tutorial](../tutorial/index.md) and
+    [function patterns](../how-to/function-patterns.md) if you're new to VGI.
+
+The transactor is a long-lived subprocess that gives worker functions transactional access to a
+database, mediated by `TransactorClient` over the `TransactorProtocol`.
+
+## Runtime requirements
+
+The bundled transactor requires both:
+
+- `pip install 'vgi-python[transactor]'` for its Python dependencies.
+- A VGI fork build of `duckdb-python` providing `DuckDBPyConnection.subcursor()`,
+  which lets reads share an open write transaction. This extension has not yet
+  been merged into Haybarn or upstream DuckDB; installing the extra alone is
+  insufficient.
+
+Engine resolution prefers `haybarn` when installed, then falls back to `duckdb`.
+Use an environment where the selected engine provides `subcursor()`; installing
+a compatible `duckdb` build alongside an incompatible `haybarn` installation
+does not override that preference.
+
+These requirements apply to the bundled transactor. Workers can implement
+[table writes](../catalog-interface.md#table-writes-and-indexes) against their own
+backing data sources without using it.
+
+## API
+
+::: vgi.transactor

{vgi_python-0.40.0 → vgi_python-0.42.0}/docs/authentication.md

@@ -24,7 +24,7 @@ the principal in the `AuthContext` available to functions.
 | Variable | Description |
 |----------|-------------|
 | `VGI_BEARER_TOKENS` | Comma-separated `token=principal` pairs for static bearer auth |
-| `VGI_JWT_ISSUER` | JWT issuer URL (requires `vgi[oauth]` extra) |
+| `VGI_JWT_ISSUER` | JWT issuer URL (requires `vgi-python[oauth]` extra) |
 | `VGI_JWT_AUDIENCE` | JWT audience string, comma-separated for multiple audiences (required when `VGI_JWT_ISSUER` is set) |
 | `VGI_JWT_JWKS_URI` | JWKS endpoint URL (auto-discovered from issuer if omitted) |
 | `VGI_OAUTH_RESOURCE` | OAuth resource URL for RFC 9728 metadata |
@@ -39,6 +39,48 @@ When both `VGI_BEARER_TOKENS` and `VGI_JWT_ISSUER` are set, they are
 chained — JWT validation is attempted first, falling back to bearer token
 lookup.
 
+## Grants and resolved tokens as bearer credentials
+
+`vgi_rpc.Identity.v1`'s `issue_grant` mints a standing credential that
+unattended automation later presents as an ordinary bearer. Over HTTP the
+worker accepts two kinds of identity credential after the deployment's own
+authenticator:
+
+1. **Sealed grants** (opt-in). Set `VGI_RPC_GRANT_KEYS` (or
+   `vgi-serve --http --grant-key KEY`, repeatable). `issue_grant` then mints a
+   `vgig1.` grant sealed with the first key, unless the worker overrides
+   `mint_grant`. Every key verifies, so to rotate you add the new key first and
+   drop the old one once its grants have expired. A request carrying
+   `Authorization: Bearer vgig1.…` is authenticated as the grant's owner, with
+   `domain="grant"` and claims `grant_id`, `scopes` and `purpose`. The claims
+   carry no `auth_time`, so a grant cannot mint another grant.
+2. **`resolve_token`**. A worker that overrides
+   [`Worker.resolve_token`][vgi.worker.Worker.resolve_token] has it consulted
+   for any other bearer, with `domain="token"`. `None` means the credential is
+   unknown, which is a 401 unless something else accepts it.
+   `AuthUnavailableError` is a 503 with the hook's `Retry-After`.
+
+The order is the deployment's authenticator (JWT, static bearer), then sealed
+grants, then `resolve_token`. A `vgig1.` token that does not verify is a 401
+that stops the chain: it never reaches `resolve_token`. A JWS-shaped token or
+one over 4096 bytes never reaches the hook either. Your own `authenticate`
+callback must raise `ValueError` for a credential it does not recognise, so the
+chain moves on. One that answers anonymous for every request ends the chain
+before grants are checked. With no authenticator configured, a request with no
+`Authorization` header stays anonymous as before.
+
+| Variable | Description |
+|----------|-------------|
+| `VGI_RPC_GRANT_KEYS` | Comma-separated standard-base64 keys, 32 bytes each. The first mints and all verify. Unset means grants are off. A malformed key stops startup |
+| `VGI_RPC_GRANT_AUDIENCE` | Bound into every grant, so two deployments that share a key still reject each other's grants |
+| `VGI_RPC_GRANT_MAX_TTL_SECONDS` | Ceiling on a grant's lifetime (default 7 days) |
+
+The startup rules are unchanged. Overriding `resolve_token` still requires
+`VGI_INTROSPECT_PRINCIPALS`. Grant keys alone need no allowlist, because
+minting answers only for the caller. Grants and `resolve_token` bearers are
+HTTP-only, like the rest of Identity. When `VGI_PROXY_PROOF_MODE` is on, both
+are accepted only *inside* the proxy-proof gate, never as an alternative to it.
+
 ## Programmatic API
 
 ```python test="skip"

{vgi_python-0.40.0 → vgi_python-0.42.0}/docs/catalog-interface.md

@@ -697,6 +697,14 @@ for obj in client.schema_contents(attach_opaque_data=attach_opaque_data, path=["
     elif isinstance(obj, FunctionInfo):
         print(f"Function: {obj.name}")
 
+# Or load the whole catalog at once. Uses one catalog_contents RPC when the
+# attach result advertises supports_catalog_contents, else the per-schema RPCs.
+snapshot = client.load_catalog(attach=result)
+for entry in snapshot.schemas:
+    print(entry.schema.path, [t.name for t in entry.tables])
+# Later: revalidate. A not_modified answer returns the same content cheaply.
+snapshot = client.load_catalog(attach=result, previous=snapshot)
+
 # Get only scalar functions using type filter
 from vgi.catalog import SchemaObjectType
 for obj in client.schema_contents(
@@ -723,7 +731,8 @@ client.catalog_detach(attach_opaque_data=attach_opaque_data)
 | `schema_create()` | Create a schema |
 | `schema_drop()` | Drop a schema |
 | `schema_contents()` | List schema contents (optional `type` filter) |
-| `contents()` | Load every schema and its contents in one call (`if_none_match` revalidates) |
+| `load_catalog()` | Load the whole catalog: one `catalog_contents` call when the attach advertises it, else (or on failure, or inside a transaction) `catalog_schemas` + the per-schema RPCs; pass `previous=` to revalidate with `if_none_match` |
+| `contents()` | The raw `catalog_contents` RPC (only valid when advertised; `if_none_match` revalidates) |
 | `table_get()` | Get table info |
 | `table_create()` | Create a table |
 | `table_drop()` | Drop a table |
@@ -935,7 +944,7 @@ if __name__ == "__main__":
 
 ---
 
-## API Limitations
+## Catalog capabilities and limitations { #api-limitations }
 
 The current CatalogInterface has the following limitations:
 
@@ -943,8 +952,33 @@ The current CatalogInterface has the following limitations:
 - **Tags**: Cannot be updated after object creation
 - **Schema metadata**: Comments and tags cannot be updated on schemas
 - **Constraints**: Only NOT NULL can be added/dropped (no ALTER for UNIQUE/CHECK)
-- **Indexes**: Not supported
-- **INSERT/UPDATE/DELETE**: Not yet implemented (metadata only)
+
+### Table writes and indexes
+
+`INSERT`, `UPDATE`, and `DELETE` are supported when the catalog supplies a write
+function for the operation. For declarative catalogs, configure the
+[`Table`][vgi.catalog.descriptors.Table] descriptor:
+
+| Operation | Descriptor field | Requirement |
+|-----------|------------------|-------------|
+| `INSERT` | `insert_function` | A `TableInOutGenerator` that accepts rows to insert |
+| `UPDATE` | `update_function` | A write function and a scan `function` that provides row IDs |
+| `DELETE` | `delete_function` | A write function and a scan `function` that provides row IDs |
+
+Each field defaults to `None`, which leaves that operation unsupported. Custom
+catalogs implement `table_insert_function_get()`, `table_update_function_get()`,
+or `table_delete_function_get()` to return the corresponding `ScanFunctionResult`.
+The worker's write functions implement the changes to the backing data source;
+declaring a table alone does not make it writable.
+
+`ReadOnlyCatalogInterface` rejects catalog DDL, but can expose these write functions
+on declarative tables. Table writes do not require the bundled
+[transactor](api/transactor.md); that is an optional database-access service with
+additional engine requirements.
+
+Index metadata can be declared with [`Index`][vgi.catalog.descriptors.Index]. Custom
+catalogs can implement `index_get()`, `index_create()`, and `index_drop()`; support
+depends on the catalog implementation and backing data source.
 
 ---
 

{vgi_python-0.40.0 → vgi_python-0.42.0}/docs/cli.md

@@ -545,7 +545,7 @@ vgi-fixture-worker --log-format json
 | `VGI_QUIET=1` | Suppress the interactive-terminal startup warning (same as `--quiet`) |
 | `VGI_FILTER_DEBUG=1` | Enable filter pushdown debug logging |
 | `VGI_BEARER_TOKENS` | Comma-separated `token=principal` pairs for static bearer auth (HTTP only) |
-| `VGI_JWT_ISSUER` | JWT issuer URL for JWT/JWKS auth (requires `vgi[oauth]` extra) |
+| `VGI_JWT_ISSUER` | JWT issuer URL for JWT/JWKS auth (requires `vgi-python[oauth]` extra) |
 | `VGI_JWT_AUDIENCE` | JWT audience string, comma-separated for multiple audiences (required when `VGI_JWT_ISSUER` is set) |
 | `VGI_JWT_JWKS_URI` | JWKS endpoint URL (auto-discovered if omitted) |
 | `VGI_OAUTH_RESOURCE` | OAuth resource URL for RFC 9728 metadata |

{vgi_python-0.40.0 → vgi_python-0.42.0}/docs/how-to/function-patterns.md

@@ -1,5 +1,5 @@
 ---
-description: "How to write each of the four VGI function patterns: scalar, table, table-in-out, and aggregate — with a complete runnable worker for each."
+description: "How to write each of the five VGI function patterns: scalar, table, table-in-out, aggregate, and buffering — with a complete runnable worker for each."
 ---
 
 # Function patterns
@@ -178,11 +178,14 @@ ATTACH 'aggregates' (TYPE vgi, LOCATION 'uv run sum_worker.py');
 SELECT category, aggregates.vgi_sum(value) FROM t GROUP BY category;
 ```
 
-??? info "State must be an `ArrowSerializableDataclass`"
-    Table generators and aggregates keep state between calls. Because that state is serialized
-    across parallel workers (aggregates) and HTTP round-trips, the framework requires it to extend
-    `ArrowSerializableDataclass`. Annotate fields with `ArrowType(...)` so the wire type is
-    explicit.
+??? info "State must be serializable"
+    Table generators and aggregates keep state between calls. A state type must provide
+    `serialize_to_bytes()` and `deserialize_from_bytes()` so the framework can carry it across
+    HTTP round-trips or persist it between aggregate calls. `ArrowSerializableDataclass`
+    implements these methods for you and is the usual starting point. Custom state types can
+    implement the same [`StreamStateCodec`][vgi.function.StreamStateCodec] contract without
+    inheriting from `ArrowSerializableDataclass`. See
+    [State Class](../aggregate-functions.md#state-class) for both approaches.
 
 ## Buffering
 

{vgi_python-0.40.0 → vgi_python-0.42.0}/docs/how-to/state-storage.md

@@ -22,9 +22,11 @@ function that coordinates partial results across workers.
 
 ## Two kinds of "state" — don't confuse them
 
-- **Generator cursor state** — the small `ArrowSerializableDataclass` a table generator keeps
-  *within one scan* (see [streaming with state](function-patterns.md#streaming-with-state)). It
-  lives in the worker for the duration of the call.
+- **Generator cursor state** — the small serializable object a table generator keeps
+  *within one scan* (see [streaming with state](function-patterns.md#streaming-with-state)).
+  Over HTTP, the framework serializes it between requests. `ArrowSerializableDataclass`
+  supplies the serialization methods; custom types can implement
+  [`StreamStateCodec`][vgi.function.StreamStateCodec] instead.
 - **Shared storage** (this page) — state that must outlive a single call or be shared across
   **separate worker processes**, e.g. combining partial aggregate results. This is backed by a
   pluggable store.

{vgi_python-0.40.0 → vgi_python-0.42.0}/docs/shared-storage.md

@@ -10,7 +10,7 @@ variable.
 | Backend | Value | Use Case | Dependencies |
 |---------|-------|----------|-------------|
 | SQLite | `sqlite` (default) | Local / subprocess transport | None (stdlib) |
-| Azure SQL | `azure-sql` | Azure cloud deployments | `vgi[azure]` |
+| Azure SQL | `azure-sql` | Azure cloud deployments | `vgi-python[azure]` |
 | Cloudflare DO | `cloudflare-do` | Edge / multi-cloud deployments | None (stdlib) |
 
 ## SQLite (Default)
@@ -36,7 +36,7 @@ workers run on separate hosts.
 1. Install the Azure extra:
 
 ```bash
-pip install vgi[azure]
+pip install 'vgi-python[azure]'
 ```
 
 2. Create an Azure SQL Database (Serverless recommended for cost):

{vgi_python-0.40.0 → vgi_python-0.42.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "vgi-python"
-version = "0.40.0"
+version = "0.42.0"
 description = "Vector Gateway Interface - Connect DuckDB to external programs via Apache Arrow"
 readme = "README.md"
 keywords = [
@@ -40,7 +40,7 @@ dependencies = [
     "pyarrow",
     "typer>=0.9",
     "platformdirs",
-    "vgi-rpc>=0.48.0",
+    "vgi-rpc>=0.49.0",
     "httpx2>=2.12.0",
 ]