vgi-python 0.23.0__tar.gz → 0.24.0__tar.gz

{vgi_python-0.23.0 → vgi_python-0.24.0}/.github/workflows/integration.yml

@@ -32,7 +32,7 @@ env:
   VGI_REF: main
   # The Haybarn release providing the prebuilt haybarn-unittest binary. Must be
   # ABI-compatible with the community-published vgi extension.
-  HAYBARN_RELEASE: haybarn-v1.5.4-rc3
+  HAYBARN_RELEASE: haybarn-v1.5.5-rc1
   # CI is not a user: its ATTACHes should not land in Query.Farm's adoption data.
   # The vgi extension fires a telemetry event per successful ATTACH, and this
   # suite performs several hundred of them per lane, so opting out keeps the

{vgi_python-0.23.0 → vgi_python-0.24.0}/CLAUDE.md

@@ -508,4 +508,5 @@ The table schema is automatically derived from the function's `output_schema`:
 - **Function lifecycle**: `docs/lifecycle.md`
 - **Generator API (advanced)**: `docs/generator-api.md`
 - **Catalog interface**: `docs/catalog-interface.md`
+- **Global functions (system.main)**: `docs/global-functions.md`
 - **Shared storage backends**: `docs/shared-storage.md`

{vgi_python-0.23.0 → vgi_python-0.24.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: vgi-python
-Version: 0.23.0
+Version: 0.24.0
 Summary: Vector Gateway Interface - Connect DuckDB to external programs via Apache Arrow
 Project-URL: Homepage, https://query.farm
 Project-URL: Repository, https://github.com/Query-farm/vgi-python
@@ -162,7 +162,7 @@ Requires-Dist: httpx>=0.24
 Requires-Dist: platformdirs
 Requires-Dist: pyarrow
 Requires-Dist: typer>=0.9
-Requires-Dist: vgi-rpc>=0.28.3
+Requires-Dist: vgi-rpc>=0.33.0
 Provides-Extra: azure
 Requires-Dist: azure-identity>=1.16.0; extra == 'azure'
 Requires-Dist: pymssql>=2.3.0; extra == 'azure'
@@ -190,8 +190,11 @@ Provides-Extra: duckdb
 Requires-Dist: duckdb>=1.5.3; extra == 'duckdb'
 Provides-Extra: fixtures
 Requires-Dist: vgi-fixtures; extra == 'fixtures'
+Provides-Extra: granian
+Requires-Dist: granian>=2.7; extra == 'granian'
+Requires-Dist: vgi-rpc[http]; extra == 'granian'
 Provides-Extra: haybarn
-Requires-Dist: haybarn>=1.5.3rc10; extra == 'haybarn'
+Requires-Dist: haybarn>=1.5.5rc1; extra == 'haybarn'
 Provides-Extra: http
 Requires-Dist: vgi-rpc[http]; extra == 'http'
 Provides-Extra: oauth

vgi_python-0.24.0/docs/global-functions.md

@@ -0,0 +1,88 @@
+# Global functions
+
+A VGI catalog normally exposes its functions under its own name — `mycat.main.my_func()`.
+Some functions aren't *about* the attached catalog at all: diagnostics, converters, format
+helpers. For those, a worker can ask the client to also publish them into the **global**
+function namespace (DuckDB's `system.main`), so they can be called unqualified:
+
+```sql
+ATTACH 'acme' AS acme (TYPE vgi, LOCATION 'acme-worker');
+SELECT * FROM acme_table_info('acme');   -- no catalog qualifier
+```
+
+This mirrors how `ducklake_table_info` is reachable after `LOAD ducklake`, except that a VGI
+worker's globals appear at `ATTACH` time rather than at extension load.
+
+## Declaring globals
+
+Add the function classes to `Catalog.global_functions` and pick a prefix:
+
+```python
+from vgi.catalog import Catalog, Schema
+
+CATALOG = Catalog(
+    name="acme",
+    global_function_prefix="acme",
+    global_functions=[TableInfoFunction, ChecksumFunction],
+    schemas=[
+        Schema(name="main", functions=[TableInfoFunction, ChecksumFunction, ...]),
+    ],
+)
+```
+
+All four function types are supported: scalar, aggregate, table, and table-buffering.
+
+**Every global must also appear in exactly one `Schema.functions`.** This is enforced in
+`Catalog.__post_init__`. Bind dispatch is keyed on `(schema_name, name)`, so a function listed
+only in `global_functions` would be registered by the client but never dispatchable. Keeping it
+schema-resident also means the qualified name (`acme.main.table_info()`) keeps working, which is
+the unambiguous fallback if the global name is unavailable.
+
+### The prefix
+
+`global_function_prefix` is applied by the client to form the globally visible name:
+
+| prefix | function name | published as |
+|---|---|---|
+| `"acme"` | `table_info` | `acme_table_info` |
+| `None` | `table_info` | `table_info` |
+
+The prefix must match `[a-z_][a-z0-9_]*` — it is concatenated into a SQL identifier.
+
+**Prefer a prefix.** `system.main` is shared by every extension and every attached VGI worker in
+the process. An unprefixed name like `table_info` is very likely to be claimed by someone else,
+in which case yours is silently skipped (see below). A prefix that identifies your worker makes
+collisions essentially impossible.
+
+## What the client does with them
+
+Registration is **best-effort and advisory**. Do not build anything that requires a global name
+to resolve — treat it as an ergonomic alias for the schema-qualified name, which is the one with
+guarantees.
+
+Specifically, on the DuckDB side:
+
+- **First attach wins.** If the name is already owned by a different worker, yours is skipped and
+  logged; your `ATTACH` still succeeds and the function stays reachable at its qualified path.
+- **Re-`ATTACH` is idempotent.** Attaching the same worker again reuses the existing registration
+  and refreshes its connection state.
+- **The user can turn it off.** `ATTACH ... (TYPE vgi, global_functions false)` suppresses
+  publishing entirely for that attach.
+- **Registration outlives `DETACH`.** DuckDB has no API to unregister a function, so the entry
+  persists for the life of the process. Calling it after `DETACH` raises an error telling the user
+  to re-`ATTACH` rather than silently reconnecting to a detached catalog.
+
+Use `vgi_global_functions()` to see what is currently published, who owns it, and whether it is
+still live.
+
+## Protocol
+
+Globals ride on the existing attach response — there is no extra round trip and no separate RPC.
+`CatalogAttachResult` carries:
+
+- `global_functions: list[bytes]` — IPC-serialized `FunctionInfo` records. `name` and
+  `schema_name` are the real dispatch coordinates; the prefix is *not* baked into `name`.
+- `global_function_prefix: str` — empty string means publish bare names.
+
+Both fields are additive and default to empty, so a worker that doesn't set them advertises
+nothing. Introduced in protocol version 1.3.0.

{vgi_python-0.23.0 → vgi_python-0.24.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "vgi-python"
-version = "0.23.0"
+version = "0.24.0"
 description = "Vector Gateway Interface - Connect DuckDB to external programs via Apache Arrow"
 readme = "README.md"
 keywords = [
@@ -40,7 +40,7 @@ dependencies = [
     "pyarrow",
     "typer>=0.9",
     "platformdirs",
-    "vgi-rpc>=0.28.3",
+    "vgi-rpc>=0.33.0",
     "httpx>=0.24",
 ]
 
@@ -49,9 +49,13 @@ dependencies = [
 # evaluation, the transactor, and the statistics fixtures. It is resolved
 # lazily in vgi/_duckdb.py: haybarn is preferred, stock duckdb is the
 # fallback. Neither is a hard dependency — install one via these extras.
-haybarn = ["haybarn>=1.5.3rc10"]
+haybarn = ["haybarn>=1.5.5rc1"]
 duckdb = ["duckdb>=1.5.3"]
 http = ["vgi-rpc[http]"]
+# Alternative HTTP server: granian moves socket I/O and HTTP parsing into Rust,
+# off the GIL. Optional because it is a compiled extension where waitress is
+# pure Python; select it with `vgi-serve --http --server granian`.
+granian = ["vgi-rpc[http]", "granian>=2.7"]
 oauth = ["vgi-rpc[oauth]"]
 otel = ["vgi-rpc[otel]"]
 sentry = ["vgi-rpc[sentry]"]
@@ -187,6 +191,12 @@ warn_unused_ignores = true
 module = "typer.*"
 ignore_missing_imports = true
 
+[[tool.mypy.overrides]]
+# granian ships no stubs, and is an optional extra used only behind
+# `--server granian`.
+module = "granian.*"
+ignore_missing_imports = true
+
 [[tool.mypy.overrides]]
 # These files have type: ignore comments for ty that mypy doesn't need
 module = ["vgi._test_fixtures.scalar.null_handling", "vgi.table_function"]

{vgi_python-0.23.0 → vgi_python-0.24.0}/tests/catalog/test_declarative.py

@@ -26,6 +26,7 @@ from vgi.catalog import (
     Catalog,
     CatalogAttachResult,
     ForeignKeyDef,
+    FunctionInfo,
     Macro,
     MacroInfo,
     MacroType,
@@ -41,6 +42,7 @@ from vgi.catalog import (
     View,
     ViewInfo,
 )
+from vgi.catalog.catalog_interface import FunctionType as CatalogInterfaceFunctionType
 from vgi.invocation import BindResponse
 from vgi.schema_utils import schema
 from vgi.table_function import (
@@ -1394,6 +1396,152 @@ class TestCatalogValidation:
             Catalog(name="myapp", schemas=[s1, s2])
 
 
+class TestGlobalFunctionsValidation:
+    """Validation for ``Catalog.global_functions`` / ``global_function_prefix``."""
+
+    def test_defaults_are_empty(self) -> None:
+        """A catalog publishes nothing globally unless it asks to."""
+        catalog = Catalog(name="myapp", schemas=[Schema(name="main")])
+        assert catalog.global_functions == ()
+        assert catalog.global_function_prefix is None
+
+    def test_accepts_schema_resident_function(self) -> None:
+        """The happy path: listed once globally, resident in exactly one schema."""
+        catalog = Catalog(
+            name="myapp",
+            schemas=[Schema(name="main", functions=[UsersFunction])],
+            global_functions=[UsersFunction],
+            global_function_prefix="myapp",
+        )
+        assert list(catalog.global_functions) == [UsersFunction]
+        assert catalog.global_function_prefix == "myapp"
+
+    def test_rejects_function_absent_from_every_schema(self) -> None:
+        """A global function that isn't schema-resident would never dispatch."""
+        with pytest.raises(ValueError, match="does not appear in any Schema.functions"):
+            Catalog(
+                name="myapp",
+                schemas=[Schema(name="main")],
+                global_functions=[UsersFunction],
+            )
+
+    def test_rejects_function_resident_in_multiple_schemas(self) -> None:
+        """Two homes make the global registration ambiguous about dispatch."""
+        with pytest.raises(ValueError, match="appears in multiple schemas"):
+            Catalog(
+                name="myapp",
+                schemas=[
+                    Schema(name="main", functions=[UsersFunction]),
+                    Schema(name="other", functions=[UsersFunction]),
+                ],
+                global_functions=[UsersFunction],
+            )
+
+    def test_rejects_duplicate_global_entries(self) -> None:
+        """Listing a class twice would publish the same name twice."""
+        with pytest.raises(ValueError, match="duplicate entry UsersFunction"):
+            Catalog(
+                name="myapp",
+                schemas=[Schema(name="main", functions=[UsersFunction])],
+                global_functions=[UsersFunction, UsersFunction],
+            )
+
+    @pytest.mark.parametrize("prefix", ["9bad", "Bad", "has-dash", "has space", "", "trailing;"])
+    def test_rejects_non_identifier_prefix(self, prefix: str) -> None:
+        """The prefix is concatenated into a SQL identifier, so it must be safe."""
+        with pytest.raises(ValueError, match="not a valid SQL identifier prefix"):
+            Catalog(
+                name="myapp",
+                schemas=[Schema(name="main", functions=[UsersFunction])],
+                global_functions=[UsersFunction],
+                global_function_prefix=prefix,
+            )
+
+    def test_prefix_validated_even_without_global_functions(self) -> None:
+        """A bad prefix is a bug worth reporting even if nothing uses it yet."""
+        with pytest.raises(ValueError, match="not a valid SQL identifier prefix"):
+            Catalog(name="myapp", schemas=[Schema(name="main")], global_function_prefix="Bad")
+
+
+class TestGlobalFunctionsAttachResult:
+    """``catalog_attach`` advertises globals so the client can publish them."""
+
+    @staticmethod
+    def _attach(catalog_obj: Catalog) -> CatalogAttachResult:
+        class _TestCatalog(ReadOnlyCatalogInterface):
+            catalog = catalog_obj
+
+        return _TestCatalog().catalog_attach(
+            name=catalog_obj.name,
+            options={},
+            data_version_spec=None,
+            implementation_version=None,
+        )
+
+    def test_absent_when_not_declared(self) -> None:
+        """Catalogs that declare nothing advertise an empty list and no prefix."""
+        result = self._attach(Catalog(name="myapp", schemas=[Schema(name="main", functions=[UsersFunction])]))
+        assert result.global_functions == []
+        assert result.global_function_prefix == ""
+
+    def test_carries_serialized_function_info(self) -> None:
+        """Each global arrives as a FunctionInfo with real dispatch coordinates.
+
+        ``name``/``schema_name`` must stay the values bind dispatches on — the
+        prefix is applied client-side, not baked into the name here.
+        """
+        result = self._attach(
+            Catalog(
+                name="myapp",
+                schemas=[Schema(name="analytics", functions=[UsersFunction])],
+                default_schema="analytics",
+                global_functions=[UsersFunction],
+                global_function_prefix="myapp",
+            )
+        )
+        assert result.global_function_prefix == "myapp"
+        assert len(result.global_functions) == 1
+
+        info = FunctionInfo.deserialize_from_bytes(result.global_functions[0])
+        assert info.name == "users"
+        assert info.schema_name == "analytics"
+        assert info.function_type == CatalogInterfaceFunctionType.TABLE
+
+    def test_prefix_empty_string_when_unset(self) -> None:
+        """``None`` prefix serializes as empty string — the wire type is str."""
+        result = self._attach(
+            Catalog(
+                name="myapp",
+                schemas=[Schema(name="main", functions=[UsersFunction])],
+                global_functions=[UsersFunction],
+            )
+        )
+        assert result.global_function_prefix == ""
+        assert len(result.global_functions) == 1
+
+    def test_example_fixture_publishes_one_of_each_function_type(self) -> None:
+        """The shipped fixture covers all four types for the C++ integration suite."""
+        from vgi._test_fixtures.worker import ExampleCatalog
+
+        result = ExampleCatalog().catalog_attach(
+            name="example",
+            options={},
+            data_version_spec=None,
+            implementation_version=None,
+        )
+        assert result.global_function_prefix == "vgi_example"
+        infos = [FunctionInfo.deserialize_from_bytes(b) for b in result.global_functions]
+        assert {i.function_type for i in infos} == {
+            CatalogInterfaceFunctionType.SCALAR,
+            CatalogInterfaceFunctionType.TABLE,
+            CatalogInterfaceFunctionType.AGGREGATE,
+            CatalogInterfaceFunctionType.TABLE_BUFFERING,
+        }
+        # Every advertised global must be schema-resident, or it could never
+        # be dispatched after the client registers it.
+        assert all(i.schema_name == "main" for i in infos)
+
+
 # =============================================================================
 # ReadOnlyCatalogInterface with Catalog Object Tests
 # =============================================================================

{vgi_python-0.23.0 → vgi_python-0.24.0}/tests/catalog/test_serialization.py

@@ -1110,7 +1110,7 @@ class TestArrowSchemaCorrectness:
     def test_catalog_attach_result_schema(self) -> None:
         """Verify CatalogAttachResult Arrow schema."""
         schema = CatalogAttachResult.ARROW_SCHEMA
-        assert len(schema) == 15
+        assert len(schema) == 17
         assert schema.field("attach_opaque_data").type == pa.binary()
         assert schema.field("supports_transactions").type == pa.bool_()
         assert schema.field("supports_time_travel").type == pa.bool_()
@@ -1121,6 +1121,8 @@ class TestArrowSchemaCorrectness:
         assert schema.field("settings").type == pa.list_(pa.binary())
         assert schema.field("secret_types").type == pa.list_(pa.binary())
         assert schema.field("attach_catalogs").type == pa.list_(pa.binary())
+        assert schema.field("global_functions").type == pa.list_(pa.binary())
+        assert schema.field("global_function_prefix").type == pa.string()
         assert schema.field("resolved_data_version").type == pa.string()
         assert schema.field("resolved_implementation_version").type == pa.string()
 

{vgi_python-0.23.0 → vgi_python-0.24.0}/tests/conformance/test_directory_parity.py

@@ -42,6 +42,19 @@ _EXEMPTIONS: dict[str, str] = {
         "path, ProcessParams.if_none_match threading, and the revalidatable "
         "fixture's 304 behavior)."
     ),
+    "global_functions": (
+        "Publishing worker-declared functions into DuckDB's system.main is "
+        "entirely a C++-extension concern: the registry, prefix application, "
+        "first-attach-wins collision policy, post-DETACH liveness, and the "
+        "vgi_global_functions() introspection surface all live in the "
+        "extension. The Python worker's only role is advertising "
+        "global_functions/global_function_prefix on the attach response, which "
+        "tests/catalog/test_declarative.py covers directly (descriptor "
+        "validation + the serialized FunctionInfo payload). The observable "
+        "behaviour is cross-language by construction and is covered by "
+        "vgi/test/sql/integration/global_functions/, which any worker "
+        "implementation runs via VGI_TEST_WORKER."
+    ),
     "filter_pushdown": (
         "Per-type filter-pushdown coverage is mirrored by tests/test_filter_pushdown.py "
         "and tests/test_filter_pushdown_extension.py, which drive the same predicate "

{vgi_python-0.23.0 → vgi_python-0.24.0}/tests/conformance/test_protocol_version.py

@@ -32,14 +32,14 @@ CLIENT_PROTOCOL_VERSION = VgiProtocol.protocol_version
 
 def test_client_protocol_version_is_canonical() -> None:
     """Sanity-check the constants this test reasons about haven't drifted apart."""
-    assert CLIENT_PROTOCOL_VERSION == "1.2.0"
+    assert CLIENT_PROTOCOL_VERSION == "1.3.0"
     # The fixture must be a genuine mismatch in the major/minor that the
     # framework compares; a patch-only difference would (correctly) pass.
     assert BAD_PROTOCOL_VERSION.split(".")[:2] != CLIENT_PROTOCOL_VERSION.split(".")[:2]
 
 
 def test_mismatched_worker_protocol_version_raises() -> None:
-    """A worker enforcing 99.0.0 must reject the 1.2.0 client at dispatch."""
+    """A worker enforcing 99.0.0 must reject the 1.3.0 client at dispatch."""
     with Client("vgi-fixture-bad-protocol-worker") as client, pytest.raises(ClientError) as exc_info:
         list(
             client.table_function(
@@ -54,5 +54,5 @@ def test_mismatched_worker_protocol_version_raises() -> None:
     assert "protocol_version mismatch" in message
     assert CLIENT_PROTOCOL_VERSION in message
     assert BAD_PROTOCOL_VERSION in message
-    # Directional hint: the client (1.2.0) is older than the worker (99.0.0).
+    # Directional hint: the client (1.3.0) is older than the worker (99.0.0).
     assert "upgrade the VGI extension/client" in message

{vgi_python-0.23.0 → vgi_python-0.24.0}/tests/landing/describe.expected.json

@@ -3,7 +3,7 @@
     {
       "attach_options": [],
       "counts": {
-        "functions": 217,
+        "functions": 221,
         "schemas": 2,
         "tables": 59,
         "views": 3
@@ -371,6 +371,19 @@
         {
           "doc": "Example functions for testing VGI",
           "functions": [
+            {
+              "args": [
+                {
+                  "desc": "Column to sum",
+                  "name": "value",
+                  "type": "int64"
+                }
+              ],
+              "doc": "Global-registration probe (aggregate)",
+              "name": "global_agg",
+              "returns": "int64",
+              "type": "aggregate"
+            },
             {
               "args": [
                 {
@@ -979,6 +992,19 @@
               "returns": "double",
               "type": "scalar"
             },
+            {
+              "args": [
+                {
+                  "desc": "Value to label",
+                  "name": "value",
+                  "type": "int64"
+                }
+              ],
+              "doc": "Global-registration probe (scalar)",
+              "name": "global_scalar",
+              "returns": "string",
+              "type": "scalar"
+            },
             {
               "args": [
                 {
@@ -2146,6 +2172,12 @@
               "name": "geo_encode",
               "type": "table"
             },
+            {
+              "args": [],
+              "doc": "Global-registration probe (table)",
+              "name": "global_table",
+              "type": "table"
+            },
             {
               "args": [
                 {
@@ -3281,6 +3313,12 @@
               "name": "exception_process",
               "type": "table_in_out"
             },
+            {
+              "args": [],
+              "doc": "Global-registration probe (table-buffering)",
+              "name": "global_buffered",
+              "type": "table_in_out"
+            },
             {
               "args": [],
               "doc": "Worker sleeps in process (manual cancel test)",

vgi_python-0.24.0/tests/table/test_wide_columns.py

@@ -0,0 +1,84 @@
+# Copyright 2025, 2026 Query Farm LLC - https://query.farm
+
+"""Tests for the wide-schema benchmark fixture.
+
+Guards the fixture the width benchmarks depend on: if ``wide`` stops
+producing the schema it claims, a profile taken against it measures
+something other than what it says on the tin.
+"""
+
+from __future__ import annotations
+
+import pyarrow as pa
+import pytest
+from vgi_rpc.rpc import RpcServer
+
+from vgi._test_fixtures.table.wide import DEFAULT_COLUMNS, WideFunction
+from vgi.arguments import Arguments
+from vgi.protocol import BindRequest, FunctionType, InitRequest, VgiProtocol
+from vgi.worker import Worker
+
+
+class _WideWorker(Worker):
+    """Worker exposing only the wide fixture, so it stays off ExampleWorker."""
+
+    functions = [WideFunction]
+
+
+def _scan(columns: int, count: int, batch_size: int) -> list[pa.RecordBatch]:
+    from vgi_rpc.http import http_connect, make_sync_client
+
+    key = b"\x11" * 32
+    worker = _WideWorker(quiet=True)
+    worker._signing_key = key
+    client = make_sync_client(RpcServer(VgiProtocol, worker, enable_describe=False), token_key=key)
+    with http_connect(VgiProtocol, client=client, compression_level=None) as proxy:  # type: ignore[type-abstract]
+        bind = BindRequest(
+            function_name="wide",
+            arguments=Arguments(
+                positional=(pa.scalar(count),),
+                named={"columns": pa.scalar(columns), "batch_size": pa.scalar(batch_size)},
+            ),
+            function_type=FunctionType.TABLE,
+            input_schema=pa.schema([]),
+        )
+        resp = proxy.bind(request=bind)
+        assert len(resp.output_schema) == columns
+        stream = proxy.init(
+            request=InitRequest(bind_call=bind, output_schema=resp.output_schema, bind_opaque_data=resp.opaque_data)
+        )
+        try:
+            return [ab.batch for ab in stream]
+        finally:
+            stream.close()
+
+
+@pytest.mark.parametrize("columns", [1, 12, 2000])
+def test_wide_emits_the_requested_width(columns: int) -> None:
+    """The bound schema and every emitted batch carry exactly ``columns``."""
+    batches = _scan(columns=columns, count=8, batch_size=4)
+    assert batches, "expected at least one batch"
+    assert sum(b.num_rows for b in batches) == 8
+    for batch in batches:
+        assert batch.num_columns == columns
+        assert batch.schema.names[0] == "c0"
+        assert batch.schema.names[-1] == f"c{columns - 1}"
+
+
+def test_wide_survives_multi_turn_state_round_trip() -> None:
+    """Rows stay contiguous across turns, so the cursor round-trips at width.
+
+    Each turn re-serializes state and rebuilds a zero-row sentinel batch of
+    the full width; this is the path the width benchmark exercises.
+    """
+    batches = _scan(columns=DEFAULT_COLUMNS, count=20, batch_size=4)
+    values = [v for b in batches for v in b.column(0).to_pylist()]
+    assert values == list(range(20))
+    assert all(b.num_columns == DEFAULT_COLUMNS for b in batches)
+
+
+def test_wide_is_not_registered_on_the_example_worker() -> None:
+    """Keep it off ExampleWorker: that count is asserted by the C++ suite."""
+    from vgi._test_fixtures.worker import ExampleWorker
+
+    assert WideFunction not in ExampleWorker.functions