graphique 2.0.2__tar.gz → 2.2__tar.gz

{graphique-2.0.2 → graphique-2.2}/PKG-INFO

@@ -1,41 +1,44 @@
 Metadata-Version: 2.4
 Name: graphique
-Version: 2.0.2
+Version: 2.2
 Summary: GraphQL service for python dataframes and parquet datasets.
-Author-email: Aric Coady <aric.coady@gmail.com>
+Author: A. Coady
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/coady/graphique
 Project-URL: Documentation, https://coady.github.io/graphique
 Project-URL: Changelog, https://github.com/coady/graphique/blob/main/CHANGELOG.md
 Project-URL: Issues, https://github.com/coady/graphique/issues
-Keywords: graphql,arrow,parquet
+Keywords: graphql,ibis,dataframes,arrow,parquet,duckdb
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
 Classifier: Topic :: Database :: Database Engines/Servers
 Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
 Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Typing :: Typed
-Requires-Python: >=3.11
+Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
-Requires-Dist: pyarrow>=20
-Requires-Dist: strawberry-graphql[asgi,cli]>=0.288
+Requires-Dist: pyarrow>=23
+Requires-Dist: strawberry-graphql[asgi,cli]>=0.328.3
 Requires-Dist: ibis-framework>=10.7
-Requires-Dist: pyarrow-hotfix
 Requires-Dist: isodate>=0.7
+Requires-Dist: pyarrow-hotfix
+Requires-Dist: sqlglot<30.18
 Provides-Extra: server
 Requires-Dist: uvicorn[standard]; extra == "server"
 Requires-Dist: duckdb; extra == "server"
 Provides-Extra: cli
+Requires-Dist: duckdb; extra == "cli"
 Requires-Dist: tqdm; extra == "cli"
+Requires-Dist: typer; extra == "cli"
 Dynamic: license-file
 
 [![image](https://img.shields.io/pypi/v/graphique.svg)](https://pypi.org/project/graphique/)
@@ -48,14 +51,9 @@ Dynamic: license-file
 [![image](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
 
-[GraphQL](https://graphql.org) service for [ibis](https://ibis-project.org) dataframes, [arrow](https://arrow.apache.org) tables, and [parquet](https://parquet.apache.org) datasets. The schema for a query API is derived automatically.
-
-## Version 2
-When this project started, there was no out-of-core execution engine with performance comparable to [PyArrow](https://arrow.apache.org/docs/python/index.html). So it effectively included one, based on datasets and [Acero](https://arrow.apache.org/docs/python/api/acero.html).
+[GraphQL](https://graphql.org) service for [ibis](https://ibis-project.org) tables. Ibis supports [20+ backends](https://ibis-project.org/why#how-does-ibis-work) — DuckDB, PostgreSQL, Polars, BigQuery, etc. — so the same query API works across local files and remote databases. The schema is derived automatically.
 
-Since then the ecosystem has grown considerably: [DuckDB](https://duckdb.org), [DataFusion](https://datafusion.apache.org), and [Ibis](https://ibis-project.org). As of version 2, graphique is based on `ibis`. It provides a common dataframe API for multiple backends, enabling graphique to also have a default but configurable backend.
-
-Being a major version upgrade, there are incompatible changes from version 1. However the overall API remains largely the same.
+[Parquet](https://parquet.apache.org) files load with `ibis.read_parquet`. For hive-partitioned data, [DuckLake](https://ducklake.select/) (`graphique.ducklake.register` / `attach`) keeps partition metadata in a catalog so DuckDB can prune and count without scanning footers. Passing a PyArrow `Dataset` as a root is deprecated.
 
 ## Usage
 There is an example app which reads a parquet dataset.
@@ -74,25 +72,30 @@ outputs the graphql schema.
 The example app uses [Starlette's config](https://www.starlette.io/config/): in environment variables or a `.env` file.
 
 * PARQUET_PATH: path to the parquet directory or file
-* FEDERATED = '': field name to extend type `Query` with a federated `Table`
-* METRICS = False: include timings from apollo tracing extension
+* NAME = '': GraphQL field on `Query`; defaults to root type
 * COLUMNS = None: list of names, or mapping of aliases, of columns to select
-* FILTERS = None: json `filter` query for which rows to read at startup
 
 Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
 
 ### App
-For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an ibis [Table](https://ibis-project.org/reference/expression-tables) or arrow [Dataset](https://arrow.apache.org/docs/python/api/dataset.html).
-Supply a mapping of names to datasets for multiple roots, and to enable federation.
+For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an ibis [Table](https://ibis-project.org/reference/expression-tables).
+Use a `Query` type with table attributes for multiple roots, and to enable federation.
 
 ```python
 import ibis
-from graphique import GraphQL
+from graphique import GraphQL, typed
+
+# any ibis backend: DuckDB, PostgreSQL, Polars, BigQuery, ...
+table = ibis.read_parquet(...)  # or `ibis.connect(...).table(...)`
+app = GraphQL(table)  # Table is root query type
 
-source = ibis.read_*(...)  # or ibis.connect(...).table(...) or pyarrow.dataset.dataset(...)
-# apply initial projections or filters to `source`
-app = GraphQL(source)  # Table is root query type
-app = GraphQL.federated({<name>: source, ...}, keys={<name>: [], ...})  # Tables on federated fields
+
+# multiple named fields, with optional federation keys
+class Query:
+    name = table  # or `typed(table, name, keys=...)`
+
+
+app = GraphQL(Query)
 ```
 
 Start like any ASGI app.
@@ -103,7 +106,7 @@ uvicorn <module>:app
 
 ### API
 #### types
-* `Dataset`: interface for an ibis table or arrow dataset.
+* `Dataset`: GraphQL interface over an ibis table.
 * `Table`: implements the `Dataset` interface. Adds typed `row`, `columns`, and `filter` fields from introspecting the schema.
 * `Column`: interface for an ibis column. Each data type has a corresponding column implementation: Boolean, Int, BigInt, Float, Decimal, Date, Datetime, Time, Duration, Base64, String, Array, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
 * `Row`: scalar fields. Tables are column-oriented, and graphique encourages that usage for performance. A single `row` field is provided for convenience, but a field for a list of rows is not. Requesting parallel columns is far more efficient.
@@ -111,7 +114,8 @@ uvicorn <module>:app
 #### selection
 * `slice`: contiguous selection of rows
 * `filter`: select rows by predicates
-* `join`: join tables by key columns
+* `join`, `asofJoin`, `crossJoin`: join tables by key columns
+* `difference`, `intersect`, `union`: set operations on tables
 * `take`: rows by index
 * `dropNull`: remove rows with nulls
 
@@ -121,27 +125,33 @@ uvicorn <module>:app
 * `column`: access a column of any type by name
 * `row`: provides a field for each scalar of a single row
 * `cast`: cast column types
+* `unpack`: project struct fields
 * `fillNull`: fill null values
 
 #### aggregation
 * `group`: group by given columns, and aggregate the others
 * `distinct`: group with all columns
-* `runs`: provisionally group by adjacency
+* `runs`: group by adjacency
 * `unnest`: unnest an array column
-* `count`: number of rows
+* `count`, `any`: number of rows
 
 #### ordering
 * `order`: sort table by given columns
-* options `limit` and `dense`: select rows with smallest or largest values
+* `first`: sort and filter by rank
+
+#### reflection
+* `schema`: field names and types
+* `optional`: nullable for errors
+* `toSql`: compiles SQL query
 
 ### Performance
-Performance is dependent on the [ibis backend](https://ibis-project.org/backends/duckdb), which defaults to [duckdb](https://duckdb.org/). There are no internal Python loops. Scalars do not become Python types until serialized.
+Performance is dependent on the Ibis backend, which defaults to DuckDB. There are no internal Python loops. Scalars do not become Python types until serialized. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
 
-[PyArrow](https://arrow.apache.org/docs/python/) is also used for partitioned dataset optimizations, and for any feature which ibis does not support. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
+For partitioned parquet datasets, [DuckLake](https://ducklake.select) provides performance benefits over `ibis.read_parquet`; see `graphique.ducklake.{register,attach}` for helpers. `python -m graphique.partition` in `graphique[cli]` writes hive-layout parquet out-of-core.
 
 ## Installation
 ```console
-pip install graphique[server]
+pip install graphique[server,cli]
 ```
 
 ## Dependencies

{graphique-2.0.2 → graphique-2.2}/README.md

@@ -8,14 +8,9 @@
 [![image](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
 
-[GraphQL](https://graphql.org) service for [ibis](https://ibis-project.org) dataframes, [arrow](https://arrow.apache.org) tables, and [parquet](https://parquet.apache.org) datasets. The schema for a query API is derived automatically.
+[GraphQL](https://graphql.org) service for [ibis](https://ibis-project.org) tables. Ibis supports [20+ backends](https://ibis-project.org/why#how-does-ibis-work) — DuckDB, PostgreSQL, Polars, BigQuery, etc. — so the same query API works across local files and remote databases. The schema is derived automatically.
 
-## Version 2
-When this project started, there was no out-of-core execution engine with performance comparable to [PyArrow](https://arrow.apache.org/docs/python/index.html). So it effectively included one, based on datasets and [Acero](https://arrow.apache.org/docs/python/api/acero.html).
-
-Since then the ecosystem has grown considerably: [DuckDB](https://duckdb.org), [DataFusion](https://datafusion.apache.org), and [Ibis](https://ibis-project.org). As of version 2, graphique is based on `ibis`. It provides a common dataframe API for multiple backends, enabling graphique to also have a default but configurable backend.
-
-Being a major version upgrade, there are incompatible changes from version 1. However the overall API remains largely the same.
+[Parquet](https://parquet.apache.org) files load with `ibis.read_parquet`. For hive-partitioned data, [DuckLake](https://ducklake.select/) (`graphique.ducklake.register` / `attach`) keeps partition metadata in a catalog so DuckDB can prune and count without scanning footers. Passing a PyArrow `Dataset` as a root is deprecated.
 
 ## Usage
 There is an example app which reads a parquet dataset.
@@ -34,25 +29,30 @@ outputs the graphql schema.
 The example app uses [Starlette's config](https://www.starlette.io/config/): in environment variables or a `.env` file.
 
 * PARQUET_PATH: path to the parquet directory or file
-* FEDERATED = '': field name to extend type `Query` with a federated `Table`
-* METRICS = False: include timings from apollo tracing extension
+* NAME = '': GraphQL field on `Query`; defaults to root type
 * COLUMNS = None: list of names, or mapping of aliases, of columns to select
-* FILTERS = None: json `filter` query for which rows to read at startup
 
 Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
 
 ### App
-For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an ibis [Table](https://ibis-project.org/reference/expression-tables) or arrow [Dataset](https://arrow.apache.org/docs/python/api/dataset.html).
-Supply a mapping of names to datasets for multiple roots, and to enable federation.
+For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an ibis [Table](https://ibis-project.org/reference/expression-tables).
+Use a `Query` type with table attributes for multiple roots, and to enable federation.
 
 ```python
 import ibis
-from graphique import GraphQL
+from graphique import GraphQL, typed
+
+# any ibis backend: DuckDB, PostgreSQL, Polars, BigQuery, ...
+table = ibis.read_parquet(...)  # or `ibis.connect(...).table(...)`
+app = GraphQL(table)  # Table is root query type
+
 
-source = ibis.read_*(...)  # or ibis.connect(...).table(...) or pyarrow.dataset.dataset(...)
-# apply initial projections or filters to `source`
-app = GraphQL(source)  # Table is root query type
-app = GraphQL.federated({<name>: source, ...}, keys={<name>: [], ...})  # Tables on federated fields
+# multiple named fields, with optional federation keys
+class Query:
+    name = table  # or `typed(table, name, keys=...)`
+
+
+app = GraphQL(Query)
 ```
 
 Start like any ASGI app.
@@ -63,7 +63,7 @@ uvicorn <module>:app
 
 ### API
 #### types
-* `Dataset`: interface for an ibis table or arrow dataset.
+* `Dataset`: GraphQL interface over an ibis table.
 * `Table`: implements the `Dataset` interface. Adds typed `row`, `columns`, and `filter` fields from introspecting the schema.
 * `Column`: interface for an ibis column. Each data type has a corresponding column implementation: Boolean, Int, BigInt, Float, Decimal, Date, Datetime, Time, Duration, Base64, String, Array, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
 * `Row`: scalar fields. Tables are column-oriented, and graphique encourages that usage for performance. A single `row` field is provided for convenience, but a field for a list of rows is not. Requesting parallel columns is far more efficient.
@@ -71,7 +71,8 @@ uvicorn <module>:app
 #### selection
 * `slice`: contiguous selection of rows
 * `filter`: select rows by predicates
-* `join`: join tables by key columns
+* `join`, `asofJoin`, `crossJoin`: join tables by key columns
+* `difference`, `intersect`, `union`: set operations on tables
 * `take`: rows by index
 * `dropNull`: remove rows with nulls
 
@@ -81,27 +82,33 @@ uvicorn <module>:app
 * `column`: access a column of any type by name
 * `row`: provides a field for each scalar of a single row
 * `cast`: cast column types
+* `unpack`: project struct fields
 * `fillNull`: fill null values
 
 #### aggregation
 * `group`: group by given columns, and aggregate the others
 * `distinct`: group with all columns
-* `runs`: provisionally group by adjacency
+* `runs`: group by adjacency
 * `unnest`: unnest an array column
-* `count`: number of rows
+* `count`, `any`: number of rows
 
 #### ordering
 * `order`: sort table by given columns
-* options `limit` and `dense`: select rows with smallest or largest values
+* `first`: sort and filter by rank
+
+#### reflection
+* `schema`: field names and types
+* `optional`: nullable for errors
+* `toSql`: compiles SQL query
 
 ### Performance
-Performance is dependent on the [ibis backend](https://ibis-project.org/backends/duckdb), which defaults to [duckdb](https://duckdb.org/). There are no internal Python loops. Scalars do not become Python types until serialized.
+Performance is dependent on the Ibis backend, which defaults to DuckDB. There are no internal Python loops. Scalars do not become Python types until serialized. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
 
-[PyArrow](https://arrow.apache.org/docs/python/) is also used for partitioned dataset optimizations, and for any feature which ibis does not support. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
+For partitioned parquet datasets, [DuckLake](https://ducklake.select) provides performance benefits over `ibis.read_parquet`; see `graphique.ducklake.{register,attach}` for helpers. `python -m graphique.partition` in `graphique[cli]` writes hive-layout parquet out-of-core.
 
 ## Installation
 ```console
-pip install graphique[server]
+pip install graphique[server,cli]
 ```
 
 ## Dependencies

graphique-2.2/graphique/__init__.py

@@ -0,0 +1,21 @@
+from .interface import Dataset
+from .middleware import GraphQL, MetricsExtension, implement, typed
+
+
+def fetch_record_batch(self, rows_per_batch):
+    return self.to_arrow_reader(rows_per_batch)
+
+
+def connect(database=":memory:", read_only=False, config={}):
+    return original(database, read_only, {"lambda_syntax": "ENABLE_SINGLE_ARROW", **config})
+
+
+try:
+    import duckdb
+except ImportError:  # pragma: no cover
+    pass
+else:  # ibis-project/ibis#12196
+    if not hasattr(duckdb.DuckDBPyConnection, "fetch_arrow_table"):  # pragma: no branch
+        duckdb.DuckDBPyConnection.fetch_arrow_table = duckdb.DuckDBPyConnection.to_arrow_table
+        duckdb.DuckDBPyConnection.fetch_record_batch = fetch_record_batch
+        original, duckdb.connect = duckdb.connect, connect  # type: ignore

graphique-2.2/graphique/ducklake.py

@@ -0,0 +1,41 @@
+"""DuckLake helpers."""
+
+import functools
+from collections.abc import Sequence
+
+import ibis
+import pyarrow.dataset as ds
+
+
+@functools.singledispatch
+def register(
+    source: str | Sequence[str],
+    table: str,
+    *partitioning: str,
+    catalog: str = ":memory:",
+    database: str = "lake.main",
+) -> ibis.Table:
+    """Attach a DuckLake catalog and register a parquet dataset."""
+    alias, _, schema = database.partition(".")
+    con = ibis.duckdb.connect(extensions=["ducklake"])
+    con.raw_sql(f"ATTACH 'ducklake:{catalog}' AS {alias}")
+    con.create_table(table, schema=con.read_parquet(source).schema(), database=database)
+    if columns := ", ".join(partitioning):
+        con.raw_sql(f"ALTER TABLE {database}.{table} SET PARTITIONED BY ({columns})")
+    parameters = alias, table, source, schema
+    con.raw_sql("CALL ducklake_add_data_files(?, ?, ?, schema => ?)", parameters=parameters)
+    return con.table(table, database=database)
+
+
+@register.register
+def _(source: ds.Dataset, table: str, filter: ds.Expression | None = None, **kwargs) -> ibis.Table:
+    paths = [frag.path for frag in source.get_fragments(filter)]
+    return register(paths, table, *source.partitioning.schema.names, **kwargs)
+
+
+def attach(table: str, catalog: str, database: str = "lake.main") -> ibis.Table:
+    """Attach a DuckLake catalog read-only and return a table."""
+    alias, _, _ = database.partition(".")
+    con = ibis.duckdb.connect(extensions=["ducklake"])
+    con.raw_sql(f"ATTACH 'ducklake:{catalog}' AS {alias} (READ_ONLY)")
+    return con.table(table, database=database)