graphique 2.0.1__tar.gz → 2.1__tar.gz

{graphique-2.0.1 → graphique-2.1}/PKG-INFO

@@ -1,20 +1,19 @@
 Metadata-Version: 2.4
 Name: graphique
-Version: 2.0.1
+Version: 2.1
 Summary: GraphQL service for python dataframes and parquet datasets.
-Author-email: Aric Coady <aric.coady@gmail.com>
+Author: A. Coady
 License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/coady/graphique
 Project-URL: Documentation, https://coady.github.io/graphique
 Project-URL: Changelog, https://github.com/coady/graphique/blob/main/CHANGELOG.md
 Project-URL: Issues, https://github.com/coady/graphique/issues
-Keywords: graphql,arrow,parquet
+Keywords: graphql,ibis,dataframes,arrow,parquet
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
@@ -24,11 +23,11 @@ Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
 Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Typing :: Typed
-Requires-Python: >=3.10
+Requires-Python: >=3.11
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
-Requires-Dist: pyarrow>=20
-Requires-Dist: strawberry-graphql[asgi,cli]>=0.285
+Requires-Dist: pyarrow>=23
+Requires-Dist: strawberry-graphql[asgi,cli]>=0.288
 Requires-Dist: ibis-framework>=10.7
 Requires-Dist: pyarrow-hotfix
 Requires-Dist: isodate>=0.7
@@ -37,6 +36,7 @@ Requires-Dist: uvicorn[standard]; extra == "server"
 Requires-Dist: duckdb; extra == "server"
 Provides-Extra: cli
 Requires-Dist: tqdm; extra == "cli"
+Requires-Dist: typer; extra == "cli"
 Dynamic: license-file
 
 [![image](https://img.shields.io/pypi/v/graphique.svg)](https://pypi.org/project/graphique/)
@@ -47,7 +47,7 @@ Dynamic: license-file
 [![image](https://codecov.io/gh/coady/graphique/branch/main/graph/badge.svg)](https://codecov.io/gh/coady/graphique/)
 [![CodeQL](https://github.com/coady/graphique/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/coady/graphique/actions/workflows/github-code-scanning/codeql)
 [![image](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
-[![image](https://mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
+[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
 
 [GraphQL](https://graphql.org) service for [ibis](https://ibis-project.org) dataframes, [arrow](https://arrow.apache.org) tables, and [parquet](https://parquet.apache.org) datasets. The schema for a query API is derived automatically.
 
@@ -75,25 +75,27 @@ outputs the graphql schema.
 The example app uses [Starlette's config](https://www.starlette.io/config/): in environment variables or a `.env` file.
 
 * PARQUET_PATH: path to the parquet directory or file
-* FEDERATED = '': field name to extend type `Query` with a federated `Table`
-* METRICS = False: include timings from apollo tracing extension
+* NAME = '': GraphQL field on `Query`; defaults to root type
 * COLUMNS = None: list of names, or mapping of aliases, of columns to select
-* FILTERS = None: json `filter` query for which rows to read at startup
 
 Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
 
 ### App
 For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an ibis [Table](https://ibis-project.org/reference/expression-tables) or arrow [Dataset](https://arrow.apache.org/docs/python/api/dataset.html).
-Supply a mapping of names to datasets for multiple roots, and to enable federation.
+Use a `Query` type with dataset attributes for multiple roots, and to enable federation.
 
 ```python
 import ibis
-from graphique import GraphQL
+from graphique import GraphQL, typed
 
-source = ibis.read_*(...)  # or ibis.connect(...).table(...) or pyarrow.dataset.dataset(...)
+source = ibis.read_*(...)  # or `ibis.connect(...).table(...)` or `pyarrow.dataset.dataset(...)`
 # apply initial projections or filters to `source`
 app = GraphQL(source)  # Table is root query type
-app = GraphQL.federated({<name>: source, ...}, keys={<name>: [], ...})  # Tables on federated fields
+
+# multiple named fields, with optional federation keys
+class Query:
+    name = source  # or `typed(source, name, keys=...)`
+app = GraphQL(Query)
 ```
 
 Start like any ASGI app.
@@ -112,7 +114,8 @@ uvicorn <module>:app
 #### selection
 * `slice`: contiguous selection of rows
 * `filter`: select rows by predicates
-* `join`: join tables by key columns
+* `join`, `asofJoin`, `crossJoin`: join tables by key columns
+* `difference`, `intersect`, `union`: set operations on tables
 * `take`: rows by index
 * `dropNull`: remove rows with nulls
 
@@ -122,6 +125,7 @@ uvicorn <module>:app
 * `column`: access a column of any type by name
 * `row`: provides a field for each scalar of a single row
 * `cast`: cast column types
+* `unpack`: project struct fields
 * `fillNull`: fill null values
 
 #### aggregation
@@ -129,20 +133,26 @@ uvicorn <module>:app
 * `distinct`: group with all columns
 * `runs`: provisionally group by adjacency
 * `unnest`: unnest an array column
-* `count`: number of rows
+* `count`, `any`: number of rows
 
 #### ordering
 * `order`: sort table by given columns
-* options `limit` and `dense`: select rows with smallest or largest values
+* `first`: provisionally sort and filter by rank
+
+#### reflection
+* `type`: type of data source
+* `schema`: field names and types
+* `optional`: nullable for errors
+* `toSql`: compiles SQL query
 
 ### Performance
-Performance is dependent on the [ibis backend](https://ibis-project.org/backends/duckdb), which defaults to [duckdb](https://duckdb.org/). There are no internal Python loops. Scalars do not become Python types until serialized.
+Performance is dependent on the [ibis backend](https://ibis-project.org/backends/duckdb), which defaults to [duckdb](https://duckdb.org/). There are no internal Python loops. Scalars do not become Python types until serialized. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
 
-[PyArrow](https://arrow.apache.org/docs/python/) is also used for partitioned dataset optimizations, and for any feature which ibis does not support. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
+[PyArrow](https://arrow.apache.org/docs/python/) is also used for partitioned dataset optimizations. `python -m graphique.partition` is a command-line script provided in `graphique[cli]`, for out-of-core partitioning.
 
 ## Installation
 ```console
-pip install graphique[server]
+pip install graphique[server,cli]
 ```
 
 ## Dependencies

{graphique-2.0.1 → graphique-2.1}/README.md

@@ -6,7 +6,7 @@
 [![image](https://codecov.io/gh/coady/graphique/branch/main/graph/badge.svg)](https://codecov.io/gh/coady/graphique/)
 [![CodeQL](https://github.com/coady/graphique/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/coady/graphique/actions/workflows/github-code-scanning/codeql)
 [![image](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
-[![image](https://mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
+[![ty](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ty/main/assets/badge/v0.json)](https://github.com/astral-sh/ty)
 
 [GraphQL](https://graphql.org) service for [ibis](https://ibis-project.org) dataframes, [arrow](https://arrow.apache.org) tables, and [parquet](https://parquet.apache.org) datasets. The schema for a query API is derived automatically.
 
@@ -34,25 +34,27 @@ outputs the graphql schema.
 The example app uses [Starlette's config](https://www.starlette.io/config/): in environment variables or a `.env` file.
 
 * PARQUET_PATH: path to the parquet directory or file
-* FEDERATED = '': field name to extend type `Query` with a federated `Table`
-* METRICS = False: include timings from apollo tracing extension
+* NAME = '': GraphQL field on `Query`; defaults to root type
 * COLUMNS = None: list of names, or mapping of aliases, of columns to select
-* FILTERS = None: json `filter` query for which rows to read at startup
 
 Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
 
 ### App
 For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an ibis [Table](https://ibis-project.org/reference/expression-tables) or arrow [Dataset](https://arrow.apache.org/docs/python/api/dataset.html).
-Supply a mapping of names to datasets for multiple roots, and to enable federation.
+Use a `Query` type with dataset attributes for multiple roots, and to enable federation.
 
 ```python
 import ibis
-from graphique import GraphQL
+from graphique import GraphQL, typed
 
-source = ibis.read_*(...)  # or ibis.connect(...).table(...) or pyarrow.dataset.dataset(...)
+source = ibis.read_*(...)  # or `ibis.connect(...).table(...)` or `pyarrow.dataset.dataset(...)`
 # apply initial projections or filters to `source`
 app = GraphQL(source)  # Table is root query type
-app = GraphQL.federated({<name>: source, ...}, keys={<name>: [], ...})  # Tables on federated fields
+
+# multiple named fields, with optional federation keys
+class Query:
+    name = source  # or `typed(source, name, keys=...)`
+app = GraphQL(Query)
 ```
 
 Start like any ASGI app.
@@ -71,7 +73,8 @@ uvicorn <module>:app
 #### selection
 * `slice`: contiguous selection of rows
 * `filter`: select rows by predicates
-* `join`: join tables by key columns
+* `join`, `asofJoin`, `crossJoin`: join tables by key columns
+* `difference`, `intersect`, `union`: set operations on tables
 * `take`: rows by index
 * `dropNull`: remove rows with nulls
 
@@ -81,6 +84,7 @@ uvicorn <module>:app
 * `column`: access a column of any type by name
 * `row`: provides a field for each scalar of a single row
 * `cast`: cast column types
+* `unpack`: project struct fields
 * `fillNull`: fill null values
 
 #### aggregation
@@ -88,20 +92,26 @@ uvicorn <module>:app
 * `distinct`: group with all columns
 * `runs`: provisionally group by adjacency
 * `unnest`: unnest an array column
-* `count`: number of rows
+* `count`, `any`: number of rows
 
 #### ordering
 * `order`: sort table by given columns
-* options `limit` and `dense`: select rows with smallest or largest values
+* `first`: provisionally sort and filter by rank
+
+#### reflection
+* `type`: type of data source
+* `schema`: field names and types
+* `optional`: nullable for errors
+* `toSql`: compiles SQL query
 
 ### Performance
-Performance is dependent on the [ibis backend](https://ibis-project.org/backends/duckdb), which defaults to [duckdb](https://duckdb.org/). There are no internal Python loops. Scalars do not become Python types until serialized.
+Performance is dependent on the [ibis backend](https://ibis-project.org/backends/duckdb), which defaults to [duckdb](https://duckdb.org/). There are no internal Python loops. Scalars do not become Python types until serialized. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
 
-[PyArrow](https://arrow.apache.org/docs/python/) is also used for partitioned dataset optimizations, and for any feature which ibis does not support. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
+[PyArrow](https://arrow.apache.org/docs/python/) is also used for partitioned dataset optimizations. `python -m graphique.partition` is a command-line script provided in `graphique[cli]`, for out-of-core partitioning.
 
 ## Installation
 ```console
-pip install graphique[server]
+pip install graphique[server,cli]
 ```
 
 ## Dependencies

graphique-2.1/graphique/__init__.py

@@ -0,0 +1,2 @@
+from .interface import Dataset
+from .middleware import GraphQL, MetricsExtension, implement, typed

graphique-2.1/graphique/core.py

@@ -0,0 +1,100 @@
+"""
+Core arrow utilities.
+"""
+
+import bisect
+import itertools
+
+import ibis
+import pyarrow as pa
+import pyarrow.dataset as ds
+
+
+class links:
+    ref = "https://ibis-project.org/reference"
+    types = f"[data type]({ref}/datatypes)"
+    schema = f"[table schema]({ref}/schemas#ibis.expr.schema.Schema)"
+
+
+def getitems(obj, *keys):
+    """Nested `getitem`."""
+    for key in keys:
+        obj = obj[key]
+    return obj
+
+
+def order_key(name: str):
+    """Parse sort order."""
+    return (ibis.desc if name.startswith("-") else ibis.asc)(ibis._[name.lstrip("-")])
+
+
+def rank_over(
+    table: ibis.Table,
+    by: list[str],
+    over: list[str],
+    index: ibis.Column,
+    rank: int = 1,
+) -> ibis.Table:
+    """Filter rows by rank within each grouping window."""
+    order_by = list(map(order_key, by))
+    table = table.mutate(_=index.over(group_by=over, order_by=order_by))
+    return table.filter(table["_"] < rank).drop("_").order_by(*order_by)
+
+
+class Parquet(ds.Dataset):
+    """Partitioned parquet dataset."""
+
+    def schema(self) -> pa.Schema:
+        """partition schema"""
+        return self.partitioning.schema if hasattr(self, "partitioning") else pa.schema([])
+
+    def keys(self, *names) -> list:
+        """Return prefix of matching partition keys."""
+        keys = set(Parquet.schema(self).names)
+        return list(itertools.takewhile(lambda name: name.lstrip("-") in keys, names))
+
+    def fragments(self, counts: str = "", path: str = "__path__") -> ibis.Table:
+        """Return partition fragments as a table."""
+        parts = []
+        for frag in self.get_fragments():
+            parts.append(ds.get_partition_keys(frag.partition_expression))
+            parts[-1][path] = frag.path
+            if counts:
+                parts[-1][counts] = frag.count_rows()
+        return ibis.memtable(pa.Table.from_pylist(parts))
+
+    def filter(self, expr: ds.Expression | None) -> ds.Dataset | None:
+        """Attempt to apply filter to partition keys."""
+        try:  # raises ValueError if filter references non-partition keys
+            ds.dataset([], schema=self.partitioning.schema).scanner(filter=expr)
+        except (AttributeError, ValueError):
+            return None
+        if expr is None:
+            return self
+        paths = [frag.path for frag in self.get_fragments(expr)]
+        return ds.dataset(paths, partitioning=self.partitioning)
+
+    def to_table(self, name: str | None = None) -> ibis.Table:
+        """Return ibis `Table` from filtered dataset."""
+        paths = [frag.path for frag in self.get_fragments()]
+        return ibis.read_parquet(paths, table_name=name)
+
+    def order(self, *names: str, limit: int | None = None) -> ds.Dataset:
+        """Return ordered partitions of the dataset."""
+        table = Parquet.fragments(self, counts="" if limit is None else "_")
+        table = table.order_by(*map(order_key, names)).cache()
+        if limit is not None:
+            limit = bisect.bisect_left(table["_"].cumsum().to_list(), limit) + 1
+        return ds.dataset(table[:limit]["__path__"].to_list(), partitioning=self.partitioning)
+
+    def first(self, *names: str, rank: int = 1, dense: bool = False) -> ds.Dataset:
+        """Return ordered partitions up to max rank (dense or sparse)."""
+        keys = {name.strip("-"): order_key(name) for name in names}
+        if dense or rank == 1:
+            table = Parquet.fragments(self).aggregate(_=ibis._["__path__"].collect(), by=list(keys))
+            paths = table.order_by(*keys.values())[:rank]["_"].unnest()
+        else:
+            table = Parquet.fragments(self, counts="_").order_by(*keys.values()).cache()
+            limit = bisect.bisect_left(table["_"].cumsum().to_list(), rank) + 1
+            paths = table.semi_join(table[:limit].select(*keys).distinct(), list(keys))["__path__"]
+        return ds.dataset(paths.to_list(), partitioning=self.partitioning)