graphique 1.8__tar.gz → 2.0__tar.gz

{graphique-1.8 → graphique-2.0}/PKG-INFO

@@ -1,22 +1,9 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: graphique
-Version: 1.8
-Summary: GraphQL service for arrow tables and parquet data sets.
+Version: 2.0
+Summary: GraphQL service for python dataframes and parquet datasets.
 Author-email: Aric Coady <aric.coady@gmail.com>
-License: Copyright 2022 Aric Coady
-        
-           Licensed under the Apache License, Version 2.0 (the "License");
-           you may not use this file except in compliance with the License.
-           You may obtain a copy of the License at
-        
-             http://www.apache.org/licenses/LICENSE-2.0
-        
-           Unless required by applicable law or agreed to in writing, software
-           distributed under the License is distributed on an "AS IS" BASIS,
-           WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-           See the License for the specific language governing permissions and
-           limitations under the License.
-        
+License-Expression: Apache-2.0
 Project-URL: Homepage, https://github.com/coady/graphique
 Project-URL: Documentation, https://coady.github.io/graphique
 Project-URL: Changelog, https://github.com/coady/graphique/blob/main/CHANGELOG.md
@@ -25,10 +12,8 @@ Keywords: graphql,arrow,parquet
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: Intended Audience :: Science/Research
-Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
@@ -38,17 +23,20 @@ Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
 Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Typing :: Typed
-Requires-Python: >=3.9
+Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE.txt
-Requires-Dist: pyarrow>=18
+Requires-Dist: pyarrow>=20
 Requires-Dist: strawberry-graphql[asgi,cli]>=0.236
-Requires-Dist: numpy
+Requires-Dist: ibis-framework>=10.7
+Requires-Dist: pyarrow-hotfix
 Requires-Dist: isodate>=0.7
 Provides-Extra: server
 Requires-Dist: uvicorn[standard]; extra == "server"
+Requires-Dist: duckdb; extra == "server"
 Provides-Extra: cli
 Requires-Dist: tqdm; extra == "cli"
+Dynamic: license-file
 
 [![image](https://img.shields.io/pypi/v/graphique.svg)](https://pypi.org/project/graphique/)
 ![image](https://img.shields.io/pypi/pyversions/graphique.svg)
@@ -57,42 +45,52 @@ Requires-Dist: tqdm; extra == "cli"
 [![build](https://github.com/coady/graphique/actions/workflows/build.yml/badge.svg)](https://github.com/coady/graphique/actions/workflows/build.yml)
 [![image](https://codecov.io/gh/coady/graphique/branch/main/graph/badge.svg)](https://codecov.io/gh/coady/graphique/)
 [![CodeQL](https://github.com/coady/graphique/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/coady/graphique/actions/workflows/github-code-scanning/codeql)
-[![CodSpeed Badge](https://img.shields.io/endpoint?url=https://codspeed.io/badge.json)](https://codspeed.io/coady/graphique)
 [![image](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
 [![image](https://mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
 
-[GraphQL](https://graphql.org) service for [arrow](https://arrow.apache.org) tables and [parquet](https://parquet.apache.org) data sets. The schema for a query API is derived automatically.
+[GraphQL](https://graphql.org) service for [ibis](https://ibis-project.org) dataframes, [arrow](https://arrow.apache.org) tables, and [parquet](https://parquet.apache.org) datasets. The schema for a query API is derived automatically.
+
+## Version 2
+When this project started, there was no out-of-core execution engine with performance comparable to [PyArrow](https://arrow.apache.org/docs/python/index.html). So it effectively included one, based on datasets and [Acero](https://arrow.apache.org/docs/python/api/acero.html).
+
+Since then the ecosystem has grown considerably: [DuckDB](https://duckdb.org), [DataFusion](https://datafusion.apache.org), and [Ibis](https://ibis-project.org). As of version 2, graphique is based on `ibis`. It provides a common dataframe API for multiple backends, enabling graphique to also have a default but configurable backend.
+
+Being a major version upgrade, there are incompatible changes from version 1. However the overall API remains largely the same.
 
 ## Usage
+There is an example app which reads a parquet dataset.
 ```console
-% env PARQUET_PATH=... uvicorn graphique.service:app
+env PARQUET_PATH=... uvicorn graphique.service:app
 ```
 
 Open http://localhost:8000/ to try out the API in [GraphiQL](https://github.com/graphql/graphiql/tree/main/packages/graphiql#readme). There is a test fixture at `./tests/fixtures/zipcodes.parquet`.
 
 ```console
-% env PARQUET_PATH=... strawberry export-schema graphique.service:app.schema
+env PARQUET_PATH=... strawberry export-schema graphique.service:app.schema
 ```
-outputs the graphql schema for a parquet data set.
+outputs the graphql schema.
 
 ### Configuration
-Graphique uses [Starlette's config](https://www.starlette.io/config/): in environment variables or a `.env` file. Config variables are used as input to a [parquet dataset](https://arrow.apache.org/docs/python/dataset.html).
+The example app uses [Starlette's config](https://www.starlette.io/config/): in environment variables or a `.env` file.
 
 * PARQUET_PATH: path to the parquet directory or file
-* FEDERATED = '': field name to extend type `Query` with a federated `Table` 
-* DEBUG = False: run service in debug mode, which includes metrics
+* FEDERATED = '': field name to extend type `Query` with a federated `Table`
+* METRICS = False: include timings from apollo tracing extension
 * COLUMNS = None: list of names, or mapping of aliases, of columns to select
 * FILTERS = None: json `filter` query for which rows to read at startup
 
-For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an arrow [Dataset](https://arrow.apache.org/docs/python/api/dataset.html), [Scanner](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Scanner.html), or [Table](https://arrow.apache.org/docs/python/generated/pyarrow.Table.html). The GraphQL `Table` type will be the root Query type.
+Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
 
+### App
+For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an ibis [Table](https://ibis-project.org/reference/expression-tables) or arrow [Dataset](https://arrow.apache.org/docs/python/api/dataset.html).
 Supply a mapping of names to datasets for multiple roots, and to enable federation.
 
 ```python
-import pyarrow.dataset as ds
+import ibis
 from graphique import GraphQL
 
-source = ds.dataset(...)
+source = ibis.read_*(...)  # or ibis.connect(...).table(...) or pyarrow.dataset.dataset(...)
+# apply initial projections or filters to `source`
 app = GraphQL(source)  # Table is root query type
 app = GraphQL.federated({<name>: source, ...}, keys={<name>: [], ...})  # Tables on federated fields
 ```
@@ -103,55 +101,53 @@ Start like any ASGI app.
 uvicorn <module>:app
 ```
 
-Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
-
 ### API
 #### types
-* `Dataset`: interface for an arrow dataset, scanner, or table.
+* `Dataset`: interface for an ibis table or arrow dataset.
 * `Table`: implements the `Dataset` interface. Adds typed `row`, `columns`, and `filter` fields from introspecting the schema.
-* `Column`: interface for an arrow column (a.k.a. ChunkedArray). Each arrow data type has a corresponding column implementation: Boolean, Int, Long, Float, Decimal, Date, Datetime, Time, Duration, Base64, String, List, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
-* `Row`: scalar fields. Arrow tables are column-oriented, and graphique encourages that usage for performance. A single `row` field is provided for convenience, but a field for a list of rows is not. Requesting parallel columns is far more efficient.
+* `Column`: interface for an ibis column. Each data type has a corresponding column implementation: Boolean, Int, BigInt, Float, Decimal, Date, Datetime, Time, Duration, Base64, String, Array, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
+* `Row`: scalar fields. Tables are column-oriented, and graphique encourages that usage for performance. A single `row` field is provided for convenience, but a field for a list of rows is not. Requesting parallel columns is far more efficient.
 
 #### selection
 * `slice`: contiguous selection of rows
-* `filter`: select rows with simple predicates
-* `scan`: select rows and project columns with expressions
+* `filter`: select rows by predicates
+* `join`: join tables by key columns
+* `take`: rows by index
+* `dropNull`: remove rows with nulls
 
 #### projection
+* `project`: project columns with expressions
 * `columns`: provides a field for every `Column` in the schema
 * `column`: access a column of any type by name
 * `row`: provides a field for each scalar of a single row
-* `apply`: transform columns by applying a function
-* `join`: join tables by key columns
+* `cast`: cast column types
+* `fillNull`: fill null values
 
 #### aggregation
 * `group`: group by given columns, and aggregate the others
-* `runs`: partition on adjacent values in given columns, transforming the others into list columns
-* `tables`: return a list of tables by splitting on the scalars in list columns
-* `flatten`: flatten list columns with repeated scalars
+* `distinct`: group with all columns
+* `runs`: provisionally group by adjacency
+* `unnest`: unnest an array column
+* `count`: number of rows
 
 #### ordering
-* `sort`: sort table by given columns
-* `rank`: select rows with smallest or largest values
+* `order`: sort table by given columns
+* options `limit` and `dense`: select rows with smallest or largest values
 
 ### Performance
-Graphique relies on native [PyArrow](https://arrow.apache.org/docs/python/index.html) routines wherever possible. Otherwise it falls back to using [NumPy](https://numpy.org/doc/stable/) or custom optimizations.
-
-By default, datasets are read on-demand, with only the necessary rows and columns scanned. Although graphique is a running service, [parquet is performant](https://arrow.apache.org/docs/python/generated/pyarrow.dataset.Dataset.html) at reading a subset of data. Optionally specify `FILTERS` in the json `filter` format to read a subset of rows at startup, trading-off memory for latency. An empty filter (`{}`) will read the whole table.
+Performance is dependent on the [ibis backend](https://ibis-project.org/backends/duckdb), which defaults to [duckdb](https://duckdb.org/). There are no internal Python loops. Scalars do not become Python types until serialized.
 
-Specifying `COLUMNS` will limit memory usage when reading at startup (`FILTERS`). There is little speed difference as unused columns are inherently ignored. Optional aliasing can also be used for camel casing.
-
-If index columns are detected in the schema metadata, then an initial `filter` will also attempt a binary search on tables.
+[PyArrow](https://arrow.apache.org/docs/python/) is also used for partitioned dataset optimizations, and for any feature which ibis does not support. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
 
 ## Installation
 ```console
-% pip install graphique[server]
+pip install graphique[server]
 ```
 
 ## Dependencies
-* pyarrow
+* ibis-framework (with duckdb or other backend)
 * strawberry-graphql[asgi,cli]
-* numpy
+* pyarrow
 * isodate
 * uvicorn (or other [ASGI server](https://asgi.readthedocs.io/en/latest/implementations.html))
 
@@ -159,5 +155,5 @@ If index columns are detected in the schema metadata, then an initial `filter` w
 100% branch coverage.
 
 ```console
-% pytest [--cov]
+pytest [--cov]
 ```

graphique-2.0/README.md

@@ -0,0 +1,119 @@
+[![image](https://img.shields.io/pypi/v/graphique.svg)](https://pypi.org/project/graphique/)
+![image](https://img.shields.io/pypi/pyversions/graphique.svg)
+[![image](https://pepy.tech/badge/graphique)](https://pepy.tech/project/graphique)
+![image](https://img.shields.io/pypi/status/graphique.svg)
+[![build](https://github.com/coady/graphique/actions/workflows/build.yml/badge.svg)](https://github.com/coady/graphique/actions/workflows/build.yml)
+[![image](https://codecov.io/gh/coady/graphique/branch/main/graph/badge.svg)](https://codecov.io/gh/coady/graphique/)
+[![CodeQL](https://github.com/coady/graphique/actions/workflows/github-code-scanning/codeql/badge.svg)](https://github.com/coady/graphique/actions/workflows/github-code-scanning/codeql)
+[![image](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/astral-sh/ruff/main/assets/badge/v2.json)](https://github.com/astral-sh/ruff)
+[![image](https://mypy-lang.org/static/mypy_badge.svg)](https://mypy-lang.org/)
+
+[GraphQL](https://graphql.org) service for [ibis](https://ibis-project.org) dataframes, [arrow](https://arrow.apache.org) tables, and [parquet](https://parquet.apache.org) datasets. The schema for a query API is derived automatically.
+
+## Version 2
+When this project started, there was no out-of-core execution engine with performance comparable to [PyArrow](https://arrow.apache.org/docs/python/index.html). So it effectively included one, based on datasets and [Acero](https://arrow.apache.org/docs/python/api/acero.html).
+
+Since then the ecosystem has grown considerably: [DuckDB](https://duckdb.org), [DataFusion](https://datafusion.apache.org), and [Ibis](https://ibis-project.org). As of version 2, graphique is based on `ibis`. It provides a common dataframe API for multiple backends, enabling graphique to also have a default but configurable backend.
+
+Being a major version upgrade, there are incompatible changes from version 1. However the overall API remains largely the same.
+
+## Usage
+There is an example app which reads a parquet dataset.
+```console
+env PARQUET_PATH=... uvicorn graphique.service:app
+```
+
+Open http://localhost:8000/ to try out the API in [GraphiQL](https://github.com/graphql/graphiql/tree/main/packages/graphiql#readme). There is a test fixture at `./tests/fixtures/zipcodes.parquet`.
+
+```console
+env PARQUET_PATH=... strawberry export-schema graphique.service:app.schema
+```
+outputs the graphql schema.
+
+### Configuration
+The example app uses [Starlette's config](https://www.starlette.io/config/): in environment variables or a `.env` file.
+
+* PARQUET_PATH: path to the parquet directory or file
+* FEDERATED = '': field name to extend type `Query` with a federated `Table`
+* METRICS = False: include timings from apollo tracing extension
+* COLUMNS = None: list of names, or mapping of aliases, of columns to select
+* FILTERS = None: json `filter` query for which rows to read at startup
+
+Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
+
+### App
+For more options create a custom [ASGI](https://asgi.readthedocs.io/en/latest/index.html) app. Call graphique's `GraphQL` on an ibis [Table](https://ibis-project.org/reference/expression-tables) or arrow [Dataset](https://arrow.apache.org/docs/python/api/dataset.html).
+Supply a mapping of names to datasets for multiple roots, and to enable federation.
+
+```python
+import ibis
+from graphique import GraphQL
+
+source = ibis.read_*(...)  # or ibis.connect(...).table(...) or pyarrow.dataset.dataset(...)
+# apply initial projections or filters to `source`
+app = GraphQL(source)  # Table is root query type
+app = GraphQL.federated({<name>: source, ...}, keys={<name>: [], ...})  # Tables on federated fields
+```
+
+Start like any ASGI app.
+
+```console
+uvicorn <module>:app
+```
+
+### API
+#### types
+* `Dataset`: interface for an ibis table or arrow dataset.
+* `Table`: implements the `Dataset` interface. Adds typed `row`, `columns`, and `filter` fields from introspecting the schema.
+* `Column`: interface for an ibis column. Each data type has a corresponding column implementation: Boolean, Int, BigInt, Float, Decimal, Date, Datetime, Time, Duration, Base64, String, Array, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
+* `Row`: scalar fields. Tables are column-oriented, and graphique encourages that usage for performance. A single `row` field is provided for convenience, but a field for a list of rows is not. Requesting parallel columns is far more efficient.
+
+#### selection
+* `slice`: contiguous selection of rows
+* `filter`: select rows by predicates
+* `join`: join tables by key columns
+* `take`: rows by index
+* `dropNull`: remove rows with nulls
+
+#### projection
+* `project`: project columns with expressions
+* `columns`: provides a field for every `Column` in the schema
+* `column`: access a column of any type by name
+* `row`: provides a field for each scalar of a single row
+* `cast`: cast column types
+* `fillNull`: fill null values
+
+#### aggregation
+* `group`: group by given columns, and aggregate the others
+* `distinct`: group with all columns
+* `runs`: provisionally group by adjacency
+* `unnest`: unnest an array column
+* `count`: number of rows
+
+#### ordering
+* `order`: sort table by given columns
+* options `limit` and `dense`: select rows with smallest or largest values
+
+### Performance
+Performance is dependent on the [ibis backend](https://ibis-project.org/backends/duckdb), which defaults to [duckdb](https://duckdb.org/). There are no internal Python loops. Scalars do not become Python types until serialized.
+
+[PyArrow](https://arrow.apache.org/docs/python/) is also used for partitioned dataset optimizations, and for any feature which ibis does not support. Table fields are lazily evaluated up until scalars are reached, and automatically cached as needed for multiple fields.
+
+## Installation
+```console
+pip install graphique[server]
+```
+
+## Dependencies
+* ibis-framework (with duckdb or other backend)
+* strawberry-graphql[asgi,cli]
+* pyarrow
+* isodate
+* uvicorn (or other [ASGI server](https://asgi.readthedocs.io/en/latest/implementations.html))
+
+## Tests
+100% branch coverage.
+
+```console
+pytest [--cov]
+```

graphique-2.0/graphique/__init__.py

@@ -0,0 +1,3 @@
+# ruff: noqa: F401
+from .interface import Dataset
+from .middleware import GraphQL

graphique-2.0/graphique/core.py

@@ -0,0 +1,83 @@
+"""
+Core arrow utilities.
+"""
+
+import itertools
+
+import ibis
+import pyarrow as pa
+import pyarrow.dataset as ds
+
+
+class links:
+    ref = "https://ibis-project.org/reference"
+    types = f"[data type]({ref}/datatypes)"
+    schema = f"[table schema]({ref}/schemas#ibis.expr.schema.Schema)"
+
+
+def getitems(obj, *keys):
+    """Nested `getitem`."""
+    for key in keys:
+        obj = obj[key]
+    return obj
+
+
+def order_key(name: str) -> ibis.Deferred:
+    """Parse sort order."""
+    return (ibis.desc if name.startswith('-') else ibis.asc)(ibis._[name.lstrip('-')])
+
+
+class Parquet(ds.Dataset):
+    """Partitioned parquet dataset."""
+
+    def schema(self) -> pa.Schema:
+        """partition schema"""
+        return self.partitioning.schema if hasattr(self, 'partitioning') else pa.schema([])
+
+    def keys(self, *names) -> list:
+        """Return prefix of matching partition keys."""
+        keys = set(Parquet.schema(self).names)
+        return list(itertools.takewhile(lambda name: name.lstrip('-') in keys, names))
+
+    def fragments(self, counts: str = '') -> ibis.Table:
+        """Return partition fragments as a table."""
+        parts = []
+        for frag in self._get_fragments(self._scan_options.get('filter')):
+            parts.append(ds.get_partition_keys(frag.partition_expression))
+            parts[-1]['__path__'] = frag.path
+            if counts:
+                parts[-1][counts] = frag.count_rows()
+        return ibis.memtable(pa.Table.from_pylist(parts))
+
+    def group(self, *names, counts: str = '') -> ibis.Table:
+        """Return grouped partitions as a table."""
+        table = Parquet.fragments(self, counts)
+        agg = {counts: table[counts].sum()} if counts else {}
+        return table.aggregate(agg, by=names).order_by(*names)
+
+    def filter(self, expr: ds.Expression) -> ds.Dataset | None:
+        """Attempt to apply filter to partition keys."""
+        try:  # raises ValueError if filter references non-partition keys
+            ds.dataset([], schema=self.partitioning.schema).scanner(filter=expr)
+        except (AttributeError, ValueError):
+            return None
+        return self if expr is None else self.filter(expr)
+
+    def to_table(self) -> ibis.Table:
+        """Return ibis `Table` from filtered dataset."""
+        paths = [frag.path for frag in self._get_fragments(self._scan_options.get('filter'))]
+        hive = isinstance(self.partitioning, ds.HivePartitioning)
+        return ibis.read_parquet(paths, hive_partitioning=hive)
+
+    def rank(self, limit: int, *names: str, dense: bool = False) -> ibis.Table:
+        """Return ordered limited partitions as a table."""
+        keys = {name.strip('-'): order_key(name) for name in names}
+        table = Parquet.fragments(self, counts='_').order_by(*keys.values()).cache()
+        groups = table.aggregate(count=ibis._.count(), total=table['_'].sum(), by=list(keys))
+        groups = groups.order_by(*keys.values()).cache()
+        if not dense:
+            totals = itertools.accumulate(groups['total'].to_list())
+            limit = next((index for index, total in enumerate(totals, 1) if total >= limit), None)  # type: ignore
+        limit = groups[:limit]['count'].sum().to_pyarrow().as_py()
+        hive = isinstance(self.partitioning, ds.HivePartitioning)
+        return ibis.read_parquet(table[:limit]['__path__'].to_list(), hive_partitioning=hive)