graphique 1.0__tar.gz → 1.2__tar.gz

{graphique-1.0 → graphique-1.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: graphique
-Version: 1.0
+Version: 1.2
 Summary: GraphQL service for arrow tables and parquet data sets.
 Author-email: Aric Coady <aric.coady@gmail.com>
 License: Copyright 2022 Aric Coady
@@ -26,16 +26,16 @@ Classifier: Intended Audience :: Science/Research
 Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Operating System :: OS Independent
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.7
 Classifier: Programming Language :: Python :: 3.8
 Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
 Classifier: Topic :: Database :: Database Engines/Servers
 Classifier: Topic :: Internet :: WWW/HTTP :: HTTP Servers
 Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Typing :: Typed
-Requires-Python: >=3.7
+Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 Provides-Extra: server
 License-File: LICENSE.txt
@@ -57,7 +57,7 @@ License-File: LICENSE.txt
 % env PARQUET_PATH=... uvicorn graphique.service:app
 ```
 
-Open http://localhost:8000/graphql to try out the API in [GraphiQL](https://github.com/graphql/graphiql/tree/main/packages/graphiql#readme). There is a test fixture at `./tests/fixtures/zipcodes.parquet`.
+Open http://localhost:8000/ to try out the API in [GraphiQL](https://github.com/graphql/graphiql/tree/main/packages/graphiql#readme). There is a test fixture at `./tests/fixtures/zipcodes.parquet`.
 
 ```console
 % env PARQUET_PATH=... strawberry export-schema graphique.service:app.schema
@@ -88,16 +88,16 @@ app = GraphQL.federated({<name>: ds.dataset(...), ...}, keys={...})  # Tables on
 Start like any ASGI app.
 
 ```console
-uvicorn <package>:app
+uvicorn <module>:app
 ```
 
 Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
 
 ### API
 #### types
-* `Dataset`: an interface for an arrow dataset or table.
-* `Table`: implements the `Dataset` interface with a known schema.
-* `Column`: an arrow Column (a.k.a. ChunkedArray). Each arrow data type has a corresponding column implementation: Boolean, Int, Long, Float, Decimal, Date, DateTime, Time, Duration, Base64, String, List, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
+* `Dataset`: interface for an arrow dataset, scanner, or table.
+* `Table`: implements the `Dataset` interface. Adds typed `row`, `columns`, and `filter` fields from introspecting the schema.
+* `Column`: interface for an arrow column (a.k.a. ChunkedArray). Each arrow data type has a corresponding column implementation: Boolean, Int, Long, Float, Decimal, Date, Datetime, Time, Duration, Base64, String, List, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
 * `Row`: scalar fields. Arrow tables are column-oriented, and graphique encourages that usage for performance. A single `row` field is provided for convenience, but a field for a list of rows is not. Requesting parallel columns is far more efficient.
 
 #### selection
@@ -138,8 +138,8 @@ If index columns are detected in the schema metadata, then an initial `filter` w
 ```
 
 ## Dependencies
-* pyarrow >=10
-* strawberry-graphql[asgi] >=0.134.4
+* pyarrow >=12
+* strawberry-graphql[asgi,cli]
 * uvicorn (or other [ASGI server](https://asgi.readthedocs.io/en/latest/implementations.html))
 
 ## Tests
@@ -150,6 +150,23 @@ If index columns are detected in the schema metadata, then an initial `filter` w
 ```
 
 ## Changes
+1.2
+
+* Pyarrow >=12 required
+* Grouping fragments optimized
+* Group by empty columns
+* Batch sorting and grouping into lists
+
+1.1
+
+* Pyarrow >=11 required
+* Python >=3.8 required
+* Scannable functions added
+* List aggregations deprecated
+* Group by fragments
+* Month day nano interval array
+* `min` and `max` fields memory optimized
+
 1.0
 
 * Pyarrow >=10 required
@@ -216,9 +233,9 @@ If index columns are detected in the schema metadata, then an initial `filter` w
 
 0.2
 
+* Pyarrow >= 2 required
 * `ListColumn` and `StructColumn` types
 * `Groups` type with `aggregate` field
 * `group` and `unique` optimized
-* pyarrow >= 2 required
 * Statistical fields: `mode`, `stddev`, `variance`
 * `is_in`, `min`, and `max` optimized

{graphique-1.0 → graphique-1.2}/README.md

@@ -15,7 +15,7 @@
 % env PARQUET_PATH=... uvicorn graphique.service:app
 ```
 
-Open http://localhost:8000/graphql to try out the API in [GraphiQL](https://github.com/graphql/graphiql/tree/main/packages/graphiql#readme). There is a test fixture at `./tests/fixtures/zipcodes.parquet`.
+Open http://localhost:8000/ to try out the API in [GraphiQL](https://github.com/graphql/graphiql/tree/main/packages/graphiql#readme). There is a test fixture at `./tests/fixtures/zipcodes.parquet`.
 
 ```console
 % env PARQUET_PATH=... strawberry export-schema graphique.service:app.schema
@@ -46,16 +46,16 @@ app = GraphQL.federated({<name>: ds.dataset(...), ...}, keys={...})  # Tables on
 Start like any ASGI app.
 
 ```console
-uvicorn <package>:app
+uvicorn <module>:app
 ```
 
 Configuration options exist to provide a convenient no-code solution, but are subject to change in the future. Using a custom app is recommended for production usage.
 
 ### API
 #### types
-* `Dataset`: an interface for an arrow dataset or table.
-* `Table`: implements the `Dataset` interface with a known schema.
-* `Column`: an arrow Column (a.k.a. ChunkedArray). Each arrow data type has a corresponding column implementation: Boolean, Int, Long, Float, Decimal, Date, DateTime, Time, Duration, Base64, String, List, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
+* `Dataset`: interface for an arrow dataset, scanner, or table.
+* `Table`: implements the `Dataset` interface. Adds typed `row`, `columns`, and `filter` fields from introspecting the schema.
+* `Column`: interface for an arrow column (a.k.a. ChunkedArray). Each arrow data type has a corresponding column implementation: Boolean, Int, Long, Float, Decimal, Date, Datetime, Time, Duration, Base64, String, List, Struct. All columns have a `values` field for their list of scalars. Additional fields vary by type.
 * `Row`: scalar fields. Arrow tables are column-oriented, and graphique encourages that usage for performance. A single `row` field is provided for convenience, but a field for a list of rows is not. Requesting parallel columns is far more efficient.
 
 #### selection
@@ -96,8 +96,8 @@ If index columns are detected in the schema metadata, then an initial `filter` w
 ```
 
 ## Dependencies
-* pyarrow >=10
-* strawberry-graphql[asgi] >=0.134.4
+* pyarrow >=12
+* strawberry-graphql[asgi,cli]
 * uvicorn (or other [ASGI server](https://asgi.readthedocs.io/en/latest/implementations.html))
 
 ## Tests
@@ -108,6 +108,23 @@ If index columns are detected in the schema metadata, then an initial `filter` w
 ```
 
 ## Changes
+1.2
+
+* Pyarrow >=12 required
+* Grouping fragments optimized
+* Group by empty columns
+* Batch sorting and grouping into lists
+
+1.1
+
+* Pyarrow >=11 required
+* Python >=3.8 required
+* Scannable functions added
+* List aggregations deprecated
+* Group by fragments
+* Month day nano interval array
+* `min` and `max` fields memory optimized
+
 1.0
 
 * Pyarrow >=10 required
@@ -174,9 +191,9 @@ If index columns are detected in the schema metadata, then an initial `filter` w
 
 0.2
 
+* Pyarrow >= 2 required
 * `ListColumn` and `StructColumn` types
 * `Groups` type with `aggregate` field
 * `group` and `unique` optimized
-* pyarrow >= 2 required
 * Statistical fields: `mode`, `stddev`, `variance`
 * `is_in`, `min`, and `max` optimized

{graphique-1.0 → graphique-1.2}/graphique/__init__.py

@@ -1,3 +1,3 @@
 from .middleware import GraphQL  # noqa: F401
 
-__version__ = '1.0'
+__version__ = '1.2'

{graphique-1.0 → graphique-1.2}/graphique/core.py

@@ -7,17 +7,18 @@ Their methods are called as functions.
 import bisect
 import contextlib
 import functools
+import inspect
 import itertools
+import operator
 import json
-from concurrent import futures
 from dataclasses import dataclass
-from typing import Callable, Iterable, Iterator, Optional, Sequence, Union
+from typing import Callable, Iterable, Iterator, Optional, Sequence, Union, get_type_hints
 import numpy as np  # type: ignore
 import pyarrow as pa
 import pyarrow.compute as pc
 import pyarrow.dataset as ds
 
-threader = futures.ThreadPoolExecutor(pa.cpu_count())
+Array = Union[pa.Array, pa.ChunkedArray]
 
 
 class Agg:
@@ -51,7 +52,19 @@ class Agg:
         self.options = options
 
     def astuple(self, func: str) -> tuple:
-        return f'hash_{func}', self.option_map[func](**self.options)
+        options = self.option_map[func](**self.options)
+        return self.name, func, options
+
+    @classmethod
+    def getfunc(cls, name: str) -> Callable:
+        """Return callable with named parameters for keyword arguments."""
+        if name in ('first', 'last', 'one'):
+            return operator.itemgetter(-1 if name == 'last' else 0)
+        if name == 'element':
+            return lambda array, index: array[index]
+        if name == 'slice':
+            return lambda array, start, stop, step: array[start:stop:step]
+        return getattr(pc, name)
 
 
 @dataclass(frozen=True)
@@ -78,15 +91,34 @@ def decode(array: pa.Array) -> pa.Array:
     return array.dictionary_decode() if isinstance(array, pa.DictionaryArray) else array
 
 
-class ListChunk(pa.lib.BaseListArray):
-    value_length = pc.list_value_length
+def register(func: Callable) -> pc.Function:
+    """Register user defined scalar function."""
+    doc = inspect.getdoc(func)
+    doc = {'summary': doc.splitlines()[0], 'description': doc}  # type: ignore
+    annotations = dict(get_type_hints(func))
+    pc.register_scalar_function(func, func.__name__, doc, annotations, annotations.pop('return'))
+    return pc.get_function(func.__name__)
+
 
+@register
+def digitize(
+    ctx, array: pa.float64(), bins: pa.list_(pa.float64()), right: pa.bool_()  # type: ignore
+) -> pa.int64():  # type: ignore
+    """Return the indices of the bins to which each value in input array belongs."""
+    return pa.array(np.digitize(array, bins.values, right.as_py()))
+
+
+class ListChunk(pa.lib.BaseListArray):
     def from_counts(counts: pa.IntegerArray, values: pa.Array) -> pa.LargeListArray:
         """Return list array by converting counts into offsets."""
         offsets = pa.concat_arrays([pa.array([0], counts.type), pc.cumulative_sum_checked(counts)])
         cls = pa.LargeListArray if offsets.type == 'int64' else pa.ListArray
         return cls.from_arrays(offsets, values)
 
+    def from_scalars(values: Iterable) -> pa.LargeListArray:
+        """Return list array from array scalars."""
+        return ListChunk.from_counts(pa.array(map(len, values)), pa.concat_arrays(values))
+
     def element(self, index: int) -> pa.Array:
         """element at index of each list scalar; defaults to null"""
         with contextlib.suppress(ValueError):
@@ -113,31 +145,41 @@ class ListChunk(pa.lib.BaseListArray):
     def map_list(self, func: Callable, **kwargs) -> pa.lib.BaseListArray:
         """Return list array by mapping function across scalars, with null handling."""
         values = [func(value, **kwargs) for value in ListChunk.scalars(self)]
-        return ListChunk.from_counts(pa.array(map(len, values)), pa.concat_arrays(values))
+        return ListChunk.from_scalars(values)
 
-    def aggregate(self, **funcs: Optional[pc.FunctionOptions]) -> pa.StructArray:
+    def inner_flatten(self) -> pa.lib.BaseListArray:
+        """Return flattened inner lists from a nested list array."""
+        offsets = self.values.offsets.take(self.offsets)
+        return type(self).from_arrays(offsets, self.values.values)
+
+    def aggregate(self, **funcs: Optional[pc.FunctionOptions]) -> pa.RecordBatch:
         """Return aggregated scalars by grouping each hash function on the parent indices.
 
         If there are empty or null scalars, then the result must be padded with null defaults and
         reordered. If the function is a `count`, then the default is 0.
         """
-        items = {f'hash_{name}': funcs[name] for name in funcs}.items()
-        indices = pc.list_parent_indices(self)
-        groups = pc._group_by([pc.list_flatten(self)] * len(funcs), [indices], items)
-        if len(groups) == len(self):  # no empty or null scalars
-            return groups
+        columns = {'key': pc.list_parent_indices(self), '': pc.list_flatten(self)}
+        items = [('', name, funcs[name]) for name in funcs]
+        table = pa.table(columns).group_by(['key']).aggregate(items)
+        indices, table = table['key'], table.remove_column(table.schema.get_field_index('key'))
+        (batch,) = table.to_batches()
+        if len(batch) == len(self):  # no empty or null scalars
+            return batch
         mask = pc.equal(pc.list_value_length(self), 0)
         empties = pc.indices_nonzero(Column.fill_null(mask, True))
-        indices = pa.concat_arrays([groups.field(-1).cast('uint64'), empties])
-        counters = [field.name for field in groups.type if 'count' in field.name]
-        empties = pa.repeat(pa.scalar(dict.fromkeys(counters, 0), groups.type), len(empties))
-        return pa.concat_arrays([groups, empties]).take(pc.sort_indices(indices))
+        indices = pa.chunked_array(indices.chunks + [empties.cast(indices.type)])
+        columns = {}
+        for field in batch.schema:
+            scalar = pa.scalar(0 if 'count' in field.name else None, field.type)
+            columns[field.name] = pa.repeat(scalar, len(empties))
+        table = pa.concat_tables([table, pa.table(columns)]).combine_chunks()
+        return table.to_batches()[0].take(pc.sort_indices(indices))
 
     def min_max(self, **options) -> pa.Array:
         if pa.types.is_dictionary(self.type.value_type):
-            self = ListChunk.aggregate(self, distinct=None).field(0)
+            (self,) = ListChunk.aggregate(self, distinct=None)
             self = type(self).from_arrays(self.offsets, self.values.dictionary_decode())
-        return ListChunk.aggregate(self, min_max=pc.ScalarAggregateOptions(**options)).field(0)
+        return ListChunk.aggregate(self, min_max=pc.ScalarAggregateOptions(**options))[0]
 
     def min(self, **options) -> pa.Array:
         """min value of each list scalar"""
@@ -157,51 +199,45 @@ class ListChunk(pa.lib.BaseListArray):
 
     def index(self, **options) -> pa.Array:
         """index for first occurrence of each list scalar"""
-        return pa.array(pc.index(value, **options).as_py() for value in ListChunk.scalars(self))
+        values = [pc.index(value, **options) for value in ListChunk.scalars(self)]
+        return Column.from_scalars(values)
+
+    @register
+    def list_all(ctx, self: pa.list_(pa.bool_())) -> pa.bool_():  # type: ignore
+        """Test whether all elements in a boolean array evaluate to true."""
+        return ListChunk.aggregate(self, all=None)[0]
+
+    @register
+    def list_any(ctx, self: pa.list_(pa.bool_())) -> pa.bool_():  # type: ignore
+        """Test whether any element in a boolean array evaluates to true."""
+        return ListChunk.aggregate(self, any=None)[0]
 
 
 class Column(pa.ChunkedArray):
     """Chunked array interface as a namespace of functions."""
 
-    def map(self, func: Callable, **kwargs) -> pa.ChunkedArray:
-        map_ = threader.map if self.num_chunks > 1 else map
-        return pa.chunked_array(map_(functools.partial(func, **kwargs), self.iterchunks()))  # type: ignore
+    def from_scalars(values: Sequence) -> pa.Array:
+        """Return array from arrow scalars."""
+        return pa.array((value.as_py() for value in values), values[0].type)
 
     def scalar_type(self):
         return self.type.value_type if pa.types.is_dictionary(self.type) else self.type
 
     def is_list_type(self):
-        return pa.types.is_list(self.type) or pa.types.is_large_list(self.type)
+        funcs = pa.types.is_list, pa.types.is_large_list, pa.types.is_fixed_size_list
+        return any(func(self.type) for func in funcs)
 
-    def combine_dictionaries(self) -> tuple:
+    def call_indices(self, func: Callable) -> Array:
         if not pa.types.is_dictionary(self.type):
-            return self, None
-        if not self or len(self) <= max(len(chunk.dictionary) for chunk in self.iterchunks()):
-            return self.cast(self.type.value_type), None
-        self = self.unify_dictionaries()
-        indices = pa.chunked_array(chunk.indices for chunk in self.iterchunks())
-        return self.chunk(0).dictionary, indices
-
-    def indices(self) -> Union[pa.Array, pa.ChunkedArray]:
-        """Return chunked indices suitable for aggregation."""
-        if isinstance(self, pa.Array):
-            return pa.array(np.arange(len(self)))
-        offsets = list(itertools.accumulate(map(len, self.iterchunks())))
-        return pa.chunked_array(map(np.arange, [0] + offsets, offsets))
-
-    def call_indices(self, func: Callable) -> pa.ChunkedArray:
-        dictionary, indices = Column.combine_dictionaries(self)
-        if indices is None:
-            return func(dictionary)
-        return pa.chunked_array(
-            pa.DictionaryArray.from_arrays(chunk, dictionary) for chunk in func(indices).chunks
-        )
+            return func(self)
+        array = self.combine_chunks()
+        return pa.DictionaryArray.from_arrays(func(array.indices), array.dictionary)
 
-    def fill_null_backward(self) -> pa.ChunkedArray:
+    def fill_null_backward(self) -> Array:
         """`fill_null_backward` with dictionary support."""
         return Column.call_indices(self, pc.fill_null_backward)
 
-    def fill_null_forward(self) -> pa.ChunkedArray:
+    def fill_null_forward(self) -> Array:
         """`fill_null_forward` with dictionary support."""
         return Column.call_indices(self, pc.fill_null_forward)
 
@@ -209,11 +245,11 @@ class Column(pa.ChunkedArray):
         """Optimized `fill_null` to check `null_count`."""
         return self.fill_null(value) if self.null_count else self
 
-    def sort_values(self) -> pa.Array:
-        dictionary, indices = Column.combine_dictionaries(self)
-        if indices is None:
-            return dictionary
-        return pc.sort_indices(pc.sort_indices(dictionary)).take(indices)
+    def sort_values(self) -> Array:
+        if not pa.types.is_dictionary(self.type):
+            return self
+        array = self if isinstance(self, pa.Array) else self.combine_chunks()
+        return pc.sort_indices(pc.sort_indices(array.dictionary)).take(array.indices)
 
     def diff(self, func: Callable = pc.subtract) -> pa.ChunkedArray:
         """Return discrete differences between adjacent values."""
@@ -224,7 +260,7 @@ class Column(pa.ChunkedArray):
 
         Args:
             predicate: binary function applied to adjacent values
-            args: apply binary function to scalar, using `subtract` as the difference function
+            *args: apply binary function to scalar, using `subtract` as the difference function
         """
         ends = [pa.array([True])]
         mask = predicate(Column.diff(self), *args) if args else Column.diff(self, predicate)
@@ -243,12 +279,6 @@ class Column(pa.ChunkedArray):
         """Return max of the values."""
         return Column.min_max(self, **options)['max']
 
-    def digitize(self, bins: Iterable, right=False) -> pa.ChunkedArray:
-        """Return the indices of the bins to which each value in input array belongs."""
-        if not isinstance(bins, (pa.Array, np.ndarray)):
-            bins = pa.array(bins, self.type)
-        return Column.map(self, np.digitize, bins=bins, right=bool(right))
-
     def index(self, value, start=0, end=None) -> int:
         """Return the first index of a value."""
         with contextlib.suppress(NotImplementedError):
@@ -283,10 +313,10 @@ class Column(pa.ChunkedArray):
 class Table(pa.Table):
     """Table interface as a namespace of functions."""
 
-    def map_batch(self, func: Callable, *rargs, **kwargs) -> pa.Table:
-        return pa.Table.from_batches(
-            threader.map(lambda batch: func(batch, *rargs, **kwargs), self.to_batches())
-        )
+    def map_batch(scanner: ds.Scanner, func: Callable, *rargs, **kwargs) -> pa.Table:
+        # TODO(apache/arrow#31612): replace with user defined function for multiple kernels
+        batches = [func(batch, *rargs, **kwargs) for batch in scanner.to_batches() if batch]
+        return pa.Table.from_batches(batches, None if batches else scanner.projected_schema)
 
     def union(*tables: pa.Table) -> pa.Table:
         """Return table with union of columns."""
@@ -316,7 +346,7 @@ class Table(pa.Table):
         Assumes the table is sorted by the column name, i.e., indexed.
         """
         (slc,) = Column.find(self[name], value)
-        return pa.concat_tables([self[: slc.start], self[slc.stop :]])  # noqa: E203
+        return pa.concat_tables([self[: slc.start], self[slc.stop :]])
 
     def from_offsets(self, offsets: pa.IntegerArray) -> pa.Table:
         """Return table with columns converted into list columns."""
@@ -333,8 +363,8 @@ class Table(pa.Table):
         """Return table partitioned by discrete differences and corresponding counts.
 
         Args:
-            names: columns to partition by `not_equal` which will return scalars
-            predicates: inequality predicates with optional args which will return list arrays;
+            *names: columns to partition by `not_equal` which will return scalars
+            **predicates: inequality predicates with optional args which will return list arrays;
                 if the predicate has args, it will be called on the differences
         """
         offsets = pa.chunked_array(
@@ -358,65 +388,89 @@ class Table(pa.Table):
         """Group by and aggregate.
 
         Args:
-            names: columns to group by
+            *names: columns to group by
             counts: alias for optional row counts
             first: columns to take first value
             last: columns to take last value
-            funcs: aggregate funcs with columns options
+            **funcs: aggregate funcs with columns options
         """
-        none = not (counts or first or last or any(funcs.values()))
-        column = self[names[0]]
-        args = [(column, 'hash_count', pc.CountOptions(mode='all'))] if none or counts else []
+        if isinstance(self, pa.Table):
+            self = self.unify_dictionaries()
+        columns = {name: self[name] for name in self.schema.names}
+        aliases, args = {}, []  # type: ignore
+        if counts:
+            aliases['count_all'] = counts
+            args.append(([], 'count_all'))
         if first or last:
-            args.append((Column.indices(column), 'hash_min_max', None))
+            columns[''] = np.arange(len(self))
+            args.append(('', 'min_max'))
         lists: Sequence[Agg] = []
         list_cols = [self[agg.name] for agg in funcs.get('list', [])]
         if any(pa.types.is_dictionary(col.type) or Column.is_list_type(col) for col in list_cols):
-            args.append((Column.indices(column), 'hash_list', None))
+            columns[''] = np.arange(len(self))
+            args.append(('', 'list'))
             lists = funcs.pop('list')
-        for func in funcs:
-            args += [(self[agg.name], *agg.astuple(func)) for agg in funcs[func]]  # type: ignore
-        values, hashes, options = zip(*args)
-        keys = map(self.column, names)
-        if isinstance(self, pa.Table):
-            keys = (key.unify_dictionaries() for key in keys)  # type: ignore
-            values = (
-                value.unify_dictionaries() if 'distinct' in func else value
-                for value, func in zip(values, hashes)
-            )
-        arrays = iter(pc._group_by(values, keys, zip(hashes, options)).flatten())
-        if none:
-            next(arrays)
-        columns = {counts: next(arrays)} if counts else {}
+        for func, aggs in funcs.items():
+            aliases.update({f'{agg.name}_{func}': agg.alias for agg in aggs})
+            args += (agg.astuple(func) for agg in aggs)  # type: ignore
+        table = pa.table(columns).group_by(list(names)).aggregate(args)
+        columns = {name: table[name].combine_chunks() for name in table.schema.names}
         if first or last:
-            for aggs, indices in zip([first, last], next(arrays).flatten()):
+            for aggs, indices in zip([first, last], columns.pop('_min_max').flatten()):
                 columns.update({agg.alias: self[agg.name].take(indices) for agg in aggs})
         if lists:
-            indices = next(arrays)
+            indices = columns.pop('_list')
             table = self.select({agg.name for agg in lists}).take(indices.values)
             table = Table.from_offsets(table, indices.offsets)
-            columns.update({agg.alias: table[agg.name] for agg in lists})
-        for aggs in funcs.values():
-            columns.update(zip([agg.alias for agg in aggs], arrays))
-        columns.update(zip(names, map(decode, arrays)))
+            columns.update({agg.alias: table[agg.name].combine_chunks() for agg in lists})
+        columns.update({name: decode(columns[name]) for name in names})
+        columns = {aliases.get(name, name): columns[name] for name in columns}
         return type(self).from_pydict(columns)
 
+    def aggregate(self, counts: str = '', **funcs: Sequence[Agg]) -> dict:
+        """Return aggregated scalars as a row of data."""
+        row = {name: self[name] for name in self.column_names}
+        if counts:
+            row[counts] = len(self)
+        aliases, args = {}, []  # type: ignore
+        for key in funcs:
+            if key in Agg.option_map:
+                aliases.update({f'{agg.name}_{key}': agg.alias for agg in funcs[key]})
+                args += (agg.astuple(key) for agg in funcs[key])
+            else:
+                func = Agg.getfunc(key)
+                row.update({agg.alias: func(self[agg.name], **agg.options) for agg in funcs[key]})
+        if args:
+            table = self.group_by([]).aggregate(args)
+            row.update({aliases[name]: table[name][0] for name in table.column_names})
+        for name, value in row.items():
+            if isinstance(value, pa.ChunkedArray):
+                row[name] = value.combine_chunks()
+        return row
+
+    def list_fields(self) -> set:
+        return {field.name for field in self.schema if Column.is_list_type(field)}
+
     def list_value_length(self) -> pa.Array:
-        lists = {name for name in self.column_names if Column.is_list_type(self[name])}
+        lists = Table.list_fields(self)
         if not lists:
-            raise ValueError(f"no list columns available: {self.column_names}")
+            raise ValueError(f"no list columns available: {self.schema.names}")
         counts, *others = (pc.list_value_length(self[name]) for name in lists)
         if any(counts != other for other in others):
             raise ValueError(f"list columns have different value lengths: {lists}")
-        return counts.chunk(0)
+        return counts if isinstance(counts, pa.Array) else counts.chunk(0)
 
-    def sort_list(self, *names, length: int = None) -> pa.Table:
+    def sort_list(
+        self, *names: str, length: Optional[int] = None, null_placement: str = 'at_end'
+    ) -> pa.Table:
         """Return table with list columns sorted within scalars."""
         keys = dict(map(sort_key, names))  # type: ignore
         columns = {name: Column.sort_values(pc.list_flatten(self[name])) for name in keys}
         columns[''] = pc.list_parent_indices(self[next(iter(keys))])
         keys = dict({'': 'ascending'}, **keys)
-        indices = pc.sort_indices(pa.table(columns), sort_keys=keys.items())
+        indices = pc.sort_indices(
+            pa.table(columns), sort_keys=keys.items(), null_placement=null_placement
+        )
         counts = Table.list_value_length(self)
         if length is not None:
             indices = pa.concat_arrays(
@@ -428,27 +482,35 @@ class Table(pa.Table):
 
     def select_list(self, apply: Callable = lambda c: c) -> pa.Table:
         """Return table with only the list columns."""
-        names = [name for name in self.column_names if Column.is_list_type(self[name])]
+        names = list(Table.list_fields(self))
         return pa.table(list(map(apply, self.select(names))), names)
 
-    def sort_indices(self, *names, length: int = None) -> pa.Table:
+    def sort_indices(
+        self, *names: str, length: Optional[int] = None, null_placement: str = 'at_end'
+    ) -> pa.Array:
         """Return indices which would sort the table by columns, optimized for fixed length."""
-        func = pc.sort_indices
+        func = functools.partial(pc.sort_indices, null_placement=null_placement)
         if length is not None:
             func = functools.partial(pc.select_k_unstable, k=length)
         keys = dict(map(sort_key, names))  # type: ignore
         table = pa.table({name: Column.sort_values(self[name]) for name in keys})
         return func(table, sort_keys=keys.items())
 
-    def sort(self, *names, length: int = None, indices: str = '') -> pa.Table:
+    def sort(
+        self,
+        *names: str,
+        length: Optional[int] = None,
+        indices: str = '',
+        null_placement: str = 'at_end',
+    ) -> pa.Table:
         """Return table sorted by columns, optimized for fixed length.
 
         Args:
-            names: columns to sort by
+            *names: columns to sort by
             length: maximum number of rows to return
             indices: include original indices in the table
         """
-        indices_ = Table.sort_indices(self, *names, length=length)
+        indices_ = Table.sort_indices(self, *names, length=length, null_placement=null_placement)
         table = self.take(indices_)
         if indices:
             table = table.append_column(indices, indices_)
@@ -456,7 +518,7 @@ class Table(pa.Table):
         metadata = {'index_columns': list(itertools.takewhile(func, names))}
         return table.replace_schema_metadata({'pandas': json.dumps(metadata)})
 
-    def filter_list(self, expr: ds.Expression) -> 'Table':
+    def filter_list(self, expr: ds.Expression) -> pa.Table:
         """Return table with list columns filtered within scalars."""
         table = Table.select_list(self)
         counts = Table.list_value_length(table)
@@ -469,15 +531,16 @@ class Table(pa.Table):
         flattened = pa.table(columns, self.column_names)
         mask = ds.dataset(flattened).to_table(columns={'mask': expr})['mask'].combine_chunks()
         masks = type(first).from_arrays(first.offsets, mask)
-        counts = Column.fill_null(ListChunk.aggregate(masks, sum=None).field(0), 0)
+        counts = Column.fill_null(ListChunk.aggregate(masks, sum=None)[0], 0)
         flattened = flattened.select(table.column_names).filter(mask)
         return Table.union(self, Table.from_counts(flattened, counts))
 
-    def matched(self, func: Callable, *names: str) -> pa.Table:
-        for name in names:
+    def min_max(self, *names: str) -> pa.Table:
+        for name, order in map(sort_key, names):
+            func = Column.min if order == 'ascending' else Column.max
             if Column.is_list_type(self[name]):
                 self = self.append_column('', getattr(ListChunk, func.__name__)(self[name]))
                 self = Table.filter_list(self, pc.field(name) == pc.field('')).drop([''])
             else:
-                self = ds.dataset(self).to_table(filter=pc.field(name) == func(self[name]))
+                self = self.filter(pc.field(name) == func(self[name]))
         return self