atdata 0.1.3b3__tar.gz → 0.1.3b4__tar.gz

{atdata-0.1.3b3 → atdata-0.1.3b4}/.github/workflows/uv-test.yml

@@ -15,6 +15,8 @@ jobs:
   uv-test:
     name: Run tests
     runs-on: ubuntu-latest
+    environment:
+      name: test
 
     steps:
       - uses: actions/checkout@v5
@@ -32,9 +34,15 @@ jobs:
         # TODO Better to use --locked for author control over versions?
         # run: uv sync --locked --all-extras --dev
 
-      - name: Run tests
-        # For example, using `pytest`
-        run: uv run pytest tests
+      - name: Run tests with coverage
+        run: uv run pytest --cov=atdata --cov-report=xml --cov-report=term
+
+      - name: Upload coverage to Codecov
+        uses: codecov/codecov-action@v5
+        with:
+          # file: ./coverage.xml # Claude hallucination -- fascinating!
+          fail_ci_if_error: false
+          token: ${{ secrets.CODECOV_TOKEN }}
 
 
 #

atdata-0.1.3b4/.vscode/settings.json

@@ -0,0 +1,10 @@
+{
+    "cSpell.words": [
+        "atdata",
+        "getattr",
+        "msgpack",
+        "pypi",
+        "pyproject",
+        "pytest"
+    ]
+}

atdata-0.1.3b4/CLAUDE.md

@@ -0,0 +1,149 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+`atdata` is a Python library that implements a loose federation of distributed, typed datasets built on top of WebDataset. It provides:
+
+- **Typed samples** with automatic serialization via msgpack
+- **Lens-based transformations** between different dataset schemas
+- **Batch aggregation** with automatic numpy array stacking
+- **WebDataset integration** for efficient large-scale dataset storage
+
+## Development Commands
+
+### Environment Setup
+```bash
+# Uses uv for dependency management
+python -m pip install uv  # if not already installed
+uv sync
+```
+
+### Testing
+```bash
+# Run all tests with coverage
+pytest
+
+# Run specific test file
+pytest tests/test_dataset.py
+pytest tests/test_lens.py
+
+# Run single test
+pytest tests/test_dataset.py::test_create_sample
+pytest tests/test_lens.py::test_lens
+```
+
+### Building
+```bash
+# Build the package
+uv build
+```
+
+## Architecture
+
+### Core Components
+
+The codebase has three main modules under `src/atdata/`:
+
+1. **dataset.py** - Core dataset and sample infrastructure
+   - `PackableSample`: Base class for samples that can be serialized with msgpack
+   - `Dataset[ST]`: Generic typed dataset wrapping WebDataset tar files
+   - `SampleBatch[DT]`: Automatic batching with attribute aggregation
+   - `@packable` decorator: Converts dataclasses into PackableSample subclasses
+
+2. **lens.py** - Type transformation system
+   - `Lens[S, V]`: Bidirectional transformations between sample types (getter/putter)
+   - `LensNetwork`: Singleton registry for lens transformations
+   - `@lens` decorator: Registers lens getters globally
+
+3. **_helpers.py** - Serialization utilities
+   - `array_to_bytes()` / `bytes_to_array()`: numpy array serialization
+
+### Key Design Patterns
+
+**Sample Type Definition**
+
+Two approaches for defining sample types:
+
+```python
+# Approach 1: Explicit inheritance
+@dataclass
+class MySample(atdata.PackableSample):
+    field1: str
+    field2: NDArray
+
+# Approach 2: Decorator (recommended)
+@atdata.packable
+class MySample:
+    field1: str
+    field2: NDArray
+```
+
+**NDArray Handling**
+
+Fields annotated as `NDArray` or `NDArray | None` are automatically:
+- Converted from bytes during deserialization
+- Converted to bytes during serialization (via `_helpers.array_to_bytes`)
+- Handled by `_ensure_good()` method in `PackableSample.__post_init__`
+
+**Lens Transformations**
+
+Lenses enable viewing datasets through different type schemas:
+
+```python
+@atdata.lens
+def my_lens(source: SourceType) -> ViewType:
+    return ViewType(...)
+
+@my_lens.putter
+def my_lens_put(view: ViewType, source: SourceType) -> SourceType:
+    return SourceType(...)
+
+# Use with datasets
+ds = atdata.Dataset[SourceType](url).as_type(ViewType)
+```
+
+The `LensNetwork` singleton (in `lens.py:183`) maintains a global registry of all lenses decorated with `@lens`.
+
+**Batch Aggregation**
+
+`SampleBatch` uses `__getattr__` magic to aggregate sample attributes:
+- For `NDArray` fields: stacks into numpy array with batch dimension
+- For other fields: creates list
+- Results are cached in `_aggregate_cache`
+
+### Dataset URLs
+
+Datasets use WebDataset brace-notation URLs:
+- Single shard: `path/to/file-000000.tar`
+- Multiple shards: `path/to/file-{000000..000009}.tar`
+
+### Important Implementation Details
+
+**Type Parameters**
+
+The codebase uses Python 3.12+ generics heavily:
+- `Dataset[ST]` where `ST` is the sample type
+- `SampleBatch[DT]` where `DT` is the sample type
+- Uses `__orig_class__.__args__[0]` at runtime to extract type parameters
+
+**Serialization Flow**
+
+1. Sample → `as_wds` property → dict with `__key__` and `msgpack` bytes
+2. Msgpack bytes created by `packed` property calling `_make_packable()` on fields
+3. Deserialization: `from_bytes()` → `from_data()` → `__init__` → `_ensure_good()`
+
+**WebDataset Integration**
+
+- Uses `wds.ShardWriter` / `wds.TarWriter` for writing
+- Dataset iteration via `wds.DataPipeline` with custom `wrap()` / `wrap_batch()` methods
+- Supports `ordered()` and `shuffled()` iteration modes
+
+## Testing Notes
+
+- Tests use parametrization heavily via `@pytest.mark.parametrize`
+- Test cases cover both decorator and inheritance syntax
+- Temporary WebDataset tar files created in `tmp_path` fixture
+- Tests verify both serialization and batch aggregation behavior
+- Lens tests verify well-behavedness (GetPut/PutGet laws)

atdata-0.1.3b4/PKG-INFO

@@ -0,0 +1,172 @@
+Metadata-Version: 2.4
+Name: atdata
+Version: 0.1.3b4
+Summary: A loose federation of distributed, typed datasets
+Author-email: Maxine Levesque <hello@maxine.science>
+License-File: LICENSE
+Requires-Python: >=3.12
+Requires-Dist: fastparquet>=2024.11.0
+Requires-Dist: msgpack>=1.1.2
+Requires-Dist: numpy>=2.3.4
+Requires-Dist: ormsgpack>=1.11.0
+Requires-Dist: pandas>=2.3.3
+Requires-Dist: tqdm>=4.67.1
+Requires-Dist: webdataset>=1.0.2
+Description-Content-Type: text/markdown
+
+# atdata
+
+[![codecov](https://codecov.io/gh/foundation-ac/atdata/branch/main/graph/badge.svg)](https://codecov.io/gh/foundation-ac/atdata)
+
+A loose federation of distributed, typed datasets built on WebDataset.
+
+**atdata** provides a type-safe, composable framework for working with large-scale datasets. It combines the efficiency of WebDataset's tar-based storage with Python's type system and functional programming patterns.
+
+## Features
+
+- **Typed Samples** - Define dataset schemas using Python dataclasses with automatic msgpack serialization
+- **Lens Transformations** - Bidirectional, composable transformations between different dataset views
+- **Automatic Batching** - Smart batch aggregation with numpy array stacking
+- **WebDataset Integration** - Efficient storage and streaming for large-scale datasets
+
+## Installation
+
+```bash
+pip install atdata
+```
+
+Requires Python 3.12 or later.
+
+## Quick Start
+
+### Defining Sample Types
+
+Use the `@packable` decorator to create typed dataset samples:
+
+```python
+import atdata
+from numpy.typing import NDArray
+
+@atdata.packable
+class ImageSample:
+    image: NDArray
+    label: str
+    metadata: dict
+```
+
+### Creating Datasets
+
+```python
+# Create a dataset
+dataset = atdata.Dataset[ImageSample]("path/to/data-{000000..000009}.tar")
+
+# Iterate over samples in order
+for sample in dataset.ordered(batch_size=None):
+    print(f"Label: {sample.label}, Image shape: {sample.image.shape}")
+
+# Iterate with shuffling and batching
+for batch in dataset.shuffled(batch_size=32):
+    # batch.image is automatically stacked into shape (32, ...)
+    # batch.label is a list of 32 labels
+    process_batch(batch.image, batch.label)
+```
+
+### Lens Transformations
+
+Define reusable transformations between sample types:
+
+```python
+@atdata.packable
+class ProcessedSample:
+    features: NDArray
+    label: str
+
+@atdata.lens
+def preprocess(sample: ImageSample) -> ProcessedSample:
+    features = extract_features(sample.image)
+    return ProcessedSample(features=features, label=sample.label)
+
+# Apply lens to view dataset as ProcessedSample
+processed_ds = dataset.as_type(ProcessedSample)
+
+for sample in processed_ds.ordered(batch_size=None):
+    # sample is now a ProcessedSample
+    print(sample.features.shape)
+```
+
+## Core Concepts
+
+### PackableSample
+
+Base class for serializable samples. Fields annotated as `NDArray` are automatically handled:
+
+```python
+@atdata.packable
+class MySample:
+    array_field: NDArray      # Automatically serialized
+    optional_array: NDArray | None
+    regular_field: str
+```
+
+### Lens
+
+Bidirectional transformations with getter/putter semantics:
+
+```python
+@atdata.lens
+def my_lens(source: SourceType) -> ViewType:
+    # Transform source -> view
+    return ViewType(...)
+
+@my_lens.putter
+def my_lens_put(view: ViewType, source: SourceType) -> SourceType:
+    # Transform view -> source
+    return SourceType(...)
+```
+
+### Dataset URLs
+
+Uses WebDataset brace expansion for sharded datasets:
+
+- Single file: `"data/dataset-000000.tar"`
+- Multiple shards: `"data/dataset-{000000..000099}.tar"`
+- Multiple patterns: `"data/{train,val}/dataset-{000000..000009}.tar"`
+
+## Development
+
+### Setup
+
+```bash
+# Install uv if not already available
+python -m pip install uv
+
+# Install dependencies
+uv sync
+```
+
+### Testing
+
+```bash
+# Run all tests with coverage
+pytest
+
+# Run specific test file
+pytest tests/test_dataset.py
+
+# Run single test
+pytest tests/test_lens.py::test_lens
+```
+
+### Building
+
+```bash
+uv build
+```
+
+## Contributing
+
+Contributions are welcome! This project is in beta, so the API may still evolve.
+
+## License
+
+This project is licensed under the Mozilla Public License 2.0. See [LICENSE](LICENSE) for details.

atdata-0.1.3b4/README.md

@@ -0,0 +1,156 @@
+# atdata
+
+[![codecov](https://codecov.io/gh/foundation-ac/atdata/branch/main/graph/badge.svg)](https://codecov.io/gh/foundation-ac/atdata)
+
+A loose federation of distributed, typed datasets built on WebDataset.
+
+**atdata** provides a type-safe, composable framework for working with large-scale datasets. It combines the efficiency of WebDataset's tar-based storage with Python's type system and functional programming patterns.
+
+## Features
+
+- **Typed Samples** - Define dataset schemas using Python dataclasses with automatic msgpack serialization
+- **Lens Transformations** - Bidirectional, composable transformations between different dataset views
+- **Automatic Batching** - Smart batch aggregation with numpy array stacking
+- **WebDataset Integration** - Efficient storage and streaming for large-scale datasets
+
+## Installation
+
+```bash
+pip install atdata
+```
+
+Requires Python 3.12 or later.
+
+## Quick Start
+
+### Defining Sample Types
+
+Use the `@packable` decorator to create typed dataset samples:
+
+```python
+import atdata
+from numpy.typing import NDArray
+
+@atdata.packable
+class ImageSample:
+    image: NDArray
+    label: str
+    metadata: dict
+```
+
+### Creating Datasets
+
+```python
+# Create a dataset
+dataset = atdata.Dataset[ImageSample]("path/to/data-{000000..000009}.tar")
+
+# Iterate over samples in order
+for sample in dataset.ordered(batch_size=None):
+    print(f"Label: {sample.label}, Image shape: {sample.image.shape}")
+
+# Iterate with shuffling and batching
+for batch in dataset.shuffled(batch_size=32):
+    # batch.image is automatically stacked into shape (32, ...)
+    # batch.label is a list of 32 labels
+    process_batch(batch.image, batch.label)
+```
+
+### Lens Transformations
+
+Define reusable transformations between sample types:
+
+```python
+@atdata.packable
+class ProcessedSample:
+    features: NDArray
+    label: str
+
+@atdata.lens
+def preprocess(sample: ImageSample) -> ProcessedSample:
+    features = extract_features(sample.image)
+    return ProcessedSample(features=features, label=sample.label)
+
+# Apply lens to view dataset as ProcessedSample
+processed_ds = dataset.as_type(ProcessedSample)
+
+for sample in processed_ds.ordered(batch_size=None):
+    # sample is now a ProcessedSample
+    print(sample.features.shape)
+```
+
+## Core Concepts
+
+### PackableSample
+
+Base class for serializable samples. Fields annotated as `NDArray` are automatically handled:
+
+```python
+@atdata.packable
+class MySample:
+    array_field: NDArray      # Automatically serialized
+    optional_array: NDArray | None
+    regular_field: str
+```
+
+### Lens
+
+Bidirectional transformations with getter/putter semantics:
+
+```python
+@atdata.lens
+def my_lens(source: SourceType) -> ViewType:
+    # Transform source -> view
+    return ViewType(...)
+
+@my_lens.putter
+def my_lens_put(view: ViewType, source: SourceType) -> SourceType:
+    # Transform view -> source
+    return SourceType(...)
+```
+
+### Dataset URLs
+
+Uses WebDataset brace expansion for sharded datasets:
+
+- Single file: `"data/dataset-000000.tar"`
+- Multiple shards: `"data/dataset-{000000..000099}.tar"`
+- Multiple patterns: `"data/{train,val}/dataset-{000000..000009}.tar"`
+
+## Development
+
+### Setup
+
+```bash
+# Install uv if not already available
+python -m pip install uv
+
+# Install dependencies
+uv sync
+```
+
+### Testing
+
+```bash
+# Run all tests with coverage
+pytest
+
+# Run specific test file
+pytest tests/test_dataset.py
+
+# Run single test
+pytest tests/test_lens.py::test_lens
+```
+
+### Building
+
+```bash
+uv build
+```
+
+## Contributing
+
+Contributions are welcome! This project is in beta, so the API may still evolve.
+
+## License
+
+This project is licensed under the Mozilla Public License 2.0. See [LICENSE](LICENSE) for details.

{atdata-0.1.3b3 → atdata-0.1.3b4}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "atdata"
-version = "0.1.3b3"
+version = "0.1.3b4"
 description = "A loose federation of distributed, typed datasets"
 readme = "README.md"
 authors = [

atdata-0.1.3b4/src/atdata/__init__.py

@@ -0,0 +1,55 @@
+"""A loose federation of distributed, typed datasets.
+
+``atdata`` provides a typed dataset abstraction built on WebDataset, with support
+for:
+
+- **Typed samples** with automatic msgpack serialization
+- **NDArray handling** with transparent bytes conversion
+- **Lens transformations** for viewing datasets through different type schemas
+- **Batch aggregation** with automatic numpy array stacking
+- **WebDataset integration** for efficient large-scale dataset storage
+
+Quick Start:
+    >>> import atdata
+    >>> import numpy as np
+    >>>
+    >>> @atdata.packable
+    ... class MyData:
+    ...     features: np.ndarray
+    ...     label: str
+    >>>
+    >>> # Create dataset from WebDataset tar files
+    >>> ds = atdata.Dataset[MyData]("path/to/data-{000000..000009}.tar")
+    >>>
+    >>> # Iterate with automatic batching
+    >>> for batch in ds.shuffled(batch_size=32):
+    ...     features = batch.features  # numpy array (32, ...)
+    ...     labels = batch.label  # list of 32 strings
+
+Main Components:
+    - ``PackableSample``: Base class for msgpack-serializable samples
+    - ``Dataset``: Typed dataset wrapper for WebDataset
+    - ``SampleBatch``: Automatic batch aggregation
+    - ``Lens``: Bidirectional type transformations
+    - ``@packable``: Decorator for creating PackableSample classes
+    - ``@lens``: Decorator for creating lens transformations
+"""
+
+##
+# Expose components
+
+from .dataset import (
+    PackableSample,
+    SampleBatch,
+    Dataset,
+    packable,
+)
+
+from .lens import (
+    Lens,
+    LensNetwork,
+    lens,
+)
+
+
+#

atdata-0.1.3b4/src/atdata/_helpers.py

@@ -0,0 +1,58 @@
+"""Helper utilities for numpy array serialization.
+
+This module provides utility functions for converting numpy arrays to and from
+bytes for msgpack serialization. The functions use numpy's native save/load
+format to preserve array dtype and shape information.
+
+Functions:
+    - ``array_to_bytes()``: Serialize numpy array to bytes
+    - ``bytes_to_array()``: Deserialize bytes to numpy array
+
+These helpers are used internally by ``PackableSample`` to enable transparent
+handling of NDArray fields during msgpack packing/unpacking.
+"""
+
+##
+# Imports
+
+from io import BytesIO
+
+import numpy as np
+
+
+##
+
+def array_to_bytes( x: np.ndarray ) -> bytes:
+    """Convert a numpy array to bytes for msgpack serialization.
+
+    Uses numpy's native ``save()`` format to preserve array dtype and shape.
+
+    Args:
+        x: A numpy array to serialize.
+
+    Returns:
+        Raw bytes representing the serialized array.
+
+    Note:
+        Uses ``allow_pickle=True`` to support object dtypes.
+    """
+    np_bytes = BytesIO()
+    np.save( np_bytes, x, allow_pickle = True )
+    return np_bytes.getvalue()
+
+def bytes_to_array( b: bytes ) -> np.ndarray:
+    """Convert serialized bytes back to a numpy array.
+
+    Reverses the serialization performed by ``array_to_bytes()``.
+
+    Args:
+        b: Raw bytes from a serialized numpy array.
+
+    Returns:
+        The deserialized numpy array with original dtype and shape.
+
+    Note:
+        Uses ``allow_pickle=True`` to support object dtypes.
+    """
+    np_bytes = BytesIO( b )
+    return np.load( np_bytes, allow_pickle = True )