dataframely 2.0.0__tar.gz → 2.3.1__tar.gz

dataframely-2.3.1/.github/copilot-instructions.md

@@ -0,0 +1,214 @@
+# Dataframely - Coding Agent Instructions
+
+## Project Overview
+
+Dataframely is a declarative, polars-native data frame validation library. It validates schemas and data content in
+polars DataFrames using native polars expressions and a custom Rust-based polars plugin for high performance. It
+supports validating individual data frames via `Schema` classes and interconnected data frames via `Collection` classes.
+
+## Tech Stack
+
+### Core Technologies
+
+- **Python**: Primary language for the public API
+- **Rust**: Backend for polars plugin and custom regex operations
+- **Polars**: Only supported data frame library
+- **pyo3 & maturin**: Rust-Python bindings and build system
+- **pixi**: Primary environment and task manager (NOT pip/conda directly)
+
+### Build System
+
+- **maturin**: Builds the Rust extension module `dataframely._native`
+- **Cargo**: Rust dependency management
+- Rust toolchain specified in `rust-toolchain.toml` with clippy and rustfmt components
+
+## Environment Setup
+
+**CRITICAL**: Always use `pixi` commands - never run `pip`, `conda`, `python`, or `cargo` directly unless specifically
+required for Rust-only operations.
+
+### Initial Setup
+
+Unless already performed via external setup steps:
+
+```bash
+# Install Rust toolchain
+rustup show
+
+# Install pixi environment and dependencies
+pixi install
+
+# Build and install the package locally (REQUIRED after Rust changes)
+pixi run postinstall
+```
+
+### After Rust Code Changes
+
+**Always run** `pixi run postinstall` after modifying any Rust code in `src/` to rebuild the native extension.
+
+## Development Workflow
+
+### Running Tests
+
+```bash
+# Run all tests (excludes S3 tests by default)
+pixi run test
+
+# Run tests with S3 backend (requires moto server)
+pixi run test -m s3
+
+# Run specific test file or directory
+pixi run test tests/schema/
+
+# Run with coverage
+pixi run test-coverage
+
+# Run benchmarks
+pixi run test-bench
+```
+
+### Code Quality
+
+**NEVER** run linters/formatters directly. Use pre-commit:
+
+```bash
+# Run all pre-commit hooks
+pixi run pre-commit run
+```
+
+Pre-commit handles:
+
+- **Python**: ruff (lint & format), mypy (type checking), docformatter
+- **Rust**: cargo fmt, cargo clippy
+- **Other**: prettier (md/yml), taplo (toml), license headers, trailing whitespace
+
+### Building Documentation
+
+```bash
+# Build documentation
+pixi run -e docs postinstall
+pixi run docs
+
+# Open in browser (macOS)
+open docs/_build/html/index.html
+```
+
+## Project Structure
+
+```
+dataframely/              # Python package
+  schema.py              # Core Schema class for DataFrame validation
+  collection/            # Collection class for validating multiple interconnected DataFrames
+  columns/               # Column type definitions (String, Integer, Float, etc.)
+  testing/               # Testing utilities (factories, masks, storage mocks)
+  _storage/              # Storage backends (Parquet, Delta Lake)
+  _rule.py               # Rule decorator for validation rules
+  _plugin.py             # Polars plugin registration
+  _native.pyi            # Type stubs for Rust extension
+
+src/                     # Rust source code
+  lib.rs                 # PyO3 module definition
+  polars_plugin/         # Custom polars plugin for validation
+  regex/                 # Custom regex operations
+
+tests/                   # Unit tests (mirrors dataframely/ structure)
+  benches/               # Benchmark tests
+  conftest.py            # Shared pytest fixtures (including s3_server)
+
+docs/                    # Sphinx documentation
+  guides/                # User guides and examples
+  api/                   # Auto-generated API reference
+```
+
+## Pixi Environments
+
+Multiple environments for different purposes:
+
+- **default**: Base Python + core dependencies
+- **dev**: Includes jupyter for notebooks
+- **test**: Testing dependencies (pytest, moto, boto3, etc.)
+- **docs**: Documentation building (sphinx, myst-parser, etc.)
+- **lint**: Linting and formatting tools
+- **optionals**: Optional dependencies (pydantic, deltalake, pyarrow, sqlalchemy)
+- **py310-py314**: Python version-specific environments
+
+Use `-e <env>` to run commands in specific environments:
+
+```bash
+pixi run -e test test
+pixi run -e docs docs
+```
+
+## API Design Principles
+
+### Critical Guidelines
+
+1. **NO BREAKING CHANGES**: Public API must remain backward compatible
+2. **100% Test Coverage**: All new code requires tests
+3. **Documentation Required**: All public features need docstrings + API docs
+4. **Cautious API Extension**: Avoid adding to public API unless necessary
+
+### Public API
+
+Public exports are in `dataframely/__init__.py`. Main components:
+
+- **Schema classes**: `Schema` for DataFrame validation
+- **Collection classes**: `Collection`, `CollectionMember` for multi-DataFrame validation
+- **Column types**: `String`, `Integer`, `Float`, `Bool`, `Date`, `Datetime`, etc.
+- **Decorators**: `@rule()`, `@filter()`
+- **Type hints**: `DataFrame[Schema]`, `LazyFrame[Schema]`, `Validation`
+
+## Common Pitfalls & Solutions
+
+### S3 Testing
+
+The `s3_server` fixture in `tests/conftest.py` uses `subprocess.Popen` to start moto_server on port 9999. This is a **workaround** for a polars issue with ThreadedMotoServer. When the polars issue is fixed, it should be replaced with ThreadedMotoServer (code is commented in the file).
+
+**Note**: CI skips S3 tests by default. Run with `pixi run test -m s3` when modifying storage backends.
+
+## Testing Strategy
+
+- Tests are organized by module, mirroring the `dataframely/` structure
+- Use `dy.Schema.sample()` for generating test data
+- Test both eager (`DataFrame`) and lazy (`LazyFrame`) execution
+- S3 tests use moto server fixture from `conftest.py`
+- Benchmark tests in `tests/benches/` use pytest-benchmark
+
+## Validation Pattern
+
+Typical usage pattern:
+
+```python
+class MySchema(dy.Schema):
+    col = dy.String(nullable=False)
+
+    @dy.rule()
+    def my_rule(cls) -> pl.Expr:
+        return pl.col("col").str.len_chars() > 0
+
+# Validate and cast
+validated_df: dy.DataFrame[MySchema] = MySchema.validate(df, cast=True)
+```
+
+## Key Configuration Files
+
+- `pixi.toml`: Environment and task definitions
+- `pyproject.toml`: Python package metadata, tool configurations (ruff, mypy, pytest)
+- `Cargo.toml`: Rust dependencies and build settings
+- `.pre-commit-config.yaml`: All code quality checks
+- `rust-toolchain.toml`: Rust nightly version specification
+
+## When Making Changes
+
+1. **Python code**: Run `pixi run pre-commit run` before committing
+2. **Rust code**: Run `pixi run postinstall` to rebuild, then run tests
+3. **Tests**: Ensure `pixi run test` passes. If changes might affect storage backends, use `pixi run test -m s3`.
+4. **Documentation**: Update docstrings
+5. **API changes**: Ensure backward compatibility or document migration path
+
+## Performance Considerations
+
+- Validation uses native polars expressions for performance
+- Custom Rust plugin for advanced validation logic
+- Lazy evaluation supported via `LazyFrame` for large datasets
+- Avoid materializing data unnecessarily in validation rules

dataframely-2.3.1/.github/instructions/tests.instructions.md

@@ -0,0 +1,9 @@
+---
+applyTo: tests/**/*.py
+---
+
+# Testing Guidelines
+
+- Instead of writing new tests, existing tests should ideally be parametrized using `@pytest.mark.parametrize`
+  unless the parametrization is impractical, e.g. by adding more parameters
+- Tests should not use docstrings unless they are _very_ complex

{dataframely-2.0.0 → dataframely-2.3.1}/.github/workflows/build.yml

@@ -17,7 +17,7 @@ jobs:
         with:
           fetch-depth: 0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: build
       - name: Set version
@@ -25,7 +25,7 @@ jobs:
       - name: Build project
         run: pixi run -e build build-sdist
       - name: Upload package
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         with:
           name: sdist
           path: dist/*
@@ -42,7 +42,7 @@ jobs:
           - target-platform: linux-aarch64
             os: ubuntu-24.04-arm
           - target-platform: osx-64
-            os: macos-13
+            os: macos-15-intel
           - target-platform: osx-arm64
             os: macos-latest
           - target-platform: win-64
@@ -52,7 +52,7 @@ jobs:
         with:
           fetch-depth: 0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: build
       - name: Set version
@@ -70,7 +70,7 @@ jobs:
       - name: Check package
         run: pixi run -e build check-wheel
       - name: Upload package
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         with:
           name: wheel-${{ matrix.target-platform }}
           path: dist/*
@@ -84,7 +84,7 @@ jobs:
       id-token: write
     environment: pypi
     steps:
-      - uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
+      - uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
         with:
           path: dist
           merge-multiple: true

{dataframely-2.0.0 → dataframely-2.3.1}/.github/workflows/ci.yml

@@ -24,7 +24,7 @@ jobs:
           # needed for 'pre-commit-mirrors-insert-license'
           fetch-depth: 0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: default lint
       - name: Install Rust
@@ -58,7 +58,7 @@ jobs:
       - name: Checkout branch
         uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: ${{ matrix.environment }}
           # FIXME: Remove when `s3_server` fixture does not start a process anymore

dataframely-2.3.1/.github/workflows/copilot-setup-steps.yml

@@ -0,0 +1,26 @@
+name: Copilot Setup Steps
+on:
+  pull_request:
+    paths:
+      - .github/workflows/copilot-setup-steps.yml
+  workflow_dispatch:
+
+jobs:
+  copilot-setup-steps:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - name: Checkout branch
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+      - name: Set up pixi
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
+        with:
+          environments: default
+      - name: Install Rust
+        run: rustup show
+      - name: Cache Rust dependencies
+        uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
+      - name: Install repository
+        run: pixi run postinstall

{dataframely-2.0.0 → dataframely-2.3.1}/.github/workflows/nightly.yml

@@ -25,7 +25,7 @@ jobs:
       - name: Checkout branch
         uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: nightly
       - name: Install polars nightly

{dataframely-2.0.0 → dataframely-2.3.1}/.github/workflows/scorecard.yml

@@ -65,7 +65,7 @@ jobs:
       # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
       # format to the repository Actions tab.
       - name: "Upload artifact"
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         with:
           name: SARIF file
           path: results.sarif
@@ -74,6 +74,6 @@ jobs:
       # Upload the results to GitHub's code scanning dashboard (optional).
       # Commenting out will disable upload of results to your repo's Code Scanning dashboard
       - name: "Upload to code-scanning"
-        uses: github/codeql-action/upload-sarif@3599b3baa15b485a2e49ef411a7a4bb2452e7f93 # v3.29.5
+        uses: github/codeql-action/upload-sarif@0499de31b99561a6d14a36a5f662c2a54f91beee # v3.29.5
         with:
           sarif_file: results.sarif

{dataframely-2.0.0 → dataframely-2.3.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataframely
-Version: 2.0.0
+Version: 2.3.1
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
@@ -9,7 +9,7 @@ Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Requires-Dist: fsspec>=2025.9
 Requires-Dist: numpy
-Requires-Dist: polars>=1.34
+Requires-Dist: polars>=1.36
 Requires-Dist: typing-extensions ; python_full_version < '3.11'
 Requires-Dist: deltalake ; extra == 'deltalake'
 Requires-Dist: pyarrow ; extra == 'pyarrow'
@@ -109,5 +109,5 @@ df = pl.DataFrame({
 validated_df: dy.DataFrame[HouseSchema] = HouseSchema.validate(df, cast=True)
 ```
 
-See more advanced usage examples in the [documentation](https://dataframely.readthedocs.io/en/latest/).
+See more advanced usage examples in the [documentation](https://dataframely.readthedocs.io/stable/).
 

{dataframely-2.0.0 → dataframely-2.3.1}/README.md

@@ -78,4 +78,4 @@ df = pl.DataFrame({
 validated_df: dy.DataFrame[HouseSchema] = HouseSchema.validate(df, cast=True)
 ```
 
-See more advanced usage examples in the [documentation](https://dataframely.readthedocs.io/en/latest/).
+See more advanced usage examples in the [documentation](https://dataframely.readthedocs.io/stable/).

{dataframely-2.0.0 → dataframely-2.3.1}/dataframely/__init__.py

@@ -51,6 +51,7 @@ from .columns import (
     UInt64,
 )
 from .config import Config
+from .exc import DeserializationError
 from .filter_result import FailureInfo
 from .functional import (
     concat_collection_members,
@@ -106,4 +107,5 @@ __all__ = [
     "Array",
     "Object",
     "Validation",
+    "DeserializationError",
 ]

{dataframely-2.0.0 → dataframely-2.3.1}/dataframely/_base_schema.py

@@ -8,7 +8,7 @@ import textwrap
 from abc import ABCMeta
 from copy import copy
 from dataclasses import dataclass, field
-from typing import Any
+from typing import TYPE_CHECKING, Any
 
 import polars as pl
 
@@ -164,12 +164,16 @@ class SchemaMeta(ABCMeta):
 
         return cls
 
-    def __getattribute__(cls, name: str) -> Any:
-        val = super().__getattribute__(name)
-        # Dynamically set the name of the column if it is a `Column` instance.
-        if isinstance(val, Column):
-            val._name = val.alias or name
-        return val
+    if not TYPE_CHECKING:
+        # Only define __getattribute__ at runtime to allow type checkers to properly
+        # validate attribute access. When TYPE_CHECKING is True, type checkers will use
+        # the default metaclass behavior which correctly identifies non-existent attributes.
+        def __getattribute__(cls, name: str) -> Any:
+            val = super().__getattribute__(name)
+            # Dynamically set the name of the column if it is a `Column` instance.
+            if isinstance(val, Column):
+                val._name = val.alias or name
+            return val
 
     @staticmethod
     def _get_metadata_recursively(kls: type[object]) -> Metadata:
@@ -199,9 +203,9 @@ class SchemaMeta(ABCMeta):
     def __repr__(cls) -> str:
         parts = [f'[Schema "{cls.__name__}"]']
         parts.append(textwrap.indent("Columns:", prefix=" " * 2))
-        for name, col in cls.columns().items():
+        for name, col in cls.columns().items():  # type: ignore[attr-defined]
             parts.append(textwrap.indent(f'- "{name}": {col!r}', prefix=" " * 4))
-        if validation_rules := cls._schema_validation_rules():
+        if validation_rules := cls._schema_validation_rules():  # type: ignore[attr-defined]
             parts.append(textwrap.indent("Rules:", prefix=" " * 2))
             for name, rule in validation_rules.items():
                 parts.append(textwrap.indent(f'- "{name}": {rule!r}', prefix=" " * 4))

{dataframely-2.0.0 → dataframely-2.3.1}/dataframely/_pydantic.py

@@ -20,12 +20,12 @@ _S = TypeVar("_S", bound=BaseSchema)
 def _dict_to_df(schema_type: type[BaseSchema], data: dict) -> pl.DataFrame:
     return pl.from_dict(
         data,
-        schema=schema_type.to_polars_schema(),
+        schema=schema_type.to_polars_schema(),  # type: ignore[attr-defined]
     )
 
 
 def _validate_df_schema(schema_type: type[_S], df: pl.DataFrame) -> DataFrame[_S]:
-    if not schema_type.is_valid(df):
+    if not schema_type.is_valid(df):  # type: ignore[attr-defined]
         raise ValueError("DataFrame violates schema")
     return df  # type: ignore
 

{dataframely-2.0.0 → dataframely-2.3.1}/dataframely/collection/_base.py

@@ -65,6 +65,11 @@ class CollectionMember:
     #: the collection's common primary key. Two members that share common column names
     #: may not both be inlined for sampling.
     inline_for_sampling: bool = False
+    #: Whether individual row failures in this member should be propagated to the
+    #: collection, i.e., cause the common primary key of the failures to be filtered
+    #: out from the entire collection. This setting is ignored if `ignored_in_filters`
+    #: is `True`.
+    propagate_row_failures: bool = False
 
 
 # --------------------------------------- UTILS -------------------------------------- #
@@ -250,6 +255,7 @@ class CollectionMeta(ABCMeta):
                 is_optional=True,
                 ignored_in_filters=collection_member.ignored_in_filters,
                 inline_for_sampling=collection_member.inline_for_sampling,
+                propagate_row_failures=collection_member.propagate_row_failures,
             )
         elif issubclass(origin, TypedLazyFrame):
             # Happy path: required member
@@ -258,6 +264,7 @@ class CollectionMeta(ABCMeta):
                 is_optional=False,
                 ignored_in_filters=collection_member.ignored_in_filters,
                 inline_for_sampling=collection_member.inline_for_sampling,
+                propagate_row_failures=collection_member.propagate_row_failures,
             )
         else:
             # Some other unknown annotation
@@ -333,6 +340,16 @@ class BaseCollection(metaclass=CollectionMeta):
             if not member.ignored_in_filters
         }
 
+    @classmethod
+    def _failure_propagating_members(cls) -> set[str]:
+        """The names of all members of the collection that propagate individual row
+        failures to the collection."""
+        return {
+            name
+            for name, member in cls.members().items()
+            if member.propagate_row_failures
+        }
+
     @classmethod
     def common_primary_key(cls) -> list[str]:
         """The primary keys shared by non ignored members of the collection."""