dataframely 2.0.0__tar.gz → 2.1.0__tar.gz

dataframely-2.1.0/.github/copilot-instructions.md

@@ -0,0 +1,214 @@
+# Dataframely - Coding Agent Instructions
+
+## Project Overview
+
+Dataframely is a declarative, polars-native data frame validation library. It validates schemas and data content in
+polars DataFrames using native polars expressions and a custom Rust-based polars plugin for high performance. It
+supports validating individual data frames via `Schema` classes and interconnected data frames via `Collection` classes.
+
+## Tech Stack
+
+### Core Technologies
+
+- **Python**: Primary language for the public API
+- **Rust**: Backend for polars plugin and custom regex operations
+- **Polars**: Only supported data frame library
+- **pyo3 & maturin**: Rust-Python bindings and build system
+- **pixi**: Primary environment and task manager (NOT pip/conda directly)
+
+### Build System
+
+- **maturin**: Builds the Rust extension module `dataframely._native`
+- **Cargo**: Rust dependency management
+- Rust toolchain specified in `rust-toolchain.toml` with clippy and rustfmt components
+
+## Environment Setup
+
+**CRITICAL**: Always use `pixi` commands - never run `pip`, `conda`, `python`, or `cargo` directly unless specifically
+required for Rust-only operations.
+
+### Initial Setup
+
+Unless already performed via external setup steps:
+
+```bash
+# Install Rust toolchain
+rustup show
+
+# Install pixi environment and dependencies
+pixi install
+
+# Build and install the package locally (REQUIRED after Rust changes)
+pixi run postinstall
+```
+
+### After Rust Code Changes
+
+**Always run** `pixi run postinstall` after modifying any Rust code in `src/` to rebuild the native extension.
+
+## Development Workflow
+
+### Running Tests
+
+```bash
+# Run all tests (excludes S3 tests by default)
+pixi run test
+
+# Run tests with S3 backend (requires moto server)
+pixi run test -m s3
+
+# Run specific test file or directory
+pixi run test tests/schema/
+
+# Run with coverage
+pixi run test-coverage
+
+# Run benchmarks
+pixi run test-bench
+```
+
+### Code Quality
+
+**NEVER** run linters/formatters directly. Use pre-commit:
+
+```bash
+# Run all pre-commit hooks
+pixi run pre-commit run
+```
+
+Pre-commit handles:
+
+- **Python**: ruff (lint & format), mypy (type checking), docformatter
+- **Rust**: cargo fmt, cargo clippy
+- **Other**: prettier (md/yml), taplo (toml), license headers, trailing whitespace
+
+### Building Documentation
+
+```bash
+# Build documentation
+pixi run -e docs postinstall
+pixi run docs
+
+# Open in browser (macOS)
+open docs/_build/html/index.html
+```
+
+## Project Structure
+
+```
+dataframely/              # Python package
+  schema.py              # Core Schema class for DataFrame validation
+  collection/            # Collection class for validating multiple interconnected DataFrames
+  columns/               # Column type definitions (String, Integer, Float, etc.)
+  testing/               # Testing utilities (factories, masks, storage mocks)
+  _storage/              # Storage backends (Parquet, Delta Lake)
+  _rule.py               # Rule decorator for validation rules
+  _plugin.py             # Polars plugin registration
+  _native.pyi            # Type stubs for Rust extension
+
+src/                     # Rust source code
+  lib.rs                 # PyO3 module definition
+  polars_plugin/         # Custom polars plugin for validation
+  regex/                 # Custom regex operations
+
+tests/                   # Unit tests (mirrors dataframely/ structure)
+  benches/               # Benchmark tests
+  conftest.py            # Shared pytest fixtures (including s3_server)
+
+docs/                    # Sphinx documentation
+  guides/                # User guides and examples
+  api/                   # Auto-generated API reference
+```
+
+## Pixi Environments
+
+Multiple environments for different purposes:
+
+- **default**: Base Python + core dependencies
+- **dev**: Includes jupyter for notebooks
+- **test**: Testing dependencies (pytest, moto, boto3, etc.)
+- **docs**: Documentation building (sphinx, myst-parser, etc.)
+- **lint**: Linting and formatting tools
+- **optionals**: Optional dependencies (pydantic, deltalake, pyarrow, sqlalchemy)
+- **py310-py314**: Python version-specific environments
+
+Use `-e <env>` to run commands in specific environments:
+
+```bash
+pixi run -e test test
+pixi run -e docs docs
+```
+
+## API Design Principles
+
+### Critical Guidelines
+
+1. **NO BREAKING CHANGES**: Public API must remain backward compatible
+2. **100% Test Coverage**: All new code requires tests
+3. **Documentation Required**: All public features need docstrings + API docs
+4. **Cautious API Extension**: Avoid adding to public API unless necessary
+
+### Public API
+
+Public exports are in `dataframely/__init__.py`. Main components:
+
+- **Schema classes**: `Schema` for DataFrame validation
+- **Collection classes**: `Collection`, `CollectionMember` for multi-DataFrame validation
+- **Column types**: `String`, `Integer`, `Float`, `Bool`, `Date`, `Datetime`, etc.
+- **Decorators**: `@rule()`, `@filter()`
+- **Type hints**: `DataFrame[Schema]`, `LazyFrame[Schema]`, `Validation`
+
+## Common Pitfalls & Solutions
+
+### S3 Testing
+
+The `s3_server` fixture in `tests/conftest.py` uses `subprocess.Popen` to start moto_server on port 9999. This is a **workaround** for a polars issue with ThreadedMotoServer. When the polars issue is fixed, it should be replaced with ThreadedMotoServer (code is commented in the file).
+
+**Note**: CI skips S3 tests by default. Run with `pixi run test -m s3` when modifying storage backends.
+
+## Testing Strategy
+
+- Tests are organized by module, mirroring the `dataframely/` structure
+- Use `dy.Schema.sample()` for generating test data
+- Test both eager (`DataFrame`) and lazy (`LazyFrame`) execution
+- S3 tests use moto server fixture from `conftest.py`
+- Benchmark tests in `tests/benches/` use pytest-benchmark
+
+## Validation Pattern
+
+Typical usage pattern:
+
+```python
+class MySchema(dy.Schema):
+    col = dy.String(nullable=False)
+
+    @dy.rule()
+    def my_rule(cls) -> pl.Expr:
+        return pl.col("col").str.len_chars() > 0
+
+# Validate and cast
+validated_df: dy.DataFrame[MySchema] = MySchema.validate(df, cast=True)
+```
+
+## Key Configuration Files
+
+- `pixi.toml`: Environment and task definitions
+- `pyproject.toml`: Python package metadata, tool configurations (ruff, mypy, pytest)
+- `Cargo.toml`: Rust dependencies and build settings
+- `.pre-commit-config.yaml`: All code quality checks
+- `rust-toolchain.toml`: Rust nightly version specification
+
+## When Making Changes
+
+1. **Python code**: Run `pixi run pre-commit run` before committing
+2. **Rust code**: Run `pixi run postinstall` to rebuild, then run tests
+3. **Tests**: Ensure `pixi run test` passes
+4. **Documentation**: Update docstrings
+5. **API changes**: Ensure backward compatibility or document migration path
+
+## Performance Considerations
+
+- Validation uses native polars expressions for performance
+- Custom Rust plugin for advanced validation logic
+- Lazy evaluation supported via `LazyFrame` for large datasets
+- Avoid materializing data unnecessarily in validation rules

dataframely-2.1.0/.github/instructions/tests.instructions.md

@@ -0,0 +1,9 @@
+---
+applyTo: tests/**/*.py
+---
+
+# Testing Guidelines
+
+- Instead of writing new tests, existing tests should ideally be parametrized using `@pytest.mark.parametrize`
+  unless the parametrization is impractical, e.g. by adding more parameters
+- Tests should not use docstrings unless they are _very_ complex

{dataframely-2.0.0 → dataframely-2.1.0}/.github/workflows/build.yml

@@ -17,7 +17,7 @@ jobs:
         with:
           fetch-depth: 0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: build
       - name: Set version
@@ -25,7 +25,7 @@ jobs:
       - name: Build project
         run: pixi run -e build build-sdist
       - name: Upload package
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         with:
           name: sdist
           path: dist/*
@@ -42,7 +42,7 @@ jobs:
           - target-platform: linux-aarch64
             os: ubuntu-24.04-arm
           - target-platform: osx-64
-            os: macos-13
+            os: macos-15-intel
           - target-platform: osx-arm64
             os: macos-latest
           - target-platform: win-64
@@ -52,7 +52,7 @@ jobs:
         with:
           fetch-depth: 0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: build
       - name: Set version
@@ -70,7 +70,7 @@ jobs:
       - name: Check package
         run: pixi run -e build check-wheel
       - name: Upload package
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         with:
           name: wheel-${{ matrix.target-platform }}
           path: dist/*
@@ -84,7 +84,7 @@ jobs:
       id-token: write
     environment: pypi
     steps:
-      - uses: actions/download-artifact@634f93cb2916e3fdff6788551b99b062d0335ce0 # v5.0.0
+      - uses: actions/download-artifact@018cc2cf5baa6db3ef3c5f8a56943fffe632ef53 # v6.0.0
         with:
           path: dist
           merge-multiple: true

{dataframely-2.0.0 → dataframely-2.1.0}/.github/workflows/ci.yml

@@ -24,7 +24,7 @@ jobs:
           # needed for 'pre-commit-mirrors-insert-license'
           fetch-depth: 0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: default lint
       - name: Install Rust
@@ -58,7 +58,7 @@ jobs:
       - name: Checkout branch
         uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: ${{ matrix.environment }}
           # FIXME: Remove when `s3_server` fixture does not start a process anymore

dataframely-2.1.0/.github/workflows/copilot-setup-steps.yml

@@ -0,0 +1,26 @@
+name: Copilot Setup Steps
+on:
+  pull_request:
+    paths:
+      - .github/workflows/copilot-setup-steps.yml
+  workflow_dispatch:
+
+jobs:
+  copilot-setup-steps:
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      id-token: write
+    steps:
+      - name: Checkout branch
+        uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
+      - name: Set up pixi
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
+        with:
+          environments: default
+      - name: Install Rust
+        run: rustup show
+      - name: Cache Rust dependencies
+        uses: Swatinem/rust-cache@f13886b937689c021905a6b90929199931d60db1 # v2.8.1
+      - name: Install repository
+        run: pixi run postinstall

{dataframely-2.0.0 → dataframely-2.1.0}/.github/workflows/nightly.yml

@@ -25,7 +25,7 @@ jobs:
       - name: Checkout branch
         uses: actions/checkout@08c6903cd8c0fde910a37f88322edcfb5dd907a8 # v5.0.0
       - name: Set up pixi
-        uses: prefix-dev/setup-pixi@194d461b21b6c5717c722ffc597fa91ed2ff29fa # v0.9.1
+        uses: prefix-dev/setup-pixi@28eb668aafebd9dede9d97c4ba1cd9989a4d0004 # v0.9.2
         with:
           environments: nightly
       - name: Install polars nightly

{dataframely-2.0.0 → dataframely-2.1.0}/.github/workflows/scorecard.yml

@@ -65,7 +65,7 @@ jobs:
       # Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
       # format to the repository Actions tab.
       - name: "Upload artifact"
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
+        uses: actions/upload-artifact@330a01c490aca151604b8cf639adc76d48f6c5d4 # v5.0.0
         with:
           name: SARIF file
           path: results.sarif
@@ -74,6 +74,6 @@ jobs:
       # Upload the results to GitHub's code scanning dashboard (optional).
       # Commenting out will disable upload of results to your repo's Code Scanning dashboard
       - name: "Upload to code-scanning"
-        uses: github/codeql-action/upload-sarif@3599b3baa15b485a2e49ef411a7a4bb2452e7f93 # v3.29.5
+        uses: github/codeql-action/upload-sarif@0499de31b99561a6d14a36a5f662c2a54f91beee # v3.29.5
         with:
           sarif_file: results.sarif

{dataframely-2.0.0 → dataframely-2.1.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: dataframely
-Version: 2.0.0
+Version: 2.1.0
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
@@ -109,5 +109,5 @@ df = pl.DataFrame({
 validated_df: dy.DataFrame[HouseSchema] = HouseSchema.validate(df, cast=True)
 ```
 
-See more advanced usage examples in the [documentation](https://dataframely.readthedocs.io/en/latest/).
+See more advanced usage examples in the [documentation](https://dataframely.readthedocs.io/stable/).
 

{dataframely-2.0.0 → dataframely-2.1.0}/README.md

@@ -78,4 +78,4 @@ df = pl.DataFrame({
 validated_df: dy.DataFrame[HouseSchema] = HouseSchema.validate(df, cast=True)
 ```
 
-See more advanced usage examples in the [documentation](https://dataframely.readthedocs.io/en/latest/).
+See more advanced usage examples in the [documentation](https://dataframely.readthedocs.io/stable/).

{dataframely-2.0.0 → dataframely-2.1.0}/dataframely/_base_schema.py

@@ -8,7 +8,7 @@ import textwrap
 from abc import ABCMeta
 from copy import copy
 from dataclasses import dataclass, field
-from typing import Any
+from typing import TYPE_CHECKING, Any
 
 import polars as pl
 
@@ -164,12 +164,16 @@ class SchemaMeta(ABCMeta):
 
         return cls
 
-    def __getattribute__(cls, name: str) -> Any:
-        val = super().__getattribute__(name)
-        # Dynamically set the name of the column if it is a `Column` instance.
-        if isinstance(val, Column):
-            val._name = val.alias or name
-        return val
+    if not TYPE_CHECKING:
+        # Only define __getattribute__ at runtime to allow type checkers to properly
+        # validate attribute access. When TYPE_CHECKING is True, type checkers will use
+        # the default metaclass behavior which correctly identifies non-existent attributes.
+        def __getattribute__(cls, name: str) -> Any:
+            val = super().__getattribute__(name)
+            # Dynamically set the name of the column if it is a `Column` instance.
+            if isinstance(val, Column):
+                val._name = val.alias or name
+            return val
 
     @staticmethod
     def _get_metadata_recursively(kls: type[object]) -> Metadata:
@@ -199,9 +203,9 @@ class SchemaMeta(ABCMeta):
     def __repr__(cls) -> str:
         parts = [f'[Schema "{cls.__name__}"]']
         parts.append(textwrap.indent("Columns:", prefix=" " * 2))
-        for name, col in cls.columns().items():
+        for name, col in cls.columns().items():  # type: ignore[attr-defined]
             parts.append(textwrap.indent(f'- "{name}": {col!r}', prefix=" " * 4))
-        if validation_rules := cls._schema_validation_rules():
+        if validation_rules := cls._schema_validation_rules():  # type: ignore[attr-defined]
             parts.append(textwrap.indent("Rules:", prefix=" " * 2))
             for name, rule in validation_rules.items():
                 parts.append(textwrap.indent(f'- "{name}": {rule!r}', prefix=" " * 4))

{dataframely-2.0.0 → dataframely-2.1.0}/dataframely/_pydantic.py

@@ -20,12 +20,12 @@ _S = TypeVar("_S", bound=BaseSchema)
 def _dict_to_df(schema_type: type[BaseSchema], data: dict) -> pl.DataFrame:
     return pl.from_dict(
         data,
-        schema=schema_type.to_polars_schema(),
+        schema=schema_type.to_polars_schema(),  # type: ignore[attr-defined]
     )
 
 
 def _validate_df_schema(schema_type: type[_S], df: pl.DataFrame) -> DataFrame[_S]:
-    if not schema_type.is_valid(df):
+    if not schema_type.is_valid(df):  # type: ignore[attr-defined]
         raise ValueError("DataFrame violates schema")
     return df  # type: ignore
 

{dataframely-2.0.0 → dataframely-2.1.0}/dataframely/collection/collection.py

@@ -531,6 +531,25 @@ class Collection(BaseCollection, ABC):
         Raises:
             ValueError: If an insufficient set of input data frames is provided, i.e. if
                 any required member of this collection is missing in the input.
+
+        Example:
+
+            .. code-block:: python
+
+                # Define collection
+                class HospitalInvoiceData(dy.Collection):
+                    invoice: dy.LazyFrame[InvoiceSchema]
+                    ...
+
+                # Filter the data and cast columns to expected types
+                good, failure = HospitalInvoiceData.filter(df, cast=True)
+
+                # Inspect the reasons for the failed rows for member `invoice`
+                print(failure.invoice.counts())
+
+                # Inspect the failed rows
+                failed_df = failure.invoice.invalid()
+                print(failed_df)
         """
         cls._validate_input_keys(data)
 

{dataframely-2.0.0 → dataframely-2.1.0}/dataframely/columns/array.py

@@ -15,7 +15,7 @@ from dataframely.random import Generator
 
 from ._base import Check, Column
 from ._registry import column_from_dict, register
-from .struct import Struct
+from .list import _list_primary_key_check
 
 if sys.version_info >= (3, 11):
     from typing import Self
@@ -42,7 +42,7 @@ class Array(Column):
     ):
         """
         Args:
-            inner: The inner column type. No validation rules on the inner type are supported yet.
+            inner: The inner column type.
             shape: The shape of the array.
             nullable: Whether this column may contain null values.
             primary_key: Whether this column is part of the primary key of the schema.
@@ -64,23 +64,6 @@ class Array(Column):
                 names, the specified alias is the only valid name.
             metadata: A dictionary of metadata to attach to the column.
         """
-        if inner.primary_key or (
-            isinstance(inner, Struct)
-            and any(col.primary_key for col in inner.inner.values())
-        ):
-            raise ValueError(
-                "`primary_key=True` is not yet supported for inner types of the Array type."
-            )
-
-        # We disallow validation rules on the inner type since Polars arrays currently don't support .eval(). Converting
-        # to a list and calling .list.eval() is possible, however, since the shape can have multiple axes, the recursive
-        # conversion could have significant performance impact. Hence, we simply disallow inner validation rules.
-        # Another option would be to allow validation rules only for sampling, but not enforce them.
-        if inner.validation_rules(pl.lit(None)):
-            raise ValueError(
-                "Validation rules on the inner type of Array are not yet supported."
-            )
-
         super().__init__(
             nullable=nullable,
             primary_key=False,
@@ -95,20 +78,39 @@ class Array(Column):
     def dtype(self) -> pl.DataType:
         return pl.Array(self.inner.dtype, self.shape)
 
+    def validation_rules(self, expr: pl.Expr) -> dict[str, pl.Expr]:
+        inner_rules = {
+            f"inner_{rule_name}": expr.arr.eval(inner_expr).arr.all()
+            for rule_name, inner_expr in self.inner.validation_rules(
+                pl.element()
+            ).items()
+        }
+
+        array_rules: dict[str, pl.Expr] = {}
+        if (rule := _list_primary_key_check(expr.arr, self.inner)) is not None:
+            array_rules["primary_key"] = rule
+
+        return {
+            **super().validation_rules(expr),
+            **array_rules,
+            **inner_rules,
+        }
+
     def sqlalchemy_dtype(self, dialect: sa.Dialect) -> sa_TypeEngine:
         # NOTE: We might want to add support for PostgreSQL's ARRAY type or use JSON in the future.
         raise NotImplementedError("SQL column cannot have 'Array' type.")
 
-    def _pyarrow_dtype_of_shape(self, shape: Sequence[int]) -> pa.DataType:
+    def _pyarrow_field_of_shape(self, shape: Sequence[int]) -> pa.Field:
         if shape:
             size, *rest = shape
-            return pa.list_(self._pyarrow_dtype_of_shape(rest), size)
+            inner_type = self._pyarrow_field_of_shape(rest)
+            return pa.field("item", pa.list_(inner_type, size), nullable=True)
         else:
-            return self.inner.pyarrow_dtype
+            return self.inner.pyarrow_field("item")
 
     @property
     def pyarrow_dtype(self) -> pa.DataType:
-        return self._pyarrow_dtype_of_shape(self.shape)
+        return self._pyarrow_field_of_shape(self.shape).type
 
     def _sample_unchecked(self, generator: Generator, n: int) -> pl.Series:
         # Sample the inner elements in a flat series

{dataframely-2.0.0 → dataframely-2.1.0}/dataframely/columns/list.py

@@ -8,6 +8,8 @@ from itertools import chain
 from typing import Any, cast
 
 import polars as pl
+from polars.expr.array import ExprArrayNameSpace
+from polars.expr.list import ExprListNameSpace
 
 from dataframely._compat import pa, sa, sa_TypeEngine
 from dataframely._polars import PolarsDataType
@@ -97,29 +99,8 @@ class List(Column):
         }
 
         list_rules: dict[str, pl.Expr] = {}
-        if self.inner.primary_key:
-            list_rules["primary_key"] = ~expr.list.eval(
-                pl.element().is_duplicated()
-            ).list.any()
-        elif isinstance(self.inner, Struct) and any(
-            col.primary_key for col in self.inner.inner.values()
-        ):
-            primary_key_columns = [
-                name for name, col in self.inner.inner.items() if col.primary_key
-            ]
-            # NOTE: We optimize for a single primary key column here as it is much
-            #  faster to run duplication checks for non-struct types in polars 1.22.
-            if len(primary_key_columns) == 1:
-                list_rules["primary_key"] = ~expr.list.eval(
-                    pl.element().struct.field(primary_key_columns[0]).is_duplicated()
-                ).list.any()
-            else:
-                list_rules["primary_key"] = ~expr.list.eval(
-                    pl.struct(
-                        pl.element().struct.field(primary_key_columns)
-                    ).is_duplicated()
-                ).list.any()
-
+        if (rule := _list_primary_key_check(expr.list, self.inner)) is not None:
+            list_rules["primary_key"] = rule
         if self.min_length is not None:
             list_rules["min_length"] = (
                 pl.when(expr.is_null())
@@ -145,7 +126,7 @@ class List(Column):
     @property
     def pyarrow_dtype(self) -> pa.DataType:
         # NOTE: Polars uses `large_list`s by default.
-        return pa.large_list(self.inner.pyarrow_dtype)
+        return pa.large_list(self.inner.pyarrow_field("item"))
 
     def _sample_unchecked(self, generator: Generator, n: int) -> pl.Series:
         # First, sample the number of items per list element
@@ -187,3 +168,35 @@ class List(Column):
     def from_dict(cls, data: dict[str, Any]) -> Self:
         data["inner"] = column_from_dict(data["inner"])
         return super().from_dict(data)
+
+
+def _list_primary_key_check(
+    list_expr: ExprListNameSpace | ExprArrayNameSpace, inner: Column
+) -> pl.Expr | None:
+    def list_any(expr: pl.Expr) -> pl.Expr:
+        if isinstance(list_expr, ExprListNameSpace):
+            return expr.list.any()
+        return expr.arr.any()
+
+    if inner.primary_key:
+        return ~list_expr.eval(pl.element().is_duplicated()).pipe(list_any)
+    elif isinstance(inner, Struct) and any(
+        col.primary_key for col in inner.inner.values()
+    ):
+        primary_key_columns = [
+            name for name, col in inner.inner.items() if col.primary_key
+        ]
+        # NOTE: We optimize for a single primary key column here as it is much
+        #  faster to run duplication checks for non-struct types in polars 1.22.
+        if len(primary_key_columns) == 1:
+            return ~list_expr.eval(
+                pl.element().struct.field(primary_key_columns[0]).is_duplicated()
+            ).pipe(list_any)
+        else:
+            return ~list_expr.eval(
+                pl.struct(
+                    pl.element().struct.field(primary_key_columns)
+                ).is_duplicated()
+            ).pipe(list_any)
+
+    return None

{dataframely-2.0.0 → dataframely-2.1.0}/dataframely/columns/struct.py

@@ -112,7 +112,7 @@ class Struct(Column):
 
     @property
     def pyarrow_dtype(self) -> pa.DataType:
-        return pa.struct({name: col.pyarrow_dtype for name, col in self.inner.items()})
+        return pa.struct([col.pyarrow_field(name) for name, col in self.inner.items()])
 
     def _sample_unchecked(self, generator: Generator, n: int) -> pl.Series:
         series = (