typedframes 0.2.1__tar.gz → 0.3.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- typedframes-0.2.1/README.md → typedframes-0.3.0/PKG-INFO +121 -43
- typedframes-0.2.1/PKG-INFO → typedframes-0.3.0/README.md +86 -67
- {typedframes-0.2.1 → typedframes-0.3.0}/pyproject.toml +36 -7
- {typedframes-0.2.1 → typedframes-0.3.0}/rust/Cargo.lock +17 -47
- {typedframes-0.2.1 → typedframes-0.3.0}/rust/Cargo.toml +2 -2
- {typedframes-0.2.1 → typedframes-0.3.0}/rust/src/lib.rs +1679 -166
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/__init__.py +1 -1
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/pandas.py +2 -2
- {typedframes-0.2.1 → typedframes-0.3.0}/rust/README.md +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/rust/benches/parser_bench.rs +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/rust/src/main.rs +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/rust/tests/integration_test.rs +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/base_schema.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/cli.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/column.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/column_group.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/column_group_error.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/column_set.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/missing_dependency_error.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/mypy.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/pandera.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/polars.py +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/py.typed +0 -0
- {typedframes-0.2.1 → typedframes-0.3.0}/src/typedframes/schema_algebra.py +0 -0
|
@@ -1,14 +1,48 @@
|
|
|
1
|
+
Metadata-Version: 2.4
|
|
2
|
+
Name: typedframes
|
|
3
|
+
Version: 0.3.0
|
|
4
|
+
Classifier: Development Status :: 3 - Alpha
|
|
5
|
+
Classifier: Intended Audience :: Developers
|
|
6
|
+
Classifier: License :: OSI Approved :: MIT License
|
|
7
|
+
Classifier: Operating System :: OS Independent
|
|
8
|
+
Classifier: Programming Language :: Python :: 3
|
|
9
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
10
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
11
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
12
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
13
|
+
Classifier: Topic :: Software Development :: Quality Assurance
|
|
14
|
+
Classifier: Topic :: Software Development :: Libraries :: Python Modules
|
|
15
|
+
Requires-Dist: mypy ; extra == 'mypy'
|
|
16
|
+
Requires-Dist: pandas>=2.0 ; extra == 'pandas'
|
|
17
|
+
Requires-Dist: pandera>=0.20.0 ; extra == 'pandera'
|
|
18
|
+
Requires-Dist: polars>=1.0 ; extra == 'polars'
|
|
19
|
+
Provides-Extra: mypy
|
|
20
|
+
Provides-Extra: pandas
|
|
21
|
+
Provides-Extra: pandera
|
|
22
|
+
Provides-Extra: polars
|
|
23
|
+
Summary: Static analysis for pandas and polars DataFrames. Catch column errors at lint-time, not runtime.
|
|
24
|
+
Keywords: pandas,polars,type-checking,static-analysis,dataframe,linter,mypy-plugin
|
|
25
|
+
Home-Page: https://github.com/w-martin/typedframes
|
|
26
|
+
Author: William Martin
|
|
27
|
+
License-Expression: MIT
|
|
28
|
+
Requires-Python: >=3.11
|
|
29
|
+
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
|
|
30
|
+
Project-URL: Documentation, https://typedframes.readthedocs.io/en/latest/
|
|
31
|
+
Project-URL: Homepage, https://github.com/w-martin/typedframes
|
|
32
|
+
Project-URL: Issue Tracker, https://github.com/w-martin/typedframes/issues
|
|
33
|
+
Project-URL: Repository, https://github.com/w-martin/typedframes
|
|
34
|
+
|
|
1
35
|
# typedframes
|
|
2
36
|
|
|
3
37
|
[](https://github.com/w-martin/typedframes/actions/workflows/publish.yml)
|
|
4
38
|
[](https://pypi.org/project/typedframes/)
|
|
5
39
|
[](https://pypi.org/project/typedframes/)
|
|
6
|
-
[](https://coveralls.io/github/w-martin/typedframes?branch=main)
|
|
7
41
|
[](https://opensource.org/licenses/MIT)
|
|
8
42
|
|
|
9
43
|
> ⚠️ **Project Status: Proof of Concept**
|
|
10
44
|
>
|
|
11
|
-
> `typedframes` (v0.
|
|
45
|
+
> `typedframes` (v0.3.0) is currently an experimental proof-of-concept. The core static analysis and mypy/Rust
|
|
12
46
|
> integrations work, but expect rough edges. The codebase prioritizes demonstrating the viability of static DataFrame
|
|
13
47
|
> schema validation over production-grade stability.
|
|
14
48
|
>
|
|
@@ -106,7 +140,7 @@ print(orders["revenue"]) # ✗ unknown-column — 'revenue' not in inferred set
|
|
|
106
140
|
|
|
107
141
|
```shell
|
|
108
142
|
typedframes check src/
|
|
109
|
-
# src/pipeline.py:7:8: error[unknown-column] Column 'revenue' not in inferred set
|
|
143
|
+
# src/pipeline.py:7:8: error[unknown-column] Column 'revenue' does not exist in inferred column set (defined at line 6)
|
|
110
144
|
# ✗ Found 1 error in 12 files (0.0s)
|
|
111
145
|
```
|
|
112
146
|
|
|
@@ -263,12 +297,39 @@ trimmed = df.drop(columns=["nonexistent"])
|
|
|
263
297
|
# ⚠ dropped-unknown-column: Dropped column 'nonexistent' does not exist in UserData
|
|
264
298
|
```
|
|
265
299
|
|
|
300
|
+
### Function Parameter Contracts
|
|
301
|
+
|
|
302
|
+
Beyond validating access at the point it happens, the checker infers a *contract* for any function's
|
|
303
|
+
first parameter: every column the function needs, drawn from what its body accesses or — taking priority —
|
|
304
|
+
from a schema annotation on the parameter itself. Calling that function with a DataFrame that doesn't
|
|
305
|
+
satisfy the contract is caught at the **call site**, across files:
|
|
306
|
+
|
|
307
|
+
```python
|
|
308
|
+
# transforms.py
|
|
309
|
+
def contact_label(customers):
|
|
310
|
+
return customers["name"] + customers["email"]
|
|
311
|
+
```
|
|
312
|
+
|
|
313
|
+
```python
|
|
314
|
+
# pipeline.py
|
|
315
|
+
customers = load_customers(path) # inferred columns: {customer_id, name, region}
|
|
316
|
+
contact_label(customers)
|
|
317
|
+
# ✗ missing-column: 'customers' passed to contact_label (transforms.py:2) is missing
|
|
318
|
+
# column(s) {email} — available: {customer_id, name, region}, required: {email, name}
|
|
319
|
+
```
|
|
320
|
+
|
|
321
|
+
The contract is resolved *transitively*: if a function only forwards its parameter to other functions
|
|
322
|
+
(`step1 = preprocess(df); step2 = enrich(step1)`), the checker follows the chain and unions their
|
|
323
|
+
requirements, catching a missing column even when no single function in the chain touches it directly.
|
|
324
|
+
Column-list slices (`df[["a", "b"]]`) contribute to the contract too.
|
|
325
|
+
|
|
266
326
|
### See Also
|
|
267
327
|
|
|
268
328
|
- [`examples/inference_example.py`](examples/inference_example.py) — single-file walkthrough of all four inference
|
|
269
329
|
scenarios with annotated ✓/✗ comments.
|
|
270
330
|
- [`examples/multi_file_inference/`](examples/multi_file_inference/) — multi-file project checked with
|
|
271
|
-
`typedframes check examples/multi_file_inference/`; no `BaseSchema` anywhere.
|
|
331
|
+
`typedframes check examples/multi_file_inference/`; no `BaseSchema` anywhere. Includes a function
|
|
332
|
+
parameter contract violation caught at the call site (`missing-column`).
|
|
272
333
|
- [`examples/multi_file_with_schema/`](examples/multi_file_with_schema/) — same scenario with `BaseSchema`
|
|
273
334
|
classes; the checker follows schemas across module boundaries via the project index.
|
|
274
335
|
|
|
@@ -293,8 +354,10 @@ typedframes check src/
|
|
|
293
354
|
**Features:**
|
|
294
355
|
- Catches column name errors
|
|
295
356
|
- Validates schema mismatches between functions
|
|
357
|
+
- Validates function parameter contracts across files, including transitively through chains of
|
|
358
|
+
helper functions (`missing-column`)
|
|
296
359
|
- Checks both pandas and polars code
|
|
297
|
-
-
|
|
360
|
+
- Significantly faster than mypy (see benchmarks below)
|
|
298
361
|
|
|
299
362
|
**Use this for:**
|
|
300
363
|
- Fast feedback during development
|
|
@@ -375,17 +438,15 @@ Fast feedback reduces development time. The typedframes Rust binary provides nea
|
|
|
375
438
|
|
|
376
439
|
**Benchmark results** (10 runs, 3 warmup, caches cleared between runs):
|
|
377
440
|
|
|
378
|
-
| Tool
|
|
379
|
-
|
|
380
|
-
| typedframes
|
|
381
|
-
| ruff
|
|
382
|
-
| ty
|
|
383
|
-
| pyrefly
|
|
384
|
-
| mypy
|
|
385
|
-
| mypy + typedframes |
|
|
386
|
-
| pyright
|
|
387
|
-
|
|
388
|
-
*† great_expectations column from previous benchmark run.*
|
|
441
|
+
| Tool | Version | What it does | typedframes (13 files) | great_expectations (485 files) |
|
|
442
|
+
|------|---------|--------------|------------------------|--------------------------------|
|
|
443
|
+
| typedframes | 0.3.0 | DataFrame column checker | 81ms ±6ms | 3.64s ±9ms |
|
|
444
|
+
| ruff | 0.15.21 | Linter (no type checking) | 48ms ±13ms | 237ms ±31ms |
|
|
445
|
+
| ty | 0.0.58 | Type checker | 119ms ±14ms | 287ms ±33ms |
|
|
446
|
+
| pyrefly | 1.1.1 | Type checker | 162ms ±18ms | 390ms ±21ms |
|
|
447
|
+
| mypy | 2.2.0 | Type checker (no plugin) | 4.56s ±22ms | 7.31s ±35ms |
|
|
448
|
+
| mypy + typedframes | 2.2.0 | Type checker + column checker | 4.69s ±61ms | 10.89s ±356ms |
|
|
449
|
+
| pyright | 1.1.411 | Type checker | 1.39s ±17ms | 8.06s ±193ms |
|
|
389
450
|
|
|
390
451
|
*Run `uv run python benchmarks/benchmark_checkers.py` to reproduce.*
|
|
391
452
|
|
|
@@ -596,63 +657,77 @@ See [`examples/schema_algebra_example.py`](examples/schema_algebra_example.py) f
|
|
|
596
657
|
Comprehensive comparison of pandas/DataFrame typing and validation tools. **typedframes focuses on static analysis**
|
|
597
658
|
—catching errors at lint-time before your code runs.
|
|
598
659
|
|
|
599
|
-
| Feature | typedframes | Pandera | Great Expectations | strictly_typed_pandas | pandas-stubs | dataenforce | pandas-type-checks | StaticFrame | narwhals |
|
|
600
|
-
|
|
601
|
-
| **Version tested** | 0.
|
|
660
|
+
| Feature | typedframes | Pandera | Great Expectations | strictly_typed_pandas | pandas-stubs | dataenforce | pandas-type-checks | StaticFrame | narwhals | dataframely | patito |
|
|
661
|
+
|---------------------------------|------------------------|-------------|--------------------|-----------------------|--------------|-------------|--------------------|------------------|----------|------------------|------------------|
|
|
662
|
+
| **Version tested** | 0.3.0 | 0.32.1 | 1.18.2 | 0.3.7 | 3.0.3 | 0.1.2 | 1.1.3 | 5.0.0 | 2.23.0 | 2.13.0 | 0.8.6 |
|
|
602
663
|
| **Analysis Type** |
|
|
603
|
-
| When errors are caught | **Static (lint-time)** | Runtime | Runtime |
|
|
664
|
+
| When errors are caught | **Static (lint-time)** | Runtime | Runtime | Runtime | Static | Runtime | Runtime | Runtime | Runtime | Runtime | Runtime |
|
|
604
665
|
| **Static Analysis (our focus)** |
|
|
605
|
-
| Mypy plugin | ✅ Yes | ⚠️ Limited | ❌ No |
|
|
606
|
-
| Standalone checker | ✅ Rust (~1ms) | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
607
|
-
| Column name checking | ✅ Yes | ⚠️ Limited | ❌ No |
|
|
608
|
-
| Column type checking | ✅ Yes | ⚠️ Limited | ❌ No |
|
|
609
|
-
| Typo suggestions | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
666
|
+
| Mypy plugin | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No | ⚠️ Basic | ❌ No | ❌ No | ❌ No |
|
|
667
|
+
| Standalone checker | ✅ Rust (~1ms) | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
668
|
+
| Column name checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
669
|
+
| Column type checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
670
|
+
| Typo suggestions | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
610
671
|
| **Runtime Validation** |
|
|
611
|
-
| Data validation | ❌ No | ✅ Excellent | ✅ Excellent | ✅ typeguard | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No |
|
|
612
|
-
| Value constraints | ❌ No | ✅ Yes | ✅ Excellent | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No |
|
|
672
|
+
| Data validation | ❌ No | ✅ Excellent | ✅ Excellent | ✅ typeguard | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
|
|
673
|
+
| Value constraints | ❌ No | ✅ Yes | ✅ Excellent | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
|
|
613
674
|
| **Schema Features** |
|
|
614
|
-
| Column grouping | ✅ ColumnGroup | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
615
|
-
| Regex column matching | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
675
|
+
| Column grouping | ✅ ColumnGroup | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
676
|
+
| Regex column matching | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
616
677
|
| **Backend Support** |
|
|
617
|
-
| Pandas | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Own | ✅ Yes |
|
|
618
|
-
| Polars | ✅ Yes | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ Own | ✅ Yes |
|
|
619
|
-
| DuckDB, cuDF, etc. | ❌ No | ❌ No | ✅ Spark, SQL | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes |
|
|
678
|
+
| Pandas | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Own | ✅ Yes | ❌ No | ⚠️ Limited |
|
|
679
|
+
| Polars | ✅ Yes | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ Own | ✅ Yes | ✅ Yes (only) | ✅ Yes |
|
|
680
|
+
| DuckDB, cuDF, etc. | ❌ No | ❌ No | ✅ Spark, SQL | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No |
|
|
620
681
|
| **Project Status (Feb 2026)** |
|
|
621
|
-
| Active development | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Low | ✅ Yes | ❌ Inactive | ⚠️ Low | ✅ Yes | ✅ Yes |
|
|
682
|
+
| Active development | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Low | ✅ Yes | ❌ Inactive | ⚠️ Low | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
|
|
622
683
|
|
|
623
684
|
**Legend:** ✅ Full support | ⚠️ Limited/Partial | ❌ Not supported
|
|
624
685
|
|
|
625
686
|
### Tool Descriptions
|
|
626
687
|
|
|
627
|
-
- **[Pandera](https://pandera.readthedocs.io/)** (v0.
|
|
688
|
+
- **[Pandera](https://pandera.readthedocs.io/)** (v0.32.1): Excellent runtime validation. Static analysis support exists
|
|
628
689
|
but has limitations—column access via `df["column"]` is not validated, and schema mismatches between functions may not
|
|
629
690
|
be caught.
|
|
630
691
|
|
|
631
|
-
- **[strictly_typed_pandas](https://strictly-typed-pandas.readthedocs.io/)** (v0.3.
|
|
632
|
-
hints
|
|
692
|
+
- **[strictly_typed_pandas](https://strictly-typed-pandas.readthedocs.io/)** (v0.3.7): Provides `DataSet[Schema]` type
|
|
693
|
+
hints for runtime validation via typeguard. Despite documentation implying mypy support, there is no mypy plugin —
|
|
694
|
+
column access errors are not caught statically. No standalone checker. No polars support.
|
|
633
695
|
|
|
634
|
-
- **[pandas-stubs](https://github.com/pandas-dev/pandas-stubs)** (v3.0.
|
|
696
|
+
- **[pandas-stubs](https://github.com/pandas-dev/pandas-stubs)** (v3.0.3): Official pandas type stubs. Provides
|
|
635
697
|
API-level types but no column-level checking.
|
|
636
698
|
|
|
637
699
|
- **[dataenforce](https://github.com/CedricFR/dataenforce)** (v0.1.2): Runtime validation via decorator. Marked as
|
|
638
|
-
experimental/not production-ready. Appears inactive.
|
|
700
|
+
experimental/not production-ready. Appears inactive. Broken on Python 3.13+ due to removal of internal typing APIs.
|
|
639
701
|
|
|
640
702
|
- **[pandas-type-checks](https://pypi.org/project/pandas-type-checks/)** (v1.1.3): Runtime validation decorator. No
|
|
641
703
|
static analysis.
|
|
642
704
|
|
|
643
|
-
- **[StaticFrame](https://github.com/static-frame/static-frame)** (
|
|
644
|
-
|
|
705
|
+
- **[StaticFrame](https://github.com/static-frame/static-frame)** (v5.0.0): Alternative immutable DataFrame library.
|
|
706
|
+
Not compatible with pandas/polars — requires a full rewrite to StaticFrame's own API. Column access is still
|
|
707
|
+
string-based; mypy does not catch column name typos. Type safety comes from immutability guarantees, not schema checking.
|
|
645
708
|
|
|
646
|
-
- **[narwhals](https://narwhals-dev.github.io/narwhals/)** (v2.
|
|
709
|
+
- **[narwhals](https://narwhals-dev.github.io/narwhals/)** (v2.23.0): Compatibility layer that provides a unified API
|
|
647
710
|
across pandas, polars, DuckDB, cuDF, and more. Solves a different problem—write-once-run-anywhere portability, not
|
|
648
711
|
type safety. See [Why Abstraction Layers Don't Solve Type Safety](#why-abstraction-layers-dont-solve-type-safety)
|
|
649
712
|
below.
|
|
650
713
|
|
|
651
|
-
- **[Great Expectations](https://greatexpectations.io/)** (v1.
|
|
714
|
+
- **[Great Expectations](https://greatexpectations.io/)** (v1.18.2): Comprehensive data quality framework. Defines
|
|
652
715
|
"expectations" (assertions) about data values, distributions, and schema properties. Excellent for runtime
|
|
653
716
|
validation, data documentation, and data quality monitoring. No static analysis or column-level type checking in
|
|
654
717
|
code. Supports pandas, Spark, and SQL backends.
|
|
655
718
|
|
|
719
|
+
- **[dataframely](https://github.com/Quantco/dataframely)** (v2.13.0): Polars-only runtime validation library from Quantco.
|
|
720
|
+
Schemas are defined as classes inheriting `dy.Schema` with typed descriptor fields (`dy.String()`, `dy.Float64()`)
|
|
721
|
+
and `@dy.rule()` decorators for cross-column and group-level constraints. Returns `dy.DataFrame[Schema]` generic
|
|
722
|
+
types that give call-site narrowing to type checkers, but does not validate column subscript access inside function
|
|
723
|
+
bodies. No lint-time or static analysis capability. Supports nullability, string constraints, numeric bounds,
|
|
724
|
+
cross-column rules, soft validation, test data generation, and SQLAlchemy/PyArrow export.
|
|
725
|
+
|
|
726
|
+
- **[patito](https://github.com/JakobGM/patito)** (v0.8.6): Runtime validation library using a Pydantic-style `patito.Model`
|
|
727
|
+
class. Polars is the primary backend; pandas is supported but works by converting to Polars via PyArrow (an
|
|
728
|
+
undeclared dependency). DuckDB is not supported despite appearing in some documentation—validation crashes immediately
|
|
729
|
+
on DuckDB relations. No static analysis or standalone checker.
|
|
730
|
+
|
|
656
731
|
### Type Checkers (Not DataFrame-Specific)
|
|
657
732
|
|
|
658
733
|
These are general Python type checkers. They don't validate DataFrame column names, but they can be used alongside
|
|
@@ -946,6 +1021,8 @@ MIT License - see [LICENSE](LICENSE)
|
|
|
946
1021
|
- [x] Pandera integration (`to_pandera_schema()`)
|
|
947
1022
|
- [x] Cross-file schema inference (project-level index, `--no-index` flag)
|
|
948
1023
|
- [x] Aggressive column inference (untracked-dataframe/dropped-unknown-column warnings, method chain propagation)
|
|
1024
|
+
- [x] Function parameter contracts (`missing-column`), resolved transitively across chains of helper
|
|
1025
|
+
functions and cross-file calls; schema-annotated parameters take priority over body-scanning
|
|
949
1026
|
|
|
950
1027
|
**Planned:**
|
|
951
1028
|
|
|
@@ -1002,6 +1079,7 @@ Inspired by the needs of ML/data science teams working with complex data pipelin
|
|
|
1002
1079
|
|
|
1003
1080
|
---
|
|
1004
1081
|
|
|
1005
|
-
**Questions? Issues? Ideas?** [Open an issue](https://github.com/
|
|
1082
|
+
**Questions? Issues? Ideas?** [Open an issue](https://github.com/w-martin/typedframes/issues)
|
|
1006
1083
|
|
|
1007
1084
|
**Ready to catch DataFrame bugs before runtime?** `pip install typedframes`
|
|
1085
|
+
|
|
@@ -1,37 +1,14 @@
|
|
|
1
|
-
Metadata-Version: 2.4
|
|
2
|
-
Name: typedframes
|
|
3
|
-
Version: 0.2.1
|
|
4
|
-
Classifier: Programming Language :: Python :: 3
|
|
5
|
-
Classifier: Programming Language :: Python :: 3.11
|
|
6
|
-
Classifier: Programming Language :: Python :: 3.12
|
|
7
|
-
Classifier: Programming Language :: Python :: 3.13
|
|
8
|
-
Classifier: Programming Language :: Python :: 3.14
|
|
9
|
-
Requires-Dist: mypy ; extra == 'mypy'
|
|
10
|
-
Requires-Dist: pandas>=2.3.3 ; extra == 'pandas'
|
|
11
|
-
Requires-Dist: pandera>=0.29.0 ; extra == 'pandera'
|
|
12
|
-
Requires-Dist: polars>=1.38.1 ; extra == 'polars'
|
|
13
|
-
Provides-Extra: mypy
|
|
14
|
-
Provides-Extra: pandas
|
|
15
|
-
Provides-Extra: pandera
|
|
16
|
-
Provides-Extra: polars
|
|
17
|
-
Summary: Static analysis for pandas and polars DataFrames. Catch column errors at lint-time, not runtime.
|
|
18
|
-
Home-Page: https://github.com/w-martin/typedframes
|
|
19
|
-
Author: William Martin
|
|
20
|
-
License: MIT
|
|
21
|
-
Requires-Python: >=3.11
|
|
22
|
-
Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
|
|
23
|
-
|
|
24
1
|
# typedframes
|
|
25
2
|
|
|
26
3
|
[](https://github.com/w-martin/typedframes/actions/workflows/publish.yml)
|
|
27
4
|
[](https://pypi.org/project/typedframes/)
|
|
28
5
|
[](https://pypi.org/project/typedframes/)
|
|
29
|
-
[](https://coveralls.io/github/w-martin/typedframes?branch=main)
|
|
30
7
|
[](https://opensource.org/licenses/MIT)
|
|
31
8
|
|
|
32
9
|
> ⚠️ **Project Status: Proof of Concept**
|
|
33
10
|
>
|
|
34
|
-
> `typedframes` (v0.
|
|
11
|
+
> `typedframes` (v0.3.0) is currently an experimental proof-of-concept. The core static analysis and mypy/Rust
|
|
35
12
|
> integrations work, but expect rough edges. The codebase prioritizes demonstrating the viability of static DataFrame
|
|
36
13
|
> schema validation over production-grade stability.
|
|
37
14
|
>
|
|
@@ -129,7 +106,7 @@ print(orders["revenue"]) # ✗ unknown-column — 'revenue' not in inferred set
|
|
|
129
106
|
|
|
130
107
|
```shell
|
|
131
108
|
typedframes check src/
|
|
132
|
-
# src/pipeline.py:7:8: error[unknown-column] Column 'revenue' not in inferred set
|
|
109
|
+
# src/pipeline.py:7:8: error[unknown-column] Column 'revenue' does not exist in inferred column set (defined at line 6)
|
|
133
110
|
# ✗ Found 1 error in 12 files (0.0s)
|
|
134
111
|
```
|
|
135
112
|
|
|
@@ -286,12 +263,39 @@ trimmed = df.drop(columns=["nonexistent"])
|
|
|
286
263
|
# ⚠ dropped-unknown-column: Dropped column 'nonexistent' does not exist in UserData
|
|
287
264
|
```
|
|
288
265
|
|
|
266
|
+
### Function Parameter Contracts
|
|
267
|
+
|
|
268
|
+
Beyond validating access at the point it happens, the checker infers a *contract* for any function's
|
|
269
|
+
first parameter: every column the function needs, drawn from what its body accesses or — taking priority —
|
|
270
|
+
from a schema annotation on the parameter itself. Calling that function with a DataFrame that doesn't
|
|
271
|
+
satisfy the contract is caught at the **call site**, across files:
|
|
272
|
+
|
|
273
|
+
```python
|
|
274
|
+
# transforms.py
|
|
275
|
+
def contact_label(customers):
|
|
276
|
+
return customers["name"] + customers["email"]
|
|
277
|
+
```
|
|
278
|
+
|
|
279
|
+
```python
|
|
280
|
+
# pipeline.py
|
|
281
|
+
customers = load_customers(path) # inferred columns: {customer_id, name, region}
|
|
282
|
+
contact_label(customers)
|
|
283
|
+
# ✗ missing-column: 'customers' passed to contact_label (transforms.py:2) is missing
|
|
284
|
+
# column(s) {email} — available: {customer_id, name, region}, required: {email, name}
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
The contract is resolved *transitively*: if a function only forwards its parameter to other functions
|
|
288
|
+
(`step1 = preprocess(df); step2 = enrich(step1)`), the checker follows the chain and unions their
|
|
289
|
+
requirements, catching a missing column even when no single function in the chain touches it directly.
|
|
290
|
+
Column-list slices (`df[["a", "b"]]`) contribute to the contract too.
|
|
291
|
+
|
|
289
292
|
### See Also
|
|
290
293
|
|
|
291
294
|
- [`examples/inference_example.py`](examples/inference_example.py) — single-file walkthrough of all four inference
|
|
292
295
|
scenarios with annotated ✓/✗ comments.
|
|
293
296
|
- [`examples/multi_file_inference/`](examples/multi_file_inference/) — multi-file project checked with
|
|
294
|
-
`typedframes check examples/multi_file_inference/`; no `BaseSchema` anywhere.
|
|
297
|
+
`typedframes check examples/multi_file_inference/`; no `BaseSchema` anywhere. Includes a function
|
|
298
|
+
parameter contract violation caught at the call site (`missing-column`).
|
|
295
299
|
- [`examples/multi_file_with_schema/`](examples/multi_file_with_schema/) — same scenario with `BaseSchema`
|
|
296
300
|
classes; the checker follows schemas across module boundaries via the project index.
|
|
297
301
|
|
|
@@ -316,8 +320,10 @@ typedframes check src/
|
|
|
316
320
|
**Features:**
|
|
317
321
|
- Catches column name errors
|
|
318
322
|
- Validates schema mismatches between functions
|
|
323
|
+
- Validates function parameter contracts across files, including transitively through chains of
|
|
324
|
+
helper functions (`missing-column`)
|
|
319
325
|
- Checks both pandas and polars code
|
|
320
|
-
-
|
|
326
|
+
- Significantly faster than mypy (see benchmarks below)
|
|
321
327
|
|
|
322
328
|
**Use this for:**
|
|
323
329
|
- Fast feedback during development
|
|
@@ -398,17 +404,15 @@ Fast feedback reduces development time. The typedframes Rust binary provides nea
|
|
|
398
404
|
|
|
399
405
|
**Benchmark results** (10 runs, 3 warmup, caches cleared between runs):
|
|
400
406
|
|
|
401
|
-
| Tool
|
|
402
|
-
|
|
403
|
-
| typedframes
|
|
404
|
-
| ruff
|
|
405
|
-
| ty
|
|
406
|
-
| pyrefly
|
|
407
|
-
| mypy
|
|
408
|
-
| mypy + typedframes |
|
|
409
|
-
| pyright
|
|
410
|
-
|
|
411
|
-
*† great_expectations column from previous benchmark run.*
|
|
407
|
+
| Tool | Version | What it does | typedframes (13 files) | great_expectations (485 files) |
|
|
408
|
+
|------|---------|--------------|------------------------|--------------------------------|
|
|
409
|
+
| typedframes | 0.3.0 | DataFrame column checker | 81ms ±6ms | 3.64s ±9ms |
|
|
410
|
+
| ruff | 0.15.21 | Linter (no type checking) | 48ms ±13ms | 237ms ±31ms |
|
|
411
|
+
| ty | 0.0.58 | Type checker | 119ms ±14ms | 287ms ±33ms |
|
|
412
|
+
| pyrefly | 1.1.1 | Type checker | 162ms ±18ms | 390ms ±21ms |
|
|
413
|
+
| mypy | 2.2.0 | Type checker (no plugin) | 4.56s ±22ms | 7.31s ±35ms |
|
|
414
|
+
| mypy + typedframes | 2.2.0 | Type checker + column checker | 4.69s ±61ms | 10.89s ±356ms |
|
|
415
|
+
| pyright | 1.1.411 | Type checker | 1.39s ±17ms | 8.06s ±193ms |
|
|
412
416
|
|
|
413
417
|
*Run `uv run python benchmarks/benchmark_checkers.py` to reproduce.*
|
|
414
418
|
|
|
@@ -619,63 +623,77 @@ See [`examples/schema_algebra_example.py`](examples/schema_algebra_example.py) f
|
|
|
619
623
|
Comprehensive comparison of pandas/DataFrame typing and validation tools. **typedframes focuses on static analysis**
|
|
620
624
|
—catching errors at lint-time before your code runs.
|
|
621
625
|
|
|
622
|
-
| Feature | typedframes | Pandera | Great Expectations | strictly_typed_pandas | pandas-stubs | dataenforce | pandas-type-checks | StaticFrame | narwhals |
|
|
623
|
-
|
|
624
|
-
| **Version tested** | 0.
|
|
626
|
+
| Feature | typedframes | Pandera | Great Expectations | strictly_typed_pandas | pandas-stubs | dataenforce | pandas-type-checks | StaticFrame | narwhals | dataframely | patito |
|
|
627
|
+
|---------------------------------|------------------------|-------------|--------------------|-----------------------|--------------|-------------|--------------------|------------------|----------|------------------|------------------|
|
|
628
|
+
| **Version tested** | 0.3.0 | 0.32.1 | 1.18.2 | 0.3.7 | 3.0.3 | 0.1.2 | 1.1.3 | 5.0.0 | 2.23.0 | 2.13.0 | 0.8.6 |
|
|
625
629
|
| **Analysis Type** |
|
|
626
|
-
| When errors are caught | **Static (lint-time)** | Runtime | Runtime |
|
|
630
|
+
| When errors are caught | **Static (lint-time)** | Runtime | Runtime | Runtime | Static | Runtime | Runtime | Runtime | Runtime | Runtime | Runtime |
|
|
627
631
|
| **Static Analysis (our focus)** |
|
|
628
|
-
| Mypy plugin | ✅ Yes | ⚠️ Limited | ❌ No |
|
|
629
|
-
| Standalone checker | ✅ Rust (~1ms) | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
630
|
-
| Column name checking | ✅ Yes | ⚠️ Limited | ❌ No |
|
|
631
|
-
| Column type checking | ✅ Yes | ⚠️ Limited | ❌ No |
|
|
632
|
-
| Typo suggestions | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
632
|
+
| Mypy plugin | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No | ⚠️ Basic | ❌ No | ❌ No | ❌ No |
|
|
633
|
+
| Standalone checker | ✅ Rust (~1ms) | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
634
|
+
| Column name checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
635
|
+
| Column type checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
636
|
+
| Typo suggestions | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
633
637
|
| **Runtime Validation** |
|
|
634
|
-
| Data validation | ❌ No | ✅ Excellent | ✅ Excellent | ✅ typeguard | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No |
|
|
635
|
-
| Value constraints | ❌ No | ✅ Yes | ✅ Excellent | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No |
|
|
638
|
+
| Data validation | ❌ No | ✅ Excellent | ✅ Excellent | ✅ typeguard | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
|
|
639
|
+
| Value constraints | ❌ No | ✅ Yes | ✅ Excellent | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
|
|
636
640
|
| **Schema Features** |
|
|
637
|
-
| Column grouping | ✅ ColumnGroup | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
638
|
-
| Regex column matching | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
641
|
+
| Column grouping | ✅ ColumnGroup | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
642
|
+
| Regex column matching | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
639
643
|
| **Backend Support** |
|
|
640
|
-
| Pandas | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Own | ✅ Yes |
|
|
641
|
-
| Polars | ✅ Yes | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ Own | ✅ Yes |
|
|
642
|
-
| DuckDB, cuDF, etc. | ❌ No | ❌ No | ✅ Spark, SQL | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes |
|
|
644
|
+
| Pandas | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Own | ✅ Yes | ❌ No | ⚠️ Limited |
|
|
645
|
+
| Polars | ✅ Yes | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ Own | ✅ Yes | ✅ Yes (only) | ✅ Yes |
|
|
646
|
+
| DuckDB, cuDF, etc. | ❌ No | ❌ No | ✅ Spark, SQL | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No |
|
|
643
647
|
| **Project Status (Feb 2026)** |
|
|
644
|
-
| Active development | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Low | ✅ Yes | ❌ Inactive | ⚠️ Low | ✅ Yes | ✅ Yes |
|
|
648
|
+
| Active development | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Low | ✅ Yes | ❌ Inactive | ⚠️ Low | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
|
|
645
649
|
|
|
646
650
|
**Legend:** ✅ Full support | ⚠️ Limited/Partial | ❌ Not supported
|
|
647
651
|
|
|
648
652
|
### Tool Descriptions
|
|
649
653
|
|
|
650
|
-
- **[Pandera](https://pandera.readthedocs.io/)** (v0.
|
|
654
|
+
- **[Pandera](https://pandera.readthedocs.io/)** (v0.32.1): Excellent runtime validation. Static analysis support exists
|
|
651
655
|
but has limitations—column access via `df["column"]` is not validated, and schema mismatches between functions may not
|
|
652
656
|
be caught.
|
|
653
657
|
|
|
654
|
-
- **[strictly_typed_pandas](https://strictly-typed-pandas.readthedocs.io/)** (v0.3.
|
|
655
|
-
hints
|
|
658
|
+
- **[strictly_typed_pandas](https://strictly-typed-pandas.readthedocs.io/)** (v0.3.7): Provides `DataSet[Schema]` type
|
|
659
|
+
hints for runtime validation via typeguard. Despite documentation implying mypy support, there is no mypy plugin —
|
|
660
|
+
column access errors are not caught statically. No standalone checker. No polars support.
|
|
656
661
|
|
|
657
|
-
- **[pandas-stubs](https://github.com/pandas-dev/pandas-stubs)** (v3.0.
|
|
662
|
+
- **[pandas-stubs](https://github.com/pandas-dev/pandas-stubs)** (v3.0.3): Official pandas type stubs. Provides
|
|
658
663
|
API-level types but no column-level checking.
|
|
659
664
|
|
|
660
665
|
- **[dataenforce](https://github.com/CedricFR/dataenforce)** (v0.1.2): Runtime validation via decorator. Marked as
|
|
661
|
-
experimental/not production-ready. Appears inactive.
|
|
666
|
+
experimental/not production-ready. Appears inactive. Broken on Python 3.13+ due to removal of internal typing APIs.
|
|
662
667
|
|
|
663
668
|
- **[pandas-type-checks](https://pypi.org/project/pandas-type-checks/)** (v1.1.3): Runtime validation decorator. No
|
|
664
669
|
static analysis.
|
|
665
670
|
|
|
666
|
-
- **[StaticFrame](https://github.com/static-frame/static-frame)** (
|
|
667
|
-
|
|
671
|
+
- **[StaticFrame](https://github.com/static-frame/static-frame)** (v5.0.0): Alternative immutable DataFrame library.
|
|
672
|
+
Not compatible with pandas/polars — requires a full rewrite to StaticFrame's own API. Column access is still
|
|
673
|
+
string-based; mypy does not catch column name typos. Type safety comes from immutability guarantees, not schema checking.
|
|
668
674
|
|
|
669
|
-
- **[narwhals](https://narwhals-dev.github.io/narwhals/)** (v2.
|
|
675
|
+
- **[narwhals](https://narwhals-dev.github.io/narwhals/)** (v2.23.0): Compatibility layer that provides a unified API
|
|
670
676
|
across pandas, polars, DuckDB, cuDF, and more. Solves a different problem—write-once-run-anywhere portability, not
|
|
671
677
|
type safety. See [Why Abstraction Layers Don't Solve Type Safety](#why-abstraction-layers-dont-solve-type-safety)
|
|
672
678
|
below.
|
|
673
679
|
|
|
674
|
-
- **[Great Expectations](https://greatexpectations.io/)** (v1.
|
|
680
|
+
- **[Great Expectations](https://greatexpectations.io/)** (v1.18.2): Comprehensive data quality framework. Defines
|
|
675
681
|
"expectations" (assertions) about data values, distributions, and schema properties. Excellent for runtime
|
|
676
682
|
validation, data documentation, and data quality monitoring. No static analysis or column-level type checking in
|
|
677
683
|
code. Supports pandas, Spark, and SQL backends.
|
|
678
684
|
|
|
685
|
+
- **[dataframely](https://github.com/Quantco/dataframely)** (v2.13.0): Polars-only runtime validation library from Quantco.
|
|
686
|
+
Schemas are defined as classes inheriting `dy.Schema` with typed descriptor fields (`dy.String()`, `dy.Float64()`)
|
|
687
|
+
and `@dy.rule()` decorators for cross-column and group-level constraints. Returns `dy.DataFrame[Schema]` generic
|
|
688
|
+
types that give call-site narrowing to type checkers, but does not validate column subscript access inside function
|
|
689
|
+
bodies. No lint-time or static analysis capability. Supports nullability, string constraints, numeric bounds,
|
|
690
|
+
cross-column rules, soft validation, test data generation, and SQLAlchemy/PyArrow export.
|
|
691
|
+
|
|
692
|
+
- **[patito](https://github.com/JakobGM/patito)** (v0.8.6): Runtime validation library using a Pydantic-style `patito.Model`
|
|
693
|
+
class. Polars is the primary backend; pandas is supported but works by converting to Polars via PyArrow (an
|
|
694
|
+
undeclared dependency). DuckDB is not supported despite appearing in some documentation—validation crashes immediately
|
|
695
|
+
on DuckDB relations. No static analysis or standalone checker.
|
|
696
|
+
|
|
679
697
|
### Type Checkers (Not DataFrame-Specific)
|
|
680
698
|
|
|
681
699
|
These are general Python type checkers. They don't validate DataFrame column names, but they can be used alongside
|
|
@@ -969,6 +987,8 @@ MIT License - see [LICENSE](LICENSE)
|
|
|
969
987
|
- [x] Pandera integration (`to_pandera_schema()`)
|
|
970
988
|
- [x] Cross-file schema inference (project-level index, `--no-index` flag)
|
|
971
989
|
- [x] Aggressive column inference (untracked-dataframe/dropped-unknown-column warnings, method chain propagation)
|
|
990
|
+
- [x] Function parameter contracts (`missing-column`), resolved transitively across chains of helper
|
|
991
|
+
functions and cross-file calls; schema-annotated parameters take priority over body-scanning
|
|
972
992
|
|
|
973
993
|
**Planned:**
|
|
974
994
|
|
|
@@ -1025,7 +1045,6 @@ Inspired by the needs of ML/data science teams working with complex data pipelin
|
|
|
1025
1045
|
|
|
1026
1046
|
---
|
|
1027
1047
|
|
|
1028
|
-
**Questions? Issues? Ideas?** [Open an issue](https://github.com/
|
|
1048
|
+
**Questions? Issues? Ideas?** [Open an issue](https://github.com/w-martin/typedframes/issues)
|
|
1029
1049
|
|
|
1030
1050
|
**Ready to catch DataFrame bugs before runtime?** `pip install typedframes`
|
|
1031
|
-
|