typedframes 0.14.0__py3-none-win_amd64.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,1266 @@
1
+ Metadata-Version: 2.4
2
+ Name: typedframes
3
+ Version: 0.14.0
4
+ Classifier: Development Status :: 3 - Alpha
5
+ Classifier: Intended Audience :: Developers
6
+ Classifier: License :: OSI Approved :: MIT License
7
+ Classifier: Operating System :: OS Independent
8
+ Classifier: Programming Language :: Python :: 3
9
+ Classifier: Programming Language :: Python :: 3.11
10
+ Classifier: Programming Language :: Python :: 3.12
11
+ Classifier: Programming Language :: Python :: 3.13
12
+ Classifier: Programming Language :: Python :: 3.14
13
+ Classifier: Programming Language :: Python :: 3.15
14
+ Classifier: Topic :: Software Development :: Quality Assurance
15
+ Classifier: Topic :: Software Development :: Libraries :: Python Modules
16
+ Requires-Dist: mypy ; extra == 'mypy'
17
+ Requires-Dist: pandas>=2.0 ; extra == 'pandas'
18
+ Requires-Dist: pandera>=0.20.0 ; extra == 'pandera'
19
+ Requires-Dist: polars>=1.0 ; extra == 'polars'
20
+ Provides-Extra: mypy
21
+ Provides-Extra: pandas
22
+ Provides-Extra: pandera
23
+ Provides-Extra: polars
24
+ License-File: LICENSE
25
+ Summary: Static analysis for pandas and polars DataFrames. Catch column errors at lint-time, not runtime.
26
+ Keywords: pandas,polars,type-checking,static-analysis,dataframe,linter,mypy-plugin
27
+ Home-Page: https://github.com/w-martin/typedframes
28
+ Author: William Martin
29
+ License-Expression: MIT
30
+ Requires-Python: >=3.11
31
+ Description-Content-Type: text/markdown; charset=UTF-8; variant=GFM
32
+ Project-URL: Documentation, https://typedframes.readthedocs.io/en/latest/
33
+ Project-URL: Homepage, https://github.com/w-martin/typedframes
34
+ Project-URL: Issue Tracker, https://github.com/w-martin/typedframes/issues
35
+ Project-URL: Repository, https://github.com/w-martin/typedframes
36
+
37
+ # typedframes
38
+
39
+ [![CI](https://github.com/w-martin/typedframes/actions/workflows/publish.yml/badge.svg)](https://github.com/w-martin/typedframes/actions/workflows/publish.yml)
40
+ [![PyPI version](https://img.shields.io/pypi/v/typedframes.svg)](https://pypi.org/project/typedframes/)
41
+ [![Python versions](https://img.shields.io/pypi/pyversions/typedframes.svg)](https://pypi.org/project/typedframes/)
42
+ [![Coverage](https://coveralls.io/repos/github/w-martin/typedframes/badge.svg?branch=main)](https://coveralls.io/github/w-martin/typedframes?branch=main)
43
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
44
+
45
+ > ⚠️ **Project Status: Proof of Concept**
46
+ >
47
+ > `typedframes` (v0.14.0) is currently an experimental proof-of-concept. The core static analysis and mypy/Rust
48
+ > integrations work, but expect rough edges. The codebase prioritizes demonstrating the viability of static DataFrame
49
+ > column checking over production-grade stability.
50
+ >
51
+ **A Rust-fast linter for pandas and polars DataFrames. Catches column errors at lint-time, and gates CI on how
52
+ much of your codebase it can actually see — no schema classes required to start.**
53
+
54
+ ```python
55
+ import pandas as pd
56
+
57
+ # Checker infers {order_id, amount, status} from usecols= — no schema class needed
58
+ orders = pd.read_csv("orders.csv", usecols=["order_id", "amount", "status"])
59
+ print(orders["amount"]) # ✓ OK
60
+ print(orders["revenue"]) # ✗ unknown-column — 'revenue' not in inferred column set
61
+ ```
62
+
63
+ ```shell
64
+ typedframes check src/ --coverage-fail-under=90
65
+ # src/pipeline.py:7:8: error[unknown-column] Column 'revenue' does not exist in inferred column set (defined at line 6)
66
+ # ✗ Found 1 error in 12 files (0.0s)
67
+ # ✗ DataFrame schema coverage 82.0% is below the required 90.0% (9/11 DataFrames had column info)
68
+ ```
69
+
70
+ `--coverage-fail-under=N` is the same idea as a test-coverage or mypy type-coverage gate, applied to how many DataFrames
71
+ the checker can resolve to a concrete column list — a DataFrame it merely recognizes but can't resolve columns for
72
+ (a Feast retrieval whose entity_df isn't itself concrete, a bare `-> pd.DataFrame` return with no attached Schema)
73
+ does not count, and any column accessed on it is flagged `unverifiable-column` rather than passed through silently.
74
+ See [DataFrame Schema Coverage Thresholds](#dataframe-schema-coverage-thresholds-opt-in).
75
+ Add `BaseSchema` classes later for cross-file awareness and IDE autocomplete — see [Quick Start](#quick-start).
76
+
77
+ ---
78
+
79
+ ## Table of Contents
80
+
81
+ - [Why typedframes?](#why-typedframes)
82
+ - [Installation](#installation)
83
+ - [Quick Start](#quick-start)
84
+ - [Column Inference](#column-inference)
85
+ - [Static Analysis](#static-analysis)
86
+ - [Jupyter Notebooks](#jupyter-notebooks)
87
+ - [DataFrame Schema Coverage Thresholds](#dataframe-schema-coverage-thresholds-opt-in)
88
+ - [Static Analysis Performance](#static-analysis-performance)
89
+ - [Type Safety With Multiple Backends](#type-safety-with-multiple-backends)
90
+ - [Advanced Usage](#advanced-usage)
91
+ - [Comparison](#comparison)
92
+ - [Pandera Integration](#pandera-integration)
93
+ - [Examples](#examples)
94
+ - [Philosophy](#philosophy)
95
+ - [FAQ](#faq)
96
+
97
+ ---
98
+
99
+ ## Why typedframes?
100
+
101
+ **The problem:** Many pandas bugs are column mismatches — you access a column that doesn't exist, pass a
102
+ DataFrame missing a column a function needs, or make a typo. These errors only surface at runtime, often in
103
+ production, and it's hard to know how much of a codebase is actually protected against them.
104
+
105
+ **The solution:** A fast standalone linter that infers column sets from your existing code (`usecols=`,
106
+ `dtype=`, method chains) and catches mismatches at lint-time — plus an opt-in coverage threshold so CI fails
107
+ when too much of the codebase is invisible to the checker. Add `BaseSchema` classes where you want cross-file
108
+ tracking and IDE autocomplete; they're a progressive enhancement, not a prerequisite.
109
+
110
+ **What you get:**
111
+
112
+ - ✅ **Works without schema annotations** - Column inference from `usecols=`, `dtype=`, and method chains catches errors on unannotated code
113
+ - ✅ **CI gate via coverage threshold** - `--coverage-fail-under=N` fails the build when too much of your codebase is invisible to the checker, the same way you'd gate on test coverage or a type checker's type-coverage number
114
+ - ✅ **Rust-fast** - Milliseconds, not seconds, even on hundreds of files; fast enough for pre-commit hooks and CI (see [benchmarks](#static-analysis-performance))
115
+ - ✅ **Cross-file awareness** - Add `BaseSchema` and typed return annotations to follow schemas across module boundaries
116
+ - ✅ **Refactor-safe access** - `df[Schema.column_group.s].mean()` (pandas) or `df.select(Schema.col.col)` (polars) instead of scattered string literals
117
+ - ✅ **Works with pandas AND polars** - Same schema API, native backend types
118
+ - ✅ **Dynamic column matching** - Regex-based ColumnSets for time-series data
119
+ - ✅ **Zero runtime overhead** - No validation, no slowdown
120
+ - ✅ **Type-safe backends** - Type checker knows pandas vs polars methods
121
+
122
+ ---
123
+
124
+ ## Installation
125
+
126
+ ```shell
127
+ pip install typedframes
128
+ ```
129
+ or
130
+ ```shell
131
+ uv add typedframes
132
+ ```
133
+
134
+ The `typedframes` command is a native Rust executable included in the wheel — no separate install
135
+ needed, and no Python process runs when you use it. Python is only needed for the schema classes and the mypy
136
+ plugin. `typedframes` itself has no required dependencies; add the extra for whichever backend(s) your code uses:
137
+
138
+ ```shell
139
+ pip install typedframes[pandas] # includes pandas
140
+ pip install typedframes[polars] # includes polars
141
+ ```
142
+
143
+ ---
144
+
145
+ ## Quick Start
146
+
147
+ ### Run on existing code
148
+
149
+ The checker works from day one without any schema classes. Pass `usecols=` / `columns=` to your read calls and
150
+ column access is validated automatically — no schema classes needed:
151
+
152
+ ```python
153
+ import pandas as pd
154
+
155
+ # Checker infers {order_id, amount, status} from usecols=
156
+ orders = pd.read_csv("orders.csv", usecols=["order_id", "amount", "status"])
157
+ print(orders["amount"]) # ✓ OK
158
+ print(orders["revenue"]) # ✗ unknown-column — 'revenue' not in inferred set
159
+ ```
160
+
161
+ ```shell
162
+ typedframes check src/
163
+ # src/pipeline.py:7:8: error[unknown-column] Column 'revenue' does not exist in inferred column set (defined at line 6)
164
+ # ✗ Found 1 error in 12 files (0.0s)
165
+ ```
166
+
167
+ See [`examples/features/multi_file_inference/`](examples/features/multi_file_inference/) for a multi-file example with no `BaseSchema`
168
+ classes at all.
169
+
170
+ ### Define Your Schema (Once)
171
+
172
+ Add `BaseSchema` classes when you want cross-file awareness and IDE autocomplete. Schemas travel with function return
173
+ types across module boundaries — the checker validates call sites even in files that have no `usecols=` of their own.
174
+
175
+ **Descriptors as a bridge:** define once in `Column(type=int)`, access as `df[UserData.user_id.s]` (pandas string
176
+ access) or `df.select(UserData.revenue.col)` (polars expression). Refactor by changing the descriptor definition —
177
+ all `.s` and `.col` references update automatically. No find-and-replace across string literals.
178
+
179
+ ```python
180
+ from typedframes import BaseSchema, Column, ColumnSet
181
+
182
+
183
+ class SalesData(BaseSchema):
184
+ date = Column(type=str)
185
+ revenue = Column(type=float)
186
+ customer_id = Column(type=int)
187
+
188
+ # Dynamic columns with regex
189
+ metrics = ColumnSet(type=float, members=r"metric_\d+", regex=True)
190
+ ```
191
+
192
+ ### Use With Pandas
193
+
194
+ ```python
195
+ from typing import Annotated
196
+ import pandas as pd
197
+
198
+ # Annotate your variable — checker validates all column access below
199
+ df: Annotated[pd.DataFrame, SalesData] = pd.read_csv("sales.csv")
200
+
201
+ # String access — validated by the standalone checker
202
+ print(df["revenue"].sum())
203
+ print(df["profit"]) # ✗ unknown-column: Column 'profit' not in SalesData
204
+
205
+ # .s gives a refactor-safe string name from the descriptor
206
+ print(df[SalesData.revenue.s].sum()) # same as df['revenue'].sum()
207
+
208
+
209
+ # Type-safe function signature
210
+ def analyze(data: Annotated[pd.DataFrame, SalesData]) -> float:
211
+ data["revenue"] # ✓ Validated by checker
212
+ data["profit"] # ✗ unknown-column: 'profit' not in SalesData
213
+ return data[SalesData.revenue.s].mean()
214
+ ```
215
+
216
+ ### Use With Polars
217
+
218
+ ```python
219
+ from typing import Annotated
220
+ import polars as pl
221
+
222
+ # Annotate your variable — checker validates pl.col() references too
223
+ df: Annotated[pl.DataFrame, SalesData] = pl.read_csv("sales.csv")
224
+
225
+ # pl.col() references are now validated by the standalone checker
226
+ print(df.filter(pl.col("revenue") > 1000))
227
+ print(df.select(pl.col("profit"))) # ✗ unknown-column: Column 'profit' not in SalesData
228
+
229
+ # .col gives a refactor-safe polars expression from the descriptor
230
+ filtered = df.filter(SalesData.revenue.col > 1000)
231
+ grouped = df.group_by("customer_id").agg(SalesData.revenue.col.sum())
232
+ ```
233
+
234
+ ---
235
+
236
+ ## Column Inference
237
+
238
+ The standalone checker works without any `BaseSchema` classes. It infers column sets directly from data
239
+ loading calls and method chains, so you get column validation even on completely unannotated code.
240
+ `BaseSchema` is a progressive enhancement: it adds cross-file awareness and IDE autocomplete, but the
241
+ checker catches real bugs from day one without it.
242
+
243
+ ### Inferred Schemas
244
+
245
+ When you pass `usecols=` (pandas) or `schema=` / `columns=` (polars), the checker builds an inferred column set
246
+ and validates all subscript access against it — no schema annotation required:
247
+
248
+ ```python
249
+ # Checker infers {user_id, email} from usecols= — no annotation needed
250
+ df = pd.read_csv("users.csv", usecols=["user_id", "email"])
251
+ print(df["user_id"]) # ✓ OK — in usecols
252
+ # print(df["age"]) # ✗ Error: 'age' not in inferred column set
253
+ ```
254
+
255
+ The checker also propagates column sets through method chains. Row-preserving operations (`filter`, `query`,
256
+ `head`, `tail`, `sort_values`, `dropna`, `fillna`, `ffill`, `bfill`, `reset_index`) pass the column set through
257
+ unchanged. Structural operations update it:
258
+
259
+ ```python
260
+ from typing import Annotated
261
+
262
+ df: Annotated[pd.DataFrame, UserData] = pd.read_csv("users.csv")
263
+
264
+ # Subscript slice — inferred column set {user_id, email}
265
+ small = df[["user_id", "email"]]
266
+ # print(small["age"]) # ✗ Error: 'age' not in inferred column set
267
+
268
+ # rename() — old name removed, new name added
269
+ renamed = small.rename(columns={"email": "email_address"})
270
+ print(renamed["email_address"]) # ✓ OK
271
+
272
+ # drop() — column removed from inferred set
273
+ trimmed = df.drop(columns=["age"])
274
+ # print(trimmed["age"]) # ✗ Error: 'age' was dropped
275
+
276
+ # assign() — new column added to inferred set
277
+ augmented = df.assign(created_at="2024-01-01")
278
+ print(augmented["created_at"]) # ✓ OK
279
+ ```
280
+
281
+ ### Inference Gaps and Warnings
282
+
283
+ **untracked-dataframe — unannotated data ingestion (on by default)**
284
+
285
+ When a DataFrame is loaded via `pd.read_csv()` without `usecols=` or a schema annotation, the checker
286
+ assumes an *Unknown* state, bypasses strict column validation on it (to avoid false positives on columns
287
+ it simply can't see), and flags the load itself as a warning-level diagnostic.
288
+
289
+ For permissive Exploratory Data Analysis (EDA) work where you don't want that noise yet, downgrade it to
290
+ a quiet info-level note with `--lenient-ingest`:
291
+
292
+ ```shell
293
+ typedframes check src/ --lenient-ingest
294
+ ```
295
+
296
+ By default, loading a DataFrame without a schema or `usecols=` produces:
297
+
298
+ ```python
299
+ df = pd.read_csv("users.csv")
300
+ # ⚠ untracked-dataframe: columns unknown at lint time; specify `usecols`/`columns`, or
301
+ # annotate the variable's type, e.g. `df: Annotated[pd.DataFrame, MySchema] = pd.read_csv(...)`
302
+ ```
303
+
304
+ Fix option 1 — annotate with a schema:
305
+ ```python
306
+ from typing import Annotated
307
+
308
+ df: Annotated[pd.DataFrame, UserData] = pd.read_csv("users.csv")
309
+ ```
310
+
311
+ Fix option 2 — pass `usecols=`:
312
+ ```python
313
+ df = pd.read_csv("users.csv", usecols=["user_id", "email"])
314
+ ```
315
+
316
+ **dropped-unknown-column — dropped column does not exist**
317
+
318
+ Emitted when `drop(columns=[...])` names a column that isn't in the inferred set:
319
+
320
+ ```python
321
+ from typing import Annotated
322
+
323
+ df: Annotated[pd.DataFrame, UserData] = pd.read_csv("users.csv")
324
+ trimmed = df.drop(columns=["nonexistent"])
325
+ # ⚠ dropped-unknown-column: Dropped column 'nonexistent' does not exist in UserData
326
+ ```
327
+
328
+ ### Function Parameter Contracts
329
+
330
+ Beyond validating access at the point it happens, the checker infers a *contract* for any function's
331
+ first parameter: every column the function needs, drawn from what its body accesses or — taking priority —
332
+ from a schema annotation on the parameter itself. Calling that function with a DataFrame that doesn't
333
+ satisfy the contract is caught at the **call site**, across files:
334
+
335
+ ```python
336
+ # transforms.py
337
+ def contact_label(customers):
338
+ return customers["name"] + customers["email"]
339
+ ```
340
+
341
+ ```python
342
+ # pipeline.py
343
+ customers = load_customers(path) # inferred columns: {customer_id, name, region}
344
+ contact_label(customers)
345
+ # ✗ missing-column: 'customers' passed to contact_label (transforms.py:2) is missing
346
+ # column(s) {email} — available: {customer_id, name, region}, required: {email, name}
347
+ ```
348
+
349
+ The contract is resolved *transitively*: if a function only forwards its parameter to other functions
350
+ (`step1 = preprocess(df); step2 = enrich(step1)`), the checker follows the chain and unions their
351
+ requirements, catching a missing column even when no single function in the chain touches it directly.
352
+ Column-list slices (`df[["a", "b"]]`) contribute to the contract too.
353
+
354
+ **Known limitations:**
355
+ - Cross-file delegate/schema resolution follows `from module import name`, plain `import module` +
356
+ `module.helper(df)` attribute access, and `from module import *` wildcard imports. A dotted import with no
357
+ alias (`import a.b.c`) only binds the first segment (`a`), matching Python's own binding rules, so a
358
+ deeply nested submodule accessed without an alias is not tracked.
359
+ - Contract inference is a single top-to-bottom pass over a function body, not full control-flow analysis.
360
+ Deeply nested control flow (nested `try`/`except`, `match`, comprehensions) may under-report a function's
361
+ true requirements.
362
+ - A cycle in the delegate graph (mutually- or self-delegating helpers) contributes only each function's own
363
+ direct requirements to the cycle, not the full transitive union — conservative rather than exhaustive.
364
+ - If two plainly-imported modules both define a same-named function, an attribute-style delegate call
365
+ (`module.helper(df)`) resolves to whichever one is found first — the checker doesn't disambiguate by which
366
+ module the call site actually used.
367
+
368
+ ### See Also
369
+
370
+ - [`examples/features/inference_example.py`](examples/features/inference_example.py) — single-file walkthrough of all four inference
371
+ scenarios with annotated ✓/✗ comments.
372
+ - [`examples/features/multi_file_inference/`](examples/features/multi_file_inference/) — multi-file project checked with
373
+ `typedframes check examples/features/multi_file_inference/`; no `BaseSchema` anywhere. Includes a function
374
+ parameter contract violation caught at the call site (`missing-column`).
375
+ - [`examples/features/multi_file_with_schema/`](examples/features/multi_file_with_schema/) — same scenario with `BaseSchema`
376
+ classes; the checker follows schemas across module boundaries via the project index.
377
+ - SQL / data-warehouse column inference — the column set is inferred from a query's `SELECT` list instead of
378
+ `usecols=`/`columns=`, including tracing the query back through a single-assignment variable or a `.sql` file,
379
+ and dialect-aware identifier case folding (`sql_dialect` in `pyproject.toml` — see [Project-level
380
+ configuration](docs/api/cli.md#project-level-configuration)):
381
+ [`examples/sql_connectors/snowflake/`](examples/sql_connectors/snowflake/), [`examples/sql_connectors/bigquery/`](examples/sql_connectors/bigquery/),
382
+ [`examples/sql_connectors/athena/`](examples/sql_connectors/athena/), [`examples/sql_connectors/redshift/`](examples/sql_connectors/redshift/),
383
+ [`examples/sql_connectors/databricks/`](examples/sql_connectors/databricks/), [`examples/sql_connectors/pyspark/`](examples/sql_connectors/pyspark/),
384
+ [`examples/sql_connectors/duckdb/`](examples/sql_connectors/duckdb/), [`examples/sql_connectors/connectorx/`](examples/sql_connectors/connectorx/),
385
+ [`examples/sql_connectors/sqlalchemy/`](examples/sql_connectors/sqlalchemy/) (Core `select()` and declarative models, not just raw SQL text),
386
+ [`examples/sql_connectors/feast/`](examples/sql_connectors/feast/) (feature-store retrieval, registered as an *open* schema since
387
+ `entity_df`'s own columns aren't enumerable in general), and
388
+ [`examples/sql_connectors/azure_synapse/`](examples/sql_connectors/azure_synapse/) (Azure's closest analog to Athena, including T-SQL's
389
+ `[bracket-quoted]` identifier convention). A wrapper function that case-folds a connector's result before
390
+ returning it (`.rename(columns=str.lower)`, `df.columns = df.columns.str.lower()`) is traced cross-file
391
+ too — see [docs/usage.md's "Supported column-set transforms"](docs/usage.md#supported-column-set-transforms).
392
+
393
+ ---
394
+
395
+ ## Static Analysis
396
+
397
+ typedframes provides **two ways** to check your code:
398
+
399
+ ### Option 1: Standalone Checker (Fast)
400
+
401
+ ```shell
402
+ # Blazing fast Rust-based checker
403
+ typedframes check src/
404
+
405
+ # Output (ty-style, auto-colored in terminals):
406
+ # src/analysis.py:23:8: error[unknown-column] Column 'profit' not in SalesData
407
+ # src/pipeline.py:56:8: error[unknown-column] Column 'user_name' not in UserData
408
+ # ✗ Found 2 errors in 47 files (0.0s)
409
+ ```
410
+
411
+ **Features:**
412
+ - Catches column name errors
413
+ - Validates schema mismatches between functions
414
+ - Validates function parameter contracts across files, including transitively through chains of
415
+ helper functions (`missing-column`)
416
+ - Checks both pandas and polars code
417
+ - Significantly faster than mypy (see benchmarks below)
418
+
419
+ **Use this for:**
420
+ - Fast feedback during development
421
+ - CI/CD pipelines
422
+ - Pre-commit hooks
423
+
424
+ **Configuration:**
425
+ ```shell
426
+ # Check specific files
427
+ typedframes check src/pipeline.py
428
+
429
+ # Check directory (builds cross-file index automatically)
430
+ typedframes check src/
431
+
432
+ # Fail on any error (for CI)
433
+ typedframes check src/ --strict
434
+
435
+ # JSON output
436
+ typedframes check src/ --output-format=json
437
+
438
+ # Check a single file (follows that file's own imports, not the whole project)
439
+ typedframes check src/pipeline.py
440
+
441
+ # Skip the cross-file index entirely (every file in isolation, fastest)
442
+ typedframes check src/ --no-index
443
+
444
+ # Suppress all warnings (untracked-dataframe, dropped-unknown-column)
445
+ typedframes check src/ --no-warnings
446
+
447
+ # Enforce minimum DataFrame schema coverage (see below)
448
+ typedframes check src/ --coverage-fail-under=90
449
+
450
+ # Show which DataFrames lack column info, per file
451
+ typedframes check src/ --coverage-detail=term-missing
452
+ ```
453
+
454
+ To suppress warnings project-wide, add to `pyproject.toml`:
455
+ ```toml
456
+ [tool.typedframes]
457
+ enabled = true
458
+ warnings = false
459
+ ```
460
+
461
+ ### Option 2: Mypy Plugin (Comprehensive)
462
+
463
+ ```shell
464
+ # Add to pyproject.toml
465
+ [tool.mypy]
466
+ plugins = ["typedframes.mypy"]
467
+
468
+ # Or mypy.ini
469
+ [mypy]
470
+ plugins = typedframes.mypy
471
+
472
+ # Run mypy
473
+ mypy src/
474
+ ```
475
+
476
+ **Features:**
477
+ - Full type checking across your codebase
478
+ - Catches column errors AND regular type errors
479
+ - IDE integration (VSCode, PyCharm)
480
+ - Works with existing mypy configuration
481
+
482
+ **Use this for:**
483
+ - Comprehensive type checking
484
+ - Integration with existing mypy setup
485
+ - IDE error highlighting
486
+
487
+ ### Supported Operations
488
+
489
+ The checker tracks schema changes through `rename`, `drop`, `assign`, `select`, `pop`,
490
+ `insert`, `del`, subscript assignment, `merge`, and `concat`. Row-passthrough operations
491
+ like `filter`, `query`, `head`, `sort_values`, and `dropna` are validated without schema
492
+ changes. Operations with runtime-dependent output (`join`, `pivot`, `melt`, `groupby`,
493
+ `apply`, etc.) are left untracked to avoid false positives.
494
+
495
+ See the full [Method Matrix](https://typedframes.readthedocs.io/en/latest/method-matrix/)
496
+ for the complete list of tracked, passthrough, and untracked operations, plus the error
497
+ code reference.
498
+
499
+ ### Pre-commit Hook
500
+
501
+ ```yaml
502
+ # .pre-commit-config.yaml
503
+ repos:
504
+ - repo: local
505
+ hooks:
506
+ - id: typedframes
507
+ name: typedframes check
508
+ entry: typedframes check . --strict
509
+ language: python
510
+ additional_dependencies: ["typedframes==0.14.0"]
511
+ types_or: [python, jupyter]
512
+ pass_filenames: false
513
+ ```
514
+
515
+ `pass_filenames: false` is deliberate: the checker takes a single path and builds its
516
+ cross-file index from it, so one run per commit still sees the modules the commit didn't
517
+ touch — a per-file invocation would lose that. `--strict` is what makes the hook block
518
+ the commit: `typedframes check` exits 0 by default even when it finds real errors. Add
519
+ the coverage gate to the same line:
520
+
521
+ ```yaml
522
+ # with coverage gating
523
+ entry: typedframes check . --strict --coverage-fail-under=90
524
+ ```
525
+
526
+ This repository also ships a `.pre-commit-hooks.yaml`, so once a tagged release contains
527
+ it you can point `repo:` straight at `https://github.com/w-martin/typedframes` with a
528
+ `rev:` and drop the boilerplate above. That form needs a Rust toolchain wherever the hook
529
+ runs — pre-commit's python installer builds the repository from source rather than
530
+ fetching a wheel — so the local hook stays the quicker one to install either way.
531
+
532
+ ### GitHub Actions
533
+
534
+ ```yaml
535
+ # .github/workflows/typedframes.yml
536
+ name: typedframes
537
+ on: [push, pull_request]
538
+
539
+ jobs:
540
+ typedframes:
541
+ runs-on: ubuntu-latest
542
+ steps:
543
+ - uses: actions/checkout@v4
544
+ # No tagged release contains action.yml yet. Repin this to that tag once one
545
+ # ships: an unpinned third-party action is a supply-chain risk, and @main is a
546
+ # stopgap rather than a recommendation.
547
+ - uses: w-martin/typedframes@main
548
+ with:
549
+ path: src/
550
+ coverage-fail-under: "90"
551
+ ```
552
+
553
+ The action installs the PyPI wheel into a throwaway virtualenv and runs the checker with
554
+ `--output-format=github`, so errors arrive as annotations on the pull request diff.
555
+
556
+ | Input | Default | |
557
+ |-------|---------|-|
558
+ | `path` | `.` | File or directory to check |
559
+ | `version` | `latest` | PyPI version to install, e.g. `"0.14.0"` |
560
+ | `strict` | `true` | Fail the step on errors — `typedframes check` exits 0 without it |
561
+ | `coverage-fail-under` | *(unset)* | Minimum DataFrame schema coverage, e.g. `"90"` |
562
+ | `coverage-detail` | `summary` | Or `term-missing` for the per-file breakdown, `explain` to diagnose a lower-than-expected total |
563
+ | `no-warnings` | `false` | Suppress warning-level diagnostics |
564
+ | `args` | *(empty)* | Escape hatch for `--no-index`, `--lenient-ingest`, `--no-info` |
565
+
566
+ ### Fixing Untyped DataFrames (`suggest`)
567
+
568
+ For a DataFrame the checker can't resolve columns for, `typedframes suggest` works out a fix from the columns the rest
569
+ of its scope reads: a `usecols=`/`columns=` added to a plain `read_csv` or `read_parquet`, or else a column selection on
570
+ the line after the load. It is a dry run that prints diffs. Each site has a two-word name (`quiet-amber`) shown next to it.
571
+
572
+ ```shell
573
+ typedframes suggest src/ # dry run: prints diffs, changes nothing
574
+ typedframes suggest src/ --apply # write the safe suggestions
575
+ typedframes suggest src/ --apply --only quiet-amber # act on just that site
576
+ typedframes suggest src/ --interactive # ask about each suggestion
577
+ ```
578
+
579
+ Edits need a clean git work tree and tracked files, each edited file is re-checked in memory before anything is written,
580
+ and the files are written together or not at all. See the
581
+ [full guide](docs/usage.md#suggesting-fixes-suggest).
582
+
583
+ ### Reporting a Problem (`report`)
584
+
585
+ If `check` or `suggest` gets a site wrong and the code can't be shared, `typedframes report quiet-amber src/` prints a
586
+ bug-report body with a redacted copy of the code around that site, checked against the checker so it reproduces the same
587
+ result. `typedframes report --all src/` does this for every untyped site, grouped by why the checker decided as it did.
588
+ Redacted is not anonymous: read it before posting, and don't use it on code you aren't allowed to share. See
589
+ [Reporting a problem](docs/usage.md#reporting-a-problem-report), and
590
+ [Security and data handling](docs/security.md) for what the tool reads, writes and connects to.
591
+
592
+ ---
593
+
594
+ ## Jupyter Notebooks
595
+
596
+ The standalone checker reads `.ipynb` files directly — no `nbconvert`, no `jupytext`, no
597
+ flag, no conversion step. Notebooks are parsed with
598
+ [`ruff_notebook`](https://github.com/astral-sh/ruff/tree/main/crates/ruff_notebook), the
599
+ same crate Ruff uses for its own notebook support:
600
+
601
+ ```shell
602
+ # Check one notebook
603
+ typedframes check analysis.ipynb
604
+
605
+ # Check a directory — .py and .ipynb files are both picked up, no flag needed
606
+ typedframes check notebooks/
607
+
608
+ # Output — `cell N` is the notebook's Nth cell; line:col are relative to that cell:
609
+ # analysis.ipynb:cell 6:2:1: warning[untracked-dataframe] columns unknown at lint time; specify
610
+ # `usecols`/`columns`, or annotate the variable's type
611
+ # analysis.ipynb:cell 10:1:1: error[unknown-column] Column 'revenue' does not exist in Orders
612
+ # {amount, customer_id, order_id} (defined at analysis.ipynb cell 2:8)
613
+ # ✗ Found 1 error, 2 warnings in 1 file (0.0s)
614
+ ```
615
+
616
+ **Features:**
617
+ - ✅ Auto-detected — every `.ipynb` under a checked path, alongside `.py` files
618
+ - ✅ Diagnostics reported as `cell N:line:col`, mapped back to the cell that produced them
619
+ rather than a line number in the raw notebook JSON — including a same-notebook schema's
620
+ `(defined at ...)` cross-reference
621
+ - ✅ Same error codes, severities, and flags as `.py` files: `--strict`,
622
+ `--output-format`, `--no-warnings`, `--lenient-ingest`, coverage thresholds
623
+ - ✅ IPython magics and shell escapes parse natively (`%matplotlib inline`, `%%time`,
624
+ `!pip install ...`) — nothing has to be stripped first
625
+ - ✅ `# typedframes: ignore` suppresses a diagnostic on a line inside a cell, same as in a
626
+ `.py` file
627
+ - ✅ A schema imported from a project `.py` module resolves through the cross-file index,
628
+ so `orders = load_orders()` in a notebook is validated against the schema on
629
+ `load_orders`' return annotation
630
+ - ✅ `.ipynb_checkpoints/` is pruned by default, so stale checkpoint copies aren't
631
+ re-reported
632
+
633
+ **Caveats:**
634
+ - ❌ The mypy plugin ([Option 2](#option-2-mypy-plugin-comprehensive)) is `.py`-only —
635
+ mypy reads Python source files, so notebook checking is standalone-checker-only
636
+ - ❌ `cell N` counts *every* cell in the notebook, markdown and raw cells included — so
637
+ the number is a cell's position in the notebook, not "the Nth code cell"
638
+ - ❌ A notebook is never a *source* of schemas for other files (Python can't import an
639
+ `.ipynb`); a schema defined in a notebook is visible only inside that notebook
640
+ - ❌ A notebook that isn't valid notebook JSON, whose kernel isn't Python, or whose code
641
+ doesn't parse is skipped with a message on stderr and the run continues — a skip alone
642
+ never fails the run, `--strict` included
643
+ - ❌ Under `--output-format=github`, the annotation's `line`/`col` stay cell-relative and
644
+ the cell number moves into the annotation title, since GitHub annotations address raw
645
+ file lines and an `.ipynb` is JSON rather than source text
646
+
647
+ ### See Also
648
+
649
+ - [`examples/features/ipynb_example.ipynb`](examples/features/ipynb_example.ipynb) — a
650
+ runnable notebook with one schema, an IPython magic, and three intentional bugs
651
+ (`untracked-dataframe`, `dropped-unknown-column`, `unknown-column`), all caught by
652
+ `typedframes check ipynb_example.ipynb` without running a cell.
653
+
654
+ ---
655
+
656
+ ## DataFrame Schema Coverage Thresholds (Opt-In)
657
+
658
+ **DataFrame schema coverage** is the fraction of DataFrames `typedframes check` could
659
+ resolve column information for — the analogue of the "type coverage" reported by mypy,
660
+ pyright, and pyre, and unrelated to test coverage. That number is informational by
661
+ default. If you want it enforced — failing the run when too much of your code is
662
+ invisible to the checker — enable a threshold:
663
+
664
+ ```toml
665
+ [tool.typedframes.coverage]
666
+ enabled = true
667
+ fail_under = 90.0
668
+ ```
669
+
670
+ or as a one-off, without touching config:
671
+
672
+ ```shell
673
+ typedframes check src/ --coverage-fail-under=90
674
+ ```
675
+
676
+ This is **entirely opt-in** — with no `[tool.typedframes.coverage]` table and no
677
+ `--coverage-fail-under`, nothing changes: no threshold, no exit-code difference. Per-path
678
+ overrides (e.g. a lower bar for `legacy/**`), a `--coverage-detail=term-missing`
679
+ breakdown of exactly which DataFrames cost you coverage (as text or, combined with
680
+ `--output-format=json`, as structured JSON), and the full config reference all live in
681
+ the
682
+ [DataFrame schema coverage thresholds guide](docs/usage.md#dataframe-schema-coverage-thresholds).
683
+
684
+ Some DataFrames are recognized but never resolve to a concrete column list — a Feast
685
+ retrieval whose `entity_df` isn't itself concretely typed, or a bare `-> pd.DataFrame`
686
+ return with no attached `Schema`. These do **not** count as covered, and accessing any
687
+ column on one is flagged `unverifiable-column` rather than passed through silently —
688
+ the checker genuinely doesn't know whether the column exists, so it says so instead of
689
+ guessing either way. A Feast retrieval whose `entity_df` argument IS a concretely typed
690
+ DataFrame is fully resolved instead: Feast's real output is `entity_df`'s own columns
691
+ plus the requested features, so both halves get checked normally. See
692
+ [Unresolved schemas](docs/usage.md#unresolved-schemas-recognized-but-not-column-checked).
693
+
694
+ ## Static Analysis Performance
695
+
696
+ Fast feedback reduces development time. The typedframes Rust binary provides near-instant column checking.
697
+
698
+ **Benchmark results** (median of 5 runs ± one standard deviation, 3 warmup, caches cleared between runs):
699
+ *2026-10-13 · Darwin 27.0.0 · arm · CPython 3.15.0 · 64GiB RAM · Great Expectations pinned @ 1.23.2*
700
+
701
+ | Tool | Version | What it does | typedframes (12 files) | great_expectations (494 files) |
702
+ |------|---------|--------------|------------------------|--------------------------------|
703
+ | typedframes | 0.14.0 | DataFrame column checker | 18.1 ± 0.2 ms | 48.6 ± 0.1 ms |
704
+ | ruff | 0.17.0 | Linter (no type checking) | 25.1 ± 1.7 ms | 214.6 ± 2.0 ms |
705
+ | ty | 0.0.84 | Type checker | 51.1 ± 10.3 ms | 754.2 ± 9.0 ms |
706
+ | pyrefly | 1.3.2 | Type checker | 99.3 ± 11.0 ms | 299.0 ± 6.4 ms |
707
+ | mypy | 2.4.0 | Type checker (no plugin) | 1.79 ± 0.01 s | 2.99 ± 0.03 s |
708
+ | mypy + typedframes | 2.4.0 | Type checker + column checker | 1.81 ± 0.01 s | 3.16 ± 0.01 s |
709
+ | pyright | 1.1.414 | Type checker | 656.8 ± 5.8 ms | 3.86 ± 0.07 s |
710
+
711
+ *Run `uv run python benchmarks/benchmark_checkers.py` to reproduce.*
712
+
713
+ The typedframes binary resolves column names within a file and, when a project index is present, across files too.
714
+ Run `typedframes check src/` to build the index automatically and catch errors like `df = load_users(); df["typo"]`
715
+ even when `load_users` is defined in another module. Checking a single file
716
+ (`typedframes check src/pipeline.py`) builds a smaller index from that file's own imports instead of walking the whole
717
+ project, so it still resolves `load_users` without paying for files it never references. Pass `--no-index` to skip the
718
+ index entirely and check each file in isolation. Full type checkers (mypy, pyright, ty) analyze all Python types across
719
+ your entire codebase. Use both: the binary for fast iteration, mypy for comprehensive checking.
720
+
721
+ The standalone checker is built with [`ruff_python_parser`](https://github.com/astral-sh/ruff) for Python AST
722
+ parsing.
723
+
724
+ **Note:** ty (Astral) does not currently support mypy plugins, so use the standalone binary for column checking with ty.
725
+
726
+ ### Index Cache and Parallelism
727
+
728
+ Files are indexed and checked in parallel, one worker per core (up to 16). Set `TYPEDFRAMES_JOBS=1` to run on a single thread, or
729
+ a number to cap the workers.
730
+
731
+ What the checker learns about each file is also cached on disk and reused for every file whose contents have not
732
+ changed, so repeat runs only re-index what you edited. A cached run reports exactly what an uncached run would. The cache
733
+ lives in your per-user cache directory by default (`~/Library/Caches/typedframes` on macOS, `~/.cache/typedframes` on
734
+ Linux), so nothing is added to your repository.
735
+
736
+ ```shell
737
+ typedframes cache path # where is it?
738
+ typedframes cache clear # delete it
739
+ typedframes check src/ --no-cache # skip it for one run
740
+ typedframes check src/ --cache-dir .typedframes_cache # keep it in the project instead
741
+ typedframes cache gitignore # add a project-local cache directory to .gitignore
742
+ ```
743
+
744
+ To set it permanently, use the `TYPEDFRAMES_CACHE_DIR` environment variable or `pyproject.toml`; a relative path is
745
+ relative to the project:
746
+
747
+ ```toml
748
+ [tool.typedframes]
749
+ cache_dir = ".typedframes_cache"
750
+ ```
751
+
752
+ The order of precedence is `--no-cache`, `--cache-dir`, `TYPEDFRAMES_CACHE_DIR`, `cache_dir`, then the default. See
753
+ `typedframes cache --help`.
754
+
755
+ ---
756
+
757
+ ## Type Safety With Multiple Backends
758
+
759
+ typedframes uses **native backend types** to ensure complete type safety:
760
+
761
+ ```python
762
+ from typing import Annotated
763
+ import pandas as pd
764
+ import polars as pl
765
+ from typedframes import BaseSchema, Column
766
+
767
+
768
+ class UserData(BaseSchema):
769
+ user_id = Column(type=int)
770
+ email = Column(type=str)
771
+
772
+
773
+ # Pandas pipeline - type checker knows pandas methods
774
+ def pandas_analyze(df: Annotated[pd.DataFrame, UserData]) -> Annotated[pd.DataFrame, UserData]:
775
+ return df[df["user_id"] > 100] # ✓ Pandas syntax
776
+
777
+
778
+ # Polars pipeline - type checker knows polars methods
779
+ def polars_analyze(df: Annotated[pl.DataFrame, UserData]) -> Annotated[pl.DataFrame, UserData]:
780
+ return df.filter(pl.col("user_id") > 100) # ✓ Polars syntax
781
+
782
+
783
+ # Use native types throughout
784
+ df_pandas: Annotated[pd.DataFrame, UserData] = pd.read_csv("data.csv")
785
+ df_polars: Annotated[pl.DataFrame, UserData] = pl.read_csv("data.csv")
786
+
787
+ pandas_analyze(df_pandas) # ✓ OK
788
+ polars_analyze(df_polars) # ✓ OK
789
+ ```
790
+
791
+ ---
792
+
793
+ ## Advanced Usage
794
+
795
+ ### Merges, Joins, and Filters
796
+
797
+ Schema-typed DataFrames preserve their type through common operations:
798
+
799
+ **Pandas:**
800
+
801
+ ```python
802
+ from typing import Annotated
803
+ import pandas as pd
804
+ from typedframes import BaseSchema, Column
805
+
806
+
807
+ class UserSchema(BaseSchema):
808
+ user_id = Column(type=int)
809
+ email = Column(type=str)
810
+
811
+
812
+ class OrderSchema(BaseSchema):
813
+ order_id = Column(type=int)
814
+ user_id = Column(type=int)
815
+ total = Column(type=float)
816
+
817
+
818
+ # Schema preserved through filtering
819
+ def get_active_users(df: Annotated[pd.DataFrame, UserSchema]) -> Annotated[pd.DataFrame, UserSchema]:
820
+ return df[df["user_id"] > 100] # ✓ Validated by checker
821
+
822
+
823
+ # Schema preserved through merges
824
+ users: Annotated[pd.DataFrame, UserSchema] = pd.read_csv("users.csv")
825
+ orders: Annotated[pd.DataFrame, OrderSchema] = pd.read_csv("orders.csv")
826
+ merged = users.merge(orders, on=UserSchema.user_id.s)
827
+ ```
828
+
829
+ **Polars:**
830
+ ```python
831
+ from typing import Annotated
832
+ import polars as pl
833
+
834
+
835
+ # Schema columns work in filter expressions
836
+ def filter_users(df: Annotated[pl.DataFrame, UserSchema]) -> pl.DataFrame:
837
+ return df.filter(pl.col("user_id") > 100)
838
+
839
+
840
+ # Schema columns work in join expressions
841
+ def join_data(
842
+ users: Annotated[pl.DataFrame, UserSchema],
843
+ orders: Annotated[pl.DataFrame, OrderSchema],
844
+ ) -> pl.DataFrame:
845
+ return users.join(
846
+ orders,
847
+ left_on=UserSchema.user_id.s,
848
+ right_on=OrderSchema.user_id.s,
849
+ )
850
+
851
+
852
+ # Schema columns work in select expressions
853
+ def select_columns(df: Annotated[pl.DataFrame, UserSchema]) -> pl.DataFrame:
854
+ return df.select([UserSchema.user_id.s, UserSchema.email.s])
855
+ ```
856
+
857
+ ### Dynamic Column Matching
858
+
859
+ Perfect for time-series data where column counts change. Regex ColumnSets document which columns belong
860
+ to a group and are validated by the static checker. The `.s` property gives you the list of column names
861
+ for explicit (non-regex) ColumnSets; for non-regex groups you can also use `.cols()` for polars expressions.
862
+
863
+ ```python
864
+ from typing import Annotated
865
+ import pandas as pd
866
+ from typedframes import BaseSchema, Column, ColumnSet, ColumnGroup
867
+
868
+
869
+ class SensorReadings(BaseSchema):
870
+ timestamp = Column(type=str)
871
+ # Explicit sensor columns — refactor-safe list access via .s
872
+ sensors = ColumnSet(type=float, members=["sensor_1", "sensor_2", "sensor_3"])
873
+
874
+
875
+ df: Annotated[pd.DataFrame, SensorReadings] = pd.read_csv("readings.csv")
876
+ df[SensorReadings.sensors.s].mean() # ✓ Expands to df[["sensor_1", "sensor_2", "sensor_3"]].mean()
877
+ ```
878
+
879
+ For logical grouping across multiple ColumnSets:
880
+
881
+ ```python
882
+ class TimeSeriesData(BaseSchema):
883
+ timestamp = Column(type=str)
884
+ temperature = ColumnSet(type=float, members=["temp_1", "temp_2", "temp_3"])
885
+ pressure = ColumnSet(type=float, members=["pressure_1", "pressure_2"])
886
+
887
+ # Group for convenient access to all sensor columns
888
+ sensors = ColumnGroup(members=[temperature, pressure])
889
+
890
+
891
+ df: Annotated[pd.DataFrame, TimeSeriesData] = pd.read_csv("sensors.csv")
892
+ avg_temp = df[TimeSeriesData.temperature.s].mean()
893
+ all_readings = df[TimeSeriesData.sensors.s].describe()
894
+ ```
895
+
896
+ ### Schema Composition
897
+
898
+ Compose upward — build bigger schemas from smaller ones via inheritance. Type checkers see all columns natively.
899
+
900
+ ```python
901
+ from typing import Annotated
902
+ import pandas as pd
903
+ from typedframes import BaseSchema, Column
904
+
905
+
906
+ # Start with the smallest useful schema
907
+ class UserPublic(BaseSchema):
908
+ user_id = Column(type=int)
909
+ email = Column(type=str)
910
+ name = Column(type=str)
911
+
912
+
913
+ # Extend it — never strip down
914
+ class UserFull(UserPublic):
915
+ password_hash = Column(type=str)
916
+
917
+
918
+ class Orders(BaseSchema):
919
+ order_id = Column(type=int)
920
+ user_id = Column(type=int)
921
+ total = Column(type=float)
922
+
923
+
924
+ # Combine via multiple inheritance
925
+ class UserOrders(UserPublic, Orders):
926
+ """Type checkers see all columns from both parents."""
927
+
928
+ ...
929
+
930
+
931
+ # Or use the + operator
932
+ UserOrdersDynamic = UserPublic + Orders
933
+
934
+ users: Annotated[pd.DataFrame, UserPublic] = pd.read_csv("users.csv")
935
+ orders: Annotated[pd.DataFrame, Orders] = pd.read_csv("orders.csv")
936
+ merged: Annotated[pd.DataFrame, UserOrders] = users.merge(orders, on=UserPublic.user_id.s)
937
+ ```
938
+
939
+ Overlapping columns with the same type are allowed (common after merges). Conflicting types raise `SchemaConflictError`.
940
+
941
+ See [`examples/features/schema_algebra_example.py`](examples/features/schema_algebra_example.py) for a complete walkthrough.
942
+
943
+ ---
944
+
945
+ ## Comparison
946
+
947
+ ### Feature Matrix (Static Analysis Focus)
948
+
949
+ Comprehensive comparison of pandas/DataFrame typing and validation tools. **typedframes focuses on static analysis**
950
+ —catching errors at lint-time before your code runs.
951
+
952
+ | Feature | typedframes | Pandera | Great Expectations | strictly_typed_pandas | pandas-stubs | dataenforce | pandas-type-checks | StaticFrame | narwhals | dataframely | patito |
953
+ |---------------------------------|------------------------|-------------|--------------------|-----------------------|--------------|-------------|--------------------|------------------|----------|------------------|------------------|
954
+ | **Version tested** | 0.12.0 | 0.34.1 | 1.23.2 | 0.3.7 | 3.0.5.260914 | 0.1.2 | 1.1.3 | 5.1.1 | 2.26.0 | 3.1.2 | 0.8.6 |
955
+ | **Analysis Type** |
956
+ | When errors are caught | **Static (lint-time)** | Runtime | Runtime | Runtime | Static | Runtime | Runtime | Runtime | Runtime | Runtime | Runtime |
957
+ | **Static Analysis (our focus)** |
958
+ | Mypy plugin | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No | ⚠️ Basic | ❌ No | ❌ No | ❌ No |
959
+ | Standalone checker | ✅ Rust (ms-scale) | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
960
+ | Column name checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
961
+ | Column type checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
962
+ | Typo suggestions | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
963
+ | Coverage gate (`--coverage-fail-under`) | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
964
+ | **Runtime Validation** |
965
+ | Data validation | ❌ No | ✅ Excellent | ✅ Excellent | ✅ typeguard | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
966
+ | Value constraints | ❌ No | ✅ Yes | ✅ Excellent | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
967
+ | **Schema Features** |
968
+ | Column grouping | ✅ ColumnGroup | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
969
+ | Regex column matching | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
970
+ | **Backend Support** |
971
+ | Pandas | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | ❌ Own | ✅ Yes | ❌ No | ⚠️ Limited |
972
+ | Polars | ✅ Yes | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ Own | ✅ Yes | ✅ Yes (only) | ✅ Yes |
973
+ | DuckDB, cuDF, etc. | ❌ No | ❌ No | ✅ Spark, SQL | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ❌ No |
974
+ | **Project Status (Oct 2026)** |
975
+ | Active development | ✅ Yes | ✅ Yes | ✅ Yes | ⚠️ Low | ✅ Yes | ❌ Inactive | ⚠️ Low | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
976
+
977
+ **Legend:** ✅ Full support | ⚠️ Limited/Partial | ❌ Not supported
978
+
979
+ ### Tool Descriptions
980
+
981
+ - **[Pandera](https://pandera.readthedocs.io/)** (v0.34.1): Excellent runtime validation. Static analysis support exists
982
+ but has limitations—column access via `df["column"]` is not validated, and schema mismatches between functions may not
983
+ be caught.
984
+
985
+ - **[strictly_typed_pandas](https://strictly-typed-pandas.readthedocs.io/)** (v0.3.7): Provides `DataSet[Schema]` type
986
+ hints for runtime validation via typeguard. Despite documentation implying mypy support, there is no mypy plugin —
987
+ column access errors are not caught statically. No standalone checker. No polars support. Pinned to
988
+ `pandas<=2.2.3` and `pandas-stubs<=2.2.3.250308`, so it can't be used with pandas 3.
989
+
990
+ - **[pandas-stubs](https://github.com/pandas-dev/pandas-stubs)** (v3.0.5.260914): Official pandas type stubs. Provides
991
+ API-level types but no column-level checking.
992
+
993
+ - **[dataenforce](https://github.com/CedricFR/dataenforce)** (v0.1.2, the only release ever published): Runtime
994
+ validation via decorator. Appears inactive/abandoned. Broken on every currently-supported Python version
995
+ (3.11 through 3.15) due to removal of internal typing APIs (`typing._TypingEmpty`) it depends on — confirmed
996
+ working only as far back as Python 3.9.
997
+
998
+ - **[pandas-type-checks](https://pypi.org/project/pandas-type-checks/)** (v1.1.3): Runtime validation decorator. No
999
+ static analysis.
1000
+
1001
+ - **[StaticFrame](https://github.com/static-frame/static-frame)** (v5.1.1): Alternative immutable DataFrame library.
1002
+ Not compatible with pandas/polars — requires a full rewrite to StaticFrame's own API. Column access is still
1003
+ string-based; mypy does not catch column name typos. Type safety comes from immutability guarantees, not schema checking.
1004
+
1005
+ - **[narwhals](https://narwhals-dev.github.io/narwhals/)** (v2.26.0): Compatibility layer that provides a unified API
1006
+ across pandas, polars, DuckDB, cuDF, and more. Solves a different problem—write-once-run-anywhere portability, not
1007
+ type safety. See [Why Abstraction Layers Don't Solve Type Safety](#why-abstraction-layers-dont-solve-type-safety)
1008
+ below.
1009
+
1010
+ - **[Great Expectations](https://greatexpectations.io/)** (v1.23.2): Comprehensive data quality framework. Defines
1011
+ "expectations" (assertions) about data values, distributions, and schema properties. Excellent for runtime
1012
+ validation, data documentation, and data quality monitoring. No static analysis or column-level type checking in
1013
+ code. Supports pandas, Spark, and SQL backends.
1014
+
1015
+ - **[dataframely](https://github.com/Quantco/dataframely)** (v3.1.2): Polars-only runtime validation library from Quantco.
1016
+ Schemas are defined as classes inheriting `dy.Schema` with typed descriptor fields (`dy.String()`, `dy.Float64()`)
1017
+ and `@dy.rule()` decorators for cross-column and group-level constraints. Returns `dy.DataFrame[Schema]` generic
1018
+ types that give call-site narrowing to type checkers, but does not validate column subscript access inside function
1019
+ bodies, and (as of 3.0) that narrowing doesn't even survive a `.filter()` call — it returns a plain `pl.DataFrame`.
1020
+ 3.0 also removed the `dy.Series` type entirely; column access now returns a plain `pl.Series`. No lint-time or
1021
+ static analysis capability. Supports nullability, string constraints, numeric bounds, cross-column rules, soft
1022
+ validation, test data generation, and SQLAlchemy/PyArrow export.
1023
+
1024
+ - **[patito](https://github.com/JakobGM/patito)** (v0.8.6): Runtime validation library using a Pydantic-style `patito.Model`
1025
+ class. Polars is the primary backend; pandas is supported but works by converting to Polars via PyArrow (an
1026
+ undeclared dependency). No static analysis or standalone checker.
1027
+
1028
+ ### Type Checkers (Not DataFrame-Specific)
1029
+
1030
+ These are general Python type checkers. They don't validate DataFrame column names, but they can be used alongside
1031
+ typedframes for comprehensive type checking:
1032
+
1033
+ - **[mypy](https://mypy-lang.org/)** (v2.4.0): The original Python type checker. typedframes provides a mypy plugin for
1034
+ column checking. See [performance benchmarks](#static-analysis-performance).
1035
+
1036
+ - **[ty](https://github.com/astral-sh/ty)** (v0.0.84, Astral): New Rust-based type checker, faster than mypy on
1037
+ large codebases. Does not support mypy plugins—use typedframes standalone checker.
1038
+
1039
+ - **[pyrefly](https://pyrefly.org/)** (v1.3.2, Meta): Rust-based type checker from Meta, replacement for Pyre. Fast,
1040
+ but no DataFrame column checking.
1041
+
1042
+ - **[pyright](https://github.com/microsoft/pyright)** (v1.1.414, Microsoft): Type checker powering Pylance/VSCode. No
1043
+ mypy plugin support—use typedframes standalone checker.
1044
+
1045
+ ### Not Directly Comparable
1046
+
1047
+ These tools serve different purposes:
1048
+
1049
+ - **[pandas_lint](https://github.com/Jean-EstevezT/pandas_lint)**: Lints pandas code patterns (performance, best
1050
+ practices). Does not check column names/types.
1051
+ - **[pandas-vet](https://github.com/deppen8/pandas-vet)**: Flake8 plugin for pandas best practices. Does not check
1052
+ column names/types.
1053
+
1054
+ ### When to Use What
1055
+
1056
+ | Use Case | Recommended Tool |
1057
+ |------------------------------------------------------|-------------------------------------|
1058
+ | Static column checking (existing pandas/polars) | **typedframes** |
1059
+ | Runtime data validation | Pandera |
1060
+ | Both static + runtime | typedframes + `to_pandera_schema()` |
1061
+ | Cross-library portability (write once, run anywhere) | narwhals |
1062
+ | Data quality monitoring / pipeline validation | Great Expectations |
1063
+ | Immutable DataFrames from scratch | StaticFrame |
1064
+ | Pandas API type hints only | pandas-stubs |
1065
+
1066
+ ---
1067
+
1068
+ ## Pandera Integration
1069
+
1070
+ Convert typedframes schemas to [Pandera](https://pandera.readthedocs.io/) schemas for runtime validation. Define your
1071
+ schema once, get both static and runtime checking.
1072
+
1073
+ ```shell
1074
+ pip install typedframes[pandera]
1075
+ ```
1076
+
1077
+ ```python
1078
+ from typedframes import BaseSchema, Column
1079
+ from typedframes.pandera import to_pandera_schema
1080
+ import pandas as pd
1081
+
1082
+
1083
+ class UserData(BaseSchema):
1084
+ user_id = Column(type=int)
1085
+ email = Column(type=str)
1086
+ age = Column(type=int, nullable=True)
1087
+
1088
+
1089
+ # Convert to pandera schema
1090
+ pandera_schema = to_pandera_schema(UserData)
1091
+
1092
+ # Validate data at runtime
1093
+ df = pd.read_csv("users.csv")
1094
+ validated_df = pandera_schema.validate(df) # Raises SchemaError on failure
1095
+ ```
1096
+
1097
+ The conversion maps:
1098
+
1099
+ - `Column` type/nullable/alias to `pa.Column` dtype/nullable/name
1100
+ - `ColumnSet` with explicit members to individual `pa.Column` entries
1101
+ - `ColumnSet` with regex to `pa.Column(regex=True)`
1102
+ - `allow_extra_columns` to pandera's `strict` mode
1103
+
1104
+ ---
1105
+
1106
+ ## Examples
1107
+
1108
+ Runnable versions of everything shown in [Quick Start](#quick-start) and
1109
+ [Advanced Usage](#advanced-usage) live under [`examples/features/`](examples/features/):
1110
+
1111
+ - [`annotated_pandas_example.py`](examples/features/annotated_pandas_example.py) —
1112
+ `Annotated[pd.DataFrame, Schema]` with string and `.s` descriptor column access
1113
+ - [`annotated_polars_example.py`](examples/features/annotated_polars_example.py) —
1114
+ the same, with `pl.col()` and `.col` descriptor expressions
1115
+ - [`typedframes_example.py`](examples/features/typedframes_example.py) — pandas and
1116
+ polars side by side against one shared schema
1117
+ - [`schema_algebra_example.py`](examples/features/schema_algebra_example.py) —
1118
+ composing schemas via inheritance and the `+` operator
1119
+ - [`inference_example.py`](examples/features/inference_example.py) — all four
1120
+ column-inference modes with annotated ✓/✗ comments, no `BaseSchema` at all
1121
+ - [`multi_file_inference/`](examples/features/multi_file_inference/) and
1122
+ [`multi_file_with_schema/`](examples/features/multi_file_with_schema/) — the same
1123
+ cross-file pipeline checked with and without schemas, see
1124
+ [`examples/features/README.md`](examples/features/README.md) for what each one catches
1125
+ - [`ipynb_example.ipynb`](examples/features/ipynb_example.ipynb) — the same checks in a
1126
+ Jupyter notebook, with `cell N:line:col` locations (see [Jupyter
1127
+ Notebooks](#jupyter-notebooks))
1128
+
1129
+ ---
1130
+
1131
+ ## Philosophy
1132
+
1133
+ ### Type Safety Over Validation
1134
+
1135
+ We believe static analysis catches bugs earlier and cheaper than runtime validation.
1136
+
1137
+ **typedframes focuses on:**
1138
+ - ✅ Catching errors at lint-time
1139
+ - ✅ Zero runtime overhead
1140
+ - ✅ Developer experience
1141
+
1142
+ **We explicitly don't focus on:**
1143
+ - ❌ Runtime data validation (use Pandera)
1144
+ - ❌ Statistical checks (use Pandera)
1145
+ - ❌ Data quality monitoring (use Great Expectations)
1146
+
1147
+ **Important:** An `Annotated[pd.DataFrame, Schema]` type annotation is a *trust assertion*, not a validation step.
1148
+ It tells the type checker "this DataFrame conforms to this schema" without verifying the actual data. The linter catches
1149
+ mistakes in your code (wrong column names, schema mismatches between functions), but it cannot verify that a CSV file
1150
+ contains the expected columns. For runtime validation of external data, use
1151
+ [`to_pandera_schema()`](#pandera-integration) to convert your typedframes schemas to Pandera schemas.
1152
+
1153
+ ### Native Backend Types
1154
+
1155
+ We use native `Annotated[pd.DataFrame, Schema]` and `Annotated[pl.DataFrame, Schema]` types because pandas and
1156
+ polars have fundamentally different APIs. By annotating native objects rather than wrapping them in custom classes,
1157
+ typedframes lets you use each library's full, native API while still getting schema-level type safety.
1158
+
1159
+ **Trade-offs we avoid:**
1160
+ - ❌ Custom wrapper classes (you lose IDE completion for native methods)
1161
+ - ❌ "Universal DataFrame" abstractions (you lose library-specific features)
1162
+ - ❌ Lowest-common-denominator APIs
1163
+
1164
+ ### Why Abstraction Layers Don't Solve Type Safety
1165
+
1166
+ Tools like [narwhals](https://narwhals-dev.github.io/narwhals/) solve a different problem: writing portable code that runs on pandas, polars, DuckDB, cuDF, and other backends. This is useful for library authors who want to support multiple backends without maintaining separate codebases.
1167
+
1168
+ However, abstraction layers don't provide column-level type safety:
1169
+
1170
+ ```python
1171
+ import narwhals as nw
1172
+
1173
+
1174
+ def process(df: nw.DataFrame) -> nw.DataFrame:
1175
+ # No static checking - "revenue" typo won't be caught until runtime
1176
+ return df.filter(nw.col("revnue") > 100) # Typo: "revnue" vs "revenue"
1177
+ ```
1178
+
1179
+ **The fundamental issue:** Abstraction layers abstract over *which library* you're using, not *what columns* your data has. They can't know at lint-time whether "revenue" is a valid column in your DataFrame.
1180
+
1181
+ typedframes solves the orthogonal problem of schema safety:
1182
+
1183
+ ```python
1184
+ from typing import Annotated
1185
+ import polars as pl
1186
+ from typedframes import BaseSchema, Column
1187
+
1188
+
1189
+ class SalesData(BaseSchema):
1190
+ revenue = Column(type=float)
1191
+
1192
+
1193
+ def process(df: Annotated[pl.DataFrame, SalesData]) -> pl.DataFrame:
1194
+ return df.filter(pl.col("revnue") > 100) # ✗ Error at lint-time: 'revnue' not in SalesData
1195
+ ```
1196
+
1197
+ **Use narwhals when:** You're writing a library that needs to work with multiple DataFrame backends.
1198
+
1199
+ **Use typedframes when:** You want to catch column name/type errors before your code runs.
1200
+
1201
+ ### Why No Built-in Validation?
1202
+
1203
+ Ideally, validation happens at the point of data ingestion rather than in Python application code. If you're validating
1204
+ DataFrames in Python, consider whether your data pipeline could enforce constraints earlier. Use Pandera for cases where
1205
+ runtime validation is genuinely necessary.
1206
+
1207
+ ---
1208
+
1209
+ ## License
1210
+
1211
+ MIT License - see [LICENSE](LICENSE)
1212
+
1213
+ ---
1214
+
1215
+ ## FAQ
1216
+
1217
+ **Q: Do I need to choose between pandas and polars?**
1218
+ A: No. Define your schema once, use it with both. Just use `Annotated[pd.DataFrame, Schema]` or `Annotated[pl.DataFrame, Schema]` in your function signatures.
1219
+
1220
+ **Q: Does this replace Pandera?**
1221
+ A: No, it complements it. Use typedframes for static analysis, and `to_pandera_schema()` to convert your schemas to
1222
+ Pandera for runtime validation. See [Pandera Integration](#pandera-integration).
1223
+
1224
+ **Q: Is the standalone checker required?**
1225
+ A: No. You can use just the mypy plugin, just the standalone checker, or both. They catch the same errors.
1226
+
1227
+ **Q: What works without any plugin?**
1228
+ A: Any type checker (mypy, pyright, ty) understands `Annotated[pd.DataFrame, Schema]` as a plain `pd.DataFrame` —
1229
+ no plugin or stubs needed for basic type checking. Column *name* validation (catching typos like `df["revnue"]` in
1230
+ string-based access) still requires the standalone checker or mypy plugin.
1231
+
1232
+ **Q: What about pyright/pylance users?**
1233
+ A: The mypy plugin doesn't work with pyright. Use the standalone checker (`typedframes check`) for column name
1234
+ validation. Schema descriptor access (`df[Schema.column]`) works natively in pyright without any plugin.
1235
+
1236
+ **Q: Do I need to write `BaseSchema` classes to get value?**
1237
+ A: No. The standalone checker works entirely from inference: `usecols=`/`columns=`/`dtype=` arguments on read
1238
+ calls give it enough information to validate column access and propagate that knowledge through method chains
1239
+ (`rename`, `drop`, `assign`, `select`, …). `BaseSchema` is a progressive enhancement that unlocks cross-file
1240
+ awareness (schemas travel with function return types across module boundaries) and IDE autocomplete via
1241
+ descriptors — but the checker catches real column errors from day one without it. See
1242
+ [`examples/features/multi_file_inference/`](examples/features/multi_file_inference/) for a complete demo with no schema classes.
1243
+
1244
+ **Q: Does this work with existing pandas/polars code?**
1245
+ A: Yes. You can gradually adopt typedframes by adding schemas to new code. Existing code continues to work.
1246
+ Start by adding `usecols=` to your read calls to get immediate column validation, then add `BaseSchema`
1247
+ classes incrementally where cross-file tracking or autocomplete is most valuable.
1248
+
1249
+ **Q: What if my column name conflicts with a pandas/polars method?**
1250
+ A: No problem. Since column access uses bracket syntax with schema descriptors (`df[Schema.mean]`), there is no conflict
1251
+ with DataFrame methods (`df.mean()`). Both work independently.
1252
+
1253
+ ---
1254
+
1255
+ ## Credits
1256
+
1257
+ Built by developers who believe DataFrame bugs should be caught at lint-time, not in production.
1258
+
1259
+ Inspired by the needs of ML/data science teams working with complex data pipelines.
1260
+
1261
+ ---
1262
+
1263
+ **Questions? Issues? Ideas?** [Open an issue](https://github.com/w-martin/typedframes/issues)
1264
+
1265
+ **Ready to catch DataFrame bugs before runtime?** `pip install typedframes`
1266
+