typedframes 0.4.1__tar.gz → 0.6.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (37) hide show
  1. {typedframes-0.4.1 → typedframes-0.6.0}/PKG-INFO +198 -244
  2. {typedframes-0.4.1 → typedframes-0.6.0}/README.md +197 -243
  3. {typedframes-0.4.1 → typedframes-0.6.0}/pyproject.toml +24 -7
  4. {typedframes-0.4.1 → typedframes-0.6.0}/rust/Cargo.lock +1 -1
  5. {typedframes-0.4.1 → typedframes-0.6.0}/rust/Cargo.toml +1 -1
  6. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/ast_extract.rs +69 -0
  7. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/config.rs +55 -5
  8. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/constants.rs +11 -0
  9. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/errors.rs +11 -1
  10. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/index.rs +669 -23
  11. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/lib.rs +4 -0
  12. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/linter.rs +1365 -177
  13. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/notebook.rs +4 -4
  14. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/pyapi.rs +25 -1
  15. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/__init__.py +1 -1
  16. typedframes-0.6.0/src/typedframes/_rust_checker.pyi +16 -0
  17. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/cli.py +216 -131
  18. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/mypy.py +2 -2
  19. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/pandas.py +7 -7
  20. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/polars.py +4 -4
  21. {typedframes-0.4.1 → typedframes-0.6.0}/rust/.cargo/config.toml +0 -0
  22. {typedframes-0.4.1 → typedframes-0.6.0}/rust/README.md +0 -0
  23. {typedframes-0.4.1 → typedframes-0.6.0}/rust/benches/parser_bench.rs +0 -0
  24. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/contract.rs +0 -0
  25. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/main.rs +0 -0
  26. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/sql.rs +0 -0
  27. {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/typo.rs +0 -0
  28. {typedframes-0.4.1 → typedframes-0.6.0}/rust/tests/integration_test.rs +0 -0
  29. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/base_schema.py +0 -0
  30. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/column.py +0 -0
  31. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/column_group.py +0 -0
  32. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/column_group_error.py +0 -0
  33. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/column_set.py +0 -0
  34. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/missing_dependency_error.py +0 -0
  35. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/pandera.py +0 -0
  36. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/py.typed +0 -0
  37. {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/schema_algebra.py +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: typedframes
3
- Version: 0.4.1
3
+ Version: 0.6.0
4
4
  Classifier: Development Status :: 3 - Alpha
5
5
  Classifier: Intended Audience :: Developers
6
6
  Classifier: License :: OSI Approved :: MIT License
@@ -42,7 +42,7 @@ Project-URL: Repository, https://github.com/w-martin/typedframes
42
42
 
43
43
  > ⚠️ **Project Status: Proof of Concept**
44
44
  >
45
- > `typedframes` (v0.4.1) is currently an experimental proof-of-concept. The core static analysis and mypy/Rust
45
+ > `typedframes` (v0.5.0) is currently an experimental proof-of-concept. The core static analysis and mypy/Rust
46
46
  > integrations work, but expect rough edges. The codebase prioritizes demonstrating the viability of static DataFrame
47
47
  > column checking over production-grade stability.
48
48
  >
@@ -59,14 +59,17 @@ print(orders["revenue"]) # ✗ unknown-column — 'revenue' not in inferred col
59
59
  ```
60
60
 
61
61
  ```shell
62
- typedframes check src/ --fail-under=90
62
+ typedframes check src/ --coverage-fail-under=90
63
63
  # src/pipeline.py:7:8: error[unknown-column] Column 'revenue' does not exist in inferred column set (defined at line 6)
64
64
  # ✗ Found 1 error in 12 files (0.0s)
65
65
  # ✗ DataFrame schema coverage 82.0% is below the required 90.0% (9/11 DataFrames had column info)
66
66
  ```
67
67
 
68
- `--fail-under=N` is the same idea as a test-coverage or mypy type-coverage gate, applied to how many DataFrames
69
- the checker can resolve columns for — see [DataFrame Schema Coverage Thresholds](#dataframe-schema-coverage-thresholds-opt-in).
68
+ `--coverage-fail-under=N` is the same idea as a test-coverage or mypy type-coverage gate, applied to how many DataFrames
69
+ the checker can resolve to a concrete column list — a DataFrame it merely recognizes but can't resolve columns for
70
+ (a Feast retrieval whose entity_df isn't itself concrete, a bare `-> pd.DataFrame` return with no attached Schema)
71
+ does not count, and any column accessed on it is flagged `unverifiable-column` rather than passed through silently.
72
+ See [DataFrame Schema Coverage Thresholds](#dataframe-schema-coverage-thresholds-opt-in).
70
73
  Add `BaseSchema` classes later for cross-file awareness and IDE autocomplete — see [Quick Start](#quick-start).
71
74
 
72
75
  ---
@@ -78,6 +81,7 @@ Add `BaseSchema` classes later for cross-file awareness and IDE autocomplete —
78
81
  - [Quick Start](#quick-start)
79
82
  - [Column Inference](#column-inference)
80
83
  - [Static Analysis](#static-analysis)
84
+ - [Jupyter Notebooks](#jupyter-notebooks)
81
85
  - [DataFrame Schema Coverage Thresholds](#dataframe-schema-coverage-thresholds-opt-in)
82
86
  - [Static Analysis Performance](#static-analysis-performance)
83
87
  - [Type Safety With Multiple Backends](#type-safety-with-multiple-backends)
@@ -104,7 +108,7 @@ tracking and IDE autocomplete; they're a progressive enhancement, not a prerequi
104
108
  **What you get:**
105
109
 
106
110
  - ✅ **Works without schema annotations** - Column inference from `usecols=`, `dtype=`, and method chains catches errors on unannotated code
107
- - ✅ **CI gate via coverage threshold** - `--fail-under=N` fails the build when too much of your codebase is invisible to the checker, the same way you'd gate on test coverage or a type checker's type-coverage number
111
+ - ✅ **CI gate via coverage threshold** - `--coverage-fail-under=N` fails the build when too much of your codebase is invisible to the checker, the same way you'd gate on test coverage or a type checker's type-coverage number
108
112
  - ✅ **Rust-fast** - Milliseconds, not seconds, even on hundreds of files; fast enough for pre-commit hooks and CI (see [benchmarks](#static-analysis-performance))
109
113
  - ✅ **Cross-file awareness** - Add `BaseSchema` and typed return annotations to follow schemas across module boundaries
110
114
  - ✅ **Refactor-safe access** - `df[Schema.column_group.s].mean()` (pandas) or `df.select(Schema.col.col)` (polars) instead of scattered string literals
@@ -125,7 +129,13 @@ or
125
129
  uv add typedframes
126
130
  ```
127
131
 
128
- The Rust-based checker is included — no separate install needed.
132
+ The Rust-based checker is included — no separate install needed. `typedframes` itself has
133
+ no required dependencies; add the extra for whichever backend(s) your code uses:
134
+
135
+ ```shell
136
+ pip install typedframes[pandas] # includes pandas
137
+ pip install typedframes[polars] # includes polars
138
+ ```
129
139
 
130
140
  ---
131
141
 
@@ -422,17 +432,20 @@ typedframes check src/ --strict
422
432
  # JSON output
423
433
  typedframes check src/ --output-format=json
424
434
 
425
- # Skip cross-file index (single-file mode, faster for quick checks)
435
+ # Check a single file (follows that file's own imports, not the whole project)
436
+ typedframes check src/pipeline.py
437
+
438
+ # Skip the cross-file index entirely (every file in isolation, fastest)
426
439
  typedframes check src/ --no-index
427
440
 
428
441
  # Suppress all warnings (untracked-dataframe, dropped-unknown-column)
429
442
  typedframes check src/ --no-warnings
430
443
 
431
444
  # Enforce minimum DataFrame schema coverage (see below)
432
- typedframes check src/ --fail-under=90
445
+ typedframes check src/ --coverage-fail-under=90
433
446
 
434
447
  # Show which DataFrames lack column info, per file
435
- typedframes check src/ --coverage-report=term-missing
448
+ typedframes check src/ --coverage-detail=term-missing
436
449
  ```
437
450
 
438
451
  To suppress warnings project-wide, add to `pyproject.toml`:
@@ -480,111 +493,174 @@ See the full [Method Matrix](https://typedframes.readthedocs.io/en/latest/method
480
493
  for the complete list of tracked, passthrough, and untracked operations, plus the error
481
494
  code reference.
482
495
 
483
- ---
484
-
485
- ## DataFrame Schema Coverage Thresholds (Opt-In)
486
-
487
- **DataFrame schema coverage** is the fraction of DataFrames `typedframes check` could
488
- resolve column information for — the analogue of the "type coverage" reported by mypy,
489
- pyright, and pyre, and unrelated to test coverage. That number is informational by
490
- default. If you want it enforced — failing the run when too much of your code is
491
- invisible to the checker — enable a threshold.
496
+ ### Pre-commit Hook
497
+
498
+ ```yaml
499
+ # .pre-commit-config.yaml
500
+ repos:
501
+ - repo: local
502
+ hooks:
503
+ - id: typedframes
504
+ name: typedframes check
505
+ entry: typedframes check . --strict
506
+ language: python
507
+ additional_dependencies: ["typedframes==0.5.0"]
508
+ types_or: [python, jupyter]
509
+ pass_filenames: false
510
+ ```
492
511
 
493
- This is **entirely opt-in**. With no `[tool.typedframes.coverage]` table and no
494
- `--fail-under`, nothing changes: no threshold, no exit-code difference.
512
+ `pass_filenames: false` is deliberate: the checker takes a single path and builds its
513
+ cross-file index from it, so one run per commit still sees the modules the commit didn't
514
+ touch — a per-file invocation would lose that. `--strict` is what makes the hook block
515
+ the commit: `typedframes check` exits 0 by default even when it finds real errors. Add
516
+ the coverage gate to the same line:
495
517
 
496
- Every supported key, at its default value:
518
+ ```yaml
519
+ # with coverage gating
520
+ entry: typedframes check . --strict --coverage-fail-under=90
521
+ ```
497
522
 
498
- ```toml
499
- [tool.typedframes.coverage]
500
- # Master switch. Coverage enforcement is off unless this is true, so adding this
501
- # table without setting it changes nothing.
502
- enabled = false
503
-
504
- # Minimum percentage of DataFrames that must have recognized column/schema info
505
- # before `typedframes check` exits 1. Only consulted when `enabled = true`.
506
- # Applies to every file not captured by a glob in [overrides] below.
507
- fail_under = 100.0
508
-
509
- # How much coverage detail to print after each check. One of:
510
- # "summary" one line (the default, unchanged from before this feature)
511
- # "term-missing" per-file table plus the DataFrame sites lacking column info
512
- # "json" machine-readable document, for CI tooling
513
- # Independent of `enabled` — a detailed report is useful without a gate, and
514
- # vice versa. Overridden by `--coverage-report`.
515
- report = "summary"
516
-
517
- [tool.typedframes.coverage.overrides]
518
- # Per-path glob overrides of `fail_under`, for holding legacy code to a lower bar
519
- # than new code. Each glob is graded on its own files as a separate group, so a
520
- # lenient legacy bucket can't drag down (or rescue) the rest of the project.
521
- # Paths are matched project-relative: `**` spans any number of directories,
522
- # `*` and `?` stay within one path segment.
523
- # When several globs match one file the most specific wins — longest literal
524
- # prefix before the first `*` or `?`. Files matching no glob use `fail_under`.
525
- # "legacy/**" = 50.0
526
- # "src/new_module/**" = 100.0
523
+ This repository also ships a `.pre-commit-hooks.yaml`, so once a tagged release contains
524
+ it you can point `repo:` straight at `https://github.com/w-martin/typedframes` with a
525
+ `rev:` and drop the boilerplate above. That form needs a Rust toolchain wherever the hook
526
+ runs — pre-commit's python installer builds the repository from source rather than
527
+ fetching a wheel — so the local hook stays the quicker one to install either way.
528
+
529
+ ### GitHub Actions
530
+
531
+ ```yaml
532
+ # .github/workflows/typedframes.yml
533
+ name: typedframes
534
+ on: [push, pull_request]
535
+
536
+ jobs:
537
+ typedframes:
538
+ runs-on: ubuntu-latest
539
+ steps:
540
+ - uses: actions/checkout@v4
541
+ # No tagged release contains action.yml yet. Repin this to that tag once one
542
+ # ships: an unpinned third-party action is a supply-chain risk, and @main is a
543
+ # stopgap rather than a recommendation.
544
+ - uses: w-martin/typedframes@main
545
+ with:
546
+ path: src/
547
+ coverage-fail-under: "90"
527
548
  ```
528
549
 
529
- Prefer to keep `pyproject.toml` clean? The same settings work in a standalone
530
- `typedframes.toml` at the project root, with the `[tool.typedframes]` prefix dropped
531
- (the way `ruff.toml` drops `[tool.ruff]`):
550
+ The action installs the PyPI wheel into a throwaway virtualenv and runs the checker with
551
+ `--output-format=github`, so errors arrive as annotations on the pull request diff.
532
552
 
533
- ```toml
534
- # typedframes.toml
535
- [coverage]
536
- enabled = false
537
- fail_under = 100.0
538
- report = "summary"
539
-
540
- [coverage.overrides]
541
- # "legacy/**" = 50.0
542
- ```
553
+ | Input | Default | |
554
+ |-------|---------|-|
555
+ | `path` | `.` | File or directory to check |
556
+ | `version` | `latest` | PyPI version to install, e.g. `"0.5.0"` |
557
+ | `strict` | `true` | Fail the step on errors — `typedframes check` exits 0 without it |
558
+ | `coverage-fail-under` | *(unset)* | Minimum DataFrame schema coverage, e.g. `"90"` |
559
+ | `coverage-detail` | `summary` | Or `term-missing` for the per-file breakdown |
560
+ | `no-warnings` | `false` | Suppress warning-level diagnostics |
561
+ | `args` | *(empty)* | Escape hatch for `--no-index`, `--lenient-ingest`, `--no-info` |
543
562
 
544
- If both files exist, `typedframes.toml` wins **entirely** — the two are never merged,
545
- so exactly one file explains the whole configuration.
563
+ ---
546
564
 
547
- ### Seeing What's Missing
565
+ ## Jupyter Notebooks
548
566
 
549
- The default one-line DataFrame schema coverage summary tells you the ratio but not what
550
- to fix. `--coverage-report=term-missing` names the DataFrames that cost you coverage:
567
+ The standalone checker reads `.ipynb` files directly — no `nbconvert`, no `jupytext`, no
568
+ flag, no conversion step. Notebooks are parsed with
569
+ [`ruff_notebook`](https://github.com/astral-sh/ruff/tree/main/crates/ruff_notebook), the
570
+ same crate Ruff uses for its own notebook support:
551
571
 
552
572
  ```shell
553
- typedframes check src/ --coverage-report=term-missing
573
+ # Check one notebook
574
+ typedframes check analysis.ipynb
575
+
576
+ # Check a directory — .py and .ipynb files are both picked up, no flag needed
577
+ typedframes check notebooks/
578
+
579
+ # Output — `cell N` is the notebook's Nth cell; line:col are relative to that cell:
580
+ # analysis.ipynb:cell 6:2:1: warning[untracked-dataframe] columns unknown at lint time; specify
581
+ # `usecols`/`columns`, or annotate the variable's type
582
+ # analysis.ipynb:cell 10:1:1: error[unknown-column] Column 'revenue' does not exist in Orders
583
+ # {amount, customer_id, order_id} (defined at analysis.ipynb cell 2:8)
584
+ # ✗ Found 1 error, 2 warnings in 1 file (0.0s)
554
585
  ```
555
586
 
556
- ```
557
- Name Typed Total Cover Missing
558
- ---------------------------------------------
559
- legacy/old.py 0 2 0% old_one:2, old_two:3
560
- src/new.py 1 2 50% bad:3
561
- ---------------------------------------------
562
- TOTAL 1 4 25%
563
- ```
587
+ **Features:**
588
+ - ✅ Auto-detected — every `.ipynb` under a checked path, alongside `.py` files
589
+ - ✅ Diagnostics reported as `cell N:line:col`, mapped back to the cell that produced them
590
+ rather than a line number in the raw notebook JSON — including a same-notebook schema's
591
+ `(defined at ...)` cross-reference
592
+ - ✅ Same error codes, severities, and flags as `.py` files: `--strict`,
593
+ `--output-format`, `--no-warnings`, `--lenient-ingest`, coverage thresholds
594
+ - ✅ IPython magics and shell escapes parse natively (`%matplotlib inline`, `%%time`,
595
+ `!pip install ...`) — nothing has to be stripped first
596
+ - ✅ `# typedframes: ignore` suppresses a diagnostic on a line inside a cell, same as in a
597
+ `.py` file
598
+ - ✅ A schema imported from a project `.py` module resolves through the cross-file index,
599
+ so `orders = load_orders()` in a notebook is validated against the schema on
600
+ `load_orders`' return annotation
601
+ - ✅ `.ipynb_checkpoints/` is pruned by default, so stale checkpoint copies aren't
602
+ re-reported
603
+
604
+ **Caveats:**
605
+ - ❌ The mypy plugin ([Option 2](#option-2-mypy-plugin-comprehensive)) is `.py`-only —
606
+ mypy reads Python source files, so notebook checking is standalone-checker-only
607
+ - ❌ `cell N` counts *every* cell in the notebook, markdown and raw cells included — so
608
+ the number is a cell's position in the notebook, not "the Nth code cell"
609
+ - ❌ A notebook is never a *source* of schemas for other files (Python can't import an
610
+ `.ipynb`); a schema defined in a notebook is visible only inside that notebook
611
+ - ❌ A notebook that isn't valid notebook JSON, whose kernel isn't Python, or whose code
612
+ doesn't parse is skipped with a message on stderr and the run continues — a skip alone
613
+ never fails the run, `--strict` included
614
+ - ❌ Under `--output-format=github`, the annotation's `line`/`col` stay cell-relative and
615
+ the cell number moves into the annotation title, since GitHub annotations address raw
616
+ file lines and an `.ipynb` is JSON rather than source text
617
+
618
+ ### See Also
564
619
 
565
- Each `Missing` entry is `variable:line` — the assignment where the checker recognized a
566
- DataFrame but couldn't resolve its columns. Fix those (add `usecols=`, name the columns
567
- in the `SELECT`, or annotate the variable) and coverage rises.
620
+ - [`examples/features/ipynb_example.ipynb`](examples/features/ipynb_example.ipynb) — a
621
+ runnable notebook with one schema, an IPython magic, and three intentional bugs
622
+ (`untracked-dataframe`, `dropped-unknown-column`, `unknown-column`), all caught by
623
+ `typedframes check ipynb_example.ipynb` without running a cell.
568
624
 
569
- For CI tooling, `--coverage-report=json` emits the same data as a document. Combine it
570
- with `--output-format=json` and the coverage report is nested under a `coverage` key so
571
- stdout stays a single valid JSON document:
625
+ ---
572
626
 
573
- ```shell
574
- typedframes check src/ --output-format=json --coverage-report=json
627
+ ## DataFrame Schema Coverage Thresholds (Opt-In)
628
+
629
+ **DataFrame schema coverage** is the fraction of DataFrames `typedframes check` could
630
+ resolve column information for — the analogue of the "type coverage" reported by mypy,
631
+ pyright, and pyre, and unrelated to test coverage. That number is informational by
632
+ default. If you want it enforced — failing the run when too much of your code is
633
+ invisible to the checker — enable a threshold:
634
+
635
+ ```toml
636
+ [tool.typedframes.coverage]
637
+ enabled = true
638
+ fail_under = 90.0
575
639
  ```
576
640
 
577
- Notes:
641
+ or as a one-off, without touching config:
642
+
643
+ ```shell
644
+ typedframes check src/ --coverage-fail-under=90
645
+ ```
578
646
 
579
- - Coverage is a **separate gate from `--strict`**. `--strict` fails on errors
580
- (correctness); a threshold fails on missing column information (completeness).
581
- Enabling one never implies the other.
582
- - `--fail-under=N` is a **total override**: it applies one threshold everywhere and
583
- ignores the config table, per-path overrides included. Handy for a one-off CI run.
584
- - A group with no recognized DataFrames passes: 0/0 means there was nothing to
585
- measure, not that something failed.
586
- - A failed threshold exits **1** and is reported even under `--no-info` — that flag
587
- silences the informational summary line, not a gate result.
647
+ This is **entirely opt-in** — with no `[tool.typedframes.coverage]` table and no
648
+ `--coverage-fail-under`, nothing changes: no threshold, no exit-code difference. Per-path
649
+ overrides (e.g. a lower bar for `legacy/**`), a `--coverage-detail=term-missing`
650
+ breakdown of exactly which DataFrames cost you coverage (as text or, combined with
651
+ `--output-format=json`, as structured JSON), and the full config reference all live in
652
+ the
653
+ [DataFrame schema coverage thresholds guide](docs/usage.md#dataframe-schema-coverage-thresholds).
654
+
655
+ Some DataFrames are recognized but never resolve to a concrete column list — a Feast
656
+ retrieval whose `entity_df` isn't itself concretely typed, or a bare `-> pd.DataFrame`
657
+ return with no attached `Schema`. These do **not** count as covered, and accessing any
658
+ column on one is flagged `unverifiable-column` rather than passed through silently —
659
+ the checker genuinely doesn't know whether the column exists, so it says so instead of
660
+ guessing either way. A Feast retrieval whose `entity_df` argument IS a concretely typed
661
+ DataFrame is fully resolved instead: Feast's real output is `entity_df`'s own columns
662
+ plus the requested features, so both halves get checked normally. See
663
+ [Unresolved schemas](docs/usage.md#unresolved-schemas-recognized-but-not-column-checked).
588
664
 
589
665
  ## Static Analysis Performance
590
666
 
@@ -607,9 +683,11 @@ Fast feedback reduces development time. The typedframes Rust binary provides nea
607
683
 
608
684
  The typedframes binary resolves column names within a file and, when a project index is present, across files too.
609
685
  Run `typedframes check src/` to build the index automatically and catch errors like `df = load_users(); df["typo"]`
610
- even when `load_users` is defined in another module. Pass `--no-index` to skip the index and check each file in
611
- isolation. Full type checkers (mypy, pyright, ty) analyze all Python types across your entire codebase. Use both: the
612
- binary for fast iteration, mypy for comprehensive checking.
686
+ even when `load_users` is defined in another module. Checking a single file
687
+ (`typedframes check src/pipeline.py`) builds a smaller index from that file's own imports instead of walking the whole
688
+ project, so it still resolves `load_users` without paying for files it never references. Pass `--no-index` to skip the
689
+ index entirely and check each file in isolation. Full type checkers (mypy, pyright, ty) analyze all Python types across
690
+ your entire codebase. Use both: the binary for fast iteration, mypy for comprehensive checking.
613
691
 
614
692
  The standalone checker is built with [`ruff_python_parser`](https://github.com/astral-sh/ruff) for Python AST
615
693
  parsing.
@@ -824,6 +902,7 @@ Comprehensive comparison of pandas/DataFrame typing and validation tools. **type
824
902
  | Column name checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
825
903
  | Column type checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
826
904
  | Typo suggestions | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
905
+ | Coverage gate (`--coverage-fail-under`) | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
827
906
  | **Runtime Validation** |
828
907
  | Data validation | ❌ No | ✅ Excellent | ✅ Excellent | ✅ typeguard | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
829
908
  | Value constraints | ❌ No | ✅ Yes | ✅ Excellent | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
@@ -967,122 +1046,26 @@ The conversion maps:
967
1046
 
968
1047
  ## Examples
969
1048
 
970
- ### Basic CSV Processing
971
-
972
- ```python
973
- from typing import Annotated
974
- import pandas as pd
975
- from typedframes import BaseSchema, Column
976
-
977
-
978
- class Orders(BaseSchema):
979
- order_id = Column(type=int)
980
- customer_id = Column(type=int)
981
- total = Column(type=float)
982
- date = Column(type=str)
983
-
984
-
985
- def calculate_revenue(orders: Annotated[pd.DataFrame, Orders]) -> float:
986
- return orders["total"].sum()
987
-
988
-
989
- df: Annotated[pd.DataFrame, Orders] = pd.read_csv("orders.csv")
990
- revenue = calculate_revenue(df)
991
- ```
992
-
993
- ### Time Series Analysis
994
-
995
- ```python
996
- from typing import Annotated
997
- import pandas as pd
998
- from typedframes import BaseSchema, Column, ColumnSet, ColumnGroup
999
-
1000
-
1001
- class SensorData(BaseSchema):
1002
- timestamp = Column(type=str)
1003
- temperature = ColumnSet(type=float, members=["temp_1", "temp_2", "temp_3"])
1004
- humidity = ColumnSet(type=float, members=["humidity_1", "humidity_2"])
1005
-
1006
- all_sensors = ColumnGroup(members=[temperature, humidity])
1007
-
1008
-
1009
- df: Annotated[pd.DataFrame, SensorData] = pd.read_csv("sensors.csv")
1010
-
1011
- # Clean, type-safe operations using .s for column name lists
1012
- avg_temp_per_row = df[SensorData.temperature.s].mean(axis=1)
1013
- all_readings_stats = df[SensorData.all_sensors.s].describe()
1014
- ```
1015
-
1016
- ### Multi-Step Pipeline
1017
-
1018
- ```python
1019
- from typing import Annotated
1020
- import pandas as pd
1021
- from typedframes import BaseSchema, Column
1022
-
1023
-
1024
- class RawSales(BaseSchema):
1025
- date = Column(type=str)
1026
- product_id = Column(type=int)
1027
- quantity = Column(type=int)
1028
- price = Column(type=float)
1029
-
1030
-
1031
- class AggregatedSales(BaseSchema):
1032
- date = Column(type=str)
1033
- total_revenue = Column(type=float)
1034
- total_quantity = Column(type=int)
1035
-
1036
-
1037
- def aggregate_daily(df: Annotated[pd.DataFrame, RawSales]) -> Annotated[pd.DataFrame, AggregatedSales]:
1038
- result = (
1039
- df.groupby(RawSales.date.s)
1040
- .agg(
1041
- {
1042
- RawSales.price.s: "sum",
1043
- RawSales.quantity.s: "sum",
1044
- }
1045
- )
1046
- .reset_index()
1047
- )
1048
- result.columns = pd.Index(["date", "total_revenue", "total_quantity"])
1049
- return result # type: ignore[return-value]
1050
-
1051
-
1052
- # Type-safe pipeline
1053
- raw: Annotated[pd.DataFrame, RawSales] = pd.read_csv("sales.csv")
1054
- aggregated = aggregate_daily(raw)
1055
-
1056
-
1057
- # Type checker validates schema transformations
1058
- def analyze(df: Annotated[pd.DataFrame, AggregatedSales]) -> float:
1059
- df["total_revenue"] # ✓ OK
1060
- df["price"] # ✗ Error: 'price' not in AggregatedSales
1061
- return df[AggregatedSales.total_revenue.s].mean()
1062
- ```
1063
-
1064
- ### Polars Performance Pipeline
1065
-
1066
- ```python
1067
- from typing import Annotated
1068
- import polars as pl
1069
- from typedframes import BaseSchema, Column
1070
-
1071
-
1072
- class LargeDataset(BaseSchema):
1073
- id = Column(type=int)
1074
- value = Column(type=float)
1075
- category = Column(type=str)
1076
-
1077
-
1078
- def efficient_aggregation(df: Annotated[pl.DataFrame, LargeDataset]) -> pl.DataFrame:
1079
- return df.filter(pl.col("value") > 100).group_by("category").agg(pl.col("value").mean())
1080
-
1081
-
1082
- # Polars handles large files efficiently
1083
- df: Annotated[pl.DataFrame, LargeDataset] = pl.read_csv("huge_file.csv")
1084
- result = efficient_aggregation(df)
1085
- ```
1049
+ Runnable versions of everything shown in [Quick Start](#quick-start) and
1050
+ [Advanced Usage](#advanced-usage) live under [`examples/features/`](examples/features/):
1051
+
1052
+ - [`annotated_pandas_example.py`](examples/features/annotated_pandas_example.py) —
1053
+ `Annotated[pd.DataFrame, Schema]` with string and `.s` descriptor column access
1054
+ - [`annotated_polars_example.py`](examples/features/annotated_polars_example.py) —
1055
+ the same, with `pl.col()` and `.col` descriptor expressions
1056
+ - [`typedframes_example.py`](examples/features/typedframes_example.py) — pandas and
1057
+ polars side by side against one shared schema
1058
+ - [`schema_algebra_example.py`](examples/features/schema_algebra_example.py) —
1059
+ composing schemas via inheritance and the `+` operator
1060
+ - [`inference_example.py`](examples/features/inference_example.py) — all four
1061
+ column-inference modes with annotated ✓/✗ comments, no `BaseSchema` at all
1062
+ - [`multi_file_inference/`](examples/features/multi_file_inference/) and
1063
+ [`multi_file_with_schema/`](examples/features/multi_file_with_schema/) — the same
1064
+ cross-file pipeline checked with and without schemas, see
1065
+ [`examples/features/README.md`](examples/features/README.md) for what each one catches
1066
+ - [`ipynb_example.ipynb`](examples/features/ipynb_example.ipynb) — the same checks in a
1067
+ Jupyter notebook, with `cell N:line:col` locations (see [Jupyter
1068
+ Notebooks](#jupyter-notebooks))
1086
1069
 
1087
1070
  ---
1088
1071
 
@@ -1170,35 +1153,6 @@ MIT License - see [LICENSE](LICENSE)
1170
1153
 
1171
1154
  ---
1172
1155
 
1173
- ## Roadmap
1174
-
1175
- **Shipped:**
1176
- - [x] Schema definition API
1177
- - [x] Pandas support
1178
- - [x] Polars support
1179
- - [x] Mypy plugin
1180
- - [x] Standalone checker (Rust)
1181
- - [x] Explicit backend types
1182
- - [x] Merge/join schema preservation
1183
- - [x] Schema Composition (multiple inheritance, `SchemaA + SchemaB`)
1184
- - [x] Column name collision warnings
1185
- - [x] Pandera integration (`to_pandera_schema()`)
1186
- - [x] Cross-file schema inference (project-level index, `--no-index` flag)
1187
- - [x] Aggressive column inference (untracked-dataframe/dropped-unknown-column warnings, method chain propagation)
1188
- - [x] Function parameter contracts (`missing-column`), resolved transitively across chains of helper
1189
- functions and cross-file calls; schema-annotated parameters take priority over body-scanning
1190
- - [x] SQL / data-warehouse column inference (`SELECT` list parsing across Snowflake, BigQuery, Athena, Redshift,
1191
- Databricks, PySpark, DuckDB, SQLAlchemy Core/ORM, Feast, and T-SQL/Synapse/Fabric dialects)
1192
- - [x] Jupyter notebook (`.ipynb`) checking — code cells are checked directly, with errors reported as
1193
- `notebook.ipynb:cell N:line:col`
1194
-
1195
- **Planned:**
1196
-
1197
- - [ ] **Opt-in data loading constraints** - `Field` class with constraints (`gt`, `ge`, `lt`, `le`), strictly isolated
1198
- to `from_schema()` ingestion boundaries
1199
-
1200
- ---
1201
-
1202
1156
  ## FAQ
1203
1157
 
1204
1158
  **Q: Do I need to choose between pandas and polars?**