typedframes 0.4.1__tar.gz → 0.6.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {typedframes-0.4.1 → typedframes-0.6.0}/PKG-INFO +198 -244
- {typedframes-0.4.1 → typedframes-0.6.0}/README.md +197 -243
- {typedframes-0.4.1 → typedframes-0.6.0}/pyproject.toml +24 -7
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/Cargo.lock +1 -1
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/Cargo.toml +1 -1
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/ast_extract.rs +69 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/config.rs +55 -5
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/constants.rs +11 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/errors.rs +11 -1
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/index.rs +669 -23
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/lib.rs +4 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/linter.rs +1365 -177
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/notebook.rs +4 -4
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/pyapi.rs +25 -1
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/__init__.py +1 -1
- typedframes-0.6.0/src/typedframes/_rust_checker.pyi +16 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/cli.py +216 -131
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/mypy.py +2 -2
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/pandas.py +7 -7
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/polars.py +4 -4
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/.cargo/config.toml +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/README.md +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/benches/parser_bench.rs +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/contract.rs +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/main.rs +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/sql.rs +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/src/typo.rs +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/rust/tests/integration_test.rs +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/base_schema.py +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/column.py +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/column_group.py +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/column_group_error.py +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/column_set.py +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/missing_dependency_error.py +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/pandera.py +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/py.typed +0 -0
- {typedframes-0.4.1 → typedframes-0.6.0}/src/typedframes/schema_algebra.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: typedframes
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.6.0
|
|
4
4
|
Classifier: Development Status :: 3 - Alpha
|
|
5
5
|
Classifier: Intended Audience :: Developers
|
|
6
6
|
Classifier: License :: OSI Approved :: MIT License
|
|
@@ -42,7 +42,7 @@ Project-URL: Repository, https://github.com/w-martin/typedframes
|
|
|
42
42
|
|
|
43
43
|
> ⚠️ **Project Status: Proof of Concept**
|
|
44
44
|
>
|
|
45
|
-
> `typedframes` (v0.
|
|
45
|
+
> `typedframes` (v0.5.0) is currently an experimental proof-of-concept. The core static analysis and mypy/Rust
|
|
46
46
|
> integrations work, but expect rough edges. The codebase prioritizes demonstrating the viability of static DataFrame
|
|
47
47
|
> column checking over production-grade stability.
|
|
48
48
|
>
|
|
@@ -59,14 +59,17 @@ print(orders["revenue"]) # ✗ unknown-column — 'revenue' not in inferred col
|
|
|
59
59
|
```
|
|
60
60
|
|
|
61
61
|
```shell
|
|
62
|
-
typedframes check src/ --fail-under=90
|
|
62
|
+
typedframes check src/ --coverage-fail-under=90
|
|
63
63
|
# src/pipeline.py:7:8: error[unknown-column] Column 'revenue' does not exist in inferred column set (defined at line 6)
|
|
64
64
|
# ✗ Found 1 error in 12 files (0.0s)
|
|
65
65
|
# ✗ DataFrame schema coverage 82.0% is below the required 90.0% (9/11 DataFrames had column info)
|
|
66
66
|
```
|
|
67
67
|
|
|
68
|
-
`--fail-under=N` is the same idea as a test-coverage or mypy type-coverage gate, applied to how many DataFrames
|
|
69
|
-
the checker can resolve
|
|
68
|
+
`--coverage-fail-under=N` is the same idea as a test-coverage or mypy type-coverage gate, applied to how many DataFrames
|
|
69
|
+
the checker can resolve to a concrete column list — a DataFrame it merely recognizes but can't resolve columns for
|
|
70
|
+
(a Feast retrieval whose entity_df isn't itself concrete, a bare `-> pd.DataFrame` return with no attached Schema)
|
|
71
|
+
does not count, and any column accessed on it is flagged `unverifiable-column` rather than passed through silently.
|
|
72
|
+
See [DataFrame Schema Coverage Thresholds](#dataframe-schema-coverage-thresholds-opt-in).
|
|
70
73
|
Add `BaseSchema` classes later for cross-file awareness and IDE autocomplete — see [Quick Start](#quick-start).
|
|
71
74
|
|
|
72
75
|
---
|
|
@@ -78,6 +81,7 @@ Add `BaseSchema` classes later for cross-file awareness and IDE autocomplete —
|
|
|
78
81
|
- [Quick Start](#quick-start)
|
|
79
82
|
- [Column Inference](#column-inference)
|
|
80
83
|
- [Static Analysis](#static-analysis)
|
|
84
|
+
- [Jupyter Notebooks](#jupyter-notebooks)
|
|
81
85
|
- [DataFrame Schema Coverage Thresholds](#dataframe-schema-coverage-thresholds-opt-in)
|
|
82
86
|
- [Static Analysis Performance](#static-analysis-performance)
|
|
83
87
|
- [Type Safety With Multiple Backends](#type-safety-with-multiple-backends)
|
|
@@ -104,7 +108,7 @@ tracking and IDE autocomplete; they're a progressive enhancement, not a prerequi
|
|
|
104
108
|
**What you get:**
|
|
105
109
|
|
|
106
110
|
- ✅ **Works without schema annotations** - Column inference from `usecols=`, `dtype=`, and method chains catches errors on unannotated code
|
|
107
|
-
- ✅ **CI gate via coverage threshold** - `--fail-under=N` fails the build when too much of your codebase is invisible to the checker, the same way you'd gate on test coverage or a type checker's type-coverage number
|
|
111
|
+
- ✅ **CI gate via coverage threshold** - `--coverage-fail-under=N` fails the build when too much of your codebase is invisible to the checker, the same way you'd gate on test coverage or a type checker's type-coverage number
|
|
108
112
|
- ✅ **Rust-fast** - Milliseconds, not seconds, even on hundreds of files; fast enough for pre-commit hooks and CI (see [benchmarks](#static-analysis-performance))
|
|
109
113
|
- ✅ **Cross-file awareness** - Add `BaseSchema` and typed return annotations to follow schemas across module boundaries
|
|
110
114
|
- ✅ **Refactor-safe access** - `df[Schema.column_group.s].mean()` (pandas) or `df.select(Schema.col.col)` (polars) instead of scattered string literals
|
|
@@ -125,7 +129,13 @@ or
|
|
|
125
129
|
uv add typedframes
|
|
126
130
|
```
|
|
127
131
|
|
|
128
|
-
The Rust-based checker is included — no separate install needed.
|
|
132
|
+
The Rust-based checker is included — no separate install needed. `typedframes` itself has
|
|
133
|
+
no required dependencies; add the extra for whichever backend(s) your code uses:
|
|
134
|
+
|
|
135
|
+
```shell
|
|
136
|
+
pip install typedframes[pandas] # includes pandas
|
|
137
|
+
pip install typedframes[polars] # includes polars
|
|
138
|
+
```
|
|
129
139
|
|
|
130
140
|
---
|
|
131
141
|
|
|
@@ -422,17 +432,20 @@ typedframes check src/ --strict
|
|
|
422
432
|
# JSON output
|
|
423
433
|
typedframes check src/ --output-format=json
|
|
424
434
|
|
|
425
|
-
#
|
|
435
|
+
# Check a single file (follows that file's own imports, not the whole project)
|
|
436
|
+
typedframes check src/pipeline.py
|
|
437
|
+
|
|
438
|
+
# Skip the cross-file index entirely (every file in isolation, fastest)
|
|
426
439
|
typedframes check src/ --no-index
|
|
427
440
|
|
|
428
441
|
# Suppress all warnings (untracked-dataframe, dropped-unknown-column)
|
|
429
442
|
typedframes check src/ --no-warnings
|
|
430
443
|
|
|
431
444
|
# Enforce minimum DataFrame schema coverage (see below)
|
|
432
|
-
typedframes check src/ --fail-under=90
|
|
445
|
+
typedframes check src/ --coverage-fail-under=90
|
|
433
446
|
|
|
434
447
|
# Show which DataFrames lack column info, per file
|
|
435
|
-
typedframes check src/ --coverage-
|
|
448
|
+
typedframes check src/ --coverage-detail=term-missing
|
|
436
449
|
```
|
|
437
450
|
|
|
438
451
|
To suppress warnings project-wide, add to `pyproject.toml`:
|
|
@@ -480,111 +493,174 @@ See the full [Method Matrix](https://typedframes.readthedocs.io/en/latest/method
|
|
|
480
493
|
for the complete list of tracked, passthrough, and untracked operations, plus the error
|
|
481
494
|
code reference.
|
|
482
495
|
|
|
483
|
-
|
|
484
|
-
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
488
|
-
|
|
489
|
-
|
|
490
|
-
|
|
491
|
-
|
|
496
|
+
### Pre-commit Hook
|
|
497
|
+
|
|
498
|
+
```yaml
|
|
499
|
+
# .pre-commit-config.yaml
|
|
500
|
+
repos:
|
|
501
|
+
- repo: local
|
|
502
|
+
hooks:
|
|
503
|
+
- id: typedframes
|
|
504
|
+
name: typedframes check
|
|
505
|
+
entry: typedframes check . --strict
|
|
506
|
+
language: python
|
|
507
|
+
additional_dependencies: ["typedframes==0.5.0"]
|
|
508
|
+
types_or: [python, jupyter]
|
|
509
|
+
pass_filenames: false
|
|
510
|
+
```
|
|
492
511
|
|
|
493
|
-
|
|
494
|
-
|
|
512
|
+
`pass_filenames: false` is deliberate: the checker takes a single path and builds its
|
|
513
|
+
cross-file index from it, so one run per commit still sees the modules the commit didn't
|
|
514
|
+
touch — a per-file invocation would lose that. `--strict` is what makes the hook block
|
|
515
|
+
the commit: `typedframes check` exits 0 by default even when it finds real errors. Add
|
|
516
|
+
the coverage gate to the same line:
|
|
495
517
|
|
|
496
|
-
|
|
518
|
+
```yaml
|
|
519
|
+
# with coverage gating
|
|
520
|
+
entry: typedframes check . --strict --coverage-fail-under=90
|
|
521
|
+
```
|
|
497
522
|
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
|
|
509
|
-
|
|
510
|
-
|
|
511
|
-
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
515
|
-
|
|
516
|
-
|
|
517
|
-
|
|
518
|
-
#
|
|
519
|
-
|
|
520
|
-
|
|
521
|
-
|
|
522
|
-
|
|
523
|
-
# When several globs match one file the most specific wins — longest literal
|
|
524
|
-
# prefix before the first `*` or `?`. Files matching no glob use `fail_under`.
|
|
525
|
-
# "legacy/**" = 50.0
|
|
526
|
-
# "src/new_module/**" = 100.0
|
|
523
|
+
This repository also ships a `.pre-commit-hooks.yaml`, so once a tagged release contains
|
|
524
|
+
it you can point `repo:` straight at `https://github.com/w-martin/typedframes` with a
|
|
525
|
+
`rev:` and drop the boilerplate above. That form needs a Rust toolchain wherever the hook
|
|
526
|
+
runs — pre-commit's python installer builds the repository from source rather than
|
|
527
|
+
fetching a wheel — so the local hook stays the quicker one to install either way.
|
|
528
|
+
|
|
529
|
+
### GitHub Actions
|
|
530
|
+
|
|
531
|
+
```yaml
|
|
532
|
+
# .github/workflows/typedframes.yml
|
|
533
|
+
name: typedframes
|
|
534
|
+
on: [push, pull_request]
|
|
535
|
+
|
|
536
|
+
jobs:
|
|
537
|
+
typedframes:
|
|
538
|
+
runs-on: ubuntu-latest
|
|
539
|
+
steps:
|
|
540
|
+
- uses: actions/checkout@v4
|
|
541
|
+
# No tagged release contains action.yml yet. Repin this to that tag once one
|
|
542
|
+
# ships: an unpinned third-party action is a supply-chain risk, and @main is a
|
|
543
|
+
# stopgap rather than a recommendation.
|
|
544
|
+
- uses: w-martin/typedframes@main
|
|
545
|
+
with:
|
|
546
|
+
path: src/
|
|
547
|
+
coverage-fail-under: "90"
|
|
527
548
|
```
|
|
528
549
|
|
|
529
|
-
|
|
530
|
-
|
|
531
|
-
(the way `ruff.toml` drops `[tool.ruff]`):
|
|
550
|
+
The action installs the PyPI wheel into a throwaway virtualenv and runs the checker with
|
|
551
|
+
`--output-format=github`, so errors arrive as annotations on the pull request diff.
|
|
532
552
|
|
|
533
|
-
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
```
|
|
553
|
+
| Input | Default | |
|
|
554
|
+
|-------|---------|-|
|
|
555
|
+
| `path` | `.` | File or directory to check |
|
|
556
|
+
| `version` | `latest` | PyPI version to install, e.g. `"0.5.0"` |
|
|
557
|
+
| `strict` | `true` | Fail the step on errors — `typedframes check` exits 0 without it |
|
|
558
|
+
| `coverage-fail-under` | *(unset)* | Minimum DataFrame schema coverage, e.g. `"90"` |
|
|
559
|
+
| `coverage-detail` | `summary` | Or `term-missing` for the per-file breakdown |
|
|
560
|
+
| `no-warnings` | `false` | Suppress warning-level diagnostics |
|
|
561
|
+
| `args` | *(empty)* | Escape hatch for `--no-index`, `--lenient-ingest`, `--no-info` |
|
|
543
562
|
|
|
544
|
-
|
|
545
|
-
so exactly one file explains the whole configuration.
|
|
563
|
+
---
|
|
546
564
|
|
|
547
|
-
|
|
565
|
+
## Jupyter Notebooks
|
|
548
566
|
|
|
549
|
-
The
|
|
550
|
-
|
|
567
|
+
The standalone checker reads `.ipynb` files directly — no `nbconvert`, no `jupytext`, no
|
|
568
|
+
flag, no conversion step. Notebooks are parsed with
|
|
569
|
+
[`ruff_notebook`](https://github.com/astral-sh/ruff/tree/main/crates/ruff_notebook), the
|
|
570
|
+
same crate Ruff uses for its own notebook support:
|
|
551
571
|
|
|
552
572
|
```shell
|
|
553
|
-
|
|
573
|
+
# Check one notebook
|
|
574
|
+
typedframes check analysis.ipynb
|
|
575
|
+
|
|
576
|
+
# Check a directory — .py and .ipynb files are both picked up, no flag needed
|
|
577
|
+
typedframes check notebooks/
|
|
578
|
+
|
|
579
|
+
# Output — `cell N` is the notebook's Nth cell; line:col are relative to that cell:
|
|
580
|
+
# analysis.ipynb:cell 6:2:1: warning[untracked-dataframe] columns unknown at lint time; specify
|
|
581
|
+
# `usecols`/`columns`, or annotate the variable's type
|
|
582
|
+
# analysis.ipynb:cell 10:1:1: error[unknown-column] Column 'revenue' does not exist in Orders
|
|
583
|
+
# {amount, customer_id, order_id} (defined at analysis.ipynb cell 2:8)
|
|
584
|
+
# ✗ Found 1 error, 2 warnings in 1 file (0.0s)
|
|
554
585
|
```
|
|
555
586
|
|
|
556
|
-
|
|
557
|
-
|
|
558
|
-
|
|
559
|
-
|
|
560
|
-
|
|
561
|
-
|
|
562
|
-
|
|
563
|
-
|
|
587
|
+
**Features:**
|
|
588
|
+
- ✅ Auto-detected — every `.ipynb` under a checked path, alongside `.py` files
|
|
589
|
+
- ✅ Diagnostics reported as `cell N:line:col`, mapped back to the cell that produced them
|
|
590
|
+
rather than a line number in the raw notebook JSON — including a same-notebook schema's
|
|
591
|
+
`(defined at ...)` cross-reference
|
|
592
|
+
- ✅ Same error codes, severities, and flags as `.py` files: `--strict`,
|
|
593
|
+
`--output-format`, `--no-warnings`, `--lenient-ingest`, coverage thresholds
|
|
594
|
+
- ✅ IPython magics and shell escapes parse natively (`%matplotlib inline`, `%%time`,
|
|
595
|
+
`!pip install ...`) — nothing has to be stripped first
|
|
596
|
+
- ✅ `# typedframes: ignore` suppresses a diagnostic on a line inside a cell, same as in a
|
|
597
|
+
`.py` file
|
|
598
|
+
- ✅ A schema imported from a project `.py` module resolves through the cross-file index,
|
|
599
|
+
so `orders = load_orders()` in a notebook is validated against the schema on
|
|
600
|
+
`load_orders`' return annotation
|
|
601
|
+
- ✅ `.ipynb_checkpoints/` is pruned by default, so stale checkpoint copies aren't
|
|
602
|
+
re-reported
|
|
603
|
+
|
|
604
|
+
**Caveats:**
|
|
605
|
+
- ❌ The mypy plugin ([Option 2](#option-2-mypy-plugin-comprehensive)) is `.py`-only —
|
|
606
|
+
mypy reads Python source files, so notebook checking is standalone-checker-only
|
|
607
|
+
- ❌ `cell N` counts *every* cell in the notebook, markdown and raw cells included — so
|
|
608
|
+
the number is a cell's position in the notebook, not "the Nth code cell"
|
|
609
|
+
- ❌ A notebook is never a *source* of schemas for other files (Python can't import an
|
|
610
|
+
`.ipynb`); a schema defined in a notebook is visible only inside that notebook
|
|
611
|
+
- ❌ A notebook that isn't valid notebook JSON, whose kernel isn't Python, or whose code
|
|
612
|
+
doesn't parse is skipped with a message on stderr and the run continues — a skip alone
|
|
613
|
+
never fails the run, `--strict` included
|
|
614
|
+
- ❌ Under `--output-format=github`, the annotation's `line`/`col` stay cell-relative and
|
|
615
|
+
the cell number moves into the annotation title, since GitHub annotations address raw
|
|
616
|
+
file lines and an `.ipynb` is JSON rather than source text
|
|
617
|
+
|
|
618
|
+
### See Also
|
|
564
619
|
|
|
565
|
-
|
|
566
|
-
|
|
567
|
-
|
|
620
|
+
- [`examples/features/ipynb_example.ipynb`](examples/features/ipynb_example.ipynb) — a
|
|
621
|
+
runnable notebook with one schema, an IPython magic, and three intentional bugs
|
|
622
|
+
(`untracked-dataframe`, `dropped-unknown-column`, `unknown-column`), all caught by
|
|
623
|
+
`typedframes check ipynb_example.ipynb` without running a cell.
|
|
568
624
|
|
|
569
|
-
|
|
570
|
-
with `--output-format=json` and the coverage report is nested under a `coverage` key so
|
|
571
|
-
stdout stays a single valid JSON document:
|
|
625
|
+
---
|
|
572
626
|
|
|
573
|
-
|
|
574
|
-
|
|
627
|
+
## DataFrame Schema Coverage Thresholds (Opt-In)
|
|
628
|
+
|
|
629
|
+
**DataFrame schema coverage** is the fraction of DataFrames `typedframes check` could
|
|
630
|
+
resolve column information for — the analogue of the "type coverage" reported by mypy,
|
|
631
|
+
pyright, and pyre, and unrelated to test coverage. That number is informational by
|
|
632
|
+
default. If you want it enforced — failing the run when too much of your code is
|
|
633
|
+
invisible to the checker — enable a threshold:
|
|
634
|
+
|
|
635
|
+
```toml
|
|
636
|
+
[tool.typedframes.coverage]
|
|
637
|
+
enabled = true
|
|
638
|
+
fail_under = 90.0
|
|
575
639
|
```
|
|
576
640
|
|
|
577
|
-
|
|
641
|
+
or as a one-off, without touching config:
|
|
642
|
+
|
|
643
|
+
```shell
|
|
644
|
+
typedframes check src/ --coverage-fail-under=90
|
|
645
|
+
```
|
|
578
646
|
|
|
579
|
-
|
|
580
|
-
|
|
581
|
-
|
|
582
|
-
|
|
583
|
-
|
|
584
|
-
|
|
585
|
-
|
|
586
|
-
|
|
587
|
-
|
|
647
|
+
This is **entirely opt-in** — with no `[tool.typedframes.coverage]` table and no
|
|
648
|
+
`--coverage-fail-under`, nothing changes: no threshold, no exit-code difference. Per-path
|
|
649
|
+
overrides (e.g. a lower bar for `legacy/**`), a `--coverage-detail=term-missing`
|
|
650
|
+
breakdown of exactly which DataFrames cost you coverage (as text or, combined with
|
|
651
|
+
`--output-format=json`, as structured JSON), and the full config reference all live in
|
|
652
|
+
the
|
|
653
|
+
[DataFrame schema coverage thresholds guide](docs/usage.md#dataframe-schema-coverage-thresholds).
|
|
654
|
+
|
|
655
|
+
Some DataFrames are recognized but never resolve to a concrete column list — a Feast
|
|
656
|
+
retrieval whose `entity_df` isn't itself concretely typed, or a bare `-> pd.DataFrame`
|
|
657
|
+
return with no attached `Schema`. These do **not** count as covered, and accessing any
|
|
658
|
+
column on one is flagged `unverifiable-column` rather than passed through silently —
|
|
659
|
+
the checker genuinely doesn't know whether the column exists, so it says so instead of
|
|
660
|
+
guessing either way. A Feast retrieval whose `entity_df` argument IS a concretely typed
|
|
661
|
+
DataFrame is fully resolved instead: Feast's real output is `entity_df`'s own columns
|
|
662
|
+
plus the requested features, so both halves get checked normally. See
|
|
663
|
+
[Unresolved schemas](docs/usage.md#unresolved-schemas-recognized-but-not-column-checked).
|
|
588
664
|
|
|
589
665
|
## Static Analysis Performance
|
|
590
666
|
|
|
@@ -607,9 +683,11 @@ Fast feedback reduces development time. The typedframes Rust binary provides nea
|
|
|
607
683
|
|
|
608
684
|
The typedframes binary resolves column names within a file and, when a project index is present, across files too.
|
|
609
685
|
Run `typedframes check src/` to build the index automatically and catch errors like `df = load_users(); df["typo"]`
|
|
610
|
-
even when `load_users` is defined in another module.
|
|
611
|
-
|
|
612
|
-
|
|
686
|
+
even when `load_users` is defined in another module. Checking a single file
|
|
687
|
+
(`typedframes check src/pipeline.py`) builds a smaller index from that file's own imports instead of walking the whole
|
|
688
|
+
project, so it still resolves `load_users` without paying for files it never references. Pass `--no-index` to skip the
|
|
689
|
+
index entirely and check each file in isolation. Full type checkers (mypy, pyright, ty) analyze all Python types across
|
|
690
|
+
your entire codebase. Use both: the binary for fast iteration, mypy for comprehensive checking.
|
|
613
691
|
|
|
614
692
|
The standalone checker is built with [`ruff_python_parser`](https://github.com/astral-sh/ruff) for Python AST
|
|
615
693
|
parsing.
|
|
@@ -824,6 +902,7 @@ Comprehensive comparison of pandas/DataFrame typing and validation tools. **type
|
|
|
824
902
|
| Column name checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
825
903
|
| Column type checking | ✅ Yes | ⚠️ Limited | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
826
904
|
| Typo suggestions | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
905
|
+
| Coverage gate (`--coverage-fail-under`) | ✅ Yes | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No | ❌ No |
|
|
827
906
|
| **Runtime Validation** |
|
|
828
907
|
| Data validation | ❌ No | ✅ Excellent | ✅ Excellent | ✅ typeguard | ❌ No | ✅ Yes | ✅ Yes | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
|
|
829
908
|
| Value constraints | ❌ No | ✅ Yes | ✅ Excellent | ❌ No | ❌ No | ❌ No | ❌ No | ✅ Yes | ❌ No | ✅ Yes | ✅ Yes |
|
|
@@ -967,122 +1046,26 @@ The conversion maps:
|
|
|
967
1046
|
|
|
968
1047
|
## Examples
|
|
969
1048
|
|
|
970
|
-
|
|
971
|
-
|
|
972
|
-
|
|
973
|
-
|
|
974
|
-
|
|
975
|
-
|
|
976
|
-
|
|
977
|
-
|
|
978
|
-
|
|
979
|
-
|
|
980
|
-
|
|
981
|
-
|
|
982
|
-
|
|
983
|
-
|
|
984
|
-
|
|
985
|
-
|
|
986
|
-
|
|
987
|
-
|
|
988
|
-
|
|
989
|
-
|
|
990
|
-
revenue = calculate_revenue(df)
|
|
991
|
-
```
|
|
992
|
-
|
|
993
|
-
### Time Series Analysis
|
|
994
|
-
|
|
995
|
-
```python
|
|
996
|
-
from typing import Annotated
|
|
997
|
-
import pandas as pd
|
|
998
|
-
from typedframes import BaseSchema, Column, ColumnSet, ColumnGroup
|
|
999
|
-
|
|
1000
|
-
|
|
1001
|
-
class SensorData(BaseSchema):
|
|
1002
|
-
timestamp = Column(type=str)
|
|
1003
|
-
temperature = ColumnSet(type=float, members=["temp_1", "temp_2", "temp_3"])
|
|
1004
|
-
humidity = ColumnSet(type=float, members=["humidity_1", "humidity_2"])
|
|
1005
|
-
|
|
1006
|
-
all_sensors = ColumnGroup(members=[temperature, humidity])
|
|
1007
|
-
|
|
1008
|
-
|
|
1009
|
-
df: Annotated[pd.DataFrame, SensorData] = pd.read_csv("sensors.csv")
|
|
1010
|
-
|
|
1011
|
-
# Clean, type-safe operations using .s for column name lists
|
|
1012
|
-
avg_temp_per_row = df[SensorData.temperature.s].mean(axis=1)
|
|
1013
|
-
all_readings_stats = df[SensorData.all_sensors.s].describe()
|
|
1014
|
-
```
|
|
1015
|
-
|
|
1016
|
-
### Multi-Step Pipeline
|
|
1017
|
-
|
|
1018
|
-
```python
|
|
1019
|
-
from typing import Annotated
|
|
1020
|
-
import pandas as pd
|
|
1021
|
-
from typedframes import BaseSchema, Column
|
|
1022
|
-
|
|
1023
|
-
|
|
1024
|
-
class RawSales(BaseSchema):
|
|
1025
|
-
date = Column(type=str)
|
|
1026
|
-
product_id = Column(type=int)
|
|
1027
|
-
quantity = Column(type=int)
|
|
1028
|
-
price = Column(type=float)
|
|
1029
|
-
|
|
1030
|
-
|
|
1031
|
-
class AggregatedSales(BaseSchema):
|
|
1032
|
-
date = Column(type=str)
|
|
1033
|
-
total_revenue = Column(type=float)
|
|
1034
|
-
total_quantity = Column(type=int)
|
|
1035
|
-
|
|
1036
|
-
|
|
1037
|
-
def aggregate_daily(df: Annotated[pd.DataFrame, RawSales]) -> Annotated[pd.DataFrame, AggregatedSales]:
|
|
1038
|
-
result = (
|
|
1039
|
-
df.groupby(RawSales.date.s)
|
|
1040
|
-
.agg(
|
|
1041
|
-
{
|
|
1042
|
-
RawSales.price.s: "sum",
|
|
1043
|
-
RawSales.quantity.s: "sum",
|
|
1044
|
-
}
|
|
1045
|
-
)
|
|
1046
|
-
.reset_index()
|
|
1047
|
-
)
|
|
1048
|
-
result.columns = pd.Index(["date", "total_revenue", "total_quantity"])
|
|
1049
|
-
return result # type: ignore[return-value]
|
|
1050
|
-
|
|
1051
|
-
|
|
1052
|
-
# Type-safe pipeline
|
|
1053
|
-
raw: Annotated[pd.DataFrame, RawSales] = pd.read_csv("sales.csv")
|
|
1054
|
-
aggregated = aggregate_daily(raw)
|
|
1055
|
-
|
|
1056
|
-
|
|
1057
|
-
# Type checker validates schema transformations
|
|
1058
|
-
def analyze(df: Annotated[pd.DataFrame, AggregatedSales]) -> float:
|
|
1059
|
-
df["total_revenue"] # ✓ OK
|
|
1060
|
-
df["price"] # ✗ Error: 'price' not in AggregatedSales
|
|
1061
|
-
return df[AggregatedSales.total_revenue.s].mean()
|
|
1062
|
-
```
|
|
1063
|
-
|
|
1064
|
-
### Polars Performance Pipeline
|
|
1065
|
-
|
|
1066
|
-
```python
|
|
1067
|
-
from typing import Annotated
|
|
1068
|
-
import polars as pl
|
|
1069
|
-
from typedframes import BaseSchema, Column
|
|
1070
|
-
|
|
1071
|
-
|
|
1072
|
-
class LargeDataset(BaseSchema):
|
|
1073
|
-
id = Column(type=int)
|
|
1074
|
-
value = Column(type=float)
|
|
1075
|
-
category = Column(type=str)
|
|
1076
|
-
|
|
1077
|
-
|
|
1078
|
-
def efficient_aggregation(df: Annotated[pl.DataFrame, LargeDataset]) -> pl.DataFrame:
|
|
1079
|
-
return df.filter(pl.col("value") > 100).group_by("category").agg(pl.col("value").mean())
|
|
1080
|
-
|
|
1081
|
-
|
|
1082
|
-
# Polars handles large files efficiently
|
|
1083
|
-
df: Annotated[pl.DataFrame, LargeDataset] = pl.read_csv("huge_file.csv")
|
|
1084
|
-
result = efficient_aggregation(df)
|
|
1085
|
-
```
|
|
1049
|
+
Runnable versions of everything shown in [Quick Start](#quick-start) and
|
|
1050
|
+
[Advanced Usage](#advanced-usage) live under [`examples/features/`](examples/features/):
|
|
1051
|
+
|
|
1052
|
+
- [`annotated_pandas_example.py`](examples/features/annotated_pandas_example.py) —
|
|
1053
|
+
`Annotated[pd.DataFrame, Schema]` with string and `.s` descriptor column access
|
|
1054
|
+
- [`annotated_polars_example.py`](examples/features/annotated_polars_example.py) —
|
|
1055
|
+
the same, with `pl.col()` and `.col` descriptor expressions
|
|
1056
|
+
- [`typedframes_example.py`](examples/features/typedframes_example.py) — pandas and
|
|
1057
|
+
polars side by side against one shared schema
|
|
1058
|
+
- [`schema_algebra_example.py`](examples/features/schema_algebra_example.py) —
|
|
1059
|
+
composing schemas via inheritance and the `+` operator
|
|
1060
|
+
- [`inference_example.py`](examples/features/inference_example.py) — all four
|
|
1061
|
+
column-inference modes with annotated ✓/✗ comments, no `BaseSchema` at all
|
|
1062
|
+
- [`multi_file_inference/`](examples/features/multi_file_inference/) and
|
|
1063
|
+
[`multi_file_with_schema/`](examples/features/multi_file_with_schema/) — the same
|
|
1064
|
+
cross-file pipeline checked with and without schemas, see
|
|
1065
|
+
[`examples/features/README.md`](examples/features/README.md) for what each one catches
|
|
1066
|
+
- [`ipynb_example.ipynb`](examples/features/ipynb_example.ipynb) — the same checks in a
|
|
1067
|
+
Jupyter notebook, with `cell N:line:col` locations (see [Jupyter
|
|
1068
|
+
Notebooks](#jupyter-notebooks))
|
|
1086
1069
|
|
|
1087
1070
|
---
|
|
1088
1071
|
|
|
@@ -1170,35 +1153,6 @@ MIT License - see [LICENSE](LICENSE)
|
|
|
1170
1153
|
|
|
1171
1154
|
---
|
|
1172
1155
|
|
|
1173
|
-
## Roadmap
|
|
1174
|
-
|
|
1175
|
-
**Shipped:**
|
|
1176
|
-
- [x] Schema definition API
|
|
1177
|
-
- [x] Pandas support
|
|
1178
|
-
- [x] Polars support
|
|
1179
|
-
- [x] Mypy plugin
|
|
1180
|
-
- [x] Standalone checker (Rust)
|
|
1181
|
-
- [x] Explicit backend types
|
|
1182
|
-
- [x] Merge/join schema preservation
|
|
1183
|
-
- [x] Schema Composition (multiple inheritance, `SchemaA + SchemaB`)
|
|
1184
|
-
- [x] Column name collision warnings
|
|
1185
|
-
- [x] Pandera integration (`to_pandera_schema()`)
|
|
1186
|
-
- [x] Cross-file schema inference (project-level index, `--no-index` flag)
|
|
1187
|
-
- [x] Aggressive column inference (untracked-dataframe/dropped-unknown-column warnings, method chain propagation)
|
|
1188
|
-
- [x] Function parameter contracts (`missing-column`), resolved transitively across chains of helper
|
|
1189
|
-
functions and cross-file calls; schema-annotated parameters take priority over body-scanning
|
|
1190
|
-
- [x] SQL / data-warehouse column inference (`SELECT` list parsing across Snowflake, BigQuery, Athena, Redshift,
|
|
1191
|
-
Databricks, PySpark, DuckDB, SQLAlchemy Core/ORM, Feast, and T-SQL/Synapse/Fabric dialects)
|
|
1192
|
-
- [x] Jupyter notebook (`.ipynb`) checking — code cells are checked directly, with errors reported as
|
|
1193
|
-
`notebook.ipynb:cell N:line:col`
|
|
1194
|
-
|
|
1195
|
-
**Planned:**
|
|
1196
|
-
|
|
1197
|
-
- [ ] **Opt-in data loading constraints** - `Field` class with constraints (`gt`, `ge`, `lt`, `le`), strictly isolated
|
|
1198
|
-
to `from_schema()` ingestion boundaries
|
|
1199
|
-
|
|
1200
|
-
---
|
|
1201
|
-
|
|
1202
1156
|
## FAQ
|
|
1203
1157
|
|
|
1204
1158
|
**Q: Do I need to choose between pandas and polars?**
|