cleanframe-engine 0.4.0__tar.gz → 0.5.1__tar.gz

{cleanframe_engine-0.4.0 → cleanframe_engine-0.5.1}/.gitignore

@@ -99,3 +99,6 @@ examples/messy_customers.recipe.yaml
 examples/messy_customers.schema.yaml
 !examples/customer.recipe.yaml
 !examples/customer.schema.yaml
+
+# `cleanframe demo` output when run inside the repository
+/cleanframe-demo/

{cleanframe_engine-0.4.0 → cleanframe_engine-0.5.1}/CHANGELOG.md

@@ -7,6 +7,132 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.5.1] — 2026-10-19
+
+One logo everywhere, a README hero made from real output, and a correct row count in `cleanframe demo`.
+
+### Changed
+
+- The HTML profiling and diff reports show the CleanFrame mark in their header, the same mark as the
+  README, the docs and the playground. `docs/assets/Logo.svg` is now the one source of the mark.
+- The README hero is rendered from a real `cleanframe demo` run by `scripts/render_brand_assets.py`, in a
+  light and a dark version: the messy file, the recipe excerpt, the clean output with its quarantined row,
+  and the drift stop on next month's file. It replaces an illustration whose steps were not CleanFrame ops.
+
+### Fixed
+
+- `cleanframe demo` said "1 duplicate row(s) dropped" for the row it had quarantined. Its summary now
+  reports the quarantined row on its own and names duplicates only when the recipe dropped some.
+
+## [0.5.0] — 2026-10-18
+
+Decision models, agents, a playground and data CI. CleanFrame can now ask a typed decision model
+(TypeSafe's Jev, an open "System One" model, or any LLM through an adapter) the judgment calls rules
+leave open, and freeze the confident answers into the recipe. Replay still makes zero AI calls. This
+release also fixes two pandas 3 bugs and a rules merge that could fold one real value into another.
+
+### Added
+
+- **Decision models: `clean(decide=...)` / `cleanframe clean --decide SPEC`** (`cleanframe/decisions.py`).
+  CleanFrame asks four kinds of typed questions: is `BLR` another name for `Bangalore` (choice); is a typo
+  merge the rules made really the same category (yes/no, and a "no" **vetoes** the merge); does an ambiguous
+  token such as `none`/`-`/`unknown` mean missing in *this* column (yes/no); which source column holds a
+  weakly-matched schema field (choice).
+  - Backends: `typesafe/<model>` (`TYPESAFE_API_KEY`), `openrouter/typesafe/<model>` (`OPENROUTER_API_KEY`),
+    `systemone/<model>` for self-hosted Jev-compatible servers (`SYSTEMONE_BASE_URL`), `rules` (offline
+    heuristics), any LLM spec through an adapter (recorded as `calibrated: false`), or a custom object with
+    `.decide()`. The System One wire protocol is implemented with the standard library: no new dependency.
+  - Policy: an answer that changes data needs `p ≥ max(mode threshold, decision_threshold)` (0.9 by default).
+    Below that it is a recorded suggestion, never executed. Undoing a rules merge only needs `p > 0.5`.
+    Confirming every typo merge in a column lifts it to 0.9 confidence so it survives `strict`.
+  - Privacy follows `llm_exposure`: under `metadata` only schema questions (names, types, patterns) and
+    null-token questions (tokens from CleanFrame's fixed vocabulary) are asked. Category questions need
+    `sample` and are only asked about columns whose values repeat.
+  - Every answer is recorded in `recipe.meta.decisions` (model, calibrated, threshold, counts, records);
+    `CleanResult.decisions` and the CLI's `--json` summary carry the same. Questions are built from sorted
+    distinct values, so row order cannot change them. `max_decisions` caps cost, 429/5xx are retried with
+    backoff, and a failing backend falls back to the rules plan with a warning (`--no-llm-fallback` raises).
+  - New flags: `--decide`, `--decision-threshold`, `--max-decisions`.
+- **`cleanframe demo`.** A guided run on sample data that now ships inside the wheel
+  (`cleanframe/demo_data/`): profile, clean into a recipe, watch next month's drifted file **stop**, then
+  replay the drafted patch. It works right after `pip install`, with no clone and no API key. Also
+  `cleanframe.demo.demo_file()` / `run_demo()`. Before this, the README's first command needed a clone.
+- `examples/next_month_customers.csv`: the drifted sample the demo uses.
+- **MCP server for AI agents**: `cleanframe-mcp`, `cleanframe mcp` or `python -m cleanframe.mcp_server`
+  (extra `[mcp]`, `mcp>=2.3`). Seven tools: `profile_data`, `suggest_recipe`, `apply_recipe`,
+  `check_drift`, `explain_recipe`, `list_capabilities`, `export_code`.
+  - Errors come back structured (`ok`/`kind`) and flagged `isError`. Output is bounded JSON, with every cap reported.
+  - The input file is never written, and drift stops a replay by default.
+  - `CLEANFRAME_AGENT_HIDE_VALUES=1` replaces quoted cell values with `"<hidden>"`.
+  - The same tools are plain functions in `cleanframe.agent_tools`, with no `mcp` dependency.
+- **Agent Skill** `skills/cleanframe/SKILL.md`, installed with `npx skills add inboxpraveen/Cleanframe`. It
+  teaches Claude Code, Codex, Cursor and other agents the recipe workflow and its hard rules, instead of
+  throwaway pandas.
+- **GitHub Action "CleanFrame Data CI"** (`action/`, `uses: inboxpraveen/Cleanframe/action@v0.5.0`).
+  - Replays committed recipes over data files on pull requests or a schedule, and fails the check on schema
+    drift (exit 3) or validation failure (exit 4).
+  - Writes a per-file job-summary table and annotations.
+  - Outputs: `drift`, `changed-cells`, `quarantined`, `failed-files`, `exit-code`, `results-file`.
+  - Inputs reach its scripts only through environment variables, and CleanFrame runs in an isolated virtualenv.
+  - See `docs/github-action.md` and `examples/github-action/data-ci.yml`.
+- **In-browser playground** (`playground/`, https://inboxpraveen.github.io/Cleanframe/).
+  - Drop a CSV/TSV/XLSX and get a Mess Score (100 − quality score), the detected problems, the YAML recipe, a
+    cell-level diff, the generated pandas code and both HTML reports.
+  - A shareable score card shows counts only, never names or values.
+  - CleanFrame runs client-side in Pyodide 314.0.7 in a Web Worker. A Content-Security-Policy limits the page
+    to its own site and the pinned Pyodide CDN path, so files never leave the browser.
+  - Deployed by `.github/workflows/pages.yml` from a wheel built from `main`; `python playground/dev_server.py`
+    serves it locally.
+- **CleanBench** (`bench/`, not shipped in the wheel or sdist): a benchmark for data-cleaning decisions.
+  - It runs `decide=` backends through CleanFrame's real pipeline on 60 hand-labelled messy tables
+    (299 questions: aliases, typo merges, null tokens, schema mapping).
+  - Gold is written as equivalence classes and checked by an offline oracle pass. Every scenario lists the
+    questions it must produce.
+  - Metrics: safe-apply precision (of the changes a system makes, how many are right), coverage@0.9, macro
+    accuracy with scenario-bootstrap CIs, ECE, Brier, cells fixed vs corrupted, latency, $ per 1k decisions.
+  - `python -m bench.cleanbench {validate,run,leaderboard,export,hash}`. `export` writes standalone System One
+    requests with gold, for other leaderboards.
+  - A private held-out split is committed only as a SHA-256.
+  - The rules baseline is on `bench/LEADERBOARD.md` with its misses visible. Docs: `docs/benchmark.md`.
+- CI: a required **pandas 3** job (pandas 3.0 shipped in January; the existing job only tried
+  pre-releases), and an **action smoke** job that runs the action on Linux, Windows and macOS runners.
+
+### Fixed
+
+- **pandas 3: `dedup` with `ignore_case` silently kept case-variant duplicates.** The key builder only
+  case-folded `object` columns, and pandas 3 reads text as dtype `str`. It now folds `str`/`string`/categorical
+  columns too, in the executor and in exported code alike.
+- **pandas 3.1: `fill_na` warns, and pandas 4 would make it fail,** when the fill value doesn't fit the
+  column: `empty` or a text value into a number or date column, `zero` into text (pandas 3's default
+  text dtype). pandas used to turn such a column into `object` silently; 3.1 deprecates that. The op and
+  exported code now cast explicitly first, so results are unchanged and nothing warns. Caught by the
+  pandas pre-release CI job on 3.1.0rc0.
+- **A typo merge could fold one real value into another.** `Austria` (rare) was merged into `Australia`
+  in `review` and `auto` mode because the names score 0.875 similar. A typo merge now also needs the two
+  spellings to be one edit apart, so look-alikes and opposites two letters apart stay separate
+  (`Hypothyroidism`/`Hyperthyroidism`, `Prednisone`/`Prednisolone`). Real typos (`Banglore`, `Kolkatta`,
+  `Califronia`) still merge; rarer two-edit typos are left to a human or a decision model.
+- **Categories that differ only in `+`, `-` or `#` were merged as case variants** (confidence 0.9, applied
+  in every mode): blood groups `A-` → `A+`, `O-` → `O+`; `C++`/`C#` → `C`. These marks now always count.
+  A hyphen between letters (`New-York`) is still cosmetic.
+- **`similarity()` scored any two non-Latin strings as identical** (both reduce to an empty ASCII key), so
+  `दिल्ली` matched `मुंबई` perfectly in schema mapping and drift. Such pairs are now compared as text.
+- Negation is detected inside phrases too (`Very dissatisfied` / `Very satisfied`).
+- **The quality score ignored date-format and money-as-text problems.** Those issues report `unparsed: 0`
+  when every value parses, and the scorer read that as "no rows affected". So a column in three date
+  formats cost nothing. The detectors now report an explicit `affected` count, and the scorer prefers it.
+  Scores for such files drop (the bundled sample goes from 87 to 80); that is the fix, not a regression.
+- `docs/plugins.md` showed `cleanframe --plugin X apply …`, which is a usage error; global flags go after
+  the subcommand.
+- Two tests assumed pandas 2 string semantics and failed on pandas 3 without a library bug behind them;
+  they now accept either representation of a missing text cell.
+
+### Changed
+
+- `pyproject.toml` and `CITATION.cff` name the maintainer (Praveen Kumar); PyPI keywords and project
+  URLs (Playground, Funding) added.
+- `docs/cli.md`: the `--json` summary documents `usage_error` and the `decisions` field.
+
 ## [0.4.0] — 2026-09-30
 
 A correctness release. An empirical review (26 hard-input probes, a 1M/3M-row benchmark and a
@@ -390,7 +516,10 @@ now fail loudly, and a few transforms that quietly corrupted data no longer run.
   deterministic executor, validation/quarantine, cell-level diff, schema drift, HTML reports,
   codegen, and CLI.
 
-[Unreleased]: https://github.com/inboxpraveen/Cleanframe/compare/v0.3.1...HEAD
+[Unreleased]: https://github.com/inboxpraveen/Cleanframe/compare/v0.5.1...HEAD
+[0.5.1]: https://github.com/inboxpraveen/Cleanframe/releases/tag/v0.5.1
+[0.5.0]: https://github.com/inboxpraveen/Cleanframe/releases/tag/v0.5.0
+[0.4.0]: https://github.com/inboxpraveen/Cleanframe/releases/tag/v0.4.0
 [0.3.1]: https://github.com/inboxpraveen/Cleanframe/releases/tag/v0.3.1
 [0.3.0]: https://github.com/inboxpraveen/Cleanframe/releases/tag/v0.3.0
 [0.2.0]: https://github.com/inboxpraveen/Cleanframe/commits/main

{cleanframe_engine-0.4.0 → cleanframe_engine-0.5.1}/CONTRIBUTING.md

@@ -27,6 +27,17 @@ than no change at all. The rules below exist to protect it.
    raw cell values by default, and its output must go through `Recipe.from_dict`
    (which validates it) — never straight to the executor.
 
+   The same holds for **decision models** (`decisions.py`, `clean(decide=...)`): they
+   answer typed questions while planning, and accepted answers become ordinary recipe
+   ops, so replay never calls them. Under the default `metadata` exposure a question may
+   carry column names, types, value *patterns* and — the one exception — a null token,
+   because the token always comes from CleanFrame's fixed vocabulary
+   (`ops.DEFAULT_NA_TOKENS`), never from free-form data. Category values are sent only
+   under `sample`. An answer that *changes* data needs `p ≥ max(mode threshold,
+   decision_threshold)`; below that it is a recorded suggestion, never executed.
+   Questions must be built from sorted distinct values so row order cannot change them
+   (`tests/test_decisions.py` checks this).
+
 3. **Recipes round-trip losslessly and idempotently.** `Recipe.from_yaml(r.to_yaml())`
    must equal `r`, and applying `to_yaml` twice must be byte-identical. If you add
    an op with parameters, you must add a `coerce`/`compact` pair such that
@@ -47,7 +58,9 @@ than no change at all. The rules below exist to protect it.
    1.5 g?) or malformed for the declared convention (`2,5` under `thousands=","`), the
    result is a counted, warned null, never a plausible-looking number. Wrong numbers
    are worse than missing ones. Likewise a clustering/merge rule must never fold together
-   values that differ in sign, decimal or grouping (`-300`/`300`).
+   values that differ in sign, decimal or grouping (`-300`/`300`), and a *typo* merge must
+   be a typo: one edit apart, so look-alike words such as `Austria`/`Australia` or
+   `Hypothyroidism`/`Hyperthyroidism` stay apart.
 
 5. **Every changed cell is tracked.** The executor assigns a stable row id and
    tracks column lineage so `CellDiff` can attribute every change. If you add a
@@ -92,6 +105,9 @@ DataFrame ─▶ profile.py ─▶ detectors/ ─▶ planner.py ─▶ Recipe (r
   unknown check name at load time. It lives apart from `validate.py` so `recipe.py`
   can use the guard without an import cycle.
 - **`llm.py`** — optional; writes a recipe from metadata only.
+- **`decisions.py`** — optional; asks a decision model (Jev, an open System One model,
+  or an LLM via an adapter) the judgment calls rules leave open, and turns confident
+  answers into ordinary proposals for the planner.
 
 The dependency direction is strictly downward (detectors import ops, planner
 imports detectors, etc.). Don't introduce an upward import — it will create a

{cleanframe_engine-0.4.0 → cleanframe_engine-0.5.1}/PKG-INFO

@@ -1,16 +1,19 @@
 Metadata-Version: 2.5
 Name: cleanframe-engine
-Version: 0.4.0
+Version: 0.5.1
 Summary: The reproducible data-cleaning engine for Python. AI writes the recipe once; the recipe runs forever — deterministic, diffable, reviewable.
 Project-URL: Homepage, https://github.com/inboxpraveen/Cleanframe
 Project-URL: Repository, https://github.com/inboxpraveen/Cleanframe
 Project-URL: Documentation, https://github.com/inboxpraveen/Cleanframe/wiki
 Project-URL: Issues, https://github.com/inboxpraveen/Cleanframe/issues
 Project-URL: Changelog, https://github.com/inboxpraveen/Cleanframe/blob/main/CHANGELOG.md
-Author: CleanFrame contributors
+Project-URL: Playground, https://inboxpraveen.github.io/Cleanframe/
+Project-URL: Funding, https://github.com/sponsors/inboxpraveen
+Author: Praveen Kumar
+Maintainer: Praveen Kumar
 License-Expression: Apache-2.0
 License-File: LICENSE
-Keywords: data-cleaning,data-quality,data-validation,etl,pandas,reproducible,schema-drift
+Keywords: ai-agents,csv,data-cleaning,data-quality,data-validation,decision-models,etl,excel,llm,mcp,pandas,reproducible,schema-drift
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3
@@ -30,6 +33,7 @@ Requires-Dist: pyyaml>=6.0
 Provides-Extra: all
 Requires-Dist: anthropic>=0.30; extra == 'all'
 Requires-Dist: charset-normalizer>=3; extra == 'all'
+Requires-Dist: mcp<3,>=2.3; extra == 'all'
 Requires-Dist: openai>=1.0; extra == 'all'
 Requires-Dist: openpyxl>=3.0; extra == 'all'
 Requires-Dist: pyarrow>=12.0; extra == 'all'
@@ -37,6 +41,7 @@ Provides-Extra: detect
 Requires-Dist: charset-normalizer>=3; extra == 'detect'
 Provides-Extra: dev
 Requires-Dist: hypothesis>=6.80; extra == 'dev'
+Requires-Dist: mcp<3,>=2.3; extra == 'dev'
 Requires-Dist: mypy>=1.8; extra == 'dev'
 Requires-Dist: openpyxl>=3.0; extra == 'dev'
 Requires-Dist: pytest-cov>=4.0; extra == 'dev'
@@ -48,23 +53,33 @@ Requires-Dist: openpyxl>=3.0; extra == 'excel'
 Provides-Extra: llm
 Requires-Dist: anthropic>=0.30; extra == 'llm'
 Requires-Dist: openai>=1.0; extra == 'llm'
+Provides-Extra: mcp
+Requires-Dist: mcp<3,>=2.3; extra == 'mcp'
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=12.0; extra == 'parquet'
 Description-Content-Type: text/markdown
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/inboxpraveen/Cleanframe/main/docs/assets/Logo.png" alt="CleanFrame logo" width="180" />
+  <img src="https://raw.githubusercontent.com/inboxpraveen/Cleanframe/main/docs/assets/Logo.svg" alt="CleanFrame logo" width="120" />
 </p>
 
 <h1 align="center">CleanFrame</h1>
 
 <p align="center">
   <b>The reproducible data-cleaning engine for Python.</b><br/>
-  AI writes the cleaning recipe once. The recipe runs forever — deterministic, diffable, reviewable.
+  AI decides once. The recipe runs forever — deterministic, diffable, reviewable.
 </p>
 
 <p align="center">
-  <img src="https://raw.githubusercontent.com/inboxpraveen/Cleanframe/main/docs/assets/Hero.png" width="auto" alt="CleanFrame"/>
+  <a href="https://inboxpraveen.github.io/Cleanframe/"><b>▶ Try it in your browser</b></a> (nothing to install, your file never leaves the tab)
+  &nbsp;·&nbsp; <code>pip install cleanframe-engine && cleanframe demo</code>
+</p>
+
+<p align="center">
+  <picture>
+    <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/inboxpraveen/Cleanframe/main/docs/assets/Hero-dark.png">
+    <img src="https://raw.githubusercontent.com/inboxpraveen/Cleanframe/main/docs/assets/Hero.png" width="100%" alt="Real output of cleanframe demo: a messy customers CSV becomes clean data through a reviewable YAML recipe. Changed cells are highlighted, a row with an invalid email is held in quarantine, and next month's renamed column stops the replay."/>
+  </picture>
 </p>
 
 <p align="center">
@@ -74,6 +89,8 @@ Description-Content-Type: text/markdown
   <img src="https://img.shields.io/badge/python-3.10%2B-blue" alt="Python"/>
   <img src="https://img.shields.io/badge/LLM-optional-success" alt="LLM optional"/>
   <a href="https://github.com/inboxpraveen/Cleanframe/wiki"><img src="https://img.shields.io/badge/docs-wiki-informational" alt="Wiki"/></a>
+  <a href="https://inboxpraveen.github.io/Cleanframe/"><img src="https://img.shields.io/badge/playground-in_browser-ff6f00" alt="Playground"/></a>
+  <a href="https://github.com/inboxpraveen/Cleanframe/blob/main/docs/decisions.md"><img src="https://img.shields.io/badge/decision_models-Jev--style-8a2be2" alt="Decision models"/></a>
 </p>
 
 ---
@@ -108,26 +125,31 @@ The plan compiles to deterministic pandas operations. Same input → same output
 
 ## 30-second demo
 
-No API key needed for this:
+No API key, no sample files to download. The demo data ships inside the package:
 
 ```bash
 pip install cleanframe-engine
-cleanframe report examples/messy_customers.csv
+cleanframe demo
 ```
 
-You get an HTML report: detected issues, quality score, column-by-column diagnosis. Then fix it:
+It profiles a messy customer file (HTML report + quality score), cleans it into a recipe, then replays
+that recipe on *next month's* file. That file has a renamed column and a new date format, so the replay
+**stops** and tells you why. Finally the demo drafts the patch that makes it replay cleanly. Every step
+prints the command to run on your own data.
+
+The same thing in Python:
 
 ```python
 import pandas as pd
 import cleanframe as cf
 
-df = pd.read_csv("examples/messy_customers.csv")
+df = pd.read_csv("cleanframe-demo/messy_customers.csv")      # written by `cleanframe demo`
 
 result = cf.clean(
     df,
-    target_schema="examples/customer.schema.yaml",  # or: cf.infer_schema(df)
-    llm="anthropic/claude-sonnet-4-6",              # optional — omit for rules-only
-    mode="review",                                  # review | auto | strict
+    target_schema="cleanframe-demo/customer.schema.yaml",    # or: cf.infer_schema(df)
+    mode="review",                                           # review | auto | strict
+    # decide="typesafe/jev-latest",   # optional decision model for judgment calls (TYPESAFE_API_KEY)
 )
 
 result.diff.show()                # cell-level before/after, git-diff style
@@ -157,6 +179,32 @@ If the new file's schema drifted (a renamed column, a new currency format), Clea
   Run `cleanframe suggest new_customers.csv --recipe <recipe.yaml> --update` to review a patch.
 ```
 
+## Decision models: AI makes the judgment calls, under a 0.9 bar
+
+Some cleaning steps are judgment calls, not parsing. Is `BLR` another name for `Bangalore`? Is `Banglore`
+a typo, or a different place? Does `none` in an *allergies* column mean "missing" or "no allergies"? Which
+column is the schema's `amount_inr`? Rules can't settle these safely. With `decide=`, a **decision model**
+answers each one as a typed question (pick an option, or yes/no) with a probability. That can be
+[Jev](https://typesafe.ai), an open "System One" model, or any LLM through an adapter.
+
+```bash
+cleanframe clean vendor.csv --decide typesafe/jev-latest --llm-exposure sample
+```
+
+- **An answer changes data only at p ≥ 0.9** (`decision_threshold`). Below that it is recorded as a
+  suggestion in the recipe and never executed. A confident "no, these differ" vetoes a typo merge the
+  rules would have made.
+- **Asked once, at planning time.** Accepted answers become ordinary recipe ops (`normalize_values`,
+  `to_na`, `rename_to`), and every answer is logged in `recipe.meta.decisions`. Replay makes zero AI calls.
+- **Privacy follows `llm_exposure`.** Category values are sent only under `sample`, and only for columns
+  whose values repeat. Dates and numbers are never asked: they stay with the deterministic parsers.
+- Backends: `typesafe/…`, `openrouter/typesafe/…`, self-hosted `systemone/…`, offline `rules`, or any
+  `--llm` spec. No SDK is needed.
+
+How good are decision models at this? **[CleanBench](https://github.com/inboxpraveen/Cleanframe/blob/main/bench/README.md)** measures it on hand-labelled messy
+columns full of traps, scoring *safe-apply precision*: of the answers that would change your data, how
+many are right. See [Decision models](https://github.com/inboxpraveen/Cleanframe/blob/main/docs/decisions.md).
+
 ## Bring your own API key (or no key at all)
 
 CleanFrame is LLM-optional and provider-agnostic.
@@ -167,6 +215,7 @@ CleanFrame is LLM-optional and provider-agnostic.
 | **Metadata** | LLM sees column names, dtypes, and *value patterns* (regex sketches) — never raw values | Only metadata |
 | **Sample** | LLM sees a small shuffled sample of *values*. Emails, phones and long strings are redacted; other short values (names, cities) are sent verbatim. Passing `llm_exposure="sample"` is the opt-in — there is no separate approval step | Sample values, per the redaction above |
 | **Replay** | Saved recipes | Never — recipes need no LLM |
+| **Decisions** (`decide=`) | Typed questions to a decision model while planning; same exposure rules | Metadata only by default; distinct category values under `sample` |
 
 - Keys come from environment variables (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`,
   `OPENROUTER_API_KEY`, `GROQ_API_KEY`, …). CleanFrame never stores or logs them, and
@@ -190,7 +239,7 @@ CleanFrame is LLM-optional and provider-agnostic.
 | Column name chaos | `Cust Name`, `customer_name`, `CustomerName` → `customer_name` |
 | Date formats | `12/01/24`, `1 Jan 2024`, `2024-01-01` → ISO dates |
 | Currency & numbers | `₹1,20,000`, `$1,200`, `-$5`, `(1,200)`, `1.234,56`, `1200 INR` → typed floats (+ a currency column); a value whose grouping is invalid or ambiguous becomes a *counted, warned* null, never a guess |
-| Category variants | `Bengaluru` / `bengaluru ` / `BENGALURU` and typos (`Banglore`) → one canonical value; abbreviations such as `BLR` need a seed map or an LLM |
+| Category variants | `Bengaluru` / `bengaluru ` / `BENGALURU` and typos (`Banglore`, one edit away) → one canonical value; look-alikes such as `Austria`/`Australia` stay apart; abbreviations such as `BLR` need a seed map or a [decision model](https://github.com/inboxpraveen/Cleanframe/blob/main/docs/decisions.md) |
 | Emails & phones | validation, normalization, `00`/`+` international prefixes, optional default country code |
 | Duplicates | exact duplicates are dropped (tracked in the diff); fuzzy near-duplicates and key collisions are *reported* with keep/drop proposals for you to act on |
 | Missing values | detected and explained; strategies proposed, never silently applied |
@@ -218,9 +267,44 @@ CleanFrame is built for pipelines, not just demos:
 - **Strict recipes** — unknown ops/params, duplicate keys and bad regexes fail at *load*; plain YAML keys like `yes`, `010` or `12:30` stay column names
 - **Generated code = executor** — exported pandas embeds the executor's own parsers, and property-based tests check they agree
 - **Plugins that travel** — custom ops/detectors load from entry points, `--plugin` or `CLEANFRAME_PLUGINS`, and can opt in to code export and streaming
+- **Gated AI decisions** — a decision model's answer changes data only at p ≥ 0.9, every answer is logged in the recipe, and replay never calls the model
+- **pandas 1.5 → 3.x** — CI runs the minimum pins, pandas 2 and pandas 3
 
 See the [Production Guide](https://github.com/inboxpraveen/Cleanframe/wiki/Production-Guide).
 
+## Data CI on GitHub
+
+Fail a pull request when a data file drifts from its committed recipe:
+
+```yaml
+- uses: actions/checkout@v7
+- uses: inboxpraveen/Cleanframe/action@v0.5.1
+  with:
+    files: data/incoming/*.csv
+    recipe: recipes/customer.recipe.yaml
+```
+
+Every file is replayed, the job summary gets a per-file table, and the check fails on schema drift
+(exit 3) or a failing validation (exit 4). See [GitHub Action](https://github.com/inboxpraveen/Cleanframe/blob/main/docs/github-action.md).
+
+## Use it from your AI agent
+
+Coding agents love to "just fix the CSV" with throwaway pandas that nobody reviews and nothing replays.
+Give them CleanFrame instead, and they write a **recipe**: reviewed in git, replayed deterministically,
+stopped by drift.
+
+```bash
+npx skills add inboxpraveen/Cleanframe          # Agent Skill: Claude Code, Codex, Cursor, …
+pip install "cleanframe-engine[mcp]"            # MCP server: profile, suggest, apply, drift, explain, export
+claude mcp add cleanframe -- cleanframe-mcp
+```
+
+See [AI agents](https://github.com/inboxpraveen/Cleanframe/blob/main/docs/agents.md) for Claude Desktop, Cursor, VS Code and Codex configs.
+Tools: `profile_data`, `suggest_recipe`, `apply_recipe`, `check_drift`, `explain_recipe`, `list_capabilities`,
+`export_code`. There is no LLM inside the server, the input file is never overwritten, and drift stops the replay.
+
+<!-- mcp-name: io.github.inboxpraveen/cleanframe -->
+
 ## The recipe: your durable artifact
 
 ```yaml
@@ -266,6 +350,7 @@ not against it.
 | Works with zero LLM / offline | ✅ | ❌ | ✅ | ✅ | ✅ | ⚠️ |
 | Cell-level diff & lineage | ✅ | ❌ | ❌ | ❌ | ⚠️ | ⚠️ |
 | Schema-drift alerts on re-import | ✅ | ❌ | ⚠️ | ❌ | ❌ | ✅ ($6k+/yr) |
+| AI judgment calls gated by probability, frozen for replay | ✅ | ❌ | ❌ | ❌ | ❌ | ⚠️ |
 | Free & open source | ✅ Apache-2.0 | ✅ | ✅ | ✅ | ✅ | ❌ |
 
 Use PandasAI to *explore*. Use pandera/GX to *guard*. Use **CleanFrame to fix — repeatably.**
@@ -276,10 +361,11 @@ Use PandasAI to *explore*. Use pandera/GX to *guard*. Use **CleanFrame to fix
 CSV / Excel / DataFrame
    │
    ▼
-Profiler ─→ Issue Detectors ─→ Planner ──────────→ Recipe (YAML)
-             (deterministic)    (rules, or            │
-                                LLM-assisted)         ▼
-                                              Executor (pure pandas)
+Profiler ─→ Issue Detectors ─→ [Decisions] ─→ Planner ──────────→ Recipe (YAML)
+             (deterministic)    (optional:      (rules, or            │
+                                 typed Qs to     LLM-assisted)        ▼
+                                 Jev / an LLM,                Executor (pure pandas)
+                                 gated p ≥ 0.9)
                                                       │
                                                       ▼
                                         Validator · Cell-diff · HTML report
@@ -307,7 +393,9 @@ pip install cleanframe-engine
 pip install "cleanframe-engine[excel]"      # Excel
 pip install "cleanframe-engine[parquet]"    # Parquet
 pip install "cleanframe-engine[llm]"        # Anthropic + OpenAI SDKs
+pip install "cleanframe-engine[mcp]"        # MCP server for AI agents
 pip install "cleanframe-engine[all]"
+# decision models (decide=) need no extra: the System One protocol uses the standard library
 
 # straight from git
 pip install "cleanframe-engine @ git+https://github.com/inboxpraveen/Cleanframe"
@@ -329,7 +417,10 @@ Python 3.10+. The distribution is [`cleanframe-engine`](https://pypi.org/project
 - [x] Read-time format auto-correction (encoding + delimiter)
 - [x] Plugin discovery; custom ops that export and stream
 - [x] `--json` run summaries, library logging, property-based invariant tests
-- [ ] `MessyData-100` public benchmark + leaderboard
+- [x] Decision models (`decide=`): typed, probability-gated judgment calls frozen into recipes
+- [x] CleanBench: a public benchmark and leaderboard for data-cleaning decisions
+- [x] MCP server + Agent Skill; GitHub Action for data CI; in-browser playground; `cleanframe demo`
+- [ ] Decision questions for more judgment calls (unit families, near-duplicate rows), with per-question calibration from CleanBench
 - [ ] pandera / GX / dbt exporters
 - [ ] Polars backend
 - [ ] Recipe registry for teams
@@ -340,7 +431,20 @@ The detector plugin system exists so the community owns the long tail of messy-d
 
 ## Sponsoring
 
-CleanFrame is free, Apache-2.0, and will stay that way for individuals — no feature is paywalled for a person with a laptop and a messy CSV. If it saves your team recurring hours, [sponsorship](https://github.com/sponsors/inboxpraveen) funds detector coverage, the benchmark, and long-term maintenance.
+CleanFrame is free, Apache-2.0, and will stay that way. No feature is paywalled for a person with a laptop
+and a messy CSV. If it saves your team recurring hours, [sponsorship](https://github.com/sponsors/inboxpraveen)
+funds detector coverage, CleanBench runs on new models, and fast fixes for anything that costs you data.
+
+| Tier | What you get |
+|---|---|
+| **$5/mo** | Your name in the sponsors list |
+| **$25/mo** | Your name and a link in this README; issues you open are looked at first |
+| **$100/mo**: retainer, patch on demand | I build the recipe for one recurring file (a vendor export, a settlement report, a CRM dump). You own it and run it; when the format changes, I patch it within 3 business days |
+| **$150/mo**: company sponsor | A short section near the top of this README with your logo, a link and two or three lines you write, plus a mention in the release notes |
+| **$200/mo**: retainer, monthly run | Patch on demand, plus I run the recipe on each new file and send the cleaned output with a cell-level diff, and one custom detector per quarter. Your data stays on my machine, is never shared or used for training, and is deleted after 30 days |
+| **$250 one-time**: recipe build | Send one file. You get a schema and a complete recipe committed to your repo, the cleaned output with a report and a diff of every changed cell, a 30-minute handover call, and 30 days of support |
+
+The [Sponsors page](https://github.com/sponsors/inboxpraveen) has the current tiers and terms.
 
 ---