cleanframe-engine 0.3.0__tar.gz → 0.4.0__tar.gz

{cleanframe_engine-0.3.0 → cleanframe_engine-0.4.0}/CHANGELOG.md

@@ -7,6 +7,162 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.4.0] — 2026-09-30
+
+A correctness release. An empirical review (26 hard-input probes, a 1M/3M-row benchmark and a
+docs audit that ran every example) found places where CleanFrame produced a **wrong number, a
+wrong merge or a wrong file with no warning**. Every one is fixed, each has a regression test, and
+the invariants that should have caught them now have property-based tests.
+
+### Fixed — silent wrong results
+
+- **A new number format no longer replays silently.** Replaying a `.`-decimal recipe on `€1.200,50`
+  produced `1.2005`, exit 0, no warning. Numbers are now parsed by structure (below), and `apply_recipe`
+  raises a `number_format_drift` finding (exit 3) when digit-bearing values stop fitting the recipe's
+  `decimal`/`thousands` convention (a small share is INFO, so a stray `12ab34` is not a stop).
+- **Signs were dropped.** `-$5`, `-€3.50`, `-₹500` and `$(8)` parsed as positives. `parse_number` now finds
+  the sign wherever it is written (`-5`, `-$5`, `$-5`, `5-`, `(5)`, `$(5)`, unicode minus).
+- **`€2,5` became 25 and `€999,99` became 99999** whenever one value in the column looked US-grouped.
+  Grouping is now validated (3-digit or Indian lakh) and the currency detector infers comma-decimal only
+  from unambiguous evidence; values that fit neither convention become a counted, warned null.
+- **The category detector merged different numbers**, in every mode including `strict`: `-1,200`→`1,200`,
+  `12.00`→`1,200`, `inf`→`-inf`, `A1`→`A-1`. Values that differ in sign, decimal or grouping are never one category.
+- **`normalize_unit` read `1,500 g` as 1.5 g** (1000x off). Ambiguous numbers are refused (null + warning);
+  `1,5 kg` and `0.500 kg` still parse, and `1.5e3 g` now does too.
+- **`cast: int` rounded IDs above 2**53** through float64; integer text is now converted exactly.
+- **The cell diff missed precision loss** (`9007199254740993` → `...992.0`) and `1` → `True`, because numpy
+  compared `int64` with `float64` after promotion. Both are now reported.
+- **`R$`, `A$`, `C$`, `HK$`, `NZ$`, `S$` were all labelled USD**: symbols are matched longest-first, and `₺ ₫ ₱ ₪ ฿` were added.
+- **Auto-dedup treated `1`, `True` and `1.0` as the same value**, and collapsed a one-column frame to its
+  distinct values. Booleans stay distinct from numbers, and one-column frames are reported, not deduplicated.
+- **Offset timestamps were shifted to the UTC calendar day** (`2024-01-01T02:00+05:30` → `2023-12-31`); a
+  single-offset column now keeps its wall-clock date.
+- **`title_case` produced `3Rd Street` and `Don'T`**, and **`normalize_phone` turned `0044…` into `+91442…` and `12345` into
+  `+9112345`**. Ordinals and apostrophes are handled; `00` is read as `+`; numbers too short to be phones never get a country code invented.
+- **Any non-UTF-8 file was accepted as latin-1** (Shift-JIS became mojibake with exit 0). The encoding ladder
+  can now refuse: a file that is neither UTF-8, cp1252 nor plausibly a single-byte Latin text raises with the likely
+  encodings and `--encoding` advice.
+- **`read_frame` silently truncated `a\x00b` to `a`**; it now raises like `clean`.
+- **Generated code diverged from the executor**: `05/01/2024` read as Jan 5 instead of May 1, offset timestamps
+  crashed it, `to_na case_insensitive: false` was ignored, `normalize_values` skipped non-string cells, integer column
+  labels and missing columns raised `KeyError`. Exported pandas now **embeds the executor's own helper source**
+  for numbers, units, ints, dates and title case, and skips missing columns like the executor.
+- **`stream_apply` leaked raw pandas errors** on a late chunk that did not fit the recorded dtypes (exit 70), and its
+  output could differ from `apply_recipe` (`2` vs `2.0`, drift missed beyond the first 200 rows). It now makes a first
+  bounded-memory pass, checks drift across every row, and writes byte-identical output or refuses with a named error.
+  Streaming also stops writing a quarantine file nobody asked for.
+- **`suggest --update` "learned" a competing reading of dates that already parse** (`03/03/2026` → `%m/%d/%Y`) and could
+  not fix the README's own `Jan 5, 26` example. It now learns only from values the recipe cannot parse.
+- **YAML 1.1 traps in hand-written recipes.** `yes:`, `no:`, `null:`, `010:`, `12:30:` keys were read as `True`, `False`,
+  `None`, `8`, `750` and the column was skipped with only a warning. Plain keys now stay text; `to_na: [no, null]` is refused
+  with a "quote it" hint.
+- **`cleanframe clean` silently overwrote a hand-edited recipe.** It now refuses to replace a recipe that differs from
+  the one just planned unless `--overwrite` is given (re-running over unchanged input is still fine).
+- `Recipe.from_yaml(r.to_yaml()) == r` was `False` for every generated recipe: equality now compares the canonical serialisation.
+- **Money in `1.234` shape is never guessed.** With no other evidence `€1.234` could be 1234 (German) or 1.234 (Irish); the currency detector now reports `ambiguous_number_format` and proposes nothing, instead of picking one and being 1000x wrong.
+- **Currency codes:** an explicit code beats a symbol (`$5 MXN` is pesos), `$`-family symbols glued to letters are no longer mislabelled (`AR$`, `MX$`, `NT$`, `CL$`, `CO$` added; an unknown `XY$5` is "no currency", not USD), `Rs` is INR, and more ISO codes are recognised.
+- **`normalize_unit` recognises the spellings people write** (`lbs`, `pounds`, `kgs`, `ounces`, `inches`, `feet`, `gallons`, …).
+- **A deliberate `to_na` was reported as "values became missing because an op could not parse them"**; it is now logged as intentional and only real parse failures warn.
+- **A derived column could be named `None_currency`** when `extract_currency` followed an op that returned an unnamed Series; op results keep the column name.
+- **`stream_apply` silently dropped the extra field of a row wider than the header** (pandas' chunked reader truncates it); streaming now refuses such a file, as `apply_recipe` does.
+- `dedup` / `drop_columns` accept numeric-looking column names (`subset: [2024]`); `on_fail: null` explains the YAML trap; the Excel writer warns when it removes illegal control characters; `read_frame` on a `;`-delimited file with commas in the text says "semicolon-delimited" instead of suggesting `header_row`; a lone `-`/`+` is no longer formula-escaped; a percent or currency symbol keeps `10 %` distinct from `10` in category clustering.
+
+### Security & trust
+
+- **`OPENAI_API_KEY` is no longer sent to other providers.** Each provider reads only its own key variable.
+- **LLM plans can never contain `fill_na` or `drop_columns`** (any mode); they are stripped with a warning, closing a
+  hole in the "nothing silently imputed or dropped" invariant. An LLM recipe is also dry-run on the first rows and falls back to the rules planner if it cannot run.
+- `sample` exposure is documented as what it is (values partly redacted, no approval step) and warns which columns are sent; `llm_exposure` typos raise a clear error; `max_tokens_budget` without `llm=` warns.
+- ReDoS-prone `replace` patterns and regex back-references to missing groups are refused **at load**.
+
+### Added
+
+- **Plugins that travel.** `cleanframe.plugins`: entry points (`cleanframe.plugins` group), `CLEANFRAME_PLUGINS`,
+  `--plugin MODULE`, `cf.load_plugins()`, and lazy discovery when a recipe names an unknown op.
+  `register_op(codegen=, helpers=, streamable=)` lets a custom op export as standalone pandas and stream (default-deny). See the new plugin guide.
+- **`--json`**: one machine-readable summary object on stdout (human output to stderr) for every command, on success and failure — including usage errors (`status: usage_error`, exit 2), and `ops`/`detectors` list themselves. A workbook `apply --chunksize` is now refused instead of ignored.
+- **`normalize_unicode` op + `unicode` detector**: NFC-normalises text and removes zero-width spaces, word joiners, BOMs and non-breaking spaces (zero-width joiners are kept). It fixes `café` typed two ways splitting a category, and invisible characters that `strip` leaves behind.
+- **Logging**: the library logs through `logging.getLogger("cleanframe")` (silent `NullHandler`); `--verbose` enables INFO.
+- **`header_row=` / `--header-row`** for files with title rows above the header, with a suggested value when the header looks misplaced; read-time warnings for ragged rows and `Total` footer rows; hidden Excel sheets stay hidden on write-back; warnings for uncached formulas and merged cells.
+- **Property-based tests (Hypothesis)** for the parser, generated-code parity, determinism, recipe round-trip, "nothing silently dropped" and streaming parity; `scripts/sync_wiki.py` + a test that keeps `docs/` and `wiki/` identical.
+- More date formats (`%b %d, %y`, `%B %d, %y`, `%d %b, %Y`, `%d.%m.%y`, `%Y.%m.%d`) — appended, so existing recipes are unaffected.
+- A `detect` extra (`charset-normalizer`) that ranks candidate encodings in the refusal message.
+- Unknown `options=` keys warn with a did-you-mean.
+
+### Performance
+
+- Pure text ops (`parse_number`, `normalize_unit`, `normalize_phone`, `normalize_values`, `extract_currency`, explicit-format `parse_date`, and the email/url/phone/regex validators) now run **once per distinct value** of an all-string column and fan the results out — identical output, 15-25x faster on repetitive columns (1M rows: `parse_number` 4.5 s → 0.17 s, `normalize_unit` 3.8 s → 0.26 s). Measured end to end on a realistic 18-column, 1M-row file: `clean()` 42 s → 29 s, replay 28 s → 16 s.
+- Peak memory is unchanged and is now documented honestly: the frame plus the op-touched columns, about 2x when most columns are cleaned. Use `stream_apply` for row-independent recipes on files that do not fit.
+
+### Changed
+
+- **Behaviour changes worth knowing when upgrading:**
+  - Numbers with invalid or ambiguous grouping now become a warned null instead of a guess (`2,5` under `thousands=","`, `1,500 g`).
+  - Streaming reads the file twice, and writes a quarantine file only when `quarantine_path` is given.
+  - Offset timestamps keep their wall-clock date; mixed-offset columns are converted to UTC.
+  - `title_case` no longer capitalises after apostrophes or digits; `normalize_phone` treats `00` as `+`.
+  - `to_na` tokens, `replace` pattern/repl and `parse_number` separators must be text; `version: true` and unknown option keys are rejected.
+  - `Recipe` defines `__eq__` on its canonical form (it was already unhashable).
+- Streaming `apply` measured at ~270 MB peak for 600k rows at `chunksize=50_000` (docs updated with the baseline).
+- The validation check registry moved into `cleanframe.checks` (internal; removes the last import cycle).
+
+## [0.3.1] — 2026-09-04
+
+First follow-up to the 0.3.0 release, from the CI and code-scanning results that
+only appear once a project is public.
+
+### Fixed
+
+- **`Severity` comparisons were alphabetical, not ordered.** Only `__lt__` was
+  defined, so the `str` mixin answered everything else: `Severity.ERROR >
+  Severity.INFO` returned `False` because `"error" < "info"` as text. All four
+  comparisons are now defined on the severity rank. Library code always used
+  `.rank` explicitly, so nothing internal was affected, but any caller comparing
+  severities directly got wrong answers. Found by CodeQL's incomplete-ordering
+  query.
+- **`mypy` failed in CI** with `numpy/__init__.pyi:737: error: Type statement is
+  only supported in Python 3.12 and greater`. The `python_version = "3.10"` pin
+  also governs how dependency stubs are parsed, and numpy 2.5's stubs use syntax
+  that needs 3.12. Removed the pin; minimum-version support is still checked by
+  ruff's `target-version` and by the Python 3.10 test job.
+- **The wiki sync workflow could not push.** GitHub Actions' built-in token can
+  clone a wiki but not write to it, so the job committed and then failed on
+  authentication. It now requires the `WIKI_TOKEN` secret and stops up front with
+  an explanatory message. `wiki/README.md` claimed no token was needed; corrected.
+- Two module-level import cycles removed: `profile` with `ops`, and `recipe` with
+  `validate`. `COMMON_DATE_FORMATS` now lives in `ops`, which owns date parsing,
+  and is re-exported from `profile` so every existing import keeps working.
+  `validate` imports `ValidationRule` for annotations only.
+- Empty exception handlers in `fingerprint` and `report` replaced with a helper
+  that returns whether a cell is missing, so the intent is in the code rather
+  than in a comment beside `pass`.
+- Redundant function-level `import re` in the dates detector removed; it now
+  reuses the module's compiled pattern instead of recompiling per call.
+
+### Changed
+
+- Workflow actions updated: checkout to v7, setup-python to v7, upload-artifact
+  to v7, download-artifact to v8, codeql-action to v4. Clears the Node 20
+  deprecation warnings.
+- Dependabot groups the `github-actions` ecosystem, so action bumps arrive as one
+  pull request instead of one per action.
+- CodeQL analyses the library and skips `tests`. Test fixtures are deliberately
+  hostile (catastrophic-backtracking patterns, injection payloads, handlers that
+  assert something must not raise) and are not part of the wheel, so reporting
+  them buries real findings. Configuration lives in
+  `.github/codeql/codeql-config.yml`.
+- Protocol methods on `LLMClient` and `Planner` carry a docstring instead of a
+  bare `...`.
+- Installation docs link the package page, show how to pin a version, and mention
+  `cleanframe --version`. `CHANGELOG.md` gained the Keep a Changelog link
+  definitions now that releases are tagged.
+
+### Verified
+
+335 tests pass on Python 3.10 with the declared minimum pins (pandas 1.5.3,
+numpy 1.23.5), on Python 3.13 with pandas 2.3.3, and on Python 3.13 with pandas
+3.0.5 and numpy 2.5.2, which is what a fresh `pip install` resolves today.
+
 ## [0.3.0] — 2026-09-04
 
 Release-readiness pass driven by a full audit of the CLI, the Python API, packaging and
@@ -233,3 +389,9 @@ now fail loudly, and a few transforms that quietly corrupted data no longer run.
 - Initial public release: profiler, detectors, rules + optional LLM planner, recipe YAML,
   deterministic executor, validation/quarantine, cell-level diff, schema drift, HTML reports,
   codegen, and CLI.
+
+[Unreleased]: https://github.com/inboxpraveen/Cleanframe/compare/v0.3.1...HEAD
+[0.3.1]: https://github.com/inboxpraveen/Cleanframe/releases/tag/v0.3.1
+[0.3.0]: https://github.com/inboxpraveen/Cleanframe/releases/tag/v0.3.0
+[0.2.0]: https://github.com/inboxpraveen/Cleanframe/commits/main
+[0.1.0]: https://github.com/inboxpraveen/Cleanframe/commits/main

{cleanframe_engine-0.3.0 → cleanframe_engine-0.4.0}/CONTRIBUTING.md

@@ -9,7 +9,7 @@ than no change at all. The rules below exist to protect it.
 
 ---
 
-## The five invariants (do not break these)
+## The six invariants (do not break these)
 
 1. **Determinism.** The same input must always produce the same recipe, the same
    cleaned frame, the same diff — on any machine, in any process. That means:
@@ -43,11 +43,25 @@ than no change at all. The rules below exist to protect it.
    name it in the op's signature (or in `aliases=`) so the loader accepts it — a
    parameter the loader silently ignores is a data-loss bug.
 
+   It applies to *guessing* as well: when a value is ambiguous (`1,500 g` — 1500 g or
+   1.5 g?) or malformed for the declared convention (`2,5` under `thousands=","`), the
+   result is a counted, warned null, never a plausible-looking number. Wrong numbers
+   are worse than missing ones. Likewise a clustering/merge rule must never fold together
+   values that differ in sign, decimal or grouping (`-300`/`300`).
+
 5. **Every changed cell is tracked.** The executor assigns a stable row id and
    tracks column lineage so `CellDiff` can attribute every change. If you add a
    transform that adds/removes/reorders columns or rows, make sure the lineage in
    `executor.py` stays correct and the diff still reconciles.
 
+6. **Exported code equals the executor — because it is the same code.** The pandas that
+   `result.code` writes must produce exactly what `apply_recipe` produces. Do not
+   re-implement an op's logic in `codegen.py`: put shared scalar logic in a small
+   self-contained module (`_numparse.py`, `_textparse.py`) or an existing `ops.py` helper
+   and *embed its source* with `inspect.getsource`, as codegen already does. Two
+   implementations drift; one implementation cannot. `tests/test_properties.py` fuzzes
+   this, and `tests/test_fix_numbers_and_parity.py` pins the known cases.
+
 If a change would weaken one of these, open an issue to discuss it first.
 
 ---
@@ -74,6 +88,9 @@ DataFrame ─▶ profile.py ─▶ detectors/ ─▶ planner.py ─▶ Recipe (r
   renames, and — critically — orders ops via `OP_ORDER`.
 - **`executor.py`** — replays a recipe in fixed phases with lineage tracking.
 - **`recipe.py` / `schema.py`** — the durable, human-reviewable artifacts.
+- **`checks.py`** — the validation check registry, plus the guard that rejects an
+  unknown check name at load time. It lives apart from `validate.py` so `recipe.py`
+  can use the guard without an import cycle.
 - **`llm.py`** — optional; writes a recipe from metadata only.
 
 The dependency direction is strictly downward (detectors import ops, planner
@@ -218,6 +235,14 @@ you.
 - **Every PR keeps `pytest` green.** New behavior needs a new test.
 - **Test the invariant, not just the happy path.** For anything order-sensitive, add
   a determinism assertion. For a new op with params, add the round-trip.
+- **Fuzz the invariants.** `tests/test_properties.py` uses [Hypothesis](https://hypothesis.readthedocs.io/)
+  to check parser totality/sign, generated-code == executor, determinism, recipe round-trip,
+  "nothing silently dropped" and streaming == whole-frame. CI runs a small deterministic
+  sample; to hunt for new bugs locally run
+  `CLEANFRAME_PROP_EXAMPLES=2000 CLEANFRAME_PROP_RANDOM=1 pytest tests/test_properties.py`.
+  If it finds one, add the minimal case as a normal example test as well.
+- **New op?** Decide `codegen=` / `streamable=` deliberately (see the [plugin guide](docs/plugins.md));
+  both default to "refuse", which is safe.
 - Prefer small, focused tests using the fixtures in `tests/conftest.py`.
 - `ruff check cleanframe tests` and `mypy` must both pass — CI runs them.
 
@@ -227,8 +252,9 @@ you.
   docstrings that explain *why* (the reader can see *what*).
 - Target Python 3.10+. Keep the core dependencies minimal (pandas, numpy, pyyaml,
   jinja2). Anything heavier goes in an optional extra in `pyproject.toml`.
-- No `print` in library code — return data or `log` to a list. `print` is for the
-  CLI only.
+- No `print` in library code — return data, add to the run `log`, or use
+  `logging.getLogger(__name__)` (the package installs a `NullHandler`). `print` is for the
+  CLI only, and under `--json` the CLI keeps stdout for the one JSON summary object.
 
 ## Pull requests
 

{cleanframe_engine-0.3.0 → cleanframe_engine-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: cleanframe-engine
-Version: 0.3.0
+Version: 0.4.0
 Summary: The reproducible data-cleaning engine for Python. AI writes the recipe once; the recipe runs forever — deterministic, diffable, reviewable.
 Project-URL: Homepage, https://github.com/inboxpraveen/Cleanframe
 Project-URL: Repository, https://github.com/inboxpraveen/Cleanframe
@@ -29,10 +29,14 @@ Requires-Dist: python-dateutil>=2.8
 Requires-Dist: pyyaml>=6.0
 Provides-Extra: all
 Requires-Dist: anthropic>=0.30; extra == 'all'
+Requires-Dist: charset-normalizer>=3; extra == 'all'
 Requires-Dist: openai>=1.0; extra == 'all'
 Requires-Dist: openpyxl>=3.0; extra == 'all'
 Requires-Dist: pyarrow>=12.0; extra == 'all'
+Provides-Extra: detect
+Requires-Dist: charset-normalizer>=3; extra == 'detect'
 Provides-Extra: dev
+Requires-Dist: hypothesis>=6.80; extra == 'dev'
 Requires-Dist: mypy>=1.8; extra == 'dev'
 Requires-Dist: openpyxl>=3.0; extra == 'dev'
 Requires-Dist: pytest-cov>=4.0; extra == 'dev'
@@ -147,8 +151,10 @@ If the new file's schema drifted (a renamed column, a new currency format), Clea
 ```
 ⚠ Schema drift detected in new_customers.csv
   • Column "Amt (INR)" is new — 94% match to recipe column "amount_inr"
-  • 312 values in "signup_date" match no allowed date format (new: "Jan 5, 26")
-  Run `cleanframe suggest new_customers.csv --recipe customer.recipe.yaml --update` to review a patch.
+  • 312 value(s) in "signup_date" match no allowed date format (new: 'Jan 5, 26')
+  • 40 of 312 numeric value(s) in "Amount" do not fit the recipe's number format
+    (decimal '.', thousands ','; new: '€1.200,50')
+  Run `cleanframe suggest new_customers.csv --recipe <recipe.yaml> --update` to review a patch.
 ```
 
 ## Bring your own API key (or no key at all)
@@ -159,30 +165,34 @@ CleanFrame is LLM-optional and provider-agnostic.
 |---|---|---|
 | **Rules-only** (default) | Deterministic detectors + heuristics | Never |
 | **Metadata** | LLM sees column names, dtypes, and *value patterns* (regex sketches) — never raw values | Only metadata |
-| **Sample** | LLM sees an anonymized, shuffled sample you approve | Only the approved sample |
+| **Sample** | LLM sees a small shuffled sample of *values*. Emails, phones and long strings are redacted; other short values (names, cities) are sent verbatim. Passing `llm_exposure="sample"` is the opt-in — there is no separate approval step | Sample values, per the redaction above |
 | **Replay** | Saved recipes | Never — recipes need no LLM |
 
 - Keys come from environment variables (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`,
-  `OPENROUTER_API_KEY`, `GROQ_API_KEY`, …). CleanFrame never stores, logs, or
-  transmits them.
+  `OPENROUTER_API_KEY`, `GROQ_API_KEY`, …). CleanFrame never stores or logs them, and
+  each provider reads **only its own** variable — `OPENAI_API_KEY` is never sent to
+  another provider.
 - Any provider that speaks OpenAI Chat Completions works out of the box —
   Anthropic (native), OpenAI, OpenRouter, Groq, Together, Fireworks, DeepSeek,
   Mistral, Google Gemini, xAI, Perplexity, Cohere, plus local Ollama / LM Studio
   or any custom `OPENAI_BASE_URL`. Spec format: `provider/model`
   (e.g. `openrouter/anthropic/claude-sonnet-4`, `groq/llama-3.3-70b-versatile`).
-- Hard cost cap: `cf.clean(..., max_tokens_budget=50_000)` aborts planning before it gets expensive.
+- Cost cap: `cf.clean(..., max_tokens_budget=50_000)` is checked before each call (against an
+  estimate) and again against the usage the provider reports — leave headroom, since a call
+  that overshoots has already been billed.
 - Fully offline / air-gapped operation is a supported first-class mode, not a degraded one.
 
 ## What it cleans
 
 | Problem | Example |
 |---|---|
+| Invisible characters | zero-width spaces, BOMs, non-breaking spaces and decomposed accents (`café` typed two ways) → one normalised form |
 | Column name chaos | `Cust Name`, `customer_name`, `CustomerName` → `customer_name` |
 | Date formats | `12/01/24`, `1 Jan 2024`, `2024-01-01` → ISO dates |
-| Currency & numbers | `₹1,20,000`, `$1,200`, `1200 INR` → typed floats + currency column |
-| Category variants | `Bengaluru` / `Bangalore` / `BLR` → one canonical value |
-| Emails & phones | validation, normalization, country codes |
-| Duplicates | exact + fuzzy matching with reviewable merge proposals |
+| Currency & numbers | `₹1,20,000`, `$1,200`, `-$5`, `(1,200)`, `1.234,56`, `1200 INR` → typed floats (+ a currency column); a value whose grouping is invalid or ambiguous becomes a *counted, warned* null, never a guess |
+| Category variants | `Bengaluru` / `bengaluru ` / `BENGALURU` and typos (`Banglore`) → one canonical value; abbreviations such as `BLR` need a seed map or an LLM |
+| Emails & phones | validation, normalization, `00`/`+` international prefixes, optional default country code |
+| Duplicates | exact duplicates are dropped (tracked in the diff); fuzzy near-duplicates and key collisions are *reported* with keep/drop proposals for you to act on |
 | Missing values | detected and explained; strategies proposed, never silently applied |
 | Units | `5kg`, `5000 g`, `5 KG` → normalized |
 | Schema mapping | messy file → your target schema, with confidence scores |
@@ -196,14 +206,18 @@ CleanFrame is built for pipelines, not just demos:
 - **Multi-sheet workbooks** — clean every Excel tab into one reviewable recipe; write-back refuses to overwrite the source in place
 - **Out-of-core streaming** — replay row-independent recipes over larger-than-RAM CSVs at chunk-bounded memory
 - **Format auto-detection** — encoding (utf-8 → cp1252) and delimiter sniffed at read time, pinned into the recipe for replay
-- **Bounded memory** — detector sampling (50k values) + capped cell-diff detail (100k); the diff snapshots only op-touched columns (peak ≈ input, not 2× the frame)
+- **Bounded memory** — detector sampling (50k values) + capped cell-diff detail (100k); the diff snapshots the op-touched columns (peak is the frame plus those columns — about 2× when most columns are cleaned; use streaming for files that do not fit)
 - **Safe CSV exports** — spreadsheet formula injection escaped by default
 - **Regex guards** — oversized / nested-quantifier patterns rejected
 - **Visible degradation** — missing columns, unparseable values and LLM fallbacks warn instead of failing silently
 - **Quarantine, don't delete** — validation failures are held with reasons
 - **Verbatim reads** — `text=True` keeps leading zeros, literal `NA` and `1e5` exactly as the file has them; otherwise CleanFrame warns about what pandas' type inference changed on read
 - **Never overwrites its input** — writing output over the source file takes an explicit `--overwrite`, and every write lands via a temporary file
-- **Scriptable exits** — distinct exit codes for usage errors, data errors, drift stops and validation failures
+- **Scriptable exits** — distinct exit codes for usage errors, data errors, drift stops and validation failures; `--json` prints one machine-readable summary on stdout
+- **Number-format drift** — a file whose numbers stop fitting the recipe (`€1.200,50` against a `.`-decimal recipe) stops the replay instead of yielding `1.2005`
+- **Strict recipes** — unknown ops/params, duplicate keys and bad regexes fail at *load*; plain YAML keys like `yes`, `010` or `12:30` stay column names
+- **Generated code = executor** — exported pandas embeds the executor's own parsers, and property-based tests check they agree
+- **Plugins that travel** — custom ops/detectors load from entry points, `--plugin` or `CLEANFRAME_PLUGINS`, and can opt in to code export and streaming
 
 See the [Production Guide](https://github.com/inboxpraveen/Cleanframe/wiki/Production-Guide).
 
@@ -271,13 +285,21 @@ Profiler ─→ Issue Detectors ─→ Planner ──────────→
                                         Validator · Cell-diff · HTML report
 ```
 
-Detectors are plugins — write your own in ~30 lines:
+Detectors and ops are plugins — write your own in ~30 lines:
 
 ```python
+@cf.register_op("normalize_iban", streamable=True,
+                codegen=lambda p, col: [f"df[{col!r}] = df[{col!r}].str.replace(' ', '').str.upper()"])
+def normalize_iban(series: pd.Series) -> pd.Series:
+    return series.str.replace(" ", "").str.upper()
+
 @cf.detector("iban")
-def detect_iban(series: pd.Series) -> cf.Issues: ...
+def detect_iban(series: pd.Series) -> cf.Issues: ...   # propose cf.Op("normalize_iban")
 ```
 
+Package it under the `cleanframe.plugins` entry-point group and `cleanframe apply` in CI
+finds it — see the [plugin guide](https://github.com/inboxpraveen/Cleanframe/wiki/Plugins).
+
 ## Install
 
 ```bash
@@ -291,7 +313,7 @@ pip install "cleanframe-engine[all]"
 pip install "cleanframe-engine @ git+https://github.com/inboxpraveen/Cleanframe"
 ```
 
-Python 3.10+. The distribution is `cleanframe-engine`; the import package is
+Python 3.10+. The distribution is [`cleanframe-engine`](https://pypi.org/project/cleanframe-engine/); the import package is
 `cleanframe` (`import cleanframe as cf`), and the CLI is `cleanframe` (or
 `python -m cleanframe`).
 
@@ -305,6 +327,8 @@ Python 3.10+. The distribution is `cleanframe-engine`; the import package is
 - [x] Selective ingestion (sheet / columns / rows)
 - [x] Out-of-core streaming replay (larger-than-RAM CSVs)
 - [x] Read-time format auto-correction (encoding + delimiter)
+- [x] Plugin discovery; custom ops that export and stream
+- [x] `--json` run summaries, library logging, property-based invariant tests
 - [ ] `MessyData-100` public benchmark + leaderboard
 - [ ] pandera / GX / dbt exporters
 - [ ] Polars backend

{cleanframe_engine-0.3.0 → cleanframe_engine-0.4.0}/README.md

@@ -97,8 +97,10 @@ If the new file's schema drifted (a renamed column, a new currency format), Clea
 ```
 ⚠ Schema drift detected in new_customers.csv
   • Column "Amt (INR)" is new — 94% match to recipe column "amount_inr"
-  • 312 values in "signup_date" match no allowed date format (new: "Jan 5, 26")
-  Run `cleanframe suggest new_customers.csv --recipe customer.recipe.yaml --update` to review a patch.
+  • 312 value(s) in "signup_date" match no allowed date format (new: 'Jan 5, 26')
+  • 40 of 312 numeric value(s) in "Amount" do not fit the recipe's number format
+    (decimal '.', thousands ','; new: '€1.200,50')
+  Run `cleanframe suggest new_customers.csv --recipe <recipe.yaml> --update` to review a patch.
 ```
 
 ## Bring your own API key (or no key at all)
@@ -109,30 +111,34 @@ CleanFrame is LLM-optional and provider-agnostic.
 |---|---|---|
 | **Rules-only** (default) | Deterministic detectors + heuristics | Never |
 | **Metadata** | LLM sees column names, dtypes, and *value patterns* (regex sketches) — never raw values | Only metadata |
-| **Sample** | LLM sees an anonymized, shuffled sample you approve | Only the approved sample |
+| **Sample** | LLM sees a small shuffled sample of *values*. Emails, phones and long strings are redacted; other short values (names, cities) are sent verbatim. Passing `llm_exposure="sample"` is the opt-in — there is no separate approval step | Sample values, per the redaction above |
 | **Replay** | Saved recipes | Never — recipes need no LLM |
 
 - Keys come from environment variables (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`,
-  `OPENROUTER_API_KEY`, `GROQ_API_KEY`, …). CleanFrame never stores, logs, or
-  transmits them.
+  `OPENROUTER_API_KEY`, `GROQ_API_KEY`, …). CleanFrame never stores or logs them, and
+  each provider reads **only its own** variable — `OPENAI_API_KEY` is never sent to
+  another provider.
 - Any provider that speaks OpenAI Chat Completions works out of the box —
   Anthropic (native), OpenAI, OpenRouter, Groq, Together, Fireworks, DeepSeek,
   Mistral, Google Gemini, xAI, Perplexity, Cohere, plus local Ollama / LM Studio
   or any custom `OPENAI_BASE_URL`. Spec format: `provider/model`
   (e.g. `openrouter/anthropic/claude-sonnet-4`, `groq/llama-3.3-70b-versatile`).
-- Hard cost cap: `cf.clean(..., max_tokens_budget=50_000)` aborts planning before it gets expensive.
+- Cost cap: `cf.clean(..., max_tokens_budget=50_000)` is checked before each call (against an
+  estimate) and again against the usage the provider reports — leave headroom, since a call
+  that overshoots has already been billed.
 - Fully offline / air-gapped operation is a supported first-class mode, not a degraded one.
 
 ## What it cleans
 
 | Problem | Example |
 |---|---|
+| Invisible characters | zero-width spaces, BOMs, non-breaking spaces and decomposed accents (`café` typed two ways) → one normalised form |
 | Column name chaos | `Cust Name`, `customer_name`, `CustomerName` → `customer_name` |
 | Date formats | `12/01/24`, `1 Jan 2024`, `2024-01-01` → ISO dates |
-| Currency & numbers | `₹1,20,000`, `$1,200`, `1200 INR` → typed floats + currency column |
-| Category variants | `Bengaluru` / `Bangalore` / `BLR` → one canonical value |
-| Emails & phones | validation, normalization, country codes |
-| Duplicates | exact + fuzzy matching with reviewable merge proposals |
+| Currency & numbers | `₹1,20,000`, `$1,200`, `-$5`, `(1,200)`, `1.234,56`, `1200 INR` → typed floats (+ a currency column); a value whose grouping is invalid or ambiguous becomes a *counted, warned* null, never a guess |
+| Category variants | `Bengaluru` / `bengaluru ` / `BENGALURU` and typos (`Banglore`) → one canonical value; abbreviations such as `BLR` need a seed map or an LLM |
+| Emails & phones | validation, normalization, `00`/`+` international prefixes, optional default country code |
+| Duplicates | exact duplicates are dropped (tracked in the diff); fuzzy near-duplicates and key collisions are *reported* with keep/drop proposals for you to act on |
 | Missing values | detected and explained; strategies proposed, never silently applied |
 | Units | `5kg`, `5000 g`, `5 KG` → normalized |
 | Schema mapping | messy file → your target schema, with confidence scores |
@@ -146,14 +152,18 @@ CleanFrame is built for pipelines, not just demos:
 - **Multi-sheet workbooks** — clean every Excel tab into one reviewable recipe; write-back refuses to overwrite the source in place
 - **Out-of-core streaming** — replay row-independent recipes over larger-than-RAM CSVs at chunk-bounded memory
 - **Format auto-detection** — encoding (utf-8 → cp1252) and delimiter sniffed at read time, pinned into the recipe for replay
-- **Bounded memory** — detector sampling (50k values) + capped cell-diff detail (100k); the diff snapshots only op-touched columns (peak ≈ input, not 2× the frame)
+- **Bounded memory** — detector sampling (50k values) + capped cell-diff detail (100k); the diff snapshots the op-touched columns (peak is the frame plus those columns — about 2× when most columns are cleaned; use streaming for files that do not fit)
 - **Safe CSV exports** — spreadsheet formula injection escaped by default
 - **Regex guards** — oversized / nested-quantifier patterns rejected
 - **Visible degradation** — missing columns, unparseable values and LLM fallbacks warn instead of failing silently
 - **Quarantine, don't delete** — validation failures are held with reasons
 - **Verbatim reads** — `text=True` keeps leading zeros, literal `NA` and `1e5` exactly as the file has them; otherwise CleanFrame warns about what pandas' type inference changed on read
 - **Never overwrites its input** — writing output over the source file takes an explicit `--overwrite`, and every write lands via a temporary file
-- **Scriptable exits** — distinct exit codes for usage errors, data errors, drift stops and validation failures
+- **Scriptable exits** — distinct exit codes for usage errors, data errors, drift stops and validation failures; `--json` prints one machine-readable summary on stdout
+- **Number-format drift** — a file whose numbers stop fitting the recipe (`€1.200,50` against a `.`-decimal recipe) stops the replay instead of yielding `1.2005`
+- **Strict recipes** — unknown ops/params, duplicate keys and bad regexes fail at *load*; plain YAML keys like `yes`, `010` or `12:30` stay column names
+- **Generated code = executor** — exported pandas embeds the executor's own parsers, and property-based tests check they agree
+- **Plugins that travel** — custom ops/detectors load from entry points, `--plugin` or `CLEANFRAME_PLUGINS`, and can opt in to code export and streaming
 
 See the [Production Guide](https://github.com/inboxpraveen/Cleanframe/wiki/Production-Guide).
 
@@ -221,13 +231,21 @@ Profiler ─→ Issue Detectors ─→ Planner ──────────→
                                         Validator · Cell-diff · HTML report
 ```
 
-Detectors are plugins — write your own in ~30 lines:
+Detectors and ops are plugins — write your own in ~30 lines:
 
 ```python
+@cf.register_op("normalize_iban", streamable=True,
+                codegen=lambda p, col: [f"df[{col!r}] = df[{col!r}].str.replace(' ', '').str.upper()"])
+def normalize_iban(series: pd.Series) -> pd.Series:
+    return series.str.replace(" ", "").str.upper()
+
 @cf.detector("iban")
-def detect_iban(series: pd.Series) -> cf.Issues: ...
+def detect_iban(series: pd.Series) -> cf.Issues: ...   # propose cf.Op("normalize_iban")
 ```
 
+Package it under the `cleanframe.plugins` entry-point group and `cleanframe apply` in CI
+finds it — see the [plugin guide](https://github.com/inboxpraveen/Cleanframe/wiki/Plugins).
+
 ## Install
 
 ```bash
@@ -241,7 +259,7 @@ pip install "cleanframe-engine[all]"
 pip install "cleanframe-engine @ git+https://github.com/inboxpraveen/Cleanframe"
 ```
 
-Python 3.10+. The distribution is `cleanframe-engine`; the import package is
+Python 3.10+. The distribution is [`cleanframe-engine`](https://pypi.org/project/cleanframe-engine/); the import package is
 `cleanframe` (`import cleanframe as cf`), and the CLI is `cleanframe` (or
 `python -m cleanframe`).
 
@@ -255,6 +273,8 @@ Python 3.10+. The distribution is `cleanframe-engine`; the import package is
 - [x] Selective ingestion (sheet / columns / rows)
 - [x] Out-of-core streaming replay (larger-than-RAM CSVs)
 - [x] Read-time format auto-correction (encoding + delimiter)
+- [x] Plugin discovery; custom ops that export and stream
+- [x] `--json` run summaries, library logging, property-based invariant tests
 - [ ] `MessyData-100` public benchmark + leaderboard
 - [ ] pandera / GX / dbt exporters
 - [ ] Polars backend

{cleanframe_engine-0.3.0 → cleanframe_engine-0.4.0}/SECURITY.md

@@ -4,9 +4,9 @@
 
 | Version | Supported |
 |---------|-----------|
-| 0.3.x   | Yes       |
-| 0.2.x   | Best-effort |
-| 0.1.x   | No        |
+| 0.4.x   | Yes       |
+| 0.3.x   | Security fixes only |
+| < 0.3   | No (never published under the `cleanframe-engine` name) |
 
 ## Reporting a vulnerability
 
@@ -28,15 +28,17 @@ and disclosure timeline with you.
 ## Design guarantees relevant to security
 
 - **No `eval` / `exec`** in the library path. Recipes are data, not code.
-- Recipes and schemas are loaded with `yaml.safe_load`.
+- Recipes and schemas are loaded with a `SafeLoader` subclass (no arbitrary Python object construction).
 - HTML reports use Jinja2 autoescaping.
-- The LLM planner never receives raw cell values in the default `metadata` exposure.
+- The LLM planner never receives raw cell values in the default `metadata` exposure, and each
+  provider reads only its own API-key environment variable (an `OPENAI_API_KEY` is never sent to another provider).
+- An LLM-authored recipe can never contain `fill_na` or `drop_columns`; they are stripped with a warning.
 - CSV/Excel exports sanitise formula-like cells **and header labels** (`=`, `@`, and
   `+`/`-` followed by anything that is not a plain number) by default. A signed
   number is left alone so normalised phone numbers and negative amounts survive.
 - Generated standalone pandas escapes column names and validation labels, so a
   crafted CSV header cannot inject code into the exported module.
-- User-supplied regex patterns in recipes are length- and complexity-bounded.
+- User-supplied regex patterns in recipes are length- and complexity-bounded, and refused at load time.
 - Recipes and schemas reject duplicate YAML keys instead of silently keeping the last.
 - Cleaned output never overwrites its own input file without an explicit opt-in.
 
@@ -46,3 +48,8 @@ CleanFrame is a **data-cleaning library**, not a sandbox. Callers who pass untru
 file paths, recipes, or schemas still control the host filesystem and process.
 Treat recipe YAML from untrusted sources like any other untrusted config: review it
 before applying it to production data.
+
+**Plugins run code.** `cleanframe.plugins` entry points, `CLEANFRAME_PLUGINS` and `--plugin MODULE` import Python
+modules, exactly like installing a package. Only install plugins you trust; set `CLEANFRAME_NO_PLUGINS=1` (or
+pass `--no-plugins`) to disable automatic discovery. A recipe cannot load a plugin by itself: it can only *name* an
+op, which is resolved against the plugins already installed or explicitly requested.

{cleanframe_engine-0.3.0 → cleanframe_engine-0.4.0}/cleanframe/__init__.py

@@ -13,6 +13,8 @@ input → same output, every time. See the README for the full tour.
 
 from __future__ import annotations
 
+import logging as _logging
+
 from ._version import __version__
 
 # -- high-level API --------------------------------------------------------
@@ -52,6 +54,7 @@ from .ops import list_ops, register_op
 
 # -- planning / execution --------------------------------------------------
 from .planner import Planner, RulesPlanner, plan_recipe
+from .plugins import load_plugins
 
 # -- profiling -------------------------------------------------------------
 from .profile import ColumnProfile, DataFrameProfile, profile_dataframe
@@ -83,6 +86,10 @@ from .workbook import (
     read_workbook,
 )
 
+# A library must not configure logging. Applications opt in with
+# ``logging.getLogger("cleanframe").setLevel(...)`` (or the CLI's --verbose).
+_logging.getLogger("cleanframe").addHandler(_logging.NullHandler())
+
 __all__ = [
     "__version__",
     # api
@@ -103,6 +110,7 @@ __all__ = [
     "detector",
     "validator",
     "register_op",
+    "load_plugins",
     "DetectorContext",
     "run_detectors",
     "list_detectors",