cleanframe-engine 0.3.0__tar.gz

cleanframe_engine-0.3.0/.gitignore

@@ -0,0 +1,101 @@
+# Byte-compile / cache
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# Environment / secrets (never commit)
+.env
+.env.*
+!.env.example
+llm-help.md
+*.pem
+*.key
+credentials.json
+service-account*.json
+.secrets/
+
+# Distribution / packaging
+build/
+dist/
+develop-eggs/
+downloads/
+eggs/
+.eggs/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+pip-wheel-metadata/
+
+# Test / coverage / linters / type checkers
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+.tox/
+.nox/
+coverage.xml
+*.cover
+.hypothesis/
+.mypy_cache/
+.dmypy.json
+dmypy.json
+.ruff_cache/
+.cache/
+
+# Jupyter
+.ipynb_checkpoints/
+
+# IDE / editors
+.idea/
+.vscode/
+*.swp
+*.swo
+*~
+*.code-workspace
+
+# OS junk (Windows / macOS / Linux)
+.DS_Store
+.DS_Store?
+._*
+.Spotlight-V100
+.Trashes
+ehthumbs.db
+Thumbs.db
+Desktop.ini
+$RECYCLE.BIN/
+
+# Local run outputs (keep committed examples/)
+/out/
+/output/
+/tmp/
+/temp/
+
+# Artifacts CleanFrame writes next to an input file. The sample recipe and schema
+# under examples/ are committed on purpose and are already tracked.
+*.report.html
+*.clean.csv
+*.clean.tsv
+*.clean.xlsx
+*.clean.json
+*.clean.parquet
+*.quarantine.csv
+*.patched.yaml
+*.cf-tmp
+examples/messy_customers.recipe.yaml
+examples/messy_customers.schema.yaml
+!examples/customer.recipe.yaml
+!examples/customer.schema.yaml

cleanframe_engine-0.3.0/CHANGELOG.md

@@ -0,0 +1,235 @@
+# Changelog
+
+All notable changes to CleanFrame are documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.3.0] — 2026-09-04
+
+Release-readiness pass driven by a full audit of the CLI, the Python API, packaging and
+the docs. Several fixes change behaviour on purpose: cases that used to pass silently
+now fail loudly, and a few transforms that quietly corrupted data no longer run.
+
+### Security
+
+- **Generated code no longer trusts column names.** `generate_code` interpolated the
+  source column into a comment and the validation label into a `raise` without escaping,
+  so a CSV header containing a newline produced an exported `clean(df)` that executed
+  arbitrary statements. Both are now sanitised.
+- **CSV/Excel exports sanitise header labels**, not just cells: a column literally named
+  `=CMD()` was written as a live formula.
+- The formula sanitiser no longer escapes plain signed numbers, so a normalised phone
+  number (`+919876543210`) and a negative amount (`-1.5`) survive export unchanged.
+- Recipes and schemas reject **duplicate YAML keys** instead of silently keeping the last.
+
+### Fixed — silent data corruption
+
+- `normalize_phone` on a numeric column (one blank cell makes pandas read the column as
+  float) appended a spurious trailing digit; `9876543210.0` became `98765432100`.
+  Trailing extensions (`ext 12`) are no longer fused into the number either.
+- ISO timestamps and dashed identifiers were classified as phone numbers and stripped to
+  digits. Phone classification now needs a phone-ish column name or an explicit `+`/`(`
+  dial prefix, and clock times and ISO dates are never phone-like.
+- European decimals were parsed with the US convention: `€1.200,50` became `1.2005` while
+  the issue reported nothing unparseable. The currency detector now infers the grouping.
+- A fuzzy category merge folded `Unapproved` into `Approved` (and `Unverified` into
+  `Verified`). A spelling that is another plus a negation prefix is never merged.
+- A ragged CSV row (one field too many, typically a trailing delimiter) shifted every
+  column left because pandas promoted the first field to the index. Reads now pass
+  `index_col=False`.
+- `in [Yes, No]` rejected every row: YAML 1.1 read the values as booleans. Membership
+  values are parsed as text.
+- Mixed day-first and month-first dates reported `unparsed: 0` while nulling the values
+  that no longer matched after the conflicting format was dropped. The count is now taken
+  after reconciliation.
+- Punctuation-only values (`-`, `?`) were collapsed into `""` by category clustering.
+- `NA` in a short all-caps code column (Namibia) is treated as ambiguous, not a null.
+- Category canonicalisation prefers short all-caps codes (`CA` over `ca`).
+- A user column named `_cf_quarantine_reason` was overwritten in the quarantine frame.
+- `skiprows` meant different things per format and could eat the CSV header. It now
+  counts **data rows** everywhere: an int drops that many leading records, a list names
+  1-based data rows.
+- `profile_dataframe` no longer takes O(n²) time on long text values, so a column holding
+  a notes field or a JSON blob does not stall `clean()`. It also survives unhashable
+  cells (lists/dicts) and categorical columns.
+- A validation rule written `values: [Yes, No]` matched nothing, because YAML 1.1 reads
+  those as booleans. Every spelling of the boolean is now matched, in the executor and in
+  the exported code alike.
+- `round: {decimals: 2}` — the documented parameter form — failed to load; only the bare
+  `round: 2` worked.
+- The schema dtype alias `str` was listed but not accepted.
+- `excel_sheet_names` left the workbook file handle open, so a later write to the same
+  path could fail with a permission error on Windows.
+
+### Fixed — errors that reached users as tracebacks
+
+- The CLI has a catch-all: an unexpected exception prints one line plus an invitation to
+  re-run with `--debug` (or `CLEANFRAME_DEBUG=1`) instead of a traceback.
+- Clean messages replace tracebacks for: a missing or directory input path on
+  `clean`/`report` (the format detector opened the file before the existence check); an
+  output path that is a directory, read-only, or locked; malformed recipe YAML through
+  `apply`; malformed schema YAML; a recipe or schema in a non-UTF-8 encoding; a
+  non-integer recipe `version`; a non-mapping `meta`; a malformed `source_fingerprint`; a
+  validation rule with an unsupported parameter; a bad `mode`, `on_drift` or `chunksize`;
+  a wrongly-typed `options`, `planner` or `recipe` argument.
+- The encoding ladder ends at latin-1, which cannot fail. Byte values cp1252 leaves
+  undefined (a Shift-JIS export, for instance) no longer raise mid-detection.
+- UTF-16/32 files are detected by byte-order mark instead of being reported as binary.
+- Excel writes strip the control characters openpyxl refuses and check its 32,767-character
+  cell limit up front, so a failed write no longer leaves a truncated workbook behind.
+
+### Changed
+
+- **The PyPI distribution is now `cleanframe-engine`** (the name `cleanframe` belongs to
+  an unrelated project). The import package, the console script and every API are
+  unchanged: `pip install cleanframe-engine`, then `import cleanframe as cf`.
+- **Exit codes are distinct:** `0` success, `1` data/recipe/output error, `2` usage,
+  `3` stopped on drift, `4` validation failed, `70` internal error, `130` interrupted.
+  Previously everything but a usage error exited `1`.
+- **Writing output over the input file is refused** unless `overwrite=True` /
+  `--overwrite`. A single-sheet clean written back over its own workbook used to leave
+  one renamed sheet and delete the rest.
+- Every write goes through a temporary file and is moved into place — cleaned data,
+  quarantine, streamed output, recipes, schemas, generated code and HTML reports.
+- `stream_apply` accepts `overwrite=True` (CLI `--overwrite`), which it previously had no
+  way to express.
+- **Unknown and misspelled op parameters are rejected at load time** with the list of
+  valid names. `parse_date: {format: ...}`, `dedup: {case_insensitive: true}` and
+  `to_na: {token: ...}` used to load and do nothing. `cast` targets, `normalize_unit`
+  units, `dedup keep` and validation check names are validated at load too.
+- Wrongly-typed parameters are refused rather than coerced: `remove_symbols: 5` used to
+  delete every digit 5, and a bare string where a list belongs used to be read one
+  character at a time.
+- Schema dtype aliases (`int`, `number`, `text`, `bool`, `id`) are normalised, so they now
+  drive the same casts as their canonical spellings instead of being silently ignored.
+- Flags that were silently ignored now error: `--code`/`--report`/`--quarantine` and the
+  file-only read flags in workbook mode, `--report` and the selection flags in streaming
+  mode. `--chunksize 0` was accepted and quietly ran a whole-frame apply.
+- `llm_exposure="none"` makes no network call at all; it plans with rules and warns.
+  Previously it sent the same metadata payload as `metadata` exposure.
+- `on_drift` is validated: a typo used to disable the drift guard silently.
+- Streaming pins column types from the recipe's fingerprint, so a streamed replay renders
+  numbers the same way a whole-frame replay does (`2` vs `2.0`), falling back to text with
+  a warning if the file no longer matches.
+- `suggest` re-applies the recipe's recorded `read:` binding and accepts the selection
+  flags, so the command the drift message recommends now works on workbooks and on
+  `;`-separated files instead of reporting every column as drifted.
+- `generate_code` raises when a recipe uses an op or check it cannot reproduce, instead of
+  emitting a silently incomplete module (`allow_partial=True` restores the old output).
+- Every warning uses the new `CleanFrameWarning` category and prints as one line from the
+  CLI. Filter them with `warnings.simplefilter("ignore", cleanframe.CleanFrameWarning)`.
+- `Mode.coerce` raises `CleanFrameError` rather than `ValueError`.
+- Unsupported input extensions are refused instead of being parsed as CSV.
+- Duplicate and blank CSV header names are refused instead of being renamed by pandas to
+  `name.1` / `Unnamed: 3`.
+- Writing legacy `.xls` is refused (pandas emits `.xlsx` bytes under the name).
+
+### Added
+
+- `text=True` / `--text` reads every field verbatim, keeping leading zeros, literal `NA`
+  and `1e5` exactly as the file has them, and records the choice in the recipe. Without
+  it, `clean`/`report` compare a bounded verbatim re-read and warn naming the columns
+  pandas' type inference changed.
+- Values an op could not parse are counted per column, logged and warned about.
+- `python -m cleanframe` as an alias for the console script.
+- CLI: `--debug`, `--verbose`, `--sep`, `--encoding`, `--text`, `--no-llm-fallback`,
+  `clean --out-dir`, and `-o` on every subcommand that has `--out`.
+- `clean(..., llm_fallback=False)` makes an LLM failure raise instead of degrading.
+- **A model-written step that cannot load no longer costs the whole plan.** It is dropped,
+  warned about, and listed in `recipe.meta["llm_dropped"]`, so the rest of the model's
+  recipe still runs. Previously one stray parameter, or a `cast` with no target, discarded
+  the entire LLM recipe and silently fell back to rules. A response that is not a recipe at
+  all still falls back. Ops a mode forbids are recorded in `recipe.meta["llm_blocked_ops"]`.
+- Recipe loading joins an op and its single argument written as two list items
+  (`["extract_currency", "cast", "float"]`), a shape models emit regularly. Two real op
+  names in a row are never joined.
+- `sep=` / `encoding=` on `clean`, `report`, `apply_recipe` and `infer_schema`;
+  `blank_lines=` and `text=` on `read_frame`; `source=`/`overwrite=` on `write_frame`.
+- `OutputError` and `CleanFrameWarning` in the public API.
+- `mypy` runs clean over the package and is part of the dev extra and CI.
+- Release workflow (PyPI trusted publishing), Dependabot, CodeQL, issue and pull-request
+  templates, a code of conduct, and a citation file.
+
+## [0.2.0] — 2026-07-19
+
+### Added — production-hardening upgrade (multi-sheet, scaling, selection, format-correction, streaming)
+
+- **Multi-sheet Excel workbooks**: `clean_workbook` / `apply_workbook` clean every tab
+  independently (one recipe + diff per sheet), collected in a `WorkbookResult` with a
+  single reviewable `WorkbookRecipe` (`sheets:` block). `read_frame` now *refuses* to
+  silently read only sheet 1 of a multi-sheet workbook — pass `sheet=` or use the
+  workbook API. Write-back preserves untouched sheets but **refuses in-place overwrite
+  of the source** (formulas/formatting are lost on pandas re-emit) unless `overwrite=True`.
+  The `cleanframe clean/apply` CLI auto-routes a multi-sheet `.xlsx`.
+- **Selective ingestion**: `sheet` / `columns` / `nrows` / `skiprows` on
+  `read_frame`/`clean`/`report`/`apply`/`infer-schema` and the CLI, recorded in the recipe's
+  new `read:` section (recipe v2, backward-compatible) so `apply` re-reads the same slice.
+- **Read-time format auto-correction** (`correct_format=True`, default; `--no-correct` to
+  opt out): deterministic encoding fallback (utf-8 → cp1252) and header-consensus delimiter
+  detection for CSV-family files, pinned into the recipe `read:` section for replay; refuses
+  on an ambiguous delimiter.
+- **Out-of-core streaming replay**: `stream_apply(recipe, in, out, chunksize=)` and
+  `cleanframe apply --chunksize N` process files larger than RAM. Row-independent recipes
+  stream with byte-identical values and bounded (chunk-sized) memory; global ops
+  (`dedup`, aggregate `fill_na`, `cast` to category/datetime, format-less `parse_date`, the
+  `unique` validator) are **refused with a clear, named error**. Streaming also honours the
+  refuse-on-drift guarantee (checked on a bounded head sample).
+- **In-RAM scaling**: ~44× faster diff extraction (bulk slicing), the executor snapshots only
+  op-touched columns (peak memory ≈ input, not 2×), a vectorised `.str` fast-path for text ops
+  (byte-identical), and deduplicated profiler signal computation.
+
+### Fixed — correctness & robustness (from a full empirical audit)
+
+- **Silent data loss / lineage**: an emitted derived column that overwrites an existing column
+  is now tracked in the diff (was reported as add+remove with the change lost); a value rewrite
+  on a row later dropped by dedup/validation keeps its change provenance; two ops emitting the
+  same column now raise instead of silently clobbering.
+- **Detectors**: ambiguous DD/MM vs MM/DD dates are resolved from the data (no silent day/month
+  swap); fuzzy category clustering no longer merges two both-frequent look-alikes
+  (`insured`/`uninsured`); datetimes keep their time-of-day; disguised-null tokens that are often
+  legitimate (`none`, `-`, `unknown`) are surfaced for review, not auto-converted.
+- **`parse_number`** rejects fused digit groups (`"12ab34"` → NaN) and handles scientific
+  notation; format-less `parse_date`/`cast(datetime)` are now deterministic (order-independent).
+- **Crash-hardening**: non-UTF-8 CSVs, ragged/empty files, directories, mislabeled extensions,
+  malformed YAML/op params, duplicate/non-string/MultiIndex column names, and a Windows cp1252
+  console now raise a clean `CleanFrameError` / render safely instead of a raw traceback.
+- **LLM planner**: any failure (bad JSON, malformed recipe, provider/network error) now falls
+  back to the deterministic rules planner as documented — previously some outputs crashed the
+  pipeline with an uncaught `KeyError`. Array-form ops are parsed leniently; the `METADATA`
+  exposure no longer leaks a raw cell value via a detector message.
+- **Codegen**: generated standalone pandas now reproduces the executor exactly (currency symbols,
+  number sign/unicode-minus, NA tokens, unit aliases, `cast` bool/date, validation row-filtering,
+  and case-insensitive dedup), rendered from a single source of truth in `cleanframe.ops`.
+- **Schema**: an unknown/typo'd dtype (`"flaot"`) is rejected instead of silently ignored.
+
+### Added — production safety guards for large datasets and untrusted exports
+
+- Detector scans sample at most 50,000 non-null values per column (`sample_non_null`)
+- Cell-level diffs cap stored detail at 100,000 changes by default (`max_diff_changes`)
+- CSV/TSV writes escape spreadsheet formula injection by default (`sanitize_csv=True`)
+- Recipe regexes (`replace`, `matches:`) reject oversized / nested-quantifier patterns
+- LLM HTTP clients use a 60s timeout; SAMPLE exposure no longer materialises entire columns
+- Missing recipe columns and LLM fallbacks emit `warnings.warn` (not silent skips)
+- Optional `parquet` extra (`pyarrow`)
+- `cleanframe/py.typed` marker (PEP 561)
+- Example schema + recipe under `examples/`
+- CI workflow (pytest + ruff on Python 3.10–3.13)
+- Full documentation under `docs/` and GitHub Wiki pages under `wiki/`
+
+### Fixed
+
+- Units detector threshold now compares against the sampled column size, not the full row count
+- Cross-platform IO: UTF-8 (+ BOM-tolerant reads), LF-only text/CSV writes, auto-create parent dirs
+- Quarantine / `save_all` CSV exports go through `write_frame` (same sanitise + encoding path)
+- Pandas 3 compatibility: detectors recognise default ``str`` / ``string`` dtypes (not only ``object``)
+
+## [0.1.0] — 2026-07-11
+
+### Added
+
+- Initial public release: profiler, detectors, rules + optional LLM planner, recipe YAML,
+  deterministic executor, validation/quarantine, cell-level diff, schema drift, HTML reports,
+  codegen, and CLI.

cleanframe_engine-0.3.0/CONTRIBUTING.md

@@ -0,0 +1,241 @@
+# Contributing to CleanFrame
+
+Thank you for helping make messy data reproducible. This guide is short on purpose.
+Read the **Invariants** section once — everything else follows from it.
+
+CleanFrame's whole promise is: *AI writes the recipe once; pure pandas replays it
+forever, deterministically.* A change that quietly breaks that promise is worse
+than no change at all. The rules below exist to protect it.
+
+---
+
+## The five invariants (do not break these)
+
+1. **Determinism.** The same input must always produce the same recipe, the same
+   cleaned frame, the same diff — on any machine, in any process. That means:
+   - No `set` iteration where order matters, no reliance on dict ordering from an
+     unstable source, no `random`, no clock, no network in the core path.
+   - Sort before you iterate when order is observable. Break ties explicitly
+     (see `_canonical` in `detectors/categories.py` for the pattern).
+   - There is a test for this (`tests/test_executor_diff.py::test_execution_is_deterministic`).
+     Add one for anything new that could vary.
+
+2. **The LLM never touches your data.** The LLM planner (`llm.py`) sends *metadata
+   and value pattern sketches only* (unless the caller explicitly opts into a
+   sample). It returns a recipe that is parsed through the *same* `Recipe` model
+   the rules planner uses. If you extend the LLM path, the model must never receive
+   raw cell values by default, and its output must go through `Recipe.from_dict`
+   (which validates it) — never straight to the executor.
+
+3. **Recipes round-trip losslessly and idempotently.** `Recipe.from_yaml(r.to_yaml())`
+   must equal `r`, and applying `to_yaml` twice must be byte-identical. If you add
+   an op with parameters, you must add a `coerce`/`compact` pair such that
+   `coerce(compact(params)) == params` (see the op registry contract below).
+
+4. **Nothing is silently imputed or dropped.** Missing values are *reported*, never
+   filled unless a human puts `fill_na` in the recipe. Validation failures go to a
+   *quarantine* frame with a reason — never deleted. Outliers are flagged, never
+   "fixed." When you must bound coverage (top-N, sampling), surface it. Values an op
+   cannot parse are counted and warned about, not quietly turned into nulls.
+
+   This extends to *inputs*: a misspelled op parameter, an unknown validation check
+   and a duplicate YAML key are all rejected at load time. If you add a parameter,
+   name it in the op's signature (or in `aliases=`) so the loader accepts it — a
+   parameter the loader silently ignores is a data-loss bug.
+
+5. **Every changed cell is tracked.** The executor assigns a stable row id and
+   tracks column lineage so `CellDiff` can attribute every change. If you add a
+   transform that adds/removes/reorders columns or rows, make sure the lineage in
+   `executor.py` stays correct and the diff still reconciles.
+
+If a change would weaken one of these, open an issue to discuss it first.
+
+---
+
+## Architecture in one screen
+
+```
+DataFrame ─▶ profile.py ─▶ detectors/ ─▶ planner.py ─▶ Recipe (recipe.py)
+             (semantic     (Issues +      (assemble +        │
+              types)        Proposals)     order + gate)      ▼
+                                                       executor.py  (pure pandas)
+                                                             │
+                              ┌──────────────────────────────┼───────────────┐
+                              ▼              ▼                ▼               ▼
+                         validate.py     diff.py          drift.py       report.py
+                         (quarantine)   (cell diff)     (replay guard)    (HTML)
+```
+
+- **`ops.py`** — the vocabulary. Every recipe op is a pure function of a pandas
+  object registered here. The deterministic heart.
+- **`detectors/`** — plugins that find problems and *propose* fixes (ops + renames).
+  Domain knowledge lives here.
+- **`planner.py`** — turns proposals into a recipe: applies mode policy, resolves
+  renames, and — critically — orders ops via `OP_ORDER`.
+- **`executor.py`** — replays a recipe in fixed phases with lineage tracking.
+- **`recipe.py` / `schema.py`** — the durable, human-reviewable artifacts.
+- **`llm.py`** — optional; writes a recipe from metadata only.
+
+The dependency direction is strictly downward (detectors import ops, planner
+imports detectors, etc.). Don't introduce an upward import — it will create a
+cycle. When you need a lower layer from a higher one at call time, use a local
+import inside the function (there are a few examples already).
+
+---
+
+Demo data lives in `examples/messy_customers.csv` — plus a sample schema and
+recipe (`customer.schema.yaml`, `customer.recipe.yaml`). Full docs:
+[`docs/`](docs/) and the [Wiki](https://github.com/inboxpraveen/Cleanframe/wiki).
+
+---
+
+## Getting set up
+
+```bash
+git clone https://github.com/inboxpraveen/Cleanframe.git
+cd Cleanframe
+pip install -e ".[dev]"      # editable install + pytest + openpyxl + ruff + mypy
+pytest                       # full suite, runs in a few seconds
+ruff check cleanframe tests  # lint
+mypy                         # the package ships py.typed, so this must stay clean
+```
+
+The distribution is published as `cleanframe-engine`; the import package is
+`cleanframe`.
+
+Windows note: the code and tests handle currency symbols (`₹`, `€`). If your
+console mangles them, run with `PYTHONUTF8=1`. The CLI sets UTF-8 output itself.
+
+---
+
+## The most common contribution: a new detector
+
+Detectors are the point of the plugin system — the community owns the long tail of
+messy-data weirdness. A detector is ~15 lines.
+
+```python
+# cleanframe/detectors/iban.py
+import re
+import pandas as pd
+from ..issues import Issues
+from ..types import Op, Severity
+from .base import DetectorContext, detector
+
+IBAN_RE = re.compile(r"^[A-Z]{2}\d{2}[A-Z0-9]{11,30}$")
+
+@detector("iban", priority=45)                 # lower priority runs earlier
+def detect_iban(series: pd.Series, ctx: DetectorContext) -> Issues:
+    """One-line summary — shows up in `cleanframe detectors`."""
+    issues = Issues()
+    cp = ctx.column_profile
+    if cp is None or cp.semantic_type not in ("text", "id"):
+        return issues                          # cheap early-out for irrelevant columns
+    bad = [v for v in series.dropna() if isinstance(v, str) and not IBAN_RE.match(v.replace(" ", ""))]
+    if bad:
+        issues.add(
+            "invalid_iban",
+            f"{len(bad)} value(s) are not valid IBANs",
+            severity=Severity.WARNING,
+            confidence=1.0,
+            evidence={"count": len(bad), "examples": bad[:5]},
+            ops=[Op("remove_symbols", {"symbols": [" "]})],   # optional proposed fix
+        )
+    return issues
+```
+
+Then register it by importing the module in `cleanframe/detectors/__init__.py`.
+
+Rules for a good detector:
+
+- **Return `Issues`.** Use `issues.add(...)`; attach a fix with `ops=[...]` and/or
+  `rename_to=...` only when you're confident. `confidence` gates inclusion by mode
+  (`strict` needs ≥ 0.85, `auto` ≥ 0.65, `review` ≥ 0.5).
+- **Be deterministic.** Sort before iterating; break ties explicitly. Never let row
+  order change your output (test it — feed the column shuffled and assert the same
+  proposal).
+- **Don't over-reach.** Propose only *safe, reversible* fixes automatically. Domain
+  guesses (e.g. "BLR means Bangalore") belong to the LLM or the human, not rules.
+- **Early-out** on columns that aren't yours (check `ctx.column_profile.semantic_type`).
+- The signature can be `(series)` or `(series, ctx)` — the runner passes `ctx` if you
+  declare it. Frame-level detectors use `@detector("name", scope="frame")` and take
+  the whole `df`.
+
+Add a test in `tests/test_profile_detectors.py` asserting your detector fires (and
+does *not* fire on clean data).
+
+---
+
+## Adding an op (a recipe transform)
+
+Ops must be pure and deterministic. If your op takes parameters, you owe a
+`coerce`/`compact` pair so recipes stay minimal *and* round-trip losslessly.
+
+```python
+@register_op(
+    "titlecase_words",
+    scope="column",                                  # or "frame"
+    coerce=lambda raw: {"min_len": (raw or {}).get("min_len", 2)},   # recipe form -> params
+    compact=lambda p: _prune(p, {"min_len": 2}),      # params -> minimal recipe form
+)
+def titlecase_words(series, min_len=2):
+    """Docstring first line shows in `cleanframe ops`."""
+    return _apply_str(series, lambda s: " ".join(w.title() if len(w) >= min_len else w for w in s.split()))
+```
+
+Then, if the planner should ever emit it, add its name to **`OP_ORDER`** in
+`planner.py` at the correct position. `OP_ORDER` is the single source of truth for
+op sequencing — getting the position right is how independently-written detectors
+compose safely (whitespace before categories, currency split before symbol
+stripping, casing last). If your op isn't ordered, it runs after all ordered ops.
+
+Column ops return a `Series` (or a `ColumnOpResult` if they also emit new columns —
+see `extract_currency`). Frame ops return a `DataFrame` and **must preserve the row
+index** of surviving rows, or the cell diff will misattribute changes.
+
+Required test: `coerce(compact(params)) == params` (the parametrized test in
+`tests/test_ops.py` will pick your op up automatically if you give it a sample).
+
+---
+
+## Adding a validator
+
+```python
+from cleanframe.validate import validator
+
+@validator("valid_iban")
+def _valid_iban(series):
+    return series.isna() | series.astype(str).str.replace(" ", "").str.match(IBAN_RE)  # True = passes
+```
+
+Return a boolean pass-mask (`True` means the value is fine; NaN should usually
+pass — `not_null` is the check for missingness). `on_fail` policy is handled for
+you.
+
+---
+
+## Testing & quality bar
+
+- **Every PR keeps `pytest` green.** New behavior needs a new test.
+- **Test the invariant, not just the happy path.** For anything order-sensitive, add
+  a determinism assertion. For a new op with params, add the round-trip.
+- Prefer small, focused tests using the fixtures in `tests/conftest.py`.
+- `ruff check cleanframe tests` and `mypy` must both pass — CI runs them.
+
+## Style
+
+- Match the surrounding code: type hints, `from __future__ import annotations`,
+  docstrings that explain *why* (the reader can see *what*).
+- Target Python 3.10+. Keep the core dependencies minimal (pandas, numpy, pyyaml,
+  jinja2). Anything heavier goes in an optional extra in `pyproject.toml`.
+- No `print` in library code — return data or `log` to a list. `print` is for the
+  CLI only.
+
+## Pull requests
+
+1. Branch from `main`. One logical change per PR.
+2. Describe the user-visible behavior change and which invariant(s) you considered.
+3. If you touched the recipe format, the executor, or `OP_ORDER`, say so
+   prominently — those are the load-bearing walls.
+4. Be kind in review. We're all here to make one messy CSV less painful.
+
+Questions or a detector idea? Open an issue tagged `good-first-detector`.