prefit 0.1.0__tar.gz

prefit-0.1.0/.gitignore

@@ -0,0 +1,31 @@
+# secrets / local env
+.env
+.venv/
+venv/
+
+# notebooks (scratch work; examples live in examples/*.py)
+*.ipynb
+.ipynb_checkpoints/
+
+# python / tooling caches
+__pycache__/
+*.py[cod]
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+
+# build artefacts
+dist/
+build/
+*.egg-info/
+
+# OS
+.DS_Store
+
+# reports generated in the repo root (examples/output/ is tracked on purpose)
+/*.html
+
+# private v0 design notes (kept locally, not published)
+ARCHITECTURE.v0.md

prefit-0.1.0/ARCHITECTURE.md

@@ -0,0 +1,98 @@
+# Architecture
+
+prefit shortens the pre-training analysis of a tabular supervised-learning
+project. It measures the data, derives classified decisions from the
+measurements, tests the decisions that can be tested, asks the user about the
+ones that cannot, and exports only what is safe, confirmed or verified.
+
+## Pipeline
+
+```
+pf.split(df, ...) -> dev_df, test_df            (explicit; same splitter as holdout=)
+dev_df, target, [estimator, groups, time_col, holdout, context, answers]
+   │
+   ▼  guard.py ─ validate inputs (misspelled target, single-class target, groups+time…);
+   │             optional internal holdout (only with holdout=…), set aside before profiling
+   ▼  fingerprint.py ─ SHA-256 digests of the input frame -> manifest (no data)
+   ▼  profiling/ ─ dev rows only, seeded sample above ProfileOptions.max_rows
+   │     kinds.py   column kind + reason (numeric, categorical, boolean,
+   │                datetime, text, id_like, constant)
+   │     stats.py   per-kind stats, bootstrap / Wilson intervals,
+   │                single-feature held-out association score
+   │     tabular.py → DataProfile (aggregates only, JSON-serializable)
+   ▼  rules/ ─ deterministic: DataProfile + RuleContext → RuleOutput
+   │     catalog.py  versioned rules with thresholds; emit Evidence → Finding → Decision (+ Question)
+   │     metrics.py  metric policy: one contextual decision driven by the cost of errors
+   ▼  figures.py ─ decision-bound plotly specs (aggregates + color tokens)
+   ▼  report/model.py ─ Report (decisions, questions, metric plan, roadmap,
+   │                    experiments, baseline) + details (findings, evidence,
+   │                    profile, figures, manifest, context, limitations, llm);
+   │                    deeply immutable (frozen.py)
+   │
+   ├─ Report.answer(...)                  re-runs the rules with new answers (no data needed)
+   ├─ Report.verify(df)                   fingerprint check, then verify.py on dev rows
+   ├─ Report.holdout(df)                  fingerprint check, re-derives the internal holdout
+   ├─ Report.register_external_holdout()  metadata + exact-overlap count, never profiled
+   ├─ Report.export()                     export.py: model recipe + preprocessor + CV + scoring
+   ├─ Report.interpret()  llm/: optional, schema-constrained interpretation
+   ├─ Report.to_html()    render/: single-file report
+   └─ pf.review(...)      posttrain.py: assumptions vs. actual errors
+```
+
+## Contracts
+
+- **Evidence** — `id`, `label`, `value`, optional 95% `ci`, `method`, `columns`,
+  linked `figure_ids`.
+- **Finding** — produced by one rule; `rule_id`, `rule_version`, `severity`,
+  `trigger` (`fired` | `uncertain`), `thresholds`, and at least one evidence id
+  (enforced by the model).
+- **Decision** — `cls` (procedural / conditional / contextual / review),
+  `status`, `stage`, `action`, `rationale`, `risk`, optional `step` (the concrete
+  pipeline change), links to findings, evidence, figures, experiments, question.
+- **Decision.eligible()** — the single rule for what enters the pipeline:
+  procedural+applied, contextual+confirmed, conditional+verified (or
+  supported_by_proxy when explicitly requested).
+
+## Why these boundaries
+
+1. **Thresholds are triggers.** A rule compares a statistic *and its interval*
+   to a threshold. Interval straddles the threshold → `uncertain`.
+2. **Data-modifying actions are never procedural.** Dropping duplicates, rows
+   or columns can be a collection artefact or legitimate data; only the user or
+   an experiment can decide. The one exception is constant columns, which carry
+   no information. Raw datetime/text/string-id columns are left out of the
+   export because v1 cannot encode them — a capability limit reported as a
+   review item, not a judgement that they are uninformative.
+3. **The data cannot choose the metric.** It can say which metrics mislead
+   (evidence-backed `avoid`); what to optimize depends on what errors cost.
+4. **Experiments are bounded, and their inference matches the split.** One
+   change at a time against a fixed base, identical folds, hard budget. The
+   test follows the protocol: corrected repeated k-fold t-test (repeated or
+   grouped k-fold), no p-value for time-ordered folds. "Improved" needs
+   direction, Holm-adjusted significance and a practical effect threshold.
+   Proxy models give sensitivity analysis; only the user's estimator verifies.
+5. **Split precedence is explicit.** time_col > groups > stratification;
+   time_col disables stratification (stated in the decision); groups and
+   time_col together raise instead of silently dropping one.
+6. **The EDA must not leak into the final estimate.** The holdout is reserved
+   before profiling and is never used by rules or experiments.
+7. **Reports are immutable, data is fingerprinted.** answer / verify /
+   interpret / register_external_holdout return a new report with an
+   incremented manifest revision and a history entry; every model is frozen
+   with tuples and read-only dicts. Steps that need data again check the
+   frame's fingerprint and refuse a different one instead of re-splitting.
+8. **The LLM is above the trust boundary.** It sees an allowlisted payload
+   (inspectable with `llm_payload()`), must cite existing ids, and can only
+   suggest; `answer()` is the only way an answer is applied.
+
+## Adding a rule
+
+See CONTRIBUTING.md. In short: write `fn(profile, ctx, out)` in
+`rules/catalog.py` with `@rule(id, version, thresholds)`, use `trigger()` with
+the statistic's interval, pick the decision class honestly, add benchmark cases
+(must fire / must not fire / near threshold), and bump the manifest version.
+
+## Not in v1
+
+Other modalities, time-series forecasting, clustering, model selection,
+multi-feature leakage detection, interaction effects between decisions.

prefit-0.1.0/CHANGELOG.md

@@ -0,0 +1,111 @@
+# Changelog
+
+All notable changes to this project are documented here. The project follows
+[Semantic Versioning](https://semver.org/); before 1.0, minor versions may
+change the public API.
+
+## 0.1.0 — 2026-11-01
+
+First release of **prefit** (`pip install prefit`, `import prefit as pf`), the
+decision-driven design. Replaces the v0 `Roadmap` API.
+
+### Added
+- `pf.analyze()` returns a `Report`: Evidence → Finding → Decision chain with
+  decision classes (procedural / conditional / contextual / review) and statuses.
+- Input guard: misspelled targets and unknown keywords raise; holdout reserved
+  before profiling (stratified, group-aware with `groups=`, time-ordered with
+  `time_col=`).
+- Column-kind inference (numeric, categorical, boolean, datetime, text,
+  id_like, constant) with reasons; bootstrap / Wilson intervals on
+  threshold-relevant statistics; near-threshold findings marked `uncertain`.
+- Task-agnostic leakage hint via a single-feature held-out score; leakage is
+  reported as "manual review required" with a question, never as a verdict.
+- Rules: task confirmation, validation split, holdout policy, small data,
+  temporal order, target missing, duplicates, conflicting duplicates, high
+  missingness, near-constant, high correlation, constant columns, unsupported
+  columns, id-like columns, leakage hint, fit-in-fold, skewed features, high
+  cardinality, class imbalance, target skew, heavy tails, zero inflation;
+  metric policy driven by the cost of errors.
+- `Report.answer()`, `Report.verify()` (bounded experiments, corrected paired
+  t-test, Holm correction, budget; user estimator verifies, proxies give
+  sensitivity analysis), `Report.export()` (sklearn Pipeline + CV + scoring from
+  eligible decisions only, with an explicit excluded list).
+- Single-file HTML report with decision cards, linked evidence, rule versions
+  and thresholds, experiments, decision-bound plotly figures with data tables,
+  light/dark themes; text, markdown and JSON renders; reproducibility manifest.
+- `pf.review()`: reference-based hypothesis checks, baselines, residual /
+  confusion / PR / segment analysis, list of never-verified assumptions;
+  standalone or embedded HTML.
+- Optional LLM layer: allowlisted payload with `llm_payload()` preview, column
+  aliasing, schema-constrained output, unknown-id claims dropped, suggestions
+  never applied; Anthropic, Gemini and local OpenAI-compatible providers.
+- Versioned benchmark (`benchmarks/`), track record cited in reports.
+- CLI: `prefit report data.csv --target y`.
+- CI (ruff, mypy, pytest with coverage, HTML smoke test).
+
+### Changed (pre-release API round)
+- Data partitions are explicit: `pf.split(df, target, test_size, strategy)` is the
+  primary path; `analyze` defaults to `holdout=None` and only sets a holdout aside
+  when asked (`holdout=0.2`). `report.split()` became `report.holdout(df)`, which
+  re-derives that partition and never splits anew.
+- Fingerprints: `verify`, `holdout` and `register_external_holdout(dev_df=...)`
+  check the frame against SHA-256 digests (column schema, dtypes, index, target,
+  row-order-sensitive content) of the frame passed to `analyze`, and explain
+  any mismatch (reordered, changed, shorter, different columns).
+- `report.register_external_holdout(test_df, dev_df=...)` records an evaluation
+  partition (metadata only, never profiled) and counts exact row overlap with
+  the development data, reported as a review item without a leakage verdict.
+- Deep immutability: all report models are frozen, lists are tuples and dicts
+  read-only; every change is a new revision. `pf.load_report()` reloads JSON.
+- API surface: reading / flow / automation on `Report`, internals under
+  `report.details` (findings, evidence, profile, figures, manifest, context,
+  limitations, rule_track_record, llm). Short reprs for all report objects,
+  a notebook summary card, and `report.decisions_table()`. v0 aliases removed.
+- Split columns (`groups`, `time_col`) stay out of the features unless
+  `split_columns_as_features=True`, which raises a warning.
+- HTML report rebuilt: Summary / Actions / Roadmap / Details navigation, short
+  decision cards (why with the decisive number, do, experiment summary, at most
+  one figure) with evidence, experiment, figure and rule/benchmark details in
+  collapsed sections, consecutive roadmap numbering, collapsed Details section.
+- Verification and baselines use the export splitter (repeated for power).
+- Feature scaling is an export option (`scale=True`), not part of the
+  procedural fit-in-fold decision.
+- Bootstrap intervals computed on subsampled resamples are rescaled to the full
+  sample size (they were ~sqrt(n/m) times too wide on large data).
+- Single-feature association is a 2-fold cross-validated score.
+- Single-class targets are rejected up front; +/-inf values produce an
+  `infinite_values` review item instead of numpy warnings and are treated as
+  missing by the exported preprocessing; duplicate detection hashes rows
+  directly (much faster on wide data).
+- `benchmarks/run.py` keeps the committed experiment section when run without
+  `--verify`.
+
+### Changed (pre-release review)
+- Raw datetime / text / string-id columns are a review item ("unsupported in
+  v1, may carry signal"), no longer a procedural drop.
+- `groups=` together with `time_col=` raises instead of silently ignoring
+  `groups`; the time split states that class stratification is off; grouped
+  classification notes that class balance is approximate.
+- Verification: the test now follows the split protocol (corrected repeated
+  k-fold t-test; single-run grouped k-fold; no p-value for time-series folds),
+  and "improved" requires a practical effect threshold
+  (`Budget.min_relative_effect` / `min_effect`). Per-fold differences and the
+  protocol are shown in the report.
+- `export()` returns `model` (full estimator recipe, possibly a
+  `TransformedTargetRegressor`) and `preprocessor` instead of `pipeline`;
+  estimator-level steps without an estimator go to `needs_estimator`, and an
+  estimator without `class_weight` moves that step to `excluded` — nothing
+  listed as included is dropped.
+- Reports record `manifest.revision` and `manifest.history` for every
+  analyze / answer / verify / interpret step.
+- Wording: "single-feature predictive leakage hint"; benchmark records are a
+  "rule detection pass rate"; the HTML report opens with a data-partition note.
+
+### Changed
+- `GroupKFold` is no longer suggested from duplicates or id-like columns.
+- Integer targets with more than 10 distinct values are inferred as regression.
+- `use_llm=`, `provider=`, `api_key=` replaced by `llm=LLMConfig(...)`.
+
+### Removed
+- `Roadmap.make_sklearn()` / `validation_recipe()` (use `Report.export()`).
+- Reserved non-tabular modalities from the public enums.

prefit-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,53 @@
+# Contributing
+
+## Setup
+
+```bash
+python -m venv .venv && source .venv/bin/activate
+pip install -e ".[dev]"
+pytest -q                      # tests (includes the benchmark cases)
+ruff check . && mypy prefit
+python benchmarks/run.py       # refreshes benchmarks/results and the track record
+```
+
+## Adding or changing a rule
+
+1. Write the rule in `prefit/rules/catalog.py` with
+   `@rule("id", "1", thresholds={...})`. Compare statistics with `trigger()` and
+   pass the statistic's interval so near-threshold cases become `uncertain`.
+2. Choose the decision class honestly:
+   - procedural only if it is about *how to evaluate* and is right for any data;
+   - anything that changes the data's content is review, conditional or contextual;
+   - anything that needs knowledge outside the data is contextual (ask a question).
+3. Never phrase a hint as a verdict ("leakage hint — manual review required",
+   not "leakage detected").
+4. If the decision has a pipeline step, make sure `export.build_model` supports
+   its `Step.kind`.
+5. Add benchmark cases in `benchmarks/fixtures.py`: one where it must fire, one
+   where it must not, and a near-threshold one where that is meaningful.
+6. Bump the rule's version when its thresholds or logic change, and bump
+   `MANIFEST_VERSION` when cases or expectations change. Re-run
+   `python benchmarks/run.py` and commit the results, including failures.
+7. If decision structure changes, refresh the golden files:
+   `UPDATE_GOLDEN=1 pytest tests/test_snapshot.py`, and review the diff.
+
+## Versioning
+
+Semantic versioning; before 1.0, minor versions may change the API. Every
+release gets a CHANGELOG entry. Rule versions are independent and appear in
+every report's manifest.
+
+## Pull requests
+
+Keep the deterministic core free of new hard dependencies. Tests, ruff and mypy
+must pass; coverage must stay above the configured floor.
+
+## Releasing
+
+Publishing runs in GitHub Actions with PyPI Trusted Publishing (no tokens).
+
+1. Bump `version` in `pyproject.toml` and `prefit/_version.py`, add a CHANGELOG entry, merge to `main`.
+2. Optional dry run: Actions → Release → Run workflow → `target=testpypi`, then
+   `pip install -i https://test.pypi.org/simple/ --extra-index-url https://pypi.org/simple/ prefit`.
+3. Tag and push: `git tag vX.Y.Z && git push origin vX.Y.Z`. The workflow checks that the tag matches the
+   package version, builds, runs `twine check` and a wheel smoke test, then publishes to PyPI.

prefit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 hkln4
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

prefit-0.1.0/PKG-INFO

@@ -0,0 +1,351 @@
+Metadata-Version: 2.5
+Name: prefit
+Version: 0.1.0
+Summary: Decision-driven EDA for tabular supervised learning: evidence-backed, verifiable pre-training decisions.
+Project-URL: Homepage, https://hkln4.github.io/prefit/
+Project-URL: Example report, https://hkln4.github.io/prefit/report.html
+Project-URL: Repository, https://github.com/hkln4/prefit
+Project-URL: Issues, https://github.com/hkln4/prefit/issues
+Project-URL: Changelog, https://github.com/hkln4/prefit/blob/main/CHANGELOG.md
+Author: hkln4
+License-Expression: MIT
+License-File: LICENSE
+Keywords: cross-validation,data-profiling,eda,evaluation,leakage,machine-learning,model-evaluation,scikit-learn
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: numpy>=1.23
+Requires-Dist: pandas>=1.5
+Requires-Dist: pydantic>=2
+Provides-Extra: all
+Requires-Dist: anthropic>=0.40; extra == 'all'
+Requires-Dist: plotly>=5; extra == 'all'
+Requires-Dist: scikit-learn>=1.4; extra == 'all'
+Requires-Dist: scipy>=1.9; extra == 'all'
+Provides-Extra: dev
+Requires-Dist: mypy>=1.8; extra == 'dev'
+Requires-Dist: pandas-stubs; extra == 'dev'
+Requires-Dist: plotly>=5; extra == 'dev'
+Requires-Dist: pytest-cov>=4; extra == 'dev'
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Requires-Dist: scikit-learn>=1.4; extra == 'dev'
+Requires-Dist: scipy>=1.9; extra == 'dev'
+Provides-Extra: gemini
+Requires-Dist: google-genai>=1.0; extra == 'gemini'
+Provides-Extra: llm
+Requires-Dist: anthropic>=0.40; extra == 'llm'
+Provides-Extra: report
+Requires-Dist: plotly>=5; extra == 'report'
+Provides-Extra: verify
+Requires-Dist: scikit-learn>=1.4; extra == 'verify'
+Requires-Dist: scipy>=1.9; extra == 'verify'
+Description-Content-Type: text/markdown
+
+# prefit
+
+**Decision-driven EDA for tabular supervised learning.**
+
+Point `prefit` at your development data before you train. It profiles the
+data, turns what it measures into a short list of *decisions* — how to split,
+which columns to question, what to preprocess, which metric to trust — and shows
+the evidence, threshold and uncertainty behind every one. Decisions that depend
+on your data and model are tested with small, bounded experiments instead of
+being assumed. At the end you get a scikit-learn model recipe that contains only
+steps that are safe, that you confirmed, or that an experiment supported.
+
+It is not AutoML: it does not choose a model or tune hyperparameters, and it
+never applies an unconfirmed assumption silently.
+
+**[Example report](https://hkln4.github.io/prefit/report.html)** ·
+[Project page](https://hkln4.github.io/prefit/) · [PyPI](https://pypi.org/project/prefit/)
+
+```bash
+pip install "prefit[verify]"        # core + experiments/export (scikit-learn, scipy)
+pip install "prefit[verify,report]" # + plotly, for fully offline HTML reports
+pip install "prefit[llm]"           # optional LLM interpretation (Anthropic)
+```
+
+The core (`pandas`, `numpy`, `pydantic`) profiles, decides and renders reports
+with no scikit-learn, no network and no API key.
+
+---
+
+## Quickstart
+
+```python
+import prefit as pf
+from sklearn.ensemble import HistGradientBoostingRegressor
+
+model = HistGradientBoostingRegressor(random_state=0)
+
+# 1. set the test data aside before anything is computed
+dev_df, test_df = pf.split(df, target="price", test_size=0.2, strategy="random", random_state=42)
+
+# 2. analyze the development data only
+report = pf.analyze(dev_df, target="price", estimator=model)
+report                                   # notebook: a short summary card with the next step
+print(report)                            # roadmap: one line of why / do per decision
+report.to_html("report.html")            # the full report
+
+# 3. every step returns a new report revision
+report = report.answer({"metric_priority": "typical_error"})
+report = report.verify(dev_df)           # bounded experiments with your model
+report = report.register_external_holdout(test_df, dev_df=dev_df, name="test set")
+
+# 4. export only what is applied, confirmed or verified
+built = report.export()
+print(built.describe())                  # what is in, what is out, and why
+fitted = built.model.fit(dev_df[built.feature_columns], dev_df["price"])
+
+# 5. once, at the end, on the untouched test data
+review = pf.review(test_df["price"], fitted.predict(test_df[built.feature_columns]), report)
+report.to_html("report.html", review=review)
+```
+
+Shortcut: `pf.analyze(df, target, holdout=0.2)` sets the holdout aside itself,
+before profiling; `dev_df, test_df = report.holdout(df)` gives it back. Without
+`holdout=`, `analyze` never splits — the default is `holdout=None`.
+
+From the shell:
+
+```bash
+prefit report dev.csv --target price --verify --answer metric_priority=typical_error --out report.html
+```
+
+An example report produced by `examples/quickstart.py` is live at
+[hkln4.github.io/prefit/report.html](https://hkln4.github.io/prefit/report.html) (source:
+[`examples/output/california_report.html`](https://github.com/hkln4/prefit/blob/main/examples/output/california_report.html)).
+
+---
+
+## The report object
+
+```python
+# reading
+print(report); report.to_html(path); report.to_markdown()
+report.roadmap; report.pending_questions(); report.decisions_table()
+report.target; report.task
+
+# flow (each returns a new revision; the old report is unchanged)
+report.answer({...}); report.verify(df); report.interpret(llm)
+report.register_external_holdout(test_df, dev_df=dev_df)
+report.export(estimator); report.holdout(df); pf.review(...)
+
+# automation
+report.decisions; report.decision(id); report.questions; report.experiments
+report.metric_plan; report.baseline; report.to_json(); report.llm_payload()
+
+# internals: the traceability chain and reproducibility record
+report.details.findings / .finding(id) / .evidence / .profile / .figures
+report.details.manifest / .context / .limitations / .rule_track_record / .llm
+```
+
+`report.decisions_table()` is the best place to start in a notebook: one row per
+decision with its status, a one-line why (with the decisive number), what to do,
+and any experiment result.
+
+**Reports are immutable.** Every model is frozen, lists are tuples and dicts are
+read-only, all the way down. `answer`, `verify`, `interpret` and
+`register_external_holdout` return a new report whose
+`details.manifest.revision` is incremented and whose `history` records the step.
+`pf.load_report(path)` reloads a report saved with `to_json`.
+
+**The data is checked, never trusted.** A report stores no rows, so `verify`
+and `holdout` take the DataFrame again. Its fingerprint — SHA-256 digests of the
+column schema, dtypes, index, target and row-order-sensitive content — must
+match the frame passed to `analyze`. A shuffled, edited, shortened or different
+frame (the test set, say) raises an error that says what differs; nothing is
+ever re-split.
+
+---
+
+## How decisions are made
+
+Every rule turns *evidence* (a measured value, its 95% interval, the method)
+into a *finding*, and the finding into one or more *decisions*. Each decision has
+a **class**, which says how much the data alone can settle it:
+
+| class | meaning | enters the model recipe when |
+|---|---|---|
+| **procedural** | about *how to evaluate*; correct regardless of the data (fit preprocessing inside folds, stratify, keep split columns and the holdout out). The only column it removes are constant ones. | always |
+| **conditional** | may help or hurt depending on data and model (log target, power transform, target encoding, class weights, dropping mostly-missing columns) | an experiment with **your** estimator verifies it |
+| **contextual** | cannot be decided from data (metric = cost of errors, is a column known at prediction time, is it an identifier, does time order matter) | you answer the question |
+| **review** | needs a human look (duplicates, label conflicts, informative missingness, infinite values, collinearity, raw datetime/text columns v1 cannot encode) | never; it is a to-do |
+
+Ground rules the catalog follows:
+
+- **Thresholds are triggers, not verdicts.** Threshold-relevant statistics carry
+  a bootstrap or Wilson interval (bootstrap intervals are computed for the full
+  sample size even when resamples are subsampled); when the interval straddles
+  the threshold the finding is *near threshold* and its decision *uncertain*.
+- **Leakage is only ever a hint.** A feature that alone predicts the target in a
+  2-fold cross-validated single-feature score raises a "single-feature
+  predictive leakage hint — manual review required" and a question ("is this
+  known at prediction time?"). Genuinely strong features look the same, so
+  nothing is dropped until you answer.
+- **Unsupported is not uninformative.** Raw datetimes, free text and string
+  identifiers are left out of the exported features because v1 has no encoder
+  for them; they are reported as a review item, never as "carries no signal".
+- **No group splitting without real groups.** ID-like columns and duplicates
+  produce review items. Group-aware splitters are used only with `groups=`;
+  `TimeSeriesSplit` only with `time_col=`. Both columns stay out of the features
+  unless `split_columns_as_features=True` (which adds a warning).
+- **Split precedence is explicit.** `time_col` turns off class stratification
+  (the report says so); `groups` with classification uses
+  `StratifiedGroupKFold`, which balances classes only approximately; `groups`
+  and `time_col` together raise an error in v1.
+- **The data you analyze is the data the report describes.** The report opens
+  with a data-partition note: which rows were analyzed, whether an internal or
+  registered external holdout exists, and that it was never read.
+
+## Experiments, and what they can claim
+
+`verify` tests each conditional decision once per model family as "current
+setup" vs. "current setup + this one change", on identical folds. The folds come
+from the same splitter as the export CV decision (repeated when it is a plain
+k-fold, for power); baselines use the same folds. A `Budget` caps the number of
+experiments, wall-clock time and rows.
+
+| protocol | when | test |
+|---|---|---|
+| `repeated_kfold` | default | corrected repeated k-fold t-test (Nadeau & Bengio 2003; Bouckaert & Frank 2004), Holm across the run |
+| `grouped_kfold` | `groups=` | same correction on a single grouped k-fold run (low power, said in the report) |
+| `time_series` | `time_col=` | none — expanding windows are not exchangeable; per-fold differences only, outcome left to you |
+
+An experiment counts as **improved** only with the right direction, a
+significant Holm-adjusted p-value, *and* a mean difference of at least the
+practical threshold (`Budget(min_relative_effect=0.01)` = 1% of the baseline
+score, or an absolute `min_effect`). Significant but smaller differences are
+**negligible**. Per-fold differences are in the report.
+
+- With `estimator=`, an improvement makes the decision **verified** (and
+  exportable); a practical-size loss makes it **refuted**.
+- Without it, a linear and a tree proxy are used. Their results are a
+  **sensitivity analysis** (`supported_by_proxy`), not a verdict on your model,
+  and are excluded from export unless you pass
+  `export(include_proxy_supported=True)`. When one family gains and the other
+  loses, the decision is marked model-family dependent.
+- All of it is **development evidence**. The generalization claim comes from the
+  untouched test data (or nested CV when you have none).
+
+## Export
+
+`report.export(estimator, scale=True)` returns:
+
+- `model` — the full recipe for `fit` / `cross_validate`: `Pipeline(preprocess,
+  estimator)`, wrapped in a `TransformedTargetRegressor` when a target transform
+  was verified (`built.model_kind` says which). `None` without an estimator.
+- `preprocessor` — the feature preprocessing alone. Feature scaling is an
+  estimator-dependent default (`scale=False` turns it off), not a decision.
+- `cv`, `scoring`, `feature_columns`, `groups`, `sort_by`.
+- `included`, `excluded` (with status and reason), `needs_estimator` (eligible
+  estimator-level steps that need an estimator), `unsupported` (columns left
+  out), `assumptions` still open.
+
+## The HTML report
+
+`report.to_html(path)` writes one file with four sections:
+
+- **Summary** — target, task, development rows; how many questions await you and
+  how many decisions are critical or need review; one next step; the data
+  partition note.
+- **Actions** — what needs your attention (open questions, review items, and
+  conditional decisions an experiment verified or refuted), the questions with
+  their options (and an LLM suggestion awaiting your confirmation, if you ran
+  one), and the metric plan.
+- **Roadmap** — one short card per decision: status, title, one-line why with
+  the decisive number, one-line do, experiment summary, and at most one figure.
+  Evidence, experiment details, related figures and rule/benchmark details are
+  one click away inside the card.
+- **Details** — limitations, then collapsed: all diagnostics, findings,
+  evidence, dataset profile, manifest and history.
+
+No raw rows are embedded; category labels are shown only for low-cardinality
+columns (`ProfileOptions(redact_values=True)` hides them all). Charts use
+plotly.js: with `plotlyjs="cdn"` (default) the report is a single file but needs
+internet for the charts; `plotlyjs="inline"` embeds plotly.js (~5 MB, needs the
+`report` extra) for fully offline use; every chart also has a data table, which
+works offline either way.
+
+## Optional LLM interpretation
+
+```python
+report.llm_payload(pf.LLMConfig(share_column_names=False))   # inspect first
+report = report.interpret(pf.LLMConfig(provider="anthropic"))
+report = report.interpret(pf.LLMConfig(provider="local", model="llama3.1"))  # Ollama etc.
+```
+
+The LLM receives only the computed report — never rows, never category labels —
+and must answer in a fixed JSON schema. It can summarize, prioritize existing
+decisions, suggest answers to open questions (shown as awaiting your
+confirmation, never applied) and raise *semantic* flags on column names, which
+are marked manual-review. Claims that cite unknown evidence or decision ids are
+dropped and counted. With `share_column_names=False` columns are aliased and
+free-text messages are withheld; the report lists exactly what was sent. The
+free-text `context=` never changes a decision by itself. Without an LLM the
+report is complete.
+
+Provider status: the interpretation logic and the local OpenAI-compatible
+provider are tested (with a fake provider and a loopback server). The Anthropic
+and Gemini providers are thin wrappers that have **not** been exercised against
+the live APIs in this repository's tests.
+
+## After training
+
+`pf.review(y_true, y_pred, report, y_score=..., segments=...)` checks the
+roadmap's hypotheses against real errors, each against an explicit reference
+(e.g. "tails dominate" vs. what Gaussian errors would give), adds residual /
+confusion / precision–recall / segment analysis, and lists every assumption
+that was never verified.
+
+---
+
+## Benchmark
+
+`python benchmarks/run.py` runs versioned fixture cases (synthetic data with one
+injected hazard each, plus datasets bundled with scikit-learn) and checks that
+each rule fires, stays silent, or reports *near threshold* as expected. Results,
+including failures, are written to `benchmarks/results/<manifest>.json|md`;
+`--verify` adds experiment outcomes on real datasets (a plain run keeps the
+committed experiment section). Reports cite a rule's "detection pass rate on
+benchmark <manifest>: k/n checks".
+
+Current manifest `2026.11-2`: 20/20 cases pass. This is a detection pass rate on
+known-hazard fixtures — not a claim about real-world accuracy, nor about how
+often a decision improves a model.
+
+## Case studies
+
+`python examples/case_studies.py` (numbers below are from that script; fixed seeds):
+
+| study | rule | before | after |
+|---|---|---|---|
+| California Housing, skewed target, log-transform (HGB) | `target_skew` | MAE 0.3125 | MAE 0.3074 (−1.6%) |
+| Breast Cancer + 20% duplicated rows, de-duplicate (RF) | `duplicate_rows` | CV accuracy 0.9736 | 0.9579 (1.57 pts of inflation) |
+| Diabetes + injected leaked feature, remove it (linear) | `leakage_hint` (critical) | CV R² 0.9999 | 0.4892 |
+
+## Scope and limitations
+
+- v1 covers tabular classification and regression. No images, text, time-series
+  forecasting, clustering or model selection.
+- Leakage screening sees single-feature association and numeric correlation
+  only; leakage through feature combinations, time, preprocessing or repeated
+  entities is not detected.
+- Column kinds and the task are inferred heuristically; check the schema table.
+- Experiments test one change at a time; interactions are not measured.
+- `groups` and `time_col` cannot be combined in v1.
+- Raw datetime, free-text and string-identifier columns are not encoded.
+
+Design notes: [ARCHITECTURE.md](https://github.com/hkln4/prefit/blob/main/ARCHITECTURE.md) · changes: [CHANGELOG.md](https://github.com/hkln4/prefit/blob/main/CHANGELOG.md) ·
+contributing: [CONTRIBUTING.md](https://github.com/hkln4/prefit/blob/main/CONTRIBUTING.md)