omegaprompt 2.1.2__tar.gz → 2.2.0__tar.gz

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/CHANGELOG.md

@@ -1,5 +1,14 @@
 # Changelog
 
+## [2.2.0] - 2026-10-28
+
+- Invalidate evaluation caches when public context changes, provide explicit fresh evaluation and cache invalidation, and keep boundary-clamp verdicts distinct.
+- Bound submitted item results to max_workers and stream input-order aggregation. Account attempted calls and returned usage even on failure; never cache incomplete evaluations.
+- Reject nonfinite gate inputs and contradictory disjoint IDs; calibrate reports input errors with exit 2 before provider calls for split overlap.
+- Add explicit PreflightReport measurement_status/probe_accounting. Failed or insufficient measurements require manual review; old reports retain unknown state.
+- Preserve CLI aliases, artifact schema 2.0, and default auto mode. classify_traps gains optional paired/disjoint inputs for mini-antemortem-cli 0.11.0+.
+- Retain advisory-only adaptation controls as advisory. See docs/release/2.2.0.md for measured scope and migration.
+
 ## [2.1.2] - 2026-10-08
 
 Bound FastMCP to SDK >=1,<2 to fix clean MCP installs. Pin the Action default to this package version. Add standalone artifact inspection and explicit preflight data handoff; remove unsupported framework comparisons.

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: omegaprompt
-Version: 2.1.2
+Version: 2.2.0
 Summary: Provider-neutral prompt calibration engine with sensitivity-ranked meta-axes, hard-gate x soft-score fitness, walk-forward ship gates, guarded/expedition profiles, and explicit capability/risk reporting across Anthropic, OpenAI, Gemini, and local OpenAI-compatible backends. Optional sub-tools (mini-omega-lock, mini-antemortem-cli) distribute separately and plug in via the omegaprompt.preflight interface.
 Project-URL: Homepage, https://github.com/hibou04-ops/omegaprompt
 Project-URL: Repository, https://github.com/hibou04-ops/omegaprompt
@@ -246,7 +246,7 @@ Description-Content-Type: text/markdown
 
 # omegaprompt
 
-**The overfit gate for your prompts.** Your prompt aced your eval set — that's exactly why you can't trust it yet. `omegaprompt` re-tests the winning prompt on examples it never tuned on, and **fails your CI build** if it doesn't hold up.
+**Provider-neutral prompt calibration with explicit gates.** Search configurations on training data, score the winner on a separately supplied test slice, and inspect declared gap, hard-gate and profile checks. Paired replay measures repeat consistency; independent holdout needs cases excluded from selection.
 
 [![CI](https://github.com/hibou04-ops/omegaprompt/actions/workflows/ci.yml/badge.svg)](https://github.com/hibou04-ops/omegaprompt/actions/workflows/ci.yml)
 [![License: Apache 2.0](https://img.shields.io/pypi/l/omegaprompt?color=blue&label=license&cacheSeconds=3600)](LICENSE)
@@ -257,6 +257,8 @@ Description-Content-Type: text/markdown
 [![MCP](https://img.shields.io/badge/MCP-server-blueviolet.svg)](#103-mcp-server-claude-code-cursor)
 [![Parent framework](https://img.shields.io/badge/framework-omega--lock-blueviolet.svg)](https://github.com/hibou04-ops/omega-lock)
 
+> **2.2.0 (2026-10-28): evaluation integrity and bounded work.** Context changes invalidate memoized observations. `PromptTarget.evaluate(..., use_cache=False)` performs fresh calls; cache replay adds no independent observation. Parallel evaluation retains at most `max_workers` submitted results and preserves input order. Failed target/judge attempts and returned usage remain in target accounting; no partial success is cached. Explicit disjoint mode rejects overlapping IDs. [Evidence and migration](docs/release/2.2.0.md).
+
 Docs: **[Easy start](https://github.com/hibou04-ops/omegaprompt/blob/main/EASY_README.md)** · [Full reference](https://github.com/hibou04-ops/omegaprompt/blob/main/README.md) · [한국어](https://github.com/hibou04-ops/omegaprompt/blob/main/README_KR.md) · [쉬운 한국어](https://github.com/hibou04-ops/omegaprompt/blob/main/EASY_README_KR.md) · [Examples gallery](examples/) · [Claim ledger (trust evidence)](docs/claims/README_CLAIMS.generated.md)
 
 Keywords: **prompt overfitting · prompt regression testing · LLM eval CI · prompt evaluation · prompt A/B test in CI · held-out validation for prompts · CI ship gate for prompts**
@@ -266,7 +268,6 @@ pip install omegaprompt              # core
 pip install "omegaprompt[mcp]"       # + MCP server (Claude Code / Cursor)
 ```
 
-> **v2.1.2 (2026-10-08)** — MCP installs retain SDK 1.x compatibility; the Action installs this exact package version by default. Standalone examples and docking instructions clarify synthetic versus live evidence. CLI/MCP names, schema 2.0 and the eight MCP tools are unchanged.
 
 <!-- public-claim-ledger:start -->
 > Claim evidence source: [docs/claims/public_claim_ledger.json](docs/claims/public_claim_ledger.json), rendered by `python tools/generate_readme_claims.py`.
@@ -285,9 +286,9 @@ Calibrate prompt configurations through omega-lock and inspect CalibrationArtifa
 Requires Python 3.11+. Installation needs internet.
 
 ```bash
-python -m pip install omegaprompt==2.1.2
+python -m pip install omegaprompt==2.2.0
 omegaprompt --help
-python -c "from urllib.request import urlretrieve; urlretrieve('https://raw.githubusercontent.com/hibou04-ops/omegaprompt/v2.1.2/examples/reference/reference_artifact.json', 'artifact.json')"
+python -c "from urllib.request import urlretrieve; urlretrieve('https://raw.githubusercontent.com/hibou04-ops/omegaprompt/v2.2.0/examples/reference/reference_artifact.json', 'artifact.json')"
 omegaprompt check-artifact artifact.json
 omegaprompt report artifact.json --format json
 ```
@@ -298,7 +299,7 @@ omega-lock>=0.3.0,<0.4.0 is required and installed automatically. Both mini pack
 
 [Docking contracts and runnable data handoff](https://github.com/hibou04-ops/omega-lock/blob/main/DOCKING.md) · [Full guide](README.md).
 
-MCP: install the distribution with `[mcp]` and use its existing server executable. FastMCP support is bounded to MCP SDK `>=1.0.0,<2.0.0`; the tool names and schemas are unchanged.
+MCP: install the distribution with `[mcp]` and use its existing server executable. FastMCP support is bounded to MCP SDK `>=1.0.0,<2.0.0`; tool names are retained; classify_traps gains optional validation inputs.
 
 
 ## Your prompt is overfit to your eval set — and you don't know it
@@ -309,7 +310,7 @@ A week later, production quality is *worse* than before. Why?
 
 Because 4.8 was the score on **the exact examples you tuned against**. You didn't measure a prompt — you memorized an answer key. That's overfitting, and your eval tool reported PASS because that's all it was asked to do. ML solved this in the 1990s with a train/test split. Most prompt workflows still ship without one.
 
-`omegaprompt` is the 60-second check that catches this **before** you ship:
+`omegaprompt` is a configurable workflow that catches this **before** you ship:
 
 1. It tunes your prompt on a **train** slice (across system prompt, few-shot, reasoning effort, output budget, response-schema mode, tool policy).
 2. It **re-tests the winner on a held-out slice it never tuned on**.
@@ -317,7 +318,7 @@ Because 4.8 was the score on **the exact examples you tuned against**. You didn'
 
 Then one line in CI makes a "small prompt tweak" fail the build if it silently degrades quality.
 
-> **Your eval says PASS. omegaprompt says it won't generalize.** That sentence is the whole product.
+> A gate failure identifies a failed declared check on supplied data; it does not establish future production behavior.
 
 ---
 
@@ -354,7 +355,7 @@ generalization_gap: 0.00%      # train vs held-out — small gap = it transferre
 kc4_status:         MISSING_PER_ITEM_SCORES
 ```
 
-**Honest read of this demo:** the bundled demo dataset has *disjoint* train/held-out items (no shared item ids), so the per-item transfer gate cannot fire — it reports `MISSING_PER_ITEM_SCORES` and the gate degenerates to the gap check alone (which here is 0.00%). The per-item transfer gate is what *actually* fires on a real **paired** dataset (train and held-out sharing item ids). So don't read this demo's clean numbers as "the transfer gate passed" — read it as "the gap check passed and the transfer gate had nothing to score." See *How it works*.
+**Demo scope:** these deterministic fixtures exercise artifact and gap-gate behavior. Missing per-item scores are unmeasured correlation. A paired replay measures repeat consistency on the same cases; an independent holdout needs cases excluded from selection. Neither proves production generalization.
 
 Turn that same artifact into a one-command CI verdict:
 
@@ -405,7 +406,7 @@ A prompt change is a code change. Gate it like one. The dedicated **`omegaprompt
 
 ```yaml
 # .github/workflows/prompt-audit.yml
-- uses: hibou04-ops/omegaprompt@v2.1.2
+- uses: hibou04-ops/omegaprompt@v2.2.0
   with:
     artifact: artifact.json          # a CalibrationArtifact you produced in a prior step
     format: json                     # machine-readable gate summary
@@ -476,7 +477,7 @@ You give it a **dataset + a judging rubric + candidate prompts**. It searches st
 
 Then it re-tests the winning configuration on the **held-out** slice and applies two pre-declared gates:
 
-- **Held-out transfer gate** (`--min-kc4`): per-item held-out scores must still track the target. If train looks great but the held-out items stop correlating, the prompt overfit — the run is flagged `FAIL_KC4_GATE`. *(This needs train and held-out to share item ids — a "paired" replay. On an ordinary disjoint split it degrades to the gap check below, which still catches overfit — that's exactly what the offline demo above shows.)*
+- **Paired consistency gate** (`--min-kc4`): Pearson correlation on the same cases must meet the declared threshold. Failure means that consistency check failed, not proof of overfitting. An independent disjoint holdout has no paired correlation and uses the gap and hard-gate checks.
 - **Max gap gate** (`--max-gap`): the train↔held-out fitness gap must stay under your declared ceiling.
 
 Both thresholds are set **before** scoring. There's no lowering the bar after you see results.
@@ -816,7 +817,7 @@ Every combination in the unlocked subspace is evaluated. Each evaluation issues
 
 ### 5.5 Held-out replay
 
-The training-best parameters are replayed on the held-out test slice. The replay uses the *same* `PromptTarget` adapter with a different dataset; there is no leakage possible because the test slice was never seen by the searcher.
+The training-best parameters are evaluated using a separate PromptTarget instance for the supplied test dataset. Independence depends on actual content and prior selection history. A paired replay deliberately reuses the same cases and is not an independent holdout.
 
 ### 5.6 Held-out transfer gate (per-item correlation)
 
@@ -824,9 +825,9 @@ The Pearson correlation between train per-item scores and test per-item scores (
 
 **Transfer-gate semantics by validation_mode (v1.5+).** The transfer gate is a per-item correlation, so it is only meaningful when train and test slices share the *same* item ids — a "paired replay". On an ordinary disjoint train/test split the slices have no shared ids and the per-item correlation is structurally unmeasurable; the gate degenerates to the gap-only check. To make this explicit, `CalibrateTuning.validation_mode` accepts:
 
-- `"auto"` (default, backward-compat): compute the per-item correlation only when slices share ≥3 ids; otherwise skip silently.
+- `"auto"` (default, backward-compat): compute the per-item correlation only when slices share ≥3 ids; otherwise record a noncomputed status.
 - `"paired"`: caller asserts shared ids by design. Raises `ValueError` if overlap < 3 — a paired run with no overlap is a setup bug, not a free pass.
-- `"disjoint"`: caller asserts no shared ids by design. The per-item correlation is never computed; the gate is gap-only.
+- `"disjoint"`: caller asserts no shared ids by design. Overlapping IDs raise an input error. For disjoint IDs the gate is gap-only; unmeasured correlation is not a correlation pass.
 
 If you run a normal held-out split (disjoint ids), set `validation_mode="disjoint"` to make the artifact's lack of `kc4_correlation` self-documenting. Use `"paired"` when you score two prompts on the *same* items and want the per-item correlation as a stability signal.
 
@@ -1403,7 +1404,7 @@ The default test suite runs with `pytest -q` and uses mocked provider clients ra
 | `judges/` | `RuleJudge` (no_refusal / non_empty / json_object / regex / duplicate-check rejection / missing-check raise); `LLMJudge` (strict-schema dispatch, payload composition, non-`JudgeResult` response raise, strict-mode ship-grade judge refusal); `EnsembleJudge` (rule-first short-circuit, LLM escalation on rule-pass, merged gate_results, non-`RuleJudge` rejection). |
 | `targets/` | `PromptTarget` end-to-end with mocked provider + judge; meta-axis resolution and clamping for out-of-range inputs; usage accumulation across evaluations; `evaluation_history` retention; latency measurement; degraded-capability propagation. |
 | `commands/` | CLI help lists `calibrate` / `report` / `diff` / `check-artifact`; `--version` shows `omegaprompt`; `report` renders schema-v2.0 artifacts; `diff` detects regressions on fitness, cost ratios, latency ratios, boundary-crossing flips; `check-artifact` performs zero-network integrity checks. |
-| `preflight/` | **Plugin interface only** — no probe or classifier code inside `omegaprompt`. `contracts`: severity ordering, status enum, `PreflightReport.worst_severity` / `any_real_or_new`, Pydantic `extra="forbid"` enforcement; bounds on `JudgeQualityMeasurement.consistency` (0..1), `EndpointMeasurement.schema_reliability` (0..1). `adaptation`: noise-adaptive `min_kc4` across four thresholds; consistency-driven `rescore_count`; schema-fallback trigger; wall-time-driven `unlock_k` reduction; small-sample gap widening; variant-skip axis marking; `apply_adaptation_plan` invariants (never weakens `min_kc4`, never widens `max_gap`, never raises `unlock_k`). Sub-tool probe + classifier implementations (with their own test suites) live in the `mini-omega-lock` and `mini-antemortem-cli` repositories. |
+| `preflight/` | **Plugin interface only** — no probe or classifier code inside `omegaprompt`. `contracts`: severity ordering, status enum, `PreflightReport.worst_severity` / `any_real_or_new`, Pydantic `extra="forbid"` enforcement; bounds on `JudgeQualityMeasurement.consistency` (0..1), `EndpointMeasurement.schema_reliability` (0..1). `adaptation`: noise-adaptive `min_kc4` across four thresholds; consistency-driven `rescore_count`; schema-fallback trigger; wall-time-driven `unlock_k` reduction; small-sample manual-review escalation; variant-skip axis marking; `apply_adaptation_plan` invariants (never weakens `min_kc4`, never widens `max_gap`, never raises `unlock_k`). Sub-tool probe + classifier implementations (with their own test suites) live in the `mini-omega-lock` and `mini-antemortem-cli` repositories. |
 | `test_calibrate_integration.py` | **Drives the real `omega_lock.run_p1`** with a deterministic in-memory `CalibrableTarget` (no mocks on the search engine). Asserts the artifact's `calibrated_params`, `neutral_baseline_params`, `walk_forward`, and `sensitivity_ranking` match `P1Result`'s actual shape — the regression that this test catches is drift between the adapter layer and the search engine that per-module unit tests cannot reach. |
 
 Run the default no-network suite with `uv run pytest -q -m "not live"` (or `python -m pytest -q -m "not live"`). The wall-clock time is dominated by Pydantic model compilation on first import.
@@ -1948,3 +1949,5 @@ Apache 2.0. See [LICENSE](LICENSE).
 ## Colophon
 
 Designed, implemented, and shipped solo. Adapter layer over `omega-lock`; zero calibration-engine reimplementation. Every non-trivial change is pre-authored through `antemortem-cli`'s recon discipline. Tests run offline; no live API calls in CI.
+
+2.2.0 runtime split contract: explicit paired mode requires matching ID sets and identical input/reference content on at least three cases; explicit disjoint mode rejects overlapping IDs or exact duplicate input content before provider calls. Historical auto mode is retained and does not establish independence.

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/README.md

@@ -1,6 +1,6 @@
 # omegaprompt
 
-**The overfit gate for your prompts.** Your prompt aced your eval set — that's exactly why you can't trust it yet. `omegaprompt` re-tests the winning prompt on examples it never tuned on, and **fails your CI build** if it doesn't hold up.
+**Provider-neutral prompt calibration with explicit gates.** Search configurations on training data, score the winner on a separately supplied test slice, and inspect declared gap, hard-gate and profile checks. Paired replay measures repeat consistency; independent holdout needs cases excluded from selection.
 
 [![CI](https://github.com/hibou04-ops/omegaprompt/actions/workflows/ci.yml/badge.svg)](https://github.com/hibou04-ops/omegaprompt/actions/workflows/ci.yml)
 [![License: Apache 2.0](https://img.shields.io/pypi/l/omegaprompt?color=blue&label=license&cacheSeconds=3600)](LICENSE)
@@ -11,6 +11,8 @@
 [![MCP](https://img.shields.io/badge/MCP-server-blueviolet.svg)](#103-mcp-server-claude-code-cursor)
 [![Parent framework](https://img.shields.io/badge/framework-omega--lock-blueviolet.svg)](https://github.com/hibou04-ops/omega-lock)
 
+> **2.2.0 (2026-10-28): evaluation integrity and bounded work.** Context changes invalidate memoized observations. `PromptTarget.evaluate(..., use_cache=False)` performs fresh calls; cache replay adds no independent observation. Parallel evaluation retains at most `max_workers` submitted results and preserves input order. Failed target/judge attempts and returned usage remain in target accounting; no partial success is cached. Explicit disjoint mode rejects overlapping IDs. [Evidence and migration](docs/release/2.2.0.md).
+
 Docs: **[Easy start](https://github.com/hibou04-ops/omegaprompt/blob/main/EASY_README.md)** · [Full reference](https://github.com/hibou04-ops/omegaprompt/blob/main/README.md) · [한국어](https://github.com/hibou04-ops/omegaprompt/blob/main/README_KR.md) · [쉬운 한국어](https://github.com/hibou04-ops/omegaprompt/blob/main/EASY_README_KR.md) · [Examples gallery](examples/) · [Claim ledger (trust evidence)](docs/claims/README_CLAIMS.generated.md)
 
 Keywords: **prompt overfitting · prompt regression testing · LLM eval CI · prompt evaluation · prompt A/B test in CI · held-out validation for prompts · CI ship gate for prompts**
@@ -20,7 +22,6 @@ pip install omegaprompt              # core
 pip install "omegaprompt[mcp]"       # + MCP server (Claude Code / Cursor)
 ```
 
-> **v2.1.2 (2026-10-08)** — MCP installs retain SDK 1.x compatibility; the Action installs this exact package version by default. Standalone examples and docking instructions clarify synthetic versus live evidence. CLI/MCP names, schema 2.0 and the eight MCP tools are unchanged.
 
 <!-- public-claim-ledger:start -->
 > Claim evidence source: [docs/claims/public_claim_ledger.json](docs/claims/public_claim_ledger.json), rendered by `python tools/generate_readme_claims.py`.
@@ -39,9 +40,9 @@ Calibrate prompt configurations through omega-lock and inspect CalibrationArtifa
 Requires Python 3.11+. Installation needs internet.
 
 ```bash
-python -m pip install omegaprompt==2.1.2
+python -m pip install omegaprompt==2.2.0
 omegaprompt --help
-python -c "from urllib.request import urlretrieve; urlretrieve('https://raw.githubusercontent.com/hibou04-ops/omegaprompt/v2.1.2/examples/reference/reference_artifact.json', 'artifact.json')"
+python -c "from urllib.request import urlretrieve; urlretrieve('https://raw.githubusercontent.com/hibou04-ops/omegaprompt/v2.2.0/examples/reference/reference_artifact.json', 'artifact.json')"
 omegaprompt check-artifact artifact.json
 omegaprompt report artifact.json --format json
 ```
@@ -52,7 +53,7 @@ omega-lock>=0.3.0,<0.4.0 is required and installed automatically. Both mini pack
 
 [Docking contracts and runnable data handoff](https://github.com/hibou04-ops/omega-lock/blob/main/DOCKING.md) · [Full guide](README.md).
 
-MCP: install the distribution with `[mcp]` and use its existing server executable. FastMCP support is bounded to MCP SDK `>=1.0.0,<2.0.0`; the tool names and schemas are unchanged.
+MCP: install the distribution with `[mcp]` and use its existing server executable. FastMCP support is bounded to MCP SDK `>=1.0.0,<2.0.0`; tool names are retained; classify_traps gains optional validation inputs.
 
 
 ## Your prompt is overfit to your eval set — and you don't know it
@@ -63,7 +64,7 @@ A week later, production quality is *worse* than before. Why?
 
 Because 4.8 was the score on **the exact examples you tuned against**. You didn't measure a prompt — you memorized an answer key. That's overfitting, and your eval tool reported PASS because that's all it was asked to do. ML solved this in the 1990s with a train/test split. Most prompt workflows still ship without one.
 
-`omegaprompt` is the 60-second check that catches this **before** you ship:
+`omegaprompt` is a configurable workflow that catches this **before** you ship:
 
 1. It tunes your prompt on a **train** slice (across system prompt, few-shot, reasoning effort, output budget, response-schema mode, tool policy).
 2. It **re-tests the winner on a held-out slice it never tuned on**.
@@ -71,7 +72,7 @@ Because 4.8 was the score on **the exact examples you tuned against**. You didn'
 
 Then one line in CI makes a "small prompt tweak" fail the build if it silently degrades quality.
 
-> **Your eval says PASS. omegaprompt says it won't generalize.** That sentence is the whole product.
+> A gate failure identifies a failed declared check on supplied data; it does not establish future production behavior.
 
 ---
 
@@ -108,7 +109,7 @@ generalization_gap: 0.00%      # train vs held-out — small gap = it transferre
 kc4_status:         MISSING_PER_ITEM_SCORES
 ```
 
-**Honest read of this demo:** the bundled demo dataset has *disjoint* train/held-out items (no shared item ids), so the per-item transfer gate cannot fire — it reports `MISSING_PER_ITEM_SCORES` and the gate degenerates to the gap check alone (which here is 0.00%). The per-item transfer gate is what *actually* fires on a real **paired** dataset (train and held-out sharing item ids). So don't read this demo's clean numbers as "the transfer gate passed" — read it as "the gap check passed and the transfer gate had nothing to score." See *How it works*.
+**Demo scope:** these deterministic fixtures exercise artifact and gap-gate behavior. Missing per-item scores are unmeasured correlation. A paired replay measures repeat consistency on the same cases; an independent holdout needs cases excluded from selection. Neither proves production generalization.
 
 Turn that same artifact into a one-command CI verdict:
 
@@ -159,7 +160,7 @@ A prompt change is a code change. Gate it like one. The dedicated **`omegaprompt
 
 ```yaml
 # .github/workflows/prompt-audit.yml
-- uses: hibou04-ops/omegaprompt@v2.1.2
+- uses: hibou04-ops/omegaprompt@v2.2.0
   with:
     artifact: artifact.json          # a CalibrationArtifact you produced in a prior step
     format: json                     # machine-readable gate summary
@@ -230,7 +231,7 @@ You give it a **dataset + a judging rubric + candidate prompts**. It searches st
 
 Then it re-tests the winning configuration on the **held-out** slice and applies two pre-declared gates:
 
-- **Held-out transfer gate** (`--min-kc4`): per-item held-out scores must still track the target. If train looks great but the held-out items stop correlating, the prompt overfit — the run is flagged `FAIL_KC4_GATE`. *(This needs train and held-out to share item ids — a "paired" replay. On an ordinary disjoint split it degrades to the gap check below, which still catches overfit — that's exactly what the offline demo above shows.)*
+- **Paired consistency gate** (`--min-kc4`): Pearson correlation on the same cases must meet the declared threshold. Failure means that consistency check failed, not proof of overfitting. An independent disjoint holdout has no paired correlation and uses the gap and hard-gate checks.
 - **Max gap gate** (`--max-gap`): the train↔held-out fitness gap must stay under your declared ceiling.
 
 Both thresholds are set **before** scoring. There's no lowering the bar after you see results.
@@ -570,7 +571,7 @@ Every combination in the unlocked subspace is evaluated. Each evaluation issues
 
 ### 5.5 Held-out replay
 
-The training-best parameters are replayed on the held-out test slice. The replay uses the *same* `PromptTarget` adapter with a different dataset; there is no leakage possible because the test slice was never seen by the searcher.
+The training-best parameters are evaluated using a separate PromptTarget instance for the supplied test dataset. Independence depends on actual content and prior selection history. A paired replay deliberately reuses the same cases and is not an independent holdout.
 
 ### 5.6 Held-out transfer gate (per-item correlation)
 
@@ -578,9 +579,9 @@ The Pearson correlation between train per-item scores and test per-item scores (
 
 **Transfer-gate semantics by validation_mode (v1.5+).** The transfer gate is a per-item correlation, so it is only meaningful when train and test slices share the *same* item ids — a "paired replay". On an ordinary disjoint train/test split the slices have no shared ids and the per-item correlation is structurally unmeasurable; the gate degenerates to the gap-only check. To make this explicit, `CalibrateTuning.validation_mode` accepts:
 
-- `"auto"` (default, backward-compat): compute the per-item correlation only when slices share ≥3 ids; otherwise skip silently.
+- `"auto"` (default, backward-compat): compute the per-item correlation only when slices share ≥3 ids; otherwise record a noncomputed status.
 - `"paired"`: caller asserts shared ids by design. Raises `ValueError` if overlap < 3 — a paired run with no overlap is a setup bug, not a free pass.
-- `"disjoint"`: caller asserts no shared ids by design. The per-item correlation is never computed; the gate is gap-only.
+- `"disjoint"`: caller asserts no shared ids by design. Overlapping IDs raise an input error. For disjoint IDs the gate is gap-only; unmeasured correlation is not a correlation pass.
 
 If you run a normal held-out split (disjoint ids), set `validation_mode="disjoint"` to make the artifact's lack of `kc4_correlation` self-documenting. Use `"paired"` when you score two prompts on the *same* items and want the per-item correlation as a stability signal.
 
@@ -1157,7 +1158,7 @@ The default test suite runs with `pytest -q` and uses mocked provider clients ra
 | `judges/` | `RuleJudge` (no_refusal / non_empty / json_object / regex / duplicate-check rejection / missing-check raise); `LLMJudge` (strict-schema dispatch, payload composition, non-`JudgeResult` response raise, strict-mode ship-grade judge refusal); `EnsembleJudge` (rule-first short-circuit, LLM escalation on rule-pass, merged gate_results, non-`RuleJudge` rejection). |
 | `targets/` | `PromptTarget` end-to-end with mocked provider + judge; meta-axis resolution and clamping for out-of-range inputs; usage accumulation across evaluations; `evaluation_history` retention; latency measurement; degraded-capability propagation. |
 | `commands/` | CLI help lists `calibrate` / `report` / `diff` / `check-artifact`; `--version` shows `omegaprompt`; `report` renders schema-v2.0 artifacts; `diff` detects regressions on fitness, cost ratios, latency ratios, boundary-crossing flips; `check-artifact` performs zero-network integrity checks. |
-| `preflight/` | **Plugin interface only** — no probe or classifier code inside `omegaprompt`. `contracts`: severity ordering, status enum, `PreflightReport.worst_severity` / `any_real_or_new`, Pydantic `extra="forbid"` enforcement; bounds on `JudgeQualityMeasurement.consistency` (0..1), `EndpointMeasurement.schema_reliability` (0..1). `adaptation`: noise-adaptive `min_kc4` across four thresholds; consistency-driven `rescore_count`; schema-fallback trigger; wall-time-driven `unlock_k` reduction; small-sample gap widening; variant-skip axis marking; `apply_adaptation_plan` invariants (never weakens `min_kc4`, never widens `max_gap`, never raises `unlock_k`). Sub-tool probe + classifier implementations (with their own test suites) live in the `mini-omega-lock` and `mini-antemortem-cli` repositories. |
+| `preflight/` | **Plugin interface only** — no probe or classifier code inside `omegaprompt`. `contracts`: severity ordering, status enum, `PreflightReport.worst_severity` / `any_real_or_new`, Pydantic `extra="forbid"` enforcement; bounds on `JudgeQualityMeasurement.consistency` (0..1), `EndpointMeasurement.schema_reliability` (0..1). `adaptation`: noise-adaptive `min_kc4` across four thresholds; consistency-driven `rescore_count`; schema-fallback trigger; wall-time-driven `unlock_k` reduction; small-sample manual-review escalation; variant-skip axis marking; `apply_adaptation_plan` invariants (never weakens `min_kc4`, never widens `max_gap`, never raises `unlock_k`). Sub-tool probe + classifier implementations (with their own test suites) live in the `mini-omega-lock` and `mini-antemortem-cli` repositories. |
 | `test_calibrate_integration.py` | **Drives the real `omega_lock.run_p1`** with a deterministic in-memory `CalibrableTarget` (no mocks on the search engine). Asserts the artifact's `calibrated_params`, `neutral_baseline_params`, `walk_forward`, and `sensitivity_ranking` match `P1Result`'s actual shape — the regression that this test catches is drift between the adapter layer and the search engine that per-module unit tests cannot reach. |
 
 Run the default no-network suite with `uv run pytest -q -m "not live"` (or `python -m pytest -q -m "not live"`). The wall-clock time is dominated by Pydantic model compilation on first import.
@@ -1702,3 +1703,5 @@ Apache 2.0. See [LICENSE](LICENSE).
 ## Colophon
 
 Designed, implemented, and shipped solo. Adapter layer over `omega-lock`; zero calibration-engine reimplementation. Every non-trivial change is pre-authored through `antemortem-cli`'s recon discipline. Tests run offline; no live API calls in CI.
+
+2.2.0 runtime split contract: explicit paired mode requires matching ID sets and identical input/reference content on at least three cases; explicit disjoint mode rejects overlapping IDs or exact duplicate input content before provider calls. Historical auto mode is retained and does not establish independence.

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/README_KR.md

@@ -1,6 +1,6 @@
 # omegaprompt
 
-**당신의 prompt를 위한 overfit gate.** prompt가 당신의 eval set에서 만점을 받았다 — 바로 그것이 아직 그 prompt를 믿어선 안 되는 이유입니다. `omegaprompt`는 이긴 prompt를 그 prompt가 한 번도 튜닝하지 않은 예시들로 다시 테스트하고, 그 결과가 유지되지 않으면 **당신의 CI 빌드를 실패**시킵니다.
+**명시적 gate를 갖춘 provider 중립 프롬프트 보정.** train 데이터에서 설정을 선택하고 별도로 제공한 test 데이터에서 gap·hard gate·실행 정책을 검사합니다. paired 재측정은 반복 일관성이며, 독립 holdout은 선택에 사용하지 않은 사례가 필요합니다.
 
 [![CI](https://github.com/hibou04-ops/omegaprompt/actions/workflows/ci.yml/badge.svg)](https://github.com/hibou04-ops/omegaprompt/actions/workflows/ci.yml)
 [![License: Apache 2.0](https://img.shields.io/pypi/l/omegaprompt?color=blue&label=license&cacheSeconds=3600)](LICENSE)
@@ -11,6 +11,8 @@
 [![MCP](https://img.shields.io/badge/MCP-server-blueviolet.svg)](https://github.com/hibou04-ops/omegaprompt/blob/main/README.md#103-mcp-server-claude-code-cursor)
 [![Parent framework](https://img.shields.io/badge/framework-omega--lock-blueviolet.svg)](https://github.com/hibou04-ops/omega-lock)
 
+> **2.2.0 (2026-10-28): 평가 무결성과 대기 작업 상한.** 데이터·rubric·variants 등 평가 컨텍스트 변경 시 캐시를 무효화합니다. `PromptTarget.evaluate(..., use_cache=False)`는 실제 재측정이며 캐시 재생은 독립 관측으로 세지 않습니다. 병렬 대기 결과는 `max_workers`개 이하이고 입력 순서를 보존합니다. 실패한 target/judge 시도와 반환된 usage도 누적 계측하며 부분 성공은 캐시하지 않습니다. 명시적 disjoint 모드는 ID 중복을 입력 오류로 처리합니다. [검증 근거와 이행 안내](docs/release/2.2.0.md).
+
 문서: **[Easy start](https://github.com/hibou04-ops/omegaprompt/blob/main/EASY_README.md)** · [Full reference](https://github.com/hibou04-ops/omegaprompt/blob/main/README.md) · [한국어](https://github.com/hibou04-ops/omegaprompt/blob/main/README_KR.md) · [쉬운 한국어](https://github.com/hibou04-ops/omegaprompt/blob/main/EASY_README_KR.md) · [Examples gallery](examples/) · [Claim ledger (신뢰 근거)](docs/claims/README_CLAIMS.generated.md)
 
 키워드: **prompt overfitting · prompt regression testing · LLM eval CI · prompt evaluation · prompt A/B test in CI · held-out validation for prompts · CI ship gate for prompts**
@@ -20,7 +22,6 @@ pip install omegaprompt              # core
 pip install "omegaprompt[mcp]"       # + MCP server (Claude Code / Cursor)
 ```
 
-> **v2.1.2 (2026-10-08)** — MCP 설치는 SDK 1.x 호환 범위를 유지하고 Action은 해당 버전을 기본 설치합니다. 단독 예제와 도킹 안내에서 합성·실제 근거를 구분합니다. CLI/MCP 이름, schema 2.0, MCP 도구 8개는 유지합니다.
 
 <!-- public-claim-ledger:start -->
 > Claim evidence source: [docs/claims/public_claim_ledger.json](docs/claims/public_claim_ledger.json), rendered by `python tools/generate_readme_claims.py`.
@@ -39,16 +40,16 @@ omega-lock 엔진으로 프롬프트 설정을 보정하고 CalibrationArtifact
 Python 3.11 이상에서 실행합니다. 설치에는 인터넷이 필요합니다.
 
 ```bash
-python -m pip install omegaprompt==2.1.2
+python -m pip install omegaprompt==2.2.0
 omegaprompt --help
-python -c "from urllib.request import urlretrieve; urlretrieve('https://raw.githubusercontent.com/hibou04-ops/omegaprompt/v2.1.2/examples/reference/reference_artifact.json', 'artifact.json')"
+python -c "from urllib.request import urlretrieve; urlretrieve('https://raw.githubusercontent.com/hibou04-ops/omegaprompt/v2.2.0/examples/reference/reference_artifact.json', 'artifact.json')"
 omegaprompt check-artifact artifact.json
 omegaprompt report artifact.json --format json
 ```
 
 artifact 다운로드는 인터넷이 필요하며 이후 검사는 오프라인입니다. 저장된 합성 artifact의 누락된 per-item 점수는 transfer 통과를 뜻하지 않습니다. 실제 보정은 provider 설정과 비용 확인이 필요합니다. omega-lock>=0.3.0,<0.4.0은 필수이며 두 mini 도구는 선택입니다.
 
-[실제 연결 방식과 입력·출력](https://github.com/hibou04-ops/omega-lock/blob/main/DOCKING.md) · [전체 안내](README.md). 두 mini 도구의 현재 검증 조합은 omegaprompt 2.1.2이며, 별도 배포 패키지입니다.
+[실제 연결 방식과 입력·출력](https://github.com/hibou04-ops/omega-lock/blob/main/DOCKING.md) · [전체 안내](README.md). 두 mini 도구의 현재 검증 조합은 omegaprompt 2.2.0이며, 별도 배포 패키지입니다.
 
 MCP는 해당 배포명의 `[mcp]` extras와 기존 서버 명령을 사용합니다. 기존 FastMCP 계약은 SDK `>=1.0.0,<2.0.0` 범위로 유지합니다.
 
@@ -157,7 +158,7 @@ prompt 변경은 코드 변경입니다. 코드처럼 gate하세요. 전용 **`o
 
 ```yaml
 # .github/workflows/prompt-audit.yml
-- uses: hibou04-ops/omegaprompt@v2.1.2
+- uses: hibou04-ops/omegaprompt@v2.2.0
   with:
     artifact: artifact.json          # a CalibrationArtifact you produced in a prior step
     format: json                     # machine-readable gate summary
@@ -568,7 +569,7 @@ unlock된 부분공간의 모든 조합이 평가됩니다. 각 평가는 datase
 
 ### 5.5 held-out replay
 
-training-best parameter가 held-out 테스트 슬라이스에서 replay됩니다. replay는 *같은* `PromptTarget` adapter를 다른 dataset과 함께 씁니다; 테스트 슬라이스는 searcher가 결코 보지 않았으므로 leakage가 가능하지 않습니다.
+선택한 parameter를 별도 PromptTarget 인스턴스와 제공된 test 데이터에서 평가합니다. 독립성은 실제 입력 내용과 과거 선택 이력에 달려 있습니다. paired 재측정은 동일 사례를 다시 사용하므로 독립 holdout이 아닙니다.
 
 ### 5.6 held-out transfer gate (per-item 상관)
 
@@ -576,7 +577,7 @@ train per-item 점수와 test per-item 점수 사이의(공유 dataset id 위에
 
 **validation_mode별 transfer-gate 의미(v1.5+).** transfer gate는 per-item 상관이므로, train과 test 슬라이스가 *같은* item id를 공유할 때 — "paired replay" — 에만 의미가 있습니다. 보통의 disjoint train/test split에서는 슬라이스에 공유 id가 없어 per-item 상관이 구조적으로 측정 불가능합니다; gate는 gap-only 검사로 축소됩니다. 이를 명시하기 위해 `CalibrateTuning.validation_mode`는 다음을 받습니다:
 
-- `"auto"`(기본값, backward-compat): 슬라이스가 ≥3개 id를 공유할 때만 per-item 상관을 계산하고; 그렇지 않으면 조용히 건너뜁니다.
+- `"auto"`(기본값, backward-compat): 슬라이스가 ≥3개 id를 공유할 때만 per-item 상관을 계산하고; 그렇지 않으면 미측정 사유를 기록합니다.
 - `"paired"`: 호출자가 설계상 공유 id를 단언합니다. overlap이 3 미만이면 `ValueError`를 던집니다 — overlap 없는 paired 실행은 무료 통과가 아니라 setup 버그입니다.
 - `"disjoint"`: 호출자가 설계상 공유 id 없음을 단언합니다. per-item 상관은 결코 계산되지 않고; gate는 gap-only입니다.
 
@@ -1672,3 +1673,5 @@ Apache 2.0. [LICENSE](LICENSE)를 참고하세요.
 ## Colophon
 
 혼자 설계하고, 구현하고, ship했습니다. `omega-lock` 위의 adapter 계층; calibration-engine 재구현 0. 모든 non-trivial한 변경은 `antemortem-cli`의 recon 원칙을 통해 사전 작성됩니다. 테스트는 offline으로 돌고; CI에 라이브 API 호출이 없습니다.
+
+2.2.0 runtime split contract: explicit paired mode requires matching ID sets and identical input/reference content on at least three cases; explicit disjoint mode rejects overlapping IDs or exact duplicate input content before provider calls. Historical auto mode is retained and does not establish independence.

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/examples/ci/ship-gate.yml

@@ -24,12 +24,12 @@ jobs:
       # the committed reference artifact path.
       - name: Ship gate
         id: ship_gate
-        uses: hibou04-ops/omegaprompt@v2.1.2
+        uses: hibou04-ops/omegaprompt@v2.2.0
         with:
           artifact: examples/reference/reference_artifact.json
           format: json
           require-generalization: "true"
-          version: "2.1.2"
+          version: "2.2.0"
 
       - name: Report verdict
         if: always()

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "omegaprompt"
-version = "2.1.2"
+version = "2.2.0"
 description = "Provider-neutral prompt calibration engine with sensitivity-ranked meta-axes, hard-gate x soft-score fitness, walk-forward ship gates, guarded/expedition profiles, and explicit capability/risk reporting across Anthropic, OpenAI, Gemini, and local OpenAI-compatible backends. Optional sub-tools (mini-omega-lock, mini-antemortem-cli) distribute separately and plug in via the omegaprompt.preflight interface."
 readme = "README.md"
 requires-python = ">=3.11"

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/src/omegaprompt/__init__.py

@@ -114,7 +114,7 @@ from omegaprompt.runtime import (
     report,
 )
 
-__version__ = "2.1.2"
+__version__ = "2.2.0"
 
 __all__ = [
     # domain

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/src/omegaprompt/commands/calibrate.py

@@ -314,6 +314,9 @@ def calibrate(
             tuning=tuning,
             adaptation_plan=plan,
         )
+    except ValueError as exc:
+        typer.secho(f"INPUT_ERROR: {exc}", fg=typer.colors.RED, err=True)
+        raise typer.Exit(code=2) from exc
     except RuntimeError as exc:
         message = str(exc)
         if "requires omega-lock" in message or "Install with" in message:

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/src/omegaprompt/core/walkforward.py

@@ -45,7 +45,7 @@ declared thresholds) so a future reader can tell ``None`` from
 
 from __future__ import annotations
 
-from math import isnan
+from math import isfinite, isnan
 from typing import Literal
 
 from omegaprompt.domain.result import WalkForwardResult
@@ -136,6 +136,15 @@ def evaluate_walk_forward(
         Default ``"auto"`` preserves historical behaviour: KC-4 is
         computed only when slices share >=3 ids, otherwise skipped.
     """
+    if validation_mode not in {"auto", "paired", "disjoint"}:
+        raise ValueError(f"Unknown validation_mode: {validation_mode!r}")
+    if not all(isfinite(v) for v in (train_best_fitness, test_fitness, max_gap, min_kc4)):
+        raise ValueError("fitness and gate thresholds must be finite")
+    if max_gap < 0 or not -1 <= min_kc4 <= 1:
+        raise ValueError("max_gap must be nonnegative and min_kc4 within [-1, 1]")
+    for scores in (per_item_train, per_item_test):
+        if scores is not None and not all(isfinite(v) for v in scores.values()):
+            raise ValueError("per-item scores must be finite")
     if train_best_fitness == 0:
         gap = float("inf")
         gap_status: GapStatus = (
@@ -152,6 +161,8 @@ def evaluate_walk_forward(
     kc4: float | None = None
     kc4_status: KC4Status
     if validation_mode == "disjoint":
+        if shared_ids:
+            raise ValueError("validation_mode='disjoint' requires disjoint item ids; overlap found")
         # Caller asserts disjoint split; do not compute KC-4 even if
         # ids happen to overlap (e.g. id collision across splits is a
         # bug to surface elsewhere, not a free pass for the gate).

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/src/omegaprompt/mcp/server.py

@@ -300,6 +300,8 @@ def classify_traps(
     judge: str | dict,
     dataset: str,
     test: str | None = None,
+    validation_mode: str = "auto",
+    selected_few_shot_indices: list[int] | None = None,
 ) -> list[dict]:
     """Classify the calibration config against known trap patterns.
 
@@ -315,6 +317,8 @@ def classify_traps(
         judge: Judge provider.
         dataset: Path to the train dataset JSONL.
         test: Optional path to the held-out test JSONL.
+        validation_mode: auto/paired/disjoint; paired is repeat measurement,
+            not evidence of unseen-input generalization.
 
     Returns:
         List of AnalyticalFinding dicts. Each has ``trap_id``, ``label``
@@ -328,5 +332,7 @@ def classify_traps(
         judge=judge,
         dataset=dataset,
         test=test,
+        validation_mode=validation_mode,
+        selected_few_shot_indices=selected_few_shot_indices,
     )
     return [f.model_dump(mode="json") for f in findings]

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/src/omegaprompt/preflight/adaptation.py

@@ -201,6 +201,9 @@ def derive_adaptation_plan(
         None,
     )
     require_manual_review_reasons: list[str] = []
+    for metric, state in report.measurement_status.items():
+        if state in {"failed", "insufficient_samples"}:
+            require_manual_review_reasons.append(f"{metric}: measurement {state}; no quality pass established")
     if small_sample_finding is not None and small_sample_finding.severity in {
         PreflightSeverity.HIGH,
         PreflightSeverity.BLOCKER,

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/src/omegaprompt/preflight/contracts.py

@@ -9,6 +9,7 @@ full sub-units must emit records conforming to these shapes.
 from __future__ import annotations
 
 from enum import Enum
+from typing import Any, Literal
 
 from pydantic import BaseModel, ConfigDict, Field
 
@@ -133,6 +134,12 @@ class PreflightReport(BaseModel):
     status: PreflightStatus = PreflightStatus.PROCEED
     blocker_reasons: list[str] = Field(default_factory=list)
     warnings: list[str] = Field(default_factory=list)
+    # Empty maps preserve unknown historical measurement state; warnings are
+    # human text and must not serve as the machine-readable measurement flag.
+    measurement_status: dict[str, Literal[
+        "measured", "unmeasured", "insufficient_samples", "failed", "estimated", "not_applicable"
+    ]] = Field(default_factory=dict)
+    probe_accounting: dict[str, Any] = Field(default_factory=dict)
 
     def any_real_or_new(self) -> bool:
         return any(f.label in {"REAL", "NEW"} for f in self.analytical_findings)

{omegaprompt-2.1.2 → omegaprompt-2.2.0}/src/omegaprompt/runtime.py

@@ -14,6 +14,7 @@ Tier 2: measure_sensitivity, grade, preflight, classify_traps.
 from __future__ import annotations
 
 from pathlib import Path
+from inspect import signature
 from typing import Literal, Union
 
 from pydantic import BaseModel, Field
@@ -298,6 +299,16 @@ def calibrate(
 
     train_ds = _resolve_dataset(train)
     test_ds = _resolve_dataset(test) if test is not None else None
+    if tuning.validation_mode == "paired":
+        train_items = {item.id: (item.input, item.reference) for item in train_ds.items}
+        test_items = {item.id: (item.input, item.reference) for item in test_ds.items} if test_ds is not None else {}
+        if len(train_items) < 3 or train_items != test_items:
+            raise ValueError("validation_mode='paired' requires the same at least 3 item ids and input/reference content")
+    if tuning.validation_mode == "disjoint" and test_ds is not None:
+        if {item.id for item in train_ds.items} & {item.id for item in test_ds.items}:
+            raise ValueError("validation_mode='disjoint' requires disjoint item ids; overlap found")
+        if {item.input for item in train_ds.items} & {item.input for item in test_ds.items}:
+            raise ValueError("validation_mode='disjoint' requires independent inputs; duplicate content found")
     rubric_obj = _resolve_rubric(rubric)
     variants_obj = _resolve_variants(variants)
     target_provider = _resolve_provider(target)
@@ -947,6 +958,8 @@ def classify_traps(
     judge: LLMProvider | ProviderSpec | str | dict,
     dataset: Dataset | Path | str,
     test: Dataset | Path | str | None = None,
+    validation_mode: Literal["auto", "paired", "disjoint"] = "auto",
+    selected_few_shot_indices: list[int] | None = None,
 ) -> list[AnalyticalFinding]:
     """Classify the calibration config against known trap patterns.
 
@@ -972,6 +985,17 @@ def classify_traps(
     train_dataset = _resolve_dataset(dataset)
     test_dataset = _resolve_dataset(test) if test is not None else None
 
+    if validation_mode not in {"auto", "paired", "disjoint"}:
+        raise ValueError(f"Unknown validation_mode: {validation_mode!r}")
+    mode_options = {}
+    if "validation_mode" in signature(analytical_preflight).parameters:
+        mode_options["validation_mode"] = validation_mode
+    elif validation_mode != "auto":
+        raise ValueError("Explicit validation_mode requires mini-antemortem-cli>=0.11.0")
+    if selected_few_shot_indices is not None:
+        if "selected_few_shot_indices" not in signature(analytical_preflight).parameters:
+            raise ValueError("Selected few-shot context requires mini-antemortem-cli>=0.11.0")
+        mode_options["selected_few_shot_indices"] = selected_few_shot_indices
     return analytical_preflight(
         target_provider=target_provider.name,
         target_model=target_provider.model,
@@ -981,6 +1005,7 @@ def classify_traps(
         test_dataset=test_dataset,
         rubric=rubric_obj,
         variants=variants_obj,
+        **mode_options,
     )