kc-beta 0.1.0

package/template/skills/en/meta-meta/dashboard-reporting/SKILL.md

@@ -0,0 +1,106 @@
+---
+name: dashboard-reporting
+description: Generate HTML dashboards for developer users to visualize verification results, system progress, and quality metrics. Use when a testing round completes, when production batches finish processing, when the developer user wants to see the system's status, or at any point where visual reporting would help communicate progress. Dashboards should be self-contained HTML files that can be opened by double-clicking. Also use when the developer user asks about results, accuracy, or system health.
+---
+
+# Dashboard Reporting
+
+The dashboard is the developer user's window into the system. They should not need to read logs or parse JSON to understand what is happening. Give them a clear, visual summary that leads with what matters.
+
+## Dashboard Types
+
+### Results Dashboard
+Generated after each batch of documents is processed.
+
+Key elements:
+- **Summary bar**: Total documents, pass rate, fail rate, missing rate, error rate.
+- **Per-rule breakdown**: Table showing each rule's pass/fail counts, accuracy, and average confidence.
+- **Failed cases**: List of documents that failed, with the rule, extracted value, expected value, and comment. Sortable and filterable.
+- **Confidence distribution**: Histogram showing how many results fall in each confidence band.
+
+### Progress Dashboard
+Generated on demand to show the system's evolution.
+
+Key elements:
+- **Lifecycle status**: Which rules are in which phase (skill testing, workflow testing, production, stable).
+- **Rule catalog**: Table of all rules with their current status, accuracy, and version.
+- **Evolution timeline**: For each rule, how many iterations it took, what was the accuracy at each step.
+- **Model tier assignments**: Which model is being used for each step of each rule.
+
+### Quality Dashboard
+Generated after quality control reviews.
+
+Key elements:
+- **Accuracy over time**: Line chart showing per-rule and overall accuracy across batches.
+- **Sampling rate over time**: Showing how monitoring is decreasing (or not).
+- **Flagged issues**: Open issues that need developer user attention.
+- **Cost metrics**: LLM calls and tokens per document, per rule.
+
+## Feedback Collection
+
+Every dashboard must include mechanisms for users to report errors and comment directly on verification results. This is not a nice-to-have — user feedback is the most valuable data source in the system.
+
+### Developer User Feedback
+
+Developer users see full result detail. Their feedback interface should support:
+- **Field-level correction**: Click on an extracted value, provide the correct value.
+- **Result override**: Change a pass to fail (or vice versa) with a reason.
+- **Rule re-evaluation request**: Flag a result for re-processing with a different approach.
+- **Comment**: Free-text annotation on any result.
+
+### End User Feedback
+
+End users of the verification app see simplified results. Their interface should support:
+- **Flag as wrong**: One-click to report a result they believe is incorrect.
+- **Add comment**: Brief text explaining what they think is wrong.
+- **Severity indicator**: How impactful is this error? (Critical / Important / Minor)
+
+### Feedback as Ground Truth
+
+User-reported errors are ground truth. They override the coding agent's judgment and the worker LLM's output. The feedback data flow:
+
+1. User submits feedback via dashboard → stored as structured record.
+2. Record schema: `{result_id, trace_id, reporter_role, feedback_type, original_result, corrected_value, comment, timestamp}`.
+3. Feedback records are fed into the `evolution-loop` as confirmed failures.
+4. Dashboard surfaces feedback trends: correction rate over time, most-reported issues, rules with highest user correction rates.
+
+Build the feedback collection mechanism alongside the dashboard generation — they are not separate features. Every generated HTML dashboard should include the feedback UI, even if it initially writes to a local JSON file that the coding agent reads on the next iteration.
+
+## Technology
+
+Self-contained HTML with embedded CSS and JavaScript. Requirements:
+- **No external dependencies.** No CDN links, no npm packages, no server. Everything is inlined.
+- **No server required.** The developer user double-clicks the HTML file to open it in their browser.
+- **Responsive layout.** Should work on both desktop and mobile screens.
+- **Dark/light mode.** Respect the system preference or provide a toggle.
+
+For charts, use inline SVG or a lightweight chart library that can be embedded (e.g., Chart.js or lightweight alternatives, inlined as a script tag).
+
+## Data Sources
+
+Dashboards read from:
+- `Output/` for verification results.
+- `logs/` for evolution and testing history.
+- `versions.json` for current system state.
+- QC review records (stored alongside Output/).
+
+The dashboard generation script should accept input paths and produce a single HTML file.
+
+## Generation Triggers
+
+Generate dashboards automatically after:
+- Each testing round completes (skill testing or workflow testing).
+- Each production batch finishes processing.
+- Each quality control review cycle.
+- Developer user explicitly requests it.
+
+Store generated dashboards in `Output/dashboards/` with timestamps in filenames for history.
+
+## Design Principles
+
+- **Lead with the summary.** The developer user should understand the system's health in 3 seconds.
+- **Drill down on demand.** Summary → rule-level → document-level. Do not overwhelm with details upfront.
+- **Color coding.** Green for pass/healthy, red for fail/critical, yellow for warning/attention. Simple and universal.
+- **Actionable.** Every flagged issue should suggest what to do next.
+
+A starter script is available in `scripts/generate_dashboard.py`. Adapt it to the specific business scenario.

package/template/skills/en/meta-meta/dashboard-reporting/scripts/generate_dashboard.py

@@ -0,0 +1,178 @@
+"""
+Dashboard Generator — Starter Script
+
+Generates a self-contained HTML dashboard from verification results.
+This is a starting point. The coding agent should customize it for the
+specific business scenario.
+
+Usage:
+    python generate_dashboard.py --input <output_dir> --output <dashboard.html>
+
+Input: A directory containing verification result JSON files.
+Output: A single self-contained HTML file.
+"""
+
+import argparse
+import json
+import os
+from datetime import datetime
+from pathlib import Path
+
+
+def load_results(input_dir: str) -> list[dict]:
+    """Load all JSON result files from the input directory."""
+    results = []
+    for f in Path(input_dir).glob("**/*.json"):
+        if "dashboard" in str(f) or "versions" in str(f):
+            continue
+        try:
+            with open(f) as fh:
+                data = json.load(fh)
+                if isinstance(data, list):
+                    results.extend(data)
+                elif isinstance(data, dict) and "results" in data:
+                    results.extend(data["results"])
+                elif isinstance(data, dict) and "rule_id" in data:
+                    results.append(data)
+        except (json.JSONDecodeError, KeyError):
+            continue
+    return results
+
+
+def compute_summary(results: list[dict]) -> dict:
+    """Compute summary statistics from results."""
+    total = len(results)
+    if total == 0:
+        return {"total": 0, "pass": 0, "fail": 0, "missing": 0, "error": 0}
+
+    summary = {
+        "total": total,
+        "pass": sum(1 for r in results if r.get("result") == "pass"),
+        "fail": sum(1 for r in results if r.get("result") == "fail"),
+        "missing": sum(1 for r in results if r.get("result") == "missing"),
+        "error": sum(1 for r in results if r.get("result") == "error"),
+    }
+    summary["pass_rate"] = round(summary["pass"] / total * 100, 1)
+    return summary
+
+
+def compute_per_rule(results: list[dict]) -> dict:
+    """Compute per-rule statistics."""
+    rules = {}
+    for r in results:
+        rule_id = r.get("rule_id", "unknown")
+        if rule_id not in rules:
+            rules[rule_id] = {"total": 0, "pass": 0, "fail": 0, "results": []}
+        rules[rule_id]["total"] += 1
+        if r.get("result") == "pass":
+            rules[rule_id]["pass"] += 1
+        elif r.get("result") == "fail":
+            rules[rule_id]["fail"] += 1
+        rules[rule_id]["results"].append(r)
+
+    for rule_id, data in rules.items():
+        data["accuracy"] = round(data["pass"] / data["total"] * 100, 1) if data["total"] > 0 else 0
+
+    return rules
+
+
+def generate_html(summary: dict, per_rule: dict, failed_cases: list[dict]) -> str:
+    """Generate a self-contained HTML dashboard."""
+
+    rule_rows = ""
+    for rule_id, data in sorted(per_rule.items()):
+        color = "#4caf50" if data["accuracy"] >= 90 else "#ff9800" if data["accuracy"] >= 70 else "#f44336"
+        rule_rows += f"""
+        <tr>
+            <td>{rule_id}</td>
+            <td>{data['total']}</td>
+            <td>{data['pass']}</td>
+            <td>{data['fail']}</td>
+            <td style="color: {color}; font-weight: bold;">{data['accuracy']}%</td>
+        </tr>"""
+
+    fail_rows = ""
+    for case in failed_cases[:100]:  # Limit to first 100
+        fail_rows += f"""
+        <tr>
+            <td>{case.get('document', 'N/A')}</td>
+            <td>{case.get('rule_id', 'N/A')}</td>
+            <td>{case.get('extracted_value', 'N/A')}</td>
+            <td>{case.get('comment', 'N/A')}</td>
+            <td>{case.get('confidence', 'N/A')}</td>
+        </tr>"""
+
+    html = f"""<!DOCTYPE html>
+<html lang="en">
+<head>
+<meta charset="UTF-8">
+<meta name="viewport" content="width=device-width, initial-scale=1.0">
+<title>KC Reborn — Verification Dashboard</title>
+<style>
+    :root {{ --bg: #1a1a2e; --surface: #16213e; --text: #e0e0e0; --accent: #4caf50; --warn: #ff9800; --err: #f44336; }}
+    @media (prefers-color-scheme: light) {{
+        :root {{ --bg: #f5f5f5; --surface: #ffffff; --text: #333333; }}
+    }}
+    * {{ margin: 0; padding: 0; box-sizing: border-box; }}
+    body {{ font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', sans-serif; background: var(--bg); color: var(--text); padding: 20px; }}
+    h1 {{ margin-bottom: 20px; }}
+    .summary {{ display: flex; gap: 16px; margin-bottom: 24px; flex-wrap: wrap; }}
+    .card {{ background: var(--surface); padding: 20px; border-radius: 8px; min-width: 140px; text-align: center; }}
+    .card .number {{ font-size: 2em; font-weight: bold; }}
+    .card .label {{ font-size: 0.85em; opacity: 0.7; margin-top: 4px; }}
+    table {{ width: 100%; border-collapse: collapse; background: var(--surface); border-radius: 8px; overflow: hidden; margin-bottom: 24px; }}
+    th, td {{ padding: 10px 14px; text-align: left; border-bottom: 1px solid rgba(255,255,255,0.1); }}
+    th {{ background: rgba(0,0,0,0.2); font-weight: 600; }}
+    h2 {{ margin: 20px 0 12px; }}
+    .timestamp {{ opacity: 0.5; font-size: 0.85em; margin-bottom: 20px; }}
+</style>
+</head>
+<body>
+<h1>Verification Dashboard</h1>
+<div class="timestamp">Generated: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}</div>
+
+<div class="summary">
+    <div class="card"><div class="number">{summary['total']}</div><div class="label">Total</div></div>
+    <div class="card"><div class="number" style="color: var(--accent);">{summary['pass']}</div><div class="label">Pass</div></div>
+    <div class="card"><div class="number" style="color: var(--err);">{summary['fail']}</div><div class="label">Fail</div></div>
+    <div class="card"><div class="number" style="color: var(--warn);">{summary.get('missing', 0)}</div><div class="label">Missing</div></div>
+    <div class="card"><div class="number">{summary.get('pass_rate', 0)}%</div><div class="label">Pass Rate</div></div>
+</div>
+
+<h2>Per-Rule Breakdown</h2>
+<table>
+    <thead><tr><th>Rule</th><th>Total</th><th>Pass</th><th>Fail</th><th>Accuracy</th></tr></thead>
+    <tbody>{rule_rows}</tbody>
+</table>
+
+<h2>Failed Cases</h2>
+<table>
+    <thead><tr><th>Document</th><th>Rule</th><th>Extracted Value</th><th>Comment</th><th>Confidence</th></tr></thead>
+    <tbody>{fail_rows if fail_rows else '<tr><td colspan="5" style="text-align:center; opacity:0.5;">No failures</td></tr>'}</tbody>
+</table>
+</body>
+</html>"""
+    return html
+
+
+def main():
+    parser = argparse.ArgumentParser(description="Generate verification dashboard")
+    parser.add_argument("--input", required=True, help="Directory containing result JSON files")
+    parser.add_argument("--output", default="dashboard.html", help="Output HTML file path")
+    args = parser.parse_args()
+
+    results = load_results(args.input)
+    summary = compute_summary(results)
+    per_rule = compute_per_rule(results)
+    failed_cases = [r for r in results if r.get("result") == "fail"]
+
+    html = generate_html(summary, per_rule, failed_cases)
+
+    output_path = Path(args.output)
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    output_path.write_text(html, encoding="utf-8")
+    print(f"Dashboard generated: {output_path}")
+
+
+if __name__ == "__main__":
+    main()

package/template/skills/en/meta-meta/evolution-loop/SKILL.md

@@ -0,0 +1,210 @@
+---
+name: evolution-loop
+description: Drive continuous improvement of skills and workflows through the diagnose-classify-fix-retest cycle. Use after any testing round reveals failures, when production quality control flags issues, or when accuracy drops below thresholds. Covers failure analysis, distinguishing systemic issues from corner cases, deciding whether to rewrite or patch, and knowing when to stop iterating. The evolution loop is the heartbeat of the system. Also use when transitioning between lifecycle phases (skill testing, workflow testing, production monitoring).
+---
+
+# Evolution Loop
+
+The evolution loop is what makes this system self-improving rather than static. Every failure is information. The question is always: what kind of failure is this, and what is the most efficient fix?
+
+## The Loop
+
+```
+Test → Observe → Reflect → Diagnose → Classify → Fix → Re-test → Log
+  ↑                                                              |
+  └──────────────────────────────────────────────────────────────┘
+```
+
+Run this loop until one of:
+- Accuracy meets the threshold in `.env` (SKILL_ACCURACY or WORKFLOW_ACCURACY, depending on the phase).
+- MAX_ITERATIONS is reached (escalate to developer user).
+- You determine that the remaining failures are irreducible given the current approach (escalate to developer user).
+- Convergence criteria are met (see Convergence Tracking below).
+
+## Step 1: Test
+
+Run the skill or workflow on the relevant document set:
+- During skill development: run on Samples/.
+- During workflow testing: run on Samples/, compare to skill-based ground truth.
+- During production: run on Input/ batches.
+
+Record every result: pass/fail/missing/error, the extracted values, the confidence scores, and any comments.
+
+## Step 2: Observe
+
+Review the results holistically:
+- What is the overall accuracy?
+- Which rules are failing most?
+- Are there patterns in the failures? Same section type? Same document format? Same kind of entity?
+- Are there documents that fail multiple rules?
+
+Do not just count. Understand.
+
+### User-Reported Errors
+
+When developer users or end users report errors on verification results, treat these corrections as ground truth. User-reported corrections override the coding agent's own quality judgments. In the evolution loop, prioritize diagnosing user-reported errors before agent-detected ones — they represent confirmed failures, not suspected ones.
+
+Collect user error reports from the feedback mechanisms built into the dashboard (see `dashboard-reporting`). Each report should be converted into a test case and added to the regression set.
+
+## Step 3: Reflect
+
+Before diagnosing new failures, review what has already been tried. This prevents cycling through the same failed fixes.
+
+Read the structured iteration logs from `logs/evolution/{rule_id}/`. Focus on:
+- Which failure categories were identified in previous iterations.
+- What fixes were attempted and their outcomes — did accuracy go up, down, or stay flat?
+- Whether any fix was reverted due to regression.
+
+**Anti-pattern detection**: If the current failures match a pattern that was already diagnosed and "fixed" in a prior iteration, the prior fix was insufficient. Escalate the approach — for example, from a prompt tweak to a logic rewrite, or from a logic rewrite to an architecture change. Do not try the same category of fix twice.
+
+**Output**: Produce a brief iteration history summary to feed into the Diagnose step. This gives the diagnosis context about what NOT to try again. On the first iteration (no history), skip this step.
+
+### Three Dimensions of Reflection
+
+When reviewing failures, analyze them along three cross-cutting dimensions:
+
+1. **Per-rule across documents**: Is a specific rule (e.g., R003) failing on a particular subset of documents? This reveals rule-specific weaknesses — perhaps the extraction prompt does not handle a particular document format.
+
+2. **Per-document across rules**: Is a specific document type causing failures across multiple rules? This reveals document-level issues — perhaps a particular template has parsing problems that affect all downstream extraction.
+
+3. **Global patterns**: Are there systemic correlations? For example:
+   - Failures clustering on documents processed by a specific model tier.
+   - Failures clustering on documents that went through a specific parser level.
+   - Failures that only appear when document length exceeds a threshold.
+
+Cross-referencing these dimensions often reveals root causes invisible from a single perspective. For example, "R003 fails on type-X documents" is a per-rule finding, but if type-X documents also cause R007 and R012 to fail, the real issue is the document type, not any individual rule.
+
+## Step 4: Diagnose
+
+For each failure, determine the root cause:
+- **Parsing failure**: the document was not parsed correctly. The text was garbled, tables were mangled, or content was missing.
+- **Extraction failure**: the entity was not found, or the wrong value was extracted. The section was located correctly but the entity extraction failed.
+- **Judgment failure**: the entity was extracted correctly but the pass/fail determination was wrong. The logic is flawed or the edge case is not handled.
+- **Scope failure**: the rule was applied to a section where it does not apply, or not applied to a section where it should.
+
+## Step 5: Classify
+
+This is the critical decision:
+
+### Systemic Issue (affects >10% of documents)
+The failure has a common cause that affects many documents. Examples:
+- The document parser consistently fails on a certain table format.
+- The extraction prompt misunderstands a common phrasing.
+- The threshold logic has a bug.
+
+**Action**: Fix the root cause. Rewrite the relevant part of the skill or workflow. This is a code/prompt change.
+
+### Corner Case (affects <10% of documents)
+The failure is specific to a few unusual documents. Examples:
+- One document uses a non-standard date format.
+- One document has the relevant information in a footnote instead of the main text.
+- One document is a special report type with different structure.
+
+**Action**: Do NOT patch the main workflow. Record the pattern and resolution in `corner_cases.json`. See `corner-case-management` skill.
+
+### The 10% Threshold
+
+This is a heuristic, not a law. Use judgment:
+- If 8% of documents fail the same way but the fix is simple, treat it as systemic.
+- If 12% of documents fail but each for a different reason, treat them as individual corner cases.
+- When in doubt, discuss with the developer user.
+
+## Step 6: Fix
+
+For systemic issues:
+1. Identify the specific component that needs to change (parser config, extraction prompt, judgment logic, regex pattern).
+2. Make the change.
+3. Version the change (see `version-control` skill).
+
+For corner cases:
+1. Document the pattern in `corner_cases.json`.
+2. Add a detection mechanism (regex, keyword, structural pattern).
+3. Define the resolution (alternative extraction method, special judgment logic).
+4. Set a confidence threshold for matching.
+
+## Step 7: Re-test
+
+After fixing:
+1. Re-run on the full document set, not just the failing documents. Fixes can introduce regressions.
+2. Compare to previous iteration results.
+3. If accuracy improved, continue. If accuracy regressed, roll back (see `version-control`).
+
+## Step 8: Log
+
+For every iteration, record the following in a structured format that the Reflect step can consume:
+
+```json
+{
+  "iteration": 3,
+  "trigger": "regression_detected",
+  "observation_summary": "3 documents failing on date extraction",
+  "reflection": "Iteration 2 attempted regex fix for same pattern, accuracy unchanged",
+  "failures": [
+    {"case_id": "doc_007", "diagnosis": "extraction", "root_cause": "non-standard date format", "fix": "added regex pattern", "outcome": "resolved"}
+  ],
+  "mutations": [
+    {"component": "scripts/check.py", "change_type": "patch", "description": "Added YYYY/MM/DD pattern"}
+  ],
+  "outcome": {"accuracy_before": 0.82, "accuracy_after": 0.91, "regressions": 0}
+}
+```
+
+This schema is a recommended starting point — adapt the fields to your specific needs. The important thing is that logs are machine-parseable (JSON) so the Reflect step can programmatically scan history.
+
+Also maintain a plain text summary alongside the JSON for developer user readability. Store both in `logs/evolution/{rule_id}/`.
+
+## Convergence Tracking
+
+Track three metrics per iteration to know when to stop:
+
+- **Correction volume**: How many test cases changed result compared to last iteration (as a percentage of total).
+- **New pattern count**: How many previously unseen failure patterns were identified this iteration.
+- **Regression count**: How many test cases that passed last iteration now fail.
+
+### Stopping Criteria
+
+Stop the loop when ALL three conditions hold for one iteration:
+
+1. Correction volume < 5% of total test cases.
+2. New pattern count = 0.
+3. Regression count = 0.
+
+If correction volume *increases* between consecutive iterations, this is a regression signal. Pause the loop and diagnose before continuing — the last fix may be destabilizing the system.
+
+### Expected Convergence
+
+Convergence is not linear. Expect rapid improvement in early iterations followed by diminishing returns:
+
+| Document count | Typical iterations to converge |
+|---------------|-------------------------------|
+| < 50 | 3–5 |
+| 50–200 | 4–7 |
+| 200–1000 | 5–10 |
+| 1000+ | 7–15 |
+
+These are empirical estimates. Actual convergence depends on rule complexity, document variability, and model capability. If convergence takes significantly more iterations than expected, revisit your approach rather than grinding through more rounds.
+
+See `references/convergence-guide.md` for diagnostic procedures and real-world convergence data.
+
+## The Three Phases of Life
+
+### Phase 1: Evolution (Active Improvement)
+The system is being built. Every testing round reveals issues. The loop runs frequently. This is where you spend most of your effort.
+
+### Phase 2: Monitoring (Quality Control)
+The system is deployed. Workflows are running on production data. The loop runs on sampled results (see `quality-control` skill). You intervene only when quality drops.
+
+### Phase 3: Stability (Near-Zero Oversight)
+The system is mature. Quality has been stable for a sustained period. The loop runs rarely, on small random samples. You intervene only when business rules change.
+
+Transition between phases is gradual, not sudden. And it can reverse — a regulatory change can push you from Phase 3 back to Phase 1.
+
+## Escalation
+
+Escalate to the developer user when:
+- MAX_ITERATIONS is reached without meeting the accuracy threshold.
+- You cannot determine the root cause of a failure.
+- The fix requires domain knowledge you do not have.
+- Multiple rules interact in ways you cannot resolve.
+
+When escalating, provide: the rule, the failing documents, the diagnosis, what you tried, and why it did not work.

package/template/skills/en/meta-meta/evolution-loop/references/convergence-guide.md

@@ -0,0 +1,62 @@
+# Convergence Guide
+
+Diagnostic procedures and real-world data for understanding when the evolution loop is converging, stalling, or regressing.
+
+## Empirical Data
+
+### The Shiji Project — Event Dating Reflection
+
+A document verification project for historical event dating across regulatory filings. Five rounds of evolution:
+
+- **Round 1**: 1,010 corrections (first pass — many extraction and judgment errors across the board).
+- **Round 2**: 431 corrections (systematic fixes applied — regex patterns, prompt refinements).
+- **Round 3**: 465 corrections (regression — round 2 fix for date normalization introduced new failures on edge-case date formats).
+- **Round 4**: 167 corrections (stabilizing — round 3 regression diagnosed and resolved, remaining issues are corner cases).
+- **Round 5**: 46 corrections (converged — below 5% threshold, no new patterns, no regressions).
+
+**Key insight**: The round 3 spike was the most informative event. It revealed that the round 2 fix was too aggressive — it normalized dates that should not have been normalized. Without convergence tracking, this regression might have been masked by overall accuracy still improving on other cases.
+
+## Diagnostic Flowchart
+
+### If correction volume increases between iterations:
+
+1. **Check for regression**: Are previously passing cases now failing? If yes, the last fix is the likely cause. Compare the diff between iterations.
+2. **Check for fix conflicts**: Does the new fix contradict a prior fix? For example, broadening a regex in round N that was narrowed in round N-1.
+3. **Check for test set changes**: Did new documents enter the test set between iterations? New documents can inflate correction volume without indicating regression.
+
+### If correction volume stays flat (not decreasing):
+
+1. **Check for oscillation**: Are the same cases flipping between pass and fail across iterations? This indicates the fix is unstable — it solves one variant but breaks another.
+2. **Check if fix is too narrow**: The fix addresses the specific failing cases but does not generalize. The next iteration reveals similar cases the fix missed.
+
+## False Convergence
+
+Metrics look stable but underlying issues are masked. The system appears converged but will fail on production data.
+
+### Common Causes
+
+- **Test set too small**: With fewer than 20 test cases, a single case changing can swing metrics by 5%. Convergence at this scale is statistically meaningless.
+- **Test set does not cover production variety**: The test set was curated from "clean" examples. Production documents include scanned PDFs, handwritten annotations, multi-language content, and formatting variations the test set never saw.
+- **Corner cases excluded from metrics**: If difficult cases are moved to `corner_cases.json` and excluded from accuracy calculation, the remaining "easy" cases converge quickly but the real problem is hidden.
+
+### Detection
+
+Compare test set distribution to production distribution on key dimensions: document type, length, format, source. If they diverge significantly, convergence on the test set does not guarantee production quality.
+
+## Estimating Remaining Rounds
+
+### Simple Heuristic
+
+If corrections approximately halve each round, expect `log2(current_corrections / threshold)` more rounds.
+
+Example: current round has 200 corrections, threshold is 5% of 1000 cases = 50 corrections.
+- Estimated remaining rounds: log2(200/50) = log2(4) = 2 rounds.
+
+### When the Heuristic Fails
+
+If corrections do not halve between rounds, the current approach may have hit its ceiling. Consider:
+- Escalating the fix strategy (prompt tweak → logic rewrite → architecture change).
+- Expanding the test set to reveal hidden patterns.
+- Consulting the developer user for domain insight on stubborn failures.
+
+Do not grind through more iterations expecting different results. If three consecutive rounds show similar correction volumes, stop and reassess.