trainproof 0.2.0__tar.gz → 0.4.0__tar.gz

{trainproof-0.2.0 → trainproof-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: trainproof
-Version: 0.2.0
+Version: 0.4.0
 Summary: A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.
 Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
 License: MIT
@@ -41,7 +41,6 @@ trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf
 [FAIL] Critical checks failed:
   [FAIL] Loss curve is diverging.
          Evidence: End loss 7.492 vs Min loss 1.398
-  [FAIL] Loss never improved over the run (dead run).
   [WARN] Gradient norm spikes detected.
          Evidence: Max gn 2649.75 > 10.0x median (0.55)
 ```
@@ -68,7 +67,7 @@ shipped in [`examples/gallery/`](examples/gallery/):
 | `lr_hot` | LR x100 (2e-2) | **FAIL** | diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55 |
 | `lr_zero` | LR = 0 | **FAIL** | dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps |
 | `fp16_nan` | fp16 + hot LR, no clipping | **FAIL** | diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed) |
-| `bad_labels` | labels shuffled per-sequence | **WARN only** — see below | grad spike 23.3 vs median 1.09 |
+| `bad_labels` | labels shuffled per-sequence | **WARN only** (single-run) — caught by `trainproof compare` (v0.3) | grad spike 23.3 vs median 1.09 |
 
 ### The honest finding: loss curves cannot see corrupted data
 
@@ -80,12 +79,18 @@ famously fit random labels). **No single-run, loss-only rule can catch this
 class of failure** — its real signature is *relative*: a loss floor ~6x higher
 than a known-good run of the same task (5.59 vs 0.94).
 
-That finding sets the roadmap: v0.3 is `trainproof compare <run> <baseline>` —
-deterministic ratio rules against the healthy baseline you already have.
-See [ROADMAP.md](ROADMAP.md). (The gallery also improved v0.2 itself: the
-dead-run rule exists because `lr_zero` initially escaped with only a WARN.)
+That finding produced v0.3: `trainproof compare <run> <baseline>` —
+deterministic ratio rules against the healthy baseline you already have —
+which catches `bad_labels` at a 6x loss-floor ratio, in 3 seeds out of 3.
+The full study was repeated with three random seeds (15 runs):
+see [EVIDENCE_MATRIX.md](EVIDENCE_MATRIX.md) for every verdict, including the
+honest miss (compare alone overlooks one lr_zero seed — the single-run
+zero-LR fatality rule owns that case; the two commands cover each other's
+blind spots). The gallery also improved the tool itself twice: the dead-run
+rule and the total-zero-LR fatality rule both exist because runs escaped
+earlier rule versions. See [ROADMAP.md](ROADMAP.md).
 
-## The three commands
+## The commands
 
 ```bash
 # 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
@@ -99,11 +104,82 @@ trainproof tokenizer my_tokenizer.model transcripts.txt
 # 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
 #    LR sanity, throughput — from log files, any framework
 trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready
+
+# 4. Compare against a baseline
+#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
+trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] loss floor ratio exceeded limit
+         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
+  [FAIL] end loss ratio exceeded limit
+         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
+========================================
+```
+
 ```
 
 Each command prints the verdict, writes a self-contained HTML report, and sets
 the process exit code — so it works as a CI gate out of the box.
 
+## Live guardian (v0.4)
+
+Don't wait for the post-mortem — catch a doomed run *while it is still burning
+GPU*. Add one line to a HuggingFace `Trainer`:
+
+```python
+from transformers import Trainer
+from trainproof.integrations.hf import TrainproofCallback
+
+trainer = Trainer(
+    ...,
+    callbacks=[TrainproofCallback(policy="stop_on_fail")],  # or policy="warn"
+)
+```
+
+Run against a real diverging QLoRA fine-tune (learning rate 100x too high), the
+guardian aborts it 20 steps into a 300-step schedule — on its own:
+
+```text
+{'loss': '1.784', 'grad_norm': '9.634',  'learning_rate': '0.007'}
+{'loss': '4.282', 'grad_norm': '53.76',  'learning_rate': '0.009'}
+{'loss': '10.6',  'grad_norm': '13.34',  'learning_rate': '0.011'}
+{'loss': '31.67', 'grad_norm': '76.67',  'learning_rate': '0.013'}
+...
+TRAINPROOF ABORT - stopping training at step 20. Findings:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 22.952 vs Min loss 1.358
+  [FAIL] Loss never improved over the run (dead run).
+         Evidence: median of first 5 losses 1.502 vs last 5 22.952
+
+  scheduled steps : 300
+  stopped at step : 20
+  run saved       : 93% of the scheduled steps never ran
+```
+
+On a two-day pre-training run, that fraction is days of GPU time. Or watch a
+growing log file from outside the process (CI-friendly, exits non-zero on FAIL):
+
+```bash
+trainproof watch logs/run.jsonl --interval 10 --until-fail
+# [21:37:44] warming up (5 records)
+# [21:37:44] n_records=15 verdict=PASS findings=1
+```
+
+**The default is safe.** `policy="warn"` (the default) only observes and reports
+— it never interrupts your run, so you can leave it on even for experiments you
+expect to fail. Aborting is strictly opt-in via `policy="stop_on_fail"`, the one
+mode that takes an irreversible action. trainproof does not make that decision
+for you unless you ask.
+
+The guardian applies the same deterministic rules as `trainproof epoch`, so it
+inherits their documented single-run limitations.
+
 ## Supported log formats
 
 - HuggingFace Trainer (`trainer_state.json`)

{trainproof-0.2.0 → trainproof-0.4.0}/README.md

@@ -29,7 +29,6 @@ trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf
 [FAIL] Critical checks failed:
   [FAIL] Loss curve is diverging.
          Evidence: End loss 7.492 vs Min loss 1.398
-  [FAIL] Loss never improved over the run (dead run).
   [WARN] Gradient norm spikes detected.
          Evidence: Max gn 2649.75 > 10.0x median (0.55)
 ```
@@ -56,7 +55,7 @@ shipped in [`examples/gallery/`](examples/gallery/):
 | `lr_hot` | LR x100 (2e-2) | **FAIL** | diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55 |
 | `lr_zero` | LR = 0 | **FAIL** | dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps |
 | `fp16_nan` | fp16 + hot LR, no clipping | **FAIL** | diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed) |
-| `bad_labels` | labels shuffled per-sequence | **WARN only** — see below | grad spike 23.3 vs median 1.09 |
+| `bad_labels` | labels shuffled per-sequence | **WARN only** (single-run) — caught by `trainproof compare` (v0.3) | grad spike 23.3 vs median 1.09 |
 
 ### The honest finding: loss curves cannot see corrupted data
 
@@ -68,12 +67,18 @@ famously fit random labels). **No single-run, loss-only rule can catch this
 class of failure** — its real signature is *relative*: a loss floor ~6x higher
 than a known-good run of the same task (5.59 vs 0.94).
 
-That finding sets the roadmap: v0.3 is `trainproof compare <run> <baseline>` —
-deterministic ratio rules against the healthy baseline you already have.
-See [ROADMAP.md](ROADMAP.md). (The gallery also improved v0.2 itself: the
-dead-run rule exists because `lr_zero` initially escaped with only a WARN.)
+That finding produced v0.3: `trainproof compare <run> <baseline>` —
+deterministic ratio rules against the healthy baseline you already have —
+which catches `bad_labels` at a 6x loss-floor ratio, in 3 seeds out of 3.
+The full study was repeated with three random seeds (15 runs):
+see [EVIDENCE_MATRIX.md](EVIDENCE_MATRIX.md) for every verdict, including the
+honest miss (compare alone overlooks one lr_zero seed — the single-run
+zero-LR fatality rule owns that case; the two commands cover each other's
+blind spots). The gallery also improved the tool itself twice: the dead-run
+rule and the total-zero-LR fatality rule both exist because runs escaped
+earlier rule versions. See [ROADMAP.md](ROADMAP.md).
 
-## The three commands
+## The commands
 
 ```bash
 # 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
@@ -87,11 +92,82 @@ trainproof tokenizer my_tokenizer.model transcripts.txt
 # 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
 #    LR sanity, throughput — from log files, any framework
 trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready
+
+# 4. Compare against a baseline
+#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
+trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] loss floor ratio exceeded limit
+         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
+  [FAIL] end loss ratio exceeded limit
+         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
+========================================
+```
+
 ```
 
 Each command prints the verdict, writes a self-contained HTML report, and sets
 the process exit code — so it works as a CI gate out of the box.
 
+## Live guardian (v0.4)
+
+Don't wait for the post-mortem — catch a doomed run *while it is still burning
+GPU*. Add one line to a HuggingFace `Trainer`:
+
+```python
+from transformers import Trainer
+from trainproof.integrations.hf import TrainproofCallback
+
+trainer = Trainer(
+    ...,
+    callbacks=[TrainproofCallback(policy="stop_on_fail")],  # or policy="warn"
+)
+```
+
+Run against a real diverging QLoRA fine-tune (learning rate 100x too high), the
+guardian aborts it 20 steps into a 300-step schedule — on its own:
+
+```text
+{'loss': '1.784', 'grad_norm': '9.634',  'learning_rate': '0.007'}
+{'loss': '4.282', 'grad_norm': '53.76',  'learning_rate': '0.009'}
+{'loss': '10.6',  'grad_norm': '13.34',  'learning_rate': '0.011'}
+{'loss': '31.67', 'grad_norm': '76.67',  'learning_rate': '0.013'}
+...
+TRAINPROOF ABORT - stopping training at step 20. Findings:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 22.952 vs Min loss 1.358
+  [FAIL] Loss never improved over the run (dead run).
+         Evidence: median of first 5 losses 1.502 vs last 5 22.952
+
+  scheduled steps : 300
+  stopped at step : 20
+  run saved       : 93% of the scheduled steps never ran
+```
+
+On a two-day pre-training run, that fraction is days of GPU time. Or watch a
+growing log file from outside the process (CI-friendly, exits non-zero on FAIL):
+
+```bash
+trainproof watch logs/run.jsonl --interval 10 --until-fail
+# [21:37:44] warming up (5 records)
+# [21:37:44] n_records=15 verdict=PASS findings=1
+```
+
+**The default is safe.** `policy="warn"` (the default) only observes and reports
+— it never interrupts your run, so you can leave it on even for experiments you
+expect to fail. Aborting is strictly opt-in via `policy="stop_on_fail"`, the one
+mode that takes an irreversible action. trainproof does not make that decision
+for you unless you ask.
+
+The guardian applies the same deterministic rules as `trainproof epoch`, so it
+inherits their documented single-run limitations.
+
 ## Supported log formats
 
 - HuggingFace Trainer (`trainer_state.json`)

{trainproof-0.2.0 → trainproof-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "trainproof"
-version = "0.2.0"
+version = "0.4.0"
 description = "A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs."
 authors = [{name = "Panagiotis (Panos) Gkilis", email = "bedvibe@bedvibe.studio"}]
 readme = "README.md"

{trainproof-0.2.0 → trainproof-0.4.0}/src/trainproof/cli.py

@@ -4,6 +4,9 @@ from pathlib import Path
 from .speech.data import check_data
 from .speech.tokenizer import check_tokenizer
 from .epoch import check_epoch
+from .epoch import check_epoch
+from .compare import check_compare
+from .watch import watch_loop
 from .report import print_verdict_console, write_html_report
 
 def main():
@@ -24,6 +27,19 @@ def main():
     epoch_parser.add_argument("logfile", type=str, help="Path to JSONL or CSV log file")
     epoch_parser.add_argument("--format", choices=["auto", "hf", "coqui", "jsonl", "csv"], default="auto", help="Log format override")
 
+    # Compare Command
+    compare_parser = subparsers.add_parser("compare", help="Compare a run log against a baseline log.")
+    compare_parser.add_argument("run", type=str, help="Path to run log file")
+    compare_parser.add_argument("baseline", type=str, help="Path to baseline log file")
+    compare_parser.add_argument("--format", choices=["auto", "hf", "coqui", "jsonl", "csv"], default="auto", help="Log format override")
+
+    # Watch Command
+    watch_parser = subparsers.add_parser("watch", help="Live guardian: poll a training log.")
+    watch_parser.add_argument("logfile", type=str, help="Path to run log file")
+    watch_parser.add_argument("--interval", type=int, default=10, help="Polling interval in seconds")
+    watch_parser.add_argument("--format", choices=["auto", "hf", "coqui", "jsonl", "csv"], default="auto", help="Log format override")
+    watch_parser.add_argument("--until-fail", action="store_true", help="Exit with code 1 as soon as verdict becomes FAIL")
+
     args = parser.parse_args()
     
     report_dict = {}
@@ -38,6 +54,15 @@ def main():
         except Exception as e:
             print(f"Error checking epoch: {e}")
             sys.exit(1)
+    elif args.command == "compare":
+        try:
+            report_dict = check_compare(args.run, args.baseline, fmt=args.format)
+        except Exception as e:
+            print(f"Error checking compare: {e}")
+            sys.exit(1)
+    elif args.command == "watch":
+        watch_loop(args.logfile, interval=args.interval, fmt=args.format, until_fail=args.until_fail)
+        sys.exit(0)
 
     print_verdict_console(report_dict.get("verdict", "FAIL"), report_dict.get("findings", []))
     

trainproof-0.4.0/src/trainproof/compare.py

@@ -0,0 +1,132 @@
+import math
+from pathlib import Path
+from typing import Any
+from . import rules
+from .adapters import parse_log_with_format
+from .epoch import check_epoch
+
+def _get_val(row, aliases):
+    for a in aliases:
+        for k in row:
+            if a in k: return row[k]
+    return None
+
+def extract_metrics(records):
+    losses = []
+    grad_norms = []
+
+    for r in records:
+        loss = _get_val(r, ["loss", "train_loss"])
+        gn = _get_val(r, ["grad_norm", "gnorm", "grad"])
+        
+        if loss is not None and not math.isnan(loss) and not math.isinf(loss):
+            losses.append(loss)
+        if gn is not None and not math.isnan(gn) and not math.isinf(gn):
+            grad_norms.append(gn)
+            
+    if len(losses) < 10:
+        return None
+        
+    floor = min(losses)
+    
+    w = rules.LOSS_IMPROVEMENT_WINDOW
+    start_med = sorted(losses[:w])[w // 2]
+    end_med = sorted(losses[-w:])[w // 2]
+    
+    improvement = 1.0 - (end_med / start_med) if start_med > 0 else 0.0
+    
+    gn_median = None
+    if len(grad_norms) > 5:
+        sorted_gns = sorted(grad_norms)
+        gn_median = sorted_gns[len(sorted_gns) // 2]
+        
+    return {
+        "floor": floor,
+        "start_med": start_med,
+        "end_med": end_med,
+        "improvement": improvement,
+        "gn_median": gn_median,
+        "losses_len": len(losses)
+    }
+
+def check_compare(run_path: str | Path, base_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
+    findings = []
+    verdict = "PASS"
+
+    # Baseline sanity — a suspect baseline downgrades the whole comparison
+    base_epoch_res = check_epoch(base_path, fmt=fmt)
+    if base_epoch_res["verdict"] == "FAIL":
+        verdict = "WARN"
+        findings.append({
+            "level": "WARN",
+            "message": "baseline itself fails single-run checks - comparison may be meaningless",
+            "evidence": str(base_path)
+        })
+
+    run_records = parse_log_with_format(run_path, fmt)
+    base_records = parse_log_with_format(base_path, fmt)
+
+    run_metrics = extract_metrics(run_records) if run_records else None
+    base_metrics = extract_metrics(base_records) if base_records else None
+
+    if not run_metrics:
+        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "Run log has fewer than 10 valid loss points.", "evidence": str(run_path)}]}
+    if not base_metrics:
+        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "Baseline log has fewer than 10 valid loss points.", "evidence": str(base_path)}]}
+
+    # Floor ratio
+    if run_metrics["floor"] > base_metrics["floor"] * rules.MAX_FLOOR_RATIO:
+        verdict = "FAIL"
+        findings.append({
+            "level": "FAIL",
+            "message": "loss floor ratio exceeded limit",
+            "evidence": f"Run floor {run_metrics['floor']:.3f} vs Baseline floor {base_metrics['floor']:.3f} (ratio {(run_metrics['floor'] / base_metrics['floor']):.1f}x > {rules.MAX_FLOOR_RATIO})"
+        })
+
+    # End-loss ratio: where the run landed vs where the baseline landed.
+    # Robust to corrupted starts (see rules.MAX_END_RATIO comment).
+    if base_metrics["end_med"] > 0 and run_metrics["end_med"] > base_metrics["end_med"] * rules.MAX_END_RATIO:
+        verdict = "FAIL"
+        findings.append({
+            "level": "FAIL",
+            "message": "end loss ratio exceeded limit",
+            "evidence": f"Run end {run_metrics['end_med']:.3f} vs Baseline end {base_metrics['end_med']:.3f} (ratio {(run_metrics['end_med'] / base_metrics['end_med']):.1f}x > {rules.MAX_END_RATIO})"
+        })
+
+    # Improvement deficit
+    run_imp = run_metrics["improvement"]
+    base_imp = base_metrics["improvement"]
+    
+    if run_imp < 0:
+        verdict = "FAIL"
+        findings.append({
+            "level": "FAIL",
+            "message": "negative improvement",
+            "evidence": f"Run improvement is {run_imp * 100:.1f}% vs Baseline {base_imp * 100:.1f}%"
+        })
+    elif base_imp > 0 and run_imp < base_imp * rules.MIN_IMPROVEMENT_FRACTION:
+        verdict = "FAIL"
+        findings.append({
+            "level": "FAIL",
+            "message": "improvement deficit",
+            "evidence": f"Run improvement {run_imp * 100:.1f}% vs Baseline {base_imp * 100:.1f}% (ratio {(run_imp / base_imp):.2f}x < {rules.MIN_IMPROVEMENT_FRACTION})"
+        })
+
+    # Gradnorm median ratio
+    if run_metrics["gn_median"] is not None and base_metrics["gn_median"] is not None and base_metrics["gn_median"] > 0:
+        if run_metrics["gn_median"] > base_metrics["gn_median"] * rules.MAX_GRADNORM_MEDIAN_RATIO:
+            if verdict == "PASS": verdict = "WARN"
+            findings.append({
+                "level": "WARN",
+                "message": "gradient norm median significantly higher than baseline",
+                "evidence": f"Run gn median {run_metrics['gn_median']:.2f} vs Baseline gn median {base_metrics['gn_median']:.2f} (ratio {(run_metrics['gn_median'] / base_metrics['gn_median']):.1f}x > {rules.MAX_GRADNORM_MEDIAN_RATIO})"
+            })
+
+    if verdict == "PASS" and not any(f["level"] == "WARN" for f in findings):
+        findings.append({
+            "level": "PASS",
+            "message": "Run compares favorably to baseline.",
+            "evidence": f"Floor ratio {(run_metrics['floor'] / base_metrics['floor']):.2f}x, Improvement {run_imp*100:.1f}% vs {base_imp*100:.1f}%"
+        })
+
+    return {"verdict": verdict, "findings": findings}

{trainproof-0.2.0 → trainproof-0.4.0}/src/trainproof/epoch.py

@@ -4,13 +4,12 @@ from typing import Any
 from . import rules
 from .adapters import parse_log_with_format
 
-def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
-    records = parse_log_with_format(log_path, fmt)
+def check_records(records: list[dict]) -> dict[str, Any]:
     findings = []
     verdict = "PASS"
     
     if not records:
-        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "No valid log records found.", "evidence": str(log_path)}]}
+        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "No valid log records found.", "evidence": ""}]}
 
     # Find relevant keys
     def get_val(row, aliases):
@@ -92,7 +91,10 @@ def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
     if lrs:
         zeros = sum(1 for lr in lrs if lr <= 0)
         zero_frac = zeros / len(lrs)
-        if zero_frac > rules.MAX_ZERO_LR_FRACTION:
+        if zero_frac >= rules.ZERO_LR_FAIL_FRACTION:
+            findings.append({"level": "FAIL", "message": "Learning rate is zero for the entire run - the optimizer never steps.", "evidence": f"{zero_frac*100:.1f}% of steps have lr=0"})
+            verdict = "FAIL"
+        elif zero_frac > rules.MAX_ZERO_LR_FRACTION:
             findings.append({"level": "WARN", "message": "Learning rate is zero for a large fraction of the run.", "evidence": f"{zero_frac*100:.1f}% of steps have lr=0"})
             if verdict == "PASS": verdict = "WARN"
 
@@ -109,3 +111,9 @@ def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
         findings.append({"level": "PASS", "message": "Loss curve shows healthy shape, grad norms are stable.", "evidence": f"{len(valid_losses)} steps analyzed."})
 
     return {"verdict": verdict, "findings": findings}
+
+def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
+    records = parse_log_with_format(log_path, fmt)
+    if not records:
+        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "No valid log records found.", "evidence": str(log_path)}]}
+    return check_records(records)

trainproof-0.4.0/src/trainproof/integrations/__init__.py

@@ -0,0 +1,3 @@
+from .hf import TrainproofCallback
+
+__all__ = ["TrainproofCallback"]

trainproof-0.4.0/src/trainproof/integrations/hf.py

@@ -0,0 +1,89 @@
+from trainproof.epoch import check_records
+
+def _convert_state_to_records(state) -> list[dict]:
+    history = getattr(state, "log_history", [])
+    records = []
+    for entry in history:
+        if not isinstance(entry, dict) or "loss" not in entry:
+            continue
+        record = {}
+        for k in ["loss", "grad_norm", "step"]:
+            if k in entry and entry[k] is not None:
+                try:
+                    record[k] = float(entry[k])
+                except (ValueError, TypeError):
+                    pass
+        if "learning_rate" in entry and entry["learning_rate"] is not None:
+            try:
+                record["lr"] = float(entry["learning_rate"])
+            except (ValueError, TypeError):
+                pass
+        records.append(record)
+    return records
+
+try:
+    from transformers import TrainerCallback
+except ImportError:
+    class TrainerCallback:
+        pass
+    _HAS_TRANSFORMERS = False
+else:
+    _HAS_TRANSFORMERS = True
+
+
+class TrainproofCallback(TrainerCallback):
+    """Judge a live HuggingFace training run with trainproof's deterministic rules.
+
+    policy:
+      "warn" (DEFAULT): observe and report only. On a FAIL verdict it prints the
+        findings and lets training continue. It NEVER interrupts your run — safe
+        to leave on by default, including for experiments you expect to fail.
+      "stop_on_fail": opt-in. Additionally sets control.should_training_stop on a
+        FAIL verdict, aborting the run to save GPU time. This is the only mode
+        that takes an irreversible action, and you must ask for it explicitly.
+
+    check_every: minimum steps between checks. min_points: minimum logged loss
+    points before any verdict is issued (avoids judging warm-up noise).
+    A FAIL is announced once, not re-announced every subsequent check.
+    """
+
+    def __init__(self, policy="warn", check_every=25, min_points=10):
+        if not _HAS_TRANSFORMERS:
+            raise ImportError("pip install transformers is required to use TrainproofCallback")
+        self.policy = policy
+        self.check_every = check_every
+        self.min_points = min_points
+        self.last_checked_step = 0
+        self.last_verdict = None
+
+    def on_log(self, args, state, control, **kwargs):
+        step = getattr(state, "global_step", 0)
+        if step < self.last_checked_step + self.check_every:
+            return
+
+        records = _convert_state_to_records(state)
+        if len(records) < self.min_points:
+            return
+
+        self.last_checked_step = step
+        
+        report = check_records(records)
+        verdict = report.get("verdict", "UNKNOWN")
+        findings = report.get("findings", [])
+
+        if verdict == "FAIL" and self.last_verdict != "FAIL":
+            if self.policy == "stop_on_fail":
+                setattr(control, "should_training_stop", True)
+                print(f"\nTRAINPROOF ABORT - stopping training at step {step}. Findings:")
+            else:
+                print("\nTRAINPROOF WARNING - this run looks doomed:")
+                
+            for f in findings:
+                level = f.get("level", "INFO")
+                msg = f.get("message", "")
+                ev = f.get("evidence", "")
+                print(f"  [{level}] {msg}")
+                if ev:
+                    print(f"         Evidence: {ev}")
+
+        self.last_verdict = verdict

{trainproof-0.2.0 → trainproof-0.4.0}/src/trainproof/rules.py

@@ -52,6 +52,12 @@ MAX_GRAD_NORM_SPIKE_RATIO = 10.0
 # If the learning rate is strictly zero for more than this fraction of the epoch, it's a warning.
 MAX_ZERO_LR_FRACTION = 0.1
 
+# If the learning rate is zero for essentially the ENTIRE run, the optimizer
+# never steps — the run cannot train, regardless of how the loss wiggles
+# (multi-seed testing showed batch-order noise can fake >5% "improvement"
+# under lr=0). Total zero-LR is a fatality, not a warning.
+ZERO_LR_FAIL_FRACTION = 0.99
+
 # Dead-run detection (added in v0.2 after fault-injection testing showed that
 # zero-LR and garbage-label runs — which can never learn — only earned WARNs):
 # the median of the last LOSS_IMPROVEMENT_WINDOW losses must be at least
@@ -61,3 +67,24 @@ MAX_ZERO_LR_FRACTION = 0.1
 MIN_LOSS_IMPROVEMENT = 0.05
 LOSS_IMPROVEMENT_WINDOW = 5
 MIN_POINTS_FOR_IMPROVEMENT_CHECK = 10
+
+# -----------------
+# COMPARE SUBCOMMAND
+# -----------------
+
+# FAIL "loss floor ratio" if run_floor / baseline_floor > 2.0
+MAX_FLOOR_RATIO = 2.0
+
+# FAIL "end loss ratio" if the run's final loss (median of last window) lands
+# more than this factor above the baseline's. Judges where the run LANDED —
+# robust both to noisy first steps and to corruption inside the first window
+# (e.g. an LR explosion at step 10 inflates the run's own start median, making
+# self-relative improvement look positive; the end ratio still catches it).
+MAX_END_RATIO = 2.0
+
+# FAIL "improvement deficit" if the run's relative improvement is < 25% of the baseline's
+# (only evaluated when baseline improvement > 0). A run with NEGATIVE improvement always trips this rule.
+MIN_IMPROVEMENT_FRACTION = 0.25
+
+# WARN if run's grad-norm median > 5x baseline's (skip if either lacks grad norms)
+MAX_GRADNORM_MEDIAN_RATIO = 5.0

trainproof-0.4.0/src/trainproof/watch.py

@@ -0,0 +1,57 @@
+import time
+import sys
+from datetime import datetime
+from pathlib import Path
+from .adapters import parse_log_with_format
+from .epoch import check_records
+from .report import print_verdict_console
+
+def poll_once(path: str | Path, fmt: str, prev_verdict: str | None) -> tuple[str | None, bool, int]:
+    try:
+        records = parse_log_with_format(path, fmt)
+    except Exception:
+        records = []
+        
+    n_records = len(records)
+    
+    valid_losses = 0
+    for r in records:
+        if "loss" in r and r["loss"] is not None:
+            valid_losses += 1
+            
+    now_str = datetime.now().strftime("%H:%M:%S")
+    
+    if valid_losses < 10:
+        print(f"[{now_str}] warming up ({n_records} records)")
+        return (None, False, n_records)
+        
+    report = check_records(records)
+    verdict = report.get("verdict", "UNKNOWN")
+    n_findings = len(report.get("findings", []))
+    
+    changed = (verdict != prev_verdict)
+    
+    print(f"[{now_str}] n_records={n_records} verdict={verdict} findings={n_findings}")
+    
+    if changed:
+        print_verdict_console(verdict, report.get("findings", []))
+        
+    return (verdict, changed, n_records)
+
+def watch_loop(path: str | Path, interval: int = 10, fmt: str = "auto", until_fail: bool = False):
+    prev_verdict = None
+    try:
+        while True:
+            verdict, changed, n = poll_once(path, fmt, prev_verdict)
+            if verdict is not None:
+                prev_verdict = verdict
+            
+            if until_fail and verdict == "FAIL":
+                sys.exit(1)
+                
+            time.sleep(interval)
+    except KeyboardInterrupt:
+        print(f"\nStopped watching. Final verdict: {prev_verdict}")
+        if prev_verdict == "FAIL":
+            sys.exit(1)
+        sys.exit(0)

{trainproof-0.2.0 → trainproof-0.4.0}/src/trainproof.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: trainproof
-Version: 0.2.0
+Version: 0.4.0
 Summary: A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.
 Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
 License: MIT
@@ -41,7 +41,6 @@ trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf
 [FAIL] Critical checks failed:
   [FAIL] Loss curve is diverging.
          Evidence: End loss 7.492 vs Min loss 1.398
-  [FAIL] Loss never improved over the run (dead run).
   [WARN] Gradient norm spikes detected.
          Evidence: Max gn 2649.75 > 10.0x median (0.55)
 ```
@@ -68,7 +67,7 @@ shipped in [`examples/gallery/`](examples/gallery/):
 | `lr_hot` | LR x100 (2e-2) | **FAIL** | diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55 |
 | `lr_zero` | LR = 0 | **FAIL** | dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps |
 | `fp16_nan` | fp16 + hot LR, no clipping | **FAIL** | diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed) |
-| `bad_labels` | labels shuffled per-sequence | **WARN only** — see below | grad spike 23.3 vs median 1.09 |
+| `bad_labels` | labels shuffled per-sequence | **WARN only** (single-run) — caught by `trainproof compare` (v0.3) | grad spike 23.3 vs median 1.09 |
 
 ### The honest finding: loss curves cannot see corrupted data
 
@@ -80,12 +79,18 @@ famously fit random labels). **No single-run, loss-only rule can catch this
 class of failure** — its real signature is *relative*: a loss floor ~6x higher
 than a known-good run of the same task (5.59 vs 0.94).
 
-That finding sets the roadmap: v0.3 is `trainproof compare <run> <baseline>` —
-deterministic ratio rules against the healthy baseline you already have.
-See [ROADMAP.md](ROADMAP.md). (The gallery also improved v0.2 itself: the
-dead-run rule exists because `lr_zero` initially escaped with only a WARN.)
+That finding produced v0.3: `trainproof compare <run> <baseline>` —
+deterministic ratio rules against the healthy baseline you already have —
+which catches `bad_labels` at a 6x loss-floor ratio, in 3 seeds out of 3.
+The full study was repeated with three random seeds (15 runs):
+see [EVIDENCE_MATRIX.md](EVIDENCE_MATRIX.md) for every verdict, including the
+honest miss (compare alone overlooks one lr_zero seed — the single-run
+zero-LR fatality rule owns that case; the two commands cover each other's
+blind spots). The gallery also improved the tool itself twice: the dead-run
+rule and the total-zero-LR fatality rule both exist because runs escaped
+earlier rule versions. See [ROADMAP.md](ROADMAP.md).
 
-## The three commands
+## The commands
 
 ```bash
 # 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
@@ -99,11 +104,82 @@ trainproof tokenizer my_tokenizer.model transcripts.txt
 # 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
 #    LR sanity, throughput — from log files, any framework
 trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready
+
+# 4. Compare against a baseline
+#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
+trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] loss floor ratio exceeded limit
+         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
+  [FAIL] end loss ratio exceeded limit
+         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
+========================================
+```
+
 ```
 
 Each command prints the verdict, writes a self-contained HTML report, and sets
 the process exit code — so it works as a CI gate out of the box.
 
+## Live guardian (v0.4)
+
+Don't wait for the post-mortem — catch a doomed run *while it is still burning
+GPU*. Add one line to a HuggingFace `Trainer`:
+
+```python
+from transformers import Trainer
+from trainproof.integrations.hf import TrainproofCallback
+
+trainer = Trainer(
+    ...,
+    callbacks=[TrainproofCallback(policy="stop_on_fail")],  # or policy="warn"
+)
+```
+
+Run against a real diverging QLoRA fine-tune (learning rate 100x too high), the
+guardian aborts it 20 steps into a 300-step schedule — on its own:
+
+```text
+{'loss': '1.784', 'grad_norm': '9.634',  'learning_rate': '0.007'}
+{'loss': '4.282', 'grad_norm': '53.76',  'learning_rate': '0.009'}
+{'loss': '10.6',  'grad_norm': '13.34',  'learning_rate': '0.011'}
+{'loss': '31.67', 'grad_norm': '76.67',  'learning_rate': '0.013'}
+...
+TRAINPROOF ABORT - stopping training at step 20. Findings:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 22.952 vs Min loss 1.358
+  [FAIL] Loss never improved over the run (dead run).
+         Evidence: median of first 5 losses 1.502 vs last 5 22.952
+
+  scheduled steps : 300
+  stopped at step : 20
+  run saved       : 93% of the scheduled steps never ran
+```
+
+On a two-day pre-training run, that fraction is days of GPU time. Or watch a
+growing log file from outside the process (CI-friendly, exits non-zero on FAIL):
+
+```bash
+trainproof watch logs/run.jsonl --interval 10 --until-fail
+# [21:37:44] warming up (5 records)
+# [21:37:44] n_records=15 verdict=PASS findings=1
+```
+
+**The default is safe.** `policy="warn"` (the default) only observes and reports
+— it never interrupts your run, so you can leave it on even for experiments you
+expect to fail. Aborting is strictly opt-in via `policy="stop_on_fail"`, the one
+mode that takes an irreversible action. trainproof does not make that decision
+for you unless you ask.
+
+The guardian applies the same deterministic rules as `trainproof epoch`, so it
+inherits their documented single-run limitations.
+
 ## Supported log formats
 
 - HuggingFace Trainer (`trainer_state.json`)

{trainproof-0.2.0 → trainproof-0.4.0}/src/trainproof.egg-info/SOURCES.txt

@@ -3,18 +3,25 @@ pyproject.toml
 src/trainproof/__init__.py
 src/trainproof/adapters.py
 src/trainproof/cli.py
+src/trainproof/compare.py
 src/trainproof/epoch.py
 src/trainproof/report.py
 src/trainproof/rules.py
+src/trainproof/watch.py
 src/trainproof.egg-info/PKG-INFO
 src/trainproof.egg-info/SOURCES.txt
 src/trainproof.egg-info/dependency_links.txt
 src/trainproof.egg-info/entry_points.txt
 src/trainproof.egg-info/requires.txt
 src/trainproof.egg-info/top_level.txt
+src/trainproof/integrations/__init__.py
+src/trainproof/integrations/hf.py
 src/trainproof/speech/__init__.py
 src/trainproof/speech/data.py
 src/trainproof/speech/tokenizer.py
 tests/test_adapters.py
+tests/test_compare.py
 tests/test_data.py
-tests/test_epoch.py
+tests/test_epoch.py
+tests/test_hf_callback.py
+tests/test_watch.py

trainproof-0.4.0/tests/test_compare.py

@@ -0,0 +1,47 @@
+import pytest
+from pathlib import Path
+from trainproof.compare import check_compare
+
+def get_gallery_log(name: str) -> str:
+    path = Path(__file__).parent.parent / "examples" / "gallery" / name / "trainer_state.json"
+    return str(path.resolve())
+
+def test_compare_healthy_vs_healthy():
+    healthy = get_gallery_log("healthy")
+    res = check_compare(healthy, healthy, fmt="hf")
+    assert res["verdict"] == "PASS"
+
+def test_compare_bad_labels_vs_healthy():
+    bad_labels = get_gallery_log("bad_labels")
+    healthy = get_gallery_log("healthy")
+    res = check_compare(bad_labels, healthy, fmt="hf")
+    assert res["verdict"] == "FAIL"
+    findings = [f for f in res["findings"] if "loss floor ratio" in f["message"]]
+    assert len(findings) > 0
+
+def test_compare_lr_zero_vs_healthy():
+    lr_zero = get_gallery_log("lr_zero")
+    healthy = get_gallery_log("healthy")
+    res = check_compare(lr_zero, healthy, fmt="hf")
+    assert res["verdict"] == "FAIL"
+    findings = [f for f in res["findings"] if "improvement deficit" in f["message"]]
+    assert len(findings) > 0
+
+def test_compare_lr_hot_vs_healthy():
+    # lr_hot's explosion happens INSIDE the first-5 window, inflating its own
+    # start median — self-relative improvement looks positive (+62%). Only the
+    # end-loss ratio (where the run LANDED: ~7x the baseline) catches it.
+    lr_hot = get_gallery_log("lr_hot")
+    healthy = get_gallery_log("healthy")
+    res = check_compare(lr_hot, healthy, fmt="hf")
+    assert res["verdict"] == "FAIL"
+    findings = [f for f in res["findings"] if "end loss ratio" in f["message"]]
+    assert len(findings) > 0
+
+def test_compare_fp16_nan_vs_healthy():
+    fp16_nan = get_gallery_log("fp16_nan")
+    healthy = get_gallery_log("healthy")
+    res = check_compare(fp16_nan, healthy, fmt="hf")
+    assert res["verdict"] == "FAIL"
+    findings = [f for f in res["findings"] if "negative improvement" in f["message"]]
+    assert len(findings) > 0

{trainproof-0.2.0 → trainproof-0.4.0}/tests/test_epoch.py

@@ -41,3 +41,11 @@ def test_epoch_dead_noisy():
     report = check_epoch(FIXTURES_DIR / "dead_noisy.jsonl")
     assert report["verdict"] == "FAIL"
     assert any("never improved" in str(f) for f in report["findings"])
+
+def test_epoch_total_zero_lr_is_fatal():
+    # lr=0 on every step: the optimizer never moves — FAIL regardless of how
+    # batch-order noise makes the loss wiggle (found by multi-seed testing)
+    gallery = Path(__file__).parent.parent / "examples" / "gallery"
+    report = check_epoch(gallery / "lr_zero" / "trainer_state.json", fmt="hf")
+    assert report["verdict"] == "FAIL"
+    assert any("optimizer never steps" in str(f) for f in report["findings"])

trainproof-0.4.0/tests/test_hf_callback.py

@@ -0,0 +1,66 @@
+import trainproof.integrations.hf as hf_mod
+from trainproof.integrations.hf import _convert_state_to_records, TrainproofCallback
+
+# Patch so tests can run without transformers installed
+hf_mod._HAS_TRANSFORMERS = True
+class MockState:
+    def __init__(self):
+        self.global_step = 0
+        self.log_history = []
+
+class MockControl:
+    def __init__(self):
+        self.should_training_stop = False
+
+def test_convert_state_to_records():
+    state = MockState()
+    state.log_history = [
+        {"loss": 1.0, "step": 10, "learning_rate": 0.01, "eval_loss": 2.0},
+        {"eval_loss": 1.5, "step": 10}, 
+        {"loss": 0.5, "step": 20, "grad_norm": 0.1}
+    ]
+    records = _convert_state_to_records(state)
+    assert len(records) == 2
+    assert records[0] == {"loss": 1.0, "step": 10.0, "lr": 0.01}
+    assert records[1] == {"loss": 0.5, "step": 20.0, "grad_norm": 0.1}
+
+def test_hf_callback_warn_policy():
+    callback = TrainproofCallback(policy="warn", check_every=10, min_points=10)
+    state = MockState()
+    control = MockControl()
+    
+    # Add < 10 points -> should do nothing
+    for i in range(5):
+        state.log_history.append({"loss": 10.0, "step": i})
+    state.global_step = 10
+    callback.on_log(None, state, control)
+    assert callback.last_verdict is None
+    
+    # Add up to 15 points (healthy) -> PASS
+    for i in range(5, 15):
+        state.log_history.append({"loss": 1.0, "step": i, "learning_rate": 1e-4})
+    state.global_step = 20
+    callback.on_log(None, state, control)
+    assert callback.last_verdict == "PASS"
+    assert not control.should_training_stop
+    
+    # Diverge -> FAIL but don't stop because policy=warn
+    for i in range(15, 25):
+        state.log_history.append({"loss": 50.0, "step": i, "learning_rate": 1e-4})
+    state.global_step = 30
+    callback.on_log(None, state, control)
+    assert callback.last_verdict == "FAIL"
+    assert not control.should_training_stop
+
+def test_hf_callback_stop_on_fail():
+    callback = TrainproofCallback(policy="stop_on_fail", check_every=10, min_points=10)
+    state = MockState()
+    control = MockControl()
+    
+    for i in range(20):
+        state.log_history.append({"loss": 50.0, "step": i, "learning_rate": 1e-4}) 
+    state.global_step = 20
+    
+    callback.on_log(None, state, control)
+    assert callback.last_verdict == "FAIL"
+    assert control.should_training_stop

trainproof-0.4.0/tests/test_watch.py

@@ -0,0 +1,42 @@
+import json
+from pathlib import Path
+from trainproof.watch import poll_once
+
+def test_watch_poll_once(tmp_path: Path):
+    log_file = tmp_path / "test.jsonl"
+    
+    # Phase 1: warming up (less than 10 valid losses)
+    logs = []
+    for i in range(5):
+        logs.append(json.dumps({"loss": 5.0, "step": i, "lr": 1e-4}))
+    log_file.write_text("\n".join(logs))
+    
+    verdict, changed, n = poll_once(log_file, "jsonl", None)
+    assert verdict is None
+    assert not changed
+    assert n == 5
+    
+    # Phase 2: healthy phase -> PASS
+    for i in range(5, 15):
+        logs.append(json.dumps({"loss": 1.0, "step": i, "lr": 1e-4}))
+    log_file.write_text("\n".join(logs))
+    
+    verdict, changed, n = poll_once(log_file, "jsonl", None)
+    assert verdict == "PASS"
+    assert changed
+    assert n == 15
+    
+    # Check that it doesn't say "changed" if prev_verdict was PASS
+    verdict2, changed2, n2 = poll_once(log_file, "jsonl", "PASS")
+    assert verdict2 == "PASS"
+    assert not changed2
+    
+    # Phase 3: append diverging losses -> verdict changes to FAIL
+    for i in range(15, 25):
+        logs.append(json.dumps({"loss": 20.0, "step": i, "lr": 1e-4}))
+    log_file.write_text("\n".join(logs))
+    
+    verdict3, changed3, n3 = poll_once(log_file, "jsonl", "PASS")
+    assert verdict3 == "FAIL"
+    assert changed3
+    assert n3 == 25