trainproof 0.1.0__tar.gz → 0.3.0__tar.gz

trainproof-0.3.0/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.4
+Name: trainproof
+Version: 0.3.0
+Summary: A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.
+Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: soundfile
+Requires-Dist: ttsproof
+
+# trainproof
+
+[![PyPI](https://img.shields.io/pypi/v/trainproof)](https://pypi.org/project/trainproof/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**A deterministic linter for ML training runs.** Point it at your dataset, your
+tokenizer, or your first-epoch logs — it returns a PASS / WARN / FAIL verdict
+with named findings and cited evidence, before you burn days of GPU time on a
+run that was doomed at step 50.
+
+No ML judging ML. No invented "confidence 97%". Every rule is a deterministic
+threshold in [one auditable module](src/trainproof/rules.py), and every finding
+cites the numbers that triggered it.
+
+```bash
+pip install trainproof
+```
+
+## See a verdict in 60 seconds
+
+This repo ships the real logs of five QLoRA fine-tuning runs (Qwen2.5-3B,
+RTX 5080 — see the gallery below). Judge one right now:
+
+```bash
+trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf
+```
+
+```text
+[FAIL] Critical checks failed:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 7.492 vs Min loss 1.398
+  [WARN] Gradient norm spikes detected.
+         Evidence: Max gn 2649.75 > 10.0x median (0.55)
+```
+
+## Why this exists
+
+The author lost a real 11-hour fine-tune to a failure nothing warned about.
+Pointed retroactively at that run's 1MB Coqui Trainer log (2,501 logged steps),
+trainproof's verdict: **FAIL — diverging**. The loss reached its minimum at 82%
+of the run and ended 1.9x above it. Translation: the final two hours of GPU
+time made the model measurably worse, and the checkpoint worth keeping had
+already existed for hours. No tool in the stack said a word.
+
+## The fault-injection gallery
+
+To validate the rules, the same QLoRA fine-tune (Qwen2.5-3B-Instruct, 4-bit,
+LoRA r=16, 300 steps on Alpaca-cleaned) was run five times — once healthy, four
+times with exactly one knob deliberately broken. Real runs, real logs, all
+shipped in [`examples/gallery/`](examples/gallery/):
+
+| Run | Sabotage | Verdict | Key evidence |
+|---|---|---|---|
+| `healthy` | none | **PASS** | loss 1.52 → 0.94, stable gradients |
+| `lr_hot` | LR x100 (2e-2) | **FAIL** | diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55 |
+| `lr_zero` | LR = 0 | **FAIL** | dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps |
+| `fp16_nan` | fp16 + hot LR, no clipping | **FAIL** | diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed) |
+| `bad_labels` | labels shuffled per-sequence | **WARN only** (single-run) — caught by `trainproof compare` (v0.3) | grad spike 23.3 vs median 1.09 |
+
+### The honest finding: loss curves cannot see corrupted data
+
+The `bad_labels` run — whose shuffled labels make real learning impossible —
+*reduced its loss by 62%* (18.9 → 5.75). The model was genuinely learning: not
+the task, but the marginal token statistics of the garbage. From its own loss
+curve, that is indistinguishable from healthy training (neural networks
+famously fit random labels). **No single-run, loss-only rule can catch this
+class of failure** — its real signature is *relative*: a loss floor ~6x higher
+than a known-good run of the same task (5.59 vs 0.94).
+
+That finding sets the roadmap: v0.3 is `trainproof compare <run> <baseline>` —
+deterministic ratio rules against the healthy baseline you already have.
+See [ROADMAP.md](ROADMAP.md). (The gallery also improved v0.2 itself: the
+dead-run rule exists because `lr_zero` initially escaped with only a WARN.)
+
+## The three commands
+
+```bash
+# 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
+#    duplicates, text-vs-audio duration mismatches
+trainproof data /path/to/dataset_or_manifest.jsonl
+
+# 2. Tokenizer preflight: vocabulary coverage, OOV rate, sequence blowouts,
+#    suspicious splits on numbers/dates
+trainproof tokenizer my_tokenizer.model transcripts.txt
+
+# 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
+#    LR sanity, throughput — from log files, any framework
+trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready
+
+# 4. Compare against a baseline
+#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
+trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] loss floor ratio exceeded limit
+         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
+  [FAIL] end loss ratio exceeded limit
+         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
+========================================
+```
+
+Each command prints the verdict, writes a self-contained HTML report, and sets
+the process exit code — so it works as a CI gate out of the box.
+
+## Supported log formats
+
+- HuggingFace Trainer (`trainer_state.json`)
+- Coqui Trainer text logs (ANSI-colored `trainer_0_log.txt`)
+- Generic JSONL / CSV (columns: step, loss, lr, grad_norm, time — all optional)
+
+Auto-detected; override with `--format hf|coqui|jsonl|csv`. TensorBoard event
+files are planned (Lightning console captures are TTY dumps, not logs, and
+will not be supported).
+
+## Philosophy
+
+1. **Deterministic.** A rule fires or it doesn't. Thresholds live in one
+   module, commented, tunable.
+2. **Evidence-cited.** Every finding names the steps and values that triggered
+   it.
+3. **Honest about limits.** What the tool cannot detect is documented in the
+   README, not discovered by the user in production.
+
+## Family
+
+trainproof judges training runs. Its sibling [ttsproof](https://github.com/Mormolykos/ttsproof)
+judges TTS model *outputs* (structural audio checks, equivalence-aware WER/CER,
+published method with DOI) — and trainproof builds on it for the speech
+dataset checks.
+
+## Author
+
+Panagiotis (Panos) Gkilis — [portfolio](https://tts.bedvibe.studio/portfolio/) ·
+[bedvibe.studio](https://bedvibe.studio/)
+
+MIT license.

trainproof-0.3.0/README.md

@@ -0,0 +1,141 @@
+# trainproof
+
+[![PyPI](https://img.shields.io/pypi/v/trainproof)](https://pypi.org/project/trainproof/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**A deterministic linter for ML training runs.** Point it at your dataset, your
+tokenizer, or your first-epoch logs — it returns a PASS / WARN / FAIL verdict
+with named findings and cited evidence, before you burn days of GPU time on a
+run that was doomed at step 50.
+
+No ML judging ML. No invented "confidence 97%". Every rule is a deterministic
+threshold in [one auditable module](src/trainproof/rules.py), and every finding
+cites the numbers that triggered it.
+
+```bash
+pip install trainproof
+```
+
+## See a verdict in 60 seconds
+
+This repo ships the real logs of five QLoRA fine-tuning runs (Qwen2.5-3B,
+RTX 5080 — see the gallery below). Judge one right now:
+
+```bash
+trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf
+```
+
+```text
+[FAIL] Critical checks failed:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 7.492 vs Min loss 1.398
+  [WARN] Gradient norm spikes detected.
+         Evidence: Max gn 2649.75 > 10.0x median (0.55)
+```
+
+## Why this exists
+
+The author lost a real 11-hour fine-tune to a failure nothing warned about.
+Pointed retroactively at that run's 1MB Coqui Trainer log (2,501 logged steps),
+trainproof's verdict: **FAIL — diverging**. The loss reached its minimum at 82%
+of the run and ended 1.9x above it. Translation: the final two hours of GPU
+time made the model measurably worse, and the checkpoint worth keeping had
+already existed for hours. No tool in the stack said a word.
+
+## The fault-injection gallery
+
+To validate the rules, the same QLoRA fine-tune (Qwen2.5-3B-Instruct, 4-bit,
+LoRA r=16, 300 steps on Alpaca-cleaned) was run five times — once healthy, four
+times with exactly one knob deliberately broken. Real runs, real logs, all
+shipped in [`examples/gallery/`](examples/gallery/):
+
+| Run | Sabotage | Verdict | Key evidence |
+|---|---|---|---|
+| `healthy` | none | **PASS** | loss 1.52 → 0.94, stable gradients |
+| `lr_hot` | LR x100 (2e-2) | **FAIL** | diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55 |
+| `lr_zero` | LR = 0 | **FAIL** | dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps |
+| `fp16_nan` | fp16 + hot LR, no clipping | **FAIL** | diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed) |
+| `bad_labels` | labels shuffled per-sequence | **WARN only** (single-run) — caught by `trainproof compare` (v0.3) | grad spike 23.3 vs median 1.09 |
+
+### The honest finding: loss curves cannot see corrupted data
+
+The `bad_labels` run — whose shuffled labels make real learning impossible —
+*reduced its loss by 62%* (18.9 → 5.75). The model was genuinely learning: not
+the task, but the marginal token statistics of the garbage. From its own loss
+curve, that is indistinguishable from healthy training (neural networks
+famously fit random labels). **No single-run, loss-only rule can catch this
+class of failure** — its real signature is *relative*: a loss floor ~6x higher
+than a known-good run of the same task (5.59 vs 0.94).
+
+That finding sets the roadmap: v0.3 is `trainproof compare <run> <baseline>` —
+deterministic ratio rules against the healthy baseline you already have.
+See [ROADMAP.md](ROADMAP.md). (The gallery also improved v0.2 itself: the
+dead-run rule exists because `lr_zero` initially escaped with only a WARN.)
+
+## The three commands
+
+```bash
+# 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
+#    duplicates, text-vs-audio duration mismatches
+trainproof data /path/to/dataset_or_manifest.jsonl
+
+# 2. Tokenizer preflight: vocabulary coverage, OOV rate, sequence blowouts,
+#    suspicious splits on numbers/dates
+trainproof tokenizer my_tokenizer.model transcripts.txt
+
+# 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
+#    LR sanity, throughput — from log files, any framework
+trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready
+
+# 4. Compare against a baseline
+#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
+trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] loss floor ratio exceeded limit
+         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
+  [FAIL] end loss ratio exceeded limit
+         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
+========================================
+```
+
+Each command prints the verdict, writes a self-contained HTML report, and sets
+the process exit code — so it works as a CI gate out of the box.
+
+## Supported log formats
+
+- HuggingFace Trainer (`trainer_state.json`)
+- Coqui Trainer text logs (ANSI-colored `trainer_0_log.txt`)
+- Generic JSONL / CSV (columns: step, loss, lr, grad_norm, time — all optional)
+
+Auto-detected; override with `--format hf|coqui|jsonl|csv`. TensorBoard event
+files are planned (Lightning console captures are TTY dumps, not logs, and
+will not be supported).
+
+## Philosophy
+
+1. **Deterministic.** A rule fires or it doesn't. Thresholds live in one
+   module, commented, tunable.
+2. **Evidence-cited.** Every finding names the steps and values that triggered
+   it.
+3. **Honest about limits.** What the tool cannot detect is documented in the
+   README, not discovered by the user in production.
+
+## Family
+
+trainproof judges training runs. Its sibling [ttsproof](https://github.com/Mormolykos/ttsproof)
+judges TTS model *outputs* (structural audio checks, equivalence-aware WER/CER,
+published method with DOI) — and trainproof builds on it for the speech
+dataset checks.
+
+## Author
+
+Panagiotis (Panos) Gkilis — [portfolio](https://tts.bedvibe.studio/portfolio/) ·
+[bedvibe.studio](https://bedvibe.studio/)
+
+MIT license.

{trainproof-0.1.0 → trainproof-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "trainproof"
-version = "0.1.0"
+version = "0.3.0"
 description = "A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs."
 authors = [{name = "Panagiotis (Panos) Gkilis", email = "bedvibe@bedvibe.studio"}]
 readme = "README.md"

{trainproof-0.1.0 → trainproof-0.3.0}/src/trainproof/cli.py

@@ -4,6 +4,7 @@ from pathlib import Path
 from .speech.data import check_data
 from .speech.tokenizer import check_tokenizer
 from .epoch import check_epoch
+from .compare import check_compare
 from .report import print_verdict_console, write_html_report
 
 def main():
@@ -24,6 +25,12 @@ def main():
     epoch_parser.add_argument("logfile", type=str, help="Path to JSONL or CSV log file")
     epoch_parser.add_argument("--format", choices=["auto", "hf", "coqui", "jsonl", "csv"], default="auto", help="Log format override")
 
+    # Compare Command
+    compare_parser = subparsers.add_parser("compare", help="Compare a run log against a baseline log.")
+    compare_parser.add_argument("run", type=str, help="Path to run log file")
+    compare_parser.add_argument("baseline", type=str, help="Path to baseline log file")
+    compare_parser.add_argument("--format", choices=["auto", "hf", "coqui", "jsonl", "csv"], default="auto", help="Log format override")
+
     args = parser.parse_args()
     
     report_dict = {}
@@ -38,6 +45,12 @@ def main():
         except Exception as e:
             print(f"Error checking epoch: {e}")
             sys.exit(1)
+    elif args.command == "compare":
+        try:
+            report_dict = check_compare(args.run, args.baseline, fmt=args.format)
+        except Exception as e:
+            print(f"Error checking compare: {e}")
+            sys.exit(1)
 
     print_verdict_console(report_dict.get("verdict", "FAIL"), report_dict.get("findings", []))
     

trainproof-0.3.0/src/trainproof/compare.py

@@ -0,0 +1,132 @@
+import math
+from pathlib import Path
+from typing import Any
+from . import rules
+from .adapters import parse_log_with_format
+from .epoch import check_epoch
+
+def _get_val(row, aliases):
+    for a in aliases:
+        for k in row:
+            if a in k: return row[k]
+    return None
+
+def extract_metrics(records):
+    losses = []
+    grad_norms = []
+
+    for r in records:
+        loss = _get_val(r, ["loss", "train_loss"])
+        gn = _get_val(r, ["grad_norm", "gnorm", "grad"])
+        
+        if loss is not None and not math.isnan(loss) and not math.isinf(loss):
+            losses.append(loss)
+        if gn is not None and not math.isnan(gn) and not math.isinf(gn):
+            grad_norms.append(gn)
+            
+    if len(losses) < 10:
+        return None
+        
+    floor = min(losses)
+    
+    w = rules.LOSS_IMPROVEMENT_WINDOW
+    start_med = sorted(losses[:w])[w // 2]
+    end_med = sorted(losses[-w:])[w // 2]
+    
+    improvement = 1.0 - (end_med / start_med) if start_med > 0 else 0.0
+    
+    gn_median = None
+    if len(grad_norms) > 5:
+        sorted_gns = sorted(grad_norms)
+        gn_median = sorted_gns[len(sorted_gns) // 2]
+        
+    return {
+        "floor": floor,
+        "start_med": start_med,
+        "end_med": end_med,
+        "improvement": improvement,
+        "gn_median": gn_median,
+        "losses_len": len(losses)
+    }
+
+def check_compare(run_path: str | Path, base_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
+    findings = []
+    verdict = "PASS"
+
+    # Baseline sanity — a suspect baseline downgrades the whole comparison
+    base_epoch_res = check_epoch(base_path, fmt=fmt)
+    if base_epoch_res["verdict"] == "FAIL":
+        verdict = "WARN"
+        findings.append({
+            "level": "WARN",
+            "message": "baseline itself fails single-run checks - comparison may be meaningless",
+            "evidence": str(base_path)
+        })
+
+    run_records = parse_log_with_format(run_path, fmt)
+    base_records = parse_log_with_format(base_path, fmt)
+
+    run_metrics = extract_metrics(run_records) if run_records else None
+    base_metrics = extract_metrics(base_records) if base_records else None
+
+    if not run_metrics:
+        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "Run log has fewer than 10 valid loss points.", "evidence": str(run_path)}]}
+    if not base_metrics:
+        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "Baseline log has fewer than 10 valid loss points.", "evidence": str(base_path)}]}
+
+    # Floor ratio
+    if run_metrics["floor"] > base_metrics["floor"] * rules.MAX_FLOOR_RATIO:
+        verdict = "FAIL"
+        findings.append({
+            "level": "FAIL",
+            "message": "loss floor ratio exceeded limit",
+            "evidence": f"Run floor {run_metrics['floor']:.3f} vs Baseline floor {base_metrics['floor']:.3f} (ratio {(run_metrics['floor'] / base_metrics['floor']):.1f}x > {rules.MAX_FLOOR_RATIO})"
+        })
+
+    # End-loss ratio: where the run landed vs where the baseline landed.
+    # Robust to corrupted starts (see rules.MAX_END_RATIO comment).
+    if base_metrics["end_med"] > 0 and run_metrics["end_med"] > base_metrics["end_med"] * rules.MAX_END_RATIO:
+        verdict = "FAIL"
+        findings.append({
+            "level": "FAIL",
+            "message": "end loss ratio exceeded limit",
+            "evidence": f"Run end {run_metrics['end_med']:.3f} vs Baseline end {base_metrics['end_med']:.3f} (ratio {(run_metrics['end_med'] / base_metrics['end_med']):.1f}x > {rules.MAX_END_RATIO})"
+        })
+
+    # Improvement deficit
+    run_imp = run_metrics["improvement"]
+    base_imp = base_metrics["improvement"]
+    
+    if run_imp < 0:
+        verdict = "FAIL"
+        findings.append({
+            "level": "FAIL",
+            "message": "negative improvement",
+            "evidence": f"Run improvement is {run_imp * 100:.1f}% vs Baseline {base_imp * 100:.1f}%"
+        })
+    elif base_imp > 0 and run_imp < base_imp * rules.MIN_IMPROVEMENT_FRACTION:
+        verdict = "FAIL"
+        findings.append({
+            "level": "FAIL",
+            "message": "improvement deficit",
+            "evidence": f"Run improvement {run_imp * 100:.1f}% vs Baseline {base_imp * 100:.1f}% (ratio {(run_imp / base_imp):.2f}x < {rules.MIN_IMPROVEMENT_FRACTION})"
+        })
+
+    # Gradnorm median ratio
+    if run_metrics["gn_median"] is not None and base_metrics["gn_median"] is not None and base_metrics["gn_median"] > 0:
+        if run_metrics["gn_median"] > base_metrics["gn_median"] * rules.MAX_GRADNORM_MEDIAN_RATIO:
+            if verdict == "PASS": verdict = "WARN"
+            findings.append({
+                "level": "WARN",
+                "message": "gradient norm median significantly higher than baseline",
+                "evidence": f"Run gn median {run_metrics['gn_median']:.2f} vs Baseline gn median {base_metrics['gn_median']:.2f} (ratio {(run_metrics['gn_median'] / base_metrics['gn_median']):.1f}x > {rules.MAX_GRADNORM_MEDIAN_RATIO})"
+            })
+
+    if verdict == "PASS" and not any(f["level"] == "WARN" for f in findings):
+        findings.append({
+            "level": "PASS",
+            "message": "Run compares favorably to baseline.",
+            "evidence": f"Floor ratio {(run_metrics['floor'] / base_metrics['floor']):.2f}x, Improvement {run_imp*100:.1f}% vs {base_imp*100:.1f}%"
+        })
+
+    return {"verdict": verdict, "findings": findings}

{trainproof-0.1.0 → trainproof-0.3.0}/src/trainproof/epoch.py

@@ -67,6 +67,16 @@ def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
             findings.append({"level": "FAIL", "message": "Loss curve is diverging.", "evidence": f"End loss {valid_losses[-1]:.3f} vs Min loss {min_loss:.3f}"})
             verdict = "FAIL"
 
+        # Check no-improvement (dead run): robust start-vs-end median comparison
+        if len(valid_losses) >= rules.MIN_POINTS_FOR_IMPROVEMENT_CHECK:
+            w = rules.LOSS_IMPROVEMENT_WINDOW
+            start_med = sorted(valid_losses[:w])[w // 2]
+            end_med = sorted(valid_losses[-w:])[w // 2]
+            if start_med > 0 and end_med >= start_med * (1 - rules.MIN_LOSS_IMPROVEMENT):
+                findings.append({"level": "FAIL", "message": "Loss never improved over the run (dead run).",
+                                 "evidence": f"median of first {w} losses {start_med:.3f} vs last {w} {end_med:.3f} (needs >={rules.MIN_LOSS_IMPROVEMENT*100:.0f}% improvement)"})
+                verdict = "FAIL"
+
     # Check Grad Norm
     valid_gns = [g for g in grad_norms if not math.isnan(g) and not math.isinf(g)]
     if valid_gns and len(valid_gns) > 5:
@@ -82,7 +92,10 @@ def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
     if lrs:
         zeros = sum(1 for lr in lrs if lr <= 0)
         zero_frac = zeros / len(lrs)
-        if zero_frac > rules.MAX_ZERO_LR_FRACTION:
+        if zero_frac >= rules.ZERO_LR_FAIL_FRACTION:
+            findings.append({"level": "FAIL", "message": "Learning rate is zero for the entire run - the optimizer never steps.", "evidence": f"{zero_frac*100:.1f}% of steps have lr=0"})
+            verdict = "FAIL"
+        elif zero_frac > rules.MAX_ZERO_LR_FRACTION:
             findings.append({"level": "WARN", "message": "Learning rate is zero for a large fraction of the run.", "evidence": f"{zero_frac*100:.1f}% of steps have lr=0"})
             if verdict == "PASS": verdict = "WARN"
 

trainproof-0.3.0/src/trainproof/rules.py

@@ -0,0 +1,90 @@
+"""Deterministic thresholds for verdicts."""
+
+# -----------------
+# DATA SUBCOMMAND
+# -----------------
+
+# Maximum allowed duration for a single audio file in seconds.
+MAX_AUDIO_DURATION_SEC = 25.0
+
+# Minimum allowed duration for a single audio file in seconds.
+MIN_AUDIO_DURATION_SEC = 0.5
+
+# Peak amplitude threshold above which we consider audio to be clipping.
+MAX_CLIPPING_PEAK = 0.99
+
+# Maximum allowed continuous silence in seconds before warning.
+MAX_SILENCE_SEC = 2.0
+
+# Amplitude threshold for considering a frame as "silent" (for silence detection).
+SILENCE_AMPLITUDE = 0.005
+
+# Flag text/audio alignment outliers whose chars-per-second rate deviates from
+# the corpus median by more than this ratio (in either direction).
+CHARS_PER_SEC_OUTLIER_RATIO = 3.0
+
+# -----------------
+# TOKENIZER SUBCOMMAND
+# -----------------
+
+# Minimum fraction of the corpus vocabulary that must be covered by the tokenizer.
+MIN_VOCAB_COVERAGE = 0.999
+
+# Maximum acceptable Out-Of-Vocabulary (OOV) rate.
+MAX_OOV_RATE = 0.001
+
+# Maximum expected tokens per second of audio (blowout warning).
+MAX_TOKENS_PER_SEC = 50.0
+
+# -----------------
+# EPOCH SUBCOMMAND
+# -----------------
+
+# If loss increases by more than this ratio from the minimum loss, flag as diverging.
+MAX_LOSS_DIVERGENCE_RATIO = 1.5
+
+# Minimum required variation (std/mean) in the loss curve to not be considered "flat/dead".
+MIN_LOSS_VARIATION = 0.001
+
+# Maximum allowable spike in gradient norm compared to the median grad norm.
+MAX_GRAD_NORM_SPIKE_RATIO = 10.0
+
+# If the learning rate is strictly zero for more than this fraction of the epoch, it's a warning.
+MAX_ZERO_LR_FRACTION = 0.1
+
+# If the learning rate is zero for essentially the ENTIRE run, the optimizer
+# never steps — the run cannot train, regardless of how the loss wiggles
+# (multi-seed testing showed batch-order noise can fake >5% "improvement"
+# under lr=0). Total zero-LR is a fatality, not a warning.
+ZERO_LR_FAIL_FRACTION = 0.99
+
+# Dead-run detection (added in v0.2 after fault-injection testing showed that
+# zero-LR and garbage-label runs — which can never learn — only earned WARNs):
+# the median of the last LOSS_IMPROVEMENT_WINDOW losses must be at least
+# MIN_LOSS_IMPROVEMENT (relative) below the median of the first window, else
+# the run never learned. Note: this targets FIRST-EPOCH linting; a model
+# resumed at its convergence floor would also trip it.
+MIN_LOSS_IMPROVEMENT = 0.05
+LOSS_IMPROVEMENT_WINDOW = 5
+MIN_POINTS_FOR_IMPROVEMENT_CHECK = 10
+
+# -----------------
+# COMPARE SUBCOMMAND
+# -----------------
+
+# FAIL "loss floor ratio" if run_floor / baseline_floor > 2.0
+MAX_FLOOR_RATIO = 2.0
+
+# FAIL "end loss ratio" if the run's final loss (median of last window) lands
+# more than this factor above the baseline's. Judges where the run LANDED —
+# robust both to noisy first steps and to corruption inside the first window
+# (e.g. an LR explosion at step 10 inflates the run's own start median, making
+# self-relative improvement look positive; the end ratio still catches it).
+MAX_END_RATIO = 2.0
+
+# FAIL "improvement deficit" if the run's relative improvement is < 25% of the baseline's
+# (only evaluated when baseline improvement > 0). A run with NEGATIVE improvement always trips this rule.
+MIN_IMPROVEMENT_FRACTION = 0.25
+
+# WARN if run's grad-norm median > 5x baseline's (skip if either lacks grad norms)
+MAX_GRADNORM_MEDIAN_RATIO = 5.0

trainproof-0.3.0/src/trainproof.egg-info/PKG-INFO

@@ -0,0 +1,153 @@
+Metadata-Version: 2.4
+Name: trainproof
+Version: 0.3.0
+Summary: A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.
+Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
+License: MIT
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: numpy
+Requires-Dist: soundfile
+Requires-Dist: ttsproof
+
+# trainproof
+
+[![PyPI](https://img.shields.io/pypi/v/trainproof)](https://pypi.org/project/trainproof/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**A deterministic linter for ML training runs.** Point it at your dataset, your
+tokenizer, or your first-epoch logs — it returns a PASS / WARN / FAIL verdict
+with named findings and cited evidence, before you burn days of GPU time on a
+run that was doomed at step 50.
+
+No ML judging ML. No invented "confidence 97%". Every rule is a deterministic
+threshold in [one auditable module](src/trainproof/rules.py), and every finding
+cites the numbers that triggered it.
+
+```bash
+pip install trainproof
+```
+
+## See a verdict in 60 seconds
+
+This repo ships the real logs of five QLoRA fine-tuning runs (Qwen2.5-3B,
+RTX 5080 — see the gallery below). Judge one right now:
+
+```bash
+trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf
+```
+
+```text
+[FAIL] Critical checks failed:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 7.492 vs Min loss 1.398
+  [WARN] Gradient norm spikes detected.
+         Evidence: Max gn 2649.75 > 10.0x median (0.55)
+```
+
+## Why this exists
+
+The author lost a real 11-hour fine-tune to a failure nothing warned about.
+Pointed retroactively at that run's 1MB Coqui Trainer log (2,501 logged steps),
+trainproof's verdict: **FAIL — diverging**. The loss reached its minimum at 82%
+of the run and ended 1.9x above it. Translation: the final two hours of GPU
+time made the model measurably worse, and the checkpoint worth keeping had
+already existed for hours. No tool in the stack said a word.
+
+## The fault-injection gallery
+
+To validate the rules, the same QLoRA fine-tune (Qwen2.5-3B-Instruct, 4-bit,
+LoRA r=16, 300 steps on Alpaca-cleaned) was run five times — once healthy, four
+times with exactly one knob deliberately broken. Real runs, real logs, all
+shipped in [`examples/gallery/`](examples/gallery/):
+
+| Run | Sabotage | Verdict | Key evidence |
+|---|---|---|---|
+| `healthy` | none | **PASS** | loss 1.52 → 0.94, stable gradients |
+| `lr_hot` | LR x100 (2e-2) | **FAIL** | diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55 |
+| `lr_zero` | LR = 0 | **FAIL** | dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps |
+| `fp16_nan` | fp16 + hot LR, no clipping | **FAIL** | diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed) |
+| `bad_labels` | labels shuffled per-sequence | **WARN only** (single-run) — caught by `trainproof compare` (v0.3) | grad spike 23.3 vs median 1.09 |
+
+### The honest finding: loss curves cannot see corrupted data
+
+The `bad_labels` run — whose shuffled labels make real learning impossible —
+*reduced its loss by 62%* (18.9 → 5.75). The model was genuinely learning: not
+the task, but the marginal token statistics of the garbage. From its own loss
+curve, that is indistinguishable from healthy training (neural networks
+famously fit random labels). **No single-run, loss-only rule can catch this
+class of failure** — its real signature is *relative*: a loss floor ~6x higher
+than a known-good run of the same task (5.59 vs 0.94).
+
+That finding sets the roadmap: v0.3 is `trainproof compare <run> <baseline>` —
+deterministic ratio rules against the healthy baseline you already have.
+See [ROADMAP.md](ROADMAP.md). (The gallery also improved v0.2 itself: the
+dead-run rule exists because `lr_zero` initially escaped with only a WARN.)
+
+## The three commands
+
+```bash
+# 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
+#    duplicates, text-vs-audio duration mismatches
+trainproof data /path/to/dataset_or_manifest.jsonl
+
+# 2. Tokenizer preflight: vocabulary coverage, OOV rate, sequence blowouts,
+#    suspicious splits on numbers/dates
+trainproof tokenizer my_tokenizer.model transcripts.txt
+
+# 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
+#    LR sanity, throughput — from log files, any framework
+trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready
+
+# 4. Compare against a baseline
+#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
+trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] loss floor ratio exceeded limit
+         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
+  [FAIL] end loss ratio exceeded limit
+         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
+========================================
+```
+
+Each command prints the verdict, writes a self-contained HTML report, and sets
+the process exit code — so it works as a CI gate out of the box.
+
+## Supported log formats
+
+- HuggingFace Trainer (`trainer_state.json`)
+- Coqui Trainer text logs (ANSI-colored `trainer_0_log.txt`)
+- Generic JSONL / CSV (columns: step, loss, lr, grad_norm, time — all optional)
+
+Auto-detected; override with `--format hf|coqui|jsonl|csv`. TensorBoard event
+files are planned (Lightning console captures are TTY dumps, not logs, and
+will not be supported).
+
+## Philosophy
+
+1. **Deterministic.** A rule fires or it doesn't. Thresholds live in one
+   module, commented, tunable.
+2. **Evidence-cited.** Every finding names the steps and values that triggered
+   it.
+3. **Honest about limits.** What the tool cannot detect is documented in the
+   README, not discovered by the user in production.
+
+## Family
+
+trainproof judges training runs. Its sibling [ttsproof](https://github.com/Mormolykos/ttsproof)
+judges TTS model *outputs* (structural audio checks, equivalence-aware WER/CER,
+published method with DOI) — and trainproof builds on it for the speech
+dataset checks.
+
+## Author
+
+Panagiotis (Panos) Gkilis — [portfolio](https://tts.bedvibe.studio/portfolio/) ·
+[bedvibe.studio](https://bedvibe.studio/)
+
+MIT license.

{trainproof-0.1.0 → trainproof-0.3.0}/src/trainproof.egg-info/SOURCES.txt

@@ -3,6 +3,7 @@ pyproject.toml
 src/trainproof/__init__.py
 src/trainproof/adapters.py
 src/trainproof/cli.py
+src/trainproof/compare.py
 src/trainproof/epoch.py
 src/trainproof/report.py
 src/trainproof/rules.py
@@ -16,5 +17,6 @@ src/trainproof/speech/__init__.py
 src/trainproof/speech/data.py
 src/trainproof/speech/tokenizer.py
 tests/test_adapters.py
+tests/test_compare.py
 tests/test_data.py
 tests/test_epoch.py

trainproof-0.3.0/tests/test_compare.py

@@ -0,0 +1,47 @@
+import pytest
+from pathlib import Path
+from trainproof.compare import check_compare
+
+def get_gallery_log(name: str) -> str:
+    path = Path(__file__).parent.parent / "examples" / "gallery" / name / "trainer_state.json"
+    return str(path.resolve())
+
+def test_compare_healthy_vs_healthy():
+    healthy = get_gallery_log("healthy")
+    res = check_compare(healthy, healthy, fmt="hf")
+    assert res["verdict"] == "PASS"
+
+def test_compare_bad_labels_vs_healthy():
+    bad_labels = get_gallery_log("bad_labels")
+    healthy = get_gallery_log("healthy")
+    res = check_compare(bad_labels, healthy, fmt="hf")
+    assert res["verdict"] == "FAIL"
+    findings = [f for f in res["findings"] if "loss floor ratio" in f["message"]]
+    assert len(findings) > 0
+
+def test_compare_lr_zero_vs_healthy():
+    lr_zero = get_gallery_log("lr_zero")
+    healthy = get_gallery_log("healthy")
+    res = check_compare(lr_zero, healthy, fmt="hf")
+    assert res["verdict"] == "FAIL"
+    findings = [f for f in res["findings"] if "improvement deficit" in f["message"]]
+    assert len(findings) > 0
+
+def test_compare_lr_hot_vs_healthy():
+    # lr_hot's explosion happens INSIDE the first-5 window, inflating its own
+    # start median — self-relative improvement looks positive (+62%). Only the
+    # end-loss ratio (where the run LANDED: ~7x the baseline) catches it.
+    lr_hot = get_gallery_log("lr_hot")
+    healthy = get_gallery_log("healthy")
+    res = check_compare(lr_hot, healthy, fmt="hf")
+    assert res["verdict"] == "FAIL"
+    findings = [f for f in res["findings"] if "end loss ratio" in f["message"]]
+    assert len(findings) > 0
+
+def test_compare_fp16_nan_vs_healthy():
+    fp16_nan = get_gallery_log("fp16_nan")
+    healthy = get_gallery_log("healthy")
+    res = check_compare(fp16_nan, healthy, fmt="hf")
+    assert res["verdict"] == "FAIL"
+    findings = [f for f in res["findings"] if "negative improvement" in f["message"]]
+    assert len(findings) > 0

{trainproof-0.1.0 → trainproof-0.3.0}/tests/test_epoch.py

@@ -34,3 +34,18 @@ def test_epoch_flat():
     report = check_epoch(FIXTURES_DIR / "flat.jsonl")
     assert report["verdict"] == "FAIL"
     assert any("completely flat" in str(f) for f in report["findings"])
+
+def test_epoch_dead_noisy():
+    # noisy-but-never-improving loss with healthy lr: only the no-improvement
+    # rule (v0.2) can catch this dead run
+    report = check_epoch(FIXTURES_DIR / "dead_noisy.jsonl")
+    assert report["verdict"] == "FAIL"
+    assert any("never improved" in str(f) for f in report["findings"])
+
+def test_epoch_total_zero_lr_is_fatal():
+    # lr=0 on every step: the optimizer never moves — FAIL regardless of how
+    # batch-order noise makes the loss wiggle (found by multi-seed testing)
+    gallery = Path(__file__).parent.parent / "examples" / "gallery"
+    report = check_epoch(gallery / "lr_zero" / "trainer_state.json", fmt="hf")
+    assert report["verdict"] == "FAIL"
+    assert any("optimizer never steps" in str(f) for f in report["findings"])

trainproof-0.1.0/PKG-INFO

@@ -1,62 +0,0 @@
-Metadata-Version: 2.4
-Name: trainproof
-Version: 0.1.0
-Summary: A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.
-Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
-License: MIT
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-Requires-Dist: numpy
-Requires-Dist: soundfile
-Requires-Dist: ttsproof
-
-# Trainproof
-
-A deterministic linter for ML training runs. Run it on your dataset, tokenizer, and first-epoch logs — it gives a deterministic PASS/WARN/FAIL verdict with named findings and suggested fixes, BEFORE you burn weeks of GPU time.
-
-## Installation
-```bash
-pip install .
-```
-
-## Usage
-
-Exactly three subcommands:
-
-### 1. Dataset Preflight
-```bash
-trainproof data /path/to/dataset
-```
-Checks audio integrity (clipping, silence, duration distribution) and transcript quality (unnormalized text, charset audit, duration correlation).
-
-### 2. Tokenizer Preflight
-```bash
-trainproof tokenizer my_model.model transcripts.txt
-```
-Checks vocabulary coverage, tokens-per-second, and suspicious splits on numbers/dates.
-
-### 3. First-Epoch Verification
-```bash
-trainproof epoch logs/epoch1.jsonl
-```
-Analyzes loss curves (divergence, flatlines, NaN), grad norms, learning rate response, and throughput.
-
-## Supported log formats
-- **Generic JSONL / CSV**
-- **HuggingFace** (`trainer_state.json`)
-- **Coqui TTS Trainer** (plain text `trainer_0_log.txt`)
-
-*Roadmap: TensorBoard event files planned (v0.2) — Lightning console captures are TTY dumps, not logs, and will not be supported.*
-
-## Verdict Rules
-All verdict rules are deterministic thresholds defined centrally in `src/trainproof/rules.py`. Examples include:
-- `MAX_CLIPPING_PEAK = 0.99`
-- `MAX_LOSS_DIVERGENCE_RATIO = 1.5`
-- `MIN_VOCAB_COVERAGE = 0.999`
-
-## Explicit Non-Goals
-- No live learning-rate auto-adjustment.
-- No PyTorch/Lightning callbacks or framework hooks (log files only).
-- No MCP server, no LLM integration.
-- No dashboards/wandb-style UI.
-- No extra features beyond this spec.

trainproof-0.1.0/README.md

@@ -1,50 +0,0 @@
-# Trainproof
-
-A deterministic linter for ML training runs. Run it on your dataset, tokenizer, and first-epoch logs — it gives a deterministic PASS/WARN/FAIL verdict with named findings and suggested fixes, BEFORE you burn weeks of GPU time.
-
-## Installation
-```bash
-pip install .
-```
-
-## Usage
-
-Exactly three subcommands:
-
-### 1. Dataset Preflight
-```bash
-trainproof data /path/to/dataset
-```
-Checks audio integrity (clipping, silence, duration distribution) and transcript quality (unnormalized text, charset audit, duration correlation).
-
-### 2. Tokenizer Preflight
-```bash
-trainproof tokenizer my_model.model transcripts.txt
-```
-Checks vocabulary coverage, tokens-per-second, and suspicious splits on numbers/dates.
-
-### 3. First-Epoch Verification
-```bash
-trainproof epoch logs/epoch1.jsonl
-```
-Analyzes loss curves (divergence, flatlines, NaN), grad norms, learning rate response, and throughput.
-
-## Supported log formats
-- **Generic JSONL / CSV**
-- **HuggingFace** (`trainer_state.json`)
-- **Coqui TTS Trainer** (plain text `trainer_0_log.txt`)
-
-*Roadmap: TensorBoard event files planned (v0.2) — Lightning console captures are TTY dumps, not logs, and will not be supported.*
-
-## Verdict Rules
-All verdict rules are deterministic thresholds defined centrally in `src/trainproof/rules.py`. Examples include:
-- `MAX_CLIPPING_PEAK = 0.99`
-- `MAX_LOSS_DIVERGENCE_RATIO = 1.5`
-- `MIN_VOCAB_COVERAGE = 0.999`
-
-## Explicit Non-Goals
-- No live learning-rate auto-adjustment.
-- No PyTorch/Lightning callbacks or framework hooks (log files only).
-- No MCP server, no LLM integration.
-- No dashboards/wandb-style UI.
-- No extra features beyond this spec.

trainproof-0.1.0/src/trainproof/rules.py

@@ -1,53 +0,0 @@
-"""Deterministic thresholds for verdicts."""
-
-# -----------------
-# DATA SUBCOMMAND
-# -----------------
-
-# Maximum allowed duration for a single audio file in seconds.
-MAX_AUDIO_DURATION_SEC = 25.0
-
-# Minimum allowed duration for a single audio file in seconds.
-MIN_AUDIO_DURATION_SEC = 0.5
-
-# Peak amplitude threshold above which we consider audio to be clipping.
-MAX_CLIPPING_PEAK = 0.99
-
-# Maximum allowed continuous silence in seconds before warning.
-MAX_SILENCE_SEC = 2.0
-
-# Amplitude threshold for considering a frame as "silent" (for silence detection).
-SILENCE_AMPLITUDE = 0.005
-
-# Flag text/audio alignment outliers whose chars-per-second rate deviates from
-# the corpus median by more than this ratio (in either direction).
-CHARS_PER_SEC_OUTLIER_RATIO = 3.0
-
-# -----------------
-# TOKENIZER SUBCOMMAND
-# -----------------
-
-# Minimum fraction of the corpus vocabulary that must be covered by the tokenizer.
-MIN_VOCAB_COVERAGE = 0.999
-
-# Maximum acceptable Out-Of-Vocabulary (OOV) rate.
-MAX_OOV_RATE = 0.001
-
-# Maximum expected tokens per second of audio (blowout warning).
-MAX_TOKENS_PER_SEC = 50.0
-
-# -----------------
-# EPOCH SUBCOMMAND
-# -----------------
-
-# If loss increases by more than this ratio from the minimum loss, flag as diverging.
-MAX_LOSS_DIVERGENCE_RATIO = 1.5
-
-# Minimum required variation (std/mean) in the loss curve to not be considered "flat/dead".
-MIN_LOSS_VARIATION = 0.001
-
-# Maximum allowable spike in gradient norm compared to the median grad norm.
-MAX_GRAD_NORM_SPIKE_RATIO = 10.0
-
-# If the learning rate is strictly zero for more than this fraction of the epoch, it's a warning.
-MAX_ZERO_LR_FRACTION = 0.1

trainproof-0.1.0/src/trainproof.egg-info/PKG-INFO

@@ -1,62 +0,0 @@
-Metadata-Version: 2.4
-Name: trainproof
-Version: 0.1.0
-Summary: A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.
-Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
-License: MIT
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-Requires-Dist: numpy
-Requires-Dist: soundfile
-Requires-Dist: ttsproof
-
-# Trainproof
-
-A deterministic linter for ML training runs. Run it on your dataset, tokenizer, and first-epoch logs — it gives a deterministic PASS/WARN/FAIL verdict with named findings and suggested fixes, BEFORE you burn weeks of GPU time.
-
-## Installation
-```bash
-pip install .
-```
-
-## Usage
-
-Exactly three subcommands:
-
-### 1. Dataset Preflight
-```bash
-trainproof data /path/to/dataset
-```
-Checks audio integrity (clipping, silence, duration distribution) and transcript quality (unnormalized text, charset audit, duration correlation).
-
-### 2. Tokenizer Preflight
-```bash
-trainproof tokenizer my_model.model transcripts.txt
-```
-Checks vocabulary coverage, tokens-per-second, and suspicious splits on numbers/dates.
-
-### 3. First-Epoch Verification
-```bash
-trainproof epoch logs/epoch1.jsonl
-```
-Analyzes loss curves (divergence, flatlines, NaN), grad norms, learning rate response, and throughput.
-
-## Supported log formats
-- **Generic JSONL / CSV**
-- **HuggingFace** (`trainer_state.json`)
-- **Coqui TTS Trainer** (plain text `trainer_0_log.txt`)
-
-*Roadmap: TensorBoard event files planned (v0.2) — Lightning console captures are TTY dumps, not logs, and will not be supported.*
-
-## Verdict Rules
-All verdict rules are deterministic thresholds defined centrally in `src/trainproof/rules.py`. Examples include:
-- `MAX_CLIPPING_PEAK = 0.99`
-- `MAX_LOSS_DIVERGENCE_RATIO = 1.5`
-- `MIN_VOCAB_COVERAGE = 0.999`
-
-## Explicit Non-Goals
-- No live learning-rate auto-adjustment.
-- No PyTorch/Lightning callbacks or framework hooks (log files only).
-- No MCP server, no LLM integration.
-- No dashboards/wandb-style UI.
-- No extra features beyond this spec.