trainproof 0.3.0__tar.gz → 0.5.0__tar.gz

{trainproof-0.3.0/src/trainproof.egg-info → trainproof-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: trainproof
-Version: 0.3.0
+Version: 0.5.0
 Summary: A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.
 Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
 License: MIT
@@ -24,6 +24,16 @@ No ML judging ML. No invented "confidence 97%". Every rule is a deterministic
 threshold in [one auditable module](src/trainproof/rules.py), and every finding
 cites the numbers that triggered it.
 
+**What it is:** the reliability layer between your training code and the GPU
+bill — checks that run *before* training (is this run safe to start?), *during*
+training (should it keep going?), and *after* (is it reproducible; did it match
+a known-good baseline?).
+
+**The one rule it never breaks:** trainproof does not infer causes, does not
+invent confidence scores, and does not guess. It reports deterministic findings
+backed by evidence, or it stays silent. Every feature earns its place by
+answering a single question — *if it were gone, would someone lose GPU hours?*
+
 ```bash
 pip install trainproof
 ```
@@ -79,12 +89,18 @@ famously fit random labels). **No single-run, loss-only rule can catch this
 class of failure** — its real signature is *relative*: a loss floor ~6x higher
 than a known-good run of the same task (5.59 vs 0.94).
 
-That finding sets the roadmap: v0.3 is `trainproof compare <run> <baseline>` —
-deterministic ratio rules against the healthy baseline you already have.
-See [ROADMAP.md](ROADMAP.md). (The gallery also improved v0.2 itself: the
-dead-run rule exists because `lr_zero` initially escaped with only a WARN.)
+That finding produced v0.3: `trainproof compare <run> <baseline>` —
+deterministic ratio rules against the healthy baseline you already have —
+which catches `bad_labels` at a 6x loss-floor ratio, in 3 seeds out of 3.
+The full study was repeated with three random seeds (15 runs):
+see [EVIDENCE_MATRIX.md](EVIDENCE_MATRIX.md) for every verdict, including the
+honest miss (compare alone overlooks one lr_zero seed — the single-run
+zero-LR fatality rule owns that case; the two commands cover each other's
+blind spots). The gallery also improved the tool itself twice: the dead-run
+rule and the total-zero-LR fatality rule both exist because runs escaped
+earlier rule versions. See [ROADMAP.md](ROADMAP.md).
 
-## The three commands
+## The commands
 
 ```bash
 # 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
@@ -119,6 +135,79 @@ TRAINPROOF VERDICT
 Each command prints the verdict, writes a self-contained HTML report, and sets
 the process exit code — so it works as a CI gate out of the box.
 
+## Live guardian (v0.4)
+
+Don't wait for the post-mortem — catch a doomed run *while it is still burning
+GPU*. Add one line to a HuggingFace `Trainer`:
+
+```python
+from transformers import Trainer
+from trainproof.integrations.hf import TrainproofCallback
+
+trainer = Trainer(
+    ...,
+    callbacks=[TrainproofCallback(policy="stop_on_fail")],  # or policy="warn"
+)
+```
+
+Run against a real diverging QLoRA fine-tune (learning rate 100x too high), the
+guardian aborts it 20 steps into a 300-step schedule — on its own:
+
+```text
+{'loss': '1.784', 'grad_norm': '9.634',  'learning_rate': '0.007'}
+{'loss': '4.282', 'grad_norm': '53.76',  'learning_rate': '0.009'}
+{'loss': '10.6',  'grad_norm': '13.34',  'learning_rate': '0.011'}
+{'loss': '31.67', 'grad_norm': '76.67',  'learning_rate': '0.013'}
+...
+TRAINPROOF ABORT - stopping training at step 20. Findings:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 22.952 vs Min loss 1.358
+  [FAIL] Loss never improved over the run (dead run).
+         Evidence: median of first 5 losses 1.502 vs last 5 22.952
+
+  scheduled steps : 300
+  stopped at step : 20
+  run saved       : 93% of the scheduled steps never ran
+```
+
+On a two-day pre-training run, that fraction is days of GPU time. Or watch a
+growing log file from outside the process (CI-friendly, exits non-zero on FAIL):
+
+```bash
+trainproof watch logs/run.jsonl --interval 10 --until-fail
+# [21:37:44] warming up (5 records)
+# [21:37:44] n_records=15 verdict=PASS findings=1
+```
+
+**The default is safe.** `policy="warn"` (the default) only observes and reports
+— it never interrupts your run, so you can leave it on even for experiments you
+expect to fail. Aborting is strictly opt-in via `policy="stop_on_fail"`, the one
+mode that takes an irreversible action. trainproof does not make that decision
+for you unless you ask.
+
+The guardian applies the same deterministic rules as `trainproof epoch`, so it
+inherits their documented single-run limitations.
+
+## Pre-flight (v0.5): stop a run before it starts
+
+The guardian saves most of a doomed run; preflight saves 100% because it never starts. Catch broken datasets and tokenizer misconfigurations instantly.
+
+```bash
+trainproof preflight data/dataset.jsonl --tokenizer mistralai/Mistral-7B-v0.1 --max-len 4096
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] Empty or whitespace-only text found.
+         Evidence: 1 records (indices [1]...)
+========================================
+```
+
+*Checks: malformed JSONL, empty text, exact duplicate text, tokenizer structural checks (EOS/PAD/BOS), and context length overflows.*
+
 ## Supported log formats
 
 - HuggingFace Trainer (`trainer_state.json`)

trainproof-0.3.0/PKG-INFO → trainproof-0.5.0/README.md

@@ -1,153 +1,230 @@
-Metadata-Version: 2.4
-Name: trainproof
-Version: 0.3.0
-Summary: A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs.
-Author-email: "Panagiotis (Panos) Gkilis" <bedvibe@bedvibe.studio>
-License: MIT
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-Requires-Dist: numpy
-Requires-Dist: soundfile
-Requires-Dist: ttsproof
-
-# trainproof
-
-[![PyPI](https://img.shields.io/pypi/v/trainproof)](https://pypi.org/project/trainproof/)
-[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-
-**A deterministic linter for ML training runs.** Point it at your dataset, your
-tokenizer, or your first-epoch logs — it returns a PASS / WARN / FAIL verdict
-with named findings and cited evidence, before you burn days of GPU time on a
-run that was doomed at step 50.
-
-No ML judging ML. No invented "confidence 97%". Every rule is a deterministic
-threshold in [one auditable module](src/trainproof/rules.py), and every finding
-cites the numbers that triggered it.
-
-```bash
-pip install trainproof
-```
-
-## See a verdict in 60 seconds
-
-This repo ships the real logs of five QLoRA fine-tuning runs (Qwen2.5-3B,
-RTX 5080 — see the gallery below). Judge one right now:
-
-```bash
-trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf
-```
-
-```text
-[FAIL] Critical checks failed:
-  [FAIL] Loss curve is diverging.
-         Evidence: End loss 7.492 vs Min loss 1.398
-  [WARN] Gradient norm spikes detected.
-         Evidence: Max gn 2649.75 > 10.0x median (0.55)
-```
-
-## Why this exists
-
-The author lost a real 11-hour fine-tune to a failure nothing warned about.
-Pointed retroactively at that run's 1MB Coqui Trainer log (2,501 logged steps),
-trainproof's verdict: **FAIL — diverging**. The loss reached its minimum at 82%
-of the run and ended 1.9x above it. Translation: the final two hours of GPU
-time made the model measurably worse, and the checkpoint worth keeping had
-already existed for hours. No tool in the stack said a word.
-
-## The fault-injection gallery
-
-To validate the rules, the same QLoRA fine-tune (Qwen2.5-3B-Instruct, 4-bit,
-LoRA r=16, 300 steps on Alpaca-cleaned) was run five times — once healthy, four
-times with exactly one knob deliberately broken. Real runs, real logs, all
-shipped in [`examples/gallery/`](examples/gallery/):
-
-| Run | Sabotage | Verdict | Key evidence |
-|---|---|---|---|
-| `healthy` | none | **PASS** | loss 1.52 → 0.94, stable gradients |
-| `lr_hot` | LR x100 (2e-2) | **FAIL** | diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55 |
-| `lr_zero` | LR = 0 | **FAIL** | dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps |
-| `fp16_nan` | fp16 + hot LR, no clipping | **FAIL** | diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed) |
-| `bad_labels` | labels shuffled per-sequence | **WARN only** (single-run) — caught by `trainproof compare` (v0.3) | grad spike 23.3 vs median 1.09 |
-
-### The honest finding: loss curves cannot see corrupted data
-
-The `bad_labels` run — whose shuffled labels make real learning impossible —
-*reduced its loss by 62%* (18.9 → 5.75). The model was genuinely learning: not
-the task, but the marginal token statistics of the garbage. From its own loss
-curve, that is indistinguishable from healthy training (neural networks
-famously fit random labels). **No single-run, loss-only rule can catch this
-class of failure** — its real signature is *relative*: a loss floor ~6x higher
-than a known-good run of the same task (5.59 vs 0.94).
-
-That finding sets the roadmap: v0.3 is `trainproof compare <run> <baseline>` —
-deterministic ratio rules against the healthy baseline you already have.
-See [ROADMAP.md](ROADMAP.md). (The gallery also improved v0.2 itself: the
-dead-run rule exists because `lr_zero` initially escaped with only a WARN.)
-
-## The three commands
-
-```bash
-# 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
-#    duplicates, text-vs-audio duration mismatches
-trainproof data /path/to/dataset_or_manifest.jsonl
-
-# 2. Tokenizer preflight: vocabulary coverage, OOV rate, sequence blowouts,
-#    suspicious splits on numbers/dates
-trainproof tokenizer my_tokenizer.model transcripts.txt
-
-# 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
-#    LR sanity, throughput — from log files, any framework
-trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready
-
-# 4. Compare against a baseline
-#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
-trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json
-```
-
-```text
-========================================
-TRAINPROOF VERDICT
-========================================
-[FAIL] Critical checks failed:
-  [FAIL] loss floor ratio exceeded limit
-         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
-  [FAIL] end loss ratio exceeded limit
-         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
-========================================
-```
-
-Each command prints the verdict, writes a self-contained HTML report, and sets
-the process exit code — so it works as a CI gate out of the box.
-
-## Supported log formats
-
-- HuggingFace Trainer (`trainer_state.json`)
-- Coqui Trainer text logs (ANSI-colored `trainer_0_log.txt`)
-- Generic JSONL / CSV (columns: step, loss, lr, grad_norm, time — all optional)
-
-Auto-detected; override with `--format hf|coqui|jsonl|csv`. TensorBoard event
-files are planned (Lightning console captures are TTY dumps, not logs, and
-will not be supported).
-
-## Philosophy
-
-1. **Deterministic.** A rule fires or it doesn't. Thresholds live in one
-   module, commented, tunable.
-2. **Evidence-cited.** Every finding names the steps and values that triggered
-   it.
-3. **Honest about limits.** What the tool cannot detect is documented in the
-   README, not discovered by the user in production.
-
-## Family
-
-trainproof judges training runs. Its sibling [ttsproof](https://github.com/Mormolykos/ttsproof)
-judges TTS model *outputs* (structural audio checks, equivalence-aware WER/CER,
-published method with DOI) — and trainproof builds on it for the speech
-dataset checks.
-
-## Author
-
-Panagiotis (Panos) Gkilis — [portfolio](https://tts.bedvibe.studio/portfolio/) ·
-[bedvibe.studio](https://bedvibe.studio/)
-
-MIT license.
+# trainproof
+
+[![PyPI](https://img.shields.io/pypi/v/trainproof)](https://pypi.org/project/trainproof/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+**A deterministic linter for ML training runs.** Point it at your dataset, your
+tokenizer, or your first-epoch logs — it returns a PASS / WARN / FAIL verdict
+with named findings and cited evidence, before you burn days of GPU time on a
+run that was doomed at step 50.
+
+No ML judging ML. No invented "confidence 97%". Every rule is a deterministic
+threshold in [one auditable module](src/trainproof/rules.py), and every finding
+cites the numbers that triggered it.
+
+**What it is:** the reliability layer between your training code and the GPU
+bill — checks that run *before* training (is this run safe to start?), *during*
+training (should it keep going?), and *after* (is it reproducible; did it match
+a known-good baseline?).
+
+**The one rule it never breaks:** trainproof does not infer causes, does not
+invent confidence scores, and does not guess. It reports deterministic findings
+backed by evidence, or it stays silent. Every feature earns its place by
+answering a single question — *if it were gone, would someone lose GPU hours?*
+
+```bash
+pip install trainproof
+```
+
+## See a verdict in 60 seconds
+
+This repo ships the real logs of five QLoRA fine-tuning runs (Qwen2.5-3B,
+RTX 5080 — see the gallery below). Judge one right now:
+
+```bash
+trainproof epoch examples/gallery/lr_hot/trainer_state.json --format hf
+```
+
+```text
+[FAIL] Critical checks failed:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 7.492 vs Min loss 1.398
+  [WARN] Gradient norm spikes detected.
+         Evidence: Max gn 2649.75 > 10.0x median (0.55)
+```
+
+## Why this exists
+
+The author lost a real 11-hour fine-tune to a failure nothing warned about.
+Pointed retroactively at that run's 1MB Coqui Trainer log (2,501 logged steps),
+trainproof's verdict: **FAIL — diverging**. The loss reached its minimum at 82%
+of the run and ended 1.9x above it. Translation: the final two hours of GPU
+time made the model measurably worse, and the checkpoint worth keeping had
+already existed for hours. No tool in the stack said a word.
+
+## The fault-injection gallery
+
+To validate the rules, the same QLoRA fine-tune (Qwen2.5-3B-Instruct, 4-bit,
+LoRA r=16, 300 steps on Alpaca-cleaned) was run five times — once healthy, four
+times with exactly one knob deliberately broken. Real runs, real logs, all
+shipped in [`examples/gallery/`](examples/gallery/):
+
+| Run | Sabotage | Verdict | Key evidence |
+|---|---|---|---|
+| `healthy` | none | **PASS** | loss 1.52 → 0.94, stable gradients |
+| `lr_hot` | LR x100 (2e-2) | **FAIL** | diverging: end 7.49 vs min 1.40; grad spike 2650 vs median 0.55 |
+| `lr_zero` | LR = 0 | **FAIL** | dead run: first-5 median 1.52 vs last-5 1.49 (<5% improvement); lr=0 on 100% of steps |
+| `fp16_nan` | fp16 + hot LR, no clipping | **FAIL** | diverging: end 7.21 vs min 1.09 (grad scaling absorbed the intended NaN — the run diverged instead; reported as observed) |
+| `bad_labels` | labels shuffled per-sequence | **WARN only** (single-run) — caught by `trainproof compare` (v0.3) | grad spike 23.3 vs median 1.09 |
+
+### The honest finding: loss curves cannot see corrupted data
+
+The `bad_labels` run — whose shuffled labels make real learning impossible —
+*reduced its loss by 62%* (18.9 → 5.75). The model was genuinely learning: not
+the task, but the marginal token statistics of the garbage. From its own loss
+curve, that is indistinguishable from healthy training (neural networks
+famously fit random labels). **No single-run, loss-only rule can catch this
+class of failure** — its real signature is *relative*: a loss floor ~6x higher
+than a known-good run of the same task (5.59 vs 0.94).
+
+That finding produced v0.3: `trainproof compare <run> <baseline>` —
+deterministic ratio rules against the healthy baseline you already have —
+which catches `bad_labels` at a 6x loss-floor ratio, in 3 seeds out of 3.
+The full study was repeated with three random seeds (15 runs):
+see [EVIDENCE_MATRIX.md](EVIDENCE_MATRIX.md) for every verdict, including the
+honest miss (compare alone overlooks one lr_zero seed — the single-run
+zero-LR fatality rule owns that case; the two commands cover each other's
+blind spots). The gallery also improved the tool itself twice: the dead-run
+rule and the total-zero-LR fatality rule both exist because runs escaped
+earlier rule versions. See [ROADMAP.md](ROADMAP.md).
+
+## The commands
+
+```bash
+# 1. Dataset preflight (speech/TTS pack): audio integrity, transcript quality,
+#    duplicates, text-vs-audio duration mismatches
+trainproof data /path/to/dataset_or_manifest.jsonl
+
+# 2. Tokenizer preflight: vocabulary coverage, OOV rate, sequence blowouts,
+#    suspicious splits on numbers/dates
+trainproof tokenizer my_tokenizer.model transcripts.txt
+
+# 3. Training-run verdict: NaN/divergence/dead-run detection, gradient spikes,
+#    LR sanity, throughput — from log files, any framework
+trainproof epoch logs/run.jsonl            # exit code 1 on FAIL: CI-ready
+
+# 4. Compare against a baseline
+#    Catch relative pathologies like the `bad_labels` run that evade single-run rules.
+trainproof compare examples/gallery/bad_labels/trainer_state.json examples/gallery/healthy/trainer_state.json
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] loss floor ratio exceeded limit
+         Evidence: Run floor 5.592 vs Baseline floor 0.937 (ratio 6.0x > 2.0)
+  [FAIL] end loss ratio exceeded limit
+         Evidence: Run end 5.750 vs Baseline end 1.082 (ratio 5.3x > 2.0)
+========================================
+```
+
+Each command prints the verdict, writes a self-contained HTML report, and sets
+the process exit code — so it works as a CI gate out of the box.
+
+## Live guardian (v0.4)
+
+Don't wait for the post-mortem — catch a doomed run *while it is still burning
+GPU*. Add one line to a HuggingFace `Trainer`:
+
+```python
+from transformers import Trainer
+from trainproof.integrations.hf import TrainproofCallback
+
+trainer = Trainer(
+    ...,
+    callbacks=[TrainproofCallback(policy="stop_on_fail")],  # or policy="warn"
+)
+```
+
+Run against a real diverging QLoRA fine-tune (learning rate 100x too high), the
+guardian aborts it 20 steps into a 300-step schedule — on its own:
+
+```text
+{'loss': '1.784', 'grad_norm': '9.634',  'learning_rate': '0.007'}
+{'loss': '4.282', 'grad_norm': '53.76',  'learning_rate': '0.009'}
+{'loss': '10.6',  'grad_norm': '13.34',  'learning_rate': '0.011'}
+{'loss': '31.67', 'grad_norm': '76.67',  'learning_rate': '0.013'}
+...
+TRAINPROOF ABORT - stopping training at step 20. Findings:
+  [FAIL] Loss curve is diverging.
+         Evidence: End loss 22.952 vs Min loss 1.358
+  [FAIL] Loss never improved over the run (dead run).
+         Evidence: median of first 5 losses 1.502 vs last 5 22.952
+
+  scheduled steps : 300
+  stopped at step : 20
+  run saved       : 93% of the scheduled steps never ran
+```
+
+On a two-day pre-training run, that fraction is days of GPU time. Or watch a
+growing log file from outside the process (CI-friendly, exits non-zero on FAIL):
+
+```bash
+trainproof watch logs/run.jsonl --interval 10 --until-fail
+# [21:37:44] warming up (5 records)
+# [21:37:44] n_records=15 verdict=PASS findings=1
+```
+
+**The default is safe.** `policy="warn"` (the default) only observes and reports
+— it never interrupts your run, so you can leave it on even for experiments you
+expect to fail. Aborting is strictly opt-in via `policy="stop_on_fail"`, the one
+mode that takes an irreversible action. trainproof does not make that decision
+for you unless you ask.
+
+The guardian applies the same deterministic rules as `trainproof epoch`, so it
+inherits their documented single-run limitations.
+
+## Pre-flight (v0.5): stop a run before it starts
+
+The guardian saves most of a doomed run; preflight saves 100% because it never starts. Catch broken datasets and tokenizer misconfigurations instantly.
+
+```bash
+trainproof preflight data/dataset.jsonl --tokenizer mistralai/Mistral-7B-v0.1 --max-len 4096
+```
+
+```text
+========================================
+TRAINPROOF VERDICT
+========================================
+[FAIL] Critical checks failed:
+  [FAIL] Empty or whitespace-only text found.
+         Evidence: 1 records (indices [1]...)
+========================================
+```
+
+*Checks: malformed JSONL, empty text, exact duplicate text, tokenizer structural checks (EOS/PAD/BOS), and context length overflows.*
+
+## Supported log formats
+
+- HuggingFace Trainer (`trainer_state.json`)
+- Coqui Trainer text logs (ANSI-colored `trainer_0_log.txt`)
+- Generic JSONL / CSV (columns: step, loss, lr, grad_norm, time — all optional)
+
+Auto-detected; override with `--format hf|coqui|jsonl|csv`. TensorBoard event
+files are planned (Lightning console captures are TTY dumps, not logs, and
+will not be supported).
+
+## Philosophy
+
+1. **Deterministic.** A rule fires or it doesn't. Thresholds live in one
+   module, commented, tunable.
+2. **Evidence-cited.** Every finding names the steps and values that triggered
+   it.
+3. **Honest about limits.** What the tool cannot detect is documented in the
+   README, not discovered by the user in production.
+
+## Family
+
+trainproof judges training runs. Its sibling [ttsproof](https://github.com/Mormolykos/ttsproof)
+judges TTS model *outputs* (structural audio checks, equivalence-aware WER/CER,
+published method with DOI) — and trainproof builds on it for the speech
+dataset checks.
+
+## Author
+
+Panagiotis (Panos) Gkilis — [portfolio](https://tts.bedvibe.studio/portfolio/) ·
+[bedvibe.studio](https://bedvibe.studio/)
+
+MIT license.

{trainproof-0.3.0 → trainproof-0.5.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "trainproof"
-version = "0.3.0"
+version = "0.5.0"
 description = "A deterministic linter for ML training runs: dataset, tokenizer, and epoch logs."
 authors = [{name = "Panagiotis (Panos) Gkilis", email = "bedvibe@bedvibe.studio"}]
 readme = "README.md"

{trainproof-0.3.0 → trainproof-0.5.0}/src/trainproof/cli.py

@@ -5,6 +5,7 @@ from .speech.data import check_data
 from .speech.tokenizer import check_tokenizer
 from .epoch import check_epoch
 from .compare import check_compare
+from .watch import watch_loop
 from .report import print_verdict_console, write_html_report
 
 def main():
@@ -31,6 +32,20 @@ def main():
     compare_parser.add_argument("baseline", type=str, help="Path to baseline log file")
     compare_parser.add_argument("--format", choices=["auto", "hf", "coqui", "jsonl", "csv"], default="auto", help="Log format override")
 
+    # Watch Command
+    watch_parser = subparsers.add_parser("watch", help="Live guardian: poll a training log.")
+    watch_parser.add_argument("logfile", type=str, help="Path to run log file")
+    watch_parser.add_argument("--interval", type=int, default=10, help="Polling interval in seconds")
+    watch_parser.add_argument("--format", choices=["auto", "hf", "coqui", "jsonl", "csv"], default="auto", help="Log format override")
+    watch_parser.add_argument("--until-fail", action="store_true", help="Exit with code 1 as soon as verdict becomes FAIL")
+
+    # Preflight Command
+    preflight_parser = subparsers.add_parser("preflight", help="Pre-flight linter for LLM fine-tuning datasets.")
+    preflight_parser.add_argument("dataset", type=str, help="Path to JSONL dataset")
+    preflight_parser.add_argument("--tokenizer", type=str, help="Tokenizer name or path")
+    preflight_parser.add_argument("--max-len", type=int, help="Max context length")
+    preflight_parser.add_argument("--text-field", type=str, help="Text field to extract")
+
     args = parser.parse_args()
     
     report_dict = {}
@@ -51,6 +66,26 @@ def main():
         except Exception as e:
             print(f"Error checking compare: {e}")
             sys.exit(1)
+    elif args.command == "watch":
+        watch_loop(args.logfile, interval=args.interval, fmt=args.format, until_fail=args.until_fail)
+        sys.exit(0)
+    elif args.command == "preflight":
+        from .preflight import preflight
+        tokenizer = None
+        if args.tokenizer:
+            try:
+                from transformers import AutoTokenizer
+                tokenizer = AutoTokenizer.from_pretrained(args.tokenizer)
+            except ImportError:
+                report_dict = {
+                    "verdict": "FAIL",
+                    "findings": [{"id": "transformers_missing", "level": "FAIL", "message": "pip install transformers is required to load tokenizer", "evidence": ""}]
+                }
+                print_verdict_console(report_dict["verdict"], report_dict["findings"])
+                sys.exit(1)
+        
+        result = preflight(args.dataset, tokenizer=tokenizer, max_len=args.max_len, text_field=args.text_field)
+        report_dict = {"verdict": result.verdict, "findings": result.findings}
 
     print_verdict_console(report_dict.get("verdict", "FAIL"), report_dict.get("findings", []))
     

{trainproof-0.3.0 → trainproof-0.5.0}/src/trainproof/epoch.py

@@ -4,13 +4,12 @@ from typing import Any
 from . import rules
 from .adapters import parse_log_with_format
 
-def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
-    records = parse_log_with_format(log_path, fmt)
+def check_records(records: list[dict]) -> dict[str, Any]:
     findings = []
     verdict = "PASS"
     
     if not records:
-        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "No valid log records found.", "evidence": str(log_path)}]}
+        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "No valid log records found.", "evidence": ""}]}
 
     # Find relevant keys
     def get_val(row, aliases):
@@ -112,3 +111,9 @@ def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
         findings.append({"level": "PASS", "message": "Loss curve shows healthy shape, grad norms are stable.", "evidence": f"{len(valid_losses)} steps analyzed."})
 
     return {"verdict": verdict, "findings": findings}
+
+def check_epoch(log_path: str | Path, fmt: str = "auto") -> dict[str, Any]:
+    records = parse_log_with_format(log_path, fmt)
+    if not records:
+        return {"verdict": "FAIL", "findings": [{"level": "FAIL", "message": "No valid log records found.", "evidence": str(log_path)}]}
+    return check_records(records)

trainproof-0.5.0/src/trainproof/integrations/__init__.py

@@ -0,0 +1,3 @@
+from .hf import TrainproofCallback
+
+__all__ = ["TrainproofCallback"]