soup-cli 0.71.30__tar.gz → 0.71.32__tar.gz

{soup_cli-0.71.30 → soup_cli-0.71.32}/.gitignore

@@ -40,9 +40,12 @@ report.xml
 .env
 *.key
 
-# Internal plan + Claude Code local dev instructions (not for repo)
+# Internal plan + local dev instructions (not for repo)
 .claude/
 .coverage
 
+# Internal brainstorming plans + design specs (local-only, not for repo)
+docs/superpowers/
+
 # Local scratch / temp scripts at repo root (e.g. _smoke.py, _scratch.py) — never committed
 /_*.py

{soup_cli-0.71.30 → soup_cli-0.71.32}/CHANGELOG.md

@@ -12,6 +12,68 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.32] - 2026-07-07
+
+### Added
+- **ASR fine-tuning (`task='asr'`, Whisper)** — fine-tune Whisper on your accent
+  or domain, locally. whisper-tiny (39M) / base (74M) train on a 4 GB GPU.
+  - New `AsrTrainerWrapper` (HF `Seq2SeqTrainer` + `WhisperProcessor`); data rows
+    are `{"audio": <path>, "text": <transcript>}` under the new `data.format='asr'`.
+    Audio decodes via the hardened `load_audio_mono` (16 kHz mono, soundfile
+    pre-probe + `O_NOFOLLOW` + symlink/size guards); transcripts become decoder
+    labels (pad → −100, decoder-start token stripped).
+  - Optional LoRA on q/v attention projections via `training.asr_lora: true`
+    (default full fine-tune); `training.asr_language` / `training.asr_task`
+    (`transcribe`|`translate`) set the decoder prefix and are persisted to an
+    `asr_generation.json` sidecar so inference restores them.
+  - **`soup infer --task asr`** — transcribe an `{"audio": path[, "text": ref]}`
+    JSONL; reports per-row and corpus WER/CER when references are present. Loads a
+    full model or a PEFT/LoRA adapter dir. Flags: `--asr-language`, `--asr-task`,
+    `--audio-dir` (audio paths are cwd/dir-contained; UNC/traversal rejected).
+  - New pure-python `utils/asr_metrics.py` — WER / CER / `word_accuracy`
+    (= 1 − WER, for a higher-is-better ship metric leg) / `corpus_wer`, with a
+    light Whisper-style text normalizer (no new dependency).
+  - Recipes: `whisper-tiny-asr`, `whisper-base-asr` (live-trainable),
+    `whisper-large-v3-asr` (parse-only, needs a larger GPU), plus
+    `smolvlm-256m-sft` (vision). Catalog 138 → 142.
+
+### Fixed
+- `model_size_from_name` now knows Whisper checkpoint sizes (tiny…large), so the
+  hardware-fit gate no longer mistakes a 39M whisper-tiny for the 7B default and
+  blocks ASR training on consumer GPUs.
+
+## [0.71.31] - 2026-07-06
+
+### Added
+- **Judge-in-the-loop suite** — put an LLM judge in the loop across the workflow:
+  - **`task='online_dpo'`** — Online DPO training (wraps TRL `OnlineDPOTrainer`):
+    the model generates two completions per prompt on-policy each step and a
+    *judge* (a pairwise LLM judge over the existing ollama/openai-compatible
+    backend) OR a `reward_model` picks the winner. Config:
+    `training.online_dpo_judge: "ollama://model"` (or set `reward_model` —
+    exactly one), `online_dpo_loss_type: sigmoid|ipo`, `online_dpo_max_new_tokens`;
+    `beta` reuses `dpo_beta`. Transformers + text only. Recipe:
+    `online-dpo-smollm2-135m`. Adapts to the installed TRL: on trl 0.19.x the
+    judge is a swap-debiased *pairwise* comparison; on trl 1.x (which removed
+    pairwise judges) the same `JudgeEvaluator` is used as a *pointwise*
+    reward function — a documented per-version behaviour difference.
+  - **`soup data best-of-n`** — Best-of-N rejection sampling (BOND-lite): sample
+    N completions from `--base` locally, a `--judge` scores each pointwise, and
+    the winner is written as an SFT chat row (with provenance). `--emit-pairs`
+    also writes winner-vs-loser DPO pairs.
+  - **`soup data evolve`** — Evol-Instruct instruction evolution (WizardLM depth
+    / breadth) over an ollama/vllm provider, completing the synthetic-data suite
+    (Magpie / Forge / Persona / evolve).
+  - **`soup ship --task-mode pairwise`** — a true pairwise judge win-rate as the
+    ship leg-1 task-win (base = 0.5 coin-flip, tuned = its win-rate; swap-debiased),
+    fusing with the catastrophic-forgetting guard into one SHIP / DON'T-SHIP verdict.
+
+### Security
+- `soup data best-of-n` / `evolve` write outputs via atomic `mkstemp` + `os.replace`
+  (re-validated cwd containment), closing the TOCTOU symlink-swap window between the
+  containment check and the write. All judge/provider URLs are SSRF-validated; model
+  loads probe `trust_remote_code`.
+
 ## [0.71.30] - 2026-07-05
 
 ### Added

{soup_cli-0.71.30 → soup_cli-0.71.32}/CONTRIBUTING.md

@@ -110,17 +110,17 @@ src/soup_cli/
   experiment/         - SQLite experiment tracking
   eval/               - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
   migrate/            - Config migration (LLaMA-Factory, Axolotl, Unsloth)
-  recipes/            - Ready-made configs for popular models (137 recipes)
+  recipes/            - Ready-made configs for popular models (142 recipes)
   autopilot/          - Zero-config decision engine (v0.25.0)
   registry/           - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0)
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
   data/traces/        - Trace-to-Preference harvester (v0.26.0)
   data/collators.py   - CrossDocCollator for sample packing (v0.33.0)
-  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink
+  utils/              - GPU, errors, MoE, GaLore, QAT, Unsloth, vLLM, SGLang, Liger, FlashAttn, FSDP, Ring Attention, long-context, quality, curriculum, freeze, dataset-registry, mlx, peft_builder, paths, topology, launcher, mii, pipeline, cut_ce, fp8, gradient_ckpt, kernel_picker, cross_doc_attn, activation_offload, hf, spec_pairing, structured_output, metrics, tracing, auto_quant, lr_finder, grad_accum, mixed_precision, warmup, spike_recovery, convergence, v028_features, multipack_sampler, multipack, neat_packing, jinja_analyzer, quant_menu, relora, peft_patches, peft_wiring, dpo_variants, optimizer_zoo, lr_groups, loftq_init, block_expansion, tts, classifier, distill, bitnet, ebft_gdpo, moe_quant, reasoning_effort, gguf_quant, kv_cache, advanced_precision, save_formats, deploy_measure, advise, advise_history, adapter_diff, adapter_merge, blame, adapter_branch, unlearning, unlearning_eval, knowledge_edit, edit_governor, edit_diff, ra_dit, steering, citation_faithful, grace_codebook, ingest_sources, prune_prompt, active_sampler, ab_test, drift_alarm, tunability, terraform_plan, env_lock, hardware_fit, completions, license_advisor, behavior_battery, capability_suite, checklist_dsl, irt, sae_diff, sleeper_probe, interference, probe_pack, cmaes_merge, vector_bank, mole_routing, adapter_pr, soup_lock, adapter_bisect, prompt_compile, prompt_distill, compile_tools, apple_adapter, local_rl, build_dag, expectations, magpie, persona_hub, brain_rot, reward_hacking, uld, minillm, rl_checkpoint, iterative_dpo, echo_trap, mod, local_rl_scheduler, spectrum_scan, ship_verdict, reward_hack_control, data_doctor, data_lint, shrink, best_of_n, evolve
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (307 files, 15334 tests)
+tests/                - Test suite (309 files, 15663 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,10 +150,9 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (306 files)
+### Test Files (307 files)
 
-> A representative sample of the suite below. The full table lives in
-> [`.claude/CLAUDE.md`](.claude/CLAUDE.md); run `pytest tests/ -v` for the complete list.
+> A representative sample of the suite below. Run `pytest tests/ -v` for the complete list.
 
 
 | File | Covers |

{soup_cli-0.71.30 → soup_cli-0.71.32}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.30
+Version: 0.71.32
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,27 +193,33 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.30 — Process-supervised RL: PRM-guided GRPO.** Use a trained Process Reward Model to score each *reasoning step* of a GRPO completion — the o1-era training signal — plus bundled toy environments so the openenv rollout path runs out-of-the-box.
-
-- **PRM as the GRPO reward.** Point `training.prm_reward` at a PRM you trained with
-  `soup train task=prm`; it splits each completion into steps, scores every step with the PRM's
-  reward head, and folds them (`min` / `prod` / `last`) into one reward that GRPO optimises —
-  replacing `reward_fn`. It rides the reward-shaping + reward-hack-mitigation seam, so the
-  v0.71.26 controller still watches it.
-- **Bundled rollout environments.** `soup_cli.envs.calculator` / `retrieval_qa` / `guess_number`
-  each expose a `rollout(prompts)` entry point; wire one with `rollout_backend=openenv` +
-  `rollout_func=soup_cli.envs.calculator:rollout` (three ready-made `grpo-env-*` recipes ship it).
-- **Producer fixes too.** `soup train task=prm` now saves its tokenizer and no longer crashes on
-  its own summary — a PRM checkpoint is loadable standalone.
-- **Proof-of-mechanism, honestly.** Live-validated on SmolLM2-135M (CPU): the PRM reward scores
-  good completions above bad and drives GRPO's advantages (`rewards/prm_reward` logged). Tiny
-  model + synthetic PRM — not a production reward-model claim (scale help wanted: #286).
+**v0.71.32 — ASR fine-tuning (Whisper).** Fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) and base (74M) train on a 4 GB GPU — a new modality that actually runs on a laptop card.
+
+- **`task='asr'`.** A new `AsrTrainerWrapper` (HF `Seq2SeqTrainer` + `WhisperProcessor`). Data rows
+  are `{"audio": <path>, "text": <transcript>}` under `data.format='asr'`; audio decodes to 16 kHz
+  mono through the hardened loader (soundfile pre-probe + symlink/size guards). Opt-in LoRA on q/v
+  via `training.asr_lora: true` (default full fine-tune); `training.asr_language` / `asr_task`
+  (`transcribe`|`translate`) set — and persist — the decoder prefix for inference.
+- **`soup infer --task asr`.** Transcribe an `{"audio"[, "text"]}` JSONL and get per-row + corpus
+  **WER / CER** when references are present. Loads a full model or a LoRA adapter dir; audio paths
+  are containment-checked (traversal / UNC rejected).
+- **Pure-python metrics.** New `soup_cli.utils.asr_metrics` (WER / CER / `word_accuracy` /
+  `corpus_wer`) — no new dependency, reusable as a higher-is-better ship metric leg.
+- **Recipes.** `whisper-tiny-asr` / `whisper-base-asr` (live-trainable), `whisper-large-v3-asr`
+  (parse-only). Catalog 138 → 142.
+- **Proof-of-mechanism, honestly.** Live-validated on an RTX 3050: a memorization task drove
+  WER 1.000 → 0.000 (loss 7.2 → 0.0005 at 0.4/4 GB). Real accent/domain gains need real audio;
+  WER/CER use a light normalizer, not the full Whisper text normalizer.
 
 ```yaml
-task: grpo
+base: openai/whisper-tiny
+task: asr
+data:
+  format: asr            # rows: {"audio": "clip.wav", "text": "hello world"}
+  audio_dir: ./data/audio
 training:
-  prm_reward: ./my-prm            # a `soup train task=prm` checkpoint dir
-  prm_aggregate: min              # weakest-link (default) | prod | last
+  asr_language: en
+  asr_lora: true         # optional; default = full fine-tune
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.30 → soup_cli-0.71.32}/README.md

@@ -49,27 +49,33 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.30 — Process-supervised RL: PRM-guided GRPO.** Use a trained Process Reward Model to score each *reasoning step* of a GRPO completion — the o1-era training signal — plus bundled toy environments so the openenv rollout path runs out-of-the-box.
-
-- **PRM as the GRPO reward.** Point `training.prm_reward` at a PRM you trained with
-  `soup train task=prm`; it splits each completion into steps, scores every step with the PRM's
-  reward head, and folds them (`min` / `prod` / `last`) into one reward that GRPO optimises —
-  replacing `reward_fn`. It rides the reward-shaping + reward-hack-mitigation seam, so the
-  v0.71.26 controller still watches it.
-- **Bundled rollout environments.** `soup_cli.envs.calculator` / `retrieval_qa` / `guess_number`
-  each expose a `rollout(prompts)` entry point; wire one with `rollout_backend=openenv` +
-  `rollout_func=soup_cli.envs.calculator:rollout` (three ready-made `grpo-env-*` recipes ship it).
-- **Producer fixes too.** `soup train task=prm` now saves its tokenizer and no longer crashes on
-  its own summary — a PRM checkpoint is loadable standalone.
-- **Proof-of-mechanism, honestly.** Live-validated on SmolLM2-135M (CPU): the PRM reward scores
-  good completions above bad and drives GRPO's advantages (`rewards/prm_reward` logged). Tiny
-  model + synthetic PRM — not a production reward-model claim (scale help wanted: #286).
+**v0.71.32 — ASR fine-tuning (Whisper).** Fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) and base (74M) train on a 4 GB GPU — a new modality that actually runs on a laptop card.
+
+- **`task='asr'`.** A new `AsrTrainerWrapper` (HF `Seq2SeqTrainer` + `WhisperProcessor`). Data rows
+  are `{"audio": <path>, "text": <transcript>}` under `data.format='asr'`; audio decodes to 16 kHz
+  mono through the hardened loader (soundfile pre-probe + symlink/size guards). Opt-in LoRA on q/v
+  via `training.asr_lora: true` (default full fine-tune); `training.asr_language` / `asr_task`
+  (`transcribe`|`translate`) set — and persist — the decoder prefix for inference.
+- **`soup infer --task asr`.** Transcribe an `{"audio"[, "text"]}` JSONL and get per-row + corpus
+  **WER / CER** when references are present. Loads a full model or a LoRA adapter dir; audio paths
+  are containment-checked (traversal / UNC rejected).
+- **Pure-python metrics.** New `soup_cli.utils.asr_metrics` (WER / CER / `word_accuracy` /
+  `corpus_wer`) — no new dependency, reusable as a higher-is-better ship metric leg.
+- **Recipes.** `whisper-tiny-asr` / `whisper-base-asr` (live-trainable), `whisper-large-v3-asr`
+  (parse-only). Catalog 138 → 142.
+- **Proof-of-mechanism, honestly.** Live-validated on an RTX 3050: a memorization task drove
+  WER 1.000 → 0.000 (loss 7.2 → 0.0005 at 0.4/4 GB). Real accent/domain gains need real audio;
+  WER/CER use a light normalizer, not the full Whisper text normalizer.
 
 ```yaml
-task: grpo
+base: openai/whisper-tiny
+task: asr
+data:
+  format: asr            # rows: {"audio": "clip.wav", "text": "hello world"}
+  audio_dir: ./data/audio
 training:
-  prm_reward: ./my-prm            # a `soup train task=prm` checkpoint dir
-  prm_aggregate: min              # weakest-link (default) | prod | last
+  asr_language: en
+  asr_lora: true         # optional; default = full fine-tune
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.30 → soup_cli-0.71.32}/docs/commands.md

@@ -19,6 +19,7 @@ soup train --config soup.yaml --push-as user/repo --hf-resume  Resume from lates
 soup train --config soup.yaml --find-lr        LR range finder: write recommended LR JSON
 soup train --config soup.yaml --cloud modal --gpu a100  Render a Modal.com app for serverless GPU training (plan-only; --cloud-submit submits live)
 soup infer --model ./output --input p.jsonl   Batch inference
+soup infer --task asr --model <whisper|adapter> --input a.jsonl --output o.jsonl [--audio-dir d --asr-language en --asr-task transcribe|translate]  Whisper transcription + WER/CER
 soup chat --model ./output                    Interactive chat
 soup push --model ./output --repo user/name   Upload to HuggingFace
 soup push --model ./output --repo user/name --collection user/coll-abc123  Add to HF Collection
@@ -133,7 +134,7 @@ soup migrate --from llamafactory config.yaml  Import config from LLaMA-Factory
 soup migrate --from axolotl config.yml        Import config from Axolotl
 soup migrate --from unsloth notebook.ipynb    Import config from Unsloth notebook
 soup migrate --from llamafactory c.yaml --dry-run  Preview without writing
-soup recipes list                             List all 137 ready-made recipes
+soup recipes list                             List all 138 ready-made recipes
 soup recipes show llama3.1-8b-sft            Print recipe YAML
 soup recipes use llama3.1-8b-sft             Copy recipe to soup.yaml
 soup recipes search "reasoning"              Search by keyword/task/size
@@ -155,6 +156,7 @@ soup why [run_id]                             Explain training anomalies (heuris
 soup ship --base <m> --adapter <lora> --task-eval t.jsonl  SHIP / DON'T-SHIP verdict: task win AND no forgetting (exit 0=SHIP / 2=DON'T / 1=error) (v0.71.25)
 soup ship --evidence ev.json [--output v.json]  Decide offline from pre-computed scores (no model load)
 soup ship ... --task-mode judge_score --judge-model ollama://llama3.1  Leg-1 via LLM-as-a-judge
+soup ship ... --task-mode pairwise --judge-model ollama://llama3.1  Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31)
 soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 + recorded base scores
 soup mcp serve                                MCP server over stdio (drive Soup from Claude Code / Cursor / Cline; requires [mcp] extra) (v0.71.28)
 soup mcp serve --allow-mutating               Also expose plan-only train_start / export tools (never execute) (v0.71.28)
@@ -250,6 +252,8 @@ soup local-rl train --db <path> --model <id> [--scheduler-dir <dir>] [--hour H]
 soup build <manifest.yaml> [--dry-run] [--output-dir <dir>]  dbt-for-SFT DAG: validate + plan + live materialise (v0.69.0; live v0.71.6)
 soup expect <data.jsonl> <suite.yaml>         Expectations suite: PII / token-length / refusal / judge (v0.69.0)
 soup data gen-magpie --base <m> --provider ollama|vllm --target N --output <jsonl> [--base-url <url>] [--quality-filter]  Magpie synthetic generator — live (v0.69.0; live v0.71.6)
+soup data best-of-n --base <m> --prompts <jsonl> --n 8 --judge <url> -o <sft.jsonl> [--emit-pairs <dpo.jsonl>]  Best-of-N rejection sampling: sample N locally, judge picks winner -> SFT (+ DPO) rows (v0.71.31)
+soup data evolve --input <seeds.jsonl> --provider ollama|vllm --model <m> --strategy depth|breadth --rounds N -o <jsonl>  Evol-Instruct (WizardLM) instruction evolution (v0.71.31)
 soup data persona-mix --prompts <jsonl> --n N --output <jsonl>  Persona-Hub diversity sampler (v0.69.0)
 soup data brain-rot <data.jsonl> [--strict]   Brain-rot detector — arXiv 2510.13928 (v0.69.0)
 soup iterative-dpo --base-model <m> --reward-model <rm> --prompts <p.jsonl> --output-dir <out> --rounds N --pairs-per-round N [--plan-only]  Iterative DPO loop driver — LIVE sample→score→pair→train (v0.70.0; live v0.71.11)

{soup_cli-0.71.30 → soup_cli-0.71.32}/docs/data.md

@@ -62,6 +62,15 @@ soup data persona-mix --prompts prompts.jsonl --n 500 --output mixed.jsonl
 
 # Brain-rot detector (arXiv 2510.13928) — refuses to train on excessive slop
 soup data brain-rot data.jsonl --strict --max-major-fraction 0.10
+
+# Best-of-N rejection sampling (v0.71.31) — sample N locally, a judge picks the winner
+soup data best-of-n --base HuggingFaceTB/SmolLM2-135M-Instruct \
+    --prompts prompts.jsonl --n 8 --judge ollama://llama3.1 \
+    -o best_of_n.jsonl --emit-pairs pairs.jsonl
+
+# Evol-Instruct (WizardLM depth/breadth, v0.71.31) — grow instruction diversity
+soup data evolve --input seeds.jsonl --provider ollama --model llama3.1 \
+    --strategy depth --rounds 2 -o evolved.jsonl
 ```
 
 Every command applies the project-wide TOCTOU policy (`os.lstat + S_ISLNK` symlink rejection before any open) and cwd containment via the shared `paths.enforce_under_cwd_and_no_symlink` helper. All five are LIVE: `soup build` materialises with five built-in transforms (`identity` / `drop_empty` / `lowercase` / `strip` / `dedup_exact`) and SQLite-tracked incremental re-transform (v0.71.6); `soup data gen-magpie` harvests via raw completion against `--provider ollama|vllm` (loopback-only; `anthropic` rejected — no raw-completion endpoint, v0.71.6).
@@ -343,6 +352,13 @@ Or use `.txt` files directly (one document per line).
 {"audio": "recording.wav", "messages": [{"role": "user", "content": "Transcribe."}, {"role": "assistant", "content": "Hello world."}]}
 ```
 
+**ASR (Whisper transcription — `data.format: asr`, v0.71.32):**
+```json
+{"audio": "clip.wav", "text": "hello world"}
+```
+Audio paths resolve under `data.audio_dir` (containment-checked). Used by
+`task: asr` and `soup infer --task asr`. See [Training → ASR](training.md).
+
 **PRM (process reward, stepwise-supervised):**
 ```json
 {"prompt": "Solve 2+2", "completions": ["First, add", "Result is 4"], "labels": [true, true]}

{soup_cli-0.71.30 → soup_cli-0.71.32}/docs/evaluation.md

@@ -330,6 +330,11 @@ soup ship --base HuggingFaceTB/SmolLM2-135M-Instruct --adapter ./out \
 soup ship --base <m> --adapter ./out --task-eval tasks.jsonl \
   --task-mode judge_score --judge-model ollama://llama3.1
 
+# Leg-1 via a true pairwise judge win-rate (v0.71.31 #284) — the judge picks
+# base vs tuned per prompt (swap-debiased); base = 0.5 coin-flip, won <=> winrate > 0.5
+soup ship --base <m> --adapter ./out --task-eval tasks.jsonl \
+  --task-mode pairwise --judge-model ollama://llama3.1
+
 # Leg-2 via lm-eval benchmarks, base scores supplied by --baseline
 soup ship --base <m> --tuned ./out --task-eval tasks.jsonl \
   --general-suite mmlu,hellaswag --baseline registry://abc123

{soup_cli-0.71.30 → soup_cli-0.71.32}/docs/models.md

@@ -40,6 +40,17 @@ Soup works with **any** of the **340,000+** text-generation models on [HuggingFa
 | Qwen2-VL-7B-Instruct | 7B | LLaVA, ShareGPT4V |
 | Pixtral-12B-2409 | 12B | LLaVA, ShareGPT4V |
 
+### ASR Models (`task: asr`, Whisper — v0.71.32)
+
+| Recipe | Base | Size | Status |
+|---|---|---|---|
+| `whisper-tiny-asr` | openai/whisper-tiny | 39M | Live on 4 GB |
+| `whisper-base-asr` | openai/whisper-base | 74M | Live on 4 GB |
+| `whisper-large-v3-asr` | openai/whisper-large-v3 | 1.5B | Parse-only (larger GPU) |
+
+Rows are `{"audio": <path>, "text": <transcript>}` with `data.format: asr`. See
+[Training → ASR fine-tuning](training.md).
+
 ### Quick Size Guide
 
 | VRAM | Max Model (QLoRA 4-bit) | Example |

{soup_cli-0.71.30 → soup_cli-0.71.32}/docs/serving-and-export.md

@@ -418,7 +418,7 @@ soup ui
 
 **Pages:**
 - **Dashboard** — view all experiment runs, loss charts, system info, multi-run comparison
-- **New Training** — create configs from templates or 134 ready-made recipes, validate, start training with live SSE log streaming and progress bar
+- **New Training** — create configs from templates or 138 ready-made recipes, validate, start training with live SSE log streaming and progress bar
 - **Data Explorer** — browse and inspect datasets (JSONL, JSON, CSV, Parquet)
 - **Model Chat** — chat with streaming responses, configurable temperature/top_p/max_tokens, system prompt, adapter selection, markdown rendering, chat export
 
@@ -427,7 +427,7 @@ soup ui
 - **Enhanced Metrics** — 2x2 chart grid (loss, LR, grad_norm, throughput) + GPU memory chart, eval results table
 - **Multi-Run Compare** — overlay loss curves from up to 5 runs side-by-side
 - **Chat Upgrade** — SSE streaming via proxy, typing indicator, cancel button, markdown renderer (bold, italic, code blocks), chat export as JSON
-- **Config Builder** — recipe dropdown (134 recipes), config schema API for dynamic form generation
+- **Config Builder** — recipe dropdown (138 recipes), config schema API for dynamic form generation
 
 **Security:** The Web UI generates a random auth token at startup (printed to console). All mutating endpoints (start/stop training, delete runs, inspect data, validate config) require `Authorization: Bearer <token>` header. CORS is restricted to the served origin. Data inspection is sandboxed to the working directory.
 

{soup_cli-0.71.30 → soup_cli-0.71.32}/docs/training.md

@@ -401,6 +401,34 @@ advantages). Not a production reward-model claim; scale validation is help-wante
 (#286).
 
 
+## Online DPO (`task='online_dpo'`) — judge in the loop (v0.71.31)
+
+Unlike offline DPO (static `prompt/chosen/rejected` rows), **Online DPO** generates
+two completions per prompt **on-policy** each step and asks a *judge* — or a
+*reward model* — which is better; the winner becomes `chosen`, the loser
+`rejected`. The judge closes the loop. Wraps TRL `OnlineDPOTrainer`; data is
+prompt-only (like GRPO). Transformers + text only.
+
+```yaml
+base: HuggingFaceTB/SmolLM2-135M-Instruct
+task: online_dpo
+data:
+  train: ./data/prompts.jsonl      # prompt-only (or any format — prompts are extracted)
+training:
+  online_dpo_judge: "ollama://llama3.1"   # a pairwise judge (ollama://|https://|http://localhost)
+  # OR: reward_model: ./my-reward-model   # exactly one of judge / reward_model
+  online_dpo_loss_type: sigmoid           # sigmoid | ipo
+  online_dpo_max_new_tokens: 64
+  dpo_beta: 0.1
+  lora: { r: 8, alpha: 16, target_modules: auto }
+```
+
+The judge is Soup's own OpenAI-compatible `JudgeEvaluator` adapted to TRL's
+`BasePairwiseJudge` (swap-debiased: a winner is only recorded when both A,B and
+B,A orders agree). Recipe: `online-dpo-smollm2-135m`. Proof-of-mechanism was
+validated on SmolLM2-135M with a synthetic judge (not a production RLHF claim; #286).
+
+
 ## Weighted Multi-Objective Preference Loss
 
 Mix DPO / SimPO / ORPO / IPO terms in one training run by setting
@@ -533,6 +561,48 @@ training:
 {"audio": "recording.wav", "messages": [{"role": "user", "content": "Transcribe this audio."}, {"role": "assistant", "content": "Hello world."}]}
 ```
 
+### ASR fine-tuning (`task='asr'`, Whisper) — v0.71.32
+
+Fine-tune Whisper on your accent or domain. whisper-tiny (39M) and base (74M)
+train on a 4 GB GPU. Rows are `{"audio": <path>, "text": <transcript>}` under
+`data.format='asr'`; audio decodes to 16 kHz mono through the hardened loader.
+
+```yaml
+base: openai/whisper-tiny
+task: asr
+data:
+  train: ./data/train.jsonl
+  format: asr                 # rows: {"audio": "clip.wav", "text": "hello world"}
+  audio_dir: ./data/audio     # audio paths resolve here (containment-checked)
+training:
+  epochs: 3
+  lr: 1e-4
+  batch_size: 2
+  asr_language: en            # optional; sets + persists the decoder prefix
+  asr_task: transcribe        # transcribe | translate
+  asr_lora: true              # optional LoRA on q/v; default = full fine-tune
+  quantization: none
+output: ./out
+```
+
+```json
+{"audio": "clip0.wav", "text": "hello world"}
+```
+
+Transcribe + score after training:
+
+```bash
+soup infer --task asr --model ./out --input eval.jsonl --output preds.jsonl --audio-dir ./data/audio
+# -> preds carry {"transcription", "wer", "cer"} per row + a corpus WER summary
+```
+
+Notes: `task='asr'` requires `backend='transformers'` and a Whisper base (a
+non-Whisper base is rejected before download). `asr_language`/`asr_task` persist
+to an `asr_generation.json` sidecar so `soup infer --task asr` restores them
+(override with `--asr-language`/`--asr-task`). WER/CER use a light normalizer —
+good for before/after deltas, not leaderboard-comparable absolutes.
+`whisper-large-v3-asr` ships parse-only (needs a larger GPU).
+
 
 ## GRPO Plus — Objective Variants, Long-Context RL, Multi-Turn Agents
 

{soup_cli-0.71.30 → soup_cli-0.71.32}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.30"
+version = "0.71.32"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.30 → soup_cli-0.71.32}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.30"
+__version__ = "0.71.32"