soup-cli 0.71.31__tar.gz → 0.71.32__tar.gz

{soup_cli-0.71.31 → soup_cli-0.71.32}/CHANGELOG.md

@@ -12,6 +12,36 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.32] - 2026-07-07
+
+### Added
+- **ASR fine-tuning (`task='asr'`, Whisper)** — fine-tune Whisper on your accent
+  or domain, locally. whisper-tiny (39M) / base (74M) train on a 4 GB GPU.
+  - New `AsrTrainerWrapper` (HF `Seq2SeqTrainer` + `WhisperProcessor`); data rows
+    are `{"audio": <path>, "text": <transcript>}` under the new `data.format='asr'`.
+    Audio decodes via the hardened `load_audio_mono` (16 kHz mono, soundfile
+    pre-probe + `O_NOFOLLOW` + symlink/size guards); transcripts become decoder
+    labels (pad → −100, decoder-start token stripped).
+  - Optional LoRA on q/v attention projections via `training.asr_lora: true`
+    (default full fine-tune); `training.asr_language` / `training.asr_task`
+    (`transcribe`|`translate`) set the decoder prefix and are persisted to an
+    `asr_generation.json` sidecar so inference restores them.
+  - **`soup infer --task asr`** — transcribe an `{"audio": path[, "text": ref]}`
+    JSONL; reports per-row and corpus WER/CER when references are present. Loads a
+    full model or a PEFT/LoRA adapter dir. Flags: `--asr-language`, `--asr-task`,
+    `--audio-dir` (audio paths are cwd/dir-contained; UNC/traversal rejected).
+  - New pure-python `utils/asr_metrics.py` — WER / CER / `word_accuracy`
+    (= 1 − WER, for a higher-is-better ship metric leg) / `corpus_wer`, with a
+    light Whisper-style text normalizer (no new dependency).
+  - Recipes: `whisper-tiny-asr`, `whisper-base-asr` (live-trainable),
+    `whisper-large-v3-asr` (parse-only, needs a larger GPU), plus
+    `smolvlm-256m-sft` (vision). Catalog 138 → 142.
+
+### Fixed
+- `model_size_from_name` now knows Whisper checkpoint sizes (tiny…large), so the
+  hardware-fit gate no longer mistakes a 39M whisper-tiny for the 7B default and
+  blocks ASR training on consumer GPUs.
+
 ## [0.71.31] - 2026-07-06
 
 ### Added

{soup_cli-0.71.31 → soup_cli-0.71.32}/CONTRIBUTING.md

@@ -110,7 +110,7 @@ src/soup_cli/
   experiment/         - SQLite experiment tracking
   eval/               - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
   migrate/            - Config migration (LLaMA-Factory, Axolotl, Unsloth)
-  recipes/            - Ready-made configs for popular models (138 recipes)
+  recipes/            - Ready-made configs for popular models (142 recipes)
   autopilot/          - Zero-config decision engine (v0.25.0)
   registry/           - Model Registry (hashing, store, diff, attach) (v0.26.0 + v0.33.0)
   cans/               - Shareable .can artifact format + run/publish orchestrator (v0.26.0 + v0.33.0)
@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (308 files, 15421 tests)
+tests/                - Test suite (309 files, 15663 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.31 → soup_cli-0.71.32}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.31
+Version: 0.71.32
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,29 +193,33 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.31 — Judge-in-the-loop suite.** Put an LLM judge in the loop across the whole workflow: train against a judge, mine winners from a base model, grow instruction diversity, and decide SHIP with a true pairwise judge win-rate.
-
-- **`task='online_dpo'`.** Online DPO (wraps TRL `OnlineDPOTrainer`): the model generates two
-  completions per prompt on-policy each step and a *judge* (a pairwise LLM judge over your local
-  ollama / OpenAI-compatible endpoint) — or a `reward_model` — picks the winner. Set
-  `training.online_dpo_judge: "ollama://llama3.1"` (or `reward_model`, exactly one).
-- **`soup data best-of-n`.** Best-of-N rejection sampling: sample N completions from `--base`
-  locally, a `--judge` scores each, and the winner becomes an SFT row (`--emit-pairs` also writes
-  winner-vs-loser DPO pairs).
-- **`soup data evolve`.** WizardLM Evol-Instruct (depth / breadth) over an ollama / vllm provider —
-  completing the synthetic-data suite (Magpie / Forge / Persona / evolve).
-- **`soup ship --task-mode pairwise` (#284).** A true swap-debiased judge win-rate as the ship
-  leg-1 task-win, fused with the catastrophic-forgetting guard into one SHIP / DON'T-SHIP verdict.
-- **Proof-of-mechanism, honestly.** Live-validated on SmolLM2-135M: online-DPO trains with a
-  synthetic judge (`rewards/*` + `objective/kl` logged), best-of-N samples + picks, evolve loops.
-  Not a production RLHF claim (scale help wanted: #286).
+**v0.71.32 — ASR fine-tuning (Whisper).** Fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) and base (74M) train on a 4 GB GPU — a new modality that actually runs on a laptop card.
+
+- **`task='asr'`.** A new `AsrTrainerWrapper` (HF `Seq2SeqTrainer` + `WhisperProcessor`). Data rows
+  are `{"audio": <path>, "text": <transcript>}` under `data.format='asr'`; audio decodes to 16 kHz
+  mono through the hardened loader (soundfile pre-probe + symlink/size guards). Opt-in LoRA on q/v
+  via `training.asr_lora: true` (default full fine-tune); `training.asr_language` / `asr_task`
+  (`transcribe`|`translate`) set — and persist — the decoder prefix for inference.
+- **`soup infer --task asr`.** Transcribe an `{"audio"[, "text"]}` JSONL and get per-row + corpus
+  **WER / CER** when references are present. Loads a full model or a LoRA adapter dir; audio paths
+  are containment-checked (traversal / UNC rejected).
+- **Pure-python metrics.** New `soup_cli.utils.asr_metrics` (WER / CER / `word_accuracy` /
+  `corpus_wer`) — no new dependency, reusable as a higher-is-better ship metric leg.
+- **Recipes.** `whisper-tiny-asr` / `whisper-base-asr` (live-trainable), `whisper-large-v3-asr`
+  (parse-only). Catalog 138 → 142.
+- **Proof-of-mechanism, honestly.** Live-validated on an RTX 3050: a memorization task drove
+  WER 1.000 → 0.000 (loss 7.2 → 0.0005 at 0.4/4 GB). Real accent/domain gains need real audio;
+  WER/CER use a light normalizer, not the full Whisper text normalizer.
 
 ```yaml
-task: online_dpo
+base: openai/whisper-tiny
+task: asr
+data:
+  format: asr            # rows: {"audio": "clip.wav", "text": "hello world"}
+  audio_dir: ./data/audio
 training:
-  online_dpo_judge: "ollama://llama3.1"   # a pairwise judge (or set reward_model)
-  online_dpo_loss_type: sigmoid           # sigmoid | ipo
-  online_dpo_max_new_tokens: 64
+  asr_language: en
+  asr_lora: true         # optional; default = full fine-tune
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.31 → soup_cli-0.71.32}/README.md

@@ -49,29 +49,33 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.31 — Judge-in-the-loop suite.** Put an LLM judge in the loop across the whole workflow: train against a judge, mine winners from a base model, grow instruction diversity, and decide SHIP with a true pairwise judge win-rate.
-
-- **`task='online_dpo'`.** Online DPO (wraps TRL `OnlineDPOTrainer`): the model generates two
-  completions per prompt on-policy each step and a *judge* (a pairwise LLM judge over your local
-  ollama / OpenAI-compatible endpoint) — or a `reward_model` — picks the winner. Set
-  `training.online_dpo_judge: "ollama://llama3.1"` (or `reward_model`, exactly one).
-- **`soup data best-of-n`.** Best-of-N rejection sampling: sample N completions from `--base`
-  locally, a `--judge` scores each, and the winner becomes an SFT row (`--emit-pairs` also writes
-  winner-vs-loser DPO pairs).
-- **`soup data evolve`.** WizardLM Evol-Instruct (depth / breadth) over an ollama / vllm provider —
-  completing the synthetic-data suite (Magpie / Forge / Persona / evolve).
-- **`soup ship --task-mode pairwise` (#284).** A true swap-debiased judge win-rate as the ship
-  leg-1 task-win, fused with the catastrophic-forgetting guard into one SHIP / DON'T-SHIP verdict.
-- **Proof-of-mechanism, honestly.** Live-validated on SmolLM2-135M: online-DPO trains with a
-  synthetic judge (`rewards/*` + `objective/kl` logged), best-of-N samples + picks, evolve loops.
-  Not a production RLHF claim (scale help wanted: #286).
+**v0.71.32 — ASR fine-tuning (Whisper).** Fine-tune Whisper on your accent or domain, locally. whisper-tiny (39M) and base (74M) train on a 4 GB GPU — a new modality that actually runs on a laptop card.
+
+- **`task='asr'`.** A new `AsrTrainerWrapper` (HF `Seq2SeqTrainer` + `WhisperProcessor`). Data rows
+  are `{"audio": <path>, "text": <transcript>}` under `data.format='asr'`; audio decodes to 16 kHz
+  mono through the hardened loader (soundfile pre-probe + symlink/size guards). Opt-in LoRA on q/v
+  via `training.asr_lora: true` (default full fine-tune); `training.asr_language` / `asr_task`
+  (`transcribe`|`translate`) set — and persist — the decoder prefix for inference.
+- **`soup infer --task asr`.** Transcribe an `{"audio"[, "text"]}` JSONL and get per-row + corpus
+  **WER / CER** when references are present. Loads a full model or a LoRA adapter dir; audio paths
+  are containment-checked (traversal / UNC rejected).
+- **Pure-python metrics.** New `soup_cli.utils.asr_metrics` (WER / CER / `word_accuracy` /
+  `corpus_wer`) — no new dependency, reusable as a higher-is-better ship metric leg.
+- **Recipes.** `whisper-tiny-asr` / `whisper-base-asr` (live-trainable), `whisper-large-v3-asr`
+  (parse-only). Catalog 138 → 142.
+- **Proof-of-mechanism, honestly.** Live-validated on an RTX 3050: a memorization task drove
+  WER 1.000 → 0.000 (loss 7.2 → 0.0005 at 0.4/4 GB). Real accent/domain gains need real audio;
+  WER/CER use a light normalizer, not the full Whisper text normalizer.
 
 ```yaml
-task: online_dpo
+base: openai/whisper-tiny
+task: asr
+data:
+  format: asr            # rows: {"audio": "clip.wav", "text": "hello world"}
+  audio_dir: ./data/audio
 training:
-  online_dpo_judge: "ollama://llama3.1"   # a pairwise judge (or set reward_model)
-  online_dpo_loss_type: sigmoid           # sigmoid | ipo
-  online_dpo_max_new_tokens: 64
+  asr_language: en
+  asr_lora: true         # optional; default = full fine-tune
 ```
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).

{soup_cli-0.71.31 → soup_cli-0.71.32}/docs/commands.md

@@ -19,6 +19,7 @@ soup train --config soup.yaml --push-as user/repo --hf-resume  Resume from lates
 soup train --config soup.yaml --find-lr        LR range finder: write recommended LR JSON
 soup train --config soup.yaml --cloud modal --gpu a100  Render a Modal.com app for serverless GPU training (plan-only; --cloud-submit submits live)
 soup infer --model ./output --input p.jsonl   Batch inference
+soup infer --task asr --model <whisper|adapter> --input a.jsonl --output o.jsonl [--audio-dir d --asr-language en --asr-task transcribe|translate]  Whisper transcription + WER/CER
 soup chat --model ./output                    Interactive chat
 soup push --model ./output --repo user/name   Upload to HuggingFace
 soup push --model ./output --repo user/name --collection user/coll-abc123  Add to HF Collection

{soup_cli-0.71.31 → soup_cli-0.71.32}/docs/data.md

@@ -352,6 +352,13 @@ Or use `.txt` files directly (one document per line).
 {"audio": "recording.wav", "messages": [{"role": "user", "content": "Transcribe."}, {"role": "assistant", "content": "Hello world."}]}
 ```
 
+**ASR (Whisper transcription — `data.format: asr`, v0.71.32):**
+```json
+{"audio": "clip.wav", "text": "hello world"}
+```
+Audio paths resolve under `data.audio_dir` (containment-checked). Used by
+`task: asr` and `soup infer --task asr`. See [Training → ASR](training.md).
+
 **PRM (process reward, stepwise-supervised):**
 ```json
 {"prompt": "Solve 2+2", "completions": ["First, add", "Result is 4"], "labels": [true, true]}

{soup_cli-0.71.31 → soup_cli-0.71.32}/docs/models.md

@@ -40,6 +40,17 @@ Soup works with **any** of the **340,000+** text-generation models on [HuggingFa
 | Qwen2-VL-7B-Instruct | 7B | LLaVA, ShareGPT4V |
 | Pixtral-12B-2409 | 12B | LLaVA, ShareGPT4V |
 
+### ASR Models (`task: asr`, Whisper — v0.71.32)
+
+| Recipe | Base | Size | Status |
+|---|---|---|---|
+| `whisper-tiny-asr` | openai/whisper-tiny | 39M | Live on 4 GB |
+| `whisper-base-asr` | openai/whisper-base | 74M | Live on 4 GB |
+| `whisper-large-v3-asr` | openai/whisper-large-v3 | 1.5B | Parse-only (larger GPU) |
+
+Rows are `{"audio": <path>, "text": <transcript>}` with `data.format: asr`. See
+[Training → ASR fine-tuning](training.md).
+
 ### Quick Size Guide
 
 | VRAM | Max Model (QLoRA 4-bit) | Example |

{soup_cli-0.71.31 → soup_cli-0.71.32}/docs/training.md

@@ -561,6 +561,48 @@ training:
 {"audio": "recording.wav", "messages": [{"role": "user", "content": "Transcribe this audio."}, {"role": "assistant", "content": "Hello world."}]}
 ```
 
+### ASR fine-tuning (`task='asr'`, Whisper) — v0.71.32
+
+Fine-tune Whisper on your accent or domain. whisper-tiny (39M) and base (74M)
+train on a 4 GB GPU. Rows are `{"audio": <path>, "text": <transcript>}` under
+`data.format='asr'`; audio decodes to 16 kHz mono through the hardened loader.
+
+```yaml
+base: openai/whisper-tiny
+task: asr
+data:
+  train: ./data/train.jsonl
+  format: asr                 # rows: {"audio": "clip.wav", "text": "hello world"}
+  audio_dir: ./data/audio     # audio paths resolve here (containment-checked)
+training:
+  epochs: 3
+  lr: 1e-4
+  batch_size: 2
+  asr_language: en            # optional; sets + persists the decoder prefix
+  asr_task: transcribe        # transcribe | translate
+  asr_lora: true              # optional LoRA on q/v; default = full fine-tune
+  quantization: none
+output: ./out
+```
+
+```json
+{"audio": "clip0.wav", "text": "hello world"}
+```
+
+Transcribe + score after training:
+
+```bash
+soup infer --task asr --model ./out --input eval.jsonl --output preds.jsonl --audio-dir ./data/audio
+# -> preds carry {"transcription", "wer", "cer"} per row + a corpus WER summary
+```
+
+Notes: `task='asr'` requires `backend='transformers'` and a Whisper base (a
+non-Whisper base is rejected before download). `asr_language`/`asr_task` persist
+to an `asr_generation.json` sidecar so `soup infer --task asr` restores them
+(override with `--asr-language`/`--asr-task`). WER/CER use a light normalizer —
+good for before/after deltas, not leaderboard-comparable absolutes.
+`whisper-large-v3-asr` ships parse-only (needs a larger GPU).
+
 
 ## GRPO Plus — Objective Variants, Long-Context RL, Multi-Turn Agents
 

{soup_cli-0.71.31 → soup_cli-0.71.32}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.31"
+version = "0.71.32"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.31 → soup_cli-0.71.32}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.31"
+__version__ = "0.71.32"