soup-cli 0.76.0__tar.gz → 0.76.1__tar.gz

soup_cli-0.76.1/.github/constraints/transformers-floor.txt

@@ -0,0 +1,25 @@
+# #571 — exact pins for the dedicated dependency-floor CI job.
+# Qwen3.8-Flash-Next text loading establishes Transformers 5.16.1 as Soup's
+# declared training floor. trl 0.29 is the first release that does not import the
+# Transformers private availability helper whose v5 return type broke trl 0.28.
+# Torch 2.6.0 deliberately exercises Qwen4-Exp's legacy int64-only scatter path.
+# Plotext 6.0.0 is pinned here so one CI cell exercises the real Figure API rather
+# than relying only on the fake 5.x/6.x contract doubles in the unit suite.
+torch==2.6.0
+torchaudio==2.6.0
+transformers==5.16.1
+trl==0.29.0
+peft==0.20.0
+plotext==6.0.0
+# #1802 — the declared floors of the CLI's own framework, so one CI cell starts the
+# CLI on them: `typer>=0.9.0` stayed declared while no command could run below
+# 0.12.4, and nothing noticed because every cell resolved the newest Typer. These two
+# pins must equal the floors in pyproject.toml's [project] dependencies
+# (tests/test_cli_dependency_floors.py refuses a pin that differs), and the job runs
+# that file to build the command tree and render every help page under them.
+# Side effect, intended: huggingface-hub requires typer>=0.20.0 from 1.14.0 and
+# click>=8.4.0 from 1.17.0, so under these pins the cell resolves huggingface-hub
+# 1.13.0 (and the newest datasets that accepts it, 5.0.1 on 2026-10-13) — the low
+# corner of the declared ranges, where every other cell runs the high one.
+typer==0.15.4
+click==8.0.0

{soup_cli-0.76.0 → soup_cli-0.76.1}/.github/workflows/ci.yml

@@ -454,6 +454,10 @@ jobs:
   # #502/#503/#522/#571/#651: install the exact Torch 2.6.0 / Transformers 5.16.1 /
   # trl 0.29 / PEFT 0.20 support stack and Plotext 6.0.0 so pip cannot silently
   # upgrade the cell, then assert every pinned compatibility boundary.
+  # #1802: the same file pins Typer and Click to the floors pyproject.toml declares,
+  # and the last step starts the CLI on them. It is the one cell that does: every
+  # other cell resolves the newest Typer, which is how a floor no command could run
+  # on stayed declared. The job keeps its name; branch protection requires it.
   transformers-floor:
     needs: plan
     if: needs.plan.outputs.full == 'true'
@@ -488,7 +492,10 @@ jobs:
           )
           expected = dict(pairs)
           assert len(expected) == len(pairs), "duplicate package pin in floor constraints"
-          for name in ("torch", "torchaudio", "transformers", "trl", "peft", "plotext"):
+          for name in (
+              "torch", "torchaudio", "transformers", "trl", "peft", "plotext",
+              "typer", "click",  # #1802: the CLI's own declared floors
+          ):
               assert name in expected, f"floor constraints missing exact {name} pin"
               want = expected[name]
               got = md.version(name)
@@ -528,6 +535,9 @@ jobs:
           tests/test_bugfixes.py::TestDiffModelLoading
           tests/test_issue636_torch_floor.py
           tests/test_issue265_xcodec2_audio_contract.py
+          tests/test_cli_dependency_floors.py
+          tests/test_click_argument_help.py
+          tests/test_cli_startup_is_light.py
 
   test:
     needs: plan

{soup_cli-0.76.0 → soup_cli-0.76.1}/CHANGELOG.md

@@ -12,6 +12,42 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.76.1] - 2026-10-14
+
+### Changed
+
+- **The README and the performance docs now state the 8B NF4 figure with its SM clock and give the post-#331 row, and no longer say the re-measurement is pending (#1207, in #1699).** Llama-3.1-8B-Instruct NF4 on an RTX 3050 Laptop 4 GB reads **119.6 tok/s at 952 MHz in a 3.32 GB peak** before the #331 repair and **208.6 tok/s at 1935–1957 MHz in a 2.40 GB peak** after it, the second row measured by @umran666 (#1103). The two are not directly comparable: 59% against 68% of the same-session GEMM ceiling is about 14% slower per unit clock, so the raw 1.74x gain is the clock, not a speedup, and large-layer streaming changed the memory pattern between the rows, so even the fraction is not like-for-like. Changed: the headline, the GIF alt text and caption and the preprint bullet (Citing Soup) in `README.md`, and the table row, the headline and the large-layer paragraph in `docs/performance-and-quantization.md`. The four README translations (`README.tr.md`, `README.ar.md`, `README.ja.md`, `README.zh.md`) carry the same sentences, with every number, link and code span kept identical, and are re-stamped; their prose is a machine-assisted draft for the translation maintainers to polish. The preprint (DOI 10.5281/zenodo.21771064) is not edited in place; #1207 tracks its amendment as a new version.
+
+- **The "What's New" block of `README.tr.md`, `README.ar.md`, `README.ja.md` and `README.zh.md`
+  now describes v0.76.0 (#1825).** All four still described v0.75.0. The new text follows the
+  block in `README.md` bullet by bullet, with every inline code span, command, flag, config key,
+  URL, link target, number and benchmark file name kept identical; the prose is a machine-assisted
+  draft for the translation maintainers to polish. This change moves no `synced-from` stamp,
+  because the section is exempt from them; the stamps of the four files were re-taken by #1699 in
+  the same release.
+
+- soup.yaml and every other YAML file Soup reads go through one loader that applies the same nesting and expansion limits (100 levels of nesting; one node per source byte plus 10,000 for anchors, aliases and merge keys, with merge keys chained at most 100 levels), and each reader keeps its own size cap: 1 MiB for the training-config readers and for readers that had no cap before, 256 KiB for recipes, `export --quant-config` files, RA-DIT configs and sweep files, 5 MiB for agent specs, 16 MiB for can manifests, 64 KiB for README frontmatter, 4 MiB for the config read by `soup ship`. A soup.yaml that is over a limit, cannot be read or is not valid YAML is refused with one line naming the file and the problem; the other readers report through their own handlers. The audit log now finds credential options from the command tree and also masks long options by name, so the values of `--slack-url` and `--discord-url` are masked and `--min-tokens`, a length threshold, is no longer masked (#1827).
+
+- **The Typer floor is `typer>=0.15.4`, and one CI cell now starts the CLI on the declared Typer
+  and Click floors (#1858).** `typer>=0.9.0` was declared while no command could run below Typer
+  0.12.4: the command tree did not build (`RuntimeError: Type not yet supported: int | None`), so
+  `soup --help` and `soup version` failed with everything else. Typer 0.12.4 - 0.15.3 start the
+  CLI only next to Click 8.1 or older and do not ask for it, so a resolver paired them with a
+  Click on which every help page raised. 0.15.4 is the lowest Typer that works with any Click it
+  can be installed with; `click>=8.0.0` was measured again and holds. A fresh install is
+  unchanged, since a resolver already picks the newest Typer. An environment that holds an older
+  Typer now has it upgraded, or the conflict named, at install time, where before the install
+  succeeded and the first command failed; `soup doctor` reports such a Typer as outdated. The
+  `transformers-floor` CI job installs exactly the two declared floors, builds the command tree,
+  renders every help page and runs `soup version` under them, and a test fails when a declared
+  floor and that job's pin differ (#1802).
+
+### Fixed
+
+- **`task: dpo`, `ipo`, `bco` and `preference` train on the bundled DPO examples again.** Both files (`examples/data/dpo_sample.jsonl`, `examples/data/chat_preferences.jsonl`) and any dataset written the same way, a string `prompt` beside `chosen` / `rejected` message lists that repeat the user turn, failed inside trl after the model had loaded, with an error that changed with `PYTHONHASHSEED` (`'list' object has no attribute 'endswith'`, or a `TypeError`). The `dpo` and `kto` converters now hand the trainer every preference row in one layout: all strings, or all message lists with the prompt turns in `prompt` only. `reward_model` no longer trains on a text with the question in it twice, and `orpo` and `simpo` prepare the same tokens as before. A row that cannot be written that way (one answer a string and the other a message list, a malformed message, no prompt to be found) is dropped at load with a reason that names the field, and a dataset that mixes text rows with conversational ones loads as conversations, with a note. A string `prompt` beside message-list answers is not read when the two answers share leading turns: those turns are the prompt, as `orpo` and `simpo` already read such a row, so a question written only in that string has to be in the conversations as well (#1849 by @dryn85-max in #1866).
+
+- A single-string `data.train` that names a Hub dataset with a dot in its name (`teknium/OpenHermes-2.5`, `org/name.v1`) is now loaded from the Hub instead of failing with `Data file not found`: the loader read the dot as a file extension. A single string now takes the same route as the same name inside a `data.train` list: a supported file extension (`.jsonl`, `.json`, `.csv`, `.parquet`, `.txt`) means a local file, a `://` string a remote URI, anything else a Hub id. A path with an unsupported suffix that does not exist (a typo such as `data/train.jsnol`) therefore goes to the Hub lookup too, instead of `Data file not found`, while an existing file with an unsupported suffix (`data.tsv`) still gets `Unsupported file format` (#1889).
+
 ## [0.76.0] - 2026-10-12
 
 ### Added
@@ -1369,7 +1405,7 @@ control characters in rejected values are escaped (#953 by @shubhambhar007 in #9
   `soup data pii` and `soup data score` (#1329 by @taljeon in #1360).
 - Email snippets now follow the bounded local-part match, so leading punctuation
   and punctuation-adjacent addresses may produce different snippet spans; row-level
-  detection remains unchanged.
+  detection remains unchanged (#1329 by @taljeon in #1360).
 
 - **Refuse incompatible pre-tokenized data transformations.** Soup data preprocessing does not apply non-text modality transforms (vision/audio/audio_out), `data.add_new_tokens`, `data.new_special_tokens`, or `data.prompt_strategy` to cached ids, so those combinations are refused at config load (#1273 by @Samearth17 in #1363)
 
@@ -1491,7 +1527,8 @@ control characters in rejected values are escaped (#953 by @shubhambhar007 in #9
   not given, so these rules apply without the flag too, and to `soup bench`,
   which loads through `soup infer`. On a CPU-only host the model now loads in
   `float32` pinned to the CPU instead of `"auto"`/`float16`, which doubles its
-  RAM use (a 7B model needs about 28 GB instead of 14 GB).
+  RAM use (a 7B model needs about 28 GB instead of 14 GB) (#1443 by @drakeo338 in
+  #1460).
 
 - **`soup expect` refuses suite keys and argument names an expectation does not take, instead of running it on its defaults (#1428 by @Srinivasan8888 in #1461).**
   `parse_suite_spec` read only `name` and `args` and dropped every other key, and the dispatcher read arguments with `args.get(key, default)`, so the `min_tokens` / `max_tokens` example in `docs/compliance.md` (written beside `name` instead of under `args:`) and a misspelled argument such as `max_token` both ran `expect_token_length_between` on its defaults of 1 and 1,048,576 tokens and passed a 5000-token row with exit 0. A key beside `name` or an argument the expectation does not take is now refused when the suite loads (exit 3, before the data file is opened), with the key named and either "belongs under `args:`" or the nearest valid name; `min_tokens` / `max_tokens` / `threshold` are validated at parse time too. The `docs/compliance.md` example moves to the `args:` form. A suite that carried one of these keys now exits 3 on its next run, which is the point.
@@ -2028,7 +2065,7 @@ control characters in rejected values are escaped (#953 by @shubhambhar007 in #9
 
 ### Security
 
-- Hardening of outbound endpoint validation. A security advisory with details and upgrade guidance will be published alongside this release.
+- Hardening of outbound endpoint validation — [GHSA-9f49-hqpf-c849](https://github.com/MakazhanAlpamys/Soup/security/advisories/GHSA-9f49-hqpf-c849). The advisory has the details and upgrade guidance.
 
 ## [0.75.1] - 2026-09-21
 
@@ -6684,7 +6721,8 @@ what a fine-tune forgets and leaks.
   `SECURITY.md` (~220 KB). `SECURITY.md` is now a concise security policy; the
   detailed hardening notes remain in git history and the GitHub Releases notes.
 
-[Unreleased]: https://github.com/MakazhanAlpamys/Soup/compare/v0.76.0...HEAD
+[Unreleased]: https://github.com/MakazhanAlpamys/Soup/compare/v0.76.1...HEAD
+[0.76.1]: https://github.com/MakazhanAlpamys/Soup/compare/v0.76.0...v0.76.1
 [0.76.0]: https://github.com/MakazhanAlpamys/Soup/compare/v0.75.2...v0.76.0
 [0.75.2]: https://github.com/MakazhanAlpamys/Soup/compare/v0.75.1...v0.75.2
 [0.75.1]: https://github.com/MakazhanAlpamys/Soup/compare/v0.75.0...v0.75.1

{soup_cli-0.76.0 → soup_cli-0.76.1}/CONTRIBUTING.md

@@ -122,6 +122,7 @@ src/soup_cli/
   monitoring/         - Callbacks and live dashboard
   experiment/         - SQLite experiment tracking
   eval/               - Eval platform (custom tasks, LLM judge, human eval, leaderboard)
+  bench/              - Benchmark instruments behind `soup bench train` (collector, train_report, train_run) (#836)
   migrate/            - Config migration (LLaMA-Factory, Axolotl, Unsloth)
   recipes/            - Ready-made configs for popular models (170 recipes)
   autopilot/          - Zero-config decision engine (v0.25.0)
@@ -289,51 +290,51 @@ Accepted runs are recorded in [`benchmarks/gpu-test-runs.md`](benchmarks/gpu-tes
 | test_inference_advanced.py | Inference Excellence: prefix caching, spec-decoding auto-pairing, LoRA hot-swap, structured output, dashboard + /metrics, OpenTelemetry tracing (v0.30.0) |
 | test_recipes_v031.py | Model & Recipe Breadth: 34 new recipes (vision/audio/reasoning/edge/domain/multimodal); catalog-wide invariants; CI workflow validation (v0.31.0) |
 | test_auto_tuning.py | Training Stability & Auto-Tuning: LR range finder, grad-accum monitor, auto mixed-precision, auto warmup, spike recovery, convergence detector, autopilot wiring (v0.32.0) |
-| test_part_f_hardening.py | Live Wire Part F: RLVR OS-level sandbox isolation + prune_checkpoints TOCTOU (v0.33.0) |
-| test_part_a_wave1.py | Live Wire Part A: live eval-gate scoring + registry attach (v0.33.0) |
-| test_part_a_wave2.py | Live Wire Part A: soup can run / publish + DeployTarget schema (v0.33.0) |
-| test_part_e.py | Live Wire Part E: --find-lr live loop + spike recovery hint + auto mixed-precision push + grad-accum advisory (v0.33.0) |
-| test_part_d.py | Historical Part D primitives: structured-output LogitsProcessor + HF push integration smoke (v0.33.0) |
-| test_part_c.py | Live Wire Part C: multi-trainer v0.28.0 features + selective ckpt hooks + CrossDocCollator (v0.33.0) |
-| test_part_b.py | Live Wire Part B: auto-reexec under accelerate launch + DeepSpeed-MII live serve (v0.33.0) |
-| test_log_level.py | Smart logging tiers `--log-level quiet/normal/verbose/debug` (v0.34.0 Part A) |
-| test_run_cost.py | Per-run cost: GPU-rate lookup + estimate + format + tracker integration (v0.34.0 Part B) |
-| test_why.py | `soup why` heuristic explainer: NaN / plateau / divergence / grad-norm / LR bounds (v0.34.0 Part C) |
-| test_crash_reporter.py | `.crash` bundle: classify, redact secrets, write under cwd, oversize truncation (v0.34.0 Part D) |
-| test_replay.py | `soup runs replay`: summarise + downsample + CLI rendering (v0.34.0 Part E) |
-| test_profiling.py | Auto-profiling: ProfilerSchedule + path containment + torch-less degradation (v0.34.0 Part F) |
-| test_tui.py | `soup tui`: CLI bounds + missing-textual error + row builders (v0.34.0 Part G) |
-| test_trainer_coverage_v035.py | Multi-trainer v0.28.0 wiring smoke matrix: every trainer × every speed/memory feature + benchmark_kernel_combos + schema-gate lift (v0.35.0 Parts A / B / C / D — #60, #61, #45) |
+| test_part_f_hardening.py | Live Wire hardening: RLVR OS-level sandbox isolation + prune_checkpoints TOCTOU (v0.33.0) |
+| test_part_a_wave1.py | Live Wire: live eval-gate scoring + registry attach (v0.33.0) |
+| test_part_a_wave2.py | Live Wire: soup can run / publish + DeployTarget schema (v0.33.0) |
+| test_part_e.py | Live Wire: --find-lr live loop + spike recovery hint + auto mixed-precision push + grad-accum advisory (v0.33.0) |
+| test_part_d.py | Historical Live Wire primitives: structured-output LogitsProcessor + HF push integration smoke (v0.33.0) |
+| test_part_c.py | Live Wire: multi-trainer v0.28.0 features + selective ckpt hooks + CrossDocCollator (v0.33.0) |
+| test_part_b.py | Live Wire: auto-reexec under accelerate launch + DeepSpeed-MII live serve (v0.33.0) |
+| test_log_level.py | Smart logging tiers `--log-level quiet/normal/verbose/debug` (v0.34.0) |
+| test_run_cost.py | Per-run cost: GPU-rate lookup + estimate + format + tracker integration (v0.34.0) |
+| test_why.py | `soup why` heuristic explainer: NaN / plateau / divergence / grad-norm / LR bounds (v0.34.0) |
+| test_crash_reporter.py | `.crash` bundle: classify, redact secrets, write under cwd, oversize truncation (v0.34.0) |
+| test_replay.py | `soup runs replay`: summarise + downsample + CLI rendering (v0.34.0) |
+| test_profiling.py | Auto-profiling: ProfilerSchedule + path containment + torch-less degradation (v0.34.0) |
+| test_tui.py | `soup tui`: CLI bounds + missing-textual error + row builders (v0.34.0) |
+| test_trainer_coverage_v035.py | Multi-trainer v0.28.0 wiring smoke matrix: every trainer × every speed/memory feature + benchmark_kernel_combos + schema-gate lift (v0.35.0 — #60, #61, #45) |
 | test_v0470_part_a.py | Synthetic Data Forge: ForgePlan / ProvenanceRecord / ForgeRow frozen dataclasses + VALID_TASKS allowlist + chunk_document + score_uncertainty + discover_documents (cwd-contained, symlink-rejecting, extension allowlist) + build_forge_plan validators (task / target_rows / teacher / NaN+Inf threshold) + synthesise_forge_rows (judge-exception swallow at DEBUG) + write_forge_dataset + write_provenance (atomic, TOCTOU-safe) + CLI smoke (v0.47.0) |
 | test_v0470_part_b.py | Data Quality Moat: BENCHMARKS MappingProxyType + ScoreReport frozen + ngram_set / ngram_overlap_ratio / decontaminate_rows (containment ratio) + detect_pii (ReDoS-hardened regexes + 50 KB pre-cap) + detect_language (6-language stopword heuristic) + score_toxicity (keyword baseline) + score_educational_value + compute_scorecard + load_jsonl_rows / write_jsonl_rows (cwd-contained, symlink-rejecting, atomic) + CLI smoke per subcommand (v0.47.0) |
 | test_v0480_part_a.py | Curriculum-Aware Trainer (BETA): DynamicCurriculumPolicy frozen + bounds; compute_bucket_weights softmax + water-fill (floor-strict invariant); validate_distributed_curriculum DDP gate; render_curve + parse_history_jsonl with 100k-row DoS cap; SoupConfig cross-validators (requires-curriculum / mlx-rejected / non-SFT-rejected / floor ≤ 1/buckets); `soup runs curriculum-curve` CLI (TOCTOU symlink reject + 50 MB cap + corrupt-JSONL exit 2) (v0.48.0) |
 | test_v0490.py | v0.49.0 Long Context & Architecture: YaRN math kernels (yarn_find_correction_dim / yarn_find_correction_range / yarn_linear_ramp_mask / yarn_get_mscale — bool/NaN/Inf rejection, auto-disambiguated low==high); YaRN schema fields (yarn_factor / yarn_attn_factor / yarn_beta_fast / yarn_beta_slow) + cross-validator rejecting yarn fields without rope_scaling_type=yarn; Dynamic NTK harden + verify; LongLoRA schema gate (Llama-family + sft + transformers + !ring_attn); `is_llama_model` word-boundary regex (mirrors v0.39.0 `is_gemma4_model` / v0.44.0 `is_llama4_model` policy); Llama 3.1 NTK-aware (`scale_inv_freq_llama3` smooth-transition / bool-rejected on every param / zero-`old_context_len` rejected; `detect_llama3_rope_in_config` with `rope_type` alias + explicit `is None` check); public-boundary input validation on `get_rope_scaling_config` (target_length / original_length / yarn_factor — security review fix) (v0.49.0) |
 | test_v0480_part_b.py | Data Mixing Optimizer (BETA): parse_budget (digits + s/m/h suffix [60s, 24h]) + validate_datasets (containment + symlink + dedup + 32-cap) + MixCandidate (simplex + finite + bool-rejected) + BudgetTracker (injectable clock) + run_mix_optimizer (isolated proxy failures + KeyboardInterrupt propagation + NaN skip + partial budget trip) + render_mix_recipe_yaml (YAML injection defence) + write_mix_recipe / load_mix_recipe (atomic + TOCTOU + 256 KB cap) + `soup data mix --optimize / --apply` CLI (v0.48.0) |
 
-| test_v0500_part_a.py | v0.50.0 Part A — GRPO objective variants (gspo / dapo / dr_grpo / bnpo / two_sided / rft / standard); closed allowlist + frozen `GRPOVariantSpec` + `MappingProxyType` immutability; `validate_grpo_variant` (bool / null-byte / oversize / unknown rejected); `validate_grpo_delta` (bool / NaN / Inf / out-of-range rejected); `apply_variant_loss` deferred-live stub; TrainingConfig integration (Literal accept + two_sided requires delta cross-validator + NaN/Inf field_validator) (v0.50.0 Part A) |
-| test_v0500_part_b.py | v0.50.0 Part B — Long-context GRPO + vLLM sleep; `validate_long_context_grpo_compat` (task / backend / ring-attention exclusivity + null-byte + bool guards); `validate_vllm_sleep_mode_compat` (transformers / unsloth allowlist + null-byte); `apply_vllm_sleep_mode` deferred stub with v0.50.1 marker; SoupConfig integration (vllm_sleep_mode requires task='grpo' — code-review HIGH fix) (v0.50.0 Part B) |
-| test_v0500_part_c.py | v0.50.0 Part C — Multi-turn agent rollout backend allowlist (art / ruler / nemo_gym / openenv); frozen `RolloutBackendSpec` + `MappingProxyType` immutability; `validate_rollout_backend` (bool rejected); `required_rollout_package` per-entry mapping; `launch_rollout` deferred stub; SoupConfig task-gate + mlx rejection (v0.50.0 Part C) |
-| test_v0500_part_d.py | v0.50.0 Part D — 7 stability/efficiency knobs (`ref_model_ema_alpha` / `replay_buffer_size` / `async_grpo_prefetch` / `tis_threshold` / `mask_truncated_completions` / `defer_rerolling` / `skip_zero_advantage` / `off_policy_mask_threshold`); explicit bool-rejection field_validator across all numeric fields (tdd-guide HIGH fix); `mask_truncated_completions` requires `tis_threshold` cross-validator; SoupConfig task-gate naming every offending field; `grpo_fp16` task-gate (code-review HIGH fix) (v0.50.0 Part D) |
-| test_v0500_part_e.py | v0.50.0 Part E — `task='prm'` (Process Reward Model) + `vision_grpo` flag; `validate_prm_compat` (data.format / modality / mlx gates); `validate_vision_grpo_compat` (task ∈ {grpo, ppo} / modality='vision' / non-mlx); `build_prm_trainer` deferred stub; SoupConfig integration with all rejection paths exercised (v0.50.0 Part E) |
+| test_v0500_part_a.py | v0.50.0 — GRPO objective variants (gspo / dapo / dr_grpo / bnpo / two_sided / rft / standard); closed allowlist + frozen `GRPOVariantSpec` + `MappingProxyType` immutability; `validate_grpo_variant` (bool / null-byte / oversize / unknown rejected); `validate_grpo_delta` (bool / NaN / Inf / out-of-range rejected); `apply_variant_loss` deferred-live stub; TrainingConfig integration (Literal accept + two_sided requires delta cross-validator + NaN/Inf field_validator) (v0.50.0) |
+| test_v0500_part_b.py | v0.50.0 — Long-context GRPO + vLLM sleep; `validate_long_context_grpo_compat` (task / backend / ring-attention exclusivity + null-byte + bool guards); `validate_vllm_sleep_mode_compat` (transformers / unsloth allowlist + null-byte); `apply_vllm_sleep_mode` deferred stub with v0.50.1 marker; SoupConfig integration (vllm_sleep_mode requires task='grpo' — code-review HIGH fix) (v0.50.0) |
+| test_v0500_part_c.py | v0.50.0 — Multi-turn agent rollout backend allowlist (art / ruler / nemo_gym / openenv); frozen `RolloutBackendSpec` + `MappingProxyType` immutability; `validate_rollout_backend` (bool rejected); `required_rollout_package` per-entry mapping; `launch_rollout` deferred stub; SoupConfig task-gate + mlx rejection (v0.50.0) |
+| test_v0500_part_d.py | v0.50.0 — 7 stability/efficiency knobs (`ref_model_ema_alpha` / `replay_buffer_size` / `async_grpo_prefetch` / `tis_threshold` / `mask_truncated_completions` / `defer_rerolling` / `skip_zero_advantage` / `off_policy_mask_threshold`); explicit bool-rejection field_validator across all numeric fields (tdd-guide HIGH fix); `mask_truncated_completions` requires `tis_threshold` cross-validator; SoupConfig task-gate naming every offending field; `grpo_fp16` task-gate (code-review HIGH fix) (v0.50.0) |
+| test_v0500_part_e.py | v0.50.0 — `task='prm'` (Process Reward Model) + `vision_grpo` flag; `validate_prm_compat` (data.format / modality / mlx gates); `validate_vision_grpo_compat` (task ∈ {grpo, ppo} / modality='vision' / non-mlx); `build_prm_trainer` deferred stub; SoupConfig integration with all rejection paths exercised (v0.50.0) |
 | test_v0520.py | v0.52.0 Modality II — TTS / classifier / distill / BitNet / EBFT-GDPO / MoE quant / reasoning_effort: TTS family allowlist + per-family emotion allowlists (Orpheus + Oute) + validate_tts_compat; classifier / reranker / cross_encoder tasks + num_labels (with field_validator bool guard) + label_names dedup + classifier-only field gates; distill divergence (kl alias canonicalised, Literal excludes alias) + teacher_model + distill_temperature bounds; BitNet 1.58 quant + bitnet/tq1_0 export-format stubs + Falcon-E recipe + is_bitnet_model org-prefix detect; EBFT (structured/strided) + GDPO (standard/length_normalized/margin) variant allowlists + task gates; MoE expert quant (nf4/int8_rowwise) + train_router_only requiring moe_lora=true; reasoning_effort + train_on_eot with SFT-family task gate; 6 new recipes (5 TTS + Falcon-E BitNet); review-fix coverage (num_labels bool guard, Oute emotion allowlist, lazy-import in classifier validator, task gates, oversize / NaN / Inf matrices). Test count: 272 (v0.52.0) |
 | test_v0531_82.py | v0.53.1 #82 autopilot pre-quantized detection: `detect_prequantized_format` + `decide_quantization(prequantized=...)` + `detect_prequantized_format_from_path` with cwd-containment + config.json symlink rejection + name/config aliases + word-boundary regex (v0.53.1) |
 | test_v0531_142.py | v0.53.1 #142 merge_4bit + export_torchao live wiring: BNB-4bit single-stage merge + TorchAO PTQ with per-scheme kwarg allowlist + CLI `--save-format` + `--quant-config` + `load_quant_config` (yaml.safe_load + 256 KB cap + extension allowlist) + path TOCTOU (v0.53.1) |
 | test_v0531_139.py | v0.53.1 #139 export_advanced_gguf live: 3-stage llama.cpp pipeline (convert → imatrix → quantize) + UD-prefix strip + subprocess argv shape + `_prepare_calibration_text` JSONL alias fallback + null-byte strip + 50 MB cap + POSIX O_NOFOLLOW + `_safe_stderr` Rich escape (v0.53.1) |
 | test_v0531_109.py | v0.53.1 #109 deploy autopilot --measure: `compute_cache_key` + `sha_of_file` + `measure_candidate` OK/MINOR/MAJOR bands + `pick_best` soft-fallback (max-by-delta) + cache round-trip with symlink rejection on load AND save + CLI integration + `_MAX_CANDIDATES=32` cap + `render_measure_table` markup escape regression (v0.53.1) |
-| test_v0530.py | v0.53.0 Quant Menu II — UD GGUFs + KV cache + NVFP4 + LF parity + save formats: Parts A+B GGUF (UD ladder 14 entries + IQ 12 + Apple/ARM 10 frozensets + non-overlap invariant + `validate_*` case-insensitive + rejection matrix + `is_advanced_gguf_format` union + `_LOWER_INDEX` O(1) lookup + MappingProxyType immutability + `validate_calibration_data_path` shape rejection + 4096-boundary + `export_advanced_gguf` v0.53.1 deferred stub); Part C KV cache (`KV_CACHE_TYPES` frozenset + `validate_kv_cache_type` case + bool/null/oversize/non-string rejection + `requires_hopper` delegates to spec + `get_kv_cache_spec` frozen + schema fp8-on-mlx rejected with specific message + q8_0-on-mlx allowed); Part D advanced precision (`fp8_attention` requires `quantization_aware='fp8'` BEFORE mlx-gate ordering + bool guards on every string param + schema rejects-without-fp8-qat; `nvfp4` mlx + vision rejection + bool guards; `unsloth_bnb_4bit` backend='unsloth' + quantization='4bit' rejection matrix; `apply_*` deferred); Part E LF parity (`bnb_4bit_use_double_quant` rejects none/8bit/gptq parametrize; `llm_int8` rejects default-none + 4bit; `quantize_ref_model` happy on dpo/grpo/kto + rejects sft/pretrain; `quantize_reward_model` happy on ppo/reward_model + rejects dpo; explicit `TypeError("v0.53.0 flag must be bool")` from `_validate_v053_bool_fields`; explicit-null surfaces as `valid boolean` ValidationError); Part F save formats (`MERGE_SAVE_FORMATS` lowercase normalisation + rejection matrix; `TORCHAO_PTQ_SCHEMES` CASE-SENSITIVE — `int4weightonly` rejected; `validate_quant_config_path` 4096-boundary; `MergeSaveSpec` + `TorchAOPTQSpec` frozen + MappingProxyType immutability; `merge_4bit` + `export_torchao` deferred); Cross-cutting (full 5-field YAML round-trip + cardinality invariant + tautological-assert replaced with allowlist + idempotent re-validate + `get_gguf_spec` unknown raises + bool guards on backend/modality/quantization across every Part D validator). Test count: 154 (v0.53.0) |
-| test_v0570_part_a.py | v0.57.0 Part A `soup adapters diff` — `effective_rank` SVD entropy (identity / concentrated / empty / 1D / zero / bool-eps / non-finite eps); `compute_layer_diffs` (identical-zero / known-norm-5 / shape-mismatch-skipped / intersection-only / non-mapping / oversize cap); `render_report_json` + `render_report_markdown` (roundtrip / non-report TypeError / only-lists section); `LayerDiff` frozen via FrozenInstanceError; end-to-end `compute_adapter_diff` (safetensors fixture + outside-cwd reject + bool top_k + 1/201 boundary + missing safetensors + .bin rejection + POSIX symlink at weights file); CLI (table/json/markdown smoke + unknown --format + --output requires non-table + outside-cwd + atomic write); no-top-level-torch source-grep guard. Test count: 36 (v0.57.0 Part A) |
-| test_v0570_part_b.py | v0.57.0 Part B `soup adapters merge` — 4 strategies in pure numpy: `merge_linear` (average / weighted / intersection / shape-mismatch / single-rejected / 17-too-many / bool weight / negative / NaN / inf rejection / zero-sum / length-mismatch); `merge_ties` (density top-half / majority-sign election / **tied-sign-defaults-positive** review fix / density bounds / density=1.0 inclusive / bool-density); `merge_dare` (deterministic-by-seed / different-seeds-diverge / bool-seed / negative-seed / density=1 equals linear); `merge_svd` (no-rank equals linear / rank reduces / non-2d passthrough / rank clamp / invalid-rank); `merge_adapters` e2e linear + unknown-strategy + output-outside-cwd + **POSIX symlink-at-output-safetensors rejection**; `SUPPORTED_STRATEGIES` frozenset + `STRATEGY_ORDER` tuple invariants; `predict_merged_verdict` stub + non-report + canary_suite type; `MergeReport` FrozenInstanceError; CLI help / linear / unknown-strategy / invalid-weights / single-adapter rejection; no-top-level-torch source-grep. Test count: 43 (v0.57.0 Part B) |
-| test_v0570_part_c.py | v0.57.0 Part C `soup adapters blame` — `parse_budget` (60/60s/5m/2h / below-floor / above-cap / invalid-format / empty / bool / null-byte / non-string); `plan_blame` (happy / infeasible-budget / shard offsets / uneven split / empty dataset / outside-cwd-adapter / outside-cwd-dataset / invalid-layer / shard bounds / **bool=True AND False shards** / **bool budget_seconds**); `run_blame` stub raises NotImplementedError v0.57.1 + non-plan TypeError; `BlamePlan` + `BlameShardWork` FrozenInstanceError; CLI help + plan-only smoke + invalid-budget + live runner advisory exit-0. Test count: 33 (v0.57.0 Part C) |
+| test_v0530.py | v0.53.0 Quant Menu II — UD GGUFs + KV cache + NVFP4 + LF parity + save formats: GGUF (UD ladder 14 entries + IQ 12 + Apple/ARM 10 frozensets + non-overlap invariant + `validate_*` case-insensitive + rejection matrix + `is_advanced_gguf_format` union + `_LOWER_INDEX` O(1) lookup + MappingProxyType immutability + `validate_calibration_data_path` shape rejection + 4096-boundary + `export_advanced_gguf` v0.53.1 deferred stub); KV cache (`KV_CACHE_TYPES` frozenset + `validate_kv_cache_type` case + bool/null/oversize/non-string rejection + `requires_hopper` delegates to spec + `get_kv_cache_spec` frozen + schema fp8-on-mlx rejected with specific message + q8_0-on-mlx allowed); advanced precision (`fp8_attention` requires `quantization_aware='fp8'` BEFORE mlx-gate ordering + bool guards on every string param + schema rejects-without-fp8-qat; `nvfp4` mlx + vision rejection + bool guards; `unsloth_bnb_4bit` backend='unsloth' + quantization='4bit' rejection matrix; `apply_*` deferred); LF parity (`bnb_4bit_use_double_quant` rejects none/8bit/gptq parametrize; `llm_int8` rejects default-none + 4bit; `quantize_ref_model` happy on dpo/grpo/kto + rejects sft/pretrain; `quantize_reward_model` happy on ppo/reward_model + rejects dpo; explicit `TypeError("v0.53.0 flag must be bool")` from `_validate_v053_bool_fields`; explicit-null surfaces as `valid boolean` ValidationError); save formats (`MERGE_SAVE_FORMATS` lowercase normalisation + rejection matrix; `TORCHAO_PTQ_SCHEMES` CASE-SENSITIVE — `int4weightonly` rejected; `validate_quant_config_path` 4096-boundary; `MergeSaveSpec` + `TorchAOPTQSpec` frozen + MappingProxyType immutability; `merge_4bit` + `export_torchao` deferred); Cross-cutting (full 5-field YAML round-trip + cardinality invariant + tautological-assert replaced with allowlist + idempotent re-validate + `get_gguf_spec` unknown raises + bool guards on backend/modality/quantization across every advanced-precision validator). Test count: 154 (v0.53.0) |
+| test_v0570_part_a.py | v0.57.0 `soup adapters diff` — `effective_rank` SVD entropy (identity / concentrated / empty / 1D / zero / bool-eps / non-finite eps); `compute_layer_diffs` (identical-zero / known-norm-5 / shape-mismatch-skipped / intersection-only / non-mapping / oversize cap); `render_report_json` + `render_report_markdown` (roundtrip / non-report TypeError / only-lists section); `LayerDiff` frozen via FrozenInstanceError; end-to-end `compute_adapter_diff` (safetensors fixture + outside-cwd reject + bool top_k + 1/201 boundary + missing safetensors + .bin rejection + POSIX symlink at weights file); CLI (table/json/markdown smoke + unknown --format + --output requires non-table + outside-cwd + atomic write); no-top-level-torch source-grep guard. Test count: 36 (v0.57.0) |
+| test_v0570_part_b.py | v0.57.0 `soup adapters merge` — 4 strategies in pure numpy: `merge_linear` (average / weighted / intersection / shape-mismatch / single-rejected / 17-too-many / bool weight / negative / NaN / inf rejection / zero-sum / length-mismatch); `merge_ties` (density top-half / majority-sign election / **tied-sign-defaults-positive** review fix / density bounds / density=1.0 inclusive / bool-density); `merge_dare` (deterministic-by-seed / different-seeds-diverge / bool-seed / negative-seed / density=1 equals linear); `merge_svd` (no-rank equals linear / rank reduces / non-2d passthrough / rank clamp / invalid-rank); `merge_adapters` e2e linear + unknown-strategy + output-outside-cwd + **POSIX symlink-at-output-safetensors rejection**; `SUPPORTED_STRATEGIES` frozenset + `STRATEGY_ORDER` tuple invariants; `predict_merged_verdict` stub + non-report + canary_suite type; `MergeReport` FrozenInstanceError; CLI help / linear / unknown-strategy / invalid-weights / single-adapter rejection; no-top-level-torch source-grep. Test count: 43 (v0.57.0) |
+| test_v0570_part_c.py | v0.57.0 `soup adapters blame` — `parse_budget` (60/60s/5m/2h / below-floor / above-cap / invalid-format / empty / bool / null-byte / non-string); `plan_blame` (happy / infeasible-budget / shard offsets / uneven split / empty dataset / outside-cwd-adapter / outside-cwd-dataset / invalid-layer / shard bounds / **bool=True AND False shards** / **bool budget_seconds**); `run_blame` stub raises NotImplementedError v0.57.1 + non-plan TypeError; `BlamePlan` + `BlameShardWork` FrozenInstanceError; CLI help + plan-only smoke + invalid-budget + live runner advisory exit-0. Test count: 33 (v0.57.0) |
 | test_v0580.py | v0.58.0 `soup loop` CLI-first data flywheel — `LoopState` frozen + 3 closed allowlist + bool/null/oversize/non-string reject; `with_status` / `bumped` (unknown counter + bool delta rejected); state file I/O atomic + cwd containment + POSIX symlink rejection + 1 MiB cap + forward-compat unknown-field-dropping; `init_state` refuses overwrite without `--force`; `CanaryPolicy` (cross-field, NaN/Inf, frozen); deterministic SHA-256 routing (stable-only / 0% / 100% / determinism / 25% split approximate / empty key / NUL key / non-policy); `rollback` (clears canary / route-after-rollback all-stable / reason / non-policy); `BucketStats` (record + verdict OK/MAJOR/UNKNOWN + bounds + invalid bucket + snapshot MappingProxyType); `parse_budget_string` (5 happy + empty/garbage/NUL/negative/overflow); `check_budget` (within/budget-blocked/daily-cap-blocked/None=unlimited + negative/NaN/bool guards); `reset_daily_counter_if_new_day` (same-day/new-day/None-prior/negative); `IterationRecord` frozen + verdict allowlist + path-separator id; iteration write/read roundtrip + outside-cwd + missing + sorted listing + non-record + invalid manifest; `new_iteration_id` unique × 20; `run_once` (defaults + budget-skip + custom callbacks + invalid types); `WatchConfig` (defaults + bad poll_interval/max_iterations/callable); `watch` finite-runs + on_iteration flips status to stop; `maybe_rollback` (OK no-op / MAJOR clears / UNKNOWN no-op / non-policy / non-str); CLI smoke (loop --help / init / refuse overwrite / --force / invalid budget / status without init / status / pause+resume cycle / pause-stopped / resume-running / watch --max-iterations / --foreground/--detach mutex / canary / canary invalid traffic / canary same-as-stable / replay list-empty / replay show / replay unknown); source-grep (cli.py registers loop / `__init__.py` == 0.58.0 / no top-level torch in 5 loop modules / typer.Typer). Review-fix coverage: 1 CRITICAL + 3 HIGH + 3 MEDIUM + 1 LOW. Test count: 160 (188 pass + 1 POSIX-skipped). (v0.58.0) |
-| test_v0570_part_d.py | v0.57.0 Part D `soup adapters branch/checkout/branches` — `create_branch` happy + with-dataset + invalid-name + null-byte name + bool name + config-outside-cwd + missing-config + empty-base-model + null-byte base + **bool base_model rejected** + oversize 1MiB config + atomic write; `list_branches` empty + sorted; `load_branch` roundtrip + missing + invalid-name + **POSIX symlink rejection**; `delete_branch` true-when-present + false-missing + **POSIX symlink rejection** + traversal rejection; `write_checkout` writes target + drift detection + outside-cwd reject + non-branch + missing source; `Branch` FrozenInstanceError; SOUP_BRANCHES_DIR env override (valid-honoured + null-byte falls-back via mocked os.environ.get + CRLF falls-back); CLI smoke create / list / checkout / invalid-name / missing-config / missing-branch / empty-list. Test count: 37 (v0.57.0 Part D, 4 POSIX-skipped on Windows) |
+| test_v0570_part_d.py | v0.57.0 `soup adapters branch/checkout/branches` — `create_branch` happy + with-dataset + invalid-name + null-byte name + bool name + config-outside-cwd + missing-config + empty-base-model + null-byte base + **bool base_model rejected** + oversize 1MiB config + atomic write; `list_branches` empty + sorted; `load_branch` roundtrip + missing + invalid-name + **POSIX symlink rejection**; `delete_branch` true-when-present + false-missing + **POSIX symlink rejection** + traversal rejection; `write_checkout` writes target + drift detection + outside-cwd reject + non-branch + missing source; `Branch` FrozenInstanceError; SOUP_BRANCHES_DIR env override (valid-honoured + null-byte falls-back via mocked os.environ.get + CRLF falls-back); CLI smoke create / list / checkout / invalid-name / missing-config / missing-branch / empty-list. Test count: 37 (v0.57.0, 4 POSIX-skipped on Windows) |
 | test_issue349_measured_fit.py | v0.73.1 #349 — measured stream VRAM fit: `training.stream_vram_probe` (bool, default false, sft-only) decides layer-streaming VRAM pre-flight by MEASURING a real forward+backward at the configured shape instead of using the fitted formula; justified on RTX 3050 where the formula under-predicts 0.934x at seq 5120 / 0.787x at seq 6144. Config validation + schema bounds (task gate sft, rejects stream_layers=false). Plus: `training.batch_size` field_validator now rejects 0 and negative values (was accepting `-4` before). Test count: 33 (v0.73.1) |
 | test_issue372_no_reexec_hint.py | #372 — `--no-reexec` hint is derived from the same argv as auto-reexec (`build_train_reexec_argv` + `hint_argv_from_reexec`); tokenised printed command is a superset of user flags parametrised over `--fsdp` / `--deepspeed` / `--config` / `--trust-remote-code`; control with no extra flags still prints a valid minimal command; CLI smoke does not skip. |
 | test_issue392_mlx_target_keys.py | v0.73.1 #392 — the MLX `adapter_config.json` serialised `target_modules` UNRESOLVED, so a default MLX adapter loaded as a silent no-op. `_apply_lora` resolved `auto` into a LOCAL variable and trained the resolved modules; the writer serialised the raw config value, so the file carried `{"keys": ["auto"]}`, `linear_to_lora_layers` matched no module and `load_weights(strict=False)` dropped all 96 LoRA tensors WITHOUT A WORD — generation with the adapter bit-identical to the base. `auto` is the schema DEFAULT, so this was every MLX run that did not name its modules by hand, and `adapter_config.json` exists specifically to promise the output dir loads with `mlx_lm.load(..., adapter_path=...)`. Reported by @armanbot-jpg with the control that isolates it: hand-editing `keys` makes the very same `adapters.safetensors` produce the tuned behaviour, i.e. the weights were fine and the config was not. **Third time in this project a healthy loss curve shipped a dead artifact** (v0.72.0's `.inner.` keys, #362's full-fine-tune adapter, this) — the loss curve structurally cannot see it, only the artifact can. Repaired as ONE shared `resolve_mlx_target_keys()` rather than teaching the writer to resolve too, since two copies of "which modules did we train?" is how they drifted. 10 red before, 10 green after; two are controls (the resolved value must never contain `auto` whatever the input, and rank/scale/dropout must still come from the config, so a repair that hardcoded the block would fail rather than pass). (v0.73.1) |
 | test_issue441_coerce_token_ids.py | #441 — `coerce_token_ids` is the public shared helper (`data_doctor` must import that name, not `_coerce_token_ids`, and must not define its own copy); duck-typed mapping (has `get`, not a `collections.abc.Mapping`) returns `[1, 2, 3]` rather than raising `input_ids[0]='input_ids'`; the same object carrying `assistant_masks` takes the mask path (labels keep only the masked token — fails if only the coerce gate is widened); missing `input_ids` still raises the existing error; both call sites share `_is_mapping_like` (`isinstance(out, Mapping)` lives in exactly one place). |
 | test_v07302.py | v0.73.2 `soup ship` leg-2 suite enhancements + noise floor: `--noise-floor N` (N in [2,10], metric-mode-only) re-runs base N times, per-axis floor = max-min spread, gates each at max(threshold, floor) — `VramFit.noise_floor_block` frozen + `measure_base_spreads` (base-only, N forward passes) + `compute_noise_floors` aggregation; evidence JSON schema extended with optional `noise_floor` block (runs + floors dict); 8th bundled suite `mini_over_refusal` (inverse safety gate, benign-not-refused) + DEFAULT_GENERAL_SUITE count 8; scorer fixes: `extract_mcq_letter` now understands `\boxed{C}` single-letter format, `tool_call_name_match` tolerates missing outer `{"function": ...}` envelope (carries BOTH name + arguments), `score_bundled_suite` raises TypeError for non-callable generator (was silently returning 0.0); baseline scale warning for the three affected suites (mini_mmlu / mini_common_sense / mini_tool_call); MCP `ship_evidence` tool honours noise_floor in evidence schema. (v0.73.2) |
-| test_v0560.py | v0.56.0 `soup diagnose` post-training failure-mode report card — Part A 6 probes (forgetting Δ-accuracy + tolerance band; refusal advbench/xstest delta + `_MAX_REFUSAL_SCAN=8192` cap; format JSON/regex/tool_call with ReDoS probe + `_VALID_KINDS` frozenset; mode_collapse pairwise n-gram Jaccard over K completions; memorization training-prefix echo via `split_prefix`; contamination v0.47 ngram-overlap reuse + combined-complexity cap N×M>1e9); Part B `FailureReport` + `FailureScore` frozen dataclasses with OK/MINOR/MAJOR taxonomy (≥0.85/≥0.60 thresholds) + `compose_report` / `build_report` SDK + atomic `write_report` (realpath containment + symlink reject) + `render_badge_svg` HTML-escaped 6-cell SVG + CLI smoke (--evidence/--output/--badge/--attach-to-registry); Part C `diagnose_report` artifact kind + `soup train --diagnose-gate` MAJOR-rejection helper. Review-fix coverage: atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), 16 MiB evidence size cap, `extract_row_text` centralisation, `tokenize` delegates to `_eval_text`, extras null-byte sanitisation, source-grep regression guards. Test count: 123 (v0.56.0) |
-| test_v0540.py | v0.54.0 `soup advise` pre-flight decision — Part A Verdict engine (TASK_CATEGORIES + CHOICES allowlists; frozen Verdict / DatasetProfile / ROIEstimate; `classify_task` keyword + tool_calls + reasoning-trace signals + goal-steers; `compute_dataset_profile` shape + diversity + chosen/rejected + reasoning detection; `build_verdict` 5-branch rubric with `_MIN_ROWS_FOR_GRPO=500`; `load_advise_dataset` cwd-containment + symlink reject + BOM strip + malformed-JSON reject); Part B Probe runner (`synth_probe_baselines` + `synth_probe_lora_delta` heuristic stubs with forward-compat `model`/`device`/`lr`/`timeout_seconds` kwargs; `format_verdict_rubric` + `next_command_for` handoff); Part C Cross-project learning (`record_verdict` + `load_history` + `_append_with_lock` cross-process fcntl/msvcrt locking; `~/.soup/advise_history.jsonl` + sidecar `.lock` on Windows; `history_path` env override containment; per-line 64 KB cap on history reads); CLI smoke (run / explain / compare subcommands + `_rewrite_advise_argv` scoped to argv[1]); review-fix coverage (atomic scratch write + symlink reject on read; concurrent 8-thread record stress; 49↔50 / 499↔500 / 4096↔4097 boundary). Test count: 136 (v0.54.0) |
-| test_v0510.py | v0.51.0 Model Catalog Expansion + Alternative Model Hubs: Part E hubs.py (`SUPPORTED_HUBS` + `validate_hub_name` + `validate_hub_endpoint` SSRF parity / CRLF rejection / IPv6 mapped private rejected / IPv6 loopback ok / control chars; `resolve_endpoint` env-var override; `default_endpoint` + `endpoint_env_var` + `required_hub_package` + `is_hf` with bool guards; MappingProxyType immutability); TrainingConfig `hub` field (default + Literal accept + None reject + case-insensitive normalisation + YAML round-trip) + SoupConfig `_validate_hub_supported` (mlx + non-hf rejected; mlx + hf accepted; modelers + transformers accepted); Part D MULTIPACK_ARCHITECTURES extension (20 new arches parametrize + legacy preserved + exact count=38 + frozenset immutability); Parts A/B/C 26 new recipes (parametrize over every name × {get_recipe / RecipeMeta / SoupConfig load / yaml.safe_load / model id no null/whitespace/empty parts / max_length bounds / GRPO required fields}); baichuan-sft uses `hub: modelscope`; total recipe count >= 105 (v0.51.0) |
+| test_v0560.py | v0.56.0 `soup diagnose` post-training failure-mode report card — 6 probes (forgetting Δ-accuracy + tolerance band; refusal advbench/xstest delta + `_MAX_REFUSAL_SCAN=8192` cap; format JSON/regex/tool_call with ReDoS probe + `_VALID_KINDS` frozenset; mode_collapse pairwise n-gram Jaccard over K completions; memorization training-prefix echo via `split_prefix`; contamination v0.47 ngram-overlap reuse + combined-complexity cap N×M>1e9); `FailureReport` + `FailureScore` frozen dataclasses with OK/MINOR/MAJOR taxonomy (≥0.85/≥0.60 thresholds) + `compose_report` / `build_report` SDK + atomic `write_report` (realpath containment + symlink reject) + `render_badge_svg` HTML-escaped 6-cell SVG + CLI smoke (--evidence/--output/--badge/--attach-to-registry); `diagnose_report` artifact kind + `soup train --diagnose-gate` MAJOR-rejection helper. Review-fix coverage: atomic+TOCTOU-safe badge write, typer.Exit (not sys.exit), 16 MiB evidence size cap, `extract_row_text` centralisation, `tokenize` delegates to `_eval_text`, extras null-byte sanitisation, source-grep regression guards. Test count: 123 (v0.56.0) |
+| test_v0540.py | v0.54.0 `soup advise` pre-flight decision — Verdict engine (TASK_CATEGORIES + CHOICES allowlists; frozen Verdict / DatasetProfile / ROIEstimate; `classify_task` keyword + tool_calls + reasoning-trace signals + goal-steers; `compute_dataset_profile` shape + diversity + chosen/rejected + reasoning detection; `build_verdict` 5-branch rubric with `_MIN_ROWS_FOR_GRPO=500`; `load_advise_dataset` cwd-containment + symlink reject + BOM strip + malformed-JSON reject); Probe runner (`synth_probe_baselines` + `synth_probe_lora_delta` heuristic stubs with forward-compat `model`/`device`/`lr`/`timeout_seconds` kwargs; `format_verdict_rubric` + `next_command_for` handoff); Cross-project learning (`record_verdict` + `load_history` + `_append_with_lock` cross-process fcntl/msvcrt locking; `~/.soup/advise_history.jsonl` + sidecar `.lock` on Windows; `history_path` env override containment; per-line 64 KB cap on history reads); CLI smoke (run / explain / compare subcommands + `_rewrite_advise_argv` scoped to argv[1]); review-fix coverage (atomic scratch write + symlink reject on read; concurrent 8-thread record stress; 49↔50 / 499↔500 / 4096↔4097 boundary). Test count: 136 (v0.54.0) |
+| test_v0510.py | v0.51.0 Model Catalog Expansion + Alternative Model Hubs: hubs.py (`SUPPORTED_HUBS` + `validate_hub_name` + `validate_hub_endpoint` SSRF parity / CRLF rejection / IPv6 mapped private rejected / IPv6 loopback ok / control chars; `resolve_endpoint` env-var override; `default_endpoint` + `endpoint_env_var` + `required_hub_package` + `is_hf` with bool guards; MappingProxyType immutability); TrainingConfig `hub` field (default + Literal accept + None reject + case-insensitive normalisation + YAML round-trip) + SoupConfig `_validate_hub_supported` (mlx + non-hf rejected; mlx + hf accepted; modelers + transformers accepted); MULTIPACK_ARCHITECTURES extension (20 new arches parametrize + legacy preserved + exact count=38 + frozenset immutability); 26 new recipes (parametrize over every name × {get_recipe / RecipeMeta / SoupConfig load / yaml.safe_load / model id no null/whitespace/empty parts / max_length bounds / GRPO required fields}); baichuan-sft uses `hub: modelscope`; total recipe count >= 105 (v0.51.0) |
 
 (Note: the test-file table above is a representative subset, not exhaustive — every `tests/test_*.py` file maps to a feature area or release.)
 

{soup_cli-0.76.0 → soup_cli-0.76.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: soup-cli
-Version: 0.76.0
+Version: 0.76.1
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -25,7 +25,7 @@ Requires-Dist: plotext>=5.2.0
 Requires-Dist: pydantic>=2.0.0
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: rich>=13.0.0
-Requires-Dist: typer<0.21.0,>=0.9.0
+Requires-Dist: typer<0.21.0,>=0.15.4
 Provides-Extra: aider
 Requires-Dist: aider-chat<0.87,>=0.86.2; extra == 'aider'
 Provides-Extra: all
@@ -223,11 +223,14 @@ soup train
 
 **Fine-tune an 8B model on a 4 GB laptop GPU.** Layer streaming keeps the frozen base out of
 VRAM and feeds it to the GPU one decoder layer at a time. Measured on an RTX 3050 Laptop 4 GB:
-Llama-3.1-8B-Instruct + NF4 at **119.6 tok/s, 3.32 GB peak** — bit-exact against a normal
-resident run, and reproduced independently on an H100 at 113.00 tok/s in the same 3.32 GB.
-(Both figures were measured on v0.72.2, before the v0.73.0 correctness repair that cost
-−4.8% at 32B; neither has been re-run on a 4 GB card since — re-measurement pending in
-issue [#361](https://github.com/MakazhanAlpamys/Soup/issues/361).) Opt-in (`stream_layers: true`)
+Llama-3.1-8B-Instruct + NF4 at **119.6 tok/s at 952 MHz, 3.32 GB peak** (pre-repair) — bit-exact
+against a normal resident run, and reproduced independently on an H100 at 113.00 tok/s in the same
+3.32 GB. (Both figures were measured on v0.72.2, before the v0.73.0 correctness repair that cost
+−4.8% at 32B. The post-repair run, on another RTX 3050 Laptop 4 GB (@umran666, #1103), reads
+**208.6 tok/s at 1935–1957 MHz in a 2.40 GB peak**. That is 59% of its same-session GEMM ceiling
+against 68%: about 14% slower per unit clock, so the raw 1.74x gain is the clock, not a speedup.
+Large-layer streaming also changed the memory pattern between the two rows, so even the fraction
+is not like-for-like — [the record](benchmarks/gate-v0.72.2-nf4.md).) Opt-in (`stream_layers: true`)
 and still BETA —
 [how it works](docs/performance-and-quantization.md#layer-streaming-beta-v0720-nf4-v0722-disk--wider-archs-v0723-preference-losses-v0724) ·
 [all measurements](benchmarks/) · [paper](https://doi.org/10.5281/zenodo.21771064) ·
@@ -235,8 +238,8 @@ and still BETA —
 4 GB, then asserts a streamed model is bit-identical to a normal one)
 
 <p align="center">
-  <a href="https://youtu.be/T1LCErE943E"><img src="docs/assets/layer-streaming.gif" alt="soup train pre-flight for Llama-3.1-8B on a 4 GB card: a 3.60 GB base store pinned in RAM across 32 layers and two 113 MB VRAM buffers, then a measured peak of 3.32 GB at 119.6 tok/s, stopping short of the 4 GB line (measured on v0.72.2, before the #331 repair; re-measurement pending in issue #361)"></a><br>
-  <sub>Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512 on an RTX 3050 Laptop 4 GB — <b>3.32 GB peak, 119.6 tok/s</b> (measured on v0.72.2, before the #331 repair; re-measurement pending in issue #361). <a href="https://youtu.be/T1LCErE943E">Full video (90s)</a></sub>
+  <a href="https://youtu.be/T1LCErE943E"><img src="docs/assets/layer-streaming.gif" alt="soup train pre-flight for Llama-3.1-8B on a 4 GB card: a 3.60 GB base store pinned in RAM across 32 layers and two 113 MB VRAM buffers, then a measured peak of 3.32 GB at 119.6 tok/s, stopping short of the 4 GB line (pre-repair row, measured on v0.72.2 at 952 MHz; the post-#331 row is in the caption below)"></a><br>
+  <sub>Llama-3.1-8B-Instruct + NF4, LoRA, batch 1, seq 512 on an RTX 3050 Laptop 4 GB — <b>3.32 GB peak, 119.6 tok/s</b> at 952 MHz (pre-repair, v0.72.2; post-#331: 208.6 tok/s at 1935 MHz in 2.40 GB, not directly comparable, @umran666, #1103). <a href="https://youtu.be/T1LCErE943E">Full video (90s)</a></sub>
 </p>
 
 ## Why Soup?
@@ -251,53 +254,36 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.76.0 — settings Soup accepted and never applied now stop the load.** Options that were
-validated and documented, and read by nothing on the path they claimed to configure, are now
-refused at load, naming the setting and what to change; the rest warn and name the release
-that will refuse them. A config that loaded before can stop loading: run
-`soup doctor --config soup.yaml` first.
-
-- **Breaking: a setting that never took effect is refused at load.** Among them
-  `training.quantization_aware: true` (it never ran quantization-aware training),
-  `training.use_longlora`, `training.ebft_variant`, `training.gdpo_variant`,
-  `training.rope_scaling_type` where no setup reads it, `freeze_layers` / `freeze_ratio`
-  outside the text `sft` / `tts` path on the transformers backend, `auto_mixed_precision`
-  outside `sft` / `tts`, and `backend: unsloth` on seven further tasks that have no unsloth
-  setup. Each message names the field and what to change. `lr_groups`, `early_stop_patience`,
-  `forgetting_eval_steps` and other unread knobs warn at load now and are refused from v0.77.
-- **Validation loss is now evaluated.** `data.val_split` held rows out of training on every
-  run, and no trainer on the transformers backend evaluated them. Every task that gets a
-  split now evaluates it at the end of each epoch, or every N steps with `training.eval_steps`,
-  and records the val loss; runs take a few percent longer. **Breaking for `grpo`, `ppo` and
-  `online_dpo`:** by default they no longer hold rows out (`data.val_split` is ignored with
-  a note), so every row trains.
-- **Breaking: every non-standard `grpo_variant` applies the `grpo_beta` KL penalty.** `gspo`,
-  `dapo`, `dr_grpo`, `bnpo`, `two_sided` and `rft` validated `grpo_beta` and dropped it, so
-  existing variant configs will not reproduce prior runs. `grpo_beta: 0` now runs KL-free and
-  skips the reference forward pass.
-- **Layer streaming reads its disk tier on a background thread, through direct I/O, and can
-  stripe it across NVMe drives.** On a 36 GB 70B-shaped NF4 store larger than RAM (one RTX 5070
-  Laptop 8 GB box), a cold step went from 110.5 s on the synchronous read to 16.2-17.3 s, and
-  from 17.4 s to 9.97 s across two drives with `SOUP_LAYER_STREAM_STRIPE_DIRS` (a sustained-load
-  probe: two rounds, 1.28x and 1.41x, no formal verdict;
-  [record](benchmarks/probe-rtx5070-two-drive-sustained.md)). A store that fits RAM is still
-  faster on the RAM tier, which `stream_source: auto` picks. Opt-in (`stream_layers: true`) and
-  still BETA; records: [direct I/O](benchmarks/gate-974-disk-tier-direct-io.md),
-  [striping](benchmarks/gate-two-drive-striping.md).
-- **New commands and options.** `soup adapters check` scans a LoRA adapter for silent failure;
-  `soup bench train` trains a fixed number of steps and exits 1 when the model was not training
-  (it covers `task: sft` on the transformers backend; `soup bench` is now a group, and
-  `soup bench infer` is the old inference benchmark);
-  `soup data clean` sanitizes and repairs a dataset without touching the input file;
-  `soup recipes verify` checks on the meta device, with no weights downloaded, that every shipped
-  recipe can attach a LoRA adapter; `soup rewind` names the dataset rows behind a loss spike.
-  `soup infer` gains opt-in `--batch-size` and an experimental `--cuda-graphs` (PyTorch 2.14 or
-  newer; on one RTX 5070 Laptop, unpinned, Qwen2.5-1.5B in FP16, the slowest run with it decoded
-  2.9x faster than the fastest without). `soup ab` runs a two-sided, variance-robust test, and
-  `soup adapters sign --backend sigstore` and `soup attest emit --sign sigstore` sign with
-  Sigstore keyless OIDC (`pip install "soup-cli[sigstore]"`).
-- **Deprecated:** `soup export --format awq` and `--format gptq` will be removed in v0.77; both
-  upstream projects are archived and neither extra installs next to `[train]`.
+**v0.76.1 — a patch: the bundled DPO examples train again, and every YAML file Soup reads has
+the same limits.** Everything in v0.76.0 is unchanged. That release refuses settings Soup
+accepted and never applied, so a config that loaded before can stop loading: coming from
+0.75.x, run `soup doctor --config soup.yaml` first. Its feature list is in the
+[v0.76.0 release notes](https://github.com/MakazhanAlpamys/Soup/releases/tag/v0.76.0). The
+settings that only warn at load are refused from v0.77, which also removes
+`soup export --format awq` and `--format gptq`.
+
+- **The bundled DPO examples train again.** `task: dpo`, `ipo`, `bco` and `preference` failed
+  inside trl, after the model had loaded, on `examples/data/dpo_sample.jsonl`,
+  `examples/data/chat_preferences.jsonl` and any dataset written the same way (a string
+  `prompt` beside `chosen` / `rejected` message lists), with an error that changed with
+  `PYTHONHASHSEED`. Every preference row now reaches the trainer in one layout, and a row that
+  cannot be written that way is dropped at load with a reason that names the field.
+- **One YAML loader, with limits, for every file Soup reads.** `soup.yaml` and the other YAML
+  readers share a nesting limit (100 levels) and an expansion limit, and each keeps its own
+  size cap (1 MiB for a `soup.yaml`). A file over a limit, unreadable or not valid YAML is
+  refused with one line naming the file and the problem. The audit log masks credential
+  options by name.
+- **The Typer floor is `typer>=0.15.4`.** `typer>=0.9.0` was declared while no command could
+  start below Typer 0.12.4, and 0.12.4 - 0.15.3 started only next to Click 8.1 or older. An
+  environment with an older Typer now has it upgraded, or the conflict named, at install time;
+  `soup doctor` reports it. A fresh install is unchanged, and one CI cell now starts the CLI on
+  the declared Typer and Click floors.
+- **A Hub dataset id with a dot loads from the Hub.** `data.train: teknium/OpenHermes-2.5`
+  failed with `Data file not found`: the loader read the dot as a file extension.
+- **Docs.** The README and the performance docs state the 8B NF4 figure with its SM clock
+  (119.6 tok/s at 952 MHz) beside the post-#331 row (208.6 tok/s at 1935-1957 MHz,
+  [#1207](https://github.com/MakazhanAlpamys/Soup/issues/1207) tracks the preprint
+  amendment), and the What's New of the four translations describes v0.76.0.
 
 > Python **3.10–3.12** only. On 3.13+, pip used to resolve untested PyTorch wheels that
 > crash in the native extension before Soup runs at all.
@@ -616,8 +602,12 @@ published**, which is also the shortest way to describe what the paper is for:
   the replication survives in a weaker form — the constraint is common to both machines and is
   not the GPU's compute.
 - **Replication on hardware nothing like the original** (added in v2): 119.6 tok/s on the RTX
-  3050 against a median 113.00 on an H100, at the same 3.32 GB peak. Both pre-date the #331
-  repair; the 4 GB re-measurement is pending in issue #361.
+  3050 (at 952 MHz) against a median 113.00 on an H100, at the same 3.32 GB peak. Both pre-date
+  the #331 repair and are the figures the paper reports. The post-#331 row (@umran666, #1103) is
+  208.6 tok/s at 1935–1957 MHz in a 2.40 GB peak, 59% against 68% of the same-session GEMM
+  ceiling: the raw 1.74x gain is the clock, not a speedup, and the memory pattern changed between
+  the rows, so even the fraction is not like-for-like
+  ([the record](benchmarks/gate-v0.72.2-nf4.md)).
 - **A silent wrong-gradient defect, found and repaired.** On NF4 above ~165 MiB per layer the
   forward stayed bit-exact and the loss curve looked healthy while the gradients were wrong. The
   cause is named in the upstream library and reported there; the repair is gated against controls