soup-cli 0.71.38__tar.gz → 0.71.39__tar.gz

{soup_cli-0.71.38 → soup_cli-0.71.39}/CHANGELOG.md

@@ -12,6 +12,53 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.39] - 2026-07-19
+
+**"CI for weights, not prompts": close the evidence loop.** `soup ship`'s verdict is
+now something Soup can emit, commit, review, and bind to the exact model that
+produced it — turning the `soup ci init` gate from "edit two numbers in a JSON file"
+into a reproducible, provenance-bound check that renders on every PR.
+
+### Added
+
+- **`soup ship --emit-evidence <path>`** — re-serialises the verdict into the
+  `--evidence` INPUT schema, so a run's output is replayable as input: feeding it
+  back through `--evidence` (same `--forgetting-threshold`) reproduces an identical
+  verdict. Output is finally input.
+- **`ShipConfig` under `eval.ship` in `soup.yaml` + `soup ship --config soup.yaml`** —
+  commit the gate policy (`task_eval` / `task_mode` / `general_suite` /
+  `forgetting_threshold` / `judge_model` / `baseline`) so the verdict is reviewable in
+  a PR diff and reproducible. An explicit CLI flag always wins (CLI > config > default).
+- **`soup ship --push owner/repo#N`** — post the verdict as a GitHub PR comment
+  (reuses the `soup adapters pr --push` `gh api` plumbing). Best-effort: a missing
+  token / `gh` failure warns but never flips the SHIP / DON'T-SHIP exit code.
+- **Evidence provenance + staleness gate.** With `--emit-evidence`, `--config` STAMPS
+  a `provenance` block (`config_sha` — a semantic, order-insensitive recipe hash — plus
+  `base_model` and a best-effort `data_sha`) onto the evidence. With `--evidence`
+  alone, `--config` GATES: it refuses (exit 3) evidence whose `config_sha` drifted
+  from the committed config, so a PR that changed `soup.yaml` but forgot to recompute
+  its evidence is caught. The gate policy (`eval.ship`) is EXCLUDED from the hash, so
+  tuning `forgetting_threshold` never falsely invalidates evidence about an unchanged
+  model.
+- **`soup ci init --config <soup.yaml>`** — binds the generated gate's `soup ship`
+  step to the committed config (provenance/staleness enforcement in CI).
+
+### Changed
+
+- **Exit-code note:** `soup ship --config` usage / staleness errors are exit `3`
+  (usage), preserving `0 = SHIP`, `2 = DON'T SHIP`, `1 = runtime` from v0.71.38.
+
+### Security
+
+- `provenance.config_sha` read from untrusted evidence is shape-validated as a hex
+  digest before being echoed (a raw value could smuggle terminal ESC bytes past
+  `rich.markup.escape`).
+- `provenance.data_sha` hashes the training file through an `O_NOFOLLOW` fd with a
+  symlink-rejecting containment check and an 8 GiB cap (was an unguarded `hash_file`).
+- `soup ci init`'s path validation now rejects `#` and the YAML 1.1 line breaks
+  (NEL / LS / PS), closing a plain-scalar comment-truncation of the generated `run:`
+  step (also hardens the pre-existing `--data` / `--suite` / `--evidence` args).
+
 ## [0.71.38] - 2026-07-17
 
 **`soup ship`'s regression leg now has teeth.** Leg 2 (the catastrophic-forgetting

{soup_cli-0.71.38 → soup_cli-0.71.39}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (316 files, 16330 tests)
+tests/                - Test suite (317 files, 16387 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.38 → soup_cli-0.71.39}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.38
+Version: 0.71.39
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,41 +193,39 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.38 — The gate grows teeth.** `soup ship` answers the one question that matters after a fine-tune — *did my task improve WITHOUT silently breaking everything else?* This release makes the "without breaking everything else" half real.
-
-- **A gate that actually catches regressions.** The regression leg used to be 15 trivia
-  prompts scored by raw substring — it credited `"B"` for "**B**erlin", `"3"` for "1**3**", and
-  had **zero** items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based
-  scorer over seven bundled, offline suites (MCQ · arithmetic · **tool-calling** · **JSON
-  validity** · **safety/refusal**). A tune that WINS your task but quietly breaks tool-calling
-  now gets a **DON'T SHIP** — the old gate waved it through.
-- **Zero new dependencies, no network.** Every suite ships in the wheel and is scored by the
-  pure scorers Soup already has — no lm-eval, no download. `soup ci init`'s core-only gate keeps
-  working.
-- **Distinct exit codes for CI.** Usage errors moved off exit `2` to `3`, so `2` now means only
-  DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
-- **Breaking, on purpose.** The fixed scorer can change an existing run's verdict — because the
-  old one was reporting false negatives. Recompute any committed `--baseline`.
+**v0.71.39 — CI for weights, not prompts.** `soup ship`'s verdict is now something Soup can emit, commit, review, and bind to the exact model that produced it — so a fine-tuning gate renders on every PR instead of asking you to hand-edit a JSON file.
+
+- **Output is finally input.** `soup ship --emit-evidence ev.json` writes the scores in the
+  `--evidence` schema, so a run replays offline into an identical verdict — the round-trip the
+  old `--output` couldn't do.
+- **Commit the gate policy.** Put `eval.ship` in your `soup.yaml` (`forgetting_threshold`,
+  `general_suite`, `task_mode`, …) and `soup ship --config soup.yaml` reads it — the gate is now
+  reviewable in a PR diff. Explicit flags still win.
+- **Provenance-bound evidence.** `--config --emit-evidence` STAMPS a `config_sha` onto the
+  evidence; `--config --evidence` GATES on it — a PR that changed the recipe but forgot to
+  recompute its evidence is refused (exit 3). Tuning the threshold never falsely invalidates it.
+- **The verdict on your PR.** `soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card as
+  a GitHub comment; `soup ci init --config soup.yaml` wires the whole thing into CI.
 
 ```bash
-# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
-soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
-#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
+# producer (train job): stamp the scores with the config that produced them
+soup ship --evidence scores.json --config soup.yaml --emit-evidence ship_evidence.json
 
-soup ship --evidence scores.json    # offline verdict from pre-computed scores
+# gate (PR CI): refuse evidence that doesn't match the committed config
+soup ship --evidence ship_evidence.json --config soup.yaml --push owner/repo#42
+soup ci init --config soup.yaml   # writes .github/workflows/soup-gate.yml
 ```
 
 <details>
-<summary>Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)</summary>
+<summary>Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)</summary>
 
-`soup data dedup --semantic` dedups by meaning; `soup data topics` shows a coverage table;
-`soup data canary insert|check` proves whether a model memorized inserted secrets (exit 2 on a
-leak); `soup train --replay old.jsonl` interleaves old data so a new task doesn't erase it.
+`soup ship`'s regression leg became real: a fixed, extraction-based scorer over seven bundled,
+offline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that
+wins your task but quietly breaks tool-calling now gets a **DON'T SHIP**. Zero new deps.
 
 ```bash
-soup data topics train.jsonl
-soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
-soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
+soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
+#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
 ```
 
 </details>

{soup_cli-0.71.38 → soup_cli-0.71.39}/README.md

@@ -49,41 +49,39 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.38 — The gate grows teeth.** `soup ship` answers the one question that matters after a fine-tune — *did my task improve WITHOUT silently breaking everything else?* This release makes the "without breaking everything else" half real.
-
-- **A gate that actually catches regressions.** The regression leg used to be 15 trivia
-  prompts scored by raw substring — it credited `"B"` for "**B**erlin", `"3"` for "1**3**", and
-  had **zero** items for tool-calling, safety, or JSON. Now it's a fixed, extraction-based
-  scorer over seven bundled, offline suites (MCQ · arithmetic · **tool-calling** · **JSON
-  validity** · **safety/refusal**). A tune that WINS your task but quietly breaks tool-calling
-  now gets a **DON'T SHIP** — the old gate waved it through.
-- **Zero new dependencies, no network.** Every suite ships in the wheel and is scored by the
-  pure scorers Soup already has — no lm-eval, no download. `soup ci init`'s core-only gate keeps
-  working.
-- **Distinct exit codes for CI.** Usage errors moved off exit `2` to `3`, so `2` now means only
-  DON'T-SHIP — a typo'd flag is no longer indistinguishable from a caught regression.
-- **Breaking, on purpose.** The fixed scorer can change an existing run's verdict — because the
-  old one was reporting false negatives. Recompute any committed `--baseline`.
+**v0.71.39 — CI for weights, not prompts.** `soup ship`'s verdict is now something Soup can emit, commit, review, and bind to the exact model that produced it — so a fine-tuning gate renders on every PR instead of asking you to hand-edit a JSON file.
+
+- **Output is finally input.** `soup ship --emit-evidence ev.json` writes the scores in the
+  `--evidence` schema, so a run replays offline into an identical verdict — the round-trip the
+  old `--output` couldn't do.
+- **Commit the gate policy.** Put `eval.ship` in your `soup.yaml` (`forgetting_threshold`,
+  `general_suite`, `task_mode`, …) and `soup ship --config soup.yaml` reads it — the gate is now
+  reviewable in a PR diff. Explicit flags still win.
+- **Provenance-bound evidence.** `--config --emit-evidence` STAMPS a `config_sha` onto the
+  evidence; `--config --evidence` GATES on it — a PR that changed the recipe but forgot to
+  recompute its evidence is refused (exit 3). Tuning the threshold never falsely invalidates it.
+- **The verdict on your PR.** `soup ship --push owner/repo#N` posts the SHIP / DON'T-SHIP card as
+  a GitHub comment; `soup ci init --config soup.yaml` wires the whole thing into CI.
 
 ```bash
-# leg 1 (task win) AND leg 2 (no regression on the bundled suite) -> one verdict
-soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
-#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
+# producer (train job): stamp the scores with the config that produced them
+soup ship --evidence scores.json --config soup.yaml --emit-evidence ship_evidence.json
 
-soup ship --evidence scores.json    # offline verdict from pre-computed scores
+# gate (PR CI): refuse evidence that doesn't match the committed config
+soup ship --evidence ship_evidence.json --config soup.yaml --push owner/repo#42
+soup ci init --config soup.yaml   # writes .github/workflows/soup-gate.yml
 ```
 
 <details>
-<summary>Previous release — v0.71.36, Data Moat II (semantic dedup · topics · canaries · replay)</summary>
+<summary>Previous release — v0.71.38, The gate grows teeth (real leg-2 regression gate)</summary>
 
-`soup data dedup --semantic` dedups by meaning; `soup data topics` shows a coverage table;
-`soup data canary insert|check` proves whether a model memorized inserted secrets (exit 2 on a
-leak); `soup train --replay old.jsonl` interleaves old data so a new task doesn't erase it.
+`soup ship`'s regression leg became real: a fixed, extraction-based scorer over seven bundled,
+offline suites (MCQ · arithmetic · tool-calling · JSON validity · safety/refusal). A tune that
+wins your task but quietly breaks tool-calling now gets a **DON'T SHIP**. Zero new deps.
 
 ```bash
-soup data topics train.jsonl
-soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
-soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
+soup ship --base ./base --adapter ./my-lora --task-eval my_task.jsonl
+#   exit 0 = SHIP · 2 = DON'T SHIP · 3 = bad flags · 1 = runtime error
 ```
 
 </details>

{soup_cli-0.71.38 → soup_cli-0.71.39}/docs/commands.md

@@ -166,9 +166,12 @@ soup ship ... --task-mode judge_score --judge-model ollama://llama3.1  Leg-1 via
 soup ship ... --task-mode pairwise --judge-model ollama://llama3.1  Leg-1 via swap-debiased judge win-rate (base=0.5) (v0.71.31)
 soup ship ...  # leg-2 default = 7 bundled offline suites (MCQ/arithmetic + tool_call/format_json/safety, extraction scorer, ~40 items each) (v0.71.38)
 soup ship ... --general-suite mmlu,gsm8k --baseline base.json  lm-eval leg-2 override + recorded base scores
+soup ship ... --emit-evidence ev.json  Re-serialise the scores as replayable --evidence input (output-is-input, #312) (v0.71.39)
+soup ship ... --config soup.yaml  Read eval.ship gate defaults; --evidence GATES on provenance, --emit-evidence STAMPS it (v0.71.39)
+soup ship ... --push owner/repo#N  Post the verdict as a GitHub PR comment (best-effort; never flips the exit code) (v0.71.39)
 soup card <registry-id> -o MODELCARD.md       HF model card from a registry entry: training config, evals, hashes, lineage, artifacts (v0.71.35)
 soup push --model ./out --repo you/m --card <registry-id>  Upload that registry-driven card as the README (HF only) (v0.71.35)
-soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--branch main --python 3.11] [--force]  Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35)
+soup ci init [--data d.jsonl --suite s.yaml --evidence ev.json] [--config soup.yaml] [--branch main --python 3.11] [--force]  Write .github/workflows/soup-gate.yml: data validate -> expect -> ship gate on every PR (v0.71.35); --config binds the gate to a committed config so it refuses stale evidence (v0.71.39)
 soup mcp serve                                MCP server over stdio (drive Soup from Claude Code / Cursor / Cline; requires [mcp] extra) (v0.71.28)
 soup mcp serve --allow-mutating               Also expose plan-only train_start / export tools (never execute) (v0.71.28)
 soup shrink --model <id|path> --drop-ratio 0.25 --calib c.jsonl -o shrunk  Depth-prune least-important layer block + SHIP/DON'T-SHIP ppl verdict (exit 0/2/1) (v0.71.29)

{soup_cli-0.71.38 → soup_cli-0.71.39}/docs/evaluation.md

@@ -378,6 +378,36 @@ Exit codes (v0.71.38): **0 = SHIP · 2 = DON'T SHIP · 3 = usage/flag error · 1
 lives in `soup_cli.utils.ship_verdict` (`decide_ship` is a pure function — the whole truth table
 is CPU-testable); the bundled suites live in `soup_cli.eval.gate_suites`.
 
+### Closing the evidence loop (v0.71.39)
+
+The verdict is now emittable, committable, and provenance-bound so a fine-tuning gate runs on
+every PR instead of relying on a hand-edited JSON file.
+
+- **`--emit-evidence <path>`** re-serialises the scores into the `--evidence` INPUT schema, so a
+  run's output is replayable as input — feeding it back through `--evidence` (same threshold)
+  reproduces an identical verdict.
+- **`--config soup.yaml`** reads a committed `eval.ship` block for the gate defaults
+  (`task_eval` / `task_mode` / `general_suite` / `forgetting_threshold` / `judge_model` /
+  `baseline`); an explicit CLI flag always wins. This makes the gate reviewable in a PR diff.
+- **Provenance + staleness.** With `--emit-evidence`, `--config` STAMPS a `provenance` block
+  (`config_sha` — a semantic, order-insensitive recipe hash that EXCLUDES the `eval.ship` gate
+  policy, so tuning the threshold never invalidates evidence — plus `base_model` and a
+  best-effort `data_sha`). With `--evidence` alone, `--config` GATES: it refuses (exit 3)
+  evidence whose `config_sha` drifted from the committed config.
+- **`--push owner/repo#N`** posts the verdict as a GitHub PR comment (best-effort — a missing
+  token or `gh` failure warns but never flips the SHIP / DON'T-SHIP exit code).
+
+```bash
+# Producer (train job): compute or stamp scores, bound to the committed recipe
+soup ship --evidence scores.json --config soup.yaml --emit-evidence ship_evidence.json
+
+# Gate (PR CI): refuse evidence that doesn't match the committed config, comment the verdict
+soup ship --evidence ship_evidence.json --config soup.yaml --push owner/repo#42
+```
+
+`soup ci init --config soup.yaml` binds the generated workflow's ship step to the committed
+config, so the whole loop runs in CI (see [commands.md](commands.md)).
+
 
 ## NLG Evaluation Metrics (BLEU + ROUGE)
 

{soup_cli-0.71.38 → soup_cli-0.71.39}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.38"
+version = "0.71.39"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.38 → soup_cli-0.71.39}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.38"
+__version__ = "0.71.39"

{soup_cli-0.71.38 → soup_cli-0.71.39}/src/soup_cli/commands/ci.py

@@ -6,6 +6,8 @@
 
 from __future__ import annotations
 
+from typing import Optional
+
 import typer
 from rich.console import Console
 from rich.markup import escape
@@ -33,6 +35,10 @@ def init(
         "3.11", "--python", help="Python version for the CI runner (e.g. 3.11)"
     ),
     branch: str = typer.Option("main", "--branch", help="Branch the workflow triggers on"),
+    config: Optional[str] = typer.Option(
+        None, "--config",
+        help="Bind the ship gate to a committed soup.yaml (refuses stale evidence)",
+    ),
     output: str = typer.Option(
         ".github/workflows/soup-gate.yml", "-o", "--output", help="Workflow output path"
     ),
@@ -48,6 +54,7 @@ def init(
             branch=branch,
             output_path=output,
             overwrite=force,
+            config_path=config,
         )
     except (ValueError, TypeError, OSError) as exc:
         console.print(f"[red]{escape(str(exc))}[/]")