soup-cli 0.71.3__tar.gz → 0.71.5__tar.gz

{soup_cli-0.71.3 → soup_cli-0.71.5}/CHANGELOG.md

@@ -12,6 +12,96 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.5] - 2026-06-02
+
+### Added
+- **`soup eval against` now reads eval metrics** — `ExperimentTracker.get_metric_series`
+  falls back to the `eval_results` table when the metric is not a per-step
+  training column (`loss` / `lr` / `grad_norm` / `speed` / `gpu_mem`). So
+  `soup eval against <base> --candidate <run> --metric task_accuracy` returns a
+  real score series (benchmark scores live in `eval_results`, not `metrics`)
+  instead of "Empty series". Per-step columns still read from `metrics` — no
+  regression for existing callers.
+- **`soup advise` learns from past project outcomes** — `soup advise` now reads
+  this project's accepted-verdict history (`~/.soup/advise_history.jsonl`) and
+  biases the rubric: 3+ successful SFT precedents flip a marginal RAG call to
+  SFT; 3+ negative GRPO outcomes suppress GRPO in favour of SFT-on-traces; an
+  encouraged choice gets a small confidence nudge. Scoped per-project (one
+  project's record never biases another). No history → identical to before.
+- **Slack/Discord webhooks on four more commands** — `--slack-url` / `--discord-url`
+  (SSRF-hardened, loopback-only HTTP, RFC1918 rejected, never crashes the
+  command) now ship on `soup ingest`, `soup prune-prompt`, `soup ab` (fires only
+  on a `reject_h0` / `accept_h0` decision, not `continue`), and
+  `soup data active-sample` — not just `soup drift-alarm`. The validator + sender
+  moved to a shared `soup_cli/utils/webhooks.py`.
+- **Tokenizer-aware `soup prune-prompt`** — `--tokenizer <model_or_path>` detects
+  and strips the shared system-prompt prefix on **token** boundaries instead of
+  characters, so a multi-byte UTF-8 prefix can never be split mid-code-point.
+  Default (no `--tokenizer`) keeps the whitespace-character behaviour.
+- **Curriculum bucketing by loss percentile** — `DynamicCurriculumCallback` now
+  buckets samples by the percentile rank of the live loss (or perplexity)
+  signal within a rolling window when `data.curriculum_metric` is `loss` /
+  `perplexity`, so a consistently-hard sample is routed to the same difficulty
+  bucket across recomputes. `length` and warm-up still use round-robin.
+- **`--hub` on `soup data push` and `soup data forge`** — `soup data push
+  --hub modelscope|modelers` uploads a dataset via the matching SDK
+  (`repo_type=dataset`, commit message sanitised); `soup data forge --hub
+  <non-hf> --teacher owner/name` pre-fetches the teacher model from that hub
+  (and warns when the teacher is not a repo id so `--hub` is never silently
+  ignored). HF stays the default.
+
+### Notes
+- Live SaaS *pull* adapters for `soup ingest` (Langfuse / LangSmith / Helicone /
+  OpenPipe / OpenAI SDKs, issue #204) remain deferred: they need credentialed
+  vendor accounts with populated trace data to validate honestly. Tracked as an
+  open, `infra-blocked` (external-account) item. `soup ingest` continues to parse
+  the JSONL export you pull from your dashboard.
+
+## [0.71.4] - 2026-06-02
+
+### Added
+- **Live canary verdict for `soup adapters merge`** — `--canary <suite.json>`
+  scores the merged adapter against the first input and classifies
+  **OK / MINOR / MAJOR** using the Quant-Lobotomy taxonomy (drop <2% OK, <5%
+  MINOR, else MAJOR). `--strict-verdict` exits 2 on MAJOR. Pre-scored
+  `{"baseline_scores","candidate_scores"}` suites run with no model load; a
+  `{"tasks":[...]}` suite uses an injectable scorer. Replaces the v0.57 `UNKNOWN`
+  stub.
+- **Live evolutionary merge** — `soup adapters merge --strategy cmaes --eval
+  <suite> --budget <t>` now runs the full CMA-ES loop: each candidate is merged,
+  materialised, scored against the eval suite, and the best-weighted merge is
+  written to `--output`. Replaces the v0.67 plan-only stub.
+- **Publish an adapter PR to GitHub** — `soup adapters pr <title> --base-sha
+  <hex> --adapter <path> --push owner/repo#N` posts the rendered PR Markdown as a
+  GitHub PR comment via `gh api` (argv-list, body over JSON stdin; no shell).
+  Token resolves from `GITHUB_TOKEN` / `GH_TOKEN`.
+- **Pre-wired `soup loop` production stages** — `soup loop init --pre-wired` (or
+  `soup loop watch --pre-wired`) swaps the v0.58 no-op stage stubs for real
+  harvest (traces → preference pairs) → DPO train → eval-gate → canary-deploy
+  callables. `soup loop status` now shows the `pre_wired` flag.
+- **Loop iterations as Soup Cans + Registry lineage** — `soup loop watch
+  --pack-cans` packs each successful iteration as a v0.26 Soup Can and appends a
+  Registry entry (tag `loop-iter`), chaining a real lineage DAG across
+  iterations visible through `soup history`. `soup loop replay <id> --extract
+  <dir>` unpacks a recorded iteration.
+- **Branch pointers into the Registry** — `soup adapters branch <name>
+  --attach-to-registry <id>` links a branch snapshot to a Registry entry (shown
+  as a `branches` node in `soup history`); `soup adapters branch <name>
+  --from-registry <id>` derives a fresh snapshot's config + base from an entry.
+
+### Security
+- The backdoor-scan gate (v0.71.2 #192) and license-conflict gate (v0.60 Part E)
+  now run for **all** merge strategies, including `--strategy cmaes` (previously
+  bypassed because cmaes returned before the gates).
+- `soup loop` canary deploy restricts `SOUP_LOOP_SERVE_ENDPOINT` to loopback /
+  RFC1918-private hosts (a serve endpoint is the operator's own box/LAN), beyond
+  the general webhook SSRF policy which permits any HTTPS host.
+- `soup adapters pr --push` builds the `gh` child environment from an allowlist
+  so unrelated secrets (`HF_TOKEN` / `OPENAI_API_KEY` / …) never reach the
+  subprocess.
+- The canary-suite JSON read uses `O_NOFOLLOW` + `os.fstat` (size cap enforced on
+  the same fd) to close the symlink/size-cap TOCTOU window.
+
 ## [0.71.3] - 2026-06-01
 
 ### Added

{soup_cli-0.71.3 → soup_cli-0.71.5}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (273 files, 12342 tests)
+tests/                - Test suite (275 files, 12581 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.71.3 → soup_cli-0.71.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.3
+Version: 0.71.5
 Summary: Fine-tune LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -183,19 +183,22 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.3 — Compliance / annex / audit / energy.** The EU AI Act + carbon + audit surfaces go live:
-
-- **Energy & CO2 measurement** — `soup train --track-energy` measures the training window with an
-  offline carbon tracker (no network) and reports kWh / CO2, feeding them into the Annex doc.
-  Install with `pip install soup-cli[carbon]`.
-- **PDF compliance docs** — `soup train --annex-xi report.pdf` now renders a real PDF of the
-  EU AI Act Annex XI/XII (a `.md` path still gives markdown). The top crawled domains of your
-  training corpus are auto-listed. `pip install soup-cli[pdf]`.
-- **Per-command audit log** — every `soup` command appends one HIPAA/SOC2 record to
-  `~/.soup/audit.jsonl` (secrets redacted). Opt out with `--no-audit-log` or `SOUP_NO_AUDIT_LOG=1`.
-- **Signed Soup Cans + airgap receipts** — `soup can pack --attest statement.json` embeds in-toto
-  attestations into a can; `soup airgap-bundle --repro-receipt receipt.json` ships a reproducibility
-  receipt inside the offline bundle.
+**v0.71.5 — Ingest, data & prompt polish.** Sharper production-loop ergonomics:
+
+- **Alternative model hubs for data** — `soup data push --hub modelscope|modelers` uploads a
+  local JSONL to ModelScope / Modelers, and `soup data forge --hub … --teacher owner/name`
+  pre-fetches the teacher from that hub.
+- **Tokenizer-aware prompt pruning** — `soup prune-prompt --tokenizer <id-or-path>` finds the
+  shared *token* prefix and decodes only the remainder, so BPE multi-byte sequences never get
+  truncated mid-token the way char-slicing can.
+- **Webhooks everywhere** — `--slack-url` / `--discord-url` now work on `soup ingest`,
+  `soup prune-prompt`, `soup ab`, and `soup data active-sample` (same SSRF-hardened validator as
+  `soup drift-alarm`). The A/B harness only pings when the sequential test actually decides.
+- **Curriculum by difficulty percentile** — dynamic curriculum can bucket by `loss` /
+  `perplexity` percentile instead of length round-robin.
+- **Smarter pre-flight `advise`** — `soup advise` now nudges its confidence using your prior
+  verdicts for the same project, and `soup runs replay` can plot a benchmark-score curve, not
+  just the loss curve.
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.3 → soup_cli-0.71.5}/README.md

@@ -49,19 +49,22 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.3 — Compliance / annex / audit / energy.** The EU AI Act + carbon + audit surfaces go live:
-
-- **Energy & CO2 measurement** — `soup train --track-energy` measures the training window with an
-  offline carbon tracker (no network) and reports kWh / CO2, feeding them into the Annex doc.
-  Install with `pip install soup-cli[carbon]`.
-- **PDF compliance docs** — `soup train --annex-xi report.pdf` now renders a real PDF of the
-  EU AI Act Annex XI/XII (a `.md` path still gives markdown). The top crawled domains of your
-  training corpus are auto-listed. `pip install soup-cli[pdf]`.
-- **Per-command audit log** — every `soup` command appends one HIPAA/SOC2 record to
-  `~/.soup/audit.jsonl` (secrets redacted). Opt out with `--no-audit-log` or `SOUP_NO_AUDIT_LOG=1`.
-- **Signed Soup Cans + airgap receipts** — `soup can pack --attest statement.json` embeds in-toto
-  attestations into a can; `soup airgap-bundle --repro-receipt receipt.json` ships a reproducibility
-  receipt inside the offline bundle.
+**v0.71.5 — Ingest, data & prompt polish.** Sharper production-loop ergonomics:
+
+- **Alternative model hubs for data** — `soup data push --hub modelscope|modelers` uploads a
+  local JSONL to ModelScope / Modelers, and `soup data forge --hub … --teacher owner/name`
+  pre-fetches the teacher from that hub.
+- **Tokenizer-aware prompt pruning** — `soup prune-prompt --tokenizer <id-or-path>` finds the
+  shared *token* prefix and decodes only the remainder, so BPE multi-byte sequences never get
+  truncated mid-token the way char-slicing can.
+- **Webhooks everywhere** — `--slack-url` / `--discord-url` now work on `soup ingest`,
+  `soup prune-prompt`, `soup ab`, and `soup data active-sample` (same SSRF-hardened validator as
+  `soup drift-alarm`). The A/B harness only pings when the sequential test actually decides.
+- **Curriculum by difficulty percentile** — dynamic curriculum can bucket by `loss` /
+  `perplexity` percentile instead of length round-robin.
+- **Smarter pre-flight `advise`** — `soup advise` now nudges its confidence using your prior
+  verdicts for the same project, and `soup runs replay` can plot a benchmark-score curve, not
+  just the loss curve.
 
 Full history: [CHANGELOG.md](CHANGELOG.md) &middot; [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 

{soup_cli-0.71.3 → soup_cli-0.71.5}/docs/adapters-and-governance.md

@@ -34,6 +34,8 @@ v0.57 shipped `adapters diff / merge / blame / branch`. v0.67 finishes the lifec
 
 ```bash
 # 1. Evolutionary merge: search the simplex of merge weights via CMA-ES.
+#    v0.71.4 makes this a LIVE loop — each candidate is merged, scored against
+#    --eval, and the best blend is written to --output.
 soup adapters merge \
   adapter-finance/ adapter-medical/ adapter-legal/ \
   --strategy cmaes \
@@ -43,6 +45,12 @@ soup adapters merge \
   --max-generations 20 \
   --output merged/
 
+# 1b. One-shot merge with a live canary verdict (v0.71.4). Scores the merged
+#     adapter vs the first input → OK / MINOR / MAJOR; --strict-verdict exits 2
+#     on a MAJOR regression. A pre-scored canary suite needs no model load.
+soup adapters merge adapter-a/ adapter-b/ -o merged/ \
+  --canary evals/canary.json --strict-verdict
+
 # 2. Render the merge as a GitHub PR for review (eval deltas + sample diffs).
 soup adapters pr "merge: 3-domain blend" \
   --base-sha $(git rev-parse HEAD) \
@@ -52,6 +60,12 @@ soup adapters pr "merge: 3-domain blend" \
   --dataset-diff data/diff.txt \
   --format markdown -o pr.md
 
+# 2b. Post it straight to a GitHub PR comment (v0.71.4). Auth via GITHUB_TOKEN
+#     / GH_TOKEN; uses `gh api` under the hood (no secret leaks to the child).
+soup adapters pr "merge: 3-domain blend" \
+  --base-sha $(git rev-parse HEAD) --adapter merged/ \
+  --eval evals/deltas.json --push your-org/your-repo#42
+
 # 3. Lock a reproducible run state. Closure = sha(base + dataset + env).
 soup env lock                                     # v0.64 — capture env hash
 soup lock write \
@@ -93,6 +107,16 @@ soup loop status
 # Run the daemon (foreground)
 soup loop watch --poll-interval 300
 
+# Pre-wired production stages (v0.71.4): real traces → DPO → eval-gate → canary,
+# instead of the v0.58 no-op stubs. Opt in once via `loop init --pre-wired` or
+# per-run via --pre-wired. Pack each iteration as a shareable Soup Can with
+# Registry lineage via --pack-cans.
+soup loop init registry://abc12 --eval evals/lock.json --baseline registry://prod --pre-wired
+soup loop watch --pre-wired --pack-cans --poll-interval 300
+
+# Unpack a recorded iteration's Soup Can for what-if analysis (v0.71.4)
+soup loop replay iter-20260515T120000-abcdef01 --extract ./iter-dump
+
 # Background subprocess (writes PID, no shell)
 soup loop watch --detach
 
@@ -240,6 +264,14 @@ soup adapters blame ./run-v18 --dataset train.jsonl --layer q_proj.7 \
 soup adapters branch v18 --config soup.yaml --base meta-llama/Llama-3.1-8B \
   --dataset train.jsonl
 
+# Link a branch into the Registry lineage DAG (v0.71.4) — shows as a
+# `branches` node under the entry in `soup history`.
+soup adapters branch v18 --config soup.yaml --base meta-llama/Llama-3.1-8B \
+  --attach-to-registry reg_20260601_abc123
+
+# Or derive a fresh snapshot's config + base straight from a Registry entry (v0.71.4)
+soup adapters branch v18-from-reg --from-registry reg_20260601_abc123
+
 # Restore the snapshot's config (refuses if source SHA drifted)
 soup adapters checkout v18 --output soup.yaml
 
@@ -264,7 +296,9 @@ soup adapters branches
 - Branch pointers live under `~/.soup/branches/` (override via `SOUP_BRANCHES_DIR`, constrained to `$HOME` / `$CWD` / `$TMPDIR`).
 - `soup adapters checkout` SHA-checks the source config — refuses to restore when the source has drifted from the snapshot, so reproducibility never silently lies.
 
-**v0.66.0:** `soup adapters blame` is now LIVE — the v0.57 `NotImplementedError` stub (#171) is lifted via a DataInf-style influence-function approximation. Pass `--top-k 50` to control the reported top-influencer count; pass a real `probe_fn` (Python API) to feed real gradients, or use the default deterministic synthetic probe for offline planning. `MergeReport.verdict` remains the `UNKNOWN` stub (live canary eval in v0.57.1).
+**v0.66.0:** `soup adapters blame` is now LIVE — the v0.57 `NotImplementedError` stub (#171) is lifted via a DataInf-style influence-function approximation. Pass `--top-k 50` to control the reported top-influencer count; pass a real `probe_fn` (Python API) to feed real gradients, or use the default deterministic synthetic probe for offline planning.
+
+**v0.71.4:** the merge verdict is now LIVE — `soup adapters merge … --canary suite.json` lifts the `MergeReport.verdict` `UNKNOWN` stub. A pre-scored `{"baseline_scores","candidate_scores"}` suite classifies the blend OK / MINOR / MAJOR with no model load; a `{"tasks":[...]}` suite uses an injectable scorer. `--strict-verdict` exits 2 on MAJOR. The backdoor-scan and license-conflict gates now run for **every** strategy, including `cmaes`. `soup adapters branch <name> --from-registry <id>` / `--attach-to-registry <id>` link training-env snapshots into the Registry lineage DAG (shown as a `branches` node in `soup history`).
 
 
 ## Soup Cans (Shareable Recipes)

{soup_cli-0.71.3 → soup_cli-0.71.5}/docs/backends-and-ops.md

@@ -459,6 +459,11 @@ Every completed run also stores an estimated cost (`$` per run) computed from th
 captured GPU device name and duration. `soup runs show` renders `—` for CPU /
 MPS / unknown GPUs (no fabricated zeros).
 
+As of v0.71.5, the metric-series lookup that powers replay (`ExperimentTracker.get_metric_series`)
+transparently falls back to the `eval_results` table when a metric has no per-step
+rows — so you can plot a benchmark-score curve (e.g. `mmlu`, `gsm8k`) the same way
+you plot `loss`, without caring which table holds the series.
+
 ### Tracker integrations (--tracker mlflow / swanlab / trackio)
 
 ```bash

{soup_cli-0.71.3 → soup_cli-0.71.5}/docs/commands.md

@@ -90,10 +90,12 @@ soup data download user/ds --samples 1000    Stream first 1000 samples
 soup data register --name my-ds --path d.jsonl --format alpaca  Register dataset
 soup data unregister --name my-ds            Remove from registry
 soup data push --input d.jsonl --hf-dataset user/name  Upload local JSONL as HF dataset
+soup data push --input d.jsonl --hf-dataset u/n --hub modelscope|modelers  Upload to an alternative hub
 soup data registry                           List all registered datasets
 soup data demo                                List bundled demo JSONL fixtures
 soup data demo alpaca_demo --output ./d.jsonl Copy a bundled demo JSONL fixture
 soup data forge --docs ./docs --task sft --target-rows 1000  Synthetic data pipeline + provenance
+soup data forge --docs ./docs --hub modelscope --teacher owner/name  Pre-fetch the teacher from an alternative hub
 soup data score --input rows.jsonl            Composite quality scorecard (PII + toxicity + lang + edu)
 soup data decontaminate --input rows.jsonl --benchmarks mmlu,gsm8k  Drop benchmark-overlap rows
 soup data toxicity --input rows.jsonl -o tox.jsonl  Flag toxic rows (keyword baseline)
@@ -109,12 +111,12 @@ soup cost --config soup.yaml --gpu H100      Estimate training cost for specific
 soup adapters list ./output/                 Scan for LoRA adapters
 soup adapters info ./output/checkpoint-500/  Show adapter metadata
 soup adapters compare adapter1/ adapter2/    Compare two adapters
-soup loop init <model> --eval <s> --baseline <b>  Create .soup/loop.yaml (data flywheel)
-soup loop status                              Counters + status (traces / pairs / runs / shipped)
-soup loop watch [--detach] [--max-iter N]    Harvest → train → gate → deploy daemon
+soup loop init <model> --eval <s> --baseline <b> [--pre-wired]  Create .soup/loop.yaml (data flywheel; --pre-wired = real stages)
+soup loop status                              Counters + status + pre_wired flag
+soup loop watch [--detach] [--max-iter N] [--pre-wired] [--pack-cans]  Harvest → train → gate → deploy daemon (pre-wired stages + Soup Can packing)
 soup loop pause / soup loop resume           Atomic status flip
 soup loop canary <adapter> --traffic 5%      Promote canary + auto-rollback on MAJOR
-soup loop replay [<iter-id>]                 Replay a recorded iteration manifest
+soup loop replay [<iter-id>] [--extract <dir>]  Replay / unpack a recorded iteration manifest
 soup serve --model m --adapters chat=./c code=./d  Multi-adapter serving
 soup migrate --from llamafactory config.yaml  Import config from LLaMA-Factory
 soup migrate --from axolotl config.yml        Import config from Axolotl
@@ -137,7 +139,7 @@ soup can publish r.can --hf-hub user/name    Publish .can to HF Hub as dataset
 soup runs                                     List training runs
 soup runs show <run_id>                       Run details + loss graph + cost
 soup runs compare <run_1> <run_2>             Compare two runs
-soup runs replay <run_id>                     Replay summary + loss curve from history
+soup runs replay <run_id>                     Replay summary + loss curve from history (also plots a benchmark-score curve when the metric lives in eval_results)
 soup why [run_id]                             Explain training anomalies (heuristic)
 soup tui                                      Full-screen Textual dashboard (requires [tui] extra)
 soup train --config soup.yaml --profile       Record torch.profiler trace to <output>/profiles/
@@ -169,8 +171,10 @@ soup edit set --base <m> --method rome|memit|alphaedit --subject "..." --target
 soup edit diff <before-run> <after-run> --probes p.jsonl  Knowledge-injection diff visualizer
 soup ingest --source langfuse|langsmith|helicone|openpipe|otel|openai-stored --logs <jsonl>  Universal trace importer (6 SaaS adapters → normalised JSONL)
 soup prune-prompt --input <jsonl> --output <jsonl> --min-frequency 0.95  Detect + strip shared system-prompt prefix
+soup prune-prompt ... --tokenizer <id-or-path>  Tokenizer-aware prefix detection (decodes remaining ids, boundary-safe)
 soup data active-sample --input <jsonl> --output <jsonl> --budget N  Top-N uncertain prod traces for human review
 soup ab --input <jsonl> --metric latency|judge_score|retry_rate  mSPRT sequential A/B (decision: continue / reject_h0 / accept_h0)
+soup ingest|prune-prompt|ab|data active-sample ... --slack-url <https> | --discord-url <https>  Shared SSRF-validated webhook on completion
 soup drift-alarm --reference <jsonl> --live <jsonl> --threshold 0.2  Rolling-KL drift alarm (exit 3 on drift)
 soup drift-alarm ... --slack-url <https> | --discord-url <https>  Optional SSRF-validated webhook on drift detected
 soup tunability --list                                   List built-in candidate-base catalogue
@@ -188,8 +192,11 @@ soup probe interference <losses.json> [--output o.json]  Pairwise N×N adapter i
 soup probe pack <base> [--output o.json]      Per-base calibrated probe pack manifest (v0.66.0)
 soup probe pack --list                        List bundled probe-pack bases (v0.66.0)
 soup adapters blame ... --top-k 50            Live DataInf-style influence runner (v0.66.0, closes #171)
-soup adapters merge ... --strategy cmaes --eval <s> --budget 1h  CMA-ES evolutionary merge (v0.67.0)
+soup adapters merge ... --strategy cmaes --eval <s> --budget 1h  CMA-ES evolutionary merge — live loop (v0.67.0 schema / v0.71.4 live)
+soup adapters merge ... --canary <suite.json> [--strict-verdict]  Live OK/MINOR/MAJOR canary verdict, exit 2 on MAJOR (v0.71.4)
 soup adapters pr <title> --base-sha <hex> --adapter <path>  GitHub-shaped adapter PR Markdown / JSON (v0.67.0)
+soup adapters pr <title> ... --push owner/repo#N  Post the PR as a GitHub comment via gh api (v0.71.4)
+soup adapters branch <name> --from-registry <id> | --attach-to-registry <id>  Branch ↔ Registry lineage (v0.71.4)
 soup adapters bisect <ckpt>... --eval-command "..."  Binary search over training history (v0.67.0)
 soup lock write --base-sha <h> --dataset-sha <h> --env-hash <h>  Write soup.lock (v0.67.0)
 soup lock write --base-sha <h> --dataset-sha <h> --env-lock soup-env.lock  Auto-derive --env-hash from soup-env.lock (v0.71.1)

{soup_cli-0.71.3 → soup_cli-0.71.5}/docs/data.md

@@ -94,6 +94,14 @@ soup prune-prompt --input traces.jsonl --output pruned.jsonl --min-frequency 0.9
 
 Binary-search over up-to-32 candidate templates finds the longest qualifying prefix (a longer threshold-meeting prefix may exist beyond the universal one — Soup does not early-exit on the 100% match). Two-pass file read with a 100 000-row DoS cap.
 
+**Tokenizer-aware mode (v0.71.5).** Pass `--tokenizer <id-or-path>` (a HuggingFace repo id, a local path, or anything `AutoTokenizer.from_pretrained` accepts) to detect the shared prefix in *token* space and decode only the remaining ids:
+
+```bash
+soup prune-prompt --input traces.jsonl --output pruned.jsonl --tokenizer Qwen/Qwen2.5-0.5B
+```
+
+Char-level stripping can cut a BPE multi-byte sequence in half when the shared prefix ends mid-token; token-aware pruning finds the longest shared *token-id* prefix and decodes the remainder, so the boundary always lands on a real token. Per-row encoding is capped at 50 000 tokens. Omit `--tokenizer` to keep the original character-level behaviour.
+
 
 ## Active-Learning Sampler (`soup data active-sample`)
 
@@ -108,6 +116,8 @@ soup data active-sample --input traces.jsonl --output for-review.jsonl --budget
 
 The output JSONL is a drop-in prompt set for `soup eval human` (v0.19). Budget is bounded `[1, 100 000]`.
 
+**Webhooks (v0.71.5).** `soup ingest`, `soup prune-prompt`, `soup ab`, and `soup data active-sample` all accept `--slack-url` / `--discord-url` and POST a one-line summary on completion through the same SSRF-hardened validator as `soup drift-alarm` (scheme allowlist, loopback-only HTTP, RFC1918 / link-local / reserved / multicast rejected; the post never raises, so a flaky webhook can't fail the command). `soup ab` only fires when the sequential test actually decides (`reject_h0` / `accept_h0`), not while it's still `continue`-ing.
+
 
 ## Synthetic Data Generation
 
@@ -479,6 +489,8 @@ Three tasks supported: `sft` (Q&A pairs), `preference` (chosen/rejected), `tool`
 
 Document discovery is one level deep over `.txt` / `.md` / `.json` / `.jsonl`; dotfiles + symlinked directories are skipped. All paths are cwd-contained, all writes are atomic via staged-tempfile + `os.replace`, and write targets are rejected if they're symlinks. **Judge providers are live**: `--judge-provider ollama` (localhost-only), `--judge-provider anthropic` (env-only API key), `--judge-provider vllm` (scheme-validated). Per-call judge exceptions logged at DEBUG.
 
+**Alternative teacher hubs (v0.71.5).** `--hub modelscope|modelers` pre-fetches the `--teacher` from that hub when the teacher is a routable repo id (`owner/name`); `--hub hf` (default) is a no-op and leaves the teacher as a provenance label. If `--hub` is non-HF but `--teacher` is not a repo id (e.g. the default `local-judge`), Soup prints a loud yellow warning rather than silently dropping the flag.
+
 
 ## Data Quality Scorecard
 

{soup_cli-0.71.3 → soup_cli-0.71.5}/docs/evaluation.md

@@ -82,6 +82,8 @@ soup advise compare
 4. Task is `factual_lookup` with high output variance → **RAG**.
 5. Otherwise → **SFT**.
 
+**Cross-project confidence bias (v0.71.5).** When `~/.soup/advise_history.jsonl` holds ≥3 prior verdicts for the *same choice* in the *same project*, `soup advise` nudges its confidence (not its decision) toward what worked before: a net-positive precedent record (you accepted it AND its recorded outcome was good) bumps confidence up by a small constant; a net-negative one bumps it down. The rubric verdict itself never changes — only how sure Soup is. Verdicts must be `--record`ed for the bias to kick in.
+
 **Why this command exists.** "Choose fine-tuning vs RAG vs prompt-engineering" is the most-mis-made decision in the space. Reddit, HN, IBM, and Google Cloud all converge on the same advice (start with prompts, escalate to RAG, fine-tune as last resort) and almost everyone ignores it because nobody has the data to prove their case is the exception. Soup `autopilot` picks hyperparameters AFTER you've decided to train; `soup advise` owns the layer above. No trainer library has an incentive to tell users *not to train* — Unsloth's funnel, Axolotl's hosted business, LLaMA-Factory's Alibaba alignment all monetise the training event.
 
 
@@ -213,6 +215,8 @@ soup ab --input ab.jsonl --metric judge_score --alpha 0.01 --beta 0.10 --effect-
 
 Input rows look like `{"arm": "control", "latency": 1.23}` or `{"arm": "treatment", "judge_score": 0.91}`. Decision is one of `continue` (keep collecting samples), `reject_h0` (real difference detected), `accept_h0` (no significant difference). Composes with `soup loop canary` (v0.58) — promote or roll back as soon as the LLR clears a decision boundary.
 
+`soup ab` accepts `--slack-url` / `--discord-url` (v0.71.5) and pings the webhook **only when the test actually decides** (`reject_h0` / `accept_h0`) — a still-running `continue` stays quiet so you're not paged on every peek. Same SSRF-hardened validator as `soup drift-alarm`.
+
 
 ## Drift Alarm (`soup drift-alarm`)
 

{soup_cli-0.71.3 → soup_cli-0.71.5}/docs/training.md

@@ -907,12 +907,15 @@ Layer dynamic re-weighting on top of the static `curriculum` bucketer. Every N s
 training:
   curriculum: true                          # static bucketer (v0.23.0)
   curriculum_buckets: 4
+  curriculum_metric: perplexity             # length (default) | loss | perplexity
   curriculum_dynamic: true                  # NEW — dynamic re-weighting
   curriculum_dynamic_recompute_steps: 50    # refresh every 50 global steps
   curriculum_dynamic_floor: 0.05            # min weight per bucket
   curriculum_dynamic_temperature: 1.0       # softmax temp on uncertainty
 ```
 
+**Bucketing by difficulty percentile (v0.71.5).** When `curriculum_metric` is `loss` or `perplexity`, the dynamic callback assigns each step's sample to a bucket by its *rank* within a rolling 512-step window of the difficulty signal (perplexity = `exp(min(loss, 50))`), instead of the round-robin fallback used for `length`. This keeps the buckets calibrated to the live loss distribution rather than a static length sort. `length` (the default) keeps the round-robin assignment.
+
 Visualise the recorded bucket-weight evolution with `soup runs curriculum-curve <run_id>`.
 
 DDP / grad-accum safety: multi-rank launches must wire an `all_reduce` hook on per-bucket stats (a cross-validator rejects un-coordinated multi-rank runs upfront). Multi-trainer expansion beyond `sft` / `pretrain` is tracked for v0.48.1.

{soup_cli-0.71.3 → soup_cli-0.71.5}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.3"
+version = "0.71.5"
 description = "Fine-tune LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.3 → soup_cli-0.71.5}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune LLMs in one command."""
 
-__version__ = "0.71.3"
+__version__ = "0.71.5"

soup_cli-0.71.5/src/soup_cli/commands/_webhook_cli.py

@@ -0,0 +1,71 @@
+"""Shared CLI glue for the --slack-url / --discord-url webhook flags (v0.71.5 #207).
+
+Keeps Typer + Rich Console concerns in the commands layer (``utils/webhooks``
+stays import-light + framework-free). Used by ``ingest`` / ``prune-prompt`` /
+``ab`` / ``data active-sample`` so the validate-then-deliver pattern is defined
+once.
+"""
+
+from __future__ import annotations
+
+from typing import Optional, Tuple
+
+import typer
+from rich.console import Console
+from rich.markup import escape
+
+
+def validate_webhook_flags(
+    slack_url: Optional[str],
+    discord_url: Optional[str],
+    *,
+    console: Console,
+) -> Tuple[Optional[str], Optional[str]]:
+    """Validate webhook URLs at the CLI boundary (``typer.Exit(2)`` on bad).
+
+    Returns the (canonical) URLs. Mirrors the v0.63.0 ``drift-alarm``
+    early-rejection pattern so a typo'd / SSRF-y URL fails fast with a
+    friendly message instead of being silently swallowed at delivery time.
+    """
+    from soup_cli.utils.webhooks import validate_webhook_url
+
+    out = []
+    for label, value in (("--slack-url", slack_url), ("--discord-url", discord_url)):
+        if value is None:
+            out.append(None)
+            continue
+        try:
+            out.append(validate_webhook_url(value))
+        except (TypeError, ValueError) as exc:
+            console.print(f"[red]{label}: {escape(str(exc))}[/]")
+            raise typer.Exit(2) from exc
+    return out[0], out[1]
+
+
+def emit_webhooks(
+    slack_url: Optional[str],
+    discord_url: Optional[str],
+    *,
+    payload: dict,
+    console: Console,
+) -> None:
+    """POST the completion payload to any configured webhooks (best-effort).
+
+    Never raises (delegates to the never-raising
+    :func:`soup_cli.utils.webhooks.send_webhooks`). Prints a per-target
+    delivered/failed line so the operator sees whether the alert landed.
+    """
+    if slack_url is None and discord_url is None:
+        return
+    from soup_cli.utils.webhooks import send_webhooks
+
+    for label, ok in send_webhooks(
+        payload, slack_url=slack_url, discord_url=discord_url
+    ):
+        colour = "green" if ok else "yellow"
+        console.print(
+            f"[{colour}]{label} webhook: {'delivered' if ok else 'failed'}[/]"
+        )
+
+
+__all__ = ["emit_webhooks", "validate_webhook_flags"]

{soup_cli-0.71.3 → soup_cli-0.71.5}/src/soup_cli/commands/ab.py

@@ -2,12 +2,15 @@
 
 from __future__ import annotations
 
+from typing import Optional
+
 import typer
 from rich.console import Console
 from rich.markup import escape
 from rich.panel import Panel
 from rich.table import Table
 
+from soup_cli.commands._webhook_cli import emit_webhooks, validate_webhook_flags
 from soup_cli.utils.ab_test import (
     MsprtConfig,
     run_msprt,
@@ -35,6 +38,20 @@ def ab(
         0.1, "--effect-size",
         help="Minimum detectable difference in means.",
     ),
+    slack_url: Optional[str] = typer.Option(
+        None, "--slack-url",
+        help=(
+            "Optional Slack webhook URL — POSTed on a reject_h0 / accept_h0 "
+            "decision (not on continue). SSRF-validated."
+        ),
+    ),
+    discord_url: Optional[str] = typer.Option(
+        None, "--discord-url",
+        help=(
+            "Optional Discord webhook URL — POSTed on a reject_h0 / accept_h0 "
+            "decision (not on continue). SSRF-validated."
+        ),
+    ),
 ) -> None:
     """Sequential A/B test with early-stop guarantees (mSPRT)."""
     try:
@@ -43,6 +60,10 @@ def ab(
         console.print(f"[red]{escape(str(exc))}[/]")
         raise typer.Exit(2) from exc
 
+    slack_url, discord_url = validate_webhook_flags(
+        slack_url, discord_url, console=console
+    )
+
     try:
         cfg = MsprtConfig(
             metric=canonical, alpha=alpha, beta=beta, effect_size=effect_size,
@@ -101,5 +122,24 @@ def ab(
             )
         )
 
+    # Webhook only fires on a terminal decision (reject_h0 / accept_h0) —
+    # a `continue` verdict carries no actionable signal (issue #207).
+    if verdict.decision != "continue":
+        emit_webhooks(
+            slack_url,
+            discord_url,
+            payload={
+                "command": "ab",
+                "metric": canonical,
+                "decision": verdict.decision,
+                "log_likelihood_ratio": verdict.log_likelihood_ratio,
+                "n_control": verdict.n_control,
+                "n_treatment": verdict.n_treatment,
+                "mean_control": verdict.mean_control,
+                "mean_treatment": verdict.mean_treatment,
+            },
+            console=console,
+        )
+
 
 __all__ = ["ab"]

{soup_cli-0.71.3 → soup_cli-0.71.5}/src/soup_cli/commands/active_sample.py

@@ -2,11 +2,14 @@
 
 from __future__ import annotations
 
+from typing import Optional
+
 import typer
 from rich.console import Console
 from rich.markup import escape
 from rich.panel import Panel
 
+from soup_cli.commands._webhook_cli import emit_webhooks, validate_webhook_flags
 from soup_cli.utils.active_sampler import sample_uncertain_rows, validate_budget
 
 console = Console()
@@ -24,6 +27,14 @@ def active_sample(
         100, "--budget",
         help="Max rows to surface for human review (1 - 100_000).",
     ),
+    slack_url: Optional[str] = typer.Option(
+        None, "--slack-url",
+        help="Optional Slack webhook URL — POSTed on completion. SSRF-validated.",
+    ),
+    discord_url: Optional[str] = typer.Option(
+        None, "--discord-url",
+        help="Optional Discord webhook URL — POSTed on completion. SSRF-validated.",
+    ),
 ) -> None:
     """Surface the most uncertain prod traces for human review."""
     try:
@@ -32,6 +43,10 @@ def active_sample(
         console.print(f"[red]{escape(str(exc))}[/]")
         raise typer.Exit(2) from exc
 
+    slack_url, discord_url = validate_webhook_flags(
+        slack_url, discord_url, console=console
+    )
+
     try:
         plan = sample_uncertain_rows(
             input_path,
@@ -55,5 +70,18 @@ def active_sample(
         )
     )
 
+    emit_webhooks(
+        slack_url,
+        discord_url,
+        payload={
+            "command": "active-sample",
+            "rows_selected": plan.rows_selected,
+            "rows_in": plan.rows_in,
+            "mean_uncertainty": plan.mean_uncertainty,
+            "budget": plan.budget,
+        },
+        console=console,
+    )
+
 
 __all__ = ["active_sample"]