soup-cli 0.71.35__tar.gz → 0.71.36__tar.gz

{soup_cli-0.71.35 → soup_cli-0.71.36}/CHANGELOG.md

@@ -12,6 +12,71 @@ reproducing 70+ versions of notes.
 
 ## [Unreleased]
 
+## [0.71.36] - 2026-07-16
+
+**Data Moat II** — a semantic layer over your training data, plus two tools for
+what a fine-tune forgets and leaks.
+
+### Added
+
+- **`soup data dedup --semantic`** — near-duplicate removal over embedding
+  cosine instead of MinHash shingling. Catches *reworded* duplicates that
+  MinHash misses (measured: 0.88–0.91 cosine on rewordings MinHash scored as
+  distinct) while correctly keeping distinct-but-similar instructions. Zero new
+  dependencies: uses `transformers` from the `[train]` extra. **Read the
+  known-limitation below before lowering `--threshold`.**
+- **`soup data topics <data>`** — cluster a dataset and label each cluster with
+  c-TF-IDF terms, plus a coverage table (`82% code · 6% math`) and a warning for
+  thin topics. Labels are *emergent term clusters*, not a fixed taxonomy.
+- **`soup data canary insert|check`** — Secret-Sharer memorization probe. Insert
+  K high-entropy secrets, then check any model/adapter: each secret's loss is
+  ranked against never-inserted controls drawn from the same space. Exit 2 on
+  MAJOR so CI can gate. Measured on SmolLM2-135M: a memorized set lands at
+  percentile 0.0 (loss 1.7–2.5) against a clean model's 4.1–6.2.
+- **`soup train --replay old.jsonl --replay-ratio 0.1`** — continual-learning
+  rehearsal. Interleaves a seeded sample of an old dataset into training so a
+  new task does not erase the old one. `r` is the fraction of the **final**
+  mixed set (`n_replay = round(r/(1-r) · n_new)`), rows are interleaved rather
+  than appended, and an undersized pool reports the shortfall instead of
+  repeating rows. Mixed into `train` only — validation stays pure new-task.
+
+### Fixed
+
+- **The hardware-fit gate refused to train any local checkpoint.** A merged
+  model (`soup merge -o ./mymodel`) has no size marker in its name, so the size
+  guesser returned its 7B default, predicted ~16 GB of VRAM and refused. Local
+  checkpoints are now *measured* from their safetensors header (0.135B actual vs
+  7.0B guessed) — this had blocked `soup merge` → train-from-merged entirely.
+  Third instance of this class after the v0.71.32 (Whisper) and v0.71.33
+  (`M` suffix) fixes.
+- **`pip install 'soup-cli[extra]'` hints printed without the extra.** Rich ate
+  the bracket, so every "install the missing dependency" message across 17 sites
+  told users to run `pip install 'soup-cli'` — which succeeds and still leaves
+  the feature broken. Affected `[eval]`, `[data]`, `[serve]`, `[ui]`, `[tui]`,
+  `[compile]`, `[mcp]`, `[carbon]` and others, including Typer help text.
+- Replay rows bypassed the image/audio path-traversal validation that the
+  primary dataset receives.
+
+### Known limitations
+
+- **Semantic dedup is not a paraphrase detector.** Measured with
+  all-MiniLM-L6-v2, paraphrase cosines (0.49–0.76) **overlap** with
+  genuinely-distinct rows (0.54–0.76): "Add two numbers" vs "Multiply two
+  numbers" scores 0.759, *higher* than the true paraphrase "reverse a string" /
+  "invert the order of characters" at 0.491. **No threshold separates them**, so
+  lowering `--threshold` to chase paraphrases deletes real training rows. The
+  default (0.8) is deliberately conservative.
+- **Replay is validated at proof-of-mechanism scale.** On SmolLM2-135M + LoRA,
+  replay retained the old task 7% better than a no-replay control — the correct
+  direction — but forgetting without it was only +4%, i.e. mild. The effect size
+  at full fine-tuning or 7B+ is unproven on a 4 GB box.
+- **Canary exposure is the sampled-control approximation**, not full-space rank
+  enumeration. "No exposure" is not proof of no memorization.
+- **`data topics` / `dedup --semantic` require `[train]`** (torch) and download
+  an embedding model. Plain MinHash `dedup` stays on the light core.
+- Replay v1 is `sft`/`pretrain` only and is incompatible with
+  `packing`/`multipack`.
+
 ## [0.71.35] - 2026-07-15
 
 ### Added

{soup_cli-0.71.35 → soup_cli-0.71.36}/CONTRIBUTING.md

@@ -120,7 +120,7 @@ src/soup_cli/
   templates/          - 21 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0, +4 compliance v0.71.35)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (313 files, 16001 tests)
+tests/                - Test suite (314 files, 16283 tests)
 examples/             - Real-world config examples and datasets
 ```
 
@@ -150,7 +150,7 @@ pytest tests/test_data.py::test_detect_alpaca_format -v
 pytest tests/ --cov=soup_cli --cov-report=html
 ```
 
-### Test Files (307 files)
+### Test Files (313 files)
 
 > A representative sample of the suite below. Run `pytest tests/ -v` for the complete list.
 
@@ -388,7 +388,7 @@ If adding a new training algorithm:
 
 1. Create `trainer/your_trainer.py` wrapping the appropriate TRL trainer
 2. Add config fields to `config/schema.py` (Pydantic v2)
-3. Add template to `config/schema.py` (see existing 15 templates)
+3. Add a template YAML under `src/soup_cli/templates/` + a `manifest.json` entry (see the existing 21 templates)
 4. Update `commands/train.py` to route to your trainer
 5. Add 30+ tests in `tests/test_your_trainer.py`
 6. Update `README.md` + the relevant page under `docs/`

{soup_cli-0.71.35 → soup_cli-0.71.36}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.71.35
+Version: 0.71.36
 Summary: Fine-tune and post-train LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -193,34 +193,36 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.35 — the compliance pack.** Ship a regulated fine-tune with the paperwork it needs: start from a regulation-shaped config, publish a provenance-carrying model card, and gate every PR in CI.
-
-- **`soup init --template hipaa|soc2|eu-ai-act|sr-11-7`.** Four regulation-shaped starting
-  configs, each documenting the exact commands for that regime — PHI scrubbing + air-gap,
-  BOM/attest/sign, EU Annex XI + energy tracking, or repro-receipt + diagnose/ship.
-- **`soup card <registry-id>` → `MODELCARD.md`.** Turn a registry entry into a publishable HF
-  model card: training config, eval scorecard, config/data hashes, lineage, and every
-  registered artifact. `soup push --card` uploads it as the README, so every public push
-  carries its provenance.
-- **`soup ci init` — fine-tuning CI.** Writes a GitHub Actions gate that runs
-  `data validate` → `expect` → `ship --evidence` on each PR; a regression exits 2 and
-  blocks the merge.
-- **Compliance quickstart** ([docs/compliance.md](docs/compliance.md)) — the whole path from
-  template to air-gapped, signed, attested, carded model.
-- **GGUF export fixed on Windows.** Validated end-to-end for the first time (SmolLM2 →
-  q4_0/q4_k_m/q8_0/f16 → Ollama → live inference), which surfaced four real bugs: the export
-  cloned llama.cpp into your *current directory*, its first run **downgraded your PyTorch to
-  CPU-only** (breaking CUDA), a correctly-built llama.cpp wasn't found under MSVC, and
-  `deploy ollama` rejected relative GGUF paths. All fixed.
-- **Hardening:** the model-card generator (including the existing `soup push` card) no longer
-  interpolates unescaped config values — a crafted `base`/`scheduler` can't smuggle HTML into
-  a card published on the Hub.
+**v0.71.36 — Data Moat II.** A semantic layer over your training data, plus two tools for what a fine-tune forgets and what it leaks.
+
+- **`soup data dedup --semantic`** — dedup by meaning, not by shared tokens. Catches
+  *reworded* duplicates MinHash's shingling scores as distinct. Zero new dependencies.
+- **`soup data topics <data>`** — cluster your data and get a coverage table
+  (`82% code · 6% math · 1% safety`) with a warning for thin topics, so you can see what
+  you are actually training on.
+- **`soup data canary insert|check`** — insert unique secrets, then prove whether a model
+  memorized them: each secret's loss is ranked against never-inserted controls. Exit 2 on a
+  leak, so CI can gate. On SmolLM2-135M a memorized set lands at percentile 0.0 against a
+  clean model's 4%–93% spread.
+- **`soup train --replay old.jsonl`** — continual-learning rehearsal: interleave your old
+  data so the new task doesn't erase it.
+- **Honest result:** semantic dedup is **not** a paraphrase detector. Measured, paraphrase
+  cosines (0.49–0.76) *overlap* with genuinely-distinct rows (0.54–0.76) — "Add two numbers"
+  vs "Multiply two numbers" scores **higher** than a real paraphrase — so no threshold
+  separates them. The default is deliberately conservative; see the CHANGELOG.
+- **Two blocking bugs fixed:** the hardware-fit gate **refused to train any model you merged
+  yourself** (a local checkpoint's name has no size marker, so it guessed 7B and predicted
+  16 GB), and every `pip install 'soup-cli[extra]'` hint printed **without the extra** —
+  17 sites where the suggested command succeeds and leaves the feature still broken.
 
 ```bash
-soup adapters arithmetic "coder - toxic" \
-  --adapter coder=./coder-lora --adapter toxic=./toxic-lora -o ./cleaned
+soup data topics train.jsonl                    # what am I actually training on?
+soup data dedup train.jsonl --semantic -o clean.jsonl
 
-soup train --config sft.yaml   # with training.lisa_enabled: true, lisa_num_layers: 2
+soup data canary insert train.jsonl -o canaried.jsonl --manifest secrets.json
+soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
+
+soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
 ```
 
 <details>
@@ -320,6 +322,7 @@ The full feature reference lives in [`docs/`](docs/). Start here:
 | [Evaluation & probes](docs/evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
 | [Adapters, registry & governance](docs/adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap) |
+| [Compliance & governance quickstart](docs/compliance.md) | HIPAA/SOC2/EU-AI-Act/SR-11-7 `init` templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (`soup card`), CI gate (`soup ci init`) |
 | [Backends, platform & ops](docs/backends-and-ops.md) | MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
 | [Command reference](docs/commands.md) | The full `soup` command list |
 | [Supported models & extras](docs/models.md) | Recommended model families, the VRAM size guide, the pip extras matrix |

{soup_cli-0.71.35 → soup_cli-0.71.36}/README.md

@@ -49,34 +49,36 @@ infrastructure instead of improving models. Soup fixes that.
 
 ## What's New
 
-**v0.71.35 — the compliance pack.** Ship a regulated fine-tune with the paperwork it needs: start from a regulation-shaped config, publish a provenance-carrying model card, and gate every PR in CI.
-
-- **`soup init --template hipaa|soc2|eu-ai-act|sr-11-7`.** Four regulation-shaped starting
-  configs, each documenting the exact commands for that regime — PHI scrubbing + air-gap,
-  BOM/attest/sign, EU Annex XI + energy tracking, or repro-receipt + diagnose/ship.
-- **`soup card <registry-id>` → `MODELCARD.md`.** Turn a registry entry into a publishable HF
-  model card: training config, eval scorecard, config/data hashes, lineage, and every
-  registered artifact. `soup push --card` uploads it as the README, so every public push
-  carries its provenance.
-- **`soup ci init` — fine-tuning CI.** Writes a GitHub Actions gate that runs
-  `data validate` → `expect` → `ship --evidence` on each PR; a regression exits 2 and
-  blocks the merge.
-- **Compliance quickstart** ([docs/compliance.md](docs/compliance.md)) — the whole path from
-  template to air-gapped, signed, attested, carded model.
-- **GGUF export fixed on Windows.** Validated end-to-end for the first time (SmolLM2 →
-  q4_0/q4_k_m/q8_0/f16 → Ollama → live inference), which surfaced four real bugs: the export
-  cloned llama.cpp into your *current directory*, its first run **downgraded your PyTorch to
-  CPU-only** (breaking CUDA), a correctly-built llama.cpp wasn't found under MSVC, and
-  `deploy ollama` rejected relative GGUF paths. All fixed.
-- **Hardening:** the model-card generator (including the existing `soup push` card) no longer
-  interpolates unescaped config values — a crafted `base`/`scheduler` can't smuggle HTML into
-  a card published on the Hub.
+**v0.71.36 — Data Moat II.** A semantic layer over your training data, plus two tools for what a fine-tune forgets and what it leaks.
+
+- **`soup data dedup --semantic`** — dedup by meaning, not by shared tokens. Catches
+  *reworded* duplicates MinHash's shingling scores as distinct. Zero new dependencies.
+- **`soup data topics <data>`** — cluster your data and get a coverage table
+  (`82% code · 6% math · 1% safety`) with a warning for thin topics, so you can see what
+  you are actually training on.
+- **`soup data canary insert|check`** — insert unique secrets, then prove whether a model
+  memorized them: each secret's loss is ranked against never-inserted controls. Exit 2 on a
+  leak, so CI can gate. On SmolLM2-135M a memorized set lands at percentile 0.0 against a
+  clean model's 4%–93% spread.
+- **`soup train --replay old.jsonl`** — continual-learning rehearsal: interleave your old
+  data so the new task doesn't erase it.
+- **Honest result:** semantic dedup is **not** a paraphrase detector. Measured, paraphrase
+  cosines (0.49–0.76) *overlap* with genuinely-distinct rows (0.54–0.76) — "Add two numbers"
+  vs "Multiply two numbers" scores **higher** than a real paraphrase — so no threshold
+  separates them. The default is deliberately conservative; see the CHANGELOG.
+- **Two blocking bugs fixed:** the hardware-fit gate **refused to train any model you merged
+  yourself** (a local checkpoint's name has no size marker, so it guessed 7B and predicted
+  16 GB), and every `pip install 'soup-cli[extra]'` hint printed **without the extra** —
+  17 sites where the suggested command succeeds and leaves the feature still broken.
 
 ```bash
-soup adapters arithmetic "coder - toxic" \
-  --adapter coder=./coder-lora --adapter toxic=./toxic-lora -o ./cleaned
+soup data topics train.jsonl                    # what am I actually training on?
+soup data dedup train.jsonl --semantic -o clean.jsonl
 
-soup train --config sft.yaml   # with training.lisa_enabled: true, lisa_num_layers: 2
+soup data canary insert train.jsonl -o canaried.jsonl --manifest secrets.json
+soup data canary check --manifest secrets.json --base ./my-model  # exit 2 = leak
+
+soup train --config sft.yaml --replay old_task.jsonl --replay-ratio 0.1
 ```
 
 <details>
@@ -176,6 +178,7 @@ The full feature reference lives in [`docs/`](docs/). Start here:
 | [Evaluation & probes](docs/evaluation.md) | Eval design/gate, eval-gated training, benchmarks, NLG metrics, calibration, Elo arena, diagnose, post-train X-ray probes, A/B, drift, tunability, `soup advise` |
 | [Serving & export](docs/serving-and-export.md) | OpenAI-compatible server, batch inference, benchmarking, merge/export, Anthropic Messages endpoint, speculative decoding (train + measure your own draft), deploy autopilot, Web UI, Agent Forge |
 | [Adapters, registry & governance](docs/adapters-and-governance.md) | Adapter lifecycle/management, model registry, Soup Cans, the data flywheel (`soup loop`), knowledge editing, steering, supply-chain controls (scan/sign/BOM/attest/audit/airgap) |
+| [Compliance & governance quickstart](docs/compliance.md) | HIPAA/SOC2/EU-AI-Act/SR-11-7 `init` templates, provenance (BOM/attest/repro-receipt), audit log, air-gap, model-card autogen (`soup card`), CI gate (`soup ci init`) |
 | [Backends, platform & ops](docs/backends-and-ops.md) | MLX/Unsloth backends, alternative hubs, HF Hub integration, autopilot, experiment tracking, plan/apply, env lockfiles, hardware-fit, completions, plugins, utility commands |
 | [Command reference](docs/commands.md) | The full `soup` command list |
 | [Supported models & extras](docs/models.md) | Recommended model families, the VRAM size guide, the pip extras matrix |

{soup_cli-0.71.35 → soup_cli-0.71.36}/docs/commands.md

@@ -11,6 +11,7 @@ soup init [--template chat|code|...|audio]       Create config
 soup init --template hipaa|soc2|eu-ai-act|sr-11-7  Compliance-shaped starting config + the commands for that regime (v0.71.35)
 soup autopilot --model <id> --data d.jsonl --goal <g>  Zero-configsoup train --config soup.yaml                 Start training
 soup train --config soup.yaml --tensorboard   Train with TensorBoard logging
+soup train --config soup.yaml --replay old.jsonl --replay-ratio 0.1  Continual-learning rehearsal: interleave old data so the new task doesn't erase it (sft/pretrain)
 soup train --config soup.yaml --fsdp full_shard  Train with FSDP2
 soup train --config soup.yaml --deepspeed zero++  DeepSpeed ZeRO++ (quantized comms)
 soup train --config soup.yaml --gpus auto|N      Multi-GPU launch hint
@@ -77,6 +78,10 @@ soup data lint <path>                         Preference-data linter: length bia
 soup data convert <path> --to chatml          Convert between formats
 soup data merge data1.jsonl data2.jsonl       Combine datasets
 soup data dedup <path> --threshold 0.8        Remove duplicates (MinHash)
+soup data dedup <path> --semantic             Dedup by embedding cosine — catches rewordings MinHash misses ([train])
+soup data topics <path> [--clusters N|auto]   Cluster + c-TF-IDF labels + coverage table + thin-topic warnings ([train])
+soup data canary insert <path> -o <out> --manifest <m>  Insert K secrets to later prove memorization (manifest = SECRET)
+soup data canary check --manifest <m> --base <model>    Rank each secret's loss vs never-inserted controls; exit 2 = leak
 soup data stats <path>                        Extended statistics
 soup data generate --prompt "..." --count 100 Generate synthetic data
 soup data generate ... --provider ollama      Use local Ollama instance

{soup_cli-0.71.35 → soup_cli-0.71.36}/docs/data.md

@@ -22,6 +22,9 @@
 - [Synthetic Data Forge](#synthetic-data-forge)
 - [Data Quality Scorecard](#data-quality-scorecard)
 - [Remote Datasets (S3 / GCS / Azure / OCI)](#remote-datasets-s3--gcs--azure--oci)
+- [Semantic dedup (`soup data dedup --semantic`)](#semantic-dedup-soup-data-dedup---semantic)
+- [Topic map (`soup data topics`)](#topic-map-soup-data-topics)
+- [Canaries (`soup data canary insertcheck`)](#canaries-soup-data-canary-insertcheck)
 - [Data Recipe DAG](#data-recipe-dag)
 - [Data Mixing Optimizer (BETA)](#data-mixing-optimizer-beta)
 - [AOT Tokenization with `soup data preprocess`](#aot-tokenization-with-soup-data-preprocess)
@@ -29,6 +32,125 @@
 
 ---
 
+## Semantic dedup (`soup data dedup --semantic`)
+
+`soup data dedup` removes near-duplicates with MinHash by default — fast, no
+torch, but **lexical**: it compares shared token shingles, so two rows that say
+the same thing in different words look unrelated to it.
+
+`--semantic` compares embedding cosine instead:
+
+```bash
+soup data dedup train.jsonl --semantic -o clean.jsonl
+soup data dedup train.jsonl --semantic --threshold 0.85 --field text -o clean.jsonl
+soup data dedup train.jsonl --semantic --embed-model sentence-transformers/all-mpnet-base-v2
+```
+
+Requires the `[train]` extra (it reuses `transformers`; there is no new
+dependency) and downloads a small embedding model on first use. Plain MinHash
+`dedup` stays on the light core.
+
+**What it buys you.** Measured against MinHash on the same rows
+(all-MiniLM-L6-v2):
+
+| pair | cosine | MinHash | `--semantic` |
+|---|---|---|---|
+| exact duplicate | 1.000 | caught | caught |
+| "sorts **a list** of integers" / "sorts **an array** of integers" | 0.908 | **missed** | caught |
+| "which sorts a list of **ints**" (reworded) | 0.880 | **missed** | caught |
+| "Add two numbers" / "Multiply two numbers" | 0.759 | kept | kept (correct) |
+
+So `--semantic` catches **rewordings** MinHash's shingling scores as distinct.
+
+### It is not a paraphrase detector — and why the default is 0.8
+
+Heavier paraphrases are **not** reliably separable. Measured, paraphrase cosines
+(0.49–0.76) *overlap* with genuinely-distinct rows (0.54–0.76):
+
+- "reverse a string" / "invert the order of characters" — a **true paraphrase** — scores **0.491**
+- "Add two numbers" / "Multiply two numbers" — **two rows you must keep** — scores **0.759**
+
+A real paraphrase can score *lower* than two rows that must both survive, so **no
+threshold cleanly separates them**. Lowering `--threshold` to chase paraphrase
+recall deletes real training rows — silent data loss, which is worse than keeping
+a duplicate. The 0.8 default is deliberately conservative. Raise or lower it only
+against your own data, and check what got dropped.
+
+`--threshold` means Jaccard for MinHash and cosine for `--semantic`. They are
+different scales; a value tuned for one is not meaningful for the other.
+
+## Topic map (`soup data topics`)
+
+See what you are actually training on:
+
+```bash
+soup data topics train.jsonl                       # 'auto' picks the cluster count
+soup data topics train.jsonl --clusters 8 -o topics.json
+```
+
+Embeds every row, clusters with k-means, and labels each cluster with c-TF-IDF
+terms — terms frequent in *that* cluster and rare elsewhere, so filler words like
+"the" never become a label. Prints a coverage table plus a warning for any topic
+under 2% of the data:
+
+```
+        Topic map — 4200 rows, 6 clusters
+┏━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━┓
+┃ Topic                    ┃ Rows ┃ Coverage ┃
+┡━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━┩
+│ function / python / code │ 3444 │    82.0% │
+│ theorem / proof / math   │  252 │     6.0% │
+│ refuse / harmful / safe  │   42 │     1.0% │
+└──────────────────────────┴──────┴──────────┘
+topic 'refuse / harmful / safe' is thin: 1.0% of rows (42/4200)
+```
+
+Labels are **emergent term clusters**, not a classification against a fixed
+taxonomy: "82% code" means 82% of rows landed in a cluster whose top terms look
+like code. Requires `[train]`.
+
+## Canaries (`soup data canary insert|check`)
+
+Prove whether a model memorized your data — for leak detection and provenance.
+
+```bash
+# 1. insert K unique secrets (keep the manifest OUT of your repo)
+soup data canary insert train.jsonl -o canaried.jsonl --count 16 --manifest secrets.json
+
+# 2. train on canaried.jsonl as usual, then:
+soup data canary check --manifest secrets.json --base ./my-model --adapter ./lora
+```
+
+`check` measures the model's loss on each inserted secret and ranks it against
+never-inserted **controls** drawn from the same secret space and sharing the same
+carrier prompt — so a low loss means the *secret* is unusually likely, not the
+prompt. Exit **2** on MAJOR, so CI can gate on a leak.
+
+This is loss-vs-controls (Carlini et al., *The Secret Sharer*), not "ask the model
+and see if it says the secret": a model can memorize a canary and still not emit
+it under greedy decoding, so "nothing came back" would be false reassurance.
+
+The verdict asks whether **more** canaries look memorized than chance explains
+(binomial tail, α=0.05) rather than whether any single one dipped low — with 16
+canaries, an "any single one" rule fires on a **clean** model about 15% of the
+time, which would make a CI gate useless.
+
+Measured on SmolLM2-135M:
+
+| model | loss | percentiles | verdict |
+|---|---|---|---|
+| trained on the canaries | 1.7–2.5 | all 0.0% | **MAJOR** (exit 2) |
+| never saw them | 4.1–6.2 | 1.6%–93% | OK (exit 0) |
+
+**The manifest is the sensitive artifact**, not the dataset: anyone holding it can
+reproduce the secrets. It is written `0600` on POSIX and must not be committed
+alongside the data it protects. `check --output` embeds the same secrets.
+
+Exposure is a **sampled-control approximation**, not full-space rank enumeration —
+"no exposure" is not proof of no memorization.
+
+---
+
 ## Data Engineering Pro
 
 The v0.69.0 release ships 5 surfaces that turn dataset prep from "throw a JSONL at the trainer" into a first-class engineering workflow.

{soup_cli-0.71.35 → soup_cli-0.71.36}/docs/peft-and-efficiency.md

@@ -248,7 +248,7 @@ configs where expert FFNs need lower rank than attention. Caps: 256 keys × valu
 `lora_dropout` strip) auto-fire when the model name and architecture match. Both are
 gated and silent on unrelated models.
 
-**Template registry** — the 16 built-in templates now live as
+**Template registry** — the 21 built-in templates now live as
 `src/soup_cli/templates/*.yaml` with a `manifest.json` index. `soup init --template <name>`
 reads the YAML; the inline copies in `schema.py` stay as a back-compat fallback,
 deprecated in favour of the YAML registry.

{soup_cli-0.71.35 → soup_cli-0.71.36}/docs/serving-and-export.md

@@ -63,6 +63,35 @@ soup export --model ./output --format gguf --llama-cpp /path/to/llama.cpp
 
 Supported quantizations: `q4_0`, `q4_k_m`, `q5_k_m`, `q8_0`, `f16`, `f32`
 
+### Building llama.cpp (required for quantized GGUF)
+
+Soup auto-clones llama.cpp (pinned tag `b5270`) to `~/.soup/llama.cpp` on first use,
+but it does **not** build it. The conversion step (`f16` / `f32`) works from the
+Python script alone; every *quantized* type additionally needs the `llama-quantize`
+binary, so build it once:
+
+```bash
+cd ~/.soup/llama.cpp
+cmake -B build -DGGML_NATIVE=OFF -DLLAMA_CURL=OFF -DCMAKE_BUILD_TYPE=Release
+cmake --build build --config Release --target llama-quantize -j 4
+```
+
+Soup finds the binary in both single-config (`build/bin/llama-quantize`, Make/Ninja)
+and multi-config (`build/bin/Release/llama-quantize.exe`, MSVC/Xcode) layouts, or on
+`PATH`. Point at an existing checkout with `--llama-cpp /path/to/llama.cpp` or the
+`LLAMA_CPP_PATH` env var.
+
+**Toolchain validated on Windows** (all six quants, then `soup deploy ollama` →
+inference): Visual Studio 2022 Build Tools with the *Desktop development with C++*
+workload (`Microsoft.VisualStudio.Component.VC.Tools.x86.x64`) + CMake ≥ 3.14, CPU-only.
+Linux/macOS need only a C++ toolchain + CMake. CUDA llama.cpp builds are untested
+(see [#144](https://github.com/MakazhanAlpamys/Soup/issues/144)).
+
+> Do **not** run `pip install -r ~/.soup/llama.cpp/requirements.txt` — it pins
+> `torch~=2.2.1` against the CPU wheel index and will downgrade a CUDA PyTorch,
+> breaking training. Soup deliberately installs only the convert script's extra
+> dependencies (`gguf`, `sentencepiece`, `protobuf`), unpinned.
+
 ### ONNX Export
 
 Export models to ONNX format for use with [ONNX Runtime](https://onnxruntime.ai/):

{soup_cli-0.71.35 → soup_cli-0.71.36}/docs/training.md

@@ -6,6 +6,7 @@
 
 **Contents:**
 
+- [Continual-learning rehearsal (`--replay`)](#continual-learning-rehearsal---replay)
 - [Loop Hardening](#loop-hardening)
 - [Unlearning (`task='unlearn'`, NPO / SimNPO / RMU)](#unlearning-taskunlearn-npo--simnpo--rmu)
 - [Continued Pre-training](#continued-pre-training)
@@ -41,6 +42,59 @@
 
 ---
 
+## Continual-learning rehearsal (`--replay`)
+
+Fine-tuning on a new task can erase the old one. Rehearsal is the standard
+defence: mix a slice of the old data back in.
+
+```bash
+soup train --config new_task.yaml --replay old_task.jsonl --replay-ratio 0.1
+```
+
+or in `soup.yaml`:
+
+```yaml
+data:
+  train: new_task.jsonl
+  replay: old_task.jsonl
+  replay_ratio: 0.1      # fraction of the FINAL mixed set
+  replay_seed: 0
+```
+
+**The ratio is a share of the final set**, not of the new data:
+`n_replay = round(r/(1-r) · n_new)`. At `0.1` over 1000 new rows that is 111
+replay rows → 1111 total → exactly 10%. The console reports what it did:
+
+```
+Replay: +26 old rows interleaved (30.2% of 86)
+```
+
+Three guarantees worth knowing:
+
+- **Interleaved, never appended.** A trailing block of old rows would mean the
+  model sees them all in the final steps — a second mini-finetune, which is the
+  failure rehearsal exists to prevent.
+- **`train` only; validation stays pure new-task**, so your eval still measures
+  the task you are learning. Measure old-task retention separately with
+  `soup eval custom` / `soup ship`.
+- **An undersized pool reports a shortfall rather than repeating rows** — a row
+  seen twice per epoch is a different experiment.
+
+The replay file gets its own format detection, so the old set may be alpaca while
+the new one is sharegpt.
+
+**Scope (v1):** `sft` and `pretrain` only, and incompatible with
+`packing`/`multipack` — those concatenate rows into fixed blocks, so the ratio
+stops being meaningful at block boundaries. Both are rejected with a clear error
+rather than silently mis-mixing.
+
+**Honest result.** Validated at proof-of-mechanism scale (SmolLM2-135M + LoRA):
+training task B from a model that knew task A, replay retained A **7% better than
+a no-replay control** (loss 0.565 → 0.526) at a ~5% cost to task B — the expected
+trade. But forgetting without replay was only **+4%**, i.e. mild: LoRA on a 135M
+model barely drifts. The direction and the mechanism are proven; the effect size
+at full fine-tuning or 7B+ is unproven on a 4 GB box.
+
 ## Loop Hardening
 
 Six surfaces protect the training loop from the failure modes that cost a real GPU-hour. The schema + math kernels shipped in v0.70.0; the live trainer-callback wiring shipped in **v0.71.11**, validated end-to-end on SmolLM2-135M.

{soup_cli-0.71.35 → soup_cli-0.71.36}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.71.35"
+version = "0.71.36"
 description = "Fine-tune and post-train LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.71.35 → soup_cli-0.71.36}/src/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune and post-train LLMs in one command."""
 
-__version__ = "0.71.35"
+__version__ = "0.71.36"

{soup_cli-0.71.35 → soup_cli-0.71.36}/src/soup_cli/cli.py

@@ -521,6 +521,15 @@ from soup_cli.commands import data_doctor as _data_doctor_cmd  # noqa: E402
 data.app.command(name="doctor")(_data_doctor_cmd.doctor)
 data.app.command(name="lint")(_data_doctor_cmd.lint)
 
+# v0.71.36 — Data Moat II: topic map + Secret-Sharer canaries.
+from soup_cli.commands import data_topics as _data_topics_cmd  # noqa: E402
+
+data.app.command(name="topics")(_data_topics_cmd.topics)
+
+from soup_cli.commands import data_canary as _data_canary_cmd  # noqa: E402
+
+data.app.add_typer(_data_canary_cmd.app, name="canary")
+
 # v0.71.28 — MCP server: drive Soup from any MCP client (Claude Code / Cursor /
 # Cline / Continue) over stdio.
 from soup_cli.commands import mcp as _mcp_cmd  # noqa: E402