soup-cli 0.40.2__tar.gz → 0.40.4__tar.gz

{soup_cli-0.40.2 → soup_cli-0.40.4}/CONTRIBUTING.md

@@ -111,7 +111,7 @@ soup_cli/
   templates/          - 17 built-in soup.yaml templates (YAML + manifest.json) with load_template loader (v0.39.0, +bco v0.40.0)
   ui/                 - Web UI (FastAPI + HTML/JS SPA)
 
-tests/                - Test suite (143 files, 4756 tests)
+tests/                - Test suite (148 files, 4930 tests)
 examples/             - Real-world config examples and datasets
 ```
 

{soup_cli-0.40.2 → soup_cli-0.40.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: soup-cli
-Version: 0.40.2
+Version: 0.40.4
 Summary: Fine-tune LLMs in one command. No SSH, no config hell.
 Project-URL: Homepage, https://github.com/MakazhanAlpamys/Soup
 Project-URL: Repository, https://github.com/MakazhanAlpamys/Soup
@@ -134,17 +134,11 @@ soup train
 
 Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 
-**v0.40.2 — Quick polish + carry-overs**: closes 3 originally-scheduled GitHub issues (#36 eval-gate dashboard row, #50 smarter `--hf-resume`, #51 custom HF Space templates) plus 7 v0.40.1 long-tail UX papercuts.
+**v0.40.4 — trust_remote_code multi-trainer + multipack live wiring**: closes two carry-over gaps from earlier releases — the v0.36.0 known gap (only SFT honoured the opt-in) and the v0.40.3-deferred multipack sampler.
 
-- **`--hf-resume` prefers local newer** — `prepare_hf_resume` checks the highest local `checkpoint-N` against the highest remote branch and skips the snapshot download when the local copy is newer or equal. Saves bandwidth and never overwrites a fresher local checkpoint with stale Hub state.
-- **Eval-gate dashboard row** — pure formatter `format_gate_row(state)` lives in `soup_cli/monitoring/display.py` and renders a one-liner like `Gate: helpfulness 7.8 ✓ | math 0.82 ✗ (-0.06) | STOP` for the live training panel. Hidden when eval-gate is disabled.
-- **`soup deploy hf-space --template-dir <path>`** — supply your own `app.py` + `README.md` (+ optional `requirements.txt`) instead of the built-in `gradio-chat` / `streamlit-chat`. Containment-checked, repo-id substitution validated *before* render, 256 KB cap per file, symlinks rejected.
-- **`soup quickstart --output DIR`** — route data, config, and run dir under any directory you choose (default keeps backwards-compat in cwd).
-- **`soup runs --cwd-only`** — restrict the listing to runs whose `output_dir` is under the current directory; the global `~/.soup/experiments.db` view is still default.
-- **`soup infer` / `soup bench` accept HF ids** — when the local model path is missing AND the value isn't path-like (no `./`, `/`, `~`, `C:\`), it falls through to a HuggingFace download via `transformers.from_pretrained`. Path-like-but-missing surfaces a friendly `FileNotFoundError`.
-- **CLI flag aliases** — `data filter --min-coherence` (alias for `--coherence`); `data split --train` accepted (informational, train is the implicit remainder); `data register / unregister` accept positional `<name>` and `<path>` alongside the `--name` / `--path` options, with conflict detection.
-- **`--log-level` plumbing complete** — `apply_logging_level` now sets the root logger so third-party libraries (transformers / peft / trl) actually respect QUIET and DEBUG. The four tiers no longer produce byte-identical output.
-- **+36 net new tests** across the new helpers, security-fix follow-ups (path-containment in `register_data`, `bench.py`, `infer.py --output`, symlink-reject in custom Space templates), and Windows cross-drive edge cases.
+- **`--trust-remote-code` everywhere** — every non-SFT trainer (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO + the unified Preference dispatcher) now defaults to `trust_remote_code=False` and only enables custom-code execution when the user explicitly opts in. Same for `soup diff`, `soup export`, `soup merge`, `soup infer`, and `soup data generate`. The v0.36.0 `KNOWN_SAFE_PREFIXES` allowlist still suppresses the warning panel for first-party orgs; unknown-org local checkpoints with `auto_map` raise a friendly `ValueError` at construction time instead of silently exec'ing on `from_pretrained`.
+- **Multipack sampler — live in HF Trainer** — `make_multipack_trainer_class` adds a `get_train_dataloader` override that installs a `MultipackBatchSampler(real_batches=False)` as the DataLoader's `batch_sampler=`. The shape mismatch that blocked v0.40.3 (`Sampler[int]` vs `list[list[int]]`) is gone — the sampler now yields flat `list[int]` per packed sequence, which is what `DataLoader.batch_sampler` actually expects. SFT and Pretrain wrappers instantiate the multipack subclass when `multipack: true`; the v0.40.3 yellow advisory is gone.
+- **+75 net new tests** across the trainer×trust_remote_code matrix (parametrize over 12 wrappers × {default-off / opt-in / unknown-org rejection}), source-level invariants (no remaining `trust_remote_code=True` literal in any trainer file), the new DataLoader override (state-missing fallback + flat-pack-yield contract), and the real `transformers.Trainer` MRO mix-in.
 
 ## Why Soup?
 
@@ -661,7 +655,7 @@ training:
 
 **Architecture allowlist** — 18 supported (Llama 3.x, Qwen 2/3, Mistral, Gemma 2/3, Phi 3/4, DeepSeek V2/V3, Mixtral, Falcon, StableLM, SmolLM2). Unknown architectures **fail loudly at config-load** instead of silently no-opping (critical fix vs Axolotl's silent-miss footgun).
 
-**v0.37.0 scope:** schema gate + helper builder ship now. Live wiring of the sampler into HF Trainer's `_get_train_sampler` lands in v0.37.1 (mirrors v0.27.0 MII stub-then-live pattern). Multipack is **sft / pretrain only** on the `transformers` backend; preference / RLHF trainers and MLX backend get distinct error messages naming the actual reason.
+**Live wiring** — landed. SFT and Pretrain trainer wrappers actually instantiate the multipack subclass when `multipack: true` is set. The factory's `get_train_dataloader` override installs `MultipackBatchSampler(real_batches=False)` (yields a flat `list[int]` per packed sequence — DataLoader-compatible) as the DataLoader's `batch_sampler=`, forwarding `dataloader_drop_last`/`num_workers`/`pin_memory` from `TrainingArguments`. The `_get_train_sampler` override stays as a defensive no-op fallback that always delegates to super, so any HF eval / prediction loop bypassing `get_train_dataloader` still gets the correct `Sampler[int]` shape (no nested-list shape mismatch). Multipack is **sft / pretrain only** on the `transformers` backend; preference / RLHF trainers and MLX backend get distinct error messages naming the actual reason. Datasets must expose `input_ids` (preferred) or `length` per row; raw text triggers an all-zeros warning.
 
 **DoS hardening** — the FFD packer caps at 1M items (algorithm is O(N²) worst-case); the 4D mask builder caps allocations at 2³¹ cells; the chat-template Jinja analyzer caps at 128KB. Every numeric input rejects `bool` explicitly (matches v0.30.0+ project policy).
 
@@ -698,12 +692,19 @@ data:
 
 When the tokenizer ships a chat template with `{% generation %}` markers, the mask is exact. Without those markers, Soup falls back to an incremental tokenize-delta walk and documents the looseness.
 
-### `--trust-remote-code` opt-in
+### `--trust-remote-code` opt-in (every command, every trainer)
 
-`soup train`, `chat`, `serve`, `data download`, `eval auto` now require `--trust-remote-code` to load any HF model that ships custom Python (`auto_map` in `config.json`). First-party orgs (Meta, Mistral, Qwen, Google, etc.) suppress the warning panel; everything else prints a `REMOTE CODE WARNING` panel before loading.
+Every command that loads a model now requires `--trust-remote-code` to execute custom Python from a model repo (`auto_map` in `config.json`). First-party orgs (Meta, Mistral, Qwen, Google, etc.) suppress the warning panel; everything else prints a `REMOTE CODE WARNING` panel before loading. Unknown-org local checkpoints with `auto_map` raise a friendly `ValueError` at construction time instead of silently exec'ing inside `from_pretrained`.
+
+Coverage:
+- `soup train` (every task — SFT, DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, Reward Model, Pretrain, Embedding, BCO, and the unified Preference dispatcher)
+- `soup chat`, `soup serve`, `soup data download`, `soup eval auto`
+- `soup diff`, `soup export`, `soup merge`, `soup infer`, `soup data generate`
 
 ```bash
 soup train --config soup.yaml --trust-remote-code
+soup infer --model my-org/custom-arch-model --input prompts.jsonl --trust-remote-code
+soup export --model ./adapter --format gguf --trust-remote-code
 ```
 
 ### Chat-template hardening
@@ -2579,6 +2580,7 @@ soup serve --model <m> --structured-output json --json-schema s.json  Constraine
 soup serve --model <m> --structured-output regex --regex-pattern '...'  Regex-constrained output
 soup serve --model <m> --dashboard            Live dashboard + /metrics endpoint
 soup serve --model <m> --trace --trace-endpoint http://localhost:4317  OpenTelemetry tracing
+soup serve --model <m> --trace-log ./serve.jsonl  Per-request JSONL log + rotation + secret redaction
 POST /v1/adapters/activate/<name>             Hot-swap active LoRA adapter
 soup sweep --config soup.yaml --param lr=...  Hyperparameter search
 soup diff --model-a ./a --model-b ./b         Compare two models
@@ -2594,7 +2596,9 @@ soup data generate ... --provider anthropic   Use Claude API
 soup data generate ... --provider vllm        Use local vLLM server
 soup data generate ... --template code        Domain templates (code/conversation/qa/preference/reasoning)
 soup data generate ... --quality-pipeline     Auto validate + filter + dedup
-soup data augment <path> --strategy rephrase|translate|style  LLM-driven augmentationsoup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from tracessoup data review prefs.jsonl --sample 10      Preview preference pairssoup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
+soup data augment <path> --strategy rephrase|translate|style  LLM-driven augmentationsoup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from traces
+soup data from-traces ... --judge --min-confidence 0.7  LLM-judge confidence filter
+soup data review prefs.jsonl --sample 10      Preview preference pairssoup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
 soup data sample <path> --n 1000             Random sample subset
 soup data sample <path> --n 1000 --strategy diverse  Cluster-based diverse sampling
 soup data sample <path> --n 1000 --strategy hard     Sample hardest examples
@@ -2791,6 +2795,44 @@ pytest tests/ -v
 pytest tests/ -m smoke -v
 ```
 
+## Live CUDA Batch-Size Probe
+
+Set `auto_batch_size_strategy: probe` in `training:` and Soup will run a real OOM-probe before training:
+
+```yaml
+training:
+  batch_size: auto
+  auto_batch_size_strategy: probe
+```
+
+For each candidate size `B`, the probe runs ONE forward + backward + step on a synthetic batch of `B` sequences of length `max_length`. On `torch.cuda.OutOfMemoryError` it halves; otherwise it doubles up to `4 × static_estimate`. The picked size is cached per `(model, max_length, quantization, lora_r, gpu)` tuple in `~/.soup/batch_cache.json` so subsequent runs skip the probe.
+
+CPU sessions and `auto_batch_size_strategy: static` skip the probe. Synthetic batch tensors are freed before the backward pass so peak VRAM reflects the realistic training step. SFT-only this release — non-SFT trainers fall back to the static estimate.
+
+## Trace-to-Preference: LLM-Judge Filter
+
+`soup data from-traces --judge` filters harvested preference pairs through an LLM judge:
+
+```bash
+soup data from-traces \
+  --logs ./prod-traces.jsonl --format langchain --signal thumbs_up \
+  --output ./prefs.jsonl \
+  --judge --judge-provider ollama --judge-model llama3 \
+  --min-confidence 0.7
+```
+
+The judge scores `chosen` and `rejected` independently against its rubric (default helpfulness/accuracy/safety on a 1-5 scale). Pairs whose normalised `(chosen - rejected)` confidence falls below `--min-confidence` are dropped. Per-pair backend exceptions are counted (not crashed) and reported. Provider allowlist `{openai, server, ollama}` validated at the CLI boundary; SSRF protection on `--judge-api-base` carries over from `soup eval judge`.
+
+## Inference Server Trace Log
+
+`soup serve --trace-log <path>` writes a passive append-only JSONL log per chat completion:
+
+```bash
+soup serve --model ./out --trace-log ./serve-trace.jsonl --trace-log-cap-mb 100
+```
+
+Each line: `{"ts": ..., "prompt": ..., "response": ..., "latency_ms": ..., "tokens": ...}`. Path-containment validated, hard rotation cap (default 100 MB, one backup retained), symlink-reject on the backup path (TOCTOU defence), and `hf_*` / `sk-*` / `Bearer …` token shapes redacted to `<redacted>` before write. Failures (disk full, serialisation errors) never crash the request handler.
+
 ## Changelog
 
 See [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases) for version history.

{soup_cli-0.40.2 → soup_cli-0.40.4}/README.md

@@ -43,17 +43,11 @@ soup train
 
 Latest highlights only. Full history: [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases).
 
-**v0.40.2 — Quick polish + carry-overs**: closes 3 originally-scheduled GitHub issues (#36 eval-gate dashboard row, #50 smarter `--hf-resume`, #51 custom HF Space templates) plus 7 v0.40.1 long-tail UX papercuts.
-
-- **`--hf-resume` prefers local newer** — `prepare_hf_resume` checks the highest local `checkpoint-N` against the highest remote branch and skips the snapshot download when the local copy is newer or equal. Saves bandwidth and never overwrites a fresher local checkpoint with stale Hub state.
-- **Eval-gate dashboard row** — pure formatter `format_gate_row(state)` lives in `soup_cli/monitoring/display.py` and renders a one-liner like `Gate: helpfulness 7.8 ✓ | math 0.82 ✗ (-0.06) | STOP` for the live training panel. Hidden when eval-gate is disabled.
-- **`soup deploy hf-space --template-dir <path>`** — supply your own `app.py` + `README.md` (+ optional `requirements.txt`) instead of the built-in `gradio-chat` / `streamlit-chat`. Containment-checked, repo-id substitution validated *before* render, 256 KB cap per file, symlinks rejected.
-- **`soup quickstart --output DIR`** — route data, config, and run dir under any directory you choose (default keeps backwards-compat in cwd).
-- **`soup runs --cwd-only`** — restrict the listing to runs whose `output_dir` is under the current directory; the global `~/.soup/experiments.db` view is still default.
-- **`soup infer` / `soup bench` accept HF ids** — when the local model path is missing AND the value isn't path-like (no `./`, `/`, `~`, `C:\`), it falls through to a HuggingFace download via `transformers.from_pretrained`. Path-like-but-missing surfaces a friendly `FileNotFoundError`.
-- **CLI flag aliases** — `data filter --min-coherence` (alias for `--coherence`); `data split --train` accepted (informational, train is the implicit remainder); `data register / unregister` accept positional `<name>` and `<path>` alongside the `--name` / `--path` options, with conflict detection.
-- **`--log-level` plumbing complete** — `apply_logging_level` now sets the root logger so third-party libraries (transformers / peft / trl) actually respect QUIET and DEBUG. The four tiers no longer produce byte-identical output.
-- **+36 net new tests** across the new helpers, security-fix follow-ups (path-containment in `register_data`, `bench.py`, `infer.py --output`, symlink-reject in custom Space templates), and Windows cross-drive edge cases.
+**v0.40.4 — trust_remote_code multi-trainer + multipack live wiring**: closes two carry-over gaps from earlier releases — the v0.36.0 known gap (only SFT honoured the opt-in) and the v0.40.3-deferred multipack sampler.
+
+- **`--trust-remote-code` everywhere** — every non-SFT trainer (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO + the unified Preference dispatcher) now defaults to `trust_remote_code=False` and only enables custom-code execution when the user explicitly opts in. Same for `soup diff`, `soup export`, `soup merge`, `soup infer`, and `soup data generate`. The v0.36.0 `KNOWN_SAFE_PREFIXES` allowlist still suppresses the warning panel for first-party orgs; unknown-org local checkpoints with `auto_map` raise a friendly `ValueError` at construction time instead of silently exec'ing on `from_pretrained`.
+- **Multipack sampler — live in HF Trainer** — `make_multipack_trainer_class` adds a `get_train_dataloader` override that installs a `MultipackBatchSampler(real_batches=False)` as the DataLoader's `batch_sampler=`. The shape mismatch that blocked v0.40.3 (`Sampler[int]` vs `list[list[int]]`) is gone — the sampler now yields flat `list[int]` per packed sequence, which is what `DataLoader.batch_sampler` actually expects. SFT and Pretrain wrappers instantiate the multipack subclass when `multipack: true`; the v0.40.3 yellow advisory is gone.
+- **+75 net new tests** across the trainer×trust_remote_code matrix (parametrize over 12 wrappers × {default-off / opt-in / unknown-org rejection}), source-level invariants (no remaining `trust_remote_code=True` literal in any trainer file), the new DataLoader override (state-missing fallback + flat-pack-yield contract), and the real `transformers.Trainer` MRO mix-in.
 
 ## Why Soup?
 
@@ -570,7 +564,7 @@ training:
 
 **Architecture allowlist** — 18 supported (Llama 3.x, Qwen 2/3, Mistral, Gemma 2/3, Phi 3/4, DeepSeek V2/V3, Mixtral, Falcon, StableLM, SmolLM2). Unknown architectures **fail loudly at config-load** instead of silently no-opping (critical fix vs Axolotl's silent-miss footgun).
 
-**v0.37.0 scope:** schema gate + helper builder ship now. Live wiring of the sampler into HF Trainer's `_get_train_sampler` lands in v0.37.1 (mirrors v0.27.0 MII stub-then-live pattern). Multipack is **sft / pretrain only** on the `transformers` backend; preference / RLHF trainers and MLX backend get distinct error messages naming the actual reason.
+**Live wiring** — landed. SFT and Pretrain trainer wrappers actually instantiate the multipack subclass when `multipack: true` is set. The factory's `get_train_dataloader` override installs `MultipackBatchSampler(real_batches=False)` (yields a flat `list[int]` per packed sequence — DataLoader-compatible) as the DataLoader's `batch_sampler=`, forwarding `dataloader_drop_last`/`num_workers`/`pin_memory` from `TrainingArguments`. The `_get_train_sampler` override stays as a defensive no-op fallback that always delegates to super, so any HF eval / prediction loop bypassing `get_train_dataloader` still gets the correct `Sampler[int]` shape (no nested-list shape mismatch). Multipack is **sft / pretrain only** on the `transformers` backend; preference / RLHF trainers and MLX backend get distinct error messages naming the actual reason. Datasets must expose `input_ids` (preferred) or `length` per row; raw text triggers an all-zeros warning.
 
 **DoS hardening** — the FFD packer caps at 1M items (algorithm is O(N²) worst-case); the 4D mask builder caps allocations at 2³¹ cells; the chat-template Jinja analyzer caps at 128KB. Every numeric input rejects `bool` explicitly (matches v0.30.0+ project policy).
 
@@ -607,12 +601,19 @@ data:
 
 When the tokenizer ships a chat template with `{% generation %}` markers, the mask is exact. Without those markers, Soup falls back to an incremental tokenize-delta walk and documents the looseness.
 
-### `--trust-remote-code` opt-in
+### `--trust-remote-code` opt-in (every command, every trainer)
+
+Every command that loads a model now requires `--trust-remote-code` to execute custom Python from a model repo (`auto_map` in `config.json`). First-party orgs (Meta, Mistral, Qwen, Google, etc.) suppress the warning panel; everything else prints a `REMOTE CODE WARNING` panel before loading. Unknown-org local checkpoints with `auto_map` raise a friendly `ValueError` at construction time instead of silently exec'ing inside `from_pretrained`.
 
-`soup train`, `chat`, `serve`, `data download`, `eval auto` now require `--trust-remote-code` to load any HF model that ships custom Python (`auto_map` in `config.json`). First-party orgs (Meta, Mistral, Qwen, Google, etc.) suppress the warning panel; everything else prints a `REMOTE CODE WARNING` panel before loading.
+Coverage:
+- `soup train` (every task — SFT, DPO, GRPO, KTO, ORPO, SimPO, IPO, PPO, Reward Model, Pretrain, Embedding, BCO, and the unified Preference dispatcher)
+- `soup chat`, `soup serve`, `soup data download`, `soup eval auto`
+- `soup diff`, `soup export`, `soup merge`, `soup infer`, `soup data generate`
 
 ```bash
 soup train --config soup.yaml --trust-remote-code
+soup infer --model my-org/custom-arch-model --input prompts.jsonl --trust-remote-code
+soup export --model ./adapter --format gguf --trust-remote-code
 ```
 
 ### Chat-template hardening
@@ -2488,6 +2489,7 @@ soup serve --model <m> --structured-output json --json-schema s.json  Constraine
 soup serve --model <m> --structured-output regex --regex-pattern '...'  Regex-constrained output
 soup serve --model <m> --dashboard            Live dashboard + /metrics endpoint
 soup serve --model <m> --trace --trace-endpoint http://localhost:4317  OpenTelemetry tracing
+soup serve --model <m> --trace-log ./serve.jsonl  Per-request JSONL log + rotation + secret redaction
 POST /v1/adapters/activate/<name>             Hot-swap active LoRA adapter
 soup sweep --config soup.yaml --param lr=...  Hyperparameter search
 soup diff --model-a ./a --model-b ./b         Compare two models
@@ -2503,7 +2505,9 @@ soup data generate ... --provider anthropic   Use Claude API
 soup data generate ... --provider vllm        Use local vLLM server
 soup data generate ... --template code        Domain templates (code/conversation/qa/preference/reasoning)
 soup data generate ... --quality-pipeline     Auto validate + filter + dedup
-soup data augment <path> --strategy rephrase|translate|style  LLM-driven augmentationsoup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from tracessoup data review prefs.jsonl --sample 10      Preview preference pairssoup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
+soup data augment <path> --strategy rephrase|translate|style  LLM-driven augmentationsoup data from-traces --logs l.jsonl --format langchain --signal thumbs_up --output p.jsonl  Preference pairs from traces
+soup data from-traces ... --judge --min-confidence 0.7  LLM-judge confidence filter
+soup data review prefs.jsonl --sample 10      Preview preference pairssoup data filter <path> --coherence 0.3       Quality filter (perplexity/coherence)
 soup data sample <path> --n 1000             Random sample subset
 soup data sample <path> --n 1000 --strategy diverse  Cluster-based diverse sampling
 soup data sample <path> --n 1000 --strategy hard     Sample hardest examples
@@ -2700,6 +2704,44 @@ pytest tests/ -v
 pytest tests/ -m smoke -v
 ```
 
+## Live CUDA Batch-Size Probe
+
+Set `auto_batch_size_strategy: probe` in `training:` and Soup will run a real OOM-probe before training:
+
+```yaml
+training:
+  batch_size: auto
+  auto_batch_size_strategy: probe
+```
+
+For each candidate size `B`, the probe runs ONE forward + backward + step on a synthetic batch of `B` sequences of length `max_length`. On `torch.cuda.OutOfMemoryError` it halves; otherwise it doubles up to `4 × static_estimate`. The picked size is cached per `(model, max_length, quantization, lora_r, gpu)` tuple in `~/.soup/batch_cache.json` so subsequent runs skip the probe.
+
+CPU sessions and `auto_batch_size_strategy: static` skip the probe. Synthetic batch tensors are freed before the backward pass so peak VRAM reflects the realistic training step. SFT-only this release — non-SFT trainers fall back to the static estimate.
+
+## Trace-to-Preference: LLM-Judge Filter
+
+`soup data from-traces --judge` filters harvested preference pairs through an LLM judge:
+
+```bash
+soup data from-traces \
+  --logs ./prod-traces.jsonl --format langchain --signal thumbs_up \
+  --output ./prefs.jsonl \
+  --judge --judge-provider ollama --judge-model llama3 \
+  --min-confidence 0.7
+```
+
+The judge scores `chosen` and `rejected` independently against its rubric (default helpfulness/accuracy/safety on a 1-5 scale). Pairs whose normalised `(chosen - rejected)` confidence falls below `--min-confidence` are dropped. Per-pair backend exceptions are counted (not crashed) and reported. Provider allowlist `{openai, server, ollama}` validated at the CLI boundary; SSRF protection on `--judge-api-base` carries over from `soup eval judge`.
+
+## Inference Server Trace Log
+
+`soup serve --trace-log <path>` writes a passive append-only JSONL log per chat completion:
+
+```bash
+soup serve --model ./out --trace-log ./serve-trace.jsonl --trace-log-cap-mb 100
+```
+
+Each line: `{"ts": ..., "prompt": ..., "response": ..., "latency_ms": ..., "tokens": ...}`. Path-containment validated, hard rotation cap (default 100 MB, one backup retained), symlink-reject on the backup path (TOCTOU defence), and `hf_*` / `sk-*` / `Bearer …` token shapes redacted to `<redacted>` before write. Failures (disk full, serialisation errors) never crash the request handler.
+
 ## Changelog
 
 See [GitHub Releases](https://github.com/MakazhanAlpamys/Soup/releases) for version history.

{soup_cli-0.40.2 → soup_cli-0.40.4}/SECURITY.md

@@ -9,7 +9,7 @@ We provide security updates for the following versions:
 - **Versions older than 3 minor versions:** No support
 
 Example:
-- v0.40.2 -- Full support (latest)
+- v0.40.4 -- Full support (latest)
 - v0.40.0-v0.40.x -- Full support
 - v0.39.0-0.39.x -- Bug-fix support only
 - v0.38.0-0.38.x -- Bug-fix support only
@@ -144,6 +144,8 @@ No known critical vulnerabilities in current releases.
 - **v0.32.0 — Training Stability & Auto-Tuning**: `--find-lr-output` containment via shared `utils/paths.is_under_cwd` (prevents writes outside cwd); `save_lr_finder_report` rejects NaN / Infinity floats in `lrs` / `losses` and serialises with `allow_nan=False` (keeps the report parser-safe); `compute_lr_schedule` rejects non-positive `start_lr`, inverted ranges, and `num_steps` outside `[2, 10_000]`; `pick_mixed_precision` rejects empty / null-byte / >200-char model names and resolves multi-version quirks (`qwen2.5` vs `qwen2`, `phi-3.5` vs `phi-3`) by longest-substring-first iteration so an added family can never accidentally make a more-specific entry dead code; `compute_warmup_steps` clamps to `[10, 1000]` with a `ratio==0.0` short-circuit matching HF Trainer's "no warmup" convention; `SpikeRecoveryStrategy` is `@dataclass(frozen=True)` (post-construction mutation cannot bypass validation), `max_attempts ∈ [1, 10]`, `lr_decay ∈ (0, 1)`, `min_lr > 0`; cross-validator `_validate_spike_recovery_requires_watchdog` rejects `loss_spike_recovery=true, loss_watchdog=false` at config-load (fails fast instead of never triggering); `convergence_window ∈ [5, 10_000]`, `convergence_rel_tol ∈ (0, 1]`, `recommend_action` reuses `detect_plateau` so plateau heuristic stays single-source-of-truth; `GradAccumMonitor.recommend()` caps doubled `accum` at `MAX_ACCUM=1024` so a runaway advisory loop cannot blow up DataLoader prefetch; `generate_config` validates BOTH the YAML output path AND the embedded `decisions["output"]` field via `is_under_cwd` (closes the gap where a crafted `decisions["output"]="../../etc"` would have silently propagated into the rendered YAML)
 - **v0.34.0 — Observability & Dev UX**: `.crash` bundle generator (`utils/crash.py`) recursively redacts `hf_*` / `sk-*` / `Bearer …` token-shaped strings in any captured `config` and metric tail before serialisation, so a `.crash` file shared on a public GitHub issue cannot leak credentials; `output_dir` is reduced to `os.path.basename` so `$HOME` doesn't leak; `write_crash_bundle` uses `os.path.realpath + commonpath` for cwd containment (Windows-safe; raises `ValueError` not `PermissionError` so callers cannot silently swallow with `except OSError`); filename appends `secrets.token_hex(4)` so two crashes in the same UTC second don't collide; bundle truncated to `MAX_BUNDLE_BYTES=1_000_000`. `train.py` crash-write surfaces failures to the user (no silent missing-bundle). `profiling.py` `resolve_trace_path` rejects empty / `.` / `..` / `/` / `\\` / null-byte `run_id` (closes the `output_dir/profiles/../trace.json` escape) and uses `os.path.realpath + is_under_cwd`; profiles dir is created only on successful torch import (no stale empty dirs on torch-less CI). `tracker.get_run` LIKE-prefix match escapes `%` / `_` / `\\` and uses `ESCAPE '\\'` so a crafted `run_id` cannot widen the match (mirrors v0.26.0 registry policy). Lazy schema migration (`_ensure_schema`) tolerates the "duplicate column" race when two CLI processes start simultaneously on a fresh DB (fork-based multi-GPU training, TUI auto-refresh). `runs.py show/replay/clean` switched user `run_id` rendering to `markup_escape` and switched `clean` containment from broken `Path.resolve() + relative_to()` to project-standard `os.path.realpath + is_under_cwd`. `tui_app.py` lazy-imports `ExperimentTracker` and `markup_escape`s every DB-sourced string before passing into Textual widgets so a crafted base_model / experiment_name cannot inject `[bold red]…[/]` markup. `run_cost.estimate_run_cost_usd` rejects `bool` in `num_gpus` (bool is a subclass of int — same defence as v0.30.0 `Candidate.__post_init__`); duration clamped to `[0, 1 year]`; unknown GPU returns `None` so callers render `—` instead of fabricating `$0.00`. `log_level.parse_log_level` rejects non-string + null-byte input.
 - **v0.33.0 — Live Wire**: RLVR `code_exec_reward` adds OS-level isolation (Linux best-effort `os.unshare(CLONE_NEWUSER|CLONE_NEWNET|CLONE_NEWPID)`, macOS `sandbox-exec` with default-deny `MACOS_SANDBOX_PROFILE` narrowed to a 3-name `mach-lookup` allowlist to prevent DNS / NSURLSession bypass of `(deny network*)`); `prune_checkpoints` switches to TOCTOU-safe `os.lstat + S_ISLNK` + `shutil.rmtree(onerror=_abort_on_symlink)` so a symlink encountered mid-walk aborts rather than escapes; `run_gate` wraps each task scorer in a typed `try/except` so backend failures produce `score=None, error=str(exc)` (never silent `score=1.0`); `_parse_judge_url` removes the bare `http://` catch-all (defence-in-depth after the Pydantic GateTask validator); `soup can run` requires `--yes` or explicit consent callback and raises `ValueError` (not `PermissionError`, which is an `OSError` subclass that broad `except` blocks would swallow); GGUF `rglob` result for ollama deploy is `realpath+commonpath` checked against extract_dir (prevents symlink escape from a crafted can); `DeployTarget.path` validator normalises mixed `\\`/`/` separators before splitting (closes a Windows `..` bypass); `CAN_FORMAT_VERSION` 1→2 (additive — v1 still loads); `soup can publish` validates `repo_id` via `utils/hf.validate_repo_id`, resolves token via `resolve_token`, sanitises commit messages (first-line, 200-char cap), uses HTTPS-only HfApi; `_write_spike_recovery_hint` adds `is_under_cwd` containment check on `args.output_dir` from raw HF `TrainingArguments`; `lookup_entry_by_output_dir` emits `ResourceWarning` when 1000-row scan limit is hit (no silent miss); `CrossDocCollator` no longer mutates input feature dicts (HF Dataset rows are cached and reused — mutation broke subsequent batches); `Candidate` rejects `bool` in `score`/`latency_ms` (was sneaking past `int` isinstance check); `evaluate_candidate` latency mean now divides by *completed* prompts (excludes crashed) so a broken candidate isn't artificially fast; `auto_quant.run_auto_quant_picker` soft-falls-back to highest-scored candidate when no candidate clears `min_score` (server still binds); `build_logits_processors` returns `[]` when neither `outlines` nor `lm-format-enforcer` is installed (server degrades to free-form rather than 500); MII server uses loopback-only CORS, max_tokens cap [1, 16384], stream rejection, generic 500 with no stack-trace leak; `os.execvp` auto-reexec uses list args (no shell), all forwarded flags pre-validated; `cleanup_extract_dir` uses `os.path.commonpath` (Windows-safe) instead of `startswith`; `_run_subprocess` catches `TimeoutExpired` and returns rc=124 (coreutils convention) instead of an unhandled traceback; new `eval_results` and `tensorrt` artifact kinds in `RegistryStore._VALID_KINDS`
+- **v0.40.4 — trust_remote_code multi-trainer + multipack live**: closes the v0.36.0 #63 known gap by extending the `--trust-remote-code` opt-in across every non-SFT trainer wrapper (DPO / GRPO / KTO / ORPO / SimPO / IPO / PPO / RewardModel / Pretrain / Embedding / BCO + the unified `PreferenceTrainerWrapper` dispatcher) and the 5 standalone commands (`soup diff`, `soup export`, `soup merge`, `soup infer`, `soup data generate`). Pattern (15 sites): each `__init__` resolves once via `model_requires_trust_remote_code(config.base) or False` + `resolve_trust_remote_code(...)` and stores `self._trust_remote_code` — every `from_pretrained` call site now reads the resolved value (no remaining `trust_remote_code=True` literal in any trainer file; source-level invariant test in `tests/test_v0404_part_a.py`). `commands/train.py` no longer carries the v0.36.0 `sft_kwargs` split; `trust_remote_code` is part of the unified `trainer_kwargs` dict that flows to every trainer regardless of task. `_load_reward_model` (module-level helper in `ppo.py`) accepts a `trust_remote_code: bool` parameter and resolves internally — design intent is that the helper is independently safe to call from outside `PPOTrainerWrapper`. `PreferenceTrainerWrapper` dispatcher forwards the raw bool to the inner DPO/SimPO/ORPO/IPO/BCO wrapper kwargs at both `_build_inner` and `_build_multi_objective` sites; the resolver fires inside the inner wrapper at construction time. `_export_onnx` / `_export_tensorrt` / `_export_awq` / `_export_gptq` and `_merge_adapter` helpers all gain a `trust_remote_code: bool = False` parameter threaded from the Typer flag. Multipack live HF Trainer wiring (#65) lands via a new `get_train_dataloader` override on `make_multipack_trainer_class` that installs `MultipackBatchSampler(real_batches=False)` (yields flat `list[int]` per pack — DataLoader-compatible) as the DataLoader's `batch_sampler=`. The override forwards `args.dataloader_drop_last` / `dataloader_num_workers` / `dataloader_pin_memory` from `TrainingArguments`. `_get_train_sampler` override stays as a defensive no-op fallback that ALWAYS delegates to super (review-fix: a multipack `list[list[int]]` from this method would cause a shape mismatch if any HF eval / prediction loop bypasses `get_train_dataloader`). The state-presence guard switched from falsy (`not max_seq`) to explicit `is None` + `not lengths` (defensively rejects only-None and empty-list cases — non-positive ints already rejected upstream by `attach_multipack_state`). Falls back to `super().get_train_dataloader()` when state is missing OR when `train_dataset` is unset (defence-in-depth so the subclass remains safe to instantiate). Known limitations: (1) `multipack: true` requires the dataset to expose `input_ids` (preferred) or `length` per row — un-tokenized text-only datasets trigger the v0.40.3 all-zeros WARNING and the `MultipackBatchSampler` will reject the run. (2) The DataLoader override does NOT thread FSDP / DeepSpeed parallelism env hints from `super().get_train_dataloader()`, so distributed `multipack: true` runs are still untested under FSDP / ZeRO; tracked for v0.40.5+ paired with v0.42.0 multi-GPU work. (3) `_live_lr_sweep_from_config` in `commands/train.py` still hardcodes `trust_remote_code=False` for the LR sweep's internal model load — defensive but means `--find-lr` cannot consume custom-code models even with the user opt-in (defence-in-depth, not a bypass). (4) Each non-SFT trainer's `__init__` repeats the resolver block (10 sites) — code-quality refactor candidate (single shared helper) deferred to a future patch to keep the v0.40.4 diff focused on the gap closure.
+- **v0.40.3 — Stub-to-live**: New `soup_cli/utils/batch_probe.py:make_cuda_probe_fn` builds a CUDA probe closure that runs ONE forward+backward+step on a synthetic batch per candidate; `model.zero_grad(set_to_none=True)` runs BEFORE forward (defends against the synthetic backward accumulating into the live training model's grad buffers — matches v0.35.0 #45 `benchmark_kernel_combos` policy); intermediate `ids/attn/labels/outputs` are `del`-ed before `loss.backward()` so peak VRAM reflects a realistic training step; `bool` rejected on `batch_size` and `max_length`; `max_length < 8` rejected; `torch.cuda.OutOfMemoryError` returns False, other exceptions propagate; returns `None` (no-op) on non-CUDA / no-torch / missing model or tokenizer. New `soup_cli/utils/multipack_trainer.py:make_multipack_trainer_class` is `lru_cache`d so two calls with the same `base_cls` return the same subclass (consistent `isinstance`, pickle-safe); `attach_multipack_state` rejects `bool` on `max_seq_len`/`batch_size`/`seed` and rejects empty `lengths`; `lengths_from_dataset` logs WARNING when every row produces 0 (loud-fail mirrors v0.37.0 multipack arch allowlist — prevents silent NaN-loss footgun); `_get_train_sampler` override accepts `*args, **kwargs` for HF >= 4.41 signature compat. **Live wiring of the sampler into SFT / Pretrain trainer wrappers is deferred to v0.40.4** — adversarial 5th-pass review surfaced a `Sampler[int]` vs `list[list[int]]` shape mismatch with HF Trainer's DataLoader; the wrappers currently print a yellow advisory and fall back to the standard sampler when `multipack: true`. New `soup_cli/data/traces/quality.py:judge_filter_pairs` reuses v0.19.0 `JudgeEvaluator` SSRF protections; threshold rejects `bool` / NaN / out-of-`[0,1]`; `_MAX_BATCH=100_000` cap applied via lazy `itertools.islice` (never fully materialises a malicious / pathological generator); per-pair backend exceptions caught and logged at DEBUG (matches v0.33.0 #47 `CrossDocCollator` policy — never silently crash the harvest); `judge_provider` validated against `VALID_PROVIDERS` allowlist at the CLI boundary BEFORE constructor, with a Rich-escape error on mismatch. New `soup_cli/monitoring/trace_logger.py:TraceLogWriter` is thread-safe (single `threading.Lock` — multi-worker `--workers 4` documented as a single-process limitation); path containment via shared `is_under_cwd`; null-byte / empty / non-string path rejected; `cap_mb` bounds `[1, 10000]` with explicit `bool` rejection; rotation: when `current + extra > cap_bytes`, rename to `<path>.1` (one backup retained); symlink at the backup path is rejected via `os.lstat + stat.S_ISLNK` (matches v0.33.0 #22 TOCTOU policy) — defends against pre-placed `<log>.1 -> /etc/cron.d/x` overwrite. Secret redaction: prompt + response strings passed through `_SECRET_RE` matching `hf_*` (≥8), `sk-*` (≥16), and `Bearer …` (≥8) — replaces matches with `<redacted>` before serialisation (mirrors v0.34.0 `crash.py` policy). `--trace-log` constructor error messages in `commands/serve.py` are `rich.markup.escape`d before printing so a crafted path name cannot inject Rich markup. Unserialisable entries dropped silently; disk-full / OSError on write never crashes the request handler (passive log). Known limitations: live CUDA probe is wired in SFT only; multipack live wiring covers SFT+Pretrain only; `TraceLogWriter` retains exactly ONE backup file (operators wanting longer retention should use external rotation); custom HF Space templates from v0.40.2 still always create the Space with `space_sdk="gradio"` (tracked for v0.40.4+).
 - **v0.40.2 — Quick polish + carry-overs**: New `soup_cli/utils/hf_space.py:render_custom_template_dir` enforces `is_under_cwd` containment on the template directory; `validate_repo_id` runs BEFORE `{MODEL_REPO}` substitution (matches v0.29.0 Part F policy); per-file 256 KB cap (matches v0.39.0 Part E template-size policy); only `app.py` / `README.md` / `requirements.txt` are read (closed allowlist — no path-from-user-data). Symlinks rejected via `os.lstat + stat.S_ISLNK` and non-regular files (FIFO / device) also rejected (matches v0.33.0 #22 prune_checkpoints TOCTOU policy) — defends against `<template_dir>/app.py -> /etc/passwd`. `_find_highest_local_checkpoint` reads `output_dir` after caller's `is_under_cwd` validation (in `prepare_hf_resume`) and silently drops non-directories + OSError. `prepare_hf_resume` skips the snapshot download when local `checkpoint-N >= remote checkpoint-N` (saves bandwidth and never overwrites a fresher local checkpoint). `commands/data.py:register_data` containment switched from `Path.resolve() + relative_to()` to shared `is_under_cwd` (Windows 8.3 short-name safety per CLAUDE.md project rule); same fix applied to `commands/bench.py` prompts-file containment. `commands/infer.py:--output` now containment-checked via `is_under_cwd` (late-evaluated after model+input validation so pre-existing `tmp_path` test contracts keep working). `commands/quickstart.py:--output` validates target dir via `is_under_cwd` before `mkdir(parents=True)`; rejects out-of-cwd targets with friendly message. `commands/runs.py:_filter_runs_by_cwd` uses `os.path.realpath + commonpath`, catches `(ValueError, OSError)` so cross-drive paths on Windows (`D:\runs` vs `C:\project`) drop silently rather than crash. `monitoring/display.py:format_gate_row` uses explicit `task.get("passed") is True` so a missing `"passed"` field renders neutrally instead of as a false-y red ✗. `commands/infer.py:_resolve_model_source` heuristic for HF-id-vs-local-path: only falls through to HF when value is NOT path-like (no `./`, `/`, `\\`, `~`, no Windows drive letter, non-empty); path-like-but-missing raises `FileNotFoundError` so users see actionable errors instead of confusing HF download attempts. Known limitation: custom HF Space templates always create the Space with `space_sdk="gradio"` regardless of the supplied `app.py` (no `--sdk` flag in this release; combine `--template streamlit-chat` with the inline registry for Streamlit Spaces). Tracked for v0.40.3+.
 - **v0.40.1 — QA Hardening**: `soup_cli/utils/encoding.force_utf8_stdio` reconfigures Windows stdout/stderr to UTF-8 before any Rich Console is constructed; `os.environ.setdefault("PYTHONIOENCODING", "utf-8")` preserves user override; `(OSError, ValueError, AttributeError)` swallowed on detached streams; POSIX no-op. `SoupConfig._remap_root_level_misplaced_keys` (model_validator, mode='before') migrates root-level `lora:` into `training.lora` so nested validators (including `lora.init_strategy: Literal["random","pissa","olora"]`) actually fire — closes a footgun where the misplaced key was silently dropped. Caller's dict is never mutated (shallow-copy policy mirroring v0.33.0 #47 / v0.40.0 Part B). `PreferenceTrainerWrapper._build_multi_objective` replaces the v0.40.0 `NotImplementedError` stub with a primary-loss approximation; `validate_weight_compat` rejects BCO mixed with paired losses at runtime (data-format incompatible). `combine_losses` rejects empty weights, propagates NaN loudly (no silent zeroing), and rejects `bool` weight values (matches v0.30.0 `Candidate` policy). `_probe_cache_param_count` rejects empty / null-byte model names before path construction (mirrors v0.26.0 registry / v0.39.0 ReLoRAPolicy policy). `commands/doctor` flags `transformers ≥ 5.0.0` as INCOMPATIBLE via `_MAX_EXCLUSIVE` table; `_version_ge` parses leading-int chunks so `5.0.0.dev0` correctly trips the cap. `_detect_gpu_hw_without_torch_cuda` calls `nvidia-smi` via argv list (no shell), 5s timeout, `OSError` / `TimeoutExpired` caught; GPU label from `nvidia-smi` stdout is `rich.markup.escape`d before embedding in Rich-markup string (a real GPU name like `NVIDIA Quadro [T4]` cannot break or inject markup). `_detect_dual_python_interpreters` uses `os.path.realpath` (not `Path.resolve()`) for Windows 8.3 short-name compat. `_pick_quickstart_model` swaps TinyLlama-1.1B → SmolLM2-135M when `total_memory ≤ 6 GB` (prevents step-0 OOM on RTX 3050 4 GB / similar). `_live_lr_sweep_from_config` switched broken `load_local` import to `load_raw_data` (previously always silently fell back to a static placeholder curve). `commands/migrate` rejects `.jsonl` input (with first-line `{` sniff) with exit-2 friendly error; `.jsonl`-only suffix gate prevents false-positives on `.ipynb` notebooks. `commands/eval custom -o` is now honored independently of `--attach-to-registry`; loop-shadow regression where `output = generate_fn(...)` overwrote the CLI option fixed (variable renamed to `response`). `_load_jsonl` switched from `utf-8` to `utf-8-sig` so PowerShell `Out-File -Encoding utf8`-produced JSONL no longer fails first-row parse. Known limitation: `--trust-remote-code` opt-in surface still excludes 10 non-SFT trainers + 5 commands (v0.36.0 #63 carry-over).
 - **v0.40.0 — Preference Variety**: New `task='bco'` (Binary Classifier Optimization) and `task='preference'` (unified dispatcher). New schema fields: `bco_beta` (gt=0), `preference_loss: Literal[dpo,simpo,orpo,ipo,bco]|None`, `preference_loss_weights: Optional[Dict[str,float]]`, `dpo_beta_schedule: Literal[linear,cosine,exponential]|None`, `dpo_beta_end: float, gt=0|None`, `dpo_ref_regen_epochs: int [1,1000]|None`. Cross-validators: `_validate_preference_dispatcher` rejects setting either `preference_loss` or `preference_loss_weights` outside `task='preference'` (closes ordering-dependency between Part B/D validators); `_validate_dpo_variants_supported_tasks` gates β-schedule + ref-regen to DPO-family tasks (`dpo`, `ipo`, or `preference` + `preference_loss in {dpo, ipo}`); rejected on mlx backend with distinct error message (matches v0.34.0 distinct-reason policy); `_validate_preference_loss_weights` enforces 2–5 entries (single-entry rejected with actionable message pointing at scalar `preference_loss`), key allowlist `{dpo, simpo, orpo, ipo, bco}`, explicit null-byte rejection on keys (matches v0.39.0 rank_pattern policy), per-value bounds `(0, 1]`, weights must sum to 1.0 (±1e-6), mutually exclusive with scalar `preference_loss`, rejected on mlx backend. `compute_beta_at_step` rejects `bool` on `step` and `total_steps` (project bool-as-int policy from v0.30.0). `BetaScheduleCallback` resolves `total_steps` lazily in `on_train_begin` so the schedule sees the real `state.max_steps` populated by HF Trainer (closes a first-cut silent-no-op bug where total_steps=0 emitted beta_end for every step). `RefModelRegenCallback._regenerate` uses `strict=True` on `load_state_dict` and logs at WARNING on mismatch (closes a first-cut silent partial-copy hazard where strict=False could produce a hybrid old-base + new-LoRA reference); epoch 0 regen suppressed (avoids copying untrained student); trainer `.beta` assignment swallow narrowed to `AttributeError` only. `PreferenceTrainerWrapper._make_inner_cfg` uses `model_copy` (not `model_dump`+`model_validate`) so re-validation never sees an inconsistent intermediate state and the caller's `cfg` is never mutated (mirrors v0.33.0 #47 immutability policy). `_split_dpo_rows_to_bco` skipped-row count emitted at DEBUG so production silent-degradation is inspectable (mirrors v0.33.0 #47 CrossDocCollator policy). Multi-objective live runtime weighted-loss combination is deferred to v0.40.1: `PreferenceTrainerWrapper.setup` raises `NotImplementedError` with a friendly message naming the deferred-version follow-up (mirrors v0.27.0 MII / v0.37.0 multipack / v0.38.0 quant menu / v0.39.0 ReLoRA stub-then-live pattern). Known limitation: `BCOTrainerWrapper._setup_transformers` still hardcodes `trust_remote_code=True` (v0.36.0 #63 known-gap family carry-over across non-SFT trainers).

{soup_cli-0.40.2 → soup_cli-0.40.4}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "soup-cli"
-version = "0.40.2"
+version = "0.40.4"
 description = "Fine-tune LLMs in one command. No SSH, no config hell."
 readme = "README.md"
 license = "Apache-2.0"

{soup_cli-0.40.2 → soup_cli-0.40.4}/soup_cli/__init__.py

@@ -1,3 +1,3 @@
 """Soup CLI — Fine-tune LLMs in one command."""
 
-__version__ = "0.40.2"
+__version__ = "0.40.4"

{soup_cli-0.40.2 → soup_cli-0.40.4}/soup_cli/commands/data.py

@@ -1586,6 +1586,26 @@ def from_traces_cmd(
         "prefs.jsonl", "--output", "-o",
         help="Output path for preference pairs (JSONL)",
     ),
+    judge: bool = typer.Option(
+        False, "--judge",
+        help="Filter pairs via LLM-as-a-judge confidence (v0.40.3 #33).",
+    ),
+    judge_provider: str = typer.Option(
+        "openai", "--judge-provider",
+        help="Judge backend: openai | server | ollama. Used with --judge.",
+    ),
+    judge_model: str = typer.Option(
+        "gpt-4o-mini", "--judge-model",
+        help="Judge model id (e.g. 'gpt-4o-mini', 'llama3', 'qwen2.5'). Used with --judge.",
+    ),
+    judge_api_base: Optional[str] = typer.Option(
+        None, "--judge-api-base",
+        help="Judge API base URL. SSRF-protected. Used with --judge.",
+    ),
+    min_confidence: float = typer.Option(
+        0.7, "--min-confidence",
+        help="Drop pairs with judge-confidence below this threshold (0.0 - 1.0).",
+    ),
 ) -> None:
     """Harvest preference pairs from production traces (v0.26.0 Part C).
 
@@ -1666,6 +1686,53 @@ def from_traces_cmd(
             trace_iter = parse_openai(events)
 
     pairs = list(build_pairs(trace_iter, signal=signal))
+
+    if judge:
+        # v0.40.3 (#33 (a)) — LLM-judge confidence filter.
+        from soup_cli.data.traces.quality import judge_filter_pairs
+        from soup_cli.eval.judge import VALID_PROVIDERS, JudgeEvaluator
+
+        # Friendly early validation matches the existing CLI conventions —
+        # fall through to the constructor only after the obvious typo is caught.
+        if judge_provider not in VALID_PROVIDERS:
+            console.print(
+                f"[red]--judge-provider '{_escape(judge_provider)}' is invalid. "
+                f"Choose: {', '.join(sorted(VALID_PROVIDERS))}[/]"
+            )
+            raise typer.Exit(1)
+
+        try:
+            judge_evaluator = JudgeEvaluator(
+                provider=judge_provider,
+                model=judge_model,
+                api_base=judge_api_base,
+            )
+        except ValueError as exc:
+            console.print(f"[red]--judge config error:[/] {_escape(str(exc))}")
+            raise typer.Exit(1) from exc
+
+        # Cost-shock warning: each pair → TWO judge calls (chosen + rejected).
+        projected = len(pairs) * 2
+        console.print(
+            f"[yellow]Judge filter will issue ~{projected} backend calls "
+            f"({len(pairs)} pairs × 2). Cost depends on provider and model. "
+            f"Use --min-confidence to tune throughput.[/]"
+        )
+
+        try:
+            filtered, report = judge_filter_pairs(
+                pairs, judge=judge_evaluator, min_confidence=min_confidence,
+            )
+        except (TypeError, ValueError) as exc:
+            console.print(f"[red]--judge runtime error:[/] {_escape(str(exc))}")
+            raise typer.Exit(1) from exc
+
+        console.print(
+            f"[green]Judge filter:[/] kept={report.kept} dropped={report.dropped} "
+            f"errors={report.errors} (min_confidence={min_confidence:.2f})"
+        )
+        pairs = filtered
+
     with output_path.open("w", encoding="utf-8") as fh:
         for pair in pairs:
             fh.write(_json.dumps(pair.to_jsonl_dict(), ensure_ascii=False) + "\n")

{soup_cli-0.40.2 → soup_cli-0.40.4}/soup_cli/commands/diff.py

@@ -69,6 +69,14 @@ def diff(
         "-o",
         help="Save results to JSONL file",
     ),
+    trust_remote_code: bool = typer.Option(
+        False,
+        "--trust-remote-code",
+        help=(
+            "Allow loading models that ship custom Python via auto_map. "
+            "Default deny (v0.36.0). Only enable if you trust the source."
+        ),
+    ),
 ):
     """Compare outputs of two models side-by-side on the same prompts."""
     # Validate model paths
@@ -105,9 +113,13 @@ def diff(
 
     # Load models
     console.print("[dim]Loading Model A...[/]")
-    model_obj_a, tokenizer_a = _load_model(str(path_a), base_a, device)
+    model_obj_a, tokenizer_a = _load_model(
+        str(path_a), base_a, device, trust_remote_code,
+    )
     console.print("[dim]Loading Model B...[/]")
-    model_obj_b, tokenizer_b = _load_model(str(path_b), base_b, device)
+    model_obj_b, tokenizer_b = _load_model(
+        str(path_b), base_b, device, trust_remote_code,
+    )
     console.print("[green]Both models loaded.[/]\n")
 
     # Run comparison
@@ -199,11 +211,21 @@ def _collect_prompts(prompts_file: Optional[str], prompt_args: Optional[list[str
     return result
 
 
-def _load_model(model_path: str, base_model: Optional[str], device: str):
+def _load_model(
+    model_path: str,
+    base_model: Optional[str],
+    device: str,
+    trust_remote_code: bool = False,
+):
     """Load a model and tokenizer."""
     import torch
     from transformers import AutoModelForCausalLM, AutoTokenizer
 
+    from soup_cli.utils.trust_remote import (
+        model_requires_trust_remote_code,
+        resolve_trust_remote_code,
+    )
+
     path = Path(model_path)
     adapter_config_path = path / "adapter_config.json"
     is_adapter = adapter_config_path.exists()
@@ -220,7 +242,16 @@ def _load_model(model_path: str, base_model: Optional[str], device: str):
         console.print(f"[red]Cannot detect base model for {path}. Use --base-a/--base-b.[/]")
         raise typer.Exit(1)
 
-    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
+    probe_target = base_model or model_path
+    requires = model_requires_trust_remote_code(model_path) or False
+    trc = resolve_trust_remote_code(
+        probe_target,
+        requested=trust_remote_code,
+        console=console,
+        requires_remote_code=requires,
+    )
+
+    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=trc)
     if tokenizer.pad_token is None:
         tokenizer.pad_token = tokenizer.eos_token
 
@@ -229,7 +260,7 @@ def _load_model(model_path: str, base_model: Optional[str], device: str):
 
         base = AutoModelForCausalLM.from_pretrained(
             base_model,
-            trust_remote_code=True,
+            trust_remote_code=trc,
             device_map="auto",
             dtype=torch.float16,
         )
@@ -237,7 +268,7 @@ def _load_model(model_path: str, base_model: Optional[str], device: str):
     else:
         model_obj = AutoModelForCausalLM.from_pretrained(
             model_path,
-            trust_remote_code=True,
+            trust_remote_code=trc,
             device_map="auto",
             dtype=torch.float16,
         )