world-model-optimizer 0.2.0__tar.gz → 0.2.1__tar.gz

{world_model_optimizer-0.2.0 → world_model_optimizer-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: world-model-optimizer
-Version: 0.2.0
+Version: 0.2.1
 Summary: Run agents, build world models from traces, and optimize agent harnesses.
 Requires-Python: >=3.12
 Requires-Dist: anthropic>=0.39
@@ -57,9 +57,9 @@ Description-Content-Type: text/markdown
 
 # World Model Optimizer
 
-`wmo` is an open-source project for running and building continuously improving agents. It
-includes a flexible agent runtime, a world model that simulates tool calls, and an optimizer that
-builds task-specific harnesses for stronger performance at lower cost.
+`wmo` turns agent traces you already collect into continuous improvement. Start with a model
+endpoint at frontier quality with 40%+ lower cost. Keep improving it with world model simulations,
+meta-harness optimization, and model distillation.
 
 ![World model, runtime agent, and optimizer connected in a continuous improvement loop](assets/world-model-agent-loop.svg)
 
@@ -71,28 +71,48 @@ builds task-specific harnesses for stronger performance at lower cost.
 
 ## Getting started
 
-### Local setup
-
-Install WMO, choose the model provider for the built-in runtime agent, and start a local run:
+**1. Register your providers.**
 
 ```bash
 pip install world-model-optimizer
 wmo providers set
-wmo run --task "Inspect this repository and explain it"
 ```
 
-Build a named world model from collected traces:
+That verifies the provider and then offers to register its models as routing candidates in
+`.wmo/pool.toml`, the roster everything below chooses from. It searches the provider's own
+catalog (OpenRouter's 338 published models included) and asks only for what that backend needs.
+Re-run it to add another provider's models beside the ones already registered.
+
+**2. Tune a router on your OTel traces.**
+
+```bash
+wmo build --file traces.jsonl --name my-endpoint
+
+# Score every registered model on held-out tasks from your traces
+wmo optimize route sweep my-endpoint --traces traces.otel.jsonl
+
+# Turn those measurements into a routing policy
+wmo optimize route fit matrix.json --kind knn \
+  --out .wmo/models/my-endpoint/policy.json
+```
+
+**3. Serve it.**
 
 ```bash
-wmo build --file traces.jsonl --name my-environment
+wmo serve --name my-endpoint
 ```
 
-Then optimize an agent harness against that model and a set of tasks:
+See what it bought you against the model you were using before:
 
 ```bash
-wmo optimize harness my-agent my-environment --tasks tasks.jsonl
+wmo optimize route report matrix.json .wmo/models/my-endpoint/policy.json \
+  --baseline gpt-5.5
 ```
 
+Distill your own small model into the pool with [`wmo optimize distill`](wmo/distill/README.md),
+serve a single model with no routing via `wmo optimize route pin`, or build an optimized harness
+for your agent with `wmo optimize harness`.
+
 ### Hosted platform
 
 Create an account at [platform.experientiallabs.ai](https://platform.experientiallabs.ai), then
@@ -121,6 +141,9 @@ wmo optimize harness my-agent my-environment --tasks tasks.jsonl --backend e2b
 
 ## Use a world model as an API
 
+`world-model-optimizer` includes world models that can be used to simulate your agent environment
+for testing and optimization.
+
 ```python
 from wmo import Action, ActionKind
 from wmo.config.store import WorldModelStore
@@ -135,7 +158,8 @@ obs = wm.step(session.id, Action(kind=ActionKind.TOOL_CALL, name="add_to_cart",
 print(obs.content)
 ```
 
-Or over HTTP (same code path), namespaced by model name: `GET /world_models`, then `POST /world_models/{name}/sessions` and `POST /world_models/{name}/sessions/{id}/step`.
+Or over HTTP (same code path), namespaced by model name: `GET /world_models`, then `POST
+/world_models/{name}/sessions` and `POST /world_models/{name}/sessions/{id}/step`.
 
 ## Run after platform login
 
@@ -176,9 +200,10 @@ the new versioned champion harness.
 
 ## Development
 
-Managed with [uv](https://docs.astral.sh/uv/); linting/formatting with [ruff](https://docs.astral.sh/ruff/); type checking with [ty](https://github.com/astral-sh/ty). Conventions live in [AGENTS.md](./AGENTS.md).
+Managed with [uv](https://docs.astral.sh/uv/); linting/formatting with [ruff](https://docs.astral.sh/ruff/); type checking with [ty](https://github.com/astral-sh/ty); tasks run through [just](https://github.com/casey/just) (`brew install just` / `cargo install just`). Conventions live in [AGENTS.md](./AGENTS.md).
 
 ```bash
+just setup               # first time: .env from the template + uv sync
 uv sync --extra dev      # env + dev tools
 uv run ruff check .      # lint
 uv run ruff format .     # format
@@ -189,7 +214,8 @@ uv run pytest -q         # tests
 ## Usage telemetry
 
 `wmo` uses anonymous usage telemetry to track the volume of usage.
-Telemetry is strictly metadata. It never includes prompts, traces, actions, observations, file paths,
+Telemetry is strictly metadata. It never includes prompts, traces, actions, observations, file
+paths,
 model names, provider credentials, or raw user content.
 
 Telemetry is enabled by default. To opt out for a project:

{world_model_optimizer-0.2.0 → world_model_optimizer-0.2.1}/README.md

@@ -1,8 +1,8 @@
 # World Model Optimizer
 
-`wmo` is an open-source project for running and building continuously improving agents. It
-includes a flexible agent runtime, a world model that simulates tool calls, and an optimizer that
-builds task-specific harnesses for stronger performance at lower cost.
+`wmo` turns agent traces you already collect into continuous improvement. Start with a model
+endpoint at frontier quality with 40%+ lower cost. Keep improving it with world model simulations,
+meta-harness optimization, and model distillation.
 
 ![World model, runtime agent, and optimizer connected in a continuous improvement loop](assets/world-model-agent-loop.svg)
 
@@ -14,28 +14,48 @@ builds task-specific harnesses for stronger performance at lower cost.
 
 ## Getting started
 
-### Local setup
-
-Install WMO, choose the model provider for the built-in runtime agent, and start a local run:
+**1. Register your providers.**
 
 ```bash
 pip install world-model-optimizer
 wmo providers set
-wmo run --task "Inspect this repository and explain it"
 ```
 
-Build a named world model from collected traces:
+That verifies the provider and then offers to register its models as routing candidates in
+`.wmo/pool.toml`, the roster everything below chooses from. It searches the provider's own
+catalog (OpenRouter's 338 published models included) and asks only for what that backend needs.
+Re-run it to add another provider's models beside the ones already registered.
+
+**2. Tune a router on your OTel traces.**
+
+```bash
+wmo build --file traces.jsonl --name my-endpoint
+
+# Score every registered model on held-out tasks from your traces
+wmo optimize route sweep my-endpoint --traces traces.otel.jsonl
+
+# Turn those measurements into a routing policy
+wmo optimize route fit matrix.json --kind knn \
+  --out .wmo/models/my-endpoint/policy.json
+```
+
+**3. Serve it.**
 
 ```bash
-wmo build --file traces.jsonl --name my-environment
+wmo serve --name my-endpoint
 ```
 
-Then optimize an agent harness against that model and a set of tasks:
+See what it bought you against the model you were using before:
 
 ```bash
-wmo optimize harness my-agent my-environment --tasks tasks.jsonl
+wmo optimize route report matrix.json .wmo/models/my-endpoint/policy.json \
+  --baseline gpt-5.5
 ```
 
+Distill your own small model into the pool with [`wmo optimize distill`](wmo/distill/README.md),
+serve a single model with no routing via `wmo optimize route pin`, or build an optimized harness
+for your agent with `wmo optimize harness`.
+
 ### Hosted platform
 
 Create an account at [platform.experientiallabs.ai](https://platform.experientiallabs.ai), then
@@ -64,6 +84,9 @@ wmo optimize harness my-agent my-environment --tasks tasks.jsonl --backend e2b
 
 ## Use a world model as an API
 
+`world-model-optimizer` includes world models that can be used to simulate your agent environment
+for testing and optimization.
+
 ```python
 from wmo import Action, ActionKind
 from wmo.config.store import WorldModelStore
@@ -78,7 +101,8 @@ obs = wm.step(session.id, Action(kind=ActionKind.TOOL_CALL, name="add_to_cart",
 print(obs.content)
 ```
 
-Or over HTTP (same code path), namespaced by model name: `GET /world_models`, then `POST /world_models/{name}/sessions` and `POST /world_models/{name}/sessions/{id}/step`.
+Or over HTTP (same code path), namespaced by model name: `GET /world_models`, then `POST
+/world_models/{name}/sessions` and `POST /world_models/{name}/sessions/{id}/step`.
 
 ## Run after platform login
 
@@ -119,9 +143,10 @@ the new versioned champion harness.
 
 ## Development
 
-Managed with [uv](https://docs.astral.sh/uv/); linting/formatting with [ruff](https://docs.astral.sh/ruff/); type checking with [ty](https://github.com/astral-sh/ty). Conventions live in [AGENTS.md](./AGENTS.md).
+Managed with [uv](https://docs.astral.sh/uv/); linting/formatting with [ruff](https://docs.astral.sh/ruff/); type checking with [ty](https://github.com/astral-sh/ty); tasks run through [just](https://github.com/casey/just) (`brew install just` / `cargo install just`). Conventions live in [AGENTS.md](./AGENTS.md).
 
 ```bash
+just setup               # first time: .env from the template + uv sync
 uv sync --extra dev      # env + dev tools
 uv run ruff check .      # lint
 uv run ruff format .     # format
@@ -132,7 +157,8 @@ uv run pytest -q         # tests
 ## Usage telemetry
 
 `wmo` uses anonymous usage telemetry to track the volume of usage.
-Telemetry is strictly metadata. It never includes prompts, traces, actions, observations, file paths,
+Telemetry is strictly metadata. It never includes prompts, traces, actions, observations, file
+paths,
 model names, provider credentials, or raw user content.
 
 Telemetry is enabled by default. To opt out for a project:

{world_model_optimizer-0.2.0 → world_model_optimizer-0.2.1}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "world-model-optimizer"
-version = "0.2.0"
+version = "0.2.1"
 description = "Run agents, build world models from traces, and optimize agent harnesses."
 readme = "README.md"
 # harbor 0.20.0 requires Python >=3.12; the harbor evaluator is a first-class optimize
@@ -95,7 +95,12 @@ build-backend = "hatchling.build"
 
 [tool.hatch.build.targets.wheel]
 packages = ["wmo", "packages/llm-waterfall/llm_waterfall"]
-exclude = ["**/*_test.py"]
+# Test-only code is named for pytest discovery (`*_test.py`), and these two are the exceptions
+# that the glob therefore misses: `fake_tinker` is the deterministic Tinker double the distill
+# tests run the whole loop against, and `wmo/conftest.py` is a pytest fixture module. Both are
+# imported only by tests, so neither belongs in the published wheel. `fake_tinker` stays where
+# it is rather than moving under a test path: five test modules import it by that name.
+exclude = ["**/*_test.py", "wmo/distill/fake_tinker.py", "wmo/conftest.py"]
 force-include = { "packages/llm-waterfall/LICENSE" = "llm_waterfall/LICENSE" }
 
 [tool.hatch.build.targets.sdist]
@@ -124,7 +129,9 @@ environment-capture = { workspace = true }
 # Tests live inline next to the modules they cover (foo.py -> foo_test.py).
 python_files = ["*_test.py"]
 # One root gate covers the flagship and every Python member (same inline-test convention).
-testpaths = ["wmo", "packages"]
+# `deploy/` carries the operated services; their tests cover the deploy scripts themselves
+# (a deploy that deletes the box's generated trees rotates the pinned TLS certificate).
+testpaths = ["wmo", "packages", "deploy"]
 # tau2-bench is a local clone of the external real-environment runner (own test suite, own
 # deps) that lives beside the tau-bench member data; it is not part of this gate.
 norecursedirs = ["tau2-bench"]
@@ -158,7 +165,9 @@ python-version = "3.12"
 [tool.ty.src]
 # `plot_trace_scaling.py` is a one-off research figure that imports matplotlib ephemerally
 # (`uv run --with matplotlib`), not a project dependency — exclude it from the type gate.
-exclude = ["examples/", ".agents/",
+# `deploy/` is box-side service code: it imports torch and transformers, which live in the
+# GPU box's venv and are deliberately not project dependencies. Ruff still lints it.
+exclude = ["examples/", ".agents/", "deploy/",
     "packages/environment-capture/*/backend/",
     "packages/environment-capture/tau-bench/",
     "packages/environment-capture/terminal-tasks/",

{world_model_optimizer-0.2.0 → world_model_optimizer-0.2.1}/wmo/cli/app.py

@@ -18,6 +18,7 @@ import uuid
 from dataclasses import dataclass
 from datetime import UTC, datetime
 from pathlib import Path
+from typing import cast
 from uuid import uuid4
 
 import typer
@@ -35,12 +36,14 @@ from rich.panel import Panel
 from rich.progress import BarColumn, DownloadColumn, Progress, TextColumn
 from rich.table import Table
 
+import wmo.cli.pool_registry as pool_registry
 import wmo.providers as providers
 from wmo.cli.agent_session import register as register_agent_session_commands
 from wmo.cli.e2b_cmds import register as register_e2b_commands
 from wmo.cli.eval_closed_loop import run_agreement, run_closed_loop
 from wmo.cli.harness_app import harness_app, optimize_app
 from wmo.cli.ingest_cmd import ingest as _ingest_command
+from wmo.cli.model_roles import configured_role_configs
 from wmo.cli.platform_cmds import register as register_platform_commands
 from wmo.cli.ui import (
     BuildParams,
@@ -75,8 +78,8 @@ from wmo.config import (
 )
 from wmo.config.card import make_build_card, save_card
 from wmo.core.types import JsonObject
+from wmo.engine.build import DEFAULT_TRAIN_SPLIT, ingest, split_traces
 from wmo.engine.build import build as run_build
-from wmo.engine.build import ingest, split_traces
 from wmo.engine.demo import run_demo
 from wmo.engine.eval_suites import (
     discover_eval_suites,
@@ -94,9 +97,10 @@ from wmo.evals.grid_plot import plot_grid, plot_grid_heatmap
 from wmo.evals.open_loop import EvalReport, OpenLoopEval
 from wmo.ingest import VendorPull, get_adapter, list_adapters
 from wmo.optimize.judge import JUDGE_VERSION, RubricJudge
-from wmo.providers import ProviderConfig, ProviderKind, verify_all, verify_embedder
+from wmo.providers import ProviderConfig, ProviderKind, VerifyResult, verify_all, verify_embedder
 from wmo.providers.base import Embedder, EmbedderKind, Provider
 from wmo.providers.models import resolve_provider_model
+from wmo.providers.pool import Tier
 from wmo.providers.retry import wrap_provider_with_retries
 from wmo.research import Side, run_concurrency_scaling
 from wmo.research.concurrency_run import build_real_runner, build_world_runner
@@ -185,6 +189,12 @@ _CONCURRENCY_ISOLATION_FLAGS: dict[str, tuple[str, ...]] = {
 _DOWNLOAD_BENCHMARKS = typer.Argument(
     None, help="Benchmark bundles to download, or 'all'. Omit for a picker."
 )
+# Repeatable option default hoisted out of the signature (ruff B008 forbids the call inline).
+_POOL_MODEL_OPTION = typer.Option(
+    None,
+    "--pool-model",
+    help="Register this model id as a routing candidate, with no prompts; repeat for several.",
+)
 
 
 @dataclass(frozen=True)
@@ -250,9 +260,55 @@ def providers_set(
     endpoint: str = typer.Option(None, "--endpoint", help="OpenAI-compatible API endpoint."),
     deployment: str = typer.Option(None, "--deployment", help="Azure deployment name."),
     api_version: str = typer.Option(None, "--api-version", help="Azure API version."),
+    pool: str = typer.Option(
+        None,
+        "--pool",
+        help="Candidate pool TOML the router picks from (default: <root>/pool.toml).",
+    ),
+    pool_model: list[str] | None = _POOL_MODEL_OPTION,
+    api_key_env: str = typer.Option(
+        None, "--api-key-env", help="Env var holding the pool entries' API key (multi-account)."
+    ),
+    tier: str = typer.Option(
+        None, "--tier", help=f"Pool entry tier: {' | '.join(pool_registry.TIERS)}."
+    ),
+    input_per_mtok: float = typer.Option(
+        None,
+        "--input-per-mtok",
+        min=0.0,
+        help="Prompt-token price of every --pool-model, USD per 1M tokens.",
+    ),
+    output_per_mtok: float = typer.Option(
+        None,
+        "--output-per-mtok",
+        min=0.0,
+        help="Completion-token price of every --pool-model, USD per 1M tokens.",
+    ),
     root: str = typer.Option(ARTIFACT_DIR, help="Project dir holding local settings."),
 ) -> None:
-    """Choose and verify the model provider used by local worker-agent runs."""
+    """Choose the local worker's provider, and register the models the router can choose from.
+
+    Two things a project needs, in one command. The worker provider lands in
+    `<root>/settings.toml` as `[models.worker]`, exactly as before. Then, on a terminal, this
+    offers to register models as ROUTING CANDIDATES in `<root>/pool.toml`, the roster
+    `wmo optimize route` selects over: pick a backend, search its catalog, and answer only what
+    that backend needs (an Azure deployment; a price for a model with no published one, never
+    for OpenRouter, which self-prices). Re-run it to add another provider's models beside the
+    ones already registered.
+
+    Scripts keep the old contract: with `--provider` and `--model` both given, nothing is
+    prompted. Registering non-interactively is `--pool-model <id>`, repeated per model, with
+    `--input-per-mtok`/`--output-per-mtok` when the model has no published price.
+    """
+    if tier is not None and tier not in pool_registry.TIERS:
+        raise typer.BadParameter(f"--tier must be one of: {', '.join(pool_registry.TIERS)}")
+    if (input_per_mtok is None) != (output_per_mtok is None):
+        # A pool entry takes both prices or neither, so half a pair would be silently dropped
+        # (interactively) or rejected as an invalid entry (with --pool-model).
+        raise typer.BadParameter(
+            "set both --input-per-mtok and --output-per-mtok, or neither; a pool entry prices "
+            "prompt and completion tokens together"
+        )
     existing = load_settings(root).models.worker
     used_picker = _console.is_terminal and (provider is None or model is None)
     if used_picker:
@@ -310,6 +366,58 @@ def providers_set(
         f"[green]set[/green] local worker provider to {config.kind.value} "
         f"({config.model_type or model}) in {settings_path(root)}"
     )
+    _register_pool_models(
+        pool_path=pool_registry.pool_path_for(root, pool),
+        kind=config.kind,
+        pool_model=list(pool_model or []),
+        options=pool_registry.EntryOptions(
+            endpoint=config.endpoint,
+            region=config.region,
+            # The RAW flags, not the worker config's: `_worker_provider_config` fills an Azure
+            # deployment in from the model id when none was given, and a pool entry must never
+            # inherit a guessed deployment name (nothing can derive an operator's own).
+            deployment=deployment,
+            api_version=api_version,
+            api_key_env=api_key_env,
+            tier=cast("Tier", tier) if tier is not None else "frontier",
+            input_per_mtok=input_per_mtok,
+            output_per_mtok=output_per_mtok,
+        ),
+        interactive=used_picker,
+    )
+
+
+def _register_pool_models(
+    *,
+    pool_path: Path,
+    kind: ProviderKind,
+    pool_model: list[str],
+    options: pool_registry.EntryOptions,
+    interactive: bool,
+) -> None:
+    """Register routing candidates after the worker provider is saved.
+
+    Explicit `--pool-model` ids win and register with no prompts, so a script gets the same
+    roster a person would build by hand. Otherwise the registry is OFFERED only on the run that
+    already prompted (a bare `wmo providers set` at a terminal): a scripted
+    `--provider ... --model ...` invocation keeps its exact pre-existing behavior and asks
+    nothing.
+    """
+    if pool_model:
+        pool_registry.register_model_ids(
+            _console, pool_path=pool_path, kind=kind, model_ids=pool_model, options=options
+        )
+        return
+    if not interactive:
+        return
+    pool_registry.run_pool_registry(
+        _console,
+        lambda text: _console.input(text),
+        lambda text: _console.input(text, password=True),
+        pool_path=pool_path,
+        default_kind=kind,
+        options=options,
+    )
 
 
 @providers_app.command("verify")
@@ -319,15 +427,18 @@ def providers_verify(
 ) -> None:
     """Ping every configured provider (completion + embed path) and report status.
 
-    Gathers provider configs from the built world models (one `--name`, or all of them, deduped by
-    kind+model), so a brand-new project with nothing built yet has nothing to verify. The phi embed
-    path of each model is checked too, unless it is the offline (creds-free) hashing embedder.
+    Two sources count as "configured", because checking that credentials work is what you do
+    BEFORE spending anything on `wmo build`: the `[models.<role>]` roles in
+    `<root>/settings.toml` (what `wmo providers set` writes), and the providers persisted inside
+    every built world model. Completion providers are deduped by kind+model across both, so a
+    role naming the same backend as a built model costs one ping, not two. The phi embed path
+    belongs to a BUILT model, so on a project with nothing built that half is skipped with a
+    note instead of aborting the command; otherwise every distinct provider-backed embedder is
+    checked (the offline hashing embedder needs no credentials and is skipped). `--name` scopes
+    the whole report to that one world model.
     """
     store = WorldModelStore(root)
     names = [name] if name is not None else store.list_names()
-    if not names:
-        _console.print("[yellow]no world models built yet[/yellow]; run `wmo build --name <name>`")
-        return
     configs: list[HarnessConfig] = []
     for model_name in names:
         try:
@@ -336,32 +447,88 @@ def providers_verify(
             raise typer.BadParameter(str(exc)) from exc
         configs.append(load_config(model_dir))
 
-    # Dedup completion providers by kind+model across all selected models.
-    seen: set[tuple[str, str]] = set()
-    providers: list[ProviderConfig] = []
-    for config in configs:
-        for pc in config.providers:
-            key = (pc.kind.value, pc.model)
-            if key not in seen:
-                seen.add(key)
-                providers.append(pc)
-    for result in verify_all(providers):
-        mark = "[green]ok[/green]" if result.ok else "[red]fail[/red]"
-        _console.print(f"{mark} {result.kind.value} ({result.model}) {result.detail}")
+    # Dedup identical completion calls across all selected models, then across the settings
+    # roles. World models come FIRST so a config present in both is pinged with the built
+    # artifact's own copy, exactly as before roles were a source.
+    labelled = [
+        (model_name, pc)
+        for model_name, cfg in zip(names, configs, strict=True)
+        for pc in cfg.providers
+    ]
+    # `--name` asks about one world model; widening it to the project's roles would answer a
+    # question the caller did not ask (and bill for it).
+    if name is None:
+        labelled += [(f"models.{role}", pc) for role, pc in configured_role_configs(root)]
+    index: dict[str, int] = {}
+    provider_configs: list[ProviderConfig] = []
+    sources: list[list[str]] = []
+    for label, pc in labelled:
+        key = _completion_identity(pc)
+        if key not in index:
+            index[key] = len(provider_configs)
+            provider_configs.append(pc)
+            sources.append([])
+        if label not in sources[index[key]]:
+            sources[index[key]].append(label)
+
+    if not provider_configs:
+        _console.print(
+            "[yellow]nothing configured[/yellow]; run `wmo providers set` to choose a provider, "
+            "or `wmo build --name <name>` to build a world model"
+        )
+        raise typer.Exit(1)
+
+    for result, origin in zip(verify_all(provider_configs), sources, strict=True):
+        _print_verify_result(result, origin)
+
+    if not configs:
+        _console.print(
+            "[dim]embed path: skipped, no world model built yet "
+            "(the embedder is chosen by `wmo build`)[/dim]"
+        )
+        return
 
     # Verify each distinct provider-backed embed path (skip the offline hashing embedder).
-    embed_seen: set[tuple[str, str]] = set()
-    for config in configs:
+    embed_seen: set[str] = set()
+    for model_name, config in zip(names, configs, strict=True):
         if config.embed_provider is EmbedderKind.HASHING:
             continue
         embed_config = config.embed_provider_config()
-        key = (embed_config.kind.value, embed_config.model)
+        key = embed_config.model_dump_json()
         if key in embed_seen:
             continue
         embed_seen.add(key)
         result = verify_embedder(embed_config)
-        mark = "[green]ok[/green]" if result.ok else "[red]fail[/red]"
-        _console.print(f"{mark} embed:{result.kind.value} ({result.model}) {result.detail}")
+        _print_verify_result(result, [model_name], prefix="embed:")
+
+
+def _completion_identity(config: ProviderConfig) -> str:
+    """What makes two provider configs the SAME completion call, for `providers verify` dedup.
+
+    Everything the config carries except the embed-only fields, which do not reach a completion
+    request. Keying on kind+model alone would collapse one model served from two Bedrock regions,
+    or one Azure model behind two deployments or endpoints, into a single ping, and then report
+    the config that was never called as verified on the other one's result. Deriving the key
+    from the whole config rather than a hand-listed subset means a field added later splits the
+    key by default instead of silently widening that collapse.
+    """
+    return config.model_copy(update={"embed_model": None, "embed_dim": None}).model_dump_json()
+
+
+def _print_verify_result(result: VerifyResult, sources: list[str], *, prefix: str = "") -> None:
+    """Print one `providers verify` line, plus the next step to take when the ping failed.
+
+    `sources` names what asked for this provider (world model names, `models.<role>` settings
+    roles) so a failure points at the thing to fix. The detail is escaped: it carries raw
+    provider error text, and an unescaped `[...]` in it would be read as rich markup and crash
+    the report.
+    """
+    mark = "[green]ok[/green]" if result.ok else "[red]fail[/red]"
+    origin = f" [dim]({', '.join(sources)})[/dim]" if sources else ""
+    detail = f" {escape(result.detail)}" if result.detail else ""
+    _console.print(f"{mark} {prefix}{result.kind.value} ({result.model}){origin}{detail}")
+    if not result.ok:
+        _console.print(f"  [yellow]{_credential_hint(result.kind, result.detail)}[/yellow]")
 
 
 @examples_app.command("list")
@@ -432,7 +599,10 @@ def build(
         None, "--chain", help="Named failover chain from .wmo/fallback.toml (default: its default)."
     ),
     train_split: float = typer.Option(
-        0.8, help="Train/held-out ratio for GEPA's internal split (lower = bigger valset)."
+        DEFAULT_TRAIN_SPLIT,
+        help="Train/held-out ratio for GEPA's internal split (lower = bigger valset). Shares "
+        "`wmo eval`'s default: both cut the same trace-id hash line, so a mismatch leaks train "
+        "traces into the eval holdout.",
     ),
     embed_provider: str = typer.Option(
         "hashing", help="phi embedder: hashing (offline) | bedrock | openai | azure."
@@ -748,19 +918,24 @@ def _verify_or_abort(config: HarnessConfig, chain: str | None = None) -> None:
             continue
         failed = True
         _console.print(f"  [red]✗ {label} ({result.model}) failed[/red]: {result.detail}")
-        if "No module named" in result.detail:
-            # SDKs are core deps; a missing module means the env is stale or hand-rolled.
-            _console.print("    [yellow]run `uv sync` to install the provider SDKs[/yellow]")
-        else:
-            envs = ", ".join(PROVIDER_ENV_VARS.get(cfg.kind, []))
-            hint = f" ({envs})" if envs else ""
-            _console.print(
-                f"    [yellow]check the model id and that your credentials are set{hint}[/yellow]"
-            )
+        _console.print(f"    [yellow]{_credential_hint(cfg.kind, result.detail)}[/yellow]")
     if failed:
         raise typer.Exit(1)
 
 
+def _credential_hint(kind: ProviderKind, detail: str) -> str:
+    """The next step for a failed provider ping: install the SDKs, or fix creds/model id.
+
+    Shared by the pre-build guard and `wmo providers verify` so both name the same env vars.
+    """
+    if "No module named" in detail:
+        # SDKs are core deps; a missing module means the env is stale or hand-rolled.
+        return "run `uv sync` to install the provider SDKs"
+    envs = ", ".join(PROVIDER_ENV_VARS.get(kind, []))
+    hint = f" ({envs})" if envs else ""
+    return f"check the model id and that your credentials are set{hint}"
+
+
 @app.command("list")
 def list_models(root: str = typer.Option(ARTIFACT_DIR, help="Project dir to list.")) -> None:
     """List every world model built under the project dir."""
@@ -823,8 +998,12 @@ def download(
             raise typer.BadParameter(str(exc)) from exc
         except urllib.error.HTTPError as exc:
             # One unpublished/broken dataset must not abort the REST of a multi-download:
-            # record it, keep fetching, and fail (with every name) at the end.
-            failures.append(f"{name}: the Hub answered {exc.code} for {exc.url}")
+            # record it, keep fetching, and fail (with every name) at the end. The reason is
+            # quoted rather than summarized here: a fetch tries more than one dataset repo id
+            # and only the error it raises knows which ones the Hub refused. Reading it off
+            # plain HTTPError attributes keeps this working against the PUBLISHED
+            # environment-capture, which the WMO wheel resolves from PyPI.
+            failures.append(f"{name}: the Hub answered {exc.code} for {exc.url} ({exc.reason})")
             _console.print(f"[yellow]skipping {name}: Hub answered {exc.code}[/yellow]")
             continue
         except urllib.error.URLError as exc:
@@ -911,7 +1090,9 @@ def eval_(  # noqa: A001 - `eval` is the user-facing command name; the builtin i
         None, "--chain", help="Named failover chain from .wmo/fallback.toml (default: its default)."
     ),
     train_split: float | None = typer.Option(
-        None, help="Train/holdout ratio per file (default: 0.7, or suite config)."
+        None,
+        help=f"Train/holdout ratio per file (default: {DEFAULT_TRAIN_SPLIT}, or suite config). "
+        "Must match the ratio the model was built with, or the holdout contains train traces.",
     ),
     val_frac: float | None = typer.Option(
         None,
@@ -1473,7 +1654,7 @@ def _eval_options(
     knowledge: bool | None = None,
     reasoning: bool | None = None,
 ) -> _EvalOptions:
-    split = 0.7 if train_split is None else train_split
+    split = DEFAULT_TRAIN_SPLIT if train_split is None else train_split
     dim = 512 if embed_dim is None else embed_dim
     retrieval = True if rag is None else rag
     turns = "all" if sample_turns is None else sample_turns