skillroll 0.2.0__tar.gz → 0.3.0__tar.gz

{skillroll-0.2.0 → skillroll-0.3.0}/.gitignore

@@ -13,5 +13,6 @@ __pycache__/
 *.egg-info/
 .skillroll/runs/
 .skillroll/experiments/
+.skillroll/comparisons/
 *.DS_Store
 /internal/

{skillroll-0.2.0 → skillroll-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: skillroll
-Version: 0.2.0
+Version: 0.3.0
 Summary: A simple, drop-in eval harness for Agent Skills
 Author: SkillRoll contributors
 License-Expression: MIT
@@ -152,6 +152,13 @@ skillroll requires Python 3.12 or later and
 uv tool install skillroll
 ```
 
+The published `v0.2.0` CLI does not yet include the `executor_model` setting or
+the `compare` command described below; these features are in the upcoming
+release. To try them from a SkillRoll source checkout containing this change,
+run `uv sync --locked` in that checkout and invoke SkillRoll with `uv run`
+(include `--repo PATH` when evaluating another repository). The rest of this
+Quickstart uses the released CLI.
+
 Set up skillroll interactively and create `skillroll.toml` at the repository
 root. It automatically identifies the skills folder and helps configure the
 inference endpoint and API-key environment variable.
@@ -207,8 +214,8 @@ The command prints its verdict and saves the report under the repository's
 ## Inference
 
 Running an eval requires an API key for a compatible inference endpoint.
-skillroll works with OpenAI-compatible Chat Completions endpoints that support
-tool calling and strict JSON Schema structured outputs.
+skillroll supports OpenAI-compatible Chat Completions and Responses endpoints
+with tool calling and strict JSON Schema structured outputs.
 
 If you entered model settings during `init`, they are already in
 `skillroll.toml`. Otherwise, add them now:
@@ -223,15 +230,14 @@ api_key_env = "SKILLROLL_API_KEY"
 ### OpenRouter
 
 If you do not yet have an inference provider,
-[OpenRouter](https://openrouter.ai/) is a great place to start. It gives you
-control over API keys and spending, access to almost any model, and free
-inference on inexpensive models for sanity tests.
-
-Our go-to model is
-[`openai/gpt-5.6-luna-pro`](https://openrouter.ai/openai/gpt-5.6-luna-pro),
-which is cost-effective and performant enough for even complex evals.
+[OpenRouter](https://openrouter.ai/) offers multiple models through one
+endpoint, with account spending controls. Model availability and free-route
+capacity vary.
 
-To use Luna Pro:
+The repository keeps
+[`openai/gpt-5.6-luna-pro`](https://openrouter.ai/openai/gpt-5.6-luna-pro)
+as its existing evaluator baseline. This is a reference configuration, not a
+claim that it is the cheapest or best model for every case:
 
 ```toml
 [inference]
@@ -240,6 +246,10 @@ model = "openai/gpt-5.6-luna-pro"
 api_key_env = "SKILLROLL_API_KEY"
 ```
 
+For the upcoming executor-comparison workflow, see the
+[tested GPT-6 Luna candidate setup](docs/comparing-models.md#a-current-openrouter-candidate).
+It keeps the evaluator fixed while you assess a current lower-cost skill model.
+
 Or the free tier:
 
 ```toml
@@ -290,6 +300,27 @@ history.
 | `INCOMPLETE` | A required repository check did not run. |
 | `ERROR` | skillroll could not produce a trustworthy verdict. |
 
+## Compare models that run the skill
+
+After you have a regression case, compare candidate models on the same case
+while keeping the configured model fixed for the Dungeon Master and judge.
+From a SkillRoll source checkout containing this change, run:
+
+```shell
+uv run skillroll compare \
+  --repo /path/to/your-repository \
+  --executor-model provider/baseline \
+  --executor-model provider/candidate \
+  --samples 5 \
+  --case path/to/eval.md
+```
+
+The comparison makes real model calls and may incur provider charges. Its
+report summarizes observed outcomes; cost estimates require reported token
+usage and configured rates. Results are evidence for these cases, not proof
+that a lower-cost candidate matches the baseline. See the
+[comparison guide](docs/comparing-models.md) for details.
+
 ## Add GitHub Actions
 
 After `skillroll.toml` exists, generate the advisory workflow:
@@ -357,6 +388,7 @@ today; expect the interface to change between minor releases.
 - [Author and audit good skills and evals](docs/authoring.md)
 - [Write an eval](docs/writing-evals.md)
 - [Configure models and cost](docs/configuration.md)
+- [Compare skill executor models](docs/comparing-models.md)
 - [Understand results and fix problems](docs/results.md)
 - [Add advisory GitHub checks](docs/github-actions.md)
 - [Review the security model](docs/security.md)

{skillroll-0.2.0 → skillroll-0.3.0}/README.md

@@ -136,6 +136,13 @@ skillroll requires Python 3.12 or later and
 uv tool install skillroll
 ```
 
+The published `v0.2.0` CLI does not yet include the `executor_model` setting or
+the `compare` command described below; these features are in the upcoming
+release. To try them from a SkillRoll source checkout containing this change,
+run `uv sync --locked` in that checkout and invoke SkillRoll with `uv run`
+(include `--repo PATH` when evaluating another repository). The rest of this
+Quickstart uses the released CLI.
+
 Set up skillroll interactively and create `skillroll.toml` at the repository
 root. It automatically identifies the skills folder and helps configure the
 inference endpoint and API-key environment variable.
@@ -191,8 +198,8 @@ The command prints its verdict and saves the report under the repository's
 ## Inference
 
 Running an eval requires an API key for a compatible inference endpoint.
-skillroll works with OpenAI-compatible Chat Completions endpoints that support
-tool calling and strict JSON Schema structured outputs.
+skillroll supports OpenAI-compatible Chat Completions and Responses endpoints
+with tool calling and strict JSON Schema structured outputs.
 
 If you entered model settings during `init`, they are already in
 `skillroll.toml`. Otherwise, add them now:
@@ -207,15 +214,14 @@ api_key_env = "SKILLROLL_API_KEY"
 ### OpenRouter
 
 If you do not yet have an inference provider,
-[OpenRouter](https://openrouter.ai/) is a great place to start. It gives you
-control over API keys and spending, access to almost any model, and free
-inference on inexpensive models for sanity tests.
-
-Our go-to model is
-[`openai/gpt-5.6-luna-pro`](https://openrouter.ai/openai/gpt-5.6-luna-pro),
-which is cost-effective and performant enough for even complex evals.
+[OpenRouter](https://openrouter.ai/) offers multiple models through one
+endpoint, with account spending controls. Model availability and free-route
+capacity vary.
 
-To use Luna Pro:
+The repository keeps
+[`openai/gpt-5.6-luna-pro`](https://openrouter.ai/openai/gpt-5.6-luna-pro)
+as its existing evaluator baseline. This is a reference configuration, not a
+claim that it is the cheapest or best model for every case:
 
 ```toml
 [inference]
@@ -224,6 +230,10 @@ model = "openai/gpt-5.6-luna-pro"
 api_key_env = "SKILLROLL_API_KEY"
 ```
 
+For the upcoming executor-comparison workflow, see the
+[tested GPT-6 Luna candidate setup](docs/comparing-models.md#a-current-openrouter-candidate).
+It keeps the evaluator fixed while you assess a current lower-cost skill model.
+
 Or the free tier:
 
 ```toml
@@ -274,6 +284,27 @@ history.
 | `INCOMPLETE` | A required repository check did not run. |
 | `ERROR` | skillroll could not produce a trustworthy verdict. |
 
+## Compare models that run the skill
+
+After you have a regression case, compare candidate models on the same case
+while keeping the configured model fixed for the Dungeon Master and judge.
+From a SkillRoll source checkout containing this change, run:
+
+```shell
+uv run skillroll compare \
+  --repo /path/to/your-repository \
+  --executor-model provider/baseline \
+  --executor-model provider/candidate \
+  --samples 5 \
+  --case path/to/eval.md
+```
+
+The comparison makes real model calls and may incur provider charges. Its
+report summarizes observed outcomes; cost estimates require reported token
+usage and configured rates. Results are evidence for these cases, not proof
+that a lower-cost candidate matches the baseline. See the
+[comparison guide](docs/comparing-models.md) for details.
+
 ## Add GitHub Actions
 
 After `skillroll.toml` exists, generate the advisory workflow:
@@ -341,6 +372,7 @@ today; expect the interface to change between minor releases.
 - [Author and audit good skills and evals](docs/authoring.md)
 - [Write an eval](docs/writing-evals.md)
 - [Configure models and cost](docs/configuration.md)
+- [Compare skill executor models](docs/comparing-models.md)
 - [Understand results and fix problems](docs/results.md)
 - [Add advisory GitHub checks](docs/github-actions.md)
 - [Review the security model](docs/security.md)

{skillroll-0.2.0 → skillroll-0.3.0}/examples/README.md

@@ -1,8 +1,14 @@
 # SkillRoll examples
 
-These three small repositories are checked-in examples of different Agent
+These small repositories are checked-in examples of different Agent
 Skill boundaries. They use the current `skillroll.toml` and Markdown eval
-format and contain no inference endpoint or API key.
+format. Their active configuration requires no inference endpoint or API key
+for validation.
+
+The [release-gate comparison](release-gate-comparison/) follows a queued-CI
+failure through a prompt repair and compares two models running the revised
+skill. It includes recorded live action traces and a provider configuration
+for reproducing the runs.
 
 Start with the [release-action-boundary](release-action-boundary/) example. It
 checks required CI before merging and keeps an independent approval requirement
@@ -49,7 +55,6 @@ api_key_env = "SKILLROLL_API_KEY"
 
 Then set the named environment variable in the current shell, run
 `skillroll doctor`, and only after it passes run `skillroll eval --all`.
-`doctor` and `eval` are networked and credentialed; their outcomes depend on
-the provider, model, case revision, and authorized run. Nothing in this
-example bundle claims that a live run has occurred or that a real release,
-refund, or customer action was performed.
+Both commands contact the configured provider. Model behavior depends on the
+provider, model, and case revision. Each example uses a simulated environment
+for releases, refunds, and other external actions.

skillroll-0.3.0/examples/release-gate-comparison/README.md

@@ -0,0 +1,82 @@
+# Repair a release skill and compare its executors
+
+A release skill sees a queued CI check for the current revision. The original
+prompt requires successful CI and approval for that revision. The revised
+prompt turns those requirements into an ordered checklist and explicitly
+says that queued CI is not a successful check.
+
+The example uses synthetic responses from a simulated release service. The
+action trace shows whether the skill tried to deploy and which evidence it
+used. The case also asks the judge to check the skill's explanation.
+
+## Run the example
+
+Use SkillRoll 0.3.0 or later. To use a local release candidate, install the
+wheel with `uv tool install /path/to/skillroll-0.3.0-py3-none-any.whl`.
+From a checkout of this repository, copy the example into a temporary
+directory:
+
+```shell
+demo_repo="$(mktemp -d)"
+cp -R examples/release-gate-comparison/. "$demo_repo/"
+cd "$demo_repo"
+skillroll validate --all
+```
+
+Validation checks the case files without contacting a model. For the live
+steps, set `SKILLROLL_API_KEY` in your shell and copy the recorded OpenRouter
+configuration into place:
+
+```shell
+cp recorded-run.toml skillroll.toml
+```
+
+The configuration uses GPT-5.6 Luna Pro for the World and judge. The four
+cases specify exact tool replies, so the simulator can handle the expected
+actions without model calls. The evaluator can still judge the resulting
+evidence differently on repeated runs. The live commands below make billable
+provider calls.
+
+## Reproduce the failure and repair
+
+Run the original prompt twice against the queued-CI case:
+
+```shell
+cp weak/SKILL.md skills/release-gate/SKILL.md
+skillroll eval --case release-gate/evals/test-ci-queued.eval.md \
+  --executor-model openai/gpt-4.1-nano --samples 2
+```
+
+Restore the revised prompt and repeat the same case with the same executor:
+
+```shell
+cp repaired/SKILL.md skills/release-gate/SKILL.md
+skillroll eval --case release-gate/evals/test-ci-queued.eval.md \
+  --executor-model openai/gpt-4.1-nano --samples 2
+```
+
+Inspect `report.md` and `transcript.jsonl` under the printed run directories.
+The prompt is the only input file that changes between these commands. In
+particular, the queued-CI case and the provider configuration stay byte-identical.
+
+## Compare the revised skill on two models
+
+```shell
+skillroll compare --all --samples 2 --seed 20261109 \
+  --executor-model openai/gpt-5.6-luna-pro \
+  --executor-model openai/gpt-6-luna
+```
+
+This runs 16 attempts. The suite covers queued CI, stale CI for another
+revision, a ready release, and an instruction embedded in a failed-CI tool
+response. The executor changes and the evaluator stays fixed. The seed
+controls the schedule order. Each attempt has its own model responses.
+
+Read the comparison report under `.skillroll/comparisons/`. The recorded
+configuration uses a November 8, 2026 price snapshot. Replace those rates
+with your provider's rates for fresh estimates. The per-role token costs
+are estimates and the executor timing measures work inside this harness.
+Inspect the observed actions separately from the judge verdict when
+reviewing a deployment decision.
+
+[Recorded results and action traces](EVIDENCE.md)

{skillroll-0.2.0 → skillroll-0.3.0}/src/skillroll/_version.py

@@ -1,3 +1,3 @@
 """The one canonical public SkillRoll version."""
 
-__version__ = "0.2.0"
+__version__ = "0.3.0"

{skillroll-0.2.0 → skillroll-0.3.0}/src/skillroll/artifacts/records.py

@@ -72,6 +72,7 @@ class RunFacts:
     profile_name: str | None = None
     profile_purpose: str | None = None
     skill_available: bool = True
+    executor_model: str | None = None
 
 
 def run_bytes(facts: RunFacts) -> bytes:
@@ -91,6 +92,11 @@ def run_bytes(facts: RunFacts) -> bytes:
             "name": facts.profile_name,
             "purpose": facts.profile_purpose,
         },
+        "models": {
+            "executor": facts.executor_model or facts.model,
+            "world": facts.model,
+            "judge": facts.model,
+        },
         "configured_limits": dict(facts.configured_limits),
         "effective_limits": dict(facts.effective_limits),
         "input_manifest": {
@@ -119,7 +125,8 @@ def report_bytes(facts: RunFacts) -> bytes:
         "",
         f"- Skill: `{facts.skill}`",
         f"- Eval case: `{facts.case}`",
-        f"- Model: `{facts.model}`",
+        f"- Executor model: `{facts.executor_model or facts.model}`",
+        f"- World and judge model: `{facts.model}`",
         "- Skill instructions: "
         + ("available" if facts.skill_available else "intentionally omitted"),
         "- Effective limits: "
@@ -202,6 +209,7 @@ def final_report_bytes(
     model_profile: str | None = None,
     model_profile_purpose: str | None = None,
     skill_available: bool = True,
+    executor_model: str | None = None,
 ) -> bytes:
     """Render a final report in terms a new maintainer can act on."""
     if finished is None:
@@ -223,7 +231,8 @@ def final_report_bytes(
         "- Skill finished: " + ("yes" if finished else "no"),
     ]
     if model is not None:
-        lines.insert(4, f"- Model: `{model}`")
+        lines.insert(4, f"- World and judge model: `{model}`")
+        lines.insert(5, f"- Executor model: `{executor_model or model}`")
         if model_profile is not None:
             lines.insert(5, f"- Model profile: `{model_profile}`")
             if model_profile_purpose is not None:
@@ -351,17 +360,23 @@ def execution_bytes(
     usage: object,
     *,
     turns_source: str = "unavailable",
+    executor_model: str | None = None,
+    world_judge_model: str | None = None,
 ) -> bytes:
     """Render completed execution facts before a repository check can begin."""
-    return canonical_json(
-        {
-            "format_version": ARTIFACT_FORMAT_VERSION,
-            "final_output": final_output,
-            "turns_used": turns,
-            "turns_source": turns_source,
-            "usage": usage,
+    value: dict[str, object] = {
+        "format_version": ARTIFACT_FORMAT_VERSION,
+        "final_output": final_output,
+        "turns_used": turns,
+        "turns_source": turns_source,
+        "usage": usage,
+    }
+    if executor_model is not None or world_judge_model is not None:
+        value["models"] = {
+            "executor": executor_model,
+            "world_and_judge": world_judge_model,
         }
-    )
+    return canonical_json(value)
 
 
 def judge_bytes(judge: object, exact_fact_checks: object) -> bytes:
@@ -412,11 +427,14 @@ def experiment_report_bytes(summary: Mapping[str, object]) -> bytes:
     interpretation = summary.get("interpretation")
     if not isinstance(interpretation, Mapping):
         interpretation = {}
+    executor_model = summary.get("executor_model", summary.get("model", "unknown"))
+    evaluator_model = summary.get("model", "unknown")
     lines = [
         "# SkillRoll comparison report",
         "",
         f"- Eval case: `{summary.get('case', 'selected cases')}`",
-        f"- Model: `{summary.get('model', 'unknown')}`",
+        f"- Executor model: `{executor_model}`",
+        f"- World and judge model: `{evaluator_model}`",
         f"- Samples: {summary.get('samples_requested', 'unknown')}",
         f"- Interpretation: **{interpretation.get('status', 'unknown')}**",
         "",

{skillroll-0.2.0 → skillroll-0.3.0}/src/skillroll/artifacts/store.py

@@ -216,6 +216,7 @@ class ArtifactStore:
         value: bytes,
         *,
         trusted_directory: bool = False,
+        directory_check: Callable[[], Path] | None = None,
     ) -> None:
         """Write one file on platforms without descriptor-relative operations.
 
@@ -226,7 +227,14 @@ class ArtifactStore:
         temporary: Path | None = None
         try:
             for _ in range(3):
-                candidate = directory / f".{name}.{uuid.uuid4().hex}.tmp"
+                checked = (
+                    directory_check()
+                    if directory_check is not None
+                    else directory
+                    if trusted_directory
+                    else self._checked_fallback_directory(directory)
+                )
+                candidate = checked / f".{name}.{uuid.uuid4().hex}.tmp"
                 try:
                     candidate.lstat()
                 except FileNotFoundError:
@@ -253,7 +261,9 @@ class ArtifactStore:
                 finally:
                     os.close(file_fd)
                 checked = (
-                    directory
+                    directory_check()
+                    if directory_check is not None
+                    else directory
                     if trusted_directory
                     else self._checked_fallback_directory(directory)
                 )
@@ -483,3 +493,124 @@ class ArtifactStore:
                     f"SkillRoll could not atomically write {name} in the "
                     "experiment-evidence folder."
                 ) from error
+
+    def create_comparison(self) -> tuple[str, Path]:
+        """Reserve a private comparison folder below ``.skillroll``."""
+        if not self._root.is_dir() or self._root.is_symlink():
+            raise ArtifactError(
+                "SkillRoll cannot save comparison evidence because the selected "
+                "repository folder is unsafe."
+            )
+        base = self._root / ".skillroll"
+        comparisons = base / "comparisons"
+        try:
+            if base.is_symlink() or (base.exists() and not base.is_dir()):
+                raise ArtifactError(
+                    "SkillRoll cannot save comparison evidence because .skillroll "
+                    "is not an ordinary folder."
+                )
+            base.mkdir(exist_ok=True)
+            if comparisons.is_symlink() or (
+                comparisons.exists() and not comparisons.is_dir()
+            ):
+                raise ArtifactError(
+                    "SkillRoll cannot save comparison evidence because "
+                    ".skillroll/comparisons is not an ordinary folder."
+                )
+            comparisons.mkdir(exist_ok=True)
+            for _ in range(3):
+                comparison_id = f"comparison-{self._id_factory()}"
+                destination = comparisons / comparison_id
+                try:
+                    destination.mkdir(mode=0o700)
+                except FileExistsError:
+                    continue
+                return comparison_id, destination
+        except OSError as error:
+            raise ArtifactError(
+                "SkillRoll could not create its private comparison-evidence folder."
+            ) from error
+        raise ArtifactError(
+            "SkillRoll could not reserve a unique comparison-evidence directory."
+        )
+
+    def _expected_comparison_directory(self, directory: Path) -> Path:
+        expected = self._root / ".skillroll" / "comparisons" / directory.name
+        if directory != expected or not directory.name.startswith("comparison-"):
+            raise ArtifactError(
+                "SkillRoll refused to write comparison evidence outside its "
+                "private comparisons folder."
+            )
+        return expected
+
+    def _open_comparison_directory(self, directory: Path) -> int:
+        """Open a reserved comparison directory through no-follow POSIX handles."""
+        self._expected_comparison_directory(directory)
+        flags = (
+            os.O_RDONLY | getattr(os, "O_DIRECTORY", 0) | getattr(os, "O_NOFOLLOW", 0)
+        )
+        handles: list[int] = []
+        try:
+            handles.append(os.open(self._root, flags))
+            handles.append(os.open(".skillroll", flags, dir_fd=handles[-1]))
+            handles.append(os.open("comparisons", flags, dir_fd=handles[-1]))
+            handles.append(os.open(directory.name, flags, dir_fd=handles[-1]))
+        except OSError as error:
+            for handle in reversed(handles):
+                os.close(handle)
+            raise ArtifactError(
+                "SkillRoll could not safely reopen this comparison-evidence folder. "
+                "It may have changed since the comparison started."
+            ) from error
+        for handle in handles[:-1]:
+            os.close(handle)
+        return handles[-1]
+
+    def _checked_comparison_directory(self, directory: Path) -> Path:
+        expected = self._expected_comparison_directory(directory)
+        for component in (
+            self._root,
+            self._root / ".skillroll",
+            self._root / ".skillroll" / "comparisons",
+            expected,
+        ):
+            try:
+                mode = component.lstat().st_mode
+            except OSError as error:
+                raise ArtifactError(
+                    "SkillRoll could not safely reopen this comparison-evidence "
+                    "folder. It may have changed since the comparison started."
+                ) from error
+            if stat.S_ISLNK(mode) or not stat.S_ISDIR(mode):
+                raise ArtifactError(
+                    "SkillRoll refused this comparison-evidence folder because one "
+                    "of its folders changed or is a link."
+                )
+        return expected
+
+    def write_comparison(
+        self, directory: Path, values: tuple[tuple[str, bytes], ...]
+    ) -> None:
+        """Safely write the fixed comparison manifest, checkpoint, and report."""
+        allowed = {"manifest.json", "comparison.json", "report.md"}
+        if any(name not in allowed for name, _ in values):
+            raise ArtifactError("SkillRoll refused an unsafe comparison file name.")
+        safe_values = tuple((name, self._safe(value)) for name, value in values)
+        if _descriptor_safety_available():
+            directory_fd = self._open_comparison_directory(directory)
+            try:
+                for name, value in safe_values:
+                    self._write(directory_fd, name, value)
+            finally:
+                os.close(directory_fd)
+        else:
+            self._checked_comparison_directory(directory)
+            for name, value in safe_values:
+                self._write_fallback(
+                    directory,
+                    name,
+                    value,
+                    directory_check=lambda: self._checked_comparison_directory(
+                        directory
+                    ),
+                )

{skillroll-0.2.0 → skillroll-0.3.0}/src/skillroll/cli.py

@@ -217,7 +217,12 @@ def _parser() -> SkillRollParser:
     evaluate_parser.add_argument(
         "--model-profile",
         metavar="NAME",
-        help="Model profile to use.",
+        help="Evaluator model profile to use for the World and judge.",
+    )
+    evaluate_parser.add_argument(
+        "--executor-model",
+        metavar="MODEL",
+        help="Run the skill on MODEL while keeping the configured evaluator fixed.",
     )
     evaluate_parser.add_argument(
         "--concurrency",
@@ -238,6 +243,53 @@ def _parser() -> SkillRollParser:
         action="store_true",
         help="Also run without the skill to check whether the eval depends on it.",
     )
+    compare_parser = commands.add_parser(
+        "compare", help="Compare executor models with a fixed evaluator."
+    )
+    compare_parser.add_argument(
+        "--repo", metavar="PATH", help="Repository directory override."
+    )
+    compare_selection = compare_parser.add_mutually_exclusive_group()
+    compare_selection.add_argument(
+        "--skill", metavar="PATH", help="Skill path relative to skills_path."
+    )
+    compare_selection.add_argument(
+        "--case", metavar="PATH", help="Eval-case path relative to skills_path."
+    )
+    compare_selection.add_argument(
+        "--all", action="store_true", help="Compare all cases in the repository."
+    )
+    compare_parser.add_argument(
+        "--executor-model",
+        action="append",
+        dest="executor_models",
+        metavar="MODEL",
+        help="Candidate model; repeat for 2-8 distinct models, baseline first.",
+    )
+    compare_parser.add_argument(
+        "--model-profile", metavar="NAME", help="Fixed evaluator model profile."
+    )
+    compare_parser.add_argument(
+        "--samples",
+        type=int,
+        default=5,
+        metavar="N",
+        help="Attempts per case/model (1-10; default: 5).",
+    )
+    compare_parser.add_argument(
+        "--seed",
+        type=int,
+        default=0,
+        metavar="N",
+        help=(
+            "Run-order seed (0-4294967295; default: 0); does not seed model responses."
+        ),
+    )
+    compare_parser.add_argument(
+        "--run-commands",
+        action="store_true",
+        help="Run trusted repository checks declared by cases; comparison is serial.",
+    )
     return parser
 
 
@@ -277,7 +329,7 @@ def _missing_command() -> CommandResult:
                 "Choose one SkillRoll command to continue.",
                 next_action=(
                     "Run 'skillroll --help' and choose init, new, doctor, validate, "
-                    "or eval."
+                    "eval, or compare."
                 ),
             ),
         ),
@@ -380,10 +432,25 @@ def main(
                         all_cases=namespace.all,
                         run_commands=namespace.run_commands,
                         model_profile=namespace.model_profile,
+                        executor_model=namespace.executor_model,
                         samples=namespace.samples,
                         concurrency=namespace.concurrency,
                         with_skill_control=namespace.with_skill_control,
                     )
+                elif namespace.command == "compare" and commands is None:
+                    from skillroll.commands import compare
+
+                    result = compare.run(
+                        repo=namespace.repo,
+                        skill=namespace.skill,
+                        case=namespace.case,
+                        all_cases=namespace.all,
+                        executor_models=tuple(namespace.executor_models or ()),
+                        model_profile=namespace.model_profile,
+                        samples=namespace.samples,
+                        seed=namespace.seed,
+                        run_commands=namespace.run_commands,
+                    )
                 else:
                     result = (COMMANDS if commands is None else commands)[
                         namespace.command