xone-cli 0.1.2__tar.gz → 0.1.3__tar.gz

{xone_cli-0.1.2 → xone_cli-0.1.3}/CHANGELOG.md

@@ -1,5 +1,11 @@
 # Changelog
 
+## 0.1.3
+
+- Add `xone eval open-source` for repeatable, sanitized open-source adoption evaluation.
+- Support metadata-only evaluation and optional `--clone-root` shallow public clone evaluation.
+- Keep the evaluator read-only by default and document that it does not run third-party scripts, containers, tests, or project commands.
+
 ## 0.1.2
 
 - Pin `xone doctor --install-plan` to the current recommended X-One tool versions.

{xone_cli-0.1.2/src/xone_cli.egg-info → xone_cli-0.1.3}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: xone-cli
-Version: 0.1.2
+Version: 0.1.3
 Summary: Unified CLI entry point for X-One Agent Evidence Loop workflows.
 Author: X-One-AI
 License-Expression: MIT
@@ -61,6 +61,14 @@ xone runbook --head HEAD --dry-run
 
 `xone runbook` auto-detects the local repository default branch. Remote GitHub URLs are not accepted yet; clone the repository first, then pass the local path with `--repo`.
 
+For repeatable open-source adoption checks, use the read-only evaluator:
+
+```bash
+xone eval open-source --repos repos.txt --output evaluation.md --format markdown
+```
+
+Add `--clone-root .xone-eval-clones` when you want X-One to shallow-clone public repositories and run only X-One dry-run commands against those local clones. It does not run third-party install scripts, tests, containers, or project commands.
+
 `xone-cli` orchestrates these X-One tools:
 
 - `agent-pr-evidence`
@@ -81,7 +89,8 @@ Current recommended tool versions:
 - It does not post GitHub comments.
 - It does not modify repositories automatically.
 - It does not make allow/deny runtime enforcement decisions.
-- It does not make hidden network calls.
+- It does not run third-party project scripts.
+- `xone eval open-source` makes explicit GitHub API and `git clone --depth 1` calls only when requested by the command.
 
 ## Docs
 

{xone_cli-0.1.2 → xone_cli-0.1.3}/README.md

@@ -34,6 +34,14 @@ xone runbook --head HEAD --dry-run
 
 `xone runbook` auto-detects the local repository default branch. Remote GitHub URLs are not accepted yet; clone the repository first, then pass the local path with `--repo`.
 
+For repeatable open-source adoption checks, use the read-only evaluator:
+
+```bash
+xone eval open-source --repos repos.txt --output evaluation.md --format markdown
+```
+
+Add `--clone-root .xone-eval-clones` when you want X-One to shallow-clone public repositories and run only X-One dry-run commands against those local clones. It does not run third-party install scripts, tests, containers, or project commands.
+
 `xone-cli` orchestrates these X-One tools:
 
 - `agent-pr-evidence`
@@ -54,7 +62,8 @@ Current recommended tool versions:
 - It does not post GitHub comments.
 - It does not modify repositories automatically.
 - It does not make allow/deny runtime enforcement decisions.
-- It does not make hidden network calls.
+- It does not run third-party project scripts.
+- `xone eval open-source` makes explicit GitHub API and `git clone --depth 1` calls only when requested by the command.
 
 ## Docs
 

{xone_cli-0.1.2 → xone_cli-0.1.3}/README.zh-CN.md

@@ -34,6 +34,14 @@ xone runbook --head HEAD --dry-run
 
 `xone runbook` 会自动探测本地仓库默认分支。当前还不直接接收远程 GitHub URL；请先 clone 仓库，再用 `--repo` 传本地路径。
 
+如果要重复执行开源样本替代真实用户反馈评测，可以使用只读评测入口：
+
+```bash
+xone eval open-source --repos repos.txt --output evaluation.md --format markdown
+```
+
+当你希望 X-One shallow clone 公开仓库并只运行 X-One dry-run 命令时，增加 `--clone-root .xone-eval-clones`。它不会运行第三方安装脚本、测试、容器或项目命令。
+
 `xone-cli` 编排这些 X-One 工具：
 
 - `agent-pr-evidence`
@@ -54,7 +62,8 @@ xone runbook --head HEAD --dry-run
 - 不自动发布 GitHub 评论。
 - 不自动修改仓库。
 - 不做 runtime allow/deny enforcement。
-- 不做隐藏网络调用。
+- 不运行第三方项目脚本。
+- `xone eval open-source` 只会在用户显式执行该命令时调用 GitHub API 和 `git clone --depth 1`。
 
 ## 文档
 

{xone_cli-0.1.2 → xone_cli-0.1.3}/docs/feedback/open-source-feedback-ledger.md

@@ -12,6 +12,8 @@ Rules:
 
 The first ledger entries live in `fixtures/open-source-samples/`.
 
+Use `xone eval open-source --repos repos.txt --output evaluation.json --format json` to refresh sanitized evaluation records. Add `--clone-root <dir>` only when shallow public clones are acceptable for the review.
+
 | Source | Stars | Forks | Type | Feedback |
 | --- | ---: | ---: | --- | --- |
 | modelcontextprotocol/servers | 87186 | 10998 | mcp-server-examples | catalog-update |

{xone_cli-0.1.2 → xone_cli-0.1.3}/docs/install.md

@@ -8,6 +8,7 @@ xone --version
 xone doctor
 xone doctor --install-plan
 xone runbook --head HEAD --dry-run
+xone eval open-source --repos repos.txt --output evaluation.md --format markdown
 ```
 
 ## User Install
@@ -17,6 +18,7 @@ python -m pip install xone-cli
 xone doctor
 xone doctor --install-plan
 xone runbook --head HEAD --dry-run
+xone eval open-source --repos repos.txt --output evaluation.md --format markdown
 ```
 
 `xone doctor` reports missing X-One tools and shows install guidance.
@@ -35,3 +37,33 @@ Current scenario groups:
 git clone --depth 1 https://github.com/owner/repo
 xone runbook --repo repo --head HEAD --dry-run
 ```
+
+## Open-Source Adoption Evaluation
+
+Use `xone eval open-source` when you want a repeatable, sanitized surrogate feedback pass over public repositories.
+
+Create a repo list:
+
+```text
+openai/codex
+https://github.com/docker/mcp-gateway
+```
+
+Metadata-only evaluation:
+
+```bash
+xone eval open-source --repos repos.txt --output evaluation.json --format json
+```
+
+Public shallow-clone evaluation:
+
+```bash
+xone eval open-source --repos repos.txt --clone-root .xone-eval-clones --output evaluation.md --format markdown
+```
+
+Safety boundary:
+
+- The command records public repository metadata and X-One command summaries.
+- It does not label third-party repositories as safe or unsafe.
+- It does not run third-party install scripts, test scripts, containers, or project commands.
+- `--clone-root` performs `git clone --depth 1` only, then runs X-One dry-run commands against the local clones.

{xone_cli-0.1.2 → xone_cli-0.1.3}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "xone-cli"
-version = "0.1.2"
+version = "0.1.3"
 description = "Unified CLI entry point for X-One Agent Evidence Loop workflows."
 readme = "README.md"
 requires-python = ">=3.11"

{xone_cli-0.1.2 → xone_cli-0.1.3}/src/xone_cli/__init__.py

@@ -1,3 +1,3 @@
 from __future__ import annotations
 
-__version__ = "0.1.2"
+__version__ = "0.1.3"

{xone_cli-0.1.2 → xone_cli-0.1.3}/src/xone_cli/cli.py

@@ -9,6 +9,7 @@ from collections.abc import Sequence
 from xone_cli import __version__
 from xone_cli.evidence import build_failure_packet, collect_evidence, collect_for_runbook, gate_evidence
 from xone_cli.lab import render_evidence_loop_lab
+from xone_cli.open_source_eval import evaluate_open_source
 from xone_cli.risk import render_risk_context
 from xone_cli.runbook import write_runbook
 from xone_cli.release import print_release_report, verify_release
@@ -71,6 +72,14 @@ def build_parser() -> argparse.ArgumentParser:
     verify.add_argument("--smoke", action="store_true")
     verify.add_argument("--json", action="store_true")
 
+    eval_parser = subparsers.add_parser("eval", help="run safe X-One evaluation workflows")
+    eval_subparsers = eval_parser.add_subparsers(dest="eval_command")
+    open_source = eval_subparsers.add_parser("open-source", help="evaluate public repositories as surrogate feedback")
+    open_source.add_argument("--repos", type=Path, required=True, help="text file containing owner/repo or GitHub URLs")
+    open_source.add_argument("--clone-root", type=Path, help="optional directory for shallow public clones")
+    open_source.add_argument("--output", type=Path, required=True, help="evaluation output path")
+    open_source.add_argument("--format", choices=("json", "markdown"), default="json")
+
     runbook = subparsers.add_parser("runbook", help="assemble a local X-One evidence runbook")
     _add_evidence_collection_args(runbook, require_base=False)
     runbook.add_argument("--output", type=Path)
@@ -145,6 +154,14 @@ def main(argv: Sequence[str] | None = None) -> int:
         print_release_report(report, as_json=args.json)
         return 0 if report["ok"] else 1
 
+    if args.command == "eval" and args.eval_command == "open-source":
+        return evaluate_open_source(
+            repos_file=args.repos,
+            clone_root=args.clone_root,
+            output=args.output,
+            output_format=args.format,
+        )
+
     if args.command == "runbook":
         code, report, message = collect_for_runbook(
             repo=args.repo,

xone_cli-0.1.3/src/xone_cli/open_source_eval.py

@@ -0,0 +1,226 @@
+from __future__ import annotations
+
+import json
+import subprocess
+import urllib.request
+from dataclasses import dataclass
+from datetime import date
+from pathlib import Path
+from urllib.parse import urlparse
+
+from xone_cli.evidence import collect_for_runbook
+from xone_cli.tooling import doctor_status
+
+
+BATCH_SCHEMA_VERSION = "xone.open_source_evaluation_batch.v1"
+RECORD_SCHEMA_VERSION = "xone.open_source_evaluation.v1"
+
+
+@dataclass(frozen=True)
+class EvalRecord:
+    source: str
+    source_url: str
+    snapshot_date: str
+    stars: int
+    forks: int
+    open_issues: int
+    default_branch: str
+    language: str | None
+    commands_run: list[dict]
+    sanitization: dict
+    decision: str
+
+    def to_dict(self) -> dict:
+        return {
+            "schema_version": RECORD_SCHEMA_VERSION,
+            "source": self.source,
+            "source_url": self.source_url,
+            "snapshot_date": self.snapshot_date,
+            "stars": self.stars,
+            "forks": self.forks,
+            "open_issues": self.open_issues,
+            "default_branch": self.default_branch,
+            "language": self.language,
+            "categories": [],
+            "xone_tools_used": ["xone-cli"],
+            "commands_run": self.commands_run,
+            "role_findings": [],
+            "sanitization": self.sanitization,
+            "decision": self.decision,
+        }
+
+
+def evaluate_open_source(
+    *,
+    repos_file: Path,
+    output: Path,
+    output_format: str,
+    clone_root: Path | None = None,
+    snapshot_date: date | None = None,
+) -> int:
+    snapshot = snapshot_date or date.today()
+    sources = read_repo_refs(repos_file)
+    records = []
+    for source in sources:
+        metadata = fetch_github_metadata(source, snapshot)
+        commands = [
+            {
+                "name": "doctor-json",
+                "command": "xone doctor --json",
+                "mode": "read-only",
+                "result": "pass" if doctor_status().ok else "needs-install",
+            }
+        ]
+        if clone_root is None:
+            commands.append(
+                {
+                    "name": "runbook-dry-run",
+                    "command": "xone runbook --repo <local-clone> --head HEAD --dry-run",
+                    "mode": "read-only",
+                    "result": "skipped",
+                    "reason": "clone-root not provided",
+                }
+            )
+        else:
+            try:
+                local_repo = clone_repo(metadata, clone_root)
+            except Exception as exc:
+                commands.append(
+                    {
+                        "name": "shallow-clone",
+                        "command": f"git clone --depth 1 {metadata['source_url']} <clone-root>",
+                        "mode": "read-only",
+                        "result": "fail",
+                        "output_summary": str(exc),
+                    }
+                )
+            else:
+                code, _report, message = collect_for_runbook(
+                    repo=str(local_repo),
+                    base=None,
+                    head="HEAD",
+                    test_logs=[],
+                    profile="strict",
+                    dry_run=True,
+                )
+                commands.append(
+                    {
+                        "name": "runbook-dry-run",
+                        "command": "xone runbook --repo <local-clone> --head HEAD --dry-run",
+                        "mode": "read-only",
+                        "result": "pass" if code == 0 else "fail",
+                        "output_summary": message.strip(),
+                    }
+                )
+        records.append(_record_from_metadata(metadata, commands).to_dict())
+
+    batch = {
+        "schema_version": BATCH_SCHEMA_VERSION,
+        "snapshot_date": snapshot.isoformat(),
+        "records": records,
+    }
+    if output_format == "json":
+        output.write_text(json.dumps(batch, indent=2, sort_keys=True) + "\n", encoding="utf-8")
+    else:
+        output.write_text(render_markdown(batch), encoding="utf-8")
+    print(f"Wrote {len(records)} open-source evaluation records to {output}")
+    return 0
+
+
+def read_repo_refs(path: Path) -> list[str]:
+    refs = []
+    for line in path.read_text(encoding="utf-8").splitlines():
+        value = line.strip()
+        if not value or value.startswith("#"):
+            continue
+        refs.append(normalize_repo_ref(value))
+    return refs
+
+
+def normalize_repo_ref(value: str) -> str:
+    parsed = urlparse(value)
+    if parsed.netloc == "github.com":
+        parts = [part for part in parsed.path.split("/") if part]
+        if len(parts) >= 2:
+            return f"{parts[0]}/{parts[1].removesuffix('.git')}"
+    return value.removesuffix(".git")
+
+
+def fetch_github_metadata(repo: str, snapshot_date: date) -> dict:
+    request = urllib.request.Request(
+        f"https://api.github.com/repos/{repo}",
+        headers={"Accept": "application/vnd.github+json", "User-Agent": "xone-cli-open-source-eval"},
+    )
+    with urllib.request.urlopen(request, timeout=20) as response:
+        payload = json.load(response)
+    return {
+        "source": payload["full_name"],
+        "source_url": payload["html_url"],
+        "snapshot_date": snapshot_date.isoformat(),
+        "stars": payload["stargazers_count"],
+        "forks": payload["forks_count"],
+        "open_issues": payload["open_issues_count"],
+        "default_branch": payload["default_branch"],
+        "language": payload.get("language"),
+    }
+
+
+def clone_repo(record: dict, root: Path) -> Path:
+    root.mkdir(parents=True, exist_ok=True)
+    target = root / record["source"].replace("/", "-")
+    if (target / ".git").exists():
+        return target
+    subprocess.run(
+        ["git", "clone", "--depth", "1", record["source_url"], str(target)],
+        check=True,
+        text=True,
+        stdout=subprocess.PIPE,
+        stderr=subprocess.PIPE,
+    )
+    return target
+
+
+def render_markdown(batch: dict) -> str:
+    lines = [
+        "# X-One Open-Source Adoption Evaluation",
+        "",
+        f"Snapshot date: {batch['snapshot_date']}",
+        "",
+        "| Source | Stars | Forks | Issues | Default Branch | Decision |",
+        "|---|---:|---:|---:|---|---|",
+    ]
+    for record in batch["records"]:
+        lines.append(
+            f"| {record['source']} | {record['stars']} | {record['forks']} | "
+            f"{record['open_issues']} | {record['default_branch']} | {record['decision']} |"
+        )
+    lines.extend(["", "## Command Evidence", ""])
+    for record in batch["records"]:
+        lines.append(f"### {record['source']}")
+        for command in record["commands_run"]:
+            lines.append(f"- {command.get('name', command['command'])}: {command['result']}")
+            lines.append(f"  - command: `{command['command']}`")
+            if command.get("output_summary"):
+                lines.append(f"  - summary: `{command['output_summary']}`")
+        lines.append("")
+    return "\n".join(lines).rstrip() + "\n"
+
+
+def _record_from_metadata(metadata: dict, commands: list[dict]) -> EvalRecord:
+    return EvalRecord(
+        source=metadata["source"],
+        source_url=metadata["source_url"],
+        snapshot_date=metadata["snapshot_date"],
+        stars=metadata["stars"],
+        forks=metadata["forks"],
+        open_issues=metadata["open_issues"],
+        default_branch=metadata["default_branch"],
+        language=metadata.get("language"),
+        commands_run=commands,
+        sanitization={
+            "raw_logs_committed": False,
+            "secrets_removed": True,
+            "third_party_safety_label_used": False,
+        },
+        decision="defer-with-reason",
+    )

{xone_cli-0.1.2 → xone_cli-0.1.3/src/xone_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: xone-cli
-Version: 0.1.2
+Version: 0.1.3
 Summary: Unified CLI entry point for X-One Agent Evidence Loop workflows.
 Author: X-One-AI
 License-Expression: MIT
@@ -61,6 +61,14 @@ xone runbook --head HEAD --dry-run
 
 `xone runbook` auto-detects the local repository default branch. Remote GitHub URLs are not accepted yet; clone the repository first, then pass the local path with `--repo`.
 
+For repeatable open-source adoption checks, use the read-only evaluator:
+
+```bash
+xone eval open-source --repos repos.txt --output evaluation.md --format markdown
+```
+
+Add `--clone-root .xone-eval-clones` when you want X-One to shallow-clone public repositories and run only X-One dry-run commands against those local clones. It does not run third-party install scripts, tests, containers, or project commands.
+
 `xone-cli` orchestrates these X-One tools:
 
 - `agent-pr-evidence`
@@ -81,7 +89,8 @@ Current recommended tool versions:
 - It does not post GitHub comments.
 - It does not modify repositories automatically.
 - It does not make allow/deny runtime enforcement decisions.
-- It does not make hidden network calls.
+- It does not run third-party project scripts.
+- `xone eval open-source` makes explicit GitHub API and `git clone --depth 1` calls only when requested by the command.
 
 ## Docs
 

{xone_cli-0.1.2 → xone_cli-0.1.3}/src/xone_cli.egg-info/SOURCES.txt

@@ -30,6 +30,7 @@ src/xone_cli/cli.py
 src/xone_cli/evidence.py
 src/xone_cli/lab.py
 src/xone_cli/model.py
+src/xone_cli/open_source_eval.py
 src/xone_cli/release.py
 src/xone_cli/repo_context.py
 src/xone_cli/risk.py
@@ -44,6 +45,7 @@ src/xone_cli.egg-info/top_level.txt
 tests/test_cli.py
 tests/test_docs.py
 tests/test_evidence.py
+tests/test_open_source_eval.py
 tests/test_release.py
 tests/test_repo_context.py
 tests/test_risk_lab.py

{xone_cli-0.1.2 → xone_cli-0.1.3}/tests/test_cli.py

@@ -12,7 +12,7 @@ def _fake_tool(bin_dir: Path, name: str, body: str = "echo tool") -> None:
 
 def test_version_outputs_package_version(capsys):
     assert main(["--version"]) == 0
-    assert capsys.readouterr().out.strip() == "xone 0.1.2"
+    assert capsys.readouterr().out.strip() == "xone 0.1.3"
 
 
 def test_doctor_json_reports_required_tools(tmp_path, monkeypatch, capsys):

{xone_cli-0.1.2 → xone_cli-0.1.3}/tests/test_docs.py

@@ -19,17 +19,20 @@ def test_docs_and_package_metadata_stay_aligned():
     assert "xone doctor" in english
     assert "xone doctor --install-plan" in english
     assert "xone runbook --head HEAD --dry-run" in english
+    assert "xone eval open-source" in english
     assert "Remote GitHub URLs are not accepted yet" in english
     assert "xone doctor" in chinese
     assert "xone doctor --install-plan" in chinese
     assert "xone runbook --head HEAD --dry-run" in chinese
+    assert "xone eval open-source" in chinese
     assert "远程 GitHub URL" in chinese
     assert "git clone --depth 1 https://github.com/owner/repo" in Path("docs/install.md").read_text(encoding="utf-8")
+    assert "xone eval open-source --repos repos.txt" in Path("docs/install.md").read_text(encoding="utf-8")
     assert "xone release verify --build --install --smoke" in Path("docs/release.md").read_text(encoding="utf-8")
     assert 'name = "xone-cli"' in pyproject
-    assert 'version = "0.1.2"' in pyproject
-    assert __version__ == "0.1.2"
-    assert "## 0.1.2" in changelog
+    assert 'version = "0.1.3"' in pyproject
+    assert __version__ == "0.1.3"
+    assert "## 0.1.3" in changelog
     assert 'xone = "xone_cli.cli:entrypoint"' in pyproject
     assert "python -m pytest -q" in ci
     assert "xone runbook --base HEAD~1 --head HEAD --dry-run" in ci

xone_cli-0.1.3/tests/test_open_source_eval.py

@@ -0,0 +1,139 @@
+import json
+from datetime import date
+
+from xone_cli.cli import main
+
+
+def test_eval_open_source_json_outputs_sanitized_records(tmp_path, monkeypatch):
+    repos = tmp_path / "repos.txt"
+    repos.write_text("openai/codex\nhttps://github.com/docker/mcp-gateway\n", encoding="utf-8")
+    output = tmp_path / "records.json"
+
+    def fake_metadata(repo, snapshot_date):
+        return {
+            "source": repo,
+            "source_url": f"https://github.com/{repo}",
+            "snapshot_date": snapshot_date.isoformat(),
+            "stars": 10,
+            "forks": 2,
+            "open_issues": 1,
+            "default_branch": "main",
+            "language": "Python",
+        }
+
+    monkeypatch.setattr("xone_cli.open_source_eval.fetch_github_metadata", fake_metadata)
+
+    assert main(["eval", "open-source", "--repos", str(repos), "--output", str(output), "--format", "json"]) == 0
+
+    payload = json.loads(output.read_text(encoding="utf-8"))
+    assert payload["schema_version"] == "xone.open_source_evaluation_batch.v1"
+    assert [record["source"] for record in payload["records"]] == ["openai/codex", "docker/mcp-gateway"]
+    assert payload["records"][0]["sanitization"] == {
+        "raw_logs_committed": False,
+        "secrets_removed": True,
+        "third_party_safety_label_used": False,
+    }
+    assert payload["records"][0]["commands_run"][0]["command"] == "xone doctor --json"
+    assert payload["records"][0]["commands_run"][1]["result"] == "skipped"
+
+
+def test_eval_open_source_with_clone_root_runs_local_runbook_dry_run(tmp_path, monkeypatch):
+    repos = tmp_path / "repos.txt"
+    repos.write_text("upstash/context7\n", encoding="utf-8")
+    output = tmp_path / "records.md"
+    clone_root = tmp_path / "clones"
+
+    def fake_metadata(repo, snapshot_date):
+        return {
+            "source": repo,
+            "source_url": f"https://github.com/{repo}",
+            "snapshot_date": snapshot_date.isoformat(),
+            "stars": 20,
+            "forks": 3,
+            "open_issues": 4,
+            "default_branch": "master",
+            "language": "TypeScript",
+        }
+
+    def fake_clone(record, root):
+        repo = root / "upstash-context7"
+        (repo / ".git").mkdir(parents=True)
+        (repo / ".git" / "HEAD").write_text("ref: refs/heads/master\n", encoding="utf-8")
+        return repo
+
+    monkeypatch.setattr("xone_cli.open_source_eval.fetch_github_metadata", fake_metadata)
+    monkeypatch.setattr("xone_cli.open_source_eval.clone_repo", fake_clone)
+
+    assert main(
+        [
+            "eval",
+            "open-source",
+            "--repos",
+            str(repos),
+            "--clone-root",
+            str(clone_root),
+            "--output",
+            str(output),
+            "--format",
+            "markdown",
+        ]
+    ) == 0
+
+    markdown = output.read_text(encoding="utf-8")
+    assert "# X-One Open-Source Adoption Evaluation" in markdown
+    assert "upstash/context7" in markdown
+    assert "runbook-dry-run" in markdown
+    assert "--base master" in markdown
+
+
+def test_eval_open_source_records_clone_failure_without_crashing(tmp_path, monkeypatch):
+    repos = tmp_path / "repos.txt"
+    repos.write_text("cline/cline\nopenai/codex\n", encoding="utf-8")
+    output = tmp_path / "records.json"
+    clone_root = tmp_path / "clones"
+
+    def fake_metadata(repo, snapshot_date):
+        return {
+            "source": repo,
+            "source_url": f"https://github.com/{repo}",
+            "snapshot_date": snapshot_date.isoformat(),
+            "stars": 10,
+            "forks": 2,
+            "open_issues": 1,
+            "default_branch": "main",
+            "language": "TypeScript",
+        }
+
+    def fake_clone(record, root):
+        if record["source"] == "cline/cline":
+            raise RuntimeError("clone failed")
+        repo = root / "openai-codex"
+        (repo / ".git").mkdir(parents=True)
+        (repo / ".git" / "HEAD").write_text("ref: refs/heads/main\n", encoding="utf-8")
+        return repo
+
+    monkeypatch.setattr("xone_cli.open_source_eval.fetch_github_metadata", fake_metadata)
+    monkeypatch.setattr("xone_cli.open_source_eval.clone_repo", fake_clone)
+
+    assert main(
+        [
+            "eval",
+            "open-source",
+            "--repos",
+            str(repos),
+            "--clone-root",
+            str(clone_root),
+            "--output",
+            str(output),
+            "--format",
+            "json",
+        ]
+    ) == 0
+
+    payload = json.loads(output.read_text(encoding="utf-8"))
+    assert [record["source"] for record in payload["records"]] == ["cline/cline", "openai/codex"]
+    failed_command = payload["records"][0]["commands_run"][1]
+    assert failed_command["name"] == "shallow-clone"
+    assert failed_command["result"] == "fail"
+    assert failed_command["mode"] == "read-only"
+    assert payload["records"][1]["commands_run"][1]["name"] == "runbook-dry-run"