contribos 0.2.0__tar.gz

contribos-0.2.0/PKG-INFO

@@ -0,0 +1,93 @@
+Metadata-Version: 2.4
+Name: contribos
+Version: 0.2.0
+Summary: Get trusted, get merged, come back: a guide for new open-source contributors in the AI era.
+Project-URL: Homepage, https://github.com/adityatiwari101104/contribos
+Project-URL: Issues, https://github.com/adityatiwari101104/contribos/issues
+Keywords: open-source,contributors,github,mcp,ai-policy,good-first-issue,onboarding
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+
+# ContribOS
+
+<!-- mcp-name: io.github.adityatiwari101104/contribos -->
+
+**Get trusted. Get merged. Come back.** ContribOS helps new open-source contributors earn trust. It finds where you're welcome, shows how this repo wants the change done, checks that you truly understand your change, coaches you through review, and turns merged PRs into a record the next maintainer can verify.
+
+In 2026, writing code is not the hard part of contributing. AI made PRs cheap, maintainers answered with PR caps, vouch lists and AI policies, and newcomer merge rates fell. ContribOS makes your PR worth a maintainer's time. See `../plan-v4.md` and `../plan-v5-research.md` for the full reasoning.
+
+ContribOS **never** opens PRs, posts comments or claims issues for you, and it never writes your explanation. It points, and you decide.
+
+## The journey
+
+| Step | Command | What you get |
+|---|---|---|
+| Find | `contribos find --lang python --topic web` | Open issues sorted into Good bet / Possible / Skip, each with evidence: claimed? open PR? maintainer active? clear? Plus repo welcome: archived, vouch list, AI policy, outside-PR merge rate, how long outside PRs wait for a first human reply, how many stall, a stale bot, and your own open PRs there |
+| Rules | `contribos policy owner/repo` | The repo's AI policy, disclosure format (`Assisted-by:` and similar trailers), vouch gate, "no AI on good first issues", issue-first and claim-first norms, tests, changelog, DCO/CLA, style tools and activity, each quoted with `file:line`. `--json` gives the same rules in a form agents can read |
+| Propose | `contribos claim <issue-url>` | A short proposal to post *before* coding: likely files, a similar-size past PR, where the test goes, and one real question. Warns about taken issues, no maintainer yet, good-first-issue AI rules and issue-first rules. If the repo has a vouch list, it drafts the introduction first |
+| Tone | `contribos tone comment.md --kind comment` | Flags machine-written tells (delve, buzzwords, assistant openers, filler), length, a missing question and leftover placeholders in anything you're about to post |
+| Agent rules | `contribos agent-rules . --ai "Claude Code"` | Teaches your coding agent this repo's rules: a SKILL.md for Claude Code, Codex, Copilot and Cursor, a hook that stops the agent opening PRs or posting comments, and a commit-msg hook for sign-off and the AI-disclosure trailer. All local-only, excluded from git |
+| Setup | `contribos setup owner/repo` | Exact steps taken from the repo's CI and contributing guide: runtime version, install, services, env vars, test and lint commands |
+| Diagnose | `contribos setup --diagnose output.txt` | What a failure means: missing dependency, wrong version, service down, env var, native build tools, flaky test, or a real test failure (and how to check it isn't yours) |
+| Understand | `contribos brief <issue-url>` | Files to read with the reasons each was picked, similar past PRs and what reviewers said, related tests, look-alike files to leave alone, house rules, and likely review questions. Add `--llm` for a plain-words summary that is grounded in the evidence |
+| Prove | `contribos check --explain explain.md --verify-test "pytest tests/test_x.py" --issue 123 --ai "Claude Code: drafted the test" --pr-draft pr.md` | Pre-submit review: scope versus similar past changes, tests, changelog, sign-off, debug leftovers, AI policy and disclosure trailers, competing PRs for the same issue. `--verify-test` runs your test with the fix and again with your source files swapped back to the base version, so you can show it fails without your change. It checks that *your* explanation covers every changed file and states what you're unsure of, then drafts the PR: what and why, how you verified it, what you're unsure about, related past changes, AI assistance |
+| Respond | `contribos review <pr-url> --path .` | Each reviewer comment classified (blocking, change, question, nit…), the code it points at, the house rule it echoes, a reply draft, what's still unanswered and for how long, and failing CI |
+| Grow | `contribos record <github-user> --html me.html` | A public, linkable record: merged PRs, change requests, whether you answered every review comment, whether you explained your change and disclosed AI use |
+| Agents | `contribos mcp` | Nine MCP tools for Claude Code, Cursor, Codex and others (policy, brief, setup, diagnose, check, review, find, claim, tone), taking text as input where an agent has text: `claude mcp add contribos -- python -m contribos mcp` |
+
+## Install
+
+```sh
+pip install -e .          # Python 3.10+, git. No other dependencies.
+export GITHUB_TOKEN=...   # needed for find, claim/brief from an issue URL, review, record
+```
+
+Everything that can run offline does: `policy`, `brief --title`, `claim --title`, `setup`, `check`, `bench` and `review --data` need only `git`. Add `--offline` to skip the API, and `--update` to fetch new commits. Clones and indexes are cached in `~/.cache/contribos` (`CONTRIBOS_CACHE`).
+
+Optional AI summaries: `CONTRIBOS_LLM=anthropic` (with `ANTHROPIC_API_KEY`) or `CONTRIBOS_LLM=openai` (with `OPENAI_API_KEY`), plus optional `CONTRIBOS_MODEL`. The model only explains evidence ContribOS already gathered. Any sentence citing a file outside that evidence is dropped.
+
+## How it works
+
+- `repo.py` handles cached clones. It reads any revision with `git show` and `git grep`, so no checkout is needed.
+- `precedent.py` builds a SQLite index of main-line changes (squash and merge PRs), the files they touched, and the issues they fixed.
+- `policy.py` is the policy radar, built from contribution docs, AI policies, templates, vouch lists, CI config and history.
+- `brief.py` ranks files by rare-term matches in code, path matches, and files touched by similar past PRs. It also finds related tests via co-change history.
+- `check.py` is the pre-submit check, the proof-of-understanding check and the PR draft.
+- `setup_doctor.py` produces setup steps from CI and docs, and diagnoses failures.
+- `find.py` checks issue takeability, repo welcome and responsiveness.
+- `claim.py` drafts the proposal and vouch introduction.
+- `proof.py` runs a test with and without your change (files restored in `finally`).
+- `tone.py` checks text you're about to post.
+- `agent_rules.py` writes the local agent skill and hooks.
+- `review.py` classifies review comments, drafts replies and tracks follow-through.
+- `record.py` builds the contribution record (markdown plus a self-contained HTML page).
+- `llm.py` is the optional provider layer (Anthropic or OpenAI) with citation grounding.
+- `mcp_server.py` is a dependency-free MCP stdio server.
+- `github.py` is the optional API client. Every caller handles its absence.
+- `bench.py` runs the history benchmark.
+
+## Benchmark (2026-09-28, offline, PR titles standing in for issues)
+
+| Repo | Brief hit@5 | Grep-only hit@5 |
+|---|---|---|
+| pallets/flask (30 PRs) | 0.77 | 0.73 |
+| pytest-dev/pytest (30 PRs) | 0.87 | 0.87 |
+
+Matching against past PRs barely improves file finding, which supports the plan's bet that finding files is a commodity. The value is in how past PRs did it and what reviewers asked. The next benchmark needs real issue text and review comments, which requires `GITHUB_TOKEN`.
+
+## Status
+
+- Tested on real repos (Flask, pytest, Ghostty): `policy`, `brief`, `claim --title`, `setup`, `check`, `bench`, `mcp`.
+- Tested with realistic fake GitHub data, but not yet against the live API (it was blocked in the build environment): `find`, `claim <url>`, `review <url>`, `record`, and the `brief` review quotes. Run them with a token before relying on them.
+- Tested on temporary git repos: `agent-rules` (git status stays clean, the guard blocks `gh pr create`, the commit-msg hook enforces sign-off and the trailer), `check --verify-test`, `tone`, policy radar v2.
+- Tests: `python3 -m unittest discover -s tests` (37 tests).
+
+## Publishing (when ready)
+
+- PyPI: `python -m build && twine upload dist/*` (version 0.2.0 in `pyproject.toml`).
+- MCP registry: `server.json` describes the package as `io.github.adityatiwari101104/contribos`; the `mcp-name` comment at the top of this README lets the registry verify the PyPI package. Check `server.json` against the current registry schema before publishing.

contribos-0.2.0/README.md

@@ -0,0 +1,78 @@
+# ContribOS
+
+<!-- mcp-name: io.github.adityatiwari101104/contribos -->
+
+**Get trusted. Get merged. Come back.** ContribOS helps new open-source contributors earn trust. It finds where you're welcome, shows how this repo wants the change done, checks that you truly understand your change, coaches you through review, and turns merged PRs into a record the next maintainer can verify.
+
+In 2026, writing code is not the hard part of contributing. AI made PRs cheap, maintainers answered with PR caps, vouch lists and AI policies, and newcomer merge rates fell. ContribOS makes your PR worth a maintainer's time. See `../plan-v4.md` and `../plan-v5-research.md` for the full reasoning.
+
+ContribOS **never** opens PRs, posts comments or claims issues for you, and it never writes your explanation. It points, and you decide.
+
+## The journey
+
+| Step | Command | What you get |
+|---|---|---|
+| Find | `contribos find --lang python --topic web` | Open issues sorted into Good bet / Possible / Skip, each with evidence: claimed? open PR? maintainer active? clear? Plus repo welcome: archived, vouch list, AI policy, outside-PR merge rate, how long outside PRs wait for a first human reply, how many stall, a stale bot, and your own open PRs there |
+| Rules | `contribos policy owner/repo` | The repo's AI policy, disclosure format (`Assisted-by:` and similar trailers), vouch gate, "no AI on good first issues", issue-first and claim-first norms, tests, changelog, DCO/CLA, style tools and activity, each quoted with `file:line`. `--json` gives the same rules in a form agents can read |
+| Propose | `contribos claim <issue-url>` | A short proposal to post *before* coding: likely files, a similar-size past PR, where the test goes, and one real question. Warns about taken issues, no maintainer yet, good-first-issue AI rules and issue-first rules. If the repo has a vouch list, it drafts the introduction first |
+| Tone | `contribos tone comment.md --kind comment` | Flags machine-written tells (delve, buzzwords, assistant openers, filler), length, a missing question and leftover placeholders in anything you're about to post |
+| Agent rules | `contribos agent-rules . --ai "Claude Code"` | Teaches your coding agent this repo's rules: a SKILL.md for Claude Code, Codex, Copilot and Cursor, a hook that stops the agent opening PRs or posting comments, and a commit-msg hook for sign-off and the AI-disclosure trailer. All local-only, excluded from git |
+| Setup | `contribos setup owner/repo` | Exact steps taken from the repo's CI and contributing guide: runtime version, install, services, env vars, test and lint commands |
+| Diagnose | `contribos setup --diagnose output.txt` | What a failure means: missing dependency, wrong version, service down, env var, native build tools, flaky test, or a real test failure (and how to check it isn't yours) |
+| Understand | `contribos brief <issue-url>` | Files to read with the reasons each was picked, similar past PRs and what reviewers said, related tests, look-alike files to leave alone, house rules, and likely review questions. Add `--llm` for a plain-words summary that is grounded in the evidence |
+| Prove | `contribos check --explain explain.md --verify-test "pytest tests/test_x.py" --issue 123 --ai "Claude Code: drafted the test" --pr-draft pr.md` | Pre-submit review: scope versus similar past changes, tests, changelog, sign-off, debug leftovers, AI policy and disclosure trailers, competing PRs for the same issue. `--verify-test` runs your test with the fix and again with your source files swapped back to the base version, so you can show it fails without your change. It checks that *your* explanation covers every changed file and states what you're unsure of, then drafts the PR: what and why, how you verified it, what you're unsure about, related past changes, AI assistance |
+| Respond | `contribos review <pr-url> --path .` | Each reviewer comment classified (blocking, change, question, nit…), the code it points at, the house rule it echoes, a reply draft, what's still unanswered and for how long, and failing CI |
+| Grow | `contribos record <github-user> --html me.html` | A public, linkable record: merged PRs, change requests, whether you answered every review comment, whether you explained your change and disclosed AI use |
+| Agents | `contribos mcp` | Nine MCP tools for Claude Code, Cursor, Codex and others (policy, brief, setup, diagnose, check, review, find, claim, tone), taking text as input where an agent has text: `claude mcp add contribos -- python -m contribos mcp` |
+
+## Install
+
+```sh
+pip install -e .          # Python 3.10+, git. No other dependencies.
+export GITHUB_TOKEN=...   # needed for find, claim/brief from an issue URL, review, record
+```
+
+Everything that can run offline does: `policy`, `brief --title`, `claim --title`, `setup`, `check`, `bench` and `review --data` need only `git`. Add `--offline` to skip the API, and `--update` to fetch new commits. Clones and indexes are cached in `~/.cache/contribos` (`CONTRIBOS_CACHE`).
+
+Optional AI summaries: `CONTRIBOS_LLM=anthropic` (with `ANTHROPIC_API_KEY`) or `CONTRIBOS_LLM=openai` (with `OPENAI_API_KEY`), plus optional `CONTRIBOS_MODEL`. The model only explains evidence ContribOS already gathered. Any sentence citing a file outside that evidence is dropped.
+
+## How it works
+
+- `repo.py` handles cached clones. It reads any revision with `git show` and `git grep`, so no checkout is needed.
+- `precedent.py` builds a SQLite index of main-line changes (squash and merge PRs), the files they touched, and the issues they fixed.
+- `policy.py` is the policy radar, built from contribution docs, AI policies, templates, vouch lists, CI config and history.
+- `brief.py` ranks files by rare-term matches in code, path matches, and files touched by similar past PRs. It also finds related tests via co-change history.
+- `check.py` is the pre-submit check, the proof-of-understanding check and the PR draft.
+- `setup_doctor.py` produces setup steps from CI and docs, and diagnoses failures.
+- `find.py` checks issue takeability, repo welcome and responsiveness.
+- `claim.py` drafts the proposal and vouch introduction.
+- `proof.py` runs a test with and without your change (files restored in `finally`).
+- `tone.py` checks text you're about to post.
+- `agent_rules.py` writes the local agent skill and hooks.
+- `review.py` classifies review comments, drafts replies and tracks follow-through.
+- `record.py` builds the contribution record (markdown plus a self-contained HTML page).
+- `llm.py` is the optional provider layer (Anthropic or OpenAI) with citation grounding.
+- `mcp_server.py` is a dependency-free MCP stdio server.
+- `github.py` is the optional API client. Every caller handles its absence.
+- `bench.py` runs the history benchmark.
+
+## Benchmark (2026-09-28, offline, PR titles standing in for issues)
+
+| Repo | Brief hit@5 | Grep-only hit@5 |
+|---|---|---|
+| pallets/flask (30 PRs) | 0.77 | 0.73 |
+| pytest-dev/pytest (30 PRs) | 0.87 | 0.87 |
+
+Matching against past PRs barely improves file finding, which supports the plan's bet that finding files is a commodity. The value is in how past PRs did it and what reviewers asked. The next benchmark needs real issue text and review comments, which requires `GITHUB_TOKEN`.
+
+## Status
+
+- Tested on real repos (Flask, pytest, Ghostty): `policy`, `brief`, `claim --title`, `setup`, `check`, `bench`, `mcp`.
+- Tested with realistic fake GitHub data, but not yet against the live API (it was blocked in the build environment): `find`, `claim <url>`, `review <url>`, `record`, and the `brief` review quotes. Run them with a token before relying on them.
+- Tested on temporary git repos: `agent-rules` (git status stays clean, the guard blocks `gh pr create`, the commit-msg hook enforces sign-off and the trailer), `check --verify-test`, `tone`, policy radar v2.
+- Tests: `python3 -m unittest discover -s tests` (37 tests).
+
+## Publishing (when ready)
+
+- PyPI: `python -m build && twine upload dist/*` (version 0.2.0 in `pyproject.toml`).
+- MCP registry: `server.json` describes the package as `io.github.adityatiwari101104/contribos`; the `mcp-name` comment at the top of this README lets the registry verify the PyPI package. Check `server.json` against the current registry schema before publishing.

contribos-0.2.0/contribos/__init__.py

@@ -0,0 +1,3 @@
+"""ContribOS: help new contributors earn trust in open source."""
+
+__version__ = "0.2.0"

contribos-0.2.0/contribos/__main__.py

@@ -0,0 +1,5 @@
+import sys
+
+from .cli import main
+
+sys.exit(main())

contribos-0.2.0/contribos/agent_rules.py

@@ -0,0 +1,148 @@
+"""Agent guardrails: make your coding agent follow this repo's rules, where slop starts.
+
+Writes, into your local checkout:
+- a skill (`.claude/skills/contribos/SKILL.md`, mirrored to `.agents/skills/`)
+  that Claude Code, Codex, Copilot and Cursor load, with this repo's rules;
+- a Claude Code hook that blocks the agent from opening PRs or posting
+  comments on its own;
+- a git `commit-msg` hook that enforces sign-off and the AI-disclosure line
+  this repo asks for.
+
+None of these files are part of your contribution: they're added to
+`.git/info/exclude` so they never end up in your PR.
+"""
+
+from __future__ import annotations
+
+import json
+import stat
+from pathlib import Path
+
+from .policy import Policy, structured
+from .setup_doctor import SetupPlan
+
+GUARD = '''#!/usr/bin/env python3
+"""ContribOS guard: stop the coding agent from acting publicly on its own."""
+import json, re, sys
+
+data = json.load(sys.stdin)
+cmd = (data.get("tool_input") or {}).get("command", "")
+if re.search(r"\\bgh\\s+(pr\\s+(create|comment|review|merge|edit|ready|close|reopen)|"
+             r"issue\\s+(create|comment|edit|close|reopen)|"
+             r"api\\b.*(-X|--method)\\s*(POST|PATCH|PUT|DELETE))", cmd, re.I):
+    print("ContribOS: opening PRs, posting comments or claiming issues must be done by the human "
+          "contributor, not the agent. Prepare the text and let the human post it.", file=sys.stderr)
+    sys.exit(2)
+sys.exit(0)
+'''
+
+COMMIT_MSG = '''#!/bin/sh
+# ContribOS commit-msg hook: enforce this repo's sign-off and AI-disclosure rules.
+msg="$1"
+{dco}
+if [ -f "$(git rev-parse --git-dir)/contribos-ai-used" ] && [ -n "{trailer}" ]; then
+  if ! grep -qi "^{trailer}:" "$msg"; then
+    echo "ContribOS: this repo asks for a '{trailer}' line when AI helped. Add, for example:" >&2
+    echo "  {trailer}: $(cat "$(git rev-parse --git-dir)/contribos-ai-used")" >&2
+    exit 1
+  fi
+fi
+exit 0
+'''
+
+DCO_CHECK = '''if ! grep -q "^Signed-off-by:" "$msg"; then
+  echo "ContribOS: this repo requires sign-off. Commit with: git commit -s" >&2
+  exit 1
+fi'''
+
+
+def skill_text(policy: Policy, setup: SetupPlan | None) -> str:
+    s = structured(policy)
+    ai = s["ai"]
+    rules = ["Never open pull requests, post comments, or claim issues yourself. Draft text; the human posts it.",
+             "Keep the change minimal and focused on the issue. Do not refactor unrelated code.",
+             "The human must understand every line. Explain your changes plainly and point out anything uncertain.",
+             "Never write the human's PR explanation, issue comments or review replies as final text; offer a "
+             "short draft they will rewrite."]
+    if ai["stance"] == "restricted":
+        rules.append("This repo restricts AI use. Read the AI policy evidence below and stop if the task is not allowed.")
+    if s["trust_gate"]:
+        rules.append("This repo only accepts PRs from vouched contributors. If the human isn't vouched yet, "
+                     "help them write a short introduction instead of code.")
+    if ai["no_ai_on_good_first_issues"]:
+        rules.append("Do not work on issues labelled good-first-issue: this repo reserves them for humans learning.")
+    if ai["disclosure_trailer"]:
+        rules.append(f"Every commit you help with must include a `{ai['disclosure_trailer']}` line naming the tool.")
+    elif ai["disclose_in_pr"]:
+        rules.append("Remind the human to disclose in the PR description which tool was used and for what.")
+    if s["dco"]:
+        rules.append("Commits must be signed off by the human (`git commit -s`). Never add Signed-off-by yourself.")
+    if s["tests_expected"]:
+        rules.append("Add or update a test that fails without the fix.")
+    if s["changelog"]:
+        rules.append("Add a changelog entry in the format the repo uses.")
+    if s["issue_first"] or s["claim_first"]:
+        rules.append("Only work on an issue a maintainer has agreed to. If unsure, stop and ask the human.")
+    steps = []
+    if setup:
+        for st in setup.steps:
+            if st.commands and ("test" in st.title.lower() or "lint" in st.title.lower() or "pre-commit" in st.title.lower()):
+                steps += [f"`{c}`" for c in st.commands[:2]]
+    L = ["---", "name: contribos", "description: Rules for contributing to this open-source repository. Use before "
+         "writing or committing any change here.", "---", "",
+         f"# Contributing to {policy.repo}", "",
+         "You are helping a human contributor. Maintainers here review human work and reject unexplained AI output.",
+         "", "## Rules", ""] + [f"- {r}" for r in rules]
+    if steps:
+        L += ["", "## Run before saying you're done", ""] + [f"- {c}" for c in steps]
+    L += ["", "## Tools", "",
+          "If the ContribOS MCP server is available, use `contribos_brief` to find relevant files and past PRs, "
+          "and `contribos_check` before the human submits.", "", "## Evidence for these rules", ""]
+    for f in policy.findings[:8]:
+        src = f" ({f.evidence[0].source})" if f.evidence else ""
+        L.append(f"- {f.topic}: {f.verdict}{src}")
+    return "\n".join(L) + "\n"
+
+
+def install(root: Path, policy: Policy, setup: SetupPlan | None, ai_note: str | None = None) -> list[str]:
+    written = []
+    text = skill_text(policy, setup)
+    for d in (root / ".claude/skills/contribos", root / ".agents/skills/contribos"):
+        d.mkdir(parents=True, exist_ok=True)
+        (d / "SKILL.md").write_text(text)
+        written.append((d / "SKILL.md").relative_to(root).as_posix())
+
+    guard = root / ".contribos/guard.py"
+    guard.parent.mkdir(exist_ok=True)
+    guard.write_text(GUARD)
+    guard.chmod(guard.stat().st_mode | stat.S_IEXEC)
+    written.append(".contribos/guard.py")
+    settings_path = root / ".claude/settings.local.json"
+    settings = json.loads(settings_path.read_text()) if settings_path.exists() else {}
+    hooks = settings.setdefault("hooks", {}).setdefault("PreToolUse", [])
+    entry = {"matcher": "Bash", "hooks": [{"type": "command", "command": "python3 .contribos/guard.py"}]}
+    if entry not in hooks:
+        hooks.append(entry)
+    settings_path.write_text(json.dumps(settings, indent=2) + "\n")
+    written.append(".claude/settings.local.json")
+
+    s = structured(policy)
+    git_dir = root / ".git"
+    if git_dir.is_dir():
+        hook = git_dir / "hooks" / "commit-msg"
+        target = hook if not hook.exists() or "ContribOS" in hook.read_text() else hook.with_name("commit-msg.contribos")
+        target.write_text(COMMIT_MSG.format(dco=DCO_CHECK if s["dco"] else "", trailer=s["ai"]["disclosure_trailer"] or ""))
+        target.chmod(target.stat().st_mode | stat.S_IEXEC)
+        written.append(target.relative_to(root).as_posix())
+        marker = git_dir / "contribos-ai-used"
+        if ai_note:
+            marker.write_text(ai_note + "\n")
+        exclude = git_dir / "info" / "exclude"
+        exclude.parent.mkdir(exist_ok=True)
+        current = exclude.read_text() if exclude.exists() else ""
+        add = [p for p in (".claude/skills/contribos/", ".agents/skills/contribos/", ".contribos/",
+                           ".claude/settings.local.json") if p not in current]
+        if add:
+            exclude.write_text(current.rstrip("\n") + ("\n" if current else "") + "# ContribOS (local only)\n"
+                               + "\n".join(add) + "\n")
+    return written

contribos-0.2.0/contribos/bench.py

@@ -0,0 +1,80 @@
+"""History benchmark: replay past merged changes and check whether the brief finds their files.
+
+For each sampled change we rebuild the brief at its parent commit, using only
+history older than the change, and the PR title as a stand-in for the issue.
+Titles are shorter and more precise than real issues, so treat scores as an
+upper-bound smoke test; the real benchmark uses linked issue text (needs the
+GitHub API).
+"""
+
+from __future__ import annotations
+
+import random
+from dataclasses import dataclass
+
+from . import brief as brief_mod
+from .policy import is_code
+from .precedent import Change
+from .repo import Repo
+
+
+@dataclass
+class Case:
+    change: Change
+    code_files: list[str]
+
+
+@dataclass
+class Result:
+    case: Case
+    full_hit: bool
+    full_recall: float
+    grep_hit: bool
+    grep_recall: float
+
+
+def sample(history: list[Change], n: int, seed: int = 7) -> list[Case]:
+    pool = []
+    for i, c in enumerate(history[:-50]):  # keep older history as precedent
+        code = [f for f in c.files if is_code(f)]
+        if c.pr and 1 <= len(code) <= 4 and len(c.title.split()) >= 4 \
+                and not c.title.lower().startswith(("bump", "release", "merge", "revert", "update dependencies")):
+            pool.append(Case(c, code))
+    random.Random(seed).shuffle(pool)
+    return pool[:n]
+
+
+def _score(top: list[str], truth: list[str]) -> tuple[bool, float]:
+    found = len(set(top) & set(truth))
+    return found > 0, found / len(truth)
+
+
+def run(repo: Repo, history: list[Change], n: int = 20, k: int = 5, log=print) -> list[Result]:
+    results = []
+    order = {c.sha: i for i, c in enumerate(history)}
+    for case in sample(history, n):
+        c = case.change
+        parent = f"{c.sha}^1"
+        older = history[order[c.sha] + 1:]
+        full = brief_mod.build(repo, c.title, "", older, None, rev=parent, top=k)
+        grep = brief_mod.build(repo, c.title, "", [], None, rev=parent, top=k)
+        fh, fr = _score([h.path for h in full.files], case.code_files)
+        gh, gr = _score([h.path for h in grep.files], case.code_files)
+        results.append(Result(case, fh, fr, gh, gr))
+        log(f"  #{c.pr} {'✓' if fh else '·'} {'✓' if gh else '·'}  {_short(c.title, 70)}")
+    return results
+
+
+def summary(results: list[Result]) -> dict:
+    n = len(results) or 1
+    return {
+        "cases": len(results),
+        "brief_hit@5": sum(r.full_hit for r in results) / n,
+        "brief_recall@5": sum(r.full_recall for r in results) / n,
+        "grep_only_hit@5": sum(r.grep_hit for r in results) / n,
+        "grep_only_recall@5": sum(r.grep_recall for r in results) / n,
+    }
+
+
+def _short(text: str, n: int) -> str:
+    return text if len(text) <= n else text[:n - 1].rstrip() + "…"