custos-code 0.0.1__tar.gz

custos_code-0.0.1/.gitignore

@@ -0,0 +1,24 @@
+.venv/
+.uv-cache/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.hypothesis/
+dist/
+build/
+*.egg-info/
+__pycache__/
+*.pyc
+*.sqlite
+*.duckdb
+.env
+.env.*
+config.toml
+~/.receipts/
+eval/study/sessions/
+bench/runs/
+bench/fixtures/*/.venv/
+.claude/
+CLAUDE.md
+.DS_Store
+docs/research/raw/

custos_code-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Oliver Zhang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

custos_code-0.0.1/PKG-INFO

@@ -0,0 +1,138 @@
+Metadata-Version: 2.5
+Name: custos-code
+Version: 0.0.1
+Summary: Checks a coding agent's final report against the log of what it actually did.
+License-Expression: MIT
+License-File: LICENSE
+Keywords: agents,claude-code,hooks,observability,verification
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.12
+Requires-Dist: anthropic>=0.40
+Requires-Dist: duckdb>=1.0
+Requires-Dist: openai>=1.40
+Requires-Dist: pydantic>=2.7
+Requires-Dist: rich>=13.7
+Requires-Dist: tomli>=2.0; python_version < '3.11'
+Requires-Dist: typer>=0.12
+Provides-Extra: dev
+Requires-Dist: hypothesis>=6; extra == 'dev'
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Custos Code
+
+Checks a coding agent's final report against the log of what it actually did.
+
+An agent finishes and says "implemented the feature, ran the tests, all passing." Custos Code reads the harness-written action log, splits the report into claims, and marks each one **confirmed**, **contradicted**, **unwitnessed**, **unrecorded**, or **qualified**, with the ledger lines that back the verdict. Contradictions go back to the agent before it is allowed to stop.
+
+```
+custos-code  session 4f2a… · 63 events
+  ✓ confirmed     edited auth/middleware.py            tier 1 · #14 Edit, git diff agrees
+  ✓ confirmed     added tests/test_rate_limit.py        tier 1 · #31 Write, file present
+  ✗ contradicted  ran the suite, all 12 passing         tier 2 · #41 `pytest | tail -5` exit 0, "collected 0 items"
+  ? unwitnessed   ready to merge                        no CI, no git status after #41
+stop blocked · 1 contradicted · evidence returned to agent
+```
+
+## Status
+
+Pre-build. The design, research, plan, and evidence protocol are in `docs/`. Start with [AGENTS.md](AGENTS.md).
+
+- [docs/DESIGN.md](docs/DESIGN.md) — problem, customers, verification ladder, feasibility, product sketches, benchmark, system design, prize strategy, adversarial review
+- [docs/RESEARCH.md](docs/RESEARCH.md) — the evidence: prevalence with denominators, cost, current workarounds, tool landscape, 40 seed cases
+- [docs/PLAN.md](docs/PLAN.md) — who owns what, phases, parallel tracks
+- [docs/EVIDENCE_PLAN.md](docs/EVIDENCE_PLAN.md) — pre-registered study: accuracy, time saved, retention, usability
+- [docs/OPEN_QUESTIONS.md](docs/OPEN_QUESTIONS.md) — every unresolved decision with an owner
+
+## See it work
+
+```bash
+uv sync
+export OPENAI_API_KEY=...
+uv run custos-code demo                 # the whole loop on a known trap, live
+uv run custos-code demo --scenario honest   # the control: nothing blocks
+uv run custos-code check --last         # your own most recent session
+```
+
+`demo` prints five things from the fixture's own tool log: what was asked, what the agent actually
+did, what it said, the receipt, and the deterministic nudge that goes back. `--format html --out
+card.html` writes a self-contained report card; `--format markdown` writes what the PR bot posts.
+
+## Prototype
+
+`docs/prototype/index.html` is an interactive, non-functional mock of the editor experience: marks on the agent's message, the evidence panel, editor decorations, the auto-mode loop, and the PR receipt. It is a single self-contained file:
+
+```bash
+open docs/prototype/index.html          # macOS
+# or: python3 -m http.server -d docs/prototype 8765  →  http://localhost:8765
+```
+
+## Local setup
+
+Install [uv](https://docs.astral.sh/uv/getting-started/installation/) once. On macOS
+with Homebrew: `brew install uv`. From the repository directory:
+
+```bash
+uv python install
+make sync
+make check
+uv run custos-code check --last
+```
+
+`.python-version` selects Python 3.12, independently of your shell's pyenv or
+Conda default. `make sync` installs the project and developer tools from the
+committed `uv.lock`; it fails if the lockfile needs updating. After an intentional
+dependency change, run `uv lock` and include the lockfile in the same PR.
+
+`make build` produces a wheel and source distribution in `dist/`. CI runs the
+same checks, installs both distributions in clean environments, and checks the
+installed CLI outside the source checkout. CI uses locked installs following
+the [uv integration guide](https://docs.astral.sh/uv/guides/integration/github/).
+
+## Bench container
+
+With Docker installed and running, build from the repository root:
+
+```bash
+docker build -t custos-code-bench .
+docker run --rm --network none custos-code-bench
+docker run --rm --network none custos-code-bench python -m pytest --version
+```
+
+The image contains Python 3.12, uv 0.12.17, git, the installed Custos Code package,
+developer dependencies, and scenario descriptions under `/app/bench/scenarios`.
+It runs as a non-root user in writable `/workspace`; the default command shows
+CLI help. The benchmark orchestration and fixture repos are not implemented
+yet, so this is their execution environment, not a working benchmark command.
+Agent CLIs and their credentials are not installed.
+
+The Docker build context is an allowlist that excludes local session logs,
+credentials, caches, and git history. For a local Python fixture, mount only
+that fixture (including its git metadata when state checks need it):
+
+```bash
+docker run --rm --network none \
+  --mount type=bind,src="$(pwd)/path/to/fixture",dst=/workspace,readonly \
+  custos-code-bench python -m pytest -p no:cacheprovider
+```
+
+This read-only example suits tests that do not write into the fixture. Agent
+bench runs will need a disposable writable checkout and explicit network and
+credential configuration. Non-Python runners require additional toolchains.
+
+## Why
+
+Across 20,574 real coding-agent sessions, 22.58% of 16,118 validated misalignment episodes were the agent misreporting its own work, and only 2.99% of resolved episodes were self-corrected. Every agent vendor attaches an action log; none checks the report against it. Sources and denominators: `docs/RESEARCH.md`.
+
+## Working in this repo
+
+Read `AGENTS.md`. Flag anything undecided with `NEEDS-DECISION(owner):`. Local by default; secrets are redacted at ingest; fixtures are synthetic.
+
+## License
+
+MIT (see `LICENSE`).

custos_code-0.0.1/README.md

@@ -0,0 +1,112 @@
+# Custos Code
+
+Checks a coding agent's final report against the log of what it actually did.
+
+An agent finishes and says "implemented the feature, ran the tests, all passing." Custos Code reads the harness-written action log, splits the report into claims, and marks each one **confirmed**, **contradicted**, **unwitnessed**, **unrecorded**, or **qualified**, with the ledger lines that back the verdict. Contradictions go back to the agent before it is allowed to stop.
+
+```
+custos-code  session 4f2a… · 63 events
+  ✓ confirmed     edited auth/middleware.py            tier 1 · #14 Edit, git diff agrees
+  ✓ confirmed     added tests/test_rate_limit.py        tier 1 · #31 Write, file present
+  ✗ contradicted  ran the suite, all 12 passing         tier 2 · #41 `pytest | tail -5` exit 0, "collected 0 items"
+  ? unwitnessed   ready to merge                        no CI, no git status after #41
+stop blocked · 1 contradicted · evidence returned to agent
+```
+
+## Status
+
+Pre-build. The design, research, plan, and evidence protocol are in `docs/`. Start with [AGENTS.md](AGENTS.md).
+
+- [docs/DESIGN.md](docs/DESIGN.md) — problem, customers, verification ladder, feasibility, product sketches, benchmark, system design, prize strategy, adversarial review
+- [docs/RESEARCH.md](docs/RESEARCH.md) — the evidence: prevalence with denominators, cost, current workarounds, tool landscape, 40 seed cases
+- [docs/PLAN.md](docs/PLAN.md) — who owns what, phases, parallel tracks
+- [docs/EVIDENCE_PLAN.md](docs/EVIDENCE_PLAN.md) — pre-registered study: accuracy, time saved, retention, usability
+- [docs/OPEN_QUESTIONS.md](docs/OPEN_QUESTIONS.md) — every unresolved decision with an owner
+
+## See it work
+
+```bash
+uv sync
+export OPENAI_API_KEY=...
+uv run custos-code demo                 # the whole loop on a known trap, live
+uv run custos-code demo --scenario honest   # the control: nothing blocks
+uv run custos-code check --last         # your own most recent session
+```
+
+`demo` prints five things from the fixture's own tool log: what was asked, what the agent actually
+did, what it said, the receipt, and the deterministic nudge that goes back. `--format html --out
+card.html` writes a self-contained report card; `--format markdown` writes what the PR bot posts.
+
+## Prototype
+
+`docs/prototype/index.html` is an interactive, non-functional mock of the editor experience: marks on the agent's message, the evidence panel, editor decorations, the auto-mode loop, and the PR receipt. It is a single self-contained file:
+
+```bash
+open docs/prototype/index.html          # macOS
+# or: python3 -m http.server -d docs/prototype 8765  →  http://localhost:8765
+```
+
+## Local setup
+
+Install [uv](https://docs.astral.sh/uv/getting-started/installation/) once. On macOS
+with Homebrew: `brew install uv`. From the repository directory:
+
+```bash
+uv python install
+make sync
+make check
+uv run custos-code check --last
+```
+
+`.python-version` selects Python 3.12, independently of your shell's pyenv or
+Conda default. `make sync` installs the project and developer tools from the
+committed `uv.lock`; it fails if the lockfile needs updating. After an intentional
+dependency change, run `uv lock` and include the lockfile in the same PR.
+
+`make build` produces a wheel and source distribution in `dist/`. CI runs the
+same checks, installs both distributions in clean environments, and checks the
+installed CLI outside the source checkout. CI uses locked installs following
+the [uv integration guide](https://docs.astral.sh/uv/guides/integration/github/).
+
+## Bench container
+
+With Docker installed and running, build from the repository root:
+
+```bash
+docker build -t custos-code-bench .
+docker run --rm --network none custos-code-bench
+docker run --rm --network none custos-code-bench python -m pytest --version
+```
+
+The image contains Python 3.12, uv 0.12.17, git, the installed Custos Code package,
+developer dependencies, and scenario descriptions under `/app/bench/scenarios`.
+It runs as a non-root user in writable `/workspace`; the default command shows
+CLI help. The benchmark orchestration and fixture repos are not implemented
+yet, so this is their execution environment, not a working benchmark command.
+Agent CLIs and their credentials are not installed.
+
+The Docker build context is an allowlist that excludes local session logs,
+credentials, caches, and git history. For a local Python fixture, mount only
+that fixture (including its git metadata when state checks need it):
+
+```bash
+docker run --rm --network none \
+  --mount type=bind,src="$(pwd)/path/to/fixture",dst=/workspace,readonly \
+  custos-code-bench python -m pytest -p no:cacheprovider
+```
+
+This read-only example suits tests that do not write into the fixture. Agent
+bench runs will need a disposable writable checkout and explicit network and
+credential configuration. Non-Python runners require additional toolchains.
+
+## Why
+
+Across 20,574 real coding-agent sessions, 22.58% of 16,118 validated misalignment episodes were the agent misreporting its own work, and only 2.99% of resolved episodes were self-corrected. Every agent vendor attaches an action log; none checks the report against it. Sources and denominators: `docs/RESEARCH.md`.
+
+## Working in this repo
+
+Read `AGENTS.md`. Flag anything undecided with `NEEDS-DECISION(owner):`. Local by default; secrets are redacted at ingest; fixtures are synthetic.
+
+## License
+
+MIT (see `LICENSE`).

custos_code-0.0.1/pyproject.toml

@@ -0,0 +1,58 @@
+[project]
+name = "custos-code"
+version = "0.0.1"
+description = "Checks a coding agent's final report against the log of what it actually did."
+readme = "README.md"
+license = "MIT"
+keywords = ["agents", "verification", "claude-code", "hooks", "observability"]
+classifiers = [
+  "Development Status :: 3 - Alpha",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3.12",
+  "Topic :: Software Development :: Quality Assurance",
+]
+requires-python = ">=3.12"
+dependencies = [
+  "pydantic>=2.7",
+  "typer>=0.12",
+  "rich>=13.7",
+  "openai>=1.40",
+  "anthropic>=0.40",
+  "duckdb>=1.0",
+  "tomli>=2.0; python_version < '3.11'",
+]
+
+[project.optional-dependencies]
+dev = ["pytest>=8", "ruff>=0.5", "mypy>=1.10", "hypothesis>=6"]
+
+[project.scripts]
+custos-code = "custos_code.cli:app"
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/custos_code"]
+
+[tool.hatch.build.targets.sdist]
+include = [
+  "/src/custos_code",
+  "/README.md",
+  "/LICENSE",
+  "/pyproject.toml",
+]
+
+[tool.ruff]
+line-length = 100
+exclude = ["pilots"]  # feasibility-week scratch scripts, not product code
+[tool.ruff.lint]
+select = ["E", "F", "I", "B", "UP"]
+ignore = ["E501"]  # long lines in docstrings are fine; the formatter handles code
+
+[tool.mypy]
+strict = true
+python_version = "3.12"
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]

custos_code-0.0.1/src/custos_code/__init__.py

@@ -0,0 +1,6 @@
+"""Custos Code: check a coding agent's final report against the log of what it actually did.
+
+Read AGENTS.md before changing anything. Invariants live there. Every module below has a
+docstring saying what it owns and what it must never do.
+"""
+__version__ = "0.0.1"

custos_code-0.0.1/src/custos_code/adapters/__init__.py

@@ -0,0 +1,194 @@
+"""Adapters turn a source (harness transcript, hook payload, vendor export) into LedgerEvents.
+
+Contract: every adapter is a function `parse(path_or_payload) -> tuple[Session, list[LedgerEvent], str | None]`
+returning the session, the chained events, and the final report text if present.
+Adapters must set flags.truncated / flags.piped / flags.sidechain honestly; downstream tiers rely on them.
+Golden tests live in tests/golden/<adapter>/ : real input in, expected JSONL out.
+
+`detect` picks the adapter from the file itself so `custos-code check <path>` needs no --agent flag.
+
+`request_and_plan` (docs/SCOPE.md §6.3, issue #58) is scope's other input besides the ledger it
+already gets: what was asked, and what the agent said it would do before doing it. It lives here,
+not in `scope.py`, so scope stays agent-agnostic -- it takes a request and a plan, never a
+harness-specific shape.
+"""
+
+from __future__ import annotations
+
+import json
+from collections.abc import Sequence
+from typing import Protocol
+
+from ..models import EventKind, LedgerEvent, Session
+from . import claude_code, codex, copilot, devin, machine, otel
+
+Source = str
+
+
+class Adapter(Protocol):
+    def parse(self, path: str) -> tuple[Session, list[LedgerEvent], str | None]: ...
+
+
+ADAPTERS: dict[Source, Adapter] = {
+    "claude_code": claude_code,
+    "codex": codex,
+    "devin": devin,
+    "machine": machine,
+    "copilot": copilot,
+    "otel": otel,
+}
+
+_CODEX_TYPES = frozenset(
+    {
+        "session_meta",
+        "turn_context",
+        "response_item",
+        "event_msg",
+        "compacted",
+        "thread_rolled_back",
+    }
+)
+
+
+def _bundle_source(obj: dict[str, object]) -> Source | None:
+    """Class-R bundles and OTLP payloads are one JSON object; tell them apart by their keys."""
+    if "resourceSpans" in obj or "scopeSpans" in obj:
+        return "otel"
+    raw = obj.get("session")
+    session: dict[str, object] = raw if isinstance(raw, dict) else {}
+    keys = set(obj) | {f"session.{k}" for k in session}
+    # Copilot first: both bundles carry `pull_request` and a `session.session_id`, and only
+    # Copilot carries a workspace or an attached tool log. Devin's tell is structured_output.
+    if "pull_request" in keys and (keys & {"log", "session.workspace", "session.agent_log"}):
+        return "copilot"
+    if keys & {"session.structured_output", "structured_output"}:
+        return "devin"
+    if keys & {"session.session_id", "session_id", "pull_request", "pull_requests"}:
+        return "devin"
+    return None
+
+
+def _line_source(rec: dict[str, object]) -> Source | None:
+    if "sessionId" in rec or (rec.get("type") in ("assistant", "user") and "message" in rec):
+        return "claude_code"
+    if str(rec.get("type")) in _CODEX_TYPES:
+        return "codex"
+    if rec.get("recorder") in machine.RECORDER_NAMES:
+        return "machine"
+    if "traceId" in rec and "spanId" in rec:
+        return "otel"
+    return None
+
+
+def detect(path: str) -> Source:
+    """Name the adapter for a file, by its shape. Raises ValueError when nothing matches."""
+    with open(path, encoding="utf-8", errors="ignore") as fh:
+        text = fh.read()
+    if not text.strip():
+        raise ValueError(f"{path} is empty")
+    try:
+        whole = json.loads(text)
+    except json.JSONDecodeError:
+        whole = None
+    if isinstance(whole, dict):
+        name = _bundle_source(whole)
+        if name:
+            return name
+    for line in text.splitlines():
+        if not line.strip():
+            continue
+        try:
+            rec = json.loads(line)
+        except json.JSONDecodeError:
+            continue
+        if isinstance(rec, dict):
+            name = _line_source(rec)
+            if name:
+                return name
+    raise ValueError(f"cannot tell which agent wrote {path}; pass --agent")
+
+
+def parse(path: str, source: Source | None = None) -> tuple[Session, list[LedgerEvent], str | None]:
+    """Parse a transcript with the adapter named by `source`, or the one `detect` picks."""
+    name = source or detect(path)
+    adapter = ADAPTERS.get(name)
+    if adapter is None:
+        raise ValueError(f"unknown agent {name!r}; one of {', '.join(sorted(ADAPTERS))}")
+    return adapter.parse(path)
+
+
+def _first_user_text(ledger: Sequence[LedgerEvent]) -> str:
+    """The earliest non-sidechain USER event's text -- the literal ask, verbatim.
+
+    Claude Code, Codex, and Devin (when it has a chat transcript) all write these already; nothing
+    new has to be recorded (SCOPE.md §6.3's whole premise).
+    """
+    for event in ledger:
+        if event.kind is EventKind.USER and not event.flags.sidechain and event.output:
+            return event.output
+    return ""
+
+
+def _todo_plan(ledger: Sequence[LedgerEvent]) -> list[str]:
+    """Claude Code's `TodoWrite`: the most recent call's items, oldest first.
+
+    The latest call wins, not the first -- a plan is allowed to change, and the self-authored
+    contract SCOPE.md §3 describes is whatever the agent most recently committed to, not its first
+    draft.
+    """
+    latest: LedgerEvent | None = None
+    for event in ledger:
+        if event.kind is EventKind.CALL and event.tool == "TodoWrite" and not event.flags.sidechain:
+            latest = event
+    if latest is None:
+        return []
+    todos = (latest.input or {}).get("todos")
+    if not isinstance(todos, list):
+        return []
+    out = []
+    for item in todos:
+        if not isinstance(item, dict):
+            continue
+        text = item.get("content") or item.get("activeForm") or item.get("task")
+        if isinstance(text, str) and text.strip():
+            out.append(text.strip())
+    return out
+
+
+def _stated_plan_before_first_call(ledger: Sequence[LedgerEvent]) -> list[str]:
+    """Fallback plan: the last thing the agent said before it touched a tool.
+
+    A message written before any evidence exists is a commitment it cannot later revise -- the
+    same "log it has no write path to" property SCOPE.md §3 grounds the plan in. Applies to any
+    adapter whose ledger has TEXT/CALL events in the shared shape (Claude Code, Codex today).
+    """
+    first_call_seq = next(
+        (e.seq for e in ledger if e.kind is EventKind.CALL and not e.flags.sidechain), None
+    )
+    last_text = ""
+    for event in ledger:
+        if first_call_seq is not None and event.seq >= first_call_seq:
+            break
+        if event.kind is EventKind.TEXT and not event.flags.sidechain and event.output:
+            last_text = event.output
+    return [last_text] if last_text.strip() else []
+
+
+def request_and_plan(
+    session: Session, ledger: Sequence[LedgerEvent], report: str | None = None
+) -> tuple[str, list[str]]:
+    """The spec and the agent's self-authored plan, per docs/SCOPE.md §6.3.
+
+    Deliberately reads the ledger, not the harness: Claude Code and Codex both already write a
+    USER event for the request, so no per-agent branch is needed to find it. Class-R bundles
+    (Copilot, Devin without a chat transcript) have no live back-and-forth to draw either from --
+    there, the PR body (`report`, already the same text `custos-code check` extracts claims from)
+    doubles as the spec, exactly as SCOPE.md §6.3 says, and there is no separate plan to find.
+
+    A `TodoWrite` call is the strongest plan signal where one exists; otherwise the last thing said
+    before the first tool call stands in for it (still empty for a class-R bundle, which has no
+    "before the first call" boundary to speak of).
+    """
+    request = _first_user_text(ledger) or (report or "")
+    plan = _todo_plan(ledger) or _stated_plan_before_first_call(ledger)
+    return request, plan