devin-qa-pack 0.2.0__tar.gz → 0.3.0__tar.gz

{devin_qa_pack-0.2.0/src/devin_qa_pack.egg-info → devin_qa_pack-0.3.0}/PKG-INFO

@@ -1,12 +1,13 @@
 Metadata-Version: 2.4
 Name: devin-qa-pack
-Version: 0.2.0
-Summary: QA audit for Devin sessions: verifies deliverable claims (tests, commits, files, pushes) against tool_call_state ground truth.
+Version: 0.3.0
+Summary: QA audit for agent sessions: verifies deliverable claims (tests, commits, files, pushes) against recorded tool-call ground truth across Devin CLI, Copilot CLI, Codex CLI and VS Code chat stores.
 Author: Icaro0310
 License: MIT
 Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 Requires-Dist: devin-internals-spec<0.4.0,>=0.3.0
+Requires-Dist: devin-install-scheduler>=0.1.0
 Provides-Extra: dev
 Requires-Dist: pytest>=8; extra == "dev"
 Requires-Dist: pyyaml>=6; extra == "dev"
@@ -53,7 +54,8 @@ Part of the [awesome-devin](https://github.com/Icaro0310/awesome-devin) ecosyste
 
 Flagship QA audit for Devin sessions: checks that what a session *claims* it
 delivered is backed by what its tool calls *actually did* — tests run,
-commits created, files written, work pushed, HTTP statuses returned —
+commits created, files written, work pushed, HTTP statuses returned,
+eval scores and coverage numbers reported, work delegated to subagents —
 and prints a verdict per session: `PASS` / `PARTIAL` / `UNVERIFIED`.
 
 ## The problem
@@ -81,8 +83,11 @@ Devin persists every tool call in `sessions.db → tool_call_state`. This
 tool reads that table via
 [devin-internals-spec](https://github.com/Icaro0310/devin-internals-spec)
 and treats it as **ground truth**: a "tests passed" claim must be backed
-by a run/execute call that ran a test runner and completed; "committed
-`sha`" must show the hash in a call or in `git log`. Text-only auditors
+by a run/execute call that ran a test runner and completed; "47 tests
+passed" must match a recorded `47 passed` in the tool output; "committed
+`sha`" must show the hash in a call or in `git log`; "coverage is at
+87%" or "the eval score was 0.92" must match a recorded percent/score
+in tool-call output. Text-only auditors
 *cannot* do this — remove Devin's store and the check disappears.
 
 ## Install
@@ -109,14 +114,25 @@ curl -fsSL https://raw.githubusercontent.com/Icaro0310/devin-assure/main/install
 ship the wheel, sdist and CycloneDX SBOM per tag.
 
 **See it in action** — run the reproducible
-[Agent Assurance demo](examples/agent-assurance): one command generates a
+[Agent Assurance demo](https://github.com/Icaro0310/devin-assure/tree/main/packages/qa-pack/examples/agent-assurance): one command generates a
 deliberately flawed agent session, then independently verifies what the
 agent actually did. No Devin install needed.
 
-<a href="examples/agent-assurance"><img src="examples/agent-assurance/demo.gif" alt="agent-assurance demo — UNVERIFIED, PARTIAL, PASS" width="80%"/></a>
+<div align="center">
+<a href="https://github.com/Icaro0310/devin-assure/tree/main/packages/qa-pack/examples/agent-assurance"><img src="https://raw.githubusercontent.com/Icaro0310/devin-assure/main/packages/qa-pack/examples/agent-assurance/demo.gif" alt="agent-assurance demo — UNVERIFIED, PARTIAL, PASS" width="80%"/></a>
+</div>
 
 ## Usage
 
+Every command reads local Devin data by default — `sessions.db`, the
+`.devin/` config tree, transcript exports — and never writes back to
+those stores. Two opt-ins do reach the network: `audit --source mcp`
+fetches the session's cloud events, and `--online` probes live URLs.
+Point `--sessions-db` at a copied database to keep reads fully off the
+original store — `session-end` still writes its verdict under the Devin
+data dir (`<data-dir>/qa/`; redirect with `--data-dir` or `--out`).
+Reports are not auto-redacted: excerpt the verdict block before sharing.
+
 ```bash
 # audit one session (exact id or unique prefix)
 devin-qa-pack audit --session <id> --sessions-db path/to/sessions.db
@@ -132,6 +148,10 @@ devin-qa-pack report --sessions-db path/to/sessions.db --out report.html
 # (hook mode — writes a JSON side file, never touches the session)
 devin-qa-pack session-end
 
+# schedule a daily audit sweep over every detected store
+# (cron / Task Scheduler / elapsed registry — opt-in)
+devin-qa-pack install --daily
+
 # intent vs. coverage (QA-4): did the agent touch what the prompt named?
 devin-qa-pack intent <session> --sessions-db path/to/sessions.db
 
@@ -148,9 +168,9 @@ devin-qa-pack audit --source mcp --session <cloud-session-id>
 Real output, on the synthetic fixture from the quick start:
 
 ```text
-UNVERIFIED  sess-unverifiable  Mystery session — 1 claim(s): 0 verified, 0 disputed, 1 unverifiable
+UNVERIFIED  sess-unverifiable  Mystery session — 1 claim(s): 0 verified, 0 disputed, 1 unverifiable — uncovered: file/docs/spec.pdf
   [unverifiable] file/docs/spec.pdf: no matching tool call and working dir not on disk
-PARTIAL  sess-disputed  Disputed session — 2 claim(s): 0 verified, 2 disputed, 0 unverifiable
+PARTIAL  sess-disputed  Disputed session — 2 claim(s): 0 verified, 2 disputed, 0 unverifiable — uncovered: tests/tests, commit/deadbee
   [disputed    ] tests/tests: no execute call matching `tests` recorded
   [disputed    ] commit/deadbee: no commit call recorded and no repo on disk to check
 PASS  sess-verified  Verified session — 4 claim(s): 4 verified, 0 disputed, 0 unverifiable
@@ -166,7 +186,10 @@ Recorded as an asciicast: [assets/demo.cast](assets/demo.cast) —
 The `report` subcommand audits all sessions (or one with `--session`,
 bounded with `--limit`) and writes a single deterministic HTML file:
 verdict counts, a per-session table and a per-claim breakdown with
-evidence and the source excerpt for every checked claim.
+evidence and the source excerpt for every checked claim. Verdicts and
+claim statuses render as labeled badges (color + text, never color
+alone) — the report stays self-contained: inline CSS, zero JS, no
+external assets.
 
 `--sessions-db` may be omitted. It auto-detects
 `%APPDATA%/devin/cli/sessions.db` on Windows and
@@ -257,6 +280,14 @@ when the session resolves. A one-line
 summary is also printed. `--limit N` bounds the number of claims
 verified.
 
+Each run also appends one compact JSON line to `verdicts.jsonl` next to
+the side file (`<data-dir>/qa/verdicts.jsonl` by default) — a
+machine-readable ledger with `session_id`, `verdict`, `audited_at`,
+claim counts, the `uncovered` claim names (disputed/unverifiable
+`kind/detail` list, omitted when empty) and the skip `reason`, for
+dashboards and triggers. The append is best-effort; the side file
+remains the contract.
+
 **Fail-soft:** `session-end` always exits `0` once it ran — the verdict
 travels in the side file, so a hook can never fail the host session. A
 session that cannot be resolved produces a `SKIPPED` verdict (still
@@ -284,6 +315,41 @@ stdin and exports `DEVIN_SESSION_ID`):
 }
 ```
 
+## Scheduled audit sweep (`install --daily`)
+
+The `SessionEnd` hook audits live; `devin-qa-pack install --daily` adds
+the complementary **scheduled sweep** — a daily `devin-qa-pack audit
+--all` over every detected surface (`--surface` narrows it) for
+sessions the hook missed (Devin closed early, hook not wired, foreign
+stores). It writes the full payload to
+`<data-dir>/qa/audit-sweep.json` and appends one compact verdict line
+per session (`origin: "sweep"`) to the same
+`<data-dir>/qa/verdicts.jsonl` ledger `session-end` feeds. Read-only on
+every store: the only writes are under the data dir's `qa/` folder, and
+the job does not need Devin closed.
+
+```bash
+devin-qa-pack install --daily                  # auto-detect backend
+devin-qa-pack install --daily --backend cron   # pin one backend
+devin-qa-pack install --daily --data-dir /custom/devin --surface devin-cli
+```
+
+Backend selection follows the shared `devin-install-scheduler`
+convention (same as devin-state's `devin-janitor`/`devin-backup`): a
+tagged `@daily` **crontab** line on Linux/macOS, a daily **Task
+Scheduler** entry on Windows, or — on locked-down machines without
+either — a job record in `<config-dir>/.devin-ecosystem/scheduled.json`
+(the `elapsed` backend) that devin-powerups' `UserPromptSubmit` hook
+ticks (`python tools/schedule.py check --run`). `--backend` pins a
+backend, `--config-dir` relocates the registry, `--data-dir` reshapes
+where the registered command writes, `--json` prints the job record.
+Both `--sessions-db` and `--out` in the registered command are baked to
+absolute paths at install time, so the job does not depend on
+`DEVIN_DATA_DIR`/XDG env surviving into the scheduler's minimal
+environment (re-run `install` if the data dir ever moves).
+The sweep exits `1` when any session is `PARTIAL`/`UNVERIFIED` — the
+verdicts land in the artifacts, the exit code is informational.
+
 ## Pre-commit gate
 
 The repo root ships `.pre-commit-hooks.yaml` — reference it from a
@@ -313,10 +379,15 @@ in the entry and the scope flags are mutually exclusive — run
 `audit --transcript` runs the same claim/evidence model over non-Devin
 transcripts via the adapters in `src/devin_qa_pack/adapters/`:
 
-- **`aider`** — `.aider.chat.history.md`. `> /run` and `> /test` records
-  become execute calls with `unknown` status (the history does not store
-  exit codes), so test claims resolve `unverifiable`; commit claims are
-  checked against `git log` in the working directory.
+- **`aider`** — `.aider.chat.history.md`. `#### /run`, `#### /test`,
+  `#### /git` and `#### !cmd` input lines become execute calls with
+  `unknown` status (the history does not store exit codes), so action
+  claims like "tests passed" resolve `unverifiable`; the `>`-blockquoted
+  tool output (`Applied edit to …`, `Commit <sha> …`, recorded test
+  output) is pooled as evidence, so value claims — pass counts, HTTP
+  statuses, coverage, eval scores — and file/commit details still
+  corroborate. Commit claims are also checked against `git log` in the
+  working directory.
 - **`claude-code`** — session `.jsonl` transcripts. `Bash` → execute,
   `Write`/`Edit`/`MultiEdit`/`NotebookEdit` → write, `tool_result`
   blocks mark calls completed/failed.
@@ -384,6 +455,14 @@ network is off by default and the audit is fully offline otherwise.
   output: a claim is `verified` when a recorded status matches,
   `disputed` when outputs record a different status and `unverifiable`
   when no output records any status. No request is ever replayed.
+- Counted test claims work the same way: "47 tests passed" is `verified`
+  when recorded output reports `47 passed`, `disputed` when it reports
+  a different count and `unverifiable` when no output records a count.
+  So do coverage and eval-score claims: "coverage is at 87%" needs a
+  recorded 87% (coverage.py's `TOTAL` row, a `coverage`/`line-rate`
+  payload field, or a "coverage …87%" output string); "the eval score
+  was 0.92" needs a recorded `0.92`. Values compare on the scale
+  written — an `85%` claim matches a recorded `85`, never `0.85`.
 - File/commit checks use `sessions.working_directory` only when it
   exists on disk and is a git repo — otherwise they rely on tool-call
   evidence alone.

devin_qa_pack-0.2.0/PKG-INFO → devin_qa_pack-0.3.0/README.md

@@ -1,18 +1,3 @@
-Metadata-Version: 2.4
-Name: devin-qa-pack
-Version: 0.2.0
-Summary: QA audit for Devin sessions: verifies deliverable claims (tests, commits, files, pushes) against tool_call_state ground truth.
-Author: Icaro0310
-License: MIT
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-Requires-Dist: devin-internals-spec<0.4.0,>=0.3.0
-Provides-Extra: dev
-Requires-Dist: pytest>=8; extra == "dev"
-Requires-Dist: pyyaml>=6; extra == "dev"
-Provides-Extra: mcp
-Requires-Dist: mcp>=1.2; extra == "mcp"
-
 <div align="center">
 
 <img src="assets/banner.svg" alt="devin-qa-pack" width="100%"/>
@@ -53,7 +38,8 @@ Part of the [awesome-devin](https://github.com/Icaro0310/awesome-devin) ecosyste
 
 Flagship QA audit for Devin sessions: checks that what a session *claims* it
 delivered is backed by what its tool calls *actually did* — tests run,
-commits created, files written, work pushed, HTTP statuses returned —
+commits created, files written, work pushed, HTTP statuses returned,
+eval scores and coverage numbers reported, work delegated to subagents —
 and prints a verdict per session: `PASS` / `PARTIAL` / `UNVERIFIED`.
 
 ## The problem
@@ -81,8 +67,11 @@ Devin persists every tool call in `sessions.db → tool_call_state`. This
 tool reads that table via
 [devin-internals-spec](https://github.com/Icaro0310/devin-internals-spec)
 and treats it as **ground truth**: a "tests passed" claim must be backed
-by a run/execute call that ran a test runner and completed; "committed
-`sha`" must show the hash in a call or in `git log`. Text-only auditors
+by a run/execute call that ran a test runner and completed; "47 tests
+passed" must match a recorded `47 passed` in the tool output; "committed
+`sha`" must show the hash in a call or in `git log`; "coverage is at
+87%" or "the eval score was 0.92" must match a recorded percent/score
+in tool-call output. Text-only auditors
 *cannot* do this — remove Devin's store and the check disappears.
 
 ## Install
@@ -109,14 +98,25 @@ curl -fsSL https://raw.githubusercontent.com/Icaro0310/devin-assure/main/install
 ship the wheel, sdist and CycloneDX SBOM per tag.
 
 **See it in action** — run the reproducible
-[Agent Assurance demo](examples/agent-assurance): one command generates a
+[Agent Assurance demo](https://github.com/Icaro0310/devin-assure/tree/main/packages/qa-pack/examples/agent-assurance): one command generates a
 deliberately flawed agent session, then independently verifies what the
 agent actually did. No Devin install needed.
 
-<a href="examples/agent-assurance"><img src="examples/agent-assurance/demo.gif" alt="agent-assurance demo — UNVERIFIED, PARTIAL, PASS" width="80%"/></a>
+<div align="center">
+<a href="https://github.com/Icaro0310/devin-assure/tree/main/packages/qa-pack/examples/agent-assurance"><img src="https://raw.githubusercontent.com/Icaro0310/devin-assure/main/packages/qa-pack/examples/agent-assurance/demo.gif" alt="agent-assurance demo — UNVERIFIED, PARTIAL, PASS" width="80%"/></a>
+</div>
 
 ## Usage
 
+Every command reads local Devin data by default — `sessions.db`, the
+`.devin/` config tree, transcript exports — and never writes back to
+those stores. Two opt-ins do reach the network: `audit --source mcp`
+fetches the session's cloud events, and `--online` probes live URLs.
+Point `--sessions-db` at a copied database to keep reads fully off the
+original store — `session-end` still writes its verdict under the Devin
+data dir (`<data-dir>/qa/`; redirect with `--data-dir` or `--out`).
+Reports are not auto-redacted: excerpt the verdict block before sharing.
+
 ```bash
 # audit one session (exact id or unique prefix)
 devin-qa-pack audit --session <id> --sessions-db path/to/sessions.db
@@ -132,6 +132,10 @@ devin-qa-pack report --sessions-db path/to/sessions.db --out report.html
 # (hook mode — writes a JSON side file, never touches the session)
 devin-qa-pack session-end
 
+# schedule a daily audit sweep over every detected store
+# (cron / Task Scheduler / elapsed registry — opt-in)
+devin-qa-pack install --daily
+
 # intent vs. coverage (QA-4): did the agent touch what the prompt named?
 devin-qa-pack intent <session> --sessions-db path/to/sessions.db
 
@@ -148,9 +152,9 @@ devin-qa-pack audit --source mcp --session <cloud-session-id>
 Real output, on the synthetic fixture from the quick start:
 
 ```text
-UNVERIFIED  sess-unverifiable  Mystery session — 1 claim(s): 0 verified, 0 disputed, 1 unverifiable
+UNVERIFIED  sess-unverifiable  Mystery session — 1 claim(s): 0 verified, 0 disputed, 1 unverifiable — uncovered: file/docs/spec.pdf
   [unverifiable] file/docs/spec.pdf: no matching tool call and working dir not on disk
-PARTIAL  sess-disputed  Disputed session — 2 claim(s): 0 verified, 2 disputed, 0 unverifiable
+PARTIAL  sess-disputed  Disputed session — 2 claim(s): 0 verified, 2 disputed, 0 unverifiable — uncovered: tests/tests, commit/deadbee
   [disputed    ] tests/tests: no execute call matching `tests` recorded
   [disputed    ] commit/deadbee: no commit call recorded and no repo on disk to check
 PASS  sess-verified  Verified session — 4 claim(s): 4 verified, 0 disputed, 0 unverifiable
@@ -166,7 +170,10 @@ Recorded as an asciicast: [assets/demo.cast](assets/demo.cast) —
 The `report` subcommand audits all sessions (or one with `--session`,
 bounded with `--limit`) and writes a single deterministic HTML file:
 verdict counts, a per-session table and a per-claim breakdown with
-evidence and the source excerpt for every checked claim.
+evidence and the source excerpt for every checked claim. Verdicts and
+claim statuses render as labeled badges (color + text, never color
+alone) — the report stays self-contained: inline CSS, zero JS, no
+external assets.
 
 `--sessions-db` may be omitted. It auto-detects
 `%APPDATA%/devin/cli/sessions.db` on Windows and
@@ -257,6 +264,14 @@ when the session resolves. A one-line
 summary is also printed. `--limit N` bounds the number of claims
 verified.
 
+Each run also appends one compact JSON line to `verdicts.jsonl` next to
+the side file (`<data-dir>/qa/verdicts.jsonl` by default) — a
+machine-readable ledger with `session_id`, `verdict`, `audited_at`,
+claim counts, the `uncovered` claim names (disputed/unverifiable
+`kind/detail` list, omitted when empty) and the skip `reason`, for
+dashboards and triggers. The append is best-effort; the side file
+remains the contract.
+
 **Fail-soft:** `session-end` always exits `0` once it ran — the verdict
 travels in the side file, so a hook can never fail the host session. A
 session that cannot be resolved produces a `SKIPPED` verdict (still
@@ -284,6 +299,41 @@ stdin and exports `DEVIN_SESSION_ID`):
 }
 ```
 
+## Scheduled audit sweep (`install --daily`)
+
+The `SessionEnd` hook audits live; `devin-qa-pack install --daily` adds
+the complementary **scheduled sweep** — a daily `devin-qa-pack audit
+--all` over every detected surface (`--surface` narrows it) for
+sessions the hook missed (Devin closed early, hook not wired, foreign
+stores). It writes the full payload to
+`<data-dir>/qa/audit-sweep.json` and appends one compact verdict line
+per session (`origin: "sweep"`) to the same
+`<data-dir>/qa/verdicts.jsonl` ledger `session-end` feeds. Read-only on
+every store: the only writes are under the data dir's `qa/` folder, and
+the job does not need Devin closed.
+
+```bash
+devin-qa-pack install --daily                  # auto-detect backend
+devin-qa-pack install --daily --backend cron   # pin one backend
+devin-qa-pack install --daily --data-dir /custom/devin --surface devin-cli
+```
+
+Backend selection follows the shared `devin-install-scheduler`
+convention (same as devin-state's `devin-janitor`/`devin-backup`): a
+tagged `@daily` **crontab** line on Linux/macOS, a daily **Task
+Scheduler** entry on Windows, or — on locked-down machines without
+either — a job record in `<config-dir>/.devin-ecosystem/scheduled.json`
+(the `elapsed` backend) that devin-powerups' `UserPromptSubmit` hook
+ticks (`python tools/schedule.py check --run`). `--backend` pins a
+backend, `--config-dir` relocates the registry, `--data-dir` reshapes
+where the registered command writes, `--json` prints the job record.
+Both `--sessions-db` and `--out` in the registered command are baked to
+absolute paths at install time, so the job does not depend on
+`DEVIN_DATA_DIR`/XDG env surviving into the scheduler's minimal
+environment (re-run `install` if the data dir ever moves).
+The sweep exits `1` when any session is `PARTIAL`/`UNVERIFIED` — the
+verdicts land in the artifacts, the exit code is informational.
+
 ## Pre-commit gate
 
 The repo root ships `.pre-commit-hooks.yaml` — reference it from a
@@ -313,10 +363,15 @@ in the entry and the scope flags are mutually exclusive — run
 `audit --transcript` runs the same claim/evidence model over non-Devin
 transcripts via the adapters in `src/devin_qa_pack/adapters/`:
 
-- **`aider`** — `.aider.chat.history.md`. `> /run` and `> /test` records
-  become execute calls with `unknown` status (the history does not store
-  exit codes), so test claims resolve `unverifiable`; commit claims are
-  checked against `git log` in the working directory.
+- **`aider`** — `.aider.chat.history.md`. `#### /run`, `#### /test`,
+  `#### /git` and `#### !cmd` input lines become execute calls with
+  `unknown` status (the history does not store exit codes), so action
+  claims like "tests passed" resolve `unverifiable`; the `>`-blockquoted
+  tool output (`Applied edit to …`, `Commit <sha> …`, recorded test
+  output) is pooled as evidence, so value claims — pass counts, HTTP
+  statuses, coverage, eval scores — and file/commit details still
+  corroborate. Commit claims are also checked against `git log` in the
+  working directory.
 - **`claude-code`** — session `.jsonl` transcripts. `Bash` → execute,
   `Write`/`Edit`/`MultiEdit`/`NotebookEdit` → write, `tool_result`
   blocks mark calls completed/failed.
@@ -384,6 +439,14 @@ network is off by default and the audit is fully offline otherwise.
   output: a claim is `verified` when a recorded status matches,
   `disputed` when outputs record a different status and `unverifiable`
   when no output records any status. No request is ever replayed.
+- Counted test claims work the same way: "47 tests passed" is `verified`
+  when recorded output reports `47 passed`, `disputed` when it reports
+  a different count and `unverifiable` when no output records a count.
+  So do coverage and eval-score claims: "coverage is at 87%" needs a
+  recorded 87% (coverage.py's `TOTAL` row, a `coverage`/`line-rate`
+  payload field, or a "coverage …87%" output string); "the eval score
+  was 0.92" needs a recorded `0.92`. Values compare on the scale
+  written — an `85%` claim matches a recorded `85`, never `0.85`.
 - File/commit checks use `sessions.working_directory` only when it
   exists on disk and is a git repo — otherwise they rely on tool-call
   evidence alone.

{devin_qa_pack-0.2.0 → devin_qa_pack-0.3.0}/pyproject.toml

@@ -4,14 +4,15 @@ build-backend = "setuptools.build_meta"
 
 [project]
 name = "devin-qa-pack"
-version = "0.2.0"
-description = "QA audit for Devin sessions: verifies deliverable claims (tests, commits, files, pushes) against tool_call_state ground truth."
+version = "0.3.0"
+description = "QA audit for agent sessions: verifies deliverable claims (tests, commits, files, pushes) against recorded tool-call ground truth across Devin CLI, Copilot CLI, Codex CLI and VS Code chat stores."
 readme = "README.md"
 license = { text = "MIT" }
 requires-python = ">=3.10"
 authors = [{ name = "Icaro0310" }]
 dependencies = [
   "devin-internals-spec>=0.3.0,<0.4.0",
+  "devin-install-scheduler>=0.1.0",
 ]
 
 [project.optional-dependencies]

devin_qa_pack-0.3.0/src/devin_qa_pack/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.3.0"

devin_qa_pack-0.3.0/src/devin_qa_pack/adapters/aider.py

@@ -0,0 +1,138 @@
+"""Experimental Aider adapter — audits ``.aider.chat.history.md``.
+
+Aider records its chat history as markdown (``io.append_chat_history``):
+``####`` lines are user input — natural-language prompts *and* slash
+commands (``#### /run pytest -q``, ``#### !ls``, ``#### /git status``);
+``>`` lines are blockquoted tool/system output (``> Applied edit to
+x.py``, ``> Commit abc1234 ...``, errors, confirmations); everything
+else is assistant prose (the claim source).
+
+Evidence is deliberately weak where the format is weak: a recorded
+``/run``/``/test``/``!``/``/git`` line becomes an execute call with
+``unknown`` status — the history does not persist exit codes — so
+action claims like "tests passed" resolve ``unverifiable``, never a
+fake ``verified``. The ``>`` output lines still corroborate
+value-style claims: a recorded "47 passed" backs a ``tests_passed``
+claim, a ``> Commit a1b2c3d`` line backs a ``commit`` claim even
+without the repo on disk (``git log`` in the working directory stays
+the stronger check). Commit claims also verify through the repo's own
+``git log`` whenever the working directory is on disk, since aider
+auto-commits.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+from devin_qa_pack.adapters.base import SourceSession
+from devin_qa_pack.claims import claims_from_pairs
+from devin_qa_pack.verify import (
+    ParsedToolCall,
+    _coverage_values,
+    _eval_scores,
+    _http_statuses,
+    _passed_counts,
+)
+
+SOURCE = "aider"
+
+# Slash commands that actually run a shell command — ``/run``/``/test``
+# shell out and ``/git X`` runs ``git X``. Everything else (``/add``,
+# ``/model``, ``/commit``…) does not execute anything checkable; its
+# evidence, if any, lands on the ``>`` output lines instead.
+_RUN_SLASH = ("/run", "/test", "/git")
+
+
+def _command_of(user_input: str) -> str | None:
+    """Map a ``####`` input line to the shell command it ran, if any.
+
+    ``/run pytest``, ``/test`` and the ``!cmd`` shell shortcut are
+    commands; ``/git push`` records as ``git push``. A bare ``/test``
+    falls back to aider's configured ``test_cmd`` — unknown here — so
+    it records no command.
+    """
+    text = user_input.strip()
+    if text.startswith("!"):
+        command = text[1:].strip()
+        return command or None
+    for slash in _RUN_SLASH:
+        if text == slash or text.startswith(slash + " "):
+            rest = text[len(slash):].strip()
+            if not rest:
+                return None
+            return f"git {rest}" if slash == "/git" else rest
+    return None
+
+
+def _parse(text: str) -> tuple[list[tuple[str, str]], list[ParsedToolCall]]:
+    pairs: list[tuple[str, str]] = []
+    calls: list[ParsedToolCall] = []
+    output: list[str] = []  # ">" blockquote lines — recorded tool output
+    for raw in text.splitlines():
+        line = raw.rstrip()
+        if not line.strip():
+            continue
+        if line.startswith("####"):
+            user_input = line[4:].strip()
+            command = _command_of(user_input)
+            if command is not None:
+                calls.append(
+                    ParsedToolCall(
+                        tool_call_id=f"aider-cmd-{len(calls) + 1}",
+                        kind="execute",
+                        status="unknown",  # history records no exit code
+                        commands=(command,),
+                        search_text=command.lower().replace("\\", "/"),
+                        http_statuses=(),
+                        passed_counts=(),  # no output on the command line
+                    )
+                )
+            else:
+                pairs.append(("user", user_input))
+            continue
+        if line.startswith("#"):
+            continue  # "# aider chat started at …" and section headers
+        if line.startswith(">"):
+            output.append(line[1:].strip())
+            continue
+        pairs.append(("assistant", line))
+    if output:
+        # Aider's tool output never ties back to a specific command
+        # (no ids, no exit codes) — pool it into one evidence call so
+        # value claims (counts, statuses, percents, scores) and
+        # file/commit details check against what was actually recorded.
+        blob = "\n".join(output)
+        calls.append(
+            ParsedToolCall(
+                tool_call_id="aider-output",
+                kind="other",
+                status="completed",
+                commands=(),
+                search_text=blob.lower().replace("\\", "/"),
+                http_statuses=tuple(sorted(set(_http_statuses(blob)))),
+                passed_counts=tuple(sorted(set(_passed_counts(blob)))),
+                coverage_values=tuple(sorted(set(_coverage_values(blob)))),
+                eval_scores=tuple(sorted(set(_eval_scores(blob)))),
+            )
+        )
+    return pairs, calls
+
+
+def load(path: str | Path, working_directory: str | None = None) -> SourceSession:
+    """Load ``.aider.chat.history.md`` into the normalized audit view.
+
+    ``working_directory`` defaults to the directory containing the
+    history file — aider writes it at the repository root.
+    """
+    p = Path(path).expanduser()
+    pairs, calls = _parse(p.read_text(encoding="utf-8", errors="replace"))
+    session_id = f"aider:{p.name}"
+    return SourceSession(
+        source=SOURCE,
+        session_id=session_id,
+        title=p.name,
+        working_directory=working_directory or str(p.parent),
+        claims=tuple(claims_from_pairs(pairs, session_id)),
+        calls=tuple(calls),
+        raw_call_count=len(calls),
+    )

{devin_qa_pack-0.2.0 → devin_qa_pack-0.3.0}/src/devin_qa_pack/adapters/claude_code.py

@@ -21,7 +21,14 @@ from typing import Any
 
 from devin_qa_pack.adapters.base import SourceSession
 from devin_qa_pack.claims import claims_from_pairs
-from devin_qa_pack.verify import ParsedToolCall, _http_statuses, _strings
+from devin_qa_pack.verify import (
+    ParsedToolCall,
+    _coverage_values,
+    _eval_scores,
+    _http_statuses,
+    _passed_counts,
+    _strings,
+)
 
 SOURCE = "claude-code"
 
@@ -63,7 +70,16 @@ def _result_text(content: Any) -> str:
     return "" if content is None else str(content)
 
 
+# Claude Code's subagent-spawning tools — ``Task`` exactly, plus any
+# custom subagent-named tool — so delegation claims verify against the
+# transcript the same way they do against devin-cli evidence.
+_SPAWN_TOOLS = {"task"}
+
+
 def _tool_kind(name: str) -> str:
+    low = name.lower()
+    if low in _SPAWN_TOOLS or "subagent" in low or "sub_agent" in low:
+        return "spawn"
     if name in _EXECUTE_TOOLS:
         return "execute"
     if name in _WRITE_TOOLS:
@@ -117,6 +133,11 @@ def load(path: str | Path, working_directory: str | None = None) -> SourceSessio
                 http_statuses=tuple(sorted(
                     {*_http_statuses(inp), *_http_statuses(output)}
                 )),
+                # counts come from the tool_result, never the tool_use
+                # input (a Write's `content` is the request, not output)
+                passed_counts=tuple(sorted(set(_passed_counts(output)))),
+                coverage_values=tuple(sorted(set(_coverage_values(output)))),
+                eval_scores=tuple(sorted(set(_eval_scores(output)))),
             )
         )
 

{devin_qa_pack-0.2.0 → devin_qa_pack-0.3.0}/src/devin_qa_pack/adapters/mcp.py

@@ -45,7 +45,14 @@ from typing import Any
 
 from devin_qa_pack.adapters.base import SourceSession
 from devin_qa_pack.claims import claims_from_pairs
-from devin_qa_pack.verify import ParsedToolCall, _http_statuses, _strings
+from devin_qa_pack.verify import (
+    ParsedToolCall,
+    _coverage_values,
+    _eval_scores,
+    _http_statuses,
+    _passed_counts,
+    _strings,
+)
 
 SOURCE = "mcp"
 
@@ -404,10 +411,19 @@ def load(
         ]
         texts: list[str] = list(_strings(started)) + outputs
         statuses: set[int] = set(_http_statuses(started))
+        joined_output = "".join(outputs)
+        # Terminal chunks are one contiguous stdout stream — join before
+        # extracting, or a "47 passed" split across updates is missed.
+        counts: set[int] = set(_passed_counts(joined_output))
+        coverage: set[str] = set(_coverage_values(joined_output))
+        scores: set[str] = set(_eval_scores(joined_output))
         status = "unknown"
         if completed is not None:
             texts.extend(_strings(completed))
             statuses |= set(_http_statuses(completed))
+            counts |= set(_passed_counts(completed))
+            coverage |= set(_coverage_values(completed))
+            scores |= set(_eval_scores(completed))
             status = _status_for("shell_process_completed", completed)
         elif started:
             status = "pending"
@@ -422,6 +438,9 @@ def load(
             search_text=" ".join(t for t in texts if t).lower().replace(
                 "\\", "/"),
             http_statuses=tuple(sorted(statuses)),
+            passed_counts=tuple(sorted(counts)),
+            coverage_values=tuple(sorted(coverage)),
+            eval_scores=tuple(sorted(scores)),
         ))
 
     for item in standalone:
@@ -434,6 +453,9 @@ def load(
             commands=_commands(contents),
             search_text=" ".join(texts).lower().replace("\\", "/"),
             http_statuses=tuple(sorted(set(_http_statuses(contents)))),
+            passed_counts=tuple(sorted(set(_passed_counts(contents)))),
+            coverage_values=tuple(sorted(set(_coverage_values(contents)))),
+            eval_scores=tuple(sorted(set(_eval_scores(contents)))),
         ))
 
     return SourceSession(