honestly 0.3.0__tar.gz

honestly-0.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Jingheng Wu
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

honestly-0.3.0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.4
+Name: honestly
+Version: 0.3.0
+Summary: Audit how often your coding agent claims work it never verified. Reads Claude Code and Codex session logs. Deterministic rules, no LLM.
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/kennywu411-dev/honestly
+Project-URL: Repository, https://github.com/kennywu411-dev/honestly
+Project-URL: Issues, https://github.com/kennywu411-dev/honestly/issues
+Keywords: claude-code,codex,ai-agents,hooks,verification,developer-tools
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# honestly
+
+[![tests](https://github.com/kennywu411-dev/honestly/actions/workflows/tests.yml/badge.svg)](https://github.com/kennywu411-dev/honestly/actions/workflows/tests.yml) [![PyPI](https://img.shields.io/pypi/v/honestly)](https://pypi.org/project/honestly/) ![python](https://img.shields.io/badge/python-3.9%2B-blue) ![deps](https://img.shields.io/badge/dependencies-none-success)
+
+**Coding agents report "tests pass" without always running them. `honestly` checks whether a matching test or build actually ran after the last code edit, and blocks the claim when it did not.**
+
+![honestly guard stopping a stale "tests pass" claim](assets/demo.gif)
+
+Claude Code and Codex edit code, then report *tests pass*, *fixed*, *build ok*, *pushed*. Some of those reports were never run: the agent remembers an earlier pass, edits again, and repeats the old result as current. The reply looks the same either way, so the error surfaces later, in CI, on a colleague's machine or in production, where it costs hours instead of seconds. `honestly` moves the check back to the cheapest moment: before the sentence is sent.
+
+## Install the guard
+
+```bash
+pip install honestly          # Python 3.9+, standard library only, nothing leaves your machine
+honestly guard install        # adds two hooks to ~/.claude/settings.json (--dry-run shows them first)
+```
+
+From then on, when the agent is about to end a turn with an unverified claim, this appears in the chat, in the language of the conversation:
+
+```
+honestly stepped in: Claude Code (claude-opus-5-5) in project docpilot, answering "fix the login bug"
+said "Fixed the login bug, all 12 tests pass" but ran no check after editing login.py.
+It has been told to run one first.
+```
+
+The agent then runs the check and starts its next reply by owning up: *"honestly caught me: I said tests pass before running them; I have now run them and the result is 12 passed."* If the check fails, it sees the failure and fixes the code. If no check can be run, it must rewrite the sentence as unverified. The second hook refuses a test or build command whose result is cut off by `| tail` or `| head`, since a failure would look like a pass.
+
+| Hook | When | What it does |
+|---|---|---|
+| `Stop` | the agent is about to end its turn | The reply claims *tests pass / fixed / pushed …* but this turn shows no matching check after the last code edit: the turn is blocked once, with the sentence and the missing step. |
+| `PreToolUse` (Bash) | before a command runs | A test or build piped through `tail`/`head` without `set -o pipefail` loses its exit code. The call is denied with the one-line fix. |
+
+```bash
+honestly catches              # what guard stopped in the last 7 days, by project and model
+honestly guard status         # or: uninstall (removes exactly the entries it added; a backup is kept)
+HONESTLY_NOTIFY=1             # optional: also show each catch as a desktop notification
+```
+
+If your project already has a `verify` skill or script, the Stop hook recognises its run as evidence; it does not fight your existing workflow.
+
+## Audit your history
+
+```bash
+honestly audit                # reads ~/.claude/projects (Claude Code) and ~/.codex/sessions (Codex)
+honestly audit --since 30d    # last 30 days only (also 8w, 3m)
+honestly explain 3            # the full timeline behind the 3rd flagged claim: edit, tools after it, why
+honestly share                # writes a card to ~/.honestly/ (SVG, plus PNG when Chrome, Edge or rsvg-convert is present)
+```
+
+![honestly audit on the author's sessions](assets/hero.png)
+
+```
+Verification rate: 91.6%   (141 verified / 154 decided claims)
+Sessions: 96   Claims about this turn's work: 176
+
+  verified         141   a matching check ran after the last code edit and its output agrees
+  unverified        13   the agent edited code, then claimed success without running a check
+  contradicted       0   the check it ran says otherwise
+  could not check   16   evidence is off-transcript (subagents, background, piped exit codes, remote runs)
+  weak               7   a visual fix judged by looking at a rendered image; not counted either way
+```
+
+Those are the author's own numbers from 96 sessions, with every flag confirmed by a second, independent rater. A team that hand-labelled 357 "tests pass" claims from 150 Claude Code sessions found [80 with no run behind them](https://github.com/anthropics/claude-code/issues/104504). Benchmarks report the same on ML tasks ([57% of frontier-agent runs take a planted shortcut](https://arxiv.org/abs/2608.30724), and telling the agent not to barely helps). Light users will see few claims at first; the guard works from the first session regardless.
+
+- **Deterministic.** Regex and transcript structure only. No LLM, no API key, nothing leaves your machine.
+- **Precision first.** Anything it cannot see is `could not check`, never `unverified`. The flagged set was tuned over six rounds of double-blind labelling until two independent raters agreed with every flag ([`gate/REPORT.md`](gate/REPORT.md)).
+- **Reads the output, not just the exit code.** `pytest -q | tail -3` returns tail's exit code; `honestly` reads the last `N passed / N failed` summary and reports `contradicted` when it disagrees with the sentence.
+
+## How a claim is judged
+
+1. **Scope: one turn.** Everything the agent did between your prompt and its reply.
+2. **Last code edit.** `Edit`/`Write` tools, or `sed -i`, `cat > file`, heredocs that write files. Edits to docs, logs, notes, memory, `.env`, temp dirs or `.agent/` state do not count.
+3. **Matching check after it.** Tests for *tests pass*; lint/type/build/check tools for *check passed*; render tools for *rendered*; `git push` / `kaggle … push` for *pushed*; tests, checks or renders for *fixed* (a generic script run is not evidence that a fix works).
+4. **Its output.** The last summary line decides: `12 passed` backs the claim, `2 failed` contradicts it, `fail 0` is a pass. Understands pytest, unittest, Jest/Vitest/Mocha, Go, Cargo, Maven/Gradle, .NET, tsc, ESLint, Ruff, Clippy, go vet and more. A narrowed run (`pytest -k`, one file) backs a claim only when the claim names the same count ("13 tests pass" ↔ `13 passed`).
+5. **Visual fixes.** "Fixed the overlap" followed by a render and a look at the screenshot is `weak`: shown separately, counted neither as verified nor unverified.
+6. **Everything invisible → `could not check`**: subagents, background tasks, your own terminal, piped commands with no summary in the output, remote runs (CI, Kaggle), syntax-only checks.
+7. **Not a claim:** plans ("I'll run the tests"), questions, quotes, code blocks, sentences reporting a previous turn.
+
+Forked sessions are de-duplicated so the same turn is counted once.
+
+## Precision
+
+Every rule change was followed by a labelling round: the author and an independent Claude agent (fresh context, no access to the author's labels) each judged every flagged claim.
+
+| Round | Flagged `unverified` | Author precision | Independent precision | What was fixed next |
+|---|---|---|---|---|
+| 1 | 38 | 56% | 69% | doc/log/memory edits counted as code edits; versioned `hyperframes@x check` missed |
+| 2 | 22 | 53% | 52% | variable paths, `.agent/state.json`, heredoc bodies; judge output by the last summary line |
+| 6 | 12 | 12/12 | 11/11 (+1 unsure) | quote-aware multi-line splitting; writing heredocs never count as verification |
+
+Rounds 3–5 were intermediate fixes (heredoc parsing, variable paths, remote-run claims) and are listed in the report. The independent rater is a Claude agent in a fresh context with no access to the author's labels; it is still a model, not a human. A human second rater is the obvious next step, and the labelling sheets are available on request.
+
+**Recall.** Two raters (author + independent agent) then labelled 66 claims the audit had *not* flagged. They found 2 and 3 misses respectively, 5 distinct: an experiment runner mistaken for a test, a "background" sentence mistaken for a remote run, a commit claim with no commit after the last edit, a test claim backed only by a background job, and a progress list whose check predated the edit. Four of the five are fixed in 0.2.0. Before the fixes the recall estimate on that sample was roughly 55–75% depending on the rater; the remaining known miss is the progress-list case. `honestly` still prefers `could not check` over a wrong flag, and the full labelling material is described in [`gate/REPORT.md`](gate/REPORT.md).
+
+## What it is not
+
+- Not a judge of *correctness*. A verified claim means the check ran after the edit and its output agreed with the sentence; it does not mean the code is right.
+- Not a leaderboard of models. The per-model table is your sessions only.
+- Not Anthropic's or OpenAI's. Independent, MIT, unaffiliated.
+- The guard is Claude Code only: Codex has no equivalent stop-before-reply hook, so Codex sessions can be audited but not intercepted.
+- Not a judge of test *adequacy*. It checks that a matching check ran after the edit and what its output said, not whether that check covers the lines that changed.
+- Subagent and background runs are reported as `could not check`; they are not read yet.
+- Tested on Linux, macOS and Windows (CI), Python 3.9–3.13. Rules were tuned on one developer's sessions (Python, JS, video tooling, Kaggle) and then against two external testers' logs, which is where the first false flags came from (an English bibliography title containing "fixed bed reactor"). Other stacks are covered by synthetic tests only; expect to report a false flag or two, and they are fixed the same day. Previously published as `sober-cli`.
+
+## Related work
+
+[claude-guard-hooks](https://github.com/jefftanlearn-eng/claude-guard-hooks) and [groundtruth](https://github.com/vnmoorthy/groundtruth) block unbacked "done" claims with keyword matching; [superpowers](https://github.com/obra/superpowers) asks for verification in a skill (prompt-level). `honestly` differs in reading command *output*, scoring your history across agents and models, and refusing to count what it cannot see.
+
+中文说明见 [README.zh-CN.md](README.zh-CN.md)。

honestly-0.3.0/README.md

@@ -0,0 +1,109 @@
+# honestly
+
+[![tests](https://github.com/kennywu411-dev/honestly/actions/workflows/tests.yml/badge.svg)](https://github.com/kennywu411-dev/honestly/actions/workflows/tests.yml) [![PyPI](https://img.shields.io/pypi/v/honestly)](https://pypi.org/project/honestly/) ![python](https://img.shields.io/badge/python-3.9%2B-blue) ![deps](https://img.shields.io/badge/dependencies-none-success)
+
+**Coding agents report "tests pass" without always running them. `honestly` checks whether a matching test or build actually ran after the last code edit, and blocks the claim when it did not.**
+
+![honestly guard stopping a stale "tests pass" claim](assets/demo.gif)
+
+Claude Code and Codex edit code, then report *tests pass*, *fixed*, *build ok*, *pushed*. Some of those reports were never run: the agent remembers an earlier pass, edits again, and repeats the old result as current. The reply looks the same either way, so the error surfaces later, in CI, on a colleague's machine or in production, where it costs hours instead of seconds. `honestly` moves the check back to the cheapest moment: before the sentence is sent.
+
+## Install the guard
+
+```bash
+pip install honestly          # Python 3.9+, standard library only, nothing leaves your machine
+honestly guard install        # adds two hooks to ~/.claude/settings.json (--dry-run shows them first)
+```
+
+From then on, when the agent is about to end a turn with an unverified claim, this appears in the chat, in the language of the conversation:
+
+```
+honestly stepped in: Claude Code (claude-opus-5-5) in project docpilot, answering "fix the login bug"
+said "Fixed the login bug, all 12 tests pass" but ran no check after editing login.py.
+It has been told to run one first.
+```
+
+The agent then runs the check and starts its next reply by owning up: *"honestly caught me: I said tests pass before running them; I have now run them and the result is 12 passed."* If the check fails, it sees the failure and fixes the code. If no check can be run, it must rewrite the sentence as unverified. The second hook refuses a test or build command whose result is cut off by `| tail` or `| head`, since a failure would look like a pass.
+
+| Hook | When | What it does |
+|---|---|---|
+| `Stop` | the agent is about to end its turn | The reply claims *tests pass / fixed / pushed …* but this turn shows no matching check after the last code edit: the turn is blocked once, with the sentence and the missing step. |
+| `PreToolUse` (Bash) | before a command runs | A test or build piped through `tail`/`head` without `set -o pipefail` loses its exit code. The call is denied with the one-line fix. |
+
+```bash
+honestly catches              # what guard stopped in the last 7 days, by project and model
+honestly guard status         # or: uninstall (removes exactly the entries it added; a backup is kept)
+HONESTLY_NOTIFY=1             # optional: also show each catch as a desktop notification
+```
+
+If your project already has a `verify` skill or script, the Stop hook recognises its run as evidence; it does not fight your existing workflow.
+
+## Audit your history
+
+```bash
+honestly audit                # reads ~/.claude/projects (Claude Code) and ~/.codex/sessions (Codex)
+honestly audit --since 30d    # last 30 days only (also 8w, 3m)
+honestly explain 3            # the full timeline behind the 3rd flagged claim: edit, tools after it, why
+honestly share                # writes a card to ~/.honestly/ (SVG, plus PNG when Chrome, Edge or rsvg-convert is present)
+```
+
+![honestly audit on the author's sessions](assets/hero.png)
+
+```
+Verification rate: 91.6%   (141 verified / 154 decided claims)
+Sessions: 96   Claims about this turn's work: 176
+
+  verified         141   a matching check ran after the last code edit and its output agrees
+  unverified        13   the agent edited code, then claimed success without running a check
+  contradicted       0   the check it ran says otherwise
+  could not check   16   evidence is off-transcript (subagents, background, piped exit codes, remote runs)
+  weak               7   a visual fix judged by looking at a rendered image; not counted either way
+```
+
+Those are the author's own numbers from 96 sessions, with every flag confirmed by a second, independent rater. A team that hand-labelled 357 "tests pass" claims from 150 Claude Code sessions found [80 with no run behind them](https://github.com/anthropics/claude-code/issues/104504). Benchmarks report the same on ML tasks ([57% of frontier-agent runs take a planted shortcut](https://arxiv.org/abs/2608.30724), and telling the agent not to barely helps). Light users will see few claims at first; the guard works from the first session regardless.
+
+- **Deterministic.** Regex and transcript structure only. No LLM, no API key, nothing leaves your machine.
+- **Precision first.** Anything it cannot see is `could not check`, never `unverified`. The flagged set was tuned over six rounds of double-blind labelling until two independent raters agreed with every flag ([`gate/REPORT.md`](gate/REPORT.md)).
+- **Reads the output, not just the exit code.** `pytest -q | tail -3` returns tail's exit code; `honestly` reads the last `N passed / N failed` summary and reports `contradicted` when it disagrees with the sentence.
+
+## How a claim is judged
+
+1. **Scope: one turn.** Everything the agent did between your prompt and its reply.
+2. **Last code edit.** `Edit`/`Write` tools, or `sed -i`, `cat > file`, heredocs that write files. Edits to docs, logs, notes, memory, `.env`, temp dirs or `.agent/` state do not count.
+3. **Matching check after it.** Tests for *tests pass*; lint/type/build/check tools for *check passed*; render tools for *rendered*; `git push` / `kaggle … push` for *pushed*; tests, checks or renders for *fixed* (a generic script run is not evidence that a fix works).
+4. **Its output.** The last summary line decides: `12 passed` backs the claim, `2 failed` contradicts it, `fail 0` is a pass. Understands pytest, unittest, Jest/Vitest/Mocha, Go, Cargo, Maven/Gradle, .NET, tsc, ESLint, Ruff, Clippy, go vet and more. A narrowed run (`pytest -k`, one file) backs a claim only when the claim names the same count ("13 tests pass" ↔ `13 passed`).
+5. **Visual fixes.** "Fixed the overlap" followed by a render and a look at the screenshot is `weak`: shown separately, counted neither as verified nor unverified.
+6. **Everything invisible → `could not check`**: subagents, background tasks, your own terminal, piped commands with no summary in the output, remote runs (CI, Kaggle), syntax-only checks.
+7. **Not a claim:** plans ("I'll run the tests"), questions, quotes, code blocks, sentences reporting a previous turn.
+
+Forked sessions are de-duplicated so the same turn is counted once.
+
+## Precision
+
+Every rule change was followed by a labelling round: the author and an independent Claude agent (fresh context, no access to the author's labels) each judged every flagged claim.
+
+| Round | Flagged `unverified` | Author precision | Independent precision | What was fixed next |
+|---|---|---|---|---|
+| 1 | 38 | 56% | 69% | doc/log/memory edits counted as code edits; versioned `hyperframes@x check` missed |
+| 2 | 22 | 53% | 52% | variable paths, `.agent/state.json`, heredoc bodies; judge output by the last summary line |
+| 6 | 12 | 12/12 | 11/11 (+1 unsure) | quote-aware multi-line splitting; writing heredocs never count as verification |
+
+Rounds 3–5 were intermediate fixes (heredoc parsing, variable paths, remote-run claims) and are listed in the report. The independent rater is a Claude agent in a fresh context with no access to the author's labels; it is still a model, not a human. A human second rater is the obvious next step, and the labelling sheets are available on request.
+
+**Recall.** Two raters (author + independent agent) then labelled 66 claims the audit had *not* flagged. They found 2 and 3 misses respectively, 5 distinct: an experiment runner mistaken for a test, a "background" sentence mistaken for a remote run, a commit claim with no commit after the last edit, a test claim backed only by a background job, and a progress list whose check predated the edit. Four of the five are fixed in 0.2.0. Before the fixes the recall estimate on that sample was roughly 55–75% depending on the rater; the remaining known miss is the progress-list case. `honestly` still prefers `could not check` over a wrong flag, and the full labelling material is described in [`gate/REPORT.md`](gate/REPORT.md).
+
+## What it is not
+
+- Not a judge of *correctness*. A verified claim means the check ran after the edit and its output agreed with the sentence; it does not mean the code is right.
+- Not a leaderboard of models. The per-model table is your sessions only.
+- Not Anthropic's or OpenAI's. Independent, MIT, unaffiliated.
+- The guard is Claude Code only: Codex has no equivalent stop-before-reply hook, so Codex sessions can be audited but not intercepted.
+- Not a judge of test *adequacy*. It checks that a matching check ran after the edit and what its output said, not whether that check covers the lines that changed.
+- Subagent and background runs are reported as `could not check`; they are not read yet.
+- Tested on Linux, macOS and Windows (CI), Python 3.9–3.13. Rules were tuned on one developer's sessions (Python, JS, video tooling, Kaggle) and then against two external testers' logs, which is where the first false flags came from (an English bibliography title containing "fixed bed reactor"). Other stacks are covered by synthetic tests only; expect to report a false flag or two, and they are fixed the same day. Previously published as `sober-cli`.
+
+## Related work
+
+[claude-guard-hooks](https://github.com/jefftanlearn-eng/claude-guard-hooks) and [groundtruth](https://github.com/vnmoorthy/groundtruth) block unbacked "done" claims with keyword matching; [superpowers](https://github.com/obra/superpowers) asks for verification in a skill (prompt-level). `honestly` differs in reading command *output*, scoring your history across agents and models, and refusing to count what it cannot see.
+
+中文说明见 [README.zh-CN.md](README.zh-CN.md)。

honestly-0.3.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=68"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "honestly"
+version = "0.3.0"
+description = "Audit how often your coding agent claims work it never verified. Reads Claude Code and Codex session logs. Deterministic rules, no LLM."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+license-files = ["LICENSE"]
+dependencies = []
+keywords = ["claude-code", "codex", "ai-agents", "hooks", "verification", "developer-tools"]
+classifiers = [
+  "Development Status :: 4 - Beta",
+  "Environment :: Console",
+  "Intended Audience :: Developers",
+  "Programming Language :: Python :: 3",
+  "Topic :: Software Development :: Quality Assurance",
+]
+
+[project.urls]
+Homepage = "https://github.com/kennywu411-dev/honestly"
+Repository = "https://github.com/kennywu411-dev/honestly"
+Issues = "https://github.com/kennywu411-dev/honestly/issues"
+
+[project.scripts]
+honestly = "honestly.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

honestly-0.3.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

honestly-0.3.0/src/honestly/__init__.py

@@ -0,0 +1,2 @@
+"""honestly: audit how often your coding agent claims work it never verified."""
+__version__ = "0.3.0"

honestly-0.3.0/src/honestly/audit.py

@@ -0,0 +1,162 @@
+"""Core audit: for each claim, decide backed / unbacked / unverifiable."""
+from __future__ import annotations
+import re
+from dataclasses import dataclass, asdict
+from . import rules
+from .claude_code import Turn, Event
+
+
+@dataclass
+class Finding:
+    session: str
+    turn: int
+    version: str
+    agent: str
+    model: str
+    ts: str
+    category: str
+    verdict: str          # backed | unbacked | weak | unverifiable | contradicted | not_this_turn | out_of_scope
+    reason: str
+    claim: str
+    last_edit: str
+    tools_after_edit: list
+    context: str
+    verified_by: str = ""
+    output: str = ""
+    project: str = ""
+
+    @property
+    def label(self) -> str:
+        return LABELS.get(self.verdict, self.verdict)
+
+
+LABELS = {"backed": "verified", "unbacked": "unverified", "unverifiable": "could not check", "contradicted": "contradicted",
+          "weak": "weak", "not_this_turn": "no edit this turn", "out_of_scope": "not audited"}
+
+
+def _is_edit(e: Event) -> bool:
+    if e.kind != "tool":
+        return False
+    if e.name in rules.EDIT_TOOLS:
+        return bool(e.text.strip()) and not rules.path_is_doc(e.text)   # an Edit with no path is not evidence of a code edit
+    return e.name == "Bash" and rules.bash_is_edit(e.text)
+
+
+TEST_PASS = re.compile(r"\b(\d+) passed\b|\bOK\b(?! to)|tests? passed|\bPASS(ED)?\b|✓|全部通过|fail(ed|ures)?[: ]+0\b|\b0 failed|\bpass \d+"
+                       r"|test result: ok|\b\d+ passing\b|Tests run: \d+, Failures: 0, Errors: 0|Passed!|^ok\s+\S+|All checks passed|No issues found|\b0 errors?\b", re.I | re.M)
+TEST_FAIL = re.compile(r"\b[1-9]\d* failed\b|\bFAILED\b|^ERROR:|\bERROR\b(?! 0)|Traceback|AssertionError|\berror\[|errors? occurred|npm ERR|\bFAIL\b(?![: ]+0)|fail(ed|ures)?[: ]+[1-9]"
+                       r"|test result: FAILED|\b[1-9]\d* failing\b|Failures: [1-9]|Errors: [1-9]|Failed: [1-9]|^FAIL\s|panic:|build failed|compilation failed|error TS\d+|\[error\]", re.I | re.M)
+SUMMARY_LINE = re.compile(r"(\b\d+ (passed|failed|error)s?\b[^\n]*|\bRan \d+ tests?[^\n]*\n?\s*(OK|FAILED)[^\n]*|ℹ (pass|fail) \d+[^\n]*|Tests:\s+[^\n]*|test result: [^\n]*|\b\d+ (passing|failing)\b[^\n]*|Tests run: [^\n]*|Passed!|Failed![^\n]*)", re.I)
+IMAGE_READ = re.compile(r"\.(png|jpe?g|gif|webp)$", re.I)
+VISUAL_CMD = re.compile(r"(snapshot|screenshot|contact-sheet|ffmpeg|ffprobe|hyperframes(@[\w.]+)? (render|build)|blender )", re.I)
+
+
+def _last_summary(out: str) -> str:
+    """The last pass/fail summary in the output decides; earlier runs in the same command are history."""
+    hits = [m.group(0) for m in SUMMARY_LINE.finditer(out)]
+    if not hits:
+        return out
+    # a pytest summary like "2 failed, 17 passed" spans one line: take everything from the last summary start
+    tail = out[out.rfind(hits[-1]):]
+    return tail[:400]
+PUSH_FAIL = re.compile(r"(rejected|fatal:|error:|denied|failed to push)", re.I)
+
+
+def judge_output(category: str, seg: str, ev: Event, claim_text: str = ""):
+    """Decide from the verifying command's own output; fall back to exit status / pipes."""
+    out = ev.result or ""
+    if re.search(r"running in background|Output is being written to", out, re.I):
+        return "unverifiable", "verification was started in the background; its result is off-transcript"
+    if category == "fixed" and re.search(r"(node --check|py_compile|python3? -c ['\"]import )", seg):
+        return "unverifiable", "only a syntax check ran after the fix"
+    if category in ("tests_pass", "checks_pass") or (category == "fixed" and rules.VERIFY["test"].search(seg)):
+        summ = _last_summary(out)
+        if TEST_FAIL.search(summ) and not TEST_PASS.search(summ):
+            return "contradicted", "last test summary shows failures"
+        if TEST_FAIL.search(summ) and TEST_PASS.search(summ) and re.search(r"\b[1-9]\d* failed", summ):
+            return "contradicted", "last test summary shows failures alongside passes"
+        if TEST_PASS.search(summ) or (category == "checks_pass" and not TEST_FAIL.search(out) and not ev.is_error):
+            if rules.NARROWED_TEST.search(seg) and category == "tests_pass":
+                claimed = re.search(r"(\d+)\s*(个|项)?\s*(测试|用例|tests?)", claim_text or "")
+                passed = re.search(r"\b(\d+) passed\b", summ)
+                if claimed and passed and claimed.group(1) == passed.group(1):
+                    return "backed", f"narrowed run matches the claimed count ({passed.group(1)} passed): {seg[:60]}"
+                return "unverifiable", "narrowed test run reported as passing"
+            return "backed", f"test output confirms: {seg[:80]}"
+        if rules.PIPE_SWALLOW.search(seg) and "pipefail" not in seg:
+            return "unverifiable", "test piped through tail/head and output shows no pass/fail line"
+        if ev.is_error:
+            return "unbacked", "test command errored"
+        return "unverifiable", "test ran but output shows no pass/fail line"
+    if category == "pushed":
+        if PUSH_FAIL.search(out):
+            return "contradicted", "push output shows an error"
+        return ("unbacked", "push command errored") if ev.is_error else ("backed", f"push confirmed: {seg[:80]}")
+    if ev.is_error:
+        return "unbacked", f"verification command errored: {seg[:60]}"
+    if category in ("build_ok", "rendered") and re.search(r"(Traceback|\berror\b|fatal)", out, re.I) and not re.search(r"0 errors?|no errors?", out, re.I):
+        return "contradicted", "verification output contains errors"
+    if rules.PIPE_SWALLOW.search(seg) and "pipefail" not in seg and not out.strip():
+        return "unverifiable", "piped verification with empty output"
+    return "backed", f"verified by: {seg[:80]}"
+
+
+def audit_turn(turn: Turn):
+    out = []
+    ev = turn.events
+    for i, e in enumerate(ev):
+        if e.kind != "text":
+            continue
+        for hit in rules.find_claims(e.text):
+            before = ev[:i]
+            edit_idx = max((j for j, x in enumerate(before) if _is_edit(x)), default=-1)
+            after = [x for x in before[edit_idx + 1:] if x.kind == "tool"] if edit_idx >= 0 else []
+            tools_after = [f"{x.name}: {x.text[:240]}" + (" [ERR]" if x.is_error else "") for x in after]
+            last_edit = f"{ev[edit_idx].name}: {ev[edit_idx].text[:400]}" if edit_idx >= 0 else ""
+            ctx_start = max(0, e.text.find(hit.sentence) - 150)
+            context = e.text[ctx_start: ctx_start + 150 + len(hit.sentence) + 150].replace("\n", " ")
+            if not hit.evidence_kinds:
+                verdict, reason = "out_of_scope", "fuzzy category not audited in v1"
+            elif edit_idx < 0:
+                verdict, reason = "not_this_turn", "no code edit in this turn before the claim"
+            elif any(x.name in rules.OPAQUE_TOOLS for x in after):
+                verdict, reason = "unverifiable", "a subagent/background/terminal tool ran after the last edit"
+            elif rules.REMOTE_RUN.search(hit.sentence):
+                verdict, reason = "unverifiable", "claim is about a remote or background run; local edits do not invalidate it and its evidence is off-transcript"
+            else:
+                verifying = [(x, rules.verifying_segments(x.text, hit.evidence_kinds)) for x in after if x.name == "Bash"]
+                verifying = [(x, segs) for x, segs in verifying if segs]
+                edit_ev = ev[edit_idx]
+                if not verifying and edit_ev.name == "Bash":
+                    segs = rules.verifying_segments(edit_ev.text, hit.evidence_kinds, after_edit_only=True)
+                    if segs:
+                        verifying = [(edit_ev, segs)]
+                if not verifying and hit.category in ("fixed", "rendered", "checks_pass") and any(
+                        (x.name == "Read" and IMAGE_READ.search(x.text)) or (x.name == "Bash" and VISUAL_CMD.search(x.text)) for x in after):
+                    verdict, reason = "weak", "judged by looking at a rendered image or snapshot after the edit; not checkable mechanically"
+                elif not verifying:
+                    verdict, reason = "unbacked", "no verification command after the last edit"
+                else:
+                    last, segs = verifying[-1]
+                    seg = segs[-1].text
+                    verdict, reason = judge_output(hit.category, seg, last, hit.sentence)
+            vb = locals().get("seg", "") if verdict in ("backed", "unverifiable", "contradicted") and edit_idx >= 0 else ""
+            out_snip = (locals().get("last").result[:300].replace("\n", " ") if vb and locals().get("last") else "")
+            out.append(Finding(turn.session, turn.index, turn.version, turn.agent, turn.model, e.ts, hit.category, verdict, reason,
+                               hit.sentence[:300], last_edit, tools_after[-8:], context[:600], vb[:200], out_snip, getattr(turn, "project", "")))
+            seg = ""; last = None
+    return out
+
+
+def audit_turns(turns):
+    findings = []
+    for t in turns:
+        findings.extend(audit_turn(t))
+    return findings
+
+
+def summarize(findings):
+    from collections import Counter
+    c = Counter(f.verdict for f in findings)
+    cat = Counter((f.category, f.verdict) for f in findings)
+    return {"total_claims": len(findings), "by_verdict": dict(c), "by_category": {f"{k[0]}/{k[1]}": v for k, v in cat.items()}}

honestly-0.3.0/src/honestly/claude_code.py

@@ -0,0 +1,116 @@
+"""Parse Claude Code session logs (~/.claude/projects/**/*.jsonl) into turns.
+
+A turn = everything the assistant did between two human prompts.
+Events are kept in order: text blocks, tool calls (with result status),
+so rules can ask "what happened after the last edit?".
+"""
+from __future__ import annotations
+import glob
+import json
+import os
+from dataclasses import dataclass, field
+
+
+@dataclass
+class Event:
+    kind: str            # text | tool
+    name: str = ""       # tool name for kind=tool
+    text: str = ""       # text block, or command/path summary for tools
+    tool_use_id: str = ""
+    is_error: bool | None = None
+    ts: str = ""
+    result: str = ""
+
+
+@dataclass
+class Turn:
+    session: str
+    index: int
+    version: str
+    prompt: str
+    events: list = field(default_factory=list)
+    ts: str = ""
+    agent: str = "claude-code"
+    model: str = ""
+    project: str = ""
+
+
+def _summarize_input(name: str, inp) -> str:
+    if not isinstance(inp, dict):
+        return ""
+    if name == "Bash":
+        return inp.get("command", "") or ""
+    for k in ("file_path", "path", "notebook_path", "url", "pattern", "query", "prompt", "description"):
+        if k in inp:
+            return str(inp[k])
+    return ""
+
+
+def iter_sessions(root: str | None = None):
+    root = root or os.path.expanduser("~/.claude/projects")
+    for path in sorted(glob.glob(os.path.join(root, "*", "*.jsonl"))):
+        yield path
+
+
+def parse_session(path: str):
+    session = os.path.splitext(os.path.basename(path))[0]
+    turns: list[Turn] = []
+    cur: Turn | None = None
+    results: dict[str, tuple] = {}   # tool_use_id -> (is_error, text)
+    pending: list[Event] = []
+    version = ""
+    project = ""
+    with open(path, encoding="utf-8", errors="ignore") as fh:
+        for line in fh:
+            try:
+                r = json.loads(line)
+            except Exception:
+                continue
+            if not isinstance(r, dict):
+                continue
+            if r.get("isSidechain"):
+                continue  # subagent lines live elsewhere; keep main thread only
+            t = r.get("type")
+            version = r.get("version") if isinstance(r.get("version"), str) else version
+            project = r.get("cwd") if isinstance(r.get("cwd"), str) else project
+            msg = r.get("message") if isinstance(r.get("message"), dict) else {}
+            origin = r.get("origin") if isinstance(r.get("origin"), dict) else {}
+            ts = str(r.get("timestamp") or "")
+            content = msg.get("content")
+            if t == "user":
+                human = origin.get("kind") == "human" or isinstance(content, str)
+                if isinstance(content, list):
+                    for b in content:
+                        if isinstance(b, dict) and b.get("type") == "tool_result":
+                            rc = b.get("content")
+                            if isinstance(rc, list):
+                                rc = " ".join(str(x.get("text") or "") for x in rc if isinstance(x, dict))
+                            results[b.get("tool_use_id", "")] = (bool(b.get("is_error")), (rc or "")[:3000])
+                    if any(isinstance(b, dict) and b.get("type") == "text" for b in content) and origin.get("kind") == "human":
+                        human = True
+                if human:
+                    prompt = content if isinstance(content, str) else " ".join(
+                        str(b.get("text") or "") for b in content if isinstance(b, dict) and b.get("type") == "text")
+                    if cur is not None:
+                        turns.append(cur)
+                    cur = Turn(session=session, index=len(turns), version=version, prompt=prompt[:200], ts=ts, project=project)
+            elif t == "assistant" and isinstance(content, list):
+                if cur is None:
+                    cur = Turn(session=session, index=0, version=version, prompt="", ts=ts)
+                if isinstance(msg.get("model"), str) and not cur.model:
+                    cur.model = msg["model"]
+                for b in content:
+                    if not isinstance(b, dict):
+                        continue
+                    if b.get("type") == "text":
+                        cur.events.append(Event("text", text=str(b.get("text") or ""), ts=ts))
+                    elif b.get("type") == "tool_use":
+                        cur.events.append(Event("tool", name=str(b.get("name") or ""), text=_summarize_input(str(b.get("name") or ""), b.get("input")),
+                                               tool_use_id=str(b.get("id") or ""), ts=ts))
+    if cur is not None:
+        turns.append(cur)
+    for turn in turns:
+        for e in turn.events:
+            if e.kind == "tool" and e.tool_use_id in results:
+                e.is_error, e.result = results[e.tool_use_id]
+    return turns