agent-polygraph 0.2.0 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -4
- package/bin/polygraph.js +82 -0
- package/package.json +1 -1
- package/src/analyze.js +13 -2
- package/src/ci.js +36 -12
- package/src/doctor.js +94 -0
- package/src/explain.js +49 -0
package/README.md
CHANGED
|
@@ -47,7 +47,21 @@ Under the tables it prints the latest examples: what the agent claimed, the last
|
|
|
47
47
|
|
|
48
48
|
**What the benchmarks actually show:** agent-polygraph detects stale test claims more reliably than a *simple any-prior-pass rule* in [25 authored replay scenarios](docs/benchmark.md) (14/14 unbacked claims versus 7/14). In [12 instrumented projects using real test executions](docs/live-evaluation.md), its selective guard requested **5** additional tests and detected **3 of 7** failing final suites; rerunning the full suite every time required **12** additional tests and detected **all 7**. That is an efficiency-versus-coverage trade-off, **not** evidence that the guard improves overall defect detection. [Public-session workload projections](docs/benchmark.md) are separate from actual executed tests. The independently labeled human study is [still open](https://github.com/EleChon-pixel/agent-polygraph/issues/3).
|
|
49
49
|
|
|
50
|
-
**npm release
|
|
50
|
+
**npm distribution:** Install the most recently *published* release with `npx --yes agent-polygraph --demo`, or pin an exact version (for example, `npx --yes agent-polygraph@0.2.0 --demo`). Check the actual public version with `npm view agent-polygraph version`: the GitHub `main` branch may be ahead of npm. Releases use GitHub OIDC trusted publishing and require maintainer 2FA approval before going live. The [automated public-install check](.github/workflows/verify-published-npm.yml) tests the download from npm, not just the repository source. See [Release instructions](docs/releasing.md).
|
|
51
|
+
|
|
52
|
+
## Explain a specific finding and check your setup
|
|
53
|
+
|
|
54
|
+
**Release note:** These commands are on GitHub `main` for the upcoming `0.3.0` release. The currently published npm `0.2.1` package does not include them. Until approval of `0.3.0`, use the repository's CLI as `node bin/polygraph.js explain` or `node bin/polygraph.js doctor`.
|
|
55
|
+
|
|
56
|
+
```sh
|
|
57
|
+
agent-polygraph explain --since 7d --limit 10 # show flagged or uncertain claims and their evidence
|
|
58
|
+
agent-polygraph explain --session abc123 # narrow by session ID prefix
|
|
59
|
+
agent-polygraph doctor # check installed hooks, logs and recorder settings
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
`explain` shows the claim, available last-test evidence, relevant subsequent edit and why a result is inconclusive. It is **read-only**, does not run tests, and may print private transcript excerpts, commands and paths. **Do not share its output without reviewing/redacting it.** Use `explain --ci` only if you want to ask GitHub for additional CI evidence.
|
|
63
|
+
|
|
64
|
+
`doctor` inspects local transcript discovery, a small sample of log formats, project/global hook settings, and recorder-directory permissions. It **never modifies settings or installs hooks**. An uninstalled direct hook is informational: the guard is optional and the Claude Code plugin may supply one separately. The default is offline; `doctor --ci` explicitly checks whether your GitHub CLI login is available, which may contact GitHub. This is a diagnostic, not a complete security audit.
|
|
51
65
|
|
|
52
66
|
## Share your numbers
|
|
53
67
|
|
|
@@ -173,7 +187,7 @@ The replay labels and scenarios were constructed by the project authors. The gua
|
|
|
173
187
|
These are the gaps that remain. `record` and `--ci` can reduce some of them, but neither replaces running tests or provides independent proof of correctness.
|
|
174
188
|
|
|
175
189
|
- Tests outside the logs. Tests the agent ran, test runs Claude Code recorded from hooks, commands you typed with `!` in Claude Code, and test output you pasted into the chat all count. So do your own bash and zsh runs with `record --shell`, and GitHub Actions runs on pushed commits with `--ci`. Other shells (fish, PowerShell, an IDE's test button), other CI services, and CI runs on commits the session never printed are still invisible.
|
|
176
|
-
- "The full suite passes"
|
|
190
|
+
- "The full suite passes" prefers an explicitly recognized full-suite command in the same folder, even when no count was printed. If the only observed run is targeted and there is no full-suite evidence, the result is **inconclusive**, not backed. Count-based heuristics still help when the runner cannot reliably identify the suite's scope. A passing test run is evidence of that run, never proof that all code works.
|
|
177
191
|
- Output with no summary. Codex keeps each command's full output. Claude Code may save long output to a separate file that the analyzer reads if it still exists. New `record` entries retain **parsed test outcomes and pass counts, not raw test output**. A run abandoned before printing a result leaves nothing to read.
|
|
178
192
|
- Files a script names at run time. Without `record` these make a claim "couldn't tell", never "backed". `record` settles them, but only in git checkouts and only for sessions after it was installed.
|
|
179
193
|
- New agent versions can change the log format. Supported: Claude Code 2.x (Bash and PowerShell tools; Linux, macOS and Windows paths) and Codex CLI. The report lists the versions it read, warns about newer ones, and counts files it couldn't parse instead of dropping them silently.
|
|
@@ -191,6 +205,8 @@ These are the gaps that remain. `record` and `--ci` can reduce some of them, but
|
|
|
191
205
|
--examples <n> examples to show per finding type (default 3, 0 to hide)
|
|
192
206
|
--html [file] write a numbers-only report to share (default polygraph.html)
|
|
193
207
|
--json full results as JSON: findings, unclear reasons, unreadable files (keep it local)
|
|
208
|
+
explain investigate flagged/uncertain verdicts (--limit, --since, --session, --project)
|
|
209
|
+
doctor check local hooks, transcript availability and recorder permissions
|
|
194
210
|
--ci also use GitHub Actions results (goes online, through your gh login)
|
|
195
211
|
--claude-dir <dir> Claude Code projects dir (default ~/.claude/projects, or $CLAUDE_CONFIG_DIR/projects)
|
|
196
212
|
--codex-dir <dir> Codex sessions dir (default ~/.codex/sessions, or $CODEX_HOME/sessions)
|
|
@@ -209,7 +225,7 @@ npm run benchmark:replay
|
|
|
209
225
|
npm run benchmark:live
|
|
210
226
|
```
|
|
211
227
|
|
|
212
|
-
Before distributing a new version, verify `npm view agent-polygraph version`, the published tarball (`npm pack --dry-run`), and that a clean install really contains the latest privacy and guard fixes. The
|
|
228
|
+
Before distributing a new version, verify `npm view agent-polygraph version`, the published tarball (`npm pack --dry-run`), and that a clean install really contains the latest privacy and guard fixes. The public npm release is currently **0.2.1**. Development on `main` may contain unreleased changes, so `npx agent-polygraph` does not necessarily have the latest GitHub fixes. A green GitHub Actions badge does **not** prove npm has been updated.
|
|
213
229
|
|
|
214
230
|
## Privacy
|
|
215
231
|
|
|
@@ -224,7 +240,7 @@ It reads files and works offline. These operations are opt-in:
|
|
|
224
240
|
|
|
225
241
|
- `guard` and `record` edit the hook config files they print when you install or uninstall them, and `record --shell` adds or removes one marked line in `~/.bashrc` and `~/.zshrc`.
|
|
226
242
|
- `record` saves snapshots and logs in `~/.agent-polygraph`.
|
|
227
|
-
- `--ci`
|
|
243
|
+
- `--ci` and `explain --ci` contact api.github.com through `gh` and send repository names and date ranges.
|
|
228
244
|
|
|
229
245
|
The terminal report and `--json` include excerpts from your sessions. `--html` doesn't.
|
|
230
246
|
|
package/bin/polygraph.js
CHANGED
|
@@ -10,6 +10,15 @@ import { renderHtml } from "../src/report/html.js";
|
|
|
10
10
|
|
|
11
11
|
const { version } = createRequire(import.meta.url)("../package.json");
|
|
12
12
|
|
|
13
|
+
if (process.argv[2] === "explain") {
|
|
14
|
+
await explainCommand(process.argv.slice(3));
|
|
15
|
+
process.exit(0);
|
|
16
|
+
}
|
|
17
|
+
if (process.argv[2] === "doctor") {
|
|
18
|
+
await doctorCommand(process.argv.slice(3));
|
|
19
|
+
process.exit(0);
|
|
20
|
+
}
|
|
21
|
+
|
|
13
22
|
if (process.argv[2] === "guard") {
|
|
14
23
|
await guard(process.argv.slice(3));
|
|
15
24
|
process.exit(0);
|
|
@@ -43,6 +52,15 @@ Usage: npx agent-polygraph [options]
|
|
|
43
52
|
-h, --help show this help
|
|
44
53
|
-v, --version show the version
|
|
45
54
|
|
|
55
|
+
Investigate evidence (read-only):
|
|
56
|
+
npx agent-polygraph explain [--limit 10] [--since 7d] [--project name]
|
|
57
|
+
--session <prefix> filter claims by session ID prefix
|
|
58
|
+
--agent claude|codex select agent (repeatable)
|
|
59
|
+
--ci optionally enrich with GitHub Actions evidence (network)
|
|
60
|
+
npx agent-polygraph doctor [--agent claude|codex] [--ci]
|
|
61
|
+
Check local transcripts, hook configuration and recorder permissions.
|
|
62
|
+
--ci optionally checks GitHub CLI authentication (may contact GitHub).
|
|
63
|
+
|
|
46
64
|
Guard (stop unbacked "tests pass" claims as they happen):
|
|
47
65
|
|
|
48
66
|
npx agent-polygraph guard --install add a Stop hook to this project (.claude/, .codex/)
|
|
@@ -59,6 +77,70 @@ Record (more evidence than the logs carry; stays on this machine):
|
|
|
59
77
|
npx agent-polygraph record --uninstall [--global] [--shell]
|
|
60
78
|
`;
|
|
61
79
|
|
|
80
|
+
async function explainCommand(argv) {
|
|
81
|
+
const { values } = parseArgs({
|
|
82
|
+
args: argv,
|
|
83
|
+
options: {
|
|
84
|
+
since: { type: "string" },
|
|
85
|
+
project: { type: "string" },
|
|
86
|
+
session: { type: "string" },
|
|
87
|
+
agent: { type: "string", multiple: true },
|
|
88
|
+
limit: { type: "string", default: "10" },
|
|
89
|
+
ci: { type: "boolean" },
|
|
90
|
+
demo: { type: "boolean" },
|
|
91
|
+
"claude-dir": { type: "string" },
|
|
92
|
+
"codex-dir": { type: "string" },
|
|
93
|
+
},
|
|
94
|
+
});
|
|
95
|
+
const agents = values.agent?.map((s) => s.toLowerCase());
|
|
96
|
+
if (agents?.some((a) => !["claude", "codex"].includes(a))) throw Error("--agent must be claude or codex");
|
|
97
|
+
const limit = Number(values.limit);
|
|
98
|
+
if (!Number.isInteger(limit) || limit < 1 || limit > 1000) throw Error("--limit must be an integer from 1 to 1000");
|
|
99
|
+
const since = parseSince(values.since);
|
|
100
|
+
if (values.since && !since) throw Error("--since must be a date or age such as 7d");
|
|
101
|
+
let roots = { claude: values["claude-dir"], codex: values["codex-dir"] };
|
|
102
|
+
let demoDir = null;
|
|
103
|
+
if (values.demo) {
|
|
104
|
+
const { makeDemo } = await import("../src/demo.js");
|
|
105
|
+
const d = makeDemo();
|
|
106
|
+
demoDir = d.root;
|
|
107
|
+
roots = { claude: d.claudeDir, codex: d.codexDir };
|
|
108
|
+
}
|
|
109
|
+
let ci;
|
|
110
|
+
if (values.ci && !values.demo) {
|
|
111
|
+
const { checkGh, makeCi } = await import("../src/ci.js");
|
|
112
|
+
await checkGh();
|
|
113
|
+
ci = makeCi();
|
|
114
|
+
}
|
|
115
|
+
try {
|
|
116
|
+
const sessions = await scan({
|
|
117
|
+
agents, since, project: values.project, roots, records: !values.demo,
|
|
118
|
+
enrich: ci?.enrich,
|
|
119
|
+
});
|
|
120
|
+
const { renderExplanations } = await import("../src/explain.js");
|
|
121
|
+
console.log(renderExplanations(sessions, { limit, sessionId: values.session }));
|
|
122
|
+
} finally {
|
|
123
|
+
if (demoDir) fs.rmSync(demoDir, { recursive: true, force: true });
|
|
124
|
+
}
|
|
125
|
+
}
|
|
126
|
+
|
|
127
|
+
async function doctorCommand(argv) {
|
|
128
|
+
const { values } = parseArgs({
|
|
129
|
+
args: argv,
|
|
130
|
+
options: {
|
|
131
|
+
agent: { type: "string", multiple: true },
|
|
132
|
+
ci: { type: "boolean" },
|
|
133
|
+
"claude-dir": { type: "string" },
|
|
134
|
+
"codex-dir": { type: "string" },
|
|
135
|
+
},
|
|
136
|
+
});
|
|
137
|
+
const agents = values.agent?.map((s) => s.toLowerCase()) || ["claude", "codex"];
|
|
138
|
+
if (agents.some((a) => !["claude", "codex"].includes(a))) throw Error("--agent must be claude or codex");
|
|
139
|
+
const { diagnose, renderDoctor } = await import("../src/doctor.js");
|
|
140
|
+
const checks = await diagnose({ agents, checkCi: values.ci, roots: { claude: values["claude-dir"], codex: values["codex-dir"] } });
|
|
141
|
+
console.log(renderDoctor(checks));
|
|
142
|
+
}
|
|
143
|
+
|
|
62
144
|
async function record(argv) {
|
|
63
145
|
const { values } = parseArgs({
|
|
64
146
|
args: argv,
|
package/package.json
CHANGED
package/src/analyze.js
CHANGED
|
@@ -236,7 +236,7 @@ export function analyzeSession(session) {
|
|
|
236
236
|
const forSha = before.filter((r) => r.sha === sha);
|
|
237
237
|
if (forSha.some((r) => r.completed && r.completed <= ev.ts && r.conclusion === "failure")) verdict = "failed";
|
|
238
238
|
else if (forSha.some((r) => !r.completed || r.completed > ev.ts)) [verdict, why] = ["unknown", "CI still running"];
|
|
239
|
-
else if (forSha.every((r) =>
|
|
239
|
+
else if (forSha.every((r) => r.conclusion === "success")) verdict = "backed";
|
|
240
240
|
else [verdict, why] = ["unknown", "CI run didn't finish normally"];
|
|
241
241
|
}
|
|
242
242
|
s.claims[verdict]++;
|
|
@@ -355,6 +355,11 @@ export function analyzeSession(session) {
|
|
|
355
355
|
// "the full suite passes" to the latest run as large as the session's full runs.
|
|
356
356
|
// Without such a hint, the latest run.
|
|
357
357
|
const ref = referencedRun(claim, runs, lastTest) || lastTest;
|
|
358
|
+
// A passing targeted test cannot, by itself, verify an explicit
|
|
359
|
+
// "full suite passes" claim. When scope isn't established, abstain.
|
|
360
|
+
const focusedOnly = claimsFullSuite(claim) && ref &&
|
|
361
|
+
!(runs.get(ref.i)?.full ?? ref.full) &&
|
|
362
|
+
![...runs.values()].some((r) => r.full && sameDir(r.dir, ref.dir));
|
|
358
363
|
let staleEdit = ref && codeEdits.findLast((e) => e.i > ref.i && within(e.abs, ref.dir));
|
|
359
364
|
// a change we couldn't follow, settled by comparing `record` snapshots of the
|
|
360
365
|
// working tree after the run and after the change
|
|
@@ -384,6 +389,7 @@ export function analyzeSession(session) {
|
|
|
384
389
|
verdict = pendingTest >= 0 || !lastCodeEdit ? "unknown" : "no-run";
|
|
385
390
|
if (verdict === "unknown") why = pendingTest >= 0 ? "test still running" : "no code work in session";
|
|
386
391
|
} else if (staleEdit) verdict = "stale";
|
|
392
|
+
else if (focusedOnly) [verdict, why] = ["unknown", "no full-suite test run observed"];
|
|
387
393
|
else if (uncertain) [verdict, why] = ["unknown", "unresolved file change after the run"];
|
|
388
394
|
// a subagent's failing run in its own worktree says little about the main checkout
|
|
389
395
|
else if (ref.outcome === "fail") [verdict, why] = ref.sub ? ["unknown", "subagent run failed"] : ["failed"];
|
|
@@ -501,7 +507,12 @@ function referencedRun(claim, runs, latest) {
|
|
|
501
507
|
* size. Judged by the counts runners printed, never by the command (too unreliable).
|
|
502
508
|
*/
|
|
503
509
|
function fullSuiteRun(runs, latest) {
|
|
504
|
-
if (!latest
|
|
510
|
+
if (!latest) return null;
|
|
511
|
+
// Prefer command-level evidence of full-suite coverage when we have it.
|
|
512
|
+
// This works even if test output omitted a reliable pass count.
|
|
513
|
+
const explicit = [...runs.values()].filter((r) => r.full && sameDir(r.dir, latest.dir));
|
|
514
|
+
if (explicit.length) return explicit.reduce((a, b) => (b.i > a.i ? b : a));
|
|
515
|
+
if (latest.count == null) return null;
|
|
505
516
|
const same = [...runs.values()].filter((r) => r.count != null && !r.external && sameDir(r.dir, latest.dir));
|
|
506
517
|
const max = Math.max(...same.map((r) => r.count));
|
|
507
518
|
// a different command, and under half the size: a focused run, not the suite shrinking
|
package/src/ci.js
CHANGED
|
@@ -82,17 +82,36 @@ export function makeCi({ gh = ghApi } = {}) {
|
|
|
82
82
|
return runsCache.get(key);
|
|
83
83
|
};
|
|
84
84
|
|
|
85
|
-
/**
|
|
86
|
-
|
|
87
|
-
|
|
85
|
+
/** Determine whether test jobs or test steps *actually* passed or failed.
|
|
86
|
+
* A test-named workflow is not evidence if its test job was skipped.
|
|
87
|
+
* Unavailable, incomplete and neutral results are not proof of a pass.
|
|
88
|
+
*/
|
|
89
|
+
const testVerdict = async (repo, r) => {
|
|
88
90
|
if (!jobsCache.has(r.id)) {
|
|
89
91
|
jobsCache.set(
|
|
90
92
|
r.id,
|
|
91
93
|
(async () => {
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
94
|
+
const results = [];
|
|
95
|
+
for (let page = 1; page <= 5; page++) {
|
|
96
|
+
stats.requests++;
|
|
97
|
+
const data = await gh(`repos/${repo}/actions/runs/${r.id}/jobs`, { per_page: "100", page: String(page) });
|
|
98
|
+
const jobs = data?.jobs || [];
|
|
99
|
+
for (const job of jobs) {
|
|
100
|
+
const steps = (job.steps || []).filter((step) => TEST_STEP.test(step.name || ""));
|
|
101
|
+
if (steps.length) {
|
|
102
|
+
for (const step of steps) results.push(step.conclusion || null);
|
|
103
|
+
} else if (TEST_STEP.test(job.name || "")) {
|
|
104
|
+
// For a test-named job, the job result itself is evidence.
|
|
105
|
+
results.push(job.conclusion || null);
|
|
106
|
+
}
|
|
107
|
+
}
|
|
108
|
+
if (jobs.length < 100) break;
|
|
109
|
+
}
|
|
110
|
+
if (!results.length) return null;
|
|
111
|
+
if (results.some((v) => v === "failure" || v === "timed_out")) return "fail";
|
|
112
|
+
if (results.every((v) => v === "success")) return "pass";
|
|
113
|
+
return "unknown";
|
|
114
|
+
})().catch(() => "unknown"),
|
|
96
115
|
);
|
|
97
116
|
}
|
|
98
117
|
return jobsCache.get(r.id);
|
|
@@ -156,12 +175,17 @@ export function makeCi({ gh = ghApi } = {}) {
|
|
|
156
175
|
forPush = forPush.filter((r) => r.sha === first);
|
|
157
176
|
}
|
|
158
177
|
const tests = [];
|
|
159
|
-
for (const r of forPush)
|
|
178
|
+
for (const r of forPush) {
|
|
179
|
+
if (!r.completed || ["skipped", "neutral", "startup_failure", "cancelled"].includes(r.conclusion)) continue;
|
|
180
|
+
const testOutcome = await testVerdict(p.repo, r);
|
|
181
|
+
if (testOutcome) tests.push({ ...r, testOutcome });
|
|
182
|
+
}
|
|
160
183
|
if (!tests.length) continue;
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
const
|
|
164
|
-
|
|
184
|
+
// A failed test step is definitive. All selected test results must be
|
|
185
|
+
// successful before we can back a claim.
|
|
186
|
+
const failed = tests.some((r) => r.testOutcome === "fail");
|
|
187
|
+
const passed = tests.every((r) => r.testOutcome === "pass");
|
|
188
|
+
if (!failed && !passed) continue; // unavailable, skipped or inconclusive test steps
|
|
165
189
|
const finished = Math.max(...tests.map((r) => r.completed));
|
|
166
190
|
const label = `CI: ${[...new Set(tests.map((r) => r.name))].join(", ")} (${tests[0].sha.slice(0, 7)})`;
|
|
167
191
|
// the run starts at the push: edits after it aren't in the pushed commit
|
package/src/doctor.js
ADDED
|
@@ -0,0 +1,94 @@
|
|
|
1
|
+
// Read-only diagnostic for transcript discovery, hook wiring and local recorder paths.
|
|
2
|
+
// Never prints transcript contents, credentials, or raw hook payloads.
|
|
3
|
+
import fs from "node:fs";
|
|
4
|
+
import os from "node:os";
|
|
5
|
+
import path from "node:path";
|
|
6
|
+
import { SOURCES } from "./scan.js";
|
|
7
|
+
import { isOurHook, targets } from "./guard.js";
|
|
8
|
+
import { readSettings } from "./safe-config.js";
|
|
9
|
+
import { home as recordHome } from "./record.js";
|
|
10
|
+
|
|
11
|
+
function checkConfig(file, sub) {
|
|
12
|
+
try {
|
|
13
|
+
const { text } = readSettings(file);
|
|
14
|
+
if (text === null) return "not installed";
|
|
15
|
+
const cfg = JSON.parse(text);
|
|
16
|
+
const event = sub === "guard" ? "Stop" : "PostToolUse";
|
|
17
|
+
const hooks = (cfg.hooks?.[event] || []).flatMap((g) => g.hooks || []);
|
|
18
|
+
return hooks.some((h) => isOurHook(h, sub)) ? "installed" : "not installed";
|
|
19
|
+
} catch (e) {
|
|
20
|
+
return `ERROR: invalid or unsafe settings (${e.code || e.name || "error"})`;
|
|
21
|
+
}
|
|
22
|
+
}
|
|
23
|
+
|
|
24
|
+
/** Diagnose local files without mutating settings or running any tests. */
|
|
25
|
+
export async function diagnose({ cwd = process.cwd(), roots = {}, agents = ["claude", "codex"], sample = 2, checkCi = false } = {}) {
|
|
26
|
+
const checks = [];
|
|
27
|
+
for (const agent of agents) {
|
|
28
|
+
const src = SOURCES[agent];
|
|
29
|
+
if (!src) continue;
|
|
30
|
+
const root = roots[agent] || src.defaultRoot();
|
|
31
|
+
let entries;
|
|
32
|
+
try { entries = src.discover(root); }
|
|
33
|
+
catch (e) {
|
|
34
|
+
checks.push({ kind: "warn", message: `${agent}: cannot list transcripts (${e.code || e.name})` });
|
|
35
|
+
continue;
|
|
36
|
+
}
|
|
37
|
+
checks.push({ kind: entries.length ? "ok" : "info", message: `${agent}: ${entries.length} session file(s) discovered in ${root}` });
|
|
38
|
+
if (entries.length) {
|
|
39
|
+
const latest = entries.sort((a, b) => b.mtime - a.mtime).slice(0, sample);
|
|
40
|
+
let unsupported = 0, unreadable = 0;
|
|
41
|
+
for (const entry of latest) {
|
|
42
|
+
try { if ((await src.parse(entry)).unrecognized) unsupported++; }
|
|
43
|
+
catch { unreadable++; }
|
|
44
|
+
}
|
|
45
|
+
checks.push({ kind: unsupported || unreadable ? "warn" : "ok", message:
|
|
46
|
+
`${agent}: sampled ${latest.length} session(s); ${unsupported} unrecognized formats, ${unreadable} unreadable` });
|
|
47
|
+
}
|
|
48
|
+
for (const global of [false, true]) {
|
|
49
|
+
const files = targets({ agents: [agent], global, cwd });
|
|
50
|
+
for (const { file } of files) {
|
|
51
|
+
const status = checkConfig(file, "guard");
|
|
52
|
+
checks.push({ kind: status.startsWith("ERROR") ? "warn" : status === "installed" ? "ok" : "info",
|
|
53
|
+
message: `${agent}: ${global ? "global" : "project"} guard ${status} (${file})` });
|
|
54
|
+
const record = checkConfig(file, "record");
|
|
55
|
+
checks.push({ kind: record.startsWith("ERROR") ? "warn" : record === "installed" ? "ok" : "info",
|
|
56
|
+
message: `${agent}: ${global ? "global" : "project"} recorder ${record}` });
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
}
|
|
60
|
+
const dir = recordHome();
|
|
61
|
+
try {
|
|
62
|
+
const st = fs.lstatSync(dir);
|
|
63
|
+
const linked = st.isSymbolicLink() || !st.isDirectory();
|
|
64
|
+
const shared = process.platform !== "win32" && (st.mode & 0o077) !== 0;
|
|
65
|
+
checks.push({ kind: linked || shared ? "warn" : "ok", message:
|
|
66
|
+
linked ? "Local recorder directory is not a safe regular directory" :
|
|
67
|
+
shared ? "Local recorder directory permits group/other access" :
|
|
68
|
+
`Local recorder directory exists (${dir})` });
|
|
69
|
+
} catch (e) {
|
|
70
|
+
checks.push({ kind: e.code === "ENOENT" ? "info" : "warn", message:
|
|
71
|
+
e.code === "ENOENT" ? "Local recorder not initialized (optional)" : `Local recorder inaccessible (${e.code || e.name})` });
|
|
72
|
+
}
|
|
73
|
+
if (checkCi) {
|
|
74
|
+
// Explicit opt-in: checking gh login can contact GitHub, unlike the default doctor.
|
|
75
|
+
try {
|
|
76
|
+
const { checkGh } = await import("./ci.js");
|
|
77
|
+
await checkGh();
|
|
78
|
+
checks.push({ kind: "ok", message: "GitHub CLI authentication available for --ci" });
|
|
79
|
+
} catch (e) {
|
|
80
|
+
checks.push({ kind: "warn", message: `GitHub CLI authentication unavailable for --ci: ${e.message}` });
|
|
81
|
+
}
|
|
82
|
+
}
|
|
83
|
+
return checks;
|
|
84
|
+
}
|
|
85
|
+
|
|
86
|
+
export function renderDoctor(checks) {
|
|
87
|
+
const severity = { ok: "OK", warn: "WARN", info: "INFO" };
|
|
88
|
+
const warnings = checks.filter((c) => c.kind === "warn").length;
|
|
89
|
+
return ["agent-polygraph doctor — read-only, local diagnostics",
|
|
90
|
+
...checks.map((c) => `[${severity[c.kind] || "INFO"}] ${c.message}`),
|
|
91
|
+
"", warnings ? `${warnings} warning(s) require investigation.` : "No diagnostic warnings in the checks performed.",
|
|
92
|
+
"Not-installed hooks are optional; a plugin can supply a separate guard.",
|
|
93
|
+
"This is not a security audit or a substitute for checking actual claim evidence."].join("\n");
|
|
94
|
+
}
|
package/src/explain.js
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
import { AGENT_NAMES } from "./report/format.js";
|
|
2
|
+
|
|
3
|
+
const LABELS = {
|
|
4
|
+
"no-run": "No recognized test run",
|
|
5
|
+
stale: "Test result is stale",
|
|
6
|
+
failed: "Latest relevant test failed",
|
|
7
|
+
unknown: "Insufficient evidence",
|
|
8
|
+
};
|
|
9
|
+
|
|
10
|
+
const when = (ms) => Number.isFinite(ms) ? new Date(ms).toISOString() : "time unavailable";
|
|
11
|
+
const printable = (s, n = 240) => String(s ?? "").replace(/[\r\n\t]+/g, " ").slice(0, n);
|
|
12
|
+
|
|
13
|
+
/**
|
|
14
|
+
* Human-readable, local-only explanation of existing verdict evidence.
|
|
15
|
+
* No commands are executed and no raw transcripts are uploaded.
|
|
16
|
+
* Only recorded findings can be explained; backed claims have no finding.
|
|
17
|
+
*/
|
|
18
|
+
export function renderExplanations(sessions, { limit = 10, sessionId = null } = {}) {
|
|
19
|
+
const selected = sessions.filter((s) => !sessionId || s.id.startsWith(sessionId));
|
|
20
|
+
const entries = selected.flatMap((s) => (s.findings || [])
|
|
21
|
+
.filter((f) => f.type === "claim" || f.type === "unclear")
|
|
22
|
+
.map((f) => ({ s, f })))
|
|
23
|
+
.sort((a, b) => (b.f.ts || 0) - (a.f.ts || 0));
|
|
24
|
+
const count = Math.max(0, Math.trunc(limit));
|
|
25
|
+
const lines = ["agent-polygraph explain — local evidence, not proof that code is correct",
|
|
26
|
+
`Sessions: ${selected.length}; flagged or uncertain claims: ${entries.length}`];
|
|
27
|
+
if (!entries.length) {
|
|
28
|
+
lines.push("No flagged/uncertain claims found. Backed claims are counted in the main report, not listed here.");
|
|
29
|
+
return lines.join("\n");
|
|
30
|
+
}
|
|
31
|
+
for (const { s, f } of entries.slice(0, count)) {
|
|
32
|
+
const d = f.detail || {};
|
|
33
|
+
const verdict = f.type === "unclear" ? "unknown" : d.verdict;
|
|
34
|
+
lines.push("", `[${LABELS[verdict] || verdict || "Unclassified"}] ${when(f.ts)} · ${AGENT_NAMES[s.agent] || s.agent} · session ${s.id}`);
|
|
35
|
+
lines.push(` Claim: ${printable(d.claim)}`);
|
|
36
|
+
if (d.lastTest) {
|
|
37
|
+
const lt = d.lastTest;
|
|
38
|
+
const at = typeof lt.ago === "number" && Number.isFinite(f.ts) ? when(f.ts - lt.ago) : "time not recorded";
|
|
39
|
+
lines.push(` Test evidence: ${printable(lt.cmd, 200)} · ${lt.outcome || "outcome unknown"} · ${at}`);
|
|
40
|
+
} else lines.push(" Test evidence: none identified");
|
|
41
|
+
if (d.editedAfter) lines.push(` Subsequent edit: ${printable(d.editedAfter, 180)}`);
|
|
42
|
+
if (d.why) lines.push(` Missing evidence: ${printable(d.why)}`);
|
|
43
|
+
if (d.ci) lines.push(" Source: optional GitHub Actions evidence");
|
|
44
|
+
if (s.file) lines.push(` Transcript (private): ${s.file}`);
|
|
45
|
+
}
|
|
46
|
+
if (entries.length > count) lines.push("", `Showing ${count} of ${entries.length} claims; increase --limit to see more.`);
|
|
47
|
+
lines.push("", "Contains transcript excerpts and local file paths. Review before sharing.");
|
|
48
|
+
return lines.join("\n");
|
|
49
|
+
}
|