driftproof 0.7.2 → 0.8.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,25 +1,28 @@
1
1
  <!-- SPDX-License-Identifier: Apache-2.0 -->
2
2
  # Driftproof
3
3
 
4
- **A dated proof that this skill, this hash, this model, still helps.**
4
+ **Your skill passed. On which model? On what date?**
5
+
6
+ A SKILL.md teaches an AI coding agent how you like things done. When a new model
7
+ ships, the same file can stop helping, or start hurting. Driftproof re-runs the
8
+ skill's tests on the new model and hands you a dated, hash-verified receipt
9
+ saying whether it still helps.
5
10
 
6
11
  [![driftproof](https://img.shields.io/endpoint?url=https://driftproofhq.com/badges/commit-message-conventions.json)](https://driftproofhq.com)
7
12
  &nbsp;— live badge for the bundled `commit-message-conventions` example, generated from its own receipt.
8
13
 
9
- Driftproof is an open **receipt spec** plus a **runner** that measures whether an
10
- agent skill actually helps — by running the skill's eval suite **with** and
11
- **without** the skill on a named model version, judging each case several times to
12
- get a confidence band, and emitting a **hash-verified**, dated **receipt**. Diff two receipts
13
- across model releases and you get a **drift report**.
14
+ **[Quickstart](https://driftproofhq.com/#quickstart)** ·
15
+ **[Latest report](https://driftproofhq.com/reports/008/)** ·
16
+ [driftproofhq.com](https://driftproofhq.com)
14
17
 
15
18
  Driftproof **consumes** the [`agentskills.io/evals`](https://agentskills.io) eval
16
19
  format; it does not invent its own.
17
20
 
18
- 📊 **Seven published reports** (each re-derived from committed receipts, nothing
19
- hand-entered), spanning six published report types, the newest being instrument
20
- re-measurement. All seven share one band-based, floor-gated verdict rule and
21
+ 📊 **Eight published reports** (each re-derived from committed receipts, nothing
22
+ hand-entered), spanning six published report types, the newest being release
23
+ drift. All eight share one band-based, floor-gated verdict rule and
21
24
  differ in what moves underneath the skill — or, in the value report, in which
22
- axes are measured; or, in the newest, in the instrument itself:
25
+ axes are measured; or, in the instrument re-measurement, in the instrument itself:
23
26
 
24
27
  - **[Report #001](https://driftproofhq.com/reports/001/)** — *release drift*: ten
25
28
  public agent skills across a current-vs-previous Sonnet release; 9 of 10 moved
@@ -62,9 +65,18 @@ axes are measured; or, in the newest, in the instrument itself:
62
65
  2026-07-27, and the truncated run it caused measured *less* variance than the
63
66
  clean re-run, which is the direction that flatters an instrument. Both runs are
64
67
  published, the broken one as evidence. Amends #005 to v1.2 and #006 to v1.1.
68
+ - **[Report #008](https://driftproofhq.com/reports/008/)** — *release drift*:
69
+ two of Report #007's cells re-measured on `claude-fable-5-1` against
70
+ `claude-fable-5`, with the skill `content_hash` and `suite_hash` asserted
71
+ identical before the first call. **Both cells came back within noise**: the
72
+ 14 cases read 0 improved, 0 regressed, 14 within noise, 0 not measured. The
73
+ first release pair in this project where both sides are generation-sampled
74
+ receipts, which is what makes the delta attributable to the model rather than
75
+ to the instrument. One case sits inside the verdict on the effect floor alone
76
+ and the report names it.
65
77
 
66
78
  ✍️ The launch essay, **[Three model releases later: what actually happens to agent
67
- skills](https://driftproofhq.com/writing/three-releases/)**, reads all seven reports
79
+ skills](https://driftproofhq.com/writing/three-releases/)**, reads all eight reports
68
80
  together: what moves underneath a skill, what the skill costs to run, and what a
69
81
  corrected instrument did to three published results. Revised 2026-09-01; every
70
82
  figure in it is gate-checked against the report page it cites.
@@ -199,6 +211,36 @@ Writing a suite that measures fairly is its own craft — see
199
211
 
200
212
  The surface and judge settings used are recorded in every receipt.
201
213
 
214
+ ### Isolation (the cli surfaces)
215
+
216
+ A `SKILL.md` you did not write is instructions to an agent that has tools. On
217
+ the two cli surfaces driftproof therefore never runs `claude` or `codex` as you.
218
+ Every spawn goes through a hop to a dedicated unprivileged unix user, from an
219
+ empty environment:
220
+
221
+ ```
222
+ /usr/bin/sudo -n -u driftproof-eval /usr/bin/env -i HOME=<its home> PATH=<its ~/.local/bin>:/usr/bin:/bin bash -lc '<wrapper>' driftproof-eval-hop claude -p ...
223
+ ```
224
+
225
+ Only `HOME` and `PATH` reach the child, both constructed from the user name, none
226
+ copied from your shell. The working directory is a fresh temp directory made for
227
+ that one call and removed after it, including on timeout. The CLI argv is passed
228
+ as separate arguments, never as a shell string. What the evaluated agent cannot
229
+ see: your environment (tokens, keys), your home directory (SSH keys, CLI session
230
+ files), your current directory, and your CLI configuration.
231
+
232
+ **Operator prerequisite, set up once** (see `RUNBOOK.md`): that user exists with a
233
+ mode-700 home and no extra groups, has its own logged-in `claude` and `codex`
234
+ under `~/.local/bin`, and a sudoers rule lets you run commands as it with
235
+ `NOPASSWD`. `DRIFTPROOF_EVAL_USER` names the user (default `driftproof-eval`).
236
+ Without the prerequisite a run fails at once, before any call, and says so.
237
+
238
+ **`--trusted-skill`** runs the legacy same-user path instead: your environment,
239
+ your cwd, your CLI login. It exists for exactly one case, a skill you authored
240
+ yourself on a machine with no eval user. Never pass it for a skill fetched from
241
+ anywhere else. The api surfaces are unaffected either way; they make HTTP calls
242
+ and spawn nothing.
243
+
202
244
  ### Cost guard
203
245
 
204
246
  Sampling multiplies calls on two axes, and the second one is easy to miss. Each
@@ -229,7 +271,7 @@ A receipt is the unit of evidence — one JSON document conforming to
229
271
  "model_release_date": "2025-10-01",
230
272
  "provider": "anthropic",
231
273
  "surface": "claude-cli",
232
- "runner_version": "0.7.2",
274
+ "runner_version": "0.8.1",
233
275
  "date_utc": "2026-07-27T…Z",
234
276
  "registry": "registered",
235
277
  "transcripts": "hashes-only",
@@ -303,7 +345,7 @@ jobs:
303
345
  runs-on: ubuntu-latest
304
346
  steps:
305
347
  - uses: actions/checkout@v4
306
- - uses: driftproofhq/driftproof@v0.7.2
348
+ - uses: driftproofhq/driftproof@v0.8.1
307
349
  with:
308
350
  skill-dir: skills/my-skill
309
351
  models: claude-haiku-4-5
@@ -319,7 +361,11 @@ fails the job on `REGRESSED` unless you set `fail-on-regression: 'false'`. For a
319
361
  free CI dry-run with **zero model calls**, set `DRIFTPROOF_STUB=1` in the job env —
320
362
  the runner returns canned receipts so the wiring can be tested without spend (this
321
363
  is exactly how the action's own [self-test](.github/workflows/action-selftest.yml)
322
- runs).
364
+ runs). That self-test is also the proof behind the input hardening: on every
365
+ run it passes one hostile value (a quote, a semicolon, `$(...)`, a backtick and
366
+ a newline) through each of the action's five inputs on the real runner and
367
+ fails unless every one is refused before anything ran, so a green check there
368
+ is a claim you can read, not one you have to take.
323
369
 
324
370
  ### Badge
325
371
 
@@ -343,7 +389,7 @@ site, so it reflects a real dated run, not a hand-set color.
343
389
 
344
390
  ## Reports
345
391
 
346
- Seven reports are published, spanning six report types. A report page lives at a
392
+ Eight reports are published, spanning six report types. A report page lives at a
347
393
  draft path — `docs/reports/NNN-draft/` — until the publish sequence renames it, and
348
394
  `scripts/build-public.sh` excludes every `*-draft/` path from the published tree
349
395
  (see the roll at the top of this README, and
package/bin/driftproof CHANGED
@@ -10,7 +10,7 @@ const { runSkillOnModel, summarizeReceipt, projectCalls } = require('../lib/run'
10
10
  const { SAMPLING } = require('../lib/sampling');
11
11
  const { validateReceipt, verifyReceiptHash } = require('../lib/receipt');
12
12
  const { buildDriftReport, revisionPairProblem } = require('../lib/diff');
13
- const { surfaceForModel, isSubscriptionSurface, resolveModel } = require('../lib/provider');
13
+ const { surfaceForModel, isSubscriptionSurface, resolveModel, evalUser } = require('../lib/provider');
14
14
  const { estimateRunCostUSD, BudgetTracker } = require('../lib/cost');
15
15
  const { registryStatus } = require('../lib/models');
16
16
  const { verdictFromReceipt, badgeEndpoint, githubOutputLines } = require('../lib/verdict');
@@ -54,6 +54,10 @@ function loadRc(skillDir) {
54
54
  return merged;
55
55
  }
56
56
 
57
+ // Flags that never take a value, so `--trusted-skill <skill-dir>` keeps the dir
58
+ // positional instead of swallowing it as the flag's value.
59
+ const BOOLEAN_FLAGS = new Set(['trusted-skill']);
60
+
57
61
  function parseArgs(argv) {
58
62
  const positional = [];
59
63
  const flags = {};
@@ -62,7 +66,7 @@ function parseArgs(argv) {
62
66
  if (a.startsWith('--')) {
63
67
  const key = a.slice(2);
64
68
  const next = argv[i + 1];
65
- if (next === undefined || next.startsWith('--')) { flags[key] = true; }
69
+ if (BOOLEAN_FLAGS.has(key) || next === undefined || next.startsWith('--')) { flags[key] = true; }
66
70
  else { flags[key] = next; i++; }
67
71
  } else positional.push(a);
68
72
  }
@@ -76,7 +80,7 @@ USAGE
76
80
  ${PROJECT_NAME} init <dir> scaffold SKILL.md + evals/evals.json + .driftproofrc
77
81
  ${PROJECT_NAME} run <skill-dir> [--models a,b] [--samples N] [--max-cases N] [--max-calls N]
78
82
  [--judge-model M] [--concurrency N] [--max-usd N]
79
- [--keep-transcripts] [--out DIR]
83
+ [--keep-transcripts] [--out DIR] [--trusted-skill]
80
84
  ${PROJECT_NAME} diff <receiptA.json> <receiptB.json> [--out FILE] [--mode release|revision]
81
85
  ${PROJECT_NAME} validate <receipt.json>
82
86
  ${PROJECT_NAME} badge <receipt.json> [--out FILE] [--github-output]
@@ -88,6 +92,7 @@ ENV
88
92
  ANTHROPIC_API_KEY required when CLAUDE_PROVIDER=api
89
93
  DRIFTPROOF_REGISTRY path to an alternate model registry (default: the packaged config/models.json)
90
94
  DRIFTPROOF_STUB=1 offline stub surface — canned receipts, zero model calls (for CI)
95
+ DRIFTPROOF_EVAL_USER the unix user the isolated hop runs the CLI as (default: driftproof-eval)
91
96
 
92
97
  NOTES
93
98
  - Default models list is 'haiku' only (cheap dev default).
@@ -112,6 +117,16 @@ NOTES
112
117
  conservative default price.
113
118
  - --concurrency runs that many (case,mode) tasks at once (default 1). Higher
114
119
  values cut wall-clock on the cli surface (cold-start dominated).
120
+ - ISOLATION (default). On the two cli surfaces every \`claude\` / \`codex\` spawn
121
+ runs as a dedicated unprivileged unix user (DRIFTPROOF_EVAL_USER) through
122
+ sudo, from an EMPTY environment (only HOME and PATH, constructed), in a fresh
123
+ temp working directory removed after the call. A third-party SKILL.md is
124
+ instructions to an agent with tools; this is what keeps it away from your
125
+ keys, tokens and files. Operator prerequisite: that user, its own logged-in
126
+ CLIs, and a NOPASSWD sudoers rule (see RUNBOOK.md).
127
+ - --trusted-skill runs the legacy same-user path instead: your environment,
128
+ your cwd, your CLI login. FOR SKILLS YOU AUTHORED YOURSELF ONLY. Never pass
129
+ it for a skill fetched from anywhere else.
115
130
  - import converts another tool's results into a valid receipt with HONEST
116
131
  epistemics: verification_level DECLARED (never TESTED), surface "external",
117
132
  source "imported/<tool>", and NO fabricated hashes. Imported receipts are
@@ -137,6 +152,10 @@ async function cmdRun(positional, flags) {
137
152
  const concurrency = flags.concurrency ? parseInt(flags.concurrency, 10) : (rc.concurrency != null ? parseInt(rc.concurrency, 10) : 1);
138
153
  const maxUsd = flags['max-usd'] ? parseFloat(flags['max-usd']) : (rc.max_usd != null ? parseFloat(rc.max_usd) : DEV_MAX_USD);
139
154
  const keepTranscripts = !!flags['keep-transcripts'];
155
+ // spec 022: the same-user spawn exists only behind this flag. Resolving the
156
+ // eval user here means a bad DRIFTPROOF_EVAL_USER is refused before the banner.
157
+ const trusted = !!flags['trusted-skill'];
158
+ const isolation = trusted ? 'same-user (--trusted-skill: self-authored skill)' : `eval-user (${evalUser()})`;
140
159
  const outDir = flags.out ? path.resolve(flags.out) : RECEIPTS_DIR;
141
160
  fs.mkdirSync(outDir, { recursive: true });
142
161
 
@@ -166,6 +185,7 @@ async function cmdRun(positional, flags) {
166
185
  console.log(` content_hash: ${skill.contentHash.slice(0, 16)}… suite: ${skill.suite.caseCount} cases (${skill.suite.suiteHash.slice(0, 12)}…)`);
167
186
  console.log(` surface: ${surface} models: ${models.join(', ')} samples/case: ${samples} concurrency: ${concurrency}`);
168
187
  console.log(` registry: ${regStatuses.join(' ')}${keepTranscripts ? ' transcripts: retained-local' : ''}`);
188
+ if (subSurfaces.length) console.log(` isolation: ${isolation}`);
169
189
  console.log(` projected calls: ${perModelCalls}/model × ${models.length} model(s) = ${totalProjected} per-model cap: ${maxCalls}`);
170
190
  console.log(` projected cost: ~$${cost.totalUSD.toFixed(2)} (rough upper bound; budget $${maxUsd.toFixed(2)}, hard-stop $${(maxUsd * 1.25).toFixed(2)})`);
171
191
  if (subSurfaces.length) console.log(` actual metered spend on ${subSurfaces.join(', ')}: $0.00 (subscription; the $ figure is the estimated-equivalent API cost, counted against the cap identically)`);
@@ -200,7 +220,7 @@ async function cmdRun(positional, flags) {
200
220
  skill,
201
221
  model,
202
222
  opts: {
203
- maxCases, maxCalls, samples, judgeModel, concurrency, budget, keepTranscripts,
223
+ maxCases, maxCalls, samples, judgeModel, concurrency, budget, keepTranscripts, trusted,
204
224
  onProgress: (p) => {
205
225
  if (p.phase === 'done') console.log(` ${p.case} / ${p.mode}: ${p.outcome} (${p.score.toFixed(2)} ± ${(p.stddev || 0).toFixed(2)})`);
206
226
  },
@@ -1,5 +1,5 @@
1
1
  {
2
- "_comment": "Driftproof model registry. The runner resolves --models ids against this list; an unknown id still runs but its receipt is marked registry:\"unregistered\" and its cost is estimated with the conservative default price in lib/models.js. Prices are STANDARD first-party USD per 1,000,000 tokens (input/output). Anthropic prices are from platform.claude.com (July 2026); Sonnet 5's introductory rate is deliberately NOT used so projections stay an upper bound. OpenAI prices are the public standard per-MTok rates as of July 2026 (see reports/phase-6-providers.md for the fetched source): GPT-5.6 Sol $5/$30, Terra $2.50/$15, Luna $1/$6; GPT-5.5 $5/$30; GPT-5.4 $2.50/$15. gpt-5.4-mini is NOT in the published table (which lists a Nano tier at $0.20/$1.25), so its price here is a deliberately conservative UPPER-bound estimate flagged price_estimate:true. `released` is best-effort (spec RECEIPT.md open question #4). `tier` drives reporting; `judge_eligible` encodes the fixed-judge policy (docs/judge-policy.html) — only the cheap Haiku judge is eligible, and NO OpenAI model is judge-eligible (the judge stays claude-haiku across providers so a cross-substrate report varies only the target model). `provider` is the two-axis provider; per-provider surface/base_url config is under `providers`. The release trigger (scripts/release-watch.js) appends auto-discovered ids here with auto_added:true.",
2
+ "_comment": "Driftproof model registry. The runner resolves --models ids against this list; an unknown id still runs but its receipt is marked registry:\"unregistered\" and its cost is estimated with the conservative default price in lib/models.js. Prices are STANDARD first-party USD per 1,000,000 tokens (input/output). Anthropic prices are from platform.claude.com (July 2026); Sonnet 5's introductory rate is deliberately NOT used so projections stay an upper bound. OpenAI prices are the public standard per-MTok rates as of July 2026 (see reports/phase-6-providers.md for the fetched source): GPT-5.6 Sol $5/$30, Terra $2.50/$15, Luna $1/$6; GPT-5.5 $5/$30; GPT-5.4 $2.50/$15. gpt-5.4-mini is NOT in the published table (which lists a Nano tier at $0.20/$1.25), so its price here is a deliberately conservative UPPER-bound estimate flagged price_estimate:true. `released` is best-effort (spec RECEIPT.md open question #4). `tier` drives reporting; `judge_eligible` encodes the fixed-judge policy (docs/judge-policy.html) — only the cheap Haiku judge is eligible, and NO OpenAI model is judge-eligible (the judge stays claude-haiku across providers so a cross-substrate report varies only the target model). `provider` is the two-axis provider; per-provider surface/base_url config is under `providers`. The release trigger (scripts/release-watch.js) appends auto-discovered ids here with auto_added:true. Three optional per-row annotations were added by spec-019b (2026-09-01) and are read by NO code path in lib/: `cache_read_price` is the documented cache-read rate per MTok, recorded for provenance and NOT consumed -- lib/cost.js prices input and output only, and the 019b gate asserts that it stays that way, so wiring caching into the projection turns that assertion red rather than leaving this note quietly false; `lifecycle: legacy` records that the docs moved an id to the Legacy list, and never means removed (claude-fable-5 is referenced by receipts/report-007 and receipts/report-007-rerun); `surface_verified` records that a live call reached the id on that surface on `surface_verified_at`, which is an OBSERVATION, not a routing setting -- the surface is chosen at call time by lib/provider.js from CLAUDE_PROVIDER. claude-fable-5-1 prices are sourced to specs/019b-fable-5-1-registration/evidence/docs-pricing-snapshot-2026-09-01.json.",
3
3
  "registry_version": "1.1",
4
4
  "provider": "anthropic",
5
5
  "providers": {
@@ -8,7 +8,8 @@
8
8
  },
9
9
  "default_price_note": "Unregistered ids are costed at the most expensive known tier (Fable, $10/$50) — a budget guard must never under-estimate. See lib/models.js DEFAULT_PRICE.",
10
10
  "models": [
11
- { "id": "claude-fable-5", "family": "fable", "provider": "anthropic", "released": "2026-06-09", "input_price": 10.0, "output_price": 50.0, "tier": "frontier", "judge_eligible": false },
11
+ { "id": "claude-fable-5-1", "family": "fable", "provider": "anthropic", "released": "2026-09-01", "input_price": 10.0, "output_price": 50.0, "cache_read_price": 0.25, "tier": "frontier", "judge_eligible": false, "surface_verified": "claude-cli", "surface_verified_at": "2026-09-01" },
12
+ { "id": "claude-fable-5", "family": "fable", "provider": "anthropic", "released": "2026-06-09", "input_price": 10.0, "output_price": 50.0, "tier": "frontier", "judge_eligible": false, "lifecycle": "legacy" },
12
13
  { "id": "claude-opus-5", "family": "opus", "provider": "anthropic", "released": null, "input_price": 5.0, "output_price": 25.0, "tier": "frontier", "judge_eligible": false },
13
14
  { "id": "claude-opus-4-8", "family": "opus", "provider": "anthropic", "released": null, "input_price": 5.0, "output_price": 25.0, "tier": "frontier", "judge_eligible": false },
14
15
  { "id": "claude-opus-4-7", "family": "opus", "provider": "anthropic", "released": null, "input_price": 5.0, "output_price": 25.0, "tier": "frontier", "judge_eligible": false },
package/config.js CHANGED
@@ -9,7 +9,7 @@ const PROJECT_NAME = 'driftproof';
9
9
  // Bumped whenever the runner's behaviour or receipt-generation semantics change
10
10
  // in a way that could affect results. Recorded into every receipt as
11
11
  // run.runner_version so a receipt is reproducible against a known engine.
12
- const RUNNER_VERSION = '0.7.2';
12
+ const RUNNER_VERSION = '0.8.1';
13
13
 
14
14
  // The eval format we CONSUME (we deliberately do not invent our own).
15
15
  const SUITE_FORMAT = 'agentskills.io/evals';
package/lib/judge.js CHANGED
@@ -81,9 +81,9 @@ function judgeSettings(samples, judgeModel) {
81
81
  // Grade one response once. Returns { score, reason, raw }.
82
82
  // `raw` is the judge's verbatim output text (hashed into the receipt for
83
83
  // transcript auditability, and optionally retained under --keep-transcripts).
84
- async function gradeOnce({ task, response, rubric, model, timeoutMs, temperature }) {
84
+ async function gradeOnce({ task, response, rubric, model, timeoutMs, temperature, trusted = false }) {
85
85
  const prompt = buildJudgePrompt({ task, response, rubric });
86
- const { text, attempts, usage } = await complete({ system: JUDGE_SYSTEM, prompt, model, maxTokens: 400, timeoutMs, temperature });
86
+ const { text, attempts, usage } = await complete({ system: JUDGE_SYSTEM, prompt, model, maxTokens: 400, timeoutMs, temperature, trusted });
87
87
  let parsed;
88
88
  try {
89
89
  parsed = extractJsonObject(text);
@@ -104,7 +104,9 @@ async function gradeOnce({ task, response, rubric, model, timeoutMs, temperature
104
104
  // JUDGE calls timed out on the api policy while running on a CLI surface, which
105
105
  // is the second shadowing site and the one #007's prep session had not found.
106
106
  // Passing `undefined` through lets lib/provider.js resolve the declared policy.
107
- async function gradeSamples({ task, response, rubric, model, samples = 5, timeoutMs }) {
107
+ // `trusted` (spec 022) is plumbed to complete() untouched: the judge takes the
108
+ // same spawn path as the generation it grades, so --trusted-skill is whole-run.
109
+ async function gradeSamples({ task, response, rubric, model, samples = 5, timeoutMs, trusted = false }) {
108
110
  const settings = judgeSettings(samples, model);
109
111
  const scores = [];
110
112
  const reasons = [];
@@ -114,7 +116,7 @@ async function gradeSamples({ task, response, rubric, model, samples = 5, timeou
114
116
  for (let i = 0; i < samples; i++) {
115
117
  let r;
116
118
  try {
117
- r = await gradeOnce({ task, response, rubric, model, timeoutMs, temperature: settings.temperature === null ? undefined : settings.temperature });
119
+ r = await gradeOnce({ task, response, rubric, model, timeoutMs, temperature: settings.temperature === null ? undefined : settings.temperature, trusted });
118
120
  } catch (e) {
119
121
  // A judge sample that persistently failed (e.g. timed out after retries):
120
122
  // tag the error so the runner can charge for the spend and mark the whole
package/lib/provider.js CHANGED
@@ -4,7 +4,9 @@
4
4
  const fs = require('fs');
5
5
  const os = require('os');
6
6
  const path = require('path');
7
- const { spawn } = require('child_process');
7
+ // Called through the module object (cp.spawn), never destructured: the gate
8
+ // records spawns by replacing that property, so nothing real runs offline.
9
+ const cp = require('child_process');
8
10
  const { withRetry, withTimeout } = require('./json');
9
11
  const { stubComplete, stubEnabled } = require('./stub');
10
12
  const {
@@ -126,7 +128,10 @@ const CODEX_OVERHEAD_NOTE =
126
128
  //
127
129
  // `temperature` is honoured ONLY on api surfaces; cli surfaces control sampling
128
130
  // themselves, so temperature is ignored there and the receipt records that fact.
129
- async function complete({ system, prompt, model, maxTokens = 1024, timeoutMs = null, temperature = undefined }) {
131
+ // `trusted` (spec 022) selects the SAME-USER legacy spawn on the two CLI lanes.
132
+ // It is false unless a caller says otherwise; bin/driftproof exposes it as
133
+ // --trusted-skill, for skills the operator authored. Api lanes ignore it.
134
+ async function complete({ system, prompt, model, maxTokens = 1024, timeoutMs = null, temperature = undefined, trusted = false }) {
130
135
  const surface = surfaceForModel(model);
131
136
  const provider = providerForSurface(surface);
132
137
  // Offline stub surface: canned completion, zero model calls. The receipt still
@@ -149,9 +154,9 @@ async function complete({ system, prompt, model, maxTokens = 1024, timeoutMs = n
149
154
  const laneRunner = () => {
150
155
  switch (surface) {
151
156
  case 'api': return completeAnthropicApi({ system, prompt, model, maxTokens, temperature });
152
- case 'claude-cli': return completeClaudeCli({ system, prompt, model, timeoutMs: effTimeout });
157
+ case 'claude-cli': return completeClaudeCli({ system, prompt, model, timeoutMs: effTimeout, trusted });
153
158
  case 'openai-api': return completeOpenaiApi({ system, prompt, model, maxTokens, temperature, timeoutMs: effTimeout });
154
- case 'openai-cli': return completeCodexCli({ system, prompt, model, timeoutMs: effTimeout });
159
+ case 'openai-cli': return completeCodexCli({ system, prompt, model, timeoutMs: effTimeout, trusted });
155
160
  default: throw new Error(`unknown surface: ${surface}`);
156
161
  }
157
162
  };
@@ -200,47 +205,182 @@ async function completeAnthropicApi({ system, prompt, model, maxTokens, temperat
200
205
  return { text, usage: parseAnthropicApiUsage(resp.usage) };
201
206
  }
202
207
 
203
- // ── anthropic/cli (claude -p) ──────────────────────────────────────────────────
204
- function completeClaudeCli({ system, prompt, model, timeoutMs }) {
205
- return new Promise((resolve, reject) => {
206
- // Strip ANTHROPIC_API_KEY so the CLI uses the subscription session, not the
207
- // metered API key. Everything else in the env is preserved.
208
+ // ── isolation: the eval-user hop (spec 022) ───────────────────────────────────
209
+ //
210
+ // Every local-CLI spawn runs as a DEDICATED UNPRIVILEGED UNIX USER by default.
211
+ // The third-party SKILL.md a run evaluates is instructions to an agent with
212
+ // tools, and until spec 022 that agent ran as the operator: same uid, whole
213
+ // environment block, operator's cwd, operator's CLI configuration. The
214
+ // 2026-09-03 security audit (findings A2, N5) confirmed a live API token in the
215
+ // inherited environment and this host's SSH keys and CLI session tokens
216
+ // readable by the same uid; this was then reproduced by execution.
217
+ //
218
+ // The boundary is a uid change reached through sudo from an EMPTY environment:
219
+ //
220
+ // /usr/bin/sudo -n -u <eval-user> /usr/bin/env -i HOME=/home/<eval-user>
221
+ // PATH=/home/<eval-user>/.local/bin:/usr/bin:/bin bash -lc '<wrapper>'
222
+ // driftproof-eval-hop <cli> <cli-args...>
223
+ //
224
+ // Operator prerequisite, NOT created here (RUNBOOK.md): the user exists with a
225
+ // mode-700 home and no extra groups, has its own logged-in `claude` and `codex`
226
+ // under ~/.local/bin, and a sudoers rule grants the operator NOPASSWD as that
227
+ // user. The runner refuses loudly when any of that is missing; it never creates
228
+ // a user, edits sudoers, or logs a CLI in.
229
+ //
230
+ // The same-user (legacy) spawn survives ONLY behind `trusted: true`, exposed by
231
+ // bin/driftproof as --trusted-skill, for skills the operator authored.
232
+ const SUDO_BIN = '/usr/bin/sudo';
233
+ const ENV_BIN = '/usr/bin/env';
234
+ const EVAL_USER_DEFAULT = 'driftproof-eval';
235
+ // A user name that can safely follow `sudo -u`: no leading dash, no shell
236
+ // metacharacter, no path separator. Anything else is refused before any spawn.
237
+ const EVAL_USER_RE = /^[a-z_][a-z0-9_-]{0,31}$/;
238
+ // The ONLY names the child environment carries, each CONSTRUCTED from the eval
239
+ // user's name in isolatedEnv() and never copied from process.env. Verified
240
+ // 2026-09-03: both CLIs run through the hop with these two alone; the login
241
+ // shell supplies LANG and TERM itself. Data, frozen, asserted by the gate.
242
+ const ISOLATED_ENV_ALLOWLIST = Object.freeze(['HOME', 'PATH']);
243
+ // $0 of the wrapper, so `ps` shows what the bash process is.
244
+ const HOP_LABEL = 'driftproof-eval-hop';
245
+ // The wrapper runs INSIDE the hop, as the eval user. It takes the CLI as
246
+ // positional parameters ("$@"), so no SKILL.md text is ever shell-interpreted.
247
+ // - the cwd is a fresh per-call directory under the system tmp, created by
248
+ // the eval user (one the parent creates is mode 700 to it);
249
+ // - the CLI runs under setsid in its own process group, stdin preserved
250
+ // (`<&0` defeats the /dev/null a non-interactive bash gives a background job);
251
+ // - TERM/INT/HUP, which sudo relays from the parent on timeout, kill that
252
+ // whole group, remove the directory and exit 143;
253
+ // - a normal exit removes the directory and returns the CLI's status.
254
+ const ISOLATED_WRAPPER = [
255
+ 'd=$(mktemp -d -t driftproof-eval.XXXXXXXX) || exit 97',
256
+ 'cd "$d" || exit 97',
257
+ 'setsid -w "$@" <&0 & pid=$!',
258
+ 'trap \'kill -TERM -- "-$pid" 2>/dev/null; sleep 1; kill -KILL -- "-$pid" 2>/dev/null; cd /; rm -rf "$d"; exit 143\' TERM INT HUP',
259
+ 'wait "$pid"; r=$?',
260
+ 'cd /; rm -rf "$d"; exit $r',
261
+ ].join('; ');
262
+
263
+ // DRIFTPROOF_EVAL_USER selects WHICH user isolates, never WHETHER isolation
264
+ // happens (that is the trusted flag, an argv decision). Empty means default.
265
+ function evalUser() {
266
+ const raw = process.env.DRIFTPROOF_EVAL_USER;
267
+ const u = raw === undefined || raw === '' ? EVAL_USER_DEFAULT : raw;
268
+ if (!EVAL_USER_RE.test(u)) throw new Error(`DRIFTPROOF_EVAL_USER is not a plausible unix user name: ${JSON.stringify(u)}`);
269
+ return u;
270
+ }
271
+
272
+ // The child environment: constructed, not copied. Keys are exactly the allowlist.
273
+ function isolatedEnv(user) {
274
+ const home = `/home/${user}`;
275
+ return { HOME: home, PATH: `${home}/.local/bin:/usr/bin:/bin` };
276
+ }
277
+
278
+ // The spawn plan for one CLI call: { mode, user, file, args, env }. Pure, so the
279
+ // gate can inspect what WOULD be spawned. Untrusted → the hop; trusted → the
280
+ // legacy same-user spawn exactly as it was (env inherited; the claude lane still
281
+ // strips ANTHROPIC_API_KEY so the CLI uses the subscription, not the metered key).
282
+ function buildSpawnPlan({ bin, args, trusted = false }) {
283
+ if (trusted) {
208
284
  const env = { ...process.env };
209
- delete env.ANTHROPIC_API_KEY;
210
-
211
- // v0.4: `--output-format json` returns ONE JSON object carrying both the
212
- // final text (`result`) and the token usage (`usage`) — the default text
213
- // output carries no usage at all, which is why usage was previously null on
214
- // this lane. The text is read from the parsed object; if the CLI ever emits
215
- // something unparseable we fall back to the raw stdout so a run degrades to
216
- // the old behaviour (text, no usage) rather than failing.
217
- const args = ['-p', '--output-format', 'json', '--model', resolveModel(model)];
218
- if (system) args.push('--append-system-prompt', system);
219
-
220
- const child = spawn('claude', args, { env, stdio: ['pipe', 'pipe', 'pipe'] });
221
- let out = '';
222
- let err = '';
223
- const killer = setTimeout(() => { child.kill('SIGKILL'); }, timeoutMs + 5000);
224
- child.stdout.on('data', (d) => { out += d; });
225
- child.stderr.on('data', (d) => { err += d; });
226
- child.on('error', (e) => {
227
- clearTimeout(killer);
228
- if (e.code === 'ENOENT') reject(new Error("the anthropic/cli surface requires the `claude` CLI on PATH. Install it or set CLAUDE_PROVIDER=api."));
229
- else reject(e);
230
- });
231
- child.on('close', (code) => {
232
- clearTimeout(killer);
233
- if (code !== 0) return reject(new Error(`claude CLI exited ${code}: ${err.slice(0, 400)}`));
234
- const parsed = parseClaudeCliJson(out);
235
- if (!parsed) return resolve({ text: out.trim(), usage: null });
236
- if (parsed.isError) return reject(new Error(`claude CLI reported is_error: ${String(parsed.text || '').slice(0, 300)}`));
237
- resolve({ text: String(parsed.text || '').trim(), usage: parsed.usage });
238
- });
239
- child.stdin.write(prompt);
285
+ if (bin === 'claude') delete env.ANTHROPIC_API_KEY;
286
+ return { mode: 'same-user', user: null, file: bin, args: [...args], env };
287
+ }
288
+ const user = evalUser();
289
+ const childEnv = isolatedEnv(user);
290
+ return {
291
+ mode: 'isolated',
292
+ user,
293
+ file: SUDO_BIN,
294
+ args: ['-n', '-u', user, ENV_BIN, '-i', ...ISOLATED_ENV_ALLOWLIST.map((k) => `${k}=${childEnv[k]}`), 'bash', '-lc', ISOLATED_WRAPPER, HOP_LABEL, bin, ...args],
295
+ // The sudo front process gets NOTHING from this process. sudo resolves
296
+ // /usr/bin/env itself; env -i then starts the child from empty.
297
+ env: {},
298
+ };
299
+ }
300
+
301
+ // Run a plan: `input` to stdin, stdout/stderr collected, settled on close.
302
+ // The timeout signal differs by mode. The legacy child gets SIGKILL as it
303
+ // always did. The hop gets SIGTERM: it is the signal the operator's uid can
304
+ // deliver to a root-owned sudo, and the one sudo relays to the wrapper's trap.
305
+ // SIGKILL would neither be permitted nor relayed, and a wrapper without the
306
+ // trap left the CLI orphaned with the stdout pipe open, so close never fired.
307
+ function runPlan(plan, { input = null, timeoutMs = 300000, graceMs = 5000 } = {}) {
308
+ return new Promise((resolve, reject) => {
309
+ let child;
310
+ try { child = cp.spawn(plan.file, plan.args, { env: plan.env, stdio: ['pipe', 'pipe', 'pipe'] }); }
311
+ catch (e) { return reject(e); }
312
+ let stdout = '';
313
+ let stderr = '';
314
+ let timedOut = false;
315
+ const killer = setTimeout(() => {
316
+ timedOut = true;
317
+ child.kill(plan.mode === 'isolated' ? 'SIGTERM' : 'SIGKILL');
318
+ }, timeoutMs + graceMs);
319
+ child.stdout.on('data', (d) => { stdout += d; });
320
+ child.stderr.on('data', (d) => { stderr += d; });
321
+ child.on('error', (e) => { clearTimeout(killer); reject(e); });
322
+ child.on('close', (code, signal) => { clearTimeout(killer); resolve({ code, signal, stdout, stderr, timedOut }); });
323
+ child.stdin.on('error', () => { /* EPIPE if the CLI exits before reading all of stdin */ });
324
+ if (input != null) child.stdin.write(input);
240
325
  child.stdin.end();
241
326
  });
242
327
  }
243
328
 
329
+ // The hop with an arbitrary command in place of the CLI, for the gate: the same
330
+ // plan builder and the same runner, never trusted. No provider is reached.
331
+ function runIsolated(cmdArgv, opts = {}) {
332
+ const [bin, ...args] = cmdArgv;
333
+ return runPlan(buildSpawnPlan({ bin, args, trusted: false }), { graceMs: 0, ...opts });
334
+ }
335
+
336
+ // A hop failure, said in terms of what to fix. Returns null when the failure is
337
+ // the CLI's own (exit status from inside the hop), which the lane reports as before.
338
+ function hopFailure(bin, r) {
339
+ const err = String(r.stderr || '').trim();
340
+ if (r.code === 127) {
341
+ return `the \`${bin}\` CLI is not on the eval user's PATH inside the isolated hop (${err.slice(0, 200)}). `
342
+ + 'Install and log it in as that user (RUNBOOK.md), or pass --trusted-skill for a skill you authored.';
343
+ }
344
+ const sudoLine = err.split('\n').find((l) => /^sudo:/.test(l));
345
+ if (sudoLine) {
346
+ return `the isolated eval-user hop failed before reaching \`${bin}\`: ${sudoLine}. The eval user and its NOPASSWD sudoers `
347
+ + 'rule are an operator prerequisite (RUNBOOK.md); pass --trusted-skill only for a skill you authored.';
348
+ }
349
+ return null;
350
+ }
351
+
352
+ // ── anthropic/cli (claude -p) ──────────────────────────────────────────────────
353
+ async function completeClaudeCli({ system, prompt, model, timeoutMs, trusted = false }) {
354
+ // v0.4: `--output-format json` returns ONE JSON object carrying both the
355
+ // final text (`result`) and the token usage (`usage`) — the default text
356
+ // output carries no usage at all, which is why usage was previously null on
357
+ // this lane. The text is read from the parsed object; if the CLI ever emits
358
+ // something unparseable we fall back to the raw stdout so a run degrades to
359
+ // the old behaviour (text, no usage) rather than failing.
360
+ const args = ['-p', '--output-format', 'json', '--model', resolveModel(model)];
361
+ if (system) args.push('--append-system-prompt', system);
362
+ const plan = buildSpawnPlan({ bin: 'claude', args, trusted });
363
+ let r;
364
+ try {
365
+ r = await runPlan(plan, { input: prompt, timeoutMs });
366
+ } catch (e) {
367
+ if (e.code === 'ENOENT') {
368
+ throw new Error(plan.mode === 'isolated'
369
+ ? `the isolated hop requires ${SUDO_BIN}`
370
+ : "the anthropic/cli surface requires the `claude` CLI on PATH. Install it or set CLAUDE_PROVIDER=api.");
371
+ }
372
+ throw e;
373
+ }
374
+ if (r.code !== 0) {
375
+ const hop = plan.mode === 'isolated' ? hopFailure('claude', r) : null;
376
+ throw new Error(hop || `claude CLI exited ${r.code}: ${r.stderr.slice(0, 400)}`);
377
+ }
378
+ const parsed = parseClaudeCliJson(r.stdout);
379
+ if (!parsed) return { text: r.stdout.trim(), usage: null };
380
+ if (parsed.isError) throw new Error(`claude CLI reported is_error: ${String(parsed.text || '').slice(0, 300)}`);
381
+ return { text: String(parsed.text || '').trim(), usage: parsed.usage };
382
+ }
383
+
244
384
  // ── openai/api (Chat Completions-compatible) ───────────────────────────────────
245
385
  // Generic OpenAI-compatible client using global fetch (Node ≥ 18). The base_url
246
386
  // is configurable — env OPENAI_BASE_URL wins, else the registry's provider
@@ -294,7 +434,7 @@ function openaiProviderConfig() {
294
434
 
295
435
  // ── openai/cli (codex exec — ChatGPT subscription) ─────────────────────────────
296
436
  // Invocation template (from the verified recon, reference_codex_cli):
297
- // codex exec --json -s read-only --skip-git-repo-check --ephemeral -o <tmp> [-m <model>] "PROMPT"
437
+ // codex exec --json -s read-only --skip-git-repo-check --ephemeral [-m <model>] -
298
438
  // Honoured facts:
299
439
  // - `-a/--ask-for-approval` is INVALID on `codex exec` — it is NEVER passed;
300
440
  // exec already defaults to approval_policy "never".
@@ -305,6 +445,12 @@ function openaiProviderConfig() {
305
445
  // - `codex exec` has no system-prompt flag, so a system prompt (the SKILL.md in
306
446
  // with_skill mode) is folded into the prompt text.
307
447
  // - stderr emits a harmless "Reading additional input from stdin…" notice.
448
+ // - `-o/--output-last-message` is NEVER passed (spec 022 AC-9, approval F-1).
449
+ // The final message is taken from the `--json` stream on the stdout pipe
450
+ // this process owns. A `-o` path is a file the child writes and the parent
451
+ // reads back; isolated, the child is the eval user running third-party
452
+ // instructions, and a symlink planted at that path would have made this
453
+ // process read an operator-owned file into the generation.
308
454
  const CODEX_EXEC_ARGS = ['exec', '--json', '-s', 'read-only', '--skip-git-repo-check', '--ephemeral'];
309
455
 
310
456
  function codexAuthPresent(homeDir) {
@@ -315,9 +461,9 @@ function codexAuthPresent(homeDir) {
315
461
  }
316
462
 
317
463
  // Build the exact argv for a codex exec call. Exposed for the gate to assert the
318
- // template (and that `-a` never appears). `outFile` receives the final message.
319
- function buildCodexArgs({ model, outFile }) {
320
- const args = [...CODEX_EXEC_ARGS, '-o', outFile];
464
+ // template (that `-a` never appears, and neither does `-o`).
465
+ function buildCodexArgs({ model } = {}) {
466
+ const args = [...CODEX_EXEC_ARGS];
321
467
  if (model) args.push('-m', resolveModel(model));
322
468
  return args;
323
469
  }
@@ -326,66 +472,58 @@ function buildCodexArgs({ model, outFile }) {
326
472
  // means "read the prompt from stdin". The prompt is NEVER a positional argv —
327
473
  // real SKILL.md files start with `---` (YAML frontmatter) and codex rejects a
328
474
  // positional beginning with `--`. Exposed so the gate can lock this in.
329
- function codexFinalArgs({ model, outFile }) {
330
- return [...buildCodexArgs({ model, outFile }), '-'];
475
+ function codexFinalArgs({ model } = {}) {
476
+ return [...buildCodexArgs({ model }), '-'];
331
477
  }
332
478
 
333
- function completeCodexCli({ system, prompt, model, timeoutMs }) {
334
- return new Promise((resolve, reject) => {
335
- if (!codexAuthPresent()) {
336
- return reject(new Error('the openai/cli surface requires codex auth (~/.codex/auth.json). Run `codex login` (or `codex login --device-auth` on a headless box).'));
479
+ async function completeCodexCli({ system, prompt, model, timeoutMs, trusted = false }) {
480
+ // The auth presence check is a PARENT-side read of ~/.codex/auth.json, which
481
+ // only means something on the same-user path; the eval user's home is not
482
+ // readable from here, and a missing login there surfaces as codex's own error.
483
+ if (trusted && !codexAuthPresent()) {
484
+ throw new Error('the openai/cli surface requires codex auth (~/.codex/auth.json). Run `codex login` (or `codex login --device-auth` on a headless box).');
485
+ }
486
+ // codex exec has no system-prompt flag → fold the system prompt into the
487
+ // prompt text (this is how the SKILL.md reaches the model in with_skill mode).
488
+ const fullPrompt = system ? `${system}\n\n---\n\n${prompt}` : prompt;
489
+ // The prompt is delivered on STDIN, not as a positional argv: real SKILL.md
490
+ // files begin with `---` (YAML frontmatter), and a positional argument that
491
+ // starts with `--` is rejected by codex's arg parser ("unexpected argument
492
+ // '---'"). `-` as the positional tells `codex exec` to read instructions from
493
+ // stdin (per its --help), which is content-agnostic.
494
+ const plan = buildSpawnPlan({ bin: 'codex', args: codexFinalArgs({ model }), trusted });
495
+ // stdout is CAPTURED. `--json` streams JSONL events there: the last
496
+ // `item.completed`/`agent_message` carries the final text and the terminal
497
+ // `turn.completed` event carries this call's token usage — the only place
498
+ // codex reports it. Nothing is read from the filesystem after the call, on
499
+ // either mode: the stdout pipe is the one channel only the child we spawned
500
+ // can write to, and the eval user cannot make this process read a file
501
+ // through it (spec 022 AC-9).
502
+ let r;
503
+ try {
504
+ r = await runPlan(plan, { input: fullPrompt, timeoutMs });
505
+ } catch (e) {
506
+ if (e.code === 'ENOENT') {
507
+ throw new Error(plan.mode === 'isolated'
508
+ ? `the isolated hop requires ${SUDO_BIN}`
509
+ : 'the openai/cli surface requires the `codex` CLI on PATH (npm i -g @openai/codex).');
337
510
  }
338
- // codex exec has no system-prompt flag → fold the system prompt into the
339
- // prompt text (this is how the SKILL.md reaches the model in with_skill mode).
340
- const fullPrompt = system ? `${system}\n\n---\n\n${prompt}` : prompt;
341
- const outFile = path.join(os.tmpdir(), `driftproof-codex-${process.pid}-${Date.now()}-${Math.floor(codexCounter())}.txt`);
342
- // The prompt is delivered on STDIN, not as a positional argv: real SKILL.md
343
- // files begin with `---` (YAML frontmatter), and a positional argument that
344
- // starts with `--` is rejected by codex's arg parser ("unexpected argument
345
- // '---'"). `-` as the positional tells `codex exec` to read instructions from
346
- // stdin (per its --help), which is content-agnostic.
347
- const args = codexFinalArgs({ model, outFile });
348
-
349
- // v0.4: stdout is CAPTURED (it was 'ignore'). `--json` streams JSONL events
350
- // there, and the terminal `turn.completed` event carries this call's token
351
- // usage — the only place codex reports it. The final message still comes from
352
- // the -o file (cleaner than scraping the stream); the JSONL is read for usage.
353
- const child = spawn('codex', args, { env: { ...process.env }, stdio: ['pipe', 'pipe', 'pipe'] });
354
- let err = '';
355
- let jsonl = '';
356
- const killer = setTimeout(() => { child.kill('SIGKILL'); }, timeoutMs + 5000);
357
- child.stdout.on('data', (d) => { jsonl += d; });
358
- child.stderr.on('data', (d) => { err += d; });
359
- child.on('error', (e) => {
360
- clearTimeout(killer);
361
- if (e.code === 'ENOENT') reject(new Error('the openai/cli surface requires the `codex` CLI on PATH (npm i -g @openai/codex).'));
362
- else reject(e);
363
- });
364
- child.on('close', (code) => {
365
- clearTimeout(killer);
366
- let text = '';
367
- try { text = fs.readFileSync(outFile, 'utf8'); } catch (_e) { /* fall through */ }
368
- try { fs.unlinkSync(outFile); } catch (_e) { /* best effort */ }
369
- if (code !== 0) return reject(new Error(`codex exec exited ${code}: ${String(err).slice(0, 400)}`));
370
- const ev = parseCodexJsonl(jsonl);
371
- // Prefer the -o file; fall back to the stream's agent_message if it is empty.
372
- const finalText = String(text || ev.text || '').trim();
373
- resolve({ text: finalText, usage: ev.usage });
374
- });
375
- child.stdin.on('error', () => { /* ignore EPIPE if codex exits before reading all input */ });
376
- child.stdin.write(fullPrompt);
377
- child.stdin.end();
378
- });
511
+ throw e;
512
+ }
513
+ if (r.code !== 0) {
514
+ const hop = plan.mode === 'isolated' ? hopFailure('codex', r) : null;
515
+ throw new Error(hop || `codex exec exited ${r.code}: ${String(r.stderr).slice(0, 400)}`);
516
+ }
517
+ const ev = parseCodexJsonl(r.stdout);
518
+ return { text: String(ev.text || '').trim(), usage: ev.usage };
379
519
  }
380
520
 
381
- // A monotonically-increasing counter for temp-file uniqueness that does not use
382
- // Math.random (kept deterministic-friendly for any harness that forbids it).
383
- let _codexCounter = 0;
384
- function codexCounter() { _codexCounter += 1; return _codexCounter; }
385
-
386
521
  module.exports = {
387
522
  complete, resolveModel, surfaceLabel, providerName, MODEL_ALIASES,
388
523
  inferProvider, surfaceForModel, providerForSurface,
389
524
  isMeteredSurface, isSubscriptionSurface, retryPolicyForSurface,
390
525
  CODEX_OVERHEAD_NOTE, CODEX_EXEC_ARGS, buildCodexArgs, codexFinalArgs, codexAuthPresent,
526
+ // spec 022 — the eval-user hop
527
+ SUDO_BIN, EVAL_USER_DEFAULT, ISOLATED_ENV_ALLOWLIST, ISOLATED_WRAPPER, HOP_LABEL,
528
+ evalUser, isolatedEnv, buildSpawnPlan, runIsolated,
391
529
  };
package/lib/run.js CHANGED
@@ -51,7 +51,7 @@ function projectCalls(caseCount, samples, draws = 1) {
51
51
  // Ask the target model to perform one eval case. `withSkill` decides whether the
52
52
  // SKILL.md is prepended as a system prompt (the whole point: measure the skill's
53
53
  // marginal effect vs a bare baseline).
54
- async function runCase({ skillMd, caseObj, model, withSkill, timeoutMs }) {
54
+ async function runCase({ skillMd, caseObj, model, withSkill, timeoutMs, trusted = false }) {
55
55
  // Test seam (gate only): force a persistent timeout for a named case id so the
56
56
  // failed_timeout path is exercised deterministically without any live call.
57
57
  if (process.env.DRIFTPROOF_TEST_TIMEOUT_CASEID && process.env.DRIFTPROOF_TEST_TIMEOUT_CASEID === caseObj.id) {
@@ -60,7 +60,7 @@ async function runCase({ skillMd, caseObj, model, withSkill, timeoutMs }) {
60
60
  throw e;
61
61
  }
62
62
  const system = withSkill ? skillMd : undefined;
63
- const { text, usage, wall_ms, attempts } = await complete({ system, prompt: caseObj.prompt, model, maxTokens: 1024, timeoutMs });
63
+ const { text, usage, wall_ms, attempts } = await complete({ system, prompt: caseObj.prompt, model, maxTokens: 1024, timeoutMs, trusted });
64
64
  return { text, usage, wall_ms, attempts };
65
65
  }
66
66
 
@@ -78,8 +78,8 @@ function outcomeFor(mean, stddev, threshold) {
78
78
  // `generationHash` binds the graded case to the exact generation text (v0.3).
79
79
  // Returns { caseResult, sampleTexts } — sampleTexts is transient (retained only
80
80
  // under --keep-transcripts; never part of the receipt).
81
- async function judgeCase({ caseObj, response, generationHash, judgeModel, mode, timeoutMs, samples }) {
82
- const g = await gradeSamples({ task: caseObj.prompt, response, rubric: caseObj.rubric, model: judgeModel, samples, timeoutMs });
81
+ async function judgeCase({ caseObj, response, generationHash, judgeModel, mode, timeoutMs, samples, trusted = false }) {
82
+ const g = await gradeSamples({ task: caseObj.prompt, response, rubric: caseObj.rubric, model: judgeModel, samples, timeoutMs, trusted });
83
83
  const outcome = outcomeFor(g.mean, g.stddev, caseObj.pass_threshold);
84
84
  const caseResult = {
85
85
  id: caseObj.id,
@@ -127,7 +127,7 @@ async function mapPool(items, concurrency, fn) {
127
127
  // a sealed receipt. Enforces a hard call cap; every model+judge call counts.
128
128
  //
129
129
  // opts: { maxCases, maxCalls, samples, judgeModel, timeoutMs, concurrency,
130
- // onProgress, budget, keepTranscripts, nowIso }
130
+ // onProgress, budget, keepTranscripts, nowIso, trusted }
131
131
  // budget — optional BudgetTracker; accumulates estimated per-call
132
132
  // USD as the run proceeds and hard-stops at 1.25× the cap.
133
133
  // keepTranscripts — when true, the run records transcripts:"retained-local"
@@ -177,6 +177,10 @@ async function runSkillOnModel({ skill, model, opts = {} }) {
177
177
  const onProgress = opts.onProgress || (() => {});
178
178
  const budget = opts.budget || null;
179
179
  const keepTranscripts = !!opts.keepTranscripts;
180
+ // spec 022: the SAME-USER legacy spawn is reachable only when a caller says
181
+ // `trusted: true` (bin/driftproof --trusted-skill). Every other caller of this
182
+ // function, the report scripts and the release watcher included, isolates.
183
+ const trusted = !!opts.trusted;
180
184
 
181
185
  let cases = skill.suite.cases;
182
186
  if (opts.maxCases && cases.length > opts.maxCases) cases = cases.slice(0, opts.maxCases);
@@ -222,12 +226,12 @@ async function runSkillOnModel({ skill, model, opts = {} }) {
222
226
  const drawIndex = draws.length;
223
227
  try {
224
228
  onProgress({ case: c.id, mode, phase: 'generate', draw: drawIndex });
225
- const gen = await runCase({ skillMd: skill.skillMd, caseObj: c, model: modelId, withSkill, timeoutMs: ctGen });
229
+ const gen = await runCase({ skillMd: skill.skillMd, caseObj: c, model: modelId, withSkill, timeoutMs: ctGen, trusted });
226
230
  calls += 1;
227
231
  if (budget) budget.add((gen.attempts || 1) * perCallCostUSD(modelId, genKind(withSkill)));
228
232
  const generationHash = sha256(String(gen.text || ''));
229
233
  onProgress({ case: c.id, mode, phase: 'judge', samples, draw: drawIndex });
230
- const jr = await judgeCase({ caseObj: c, response: gen.text, generationHash, judgeModel, mode, timeoutMs: ctJudge, samples });
234
+ const jr = await judgeCase({ caseObj: c, response: gen.text, generationHash, judgeModel, mode, timeoutMs: ctJudge, samples, trusted });
231
235
  calls += samples;
232
236
  if (budget) budget.add((jr.attempts || samples) * perCallCostUSD(judgeModel, 'judge'));
233
237
  const draw = {
package/lib/verdict.js CHANGED
@@ -1,6 +1,7 @@
1
1
  // SPDX-License-Identifier: Apache-2.0
2
2
  'use strict';
3
3
 
4
+ const crypto = require('crypto');
4
5
  const { EFFECT_FLOOR } = require('../config');
5
6
 
6
7
  // Single-receipt verdict + shields.io badge.
@@ -72,14 +73,39 @@ function badgeEndpoint(receipt, { label = 'driftproof' } = {}) {
72
73
  }
73
74
 
74
75
  // Lines suitable for appending to a GitHub Actions $GITHUB_OUTPUT file.
76
+ //
77
+ // Spec 023 (audit A6). Each entry is written in GitHub's multiline form,
78
+ //
79
+ // name<<ghadelim_<32 hex>
80
+ // value
81
+ // ghadelim_<32 hex>
82
+ //
83
+ // with a delimiter drawn fresh from crypto.randomBytes for EVERY entry, so no
84
+ // value can close a block early; and any value carrying \r or \n is refused
85
+ // with a throw before anything is written. The bare `name=value` form this used
86
+ // to emit let a model_id with a newline in it (any string a receipt carries; a
87
+ // .driftproofrc in a skill directory sets it) append a second, attacker-chosen
88
+ // entry, and that entry was then interpolated into the enforce step's shell.
89
+ // Refusing is the honest writer: a line break in a model id is not a value
90
+ // anybody meant to publish. Entries are joined by '\n' with no trailing newline
91
+ // so the caller's console.log stays the one that terminates the text.
92
+ function githubOutputEntry(name, value) {
93
+ const val = String(value);
94
+ if (/[\r\n]/.test(val)) {
95
+ throw new Error(`refusing to write $GITHUB_OUTPUT: the value of "${name}" contains a line break (${JSON.stringify(val)})`);
96
+ }
97
+ const delim = `ghadelim_${crypto.randomBytes(16).toString('hex')}`;
98
+ return `${name}<<${delim}\n${val}\n${delim}`;
99
+ }
100
+
75
101
  function githubOutputLines(receipt) {
76
102
  const v = verdictFromReceipt(receipt);
77
103
  return [
78
- `verdict=${v.verdict}`,
79
- `delta=${v.delta}`,
80
- `message=${v.message}`,
81
- `color=${v.color}`,
82
- ].join('\n');
104
+ ['verdict', v.verdict],
105
+ ['delta', v.delta],
106
+ ['message', v.message],
107
+ ['color', v.color],
108
+ ].map(([k, val]) => githubOutputEntry(k, val)).join('\n');
83
109
  }
84
110
 
85
- module.exports = { verdictFromReceipt, badgeEndpoint, githubOutputLines, shortModel, VERDICTS };
111
+ module.exports = { verdictFromReceipt, badgeEndpoint, githubOutputLines, githubOutputEntry, shortModel, VERDICTS };
package/package.json CHANGED
@@ -1,14 +1,16 @@
1
1
  {
2
2
  "name": "driftproof",
3
- "version": "0.7.2",
3
+ "version": "0.8.1",
4
4
  "description": "A dated proof that this skill, this hash, this model, still helps: run a skill's eval suite with and without the skill across model versions, emit hash-verified dated receipts, and diff receipts into drift reports.",
5
5
  "license": "Apache-2.0",
6
6
  "keywords": [
7
7
  "agent-skills",
8
- "evals",
9
- "llm",
8
+ "skill-md",
9
+ "skill-eval",
10
+ "llm-evaluation",
11
+ "claude-code",
10
12
  "drift",
11
- "verification"
13
+ "receipts"
12
14
  ],
13
15
  "homepage": "https://driftproofhq.com",
14
16
  "repository": {
@@ -41,5 +43,8 @@
41
43
  },
42
44
  "engines": {
43
45
  "node": ">=22"
46
+ },
47
+ "devDependencies": {
48
+ "html-validate": "^11.11.0"
44
49
  }
45
50
  }