tewip-ai 0.24.0 → 0.26.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -444,6 +444,115 @@ Every change is small (1 to 16 lines, median 4) and most are what the developers
444
444
  **No wrong fix was offered**: anything that could not be proved was thrown away and the file put
445
445
  back byte for byte.
446
446
 
447
+ ## One bundle a release manager can read (Phase 15)
448
+
449
+ ```bash
450
+ tewip attest --out attestation.md
451
+ ```
452
+
453
+ Everything Tewip knows about a build, in one file: which tests ran, what each failure was, which
454
+ rule or which model decided it, **what it refused to decide**, and what it never looked at. The
455
+ refusals are the point - a tool that never says "I could not tell" has nothing worth attesting,
456
+ because every line it writes might be a guess.
457
+
458
+ The bundle carries a digest, and an Ed25519 signature when you give it a key:
459
+
460
+ ```bash
461
+ TEWIP_ATTEST_KEY=./private.pem tewip attest # sign it
462
+ tewip attest --verify attestation.json --key public.pem # check it
463
+ ```
464
+
465
+ A digest is not a signature and Tewip says so: without a key it tells you the bundle is undamaged,
466
+ not that nobody edited it. Editing a claim out of a signed bundle makes verification fail and exit
467
+ non-zero. Every CI run writes one of these automatically.
468
+
469
+ ## Is this incident covered by a test? (Phase 16)
470
+
471
+ ```bash
472
+ tewip incident sentry-issue.json
473
+ ```
474
+
475
+ Reads a Sentry payload, an OpenTelemetry span or a plain stack trace, finds the test that should
476
+ have caught it, and answers with one of four verdicts: **reproduced**, **passed** (the test ran and
477
+ did not catch it, which is worse news), **failed differently**, or **never ran**. It does not say
478
+ "probably covered".
479
+
480
+ ## What did we decide about this last time? (Phase 17)
481
+
482
+ ```bash
483
+ tewip recall "the login test that keeps flaking"
484
+ ```
485
+
486
+ Every correction your team ever gave Tewip is kept. Ask, and it answers with your own decision from
487
+ then, and says it was yours - not its. It never acts on a correction by itself.
488
+
489
+ ## From the ticket to the run (Phase 18)
490
+
491
+ ```bash
492
+ tewip trace --only "SHOP-12"
493
+ ```
494
+
495
+ Joins the test plans Tewip wrote to the tests that cover them and the runs that attempted them. A
496
+ scenario counts as covered only when a real test can be named, and passing only when a real run
497
+ attempted it; everything else says what is missing. No model, no cost.
498
+
499
+ ## Could these tests ever fail? (Phase 22)
500
+
501
+ ```bash
502
+ tewip vouch tests/ # or a file, or the whole project
503
+ tewip vouch tests/ --strict # exits non-zero, for a merge gate
504
+ ```
505
+
506
+ Most tests in a pull request are now written by an agent, and the question that decides whether they
507
+ are worth anything goes unasked: **could this test ever fail?** Coverage counts a test that checks
508
+ nothing. CI goes green on it. A reviewer skims something plausible and approves it.
509
+
510
+ `vouch` reads tests somebody else wrote and answers per test, quoting the line it rests on. It finds
511
+ tests that leave an expectation unfinished (`expect(value);` with no matcher - it reads exactly like a
512
+ check), tests that compare a value with itself, tests that catch and discard their own failure, tests
513
+ that never run, and tests that compare no value at all. No model, no cost.
514
+
515
+ It is also an MCP tool, so a coding agent can ask about the tests it just wrote before claiming a
516
+ change is covered.
517
+
518
+ **Measured on 5,448 tests in 8 public projects across 5 languages** (express, axios, chalk, requests,
519
+ click, gorilla/mux, cobra, gson): 19 tests were called unable to fail, and **all 19 were correct** when
520
+ read against the files - one unconditional skip in psf/requests, eighteen `@Ignore` methods in gson.
521
+
522
+ Two things it refuses to do, which is why the list is worth reading:
523
+
524
+ - **It will not guess.** A test whose checking happens inside a helper it cannot follow is reported as
525
+ undecided, by name, never as worthless. A wrong accusation about somebody's test costs more than a
526
+ missed one.
527
+ - **It will not overclaim.** A test with no assertion still fails when the code throws - in axios, a
528
+ test that compiles a TypeScript fixture and runs it asserts nothing *on purpose*. Those are reported
529
+ separately as "compares no value", not as broken, and they do not trip `--strict`.
530
+
531
+ Honest limit, also printed at the bottom of every report: its **recall is unmeasured**. A clean report
532
+ means nothing it found was wrong, not that every test in the file is worth something.
533
+
534
+ ## A goal instead of a verb (Phases 19 and 21)
535
+
536
+ ```bash
537
+ tewip pursue "keep the checkout flow green"
538
+ tewip pursue "keep the checkout flow green" --act
539
+ ```
540
+
541
+ Give Tewip a goal and it says what it would do next, and why, from what it can actually see. The
542
+ order is fixed, published and testable - **not chosen by a model**: a real bug first, then the
543
+ environment, then flakiness, then anything failing it cannot explain, then a coverage gap, and only
544
+ then is the goal worth attesting. Deciding the same state twice gives the same answer.
545
+
546
+ With `--act` it carries the decision out, and this is where the refusals matter most:
547
+
548
+ - It **never does the work itself.** Each action is an existing command that already proves what it
549
+ changes - `write` keeps a test only when it passes *and* fails when broken.
550
+ - **Two actions are on the allowlist**: write a missing test, and attest a green goal. Everything
551
+ else is refused by name. It will not work around a real bug.
552
+ - **One action, then stop.** There is no unattended loop.
553
+ - Everything it did or refused goes to `.tewip/actions.jsonl`, with the run that justified it.
554
+ - It **will not repeat itself**: the same attempt is refused until a new run gives it new evidence.
555
+
447
556
  ## Where it has been tried
448
557
 
449
558
  | Project | How | Result |
package/dist/cli/help.js CHANGED
@@ -105,8 +105,19 @@ Usage:
105
105
  green" - and it says what it would do next and why, from what
106
106
  it can actually see. The order is fixed and written down: a
107
107
  real bug first, then the environment, then flakiness, then a
108
- coverage gap. It decides; it does not act unless you ask, and
109
- even then only through the gates that prove a change.
108
+ coverage gap. It decides; it does not act unless you ask. With
109
+ --act it carries out the two safe ones - writing a missing test,
110
+ or attesting a green goal - by running the command that already
111
+ proves them. It will not touch a real bug, and it writes what it
112
+ did or refused to .tewip/actions.jsonl.
113
+ tewip vouch [<paths>] [--strict] [--out vouch.md] [--json vouch.json]
114
+ Could these tests ever fail? Reads tests somebody else wrote -
115
+ an agent, a contractor, last year's team - and names the ones
116
+ that check nothing, compare a value with itself, leave an
117
+ expectation unfinished, swallow their own failure, or never run.
118
+ Where the checking is in a helper it cannot follow it says so
119
+ instead of accusing. --strict exits non-zero on a finding, for a
120
+ gate. No model, no cost.
110
121
  tewip trace [--only "<ticket or title>"] [--out trace.md]
111
122
  Is this requirement covered, and is it passing right now? Joins
112
123
  the plans Tewip wrote to the tests that cover them and the runs
package/dist/cli/tewip.js CHANGED
@@ -185,6 +185,12 @@ async function main() {
185
185
  runPursue({ root: process.cwd(), goal, act: args.includes('--act'), name: flag(args, 'name'), out: flag(args, 'out') });
186
186
  return;
187
187
  }
188
+ case 'vouch': {
189
+ const paths = args.filter((a, i) => !a.startsWith('--') && !['--out', '--json'].includes(args[i - 1] ?? ''));
190
+ const { runVouch } = await import('../engine/vouch/run-vouch.js');
191
+ runVouch({ root: process.cwd(), paths: paths.length > 0 ? paths : ['.'], out: flag(args, 'out'), json: flag(args, 'json'), strict: args.includes('--strict') });
192
+ return;
193
+ }
188
194
  case 'trace': {
189
195
  const { runTrace } = await import('../engine/trace/trace-command.js');
190
196
  runTrace({ root: process.cwd(), only: flag(args, 'only'), name: flag(args, 'name'), out: flag(args, 'out') });
@@ -46,9 +46,18 @@ function history(root, only) {
46
46
  function whatWasChecked(records) {
47
47
  const checked = [];
48
48
  const notChecked = [];
49
- checked.push('every failing test in the run was read, and each was given a cause or left undecided');
50
- const byRules = records.filter((r) => r.diagnosis?.source === 'rules').length;
51
- const byModel = records.filter((r) => r.diagnosis?.source === 'llm').length;
49
+ // Decided means settled: a rule that looked and could not say is a refusal, and counting it here
50
+ // would have a bundle claim more decisions than it lists claims.
51
+ const settledBy = (source) => records.filter((r) => {
52
+ const d = r.diagnosis;
53
+ return d?.source === source && d.category && d.category !== 'UNKNOWN';
54
+ }).length;
55
+ if (records.length === 0)
56
+ checked.push('every test in the run passed, so there was nothing to diagnose');
57
+ else
58
+ checked.push('every failing test in the run was read, and each was given a cause or left undecided');
59
+ const byRules = settledBy('rules');
60
+ const byModel = settledBy('llm');
52
61
  if (byRules > 0)
53
62
  checked.push(`${byRules} decided by deterministic rules (${ENGINE_VERSION}), which call no model and give the same answer every time`);
54
63
  if (byModel > 0)
@@ -0,0 +1,39 @@
1
+ /**
2
+ * What Tewip may carry out, and with what. Anything absent is refused: reporting a real bug, for
3
+ * instance, is deliberately not an action - it is for a person to read.
4
+ */
5
+ export function planFor(decision) {
6
+ switch (decision.act) {
7
+ case 'cover-gap': {
8
+ // `write` drafts the test, runs it, breaks it on purpose to see it fail, and keeps it only
9
+ // if both hold. The scenario id is in the decision's own next step.
10
+ const id = /write a test for ([\w-]+)/.exec(decision.next)?.[1];
11
+ return id ? { command: ['write', '--only', id], target: `scenario ${id}` } : null;
12
+ }
13
+ case 'attest':
14
+ return { command: ['attest'], target: 'the evidence bundle' };
15
+ default:
16
+ return null;
17
+ }
18
+ }
19
+ /** Why an action was not carried out, in words a person can act on. */
20
+ export const cannotAct = {
21
+ 'look-first': 'there is nothing to act on until a report has been read',
22
+ 'triage-real-bug': 'a real bug is for a person to see, and Tewip will not work around one on its own',
23
+ 'rerun-environment': 'a re-run belongs to whatever runs the tests, not to Tewip',
24
+ 'settle-flaky': 'settling a flaky test changes a test file, which Tewip does only when asked directly with `tewip fix`',
25
+ 'triage-undecided': 'Tewip cannot say why these failed, so it has nothing it could safely do about them',
26
+ 'cover-gap': '',
27
+ attest: '',
28
+ 'stand-down': 'there is nothing here that matches the goal',
29
+ };
30
+ /**
31
+ * Has this exact action already been tried against this exact evidence? A scheduled run that keeps
32
+ * attempting an impossible task is the failure mode that makes people turn an agent off. A new run
33
+ * is new evidence and unblocks it; running again over the same report does not.
34
+ *
35
+ * Refusals are not counted: refusing costs nothing and says nothing about whether it would work.
36
+ */
37
+ export function alreadyTried(records, act, target, runId) {
38
+ return records.find((r) => r.act === act && r.target === target && r.runId === runId && r.outcome !== 'refused') ?? null;
39
+ }
@@ -4,9 +4,11 @@
4
4
  // it chose, and only through the gates that already prove a change before keeping it. There is no
5
5
  // mode in which it loops unattended: an agent that keeps going until it decides to stop is the
6
6
  // thing teams roll back, and nothing here needs it.
7
- import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
7
+ import { spawnSync } from 'node:child_process';
8
+ import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
8
9
  import { dirname, join, resolve } from 'node:path';
9
10
  import { datasetFiles, readJsonl, readTriageRecords } from '../../reporters/store.js';
11
+ import { alreadyTried, cannotAct, planFor } from './act.js';
10
12
  import { decide } from './policy.js';
11
13
  const words = (text) => (text.toLowerCase().match(/[a-z][a-z0-9]{2,}/g) ?? []).filter((w) => !['the', 'and', 'for', 'this', 'that', 'with', 'keep', 'green', 'sprint', 'flow', 'test', 'tests'].includes(w));
12
14
  /** Does this test or scenario belong to the goal? Deterministic, so a person can predict it. */
@@ -53,6 +55,7 @@ export function observe(root, goal, only) {
53
55
  uncovered: uncoveredScenarios(root, wanted, attempted),
54
56
  passing: attempted.length - failing.length,
55
57
  noHistory: runs.length === 0,
58
+ runId: newest?.runId ?? '',
56
59
  };
57
60
  }
58
61
  /** Scenarios of a kept plan that no attempted test can be matched to, worst priority first. */
@@ -108,15 +111,100 @@ export function renderPursuit(goal, seen, decision, acting) {
108
111
  : 'Tewip has not done anything: this is a decision, and the action is yours.', '', 'The order of these decisions is fixed and written down, not chosen by a model: a real bug first,', 'then the environment, then flakiness, then a coverage gap, and only then is a goal worth attesting.');
109
112
  return lines.join('\n');
110
113
  }
114
+ /** The audit file the team owns: every action Tewip took on its own, and every one it refused. */
115
+ const auditPath = (root) => join(root, '.tewip', 'actions.jsonl');
116
+ export function actionsTaken(root) {
117
+ try {
118
+ return readFileSync(auditPath(root), 'utf8')
119
+ .split(String.fromCharCode(10))
120
+ .filter(Boolean)
121
+ .map((l) => JSON.parse(l));
122
+ }
123
+ catch {
124
+ return [];
125
+ }
126
+ }
127
+ function record(root, entry) {
128
+ mkdirSync(dirname(auditPath(root)), { recursive: true });
129
+ appendFileSync(auditPath(root), JSON.stringify(entry) + String.fromCharCode(10), 'utf8');
130
+ }
131
+ /**
132
+ * The Tewip command line belonging to *this* build, found from this file rather than from how the
133
+ * process was started.
134
+ *
135
+ * This is not a detail. The first version used process.argv[1], which is the command line only when
136
+ * a person typed it: under a test runner argv[1] is the test file, so acting re-ran the test file,
137
+ * which acted again, which re-ran it - a fork bomb that froze the machine it was written on. What
138
+ * re-enters Tewip has to be Tewip, established here, or nothing is carried out at all.
139
+ */
140
+ export function cliEntry() {
141
+ const here = import.meta.dirname;
142
+ for (const candidate of [join(here, '..', '..', 'cli', 'tewip.ts'), join(here, '..', '..', 'cli', 'tewip.js')]) {
143
+ if (existsSync(candidate))
144
+ return candidate;
145
+ }
146
+ return null;
147
+ }
148
+ /**
149
+ * Carries out the one action that was decided, by running the command that already proves it.
150
+ * Tewip does not do the work here; it asks the gate to.
151
+ */
152
+ function carryOut(cmd, decision, observed) {
153
+ const plan = planFor(decision);
154
+ const runId = observed.runId ?? '';
155
+ const note = (outcome, detail, target = plan?.target ?? '') => ({
156
+ at: new Date().toISOString(),
157
+ goal: cmd.goal,
158
+ act: decision.act,
159
+ target,
160
+ command: plan ? 'tewip ' + plan.command.join(' ') : '',
161
+ runId,
162
+ outcome,
163
+ detail,
164
+ });
165
+ if (!plan) {
166
+ const why = cannotAct[decision.act] || 'this is not something Tewip carries out on its own';
167
+ record(cmd.root, note('refused', why, decision.act));
168
+ return 'Not carried out: ' + why + '.';
169
+ }
170
+ const tried = alreadyTried(actionsTaken(cmd.root), decision.act, plan.target, runId);
171
+ if (tried) {
172
+ record(cmd.root, note('refused', 'already attempted at ' + tried.at + ', on the same evidence'));
173
+ return ['Not carried out: Tewip already attempted this at ' + tried.at.slice(0, 19).replace('T', ' ') + ' UTC, and nothing has happened since that would make the outcome different.', '', 'That attempt ' + (tried.outcome === 'done' ? 'succeeded' : 'did not finish') + ': ' + tried.detail, '', 'Run ' + tried.command + ' yourself if you want it tried again.'].join(String.fromCharCode(10));
174
+ }
175
+ // Re-enter this same build, so an installed Tewip runs the installed gates and a checkout runs
176
+ // the checkout's - never a different version from the one that decided, and never whatever
177
+ // happens to have started this process.
178
+ const entry = cliEntry();
179
+ if (!entry) {
180
+ record(cmd.root, note('refused', 'the Tewip command line could not be found next to this build, so there is nothing safe to re-enter'));
181
+ return 'Not carried out: Tewip could not find its own command line beside this build, and it will not guess at one. Run ' + 'tewip ' + plan.command.join(' ') + ' yourself.';
182
+ }
183
+ const ran = spawnSync(process.execPath, [entry, ...plan.command], { cwd: cmd.root, encoding: 'utf8', timeout: 20 * 60_000, maxBuffer: 32 * 1024 * 1024 });
184
+ const output = ((ran.stdout ?? '') + (ran.stderr ?? '')).trim();
185
+ const ok = ran.status === 0;
186
+ // What the audit file remembers: the headline of a success, the tail of a failure. A failure's
187
+ // reason is at the end of what it printed; a success's is at the start.
188
+ const lines = output.split(String.fromCharCode(10)).map((l) => l.trim()).filter(Boolean);
189
+ const detail = (ok ? lines[0] ?? 'it printed nothing' : lines.slice(-3).join(' ')).slice(0, 300);
190
+ record(cmd.root, note(ok ? 'done' : 'failed', detail));
191
+ return [
192
+ ok ? 'Carried out: tewip ' + plan.command.join(' ') : 'Tried and did not finish: tewip ' + plan.command.join(' '),
193
+ '',
194
+ output.slice(-2000),
195
+ '',
196
+ ok
197
+ ? 'Anything kept by that command was proved by it; anything unproved was thrown away. Written to .tewip/actions.jsonl.'
198
+ : 'Nothing was kept. Written to .tewip/actions.jsonl.',
199
+ ].join(String.fromCharCode(10));
200
+ }
111
201
  export function runPursue(cmd) {
112
202
  const observed = observe(cmd.root, cmd.goal, cmd.name);
113
203
  const decision = decide(cmd.goal, observed);
114
204
  const report = renderPursuit(cmd.goal, observed, decision, Boolean(cmd.act));
115
205
  console.log(report);
116
- if (cmd.act && decision.changesFiles) {
117
- console.error('\n`--act` is accepted, but Tewip carries out a change only through the command that proves it.');
118
- console.error(`Run: ${decision.next}`);
119
- }
206
+ if (cmd.act)
207
+ console.log(String.fromCharCode(10) + '---' + String.fromCharCode(10) + carryOut(cmd, decision, observed));
120
208
  if (cmd.out) {
121
209
  mkdirSync(dirname(resolve(cmd.out)), { recursive: true });
122
210
  writeFileSync(resolve(cmd.out), `${report}\n`, 'utf8');
@@ -0,0 +1,138 @@
1
+ // What counts as checking something, across the frameworks people actually use.
2
+ //
3
+ // This vocabulary is the whole accuracy of Phase 22, and its width is a lesson already paid for:
4
+ // in Phase 8 the wait fix refused six real Cypress fixes because "asserted" meant `expect(` and
5
+ // Cypress asserts with `.should(`, and refused the commonest shape of JavaScript unit test because
6
+ // `assert.equal` was not on the list either. A missing entry here does not produce a smaller
7
+ // answer, it produces a confident wrong one - Tewip telling somebody their test checks nothing when
8
+ // it checks plenty. So the list is broad, and anything not on it is undecided rather than damning.
9
+ const CHECKS = [
10
+ // JavaScript and TypeScript: Jest, Vitest, Chai, node:test, Playwright, Cypress, Supertest.
11
+ /\bexpect\s*\(/,
12
+ /\bexpect\s*\.\s*(soft|poll)\s*\(/,
13
+ /\.\s*should\s*[.(]/,
14
+ /\bshould\s*\.\s*not\b/,
15
+ /\bassert\b\s*[.(]/,
16
+ /\bassert(Equals?|True|False|Null|NotNull|That|Throws|Match|Contains|Rejects|Fails)\b/i,
17
+ /\.\s*to\s*\.\s*(equal|be|deep|have|include|throw|match|exist)\b/,
18
+ /\btoMatch(Snapshot|InlineSnapshot|ImageSnapshot)\s*\(/,
19
+ /\bexpect(Type|TypeOf)\s*[<(]/,
20
+ // AVA, tap and node-tap put their assertions on the test's own context object. Missing this made
21
+ // Tewip condemn all 59 tests in chalk - a third repeat of the Phase 8 lesson, so the pattern here
22
+ // is deliberately generous about what an assertion on `t` can be called.
23
+ /\bt\s*\.\s*(is|not|deepEqual|notDeepEqual|like|true|false|truthy|falsy|throws(Async)?|notThrows(Async)?|regex|notRegex|snapshot|pass|equal|same|strictSame|ok|notOk|match|type|has|hasStrict|rejects|resolves)\b/,
24
+ // Mocking libraries: verifying a call is a real check.
25
+ /\b(verify|verifyNoMoreInteractions|toHaveBeenCalled)\b/,
26
+ /\bsinon\s*\.\s*assert\b/,
27
+ // Python: pytest, unittest, hypothesis.
28
+ /(^|[^\w.])assert\s+[^=]/m,
29
+ /\bself\s*\.\s*assert[A-Z]/,
30
+ /\bpytest\s*\.\s*(raises|warns|approx|fail)\b/,
31
+ /\bnp\s*\.\s*testing\s*\.\s*assert/,
32
+ /\bassert_(frame|series|index)_equal\b/,
33
+ // Java, Kotlin, C#, Scala.
34
+ /\bAssert(ions)?\s*\./,
35
+ /\bassertThat\s*\(/,
36
+ /\bShould(Be|NotBe|Throw|Contain)\b/,
37
+ /\bFluentActions\b/,
38
+ // Go.
39
+ /\bt\s*\.\s*(Error|Errorf|Fatal|Fatalf|Fail|FailNow)\b/,
40
+ /\b(require|assert)\s*\.\s*[A-Z]\w+\s*\(/,
41
+ // Ruby, PHP, Rust.
42
+ /\bexpect\s*\(.*\)\s*\.\s*to\b/,
43
+ /\b\$this\s*->\s*assert/,
44
+ /\bassert(_eq|_ne|_matches)?!\s*\(/,
45
+ // Robot Framework and other keyword styles: a line that reads as a check.
46
+ /^\s*Should\s+(Be|Not|Contain|Match)\b/im,
47
+ ];
48
+ /** Does this text check anything at all, by any framework's idea of checking? */
49
+ export function checksSomething(text) {
50
+ return CHECKS.some((pattern) => pattern.test(text));
51
+ }
52
+ /**
53
+ * An `expect(...)` that was never finished: no matcher was called on it, so it runs clean and proves
54
+ * nothing. `expect(user).toBe(x)` checks; `expect(user);` does not, and neither does a matcher left
55
+ * as a property - `expect(user).toBeDefined;` is a reference to a function nobody called.
56
+ *
57
+ * Both shapes are ordinary typos that no test run can catch, which is exactly why they survive.
58
+ */
59
+ export function unfinishedExpectations(body) {
60
+ const found = [];
61
+ for (const line of body.split(String.fromCharCode(10))) {
62
+ const text = line.trim();
63
+ if (!/^(await\s+)?expect(\.soft)?\s*\(/.test(text))
64
+ continue;
65
+ const after = afterBalanced(text, text.indexOf('('));
66
+ if (after === null)
67
+ continue; // the call spans more lines; judged as part of the whole body
68
+ const rest = after.trim();
69
+ if (rest === '' || rest === ';')
70
+ found.push(text);
71
+ // A matcher named but not called: `.toBeVisible;` or `.toBeVisible` at the end of a statement.
72
+ else if (/^\.\s*[A-Za-z$_][\w$]*\s*;?$/.test(rest))
73
+ found.push(text);
74
+ }
75
+ return found;
76
+ }
77
+ /** What follows the balanced bracket that opens at `from`, or null when it never closes here. */
78
+ function afterBalanced(text, from) {
79
+ let depth = 0;
80
+ let quote = '';
81
+ for (let i = from; i < text.length; i += 1) {
82
+ const ch = text[i];
83
+ if (quote) {
84
+ if (ch === '\\')
85
+ i += 1;
86
+ else if (ch === quote)
87
+ quote = '';
88
+ continue;
89
+ }
90
+ if (ch === '"' || ch === "'" || ch === '`')
91
+ quote = ch;
92
+ else if (ch === '(' || ch === '[')
93
+ depth += 1;
94
+ else if (ch === ')' || ch === ']') {
95
+ depth -= 1;
96
+ if (depth === 0)
97
+ return text.slice(i + 1);
98
+ }
99
+ }
100
+ return null;
101
+ }
102
+ /**
103
+ * A check that cannot fail because both sides are the same literal, or the same expression.
104
+ * `expect(true).toBe(true)`, `assertEquals(1, 1)`, `assert x == x`. These appear in generated tests
105
+ * and in tests somebody left as a placeholder, and they pass forever.
106
+ */
107
+ export function tautologies(body) {
108
+ const found = [];
109
+ const literal = String.raw `(?:true|false|null|None|nil|undefined|-?\d+(?:\.\d+)?|'[^']*'|"[^"]*")`;
110
+ const patterns = [
111
+ new RegExp(String.raw `expect\s*\(\s*(${literal})\s*\)\s*\.\s*(?:to\s*\.\s*)?(?:toBe|toEqual|toStrictEqual|equal|eq|is)\s*\(\s*(${literal})\s*\)`),
112
+ new RegExp(String.raw `assert(?:Equals?|\.equal|\.strictEqual|\.deepEqual)\s*\(\s*(${literal})\s*,\s*(${literal})\s*\)`),
113
+ new RegExp(String.raw `^\s*assert\s+(${literal})\s*==\s*(${literal})\s*$`),
114
+ ];
115
+ for (const line of body.split(String.fromCharCode(10))) {
116
+ for (const pattern of patterns) {
117
+ const hit = pattern.exec(line);
118
+ if (hit && same(hit[1], hit[2]))
119
+ found.push(line.trim());
120
+ }
121
+ // The same name on both sides of a comparison, which no value can falsify.
122
+ const self = /(?:expect\s*\(\s*([\w.$]+)\s*\)\s*\.\s*(?:toBe|toEqual|equal)\s*\(\s*([\w.$]+)\s*\)|^\s*assert\s+([\w.$]+)\s*==\s*([\w.$]+)\s*$)/.exec(line);
123
+ if (self) {
124
+ const [left, right] = [self[1] ?? self[3], self[2] ?? self[4]];
125
+ if (left && right && left === right && !/^(true|false|null|undefined|None)$/.test(left))
126
+ found.push(line.trim());
127
+ }
128
+ // `assert True` / `assert 1`: a constant truth written as a statement.
129
+ if (/^\s*assert\s+(True|true|1)\s*(,|$)/.test(line))
130
+ found.push(line.trim());
131
+ }
132
+ return [...new Set(found)];
133
+ }
134
+ /** Two literals that mean the same value, allowing for how each language spells truth. */
135
+ function same(a, b) {
136
+ const clean = (v) => v.trim().replace(/^['"]|['"]$/g, '');
137
+ return clean(a) === clean(b);
138
+ }
@@ -0,0 +1,35 @@
1
+ // The command around `vouch`: read the paths, print the report, and exit non-zero under --strict so
2
+ // it can sit in front of a merge. Kept apart from the report itself so the MCP server can ask the
3
+ // same question without anything being printed to the stream it speaks on.
4
+ import { mkdirSync, writeFileSync } from 'node:fs';
5
+ import { dirname, resolve } from 'node:path';
6
+ import { CANNOT_FAIL } from './vouch.js';
7
+ import { renderVouch, vouchAll } from './vouch-command.js';
8
+ /** The silent core: every MCP caller and the report itself use this. */
9
+ export function vouchReport(root, paths) {
10
+ const files = vouchAll(root, paths);
11
+ const all = files.flatMap((f) => f.judgements);
12
+ return {
13
+ report: renderVouch(files),
14
+ cannotFail: all.filter((j) => CANNOT_FAIL.includes(j.verdict)).length,
15
+ read: all.length,
16
+ };
17
+ }
18
+ export function runVouch(cmd) {
19
+ const files = vouchAll(cmd.root, cmd.paths);
20
+ const report = renderVouch(files);
21
+ console.log(report);
22
+ if (cmd.out) {
23
+ mkdirSync(dirname(resolve(cmd.out)), { recursive: true });
24
+ writeFileSync(resolve(cmd.out), report + String.fromCharCode(10), 'utf8');
25
+ }
26
+ if (cmd.json) {
27
+ mkdirSync(dirname(resolve(cmd.json)), { recursive: true });
28
+ writeFileSync(resolve(cmd.json), JSON.stringify(files, null, 2), 'utf8');
29
+ }
30
+ const cannotFail = files.flatMap((f) => f.judgements).filter((j) => CANNOT_FAIL.includes(j.verdict)).length;
31
+ if (cmd.strict && cannotFail > 0) {
32
+ console.error(String.fromCharCode(10) + `${cannotFail} test(s) here cannot fail. Fix them or take them out; a test that cannot fail is worse than no test, because it reads as coverage.`);
33
+ process.exitCode = 1;
34
+ }
35
+ }
@@ -0,0 +1,275 @@
1
+ /** Which language this file is, by extension. Unknown extensions are not guessed at. */
2
+ export function languageOf(file) {
3
+ const lower = file.toLowerCase();
4
+ if (/\.(js|jsx|mjs|cjs|ts|tsx|mts|cts)$/.test(lower))
5
+ return 'js';
6
+ if (lower.endsWith('.py'))
7
+ return 'python';
8
+ if (/\.(java|kt|kts|scala|groovy)$/.test(lower))
9
+ return 'jvm';
10
+ if (lower.endsWith('.go'))
11
+ return 'go';
12
+ if (/\.(cs|fs)$/.test(lower))
13
+ return 'csharp';
14
+ return null;
15
+ }
16
+ /** Does this file look like tests at all? Vouching for a source file is a category error. */
17
+ export function looksLikeTests(file, text) {
18
+ const named = /(^|[\\/])(tests?|spec|specs|__tests__|e2e|it|integration)[\\/]|[._-](test|tests|spec|specs|it)\.[a-z]+$|(^|[\\/])test_[^\\/]+\.py$|_test\.(go|py|rb)$|Tests?\.(java|kt|cs|scala)$/i.test(file);
19
+ const declares = /\b(it|test|describe|Scenario)\s*\(|^\s*def\s+test_|@Test\b|\[(Test|Fact|Theory)\]|\bfunc\s+Test[A-Z]/m.test(text);
20
+ return named || declares;
21
+ }
22
+ /**
23
+ * The offsets of this file that are inside a string, a template literal or a comment - text, not
24
+ * code. Test files are full of test code that is not test code: fixtures quoting a fake test,
25
+ * snapshots, documentation examples, and the input to tools like Tewip itself. Judging those is how
26
+ * a checker ends up telling somebody that a string in their fixture checks nothing. Found by
27
+ * pointing this at Tewip's own test suite, where it produced 27 confident false accusations.
28
+ */
29
+ export function textRanges(text) {
30
+ const ranges = [];
31
+ const TRIPLES = ['"'.repeat(3), "'".repeat(3)];
32
+ for (let i = 0; i < text.length; i += 1) {
33
+ const ch = text[i];
34
+ const next = text[i + 1];
35
+ if (ch === '/' && next === '/') {
36
+ const end = indexOrEnd(text, String.fromCharCode(10), i);
37
+ ranges.push([i, end]);
38
+ i = end;
39
+ }
40
+ else if (ch === '/' && next === '*') {
41
+ const end = indexOrEnd(text, '*/', i) + 1;
42
+ ranges.push([i, end]);
43
+ i = end;
44
+ }
45
+ else if (ch === '#' && /[#\s]/.test(next ?? '')) {
46
+ const end = indexOrEnd(text, String.fromCharCode(10), i);
47
+ ranges.push([i, end]);
48
+ i = end;
49
+ }
50
+ else if (ch === '"' || ch === "'" || ch === '`') {
51
+ // A Python docstring runs to the next triple quote, not to the next single one.
52
+ const triple = TRIPLES.find((t) => text.startsWith(t, i));
53
+ const end = triple ? indexOrEnd(text, triple, i + 2) + 2 : closingQuote(text, i);
54
+ ranges.push([i, end]);
55
+ i = end;
56
+ }
57
+ }
58
+ return ranges;
59
+ }
60
+ /** Is this offset inside text rather than code? */
61
+ const isText = (ranges, at) => ranges.some(([from, to]) => at > from && at < to);
62
+ export function testsIn(language, text) {
63
+ if (language === 'python')
64
+ return pythonTests(text);
65
+ if (language === 'js')
66
+ return jsTests(text);
67
+ return bracedTests(language, text);
68
+ }
69
+ /** The line number of an offset, 1-based. */
70
+ const lineAt = (text, index) => text.slice(0, index).split(String.fromCharCode(10)).length;
71
+ /**
72
+ * `it('...', async () => { ... })` and its many spellings, including `test.each`, `it.skip` and a
73
+ * plain `function ()` callback. The body is taken by matching braces from the callback's opening one.
74
+ */
75
+ function jsTests(text) {
76
+ const found = [];
77
+ const ranges = textRanges(text);
78
+ const declaration = /\b(it|test|specify)\s*(\.\s*(skip|only|todo|failing|concurrent|each|for)\s*(\([^)]*\)|`[^`]*`)?\s*)*\(\s*(['"`])((?:\\.|(?!\5)[^\\])*)\5/g;
79
+ for (const hit of text.matchAll(declaration)) {
80
+ const at = hit.index ?? 0;
81
+ if (isText(ranges, at))
82
+ continue;
83
+ const open = text.indexOf('{', at + hit[0].length);
84
+ if (open === -1)
85
+ continue;
86
+ const body = braced(text, open);
87
+ if (body === null)
88
+ continue;
89
+ const signature = text.slice(at + hit[0].length, open);
90
+ found.push({
91
+ name: hit[6],
92
+ line: lineAt(text, at),
93
+ body,
94
+ skipped: /\.\s*(skip|todo|failing)\b/.test(hit[0]) || /\bxit\s*\(/.test(hit[0]),
95
+ callback: /\(\s*([A-Za-z_$][\w$]*)\s*\)\s*(=>)?\s*$/.exec(signature.trim())?.[1] ?? '',
96
+ });
97
+ }
98
+ // `xit(...)` and `xtest(...)`, which the pattern above deliberately does not match.
99
+ for (const hit of text.matchAll(/\bx(it|test)\s*\(\s*(['"`])((?:\\.|(?!\2)[^\\])*)\2/g)) {
100
+ const at = hit.index ?? 0;
101
+ if (isText(ranges, at))
102
+ continue;
103
+ const open = text.indexOf('{', at + hit[0].length);
104
+ const body = open === -1 ? null : braced(text, open);
105
+ found.push({ name: hit[3], line: lineAt(text, at), body: body ?? '', skipped: true, callback: '' });
106
+ }
107
+ return found.sort((a, b) => a.line - b.line);
108
+ }
109
+ /** Python has no braces: a test's body is everything indented further than its `def`. */
110
+ function pythonTests(text) {
111
+ const lines = text.split(String.fromCharCode(10));
112
+ const ranges = textRanges(text);
113
+ const offsets = [];
114
+ let seen = 0;
115
+ for (const line of lines) {
116
+ offsets.push(seen);
117
+ seen += line.length + 1;
118
+ }
119
+ const found = [];
120
+ for (let i = 0; i < lines.length; i += 1) {
121
+ const hit = /^(\s*)(?:async\s+)?def\s+(test_\w+|test)\s*\(/.exec(lines[i]);
122
+ if (!hit)
123
+ continue;
124
+ if (isText(ranges, offsets[i] + hit[1].length + 1))
125
+ continue;
126
+ const indent = hit[1].length;
127
+ // Nested in another def? Then it is a helper or, in Click's tests, a CLI command that happens to
128
+ // be named `test`. Only a module-level def, or a method of a test class, is a test.
129
+ if (nestedInFunction(lines, i, indent))
130
+ continue;
131
+ if (hit[2] === 'test' && indent > 0)
132
+ continue;
133
+ // A def whose parameters run over several lines ends at the line that closes them, and the body
134
+ // starts after that. Without this, the parameter list was read as the body and the real body was
135
+ // thrown away: psf/requests has several, and each came back as a test that compares no value.
136
+ let signatureEnds = i;
137
+ let open = 0;
138
+ for (let j = i; j < lines.length && j < i + 20; j += 1) {
139
+ open += (lines[j].match(/\(/g) ?? []).length - (lines[j].match(/\)/g) ?? []).length;
140
+ signatureEnds = j;
141
+ if (open <= 0)
142
+ break;
143
+ }
144
+ const body = [];
145
+ for (let j = signatureEnds + 1; j < lines.length; j += 1) {
146
+ const line = lines[j];
147
+ if (line.trim() === '') {
148
+ body.push(line);
149
+ continue;
150
+ }
151
+ const width = line.length - line.trimStart().length;
152
+ if (width <= indent)
153
+ break;
154
+ body.push(line);
155
+ }
156
+ // Decorators sit above the def, so skipping is read upwards.
157
+ const above = lines.slice(Math.max(0, i - 4), i).join(String.fromCharCode(10));
158
+ // A conditional skip runs somewhere - `skipif` on Windows registry tests in psf/requests runs
159
+ // on Windows - so only an unconditional skip means the test never runs. `xfail` runs too.
160
+ const skipped = /@\s*(pytest\s*\.\s*mark\s*\.\s*skip|unittest\s*\.\s*skip)\b/.test(above) && !/skipif|skipUnless|xfail/.test(above);
161
+ found.push({ name: hit[2], line: i + 1, body: body.join(String.fromCharCode(10)), skipped, callback: '' });
162
+ }
163
+ return found;
164
+ }
165
+ /** Is the def on line `i` nested inside another def, rather than inside a class or at the top? */
166
+ function nestedInFunction(lines, i, indent) {
167
+ for (let j = i - 1; j >= 0; j -= 1) {
168
+ const line = lines[j];
169
+ if (line.trim() === '')
170
+ continue;
171
+ const width = line.length - line.trimStart().length;
172
+ if (width >= indent)
173
+ continue;
174
+ return /^\s*(async\s+)?def\s/.test(line);
175
+ }
176
+ return false;
177
+ }
178
+ /** JUnit, Kotlin, Scala, Go and C#: an annotation or a naming convention, then a braced body. */
179
+ function bracedTests(language, text) {
180
+ const found = [];
181
+ const ranges = textRanges(text);
182
+ const declaration = language === 'go'
183
+ ? // Only a Test function. A Benchmark asserts nothing by design, an Example is checked against
184
+ // its own Output comment, and a Fuzz target is not a test - all three were condemned as
185
+ // "checks nothing" in gorilla/mux before this narrowed.
186
+ /\bfunc\s+(Test)(\w+)\s*\(\s*\w+\s+\*testing\.[TB]\s*\)\s*\{/g
187
+ : /(@Test\b[^\n]*|\[\s*(?:Test|Fact|Theory)[^\]]*\]|@(?:ParameterizedTest|RepeatedTest)\b)/g;
188
+ for (const hit of text.matchAll(declaration)) {
189
+ const at = hit.index ?? 0;
190
+ if (isText(ranges, at))
191
+ continue;
192
+ const open = language === 'go' ? text.indexOf('{', at) : text.indexOf('{', text.indexOf('(', at + hit[0].length));
193
+ if (open === -1)
194
+ continue;
195
+ const body = braced(text, open);
196
+ if (body === null)
197
+ continue;
198
+ const signature = text.slice(at, open);
199
+ // `void foo() throws IOException` still names foo. Without allowing the throws clause, every
200
+ // checked-exception test in gson was reported as 'unnamed'.
201
+ const name = language === 'go' ? hit[1] + hit[2] : /([\w]+)\s*\([^)]*\)\s*(throws\s+[\w.,\s]+)?$/.exec(signature.trim())?.[1] ?? 'unnamed';
202
+ const around = text.slice(Math.max(0, at - 200), open);
203
+ // An annotation disables a test outright. A `t.Skip()` may be conditional - cobra's
204
+ // TestDeadcodeElimination skips on Windows and runs everywhere else - so it counts only when it
205
+ // is unconditional, at the top level of the body.
206
+ const skipped = /@(Disabled|Ignore)\b|\[\s*Ignore\b/.test(around) || skipsUnconditionally(body);
207
+ found.push({ name, line: lineAt(text, at), body, skipped, callback: '' });
208
+ }
209
+ return found;
210
+ }
211
+ /**
212
+ * A `t.Skip()` that runs whatever happens: at the top level of the body, not inside an `if`. Go
213
+ * tests commonly skip on one platform and run on the others, and calling those "never runs" is
214
+ * wrong.
215
+ */
216
+ function skipsUnconditionally(body) {
217
+ let depth = 0;
218
+ for (const line of body.split(String.fromCharCode(10))) {
219
+ const text = line.trim();
220
+ if (depth === 0 && /^\w+\s*\.\s*Skip(Now)?\s*\(/.test(text))
221
+ return true;
222
+ depth += (line.match(/\{/g) ?? []).length - (line.match(/\}/g) ?? []).length;
223
+ }
224
+ return false;
225
+ }
226
+ /**
227
+ * The text between the brace at `open` and the one that closes it, skipping over strings, template
228
+ * literals, comments and regular expressions - all of which can hold a brace that closes nothing.
229
+ */
230
+ export function braced(text, open) {
231
+ let depth = 0;
232
+ for (let i = open; i < text.length; i += 1) {
233
+ const ch = text[i];
234
+ const next = text[i + 1];
235
+ if (ch === '/' && next === '/') {
236
+ i = indexOrEnd(text, String.fromCharCode(10), i);
237
+ continue;
238
+ }
239
+ if (ch === '/' && next === '*') {
240
+ i = indexOrEnd(text, '*/', i) + 1;
241
+ continue;
242
+ }
243
+ if (ch === '#' && /[#\s]/.test(next ?? '')) {
244
+ i = indexOrEnd(text, String.fromCharCode(10), i);
245
+ continue;
246
+ }
247
+ if (ch === '"' || ch === "'" || ch === '`') {
248
+ i = closingQuote(text, i);
249
+ continue;
250
+ }
251
+ if (ch === '{')
252
+ depth += 1;
253
+ else if (ch === '}') {
254
+ depth -= 1;
255
+ if (depth === 0)
256
+ return text.slice(open + 1, i);
257
+ }
258
+ }
259
+ return null;
260
+ }
261
+ const indexOrEnd = (text, needle, from) => {
262
+ const at = text.indexOf(needle, from + 1);
263
+ return at === -1 ? text.length : at;
264
+ };
265
+ /** Where the string opening at `from` ends, honouring escapes. */
266
+ function closingQuote(text, from) {
267
+ const quote = text[from];
268
+ for (let i = from + 1; i < text.length; i += 1) {
269
+ if (text[i] === '\\')
270
+ i += 1;
271
+ else if (text[i] === quote)
272
+ return i;
273
+ }
274
+ return text.length;
275
+ }
@@ -0,0 +1,135 @@
1
+ // `tewip vouch <paths>`: which of these tests could never fail.
2
+ //
3
+ // Costs nothing and calls no model: every verdict is a line of the file quoted back. The report is
4
+ // ordered for a reviewer with ten minutes - what cannot fail, then what does not run, then what
5
+ // Tewip would not judge - and it ends with what it refused, because that is what makes the rest of
6
+ // it believable.
7
+ import { existsSync, readFileSync, readdirSync, statSync } from 'node:fs';
8
+ import { join, relative } from 'node:path';
9
+ import { CANNOT_FAIL, vouchFile } from './vouch.js';
10
+ const SKIP = new Set(['node_modules', '.git', 'dist', 'build', 'coverage', 'vendor', 'target', '.venv', '__pycache__', '.tewip']);
11
+ /** Every file under these paths that could hold tests. */
12
+ export function testFiles(root, paths) {
13
+ const found = [];
14
+ const walk = (where, depth) => {
15
+ if (depth > 12)
16
+ return;
17
+ let entries;
18
+ try {
19
+ entries = readdirSync(where);
20
+ }
21
+ catch {
22
+ return;
23
+ }
24
+ for (const entry of entries) {
25
+ if (SKIP.has(entry))
26
+ continue;
27
+ const full = join(where, entry);
28
+ let stat;
29
+ try {
30
+ stat = statSync(full);
31
+ }
32
+ catch {
33
+ continue;
34
+ }
35
+ if (stat.isDirectory())
36
+ walk(full, depth + 1);
37
+ else if (/\.(js|jsx|mjs|cjs|ts|tsx|mts|cts|py|java|kt|kts|scala|groovy|go|cs|fs)$/i.test(entry))
38
+ found.push(full);
39
+ }
40
+ };
41
+ for (const path of paths) {
42
+ const full = join(root, path);
43
+ if (!existsSync(full))
44
+ continue;
45
+ if (statSync(full).isDirectory())
46
+ walk(full, 0);
47
+ else
48
+ found.push(full);
49
+ }
50
+ return found;
51
+ }
52
+ export function vouchAll(root, paths) {
53
+ const out = [];
54
+ for (const file of testFiles(root, paths)) {
55
+ let text;
56
+ try {
57
+ text = readFileSync(file, 'utf8');
58
+ }
59
+ catch {
60
+ continue;
61
+ }
62
+ if (text.length > 2_000_000)
63
+ continue; // a generated file, not a test somebody wrote
64
+ const vouched = vouchFile(relative(root, file).split('\\').join('/'), text);
65
+ if (vouched.judgements.length > 0 || vouched.focused)
66
+ out.push(vouched);
67
+ }
68
+ return out;
69
+ }
70
+ const count = (all, verdict) => all.filter((j) => j.verdict === verdict).length;
71
+ export function renderVouch(files) {
72
+ const all = files.flatMap((f) => f.judgements);
73
+ const cannotFail = all.filter((j) => CANNOT_FAIL.includes(j.verdict));
74
+ const crashOnly = count(all, 'crash-only');
75
+ const lines = [
76
+ '# Which of these tests could fail?',
77
+ '',
78
+ `${all.length} test(s) read in ${files.length} file(s). **${cannotFail.length} cannot fail at all.** ${crashOnly} compare no value, so only a crash would fail them. ${count(all, 'cannot-tell')} Tewip would not judge.`,
79
+ '',
80
+ 'A test that cannot fail still passes, still counts towards coverage, and still looks like work.',
81
+ '',
82
+ ];
83
+ if (cannotFail.length === 0) {
84
+ lines.push('## Nothing here is unable to fail', '', 'Every test read can be broken by a change to the code, or by the code throwing.', '');
85
+ }
86
+ for (const verdict of CANNOT_FAIL) {
87
+ const group = cannotFail.filter((j) => j.verdict === verdict);
88
+ if (group.length === 0)
89
+ continue;
90
+ lines.push(`## ${TITLES[verdict]} (${group.length})`, '');
91
+ for (const j of group) {
92
+ lines.push(`- **${j.name}** — ${j.file}:${j.line}`);
93
+ if (j.evidence)
94
+ lines.push(` - \`${j.evidence.replace(/`/g, "'")}\``);
95
+ lines.push(` - ${j.why}`);
96
+ }
97
+ lines.push('');
98
+ }
99
+ const focused = files.filter((f) => f.focused);
100
+ if (focused.length > 0) {
101
+ lines.push(`## Files where most tests do not run (${focused.length})`, '');
102
+ for (const f of focused)
103
+ lines.push(`- ${f.file}: ${f.focused}`);
104
+ lines.push('');
105
+ }
106
+ const weak = all.filter((j) => j.verdict === 'crash-only');
107
+ if (weak.length > 0) {
108
+ lines.push(`## Compare no value (${weak.length})`, '', 'These are not broken. They fail if the code throws and pass whatever it returns, which is', 'sometimes exactly the intention - a test that compiles a fixture and runs it is checking that', 'nothing blew up. Worth a look where a value was meant to be checked.', '');
109
+ for (const j of weak.slice(0, 40))
110
+ lines.push(`- ${j.name} — ${j.file}:${j.line}: \`${j.evidence.replace(/`/g, "'")}\``);
111
+ if (weak.length > 40)
112
+ lines.push(`- …and ${weak.length - 40} more`);
113
+ lines.push('');
114
+ }
115
+ const undecided = all.filter((j) => j.verdict === 'cannot-tell');
116
+ if (undecided.length > 0) {
117
+ lines.push(`## Not judged (${undecided.length})`, '', 'Tewip will not call these either way, because the checking may be somewhere it cannot follow.', '');
118
+ for (const j of undecided.slice(0, 25))
119
+ lines.push(`- ${j.name} — ${j.file}:${j.line}: ${j.evidence}`);
120
+ if (undecided.length > 25)
121
+ lines.push(`- …and ${undecided.length - 25} more`);
122
+ lines.push('');
123
+ }
124
+ lines.push('---', '', 'Every verdict above quotes the line it is based on, and nothing was inferred by a model: this', 'costs nothing to run. Where the checking happens in a helper Tewip cannot follow it says so', 'rather than guessing, which is why the list above is worth reading.', '', 'What this does not tell you is how many it missed. Everything it reports here has been measured', 'for correctness on real projects; its recall has not been measured at all, so a clean report is', 'not proof that every test in these files is worth something.');
125
+ return lines.join(String.fromCharCode(10));
126
+ }
127
+ const TITLES = {
128
+ 'crash-only': 'Compares no value: only a crash can fail it',
129
+ 'unfinished-expectation': 'An expectation nobody finished',
130
+ tautology: 'Compares a value with itself',
131
+ 'swallows-failure': 'Discards its own failure',
132
+ 'never-runs': 'Never runs',
133
+ 'cannot-tell': 'Not judged',
134
+ checks: 'Can fail',
135
+ };
@@ -0,0 +1,193 @@
1
+ // Phase 22: does this test prove anything?
2
+ //
3
+ // By 2026 most tests in a pull request were written by an agent, and the one question that decides
4
+ // whether they are worth anything goes unasked: could this test ever fail? Coverage counts a test
5
+ // that checks nothing. CI goes green on it. A reviewer skims something plausible and approves it.
6
+ //
7
+ // This file answers that per test, from the text alone, and its accuracy rests on refusing to guess.
8
+ // Telling somebody their test is worthless when it is not costs more than missing one: it spends
9
+ // their time and their trust, and a reviewer who is wrong once is ignored afterwards. So a body whose
10
+ // checking happens somewhere this cannot follow is reported as undecided, by name, and never as
11
+ // vacuous.
12
+ import { checksSomething, tautologies, unfinishedExpectations } from './assertions.js';
13
+ import { languageOf, looksLikeTests, testsIn, textRanges } from './tests-in.js';
14
+ /** Everything a verdict other than `checks` means, in the words the report uses. */
15
+ export const MEANS = {
16
+ checks: 'it asserts something, so it can fail',
17
+ 'crash-only': 'it compares no value, so it passes whatever the code returns; only an exception can fail it',
18
+ 'unfinished-expectation': 'an expectation was written but no matcher was called on it, so nothing is compared',
19
+ tautology: 'the check compares a value with itself, which no change to the code can falsify',
20
+ 'swallows-failure': 'its own failure is caught and discarded, so it reports success either way',
21
+ 'never-runs': 'it is skipped, so it proves nothing until somebody turns it back on',
22
+ 'cannot-tell': 'the checking may happen inside a helper this cannot follow, so Tewip is not calling it either way',
23
+ };
24
+ /**
25
+ * Tests that cannot fail at all, worst first.
26
+ *
27
+ * `crash-only` is deliberately not in this list, and that correction came from reading real code.
28
+ * "It checks nothing, so it cannot fail" is false: a test with no assertion still fails when the
29
+ * code throws, and in axios a test that compiles a TypeScript fixture and runs it asserts nothing
30
+ * on purpose - the crash *is* the check. Tewip would have called 143 such tests worthless across
31
+ * seven real projects. Overclaiming about somebody's tests is the exact failure this project exists
32
+ * not to commit, so the two are reported apart and worded for what each really is.
33
+ */
34
+ export const CANNOT_FAIL = ['unfinished-expectation', 'tautology', 'swallows-failure', 'never-runs'];
35
+ /** Weaker than a check, and worth knowing, but not a defect on its own. */
36
+ export const CRASH_ONLY = ['crash-only'];
37
+ export function vouchFile(file, text) {
38
+ const language = languageOf(file);
39
+ if (!language)
40
+ return { file, judgements: [], unread: 'Tewip does not read this kind of file', focused: '' };
41
+ if (!looksLikeTests(file, text))
42
+ return { file, judgements: [], unread: 'this does not look like a test file, and vouching for source code is a different question', focused: '' };
43
+ const tests = testsIn(language, text);
44
+ if (tests.length === 0) {
45
+ return { file, judgements: [], unread: 'no test in a shape Tewip recognises, so it says nothing about this file rather than clearing it', focused: '' };
46
+ }
47
+ const helpers = helpersThatCheck(text, language);
48
+ // Masked the same way the extractor is: a `.only` quoted inside a fixture focuses nothing.
49
+ const ranges = textRanges(text);
50
+ const focus = [...text.matchAll(/\b(it|test|describe)\s*\.\s*only\s*\(|\bfdescribe\s*\(|\bfit\s*\(/g)].find((hit) => !ranges.some(([from, to]) => (hit.index ?? 0) > from && (hit.index ?? 0) < to));
51
+ return {
52
+ file,
53
+ judgements: tests.map((test) => judge(file, test, helpers)),
54
+ unread: '',
55
+ focused: focus ? `${focus[0].trim()}…) is focused, so the rest of this file does not run` : '',
56
+ };
57
+ }
58
+ function judge(file, test, helpers) {
59
+ const at = { file, name: test.name, line: test.line };
60
+ if (test.skipped)
61
+ return { ...at, verdict: 'never-runs', evidence: test.name, why: MEANS['never-runs'] };
62
+ const body = withoutComments(test.body);
63
+ const tautology = tautologies(body);
64
+ const unfinished = unfinishedExpectations(body);
65
+ const checks = checksSomething(body);
66
+ // A body that checks nothing else and contains only a tautology cannot fail. One alongside real
67
+ // checks is untidy, not broken, so it is left alone.
68
+ if (tautology.length > 0 && !checksSomething(stripLines(body, tautology))) {
69
+ return { ...at, verdict: 'tautology', evidence: tautology[0], why: MEANS.tautology };
70
+ }
71
+ if (unfinished.length > 0 && !checksSomething(stripLines(body, unfinished))) {
72
+ return { ...at, verdict: 'unfinished-expectation', evidence: unfinished[0], why: MEANS['unfinished-expectation'] };
73
+ }
74
+ const swallowed = swallowsFailure(body, checks);
75
+ if (swallowed)
76
+ return { ...at, verdict: 'swallows-failure', evidence: swallowed, why: MEANS['swallows-failure'] };
77
+ if (checks)
78
+ return { ...at, verdict: 'checks', evidence: '', why: MEANS.checks };
79
+ // Mocha's `done`: the test fails when the callback never arrives, which is a real check with
80
+ // nothing asserted. express has dozens, and calling them unchecked was simply wrong.
81
+ if (test.callback && new RegExp('\\b' + test.callback + '\\b').test(body)) {
82
+ return { ...at, verdict: 'checks', evidence: test.callback, why: `it fails if ${test.callback}() is never called, which is what it is checking` };
83
+ }
84
+ // Nothing in the body checks anything. Before saying so, look for the checking somewhere else:
85
+ // a helper in this file that asserts, or a call this cannot follow at all.
86
+ const called = [...body.matchAll(/([A-Za-z_$][\w$]*)\s*\(/g)].map((m) => m[1]).filter((name) => !LANGUAGE_NOISE.has(name));
87
+ const helping = called.find((name) => helpers.has(name));
88
+ if (helping)
89
+ return { ...at, verdict: 'checks', evidence: `${helping}(…)`, why: `it checks through ${helping}(), which asserts` };
90
+ const unknown = called.find((name) => /^(check|verify|ensure|validate|assert|expect|should|confirm)/i.test(name));
91
+ if (unknown)
92
+ return { ...at, verdict: 'cannot-tell', evidence: `${unknown}(…)`, why: `${unknown}() may assert, and it is not in this file, so Tewip will not call this either way` };
93
+ if (unfinished.length > 0)
94
+ return { ...at, verdict: 'unfinished-expectation', evidence: unfinished[0], why: MEANS['unfinished-expectation'] };
95
+ if (tautology.length > 0)
96
+ return { ...at, verdict: 'tautology', evidence: tautology[0], why: MEANS.tautology };
97
+ if (body.trim() === '')
98
+ return { ...at, verdict: 'crash-only', evidence: '(the body is empty)', why: 'the test has no body at all, so nothing can fail it' };
99
+ return { ...at, verdict: 'crash-only', evidence: firstRealLine(body), why: MEANS['crash-only'] };
100
+ }
101
+ /**
102
+ * Everything inside a `try` whose `catch` discards what it caught. An assertion failure is an
103
+ * exception, so a test shaped this way reports success however the code behaves - and a `catch` that
104
+ * asserts, re-throws or fails the test is the legitimate version, which this leaves alone.
105
+ */
106
+ function swallowsFailure(body, checks) {
107
+ if (!checks)
108
+ return ''; // nothing to swallow; judged as no-assertion instead
109
+ for (const hit of body.matchAll(/catch\s*(\([^)]*\))?\s*\{/g)) {
110
+ const open = body.indexOf('{', hit.index ?? 0);
111
+ const handler = balanced(body, open);
112
+ if (handler === null)
113
+ continue;
114
+ const meaningful = withoutComments(handler).trim();
115
+ const rescues = checksSomething(handler) || /\b(throw|raise|rethrow|fail|t\s*\.\s*(Error|Fatal)|pytest\s*\.\s*fail)\b/.test(handler);
116
+ if (rescues)
117
+ continue;
118
+ // A handler that only logs, or does nothing at all, discards the failure.
119
+ if (meaningful === '' || /^(console\s*\.\s*\w+\s*\([^;]*\)\s*;?|print\s*\([^;]*\)\s*;?|pass|;)$/.test(meaningful.replace(/\s+/g, ' ').trim())) {
120
+ return meaningful === '' ? 'catch { }' : meaningful.slice(0, 120);
121
+ }
122
+ }
123
+ return '';
124
+ }
125
+ /** Functions defined in this file that assert, so a test calling one really is checking something. */
126
+ function helpersThatCheck(text, language) {
127
+ const names = new Set();
128
+ const declarations = language === 'python'
129
+ ? /^\s*(?:async\s+)?def\s+(\w+)\s*\(/gm
130
+ : /(?:function\s+([A-Za-z_$][\w$]*)|(?:const|let|var)\s+([A-Za-z_$][\w$]*)\s*=\s*(?:async\s*)?(?:\([^)]*\)|[\w$]+)\s*=>|(?:public|private|protected|static|\s)+[\w<>,\[\]]+\s+([A-Za-z_$][\w$]*)\s*\([^)]*\)\s*\{)/g;
131
+ for (const hit of text.matchAll(declarations)) {
132
+ const name = hit[1] ?? hit[2] ?? hit[3];
133
+ if (!name)
134
+ continue;
135
+ const from = (hit.index ?? 0) + hit[0].length;
136
+ const body = language === 'python' ? indentedFrom(text, from) : balanced(text, text.indexOf('{', from - 1));
137
+ if (body && checksSomething(body))
138
+ names.add(name);
139
+ }
140
+ return names;
141
+ }
142
+ /** Python's block after an offset: every line indented further than the line the offset is on. */
143
+ function indentedFrom(text, from) {
144
+ const lines = text.slice(from).split(String.fromCharCode(10));
145
+ const start = text.slice(0, from).split(String.fromCharCode(10)).pop() ?? '';
146
+ const indent = start.length - start.trimStart().length;
147
+ const body = [];
148
+ for (const line of lines.slice(1)) {
149
+ if (line.trim() === '')
150
+ continue;
151
+ if (line.length - line.trimStart().length <= indent)
152
+ break;
153
+ body.push(line);
154
+ }
155
+ return body.join(String.fromCharCode(10));
156
+ }
157
+ /** Names that are calls but not helpers: language and framework machinery. */
158
+ const LANGUAGE_NOISE = new Set(['if', 'for', 'while', 'switch', 'catch', 'return', 'function', 'await', 'typeof', 'new', 'require', 'import', 'print', 'console', 'log', 'describe', 'beforeEach', 'afterEach', 'it', 'test', 'range', 'len', 'str', 'int', 'super', 'Error', 'Promise', 'Array', 'Object', 'String', 'Number', 'Boolean', 'Date', 'JSON', 'Math', 'Map', 'Set']);
159
+ const balanced = (text, open) => {
160
+ if (open === -1)
161
+ return null;
162
+ let depth = 0;
163
+ for (let i = open; i < text.length; i += 1) {
164
+ if (text[i] === '{')
165
+ depth += 1;
166
+ else if (text[i] === '}') {
167
+ depth -= 1;
168
+ if (depth === 0)
169
+ return text.slice(open + 1, i);
170
+ }
171
+ }
172
+ return null;
173
+ };
174
+ /** Comments cannot assert, and a commented-out check is the classic way a test stops checking. */
175
+ function withoutComments(body) {
176
+ return body
177
+ .replace(/\/\*[\s\S]*?\*\//g, ' ')
178
+ .split(String.fromCharCode(10))
179
+ .map((line) => line.replace(/(^|\s)\/\/.*$/, '').replace(/(^|\s)#(?!\{).*$/, ''))
180
+ .join(String.fromCharCode(10));
181
+ }
182
+ const stripLines = (body, lines) => body
183
+ .split(String.fromCharCode(10))
184
+ .filter((line) => !lines.some((drop) => line.trim() === drop.trim()))
185
+ .join(String.fromCharCode(10));
186
+ /** The first line that does something, for the report to quote. */
187
+ function firstRealLine(body) {
188
+ const line = body
189
+ .split(String.fromCharCode(10))
190
+ .map((l) => l.trim())
191
+ .find((l) => l !== '' && l !== '{' && l !== '}');
192
+ return (line ?? '').slice(0, 120);
193
+ }
@@ -123,4 +123,18 @@ server.registerTool('next_action', {
123
123
  const observed = observe(root, goal);
124
124
  return renderPursuit(goal, observed, decide(goal, observed), false);
125
125
  }));
126
+ server.registerTool('vouch_tests', {
127
+ title: 'Could these tests ever fail?',
128
+ description: "Reads test files and says, per test, whether a change to the code could break them: which compare no value, which leave an expectation unfinished, which compare something with itself, which catch and discard their own failure, and which never run. Where the checking happens in a helper it cannot follow it says so instead of accusing. Costs nothing and uses no model. Use it on tests that were just written - including tests you wrote yourself - before claiming a change is covered.",
129
+ inputSchema: {
130
+ project,
131
+ paths: z.array(z.string()).optional().describe('Paths inside the project to read, relative to it. Defaults to the whole project.'),
132
+ },
133
+ annotations: readOnly,
134
+ }, ({ project: root, paths }) => answer(async () => {
135
+ if (!isAbsolute(root) || !existsSync(root))
136
+ throw new Error(`project must be an existing absolute path, got "${root}"`);
137
+ const { vouchReport } = await import('../engine/vouch/run-vouch.js');
138
+ return vouchReport(root, paths && paths.length > 0 ? paths : ['.']).report;
139
+ }));
126
140
  await server.connect(new StdioServerTransport());
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "tewip-ai",
3
- "version": "0.24.0",
3
+ "version": "0.26.0",
4
4
  "description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
5
5
  "keywords": [
6
6
  "test-failures",