tewip-ai 0.24.0 → 0.25.0

package/README.md

@@ -444,6 +444,80 @@ Every change is small (1 to 16 lines, median 4) and most are what the developers
 **No wrong fix was offered**: anything that could not be proved was thrown away and the file put
 back byte for byte.
 
+## One bundle a release manager can read (Phase 15)
+
+```bash
+tewip attest --out attestation.md
+```
+
+Everything Tewip knows about a build, in one file: which tests ran, what each failure was, which
+rule or which model decided it, **what it refused to decide**, and what it never looked at. The
+refusals are the point - a tool that never says "I could not tell" has nothing worth attesting,
+because every line it writes might be a guess.
+
+The bundle carries a digest, and an Ed25519 signature when you give it a key:
+
+```bash
+TEWIP_ATTEST_KEY=./private.pem tewip attest            # sign it
+tewip attest --verify attestation.json --key public.pem  # check it
+```
+
+A digest is not a signature and Tewip says so: without a key it tells you the bundle is undamaged,
+not that nobody edited it. Editing a claim out of a signed bundle makes verification fail and exit
+non-zero. Every CI run writes one of these automatically.
+
+## Is this incident covered by a test? (Phase 16)
+
+```bash
+tewip incident sentry-issue.json
+```
+
+Reads a Sentry payload, an OpenTelemetry span or a plain stack trace, finds the test that should
+have caught it, and answers with one of four verdicts: **reproduced**, **passed** (the test ran and
+did not catch it, which is worse news), **failed differently**, or **never ran**. It does not say
+"probably covered".
+
+## What did we decide about this last time? (Phase 17)
+
+```bash
+tewip recall "the login test that keeps flaking"
+```
+
+Every correction your team ever gave Tewip is kept. Ask, and it answers with your own decision from
+then, and says it was yours - not its. It never acts on a correction by itself.
+
+## From the ticket to the run (Phase 18)
+
+```bash
+tewip trace --only "SHOP-12"
+```
+
+Joins the test plans Tewip wrote to the tests that cover them and the runs that attempted them. A
+scenario counts as covered only when a real test can be named, and passing only when a real run
+attempted it; everything else says what is missing. No model, no cost.
+
+## A goal instead of a verb (Phases 19 and 21)
+
+```bash
+tewip pursue "keep the checkout flow green"
+tewip pursue "keep the checkout flow green" --act
+```
+
+Give Tewip a goal and it says what it would do next, and why, from what it can actually see. The
+order is fixed, published and testable - **not chosen by a model**: a real bug first, then the
+environment, then flakiness, then anything failing it cannot explain, then a coverage gap, and only
+then is the goal worth attesting. Deciding the same state twice gives the same answer.
+
+With `--act` it carries the decision out, and this is where the refusals matter most:
+
+- It **never does the work itself.** Each action is an existing command that already proves what it
+  changes - `write` keeps a test only when it passes *and* fails when broken.
+- **Two actions are on the allowlist**: write a missing test, and attest a green goal. Everything
+  else is refused by name. It will not work around a real bug.
+- **One action, then stop.** There is no unattended loop.
+- Everything it did or refused goes to `.tewip/actions.jsonl`, with the run that justified it.
+- It **will not repeat itself**: the same attempt is refused until a new run gives it new evidence.
+
 ## Where it has been tried
 
 | Project | How | Result |

package/dist/cli/help.js

@@ -105,8 +105,11 @@ Usage:
                                     green" - and it says what it would do next and why, from what
                                     it can actually see. The order is fixed and written down: a
                                     real bug first, then the environment, then flakiness, then a
-                                    coverage gap. It decides; it does not act unless you ask, and
-                                    even then only through the gates that prove a change.
+                                    coverage gap. It decides; it does not act unless you ask. With
+                                    --act it carries out the two safe ones - writing a missing test,
+                                    or attesting a green goal - by running the command that already
+                                    proves them. It will not touch a real bug, and it writes what it
+                                    did or refused to .tewip/actions.jsonl.
   tewip trace [--only "<ticket or title>"] [--out trace.md]
                                     Is this requirement covered, and is it passing right now? Joins
                                     the plans Tewip wrote to the tests that cover them and the runs

package/dist/engine/attest/attest-command.js

@@ -46,9 +46,18 @@ function history(root, only) {
 function whatWasChecked(records) {
     const checked = [];
     const notChecked = [];
-    checked.push('every failing test in the run was read, and each was given a cause or left undecided');
-    const byRules = records.filter((r) => r.diagnosis?.source === 'rules').length;
-    const byModel = records.filter((r) => r.diagnosis?.source === 'llm').length;
+    // Decided means settled: a rule that looked and could not say is a refusal, and counting it here
+    // would have a bundle claim more decisions than it lists claims.
+    const settledBy = (source) => records.filter((r) => {
+        const d = r.diagnosis;
+        return d?.source === source && d.category && d.category !== 'UNKNOWN';
+    }).length;
+    if (records.length === 0)
+        checked.push('every test in the run passed, so there was nothing to diagnose');
+    else
+        checked.push('every failing test in the run was read, and each was given a cause or left undecided');
+    const byRules = settledBy('rules');
+    const byModel = settledBy('llm');
     if (byRules > 0)
         checked.push(`${byRules} decided by deterministic rules (${ENGINE_VERSION}), which call no model and give the same answer every time`);
     if (byModel > 0)

package/dist/engine/pursue/act.js

@@ -0,0 +1,39 @@
+/**
+ * What Tewip may carry out, and with what. Anything absent is refused: reporting a real bug, for
+ * instance, is deliberately not an action - it is for a person to read.
+ */
+export function planFor(decision) {
+    switch (decision.act) {
+        case 'cover-gap': {
+            // `write` drafts the test, runs it, breaks it on purpose to see it fail, and keeps it only
+            // if both hold. The scenario id is in the decision's own next step.
+            const id = /write a test for ([\w-]+)/.exec(decision.next)?.[1];
+            return id ? { command: ['write', '--only', id], target: `scenario ${id}` } : null;
+        }
+        case 'attest':
+            return { command: ['attest'], target: 'the evidence bundle' };
+        default:
+            return null;
+    }
+}
+/** Why an action was not carried out, in words a person can act on. */
+export const cannotAct = {
+    'look-first': 'there is nothing to act on until a report has been read',
+    'triage-real-bug': 'a real bug is for a person to see, and Tewip will not work around one on its own',
+    'rerun-environment': 'a re-run belongs to whatever runs the tests, not to Tewip',
+    'settle-flaky': 'settling a flaky test changes a test file, which Tewip does only when asked directly with `tewip fix`',
+    'triage-undecided': 'Tewip cannot say why these failed, so it has nothing it could safely do about them',
+    'cover-gap': '',
+    attest: '',
+    'stand-down': 'there is nothing here that matches the goal',
+};
+/**
+ * Has this exact action already been tried against this exact evidence? A scheduled run that keeps
+ * attempting an impossible task is the failure mode that makes people turn an agent off. A new run
+ * is new evidence and unblocks it; running again over the same report does not.
+ *
+ * Refusals are not counted: refusing costs nothing and says nothing about whether it would work.
+ */
+export function alreadyTried(records, act, target, runId) {
+    return records.find((r) => r.act === act && r.target === target && r.runId === runId && r.outcome !== 'refused') ?? null;
+}

package/dist/engine/pursue/pursue-command.js

@@ -4,9 +4,11 @@
 // it chose, and only through the gates that already prove a change before keeping it. There is no
 // mode in which it loops unattended: an agent that keeps going until it decides to stop is the
 // thing teams roll back, and nothing here needs it.
-import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
+import { spawnSync } from 'node:child_process';
+import { appendFileSync, existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
 import { dirname, join, resolve } from 'node:path';
 import { datasetFiles, readJsonl, readTriageRecords } from '../../reporters/store.js';
+import { alreadyTried, cannotAct, planFor } from './act.js';
 import { decide } from './policy.js';
 const words = (text) => (text.toLowerCase().match(/[a-z][a-z0-9]{2,}/g) ?? []).filter((w) => !['the', 'and', 'for', 'this', 'that', 'with', 'keep', 'green', 'sprint', 'flow', 'test', 'tests'].includes(w));
 /** Does this test or scenario belong to the goal? Deterministic, so a person can predict it. */
@@ -53,6 +55,7 @@ export function observe(root, goal, only) {
         uncovered: uncoveredScenarios(root, wanted, attempted),
         passing: attempted.length - failing.length,
         noHistory: runs.length === 0,
+        runId: newest?.runId ?? '',
     };
 }
 /** Scenarios of a kept plan that no attempted test can be matched to, worst priority first. */
@@ -108,15 +111,100 @@ export function renderPursuit(goal, seen, decision, acting) {
         : 'Tewip has not done anything: this is a decision, and the action is yours.', '', 'The order of these decisions is fixed and written down, not chosen by a model: a real bug first,', 'then the environment, then flakiness, then a coverage gap, and only then is a goal worth attesting.');
     return lines.join('\n');
 }
+/** The audit file the team owns: every action Tewip took on its own, and every one it refused. */
+const auditPath = (root) => join(root, '.tewip', 'actions.jsonl');
+export function actionsTaken(root) {
+    try {
+        return readFileSync(auditPath(root), 'utf8')
+            .split(String.fromCharCode(10))
+            .filter(Boolean)
+            .map((l) => JSON.parse(l));
+    }
+    catch {
+        return [];
+    }
+}
+function record(root, entry) {
+    mkdirSync(dirname(auditPath(root)), { recursive: true });
+    appendFileSync(auditPath(root), JSON.stringify(entry) + String.fromCharCode(10), 'utf8');
+}
+/**
+ * The Tewip command line belonging to *this* build, found from this file rather than from how the
+ * process was started.
+ *
+ * This is not a detail. The first version used process.argv[1], which is the command line only when
+ * a person typed it: under a test runner argv[1] is the test file, so acting re-ran the test file,
+ * which acted again, which re-ran it - a fork bomb that froze the machine it was written on. What
+ * re-enters Tewip has to be Tewip, established here, or nothing is carried out at all.
+ */
+export function cliEntry() {
+    const here = import.meta.dirname;
+    for (const candidate of [join(here, '..', '..', 'cli', 'tewip.ts'), join(here, '..', '..', 'cli', 'tewip.js')]) {
+        if (existsSync(candidate))
+            return candidate;
+    }
+    return null;
+}
+/**
+ * Carries out the one action that was decided, by running the command that already proves it.
+ * Tewip does not do the work here; it asks the gate to.
+ */
+function carryOut(cmd, decision, observed) {
+    const plan = planFor(decision);
+    const runId = observed.runId ?? '';
+    const note = (outcome, detail, target = plan?.target ?? '') => ({
+        at: new Date().toISOString(),
+        goal: cmd.goal,
+        act: decision.act,
+        target,
+        command: plan ? 'tewip ' + plan.command.join(' ') : '',
+        runId,
+        outcome,
+        detail,
+    });
+    if (!plan) {
+        const why = cannotAct[decision.act] || 'this is not something Tewip carries out on its own';
+        record(cmd.root, note('refused', why, decision.act));
+        return 'Not carried out: ' + why + '.';
+    }
+    const tried = alreadyTried(actionsTaken(cmd.root), decision.act, plan.target, runId);
+    if (tried) {
+        record(cmd.root, note('refused', 'already attempted at ' + tried.at + ', on the same evidence'));
+        return ['Not carried out: Tewip already attempted this at ' + tried.at.slice(0, 19).replace('T', ' ') + ' UTC, and nothing has happened since that would make the outcome different.', '', 'That attempt ' + (tried.outcome === 'done' ? 'succeeded' : 'did not finish') + ': ' + tried.detail, '', 'Run ' + tried.command + ' yourself if you want it tried again.'].join(String.fromCharCode(10));
+    }
+    // Re-enter this same build, so an installed Tewip runs the installed gates and a checkout runs
+    // the checkout's - never a different version from the one that decided, and never whatever
+    // happens to have started this process.
+    const entry = cliEntry();
+    if (!entry) {
+        record(cmd.root, note('refused', 'the Tewip command line could not be found next to this build, so there is nothing safe to re-enter'));
+        return 'Not carried out: Tewip could not find its own command line beside this build, and it will not guess at one. Run ' + 'tewip ' + plan.command.join(' ') + ' yourself.';
+    }
+    const ran = spawnSync(process.execPath, [entry, ...plan.command], { cwd: cmd.root, encoding: 'utf8', timeout: 20 * 60_000, maxBuffer: 32 * 1024 * 1024 });
+    const output = ((ran.stdout ?? '') + (ran.stderr ?? '')).trim();
+    const ok = ran.status === 0;
+    // What the audit file remembers: the headline of a success, the tail of a failure. A failure's
+    // reason is at the end of what it printed; a success's is at the start.
+    const lines = output.split(String.fromCharCode(10)).map((l) => l.trim()).filter(Boolean);
+    const detail = (ok ? lines[0] ?? 'it printed nothing' : lines.slice(-3).join(' ')).slice(0, 300);
+    record(cmd.root, note(ok ? 'done' : 'failed', detail));
+    return [
+        ok ? 'Carried out: tewip ' + plan.command.join(' ') : 'Tried and did not finish: tewip ' + plan.command.join(' '),
+        '',
+        output.slice(-2000),
+        '',
+        ok
+            ? 'Anything kept by that command was proved by it; anything unproved was thrown away. Written to .tewip/actions.jsonl.'
+            : 'Nothing was kept. Written to .tewip/actions.jsonl.',
+    ].join(String.fromCharCode(10));
+}
 export function runPursue(cmd) {
     const observed = observe(cmd.root, cmd.goal, cmd.name);
     const decision = decide(cmd.goal, observed);
     const report = renderPursuit(cmd.goal, observed, decision, Boolean(cmd.act));
     console.log(report);
-    if (cmd.act && decision.changesFiles) {
-        console.error('\n`--act` is accepted, but Tewip carries out a change only through the command that proves it.');
-        console.error(`Run: ${decision.next}`);
-    }
+    if (cmd.act)
+        console.log(String.fromCharCode(10) + '---' + String.fromCharCode(10) + carryOut(cmd, decision, observed));
     if (cmd.out) {
         mkdirSync(dirname(resolve(cmd.out)), { recursive: true });
         writeFileSync(resolve(cmd.out), `${report}\n`, 'utf8');

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "tewip-ai",
-  "version": "0.24.0",
+  "version": "0.25.0",
   "description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
   "keywords": [
     "test-failures",