tewip-ai 0.29.0 → 0.31.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +98 -0
- package/dist/cli/help.js +16 -0
- package/dist/cli/tewip.js +24 -0
- package/dist/engine/pick/pick-command.js +198 -0
- package/dist/engine/pick/pick.js +117 -0
- package/dist/engine/prove/mutants.js +8 -4
- package/dist/engine/prove/prove.js +42 -5
- package/dist/engine/prove/reaches.js +111 -1
- package/dist/engine/schema/frameworks.js +230 -0
- package/dist/engine/schema/schema-command.js +174 -0
- package/dist/engine/schema/statements.js +164 -0
- package/dist/mcp/server.js +40 -0
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -552,6 +552,104 @@ With `--json` the verdict is **sealed**: editing `not-ready` to `ready` in the f
|
|
|
552
552
|
fail, so a verdict that travels into a ticket or a release note can be checked rather than trusted. It is
|
|
553
553
|
also an MCP tool, so an agent can ask whether its own work is done before saying that it is.
|
|
554
554
|
|
|
555
|
+
## What will this migration do to a live database? (Phase 26)
|
|
556
|
+
|
|
557
|
+
```bash
|
|
558
|
+
tewip schema --diff main...HEAD # or: tewip schema migrations/0002_drop_nickname.sql
|
|
559
|
+
tewip schema --strict # fails the build on a change that breaks the running version
|
|
560
|
+
```
|
|
561
|
+
|
|
562
|
+
The most expensive change in a pull request is usually not a test — it is a migration. A column dropped
|
|
563
|
+
while the previous deployment is still selecting it. An index built without `CONCURRENTLY`, blocking every
|
|
564
|
+
write to the busiest table. A `NOT NULL` added to a populated table. These are known, documented, and a
|
|
565
|
+
tool can name every one of them from the text of the migration.
|
|
566
|
+
|
|
567
|
+
```
|
|
568
|
+
1 of 1 migration file(s) read. 1 change(s) would break the running version or destroy data
|
|
569
|
+
|
|
570
|
+
## Breaks the version that is still running (1)
|
|
571
|
+
- drops the column nickname from orders — migrations/0002_drop_nickname.sql:1
|
|
572
|
+
- Instead: deploy code that no longer selects or writes the column, then drop it in a later
|
|
573
|
+
migration. During a rolling deploy the old version is still running and will fail on a column
|
|
574
|
+
that is gone
|
|
575
|
+
|
|
576
|
+
## Code that still uses what this takes away (1)
|
|
577
|
+
- orders.nickname is still named in 1 file(s): app/orders.py
|
|
578
|
+
```
|
|
579
|
+
|
|
580
|
+
That last section is the part no SQL linter does: it searches your own code for what the migration takes
|
|
581
|
+
away. "This drops `users.nickname`, and four files still read it" is a different sentence from "dropping a
|
|
582
|
+
column is risky", and it is the one that stops the outage.
|
|
583
|
+
|
|
584
|
+
It reads **SQL, Django, Rails and Alembic** migrations, judges **only the forward path** (a rollback step
|
|
585
|
+
contains the mirror image of the change, so reading it says a migration that creates a table destroys
|
|
586
|
+
one), and stays quiet when the team already did the safe thing — `atomic = False`, `algorithm:
|
|
587
|
+
:concurrently`, `NOT VALID`, `postgresql_concurrently=True`. When it cannot read a migration it says so
|
|
588
|
+
rather than passing it, and "read and nothing found" never looks like "not read".
|
|
589
|
+
|
|
590
|
+
**Tried on real projects.** 200 Zulip migrations: 199 read, 19 changes that break the running version or
|
|
591
|
+
destroy data, each sampled finding correct when read against the file — including a `DROP COLUMN` buried
|
|
592
|
+
in raw SQL inside a Python data migration. 60 Mastodon migrations: it caught a `change_column` that
|
|
593
|
+
rewrites a table, in a migration whose *name* says it only adds one.
|
|
594
|
+
|
|
595
|
+
What it does not know, and says so: your table sizes and how busy the database is, so it reports what a
|
|
596
|
+
statement **does** — rewrites the table, scans every row, blocks writes — and leaves the duration to the
|
|
597
|
+
person who knows the data. In CI the findings go into the pull request comment, and anything that breaks
|
|
598
|
+
the running version enters the evidence bundle as a refusal: a passing test suite does not verify a
|
|
599
|
+
migration.
|
|
600
|
+
|
|
601
|
+
## Run only the tests that could notice this change (Phase 25)
|
|
602
|
+
|
|
603
|
+
```bash
|
|
604
|
+
tewip pick --diff main...HEAD # or: tewip pick src/pricing.js
|
|
605
|
+
tewip pick --list # just the files, for piping into your runner
|
|
606
|
+
```
|
|
607
|
+
|
|
608
|
+
A forty-minute suite is a suite people stop running. Vendors sell this as a prediction - a model guesses
|
|
609
|
+
which tests matter - and a guess is the wrong shape for the job, because the cost of being wrong is a bug
|
|
610
|
+
reaching production and nobody can audit a guess.
|
|
611
|
+
|
|
612
|
+
Here, **skipping a test is a claim that it could not have noticed the change**, and a claim needs
|
|
613
|
+
evidence. A test runs when it imports a changed file, reaches one through another module, is edited by the
|
|
614
|
+
change itself, reads the repository at run time (a guard that scans every source file can be broken by any
|
|
615
|
+
file), or has failed recently — an unknown is never a skip. Everything else is left out with the reason
|
|
616
|
+
printed beside it.
|
|
617
|
+
|
|
618
|
+
**It refuses to narrow at all** when the change touches dependencies, build or test configuration, shared
|
|
619
|
+
setup or fixtures; when the suite drives a browser, because what a browser test exercises is not written
|
|
620
|
+
in its imports; or when a changed file cannot be traced to any test, since code is reachable in ways an
|
|
621
|
+
import does not show.
|
|
622
|
+
|
|
623
|
+
**Measured by breaking the code** (`eval/pick-safety.ts`): make a real mutation, ask `pick` what could
|
|
624
|
+
notice it, then run the **whole** suite to find out which tests really catch it. A failure is dropping a
|
|
625
|
+
catching test.
|
|
626
|
+
|
|
627
|
+
| Project | Judged cases | Catching test lost | Suite selected |
|
|
628
|
+
| --- | --- | --- | --- |
|
|
629
|
+
| This repository (node --test, 76 test files) | 8 | **0** | 7–9% |
|
|
630
|
+
| expressjs/express (Mocha, 88 test files) | 2 | **0** | refused to narrow — every test reaches the change |
|
|
631
|
+
|
|
632
|
+
Those two rows are the point. On a layered codebase it runs under a tenth of the suite and keeps every
|
|
633
|
+
test that would have caught the bug; on a monolith where every test imports the package front door it says
|
|
634
|
+
there is nothing to skip, rather than inventing a saving. It is also an MCP tool, so an agent can ask what
|
|
635
|
+
to run before running anything.
|
|
636
|
+
|
|
637
|
+
**In CI**, `mode: pick` runs before your tests and hands the workflow the files to run:
|
|
638
|
+
|
|
639
|
+
```yaml
|
|
640
|
+
- id: pick
|
|
641
|
+
uses: hmamut39/tewip@main
|
|
642
|
+
with: { mode: pick } # needs fetch-depth: 0 on actions/checkout
|
|
643
|
+
- run: npx jest ${{ steps.pick.outputs.tests }}
|
|
644
|
+
if: steps.pick.outputs.narrowed == 'true'
|
|
645
|
+
- run: npx jest
|
|
646
|
+
if: steps.pick.outputs.narrowed != 'true'
|
|
647
|
+
```
|
|
648
|
+
|
|
649
|
+
Whatever it leaves out is written into the job summary with the reason beside it, so "why did CI not run
|
|
650
|
+
that test" is answered by the run itself. When it will not narrow safely, `narrowed` is `false` and
|
|
651
|
+
`tests` is empty - so a workflow that ignores the flag still runs everything, which is the safe default.
|
|
652
|
+
|
|
555
653
|
## Would your tests catch a bug? (Phase 23)
|
|
556
654
|
|
|
557
655
|
```bash
|
package/dist/cli/help.js
CHANGED
|
@@ -118,6 +118,22 @@ Usage:
|
|
|
118
118
|
is what was asked for covered. Answers ready, not ready, or cannot
|
|
119
119
|
say - and cannot say is never rounded up to ready. --json writes a
|
|
120
120
|
sealed verdict that can be checked later. No model, no cost.
|
|
121
|
+
tewip schema [<files>] [--diff main...HEAD] [--strict] [--out schema.md]
|
|
122
|
+
What will this migration do to a live database? Names the changes
|
|
123
|
+
that break the version still running during a deploy, destroy data,
|
|
124
|
+
block writes while they run, or cannot be undone - and searches your
|
|
125
|
+
code for what still uses the column it takes away. Reads SQL, Django,
|
|
126
|
+
Rails and Alembic migrations, and says so when it cannot read one.
|
|
127
|
+
No model, no cost.
|
|
128
|
+
tewip pick [--diff main...HEAD] [--list] [--out picked.md]
|
|
129
|
+
Which tests could notice this change? Narrows a test run using
|
|
130
|
+
facts rather than a prediction: a test runs when it imports a
|
|
131
|
+
changed file or reaches one through another module, when the change
|
|
132
|
+
edits the test itself, or when it failed recently - an unknown is
|
|
133
|
+
never a skip. It runs the whole suite when the change touches
|
|
134
|
+
dependencies, build or test configuration, shared setup or
|
|
135
|
+
fixtures, or when a changed file cannot be traced to any test.
|
|
136
|
+
--list prints the files for piping. No model, no cost.
|
|
121
137
|
tewip prove <file> [--diff main...HEAD] [--limit 6] [--budget 600] [--strict]
|
|
122
138
|
What would your tests NOT catch? Breaks your code in small,
|
|
123
139
|
deliberate ways - a boundary moved by one, a condition inverted,
|
package/dist/cli/tewip.js
CHANGED
|
@@ -199,6 +199,30 @@ async function main() {
|
|
|
199
199
|
});
|
|
200
200
|
return;
|
|
201
201
|
}
|
|
202
|
+
case 'schema': {
|
|
203
|
+
const { runSchema } = await import('../engine/schema/schema-command.js');
|
|
204
|
+
runSchema({
|
|
205
|
+
root: process.cwd(),
|
|
206
|
+
diff: flag(args, 'diff') ?? 'main...HEAD',
|
|
207
|
+
files: args.filter((a, i) => !a.startsWith('--') && args[i - 1] !== '--diff' && args[i - 1] !== '--out'),
|
|
208
|
+
out: flag(args, 'out'),
|
|
209
|
+
strict: args.includes('--strict'),
|
|
210
|
+
});
|
|
211
|
+
return;
|
|
212
|
+
}
|
|
213
|
+
case 'pick': {
|
|
214
|
+
const { runPick } = await import('../engine/pick/pick-command.js');
|
|
215
|
+
runPick({
|
|
216
|
+
root: process.cwd(),
|
|
217
|
+
diff: flag(args, 'diff') ?? 'main...HEAD',
|
|
218
|
+
files: args.filter((a, i) => !a.startsWith('--') && args[i - 1] !== '--diff' && args[i - 1] !== '--out' && args[i - 1] !== '--recent' && args[i - 1] !== '--name'),
|
|
219
|
+
out: flag(args, 'out'),
|
|
220
|
+
list: args.includes('--list'),
|
|
221
|
+
recentRuns: Number(flag(args, 'recent') ?? '5'),
|
|
222
|
+
name: flag(args, 'name'),
|
|
223
|
+
});
|
|
224
|
+
return;
|
|
225
|
+
}
|
|
202
226
|
case 'prove': {
|
|
203
227
|
const paths = args.filter((a, i) => !a.startsWith('--') && !['--out', '--diff', '--limit', '--budget'].includes(args[i - 1] ?? ''));
|
|
204
228
|
const { runProve } = await import('../engine/prove/prove-command.js');
|
|
@@ -0,0 +1,198 @@
|
|
|
1
|
+
// `tewip pick --diff main...HEAD`: the tests that could notice a change, and the command that runs them.
|
|
2
|
+
//
|
|
3
|
+
// Everything here is a fact read from the project: the diff from git, the test files on disk, the import
|
|
4
|
+
// graph followed outwards from each changed file, and the failures Tewip has kept. Nothing is predicted.
|
|
5
|
+
import { spawnSync } from 'node:child_process';
|
|
6
|
+
import { existsSync, mkdirSync, readFileSync, readdirSync, writeFileSync } from 'node:fs';
|
|
7
|
+
import { dirname, join, relative, resolve } from 'node:path';
|
|
8
|
+
import { chooseFramework, detectFrameworks } from '../quality/frameworks.js';
|
|
9
|
+
import { changeSummary } from '../quality/grounding.js';
|
|
10
|
+
import { coverageFor, testFiles } from '../review/coverage.js';
|
|
11
|
+
import { importingTests } from '../prove/prove.js';
|
|
12
|
+
import { allTestsReaching } from '../prove/reaches.js';
|
|
13
|
+
import { datasetFiles, readJsonl, readTriageRecords } from '../../reporters/store.js';
|
|
14
|
+
import { pick, renderPick } from './pick.js';
|
|
15
|
+
const slash = (p) => p.split(String.fromCharCode(92)).join('/');
|
|
16
|
+
/** Tests that failed in the most recent runs, by the file they live in. */
|
|
17
|
+
function recentFailures(root, only, howMany) {
|
|
18
|
+
const dir = join(root, '.tewip', 'history');
|
|
19
|
+
const byFile = new Map();
|
|
20
|
+
if (!existsSync(dir))
|
|
21
|
+
return { byFile, noHistory: true };
|
|
22
|
+
const runs = [];
|
|
23
|
+
const records = [];
|
|
24
|
+
for (const project of only ? [only] : readdirSync(dir)) {
|
|
25
|
+
const where = join(dir, project);
|
|
26
|
+
if (!existsSync(where))
|
|
27
|
+
continue;
|
|
28
|
+
runs.push(...readJsonl(datasetFiles(where).runs));
|
|
29
|
+
records.push(...readTriageRecords(datasetFiles(where).triage));
|
|
30
|
+
}
|
|
31
|
+
if (runs.length === 0)
|
|
32
|
+
return { byFile, noHistory: true };
|
|
33
|
+
const recent = new Set(runs
|
|
34
|
+
.slice()
|
|
35
|
+
.sort((a, b) => (a.finishedAt > b.finishedAt ? -1 : 1))
|
|
36
|
+
.slice(0, howMany)
|
|
37
|
+
.map((r) => r.runId));
|
|
38
|
+
for (const record of records) {
|
|
39
|
+
if (!recent.has(record.runId))
|
|
40
|
+
continue;
|
|
41
|
+
const file = slash(String(record.file ?? ''));
|
|
42
|
+
if (!file)
|
|
43
|
+
continue;
|
|
44
|
+
byFile.set(file, [...(byFile.get(file) ?? []), record.testId]);
|
|
45
|
+
}
|
|
46
|
+
return { byFile, noHistory: false };
|
|
47
|
+
}
|
|
48
|
+
/** A test file's path as the history records it, which may be a bare name rather than a full path. */
|
|
49
|
+
const sameTest = (testFile, recorded) => testFile === recorded || testFile.endsWith(`/${recorded}`) || recorded.endsWith(`/${testFile}`);
|
|
50
|
+
/**
|
|
51
|
+
* Does this test walk the repository at run time?
|
|
52
|
+
*
|
|
53
|
+
* Such a test - a guard that scans every source file, a check on licence headers, a snapshot of the
|
|
54
|
+
* public API - can be broken by any file in the project, so it must never be skipped. This is not a
|
|
55
|
+
* guess at coverage: it is a statement about what the test reads. Found the hard way, by a safety
|
|
56
|
+
* measurement that caught this project's own control-character guard failing over a change to a module
|
|
57
|
+
* it does not import.
|
|
58
|
+
*/
|
|
59
|
+
function readsTheRepo(root, testFile) {
|
|
60
|
+
try {
|
|
61
|
+
const text = readFileSync(resolve(root, testFile), 'utf8');
|
|
62
|
+
const walks = /\breaddirSync\b|\bglob(?:Sync)?\s*\(|\bwalk\s*\(|os\.walk|Files\.walk|filepath\.Walk/.test(text);
|
|
63
|
+
const reads = /\breadFileSync\b|\bopen\s*\(|readFile\b/.test(text);
|
|
64
|
+
return walks && reads;
|
|
65
|
+
}
|
|
66
|
+
catch {
|
|
67
|
+
return false;
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
/**
|
|
71
|
+
* One command rather than a chain of them. Runners take several files at once, and a line of forty
|
|
72
|
+
* `&&`-joined invocations pays the start-up cost forty times - which is most of the saving the selection
|
|
73
|
+
* was for. The shared prefix is taken from the framework's own command for two files, so nothing is
|
|
74
|
+
* assumed about how it is spelled.
|
|
75
|
+
*/
|
|
76
|
+
function oneCommand(root, picked, framework) {
|
|
77
|
+
if (!framework)
|
|
78
|
+
return 'your test command';
|
|
79
|
+
if (picked.mustRun.length === 0)
|
|
80
|
+
return framework.suite;
|
|
81
|
+
const paths = picked.mustRun.map((m) => resolve(root, m.testFile));
|
|
82
|
+
const first = framework.command(paths[0]);
|
|
83
|
+
if (paths.length === 1)
|
|
84
|
+
return withCwd(root, framework.cwd, first);
|
|
85
|
+
// Everything before the first quoted argument is the runner. A longest-common-prefix of two commands
|
|
86
|
+
// looked cleverer and cut inside the first file name, producing `node --test "test/ "test/a.ts"`.
|
|
87
|
+
const quote = first.indexOf('"');
|
|
88
|
+
const prefix = quote > 0 ? first.slice(0, quote).trimEnd() : '';
|
|
89
|
+
if (!prefix)
|
|
90
|
+
return withCwd(root, framework.cwd, picked.mustRun.map((m) => framework.command(resolve(root, m.testFile))).join(' && '));
|
|
91
|
+
const quoted = paths.map((p) => JSON.stringify(relative(framework.cwd, p).split(String.fromCharCode(92)).join('/'))).join(' ');
|
|
92
|
+
return withCwd(root, framework.cwd, `${prefix} ${quoted}`);
|
|
93
|
+
}
|
|
94
|
+
/** Says where to run it, when that is not the project root. */
|
|
95
|
+
const withCwd = (root, cwd, command) => (resolve(cwd) === resolve(root) ? command : `cd ${JSON.stringify(relative(root, cwd).split(String.fromCharCode(92)).join('/'))} && ${command}`);
|
|
96
|
+
/** Test files that mention this file by path or by name, which is weaker than an import but not nothing. */
|
|
97
|
+
function mentionsOf(root, file, tests) {
|
|
98
|
+
// The name **with its extension**, or the path. Matching the bare stem was useless: for `cli/tewip.ts`
|
|
99
|
+
// the stem is "tewip", which appears in nearly every file of this project, so 28 test files were
|
|
100
|
+
// selected on the grounds that they had mentioned the product's own name.
|
|
101
|
+
const named = file.split('/').pop() ?? '';
|
|
102
|
+
if (named.length < 5)
|
|
103
|
+
return [];
|
|
104
|
+
const out = [];
|
|
105
|
+
for (const test of tests) {
|
|
106
|
+
try {
|
|
107
|
+
const text = readFileSync(resolve(root, test), 'utf8');
|
|
108
|
+
if (text.includes(file) || text.includes(named))
|
|
109
|
+
out.push(test);
|
|
110
|
+
}
|
|
111
|
+
catch {
|
|
112
|
+
// Unreadable: it cannot be shown to mention anything.
|
|
113
|
+
}
|
|
114
|
+
}
|
|
115
|
+
return out;
|
|
116
|
+
}
|
|
117
|
+
/** Does anything at all in the project refer to this file? Used only to tell "unknown" from "unreachable". */
|
|
118
|
+
function mentionedAnywhere(root, file) {
|
|
119
|
+
const named = file.split('/').pop() ?? '';
|
|
120
|
+
const where = spawnSync('git', ['grep', '-l', '--', named], { cwd: root, encoding: 'utf8', timeout: 60_000 });
|
|
121
|
+
const hits = (where.stdout ?? '')
|
|
122
|
+
.split(String.fromCharCode(10))
|
|
123
|
+
.map((l) => l.trim())
|
|
124
|
+
.filter((l) => l && l.split(String.fromCharCode(92)).join('/') !== file);
|
|
125
|
+
return hits.length > 0;
|
|
126
|
+
}
|
|
127
|
+
export function choose(cmd) {
|
|
128
|
+
const change = cmd.files && cmd.files.length > 0 ? { files: cmd.files } : changeSummary(cmd.root, cmd.diff);
|
|
129
|
+
if (!change)
|
|
130
|
+
return null;
|
|
131
|
+
const changed = change.files.map(slash);
|
|
132
|
+
const known = testFiles(cmd.root).map(slash);
|
|
133
|
+
const relative = known.map((t) => slash(t).replace(`${slash(cmd.root)}/`, ''));
|
|
134
|
+
const { byFile, noHistory } = recentFailures(cmd.root, cmd.name, cmd.recentRuns ?? 5);
|
|
135
|
+
const candidates = relative.map((testFile) => {
|
|
136
|
+
const failed = [];
|
|
137
|
+
for (const [file, ids] of byFile)
|
|
138
|
+
if (sameTest(testFile, file))
|
|
139
|
+
failed.push(...ids);
|
|
140
|
+
return { testFile, recentlyFailed: failed, readsTheRepo: readsTheRepo(cmd.root, testFile) };
|
|
141
|
+
});
|
|
142
|
+
// The import graph, followed outwards from each changed source file.
|
|
143
|
+
const reaching = new Map();
|
|
144
|
+
const sources = changed.filter((f) => existsSync(resolve(cmd.root, f)));
|
|
145
|
+
for (const file of sources) {
|
|
146
|
+
// Both kinds, together: a test may import the file itself and others reach it through the package.
|
|
147
|
+
// Taking only the direct importers when there are any is how a selection quietly drops the tests
|
|
148
|
+
// that exercise the code through the application's own front door.
|
|
149
|
+
const direct = importingTests(cmd.root, coverageFor(cmd.root, [file], testFiles(cmd.root)), file);
|
|
150
|
+
const reached = allTestsReaching(cmd.root, file, relative);
|
|
151
|
+
const byFile = new Map();
|
|
152
|
+
for (const r of reached)
|
|
153
|
+
byFile.set(slash(r.testFile), { testFile: slash(r.testFile), how: `reaches, through ${r.steps} import(s),` });
|
|
154
|
+
for (const testFile of direct)
|
|
155
|
+
byFile.set(slash(testFile), { testFile: slash(testFile), how: 'imports' });
|
|
156
|
+
reaching.set(file, [...byFile.values()]);
|
|
157
|
+
}
|
|
158
|
+
// A file nothing imports may still be named somewhere - a test that reads it as text, a path in a
|
|
159
|
+
// script, a dynamic import. A mention is not proof that a test exercises it, so a test that mentions
|
|
160
|
+
// it runs; and when nothing in the project mentions it at all, nothing can notice it.
|
|
161
|
+
const unreferenced = [];
|
|
162
|
+
for (const [file, reaches] of reaching) {
|
|
163
|
+
if (reaches.length > 0)
|
|
164
|
+
continue;
|
|
165
|
+
const named = mentionsOf(cmd.root, file, relative);
|
|
166
|
+
if (named.length > 0)
|
|
167
|
+
reaching.set(file, named.map((testFile) => ({ testFile, how: 'names' })));
|
|
168
|
+
else if (!mentionedAnywhere(cmd.root, file))
|
|
169
|
+
unreferenced.push(file);
|
|
170
|
+
}
|
|
171
|
+
const framework = chooseFramework(detectFrameworks(cmd.root), undefined, false) ?? detectFrameworks(cmd.root)[0] ?? null;
|
|
172
|
+
const picked = pick({ changed, tests: candidates, reaching, browserTests: Boolean(framework?.browser), noHistory, unreferenced });
|
|
173
|
+
const command = oneCommand(cmd.root, picked, framework);
|
|
174
|
+
return { picked, total: candidates.length, command, noHistory };
|
|
175
|
+
}
|
|
176
|
+
export function runPick(cmd) {
|
|
177
|
+
const chosen = choose(cmd);
|
|
178
|
+
if (!chosen)
|
|
179
|
+
throw new Error(`could not read the change "${cmd.diff}". Is this a git checkout, and does that range exist? (try --diff main...HEAD)`);
|
|
180
|
+
if (cmd.list) {
|
|
181
|
+
// For piping into a runner: the files, and nothing else. A run-everything decision prints nothing,
|
|
182
|
+
// so a script that pipes this cannot mistake "cannot narrow" for "no tests to run".
|
|
183
|
+
if (chosen.picked.runEverything) {
|
|
184
|
+
console.error(`tewip pick: ${chosen.picked.runEverything}. Run the whole suite.`);
|
|
185
|
+
process.exitCode = 2;
|
|
186
|
+
return;
|
|
187
|
+
}
|
|
188
|
+
for (const must of chosen.picked.mustRun)
|
|
189
|
+
console.log(must.testFile);
|
|
190
|
+
return;
|
|
191
|
+
}
|
|
192
|
+
const report = renderPick(chosen.picked, chosen.total, chosen.command, chosen.noHistory);
|
|
193
|
+
console.log(report);
|
|
194
|
+
if (cmd.out) {
|
|
195
|
+
mkdirSync(dirname(resolve(cmd.out)), { recursive: true });
|
|
196
|
+
writeFileSync(resolve(cmd.out), report + String.fromCharCode(10), 'utf8');
|
|
197
|
+
}
|
|
198
|
+
}
|
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
/** A changed file that can affect anything, so no test can be ruled out because of it. */
|
|
2
|
+
const GLOBAL = [
|
|
3
|
+
{ match: /(^|\/)(package|package-lock|yarn|pnpm-lock|bun\.lock|requirements|pyproject|poetry\.lock|go\.mod|go\.sum|pom\.xml|build\.gradle|Gemfile|composer\.json|Cargo\.toml)/i, why: 'it changes the dependencies, which every test runs against' },
|
|
4
|
+
{ match: /(^|\/)(\.github|\.gitlab-ci|Jenkinsfile|azure-pipelines|\.circleci|Dockerfile|docker-compose|Makefile)/i, why: 'it changes how the suite is built or run' },
|
|
5
|
+
{ match: /(^|\/)(jest|vitest|playwright|cypress|karma|babel|tsconfig|webpack|vite|rollup|eslint|tox|pytest|conftest)\b[\w.]*\.(json|js|ts|cjs|mjs|ini|cfg|toml|yaml|yml)$/i, why: 'it is test or build configuration, which applies to the whole suite' },
|
|
6
|
+
{ match: /(^|\/)(conftest\.py|setup\.(js|ts|py)|global-setup\.[jt]s|jest\.setup\.[jt]s|test-setup\.[jt]s)$/i, why: 'it is shared setup, which runs before every test' },
|
|
7
|
+
{ match: /(^|\/)(fixtures?|helpers?|support|mocks?|__mocks__|testdata|factories)\//i, why: 'it is shared test support, which any test may use' },
|
|
8
|
+
];
|
|
9
|
+
const SOURCE = /\.(js|jsx|mjs|cjs|ts|tsx|mts|cts|py|java|kt|go|cs|rb|php|vue|svelte|scala|swift)$/i;
|
|
10
|
+
const IS_TEST = /(^|\/)(tests?|spec|specs|__tests__|e2e)\/|[._-](test|tests|spec|specs)\.[a-z]+$|(^|\/)test_[^/]+\.py$|_test\.(go|py|rb)$|Tests?\.(java|kt|cs)$/i;
|
|
11
|
+
/** Documentation and assets: a test cannot notice these, so they never widen the run. */
|
|
12
|
+
const INERT = /\.(md|mdx|txt|rst|png|jpe?g|gif|svg|ico|webp|woff2?|ttf|eot|mp4|pdf|lock|csv)$|(^|\/)(LICENSE|CODEOWNERS|\.gitignore|\.editorconfig|\.prettierrc)/i;
|
|
13
|
+
export function pick(input) {
|
|
14
|
+
// Global first, then inert. The other order called `requirements.txt` documentation because of its
|
|
15
|
+
// extension and then reported that a dependency change could not affect any test.
|
|
16
|
+
const changed = input.changed.filter((f) => GLOBAL.some((g) => g.match.test(f)) || !INERT.test(f));
|
|
17
|
+
const picked = { mustRun: [], skipped: [], runEverything: '', opaque: [] };
|
|
18
|
+
if (changed.length === 0) {
|
|
19
|
+
return { ...picked, runEverything: 'nothing in this change can affect a test result - only documentation, assets or lockless files were touched. Running nothing is also an option.' };
|
|
20
|
+
}
|
|
21
|
+
// Anything global stops selection outright, and names the file that did it.
|
|
22
|
+
for (const file of changed) {
|
|
23
|
+
const global = GLOBAL.find((g) => g.match.test(file));
|
|
24
|
+
if (global)
|
|
25
|
+
return { ...picked, runEverything: `${file} cannot be narrowed around: ${global.why}` };
|
|
26
|
+
}
|
|
27
|
+
const changedSource = changed.filter((f) => SOURCE.test(f) && !IS_TEST.test(f));
|
|
28
|
+
const changedTests = changed.filter((f) => IS_TEST.test(f));
|
|
29
|
+
// A changed file nothing can be traced from. For a browser suite this is the normal case rather than a
|
|
30
|
+
// gap - the test reaches the code through the running application - so the honest answer is to run
|
|
31
|
+
// everything and say why, not to claim the tests are unrelated.
|
|
32
|
+
// A file no test imports may still be reachable in a way imports do not show. But a file that nothing
|
|
33
|
+
// in the project refers to **at all** - not imported, not named in any source or test - cannot be
|
|
34
|
+
// noticed by any test, and treating that as "cannot tell" made the command useless: in this very
|
|
35
|
+
// repository a change to the command-line dispatcher, which tests only read as text, forced the whole
|
|
36
|
+
// suite. So the two cases are separated, and only a genuine unknown stops selection.
|
|
37
|
+
picked.opaque = changedSource.filter((f) => (input.reaching.get(f) ?? []).length === 0 && !input.unreferenced.includes(f));
|
|
38
|
+
if (picked.opaque.length > 0) {
|
|
39
|
+
return {
|
|
40
|
+
...picked,
|
|
41
|
+
runEverything: input.browserTests
|
|
42
|
+
? `these tests drive a browser, so what a test exercises is not written in its imports: ${picked.opaque.slice(0, 3).join(', ')} cannot be traced to any test, and skipping on that basis would be a guess`
|
|
43
|
+
: `no test can be traced to ${picked.opaque.slice(0, 3).join(', ')}${picked.opaque.length > 3 ? ` and ${picked.opaque.length - 3} more` : ''}. Tewip will not take that as proof the suite is unaffected - code can be reached in ways an import does not show (dependency injection, a plugin registry, a dynamic import)`,
|
|
44
|
+
};
|
|
45
|
+
}
|
|
46
|
+
const why = new Map();
|
|
47
|
+
const add = (testFile, reason) => {
|
|
48
|
+
if (!why.has(testFile))
|
|
49
|
+
why.set(testFile, reason);
|
|
50
|
+
};
|
|
51
|
+
// A test the change itself edited always runs: whatever it now checks, nobody has seen it run yet.
|
|
52
|
+
for (const file of changedTests)
|
|
53
|
+
add(file, 'this change edits it');
|
|
54
|
+
for (const [file, reaches] of input.reaching) {
|
|
55
|
+
if (!changedSource.includes(file))
|
|
56
|
+
continue;
|
|
57
|
+
for (const reach of reaches)
|
|
58
|
+
add(reach.testFile, `it ${reach.how} ${file}`);
|
|
59
|
+
}
|
|
60
|
+
// A test that walks the repository can be broken by any file in it, so it always runs.
|
|
61
|
+
for (const test of input.tests) {
|
|
62
|
+
if (test.readsTheRepo)
|
|
63
|
+
add(test.testFile, 'it reads the repository itself, so any changed file can break it');
|
|
64
|
+
}
|
|
65
|
+
// An unknown is not a skip. A test that has been failing or flaking lately gets another run, because
|
|
66
|
+
// "it is unrelated to this change" and "it is broken for another reason" are different statements and
|
|
67
|
+
// only the first is a reason to skip.
|
|
68
|
+
for (const test of input.tests) {
|
|
69
|
+
if (test.recentlyFailed.length > 0)
|
|
70
|
+
add(test.testFile, `it failed recently (${test.recentlyFailed.length} time(s)), so its state is unknown rather than unrelated`);
|
|
71
|
+
}
|
|
72
|
+
picked.mustRun = [...why.entries()].map(([testFile, reason]) => ({ testFile, why: reason })).sort((a, b) => a.testFile.localeCompare(b.testFile));
|
|
73
|
+
picked.skipped = input.tests
|
|
74
|
+
.filter((t) => !why.has(t.testFile))
|
|
75
|
+
.map((t) => ({ testFile: t.testFile, why: 'nothing it imports, directly or through another module, reaches anything this change touched' }))
|
|
76
|
+
.sort((a, b) => a.testFile.localeCompare(b.testFile));
|
|
77
|
+
if (picked.mustRun.length === 0) {
|
|
78
|
+
return {
|
|
79
|
+
...picked,
|
|
80
|
+
runEverything: `nothing in this project refers to ${changed.slice(0, 3).join(', ')}, so no test can notice the change. Running nothing is defensible; running the suite costs nothing but time.`,
|
|
81
|
+
};
|
|
82
|
+
}
|
|
83
|
+
// Selecting everything is not a failure, but it should be said plainly rather than dressed up as a saving.
|
|
84
|
+
if (picked.skipped.length === 0 && picked.mustRun.length > 0) {
|
|
85
|
+
picked.runEverything = 'every test in the project reaches something this change touched, so there is nothing to skip';
|
|
86
|
+
}
|
|
87
|
+
if (input.noHistory && picked.skipped.length > 0) {
|
|
88
|
+
picked.runEverything = picked.runEverything || '';
|
|
89
|
+
}
|
|
90
|
+
return picked;
|
|
91
|
+
}
|
|
92
|
+
/** What the selection is worth, and what it rests on. */
|
|
93
|
+
export function renderPick(picked, total, command, noHistory) {
|
|
94
|
+
const lines = ['# Which tests could notice this change?', ''];
|
|
95
|
+
if (picked.runEverything) {
|
|
96
|
+
lines.push(`**Run the whole suite.** ${picked.runEverything}.`, '', 'Tewip narrows a test run only when it can show that the tests it leaves out could not have noticed', 'the change. It cannot show that here, so it is not asking you to take the risk.', '');
|
|
97
|
+
if (picked.opaque.length > 0) {
|
|
98
|
+
lines.push('Changed files nothing could be traced from:', '', ...picked.opaque.slice(0, 10).map((f) => `- ${f}`), '');
|
|
99
|
+
}
|
|
100
|
+
return lines.concat(tail(noHistory)).join(String.fromCharCode(10));
|
|
101
|
+
}
|
|
102
|
+
const kept = picked.mustRun.length;
|
|
103
|
+
const share = total > 0 ? Math.round((kept / total) * 100) : 100;
|
|
104
|
+
lines.push(`**${kept} of ${total} test file(s)** could notice it - ${share}% of the suite. ${picked.skipped.length} can be left out.`, '', '```', command, '```', '', `## Why these ${kept} run`, '', ...picked.mustRun.slice(0, 30).map((m) => `- \`${m.testFile}\` — ${m.why}`), ...(picked.mustRun.length > 30 ? [`- …and ${picked.mustRun.length - 30} more`] : []), '', `## What you are giving up (${picked.skipped.length})`, '', 'Each of these is left out because of a fact about the source, not a prediction:', '', ...picked.skipped.slice(0, 15).map((s) => `- \`${s.testFile}\``), ...(picked.skipped.length > 15 ? [`- …and ${picked.skipped.length - 15} more`] : []), '');
|
|
105
|
+
return lines.concat(tail(noHistory)).join(String.fromCharCode(10));
|
|
106
|
+
}
|
|
107
|
+
const tail = (noHistory) => [
|
|
108
|
+
'---',
|
|
109
|
+
'',
|
|
110
|
+
'How this decides: a test runs when it imports a changed file, or reaches one through another module,',
|
|
111
|
+
'or when the change edits the test itself, or when it has failed recently - an unknown is never a skip.',
|
|
112
|
+
'It runs everything when the change touches dependencies, build or test configuration, shared setup or',
|
|
113
|
+
'shared fixtures, or when a changed file cannot be traced to any test at all. No model is called.',
|
|
114
|
+
...(noHistory
|
|
115
|
+
? ['', 'This project keeps no run history, so "it has not failed lately" could not be checked for any test.', 'Run `tewip explain` on your CI reports and the next selection will know more.']
|
|
116
|
+
: []),
|
|
117
|
+
];
|
|
@@ -24,10 +24,14 @@ const NOT_BEHAVIOUR = /^\s*(\/\/|#|\*|\/\*|@|import\b|from\b|export\s+(type|inte
|
|
|
24
24
|
* sign the wrong way round, a flag returned the wrong way.
|
|
25
25
|
*/
|
|
26
26
|
const OPERATORS = [
|
|
27
|
-
|
|
28
|
-
{
|
|
29
|
-
|
|
30
|
-
|
|
27
|
+
// Whitespace on both sides, which is what separates a comparison from a generic type. Without it,
|
|
28
|
+
// `Array<{ id: string }>` was read as a comparison and mutated into `Array<={ … }>`: the file stopped
|
|
29
|
+
// compiling, the whole suite failed to load, and the measurement counted a mutant nothing caught.
|
|
30
|
+
// A codebase that writes `a<b` loses a few mutants; one that breaks its build loses the answer.
|
|
31
|
+
{ kind: 'a boundary moved by one', find: /(?<=\s)<=(?=\s)/, swap: () => '<' },
|
|
32
|
+
{ kind: 'a boundary moved by one', find: /(?<=\s)>=(?=\s)/, swap: () => '>' },
|
|
33
|
+
{ kind: 'a boundary moved by one', find: /(?<=\s)<(?=\s)/, swap: () => '<=' },
|
|
34
|
+
{ kind: 'a boundary moved by one', find: /(?<=\s)>(?=\s)/, swap: () => '>=' },
|
|
31
35
|
{ kind: 'a comparison inverted', find: /===/, swap: () => '!==' },
|
|
32
36
|
{ kind: 'a comparison inverted', find: /(?<![!=<>])==(?!=)/, swap: () => '!=' },
|
|
33
37
|
{ kind: 'a comparison inverted', find: /!==/, swap: () => '===' },
|
|
@@ -80,10 +80,11 @@ export function proveFile(root, file, covering, framework, opts = {}) {
|
|
|
80
80
|
return finish(proof, started);
|
|
81
81
|
}
|
|
82
82
|
const original = readFileSync(full, 'utf8');
|
|
83
|
+
const stopGuarding = guardAgainstBeingKilled(full, original);
|
|
83
84
|
const mutants = mutantsFor(original, limit);
|
|
84
85
|
if (mutants.length === 0) {
|
|
85
86
|
proof.blocked = 'nothing in this file could be changed in a way a test should be expected to catch - no comparison, no condition, no returned value outside of strings and comments';
|
|
86
|
-
return finish(proof, started);
|
|
87
|
+
return finish(proof, started, stopGuarding);
|
|
87
88
|
}
|
|
88
89
|
// A test has to pass before it can be asked to notice anything, or a red suite turns into "your
|
|
89
90
|
// tests catch nothing", which is the worst kind of lie this command could tell.
|
|
@@ -103,7 +104,7 @@ export function proveFile(root, file, covering, framework, opts = {}) {
|
|
|
103
104
|
}
|
|
104
105
|
if (usable.length === 0) {
|
|
105
106
|
proof.blocked = `none of the ${covering.length} test file(s) covering this pass as they are, so none can be asked whether they would notice a change. Fix the run first.`;
|
|
106
|
-
return finish(proof, started);
|
|
107
|
+
return finish(proof, started, stopGuarding);
|
|
107
108
|
}
|
|
108
109
|
proof.covering = usable;
|
|
109
110
|
// The control: break the file outright and check that a covering test notices. Without this, a file
|
|
@@ -111,7 +112,7 @@ export function proveFile(root, file, covering, framework, opts = {}) {
|
|
|
111
112
|
say('checking the tests really load this file');
|
|
112
113
|
if (!testsLoadIt(root, full, original, usable[0], framework, runMs)) {
|
|
113
114
|
proof.blocked = `${usable[0]} passes even when this file is made unusable, so the tests are not loading the file that was edited - they are reading an installed copy or a build output. Nothing here can be proved until they run against this source (for a Python src layout, install it in editable mode: pip install -e .).`;
|
|
114
|
-
return finish(proof, started);
|
|
115
|
+
return finish(proof, started, stopGuarding);
|
|
115
116
|
}
|
|
116
117
|
// Only the tests that pass may judge a mutant. A test that is already failing fails for every
|
|
117
118
|
// change too, which would have it "catch" everything - the exact false confidence this command
|
|
@@ -156,7 +157,7 @@ export function proveFile(root, file, covering, framework, opts = {}) {
|
|
|
156
157
|
}
|
|
157
158
|
proof.tried.push({ mutant, fate, noticedBy, seconds: Math.round((Date.now() - at) / 100) / 10 });
|
|
158
159
|
}
|
|
159
|
-
return finish(proof, started);
|
|
160
|
+
return finish(proof, started, stopGuarding);
|
|
160
161
|
}
|
|
161
162
|
/**
|
|
162
163
|
* Puts the file back and checks that it really went back. Anything less would mean this command can
|
|
@@ -168,7 +169,43 @@ function restore(full, original) {
|
|
|
168
169
|
throw new Error(`${full} could not be put back as it was. Check it before doing anything else: git checkout -- "${full}"`);
|
|
169
170
|
}
|
|
170
171
|
}
|
|
171
|
-
|
|
172
|
+
/**
|
|
173
|
+
* Put the file back even when the process is killed.
|
|
174
|
+
*
|
|
175
|
+
* `finally` does not run for SIGINT or SIGTERM, so a developer pressing Ctrl-C during a run - which they
|
|
176
|
+
* will, because this is the slow command - was left with a mutated source file and no idea why their
|
|
177
|
+
* code had stopped working. Found by killing one of these runs during development and finding a flipped
|
|
178
|
+
* comparison still sitting in the repository.
|
|
179
|
+
*/
|
|
180
|
+
function guardAgainstBeingKilled(full, original) {
|
|
181
|
+
const putBack = (signal) => () => {
|
|
182
|
+
try {
|
|
183
|
+
restore(full, original);
|
|
184
|
+
process.stderr.write(`${String.fromCharCode(10)}${signal}: ${full} was put back as it was.${String.fromCharCode(10)}`);
|
|
185
|
+
}
|
|
186
|
+
catch {
|
|
187
|
+
process.stderr.write(`${String.fromCharCode(10)}${signal}: could not put ${full} back. Run: git checkout -- "${full}"${String.fromCharCode(10)}`);
|
|
188
|
+
}
|
|
189
|
+
process.exit(130);
|
|
190
|
+
};
|
|
191
|
+
const handlers = ['SIGINT', 'SIGTERM', 'SIGHUP', 'SIGBREAK'].map((signal) => [signal, putBack(signal)]);
|
|
192
|
+
for (const [signal, handler] of handlers) {
|
|
193
|
+
try {
|
|
194
|
+
process.on(signal, handler);
|
|
195
|
+
}
|
|
196
|
+
catch {
|
|
197
|
+
// Not every signal exists on every platform.
|
|
198
|
+
}
|
|
199
|
+
}
|
|
200
|
+
return () => {
|
|
201
|
+
for (const [signal, handler] of handlers)
|
|
202
|
+
process.off(signal, handler);
|
|
203
|
+
};
|
|
204
|
+
}
|
|
205
|
+
const finish = (proof, started, stopGuarding) => {
|
|
206
|
+
stopGuarding?.();
|
|
207
|
+
return { ...proof, seconds: Math.round((Date.now() - started) / 1000) };
|
|
208
|
+
};
|
|
172
209
|
const firstLine = (output) => output.split(String.fromCharCode(10)).map((l) => l.trim()).find(Boolean)?.slice(0, 160) ?? '';
|
|
173
210
|
/** Which of the tests that cover a file really import it, and so could notice a change. */
|
|
174
211
|
export function importingTests(root, coverage, file) {
|