tewip-ai 0.70.14 → 0.70.16
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +60 -14
- package/dist/engine/prove/mutants.js +18 -2
- package/dist/engine/release/channels.js +85 -0
- package/dist/engine/rules.js +32 -1
- package/dist/engine/schema/frameworks.js +32 -1
- package/dist/engine/schema/kept.js +87 -0
- package/dist/engine/schema/migration-files.js +27 -3
- package/dist/engine/schema/schema-command.js +6 -56
- package/dist/engine/schema/statements.js +43 -2
- package/dist/engine/secure/secrets.js +81 -4
- package/dist/engine/secure/triage.js +9 -0
- package/dist/engine/vouch/find-tests.js +2 -2
- package/dist/engine/vouch/tests-in.js +32 -0
- package/dist/engine/vouch/text.js +70 -6
- package/dist/engine/watch/watch-command.js +16 -4
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -1035,20 +1035,32 @@ import does not show.
|
|
|
1035
1035
|
notice it, then run the **whole** suite to find out which tests really catch it. A failure is dropping a
|
|
1036
1036
|
catching test.
|
|
1037
1037
|
|
|
1038
|
-
Re-derived on 2026-10-
|
|
1039
|
-
|
|
1040
|
-
|
|
1041
|
-
|
|
1042
|
-
|
|
1043
|
-
|
|
1044
|
-
|
|
1045
|
-
|
|
1046
|
-
|
|
1047
|
-
|
|
1048
|
-
|
|
1049
|
-
|
|
1050
|
-
|
|
1051
|
-
|
|
1038
|
+
Re-derived on 2026-10-19 over ten modules against a suite of 1,288 tests, and redone from scratch
|
|
1039
|
+
because Phase 92 found the mutation engine could write something other than what it reported: **28
|
|
1040
|
+
real changes, 19 of them caught by the full suite and so able to judge a selection, 0 catching tests
|
|
1041
|
+
lost.** The other 9 were caught by nothing at all and are excluded rather than counted as successes -
|
|
1042
|
+
a selection cannot be shown safe by a change no test noticed.
|
|
1043
|
+
|
|
1044
|
+
**The saving is not one number, and the 5.7% published before was not representative.** It depends on
|
|
1045
|
+
how central the changed file is, which is the truth about what your tests can reach rather than a
|
|
1046
|
+
weakness in the selection:
|
|
1047
|
+
|
|
1048
|
+
| changed file | test files selected (of 101) | share |
|
|
1049
|
+
| --- | --- | --- |
|
|
1050
|
+
| `engine/release/channels.ts` | 4 | 4% |
|
|
1051
|
+
| `engine/flaky/flaky.ts`, `engine/perf/compare.ts` | 5 | 5% |
|
|
1052
|
+
| `engine/secure/secrets.ts` | 12 | 12% |
|
|
1053
|
+
| `engine/prove/mutants.ts` | 31 | 31% |
|
|
1054
|
+
| `engine/vouch/vouch.ts` | 44 | 44% |
|
|
1055
|
+
| `engine/schema/statements.ts` | 47 | 47% |
|
|
1056
|
+
| `engine/rules.ts`, `engine/import/junit.ts` | 52 | 52% |
|
|
1057
|
+
|
|
1058
|
+
Average across the nineteen judged changes: **27.9%**. Change a leaf module and the run narrows to a
|
|
1059
|
+
twentieth of the suite; change the rule engine and half of it legitimately reaches the change. The
|
|
1060
|
+
earlier figure was measured on peripheral files and sat at the bottom of that range, which is the
|
|
1061
|
+
wrong number to plan a CI budget against. On a monolith where every test imports the package front
|
|
1062
|
+
door it says there is nothing to skip, rather than inventing a saving. It is also an MCP tool, so an
|
|
1063
|
+
agent can ask what to run before running anything.
|
|
1052
1064
|
|
|
1053
1065
|
**In CI**, `mode: pick` runs before your tests and hands the workflow the files to run:
|
|
1054
1066
|
|
|
@@ -1276,6 +1288,40 @@ The first matching rule, in this order, decides the category. If a lower-priorit
|
|
|
1276
1288
|
| `timing-flaky` | TIMING | the failure rate is 20%-80% over at least 3 runs, and the error message varies |
|
|
1277
1289
|
| `real-bug-assertion` | REAL_BUG | a value mismatch on a test whose history is stable: it was clean before and has failed the same way since |
|
|
1278
1290
|
|
|
1291
|
+
### How often each rule is right
|
|
1292
|
+
|
|
1293
|
+
Measured over the six labelled datasets with `node eval/calibration.ts`, rules only, no model. The
|
|
1294
|
+
`claimed` column is the confidence the rule prints beside its answer; `observed` is how often that
|
|
1295
|
+
answer matched a hand label. A rule that claims more than it delivers is the one kind of error this
|
|
1296
|
+
project will not keep.
|
|
1297
|
+
|
|
1298
|
+
| rule | cases | right | observed | claimed | projects |
|
|
1299
|
+
|---|---|---|---|---|---|
|
|
1300
|
+
| `timing-retry-pass` | 1478 | 1463 | 99.0% | 0.74 | 6 |
|
|
1301
|
+
| `env-network` | 395 | 395 | 100.0% | 0.80 | 5 |
|
|
1302
|
+
| `env-server-error` | 43 | 43 | 100.0% | 0.60 | 5 |
|
|
1303
|
+
| `timing-intercepted` | 32 | 32 | 100.0% | 0.80 | 5 |
|
|
1304
|
+
| `real-bug-assertion` | 26 | 10 | 38.5% | 0.40 | 5 |
|
|
1305
|
+
| `locator-drift-strict` | 17 | 17 | 100.0% | 0.89 | 2 |
|
|
1306
|
+
| `timing-flaky` | 9 | 9 | 100.0% | 0.90 | 1 |
|
|
1307
|
+
| `test-data-collision` | 6 | 6 | 100.0% | 0.70 | 1 |
|
|
1308
|
+
| `locator-drift` | 5 | 3 | 60.0% | 0.80 | 2 |
|
|
1309
|
+
|
|
1310
|
+
Every rule but two understates itself, which is safe and not free: the merge gate trusts 0.7 and
|
|
1311
|
+
auto-fix needs 0.85, so a rule right 395 times out of 395 while claiming 0.80 sends work to a person
|
|
1312
|
+
who did not need to see it.
|
|
1313
|
+
|
|
1314
|
+
`real-bug-assertion` was the exception, and it is the rule that tells a team their application is
|
|
1315
|
+
broken. It claimed `0.65 + 0.05 x failingStreak`, up to 0.9, and was right 10 times in 26 - all 16
|
|
1316
|
+
misses a flaky test called a real bug. The streak term was not merely unearned but inverted: at a
|
|
1317
|
+
streak of 1 the rule is right 62.5% across five projects, and at 2 or more it is right none of ten.
|
|
1318
|
+
It now claims 0.40, which is how often it is right, and sits below both thresholds - in front of a
|
|
1319
|
+
person, where a two-in-five signal belongs. Nothing it decides changed: the rules layer answers the
|
|
1320
|
+
same 2,012 failures and gets the same 1,979 right.
|
|
1321
|
+
|
|
1322
|
+
`locator-drift` at 60% is measured and left alone: five cases across two projects is not enough to
|
|
1323
|
+
move a number, and saying so is better than either ignoring it or inventing a fix.
|
|
1324
|
+
|
|
1279
1325
|
History is **point-in-time**: each failure is judged only with runs up to and including its own run, never later ones. A test's first-ever failure therefore has no history, and history-based rules decline it.
|
|
1280
1326
|
|
|
1281
1327
|
## Usage
|
|
@@ -153,12 +153,28 @@ export function spread(all, limit) {
|
|
|
153
153
|
const step = all.length / limit;
|
|
154
154
|
return Array.from({ length: limit }, (unused, i) => all[Math.floor(i * step)]);
|
|
155
155
|
}
|
|
156
|
-
/**
|
|
156
|
+
/**
|
|
157
|
+
* The file with this one mutant applied, ready to be written and then put back.
|
|
158
|
+
*
|
|
159
|
+
* The replacement is spliced in by index rather than through `String.replace`. With a string
|
|
160
|
+
* pattern, `replace` still reads `$$`, `$&`, `` $` `` and `$'` in the *replacement* as substitution
|
|
161
|
+
* patterns, so the text written to the file was not always the text the mutant said it was.
|
|
162
|
+
*
|
|
163
|
+
* Swept over 61,630 mutants across 3,696 real files from the corpus, 8 came out wrong, in three
|
|
164
|
+
* shapes: Leaflet's bundled `touch$$1` became `touch$1`, which is a different variable; and two of
|
|
165
|
+
* Saleor's workflow patterns, `'^\.github/workflows/e2e\.yml$'` and `'^[[:space:]]*$'`, lost their
|
|
166
|
+
* trailing `$'` to the "everything after the match" pattern.
|
|
167
|
+
*
|
|
168
|
+
* Eight in sixty thousand sounds like nothing until you read what it is: this is the one place
|
|
169
|
+
* Tewip writes to a file nobody asked it to touch, and it was reporting one change while making
|
|
170
|
+
* another. A mutant that mangles an identifier is not the mutant whose survival is being judged.
|
|
171
|
+
*/
|
|
157
172
|
export function applyMutant(text, mutant) {
|
|
158
173
|
const lines = text.split(String.fromCharCode(10));
|
|
159
174
|
const line = lines[mutant.line - 1];
|
|
160
175
|
if (line === undefined || line.trim() !== mutant.was)
|
|
161
176
|
throw new Error(`line ${mutant.line} is not what the mutant was made from, so nothing was written`);
|
|
162
|
-
|
|
177
|
+
const at = line.indexOf(mutant.was);
|
|
178
|
+
lines[mutant.line - 1] = at < 0 ? line : `${line.slice(0, at)}${mutant.now}${line.slice(at + mutant.was.length)}`;
|
|
163
179
|
return lines.join(String.fromCharCode(10));
|
|
164
180
|
}
|
|
@@ -0,0 +1,85 @@
|
|
|
1
|
+
// Is every place Tewip is published actually carrying this release?
|
|
2
|
+
//
|
|
3
|
+
// There is a standing rule on this project: check every distribution channel after every
|
|
4
|
+
// improvement, because a stale channel looks exactly like a current one. It was being done by hand,
|
|
5
|
+
// and by hand it failed twice in one day - not because the check was skipped, but because the
|
|
6
|
+
// question was wrong. The MCP registry was asked `?search=<name>&limit=100`, which returns a capped
|
|
7
|
+
// version history and came back without the two newest releases in it. It answered with a version
|
|
8
|
+
// number, the version number looked plausible, and it would have looked just as plausible if
|
|
9
|
+
// nothing were wrong.
|
|
10
|
+
//
|
|
11
|
+
// So the verdict here has three values, not two. `unverified` is the whole point of the file: a
|
|
12
|
+
// channel nobody could read is not a channel that is up to date, and must never be rendered as one.
|
|
13
|
+
/** A gap younger than this is a release in flight, not a release that failed. */
|
|
14
|
+
export const PATIENCE_HOURS = 12;
|
|
15
|
+
/** Compares two dotted versions. Missing parts count as zero, so 0.70 is older than 0.70.1. */
|
|
16
|
+
export function compareVersions(a, b) {
|
|
17
|
+
const parts = (v) => v.split('.').map((n) => Number.parseInt(n, 10) || 0);
|
|
18
|
+
const [x, y] = [parts(a), parts(b)];
|
|
19
|
+
for (let i = 0; i < Math.max(x.length, y.length); i++) {
|
|
20
|
+
const d = (x[i] ?? 0) - (y[i] ?? 0);
|
|
21
|
+
if (d !== 0)
|
|
22
|
+
return d < 0 ? -1 : 1;
|
|
23
|
+
}
|
|
24
|
+
return 0;
|
|
25
|
+
}
|
|
26
|
+
export function judge(reading) {
|
|
27
|
+
if (reading.actual === null) {
|
|
28
|
+
return {
|
|
29
|
+
...reading,
|
|
30
|
+
verdict: 'unverified',
|
|
31
|
+
says: `could not be read${reading.problem ? `: ${reading.problem}` : ''}. This is not the same as being up to date - nothing here says whether ${reading.name} has ${reading.expected} or not.`,
|
|
32
|
+
};
|
|
33
|
+
}
|
|
34
|
+
const order = compareVersions(reading.actual, reading.expected);
|
|
35
|
+
if (order === 0)
|
|
36
|
+
return { ...reading, verdict: 'current', says: `has ${reading.actual}.` };
|
|
37
|
+
if (order < 0) {
|
|
38
|
+
const age = reading.ageHours;
|
|
39
|
+
// Unknown age is treated as old. Not knowing how long a channel has been behind is not a reason
|
|
40
|
+
// to assume it only just happened.
|
|
41
|
+
if (age !== null && age !== undefined && age < PATIENCE_HOURS) {
|
|
42
|
+
const how = age < 1 ? `${Math.max(1, Math.round(age * 60))} minute(s)` : `${age.toFixed(1)} hour(s)`;
|
|
43
|
+
return {
|
|
44
|
+
...reading,
|
|
45
|
+
verdict: 'waiting',
|
|
46
|
+
says: `has ${reading.actual}; ${reading.expected} was cut ${how} ago and has not reached it yet. Normal this soon after a release - a problem if it is still true tomorrow.`,
|
|
47
|
+
};
|
|
48
|
+
}
|
|
49
|
+
const how = age === null || age === undefined ? '' : ` It has been ${age.toFixed(0)} hour(s).`;
|
|
50
|
+
return { ...reading, verdict: 'behind', says: `has ${reading.actual}, but this repository is at ${reading.expected}. The release did not reach it.${how}` };
|
|
51
|
+
}
|
|
52
|
+
// Ahead is not an error on its own: the editor extension keeps its own version line, and a
|
|
53
|
+
// channel can legitimately carry something this checkout has not caught up with.
|
|
54
|
+
return { ...reading, verdict: 'ahead', says: `has ${reading.actual}, which is newer than this checkout's ${reading.expected}.` };
|
|
55
|
+
}
|
|
56
|
+
const MARK = { current: 'ok', behind: 'BEHIND', waiting: 'waiting', ahead: 'ahead', unverified: 'UNVERIFIED' };
|
|
57
|
+
/** The whole answer, written so the two states that need a person are impossible to skim past. */
|
|
58
|
+
export function renderChannels(readings) {
|
|
59
|
+
const judged = readings.map(judge);
|
|
60
|
+
const width = Math.max(...judged.map((c) => c.name.length), 12);
|
|
61
|
+
const lines = ['Where Tewip is published', ''];
|
|
62
|
+
for (const c of judged)
|
|
63
|
+
lines.push(` ${MARK[c.verdict].padEnd(11)} ${c.name.padEnd(width)} ${c.says}`);
|
|
64
|
+
const behind = judged.filter((c) => c.verdict === 'behind');
|
|
65
|
+
const unverified = judged.filter((c) => c.verdict === 'unverified');
|
|
66
|
+
lines.push('');
|
|
67
|
+
if (behind.length > 0) {
|
|
68
|
+
lines.push(`${behind.length} channel(s) did not get this release: ${behind.map((c) => c.name).join(', ')}.`, '');
|
|
69
|
+
}
|
|
70
|
+
if (unverified.length > 0) {
|
|
71
|
+
// Named separately from `behind` on purpose. Folding the two together is how an unanswered
|
|
72
|
+
// query becomes a clean report.
|
|
73
|
+
lines.push(`${unverified.length} channel(s) could not be checked at all: ${unverified.map((c) => c.name).join(', ')}. Nothing above says whether those are current.`, '');
|
|
74
|
+
}
|
|
75
|
+
const waiting = judged.filter((c) => c.verdict === 'waiting');
|
|
76
|
+
if (waiting.length > 0) {
|
|
77
|
+
lines.push(`${waiting.length} channel(s) have not caught up with this release yet: ${waiting.map((c) => c.name).join(', ')}. Nothing is wrong unless this is still true tomorrow.`, '');
|
|
78
|
+
}
|
|
79
|
+
if (behind.length === 0 && unverified.length === 0 && waiting.length === 0)
|
|
80
|
+
lines.push('Every channel answered, and every one has this release.', '');
|
|
81
|
+
lines.push('How each was asked:');
|
|
82
|
+
for (const c of judged)
|
|
83
|
+
lines.push(` ${c.name}: ${c.asked}`);
|
|
84
|
+
return { text: lines.join(String.fromCharCode(10)), behind: behind.length, unverified: unverified.length, waiting: waiting.length };
|
|
85
|
+
}
|
package/dist/engine/rules.js
CHANGED
|
@@ -171,6 +171,37 @@ const envServerError = {
|
|
|
171
171
|
* is what the streak conditions below are reading and what one CI run cannot supply. Re-derive with
|
|
172
172
|
* `node eval/mismatch-precision.ts` before changing any of this.
|
|
173
173
|
*/
|
|
174
|
+
/**
|
|
175
|
+
* How often this rule is right, measured rather than argued.
|
|
176
|
+
*
|
|
177
|
+
* It used to claim `0.65 + 0.05 * failingStreak`, up to 0.9, and across the six labelled datasets
|
|
178
|
+
* it was right 10 times out of 26 - the only overconfident rule in the set, and the one that tells
|
|
179
|
+
* a team their application is broken. All 16 misses were the same: a flaky test called a real bug,
|
|
180
|
+
* which is the failure the comment above this rule exists to warn about.
|
|
181
|
+
*
|
|
182
|
+
* The streak term was worse than unearned, it was inverted. Split by streak length:
|
|
183
|
+
*
|
|
184
|
+
* streak 1 10/16 62.5% across 5 projects
|
|
185
|
+
* streak 2+ 0/10 0.0% across 1 project
|
|
186
|
+
*
|
|
187
|
+
* So the formula paid the most confidence exactly where the rule was never right. The streak is
|
|
188
|
+
* gone from the number rather than used to refuse those cases: 0 of 10 is ten failures from one
|
|
189
|
+
* project, and silencing a category on one project's data is how this rule got here. Lowering an
|
|
190
|
+
* overstated number is safe; suppressing findings on thin evidence is not.
|
|
191
|
+
*
|
|
192
|
+
* A third change was tried and taken back out. Requiring the test to have passed before - the
|
|
193
|
+
* condition `env-server-error` carries, and the rule was wrong all 3 times it fired without one -
|
|
194
|
+
* breaks the behaviour six importers deliberately test, that a wrong value failing every attempt in
|
|
195
|
+
* a single run is a real bug. Three cases is not enough to remove a designed answer, and Phase 64
|
|
196
|
+
* already recorded the cost of this rule declining too readily: `tewip bugs` filed one ticket out of
|
|
197
|
+
* 1,964 real failures. Lowering an overstated number is safe. Refusing to answer is not, and it was
|
|
198
|
+
* not earned.
|
|
199
|
+
*
|
|
200
|
+
* 0.40 is simply how often the rule is right: 10 of 26. It sits below the 0.7 the merge gate trusts
|
|
201
|
+
* and the 0.85 auto-fix needs, which is where a signal right two times in five belongs - in front of
|
|
202
|
+
* a person, which is what the report already calls it. Re-derive with `node eval/calibration.ts`.
|
|
203
|
+
*/
|
|
204
|
+
const REAL_BUG_ASSERTION_CONFIDENCE = 0.4;
|
|
174
205
|
const realBugAssertion = {
|
|
175
206
|
id: 'real-bug-assertion',
|
|
176
207
|
evaluate({ record, history, loadFailedRequests }) {
|
|
@@ -196,7 +227,7 @@ const realBugAssertion = {
|
|
|
196
227
|
: `failed all ${failingStreak} recorded runs with the same mismatch: ${values}`;
|
|
197
228
|
return {
|
|
198
229
|
category: 'REAL_BUG',
|
|
199
|
-
confidence:
|
|
230
|
+
confidence: REAL_BUG_ASSERTION_CONFIDENCE,
|
|
200
231
|
evidence: [{ source: this.id, message, data: { ...mismatch, failingStreak, passesBeforeStreak } }],
|
|
201
232
|
};
|
|
202
233
|
},
|
|
@@ -1,5 +1,15 @@
|
|
|
1
1
|
import { ALEMBIC, DJANGO, LARAVEL, RAILS } from './dialect-rules.js';
|
|
2
2
|
import { EFCORE, KNEX, LIQUIBASE, SEQUELIZE } from './dialect-rules-node.js';
|
|
3
|
+
/**
|
|
4
|
+
* Every way a knex migration declares its rollback.
|
|
5
|
+
*
|
|
6
|
+
* `exports.down =` is the CommonJS form the first three projects this was verified against all
|
|
7
|
+
* used - opencollective-api, Matterwiki and Coflnet. TypeScript projects do not write that, and
|
|
8
|
+
* nocodb writes `const down = async (knex) => {` in 236 of its 219 migration files (some declare
|
|
9
|
+
* more than one). Missing the declaration means the rollback is read as part of the change, and a
|
|
10
|
+
* migration that adds tables is reported as one that drops them.
|
|
11
|
+
*/
|
|
12
|
+
const KNEX_DOWN = /^\s*(?:export\s+)?(?:const|let|var)\s+down\s*=|^\s*(?:export\s+)?(?:async\s+)?function\s+down\s*\(|^\s*(?:module\.)?exports\.down\s*=/;
|
|
3
13
|
export function dialectOf(file, text) {
|
|
4
14
|
const path = file.split('\\').join('/').toLowerCase();
|
|
5
15
|
if (path.endsWith('.sql'))
|
|
@@ -13,6 +23,17 @@ export function dialectOf(file, text) {
|
|
|
13
23
|
return 'rails';
|
|
14
24
|
if (/\bop\.(?:drop_column|add_column|create_index|alter_column|drop_table|rename_table|execute)\b/.test(text))
|
|
15
25
|
return 'alembic';
|
|
26
|
+
// Kysely, which immich moved to from TypeORM. The DDL is raw SQL inside tagged template
|
|
27
|
+
// literals - await sql`ALTER TABLE ... DROP COLUMN ...`.execute(db) - so it is perfectly readable
|
|
28
|
+
// once the wrapper is recognised, exactly as it was for TypeORM.
|
|
29
|
+
//
|
|
30
|
+
// Before this, immich's 100 migrations were not reported as unreadable. They were not reported
|
|
31
|
+
// at all: migrationsUnder drops a file whose dialect is unknown, on the grounds that such a file
|
|
32
|
+
// is not a migration - which is right for a library module called helpers.rb and wrong for a
|
|
33
|
+
// whole dialect. The 37 files Tewip did find there were generated query files under src/queries,
|
|
34
|
+
// and the real migration that drops assets.isVisible and assets.isArchived was silent.
|
|
35
|
+
if (/from\s+['\"]kysely['\"]|\bKysely\s*</.test(text) && /\bsql`/.test(text))
|
|
36
|
+
return 'kysely';
|
|
16
37
|
// TypeORM writes a migration as a class with up() and down(), and puts the real DDL in template
|
|
17
38
|
// strings passed to queryRunner.query(). The SQL is right there and perfectly readable; only the
|
|
18
39
|
// wrapper was unfamiliar, so every TypeORM project was told Tewip could not read its migrations
|
|
@@ -40,6 +61,10 @@ const BY_DIALECT = {
|
|
|
40
61
|
rails: RAILS,
|
|
41
62
|
alembic: ALEMBIC,
|
|
42
63
|
typeorm: [],
|
|
64
|
+
// Neither has operations of its own: the whole migration is raw SQL, read by `readSql`. TypeORM
|
|
65
|
+
// hands it over through queryRunner.query(...), Kysely through a tagged template, and Kysely is
|
|
66
|
+
// routed straight to the SQL reader before this table is consulted.
|
|
67
|
+
kysely: [],
|
|
43
68
|
sequelize: SEQUELIZE,
|
|
44
69
|
knex: KNEX,
|
|
45
70
|
efcore: EFCORE,
|
|
@@ -136,7 +161,13 @@ export function forwardOnly(dialect, text) {
|
|
|
136
161
|
// ecosystems as dropping a column. Verified on real migrations from opencollective-api,
|
|
137
162
|
// Matterwiki and Coflnet, all three of which have their only destructive call in down.
|
|
138
163
|
(dialect === 'sequelize' && /^\s*down\s*:/.test(line)) ||
|
|
139
|
-
|
|
164
|
+
// Knex is declared four ways and the pattern only knew two of them. nocodb writes
|
|
165
|
+
// `const down = async (knex) => {` in 236 of its migrations and `exports.down` in none, so
|
|
166
|
+
// every rollback was read as part of the change: 122 findings, each one a dropTable in a
|
|
167
|
+
// `down` that undoes its own `up`, which is what a reversible migration is.
|
|
168
|
+
(dialect === 'knex' && KNEX_DOWN.test(line)) ||
|
|
169
|
+
// Kysely declares its rollback the same way: export async function down(db) {
|
|
170
|
+
(dialect === 'kysely' && /^\s*(?:export\s+)?(?:async\s+)?function\s+down\s*\(/.test(line)) ||
|
|
140
171
|
(dialect === 'efcore' && /^\s*protected\s+override\s+void\s+Down\s*\(/.test(line)) ||
|
|
141
172
|
// Laravel's rollback, which in real files often comes *before* up().
|
|
142
173
|
(dialect === 'laravel' && /^\s*(?:public\s+)?function\s+down\s*\(/.test(line)) ||
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Tables this migration creates itself.
|
|
3
|
+
*
|
|
4
|
+
* A lock is only expensive because of the rows it holds up, and a table created three statements
|
|
5
|
+
* earlier has none. This matters enormously on the tools that generate their SQL: a Prisma project
|
|
6
|
+
* emits `CREATE TABLE` followed by an `ADD CONSTRAINT ... FOREIGN KEY` for every relation, so
|
|
7
|
+
* documenso's 165 real migrations produced 275 "blocks writes while it runs" findings - 1.7 per
|
|
8
|
+
* migration, nearly all of them about tables that did not exist a moment before. That is the volume
|
|
9
|
+
* at which a team stops reading the report, which costs more than the warnings are worth.
|
|
10
|
+
*/
|
|
11
|
+
function createdHere(text) {
|
|
12
|
+
const names = new Set();
|
|
13
|
+
// The frameworks say "create this table" in their own words, and the suppression is worth nothing
|
|
14
|
+
// in a dialect it cannot read. Only SQL was read, so bitwarden/server's 456 EF Core migrations -
|
|
15
|
+
// which do exactly what Prisma does, a CreateTable followed by a CreateIndex for every relation -
|
|
16
|
+
// produced 647 "blocks writes" findings about tables that had not existed a statement earlier.
|
|
17
|
+
// EF Core puts the name on the line after the call, so these have to reach across a newline.
|
|
18
|
+
const FRAMEWORK_CREATES = [
|
|
19
|
+
/migrationBuilder\s*\.\s*CreateTable\s*\([\s\S]{0,80}?name\s*:\s*["']([\w$.]+)["']/gi,
|
|
20
|
+
// Django names the model rather than the table, and so does its AddIndex, so the two match each
|
|
21
|
+
// other. Saleor never exercises this - Django convention puts CreateModel and AddIndex in
|
|
22
|
+
// separate migrations - but every other dialect here has its create form registered and this
|
|
23
|
+
// one did not, which is the kind of gap that waits for a project with different habits.
|
|
24
|
+
/migrations\s*\.\s*CreateModel\s*\([\s\S]{0,60}?name\s*=\s*["']([\w]+)["']/gi,
|
|
25
|
+
/queryInterface\s*\.\s*createTable\s*\(\s*["']([\w$.]+)["']/gi,
|
|
26
|
+
/knex\s*\.\s*schema\s*\.\s*createTable\s*\(\s*["']([\w$.]+)["']/gi,
|
|
27
|
+
/Schema\s*::\s*create\s*\(\s*["']([\w$.]+)["']/gi,
|
|
28
|
+
/\bcreate_table\s+[:'"]([\w$.]+)/gi,
|
|
29
|
+
/\bop\s*\.\s*create_table\s*\(\s*["']([\w$.]+)["']/gi,
|
|
30
|
+
/<createTable[^>]*tableName\s*=\s*["']([\w$.]+)["']/gi,
|
|
31
|
+
];
|
|
32
|
+
for (const pattern of FRAMEWORK_CREATES)
|
|
33
|
+
for (const m of text.matchAll(pattern))
|
|
34
|
+
names.add(m[1].toLowerCase().replace(/^.*\./, ''));
|
|
35
|
+
// Every dialect's way of quoting a name, because the suppression is useless in the one that does
|
|
36
|
+
// not match. This read PostgreSQL's "quoted" form and plain names, and not SQL Server's
|
|
37
|
+
// [dbo].[Grant] - which is standard T-SQL and what bitwarden/server is written in throughout. No
|
|
38
|
+
// table was ever recognised as freshly created there, so every index built on a table created two
|
|
39
|
+
// statements earlier was reported as blocking writes: 714 findings, nearly all of them free.
|
|
40
|
+
for (const m of text.matchAll(/\bCREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?([\w$.\"\[\]`]+)/gi)) {
|
|
41
|
+
// `[dbo].[Grant]` and `"public"."grant"` and `grant` all have to arrive here as the same word,
|
|
42
|
+
// because that is the form the findings report a table in.
|
|
43
|
+
names.add(m[1].replace(/[\"\[\]`]/g, '').toLowerCase().replace(/^.*\./, ''));
|
|
44
|
+
}
|
|
45
|
+
return names;
|
|
46
|
+
}
|
|
47
|
+
/** Risks that are only risks because rows already exist. */
|
|
48
|
+
const ONLY_HURTS_EXISTING_ROWS = new Set(['locks-writes', 'slow']);
|
|
49
|
+
/**
|
|
50
|
+
* Tables that something is renamed *to* in this migration.
|
|
51
|
+
*
|
|
52
|
+
* SQLite cannot alter a column, so the way to change one is to build a new table, copy every row
|
|
53
|
+
* into it, drop the original and rename the copy into its place. TypeORM generates exactly that,
|
|
54
|
+
* automatically, for every column change on SQLite, and n8n's migrations are full of it:
|
|
55
|
+
*
|
|
56
|
+
* CREATE TABLE temporary_credentials_entity (...)
|
|
57
|
+
* INSERT INTO temporary_credentials_entity SELECT ... FROM credentials_entity
|
|
58
|
+
* DROP TABLE credentials_entity
|
|
59
|
+
* ALTER TABLE temporary_credentials_entity RENAME TO credentials_entity
|
|
60
|
+
*
|
|
61
|
+
* The drop is real and the table is gone for an instant, but nothing is lost: the rows were copied
|
|
62
|
+
* first and the name is given straight back. Calling that "drops the table credentials_entity and
|
|
63
|
+
* everything in it" is wrong, and it would be wrong on nearly every TypeORM migration ever
|
|
64
|
+
* generated for SQLite.
|
|
65
|
+
*/
|
|
66
|
+
function renamedInto(text) {
|
|
67
|
+
const names = new Set();
|
|
68
|
+
for (const m of text.matchAll(/\bRENAME\s+TO\s+([\w."'`$[\]{}]+)/gi)) {
|
|
69
|
+
names.add(m[1].replace(/\$\{[^}]*\}/g, '').replace(/["'`[\]]/g, '').toLowerCase().replace(/^.*\./, ''));
|
|
70
|
+
}
|
|
71
|
+
return names;
|
|
72
|
+
}
|
|
73
|
+
export function droppingWhatIsNew(findings, text) {
|
|
74
|
+
const fresh = createdHere(text);
|
|
75
|
+
const rebuilt = renamedInto(text);
|
|
76
|
+
if (fresh.size === 0 && rebuilt.size === 0)
|
|
77
|
+
return findings;
|
|
78
|
+
const nameOf = (f) => f.table.toLowerCase().replace(/^.*\./, '');
|
|
79
|
+
return findings.filter((f) => {
|
|
80
|
+
if (ONLY_HURTS_EXISTING_ROWS.has(f.risk) && fresh.has(nameOf(f)))
|
|
81
|
+
return false;
|
|
82
|
+
// A table dropped and immediately given its name back by a rename is a rebuild, not a loss.
|
|
83
|
+
if (f.risk === 'destroys-data' && rebuilt.has(nameOf(f)))
|
|
84
|
+
return false;
|
|
85
|
+
return true;
|
|
86
|
+
});
|
|
87
|
+
}
|
|
@@ -15,14 +15,37 @@ import { join, relative } from 'node:path';
|
|
|
15
15
|
// is mostly documentation teaches the reader to skip the list.
|
|
16
16
|
//
|
|
17
17
|
// So the directory decides *where*, and the extension decides *whether it could hold DDL at all*.
|
|
18
|
-
|
|
18
|
+
// `versions` is Alembic's revision directory in every configuration it has - `alembic/versions`,
|
|
19
|
+
// `migrations/versions`, or whatever `script_location` points at. Netflix/dispatch keeps 162 of them
|
|
20
|
+
// under `src/dispatch/database/revisions/<name>/versions/`, where neither `alembic` nor `migrations`
|
|
21
|
+
// appears in the path, and Tewip found two files in that repository - both `.sql`, one of them a
|
|
22
|
+
// search-expressions file that is not a migration at all. A project with 162 migrations was read as
|
|
23
|
+
// a project with none of them.
|
|
24
|
+
//
|
|
25
|
+
// Letting the directory through costs little, because `dialectOf` still has to find `op.drop_column`
|
|
26
|
+
// or its siblings before anything is read as a migration - the same argument that lets a
|
|
27
|
+
// release-notes CHANGELOG.xml through below.
|
|
28
|
+
const MIGRATION_DIR = /(^|\/)(migrations?|migration|db\/migrate|prisma\/migrations|alembic|versions|flyway|changelogs?|sql)\//i;
|
|
19
29
|
const COULD_HOLD_DDL = /\.(sql|py|rb|js|mjs|cjs|ts|php|cs|xml|ya?ml|json|go|exs?)$/i;
|
|
20
30
|
// Flyway's Java migrations are real, and they are named the way Flyway requires: `V2__add_index.java`
|
|
21
31
|
// or `R__rebuild_views.java`. Keycloak's `MigrateTo12_0_0.java` is not one. Taking `.java` on the
|
|
22
32
|
// strength of its directory alone is what filled that list with source files.
|
|
23
33
|
const FLYWAY_JAVA = /(^|\/)[VR][\w.]*__[\w.-]+\.(java|kt|scala)$/;
|
|
34
|
+
/**
|
|
35
|
+
* Directories whose SQL is a fixture, not something anybody deploys.
|
|
36
|
+
*
|
|
37
|
+
* `SCHEMA_FILE` takes any `.sql` file anywhere, because a migration really can sit outside a
|
|
38
|
+
* directory that says so. nocodb ships the Sakila sample database for its tests, and eleven of
|
|
39
|
+
* those dumps - `mysql-sakila-drop-objects.sql`, `oracle-sakila-delete-data.sql` and the rest -
|
|
40
|
+
* were read as migrations that destroy data. They were 102 of the 106 findings in that repository,
|
|
41
|
+
* and not one of them will ever run against a live database.
|
|
42
|
+
*
|
|
43
|
+
* This command answers what a migration will do to a running system, so a file under `tests` is
|
|
44
|
+
* out of scope by the question's own definition.
|
|
45
|
+
*/
|
|
46
|
+
const FIXTURE_DIR = /(^|\/)(tests?|__tests__|specs?|fixtures?|testdata|mocks?)\//i;
|
|
24
47
|
export const MIGRATION = {
|
|
25
|
-
test: (path) => MIGRATION_DIR.test(path) && (COULD_HOLD_DDL.test(path) || FLYWAY_JAVA.test(path)),
|
|
48
|
+
test: (path) => !FIXTURE_DIR.test(path) && MIGRATION_DIR.test(path) && (COULD_HOLD_DDL.test(path) || FLYWAY_JAVA.test(path)),
|
|
26
49
|
};
|
|
27
50
|
// A changelog is often not in a directory that says so. OpenMRS keeps 44 Liquibase changelogs in
|
|
28
51
|
// `api/src/main/resources/` named `liquibase-*.xml`, and the path filter saw none of them - so a
|
|
@@ -34,7 +57,8 @@ export const MIGRATION = {
|
|
|
34
57
|
// project has more than one database. Letting a release-notes `CHANGELOG.xml` through costs nothing,
|
|
35
58
|
// because `dialectOf` still has to find `<databaseChangeLog` in it before it is read as a migration.
|
|
36
59
|
export const NAMED_MIGRATION = /(^|\/)[\w.-]*(liquibase|changelog)[\w.-]*\.(xml|ya?ml|json)$/i;
|
|
37
|
-
|
|
60
|
+
const SCHEMA_PATH = /\.(sql)$|(^|\/)(migrations?|migration|db\/migrate|versions)\/.*\.(py|rb|js|ts|php|cs)$/i;
|
|
61
|
+
export const SCHEMA_FILE = { test: (path) => !FIXTURE_DIR.test(path) && SCHEMA_PATH.test(path) };
|
|
38
62
|
/**
|
|
39
63
|
* Every migration in a whole repository, by the same filters a diff is read through.
|
|
40
64
|
*
|
|
@@ -13,9 +13,11 @@ import { normalise } from '../quality/source-text.js';
|
|
|
13
13
|
import { dirname, resolve } from 'node:path';
|
|
14
14
|
import { changeSummary } from '../quality/grounding.js';
|
|
15
15
|
import { dialectOf, forwardOnly, irreversible, readFramework } from './frameworks.js';
|
|
16
|
+
import { droppingWhatIsNew } from './kept.js';
|
|
16
17
|
import { readSql } from './statements.js';
|
|
17
18
|
import { MIGRATION, NAMED_MIGRATION, SCHEMA_FILE, migrationsUnder } from './migration-files.js';
|
|
18
19
|
export { migrationsUnder };
|
|
20
|
+
export { droppingWhatIsNew };
|
|
19
21
|
/** The schema files in this change. */
|
|
20
22
|
export function schemaFiles(cmd) {
|
|
21
23
|
const named = cmd.files !== undefined && cmd.files.length > 0;
|
|
@@ -27,60 +29,6 @@ export function schemaFiles(cmd) {
|
|
|
27
29
|
// sitting outside a migrations/ directory was being answered with "no migration in this change".
|
|
28
30
|
.filter((f) => (named || SCHEMA_FILE.test(f) || MIGRATION.test(f) || NAMED_MIGRATION.test(f)) && existsSync(resolve(cmd.root, f)));
|
|
29
31
|
}
|
|
30
|
-
/**
|
|
31
|
-
* Tables this migration creates itself.
|
|
32
|
-
*
|
|
33
|
-
* A lock is only expensive because of the rows it holds up, and a table created three statements
|
|
34
|
-
* earlier has none. This matters enormously on the tools that generate their SQL: a Prisma project
|
|
35
|
-
* emits `CREATE TABLE` followed by an `ADD CONSTRAINT ... FOREIGN KEY` for every relation, so
|
|
36
|
-
* documenso's 165 real migrations produced 275 "blocks writes while it runs" findings - 1.7 per
|
|
37
|
-
* migration, nearly all of them about tables that did not exist a moment before. That is the volume
|
|
38
|
-
* at which a team stops reading the report, which costs more than the warnings are worth.
|
|
39
|
-
*/
|
|
40
|
-
function createdHere(text) {
|
|
41
|
-
const names = new Set();
|
|
42
|
-
// The frameworks say "create this table" in their own words, and the suppression is worth nothing
|
|
43
|
-
// in a dialect it cannot read. Only SQL was read, so bitwarden/server's 456 EF Core migrations -
|
|
44
|
-
// which do exactly what Prisma does, a CreateTable followed by a CreateIndex for every relation -
|
|
45
|
-
// produced 647 "blocks writes" findings about tables that had not existed a statement earlier.
|
|
46
|
-
// EF Core puts the name on the line after the call, so these have to reach across a newline.
|
|
47
|
-
const FRAMEWORK_CREATES = [
|
|
48
|
-
/migrationBuilder\s*\.\s*CreateTable\s*\([\s\S]{0,80}?name\s*:\s*["']([\w$.]+)["']/gi,
|
|
49
|
-
// Django names the model rather than the table, and so does its AddIndex, so the two match each
|
|
50
|
-
// other. Saleor never exercises this - Django convention puts CreateModel and AddIndex in
|
|
51
|
-
// separate migrations - but every other dialect here has its create form registered and this
|
|
52
|
-
// one did not, which is the kind of gap that waits for a project with different habits.
|
|
53
|
-
/migrations\s*\.\s*CreateModel\s*\([\s\S]{0,60}?name\s*=\s*["']([\w]+)["']/gi,
|
|
54
|
-
/queryInterface\s*\.\s*createTable\s*\(\s*["']([\w$.]+)["']/gi,
|
|
55
|
-
/knex\s*\.\s*schema\s*\.\s*createTable\s*\(\s*["']([\w$.]+)["']/gi,
|
|
56
|
-
/Schema\s*::\s*create\s*\(\s*["']([\w$.]+)["']/gi,
|
|
57
|
-
/\bcreate_table\s+[:'"]([\w$.]+)/gi,
|
|
58
|
-
/\bop\s*\.\s*create_table\s*\(\s*["']([\w$.]+)["']/gi,
|
|
59
|
-
/<createTable[^>]*tableName\s*=\s*["']([\w$.]+)["']/gi,
|
|
60
|
-
];
|
|
61
|
-
for (const pattern of FRAMEWORK_CREATES)
|
|
62
|
-
for (const m of text.matchAll(pattern))
|
|
63
|
-
names.add(m[1].toLowerCase().replace(/^.*\./, ''));
|
|
64
|
-
// Every dialect's way of quoting a name, because the suppression is useless in the one that does
|
|
65
|
-
// not match. This read PostgreSQL's "quoted" form and plain names, and not SQL Server's
|
|
66
|
-
// [dbo].[Grant] - which is standard T-SQL and what bitwarden/server is written in throughout. No
|
|
67
|
-
// table was ever recognised as freshly created there, so every index built on a table created two
|
|
68
|
-
// statements earlier was reported as blocking writes: 714 findings, nearly all of them free.
|
|
69
|
-
for (const m of text.matchAll(/\bCREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?([\w$.\"\[\]`]+)/gi)) {
|
|
70
|
-
// `[dbo].[Grant]` and `"public"."grant"` and `grant` all have to arrive here as the same word,
|
|
71
|
-
// because that is the form the findings report a table in.
|
|
72
|
-
names.add(m[1].replace(/[\"\[\]`]/g, '').toLowerCase().replace(/^.*\./, ''));
|
|
73
|
-
}
|
|
74
|
-
return names;
|
|
75
|
-
}
|
|
76
|
-
/** Risks that are only risks because rows already exist. */
|
|
77
|
-
const ONLY_HURTS_EXISTING_ROWS = new Set(['locks-writes', 'slow']);
|
|
78
|
-
export function droppingWhatIsNew(findings, text) {
|
|
79
|
-
const fresh = createdHere(text);
|
|
80
|
-
if (fresh.size === 0)
|
|
81
|
-
return findings;
|
|
82
|
-
return findings.filter((f) => !(ONLY_HURTS_EXISTING_ROWS.has(f.risk) && fresh.has(f.table.toLowerCase().replace(/^.*\./, ''))));
|
|
83
|
-
}
|
|
84
32
|
export function readMigration(root, file) {
|
|
85
33
|
let text = '';
|
|
86
34
|
try {
|
|
@@ -93,14 +41,16 @@ export function readMigration(root, file) {
|
|
|
93
41
|
// Only the forward path: a rollback step contains the mirror image of the change, and reading it as
|
|
94
42
|
// part of the change says a migration that creates a table destroys one.
|
|
95
43
|
const forward = forwardOnly(dialect, text);
|
|
96
|
-
|
|
44
|
+
// Kysely's DDL is raw SQL in a template literal, so once the rollback is removed the same
|
|
45
|
+
// reader that handles a .sql file handles it.
|
|
46
|
+
if (dialect === 'sql' || dialect === 'kysely')
|
|
97
47
|
return { file, dialect, findings: droppingWhatIsNew(readSql(forward), forward), unread: '', irreversible: false };
|
|
98
48
|
if (dialect === 'unknown') {
|
|
99
49
|
return {
|
|
100
50
|
file,
|
|
101
51
|
dialect,
|
|
102
52
|
findings: [],
|
|
103
|
-
unread: 'Tewip does not recognise how this migration is written, so it says nothing about it rather than passing it. It reads SQL, Django, Rails, Alembic, TypeORM, Sequelize, Knex, Laravel, EF Core and Liquibase migrations',
|
|
53
|
+
unread: 'Tewip does not recognise how this migration is written, so it says nothing about it rather than passing it. It reads SQL, Django, Rails, Alembic, TypeORM, Sequelize, Knex, Kysely, Laravel, EF Core and Liquibase migrations',
|
|
104
54
|
irreversible: false,
|
|
105
55
|
};
|
|
106
56
|
}
|
|
@@ -1,5 +1,13 @@
|
|
|
1
1
|
const CLEAN = (s) => s.replace(/\s+/g, ' ').trim();
|
|
2
|
-
|
|
2
|
+
// A table name as it is written, including a JavaScript template interpolation.
|
|
3
|
+
//
|
|
4
|
+
// n8n prefixes every table through `${tablePrefix}`, and without the interpolation in here two
|
|
5
|
+
// things went wrong on its 106 TypeORM migrations. `DROP TABLE IF EXISTS ${tablePrefix}x`
|
|
6
|
+
// backtracked past the optional IF EXISTS and matched `IF`, so Tewip reported "drops the table IF".
|
|
7
|
+
// `DROP TABLE "${tablePrefix}x"` matched the opening quote alone, which `bare` then stripped to
|
|
8
|
+
// nothing, so Tewip reported "drops the table and everything in it" with a hole where the name
|
|
9
|
+
// should be. Both are worse than saying nothing: a reader cannot check either of them.
|
|
10
|
+
const NAME = String.raw `(?:\$\{[^}]*\}|[\w."\[\]` + "`" + String.raw `])+`;
|
|
3
11
|
/**
|
|
4
12
|
* What a statement does. Ordered: the first rule that matches decides, so the most serious reading of a
|
|
5
13
|
* statement wins over a milder one.
|
|
@@ -114,7 +122,40 @@ const RULES = [
|
|
|
114
122
|
},
|
|
115
123
|
];
|
|
116
124
|
/** `"users"` and `` `users` `` and `[users]` are all the table users. */
|
|
117
|
-
|
|
125
|
+
/**
|
|
126
|
+
* The name without its quoting, and without the prefix expression a project may wrap it in.
|
|
127
|
+
*
|
|
128
|
+
* `${tablePrefix}credentials_entity` is reported as `credentials_entity`, the name a reader will
|
|
129
|
+
* recognise and search for: the prefix is configuration, identical for every table in the
|
|
130
|
+
* migration, and carrying it through only makes every finding harder to read.
|
|
131
|
+
*
|
|
132
|
+
* Three things real projects do that this has to survive, all found on n8n and immich:
|
|
133
|
+
*
|
|
134
|
+
* A name that is nothing but the expression. `ALTER TABLE ${tempInsightsRawTable} RENAME TO
|
|
135
|
+
* ${insightsRawTable}` leaves nothing once the interpolation is stripped, which read as "renames
|
|
136
|
+
* the table to ". A finding with a hole where the name belongs cannot be checked by the person
|
|
137
|
+
* reading it, so when nothing survives, the expression is the name.
|
|
138
|
+
*
|
|
139
|
+
* A name that ends the template. NAME accepts a backtick because MySQL quotes with one, so in
|
|
140
|
+
* sql`truncate ${sql.table('ocr_search')}`.execute(db) it ran straight past the closing backtick
|
|
141
|
+
* and reported "empties the table execute". The template ends where its backtick is.
|
|
142
|
+
*
|
|
143
|
+
* And a builder call that names the table inside the expression. `sql.table('ocr_search')` says
|
|
144
|
+
* which table plainly, and quoting the whole expression back at the reader when the answer is
|
|
145
|
+
* right there would be needlessly unhelpful.
|
|
146
|
+
*/
|
|
147
|
+
const bare = (name) => {
|
|
148
|
+
const inTemplate = name.includes('${') ? name.split('`')[0] : name;
|
|
149
|
+
const named = /\$\{[^}]*\(\s*['"]([\w.$]+)['"]/.exec(inTemplate);
|
|
150
|
+
if (named)
|
|
151
|
+
return named[1];
|
|
152
|
+
const unquoted = inTemplate.replace(/["`[\]]/g, '').trim();
|
|
153
|
+
const withoutPrefix = unquoted
|
|
154
|
+
.replace(/\$\{[^}]*\}/g, '')
|
|
155
|
+
.replace(/^\.+|\.+$/g, '')
|
|
156
|
+
.trim();
|
|
157
|
+
return withoutPrefix || unquoted;
|
|
158
|
+
};
|
|
118
159
|
/** The statements of a SQL file, with the line each one starts on and comments removed. */
|
|
119
160
|
export function statementsOf(sql) {
|
|
120
161
|
// Comments are blanked rather than removed, so every remaining character keeps its original position
|
|
@@ -71,10 +71,74 @@ const DOC_FILE = /(?:^|[\\/])(?:\.env\.example|\.env\.sample|\.env\.template|[^\
|
|
|
71
71
|
* there is, and a key pasted into an example file by accident is one of the commonest ways
|
|
72
72
|
* credentials leak. That case is now shown as a template finding instead of being dismissed.
|
|
73
73
|
*/
|
|
74
|
-
|
|
74
|
+
/**
|
|
75
|
+
* A URL-shaped credential whose destination cannot be a real system, or whose user and password
|
|
76
|
+
* are the same word.
|
|
77
|
+
*
|
|
78
|
+
* A credential matters because it is a way in to something. `postgres://saleor:saleor@localhost`
|
|
79
|
+
* is a way in to the machine it is already running on, and `user:password@example.com` points at
|
|
80
|
+
* a domain reserved by RFC 2606 precisely so that it cannot be anybody's system.
|
|
81
|
+
*
|
|
82
|
+
* Measured on Saleor at its pinned commit: six findings were reported as needing someone now, at
|
|
83
|
+
* high severity, each telling the team to "revoke it at the issuer first". Three were the Django
|
|
84
|
+
* local-development default in settings.py, one of them on a commented-out line, and the evidence
|
|
85
|
+
* offered was that the value "has the randomness of a real credential" - about the word `saleor`.
|
|
86
|
+
*
|
|
87
|
+
* Nothing is dropped. The finding becomes a placeholder finding, shown as noise with its reason
|
|
88
|
+
* attached. Suppressing a real secret is far worse than reporting a false one, so the two tests
|
|
89
|
+
* are narrow: a destination that provably reaches nobody, and a password identical to its own
|
|
90
|
+
* username, which nobody chooses for a credential that guards anything.
|
|
91
|
+
*/
|
|
92
|
+
/** Domains that exist so they cannot be anybody's system (RFC 2606, RFC 6761). */
|
|
93
|
+
const RESERVED = /^(?:[\w-]+\.)*example\.(?:com|org|net)$|\.(?:invalid|test|localhost|local)$/i;
|
|
94
|
+
/** Addresses that only ever mean the machine the code is running on. */
|
|
95
|
+
const THIS_MACHINE = /^(?:localhost|127\.0\.0\.1|\[?::1\]?|0\.0\.0\.0)$/i;
|
|
96
|
+
const CREDENTIALLED_URL = /[a-z][a-z0-9+.-]*:\/\/([^:/\s'\"]+):([^@\s'\"]+)@([^/\s'\"?#]+)/gi;
|
|
97
|
+
/**
|
|
98
|
+
* A credential whose destination is the machine it runs on.
|
|
99
|
+
*
|
|
100
|
+
* This is deliberately not treated as a placeholder. A hardcoded password in settings.py is a real
|
|
101
|
+
* string that a person should see - an earlier phase decided exactly that, about this exact case -
|
|
102
|
+
* and calling it noise would hide it. What was wrong was only the urgency: Saleor's Django default
|
|
103
|
+
* was reported as needing someone now, at high severity, told to "revoke it at the issuer first".
|
|
104
|
+
* There is no issuer. It belongs in "real, but not reachable", which is the bucket for something
|
|
105
|
+
* true of the repository and not of what it runs.
|
|
106
|
+
*/
|
|
107
|
+
export function credentialGoesNowhere(value, line) {
|
|
108
|
+
for (const [, , password, hostport] of [...value.matchAll(CREDENTIALLED_URL), ...[...line.matchAll(CREDENTIALLED_URL)].filter((m) => m[2] === value)]) {
|
|
109
|
+
if (password !== value && !value.includes(password))
|
|
110
|
+
continue;
|
|
111
|
+
const host = hostport.replace(/:\d+$/, '');
|
|
112
|
+
if (THIS_MACHINE.test(host))
|
|
113
|
+
return `the URL points at ${host}, so it reaches only the machine it runs on`;
|
|
114
|
+
}
|
|
115
|
+
return null;
|
|
116
|
+
}
|
|
117
|
+
export function urlCredentialIsHarmless(value, line) {
|
|
118
|
+
// The URL rules capture the whole URL as the value, not the password inside it - which is also
|
|
119
|
+
// why these pass the entropy gate at all, and why the report offered "the randomness of a real
|
|
120
|
+
// credential (3.82 bits per character)" as evidence about a string that is mostly the word
|
|
121
|
+
// `postgres` and a hostname. So the value is read as a URL first, and only then looked for inside
|
|
122
|
+
// the line, which is how the patterns that do capture a bare password arrive here.
|
|
123
|
+
const inValue = [...value.matchAll(CREDENTIALLED_URL)];
|
|
124
|
+
const inLine = [...line.matchAll(CREDENTIALLED_URL)].filter((m) => m[2] === value);
|
|
125
|
+
for (const [, user, password, hostport] of [...inValue, ...inLine]) {
|
|
126
|
+
const host = hostport.replace(/:\d+$/, '');
|
|
127
|
+
if (RESERVED.test(host))
|
|
128
|
+
return `the URL points at ${host}, a domain reserved so that it cannot be anybody's system`;
|
|
129
|
+
if (user.toLowerCase() === password.toLowerCase())
|
|
130
|
+
return 'the user and the password are the same word, which is a stand-in rather than a credential';
|
|
131
|
+
}
|
|
132
|
+
return null;
|
|
133
|
+
}
|
|
134
|
+
export function looksPlanted(value, line, file, before = '', whole = value) {
|
|
75
135
|
if (PLACEHOLDER_WORDS.test(value))
|
|
76
136
|
return 'the value itself says it is an example';
|
|
77
|
-
|
|
137
|
+
// Only the words *around* the credential are commentary about it. A hostname or a database name
|
|
138
|
+
// inside the URL is part of the credential: a real password at db.acme-prod.example-corp.io was
|
|
139
|
+
// dismissed because "example" was in the line, and it was in the line because it was in the host.
|
|
140
|
+
const around = line.split(whole).join(' ');
|
|
141
|
+
if (PLACEHOLDER_WORDS.test(around))
|
|
78
142
|
return 'the line around it says it is an example';
|
|
79
143
|
// JSON and YAML put the key on its own line when the value is long, and an OpenAPI schema is all
|
|
80
144
|
// long values. nocodb declares `"example":` and then the token underneath it, so reading only the
|
|
@@ -83,6 +147,9 @@ export function looksPlanted(value, line, file, before = '') {
|
|
|
83
147
|
return 'the key above it says it is an example';
|
|
84
148
|
if (/(.)\1{5,}/.test(value))
|
|
85
149
|
return 'the value repeats one character, so it is a stand-in';
|
|
150
|
+
const harmless = urlCredentialIsHarmless(value, line);
|
|
151
|
+
if (harmless)
|
|
152
|
+
return harmless;
|
|
86
153
|
return null;
|
|
87
154
|
}
|
|
88
155
|
/** True when only the file's name suggests a template, whatever the value itself looks like. */
|
|
@@ -150,7 +217,16 @@ export function scanSecrets(root, files = sourceFiles(root)) {
|
|
|
150
217
|
for (const { kind, what, re, minEntropy } of PATTERNS) {
|
|
151
218
|
for (const match of text.matchAll(re)) {
|
|
152
219
|
const value = match[1];
|
|
153
|
-
|
|
220
|
+
// Where a pattern captures the credential separately, that is what gets judged. The two
|
|
221
|
+
// URL rules capture the whole URL first and the password second, and judging the URL meant
|
|
222
|
+
// a hostname or a database name decided whether a password was real: a genuine credential
|
|
223
|
+
// at db.acme-prod.example-corp.io was dismissed because its host contains "example", and
|
|
224
|
+
// any connection string to a database called sample or demo would go the same way.
|
|
225
|
+
// Suppressing a real secret is the worst thing this scanner can do, and it was one word in
|
|
226
|
+
// an unrelated part of the string away. The whole match is still what gets redacted, so
|
|
227
|
+
// nothing of the URL is printed either way.
|
|
228
|
+
const secret = match[2] ?? value;
|
|
229
|
+
const entropy = entropyOf(secret);
|
|
154
230
|
if (entropy < (minEntropy ?? 3))
|
|
155
231
|
continue;
|
|
156
232
|
const index = match.index ?? 0;
|
|
@@ -170,7 +246,8 @@ export function scanSecrets(root, files = sourceFiles(root)) {
|
|
|
170
246
|
line,
|
|
171
247
|
evidence: redact(lines[line - 1] ?? '', value),
|
|
172
248
|
entropy: Number(entropy.toFixed(2)),
|
|
173
|
-
placeholder: looksPlanted(
|
|
249
|
+
placeholder: looksPlanted(secret, lines[line - 1] ?? '', rel, lines[line - 2] ?? '', value),
|
|
250
|
+
localOnly: credentialGoesNowhere(value, lines[line - 1] ?? ''),
|
|
174
251
|
template: inATemplate(rel),
|
|
175
252
|
});
|
|
176
253
|
}
|
|
@@ -63,6 +63,15 @@ export function triageSecret(finding, root, accepted) {
|
|
|
63
63
|
if (finding.placeholder) {
|
|
64
64
|
return { ...base, verdict: 'noise', severity: 'low', why: `${finding.placeholder}: ${finding.evidence}`, next: 'nothing: this is not a credential' };
|
|
65
65
|
}
|
|
66
|
+
if (finding.localOnly) {
|
|
67
|
+
return {
|
|
68
|
+
...base,
|
|
69
|
+
verdict: 'not-reachable',
|
|
70
|
+
severity: 'low',
|
|
71
|
+
why: `${finding.evidence} - ${finding.localOnly}`,
|
|
72
|
+
next: 'nothing to revoke, because nothing outside this machine accepts it. Worth replacing with a value read from the environment, on a normal schedule.',
|
|
73
|
+
};
|
|
74
|
+
}
|
|
66
75
|
if (finding.template) {
|
|
67
76
|
// Only the file's name says this is a template; the value itself looks like a real credential.
|
|
68
77
|
// Dismissing it would hide the commonest accident there is - a working key pasted into an
|
|
@@ -4,7 +4,7 @@
|
|
|
4
4
|
// added. tests-in.ts keeps the questions every language shares - which language is this, does it
|
|
5
5
|
// look like tests at all, which parts of it are text rather than code - and this file holds the
|
|
6
6
|
// four readers: callbacks (JavaScript), indentation (Python), `end` (Ruby), and braces (the rest).
|
|
7
|
-
import { braced, isText, lineAt, textRanges } from './tests-in.js';
|
|
7
|
+
import { bodyBrace, braced, isText, lineAt, textRanges } from './tests-in.js';
|
|
8
8
|
/**
|
|
9
9
|
* `it('...', async () => { ... })` and its many spellings, including `test.each`, `it.skip` and a
|
|
10
10
|
* plain `function ()` callback. The body is taken by matching braces from the callback's opening one.
|
|
@@ -17,7 +17,7 @@ function jsTests(text) {
|
|
|
17
17
|
const at = hit.index ?? 0;
|
|
18
18
|
if (isText(ranges, at))
|
|
19
19
|
continue;
|
|
20
|
-
const open = text
|
|
20
|
+
const open = bodyBrace(text, at + hit[0].length);
|
|
21
21
|
if (open === -1)
|
|
22
22
|
continue;
|
|
23
23
|
const body = braced(text, open);
|
|
@@ -185,3 +185,35 @@ function closingQuote(text, from) {
|
|
|
185
185
|
}
|
|
186
186
|
return text.length;
|
|
187
187
|
}
|
|
188
|
+
/**
|
|
189
|
+
* Where a test's body actually starts.
|
|
190
|
+
*
|
|
191
|
+
* Taking the first `{` after the title is right until the callback destructures its argument:
|
|
192
|
+
*
|
|
193
|
+
* it.each(components)('transforms $name', ({ name, className }) => {
|
|
194
|
+
* expect(componentNameToClassName(name)).toBe(className)
|
|
195
|
+
* })
|
|
196
|
+
*
|
|
197
|
+
* The first `{` there belongs to the parameter list, so the body read back was `{ name, className }`
|
|
198
|
+
* - which contains no assertion, and the test was reported as one only a crash could fail.
|
|
199
|
+
*
|
|
200
|
+
* Measured on alphagov/govuk-frontend, the UK government's design system and the first real
|
|
201
|
+
* JavaScript project this was ever pointed at: 21 accusations across 1,817 tests, and 19 of them
|
|
202
|
+
* were this. Nineteen correct tests told they check nothing.
|
|
203
|
+
*
|
|
204
|
+
* A brace is a parameter list when what follows its match is `)` and then an arrow or the body's
|
|
205
|
+
* own brace. Anything else is the body.
|
|
206
|
+
*/
|
|
207
|
+
export function bodyBrace(text, from) {
|
|
208
|
+
let open = text.indexOf('{', from);
|
|
209
|
+
for (let guard = 0; open !== -1 && guard < 4; guard += 1) {
|
|
210
|
+
const inside = braced(text, open);
|
|
211
|
+
if (inside === null)
|
|
212
|
+
return -1;
|
|
213
|
+
const after = text.slice(open + inside.length + 2, open + inside.length + 24);
|
|
214
|
+
if (!/^\s*\)\s*(=>|\{)/.test(after))
|
|
215
|
+
return open;
|
|
216
|
+
open = text.indexOf('{', open + inside.length + 2);
|
|
217
|
+
}
|
|
218
|
+
return open;
|
|
219
|
+
}
|
|
@@ -2,14 +2,78 @@
|
|
|
2
2
|
//
|
|
3
3
|
// Split out in Phase 44 so that `swallowed.ts` could use them without importing the module that
|
|
4
4
|
// imports it. Nothing here knows what a test is; they are string surgery and nothing more.
|
|
5
|
-
|
|
5
|
+
// Comments cannot assert, and a commented-out check is the classic way a test stops checking.
|
|
6
|
+
//
|
|
7
|
+
// Written with line comments rather than a block, because the example below contains the exact
|
|
8
|
+
// character sequence that ends one - which is the whole point of the bug.
|
|
9
|
+
//
|
|
10
|
+
// This used to be a regex that replaced everything from a slash-star to the next star-slash, and a
|
|
11
|
+
// regex cannot tell a comment from a string that happens to contain one. gin's TestParseAccept
|
|
12
|
+
// parses an HTTP Accept header, and the media type `*/*` contains both sequences:
|
|
13
|
+
//
|
|
14
|
+
// parts := parseAccept("text/html , application/xml;q=0.9, */* ;q=0.8")
|
|
15
|
+
// assert.Len(t, parts, 4)
|
|
16
|
+
// assert.Equal(t, "text/html", parts[0])
|
|
17
|
+
// ...
|
|
18
|
+
// assert.Equal(t, "*/*", parts[3])
|
|
19
|
+
//
|
|
20
|
+
// A comment was opened inside the first string and closed in the last assertion. Everything
|
|
21
|
+
// between - all five assertions - became a single space, and a thoroughly checked test was
|
|
22
|
+
// reported as one that compares no value and only a crash could fail. Found on gin-gonic/gin, the
|
|
23
|
+
// first real Go project this was ever pointed at.
|
|
24
|
+
//
|
|
25
|
+
// So the text is walked instead, and a quoted string is skipped whole - which is what the rules
|
|
26
|
+
// above already assumed: "Strings are blanked as well as comments".
|
|
6
27
|
export function withoutComments(body) {
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
28
|
+
let out = '';
|
|
29
|
+
for (let i = 0; i < body.length; i += 1) {
|
|
30
|
+
const ch = body[i];
|
|
31
|
+
const next = body[i + 1] ?? '';
|
|
32
|
+
if (ch === '"' || ch === "'" || ch === '`') {
|
|
33
|
+
const end = endOfString(body, i);
|
|
34
|
+
out += body.slice(i, end + 1);
|
|
35
|
+
i = end;
|
|
36
|
+
continue;
|
|
37
|
+
}
|
|
38
|
+
if (ch === '/' && next === '*') {
|
|
39
|
+
const close = body.indexOf('*/', i + 2);
|
|
40
|
+
// An unclosed block comment runs to the end, which is what a compiler would do too.
|
|
41
|
+
out += ' ';
|
|
42
|
+
i = close === -1 ? body.length : close + 1;
|
|
43
|
+
continue;
|
|
44
|
+
}
|
|
45
|
+
const atLineStart = i === 0 || /\s/.test(body[i - 1] ?? '');
|
|
46
|
+
// `http://x` is not a comment, which is why the start of the line or a space is required.
|
|
47
|
+
if (ch === '/' && next === '/' && atLineStart) {
|
|
48
|
+
i = indexOrEnd(body, i) - 1;
|
|
49
|
+
continue;
|
|
50
|
+
}
|
|
51
|
+
if (ch === '#' && next !== '{' && atLineStart) {
|
|
52
|
+
i = indexOrEnd(body, i) - 1;
|
|
53
|
+
continue;
|
|
54
|
+
}
|
|
55
|
+
out += ch;
|
|
56
|
+
}
|
|
57
|
+
return out;
|
|
12
58
|
}
|
|
59
|
+
/** Where the string opening at `from` ends, honouring backslash escapes. */
|
|
60
|
+
function endOfString(text, from) {
|
|
61
|
+
const quote = text[from];
|
|
62
|
+
for (let i = from + 1; i < text.length; i += 1) {
|
|
63
|
+
if (text[i] === '\\') {
|
|
64
|
+
i += 1;
|
|
65
|
+
continue;
|
|
66
|
+
}
|
|
67
|
+
if (text[i] === quote)
|
|
68
|
+
return i;
|
|
69
|
+
}
|
|
70
|
+
return text.length - 1;
|
|
71
|
+
}
|
|
72
|
+
/** The newline at or after `from`, or the end of the text. */
|
|
73
|
+
const indexOrEnd = (text, from) => {
|
|
74
|
+
const at = text.indexOf(String.fromCharCode(10), from);
|
|
75
|
+
return at === -1 ? text.length : at;
|
|
76
|
+
};
|
|
13
77
|
export const stripLines = (body, lines) => body
|
|
14
78
|
.split(String.fromCharCode(10))
|
|
15
79
|
.filter((line) => !lines.some((drop) => line.trim() === drop.trim()))
|
|
@@ -129,12 +129,24 @@ export function howToWriteAReport(root) {
|
|
|
129
129
|
gradle: './gradlew test (already writes XML to build/test-results)',
|
|
130
130
|
playwright: 'npx playwright test --reporter=blob',
|
|
131
131
|
};
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
132
|
+
// Languages Tewip reads reports for but cannot run a suite in, so `detectFrameworks` - which
|
|
133
|
+
// exists to *run* tests - never names them. Without this, a Rust project was told to configure
|
|
134
|
+
// pytest, jest or dotnet, which is the exact advice the comment above calls the kind that gets a
|
|
135
|
+
// tool closed. Found by running `tewip watch` in BurntSushi/ripgrep.
|
|
136
|
+
const BY_MARKER = [
|
|
137
|
+
{ file: 'Cargo.toml', line: 'cargo nextest run --profile ci (with [profile.ci.junit] path = "report.xml" in .config/nextest.toml)' },
|
|
138
|
+
{ file: 'composer.json', line: './vendor/bin/phpunit --log-junit report.xml' },
|
|
139
|
+
{ file: 'Gemfile', line: 'bundle exec rspec --format RspecJunitFormatter --out report.xml (needs the rspec_junit_formatter gem)' },
|
|
140
|
+
];
|
|
141
|
+
const here = [
|
|
142
|
+
...detectFrameworks(root).map((f) => JUNIT[f.id]),
|
|
143
|
+
...BY_MARKER.filter((m) => existsSync(join(root, m.file))).map((m) => m.line),
|
|
144
|
+
].filter((line, i, all) => Boolean(line) && all.indexOf(line) === i);
|
|
135
145
|
const head = 'The tests failed but wrote no report Tewip can read.';
|
|
136
146
|
if (here.length === 0) {
|
|
137
|
-
|
|
147
|
+
// Nothing here was recognised, so this is a list of examples and says so. Naming three runners
|
|
148
|
+
// as though they were this project's is how the advice went wrong in the first place.
|
|
149
|
+
return `${head} Tewip did not recognise the runner here, so it cannot name the command. Whatever you use, make it write JUnit XML - for example pytest --junitxml=report.xml, jest --reporters=jest-junit, or dotnet test --logger trx.`;
|
|
138
150
|
}
|
|
139
151
|
return [`${head} For the runner${here.length === 1 ? '' : 's'} this project has, that is:`, ...here.map((line) => ` ${line}`)].join(String.fromCharCode(10));
|
|
140
152
|
}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tewip-ai",
|
|
3
|
-
"version": "0.70.
|
|
3
|
+
"version": "0.70.16",
|
|
4
4
|
"mcpName": "io.github.hmamut39/tewip",
|
|
5
5
|
"description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
|
|
6
6
|
"keywords": [
|
|
@@ -65,6 +65,7 @@
|
|
|
65
65
|
"prepack": "npm run build",
|
|
66
66
|
"tewip": "node cli/tewip.ts",
|
|
67
67
|
"version": "node scripts/sync-version.ts && git add server.json",
|
|
68
|
+
"channels": "node scripts/channels.ts",
|
|
68
69
|
"corpus": "node scripts/corpus.ts"
|
|
69
70
|
},
|
|
70
71
|
"license": "Apache-2.0",
|