tewip-ai 0.35.0 → 0.37.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +99 -9
- package/dist/cli/help.js +16 -2
- package/dist/cli/verbs-prove.js +20 -1
- package/dist/engine/access/access-command.js +82 -13
- package/dist/engine/access/merge.js +119 -0
- package/dist/engine/access/reports.js +206 -0
- package/dist/engine/access/wcag.js +11 -1
- package/dist/engine/perf/compare.js +97 -0
- package/dist/engine/perf/history.js +74 -0
- package/dist/engine/perf/k6.js +111 -0
- package/dist/engine/perf/perf-command.js +120 -0
- package/dist/mcp/tools-prove.js +51 -3
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -138,17 +138,19 @@ and you always get the current version.
|
|
|
138
138
|
|
|
139
139
|
### The editor extension (VS Code, Cursor, Windsurf, VSCodium)
|
|
140
140
|
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
`npm install -g tewip-ai`.
|
|
141
|
+
**Open the Extensions pane and search for "Tewip".** It is in both registries, so that works in
|
|
142
|
+
VS Code, Cursor, Windsurf, VSCodium, Gitpod and Eclipse Theia:
|
|
144
143
|
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
dependencies*, and *What our corrections add up to*. The editor's AI assistant also picks Tewip up
|
|
148
|
-
as an MCP server with nothing to configure.
|
|
144
|
+
- **VS Code** — [Visual Studio Marketplace](https://marketplace.visualstudio.com/items?itemName=hmamut39.tewip)
|
|
145
|
+
- **Cursor, Windsurf, VSCodium, Gitpod, Theia** — [Open VSX](https://open-vsx.org/extension/hmamut39/tewip)
|
|
149
146
|
|
|
150
|
-
|
|
151
|
-
|
|
147
|
+
Or install the `.vsix` attached to any [release](https://github.com/hmamut39/tewip/releases) with
|
|
148
|
+
**Extensions: Install from VSIX…**. Either way it needs the command line: `npm install -g tewip-ai`.
|
|
149
|
+
|
|
150
|
+
Press `Ctrl+Shift+P` and type "Tewip" for: *Is this work done?*, *Can everybody use this?*, *Could
|
|
151
|
+
these tests ever fail?*, *Would the tests catch a bug in this file?*, *Which tests could notice this
|
|
152
|
+
change?*, *What will this migration do to a live database?*, *Why did CI fail on this branch?*, and
|
|
153
|
+
more. The editor's AI assistant also picks Tewip up as an MCP server with nothing to configure.
|
|
152
154
|
|
|
153
155
|
### 2. From any AI assistant, through MCP (the whole IT team)
|
|
154
156
|
|
|
@@ -574,6 +576,58 @@ With `--json` the verdict is **sealed**: editing `not-ready` to `ready` in the f
|
|
|
574
576
|
fail, so a verdict that travels into a ticket or a release note can be checked rather than trusted. It is
|
|
575
577
|
also an MCP tool, so an agent can ask whether its own work is done before saying that it is.
|
|
576
578
|
|
|
579
|
+
## Is this change slower? (Phase 30, new)
|
|
580
|
+
|
|
581
|
+
```bash
|
|
582
|
+
tewip perf summary.json --record # on main: add this run to the baseline
|
|
583
|
+
tewip perf summary.json --strict # on a branch: fail the build on a real regression
|
|
584
|
+
```
|
|
585
|
+
|
|
586
|
+
Tewip has imported k6 results for a while and never had an opinion about them. Now it answers the
|
|
587
|
+
question every team asks before a release.
|
|
588
|
+
|
|
589
|
+
**The hard part is not the arithmetic.** "p95 went from 180ms to 210ms, +16%" is four lines of code
|
|
590
|
+
and worthless, because nobody knows whether 16% is the change or the laptop. Here are **four real k6
|
|
591
|
+
runs of code that did not change**:
|
|
592
|
+
|
|
593
|
+
```
|
|
594
|
+
run 1 p95 = 60.9ms
|
|
595
|
+
run 2 p95 = 60.8ms
|
|
596
|
+
run 3 p95 = 62.8ms
|
|
597
|
+
run 4 p95 = 61.4ms
|
|
598
|
+
```
|
|
599
|
+
|
|
600
|
+
A 3% spread from nothing at all. A tool that calls that a regression gets switched off.
|
|
601
|
+
|
|
602
|
+
So Tewip answers **two questions separately**, because they fail separately:
|
|
603
|
+
|
|
604
|
+
| | |
|
|
605
|
+
|---|---|
|
|
606
|
+
| **Did your own thresholds hold?** | You already wrote what "fast enough" means in your k6 script. Tewip reads the result instead of asking you to write a budget twice. |
|
|
607
|
+
| **Is it slower than it was?** | Answered **only** from three or more recorded baselines. Below that: *cannot say*. |
|
|
608
|
+
|
|
609
|
+
Given those three real baselines, a fourth identical run at 61.4ms is **not** called a regression —
|
|
610
|
+
and a genuine one is:
|
|
611
|
+
|
|
612
|
+
```
|
|
613
|
+
- `http_req_duration` p95 282.5ms — **slower**
|
|
614
|
+
- baseline 60.8ms–62.8ms over 3 runs, middle 60.9ms
|
|
615
|
+
- 282.5ms is outside everything 3 baseline runs produced (60.8-62.8ms) by more than
|
|
616
|
+
their own spread, so the runs cannot explain it. That is +364% on the middle run.
|
|
617
|
+
```
|
|
618
|
+
|
|
619
|
+
**Recording is deliberately not automatic.** A tool that files every run into the baseline ratchets:
|
|
620
|
+
today's regression becomes tomorrow's normal, the comparison keeps saying "within range", and a
|
|
621
|
+
service triples in latency over a quarter without one report ever being wrong. `--record` on main,
|
|
622
|
+
compare everywhere else.
|
|
623
|
+
|
|
624
|
+
**It never predicts a duration, a capacity, or what a user will feel.** It cannot see your production
|
|
625
|
+
hardware, your real data volume or your real traffic, and every report says so.
|
|
626
|
+
|
|
627
|
+
One more thing it refuses: if your summary came from `--new-machine-readable-summary`, that export
|
|
628
|
+
records **no threshold results at all** — the word appears in it only inside a copy of your script.
|
|
629
|
+
Tewip says "this export cannot tell me" rather than "nothing was breached".
|
|
630
|
+
|
|
577
631
|
## Can everybody use it? (Phase 28, new)
|
|
578
632
|
|
|
579
633
|
```bash
|
|
@@ -629,6 +683,42 @@ Measured on four public projects, never on fixtures — excalidraw, bootstrap, v
|
|
|
629
683
|
classes were found and fixed along the way, every one of them a defect in Tewip rather than in the
|
|
630
684
|
projects, and none of them reachable from a fixture. They are listed in [ROADMAP.md](ROADMAP.md).
|
|
631
685
|
|
|
686
|
+
### Already run axe? Give Tewip the JSON (Phase 29, new)
|
|
687
|
+
|
|
688
|
+
```bash
|
|
689
|
+
tewip access --report axe-results.json
|
|
690
|
+
```
|
|
691
|
+
|
|
692
|
+
Reading markup cannot see contrast. A browser can, so if your Playwright, Cypress or jest-axe suite
|
|
693
|
+
already writes an axe report, hand it over — and Tewip will triage it instead of relaying it.
|
|
694
|
+
|
|
695
|
+
This is where the numbers get uncomfortable. axe 4.14 over nine real pages returned **174 findings**:
|
|
696
|
+
|
|
697
|
+
| | |
|
|
698
|
+
|---|---|
|
|
699
|
+
| **53 barriers** | a real WCAG criterion, with the **EN 301 549** clause the EU Accessibility Act cites |
|
|
700
|
+
| **51 advice** | axe rules tagged `best-practice` that map to **no criterion at all** |
|
|
701
|
+
| **70 questions** | axe's own `incomplete` — it could not determine them |
|
|
702
|
+
|
|
703
|
+
**42% of what axe hands you tests no standard.** `region` alone — "all content should be contained by
|
|
704
|
+
landmarks" — was 40 findings, outnumbering every real barrier on the page. Tewip reports those as advice,
|
|
705
|
+
never as barriers, and they never fail a build. That separation is most of the point.
|
|
706
|
+
|
|
707
|
+
And the unchecked list shrinks, honestly: **one** of its seven entries — 1.4.3 Contrast — moves to
|
|
708
|
+
checked. The other six stay, because no tool settles them. The report also names how many pages the run
|
|
709
|
+
covered, and says plainly that nothing is known about a page it did not open or a state it did not reach.
|
|
710
|
+
|
|
711
|
+
```
|
|
712
|
+
## Checked here only because a browser run was given
|
|
713
|
+
|
|
714
|
+
- **1.4.3 Contrast (Minimum)** - would otherwise be unreachable: needs the computed colours of a
|
|
715
|
+
rendered page, not the markup
|
|
716
|
+
```
|
|
717
|
+
|
|
718
|
+
A source finding is set aside only when the run opened **that same file** and re-ran the equivalent rule.
|
|
719
|
+
In a React or Vue project that means nothing is set aside — the source and the rendered output are two
|
|
720
|
+
different artefacts, and neither is evidence about the other.
|
|
721
|
+
|
|
632
722
|
## Teach it your house rules (Phase 27)
|
|
633
723
|
|
|
634
724
|
```bash
|
package/dist/cli/help.js
CHANGED
|
@@ -118,7 +118,17 @@ Usage:
|
|
|
118
118
|
is what was asked for covered. Answers ready, not ready, or cannot
|
|
119
119
|
say - and cannot say is never rounded up to ready. --json writes a
|
|
120
120
|
sealed verdict that can be checked later. No model, no cost.
|
|
121
|
-
tewip
|
|
121
|
+
tewip perf <k6-summary.json> [--record] [--label checkout] [--strict] [--out perf.md]
|
|
122
|
+
Is this change slower, and does it matter? Reads a k6 run and
|
|
123
|
+
answers two questions separately: did the thresholds you wrote
|
|
124
|
+
hold, and is this slower than it was. The second is answered only
|
|
125
|
+
from repeated baselines - with fewer than three it says cannot
|
|
126
|
+
say, because one run against one run cannot tell a regression
|
|
127
|
+
from a busy machine. --record adds a run to the baseline, and is
|
|
128
|
+
deliberately not automatic: a regression recorded by accident
|
|
129
|
+
becomes next week's normal. It never predicts a duration or a
|
|
130
|
+
capacity. No model, no cost.
|
|
131
|
+
tewip access [<file or directory>] [--report axe.json] [--strict] [--out access.md]
|
|
122
132
|
Can everybody use this? Reads your markup and names the barriers
|
|
123
133
|
that reading alone can prove - a picture with no alt text, a field
|
|
124
134
|
with no label, a div only a mouse can use - each against its WCAG
|
|
@@ -127,7 +137,11 @@ Usage:
|
|
|
127
137
|
criterion it cannot reach without rendering the page (contrast,
|
|
128
138
|
focus order, whether the alt text is any good) is printed every
|
|
129
139
|
time. It never says a page is accessible, only what was checked.
|
|
130
|
-
|
|
140
|
+
Give it --report with the JSON from an axe run and it reads that
|
|
141
|
+
too: contrast moves out of the unchecked list, axe rules that map
|
|
142
|
+
to no WCAG criterion are reported as advice rather than barriers
|
|
143
|
+
(51 of 174 findings on nine real pages), and what axe could not
|
|
144
|
+
determine becomes a question. No browser, no network, no model.
|
|
131
145
|
tewip standards [--diff main...HEAD] [--strict] [--out standards.md]
|
|
132
146
|
Teach Tewip your house rules. Markdown files in
|
|
133
147
|
.tewip/standards/ - a forbid or require line is checked and
|
package/dist/cli/verbs-prove.js
CHANGED
|
@@ -20,13 +20,32 @@ export async function proveVerb(verb, args) {
|
|
|
20
20
|
});
|
|
21
21
|
return true;
|
|
22
22
|
}
|
|
23
|
+
case 'perf': {
|
|
24
|
+
const { runPerf } = await import('../engine/perf/perf-command.js');
|
|
25
|
+
const paths = args.filter((a, i) => !a.startsWith('--') && !['--out', '--label'].includes(args[i - 1] ?? ''));
|
|
26
|
+
if (paths.length === 0) {
|
|
27
|
+
console.error('tewip perf needs the JSON from a k6 run: tewip perf summary.json [--record] [--label checkout]');
|
|
28
|
+
process.exitCode = 1;
|
|
29
|
+
return true;
|
|
30
|
+
}
|
|
31
|
+
runPerf({
|
|
32
|
+
root: process.cwd(),
|
|
33
|
+
summary: paths[0],
|
|
34
|
+
label: flag(args, 'label'),
|
|
35
|
+
record: args.includes('--record'),
|
|
36
|
+
out: flag(args, 'out'),
|
|
37
|
+
gate: args.includes('--strict'),
|
|
38
|
+
});
|
|
39
|
+
return true;
|
|
40
|
+
}
|
|
23
41
|
case 'access': {
|
|
24
42
|
const { runAccess } = await import('../engine/access/access-command.js');
|
|
25
|
-
const paths = args.filter((a, i) => !a.startsWith('--') && args[i - 1] !== '--out');
|
|
43
|
+
const paths = args.filter((a, i) => !a.startsWith('--') && args[i - 1] !== '--out' && args[i - 1] !== '--report');
|
|
26
44
|
runAccess({
|
|
27
45
|
root: process.cwd(),
|
|
28
46
|
only: paths[0],
|
|
29
47
|
out: flag(args, 'out'),
|
|
48
|
+
report: flag(args, 'report'),
|
|
30
49
|
gate: args.includes('--strict'),
|
|
31
50
|
});
|
|
32
51
|
return true;
|
|
@@ -13,6 +13,8 @@ import { dirname, join, relative, resolve } from 'node:path';
|
|
|
13
13
|
import { sourceFiles } from '../secure/secrets.js';
|
|
14
14
|
import { JS_LIKE, MARKUP, tags } from './markup.js';
|
|
15
15
|
import { checkFile } from './checks.js';
|
|
16
|
+
import { readAxeReport } from './reports.js';
|
|
17
|
+
import { merge, nowChecked, pagesIn, renderBrowserSection, standardOf, stillNotChecked } from './merge.js';
|
|
16
18
|
import { blocks, criterionOf, NOT_CHECKED, RULES } from './wcag.js';
|
|
17
19
|
const ORDER = { blocker: 0, serious: 1, moderate: 2, minor: 3 };
|
|
18
20
|
/** One line per finding a person has already judged: "<rule> <file>:<line> - reason". */
|
|
@@ -101,10 +103,14 @@ function section(title, findings, lines) {
|
|
|
101
103
|
}
|
|
102
104
|
}
|
|
103
105
|
export function renderAccess(result, scanned) {
|
|
104
|
-
const { barriers, questions, accepted, filesRead } = result;
|
|
106
|
+
const { barriers, questions, accepted, filesRead, browser } = result;
|
|
105
107
|
const lines = ['# Tewip accessibility check', ''];
|
|
106
108
|
lines.push(`Read ${filesRead} markup ${filesRead === 1 ? 'file' : 'files'} under \`${scanned}\`, without rendering any of them.`, '');
|
|
107
|
-
if (
|
|
109
|
+
if (browser) {
|
|
110
|
+
lines.push(`Also read an axe run over ${browser.pages.length} rendered ${browser.pages.length === 1 ? 'page' : 'pages'}, ` +
|
|
111
|
+
'which settles what reading markup cannot.', '');
|
|
112
|
+
}
|
|
113
|
+
if (barriers.length === 0 && questions.length === 0 && !browser) {
|
|
108
114
|
lines.push('No barrier was found that reading the markup can settle, and nothing needs a decision.', '');
|
|
109
115
|
}
|
|
110
116
|
else {
|
|
@@ -115,22 +121,67 @@ export function renderAccess(result, scanned) {
|
|
|
115
121
|
];
|
|
116
122
|
if (accepted.length > 0)
|
|
117
123
|
counts.push(`${accepted.length} already accepted`);
|
|
124
|
+
if (browser) {
|
|
125
|
+
counts.unshift(`${browser.barriers.length} barriers the browser run proves`);
|
|
126
|
+
if (browser.questions.length > 0)
|
|
127
|
+
counts.push(`${browser.questions.length} the browser could not determine`);
|
|
128
|
+
}
|
|
118
129
|
lines.push(`**${counts.join(', ')}.**`, '');
|
|
119
130
|
}
|
|
120
|
-
|
|
131
|
+
if (browser) {
|
|
132
|
+
renderBrowserSection('Barriers seen in the browser', browser.barriers, lines);
|
|
133
|
+
renderBrowserSection('The browser could not determine these', browser.questions, lines);
|
|
134
|
+
}
|
|
135
|
+
section('Barriers read from the source', [...barriers].sort(worstFirst), lines);
|
|
121
136
|
section('Only a person can settle these', [...questions].sort(worstFirst), lines);
|
|
137
|
+
if (browser && browser.advice.length > 0) {
|
|
138
|
+
// Kept out of the counts above on purpose. These are axe's own best-practice rules, which test
|
|
139
|
+
// no standard: on nine real pages they were 51 of 174 findings, outnumbering every real barrier.
|
|
140
|
+
// Reporting them as equals is how a team learns to ignore the whole report.
|
|
141
|
+
const rules = [...new Set(browser.advice.map((f) => f.rule))].sort();
|
|
142
|
+
lines.push('## Advice, not barriers', '');
|
|
143
|
+
lines.push(`${browser.advice.length} findings come from axe rules that map to no WCAG criterion - ` +
|
|
144
|
+
`${rules.join(', ')}. They are somebody's good idea rather than a standard, so they are not counted as barriers ` +
|
|
145
|
+
'and never fail a build. Worth reading once; not worth a release.', '');
|
|
146
|
+
}
|
|
147
|
+
if (browser && browser.superseded.length > 0) {
|
|
148
|
+
lines.push('## Set aside, because the browser run checked them', '');
|
|
149
|
+
lines.push(`${browser.superseded.length} source ${browser.superseded.length === 1 ? 'finding' : 'findings'} ` +
|
|
150
|
+
'are not listed above: the axe run checked their criterion on a rendered page, so its answer is the better ' +
|
|
151
|
+
'one and repeating them here would count the same barrier twice. They are in the JSON beside this report.', '');
|
|
152
|
+
}
|
|
122
153
|
if (accepted.length > 0) {
|
|
123
154
|
lines.push('## Already accepted', '');
|
|
124
155
|
lines.push(`${accepted.length} ${accepted.length === 1 ? 'finding is' : 'findings are'} listed in \`.tewip/access-accepted.txt\`, so they are not repeated here.`, '');
|
|
125
156
|
}
|
|
157
|
+
// Coverage comes from every rule that ran, not from the ones that found something: a contrast
|
|
158
|
+
// check that ran and passed has still checked contrast.
|
|
159
|
+
const ran = browser?.rulesRun ?? new Set();
|
|
160
|
+
const settled = nowChecked(ran);
|
|
161
|
+
const remaining = stillNotChecked(ran);
|
|
162
|
+
if (settled.length > 0) {
|
|
163
|
+
lines.push('## Checked here only because a browser run was given', '');
|
|
164
|
+
for (const n of settled)
|
|
165
|
+
lines.push(`- **${n.criterion}** - would otherwise be unreachable: ${n.why}`);
|
|
166
|
+
lines.push('');
|
|
167
|
+
}
|
|
126
168
|
lines.push('## What was not checked', '');
|
|
127
|
-
lines.push(
|
|
128
|
-
|
|
169
|
+
lines.push(browser
|
|
170
|
+
? 'Even with the browser run, none of the following was tested:'
|
|
171
|
+
: 'This read the markup. It did not render a page, so none of the following was tested:', '');
|
|
172
|
+
for (const n of remaining)
|
|
129
173
|
lines.push(`- **${n.criterion}** - ${n.why}`);
|
|
174
|
+
if (browser) {
|
|
175
|
+
lines.push('');
|
|
176
|
+
lines.push(`The run covered ${browser.pages.length} ${browser.pages.length === 1 ? 'page' : 'pages'}. ` +
|
|
177
|
+
'Nothing is known about a page it did not open, or about a state it did not reach - a dialog that was never ' +
|
|
178
|
+
'opened, a form that was never submitted, an error that was never triggered.');
|
|
179
|
+
}
|
|
130
180
|
lines.push('');
|
|
131
181
|
lines.push('A clean result here means these checks found nothing, not that the product is accessible. ' +
|
|
132
|
-
'Roughly a third of WCAG can be judged by a machine at all
|
|
133
|
-
|
|
182
|
+
'Roughly a third of WCAG can be judged by a machine at all' +
|
|
183
|
+
(browser ? '' : ', and reading source rather than a rendered page reaches less than that') +
|
|
184
|
+
'. The rest needs a person, ideally one who uses a screen reader.', '');
|
|
134
185
|
lines.push('To accept a finding, add its key to `.tewip/access-accepted.txt`, one per line, with a reason after ` - `.', '');
|
|
135
186
|
return lines.join('\n');
|
|
136
187
|
}
|
|
@@ -140,18 +191,36 @@ export function runAccess(cmd) {
|
|
|
140
191
|
const alreadyJudged = readAccepted(root);
|
|
141
192
|
const accepted = findings.filter((f) => alreadyJudged.has(keyOf(f)));
|
|
142
193
|
const open = findings.filter((f) => !alreadyJudged.has(keyOf(f)));
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
const
|
|
194
|
+
// Phase 29: a browser run, where one was given. It is the better witness for every criterion it
|
|
195
|
+
// checked, so the source findings for those criteria are set aside rather than repeated.
|
|
196
|
+
const run = cmd.report ? readAxeReport(resolve(root, cmd.report)) : undefined;
|
|
197
|
+
const graded = run ? merge(open, run, root) : undefined;
|
|
198
|
+
const kept = graded ? graded.kept : open;
|
|
199
|
+
const barriers = kept.filter((f) => f.confidence === 'barrier');
|
|
200
|
+
const questions = kept.filter((f) => f.confidence === 'ask-a-person');
|
|
201
|
+
const browser = run && graded
|
|
202
|
+
? {
|
|
203
|
+
barriers: run.findings.filter((f) => f.confidence === 'barrier'),
|
|
204
|
+
questions: run.findings.filter((f) => f.confidence === 'ask-a-person'),
|
|
205
|
+
advice: run.findings.filter((f) => f.confidence === 'advice'),
|
|
206
|
+
superseded: graded.superseded,
|
|
207
|
+
pages: pagesIn(run),
|
|
208
|
+
rulesRun: run.rulesRun,
|
|
209
|
+
}
|
|
210
|
+
: undefined;
|
|
211
|
+
const result = { barriers, questions, accepted, filesRead, ...(browser ? { browser } : {}) };
|
|
146
212
|
const report = renderAccess(result, cmd.only ? resolve(root, cmd.only) : root);
|
|
147
213
|
console.log(report);
|
|
148
214
|
// What fails a build is deliberately narrower than what gets reported. A question never fails it:
|
|
149
215
|
// a build that breaks because something needs a human decision teaches a team to stop asking. Nor
|
|
150
216
|
// does a moderate finding - bootstrap's example pages skip a heading level, which is worth knowing
|
|
151
217
|
// and not worth blocking a release over. Only the barriers that stop somebody outright.
|
|
152
|
-
|
|
153
|
-
|
|
154
|
-
|
|
218
|
+
// A browser barrier counts the same way: axe saw it on a rendered page, which is stronger
|
|
219
|
+
// evidence than reading source, not weaker. Advice never counts, whatever axe's impact says.
|
|
220
|
+
const blocking = barriers.filter((f) => blocks(f.rule)).length + (browser?.barriers.filter((f) => f.severity === 'blocker').length ?? 0);
|
|
221
|
+
if (cmd.gate && blocking > 0) {
|
|
222
|
+
const found = barriers.length + (browser?.barriers.length ?? 0);
|
|
223
|
+
console.error(`${String.fromCharCode(10)}${blocking} accessibility barrier(s) that stop somebody outright, out of ${found} found.`);
|
|
155
224
|
process.exitCode = 1;
|
|
156
225
|
}
|
|
157
226
|
if (cmd.out) {
|
|
@@ -0,0 +1,119 @@
|
|
|
1
|
+
import { EQUIVALENT_AXE_RULE } from './reports.js';
|
|
2
|
+
import { NOT_CHECKED, RULES } from './wcag.js';
|
|
3
|
+
/** `1.1.1 Non-text Content` -> `1.1.1`, so a criterion from either source can be compared. */
|
|
4
|
+
export function criterionNumber(criterion) {
|
|
5
|
+
return /^\d+(?:\.\d+)*/.exec(criterion.trim())?.[0] ?? '';
|
|
6
|
+
}
|
|
7
|
+
/** The criterion a source-read rule belongs to, as a bare number. */
|
|
8
|
+
export function criterionOfRule(rule) {
|
|
9
|
+
const r = RULES[rule];
|
|
10
|
+
return r ? criterionNumber(r.criterion) : '';
|
|
11
|
+
}
|
|
12
|
+
/**
|
|
13
|
+
* Sets aside the source findings the run actually re-checked, rule by rule.
|
|
14
|
+
*
|
|
15
|
+
* The test is whether the *equivalent axe rule ran*, not whether axe found something and not
|
|
16
|
+
* whether some other rule under the same criterion ran. A rule that ran and passed still
|
|
17
|
+
* supersedes - axe looked at the rendered page and disagreed, which is the better evidence. A rule
|
|
18
|
+
* with no axe equivalent, like `click-without-keyboard`, is never superseded by anything.
|
|
19
|
+
*/
|
|
20
|
+
export function merge(source, run, root) {
|
|
21
|
+
const opened = pagesAsFiles(run, root);
|
|
22
|
+
const kept = [];
|
|
23
|
+
const superseded = [];
|
|
24
|
+
for (const f of source) {
|
|
25
|
+
const equivalent = EQUIVALENT_AXE_RULE[f.rule];
|
|
26
|
+
// Both conditions, and the second is the one that was missing. A run of nine pages does not
|
|
27
|
+
// supersede a finding in the hundredth file it never opened: on bootstrap that silently
|
|
28
|
+
// discarded all 44 source findings, including every one in the example pages axe never saw.
|
|
29
|
+
// A component file is never a page a browser opens, so in a React or Vue project nothing is
|
|
30
|
+
// superseded at all - correctly, because the source and the rendered output are two artefacts
|
|
31
|
+
// and neither is evidence about the other.
|
|
32
|
+
if (equivalent !== undefined && run.rulesRun.has(equivalent) && opened.has(f.file))
|
|
33
|
+
superseded.push(f);
|
|
34
|
+
else
|
|
35
|
+
kept.push(f);
|
|
36
|
+
}
|
|
37
|
+
return { kept, superseded, covered: run.criteriaCovered };
|
|
38
|
+
}
|
|
39
|
+
/** The pages a run opened, as repository-relative paths, for the ones that were local files. */
|
|
40
|
+
export function pagesAsFiles(run, root) {
|
|
41
|
+
const out = new Set();
|
|
42
|
+
const base = root.replaceAll('\\', '/').replace(/\/$/, '').toLowerCase();
|
|
43
|
+
for (const page of run.pages) {
|
|
44
|
+
// `file:///C:/repo/a.html` -> `C:/repo/a.html`, but `file:///repo/a.html` -> `/repo/a.html`.
|
|
45
|
+
// Stripping every slash after the scheme is wrong on anything but Windows: the third one is
|
|
46
|
+
// the root, not padding.
|
|
47
|
+
const stripped = decodeURIComponent(page.replace(/^file:\/\//i, '')).replaceAll('\\', '/');
|
|
48
|
+
const path = /^\/[A-Za-z]:/.test(stripped) ? stripped.slice(1) : stripped;
|
|
49
|
+
const lower = path.toLowerCase();
|
|
50
|
+
if (lower.startsWith(`${base}/`))
|
|
51
|
+
out.add(path.slice(base.length + 1));
|
|
52
|
+
}
|
|
53
|
+
return out;
|
|
54
|
+
}
|
|
55
|
+
/**
|
|
56
|
+
* What is still unchecked once the browser run is counted.
|
|
57
|
+
*
|
|
58
|
+
* This is the payoff of the phase and the reason it is worth building: the list of criteria Tewip
|
|
59
|
+
* admits it cannot reach should get shorter when real evidence arrives, rather than being printed
|
|
60
|
+
* unchanged beneath a browser run that already settled half of it.
|
|
61
|
+
*/
|
|
62
|
+
export function stillNotChecked(rulesRun) {
|
|
63
|
+
return NOT_CHECKED.filter((n) => n.settledBy === undefined || !rulesRun.has(n.settledBy));
|
|
64
|
+
}
|
|
65
|
+
/** The criteria from NOT_CHECKED that this run did settle, for the report to say so. */
|
|
66
|
+
export function nowChecked(rulesRun) {
|
|
67
|
+
return NOT_CHECKED.filter((n) => n.settledBy !== undefined && rulesRun.has(n.settledBy));
|
|
68
|
+
}
|
|
69
|
+
const SEVERITY_ORDER = { blocker: 0, serious: 1, moderate: 2, minor: 3 };
|
|
70
|
+
export function worstBrowserFirst(a, b) {
|
|
71
|
+
const d = SEVERITY_ORDER[a.severity] - SEVERITY_ORDER[b.severity];
|
|
72
|
+
if (d !== 0)
|
|
73
|
+
return d;
|
|
74
|
+
if (a.rule !== b.rule)
|
|
75
|
+
return a.rule < b.rule ? -1 : 1;
|
|
76
|
+
return a.selector < b.selector ? -1 : 1;
|
|
77
|
+
}
|
|
78
|
+
/** Groups by rule, keeping the order the comparator produced. */
|
|
79
|
+
function byRule(findings) {
|
|
80
|
+
const out = new Map();
|
|
81
|
+
for (const f of [...findings].sort(worstBrowserFirst)) {
|
|
82
|
+
const list = out.get(f.rule);
|
|
83
|
+
if (list)
|
|
84
|
+
list.push(f);
|
|
85
|
+
else
|
|
86
|
+
out.set(f.rule, [f]);
|
|
87
|
+
}
|
|
88
|
+
return out;
|
|
89
|
+
}
|
|
90
|
+
/** How a browser finding names its standard: both WCAG and the clause the EAA cites. */
|
|
91
|
+
export function standardOf(f) {
|
|
92
|
+
if (!f.criterion)
|
|
93
|
+
return 'no criterion - this is a best-practice rule, not a standard';
|
|
94
|
+
const en = f.en301549 ? `, EN 301 549 clause ${f.en301549}` : '';
|
|
95
|
+
return `WCAG 2.2 ${f.criterion} (${f.level ?? 'A'})${en}`;
|
|
96
|
+
}
|
|
97
|
+
export function renderBrowserSection(title, findings, lines, limit = 8) {
|
|
98
|
+
if (findings.length === 0)
|
|
99
|
+
return;
|
|
100
|
+
lines.push(`## ${title}`, '');
|
|
101
|
+
for (const [rule, found] of byRule(findings)) {
|
|
102
|
+
lines.push(`### ${rule} - ${found.length} ${found.length === 1 ? 'place' : 'places'}`, '');
|
|
103
|
+
lines.push(`${standardOf(found[0])}, ${found[0].severity}.`, '');
|
|
104
|
+
lines.push(`${found[0].why}`, '');
|
|
105
|
+
for (const f of found.slice(0, limit)) {
|
|
106
|
+
lines.push(`- \`${f.selector}\``);
|
|
107
|
+
lines.push(` - \`${f.html}\``);
|
|
108
|
+
if (f.next && f.next !== found[0].why)
|
|
109
|
+
lines.push(` - ${f.next}`);
|
|
110
|
+
}
|
|
111
|
+
if (found.length > limit)
|
|
112
|
+
lines.push(`- ...and ${found.length - limit} more of the same.`);
|
|
113
|
+
lines.push('');
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
/** The pages a browser run covered, which bounds everything it can be said to prove. */
|
|
117
|
+
export function pagesIn(run) {
|
|
118
|
+
return [...run.pages].sort();
|
|
119
|
+
}
|
|
@@ -0,0 +1,206 @@
|
|
|
1
|
+
// Phase 29: reading a real browser run, so the criteria Phase 28 says it cannot reach stop being
|
|
2
|
+
// unreachable.
|
|
3
|
+
//
|
|
4
|
+
// `tewip access` reads markup and is honest that contrast, focus order and anything else needing a
|
|
5
|
+
// rendered page is outside it. The answer is not for Tewip to ship a browser. It is to read the run
|
|
6
|
+
// a team already produces - axe, in a Playwright, Cypress or jest-axe suite - and do to its output
|
|
7
|
+
// what Phase 9 does to a security scanner's: decide which findings matter, and say why.
|
|
8
|
+
//
|
|
9
|
+
// Why that is worth doing, measured on axe 4.14 over nine real pages (bootstrap's own component
|
|
10
|
+
// pages and excalidraw's index): 122 findings, of which **51 carry no WCAG criterion at all**. Forty
|
|
11
|
+
// are `region` ("all content should be contained by landmarks") and nine are `landmark-one-main`.
|
|
12
|
+
// Both are axe's own `best-practice` rules, not failures of any standard, and they outnumber every
|
|
13
|
+
// real barrier on the page. A team that is handed all 122 as equals stops reading them, which is the
|
|
14
|
+
// reported reason accessibility tooling gets switched off.
|
|
15
|
+
//
|
|
16
|
+
// So the tags decide, not the impact:
|
|
17
|
+
// a wcag* tag -> a numbered criterion a regulator can cite, and a barrier;
|
|
18
|
+
// best-practice only -> advice, reported apart and never failing a build;
|
|
19
|
+
// `incomplete` -> axe could not determine it, which is exactly Tewip's "ask a person".
|
|
20
|
+
import { readFileSync } from 'node:fs';
|
|
21
|
+
/** axe's impact, in Tewip's words. A WCAG-tagged `moderate` is still worth fixing, not a blocker. */
|
|
22
|
+
const SEVERITY = {
|
|
23
|
+
critical: 'blocker',
|
|
24
|
+
serious: 'blocker',
|
|
25
|
+
moderate: 'moderate',
|
|
26
|
+
minor: 'minor',
|
|
27
|
+
};
|
|
28
|
+
/**
|
|
29
|
+
* `wcag143` -> `1.4.3`. The first digit is the principle, the second the guideline, the rest the
|
|
30
|
+
* criterion - so `wcag1411` is 1.4.11 and not 14.1.1, there being no fourteenth principle.
|
|
31
|
+
*/
|
|
32
|
+
function criterionFrom(tags) {
|
|
33
|
+
for (const tag of tags) {
|
|
34
|
+
const m = /^wcag(\d)(\d)(\d+)$/.exec(tag);
|
|
35
|
+
if (m)
|
|
36
|
+
return `${m[1]}.${m[2]}.${m[3]}`;
|
|
37
|
+
}
|
|
38
|
+
return null;
|
|
39
|
+
}
|
|
40
|
+
/** `wcag2a`, `wcag21aa`, `wcag22aa` -> the conformance level being claimed. */
|
|
41
|
+
function levelFrom(tags) {
|
|
42
|
+
let best = null;
|
|
43
|
+
for (const tag of tags) {
|
|
44
|
+
const m = /^wcag\d{1,2}(a{1,3})$/.exec(tag);
|
|
45
|
+
if (!m)
|
|
46
|
+
continue;
|
|
47
|
+
const level = m[1].toUpperCase();
|
|
48
|
+
// The strictest statement wins, so a rule tagged both A and AA is reported as AA.
|
|
49
|
+
if (best === null || level.length > best.length)
|
|
50
|
+
best = level;
|
|
51
|
+
}
|
|
52
|
+
return best;
|
|
53
|
+
}
|
|
54
|
+
/** `EN-9.1.4.3` -> `9.1.4.3`. */
|
|
55
|
+
function enClauseFrom(tags) {
|
|
56
|
+
for (const tag of tags) {
|
|
57
|
+
const m = /^EN-(\d+(?:\.\d+)+)$/.exec(tag);
|
|
58
|
+
if (m)
|
|
59
|
+
return m[1];
|
|
60
|
+
}
|
|
61
|
+
return null;
|
|
62
|
+
}
|
|
63
|
+
/** axe writes its guidance as "Fix any of the following:\n <reason>". One line reads better. */
|
|
64
|
+
function tidySummary(summary, fallback) {
|
|
65
|
+
const cleaned = summary
|
|
66
|
+
.split(/\r?\n/)
|
|
67
|
+
.map((l) => l.trim())
|
|
68
|
+
.filter((l) => l !== '' && !/^Fix (any|all) of the following:$/.test(l))
|
|
69
|
+
.join('; ');
|
|
70
|
+
return cleaned === '' ? fallback : cleaned;
|
|
71
|
+
}
|
|
72
|
+
const strings = (v) => (Array.isArray(v) ? v.filter((x) => typeof x === 'string') : []);
|
|
73
|
+
const text = (v) => (typeof v === 'string' ? v : '');
|
|
74
|
+
function fromRule(rule, page, kind) {
|
|
75
|
+
const tags = strings(rule.tags);
|
|
76
|
+
const criterion = criterionFrom(tags);
|
|
77
|
+
const help = text(rule.help) || text(rule.description);
|
|
78
|
+
const id = text(rule.id);
|
|
79
|
+
// No criterion and axe's own best-practice tag: somebody's good idea, not a standard. Reported,
|
|
80
|
+
// but never as a barrier and never as something that fails a build.
|
|
81
|
+
const advice = criterion === null && tags.includes('best-practice');
|
|
82
|
+
const confidence = kind === 'incomplete' ? 'ask-a-person' : advice ? 'advice' : 'barrier';
|
|
83
|
+
const nodes = Array.isArray(rule.nodes) ? rule.nodes : [];
|
|
84
|
+
return nodes.map((node) => ({
|
|
85
|
+
rule: id,
|
|
86
|
+
tool: 'axe',
|
|
87
|
+
confidence,
|
|
88
|
+
criterion,
|
|
89
|
+
level: levelFrom(tags),
|
|
90
|
+
en301549: enClauseFrom(tags),
|
|
91
|
+
severity: advice ? 'minor' : (SEVERITY[text(rule.impact)] ?? 'moderate'),
|
|
92
|
+
html: text(node.html).replace(/\s+/g, ' ').slice(0, 160),
|
|
93
|
+
selector: strings(node.target).join(' '),
|
|
94
|
+
page,
|
|
95
|
+
why: kind === 'incomplete'
|
|
96
|
+
? `${help}. axe could not determine this one: ${tidySummary(text(node.failureSummary), 'no reason given')}`
|
|
97
|
+
: help,
|
|
98
|
+
next: tidySummary(text(node.failureSummary), help),
|
|
99
|
+
}));
|
|
100
|
+
}
|
|
101
|
+
const OUTCOMES = ['violations', 'incomplete', 'passes', 'inapplicable'];
|
|
102
|
+
/** One axe result object: what `axe.run()` resolves to. */
|
|
103
|
+
function fromResult(result, page, run) {
|
|
104
|
+
// Recorded before any finding is read: a page on which every rule passed has still been opened,
|
|
105
|
+
// and leaving it out made a clean page look like one the browser never visited.
|
|
106
|
+
run.pages.add(page);
|
|
107
|
+
for (const outcome of OUTCOMES) {
|
|
108
|
+
for (const rule of (Array.isArray(result[outcome]) ? result[outcome] : [])) {
|
|
109
|
+
const id = text(rule.id);
|
|
110
|
+
if (id !== '')
|
|
111
|
+
run.rulesRun.add(id);
|
|
112
|
+
const criterion = criterionFrom(strings(rule.tags));
|
|
113
|
+
if (criterion)
|
|
114
|
+
run.criteriaCovered.add(criterion);
|
|
115
|
+
// Only the two outcomes that need a person's attention become findings. A rule that passed
|
|
116
|
+
// or did not apply is evidence of coverage, not something to report.
|
|
117
|
+
if (outcome === 'violations')
|
|
118
|
+
run.findings.push(...fromRule(rule, page, 'violation'));
|
|
119
|
+
if (outcome === 'incomplete')
|
|
120
|
+
run.findings.push(...fromRule(rule, page, 'incomplete'));
|
|
121
|
+
}
|
|
122
|
+
}
|
|
123
|
+
}
|
|
124
|
+
/** True of anything shaped like the object `axe.run()` resolves to. */
|
|
125
|
+
function looksLikeAxe(value) {
|
|
126
|
+
if (typeof value !== 'object' || value === null)
|
|
127
|
+
return false;
|
|
128
|
+
const v = value;
|
|
129
|
+
return Array.isArray(v.violations) || Array.isArray(v.incomplete) || Array.isArray(v.passes);
|
|
130
|
+
}
|
|
131
|
+
/**
|
|
132
|
+
* Reads an axe report, in any of the shapes real suites write it.
|
|
133
|
+
*
|
|
134
|
+
* One run; an array of runs, which is what a suite checking several pages produces; or an array of
|
|
135
|
+
* `{ url, result }` wrappers, which is what people write when they want to keep the page alongside
|
|
136
|
+
* its findings. Guessing wrong here means reporting a clean page, so an unreadable file throws
|
|
137
|
+
* rather than returning nothing.
|
|
138
|
+
*/
|
|
139
|
+
export function readAxeReport(path) {
|
|
140
|
+
let parsed;
|
|
141
|
+
const raw = readFileSync(path, 'utf8');
|
|
142
|
+
try {
|
|
143
|
+
parsed = JSON.parse(raw);
|
|
144
|
+
}
|
|
145
|
+
catch {
|
|
146
|
+
throw new Error(`${path} is not JSON. Tewip reads the JSON an axe run writes.`);
|
|
147
|
+
}
|
|
148
|
+
const run = { findings: [], pages: new Set(), rulesRun: new Set(), criteriaCovered: new Set() };
|
|
149
|
+
const pageOf = (v, fallback) => text(v.url) || text(v.page) || fallback;
|
|
150
|
+
if (looksLikeAxe(parsed)) {
|
|
151
|
+
fromResult(parsed, pageOf(parsed, path), run);
|
|
152
|
+
return run;
|
|
153
|
+
}
|
|
154
|
+
if (Array.isArray(parsed)) {
|
|
155
|
+
let read = 0;
|
|
156
|
+
for (const entry of parsed) {
|
|
157
|
+
if (looksLikeAxe(entry)) {
|
|
158
|
+
fromResult(entry, pageOf(entry, path), run);
|
|
159
|
+
read++;
|
|
160
|
+
continue;
|
|
161
|
+
}
|
|
162
|
+
// A wrapper: { page | url, result: <axe run> }.
|
|
163
|
+
if (typeof entry !== 'object' || entry === null)
|
|
164
|
+
continue;
|
|
165
|
+
const wrapper = entry;
|
|
166
|
+
// axe records the page it actually ran against in `result.url`, which is authoritative.
|
|
167
|
+
// A wrapper's own label is only a fallback for runs that did not keep it.
|
|
168
|
+
if (looksLikeAxe(wrapper.result)) {
|
|
169
|
+
fromResult(wrapper.result, pageOf(wrapper.result, pageOf(wrapper, path)), run);
|
|
170
|
+
read++;
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
if (read > 0 || parsed.length === 0)
|
|
174
|
+
return run;
|
|
175
|
+
}
|
|
176
|
+
throw new Error(`${path} is JSON, but not an axe report: Tewip looked for a "violations" array, a list of runs, ` +
|
|
177
|
+
'or a list of { page, result } wrappers. Write one with axe.run() in your Playwright, Cypress or jest-axe suite.');
|
|
178
|
+
}
|
|
179
|
+
/**
|
|
180
|
+
* The axe rule that tests the same thing as one of Tewip's source-read rules.
|
|
181
|
+
*
|
|
182
|
+
* Supersession is decided here, rule by rule, rather than by criterion. By criterion it is too
|
|
183
|
+
* coarse and loses real findings: axe reporting `aria-required-children` means it checked *that*
|
|
184
|
+
* part of 1.3.1, not that it looked at heading structure, which axe treats as best-practice and
|
|
185
|
+
* tests under no criterion at all. Nineteen real heading findings disappeared that way before this
|
|
186
|
+
* map existed.
|
|
187
|
+
*
|
|
188
|
+
* `click-without-keyboard` has no entry on purpose: axe has no equivalent rule, so a source finding
|
|
189
|
+
* for it is never superseded by any run.
|
|
190
|
+
*/
|
|
191
|
+
export const EQUIVALENT_AXE_RULE = {
|
|
192
|
+
'image-alt': 'image-alt',
|
|
193
|
+
'input-label': 'label',
|
|
194
|
+
'button-name': 'button-name',
|
|
195
|
+
'link-name': 'link-name',
|
|
196
|
+
'html-lang': 'html-has-lang',
|
|
197
|
+
'iframe-title': 'frame-title',
|
|
198
|
+
'zoom-blocked': 'meta-viewport',
|
|
199
|
+
'nested-interactive': 'nested-interactive',
|
|
200
|
+
'aria-hidden-focusable': 'aria-hidden-focus',
|
|
201
|
+
'duplicate-id': 'duplicate-id',
|
|
202
|
+
'aria-unknown': 'aria-valid-attr',
|
|
203
|
+
'positive-tabindex': 'tabindex',
|
|
204
|
+
'autoplay-sound': 'no-autoplay-audio',
|
|
205
|
+
'heading-skip': 'heading-order',
|
|
206
|
+
};
|
|
@@ -106,7 +106,17 @@ export const RULES = {
|
|
|
106
106
|
* every time is what stops "Tewip found no barriers" from being heard as "this is accessible".
|
|
107
107
|
*/
|
|
108
108
|
export const NOT_CHECKED = [
|
|
109
|
-
{
|
|
109
|
+
{
|
|
110
|
+
criterion: '1.4.3 Contrast (Minimum)',
|
|
111
|
+
why: 'needs the computed colours of a rendered page, not the markup',
|
|
112
|
+
// The one entry on this list a browser run genuinely closes (Phase 29).
|
|
113
|
+
settledBy: 'color-contrast',
|
|
114
|
+
},
|
|
115
|
+
// Everything below has no `settledBy` on purpose, and must not acquire one because a tool
|
|
116
|
+
// reports something under the same criterion number. axe's `image-alt` checks that an alt
|
|
117
|
+
// attribute exists; it has no opinion on whether the words describe the picture, which is the
|
|
118
|
+
// part named here. Matching on the number alone briefly made Tewip claim it had checked alt-text
|
|
119
|
+
// quality and timing behaviour, which no tool can.
|
|
110
120
|
{ criterion: '1.1.1 Non-text Content, quality', why: 'whether alt text actually describes the picture is a human judgement' },
|
|
111
121
|
{ criterion: '2.4.7 Focus Visible', why: 'needs a rendered page to see whether the focus ring survives the stylesheet' },
|
|
112
122
|
{ criterion: '1.3.2 Meaningful Sequence', why: 'needs the rendered order, which CSS can change from the source order' },
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
// Phase 30: deciding whether a run is actually slower.
|
|
2
|
+
//
|
|
3
|
+
// This is the whole phase. Reporting "p95 went from 180ms to 210ms, +16%" is trivial and worthless:
|
|
4
|
+
// nobody knows whether 16% is the change or the laptop. Performance numbers move between identical
|
|
5
|
+
// runs - a background process, a warm cache, a noisy CI runner - and a tool that calls every
|
|
6
|
+
// movement a regression gets muted within a week, which is the same failure mode as the
|
|
7
|
+
// accessibility scanners in Phase 28.
|
|
8
|
+
//
|
|
9
|
+
// So Tewip refuses unless it can tell the difference, and it can only tell the difference from
|
|
10
|
+
// **repeated baseline runs**. One run against one run says nothing, however large the gap looks,
|
|
11
|
+
// and Tewip says that rather than guessing. With three or more baselines there is an observed
|
|
12
|
+
// spread to compare against, and a change outside the baseline's own range by more than the width
|
|
13
|
+
// of that range is a change the runs themselves cannot explain.
|
|
14
|
+
//
|
|
15
|
+
// A k6 summary carries no raw samples - only avg, min, med, p90, p95, max - so a proper confidence
|
|
16
|
+
// interval is not available, and nothing here pretends otherwise. The test is deliberately crude
|
|
17
|
+
// and deliberately conservative: it is better to say "cannot say" to a real 4% regression than to
|
|
18
|
+
// cry wolf at noise, because the second one costs the team the tool.
|
|
19
|
+
import { MAIN_TIMING } from './k6.js';
|
|
20
|
+
/** The fewest baseline runs from which run-to-run noise can be estimated at all. */
|
|
21
|
+
export const ENOUGH_BASELINES = 3;
|
|
22
|
+
/**
|
|
23
|
+
* A change smaller than this is not called even when the baselines are unusually tight, because a
|
|
24
|
+
* handful of identical runs can make the observed spread implausibly small and a 1% move is not
|
|
25
|
+
* worth a person's afternoon.
|
|
26
|
+
*/
|
|
27
|
+
const SMALLEST_CHANGE = 0.05;
|
|
28
|
+
const median = (values) => {
|
|
29
|
+
const sorted = [...values].sort((a, b) => a - b);
|
|
30
|
+
const mid = Math.floor(sorted.length / 2);
|
|
31
|
+
return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid];
|
|
32
|
+
};
|
|
33
|
+
const round = (n) => Math.round(n * 10) / 10;
|
|
34
|
+
/**
|
|
35
|
+
* Compares one metric of a candidate run against the same metric in the baseline runs.
|
|
36
|
+
*
|
|
37
|
+
* `baseline` is every stored value for the metric, oldest first. It is not averaged into a single
|
|
38
|
+
* number, because its spread is the only evidence available about what noise looks like here.
|
|
39
|
+
*/
|
|
40
|
+
export function compareMetric(metric, now, baseline) {
|
|
41
|
+
const runs = baseline.length;
|
|
42
|
+
const before = runs > 0 ? median(baseline) : 0;
|
|
43
|
+
const low = runs > 0 ? Math.min(...baseline) : 0;
|
|
44
|
+
const high = runs > 0 ? Math.max(...baseline) : 0;
|
|
45
|
+
const spread = high - low;
|
|
46
|
+
const margin = Math.max(spread, before * SMALLEST_CHANGE);
|
|
47
|
+
const base = {
|
|
48
|
+
metric,
|
|
49
|
+
now: round(now),
|
|
50
|
+
before: round(before),
|
|
51
|
+
baselineRuns: runs,
|
|
52
|
+
baselineLow: round(low),
|
|
53
|
+
baselineHigh: round(high),
|
|
54
|
+
margin: round(margin),
|
|
55
|
+
};
|
|
56
|
+
if (runs < ENOUGH_BASELINES) {
|
|
57
|
+
return {
|
|
58
|
+
...base,
|
|
59
|
+
direction: 'cannot-say',
|
|
60
|
+
why: runs === 0
|
|
61
|
+
? 'there is nothing to compare this against yet'
|
|
62
|
+
: `${runs} baseline ${runs === 1 ? 'run is' : 'runs are'} not enough to know what noise looks like here; ` +
|
|
63
|
+
`Tewip needs ${ENOUGH_BASELINES}. Two runs can differ by this much for reasons that have nothing to do with the code.`,
|
|
64
|
+
};
|
|
65
|
+
}
|
|
66
|
+
const percent = before > 0 ? Math.round(((now - before) / before) * 100) : 0;
|
|
67
|
+
if (now > high + margin) {
|
|
68
|
+
return {
|
|
69
|
+
...base,
|
|
70
|
+
direction: 'slower',
|
|
71
|
+
why: `${round(now)}ms is outside everything ${runs} baseline runs produced (${round(low)}-${round(high)}ms) ` +
|
|
72
|
+
`by more than their own spread, so the runs cannot explain it. That is ${percent > 0 ? '+' : ''}${percent}% on the middle run.`,
|
|
73
|
+
};
|
|
74
|
+
}
|
|
75
|
+
if (now < low - margin) {
|
|
76
|
+
return {
|
|
77
|
+
...base,
|
|
78
|
+
direction: 'faster',
|
|
79
|
+
why: `${round(now)}ms is below everything ${runs} baseline runs produced (${round(low)}-${round(high)}ms) by more than their own spread.`,
|
|
80
|
+
};
|
|
81
|
+
}
|
|
82
|
+
return {
|
|
83
|
+
...base,
|
|
84
|
+
direction: 'cannot-say',
|
|
85
|
+
why: `${round(now)}ms is within what ${runs} baseline runs already did on their own (${round(low)}-${round(high)}ms, ` +
|
|
86
|
+
`allowing ${round(margin)}ms either side). A difference the baseline runs produce by themselves is not evidence about this change.`,
|
|
87
|
+
};
|
|
88
|
+
}
|
|
89
|
+
/** Enough requests for a percentile to mean anything. Below this, p95 is three data points. */
|
|
90
|
+
export const ENOUGH_REQUESTS = 100;
|
|
91
|
+
/** Timings worth comparing, in the order a person would want them. */
|
|
92
|
+
export function timingsToCompare(run) {
|
|
93
|
+
const named = Object.keys(run.timings).filter((m) => !m.includes('{'));
|
|
94
|
+
const first = named.filter((m) => m === MAIN_TIMING);
|
|
95
|
+
const rest = named.filter((m) => m !== MAIN_TIMING && /duration|waiting|blocked/.test(m)).sort();
|
|
96
|
+
return [...first, ...rest];
|
|
97
|
+
}
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
// Phase 30: the stored baseline.
|
|
2
|
+
//
|
|
3
|
+
// Recording is deliberately **not** automatic, and that is the most important decision in this
|
|
4
|
+
// file. A tool that files every run it sees into the baseline ratchets: today's regression becomes
|
|
5
|
+
// tomorrow's normal, the comparison keeps saying "within range", and a service gets three times
|
|
6
|
+
// slower over a quarter without one report ever being wrong. Tewip records only when asked, so the
|
|
7
|
+
// intended use is `--record` on the main branch and a plain comparison everywhere else.
|
|
8
|
+
//
|
|
9
|
+
// The file is JSON Lines so that a run can be appended without reading or rewriting what is already
|
|
10
|
+
// there, and so a person can read it or delete a bad line with an editor.
|
|
11
|
+
import { appendFileSync, existsSync, mkdirSync, readFileSync } from 'node:fs';
|
|
12
|
+
import { dirname, join } from 'node:path';
|
|
13
|
+
const file = (root) => join(root, '.tewip', 'perf', 'runs.jsonl');
|
|
14
|
+
/** Everything recorded for this label, oldest first. */
|
|
15
|
+
export function storedRuns(root, label) {
|
|
16
|
+
const path = file(root);
|
|
17
|
+
if (!existsSync(path))
|
|
18
|
+
return [];
|
|
19
|
+
const out = [];
|
|
20
|
+
for (const line of readFileSync(path, 'utf8').split('\n')) {
|
|
21
|
+
const trimmed = line.trim();
|
|
22
|
+
if (trimmed === '')
|
|
23
|
+
continue;
|
|
24
|
+
try {
|
|
25
|
+
const parsed = JSON.parse(trimmed);
|
|
26
|
+
// A line Tewip cannot read is skipped rather than throwing: one bad line written by a crashed
|
|
27
|
+
// run must not make the whole baseline unreadable.
|
|
28
|
+
if (parsed && typeof parsed.at === 'string' && parsed.label === label && parsed.timings)
|
|
29
|
+
out.push(parsed);
|
|
30
|
+
}
|
|
31
|
+
catch {
|
|
32
|
+
continue;
|
|
33
|
+
}
|
|
34
|
+
}
|
|
35
|
+
return out;
|
|
36
|
+
}
|
|
37
|
+
export function recordRun(root, label, run, requests, gitSha) {
|
|
38
|
+
const stored = {
|
|
39
|
+
at: run.at,
|
|
40
|
+
label,
|
|
41
|
+
...(gitSha ? { gitSha } : {}),
|
|
42
|
+
timings: Object.fromEntries(Object.entries(run.timings).map(([m, v]) => [m, { p95: v.p95, med: v.med, avg: v.avg }])),
|
|
43
|
+
requests,
|
|
44
|
+
};
|
|
45
|
+
const path = file(root);
|
|
46
|
+
mkdirSync(dirname(path), { recursive: true });
|
|
47
|
+
appendFileSync(path, `${JSON.stringify(stored)}\n`, 'utf8');
|
|
48
|
+
return stored;
|
|
49
|
+
}
|
|
50
|
+
/** Every stored p95 for one metric, oldest first, for the comparison to read the spread from. */
|
|
51
|
+
export function baselineFor(runs, metric) {
|
|
52
|
+
return runs.map((r) => r.timings[metric]?.p95).filter((v) => typeof v === 'number' && Number.isFinite(v));
|
|
53
|
+
}
|
|
54
|
+
/** The labels this project has recorded, so a typo in --label does not look like an empty history. */
|
|
55
|
+
export function knownLabels(root) {
|
|
56
|
+
const path = file(root);
|
|
57
|
+
if (!existsSync(path))
|
|
58
|
+
return [];
|
|
59
|
+
const labels = new Set();
|
|
60
|
+
for (const line of readFileSync(path, 'utf8').split('\n')) {
|
|
61
|
+
const trimmed = line.trim();
|
|
62
|
+
if (trimmed === '')
|
|
63
|
+
continue;
|
|
64
|
+
try {
|
|
65
|
+
const parsed = JSON.parse(trimmed);
|
|
66
|
+
if (typeof parsed?.label === 'string')
|
|
67
|
+
labels.add(parsed.label);
|
|
68
|
+
}
|
|
69
|
+
catch {
|
|
70
|
+
continue;
|
|
71
|
+
}
|
|
72
|
+
}
|
|
73
|
+
return [...labels].sort();
|
|
74
|
+
}
|
|
@@ -0,0 +1,111 @@
|
|
|
1
|
+
// Phase 30: reading a k6 run.
|
|
2
|
+
//
|
|
3
|
+
// k6 writes two different summary shapes and Tewip reads both, because which one a team has depends
|
|
4
|
+
// on a flag they probably did not choose deliberately:
|
|
5
|
+
//
|
|
6
|
+
// --summary-export the classic shape: metrics as a map, percentiles keyed
|
|
7
|
+
// "p(95)", and - the part that matters - the result of every
|
|
8
|
+
// threshold the team wrote.
|
|
9
|
+
// --new-machine-readable-summary metrics as an array, percentiles keyed p95, and **no
|
|
10
|
+
// threshold results at all**.
|
|
11
|
+
//
|
|
12
|
+
// Both facts below were established by running k6 v2.3.0 against a real server rather than read
|
|
13
|
+
// from documentation, and both would have been got wrong from memory:
|
|
14
|
+
//
|
|
15
|
+
// 1. In the classic shape a threshold entry is `{"p(95)<120": true}` where **true means breached**.
|
|
16
|
+
// It reads like "the condition holds". It means the opposite. A run whose p95 was 282ms against
|
|
17
|
+
// a 120ms threshold is the one that produced `true`.
|
|
18
|
+
// 2. The machine-readable shape drops thresholds entirely. The word appears in the file only inside
|
|
19
|
+
// `config.script`, which is the source text of the test. So a team on that flag has no pass or
|
|
20
|
+
// fail recorded anywhere in their export, and Tewip has to say so rather than report "no
|
|
21
|
+
// threshold was breached", which would be a lie of omission.
|
|
22
|
+
import { readFileSync } from 'node:fs';
|
|
23
|
+
const num = (v) => (typeof v === 'number' && Number.isFinite(v) ? v : 0);
|
|
24
|
+
function measurement(metric, values) {
|
|
25
|
+
return {
|
|
26
|
+
metric,
|
|
27
|
+
avg: num(values.avg),
|
|
28
|
+
min: num(values.min),
|
|
29
|
+
med: num(values.med),
|
|
30
|
+
// The two shapes disagree on how a percentile is spelled.
|
|
31
|
+
p90: num(values['p(90)'] ?? values.p90),
|
|
32
|
+
p95: num(values['p(95)'] ?? values.p95),
|
|
33
|
+
max: num(values.max),
|
|
34
|
+
};
|
|
35
|
+
}
|
|
36
|
+
/** A trend metric is one with a median: a timing, rather than a counter or a rate. */
|
|
37
|
+
const isTrend = (v) => 'med' in v || 'p(95)' in v || 'p95' in v;
|
|
38
|
+
function classic(parsed, at) {
|
|
39
|
+
const metrics = (parsed.metrics ?? {});
|
|
40
|
+
const run = { tool: 'k6', format: 'classic', at, timings: {}, rates: {}, counts: {}, thresholds: [] };
|
|
41
|
+
for (const [name, values] of Object.entries(metrics)) {
|
|
42
|
+
if (typeof values !== 'object' || values === null)
|
|
43
|
+
continue;
|
|
44
|
+
for (const [expression, breached] of Object.entries((values.thresholds ?? {}))) {
|
|
45
|
+
// `true` is a breach. See the note at the top: this is the opposite of how it reads.
|
|
46
|
+
run.thresholds.push({ metric: name, expression, breached: breached === true });
|
|
47
|
+
}
|
|
48
|
+
if (isTrend(values))
|
|
49
|
+
run.timings[name] = measurement(name, values);
|
|
50
|
+
else if ('rate' in values || ('passes' in values && 'fails' in values)) {
|
|
51
|
+
const passes = num(values.passes);
|
|
52
|
+
const fails = num(values.fails);
|
|
53
|
+
run.rates[name] = { rate: num(values.rate ?? values.value), total: passes + fails };
|
|
54
|
+
}
|
|
55
|
+
else if ('count' in values)
|
|
56
|
+
run.counts[name] = num(values.count);
|
|
57
|
+
}
|
|
58
|
+
return run;
|
|
59
|
+
}
|
|
60
|
+
function machineReadable(parsed, at) {
|
|
61
|
+
const results = (parsed.results ?? {});
|
|
62
|
+
const metrics = Array.isArray(results.metrics) ? results.metrics : [];
|
|
63
|
+
// Not an empty list: this export records no threshold results, which is a different statement.
|
|
64
|
+
const run = { tool: 'k6', format: 'machine-readable', at, timings: {}, rates: {}, counts: {}, thresholds: null };
|
|
65
|
+
for (const entry of metrics) {
|
|
66
|
+
const name = typeof entry.name === 'string' ? entry.name : '';
|
|
67
|
+
const values = (entry.values ?? {});
|
|
68
|
+
if (name === '' || typeof values !== 'object' || values === null)
|
|
69
|
+
continue;
|
|
70
|
+
if (entry.type === 'trend')
|
|
71
|
+
run.timings[name] = measurement(name, values);
|
|
72
|
+
else if (entry.type === 'rate')
|
|
73
|
+
run.rates[name] = { rate: num(values.rate), total: num(values.total) };
|
|
74
|
+
else if (entry.type === 'counter' || 'count' in values)
|
|
75
|
+
run.counts[name] = num(values.count);
|
|
76
|
+
}
|
|
77
|
+
return run;
|
|
78
|
+
}
|
|
79
|
+
/** True of either shape k6 writes. */
|
|
80
|
+
function looksLikeK6Summary(v) {
|
|
81
|
+
if (typeof v !== 'object' || v === null)
|
|
82
|
+
return false;
|
|
83
|
+
const o = v;
|
|
84
|
+
if (typeof o.metrics === 'object' && o.metrics !== null && !Array.isArray(o.metrics))
|
|
85
|
+
return true;
|
|
86
|
+
const results = o.results;
|
|
87
|
+
return typeof results === 'object' && results !== null && Array.isArray(results.metrics);
|
|
88
|
+
}
|
|
89
|
+
export function readK6Summary(path) {
|
|
90
|
+
let parsed;
|
|
91
|
+
try {
|
|
92
|
+
parsed = JSON.parse(readFileSync(path, 'utf8'));
|
|
93
|
+
}
|
|
94
|
+
catch {
|
|
95
|
+
throw new Error(`${path} is not JSON. Tewip reads the JSON k6 writes with --summary-export.`);
|
|
96
|
+
}
|
|
97
|
+
if (!looksLikeK6Summary(parsed)) {
|
|
98
|
+
throw new Error(`${path} is JSON, but not a k6 summary: Tewip looked for a "metrics" object or a "results.metrics" array. ` +
|
|
99
|
+
'Write one with: k6 run --summary-export=summary.json your-test.js');
|
|
100
|
+
}
|
|
101
|
+
const o = parsed;
|
|
102
|
+
const metadata = (o.metadata ?? {});
|
|
103
|
+
const at = typeof metadata.generated_at === 'string' ? metadata.generated_at : new Date().toISOString();
|
|
104
|
+
return Array.isArray(o.results?.metrics) ? machineReadable(o, at) : classic(o, at);
|
|
105
|
+
}
|
|
106
|
+
/** The metric a team means by "how slow is it", when they have not said which. */
|
|
107
|
+
export const MAIN_TIMING = 'http_req_duration';
|
|
108
|
+
/** How much work the run did. A verdict from a handful of requests is not worth printing. */
|
|
109
|
+
export function requestCount(run) {
|
|
110
|
+
return run.counts.http_reqs ?? run.rates.http_req_failed?.total ?? run.counts.iterations ?? 0;
|
|
111
|
+
}
|
|
@@ -0,0 +1,120 @@
|
|
|
1
|
+
// Phase 30: `tewip perf` - is this change slower, and does it matter?
|
|
2
|
+
//
|
|
3
|
+
// Two questions, answered separately because they fail separately:
|
|
4
|
+
//
|
|
5
|
+
// 1. **Did your own thresholds hold?** The team already wrote what "fast enough" means, in the k6
|
|
6
|
+
// script. Tewip reads the result rather than asking them to write a budget twice. When the
|
|
7
|
+
// export does not record thresholds - which the machine-readable one does not - the answer is
|
|
8
|
+
// "this export cannot tell me", never "nothing was breached".
|
|
9
|
+
// 2. **Is it slower than it was?** Answered only from repeated baseline runs, and refused when
|
|
10
|
+
// there are not enough of them, because one run against one run is not evidence.
|
|
11
|
+
//
|
|
12
|
+
// It never estimates a duration, a capacity or a user impact. It has no idea what the production
|
|
13
|
+
// hardware is, what the data volume is, or how many people use the endpoint, and a number invented
|
|
14
|
+
// for any of those is worse than silence.
|
|
15
|
+
import { mkdirSync, writeFileSync } from 'node:fs';
|
|
16
|
+
import { dirname, resolve } from 'node:path';
|
|
17
|
+
import { MAIN_TIMING, readK6Summary, requestCount } from './k6.js';
|
|
18
|
+
import { compareMetric, ENOUGH_BASELINES, ENOUGH_REQUESTS, timingsToCompare } from './compare.js';
|
|
19
|
+
import { baselineFor, knownLabels, recordRun, storedRuns } from './history.js';
|
|
20
|
+
const ms = (n) => `${Math.round(n * 10) / 10}ms`;
|
|
21
|
+
export function renderPerf(run, comparisons, requests, label, baselineCount, recorded) {
|
|
22
|
+
const lines = ['# Tewip performance check', ''];
|
|
23
|
+
const breaches = (run.thresholds ?? []).filter((t) => t.breached);
|
|
24
|
+
const slower = comparisons.filter((c) => c.direction === 'slower');
|
|
25
|
+
const faster = comparisons.filter((c) => c.direction === 'faster');
|
|
26
|
+
const unsure = comparisons.filter((c) => c.direction === 'cannot-say');
|
|
27
|
+
lines.push(`Read a k6 run of **${requests} requests**, compared against **${baselineCount}** recorded baseline ${baselineCount === 1 ? 'run' : 'runs'} for \`${label}\`.`, '');
|
|
28
|
+
// Headline, in the order that decides what a person does next.
|
|
29
|
+
if (breaches.length > 0) {
|
|
30
|
+
lines.push(`**${breaches.length} of your own thresholds ${breaches.length === 1 ? 'was' : 'were'} breached.**`, '');
|
|
31
|
+
}
|
|
32
|
+
if (slower.length > 0)
|
|
33
|
+
lines.push(`**${slower.length} ${slower.length === 1 ? 'metric is' : 'metrics are'} slower than the baseline runs can explain.**`, '');
|
|
34
|
+
if (breaches.length === 0 && slower.length === 0) {
|
|
35
|
+
lines.push(run.thresholds === null
|
|
36
|
+
? '**No regression this baseline can detect.** Your thresholds are not in this export, so nothing is said about them.'
|
|
37
|
+
: '**No threshold breached, and no regression this baseline can detect.**', '');
|
|
38
|
+
}
|
|
39
|
+
lines.push('## Did your own thresholds hold?', '');
|
|
40
|
+
if (run.thresholds === null) {
|
|
41
|
+
lines.push('**Cannot say.** This summary came from `--new-machine-readable-summary`, and that export records no threshold ' +
|
|
42
|
+
'results at all - the word appears in it only inside the copy of your script. Tewip will not report "nothing was ' +
|
|
43
|
+
'breached" from a file that does not say. Re-run with plain `--summary-export` to get the answer.', '');
|
|
44
|
+
}
|
|
45
|
+
else if (run.thresholds.length === 0) {
|
|
46
|
+
lines.push('This run has no thresholds. Add them to your k6 script and the team decides what "fast enough" means, rather than Tewip.', '');
|
|
47
|
+
}
|
|
48
|
+
else {
|
|
49
|
+
for (const t of run.thresholds) {
|
|
50
|
+
lines.push(`- ${t.breached ? '**breached**' : 'held'} — \`${t.metric}\`: \`${t.expression}\``);
|
|
51
|
+
}
|
|
52
|
+
lines.push('');
|
|
53
|
+
}
|
|
54
|
+
lines.push('## Is it slower than it was?', '');
|
|
55
|
+
if (baselineCount < ENOUGH_BASELINES) {
|
|
56
|
+
lines.push(`**Cannot say.** There ${baselineCount === 1 ? 'is' : 'are'} ${baselineCount} recorded ${baselineCount === 1 ? 'run' : 'runs'} for \`${label}\`, ` +
|
|
57
|
+
`and Tewip needs ${ENOUGH_BASELINES} before it will call anything a regression. Performance numbers move between identical ` +
|
|
58
|
+
'runs, and without several of them there is no way to tell a change from the machine having a busy moment. ' +
|
|
59
|
+
`Record baselines on your main branch with \`tewip perf <summary.json> --record\`.`, '');
|
|
60
|
+
}
|
|
61
|
+
for (const c of [...slower, ...faster, ...unsure]) {
|
|
62
|
+
const verdict = c.direction === 'slower' ? '**slower**' : c.direction === 'faster' ? 'faster' : 'cannot say';
|
|
63
|
+
lines.push(`- \`${c.metric}\` p95 ${ms(c.now)} — ${verdict}`);
|
|
64
|
+
lines.push(` - baseline ${ms(c.baselineLow)}–${ms(c.baselineHigh)} over ${c.baselineRuns} runs, middle ${ms(c.before)}`);
|
|
65
|
+
lines.push(` - ${c.why}`);
|
|
66
|
+
}
|
|
67
|
+
lines.push('');
|
|
68
|
+
if (requests < ENOUGH_REQUESTS) {
|
|
69
|
+
lines.push(`**${requests} requests is thin.** A p95 from this few is a handful of data points and will move on its own. ` +
|
|
70
|
+
'Treat everything above as indicative and lengthen the run before deciding anything.', '');
|
|
71
|
+
}
|
|
72
|
+
lines.push('## What this does not tell you', '');
|
|
73
|
+
lines.push('- **Nothing about production.** This is the hardware the run happened on, with that data volume and that ' +
|
|
74
|
+
'concurrency. Tewip cannot see your real machines, your real data or your real traffic, and will not guess.', '- **No duration is predicted.** How long a real user waits depends on network, device and cache, none of which are here.', '- **Only what the script exercised.** An endpoint the test never called is not covered, however green this looks.', '- **Not capacity.** This says whether it changed, not how many users it would hold.', '');
|
|
75
|
+
if (recorded)
|
|
76
|
+
lines.push(`This run was added to the baseline for \`${label}\`.`, '');
|
|
77
|
+
else
|
|
78
|
+
lines.push(`This run was **not** recorded. Add \`--record\` on your main branch to make it part of the baseline; a regression recorded by accident becomes next week's normal.`, '');
|
|
79
|
+
return lines.join('\n');
|
|
80
|
+
}
|
|
81
|
+
export function runPerf(cmd) {
|
|
82
|
+
const root = resolve(cmd.root);
|
|
83
|
+
const label = cmd.label ?? 'default';
|
|
84
|
+
const run = readK6Summary(resolve(root, cmd.summary));
|
|
85
|
+
const requests = requestCount(run);
|
|
86
|
+
const history = storedRuns(root, label);
|
|
87
|
+
if (history.length === 0) {
|
|
88
|
+
const known = knownLabels(root);
|
|
89
|
+
// A typo in --label would otherwise look exactly like "no history yet", and the team would
|
|
90
|
+
// record a second baseline under the wrong name without noticing.
|
|
91
|
+
if (known.length > 0 && !known.includes(label)) {
|
|
92
|
+
console.error(`[tewip] no recorded runs for label "${label}". This project has: ${known.join(', ')}.`);
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
const comparisons = timingsToCompare(run)
|
|
96
|
+
.map((metric) => compareMetric(metric, run.timings[metric].p95, baselineFor(history, metric)))
|
|
97
|
+
// A metric that is zero now and was zero every time before says nothing. http_req_blocked is
|
|
98
|
+
// zero on a local run, and printing "0ms is within 0-0ms" three times buries the real lines.
|
|
99
|
+
.filter((c) => c.now > 0 || c.baselineHigh > 0);
|
|
100
|
+
const recorded = cmd.record === true;
|
|
101
|
+
if (recorded)
|
|
102
|
+
recordRun(root, label, run, requests, cmd.gitSha);
|
|
103
|
+
const breached = (run.thresholds ?? []).some((t) => t.breached);
|
|
104
|
+
const regressed = comparisons.some((c) => c.direction === 'slower');
|
|
105
|
+
const report = renderPerf(run, comparisons, requests, label, history.length, recorded);
|
|
106
|
+
console.log(report);
|
|
107
|
+
if (cmd.gate && (breached || regressed)) {
|
|
108
|
+
const why = [breached ? 'a threshold you wrote was breached' : '', regressed ? 'something is slower than the baselines explain' : '']
|
|
109
|
+
.filter(Boolean)
|
|
110
|
+
.join(', and ');
|
|
111
|
+
console.error(`${String.fromCharCode(10)}Performance gate failed: ${why}.`);
|
|
112
|
+
process.exitCode = 1;
|
|
113
|
+
}
|
|
114
|
+
if (cmd.out) {
|
|
115
|
+
const path = resolve(root, cmd.out);
|
|
116
|
+
mkdirSync(dirname(path), { recursive: true });
|
|
117
|
+
writeFileSync(path, report, 'utf8');
|
|
118
|
+
}
|
|
119
|
+
return { run, comparisons, requests, regressed, breached, report };
|
|
120
|
+
}
|
package/dist/mcp/tools-prove.js
CHANGED
|
@@ -95,27 +95,75 @@ export function registerProofTools(server, { answer, readOnly, project }) {
|
|
|
95
95
|
return 'No migration or schema file in that change.';
|
|
96
96
|
return renderSchema(migrations, stillUsed(root, migrations.flatMap((m) => m.findings)));
|
|
97
97
|
}));
|
|
98
|
+
server.registerTool('is_this_slower', {
|
|
99
|
+
title: 'Is this change slower, and does it matter?',
|
|
100
|
+
description: "Judges a k6 load-test run, rather than restating its numbers. Two answers kept apart: whether the thresholds the team wrote in their own k6 script held, and whether this run is slower than the recorded baseline. The second is answered only from three or more baseline runs - below that it returns 'cannot say', because one run against one run cannot distinguish a regression from a busy machine, and a percentage printed without that distinction is how performance tooling gets ignored. It never predicts a duration, a capacity or a user impact, and it says what the run does not cover. Use it when asked whether a change made something slower.",
|
|
101
|
+
inputSchema: {
|
|
102
|
+
project,
|
|
103
|
+
summary: z.string().describe('Path to the JSON from `k6 run --summary-export=summary.json`'),
|
|
104
|
+
label: z.string().optional().describe('Which load test this is, so two different suites do not become each other baseline'),
|
|
105
|
+
},
|
|
106
|
+
annotations: readOnly,
|
|
107
|
+
}, ({ project: root, summary, label }) => answer(async () => {
|
|
108
|
+
if (!isAbsolute(root) || !existsSync(root))
|
|
109
|
+
throw new Error(`project must be an existing absolute path, got "${root}"`);
|
|
110
|
+
// Never records: an assistant adding runs to a team's baseline unasked is how a regression
|
|
111
|
+
// silently becomes the new normal. Recording stays a deliberate act on the command line.
|
|
112
|
+
const { readK6Summary, requestCount } = await import('../engine/perf/k6.js');
|
|
113
|
+
const { compareMetric, timingsToCompare } = await import('../engine/perf/compare.js');
|
|
114
|
+
const { baselineFor, storedRuns } = await import('../engine/perf/history.js');
|
|
115
|
+
const { renderPerf } = await import('../engine/perf/perf-command.js');
|
|
116
|
+
const { isAbsolute: abs, resolve } = await import('node:path');
|
|
117
|
+
const run = readK6Summary(abs(summary) ? summary : resolve(root, summary));
|
|
118
|
+
const name = label ?? 'default';
|
|
119
|
+
const history = storedRuns(root, name);
|
|
120
|
+
const comparisons = timingsToCompare(run)
|
|
121
|
+
.map((m) => compareMetric(m, run.timings[m].p95, baselineFor(history, m)))
|
|
122
|
+
.filter((c) => c.now > 0 || c.baselineHigh > 0);
|
|
123
|
+
return renderPerf(run, comparisons, requestCount(run), name, history.length, false);
|
|
124
|
+
}));
|
|
98
125
|
server.registerTool('accessibility_barriers', {
|
|
99
126
|
title: 'Can everybody use this interface?',
|
|
100
127
|
description: "Accessibility barriers in a project's markup, each named against the WCAG 2.2 criterion a regulator would cite, with the evidence to check it. Two kinds of answer, kept apart on purpose: barriers that reading the markup proves (no alt attribute at all, a field with no label anywhere, a div that only a mouse can activate), and the ones only a person can settle, where something is written as an expression or a component and the markup cannot say. It also returns every criterion it did NOT check - contrast, focus order, whether alt text is any good - because those need a rendered page. Use it before shipping an interface, or when asked whether something is accessible; a clean result means these checks found nothing, never that the product is accessible. No browser, no network, no model.",
|
|
101
128
|
inputSchema: {
|
|
102
129
|
project,
|
|
103
130
|
path: z.string().optional().describe('A file or directory inside the project to check instead of all of it'),
|
|
131
|
+
axeReport: z
|
|
132
|
+
.string()
|
|
133
|
+
.optional()
|
|
134
|
+
.describe('The JSON from an axe run, if the project has one. It settles contrast, which reading markup cannot, and its findings are triaged rather than relayed: axe rules that map to no WCAG criterion are reported as advice, and what axe could not determine becomes a question.'),
|
|
104
135
|
},
|
|
105
136
|
annotations: readOnly,
|
|
106
|
-
}, ({ project: root, path }) => answer(async () => {
|
|
137
|
+
}, ({ project: root, path, axeReport }) => answer(async () => {
|
|
107
138
|
if (!isAbsolute(root) || !existsSync(root))
|
|
108
139
|
throw new Error(`project must be an existing absolute path, got "${root}"`);
|
|
109
140
|
// The pure functions, not runAccess: that one prints, and anything on stdout breaks JSON-RPC.
|
|
110
141
|
const { findBarriers, keyOf, readAccepted, renderAccess } = await import('../engine/access/access-command.js');
|
|
142
|
+
const { merge, pagesIn } = await import('../engine/access/merge.js');
|
|
143
|
+
const { readAxeReport } = await import('../engine/access/reports.js');
|
|
111
144
|
const { findings, filesRead } = findBarriers(root, path);
|
|
112
145
|
const judged = readAccepted(root);
|
|
113
146
|
const open = findings.filter((f) => !judged.has(keyOf(f)));
|
|
147
|
+
const run = axeReport ? readAxeReport(isAbsolute(axeReport) ? axeReport : `${root}/${axeReport}`) : undefined;
|
|
148
|
+
const graded = run ? merge(open, run, root) : undefined;
|
|
149
|
+
const kept = graded ? graded.kept : open;
|
|
114
150
|
return renderAccess({
|
|
115
|
-
barriers:
|
|
116
|
-
questions:
|
|
151
|
+
barriers: kept.filter((f) => f.confidence === 'barrier'),
|
|
152
|
+
questions: kept.filter((f) => f.confidence === 'ask-a-person'),
|
|
117
153
|
accepted: findings.filter((f) => judged.has(keyOf(f))),
|
|
118
154
|
filesRead,
|
|
155
|
+
...(run && graded
|
|
156
|
+
? {
|
|
157
|
+
browser: {
|
|
158
|
+
barriers: run.findings.filter((f) => f.confidence === 'barrier'),
|
|
159
|
+
questions: run.findings.filter((f) => f.confidence === 'ask-a-person'),
|
|
160
|
+
advice: run.findings.filter((f) => f.confidence === 'advice'),
|
|
161
|
+
superseded: graded.superseded,
|
|
162
|
+
pages: pagesIn(run),
|
|
163
|
+
rulesRun: run.rulesRun,
|
|
164
|
+
},
|
|
165
|
+
}
|
|
166
|
+
: {}),
|
|
119
167
|
}, path ?? root);
|
|
120
168
|
}));
|
|
121
169
|
server.registerTool('project_standards', {
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tewip-ai",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.37.0",
|
|
4
4
|
"mcpName": "io.github.hmamut39/tewip",
|
|
5
5
|
"description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
|
|
6
6
|
"keywords": [
|