tewip-ai 0.37.0 → 0.39.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +51 -0
- package/dist/cli/help.js +20 -16
- package/dist/engine/access/access-command.js +27 -126
- package/dist/engine/access/axe-rules.js +128 -0
- package/dist/engine/access/from-lighthouse.js +74 -0
- package/dist/engine/access/render.js +133 -0
- package/dist/engine/access/reports.js +1 -1
- package/dist/engine/perf/history.js +21 -6
- package/dist/engine/perf/lighthouse.js +155 -0
- package/dist/engine/perf/perf-command.js +102 -57
- package/dist/engine/perf/run.js +39 -0
- package/dist/mcp/tools-prove.js +13 -13
- package/package.json +1 -1
package/README.md
CHANGED
|
@@ -624,6 +624,37 @@ compare everywhere else.
|
|
|
624
624
|
**It never predicts a duration, a capacity, or what a user will feel.** It cannot see your production
|
|
625
625
|
hardware, your real data volume or your real traffic, and every report says so.
|
|
626
626
|
|
|
627
|
+
### Lighthouse too (Phase 31, new)
|
|
628
|
+
|
|
629
|
+
```bash
|
|
630
|
+
tewip perf lighthouse-report.json --record
|
|
631
|
+
```
|
|
632
|
+
|
|
633
|
+
k6 measures what your server does under load. Lighthouse measures what **one page load costs in a real
|
|
634
|
+
browser** — Largest Contentful Paint, layout shift, blocking time — which is nearer to what a person
|
|
635
|
+
actually waits for. `tewip perf` takes either and works out which it was given.
|
|
636
|
+
|
|
637
|
+
Lighthouse needs the same discipline *more*, not less. **Four real Lighthouse runs of an unchanged
|
|
638
|
+
page gave LCP of 922, 912, 936 and 916ms.** Same rule: three baselines, outside the range by more than
|
|
639
|
+
its own width.
|
|
640
|
+
|
|
641
|
+
**And Tewip does not invent a verdict.** "Is 2112ms bad?" has a published answer and it is Google's:
|
|
642
|
+
Lighthouse scores every metric on its own curve. Tewip reads that score rather than hard-coding
|
|
643
|
+
thresholds it remembered — which matters, because on the real slowed run FCP and LCP had the *identical*
|
|
644
|
+
value of 2112ms and Lighthouse rates them differently:
|
|
645
|
+
|
|
646
|
+
```
|
|
647
|
+
- good — `largest-contentful-paint`: 2112ms
|
|
648
|
+
- needs improvement — `first-contentful-paint`: 2112ms
|
|
649
|
+
- **poor** — `server-response-time`: 611ms
|
|
650
|
+
```
|
|
651
|
+
|
|
652
|
+
One remembered threshold applied to both would have been wrong.
|
|
653
|
+
|
|
654
|
+
It also surfaces the **ten accessibility audits Lighthouse refuses to score** — logical tab order, focus
|
|
655
|
+
traps, whether visual order follows the DOM. Those are neither failures nor passes, and a page with no
|
|
656
|
+
failures and ten unanswered manual checks is not a page that passed.
|
|
657
|
+
|
|
627
658
|
One more thing it refuses: if your summary came from `--new-machine-readable-summary`, that export
|
|
628
659
|
records **no threshold results at all** — the word appears in it only inside a copy of your script.
|
|
629
660
|
Tewip says "this export cannot tell me" rather than "nothing was breached".
|
|
@@ -719,6 +750,26 @@ A source finding is set aside only when the run opened **that same file** and re
|
|
|
719
750
|
In a React or Vue project that means nothing is set aside — the source and the rendered output are two
|
|
720
751
|
different artefacts, and neither is evidence about the other.
|
|
721
752
|
|
|
753
|
+
**Already run Lighthouse instead? That works too** (Phase 32, new):
|
|
754
|
+
|
|
755
|
+
```bash
|
|
756
|
+
tewip access --report lighthouse-report.json
|
|
757
|
+
```
|
|
758
|
+
|
|
759
|
+
Lighthouse's accessibility audits *are* axe rules — same rule ids — so the same triage applies. Most
|
|
760
|
+
teams have Lighthouse in CI and no separate axe step; they get the whole of the above from a tool they
|
|
761
|
+
already run.
|
|
762
|
+
|
|
763
|
+
The catch was that a Lighthouse report strips axe's tags, so it says `landmark-one-main` and nothing
|
|
764
|
+
about which WCAG criterion that is, or whether it is one at all. That distinction covers **30 of axe's
|
|
765
|
+
105 rules**, and a remembered table of a hundred mappings is a table with mistakes in it. So the table
|
|
766
|
+
is **generated from axe-core itself** (`scripts/axe-rules.ts`) — 105 rules, 75 carrying a criterion, 30
|
|
767
|
+
best-practice only, none unclassified.
|
|
768
|
+
|
|
769
|
+
It also surfaces the **ten audits Lighthouse marks `manual`** — logical tab order, focus traps, whether
|
|
770
|
+
visual order follows the DOM. Those become questions, never passes. A page with no failures and ten
|
|
771
|
+
unanswered manual checks has not passed.
|
|
772
|
+
|
|
722
773
|
## Teach it your house rules (Phase 27)
|
|
723
774
|
|
|
724
775
|
```bash
|
package/dist/cli/help.js
CHANGED
|
@@ -118,17 +118,19 @@ Usage:
|
|
|
118
118
|
is what was asked for covered. Answers ready, not ready, or cannot
|
|
119
119
|
say - and cannot say is never rounded up to ready. --json writes a
|
|
120
120
|
sealed verdict that can be checked later. No model, no cost.
|
|
121
|
-
tewip perf <k6-summary.json> [--record] [--label checkout] [--strict] [--out perf.md]
|
|
122
|
-
Is this change slower, and does it matter? Reads a k6 run
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
121
|
+
tewip perf <k6-summary.json | lighthouse.json> [--record] [--label checkout] [--strict] [--out perf.md]
|
|
122
|
+
Is this change slower, and does it matter? Reads a k6 run (what
|
|
123
|
+
the server does under load) or a Lighthouse report (what one
|
|
124
|
+
page load costs in a browser), and answers two questions apart:
|
|
125
|
+
is anything in a bad state now - from the thresholds you wrote,
|
|
126
|
+
or from Lighthouse's own scoring, never a budget Tewip invented
|
|
127
|
+
- and is this slower than it was. The second only from repeated
|
|
128
|
+
baselines: with fewer than three it says cannot say, because one
|
|
129
|
+
run against one cannot tell a regression from a busy machine.
|
|
130
|
+
--record adds a run to the baseline, and is deliberately not
|
|
131
|
+
automatic: a regression recorded by accident becomes next week's
|
|
132
|
+
normal. It never predicts a duration or a capacity. No cost.
|
|
133
|
+
tewip access [<file or directory>] [--report axe.json|lighthouse.json] [--strict]
|
|
132
134
|
Can everybody use this? Reads your markup and names the barriers
|
|
133
135
|
that reading alone can prove - a picture with no alt text, a field
|
|
134
136
|
with no label, a div only a mouse can use - each against its WCAG
|
|
@@ -137,11 +139,13 @@ Usage:
|
|
|
137
139
|
criterion it cannot reach without rendering the page (contrast,
|
|
138
140
|
focus order, whether the alt text is any good) is printed every
|
|
139
141
|
time. It never says a page is accessible, only what was checked.
|
|
140
|
-
Give it --report with the JSON from an axe run
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
142
|
+
Give it --report with the JSON from an axe run or a Lighthouse
|
|
143
|
+
report and it reads that too - Lighthouse runs axe rules, so
|
|
144
|
+
either works. Contrast moves out of the unchecked list, rules
|
|
145
|
+
that map to no WCAG criterion are reported as advice rather than
|
|
146
|
+
barriers (51 of 174 findings on nine real pages), and what the
|
|
147
|
+
tool could not determine becomes a question. Lighthouse's ten
|
|
148
|
+
manual checks are questions too, never passes. No model, no cost.
|
|
145
149
|
tewip standards [--diff main...HEAD] [--strict] [--out standards.md]
|
|
146
150
|
Teach Tewip your house rules. Markdown files in
|
|
147
151
|
.tewip/standards/ - a forbid or require line is checked and
|
|
@@ -14,9 +14,13 @@ import { sourceFiles } from '../secure/secrets.js';
|
|
|
14
14
|
import { JS_LIKE, MARKUP, tags } from './markup.js';
|
|
15
15
|
import { checkFile } from './checks.js';
|
|
16
16
|
import { readAxeReport } from './reports.js';
|
|
17
|
-
import {
|
|
18
|
-
import {
|
|
19
|
-
|
|
17
|
+
import { isLighthouseFile, readLighthouse } from '../perf/lighthouse.js';
|
|
18
|
+
import { accessibilityFromLighthouse, criteriaFromLighthouse, rulesRunByLighthouse } from './from-lighthouse.js';
|
|
19
|
+
import { merge, pagesIn } from './merge.js';
|
|
20
|
+
import { renderAccess, worstFirst } from './render.js';
|
|
21
|
+
// Re-exported so callers do not have to know the report moved into its own file.
|
|
22
|
+
export { renderAccess, worstFirst };
|
|
23
|
+
import { blocks } from './wcag.js';
|
|
20
24
|
/** One line per finding a person has already judged: "<rule> <file>:<line> - reason". */
|
|
21
25
|
export function readAccepted(root) {
|
|
22
26
|
const file = join(root, '.tewip', 'access-accepted.txt');
|
|
@@ -30,15 +34,6 @@ export function readAccepted(root) {
|
|
|
30
34
|
export function keyOf(f) {
|
|
31
35
|
return `${f.rule} ${f.file}:${f.line}`;
|
|
32
36
|
}
|
|
33
|
-
export function worstFirst(a, b) {
|
|
34
|
-
const sa = ORDER[RULES[a.rule]?.severity ?? 'minor'];
|
|
35
|
-
const sb = ORDER[RULES[b.rule]?.severity ?? 'minor'];
|
|
36
|
-
if (sa !== sb)
|
|
37
|
-
return sa - sb;
|
|
38
|
-
if (a.file !== b.file)
|
|
39
|
-
return a.file < b.file ? -1 : 1;
|
|
40
|
-
return a.line - b.line;
|
|
41
|
-
}
|
|
42
37
|
/** Reads every markup file under `root` and returns what the markup alone can settle. */
|
|
43
38
|
/**
|
|
44
39
|
* Markup written to exercise code, not to be used by a person. An `<img>` with no alt in a
|
|
@@ -71,119 +66,24 @@ export function findBarriers(root, only) {
|
|
|
71
66
|
}
|
|
72
67
|
return { findings, filesRead };
|
|
73
68
|
}
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
lines.push(`${criterionOf(rule)}, ${meta?.severity ?? 'unrated'}.`, '');
|
|
93
|
-
if (meta)
|
|
94
|
-
lines.push(`Who it stops: ${meta.stops}.`, '');
|
|
95
|
-
for (const f of found.slice(0, 20)) {
|
|
96
|
-
lines.push(`- \`${f.file}:${f.line}\` ${f.element}`);
|
|
97
|
-
lines.push(` - ${f.why}`);
|
|
98
|
-
lines.push(` - do this: ${f.next}`);
|
|
99
|
-
}
|
|
100
|
-
if (found.length > 20)
|
|
101
|
-
lines.push(`- ...and ${found.length - 20} more of the same, in the JSON beside this report.`);
|
|
102
|
-
lines.push('');
|
|
103
|
-
}
|
|
104
|
-
}
|
|
105
|
-
export function renderAccess(result, scanned) {
|
|
106
|
-
const { barriers, questions, accepted, filesRead, browser } = result;
|
|
107
|
-
const lines = ['# Tewip accessibility check', ''];
|
|
108
|
-
lines.push(`Read ${filesRead} markup ${filesRead === 1 ? 'file' : 'files'} under \`${scanned}\`, without rendering any of them.`, '');
|
|
109
|
-
if (browser) {
|
|
110
|
-
lines.push(`Also read an axe run over ${browser.pages.length} rendered ${browser.pages.length === 1 ? 'page' : 'pages'}, ` +
|
|
111
|
-
'which settles what reading markup cannot.', '');
|
|
112
|
-
}
|
|
113
|
-
if (barriers.length === 0 && questions.length === 0 && !browser) {
|
|
114
|
-
lines.push('No barrier was found that reading the markup can settle, and nothing needs a decision.', '');
|
|
115
|
-
}
|
|
116
|
-
else {
|
|
117
|
-
const stopping = barriers.filter((f) => blocks(f.rule)).length;
|
|
118
|
-
const counts = [
|
|
119
|
-
`${barriers.length} ${barriers.length === 1 ? 'barrier' : 'barriers'} (${stopping} that stop somebody outright)`,
|
|
120
|
-
`${questions.length} for a person to settle`,
|
|
121
|
-
];
|
|
122
|
-
if (accepted.length > 0)
|
|
123
|
-
counts.push(`${accepted.length} already accepted`);
|
|
124
|
-
if (browser) {
|
|
125
|
-
counts.unshift(`${browser.barriers.length} barriers the browser run proves`);
|
|
126
|
-
if (browser.questions.length > 0)
|
|
127
|
-
counts.push(`${browser.questions.length} the browser could not determine`);
|
|
128
|
-
}
|
|
129
|
-
lines.push(`**${counts.join(', ')}.**`, '');
|
|
130
|
-
}
|
|
131
|
-
if (browser) {
|
|
132
|
-
renderBrowserSection('Barriers seen in the browser', browser.barriers, lines);
|
|
133
|
-
renderBrowserSection('The browser could not determine these', browser.questions, lines);
|
|
134
|
-
}
|
|
135
|
-
section('Barriers read from the source', [...barriers].sort(worstFirst), lines);
|
|
136
|
-
section('Only a person can settle these', [...questions].sort(worstFirst), lines);
|
|
137
|
-
if (browser && browser.advice.length > 0) {
|
|
138
|
-
// Kept out of the counts above on purpose. These are axe's own best-practice rules, which test
|
|
139
|
-
// no standard: on nine real pages they were 51 of 174 findings, outnumbering every real barrier.
|
|
140
|
-
// Reporting them as equals is how a team learns to ignore the whole report.
|
|
141
|
-
const rules = [...new Set(browser.advice.map((f) => f.rule))].sort();
|
|
142
|
-
lines.push('## Advice, not barriers', '');
|
|
143
|
-
lines.push(`${browser.advice.length} findings come from axe rules that map to no WCAG criterion - ` +
|
|
144
|
-
`${rules.join(', ')}. They are somebody's good idea rather than a standard, so they are not counted as barriers ` +
|
|
145
|
-
'and never fail a build. Worth reading once; not worth a release.', '');
|
|
146
|
-
}
|
|
147
|
-
if (browser && browser.superseded.length > 0) {
|
|
148
|
-
lines.push('## Set aside, because the browser run checked them', '');
|
|
149
|
-
lines.push(`${browser.superseded.length} source ${browser.superseded.length === 1 ? 'finding' : 'findings'} ` +
|
|
150
|
-
'are not listed above: the axe run checked their criterion on a rendered page, so its answer is the better ' +
|
|
151
|
-
'one and repeating them here would count the same barrier twice. They are in the JSON beside this report.', '');
|
|
152
|
-
}
|
|
153
|
-
if (accepted.length > 0) {
|
|
154
|
-
lines.push('## Already accepted', '');
|
|
155
|
-
lines.push(`${accepted.length} ${accepted.length === 1 ? 'finding is' : 'findings are'} listed in \`.tewip/access-accepted.txt\`, so they are not repeated here.`, '');
|
|
156
|
-
}
|
|
157
|
-
// Coverage comes from every rule that ran, not from the ones that found something: a contrast
|
|
158
|
-
// check that ran and passed has still checked contrast.
|
|
159
|
-
const ran = browser?.rulesRun ?? new Set();
|
|
160
|
-
const settled = nowChecked(ran);
|
|
161
|
-
const remaining = stillNotChecked(ran);
|
|
162
|
-
if (settled.length > 0) {
|
|
163
|
-
lines.push('## Checked here only because a browser run was given', '');
|
|
164
|
-
for (const n of settled)
|
|
165
|
-
lines.push(`- **${n.criterion}** - would otherwise be unreachable: ${n.why}`);
|
|
166
|
-
lines.push('');
|
|
167
|
-
}
|
|
168
|
-
lines.push('## What was not checked', '');
|
|
169
|
-
lines.push(browser
|
|
170
|
-
? 'Even with the browser run, none of the following was tested:'
|
|
171
|
-
: 'This read the markup. It did not render a page, so none of the following was tested:', '');
|
|
172
|
-
for (const n of remaining)
|
|
173
|
-
lines.push(`- **${n.criterion}** - ${n.why}`);
|
|
174
|
-
if (browser) {
|
|
175
|
-
lines.push('');
|
|
176
|
-
lines.push(`The run covered ${browser.pages.length} ${browser.pages.length === 1 ? 'page' : 'pages'}. ` +
|
|
177
|
-
'Nothing is known about a page it did not open, or about a state it did not reach - a dialog that was never ' +
|
|
178
|
-
'opened, a form that was never submitted, an error that was never triggered.');
|
|
179
|
-
}
|
|
180
|
-
lines.push('');
|
|
181
|
-
lines.push('A clean result here means these checks found nothing, not that the product is accessible. ' +
|
|
182
|
-
'Roughly a third of WCAG can be judged by a machine at all' +
|
|
183
|
-
(browser ? '' : ', and reading source rather than a rendered page reaches less than that') +
|
|
184
|
-
'. The rest needs a person, ideally one who uses a screen reader.', '');
|
|
185
|
-
lines.push('To accept a finding, add its key to `.tewip/access-accepted.txt`, one per line, with a reason after ` - `.', '');
|
|
186
|
-
return lines.join('\n');
|
|
69
|
+
/**
|
|
70
|
+
* Reads whichever browser evidence the team actually has.
|
|
71
|
+
*
|
|
72
|
+
* axe and Lighthouse are the same rules - Lighthouse runs axe - so both become one shape and are
|
|
73
|
+
* triaged identically. A team with Lighthouse in CI and no separate axe step gets the same answers
|
|
74
|
+
* without adding a tool (Phase 32).
|
|
75
|
+
*/
|
|
76
|
+
export function readBrowserRun(path) {
|
|
77
|
+
if (!isLighthouseFile(path))
|
|
78
|
+
return readAxeReport(path);
|
|
79
|
+
const report = readLighthouse(path);
|
|
80
|
+
return {
|
|
81
|
+
source: 'lighthouse',
|
|
82
|
+
findings: accessibilityFromLighthouse(report),
|
|
83
|
+
pages: new Set(report.url === '' ? [] : [report.url]),
|
|
84
|
+
rulesRun: rulesRunByLighthouse(report),
|
|
85
|
+
criteriaCovered: criteriaFromLighthouse(report),
|
|
86
|
+
};
|
|
187
87
|
}
|
|
188
88
|
export function runAccess(cmd) {
|
|
189
89
|
const root = resolve(cmd.root);
|
|
@@ -193,7 +93,7 @@ export function runAccess(cmd) {
|
|
|
193
93
|
const open = findings.filter((f) => !alreadyJudged.has(keyOf(f)));
|
|
194
94
|
// Phase 29: a browser run, where one was given. It is the better witness for every criterion it
|
|
195
95
|
// checked, so the source findings for those criteria are set aside rather than repeated.
|
|
196
|
-
const run = cmd.report ?
|
|
96
|
+
const run = cmd.report ? readBrowserRun(resolve(root, cmd.report)) : undefined;
|
|
197
97
|
const graded = run ? merge(open, run, root) : undefined;
|
|
198
98
|
const kept = graded ? graded.kept : open;
|
|
199
99
|
const barriers = kept.filter((f) => f.confidence === 'barrier');
|
|
@@ -206,6 +106,7 @@ export function runAccess(cmd) {
|
|
|
206
106
|
superseded: graded.superseded,
|
|
207
107
|
pages: pagesIn(run),
|
|
208
108
|
rulesRun: run.rulesRun,
|
|
109
|
+
tool: run.source,
|
|
209
110
|
}
|
|
210
111
|
: undefined;
|
|
211
112
|
const result = { barriers, questions, accepted, filesRead, ...(browser ? { browser } : {}) };
|
|
@@ -0,0 +1,128 @@
|
|
|
1
|
+
// Phase 32: what each axe rule tests, taken from axe-core itself.
|
|
2
|
+
//
|
|
3
|
+
// Lighthouse's accessibility audits are axe rules - the ids are identical - but a Lighthouse report
|
|
4
|
+
// does not carry axe's tags, so it says "landmark-one-main" and nothing about which criterion that
|
|
5
|
+
// is, or whether it is a criterion at all. Phase 29 needs that to tell a legal failure from somebody's
|
|
6
|
+
// good idea, and 42% of axe's output is the latter.
|
|
7
|
+
//
|
|
8
|
+
// Writing the table by hand was the reason Phase 31 left this undone: a remembered mapping between a
|
|
9
|
+
// hundred rule ids and WCAG numbers is a mapping with mistakes in it, and a wrong criterion beside a
|
|
10
|
+
// finding is worse than no criterion. So the table is **generated**: axe-core is loaded in a browser
|
|
11
|
+
// and asked for its own rule catalogue, and these are its tags, parsed the same way Phase 29 parses
|
|
12
|
+
// the tags on a live run.
|
|
13
|
+
//
|
|
14
|
+
// Generated from axe-core 4.14.0. 105 rules: 75 carry a WCAG criterion,
|
|
15
|
+
// 30 are tagged best-practice only and test no standard at all.
|
|
16
|
+
// Regenerate with: node scripts/axe-rules.ts - never by editing this file.
|
|
17
|
+
export const AXE_CORE_VERSION = '4.14.0';
|
|
18
|
+
export const AXE_RULES = {
|
|
19
|
+
'accesskeys': { criterion: null, level: null, en301549: null },
|
|
20
|
+
'area-alt': { criterion: '2.4.4', level: 'A', en301549: '9.2.4.4' },
|
|
21
|
+
'aria-allowed-attr': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
22
|
+
'aria-allowed-role': { criterion: null, level: null, en301549: null },
|
|
23
|
+
'aria-braille-equivalent': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
24
|
+
'aria-command-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
25
|
+
'aria-conditional-attr': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
26
|
+
'aria-deprecated-role': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
27
|
+
'aria-dialog-name': { criterion: null, level: null, en301549: null },
|
|
28
|
+
'aria-hidden-body': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
29
|
+
'aria-hidden-focus': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
30
|
+
'aria-input-field-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
31
|
+
'aria-meter-name': { criterion: '1.1.1', level: 'A', en301549: '9.1.1.1' },
|
|
32
|
+
'aria-progressbar-name': { criterion: '1.1.1', level: 'A', en301549: '9.1.1.1' },
|
|
33
|
+
'aria-prohibited-attr': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
34
|
+
'aria-required-attr': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
35
|
+
'aria-required-children': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
36
|
+
'aria-required-parent': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
37
|
+
'aria-roledescription': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
38
|
+
'aria-roles': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
39
|
+
'aria-tab-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
40
|
+
'aria-text': { criterion: null, level: null, en301549: null },
|
|
41
|
+
'aria-toggle-field-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
42
|
+
'aria-tooltip-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
43
|
+
'aria-treeitem-name': { criterion: null, level: null, en301549: null },
|
|
44
|
+
'aria-valid-attr': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
45
|
+
'aria-valid-attr-value': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
46
|
+
'audio-caption': { criterion: '1.2.1', level: 'A', en301549: '9.1.2.1' },
|
|
47
|
+
'autocomplete-valid': { criterion: '1.3.5', level: 'AA', en301549: '9.1.3.5' },
|
|
48
|
+
'avoid-inline-spacing': { criterion: '1.4.12', level: 'AA', en301549: '9.1.4.12' },
|
|
49
|
+
'blink': { criterion: '2.2.2', level: 'A', en301549: '9.2.2.2' },
|
|
50
|
+
'button-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
51
|
+
'bypass': { criterion: '2.4.1', level: 'A', en301549: '9.2.4.1' },
|
|
52
|
+
'color-contrast': { criterion: '1.4.3', level: 'AA', en301549: '9.1.4.3' },
|
|
53
|
+
'color-contrast-enhanced': { criterion: '1.4.6', level: 'AAA', en301549: null },
|
|
54
|
+
'css-orientation-lock': { criterion: '1.3.4', level: 'AA', en301549: '9.1.3.4' },
|
|
55
|
+
'definition-list': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
56
|
+
'dlitem': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
57
|
+
'document-title': { criterion: '2.4.2', level: 'A', en301549: '9.2.4.2' },
|
|
58
|
+
'duplicate-id': { criterion: '4.1.1', level: null, en301549: null },
|
|
59
|
+
'duplicate-id-active': { criterion: '4.1.1', level: null, en301549: null },
|
|
60
|
+
'duplicate-id-aria': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
61
|
+
'empty-heading': { criterion: null, level: null, en301549: null },
|
|
62
|
+
'empty-table-header': { criterion: null, level: null, en301549: null },
|
|
63
|
+
'focus-order-semantics': { criterion: null, level: null, en301549: null },
|
|
64
|
+
'form-field-multiple-labels': { criterion: '3.3.2', level: 'A', en301549: '9.3.3.2' },
|
|
65
|
+
'frame-focusable-content': { criterion: '2.1.1', level: 'A', en301549: '9.2.1.1' },
|
|
66
|
+
'frame-tested': { criterion: null, level: null, en301549: null },
|
|
67
|
+
'frame-title': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
68
|
+
'frame-title-unique': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
69
|
+
'heading-order': { criterion: null, level: null, en301549: null },
|
|
70
|
+
'hidden-content': { criterion: null, level: null, en301549: null },
|
|
71
|
+
'html-has-lang': { criterion: '3.1.1', level: 'A', en301549: '9.3.1.1' },
|
|
72
|
+
'html-lang-valid': { criterion: '3.1.1', level: 'A', en301549: '9.3.1.1' },
|
|
73
|
+
'html-xml-lang-mismatch': { criterion: '3.1.1', level: 'A', en301549: '9.3.1.1' },
|
|
74
|
+
'identical-links-same-purpose': { criterion: '2.4.9', level: 'AAA', en301549: null },
|
|
75
|
+
'image-alt': { criterion: '1.1.1', level: 'A', en301549: '9.1.1.1' },
|
|
76
|
+
'image-redundant-alt': { criterion: null, level: null, en301549: null },
|
|
77
|
+
'input-button-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
78
|
+
'input-image-alt': { criterion: '1.1.1', level: 'A', en301549: '9.1.1.1' },
|
|
79
|
+
'label': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
80
|
+
'label-content-name-mismatch': { criterion: '2.5.3', level: 'A', en301549: '9.2.5.3' },
|
|
81
|
+
'label-title-only': { criterion: null, level: null, en301549: null },
|
|
82
|
+
'landmark-banner-is-top-level': { criterion: null, level: null, en301549: null },
|
|
83
|
+
'landmark-complementary-is-top-level': { criterion: null, level: null, en301549: null },
|
|
84
|
+
'landmark-contentinfo-is-top-level': { criterion: null, level: null, en301549: null },
|
|
85
|
+
'landmark-main-is-top-level': { criterion: null, level: null, en301549: null },
|
|
86
|
+
'landmark-no-duplicate-banner': { criterion: null, level: null, en301549: null },
|
|
87
|
+
'landmark-no-duplicate-contentinfo': { criterion: null, level: null, en301549: null },
|
|
88
|
+
'landmark-no-duplicate-main': { criterion: null, level: null, en301549: null },
|
|
89
|
+
'landmark-one-main': { criterion: null, level: null, en301549: null },
|
|
90
|
+
'landmark-unique': { criterion: null, level: null, en301549: null },
|
|
91
|
+
'link-in-text-block': { criterion: '1.4.1', level: 'A', en301549: '9.1.4.1' },
|
|
92
|
+
'link-name': { criterion: '2.4.4', level: 'A', en301549: '9.2.4.4' },
|
|
93
|
+
'list': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
94
|
+
'listitem': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
95
|
+
'marquee': { criterion: '2.2.2', level: 'A', en301549: '9.2.2.2' },
|
|
96
|
+
'meta-refresh': { criterion: '2.2.1', level: 'A', en301549: '9.2.2.1' },
|
|
97
|
+
'meta-refresh-no-exceptions': { criterion: '2.2.4', level: 'AAA', en301549: null },
|
|
98
|
+
'meta-viewport': { criterion: '1.4.4', level: 'AA', en301549: '9.1.4.4' },
|
|
99
|
+
'meta-viewport-large': { criterion: null, level: null, en301549: null },
|
|
100
|
+
'nested-interactive': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
101
|
+
'no-autoplay-audio': { criterion: '1.4.2', level: 'A', en301549: '9.1.4.2' },
|
|
102
|
+
'object-alt': { criterion: '1.1.1', level: 'A', en301549: '9.1.1.1' },
|
|
103
|
+
'p-as-heading': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
104
|
+
'page-has-heading-one': { criterion: null, level: null, en301549: null },
|
|
105
|
+
'presentation-role-conflict': { criterion: null, level: null, en301549: null },
|
|
106
|
+
'region': { criterion: null, level: null, en301549: null },
|
|
107
|
+
'role-img-alt': { criterion: '1.1.1', level: 'A', en301549: '9.1.1.1' },
|
|
108
|
+
'scope-attr-valid': { criterion: null, level: null, en301549: null },
|
|
109
|
+
'scrollable-region-focusable': { criterion: '2.1.1', level: 'A', en301549: '9.2.1.1' },
|
|
110
|
+
'select-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
111
|
+
'server-side-image-map': { criterion: '2.1.1', level: 'A', en301549: '9.2.1.1' },
|
|
112
|
+
'skip-link': { criterion: null, level: null, en301549: null },
|
|
113
|
+
'summary-name': { criterion: '4.1.2', level: 'A', en301549: '9.4.1.2' },
|
|
114
|
+
'svg-img-alt': { criterion: '1.1.1', level: 'A', en301549: '9.1.1.1' },
|
|
115
|
+
'tabindex': { criterion: null, level: null, en301549: null },
|
|
116
|
+
'table-duplicate-name': { criterion: null, level: null, en301549: null },
|
|
117
|
+
'table-fake-caption': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
118
|
+
'target-size': { criterion: '2.5.8', level: 'AA', en301549: '9.2.5.8' },
|
|
119
|
+
'td-has-header': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
120
|
+
'td-headers-attr': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
121
|
+
'th-has-data-cells': { criterion: '1.3.1', level: 'A', en301549: '9.1.3.1' },
|
|
122
|
+
'valid-lang': { criterion: '3.1.2', level: 'AA', en301549: '9.3.1.2' },
|
|
123
|
+
'video-caption': { criterion: '1.2.2', level: 'A', en301549: '9.1.2.2' },
|
|
124
|
+
};
|
|
125
|
+
/** What a rule tests, or undefined when axe has no rule by that name. */
|
|
126
|
+
export function factsFor(ruleId) {
|
|
127
|
+
return AXE_RULES[ruleId];
|
|
128
|
+
}
|
|
@@ -0,0 +1,74 @@
|
|
|
1
|
+
import { factsFor } from './axe-rules.js';
|
|
2
|
+
/** WCAG's own ordering: level A is the most basic requirement, so its failures come first. */
|
|
3
|
+
const BY_LEVEL = { A: 'blocker', AA: 'serious', AAA: 'moderate' };
|
|
4
|
+
/**
|
|
5
|
+
* The accessibility half of a Lighthouse report, as findings the Phase 29 report can print.
|
|
6
|
+
*
|
|
7
|
+
* Three kinds come out, the same three an axe run produces:
|
|
8
|
+
* - a failing audit whose rule tests a WCAG criterion -> a barrier;
|
|
9
|
+
* - a failing audit whose rule tests nobody's standard -> advice;
|
|
10
|
+
* - an audit Lighthouse marks `manual` -> a question, because Lighthouse itself
|
|
11
|
+
* declines to answer it. Those are not passes, and a page with no failures and ten unanswered
|
|
12
|
+
* manual checks has not passed.
|
|
13
|
+
*/
|
|
14
|
+
export function accessibilityFromLighthouse(report) {
|
|
15
|
+
const out = [];
|
|
16
|
+
const page = report.url;
|
|
17
|
+
for (const failure of report.accessibilityFailures) {
|
|
18
|
+
const facts = factsFor(failure.id);
|
|
19
|
+
const criterion = facts?.criterion ?? null;
|
|
20
|
+
out.push({
|
|
21
|
+
rule: failure.id,
|
|
22
|
+
tool: 'axe',
|
|
23
|
+
confidence: criterion === null ? 'advice' : 'barrier',
|
|
24
|
+
criterion,
|
|
25
|
+
level: facts?.level ?? null,
|
|
26
|
+
en301549: facts?.en301549 ?? null,
|
|
27
|
+
severity: criterion === null ? 'minor' : (BY_LEVEL[facts?.level ?? 'A'] ?? 'serious'),
|
|
28
|
+
html: failure.snippet,
|
|
29
|
+
selector: failure.selector,
|
|
30
|
+
page,
|
|
31
|
+
why: failure.title,
|
|
32
|
+
next: failure.explanation || failure.title,
|
|
33
|
+
});
|
|
34
|
+
}
|
|
35
|
+
for (const check of report.manual) {
|
|
36
|
+
const facts = factsFor(check.id);
|
|
37
|
+
out.push({
|
|
38
|
+
rule: check.id,
|
|
39
|
+
tool: 'axe',
|
|
40
|
+
confidence: 'ask-a-person',
|
|
41
|
+
criterion: facts?.criterion ?? null,
|
|
42
|
+
level: facts?.level ?? null,
|
|
43
|
+
en301549: facts?.en301549 ?? null,
|
|
44
|
+
severity: 'moderate',
|
|
45
|
+
html: '',
|
|
46
|
+
selector: '',
|
|
47
|
+
page,
|
|
48
|
+
why: `${check.title}. Lighthouse does not score this one and hands it to a person`,
|
|
49
|
+
next: 'check it yourself, ideally with a keyboard and a screen reader; it is neither a pass nor a failure until somebody looks',
|
|
50
|
+
});
|
|
51
|
+
}
|
|
52
|
+
return out;
|
|
53
|
+
}
|
|
54
|
+
/**
|
|
55
|
+
* Which axe rules a Lighthouse run actually exercised.
|
|
56
|
+
*
|
|
57
|
+
* Everything in the accessibility category ran, including the audits that passed and the ones that
|
|
58
|
+
* did not apply to this page - both are evidence the rule was checked. This is what lets a
|
|
59
|
+
* Lighthouse run move contrast off the "not checked" list, exactly as an axe run does, since
|
|
60
|
+
* `color-contrast` is one of the rules it runs.
|
|
61
|
+
*/
|
|
62
|
+
export function rulesRunByLighthouse(report) {
|
|
63
|
+
return new Set(report.accessibilityAuditIds);
|
|
64
|
+
}
|
|
65
|
+
/** The criteria those rules test, for shrinking what the report has to call unchecked. */
|
|
66
|
+
export function criteriaFromLighthouse(report) {
|
|
67
|
+
const out = new Set();
|
|
68
|
+
for (const id of report.accessibilityAuditIds) {
|
|
69
|
+
const criterion = factsFor(id)?.criterion;
|
|
70
|
+
if (criterion)
|
|
71
|
+
out.add(criterion);
|
|
72
|
+
}
|
|
73
|
+
return out;
|
|
74
|
+
}
|
|
@@ -0,0 +1,133 @@
|
|
|
1
|
+
// Phase 28-32: writing the accessibility report.
|
|
2
|
+
//
|
|
3
|
+
// Split out of access-command.ts to keep every file in this project under 300 lines, which the
|
|
4
|
+
// Lighthouse reader pushed it past. The rendering is the part most worth keeping readable: it is
|
|
5
|
+
// where the distinction between a barrier, a question and a piece of advice is actually made
|
|
6
|
+
// visible to a person, and where what was *not* checked gets said.
|
|
7
|
+
import { nowChecked, renderBrowserSection, stillNotChecked } from './merge.js';
|
|
8
|
+
import { blocks, criterionOf, RULES } from './wcag.js';
|
|
9
|
+
const ORDER = { blocker: 0, serious: 1, moderate: 2, minor: 3 };
|
|
10
|
+
export function worstFirst(a, b) {
|
|
11
|
+
const sa = ORDER[RULES[a.rule]?.severity ?? 'minor'];
|
|
12
|
+
const sb = ORDER[RULES[b.rule]?.severity ?? 'minor'];
|
|
13
|
+
if (sa !== sb)
|
|
14
|
+
return sa - sb;
|
|
15
|
+
if (a.file !== b.file)
|
|
16
|
+
return a.file < b.file ? -1 : 1;
|
|
17
|
+
return a.line - b.line;
|
|
18
|
+
}
|
|
19
|
+
function group(findings) {
|
|
20
|
+
const by = new Map();
|
|
21
|
+
for (const f of findings) {
|
|
22
|
+
const list = by.get(f.rule);
|
|
23
|
+
if (list)
|
|
24
|
+
list.push(f);
|
|
25
|
+
else
|
|
26
|
+
by.set(f.rule, [f]);
|
|
27
|
+
}
|
|
28
|
+
return by;
|
|
29
|
+
}
|
|
30
|
+
function section(title, findings, lines) {
|
|
31
|
+
if (findings.length === 0)
|
|
32
|
+
return;
|
|
33
|
+
lines.push(`## ${title}`, '');
|
|
34
|
+
for (const [rule, found] of group(findings)) {
|
|
35
|
+
const meta = RULES[rule];
|
|
36
|
+
lines.push(`### ${rule} - ${found.length} ${found.length === 1 ? 'place' : 'places'}`, '');
|
|
37
|
+
lines.push(`${criterionOf(rule)}, ${meta?.severity ?? 'unrated'}.`, '');
|
|
38
|
+
if (meta)
|
|
39
|
+
lines.push(`Who it stops: ${meta.stops}.`, '');
|
|
40
|
+
for (const f of found.slice(0, 20)) {
|
|
41
|
+
lines.push(`- \`${f.file}:${f.line}\` ${f.element}`);
|
|
42
|
+
lines.push(` - ${f.why}`);
|
|
43
|
+
lines.push(` - do this: ${f.next}`);
|
|
44
|
+
}
|
|
45
|
+
if (found.length > 20)
|
|
46
|
+
lines.push(`- ...and ${found.length - 20} more of the same, in the JSON beside this report.`);
|
|
47
|
+
lines.push('');
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
export function renderAccess(result, scanned) {
|
|
51
|
+
const { barriers, questions, accepted, filesRead, browser } = result;
|
|
52
|
+
const lines = ['# Tewip accessibility check', ''];
|
|
53
|
+
lines.push(`Read ${filesRead} markup ${filesRead === 1 ? 'file' : 'files'} under \`${scanned}\`, without rendering any of them.`, '');
|
|
54
|
+
if (browser) {
|
|
55
|
+
lines.push(`Also read a ${browser.tool === 'lighthouse' ? 'Lighthouse' : 'axe'} run over ${browser.pages.length} rendered ` +
|
|
56
|
+
`${browser.pages.length === 1 ? 'page' : 'pages'}, ` +
|
|
57
|
+
'which settles what reading markup cannot.', '');
|
|
58
|
+
}
|
|
59
|
+
if (barriers.length === 0 && questions.length === 0 && !browser) {
|
|
60
|
+
lines.push('No barrier was found that reading the markup can settle, and nothing needs a decision.', '');
|
|
61
|
+
}
|
|
62
|
+
else {
|
|
63
|
+
const stopping = barriers.filter((f) => blocks(f.rule)).length;
|
|
64
|
+
const counts = [
|
|
65
|
+
`${barriers.length} ${barriers.length === 1 ? 'barrier' : 'barriers'} (${stopping} that stop somebody outright)`,
|
|
66
|
+
`${questions.length} for a person to settle`,
|
|
67
|
+
];
|
|
68
|
+
if (accepted.length > 0)
|
|
69
|
+
counts.push(`${accepted.length} already accepted`);
|
|
70
|
+
if (browser) {
|
|
71
|
+
counts.unshift(`${browser.barriers.length} barriers the browser run proves`);
|
|
72
|
+
if (browser.questions.length > 0)
|
|
73
|
+
counts.push(`${browser.questions.length} the browser could not determine`);
|
|
74
|
+
}
|
|
75
|
+
lines.push(`**${counts.join(', ')}.**`, '');
|
|
76
|
+
}
|
|
77
|
+
if (browser) {
|
|
78
|
+
renderBrowserSection('Barriers seen in the browser', browser.barriers, lines);
|
|
79
|
+
renderBrowserSection('The browser could not determine these', browser.questions, lines);
|
|
80
|
+
}
|
|
81
|
+
section('Barriers read from the source', [...barriers].sort(worstFirst), lines);
|
|
82
|
+
section('Only a person can settle these', [...questions].sort(worstFirst), lines);
|
|
83
|
+
if (browser && browser.advice.length > 0) {
|
|
84
|
+
// Kept out of the counts above on purpose. These are axe's own best-practice rules, which test
|
|
85
|
+
// no standard: on nine real pages they were 51 of 174 findings, outnumbering every real barrier.
|
|
86
|
+
// Reporting them as equals is how a team learns to ignore the whole report.
|
|
87
|
+
const rules = [...new Set(browser.advice.map((f) => f.rule))].sort();
|
|
88
|
+
lines.push('## Advice, not barriers', '');
|
|
89
|
+
lines.push(`${browser.advice.length} ${browser.advice.length === 1 ? 'finding comes' : 'findings come'} from axe rules that map to no WCAG criterion - ` +
|
|
90
|
+
`${rules.join(', ')}. They are somebody's good idea rather than a standard, so they are not counted as barriers ` +
|
|
91
|
+
'and never fail a build. Worth reading once; not worth a release.', '');
|
|
92
|
+
}
|
|
93
|
+
if (browser && browser.superseded.length > 0) {
|
|
94
|
+
lines.push('## Set aside, because the browser run checked them', '');
|
|
95
|
+
lines.push(`${browser.superseded.length} source ${browser.superseded.length === 1 ? 'finding' : 'findings'} ` +
|
|
96
|
+
'are not listed above: the axe run checked their criterion on a rendered page, so its answer is the better ' +
|
|
97
|
+
'one and repeating them here would count the same barrier twice. They are in the JSON beside this report.', '');
|
|
98
|
+
}
|
|
99
|
+
if (accepted.length > 0) {
|
|
100
|
+
lines.push('## Already accepted', '');
|
|
101
|
+
lines.push(`${accepted.length} ${accepted.length === 1 ? 'finding is' : 'findings are'} listed in \`.tewip/access-accepted.txt\`, so they are not repeated here.`, '');
|
|
102
|
+
}
|
|
103
|
+
// Coverage comes from every rule that ran, not from the ones that found something: a contrast
|
|
104
|
+
// check that ran and passed has still checked contrast.
|
|
105
|
+
const ran = browser?.rulesRun ?? new Set();
|
|
106
|
+
const settled = nowChecked(ran);
|
|
107
|
+
const remaining = stillNotChecked(ran);
|
|
108
|
+
if (settled.length > 0) {
|
|
109
|
+
lines.push('## Checked here only because a browser run was given', '');
|
|
110
|
+
for (const n of settled)
|
|
111
|
+
lines.push(`- **${n.criterion}** - would otherwise be unreachable: ${n.why}`);
|
|
112
|
+
lines.push('');
|
|
113
|
+
}
|
|
114
|
+
lines.push('## What was not checked', '');
|
|
115
|
+
lines.push(browser
|
|
116
|
+
? 'Even with the browser run, none of the following was tested:'
|
|
117
|
+
: 'This read the markup. It did not render a page, so none of the following was tested:', '');
|
|
118
|
+
for (const n of remaining)
|
|
119
|
+
lines.push(`- **${n.criterion}** - ${n.why}`);
|
|
120
|
+
if (browser) {
|
|
121
|
+
lines.push('');
|
|
122
|
+
lines.push(`The run covered ${browser.pages.length} ${browser.pages.length === 1 ? 'page' : 'pages'}. ` +
|
|
123
|
+
'Nothing is known about a page it did not open, or about a state it did not reach - a dialog that was never ' +
|
|
124
|
+
'opened, a form that was never submitted, an error that was never triggered.');
|
|
125
|
+
}
|
|
126
|
+
lines.push('');
|
|
127
|
+
lines.push('A clean result here means these checks found nothing, not that the product is accessible. ' +
|
|
128
|
+
'Roughly a third of WCAG can be judged by a machine at all' +
|
|
129
|
+
(browser ? '' : ', and reading source rather than a rendered page reaches less than that') +
|
|
130
|
+
'. The rest needs a person, ideally one who uses a screen reader.', '');
|
|
131
|
+
lines.push('To accept a finding, add its key to `.tewip/access-accepted.txt`, one per line, with a reason after ` - `.', '');
|
|
132
|
+
return lines.join('\n');
|
|
133
|
+
}
|
|
@@ -145,7 +145,7 @@ export function readAxeReport(path) {
|
|
|
145
145
|
catch {
|
|
146
146
|
throw new Error(`${path} is not JSON. Tewip reads the JSON an axe run writes.`);
|
|
147
147
|
}
|
|
148
|
-
const run = { findings: [], pages: new Set(), rulesRun: new Set(), criteriaCovered: new Set() };
|
|
148
|
+
const run = { source: 'axe', findings: [], pages: new Set(), rulesRun: new Set(), criteriaCovered: new Set() };
|
|
149
149
|
const pageOf = (v, fallback) => text(v.url) || text(v.page) || fallback;
|
|
150
150
|
if (looksLikeAxe(parsed)) {
|
|
151
151
|
fromResult(parsed, pageOf(parsed, path), run);
|
|
@@ -25,7 +25,8 @@ export function storedRuns(root, label) {
|
|
|
25
25
|
const parsed = JSON.parse(trimmed);
|
|
26
26
|
// A line Tewip cannot read is skipped rather than throwing: one bad line written by a crashed
|
|
27
27
|
// run must not make the whole baseline unreadable.
|
|
28
|
-
|
|
28
|
+
// Either shape counts: `metrics` is what is written now, `timings` what older versions wrote.
|
|
29
|
+
if (parsed && typeof parsed.at === 'string' && parsed.label === label && (parsed.metrics ?? parsed.timings))
|
|
29
30
|
out.push(parsed);
|
|
30
31
|
}
|
|
31
32
|
catch {
|
|
@@ -34,22 +35,36 @@ export function storedRuns(root, label) {
|
|
|
34
35
|
}
|
|
35
36
|
return out;
|
|
36
37
|
}
|
|
37
|
-
export function recordRun(root, label, run,
|
|
38
|
+
export function recordRun(root, label, run, gitSha) {
|
|
38
39
|
const stored = {
|
|
39
40
|
at: run.at,
|
|
40
41
|
label,
|
|
41
42
|
...(gitSha ? { gitSha } : {}),
|
|
42
|
-
|
|
43
|
-
|
|
43
|
+
source: run.source,
|
|
44
|
+
metrics: Object.fromEntries(Object.entries(run.metrics).map(([m, v]) => [m, v.value])),
|
|
45
|
+
requests: run.volume.n,
|
|
44
46
|
};
|
|
45
47
|
const path = file(root);
|
|
46
48
|
mkdirSync(dirname(path), { recursive: true });
|
|
47
49
|
appendFileSync(path, `${JSON.stringify(stored)}\n`, 'utf8');
|
|
48
50
|
return stored;
|
|
49
51
|
}
|
|
50
|
-
/**
|
|
52
|
+
/**
|
|
53
|
+
* Every stored value for one metric, oldest first, for the comparison to read the spread from.
|
|
54
|
+
*
|
|
55
|
+
* Reads the older shape as well, so upgrading Tewip does not quietly empty a project's baseline -
|
|
56
|
+
* which would drop it below three runs and make Tewip refuse to judge a project it judged fine the
|
|
57
|
+
* day before.
|
|
58
|
+
*/
|
|
51
59
|
export function baselineFor(runs, metric) {
|
|
52
|
-
return runs
|
|
60
|
+
return runs
|
|
61
|
+
.map((r) => r.metrics?.[metric] ?? r.timings?.[metric]?.p95)
|
|
62
|
+
.filter((v) => typeof v === 'number' && Number.isFinite(v));
|
|
63
|
+
}
|
|
64
|
+
/** Runs from the same tool. A k6 p95 and a Lighthouse LCP are not each other's baseline. */
|
|
65
|
+
export function fromSameTool(runs, source) {
|
|
66
|
+
// Runs recorded before the field existed were all k6, which is what the absence means.
|
|
67
|
+
return runs.filter((r) => (r.source ?? 'k6') === source);
|
|
53
68
|
}
|
|
54
69
|
/** The labels this project has recorded, so a typo in --label does not look like an empty history. */
|
|
55
70
|
export function knownLabels(root) {
|
|
@@ -0,0 +1,155 @@
|
|
|
1
|
+
// Phase 31: reading a Lighthouse report.
|
|
2
|
+
//
|
|
3
|
+
// k6 measures what a server does under load. Lighthouse measures what one page load costs in a real
|
|
4
|
+
// browser - Largest Contentful Paint, layout shift, blocking time - which is nearer to what a person
|
|
5
|
+
// actually waits for, and is what most teams already run. Tewip could read neither until now.
|
|
6
|
+
//
|
|
7
|
+
// The same discipline as everywhere else applies, and Lighthouse needs it more than k6 does. Four
|
|
8
|
+
// real Lighthouse runs of an unchanged page, measured here, gave LCP of 922, 912, 936 and 916ms: a
|
|
9
|
+
// 2.6% spread from nothing happening. Lighthouse's own score moved between runs too. So the
|
|
10
|
+
// comparison is the same one k6 gets - outside the baseline range by more than its own width, with
|
|
11
|
+
// three baselines required before anything is called at all.
|
|
12
|
+
//
|
|
13
|
+
// One thing is deliberately not invented. "Is 2112ms bad?" has a published answer, and it is
|
|
14
|
+
// Google's, not Tewip's: Lighthouse scores every metric 0 to 1 on its own curve and renders the
|
|
15
|
+
// bands at 0.9 and 0.5. Tewip reads that score rather than hard-coding thresholds it remembered,
|
|
16
|
+
// so the word "poor" in a report is Lighthouse's judgement and is traceable to it.
|
|
17
|
+
import { readFileSync } from 'node:fs';
|
|
18
|
+
/** The audits worth comparing between runs: what the page load cost, not advice about it. */
|
|
19
|
+
const MEASURED = new Set([
|
|
20
|
+
'largest-contentful-paint',
|
|
21
|
+
'first-contentful-paint',
|
|
22
|
+
'total-blocking-time',
|
|
23
|
+
'cumulative-layout-shift',
|
|
24
|
+
'speed-index',
|
|
25
|
+
'interaction-to-next-paint',
|
|
26
|
+
'server-response-time',
|
|
27
|
+
'time-to-first-byte',
|
|
28
|
+
]);
|
|
29
|
+
/** Lighthouse renders green at 0.9 and orange at 0.5. The bands are its own, and so is the word. */
|
|
30
|
+
function rating(score) {
|
|
31
|
+
if (typeof score !== 'number' || !Number.isFinite(score))
|
|
32
|
+
return null;
|
|
33
|
+
if (score >= 0.9)
|
|
34
|
+
return 'good';
|
|
35
|
+
if (score >= 0.5)
|
|
36
|
+
return 'needs-improvement';
|
|
37
|
+
return 'poor';
|
|
38
|
+
}
|
|
39
|
+
const text = (v) => (typeof v === 'string' ? v : '');
|
|
40
|
+
const num = (v) => (typeof v === 'number' && Number.isFinite(v) ? v : 0);
|
|
41
|
+
function looksLikeLighthouse(v) {
|
|
42
|
+
if (typeof v !== 'object' || v === null)
|
|
43
|
+
return false;
|
|
44
|
+
const o = v;
|
|
45
|
+
return typeof o.lighthouseVersion === 'string' && typeof o.audits === 'object' && o.audits !== null;
|
|
46
|
+
}
|
|
47
|
+
/** The nodes a failing accessibility audit points at. */
|
|
48
|
+
function failingNodes(audit) {
|
|
49
|
+
const details = (audit.details ?? {});
|
|
50
|
+
const items = Array.isArray(details.items) ? details.items : [];
|
|
51
|
+
const out = [];
|
|
52
|
+
for (const item of items) {
|
|
53
|
+
const node = (item?.node ?? {});
|
|
54
|
+
out.push({
|
|
55
|
+
selector: text(node.selector),
|
|
56
|
+
snippet: text(node.snippet).replace(/\s+/g, ' ').slice(0, 160),
|
|
57
|
+
explanation: text(node.explanation)
|
|
58
|
+
.split(/\r?\n/)
|
|
59
|
+
.map((l) => l.trim())
|
|
60
|
+
.filter((l) => l !== '' && !/^Fix (any|all) of the following:$/.test(l))
|
|
61
|
+
.join('; '),
|
|
62
|
+
});
|
|
63
|
+
}
|
|
64
|
+
return out;
|
|
65
|
+
}
|
|
66
|
+
export function readLighthouse(path) {
|
|
67
|
+
let parsed;
|
|
68
|
+
try {
|
|
69
|
+
parsed = JSON.parse(readFileSync(path, 'utf8'));
|
|
70
|
+
}
|
|
71
|
+
catch {
|
|
72
|
+
throw new Error(`${path} is not JSON. Tewip reads the JSON Lighthouse writes with --output=json.`);
|
|
73
|
+
}
|
|
74
|
+
if (!looksLikeLighthouse(parsed)) {
|
|
75
|
+
throw new Error(`${path} is JSON, but not a Lighthouse report: Tewip looked for "lighthouseVersion" and an "audits" object. ` +
|
|
76
|
+
'Write one with: lighthouse <url> --output=json --output-path=report.json');
|
|
77
|
+
}
|
|
78
|
+
const o = parsed;
|
|
79
|
+
// A report can carry a runtime error and still be valid JSON with empty audits. Reporting "no
|
|
80
|
+
// regression" from a run that never loaded the page would be the worst kind of wrong.
|
|
81
|
+
const runtimeError = (o.runtimeError ?? null);
|
|
82
|
+
if (runtimeError && text(runtimeError.code) !== 'NO_ERROR' && text(runtimeError.message) !== '') {
|
|
83
|
+
throw new Error(`that Lighthouse run did not complete: ${text(runtimeError.message)}`);
|
|
84
|
+
}
|
|
85
|
+
const audits = (o.audits ?? {});
|
|
86
|
+
const categories = (o.categories ?? {});
|
|
87
|
+
const metrics = {};
|
|
88
|
+
for (const [id, audit] of Object.entries(audits)) {
|
|
89
|
+
if (!MEASURED.has(id) || typeof audit.numericValue !== 'number')
|
|
90
|
+
continue;
|
|
91
|
+
metrics[id] = {
|
|
92
|
+
value: num(audit.numericValue),
|
|
93
|
+
unit: audit.numericUnit === 'millisecond' ? 'ms' : 'unitless',
|
|
94
|
+
rating: rating(audit.score),
|
|
95
|
+
};
|
|
96
|
+
}
|
|
97
|
+
// The category score itself is worth tracking: it is the number teams quote at each other.
|
|
98
|
+
for (const [name, category] of Object.entries(categories)) {
|
|
99
|
+
if (typeof category?.score === 'number') {
|
|
100
|
+
metrics[`${name}-score`] = { value: category.score, unit: 'score', rating: rating(category.score) };
|
|
101
|
+
}
|
|
102
|
+
}
|
|
103
|
+
const accessibilityFailures = [];
|
|
104
|
+
const manual = [];
|
|
105
|
+
const accessibilityAuditIds = [];
|
|
106
|
+
const a11yRefs = (categories.accessibility?.auditRefs ?? []);
|
|
107
|
+
for (const ref of a11yRefs) {
|
|
108
|
+
const audit = audits[text(ref.id)];
|
|
109
|
+
if (!audit)
|
|
110
|
+
continue;
|
|
111
|
+
const id = text(audit.id);
|
|
112
|
+
// Recorded before the outcome is looked at: a rule that passed, or did not apply to this page,
|
|
113
|
+
// has still been checked, and that is what coverage means.
|
|
114
|
+
if (id !== '')
|
|
115
|
+
accessibilityAuditIds.push(id);
|
|
116
|
+
if (audit.scoreDisplayMode === 'manual') {
|
|
117
|
+
manual.push({ id, title: text(audit.title) });
|
|
118
|
+
continue;
|
|
119
|
+
}
|
|
120
|
+
if (typeof audit.score === 'number' && audit.score < 1) {
|
|
121
|
+
const nodes = failingNodes(audit);
|
|
122
|
+
if (nodes.length === 0)
|
|
123
|
+
accessibilityFailures.push({ id, title: text(audit.title), selector: '', snippet: '', explanation: '' });
|
|
124
|
+
for (const node of nodes)
|
|
125
|
+
accessibilityFailures.push({ id, title: text(audit.title), ...node });
|
|
126
|
+
}
|
|
127
|
+
}
|
|
128
|
+
return {
|
|
129
|
+
run: {
|
|
130
|
+
source: 'lighthouse',
|
|
131
|
+
at: text(o.fetchTime) || new Date().toISOString(),
|
|
132
|
+
metrics,
|
|
133
|
+
// Lighthouse has no user-written thresholds the way a k6 script does. That is an absence of
|
|
134
|
+
// the feature, not an export that forgot to record them, so an empty list is the honest value.
|
|
135
|
+
thresholds: [],
|
|
136
|
+
volume: { kind: 'page load', n: 1 },
|
|
137
|
+
},
|
|
138
|
+
categories: Object.fromEntries(Object.entries(categories).map(([k, v]) => [k, num(v?.score)])),
|
|
139
|
+
accessibilityFailures,
|
|
140
|
+
manual,
|
|
141
|
+
accessibilityAuditIds,
|
|
142
|
+
url: text(o.finalDisplayedUrl) || text(o.finalUrl) || text(o.requestedUrl),
|
|
143
|
+
lighthouseVersion: text(o.lighthouseVersion),
|
|
144
|
+
};
|
|
145
|
+
}
|
|
146
|
+
/** True of a Lighthouse report, for a command that accepts either tool's output. */
|
|
147
|
+
export function isLighthouseFile(path) {
|
|
148
|
+
try {
|
|
149
|
+
const parsed = JSON.parse(readFileSync(path, 'utf8'));
|
|
150
|
+
return looksLikeLighthouse(parsed);
|
|
151
|
+
}
|
|
152
|
+
catch {
|
|
153
|
+
return false;
|
|
154
|
+
}
|
|
155
|
+
}
|
|
@@ -1,111 +1,156 @@
|
|
|
1
|
-
// Phase 30: `tewip perf` - is this change slower, and does it matter?
|
|
1
|
+
// Phase 30, extended in Phase 31: `tewip perf` - is this change slower, and does it matter?
|
|
2
2
|
//
|
|
3
|
-
//
|
|
3
|
+
// Reads a k6 run (what a server does under load) or a Lighthouse report (what one page load costs in
|
|
4
|
+
// a browser), whichever it is handed, and answers two questions separately because they fail
|
|
5
|
+
// separately:
|
|
4
6
|
//
|
|
5
|
-
// 1. **
|
|
6
|
-
//
|
|
7
|
-
//
|
|
8
|
-
//
|
|
9
|
-
//
|
|
10
|
-
//
|
|
7
|
+
// 1. **Is anything in a bad state right now?** From the team's own k6 thresholds, or from
|
|
8
|
+
// Lighthouse's published scoring bands. Never from a budget Tewip invented.
|
|
9
|
+
// 2. **Is it slower than it was?** Only from repeated baselines, and refused below three, because
|
|
10
|
+
// one run against one run is not evidence. Four real k6 runs of unchanged code gave p95 of
|
|
11
|
+
// 60.8-62.8ms; four real Lighthouse runs of an unchanged page gave LCP of 912-936ms. Both move
|
|
12
|
+
// on their own, and a tool that calls that a regression is muted within a week.
|
|
11
13
|
//
|
|
12
|
-
// It never
|
|
13
|
-
//
|
|
14
|
-
//
|
|
14
|
+
// It never predicts a duration, a capacity or a user impact: it cannot see the production hardware,
|
|
15
|
+
// the real data volume or the real traffic, and an invented number for any of those is worse than
|
|
16
|
+
// silence.
|
|
15
17
|
import { mkdirSync, writeFileSync } from 'node:fs';
|
|
16
18
|
import { dirname, resolve } from 'node:path';
|
|
17
|
-
import {
|
|
18
|
-
import {
|
|
19
|
-
import {
|
|
20
|
-
|
|
21
|
-
|
|
19
|
+
import { readK6Summary, requestCount } from './k6.js';
|
|
20
|
+
import { isLighthouseFile, readLighthouse } from './lighthouse.js';
|
|
21
|
+
import { fromK6, metricsInOrder, show } from './run.js';
|
|
22
|
+
import { compareMetric, ENOUGH_BASELINES } from './compare.js';
|
|
23
|
+
import { baselineFor, fromSameTool, knownLabels, recordRun, storedRuns } from './history.js';
|
|
24
|
+
/** Enough requests for a percentile to mean anything. Below this, p95 is three data points. */
|
|
25
|
+
export const ENOUGH_REQUESTS = 100;
|
|
26
|
+
export function readPerfFile(path) {
|
|
27
|
+
if (isLighthouseFile(path)) {
|
|
28
|
+
const report = readLighthouse(path);
|
|
29
|
+
return { run: report.run, manual: report.manual, url: report.url };
|
|
30
|
+
}
|
|
31
|
+
const k6 = readK6Summary(path);
|
|
32
|
+
return { run: fromK6(k6, requestCount(k6)), manual: [], url: '' };
|
|
33
|
+
}
|
|
34
|
+
export function renderPerf(run, comparisons, label, baselineCount, recorded, manual = []) {
|
|
22
35
|
const lines = ['# Tewip performance check', ''];
|
|
23
36
|
const breaches = (run.thresholds ?? []).filter((t) => t.breached);
|
|
37
|
+
const poor = Object.entries(run.metrics).filter(([, m]) => m.rating === 'poor');
|
|
24
38
|
const slower = comparisons.filter((c) => c.direction === 'slower');
|
|
25
39
|
const faster = comparisons.filter((c) => c.direction === 'faster');
|
|
26
40
|
const unsure = comparisons.filter((c) => c.direction === 'cannot-say');
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
41
|
+
const tool = run.source === 'k6' ? 'k6' : 'Lighthouse';
|
|
42
|
+
lines.push(`Read a ${tool} run of **${run.volume.n} ${run.volume.kind}${run.volume.n === 1 ? '' : 's'}**, compared against **${baselineCount}** recorded ` +
|
|
43
|
+
`${tool} baseline ${baselineCount === 1 ? 'run' : 'runs'} for \`${label}\`.`, '');
|
|
44
|
+
if (breaches.length > 0)
|
|
30
45
|
lines.push(`**${breaches.length} of your own thresholds ${breaches.length === 1 ? 'was' : 'were'} breached.**`, '');
|
|
31
|
-
|
|
46
|
+
if (poor.length > 0)
|
|
47
|
+
lines.push(`**${poor.length} ${poor.length === 1 ? 'metric is' : 'metrics are'} in ${tool}'s worst band.**`, '');
|
|
32
48
|
if (slower.length > 0)
|
|
33
49
|
lines.push(`**${slower.length} ${slower.length === 1 ? 'metric is' : 'metrics are'} slower than the baseline runs can explain.**`, '');
|
|
34
|
-
if (breaches.length === 0 && slower.length === 0) {
|
|
50
|
+
if (breaches.length === 0 && poor.length === 0 && slower.length === 0) {
|
|
35
51
|
lines.push(run.thresholds === null
|
|
36
52
|
? '**No regression this baseline can detect.** Your thresholds are not in this export, so nothing is said about them.'
|
|
37
|
-
: '**
|
|
53
|
+
: '**Nothing in a bad state, and no regression this baseline can detect.**', '');
|
|
38
54
|
}
|
|
39
|
-
lines.push('##
|
|
40
|
-
if (run.
|
|
41
|
-
|
|
42
|
-
'
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
55
|
+
lines.push('## Is anything in a bad state right now?', '');
|
|
56
|
+
if (run.source === 'k6') {
|
|
57
|
+
if (run.thresholds === null) {
|
|
58
|
+
lines.push('**Cannot say.** This summary came from `--new-machine-readable-summary`, and that export records no threshold ' +
|
|
59
|
+
'results at all - the word appears in it only inside the copy of your script. Tewip will not report "nothing was ' +
|
|
60
|
+
'breached" from a file that does not say. Re-run with plain `--summary-export` to get the answer.', '');
|
|
61
|
+
}
|
|
62
|
+
else if (run.thresholds.length === 0) {
|
|
63
|
+
lines.push('This run has no thresholds. Add them to your k6 script and the team decides what "fast enough" means, rather than Tewip.', '');
|
|
64
|
+
}
|
|
65
|
+
else {
|
|
66
|
+
for (const t of run.thresholds)
|
|
67
|
+
lines.push(`- ${t.breached ? '**breached**' : 'held'} — \`${t.metric}\`: \`${t.expression}\``);
|
|
68
|
+
lines.push('');
|
|
69
|
+
}
|
|
47
70
|
}
|
|
48
71
|
else {
|
|
49
|
-
|
|
50
|
-
|
|
72
|
+
// The bands are Lighthouse's own, computed by Lighthouse. Tewip is relaying a judgement with a
|
|
73
|
+
// published basis, not applying numbers it chose.
|
|
74
|
+
lines.push("Rated by Lighthouse's own scoring, not by any threshold Tewip invented:", '');
|
|
75
|
+
for (const name of metricsInOrder(run)) {
|
|
76
|
+
const m = run.metrics[name];
|
|
77
|
+
if (m.rating === null)
|
|
78
|
+
continue;
|
|
79
|
+
const mark = m.rating === 'poor' ? '**poor**' : m.rating === 'needs-improvement' ? 'needs improvement' : 'good';
|
|
80
|
+
lines.push(`- ${mark} — \`${name}\`: ${show(m)}`);
|
|
51
81
|
}
|
|
52
82
|
lines.push('');
|
|
53
83
|
}
|
|
54
84
|
lines.push('## Is it slower than it was?', '');
|
|
55
85
|
if (baselineCount < ENOUGH_BASELINES) {
|
|
56
|
-
lines.push(`**Cannot say.** There ${baselineCount === 1 ? 'is' : 'are'} ${baselineCount} recorded ${baselineCount === 1 ? 'run' : 'runs'}
|
|
57
|
-
`and Tewip needs ${ENOUGH_BASELINES} before it will call anything a regression. Performance numbers move
|
|
58
|
-
'runs, and without several of them there is no way to tell a change from the machine having a busy moment. ' +
|
|
59
|
-
|
|
86
|
+
lines.push(`**Cannot say.** There ${baselineCount === 1 ? 'is' : 'are'} ${baselineCount} recorded ${tool} ${baselineCount === 1 ? 'run' : 'runs'} ` +
|
|
87
|
+
`for \`${label}\`, and Tewip needs ${ENOUGH_BASELINES} before it will call anything a regression. Performance numbers move ` +
|
|
88
|
+
'between identical runs, and without several of them there is no way to tell a change from the machine having a busy moment. ' +
|
|
89
|
+
'Record baselines on your main branch with `tewip perf <file> --record`.', '');
|
|
60
90
|
}
|
|
61
91
|
for (const c of [...slower, ...faster, ...unsure]) {
|
|
62
92
|
const verdict = c.direction === 'slower' ? '**slower**' : c.direction === 'faster' ? 'faster' : 'cannot say';
|
|
63
|
-
|
|
64
|
-
|
|
93
|
+
const unit = run.metrics[c.metric]?.unit ?? 'ms';
|
|
94
|
+
const u = (n) => (unit === 'ms' ? `${n}ms` : `${n}`);
|
|
95
|
+
lines.push(`- \`${c.metric}\` ${u(c.now)} — ${verdict}`);
|
|
96
|
+
lines.push(` - baseline ${u(c.baselineLow)}–${u(c.baselineHigh)} over ${c.baselineRuns} runs, middle ${u(c.before)}`);
|
|
65
97
|
lines.push(` - ${c.why}`);
|
|
66
98
|
}
|
|
67
99
|
lines.push('');
|
|
68
|
-
if (
|
|
69
|
-
lines.push(`**${
|
|
100
|
+
if (run.source === 'k6' && run.volume.n < ENOUGH_REQUESTS) {
|
|
101
|
+
lines.push(`**${run.volume.n} requests is thin.** A p95 from this few is a handful of data points and will move on its own. ` +
|
|
70
102
|
'Treat everything above as indicative and lengthen the run before deciding anything.', '');
|
|
71
103
|
}
|
|
72
104
|
lines.push('## What this does not tell you', '');
|
|
73
105
|
lines.push('- **Nothing about production.** This is the hardware the run happened on, with that data volume and that ' +
|
|
74
|
-
'concurrency. Tewip cannot see your real machines, your real data or your real traffic, and will not guess.', '- **No duration is predicted.** How long a real user waits depends on network, device and cache, none of which are here.', '- **Only what the
|
|
106
|
+
'concurrency. Tewip cannot see your real machines, your real data or your real traffic, and will not guess.', '- **No duration is predicted.** How long a real user waits depends on network, device and cache, none of which are here.', '- **Only what the run exercised.** A page or endpoint it never touched is not covered, however green this looks.', '- **Not capacity.** This says whether it changed, not how many users it would hold.');
|
|
107
|
+
if (run.source === 'lighthouse') {
|
|
108
|
+
lines.push(`- **One page load.** k6 reports a percentile over many requests; this is a single load of one page, and a single ` +
|
|
109
|
+
'load is the noisiest measurement either tool offers.');
|
|
110
|
+
if (manual.length > 0) {
|
|
111
|
+
lines.push(`- **${manual.length} accessibility checks Lighthouse refuses to score** and hands to a person - ` +
|
|
112
|
+
`${manual.slice(0, 3).map((m) => `\`${m.id}\``).join(', ')} and others. They are not failures and not passes. ` +
|
|
113
|
+
'Run `tewip access` for the accessibility side.');
|
|
114
|
+
}
|
|
115
|
+
}
|
|
116
|
+
lines.push('');
|
|
75
117
|
if (recorded)
|
|
76
118
|
lines.push(`This run was added to the baseline for \`${label}\`.`, '');
|
|
77
119
|
else
|
|
78
|
-
lines.push(
|
|
120
|
+
lines.push('This run was **not** recorded. Add `--record` on your main branch to make it part of the baseline; a regression recorded by accident becomes next week\'s normal.', '');
|
|
79
121
|
return lines.join('\n');
|
|
80
122
|
}
|
|
81
123
|
export function runPerf(cmd) {
|
|
82
124
|
const root = resolve(cmd.root);
|
|
83
125
|
const label = cmd.label ?? 'default';
|
|
84
|
-
const run =
|
|
85
|
-
const
|
|
86
|
-
const history =
|
|
87
|
-
if (
|
|
126
|
+
const { run, manual } = readPerfFile(resolve(root, cmd.summary));
|
|
127
|
+
const all = storedRuns(root, label);
|
|
128
|
+
const history = fromSameTool(all, run.source);
|
|
129
|
+
if (all.length === 0) {
|
|
88
130
|
const known = knownLabels(root);
|
|
89
|
-
// A typo in --label would otherwise look exactly like "no history yet", and the team would
|
|
90
|
-
// record a second baseline under the wrong name without noticing.
|
|
91
131
|
if (known.length > 0 && !known.includes(label)) {
|
|
92
132
|
console.error(`[tewip] no recorded runs for label "${label}". This project has: ${known.join(', ')}.`);
|
|
93
133
|
}
|
|
94
134
|
}
|
|
95
|
-
const comparisons =
|
|
96
|
-
.map((metric) => compareMetric(metric, run.
|
|
97
|
-
// A metric that is zero now and was zero every time before says nothing
|
|
98
|
-
//
|
|
135
|
+
const comparisons = metricsInOrder(run)
|
|
136
|
+
.map((metric) => compareMetric(metric, run.metrics[metric].value, baselineFor(history, metric)))
|
|
137
|
+
// A metric that is zero now and was zero every time before says nothing, and printing
|
|
138
|
+
// "0ms is within 0-0ms" three times buries the real lines.
|
|
99
139
|
.filter((c) => c.now > 0 || c.baselineHigh > 0);
|
|
100
140
|
const recorded = cmd.record === true;
|
|
101
141
|
if (recorded)
|
|
102
|
-
recordRun(root, label, run,
|
|
142
|
+
recordRun(root, label, run, cmd.gitSha);
|
|
103
143
|
const breached = (run.thresholds ?? []).some((t) => t.breached);
|
|
144
|
+
const poor = Object.values(run.metrics).some((m) => m.rating === 'poor');
|
|
104
145
|
const regressed = comparisons.some((c) => c.direction === 'slower');
|
|
105
|
-
const report = renderPerf(run, comparisons,
|
|
146
|
+
const report = renderPerf(run, comparisons, label, history.length, recorded, manual);
|
|
106
147
|
console.log(report);
|
|
107
|
-
if (cmd.gate && (breached || regressed)) {
|
|
108
|
-
const why = [
|
|
148
|
+
if (cmd.gate && (breached || regressed || poor)) {
|
|
149
|
+
const why = [
|
|
150
|
+
breached ? 'a threshold you wrote was breached' : '',
|
|
151
|
+
poor ? "something is in Lighthouse's worst band" : '',
|
|
152
|
+
regressed ? 'something is slower than the baselines explain' : '',
|
|
153
|
+
]
|
|
109
154
|
.filter(Boolean)
|
|
110
155
|
.join(', and ');
|
|
111
156
|
console.error(`${String.fromCharCode(10)}Performance gate failed: ${why}.`);
|
|
@@ -116,5 +161,5 @@ export function runPerf(cmd) {
|
|
|
116
161
|
mkdirSync(dirname(path), { recursive: true });
|
|
117
162
|
writeFileSync(path, report, 'utf8');
|
|
118
163
|
}
|
|
119
|
-
return { run, comparisons,
|
|
164
|
+
return { run, comparisons, regressed, breached: breached || poor, report };
|
|
120
165
|
}
|
|
@@ -0,0 +1,39 @@
|
|
|
1
|
+
/** k6's percentiles, reduced to the one number a comparison needs: p95. */
|
|
2
|
+
export function fromK6(run, requests) {
|
|
3
|
+
const metrics = {};
|
|
4
|
+
for (const [name, m] of Object.entries(run.timings)) {
|
|
5
|
+
// Tagged duplicates (`http_req_duration{expected_response:true}`) are the same measurement
|
|
6
|
+
// sliced, and comparing both says the same thing twice.
|
|
7
|
+
if (name.includes('{'))
|
|
8
|
+
continue;
|
|
9
|
+
// k6 publishes no bands. It reports numbers and leaves "fast enough" to the team's thresholds,
|
|
10
|
+
// which is the right division and the reason rating is null here.
|
|
11
|
+
metrics[name] = { value: m.p95, unit: 'ms', rating: null };
|
|
12
|
+
}
|
|
13
|
+
return { source: 'k6', at: run.at, metrics, thresholds: run.thresholds, volume: { kind: 'request', n: requests } };
|
|
14
|
+
}
|
|
15
|
+
/** Metrics worth putting in front of a person, most important first. */
|
|
16
|
+
const ORDER = [
|
|
17
|
+
'largest-contentful-paint',
|
|
18
|
+
'cumulative-layout-shift',
|
|
19
|
+
'total-blocking-time',
|
|
20
|
+
'interaction-to-next-paint',
|
|
21
|
+
'first-contentful-paint',
|
|
22
|
+
'speed-index',
|
|
23
|
+
'server-response-time',
|
|
24
|
+
'http_req_duration',
|
|
25
|
+
];
|
|
26
|
+
export function metricsInOrder(run) {
|
|
27
|
+
const names = Object.keys(run.metrics);
|
|
28
|
+
const known = ORDER.filter((m) => names.includes(m));
|
|
29
|
+
const rest = names.filter((m) => !ORDER.includes(m)).sort();
|
|
30
|
+
return [...known, ...rest];
|
|
31
|
+
}
|
|
32
|
+
/** How a metric is written in a report: 2112ms, 0.08, 97. */
|
|
33
|
+
export function show(metric) {
|
|
34
|
+
if (metric.unit === 'ms')
|
|
35
|
+
return `${Math.round(metric.value)}ms`;
|
|
36
|
+
if (metric.unit === 'score')
|
|
37
|
+
return `${Math.round(metric.value * 100)}`;
|
|
38
|
+
return `${Math.round(metric.value * 1000) / 1000}`;
|
|
39
|
+
}
|
package/dist/mcp/tools-prove.js
CHANGED
|
@@ -109,18 +109,18 @@ export function registerProofTools(server, { answer, readOnly, project }) {
|
|
|
109
109
|
throw new Error(`project must be an existing absolute path, got "${root}"`);
|
|
110
110
|
// Never records: an assistant adding runs to a team's baseline unasked is how a regression
|
|
111
111
|
// silently becomes the new normal. Recording stays a deliberate act on the command line.
|
|
112
|
-
const {
|
|
113
|
-
const {
|
|
114
|
-
const {
|
|
115
|
-
const {
|
|
112
|
+
const { compareMetric } = await import('../engine/perf/compare.js');
|
|
113
|
+
const { baselineFor, fromSameTool, storedRuns } = await import('../engine/perf/history.js');
|
|
114
|
+
const { readPerfFile, renderPerf } = await import('../engine/perf/perf-command.js');
|
|
115
|
+
const { metricsInOrder } = await import('../engine/perf/run.js');
|
|
116
116
|
const { isAbsolute: abs, resolve } = await import('node:path');
|
|
117
|
-
const run =
|
|
117
|
+
const { run, manual } = readPerfFile(abs(summary) ? summary : resolve(root, summary));
|
|
118
118
|
const name = label ?? 'default';
|
|
119
|
-
const history = storedRuns(root, name);
|
|
120
|
-
const comparisons =
|
|
121
|
-
.map((m) => compareMetric(m, run.
|
|
119
|
+
const history = fromSameTool(storedRuns(root, name), run.source);
|
|
120
|
+
const comparisons = metricsInOrder(run)
|
|
121
|
+
.map((m) => compareMetric(m, run.metrics[m].value, baselineFor(history, m)))
|
|
122
122
|
.filter((c) => c.now > 0 || c.baselineHigh > 0);
|
|
123
|
-
return renderPerf(run, comparisons,
|
|
123
|
+
return renderPerf(run, comparisons, name, history.length, false, manual);
|
|
124
124
|
}));
|
|
125
125
|
server.registerTool('accessibility_barriers', {
|
|
126
126
|
title: 'Can everybody use this interface?',
|
|
@@ -131,20 +131,19 @@ export function registerProofTools(server, { answer, readOnly, project }) {
|
|
|
131
131
|
axeReport: z
|
|
132
132
|
.string()
|
|
133
133
|
.optional()
|
|
134
|
-
.describe('The JSON from an axe run, if the project has
|
|
134
|
+
.describe('The JSON from an axe run OR a Lighthouse report, if the project has either - Lighthouse runs axe rules, so both work. It settles contrast, which reading markup cannot, and its findings are triaged rather than relayed: rules that map to no WCAG criterion are reported as advice, and what the tool could not determine becomes a question.'),
|
|
135
135
|
},
|
|
136
136
|
annotations: readOnly,
|
|
137
137
|
}, ({ project: root, path, axeReport }) => answer(async () => {
|
|
138
138
|
if (!isAbsolute(root) || !existsSync(root))
|
|
139
139
|
throw new Error(`project must be an existing absolute path, got "${root}"`);
|
|
140
140
|
// The pure functions, not runAccess: that one prints, and anything on stdout breaks JSON-RPC.
|
|
141
|
-
const { findBarriers, keyOf, readAccepted, renderAccess } = await import('../engine/access/access-command.js');
|
|
141
|
+
const { findBarriers, keyOf, readAccepted, readBrowserRun, renderAccess } = await import('../engine/access/access-command.js');
|
|
142
142
|
const { merge, pagesIn } = await import('../engine/access/merge.js');
|
|
143
|
-
const { readAxeReport } = await import('../engine/access/reports.js');
|
|
144
143
|
const { findings, filesRead } = findBarriers(root, path);
|
|
145
144
|
const judged = readAccepted(root);
|
|
146
145
|
const open = findings.filter((f) => !judged.has(keyOf(f)));
|
|
147
|
-
const run = axeReport ?
|
|
146
|
+
const run = axeReport ? readBrowserRun(isAbsolute(axeReport) ? axeReport : `${root}/${axeReport}`) : undefined;
|
|
148
147
|
const graded = run ? merge(open, run, root) : undefined;
|
|
149
148
|
const kept = graded ? graded.kept : open;
|
|
150
149
|
return renderAccess({
|
|
@@ -161,6 +160,7 @@ export function registerProofTools(server, { answer, readOnly, project }) {
|
|
|
161
160
|
superseded: graded.superseded,
|
|
162
161
|
pages: pagesIn(run),
|
|
163
162
|
rulesRun: run.rulesRun,
|
|
163
|
+
tool: run.source,
|
|
164
164
|
},
|
|
165
165
|
}
|
|
166
166
|
: {}),
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "tewip-ai",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.39.0",
|
|
4
4
|
"mcpName": "io.github.hmamut39/tewip",
|
|
5
5
|
"description": "Tewip finds why each failing test failed - real bug, flaky test, environment, test data or changed selector - for any stack, from the command line, CI, or your editor.",
|
|
6
6
|
"keywords": [
|