tolkira 0.2.0 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/DESIGN.md +131 -9
- package/GLOSSARY.md +201 -1
- package/README.md +413 -29
- package/SECURITY.md +3 -0
- package/dist/attachment.d.ts +33 -0
- package/dist/attempt-diagnostics.d.ts +2 -0
- package/dist/ci.d.ts +12 -0
- package/dist/cli.js +6881 -2691
- package/dist/cli.js.map +4 -4
- package/dist/compat/cucumber-html-reporter.cjs +12 -0
- package/dist/compat/cucumber-html-reporter.d.cts +10 -0
- package/dist/compat/cucumber-html-reporter.js +263 -26
- package/dist/compat/cucumber-html-reporter.js.map +4 -4
- package/dist/compat/options.d.ts +4 -0
- package/dist/email.d.ts +18 -0
- package/dist/history.d.ts +41 -0
- package/dist/i18n/catalogs.d.ts +7 -0
- package/dist/index.d.ts +15 -7
- package/dist/index.js +5301 -1709
- package/dist/index.js.map +4 -4
- package/dist/input/background.d.ts +4 -0
- package/dist/input/detect.d.ts +30 -0
- package/dist/input/load.d.ts +3 -0
- package/dist/input/playwright.d.ts +14 -0
- package/dist/options.d.ts +18 -0
- package/dist/output.d.ts +9 -1
- package/dist/pdf.d.ts +9 -0
- package/dist/render/ansi.d.ts +2 -0
- package/dist/render/attachments.d.ts +3 -0
- package/dist/render/ci.d.ts +4 -0
- package/dist/render/client-bundle.d.ts +1 -0
- package/dist/render/client-i18n.d.ts +7 -0
- package/dist/render/details.d.ts +5 -0
- package/dist/render/dom-contract.d.ts +42 -0
- package/dist/render/format.d.ts +34 -0
- package/dist/render/html.d.ts +2 -9
- package/dist/render/i18n.d.ts +30 -0
- package/dist/render/markdown.d.ts +2 -0
- package/dist/render/matrix.d.ts +3 -0
- package/dist/render/print-script.d.ts +1 -0
- package/dist/render/report-client.d.ts +1 -0
- package/dist/render/report.d.ts +4 -0
- package/dist/render/search.d.ts +21 -0
- package/dist/render/trends.d.ts +7 -0
- package/dist/render/triage.d.ts +5 -0
- package/dist/render/types.d.ts +26 -0
- package/dist/report/index-run.d.ts +69 -0
- package/dist/report/model.d.ts +96 -6
- package/dist/report/status.d.ts +12 -0
- package/dist/stream.d.ts +15 -0
- package/dist/summary.d.ts +17 -2
- package/dist/text.d.ts +15 -0
- package/dist/warnings.d.ts +6 -0
- package/docs/AGENT-INTERFACE.md +504 -33
- package/docs/CI.md +185 -0
- package/docs/COMPATIBILITY.md +121 -16
- package/docs/CUCUMBER-COMPATIBILITY.md +232 -15
- package/docs/EMAIL.md +273 -0
- package/docs/I18N.md +241 -0
- package/docs/JUNIT-XML.md +46 -4
- package/docs/LIMITATIONS.md +21 -0
- package/docs/PDF.md +255 -0
- package/llms.txt +63 -0
- package/package.json +46 -8
package/docs/AGENT-INTERFACE.md
CHANGED
|
@@ -1,12 +1,17 @@
|
|
|
1
1
|
# Tolkira for agents
|
|
2
2
|
|
|
3
|
-
Tolkira produces
|
|
3
|
+
Tolkira produces five things from the same input:
|
|
4
4
|
|
|
5
5
|
- a self-contained HTML report for humans;
|
|
6
|
-
- a stable JSON summary for programs and AI agents
|
|
6
|
+
- a stable JSON summary for programs and AI agents;
|
|
7
|
+
- deterministic CI text: GitHub Actions annotations or a Markdown summary;
|
|
8
|
+
- an opt-in HTML email summary for a status mail or chat message;
|
|
9
|
+
- an opt-in PDF print archive when the optional Playwright Chromium engine is
|
|
10
|
+
installed.
|
|
7
11
|
|
|
8
|
-
The HTML report is the artifact. The JSON summary is the interface.
|
|
9
|
-
the HTML to decide whether
|
|
12
|
+
The HTML report is the artifact. The JSON summary is the interface. The CI text
|
|
13
|
+
is a presentation of that same summary. Do not parse the HTML to decide whether
|
|
14
|
+
a test run passed.
|
|
10
15
|
|
|
11
16
|
## CLI
|
|
12
17
|
|
|
@@ -34,45 +39,356 @@ stderr. Exit codes are about report generation, not test status:
|
|
|
34
39
|
Always read `status` from the JSON summary for the test outcome. A failed test
|
|
35
40
|
suite is exit code 0 by design.
|
|
36
41
|
|
|
42
|
+
## CI output
|
|
43
|
+
|
|
44
|
+
Format the same summary for CI instead of, or alongside, HTML and JSON:
|
|
45
|
+
|
|
46
|
+
```shell
|
|
47
|
+
# GitHub Actions workflow commands on stdout; no HTML required
|
|
48
|
+
tolkira test-results/ --ci github
|
|
49
|
+
|
|
50
|
+
# Markdown on stdout, or in a standalone file
|
|
51
|
+
tolkira test-results/ --ci markdown
|
|
52
|
+
tolkira test-results/ --out report.html --ci markdown --ci-out summary.md
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
- `--ci github` writes `::error` and `::notice` workflow commands. It needs no
|
|
56
|
+
`--out` and rejects `--ci-out`: GitHub only reads workflow commands from
|
|
57
|
+
stdout.
|
|
58
|
+
- `--ci markdown` writes deterministic Markdown with complete counts, a failed
|
|
59
|
+
list and skipped names. `--title` sets the heading.
|
|
60
|
+
- Without `--ci-out`, CI text is the only stdout and the human confirmation is
|
|
61
|
+
suppressed. With `--ci-out`, the file receives the CI text and stdout keeps
|
|
62
|
+
its normal confirmation; with `--json`, stdout stays JSON only.
|
|
63
|
+
- `--ci-out` requires `--ci markdown`, is written atomically as a standalone
|
|
64
|
+
file, and must not overwrite an input, the logo, `--out` or the `--email`
|
|
65
|
+
file. To add to
|
|
66
|
+
`$GITHUB_STEP_SUMMARY`, append stdout with `>>`; do not point `--ci-out` at a
|
|
67
|
+
shared file.
|
|
68
|
+
- Exit codes are unchanged. A failed test run still exits 0; invalid
|
|
69
|
+
combinations exit 2 before any file is written.
|
|
70
|
+
- Messages are copied verbatim and are not redacted. See [`CI.md`](CI.md).
|
|
71
|
+
|
|
72
|
+
GitHub mode caps at 50 error and 20 notice annotations and states the
|
|
73
|
+
remainder; the final count notice keeps the exact totals. A failed run status
|
|
74
|
+
without a scenario-level error produces one run-level error annotation.
|
|
75
|
+
Markdown caps the failed list at 50 scenarios, each message at 500 raw
|
|
76
|
+
characters and the skipped name list at 20, while the counts and the overall
|
|
77
|
+
`Status` row stay complete.
|
|
78
|
+
|
|
79
|
+
## Email summary
|
|
80
|
+
|
|
81
|
+
`--email <file>` writes one self-contained HTML mail body from the same summary.
|
|
82
|
+
It is another output, not a mode, and composes with `--out`, `--json`, `--ci` and
|
|
83
|
+
`--ci-out` (`--ci-out` still requires `--ci markdown`):
|
|
84
|
+
|
|
85
|
+
```shell
|
|
86
|
+
tolkira test-results/ --out report.html --email email.html
|
|
87
|
+
tolkira test-results/ --json --email email.html > summary.json
|
|
88
|
+
tolkira test-results/ --email email.html --locale de
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
- The file is a 600 px table with an inline system font stack and a small style
|
|
92
|
+
block below 480 px, no scripts, images, links, remote assets or hidden
|
|
93
|
+
preheader. The heading, note and failed names wrap anywhere; displayed feature,
|
|
94
|
+
scenario and report-name strings are capped at 200 characters with an ellipsis.
|
|
95
|
+
A standard summary stays under 100 KB.
|
|
96
|
+
- It shows the title and status, generation time, attempted pass rate as a
|
|
97
|
+
percentage with a full-sentence note that the rate counts only executed
|
|
98
|
+
(non-skipped) scenarios, every non-zero scenario status plus unexecuted, the
|
|
99
|
+
total as scenarios with results, feature/step/duration/attachment totals and
|
|
100
|
+
the first five failed scenarios in summary order
|
|
101
|
+
(feature, name, status, duration; no steps or errors). The attachment count is
|
|
102
|
+
labelled `Attachments in the run`. English digits are grouped (`8,509 bytes`)
|
|
103
|
+
and durations are human (`12 ms`, `12.3 s`, `1 min 5 s`, `1 h 2 min`). An
|
|
104
|
+
empty run and an all-unexecuted run are distinct; further failures are a
|
|
105
|
+
count.
|
|
106
|
+
- `--title` sets the heading and suggested subject; `--accent` sets the top
|
|
107
|
+
border even without `--out`. With `--out`, the report note names only the
|
|
108
|
+
report basename and says to attach or share it separately. Without `--out`
|
|
109
|
+
but with `--pdf`, it names the PDF basename because the PDF is the full
|
|
110
|
+
artifact and is written first. With neither, it says that no full report is
|
|
111
|
+
attached, so the mail never points at a file that does not exist.
|
|
112
|
+
- The email path is validated like an output: no collision with an input, the
|
|
113
|
+
logo, `--out`, `--ci-out`, `--history` or `--pdf`, and no path inside an input
|
|
114
|
+
directory (including through a junction or symlink); an empty path exits 2 and
|
|
115
|
+
writes nothing. The write order is report, PDF, email, history, `--ci-out`,
|
|
116
|
+
stdout. A write failure exits 1, names the email path, keeps the report and
|
|
117
|
+
skips later outputs.
|
|
118
|
+
- `Email written to <path>` is printed only when stdout is free. With `--json`
|
|
119
|
+
or `--ci` on stdout, the machine bytes are unchanged and no email line is
|
|
120
|
+
added; with `--ci-out` and no `--json`, the line is printed.
|
|
121
|
+
- Tolkira writes the file only. There is no SMTP, sending, scheduling or
|
|
122
|
+
network access. Feature and scenario names, the title (which may carry a
|
|
123
|
+
branch or build name) and the report basename are included and not redacted.
|
|
124
|
+
See [`EMAIL.md`](EMAIL.md).
|
|
125
|
+
|
|
126
|
+
## PDF export
|
|
127
|
+
|
|
128
|
+
`--pdf <file>` prints the complete report archive through the optional
|
|
129
|
+
Playwright Chromium engine. It is another output, not a mode, and composes with
|
|
130
|
+
`--out`, `--email`, `--json`, `--ci`/`--ci-out` and `--history`:
|
|
131
|
+
|
|
132
|
+
```shell
|
|
133
|
+
npm i -D playwright-core@^1.65.0
|
|
134
|
+
npx playwright-core install chromium-headless-shell
|
|
135
|
+
tolkira test-results/ --out report.html --pdf report.pdf
|
|
136
|
+
tolkira test-results/ --pdf report.pdf --json > summary.json
|
|
137
|
+
tolkira test-results/ --pdf report.pdf --locale de
|
|
138
|
+
tolkira test-results/ --pdf report.pdf --pdf-timeout 600000
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
- Run the install in the project where Tolkira runs; a project-local Tolkira
|
|
142
|
+
install is recommended. The engine resolves beside Tolkira, then from the
|
|
143
|
+
current project directory. Only `playwright-core` and the Chromium headless
|
|
144
|
+
shell are needed; the `playwright` package and full Chromium are not.
|
|
145
|
+
- The PDF is A4 portrait (`format: 'A4'`) with 12 mm `@page` print CSS margins,
|
|
146
|
+
the light print palette and no browser header, footer, URL or page number.
|
|
147
|
+
Chromium only in this slice; Letter/custom paper and landscape are not exposed.
|
|
148
|
+
- It is the complete archive (all views and details, screenshots, fitted tables)
|
|
149
|
+
from a fresh page: the CLI search/filter does not apply. The email is the
|
|
150
|
+
concise summary; video/audio do not print. HTTP(S) images in caller HTML are
|
|
151
|
+
aborted; inline `data:` images render.
|
|
152
|
+
- `--pdf-timeout <ms>` needs `--pdf`, is a positive integer up to 2,147,483,647
|
|
153
|
+
and overrides the five-minute render deadline
|
|
154
|
+
(`renderPdf(html, { timeoutMs })`). `page.pdf` has no timeout option and there
|
|
155
|
+
is no 30 s browser print default; Tolkira bounds the font/image wait and the
|
|
156
|
+
print explicitly.
|
|
157
|
+
- `--locale` is accepted with `--pdf` even without `--out` or `--email`; the PDF
|
|
158
|
+
follows the report locale while CLI/JSON/CI stay English. `--accent` applies.
|
|
159
|
+
- Without `--out`, no report HTML is written. `--logo` and `--history` still
|
|
160
|
+
require `--out` and exit 2.
|
|
161
|
+
- Write order: **report → PDF → email → history → `--ci-out` → stdout**. A PDF
|
|
162
|
+
render or write failure exits 1, names the PDF path, keeps the report and skips
|
|
163
|
+
the email/history/`--ci-out`/stdout; no partial PDF remains. A missing engine
|
|
164
|
+
exits 1 with the install command.
|
|
165
|
+
- `PDF written to <path>` is printed only when stdout is free, like the report and
|
|
166
|
+
email lines. JSON/CI stdout stays byte-identical; `--json --ci-out` also
|
|
167
|
+
suppresses the PDF line.
|
|
168
|
+
- The PDF path is validated like an output: no collision with an input, the logo,
|
|
169
|
+
`--out`, `--ci-out`, `--history` or `--email`, and no path inside an input
|
|
170
|
+
directory; a collision exits 2.
|
|
171
|
+
- The email with `--out` names the report; without `--out` it names the PDF
|
|
172
|
+
because the PDF is written first.
|
|
173
|
+
- The engine is optional and dynamically imported; the default install and
|
|
174
|
+
runtime dependency surface, report, JSON, email and CI bytes are unchanged;
|
|
175
|
+
the package gains `docs/PDF.md`. See [`PDF.md`](PDF.md).
|
|
176
|
+
|
|
177
|
+
## Report locale
|
|
178
|
+
|
|
179
|
+
`--locale <tag>` localizes the HTML report, the optional email summary and the
|
|
180
|
+
PDF. It requires `--out`, `--pdf` or `--email`, defaults to `en`, and accepts
|
|
181
|
+
`en` or `de`;
|
|
182
|
+
`de-AT` keeps the regional tag for `<html lang>` and uses the German catalog. An
|
|
183
|
+
empty or unknown tag exits 2 with the supported list before anything is written;
|
|
184
|
+
the API throws instead. The JSON summary, CI text, exit codes, summary keys and
|
|
185
|
+
the default title `Tolkira report` are unchanged. Test data renders verbatim in
|
|
186
|
+
every locale. With only `--json` or `--ci`, `--locale` stays a usage error. See
|
|
187
|
+
[`I18N.md`](I18N.md).
|
|
188
|
+
|
|
37
189
|
## API
|
|
38
190
|
|
|
39
191
|
```ts
|
|
40
192
|
import {
|
|
41
193
|
generateReport,
|
|
194
|
+
readHistory,
|
|
195
|
+
renderEmail,
|
|
196
|
+
renderPdf,
|
|
42
197
|
renderReport,
|
|
43
198
|
summarizeReport,
|
|
44
199
|
} from 'tolkira';
|
|
45
200
|
|
|
46
201
|
const inputs = ['test-results/'];
|
|
47
202
|
const summary = await summarizeReport(inputs);
|
|
48
|
-
const
|
|
203
|
+
const previousRuns = await readHistory('.history/history.json');
|
|
204
|
+
const { html, summary: sameSummary, skippedAttachments, history } =
|
|
205
|
+
await generateReport(inputs, {
|
|
206
|
+
title: 'Nightly',
|
|
207
|
+
locale: 'de',
|
|
208
|
+
metadata: { CI: 'true', Commit: process.env.GITHUB_SHA ?? '' },
|
|
209
|
+
logo: 'data:image/png;base64,...',
|
|
210
|
+
accent: '#1b5e20',
|
|
211
|
+
history: previousRuns,
|
|
212
|
+
});
|
|
213
|
+
const htmlOnly = await renderReport(inputs, { title: 'Nightly', locale: 'de' });
|
|
214
|
+
const email = renderEmail(summary, {
|
|
49
215
|
title: 'Nightly',
|
|
50
|
-
|
|
51
|
-
|
|
216
|
+
locale: 'de',
|
|
217
|
+
reportName: 'report.html',
|
|
52
218
|
accent: '#1b5e20',
|
|
219
|
+
now: new Date('2026-11-20T09:30:00.000Z'),
|
|
53
220
|
});
|
|
54
|
-
const
|
|
221
|
+
const pdf = await renderPdf(html, { timeoutMs: 300_000 });
|
|
55
222
|
```
|
|
56
223
|
|
|
224
|
+
`renderEmail(summary, options)` is synchronous and returns the same HTML the CLI
|
|
225
|
+
writes for the same summary, options and clock. `EmailOptions` has `title`,
|
|
226
|
+
`locale`, `reportName`, `accent` and `now`; there is no logo option. It writes
|
|
227
|
+
nothing and sends nothing. Invalid accents are ignored, an unrecognised summary
|
|
228
|
+
status falls back to the `Unknown` label and colour, and an invalid `now` omits
|
|
229
|
+
the generated date. The title, test names and report name are escaped; the report
|
|
230
|
+
note keeps only the basename. The CLI strips control characters from its
|
|
231
|
+
`Email written to` confirmation.
|
|
232
|
+
|
|
233
|
+
`renderPdf(html, options)` returns the PDF bytes for already rendered report
|
|
234
|
+
HTML. `PdfOptions` has only the optional `timeoutMs` (default five minutes). It
|
|
235
|
+
writes no file, starts no server, aborts HTTP(S) subresources and uses the same
|
|
236
|
+
optional Chromium engine as the CLI; a missing engine throws an install error.
|
|
237
|
+
The CLI and API produce equivalent PDFs for the same HTML and engine.
|
|
238
|
+
|
|
239
|
+
`skippedAttachments` is the number of Playwright path attachments skipped
|
|
240
|
+
because they resolve outside every allowed root. It is a plain number, additive
|
|
241
|
+
to the returned object; the run-level and per-scenario HTML notices and the CLI stderr warning use it, while
|
|
242
|
+
the summary document keeps schema version 2 and no warning field.
|
|
243
|
+
|
|
244
|
+
`history` is the aggregate run list the HTML was rendered with; it already
|
|
245
|
+
contains the current run. `generateReport` appends the current run to
|
|
246
|
+
`options.history` and returns the appended list. `renderReport` and
|
|
247
|
+
`CucumberHtmlStream` render `options.history` exactly as given and record
|
|
248
|
+
nothing. Both fields are additive.
|
|
249
|
+
|
|
57
250
|
The package is ESM, Node.js 22 or newer, and has no network or telemetry code.
|
|
58
251
|
|
|
59
252
|
| Option | CLI | API |
|
|
60
253
|
| ------ | --- | --- |
|
|
61
|
-
| title | `--title` | `title` |
|
|
254
|
+
| title | `--title` sets the report heading, Markdown CI heading and email heading/suggested subject | `title` |
|
|
62
255
|
| metadata | `--meta key=value` (repeatable) | `metadata` object |
|
|
63
256
|
| logo | `--logo <file>` reads a local image | `logo` base64 image data URI |
|
|
64
257
|
| accent | `--accent #rrggbb` | `accent` hex string; invalid values are ignored |
|
|
258
|
+
| history | `--history <file>` reads and appends a sidecar; needs `--out` | `history` array; `generateReport` appends the current run, `renderReport` and the stream render it as given |
|
|
259
|
+
| locale | `--locale <tag>` (`en` or `de`; needs `--out`, `--pdf` or `--email`) | `locale` string; default `en` |
|
|
260
|
+
| email | `--email <file>` writes an HTML summary | `renderEmail(summary, options)` |
|
|
261
|
+
| pdf | `--pdf <file>` prints the report with optional Playwright Chromium; `--pdf-timeout <ms>` (needs `--pdf`) overrides the deadline | `renderPdf(html, { timeoutMs })` |
|
|
65
262
|
| size warning | `--max-size <mb>` (default 10) | not applicable; the API returns HTML |
|
|
263
|
+
| CI output | `--ci github\|markdown`, `--ci-out <file>` | not applicable; format the summary yourself |
|
|
66
264
|
|
|
67
265
|
The CLI validates logo bytes and rejects unsupported types or a size over
|
|
68
266
|
2 MB. The API only accepts a `data:image/...;base64,...` logo; anything else is
|
|
69
267
|
omitted.
|
|
70
268
|
|
|
71
|
-
##
|
|
269
|
+
## Messages stream
|
|
270
|
+
|
|
271
|
+
`CucumberHtmlStream` is a shape-compatible Node `Transform` for callers that
|
|
272
|
+
already hold Cucumber Messages envelopes:
|
|
273
|
+
|
|
274
|
+
```ts
|
|
275
|
+
import { Readable } from 'node:stream';
|
|
276
|
+
import { CucumberHtmlStream } from 'tolkira';
|
|
277
|
+
|
|
278
|
+
const html = await new Promise<string>((resolve, reject) => {
|
|
279
|
+
let output = '';
|
|
280
|
+
Readable.from(envelopes)
|
|
281
|
+
.pipe(new CucumberHtmlStream({ title: 'Nightly' }))
|
|
282
|
+
.on('data', (chunk) => (output += chunk))
|
|
283
|
+
.on('error', reject)
|
|
284
|
+
.on('end', () => resolve(output));
|
|
285
|
+
});
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
The stream accepts `Envelope` objects in object mode, buffers the run and
|
|
289
|
+
renders one HTML document on flush. It takes `RenderReportOptions` (title,
|
|
290
|
+
metadata, logo, accent, locale and the optional history list) and produces the
|
|
291
|
+
same HTML for the same envelopes and options. A supplied `history` is rendered
|
|
292
|
+
exactly as given; only `generateReport` appends the current run. The stream is
|
|
293
|
+
not an incremental renderer, writes no files and emits no summary; use
|
|
294
|
+
`summarizeReport` or `generateReport` when you also need the stable JSON. The
|
|
295
|
+
whole run is held in memory, exactly like the file API.
|
|
296
|
+
|
|
297
|
+
## Run history
|
|
298
|
+
|
|
299
|
+
`--history <file>` keeps an aggregate sidecar of prior runs and renders a
|
|
300
|
+
`Trends` tab once the report is generated with two or more entries. It requires
|
|
301
|
+
`--out`; without the flag no history is read or written and the report contains
|
|
302
|
+
no history data or Trends tab.
|
|
303
|
+
|
|
304
|
+
```shell
|
|
305
|
+
tolkira test-results/ --out report.html --history .history/history.json
|
|
306
|
+
```
|
|
307
|
+
|
|
308
|
+
The sidecar is UTF-8 JSON with schema version 1:
|
|
72
309
|
|
|
73
310
|
```json
|
|
74
311
|
{
|
|
75
312
|
"schemaVersion": 1,
|
|
313
|
+
"runs": [
|
|
314
|
+
{
|
|
315
|
+
"id": "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9",
|
|
316
|
+
"recordedAt": "2026-11-19T06:00:00.000Z",
|
|
317
|
+
"title": "Nightly",
|
|
318
|
+
"status": "failed",
|
|
319
|
+
"counts": {
|
|
320
|
+
"features": 3,
|
|
321
|
+
"scenarios": {
|
|
322
|
+
"total": 10,
|
|
323
|
+
"passed": 6,
|
|
324
|
+
"failed": 2,
|
|
325
|
+
"skipped": 1,
|
|
326
|
+
"pending": 1,
|
|
327
|
+
"undefined": 0,
|
|
328
|
+
"ambiguous": 0,
|
|
329
|
+
"unknown": 0
|
|
330
|
+
},
|
|
331
|
+
"steps": {
|
|
332
|
+
"total": 42,
|
|
333
|
+
"passed": 34,
|
|
334
|
+
"failed": 3,
|
|
335
|
+
"skipped": 2,
|
|
336
|
+
"pending": 1,
|
|
337
|
+
"undefined": 1,
|
|
338
|
+
"ambiguous": 1,
|
|
339
|
+
"unknown": 0
|
|
340
|
+
}
|
|
341
|
+
},
|
|
342
|
+
"durationMs": 12345.678,
|
|
343
|
+
"attachments": { "total": 2, "bytes": 4096 }
|
|
344
|
+
}
|
|
345
|
+
]
|
|
346
|
+
}
|
|
347
|
+
```
|
|
348
|
+
|
|
349
|
+
Contract:
|
|
350
|
+
|
|
351
|
+
- `schemaVersion` is `1`. A wrong version is rejected; unknown fields inside
|
|
352
|
+
version 1 are ignored and the reader copies only the documented fields.
|
|
353
|
+
- `id` is the SHA-256 of the aggregates and the resolved title, excluding
|
|
354
|
+
`recordedAt`. When the last entry has the same id, the new run is not
|
|
355
|
+
appended and the old `recordedAt` stays.
|
|
356
|
+
- The sidecar keeps the newest 30 runs; older entries are dropped on read, on
|
|
357
|
+
append and on write. A file above 1 MiB is rejected before it is read.
|
|
358
|
+
- `title` is the resolved report title (`Tolkira report` when `--title` is
|
|
359
|
+
absent), capped at 200 characters including a trailing `…`; a longer title is
|
|
360
|
+
truncated before it is hashed or written. `durationMs` is the summary
|
|
361
|
+
duration, not wall-clock time.
|
|
362
|
+
- Only aggregates are stored: no scenario names, step text, paths, tags or
|
|
363
|
+
attachment content. The report embeds the prior-run aggregates it was
|
|
364
|
+
generated with, so anyone who receives the HTML can read the pass-rate
|
|
365
|
+
history. Use one sidecar per suite and variant; mixing runs makes the trend
|
|
366
|
+
meaningless.
|
|
367
|
+
- A missing file starts empty. Invalid JSON, a wrong schema version, an
|
|
368
|
+
oversized file, a malformed entry or a write failure exits 1 and names the
|
|
369
|
+
file. A bad file is never overwritten. The HTML is written first; a failed
|
|
370
|
+
history write leaves it in place and leaves no partial sidecar.
|
|
371
|
+
- `--history` follows the output-path rules: it must not name an input, the
|
|
372
|
+
logo, `--out`, `--ci-out`, `--email` or a path inside an input directory, even
|
|
373
|
+
when the history file does not exist yet.
|
|
374
|
+
- The CLI (through `generateReport`) is the only automatic writer. API callers
|
|
375
|
+
write through `writeHistory`; `renderReport` and `CucumberHtmlStream` render a
|
|
376
|
+
given list and record nothing. The writer reads, appends and writes atomically
|
|
377
|
+
with a rename; that prevents a corrupt file, not lost updates from two
|
|
378
|
+
concurrent processes. Run one generation per sidecar at a time.
|
|
379
|
+
- The 30-run bound applies to the CLI and `generateReport` path through
|
|
380
|
+
`readHistory` and `appendHistory`; `renderReport` and `CucumberHtmlStream`
|
|
381
|
+
accept the list they are given.
|
|
382
|
+
|
|
383
|
+
The package exports `HistoryRun`, `HISTORY_SCHEMA_VERSION`, `HISTORY_LIMIT`,
|
|
384
|
+
`HISTORY_MAX_BYTES`, `HISTORY_MAX_TITLE`, `buildHistoryRun`, `appendHistory`,
|
|
385
|
+
`readHistory` and `writeHistory` for programmatic use.
|
|
386
|
+
|
|
387
|
+
## Summary schema, version 2
|
|
388
|
+
|
|
389
|
+
```json
|
|
390
|
+
{
|
|
391
|
+
"schemaVersion": 2,
|
|
76
392
|
"status": "failed",
|
|
77
393
|
"durationMs": 12.4,
|
|
78
394
|
"counts": {
|
|
@@ -108,6 +424,7 @@ omitted.
|
|
|
108
424
|
"uri": "features/screenshot.feature",
|
|
109
425
|
"feature": "Screenshot attachment",
|
|
110
426
|
"status": "failed",
|
|
427
|
+
"executed": true,
|
|
111
428
|
"durationMs": 10,
|
|
112
429
|
"attempt": 0,
|
|
113
430
|
"tags": ["@failing"],
|
|
@@ -129,51 +446,205 @@ omitted.
|
|
|
129
446
|
This example is abridged: it shows only the failing scenario, not the passing
|
|
130
447
|
one that carries the screenshot. The totals reflect the whole input.
|
|
131
448
|
|
|
449
|
+
Undefined and ambiguous Messages steps add optional diagnostic fields:
|
|
450
|
+
|
|
451
|
+
```json
|
|
452
|
+
{
|
|
453
|
+
"keyword": "Given",
|
|
454
|
+
"text": "a list of 8 things",
|
|
455
|
+
"status": "undefined",
|
|
456
|
+
"durationMs": 0,
|
|
457
|
+
"suggestions": [
|
|
458
|
+
{
|
|
459
|
+
"language": "typescript",
|
|
460
|
+
"code": "Given(\"a list of {int} things\", (int: number) => {\n return \"pending\"\n})"
|
|
461
|
+
},
|
|
462
|
+
{
|
|
463
|
+
"language": "typescript",
|
|
464
|
+
"code": "Given(\"a list of {float} things\", (float: number) => {\n return \"pending\"\n})"
|
|
465
|
+
}
|
|
466
|
+
]
|
|
467
|
+
}
|
|
468
|
+
```
|
|
469
|
+
|
|
470
|
+
```json
|
|
471
|
+
{
|
|
472
|
+
"keyword": "Given",
|
|
473
|
+
"text": "a step with multiple definitions",
|
|
474
|
+
"status": "ambiguous",
|
|
475
|
+
"durationMs": 0,
|
|
476
|
+
"definitions": [
|
|
477
|
+
{
|
|
478
|
+
"pattern": "^a (.*?) with (.*?)$",
|
|
479
|
+
"source": "samples/ambiguous/ambiguous.ts:3"
|
|
480
|
+
},
|
|
481
|
+
{
|
|
482
|
+
"pattern": "^a step with (.*?)$",
|
|
483
|
+
"source": "samples/ambiguous/ambiguous.ts:7"
|
|
484
|
+
}
|
|
485
|
+
]
|
|
486
|
+
}
|
|
487
|
+
```
|
|
488
|
+
|
|
132
489
|
Field contract:
|
|
133
490
|
|
|
134
|
-
- `schemaVersion` is `
|
|
491
|
+
- `schemaVersion` is `2`. A breaking change increments it.
|
|
135
492
|
- `status` is one of `passed`, `failed`, `skipped`, `pending`, `undefined`,
|
|
136
493
|
`ambiguous`, `unknown`.
|
|
137
|
-
- `status` is the worst scenario status
|
|
138
|
-
using the standard Cucumber
|
|
139
|
-
|
|
140
|
-
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
494
|
+
- `status` is the worst executed scenario status, or `failed` when a test-run
|
|
495
|
+
hook fails. A scenario is the worst step status, using the standard Cucumber
|
|
496
|
+
severity order. When nothing executed, the run status is `unknown` unless a
|
|
497
|
+
test-run hook failed.
|
|
498
|
+
- `counts.features` is the number of features with at least one scenario row,
|
|
499
|
+
executed or unexecuted, matching the HTML feature list. A document whose
|
|
500
|
+
pickles never started contributes no row and is not counted.
|
|
501
|
+
- `counts.scenarios` counts executed scenarios only, so `total` can be smaller
|
|
502
|
+
than `scenarios.length`. `counts.scenarios` and `counts.steps` always contain
|
|
503
|
+
every status key, even when the value is `0`. `total` is the sum. The HTML
|
|
504
|
+
pass rate and feature pass text use attempted (non-skipped) scenarios, while
|
|
505
|
+
counts, pills and bars include skipped results.
|
|
506
|
+
- `scenarios[].executed` is `true` when the selected test case reported at
|
|
507
|
+
least one raw step result, including a hook result. Adapters represent a
|
|
508
|
+
reported skip without steps as a synthetic skipped step, so the summary
|
|
509
|
+
reports it as executed. An unexecuted scenario keeps `status: "unknown"` and
|
|
510
|
+
`executed: false`; it is not counted in `counts.scenarios`. A pickle that
|
|
511
|
+
never started has no entry at all.
|
|
512
|
+
- `durationMs` is a number of milliseconds and sums the final attempt per
|
|
513
|
+
scenario plus test-run hooks. It is not wall-clock time.
|
|
514
|
+
`steps[].durationMs` is rounded to three decimals.
|
|
144
515
|
- `scenarios` is sorted by `uri`, then `name`, so output is stable across runs.
|
|
145
|
-
- `
|
|
516
|
+
- `scenarios[].name` is the Cucumber pickle name. Examples of one scenario
|
|
517
|
+
outline can share the same name; the HTML report labels rows with `#1.1` and
|
|
518
|
+
shows the examples block name on the examples heading and in triage and
|
|
519
|
+
copy-failure text, but the summary does not expose them yet.
|
|
520
|
+
- `scenarios[].tags` is the inherited pickle tag set, including feature, rule,
|
|
521
|
+
scenario and examples tags. The HTML report uses the same set for its tag
|
|
522
|
+
filter and search. Feature and rule tags render once at their section;
|
|
523
|
+
scenario tags render on the scenario row or, for an outline, on the outline
|
|
524
|
+
heading, and examples tags on the examples heading. Example rows repeat
|
|
525
|
+
neither scenario nor examples tags. The summary does not expose rules,
|
|
526
|
+
backgrounds or the own/inherited distinction. A scenario whose pickle tag set
|
|
527
|
+
(own or inherited) includes `@flaky` (lowercase, exact match) gets a
|
|
528
|
+
report-only (HTML/PDF) badge, and when at least one exists the run metadata
|
|
529
|
+
gets a separate count such as `1 tagged @flaky`, distinct from the status
|
|
530
|
+
counts. Feature, rule and outline-level `@flaky` tags badge every affected row,
|
|
531
|
+
including every outline example, so the count can exceed the number of visible
|
|
532
|
+
tag labels; a failed or unexecuted tagged scenario also counts. The count is
|
|
533
|
+
not a computed flake rate or detection, and neither badge nor count appears in
|
|
534
|
+
the summary, CI or email or changes a status.
|
|
535
|
+
- `attempt` is the latest started Cucumber attempt for that scenario, even when
|
|
536
|
+
that attempt reported no result. An earlier retried attempt is ignored. For a
|
|
537
|
+
Playwright JSON input it is the Playwright retry index of the final result;
|
|
538
|
+
earlier attempts are not exposed as scenario rows.
|
|
146
539
|
- `attachments.bytes` is the decoded size, not the base64 length.
|
|
147
540
|
- `steps[].message` is present only when the step result has a message. For
|
|
148
|
-
hooks, `keyword` is `"Hook"
|
|
541
|
+
hooks, `keyword` is `"Hook"`, `text` is the hook type and `name` is present
|
|
542
|
+
when Cucumber provides one. Summary step counts include hook entries; the
|
|
543
|
+
HTML report counts steps and hooks separately. A failed test-run hook makes
|
|
544
|
+
the overall summary `status` failed.
|
|
545
|
+
- `steps[].suggestions` is present only for an undefined Messages step with
|
|
546
|
+
`suggestion` envelopes. Each entry has an optional `language` and the `code`
|
|
547
|
+
to implement. `steps[].definitions` is present only for an ambiguous Messages
|
|
548
|
+
step with `stepDefinition` envelopes. Each entry has the `pattern` and an
|
|
549
|
+
optional `source` in `uri:line` form with absolute paths reduced to a
|
|
550
|
+
relative form. Both fields are additive, so `schemaVersion` stays 2;
|
|
551
|
+
a passed step never carries them.
|
|
149
552
|
- Attachment file names are not part of the summary. `scenarios[].attachments`
|
|
150
|
-
is a count; open the HTML report to see
|
|
553
|
+
is a count and is 0 for an unexecuted scenario; open the HTML report to see
|
|
554
|
+
the image or file. Unnamed attachments use a deterministic
|
|
555
|
+
`attachment.<ext>` fallback in HTML; unknown media types normalise to
|
|
556
|
+
`application/octet-stream` and `attachment.bin`.
|
|
557
|
+
- CI metadata from Messages `meta.ci` is HTML-only. It renders in the report
|
|
558
|
+
header and Details, but no CI field appears in the summary and
|
|
559
|
+
`schemaVersion` stays 2. When `ci.name` is missing, the http(s) `ci.url` is
|
|
560
|
+
the link text. In a mixed input the first non-empty CI block wins, and
|
|
561
|
+
meta synthesized by the legacy adapter cannot erase it. Legacy JSON and
|
|
562
|
+
JUnit never invent CI metadata; use the `metadata` option for a legacy path.
|
|
563
|
+
|
|
564
|
+
## Migrating to schema version 2
|
|
565
|
+
|
|
566
|
+
Version 2 adds `executed` to every scenario and changes scenario counts to
|
|
567
|
+
cover executed scenarios only:
|
|
568
|
+
|
|
569
|
+
- `counts.scenarios.total` can be smaller than `scenarios.length`; use the
|
|
570
|
+
counts for aggregate decisions.
|
|
571
|
+
- An unexecuted scenario has `status: "unknown"` and `executed: false`. That
|
|
572
|
+
`unknown` is not a failure signal.
|
|
573
|
+
- Filter on `executed` (or use `counts.scenarios`) before treating a
|
|
574
|
+
scenario's `status` as an outcome. A legacy Cucumber JSON scenario with no
|
|
575
|
+
steps and no hooks is now unexecuted, so its scenario total drops by one
|
|
576
|
+
compared with version 1.
|
|
577
|
+
- Never-started pickles remain absent from `scenarios` and the counts.
|
|
151
578
|
|
|
152
579
|
The summary does not expose absolute input paths: absolute `uri` values are
|
|
153
|
-
reduced to a
|
|
580
|
+
reduced to a relative form (`features/...`, or the last path segments for files
|
|
581
|
+
outside `features/`).
|
|
154
582
|
|
|
155
583
|
## Recommended agent workflow
|
|
156
584
|
|
|
157
585
|
1. Run the tests and keep the raw Cucumber JSON or Messages.
|
|
158
|
-
2. Run `tolkira <inputs> --out report.html --json
|
|
159
|
-
|
|
160
|
-
|
|
586
|
+
2. Run `tolkira <inputs> --out report.html --json`; add `--email email.html`
|
|
587
|
+
when a status mail or chat message is part of the job.
|
|
588
|
+
3. Parse stdout once. Filter on `executed` (or use `counts`) and use `status`,
|
|
589
|
+
counts and `scenarios[].steps[].message` for diagnosis.
|
|
161
590
|
4. If you need screenshots or the full Gherkin view, open `report.html`; do not
|
|
162
591
|
scrape it for pass/fail data.
|
|
163
|
-
5. Attach or upload the HTML report as the human-facing artifact
|
|
592
|
+
5. Attach or upload the HTML report as the human-facing artifact; use
|
|
593
|
+
`email.html` as optional mail body, never as the only record.
|
|
164
594
|
|
|
165
595
|
## Current limits
|
|
166
596
|
|
|
167
597
|
- Remote attachments (`attachment.url`, `externalAttachment`) are rejected on
|
|
168
598
|
purpose; the report must work offline. Inline the bytes.
|
|
169
|
-
- Markdown
|
|
599
|
+
- Description Markdown renders only in the HTML report. Remote image
|
|
600
|
+
destinations are neutralised and images render as alt text, so the report
|
|
601
|
+
loads no external asset. JUnit `description` attributes are a best-effort,
|
|
602
|
+
non-standard mapping; they never appear in the summary.
|
|
603
|
+
- Tag-expression search, the tag dropdown and `?query=`/`?hide=`/`?tag=` URL
|
|
604
|
+
state are HTML-only progressive enhancements. `?hide=` also accepts
|
|
605
|
+
`unexecuted`, a pseudo-status for the failed-only preset that hides rows
|
|
606
|
+
without an outcome; it is restored only with the preset chip state and a
|
|
607
|
+
pruned value is removed from the URL. The summary schema and counts are
|
|
608
|
+
unchanged; the static no-JS report keeps the inert control and all rows.
|
|
170
609
|
- Inputs are capped at 100 MB in total; attachments at 50 MB each and 100 MB
|
|
171
610
|
total.
|
|
172
611
|
- The HTML report includes a hash-based CSP and never executes attachments.
|
|
173
|
-
- History
|
|
174
|
-
|
|
612
|
+
- History is an opt-in aggregate sidecar (`--history`); there is no
|
|
613
|
+
scenario-level history, dashboard or server. Retries beyond the latest attempt
|
|
614
|
+
and a trace viewer are not supported. Playwright JSON has a native adapter;
|
|
615
|
+
the final attempt is the rendered one, and `@flaky` is report-only
|
|
616
|
+
(HTML/PDF): the summary, CI and email keep the final status. JUnit XML is a basic input mode; see
|
|
617
|
+
[`LIMITATIONS.md`](LIMITATIONS.md). Playwright producer metadata and
|
|
618
|
+
annotation descriptions are HTML-only and do not change schema version 2.
|
|
619
|
+
- The email summary is opt-in and file-only. It has no sending, recipients,
|
|
620
|
+
scheduling, `.eml` draft, logo, error snippets or redaction mode; the report
|
|
621
|
+
basename (or the no-report note) is the only path-like content.
|
|
622
|
+
- PDF export is opt-in, Chromium-only and file-only. It is A4 portrait only
|
|
623
|
+
(no Letter/custom paper), with no page numbers, custom header/footer, tagged
|
|
624
|
+
PDF, CLI filter, forced landscape, PDF-only logo or PDF-only Trends; it needs a
|
|
625
|
+
separate project-local engine and browser install. The default Chromium launch
|
|
626
|
+
is not OS-sandboxed; see [`PDF.md`](PDF.md).
|
|
627
|
+
- Playwright path attachments are embedded only when the resolved regular file
|
|
628
|
+
is inside the input file's directory or the process working directory.
|
|
629
|
+
Symlinked, missing, non-regular and outside-tree paths are skipped; inline
|
|
630
|
+
base64 attachments are always preferred. An outside-tree path that exists as a
|
|
631
|
+
regular file is counted in `generateReport().skippedAttachments` and surfaces
|
|
632
|
+
as run-level and per-scenario HTML notices plus a CLI stderr warning; later-missing files, symlinks and
|
|
633
|
+
the other skipped causes are not counted. Trace zips and HTML-report replays
|
|
634
|
+
are not read.
|
|
175
635
|
- Legacy Cucumber JSON outlines arrive as expanded scenarios; the original
|
|
176
|
-
outline `examples` table is not reconstructed. Behave
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
636
|
+
outline `examples` table is not reconstructed. Behave durations are seconds
|
|
637
|
+
and are normalised to milliseconds; multiline doc strings and error messages
|
|
638
|
+
arrive as line arrays and are joined with newlines. Behave's `error` and
|
|
639
|
+
`hook_error` step-result statuses map to failed; `cleanup_error`, which Behave
|
|
640
|
+
records on the scenario/feature during context cleanup, is accepted
|
|
641
|
+
defensively as a failed step result.
|
|
642
|
+
- Rules and backgrounds are an HTML presentation feature. The version-2 summary
|
|
643
|
+
keeps the flat scenario list; `scenarios[].tags` remains the inherited tag
|
|
644
|
+
set. Legacy JSON and JUnit never invent rule or background structure.
|
|
645
|
+
- Legacy Cucumber JSON and JUnit have no suggestion or step-definition
|
|
646
|
+
envelopes. An undefined or ambiguous legacy step still shows the status lead
|
|
647
|
+
in HTML, but never invented snippets or definitions; JUnit renders no
|
|
648
|
+
diagnostics.
|
|
649
|
+
- cucumber-js, Behave, cucumber-jvm, cucumber-ruby and Messages inputs are
|
|
650
|
+
covered by fixtures.
|