tolkira 0.2.0 → 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (64) hide show
  1. package/DESIGN.md +131 -9
  2. package/GLOSSARY.md +201 -1
  3. package/README.md +413 -29
  4. package/SECURITY.md +3 -0
  5. package/dist/attachment.d.ts +33 -0
  6. package/dist/attempt-diagnostics.d.ts +2 -0
  7. package/dist/ci.d.ts +12 -0
  8. package/dist/cli.js +6881 -2691
  9. package/dist/cli.js.map +4 -4
  10. package/dist/compat/cucumber-html-reporter.cjs +12 -0
  11. package/dist/compat/cucumber-html-reporter.d.cts +10 -0
  12. package/dist/compat/cucumber-html-reporter.js +263 -26
  13. package/dist/compat/cucumber-html-reporter.js.map +4 -4
  14. package/dist/compat/options.d.ts +4 -0
  15. package/dist/email.d.ts +18 -0
  16. package/dist/history.d.ts +41 -0
  17. package/dist/i18n/catalogs.d.ts +7 -0
  18. package/dist/index.d.ts +15 -7
  19. package/dist/index.js +5301 -1709
  20. package/dist/index.js.map +4 -4
  21. package/dist/input/background.d.ts +4 -0
  22. package/dist/input/detect.d.ts +30 -0
  23. package/dist/input/load.d.ts +3 -0
  24. package/dist/input/playwright.d.ts +14 -0
  25. package/dist/options.d.ts +18 -0
  26. package/dist/output.d.ts +9 -1
  27. package/dist/pdf.d.ts +9 -0
  28. package/dist/render/ansi.d.ts +2 -0
  29. package/dist/render/attachments.d.ts +3 -0
  30. package/dist/render/ci.d.ts +4 -0
  31. package/dist/render/client-bundle.d.ts +1 -0
  32. package/dist/render/client-i18n.d.ts +7 -0
  33. package/dist/render/details.d.ts +5 -0
  34. package/dist/render/dom-contract.d.ts +42 -0
  35. package/dist/render/format.d.ts +34 -0
  36. package/dist/render/html.d.ts +2 -9
  37. package/dist/render/i18n.d.ts +30 -0
  38. package/dist/render/markdown.d.ts +2 -0
  39. package/dist/render/matrix.d.ts +3 -0
  40. package/dist/render/print-script.d.ts +1 -0
  41. package/dist/render/report-client.d.ts +1 -0
  42. package/dist/render/report.d.ts +4 -0
  43. package/dist/render/search.d.ts +21 -0
  44. package/dist/render/trends.d.ts +7 -0
  45. package/dist/render/triage.d.ts +5 -0
  46. package/dist/render/types.d.ts +26 -0
  47. package/dist/report/index-run.d.ts +69 -0
  48. package/dist/report/model.d.ts +96 -6
  49. package/dist/report/status.d.ts +12 -0
  50. package/dist/stream.d.ts +15 -0
  51. package/dist/summary.d.ts +17 -2
  52. package/dist/text.d.ts +15 -0
  53. package/dist/warnings.d.ts +6 -0
  54. package/docs/AGENT-INTERFACE.md +504 -33
  55. package/docs/CI.md +185 -0
  56. package/docs/COMPATIBILITY.md +121 -16
  57. package/docs/CUCUMBER-COMPATIBILITY.md +232 -15
  58. package/docs/EMAIL.md +273 -0
  59. package/docs/I18N.md +241 -0
  60. package/docs/JUNIT-XML.md +46 -4
  61. package/docs/LIMITATIONS.md +21 -0
  62. package/docs/PDF.md +255 -0
  63. package/llms.txt +63 -0
  64. package/package.json +46 -8
@@ -1,12 +1,17 @@
1
1
  # Tolkira for agents
2
2
 
3
- Tolkira produces two things from the same input:
3
+ Tolkira produces five things from the same input:
4
4
 
5
5
  - a self-contained HTML report for humans;
6
- - a stable JSON summary for programs and AI agents.
6
+ - a stable JSON summary for programs and AI agents;
7
+ - deterministic CI text: GitHub Actions annotations or a Markdown summary;
8
+ - an opt-in HTML email summary for a status mail or chat message;
9
+ - an opt-in PDF print archive when the optional Playwright Chromium engine is
10
+ installed.
7
11
 
8
- The HTML report is the artifact. The JSON summary is the interface. Do not parse
9
- the HTML to decide whether a test run passed.
12
+ The HTML report is the artifact. The JSON summary is the interface. The CI text
13
+ is a presentation of that same summary. Do not parse the HTML to decide whether
14
+ a test run passed.
10
15
 
11
16
  ## CLI
12
17
 
@@ -34,45 +39,356 @@ stderr. Exit codes are about report generation, not test status:
34
39
  Always read `status` from the JSON summary for the test outcome. A failed test
35
40
  suite is exit code 0 by design.
36
41
 
42
+ ## CI output
43
+
44
+ Format the same summary for CI instead of, or alongside, HTML and JSON:
45
+
46
+ ```shell
47
+ # GitHub Actions workflow commands on stdout; no HTML required
48
+ tolkira test-results/ --ci github
49
+
50
+ # Markdown on stdout, or in a standalone file
51
+ tolkira test-results/ --ci markdown
52
+ tolkira test-results/ --out report.html --ci markdown --ci-out summary.md
53
+ ```
54
+
55
+ - `--ci github` writes `::error` and `::notice` workflow commands. It needs no
56
+ `--out` and rejects `--ci-out`: GitHub only reads workflow commands from
57
+ stdout.
58
+ - `--ci markdown` writes deterministic Markdown with complete counts, a failed
59
+ list and skipped names. `--title` sets the heading.
60
+ - Without `--ci-out`, CI text is the only stdout and the human confirmation is
61
+ suppressed. With `--ci-out`, the file receives the CI text and stdout keeps
62
+ its normal confirmation; with `--json`, stdout stays JSON only.
63
+ - `--ci-out` requires `--ci markdown`, is written atomically as a standalone
64
+ file, and must not overwrite an input, the logo, `--out` or the `--email`
65
+ file. To add to
66
+ `$GITHUB_STEP_SUMMARY`, append stdout with `>>`; do not point `--ci-out` at a
67
+ shared file.
68
+ - Exit codes are unchanged. A failed test run still exits 0; invalid
69
+ combinations exit 2 before any file is written.
70
+ - Messages are copied verbatim and are not redacted. See [`CI.md`](CI.md).
71
+
72
+ GitHub mode caps at 50 error and 20 notice annotations and states the
73
+ remainder; the final count notice keeps the exact totals. A failed run status
74
+ without a scenario-level error produces one run-level error annotation.
75
+ Markdown caps the failed list at 50 scenarios, each message at 500 raw
76
+ characters and the skipped name list at 20, while the counts and the overall
77
+ `Status` row stay complete.
78
+
79
+ ## Email summary
80
+
81
+ `--email <file>` writes one self-contained HTML mail body from the same summary.
82
+ It is another output, not a mode, and composes with `--out`, `--json`, `--ci` and
83
+ `--ci-out` (`--ci-out` still requires `--ci markdown`):
84
+
85
+ ```shell
86
+ tolkira test-results/ --out report.html --email email.html
87
+ tolkira test-results/ --json --email email.html > summary.json
88
+ tolkira test-results/ --email email.html --locale de
89
+ ```
90
+
91
+ - The file is a 600 px table with an inline system font stack and a small style
92
+ block below 480 px, no scripts, images, links, remote assets or hidden
93
+ preheader. The heading, note and failed names wrap anywhere; displayed feature,
94
+ scenario and report-name strings are capped at 200 characters with an ellipsis.
95
+ A standard summary stays under 100 KB.
96
+ - It shows the title and status, generation time, attempted pass rate as a
97
+ percentage with a full-sentence note that the rate counts only executed
98
+ (non-skipped) scenarios, every non-zero scenario status plus unexecuted, the
99
+ total as scenarios with results, feature/step/duration/attachment totals and
100
+ the first five failed scenarios in summary order
101
+ (feature, name, status, duration; no steps or errors). The attachment count is
102
+ labelled `Attachments in the run`. English digits are grouped (`8,509 bytes`)
103
+ and durations are human (`12 ms`, `12.3 s`, `1 min 5 s`, `1 h 2 min`). An
104
+ empty run and an all-unexecuted run are distinct; further failures are a
105
+ count.
106
+ - `--title` sets the heading and suggested subject; `--accent` sets the top
107
+ border even without `--out`. With `--out`, the report note names only the
108
+ report basename and says to attach or share it separately. Without `--out`
109
+ but with `--pdf`, it names the PDF basename because the PDF is the full
110
+ artifact and is written first. With neither, it says that no full report is
111
+ attached, so the mail never points at a file that does not exist.
112
+ - The email path is validated like an output: no collision with an input, the
113
+ logo, `--out`, `--ci-out`, `--history` or `--pdf`, and no path inside an input
114
+ directory (including through a junction or symlink); an empty path exits 2 and
115
+ writes nothing. The write order is report, PDF, email, history, `--ci-out`,
116
+ stdout. A write failure exits 1, names the email path, keeps the report and
117
+ skips later outputs.
118
+ - `Email written to <path>` is printed only when stdout is free. With `--json`
119
+ or `--ci` on stdout, the machine bytes are unchanged and no email line is
120
+ added; with `--ci-out` and no `--json`, the line is printed.
121
+ - Tolkira writes the file only. There is no SMTP, sending, scheduling or
122
+ network access. Feature and scenario names, the title (which may carry a
123
+ branch or build name) and the report basename are included and not redacted.
124
+ See [`EMAIL.md`](EMAIL.md).
125
+
126
+ ## PDF export
127
+
128
+ `--pdf <file>` prints the complete report archive through the optional
129
+ Playwright Chromium engine. It is another output, not a mode, and composes with
130
+ `--out`, `--email`, `--json`, `--ci`/`--ci-out` and `--history`:
131
+
132
+ ```shell
133
+ npm i -D playwright-core@^1.65.0
134
+ npx playwright-core install chromium-headless-shell
135
+ tolkira test-results/ --out report.html --pdf report.pdf
136
+ tolkira test-results/ --pdf report.pdf --json > summary.json
137
+ tolkira test-results/ --pdf report.pdf --locale de
138
+ tolkira test-results/ --pdf report.pdf --pdf-timeout 600000
139
+ ```
140
+
141
+ - Run the install in the project where Tolkira runs; a project-local Tolkira
142
+ install is recommended. The engine resolves beside Tolkira, then from the
143
+ current project directory. Only `playwright-core` and the Chromium headless
144
+ shell are needed; the `playwright` package and full Chromium are not.
145
+ - The PDF is A4 portrait (`format: 'A4'`) with 12 mm `@page` print CSS margins,
146
+ the light print palette and no browser header, footer, URL or page number.
147
+ Chromium only in this slice; Letter/custom paper and landscape are not exposed.
148
+ - It is the complete archive (all views and details, screenshots, fitted tables)
149
+ from a fresh page: the CLI search/filter does not apply. The email is the
150
+ concise summary; video/audio do not print. HTTP(S) images in caller HTML are
151
+ aborted; inline `data:` images render.
152
+ - `--pdf-timeout <ms>` needs `--pdf`, is a positive integer up to 2,147,483,647
153
+ and overrides the five-minute render deadline
154
+ (`renderPdf(html, { timeoutMs })`). `page.pdf` has no timeout option and there
155
+ is no 30 s browser print default; Tolkira bounds the font/image wait and the
156
+ print explicitly.
157
+ - `--locale` is accepted with `--pdf` even without `--out` or `--email`; the PDF
158
+ follows the report locale while CLI/JSON/CI stay English. `--accent` applies.
159
+ - Without `--out`, no report HTML is written. `--logo` and `--history` still
160
+ require `--out` and exit 2.
161
+ - Write order: **report → PDF → email → history → `--ci-out` → stdout**. A PDF
162
+ render or write failure exits 1, names the PDF path, keeps the report and skips
163
+ the email/history/`--ci-out`/stdout; no partial PDF remains. A missing engine
164
+ exits 1 with the install command.
165
+ - `PDF written to <path>` is printed only when stdout is free, like the report and
166
+ email lines. JSON/CI stdout stays byte-identical; `--json --ci-out` also
167
+ suppresses the PDF line.
168
+ - The PDF path is validated like an output: no collision with an input, the logo,
169
+ `--out`, `--ci-out`, `--history` or `--email`, and no path inside an input
170
+ directory; a collision exits 2.
171
+ - The email with `--out` names the report; without `--out` it names the PDF
172
+ because the PDF is written first.
173
+ - The engine is optional and dynamically imported; the default install and
174
+ runtime dependency surface, report, JSON, email and CI bytes are unchanged;
175
+ the package gains `docs/PDF.md`. See [`PDF.md`](PDF.md).
176
+
177
+ ## Report locale
178
+
179
+ `--locale <tag>` localizes the HTML report, the optional email summary and the
180
+ PDF. It requires `--out`, `--pdf` or `--email`, defaults to `en`, and accepts
181
+ `en` or `de`;
182
+ `de-AT` keeps the regional tag for `<html lang>` and uses the German catalog. An
183
+ empty or unknown tag exits 2 with the supported list before anything is written;
184
+ the API throws instead. The JSON summary, CI text, exit codes, summary keys and
185
+ the default title `Tolkira report` are unchanged. Test data renders verbatim in
186
+ every locale. With only `--json` or `--ci`, `--locale` stays a usage error. See
187
+ [`I18N.md`](I18N.md).
188
+
37
189
  ## API
38
190
 
39
191
  ```ts
40
192
  import {
41
193
  generateReport,
194
+ readHistory,
195
+ renderEmail,
196
+ renderPdf,
42
197
  renderReport,
43
198
  summarizeReport,
44
199
  } from 'tolkira';
45
200
 
46
201
  const inputs = ['test-results/'];
47
202
  const summary = await summarizeReport(inputs);
48
- const { html, summary: sameSummary } = await generateReport(inputs, {
203
+ const previousRuns = await readHistory('.history/history.json');
204
+ const { html, summary: sameSummary, skippedAttachments, history } =
205
+ await generateReport(inputs, {
206
+ title: 'Nightly',
207
+ locale: 'de',
208
+ metadata: { CI: 'true', Commit: process.env.GITHUB_SHA ?? '' },
209
+ logo: 'data:image/png;base64,...',
210
+ accent: '#1b5e20',
211
+ history: previousRuns,
212
+ });
213
+ const htmlOnly = await renderReport(inputs, { title: 'Nightly', locale: 'de' });
214
+ const email = renderEmail(summary, {
49
215
  title: 'Nightly',
50
- metadata: { CI: 'true', Commit: process.env.GITHUB_SHA ?? '' },
51
- logo: 'data:image/png;base64,...',
216
+ locale: 'de',
217
+ reportName: 'report.html',
52
218
  accent: '#1b5e20',
219
+ now: new Date('2026-11-20T09:30:00.000Z'),
53
220
  });
54
- const htmlOnly = await renderReport(inputs, { title: 'Nightly' });
221
+ const pdf = await renderPdf(html, { timeoutMs: 300_000 });
55
222
  ```
56
223
 
224
+ `renderEmail(summary, options)` is synchronous and returns the same HTML the CLI
225
+ writes for the same summary, options and clock. `EmailOptions` has `title`,
226
+ `locale`, `reportName`, `accent` and `now`; there is no logo option. It writes
227
+ nothing and sends nothing. Invalid accents are ignored, an unrecognised summary
228
+ status falls back to the `Unknown` label and colour, and an invalid `now` omits
229
+ the generated date. The title, test names and report name are escaped; the report
230
+ note keeps only the basename. The CLI strips control characters from its
231
+ `Email written to` confirmation.
232
+
233
+ `renderPdf(html, options)` returns the PDF bytes for already rendered report
234
+ HTML. `PdfOptions` has only the optional `timeoutMs` (default five minutes). It
235
+ writes no file, starts no server, aborts HTTP(S) subresources and uses the same
236
+ optional Chromium engine as the CLI; a missing engine throws an install error.
237
+ The CLI and API produce equivalent PDFs for the same HTML and engine.
238
+
239
+ `skippedAttachments` is the number of Playwright path attachments skipped
240
+ because they resolve outside every allowed root. It is a plain number, additive
241
+ to the returned object; the run-level and per-scenario HTML notices and the CLI stderr warning use it, while
242
+ the summary document keeps schema version 2 and no warning field.
243
+
244
+ `history` is the aggregate run list the HTML was rendered with; it already
245
+ contains the current run. `generateReport` appends the current run to
246
+ `options.history` and returns the appended list. `renderReport` and
247
+ `CucumberHtmlStream` render `options.history` exactly as given and record
248
+ nothing. Both fields are additive.
249
+
57
250
  The package is ESM, Node.js 22 or newer, and has no network or telemetry code.
58
251
 
59
252
  | Option | CLI | API |
60
253
  | ------ | --- | --- |
61
- | title | `--title` | `title` |
254
+ | title | `--title` sets the report heading, Markdown CI heading and email heading/suggested subject | `title` |
62
255
  | metadata | `--meta key=value` (repeatable) | `metadata` object |
63
256
  | logo | `--logo <file>` reads a local image | `logo` base64 image data URI |
64
257
  | accent | `--accent #rrggbb` | `accent` hex string; invalid values are ignored |
258
+ | history | `--history <file>` reads and appends a sidecar; needs `--out` | `history` array; `generateReport` appends the current run, `renderReport` and the stream render it as given |
259
+ | locale | `--locale <tag>` (`en` or `de`; needs `--out`, `--pdf` or `--email`) | `locale` string; default `en` |
260
+ | email | `--email <file>` writes an HTML summary | `renderEmail(summary, options)` |
261
+ | pdf | `--pdf <file>` prints the report with optional Playwright Chromium; `--pdf-timeout <ms>` (needs `--pdf`) overrides the deadline | `renderPdf(html, { timeoutMs })` |
65
262
  | size warning | `--max-size <mb>` (default 10) | not applicable; the API returns HTML |
263
+ | CI output | `--ci github\|markdown`, `--ci-out <file>` | not applicable; format the summary yourself |
66
264
 
67
265
  The CLI validates logo bytes and rejects unsupported types or a size over
68
266
  2 MB. The API only accepts a `data:image/...;base64,...` logo; anything else is
69
267
  omitted.
70
268
 
71
- ## Summary schema, version 1
269
+ ## Messages stream
270
+
271
+ `CucumberHtmlStream` is a shape-compatible Node `Transform` for callers that
272
+ already hold Cucumber Messages envelopes:
273
+
274
+ ```ts
275
+ import { Readable } from 'node:stream';
276
+ import { CucumberHtmlStream } from 'tolkira';
277
+
278
+ const html = await new Promise<string>((resolve, reject) => {
279
+ let output = '';
280
+ Readable.from(envelopes)
281
+ .pipe(new CucumberHtmlStream({ title: 'Nightly' }))
282
+ .on('data', (chunk) => (output += chunk))
283
+ .on('error', reject)
284
+ .on('end', () => resolve(output));
285
+ });
286
+ ```
287
+
288
+ The stream accepts `Envelope` objects in object mode, buffers the run and
289
+ renders one HTML document on flush. It takes `RenderReportOptions` (title,
290
+ metadata, logo, accent, locale and the optional history list) and produces the
291
+ same HTML for the same envelopes and options. A supplied `history` is rendered
292
+ exactly as given; only `generateReport` appends the current run. The stream is
293
+ not an incremental renderer, writes no files and emits no summary; use
294
+ `summarizeReport` or `generateReport` when you also need the stable JSON. The
295
+ whole run is held in memory, exactly like the file API.
296
+
297
+ ## Run history
298
+
299
+ `--history <file>` keeps an aggregate sidecar of prior runs and renders a
300
+ `Trends` tab once the report is generated with two or more entries. It requires
301
+ `--out`; without the flag no history is read or written and the report contains
302
+ no history data or Trends tab.
303
+
304
+ ```shell
305
+ tolkira test-results/ --out report.html --history .history/history.json
306
+ ```
307
+
308
+ The sidecar is UTF-8 JSON with schema version 1:
72
309
 
73
310
  ```json
74
311
  {
75
312
  "schemaVersion": 1,
313
+ "runs": [
314
+ {
315
+ "id": "0a1b2c3d4e5f60718293a4b5c6d7e8f90a1b2c3d4e5f60718293a4b5c6d7e8f9",
316
+ "recordedAt": "2026-11-19T06:00:00.000Z",
317
+ "title": "Nightly",
318
+ "status": "failed",
319
+ "counts": {
320
+ "features": 3,
321
+ "scenarios": {
322
+ "total": 10,
323
+ "passed": 6,
324
+ "failed": 2,
325
+ "skipped": 1,
326
+ "pending": 1,
327
+ "undefined": 0,
328
+ "ambiguous": 0,
329
+ "unknown": 0
330
+ },
331
+ "steps": {
332
+ "total": 42,
333
+ "passed": 34,
334
+ "failed": 3,
335
+ "skipped": 2,
336
+ "pending": 1,
337
+ "undefined": 1,
338
+ "ambiguous": 1,
339
+ "unknown": 0
340
+ }
341
+ },
342
+ "durationMs": 12345.678,
343
+ "attachments": { "total": 2, "bytes": 4096 }
344
+ }
345
+ ]
346
+ }
347
+ ```
348
+
349
+ Contract:
350
+
351
+ - `schemaVersion` is `1`. A wrong version is rejected; unknown fields inside
352
+ version 1 are ignored and the reader copies only the documented fields.
353
+ - `id` is the SHA-256 of the aggregates and the resolved title, excluding
354
+ `recordedAt`. When the last entry has the same id, the new run is not
355
+ appended and the old `recordedAt` stays.
356
+ - The sidecar keeps the newest 30 runs; older entries are dropped on read, on
357
+ append and on write. A file above 1 MiB is rejected before it is read.
358
+ - `title` is the resolved report title (`Tolkira report` when `--title` is
359
+ absent), capped at 200 characters including a trailing `…`; a longer title is
360
+ truncated before it is hashed or written. `durationMs` is the summary
361
+ duration, not wall-clock time.
362
+ - Only aggregates are stored: no scenario names, step text, paths, tags or
363
+ attachment content. The report embeds the prior-run aggregates it was
364
+ generated with, so anyone who receives the HTML can read the pass-rate
365
+ history. Use one sidecar per suite and variant; mixing runs makes the trend
366
+ meaningless.
367
+ - A missing file starts empty. Invalid JSON, a wrong schema version, an
368
+ oversized file, a malformed entry or a write failure exits 1 and names the
369
+ file. A bad file is never overwritten. The HTML is written first; a failed
370
+ history write leaves it in place and leaves no partial sidecar.
371
+ - `--history` follows the output-path rules: it must not name an input, the
372
+ logo, `--out`, `--ci-out`, `--email` or a path inside an input directory, even
373
+ when the history file does not exist yet.
374
+ - The CLI (through `generateReport`) is the only automatic writer. API callers
375
+ write through `writeHistory`; `renderReport` and `CucumberHtmlStream` render a
376
+ given list and record nothing. The writer reads, appends and writes atomically
377
+ with a rename; that prevents a corrupt file, not lost updates from two
378
+ concurrent processes. Run one generation per sidecar at a time.
379
+ - The 30-run bound applies to the CLI and `generateReport` path through
380
+ `readHistory` and `appendHistory`; `renderReport` and `CucumberHtmlStream`
381
+ accept the list they are given.
382
+
383
+ The package exports `HistoryRun`, `HISTORY_SCHEMA_VERSION`, `HISTORY_LIMIT`,
384
+ `HISTORY_MAX_BYTES`, `HISTORY_MAX_TITLE`, `buildHistoryRun`, `appendHistory`,
385
+ `readHistory` and `writeHistory` for programmatic use.
386
+
387
+ ## Summary schema, version 2
388
+
389
+ ```json
390
+ {
391
+ "schemaVersion": 2,
76
392
  "status": "failed",
77
393
  "durationMs": 12.4,
78
394
  "counts": {
@@ -108,6 +424,7 @@ omitted.
108
424
  "uri": "features/screenshot.feature",
109
425
  "feature": "Screenshot attachment",
110
426
  "status": "failed",
427
+ "executed": true,
111
428
  "durationMs": 10,
112
429
  "attempt": 0,
113
430
  "tags": ["@failing"],
@@ -129,51 +446,205 @@ omitted.
129
446
  This example is abridged: it shows only the failing scenario, not the passing
130
447
  one that carries the screenshot. The totals reflect the whole input.
131
448
 
449
+ Undefined and ambiguous Messages steps add optional diagnostic fields:
450
+
451
+ ```json
452
+ {
453
+ "keyword": "Given",
454
+ "text": "a list of 8 things",
455
+ "status": "undefined",
456
+ "durationMs": 0,
457
+ "suggestions": [
458
+ {
459
+ "language": "typescript",
460
+ "code": "Given(\"a list of {int} things\", (int: number) => {\n return \"pending\"\n})"
461
+ },
462
+ {
463
+ "language": "typescript",
464
+ "code": "Given(\"a list of {float} things\", (float: number) => {\n return \"pending\"\n})"
465
+ }
466
+ ]
467
+ }
468
+ ```
469
+
470
+ ```json
471
+ {
472
+ "keyword": "Given",
473
+ "text": "a step with multiple definitions",
474
+ "status": "ambiguous",
475
+ "durationMs": 0,
476
+ "definitions": [
477
+ {
478
+ "pattern": "^a (.*?) with (.*?)$",
479
+ "source": "samples/ambiguous/ambiguous.ts:3"
480
+ },
481
+ {
482
+ "pattern": "^a step with (.*?)$",
483
+ "source": "samples/ambiguous/ambiguous.ts:7"
484
+ }
485
+ ]
486
+ }
487
+ ```
488
+
132
489
  Field contract:
133
490
 
134
- - `schemaVersion` is `1`. A breaking change increments it.
491
+ - `schemaVersion` is `2`. A breaking change increments it.
135
492
  - `status` is one of `passed`, `failed`, `skipped`, `pending`, `undefined`,
136
493
  `ambiguous`, `unknown`.
137
- - `status` is the worst scenario status. A scenario is the worst step status,
138
- using the standard Cucumber severity order.
139
- - `counts.features` is the number of distinct Gherkin documents.
140
- - `counts.scenarios` and `counts.steps` always contain every status key, even
141
- when the value is `0`. `total` is the sum.
142
- - `durationMs` is a number of milliseconds. `steps[].durationMs` is rounded to
143
- three decimals.
494
+ - `status` is the worst executed scenario status, or `failed` when a test-run
495
+ hook fails. A scenario is the worst step status, using the standard Cucumber
496
+ severity order. When nothing executed, the run status is `unknown` unless a
497
+ test-run hook failed.
498
+ - `counts.features` is the number of features with at least one scenario row,
499
+ executed or unexecuted, matching the HTML feature list. A document whose
500
+ pickles never started contributes no row and is not counted.
501
+ - `counts.scenarios` counts executed scenarios only, so `total` can be smaller
502
+ than `scenarios.length`. `counts.scenarios` and `counts.steps` always contain
503
+ every status key, even when the value is `0`. `total` is the sum. The HTML
504
+ pass rate and feature pass text use attempted (non-skipped) scenarios, while
505
+ counts, pills and bars include skipped results.
506
+ - `scenarios[].executed` is `true` when the selected test case reported at
507
+ least one raw step result, including a hook result. Adapters represent a
508
+ reported skip without steps as a synthetic skipped step, so the summary
509
+ reports it as executed. An unexecuted scenario keeps `status: "unknown"` and
510
+ `executed: false`; it is not counted in `counts.scenarios`. A pickle that
511
+ never started has no entry at all.
512
+ - `durationMs` is a number of milliseconds and sums the final attempt per
513
+ scenario plus test-run hooks. It is not wall-clock time.
514
+ `steps[].durationMs` is rounded to three decimals.
144
515
  - `scenarios` is sorted by `uri`, then `name`, so output is stable across runs.
145
- - `attempt` is the latest Cucumber retry attempt for that scenario.
516
+ - `scenarios[].name` is the Cucumber pickle name. Examples of one scenario
517
+ outline can share the same name; the HTML report labels rows with `#1.1` and
518
+ shows the examples block name on the examples heading and in triage and
519
+ copy-failure text, but the summary does not expose them yet.
520
+ - `scenarios[].tags` is the inherited pickle tag set, including feature, rule,
521
+ scenario and examples tags. The HTML report uses the same set for its tag
522
+ filter and search. Feature and rule tags render once at their section;
523
+ scenario tags render on the scenario row or, for an outline, on the outline
524
+ heading, and examples tags on the examples heading. Example rows repeat
525
+ neither scenario nor examples tags. The summary does not expose rules,
526
+ backgrounds or the own/inherited distinction. A scenario whose pickle tag set
527
+ (own or inherited) includes `@flaky` (lowercase, exact match) gets a
528
+ report-only (HTML/PDF) badge, and when at least one exists the run metadata
529
+ gets a separate count such as `1 tagged @flaky`, distinct from the status
530
+ counts. Feature, rule and outline-level `@flaky` tags badge every affected row,
531
+ including every outline example, so the count can exceed the number of visible
532
+ tag labels; a failed or unexecuted tagged scenario also counts. The count is
533
+ not a computed flake rate or detection, and neither badge nor count appears in
534
+ the summary, CI or email or changes a status.
535
+ - `attempt` is the latest started Cucumber attempt for that scenario, even when
536
+ that attempt reported no result. An earlier retried attempt is ignored. For a
537
+ Playwright JSON input it is the Playwright retry index of the final result;
538
+ earlier attempts are not exposed as scenario rows.
146
539
  - `attachments.bytes` is the decoded size, not the base64 length.
147
540
  - `steps[].message` is present only when the step result has a message. For
148
- hooks, `keyword` is `"Hook"` and `text` is the hook type.
541
+ hooks, `keyword` is `"Hook"`, `text` is the hook type and `name` is present
542
+ when Cucumber provides one. Summary step counts include hook entries; the
543
+ HTML report counts steps and hooks separately. A failed test-run hook makes
544
+ the overall summary `status` failed.
545
+ - `steps[].suggestions` is present only for an undefined Messages step with
546
+ `suggestion` envelopes. Each entry has an optional `language` and the `code`
547
+ to implement. `steps[].definitions` is present only for an ambiguous Messages
548
+ step with `stepDefinition` envelopes. Each entry has the `pattern` and an
549
+ optional `source` in `uri:line` form with absolute paths reduced to a
550
+ relative form. Both fields are additive, so `schemaVersion` stays 2;
551
+ a passed step never carries them.
149
552
  - Attachment file names are not part of the summary. `scenarios[].attachments`
150
- is a count; open the HTML report to see the image or file.
553
+ is a count and is 0 for an unexecuted scenario; open the HTML report to see
554
+ the image or file. Unnamed attachments use a deterministic
555
+ `attachment.<ext>` fallback in HTML; unknown media types normalise to
556
+ `application/octet-stream` and `attachment.bin`.
557
+ - CI metadata from Messages `meta.ci` is HTML-only. It renders in the report
558
+ header and Details, but no CI field appears in the summary and
559
+ `schemaVersion` stays 2. When `ci.name` is missing, the http(s) `ci.url` is
560
+ the link text. In a mixed input the first non-empty CI block wins, and
561
+ meta synthesized by the legacy adapter cannot erase it. Legacy JSON and
562
+ JUnit never invent CI metadata; use the `metadata` option for a legacy path.
563
+
564
+ ## Migrating to schema version 2
565
+
566
+ Version 2 adds `executed` to every scenario and changes scenario counts to
567
+ cover executed scenarios only:
568
+
569
+ - `counts.scenarios.total` can be smaller than `scenarios.length`; use the
570
+ counts for aggregate decisions.
571
+ - An unexecuted scenario has `status: "unknown"` and `executed: false`. That
572
+ `unknown` is not a failure signal.
573
+ - Filter on `executed` (or use `counts.scenarios`) before treating a
574
+ scenario's `status` as an outcome. A legacy Cucumber JSON scenario with no
575
+ steps and no hooks is now unexecuted, so its scenario total drops by one
576
+ compared with version 1.
577
+ - Never-started pickles remain absent from `scenarios` and the counts.
151
578
 
152
579
  The summary does not expose absolute input paths: absolute `uri` values are
153
- reduced to a report-relative form (`features/...`).
580
+ reduced to a relative form (`features/...`, or the last path segments for files
581
+ outside `features/`).
154
582
 
155
583
  ## Recommended agent workflow
156
584
 
157
585
  1. Run the tests and keep the raw Cucumber JSON or Messages.
158
- 2. Run `tolkira <inputs> --out report.html --json`.
159
- 3. Parse stdout once. Use `status`, `counts` and `scenarios[].steps[].message`
160
- for diagnosis.
586
+ 2. Run `tolkira <inputs> --out report.html --json`; add `--email email.html`
587
+ when a status mail or chat message is part of the job.
588
+ 3. Parse stdout once. Filter on `executed` (or use `counts`) and use `status`,
589
+ counts and `scenarios[].steps[].message` for diagnosis.
161
590
  4. If you need screenshots or the full Gherkin view, open `report.html`; do not
162
591
  scrape it for pass/fail data.
163
- 5. Attach or upload the HTML report as the human-facing artifact.
592
+ 5. Attach or upload the HTML report as the human-facing artifact; use
593
+ `email.html` as optional mail body, never as the only record.
164
594
 
165
595
  ## Current limits
166
596
 
167
597
  - Remote attachments (`attachment.url`, `externalAttachment`) are rejected on
168
598
  purpose; the report must work offline. Inline the bytes.
169
- - Markdown images with remote URLs are stripped from descriptions.
599
+ - Description Markdown renders only in the HTML report. Remote image
600
+ destinations are neutralised and images render as alt text, so the report
601
+ loads no external asset. JUnit `description` attributes are a best-effort,
602
+ non-standard mapping; they never appear in the summary.
603
+ - Tag-expression search, the tag dropdown and `?query=`/`?hide=`/`?tag=` URL
604
+ state are HTML-only progressive enhancements. `?hide=` also accepts
605
+ `unexecuted`, a pseudo-status for the failed-only preset that hides rows
606
+ without an outcome; it is restored only with the preset chip state and a
607
+ pruned value is removed from the URL. The summary schema and counts are
608
+ unchanged; the static no-JS report keeps the inert control and all rows.
170
609
  - Inputs are capped at 100 MB in total; attachments at 50 MB each and 100 MB
171
610
  total.
172
611
  - The HTML report includes a hash-based CSP and never executes attachments.
173
- - History, trends, retries beyond the latest attempt and JUnit/Playwright
174
- adapters are not part of v0.1.
612
+ - History is an opt-in aggregate sidecar (`--history`); there is no
613
+ scenario-level history, dashboard or server. Retries beyond the latest attempt
614
+ and a trace viewer are not supported. Playwright JSON has a native adapter;
615
+ the final attempt is the rendered one, and `@flaky` is report-only
616
+ (HTML/PDF): the summary, CI and email keep the final status. JUnit XML is a basic input mode; see
617
+ [`LIMITATIONS.md`](LIMITATIONS.md). Playwright producer metadata and
618
+ annotation descriptions are HTML-only and do not change schema version 2.
619
+ - The email summary is opt-in and file-only. It has no sending, recipients,
620
+ scheduling, `.eml` draft, logo, error snippets or redaction mode; the report
621
+ basename (or the no-report note) is the only path-like content.
622
+ - PDF export is opt-in, Chromium-only and file-only. It is A4 portrait only
623
+ (no Letter/custom paper), with no page numbers, custom header/footer, tagged
624
+ PDF, CLI filter, forced landscape, PDF-only logo or PDF-only Trends; it needs a
625
+ separate project-local engine and browser install. The default Chromium launch
626
+ is not OS-sandboxed; see [`PDF.md`](PDF.md).
627
+ - Playwright path attachments are embedded only when the resolved regular file
628
+ is inside the input file's directory or the process working directory.
629
+ Symlinked, missing, non-regular and outside-tree paths are skipped; inline
630
+ base64 attachments are always preferred. An outside-tree path that exists as a
631
+ regular file is counted in `generateReport().skippedAttachments` and surfaces
632
+ as run-level and per-scenario HTML notices plus a CLI stderr warning; later-missing files, symlinks and
633
+ the other skipped causes are not counted. Trace zips and HTML-report replays
634
+ are not read.
175
635
  - Legacy Cucumber JSON outlines arrive as expanded scenarios; the original
176
- outline `examples` table is not reconstructed. Behave-shaped durations are
177
- interpreted as nanoseconds, like cucumber-js.
178
- - cucumber-js, Behave-shaped and Messages inputs are covered by fixtures;
179
- cucumber-jvm and cucumber-ruby fixtures are follow-ups.
636
+ outline `examples` table is not reconstructed. Behave durations are seconds
637
+ and are normalised to milliseconds; multiline doc strings and error messages
638
+ arrive as line arrays and are joined with newlines. Behave's `error` and
639
+ `hook_error` step-result statuses map to failed; `cleanup_error`, which Behave
640
+ records on the scenario/feature during context cleanup, is accepted
641
+ defensively as a failed step result.
642
+ - Rules and backgrounds are an HTML presentation feature. The version-2 summary
643
+ keeps the flat scenario list; `scenarios[].tags` remains the inherited tag
644
+ set. Legacy JSON and JUnit never invent rule or background structure.
645
+ - Legacy Cucumber JSON and JUnit have no suggestion or step-definition
646
+ envelopes. An undefined or ambiguous legacy step still shows the status lead
647
+ in HTML, but never invented snippets or definitions; JUnit renders no
648
+ diagnostics.
649
+ - cucumber-js, Behave, cucumber-jvm, cucumber-ruby and Messages inputs are
650
+ covered by fixtures.