e2e 0.15.1 → 0.16.0

package/docs/reference/reporters.mdx

@@ -0,0 +1,216 @@
+---
+title: Reporters
+description: Receive run events and publish results with a custom reporter.
+---
+
+Add reporter objects to `reporters` in `e2e.config.ts`, alongside the
+[built-in reporters](/reference/cli#output). Use `onEvent` for live progress
+and `onRunFinished` to process the final report and artifacts.
+
+```ts
+import type { E2EConfig, Reporter } from 'e2e';
+import { web } from '@e2e-dev/web';
+
+const upload: Reporter = {
+  name: 'upload',
+  async onRunFinished(run) {
+    const response = await fetch('https://example.test/runs', {
+      method: 'POST',
+      body: JSON.stringify(run.report),
+    });
+    const { url } = (await response.json()) as { url: string };
+    return [{ label: 'Results', text: url }];
+  },
+};
+
+export default {
+  targets: [{ engine: web(), app: { url: 'localhost:3000' } }],
+  reporters: ['list', upload],
+} satisfies E2EConfig;
+```
+
+## The contract
+
+```ts
+interface Reporter {
+  readonly name: string;
+  onEvent?(event: RunEvent): void;
+  onRunFinished?(run: FinishedRun, signal: AbortSignal): Promise<ReporterSummary | void>;
+}
+
+interface FinishedRun {
+  readonly report: Report;                 // the report-1 document, as written
+  readonly status: RunStatus;              // report.run.status
+  readonly exitCode: RunExitCode;
+  readonly projectRoot: string;            // what the terminal shows paths relative to
+  readonly reportPath: string | undefined; // absent when the write failed
+  readonly artifactsRoot: string;          // what the report's artifact paths are relative to
+  readonly aiTracePath: string | undefined;
+  readonly lastRun?: Report;               // with --last-failed, the report the rerun selected from
+}
+
+type ReporterSummary = readonly { label: string; text: string }[];
+```
+
+`Reporter`, `FinishedRun`, `ReporterSummary`, `Report`, `RunEvent`, and
+`RunEventOf` are exported from the main entrypoint.
+
+The built-in `junit` reporter is the smallest complete example: it reads
+`run.report`, writes `junit.xml` beside `run.reportPath`, and returns one row
+naming the file. `markdown` does the same with `summary.md` through
+`renderMarkdownReport(report, options)`, also exported, so your own reporter
+can post the same page elsewhere; `failurePages` links each block to a page
+by result id. The built-in reporter also writes one page per failed or flaky
+test under `failures/`, with the screen at failure inline. [`@e2e-dev/github`](/github) is that
+reporter for GitHub Actions.
+
+A name and at least one handler are required; anything else is
+`INVALID_CONFIG`.
+
+## The event stream
+
+`onEvent` receives run progress as plain JSON. The `list` reporter uses the
+same events. Each event has a sequence number, `seq`, and an ISO timestamp,
+`at`.
+
+```ts
+type RunEventFact =
+  | { type: 'run-started'; runId; projectId; projectRoot; artifactsRoot; ci; targets; agents?; model?; judge? }
+  | { type: 'plan'; total; files: { file; target; tests }[] }
+  | { type: 'notice'; target; message }
+  | { type: 'output'; target; pair; stream: 'stdout' | 'stderr'; text }
+  | { type: 'setup'; step: SetupStep; state: 'started' }
+  | { type: 'setup'; step: SetupStep; state: 'finished'; durationMs; outcome?: 'reused' }
+  | { type: 'test-started'; testId; agent; repeat; title; file; serialId; target }
+  | { type: 'step'; testId; agent; repeat; target; progress: StepProgress }
+  | { type: 'test-finished'; result: RunEventResult }
+  | { type: 'serial-group'; group: SerialGroupRecord }
+  | { type: 'explore'; progress: ExploreProgress }
+  | { type: 'run-error'; error: SerializedError }
+  | { type: 'run-interrupted'; mode: 'graceful' | 'forced' }
+  | { type: 'run-stopped'; failures; limit }
+  | { type: 'run-finished'; status; exitCode; reportPath?; aiTracePath? };
+```
+
+`output` is one write a test made to stdout or stderr; `pair` names the test
+running at the time and is undefined between tests. `judge` on `run-started`
+is the run agent's judge model, present only when judgments use a model other
+than `model`.
+
+The `phase` field describes step progress:
+
+- `start` and `end` mark the step's boundaries.
+- `event` reports a completed model turn, tool call, or observation.
+- `activity` reports work starting: `observe`, `action`, or `model`.
+- `replay` reports when replay from the cache starts or returns control to the agent.
+
+```ts
+type StepProgress = (
+  | { phase: 'start'; kind: StepKind; api: string; label: string }
+  | {
+      phase: 'end';
+      kind: StepKind;
+      api: string;
+      label: string;
+      status: 'passed' | 'failed' | 'blocked' | 'timed-out' | 'cancelled';
+      durationMs: number;
+      modelCalls: number;
+      error?: SerializedError;
+      explanation?: string;
+    }
+  | { phase: 'event'; api: string; event: StepEvent }
+  | { phase: 'activity'; api: string; activity: 'observe' | 'model' | 'action' }
+  | { phase: 'replay'; api: string; active: boolean }
+) & { identity?: StepProgressIdentity };
+```
+
+Every step phase emitted by the runner includes `progress.identity`:
+
+```ts
+{ attemptId: string; attemptIndex: number; stepId: string; stepIndex: number }
+```
+
+The IDs match the final report. Both indexes start at zero. A retry has a
+new attempt ID and a higher attempt index; its step indexes start again at
+zero. Serial-group members share the group's attempt ID and index, while
+each member's step IDs remain distinct. Use `stepId` to match start, activity,
+event, replay, and end phases, including nested steps. `identity` is optional
+in the type so a reporter can also accept older streams.
+
+An `end` phase also includes the step's redacted `error` and `explanation`
+when present. Preserve its `status`, including `blocked` and `cancelled`,
+instead of treating every non-passed step as a product failure.
+
+`explore` appears only in an `e2e explore` run: `started`, `planning`,
+`step-started`, `step-finished`, `finding`, and `finished`. The record of all
+of it is `run.explore` in the report.
+
+New event types are added over time. Handle the types you know and ignore
+the rest; leave the `default` branch of a `switch` empty.
+
+`run-started` comes first and `run-finished` last. `test-finished` carries the
+full result record, including every attempt's artifacts. A test pinned to
+several agents starts and finishes once per agent, so key running state by
+test id, target, and agent together. Ordinary tests emit `test-started` once
+for that combination. Serial members emit it on each group attempt they
+execute. Use step progress identity to distinguish retries. An attempt
+that fails before any step emits no step progress and appears in the final
+result. Secrets are redacted before events are
+built. Error messages are only stripped of control characters; treat them as
+untrusted.
+
+`onEvent` must not block and must not throw. A throw quarantines the reporter
+for the rest of the run and prints one line on stderr naming the reporter, the
+event type, and the error; the run's outcome is unchanged. Slow work belongs
+in `onRunFinished`.
+
+## The finished run
+
+`onRunFinished` runs after the report is written and the terminal summary
+prints. All reporters run concurrently, each with a one-minute deadline.
+A timeout, exception, or invalid return value is reported on stderr without
+changing the run's status, exit code, or report.
+
+One Ctrl-C still lets reporters finish. A second Ctrl-C stops waiting for them.
+
+The second argument is an `AbortSignal` that aborts when the budget runs out
+or the run is forced to stop. Hand it to every request and timer, or an
+abandoned reporter keeps the CLI alive after the run has ended.
+
+With `--last-failed`, `lastRun` is the report the rerun selected from. The
+rerun's own report lists the tests it left out as unselected, so a reporter
+that keeps one place current, the way `@e2e-dev/github` keeps a pull request
+comment, folds the rerun into `lastRun` to show both runs as one.
+
+The rows a reporter resolves with print under the `list` summary in the same
+layout as `Report`; the `json` reporter does not carry them:
+
+```
+   Test Files  2 passed (2)
+        Tests  6 passed (6)
+     Start at  14:02:11
+     Duration  8.41s
+       Report  .e2e/report.json
+
+        JUnit  .e2e/junit.xml
+      Results  https://example.test/runs/0192c…
+```
+
+Artifact paths in the report are POSIX paths relative to `artifactsRoot`.
+`path`, `size`, and `sha256` are optional: an artifact whose file never
+appeared, or that a redaction withheld, is recorded without them. Check for
+`path` before reading. A video a hosted service keeps has an `http(s)` `url`
+in place of `path`.
+
+## What `--reporter` does to reporter objects
+
+`--reporter` replaces only the built-in reporters; configured reporter
+objects still run. A config with only reporter objects has no built-in
+terminal output.
+
+## Where a reporter runs
+
+Handlers run in the runner process. Workers also load the config, so reporter
+construction must have no side effects. Read tokens and start requests inside
+handlers. Adding a reporter leaves [cached traces](/cache#what-must-match)
+valid.