squally-reporter 0.19.0 → 0.19.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (2) hide show
  1. package/README.md +751 -751
  2. package/package.json +54 -54
package/README.md CHANGED
@@ -1,751 +1,751 @@
1
- # squally-reporter
2
-
3
- Playwright reporter that sends test results to your Squally dashboard.
4
-
5
- Learn more at [app.squally.dev](https://app.squally.dev).
6
-
7
- ## Requirements
8
-
9
- - Node.js >= 20.18.1
10
- - `@playwright/test` >= 1.30.0
11
-
12
- ## Install
13
-
14
- ```bash
15
- npm install -D squally-reporter
16
- ```
17
-
18
- ## Config
19
-
20
- Set one env var (e.g. as a CI secret):
21
-
22
- ```bash
23
- SQUALLY_INGEST_KEY=your-project-ingest-key
24
- ```
25
-
26
- Find your ingest key in the Squally dashboard under **Settings › Project ›
27
- Ingest key**.
28
-
29
- **Since 0.12.0 the URL defaults to `https://app.squally.dev`.** Set
30
- `SQUALLY_INGEST_URL` only to report to another Squally. Without a key the
31
- reporter sends nothing and says so once.
32
-
33
- **Renamed in 0.9.0.** These were `SQUALLY_API_URL` and `SQUALLY_API_KEY`. The
34
- old names still work for now, with a deprecation warning in the CI log - but
35
- they belong to Squally's read API now, so rename them. `SQUALLY_API_URL` is
36
- only used together with `SQUALLY_API_KEY`, never on its own - and when it is
37
- set but not used, the reporter sends nothing rather than falling back to the
38
- default URL, since the variable says you meant another target.
39
-
40
- **A read key is never sent.** A key starting with `sqly_ro_` is a read API key,
41
- for scripts and agents; reporting needs the ingest key. Since 0.12.0 the
42
- reporter refuses one in any of `SQUALLY_INGEST_KEY`, `SQUALLY_API_KEY` or the
43
- `apiKey` option before sending anything, and says which one to fix.
44
-
45
- Then add the reporter to `playwright.config.ts`:
46
-
47
- ```ts
48
- import { defineConfig } from "@playwright/test";
49
-
50
- export default defineConfig({
51
- reporter: [
52
- ["html"],
53
- ["squally-reporter"],
54
- ],
55
- });
56
- ```
57
-
58
- **Replace your `reporter` line, don't add a second one.** `npm init playwright`
59
- writes `reporter: 'html',`, and with two `reporter:` keys in the config the
60
- later one wins without a warning - either the HTML report or Squally silently
61
- goes. Using other reporters? Keep them in the list and add
62
- `["squally-reporter"]`.
63
-
64
- Alternatively, pass config explicitly instead of using env vars - but **prefer
65
- the env vars above for `apiKey`, and not only because a literal in a committed
66
- file is obviously wrong.** Playwright's `json` and `blob` reporters serialize
67
- your resolved config into their output file, options and all: an `apiKey`
68
- passed here ends up in that file in plain text, even when its value came from
69
- an environment variable at runtime. What gets written is the option, not where
70
- it came from. The reporter warns, naming the file, when it sees this
71
- combination. Measured against a real report, 20.09.2026.
72
-
73
- ```ts
74
- reporter: [
75
- ["squally-reporter", { apiUrl: "https://app.squally.dev", apiKey: process.env.SQUALLY_INGEST_KEY }],
76
- ],
77
- ```
78
-
79
- That's it - run your tests as usual (`npx playwright test`); results stream
80
- to Squally while the suite runs (see *How results reach Squally* below).
81
-
82
- ## Artifact upload (screenshot, trace, browser console)
83
-
84
- If your `playwright.config.ts` captures screenshots or traces, the reporter
85
- uploads them to Squally so they show up next to the failing test in the
86
- dashboard - no separate setup beyond the two env vars above.
87
-
88
- Three kinds are uploaded:
89
-
90
- | Kind | Where it comes from |
91
- | --- | --- |
92
- | **Screenshot** | the file Playwright writes for a failed attempt |
93
- | **Trace** | the trace zip Playwright writes |
94
- | **Browser console** | read out of that trace zip by this reporter |
95
-
96
- **The browser console needs a trace.** It is not something Playwright hands a
97
- reporter: `result.stdout`/`stderr` are the output of the *Node* process - what
98
- your test file printed - while what the *page* printed only exists inside the
99
- trace. So the reporter opens the trace and reads it out, which means an
100
- attempt with no trace has no browser console, and nothing can change that but
101
- turning tracing on. Squally's attempt page says which of the two it is rather
102
- than showing an empty section.
103
-
104
- Supported trace schema versions: **5 to 10**, which covers Playwright 1.30
105
- through 1.62 (1.62 writes version 8). A trace outside that range is left alone
106
- and the reporter prints one warning naming the version - it never uploads an
107
- empty console file, because "no messages" and "this reporter could not read
108
- it" must not look the same in the dashboard.
109
-
110
- **Only from attempts worth looking at:** artifacts are uploaded only for
111
- attempts that **failed**, or that **needed a retry** (even if that retry then
112
- passed) - never for an attempt that passed on its first try, regardless of how
113
- `screenshot`/`trace` are configured. Nobody looks at a screenshot of a test
114
- that just passed, and uploading one anyway would only eat into your
115
- organization's storage for no benefit.
116
-
117
- **Uploaded while the suite runs (0.18.0).** An attempt's artifacts go up as
118
- soon as Squally has accepted the batch that carries its result - not only
119
- after the last test, as until 0.17.0. A job that is cancelled or killed
120
- partway keeps the artifacts of every result that already arrived. When the
121
- suite ends, the reporter waits **up to 3 minutes** for the uploads still
122
- running; the last 15 seconds of that only finish the confirmation, no new
123
- upload starts in them. A file is sent as a stream, not read into memory
124
- first.
125
-
126
- **You need to actually enable capture in `playwright.config.ts`** - without
127
- it, Playwright doesn't produce any attachments, so there's nothing for the
128
- reporter to find or upload (no error either way, it's a silent no-op):
129
-
130
- ```ts
131
- export default defineConfig({
132
- use: {
133
- screenshot: "only-on-failure",
134
- trace: "retain-on-failure",
135
- },
136
- // ...
137
- });
138
- ```
139
-
140
- **Limits.** None of these are enforced by the reporter - Squally applies them
141
- when the upload is requested and answers with a reason, which is what the
142
- reporter prints. The numbers below are therefore Squally's current ones rather
143
- than anything this package guarantees, and can change server-side without a new
144
- release of it:
145
-
146
- - **50 MB per screenshot or trace, 4 MB per browser console**, the same on
147
- every plan. A larger file is rejected before any bytes are uploaded; the
148
- reporter logs a `console.warn` naming the file and its limit and moves on.
149
- The test result itself is reported as usual, and the run's outcome and exit
150
- code are untouched. In the dashboard that artifact is shown as skipped, with
151
- the reason, rather than silently missing.
152
- - **At most 250 artifacts per request.** Since 0.18.0 the reporter sends them
153
- in rounds of at most 250, so a shard with more no longer loses the rest.
154
- Until 0.17.0 a shard's artifacts went in one request, and Squally kept the
155
- first 250.
156
- - **A total storage cap per organization, depending on your Squally plan.**
157
- Once it is reached, further uploads are rejected the same way until you free
158
- up space or upgrade. Note that a fully instrumented suite now produces up to
159
- three artifacts per failed attempt rather than two.
160
- - The browser console is a text file of a few KB - it is extracted from the
161
- trace you are already uploading, so it adds no capture cost to your run.
162
- - **Your test's own `stdout`/`stderr` is cut at 16 KB** before it is sent. That
163
- is this reporter's own limit, not Squally's, and it applies whether or not
164
- you capture artifacts at all.
165
- - Two minutes per upload, and 30 seconds for the calls that surround it. Since
166
- 0.18.0 a network error, an HTTP 5xx or a 429 is retried like the results
167
- (see *When Squally cannot be reached for a while*), and the 3 minutes at the
168
- end bound the whole wait. What did not arrive is said once, at the end:
169
-
170
- ```
171
- [squally-reporter] 2 of 9 artifacts did not reach Squally - the storage answered HTTP 503, and the retries ran out. Your test run and its exit code are not affected.
172
- ```
173
-
174
- Upload is best-effort and fully non-blocking, same principle as the rest of
175
- this reporter - a failed or skipped upload never fails your test run or
176
- changes its exit code, it only logs a `console.warn` explaining what
177
- happened.
178
-
179
- ## How results reach Squally
180
-
181
- As soon as the first test starts, the reporter creates the run in Squally and
182
- prints where to watch it (0.15.0), on stderr:
183
-
184
- ```
185
- [squally-reporter] Reporting to Squally: run #128 · https://app.squally.dev/o/…/runs/…
186
- ```
187
-
188
- Under `--shard` the line reads `run #128, shard 2 of 3`; every shard of a CI
189
- run has the same number.
190
-
191
- Results stream to Squally as your suite runs, not all at once at the end:
192
- each test's result is sent in a small batch, so the "running" view in the
193
- dashboard stays close to real time even for a long suite. A batch goes out on
194
- whichever of these comes first:
195
-
196
- - **25 results.** Fixed, and the same whether you shard or not.
197
- - **A timer: 2 seconds, or 0.4 seconds × the number of shards when that is
198
- longer, capped at 30 seconds** (0.12.0). Up to five shards every shard
199
- flushes every 2 seconds; eight shards every 3.2; sixteen every 6.4; from 75
200
- shards the cap holds it at 30.
201
-
202
- Only the timer scales. All shards of a run share one rate budget on Squally's
203
- side (300 requests a minute per key), and the timer keeps all of them
204
- together at about half of it. The 25-result threshold is left alone
205
- deliberately: it bounds how long a *fast* suite waits, and stretching it would
206
- delay results without bounding the burst any better.
207
-
208
- A batch also stays under 3 MB (0.12.0): results with long error texts or many
209
- steps simply go out in more, smaller requests.
210
-
211
- There is no longer a single-request size limit that could discard a whole run -
212
- a very large suite simply takes more, smaller requests. As a courtesy the
213
- reporter still warns if a run's result count climbs well past 5,000, since that
214
- usually means retries are configured higher than intended rather than a suite
215
- that's actually that large.
216
-
217
- If Squally is unreachable partway through, results already sent stay in the
218
- dashboard; only what could not be delivered is lost, and your test run's exit
219
- code is never affected either way.
220
-
221
- ## The end of a run (0.18.0)
222
-
223
- When every result arrived and the run is marked complete, the last line says
224
- where it is and what got there:
225
-
226
- ```
227
- [squally-reporter] Reported to Squally: run #128 · 412 results, 9 artifacts · https://app.squally.dev/o/…/runs/…
228
- ```
229
-
230
- If something did not arrive, that line is replaced by the one that says what
231
- is missing (see below).
232
-
233
- **`SQUALLY_OUTPUT_FILE`** writes the run's link and counts as JSON, for a
234
- later CI step that posts the link somewhere:
235
-
236
- ```bash
237
- SQUALLY_OUTPUT_FILE=test-results/squally.json
238
- ```
239
-
240
- ```json
241
- {
242
- "version": 1,
243
- "runUrl": "https://app.squally.dev/o/…/runs/…",
244
- "runNumber": 128,
245
- "runId": "…",
246
- "shard": null,
247
- "complete": true,
248
- "tests": { "passed": 40, "failed": 1, "flaky": 1, "skipped": 0 },
249
- "results": { "produced": 44, "delivered": 44 },
250
- "artifacts": { "uploaded": 3, "notUploaded": 0 }
251
- }
252
- ```
253
-
254
- - Written at the very end of every run the reporter took part in - also one
255
- Squally refused or could not be reached for, then with `runUrl: null` and
256
- `complete: false`, so the file is always there for the next step. Missing
257
- directories are created.
258
- - `tests` counts this process's tests the way Playwright does at the end
259
- (`expected` as passed, `unexpected` as failed, `flaky`, `skipped`).
260
- `results` counts attempts: produced, and accepted by Squally.
261
- - One file per process: under `--shard`, give each shard its own path (or
262
- read them one after another), and `shard` says which one it was.
263
- - A path that cannot be written is one warning; the exit code stays yours.
264
- - New fields may be added; `version` changes only if one changes meaning.
265
-
266
- ## When Squally cannot be reached for a while (0.17.0)
267
-
268
- Some failures pass on their own, and the reporter waits them out instead of
269
- dropping results:
270
-
271
- - a network error or a request without an answer within 30 seconds,
272
- - an HTTP 5xx,
273
- - HTTP 429 (Squally's rate limit),
274
- - HTTP 403 from Vercel's firewall in front of Squally - the answer carries
275
- `x-vercel-mitigated: challenge` (or `deny`). The firewall can challenge one
276
- runner for a few minutes; it says nothing about your key.
277
-
278
- Each request is retried after 1 second, then 2, 4, 8 and so on up to 30
279
- seconds between tries (each wait shortened by a random part, so shards
280
- refused together do not come back together), for up to 90 seconds. A
281
- `Retry-After` header is honoured. The start of the run gets the same 90
282
- seconds; your tests run meanwhile and their results are held until Squally
283
- answers. While Squally keeps refusing, results go out one request at a time.
284
-
285
- Not retried, because waiting does not change them: Squally's own error
286
- answers (a refused key, an exhausted quota, an unknown or finished run), an
287
- HTTP 403 *without* the firewall's header, and a redirect. Those keep their own
288
- line, as before.
289
-
290
- The artifact uploads follow the same rules since 0.18.0 (until then each was
291
- tried once): the request for the upload URL, the upload to storage (a network
292
- error, HTTP 5xx or 429) and the confirmation. They have their own line at the
293
- end and never delay the results.
294
-
295
- When your suite ends, the reporter normally waits up to 5 seconds for the last
296
- results. After one of the failures above it waits up to 60 seconds, for the
297
- last results and for marking the run complete.
298
-
299
- What did not arrive is said once, at the end - not once per batch:
300
-
301
- ```
302
- [squally-reporter] 37 of 412 results did not reach Squally - Vercel's firewall in front of Squally challenged this runner (HTTP 403, x-vercel-mitigated: challenge), and the retries ran out. Your test run and its exit code are not affected. The run page in Squally shows how many are missing.
303
- ```
304
-
305
- If the run could not be marked complete either, the line says so: the run
306
- then shows as **incomplete** in Squally once the project's run timeout has
307
- passed, with the results that did arrive. If the run could not be started at
308
- all, the line reads `Squally did not start this run - ...` and nothing was
309
- sent.
310
-
311
- The exit code of your test run is never changed by any of this.
312
-
313
- ## Steps (0.12.0)
314
-
315
- For an attempt that **failed, or needed a retry**, the reporter also sends the
316
- steps Playwright recorded - hooks, fixtures, actions, assertions and your own
317
- `test.step`s, with their durations and `file:line`. Squally shows them on the
318
- attempt page, the failing step marked. An attempt that passed on its first try
319
- sends none.
320
-
321
- - **Typed values never leave your runner.** The value of a `fill`, `type` or
322
- `pressSequentially` is replaced: `Fill "•••" getByLabel('Password')`. Every
323
- title also goes through the same secret masking as error texts.
324
- - **At most 300 steps per attempt, 200 characters per title.** A longer list
325
- keeps the failing step and the steps it sits in, then the rest in order, and
326
- tells Squally how many were left out.
327
- - Nothing to configure: steps come from Playwright's own report, with or
328
- without tracing.
329
-
330
- **Only actual test runs appear.** A run shows up in Squally once your first
331
- test starts, so commands that never run a test create nothing: `--list`,
332
- a `globalSetup` or `webServer` that fails to start, a `--grep` that matches
333
- nothing. A run that *does* execute and legitimately contains no tests — an
334
- empty shard, `--only-changed` with no changed test file, `--pass-with-no-tests`
335
- — is still reported, as a run with zero results.
336
-
337
- **Timeouts and interrupted tests.** A test that runs out of time is reported
338
- as failed, and Squally is told that it timed out and what its timeout was -
339
- including a `test.setTimeout()` or `test.slow()` from inside the test. A test
340
- that Playwright cuts off because the whole run is stopping (`maxFailures`
341
- reached, Ctrl+C) is reported as skipped: it did not fail, and it does not
342
- count against the test.
343
-
344
- ## Annotations, the test's line, worker and start (0.19.0)
345
-
346
- Every attempt also sends:
347
-
348
- - **Its annotations** - `test.info().annotations`, the `{ annotation }`
349
- option, and the ones `test.skip()`, `test.fixme()`, `test.slow()` and
350
- `test.fail()` add: type, description, and where it was added (`file:line`).
351
- Squally shows them on the run's Tests tab, the attempt page and the test
352
- page; an `issue` whose description is a link becomes a link there.
353
- - **Descriptions go through the same secret masking as error texts**, and
354
- are cut at 1,000 characters.
355
- - **At most 20 per attempt**, types cut at 64 characters; Squally is told
356
- how many were left out.
357
- - On Playwright versions without per-attempt annotations, the test's own
358
- list is sent with every attempt.
359
- - **The test's line and column** in its file, for the attempt page's
360
- *Open in repository* link.
361
- - **The worker that ran it, its slot and when it started** (`workerIndex`,
362
- `parallelIndex`, `startTime`). Squally stores them for a later timeline of
363
- the run; nothing shows them yet.
364
-
365
- Nothing to configure. An older Squally ignores these fields.
366
-
367
- ## Cancelled runs
368
-
369
- **Cancelled in Squally.** An admin can cancel a running run from its page in
370
- Squally. That marks the run there; it does not stop your CI job. What the job
371
- sends afterwards is discarded, and from 0.12.0 the reporter says so in one
372
- line - "This run was cancelled in Squally" - instead of advice about job keys.
373
-
374
- A run stopped with SIGINT - Ctrl+C, or a cancelled CI job - shows in Squally
375
- as **Cancelled**: the reporter sends what finished and tells Squally the run
376
- was interrupted. Your exit code stays Playwright's own (130). This needs
377
- Playwright itself to receive the signal.
378
-
379
- **On GitHub Actions** a cancelled workflow sends the signal to the step's
380
- shell, not to Playwright, so the reporter never reports the end. Add one step
381
- after the tests (0.14.0) that marks the run cancelled in Squally:
382
-
383
- ```yaml
384
- jobs:
385
- test:
386
- runs-on: ubuntu-latest
387
- env:
388
- SQUALLY_INGEST_KEY: ${{ secrets.SQUALLY_INGEST_KEY }}
389
- steps:
390
- # ...checkout, setup-node, npm ci, playwright install
391
- - name: Run Playwright tests
392
- run: npx playwright test
393
- - name: Mark Squally run cancelled
394
- if: ${{ cancelled() }}
395
- run: npx squally-reporter cancel
396
- ```
397
-
398
- `npx squally-reporter cancel` finds the run by the same CI run id the reporter
399
- sent from this job, marks every shard of it that is still running as
400
- cancelled, and prints one line: "Marked run #N as cancelled in Squally
401
- (X shards)", "Nothing to cancel: no run recorded for <id>" when the job was
402
- cancelled before its first test, or "Run already finished". It never fails the
403
- step: a missing key, an unreachable Squally, a 401 or a 5xx is one warning and
404
- exit code 0. It needs the key in the step's environment - set it at job
405
- level, as above, or on the step too. A Squally without the cancel endpoint
406
- answers 404, which is a warning as well.
407
-
408
- Two things to know:
409
-
410
- - **Set `SQUALLY_CI_JOB_KEY` at job level**, not only on the test step. The
411
- cancel step builds the id the same way the reporter did; without the key it
412
- names another run - or none. `npx squally-reporter cancel --job-key <key>`
413
- passes it explicitly.
414
- - **A fail-fast matrix marks the whole group cancelled.** With GitHub's
415
- default `fail-fast: true`, one failing shard cancels the others, their
416
- cancel steps run, and the whole run shows as **Cancelled** instead of
417
- **Failed**. Set `fail-fast: false` on a sharded job so a
418
- real failure stays a failure.
419
-
420
- Without the step, a cancelled workflow shows as **incomplete** in Squally.
421
-
422
- Without that - or after any hard kill (SIGKILL) - the reporter never gets to
423
- report the end: the run stays *running* until Squally's hourly check marks it
424
- **incomplete** once the project's run timeout (60 minutes by default) has
425
- passed. A run stopped before its first test starts sends nothing, like any
426
- run that never ran a test, and `globalTimeout` running out is not a cancel -
427
- that run is simply failed.
428
-
429
- ## Which CI, branch and commit a run is reported under
430
-
431
- Squally counts **CI runs** in its test metrics, so the reporter works out
432
- which CI system it is running on from that system's own environment:
433
-
434
- | Provider | Recognised by | Shards grouped by | Branch | Commit |
435
- |---|---|---|---|---|
436
- | GitHub Actions | `GITHUB_ACTIONS` | `GITHUB_RUN_ID`-`GITHUB_RUN_ATTEMPT` | `GITHUB_REF_NAME`; pull requests: `GITHUB_HEAD_REF` | `GITHUB_SHA`; pull requests: the PR's head commit |
437
- | GitLab CI | `GITLAB_CI` | `CI_PIPELINE_ID` | `CI_COMMIT_BRANCH`; merge requests: `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | `CI_COMMIT_SHA`; merged-results pipelines: `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` |
438
- | CircleCI | `CIRCLECI` | `CIRCLE_WORKFLOW_ID` | `CIRCLE_BRANCH` | `CIRCLE_SHA1` |
439
- | Jenkins | `BUILD_TAG` (`jenkins-…`) | `BUILD_TAG` | `CHANGE_BRANCH`, else `BRANCH_NAME`, else `GIT_BRANCH` | `GIT_COMMIT` |
440
- | Azure Pipelines | `TF_BUILD` | `BUILD_BUILDID` | `BUILD_SOURCEBRANCH`; pull requests: `SYSTEM_PULLREQUEST_SOURCEBRANCH` | `BUILD_SOURCEVERSION`; pull requests: `SYSTEM_PULLREQUEST_SOURCECOMMITID` |
441
- | Bitbucket Pipelines | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_BRANCH` | `BITBUCKET_COMMIT` |
442
- | Buildkite | `BUILDKITE` | `BUILDKITE_BUILD_ID` | `BUILDKITE_BRANCH` | `BUILDKITE_COMMIT` |
443
- | TeamCity | `TEAMCITY_VERSION` | - (set `SQUALLY_CI_RUN_ID`) | from git | `BUILD_VCS_NUMBER` |
444
- | AWS CodeBuild | `CODEBUILD_BUILD_ID` | `CODEBUILD_BUILD_ID` | `CODEBUILD_WEBHOOK_HEAD_REF` | `CODEBUILD_RESOLVED_SOURCE_VERSION` |
445
- | Travis CI | `TRAVIS` | `TRAVIS_BUILD_ID` | `TRAVIS_BRANCH`; pull requests: `TRAVIS_PULL_REQUEST_BRANCH` | `TRAVIS_COMMIT`; pull requests: `TRAVIS_PULL_REQUEST_SHA` |
446
- | Drone | `DRONE` | `DRONE_BUILD_NUMBER` | `DRONE_BRANCH`; pull requests: `DRONE_SOURCE_BRANCH` | `DRONE_COMMIT_SHA` |
447
- | any other CI | `CI=true` or `CI=1` | - (set `SQUALLY_CI_RUN_ID`) | from git | from git |
448
-
449
- - **`CI` counts only as `true` or `1`.** `CI=false` or `CI=0` is a local run -
450
- unless one of the providers above is recognised, which always wins. A run
451
- that is none of these is local. Local runs are listed with all other runs,
452
- and have numbers of their own: Squally's figures count CI runs by default,
453
- and the **All · CI · Local** switch shows local runs on their own, or both.
454
- A local run never posts a GitHub check, a pull-request comment or a Slack
455
- message.
456
- - **Pull requests are reported under their own head commit and source
457
- branch**, not the merge commit CI checks out - that is the commit Squally's
458
- GitHub check and pull-request comment have to find. The commit message and
459
- author are that commit's too - never the "Merge … into …" message of the
460
- checkout.
461
- - **The head commit is fetched when the checkout lacks it.** With
462
- `actions/checkout`'s default `fetch-depth: 1` the head commit is not in the
463
- clone, so the reporter fetches that one commit from `origin`
464
- (`git fetch --depth=1 --no-tags`), waits at most 3 seconds, and reads its
465
- message and author. It only does so in a shallow checkout - fetching into a
466
- full clone would make it shallow for the rest of your job - and never
467
- changes `HEAD` or any branch. If the fetch fails or takes longer, the
468
- message and author stay empty and the reporter says so once; the run and
469
- its exit code are unaffected. Alternatives: `fetch-depth: 2` on the checkout
470
- step puts the head commit in the clone, so nothing is fetched, and
471
- `SQUALLY_DISABLE_HEAD_COMMIT_FETCH=true` turns the fetch off.
472
- - **A tag build has no branch.** A detached checkout never reports `HEAD` as a
473
- branch name, and `refs/heads/` or `origin/` prefixes are removed.
474
- - **A link to the CI run (0.11.0).** The reporter sends the page of the CI run,
475
- so Squally can link a run to it. It is read from each provider's own
476
- variable, sent only as an `https://` URL of at most 2,048 characters, and
477
- never opened by Squally - only shown as a link:
478
-
479
- | Provider | Link |
480
- |---|---|
481
- | GitHub Actions | `GITHUB_SERVER_URL`/`GITHUB_REPOSITORY`/actions/runs/`GITHUB_RUN_ID`/attempts/`GITHUB_RUN_ATTEMPT` |
482
- | GitLab CI | `CI_PIPELINE_URL` |
483
- | Jenkins | `BUILD_URL` (only with Jenkins served over https) |
484
- | Buildkite | `BUILDKITE_BUILD_URL` |
485
- | Travis CI | `TRAVIS_BUILD_WEB_URL` |
486
- | Drone | `DRONE_BUILD_LINK` |
487
-
488
- Other providers send no link. It is the run's page, never one job's, and it
489
- is the same with `SQUALLY_CI_RUN_ID` set.
490
-
491
- ## Tests marked with `test.fail()`
492
-
493
- Playwright's `test.fail()` marks a test as *expected to fail*: it guards a bug
494
- you already know about, and Playwright runs the test anyway to check that it
495
- does still fail. The interesting moment is the opposite one - when someone
496
- fixes the bug, the test starts passing and Playwright turns it red so you
497
- remember to delete the marker.
498
-
499
- The reporter passes that marker on, so Squally reads such a test the same way
500
- Playwright does:
501
-
502
- - **Marked and still failing** - shown as passing, with an "expected failure"
503
- label. It counts as a pass in the test's stability, not against its failure
504
- rate, and it does not fail Squally's pull-request check.
505
- - **Marked and now passing** - shown as a failure, because the marker is out of
506
- date and should be removed.
507
- - **Marked and timed out** - shown as a failure. Playwright does not count a
508
- timeout as the expected failure either, so the marker is not passed on for
509
- it.
510
-
511
- Both ways of marking a test are recognised: the `test.fail()` modifier and a
512
- `testInfo.fail()` call inside the test body. `test.skip()` and `test.fixme()`
513
- are a different statement and are not affected.
514
-
515
- Nothing is reported for an unmarked test, so an ordinary suite is unchanged.
516
- Note also that Playwright itself treats a marked, failing test as expected, so
517
- with `screenshot: "only-on-failure"` it captures nothing for it - there is no
518
- artifact for the reporter to upload, which is consistent rather than missing.
519
-
520
- ## Sharding
521
-
522
- **Handled automatically - no extra setup needed.** Each shard is still a
523
- separate process with its own reporter instance, but Squally now correlates
524
- them itself: all shards of one CI run are grouped together and shown as one
525
- run once every shard has finished, with the flip side of that guarantee being
526
- that the run only completes once every shard *has* reported.
527
-
528
- Correlation is based on an id read from your CI provider's own environment
529
- (the "Shards grouped by" column above), so it costs you nothing on every
530
- provider in that table except TeamCity. On TeamCity, on a CI system not in the
531
- table, or when you shard locally across several separate `--shard`
532
- invocations, set `SQUALLY_CI_RUN_ID` to the same value for every shard of one
533
- run - anything unique per CI run and shared by its shards works, e.g. a build
534
- number your CI system already exposes.
535
-
536
- **Several jobs in one CI run need their own key (0.11.0).** The id is shared
537
- by everything in one CI run, so two jobs that are *not* shards of each other -
538
- a GitHub Actions matrix over browsers run without `--shard`, or an `api` and a
539
- `ui` suite - would report into the same Squally run. Give each job its own
540
- `SQUALLY_CI_JOB_KEY`, shared only by that job's own shards:
541
-
542
- ```yaml
543
- jobs:
544
- test:
545
- strategy:
546
- matrix:
547
- browser: [chromium, firefox]
548
- env:
549
- SQUALLY_CI_JOB_KEY: ${{ matrix.browser }}
550
- steps:
551
- - run: npx playwright test --project=${{ matrix.browser }}
552
- ```
553
-
554
- The reporter puts the key into the id - `<GITHUB_RUN_ID>-<key>-<GITHUB_RUN_ATTEMPT>`
555
- on GitHub Actions, `<provider id>-<key>` elsewhere - so each job is its own run
556
- in Squally, and a re-run of one job links to that job's own earlier attempt.
557
- The key may hold letters, digits, `.`, `_` and `-`, at most 64 characters; an
558
- invalid key is reported once and ignored, and your test run is unaffected.
559
- Without a key the id is exactly what it was before 0.11.0. A suite that is
560
- sharded *and* split into jobs uses the same key for all shards of one job:
561
- `SQUALLY_CI_JOB_KEY: ${{ matrix.suite }}` beside `--shard=${{ matrix.shard }}/2`.
562
-
563
- **Replaces the hand-built id.** Until 0.11.0 this section suggested
564
- `SQUALLY_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.browser }}`.
565
- It still groups each job correctly, but Squally cannot tell its attempts apart,
566
- so a re-run is never linked to the attempt before it. Switch to
567
- `SQUALLY_CI_JOB_KEY`. An explicit `SQUALLY_CI_RUN_ID` still wins for the id;
568
- the key is then only sent alongside it.
569
-
570
- **When two jobs collide anyway, Squally refuses the second one at the start**
571
- (squally-app with block 1b, reporter 0.11.0). The reporter prints one block,
572
- sends nothing more for that job, and leaves your test run and its exit code
573
- alone:
574
-
575
- - `ci_run_collision` - another job already reports under this CI run id (and
576
- shard). Give every job its own `SQUALLY_CI_JOB_KEY`.
577
- - `ci_run_mixed` - the id would put a sharded job and an unsharded one, or
578
- shards of different totals, into one run. The same fix.
579
-
580
- Before 0.11.0 the second job was handed the first job's run instead, and its
581
- results were refused once that run completed; a reporter older than 0.11.0
582
- still behaves that way, and prints "Squally has already completed run …".
583
-
584
- **Re-runs.** On GitHub Actions every re-run attempt is its own run in Squally,
585
- linked to the attempt before it ("Re-run of #N"). "Re-run failed jobs" runs only
586
- the shards that did not pass; that attempt waits only for those shards,
587
- completes once they have reported, and passes or fails on the tests they ran -
588
- the reused shards are neither waited for nor counted, and the run page shows
589
- them as "passed in #N · not re-run". A re-run shard that never reports still
590
- leaves the run incomplete after the run timeout.
591
-
592
- Other CI systems re-run a job under the id of the run it belongs to - a GitLab
593
- job retry, an Azure Pipelines re-run, a Bitbucket re-run of failed steps, a
594
- Buildkite retry, a Travis CI restart. Since 0.11.0 Squally refuses such a
595
- retry at the start with `ci_run_collision`: the slot is taken by the first
596
- attempt. **`SQUALLY_CI_JOB_KEY` does not help here** - the retry is the same
597
- job - so a retry on these systems is not reported; the first attempt stays the
598
- run. A Drone restart is a new build with a new number, and so a new run in
599
- Squally, not linked to the one it repeats.
600
-
601
- **`playwright merge-reports` is not part of this and is not recommended** for
602
- getting a single run in Squally - the automatic correlation above is the
603
- supported path, and produces correct per-shard timing/branch/commit
604
- information that a merge job cannot (see the historical note below). Nothing
605
- stops you from using `merge-reports` for Playwright's own HTML report if you
606
- want that separately; just don't point this reporter at the merged result
607
- expecting Squally to treat it specially.
608
-
609
- <details>
610
- <summary>Why not just use `merge-reports`, historically</summary>
611
-
612
- An earlier version of this reporter recommended `playwright merge-reports` as
613
- the way to get one Squally run out of a sharded job. That path has a real
614
- problem: branch, commit, commit message and author are all read on whichever
615
- machine the reporter actually runs on - during a merge that is the merge job,
616
- not the shards - and without a checkout there (and without the CI provider's
617
- own commit variables) they all come out empty. A run with no branch is in
618
- no branch's numbers and no branch filter finds it, so a "merged" run could
619
- exist in the dashboard while missing from everything Squally measures per
620
- branch: suite
621
- stability, and each test's stability, flaky rate and failure rate on that
622
- branch. The automatic correlation above reads that information from each
623
- shard's own environment instead, so this problem does not arise.
624
-
625
- </details>
626
-
627
- ## The run's Playwright config and repository (0.16.0)
628
-
629
- At the start of a run the reporter sends what Squally's **Metadata** tab
630
- shows under *Settings* and *Playwright projects*, read field by field from
631
- Playwright's config - never the config itself:
632
-
633
- - **Settings:** `workers` as configured and the workers Playwright actually
634
- started (newer Playwright versions only; 1.30 does not report it),
635
- `fullyParallel`, `forbidOnly`, `globalTimeout`, `maxFailures`, and the
636
- reporters by name - Playwright's own (`html`, `list`, ...) and npm packages
637
- by package name; a reporter file of your own is sent as `custom`, never its
638
- path.
639
- - **Per project:** its name, `browserName`, `channel`, `viewport`, the
640
- `trace`/`video`/`screenshot` mode, `retries` and `timeout`.
641
- - **The repository** as host and path, e.g. `github.com/acme/shop`: from your
642
- CI's variables (`GITHUB_REPOSITORY`, `CI_PROJECT_PATH`,
643
- `BITBUCKET_REPO_FULL_NAME`, `BUILD_REPOSITORY_URI`, `BUILDKITE_REPO`,
644
- `CIRCLE_REPOSITORY_URL`, `CODEBUILD_SOURCE_REPO_URL`, `DRONE_GIT_HTTP_URL`,
645
- `TRAVIS_REPO_SLUG`, `GIT_URL`), else `git remote get-url origin`.
646
- Credentials, a query and `.git` are removed before it is sent.
647
- - **Where in the repository the run started** (0.19.0): the directory
648
- Playwright runs in, relative to the repository's root, as
649
- `git rev-parse --show-prefix` prints it - `e2e/` in a monorepo, empty at
650
- the root. Test file paths are relative to that directory, so Squally needs
651
- it to link a test to its file. Not sent when git cannot say.
652
-
653
- Two switches:
654
-
655
- ```bash
656
- # The base URL of each project (use.baseURL) - off unless you turn it on.
657
- # Only scheme, host and path are sent, never a login, query or fragment.
658
- SQUALLY_SEND_BASE_URL=true
659
-
660
- # The repository and where in it the run started - on unless you turn it off.
661
- SQUALLY_SEND_REPOSITORY=false
662
- ```
663
-
664
- **Never sent:** the rest of `use` (extra HTTP headers, HTTP credentials,
665
- `storageState`, proxy, launch and context options), `config.metadata`, the
666
- command line, the config file and every absolute directory path, `webServer`, `grep`,
667
- global setup and teardown, and each project's own `metadata`.
668
-
669
- ## Tested revision (optional)
670
-
671
- The tested revision is the state of your **application** a run tested, as
672
- opposed to the commit the run is reported under. Squally does not know it by
673
- itself - it is **opt-in**, with one of these two:
674
-
675
- ```bash
676
- # The tests live in the same repo as the application: the commit this CI job
677
- # checked out is the tested revision.
678
- SAME_REPO_AS_APP=true
679
-
680
- # The tests live in a separate repo (e.g. a dedicated E2E repo pointed at a
681
- # deployed app): pass the tested revision itself - a commit SHA, tag, or
682
- # branch name/ref.
683
- TESTED_REVISION=v1.2.0
684
- ```
685
-
686
- - **`SAME_REPO_AS_APP=true`** sends the commit your job checked out: your CI
687
- provider's own variable for it (`GITHUB_SHA`, `CI_COMMIT_SHA`,
688
- `CIRCLE_SHA1`, `GIT_COMMIT`, `BUILD_SOURCEVERSION`, `BITBUCKET_COMMIT`,
689
- `BUILDKITE_COMMIT`, `BUILD_VCS_NUMBER`, `CODEBUILD_RESOLVED_SOURCE_VERSION`,
690
- `TRAVIS_COMMIT`, `DRONE_COMMIT_SHA`), or `git rev-parse HEAD` where there is
691
- none - including local runs. Only a commit SHA is sent; if neither yields
692
- one, the run has no tested revision.
693
- - **On a pull request the two commits differ, on purpose.** The run is
694
- reported under the pull request's head commit (see the provider table
695
- above) - that is what Squally's check and PR comment need. The tested
696
- revision is the merge commit CI checked out, because that is what the tests
697
- ran against. On a Bitbucket pull request and a GitHub `pull_request_target`
698
- run it comes from `git rev-parse HEAD`.
699
- - **`TESTED_REVISION` wins** when both are set.
700
- - Neither variable is required - without them, test results are still sent
701
- and stored exactly as before.
702
-
703
- **The value is normalised before it is stored**, so that the same revision
704
- written two ways still compares equal: it is lowercased, and a leading `v` or a
705
- `refs/heads/` prefix is removed. The example above is exactly such a case -
706
- `v1.2.0` is stored as `1.2.0`. The reporter says so once per run when it
707
- happens, so a value that looks different in the dashboard is not a mystery.
708
-
709
- ## Behind a proxy (0.18.0)
710
-
711
- Every request - to Squally, the artifact uploads to storage, and
712
- `npx squally-reporter cancel` - goes through the proxy in `HTTPS_PROXY` (for
713
- `https://` addresses) or `HTTP_PROXY` (for `http://`), lower-case names
714
- first, the way curl reads them. Hosts in `NO_PROXY` are reached directly:
715
- comma-separated, each name with all its subdomains (`internal` and
716
- `.internal` both cover `squally.internal`), `host:port` for one port only, `*`
717
- for everything - no IP ranges:
718
-
719
- ```bash
720
- HTTPS_PROXY=http://proxy.internal:3128
721
- NO_PROXY=localhost,127.0.0.1,.internal
722
- ```
723
-
724
- The proxy is asked to open a tunnel (`CONNECT`), so TLS still goes end to end
725
- to Squally. A login in the proxy URL (`http://user:pass@host:port`) is sent as
726
- `Proxy-Authorization`. A value that is not a URL is one warning, naming the
727
- variable but not its value, and the reporter connects directly. Without any of
728
- these variables nothing changes. Until 0.17.0 the variables were ignored.
729
-
730
- ## Troubleshooting
731
-
732
- Set `SQUALLY_DEBUG=1` to print detail that is otherwise kept quiet - the
733
- browser-console extraction (how many messages were found, or that an attempt's
734
- trace held none), since 0.11.0 a CI run link that was not sent because it was
735
- not an `https://` URL, and since 0.18.0 which proxy variable is in use and a
736
- quarantine rules request that failed (nothing uses the rules yet, so that is
737
- no longer a warning). Everything that is *not* ordinary (a trace that cannot
738
- be read, an unsupported schema version, a rejected upload) is printed without
739
- it.
740
-
741
- ## Support
742
-
743
- Found a bug or have a question? Please open an issue on
744
- [GitHub Issues](https://github.com/squallydev-admin/squally-reporter/issues) -
745
- that's the primary way to report problems or ask questions. For anything you
746
- would rather not post publicly, write to
747
- [support@squally.dev](mailto:support@squally.dev).
748
-
749
- ## License
750
-
751
- MIT - see [LICENSE](./LICENSE).
1
+ # squally-reporter
2
+
3
+ Playwright reporter that sends test results to your Squally dashboard.
4
+
5
+ Learn more at [app.squally.dev](https://app.squally.dev).
6
+
7
+ ## Requirements
8
+
9
+ - Node.js >= 20.18.1
10
+ - `@playwright/test` >= 1.30.0
11
+
12
+ ## Install
13
+
14
+ ```bash
15
+ npm install -D squally-reporter
16
+ ```
17
+
18
+ ## Config
19
+
20
+ Set one env var (e.g. as a CI secret):
21
+
22
+ ```bash
23
+ SQUALLY_INGEST_KEY=your-project-ingest-key
24
+ ```
25
+
26
+ Find your ingest key in the Squally dashboard under **Settings › Project ›
27
+ Ingest key**.
28
+
29
+ **Since 0.12.0 the URL defaults to `https://app.squally.dev`.** Set
30
+ `SQUALLY_INGEST_URL` only to report to another Squally. Without a key the
31
+ reporter sends nothing and says so once.
32
+
33
+ **Renamed in 0.9.0.** These were `SQUALLY_API_URL` and `SQUALLY_API_KEY`. The
34
+ old names still work for now, with a deprecation warning in the CI log - but
35
+ they belong to Squally's read API now, so rename them. `SQUALLY_API_URL` is
36
+ only used together with `SQUALLY_API_KEY`, never on its own - and when it is
37
+ set but not used, the reporter sends nothing rather than falling back to the
38
+ default URL, since the variable says you meant another target.
39
+
40
+ **A read key is never sent.** A key starting with `sqly_ro_` is a read API key,
41
+ for scripts and agents; reporting needs the ingest key. Since 0.12.0 the
42
+ reporter refuses one in any of `SQUALLY_INGEST_KEY`, `SQUALLY_API_KEY` or the
43
+ `apiKey` option before sending anything, and says which one to fix.
44
+
45
+ Then add the reporter to `playwright.config.ts`:
46
+
47
+ ```ts
48
+ import { defineConfig } from "@playwright/test";
49
+
50
+ export default defineConfig({
51
+ reporter: [
52
+ ["html"],
53
+ ["squally-reporter"],
54
+ ],
55
+ });
56
+ ```
57
+
58
+ **Replace your `reporter` line, don't add a second one.** `npm init playwright`
59
+ writes `reporter: 'html',`, and with two `reporter:` keys in the config the
60
+ later one wins without a warning - either the HTML report or Squally silently
61
+ goes. Using other reporters? Keep them in the list and add
62
+ `["squally-reporter"]`.
63
+
64
+ Alternatively, pass config explicitly instead of using env vars - but **prefer
65
+ the env vars above for `apiKey`, and not only because a literal in a committed
66
+ file is obviously wrong.** Playwright's `json` and `blob` reporters serialize
67
+ your resolved config into their output file, options and all: an `apiKey`
68
+ passed here ends up in that file in plain text, even when its value came from
69
+ an environment variable at runtime. What gets written is the option, not where
70
+ it came from. The reporter warns, naming the file, when it sees this
71
+ combination. Measured against a real report, 20.09.2026.
72
+
73
+ ```ts
74
+ reporter: [
75
+ ["squally-reporter", { apiUrl: "https://app.squally.dev", apiKey: process.env.SQUALLY_INGEST_KEY }],
76
+ ],
77
+ ```
78
+
79
+ That's it - run your tests as usual (`npx playwright test`); results stream
80
+ to Squally while the suite runs (see *How results reach Squally* below).
81
+
82
+ ## Artifact upload (screenshot, trace, browser console)
83
+
84
+ If your `playwright.config.ts` captures screenshots or traces, the reporter
85
+ uploads them to Squally so they show up next to the failing test in the
86
+ dashboard - no separate setup beyond the two env vars above.
87
+
88
+ Three kinds are uploaded:
89
+
90
+ | Kind | Where it comes from |
91
+ | --- | --- |
92
+ | **Screenshot** | the file Playwright writes for a failed attempt |
93
+ | **Trace** | the trace zip Playwright writes |
94
+ | **Browser console** | read out of that trace zip by this reporter |
95
+
96
+ **The browser console needs a trace.** It is not something Playwright hands a
97
+ reporter: `result.stdout`/`stderr` are the output of the *Node* process - what
98
+ your test file printed - while what the *page* printed only exists inside the
99
+ trace. So the reporter opens the trace and reads it out, which means an
100
+ attempt with no trace has no browser console, and nothing can change that but
101
+ turning tracing on. Squally's attempt page says which of the two it is rather
102
+ than showing an empty section.
103
+
104
+ Supported trace schema versions: **5 to 10**, which covers Playwright 1.30
105
+ through 1.62 (1.62 writes version 8). A trace outside that range is left alone
106
+ and the reporter prints one warning naming the version - it never uploads an
107
+ empty console file, because "no messages" and "this reporter could not read
108
+ it" must not look the same in the dashboard.
109
+
110
+ **Only from attempts worth looking at:** artifacts are uploaded only for
111
+ attempts that **failed**, or that **needed a retry** (even if that retry then
112
+ passed) - never for an attempt that passed on its first try, regardless of how
113
+ `screenshot`/`trace` are configured. Nobody looks at a screenshot of a test
114
+ that just passed, and uploading one anyway would only eat into your
115
+ organization's storage for no benefit.
116
+
117
+ **Uploaded while the suite runs (0.18.0).** An attempt's artifacts go up as
118
+ soon as Squally has accepted the batch that carries its result - not only
119
+ after the last test, as until 0.17.0. A job that is cancelled or killed
120
+ partway keeps the artifacts of every result that already arrived. When the
121
+ suite ends, the reporter waits **up to 3 minutes** for the uploads still
122
+ running; the last 15 seconds of that only finish the confirmation, no new
123
+ upload starts in them. A file is sent as a stream, not read into memory
124
+ first.
125
+
126
+ **You need to actually enable capture in `playwright.config.ts`** - without
127
+ it, Playwright doesn't produce any attachments, so there's nothing for the
128
+ reporter to find or upload (no error either way, it's a silent no-op):
129
+
130
+ ```ts
131
+ export default defineConfig({
132
+ use: {
133
+ screenshot: "only-on-failure",
134
+ trace: "retain-on-failure",
135
+ },
136
+ // ...
137
+ });
138
+ ```
139
+
140
+ **Limits.** None of these are enforced by the reporter - Squally applies them
141
+ when the upload is requested and answers with a reason, which is what the
142
+ reporter prints. The numbers below are therefore Squally's current ones rather
143
+ than anything this package guarantees, and can change server-side without a new
144
+ release of it:
145
+
146
+ - **50 MB per screenshot or trace, 4 MB per browser console**, the same on
147
+ every plan. A larger file is rejected before any bytes are uploaded; the
148
+ reporter logs a `console.warn` naming the file and its limit and moves on.
149
+ The test result itself is reported as usual, and the run's outcome and exit
150
+ code are untouched. In the dashboard that artifact is shown as skipped, with
151
+ the reason, rather than silently missing.
152
+ - **At most 250 artifacts per request.** Since 0.18.0 the reporter sends them
153
+ in rounds of at most 250, so a shard with more no longer loses the rest.
154
+ Until 0.17.0 a shard's artifacts went in one request, and Squally kept the
155
+ first 250.
156
+ - **A total storage cap per organization, depending on your Squally plan.**
157
+ Once it is reached, further uploads are rejected the same way until you free
158
+ up space or upgrade. Note that a fully instrumented suite now produces up to
159
+ three artifacts per failed attempt rather than two.
160
+ - The browser console is a text file of a few KB - it is extracted from the
161
+ trace you are already uploading, so it adds no capture cost to your run.
162
+ - **Your test's own `stdout`/`stderr` is cut at 16 KB** before it is sent. That
163
+ is this reporter's own limit, not Squally's, and it applies whether or not
164
+ you capture artifacts at all.
165
+ - Two minutes per upload, and 30 seconds for the calls that surround it. Since
166
+ 0.18.0 a network error, an HTTP 5xx or a 429 is retried like the results
167
+ (see *When Squally cannot be reached for a while*), and the 3 minutes at the
168
+ end bound the whole wait. What did not arrive is said once, at the end:
169
+
170
+ ```
171
+ [squally-reporter] 2 of 9 artifacts did not reach Squally - the storage answered HTTP 503, and the retries ran out. Your test run and its exit code are not affected.
172
+ ```
173
+
174
+ Upload is best-effort and fully non-blocking, same principle as the rest of
175
+ this reporter - a failed or skipped upload never fails your test run or
176
+ changes its exit code, it only logs a `console.warn` explaining what
177
+ happened.
178
+
179
+ ## How results reach Squally
180
+
181
+ As soon as the first test starts, the reporter creates the run in Squally and
182
+ prints where to watch it (0.15.0), on stderr:
183
+
184
+ ```
185
+ [squally-reporter] Reporting to Squally: run #128 · https://app.squally.dev/o/…/runs/…
186
+ ```
187
+
188
+ Under `--shard` the line reads `run #128, shard 2 of 3`; every shard of a CI
189
+ run has the same number.
190
+
191
+ Results stream to Squally as your suite runs, not all at once at the end:
192
+ each test's result is sent in a small batch, so the "running" view in the
193
+ dashboard stays close to real time even for a long suite. A batch goes out on
194
+ whichever of these comes first:
195
+
196
+ - **25 results.** Fixed, and the same whether you shard or not.
197
+ - **A timer: 2 seconds, or 0.4 seconds × the number of shards when that is
198
+ longer, capped at 30 seconds** (0.12.0). Up to five shards every shard
199
+ flushes every 2 seconds; eight shards every 3.2; sixteen every 6.4; from 75
200
+ shards the cap holds it at 30.
201
+
202
+ Only the timer scales. All shards of a run share one rate budget on Squally's
203
+ side (300 requests a minute per key), and the timer keeps all of them
204
+ together at about half of it. The 25-result threshold is left alone
205
+ deliberately: it bounds how long a *fast* suite waits, and stretching it would
206
+ delay results without bounding the burst any better.
207
+
208
+ A batch also stays under 3 MB (0.12.0): results with long error texts or many
209
+ steps simply go out in more, smaller requests.
210
+
211
+ There is no longer a single-request size limit that could discard a whole run -
212
+ a very large suite simply takes more, smaller requests. As a courtesy the
213
+ reporter still warns if a run's result count climbs well past 5,000, since that
214
+ usually means retries are configured higher than intended rather than a suite
215
+ that's actually that large.
216
+
217
+ If Squally is unreachable partway through, results already sent stay in the
218
+ dashboard; only what could not be delivered is lost, and your test run's exit
219
+ code is never affected either way.
220
+
221
+ ## The end of a run (0.18.0)
222
+
223
+ When every result arrived and the run is marked complete, the last line says
224
+ where it is and what got there:
225
+
226
+ ```
227
+ [squally-reporter] Reported to Squally: run #128 · 412 results, 9 artifacts · https://app.squally.dev/o/…/runs/…
228
+ ```
229
+
230
+ If something did not arrive, that line is replaced by the one that says what
231
+ is missing (see below).
232
+
233
+ **`SQUALLY_OUTPUT_FILE`** writes the run's link and counts as JSON, for a
234
+ later CI step that posts the link somewhere:
235
+
236
+ ```bash
237
+ SQUALLY_OUTPUT_FILE=test-results/squally.json
238
+ ```
239
+
240
+ ```json
241
+ {
242
+ "version": 1,
243
+ "runUrl": "https://app.squally.dev/o/…/runs/…",
244
+ "runNumber": 128,
245
+ "runId": "…",
246
+ "shard": null,
247
+ "complete": true,
248
+ "tests": { "passed": 40, "failed": 1, "flaky": 1, "skipped": 0 },
249
+ "results": { "produced": 44, "delivered": 44 },
250
+ "artifacts": { "uploaded": 3, "notUploaded": 0 }
251
+ }
252
+ ```
253
+
254
+ - Written at the very end of every run the reporter took part in - also one
255
+ Squally refused or could not be reached for, then with `runUrl: null` and
256
+ `complete: false`, so the file is always there for the next step. Missing
257
+ directories are created.
258
+ - `tests` counts this process's tests the way Playwright does at the end
259
+ (`expected` as passed, `unexpected` as failed, `flaky`, `skipped`).
260
+ `results` counts attempts: produced, and accepted by Squally.
261
+ - One file per process: under `--shard`, give each shard its own path (or
262
+ read them one after another), and `shard` says which one it was.
263
+ - A path that cannot be written is one warning; the exit code stays yours.
264
+ - New fields may be added; `version` changes only if one changes meaning.
265
+
266
+ ## When Squally cannot be reached for a while (0.17.0)
267
+
268
+ Some failures pass on their own, and the reporter waits them out instead of
269
+ dropping results:
270
+
271
+ - a network error or a request without an answer within 30 seconds,
272
+ - an HTTP 5xx,
273
+ - HTTP 429 (Squally's rate limit),
274
+ - HTTP 403 from Vercel's firewall in front of Squally - the answer carries
275
+ `x-vercel-mitigated: challenge` (or `deny`). The firewall can challenge one
276
+ runner for a few minutes; it says nothing about your key.
277
+
278
+ Each request is retried after 1 second, then 2, 4, 8 and so on up to 30
279
+ seconds between tries (each wait shortened by a random part, so shards
280
+ refused together do not come back together), for up to 90 seconds. A
281
+ `Retry-After` header is honoured. The start of the run gets the same 90
282
+ seconds; your tests run meanwhile and their results are held until Squally
283
+ answers. While Squally keeps refusing, results go out one request at a time.
284
+
285
+ Not retried, because waiting does not change them: Squally's own error
286
+ answers (a refused key, an exhausted quota, an unknown or finished run), an
287
+ HTTP 403 *without* the firewall's header, and a redirect. Those keep their own
288
+ line, as before.
289
+
290
+ The artifact uploads follow the same rules since 0.18.0 (until then each was
291
+ tried once): the request for the upload URL, the upload to storage (a network
292
+ error, HTTP 5xx or 429) and the confirmation. They have their own line at the
293
+ end and never delay the results.
294
+
295
+ When your suite ends, the reporter normally waits up to 5 seconds for the last
296
+ results. After one of the failures above it waits up to 60 seconds, for the
297
+ last results and for marking the run complete.
298
+
299
+ What did not arrive is said once, at the end - not once per batch:
300
+
301
+ ```
302
+ [squally-reporter] 37 of 412 results did not reach Squally - Vercel's firewall in front of Squally challenged this runner (HTTP 403, x-vercel-mitigated: challenge), and the retries ran out. Your test run and its exit code are not affected. The run page in Squally shows how many are missing.
303
+ ```
304
+
305
+ If the run could not be marked complete either, the line says so: the run
306
+ then shows as **incomplete** in Squally once the project's run timeout has
307
+ passed, with the results that did arrive. If the run could not be started at
308
+ all, the line reads `Squally did not start this run - ...` and nothing was
309
+ sent.
310
+
311
+ The exit code of your test run is never changed by any of this.
312
+
313
+ ## Steps (0.12.0)
314
+
315
+ For an attempt that **failed, or needed a retry**, the reporter also sends the
316
+ steps Playwright recorded - hooks, fixtures, actions, assertions and your own
317
+ `test.step`s, with their durations and `file:line`. Squally shows them on the
318
+ attempt page, the failing step marked. An attempt that passed on its first try
319
+ sends none.
320
+
321
+ - **Typed values never leave your runner.** The value of a `fill`, `type` or
322
+ `pressSequentially` is replaced: `Fill "•••" getByLabel('Password')`. Every
323
+ title also goes through the same secret masking as error texts.
324
+ - **At most 300 steps per attempt, 200 characters per title.** A longer list
325
+ keeps the failing step and the steps it sits in, then the rest in order, and
326
+ tells Squally how many were left out.
327
+ - Nothing to configure: steps come from Playwright's own report, with or
328
+ without tracing.
329
+
330
+ **Only actual test runs appear.** A run shows up in Squally once your first
331
+ test starts, so commands that never run a test create nothing: `--list`,
332
+ a `globalSetup` or `webServer` that fails to start, a `--grep` that matches
333
+ nothing. A run that *does* execute and legitimately contains no tests — an
334
+ empty shard, `--only-changed` with no changed test file, `--pass-with-no-tests`
335
+ — is still reported, as a run with zero results.
336
+
337
+ **Timeouts and interrupted tests.** A test that runs out of time is reported
338
+ as failed, and Squally is told that it timed out and what its timeout was -
339
+ including a `test.setTimeout()` or `test.slow()` from inside the test. A test
340
+ that Playwright cuts off because the whole run is stopping (`maxFailures`
341
+ reached, Ctrl+C) is reported as skipped: it did not fail, and it does not
342
+ count against the test.
343
+
344
+ ## Annotations, the test's line, worker and start (0.19.0)
345
+
346
+ Every attempt also sends:
347
+
348
+ - **Its annotations** - `test.info().annotations`, the `{ annotation }`
349
+ option, and the ones `test.skip()`, `test.fixme()`, `test.slow()` and
350
+ `test.fail()` add: type, description, and where it was added (`file:line`).
351
+ Squally shows them on the run's Tests tab, the attempt page and the test
352
+ page; an `issue` whose description is a link becomes a link there.
353
+ - **Descriptions go through the same secret masking as error texts**, and
354
+ are cut at 1,000 characters.
355
+ - **At most 20 per attempt**, types cut at 64 characters; Squally is told
356
+ how many were left out.
357
+ - On Playwright versions without per-attempt annotations, the test's own
358
+ list is sent with every attempt.
359
+ - **The test's line and column** in its file, for the attempt page's
360
+ *Open in repository* link.
361
+ - **The worker that ran it, its slot and when it started** (`workerIndex`,
362
+ `parallelIndex`, `startTime`). Squally stores them for a later timeline of
363
+ the run; nothing shows them yet.
364
+
365
+ Nothing to configure. An older Squally ignores these fields.
366
+
367
+ ## Cancelled runs
368
+
369
+ **Cancelled in Squally.** An admin can cancel a running run from its page in
370
+ Squally. That marks the run there; it does not stop your CI job. What the job
371
+ sends afterwards is discarded, and from 0.12.0 the reporter says so in one
372
+ line - "This run was cancelled in Squally" - instead of advice about job keys.
373
+
374
+ A run stopped with SIGINT - Ctrl+C, or a cancelled CI job - shows in Squally
375
+ as **Cancelled**: the reporter sends what finished and tells Squally the run
376
+ was interrupted. Your exit code stays Playwright's own (130). This needs
377
+ Playwright itself to receive the signal.
378
+
379
+ **On GitHub Actions** a cancelled workflow sends the signal to the step's
380
+ shell, not to Playwright, so the reporter never reports the end. Add one step
381
+ after the tests (0.14.0) that marks the run cancelled in Squally:
382
+
383
+ ```yaml
384
+ jobs:
385
+ test:
386
+ runs-on: ubuntu-latest
387
+ env:
388
+ SQUALLY_INGEST_KEY: ${{ secrets.SQUALLY_INGEST_KEY }}
389
+ steps:
390
+ # ...checkout, setup-node, npm ci, playwright install
391
+ - name: Run Playwright tests
392
+ run: npx playwright test
393
+ - name: Mark Squally run cancelled
394
+ if: ${{ cancelled() }}
395
+ run: npx squally-reporter cancel
396
+ ```
397
+
398
+ `npx squally-reporter cancel` finds the run by the same CI run id the reporter
399
+ sent from this job, marks every shard of it that is still running as
400
+ cancelled, and prints one line: "Marked run #N as cancelled in Squally
401
+ (X shards)", "Nothing to cancel: no run recorded for <id>" when the job was
402
+ cancelled before its first test, or "Run already finished". It never fails the
403
+ step: a missing key, an unreachable Squally, a 401 or a 5xx is one warning and
404
+ exit code 0. It needs the key in the step's environment - set it at job
405
+ level, as above, or on the step too. A Squally without the cancel endpoint
406
+ answers 404, which is a warning as well.
407
+
408
+ Two things to know:
409
+
410
+ - **Set `SQUALLY_CI_JOB_KEY` at job level**, not only on the test step. The
411
+ cancel step builds the id the same way the reporter did; without the key it
412
+ names another run - or none. `npx squally-reporter cancel --job-key <key>`
413
+ passes it explicitly.
414
+ - **A fail-fast matrix marks the whole group cancelled.** With GitHub's
415
+ default `fail-fast: true`, one failing shard cancels the others, their
416
+ cancel steps run, and the whole run shows as **Cancelled** instead of
417
+ **Failed**. Set `fail-fast: false` on a sharded job so a
418
+ real failure stays a failure.
419
+
420
+ Without the step, a cancelled workflow shows as **incomplete** in Squally.
421
+
422
+ Without that - or after any hard kill (SIGKILL) - the reporter never gets to
423
+ report the end: the run stays *running* until Squally's hourly check marks it
424
+ **incomplete** once the project's run timeout (60 minutes by default) has
425
+ passed. A run stopped before its first test starts sends nothing, like any
426
+ run that never ran a test, and `globalTimeout` running out is not a cancel -
427
+ that run is simply failed.
428
+
429
+ ## Which CI, branch and commit a run is reported under
430
+
431
+ Squally counts **CI runs** in its test metrics, so the reporter works out
432
+ which CI system it is running on from that system's own environment:
433
+
434
+ | Provider | Recognised by | Shards grouped by | Branch | Commit |
435
+ |---|---|---|---|---|
436
+ | GitHub Actions | `GITHUB_ACTIONS` | `GITHUB_RUN_ID`-`GITHUB_RUN_ATTEMPT` | `GITHUB_REF_NAME`; pull requests: `GITHUB_HEAD_REF` | `GITHUB_SHA`; pull requests: the PR's head commit |
437
+ | GitLab CI | `GITLAB_CI` | `CI_PIPELINE_ID` | `CI_COMMIT_BRANCH`; merge requests: `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | `CI_COMMIT_SHA`; merged-results pipelines: `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` |
438
+ | CircleCI | `CIRCLECI` | `CIRCLE_WORKFLOW_ID` | `CIRCLE_BRANCH` | `CIRCLE_SHA1` |
439
+ | Jenkins | `BUILD_TAG` (`jenkins-…`) | `BUILD_TAG` | `CHANGE_BRANCH`, else `BRANCH_NAME`, else `GIT_BRANCH` | `GIT_COMMIT` |
440
+ | Azure Pipelines | `TF_BUILD` | `BUILD_BUILDID` | `BUILD_SOURCEBRANCH`; pull requests: `SYSTEM_PULLREQUEST_SOURCEBRANCH` | `BUILD_SOURCEVERSION`; pull requests: `SYSTEM_PULLREQUEST_SOURCECOMMITID` |
441
+ | Bitbucket Pipelines | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_BRANCH` | `BITBUCKET_COMMIT` |
442
+ | Buildkite | `BUILDKITE` | `BUILDKITE_BUILD_ID` | `BUILDKITE_BRANCH` | `BUILDKITE_COMMIT` |
443
+ | TeamCity | `TEAMCITY_VERSION` | - (set `SQUALLY_CI_RUN_ID`) | from git | `BUILD_VCS_NUMBER` |
444
+ | AWS CodeBuild | `CODEBUILD_BUILD_ID` | `CODEBUILD_BUILD_ID` | `CODEBUILD_WEBHOOK_HEAD_REF` | `CODEBUILD_RESOLVED_SOURCE_VERSION` |
445
+ | Travis CI | `TRAVIS` | `TRAVIS_BUILD_ID` | `TRAVIS_BRANCH`; pull requests: `TRAVIS_PULL_REQUEST_BRANCH` | `TRAVIS_COMMIT`; pull requests: `TRAVIS_PULL_REQUEST_SHA` |
446
+ | Drone | `DRONE` | `DRONE_BUILD_NUMBER` | `DRONE_BRANCH`; pull requests: `DRONE_SOURCE_BRANCH` | `DRONE_COMMIT_SHA` |
447
+ | any other CI | `CI=true` or `CI=1` | - (set `SQUALLY_CI_RUN_ID`) | from git | from git |
448
+
449
+ - **`CI` counts only as `true` or `1`.** `CI=false` or `CI=0` is a local run -
450
+ unless one of the providers above is recognised, which always wins. A run
451
+ that is none of these is local. Local runs are listed with all other runs,
452
+ and have numbers of their own: Squally's figures count CI runs by default,
453
+ and the **All · CI · Local** switch shows local runs on their own, or both.
454
+ A local run never posts a GitHub check, a pull-request comment or a Slack
455
+ message.
456
+ - **Pull requests are reported under their own head commit and source
457
+ branch**, not the merge commit CI checks out - that is the commit Squally's
458
+ GitHub check and pull-request comment have to find. The commit message and
459
+ author are that commit's too - never the "Merge … into …" message of the
460
+ checkout.
461
+ - **The head commit is fetched when the checkout lacks it.** With
462
+ `actions/checkout`'s default `fetch-depth: 1` the head commit is not in the
463
+ clone, so the reporter fetches that one commit from `origin`
464
+ (`git fetch --depth=1 --no-tags`), waits at most 3 seconds, and reads its
465
+ message and author. It only does so in a shallow checkout - fetching into a
466
+ full clone would make it shallow for the rest of your job - and never
467
+ changes `HEAD` or any branch. If the fetch fails or takes longer, the
468
+ message and author stay empty and the reporter says so once; the run and
469
+ its exit code are unaffected. Alternatives: `fetch-depth: 2` on the checkout
470
+ step puts the head commit in the clone, so nothing is fetched, and
471
+ `SQUALLY_DISABLE_HEAD_COMMIT_FETCH=true` turns the fetch off.
472
+ - **A tag build has no branch.** A detached checkout never reports `HEAD` as a
473
+ branch name, and `refs/heads/` or `origin/` prefixes are removed.
474
+ - **A link to the CI run (0.11.0).** The reporter sends the page of the CI run,
475
+ so Squally can link a run to it. It is read from each provider's own
476
+ variable, sent only as an `https://` URL of at most 2,048 characters, and
477
+ never opened by Squally - only shown as a link:
478
+
479
+ | Provider | Link |
480
+ |---|---|
481
+ | GitHub Actions | `GITHUB_SERVER_URL`/`GITHUB_REPOSITORY`/actions/runs/`GITHUB_RUN_ID`/attempts/`GITHUB_RUN_ATTEMPT` |
482
+ | GitLab CI | `CI_PIPELINE_URL` |
483
+ | Jenkins | `BUILD_URL` (only with Jenkins served over https) |
484
+ | Buildkite | `BUILDKITE_BUILD_URL` |
485
+ | Travis CI | `TRAVIS_BUILD_WEB_URL` |
486
+ | Drone | `DRONE_BUILD_LINK` |
487
+
488
+ Other providers send no link. It is the run's page, never one job's, and it
489
+ is the same with `SQUALLY_CI_RUN_ID` set.
490
+
491
+ ## Tests marked with `test.fail()`
492
+
493
+ Playwright's `test.fail()` marks a test as *expected to fail*: it guards a bug
494
+ you already know about, and Playwright runs the test anyway to check that it
495
+ does still fail. The interesting moment is the opposite one - when someone
496
+ fixes the bug, the test starts passing and Playwright turns it red so you
497
+ remember to delete the marker.
498
+
499
+ The reporter passes that marker on, so Squally reads such a test the same way
500
+ Playwright does:
501
+
502
+ - **Marked and still failing** - shown as passing, with an "expected failure"
503
+ label. It counts as a pass in the test's stability, not against its failure
504
+ rate, and it does not fail Squally's pull-request check.
505
+ - **Marked and now passing** - shown as a failure, because the marker is out of
506
+ date and should be removed.
507
+ - **Marked and timed out** - shown as a failure. Playwright does not count a
508
+ timeout as the expected failure either, so the marker is not passed on for
509
+ it.
510
+
511
+ Both ways of marking a test are recognised: the `test.fail()` modifier and a
512
+ `testInfo.fail()` call inside the test body. `test.skip()` and `test.fixme()`
513
+ are a different statement and are not affected.
514
+
515
+ Nothing is reported for an unmarked test, so an ordinary suite is unchanged.
516
+ Note also that Playwright itself treats a marked, failing test as expected, so
517
+ with `screenshot: "only-on-failure"` it captures nothing for it - there is no
518
+ artifact for the reporter to upload, which is consistent rather than missing.
519
+
520
+ ## Sharding
521
+
522
+ **Handled automatically - no extra setup needed.** Each shard is still a
523
+ separate process with its own reporter instance, but Squally now correlates
524
+ them itself: all shards of one CI run are grouped together and shown as one
525
+ run once every shard has finished, with the flip side of that guarantee being
526
+ that the run only completes once every shard *has* reported.
527
+
528
+ Correlation is based on an id read from your CI provider's own environment
529
+ (the "Shards grouped by" column above), so it costs you nothing on every
530
+ provider in that table except TeamCity. On TeamCity, on a CI system not in the
531
+ table, or when you shard locally across several separate `--shard`
532
+ invocations, set `SQUALLY_CI_RUN_ID` to the same value for every shard of one
533
+ run - anything unique per CI run and shared by its shards works, e.g. a build
534
+ number your CI system already exposes.
535
+
536
+ **Several jobs in one CI run need their own key (0.11.0).** The id is shared
537
+ by everything in one CI run, so two jobs that are *not* shards of each other -
538
+ a GitHub Actions matrix over browsers run without `--shard`, or an `api` and a
539
+ `ui` suite - would report into the same Squally run. Give each job its own
540
+ `SQUALLY_CI_JOB_KEY`, shared only by that job's own shards:
541
+
542
+ ```yaml
543
+ jobs:
544
+ test:
545
+ strategy:
546
+ matrix:
547
+ browser: [chromium, firefox]
548
+ env:
549
+ SQUALLY_CI_JOB_KEY: ${{ matrix.browser }}
550
+ steps:
551
+ - run: npx playwright test --project=${{ matrix.browser }}
552
+ ```
553
+
554
+ The reporter puts the key into the id - `<GITHUB_RUN_ID>-<key>-<GITHUB_RUN_ATTEMPT>`
555
+ on GitHub Actions, `<provider id>-<key>` elsewhere - so each job is its own run
556
+ in Squally, and a re-run of one job links to that job's own earlier attempt.
557
+ The key may hold letters, digits, `.`, `_` and `-`, at most 64 characters; an
558
+ invalid key is reported once and ignored, and your test run is unaffected.
559
+ Without a key the id is exactly what it was before 0.11.0. A suite that is
560
+ sharded *and* split into jobs uses the same key for all shards of one job:
561
+ `SQUALLY_CI_JOB_KEY: ${{ matrix.suite }}` beside `--shard=${{ matrix.shard }}/2`.
562
+
563
+ **Replaces the hand-built id.** Until 0.11.0 this section suggested
564
+ `SQUALLY_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.browser }}`.
565
+ It still groups each job correctly, but Squally cannot tell its attempts apart,
566
+ so a re-run is never linked to the attempt before it. Switch to
567
+ `SQUALLY_CI_JOB_KEY`. An explicit `SQUALLY_CI_RUN_ID` still wins for the id;
568
+ the key is then only sent alongside it.
569
+
570
+ **When two jobs collide anyway, Squally refuses the second one at the start**
571
+ (squally-app with block 1b, reporter 0.11.0). The reporter prints one block,
572
+ sends nothing more for that job, and leaves your test run and its exit code
573
+ alone:
574
+
575
+ - `ci_run_collision` - another job already reports under this CI run id (and
576
+ shard). Give every job its own `SQUALLY_CI_JOB_KEY`.
577
+ - `ci_run_mixed` - the id would put a sharded job and an unsharded one, or
578
+ shards of different totals, into one run. The same fix.
579
+
580
+ Before 0.11.0 the second job was handed the first job's run instead, and its
581
+ results were refused once that run completed; a reporter older than 0.11.0
582
+ still behaves that way, and prints "Squally has already completed run …".
583
+
584
+ **Re-runs.** On GitHub Actions every re-run attempt is its own run in Squally,
585
+ linked to the attempt before it ("Re-run of #N"). "Re-run failed jobs" runs only
586
+ the shards that did not pass; that attempt waits only for those shards,
587
+ completes once they have reported, and passes or fails on the tests they ran -
588
+ the reused shards are neither waited for nor counted, and the run page shows
589
+ them as "passed in #N · not re-run". A re-run shard that never reports still
590
+ leaves the run incomplete after the run timeout.
591
+
592
+ Other CI systems re-run a job under the id of the run it belongs to - a GitLab
593
+ job retry, an Azure Pipelines re-run, a Bitbucket re-run of failed steps, a
594
+ Buildkite retry, a Travis CI restart. Since 0.11.0 Squally refuses such a
595
+ retry at the start with `ci_run_collision`: the slot is taken by the first
596
+ attempt. **`SQUALLY_CI_JOB_KEY` does not help here** - the retry is the same
597
+ job - so a retry on these systems is not reported; the first attempt stays the
598
+ run. A Drone restart is a new build with a new number, and so a new run in
599
+ Squally, not linked to the one it repeats.
600
+
601
+ **`playwright merge-reports` is not part of this and is not recommended** for
602
+ getting a single run in Squally - the automatic correlation above is the
603
+ supported path, and produces correct per-shard timing/branch/commit
604
+ information that a merge job cannot (see the historical note below). Nothing
605
+ stops you from using `merge-reports` for Playwright's own HTML report if you
606
+ want that separately; just don't point this reporter at the merged result
607
+ expecting Squally to treat it specially.
608
+
609
+ <details>
610
+ <summary>Why not just use `merge-reports`, historically</summary>
611
+
612
+ An earlier version of this reporter recommended `playwright merge-reports` as
613
+ the way to get one Squally run out of a sharded job. That path has a real
614
+ problem: branch, commit, commit message and author are all read on whichever
615
+ machine the reporter actually runs on - during a merge that is the merge job,
616
+ not the shards - and without a checkout there (and without the CI provider's
617
+ own commit variables) they all come out empty. A run with no branch is in
618
+ no branch's numbers and no branch filter finds it, so a "merged" run could
619
+ exist in the dashboard while missing from everything Squally measures per
620
+ branch: suite
621
+ stability, and each test's stability, flaky rate and failure rate on that
622
+ branch. The automatic correlation above reads that information from each
623
+ shard's own environment instead, so this problem does not arise.
624
+
625
+ </details>
626
+
627
+ ## The run's Playwright config and repository (0.16.0)
628
+
629
+ At the start of a run the reporter sends what Squally's **Metadata** tab
630
+ shows under *Settings* and *Playwright projects*, read field by field from
631
+ Playwright's config - never the config itself:
632
+
633
+ - **Settings:** `workers` as configured and the workers Playwright actually
634
+ started (newer Playwright versions only; 1.30 does not report it),
635
+ `fullyParallel`, `forbidOnly`, `globalTimeout`, `maxFailures`, and the
636
+ reporters by name - Playwright's own (`html`, `list`, ...) and npm packages
637
+ by package name; a reporter file of your own is sent as `custom`, never its
638
+ path.
639
+ - **Per project:** its name, `browserName`, `channel`, `viewport`, the
640
+ `trace`/`video`/`screenshot` mode, `retries` and `timeout`.
641
+ - **The repository** as host and path, e.g. `github.com/acme/shop`: from your
642
+ CI's variables (`GITHUB_REPOSITORY`, `CI_PROJECT_PATH`,
643
+ `BITBUCKET_REPO_FULL_NAME`, `BUILD_REPOSITORY_URI`, `BUILDKITE_REPO`,
644
+ `CIRCLE_REPOSITORY_URL`, `CODEBUILD_SOURCE_REPO_URL`, `DRONE_GIT_HTTP_URL`,
645
+ `TRAVIS_REPO_SLUG`, `GIT_URL`), else `git remote get-url origin`.
646
+ Credentials, a query and `.git` are removed before it is sent.
647
+ - **Where in the repository the run started** (0.19.0): the directory
648
+ Playwright runs in, relative to the repository's root, as
649
+ `git rev-parse --show-prefix` prints it - `e2e/` in a monorepo, empty at
650
+ the root. Test file paths are relative to that directory, so Squally needs
651
+ it to link a test to its file. Not sent when git cannot say.
652
+
653
+ Two switches:
654
+
655
+ ```bash
656
+ # The base URL of each project (use.baseURL) - off unless you turn it on.
657
+ # Only scheme, host and path are sent, never a login, query or fragment.
658
+ SQUALLY_SEND_BASE_URL=true
659
+
660
+ # The repository and where in it the run started - on unless you turn it off.
661
+ SQUALLY_SEND_REPOSITORY=false
662
+ ```
663
+
664
+ **Never sent:** the rest of `use` (extra HTTP headers, HTTP credentials,
665
+ `storageState`, proxy, launch and context options), `config.metadata`, the
666
+ command line, the config file and every absolute directory path, `webServer`, `grep`,
667
+ global setup and teardown, and each project's own `metadata`.
668
+
669
+ ## Tested revision (optional)
670
+
671
+ The tested revision is the state of your **application** a run tested, as
672
+ opposed to the commit the run is reported under. Squally does not know it by
673
+ itself - it is **opt-in**, with one of these two:
674
+
675
+ ```bash
676
+ # The tests live in the same repo as the application: the commit this CI job
677
+ # checked out is the tested revision.
678
+ SAME_REPO_AS_APP=true
679
+
680
+ # The tests live in a separate repo (e.g. a dedicated E2E repo pointed at a
681
+ # deployed app): pass the tested revision itself - a commit SHA, tag, or
682
+ # branch name/ref.
683
+ TESTED_REVISION=v1.2.0
684
+ ```
685
+
686
+ - **`SAME_REPO_AS_APP=true`** sends the commit your job checked out: your CI
687
+ provider's own variable for it (`GITHUB_SHA`, `CI_COMMIT_SHA`,
688
+ `CIRCLE_SHA1`, `GIT_COMMIT`, `BUILD_SOURCEVERSION`, `BITBUCKET_COMMIT`,
689
+ `BUILDKITE_COMMIT`, `BUILD_VCS_NUMBER`, `CODEBUILD_RESOLVED_SOURCE_VERSION`,
690
+ `TRAVIS_COMMIT`, `DRONE_COMMIT_SHA`), or `git rev-parse HEAD` where there is
691
+ none - including local runs. Only a commit SHA is sent; if neither yields
692
+ one, the run has no tested revision.
693
+ - **On a pull request the two commits differ, on purpose.** The run is
694
+ reported under the pull request's head commit (see the provider table
695
+ above) - that is what Squally's check and PR comment need. The tested
696
+ revision is the merge commit CI checked out, because that is what the tests
697
+ ran against. On a Bitbucket pull request and a GitHub `pull_request_target`
698
+ run it comes from `git rev-parse HEAD`.
699
+ - **`TESTED_REVISION` wins** when both are set.
700
+ - Neither variable is required - without them, test results are still sent
701
+ and stored exactly as before.
702
+
703
+ **The value is normalised before it is stored**, so that the same revision
704
+ written two ways still compares equal: it is lowercased, and a leading `v` or a
705
+ `refs/heads/` prefix is removed. The example above is exactly such a case -
706
+ `v1.2.0` is stored as `1.2.0`. The reporter says so once per run when it
707
+ happens, so a value that looks different in the dashboard is not a mystery.
708
+
709
+ ## Behind a proxy (0.18.0)
710
+
711
+ Every request - to Squally, the artifact uploads to storage, and
712
+ `npx squally-reporter cancel` - goes through the proxy in `HTTPS_PROXY` (for
713
+ `https://` addresses) or `HTTP_PROXY` (for `http://`), lower-case names
714
+ first, the way curl reads them. Hosts in `NO_PROXY` are reached directly:
715
+ comma-separated, each name with all its subdomains (`internal` and
716
+ `.internal` both cover `squally.internal`), `host:port` for one port only, `*`
717
+ for everything - no IP ranges:
718
+
719
+ ```bash
720
+ HTTPS_PROXY=http://proxy.internal:3128
721
+ NO_PROXY=localhost,127.0.0.1,.internal
722
+ ```
723
+
724
+ The proxy is asked to open a tunnel (`CONNECT`), so TLS still goes end to end
725
+ to Squally. A login in the proxy URL (`http://user:pass@host:port`) is sent as
726
+ `Proxy-Authorization`. A value that is not a URL is one warning, naming the
727
+ variable but not its value, and the reporter connects directly. Without any of
728
+ these variables nothing changes. Until 0.17.0 the variables were ignored.
729
+
730
+ ## Troubleshooting
731
+
732
+ Set `SQUALLY_DEBUG=1` to print detail that is otherwise kept quiet - the
733
+ browser-console extraction (how many messages were found, or that an attempt's
734
+ trace held none), since 0.11.0 a CI run link that was not sent because it was
735
+ not an `https://` URL, and since 0.18.0 which proxy variable is in use and a
736
+ quarantine rules request that failed (nothing uses the rules yet, so that is
737
+ no longer a warning). Everything that is *not* ordinary (a trace that cannot
738
+ be read, an unsupported schema version, a rejected upload) is printed without
739
+ it.
740
+
741
+ ## Support
742
+
743
+ Found a bug or have a question? Please open an issue on
744
+ [GitHub Issues](https://github.com/squallydev-admin/squally-reporter/issues) -
745
+ that's the primary way to report problems or ask questions. For anything you
746
+ would rather not post publicly, write to
747
+ [support@squally.dev](mailto:support@squally.dev).
748
+
749
+ ## License
750
+
751
+ MIT - see [LICENSE](./LICENSE).