squally-reporter 0.19.0 → 0.19.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +751 -751
- package/package.json +54 -54
package/README.md
CHANGED
|
@@ -1,751 +1,751 @@
|
|
|
1
|
-
# squally-reporter
|
|
2
|
-
|
|
3
|
-
Playwright reporter that sends test results to your Squally dashboard.
|
|
4
|
-
|
|
5
|
-
Learn more at [app.squally.dev](https://app.squally.dev).
|
|
6
|
-
|
|
7
|
-
## Requirements
|
|
8
|
-
|
|
9
|
-
- Node.js >= 20.18.1
|
|
10
|
-
- `@playwright/test` >= 1.30.0
|
|
11
|
-
|
|
12
|
-
## Install
|
|
13
|
-
|
|
14
|
-
```bash
|
|
15
|
-
npm install -D squally-reporter
|
|
16
|
-
```
|
|
17
|
-
|
|
18
|
-
## Config
|
|
19
|
-
|
|
20
|
-
Set one env var (e.g. as a CI secret):
|
|
21
|
-
|
|
22
|
-
```bash
|
|
23
|
-
SQUALLY_INGEST_KEY=your-project-ingest-key
|
|
24
|
-
```
|
|
25
|
-
|
|
26
|
-
Find your ingest key in the Squally dashboard under **Settings › Project ›
|
|
27
|
-
Ingest key**.
|
|
28
|
-
|
|
29
|
-
**Since 0.12.0 the URL defaults to `https://app.squally.dev`.** Set
|
|
30
|
-
`SQUALLY_INGEST_URL` only to report to another Squally. Without a key the
|
|
31
|
-
reporter sends nothing and says so once.
|
|
32
|
-
|
|
33
|
-
**Renamed in 0.9.0.** These were `SQUALLY_API_URL` and `SQUALLY_API_KEY`. The
|
|
34
|
-
old names still work for now, with a deprecation warning in the CI log - but
|
|
35
|
-
they belong to Squally's read API now, so rename them. `SQUALLY_API_URL` is
|
|
36
|
-
only used together with `SQUALLY_API_KEY`, never on its own - and when it is
|
|
37
|
-
set but not used, the reporter sends nothing rather than falling back to the
|
|
38
|
-
default URL, since the variable says you meant another target.
|
|
39
|
-
|
|
40
|
-
**A read key is never sent.** A key starting with `sqly_ro_` is a read API key,
|
|
41
|
-
for scripts and agents; reporting needs the ingest key. Since 0.12.0 the
|
|
42
|
-
reporter refuses one in any of `SQUALLY_INGEST_KEY`, `SQUALLY_API_KEY` or the
|
|
43
|
-
`apiKey` option before sending anything, and says which one to fix.
|
|
44
|
-
|
|
45
|
-
Then add the reporter to `playwright.config.ts`:
|
|
46
|
-
|
|
47
|
-
```ts
|
|
48
|
-
import { defineConfig } from "@playwright/test";
|
|
49
|
-
|
|
50
|
-
export default defineConfig({
|
|
51
|
-
reporter: [
|
|
52
|
-
["html"],
|
|
53
|
-
["squally-reporter"],
|
|
54
|
-
],
|
|
55
|
-
});
|
|
56
|
-
```
|
|
57
|
-
|
|
58
|
-
**Replace your `reporter` line, don't add a second one.** `npm init playwright`
|
|
59
|
-
writes `reporter: 'html',`, and with two `reporter:` keys in the config the
|
|
60
|
-
later one wins without a warning - either the HTML report or Squally silently
|
|
61
|
-
goes. Using other reporters? Keep them in the list and add
|
|
62
|
-
`["squally-reporter"]`.
|
|
63
|
-
|
|
64
|
-
Alternatively, pass config explicitly instead of using env vars - but **prefer
|
|
65
|
-
the env vars above for `apiKey`, and not only because a literal in a committed
|
|
66
|
-
file is obviously wrong.** Playwright's `json` and `blob` reporters serialize
|
|
67
|
-
your resolved config into their output file, options and all: an `apiKey`
|
|
68
|
-
passed here ends up in that file in plain text, even when its value came from
|
|
69
|
-
an environment variable at runtime. What gets written is the option, not where
|
|
70
|
-
it came from. The reporter warns, naming the file, when it sees this
|
|
71
|
-
combination. Measured against a real report, 20.09.2026.
|
|
72
|
-
|
|
73
|
-
```ts
|
|
74
|
-
reporter: [
|
|
75
|
-
["squally-reporter", { apiUrl: "https://app.squally.dev", apiKey: process.env.SQUALLY_INGEST_KEY }],
|
|
76
|
-
],
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
That's it - run your tests as usual (`npx playwright test`); results stream
|
|
80
|
-
to Squally while the suite runs (see *How results reach Squally* below).
|
|
81
|
-
|
|
82
|
-
## Artifact upload (screenshot, trace, browser console)
|
|
83
|
-
|
|
84
|
-
If your `playwright.config.ts` captures screenshots or traces, the reporter
|
|
85
|
-
uploads them to Squally so they show up next to the failing test in the
|
|
86
|
-
dashboard - no separate setup beyond the two env vars above.
|
|
87
|
-
|
|
88
|
-
Three kinds are uploaded:
|
|
89
|
-
|
|
90
|
-
| Kind | Where it comes from |
|
|
91
|
-
| --- | --- |
|
|
92
|
-
| **Screenshot** | the file Playwright writes for a failed attempt |
|
|
93
|
-
| **Trace** | the trace zip Playwright writes |
|
|
94
|
-
| **Browser console** | read out of that trace zip by this reporter |
|
|
95
|
-
|
|
96
|
-
**The browser console needs a trace.** It is not something Playwright hands a
|
|
97
|
-
reporter: `result.stdout`/`stderr` are the output of the *Node* process - what
|
|
98
|
-
your test file printed - while what the *page* printed only exists inside the
|
|
99
|
-
trace. So the reporter opens the trace and reads it out, which means an
|
|
100
|
-
attempt with no trace has no browser console, and nothing can change that but
|
|
101
|
-
turning tracing on. Squally's attempt page says which of the two it is rather
|
|
102
|
-
than showing an empty section.
|
|
103
|
-
|
|
104
|
-
Supported trace schema versions: **5 to 10**, which covers Playwright 1.30
|
|
105
|
-
through 1.62 (1.62 writes version 8). A trace outside that range is left alone
|
|
106
|
-
and the reporter prints one warning naming the version - it never uploads an
|
|
107
|
-
empty console file, because "no messages" and "this reporter could not read
|
|
108
|
-
it" must not look the same in the dashboard.
|
|
109
|
-
|
|
110
|
-
**Only from attempts worth looking at:** artifacts are uploaded only for
|
|
111
|
-
attempts that **failed**, or that **needed a retry** (even if that retry then
|
|
112
|
-
passed) - never for an attempt that passed on its first try, regardless of how
|
|
113
|
-
`screenshot`/`trace` are configured. Nobody looks at a screenshot of a test
|
|
114
|
-
that just passed, and uploading one anyway would only eat into your
|
|
115
|
-
organization's storage for no benefit.
|
|
116
|
-
|
|
117
|
-
**Uploaded while the suite runs (0.18.0).** An attempt's artifacts go up as
|
|
118
|
-
soon as Squally has accepted the batch that carries its result - not only
|
|
119
|
-
after the last test, as until 0.17.0. A job that is cancelled or killed
|
|
120
|
-
partway keeps the artifacts of every result that already arrived. When the
|
|
121
|
-
suite ends, the reporter waits **up to 3 minutes** for the uploads still
|
|
122
|
-
running; the last 15 seconds of that only finish the confirmation, no new
|
|
123
|
-
upload starts in them. A file is sent as a stream, not read into memory
|
|
124
|
-
first.
|
|
125
|
-
|
|
126
|
-
**You need to actually enable capture in `playwright.config.ts`** - without
|
|
127
|
-
it, Playwright doesn't produce any attachments, so there's nothing for the
|
|
128
|
-
reporter to find or upload (no error either way, it's a silent no-op):
|
|
129
|
-
|
|
130
|
-
```ts
|
|
131
|
-
export default defineConfig({
|
|
132
|
-
use: {
|
|
133
|
-
screenshot: "only-on-failure",
|
|
134
|
-
trace: "retain-on-failure",
|
|
135
|
-
},
|
|
136
|
-
// ...
|
|
137
|
-
});
|
|
138
|
-
```
|
|
139
|
-
|
|
140
|
-
**Limits.** None of these are enforced by the reporter - Squally applies them
|
|
141
|
-
when the upload is requested and answers with a reason, which is what the
|
|
142
|
-
reporter prints. The numbers below are therefore Squally's current ones rather
|
|
143
|
-
than anything this package guarantees, and can change server-side without a new
|
|
144
|
-
release of it:
|
|
145
|
-
|
|
146
|
-
- **50 MB per screenshot or trace, 4 MB per browser console**, the same on
|
|
147
|
-
every plan. A larger file is rejected before any bytes are uploaded; the
|
|
148
|
-
reporter logs a `console.warn` naming the file and its limit and moves on.
|
|
149
|
-
The test result itself is reported as usual, and the run's outcome and exit
|
|
150
|
-
code are untouched. In the dashboard that artifact is shown as skipped, with
|
|
151
|
-
the reason, rather than silently missing.
|
|
152
|
-
- **At most 250 artifacts per request.** Since 0.18.0 the reporter sends them
|
|
153
|
-
in rounds of at most 250, so a shard with more no longer loses the rest.
|
|
154
|
-
Until 0.17.0 a shard's artifacts went in one request, and Squally kept the
|
|
155
|
-
first 250.
|
|
156
|
-
- **A total storage cap per organization, depending on your Squally plan.**
|
|
157
|
-
Once it is reached, further uploads are rejected the same way until you free
|
|
158
|
-
up space or upgrade. Note that a fully instrumented suite now produces up to
|
|
159
|
-
three artifacts per failed attempt rather than two.
|
|
160
|
-
- The browser console is a text file of a few KB - it is extracted from the
|
|
161
|
-
trace you are already uploading, so it adds no capture cost to your run.
|
|
162
|
-
- **Your test's own `stdout`/`stderr` is cut at 16 KB** before it is sent. That
|
|
163
|
-
is this reporter's own limit, not Squally's, and it applies whether or not
|
|
164
|
-
you capture artifacts at all.
|
|
165
|
-
- Two minutes per upload, and 30 seconds for the calls that surround it. Since
|
|
166
|
-
0.18.0 a network error, an HTTP 5xx or a 429 is retried like the results
|
|
167
|
-
(see *When Squally cannot be reached for a while*), and the 3 minutes at the
|
|
168
|
-
end bound the whole wait. What did not arrive is said once, at the end:
|
|
169
|
-
|
|
170
|
-
```
|
|
171
|
-
[squally-reporter] 2 of 9 artifacts did not reach Squally - the storage answered HTTP 503, and the retries ran out. Your test run and its exit code are not affected.
|
|
172
|
-
```
|
|
173
|
-
|
|
174
|
-
Upload is best-effort and fully non-blocking, same principle as the rest of
|
|
175
|
-
this reporter - a failed or skipped upload never fails your test run or
|
|
176
|
-
changes its exit code, it only logs a `console.warn` explaining what
|
|
177
|
-
happened.
|
|
178
|
-
|
|
179
|
-
## How results reach Squally
|
|
180
|
-
|
|
181
|
-
As soon as the first test starts, the reporter creates the run in Squally and
|
|
182
|
-
prints where to watch it (0.15.0), on stderr:
|
|
183
|
-
|
|
184
|
-
```
|
|
185
|
-
[squally-reporter] Reporting to Squally: run #128 · https://app.squally.dev/o/…/runs/…
|
|
186
|
-
```
|
|
187
|
-
|
|
188
|
-
Under `--shard` the line reads `run #128, shard 2 of 3`; every shard of a CI
|
|
189
|
-
run has the same number.
|
|
190
|
-
|
|
191
|
-
Results stream to Squally as your suite runs, not all at once at the end:
|
|
192
|
-
each test's result is sent in a small batch, so the "running" view in the
|
|
193
|
-
dashboard stays close to real time even for a long suite. A batch goes out on
|
|
194
|
-
whichever of these comes first:
|
|
195
|
-
|
|
196
|
-
- **25 results.** Fixed, and the same whether you shard or not.
|
|
197
|
-
- **A timer: 2 seconds, or 0.4 seconds × the number of shards when that is
|
|
198
|
-
longer, capped at 30 seconds** (0.12.0). Up to five shards every shard
|
|
199
|
-
flushes every 2 seconds; eight shards every 3.2; sixteen every 6.4; from 75
|
|
200
|
-
shards the cap holds it at 30.
|
|
201
|
-
|
|
202
|
-
Only the timer scales. All shards of a run share one rate budget on Squally's
|
|
203
|
-
side (300 requests a minute per key), and the timer keeps all of them
|
|
204
|
-
together at about half of it. The 25-result threshold is left alone
|
|
205
|
-
deliberately: it bounds how long a *fast* suite waits, and stretching it would
|
|
206
|
-
delay results without bounding the burst any better.
|
|
207
|
-
|
|
208
|
-
A batch also stays under 3 MB (0.12.0): results with long error texts or many
|
|
209
|
-
steps simply go out in more, smaller requests.
|
|
210
|
-
|
|
211
|
-
There is no longer a single-request size limit that could discard a whole run -
|
|
212
|
-
a very large suite simply takes more, smaller requests. As a courtesy the
|
|
213
|
-
reporter still warns if a run's result count climbs well past 5,000, since that
|
|
214
|
-
usually means retries are configured higher than intended rather than a suite
|
|
215
|
-
that's actually that large.
|
|
216
|
-
|
|
217
|
-
If Squally is unreachable partway through, results already sent stay in the
|
|
218
|
-
dashboard; only what could not be delivered is lost, and your test run's exit
|
|
219
|
-
code is never affected either way.
|
|
220
|
-
|
|
221
|
-
## The end of a run (0.18.0)
|
|
222
|
-
|
|
223
|
-
When every result arrived and the run is marked complete, the last line says
|
|
224
|
-
where it is and what got there:
|
|
225
|
-
|
|
226
|
-
```
|
|
227
|
-
[squally-reporter] Reported to Squally: run #128 · 412 results, 9 artifacts · https://app.squally.dev/o/…/runs/…
|
|
228
|
-
```
|
|
229
|
-
|
|
230
|
-
If something did not arrive, that line is replaced by the one that says what
|
|
231
|
-
is missing (see below).
|
|
232
|
-
|
|
233
|
-
**`SQUALLY_OUTPUT_FILE`** writes the run's link and counts as JSON, for a
|
|
234
|
-
later CI step that posts the link somewhere:
|
|
235
|
-
|
|
236
|
-
```bash
|
|
237
|
-
SQUALLY_OUTPUT_FILE=test-results/squally.json
|
|
238
|
-
```
|
|
239
|
-
|
|
240
|
-
```json
|
|
241
|
-
{
|
|
242
|
-
"version": 1,
|
|
243
|
-
"runUrl": "https://app.squally.dev/o/…/runs/…",
|
|
244
|
-
"runNumber": 128,
|
|
245
|
-
"runId": "…",
|
|
246
|
-
"shard": null,
|
|
247
|
-
"complete": true,
|
|
248
|
-
"tests": { "passed": 40, "failed": 1, "flaky": 1, "skipped": 0 },
|
|
249
|
-
"results": { "produced": 44, "delivered": 44 },
|
|
250
|
-
"artifacts": { "uploaded": 3, "notUploaded": 0 }
|
|
251
|
-
}
|
|
252
|
-
```
|
|
253
|
-
|
|
254
|
-
- Written at the very end of every run the reporter took part in - also one
|
|
255
|
-
Squally refused or could not be reached for, then with `runUrl: null` and
|
|
256
|
-
`complete: false`, so the file is always there for the next step. Missing
|
|
257
|
-
directories are created.
|
|
258
|
-
- `tests` counts this process's tests the way Playwright does at the end
|
|
259
|
-
(`expected` as passed, `unexpected` as failed, `flaky`, `skipped`).
|
|
260
|
-
`results` counts attempts: produced, and accepted by Squally.
|
|
261
|
-
- One file per process: under `--shard`, give each shard its own path (or
|
|
262
|
-
read them one after another), and `shard` says which one it was.
|
|
263
|
-
- A path that cannot be written is one warning; the exit code stays yours.
|
|
264
|
-
- New fields may be added; `version` changes only if one changes meaning.
|
|
265
|
-
|
|
266
|
-
## When Squally cannot be reached for a while (0.17.0)
|
|
267
|
-
|
|
268
|
-
Some failures pass on their own, and the reporter waits them out instead of
|
|
269
|
-
dropping results:
|
|
270
|
-
|
|
271
|
-
- a network error or a request without an answer within 30 seconds,
|
|
272
|
-
- an HTTP 5xx,
|
|
273
|
-
- HTTP 429 (Squally's rate limit),
|
|
274
|
-
- HTTP 403 from Vercel's firewall in front of Squally - the answer carries
|
|
275
|
-
`x-vercel-mitigated: challenge` (or `deny`). The firewall can challenge one
|
|
276
|
-
runner for a few minutes; it says nothing about your key.
|
|
277
|
-
|
|
278
|
-
Each request is retried after 1 second, then 2, 4, 8 and so on up to 30
|
|
279
|
-
seconds between tries (each wait shortened by a random part, so shards
|
|
280
|
-
refused together do not come back together), for up to 90 seconds. A
|
|
281
|
-
`Retry-After` header is honoured. The start of the run gets the same 90
|
|
282
|
-
seconds; your tests run meanwhile and their results are held until Squally
|
|
283
|
-
answers. While Squally keeps refusing, results go out one request at a time.
|
|
284
|
-
|
|
285
|
-
Not retried, because waiting does not change them: Squally's own error
|
|
286
|
-
answers (a refused key, an exhausted quota, an unknown or finished run), an
|
|
287
|
-
HTTP 403 *without* the firewall's header, and a redirect. Those keep their own
|
|
288
|
-
line, as before.
|
|
289
|
-
|
|
290
|
-
The artifact uploads follow the same rules since 0.18.0 (until then each was
|
|
291
|
-
tried once): the request for the upload URL, the upload to storage (a network
|
|
292
|
-
error, HTTP 5xx or 429) and the confirmation. They have their own line at the
|
|
293
|
-
end and never delay the results.
|
|
294
|
-
|
|
295
|
-
When your suite ends, the reporter normally waits up to 5 seconds for the last
|
|
296
|
-
results. After one of the failures above it waits up to 60 seconds, for the
|
|
297
|
-
last results and for marking the run complete.
|
|
298
|
-
|
|
299
|
-
What did not arrive is said once, at the end - not once per batch:
|
|
300
|
-
|
|
301
|
-
```
|
|
302
|
-
[squally-reporter] 37 of 412 results did not reach Squally - Vercel's firewall in front of Squally challenged this runner (HTTP 403, x-vercel-mitigated: challenge), and the retries ran out. Your test run and its exit code are not affected. The run page in Squally shows how many are missing.
|
|
303
|
-
```
|
|
304
|
-
|
|
305
|
-
If the run could not be marked complete either, the line says so: the run
|
|
306
|
-
then shows as **incomplete** in Squally once the project's run timeout has
|
|
307
|
-
passed, with the results that did arrive. If the run could not be started at
|
|
308
|
-
all, the line reads `Squally did not start this run - ...` and nothing was
|
|
309
|
-
sent.
|
|
310
|
-
|
|
311
|
-
The exit code of your test run is never changed by any of this.
|
|
312
|
-
|
|
313
|
-
## Steps (0.12.0)
|
|
314
|
-
|
|
315
|
-
For an attempt that **failed, or needed a retry**, the reporter also sends the
|
|
316
|
-
steps Playwright recorded - hooks, fixtures, actions, assertions and your own
|
|
317
|
-
`test.step`s, with their durations and `file:line`. Squally shows them on the
|
|
318
|
-
attempt page, the failing step marked. An attempt that passed on its first try
|
|
319
|
-
sends none.
|
|
320
|
-
|
|
321
|
-
- **Typed values never leave your runner.** The value of a `fill`, `type` or
|
|
322
|
-
`pressSequentially` is replaced: `Fill "•••" getByLabel('Password')`. Every
|
|
323
|
-
title also goes through the same secret masking as error texts.
|
|
324
|
-
- **At most 300 steps per attempt, 200 characters per title.** A longer list
|
|
325
|
-
keeps the failing step and the steps it sits in, then the rest in order, and
|
|
326
|
-
tells Squally how many were left out.
|
|
327
|
-
- Nothing to configure: steps come from Playwright's own report, with or
|
|
328
|
-
without tracing.
|
|
329
|
-
|
|
330
|
-
**Only actual test runs appear.** A run shows up in Squally once your first
|
|
331
|
-
test starts, so commands that never run a test create nothing: `--list`,
|
|
332
|
-
a `globalSetup` or `webServer` that fails to start, a `--grep` that matches
|
|
333
|
-
nothing. A run that *does* execute and legitimately contains no tests — an
|
|
334
|
-
empty shard, `--only-changed` with no changed test file, `--pass-with-no-tests`
|
|
335
|
-
— is still reported, as a run with zero results.
|
|
336
|
-
|
|
337
|
-
**Timeouts and interrupted tests.** A test that runs out of time is reported
|
|
338
|
-
as failed, and Squally is told that it timed out and what its timeout was -
|
|
339
|
-
including a `test.setTimeout()` or `test.slow()` from inside the test. A test
|
|
340
|
-
that Playwright cuts off because the whole run is stopping (`maxFailures`
|
|
341
|
-
reached, Ctrl+C) is reported as skipped: it did not fail, and it does not
|
|
342
|
-
count against the test.
|
|
343
|
-
|
|
344
|
-
## Annotations, the test's line, worker and start (0.19.0)
|
|
345
|
-
|
|
346
|
-
Every attempt also sends:
|
|
347
|
-
|
|
348
|
-
- **Its annotations** - `test.info().annotations`, the `{ annotation }`
|
|
349
|
-
option, and the ones `test.skip()`, `test.fixme()`, `test.slow()` and
|
|
350
|
-
`test.fail()` add: type, description, and where it was added (`file:line`).
|
|
351
|
-
Squally shows them on the run's Tests tab, the attempt page and the test
|
|
352
|
-
page; an `issue` whose description is a link becomes a link there.
|
|
353
|
-
- **Descriptions go through the same secret masking as error texts**, and
|
|
354
|
-
are cut at 1,000 characters.
|
|
355
|
-
- **At most 20 per attempt**, types cut at 64 characters; Squally is told
|
|
356
|
-
how many were left out.
|
|
357
|
-
- On Playwright versions without per-attempt annotations, the test's own
|
|
358
|
-
list is sent with every attempt.
|
|
359
|
-
- **The test's line and column** in its file, for the attempt page's
|
|
360
|
-
*Open in repository* link.
|
|
361
|
-
- **The worker that ran it, its slot and when it started** (`workerIndex`,
|
|
362
|
-
`parallelIndex`, `startTime`). Squally stores them for a later timeline of
|
|
363
|
-
the run; nothing shows them yet.
|
|
364
|
-
|
|
365
|
-
Nothing to configure. An older Squally ignores these fields.
|
|
366
|
-
|
|
367
|
-
## Cancelled runs
|
|
368
|
-
|
|
369
|
-
**Cancelled in Squally.** An admin can cancel a running run from its page in
|
|
370
|
-
Squally. That marks the run there; it does not stop your CI job. What the job
|
|
371
|
-
sends afterwards is discarded, and from 0.12.0 the reporter says so in one
|
|
372
|
-
line - "This run was cancelled in Squally" - instead of advice about job keys.
|
|
373
|
-
|
|
374
|
-
A run stopped with SIGINT - Ctrl+C, or a cancelled CI job - shows in Squally
|
|
375
|
-
as **Cancelled**: the reporter sends what finished and tells Squally the run
|
|
376
|
-
was interrupted. Your exit code stays Playwright's own (130). This needs
|
|
377
|
-
Playwright itself to receive the signal.
|
|
378
|
-
|
|
379
|
-
**On GitHub Actions** a cancelled workflow sends the signal to the step's
|
|
380
|
-
shell, not to Playwright, so the reporter never reports the end. Add one step
|
|
381
|
-
after the tests (0.14.0) that marks the run cancelled in Squally:
|
|
382
|
-
|
|
383
|
-
```yaml
|
|
384
|
-
jobs:
|
|
385
|
-
test:
|
|
386
|
-
runs-on: ubuntu-latest
|
|
387
|
-
env:
|
|
388
|
-
SQUALLY_INGEST_KEY: ${{ secrets.SQUALLY_INGEST_KEY }}
|
|
389
|
-
steps:
|
|
390
|
-
# ...checkout, setup-node, npm ci, playwright install
|
|
391
|
-
- name: Run Playwright tests
|
|
392
|
-
run: npx playwright test
|
|
393
|
-
- name: Mark Squally run cancelled
|
|
394
|
-
if: ${{ cancelled() }}
|
|
395
|
-
run: npx squally-reporter cancel
|
|
396
|
-
```
|
|
397
|
-
|
|
398
|
-
`npx squally-reporter cancel` finds the run by the same CI run id the reporter
|
|
399
|
-
sent from this job, marks every shard of it that is still running as
|
|
400
|
-
cancelled, and prints one line: "Marked run #N as cancelled in Squally
|
|
401
|
-
(X shards)", "Nothing to cancel: no run recorded for <id>" when the job was
|
|
402
|
-
cancelled before its first test, or "Run already finished". It never fails the
|
|
403
|
-
step: a missing key, an unreachable Squally, a 401 or a 5xx is one warning and
|
|
404
|
-
exit code 0. It needs the key in the step's environment - set it at job
|
|
405
|
-
level, as above, or on the step too. A Squally without the cancel endpoint
|
|
406
|
-
answers 404, which is a warning as well.
|
|
407
|
-
|
|
408
|
-
Two things to know:
|
|
409
|
-
|
|
410
|
-
- **Set `SQUALLY_CI_JOB_KEY` at job level**, not only on the test step. The
|
|
411
|
-
cancel step builds the id the same way the reporter did; without the key it
|
|
412
|
-
names another run - or none. `npx squally-reporter cancel --job-key <key>`
|
|
413
|
-
passes it explicitly.
|
|
414
|
-
- **A fail-fast matrix marks the whole group cancelled.** With GitHub's
|
|
415
|
-
default `fail-fast: true`, one failing shard cancels the others, their
|
|
416
|
-
cancel steps run, and the whole run shows as **Cancelled** instead of
|
|
417
|
-
**Failed**. Set `fail-fast: false` on a sharded job so a
|
|
418
|
-
real failure stays a failure.
|
|
419
|
-
|
|
420
|
-
Without the step, a cancelled workflow shows as **incomplete** in Squally.
|
|
421
|
-
|
|
422
|
-
Without that - or after any hard kill (SIGKILL) - the reporter never gets to
|
|
423
|
-
report the end: the run stays *running* until Squally's hourly check marks it
|
|
424
|
-
**incomplete** once the project's run timeout (60 minutes by default) has
|
|
425
|
-
passed. A run stopped before its first test starts sends nothing, like any
|
|
426
|
-
run that never ran a test, and `globalTimeout` running out is not a cancel -
|
|
427
|
-
that run is simply failed.
|
|
428
|
-
|
|
429
|
-
## Which CI, branch and commit a run is reported under
|
|
430
|
-
|
|
431
|
-
Squally counts **CI runs** in its test metrics, so the reporter works out
|
|
432
|
-
which CI system it is running on from that system's own environment:
|
|
433
|
-
|
|
434
|
-
| Provider | Recognised by | Shards grouped by | Branch | Commit |
|
|
435
|
-
|---|---|---|---|---|
|
|
436
|
-
| GitHub Actions | `GITHUB_ACTIONS` | `GITHUB_RUN_ID`-`GITHUB_RUN_ATTEMPT` | `GITHUB_REF_NAME`; pull requests: `GITHUB_HEAD_REF` | `GITHUB_SHA`; pull requests: the PR's head commit |
|
|
437
|
-
| GitLab CI | `GITLAB_CI` | `CI_PIPELINE_ID` | `CI_COMMIT_BRANCH`; merge requests: `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | `CI_COMMIT_SHA`; merged-results pipelines: `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` |
|
|
438
|
-
| CircleCI | `CIRCLECI` | `CIRCLE_WORKFLOW_ID` | `CIRCLE_BRANCH` | `CIRCLE_SHA1` |
|
|
439
|
-
| Jenkins | `BUILD_TAG` (`jenkins-…`) | `BUILD_TAG` | `CHANGE_BRANCH`, else `BRANCH_NAME`, else `GIT_BRANCH` | `GIT_COMMIT` |
|
|
440
|
-
| Azure Pipelines | `TF_BUILD` | `BUILD_BUILDID` | `BUILD_SOURCEBRANCH`; pull requests: `SYSTEM_PULLREQUEST_SOURCEBRANCH` | `BUILD_SOURCEVERSION`; pull requests: `SYSTEM_PULLREQUEST_SOURCECOMMITID` |
|
|
441
|
-
| Bitbucket Pipelines | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_BRANCH` | `BITBUCKET_COMMIT` |
|
|
442
|
-
| Buildkite | `BUILDKITE` | `BUILDKITE_BUILD_ID` | `BUILDKITE_BRANCH` | `BUILDKITE_COMMIT` |
|
|
443
|
-
| TeamCity | `TEAMCITY_VERSION` | - (set `SQUALLY_CI_RUN_ID`) | from git | `BUILD_VCS_NUMBER` |
|
|
444
|
-
| AWS CodeBuild | `CODEBUILD_BUILD_ID` | `CODEBUILD_BUILD_ID` | `CODEBUILD_WEBHOOK_HEAD_REF` | `CODEBUILD_RESOLVED_SOURCE_VERSION` |
|
|
445
|
-
| Travis CI | `TRAVIS` | `TRAVIS_BUILD_ID` | `TRAVIS_BRANCH`; pull requests: `TRAVIS_PULL_REQUEST_BRANCH` | `TRAVIS_COMMIT`; pull requests: `TRAVIS_PULL_REQUEST_SHA` |
|
|
446
|
-
| Drone | `DRONE` | `DRONE_BUILD_NUMBER` | `DRONE_BRANCH`; pull requests: `DRONE_SOURCE_BRANCH` | `DRONE_COMMIT_SHA` |
|
|
447
|
-
| any other CI | `CI=true` or `CI=1` | - (set `SQUALLY_CI_RUN_ID`) | from git | from git |
|
|
448
|
-
|
|
449
|
-
- **`CI` counts only as `true` or `1`.** `CI=false` or `CI=0` is a local run -
|
|
450
|
-
unless one of the providers above is recognised, which always wins. A run
|
|
451
|
-
that is none of these is local. Local runs are listed with all other runs,
|
|
452
|
-
and have numbers of their own: Squally's figures count CI runs by default,
|
|
453
|
-
and the **All · CI · Local** switch shows local runs on their own, or both.
|
|
454
|
-
A local run never posts a GitHub check, a pull-request comment or a Slack
|
|
455
|
-
message.
|
|
456
|
-
- **Pull requests are reported under their own head commit and source
|
|
457
|
-
branch**, not the merge commit CI checks out - that is the commit Squally's
|
|
458
|
-
GitHub check and pull-request comment have to find. The commit message and
|
|
459
|
-
author are that commit's too - never the "Merge … into …" message of the
|
|
460
|
-
checkout.
|
|
461
|
-
- **The head commit is fetched when the checkout lacks it.** With
|
|
462
|
-
`actions/checkout`'s default `fetch-depth: 1` the head commit is not in the
|
|
463
|
-
clone, so the reporter fetches that one commit from `origin`
|
|
464
|
-
(`git fetch --depth=1 --no-tags`), waits at most 3 seconds, and reads its
|
|
465
|
-
message and author. It only does so in a shallow checkout - fetching into a
|
|
466
|
-
full clone would make it shallow for the rest of your job - and never
|
|
467
|
-
changes `HEAD` or any branch. If the fetch fails or takes longer, the
|
|
468
|
-
message and author stay empty and the reporter says so once; the run and
|
|
469
|
-
its exit code are unaffected. Alternatives: `fetch-depth: 2` on the checkout
|
|
470
|
-
step puts the head commit in the clone, so nothing is fetched, and
|
|
471
|
-
`SQUALLY_DISABLE_HEAD_COMMIT_FETCH=true` turns the fetch off.
|
|
472
|
-
- **A tag build has no branch.** A detached checkout never reports `HEAD` as a
|
|
473
|
-
branch name, and `refs/heads/` or `origin/` prefixes are removed.
|
|
474
|
-
- **A link to the CI run (0.11.0).** The reporter sends the page of the CI run,
|
|
475
|
-
so Squally can link a run to it. It is read from each provider's own
|
|
476
|
-
variable, sent only as an `https://` URL of at most 2,048 characters, and
|
|
477
|
-
never opened by Squally - only shown as a link:
|
|
478
|
-
|
|
479
|
-
| Provider | Link |
|
|
480
|
-
|---|---|
|
|
481
|
-
| GitHub Actions | `GITHUB_SERVER_URL`/`GITHUB_REPOSITORY`/actions/runs/`GITHUB_RUN_ID`/attempts/`GITHUB_RUN_ATTEMPT` |
|
|
482
|
-
| GitLab CI | `CI_PIPELINE_URL` |
|
|
483
|
-
| Jenkins | `BUILD_URL` (only with Jenkins served over https) |
|
|
484
|
-
| Buildkite | `BUILDKITE_BUILD_URL` |
|
|
485
|
-
| Travis CI | `TRAVIS_BUILD_WEB_URL` |
|
|
486
|
-
| Drone | `DRONE_BUILD_LINK` |
|
|
487
|
-
|
|
488
|
-
Other providers send no link. It is the run's page, never one job's, and it
|
|
489
|
-
is the same with `SQUALLY_CI_RUN_ID` set.
|
|
490
|
-
|
|
491
|
-
## Tests marked with `test.fail()`
|
|
492
|
-
|
|
493
|
-
Playwright's `test.fail()` marks a test as *expected to fail*: it guards a bug
|
|
494
|
-
you already know about, and Playwright runs the test anyway to check that it
|
|
495
|
-
does still fail. The interesting moment is the opposite one - when someone
|
|
496
|
-
fixes the bug, the test starts passing and Playwright turns it red so you
|
|
497
|
-
remember to delete the marker.
|
|
498
|
-
|
|
499
|
-
The reporter passes that marker on, so Squally reads such a test the same way
|
|
500
|
-
Playwright does:
|
|
501
|
-
|
|
502
|
-
- **Marked and still failing** - shown as passing, with an "expected failure"
|
|
503
|
-
label. It counts as a pass in the test's stability, not against its failure
|
|
504
|
-
rate, and it does not fail Squally's pull-request check.
|
|
505
|
-
- **Marked and now passing** - shown as a failure, because the marker is out of
|
|
506
|
-
date and should be removed.
|
|
507
|
-
- **Marked and timed out** - shown as a failure. Playwright does not count a
|
|
508
|
-
timeout as the expected failure either, so the marker is not passed on for
|
|
509
|
-
it.
|
|
510
|
-
|
|
511
|
-
Both ways of marking a test are recognised: the `test.fail()` modifier and a
|
|
512
|
-
`testInfo.fail()` call inside the test body. `test.skip()` and `test.fixme()`
|
|
513
|
-
are a different statement and are not affected.
|
|
514
|
-
|
|
515
|
-
Nothing is reported for an unmarked test, so an ordinary suite is unchanged.
|
|
516
|
-
Note also that Playwright itself treats a marked, failing test as expected, so
|
|
517
|
-
with `screenshot: "only-on-failure"` it captures nothing for it - there is no
|
|
518
|
-
artifact for the reporter to upload, which is consistent rather than missing.
|
|
519
|
-
|
|
520
|
-
## Sharding
|
|
521
|
-
|
|
522
|
-
**Handled automatically - no extra setup needed.** Each shard is still a
|
|
523
|
-
separate process with its own reporter instance, but Squally now correlates
|
|
524
|
-
them itself: all shards of one CI run are grouped together and shown as one
|
|
525
|
-
run once every shard has finished, with the flip side of that guarantee being
|
|
526
|
-
that the run only completes once every shard *has* reported.
|
|
527
|
-
|
|
528
|
-
Correlation is based on an id read from your CI provider's own environment
|
|
529
|
-
(the "Shards grouped by" column above), so it costs you nothing on every
|
|
530
|
-
provider in that table except TeamCity. On TeamCity, on a CI system not in the
|
|
531
|
-
table, or when you shard locally across several separate `--shard`
|
|
532
|
-
invocations, set `SQUALLY_CI_RUN_ID` to the same value for every shard of one
|
|
533
|
-
run - anything unique per CI run and shared by its shards works, e.g. a build
|
|
534
|
-
number your CI system already exposes.
|
|
535
|
-
|
|
536
|
-
**Several jobs in one CI run need their own key (0.11.0).** The id is shared
|
|
537
|
-
by everything in one CI run, so two jobs that are *not* shards of each other -
|
|
538
|
-
a GitHub Actions matrix over browsers run without `--shard`, or an `api` and a
|
|
539
|
-
`ui` suite - would report into the same Squally run. Give each job its own
|
|
540
|
-
`SQUALLY_CI_JOB_KEY`, shared only by that job's own shards:
|
|
541
|
-
|
|
542
|
-
```yaml
|
|
543
|
-
jobs:
|
|
544
|
-
test:
|
|
545
|
-
strategy:
|
|
546
|
-
matrix:
|
|
547
|
-
browser: [chromium, firefox]
|
|
548
|
-
env:
|
|
549
|
-
SQUALLY_CI_JOB_KEY: ${{ matrix.browser }}
|
|
550
|
-
steps:
|
|
551
|
-
- run: npx playwright test --project=${{ matrix.browser }}
|
|
552
|
-
```
|
|
553
|
-
|
|
554
|
-
The reporter puts the key into the id - `<GITHUB_RUN_ID>-<key>-<GITHUB_RUN_ATTEMPT>`
|
|
555
|
-
on GitHub Actions, `<provider id>-<key>` elsewhere - so each job is its own run
|
|
556
|
-
in Squally, and a re-run of one job links to that job's own earlier attempt.
|
|
557
|
-
The key may hold letters, digits, `.`, `_` and `-`, at most 64 characters; an
|
|
558
|
-
invalid key is reported once and ignored, and your test run is unaffected.
|
|
559
|
-
Without a key the id is exactly what it was before 0.11.0. A suite that is
|
|
560
|
-
sharded *and* split into jobs uses the same key for all shards of one job:
|
|
561
|
-
`SQUALLY_CI_JOB_KEY: ${{ matrix.suite }}` beside `--shard=${{ matrix.shard }}/2`.
|
|
562
|
-
|
|
563
|
-
**Replaces the hand-built id.** Until 0.11.0 this section suggested
|
|
564
|
-
`SQUALLY_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.browser }}`.
|
|
565
|
-
It still groups each job correctly, but Squally cannot tell its attempts apart,
|
|
566
|
-
so a re-run is never linked to the attempt before it. Switch to
|
|
567
|
-
`SQUALLY_CI_JOB_KEY`. An explicit `SQUALLY_CI_RUN_ID` still wins for the id;
|
|
568
|
-
the key is then only sent alongside it.
|
|
569
|
-
|
|
570
|
-
**When two jobs collide anyway, Squally refuses the second one at the start**
|
|
571
|
-
(squally-app with block 1b, reporter 0.11.0). The reporter prints one block,
|
|
572
|
-
sends nothing more for that job, and leaves your test run and its exit code
|
|
573
|
-
alone:
|
|
574
|
-
|
|
575
|
-
- `ci_run_collision` - another job already reports under this CI run id (and
|
|
576
|
-
shard). Give every job its own `SQUALLY_CI_JOB_KEY`.
|
|
577
|
-
- `ci_run_mixed` - the id would put a sharded job and an unsharded one, or
|
|
578
|
-
shards of different totals, into one run. The same fix.
|
|
579
|
-
|
|
580
|
-
Before 0.11.0 the second job was handed the first job's run instead, and its
|
|
581
|
-
results were refused once that run completed; a reporter older than 0.11.0
|
|
582
|
-
still behaves that way, and prints "Squally has already completed run …".
|
|
583
|
-
|
|
584
|
-
**Re-runs.** On GitHub Actions every re-run attempt is its own run in Squally,
|
|
585
|
-
linked to the attempt before it ("Re-run of #N"). "Re-run failed jobs" runs only
|
|
586
|
-
the shards that did not pass; that attempt waits only for those shards,
|
|
587
|
-
completes once they have reported, and passes or fails on the tests they ran -
|
|
588
|
-
the reused shards are neither waited for nor counted, and the run page shows
|
|
589
|
-
them as "passed in #N · not re-run". A re-run shard that never reports still
|
|
590
|
-
leaves the run incomplete after the run timeout.
|
|
591
|
-
|
|
592
|
-
Other CI systems re-run a job under the id of the run it belongs to - a GitLab
|
|
593
|
-
job retry, an Azure Pipelines re-run, a Bitbucket re-run of failed steps, a
|
|
594
|
-
Buildkite retry, a Travis CI restart. Since 0.11.0 Squally refuses such a
|
|
595
|
-
retry at the start with `ci_run_collision`: the slot is taken by the first
|
|
596
|
-
attempt. **`SQUALLY_CI_JOB_KEY` does not help here** - the retry is the same
|
|
597
|
-
job - so a retry on these systems is not reported; the first attempt stays the
|
|
598
|
-
run. A Drone restart is a new build with a new number, and so a new run in
|
|
599
|
-
Squally, not linked to the one it repeats.
|
|
600
|
-
|
|
601
|
-
**`playwright merge-reports` is not part of this and is not recommended** for
|
|
602
|
-
getting a single run in Squally - the automatic correlation above is the
|
|
603
|
-
supported path, and produces correct per-shard timing/branch/commit
|
|
604
|
-
information that a merge job cannot (see the historical note below). Nothing
|
|
605
|
-
stops you from using `merge-reports` for Playwright's own HTML report if you
|
|
606
|
-
want that separately; just don't point this reporter at the merged result
|
|
607
|
-
expecting Squally to treat it specially.
|
|
608
|
-
|
|
609
|
-
<details>
|
|
610
|
-
<summary>Why not just use `merge-reports`, historically</summary>
|
|
611
|
-
|
|
612
|
-
An earlier version of this reporter recommended `playwright merge-reports` as
|
|
613
|
-
the way to get one Squally run out of a sharded job. That path has a real
|
|
614
|
-
problem: branch, commit, commit message and author are all read on whichever
|
|
615
|
-
machine the reporter actually runs on - during a merge that is the merge job,
|
|
616
|
-
not the shards - and without a checkout there (and without the CI provider's
|
|
617
|
-
own commit variables) they all come out empty. A run with no branch is in
|
|
618
|
-
no branch's numbers and no branch filter finds it, so a "merged" run could
|
|
619
|
-
exist in the dashboard while missing from everything Squally measures per
|
|
620
|
-
branch: suite
|
|
621
|
-
stability, and each test's stability, flaky rate and failure rate on that
|
|
622
|
-
branch. The automatic correlation above reads that information from each
|
|
623
|
-
shard's own environment instead, so this problem does not arise.
|
|
624
|
-
|
|
625
|
-
</details>
|
|
626
|
-
|
|
627
|
-
## The run's Playwright config and repository (0.16.0)
|
|
628
|
-
|
|
629
|
-
At the start of a run the reporter sends what Squally's **Metadata** tab
|
|
630
|
-
shows under *Settings* and *Playwright projects*, read field by field from
|
|
631
|
-
Playwright's config - never the config itself:
|
|
632
|
-
|
|
633
|
-
- **Settings:** `workers` as configured and the workers Playwright actually
|
|
634
|
-
started (newer Playwright versions only; 1.30 does not report it),
|
|
635
|
-
`fullyParallel`, `forbidOnly`, `globalTimeout`, `maxFailures`, and the
|
|
636
|
-
reporters by name - Playwright's own (`html`, `list`, ...) and npm packages
|
|
637
|
-
by package name; a reporter file of your own is sent as `custom`, never its
|
|
638
|
-
path.
|
|
639
|
-
- **Per project:** its name, `browserName`, `channel`, `viewport`, the
|
|
640
|
-
`trace`/`video`/`screenshot` mode, `retries` and `timeout`.
|
|
641
|
-
- **The repository** as host and path, e.g. `github.com/acme/shop`: from your
|
|
642
|
-
CI's variables (`GITHUB_REPOSITORY`, `CI_PROJECT_PATH`,
|
|
643
|
-
`BITBUCKET_REPO_FULL_NAME`, `BUILD_REPOSITORY_URI`, `BUILDKITE_REPO`,
|
|
644
|
-
`CIRCLE_REPOSITORY_URL`, `CODEBUILD_SOURCE_REPO_URL`, `DRONE_GIT_HTTP_URL`,
|
|
645
|
-
`TRAVIS_REPO_SLUG`, `GIT_URL`), else `git remote get-url origin`.
|
|
646
|
-
Credentials, a query and `.git` are removed before it is sent.
|
|
647
|
-
- **Where in the repository the run started** (0.19.0): the directory
|
|
648
|
-
Playwright runs in, relative to the repository's root, as
|
|
649
|
-
`git rev-parse --show-prefix` prints it - `e2e/` in a monorepo, empty at
|
|
650
|
-
the root. Test file paths are relative to that directory, so Squally needs
|
|
651
|
-
it to link a test to its file. Not sent when git cannot say.
|
|
652
|
-
|
|
653
|
-
Two switches:
|
|
654
|
-
|
|
655
|
-
```bash
|
|
656
|
-
# The base URL of each project (use.baseURL) - off unless you turn it on.
|
|
657
|
-
# Only scheme, host and path are sent, never a login, query or fragment.
|
|
658
|
-
SQUALLY_SEND_BASE_URL=true
|
|
659
|
-
|
|
660
|
-
# The repository and where in it the run started - on unless you turn it off.
|
|
661
|
-
SQUALLY_SEND_REPOSITORY=false
|
|
662
|
-
```
|
|
663
|
-
|
|
664
|
-
**Never sent:** the rest of `use` (extra HTTP headers, HTTP credentials,
|
|
665
|
-
`storageState`, proxy, launch and context options), `config.metadata`, the
|
|
666
|
-
command line, the config file and every absolute directory path, `webServer`, `grep`,
|
|
667
|
-
global setup and teardown, and each project's own `metadata`.
|
|
668
|
-
|
|
669
|
-
## Tested revision (optional)
|
|
670
|
-
|
|
671
|
-
The tested revision is the state of your **application** a run tested, as
|
|
672
|
-
opposed to the commit the run is reported under. Squally does not know it by
|
|
673
|
-
itself - it is **opt-in**, with one of these two:
|
|
674
|
-
|
|
675
|
-
```bash
|
|
676
|
-
# The tests live in the same repo as the application: the commit this CI job
|
|
677
|
-
# checked out is the tested revision.
|
|
678
|
-
SAME_REPO_AS_APP=true
|
|
679
|
-
|
|
680
|
-
# The tests live in a separate repo (e.g. a dedicated E2E repo pointed at a
|
|
681
|
-
# deployed app): pass the tested revision itself - a commit SHA, tag, or
|
|
682
|
-
# branch name/ref.
|
|
683
|
-
TESTED_REVISION=v1.2.0
|
|
684
|
-
```
|
|
685
|
-
|
|
686
|
-
- **`SAME_REPO_AS_APP=true`** sends the commit your job checked out: your CI
|
|
687
|
-
provider's own variable for it (`GITHUB_SHA`, `CI_COMMIT_SHA`,
|
|
688
|
-
`CIRCLE_SHA1`, `GIT_COMMIT`, `BUILD_SOURCEVERSION`, `BITBUCKET_COMMIT`,
|
|
689
|
-
`BUILDKITE_COMMIT`, `BUILD_VCS_NUMBER`, `CODEBUILD_RESOLVED_SOURCE_VERSION`,
|
|
690
|
-
`TRAVIS_COMMIT`, `DRONE_COMMIT_SHA`), or `git rev-parse HEAD` where there is
|
|
691
|
-
none - including local runs. Only a commit SHA is sent; if neither yields
|
|
692
|
-
one, the run has no tested revision.
|
|
693
|
-
- **On a pull request the two commits differ, on purpose.** The run is
|
|
694
|
-
reported under the pull request's head commit (see the provider table
|
|
695
|
-
above) - that is what Squally's check and PR comment need. The tested
|
|
696
|
-
revision is the merge commit CI checked out, because that is what the tests
|
|
697
|
-
ran against. On a Bitbucket pull request and a GitHub `pull_request_target`
|
|
698
|
-
run it comes from `git rev-parse HEAD`.
|
|
699
|
-
- **`TESTED_REVISION` wins** when both are set.
|
|
700
|
-
- Neither variable is required - without them, test results are still sent
|
|
701
|
-
and stored exactly as before.
|
|
702
|
-
|
|
703
|
-
**The value is normalised before it is stored**, so that the same revision
|
|
704
|
-
written two ways still compares equal: it is lowercased, and a leading `v` or a
|
|
705
|
-
`refs/heads/` prefix is removed. The example above is exactly such a case -
|
|
706
|
-
`v1.2.0` is stored as `1.2.0`. The reporter says so once per run when it
|
|
707
|
-
happens, so a value that looks different in the dashboard is not a mystery.
|
|
708
|
-
|
|
709
|
-
## Behind a proxy (0.18.0)
|
|
710
|
-
|
|
711
|
-
Every request - to Squally, the artifact uploads to storage, and
|
|
712
|
-
`npx squally-reporter cancel` - goes through the proxy in `HTTPS_PROXY` (for
|
|
713
|
-
`https://` addresses) or `HTTP_PROXY` (for `http://`), lower-case names
|
|
714
|
-
first, the way curl reads them. Hosts in `NO_PROXY` are reached directly:
|
|
715
|
-
comma-separated, each name with all its subdomains (`internal` and
|
|
716
|
-
`.internal` both cover `squally.internal`), `host:port` for one port only, `*`
|
|
717
|
-
for everything - no IP ranges:
|
|
718
|
-
|
|
719
|
-
```bash
|
|
720
|
-
HTTPS_PROXY=http://proxy.internal:3128
|
|
721
|
-
NO_PROXY=localhost,127.0.0.1,.internal
|
|
722
|
-
```
|
|
723
|
-
|
|
724
|
-
The proxy is asked to open a tunnel (`CONNECT`), so TLS still goes end to end
|
|
725
|
-
to Squally. A login in the proxy URL (`http://user:pass@host:port`) is sent as
|
|
726
|
-
`Proxy-Authorization`. A value that is not a URL is one warning, naming the
|
|
727
|
-
variable but not its value, and the reporter connects directly. Without any of
|
|
728
|
-
these variables nothing changes. Until 0.17.0 the variables were ignored.
|
|
729
|
-
|
|
730
|
-
## Troubleshooting
|
|
731
|
-
|
|
732
|
-
Set `SQUALLY_DEBUG=1` to print detail that is otherwise kept quiet - the
|
|
733
|
-
browser-console extraction (how many messages were found, or that an attempt's
|
|
734
|
-
trace held none), since 0.11.0 a CI run link that was not sent because it was
|
|
735
|
-
not an `https://` URL, and since 0.18.0 which proxy variable is in use and a
|
|
736
|
-
quarantine rules request that failed (nothing uses the rules yet, so that is
|
|
737
|
-
no longer a warning). Everything that is *not* ordinary (a trace that cannot
|
|
738
|
-
be read, an unsupported schema version, a rejected upload) is printed without
|
|
739
|
-
it.
|
|
740
|
-
|
|
741
|
-
## Support
|
|
742
|
-
|
|
743
|
-
Found a bug or have a question? Please open an issue on
|
|
744
|
-
[GitHub Issues](https://github.com/squallydev-admin/squally-reporter/issues) -
|
|
745
|
-
that's the primary way to report problems or ask questions. For anything you
|
|
746
|
-
would rather not post publicly, write to
|
|
747
|
-
[support@squally.dev](mailto:support@squally.dev).
|
|
748
|
-
|
|
749
|
-
## License
|
|
750
|
-
|
|
751
|
-
MIT - see [LICENSE](./LICENSE).
|
|
1
|
+
# squally-reporter
|
|
2
|
+
|
|
3
|
+
Playwright reporter that sends test results to your Squally dashboard.
|
|
4
|
+
|
|
5
|
+
Learn more at [app.squally.dev](https://app.squally.dev).
|
|
6
|
+
|
|
7
|
+
## Requirements
|
|
8
|
+
|
|
9
|
+
- Node.js >= 20.18.1
|
|
10
|
+
- `@playwright/test` >= 1.30.0
|
|
11
|
+
|
|
12
|
+
## Install
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
npm install -D squally-reporter
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
## Config
|
|
19
|
+
|
|
20
|
+
Set one env var (e.g. as a CI secret):
|
|
21
|
+
|
|
22
|
+
```bash
|
|
23
|
+
SQUALLY_INGEST_KEY=your-project-ingest-key
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Find your ingest key in the Squally dashboard under **Settings › Project ›
|
|
27
|
+
Ingest key**.
|
|
28
|
+
|
|
29
|
+
**Since 0.12.0 the URL defaults to `https://app.squally.dev`.** Set
|
|
30
|
+
`SQUALLY_INGEST_URL` only to report to another Squally. Without a key the
|
|
31
|
+
reporter sends nothing and says so once.
|
|
32
|
+
|
|
33
|
+
**Renamed in 0.9.0.** These were `SQUALLY_API_URL` and `SQUALLY_API_KEY`. The
|
|
34
|
+
old names still work for now, with a deprecation warning in the CI log - but
|
|
35
|
+
they belong to Squally's read API now, so rename them. `SQUALLY_API_URL` is
|
|
36
|
+
only used together with `SQUALLY_API_KEY`, never on its own - and when it is
|
|
37
|
+
set but not used, the reporter sends nothing rather than falling back to the
|
|
38
|
+
default URL, since the variable says you meant another target.
|
|
39
|
+
|
|
40
|
+
**A read key is never sent.** A key starting with `sqly_ro_` is a read API key,
|
|
41
|
+
for scripts and agents; reporting needs the ingest key. Since 0.12.0 the
|
|
42
|
+
reporter refuses one in any of `SQUALLY_INGEST_KEY`, `SQUALLY_API_KEY` or the
|
|
43
|
+
`apiKey` option before sending anything, and says which one to fix.
|
|
44
|
+
|
|
45
|
+
Then add the reporter to `playwright.config.ts`:
|
|
46
|
+
|
|
47
|
+
```ts
|
|
48
|
+
import { defineConfig } from "@playwright/test";
|
|
49
|
+
|
|
50
|
+
export default defineConfig({
|
|
51
|
+
reporter: [
|
|
52
|
+
["html"],
|
|
53
|
+
["squally-reporter"],
|
|
54
|
+
],
|
|
55
|
+
});
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
**Replace your `reporter` line, don't add a second one.** `npm init playwright`
|
|
59
|
+
writes `reporter: 'html',`, and with two `reporter:` keys in the config the
|
|
60
|
+
later one wins without a warning - either the HTML report or Squally silently
|
|
61
|
+
goes. Using other reporters? Keep them in the list and add
|
|
62
|
+
`["squally-reporter"]`.
|
|
63
|
+
|
|
64
|
+
Alternatively, pass config explicitly instead of using env vars - but **prefer
|
|
65
|
+
the env vars above for `apiKey`, and not only because a literal in a committed
|
|
66
|
+
file is obviously wrong.** Playwright's `json` and `blob` reporters serialize
|
|
67
|
+
your resolved config into their output file, options and all: an `apiKey`
|
|
68
|
+
passed here ends up in that file in plain text, even when its value came from
|
|
69
|
+
an environment variable at runtime. What gets written is the option, not where
|
|
70
|
+
it came from. The reporter warns, naming the file, when it sees this
|
|
71
|
+
combination. Measured against a real report, 20.09.2026.
|
|
72
|
+
|
|
73
|
+
```ts
|
|
74
|
+
reporter: [
|
|
75
|
+
["squally-reporter", { apiUrl: "https://app.squally.dev", apiKey: process.env.SQUALLY_INGEST_KEY }],
|
|
76
|
+
],
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
That's it - run your tests as usual (`npx playwright test`); results stream
|
|
80
|
+
to Squally while the suite runs (see *How results reach Squally* below).
|
|
81
|
+
|
|
82
|
+
## Artifact upload (screenshot, trace, browser console)
|
|
83
|
+
|
|
84
|
+
If your `playwright.config.ts` captures screenshots or traces, the reporter
|
|
85
|
+
uploads them to Squally so they show up next to the failing test in the
|
|
86
|
+
dashboard - no separate setup beyond the two env vars above.
|
|
87
|
+
|
|
88
|
+
Three kinds are uploaded:
|
|
89
|
+
|
|
90
|
+
| Kind | Where it comes from |
|
|
91
|
+
| --- | --- |
|
|
92
|
+
| **Screenshot** | the file Playwright writes for a failed attempt |
|
|
93
|
+
| **Trace** | the trace zip Playwright writes |
|
|
94
|
+
| **Browser console** | read out of that trace zip by this reporter |
|
|
95
|
+
|
|
96
|
+
**The browser console needs a trace.** It is not something Playwright hands a
|
|
97
|
+
reporter: `result.stdout`/`stderr` are the output of the *Node* process - what
|
|
98
|
+
your test file printed - while what the *page* printed only exists inside the
|
|
99
|
+
trace. So the reporter opens the trace and reads it out, which means an
|
|
100
|
+
attempt with no trace has no browser console, and nothing can change that but
|
|
101
|
+
turning tracing on. Squally's attempt page says which of the two it is rather
|
|
102
|
+
than showing an empty section.
|
|
103
|
+
|
|
104
|
+
Supported trace schema versions: **5 to 10**, which covers Playwright 1.30
|
|
105
|
+
through 1.62 (1.62 writes version 8). A trace outside that range is left alone
|
|
106
|
+
and the reporter prints one warning naming the version - it never uploads an
|
|
107
|
+
empty console file, because "no messages" and "this reporter could not read
|
|
108
|
+
it" must not look the same in the dashboard.
|
|
109
|
+
|
|
110
|
+
**Only from attempts worth looking at:** artifacts are uploaded only for
|
|
111
|
+
attempts that **failed**, or that **needed a retry** (even if that retry then
|
|
112
|
+
passed) - never for an attempt that passed on its first try, regardless of how
|
|
113
|
+
`screenshot`/`trace` are configured. Nobody looks at a screenshot of a test
|
|
114
|
+
that just passed, and uploading one anyway would only eat into your
|
|
115
|
+
organization's storage for no benefit.
|
|
116
|
+
|
|
117
|
+
**Uploaded while the suite runs (0.18.0).** An attempt's artifacts go up as
|
|
118
|
+
soon as Squally has accepted the batch that carries its result - not only
|
|
119
|
+
after the last test, as until 0.17.0. A job that is cancelled or killed
|
|
120
|
+
partway keeps the artifacts of every result that already arrived. When the
|
|
121
|
+
suite ends, the reporter waits **up to 3 minutes** for the uploads still
|
|
122
|
+
running; the last 15 seconds of that only finish the confirmation, no new
|
|
123
|
+
upload starts in them. A file is sent as a stream, not read into memory
|
|
124
|
+
first.
|
|
125
|
+
|
|
126
|
+
**You need to actually enable capture in `playwright.config.ts`** - without
|
|
127
|
+
it, Playwright doesn't produce any attachments, so there's nothing for the
|
|
128
|
+
reporter to find or upload (no error either way, it's a silent no-op):
|
|
129
|
+
|
|
130
|
+
```ts
|
|
131
|
+
export default defineConfig({
|
|
132
|
+
use: {
|
|
133
|
+
screenshot: "only-on-failure",
|
|
134
|
+
trace: "retain-on-failure",
|
|
135
|
+
},
|
|
136
|
+
// ...
|
|
137
|
+
});
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
**Limits.** None of these are enforced by the reporter - Squally applies them
|
|
141
|
+
when the upload is requested and answers with a reason, which is what the
|
|
142
|
+
reporter prints. The numbers below are therefore Squally's current ones rather
|
|
143
|
+
than anything this package guarantees, and can change server-side without a new
|
|
144
|
+
release of it:
|
|
145
|
+
|
|
146
|
+
- **50 MB per screenshot or trace, 4 MB per browser console**, the same on
|
|
147
|
+
every plan. A larger file is rejected before any bytes are uploaded; the
|
|
148
|
+
reporter logs a `console.warn` naming the file and its limit and moves on.
|
|
149
|
+
The test result itself is reported as usual, and the run's outcome and exit
|
|
150
|
+
code are untouched. In the dashboard that artifact is shown as skipped, with
|
|
151
|
+
the reason, rather than silently missing.
|
|
152
|
+
- **At most 250 artifacts per request.** Since 0.18.0 the reporter sends them
|
|
153
|
+
in rounds of at most 250, so a shard with more no longer loses the rest.
|
|
154
|
+
Until 0.17.0 a shard's artifacts went in one request, and Squally kept the
|
|
155
|
+
first 250.
|
|
156
|
+
- **A total storage cap per organization, depending on your Squally plan.**
|
|
157
|
+
Once it is reached, further uploads are rejected the same way until you free
|
|
158
|
+
up space or upgrade. Note that a fully instrumented suite now produces up to
|
|
159
|
+
three artifacts per failed attempt rather than two.
|
|
160
|
+
- The browser console is a text file of a few KB - it is extracted from the
|
|
161
|
+
trace you are already uploading, so it adds no capture cost to your run.
|
|
162
|
+
- **Your test's own `stdout`/`stderr` is cut at 16 KB** before it is sent. That
|
|
163
|
+
is this reporter's own limit, not Squally's, and it applies whether or not
|
|
164
|
+
you capture artifacts at all.
|
|
165
|
+
- Two minutes per upload, and 30 seconds for the calls that surround it. Since
|
|
166
|
+
0.18.0 a network error, an HTTP 5xx or a 429 is retried like the results
|
|
167
|
+
(see *When Squally cannot be reached for a while*), and the 3 minutes at the
|
|
168
|
+
end bound the whole wait. What did not arrive is said once, at the end:
|
|
169
|
+
|
|
170
|
+
```
|
|
171
|
+
[squally-reporter] 2 of 9 artifacts did not reach Squally - the storage answered HTTP 503, and the retries ran out. Your test run and its exit code are not affected.
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
Upload is best-effort and fully non-blocking, same principle as the rest of
|
|
175
|
+
this reporter - a failed or skipped upload never fails your test run or
|
|
176
|
+
changes its exit code, it only logs a `console.warn` explaining what
|
|
177
|
+
happened.
|
|
178
|
+
|
|
179
|
+
## How results reach Squally
|
|
180
|
+
|
|
181
|
+
As soon as the first test starts, the reporter creates the run in Squally and
|
|
182
|
+
prints where to watch it (0.15.0), on stderr:
|
|
183
|
+
|
|
184
|
+
```
|
|
185
|
+
[squally-reporter] Reporting to Squally: run #128 · https://app.squally.dev/o/…/runs/…
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Under `--shard` the line reads `run #128, shard 2 of 3`; every shard of a CI
|
|
189
|
+
run has the same number.
|
|
190
|
+
|
|
191
|
+
Results stream to Squally as your suite runs, not all at once at the end:
|
|
192
|
+
each test's result is sent in a small batch, so the "running" view in the
|
|
193
|
+
dashboard stays close to real time even for a long suite. A batch goes out on
|
|
194
|
+
whichever of these comes first:
|
|
195
|
+
|
|
196
|
+
- **25 results.** Fixed, and the same whether you shard or not.
|
|
197
|
+
- **A timer: 2 seconds, or 0.4 seconds × the number of shards when that is
|
|
198
|
+
longer, capped at 30 seconds** (0.12.0). Up to five shards every shard
|
|
199
|
+
flushes every 2 seconds; eight shards every 3.2; sixteen every 6.4; from 75
|
|
200
|
+
shards the cap holds it at 30.
|
|
201
|
+
|
|
202
|
+
Only the timer scales. All shards of a run share one rate budget on Squally's
|
|
203
|
+
side (300 requests a minute per key), and the timer keeps all of them
|
|
204
|
+
together at about half of it. The 25-result threshold is left alone
|
|
205
|
+
deliberately: it bounds how long a *fast* suite waits, and stretching it would
|
|
206
|
+
delay results without bounding the burst any better.
|
|
207
|
+
|
|
208
|
+
A batch also stays under 3 MB (0.12.0): results with long error texts or many
|
|
209
|
+
steps simply go out in more, smaller requests.
|
|
210
|
+
|
|
211
|
+
There is no longer a single-request size limit that could discard a whole run -
|
|
212
|
+
a very large suite simply takes more, smaller requests. As a courtesy the
|
|
213
|
+
reporter still warns if a run's result count climbs well past 5,000, since that
|
|
214
|
+
usually means retries are configured higher than intended rather than a suite
|
|
215
|
+
that's actually that large.
|
|
216
|
+
|
|
217
|
+
If Squally is unreachable partway through, results already sent stay in the
|
|
218
|
+
dashboard; only what could not be delivered is lost, and your test run's exit
|
|
219
|
+
code is never affected either way.
|
|
220
|
+
|
|
221
|
+
## The end of a run (0.18.0)
|
|
222
|
+
|
|
223
|
+
When every result arrived and the run is marked complete, the last line says
|
|
224
|
+
where it is and what got there:
|
|
225
|
+
|
|
226
|
+
```
|
|
227
|
+
[squally-reporter] Reported to Squally: run #128 · 412 results, 9 artifacts · https://app.squally.dev/o/…/runs/…
|
|
228
|
+
```
|
|
229
|
+
|
|
230
|
+
If something did not arrive, that line is replaced by the one that says what
|
|
231
|
+
is missing (see below).
|
|
232
|
+
|
|
233
|
+
**`SQUALLY_OUTPUT_FILE`** writes the run's link and counts as JSON, for a
|
|
234
|
+
later CI step that posts the link somewhere:
|
|
235
|
+
|
|
236
|
+
```bash
|
|
237
|
+
SQUALLY_OUTPUT_FILE=test-results/squally.json
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
```json
|
|
241
|
+
{
|
|
242
|
+
"version": 1,
|
|
243
|
+
"runUrl": "https://app.squally.dev/o/…/runs/…",
|
|
244
|
+
"runNumber": 128,
|
|
245
|
+
"runId": "…",
|
|
246
|
+
"shard": null,
|
|
247
|
+
"complete": true,
|
|
248
|
+
"tests": { "passed": 40, "failed": 1, "flaky": 1, "skipped": 0 },
|
|
249
|
+
"results": { "produced": 44, "delivered": 44 },
|
|
250
|
+
"artifacts": { "uploaded": 3, "notUploaded": 0 }
|
|
251
|
+
}
|
|
252
|
+
```
|
|
253
|
+
|
|
254
|
+
- Written at the very end of every run the reporter took part in - also one
|
|
255
|
+
Squally refused or could not be reached for, then with `runUrl: null` and
|
|
256
|
+
`complete: false`, so the file is always there for the next step. Missing
|
|
257
|
+
directories are created.
|
|
258
|
+
- `tests` counts this process's tests the way Playwright does at the end
|
|
259
|
+
(`expected` as passed, `unexpected` as failed, `flaky`, `skipped`).
|
|
260
|
+
`results` counts attempts: produced, and accepted by Squally.
|
|
261
|
+
- One file per process: under `--shard`, give each shard its own path (or
|
|
262
|
+
read them one after another), and `shard` says which one it was.
|
|
263
|
+
- A path that cannot be written is one warning; the exit code stays yours.
|
|
264
|
+
- New fields may be added; `version` changes only if one changes meaning.
|
|
265
|
+
|
|
266
|
+
## When Squally cannot be reached for a while (0.17.0)
|
|
267
|
+
|
|
268
|
+
Some failures pass on their own, and the reporter waits them out instead of
|
|
269
|
+
dropping results:
|
|
270
|
+
|
|
271
|
+
- a network error or a request without an answer within 30 seconds,
|
|
272
|
+
- an HTTP 5xx,
|
|
273
|
+
- HTTP 429 (Squally's rate limit),
|
|
274
|
+
- HTTP 403 from Vercel's firewall in front of Squally - the answer carries
|
|
275
|
+
`x-vercel-mitigated: challenge` (or `deny`). The firewall can challenge one
|
|
276
|
+
runner for a few minutes; it says nothing about your key.
|
|
277
|
+
|
|
278
|
+
Each request is retried after 1 second, then 2, 4, 8 and so on up to 30
|
|
279
|
+
seconds between tries (each wait shortened by a random part, so shards
|
|
280
|
+
refused together do not come back together), for up to 90 seconds. A
|
|
281
|
+
`Retry-After` header is honoured. The start of the run gets the same 90
|
|
282
|
+
seconds; your tests run meanwhile and their results are held until Squally
|
|
283
|
+
answers. While Squally keeps refusing, results go out one request at a time.
|
|
284
|
+
|
|
285
|
+
Not retried, because waiting does not change them: Squally's own error
|
|
286
|
+
answers (a refused key, an exhausted quota, an unknown or finished run), an
|
|
287
|
+
HTTP 403 *without* the firewall's header, and a redirect. Those keep their own
|
|
288
|
+
line, as before.
|
|
289
|
+
|
|
290
|
+
The artifact uploads follow the same rules since 0.18.0 (until then each was
|
|
291
|
+
tried once): the request for the upload URL, the upload to storage (a network
|
|
292
|
+
error, HTTP 5xx or 429) and the confirmation. They have their own line at the
|
|
293
|
+
end and never delay the results.
|
|
294
|
+
|
|
295
|
+
When your suite ends, the reporter normally waits up to 5 seconds for the last
|
|
296
|
+
results. After one of the failures above it waits up to 60 seconds, for the
|
|
297
|
+
last results and for marking the run complete.
|
|
298
|
+
|
|
299
|
+
What did not arrive is said once, at the end - not once per batch:
|
|
300
|
+
|
|
301
|
+
```
|
|
302
|
+
[squally-reporter] 37 of 412 results did not reach Squally - Vercel's firewall in front of Squally challenged this runner (HTTP 403, x-vercel-mitigated: challenge), and the retries ran out. Your test run and its exit code are not affected. The run page in Squally shows how many are missing.
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
If the run could not be marked complete either, the line says so: the run
|
|
306
|
+
then shows as **incomplete** in Squally once the project's run timeout has
|
|
307
|
+
passed, with the results that did arrive. If the run could not be started at
|
|
308
|
+
all, the line reads `Squally did not start this run - ...` and nothing was
|
|
309
|
+
sent.
|
|
310
|
+
|
|
311
|
+
The exit code of your test run is never changed by any of this.
|
|
312
|
+
|
|
313
|
+
## Steps (0.12.0)
|
|
314
|
+
|
|
315
|
+
For an attempt that **failed, or needed a retry**, the reporter also sends the
|
|
316
|
+
steps Playwright recorded - hooks, fixtures, actions, assertions and your own
|
|
317
|
+
`test.step`s, with their durations and `file:line`. Squally shows them on the
|
|
318
|
+
attempt page, the failing step marked. An attempt that passed on its first try
|
|
319
|
+
sends none.
|
|
320
|
+
|
|
321
|
+
- **Typed values never leave your runner.** The value of a `fill`, `type` or
|
|
322
|
+
`pressSequentially` is replaced: `Fill "•••" getByLabel('Password')`. Every
|
|
323
|
+
title also goes through the same secret masking as error texts.
|
|
324
|
+
- **At most 300 steps per attempt, 200 characters per title.** A longer list
|
|
325
|
+
keeps the failing step and the steps it sits in, then the rest in order, and
|
|
326
|
+
tells Squally how many were left out.
|
|
327
|
+
- Nothing to configure: steps come from Playwright's own report, with or
|
|
328
|
+
without tracing.
|
|
329
|
+
|
|
330
|
+
**Only actual test runs appear.** A run shows up in Squally once your first
|
|
331
|
+
test starts, so commands that never run a test create nothing: `--list`,
|
|
332
|
+
a `globalSetup` or `webServer` that fails to start, a `--grep` that matches
|
|
333
|
+
nothing. A run that *does* execute and legitimately contains no tests — an
|
|
334
|
+
empty shard, `--only-changed` with no changed test file, `--pass-with-no-tests`
|
|
335
|
+
— is still reported, as a run with zero results.
|
|
336
|
+
|
|
337
|
+
**Timeouts and interrupted tests.** A test that runs out of time is reported
|
|
338
|
+
as failed, and Squally is told that it timed out and what its timeout was -
|
|
339
|
+
including a `test.setTimeout()` or `test.slow()` from inside the test. A test
|
|
340
|
+
that Playwright cuts off because the whole run is stopping (`maxFailures`
|
|
341
|
+
reached, Ctrl+C) is reported as skipped: it did not fail, and it does not
|
|
342
|
+
count against the test.
|
|
343
|
+
|
|
344
|
+
## Annotations, the test's line, worker and start (0.19.0)
|
|
345
|
+
|
|
346
|
+
Every attempt also sends:
|
|
347
|
+
|
|
348
|
+
- **Its annotations** - `test.info().annotations`, the `{ annotation }`
|
|
349
|
+
option, and the ones `test.skip()`, `test.fixme()`, `test.slow()` and
|
|
350
|
+
`test.fail()` add: type, description, and where it was added (`file:line`).
|
|
351
|
+
Squally shows them on the run's Tests tab, the attempt page and the test
|
|
352
|
+
page; an `issue` whose description is a link becomes a link there.
|
|
353
|
+
- **Descriptions go through the same secret masking as error texts**, and
|
|
354
|
+
are cut at 1,000 characters.
|
|
355
|
+
- **At most 20 per attempt**, types cut at 64 characters; Squally is told
|
|
356
|
+
how many were left out.
|
|
357
|
+
- On Playwright versions without per-attempt annotations, the test's own
|
|
358
|
+
list is sent with every attempt.
|
|
359
|
+
- **The test's line and column** in its file, for the attempt page's
|
|
360
|
+
*Open in repository* link.
|
|
361
|
+
- **The worker that ran it, its slot and when it started** (`workerIndex`,
|
|
362
|
+
`parallelIndex`, `startTime`). Squally stores them for a later timeline of
|
|
363
|
+
the run; nothing shows them yet.
|
|
364
|
+
|
|
365
|
+
Nothing to configure. An older Squally ignores these fields.
|
|
366
|
+
|
|
367
|
+
## Cancelled runs
|
|
368
|
+
|
|
369
|
+
**Cancelled in Squally.** An admin can cancel a running run from its page in
|
|
370
|
+
Squally. That marks the run there; it does not stop your CI job. What the job
|
|
371
|
+
sends afterwards is discarded, and from 0.12.0 the reporter says so in one
|
|
372
|
+
line - "This run was cancelled in Squally" - instead of advice about job keys.
|
|
373
|
+
|
|
374
|
+
A run stopped with SIGINT - Ctrl+C, or a cancelled CI job - shows in Squally
|
|
375
|
+
as **Cancelled**: the reporter sends what finished and tells Squally the run
|
|
376
|
+
was interrupted. Your exit code stays Playwright's own (130). This needs
|
|
377
|
+
Playwright itself to receive the signal.
|
|
378
|
+
|
|
379
|
+
**On GitHub Actions** a cancelled workflow sends the signal to the step's
|
|
380
|
+
shell, not to Playwright, so the reporter never reports the end. Add one step
|
|
381
|
+
after the tests (0.14.0) that marks the run cancelled in Squally:
|
|
382
|
+
|
|
383
|
+
```yaml
|
|
384
|
+
jobs:
|
|
385
|
+
test:
|
|
386
|
+
runs-on: ubuntu-latest
|
|
387
|
+
env:
|
|
388
|
+
SQUALLY_INGEST_KEY: ${{ secrets.SQUALLY_INGEST_KEY }}
|
|
389
|
+
steps:
|
|
390
|
+
# ...checkout, setup-node, npm ci, playwright install
|
|
391
|
+
- name: Run Playwright tests
|
|
392
|
+
run: npx playwright test
|
|
393
|
+
- name: Mark Squally run cancelled
|
|
394
|
+
if: ${{ cancelled() }}
|
|
395
|
+
run: npx squally-reporter cancel
|
|
396
|
+
```
|
|
397
|
+
|
|
398
|
+
`npx squally-reporter cancel` finds the run by the same CI run id the reporter
|
|
399
|
+
sent from this job, marks every shard of it that is still running as
|
|
400
|
+
cancelled, and prints one line: "Marked run #N as cancelled in Squally
|
|
401
|
+
(X shards)", "Nothing to cancel: no run recorded for <id>" when the job was
|
|
402
|
+
cancelled before its first test, or "Run already finished". It never fails the
|
|
403
|
+
step: a missing key, an unreachable Squally, a 401 or a 5xx is one warning and
|
|
404
|
+
exit code 0. It needs the key in the step's environment - set it at job
|
|
405
|
+
level, as above, or on the step too. A Squally without the cancel endpoint
|
|
406
|
+
answers 404, which is a warning as well.
|
|
407
|
+
|
|
408
|
+
Two things to know:
|
|
409
|
+
|
|
410
|
+
- **Set `SQUALLY_CI_JOB_KEY` at job level**, not only on the test step. The
|
|
411
|
+
cancel step builds the id the same way the reporter did; without the key it
|
|
412
|
+
names another run - or none. `npx squally-reporter cancel --job-key <key>`
|
|
413
|
+
passes it explicitly.
|
|
414
|
+
- **A fail-fast matrix marks the whole group cancelled.** With GitHub's
|
|
415
|
+
default `fail-fast: true`, one failing shard cancels the others, their
|
|
416
|
+
cancel steps run, and the whole run shows as **Cancelled** instead of
|
|
417
|
+
**Failed**. Set `fail-fast: false` on a sharded job so a
|
|
418
|
+
real failure stays a failure.
|
|
419
|
+
|
|
420
|
+
Without the step, a cancelled workflow shows as **incomplete** in Squally.
|
|
421
|
+
|
|
422
|
+
Without that - or after any hard kill (SIGKILL) - the reporter never gets to
|
|
423
|
+
report the end: the run stays *running* until Squally's hourly check marks it
|
|
424
|
+
**incomplete** once the project's run timeout (60 minutes by default) has
|
|
425
|
+
passed. A run stopped before its first test starts sends nothing, like any
|
|
426
|
+
run that never ran a test, and `globalTimeout` running out is not a cancel -
|
|
427
|
+
that run is simply failed.
|
|
428
|
+
|
|
429
|
+
## Which CI, branch and commit a run is reported under
|
|
430
|
+
|
|
431
|
+
Squally counts **CI runs** in its test metrics, so the reporter works out
|
|
432
|
+
which CI system it is running on from that system's own environment:
|
|
433
|
+
|
|
434
|
+
| Provider | Recognised by | Shards grouped by | Branch | Commit |
|
|
435
|
+
|---|---|---|---|---|
|
|
436
|
+
| GitHub Actions | `GITHUB_ACTIONS` | `GITHUB_RUN_ID`-`GITHUB_RUN_ATTEMPT` | `GITHUB_REF_NAME`; pull requests: `GITHUB_HEAD_REF` | `GITHUB_SHA`; pull requests: the PR's head commit |
|
|
437
|
+
| GitLab CI | `GITLAB_CI` | `CI_PIPELINE_ID` | `CI_COMMIT_BRANCH`; merge requests: `CI_MERGE_REQUEST_SOURCE_BRANCH_NAME` | `CI_COMMIT_SHA`; merged-results pipelines: `CI_MERGE_REQUEST_SOURCE_BRANCH_SHA` |
|
|
438
|
+
| CircleCI | `CIRCLECI` | `CIRCLE_WORKFLOW_ID` | `CIRCLE_BRANCH` | `CIRCLE_SHA1` |
|
|
439
|
+
| Jenkins | `BUILD_TAG` (`jenkins-…`) | `BUILD_TAG` | `CHANGE_BRANCH`, else `BRANCH_NAME`, else `GIT_BRANCH` | `GIT_COMMIT` |
|
|
440
|
+
| Azure Pipelines | `TF_BUILD` | `BUILD_BUILDID` | `BUILD_SOURCEBRANCH`; pull requests: `SYSTEM_PULLREQUEST_SOURCEBRANCH` | `BUILD_SOURCEVERSION`; pull requests: `SYSTEM_PULLREQUEST_SOURCECOMMITID` |
|
|
441
|
+
| Bitbucket Pipelines | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_BUILD_NUMBER` | `BITBUCKET_BRANCH` | `BITBUCKET_COMMIT` |
|
|
442
|
+
| Buildkite | `BUILDKITE` | `BUILDKITE_BUILD_ID` | `BUILDKITE_BRANCH` | `BUILDKITE_COMMIT` |
|
|
443
|
+
| TeamCity | `TEAMCITY_VERSION` | - (set `SQUALLY_CI_RUN_ID`) | from git | `BUILD_VCS_NUMBER` |
|
|
444
|
+
| AWS CodeBuild | `CODEBUILD_BUILD_ID` | `CODEBUILD_BUILD_ID` | `CODEBUILD_WEBHOOK_HEAD_REF` | `CODEBUILD_RESOLVED_SOURCE_VERSION` |
|
|
445
|
+
| Travis CI | `TRAVIS` | `TRAVIS_BUILD_ID` | `TRAVIS_BRANCH`; pull requests: `TRAVIS_PULL_REQUEST_BRANCH` | `TRAVIS_COMMIT`; pull requests: `TRAVIS_PULL_REQUEST_SHA` |
|
|
446
|
+
| Drone | `DRONE` | `DRONE_BUILD_NUMBER` | `DRONE_BRANCH`; pull requests: `DRONE_SOURCE_BRANCH` | `DRONE_COMMIT_SHA` |
|
|
447
|
+
| any other CI | `CI=true` or `CI=1` | - (set `SQUALLY_CI_RUN_ID`) | from git | from git |
|
|
448
|
+
|
|
449
|
+
- **`CI` counts only as `true` or `1`.** `CI=false` or `CI=0` is a local run -
|
|
450
|
+
unless one of the providers above is recognised, which always wins. A run
|
|
451
|
+
that is none of these is local. Local runs are listed with all other runs,
|
|
452
|
+
and have numbers of their own: Squally's figures count CI runs by default,
|
|
453
|
+
and the **All · CI · Local** switch shows local runs on their own, or both.
|
|
454
|
+
A local run never posts a GitHub check, a pull-request comment or a Slack
|
|
455
|
+
message.
|
|
456
|
+
- **Pull requests are reported under their own head commit and source
|
|
457
|
+
branch**, not the merge commit CI checks out - that is the commit Squally's
|
|
458
|
+
GitHub check and pull-request comment have to find. The commit message and
|
|
459
|
+
author are that commit's too - never the "Merge … into …" message of the
|
|
460
|
+
checkout.
|
|
461
|
+
- **The head commit is fetched when the checkout lacks it.** With
|
|
462
|
+
`actions/checkout`'s default `fetch-depth: 1` the head commit is not in the
|
|
463
|
+
clone, so the reporter fetches that one commit from `origin`
|
|
464
|
+
(`git fetch --depth=1 --no-tags`), waits at most 3 seconds, and reads its
|
|
465
|
+
message and author. It only does so in a shallow checkout - fetching into a
|
|
466
|
+
full clone would make it shallow for the rest of your job - and never
|
|
467
|
+
changes `HEAD` or any branch. If the fetch fails or takes longer, the
|
|
468
|
+
message and author stay empty and the reporter says so once; the run and
|
|
469
|
+
its exit code are unaffected. Alternatives: `fetch-depth: 2` on the checkout
|
|
470
|
+
step puts the head commit in the clone, so nothing is fetched, and
|
|
471
|
+
`SQUALLY_DISABLE_HEAD_COMMIT_FETCH=true` turns the fetch off.
|
|
472
|
+
- **A tag build has no branch.** A detached checkout never reports `HEAD` as a
|
|
473
|
+
branch name, and `refs/heads/` or `origin/` prefixes are removed.
|
|
474
|
+
- **A link to the CI run (0.11.0).** The reporter sends the page of the CI run,
|
|
475
|
+
so Squally can link a run to it. It is read from each provider's own
|
|
476
|
+
variable, sent only as an `https://` URL of at most 2,048 characters, and
|
|
477
|
+
never opened by Squally - only shown as a link:
|
|
478
|
+
|
|
479
|
+
| Provider | Link |
|
|
480
|
+
|---|---|
|
|
481
|
+
| GitHub Actions | `GITHUB_SERVER_URL`/`GITHUB_REPOSITORY`/actions/runs/`GITHUB_RUN_ID`/attempts/`GITHUB_RUN_ATTEMPT` |
|
|
482
|
+
| GitLab CI | `CI_PIPELINE_URL` |
|
|
483
|
+
| Jenkins | `BUILD_URL` (only with Jenkins served over https) |
|
|
484
|
+
| Buildkite | `BUILDKITE_BUILD_URL` |
|
|
485
|
+
| Travis CI | `TRAVIS_BUILD_WEB_URL` |
|
|
486
|
+
| Drone | `DRONE_BUILD_LINK` |
|
|
487
|
+
|
|
488
|
+
Other providers send no link. It is the run's page, never one job's, and it
|
|
489
|
+
is the same with `SQUALLY_CI_RUN_ID` set.
|
|
490
|
+
|
|
491
|
+
## Tests marked with `test.fail()`
|
|
492
|
+
|
|
493
|
+
Playwright's `test.fail()` marks a test as *expected to fail*: it guards a bug
|
|
494
|
+
you already know about, and Playwright runs the test anyway to check that it
|
|
495
|
+
does still fail. The interesting moment is the opposite one - when someone
|
|
496
|
+
fixes the bug, the test starts passing and Playwright turns it red so you
|
|
497
|
+
remember to delete the marker.
|
|
498
|
+
|
|
499
|
+
The reporter passes that marker on, so Squally reads such a test the same way
|
|
500
|
+
Playwright does:
|
|
501
|
+
|
|
502
|
+
- **Marked and still failing** - shown as passing, with an "expected failure"
|
|
503
|
+
label. It counts as a pass in the test's stability, not against its failure
|
|
504
|
+
rate, and it does not fail Squally's pull-request check.
|
|
505
|
+
- **Marked and now passing** - shown as a failure, because the marker is out of
|
|
506
|
+
date and should be removed.
|
|
507
|
+
- **Marked and timed out** - shown as a failure. Playwright does not count a
|
|
508
|
+
timeout as the expected failure either, so the marker is not passed on for
|
|
509
|
+
it.
|
|
510
|
+
|
|
511
|
+
Both ways of marking a test are recognised: the `test.fail()` modifier and a
|
|
512
|
+
`testInfo.fail()` call inside the test body. `test.skip()` and `test.fixme()`
|
|
513
|
+
are a different statement and are not affected.
|
|
514
|
+
|
|
515
|
+
Nothing is reported for an unmarked test, so an ordinary suite is unchanged.
|
|
516
|
+
Note also that Playwright itself treats a marked, failing test as expected, so
|
|
517
|
+
with `screenshot: "only-on-failure"` it captures nothing for it - there is no
|
|
518
|
+
artifact for the reporter to upload, which is consistent rather than missing.
|
|
519
|
+
|
|
520
|
+
## Sharding
|
|
521
|
+
|
|
522
|
+
**Handled automatically - no extra setup needed.** Each shard is still a
|
|
523
|
+
separate process with its own reporter instance, but Squally now correlates
|
|
524
|
+
them itself: all shards of one CI run are grouped together and shown as one
|
|
525
|
+
run once every shard has finished, with the flip side of that guarantee being
|
|
526
|
+
that the run only completes once every shard *has* reported.
|
|
527
|
+
|
|
528
|
+
Correlation is based on an id read from your CI provider's own environment
|
|
529
|
+
(the "Shards grouped by" column above), so it costs you nothing on every
|
|
530
|
+
provider in that table except TeamCity. On TeamCity, on a CI system not in the
|
|
531
|
+
table, or when you shard locally across several separate `--shard`
|
|
532
|
+
invocations, set `SQUALLY_CI_RUN_ID` to the same value for every shard of one
|
|
533
|
+
run - anything unique per CI run and shared by its shards works, e.g. a build
|
|
534
|
+
number your CI system already exposes.
|
|
535
|
+
|
|
536
|
+
**Several jobs in one CI run need their own key (0.11.0).** The id is shared
|
|
537
|
+
by everything in one CI run, so two jobs that are *not* shards of each other -
|
|
538
|
+
a GitHub Actions matrix over browsers run without `--shard`, or an `api` and a
|
|
539
|
+
`ui` suite - would report into the same Squally run. Give each job its own
|
|
540
|
+
`SQUALLY_CI_JOB_KEY`, shared only by that job's own shards:
|
|
541
|
+
|
|
542
|
+
```yaml
|
|
543
|
+
jobs:
|
|
544
|
+
test:
|
|
545
|
+
strategy:
|
|
546
|
+
matrix:
|
|
547
|
+
browser: [chromium, firefox]
|
|
548
|
+
env:
|
|
549
|
+
SQUALLY_CI_JOB_KEY: ${{ matrix.browser }}
|
|
550
|
+
steps:
|
|
551
|
+
- run: npx playwright test --project=${{ matrix.browser }}
|
|
552
|
+
```
|
|
553
|
+
|
|
554
|
+
The reporter puts the key into the id - `<GITHUB_RUN_ID>-<key>-<GITHUB_RUN_ATTEMPT>`
|
|
555
|
+
on GitHub Actions, `<provider id>-<key>` elsewhere - so each job is its own run
|
|
556
|
+
in Squally, and a re-run of one job links to that job's own earlier attempt.
|
|
557
|
+
The key may hold letters, digits, `.`, `_` and `-`, at most 64 characters; an
|
|
558
|
+
invalid key is reported once and ignored, and your test run is unaffected.
|
|
559
|
+
Without a key the id is exactly what it was before 0.11.0. A suite that is
|
|
560
|
+
sharded *and* split into jobs uses the same key for all shards of one job:
|
|
561
|
+
`SQUALLY_CI_JOB_KEY: ${{ matrix.suite }}` beside `--shard=${{ matrix.shard }}/2`.
|
|
562
|
+
|
|
563
|
+
**Replaces the hand-built id.** Until 0.11.0 this section suggested
|
|
564
|
+
`SQUALLY_CI_RUN_ID: ${{ github.run_id }}-${{ github.run_attempt }}-${{ matrix.browser }}`.
|
|
565
|
+
It still groups each job correctly, but Squally cannot tell its attempts apart,
|
|
566
|
+
so a re-run is never linked to the attempt before it. Switch to
|
|
567
|
+
`SQUALLY_CI_JOB_KEY`. An explicit `SQUALLY_CI_RUN_ID` still wins for the id;
|
|
568
|
+
the key is then only sent alongside it.
|
|
569
|
+
|
|
570
|
+
**When two jobs collide anyway, Squally refuses the second one at the start**
|
|
571
|
+
(squally-app with block 1b, reporter 0.11.0). The reporter prints one block,
|
|
572
|
+
sends nothing more for that job, and leaves your test run and its exit code
|
|
573
|
+
alone:
|
|
574
|
+
|
|
575
|
+
- `ci_run_collision` - another job already reports under this CI run id (and
|
|
576
|
+
shard). Give every job its own `SQUALLY_CI_JOB_KEY`.
|
|
577
|
+
- `ci_run_mixed` - the id would put a sharded job and an unsharded one, or
|
|
578
|
+
shards of different totals, into one run. The same fix.
|
|
579
|
+
|
|
580
|
+
Before 0.11.0 the second job was handed the first job's run instead, and its
|
|
581
|
+
results were refused once that run completed; a reporter older than 0.11.0
|
|
582
|
+
still behaves that way, and prints "Squally has already completed run …".
|
|
583
|
+
|
|
584
|
+
**Re-runs.** On GitHub Actions every re-run attempt is its own run in Squally,
|
|
585
|
+
linked to the attempt before it ("Re-run of #N"). "Re-run failed jobs" runs only
|
|
586
|
+
the shards that did not pass; that attempt waits only for those shards,
|
|
587
|
+
completes once they have reported, and passes or fails on the tests they ran -
|
|
588
|
+
the reused shards are neither waited for nor counted, and the run page shows
|
|
589
|
+
them as "passed in #N · not re-run". A re-run shard that never reports still
|
|
590
|
+
leaves the run incomplete after the run timeout.
|
|
591
|
+
|
|
592
|
+
Other CI systems re-run a job under the id of the run it belongs to - a GitLab
|
|
593
|
+
job retry, an Azure Pipelines re-run, a Bitbucket re-run of failed steps, a
|
|
594
|
+
Buildkite retry, a Travis CI restart. Since 0.11.0 Squally refuses such a
|
|
595
|
+
retry at the start with `ci_run_collision`: the slot is taken by the first
|
|
596
|
+
attempt. **`SQUALLY_CI_JOB_KEY` does not help here** - the retry is the same
|
|
597
|
+
job - so a retry on these systems is not reported; the first attempt stays the
|
|
598
|
+
run. A Drone restart is a new build with a new number, and so a new run in
|
|
599
|
+
Squally, not linked to the one it repeats.
|
|
600
|
+
|
|
601
|
+
**`playwright merge-reports` is not part of this and is not recommended** for
|
|
602
|
+
getting a single run in Squally - the automatic correlation above is the
|
|
603
|
+
supported path, and produces correct per-shard timing/branch/commit
|
|
604
|
+
information that a merge job cannot (see the historical note below). Nothing
|
|
605
|
+
stops you from using `merge-reports` for Playwright's own HTML report if you
|
|
606
|
+
want that separately; just don't point this reporter at the merged result
|
|
607
|
+
expecting Squally to treat it specially.
|
|
608
|
+
|
|
609
|
+
<details>
|
|
610
|
+
<summary>Why not just use `merge-reports`, historically</summary>
|
|
611
|
+
|
|
612
|
+
An earlier version of this reporter recommended `playwright merge-reports` as
|
|
613
|
+
the way to get one Squally run out of a sharded job. That path has a real
|
|
614
|
+
problem: branch, commit, commit message and author are all read on whichever
|
|
615
|
+
machine the reporter actually runs on - during a merge that is the merge job,
|
|
616
|
+
not the shards - and without a checkout there (and without the CI provider's
|
|
617
|
+
own commit variables) they all come out empty. A run with no branch is in
|
|
618
|
+
no branch's numbers and no branch filter finds it, so a "merged" run could
|
|
619
|
+
exist in the dashboard while missing from everything Squally measures per
|
|
620
|
+
branch: suite
|
|
621
|
+
stability, and each test's stability, flaky rate and failure rate on that
|
|
622
|
+
branch. The automatic correlation above reads that information from each
|
|
623
|
+
shard's own environment instead, so this problem does not arise.
|
|
624
|
+
|
|
625
|
+
</details>
|
|
626
|
+
|
|
627
|
+
## The run's Playwright config and repository (0.16.0)
|
|
628
|
+
|
|
629
|
+
At the start of a run the reporter sends what Squally's **Metadata** tab
|
|
630
|
+
shows under *Settings* and *Playwright projects*, read field by field from
|
|
631
|
+
Playwright's config - never the config itself:
|
|
632
|
+
|
|
633
|
+
- **Settings:** `workers` as configured and the workers Playwright actually
|
|
634
|
+
started (newer Playwright versions only; 1.30 does not report it),
|
|
635
|
+
`fullyParallel`, `forbidOnly`, `globalTimeout`, `maxFailures`, and the
|
|
636
|
+
reporters by name - Playwright's own (`html`, `list`, ...) and npm packages
|
|
637
|
+
by package name; a reporter file of your own is sent as `custom`, never its
|
|
638
|
+
path.
|
|
639
|
+
- **Per project:** its name, `browserName`, `channel`, `viewport`, the
|
|
640
|
+
`trace`/`video`/`screenshot` mode, `retries` and `timeout`.
|
|
641
|
+
- **The repository** as host and path, e.g. `github.com/acme/shop`: from your
|
|
642
|
+
CI's variables (`GITHUB_REPOSITORY`, `CI_PROJECT_PATH`,
|
|
643
|
+
`BITBUCKET_REPO_FULL_NAME`, `BUILD_REPOSITORY_URI`, `BUILDKITE_REPO`,
|
|
644
|
+
`CIRCLE_REPOSITORY_URL`, `CODEBUILD_SOURCE_REPO_URL`, `DRONE_GIT_HTTP_URL`,
|
|
645
|
+
`TRAVIS_REPO_SLUG`, `GIT_URL`), else `git remote get-url origin`.
|
|
646
|
+
Credentials, a query and `.git` are removed before it is sent.
|
|
647
|
+
- **Where in the repository the run started** (0.19.0): the directory
|
|
648
|
+
Playwright runs in, relative to the repository's root, as
|
|
649
|
+
`git rev-parse --show-prefix` prints it - `e2e/` in a monorepo, empty at
|
|
650
|
+
the root. Test file paths are relative to that directory, so Squally needs
|
|
651
|
+
it to link a test to its file. Not sent when git cannot say.
|
|
652
|
+
|
|
653
|
+
Two switches:
|
|
654
|
+
|
|
655
|
+
```bash
|
|
656
|
+
# The base URL of each project (use.baseURL) - off unless you turn it on.
|
|
657
|
+
# Only scheme, host and path are sent, never a login, query or fragment.
|
|
658
|
+
SQUALLY_SEND_BASE_URL=true
|
|
659
|
+
|
|
660
|
+
# The repository and where in it the run started - on unless you turn it off.
|
|
661
|
+
SQUALLY_SEND_REPOSITORY=false
|
|
662
|
+
```
|
|
663
|
+
|
|
664
|
+
**Never sent:** the rest of `use` (extra HTTP headers, HTTP credentials,
|
|
665
|
+
`storageState`, proxy, launch and context options), `config.metadata`, the
|
|
666
|
+
command line, the config file and every absolute directory path, `webServer`, `grep`,
|
|
667
|
+
global setup and teardown, and each project's own `metadata`.
|
|
668
|
+
|
|
669
|
+
## Tested revision (optional)
|
|
670
|
+
|
|
671
|
+
The tested revision is the state of your **application** a run tested, as
|
|
672
|
+
opposed to the commit the run is reported under. Squally does not know it by
|
|
673
|
+
itself - it is **opt-in**, with one of these two:
|
|
674
|
+
|
|
675
|
+
```bash
|
|
676
|
+
# The tests live in the same repo as the application: the commit this CI job
|
|
677
|
+
# checked out is the tested revision.
|
|
678
|
+
SAME_REPO_AS_APP=true
|
|
679
|
+
|
|
680
|
+
# The tests live in a separate repo (e.g. a dedicated E2E repo pointed at a
|
|
681
|
+
# deployed app): pass the tested revision itself - a commit SHA, tag, or
|
|
682
|
+
# branch name/ref.
|
|
683
|
+
TESTED_REVISION=v1.2.0
|
|
684
|
+
```
|
|
685
|
+
|
|
686
|
+
- **`SAME_REPO_AS_APP=true`** sends the commit your job checked out: your CI
|
|
687
|
+
provider's own variable for it (`GITHUB_SHA`, `CI_COMMIT_SHA`,
|
|
688
|
+
`CIRCLE_SHA1`, `GIT_COMMIT`, `BUILD_SOURCEVERSION`, `BITBUCKET_COMMIT`,
|
|
689
|
+
`BUILDKITE_COMMIT`, `BUILD_VCS_NUMBER`, `CODEBUILD_RESOLVED_SOURCE_VERSION`,
|
|
690
|
+
`TRAVIS_COMMIT`, `DRONE_COMMIT_SHA`), or `git rev-parse HEAD` where there is
|
|
691
|
+
none - including local runs. Only a commit SHA is sent; if neither yields
|
|
692
|
+
one, the run has no tested revision.
|
|
693
|
+
- **On a pull request the two commits differ, on purpose.** The run is
|
|
694
|
+
reported under the pull request's head commit (see the provider table
|
|
695
|
+
above) - that is what Squally's check and PR comment need. The tested
|
|
696
|
+
revision is the merge commit CI checked out, because that is what the tests
|
|
697
|
+
ran against. On a Bitbucket pull request and a GitHub `pull_request_target`
|
|
698
|
+
run it comes from `git rev-parse HEAD`.
|
|
699
|
+
- **`TESTED_REVISION` wins** when both are set.
|
|
700
|
+
- Neither variable is required - without them, test results are still sent
|
|
701
|
+
and stored exactly as before.
|
|
702
|
+
|
|
703
|
+
**The value is normalised before it is stored**, so that the same revision
|
|
704
|
+
written two ways still compares equal: it is lowercased, and a leading `v` or a
|
|
705
|
+
`refs/heads/` prefix is removed. The example above is exactly such a case -
|
|
706
|
+
`v1.2.0` is stored as `1.2.0`. The reporter says so once per run when it
|
|
707
|
+
happens, so a value that looks different in the dashboard is not a mystery.
|
|
708
|
+
|
|
709
|
+
## Behind a proxy (0.18.0)
|
|
710
|
+
|
|
711
|
+
Every request - to Squally, the artifact uploads to storage, and
|
|
712
|
+
`npx squally-reporter cancel` - goes through the proxy in `HTTPS_PROXY` (for
|
|
713
|
+
`https://` addresses) or `HTTP_PROXY` (for `http://`), lower-case names
|
|
714
|
+
first, the way curl reads them. Hosts in `NO_PROXY` are reached directly:
|
|
715
|
+
comma-separated, each name with all its subdomains (`internal` and
|
|
716
|
+
`.internal` both cover `squally.internal`), `host:port` for one port only, `*`
|
|
717
|
+
for everything - no IP ranges:
|
|
718
|
+
|
|
719
|
+
```bash
|
|
720
|
+
HTTPS_PROXY=http://proxy.internal:3128
|
|
721
|
+
NO_PROXY=localhost,127.0.0.1,.internal
|
|
722
|
+
```
|
|
723
|
+
|
|
724
|
+
The proxy is asked to open a tunnel (`CONNECT`), so TLS still goes end to end
|
|
725
|
+
to Squally. A login in the proxy URL (`http://user:pass@host:port`) is sent as
|
|
726
|
+
`Proxy-Authorization`. A value that is not a URL is one warning, naming the
|
|
727
|
+
variable but not its value, and the reporter connects directly. Without any of
|
|
728
|
+
these variables nothing changes. Until 0.17.0 the variables were ignored.
|
|
729
|
+
|
|
730
|
+
## Troubleshooting
|
|
731
|
+
|
|
732
|
+
Set `SQUALLY_DEBUG=1` to print detail that is otherwise kept quiet - the
|
|
733
|
+
browser-console extraction (how many messages were found, or that an attempt's
|
|
734
|
+
trace held none), since 0.11.0 a CI run link that was not sent because it was
|
|
735
|
+
not an `https://` URL, and since 0.18.0 which proxy variable is in use and a
|
|
736
|
+
quarantine rules request that failed (nothing uses the rules yet, so that is
|
|
737
|
+
no longer a warning). Everything that is *not* ordinary (a trace that cannot
|
|
738
|
+
be read, an unsupported schema version, a rejected upload) is printed without
|
|
739
|
+
it.
|
|
740
|
+
|
|
741
|
+
## Support
|
|
742
|
+
|
|
743
|
+
Found a bug or have a question? Please open an issue on
|
|
744
|
+
[GitHub Issues](https://github.com/squallydev-admin/squally-reporter/issues) -
|
|
745
|
+
that's the primary way to report problems or ask questions. For anything you
|
|
746
|
+
would rather not post publicly, write to
|
|
747
|
+
[support@squally.dev](mailto:support@squally.dev).
|
|
748
|
+
|
|
749
|
+
## License
|
|
750
|
+
|
|
751
|
+
MIT - see [LICENSE](./LICENSE).
|