squally-reporter 0.2.2 → 0.4.0

package/README.md

@@ -1,82 +1,211 @@
-# squally-reporter
-
-Playwright reporter that sends test results to your Squally dashboard.
-
-Learn more at [squally-app.vercel.app](https://squally-app.vercel.app).
-
-## Requirements
-
-- Node.js >= 22.19.0 (required by the `undici` dependency)
-- `@playwright/test` >= 1.30.0
-
-## Install
-
-```bash
-npm install squally-reporter
-```
-
-## Config
-
-Set two env vars (e.g. in CI secrets or `.env`):
-
-```bash
-SQUALLY_API_URL=https://squally-app.vercel.app
-SQUALLY_API_KEY=your-project-api-key
-```
-
-Find your API key on your project's Settings page in the Squally dashboard.
-
-Then add the reporter to `playwright.config.ts`:
-
-```ts
-import { defineConfig } from "@playwright/test";
-
-export default defineConfig({
-  reporter: [
-    ["list"],
-    ["squally-reporter"],
-  ],
-});
-```
-
-Alternatively, pass config explicitly instead of using env vars:
-
-```ts
-reporter: [
-  ["squally-reporter", { apiUrl: "https://squally-app.vercel.app", apiKey: "..." }],
-],
-```
-
-That's it - run your tests as usual (`npx playwright test`) and results will
-be sent to Squally after the run finishes.
-
-## Tested revision (optional)
-
-If your test-automation code lives in the same repo as the application it
-tests, Squally can already tell which commit was tested from the CI
-provider's own commit env vars. If they're **separate repos** (e.g. a
-dedicated E2E-test repo pointed at a deployed app), set one of the following
-so Squally knows which application state was actually tested:
-
-```bash
-# Same repo as the application under test - Squally will trust this CI
-# job's own commit (GITHUB_SHA / CI_COMMIT_SHA) as the tested revision.
-SAME_REPO_AS_APP=true
-
-# Separate repos - pass the tested revision explicitly (a commit SHA, tag,
-# or branch name/ref).
-TESTED_REVISION=v1.2.0
-```
-
-Neither variable is required - without them, test results are still sent
-and stored exactly as before.
-
-## Support
-
-Found a bug or have a question? Please open an issue on
-[GitHub Issues](https://github.com/squallydev-admin/squally-reporter/issues) -
-that's the primary way to report problems or ask questions.
-
-## License
-
-MIT - see [LICENSE](./LICENSE).
+# squally-reporter
+
+Playwright reporter that sends test results to your Squally dashboard.
+
+Learn more at [squally-app.vercel.app](https://squally-app.vercel.app).
+
+## Requirements
+
+- Node.js >= 22.19.0 (required by the `undici` dependency)
+- `@playwright/test` >= 1.30.0
+
+## Install
+
+```bash
+npm install -D squally-reporter
+```
+
+## Config
+
+Set two env vars (e.g. in CI secrets or `.env`):
+
+```bash
+SQUALLY_API_URL=https://squally-app.vercel.app
+SQUALLY_API_KEY=your-project-api-key
+```
+
+Find your API key on your project's Settings page in the Squally dashboard.
+
+Then add the reporter to `playwright.config.ts`:
+
+```ts
+import { defineConfig } from "@playwright/test";
+
+export default defineConfig({
+  reporter: [
+    ["list"],
+    ["squally-reporter"],
+  ],
+});
+```
+
+Alternatively, pass config explicitly instead of using env vars:
+
+```ts
+reporter: [
+  ["squally-reporter", { apiUrl: "https://squally-app.vercel.app", apiKey: "..." }],
+],
+```
+
+That's it - run your tests as usual (`npx playwright test`) and results will
+be sent to Squally after the run finishes.
+
+## Screenshot/trace upload
+
+If your `playwright.config.ts` captures screenshots or traces, the reporter
+uploads them to Squally so they show up next to the failing test in the
+dashboard - no separate setup beyond the two env vars above.
+
+**Only from attempts worth looking at:** screenshots/traces are uploaded
+only for attempts that **failed**, or that **needed a retry** (even if that
+retry then passed) - never for an attempt that passed on its first try,
+regardless of how `screenshot`/`trace` are configured. Nobody looks at a
+screenshot of a test that just passed, and uploading one anyway would only
+eat into your organization's storage for no benefit.
+
+**You need to actually enable capture in `playwright.config.ts`** - without
+it, Playwright doesn't produce any attachments, so there's nothing for the
+reporter to find or upload (no error either way, it's a silent no-op):
+
+```ts
+export default defineConfig({
+  use: {
+    screenshot: "only-on-failure",
+    trace: "retain-on-failure",
+  },
+  // ...
+});
+```
+
+**Limits:**
+- 50 MB per individual file. Larger files are skipped (logged as a
+  `console.warn`, the test result itself is still reported as usual).
+- A total storage cap per organization, depending on your Squally plan.
+  Once reached, further uploads are skipped the same way until you either
+  free up space or upgrade.
+- Two minutes per upload, and 30 seconds for the calls that surround it. An
+  unreachable or very slow server means a skipped upload and a `console.warn`,
+  not a CI job waiting indefinitely.
+
+Upload is best-effort and fully non-blocking, same principle as the rest of
+this reporter - a failed or skipped upload never fails your test run or
+changes its exit code, it only logs a `console.warn` explaining what
+happened.
+
+## Limits per run
+
+Squally accepts at most **5,000 test results in a single run**. Beyond that the
+whole report is rejected, so the reporter checks before sending and skips the
+upload with a warning rather than letting the server discard it after the fact.
+Nothing else about your test run is affected - this only means that one run
+does not appear in the dashboard.
+
+**Retries count separately.** Each attempt is its own result, so a suite of
+2,000 tests having a bad day reaches the ceiling sooner than the test count
+suggests.
+
+If you run into this, the practical way around it today is to let separate
+parts of the suite report separately - for example as separate Playwright
+projects or separate CI jobs, each producing its own run. Splitting one report
+across several requests is not supported yet.
+
+There is also a hard **~4.5 MB limit on the request itself**, coming from the
+hosting platform rather than from Squally. Very large error messages, stack
+traces or console output can hit it before the 5,000 mark; the reporter already
+truncates each field to what the server accepts, which keeps normal runs well
+clear of it.
+
+## Sharding
+
+**Playwright's `--shard` is not merged automatically.** Each shard is a
+separate process with its own reporter instance, so each one reports its own
+run: a job split four ways shows up in Squally as four runs from the same
+commit, each holding only its own slice of the suite. The reporter says so in
+the log when it sees `--shard`.
+
+What that does and does not affect:
+
+- **Per-test flakiness stays correct.** A test lives in exactly one shard, so
+  it still contributes one data point per CI run, and its history, flip rate
+  and retry rate are unaffected.
+- **Run-level numbers are off.** The run count, "runs per day", and each run's
+  own pass/fail (it only covers its slice) all reflect shards rather than CI
+  runs.
+
+### Reporting sharded runs as one run
+
+Use Playwright's own blob reports: have each shard write a blob, then merge
+them in one final job and let this reporter run on the merged result. That
+produces exactly one run with every test in it.
+
+```yaml
+test:
+  strategy:
+    matrix:
+      shard: [1, 2, 3, 4]
+  steps:
+    - uses: actions/checkout@v4
+    - run: npx playwright test --shard=${{ matrix.shard }}/4 --reporter=blob
+    - uses: actions/upload-artifact@v4
+      with:
+        name: blob-${{ matrix.shard }}
+        path: blob-report
+
+merge:
+  needs: test
+  steps:
+    # Required - see the note below.
+    - uses: actions/checkout@v4
+    - uses: actions/download-artifact@v4
+      with:
+        pattern: blob-*
+        merge-multiple: true
+        path: all-blobs
+    - run: npx playwright merge-reports --reporter=squally-reporter ./all-blobs
+      env:
+        SQUALLY_API_URL: ${{ secrets.SQUALLY_API_URL }}
+        SQUALLY_API_KEY: ${{ secrets.SQUALLY_API_KEY }}
+```
+
+**The merge job needs the repository checked out.** Branch, commit, commit
+message and author are read on whichever machine the reporter runs on - during
+a merge that is the merge job, not the shards. Without a checkout (and without
+the CI provider's own commit variables) they all come out empty, and a run
+without a branch is never counted as a run on your stable branch, which is what
+Squally's broken/healthy verdicts are based on. `actions/checkout` in the merge
+job is enough.
+
+The run's start time is taken from the earliest test, not from when the
+reporter started, so a merge that happens minutes after the tests still dates
+the run correctly.
+
+## Tested revision (optional)
+
+If your test-automation code lives in the same repo as the application it
+tests, Squally can already tell which commit was tested from the CI
+provider's own commit env vars. If they're **separate repos** (e.g. a
+dedicated E2E-test repo pointed at a deployed app), set one of the following
+so Squally knows which application state was actually tested:
+
+```bash
+# Same repo as the application under test - Squally will trust this CI
+# job's own commit (GITHUB_SHA / CI_COMMIT_SHA) as the tested revision.
+SAME_REPO_AS_APP=true
+
+# Separate repos - pass the tested revision explicitly (a commit SHA, tag,
+# or branch name/ref).
+TESTED_REVISION=v1.2.0
+```
+
+Neither variable is required - without them, test results are still sent
+and stored exactly as before.
+
+## Support
+
+Found a bug or have a question? Please open an issue on
+[GitHub Issues](https://github.com/squallydev-admin/squally-reporter/issues) -
+that's the primary way to report problems or ask questions.
+
+## License
+
+MIT - see [LICENSE](./LICENSE).

package/dist/index.d.ts

@@ -9,9 +9,14 @@ export default class SquallyReporter implements Reporter {
     private enabled;
     private startedAt;
     private results;
+    private artifacts;
+    private earliestTestStart;
     constructor(options?: SquallyReporterOptions);
-    onBegin(_config: FullConfig, _suite: Suite): void;
+    onBegin(config: FullConfig, _suite: Suite): void;
     onTestEnd(test: TestCase, result: TestResult): void;
     onEnd(result: FullResult): Promise<void>;
+    private runStartedAt;
+    private uploadArtifacts;
+    private presignAndUpload;
 }
 export {};

package/dist/index.js

@@ -4,8 +4,113 @@ var __importDefault = (this && this.__importDefault) || function (mod) {
 };
 Object.defineProperty(exports, "__esModule", { value: true });
 const node_child_process_1 = require("node:child_process");
+const node_fs_1 = require("node:fs");
+const promises_1 = require("node:fs/promises");
 const node_path_1 = __importDefault(require("node:path"));
 const undici_1 = require("undici");
+// Mirrors squally-app's ingest contract (app/api/ingest/route.ts,
+// FIELD_LIMITS / checkStringField). Kept as a copy on purpose - the reporter
+// ships independently of the app and cannot import from it - which means this
+// table has to be updated when that one changes. It is deliberately not
+// "roughly right": the server rejects the ENTIRE payload on the FIRST field
+// that is too long, so one over-long value used to cost the whole run, every
+// test result in it, with a bare "400 Bad Request" as the only clue.
+//
+// ciProvider is absent because the server does not length-check it, and this
+// reporter only ever sends one of four fixed strings for it.
+const FIELD_LIMITS = {
+    branch: 250,
+    commitSha: 100,
+    commitMessage: 500,
+    authorName: 100,
+    testedRevision: 200,
+    eventType: 100,
+    testName: 500,
+    filePath: 500,
+    errorMessage: 5000,
+    errorStack: 20000,
+    consoleOutput: 20000,
+};
+// ASCII only, like everything else in this file, and short: it has to fit
+// INSIDE the limit it is protecting, and authorName only has 100 characters
+// to give away.
+const TRUNCATION_MARKER = "... [truncated]";
+// Squally rejects a payload with more than this many entries outright - the
+// whole run, not the surplus. Mirrored here so the reporter can say so up
+// front instead of the customer finding out from a 400 after the suite has
+// already finished.
+//
+// Deliberately NOT enforced by trimming: unlike an over-long error message,
+// the surplus here is test results, and dropping them would turn a loud
+// failure into a run that quietly looks like those tests never existed. The
+// reporter reports what happened or says why it could not - it does not
+// invent a smaller truth.
+//
+// Retries count as separate entries, so a suite of 2 000 tests reaches this
+// faster than the number suggests.
+const MAX_TEST_RESULTS = 5000;
+// Visible rather than silent: a value that was cut says so, so nobody debugs
+// a "wrong" commit message that is merely shortened. For testName/filePath
+// this also alters Squally's test identity - unavoidable, since the
+// alternative is losing the run outright, and two names sharing their first
+// 500 characters would collide either way.
+function truncate(value, max) {
+    if (value.length <= max)
+        return value;
+    if (max <= TRUNCATION_MARKER.length)
+        return value.slice(0, max);
+    return value.slice(0, max - TRUNCATION_MARKER.length) + TRUNCATION_MARKER;
+}
+function truncateOrNull(value, max) {
+    return value === null ? null : truncate(value, max);
+}
+// How much of a server error body to put in the log. Enough for the server's
+// own one-line explanation, bounded so that an unexpected HTML error page
+// from a proxy cannot dump kilobytes into the customer's CI output.
+// undici leaves headersTimeout and bodyTimeout at 300 000 ms each when they
+// are not set, and nothing here used to set them. A hung server therefore cost
+// 5 minutes for the run POST, 5 for presign, 5 for confirm and 5 per upload
+// round - with a hundred artifacts at five in parallel, well over an hour of a
+// CI job sitting there with no output explaining what it is waiting for. That
+// is worse for a customer than an error: an error at least says something.
+//
+// Two budgets, because the two kinds of request are not comparable. The JSON
+// calls exchange kilobytes and have no business taking longer than half a
+// minute. An upload can be a 50 MB trace over a CI runner's modest uplink,
+// where two minutes is tight-ish but still far short of "hung".
+const API_REQUEST_TIMEOUT_MS = 30000;
+const UPLOAD_REQUEST_TIMEOUT_MS = 120000;
+const ERROR_BODY_LOG_LIMIT = 500;
+// Turns a failed response into something a customer can act on. Today the
+// caller only sees "400 Bad Request"; the reason ("commitMessage exceeds max
+// length of 500 characters") sits in the body and was being thrown away.
+//
+// Nothing from the REQUEST is logged, only the server's own response - the
+// Authorization header must never reach the CI log.
+// A timed-out request throws like any other network failure, and is handled
+// like one - warn, carry on, never touch the exit code. It only gets its own
+// wording because "fetch failed" would leave the customer guessing whether the
+// server said no or simply never answered.
+function describeRequestError(err, timeoutMs) {
+    const name = err instanceof Error ? err.name : "";
+    if (name === "TimeoutError" || name === "AbortError") {
+        return `no response within ${timeoutMs} ms - giving up`;
+    }
+    return String(err);
+}
+async function describeResponseError(response) {
+    let detail = "";
+    try {
+        const body = (await response.text()).trim();
+        if (body)
+            detail = ` - ${truncate(body, ERROR_BODY_LOG_LIMIT)}`;
+    }
+    catch {
+        // Reading the body is best-effort: a body that cannot be read must not
+        // turn a logged warning into a thrown error.
+    }
+    return `${response.status} ${response.statusText}${detail}`;
+}
 // Maps Playwright's more granular statuses down to the three
 // statuses Squally stores (passed | failed | skipped).
 // "timedOut" and "interrupted" both count as failures for reporting purposes.
@@ -143,6 +248,29 @@ function detectEventType() {
 function toPosixPath(p) {
     return p.split(node_path_1.default.sep).join("/");
 }
+// Playwright own assertion errors (expect(...).toHaveText(...) etc.) are
+// colorized with ANSI escape codes in error.message/error.stack - useful in
+// a terminal, but rendered as visible garbage once stored as plain text and
+// shown in Squally UI. Test code that colors its own console output (e.g.
+// via chalk/picocolors) can do the same to stdout/stderr. Stripped here,
+// before anything is ever sent - not server/UI-side - so the raw escape
+// codes never land in the database in the first place, and every later
+// consumer (the existing keyword-based error categorization, the planned
+// AI analysis, a fingerprint cache over normalized error signatures)
+// does not have to strip them again itself.
+//
+// Matches the CSI form ANSI color/style codes actually take: an ESC
+// character (code point 27), then an opening bracket, then any run of
+// digits/semicolons, then one letter (e.g. ESC, bracket, 3, 1, m for red).
+// No new dependency (e.g. strip-ansi) for a one-line regex, same reasoning
+// as toPosixPath/the upload worker pool above. Built via String.fromCharCode
+// rather than a hex escape directly in the source: this file has to stay
+// plain, readable ASCII - a raw ESC byte pasted into source code is
+// invisible in most editors/diffs and easy to mistake for empty content.
+function stripAnsi(value) {
+    const ansiPattern = new RegExp(String.fromCharCode(27) + String.fromCharCode(92) + "[[0-9;]*[a-zA-Z]", "g");
+    return value.replace(ansiPattern, "");
+}
 // Playwright captures stdout/stderr as two separate arrays (each internally
 // chronological, but not merged with each other), so a true byte-accurate
 // interleaving between the two streams isn't available here - stdout lines
@@ -152,13 +280,99 @@ function toPosixPath(p) {
 function formatConsoleOutput(result) {
     const lines = [];
     for (const chunk of result.stdout) {
-        lines.push(`[log] ${chunk.toString().replace(/\s+$/, "")}`);
+        lines.push(`[log] ${stripAnsi(chunk.toString()).replace(/\s+$/, "")}`);
     }
     for (const chunk of result.stderr) {
-        lines.push(`[error] ${chunk.toString().replace(/\s+$/, "")}`);
+        lines.push(`[error] ${stripAnsi(chunk.toString()).replace(/\s+$/, "")}`);
     }
     return lines.length > 0 ? lines.join("\n") : null;
 }
+// Presigned upload URLs are valid for 15 minutes (server-side, see
+// squally-app's lib/r2.ts). Uploading fully sequentially could get close to
+// that window for a suite with a very large number of failing/retried
+// tests, but firing everything at once isn't right either - CI runners
+// often have modest outbound bandwidth/file-descriptor limits, and a
+// thundering herd of large trace uploads competes with the test run itself
+// for those resources. 5 concurrent uploads is a pragmatic middle ground:
+// fast enough that even a few hundred artifacts finish in well under 15
+// minutes, bounded enough not to overwhelm a typical CI runner.
+const UPLOAD_CONCURRENCY = 5;
+// Playwright does not let the process end on its own: the runner calls
+// gracefullyCloseAll().then(() => process.exit(code)) (playwright-core's
+// gracefullyProcessExitDoNotHang). process.exit() tears the event loop down
+// immediately, and on Windows that aborts the whole process with
+//   Assertion failed: !(handle->flags & UV_HANDLE_CLOSING), src/win/async.c
+// if any of our own I/O is still winding down at that moment - the exit code
+// becomes 127 instead of 0, i.e. a green test run reports as failed.
+//
+// Measured, so the number is not a guess: with the run POST alone the process
+// exits cleanly; from the second request on (presign) it aborts reproducibly.
+// A 0 ms or 10 ms wait does not help, 100 ms survives two requests, and 250 ms
+// survives the full path (ingest -> presign -> upload -> confirm) over
+// repeated runs.
+//
+// This is a mitigation, not a cure: it widens the window rather than removing
+// the race, because nothing in Node's public API reports "all my handles are
+// really gone". Do not shrink it without re-running the Windows job in CI -
+// that job exists specifically to catch this (see .github/workflows/ci.yml).
+// The same crash was fixed once before for the single-request path and came
+// back when 0.3.0 added the artifact requests.
+const EXIT_DRAIN_MS = 250;
+// No new dependency for this - a fixed-size worker pool that keeps pulling
+// the next item until the list is exhausted is ~10 lines, not worth pulling
+// in e.g. p-limit for.
+async function runWithConcurrencyLimit(items, limit, worker) {
+    let nextIndex = 0;
+    async function runNext() {
+        const currentIndex = nextIndex++;
+        if (currentIndex >= items.length)
+            return;
+        await worker(items[currentIndex]);
+        return runNext();
+    }
+    const workerCount = Math.min(limit, items.length);
+    await Promise.all(Array.from({ length: workerCount }, () => runNext()));
+}
+// Only screenshots/traces from attempts worth looking at get uploaded -
+// deliberately independent of how the customer configured
+// screenshot/trace in playwright.config.ts. A "screenshot: 'on'" config
+// would otherwise attach an image to every single passing test too: nobody
+// opens a screenshot of a test that just passed, and every one of those
+// uploads eats into the organization's artifact cap (Free tier: 1 GB) for
+// zero benefit. "failed" already covers Playwright's timedOut/interrupted
+// (mapStatus above collapses those down) - retryCount > 0 additionally
+// covers a passed-on-retry attempt, since seeing what an earlier, failing
+// attempt looked like is still useful even if the test eventually passed.
+function attemptQualifiesForArtifacts(mappedStatus, retryCount) {
+    return mappedStatus === "failed" || retryCount > 0;
+}
+// A control character as separator, not an empty string or a slash -
+// testName/filePath can contain arbitrary characters, and this key only
+// needs to be unique, not human-readable (it is purely an internal lookup
+// back to the local CollectedArtifact once the server round-trips these
+// identity fields).
+function artifactKey(testName, filePath, retryCount, type) {
+    return [testName, filePath, String(retryCount), type].join(String.fromCharCode(1));
+}
+function skipReasonMessage(reason) {
+    if (reason === "file_too_large")
+        return "file_too_large (limit 50 MB)";
+    if (reason === "cap_exceeded")
+        return "cap_exceeded (storage limit reached, upgrade your plan)";
+    if (reason === "not_found")
+        return "not_found (test result not found on server)";
+    return reason;
+}
+function confirmReasonMessage(reason) {
+    if (reason === "not_uploaded")
+        return "not_uploaded (upload did not reach storage)";
+    if (reason === "not_found")
+        return "not_found (test result not found on server)";
+    return reason;
+}
+function artifactLabel(type) {
+    return type === "screenshot" ? "Screenshot" : "Trace";
+}
 class SquallyReporter {
     constructor(options = {}) {
         this.apiUrl = "";
@@ -169,6 +383,9 @@ class SquallyReporter {
         this.enabled = false;
         this.startedAt = new Date();
         this.results = [];
+        this.artifacts = [];
+        // Earliest TestResult.startTime seen, see the startedAt note in onEnd().
+        this.earliestTestStart = null;
         const apiUrl = options.apiUrl ?? process.env.SQUALLY_API_URL;
         const apiKey = options.apiKey ?? process.env.SQUALLY_API_KEY;
         if (!apiUrl || !apiKey) {
@@ -185,25 +402,88 @@ class SquallyReporter {
         this.apiKey = apiKey;
         this.enabled = true;
     }
-    onBegin(_config, _suite) {
+    onBegin(config, _suite) {
         if (!this.enabled)
             return;
         this.startedAt = new Date();
+        // Each shard is its own process with its own reporter instance, so each
+        // one reports a run of its own. Measured: two shards over two spec files
+        // produce two runs holding two results each; a shard that receives no
+        // tests produces none at all.
+        //
+        // Worth saying out loud what this does and does not break: per-test
+        // history is unaffected, because a test lives in exactly one shard and so
+        // still contributes one data point per CI run. What gets distorted is the
+        // run level - the number of runs, each run's pass/fail (it only covers
+        // its own slice), and anything counted per run in the dashboard.
+        if (config.shard) {
+            console.warn(`[squally-reporter] Sharding detected (--shard=${config.shard.current}/${config.shard.total}). ` +
+                `Each shard reports separately, so this CI run will show up in Squally as up to ` +
+                `${config.shard.total} runs, each holding only its own slice of the suite. ` +
+                "Per-test flakiness stays correct; run counts and per-run pass/fail do not. " +
+                "For a single run, write a blob report per shard (--reporter=blob) and merge " +
+                "afterwards: npx playwright merge-reports --reporter=squally-reporter ./blob-report");
+        }
     }
     onTestEnd(test, result) {
         if (!this.enabled)
             return;
         const error = result.errors[0];
+        // Truncated HERE, once, not when the payload is built: testName/filePath
+        // are Squally's test identity, and /api/artifacts/presign resolves each
+        // artifact back to a TestResult through exactly these two values plus
+        // retryCount. Cutting them in two places, or cutting them only for the
+        // ingest payload, would make every artifact of a long-named test
+        // unmatchable server-side.
+        const testName = truncate(test.titlePath().slice(1).join(" > "), FIELD_LIMITS.testName);
+        const filePath = truncate(toPosixPath(node_path_1.default.relative(process.cwd(), test.location.file)), FIELD_LIMITS.filePath);
+        const status = mapStatus(result.status);
+        if (this.earliestTestStart === null || result.startTime < this.earliestTestStart) {
+            this.earliestTestStart = result.startTime;
+        }
         this.results.push({
-            testName: test.titlePath().slice(1).join(" > "),
-            filePath: toPosixPath(node_path_1.default.relative(process.cwd(), test.location.file)),
-            status: mapStatus(result.status),
+            testName,
+            filePath,
+            status,
             durationMs: result.duration,
             retryCount: result.retry,
-            errorMessage: error?.message ?? null,
-            errorStack: error?.stack ?? null,
-            consoleOutput: formatConsoleOutput(result),
+            errorMessage: truncateOrNull(error?.message ? stripAnsi(error.message) : null, FIELD_LIMITS.errorMessage),
+            errorStack: truncateOrNull(error?.stack ? stripAnsi(error.stack) : null, FIELD_LIMITS.errorStack),
+            consoleOutput: truncateOrNull(formatConsoleOutput(result), FIELD_LIMITS.consoleOutput),
         });
+        if (!attemptQualifiesForArtifacts(status, result.retry))
+            return;
+        for (const attachment of result.attachments) {
+            if (attachment.name !== "screenshot" && attachment.name !== "trace")
+                continue;
+            // Attachments can be in-memory (body, no path) when added manually via
+            // testInfo.attach() - only file-backed ones (screenshot/trace always
+            // are, when Playwright itself captures them) can be uploaded here.
+            if (!attachment.path)
+                continue;
+            // Synchronous, not fs/promises - onTestEnd's return value isn't
+            // awaited by Playwright (Reporter['onTestEnd'] is typed as void, not
+            // Promise<void>), so an async stat call here would race with
+            // onEnd() reading this.artifacts before it resolved.
+            let sizeBytes;
+            try {
+                sizeBytes = (0, node_fs_1.statSync)(attachment.path).size;
+            }
+            catch {
+                // File already gone (e.g. cleaned up by a parallel worker) -
+                // nothing to upload, silently skip rather than warn for what's an
+                // expected, harmless race in some CI setups.
+                continue;
+            }
+            this.artifacts.push({
+                testName,
+                filePath,
+                retryCount: result.retry,
+                type: attachment.name,
+                localPath: attachment.path,
+                sizeBytes,
+            });
+        }
     }
     async onEnd(result) {
         if (!this.enabled)
@@ -215,53 +495,243 @@ class SquallyReporter {
         // sending anything.
         if (this.results.length === 0)
             return;
+        // Checked before the request, not after: the server would answer 400 and
+        // discard the entire run, and the customer would be left correlating a
+        // status code with a suite that finished minutes ago.
+        if (this.results.length > MAX_TEST_RESULTS) {
+            console.warn(`[squally-reporter] This run has ${this.results.length} test results, and Squally ` +
+                `accepts at most ${MAX_TEST_RESULTS} per run - nothing was sent. Note that retries ` +
+                "count separately. Split the suite across several Playwright projects or " +
+                "configurations so each reports its own run, or get in touch if you need a " +
+                "single run this size.");
+            return;
+        }
         const resolvedRevision = resolveTestedRevision();
+        // commitMessage is the one that bites in practice: `git log -1 %B` is the
+        // full body, and a squash merge carrying a PR description is routinely
+        // past 500 characters. Before this, that single value discarded every test
+        // result in the run.
         const payload = {
-            branch: detectBranch(),
-            commitSha: detectCommitSha(),
-            commitMessage: detectCommitMessage(),
-            authorName: detectAuthorName(),
+            branch: truncateOrNull(detectBranch(), FIELD_LIMITS.branch),
+            commitSha: truncateOrNull(detectCommitSha(), FIELD_LIMITS.commitSha),
+            commitMessage: truncateOrNull(detectCommitMessage(), FIELD_LIMITS.commitMessage),
+            authorName: truncateOrNull(detectAuthorName(), FIELD_LIMITS.authorName),
             ciProvider: detectCiProvider(),
             status: result.status === "passed" ? "passed" : "failed",
-            startedAt: this.startedAt.toISOString(),
+            startedAt: this.runStartedAt().toISOString(),
             finishedAt: new Date().toISOString(),
             testResults: this.results,
-            testedRevision: resolvedRevision?.revision ?? null,
+            testedRevision: truncateOrNull(resolvedRevision?.revision ?? null, FIELD_LIMITS.testedRevision),
             revisionSource: resolvedRevision?.source ?? null,
-            eventType: detectEventType(),
+            eventType: truncateOrNull(detectEventType(), FIELD_LIMITS.eventType),
         };
-        // Node's global fetch() pools connections through a shared, long-lived
-        // keep-alive dispatcher by default. In a short-lived CLI process like
-        // this reporter, that pooled socket can still be mid-teardown when
-        // Playwright exits the process, which crashes with a libuv assertion on
-        // Windows ("UV_HANDLE_CLOSING", src/win/async.c). Using our own Agent
-        // for this one request and closing it explicitly guarantees no open
-        // handle is left behind by the time onEnd() resolves. Importing fetch
-        // from undici too (not Node's global one) avoids a possible instanceof
-        // mismatch between Node's internal bundled undici and this installed
-        // copy when passing our Agent as the dispatcher.
-        const dispatcher = new undici_1.Agent({ keepAliveTimeout: 1, keepAliveMaxTimeout: 1 });
+        // Our own Agent rather than the global one: Node's global fetch() pools
+        // connections through a shared, long-lived keep-alive dispatcher, and a
+        // pooled socket outliving this run is exactly what we do not want in a
+        // short-lived CLI process. Importing fetch from undici too (not Node's
+        // global one) avoids a possible instanceof mismatch between Node's
+        // internal bundled undici and this installed copy when passing our Agent
+        // as the dispatcher.
+        //
+        // NOTE: this does NOT on its own prevent the Windows exit crash - that
+        // claim used to stand here and was wrong. Closing the dispatcher resolves
+        // before the sockets are actually gone; see EXIT_DRAIN_MS above. Measured
+        // with global fetch as well: dropping undici entirely does not help
+        // either, so the client is not what makes the difference.
+        // The per-request AbortSignals below are the real bound; these are a
+        // backstop so that nothing can outlive the longest budget even if a signal
+        // never fires.
+        const dispatcher = new undici_1.Agent({
+            keepAliveTimeout: 1,
+            keepAliveMaxTimeout: 1,
+            headersTimeout: UPLOAD_REQUEST_TIMEOUT_MS,
+            bodyTimeout: UPLOAD_REQUEST_TIMEOUT_MS,
+        });
         try {
-            const response = await (0, undici_1.fetch)(`${this.apiUrl}/api/ingest`, {
+            let runId = null;
+            try {
+                const response = await (0, undici_1.fetch)(`${this.apiUrl}/api/ingest`, {
+                    method: "POST",
+                    headers: {
+                        "Content-Type": "application/json",
+                        Authorization: `Bearer ${this.apiKey}`,
+                    },
+                    body: JSON.stringify(payload),
+                    dispatcher,
+                    signal: AbortSignal.timeout(API_REQUEST_TIMEOUT_MS),
+                });
+                if (!response.ok) {
+                    // Never throw here - a failing reporter upload should not fail the
+                    // whole CI run just because the dashboard was unreachable.
+                    console.error(`[squally-reporter] Ingest failed: ${await describeResponseError(response)}`);
+                }
+                else {
+                    const body = (await response.json());
+                    runId = body.runId ?? null;
+                }
+            }
+            catch (err) {
+                console.error("[squally-reporter] Failed to reach Squally API:", describeRequestError(err, API_REQUEST_TIMEOUT_MS));
+            }
+            // Artifact upload is a separate, best-effort step layered on top of
+            // ingest, never a reason to fail it: it needs a runId (presign looks
+            // TestResults up by runId + identity fields, which only exist once
+            // ingest has actually inserted them), and there's nothing to do if
+            // this run had no failing/retried attempts with a screenshot/trace
+            // attached (attemptQualifiesForArtifacts already filtered that in
+            // onTestEnd) - no reason to make an empty presign call.
+            if (runId && this.artifacts.length > 0) {
+                await this.uploadArtifacts(runId, dispatcher);
+            }
+        }
+        finally {
+            await dispatcher.close();
+            // See EXIT_DRAIN_MS - closing the dispatcher resolves before the
+            // underlying sockets are fully gone, and Playwright exits the process
+            // the moment onEnd() resolves.
+            await new Promise((resolve) => setTimeout(resolve, EXIT_DRAIN_MS));
+        }
+    }
+    // onBegin's timestamp is normally the right answer: it fires before the
+    // first test and therefore includes global setup, webServer startup and so
+    // on. It is wrong in exactly one case - `npx playwright merge-reports`,
+    // where onBegin fires on the merging machine, minutes or hours after the
+    // tests actually ran, and would date the whole run to the merge.
+    //
+    // The earlier of the two is right in both situations: during a normal run
+    // onBegin precedes every test, during a merge it follows all of them. No
+    // detection of which mode we are in is needed.
+    runStartedAt() {
+        if (this.earliestTestStart === null)
+            return this.startedAt;
+        return this.earliestTestStart < this.startedAt ? this.earliestTestStart : this.startedAt;
+    }
+    // Top-level safety net - presignAndUpload already handles presign/
+    // upload/confirm failures individually (see Aufgabe 5's requirement that
+    // none of this ever affects the test suite's exit code), but this catch
+    // still exists in case something in there throws in a way that wasn't
+    // anticipated. The whole artifact feature must be additive: worst case,
+    // some screenshots/traces don't make it to Squally, but the customer's
+    // CI run itself is never at risk.
+    async uploadArtifacts(runId, dispatcher) {
+        try {
+            await this.presignAndUpload(runId, dispatcher);
+        }
+        catch (err) {
+            console.warn("[squally-reporter] Artifact upload failed unexpectedly:", err);
+        }
+    }
+    async presignAndUpload(runId, dispatcher) {
+        const presignArtifacts = this.artifacts.map((artifact) => ({
+            testName: artifact.testName,
+            filePath: artifact.filePath,
+            retryCount: artifact.retryCount,
+            type: artifact.type,
+            sizeBytes: artifact.sizeBytes,
+        }));
+        let presignResults;
+        try {
+            const response = await (0, undici_1.fetch)(`${this.apiUrl}/api/artifacts/presign`, {
                 method: "POST",
                 headers: {
                     "Content-Type": "application/json",
                     Authorization: `Bearer ${this.apiKey}`,
                 },
-                body: JSON.stringify(payload),
+                body: JSON.stringify({ runId, artifacts: presignArtifacts }),
                 dispatcher,
+                signal: AbortSignal.timeout(API_REQUEST_TIMEOUT_MS),
             });
             if (!response.ok) {
-                // Never throw here - a failing reporter upload should not fail the
-                // whole CI run just because the dashboard was unreachable.
-                console.error(`[squally-reporter] Ingest failed: ${response.status} ${response.statusText}`);
+                console.warn(`[squally-reporter] Artifact presign failed: ${await describeResponseError(response)}` +
+                    " - skipping artifact upload for this run");
+                return;
             }
+            const body = (await response.json());
+            presignResults = body.results;
         }
         catch (err) {
-            console.error("[squally-reporter] Failed to reach Squally API:", err);
+            console.warn("[squally-reporter] Artifact presign request failed:", describeRequestError(err, API_REQUEST_TIMEOUT_MS));
+            return;
         }
-        finally {
-            await dispatcher.close();
+        // Maps a presign result back to the local file it came from -
+        // testName/filePath/retryCount/type is exactly what was sent, so it's
+        // guaranteed to match one of this.artifacts (barring a server bug),
+        // but a missing lookup is handled defensively below instead of assumed.
+        const artifactsByKey = new Map(this.artifacts.map((artifact) => [
+            artifactKey(artifact.testName, artifact.filePath, artifact.retryCount, artifact.type),
+            artifact,
+        ]));
+        const uploads = [];
+        for (const item of presignResults) {
+            const local = artifactsByKey.get(artifactKey(item.testName, item.filePath ?? "", item.retryCount, item.type));
+            if (!local)
+                continue;
+            if (item.skip) {
+                console.warn(`[squally-reporter] ${artifactLabel(item.type)} for '${item.testName}' skipped: ${skipReasonMessage(item.reason)}`);
+                continue;
+            }
+            uploads.push({ artifact: local, uploadUrl: item.uploadUrl });
+        }
+        if (uploads.length === 0)
+            return;
+        // See UPLOAD_CONCURRENCY's own comment above for why this is bounded
+        // rather than either fully sequential or fully parallel.
+        await runWithConcurrencyLimit(uploads, UPLOAD_CONCURRENCY, async ({ artifact, uploadUrl }) => {
+            try {
+                const fileBody = await (0, promises_1.readFile)(artifact.localPath);
+                const response = await (0, undici_1.fetch)(uploadUrl, {
+                    method: "PUT",
+                    body: fileBody,
+                    dispatcher,
+                    signal: AbortSignal.timeout(UPLOAD_REQUEST_TIMEOUT_MS),
+                });
+                if (!response.ok) {
+                    console.warn(`[squally-reporter] Upload failed for ${artifact.type} of '${artifact.testName}': ` +
+                        (await describeResponseError(response)));
+                }
+            }
+            catch (err) {
+                console.warn(`[squally-reporter] Upload failed for ${artifact.type} of '${artifact.testName}':`, describeRequestError(err, UPLOAD_REQUEST_TIMEOUT_MS));
+            }
+        });
+        // Confirm every artifact that was actually presigned (not the ones
+        // presign already skipped - those never got a key, and re-checking
+        // them here would just produce a confusing second "not_uploaded"
+        // warning for something that was never attempted in the first place).
+        // Confirm's own headObject check is the source of truth for whether an
+        // upload really landed - deliberately not tracking PUT success/failure
+        // above to decide what to confirm, since a PUT that reported success
+        // but didn't truly land (or vice versa) should still be caught here.
+        const confirmArtifacts = uploads.map(({ artifact }) => ({
+            testName: artifact.testName,
+            filePath: artifact.filePath,
+            retryCount: artifact.retryCount,
+            type: artifact.type,
+        }));
+        try {
+            const response = await (0, undici_1.fetch)(`${this.apiUrl}/api/artifacts/confirm`, {
+                method: "POST",
+                headers: {
+                    "Content-Type": "application/json",
+                    Authorization: `Bearer ${this.apiKey}`,
+                },
+                body: JSON.stringify({ runId, artifacts: confirmArtifacts }),
+                dispatcher,
+                signal: AbortSignal.timeout(API_REQUEST_TIMEOUT_MS),
+            });
+            if (!response.ok) {
+                console.warn(`[squally-reporter] Artifact confirm failed: ${await describeResponseError(response)}`);
+                return;
+            }
+            const body = (await response.json());
+            for (const item of body.results) {
+                if (item.confirmed)
+                    continue;
+                console.warn(`[squally-reporter] ${artifactLabel(item.type)} for '${item.testName}' not confirmed: ${confirmReasonMessage(item.reason)}`);
+            }
+        }
+        catch (err) {
+            console.warn("[squally-reporter] Artifact confirm request failed:", describeRequestError(err, API_REQUEST_TIMEOUT_MS));
         }
     }
 }

package/package.json

@@ -1,38 +1,40 @@
-{
-  "name": "squally-reporter",
-  "version": "0.2.2",
-  "description": "Playwright reporter that sends test results to your Squally dashboard.",
-  "author": "Squally <squally.dev@gmail.com>",
-  "keywords": [
-    "playwright",
-    "playwright-reporter",
-    "flaky-tests",
-    "test-results",
-    "ci"
-  ],
-  "main": "dist/index.js",
-  "types": "dist/index.d.ts",
-  "files": [
-    "dist"
-  ],
-  "engines": {
-    "node": ">=22.19.0"
-  },
-  "scripts": {
-    "build": "tsc",
-    "prepublishOnly": "npm run build",
-    "prepare": "npm run build"
-  },
-  "peerDependencies": {
-    "@playwright/test": ">=1.30.0"
-  },
-  "devDependencies": {
-    "@playwright/test": "^1.48.0",
-    "@types/node": "^20.14.0",
-    "typescript": "^5.6.0"
-  },
-  "license": "MIT",
-  "dependencies": {
-    "undici": "^8.10.0"
-  }
-}
+{
+  "name": "squally-reporter",
+  "version": "0.4.0",
+  "description": "Playwright reporter that sends test results to your Squally dashboard.",
+  "author": "Squally <squally.dev@gmail.com>",
+  "keywords": [
+    "playwright",
+    "playwright-reporter",
+    "flaky-tests",
+    "test-results",
+    "ci"
+  ],
+  "main": "dist/index.js",
+  "types": "dist/index.d.ts",
+  "files": [
+    "dist"
+  ],
+  "engines": {
+    "node": ">=22.19.0"
+  },
+  "scripts": {
+    "build": "tsc",
+    "prepublishOnly": "npm run build",
+    "prepare": "npm run build",
+    "test:exit-code": "node test/exit-code/run.mjs",
+    "test": "npm run build && npm run test:exit-code"
+  },
+  "peerDependencies": {
+    "@playwright/test": ">=1.30.0"
+  },
+  "devDependencies": {
+    "@playwright/test": "^1.48.0",
+    "@types/node": "^20.14.0",
+    "typescript": "^5.6.0"
+  },
+  "license": "MIT",
+  "dependencies": {
+    "undici": "^8.10.0"
+  }
+}