@piwitests/reporter 0.25.0 → 0.26.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +7 -7
- package/dist/cli/index.js +430 -8
- package/dist/global-setup-module.js +4 -1
- package/dist/index.d.ts +51 -2
- package/dist/index.js +242 -28
- package/package.json +2 -2
- package/templates/skills/investigate-failure/SKILL.md +1 -1
- package/templates/skills/run-the-right-tests/SKILL.md +38 -0
- package/templates/skills/setup-piwi/SKILL.md +1 -1
package/README.md
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
1
|
# Piwi Dashboard Reporter
|
|
2
2
|
|
|
3
|
-
A custom Playwright reporter that sends test results to a [Piwi Dashboard](https://piwitests.
|
|
3
|
+
A custom Playwright reporter that sends test results to a [Piwi Dashboard](https://piwitests.dev) server. It handles uploading test results, HTML reports, trace files, and performance metrics — with optional live streaming of results as tests execute.
|
|
4
4
|
|
|
5
|
-
📖 **[Full documentation](https://piwitests.
|
|
5
|
+
📖 **[Full documentation](https://piwitests.dev/reporter)**
|
|
6
6
|
|
|
7
7
|
## Installation
|
|
8
8
|
|
|
@@ -18,14 +18,14 @@ From your Playwright project, `npx @piwitests/reporter init` installs the report
|
|
|
18
18
|
npx @piwitests/reporter init --server-url http://localhost:3000 --project my-project
|
|
19
19
|
```
|
|
20
20
|
|
|
21
|
-
Every step is idempotent (safe to re-run); a config shape it will not rewrite is reported as `manual` with the exact change to make, never mangled. Add `--dry-run` to preview or `--json` for a machine-readable plan an agent can act on. It also installs the [Piwi agent skills](https://piwitests.
|
|
21
|
+
Every step is idempotent (safe to re-run); a config shape it will not rewrite is reported as `manual` with the exact change to make, never mangled. Add `--dry-run` to preview or `--json` for a machine-readable plan an agent can act on. It also installs the [Piwi agent skills](https://piwitests.dev/mcp#agent-skills) so your coding agent can investigate failures, heal locators, and stabilize flaky tests. Run `npx @piwitests/reporter init --help` for all options, or wire it up by hand with the steps below.
|
|
22
22
|
|
|
23
23
|
> The package is `@piwitests/reporter`; its command is `piwi`. Invoke it through the package name (`npx @piwitests/reporter <command>`) so npx resolves this package — `npx piwi` would fetch an unrelated `piwi` from npm. Once the reporter is a project dependency, `npx piwi <command>` resolves the local binary and works too.
|
|
24
24
|
|
|
25
25
|
## Quick start
|
|
26
26
|
|
|
27
27
|
`wrapConfig` is the recommended setup. It injects the reporter **and** a global
|
|
28
|
-
setup step (so the run shows up as "
|
|
28
|
+
setup step (so the run shows up as "initializing" while your `globalSetup` runs),
|
|
29
29
|
and forwards your options to that setup:
|
|
30
30
|
|
|
31
31
|
```typescript
|
|
@@ -184,7 +184,7 @@ Then import `test` from your fixtures file in every spec — a spec that imports
|
|
|
184
184
|
|
|
185
185
|
Capture works for the `page` fixture, `browser.newPage()`, `browser.newContext().newPage()`, and popups. Everything is only collected when `collectPerformanceMetrics` is `true` (the default); locator snapshots can be disabled separately with `captureLocators: false`.
|
|
186
186
|
|
|
187
|
-
Without the fixtures you still get full run history, statuses, errors, traces, reports, streaming, and clustering — the fixtures add the slow-endpoint, Web Vitals, console, ARIA, and locator-healing layers. See the [capture fixtures guide](https://piwitests.
|
|
187
|
+
Without the fixtures you still get full run history, statuses, errors, traces, reports, streaming, and clustering — the fixtures add the slow-endpoint, Web Vitals, console, ARIA, and locator-healing layers. See the [capture fixtures guide](https://piwitests.dev/capture-fixtures) for the full feature matrix and composition patterns.
|
|
188
188
|
|
|
189
189
|
## AI steps
|
|
190
190
|
|
|
@@ -204,7 +204,7 @@ export const test = extendPiwiAi(extendPiwiFixtures(base))
|
|
|
204
204
|
|
|
205
205
|
Author missing entries once in `resolve` mode (`PIWI_AI=resolve`, pointed at a dashboard with an AI provider configured), commit the artifacts, and CI replays them offline. `{param}` placeholders are type-checked and masked out of everything sent to the model. Manage the committed entries with `piwi ai check | resolve | prune`.
|
|
206
206
|
|
|
207
|
-
See the [AI steps guide](https://piwitests.
|
|
207
|
+
See the [AI steps guide](https://piwitests.dev/ai-steps) for the authoring/replay lifecycle, the safety model (allowlisted, drift-guarded, postcondition-verified), and the full option/env-var reference.
|
|
208
208
|
|
|
209
209
|
## Authentication
|
|
210
210
|
|
|
@@ -298,7 +298,7 @@ addition:
|
|
|
298
298
|
|
|
299
299
|
## Requirements
|
|
300
300
|
|
|
301
|
-
- Node.js 18 or higher (the reporter runs inside your test project — the dashboard *server* itself targets Node
|
|
301
|
+
- Node.js 18 or higher (the reporter runs inside your test project — the dashboard *server* itself targets Node 22+, or use its Docker image)
|
|
302
302
|
- Playwright Test 1.40 or higher
|
|
303
303
|
- Running Piwi Dashboard server
|
|
304
304
|
|
package/dist/cli/index.js
CHANGED
|
@@ -414,7 +414,7 @@ function formatGateResult(result) {
|
|
|
414
414
|
const { facts } = result;
|
|
415
415
|
const lines = [
|
|
416
416
|
result.passed ? `\u2714 Piwi gate passed \u2014 ${facts.projectName} run #${facts.runId}` : `\u2716 Piwi gate failed \u2014 ${facts.projectName} run #${facts.runId}`,
|
|
417
|
-
` ${facts.totalTests} tests, ${facts.failedTests} failed, ${facts.newRegressions} new, ${facts.newFlaky} newly flaky`
|
|
417
|
+
` ${facts.totalTests} tests, ${facts.failedTests} failed, ${facts.newRegressions} new, ${facts.newFlaky} newly flaky, ${facts.flakyTests} flaky`
|
|
418
418
|
];
|
|
419
419
|
if (facts.quarantinedFailures > 0) {
|
|
420
420
|
lines.push(
|
|
@@ -453,6 +453,8 @@ Policy (at least one is required):
|
|
|
453
453
|
--max-new-flaky <n> Fail when more than n tests newly became flaky
|
|
454
454
|
--max-quarantined <n> Fail when more than n tests are quarantined
|
|
455
455
|
--fail-on-new-cluster Fail when this run introduced a new failure cluster
|
|
456
|
+
--fail-on-flaky Fail when this run contains any flaky test
|
|
457
|
+
--require-selection <key> Fail when a test the named selection matches did not run or failed
|
|
456
458
|
|
|
457
459
|
Other:
|
|
458
460
|
--json Print the raw result as JSON instead of a summary
|
|
@@ -475,9 +477,9 @@ function readCount(argv, name) {
|
|
|
475
477
|
if (!Number.isFinite(n) || n < 0) throw new Error(`${name} expects a non-negative number, got "${raw}"`);
|
|
476
478
|
return Math.floor(n);
|
|
477
479
|
}
|
|
478
|
-
function readRunIdFromFile(
|
|
480
|
+
function readRunIdFromFile(path10) {
|
|
479
481
|
try {
|
|
480
|
-
const parsed = JSON.parse(fs3.readFileSync(
|
|
482
|
+
const parsed = JSON.parse(fs3.readFileSync(path10, "utf-8"));
|
|
481
483
|
const runId = Number(parsed.runId);
|
|
482
484
|
return Number.isFinite(runId) && runId > 0 ? runId : null;
|
|
483
485
|
} catch {
|
|
@@ -504,7 +506,9 @@ function parseGateArgs(argv, env) {
|
|
|
504
506
|
maxNewRegressions: readCount(argv, "--max-new-regressions"),
|
|
505
507
|
maxNewFlaky: readCount(argv, "--max-new-flaky"),
|
|
506
508
|
maxQuarantined: readCount(argv, "--max-quarantined"),
|
|
507
|
-
failOnNewCluster: argv.includes("--fail-on-new-cluster")
|
|
509
|
+
failOnNewCluster: argv.includes("--fail-on-new-cluster"),
|
|
510
|
+
failOnFlaky: argv.includes("--fail-on-flaky"),
|
|
511
|
+
requireSelection: readOption2(argv, "--require-selection")
|
|
508
512
|
};
|
|
509
513
|
return { serverUrl, apiKey: readOption2(argv, "--api-key") ?? env.PIWI_API_KEY ?? null, runId, policy };
|
|
510
514
|
}
|
|
@@ -770,7 +774,12 @@ function ensureGitignoreEntry(existing, entry = ".env") {
|
|
|
770
774
|
var fs5 = __toESM(require("fs"));
|
|
771
775
|
var path7 = __toESM(require("path"));
|
|
772
776
|
var SETUP_SKILL = "setup-piwi";
|
|
773
|
-
var WORKFLOW_SKILLS = [
|
|
777
|
+
var WORKFLOW_SKILLS = [
|
|
778
|
+
"investigate-failure",
|
|
779
|
+
"apply-locator-healing",
|
|
780
|
+
"stabilize-flaky-tests",
|
|
781
|
+
"run-the-right-tests"
|
|
782
|
+
];
|
|
774
783
|
var ALL_SKILLS = [SETUP_SKILL, ...WORKFLOW_SKILLS];
|
|
775
784
|
var DEFAULT_SKILLS_DIR = path7.join(".claude", "skills");
|
|
776
785
|
function findTemplatesDir(fromDir) {
|
|
@@ -1186,8 +1195,415 @@ Piwi setup${opts.dryRun ? " (dry run \u2014 nothing written)" : ""} for "${opts.
|
|
|
1186
1195
|
return steps.some((s) => s.status === "error") ? 1 : 0;
|
|
1187
1196
|
}
|
|
1188
1197
|
|
|
1189
|
-
// src/cli/
|
|
1198
|
+
// src/cli/select.ts
|
|
1199
|
+
var fs7 = __toESM(require("fs"));
|
|
1200
|
+
var path9 = __toESM(require("path"));
|
|
1201
|
+
var import_node_child_process3 = require("child_process");
|
|
1202
|
+
var import_node_module = require("module");
|
|
1203
|
+
|
|
1204
|
+
// src/internal/support/selection-client.ts
|
|
1205
|
+
function authHeaders(apiKey) {
|
|
1206
|
+
return apiKey ? { "X-API-Key": apiKey } : {};
|
|
1207
|
+
}
|
|
1208
|
+
async function resolveProjectId(options) {
|
|
1209
|
+
if (!options.project) throw new Error("No project \u2014 pass --project or set PIWI_PROJECT_NAME");
|
|
1210
|
+
if (/^\d+$/.test(options.project)) return Number(options.project);
|
|
1211
|
+
const res = await fetch(`${options.serverUrl}/api/projects/menu`, { headers: authHeaders(options.apiKey) });
|
|
1212
|
+
if (!res.ok) throw new Error(`Could not list projects (dashboard returned ${res.status})`);
|
|
1213
|
+
const body = await res.json();
|
|
1214
|
+
const match = body.items?.find((p) => p.name.toLowerCase() === options.project.toLowerCase());
|
|
1215
|
+
if (!match) throw new Error(`No project named "${options.project}" on this dashboard`);
|
|
1216
|
+
return match.id;
|
|
1217
|
+
}
|
|
1218
|
+
async function fetchResolution(options, projectId) {
|
|
1219
|
+
const params = new URLSearchParams({ format: options.format ?? "args" });
|
|
1220
|
+
if (options.pkgRunner) params.set("pkgRunner", options.pkgRunner);
|
|
1221
|
+
if (options.budgetMs != null) params.set("budgetMs", String(options.budgetMs));
|
|
1222
|
+
if (options.shard) params.set("shard", options.shard);
|
|
1223
|
+
if (options.order) params.set("order", options.order);
|
|
1224
|
+
const url = `${options.serverUrl}/api/projects/${projectId}/selections/${encodeURIComponent(options.key)}/resolve?${params}`;
|
|
1225
|
+
const res = await fetch(url, { headers: authHeaders(options.apiKey) });
|
|
1226
|
+
if (!res.ok) {
|
|
1227
|
+
const body = await res.json().catch(() => ({}));
|
|
1228
|
+
throw new Error(body.message || `Dashboard returned ${res.status} resolving "${options.key}"`);
|
|
1229
|
+
}
|
|
1230
|
+
return await res.json();
|
|
1231
|
+
}
|
|
1232
|
+
async function fetchImpact(options, projectId, changedFiles) {
|
|
1233
|
+
const url = `${options.serverUrl}/api/projects/${projectId}/selections/impact`;
|
|
1234
|
+
const res = await fetch(url, {
|
|
1235
|
+
method: "POST",
|
|
1236
|
+
headers: { "Content-Type": "application/json", ...authHeaders(options.apiKey) },
|
|
1237
|
+
body: JSON.stringify({
|
|
1238
|
+
changedFiles,
|
|
1239
|
+
format: options.format ?? "args",
|
|
1240
|
+
shard: options.shard ?? void 0,
|
|
1241
|
+
order: options.order ?? void 0
|
|
1242
|
+
})
|
|
1243
|
+
});
|
|
1244
|
+
if (!res.ok) {
|
|
1245
|
+
const body = await res.json().catch(() => ({}));
|
|
1246
|
+
throw new Error(body.message || `Dashboard returned ${res.status} resolving impact`);
|
|
1247
|
+
}
|
|
1248
|
+
return await res.json();
|
|
1249
|
+
}
|
|
1250
|
+
|
|
1251
|
+
// src/cli/select.ts
|
|
1252
|
+
var EXIT_OK3 = 0;
|
|
1253
|
+
var EXIT_ERROR3 = 2;
|
|
1190
1254
|
var USAGE5 = `
|
|
1255
|
+
piwi select / piwi run \u2014 run a saved selection of tests
|
|
1256
|
+
|
|
1257
|
+
Usage:
|
|
1258
|
+
npx @piwitests/reporter select <key> [options] print the Playwright args
|
|
1259
|
+
npx @piwitests/reporter run <key> [options] [-- <playwright args>]
|
|
1260
|
+
npx @piwitests/reporter run impact --base <ref> run the tests your diff impacts
|
|
1261
|
+
|
|
1262
|
+
Connection:
|
|
1263
|
+
--server-url <url> Dashboard URL (env PIWI_DASHBOARD_URL)
|
|
1264
|
+
--api-key <key> API key (env PIWI_API_KEY)
|
|
1265
|
+
--project <name|id> Project (env PIWI_PROJECT_NAME)
|
|
1266
|
+
|
|
1267
|
+
Selection:
|
|
1268
|
+
--format <fmt> args (file:line, default) | grep | files | json
|
|
1269
|
+
--budget <duration> Cap total time, e.g. 5m, 90s, 300000 (ms)
|
|
1270
|
+
--shard <i/n> Keep only shard i of n, balanced by test duration
|
|
1271
|
+
--fail-fast Order the least-reliable tests first (fail-fast)
|
|
1272
|
+
--base <ref> For "impact": the ref to diff the working tree against
|
|
1273
|
+
|
|
1274
|
+
Behavior:
|
|
1275
|
+
--strict Fail (exit 2) instead of falling back when unreachable
|
|
1276
|
+
--pkg-runner <cmd> Package runner for the printed command (default npx)
|
|
1277
|
+
--json Print the full resolution as JSON (select only)
|
|
1278
|
+
-h, --help Show this help
|
|
1279
|
+
|
|
1280
|
+
Exit codes: 0 ok, 1 the test run failed (run only), 2 could not resolve.
|
|
1281
|
+
`.trim();
|
|
1282
|
+
var IMPACT_KEY = "impact";
|
|
1283
|
+
var VALUE_FLAGS = /* @__PURE__ */ new Set([
|
|
1284
|
+
"--server-url",
|
|
1285
|
+
"--api-key",
|
|
1286
|
+
"--project",
|
|
1287
|
+
"--format",
|
|
1288
|
+
"--budget",
|
|
1289
|
+
"--shard",
|
|
1290
|
+
"--base",
|
|
1291
|
+
"--pkg-runner"
|
|
1292
|
+
]);
|
|
1293
|
+
function readOption5(argv, name) {
|
|
1294
|
+
const withEquals = argv.find((arg) => arg.startsWith(`${name}=`));
|
|
1295
|
+
if (withEquals) return withEquals.slice(name.length + 1);
|
|
1296
|
+
const index = argv.indexOf(name);
|
|
1297
|
+
if (index === -1) return void 0;
|
|
1298
|
+
const value = argv[index + 1];
|
|
1299
|
+
return value && !value.startsWith("-") ? value : void 0;
|
|
1300
|
+
}
|
|
1301
|
+
function findKey(argv) {
|
|
1302
|
+
for (let i = 0; i < argv.length; i++) {
|
|
1303
|
+
const tok = argv[i];
|
|
1304
|
+
if (tok.startsWith("-")) {
|
|
1305
|
+
if (!tok.includes("=") && VALUE_FLAGS.has(tok)) i++;
|
|
1306
|
+
continue;
|
|
1307
|
+
}
|
|
1308
|
+
return tok;
|
|
1309
|
+
}
|
|
1310
|
+
return void 0;
|
|
1311
|
+
}
|
|
1312
|
+
function parseDuration(raw) {
|
|
1313
|
+
const match = raw.trim().match(/^(\d+(?:\.\d+)?)(ms|s|m|h)?$/);
|
|
1314
|
+
if (!match) return null;
|
|
1315
|
+
const value = Number(match[1]);
|
|
1316
|
+
const unit = match[2];
|
|
1317
|
+
const factor = unit === "h" ? 36e5 : unit === "m" ? 6e4 : unit === "s" ? 1e3 : 1;
|
|
1318
|
+
const ms = Math.round(value * factor);
|
|
1319
|
+
return ms > 0 ? ms : null;
|
|
1320
|
+
}
|
|
1321
|
+
function parseSelectArgs(argv, env) {
|
|
1322
|
+
const dashIndex = argv.indexOf("--");
|
|
1323
|
+
const own = dashIndex === -1 ? argv : argv.slice(0, dashIndex);
|
|
1324
|
+
const extra = dashIndex === -1 ? [] : argv.slice(dashIndex + 1);
|
|
1325
|
+
const serverUrl = (readOption5(own, "--server-url") ?? env.PIWI_DASHBOARD_URL ?? "").replace(/\/$/, "");
|
|
1326
|
+
if (!serverUrl) throw new Error("No dashboard URL \u2014 pass --server-url or set PIWI_DASHBOARD_URL");
|
|
1327
|
+
const key = findKey(own);
|
|
1328
|
+
if (!key) throw new Error("No selection key \u2014 usage: piwi select <key>");
|
|
1329
|
+
const budgetRaw = readOption5(own, "--budget");
|
|
1330
|
+
let budgetMs = null;
|
|
1331
|
+
if (budgetRaw !== void 0) {
|
|
1332
|
+
budgetMs = parseDuration(budgetRaw);
|
|
1333
|
+
if (budgetMs === null) throw new Error(`--budget expects a duration like 5m or 90s, got "${budgetRaw}"`);
|
|
1334
|
+
}
|
|
1335
|
+
const shard = readOption5(own, "--shard") ?? null;
|
|
1336
|
+
if (shard !== null && !/^\d+\s*\/\s*\d+$/.test(shard)) {
|
|
1337
|
+
throw new Error(`--shard expects an "i/n" spec like 2/4, got "${shard}"`);
|
|
1338
|
+
}
|
|
1339
|
+
return {
|
|
1340
|
+
serverUrl,
|
|
1341
|
+
apiKey: readOption5(own, "--api-key") ?? env.PIWI_API_KEY ?? null,
|
|
1342
|
+
project: readOption5(own, "--project") ?? env.PIWI_PROJECT_NAME ?? "",
|
|
1343
|
+
key,
|
|
1344
|
+
format: readOption5(own, "--format") ?? "args",
|
|
1345
|
+
budgetMs,
|
|
1346
|
+
shard,
|
|
1347
|
+
order: own.includes("--fail-fast") ? "failureLikelihood" : null,
|
|
1348
|
+
base: readOption5(own, "--base") ?? null,
|
|
1349
|
+
strict: own.includes("--strict"),
|
|
1350
|
+
pkgRunner: readOption5(own, "--pkg-runner") ?? "npx",
|
|
1351
|
+
json: own.includes("--json"),
|
|
1352
|
+
extra
|
|
1353
|
+
};
|
|
1354
|
+
}
|
|
1355
|
+
var CACHE_FILE = path9.join(".piwi", "selection-cache.json");
|
|
1356
|
+
function cacheKey(projectId, args) {
|
|
1357
|
+
return `${projectId}:${args.key}:${args.format}:${args.budgetMs ?? 0}:${args.shard ?? ""}:${args.order ?? ""}`;
|
|
1358
|
+
}
|
|
1359
|
+
function readCache(projectId, args) {
|
|
1360
|
+
try {
|
|
1361
|
+
const store = JSON.parse(fs7.readFileSync(CACHE_FILE, "utf-8"));
|
|
1362
|
+
return store[cacheKey(projectId, args)] ?? null;
|
|
1363
|
+
} catch {
|
|
1364
|
+
return null;
|
|
1365
|
+
}
|
|
1366
|
+
}
|
|
1367
|
+
function writeCache(projectId, args, resolution) {
|
|
1368
|
+
try {
|
|
1369
|
+
let store = {};
|
|
1370
|
+
try {
|
|
1371
|
+
store = JSON.parse(fs7.readFileSync(CACHE_FILE, "utf-8"));
|
|
1372
|
+
} catch {
|
|
1373
|
+
}
|
|
1374
|
+
store[cacheKey(projectId, args)] = resolution;
|
|
1375
|
+
fs7.mkdirSync(path9.dirname(CACHE_FILE), { recursive: true });
|
|
1376
|
+
fs7.writeFileSync(CACHE_FILE, JSON.stringify(store, null, 2));
|
|
1377
|
+
} catch {
|
|
1378
|
+
}
|
|
1379
|
+
}
|
|
1380
|
+
async function resolveWithCache(args, projectId) {
|
|
1381
|
+
try {
|
|
1382
|
+
const resolution = await fetchResolution(args, projectId);
|
|
1383
|
+
writeCache(projectId, args, resolution);
|
|
1384
|
+
return { resolution, fromCache: false };
|
|
1385
|
+
} catch (e) {
|
|
1386
|
+
if (args.strict) throw e;
|
|
1387
|
+
const cached = readCache(projectId, args);
|
|
1388
|
+
if (cached) {
|
|
1389
|
+
console.error(`piwi: dashboard unreachable, using cached resolution \u2014 ${e.message}`);
|
|
1390
|
+
return { resolution: cached, fromCache: true };
|
|
1391
|
+
}
|
|
1392
|
+
return null;
|
|
1393
|
+
}
|
|
1394
|
+
}
|
|
1395
|
+
function printWarnings(resolution) {
|
|
1396
|
+
for (const w of resolution.warnings) console.error(`piwi: warning [${w.code}] ${w.message}`);
|
|
1397
|
+
}
|
|
1398
|
+
function gitChangedFiles(base) {
|
|
1399
|
+
const out = (0, import_node_child_process3.execFileSync)("git", ["diff", "--name-only", base], {
|
|
1400
|
+
encoding: "utf-8",
|
|
1401
|
+
stdio: ["ignore", "pipe", "ignore"]
|
|
1402
|
+
});
|
|
1403
|
+
return [
|
|
1404
|
+
...new Set(
|
|
1405
|
+
out.split("\n").map((line) => line.trim()).filter(Boolean)
|
|
1406
|
+
)
|
|
1407
|
+
];
|
|
1408
|
+
}
|
|
1409
|
+
async function loadImpact(args, projectId) {
|
|
1410
|
+
const files = gitChangedFiles(args.base);
|
|
1411
|
+
return fetchImpact(args, projectId, files);
|
|
1412
|
+
}
|
|
1413
|
+
async function runSelectImpact(args) {
|
|
1414
|
+
let impact;
|
|
1415
|
+
try {
|
|
1416
|
+
const projectId = await resolveProjectId(args);
|
|
1417
|
+
impact = await loadImpact(args, projectId);
|
|
1418
|
+
} catch (e) {
|
|
1419
|
+
console.error(`piwi select: ${e.message}`);
|
|
1420
|
+
return EXIT_ERROR3;
|
|
1421
|
+
}
|
|
1422
|
+
printWarnings(impact);
|
|
1423
|
+
console.error(
|
|
1424
|
+
`piwi select: ${impact.impact.changedFiles} changed file(s) \u2192 ${impact.estimate.count} impacted test(s)${impact.impact.widened ? " (widened to full suite)" : ""}`
|
|
1425
|
+
);
|
|
1426
|
+
if (args.json) {
|
|
1427
|
+
console.log(JSON.stringify(impact, null, 2));
|
|
1428
|
+
return EXIT_OK3;
|
|
1429
|
+
}
|
|
1430
|
+
console.log(impact.impact.widened ? "" : impact.materialization.args.join(" "));
|
|
1431
|
+
return EXIT_OK3;
|
|
1432
|
+
}
|
|
1433
|
+
async function runSelect(argv, env = process.env) {
|
|
1434
|
+
if (argv.includes("-h") || argv.includes("--help")) {
|
|
1435
|
+
console.log(USAGE5);
|
|
1436
|
+
return EXIT_OK3;
|
|
1437
|
+
}
|
|
1438
|
+
let args;
|
|
1439
|
+
try {
|
|
1440
|
+
args = parseSelectArgs(argv, env);
|
|
1441
|
+
} catch (e) {
|
|
1442
|
+
console.error(`piwi select: ${e.message}
|
|
1443
|
+
`);
|
|
1444
|
+
console.error(USAGE5);
|
|
1445
|
+
return EXIT_ERROR3;
|
|
1446
|
+
}
|
|
1447
|
+
if (args.key === IMPACT_KEY) {
|
|
1448
|
+
if (!args.base) {
|
|
1449
|
+
console.error("piwi select: impact needs a base ref \u2014 pass --base <ref> (e.g. --base origin/main)");
|
|
1450
|
+
return EXIT_ERROR3;
|
|
1451
|
+
}
|
|
1452
|
+
return runSelectImpact(args);
|
|
1453
|
+
}
|
|
1454
|
+
let resolution;
|
|
1455
|
+
try {
|
|
1456
|
+
const projectId = await resolveProjectId(args);
|
|
1457
|
+
resolution = await fetchResolution(args, projectId);
|
|
1458
|
+
} catch (e) {
|
|
1459
|
+
console.error(`piwi select: ${e.message}`);
|
|
1460
|
+
return EXIT_ERROR3;
|
|
1461
|
+
}
|
|
1462
|
+
printWarnings(resolution);
|
|
1463
|
+
if (resolution.estimate.count === 0) {
|
|
1464
|
+
console.error(`piwi select: "${args.key}" resolved to 0 tests`);
|
|
1465
|
+
return EXIT_ERROR3;
|
|
1466
|
+
}
|
|
1467
|
+
if (args.json) console.log(JSON.stringify(resolution, null, 2));
|
|
1468
|
+
else console.log(resolution.materialization.args.join(" "));
|
|
1469
|
+
return EXIT_OK3;
|
|
1470
|
+
}
|
|
1471
|
+
function resolvePlaywrightCli() {
|
|
1472
|
+
const require2 = (0, import_node_module.createRequire)(path9.join(process.cwd(), "noop.js"));
|
|
1473
|
+
for (const id of ["playwright/cli", "@playwright/test/cli", "playwright/lib/cli/cli"]) {
|
|
1474
|
+
try {
|
|
1475
|
+
return require2.resolve(id);
|
|
1476
|
+
} catch {
|
|
1477
|
+
}
|
|
1478
|
+
}
|
|
1479
|
+
return null;
|
|
1480
|
+
}
|
|
1481
|
+
function spawnPlaywright(pkgRunner, playwrightArgs, env) {
|
|
1482
|
+
const cli = resolvePlaywrightCli();
|
|
1483
|
+
const child = cli ? (0, import_node_child_process3.spawn)(process.execPath, [cli, "test", ...playwrightArgs], { stdio: "inherit", env }) : (0, import_node_child_process3.spawn)(process.platform === "win32" ? `${pkgRunner}.cmd` : pkgRunner, ["playwright", "test", ...playwrightArgs], {
|
|
1484
|
+
stdio: "inherit",
|
|
1485
|
+
env
|
|
1486
|
+
});
|
|
1487
|
+
return new Promise((resolve5) => {
|
|
1488
|
+
child.on("error", (err) => {
|
|
1489
|
+
console.error(`piwi run: could not start Playwright \u2014 ${err.message}`);
|
|
1490
|
+
resolve5(EXIT_ERROR3);
|
|
1491
|
+
});
|
|
1492
|
+
child.on("exit", (code) => resolve5(code ?? EXIT_ERROR3));
|
|
1493
|
+
});
|
|
1494
|
+
}
|
|
1495
|
+
async function runRunImpact(args, env) {
|
|
1496
|
+
let projectId;
|
|
1497
|
+
try {
|
|
1498
|
+
projectId = await resolveProjectId(args);
|
|
1499
|
+
} catch (e) {
|
|
1500
|
+
if (args.strict) {
|
|
1501
|
+
console.error(`piwi run: ${e.message}`);
|
|
1502
|
+
return EXIT_ERROR3;
|
|
1503
|
+
}
|
|
1504
|
+
console.error(`piwi run: ${e.message} \u2014 running the full suite`);
|
|
1505
|
+
return spawnPlaywright(args.pkgRunner, args.extra, env);
|
|
1506
|
+
}
|
|
1507
|
+
let impact;
|
|
1508
|
+
try {
|
|
1509
|
+
impact = await loadImpact(args, projectId);
|
|
1510
|
+
} catch (e) {
|
|
1511
|
+
if (args.strict) {
|
|
1512
|
+
console.error(`piwi run: ${e.message}`);
|
|
1513
|
+
return EXIT_ERROR3;
|
|
1514
|
+
}
|
|
1515
|
+
console.error(`piwi run: ${e.message} \u2014 running the full suite`);
|
|
1516
|
+
return spawnPlaywright(args.pkgRunner, args.extra, env);
|
|
1517
|
+
}
|
|
1518
|
+
printWarnings(impact);
|
|
1519
|
+
if (impact.impact.widened) {
|
|
1520
|
+
console.error(
|
|
1521
|
+
`piwi run: impact widened to the full suite (${impact.impact.unmappedSourceFiles.length} unmapped source file(s))`
|
|
1522
|
+
);
|
|
1523
|
+
return spawnPlaywright(args.pkgRunner, args.extra, env);
|
|
1524
|
+
}
|
|
1525
|
+
if (impact.estimate.count === 0) {
|
|
1526
|
+
console.error(`piwi run: no tests impacted by ${impact.impact.changedFiles} changed file(s) \u2014 nothing to run`);
|
|
1527
|
+
return EXIT_OK3;
|
|
1528
|
+
}
|
|
1529
|
+
const runEnv = {
|
|
1530
|
+
...env,
|
|
1531
|
+
PIWI_SELECTION: IMPACT_KEY,
|
|
1532
|
+
PIWI_SELECTION_VERSION: "0",
|
|
1533
|
+
PIWI_SELECTION_HASH: impact.resolvedHash,
|
|
1534
|
+
PIWI_SELECTION_COUNT: String(impact.estimate.count)
|
|
1535
|
+
};
|
|
1536
|
+
console.error(`piwi run: impact \u2192 ${impact.estimate.count} test(s)`);
|
|
1537
|
+
return spawnPlaywright(args.pkgRunner, [...impact.materialization.args, ...args.extra], runEnv);
|
|
1538
|
+
}
|
|
1539
|
+
async function runRun(argv, env = process.env) {
|
|
1540
|
+
if (argv.includes("-h") || argv.includes("--help")) {
|
|
1541
|
+
console.log(USAGE5);
|
|
1542
|
+
return EXIT_OK3;
|
|
1543
|
+
}
|
|
1544
|
+
let args;
|
|
1545
|
+
try {
|
|
1546
|
+
args = parseSelectArgs(argv, env);
|
|
1547
|
+
} catch (e) {
|
|
1548
|
+
console.error(`piwi run: ${e.message}
|
|
1549
|
+
`);
|
|
1550
|
+
console.error(USAGE5);
|
|
1551
|
+
return EXIT_ERROR3;
|
|
1552
|
+
}
|
|
1553
|
+
if (args.format === "json") {
|
|
1554
|
+
console.error("piwi run: --format json cannot be run; use args, grep or files");
|
|
1555
|
+
return EXIT_ERROR3;
|
|
1556
|
+
}
|
|
1557
|
+
if (args.key === IMPACT_KEY) {
|
|
1558
|
+
if (!args.base) {
|
|
1559
|
+
console.error("piwi run: impact needs a base ref \u2014 pass --base <ref> (e.g. --base origin/main)");
|
|
1560
|
+
return EXIT_ERROR3;
|
|
1561
|
+
}
|
|
1562
|
+
return runRunImpact(args, env);
|
|
1563
|
+
}
|
|
1564
|
+
let projectId;
|
|
1565
|
+
try {
|
|
1566
|
+
projectId = await resolveProjectId(args);
|
|
1567
|
+
} catch (e) {
|
|
1568
|
+
if (args.strict) {
|
|
1569
|
+
console.error(`piwi run: ${e.message}`);
|
|
1570
|
+
return EXIT_ERROR3;
|
|
1571
|
+
}
|
|
1572
|
+
console.error(`piwi run: ${e.message} \u2014 running the full suite`);
|
|
1573
|
+
return spawnPlaywright(args.pkgRunner, args.extra, env);
|
|
1574
|
+
}
|
|
1575
|
+
let outcome;
|
|
1576
|
+
try {
|
|
1577
|
+
outcome = await resolveWithCache(args, projectId);
|
|
1578
|
+
} catch (e) {
|
|
1579
|
+
console.error(`piwi run: ${e.message}`);
|
|
1580
|
+
return EXIT_ERROR3;
|
|
1581
|
+
}
|
|
1582
|
+
if (!outcome) {
|
|
1583
|
+
console.error("piwi run: dashboard unreachable and no cached resolution \u2014 running the full suite");
|
|
1584
|
+
return spawnPlaywright(args.pkgRunner, args.extra, env);
|
|
1585
|
+
}
|
|
1586
|
+
const { resolution } = outcome;
|
|
1587
|
+
printWarnings(resolution);
|
|
1588
|
+
if (resolution.estimate.count === 0) {
|
|
1589
|
+
console.error(`piwi run: "${args.key}" resolved to 0 tests`);
|
|
1590
|
+
return EXIT_ERROR3;
|
|
1591
|
+
}
|
|
1592
|
+
const runEnv = {
|
|
1593
|
+
...env,
|
|
1594
|
+
PIWI_SELECTION: resolution.key ?? args.key,
|
|
1595
|
+
PIWI_SELECTION_VERSION: String(resolution.version ?? 0),
|
|
1596
|
+
PIWI_SELECTION_HASH: resolution.resolvedHash,
|
|
1597
|
+
PIWI_SELECTION_COUNT: String(resolution.estimate.count)
|
|
1598
|
+
};
|
|
1599
|
+
console.error(
|
|
1600
|
+
`piwi run: ${args.key} \u2192 ${resolution.estimate.count} tests${resolution.materialization.format !== args.format ? ` (materialized as ${resolution.materialization.format})` : ""}`
|
|
1601
|
+
);
|
|
1602
|
+
return spawnPlaywright(args.pkgRunner, [...resolution.materialization.args, ...args.extra], runEnv);
|
|
1603
|
+
}
|
|
1604
|
+
|
|
1605
|
+
// src/cli/index.ts
|
|
1606
|
+
var USAGE6 = `
|
|
1191
1607
|
piwi \u2014 companion commands for the Piwi Dashboard reporter
|
|
1192
1608
|
|
|
1193
1609
|
Usage:
|
|
@@ -1197,6 +1613,8 @@ Commands:
|
|
|
1197
1613
|
init Wire a Playwright project up to a Piwi Dashboard
|
|
1198
1614
|
skills Install the Piwi agent skills into this project
|
|
1199
1615
|
gate Fail a CI job on the dashboard's analysis of a run
|
|
1616
|
+
select Print the Playwright args for a saved test selection
|
|
1617
|
+
run Run a saved test selection with playwright test
|
|
1200
1618
|
ai Manage committed natural-language AI-step artifacts
|
|
1201
1619
|
|
|
1202
1620
|
Run \`npx @piwitests/reporter <command> --help\` for a command's options.
|
|
@@ -1212,17 +1630,21 @@ async function main() {
|
|
|
1212
1630
|
return runSkills(rest, findTemplatesDir(__dirname));
|
|
1213
1631
|
case "gate":
|
|
1214
1632
|
return runGate(rest);
|
|
1633
|
+
case "select":
|
|
1634
|
+
return runSelect(rest);
|
|
1635
|
+
case "run":
|
|
1636
|
+
return runRun(rest);
|
|
1215
1637
|
case "ai":
|
|
1216
1638
|
return runAi(rest);
|
|
1217
1639
|
case void 0:
|
|
1218
1640
|
case "-h":
|
|
1219
1641
|
case "--help":
|
|
1220
|
-
console.log(
|
|
1642
|
+
console.log(USAGE6);
|
|
1221
1643
|
return 0;
|
|
1222
1644
|
default:
|
|
1223
1645
|
console.error(`piwi: unknown command "${command}"
|
|
1224
1646
|
`);
|
|
1225
|
-
console.error(
|
|
1647
|
+
console.error(USAGE6);
|
|
1226
1648
|
return 2;
|
|
1227
1649
|
}
|
|
1228
1650
|
}
|
|
@@ -81,6 +81,7 @@ var DEFAULTS = {
|
|
|
81
81
|
streaming: true,
|
|
82
82
|
streamingBatchSize: 5,
|
|
83
83
|
streamingBatchDelay: 2e3,
|
|
84
|
+
failOnFlakyTests: false,
|
|
84
85
|
username: null,
|
|
85
86
|
password: null,
|
|
86
87
|
apiKey: null,
|
|
@@ -100,6 +101,7 @@ var PIWI_ENV_KEYS = {
|
|
|
100
101
|
streamingBatchSize: "PIWI_STREAMING_BATCH_SIZE",
|
|
101
102
|
streamingBatchDelay: "PIWI_STREAMING_BATCH_DELAY",
|
|
102
103
|
liveFileUploads: "PIWI_LIVE_FILE_UPLOADS",
|
|
104
|
+
failOnFlakyTests: "PIWI_FAIL_ON_FLAKY_TESTS",
|
|
103
105
|
uploadTraces: "PIWI_UPLOAD_TRACES",
|
|
104
106
|
uploadReport: "PIWI_UPLOAD_REPORT",
|
|
105
107
|
captureLocators: "PIWI_CAPTURE_LOCATORS",
|
|
@@ -135,6 +137,7 @@ var ENV_FALLBACK_SPECS = [
|
|
|
135
137
|
{ option: "streamingBatchSize", env: PIWI_ENV_KEYS.streamingBatchSize, kind: "number" },
|
|
136
138
|
{ option: "streamingBatchDelay", env: PIWI_ENV_KEYS.streamingBatchDelay, kind: "number" },
|
|
137
139
|
{ option: "liveFileUploads", env: PIWI_ENV_KEYS.liveFileUploads, kind: "bool" },
|
|
140
|
+
{ option: "failOnFlakyTests", env: PIWI_ENV_KEYS.failOnFlakyTests, kind: "bool" },
|
|
138
141
|
{ option: "uploadTraces", env: PIWI_ENV_KEYS.uploadTraces, kind: "bool" },
|
|
139
142
|
{ option: "uploadReport", env: PIWI_ENV_KEYS.uploadReport, kind: "bool" },
|
|
140
143
|
{ option: "captureLocators", env: PIWI_ENV_KEYS.captureLocators, kind: "bool" },
|
|
@@ -506,7 +509,7 @@ function createGlobalSetup(options, userSetup) {
|
|
|
506
509
|
projectName: opts.projectName
|
|
507
510
|
})
|
|
508
511
|
);
|
|
509
|
-
logger.debug(`Global setup:
|
|
512
|
+
logger.debug(`Global setup: initializing run #${response.runId}`);
|
|
510
513
|
}
|
|
511
514
|
} catch (error) {
|
|
512
515
|
logger.warn(`Could not register global setup: ${errorMessage(error)}`);
|
package/dist/index.d.ts
CHANGED
|
@@ -88,6 +88,14 @@ interface PiwiDashboardOptions {
|
|
|
88
88
|
streamingBatchSize?: number;
|
|
89
89
|
/** Max delay (ms) before flushing pending events during streaming. Defaults to `2000`. */
|
|
90
90
|
streamingBatchDelay?: number;
|
|
91
|
+
/**
|
|
92
|
+
* Fail the run when any test was flaky (passed only after a retry). Forwarded
|
|
93
|
+
* to Playwright's native `failOnFlakyTests` config option (Playwright 1.52+)
|
|
94
|
+
* when the reporter is installed via `wrapConfig`, so a flaky-only run exits
|
|
95
|
+
* non-zero without any server round-trip. Defaults to `false`. Can also be
|
|
96
|
+
* set with `PIWI_FAIL_ON_FLAKY_TESTS`.
|
|
97
|
+
*/
|
|
98
|
+
failOnFlakyTests?: boolean;
|
|
91
99
|
/** Username for dashboard login (use `apiKey` instead when possible) */
|
|
92
100
|
username?: string | null;
|
|
93
101
|
/** Password for dashboard login (used with `username`) */
|
|
@@ -239,6 +247,8 @@ declare class PiwiDashboardReporter {
|
|
|
239
247
|
private plannedTests;
|
|
240
248
|
/** Ids of tests that actually reported via `onTestEnd`, to find the ones that never ran. */
|
|
241
249
|
private reportedTestIds;
|
|
250
|
+
/** Per-test attempt history, keyed by `test.id`; snapshotted onto every attempt's payload. */
|
|
251
|
+
private attemptsByTest;
|
|
242
252
|
private instanceId;
|
|
243
253
|
private runLabel;
|
|
244
254
|
private shardInfo;
|
|
@@ -248,6 +258,8 @@ declare class PiwiDashboardReporter {
|
|
|
248
258
|
private viaDesktopApp;
|
|
249
259
|
private isFullRun;
|
|
250
260
|
private filterDetails;
|
|
261
|
+
/** Configured `maxFailures` (0 = unlimited) — disambiguates an interrupted run's unrun reason. */
|
|
262
|
+
private maxFailures;
|
|
251
263
|
private httpClient;
|
|
252
264
|
private uploader;
|
|
253
265
|
private fileHandler;
|
|
@@ -265,6 +277,12 @@ declare class PiwiDashboardReporter {
|
|
|
265
277
|
onTestBegin(test: TestCase, result: TestResult): void;
|
|
266
278
|
/** Track suite-level setup steps (beforeAll/afterAll) not tied to any test */
|
|
267
279
|
private setupSteps;
|
|
280
|
+
/**
|
|
281
|
+
* Step categories streamed live while the run executes. `pw:assert` is
|
|
282
|
+
* excluded: it is the polling noise of `expect()`, not a step a human
|
|
283
|
+
* watches; the meaningful readout is the `pw:expect` wrapper around it.
|
|
284
|
+
*/
|
|
285
|
+
private static readonly LIVE_STEP_CATEGORIES;
|
|
268
286
|
/** Playwright reporter hook: called when a step (including hook/fixture) begins */
|
|
269
287
|
onStepBegin(test: TestCase | undefined, _result: TestResult | undefined, step: any): void;
|
|
270
288
|
/** Playwright reporter hook: called when a step (including hook/fixture) ends */
|
|
@@ -276,13 +294,44 @@ declare class PiwiDashboardReporter {
|
|
|
276
294
|
* (no `onTestEnd`) — typically because `maxFailures` cut the run short. These
|
|
277
295
|
* carry no result, so they're emitted with zero duration and no error. In
|
|
278
296
|
* streaming mode they're queued as complete events so the pre-finish drain
|
|
279
|
-
* sends them alongside the rest.
|
|
297
|
+
* sends them alongside the rest. The `reason` (global timeout / max failures /
|
|
298
|
+
* interrupted) is resolved once from the overall run status by the caller.
|
|
280
299
|
*/
|
|
281
300
|
private materializeUnrunTests;
|
|
282
301
|
/** Playwright reporter hook: called when the full test run finishes */
|
|
283
302
|
onEnd(result: FullResult): Promise<void>;
|
|
284
303
|
}
|
|
285
304
|
|
|
305
|
+
interface ResolveSelectionResult {
|
|
306
|
+
key: string;
|
|
307
|
+
version: number;
|
|
308
|
+
resolvedHash: string;
|
|
309
|
+
resolvedCount: number;
|
|
310
|
+
/** A `--grep` regex that selects the resolved tests, when the grep format was produced. */
|
|
311
|
+
grep?: string;
|
|
312
|
+
/** File (or `file:line`) tokens, when grep fell back to a file materialization. */
|
|
313
|
+
files?: string[];
|
|
314
|
+
}
|
|
315
|
+
interface ResolveSelectionOptions {
|
|
316
|
+
/** Selection key; defaults to the `PIWI_SELECTION` env var. */
|
|
317
|
+
key?: string;
|
|
318
|
+
serverUrl?: string;
|
|
319
|
+
apiKey?: string | null;
|
|
320
|
+
/** Project name or id; defaults to `PIWI_PROJECT_NAME`. */
|
|
321
|
+
project?: string;
|
|
322
|
+
budgetMs?: number | null;
|
|
323
|
+
/** Throw instead of falling back to the full suite when resolution fails. */
|
|
324
|
+
strict?: boolean;
|
|
325
|
+
env?: NodeJS.ProcessEnv;
|
|
326
|
+
}
|
|
327
|
+
/**
|
|
328
|
+
* Resolve the configured selection. Returns `undefined` when no selection is
|
|
329
|
+
* named or (in non-strict mode) when the dashboard cannot be reached, so the
|
|
330
|
+
* config runs the full suite. On success it also writes `PIWI_SELECTION_*` so
|
|
331
|
+
* the reporter stamps the run with the resolved selection.
|
|
332
|
+
*/
|
|
333
|
+
declare function resolveSelection(options?: ResolveSelectionOptions): Promise<ResolveSelectionResult | undefined>;
|
|
334
|
+
|
|
286
335
|
/**
|
|
287
336
|
* Parameters are first-class: a template like `row for {name}` is the cache key,
|
|
288
337
|
* so one entry serves every value. Placeholders survive compilation as `{{name}}`
|
|
@@ -345,4 +394,4 @@ declare function extendPiwiAi<TestArgs extends FixtureArgs, WorkerArgs extends F
|
|
|
345
394
|
* exported from this file, it isn't part of the supported API.
|
|
346
395
|
*/
|
|
347
396
|
|
|
348
|
-
export { type AiMode, type AiOnMiss, type PiwiAi, type PiwiDashboardOptions, PiwiDashboardReporter, createGlobalSetup, PiwiDashboardReporter as default, extendPiwiAi, piwiAiFixtures, wrapConfig };
|
|
397
|
+
export { type AiMode, type AiOnMiss, type PiwiAi, type PiwiDashboardOptions, PiwiDashboardReporter, type ResolveSelectionOptions, type ResolveSelectionResult, createGlobalSetup, PiwiDashboardReporter as default, extendPiwiAi, piwiAiFixtures, resolveSelection, wrapConfig };
|
package/dist/index.js
CHANGED
|
@@ -37,6 +37,7 @@ __export(index_exports, {
|
|
|
37
37
|
extendPiwiFixtures: () => extendPiwiFixtures,
|
|
38
38
|
piwiAiFixtures: () => piwiAiFixtures,
|
|
39
39
|
piwiFixtures: () => piwiFixtures,
|
|
40
|
+
resolveSelection: () => resolveSelection,
|
|
40
41
|
wrapConfig: () => wrapConfig
|
|
41
42
|
});
|
|
42
43
|
module.exports = __toCommonJS(index_exports);
|
|
@@ -79,6 +80,7 @@ var DEFAULTS = {
|
|
|
79
80
|
streaming: true,
|
|
80
81
|
streamingBatchSize: 5,
|
|
81
82
|
streamingBatchDelay: 2e3,
|
|
83
|
+
failOnFlakyTests: false,
|
|
82
84
|
username: null,
|
|
83
85
|
password: null,
|
|
84
86
|
apiKey: null,
|
|
@@ -98,6 +100,7 @@ var PIWI_ENV_KEYS = {
|
|
|
98
100
|
streamingBatchSize: "PIWI_STREAMING_BATCH_SIZE",
|
|
99
101
|
streamingBatchDelay: "PIWI_STREAMING_BATCH_DELAY",
|
|
100
102
|
liveFileUploads: "PIWI_LIVE_FILE_UPLOADS",
|
|
103
|
+
failOnFlakyTests: "PIWI_FAIL_ON_FLAKY_TESTS",
|
|
101
104
|
uploadTraces: "PIWI_UPLOAD_TRACES",
|
|
102
105
|
uploadReport: "PIWI_UPLOAD_REPORT",
|
|
103
106
|
captureLocators: "PIWI_CAPTURE_LOCATORS",
|
|
@@ -116,6 +119,12 @@ var PIWI_ENV_KEYS = {
|
|
|
116
119
|
aiScreenshotFallback: "PIWI_AI_SCREENSHOT_FALLBACK"
|
|
117
120
|
};
|
|
118
121
|
var PIWI_DESKTOP_CONFIG_ENV = "PIWI_DESKTOP_CONFIG";
|
|
122
|
+
var PIWI_SELECTION_ENV = {
|
|
123
|
+
key: "PIWI_SELECTION",
|
|
124
|
+
version: "PIWI_SELECTION_VERSION",
|
|
125
|
+
hash: "PIWI_SELECTION_HASH",
|
|
126
|
+
count: "PIWI_SELECTION_COUNT"
|
|
127
|
+
};
|
|
119
128
|
function readBool(val) {
|
|
120
129
|
if (val === void 0) return void 0;
|
|
121
130
|
return val === "true";
|
|
@@ -133,6 +142,7 @@ var ENV_FALLBACK_SPECS = [
|
|
|
133
142
|
{ option: "streamingBatchSize", env: PIWI_ENV_KEYS.streamingBatchSize, kind: "number" },
|
|
134
143
|
{ option: "streamingBatchDelay", env: PIWI_ENV_KEYS.streamingBatchDelay, kind: "number" },
|
|
135
144
|
{ option: "liveFileUploads", env: PIWI_ENV_KEYS.liveFileUploads, kind: "bool" },
|
|
145
|
+
{ option: "failOnFlakyTests", env: PIWI_ENV_KEYS.failOnFlakyTests, kind: "bool" },
|
|
136
146
|
{ option: "uploadTraces", env: PIWI_ENV_KEYS.uploadTraces, kind: "bool" },
|
|
137
147
|
{ option: "uploadReport", env: PIWI_ENV_KEYS.uploadReport, kind: "bool" },
|
|
138
148
|
{ option: "captureLocators", env: PIWI_ENV_KEYS.captureLocators, kind: "bool" },
|
|
@@ -430,8 +440,8 @@ function resolveOverallStatus(result, counters) {
|
|
|
430
440
|
const STATUS_MAP = {
|
|
431
441
|
passed: "passed",
|
|
432
442
|
failed: "failed",
|
|
433
|
-
timedout: "
|
|
434
|
-
interrupted: "
|
|
443
|
+
timedout: "timedout",
|
|
444
|
+
interrupted: "interrupted"
|
|
435
445
|
};
|
|
436
446
|
if (result?.status) return STATUS_MAP[result.status] ?? "failed";
|
|
437
447
|
if (counters.failedTests === 0 && counters.timedOutTests === 0 && counters.totalTests > 0) return "passed";
|
|
@@ -448,6 +458,7 @@ function toWireTestCase(tc) {
|
|
|
448
458
|
timeout: rest.timeout ?? null,
|
|
449
459
|
error: rest.error,
|
|
450
460
|
retries: rest.retries,
|
|
461
|
+
attempts: rest.attempts || null,
|
|
451
462
|
workerIndex: rest.workerIndex ?? null,
|
|
452
463
|
shardIndex: rest.shardIndex ?? null,
|
|
453
464
|
startedAt: rest.startedAt ?? null,
|
|
@@ -470,7 +481,9 @@ function toWireTestCase(tc) {
|
|
|
470
481
|
testAnnotations: rest.testAnnotations ?? null,
|
|
471
482
|
tags: rest.tags ?? null,
|
|
472
483
|
testMeta: rest.testMeta ?? null,
|
|
473
|
-
locatorSnapshots: rest.locatorSnapshots || null
|
|
484
|
+
locatorSnapshots: rest.locatorSnapshots || null,
|
|
485
|
+
didNotRunReason: rest.didNotRunReason ?? null,
|
|
486
|
+
blockedBy: rest.blockedBy ?? null
|
|
474
487
|
};
|
|
475
488
|
}
|
|
476
489
|
function serializeRun(payload, opts) {
|
|
@@ -523,8 +536,8 @@ var Uploader = class {
|
|
|
523
536
|
auth
|
|
524
537
|
);
|
|
525
538
|
this.logger.info(`Successfully uploaded test results`);
|
|
526
|
-
if (response.
|
|
527
|
-
this.logger.info(`Test Run ID: ${response.
|
|
539
|
+
if (response.runId) {
|
|
540
|
+
this.logger.info(`Test Run ID: ${response.runId}, Project ID: ${response.projectId}`);
|
|
528
541
|
}
|
|
529
542
|
return response;
|
|
530
543
|
}
|
|
@@ -538,8 +551,8 @@ var Uploader = class {
|
|
|
538
551
|
await this.appendFilesToForm(form, payload.testCases, reportOptions.uploadTraces);
|
|
539
552
|
const response = await this.httpClient.postFormData("/api/test-runs/upload", form, auth);
|
|
540
553
|
this.logger.info(`Successfully uploaded test results with files`);
|
|
541
|
-
if (response.
|
|
542
|
-
this.logger.info(`Test Run ID: ${response.
|
|
554
|
+
if (response.runId) {
|
|
555
|
+
this.logger.info(`Test Run ID: ${response.runId}, Project ID: ${response.projectId}`);
|
|
543
556
|
}
|
|
544
557
|
if (response.reports) {
|
|
545
558
|
for (const r of response.reports) this.logger.info(`${r.label}: ${r.path}`);
|
|
@@ -549,7 +562,7 @@ var Uploader = class {
|
|
|
549
562
|
/** Upload report files for an already-submitted streaming run */
|
|
550
563
|
async uploadReportsForStreamingRun(projectName, runId, reportOptions, startTime, auth) {
|
|
551
564
|
const form = new import_form_data2.default();
|
|
552
|
-
form.append("
|
|
565
|
+
form.append("runId", String(runId));
|
|
553
566
|
form.append("projectName", projectName);
|
|
554
567
|
form.append(
|
|
555
568
|
"testRun",
|
|
@@ -1012,6 +1025,39 @@ var FileHandler = class {
|
|
|
1012
1025
|
|
|
1013
1026
|
// src/internal/collect/metadata-collector.ts
|
|
1014
1027
|
var import_node_child_process = require("child_process");
|
|
1028
|
+
var DETACHED_HEAD = "HEAD";
|
|
1029
|
+
function resolveScmBranch(env, gitBranch) {
|
|
1030
|
+
const pick = (...names) => {
|
|
1031
|
+
for (const name of names) {
|
|
1032
|
+
const value = env[name]?.trim();
|
|
1033
|
+
if (value) return value;
|
|
1034
|
+
}
|
|
1035
|
+
return void 0;
|
|
1036
|
+
};
|
|
1037
|
+
const override = env.PIWI_BRANCH?.trim();
|
|
1038
|
+
if (override) return override;
|
|
1039
|
+
let provider;
|
|
1040
|
+
if (env.JENKINS_URL) provider = pick("CHANGE_BRANCH", "BRANCH_NAME");
|
|
1041
|
+
else if (env.GITHUB_ACTIONS) provider = pick("GITHUB_HEAD_REF", "GITHUB_REF_NAME");
|
|
1042
|
+
else if (env.GITLAB_CI) provider = pick("CI_MERGE_REQUEST_SOURCE_BRANCH_NAME", "CI_COMMIT_REF_NAME");
|
|
1043
|
+
else if (env.CIRCLECI) provider = pick("CIRCLE_BRANCH");
|
|
1044
|
+
else if (env.TRAVIS) provider = pick("TRAVIS_PULL_REQUEST_BRANCH", "TRAVIS_BRANCH");
|
|
1045
|
+
else if (env.TF_BUILD) provider = pick("SYSTEM_PULLREQUEST_SOURCEBRANCH", "BUILD_SOURCEBRANCHNAME");
|
|
1046
|
+
else if (env.BITBUCKET_BUILD_NUMBER) provider = pick("BITBUCKET_BRANCH");
|
|
1047
|
+
if (provider) return normalizeRef(provider);
|
|
1048
|
+
const trimmed = gitBranch?.trim();
|
|
1049
|
+
if (trimmed && trimmed !== DETACHED_HEAD) return trimmed;
|
|
1050
|
+
return void 0;
|
|
1051
|
+
}
|
|
1052
|
+
function resolveScmPrNumber(env) {
|
|
1053
|
+
const githubRef = env.GITHUB_ACTIONS ? env.GITHUB_REF?.match(/^refs\/pull\/(\d+)\/(?:merge|head)$/) : void 0;
|
|
1054
|
+
const raw = githubRef?.[1] ?? env.CI_MERGE_REQUEST_IID ?? env.BITBUCKET_PR_ID ?? env.SYSTEM_PULLREQUEST_PULLREQUESTNUMBER ?? env.CHANGE_ID;
|
|
1055
|
+
const trimmed = raw?.trim();
|
|
1056
|
+
return trimmed && /^\d+$/.test(trimmed) ? trimmed : void 0;
|
|
1057
|
+
}
|
|
1058
|
+
function normalizeRef(ref) {
|
|
1059
|
+
return ref.replace(/^refs\/heads\//, "");
|
|
1060
|
+
}
|
|
1015
1061
|
var MetadataCollector = class {
|
|
1016
1062
|
constructor(logger = new Logger()) {
|
|
1017
1063
|
this.logger = logger;
|
|
@@ -1117,6 +1163,7 @@ var MetadataCollector = class {
|
|
|
1117
1163
|
}
|
|
1118
1164
|
collectScmInfo(_options) {
|
|
1119
1165
|
const scm = {};
|
|
1166
|
+
let gitBranch;
|
|
1120
1167
|
try {
|
|
1121
1168
|
const execOpts = {
|
|
1122
1169
|
encoding: "utf8",
|
|
@@ -1125,7 +1172,7 @@ var MetadataCollector = class {
|
|
|
1125
1172
|
stdio: ["ignore", "pipe", "ignore"]
|
|
1126
1173
|
};
|
|
1127
1174
|
scm.commit = (0, import_node_child_process.execSync)("git rev-parse HEAD", execOpts).trim();
|
|
1128
|
-
|
|
1175
|
+
gitBranch = (0, import_node_child_process.execSync)("git rev-parse --abbrev-ref HEAD", execOpts).trim();
|
|
1129
1176
|
scm.author = (0, import_node_child_process.execSync)('git log -1 --pretty=format:"%an"', execOpts).trim();
|
|
1130
1177
|
scm.commitMessage = (0, import_node_child_process.execSync)('git log -1 --pretty=format:"%s"', execOpts).trim();
|
|
1131
1178
|
try {
|
|
@@ -1135,6 +1182,10 @@ var MetadataCollector = class {
|
|
|
1135
1182
|
} catch (error) {
|
|
1136
1183
|
this.logger.debug(`Git info not available: ${errorMessage(error)}`);
|
|
1137
1184
|
}
|
|
1185
|
+
const branch = resolveScmBranch(process.env, gitBranch);
|
|
1186
|
+
if (branch) scm.branch = branch;
|
|
1187
|
+
const prNumber = resolveScmPrNumber(process.env);
|
|
1188
|
+
if (prNumber) scm.prNumber = prNumber;
|
|
1138
1189
|
return Object.keys(scm).length > 0 ? scm : void 0;
|
|
1139
1190
|
}
|
|
1140
1191
|
collectCiInfo() {
|
|
@@ -1421,8 +1472,8 @@ var StreamManager = class {
|
|
|
1421
1472
|
if (this.pendingBeginEvents.length > 0) {
|
|
1422
1473
|
this.pendingEvents = [...this.pendingBeginEvents, ...this.pendingEvents];
|
|
1423
1474
|
this.pendingBeginEvents = [];
|
|
1424
|
-
this.flush();
|
|
1425
1475
|
}
|
|
1476
|
+
if (this.pendingEvents.length > 0) this.flush();
|
|
1426
1477
|
}
|
|
1427
1478
|
} catch (error) {
|
|
1428
1479
|
if (error instanceof HttpError && (error.status === 401 || error.status === 403)) {
|
|
@@ -1934,6 +1985,17 @@ function detectCliFileFilters(argv = process.argv) {
|
|
|
1934
1985
|
return files;
|
|
1935
1986
|
}
|
|
1936
1987
|
|
|
1988
|
+
// src/internal/support/selection-env.ts
|
|
1989
|
+
function readSelectionStamp(env = process.env) {
|
|
1990
|
+
const key = env[PIWI_SELECTION_ENV.key];
|
|
1991
|
+
if (!key) return null;
|
|
1992
|
+
const version = Number(env[PIWI_SELECTION_ENV.version]);
|
|
1993
|
+
const resolvedCount = Number(env[PIWI_SELECTION_ENV.count]);
|
|
1994
|
+
const resolvedHash = env[PIWI_SELECTION_ENV.hash] ?? "";
|
|
1995
|
+
if (!Number.isFinite(version) || !Number.isFinite(resolvedCount) || !resolvedHash) return null;
|
|
1996
|
+
return { key, version, resolvedHash, resolvedCount };
|
|
1997
|
+
}
|
|
1998
|
+
|
|
1937
1999
|
// src/public/global-setup.ts
|
|
1938
2000
|
var path10 = __toESM(require("path"));
|
|
1939
2001
|
var fs11 = __toESM(require("fs"));
|
|
@@ -2024,7 +2086,7 @@ function createGlobalSetup(options, userSetup) {
|
|
|
2024
2086
|
projectName: opts.projectName
|
|
2025
2087
|
})
|
|
2026
2088
|
);
|
|
2027
|
-
logger.debug(`Global setup:
|
|
2089
|
+
logger.debug(`Global setup: initializing run #${response.runId}`);
|
|
2028
2090
|
}
|
|
2029
2091
|
} catch (error) {
|
|
2030
2092
|
logger.warn(`Could not register global setup: ${errorMessage(error)}`);
|
|
@@ -2067,8 +2129,12 @@ function wrapConfig(config, piwiOptions) {
|
|
|
2067
2129
|
globalSetupModules.push(...orig);
|
|
2068
2130
|
}
|
|
2069
2131
|
globalSetupModules.push(resolveSetupModule());
|
|
2132
|
+
const forwarded = {};
|
|
2133
|
+
const failOnFlaky = piwiOptions?.failOnFlakyTests ?? readBool(process.env[PIWI_ENV_KEYS.failOnFlakyTests]);
|
|
2134
|
+
if (failOnFlaky === true) forwarded.failOnFlakyTests = true;
|
|
2070
2135
|
return {
|
|
2071
2136
|
...config,
|
|
2137
|
+
...forwarded,
|
|
2072
2138
|
reporter: injectReporter(config.reporter, piwiOptions),
|
|
2073
2139
|
globalSetup: globalSetupModules.length === 1 ? globalSetupModules[0] : globalSetupModules
|
|
2074
2140
|
};
|
|
@@ -2093,6 +2159,45 @@ function classifyStatus(rawStatus, annotations) {
|
|
|
2093
2159
|
const intentional = annotations.some((a) => a.type === "skip" || a.type === "fixme");
|
|
2094
2160
|
return intentional ? "skipped" : "didnotrun";
|
|
2095
2161
|
}
|
|
2162
|
+
function resolveUnrunReason(runStatus, opts) {
|
|
2163
|
+
if (runStatus === "timedout") return "global-timeout";
|
|
2164
|
+
if (opts.maxFailures > 0 && opts.failures >= opts.maxFailures) return "max-failures";
|
|
2165
|
+
return "interrupted";
|
|
2166
|
+
}
|
|
2167
|
+
function fileOf(location) {
|
|
2168
|
+
return location.replace(/:\d+:\d+$/, "");
|
|
2169
|
+
}
|
|
2170
|
+
function suiteKey(suitePath) {
|
|
2171
|
+
return (suitePath ?? []).join("");
|
|
2172
|
+
}
|
|
2173
|
+
function sharedPrefixDepth(a, b) {
|
|
2174
|
+
let depth = 0;
|
|
2175
|
+
while (depth < a.length && depth < b.length && a[depth] === b[depth]) depth++;
|
|
2176
|
+
return depth;
|
|
2177
|
+
}
|
|
2178
|
+
function linkBlockedTests(cases) {
|
|
2179
|
+
const blockers = cases.filter((c) => c.status === "failed" || c.status === "timedOut" || c.status === "timedout");
|
|
2180
|
+
if (blockers.length === 0) return;
|
|
2181
|
+
for (const c of cases) {
|
|
2182
|
+
if (c.didNotRunReason !== "previous-failure") continue;
|
|
2183
|
+
const file = fileOf(c.location);
|
|
2184
|
+
const inFile = blockers.filter((b) => fileOf(b.location) === file);
|
|
2185
|
+
if (inFile.length === 0) continue;
|
|
2186
|
+
const key = suiteKey(c.suitePath);
|
|
2187
|
+
let blocker = inFile.find((b) => suiteKey(b.suitePath) === key);
|
|
2188
|
+
if (!blocker) {
|
|
2189
|
+
let bestDepth = 0;
|
|
2190
|
+
for (const b of inFile) {
|
|
2191
|
+
const depth = sharedPrefixDepth(c.suitePath ?? [], b.suitePath ?? []);
|
|
2192
|
+
if (depth > bestDepth) {
|
|
2193
|
+
bestDepth = depth;
|
|
2194
|
+
blocker = b;
|
|
2195
|
+
}
|
|
2196
|
+
}
|
|
2197
|
+
}
|
|
2198
|
+
if (blocker) c.blockedBy = blocker.location;
|
|
2199
|
+
}
|
|
2200
|
+
}
|
|
2096
2201
|
|
|
2097
2202
|
// ../core/src/test-meta.ts
|
|
2098
2203
|
var PIWI_ANNOTATION_PREFIX = "piwi:";
|
|
@@ -2462,7 +2567,7 @@ var RunSubmitter = class {
|
|
|
2462
2567
|
auth
|
|
2463
2568
|
);
|
|
2464
2569
|
this.recovery.clear();
|
|
2465
|
-
return { done: true, output: this.buildOutput(response?.
|
|
2570
|
+
return { done: true, output: this.buildOutput(response?.runId, response?.projectId, run, overallStatus) };
|
|
2466
2571
|
} catch (error) {
|
|
2467
2572
|
if (error instanceof HttpError && error.status === 401 && !auth) {
|
|
2468
2573
|
this.logAuthRequired(run.options.serverUrl);
|
|
@@ -2478,7 +2583,7 @@ var RunSubmitter = class {
|
|
|
2478
2583
|
try {
|
|
2479
2584
|
const response = await this.uploader.uploadJSON(payload, auth);
|
|
2480
2585
|
this.recovery.clear();
|
|
2481
|
-
return { done: true, output: this.buildOutput(response?.
|
|
2586
|
+
return { done: true, output: this.buildOutput(response?.runId, response?.projectId, run, overallStatus) };
|
|
2482
2587
|
} catch (error) {
|
|
2483
2588
|
if (error instanceof HttpError && error.status === 401 && !auth) {
|
|
2484
2589
|
this.logAuthRequired(run.options.serverUrl);
|
|
@@ -2505,7 +2610,7 @@ function testLocation(test) {
|
|
|
2505
2610
|
const relativeFilePath = path14.relative(process.cwd(), test.location.file).split(path14.sep).join("/");
|
|
2506
2611
|
return `${relativeFilePath}:${test.location.line}:${test.location.column}`;
|
|
2507
2612
|
}
|
|
2508
|
-
var PiwiDashboardReporter = class {
|
|
2613
|
+
var PiwiDashboardReporter = class _PiwiDashboardReporter {
|
|
2509
2614
|
constructor(rawOptions = {}) {
|
|
2510
2615
|
this.testCases = [];
|
|
2511
2616
|
this.startTime = null;
|
|
@@ -2521,11 +2626,15 @@ var PiwiDashboardReporter = class {
|
|
|
2521
2626
|
this.plannedTests = [];
|
|
2522
2627
|
/** Ids of tests that actually reported via `onTestEnd`, to find the ones that never ran. */
|
|
2523
2628
|
this.reportedTestIds = /* @__PURE__ */ new Set();
|
|
2629
|
+
/** Per-test attempt history, keyed by `test.id`; snapshotted onto every attempt's payload. */
|
|
2630
|
+
this.attemptsByTest = /* @__PURE__ */ new Map();
|
|
2524
2631
|
this.runLabel = null;
|
|
2525
2632
|
this.shardInfo = null;
|
|
2526
2633
|
this.metadata = {};
|
|
2527
2634
|
this.isFullRun = true;
|
|
2528
2635
|
this.filterDetails = null;
|
|
2636
|
+
/** Configured `maxFailures` (0 = unlimited) — disambiguates an interrupted run's unrun reason. */
|
|
2637
|
+
this.maxFailures = 0;
|
|
2529
2638
|
this.streamManager = null;
|
|
2530
2639
|
/** Track suite-level setup steps (beforeAll/afterAll) not tied to any test */
|
|
2531
2640
|
this.setupSteps = [];
|
|
@@ -2573,6 +2682,7 @@ var PiwiDashboardReporter = class {
|
|
|
2573
2682
|
}
|
|
2574
2683
|
this.startTime = (/* @__PURE__ */ new Date()).toISOString();
|
|
2575
2684
|
this.playwrightVersion = config.version;
|
|
2685
|
+
this.maxFailures = config.maxFailures ?? 0;
|
|
2576
2686
|
this.logger.info(
|
|
2577
2687
|
`Starting test run for project: ${this.options.projectName} (Playwright v${this.playwrightVersion})`
|
|
2578
2688
|
);
|
|
@@ -2582,14 +2692,18 @@ var PiwiDashboardReporter = class {
|
|
|
2582
2692
|
const grep = grepRe && grepRe.source !== ".*" ? grepRe.source : void 0;
|
|
2583
2693
|
const grepInvert = grepInvertRe?.source;
|
|
2584
2694
|
const fileFilters = detectCliFileFilters();
|
|
2585
|
-
|
|
2695
|
+
const selection = readSelectionStamp();
|
|
2696
|
+
if (grep || grepInvert || fileFilters.length > 0 || selection) {
|
|
2586
2697
|
this.isFullRun = false;
|
|
2587
2698
|
this.filterDetails = {
|
|
2588
2699
|
...grep ? { grep } : {},
|
|
2589
2700
|
...grepInvert ? { grepInvert } : {},
|
|
2590
|
-
...fileFilters.length > 0 ? { files: fileFilters } : {}
|
|
2701
|
+
...fileFilters.length > 0 ? { files: fileFilters } : {},
|
|
2702
|
+
...selection ? { selection } : {}
|
|
2591
2703
|
};
|
|
2592
|
-
this.logger.info(
|
|
2704
|
+
this.logger.info(
|
|
2705
|
+
selection ? `Selection run detected (${selection.key})` : "Partial run detected (filter active)"
|
|
2706
|
+
);
|
|
2593
2707
|
}
|
|
2594
2708
|
this.metadata = this.metadataCollector.collect(config, suite, this.options);
|
|
2595
2709
|
this.plannedTests = suite.allTests();
|
|
@@ -2626,11 +2740,19 @@ var PiwiDashboardReporter = class {
|
|
|
2626
2740
|
this.streamManager.queueBeginEvent(toWireTestCase(beginEvent));
|
|
2627
2741
|
}
|
|
2628
2742
|
}
|
|
2743
|
+
static {
|
|
2744
|
+
/**
|
|
2745
|
+
* Step categories streamed live while the run executes. `pw:assert` is
|
|
2746
|
+
* excluded: it is the polling noise of `expect()`, not a step a human
|
|
2747
|
+
* watches; the meaningful readout is the `pw:expect` wrapper around it.
|
|
2748
|
+
*/
|
|
2749
|
+
this.LIVE_STEP_CATEGORIES = /* @__PURE__ */ new Set(["hook", "fixture", "pw:api", "pw:expect"]);
|
|
2750
|
+
}
|
|
2629
2751
|
/** Playwright reporter hook: called when a step (including hook/fixture) begins */
|
|
2630
2752
|
onStepBegin(test, _result, step) {
|
|
2631
|
-
if (!this.enabled || !this.streamManager
|
|
2753
|
+
if (!this.enabled || !this.streamManager) return;
|
|
2632
2754
|
const cat = step.category;
|
|
2633
|
-
if (cat
|
|
2755
|
+
if (!_PiwiDashboardReporter.LIVE_STEP_CATEGORIES.has(cat)) return;
|
|
2634
2756
|
const event = {
|
|
2635
2757
|
type: "step-begin",
|
|
2636
2758
|
title: step.title,
|
|
@@ -2644,10 +2766,9 @@ var PiwiDashboardReporter = class {
|
|
|
2644
2766
|
}
|
|
2645
2767
|
/** Playwright reporter hook: called when a step (including hook/fixture) ends */
|
|
2646
2768
|
onStepEnd(test, _result, step) {
|
|
2769
|
+
if (!this.enabled || !this.streamManager) return;
|
|
2647
2770
|
const cat = step.category;
|
|
2648
|
-
if (cat
|
|
2649
|
-
if (!this.enabled || !this.streamManager?.enabled) return;
|
|
2650
|
-
if (cat !== "hook" && cat !== "fixture") return;
|
|
2771
|
+
if (!_PiwiDashboardReporter.LIVE_STEP_CATEGORIES.has(cat)) return;
|
|
2651
2772
|
const workerIndex = workerIndexOf(_result);
|
|
2652
2773
|
const startedAt = step.startTime instanceof Date ? step.startTime.getTime() : null;
|
|
2653
2774
|
const event = {
|
|
@@ -2662,7 +2783,7 @@ var PiwiDashboardReporter = class {
|
|
|
2662
2783
|
startedAt
|
|
2663
2784
|
};
|
|
2664
2785
|
this.streamManager?.queueEvent(event);
|
|
2665
|
-
if (!test && startedAt) {
|
|
2786
|
+
if (!test && startedAt && (cat === "hook" || cat === "fixture")) {
|
|
2666
2787
|
this.setupSteps.push({
|
|
2667
2788
|
title: step.title,
|
|
2668
2789
|
category: cat,
|
|
@@ -2682,6 +2803,14 @@ var PiwiDashboardReporter = class {
|
|
|
2682
2803
|
const annotations = mergeAnnotations(test, result);
|
|
2683
2804
|
const status = classifyStatus(result.status, annotations);
|
|
2684
2805
|
const tags = collectTestTags(test);
|
|
2806
|
+
const attempts = this.attemptsByTest.get(test.id) ?? [];
|
|
2807
|
+
attempts.push({
|
|
2808
|
+
retry: result.retry,
|
|
2809
|
+
status,
|
|
2810
|
+
duration: result.duration,
|
|
2811
|
+
startedAt: result.startTime ? result.startTime.getTime() : null
|
|
2812
|
+
});
|
|
2813
|
+
this.attemptsByTest.set(test.id, attempts);
|
|
2685
2814
|
const testCase = {
|
|
2686
2815
|
type: "complete",
|
|
2687
2816
|
title: test.title,
|
|
@@ -2693,6 +2822,7 @@ var PiwiDashboardReporter = class {
|
|
|
2693
2822
|
timeout: test.timeout ?? null,
|
|
2694
2823
|
error: buildErrorText(result),
|
|
2695
2824
|
retries: result.retry,
|
|
2825
|
+
attempts: attempts.map((a) => ({ ...a })),
|
|
2696
2826
|
workerIndex: workerIndexOf(result),
|
|
2697
2827
|
shardIndex: this.shardInfo?.current ?? null,
|
|
2698
2828
|
startedAt: result.startTime ? result.startTime.getTime() : null,
|
|
@@ -2702,7 +2832,10 @@ var PiwiDashboardReporter = class {
|
|
|
2702
2832
|
suiteConfig,
|
|
2703
2833
|
testAnnotations: annotations.length ? annotations : null,
|
|
2704
2834
|
tags: tags.length ? tags : null,
|
|
2705
|
-
testMeta: collectTestMetadata(annotations)
|
|
2835
|
+
testMeta: collectTestMetadata(annotations),
|
|
2836
|
+
// An annotation-less skip reclassified to `didnotrun` is a serial-group
|
|
2837
|
+
// cascade: an earlier test failed and Playwright skipped the rest.
|
|
2838
|
+
didNotRunReason: status === "didnotrun" ? "previous-failure" : null
|
|
2706
2839
|
};
|
|
2707
2840
|
if (result.status === "failed" || result.status === "timedOut") {
|
|
2708
2841
|
const failingLine = extractFailingLine(testCase.error, test.location.file, test.location.line);
|
|
@@ -2746,6 +2879,7 @@ var PiwiDashboardReporter = class {
|
|
|
2746
2879
|
break;
|
|
2747
2880
|
}
|
|
2748
2881
|
this.testCases.push(testCase);
|
|
2882
|
+
if (status === "didnotrun") linkBlockedTests(this.testCases);
|
|
2749
2883
|
if (this.streamManager) {
|
|
2750
2884
|
this.streamManager.queueEvent(toWireTestCase(testCase));
|
|
2751
2885
|
if (this.options.liveFileUploads) this.streamManager.scheduleLiveUpload(testCase);
|
|
@@ -2756,9 +2890,10 @@ var PiwiDashboardReporter = class {
|
|
|
2756
2890
|
* (no `onTestEnd`) — typically because `maxFailures` cut the run short. These
|
|
2757
2891
|
* carry no result, so they're emitted with zero duration and no error. In
|
|
2758
2892
|
* streaming mode they're queued as complete events so the pre-finish drain
|
|
2759
|
-
* sends them alongside the rest.
|
|
2893
|
+
* sends them alongside the rest. The `reason` (global timeout / max failures /
|
|
2894
|
+
* interrupted) is resolved once from the overall run status by the caller.
|
|
2760
2895
|
*/
|
|
2761
|
-
materializeUnrunTests() {
|
|
2896
|
+
materializeUnrunTests(reason) {
|
|
2762
2897
|
for (const test of this.plannedTests) {
|
|
2763
2898
|
if (this.reportedTestIds.has(test.id)) continue;
|
|
2764
2899
|
const { suitePath, suiteConfig } = this.metadataCollector.getSuiteInfo(test);
|
|
@@ -2782,7 +2917,8 @@ var PiwiDashboardReporter = class {
|
|
|
2782
2917
|
suiteConfig,
|
|
2783
2918
|
testAnnotations: declaredAnnotations.length ? declaredAnnotations : null,
|
|
2784
2919
|
tags: tags.length ? tags : null,
|
|
2785
|
-
testMeta: collectTestMetadata(declaredAnnotations)
|
|
2920
|
+
testMeta: collectTestMetadata(declaredAnnotations),
|
|
2921
|
+
didNotRunReason: reason
|
|
2786
2922
|
};
|
|
2787
2923
|
this.testCases.push(testCase);
|
|
2788
2924
|
this.totalTests++;
|
|
@@ -2795,7 +2931,11 @@ var PiwiDashboardReporter = class {
|
|
|
2795
2931
|
/** Playwright reporter hook: called when the full test run finishes */
|
|
2796
2932
|
async onEnd(result) {
|
|
2797
2933
|
if (!this.enabled) return;
|
|
2798
|
-
|
|
2934
|
+
const unrunReason = resolveUnrunReason(result?.status, {
|
|
2935
|
+
maxFailures: this.maxFailures,
|
|
2936
|
+
failures: this.failedTests + this.timedOutTests
|
|
2937
|
+
});
|
|
2938
|
+
this.materializeUnrunTests(unrunReason);
|
|
2799
2939
|
await this.submitter.submit(
|
|
2800
2940
|
{
|
|
2801
2941
|
options: this.options,
|
|
@@ -2821,6 +2961,79 @@ var PiwiDashboardReporter = class {
|
|
|
2821
2961
|
}
|
|
2822
2962
|
};
|
|
2823
2963
|
|
|
2964
|
+
// src/internal/support/selection-client.ts
|
|
2965
|
+
function authHeaders(apiKey) {
|
|
2966
|
+
return apiKey ? { "X-API-Key": apiKey } : {};
|
|
2967
|
+
}
|
|
2968
|
+
async function resolveProjectId(options) {
|
|
2969
|
+
if (!options.project) throw new Error("No project \u2014 pass --project or set PIWI_PROJECT_NAME");
|
|
2970
|
+
if (/^\d+$/.test(options.project)) return Number(options.project);
|
|
2971
|
+
const res = await fetch(`${options.serverUrl}/api/projects/menu`, { headers: authHeaders(options.apiKey) });
|
|
2972
|
+
if (!res.ok) throw new Error(`Could not list projects (dashboard returned ${res.status})`);
|
|
2973
|
+
const body = await res.json();
|
|
2974
|
+
const match = body.items?.find((p) => p.name.toLowerCase() === options.project.toLowerCase());
|
|
2975
|
+
if (!match) throw new Error(`No project named "${options.project}" on this dashboard`);
|
|
2976
|
+
return match.id;
|
|
2977
|
+
}
|
|
2978
|
+
async function fetchResolution(options, projectId) {
|
|
2979
|
+
const params = new URLSearchParams({ format: options.format ?? "args" });
|
|
2980
|
+
if (options.pkgRunner) params.set("pkgRunner", options.pkgRunner);
|
|
2981
|
+
if (options.budgetMs != null) params.set("budgetMs", String(options.budgetMs));
|
|
2982
|
+
if (options.shard) params.set("shard", options.shard);
|
|
2983
|
+
if (options.order) params.set("order", options.order);
|
|
2984
|
+
const url = `${options.serverUrl}/api/projects/${projectId}/selections/${encodeURIComponent(options.key)}/resolve?${params}`;
|
|
2985
|
+
const res = await fetch(url, { headers: authHeaders(options.apiKey) });
|
|
2986
|
+
if (!res.ok) {
|
|
2987
|
+
const body = await res.json().catch(() => ({}));
|
|
2988
|
+
throw new Error(body.message || `Dashboard returned ${res.status} resolving "${options.key}"`);
|
|
2989
|
+
}
|
|
2990
|
+
return await res.json();
|
|
2991
|
+
}
|
|
2992
|
+
|
|
2993
|
+
// src/public/selection.ts
|
|
2994
|
+
async function resolveSelection(options = {}) {
|
|
2995
|
+
const env = options.env ?? process.env;
|
|
2996
|
+
const key = options.key ?? env[PIWI_SELECTION_ENV.key];
|
|
2997
|
+
if (!key) return void 0;
|
|
2998
|
+
const serverUrl = (options.serverUrl ?? env[PIWI_ENV_KEYS.serverUrl] ?? "").replace(/\/$/, "");
|
|
2999
|
+
const apiKey = options.apiKey ?? env[PIWI_ENV_KEYS.apiKey] ?? null;
|
|
3000
|
+
const project = options.project ?? env[PIWI_ENV_KEYS.projectName] ?? "";
|
|
3001
|
+
if (!serverUrl) {
|
|
3002
|
+
if (options.strict) throw new Error("resolveSelection: no dashboard URL (set PIWI_DASHBOARD_URL)");
|
|
3003
|
+
return void 0;
|
|
3004
|
+
}
|
|
3005
|
+
const clientOptions = { serverUrl, apiKey, project, key, format: "grep", budgetMs: options.budgetMs ?? null };
|
|
3006
|
+
try {
|
|
3007
|
+
const projectId = await resolveProjectId(clientOptions);
|
|
3008
|
+
const resolution = await fetchResolution(clientOptions, projectId);
|
|
3009
|
+
if (resolution.estimate.count === 0) {
|
|
3010
|
+
if (options.strict) throw new Error(`resolveSelection: "${key}" resolved to 0 tests`);
|
|
3011
|
+
console.warn(`[Piwi Dashboard] selection "${key}" resolved to 0 tests \u2014 running the full suite`);
|
|
3012
|
+
return void 0;
|
|
3013
|
+
}
|
|
3014
|
+
env[PIWI_SELECTION_ENV.key] = resolution.key ?? key;
|
|
3015
|
+
env[PIWI_SELECTION_ENV.version] = String(resolution.version ?? 0);
|
|
3016
|
+
env[PIWI_SELECTION_ENV.hash] = resolution.resolvedHash;
|
|
3017
|
+
env[PIWI_SELECTION_ENV.count] = String(resolution.estimate.count);
|
|
3018
|
+
const args = resolution.materialization.args;
|
|
3019
|
+
const result = {
|
|
3020
|
+
key: resolution.key ?? key,
|
|
3021
|
+
version: resolution.version ?? 0,
|
|
3022
|
+
resolvedHash: resolution.resolvedHash,
|
|
3023
|
+
resolvedCount: resolution.estimate.count
|
|
3024
|
+
};
|
|
3025
|
+
if (args[0] === "--grep") result.grep = args[1];
|
|
3026
|
+
else result.files = args;
|
|
3027
|
+
return result;
|
|
3028
|
+
} catch (e) {
|
|
3029
|
+
if (options.strict) throw e;
|
|
3030
|
+
console.warn(
|
|
3031
|
+
`[Piwi Dashboard] selection "${key}" could not be resolved \u2014 running the full suite (${e.message})`
|
|
3032
|
+
);
|
|
3033
|
+
return void 0;
|
|
3034
|
+
}
|
|
3035
|
+
}
|
|
3036
|
+
|
|
2824
3037
|
// src/internal/capture/capture-fixtures.ts
|
|
2825
3038
|
var import_node_zlib = require("zlib");
|
|
2826
3039
|
|
|
@@ -6362,5 +6575,6 @@ var index_default = PiwiDashboardReporter;
|
|
|
6362
6575
|
extendPiwiFixtures,
|
|
6363
6576
|
piwiAiFixtures,
|
|
6364
6577
|
piwiFixtures,
|
|
6578
|
+
resolveSelection,
|
|
6365
6579
|
wrapConfig
|
|
6366
6580
|
});
|
package/package.json
CHANGED
|
@@ -1,9 +1,9 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@piwitests/reporter",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.26.1",
|
|
4
4
|
"description": "Playwright reporter that streams results, traces and HTML reports to a Piwi Dashboard instance",
|
|
5
5
|
"url": "https://github.com/PiwiTests/platform",
|
|
6
|
-
"homepage": "https://piwitests.
|
|
6
|
+
"homepage": "https://piwitests.dev",
|
|
7
7
|
"repository": {
|
|
8
8
|
"type": "git",
|
|
9
9
|
"url": "https://github.com/PiwiTests/platform"
|
|
@@ -5,7 +5,7 @@ description: Investigate a failed test run recorded in Piwi Dashboard and propos
|
|
|
5
5
|
|
|
6
6
|
# Investigate a Piwi failure
|
|
7
7
|
|
|
8
|
-
Turn a failed run in [Piwi Dashboard](https://piwitests.
|
|
8
|
+
Turn a failed run in [Piwi Dashboard](https://piwitests.dev) into a grounded diagnosis and a concrete fix. Piwi has already gathered the evidence — the error text, the steps that ran, console output, failing network calls, a suggested locator, and the source diff since the last passing run. Use that instead of guessing.
|
|
9
9
|
|
|
10
10
|
## How you reach Piwi
|
|
11
11
|
|
|
@@ -0,0 +1,38 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: run-the-right-tests
|
|
3
|
+
description: Pick and run the right subset of Playwright tests using Piwi's data-driven selections, instead of always running the whole suite. Use when the user asks to "run smoke tests", "run the right tests", "just run what's relevant", "verify this fix", or wants a fast, targeted test loop rather than the full run.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Run the right tests with Piwi selections
|
|
7
|
+
|
|
8
|
+
Running the whole suite for every change is slow, and hand-maintained grep patterns go stale. A Piwi **selection** is a named, declarative subset of the suite that Piwi resolves from run history — smoke, critical-and-stable, recently-broken, the best five minutes. This skill picks the selection that fits the task and runs exactly those tests.
|
|
9
|
+
|
|
10
|
+
## How you reach Piwi
|
|
11
|
+
|
|
12
|
+
Prefer the **Piwi MCP server** if it is connected (`list_selections`, `resolve_selection`, `preview_selection`). Otherwise use the reporter CLI (`npx @piwitests/reporter select|run`) with `PIWI_DASHBOARD_URL` / `PIWI_API_KEY` / `PIWI_PROJECT_NAME` set, or the dashboard's **Selections** tab.
|
|
13
|
+
|
|
14
|
+
## Steps
|
|
15
|
+
|
|
16
|
+
1. **See what selections exist.** Call `list_selections` for the project. Every project has the built-ins `failed` (tests whose latest run failed) and `quarantine-free`, plus any the team saved (often `smoke`). Pick the one that matches the intent:
|
|
17
|
+
- About to make a broad change, or want a quick confidence check → `smoke` (or `preview_selection` with `{ "include": [{ "tags": ["smoke"] }] }`).
|
|
18
|
+
- Iterating on a fix → `failed`, or a `{ "include": [{ "failedInLastRuns": 5 }] }` preview.
|
|
19
|
+
- Time-boxed → resolve any selection with a `budgetMs` (e.g. 300000 for five minutes).
|
|
20
|
+
|
|
21
|
+
2. **Resolve it to a command.** Call `resolve_selection` with the key (and optional `budgetMs`). You get back the matching tests, an estimate, and a ready-to-run `playwright test` command. Check the count and warnings first — a selection that matches **zero** tests is a red flag (too narrow, or nothing qualifies), and a `quarantined-included` warning means a flaky test is in the set.
|
|
22
|
+
|
|
23
|
+
3. **Run it.** The most reliable way is the reporter CLI, which resolves and runs in one step and stamps the run so the dashboard names the subset:
|
|
24
|
+
```
|
|
25
|
+
npx @piwitests/reporter run <key>
|
|
26
|
+
```
|
|
27
|
+
Or run the materialized command `resolve_selection` returned directly with `playwright test`. Pass extra Playwright args after `--` (e.g. `-- --workers=4`).
|
|
28
|
+
|
|
29
|
+
4. **After a fix, verify with the same selection.** Re-run the selection that covered the failure and confirm it now passes. In CI, `npx @piwitests/reporter gate --require-selection <key>` fails the build if any test the selection currently matches did not run or did not pass — catching a smoke job that silently shrank.
|
|
30
|
+
|
|
31
|
+
5. **Report.** State which selection you ran, how many tests it resolved to, and the outcome. If no saved selection fit, suggest one (a `preview_selection` definition the team could save) rather than falling back to the whole suite silently.
|
|
32
|
+
|
|
33
|
+
## Guardrails
|
|
34
|
+
|
|
35
|
+
- A selection that resolves to **zero** tests is never "nothing to do" — it means the definition is wrong. Stop and say so.
|
|
36
|
+
- Selections choose what to *run*; they never hide a failure. Quarantine is the tool for a test whose verdict should not block a merge — do not use a selection to route around a red test.
|
|
37
|
+
- Don't invent a selection key. Use `list_selections` to see the real ones, and `preview_selection` for an ad-hoc subset.
|
|
38
|
+
- The full suite stays the baseline. Selections are for the fast loops between full runs, not a permanent replacement.
|
|
@@ -5,7 +5,7 @@ description: Wire a Playwright project up to a Piwi Dashboard — install the re
|
|
|
5
5
|
|
|
6
6
|
# Set up Piwi in a Playwright project
|
|
7
7
|
|
|
8
|
-
Connect a Playwright test suite to a [Piwi Dashboard](https://piwitests.
|
|
8
|
+
Connect a Playwright test suite to a [Piwi Dashboard](https://piwitests.dev) so every run is uploaded, kept, and analyzed. The mechanical work is done by a deterministic command; your job is to gather the right inputs, run it, finish anything it flags, and prove a run reaches the dashboard.
|
|
9
9
|
|
|
10
10
|
## Before you start
|
|
11
11
|
|