flake-ledger 0.1.0 → 0.1.1

package/README.md

@@ -1,6 +1,7 @@
 # flake-ledger
 
 [![CI](https://github.com/WJDdidi-1/flake-ledger/actions/workflows/ci.yml/badge.svg)](https://github.com/WJDdidi-1/flake-ledger/actions/workflows/ci.yml)
+[![version](https://img.shields.io/github/package-json/v/WJDdidi-1/flake-ledger)](https://github.com/WJDdidi-1/flake-ledger/releases)
 
 Ranks flaky tests from JUnit XML across CI runs, including passed-on-retry.
 
@@ -114,6 +115,61 @@ com.example.CheckoutTest::appliesCoupon
 `--fail-on` and `--min-runs` are opt-in; without them the command is
 report-only and exits 0.
 
+## Measured on real test runners
+
+The checked-in example is hand-written. To check the parser against output it
+did not create, the same CLI was run against JUnit XML produced by pytest
+9.2.0 with pytest-rerunfailures 16.7 and by Playwright 1.65.0. Each fixture had
+ten runs, one directory per run; the test files, configs, and commands are in
+[`docs/real-world.md`](https://github.com/WJDdidi-1/flake-ledger/blob/main/docs/real-world.md).
+
+pytest-rerunfailures writes one `<testcase>` per attempt, so a test that failed
+once and then passed on retry appears as two passing testcases. From the
+fixture directory, `flake-ledger scan runs` printed:
+
+```console
+$ flake-ledger scan runs
+TEST ID                              RUNS  ATTEMPTS  UNSTABLE  PASS-ON-RETRY  FAILED  RATE    SCORE  CLASS
+test_checkout::test_always_failing   10    20        10        0              10      100.0%  0.722  failed-on-retry
+test_checkout::test_applies_coupon   10    20        10        10             0       100.0%  0.722  passed-on-retry
+test_checkout::test_cross_run_flake  10    10        5         0              5       50.0%   0.237  cross-run flake
+test_checkout::test_skipped          10    10        0         0              0       0.0%    0.000  skipped
+test_checkout::test_stable           10    10        0         0              0       0.0%    0.000  stable
+
+10 runs, 5 tests, 70 attempts, 10 XML files.
+```
+
+The recovered test is `passed-on-retry` in all 10 runs, the alternating test is
+`cross-run flake` in 5 of 10, and the stable and skipped tests stay in their
+classes.
+
+Playwright's JUnit reporter writes `<flakyError>`/`<rerunError>` only when
+`includeRetries: true` is set. With that option, two projects (chromium and
+firefox) and ten runs produced:
+
+```console
+$ flake-ledger scan runs
+TEST ID                                      RUNS  ATTEMPTS  UNSTABLE  PASS-ON-RETRY  FAILED  RATE    SCORE  CLASS
+checkout.spec.js::failures › always failing  10    40        10        0              10      100.0%  0.722  always-failing
+checkout.spec.js::checkout › applies coupon  10    40        10        10             0       100.0%  0.722  passed-on-retry
+checkout.spec.js::cross-run flake            10    20        5         0              5       50.0%   0.237  cross-run flake
+checkout.spec.js::skipped                    10    20        0         0              0       0.0%    0.000  skipped
+checkout.spec.js::stable                     10    20        0         0              0       0.0%    0.000  stable
+
+10 runs, 5 tests, 140 attempts, 10 XML files.
+```
+
+The recovered test is `passed-on-retry` in all 10 runs. `stable` has 20 attempts
+because both projects run it, but it is still `stable`: duplicate executions in
+separate Playwright project suites are not retries.
+
+**Playwright caveat:** with the default JUnit reporter, Playwright writes only
+the final attempt for each test, so the retry is not in the XML at all. With the
+same ten runs, the row for `checkout.spec.js::checkout › applies coupon` had 20
+attempts, 0 passed-on-retry runs, and class `stable`. `flake-ledger` cannot
+recover that retry from the file; set `includeRetries: true` (or
+`PLAYWRIGHT_JUNIT_INCLUDE_RETRIES=1`) if you want retry evidence in the ledger.
+
 ## Who it is for
 
 Maintainers of multi-runner CI suites (pytest, Playwright, xUnit,
@@ -128,13 +184,6 @@ free, offline, cross-runner ledger plus a quarantine file.
 
 ## Install
 
-From npm:
-
-```sh
-npm install --global flake-ledger
-flake-ledger --version
-```
-
 From a checkout:
 
 ```sh
@@ -144,6 +193,14 @@ npm install --global .
 flake-ledger --version
 ```
 
+The npm package is published by the release workflow when a `v*` tag is
+pushed. The latest release is `v0.1.1`:
+
+```sh
+npm install --global flake-ledger@0.1.1
+flake-ledger --version   # 0.1.1
+```
+
 To try it without a global install, run the checked-in five-run example:
 
 ```sh
@@ -226,6 +283,13 @@ Supported inputs are Surefire and compatible JUnit XML, including
 attributes, and repeated `<testcase>` elements for the same test. A malformed
 XML file produces a warning and does not hide healthy files.
 
+Real runners differ in how much of that they emit. pytest-rerunfailures
+recovers a test by writing the passing `<testcase>` again, so repeated passing
+testcases in one suite count as retry evidence. Playwright writes
+`<flakyError>`/`<rerunError>` only with `includeRetries: true`; without it its
+JUnit XML contains only the final attempt. The measured output from both
+runners is in [`docs/real-world.md`](https://github.com/WJDdidi-1/flake-ledger/blob/main/docs/real-world.md).
+
 ## What it reports
 
 `scan` collapses all of those attempts into one record per `classname::name`
@@ -316,6 +380,10 @@ readable path through the same CLI: `npm run example` runs `scan`, `explain`,
 `--quarantine`, and the CI gate against it, and exits nonzero if the table,
 timeline, quarantine list, or exit codes change.
 
+The real-runner measurement in [`docs/real-world.md`](https://github.com/WJDdidi-1/flake-ledger/blob/main/docs/real-world.md)
+records the exact commands and output from pytest-rerunfailures and Playwright,
+including the Playwright `includeRetries` caveat.
+
 CI runs the suite on Ubuntu, macOS (Intel), and Windows for Node.js 18, 20, 22,
 and 24. Each job also packs the tarball, installs it into a clean directory,
 and runs the installed binary, so the published package is tested rather than
@@ -328,6 +396,20 @@ npm test
 node bin/flake-ledger.mjs scan path/to/runs
 ```
 
+## Releases
+
+Versions are SemVer tags (`vMAJOR.MINOR.PATCH`) and must match `version` in
+`package.json`. Pushing a `v*` tag runs the publish workflow, which tests,
+packs the tarball, installs it into a clean directory, and publishes it to npm
+with provenance. The latest release is `v0.1.1`, matching `version` `0.1.1` in
+`package.json`.
+
+The changelog for each version lives in [`CHANGELOG.md`](CHANGELOG.md), the
+per-release GitHub Release bodies are in
+[`docs/releases`](https://github.com/WJDdidi-1/flake-ledger/tree/main/docs/releases),
+and the maintainer steps are in
+[`docs/RELEASING.md`](https://github.com/WJDdidi-1/flake-ledger/blob/main/docs/RELEASING.md).
+
 ## Status
 
 Early MVP. The features above are the scope; anything not listed is not built

package/lib/junit.mjs

@@ -241,7 +241,7 @@ function testcaseIdentity(node, suiteName) {
   };
 }
 
-function addTestcase(node, suiteName, sourcePath, output) {
+function addTestcase(node, suiteName, sourcePath, output, suiteKey) {
   const identity = testcaseIdentity(node, suiteName);
   const attempts = attemptsForTestcase(node);
   output.testcaseCount += 1;
@@ -251,6 +251,7 @@ function addTestcase(node, suiteName, sourcePath, output) {
       ...attempt,
       source: sourcePath,
       suite: suiteName,
+      suiteKey,
     });
   }
 }
@@ -264,13 +265,25 @@ function visitContainer(node, context, output) {
     } else if (tag === 'testsuites') {
       visitContainer(child, context, output);
     } else if (tag === 'testcase') {
-      addTestcase(child, context.suiteName, context.sourcePath, output);
+      addTestcase(
+        child,
+        context.suiteName,
+        context.sourcePath,
+        output,
+        context.suiteKey || `${context.sourcePath}#root`,
+      );
     }
   }
 }
 
 function visitSuite(node, context, output) {
   const ownName = getAttribute(node, 'name') || context.suiteName || null;
+  // A report can contain several <testsuite> elements with the same name:
+  // Playwright, for example, writes one per project (hostname). The key keeps
+  // repeated testcases from one suite together while keeping duplicate runs of
+  // the same test in different projects apart.
+  const suiteKey = `${context.sourcePath}#suite-${context.suiteCounter.value}`;
+  context.suiteCounter.value += 1;
   output.suites.push(suiteMetadata(node, context.sourcePath));
   const timestamp = getAttribute(node, 'timestamp');
   if (timestamp) {
@@ -285,11 +298,11 @@ function visitSuite(node, context, output) {
     if (!isElement(child)) continue;
     const tag = lower(localName(child.name));
     if (tag === 'testsuite') {
-      visitSuite(child, { ...context, suiteName: ownName }, output);
+      visitSuite(child, { ...context, suiteName: ownName, suiteKey }, output);
     } else if (tag === 'testsuites') {
-      visitContainer(child, { ...context, suiteName: ownName }, output);
+      visitContainer(child, { ...context, suiteName: ownName, suiteKey }, output);
     } else if (tag === 'testcase') {
-      addTestcase(child, ownName, context.sourcePath, output);
+      addTestcase(child, ownName, context.sourcePath, output, suiteKey);
     }
   }
 }
@@ -311,16 +324,17 @@ export function parseJUnit(source, sourcePath = '<junit.xml>') {
     timestamp: null,
     timestampMs: null,
   };
+  const suiteCounter = { value: 0 };
 
   for (const root of elementChildren(document)) {
     if (!isElement(root)) continue;
     const tag = lower(localName(root.name));
     if (tag === 'testsuite') {
-      visitSuite(root, { suiteName: null, sourcePath }, output);
+      visitSuite(root, { suiteName: null, sourcePath, suiteCounter }, output);
     } else if (tag === 'testsuites') {
-      visitContainer(root, { suiteName: null, sourcePath }, output);
+      visitContainer(root, { suiteName: null, sourcePath, suiteCounter }, output);
     } else if (tag === 'testcase') {
-      addTestcase(root, null, sourcePath, output);
+      addTestcase(root, null, sourcePath, output, `${sourcePath}#root`);
     }
   }
 

package/lib/scan.mjs

@@ -207,6 +207,24 @@ function statusesOf(attempts) {
   return attempts.map((attempt) => attempt.status);
 }
 
+function attemptGroupKey(attempt) {
+  // Newer scanners attach suiteKey, which distinguishes separate <testsuite>
+  // elements (Playwright projects) from repeated testcases in one suite.
+  // Fall back to the source file so older callers with plain attempt objects
+  // still merge duplicates from the same report.
+  return attempt.suiteKey || attempt.source || '';
+}
+
+function hasRepeatedAttemptInSameSuite(attempts) {
+  const seen = new Set();
+  for (const attempt of attempts) {
+    const key = attemptGroupKey(attempt);
+    if (seen.has(key)) return true;
+    seen.add(key);
+  }
+  return false;
+}
+
 function outcomeForAttempts(attempts) {
   const statuses = statusesOf(attempts);
   const failures = statuses.filter((status) => status === 'failed' || status === 'error');
@@ -219,7 +237,15 @@ function outcomeForAttempts(attempts) {
   }
   if (failures.length > 0) return failures[failures.length - 1];
   if (attempts.some((attempt) => attempt.retried)) return 'passed-on-retry';
-  if (passes.length > 0) return 'passed';
+  if (passes.length > 0) {
+    // pytest-rerunfailures writes one <testcase> per attempt but does not put
+    // the earlier failure in the XML: a recovered test appears as the same
+    // passing testcase twice. Repeated passes in one suite are therefore retry
+    // evidence. The same test in two different Playwright project suites is a
+    // duplicate execution, not a retry, so only repeats within one suite count.
+    if (hasRepeatedAttemptInSameSuite(attempts)) return 'passed-on-retry';
+    return 'passed';
+  }
   return statuses.length > 0 ? 'skipped' : 'unknown';
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "flake-ledger",
-  "version": "0.1.0",
+  "version": "0.1.1",
   "description": "Ranks flaky tests from JUnit XML across CI runs, including passed-on-retry.",
   "license": "Apache-2.0",
   "type": "module",