@gfdelarue/testkit 0.6.3 → 0.6.5

package/docs/release-notes.md

@@ -1,3 +1,98 @@
+# 0.6.5
+
+## `testkit bug bisect` finds the commit that broke a recorded test
+
+`testkit bug bisect <id>` runs a recorded bug's regression test at older
+commits, each in a temporary copy of the checkout with the test added, and
+finds the last commit where it passes and the first where it fails. It checks
+the commit that `git blame` names first, so a correct blame costs two runs,
+and searches the first-parent line when the blame is wrong, for example when a
+later commit only reformatted the faulty line. Every commit it runs the
+test at, the checked-out one included, gets its own dependencies from a
+frozen install of its lockfiles, with the checkout's ignored `.env` files and
+package manager settings in place. The draft keeps both commits,
+and `testkit bug file` adds them to the issue and to the facts block as
+`bisect.good`, `bisect.bad`, and `bisect.exact`, with the list of commits
+it ran the test at. Readers that don't know `bisect` ignore it, as
+`docs/bug-records.md` allows. See
+[Find the commit that broke it](../README.md#find-the-commit-that-broke-it).
+
+## `testkit shake` finds the condition that makes a test flaky
+
+`testkit shake --file <test>` reruns test files under conditions that make
+flaky tests fail: a seeded random order, copies of each file running at the same time
+on the same services and database, a time zone 14 hours ahead of UTC and one
+11 hours behind, and every CPU core busy. Each condition runs `--runs` times
+beside a control. Fisher's exact test against the control gives each condition
+a verdict such as `reproduces` or `raises failures`, with p corrected for the
+number of tests compared and a 95% Wilson interval for each failure rate. The
+report gives a command that reproduces each failure, and Testkit runs it once
+more to say whether the failure came back. The run bundles of failed runs stay
+under `.testkit/shake/<id>/runs/`. `--since <ref>` shakes the test files
+changed since the ref, so a job can check new tests before they merge. See
+[Shake out flaky tests](../README.md#shake-out-flaky-tests).
+
+`testkit run` takes `--vitest-shuffle <seed>`, which shuffles plain Vitest
+files and their tests in the order that seed gives, and `--timezone <zone>`,
+which sets `TZ` for the tests, the services Testkit starts, and a `ui` test's
+browser. `--retries <n>` overrides the retries that the Vitest and Playwright
+configs set, including in each project. `testkit bug record` takes all three,
+and `testkit bug file` and the issue's reproduce command carry them. Testkit now passes `TZ` from its own
+environment to the test processes it starts, as it passes `LANG` and `LC_ALL`.
+
+## `testkit prove` checks that the tests notice a change
+
+`testkit prove --since <ref>` mutates the product code changed since the ref,
+one place at a time, and runs the selected tests against each mutant in a
+temporary copy of the checkout. Testkit starts the services for every run, so
+an HTTP test of a service or a `ui` test of the page a service sends can catch
+a mutant. Every mutated place records when a test process, a service, or a
+browser page runs it, so a mutant no test noticed reads as `survived` only when
+its code ran, and as `not-reached` when nothing ran it, such as code a service
+serves from an old build. `--mutants` runs mutants an agent wrote, and
+`--fail-under` fails a job below a mutation score. See
+[Prove the tests catch a change](../README.md#prove-the-tests-catch-a-change).
+
+# 0.6.4
+
+## Testkit supports Node 26 LTS
+
+Testkit runs on Node 26 from 26.13.0, the first Node 26 LTS release. Before
+this release, `testkit` exited with `TESTKIT_HOST_NODE_UNSUPPORTED` on any
+Node 26, which Homebrew, Arch, and Scoop now install.
+`engines.node` is now `>=22.15.0 <23.0.0 || >=24.14.1 <25.0.0 || >=26.13.0 <27.0.0`.
+Support for Node 22 and 24 is unchanged.
+
+When an engine process exited before it wrote its report, Testkit took the
+failure message from the first line of its output. On Node 26, Playwright
+1.59.1 printed a `DEP0205` deprecation warning first, so Testkit reported the
+warning instead of the error. Testkit now skips Node's warning lines.
+
+## Testkit runs Playwright 1.64.1
+
+Testkit's Playwright moves from 1.59.1 to 1.64.1, and its Chromium from 147 to 156. Run `testkit setup playwright` once after you upgrade. Until you do, a ui
+task stops with `TESTKIT_PLAYWRIGHT_INSTALLATION_INVALID`, because Chromium 156
+isn't installed yet.
+
+Playwright 1.59.1 couldn't unpack Chromium on Node 26, so
+`testkit setup playwright` hung there until its 20-minute limit. Playwright
+1.64.1 installs it on Node 26 as it does on Node 22 and 24.
+
+Playwright 1.60 to 1.64 change some behavior that ui tests may rely on:
+
+- Device descriptors now set `screen`, so `window.screen` and media queries
+  see the emulated size.
+- Actions, `isVisible()`, and `toBeVisible()` treat elements inside hidden
+  iframes as hidden.
+- In headless mode, the clipboard is separate from the host's clipboard.
+- `locator.ariaRef()`, the `handle` option of `exposeBinding`, and the
+  `videosPath` and `videoSize` context options are gone.
+- `testkit setup playwright --with-deps` no longer supports Ubuntu 20.04 or
+  Debian 11.
+
+[Playwright's release notes](https://playwright.dev/docs/release-notes) list
+every change.
+
 # 0.6.3
 
 ## A global Testkit runs the product's own install

package/package.json

@@ -1,7 +1,22 @@
 {
   "name": "@gfdelarue/testkit",
-  "version": "0.6.3",
-  "description": "Test orchestration platform for product acceptance and QA workflows",
+  "version": "0.6.5",
+  "description": "Test runner and CLI for coding agents and CI. Your agent hunts bugs, and Testkit files each one in GitHub or Gitea with a failing test.",
+  "keywords": [
+    "testing",
+    "test-runner",
+    "cli",
+    "ai",
+    "coding-agents",
+    "bug-hunting",
+    "regression-tests",
+    "vitest",
+    "playwright",
+    "ci",
+    "github",
+    "gitea"
+  ],
+  "homepage": "https://turnstone.cloud",
   "license": "SEE LICENSE IN LICENSE",
   "type": "module",
   "repository": {
@@ -96,7 +111,7 @@
   ],
   "devDependencies": {
     "@eslint/js": "^10.0.1",
-    "@homebridge/node-pty-prebuilt-multiarch": "^0.13.1",
+    "@homebridge/node-pty-prebuilt-multiarch": "^0.15.0",
     "@kitschpatrol/tokei": "2.0.2",
     "@moontern-labs/code-budget": "0.1.3",
     "@types/picomatch": "4.0.3",
@@ -113,7 +128,7 @@
     "verdaccio": "6.10.3"
   },
   "dependencies": {
-    "@playwright/test": "1.59.1",
+    "@playwright/test": "1.64.1",
     "@types/node": "22.20.1",
     "@typescript-eslint/parser": "8.65.0",
     "@typescript-eslint/types": "8.65.0",
@@ -124,8 +139,8 @@
     "execa": "10.0.1",
     "minimatch": "10.2.6",
     "picocolors": "^1.1.1",
-    "playwright": "1.59.1",
-    "playwright-core": "1.59.1",
+    "playwright": "1.64.1",
+    "playwright-core": "1.64.1",
     "string-width": "^8.1.0",
     "tar": "7.5.21",
     "tinyglobby": "0.2.17",
@@ -151,7 +166,7 @@
     }
   },
   "engines": {
-    "node": ">=22.15.0 <23.0.0 || >=24.14.1 <25.0.0",
+    "node": ">=22.15.0 <23.0.0 || >=24.14.1 <25.0.0 || >=26.13.0 <27.0.0",
     "npm": ">=11.6.0 <12.0.0"
   },
   "volta": {