workers-port 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +308 -236
- package/bin/workers.js +19 -3
- package/demo/serve.js +9 -3
- package/package.json +1 -1
- package/src/loop.js +2 -2
- package/src/mutants.js +5 -1
- package/src/port.js +191 -180
- package/src/report.js +1 -1
- package/src/rules.md +1 -0
- package/src/run-kane.js +9 -5
- package/src/run-playwright.js +7 -3
- package/src/server.js +24 -23
- package/src/translate.js +83 -5
- package/src/util.js +9 -5
package/README.md
CHANGED
|
@@ -1,236 +1,308 @@
|
|
|
1
|
-
# Workers
|
|
2
|
-
|
|
3
|
-
Workers ports your Playwright suite to Kane CLI tests and proves the new tests catch the same bugs, by breaking your app on purpose. When a ported test is weaker, your coding agent rewrites it from Kane's own result.
|
|
4
|
-
|
|
5
|
-
**[
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
##
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
|
48
|
-
|
|
49
|
-
|
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
|
68
|
-
|
|
69
|
-
|
|
|
70
|
-
| Kane
|
|
71
|
-
|
|
|
72
|
-
|
|
|
73
|
-
|
|
|
74
|
-
| Kane
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
136
|
-
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
The
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
158
|
-
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
]
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
|
|
180
|
-
|
|
181
|
-
|
|
182
|
-
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
|
|
207
|
-
| `
|
|
208
|
-
|
|
209
|
-
|
|
210
|
-
|
|
211
|
-
|
|
212
|
-
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
223
|
-
|
|
224
|
-
|
|
225
|
-
|
|
226
|
-
|
|
227
|
-
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
236
|
-
|
|
1
|
+
# Workers
|
|
2
|
+
|
|
3
|
+
Workers ports your Playwright suite to Kane CLI tests and proves the new tests catch the same bugs, by breaking your app on purpose. When a ported test is weaker, your coding agent rewrites it from Kane's own result.
|
|
4
|
+
|
|
5
|
+
**[Live site](https://workers-port.vercel.app)** · **[Report: TodoMVC](https://workers-port.vercel.app/report/)** · **[Report: shop app](https://workers-port.vercel.app/shop-report/)** · **[npm: workers-port](https://www.npmjs.com/package/workers-port)** · [Source](https://github.com/Techkeyy/workers)
|
|
6
|
+
|
|
7
|
+
**[Watch the 3-minute film](https://youtu.be/Ch378w02KHE)**
|
|
8
|
+
|
|
9
|
+
[](https://youtu.be/Ch378w02KHE)
|
|
10
|
+
|
|
11
|
+
Teams want off brittle selector tests. Rewriting a suite by hand takes weeks, and nobody trusts an AI rewrite they cannot check. TestMu's own [*State of AI in Quality Engineering 2026*](https://www.testmuai.com/research/state-of-ai-in-quality-engineering.md) puts the median trust in AI-generated tests at 54 out of 100, and finds teams spend 50% of their AI testing budget on generating tests but only 17% on verifying them.
|
|
12
|
+
|
|
13
|
+
> *"The agent says the new tests are equivalent. Would they actually catch the bug the old ones caught?"*
|
|
14
|
+
|
|
15
|
+
Workers answers that question with a run, not with the agent's word. Every ported test is checked against deliberate breaks of your app (mutants), side by side with the original Playwright test. **A ported test is only accepted if it catches the same bugs, and it avoids the false alarms the selector tests raise.**
|
|
16
|
+
|
|
17
|
+
Built for the Kane CLI hackathon.
|
|
18
|
+
|
|
19
|
+

|
|
20
|
+
|
|
21
|
+
## For judges: the 60-second tour
|
|
22
|
+
|
|
23
|
+
| Criterion | Where to see it |
|
|
24
|
+
|---|---|
|
|
25
|
+
| **Bar 1: works end to end** | Two real runs, published as reports: [TodoMVC](https://workers-port.vercel.app/report/) (6 tests, 7 mutants) and a second app, [shop](https://workers-port.vercel.app/shop-report/) (10 tests, 7 mutants). Each ran the full path: list, translate with a coding agent, Playwright on every mutant, Kane on every mutant, parity diff, repair, report. See the [Quickstart](#quickstart). |
|
|
26
|
+
| **Bar 2: Kane caught or proved something** | Kane failed on the same planted bugs Playwright failed on: 7 of 7 test and bug pairs in TodoMVC, 8 of 8 in the shop ([how to read these](#results-run-2-on-the-bundled-todomvc-demo)). It also passed 2 and 5 harmless refactors that broke the Playwright selectors. Screenshots: [`report-matrix.jpg`](docs/images/report-matrix.jpg), [`shop-report.jpg`](docs/images/shop-report.jpg). |
|
|
27
|
+
| **Bar 3: agent and Kane talk to each other** | The [repair loop](#the-repair-loop-kane-talks-back-to-the-agent): Kane's result on a mutant goes into the agent's prompt, the agent rewrites the test, Kane runs it again. Screenshot: [`repair-loop.jpg`](docs/images/repair-loop.jpg). Live: the Repair loop section of the [TodoMVC report](https://workers-port.vercel.app/report/). |
|
|
28
|
+
| **Ships** | On npm as [`workers-port`](https://www.npmjs.com/package/workers-port), a live [site](https://workers-port.vercel.app) with a [break-it demo](https://workers-port.vercel.app/#break-it), a Claude Code [skill](skill/workers/SKILL.md), 87 passing unit tests. |
|
|
29
|
+
| **Verified** | The check is the product: a bug mutant counts as caught only when Playwright and Kane both fail it, from exit codes and JSON. A bug that slipped past both suites (B2) is reported, not hidden. See [How I tried to break it](#how-i-tried-to-break-it). |
|
|
30
|
+
| **Closed loop** | Fired twice, both real. In run 2, "append new items" was `kane-weaker` on B3: round 1's rewrite broke on the clean app and was rejected, round 2 caught B3. In a [deliberate attack](#attack-a-deliberately-useless-test), a planted useless test was flagged `kane-weaker` on B1 and B3 and repaired in one round ([report](https://workers-port.vercel.app/attack-report/)). |
|
|
31
|
+
| **Craft** | `npm install workers-port`, then a [dry run that costs no Kane credits](#quickstart). `--resume` after an interrupted run. Findings for the Kane team: [K1, T1, K2](#platform-feedback-for-kane-cli). |
|
|
32
|
+
|
|
33
|
+
Raw data behind every number: [TodoMVC `report.json`](https://workers-port.vercel.app/report/report.json) and [shop `report.json`](https://workers-port.vercel.app/shop-report/report.json).
|
|
34
|
+
|
|
35
|
+
## Why this exists
|
|
36
|
+
|
|
37
|
+
Playwright tests pin to selectors and `data-testid`s. A harmless markup change breaks them, so teams look at Kane CLI, where tests are plain English steps. The move is the hard part:
|
|
38
|
+
|
|
39
|
+
- A hand rewrite is weeks of work.
|
|
40
|
+
- An agent rewrite is fast, but a test that passes on the clean app proves nothing. A test that asserts nothing passes too.
|
|
41
|
+
- Kane's own recording can also be weaker than it looks (see [K1](#platform-feedback-for-kane-cli): it rebuilt "N items left" from a bare number).
|
|
42
|
+
|
|
43
|
+
What is missing is a check on the port itself. Workers supplies it.
|
|
44
|
+
|
|
45
|
+
```
|
|
46
|
+
old Playwright test new Kane test
|
|
47
|
+
| |
|
|
48
|
+
+---------- same app --------+
|
|
49
|
+
|
|
|
50
|
+
clean app: both must pass
|
|
51
|
+
bug mutant: both must FAIL <- the new test is as strong
|
|
52
|
+
refactor mutant: Playwright fails, Kane passes <- the false alarm is gone
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
## What it does
|
|
56
|
+
|
|
57
|
+
1. **Lists** the Playwright tests (Playwright's own `--list` JSON).
|
|
58
|
+
2. **Translates** each test with your coding agent into a Kane `_test.md` in plain English, following `src/rules.md`. Every `expect` becomes an explicit "Verify" line. Local files a spec imports (page objects, fixtures) are sent along, two levels deep, capped at 40 KB. The agent returns text; Workers writes the files.
|
|
59
|
+
3. **Runs Playwright** on the clean app, then on every mutant. Each mutant is applied to the app, the suite runs, and the mutant is reverted (also on errors and Ctrl+C).
|
|
60
|
+
4. **Runs Kane**: authors each test on the clean app, then runs it on every mutant with Kane's default self-healing, the way it runs in CI. The test's local recordings are restored from a clean snapshot before and after every run.
|
|
61
|
+
5. **Diffs parity** with plain code, no LLM. Verdicts come from Playwright's JSON and Kane's exit code and `run_end` status.
|
|
62
|
+
6. **Repairs** weak tests. For each test Kane is weaker on, the agent gets what the old test asserted and what Kane did, rewrites it, and Workers re-runs it, up to 2 rounds. The agent never sees the mutant. A test still weak after the cap is marked "needs human".
|
|
63
|
+
7. **Reports** `report.html` and `report.json`: the matrix, findings, repair log with a real line diff, Kane dashboard links, before and after line counts.
|
|
64
|
+
|
|
65
|
+
Cell classes:
|
|
66
|
+
|
|
67
|
+
| Class | Meaning |
|
|
68
|
+
|---|---|
|
|
69
|
+
| `ok` | passes on the clean app in both suites |
|
|
70
|
+
| `broken-on-clean` | the Kane test fails on the clean app, so it needs a repair |
|
|
71
|
+
| `parity-caught` | bug mutant: Playwright and Kane both fail it |
|
|
72
|
+
| `parity-missed` | bug mutant: neither suite fails it (this test does not check that behaviour) |
|
|
73
|
+
| `kane-weaker` | bug mutant: Playwright fails it, Kane passes. Repair round |
|
|
74
|
+
| `kane-stronger` | bug mutant: Kane fails it, Playwright passes |
|
|
75
|
+
| `parity` | refactor mutant: both pass |
|
|
76
|
+
| `false-alarm-avoided` | refactor mutant: Playwright fails, Kane passes |
|
|
77
|
+
| `kane-false-alarm` | refactor mutant: Kane fails |
|
|
78
|
+
|
|
79
|
+
The pitch is plain English and no selectors, not shorter files: in run 2 the six `_test.md` files total 116 lines against 79 for the Playwright originals, and in the shop run 243 against 77.
|
|
80
|
+
|
|
81
|
+
## Results: run 2 on the bundled TodoMVC demo
|
|
82
|
+
|
|
83
|
+
Six tests from Playwright's example spec, 7 mutants (5 bugs B1 to B5, 2 refactors R1, R2), Kane with self-healing on, local recordings restored per run. Source: [`report.json`](https://workers-port.vercel.app/report/report.json), rendered at [/report/](https://workers-port.vercel.app/report/).
|
|
84
|
+
|
|
85
|
+
"Bugs caught" counts test and bug pairs: one test failing on one bug. There are 6 tests times 5 bugs, so 30 pairs. Playwright failed on 7 of them, and Kane failed on the same 7. Across the suite, 4 of the 5 distinct bugs were caught (B1, B3, B4, B5) and B2 by none.
|
|
86
|
+
|
|
87
|
+
| | 6 tests, 7 mutants |
|
|
88
|
+
|---|---|
|
|
89
|
+
| Test and bug pairs caught, old suite | 7 of 30 |
|
|
90
|
+
| Test and bug pairs caught, new suite | 7 of 30 |
|
|
91
|
+
| False alarms avoided | 2 |
|
|
92
|
+
| Kane weaker | 0 |
|
|
93
|
+
| Needs human | 0 |
|
|
94
|
+
| Bugs that slipped past both suites | 1 (B2) |
|
|
95
|
+
| Repaired tests | 1 (2 rounds) |
|
|
96
|
+
| Kane credits | 389.8 |
|
|
97
|
+
|
|
98
|
+
Mutants: B1 active count counts completed items, B2 counter always says "items", B3 counter always says "item", B4 "Clear completed" removes active items, B5 Escape saves the edit, R1 counter test-id renamed, R2 extra class on the clear button.
|
|
99
|
+
|
|
100
|
+
`C` = bug caught by both Playwright and Kane. `.` = both pass (the test does not check that). `A` = refactor, Playwright fails, Kane passes (false alarm avoided). `ok` = both pass on the clean app.
|
|
101
|
+
|
|
102
|
+
| Test | clean | B1 | B2 | B3 | B4 | B5 | R1 | R2 |
|
|
103
|
+
|---|---|---|---|---|---|---|---|---|
|
|
104
|
+
| add todo items | ok | . | . | . | . | . | . | . |
|
|
105
|
+
| append new items to the bottom (repaired) | ok | C | . | C | . | . | A | . |
|
|
106
|
+
| cancel edits on escape | ok | . | . | . | . | C | . | . |
|
|
107
|
+
| display the current number of todo items | ok | C | . | . | . | . | A | . |
|
|
108
|
+
| remove completed items when clicked | ok | . | . | . | C | . | . | . |
|
|
109
|
+
| hidden when there are no completed items | ok | C | . | . | C | . | . | . |
|
|
110
|
+
|
|
111
|
+

|
|
112
|
+
|
|
113
|
+
The live site replays these recorded results. Pick a bug and it shows what each suite said. Nothing on that page runs live.
|
|
114
|
+
|
|
115
|
+
### The repair loop: Kane talks back to the agent
|
|
116
|
+
|
|
117
|
+
"should append new items to the bottom of the list" asserts `toHaveText('3 items left')`.
|
|
118
|
+
|
|
119
|
+
- **First translation:** `kane-weaker` on B3 (the counter shows "3 item left"). Playwright caught it, Kane did not. In an earlier run the same test caught B3 first time, so Kane's authoring is not deterministic. That is why the port has to be verified.
|
|
120
|
+
- **Round 1:** the agent was given what the old test asserted and Kane's step results, and rewrote the test. The rewrite broke on the clean app (`broken-on-clean`) and was rejected. The agent's word was not trusted, the run was.
|
|
121
|
+
- **Round 2:** the agent fixed it, using the wording "exact visible text of the todo counter ... is '3 items left'". Kane re-authored it, caught B3, and every other cell held. Final status `repaired`.
|
|
122
|
+
|
|
123
|
+

|
|
124
|
+
|
|
125
|
+
### Attack: a deliberately useless test
|
|
126
|
+
|
|
127
|
+
The obvious way to cheat a port is a test that asserts nothing. It passes on the clean app, so a tool that only checks "green" accepts it. So I planted one: I replaced the translation of "should append new items to the bottom of the list" with a test that does the real setup (adds the three todos) but only checks `Verify the todo list is visible.`. Then I let Workers pick it up with `--resume` and ran it for real against Kane.
|
|
128
|
+
|
|
129
|
+
| Step | What happened |
|
|
130
|
+
| --- | --- |
|
|
131
|
+
| Clean app | The useless test passed, as a useless test would. |
|
|
132
|
+
| Mutants | It missed B1 and B3, the two bugs the original Playwright test catches. Workers marked it `kane-weaker` on both. |
|
|
133
|
+
| Repair round 1 | The agent got the original test's assertions and Kane's step results, never the mutants, and rewrote the test with three real checks: the exact counter text `"3 items left"`, the exact list order, and the same list after a reload. |
|
|
134
|
+
| Re-run | Kane re-authored the rewrite and **caught B1 and B3**. Every other cell held, including the R1 false alarm avoided. Final status `repaired`, 71.4 credits. |
|
|
135
|
+
|
|
136
|
+
Report: [attack-report](https://workers-port.vercel.app/attack-report/) · [raw `report.json`](https://workers-port.vercel.app/attack-report/report.json)
|
|
137
|
+
|
|
138
|
+
## Results: a second app with a real-world suite
|
|
139
|
+
|
|
140
|
+
TodoMVC's example suite is tidy. Real suites are not. So I ran Workers on a second app, [`examples/shop`](examples/shop) ("Inkwell Supply": login, cart, coupon, checkout), with a Playwright suite written the way teams write them:
|
|
141
|
+
|
|
142
|
+
- page objects in their own files (`pages/LoginPage.ts`, `ShopPage.ts`, `CartPage.ts`)
|
|
143
|
+
- a `loggedInPage` fixture built with `test.extend`
|
|
144
|
+
- role-based locators
|
|
145
|
+
- three spec files, 10 tests in total
|
|
146
|
+
|
|
147
|
+
**What this exposed and fixed:** the translator only sent the agent the test block, so imported page objects were invisible to it. It now includes the local files a spec imports. In the translations:
|
|
148
|
+
|
|
149
|
+
- the fixture became the real login steps with the demo account
|
|
150
|
+
- `addToCart('Fountain Pen')` became `Click the button labeled "Add Fountain Pen to cart".`
|
|
151
|
+
|
|
152
|
+
**The result** (source: [`report.json`](https://workers-port.vercel.app/shop-report/report.json)):
|
|
153
|
+
|
|
154
|
+
| | |
|
|
155
|
+
| --- | --- |
|
|
156
|
+
| Tests ported | 10/10 |
|
|
157
|
+
| Test and bug pairs caught, old suite / Kane (of 50) | 8 / 8 |
|
|
158
|
+
| Distinct bugs caught (B1 to B5) | all 5 |
|
|
159
|
+
| False alarms avoided (a renamed `data-testid`, restructured markup) | 5 |
|
|
160
|
+
| Kane weaker / needs human | 0 / 0 |
|
|
161
|
+
|
|
162
|
+

|
|
163
|
+
|
|
164
|
+
[Read the full shop report](https://workers-port.vercel.app/shop-report/).
|
|
165
|
+
|
|
166
|
+
This run was interrupted halfway by a machine restart and finished with `--resume`, so the credits figure in that report (191.1) only covers the resumed part.
|
|
167
|
+
|
|
168
|
+
## Quickstart
|
|
169
|
+
|
|
170
|
+
Prerequisites:
|
|
171
|
+
|
|
172
|
+
- Node 22 or newer (`engines` in `package.json`)
|
|
173
|
+
- `kane-cli` (`npm install -g @testmuai/kane-cli`), logged in, with credits
|
|
174
|
+
- `claude` or `codex` CLI, logged in
|
|
175
|
+
- Chromium for Playwright (installed by the command below)
|
|
176
|
+
|
|
177
|
+
From npm, in any empty folder (the demo app and its Playwright suite are copied to `./workers-demo`):
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
npm install workers-port
|
|
181
|
+
npx playwright install chromium
|
|
182
|
+
|
|
183
|
+
# everything except Kane: list, translate, run Playwright on every mutant, print a credit estimate
|
|
184
|
+
npx workers-port demo --dry-run --max-tests 1
|
|
185
|
+
|
|
186
|
+
# the real run (spends Kane credits)
|
|
187
|
+
npx workers-port demo --max-tests 3
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
Or from a clone of this repo:
|
|
191
|
+
|
|
192
|
+
```bash
|
|
193
|
+
git clone https://github.com/Techkeyy/workers
|
|
194
|
+
cd workers
|
|
195
|
+
npm ci
|
|
196
|
+
npx playwright install chromium
|
|
197
|
+
|
|
198
|
+
# everything except Kane: list, translate, run Playwright on every mutant, print a credit estimate
|
|
199
|
+
node bin/workers.js demo --dry-run --max-tests 1
|
|
200
|
+
|
|
201
|
+
# the real run (spends Kane credits)
|
|
202
|
+
node bin/workers.js demo --max-tests 3
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
The claude adapter is the default (`--agent claude`, falling back to codex if claude is not logged in and codex is). To use Codex: `--agent codex`. If Codex's configured default model is not available on your account, pass `--agent-model <a model your account supports>`. My runs used `--agent codex --agent-model gpt-5.6-terra`.
|
|
206
|
+
|
|
207
|
+
`--grep "title one|title two"` ports only tests whose title contains one of the strings. Run 2 used it to pick the six tests that cover every mutant. Without `--max-tests` or `--grep`, the demo ports the whole 24-test spec.
|
|
208
|
+
|
|
209
|
+
If a run is interrupted, `--resume` with the same `--out` folder reuses the `_test.md` files already written, and Kane replays their recordings instead of authoring them again.
|
|
210
|
+
|
|
211
|
+
Credits, as I measured them: about 40 per authored test, 0 per clean replay, 10 to 30 per mutant run where self-healing kicks in. Run 2 cost 389.8 credits for 6 tests.
|
|
212
|
+
|
|
213
|
+
## Using it on your own project
|
|
214
|
+
|
|
215
|
+
```bash
|
|
216
|
+
node bin/workers.js port --config playwright.config.ts --app ./my-app --mutants workers.mutants.json
|
|
217
|
+
```
|
|
218
|
+
|
|
219
|
+
Options: `--url`, `--out`, `--grep`, `--max-tests`, `--repair-rounds`, `--agent`, `--agent-model`, `--resume`, `--dry-run`. Run `node bin/workers.js --help` for the full list. Run `--dry-run` first.
|
|
220
|
+
|
|
221
|
+
The mutants file is a JSON array. Each mutant edits one file in `--app` with an exact find and replace, temporarily:
|
|
222
|
+
|
|
223
|
+
```json
|
|
224
|
+
[
|
|
225
|
+
{
|
|
226
|
+
"id": "B3",
|
|
227
|
+
"kind": "bug",
|
|
228
|
+
"description": "Counter always says \"item\" (so \"3 item left\")",
|
|
229
|
+
"file": "app.js",
|
|
230
|
+
"find": "const itemWord = activeCount === 1 ? 'item' : 'items';",
|
|
231
|
+
"replace": "const itemWord = 'item';"
|
|
232
|
+
}
|
|
233
|
+
]
|
|
234
|
+
```
|
|
235
|
+
|
|
236
|
+
`kind` is `bug` (behaviour changes, both suites should fail) or `refactor` (markup changes, behaviour stays, Playwright may fail and Kane should pass). Bundled files: [`demo/workers.mutants.json`](demo/workers.mutants.json) and [`examples/shop/workers.mutants.json`](examples/shop/workers.mutants.json). Originals are kept in `<file>.workers-backup` and restored when the run ends, fails, is interrupted, or at the next startup.
|
|
237
|
+
|
|
238
|
+
## Claude Code skill
|
|
239
|
+
|
|
240
|
+
`skill/workers/` lets you say "port my Playwright tests to Kane" inside Claude Code. Install it by copying the folder:
|
|
241
|
+
|
|
242
|
+
```bash
|
|
243
|
+
mkdir -p ~/.claude/skills
|
|
244
|
+
cp -r skill/workers ~/.claude/skills/
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
Use `.claude/skills/` instead for a single project, then start a new Claude Code session. See [`skill/README.md`](skill/README.md).
|
|
248
|
+
|
|
249
|
+
## How I tried to break it
|
|
250
|
+
|
|
251
|
+
| Attack | Result |
|
|
252
|
+
|---|---|
|
|
253
|
+
| The agent sees the mutant and tunes the test to it | It cannot. The repair prompt holds the old test's Playwright error and source plus Kane's step results, never the mutant's id, description, file or diff. A unit test pins this. |
|
|
254
|
+
| The agent writes a test that asserts nothing | It has to fail the mutants the old test fails. In run 2, round 1's weaker rewrite of the append test did not survive the runs: it broke on the clean app and was rejected. A direct attack with a useless test is [reported above](#attack-a-deliberately-useless-test). |
|
|
255
|
+
| Kane heals around a bug and passes it | A heal that passes a bug mutant still counts as a miss (`kane-weaker`). A heal that passes a refactor counts as a false alarm avoided, with the heal flagged. |
|
|
256
|
+
| Strict replay hides the brittleness story | In strict replay Kane's recorded checks use `data-testid` selectors like Playwright does (R1 was a `kane-false-alarm` with healing off). So Workers runs Kane the way CI does, with self-healing on, and restores the clean recordings before every run. |
|
|
257
|
+
| Kane authoring changes between runs | It does. The same test caught B3 in run 1 and missed it in run 2 before repair. That is the case for verifying every port. |
|
|
258
|
+
| A bug neither suite catches | B2 ("1 items left") slipped past both suites in TodoMVC. It is reported as missed, not hidden. |
|
|
259
|
+
| Imported page objects are invisible to the agent | They were, in the first version. The shop run exposed it, and the translator now sends the imported files. A unit test pins that the shop spec pulls in its page objects and fixtures. |
|
|
260
|
+
| Mutant left in the app after a crash | Reverted on errors and SIGINT, and at next startup. `demo/app` was clean after run 2. |
|
|
261
|
+
|
|
262
|
+
**A bug-mutant cell is only `parity-caught` when both suites actually failed. Verdicts come from exit codes and JSON, never from the agent.** Rules decide, the agent only writes tests.
|
|
263
|
+
|
|
264
|
+
The Workers code itself has 87 unit tests (`npm test`, which runs `node --test`), plus the real runs above.
|
|
265
|
+
|
|
266
|
+
## Architecture
|
|
267
|
+
|
|
268
|
+
| File | Job |
|
|
269
|
+
|---|---|
|
|
270
|
+
| `bin/workers.js` | CLI entry: `port` and `demo` |
|
|
271
|
+
| `src/port.js` | Orchestrates the full run: translate, baseline, mutants, parity, repair, report |
|
|
272
|
+
| `src/inventory.js` | Lists Playwright tests from `--list` JSON |
|
|
273
|
+
| `src/translate.js` | Extracts a test's source, collects its local imports, asks the agent, validates the `_test.md` |
|
|
274
|
+
| `src/rules.md` | The translation rules the agent follows |
|
|
275
|
+
| `src/agents.js` | Adapters for claude and codex |
|
|
276
|
+
| `src/run-playwright.js` | Parses Playwright results into per-test verdicts |
|
|
277
|
+
| `src/run-kane.js` | Runs Kane and parses its NDJSON into a result |
|
|
278
|
+
| `src/kane-cache.js` | Snapshots and restores Kane's per-test recordings |
|
|
279
|
+
| `src/mutants.js` | Loads, applies and reverts mutants |
|
|
280
|
+
| `src/parity.js` | Pure function: verdicts to cell class |
|
|
281
|
+
| `src/loop.js` | Builds the repair prompt (no mutant details) and runs the rounds |
|
|
282
|
+
| `src/diff.js` | Line-based LCS diff for the repair log |
|
|
283
|
+
| `src/report.js` | Renders `report.html` and `report.json` |
|
|
284
|
+
| `src/server.js` | Makes sure the app is being served |
|
|
285
|
+
| `src/util.js` | Runs child processes |
|
|
286
|
+
|
|
287
|
+
## Platform feedback for Kane CLI
|
|
288
|
+
|
|
289
|
+
I built on `kane-cli` 0.10.4 (`testmd run`, NDJSON `--agent` output, `output-<stem>/` recordings, self-healing). Mutation checks turn up things normal use rarely does. These are observations from my runs; I have not filed them as issues yet.
|
|
290
|
+
|
|
291
|
+
- **K1, plural label rebuilt in extraction.** The recorded `code_js` for "Verify the counter reads '1 item left'" reads only the number and rebuilds the text: ``const n=Number(strong.textContent); return `${n} item${n===1?'':'s'} left` ``. A replay with `--no-adaptive-heal` on B2 passed, while Playwright's read of the DOM gave "1 items left". So a label or plural bug cannot be caught by that check. Wording like "the exact visible text of X is ..." helps (B3 was caught), and Workers uses it, but the recorded check can still differ between authoring runs.
|
|
292
|
+
- **T1, localStorage reads time out.** Authoring a test that verified localStorage failed with "verification setup repeatedly timed out reading browser storage". Workers translates those checks into reload then verify what is visible.
|
|
293
|
+
- **K2, a failed healed run replaced the local recordings.** A failed run with healing on set `artifacts.replaced: true` for B1. The next run (B2) had 0 replay decisions and re-authored on the mutant, about 49 credits, committing that version. Restoring `output-<stem>/` from a snapshot fixed it locally (a clean replay after restore cost 0 credits). It would help if a failed run left the recordings alone.
|
|
294
|
+
|
|
295
|
+
## Limitations
|
|
296
|
+
|
|
297
|
+
- Mutants are hand-written. Workers does not generate them.
|
|
298
|
+
- The claude adapter is the default and is unit-tested, but the published runs used codex (model `gpt-5.6-terra`). I have not exercised the claude adapter end to end.
|
|
299
|
+
- Kane runs sequentially, about a minute each.
|
|
300
|
+
- Desktop web only.
|
|
301
|
+
- Tests run against a local app.
|
|
302
|
+
- One planted bug (B2) slipped past both suites in the TodoMVC demo.
|
|
303
|
+
- Ctrl+C reverts any applied mutant, but a dev server that Workers started itself may keep running; stop it by hand.
|
|
304
|
+
- `--resume` reuses existing `_test.md` files as they are. If you edit a Playwright spec between runs, delete its folder in `--out` first.
|
|
305
|
+
|
|
306
|
+
## License
|
|
307
|
+
|
|
308
|
+
MIT, see [LICENSE](LICENSE). `demo/tests/demo-todo-app.spec.ts` is Playwright's example spec from the `create-playwright` package (version 1.17.130, Microsoft, MIT). Only its URL was changed, see [`demo/tests/NOTICE.md`](demo/tests/NOTICE.md).
|