staysfixed 0.3.1 → 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +534 -402
- package/package.json +8 -3
- package/src/cli/index.js +14 -0
- package/src/v2/adapters/android-driver.js +1705 -0
- package/src/v2/adapters/android.js +1117 -0
- package/src/v2/adapters/contract.js +565 -0
- package/src/v2/adapters/electron.js +1594 -0
- package/src/v2/adapters/http.js +733 -0
- package/src/v2/adapters/ios-driver.js +1551 -0
- package/src/v2/adapters/ios.js +989 -0
- package/src/v2/adapters/isolate.js +739 -0
- package/src/v2/adapters/process.js +920 -0
- package/src/v2/adapters/source.js +1241 -0
- package/src/v2/adapters/web-driver.js +1532 -0
- package/src/v2/adapters/web.js +1009 -0
- package/src/v2/adapters/windows.js +1329 -0
- package/src/v2/browsers.js +1203 -0
- package/src/v2/cause.js +364 -0
- package/src/v2/check.js +1331 -0
- package/src/v2/ci.js +1209 -0
- package/src/v2/cli.js +657 -0
- package/src/v2/cluster.js +372 -0
- package/src/v2/coverage.js +1116 -0
- package/src/v2/detect.js +1199 -0
- package/src/v2/doctor.js +1690 -0
- package/src/v2/escalate.js +679 -0
- package/src/v2/init.js +1394 -0
- package/src/v2/intent.js +659 -0
- package/src/v2/journeys/from-routes.js +498 -0
- package/src/v2/journeys/from-suite.js +988 -0
- package/src/v2/journeys/index.js +651 -0
- package/src/v2/journeys/record.js +516 -0
- package/src/v2/mcp/server.js +374 -0
- package/src/v2/mcp/tools.js +1571 -0
- package/src/v2/normalise.js +783 -0
- package/src/v2/observation.js +877 -0
- package/src/v2/rank.js +672 -0
- package/src/v2/reference.js +1051 -0
- package/src/v2/remote.js +911 -0
- package/src/v2/run.js +964 -0
- package/src/v2/sealed.js +564 -0
- package/src/v2/selfcheck.js +564 -0
- package/src/v2/ship.js +684 -0
- package/src/v2/store.js +703 -0
- package/src/v2/types.js +503 -0
- package/src/v2/waiver.js +511 -0
package/README.md
CHANGED
|
@@ -1,304 +1,459 @@
|
|
|
1
1
|
# Stays Fixed
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
3
|
+
**Prove that nothing which already worked has changed — and let the agent do the
|
|
4
|
+
reviewing.**
|
|
5
|
+
|
|
6
|
+
An AI agent changes twenty files in four minutes. What breaks is almost never the
|
|
7
|
+
thing it was working on: it is something in a corner that had already been built,
|
|
8
|
+
already been fixed, and that nobody thought to look at again. The agent cannot
|
|
9
|
+
check that corner, because it does not know the corner exists.
|
|
10
|
+
|
|
11
|
+
So Stays Fixed runs your product through the same steps twice, compares it
|
|
12
|
+
against the build you were last happy with, subtracts everything the product
|
|
13
|
+
disagrees with itself about, and hands back **only the differences nobody asked
|
|
14
|
+
for**. Everything unchanged is skipped and never mentioned. The agent already
|
|
15
|
+
knows what it *meant* to change, so what is left is its work queue.
|
|
16
|
+
|
|
17
|
+
You are not in that loop, and you never approve anything. **What "working" means
|
|
18
|
+
is cut by something you already do: saying ship.** You hear about it only when a
|
|
19
|
+
difference lands in a class no agent may wave through — money, signing in, lost
|
|
20
|
+
data, a crash, or a bug you already reported once — and then it arrives as three
|
|
21
|
+
plain sentences inside the summary you were reading anyway.
|
|
5
22
|
|
|
6
|
-
|
|
7
|
-
never the thing they were working on — it is something in the corner that had
|
|
8
|
-
already been built, already been fixed, and that nobody thought to look at again.
|
|
9
|
-
Ordinary tests read the code, and code that reads fine can still render a page
|
|
10
|
-
with no stylesheet, a collapsed sidebar or a button pushed off the screen. A
|
|
11
|
-
picture can see that. So Stays Fixed opens your real app, photographs the screens
|
|
12
|
-
that matter, and compares them against pictures a human approved.
|
|
23
|
+
---
|
|
13
24
|
|
|
14
|
-
|
|
25
|
+
## The loop
|
|
15
26
|
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
27
|
+
1. The agent seals what it **meant** to change, before it runs anything.
|
|
28
|
+
2. It changes some code.
|
|
29
|
+
3. It calls `staysfixed_check` over MCP.
|
|
30
|
+
4. Everything unchanged is skipped and never reaches its context.
|
|
31
|
+
5. What comes back is what changed. It already knows what it intended, so the
|
|
32
|
+
targets are the differences it did **not** intend.
|
|
33
|
+
6. It fixes those and runs again. Anything it genuinely meant, it can record as
|
|
34
|
+
intended — five times, never in a sealed class, and only inside what it
|
|
35
|
+
sealed in step 1.
|
|
22
36
|
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
37
|
+
> "You will call the MCP, make it run the tests, the results come back, then you
|
|
38
|
+
> decide. Whatever is unchanged will be skipped. The things that changed other
|
|
39
|
+
> than the ones you actually did yourself — those ones are the targets. So you
|
|
40
|
+
> will not burn your tokens reviewing every single thing. You will only review
|
|
41
|
+
> what actually matters."
|
|
26
42
|
|
|
27
43
|
---
|
|
28
44
|
|
|
29
|
-
## Install
|
|
45
|
+
## Install
|
|
30
46
|
|
|
31
47
|
```
|
|
32
48
|
npx staysfixed init
|
|
33
49
|
npx staysfixed check
|
|
34
50
|
```
|
|
35
51
|
|
|
36
|
-
No
|
|
37
|
-
it only needs to be able to
|
|
38
|
-
|
|
39
|
-
`init` asks what your app is, writes a `staysfixed.config.js` you can read, makes
|
|
40
|
-
the `.staysfixed/` folder and adds the two lines to your `.gitignore` that keep
|
|
41
|
-
the throwaway results out of git.
|
|
52
|
+
No account, no sign-up, no server anywhere, nothing uploaded. It works in any
|
|
53
|
+
project in any language — it only needs to be able to run your product.
|
|
42
54
|
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
right:
|
|
55
|
+
Requirements: **Node 22 or newer**. Everything else depends on what you are
|
|
56
|
+
watching, and the tool works out what it has:
|
|
46
57
|
|
|
47
58
|
```
|
|
48
|
-
npx staysfixed
|
|
59
|
+
npx staysfixed doctor
|
|
60
|
+
npx staysfixed doctor --json # the same answer, for an agent
|
|
49
61
|
```
|
|
50
62
|
|
|
51
|
-
|
|
63
|
+
`doctor` is the first thing you should run and the first thing an agent should
|
|
64
|
+
call. It says what it can check on this machine, what it cannot, what is missing,
|
|
65
|
+
and the exact command that would fix each gap — and it never suggests setting up
|
|
66
|
+
something that already works, because everything it lists as missing failed a
|
|
67
|
+
real check first.
|
|
52
68
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
inside your Electron build). `npx staysfixed doctor` tells you what it
|
|
56
|
-
found and what it is missing.
|
|
69
|
+
Every kind of product comes back in one of four states, and the last two are the
|
|
70
|
+
ones that matter:
|
|
57
71
|
|
|
58
|
-
|
|
59
|
-
|
|
72
|
+
| State | What it means for you |
|
|
73
|
+
| --- | --- |
|
|
74
|
+
| **ready** | It works here now. Nothing to say. |
|
|
75
|
+
| **the agent can fix this** | Everything in the way installs with a command. The agent runs it and never mentions it to you. |
|
|
76
|
+
| **only a person can do this** | A licence, a device, a password, a pair of hands. You get *what to do, why it is needed, and what it unlocks* — one sentence each, so being asked for half an hour of your time comes with what you get for it. |
|
|
77
|
+
| **not possible here** | No command on this machine changes the answer. It says so, says the nearest honest alternative, and **stops offering it**. |
|
|
60
78
|
|
|
61
|
-
|
|
79
|
+
Two separate questions are answered separately, because folding them together is
|
|
80
|
+
how a surface gets called ready while nothing is ever walked on it:
|
|
81
|
+
|
|
82
|
+
- **can this machine run it** — is there a simulator, an emulator, a browser, a
|
|
83
|
+
Java runtime, another machine it can already reach over SSH;
|
|
84
|
+
- **can this copy drive it** — is the adapter for that kind of product actually
|
|
85
|
+
in this build of the tool.
|
|
62
86
|
|
|
63
|
-
|
|
87
|
+
A Mac with Xcode on it can run an iPhone app. That says nothing about whether
|
|
88
|
+
there is anything here that knows how to open one, and `doctor --json` answers
|
|
89
|
+
both under `surfaces` and `drivers`.
|
|
64
90
|
|
|
65
|
-
|
|
91
|
+
And it never sends you shopping for nothing. A project with no phone app in it is
|
|
92
|
+
never told to install Java or thirty gigabytes of Xcode — it is told there is no
|
|
93
|
+
phone app here to check.
|
|
66
94
|
|
|
67
|
-
|
|
68
|
-
|
|
95
|
+
### What a fresh install downloads, and what it does not
|
|
96
|
+
|
|
97
|
+
`npm install staysfixed` pulls **two small packages and nothing else** —
|
|
98
|
+
`pixelmatch` and `pngjs`, under a megabyte together. No browser, no runtime,
|
|
99
|
+
nothing that takes minutes.
|
|
100
|
+
|
|
101
|
+
Checking a **website** needs a browser. Rather than make everybody who only
|
|
102
|
+
wanted to check a command-line tool wait for one, that is a separate step you
|
|
103
|
+
take when you need it:
|
|
69
104
|
|
|
70
105
|
```
|
|
71
|
-
|
|
106
|
+
npm install --save-dev playwright && npx playwright install chromium
|
|
107
|
+
```
|
|
72
108
|
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
look at it, then run: staysfixed approve pricing-card-pro
|
|
79
|
-
✓ the sidebar still collapses still holds 400ms
|
|
80
|
-
✓ logging out clears the session still holds 700ms
|
|
109
|
+
Measured on a Mac in August 2026, that is about **18MB of packages** in your
|
|
110
|
+
project and about **570MB of browser** in a shared cache outside it — 371MB for
|
|
111
|
+
Chrome for Testing and 196MB for its headless shell — downloaded once per
|
|
112
|
+
machine, not once per project. `doctor` tells you when you need it, and it is one
|
|
113
|
+
of the things an agent can simply do without asking you.
|
|
81
114
|
|
|
82
|
-
|
|
83
|
-
|
|
115
|
+
Checking a **desktop app** needs no browser at all and no download: the app is
|
|
116
|
+
its own Chromium, and the tool drives it over its own debugging port.
|
|
84
117
|
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
118
|
+
**Checks never open the browser you use.** Given the choice they open Chrome for
|
|
119
|
+
Testing or Chromium — a separate application — because on a Mac two copies of one
|
|
120
|
+
browser share a single slot, and a check running in the background can end up
|
|
121
|
+
answering when you click your own browser icon. If your machine has nothing but
|
|
122
|
+
your everyday browser, it is used, invisibly and on a throwaway profile, and
|
|
123
|
+
every run says so out loud rather than borrowing it quietly. Nothing it opens
|
|
124
|
+
uses your profile, nothing it opens survives the run, and nothing it did not
|
|
125
|
+
start is ever closed:
|
|
89
126
|
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
staysfixed approve pricing-card-pro
|
|
94
|
-
Or accept every one of them: staysfixed approve --all
|
|
95
|
-
The pictures, side by side: .staysfixed/report.html
|
|
127
|
+
```
|
|
128
|
+
npx staysfixed browsers # which browser checks open, and why
|
|
129
|
+
npx staysfixed browsers --clean # clear up after a run that was interrupted
|
|
96
130
|
```
|
|
97
131
|
|
|
98
|
-
|
|
99
|
-
the new one and the difference side by side. If the new look is what you meant,
|
|
100
|
-
approve it and it becomes the picture everything is measured against from now on.
|
|
132
|
+
---
|
|
101
133
|
|
|
102
|
-
|
|
103
|
-
promise. The `results/` folder is only evidence from the last run and is ignored.
|
|
134
|
+
## What is real today
|
|
104
135
|
|
|
105
|
-
|
|
136
|
+
This is a repository in the middle of a rebuild, and the README is not going to
|
|
137
|
+
pretend otherwise.
|
|
106
138
|
|
|
107
|
-
|
|
108
|
-
|
|
139
|
+
| | State |
|
|
140
|
+
| --- | --- |
|
|
141
|
+
| Picture checks, guards, walk, markers, flake register, MCP server | **Shipped.** Published as `staysfixed` 0.3.x and in use. |
|
|
142
|
+
| `doctor` describing this machine in plain English and as JSON | **Shipped.** |
|
|
143
|
+
| The difference engine — the address space, normalisation, wobble subtraction, clustering, ranking, causal proof, the store, the MCP tools, the self-check corpus | **Works.** |
|
|
144
|
+
| Command-line tools, libraries and HTTP servers | **Works.** |
|
|
145
|
+
| Reading the contract straight out of your source — routes, exports, message channels — without running anything | **Works.** 5,785 doors read out of one desktop app in 1.4 seconds. |
|
|
146
|
+
| Websites, through a browser of the tool's own | **Works.** |
|
|
147
|
+
| Electron desktop apps, over their own debugging port | **Works.** |
|
|
148
|
+
| The reference cut when you ship, sealed intents, the waiver budget, and escalations in your closing summary | **Works.** This page describes what it actually does. |
|
|
149
|
+
| The coverage ledger — every door counted, the unopened ones named, and the sentence saying so on every reply | **Works.** See [what it did not check](#what-it-did-not-check). |
|
|
150
|
+
| Aiming a check at one kind of product, and refusing by name rather than checking something else | **Works.** |
|
|
151
|
+
| Android APKs on an emulator | **The adapter is here.** It reads everything the APK declares with nothing installed and no Java, and where there is an emulator it installs one build at a time and walks it. Whether *this* machine can run one is a separate question, and `doctor` asks the adapter itself rather than keeping a second opinion — most of what it wants installs with a command; accepting Google's licence, once, needs a person. Two emulator snapshots restoring byte-identically is unproven, so Android compares against the stored record and says which mode it used. |
|
|
152
|
+
| The iOS simulator | Not yet. `doctor` says so rather than reporting a green run that never touched the phone. |
|
|
153
|
+
| Native Windows GUI (a real Win32 app, not an Electron one) | **The probe is here**, driven over ssh to any machine that reaches a Windows desktop — a WSL shell on one counts, and nothing is installed on it. Windows shows one desktop, so two builds can never run at once: the comparison is genuinely weaker here than anywhere else. |
|
|
154
|
+
|
|
155
|
+
`staysfixed check` is the front door for both. Version 1's flags still mean
|
|
156
|
+
exactly what they meant yesterday — `--pictures`, `--guards`, `--watch` and
|
|
157
|
+
`--only` reach the same code they always did. Nobody who installed this last week
|
|
158
|
+
has to change anything.
|
|
159
|
+
|
|
160
|
+
## How it proves nothing changed
|
|
161
|
+
|
|
162
|
+
Three ideas, in order of how much weight they carry.
|
|
163
|
+
|
|
164
|
+
### 1. Measure the wobble. Never guess a tolerance.
|
|
165
|
+
|
|
166
|
+
Every product disagrees with itself a little between runs — a timestamp, an
|
|
167
|
+
animation frame, an id. So the tool runs the **new build twice**. Anything that
|
|
168
|
+
differs between two runs of the same build was not caused by your change: it is
|
|
169
|
+
the product's own wobble, and it is subtracted arithmetically.
|
|
170
|
+
|
|
171
|
+
There is no tolerance setting in version 2 and there is not going to be one.
|
|
172
|
+
Tolerance knobs are how tools like this die — too loose to catch the real thing,
|
|
173
|
+
too tight to leave switched on.
|
|
174
|
+
|
|
175
|
+
It also catches a bug class no screenshot tool has ever caught. A path that was
|
|
176
|
+
**steady** in the old build and **wobbles** in the new one means the change made
|
|
177
|
+
something unpredictable. That is a finding, even though no single value can be
|
|
178
|
+
pointed at.
|
|
179
|
+
|
|
180
|
+
### 2. Cheap suspicion, expensive proof.
|
|
181
|
+
|
|
182
|
+
Comparing against the stored record is fast and needs no rebuild, so that runs
|
|
183
|
+
first. Every path that then looks different gets the old build **booted live, on
|
|
184
|
+
the same machine, in the same minute**, and walked again. Only differences that
|
|
185
|
+
survive that live re-run are reported.
|
|
186
|
+
|
|
187
|
+
`--paired` goes straight to the expensive half — old build live from the start.
|
|
188
|
+
That is for pre-release, and for the first run on a product with nothing recorded.
|
|
189
|
+
|
|
190
|
+
### 3. Sequential, never simultaneous.
|
|
191
|
+
|
|
192
|
+
Two builds at the same instant fight over ports, single-instance locks, user data
|
|
193
|
+
directories, databases and relay slots. The value was never in the same *second*:
|
|
194
|
+
it is in the same machine, same fonts, same operating system, same data, minutes
|
|
195
|
+
apart. Runs are sequential with a full state reset between them, interleaved
|
|
196
|
+
journey by journey so drift cannot accumulate.
|
|
109
197
|
|
|
110
|
-
|
|
111
|
-
// .staysfixed/guards/the-sidebar-still-collapses.js
|
|
112
|
-
export default {
|
|
113
|
-
name: 'the sidebar still collapses',
|
|
114
|
-
because: 'A CSS rename broke the toggle handler and it shipped unnoticed for four days.',
|
|
115
|
-
async run(app) {
|
|
116
|
-
await app.open('/');
|
|
117
|
-
await app.click('[data-action="toggle-sidebar"]');
|
|
118
|
-
await app.expect('the sidebar is hidden', async () => !(await app.page.visible('.sidebar')));
|
|
119
|
-
},
|
|
120
|
-
};
|
|
121
|
-
```
|
|
198
|
+
---
|
|
122
199
|
|
|
123
|
-
|
|
124
|
-
|
|
200
|
+
## What it looks at
|
|
201
|
+
|
|
202
|
+
Seven channels, all flattened to one shape — a path, a channel, and a value — so
|
|
203
|
+
one comparison engine serves every platform. Pixels are last and are only ever
|
|
204
|
+
evidence for something another channel already found.
|
|
125
205
|
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
206
|
+
| Channel | What it holds |
|
|
207
|
+
| --- | --- |
|
|
208
|
+
| `meaning` | What the interface says a control is and does — its role, its name, whether it is on, off or disabled. Not the underlying markup, because markup changes when nothing did. |
|
|
209
|
+
| `effects` | What the product sent out into the world: calls made, files written, processes started, things saved. |
|
|
210
|
+
| `complaints` | What the product complained about: console messages, errors, crashes, the code it exited with. |
|
|
211
|
+
| `results` | What the product gave back: what it printed, what it answered, what it offers other code. |
|
|
212
|
+
| `contract` | The doors the source says exist: routes, exported functions, message channels. Read without running anything. Free, and exact. |
|
|
213
|
+
| `counters` | Rough counts and rough timings. Deliberately rough — precise timing is noise, not information. |
|
|
214
|
+
| `pixels` | What it looked like. Used to show a person a problem another channel already found. |
|
|
215
|
+
|
|
216
|
+
An address reads left to right, widest thing first:
|
|
217
|
+
|
|
218
|
+
```
|
|
219
|
+
api.GET./users.status
|
|
220
|
+
cli.build.exit
|
|
221
|
+
ipc.session:create.registered
|
|
222
|
+
screen.home.tree.button:Save.enabled
|
|
129
223
|
```
|
|
130
224
|
|
|
131
|
-
|
|
132
|
-
|
|
133
|
-
|
|
134
|
-
|
|
135
|
-
|
|
225
|
+
Three kinds of difference come out, and the last one is the kind no screenshot
|
|
226
|
+
comparison has ever noticed: **changed**, **appeared**, and **vanished** — a
|
|
227
|
+
door that closed.
|
|
228
|
+
|
|
229
|
+
**Where the steps come from**, ranked, because this is the real workload
|
|
230
|
+
question: read the code (free, exact) → run the project's own existing test suite
|
|
231
|
+
under instrumentation → recorded real sessions → the agent exploring one named
|
|
232
|
+
gap and freezing it into a replayable file → never a person clicking through an
|
|
233
|
+
app.
|
|
234
|
+
|
|
235
|
+
## Keeping it quiet
|
|
236
|
+
|
|
237
|
+
Four layers before anything reaches the agent: **normalise** volatile shapes by
|
|
238
|
+
rules kept in git, so a version bump in a footer reports zero differences instead
|
|
239
|
+
of five hundred; **cluster** by signature, so one cause reads as one finding;
|
|
240
|
+
**rank by distance from the changed code**, so a difference far from your edit
|
|
241
|
+
sorts to the **top** — that is the definition of a side effect; then let the agent
|
|
242
|
+
**prove causation** by reverting the suspect hunk and running again. That last
|
|
243
|
+
step is a proof, not a heuristic.
|
|
244
|
+
|
|
245
|
+
Every normalisation rule buys quiet by making some real differences invisible.
|
|
246
|
+
The rule that quietens a wobbling clock also hides a genuinely wrong date. So the
|
|
247
|
+
rules are data, not code: they live in git, they get reviewed like any other
|
|
248
|
+
change, and every one carries a `wouldHide` field in plain English saying what it
|
|
249
|
+
covers up. `explain()` answers for any value it changed — what was replaced,
|
|
250
|
+
where, by which rule, and what that rule admits it might be hiding. The test
|
|
251
|
+
suite fails if a rule ships without one. See
|
|
252
|
+
[docs/how-v2-works.md](docs/how-v2-works.md).
|
|
136
253
|
|
|
137
|
-
|
|
138
|
-
|
|
139
|
-
|
|
140
|
-
are identical either way — bring it to the front yourself whenever you want to
|
|
141
|
-
watch it work. Set `app.foreground: true` if you would rather it came forward.
|
|
254
|
+
---
|
|
255
|
+
|
|
256
|
+
## What it did **not** check
|
|
142
257
|
|
|
143
|
-
|
|
258
|
+
This is the most important thing the tool says, and the reason is arithmetic
|
|
259
|
+
rather than modesty. A tool that reports "nothing changed" is indistinguishable,
|
|
260
|
+
from the outside, from a tool that looked at nothing — and the more useful it
|
|
261
|
+
becomes, the less anybody reads past the headline.
|
|
144
262
|
|
|
145
|
-
|
|
146
|
-
|
|
263
|
+
So every reply carries what was left out, **in the same breath as the good
|
|
264
|
+
news**, on clean runs as loudly as on dirty ones. There is no version of the
|
|
265
|
+
answer that omits it.
|
|
147
266
|
|
|
148
267
|
```
|
|
149
|
-
|
|
268
|
+
ok Nothing that worked has changed. 601 addresses checked against 0.13.0, run
|
|
269
|
+
live. … NOT EVERYTHING WAS CHECKED: 391 of the 452 ways into this product
|
|
270
|
+
have never been walked through, so nothing here says anything about them,
|
|
271
|
+
and 2 other things were not looked at. A clean result only covers what was
|
|
272
|
+
walked.
|
|
273
|
+
|
|
274
|
+
What this run did not check
|
|
150
275
|
|
|
151
|
-
|
|
152
|
-
|
|
276
|
+
391 of the 452 ways into this product have never been walked through.
|
|
277
|
+
A break behind any of them is invisible to this tool. Point a journey at
|
|
278
|
+
them, or run the project's own test suite as journeys.
|
|
153
279
|
```
|
|
154
280
|
|
|
155
|
-
|
|
156
|
-
|
|
157
|
-
|
|
281
|
+
A **door** is any way into your product the source declares: an HTTP route, an
|
|
282
|
+
exported function, a message channel between a desktop app's two halves, a
|
|
283
|
+
command in your `package.json`. They are read straight out of the code without
|
|
284
|
+
running anything — 5,785 of them out of one desktop app in 1.4 seconds, 452 of
|
|
285
|
+
those message channels.
|
|
158
286
|
|
|
159
|
-
|
|
287
|
+
Three rules hold the count honest:
|
|
160
288
|
|
|
161
|
-
|
|
162
|
-
|
|
163
|
-
|
|
289
|
+
- **A door read out of the source is not a door that was walked.** Knowing a
|
|
290
|
+
door exists is not evidence that anything opened it, and getting that backwards
|
|
291
|
+
would report perfect coverage on a product nobody ever ran.
|
|
292
|
+
- **Never a percentage.** A percentage invites a target, a target invites gaming,
|
|
293
|
+
and a gamed coverage number is worse than no number because somebody believes
|
|
294
|
+
it. Counts, and the names of what is missing.
|
|
295
|
+
- **Undercount rather than overcount.** Where the evidence is ambiguous the door
|
|
296
|
+
is recorded as unopened, and the reason is written down beside it.
|
|
297
|
+
|
|
298
|
+
Even a run that walked every door it knows about refuses to claim it checked
|
|
299
|
+
everything, because it did not: it checked every way in *this tool knows about*.
|
|
164
300
|
|
|
165
301
|
```
|
|
166
|
-
|
|
302
|
+
staysfixed check --json # coverage.doorsKnown, coverage.doorsWalked, coverage.gaps
|
|
167
303
|
```
|
|
168
304
|
|
|
169
|
-
|
|
305
|
+
Over MCP it is `staysfixed_coverage`. Every `staysfixed_check` reply says it in
|
|
306
|
+
words directly under the headline, and the JSON form carries `notChecked` and
|
|
307
|
+
`doorsNeverOpened` as fields of their own rather than only as prose — a number an
|
|
308
|
+
agent has to go looking for is a number it skips.
|
|
170
309
|
|
|
171
|
-
|
|
172
|
-
|
|
310
|
+
And on a clean run only, the reply also says what a clean result **on this
|
|
311
|
+
machine** actually means: *"this covers your website; your iPhone app is not
|
|
312
|
+
being checked at all, and here is why."* Nothing inside a run can know that — a
|
|
313
|
+
run only knows what it walked — so it comes from the machine survey and lands
|
|
314
|
+
beside the good news, which is the one place it cannot be missed.
|
|
173
315
|
|
|
174
|
-
|
|
175
|
-
It was still right at "v0.1.0" and already different by "v0.2.0". The change is in between.
|
|
176
|
-
2 commits landed in between:
|
|
177
|
-
3f9c1ab 2026-08-24 Move the empty state into its own component Asad Iqbal
|
|
178
|
-
77d0e42 2026-08-25 Tidy the card styles Asad Iqbal
|
|
179
|
-
files those commits touched:
|
|
180
|
-
src/billing/EmptyState.jsx
|
|
181
|
-
src/styles/cards.css
|
|
316
|
+
### A run that compared nothing is not a pass
|
|
182
317
|
|
|
183
|
-
|
|
184
|
-
|
|
318
|
+
There is one shape of clean result that would be a lie: every journey walked on
|
|
319
|
+
the new build, nothing on record from the old one, zero differences found, and a
|
|
320
|
+
verdict reading *nothing that worked has changed*. It is arithmetically true and
|
|
321
|
+
it would let a real regression through. That run comes back as **`NOTHING WAS
|
|
322
|
+
ACTUALLY COMPARED`**, it is not a pass, and it exits non-zero.
|
|
185
323
|
|
|
186
|
-
|
|
324
|
+
---
|
|
325
|
+
|
|
326
|
+
## Aiming a check at one thing
|
|
327
|
+
|
|
328
|
+
By default a check walks everything your settings describe. To aim it at one kind
|
|
329
|
+
of product:
|
|
187
330
|
|
|
188
331
|
```
|
|
189
|
-
staysfixed check --
|
|
190
|
-
staysfixed check --
|
|
191
|
-
staysfixed
|
|
192
|
-
staysfixed
|
|
193
|
-
staysfixed doctor what is missing before any of this can run
|
|
194
|
-
staysfixed mcp serve to an AI agent (see below)
|
|
332
|
+
staysfixed check --surface web --at http://localhost:3000
|
|
333
|
+
staysfixed check --surface electron --at ./release/mac-arm64/YourApp.app
|
|
334
|
+
staysfixed check --surface android --at ./app/build/outputs/apk/release/app.apk
|
|
335
|
+
staysfixed check --surface ios --at ./build/YourApp.app
|
|
195
336
|
```
|
|
196
337
|
|
|
197
|
-
|
|
198
|
-
|
|
338
|
+
The important half is what happens when it cannot go there. A tool that quietly
|
|
339
|
+
ignored an option it did not understand would check whatever it was going to
|
|
340
|
+
check anyway and hand back a **perfectly clean result about the wrong thing** —
|
|
341
|
+
the most dangerous shape a reply can have. So:
|
|
199
342
|
|
|
200
|
-
|
|
201
|
-
|
|
202
|
-
|
|
343
|
+
- aimed at a kind of product this project does not contain → it refuses, by name,
|
|
344
|
+
and nothing is checked;
|
|
345
|
+
- aimed at a kind of product this copy of the tool has no adapter for → it
|
|
346
|
+
refuses, and names the adapter that is missing;
|
|
347
|
+
- given an address no adapter would ever read → it refuses rather than dropping it;
|
|
348
|
+
- and a run that *did* go where it was aimed says so, so a clean result can be
|
|
349
|
+
trusted to be about the thing you named.
|
|
203
350
|
|
|
204
|
-
|
|
205
|
-
6 screens in the settings
|
|
206
|
-
3 guards
|
|
207
|
-
2 known-good markers
|
|
208
|
-
newest marker: v0.1.0 — pinned 2 days ago
|
|
351
|
+
---
|
|
209
352
|
|
|
210
|
-
|
|
211
|
-
✓ Everything that worked still works.
|
|
212
|
-
```
|
|
353
|
+
## Where the approval line sits
|
|
213
354
|
|
|
214
|
-
|
|
355
|
+
The word "approve" was hiding four different decisions.
|
|
356
|
+
|
|
357
|
+
1. **What counts as working** — you, and only you. But never by opening this
|
|
358
|
+
tool. The reference is cut by something you already do: saying ship. You
|
|
359
|
+
approve in bulk, retrospectively, by shipping.
|
|
360
|
+
2. **Is this difference real or is it noise** — the machine, arithmetically, from
|
|
361
|
+
running the new build twice. No judgement, nobody's opinion.
|
|
362
|
+
3. **Did my own edit cause this** — the agent. That is a *causal* claim, which is
|
|
363
|
+
checkable: revert the suspect hunk, run again, and if the difference survives
|
|
364
|
+
the revert the agent was wrong and it escalates.
|
|
365
|
+
4. **Is an unintended difference acceptable anyway** — you. This is the only thing
|
|
366
|
+
that reaches a person, and it should be a handful of items a month.
|
|
367
|
+
|
|
368
|
+
**An agent can check, and it can waive within limits. It can never decide what
|
|
369
|
+
"working" means.** That is cut by shipping, by a person, and there is no tool on
|
|
370
|
+
the MCP surface that could move it — not refused, not on the list.
|
|
371
|
+
|
|
372
|
+
An agent's only door is a waiver, and it passes four machine-checked gates:
|
|
373
|
+
|
|
374
|
+
1. **Sealed classes are unwaivable.** Whatever the reason, whoever is asking.
|
|
375
|
+
2. **The waiver has to agree with an intent sealed *before* the run** — so the
|
|
376
|
+
agent says what it meant to change before it sees what broke. Sealing one
|
|
377
|
+
afterwards is refused, and the refusal says so in those words.
|
|
378
|
+
3. **Five waivers between one ship and the next.** Past five it is not a change
|
|
379
|
+
with side effects, it is a rewrite, and a person looks at a rewrite. Sealing
|
|
380
|
+
another intent does not buy five more.
|
|
381
|
+
4. **Every waiver is fingerprinted to one exact difference** and dies the moment
|
|
382
|
+
the reference moves. Change the value it was written about and it stops
|
|
383
|
+
covering anything.
|
|
215
384
|
|
|
216
|
-
|
|
385
|
+
Every waiver is counted out loud in the reply. "Nothing changed", "nothing ran"
|
|
386
|
+
and "everything was waived" read identically otherwise, and two of those three
|
|
387
|
+
are a safety net quietly announcing success.
|
|
217
388
|
|
|
218
|
-
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
|
|
222
|
-
|
|
389
|
+
### The five sealed classes, in full
|
|
390
|
+
|
|
391
|
+
These go to a person whatever any agent believes, and there is no setting that
|
|
392
|
+
turns them off:
|
|
393
|
+
|
|
394
|
+
| | |
|
|
395
|
+
| --- | --- |
|
|
396
|
+
| **a bug somebody already reported once** | A guard exists because this exact thing broke before and somebody had to say so. A difference here means it is back. |
|
|
397
|
+
| **a crash** | The product stopped, or started stopping. Nothing about that can be intended. |
|
|
398
|
+
| **losing data** | Code can be edited back. Data that was deleted cannot. |
|
|
399
|
+
| **money** | A charge, a price or a refund that goes out wrong costs a real person real money. |
|
|
400
|
+
| **signing in** | Getting this wrong locks the right people out, or lets the wrong people in. |
|
|
401
|
+
|
|
402
|
+
### Saying ship
|
|
223
403
|
|
|
224
404
|
```
|
|
225
|
-
staysfixed
|
|
405
|
+
staysfixed ship --why "0.14.0 to TestFlight"
|
|
226
406
|
```
|
|
227
407
|
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
which needs no window at all and prints where the seconds went when the run ends:
|
|
233
|
-
[docs/watching.md](docs/watching.md).
|
|
408
|
+
One line at the end of your release script, after the thing has actually gone
|
|
409
|
+
out. The build you shipped becomes the standard every later check is compared
|
|
410
|
+
against; every outstanding waiver is retired, because what it covered has either
|
|
411
|
+
shipped and become normal or has to be decided again.
|
|
234
412
|
|
|
235
|
-
|
|
413
|
+
It never fails your release. And it **refuses to make a build the standard if
|
|
414
|
+
that build was never checked, or was checked and found broken** — your release
|
|
415
|
+
still succeeds, it just tells you what "working" means did not move, and why.
|
|
416
|
+
Cutting a broken build as the standard is exactly how a safety net turns into a
|
|
417
|
+
rubber stamp.
|
|
236
418
|
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
A
|
|
240
|
-
|
|
241
|
-
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
|
|
257
|
-
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
|
|
262
|
-
7. **Blackout boxes.** Anything genuinely allowed to change — a live clock, a
|
|
263
|
-
session id, a "3 minutes ago" — is painted over on **both** pictures, so
|
|
264
|
-
adding a mask never forces a re-approval.
|
|
265
|
-
8. **Sensible tolerance.** 0.05% of pixels by default: enough to absorb hinting
|
|
266
|
-
noise, nowhere near enough to hide a missing stylesheet or a shifted column.
|
|
267
|
-
9. **A flake register.** Every run is remembered. A check that changes its mind
|
|
268
|
-
while the code stood still is recorded, and past the limit it is condemned and
|
|
269
|
-
says so in red until a person fixes it or deletes it. There is no option to
|
|
270
|
-
tolerate one.
|
|
271
|
-
|
|
272
|
-
**The honest caveat.** A picture is tied to the operating system that took it. A
|
|
273
|
-
picture approved on macOS will not match on Linux — the font stack is different,
|
|
274
|
-
the fallback faces are different, and the text rasteriser is a different piece of
|
|
275
|
-
code. No flag fixes this. Approved pictures are stamped with the platform that
|
|
276
|
-
took them and comparing across platforms warns you.
|
|
277
|
-
|
|
278
|
-
Two ways to live with it. **Take the pictures in one place** — approve on CI, or
|
|
279
|
-
on one machine everyone shares. That is simpler and it is what most projects
|
|
280
|
-
should do. Or **approve per platform**, by setting `dir` from an environment
|
|
281
|
-
variable so each platform keeps its own approved folder.
|
|
282
|
-
|
|
283
|
-
The long version, with what each trick cannot fix, is in
|
|
284
|
-
[docs/how-it-stays-stable.md](docs/how-it-stays-stable.md).
|
|
419
|
+
### What actually reaches you
|
|
420
|
+
|
|
421
|
+
A handful of things a month, in three sentences each — what changed, why no agent
|
|
422
|
+
could wave it through, and what to do — inside the closing summary you were
|
|
423
|
+
reading anyway. Not a report, not a dashboard, not a link:
|
|
424
|
+
|
|
425
|
+
```
|
|
426
|
+
Stays Fixed: 1 thing needs your word on terminal-deck.
|
|
427
|
+
|
|
428
|
+
1. The checkout total now says 9.99 where it said 10.00
|
|
429
|
+
No agent may wave this through: it touches money.
|
|
430
|
+
Say whether that is the amount you wanted. If it is, shipping makes it the
|
|
431
|
+
new normal; if it is not, nothing ships.
|
|
432
|
+
```
|
|
433
|
+
|
|
434
|
+
Five things can land there and four of them are rare on purpose: a sealed class;
|
|
435
|
+
the waiver budget running out with differences still outstanding; something that
|
|
436
|
+
used to give the same answer every run and now does not; a check that could not
|
|
437
|
+
run at all; and, once, the fact that you have never shipped with the hook in
|
|
438
|
+
place so there is nothing to compare against yet. A clean run produces none of
|
|
439
|
+
them and says one line: *nothing needs your word.*
|
|
440
|
+
|
|
441
|
+
```
|
|
442
|
+
staysfixed check --escalations notes.txt # the same block, in a file
|
|
443
|
+
```
|
|
285
444
|
|
|
286
445
|
---
|
|
287
446
|
|
|
288
447
|
## For AI agents (MCP)
|
|
289
448
|
|
|
290
|
-
An agent that has just changed twenty files has no way of knowing whether it
|
|
291
|
-
broke the settings page, because it never opened the settings page. With this
|
|
292
|
-
wired in, it can check before it tells you it is done.
|
|
293
|
-
|
|
294
449
|
Claude Code:
|
|
295
450
|
|
|
296
451
|
```
|
|
297
452
|
claude mcp add staysfixed -- npx -y staysfixed mcp
|
|
298
453
|
```
|
|
299
454
|
|
|
300
|
-
Cursor, Gemini CLI
|
|
301
|
-
|
|
455
|
+
Cursor, Gemini CLI and most other clients take the same block — `.cursor/mcp.json`,
|
|
456
|
+
`~/.gemini/settings.json`, or your project's `.mcp.json`:
|
|
302
457
|
|
|
303
458
|
```json
|
|
304
459
|
{
|
|
@@ -321,227 +476,204 @@ args = ["-y", "staysfixed", "mcp"]
|
|
|
321
476
|
cwd = "/absolute/path/to/your/project"
|
|
322
477
|
```
|
|
323
478
|
|
|
324
|
-
The tools an agent gets:
|
|
325
|
-
|
|
326
479
|
| Tool | What it does |
|
|
327
480
|
| --- | --- |
|
|
328
|
-
| `
|
|
329
|
-
| `
|
|
330
|
-
| `
|
|
331
|
-
| `
|
|
332
|
-
| `
|
|
333
|
-
|
|
334
|
-
**
|
|
335
|
-
|
|
336
|
-
|
|
337
|
-
|
|
338
|
-
|
|
339
|
-
|
|
340
|
-
|
|
341
|
-
|
|
481
|
+
| `staysfixed_capabilities` | **Call this first, once per session.** What it can check on this machine right now, what it cannot and why, what is missing that would unlock more, and the exact shape of every reply. It runs nothing. After this call an agent should not need to read any documentation about this tool. |
|
|
482
|
+
| `staysfixed_intent` | Seal what you **meant** to change, before you run a check. This is what makes a later "that one was me" claim checkable instead of a story. |
|
|
483
|
+
| `staysfixed_check` | Run it. Returns only the differences you did not account for, ranked with the ones furthest from your edit at the top. Unchanged paths never reach you; the reply says how many were skipped, and what was never looked at. Takes `surface` and `at` to aim it at a web page, a desktop app, an APK or a simulator build — and refuses by name rather than checking something else. |
|
|
484
|
+
| `staysfixed_explain` | One finding, in depth — both values in full, the journey that reached it, the code around it, the evidence. Never pushed into a check reply, so ask for it on the two or three you intend to act on. |
|
|
485
|
+
| `staysfixed_prove` | Test a causal claim by undoing a change and running again. If the difference survives the revert, your edit did not cause it and you were about to fix the wrong thing. |
|
|
486
|
+
| `staysfixed_waive` | Record that a difference was intended. Not approval, and it makes nothing the new normal — only shipping does that. Four gates, and a refusal is final. |
|
|
487
|
+
| `staysfixed_coverage` | What was **not** checked: the doors no journey has ever opened, the surfaces this machine cannot reach, the surfaces this *copy* has no adapter for, anything refused for being irreversible, and what it can never see on any machine. Read it before telling anyone a change is safe. |
|
|
488
|
+
|
|
489
|
+
**An agent can check; only a person can approve.** `staysfixed_approve` is not
|
|
490
|
+
merely refused — it is not on the tool list at all unless the project explicitly
|
|
491
|
+
opts in, so the agent never sees a door to push on. An agent that could bless its
|
|
492
|
+
own results would edit the code, notice something moved, approve it, and report
|
|
493
|
+
success, and your safety net would have become a rubber stamp.
|
|
494
|
+
|
|
495
|
+
Full wiring for every client: [docs/mcp.md](docs/mcp.md).
|
|
496
|
+
|
|
497
|
+
### Nothing here should need a human to read documentation
|
|
498
|
+
|
|
499
|
+
Every version ships knowing, in machine-readable form and in plain English: what
|
|
500
|
+
it can check on this machine right now and what it cannot; what is missing that
|
|
501
|
+
would unlock more, and whether the tool can install it itself or a person has to;
|
|
502
|
+
which other machines it can already reach, **found by dialling them** rather than
|
|
503
|
+
by asking you; and the shape of its own results, so an agent can act on them
|
|
504
|
+
without being taught. That is `staysfixed doctor --json`, and it is
|
|
505
|
+
`staysfixed_capabilities` over MCP.
|
|
342
506
|
|
|
343
507
|
---
|
|
344
508
|
|
|
345
|
-
##
|
|
509
|
+
## The nets that are already shipped
|
|
346
510
|
|
|
347
|
-
|
|
511
|
+
### Guards — one check per bug that was already fixed once
|
|
348
512
|
|
|
349
513
|
```js
|
|
350
|
-
|
|
514
|
+
// .staysfixed/guards/the-sidebar-still-collapses.js
|
|
351
515
|
export default {
|
|
352
|
-
|
|
353
|
-
|
|
354
|
-
|
|
355
|
-
|
|
356
|
-
|
|
357
|
-
|
|
358
|
-
start: 'npm run preview', // optional command that starts the app
|
|
359
|
-
browser: '/path/to/chrome', // optional; found on the system by default
|
|
360
|
-
headless: true, // default true
|
|
361
|
-
|
|
362
|
-
// electron:
|
|
363
|
-
// binary: '/Applications/Your App.app/Contents/MacOS/Your App',
|
|
364
|
-
// args: ['--skip-onboarding'],
|
|
365
|
-
// windowMatch: 'Your App', // only drive the window whose title/url contains this
|
|
366
|
-
|
|
367
|
-
cwd: '.', // working directory for start / binary
|
|
368
|
-
env: { NODE_ENV: 'production' }, // extra environment for the launched process
|
|
369
|
-
startTimeoutMs: 60000, // how long to wait for the app to answer
|
|
370
|
-
debugPort: 9333, // default: a free one is picked
|
|
371
|
-
// attach: 'http://127.0.0.1:9333' // drive something already running instead of launching
|
|
516
|
+
name: 'the sidebar still collapses',
|
|
517
|
+
because: 'A CSS rename broke the toggle handler and it shipped unnoticed for four days.',
|
|
518
|
+
async run(app) {
|
|
519
|
+
await app.open('/');
|
|
520
|
+
await app.click('[data-action="toggle-sidebar"]');
|
|
521
|
+
await app.expect('the sidebar is hidden', async () => !(await app.page.visible('.sidebar')));
|
|
372
522
|
},
|
|
523
|
+
};
|
|
524
|
+
```
|
|
373
525
|
|
|
374
|
-
|
|
375
|
-
|
|
376
|
-
|
|
377
|
-
|
|
378
|
-
height: 900,
|
|
379
|
-
deviceScaleFactor: 2, // 2 = retina-sharp, still deterministic
|
|
380
|
-
mobile: false, // emulate a touch device
|
|
381
|
-
},
|
|
526
|
+
The name is not decoration. It is what prints when the guard fails and what an
|
|
527
|
+
agent reads before deciding whether it broke something. So names are enforced:
|
|
528
|
+
`sidebar_collapse_test` and `#4412` are refused, with an explanation and, where
|
|
529
|
+
one can honestly be built, a rewrite. See [docs/guards.md](docs/guards.md).
|
|
382
530
|
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
clock: '2026-01-01T12:00:00.000Z', // the instant the app believes it is; false = leave time alone
|
|
386
|
-
timezone: 'UTC',
|
|
387
|
-
locale: 'en-US',
|
|
388
|
-
motion: true, // kill animations, transitions, video, smooth scroll
|
|
389
|
-
random: 'seeded', // 'seeded' or 'off'
|
|
390
|
-
seed: 20260101,
|
|
391
|
-
fonts: true, // wait for fonts and images, pin text rendering
|
|
392
|
-
network: 'block-external', // 'block-external' | 'replay' | 'live'
|
|
393
|
-
networkAllow: ['https://fonts.gstatic.com/**'], // globs let out even when blocking
|
|
394
|
-
hideScrollbars: true,
|
|
395
|
-
hideCaret: true, // the text cursor blinks; hide it
|
|
396
|
-
settle: {
|
|
397
|
-
frames: 2, // identical photos in a row before we accept one
|
|
398
|
-
intervalMs: 250,
|
|
399
|
-
timeoutMs: 10000,
|
|
400
|
-
maxDriftPixels: 0, // pixels allowed to differ and still count as identical
|
|
401
|
-
},
|
|
402
|
-
},
|
|
531
|
+
Guards are the third net, for the case both engines are blind to: **the old build
|
|
532
|
+
was already wrong.** A difference against a guard is sealed — it goes to a person.
|
|
403
533
|
|
|
404
|
-
|
|
405
|
-
tolerance: {
|
|
406
|
-
pixels: 0.0005, // share of pixels allowed to differ, 0..1
|
|
407
|
-
threshold: 0.12, // per-pixel colour sensitivity, lower = stricter
|
|
408
|
-
antialiasing: true, // ignore anti-aliasing noise
|
|
409
|
-
maxPixels: 500, // a hard cap; overrides `pixels` when set
|
|
410
|
-
},
|
|
534
|
+
### Walk — the last look before a release
|
|
411
535
|
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
masks: ['[data-live-clock]', { x: 0, y: 0, width: 240, height: 32 }],
|
|
416
|
-
|
|
417
|
-
// --- The screens ---------------------------------------------------------
|
|
418
|
-
screens: [
|
|
419
|
-
{
|
|
420
|
-
name: 'billing-empty', // file-safe id; becomes the picture's file name
|
|
421
|
-
describe: 'Billing with no invoices yet', // shown to humans
|
|
422
|
-
url: '/billing', // shorthand for a single goto step
|
|
423
|
-
|
|
424
|
-
// Or a list of steps, which also works in staysfixed.config.json:
|
|
425
|
-
steps: [
|
|
426
|
-
{ goto: '/billing' }, // navigate; relative resolves against app.url
|
|
427
|
-
{ waitFor: '.invoice-list' }, // wait for a selector
|
|
428
|
-
{ waitForGone: '.spinner' }, // wait for one to disappear
|
|
429
|
-
{ scrollTo: '#totals' }, // scroll an element into view
|
|
430
|
-
{ hover: '.plan-card' },
|
|
431
|
-
{ click: 'button.new' },
|
|
432
|
-
{ type: 'input[name="q"]', text: 'hello' }, // type into a field
|
|
433
|
-
{ press: 'Enter' },
|
|
434
|
-
{ evaluate: 'window.scrollTo(0, 0)' }, // run JavaScript in the page
|
|
435
|
-
{ wait: 200 }, // last resort; settle usually beats this
|
|
436
|
-
{ note: 'A human note, shown in reports.' },
|
|
437
|
-
],
|
|
438
|
-
|
|
439
|
-
// Or code, when the steps need a decision (JS config only):
|
|
440
|
-
// async do(page) { await page.goto('/billing'); await page.click('#tab'); },
|
|
441
|
-
|
|
442
|
-
masks: ['.invoice-date'], // extra masks for this screen only
|
|
443
|
-
tolerance: { pixels: 0.001 }, // override tolerance for this screen only
|
|
444
|
-
viewport: { width: 720 }, // override the size for this screen only
|
|
445
|
-
freeze: { settle: { timeoutMs: 20000 } }, // per-screen freeze overrides
|
|
446
|
-
clip: '[data-plan="pro"]', // photograph only this element
|
|
447
|
-
fullPage: false, // photograph the whole scrollable page
|
|
448
|
-
skip: false, // leave it out for now, without deleting it
|
|
449
|
-
},
|
|
450
|
-
],
|
|
451
|
-
|
|
452
|
-
// --- Guards: one check per bug already fixed once ------------------------
|
|
453
|
-
guards: '.staysfixed/guards', // folder of plain JavaScript files
|
|
454
|
-
|
|
455
|
-
// --- The pre-release walk -----------------------------------------------
|
|
456
|
-
walk: {
|
|
457
|
-
describe: 'What a reviewer clicks through before a release',
|
|
458
|
-
steps: [ /* same shape as screens; defaults to `screens` */ ],
|
|
459
|
-
},
|
|
536
|
+
```
|
|
537
|
+
staysfixed walk --open
|
|
538
|
+
```
|
|
460
539
|
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
|
|
464
|
-
|
|
465
|
-
},
|
|
540
|
+
Opens the real built app, visits each screen, photographs every step onto one
|
|
541
|
+
page you can scroll in thirty seconds. Nothing is compared and nothing can fail
|
|
542
|
+
on a pixel. This net answers the question a comparison cannot: *does the thing I
|
|
543
|
+
am about to ship actually open?*
|
|
466
544
|
|
|
467
|
-
|
|
468
|
-
|
|
469
|
-
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
};
|
|
545
|
+
### Markers — pin a known-good moment
|
|
546
|
+
|
|
547
|
+
```
|
|
548
|
+
staysfixed mark v0.15.0 --note "before the store work"
|
|
549
|
+
staysfixed trace billing-empty
|
|
473
550
|
```
|
|
474
551
|
|
|
475
|
-
A
|
|
476
|
-
|
|
477
|
-
without anybody writing JavaScript.
|
|
552
|
+
A marker defines what "old" means. Comparison runs over stored build artifacts,
|
|
553
|
+
so tracing a regression to a commit does not need every commit rebuilt.
|
|
478
554
|
|
|
479
|
-
|
|
480
|
-
[`examples/`](examples/): [a web app](examples/staysfixed.config.web.js),
|
|
481
|
-
[an Electron app](examples/staysfixed.config.electron.js), and
|
|
482
|
-
[a guard](examples/guards/the-sidebar-still-collapses.js).
|
|
555
|
+
### Picture checks — version 1, unchanged
|
|
483
556
|
|
|
484
|
-
|
|
557
|
+
```
|
|
558
|
+
staysfixed check --pictures
|
|
559
|
+
staysfixed approve --all
|
|
560
|
+
```
|
|
485
561
|
|
|
486
|
-
|
|
562
|
+
Pixels dropped from the accusation to the evidence, but the version 1 picture
|
|
563
|
+
check is still here, still works, and still requires a person to approve. Nobody
|
|
564
|
+
who was using it has to stop.
|
|
487
565
|
|
|
488
|
-
|
|
566
|
+
### The freeze layer, which everything rests on
|
|
489
567
|
|
|
490
|
-
|
|
491
|
-
|
|
492
|
-
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
nothing else in the suite matters.
|
|
568
|
+
Frozen clock, killed motion, seeded randomness, pinned fonts and text rendering,
|
|
569
|
+
blocked or replayed network, and capture-until-two-frames-agree. Paired running
|
|
570
|
+
does not make this redundant: paired running removes differences between the two
|
|
571
|
+
builds, and this removes the product's own internal nondeterminism, which is what
|
|
572
|
+
keeps the measured wobble small enough to be useful. The long version, with what
|
|
573
|
+
each trick cannot fix, is in
|
|
574
|
+
[docs/how-it-stays-stable.md](docs/how-it-stays-stable.md).
|
|
498
575
|
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
stylesheet. Every one of the 11 pictures failed, and so did the guard written for
|
|
503
|
-
exactly that bug:
|
|
576
|
+
The network interceptor has been promoted from a determinism trick to the
|
|
577
|
+
**safety boundary**. It is the answer to "what about a payment": the old build
|
|
578
|
+
replays recorded traffic and never reaches the real world.
|
|
504
579
|
|
|
505
|
-
|
|
506
|
-
|
|
507
|
-
|
|
508
|
-
✗ files looks different — 187,242 pixels changed
|
|
509
|
-
...
|
|
510
|
-
✗ the app still has its styling
|
|
511
|
-
expected: the window is not plain white
|
|
512
|
-
why this guard exists: one release shipped with the whole app unstyled and
|
|
513
|
-
every one of its ~3,600 tests passed, because none of them could see it.
|
|
514
|
-
```
|
|
580
|
+
---
|
|
581
|
+
|
|
582
|
+
## Settings
|
|
515
583
|
|
|
516
|
-
|
|
584
|
+
Everything is optional except `app`. A five-line file works, and
|
|
585
|
+
`staysfixed init` writes one you can read. A `staysfixed.config.json` works too,
|
|
586
|
+
with a declarative `steps` form and no functions — so a Rust, Python or Go
|
|
587
|
+
project can use the tool without anybody writing JavaScript.
|
|
588
|
+
|
|
589
|
+
Two fully commented examples are in [`examples/`](examples/):
|
|
590
|
+
[a web app](examples/staysfixed.config.web.js),
|
|
591
|
+
[an Electron app](examples/staysfixed.config.electron.js), and
|
|
592
|
+
[a guard](examples/guards/the-sidebar-still-collapses.js).
|
|
517
593
|
|
|
518
|
-
|
|
519
|
-
|
|
520
|
-
That rule applies to the tool's own checks too.
|
|
594
|
+
The full reference lives with the code it configures, and the design behind all
|
|
595
|
+
of it is in [docs/how-v2-works.md](docs/how-v2-works.md).
|
|
521
596
|
|
|
522
597
|
---
|
|
523
598
|
|
|
524
|
-
##
|
|
599
|
+
## Does it actually work?
|
|
600
|
+
|
|
601
|
+
A tool that reports "nothing changed" looks exactly like a tool that is broken,
|
|
602
|
+
and there is no way to tell the two apart from the outside. So:
|
|
603
|
+
|
|
604
|
+
**It has to prove it still catches things.** `staysfixed check --selfcheck` runs
|
|
605
|
+
a corpus of deliberately broken builds and requires the engine to catch every
|
|
606
|
+
one. If it misses any, it says so, and until that is fixed a clean check means
|
|
607
|
+
nothing.
|
|
525
608
|
|
|
526
|
-
|
|
609
|
+
**The unstable app.** `fixtures/unstable-app` is a page built to be impossible to
|
|
610
|
+
observe consistently: a clock ticking ten times a second, an endless spinner, a
|
|
611
|
+
tween, a shuffled list, a random uuid, a blinking caret, a late image, and a feed
|
|
612
|
+
the server answers differently every time. The suite runs it twenty times and
|
|
613
|
+
requires every result to be identical. If that fails, nothing else in the suite
|
|
614
|
+
matters.
|
|
615
|
+
|
|
616
|
+
**A real desktop app.** Pointed at a real Electron application: eleven screens,
|
|
617
|
+
two guards, about twenty-five seconds a run, five consecutive runs with nothing
|
|
618
|
+
different. Then one line was deleted from the built app — the `<link>` to its
|
|
619
|
+
stylesheet. All eleven failed, and so did the guard written for exactly that bug,
|
|
620
|
+
whose reason reads: *one release shipped with the whole app unstyled and every one
|
|
621
|
+
of its ~3,600 tests passed, because none of them could see it.*
|
|
622
|
+
|
|
623
|
+
---
|
|
527
624
|
|
|
625
|
+
## What it will never do
|
|
626
|
+
|
|
627
|
+
Honestly, so you know before you invest an afternoon.
|
|
628
|
+
|
|
629
|
+
- **Nothing irreversible, ever.** Anything that spends money, sends a message or
|
|
630
|
+
destroys data is watched at the moment it is **asked for** — the same charge,
|
|
631
|
+
the same amount, the same place — and refused at the effect. If a bug only
|
|
632
|
+
appears after the payment settles or the email lands, this tool is blind to it,
|
|
633
|
+
by design and permanently. A refusal is reported as a gap in coverage, never as
|
|
634
|
+
a pass.
|
|
635
|
+
- **A migration that destroys data is refused, not run twice.**
|
|
636
|
+
- **A race that already existed will not show.** Subtracting the wobble floor
|
|
637
|
+
actively hides intermittent bugs. Running the new build twice recovers half of
|
|
638
|
+
this by flagging anything newly unstable. Only half. That is the sharpest
|
|
639
|
+
weakness in the whole architecture and it is not going to be dressed up.
|
|
640
|
+
- **A waiver is a judgement, and judgements can be wrong.** The gates make an
|
|
641
|
+
agent's claim falsifiable — it has to be written before the damage is visible,
|
|
642
|
+
it has to fall inside what was named, there are five of them, and they all die
|
|
643
|
+
when you ship. What they cannot do is read the agent's mind. The five sealed
|
|
644
|
+
classes are the answer to that: in the places where being wrong is expensive,
|
|
645
|
+
no judgement is accepted from any agent at all.
|
|
646
|
+
- **Real phones cannot be paired.** No paired run is possible on a device in your
|
|
647
|
+
hand. Real iPhones and real Android handsets fall back to comparing against the
|
|
648
|
+
stored record, and say so out loud on every run.
|
|
649
|
+
- **Native Windows cannot run two builds at once, even in principle,** because
|
|
650
|
+
Windows shows one desktop at a time.
|
|
651
|
+
- **It is not every possible state.** "Deep" means every door the code exposes and
|
|
652
|
+
every journey your suite already walks. Nothing can enumerate every state, and
|
|
653
|
+
any tool claiming otherwise is lying. The coverage ledger names the doors it has
|
|
654
|
+
never opened, so the hole is visible instead of pretended away.
|
|
528
655
|
- **No hosted service, no dashboard, no accounts, no teams, nothing paid.** It is
|
|
529
|
-
a command and a folder of files in your repository.
|
|
530
|
-
|
|
531
|
-
|
|
532
|
-
|
|
533
|
-
- **
|
|
534
|
-
|
|
535
|
-
|
|
536
|
-
|
|
537
|
-
|
|
538
|
-
|
|
539
|
-
|
|
540
|
-
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
544
|
-
|
|
656
|
+
a command and a folder of files in your repository.
|
|
657
|
+
- **Pictures still do not travel between operating systems.** Text is drawn
|
|
658
|
+
differently on every system. Pixels are evidence now rather than the accusation,
|
|
659
|
+
which makes this matter far less than it did — but it has not gone away.
|
|
660
|
+
- **Phones cannot be paired, and the iPhone is not covered yet.** Android runs on
|
|
661
|
+
an emulator against the stored record, which is weaker than a paired run and
|
|
662
|
+
says so every time. The iOS simulator is designed and not built. If you ship a
|
|
663
|
+
phone app, ask `doctor` what it is actually covering before you trust a clean
|
|
664
|
+
result — it will tell you plainly rather than let a green run mean less than it
|
|
665
|
+
looks like.
|
|
666
|
+
- **Native Windows shows one desktop, so two builds cannot run at once even in
|
|
667
|
+
principle.** Runs are one after the other and the same-minute guarantee is
|
|
668
|
+
weaker there than on any platform. If your Windows product is Electron — most
|
|
669
|
+
are — it is covered properly instead, over the debug port, from any machine.
|
|
670
|
+
Nothing irreversible can be refused on Windows either: there is no way to block
|
|
671
|
+
a compiled program's network call without administrator rights, so a journey
|
|
672
|
+
marked irreversible is refused outright rather than walked.
|
|
673
|
+
- **Not battle-tested.** It works, it is used, and it has not yet met the thousand
|
|
674
|
+
strange apps a widely-used tool meets. If it reports something that is not true,
|
|
675
|
+
that is the most serious kind of bug it can have — please
|
|
676
|
+
[open an issue](https://github.com/asadev/staysfixed/issues).
|
|
545
677
|
|
|
546
678
|
---
|
|
547
679
|
|