@sfqa/scar 0.0.0-stage → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +29 -0
- package/LICENSE +21 -0
- package/README.md +863 -2
- package/dist/capture.d.ts +20 -0
- package/dist/capture.js +42 -0
- package/dist/cli.d.ts +80 -0
- package/dist/cli.js +622 -0
- package/dist/harvest.d.ts +82 -0
- package/dist/harvest.js +511 -0
- package/dist/hash.d.ts +1 -0
- package/dist/hash.js +10 -0
- package/dist/ignore.d.ts +17 -0
- package/dist/ignore.js +38 -0
- package/dist/index.d.ts +13 -0
- package/dist/index.js +14 -0
- package/dist/local.d.ts +17 -0
- package/dist/local.js +163 -0
- package/dist/new-rule.d.ts +39 -0
- package/dist/new-rule.js +209 -0
- package/dist/prove.d.ts +37 -0
- package/dist/prove.js +158 -0
- package/dist/redact.d.ts +3 -0
- package/dist/redact.js +20 -0
- package/dist/report.d.ts +23 -0
- package/dist/report.js +137 -0
- package/dist/rules.d.ts +38 -0
- package/dist/rules.js +166 -0
- package/dist/scan.d.ts +38 -0
- package/dist/scan.js +404 -0
- package/dist/types.d.ts +77 -0
- package/dist/types.js +11 -0
- package/package.json +59 -4
- package/prompts/extract-rules.md +80 -0
- package/prompts/turn-incident-into-rule.md +82 -0
- package/rules/generic.json +518 -0
package/README.md
CHANGED
|
@@ -1,3 +1,864 @@
|
|
|
1
|
-
#
|
|
1
|
+
# @sfqa/scar
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Your test suite already knows what went wrong. This turns those failures into
|
|
4
|
+
rules a reviewer can run.
|
|
5
|
+
|
|
6
|
+
A rule is one past incident: the line not to repeat, how bad recurrence would be,
|
|
7
|
+
and a pattern a machine can match. A finding is that scar firing again - with the
|
|
8
|
+
origin and the fix that already worked once. No model at review time. Same input,
|
|
9
|
+
same findings, every time.
|
|
10
|
+
|
|
11
|
+
```bash
|
|
12
|
+
npm install -D @sfqa/scar
|
|
13
|
+
npx sfqa-scar . # scan with the shipped template (or your scar-tissue.json)
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
```text
|
|
17
|
+
[HIGH] ST-004 - Never pass a timeout to isVisible() - the option is ignored and the call returns instantly. Use waitFor({ state: 'visible', timeout }) when render timing matters.
|
|
18
|
+
where : test-pages/web/LeadPage.ts:212
|
|
19
|
+
matched: const visible = await label.isVisible({ timeout: 5000 });
|
|
20
|
+
context: All Playwright page objects (test-pages/web/**), field readers and verification helpers reading Lightning labels.
|
|
21
|
+
origin : incident
|
|
22
|
+
A field reader called isVisible({ timeout: 5000 }); Playwright ignores options there and returns immediately, so a Lightning label still rendering read as absent and the value was silently reported as empty.
|
|
23
|
+
do this: Replace with `await locator.waitFor({ state: 'visible', timeout: N }).then(() => true).catch(() => false)` - a visible wait that cannot silently report a stale DOM.
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Exit `0` when nothing at or above `--fail-on` matches, `1` otherwise. Put it in a
|
|
27
|
+
hook or a CI job and it runs without anyone remembering.
|
|
28
|
+
|
|
29
|
+
## Two files, two jobs
|
|
30
|
+
|
|
31
|
+
| File | What it is | When it appears |
|
|
32
|
+
| --- | --- | --- |
|
|
33
|
+
| `packages/scar/rules/generic.json` (packaged) | The **template** - 19 worked-example rules shipped with the tool. Used automatically when a repo has no local rule file. | Ships with the package. |
|
|
34
|
+
| `scar-tissue.json` (in your repo root) | **Your** rule set - the rules you wrote, tuned, or accepted-until-later. Picked up automatically, no `--rules` flag. | Created the first time you run `sfqa-scar new …`, or by copying the template to adopt and tune it. |
|
|
35
|
+
|
|
36
|
+
Plain scan never writes a rule file. Copying the template is how you take
|
|
37
|
+
ownership of the rules (and demote an accepted finding with a reason, instead of
|
|
38
|
+
turning it off):
|
|
39
|
+
|
|
40
|
+
```bash
|
|
41
|
+
cp node_modules/@sfqa/scar/rules/generic.json ./scar-tissue.json # adopt + tune (optional)
|
|
42
|
+
npx sfqa-scar . # reads scar-tissue.json automatically
|
|
43
|
+
```
|
|
44
|
+
|
|
45
|
+
## Why this, and not a linter or an AI reviewer
|
|
46
|
+
|
|
47
|
+
- **A linter** knows syntax and style. It does not know that `isVisible({timeout})`
|
|
48
|
+
silently lied to you in March. A scar rule cites the incident and the fix.
|
|
49
|
+
- **An AI reviewer** is non-deterministic and forgets. This is plain matching over
|
|
50
|
+
local files - no network, no keys - so the same input gives the same findings
|
|
51
|
+
every time, which is the only way a merge gate earns the right to fail a build.
|
|
52
|
+
- **AI has one, bounded job:** drafting. `harvest` surfaces candidates from
|
|
53
|
+
history, and the shipped prompts (`prompts/extract-rules.md`,
|
|
54
|
+
`prompts/turn-incident-into-rule.md`) help turn an incident into a rule. The
|
|
55
|
+
result is still just data you review and commit.
|
|
56
|
+
- **Every rule is proved, not believed.** A fixture must make it fire and a clean
|
|
57
|
+
fixture must keep it silent (`sfqa-scar prove`). A rule with no detector is
|
|
58
|
+
reported as a gap, never counted as coverage. The proof caught a rule in this
|
|
59
|
+
very set whose detector could never fire - and, on a real run, caught the
|
|
60
|
+
detectors themselves misreading clean code (see [Proof and honesty](#proof-and-honesty)).
|
|
61
|
+
|
|
62
|
+
## Adding a rule is a prompt, not a regex
|
|
63
|
+
|
|
64
|
+
Most rules tools ask you to write a detector. This one asks you to describe the
|
|
65
|
+
incident, because the incident is the part you already know:
|
|
66
|
+
|
|
67
|
+
> "In March a field reader called `isVisible({ timeout: 5000 })`. Playwright
|
|
68
|
+
> ignores options there and returns immediately, so a label that was still
|
|
69
|
+
> rendering read as absent and the test passed with an empty field. Never pass
|
|
70
|
+
> options to `isVisible()`."
|
|
71
|
+
|
|
72
|
+
That paragraph is the whole job. It becomes a rule like this:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
npx sfqa-scar new --rule "Never pass a timeout to isVisible() - it is ignored" --scope "test-pages/**"
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
which writes three files: the rule entry, one fixture that must match, and one
|
|
79
|
+
that must not. You fill in the sentence and one line of pattern; `sfqa-scar
|
|
80
|
+
prove` tells you whether the rule fires where it should and stays silent where
|
|
81
|
+
it should not. If you have no idea what the pattern should be, hand the
|
|
82
|
+
paragraph to an assistant with
|
|
83
|
+
[`prompts/turn-incident-into-rule.md`](prompts/turn-incident-into-rule.md) - it
|
|
84
|
+
asks for exactly the rule, the detector and the two fixtures, and refuses to
|
|
85
|
+
accept a rule that cites no incident.
|
|
86
|
+
|
|
87
|
+
There is a full worked example, from incident to committed rule, in
|
|
88
|
+
[Add a rule after an incident](#add-a-rule-after-an-incident). The rest of this
|
|
89
|
+
README is detail you can read when you need it.
|
|
90
|
+
|
|
91
|
+
## Why this exists
|
|
92
|
+
|
|
93
|
+
A Salesforce suite accumulates scar tissue: the toast that vanishes on the
|
|
94
|
+
slower sandbox, the `isVisible` call that ignores its own timeout, the picklist
|
|
95
|
+
label that drifted one capital letter, the credential that reached git history.
|
|
96
|
+
Each one cost a day. Each one is written down in a retro and then forgotten.
|
|
97
|
+
|
|
98
|
+
`@sfqa/scar` keeps them as data. The rule set is JSON; the detectors are
|
|
99
|
+
declarative; the scanner is dependency-free Node. A new team member can read
|
|
100
|
+
the history without reading the commit log.
|
|
101
|
+
|
|
102
|
+
## Not just Salesforce
|
|
103
|
+
|
|
104
|
+
The scanner is a text-file engine: scoped globs, regex detectors, no Salesforce
|
|
105
|
+
API and no browser. It will run over Cypress, Selenium, Appium, API tests, or
|
|
106
|
+
plain application code.
|
|
107
|
+
|
|
108
|
+
The shipped rules are the part that is Salesforce/Playwright-shaped, because that
|
|
109
|
+
is the history they were mined from. A rule like `.isVisible({` is inert in a
|
|
110
|
+
Cypress repo. That is not a limitation of the engine; it is the point. Regenerate
|
|
111
|
+
the rule set from your own history with `prompts/extract-rules.md`, and the same
|
|
112
|
+
binary reviews your stack. The 19 rules are a worked example of the format, not a
|
|
113
|
+
universal pack.
|
|
114
|
+
|
|
115
|
+
## What it does not do
|
|
116
|
+
|
|
117
|
+
- It does not learn on its own. A new incident becomes a rule only when you add
|
|
118
|
+
one, or re-run extraction. The runtime is deterministic.
|
|
119
|
+
- It does not read live runs or server-side logs. It reads source files, diffs,
|
|
120
|
+
and the local report artifacts you point at with `--report`.
|
|
121
|
+
- It cannot name the offending source line from a runtime failure. A failing test
|
|
122
|
+
gives you the test, the file and the error - that is a brief, not a draft. Turning
|
|
123
|
+
a brief into a rule is the step that needs a person (or a model working for one).
|
|
124
|
+
- It does not run itself. Wire it into a pre-commit hook, a commit-msg hook, or a
|
|
125
|
+
CI job; then it runs without anyone remembering.
|
|
126
|
+
|
|
127
|
+
## Run it on every commit
|
|
128
|
+
|
|
129
|
+
Three places, all short.
|
|
130
|
+
|
|
131
|
+
**Pre-commit** (husky v9):
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
npm install -D husky
|
|
135
|
+
npx husky init
|
|
136
|
+
echo "npx sfqa-scar --staged --fail-on high" > .husky/pre-commit
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
Older husky: `npx husky add .husky/pre-commit "npx sfqa-scar --staged --fail-on high"`.
|
|
140
|
+
|
|
141
|
+
**Commit-msg** (optional, for incident capture):
|
|
142
|
+
|
|
143
|
+
```bash
|
|
144
|
+
echo 'npx sfqa-scar harvest --staged --message-file "$1" --ask' > .husky/commit-msg
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
This one asks a single question when the commit message looks like a fix, and
|
|
148
|
+
records a "no" so it is not asked again. See
|
|
149
|
+
[Ask at commit time](#ask-at-commit-time-remember-the-no).
|
|
150
|
+
|
|
151
|
+
**CI** (GitHub Actions):
|
|
152
|
+
|
|
153
|
+
```yaml
|
|
154
|
+
name: scar
|
|
155
|
+
on: [pull_request]
|
|
156
|
+
|
|
157
|
+
jobs:
|
|
158
|
+
review:
|
|
159
|
+
runs-on: ubuntu-latest
|
|
160
|
+
steps:
|
|
161
|
+
- uses: actions/checkout@v4
|
|
162
|
+
with:
|
|
163
|
+
fetch-depth: 0 # --base needs the history
|
|
164
|
+
- uses: actions/setup-node@v4
|
|
165
|
+
with:
|
|
166
|
+
node-version: 20
|
|
167
|
+
- run: npm ci
|
|
168
|
+
- run: npx sfqa-scar --base origin/${{ github.base_ref }} --fail-on high
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
**CI** (Azure DevOps, advisory - reports, never gates the build):
|
|
172
|
+
|
|
173
|
+
```yaml
|
|
174
|
+
- job: ScarReview
|
|
175
|
+
displayName: 'Scar Tissue Review (advisory)'
|
|
176
|
+
steps:
|
|
177
|
+
- task: NodeTool@0
|
|
178
|
+
inputs: { versionSpec: '20.x' }
|
|
179
|
+
- script: |
|
|
180
|
+
out=scar-review.md
|
|
181
|
+
npx --yes @sfqa/scar --markdown --fail-on high . > "$out" || true
|
|
182
|
+
cat "$out"
|
|
183
|
+
echo "##vso[task.complete result=SucceededWithIssues;]Scar review written (advisory)."
|
|
184
|
+
continueOnError: true
|
|
185
|
+
- task: PublishBuildArtifacts@1
|
|
186
|
+
inputs: { PathtoPublish: 'scar-review.md', ArtifactName: 'scar-review' }
|
|
187
|
+
condition: always()
|
|
188
|
+
```
|
|
189
|
+
|
|
190
|
+
To make it a gate, drop `|| true` and `continueOnError` and let the exit code
|
|
191
|
+
fail the job. Advisory is a good first step: the findings are real, but you
|
|
192
|
+
decide when a red build is earned.
|
|
193
|
+
|
|
194
|
+
That is the whole "automatic" story. No daemon, no network, no API key. Once
|
|
195
|
+
those hooks exist, nobody has to remember to run it.
|
|
196
|
+
|
|
197
|
+
## Day one: the hook does not add rules
|
|
198
|
+
|
|
199
|
+
The pre-commit hook and the CI job are **enforcement**. They apply rules that
|
|
200
|
+
already exist. They never write one.
|
|
201
|
+
|
|
202
|
+
The commit-msg hook is adjacent: it never blocks either. It only asks whether a
|
|
203
|
+
fix-shaped change still owes a rule, and a "no" records itself.
|
|
204
|
+
|
|
205
|
+
That is deliberate. A scanner that invented a rule from a failed commit would be
|
|
206
|
+
guessing about intent, and a rule you cannot defend from history is a style
|
|
207
|
+
opinion. Writing the rule is the one step that needs a person - or a model
|
|
208
|
+
working for one - who knows what actually broke.
|
|
209
|
+
|
|
210
|
+
So on day one the rule file comes from one of three places:
|
|
211
|
+
|
|
212
|
+
1. **Copy the template.** Use this if your stack is close to Salesforce +
|
|
213
|
+
Playwright. If you just want to evaluate the shipped set, skip the copy
|
|
214
|
+
entirely: the packaged `rules/generic.json` is the default.
|
|
215
|
+
2. **Extract from history.** One pass with
|
|
216
|
+
[`prompts/extract-rules.md`](prompts/extract-rules.md) before you wire the
|
|
217
|
+
hook. This is the right start for any other stack.
|
|
218
|
+
3. **An empty rule file.** Valid, and it enforces nothing. The hook runs, finds
|
|
219
|
+
zero findings, exits 0. Honest, but useless.
|
|
220
|
+
|
|
221
|
+
The order that actually works:
|
|
222
|
+
|
|
223
|
+
```bash
|
|
224
|
+
# 1. seed a rule file you own (skip if you want the packaged default)
|
|
225
|
+
cp node_modules/@sfqa/scar/rules/generic.json ./scar-tissue.json
|
|
226
|
+
|
|
227
|
+
# 2. see what it says before it can fail a commit
|
|
228
|
+
npx sfqa-scar .
|
|
229
|
+
|
|
230
|
+
# 3. wire it in once you trust the output (see "Run it on every commit")
|
|
231
|
+
npx husky init
|
|
232
|
+
echo "npx sfqa-scar --staged --fail-on high" > .husky/pre-commit
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
A `scar-tissue.json` in the working directory is picked up automatically, so no
|
|
236
|
+
`--rules` flag is needed. Pass `--rules` only to point somewhere else.
|
|
237
|
+
|
|
238
|
+
After that, the loop is:
|
|
239
|
+
|
|
240
|
+
```text
|
|
241
|
+
a test fails for a real reason
|
|
242
|
+
-> the postmortem names the bad line
|
|
243
|
+
-> one rule + two fixtures (prompts/turn-incident-into-rule.md)
|
|
244
|
+
-> commit the rule
|
|
245
|
+
-> the hook enforces it on every future commit
|
|
246
|
+
```
|
|
247
|
+
|
|
248
|
+
The hook did not add the rule. **The incident did.** The postmortem is the
|
|
249
|
+
teacher; the hook is the memory. If a team installs the hook with an empty rule
|
|
250
|
+
file and never runs the retro step, it will correctly do nothing forever - and
|
|
251
|
+
that is the boundary of the product, not a bug in it.
|
|
252
|
+
|
|
253
|
+
## The standing loop
|
|
254
|
+
|
|
255
|
+
Adoption is one loop, and it repeats:
|
|
256
|
+
|
|
257
|
+
1. **A failure happens.** Someone fixes it, or a test fails for a real reason and
|
|
258
|
+
the retro names the bad line.
|
|
259
|
+
2. **A rule appears.** Hand-written, or harvested from history and drafted.
|
|
260
|
+
3. **The rule gets proved.** Positive fixture fires, clean fixture stays quiet.
|
|
261
|
+
4. **The rule gets committed.** With the incident, in the same commit.
|
|
262
|
+
5. **The hook enforces.** Pre-commit and CI, on every commit after that.
|
|
263
|
+
6. **Harvest keeps watch.** Weekly sweep for unclaimed incidents; CI goes red if
|
|
264
|
+
any remain.
|
|
265
|
+
|
|
266
|
+
Nothing in this loop depends on anyone remembering to do anything. The only
|
|
267
|
+
remembering left is the one decision the tool cannot make for you: whether a
|
|
268
|
+
draft is worth turning into a rule. Everything else is wired.
|
|
269
|
+
|
|
270
|
+
## Find the rules you never wrote
|
|
271
|
+
|
|
272
|
+
The gap in every "write a rule after an incident" workflow is the incident
|
|
273
|
+
nobody wrote a rule for. `harvest` closes that gap from three sources:
|
|
274
|
+
|
|
275
|
+
| Source | What it shows | Command |
|
|
276
|
+
|---|---|---|
|
|
277
|
+
| Git history | fix-shaped commits no rule accounts for | `harvest` |
|
|
278
|
+
| The staged change | the incident you are committing right now | `harvest --staged --message ...` |
|
|
279
|
+
| Local test reports | failures you are living with, repeated across runs | `harvest --report test-results/` |
|
|
280
|
+
|
|
281
|
+
```bash
|
|
282
|
+
npx sfqa-scar harvest # last 300 commits
|
|
283
|
+
npx sfqa-scar harvest --since 2026-01-01 # a date range
|
|
284
|
+
npx sfqa-scar harvest --all # also show incidents a rule covers
|
|
285
|
+
npx sfqa-scar harvest --write scar-candidates.json --json
|
|
286
|
+
npx sfqa-scar harvest --strict # exit 1 if any incident has no rule
|
|
287
|
+
|
|
288
|
+
npx sfqa-scar harvest --staged --message "fix: replace the fixed sleep"
|
|
289
|
+
npx sfqa-scar harvest --report test-results/ --report-format auto
|
|
290
|
+
```
|
|
291
|
+
|
|
292
|
+
What it does:
|
|
293
|
+
|
|
294
|
+
- reads commits whose message says fix, revert, hotfix, flaky, race, timeout,
|
|
295
|
+
workaround, credential, secret, rollback, regression, bug, defect or patch;
|
|
296
|
+
- ignores an incident already claimed by a rule's `origin.commit`, or whose
|
|
297
|
+
removed lines match an existing detector;
|
|
298
|
+
- for each unclaimed incident, shows the commit, the removed bad line, the added
|
|
299
|
+
fix, a proposed `pathInclude`, a proposed `linePatterns` regex, and a
|
|
300
|
+
confidence;
|
|
301
|
+
- writes nothing. You review the draft, then add it.
|
|
302
|
+
|
|
303
|
+
```text
|
|
304
|
+
CANDIDATE 1/1
|
|
305
|
+
commit : 402ef2d 2026-11-09
|
|
306
|
+
subject: fix: replace fixed sleep with an element wait
|
|
307
|
+
files : tests/orders.ts
|
|
308
|
+
removed: await page.waitForTimeout(2000);
|
|
309
|
+
added : await locator.waitFor({ state: 'visible', timeout: 5000 });
|
|
310
|
+
detect : page\.waitForTimeout\( [confidence high]
|
|
311
|
+
scope : tests/**
|
|
312
|
+
id : commit:402ef2d9c1...
|
|
313
|
+
```
|
|
314
|
+
|
|
315
|
+
From there it is the loop from the previous section: prove the draft with a
|
|
316
|
+
positive fixture and a clean fixture, then commit. `--strict` in CI turns "we
|
|
317
|
+
forgot to write a rule for last month's hotfix" into a red build, which is the
|
|
318
|
+
only way the gap becomes visible. Run the history pass on a schedule, not on every
|
|
319
|
+
push, so the gate stays fast:
|
|
320
|
+
|
|
321
|
+
```yaml
|
|
322
|
+
name: scar-harvest
|
|
323
|
+
on:
|
|
324
|
+
schedule:
|
|
325
|
+
- cron: '0 6 * * 1' # Mondays
|
|
326
|
+
workflow_dispatch:
|
|
327
|
+
|
|
328
|
+
jobs:
|
|
329
|
+
gaps:
|
|
330
|
+
runs-on: ubuntu-latest
|
|
331
|
+
steps:
|
|
332
|
+
- uses: actions/checkout@v4
|
|
333
|
+
with:
|
|
334
|
+
fetch-depth: 0
|
|
335
|
+
- uses: actions/setup-node@v4
|
|
336
|
+
with:
|
|
337
|
+
node-version: 20
|
|
338
|
+
- run: npm ci
|
|
339
|
+
- run: npx sfqa-scar harvest --since 2026-01-01 --strict
|
|
340
|
+
```
|
|
341
|
+
|
|
342
|
+
## Ask at commit time, remember the no
|
|
343
|
+
|
|
344
|
+
The best moment to capture an incident is while the person still remembers it:
|
|
345
|
+
the commit that fixes it. A `commit-msg` hook has the staged diff *and* the
|
|
346
|
+
message, so it can ask a single question and then get out of the way.
|
|
347
|
+
|
|
348
|
+
```bash
|
|
349
|
+
# .husky/pre-commit - fast, no prompts, no writes
|
|
350
|
+
npx sfqa-scar --staged --fail-on high
|
|
351
|
+
|
|
352
|
+
# .husky/commit-msg - one question, only when the message looks like an incident
|
|
353
|
+
npx sfqa-scar harvest --staged --message-file "$1" --ask
|
|
354
|
+
```
|
|
355
|
+
|
|
356
|
+
Why two hooks instead of one:
|
|
357
|
+
|
|
358
|
+
- **Pre-commit scans.** It only reads the staged files and the existing rules.
|
|
359
|
+
No `git log` walk, no network, no prompt. It stays fast enough to run on every
|
|
360
|
+
commit.
|
|
361
|
+
- **Commit-msg harvests.** Git has no commit message until this hook, so
|
|
362
|
+
message-based incident detection cannot happen earlier. It uses the staged
|
|
363
|
+
diff plus `COMMIT_EDITMSG`, asks once, and exits `0` either way - a `commit-msg`
|
|
364
|
+
hook must not block a commit because someone declined to write a rule.
|
|
365
|
+
|
|
366
|
+
When the message matches the incident keywords, `--ask` shows the candidate and
|
|
367
|
+
offers three answers:
|
|
368
|
+
|
|
369
|
+
```text
|
|
370
|
+
capture a draft for staged "fix: replace the fixed sleep with an element wait"? [y]es / [i]gnore / [s]kip:
|
|
371
|
+
```
|
|
372
|
+
|
|
373
|
+
| Answer | What happens | Next run |
|
|
374
|
+
|---|---|---|
|
|
375
|
+
| `y` | the draft is saved to `scar-tissue.candidates.json` | **not asked again**; still shown as `captured: yes`, because no rule exists yet |
|
|
376
|
+
| `i` | the decision is written to `scar-tissue.ignore.json` | **not asked again**; listed with `--all` |
|
|
377
|
+
| `s` | nothing is written | asked again next time |
|
|
378
|
+
|
|
379
|
+
So a "yes" is not a vibe. It is a queued draft with the incident, the scope and
|
|
380
|
+
the proposed detector, waiting for someone to move it into the rule set. A "no"
|
|
381
|
+
is a recorded decision. Both are files you can review in a PR.
|
|
382
|
+
|
|
383
|
+
```json
|
|
384
|
+
// scar-tissue.candidates.json - yes
|
|
385
|
+
{
|
|
386
|
+
"version": 1,
|
|
387
|
+
"candidates": [
|
|
388
|
+
{
|
|
389
|
+
"fingerprint": "staged:9f2c...",
|
|
390
|
+
"capturedAt": "2026-11-09T09:12:00.000Z",
|
|
391
|
+
"subject": "fix: replace the fixed sleep with an element wait",
|
|
392
|
+
"files": ["tests/orders.ts"],
|
|
393
|
+
"draft": { "id": "ST-020", "match": { "linePatterns": ["page\\.waitForTimeout\\("] } }
|
|
394
|
+
}
|
|
395
|
+
]
|
|
396
|
+
}
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
```json
|
|
400
|
+
// scar-tissue.ignore.json - no
|
|
401
|
+
{
|
|
402
|
+
"version": 1,
|
|
403
|
+
"ignored": [
|
|
404
|
+
{
|
|
405
|
+
"fingerprint": "staged:9f2c...",
|
|
406
|
+
"reason": "the fix is already covered by the code review",
|
|
407
|
+
"at": "2026-11-09T09:12:00.000Z",
|
|
408
|
+
"subject": "fix: replace the fixed sleep with an element wait"
|
|
409
|
+
}
|
|
410
|
+
]
|
|
411
|
+
}
|
|
412
|
+
```
|
|
413
|
+
|
|
414
|
+
That means you can audit a decision, diff it in a PR, and reverse it by deleting
|
|
415
|
+
the entry. `--ignore <id>` and `--capture <id>` record the same decisions
|
|
416
|
+
non-interactively, which is useful from a CI job or when someone dismisses a
|
|
417
|
+
candidate they saw in a report. `--all` brings dismissed candidates back, with
|
|
418
|
+
`ignoredAt` and `ignoreReason`.
|
|
419
|
+
|
|
420
|
+
A captured candidate disappears on its own once a rule covers it: the coverage
|
|
421
|
+
check runs first, so adding the rule with the right `origin.commit` or detector
|
|
422
|
+
closes the loop without hand-editing the queue.
|
|
423
|
+
|
|
424
|
+
`--ask` only prompts for `--staged`. History is reviewed in bulk with `--write`;
|
|
425
|
+
a hundred interactive questions about last year's commits is not a workflow.
|
|
426
|
+
|
|
427
|
+
## Local test runs are the third source
|
|
428
|
+
|
|
429
|
+
A test report shows the failures a team is currently living with. Point
|
|
430
|
+
`--report` at a file or a directory and harvest the two formats runners emit
|
|
431
|
+
most: Playwright JSON and JUnit XML. Format is sniffed from the file unless you
|
|
432
|
+
pass `--report-format playwright|junit`.
|
|
433
|
+
|
|
434
|
+
```bash
|
|
435
|
+
npx playwright test --reporter=json
|
|
436
|
+
npx sfqa-scar harvest --report test-results/ --json --write local-failures.json
|
|
437
|
+
```
|
|
438
|
+
|
|
439
|
+
The same failure across several runs is merged and counted, because repetition
|
|
440
|
+
is the signal:
|
|
441
|
+
|
|
442
|
+
```text
|
|
443
|
+
LOCAL FAILURE 1/1
|
|
444
|
+
test : orders.spec.ts > creates an order > creates an order
|
|
445
|
+
file : tests/orders.spec.ts
|
|
446
|
+
status : failed [4 occurrence(s)]
|
|
447
|
+
error : Error: expect(locator).toBeVisible() failed
|
|
448
|
+
Locator: getByText('Order created')
|
|
449
|
+
next : find the source line behind this failure, then write one rule with two fixtures
|
|
450
|
+
id : test:3a91...
|
|
451
|
+
```
|
|
452
|
+
|
|
453
|
+
These are **briefs, not drafts**. A runtime error tells you that something
|
|
454
|
+
broke; it does not tell you which source line caused it, so the tool will not
|
|
455
|
+
invent a regex for it. The brief gives you the test, the file, the error and a
|
|
456
|
+
stable id, which is exactly the input the
|
|
457
|
+
[incident-to-rule prompt](prompts/turn-incident-into-rule.md) expects. Local
|
|
458
|
+
failures are also filtered by the same ignore ledger, so `--ignore <id>` works
|
|
459
|
+
on them too.
|
|
460
|
+
|
|
461
|
+
Two limits, stated plainly:
|
|
462
|
+
|
|
463
|
+
- History and staged mode read a **diff**. A failure that was never fixed and
|
|
464
|
+
never staged is invisible there; the local-report source is how you see it.
|
|
465
|
+
- The proposed regex is a **draft**. A low-confidence draft deliberately has no
|
|
466
|
+
detector; the tool will not guess a pattern it cannot defend.
|
|
467
|
+
|
|
468
|
+
Credential-shaped lines are redacted before they reach stdout, the JSON or a
|
|
469
|
+
draft. The harvester will not write the secret it found into a rule file.
|
|
470
|
+
|
|
471
|
+
## The rule set is a template, not a product
|
|
472
|
+
|
|
473
|
+
The shipped set (`rules/generic.json`) has 19 rules. Their **detectors** are
|
|
474
|
+
generic. Where a detector needs to know a value, the value list is an example
|
|
475
|
+
that you replace:
|
|
476
|
+
|
|
477
|
+
| Detector input | Shipped example | Replace with |
|
|
478
|
+
|---|---|---|
|
|
479
|
+
| `labelRules.canonical` | `Web`, `Event`, `Referral`, `Phone`, `Email`, `SMS` | the exact labels your org renders |
|
|
480
|
+
| inventory-code shape | `SKU-A12-0042` | the identifier shape your product uses |
|
|
481
|
+
| local-only file patterns | `local-notes-*.md`, `*.zip` | the paths your team keeps local |
|
|
482
|
+
| timeout ceiling | `< 360000` ms | your measured longest journey, minus headroom |
|
|
483
|
+
|
|
484
|
+
That is deliberate. A generic rule set that pretended to know your org would be
|
|
485
|
+
worse than no rule set. Extract once from your own history, keep the examples
|
|
486
|
+
only as documentation.
|
|
487
|
+
|
|
488
|
+
## Where the rules come from
|
|
489
|
+
|
|
490
|
+
Three sources, in increasing order of value:
|
|
491
|
+
|
|
492
|
+
1. **The shipped template.** Eighteen rules mined from one team's
|
|
493
|
+
Salesforce/Playwright history. Treat it as a worked example of the format,
|
|
494
|
+
and as a usable starting point only if your stack is close to that one.
|
|
495
|
+
2. **History harvest.** Run `npx sfqa-scar harvest` first. It reads your git
|
|
496
|
+
log, finds fix-shaped commits that no rule covers, and prints a draft rule for
|
|
497
|
+
each one. Deterministic, no LLM. For lessons that need judgement across a
|
|
498
|
+
whole diff, the deeper pass is
|
|
499
|
+
[`prompts/extract-rules.md`](prompts/extract-rules.md): it reads `git log`,
|
|
500
|
+
opens commits whose messages say fix, revert, hotfix, flaky, race, timeout,
|
|
501
|
+
workaround or credentials, and emits rules that cite a commit hash. A lesson
|
|
502
|
+
without a hash is dropped.
|
|
503
|
+
3. **One rule per incident.** This is the loop that keeps the rule set alive.
|
|
504
|
+
After a postmortem, write one rule before the memory fades. Ten minutes, and
|
|
505
|
+
it is the only ongoing maintenance the tool needs.
|
|
506
|
+
|
|
507
|
+
So how do you know which rules to add? You do not pick them from a catalogue.
|
|
508
|
+
You take them from incidents that already cost you a day. If you cannot name the
|
|
509
|
+
bad line and the day it burned you, it is a lint opinion, not scar tissue. That
|
|
510
|
+
is the whole selection criterion.
|
|
511
|
+
|
|
512
|
+
`harvest` exists because that criterion still requires someone to remember the
|
|
513
|
+
incident. The tool reads the history for you and puts the unclaimed ones on
|
|
514
|
+
screen. You still approve each rule; you no longer have to notice the gap
|
|
515
|
+
yourself.
|
|
516
|
+
|
|
517
|
+
## Add a rule after an incident
|
|
518
|
+
|
|
519
|
+
One worked example, end to end, using a failure every automation suite has seen:
|
|
520
|
+
a test that waited on a fixed sleep instead of the element, passed locally, and
|
|
521
|
+
failed in CI when the environment was slower.
|
|
522
|
+
|
|
523
|
+
**1. Start from the incident, not from the regex.** Write the sentence a reviewer
|
|
524
|
+
would need, then let the tool scaffold the rest. Seed a rule file you own first,
|
|
525
|
+
so the 19 shipped rules stay and your new one becomes ST-020:
|
|
526
|
+
|
|
527
|
+
```bash
|
|
528
|
+
cp node_modules/@sfqa/scar/rules/generic.json scar-tissue.json
|
|
529
|
+
|
|
530
|
+
npx sfqa-scar new \
|
|
531
|
+
--rule "Never wait on a fixed sleep for a UI element; poll the element with a real timeout." \
|
|
532
|
+
--scope "tests/" \
|
|
533
|
+
--severity med
|
|
534
|
+
```
|
|
535
|
+
|
|
536
|
+
```text
|
|
537
|
+
created ST-020
|
|
538
|
+
|
|
539
|
+
rule scar-tissue.json
|
|
540
|
+
positive scar-fixtures/positive/tests/st-020.ts
|
|
541
|
+
clean scar-fixtures/negative/tests/st-020.ts
|
|
542
|
+
|
|
543
|
+
what to do next
|
|
544
|
+
1. open scar-tissue.json, replace the TODOs, and put the pattern in match.linePatterns
|
|
545
|
+
2. not sure of the pattern? prompts/turn-incident-into-rule.md (in the package) drafts it from the incident
|
|
546
|
+
3. paste the bad line into scar-fixtures/positive/tests/st-020.ts
|
|
547
|
+
4. paste the same code with the fix into scar-fixtures/negative/tests/st-020.ts
|
|
548
|
+
5. prove just this rule: sfqa-scar prove --id ST-020
|
|
549
|
+
6. it is scoped to tests/**; a fixture outside that path is never scanned
|
|
550
|
+
```
|
|
551
|
+
|
|
552
|
+
`--scope` takes a plain directory (`tests/` means everything under it) or a glob
|
|
553
|
+
(`tests/**/*.ts`). Repeat it if the rule belongs in more than one place; a
|
|
554
|
+
fixture pair is written for each.
|
|
555
|
+
|
|
556
|
+
Three files exist, the id is free, and the fixtures are already in a path the
|
|
557
|
+
rule scans - which is the mistake people make when they place them by hand. The
|
|
558
|
+
rule starts with an empty detector on purpose, so nothing can claim it is
|
|
559
|
+
enforced before it is.
|
|
560
|
+
|
|
561
|
+
**2. Fill in the two TODOs.** The scaffold gives you the shape; the incident
|
|
562
|
+
gives you the words:
|
|
563
|
+
|
|
564
|
+
```json
|
|
565
|
+
{
|
|
566
|
+
"id": "ST-020",
|
|
567
|
+
"rule": "Never wait on a fixed sleep for a UI element; poll the element with a real timeout.",
|
|
568
|
+
"context": "Playwright page objects and helpers under tests/**",
|
|
569
|
+
"origin": {
|
|
570
|
+
"commit": "9f2c41d",
|
|
571
|
+
"summary": "waitForTimeout(2000) passed locally and failed in CI on a slower environment; a fixed sleep is a bet on machine speed."
|
|
572
|
+
},
|
|
573
|
+
"severity": "med",
|
|
574
|
+
"detection_pattern": "page.waitForTimeout(...) used to wait for UI state",
|
|
575
|
+
"match": {
|
|
576
|
+
"pathInclude": ["tests/**"],
|
|
577
|
+
"pathExclude": ["**/node_modules/**"],
|
|
578
|
+
"linePatterns": ["\\.waitForTimeout\\(\\s*\\d+\\s*\\)"]
|
|
579
|
+
},
|
|
580
|
+
"recommendation": "Replace with await locator.waitFor({ state: 'visible', timeout: N })."
|
|
581
|
+
}
|
|
582
|
+
```
|
|
583
|
+
|
|
584
|
+
**3. Prove it both ways.** Put the bad line in the positive fixture, the same
|
|
585
|
+
code with the fix in the clean one, then ask for a verdict on one rule:
|
|
586
|
+
|
|
587
|
+
```bash
|
|
588
|
+
npx sfqa-scar prove --id ST-020
|
|
589
|
+
```
|
|
590
|
+
|
|
591
|
+
```text
|
|
592
|
+
sfqa-scar prove - 1 rule(s), 1 executable
|
|
593
|
+
rule set: scar-tissue.json
|
|
594
|
+
fixtures: scar-fixtures
|
|
595
|
+
|
|
596
|
+
rule severity detector positive clean
|
|
597
|
+
------ -------- --------- --------- -------
|
|
598
|
+
ST-020 med yes fires clean
|
|
599
|
+
|
|
600
|
+
PROOF: 1/1 executable rules fire on a positive fixture and stay silent on the clean set.
|
|
601
|
+
Coverage: 1/1 rules enforced today; 0 rule(s) have no detector.
|
|
602
|
+
```
|
|
603
|
+
|
|
604
|
+
Read it as a sentence:
|
|
605
|
+
|
|
606
|
+
- `fires` - the bad line was found. If this says `missing`, the detector is wrong
|
|
607
|
+
or the fixture is outside the rule's scope; the message prints that scope.
|
|
608
|
+
- `clean` - the fixed line was not found. If this says `leak`, the pattern is too
|
|
609
|
+
broad and would fail good code.
|
|
610
|
+
- `detector: NO` - the rule is prose. `prove` reports it as a gap and exits 1 if
|
|
611
|
+
nothing else fires, because a rule that cannot fire is a comment with a
|
|
612
|
+
version number.
|
|
613
|
+
|
|
614
|
+
**4. Scan for real.**
|
|
615
|
+
|
|
616
|
+
```bash
|
|
617
|
+
npx sfqa-scar --fail-on med tests/
|
|
618
|
+
```
|
|
619
|
+
|
|
620
|
+
If the rule set fails to load, the error names the rule and the field; detectors
|
|
621
|
+
are validated before any file is read.
|
|
622
|
+
|
|
623
|
+
**5. Commit the rule, the fixtures, and the incident summary together.** The
|
|
624
|
+
commit message is the postmortem. Reviewers can see what changed and why, and
|
|
625
|
+
the next person inherits the lesson without reading the incident channel.
|
|
626
|
+
|
|
627
|
+
That is the loop: incident, retro, one rule, two fixtures, commit. It runs once
|
|
628
|
+
per incident, not once per commit.
|
|
629
|
+
|
|
630
|
+
### If you do not want the scaffold
|
|
631
|
+
|
|
632
|
+
The format is plain JSON and nothing depends on `new`. Append the object above
|
|
633
|
+
to the `rules` array in `scar-tissue.json` by hand, put the two fixtures wherever
|
|
634
|
+
your rule's `pathInclude` points, and run `sfqa-scar prove`. The scaffold only
|
|
635
|
+
removes the two mechanical decisions - which id is free, and where the fixture
|
|
636
|
+
belongs.
|
|
637
|
+
|
|
638
|
+
[`prompts/turn-incident-into-rule.md`](prompts/turn-incident-into-rule.md) is a
|
|
639
|
+
fill-in-the-blanks prompt for step 1 if you want an assistant to draft the rule
|
|
640
|
+
from an incident write-up.
|
|
641
|
+
|
|
642
|
+
## Choosing a starter set
|
|
643
|
+
|
|
644
|
+
The engine does not change between stacks; the rule set does. What survives
|
|
645
|
+
depends on what your repo actually contains:
|
|
646
|
+
|
|
647
|
+
| Your repo | Start with | Then extract your own |
|
|
648
|
+
|---|---|---|
|
|
649
|
+
| Salesforce + Playwright | Most of the shipped set (ST-001 to ST-017, ST-019). Re-check `labelRules.canonical` against your org first. | Auth and session reuse, record navigation, picklist drift, toast races |
|
|
650
|
+
| Playwright, non-Salesforce | ST-004, ST-009, ST-017, ST-018. Drop every `labelRules` and record/Lightning rule. | Waits, fixture reuse, your real timeout ceiling, selector drift |
|
|
651
|
+
| Cypress, Selenium or Appium | ST-009, ST-017, ST-018 only. `.isVisible({` is inert here. | Framework-specific waits, session reuse, selector strategy |
|
|
652
|
+
| API or integration tests | ST-001, ST-002, ST-007, ST-008, ST-018. Nothing DOM-shaped survives. | Retry and idempotency, environment config bleed, hardcoded fixture ids |
|
|
653
|
+
| Application code | ST-001, ST-002, ST-018. The engine handles it; the rules must too. | Error swallowing, silent fallbacks, config precedence |
|
|
654
|
+
|
|
655
|
+
The pattern behind the table: keep the rules that describe **process and safety**
|
|
656
|
+
(secrets, config bleed, hardcoded mutable ids, fixed sleeps, duplicated helpers),
|
|
657
|
+
drop the rules that name **another team's product** (Lightning, lead conversion,
|
|
658
|
+
Salesforce picklists). Then grow the set from your own incidents, one at a time.
|
|
659
|
+
|
|
660
|
+
What not to do: copy all 19 into a Cypress repo and treat a silent scan as
|
|
661
|
+
passing. A rule that can never fire is coverage theatre. If a rule is inert in
|
|
662
|
+
your stack, delete it rather than leave it in the file.
|
|
663
|
+
|
|
664
|
+
## The 19 rules
|
|
665
|
+
|
|
666
|
+
| ID | Severity | Rule |
|
|
667
|
+
|---|---|---|
|
|
668
|
+
| ST-001 | high | Never hardcode credentials, tokens or secrets in code, config fallbacks or committed `.env` files. |
|
|
669
|
+
| ST-002 | high | Each environment config reads its own environment's variables, never a sibling's. |
|
|
670
|
+
| ST-003 | high | Picklist values match the Salesforce label exactly: case, spaces, hyphens. |
|
|
671
|
+
| ST-004 | high | `isVisible()` ignores any `timeout`; use `waitFor({ state: 'visible' })`. |
|
|
672
|
+
| ST-005 | med | Never treat a success toast as the only success signal. |
|
|
673
|
+
| ST-006 | med | Record navigation tolerates Lightning URL variants; no single strict `waitForURL`. |
|
|
674
|
+
| ST-007 | med | Server-side integration fields need periodic reload, not just a deadline poll. |
|
|
675
|
+
| ST-008 | high | Never hardcode mutable inventory, accounts or orders; use a sentinel. |
|
|
676
|
+
| ST-009 | high | Timeout must exceed the measured journey, or retries write duplicate records. |
|
|
677
|
+
| ST-010 | med | Keep teardown out of the reporter-finalization window; no broad `pkill`. |
|
|
678
|
+
| ST-011 | high | Auth goes through the shared login flow; no hardcoded standard-login selectors. |
|
|
679
|
+
| ST-012 | med | Human-in-the-loop logins need race tolerance: adopt an existing session. |
|
|
680
|
+
| ST-013 | med | After impersonation or a tab switch, rebuild page objects on the active page. |
|
|
681
|
+
| ST-014 | high | Lead conversion has two outcomes; branch on duplicate detection and read identity from the record. |
|
|
682
|
+
| ST-015 | med | Field-dependency maps and picklist constants need live-org provenance. |
|
|
683
|
+
| ST-016 | low | Correlated generated fields must be co-constrained. |
|
|
684
|
+
| ST-017 | med | Search for an existing helper before adding a duplicate. |
|
|
685
|
+
| ST-018 | low | Local-only artifacts are gitignored from birth. *(no detector yet)* |
|
|
686
|
+
| ST-019 | med | Read a record header from a scoped container, never a bare comma-list that mixes a container with a value element plus `.first()`. |
|
|
687
|
+
|
|
688
|
+
18 of 19 have a machine detector and are proven both ways: a fixture makes each
|
|
689
|
+
one fire, and a clean fixture set makes none of them fire. ST-018 is reported as
|
|
690
|
+
a gap rather than counted as coverage. The source repo keeps the fixture set the packaged rules are proven against in
|
|
691
|
+
`scar-validation/`, and runs it in CI with the same `prove` command a consumer
|
|
692
|
+
uses on their own rules. The fixtures are not shipped in the tarball, because
|
|
693
|
+
they only make sense next to a rule set you are editing - `sfqa-scar new` creates
|
|
694
|
+
the first pair for you.
|
|
695
|
+
|
|
696
|
+
## Rule format
|
|
697
|
+
|
|
698
|
+
```json
|
|
699
|
+
{
|
|
700
|
+
"id": "ST-004",
|
|
701
|
+
"rule": "Never pass options to isVisible() ...",
|
|
702
|
+
"context": "All Playwright page objects ...",
|
|
703
|
+
"origin": { "commit": null, "summary": "what went wrong, as a class of incident" },
|
|
704
|
+
"severity": "high",
|
|
705
|
+
"detection_pattern": ".isVisible({ ... }) - any argument passed to isVisible, unless the call is .catch()-guarded.",
|
|
706
|
+
"match": {
|
|
707
|
+
"pathInclude": ["**/*.ts"],
|
|
708
|
+
"pathExclude": ["**/node_modules/**"],
|
|
709
|
+
"linePatterns": ["\\.isVisible\\(\\s*\\{[^{}]{0,200}?\\}\\)(?!\\s*\\.catch)"],
|
|
710
|
+
"fileExemptSignals": ["Promise\\.race"]
|
|
711
|
+
},
|
|
712
|
+
"recommendation": "Replace with waitFor({ state: 'visible', timeout: N })."
|
|
713
|
+
}
|
|
714
|
+
```
|
|
715
|
+
|
|
716
|
+
Detectors available in `match`:
|
|
717
|
+
|
|
718
|
+
| Key | Fires when |
|
|
719
|
+
|---|---|
|
|
720
|
+
| `linePatterns` | a non-comment line matches (a pattern containing `[\s\S]` may span lines) |
|
|
721
|
+
| `numericRules` | capture group 1 falls inside `[min, max]`, e.g. a too-short timeout |
|
|
722
|
+
| `fileNameRules` | the file's base name matches, e.g. a tracked `.env` |
|
|
723
|
+
| `labelRules` | a quoted literal is a near-miss of a canonical label (case, spacing, hyphen) |
|
|
724
|
+
| `fileRequiredSignals` | **none** of the signals exist in a file that is in scope |
|
|
725
|
+
|
|
726
|
+
A `linePatterns` entry sees the line plus the next three lines, and the finding
|
|
727
|
+
is reported only when the match *starts* on that line. That is how a guard
|
|
728
|
+
written underneath -
|
|
729
|
+
|
|
730
|
+
```ts
|
|
731
|
+
const there = await label.isVisible({ timeout: 5000 })
|
|
732
|
+
.catch(() => false);
|
|
733
|
+
```
|
|
734
|
+
|
|
735
|
+
- is read as a deliberate probe rather than a silent assertion. The template's
|
|
736
|
+
ST-004 expresses exactly that with a trailing `(?!\s*\.catch)`.
|
|
737
|
+
|
|
738
|
+
Two exemption scopes exist on purpose:
|
|
739
|
+
|
|
740
|
+
- `fileExemptSignals` clears the whole file when a mitigation is present
|
|
741
|
+
(a `Promise.race` guarding a toast wait).
|
|
742
|
+
- `fileExemptLines` clears one line (an empty `process.env` fallback) while a
|
|
743
|
+
sibling hardcoded literal in the same file still fires.
|
|
744
|
+
|
|
745
|
+
## Extracting your own rule set
|
|
746
|
+
|
|
747
|
+
The shipped set is a starting point. To mine your own history, give a capable
|
|
748
|
+
model the prompt in
|
|
749
|
+
[`prompts/extract-rules.md`](prompts/extract-rules.md). It asks for a commit
|
|
750
|
+
reference per rule, JavaScript regexes, exact canonical labels, and a documented
|
|
751
|
+
gap instead of an invented detector. Review the output; a rule you cannot defend
|
|
752
|
+
from history is a style opinion, not scar tissue.
|
|
753
|
+
|
|
754
|
+
After that first pass, do not re-run extraction for every small bug. Use
|
|
755
|
+
[`prompts/turn-incident-into-rule.md`](prompts/turn-incident-into-rule.md) to
|
|
756
|
+
turn one incident into one rule plus its two fixtures, and save the bulk
|
|
757
|
+
extraction for a quarterly cleanup or a new repository.
|
|
758
|
+
|
|
759
|
+
## Usage
|
|
760
|
+
|
|
761
|
+
```bash
|
|
762
|
+
# scan paths
|
|
763
|
+
npx sfqa-scar test-pages/ features/
|
|
764
|
+
|
|
765
|
+
# review only what changed
|
|
766
|
+
npx sfqa-scar --base origin/main
|
|
767
|
+
npx sfqa-scar --staged
|
|
768
|
+
|
|
769
|
+
# machine-readable, for CI or a PR comment
|
|
770
|
+
npx sfqa-scar --json --fail-on med .
|
|
771
|
+
|
|
772
|
+
# markdown, for a PR comment or a write-up (relative paths, one row per finding)
|
|
773
|
+
npx sfqa-scar --markdown --fail-on med .
|
|
774
|
+
|
|
775
|
+
# start a rule: writes the rule and both fixtures at a path the rule scans
|
|
776
|
+
npx sfqa-scar new --rule "Never pass options to isVisible()" --scope "test-pages/**"
|
|
777
|
+
|
|
778
|
+
# prove every rule against its fixtures (exit 1 if one does not)
|
|
779
|
+
npx sfqa-scar prove
|
|
780
|
+
npx sfqa-scar prove --id ST-020 # one rule, the one you just wrote
|
|
781
|
+
|
|
782
|
+
# a different rule set - one in the working directory is used automatically
|
|
783
|
+
npx sfqa-scar --rules ../shared/scar-tissue.json src/
|
|
784
|
+
|
|
785
|
+
# find incidents that never became rules
|
|
786
|
+
npx sfqa-scar harvest --since 2026-01-01
|
|
787
|
+
npx sfqa-scar harvest --write scar-candidates.json --json
|
|
788
|
+
npx sfqa-scar harvest --staged --message "fix: replace the fixed sleep" --ask
|
|
789
|
+
npx sfqa-scar harvest --report test-results/ --report-format auto
|
|
790
|
+
npx sfqa-scar harvest --all # include dismissed candidates
|
|
791
|
+
|
|
792
|
+
# dismiss one candidate, permanently, in committed data
|
|
793
|
+
npx sfqa-scar harvest --ignore staged:9f2c...
|
|
794
|
+
|
|
795
|
+
# keep one candidate as a queued draft
|
|
796
|
+
npx sfqa-scar harvest --capture staged:9f2c...
|
|
797
|
+
```
|
|
798
|
+
|
|
799
|
+
Exit codes: `0` clean at the threshold, `1` at least one finding at or above
|
|
800
|
+
`--fail-on` (default `high`), `2` bad usage or an invalid rule set.
|
|
801
|
+
|
|
802
|
+
`harvest` is advisory: it exits `0` even when it finds an unclaimed incident.
|
|
803
|
+
Add `--strict` to make a gap exit `1`, for a CI job that refuses to let last
|
|
804
|
+
month's hotfix go unrecorded.
|
|
805
|
+
|
|
806
|
+
Findings inside gitignored files are downgraded to `low` and marked
|
|
807
|
+
`[gitignored - informational]`. A properly ignored `.env` is not a leak; a
|
|
808
|
+
tracked one is.
|
|
809
|
+
|
|
810
|
+
## Design rules
|
|
811
|
+
|
|
812
|
+
- **The rule set is data.** Regenerate it, version it, review it in a PR. No
|
|
813
|
+
detector logic hides in the scanner.
|
|
814
|
+
- **Nothing is silently dropped.** A rule with no detector is listed as a gap.
|
|
815
|
+
A finding in an ignored file is shown, not hidden.
|
|
816
|
+
- **Fail at load, not at scan.** Every regex compiles and every rule is
|
|
817
|
+
validated (unique id, known severity, required prose) before a file is read.
|
|
818
|
+
- **Comments are not recurrences.** A comment that mentions a bad pattern is a
|
|
819
|
+
warning to the reader, not a hit.
|
|
820
|
+
- **Discovery never writes rules.** `harvest` proposes; a person approves. The
|
|
821
|
+
scanner has no side effects. The only file harvest writes is the ignore
|
|
822
|
+
ledger, and only when a person says "do not ask again".
|
|
823
|
+
- **A decision is data.** `scar-tissue.candidates.json` (captured) and
|
|
824
|
+
`scar-tissue.ignore.json` (dismissed) are reviewed and committed like code,
|
|
825
|
+
so a decision can be audited and reversed.
|
|
826
|
+
- **No org vocabulary in the package.** The shipped rule set is client-free.
|
|
827
|
+
|
|
828
|
+
## Tune, do not turn off
|
|
829
|
+
|
|
830
|
+
A finding you choose to live with is still information. Demote it in **your**
|
|
831
|
+
`scar-tissue.json`, not in the template: set its `severity` to `low` and record
|
|
832
|
+
the reason in `context` (for example, "accepted until test-data setup exists").
|
|
833
|
+
The scan keeps reporting it, the history stays honest, and you can reverse it the
|
|
834
|
+
day the reason stops being true. A rule lowered in the template protects nobody.
|
|
835
|
+
|
|
836
|
+
## Proof and honesty
|
|
837
|
+
|
|
838
|
+
The build fails on proof, not just on tests. `npm run validate:scar` (the same
|
|
839
|
+
`sfqa-scar prove` a consumer runs on their own rules) proves every executable rule
|
|
840
|
+
fires on a positive fixture and stays silent on a clean set; a rule with no
|
|
841
|
+
detector is reported as a gap, never counted as coverage. The first run of that
|
|
842
|
+
harness caught a rule whose cross-line detector could never fire - it sat in the
|
|
843
|
+
report looking thorough and could flag nothing. The fix and a regression fixture
|
|
844
|
+
landed before anything shipped.
|
|
845
|
+
|
|
846
|
+
Fixtures prove a detector *can* fire. They do not prove it fires on the right
|
|
847
|
+
thing, so the set was run against the repository it was mined from. 150 findings,
|
|
848
|
+
37 `high` - and about eight were real. The rest were the detectors misreading
|
|
849
|
+
code (a guarded `.isVisible(...).catch(...)` probe read as a silent assertion, a
|
|
850
|
+
field-map value read as label drift, a `map.get("password")` read as a secret).
|
|
851
|
+
Each misread became a **detector fix plus a regression test**, never a lower
|
|
852
|
+
severity in the template. Re-run: 113 findings, 11 `high`. A tool that opens with
|
|
853
|
+
false alarms gets switched off in a week; the point is to be wrong in a way you
|
|
854
|
+
can read, fix, and prove.
|
|
855
|
+
|
|
856
|
+
Read the numbers as method, not a promise: `scar-validation/REPORT.md` records the
|
|
857
|
+
breakdown, and the blog tells the story of the three catches (the scar that
|
|
858
|
+
spread, the rule that could never fire, the detectors that were the problem).
|
|
859
|
+
|
|
860
|
+
## Status
|
|
861
|
+
|
|
862
|
+
`0.1.1`. Dependency-free, Node >= 20, TypeScript types included. 18 of 19 rules
|
|
863
|
+
have a machine detector and are proven both ways; the 19th is a documented gap.
|
|
864
|
+
The public proof runs on synthetic fixtures, not a field trial. MIT licensed.
|