@sfqa/scar 0.0.0-stage → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,3 +1,864 @@
1
- # Temporary Holding Version
1
+ # @sfqa/scar
2
2
 
3
- This version is a temporary placeholder for this package. An operational version to replace this has been submitted for review and is awaiting a staged release.
3
+ Your test suite already knows what went wrong. This turns those failures into
4
+ rules a reviewer can run.
5
+
6
+ A rule is one past incident: the line not to repeat, how bad recurrence would be,
7
+ and a pattern a machine can match. A finding is that scar firing again - with the
8
+ origin and the fix that already worked once. No model at review time. Same input,
9
+ same findings, every time.
10
+
11
+ ```bash
12
+ npm install -D @sfqa/scar
13
+ npx sfqa-scar . # scan with the shipped template (or your scar-tissue.json)
14
+ ```
15
+
16
+ ```text
17
+ [HIGH] ST-004 - Never pass a timeout to isVisible() - the option is ignored and the call returns instantly. Use waitFor({ state: 'visible', timeout }) when render timing matters.
18
+ where : test-pages/web/LeadPage.ts:212
19
+ matched: const visible = await label.isVisible({ timeout: 5000 });
20
+ context: All Playwright page objects (test-pages/web/**), field readers and verification helpers reading Lightning labels.
21
+ origin : incident
22
+ A field reader called isVisible({ timeout: 5000 }); Playwright ignores options there and returns immediately, so a Lightning label still rendering read as absent and the value was silently reported as empty.
23
+ do this: Replace with `await locator.waitFor({ state: 'visible', timeout: N }).then(() => true).catch(() => false)` - a visible wait that cannot silently report a stale DOM.
24
+ ```
25
+
26
+ Exit `0` when nothing at or above `--fail-on` matches, `1` otherwise. Put it in a
27
+ hook or a CI job and it runs without anyone remembering.
28
+
29
+ ## Two files, two jobs
30
+
31
+ | File | What it is | When it appears |
32
+ | --- | --- | --- |
33
+ | `packages/scar/rules/generic.json` (packaged) | The **template** - 19 worked-example rules shipped with the tool. Used automatically when a repo has no local rule file. | Ships with the package. |
34
+ | `scar-tissue.json` (in your repo root) | **Your** rule set - the rules you wrote, tuned, or accepted-until-later. Picked up automatically, no `--rules` flag. | Created the first time you run `sfqa-scar new …`, or by copying the template to adopt and tune it. |
35
+
36
+ Plain scan never writes a rule file. Copying the template is how you take
37
+ ownership of the rules (and demote an accepted finding with a reason, instead of
38
+ turning it off):
39
+
40
+ ```bash
41
+ cp node_modules/@sfqa/scar/rules/generic.json ./scar-tissue.json # adopt + tune (optional)
42
+ npx sfqa-scar . # reads scar-tissue.json automatically
43
+ ```
44
+
45
+ ## Why this, and not a linter or an AI reviewer
46
+
47
+ - **A linter** knows syntax and style. It does not know that `isVisible({timeout})`
48
+ silently lied to you in March. A scar rule cites the incident and the fix.
49
+ - **An AI reviewer** is non-deterministic and forgets. This is plain matching over
50
+ local files - no network, no keys - so the same input gives the same findings
51
+ every time, which is the only way a merge gate earns the right to fail a build.
52
+ - **AI has one, bounded job:** drafting. `harvest` surfaces candidates from
53
+ history, and the shipped prompts (`prompts/extract-rules.md`,
54
+ `prompts/turn-incident-into-rule.md`) help turn an incident into a rule. The
55
+ result is still just data you review and commit.
56
+ - **Every rule is proved, not believed.** A fixture must make it fire and a clean
57
+ fixture must keep it silent (`sfqa-scar prove`). A rule with no detector is
58
+ reported as a gap, never counted as coverage. The proof caught a rule in this
59
+ very set whose detector could never fire - and, on a real run, caught the
60
+ detectors themselves misreading clean code (see [Proof and honesty](#proof-and-honesty)).
61
+
62
+ ## Adding a rule is a prompt, not a regex
63
+
64
+ Most rules tools ask you to write a detector. This one asks you to describe the
65
+ incident, because the incident is the part you already know:
66
+
67
+ > "In March a field reader called `isVisible({ timeout: 5000 })`. Playwright
68
+ > ignores options there and returns immediately, so a label that was still
69
+ > rendering read as absent and the test passed with an empty field. Never pass
70
+ > options to `isVisible()`."
71
+
72
+ That paragraph is the whole job. It becomes a rule like this:
73
+
74
+ ```bash
75
+ npx sfqa-scar new --rule "Never pass a timeout to isVisible() - it is ignored" --scope "test-pages/**"
76
+ ```
77
+
78
+ which writes three files: the rule entry, one fixture that must match, and one
79
+ that must not. You fill in the sentence and one line of pattern; `sfqa-scar
80
+ prove` tells you whether the rule fires where it should and stays silent where
81
+ it should not. If you have no idea what the pattern should be, hand the
82
+ paragraph to an assistant with
83
+ [`prompts/turn-incident-into-rule.md`](prompts/turn-incident-into-rule.md) - it
84
+ asks for exactly the rule, the detector and the two fixtures, and refuses to
85
+ accept a rule that cites no incident.
86
+
87
+ There is a full worked example, from incident to committed rule, in
88
+ [Add a rule after an incident](#add-a-rule-after-an-incident). The rest of this
89
+ README is detail you can read when you need it.
90
+
91
+ ## Why this exists
92
+
93
+ A Salesforce suite accumulates scar tissue: the toast that vanishes on the
94
+ slower sandbox, the `isVisible` call that ignores its own timeout, the picklist
95
+ label that drifted one capital letter, the credential that reached git history.
96
+ Each one cost a day. Each one is written down in a retro and then forgotten.
97
+
98
+ `@sfqa/scar` keeps them as data. The rule set is JSON; the detectors are
99
+ declarative; the scanner is dependency-free Node. A new team member can read
100
+ the history without reading the commit log.
101
+
102
+ ## Not just Salesforce
103
+
104
+ The scanner is a text-file engine: scoped globs, regex detectors, no Salesforce
105
+ API and no browser. It will run over Cypress, Selenium, Appium, API tests, or
106
+ plain application code.
107
+
108
+ The shipped rules are the part that is Salesforce/Playwright-shaped, because that
109
+ is the history they were mined from. A rule like `.isVisible({` is inert in a
110
+ Cypress repo. That is not a limitation of the engine; it is the point. Regenerate
111
+ the rule set from your own history with `prompts/extract-rules.md`, and the same
112
+ binary reviews your stack. The 19 rules are a worked example of the format, not a
113
+ universal pack.
114
+
115
+ ## What it does not do
116
+
117
+ - It does not learn on its own. A new incident becomes a rule only when you add
118
+ one, or re-run extraction. The runtime is deterministic.
119
+ - It does not read live runs or server-side logs. It reads source files, diffs,
120
+ and the local report artifacts you point at with `--report`.
121
+ - It cannot name the offending source line from a runtime failure. A failing test
122
+ gives you the test, the file and the error - that is a brief, not a draft. Turning
123
+ a brief into a rule is the step that needs a person (or a model working for one).
124
+ - It does not run itself. Wire it into a pre-commit hook, a commit-msg hook, or a
125
+ CI job; then it runs without anyone remembering.
126
+
127
+ ## Run it on every commit
128
+
129
+ Three places, all short.
130
+
131
+ **Pre-commit** (husky v9):
132
+
133
+ ```bash
134
+ npm install -D husky
135
+ npx husky init
136
+ echo "npx sfqa-scar --staged --fail-on high" > .husky/pre-commit
137
+ ```
138
+
139
+ Older husky: `npx husky add .husky/pre-commit "npx sfqa-scar --staged --fail-on high"`.
140
+
141
+ **Commit-msg** (optional, for incident capture):
142
+
143
+ ```bash
144
+ echo 'npx sfqa-scar harvest --staged --message-file "$1" --ask' > .husky/commit-msg
145
+ ```
146
+
147
+ This one asks a single question when the commit message looks like a fix, and
148
+ records a "no" so it is not asked again. See
149
+ [Ask at commit time](#ask-at-commit-time-remember-the-no).
150
+
151
+ **CI** (GitHub Actions):
152
+
153
+ ```yaml
154
+ name: scar
155
+ on: [pull_request]
156
+
157
+ jobs:
158
+ review:
159
+ runs-on: ubuntu-latest
160
+ steps:
161
+ - uses: actions/checkout@v4
162
+ with:
163
+ fetch-depth: 0 # --base needs the history
164
+ - uses: actions/setup-node@v4
165
+ with:
166
+ node-version: 20
167
+ - run: npm ci
168
+ - run: npx sfqa-scar --base origin/${{ github.base_ref }} --fail-on high
169
+ ```
170
+
171
+ **CI** (Azure DevOps, advisory - reports, never gates the build):
172
+
173
+ ```yaml
174
+ - job: ScarReview
175
+ displayName: 'Scar Tissue Review (advisory)'
176
+ steps:
177
+ - task: NodeTool@0
178
+ inputs: { versionSpec: '20.x' }
179
+ - script: |
180
+ out=scar-review.md
181
+ npx --yes @sfqa/scar --markdown --fail-on high . > "$out" || true
182
+ cat "$out"
183
+ echo "##vso[task.complete result=SucceededWithIssues;]Scar review written (advisory)."
184
+ continueOnError: true
185
+ - task: PublishBuildArtifacts@1
186
+ inputs: { PathtoPublish: 'scar-review.md', ArtifactName: 'scar-review' }
187
+ condition: always()
188
+ ```
189
+
190
+ To make it a gate, drop `|| true` and `continueOnError` and let the exit code
191
+ fail the job. Advisory is a good first step: the findings are real, but you
192
+ decide when a red build is earned.
193
+
194
+ That is the whole "automatic" story. No daemon, no network, no API key. Once
195
+ those hooks exist, nobody has to remember to run it.
196
+
197
+ ## Day one: the hook does not add rules
198
+
199
+ The pre-commit hook and the CI job are **enforcement**. They apply rules that
200
+ already exist. They never write one.
201
+
202
+ The commit-msg hook is adjacent: it never blocks either. It only asks whether a
203
+ fix-shaped change still owes a rule, and a "no" records itself.
204
+
205
+ That is deliberate. A scanner that invented a rule from a failed commit would be
206
+ guessing about intent, and a rule you cannot defend from history is a style
207
+ opinion. Writing the rule is the one step that needs a person - or a model
208
+ working for one - who knows what actually broke.
209
+
210
+ So on day one the rule file comes from one of three places:
211
+
212
+ 1. **Copy the template.** Use this if your stack is close to Salesforce +
213
+ Playwright. If you just want to evaluate the shipped set, skip the copy
214
+ entirely: the packaged `rules/generic.json` is the default.
215
+ 2. **Extract from history.** One pass with
216
+ [`prompts/extract-rules.md`](prompts/extract-rules.md) before you wire the
217
+ hook. This is the right start for any other stack.
218
+ 3. **An empty rule file.** Valid, and it enforces nothing. The hook runs, finds
219
+ zero findings, exits 0. Honest, but useless.
220
+
221
+ The order that actually works:
222
+
223
+ ```bash
224
+ # 1. seed a rule file you own (skip if you want the packaged default)
225
+ cp node_modules/@sfqa/scar/rules/generic.json ./scar-tissue.json
226
+
227
+ # 2. see what it says before it can fail a commit
228
+ npx sfqa-scar .
229
+
230
+ # 3. wire it in once you trust the output (see "Run it on every commit")
231
+ npx husky init
232
+ echo "npx sfqa-scar --staged --fail-on high" > .husky/pre-commit
233
+ ```
234
+
235
+ A `scar-tissue.json` in the working directory is picked up automatically, so no
236
+ `--rules` flag is needed. Pass `--rules` only to point somewhere else.
237
+
238
+ After that, the loop is:
239
+
240
+ ```text
241
+ a test fails for a real reason
242
+ -> the postmortem names the bad line
243
+ -> one rule + two fixtures (prompts/turn-incident-into-rule.md)
244
+ -> commit the rule
245
+ -> the hook enforces it on every future commit
246
+ ```
247
+
248
+ The hook did not add the rule. **The incident did.** The postmortem is the
249
+ teacher; the hook is the memory. If a team installs the hook with an empty rule
250
+ file and never runs the retro step, it will correctly do nothing forever - and
251
+ that is the boundary of the product, not a bug in it.
252
+
253
+ ## The standing loop
254
+
255
+ Adoption is one loop, and it repeats:
256
+
257
+ 1. **A failure happens.** Someone fixes it, or a test fails for a real reason and
258
+ the retro names the bad line.
259
+ 2. **A rule appears.** Hand-written, or harvested from history and drafted.
260
+ 3. **The rule gets proved.** Positive fixture fires, clean fixture stays quiet.
261
+ 4. **The rule gets committed.** With the incident, in the same commit.
262
+ 5. **The hook enforces.** Pre-commit and CI, on every commit after that.
263
+ 6. **Harvest keeps watch.** Weekly sweep for unclaimed incidents; CI goes red if
264
+ any remain.
265
+
266
+ Nothing in this loop depends on anyone remembering to do anything. The only
267
+ remembering left is the one decision the tool cannot make for you: whether a
268
+ draft is worth turning into a rule. Everything else is wired.
269
+
270
+ ## Find the rules you never wrote
271
+
272
+ The gap in every "write a rule after an incident" workflow is the incident
273
+ nobody wrote a rule for. `harvest` closes that gap from three sources:
274
+
275
+ | Source | What it shows | Command |
276
+ |---|---|---|
277
+ | Git history | fix-shaped commits no rule accounts for | `harvest` |
278
+ | The staged change | the incident you are committing right now | `harvest --staged --message ...` |
279
+ | Local test reports | failures you are living with, repeated across runs | `harvest --report test-results/` |
280
+
281
+ ```bash
282
+ npx sfqa-scar harvest # last 300 commits
283
+ npx sfqa-scar harvest --since 2026-01-01 # a date range
284
+ npx sfqa-scar harvest --all # also show incidents a rule covers
285
+ npx sfqa-scar harvest --write scar-candidates.json --json
286
+ npx sfqa-scar harvest --strict # exit 1 if any incident has no rule
287
+
288
+ npx sfqa-scar harvest --staged --message "fix: replace the fixed sleep"
289
+ npx sfqa-scar harvest --report test-results/ --report-format auto
290
+ ```
291
+
292
+ What it does:
293
+
294
+ - reads commits whose message says fix, revert, hotfix, flaky, race, timeout,
295
+ workaround, credential, secret, rollback, regression, bug, defect or patch;
296
+ - ignores an incident already claimed by a rule's `origin.commit`, or whose
297
+ removed lines match an existing detector;
298
+ - for each unclaimed incident, shows the commit, the removed bad line, the added
299
+ fix, a proposed `pathInclude`, a proposed `linePatterns` regex, and a
300
+ confidence;
301
+ - writes nothing. You review the draft, then add it.
302
+
303
+ ```text
304
+ CANDIDATE 1/1
305
+ commit : 402ef2d 2026-11-09
306
+ subject: fix: replace fixed sleep with an element wait
307
+ files : tests/orders.ts
308
+ removed: await page.waitForTimeout(2000);
309
+ added : await locator.waitFor({ state: 'visible', timeout: 5000 });
310
+ detect : page\.waitForTimeout\( [confidence high]
311
+ scope : tests/**
312
+ id : commit:402ef2d9c1...
313
+ ```
314
+
315
+ From there it is the loop from the previous section: prove the draft with a
316
+ positive fixture and a clean fixture, then commit. `--strict` in CI turns "we
317
+ forgot to write a rule for last month's hotfix" into a red build, which is the
318
+ only way the gap becomes visible. Run the history pass on a schedule, not on every
319
+ push, so the gate stays fast:
320
+
321
+ ```yaml
322
+ name: scar-harvest
323
+ on:
324
+ schedule:
325
+ - cron: '0 6 * * 1' # Mondays
326
+ workflow_dispatch:
327
+
328
+ jobs:
329
+ gaps:
330
+ runs-on: ubuntu-latest
331
+ steps:
332
+ - uses: actions/checkout@v4
333
+ with:
334
+ fetch-depth: 0
335
+ - uses: actions/setup-node@v4
336
+ with:
337
+ node-version: 20
338
+ - run: npm ci
339
+ - run: npx sfqa-scar harvest --since 2026-01-01 --strict
340
+ ```
341
+
342
+ ## Ask at commit time, remember the no
343
+
344
+ The best moment to capture an incident is while the person still remembers it:
345
+ the commit that fixes it. A `commit-msg` hook has the staged diff *and* the
346
+ message, so it can ask a single question and then get out of the way.
347
+
348
+ ```bash
349
+ # .husky/pre-commit - fast, no prompts, no writes
350
+ npx sfqa-scar --staged --fail-on high
351
+
352
+ # .husky/commit-msg - one question, only when the message looks like an incident
353
+ npx sfqa-scar harvest --staged --message-file "$1" --ask
354
+ ```
355
+
356
+ Why two hooks instead of one:
357
+
358
+ - **Pre-commit scans.** It only reads the staged files and the existing rules.
359
+ No `git log` walk, no network, no prompt. It stays fast enough to run on every
360
+ commit.
361
+ - **Commit-msg harvests.** Git has no commit message until this hook, so
362
+ message-based incident detection cannot happen earlier. It uses the staged
363
+ diff plus `COMMIT_EDITMSG`, asks once, and exits `0` either way - a `commit-msg`
364
+ hook must not block a commit because someone declined to write a rule.
365
+
366
+ When the message matches the incident keywords, `--ask` shows the candidate and
367
+ offers three answers:
368
+
369
+ ```text
370
+ capture a draft for staged "fix: replace the fixed sleep with an element wait"? [y]es / [i]gnore / [s]kip:
371
+ ```
372
+
373
+ | Answer | What happens | Next run |
374
+ |---|---|---|
375
+ | `y` | the draft is saved to `scar-tissue.candidates.json` | **not asked again**; still shown as `captured: yes`, because no rule exists yet |
376
+ | `i` | the decision is written to `scar-tissue.ignore.json` | **not asked again**; listed with `--all` |
377
+ | `s` | nothing is written | asked again next time |
378
+
379
+ So a "yes" is not a vibe. It is a queued draft with the incident, the scope and
380
+ the proposed detector, waiting for someone to move it into the rule set. A "no"
381
+ is a recorded decision. Both are files you can review in a PR.
382
+
383
+ ```json
384
+ // scar-tissue.candidates.json - yes
385
+ {
386
+ "version": 1,
387
+ "candidates": [
388
+ {
389
+ "fingerprint": "staged:9f2c...",
390
+ "capturedAt": "2026-11-09T09:12:00.000Z",
391
+ "subject": "fix: replace the fixed sleep with an element wait",
392
+ "files": ["tests/orders.ts"],
393
+ "draft": { "id": "ST-020", "match": { "linePatterns": ["page\\.waitForTimeout\\("] } }
394
+ }
395
+ ]
396
+ }
397
+ ```
398
+
399
+ ```json
400
+ // scar-tissue.ignore.json - no
401
+ {
402
+ "version": 1,
403
+ "ignored": [
404
+ {
405
+ "fingerprint": "staged:9f2c...",
406
+ "reason": "the fix is already covered by the code review",
407
+ "at": "2026-11-09T09:12:00.000Z",
408
+ "subject": "fix: replace the fixed sleep with an element wait"
409
+ }
410
+ ]
411
+ }
412
+ ```
413
+
414
+ That means you can audit a decision, diff it in a PR, and reverse it by deleting
415
+ the entry. `--ignore <id>` and `--capture <id>` record the same decisions
416
+ non-interactively, which is useful from a CI job or when someone dismisses a
417
+ candidate they saw in a report. `--all` brings dismissed candidates back, with
418
+ `ignoredAt` and `ignoreReason`.
419
+
420
+ A captured candidate disappears on its own once a rule covers it: the coverage
421
+ check runs first, so adding the rule with the right `origin.commit` or detector
422
+ closes the loop without hand-editing the queue.
423
+
424
+ `--ask` only prompts for `--staged`. History is reviewed in bulk with `--write`;
425
+ a hundred interactive questions about last year's commits is not a workflow.
426
+
427
+ ## Local test runs are the third source
428
+
429
+ A test report shows the failures a team is currently living with. Point
430
+ `--report` at a file or a directory and harvest the two formats runners emit
431
+ most: Playwright JSON and JUnit XML. Format is sniffed from the file unless you
432
+ pass `--report-format playwright|junit`.
433
+
434
+ ```bash
435
+ npx playwright test --reporter=json
436
+ npx sfqa-scar harvest --report test-results/ --json --write local-failures.json
437
+ ```
438
+
439
+ The same failure across several runs is merged and counted, because repetition
440
+ is the signal:
441
+
442
+ ```text
443
+ LOCAL FAILURE 1/1
444
+ test : orders.spec.ts > creates an order > creates an order
445
+ file : tests/orders.spec.ts
446
+ status : failed [4 occurrence(s)]
447
+ error : Error: expect(locator).toBeVisible() failed
448
+ Locator: getByText('Order created')
449
+ next : find the source line behind this failure, then write one rule with two fixtures
450
+ id : test:3a91...
451
+ ```
452
+
453
+ These are **briefs, not drafts**. A runtime error tells you that something
454
+ broke; it does not tell you which source line caused it, so the tool will not
455
+ invent a regex for it. The brief gives you the test, the file, the error and a
456
+ stable id, which is exactly the input the
457
+ [incident-to-rule prompt](prompts/turn-incident-into-rule.md) expects. Local
458
+ failures are also filtered by the same ignore ledger, so `--ignore <id>` works
459
+ on them too.
460
+
461
+ Two limits, stated plainly:
462
+
463
+ - History and staged mode read a **diff**. A failure that was never fixed and
464
+ never staged is invisible there; the local-report source is how you see it.
465
+ - The proposed regex is a **draft**. A low-confidence draft deliberately has no
466
+ detector; the tool will not guess a pattern it cannot defend.
467
+
468
+ Credential-shaped lines are redacted before they reach stdout, the JSON or a
469
+ draft. The harvester will not write the secret it found into a rule file.
470
+
471
+ ## The rule set is a template, not a product
472
+
473
+ The shipped set (`rules/generic.json`) has 19 rules. Their **detectors** are
474
+ generic. Where a detector needs to know a value, the value list is an example
475
+ that you replace:
476
+
477
+ | Detector input | Shipped example | Replace with |
478
+ |---|---|---|
479
+ | `labelRules.canonical` | `Web`, `Event`, `Referral`, `Phone`, `Email`, `SMS` | the exact labels your org renders |
480
+ | inventory-code shape | `SKU-A12-0042` | the identifier shape your product uses |
481
+ | local-only file patterns | `local-notes-*.md`, `*.zip` | the paths your team keeps local |
482
+ | timeout ceiling | `< 360000` ms | your measured longest journey, minus headroom |
483
+
484
+ That is deliberate. A generic rule set that pretended to know your org would be
485
+ worse than no rule set. Extract once from your own history, keep the examples
486
+ only as documentation.
487
+
488
+ ## Where the rules come from
489
+
490
+ Three sources, in increasing order of value:
491
+
492
+ 1. **The shipped template.** Eighteen rules mined from one team's
493
+ Salesforce/Playwright history. Treat it as a worked example of the format,
494
+ and as a usable starting point only if your stack is close to that one.
495
+ 2. **History harvest.** Run `npx sfqa-scar harvest` first. It reads your git
496
+ log, finds fix-shaped commits that no rule covers, and prints a draft rule for
497
+ each one. Deterministic, no LLM. For lessons that need judgement across a
498
+ whole diff, the deeper pass is
499
+ [`prompts/extract-rules.md`](prompts/extract-rules.md): it reads `git log`,
500
+ opens commits whose messages say fix, revert, hotfix, flaky, race, timeout,
501
+ workaround or credentials, and emits rules that cite a commit hash. A lesson
502
+ without a hash is dropped.
503
+ 3. **One rule per incident.** This is the loop that keeps the rule set alive.
504
+ After a postmortem, write one rule before the memory fades. Ten minutes, and
505
+ it is the only ongoing maintenance the tool needs.
506
+
507
+ So how do you know which rules to add? You do not pick them from a catalogue.
508
+ You take them from incidents that already cost you a day. If you cannot name the
509
+ bad line and the day it burned you, it is a lint opinion, not scar tissue. That
510
+ is the whole selection criterion.
511
+
512
+ `harvest` exists because that criterion still requires someone to remember the
513
+ incident. The tool reads the history for you and puts the unclaimed ones on
514
+ screen. You still approve each rule; you no longer have to notice the gap
515
+ yourself.
516
+
517
+ ## Add a rule after an incident
518
+
519
+ One worked example, end to end, using a failure every automation suite has seen:
520
+ a test that waited on a fixed sleep instead of the element, passed locally, and
521
+ failed in CI when the environment was slower.
522
+
523
+ **1. Start from the incident, not from the regex.** Write the sentence a reviewer
524
+ would need, then let the tool scaffold the rest. Seed a rule file you own first,
525
+ so the 19 shipped rules stay and your new one becomes ST-020:
526
+
527
+ ```bash
528
+ cp node_modules/@sfqa/scar/rules/generic.json scar-tissue.json
529
+
530
+ npx sfqa-scar new \
531
+ --rule "Never wait on a fixed sleep for a UI element; poll the element with a real timeout." \
532
+ --scope "tests/" \
533
+ --severity med
534
+ ```
535
+
536
+ ```text
537
+ created ST-020
538
+
539
+ rule scar-tissue.json
540
+ positive scar-fixtures/positive/tests/st-020.ts
541
+ clean scar-fixtures/negative/tests/st-020.ts
542
+
543
+ what to do next
544
+ 1. open scar-tissue.json, replace the TODOs, and put the pattern in match.linePatterns
545
+ 2. not sure of the pattern? prompts/turn-incident-into-rule.md (in the package) drafts it from the incident
546
+ 3. paste the bad line into scar-fixtures/positive/tests/st-020.ts
547
+ 4. paste the same code with the fix into scar-fixtures/negative/tests/st-020.ts
548
+ 5. prove just this rule: sfqa-scar prove --id ST-020
549
+ 6. it is scoped to tests/**; a fixture outside that path is never scanned
550
+ ```
551
+
552
+ `--scope` takes a plain directory (`tests/` means everything under it) or a glob
553
+ (`tests/**/*.ts`). Repeat it if the rule belongs in more than one place; a
554
+ fixture pair is written for each.
555
+
556
+ Three files exist, the id is free, and the fixtures are already in a path the
557
+ rule scans - which is the mistake people make when they place them by hand. The
558
+ rule starts with an empty detector on purpose, so nothing can claim it is
559
+ enforced before it is.
560
+
561
+ **2. Fill in the two TODOs.** The scaffold gives you the shape; the incident
562
+ gives you the words:
563
+
564
+ ```json
565
+ {
566
+ "id": "ST-020",
567
+ "rule": "Never wait on a fixed sleep for a UI element; poll the element with a real timeout.",
568
+ "context": "Playwright page objects and helpers under tests/**",
569
+ "origin": {
570
+ "commit": "9f2c41d",
571
+ "summary": "waitForTimeout(2000) passed locally and failed in CI on a slower environment; a fixed sleep is a bet on machine speed."
572
+ },
573
+ "severity": "med",
574
+ "detection_pattern": "page.waitForTimeout(...) used to wait for UI state",
575
+ "match": {
576
+ "pathInclude": ["tests/**"],
577
+ "pathExclude": ["**/node_modules/**"],
578
+ "linePatterns": ["\\.waitForTimeout\\(\\s*\\d+\\s*\\)"]
579
+ },
580
+ "recommendation": "Replace with await locator.waitFor({ state: 'visible', timeout: N })."
581
+ }
582
+ ```
583
+
584
+ **3. Prove it both ways.** Put the bad line in the positive fixture, the same
585
+ code with the fix in the clean one, then ask for a verdict on one rule:
586
+
587
+ ```bash
588
+ npx sfqa-scar prove --id ST-020
589
+ ```
590
+
591
+ ```text
592
+ sfqa-scar prove - 1 rule(s), 1 executable
593
+ rule set: scar-tissue.json
594
+ fixtures: scar-fixtures
595
+
596
+ rule severity detector positive clean
597
+ ------ -------- --------- --------- -------
598
+ ST-020 med yes fires clean
599
+
600
+ PROOF: 1/1 executable rules fire on a positive fixture and stay silent on the clean set.
601
+ Coverage: 1/1 rules enforced today; 0 rule(s) have no detector.
602
+ ```
603
+
604
+ Read it as a sentence:
605
+
606
+ - `fires` - the bad line was found. If this says `missing`, the detector is wrong
607
+ or the fixture is outside the rule's scope; the message prints that scope.
608
+ - `clean` - the fixed line was not found. If this says `leak`, the pattern is too
609
+ broad and would fail good code.
610
+ - `detector: NO` - the rule is prose. `prove` reports it as a gap and exits 1 if
611
+ nothing else fires, because a rule that cannot fire is a comment with a
612
+ version number.
613
+
614
+ **4. Scan for real.**
615
+
616
+ ```bash
617
+ npx sfqa-scar --fail-on med tests/
618
+ ```
619
+
620
+ If the rule set fails to load, the error names the rule and the field; detectors
621
+ are validated before any file is read.
622
+
623
+ **5. Commit the rule, the fixtures, and the incident summary together.** The
624
+ commit message is the postmortem. Reviewers can see what changed and why, and
625
+ the next person inherits the lesson without reading the incident channel.
626
+
627
+ That is the loop: incident, retro, one rule, two fixtures, commit. It runs once
628
+ per incident, not once per commit.
629
+
630
+ ### If you do not want the scaffold
631
+
632
+ The format is plain JSON and nothing depends on `new`. Append the object above
633
+ to the `rules` array in `scar-tissue.json` by hand, put the two fixtures wherever
634
+ your rule's `pathInclude` points, and run `sfqa-scar prove`. The scaffold only
635
+ removes the two mechanical decisions - which id is free, and where the fixture
636
+ belongs.
637
+
638
+ [`prompts/turn-incident-into-rule.md`](prompts/turn-incident-into-rule.md) is a
639
+ fill-in-the-blanks prompt for step 1 if you want an assistant to draft the rule
640
+ from an incident write-up.
641
+
642
+ ## Choosing a starter set
643
+
644
+ The engine does not change between stacks; the rule set does. What survives
645
+ depends on what your repo actually contains:
646
+
647
+ | Your repo | Start with | Then extract your own |
648
+ |---|---|---|
649
+ | Salesforce + Playwright | Most of the shipped set (ST-001 to ST-017, ST-019). Re-check `labelRules.canonical` against your org first. | Auth and session reuse, record navigation, picklist drift, toast races |
650
+ | Playwright, non-Salesforce | ST-004, ST-009, ST-017, ST-018. Drop every `labelRules` and record/Lightning rule. | Waits, fixture reuse, your real timeout ceiling, selector drift |
651
+ | Cypress, Selenium or Appium | ST-009, ST-017, ST-018 only. `.isVisible({` is inert here. | Framework-specific waits, session reuse, selector strategy |
652
+ | API or integration tests | ST-001, ST-002, ST-007, ST-008, ST-018. Nothing DOM-shaped survives. | Retry and idempotency, environment config bleed, hardcoded fixture ids |
653
+ | Application code | ST-001, ST-002, ST-018. The engine handles it; the rules must too. | Error swallowing, silent fallbacks, config precedence |
654
+
655
+ The pattern behind the table: keep the rules that describe **process and safety**
656
+ (secrets, config bleed, hardcoded mutable ids, fixed sleeps, duplicated helpers),
657
+ drop the rules that name **another team's product** (Lightning, lead conversion,
658
+ Salesforce picklists). Then grow the set from your own incidents, one at a time.
659
+
660
+ What not to do: copy all 19 into a Cypress repo and treat a silent scan as
661
+ passing. A rule that can never fire is coverage theatre. If a rule is inert in
662
+ your stack, delete it rather than leave it in the file.
663
+
664
+ ## The 19 rules
665
+
666
+ | ID | Severity | Rule |
667
+ |---|---|---|
668
+ | ST-001 | high | Never hardcode credentials, tokens or secrets in code, config fallbacks or committed `.env` files. |
669
+ | ST-002 | high | Each environment config reads its own environment's variables, never a sibling's. |
670
+ | ST-003 | high | Picklist values match the Salesforce label exactly: case, spaces, hyphens. |
671
+ | ST-004 | high | `isVisible()` ignores any `timeout`; use `waitFor({ state: 'visible' })`. |
672
+ | ST-005 | med | Never treat a success toast as the only success signal. |
673
+ | ST-006 | med | Record navigation tolerates Lightning URL variants; no single strict `waitForURL`. |
674
+ | ST-007 | med | Server-side integration fields need periodic reload, not just a deadline poll. |
675
+ | ST-008 | high | Never hardcode mutable inventory, accounts or orders; use a sentinel. |
676
+ | ST-009 | high | Timeout must exceed the measured journey, or retries write duplicate records. |
677
+ | ST-010 | med | Keep teardown out of the reporter-finalization window; no broad `pkill`. |
678
+ | ST-011 | high | Auth goes through the shared login flow; no hardcoded standard-login selectors. |
679
+ | ST-012 | med | Human-in-the-loop logins need race tolerance: adopt an existing session. |
680
+ | ST-013 | med | After impersonation or a tab switch, rebuild page objects on the active page. |
681
+ | ST-014 | high | Lead conversion has two outcomes; branch on duplicate detection and read identity from the record. |
682
+ | ST-015 | med | Field-dependency maps and picklist constants need live-org provenance. |
683
+ | ST-016 | low | Correlated generated fields must be co-constrained. |
684
+ | ST-017 | med | Search for an existing helper before adding a duplicate. |
685
+ | ST-018 | low | Local-only artifacts are gitignored from birth. *(no detector yet)* |
686
+ | ST-019 | med | Read a record header from a scoped container, never a bare comma-list that mixes a container with a value element plus `.first()`. |
687
+
688
+ 18 of 19 have a machine detector and are proven both ways: a fixture makes each
689
+ one fire, and a clean fixture set makes none of them fire. ST-018 is reported as
690
+ a gap rather than counted as coverage. The source repo keeps the fixture set the packaged rules are proven against in
691
+ `scar-validation/`, and runs it in CI with the same `prove` command a consumer
692
+ uses on their own rules. The fixtures are not shipped in the tarball, because
693
+ they only make sense next to a rule set you are editing - `sfqa-scar new` creates
694
+ the first pair for you.
695
+
696
+ ## Rule format
697
+
698
+ ```json
699
+ {
700
+ "id": "ST-004",
701
+ "rule": "Never pass options to isVisible() ...",
702
+ "context": "All Playwright page objects ...",
703
+ "origin": { "commit": null, "summary": "what went wrong, as a class of incident" },
704
+ "severity": "high",
705
+ "detection_pattern": ".isVisible({ ... }) - any argument passed to isVisible, unless the call is .catch()-guarded.",
706
+ "match": {
707
+ "pathInclude": ["**/*.ts"],
708
+ "pathExclude": ["**/node_modules/**"],
709
+ "linePatterns": ["\\.isVisible\\(\\s*\\{[^{}]{0,200}?\\}\\)(?!\\s*\\.catch)"],
710
+ "fileExemptSignals": ["Promise\\.race"]
711
+ },
712
+ "recommendation": "Replace with waitFor({ state: 'visible', timeout: N })."
713
+ }
714
+ ```
715
+
716
+ Detectors available in `match`:
717
+
718
+ | Key | Fires when |
719
+ |---|---|
720
+ | `linePatterns` | a non-comment line matches (a pattern containing `[\s\S]` may span lines) |
721
+ | `numericRules` | capture group 1 falls inside `[min, max]`, e.g. a too-short timeout |
722
+ | `fileNameRules` | the file's base name matches, e.g. a tracked `.env` |
723
+ | `labelRules` | a quoted literal is a near-miss of a canonical label (case, spacing, hyphen) |
724
+ | `fileRequiredSignals` | **none** of the signals exist in a file that is in scope |
725
+
726
+ A `linePatterns` entry sees the line plus the next three lines, and the finding
727
+ is reported only when the match *starts* on that line. That is how a guard
728
+ written underneath -
729
+
730
+ ```ts
731
+ const there = await label.isVisible({ timeout: 5000 })
732
+ .catch(() => false);
733
+ ```
734
+
735
+ - is read as a deliberate probe rather than a silent assertion. The template's
736
+ ST-004 expresses exactly that with a trailing `(?!\s*\.catch)`.
737
+
738
+ Two exemption scopes exist on purpose:
739
+
740
+ - `fileExemptSignals` clears the whole file when a mitigation is present
741
+ (a `Promise.race` guarding a toast wait).
742
+ - `fileExemptLines` clears one line (an empty `process.env` fallback) while a
743
+ sibling hardcoded literal in the same file still fires.
744
+
745
+ ## Extracting your own rule set
746
+
747
+ The shipped set is a starting point. To mine your own history, give a capable
748
+ model the prompt in
749
+ [`prompts/extract-rules.md`](prompts/extract-rules.md). It asks for a commit
750
+ reference per rule, JavaScript regexes, exact canonical labels, and a documented
751
+ gap instead of an invented detector. Review the output; a rule you cannot defend
752
+ from history is a style opinion, not scar tissue.
753
+
754
+ After that first pass, do not re-run extraction for every small bug. Use
755
+ [`prompts/turn-incident-into-rule.md`](prompts/turn-incident-into-rule.md) to
756
+ turn one incident into one rule plus its two fixtures, and save the bulk
757
+ extraction for a quarterly cleanup or a new repository.
758
+
759
+ ## Usage
760
+
761
+ ```bash
762
+ # scan paths
763
+ npx sfqa-scar test-pages/ features/
764
+
765
+ # review only what changed
766
+ npx sfqa-scar --base origin/main
767
+ npx sfqa-scar --staged
768
+
769
+ # machine-readable, for CI or a PR comment
770
+ npx sfqa-scar --json --fail-on med .
771
+
772
+ # markdown, for a PR comment or a write-up (relative paths, one row per finding)
773
+ npx sfqa-scar --markdown --fail-on med .
774
+
775
+ # start a rule: writes the rule and both fixtures at a path the rule scans
776
+ npx sfqa-scar new --rule "Never pass options to isVisible()" --scope "test-pages/**"
777
+
778
+ # prove every rule against its fixtures (exit 1 if one does not)
779
+ npx sfqa-scar prove
780
+ npx sfqa-scar prove --id ST-020 # one rule, the one you just wrote
781
+
782
+ # a different rule set - one in the working directory is used automatically
783
+ npx sfqa-scar --rules ../shared/scar-tissue.json src/
784
+
785
+ # find incidents that never became rules
786
+ npx sfqa-scar harvest --since 2026-01-01
787
+ npx sfqa-scar harvest --write scar-candidates.json --json
788
+ npx sfqa-scar harvest --staged --message "fix: replace the fixed sleep" --ask
789
+ npx sfqa-scar harvest --report test-results/ --report-format auto
790
+ npx sfqa-scar harvest --all # include dismissed candidates
791
+
792
+ # dismiss one candidate, permanently, in committed data
793
+ npx sfqa-scar harvest --ignore staged:9f2c...
794
+
795
+ # keep one candidate as a queued draft
796
+ npx sfqa-scar harvest --capture staged:9f2c...
797
+ ```
798
+
799
+ Exit codes: `0` clean at the threshold, `1` at least one finding at or above
800
+ `--fail-on` (default `high`), `2` bad usage or an invalid rule set.
801
+
802
+ `harvest` is advisory: it exits `0` even when it finds an unclaimed incident.
803
+ Add `--strict` to make a gap exit `1`, for a CI job that refuses to let last
804
+ month's hotfix go unrecorded.
805
+
806
+ Findings inside gitignored files are downgraded to `low` and marked
807
+ `[gitignored - informational]`. A properly ignored `.env` is not a leak; a
808
+ tracked one is.
809
+
810
+ ## Design rules
811
+
812
+ - **The rule set is data.** Regenerate it, version it, review it in a PR. No
813
+ detector logic hides in the scanner.
814
+ - **Nothing is silently dropped.** A rule with no detector is listed as a gap.
815
+ A finding in an ignored file is shown, not hidden.
816
+ - **Fail at load, not at scan.** Every regex compiles and every rule is
817
+ validated (unique id, known severity, required prose) before a file is read.
818
+ - **Comments are not recurrences.** A comment that mentions a bad pattern is a
819
+ warning to the reader, not a hit.
820
+ - **Discovery never writes rules.** `harvest` proposes; a person approves. The
821
+ scanner has no side effects. The only file harvest writes is the ignore
822
+ ledger, and only when a person says "do not ask again".
823
+ - **A decision is data.** `scar-tissue.candidates.json` (captured) and
824
+ `scar-tissue.ignore.json` (dismissed) are reviewed and committed like code,
825
+ so a decision can be audited and reversed.
826
+ - **No org vocabulary in the package.** The shipped rule set is client-free.
827
+
828
+ ## Tune, do not turn off
829
+
830
+ A finding you choose to live with is still information. Demote it in **your**
831
+ `scar-tissue.json`, not in the template: set its `severity` to `low` and record
832
+ the reason in `context` (for example, "accepted until test-data setup exists").
833
+ The scan keeps reporting it, the history stays honest, and you can reverse it the
834
+ day the reason stops being true. A rule lowered in the template protects nobody.
835
+
836
+ ## Proof and honesty
837
+
838
+ The build fails on proof, not just on tests. `npm run validate:scar` (the same
839
+ `sfqa-scar prove` a consumer runs on their own rules) proves every executable rule
840
+ fires on a positive fixture and stays silent on a clean set; a rule with no
841
+ detector is reported as a gap, never counted as coverage. The first run of that
842
+ harness caught a rule whose cross-line detector could never fire - it sat in the
843
+ report looking thorough and could flag nothing. The fix and a regression fixture
844
+ landed before anything shipped.
845
+
846
+ Fixtures prove a detector *can* fire. They do not prove it fires on the right
847
+ thing, so the set was run against the repository it was mined from. 150 findings,
848
+ 37 `high` - and about eight were real. The rest were the detectors misreading
849
+ code (a guarded `.isVisible(...).catch(...)` probe read as a silent assertion, a
850
+ field-map value read as label drift, a `map.get("password")` read as a secret).
851
+ Each misread became a **detector fix plus a regression test**, never a lower
852
+ severity in the template. Re-run: 113 findings, 11 `high`. A tool that opens with
853
+ false alarms gets switched off in a week; the point is to be wrong in a way you
854
+ can read, fix, and prove.
855
+
856
+ Read the numbers as method, not a promise: `scar-validation/REPORT.md` records the
857
+ breakdown, and the blog tells the story of the three catches (the scar that
858
+ spread, the rule that could never fire, the detectors that were the problem).
859
+
860
+ ## Status
861
+
862
+ `0.1.1`. Dependency-free, Node >= 20, TypeScript types included. 18 of 19 rules
863
+ have a machine detector and are proven both ways; the 19th is a documented gap.
864
+ The public proof runs on synthetic fixtures, not a field trial. MIT licensed.