@1aboveio/skills 1.0.10 → 1.0.11
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.codex-plugin/plugin.json +1 -1
- package/.codex-plugin/resources.json +297 -216
- package/README.md +3 -3
- package/package.json +10 -8
- package/skills/app-dev/airflow/SKILL.md +1 -1
- package/skills/app-dev/android-foundation/SKILL.md +35 -37
- package/skills/app-dev/android-foundation/references/delivery-focus.md +4 -2
- package/skills/app-dev/android-foundation/references/feature-work.md +50 -0
- package/skills/app-dev/android-foundation/references/setup.md +24 -9
- package/skills/app-dev/android-foundation/references/workspace.md +81 -0
- package/skills/app-dev/android-foundation/scripts/workspace-runtime/bin/workspaces.mjs +52 -0
- package/skills/app-dev/android-foundation/scripts/workspace-runtime/native-workspace.mjs +239 -0
- package/skills/app-dev/android-foundation/scripts/workspace-runtime/workspace-lifecycle.mjs +2154 -0
- package/skills/app-dev/app-debug/SKILL.md +1 -1
- package/skills/app-dev/better-auth/SKILL.md +1 -1
- package/skills/app-dev/data-catalog/SKILL.md +1 -1
- package/skills/app-dev/data-quality/SKILL.md +1 -1
- package/skills/app-dev/ios-foundation/SKILL.md +15 -10
- package/skills/app-dev/ios-foundation/references/setup.md +25 -9
- package/skills/app-dev/ios-foundation/references/workspace.md +82 -0
- package/skills/app-dev/ios-foundation/scripts/workspace-runtime/bin/workspaces.mjs +52 -0
- package/skills/app-dev/ios-foundation/scripts/workspace-runtime/native-workspace.mjs +239 -0
- package/skills/app-dev/ios-foundation/scripts/workspace-runtime/workspace-lifecycle.mjs +2154 -0
- package/skills/app-dev/monorepo/SKILL.md +1 -1
- package/skills/app-dev/nextjs-fullstack/SKILL.md +1 -1
- package/skills/app-dev/prisma-setup/SKILL.md +1 -1
- package/skills/app-dev/pyspark/SKILL.md +1 -1
- package/skills/app-dev/python-backend/SKILL.md +1 -1
- package/skills/app-dev/shadcn/SKILL.md +1 -1
- package/skills/app-dev/zod-v4/SKILL.md +1 -1
- package/skills/cicd/cloud-build/SKILL.md +1 -1
- package/skills/cicd/cloud-debug/SKILL.md +1 -1
- package/skills/cicd/cloud-deploy/SKILL.md +1 -1
- package/skills/cicd/docker/SKILL.md +1 -1
- package/skills/cicd/github-actions/SKILL.md +1 -1
- package/skills/cicd/github-flow/SKILL.md +1 -1
- package/skills/cicd/google-cloud/SKILL.md +1 -1
- package/skills/cicd/mergify/SKILL.md +1 -1
- package/skills/cicd/podman/SKILL.md +1 -1
- package/skills/content/article-review/SKILL.md +1 -1
- package/skills/content/article-to-image/SKILL.md +1 -1
- package/skills/content/flowchart-svg/SKILL.md +1 -1
- package/skills/content/image-gen/SKILL.md +1 -1
- package/skills/content/mermaid/SKILL.md +1 -1
- package/skills/content/wechat-article-writer/SKILL.md +1 -1
- package/skills/content/wechat-publisher/SKILL.md +1 -1
- package/skills/content/wechat-search/SKILL.md +1 -1
- package/skills/content/xai-search/SKILL.md +1 -1
- package/skills/core/distill-writing/SKILL.md +3 -3
- package/skills/core/distill-writing/references/extraction.md +1 -1
- package/skills/core/doc-to-md/SKILL.md +1 -1
- package/skills/core/editorial-review/SKILL.md +82 -12
- package/skills/core/editorial-review/references/checklist.md +26 -1
- package/skills/core/house-style/SKILL.md +8 -3
- package/skills/core/house-style/references/analytical-reporting.md +123 -0
- package/skills/core/house-style/references/analytical-writing-patterns.md +79 -0
- package/skills/core/house-style/references/analytical-writing-research.md +129 -0
- package/skills/core/{write-doc → house-style}/references/bilingual.md +7 -5
- package/skills/core/house-style/references/definitions-and-metrics.md +86 -0
- package/skills/core/house-style/references/document-workflow.md +193 -0
- package/skills/core/house-style/references/initialization.md +66 -0
- package/skills/core/house-style/references/persistence.md +9 -4
- package/skills/core/house-style/references/rules.md +11 -4
- package/skills/core/house-style/scripts/init-guidance.mjs +213 -0
- package/skills/core/house-style/templates/analytical-report.md +54 -0
- package/skills/core/house-style/templates/gap-analysis.md +1 -1
- package/skills/core/plain-language/SKILL.md +2 -2
- package/skills/core/polish-cn/SKILL.md +9 -2
- package/skills/core/reg-parser/SKILL.md +3 -3
- package/skills/core/terminology/SKILL.md +8 -4
- package/skills/core/write-doc/SKILL.md +40 -151
- package/skills/core/write-report/SKILL.md +129 -0
- package/skills/core/{write-doc → write-report}/references/compliance-reports.md +4 -4
- package/skills/engineering/better-review/SKILL.md +37 -37
- package/skills/engineering/better-review/references/passes.md +35 -41
- package/skills/engineering/better-review/references/result.md +10 -9
- package/skills/engineering/e2e-test/SKILL.md +17 -17
- package/skills/engineering/e2e-test/references/authoring/playwright-config.md +11 -13
- package/skills/engineering/e2e-test/references/presentation-sweep.md +7 -7
- package/skills/engineering/e2e-test/references/reviewing.md +9 -11
- package/skills/engineering/e2e-test/references/route-manifest.md +1 -1
- package/skills/engineering/engineering/SKILL.md +124 -0
- package/skills/engineering/engineering/references/engineering-conventions.md +32 -0
- package/skills/engineering/engineering/references/feature-delivery.md +8 -0
- package/skills/engineering/engineering/references/init-scope.md +37 -0
- package/skills/engineering/engineering/scripts/init.mjs +160 -0
- package/skills/engineering/{contributing → engineering}/scripts/persist-conventions.mjs +1 -1
- package/skills/engineering/engineering-runtime/SKILL.md +1 -1
- package/skills/engineering/engineering-runtime/coherence/workflow.json +19 -19
- package/skills/engineering/engineering-runtime/scripts/workflow-policy.mjs +1 -1
- package/skills/engineering/ensure-coverage/SKILL.md +22 -109
- package/skills/engineering/ensure-coverage/references/breadth/coverage-ledger.md +7 -5
- package/skills/engineering/ensure-coverage/references/breadth/surface-baseline.md +4 -4
- package/skills/engineering/ensure-coverage/references/breadth/surface-discovery.md +3 -3
- package/skills/engineering/ensure-coverage/references/depth/characterization.md +4 -65
- package/skills/engineering/ensure-coverage/references/depth/grading.md +5 -7
- package/skills/engineering/ensure-coverage/references/depth/mock-policy.md +6 -12
- package/skills/engineering/ensure-coverage/references/enforcement/ci-contract.md +1 -1
- package/skills/engineering/ensure-coverage/references/presentation/axis.md +6 -6
- package/skills/engineering/ensure-coverage/references/process/audit-mode.md +7 -23
- package/skills/engineering/ensure-coverage/references/process/output-template.md +18 -133
- package/skills/engineering/ensure-coverage/references/process/test-strategy.md +7 -12
- package/skills/engineering/ensure-coverage/scripts/README.md +12 -4
- package/skills/engineering/golden-path/SKILL.md +1 -1
- package/skills/engineering/harness-runtime/SKILL.md +3 -1
- package/skills/engineering/harness-runtime/bin/workspaces.mjs +52 -0
- package/skills/engineering/harness-runtime/native-workspace.mjs +239 -0
- package/skills/engineering/harness-runtime/references/workspaces.md +78 -0
- package/skills/engineering/harness-runtime/workspace-lifecycle.mjs +2154 -0
- package/skills/engineering/resolve-issues/SKILL.md +40 -61
- package/skills/engineering/resolve-issues/references/ci.md +13 -3
- package/skills/engineering/resolve-issues/references/delivery-progress.md +4 -3
- package/skills/engineering/resolve-issues/references/done.md +11 -2
- package/skills/engineering/resolve-issues/references/expire.md +9 -2
- package/skills/engineering/resolve-issues/references/implementation.md +33 -17
- package/skills/engineering/resolve-issues/references/preflight.md +3 -1
- package/skills/engineering/resolve-issues/references/principles.md +52 -0
- package/skills/engineering/resolve-issues/references/review.md +25 -15
- package/skills/engineering/resolve-issues/references/task-plan.md +64 -27
- package/skills/engineering/resolve-issues/references/workspaces.md +164 -0
- package/skills/engineering/resolve-issues/scripts/cleanup-runs.mjs +185 -317
- package/skills/engineering/resolve-issues/scripts/next-actions.mjs +44 -7
- package/skills/engineering/resolve-issues/scripts/rehydrate.mjs +36 -2
- package/skills/engineering/resolve-issues/scripts/review-verdict.mjs +9 -8
- package/skills/engineering/resolve-issues/scripts/workspace-lifecycle.mjs +1 -0
- package/skills/engineering/resolve-release/SKILL.md +1 -1
- package/skills/engineering/rush-issues/SKILL.md +7 -5
- package/skills/engineering/rush-release/SKILL.md +1 -1
- package/skills/engineering/test-quality/SKILL.md +19 -4
- package/skills/engineering/test-quality/references/lint.md +1 -1
- package/skills/engineering/test-quality/references/principles.md +2 -2
- package/skills/engineering/test-quality/references/testing-conventions.md +1 -1
- package/skills/risk-compliance/compliance-audit/SKILL.md +4 -8
- package/skills/risk-compliance/compliance-audit/references/audience-and-reports.md +12 -0
- package/skills/risk-compliance/contract-compare/SKILL.md +17 -1
- package/skills/risk-compliance/contract-review/SKILL.md +6 -1
- package/skills/risk-compliance/contract-review/references/report-stage.md +17 -0
- package/skills/risk-compliance/doc-authenticity/SKILL.md +11 -1
- package/skills/risk-compliance/fraud-detection/LICENSE +3 -0
- package/skills/risk-compliance/fraud-detection/SKILL.md +148 -0
- package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/archetypes/first-party-fraud.md +3 -2
- package/skills/risk-compliance/fraud-detection/references/archetypes/third-party-fraud.md +47 -0
- package/skills/risk-compliance/fraud-detection/references/collection-payout-features.md +151 -0
- package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/contexts/bank-transfer.md +4 -3
- package/skills/risk-compliance/fraud-detection/references/contexts/card-payment.md +70 -0
- package/skills/risk-compliance/fraud-detection/references/contexts/payment-collection.md +34 -0
- package/skills/risk-compliance/fraud-detection/references/contexts/payout.md +34 -0
- package/skills/risk-compliance/fraud-detection/references/enrichment.md +74 -0
- package/skills/risk-compliance/fraud-detection/references/entities.md +48 -0
- package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/feature-engineering.md +69 -33
- package/skills/risk-compliance/fraud-detection/references/feature-sql.md +150 -0
- package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/report-rationale.md +4 -2
- package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/report-template.md +45 -23
- package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/review-checklist.md +60 -20
- package/skills/risk-compliance/fraud-detection/references/workflow.md +216 -0
- package/skills/risk-compliance/fraud-detection/release-check.mjs +95 -0
- package/skills/risk-compliance/payment-analysis/SKILL.md +36 -17
- package/skills/risk-compliance/payment-analysis/references/auth-rate-actions.md +2 -2
- package/skills/risk-compliance/payment-analysis/references/report-template.md +10 -2
- package/skills/risk-compliance/payment-analysis/references/visualization.md +11 -1
- package/skills/risk-compliance/portfolio-payment-analysis/LICENSE +3 -0
- package/skills/risk-compliance/portfolio-payment-analysis/SKILL.md +170 -0
- package/skills/risk-compliance/portfolio-payment-analysis/references/identity-and-reconciliation.md +46 -0
- package/skills/risk-compliance/portfolio-payment-analysis/references/metrics-and-reporting.md +170 -0
- package/skills/risk-compliance/portfolio-payment-analysis/references/risk-history.md +89 -0
- package/skills/risk-compliance/portfolio-payment-analysis/release-check.mjs +95 -0
- package/skills/risk-compliance/portfolio-payment-analysis/scripts/analyze.py +636 -0
- package/skills/risk-compliance/portfolio-payment-analysis/scripts/build_report.py +386 -0
- package/skills/risk-compliance/portfolio-payment-analysis/scripts/comparison_charts.py +425 -0
- package/skills/risk-compliance/portfolio-payment-analysis/scripts/csv_export.py +52 -0
- package/skills/risk-compliance/portfolio-payment-analysis/scripts/render.py +160 -0
- package/skills/risk-compliance/portfolio-payment-analysis/scripts/risk_history.py +435 -0
- package/skills/risk-compliance/similar-domain/SKILL.md +9 -4
- package/skills/risk-compliance/tld-detection/SKILL.md +124 -239
- package/skills/risk-compliance/tld-detection/agents/openai.yaml +1 -1
- package/skills/risk-compliance/tld-detection/references/clean.md +18 -51
- package/skills/risk-compliance/tld-detection/references/fx-conversion.md +56 -219
- package/skills/risk-compliance/tld-detection/references/metric-contract.md +109 -0
- package/skills/risk-compliance/tld-detection/references/pattern.md +99 -0
- package/skills/risk-compliance/tld-detection/references/report-template.md +49 -272
- package/skills/risk-compliance/tld-detection/scripts/detect_ladders.py +46 -13
- package/skills/risk-compliance/tld-detection/scripts/detect_patterns.py +346 -0
- package/skills/risk-compliance/tld-detection/scripts/detect_streams.py +51 -19
- package/skills/risk-compliance/tld-detection/scripts/profile_groups.py +109 -0
- package/skills/risk-compliance/tld-detection/scripts/render_charts.py +218 -0
- package/skills/risk-compliance/tld-detection/scripts/tld_common.py +243 -0
- package/skills/engineering/contributing/SKILL.md +0 -70
- package/skills/risk-compliance/fraud-analysis/SKILL.md +0 -114
- package/skills/risk-compliance/fraud-analysis/references/archetypes/third-party-fraud.md +0 -27
- package/skills/risk-compliance/fraud-analysis/references/contexts/card-payment.md +0 -30
- package/skills/risk-compliance/fraud-analysis/references/contexts/payment-collection.md +0 -20
- package/skills/risk-compliance/fraud-analysis/references/contexts/payout.md +0 -20
- package/skills/risk-compliance/fraud-analysis/references/workflow.md +0 -175
- package/skills/risk-compliance/tld-detection/references/chinese-polish.md +0 -55
- package/skills/risk-compliance/tld-detection/references/clustering.md +0 -235
- package/skills/risk-compliance/tld-detection/references/decision.md +0 -189
- package/skills/risk-compliance/tld-detection/references/streams.md +0 -141
- package/skills/risk-compliance/tld-detection/references/terminology.md +0 -95
- package/skills/risk-compliance/tld-detection/references/workflow.md +0 -95
- /package/skills/{engineering/contributing → core/write-report}/LICENSE +0 -0
- /package/skills/core/{write-doc → write-report}/references/reports/audit-report-template.md +0 -0
- /package/skills/core/{write-doc → write-report}/references/reports/external-assessment-template.md +0 -0
- /package/skills/core/{write-doc → write-report}/references/reports/management-report-template.md +0 -0
- /package/skills/core/{write-doc → write-report}/references/reports/vars-report-template.md +0 -0
- /package/skills/{engineering/contributing → core/write-report}/release-check.mjs +0 -0
- /package/skills/{risk-compliance/fraud-analysis → engineering/engineering}/LICENSE +0 -0
- /package/skills/engineering/{contributing → engineering}/references/tracker-conventions.md +0 -0
- /package/skills/{risk-compliance/fraud-analysis → engineering/engineering}/release-check.mjs +0 -0
- /package/skills/engineering/{contributing → engineering}/scripts/persist-conventions-core.mjs +0 -0
- /package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/archetypes/authorized-payment-scam.md +0 -0
- /package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/mechanisms/account-takeover.md +0 -0
- /package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/taxonomy.md +0 -0
- /package/skills/risk-compliance/{fraud-analysis → fraud-detection}/references/terminology.md +0 -0
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
> Read when: computing the minimum required-level floor a surface owes. The breadth axis enumerates surfaces; this is the floor each one carries the instant it's discovered. Sibling refs: [coverage-ledger.md](coverage-ledger.md) (the ledger that holds these obligations), [surface-discovery.md](surface-discovery.md) (how surfaces and their `reachability` are found), [inventory-contract.md](inventory-contract.md) (the adapter that emits the taxonomy).
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Use the table as guidance when selecting proof for the agreed scope and material risks, not as a universal obligation for newly discovered surfaces. Existing repository-adopted floors and required checks remain binding unless changed through approval. A requirement document is not needed to recognize a concrete risk. The rules below describe the bundled ledger's mechanical defaults when it is used, not new policy for every feature.
|
|
6
6
|
|
|
7
7
|
**The baseline keys off an adapter-emitted taxonomy, never inference at grading time** — inference-per-session is how "covered" came to mean different things. The adapter tags each surface deterministically so two agents on the same code compute the same floor.
|
|
8
8
|
|
|
@@ -27,9 +27,9 @@ Baseline only **entry surfaces** (`route`, `api`). **Internal surfaces** (`table
|
|
|
27
27
|
| permission | — | `integration` | resolves an auth test |
|
|
28
28
|
| integration (external) | — | fake adapter + contract | — |
|
|
29
29
|
|
|
30
|
-
|
|
30
|
+
In the ledger, every **live browser route is `browser-journey` at minimum** — `class` changes only *what the journey additionally proves*, not *whether* you drive the page. A genuinely static/content page (marketing, docs) is downgraded via the **`static` exclusion category** to a `render-health` floor — a cheap reachability guarantee, never weakening the floor for real app pages. (`smoke` is accepted as an alias for this level; it is not the curated `smoke` gate, which is a cross-flow liveness check, not a per-route level.) **`render-health` is a browser-level check, not an API ping**: a real browser must load the rendered page and confirm it renders without error (externals may be faked). An HTTP-200 / API-level test does **not** satisfy it. The other exclusion categories (`seed`/`deprecated`/`internal`) drop to manifest-only. `render-health` is the one level a present-but-shallow test can satisfy, precisely because it is definitionally shallow. (Same "page loads/renders" concept as e2e-test's Tier-1 and the `presentation:render-health` spine.)
|
|
31
31
|
|
|
32
|
-
|
|
32
|
+
In the ledger, every live browser route *also* auto-carries the **Presentation floor** on top of the functional floor above — see [../presentation/axis.md](../presentation/axis.md).
|
|
33
33
|
|
|
34
34
|
## Exclusion is gated, not honored on a reason
|
|
35
35
|
|
|
@@ -41,4 +41,4 @@ An exclusion downgrades a surface to manifest-only — the exact move that re-op
|
|
|
41
41
|
|
|
42
42
|
## The feature card is a deviation record, not a precondition
|
|
43
43
|
|
|
44
|
-
|
|
44
|
+
When using the ledger, because its baseline auto-applies, a surface with no requirement is **not blocked on authoring a file** — it is already graded at its floor. A curated `…coverage.yml` (the "feature card") is required only to **deviate**: raise above baseline, mark `sensitive`, exclude, attest (`verified`), or attach `owner`/`purpose`/`userVisibleContract` ownership. Pure happy-path baseline surfaces need no card. Requirement *ownership* (traceability) is a `warn`; the baseline *test obligation* is the hard gate.
|
|
@@ -4,13 +4,13 @@
|
|
|
4
4
|
>
|
|
5
5
|
> **Guard-chain vs navigational reachability.** This protocol determines **guard-chain** reachability — "can the URL load past middleware/auth." That is *different* from `presentation:reachability` ([../presentation/axis.md](../presentation/axis.md)), which is **navigational** — "can a user *click* their way here from an entry point." A route can be guard-reachable yet navigation-orphaned. Both matter; they are different checks.
|
|
6
6
|
|
|
7
|
-
Use this
|
|
7
|
+
Use this protocol within the agreed audit scope, even if `e2e-test` is unavailable. Enumerate the whole repository only for an explicitly requested broad audit. For scoped work, inspect relevant routes and their reachability as needed; the following describes comprehensive discovery for Next.js App Router and similar file-based routers:
|
|
8
8
|
|
|
9
9
|
1. Enumerate all `page.*` and `route.*` files, including route groups, dynamic segments, parallel/intercepting routes, and exported HTTP methods for `route.*`. Tag each by kind — **browser route** (`page.*`) or **API route** (`route.*`) — because they carry different depth obligations and proof types ([../depth/grading.md](../depth/grading.md)). Never let one stand in for the other.
|
|
10
10
|
2. For each page route, walk the layout chain root → route group → segment; inspect `layout.*`, `middleware.*`, redirects, `notFound`, auth/session checks, role/tenant guards, feature flags, rewrites.
|
|
11
11
|
3. Classify reachability as `Reachable`, `Redirected`, `Guarded`, `Blocked`, or `Unknown`, with the guard/layout/middleware evidence.
|
|
12
|
-
4. For
|
|
12
|
+
4. For relevant `Reachable` pages, identify user-actionable mutations and their backend targets. Distinguish rendering from submission and durable-state evidence; select proof based on agreed criteria and material risks, not an automatic journey obligation per mutation.
|
|
13
13
|
5. Cross-reference discovered routes against PRD/spec claims, route manifests, navigation links, product journeys, and the golden-path candidate list. Discovery may suggest golden-path candidates, but the golden-path contract is curated: a route is not release-blocking merely for being a mutation route or containing an auth/payment-looking word.
|
|
14
|
-
6.
|
|
14
|
+
6. Report relevant concrete gaps and unavailable evidence. Treat missing journey coverage according to agreed criteria and existing required floors, not as an automatic feature blocker.
|
|
15
15
|
|
|
16
16
|
Do not classify a route as covered, missing, or part of a valid journey solely because a `page.*`/`route.*` file exists — a route file is only a declared route until its guard chain and journey validity are checked.
|
|
@@ -1,68 +1,7 @@
|
|
|
1
|
-
#
|
|
1
|
+
# Characterization and fails-first evidence
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
When changing existing behavior, consider whether a relevant existing test already protects the behavior that should stay unchanged. Reuse it if sufficient. Add a characterization test when the risk of unintended change warrants it; it captures actual behavior at an observable boundary, including quirks. Do not require a new test or pre-change run for every source edit, or introduce a seam solely to satisfy a blanket rule.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
For a bug fix, a test that fails before the fix and passes afterward can show that the fix addresses the defect. Use it when practical and useful or when repository policy requires it; otherwise explain the proof and any limitation. A feature test should make meaningful assertions about the agreed outcome, not just a mock call or a fixture that bypasses the real behavior. Inspect it before claiming coverage; a broken-tree run is not automatically required.
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
## The obligation: when it is owed
|
|
10
|
-
|
|
11
|
-
A change owes a characterization test when **all** hold:
|
|
12
|
-
|
|
13
|
-
- It **edits or removes** existing lines of a non-test **source** file (a pure addition into a new file, or appended-only code, is new behavior → TDD path).
|
|
14
|
-
- The touched code has **observable behavior** — return value, durable-state effect, emitted event, HTTP contract, rendered output. (Internal-only renames with no behavioral surface don't.)
|
|
15
|
-
- The surface is **non-trivial** — anything carrying a Surface Baseline floor, or logic a user/caller depends on.
|
|
16
|
-
|
|
17
|
-
When owed, the test must: **(1) capture actual, not intended, behavior** including quirks/bugs (if current behavior is wrong, characterize it *then* fix it in a separate declared step); **(2) pass on the pre-change tree** (GREEN on `HEAD~`/base before your edit — capture as evidence); **(3) still pass after the change**, unless the change is a declared intended behavior change.
|
|
18
|
-
|
|
19
|
-
## When it is waived
|
|
20
|
-
|
|
21
|
-
Mark the waiver explicitly in the PR's evidence, never silently:
|
|
22
|
-
|
|
23
|
-
- **Already pinned** — an existing behavior test covers the exact changed path and still passes unchanged. Cite it (`file:line`); that test *is* the characterization.
|
|
24
|
-
- **Declared intended behavior change** — deliberately alters observable behavior per an AC/bug brief. The obligation inverts: update/author the test to assert the **new** behavior, and the diff must show the old assertion changing to the new (a behavior change with no test delta is the suspicious case). Cite the AC.
|
|
25
|
-
- **Trivial / generated / non-behavioral** — formatting, comments, type-only edits, generated code with no behavior delta.
|
|
26
|
-
|
|
27
|
-
## The `@characterizes` convention
|
|
28
|
-
|
|
29
|
-
Tag a characterization test so the gate can resolve it, parallel to `@covers`:
|
|
30
|
-
|
|
31
|
-
```js
|
|
32
|
-
// @characterizes <surface-id> // pins prior behavior of a surface being modified
|
|
33
|
-
// @covers <surface-id> // (a characterization test usually also @covers it)
|
|
34
|
-
// @level <level>
|
|
35
|
-
```
|
|
36
|
-
|
|
37
|
-
`@characterizes` is the signal that *this* test exists to hold a modification's behavior boundary. A modifying diff with no `@characterizes` test and no declared waiver is the gap this obligation names.
|
|
38
|
-
|
|
39
|
-
## Grading
|
|
40
|
-
|
|
41
|
-
Characterization is a **depth** concern, graded with the [shared vocabulary](grading.md):
|
|
42
|
-
|
|
43
|
-
- A modifying diff whose changed behavior is pinned by a passing characterization test (or cited pre-existing behavior test) → that surface can reach `Covered` on the normal evidence rules.
|
|
44
|
-
- A modifying diff with **no** characterization test and **no** declared intended-change → `Partial` at best; the reviewer raises the `modifies-existing-behavior` lead (below). Not a hard `Missing` (existing tests may incidentally cover it) — a **lead to adjudicate**, because whether a characterization test is *needed and adequate* is judgment.
|
|
45
|
-
|
|
46
|
-
## Detection (mechanical) → adjudication (judgment)
|
|
47
|
-
|
|
48
|
-
The decidable half is mechanical: any non-test source file with **removed/changed** lines emits a `modifies-existing-behavior` **lead** ("look here and verify," never an auto-finding). The reviewer checks for a `@characterizes` test (or a declared change) and adjudicates, citing `file:line`. The reviewer resolves it with a locator, not prose. The split matches the GATE-vs-LEAD line everywhere in this skill: detecting *that behavior changed* is mechanical; deciding *whether the pinning test is adequate* is semantic.
|
|
49
|
-
|
|
50
|
-
## Seams: where to pin
|
|
51
|
-
|
|
52
|
-
A characterization test is writable only where you can observe behavior without rewriting the thing you're about to change — a **seam** (*"a place where you can alter behavior without editing the code in that place"*). Pin at the nearest observable seam: the public function's return/throw, the endpoint's HTTP contract, the rendered page's output, the emitted event. If there's no seam, that itself is the finding — introduce one first (extract the call, expose the boundary), pin, *then* change. Do not skip characterization because "it's hard to test as written"; hard-to-pin is the legacy-code smell the obligation exists to surface.
|
|
53
|
-
|
|
54
|
-
## Fails-first: bug-fix AND feature-AC — prove the test fails when the behavior is broken
|
|
55
|
-
|
|
56
|
-
The shared property: **a test is evidence for a behavior only if it would go red were that behavior broken.** A green test that passes with *and* without the behavior it claims is **vacuous coverage** — it locks in nothing. Two cases:
|
|
57
|
-
|
|
58
|
-
**Bug-fix sibling.** A bug fix changes behavior that's *already wrong*, so it owes the inverse of characterization: at least one test demonstrated to **fail on the unfixed code and pass on the fixed code** (RED→GREEN, run in that order), tagged `// @fails-without-fix <surface>`. Reverting the fix must turn the test red. Waive only with a cited reason the failure can't be reproduced in-suite (a tracked `Unverified` manual check, not a silent skip).
|
|
59
|
-
|
|
60
|
-
**Feature-AC sibling.** The same test applies to a *new feature's* AC, where it's most-missed: a green test cited for an AC clears it only if it would go red were the AC's behavior removed — it exercises the AC's real code path, not a proxy. Recurring vacuous forms:
|
|
61
|
-
- a fixture that **early-returns before the code runs** (a 0-step workflow "proving" a multi-step reset that never executes);
|
|
62
|
-
- a `.fill()`/`setValue` that sets state in one shot and so **masks a re-render/defocus bug** a character-by-character `pressSequentially` would expose;
|
|
63
|
-
- an assertion **pinned to a stand-in value the real producer never emits**;
|
|
64
|
-
- asserting **the mock was called** rather than that the outcome happened.
|
|
65
|
-
|
|
66
|
-
**Grading the two outcomes** (they differ from [grading.md](grading.md)'s `Unverified`): can't-yet-confirm it would go red → `Unverified` (a warning unless `unverifiedBlocks: true`); confirm it would **still pass** with the behavior gone → `Missing`/`Over-mocked` (a proven-vacuous test is *worse* than unchecked).
|
|
67
|
-
|
|
68
|
-
The two **decidable** vacuous forms (a swallowed assertion, a fixed timeout) are gated by `lint-tests`; the semantic "would it have been red?" property is verified at review and by running the test against the broken tree. Named incident: the login regression (#237), where a green sweep asserted nothing the bug could trip.
|
|
7
|
+
If an existing adopted gate requires annotations, a pre-change run, or other evidence, follow it unless an approved change alters the gate. Grade evidence honestly using [grading](grading.md).
|
|
@@ -2,11 +2,9 @@
|
|
|
2
2
|
|
|
3
3
|
> Read when: grading whether a test actually proves the behavior against real state. Sibling: [mock-policy.md](mock-policy.md) (the boundary principle this grading enforces). Depth grades each `Aligned` row — and each `Orphan surface` carrying a [Surface Baseline Contract](../breadth/surface-baseline.md) floor — by inspecting its actual tests.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
A depth failure occurs when a substitute bypasses the behavior a test claims to prove. For example, a Playwright interception of an endpoint cannot prove that endpoint's real behavior. Grade that claim `Over-mocked`, not `Covered`; a focused UI test may still prove its own scope. See the [mock policy](mock-policy.md).
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
**For any write form, require one test that fills the rendered UI and asserts the mutation end to end** — operator fills the actual form and submits → backend accepts → durable state changes. Keep the render-health watcher attached **through** the submit (compose with `presentation:render-health`, [../presentation/axis.md](../presentation/axis.md)), so an uncaught error / `console.error` / first-party 5xx fired *during* the write fails the test even when the durable-state assertion would pass — the "mutate-health" gap. Do not accept seam tests in its place: a schema endpoint returning fields, a validator rejecting bad input, and an API forwarding a hand-built payload all pass while the rendered form silently drops a backend-required field. This is acute for the **split-contract form**, whose fields come from one source (schema endpoint, config descriptor, generated form spec) and validation from another (the backend contract); only a rendered-form submission proves the form emits a payload the backend accepts.
|
|
7
|
+
Distinguish what API and browser evidence actually proves. An API test alone does not prove a rendered form works; a UI test that fakes the backend does not prove durable state. Select the proof needed for the agreed outcome and material risk, reusing existing evidence where sufficient. A rendered end-to-end submission can expose split-contract form failures, but is not an automatic new obligation for every form. Existing adopted floors and required checks remain binding.
|
|
10
8
|
|
|
11
9
|
## Evidence Hierarchy
|
|
12
10
|
|
|
@@ -25,11 +23,11 @@ Classify coverage as `Covered` only after inspecting the relevant test file/case
|
|
|
25
23
|
|
|
26
24
|
A self-declared `@level` is a claim the validator must try to **refute**, and can never alone reach `Covered`. The three terminal states are marked below.
|
|
27
25
|
|
|
28
|
-
- `Covered` (green, terminal): inspected test code validates the AC at the
|
|
29
|
-
- `Partial`: some behavior
|
|
26
|
+
- `Covered` (green, terminal): inspected test code validates the agreed AC at the appropriate level with meaningful assertions and acceptable mock/fake use. A self-declared `@level` alone never reaches it. The ledger additionally requires its `verified: true` attestation.
|
|
27
|
+
- `Partial`: some agreed behavior or material-risk proof is incomplete. An untested hypothetical state alone is not an automatic blocker.
|
|
30
28
|
- `Missing`: no meaningful test maps to the AC.
|
|
31
29
|
- `Wrong Level`: test exists but is too shallow for the risk — e.g. unit-only for stateful workflow, API contract, auth, money movement, concurrency, or event behavior.
|
|
32
|
-
- `Over-mocked` (red, terminal):
|
|
30
|
+
- `Over-mocked` (red, terminal): substitutes bypass the behavior the agreed AC must prove. For example, an intercepted endpoint cannot prove its real durable-state path; the test may still prove a narrower UI behavior.
|
|
33
31
|
- `Unverified` (yellow, terminal): the level is satisfied statically (or claimed by docs/reports/CI) but nobody has read the test yet. It is a **warning** by default; it blocks only when the user sets `unverifiedBlocks: true`. Marking `verified: true` after reading the test promotes it to `Covered`; no evidence is required unless the user asks for it. The residual variance is over *quality of an existing test*, not *whether a surface exists*.
|
|
34
32
|
- `Excluded`: a human marked the surface out of scope with category + reason and the adapter signals permit it; surfaced in the exclusion census.
|
|
35
33
|
- `BLOCKED`: required source, code, tests, route discovery, environment, or product decision is unavailable.
|
|
@@ -4,11 +4,7 @@
|
|
|
4
4
|
|
|
5
5
|
## Mocking Stance
|
|
6
6
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
- A test substitute is permitted **only at a trust boundary you do not own**: third-party HTTP APIs, payment/KYC providers, email/SMS, the clock, randomness.
|
|
10
|
-
- Everything **inside** your own system runs for real: database, repositories, service layer, internal endpoints, event bus. A substitute for an internal seam is a design smell or coverage gap, not a testing choice.
|
|
11
|
-
- The rule is **two-sided**: don't mock inward, don't reach outward — externals must be replaced by a fake/stub adapter so tests stay deterministic and never call a live third party in CI. "Be pure by calling the real provider" is the opposite failure, also rejected.
|
|
7
|
+
Choose substitutes based on what the test must prove and the material risk. Unit tests may isolate internal collaborators, but a substitute cannot prove the behavior it replaces. Use real boundaries where needed to prove security, tenancy, money, or durable-state behavior, subject to existing repository policy. Fake external providers when necessary for safe, deterministic tests; follow the repository's rules for live services.
|
|
12
8
|
|
|
13
9
|
The problem we reject is **mock-only coverage of behavior that must be proven against real state** — not the presence of mocks. Fakes/stubs that keep failure-path and edge tests fast and deterministic are good and expected.
|
|
14
10
|
|
|
@@ -43,15 +39,13 @@ These are **internal** and are therefore never covered by substitutes alone:
|
|
|
43
39
|
|
|
44
40
|
Prefer integration tests with real app/service code, real DB or test containers with migrations, queue/pub-sub emulators where practical, fake external provider adapters, and assertions on final durable state, emitted events, and externally visible outcomes.
|
|
45
41
|
|
|
46
|
-
##
|
|
42
|
+
## Optional scan: `mock-internal-seam`
|
|
47
43
|
|
|
48
|
-
The
|
|
44
|
+
The following describes the scanner's strict mechanics, not a universal ban on internal test doubles. Run it when requested or required by the repository; a whole-tree scan belongs to an explicitly requested broad audit. Assess hits against the behavior and risk rather than counting them as automatic feature blockers. Existing required gates remain binding.
|
|
49
45
|
|
|
50
|
-
|
|
51
|
-
- **
|
|
52
|
-
- **Three resolutions, and "keep it" is not one:** replace with a real-boundary test, fake only the external behind it, or **delete** the test. Over-mock is worse than no test: an untested route is an honest gap, an over-mocked one is dishonest green. Deletion reverts the surface to its honest breadth status (`Missing` or `Excluded`).
|
|
53
|
-
- **In audit mode, run it whole-tree from this skill against the target repo** and count every hit as an `Over-mocked` row and a no-ship contributor. A missing `coverage.config.json` means "run on defaults", not "not applicable"; the scripts' absence from the audited repo means nothing.
|
|
46
|
+
The scanner uses an **externals-allowlist inversion**: it flags mocks resolving to first-party code and exempts declared externals, bare npm packages, and the standard library. Its default configuration is strict; a flag does not itself establish that a test is inadequate. Where an adopted required gate fails, report the failure and seek the appropriate approved resolution rather than bypassing it. Do not delete a useful test merely to silence an optional scan.
|
|
47
|
+
- **When run**, a missing `coverage.config.json` uses defaults. The scripts live in this skill, not the audited repo. A hit is a lead; determine whether the test actually bypasses the behavior it claims to prove.
|
|
54
48
|
|
|
55
|
-
|
|
49
|
+
Whether a real-boundary sibling exists, assertions are shallow, or fakes are wrong requires judgment. The scanner detects patterns, not their full meaning. The ledger's over-mock rate is a trend signal; do not invent a zero-count merge bar.
|
|
56
50
|
|
|
57
51
|
CLI, rules, and the `coverage.config.json` topology keys: [`scripts/README.md`](../../scripts/README.md).
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
> Read when: the user asks what this skill should require, wire, or audit in CI.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
When asked to propose a CI gate, this skill supports curated golden-path smoke. Do not propose, require, wire, or audit additional gates without an explicit request. Existing repository-required checks, including checks outside this menu, remain binding; do not bypass or remove them.
|
|
6
6
|
|
|
7
7
|
## What smoke proves
|
|
8
8
|
|
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
> Read when: working any `presentation:*` invariant. Sibling: `e2e-test/references/presentation-sweep.md` (the runner and the result shape it emits). This skill *grades* the axis; the `e2e-test` sweep *runs* it.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
Consider presentation evidence when it matters to the agreed outcome or material risks. The following describes the bundled ledger and sweep's per-route mechanics, not a new universal requirement or an automatic presentation campaign. Existing repository-adopted floors and required checks remain binding unless changed through approval. In the ledger, every live browser route carries a presentation floor at `[375, 1280]` (default theme, seeded data).
|
|
6
6
|
|
|
7
7
|
> **This skill grades the axis; it does not run it.** The tokens below are *graded* by the Coverage Ledger (does a qualifying real-render test exist, is it non-intercepting, is it attested). The *instruments* that open a browser and observe these invariants live in the **`e2e-test` skill's presentation sweep**. ensure-coverage owns the **obligation**; the runner and the per-`(route, level)` result shape the ledger consumes are documented with the runner (`e2e-test/references/presentation-sweep.md`). This mirrors how breadth delegates surface discovery to an inventory adapter ([../breadth/inventory-contract.md](../breadth/inventory-contract.md)).
|
|
8
8
|
|
|
@@ -14,7 +14,7 @@ Invariants, **spine first**:
|
|
|
14
14
|
- **`presentation:reachability`** — a user can get to the page and tell its controls are actionable: every manifest route is reachable by crawling the rendered nav (no **navigational orphan**), interactive controls show a visible focus indicator and meet the ≥44px target floor. Negative form: a modal traps focus (no click-through / focus escape); a privilege-gated route is not navigationally reachable for the wrong role.
|
|
15
15
|
- **`presentation:temporal`** — the layout holds still as things inject. **Split by who renders the shift** — *first-party* (lazy images, async data, our dynamic fields) is a **hermetic gate** (real render, no network; before/after geometry gated on a stable condition, never a fixed sleep, or a `PerformanceObserver` CLS score). *Third-party* (Stripe/Adyen payfields) runs against the provider **sandbox iframe** in a **non-blocking PR lane**: a failure posts a visible annotation and the surface stays `Unverified` until the test is read and marked `verified: true`. Top-page CLS is **structurally blind to cross-origin iframes** (WICG excludes descendant frames), so this is a before/after geometry assertion gated on a provider-ready signal, not CLS. The mobile **dynamic-viewport (`dvh`/`100vh`) cell** is gated via Playwright emulation with a tuned tolerance (staging fallback if flake is unmanageable).
|
|
16
16
|
|
|
17
|
-
"Graded" means the ledger marks a route's floor `Missing` / `Over-mocked` / `Unverified` / `Covered` from the tests that exist — it cannot itself detect a runtime crash, which is why the sweep is a **separate evidence run**, not a side effect of the functional E2E suite. It is not a CI gate;
|
|
17
|
+
"Graded" means the ledger marks a route's floor `Missing` / `Over-mocked` / `Unverified` / `Covered` from the tests that exist — it cannot itself detect a runtime crash, which is why the sweep is a **separate evidence run**, not a side effect of the functional E2E suite. It is not a new CI gate; existing repository-required checks remain binding ([CI contract](../enforcement/ci-contract.md)).
|
|
18
18
|
|
|
19
19
|
The floor runs at the screen viewport floor (`[375, 1280]`). The **`media=print` cell is warn-only by default** — a broken print stylesheet is logged in the census, not a blocking gate (lower-frequency, harder to assert deterministically). A surface may escalate it to a gate when print output is a real product surface (e.g. an invoice).
|
|
20
20
|
|
|
@@ -37,7 +37,7 @@ A cell-qualified level grades exactly like its base (real render, exact-match, D
|
|
|
37
37
|
|
|
38
38
|
A presentation level is **exact-match** and proven only against a **real rendered page**:
|
|
39
39
|
|
|
40
|
-
-
|
|
40
|
+
- In the ledger, a functional `browser-journey` does not automatically satisfy a separate presentation level. For scoped proof, a single test can demonstrate multiple outcomes if its assertions actually cover them.
|
|
41
41
|
- A DOM-only / JSDOM assertion of a presentation level grades **`Over-mocked`** — the axis's proxy-wiring (it can't see what a human sees). The proving test must drive a real browser.
|
|
42
42
|
- Like any obligation, a statically-satisfied presentation level lands at `Unverified` until the test is read and marked `verified: true`.
|
|
43
43
|
|
|
@@ -51,7 +51,7 @@ Every level above is **self-referential**: render-health asks whether the page r
|
|
|
51
51
|
|
|
52
52
|
That gap is not theoretical. A baseline captured from a wrong implementation is green forever — in the incident below it would have locked in a card plane the design did not have and a flattened type scale, and reported success. Drift detection cannot substitute for fidelity, because it takes the implementation as its own authority.
|
|
53
53
|
|
|
54
|
-
**When it is
|
|
54
|
+
**When it is relevant.** Compare the implementation with a cited design where the agreed outcome calls for fidelity. Use the parity workflow when requested or required by the repository; citing a design alone does not automatically mandate a campaign.
|
|
55
55
|
|
|
56
56
|
**The rule that makes it fidelity.** Every expected value is read off the **source**, with it rendered, and cites the source line it came from. A value copied off the page you built is a baseline wearing a fidelity label.
|
|
57
57
|
|
|
@@ -59,6 +59,6 @@ That gap is not theoretical. A baseline captured from a wrong implementation is
|
|
|
59
59
|
|
|
60
60
|
**No rewriting the target to match the page.** The cheapest way to green this gate is to edit `expected` instead of fixing the page. Because the source is pinned, a changed `expected` under an unmoved `source.sha` is mechanically detectable and blocks; a genuine mis-read is declared with `rebase: <reason>`. The reviewer treats an undeclared rewrite as a blocker, exactly as it does a silent re-baseline.
|
|
61
61
|
|
|
62
|
-
|
|
62
|
+
When the parity workflow is agreed or required, rendering source and deliverable side by side can reveal differences that a property list misses.
|
|
63
63
|
|
|
64
|
-
|
|
64
|
+
The fmm-express #521 incident shows why a presentation-focused change may need direct comparison with its design source. Agree that proof before implementation; do not impose the full parity workflow on unrelated features.
|
|
@@ -1,27 +1,11 @@
|
|
|
1
1
|
# Audit Mode (process)
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Audit the agreed feature scope against its observable criteria and material risks. A whole-repo census is only for an explicitly requested broad audit. Existing required checks and adopted floors remain binding.
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
1. Establish the source, scope, criteria, and required repository checks. Identify product or architecture questions for their owners.
|
|
6
|
+
2. Inspect relevant behavior and existing tests; map evidence to the agreed criteria. A prior report is a lead, not proof of test content. Reuse sufficient proof rather than requiring redundant layers.
|
|
7
|
+
3. Grade honestly using [grading](../depth/grading.md). Describe concrete material gaps and any unavailable evidence. A hypothetical untested state alone is not a feature blocker.
|
|
8
|
+
4. Report what was verified, failed or unavailable, distinguishing required fixes and decisions from speculative follow-ups. A newly discovered concrete defect may need a fix; do not silently add criteria after implementation starts.
|
|
9
|
+
5. Audit CI wiring only when asked. A missing optional coverage gate is not a finding. Required repository checks still apply.
|
|
6
10
|
|
|
7
|
-
|
|
8
|
-
2. Identify public interfaces, workflows, state transitions, side effects, code paths, and existing tests.
|
|
9
|
-
3. For E2E audit, run the [Route Discovery & Reachability Protocol](../breadth/surface-discovery.md) before judging coverage.
|
|
10
|
-
4. Identify critical journeys with actor/role, start state, trigger, major steps, expected end state, route reachability, and observable side effects.
|
|
11
|
-
5. Map each AC and journey to existing tests only after inspecting the actual test files/cases/assertions; prior docs, PR summaries, or agent reports are leads, not proof.
|
|
12
|
-
6. Reject coverage that only validates private functions, implementation structure, or shallow route render-health checks for business journeys.
|
|
13
|
-
7. Create follow-up items for every gap with required test level, setup/input, assertions, and mock/fake policy.
|
|
14
|
-
8. Reach a production-readiness verdict — an explicit ship / no-ship call, the specific scenarios a real user would hit in the first week, and the evidence that would raise confidence — and **put it at the TOP of the deliverable** (the reader's first question is "what's the call?"; the census and detail justify it below). Do not let mapped ACs alone earn a ship call. Separate **blocking decisions** (product/scope choices that change the verdict and need a human answer) from **follow-up actions** (work to schedule) — see the output template; don't dump both into one vague "open questions" list.
|
|
15
|
-
9. **Do not audit CI wiring** unless the user asks. A repo without coverage gates in CI is not a finding. When the user does ask, run `ci-audit.mjs` against the repo's CI config and report curated golden-path smoke as present / warn-only / absent. Do not turn the other audit checks into CI findings; they contribute directly to the coverage verdict.
|
|
16
|
-
10. **Run the over-mock scan whole-tree** (`lint-tests.mjs --rules mock-internal-seam`, no `--diff`) from this skill against the target repo. Every hit is an `Over-mocked` census row and a no-ship contributor; **count them in the verdict**. A missing `coverage.config.json` means "run on defaults", not "not applicable"; a non-conventional topology earns a "declare topology" follow-up, never a downgrade of the finding ([../depth/mock-policy.md](../depth/mock-policy.md)).
|
|
17
|
-
11. If code, tests, or route discovery cannot be inspected or run, mark the assessment `BLOCKED` and state the missing access or command failure. "The lint script isn't in the audited repo" is not `BLOCKED`: the scripts are part of this skill ([../../scripts/README.md](../../scripts/README.md)).
|
|
18
|
-
|
|
19
|
-
The coverage classifications (`Covered` / `Partial` / `Missing` / `Wrong Level` / `Over-mocked` / `Unverified` / `Excluded` / `BLOCKED`) are defined in [../depth/grading.md](../depth/grading.md). Use that vocabulary, not a vague "covered."
|
|
20
|
-
|
|
21
|
-
## Completeness: enumerate every surface, not just the salient gaps
|
|
22
|
-
|
|
23
|
-
**Gap-finding must be exhaustive, not a highlight reel.** An audit that lists "the top 3 problems" silently leaves the long tail uncovered — where the un-ACed orphan hides. Completeness comes from **closed-world enumeration**:
|
|
24
|
-
|
|
25
|
-
1. **Start from the full surface inventory** (every route, API, write form, table, event, job — from [surface discovery](../breadth/surface-discovery.md) / the generated inventory), not the list of things you happened to notice.
|
|
26
|
-
2. **Emit a complete surface census**: one row per inventoried surface with its status. `gaps = inventory − {surfaces at Covered}`, so every surface is accounted for *by construction*. The census is the deliverable's backbone (the **Surface Census** table in [output-template.md](output-template.md)); the prose highlights the worst rows, but the table proves nothing was skipped.
|
|
27
|
-
3. **Every non-`Covered`/non-`Excluded` row is a gap** with its own follow-up item. If the inventory cannot be fully enumerated (no adapter, partial access), say so and mark the unenumerated portion `BLOCKED`; do not silently scope to what was easy to see.
|
|
11
|
+
For an explicitly requested broad audit, define its boundaries and use [discovery](../breadth/surface-discovery.md) or a census when useful. Report incomplete access or inventory rather than claiming completeness. The optional whole-tree over-mock scan produces leads to assess in context, not automatic feature no-ship findings; an existing required gate retains its own result.
|
|
@@ -1,139 +1,24 @@
|
|
|
1
|
-
# Output Template, Writing Rules & Validation
|
|
1
|
+
# Output Template, Writing Rules & Validation
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
## Lead with the verdict
|
|
6
|
-
|
|
7
|
-
An **audit deliverable opens with the verdict**, not buries it under tables. Put a one-line `## Verdict: Ship / No-ship` (with the one-sentence reason) as the **first section**, before the Surface Census and the detailed assessment, which then *justify* it below. (Create-mode criteria docs lead with Source/Scope — there's no verdict yet.)
|
|
8
|
-
|
|
9
|
-
**A no-ship is a number, not a vibe.** The verdict carries the counts that make it no-ship: uncovered surfaces, orphan surfaces, ledger warnings, skipped tests, over-mock findings. If a finding has a count, the count goes in the summary; "it doesn't block CI yet" is never a license to omit it.
|
|
10
|
-
|
|
11
|
-
## Writing Rules
|
|
12
|
-
|
|
13
|
-
- Make each AC independently testable.
|
|
14
|
-
- Prefer observable outcomes over internal implementation details.
|
|
15
|
-
- Include negative cases and misuse paths, not only happy paths.
|
|
16
|
-
- For async/pub-sub/orchestrator systems, require correlation IDs and final durable state assertions.
|
|
17
|
-
- For idempotent APIs, require duplicate request tests and concurrent request tests when money/state transitions are involved.
|
|
18
|
-
- For security-sensitive flows, require authorization tests for cross-tenant/cross-merchant access.
|
|
19
|
-
- For provider integrations, use fake adapters in integration tests and real providers only in staging smoke tests.
|
|
20
|
-
- Avoid vague criteria like “works correctly,” “fast,” or “secure”; replace them with measurable expectations.
|
|
21
|
-
|
|
22
|
-
## Output Template
|
|
23
|
-
|
|
24
|
-
Use this structure unless the user asks for another format:
|
|
3
|
+
Use the issue or conversation by default. Create a document only when requested or required by the repository. Adapt this optional structure to the agreed scope rather than filling every section for routine work.
|
|
25
4
|
|
|
26
5
|
```markdown
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
-
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
-
|
|
38
|
-
-
|
|
39
|
-
-
|
|
40
|
-
|
|
41
|
-
> **"Out of scope" excludes behaviors, never the axis the unit is about.** A unit that cites a **Design source** owes `presentation:fidelity` ([../presentation/axis.md](../presentation/axis.md)); listing that evidence as a non-goal declares the point of the work out of scope (named incident: fmm-express #521). A unit with no design source simply says `none`.
|
|
42
|
-
|
|
43
|
-
## Acceptance Criteria
|
|
44
|
-
- AC1: <actor/system does ... under condition ... with expected observable result>
|
|
45
|
-
- AC2: <...>
|
|
46
|
-
|
|
47
|
-
## Critical User/System Journeys
|
|
48
|
-
- J1: <actor/role starts from ... does ... and reaches ... with expected side effects>
|
|
49
|
-
- J2: <...>
|
|
50
|
-
|
|
51
|
-
## Test Matrix
|
|
52
|
-
| Scenario | AC/Journey | Test Level | Mock/Fake Policy | Setup/Input | Assertions | Proving Test |
|
|
53
|
-
|---|---|---|---|---|---|---|
|
|
54
|
-
| Happy path | AC1 / J1 | Integration/API | Real DB + fake external provider | ... | ... | test name |
|
|
55
|
-
| Invalid input | AC2 | Unit + API | Mock provider only to assert it is not called | ... | ... | test name |
|
|
56
|
-
|
|
57
|
-
## Mock And Integration Policy
|
|
58
|
-
- Mock acceptable: <pure decision logic, provider failures, asserting side effect is not called>
|
|
59
|
-
- Integration required: <DB transactions, idempotency, concurrency, event/outbox, auth/tenant isolation, schema compatibility>
|
|
60
|
-
- External dependency strategy: <fake provider adapter, emulator, test container, staging smoke test>
|
|
61
|
-
- Mock-only exceptions: <AC/scenario + justification + reviewer approval; mocking a first-party internal seam is never an approvable exception — [../depth/mock-policy.md](../depth/mock-policy.md)>
|
|
62
|
-
|
|
63
|
-
## Required Automated Tests
|
|
64
|
-
- Unit: <specific logic and edge cases>
|
|
65
|
-
- Integration: <DB/broker/provider fakes/state transitions>
|
|
66
|
-
- Contract/API: <request/response/schema guarantees>
|
|
67
|
-
- API E2E: <endpoint behavior with state/side-effect assertions, if needed>
|
|
68
|
-
- UI E2E: <page route render-health/functional browser behavior, if needed>
|
|
69
|
-
- Journey E2E: <cross-page/core business flow with real APIs where practical, if needed>
|
|
70
|
-
|
|
71
|
-
## Coverage Mapping
|
|
72
|
-
| Requirement/Journey | ACs | Test Case/File | Status |
|
|
73
|
-
|---|---|---|---|
|
|
74
|
-
| <REQ-1 / J1> | AC1, AC2 | <path::test_name> | Required |
|
|
75
|
-
|
|
76
|
-
## Surface Census (Audit Mode Only — every surface, not just the gaps)
|
|
77
|
-
One row per inventoried surface so completeness is provable by construction: gaps = every row not `Covered`/`Excluded`. If the inventory can't be fully enumerated, say so and mark the rest `BLOCKED` — do not quietly scope to what was easy to see.
|
|
78
|
-
| Surface | kind/class | Required level | Status | Proving test (or — ) |
|
|
79
|
-
|---|---|---|---|---|
|
|
80
|
-
| /settings/team | route/mutation | browser-journey + presentation | Over-mocked | e2e/team.spec.ts (intercepts own API) |
|
|
81
|
-
| /settings/team/members | route/mutation | browser-journey + presentation | Missing | — (orphan, no test) |
|
|
82
|
-
| /api/team/invite | api/mutation | api + integration | Partial | tests/api/invite.test.ts |
|
|
83
|
-
| ... (every remaining surface) | ... | ... | ... | ... |
|
|
84
|
-
|
|
85
|
-
## Existing Coverage Assessment (Audit Mode Only)
|
|
86
|
-
- Production-readiness verdict: <Ship / No-ship>, with the specific user scenarios likely to generate bug reports in the first week and the evidence that would raise confidence. **State the driving counts** (uncovered/orphan surfaces, ledger warnings, skips, over-mock findings) — a no-ship names its numbers, including advisory-scan counts; it does not hide them behind "partial."
|
|
87
|
-
- Current coverage status: <Covered / Partial / Missing / Wrong Level / Over-mocked / Unverified / BLOCKED>
|
|
88
|
-
- Existing tests reviewed: <actual paths/test names/cases/assertions inspected; do not list docs-only claims as reviewed tests>
|
|
89
|
-
- Code paths reviewed: <paths/modules>
|
|
90
|
-
- Route/endpoint discovery: <method/command/source used, discovered count, manifest count, missing/stale/uncovered entries>
|
|
91
|
-
- Route reachability: <Reachable / Redirected / Guarded / Blocked / Unknown per relevant route, with layout/middleware/guard evidence>
|
|
92
|
-
- E2E journeys identified: <journey list or Not applicable>
|
|
93
|
-
- E2E journey coverage: <journey -> API/UI/Journey E2E evidence/status>
|
|
94
|
-
- Prior coverage docs/reports checked: <None or list; state which claims were verified against test code>
|
|
95
|
-
- Coverage gaps: <missing scenarios, weak assertions, over-mocking, wrong level, uncovered journeys, unverified claims, or None>
|
|
96
|
-
- Follow-up test criteria items: <specific tests to add/update, with AC/journey mapping>
|
|
97
|
-
|
|
98
|
-
## Test Skipping
|
|
99
|
-
- Skipped tests: <None or list with allowed category, evidence/blocker link, and replacement verification>
|
|
100
|
-
|
|
101
|
-
## Behavior-First Validation
|
|
102
|
-
- ACs describe externally observable behavior, not private functions.
|
|
103
|
-
- Test matrix scenarios validate outcomes, durable state, contracts, side effects, permissions, and failure behavior.
|
|
104
|
-
- Unit tests are scoped to public behavior, pure decision logic, or stable public contracts.
|
|
105
|
-
- Mock-only coverage does not bypass the core behavior under test.
|
|
106
|
-
- Existing coverage claims were verified against actual test code, not trusted from prior docs/reports.
|
|
107
|
-
- Route discovery and reachability analysis were performed before E2E audit, if applicable.
|
|
108
|
-
- Production-readiness verdict is stated as an explicit ship/no-ship call with named first-week bug-report scenarios, not a vibe.
|
|
109
|
-
- Status: PASS / FAIL / BLOCKED
|
|
110
|
-
- Required revisions before approval: <None or list>
|
|
111
|
-
|
|
112
|
-
## Reviewer Checklist
|
|
113
|
-
- Obligation impact stated: which obligations/surfaces this change touches, their `requirementRefs`, and the test that proves each.
|
|
114
|
-
- No `Orphan surface` introduced: every changed/new surface is referenced by at least one requirement and has a Coverage Ledger row.
|
|
115
|
-
- Every changed/new page with a write form has a mutation-form journey test that fills the real UI and asserts the backend accepts the submission; split-contract forms (fields and validator from different sources) are not signed off on seam tests alone.
|
|
116
|
-
- Every AC maps to an automated test or approved exception.
|
|
117
|
-
- Tests assert behavior/state, not implementation details only.
|
|
118
|
-
- No tests skipped/weakened to make CI pass; every (inherited) skip has an allowed category, evidence/blocker link, and replacement verification.
|
|
119
|
-
- Substitutes appear only at external trust boundaries; no internal seam (DB, service layer, internal endpoint, event bus) is mocked. Fakes/stubs keep failure-path/edge cases fast and deterministic and never call a live third party in CI; mocks/fakes don't bypass the core behavior.
|
|
120
|
-
- Mock-only coverage is rejected for stateful workflow, money movement, auth, idempotency, concurrency, event/outbox, or orchestrator recovery unless explicitly approved; money movement, auth, and idempotency can never be mock-only even with approval. Mocking a first-party internal seam is never approvable ([the mock policy](../depth/mock-policy.md)).
|
|
121
|
-
- Failure, retry, duplicate, permission, and concurrency cases are covered where relevant.
|
|
122
|
-
- Each proving test ran green (exit 0). No logs, screenshots, traces, or read-backs attached on top.
|
|
123
|
-
|
|
124
|
-
## Blocking Decisions (need a human answer to finalize the criteria)
|
|
125
|
-
Each entry is a product/scope decision you could NOT make yourself — phrase it so the reader knows exactly what to decide and why it matters. These are distinct from follow-up actions (which are just work to schedule); a blocking decision changes the criteria/verdict depending on the answer.
|
|
126
|
-
- Decision: <the choice to make> · Why it blocks: <which AC/scope/verdict is undetermined until this is answered> · If A → <consequence for the criteria/tests> · If B → <consequence>. Default if no answer: <the assumption you'll proceed on, marked `Assumption`>.
|
|
6
|
+
## Outcome and scope
|
|
7
|
+
- Source, agreed behavior, material risks, assumptions, and out-of-scope work.
|
|
8
|
+
- Product or architecture decisions needing their owners, if any.
|
|
9
|
+
|
|
10
|
+
## Agreed acceptance criteria and proof
|
|
11
|
+
- Observable criteria agreed before implementation.
|
|
12
|
+
- Existing tests or checks that suffice; focused additional proof if needed.
|
|
13
|
+
- Required repository checks and applicable security or data-integrity protections.
|
|
14
|
+
|
|
15
|
+
## Assessment (when auditing)
|
|
16
|
+
- Evidence inspected and execution results; what each proves and does not prove.
|
|
17
|
+
- Coverage grades for the agreed scope, with concrete gaps and unavailable evidence.
|
|
18
|
+
- Required fixes or decisions, separate from speculative follow-ups.
|
|
19
|
+
- What was verified and what was not.
|
|
127
20
|
```
|
|
128
21
|
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
After drafting, validate and revise before saving. Mark `BLOCKED` only when the source lacks enough information/access to define observable behavior. Reject criteria or audit claims that rely on:
|
|
22
|
+
Use a surface census only for an explicitly requested broad audit; state its boundaries and any incomplete inventory. Do not invent a ship/no-ship verdict where the evidence or an owner decision is missing. If a verdict is requested, explain its concrete basis; do not make an untested hypothetical state an automatic blocker. A newly found concrete defect may require a fix without silently changing the agreed criteria.
|
|
132
23
|
|
|
133
|
-
|
|
134
|
-
- Mock-heavy plans that never exercise state changes, API contracts, permissions, events, or side effects; or substitutes standing in for internal seams (DB, service layer, internal endpoints, event bus) instead of external trust boundaries.
|
|
135
|
-
- Requirement coverage mapped only to private-method/unit tests when integration, contract, or E2E is required.
|
|
136
|
-
- Coverage classified `Covered` from prior docs/PR summaries/audit reports/issue comments/CI/agent claims without inspecting actual test code/assertions.
|
|
137
|
-
- E2E audits that skipped deterministic route/endpoint discovery or reachability analysis.
|
|
138
|
-
- Audits driven only test-outward (mock census, existing-test mapping) without requirement-inward reconciliation, so `Orphan surface` gaps stay invisible.
|
|
139
|
-
- Reachable pages with write forms treated as covered by route/API/validator tests, without a mutation-form journey that fills the rendered UI and asserts the backend accepts the submission (split-contract form gap).
|
|
24
|
+
Prefer observable behavior to implementation-shaped assertions. Choose relevant failure, authorization, tenant-isolation, money, or durable-state cases based on material risk and existing policy. Preserve required repository checks and adopted floors. For design work, compare against the agreed design where needed; a cited source does not automatically require a separate presentation campaign. Inspect tests before saying they prove a behavior, and do not claim that an unavailable or failed check passed.
|
|
@@ -4,15 +4,15 @@
|
|
|
4
4
|
|
|
5
5
|
## Test Level Selection
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+
Choose sufficient proof for each agreed AC and material risk, reusing existing evidence. The examples below are options, not automatic extra layers; existing required checks and adopted floors remain binding:
|
|
8
8
|
|
|
9
9
|
- Logic AC: unit test is acceptable for pure decision logic or stable public contracts.
|
|
10
|
-
- Boundary AC: contract/API test
|
|
11
|
-
- Stateful workflow AC: integration test
|
|
12
|
-
- Async/pub-sub/orchestrator AC: integration test
|
|
10
|
+
- Boundary AC: consider a contract/API test for request/response, validation, schema, compatibility, and error contracts.
|
|
11
|
+
- Stateful workflow AC: consider an integration test for DB writes, transactions, state transitions, idempotency, and recovery.
|
|
12
|
+
- Async/pub-sub/orchestrator AC: consider an integration test with real state store plus broker emulator or fake event adapter; assert final durable state and emitted events.
|
|
13
13
|
- Critical money/security AC: integration, API E2E, or Journey E2E is required; mock-only coverage is never acceptable here, even with approval.
|
|
14
|
-
- Browser/UI AC: UI E2E
|
|
15
|
-
- Cross-page/core business journey: Journey E2E
|
|
14
|
+
- Browser/UI AC: consider UI E2E for page behavior, navigation, user-visible errors, forms, and role-dependent visibility.
|
|
15
|
+
- Cross-page/core business journey: consider Journey E2E with real APIs where practical; do not mock internal endpoints unless explicitly approved.
|
|
16
16
|
|
|
17
17
|
## E2E Coverage Rules
|
|
18
18
|
|
|
@@ -31,12 +31,7 @@ Smoke is not a separate browser-test level. A browser smoke path is a **curated
|
|
|
31
31
|
|
|
32
32
|
You can use the generated surface inventory to suggest candidates (mutation routes, auth/payments-looking routes, health endpoints), but do not let code discovery decide the golden-path manifest. Product/release judgment must promote candidates into `golden-paths.json` and write concrete observables.
|
|
33
33
|
|
|
34
|
-
For audit mode
|
|
35
|
-
|
|
36
|
-
- Run deterministic route/endpoint discovery before coverage judgment.
|
|
37
|
-
- Include discovery method, discovered count, manifest count, missing/stale/uncovered entries, and changed-route coverage status.
|
|
38
|
-
- Build the user journey inventory before mapping tests; route render-health tests do not prove a business journey unless they exercise the full flow and assert final outcomes.
|
|
39
|
-
- Decompose every reachable dashboard/settings page into its mutation-form journeys (e.g. "operator creates a GitHub connector account"). For each, require UI/Journey E2E proof that the form renders every backend-required field and that submitting it produces a payload the backend accepts plus the expected durable state change. Reject backend-validator, schema-source, and API-forwarding tests as a substitute — they are proxy wiring for a split-contract form.
|
|
34
|
+
For audit mode, inspect the routes and journeys relevant to the agreed scope. A render-health test does not by itself prove a business journey; a backend test alone does not prove a form submits correctly. Use broader discovery only for an explicitly requested broad audit, and do not require a separate browser journey for every form by default.
|
|
40
35
|
|
|
41
36
|
## Test Skipping Policy (no-skip)
|
|
42
37
|
|
|
@@ -1,8 +1,9 @@
|
|
|
1
1
|
# ensure-coverage scripts
|
|
2
2
|
|
|
3
|
-
|
|
4
|
-
|
|
5
|
-
|
|
3
|
+
This is the CLI reference for optional coverage tools. `SKILL.md` and `references/`
|
|
4
|
+
define when a tool is relevant; the checks below describe its unchanged mechanics,
|
|
5
|
+
not new acceptance conditions. Run a tool when requested or required by the repository.
|
|
6
|
+
Preserve adopted gates, but do not turn an optional scan into a feature blocker.
|
|
6
7
|
|
|
7
8
|
**Resolution.** Every script is under this skill's own directory, not the target
|
|
8
9
|
repo. Set `SKILL_DIR` to the directory you read `SKILL.md` from and run
|
|
@@ -45,7 +46,14 @@ Static test-lint gate engine. Deterministic, near-zero false positives, blocks o
|
|
|
45
46
|
| `no-swallowed-assertion` | an assertion whose rejection is eaten by a chained `.catch(...)` |
|
|
46
47
|
| `mock-internal-seam` | a `vi.mock` / `jest.mock` / `vitest.mock` whose target is first-party code |
|
|
47
48
|
|
|
48
|
-
`mock-internal-seam`
|
|
49
|
+
The scanner mechanically treats `mock-internal-seam` as default-on and zero-tolerance:
|
|
50
|
+
`@mock-ok` and inherited code do not suppress its exit status. Whole-tree scanning
|
|
51
|
+
belongs to an explicitly requested broad audit or an existing repo requirement;
|
|
52
|
+
a scoped task uses scoped checks when needed. Assess optional hits against the
|
|
53
|
+
behavior claimed; the classifier alone cannot determine test adequacy. Do not
|
|
54
|
+
require remediation merely because an optional scan flags a useful isolated unit
|
|
55
|
+
test. An adopted required gate remains binding. Correct topology declarations
|
|
56
|
+
when the scanner misclassifies an external boundary:
|
|
49
57
|
|
|
50
58
|
```json
|
|
51
59
|
// coverage.config.json at the repo root — declares what counts as first-party
|
|
@@ -3,7 +3,7 @@ name: golden-path
|
|
|
3
3
|
description: "Golden-path E2E: the short list of user journeys whose failure blocks a release, kept in a repo as one contract (golden-paths.json) with a generated GOLDEN_PATHS.md and a generated GitHub Actions workflow, and run against a local build or a deployed URL for an ALIVE/DEAD verdict. Use to set golden paths up in a repo ('init golden paths', 'add golden-path CI'), to migrate an older smoke.manifest.json or a hand-built smoke setup ('audit', 'migrate smoke'), to add, change, or remove a golden path, to ask 'is the build alive?' or run them before a PR or release, or when a change touches a flow listed in GOLDEN_PATHS.md. NOT for writing the Playwright specs themselves (e2e-test), deciding what a change must prove (ensure-coverage), or general workflow YAML (github-actions)."
|
|
4
4
|
dependencies:
|
|
5
5
|
- e2e-test
|
|
6
|
-
release-version: 0.
|
|
6
|
+
release-version: 0.45.0
|
|
7
7
|
---
|
|
8
8
|
**Updates:** Once/session run `node <skill-base-dir>/release-check.mjs`.
|
|
9
9
|
# Golden path
|