@xenos1996/usat 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (129) hide show
  1. package/LICENSE +21 -0
  2. package/README.md +335 -0
  3. package/USAT.md +596 -0
  4. package/dist/cli.d.ts +3 -0
  5. package/dist/cli.d.ts.map +1 -0
  6. package/dist/cli.js +438 -0
  7. package/dist/cli.js.map +1 -0
  8. package/dist/config.d.ts +7 -0
  9. package/dist/config.d.ts.map +1 -0
  10. package/dist/config.js +72 -0
  11. package/dist/config.js.map +1 -0
  12. package/dist/detect/index.d.ts +58 -0
  13. package/dist/detect/index.d.ts.map +1 -0
  14. package/dist/detect/index.js +329 -0
  15. package/dist/detect/index.js.map +1 -0
  16. package/dist/engine/audit.d.ts +21 -0
  17. package/dist/engine/audit.d.ts.map +1 -0
  18. package/dist/engine/audit.js +295 -0
  19. package/dist/engine/audit.js.map +1 -0
  20. package/dist/engine/diff.d.ts +24 -0
  21. package/dist/engine/diff.d.ts.map +1 -0
  22. package/dist/engine/diff.js +136 -0
  23. package/dist/engine/diff.js.map +1 -0
  24. package/dist/engine/evaluate.d.ts +19 -0
  25. package/dist/engine/evaluate.d.ts.map +1 -0
  26. package/dist/engine/evaluate.js +377 -0
  27. package/dist/engine/evaluate.js.map +1 -0
  28. package/dist/engine/gate.d.ts +19 -0
  29. package/dist/engine/gate.d.ts.map +1 -0
  30. package/dist/engine/gate.js +47 -0
  31. package/dist/engine/gate.js.map +1 -0
  32. package/dist/engine/loader.d.ts +22 -0
  33. package/dist/engine/loader.d.ts.map +1 -0
  34. package/dist/engine/loader.js +360 -0
  35. package/dist/engine/loader.js.map +1 -0
  36. package/dist/engine/maturity.d.ts +20 -0
  37. package/dist/engine/maturity.d.ts.map +1 -0
  38. package/dist/engine/maturity.js +157 -0
  39. package/dist/engine/maturity.js.map +1 -0
  40. package/dist/engine/score.d.ts +19 -0
  41. package/dist/engine/score.d.ts.map +1 -0
  42. package/dist/engine/score.js +141 -0
  43. package/dist/engine/score.js.map +1 -0
  44. package/dist/engine/sections.d.ts +14 -0
  45. package/dist/engine/sections.d.ts.map +1 -0
  46. package/dist/engine/sections.js +50 -0
  47. package/dist/engine/sections.js.map +1 -0
  48. package/dist/index.d.ts +23 -0
  49. package/dist/index.d.ts.map +1 -0
  50. package/dist/index.js +17 -0
  51. package/dist/index.js.map +1 -0
  52. package/dist/report/markdown.d.ts +16 -0
  53. package/dist/report/markdown.d.ts.map +1 -0
  54. package/dist/report/markdown.js +488 -0
  55. package/dist/report/markdown.js.map +1 -0
  56. package/dist/types.d.ts +282 -0
  57. package/dist/types.d.ts.map +1 -0
  58. package/dist/types.js +16 -0
  59. package/dist/types.js.map +1 -0
  60. package/dist/util/glob.d.ts +11 -0
  61. package/dist/util/glob.d.ts.map +1 -0
  62. package/dist/util/glob.js +86 -0
  63. package/dist/util/glob.js.map +1 -0
  64. package/dist/util/project.d.ts +72 -0
  65. package/dist/util/project.d.ts.map +1 -0
  66. package/dist/util/project.js +586 -0
  67. package/dist/util/project.js.map +1 -0
  68. package/dist/util/yaml.d.ts +28 -0
  69. package/dist/util/yaml.d.ts.map +1 -0
  70. package/dist/util/yaml.js +48 -0
  71. package/dist/util/yaml.js.map +1 -0
  72. package/docs/ARCHITECTURE.md +119 -0
  73. package/docs/README.md +27 -0
  74. package/docs/adr/0001-record-architecture-decisions.md +35 -0
  75. package/docs/adr/0002-typescript-engine-over-python.md +49 -0
  76. package/docs/adr/0003-rules-are-data-not-code.md +50 -0
  77. package/docs/adr/0004-markdown-only-output.md +41 -0
  78. package/docs/adr/0005-maturity-dampens-severity.md +56 -0
  79. package/docs/adr/0006-severity-status-two-axes.md +48 -0
  80. package/docs/adr/0007-suppressions-are-visible-and-expiring.md +46 -0
  81. package/docs/adr/0008-report-trailer-and-diff.md +48 -0
  82. package/docs/adr/0009-fail-closed-on-malformed-input.md +56 -0
  83. package/docs/adr/0010-detector-design.md +54 -0
  84. package/docs/adr/0011-coexist-with-deep-scanners.md +52 -0
  85. package/docs/adr/README.md +32 -0
  86. package/docs/agent-integration.md +129 -0
  87. package/docs/ci-integration.md +163 -0
  88. package/docs/concepts.md +196 -0
  89. package/docs/configuration.md +105 -0
  90. package/docs/detectors.md +186 -0
  91. package/docs/getting-started.md +99 -0
  92. package/docs/maturity-profiles.md +156 -0
  93. package/docs/reference/api.md +193 -0
  94. package/docs/reference/cli.md +412 -0
  95. package/docs/rule-packs.md +278 -0
  96. package/docs/standards-mapping.md +171 -0
  97. package/package.json +96 -0
  98. package/rules/core/architecture.yaml +190 -0
  99. package/rules/core/cicd.yaml +212 -0
  100. package/rules/core/code-quality.yaml +261 -0
  101. package/rules/core/dependencies.yaml +132 -0
  102. package/rules/core/documentation.yaml +119 -0
  103. package/rules/core/future-readiness.yaml +91 -0
  104. package/rules/core/release.yaml +131 -0
  105. package/rules/core/repo.yaml +303 -0
  106. package/rules/core/security.yaml +494 -0
  107. package/rules/core/supply-chain.yaml +357 -0
  108. package/rules/core/testing.yaml +260 -0
  109. package/rules/detectors.yaml +2255 -0
  110. package/rules/index.yaml +41 -0
  111. package/rules/profiles/maturity.yaml +135 -0
  112. package/rules/stacks/ai-era.yaml +193 -0
  113. package/rules/stacks/api-backend.yaml +121 -0
  114. package/rules/stacks/cli.yaml +105 -0
  115. package/rules/stacks/compliance.yaml +157 -0
  116. package/rules/stacks/containers.yaml +124 -0
  117. package/rules/stacks/data.yaml +165 -0
  118. package/rules/stacks/go.yaml +134 -0
  119. package/rules/stacks/iac.yaml +128 -0
  120. package/rules/stacks/jvm.yaml +150 -0
  121. package/rules/stacks/ml-ai.yaml +140 -0
  122. package/rules/stacks/mobile.yaml +121 -0
  123. package/rules/stacks/node-typescript.yaml +166 -0
  124. package/rules/stacks/python.yaml +175 -0
  125. package/rules/stacks/rust.yaml +129 -0
  126. package/rules/stacks/solidity.yaml +139 -0
  127. package/rules/stacks/web-frontend.yaml +159 -0
  128. package/templates/AGENTS.audit.md +135 -0
  129. package/templates/AUDIT_REPORT.md +155 -0
@@ -0,0 +1,54 @@
1
+ # 10. Detectors are cheap syntactic signals resolved to a fixed point
2
+
3
+ - **Date:** 2026-09-09
4
+ - **Status:** Accepted
5
+
6
+ ## Context
7
+
8
+ Rule selection needs facts ("is this a Node project? does it have a
9
+ database?"), and facts must come from the tree itself with no network, no
10
+ build, and no language server. The temptation is a clever detector: parse
11
+ `pyproject.toml` properly, resolve `extends` chains in tsconfig, understand
12
+ TOML tables. The constraint is that detectors run over arbitrary repos in
13
+ milliseconds and must never crash the audit.
14
+
15
+ The opposing risk is over-firing: docs, tests, examples, and templates _talk
16
+ about_ technology without _using_ it. A detector that counts the word
17
+ "postgres" in `docs/` reports a database the project does not have.
18
+
19
+ ## Decision
20
+
21
+ 1. **Syntactic signals only.** Detectors match file names, directory names,
22
+ manifest keys, and single-line content patterns. No parsing beyond
23
+ section-scoped text search and JSON path lookup.
24
+ 2. **Prose is excluded from content signals.** Markdown, docs, examples,
25
+ templates, tests, fixtures, and mocks never count as technology evidence
26
+ (`CONTENT_EXCLUDES`). A deliberately conservative list, extended only
27
+ with evidence of a miss — `__tests__/`, `spec/`, and `*.stories.*` were
28
+ added after review showed Jest and Storybook layouts leaking through.
29
+ 3. **Manifest queries are section-scoped where sections exist.** A dotted
30
+ key (`tool.poetry.dependencies`) must match a full header segment
31
+ (case-insensitive; `[dev-dependencies]` is not `[dependencies]`), the
32
+ leaf must match on word boundaries (`test` is not `latest`), and
33
+ `contains` is checked against the matched block — never the whole file.
34
+ Flat files with no headers keep whole-text search (YAML has no sections
35
+ to scope to).
36
+ 4. **`implies` chains resolve to a fixed point**, bounded by
37
+ `detectors.length + 1` passes with early exit. Two hardcoded passes
38
+ silently dropped chains of length 3+ declared worst-first, inflating
39
+ scores — the bound is now structural, not a magic number.
40
+
41
+ ## Consequences
42
+
43
+ **Good:** detection is fast, total (no network), crash-free, and order-
44
+ independent. The precision rules above each cite the false positive they
45
+ killed, so future relaxations know what they are re-introducing.
46
+
47
+ **Bad:** syntactic detection has a ceiling: `extends` chains leaving the
48
+ repo, aliased imports, and commented-out config are invisible. Detectors
49
+ report evidence, not proof — rules that need proof belong in the judgement
50
+ queue, not in a cleverer regex.
51
+
52
+ **Neutral:** the fixed-point bound is `detectors.length + 1` passes worst
53
+ case; typical runs exit after two. Performance is unchanged for existing
54
+ registries.
@@ -0,0 +1,52 @@
1
+ # 11. Coexist with deep scanners: verify configuration, consume evidence
2
+
3
+ - **Date:** 2026-09-09
4
+ - **Status:** Accepted
5
+
6
+ ## Context
7
+
8
+ A regex engine cannot do dataflow analysis (CodeQL), reachability-filtered
9
+ CVE prioritisation (Snyk, Semgrep Supply Chain), whole-program taint
10
+ tracking, or API-verified hygiene (Scorecard branch protection, SLSA
11
+ attestation verification). Pretending otherwise would mean shipping weak
12
+ copies of those tools and claiming parity — the fastest way to destroy an
13
+ auditor's credibility.
14
+
15
+ But USAT's value proposition includes supply chain, vulnerability posture,
16
+ and hygiene verdicts. Ignoring those dimensions leaves the report with
17
+ holes exactly where executives look first.
18
+
19
+ ## Decision
20
+
21
+ USAT **verifies configuration and consumes evidence; it does not duplicate
22
+ analysis**:
23
+
24
+ 1. Where a deep scanner exists, the rule checks that it is **configured and
25
+ running** (lockfile + frozen install, scanner in CI, SBOM published,
26
+ attestations present) — never that USAT re-derives its findings.
27
+ 2. Where a scanner emits machine output, future check kinds may **ingest**
28
+ it (SARIF/JSON: coverage numbers, HIGH/CRITICAL CVE counts, attestation
29
+ verification success) as evidence for the corresponding rules.
30
+ 3. USAT never claims what an oracle would contradict: API-verifiable checks
31
+ (branch protection, token permissions, signed releases) are file-text
32
+ approximations, documented as such, and the report prefers MISSING
33
+ ("not verified") over a confident verdict it cannot support.
34
+ 4. SARIF (and JSON) renderers are the approved interchange in both
35
+ directions: USAT findings out to dashboards and code scanning, oracle
36
+ findings in as evidence. The internal finding model already carries
37
+ file/line/severity/rule identity, so this is a renderer, not a redesign.
38
+
39
+ ## Consequences
40
+
41
+ **Good:** no vulnerability database to maintain, no resolver to build, no
42
+ dataflow engine to write — and no weak-duplicate claims. Each tool does
43
+ what it is good at; the audit composes them.
44
+
45
+ **Bad:** USAT reports are only as strong as the oracles the project runs.
46
+ A repo with no scanners gets MISSING verdicts where a bundled scanner
47
+ would give answers. That is honest but less satisfying, and must be
48
+ explained in every such report (the confidence column exists for this).
49
+
50
+ **Neutral:** this ADR draws the scope boundary that keeps USAT small. Any
51
+ proposal to add a CVE database, a resolver, or interprocedural analysis
52
+ re-opens this decision explicitly.
@@ -0,0 +1,32 @@
1
+ # Architecture Decision Records
2
+
3
+ Short, numbered, immutable records of the non-obvious choices in this
4
+ repository — see [ADR-0001](0001-record-architecture-decisions.md) for why
5
+ they exist and the rules for writing them (context, decision, consequences;
6
+ a changed decision gets a _new_ ADR that supersedes the old one).
7
+
8
+ ## Index
9
+
10
+ | # | Title | Status | Supersedes |
11
+ | ---- | --------------------------------------------------- | -------- | ---------- |
12
+ | 0001 | Record architecture decisions | Accepted | — |
13
+ | 0002 | Build the engine in TypeScript, not Python | Accepted | — |
14
+ | 0003 | Rule packs are YAML data, not TypeScript code | Accepted | — |
15
+ | 0004 | Markdown is the only output format | Accepted | — |
16
+ | 0005 | Lifecycle stage dampens severity; CRITICAL exempt | Accepted | — |
17
+ | 0006 | Severity/status two axes; WRONG outranks MISSING | Accepted | — |
18
+ | 0007 | Accepted risk stays visible, stops scoring, expires | Accepted | — |
19
+ | 0008 | Machine-readable trailer; audits are diffable | Accepted | — |
20
+ | 0009 | Malformed input fails closed and loudly | Accepted | — |
21
+ | 0010 | Detectors are cheap signals, fixed-point resolved | Accepted | — |
22
+ | 0011 | Coexist with deep scanners; consume evidence | Accepted | — |
23
+
24
+ ## Coverage map (what has no ADR, and why)
25
+
26
+ - **Scoring maths** — specified normatively in `docs/concepts.md`, not an
27
+ ADR: it is a formula with tests, not a judgement call. Disagreeing with it
28
+ means changing code, and the tests adjudicate.
29
+ - **Individual rule wordings** — rules are data (ADR-0003). Disputed rules
30
+ get fixed in `rules/` with a regression test, not an ADR.
31
+ - **Release mechanics** — tracked in `ROADMAP.md` until decided, then
32
+ recorded here (e.g. the eventual release-automation choice).
@@ -0,0 +1,129 @@
1
+ # Agent integration
2
+
3
+ USAT is built to be driven by an AI agent. The division of labour is explicit:
4
+
5
+ | | Who | Why |
6
+ | -------------------- | ------------- | ------------------------------------------------------------------------------- |
7
+ | Deterministic checks | **the tool** | An LLM grepping 400 files is slow, expensive, and non-deterministic |
8
+ | Judgement checks | **the agent** | Coupling, authorisation, threat modelling — nothing settles these but reasoning |
9
+ | The report | **both** | Tool renders it; agent fills the judgement queue |
10
+
11
+ The tool settles roughly 70% of applicable rules on a typical project. The agent's
12
+ job is the remaining 30% — the interesting 30%.
13
+
14
+ ## The workflow
15
+
16
+ ```bash
17
+ usat audit . --out AUDIT.md # 1. deterministic pass
18
+ ```
19
+
20
+ Then, to the agent:
21
+
22
+ > Read `USAT.md` and `AUDIT.md`.
23
+ > Work the **Judgement Queue** at the bottom of the report.
24
+ > For each item, find the evidence and record `file:line` plus one sentence of
25
+ > reasoning. Do not mark anything ✅ without evidence (Rule 4).
26
+ > Then rewrite the report with your findings folded in.
27
+
28
+ ## Why this is better than "audit this codebase"
29
+
30
+ A bare prompt produces three failure modes, and the template exists to prevent all
31
+ three:
32
+
33
+ | Failure | How USAT prevents it |
34
+ | ------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
35
+ | **Confident invention** — "✅ MFA is implemented" with no such code | Rule 4: no ✅ without evidence; every judgement item states what proof looks like |
36
+ | **Wrong scope** — Solidity reentrancy checks on a Next.js blog | Detection + `applies_when`; non-applicable sections are skipped silently (Rule 1) |
37
+ | **Wrong severity** — a prototype graded like a bank, or vice versa | Maturity profiles, with a hard floor on CRITICAL |
38
+
39
+ ## Shipped packs
40
+
41
+ ### Agent Skills (`SKILL.md`)
42
+
43
+ ```bash
44
+ cp -r skills/usat-audit ~/.claude/skills/ # Claude Code
45
+ cp -r skills/usat-audit .cursor/skills/ # Cursor
46
+ ```
47
+
48
+ Follows the Agent Skills convention — YAML frontmatter with `name` and `description`,
49
+ detailed content in `references/` [1](https://github.com/addyosmani/agent-skills/blob/main/CLAUDE.md).
50
+ The same directory works across Claude Code, Cursor, Codex, Gemini CLI, Copilot, and
51
+ ~60 other tools.
52
+
53
+ ### `AGENTS.md` for a target repository
54
+
55
+ ```bash
56
+ usat init .
57
+ cp templates/AGENTS.audit.md AGENTS.md
58
+ ```
59
+
60
+ Gives any agent working in that repo the build/test/lint commands, the conventions,
61
+ and an explicit denylist — which is worth doing on its own. Agent instruction files
62
+ are executable-ish: they commit as easily as code and agents follow them literally
63
+ (see `AI-011`).
64
+
65
+ ## Prompt you can paste
66
+
67
+ ```
68
+ You are auditing this repository with USAT (Universal Software Audit Template).
69
+
70
+ 1. Run: usat detect .
71
+ Confirm the detected facts. If any are wrong, say so and note the correct ones.
72
+
73
+ 2. Run: usat audit . --out AUDIT.md
74
+
75
+ 3. Read USAT.md (the framework) and AUDIT.md (the deterministic results).
76
+
77
+ 4. Work the Judgement Queue in AUDIT.md. For each item:
78
+ - locate the relevant code
79
+ - decide PASS / WRONG / MISSING
80
+ - record file:line evidence and one sentence of reasoning
81
+ - if you cannot determine it, say UNKNOWN and say what you would need
82
+
83
+ 5. Produce a final report that follows the Section 14 template in USAT.md:
84
+ - CRITICAL and HIGH findings first, with location and fix
85
+ - then section-by-section
86
+ - then the roadmap
87
+ - then the judgement queue with your recorded evidence
88
+
89
+ Rules:
90
+ - Never mark ✅ without evidence (file:line or command output).
91
+ - Skip sections that do not apply — do not explain why.
92
+ - ⚠️ WRONG is worse than 🚫 MISSING. Surface it.
93
+ - Priority: security > correctness > maintainability > style.
94
+ ```
95
+
96
+ ## Using the API directly
97
+
98
+ ```ts
99
+ import { runAudit, renderMarkdown, loadProfiles } from 'usat';
100
+
101
+ const { report, profile, warnings } = runAudit({
102
+ target: process.cwd(),
103
+ rulesDir: 'rules',
104
+ depth: 'deep',
105
+ profile: 'auto',
106
+ config: { version: 1 },
107
+ allowCommands: false,
108
+ usatVersion: '1.0.0',
109
+ });
110
+
111
+ console.log(report.score.overall); // 71.4
112
+ console.log(report.score.counts.UNKNOWN); // 18 → send these to the agent
113
+ const md = renderMarkdown(report, profile);
114
+ ```
115
+
116
+ `report.findings.filter(f => f.status === 'UNKNOWN')` is precisely the agent's task
117
+ list, and each item carries `why` and `evidenceHint`.
118
+
119
+ ## Notes for agent authors
120
+
121
+ - **Do not re-run greps the tool already did.** If a finding is already ✅ or 🚫 with
122
+ locations, the tool settled it.
123
+ - **Trust `suppressedReason`.** A suppressed finding was a decision someone made. Mention
124
+ it only if the reason has clearly expired.
125
+ - **Respect the maturity profile.** If a finding is marked
126
+ _"Downgraded HIGH → MEDIUM by the MVP profile"_, do not re-escalate it — unless it is
127
+ CRITICAL, which is never downgraded.
128
+ - **Say UNKNOWN when you do not know.** It is a valid answer and it keeps confidence
129
+ honest. Guessing is the failure mode this whole framework exists to prevent.
@@ -0,0 +1,163 @@
1
+ # CI integration
2
+
3
+ ## GitHub Actions
4
+
5
+ `usat init` writes a ready-made workflow. The minimum viable version:
6
+
7
+ ```yaml
8
+ name: USAT Audit
9
+ on: [pull_request]
10
+
11
+ permissions:
12
+ contents: read
13
+ pull-requests: write
14
+
15
+ jobs:
16
+ audit:
17
+ runs-on: ubuntu-latest
18
+ steps:
19
+ - uses: actions/checkout@v4
20
+ - uses: actions/setup-node@v4
21
+ with:
22
+ node-version: '20'
23
+
24
+ - name: Run USAT
25
+ id: usat
26
+ run: npx --yes @xenos1996/usat@1 audit . --depth standard --out AUDIT.md
27
+
28
+ - name: Publish to job summary
29
+ if: always()
30
+ run: cat AUDIT.md >> "$GITHUB_STEP_SUMMARY"
31
+
32
+ - name: Upload report
33
+ if: always()
34
+ uses: actions/upload-artifact@v4
35
+ with:
36
+ name: usat-audit
37
+ path: AUDIT.md
38
+ ```
39
+
40
+ ### Quality gate
41
+
42
+ ```yaml
43
+ - name: Quality gate
44
+ run: npx --yes @xenos1996/usat@1 audit . --fail-on high
45
+ ```
46
+
47
+ | Exit | Meaning |
48
+ | ---- | ----------------------------------------- |
49
+ | `0` | No findings at or above the threshold |
50
+ | `1` | Gate tripped — findings printed to stderr |
51
+ | `2` | Usage or configuration error |
52
+
53
+ **Roll-out advice:** start with `--fail-on critical`. Move to `high` once the backlog
54
+ is clear. Never start at `medium` — you will teach the team to bypass the check.
55
+
56
+ ### Comment the score on the PR
57
+
58
+ ```yaml
59
+ - name: Comment
60
+ if: github.event_name == 'pull_request'
61
+ uses: marocchino/sticky-pull-request-comment@v2
62
+ with:
63
+ path: AUDIT.md
64
+ ```
65
+
66
+ ### Composite action
67
+
68
+ ```yaml
69
+ - uses: Er-Sajan-PLG/software-auditing-template@v1
70
+ with:
71
+ depth: standard
72
+ fail-on: high
73
+ ```
74
+
75
+ See [`action.yml`](../action.yml).
76
+
77
+ ---
78
+
79
+ ## GitLab CI
80
+
81
+ ```yaml
82
+ usat-audit:
83
+ image: node:20
84
+ stage: test
85
+ script:
86
+ - npx --yes @xenos1996/usat@1 audit . --out usat-report.md --fail-on critical
87
+ artifacts:
88
+ when: always
89
+ paths: [usat-report.md]
90
+ expose_as: 'USAT Audit'
91
+ ```
92
+
93
+ ---
94
+
95
+ ## Scheduled drift detection
96
+
97
+ The highest-value CI job is not the PR gate — it is a monthly audit that shows
98
+ movement:
99
+
100
+ ```yaml
101
+ on:
102
+ schedule:
103
+ - cron: '0 6 1 * *' # 1st of the month
104
+ workflow_dispatch:
105
+
106
+ jobs:
107
+ audit:
108
+ steps:
109
+ - uses: actions/checkout@v4
110
+ with: { fetch-depth: 0 }
111
+ - run: npx --yes @xenos1996/usat@1 audit . --out reports/$(date +%Y-%m).md
112
+ - run: |
113
+ PREV=$(ls reports/*.md | tail -2 | head -1)
114
+ npx --yes @xenos1996/usat@1 diff "$PREV" "reports/$(date +%Y-%m).md" --out DIFF.md || true
115
+ cat DIFF.md >> "$GITHUB_STEP_SUMMARY"
116
+ - uses: peter-evans/create-pull-request@v6
117
+ with:
118
+ title: 'chore: monthly USAT audit'
119
+ body-path: DIFF.md
120
+ ```
121
+
122
+ A dated report per month plus `usat diff` gives you an audit trail that shows
123
+ improvement — the thing a single audit can never do.
124
+
125
+ ---
126
+
127
+ ## Reference: this repo's own CI (`.github/workflows/`)
128
+
129
+ USAT audits itself with the full stack — copy what fits:
130
+
131
+ | Workflow | What it does |
132
+ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
133
+ | `ci.yml` | lint+format+typecheck · Vitest with coverage thresholds · build + CLI smoke · rule-pack validation · npm audit + gitleaks + license scan · **hygiene** (`check-adrs` + `check-docs`) |
134
+ | `self-audit.yml` | `usat audit . --depth deep --fail-on critical` on every PR, score as PR comment |
135
+ | `scorecard.yml` | OpenSSF Scorecard monthly + on push (API-verified hygiene; SARIF to Security tab) |
136
+ | `automerge.yml` | Dependabot patch/minor auto-merge once CI is green (majors stay manual) |
137
+ | `gitleaks-pin.yml` | Monthly check that the curl-pinned gitleaks binary in `ci.yml` is current (no bot watches it) — opens a deduped issue when stale |
138
+ | `release.yml` | Tag push `v*` → OIDC trusted publishing (no long-lived token) + `--provenance` + CycloneDX SBOM artifact |
139
+
140
+ Release setup note: trusted publishing needs a one-time owner step on
141
+ npmjs.com (package Settings → Trusted Publisher → this repo + workflow)
142
+ before the first OIDC publish succeeds.
143
+
144
+ ## Choosing a depth in CI
145
+
146
+ | Depth | Rules | Runtime (≈10k files) | Use for |
147
+ | ---------- | ---------------------------------------------- | -------------------- | -------------------- |
148
+ | `quick` | high-signal only | ~1–3 s | Every PR |
149
+ | `standard` | default | ~3–10 s | PRs + nightly |
150
+ | `deep` | adds cycles, duplication, complexity, mutation | ~10–30 s | Weekly / pre-release |
151
+
152
+ `deep` is where the judgement-heavy rules live; the extra cost is mostly I/O.
153
+
154
+ ## Notes
155
+
156
+ - **No network at audit time.** USAT reads files and writes Markdown. It never uploads
157
+ anything, which is why it is safe on private repositories.
158
+ - **`--allow-commands` in CI.** Only if you trust the target repo — it shells out for
159
+ checks like `npm audit`. Off by default; those rules report ❓ NEEDS REVIEW instead.
160
+ - **Pin the version** in production pipelines (`@xenos1996/usat@1`, not `@latest`) so a
161
+ rule-pack change cannot fail your build without a commit.
162
+ - **Commit `.usat.yaml`.** Suppressions and overrides without a commit are invisible
163
+ decisions, and they are the first thing a reviewer asks about.
@@ -0,0 +1,196 @@
1
+ # Concepts
2
+
3
+ ## Two axes: severity and status
4
+
5
+ Most audit checklists merge "how bad is it" and "what did we find" into one column.
6
+ That makes scoring impossible to reproduce — two auditors tag the same observation
7
+ differently, and the totals diverge. USAT keeps them apart.
8
+
9
+ ### Severity — a property of the rule
10
+
11
+ Set by whoever wrote the rule. It says: _if this is violated, how bad is that?_
12
+
13
+ | Tag | Severity | Meaning | Default weight |
14
+ | --- | ---------- | ------------------------------------------------- | -------------- |
15
+ | 🔴 | `CRITICAL` | Breaks security, data integrity, or core function | 10 |
16
+ | 🟠 | `HIGH` | Major risk or substantial debt | 6 |
17
+ | 🟡 | `MEDIUM` | Important gap | 3 |
18
+ | 🟢 | `LOW` | Minor improvement | 1.5 |
19
+ | 🔵 | `FUTURE` | Not needed now, needed at scale | 0.5 |
20
+
21
+ ### Status — a property of the observation
22
+
23
+ Produced by the engine (or by a reviewer, for judgement checks).
24
+
25
+ | Tag | Status | Score credit | When |
26
+ | --- | -------------- | ------------ | -------------------------------------------------- |
27
+ | ✅ | `GOOD` | 1.00 | Verified present and correct |
28
+ | 🧪 | `EXPERIMENTAL` | 0.50 | Present, unvalidated (`grep_experimental`) |
29
+ | 💀 | `DEPRECATED` | 0.40 | Present but EOL (`grep_deprecated`) |
30
+ | ⚠️ | `WRONG` | 0.15 | Present but implemented incorrectly (`grep_wrong`) |
31
+ | 🚫 | `MISSING` | 0.00 | Required and absent |
32
+ | ❓ | `NEEDS REVIEW` | _excluded_ | Judgement required; no evidence recorded |
33
+ | ➖ | `SKIPPED` | _excluded_ | Not applicable to this project |
34
+
35
+ **Why WRONG scores 0.15 and not 0.00.** Something exists, so there is partial
36
+ credit — the intent was right and the surface is smaller than a greenfield fix.
37
+ But a wrong implementation is _more dangerous_ than an absent one, because it looks
38
+ finished: nobody puts "add CSRF protection" on the roadmap when CSRF protection
39
+ appears to be there. Rule 9 of the agent behaviour rules exists for exactly this.
40
+
41
+ **Why NEEDS REVIEW is excluded rather than scored zero.** An unanswered question is
42
+ not a failure. It is a hole in the audit, and the place it shows up is
43
+ **confidence**, not the score.
44
+
45
+ ---
46
+
47
+ ## Scoring
48
+
49
+ ### Per rule
50
+
51
+ ```
52
+ creditᵢ = weightᵢ × CREDIT[statusᵢ]
53
+ ```
54
+
55
+ ### Per section
56
+
57
+ ```
58
+ sectionScore = 10 × Σ creditᵢ / Σ weightᵢ (over resolved, applicable rules)
59
+ ```
60
+
61
+ ### Overall
62
+
63
+ ```
64
+ overall = 100 × Σ (creditᵢ × sectionWeight) / Σ (weightᵢ × sectionWeight)
65
+ ```
66
+
67
+ Rule-level weighting, rather than averaging section scores, means a section with
68
+ three rules cannot swing the total as hard as one with thirty.
69
+
70
+ ### Section weights
71
+
72
+ Security, supply chain, and testing carry the heaviest multipliers — a repo that
73
+ fails an authorisation control is worse than one with a slightly untidy README, and
74
+ the number should say so.
75
+
76
+ | | Section | Weight |
77
+ | ------------------- | ------------------------------------------------- | ------- |
78
+ | **S2** | Security | **1.7** |
79
+ | **S3** | Supply Chain & Build Provenance | 1.3 |
80
+ | **S7** | Testing & Quality Assurance | 1.3 |
81
+ | **S10** | Dependencies & Third-Party | 1.2 |
82
+ | **S14** | AI / LLM-Era Risks | 1.1 |
83
+ | S4, S5, S6, S8, S15 | Architecture, Code Quality, Data, CI/CD, Platform | 1.0 |
84
+ | **S13** | Accessibility, i18n & Compliance | 0.9 |
85
+ | **S11** | Performance & Resilience | 0.8 |
86
+ | S1, S9, S12 | Repository, Release, Documentation | 0.7 |
87
+ | **S16** | Future Readiness | 0.4 |
88
+
89
+ Override any of them in `rules/sections.yaml`.
90
+
91
+ ### Confidence
92
+
93
+ ```
94
+ confidence = resolved / applicable
95
+ ```
96
+
97
+ `resolved` counts rules the engine could settle (PASS, FAIL, WRONG, MISSING,
98
+ DEPRECATED, EXPERIMENTAL). `applicable` counts everything `applies_when` let through.
99
+
100
+ - Sections with **zero** resolved rules are reported as _"— not verified"_ and
101
+ excluded from the overall score. You cannot earn points for questions nobody answered.
102
+ - A high score at low confidence is the single most misleading thing an audit tool
103
+ can produce, which is why confidence sits next to every score in the report.
104
+
105
+ ---
106
+
107
+ ## Maturity dampening
108
+
109
+ Severity is reduced by the detected lifecycle stage, per rule class — but
110
+ **CRITICAL is never dampened, at any stage.**
111
+
112
+ | Stage | security | supply-chain | correctness | maintainability | operations | performance | compliance | documentation | style |
113
+ | ---------- | -------- | ------------ | ----------- | --------------- | ---------- | ----------- | ---------- | ------------- | ----- |
114
+ | Prototype | −1 | −1 | −1 | −2 | −2 | −2 | −2 | −2 | −2 |
115
+ | MVP | −0 | −1 | −1 | −1 | −1 | −1 | −1 | −2 | −2 |
116
+ | Beta | −0 | −0 | −0 | −1 | −1 | −1 | −1 | −1 | −1 |
117
+ | Production | −0 | −0 | −0 | −0 | −0 | −0 | −0 | −0 | −0 |
118
+ | Legacy | −0 | −0 | −0 | −1 | −0 | −1 | −0 | −1 | −1 |
119
+
120
+ Steps move down the ladder `FUTURE < LOW < MEDIUM < HIGH < CRITICAL`.
121
+
122
+ Every dampened finding says so in the report:
123
+
124
+ ```
125
+ - 🚫 **LICENSE present** `REPO-005`
126
+ - 🪶 Downgraded HIGH → LOW by the Prototype / Spike profile
127
+ ```
128
+
129
+ so a prototype reading knows which items will climb on their own as it grows up.
130
+ See [maturity-profiles.md](maturity-profiles.md).
131
+
132
+ ---
133
+
134
+ ## Applicability
135
+
136
+ A rule participates when **all** of these hold:
137
+
138
+ 1. Its pack is not skipped (`skip_when` is false)
139
+ 2. `applies_when` evaluates true against the detected facts
140
+ 3. Its `depths` (if any) include the current `--depth`
141
+ 4. It is not disabled in `.usat.yaml`
142
+
143
+ Two escape hatches, both loud: a force-included pack (`include:`) applies
144
+ all its rules regardless of 2 and 3 — the operator asked for it — and a
145
+ malformed `applies_when` (unknown key or operator, bad regex, non-mapping)
146
+ warns at load time and evaluates to false, so a typo can never silently
147
+ include rules. See ADR-0009.
148
+
149
+ ```yaml
150
+ - id: DATA-007
151
+ applies_when: { fact: 'has:database' }
152
+ depths: [deep]
153
+ check: { kind: manual }
154
+ ```
155
+
156
+ Packs can contribute facts (`provides:`), and facts can make further packs apply.
157
+ The engine resolves this to a fixed point (max 3 passes) so a monorepo pack can pull
158
+ in boundary-enforcement rules without either pack knowing about the other.
159
+
160
+ ---
161
+
162
+ ## Suppressions
163
+
164
+ Anything suppressed needs a reason. That is not bureaucracy — it is the difference
165
+ between "we decided not to do this" and "we forgot".
166
+
167
+ ```yaml
168
+ suppressions:
169
+ - rule: PERF-005
170
+ reason: 'Known N+1 in the admin panel; 40 rows max.'
171
+ until: '2026-12-31'
172
+ ```
173
+
174
+ Suppressed findings are **excluded from the score, the severity tallies, and
175
+ the action sections, but still listed** under _Accepted Risk_. Anyone reading
176
+ the report can see what was waived and why — and the Findings Summary can
177
+ never contradict Immediate Action over a waived item.
178
+
179
+ Waivers **expire**: an `until` date in the past (or one that cannot be
180
+ parsed) is ignored with a warning, and the finding reports normally. See
181
+ ADR-0007.
182
+
183
+ ---
184
+
185
+ ## Reproducibility
186
+
187
+ The same tree, rules, depth, and profile always produce the same score. The only
188
+ inputs that can change a result between runs:
189
+
190
+ - file contents (obviously)
191
+ - the rule packs in `rules/`
192
+ - `--depth`, `--profile`, `--allow-commands`
193
+ - `.usat.yaml` suppressions and overrides
194
+
195
+ This is what makes `usat diff` meaningful, and it is why the report records the
196
+ commit SHA, the USAT version, and every option it ran with.