@xenos1996/usat 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +335 -0
- package/USAT.md +596 -0
- package/dist/cli.d.ts +3 -0
- package/dist/cli.d.ts.map +1 -0
- package/dist/cli.js +438 -0
- package/dist/cli.js.map +1 -0
- package/dist/config.d.ts +7 -0
- package/dist/config.d.ts.map +1 -0
- package/dist/config.js +72 -0
- package/dist/config.js.map +1 -0
- package/dist/detect/index.d.ts +58 -0
- package/dist/detect/index.d.ts.map +1 -0
- package/dist/detect/index.js +329 -0
- package/dist/detect/index.js.map +1 -0
- package/dist/engine/audit.d.ts +21 -0
- package/dist/engine/audit.d.ts.map +1 -0
- package/dist/engine/audit.js +295 -0
- package/dist/engine/audit.js.map +1 -0
- package/dist/engine/diff.d.ts +24 -0
- package/dist/engine/diff.d.ts.map +1 -0
- package/dist/engine/diff.js +136 -0
- package/dist/engine/diff.js.map +1 -0
- package/dist/engine/evaluate.d.ts +19 -0
- package/dist/engine/evaluate.d.ts.map +1 -0
- package/dist/engine/evaluate.js +377 -0
- package/dist/engine/evaluate.js.map +1 -0
- package/dist/engine/gate.d.ts +19 -0
- package/dist/engine/gate.d.ts.map +1 -0
- package/dist/engine/gate.js +47 -0
- package/dist/engine/gate.js.map +1 -0
- package/dist/engine/loader.d.ts +22 -0
- package/dist/engine/loader.d.ts.map +1 -0
- package/dist/engine/loader.js +360 -0
- package/dist/engine/loader.js.map +1 -0
- package/dist/engine/maturity.d.ts +20 -0
- package/dist/engine/maturity.d.ts.map +1 -0
- package/dist/engine/maturity.js +157 -0
- package/dist/engine/maturity.js.map +1 -0
- package/dist/engine/score.d.ts +19 -0
- package/dist/engine/score.d.ts.map +1 -0
- package/dist/engine/score.js +141 -0
- package/dist/engine/score.js.map +1 -0
- package/dist/engine/sections.d.ts +14 -0
- package/dist/engine/sections.d.ts.map +1 -0
- package/dist/engine/sections.js +50 -0
- package/dist/engine/sections.js.map +1 -0
- package/dist/index.d.ts +23 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +17 -0
- package/dist/index.js.map +1 -0
- package/dist/report/markdown.d.ts +16 -0
- package/dist/report/markdown.d.ts.map +1 -0
- package/dist/report/markdown.js +488 -0
- package/dist/report/markdown.js.map +1 -0
- package/dist/types.d.ts +282 -0
- package/dist/types.d.ts.map +1 -0
- package/dist/types.js +16 -0
- package/dist/types.js.map +1 -0
- package/dist/util/glob.d.ts +11 -0
- package/dist/util/glob.d.ts.map +1 -0
- package/dist/util/glob.js +86 -0
- package/dist/util/glob.js.map +1 -0
- package/dist/util/project.d.ts +72 -0
- package/dist/util/project.d.ts.map +1 -0
- package/dist/util/project.js +586 -0
- package/dist/util/project.js.map +1 -0
- package/dist/util/yaml.d.ts +28 -0
- package/dist/util/yaml.d.ts.map +1 -0
- package/dist/util/yaml.js +48 -0
- package/dist/util/yaml.js.map +1 -0
- package/docs/ARCHITECTURE.md +119 -0
- package/docs/README.md +27 -0
- package/docs/adr/0001-record-architecture-decisions.md +35 -0
- package/docs/adr/0002-typescript-engine-over-python.md +49 -0
- package/docs/adr/0003-rules-are-data-not-code.md +50 -0
- package/docs/adr/0004-markdown-only-output.md +41 -0
- package/docs/adr/0005-maturity-dampens-severity.md +56 -0
- package/docs/adr/0006-severity-status-two-axes.md +48 -0
- package/docs/adr/0007-suppressions-are-visible-and-expiring.md +46 -0
- package/docs/adr/0008-report-trailer-and-diff.md +48 -0
- package/docs/adr/0009-fail-closed-on-malformed-input.md +56 -0
- package/docs/adr/0010-detector-design.md +54 -0
- package/docs/adr/0011-coexist-with-deep-scanners.md +52 -0
- package/docs/adr/README.md +32 -0
- package/docs/agent-integration.md +129 -0
- package/docs/ci-integration.md +163 -0
- package/docs/concepts.md +196 -0
- package/docs/configuration.md +105 -0
- package/docs/detectors.md +186 -0
- package/docs/getting-started.md +99 -0
- package/docs/maturity-profiles.md +156 -0
- package/docs/reference/api.md +193 -0
- package/docs/reference/cli.md +412 -0
- package/docs/rule-packs.md +278 -0
- package/docs/standards-mapping.md +171 -0
- package/package.json +96 -0
- package/rules/core/architecture.yaml +190 -0
- package/rules/core/cicd.yaml +212 -0
- package/rules/core/code-quality.yaml +261 -0
- package/rules/core/dependencies.yaml +132 -0
- package/rules/core/documentation.yaml +119 -0
- package/rules/core/future-readiness.yaml +91 -0
- package/rules/core/release.yaml +131 -0
- package/rules/core/repo.yaml +303 -0
- package/rules/core/security.yaml +494 -0
- package/rules/core/supply-chain.yaml +357 -0
- package/rules/core/testing.yaml +260 -0
- package/rules/detectors.yaml +2255 -0
- package/rules/index.yaml +41 -0
- package/rules/profiles/maturity.yaml +135 -0
- package/rules/stacks/ai-era.yaml +193 -0
- package/rules/stacks/api-backend.yaml +121 -0
- package/rules/stacks/cli.yaml +105 -0
- package/rules/stacks/compliance.yaml +157 -0
- package/rules/stacks/containers.yaml +124 -0
- package/rules/stacks/data.yaml +165 -0
- package/rules/stacks/go.yaml +134 -0
- package/rules/stacks/iac.yaml +128 -0
- package/rules/stacks/jvm.yaml +150 -0
- package/rules/stacks/ml-ai.yaml +140 -0
- package/rules/stacks/mobile.yaml +121 -0
- package/rules/stacks/node-typescript.yaml +166 -0
- package/rules/stacks/python.yaml +175 -0
- package/rules/stacks/rust.yaml +129 -0
- package/rules/stacks/solidity.yaml +139 -0
- package/rules/stacks/web-frontend.yaml +159 -0
- package/templates/AGENTS.audit.md +135 -0
- package/templates/AUDIT_REPORT.md +155 -0
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# 10. Detectors are cheap syntactic signals resolved to a fixed point
|
|
2
|
+
|
|
3
|
+
- **Date:** 2026-09-09
|
|
4
|
+
- **Status:** Accepted
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
Rule selection needs facts ("is this a Node project? does it have a
|
|
9
|
+
database?"), and facts must come from the tree itself with no network, no
|
|
10
|
+
build, and no language server. The temptation is a clever detector: parse
|
|
11
|
+
`pyproject.toml` properly, resolve `extends` chains in tsconfig, understand
|
|
12
|
+
TOML tables. The constraint is that detectors run over arbitrary repos in
|
|
13
|
+
milliseconds and must never crash the audit.
|
|
14
|
+
|
|
15
|
+
The opposing risk is over-firing: docs, tests, examples, and templates _talk
|
|
16
|
+
about_ technology without _using_ it. A detector that counts the word
|
|
17
|
+
"postgres" in `docs/` reports a database the project does not have.
|
|
18
|
+
|
|
19
|
+
## Decision
|
|
20
|
+
|
|
21
|
+
1. **Syntactic signals only.** Detectors match file names, directory names,
|
|
22
|
+
manifest keys, and single-line content patterns. No parsing beyond
|
|
23
|
+
section-scoped text search and JSON path lookup.
|
|
24
|
+
2. **Prose is excluded from content signals.** Markdown, docs, examples,
|
|
25
|
+
templates, tests, fixtures, and mocks never count as technology evidence
|
|
26
|
+
(`CONTENT_EXCLUDES`). A deliberately conservative list, extended only
|
|
27
|
+
with evidence of a miss — `__tests__/`, `spec/`, and `*.stories.*` were
|
|
28
|
+
added after review showed Jest and Storybook layouts leaking through.
|
|
29
|
+
3. **Manifest queries are section-scoped where sections exist.** A dotted
|
|
30
|
+
key (`tool.poetry.dependencies`) must match a full header segment
|
|
31
|
+
(case-insensitive; `[dev-dependencies]` is not `[dependencies]`), the
|
|
32
|
+
leaf must match on word boundaries (`test` is not `latest`), and
|
|
33
|
+
`contains` is checked against the matched block — never the whole file.
|
|
34
|
+
Flat files with no headers keep whole-text search (YAML has no sections
|
|
35
|
+
to scope to).
|
|
36
|
+
4. **`implies` chains resolve to a fixed point**, bounded by
|
|
37
|
+
`detectors.length + 1` passes with early exit. Two hardcoded passes
|
|
38
|
+
silently dropped chains of length 3+ declared worst-first, inflating
|
|
39
|
+
scores — the bound is now structural, not a magic number.
|
|
40
|
+
|
|
41
|
+
## Consequences
|
|
42
|
+
|
|
43
|
+
**Good:** detection is fast, total (no network), crash-free, and order-
|
|
44
|
+
independent. The precision rules above each cite the false positive they
|
|
45
|
+
killed, so future relaxations know what they are re-introducing.
|
|
46
|
+
|
|
47
|
+
**Bad:** syntactic detection has a ceiling: `extends` chains leaving the
|
|
48
|
+
repo, aliased imports, and commented-out config are invisible. Detectors
|
|
49
|
+
report evidence, not proof — rules that need proof belong in the judgement
|
|
50
|
+
queue, not in a cleverer regex.
|
|
51
|
+
|
|
52
|
+
**Neutral:** the fixed-point bound is `detectors.length + 1` passes worst
|
|
53
|
+
case; typical runs exit after two. Performance is unchanged for existing
|
|
54
|
+
registries.
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# 11. Coexist with deep scanners: verify configuration, consume evidence
|
|
2
|
+
|
|
3
|
+
- **Date:** 2026-09-09
|
|
4
|
+
- **Status:** Accepted
|
|
5
|
+
|
|
6
|
+
## Context
|
|
7
|
+
|
|
8
|
+
A regex engine cannot do dataflow analysis (CodeQL), reachability-filtered
|
|
9
|
+
CVE prioritisation (Snyk, Semgrep Supply Chain), whole-program taint
|
|
10
|
+
tracking, or API-verified hygiene (Scorecard branch protection, SLSA
|
|
11
|
+
attestation verification). Pretending otherwise would mean shipping weak
|
|
12
|
+
copies of those tools and claiming parity — the fastest way to destroy an
|
|
13
|
+
auditor's credibility.
|
|
14
|
+
|
|
15
|
+
But USAT's value proposition includes supply chain, vulnerability posture,
|
|
16
|
+
and hygiene verdicts. Ignoring those dimensions leaves the report with
|
|
17
|
+
holes exactly where executives look first.
|
|
18
|
+
|
|
19
|
+
## Decision
|
|
20
|
+
|
|
21
|
+
USAT **verifies configuration and consumes evidence; it does not duplicate
|
|
22
|
+
analysis**:
|
|
23
|
+
|
|
24
|
+
1. Where a deep scanner exists, the rule checks that it is **configured and
|
|
25
|
+
running** (lockfile + frozen install, scanner in CI, SBOM published,
|
|
26
|
+
attestations present) — never that USAT re-derives its findings.
|
|
27
|
+
2. Where a scanner emits machine output, future check kinds may **ingest**
|
|
28
|
+
it (SARIF/JSON: coverage numbers, HIGH/CRITICAL CVE counts, attestation
|
|
29
|
+
verification success) as evidence for the corresponding rules.
|
|
30
|
+
3. USAT never claims what an oracle would contradict: API-verifiable checks
|
|
31
|
+
(branch protection, token permissions, signed releases) are file-text
|
|
32
|
+
approximations, documented as such, and the report prefers MISSING
|
|
33
|
+
("not verified") over a confident verdict it cannot support.
|
|
34
|
+
4. SARIF (and JSON) renderers are the approved interchange in both
|
|
35
|
+
directions: USAT findings out to dashboards and code scanning, oracle
|
|
36
|
+
findings in as evidence. The internal finding model already carries
|
|
37
|
+
file/line/severity/rule identity, so this is a renderer, not a redesign.
|
|
38
|
+
|
|
39
|
+
## Consequences
|
|
40
|
+
|
|
41
|
+
**Good:** no vulnerability database to maintain, no resolver to build, no
|
|
42
|
+
dataflow engine to write — and no weak-duplicate claims. Each tool does
|
|
43
|
+
what it is good at; the audit composes them.
|
|
44
|
+
|
|
45
|
+
**Bad:** USAT reports are only as strong as the oracles the project runs.
|
|
46
|
+
A repo with no scanners gets MISSING verdicts where a bundled scanner
|
|
47
|
+
would give answers. That is honest but less satisfying, and must be
|
|
48
|
+
explained in every such report (the confidence column exists for this).
|
|
49
|
+
|
|
50
|
+
**Neutral:** this ADR draws the scope boundary that keeps USAT small. Any
|
|
51
|
+
proposal to add a CVE database, a resolver, or interprocedural analysis
|
|
52
|
+
re-opens this decision explicitly.
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
# Architecture Decision Records
|
|
2
|
+
|
|
3
|
+
Short, numbered, immutable records of the non-obvious choices in this
|
|
4
|
+
repository — see [ADR-0001](0001-record-architecture-decisions.md) for why
|
|
5
|
+
they exist and the rules for writing them (context, decision, consequences;
|
|
6
|
+
a changed decision gets a _new_ ADR that supersedes the old one).
|
|
7
|
+
|
|
8
|
+
## Index
|
|
9
|
+
|
|
10
|
+
| # | Title | Status | Supersedes |
|
|
11
|
+
| ---- | --------------------------------------------------- | -------- | ---------- |
|
|
12
|
+
| 0001 | Record architecture decisions | Accepted | — |
|
|
13
|
+
| 0002 | Build the engine in TypeScript, not Python | Accepted | — |
|
|
14
|
+
| 0003 | Rule packs are YAML data, not TypeScript code | Accepted | — |
|
|
15
|
+
| 0004 | Markdown is the only output format | Accepted | — |
|
|
16
|
+
| 0005 | Lifecycle stage dampens severity; CRITICAL exempt | Accepted | — |
|
|
17
|
+
| 0006 | Severity/status two axes; WRONG outranks MISSING | Accepted | — |
|
|
18
|
+
| 0007 | Accepted risk stays visible, stops scoring, expires | Accepted | — |
|
|
19
|
+
| 0008 | Machine-readable trailer; audits are diffable | Accepted | — |
|
|
20
|
+
| 0009 | Malformed input fails closed and loudly | Accepted | — |
|
|
21
|
+
| 0010 | Detectors are cheap signals, fixed-point resolved | Accepted | — |
|
|
22
|
+
| 0011 | Coexist with deep scanners; consume evidence | Accepted | — |
|
|
23
|
+
|
|
24
|
+
## Coverage map (what has no ADR, and why)
|
|
25
|
+
|
|
26
|
+
- **Scoring maths** — specified normatively in `docs/concepts.md`, not an
|
|
27
|
+
ADR: it is a formula with tests, not a judgement call. Disagreeing with it
|
|
28
|
+
means changing code, and the tests adjudicate.
|
|
29
|
+
- **Individual rule wordings** — rules are data (ADR-0003). Disputed rules
|
|
30
|
+
get fixed in `rules/` with a regression test, not an ADR.
|
|
31
|
+
- **Release mechanics** — tracked in `ROADMAP.md` until decided, then
|
|
32
|
+
recorded here (e.g. the eventual release-automation choice).
|
|
@@ -0,0 +1,129 @@
|
|
|
1
|
+
# Agent integration
|
|
2
|
+
|
|
3
|
+
USAT is built to be driven by an AI agent. The division of labour is explicit:
|
|
4
|
+
|
|
5
|
+
| | Who | Why |
|
|
6
|
+
| -------------------- | ------------- | ------------------------------------------------------------------------------- |
|
|
7
|
+
| Deterministic checks | **the tool** | An LLM grepping 400 files is slow, expensive, and non-deterministic |
|
|
8
|
+
| Judgement checks | **the agent** | Coupling, authorisation, threat modelling — nothing settles these but reasoning |
|
|
9
|
+
| The report | **both** | Tool renders it; agent fills the judgement queue |
|
|
10
|
+
|
|
11
|
+
The tool settles roughly 70% of applicable rules on a typical project. The agent's
|
|
12
|
+
job is the remaining 30% — the interesting 30%.
|
|
13
|
+
|
|
14
|
+
## The workflow
|
|
15
|
+
|
|
16
|
+
```bash
|
|
17
|
+
usat audit . --out AUDIT.md # 1. deterministic pass
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
Then, to the agent:
|
|
21
|
+
|
|
22
|
+
> Read `USAT.md` and `AUDIT.md`.
|
|
23
|
+
> Work the **Judgement Queue** at the bottom of the report.
|
|
24
|
+
> For each item, find the evidence and record `file:line` plus one sentence of
|
|
25
|
+
> reasoning. Do not mark anything ✅ without evidence (Rule 4).
|
|
26
|
+
> Then rewrite the report with your findings folded in.
|
|
27
|
+
|
|
28
|
+
## Why this is better than "audit this codebase"
|
|
29
|
+
|
|
30
|
+
A bare prompt produces three failure modes, and the template exists to prevent all
|
|
31
|
+
three:
|
|
32
|
+
|
|
33
|
+
| Failure | How USAT prevents it |
|
|
34
|
+
| ------------------------------------------------------------------- | --------------------------------------------------------------------------------- |
|
|
35
|
+
| **Confident invention** — "✅ MFA is implemented" with no such code | Rule 4: no ✅ without evidence; every judgement item states what proof looks like |
|
|
36
|
+
| **Wrong scope** — Solidity reentrancy checks on a Next.js blog | Detection + `applies_when`; non-applicable sections are skipped silently (Rule 1) |
|
|
37
|
+
| **Wrong severity** — a prototype graded like a bank, or vice versa | Maturity profiles, with a hard floor on CRITICAL |
|
|
38
|
+
|
|
39
|
+
## Shipped packs
|
|
40
|
+
|
|
41
|
+
### Agent Skills (`SKILL.md`)
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
cp -r skills/usat-audit ~/.claude/skills/ # Claude Code
|
|
45
|
+
cp -r skills/usat-audit .cursor/skills/ # Cursor
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
Follows the Agent Skills convention — YAML frontmatter with `name` and `description`,
|
|
49
|
+
detailed content in `references/` [1](https://github.com/addyosmani/agent-skills/blob/main/CLAUDE.md).
|
|
50
|
+
The same directory works across Claude Code, Cursor, Codex, Gemini CLI, Copilot, and
|
|
51
|
+
~60 other tools.
|
|
52
|
+
|
|
53
|
+
### `AGENTS.md` for a target repository
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
usat init .
|
|
57
|
+
cp templates/AGENTS.audit.md AGENTS.md
|
|
58
|
+
```
|
|
59
|
+
|
|
60
|
+
Gives any agent working in that repo the build/test/lint commands, the conventions,
|
|
61
|
+
and an explicit denylist — which is worth doing on its own. Agent instruction files
|
|
62
|
+
are executable-ish: they commit as easily as code and agents follow them literally
|
|
63
|
+
(see `AI-011`).
|
|
64
|
+
|
|
65
|
+
## Prompt you can paste
|
|
66
|
+
|
|
67
|
+
```
|
|
68
|
+
You are auditing this repository with USAT (Universal Software Audit Template).
|
|
69
|
+
|
|
70
|
+
1. Run: usat detect .
|
|
71
|
+
Confirm the detected facts. If any are wrong, say so and note the correct ones.
|
|
72
|
+
|
|
73
|
+
2. Run: usat audit . --out AUDIT.md
|
|
74
|
+
|
|
75
|
+
3. Read USAT.md (the framework) and AUDIT.md (the deterministic results).
|
|
76
|
+
|
|
77
|
+
4. Work the Judgement Queue in AUDIT.md. For each item:
|
|
78
|
+
- locate the relevant code
|
|
79
|
+
- decide PASS / WRONG / MISSING
|
|
80
|
+
- record file:line evidence and one sentence of reasoning
|
|
81
|
+
- if you cannot determine it, say UNKNOWN and say what you would need
|
|
82
|
+
|
|
83
|
+
5. Produce a final report that follows the Section 14 template in USAT.md:
|
|
84
|
+
- CRITICAL and HIGH findings first, with location and fix
|
|
85
|
+
- then section-by-section
|
|
86
|
+
- then the roadmap
|
|
87
|
+
- then the judgement queue with your recorded evidence
|
|
88
|
+
|
|
89
|
+
Rules:
|
|
90
|
+
- Never mark ✅ without evidence (file:line or command output).
|
|
91
|
+
- Skip sections that do not apply — do not explain why.
|
|
92
|
+
- ⚠️ WRONG is worse than 🚫 MISSING. Surface it.
|
|
93
|
+
- Priority: security > correctness > maintainability > style.
|
|
94
|
+
```
|
|
95
|
+
|
|
96
|
+
## Using the API directly
|
|
97
|
+
|
|
98
|
+
```ts
|
|
99
|
+
import { runAudit, renderMarkdown, loadProfiles } from 'usat';
|
|
100
|
+
|
|
101
|
+
const { report, profile, warnings } = runAudit({
|
|
102
|
+
target: process.cwd(),
|
|
103
|
+
rulesDir: 'rules',
|
|
104
|
+
depth: 'deep',
|
|
105
|
+
profile: 'auto',
|
|
106
|
+
config: { version: 1 },
|
|
107
|
+
allowCommands: false,
|
|
108
|
+
usatVersion: '1.0.0',
|
|
109
|
+
});
|
|
110
|
+
|
|
111
|
+
console.log(report.score.overall); // 71.4
|
|
112
|
+
console.log(report.score.counts.UNKNOWN); // 18 → send these to the agent
|
|
113
|
+
const md = renderMarkdown(report, profile);
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
`report.findings.filter(f => f.status === 'UNKNOWN')` is precisely the agent's task
|
|
117
|
+
list, and each item carries `why` and `evidenceHint`.
|
|
118
|
+
|
|
119
|
+
## Notes for agent authors
|
|
120
|
+
|
|
121
|
+
- **Do not re-run greps the tool already did.** If a finding is already ✅ or 🚫 with
|
|
122
|
+
locations, the tool settled it.
|
|
123
|
+
- **Trust `suppressedReason`.** A suppressed finding was a decision someone made. Mention
|
|
124
|
+
it only if the reason has clearly expired.
|
|
125
|
+
- **Respect the maturity profile.** If a finding is marked
|
|
126
|
+
_"Downgraded HIGH → MEDIUM by the MVP profile"_, do not re-escalate it — unless it is
|
|
127
|
+
CRITICAL, which is never downgraded.
|
|
128
|
+
- **Say UNKNOWN when you do not know.** It is a valid answer and it keeps confidence
|
|
129
|
+
honest. Guessing is the failure mode this whole framework exists to prevent.
|
|
@@ -0,0 +1,163 @@
|
|
|
1
|
+
# CI integration
|
|
2
|
+
|
|
3
|
+
## GitHub Actions
|
|
4
|
+
|
|
5
|
+
`usat init` writes a ready-made workflow. The minimum viable version:
|
|
6
|
+
|
|
7
|
+
```yaml
|
|
8
|
+
name: USAT Audit
|
|
9
|
+
on: [pull_request]
|
|
10
|
+
|
|
11
|
+
permissions:
|
|
12
|
+
contents: read
|
|
13
|
+
pull-requests: write
|
|
14
|
+
|
|
15
|
+
jobs:
|
|
16
|
+
audit:
|
|
17
|
+
runs-on: ubuntu-latest
|
|
18
|
+
steps:
|
|
19
|
+
- uses: actions/checkout@v4
|
|
20
|
+
- uses: actions/setup-node@v4
|
|
21
|
+
with:
|
|
22
|
+
node-version: '20'
|
|
23
|
+
|
|
24
|
+
- name: Run USAT
|
|
25
|
+
id: usat
|
|
26
|
+
run: npx --yes @xenos1996/usat@1 audit . --depth standard --out AUDIT.md
|
|
27
|
+
|
|
28
|
+
- name: Publish to job summary
|
|
29
|
+
if: always()
|
|
30
|
+
run: cat AUDIT.md >> "$GITHUB_STEP_SUMMARY"
|
|
31
|
+
|
|
32
|
+
- name: Upload report
|
|
33
|
+
if: always()
|
|
34
|
+
uses: actions/upload-artifact@v4
|
|
35
|
+
with:
|
|
36
|
+
name: usat-audit
|
|
37
|
+
path: AUDIT.md
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
### Quality gate
|
|
41
|
+
|
|
42
|
+
```yaml
|
|
43
|
+
- name: Quality gate
|
|
44
|
+
run: npx --yes @xenos1996/usat@1 audit . --fail-on high
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
| Exit | Meaning |
|
|
48
|
+
| ---- | ----------------------------------------- |
|
|
49
|
+
| `0` | No findings at or above the threshold |
|
|
50
|
+
| `1` | Gate tripped — findings printed to stderr |
|
|
51
|
+
| `2` | Usage or configuration error |
|
|
52
|
+
|
|
53
|
+
**Roll-out advice:** start with `--fail-on critical`. Move to `high` once the backlog
|
|
54
|
+
is clear. Never start at `medium` — you will teach the team to bypass the check.
|
|
55
|
+
|
|
56
|
+
### Comment the score on the PR
|
|
57
|
+
|
|
58
|
+
```yaml
|
|
59
|
+
- name: Comment
|
|
60
|
+
if: github.event_name == 'pull_request'
|
|
61
|
+
uses: marocchino/sticky-pull-request-comment@v2
|
|
62
|
+
with:
|
|
63
|
+
path: AUDIT.md
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
### Composite action
|
|
67
|
+
|
|
68
|
+
```yaml
|
|
69
|
+
- uses: Er-Sajan-PLG/software-auditing-template@v1
|
|
70
|
+
with:
|
|
71
|
+
depth: standard
|
|
72
|
+
fail-on: high
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
See [`action.yml`](../action.yml).
|
|
76
|
+
|
|
77
|
+
---
|
|
78
|
+
|
|
79
|
+
## GitLab CI
|
|
80
|
+
|
|
81
|
+
```yaml
|
|
82
|
+
usat-audit:
|
|
83
|
+
image: node:20
|
|
84
|
+
stage: test
|
|
85
|
+
script:
|
|
86
|
+
- npx --yes @xenos1996/usat@1 audit . --out usat-report.md --fail-on critical
|
|
87
|
+
artifacts:
|
|
88
|
+
when: always
|
|
89
|
+
paths: [usat-report.md]
|
|
90
|
+
expose_as: 'USAT Audit'
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
---
|
|
94
|
+
|
|
95
|
+
## Scheduled drift detection
|
|
96
|
+
|
|
97
|
+
The highest-value CI job is not the PR gate — it is a monthly audit that shows
|
|
98
|
+
movement:
|
|
99
|
+
|
|
100
|
+
```yaml
|
|
101
|
+
on:
|
|
102
|
+
schedule:
|
|
103
|
+
- cron: '0 6 1 * *' # 1st of the month
|
|
104
|
+
workflow_dispatch:
|
|
105
|
+
|
|
106
|
+
jobs:
|
|
107
|
+
audit:
|
|
108
|
+
steps:
|
|
109
|
+
- uses: actions/checkout@v4
|
|
110
|
+
with: { fetch-depth: 0 }
|
|
111
|
+
- run: npx --yes @xenos1996/usat@1 audit . --out reports/$(date +%Y-%m).md
|
|
112
|
+
- run: |
|
|
113
|
+
PREV=$(ls reports/*.md | tail -2 | head -1)
|
|
114
|
+
npx --yes @xenos1996/usat@1 diff "$PREV" "reports/$(date +%Y-%m).md" --out DIFF.md || true
|
|
115
|
+
cat DIFF.md >> "$GITHUB_STEP_SUMMARY"
|
|
116
|
+
- uses: peter-evans/create-pull-request@v6
|
|
117
|
+
with:
|
|
118
|
+
title: 'chore: monthly USAT audit'
|
|
119
|
+
body-path: DIFF.md
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
A dated report per month plus `usat diff` gives you an audit trail that shows
|
|
123
|
+
improvement — the thing a single audit can never do.
|
|
124
|
+
|
|
125
|
+
---
|
|
126
|
+
|
|
127
|
+
## Reference: this repo's own CI (`.github/workflows/`)
|
|
128
|
+
|
|
129
|
+
USAT audits itself with the full stack — copy what fits:
|
|
130
|
+
|
|
131
|
+
| Workflow | What it does |
|
|
132
|
+
| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
|
|
133
|
+
| `ci.yml` | lint+format+typecheck · Vitest with coverage thresholds · build + CLI smoke · rule-pack validation · npm audit + gitleaks + license scan · **hygiene** (`check-adrs` + `check-docs`) |
|
|
134
|
+
| `self-audit.yml` | `usat audit . --depth deep --fail-on critical` on every PR, score as PR comment |
|
|
135
|
+
| `scorecard.yml` | OpenSSF Scorecard monthly + on push (API-verified hygiene; SARIF to Security tab) |
|
|
136
|
+
| `automerge.yml` | Dependabot patch/minor auto-merge once CI is green (majors stay manual) |
|
|
137
|
+
| `gitleaks-pin.yml` | Monthly check that the curl-pinned gitleaks binary in `ci.yml` is current (no bot watches it) — opens a deduped issue when stale |
|
|
138
|
+
| `release.yml` | Tag push `v*` → OIDC trusted publishing (no long-lived token) + `--provenance` + CycloneDX SBOM artifact |
|
|
139
|
+
|
|
140
|
+
Release setup note: trusted publishing needs a one-time owner step on
|
|
141
|
+
npmjs.com (package Settings → Trusted Publisher → this repo + workflow)
|
|
142
|
+
before the first OIDC publish succeeds.
|
|
143
|
+
|
|
144
|
+
## Choosing a depth in CI
|
|
145
|
+
|
|
146
|
+
| Depth | Rules | Runtime (≈10k files) | Use for |
|
|
147
|
+
| ---------- | ---------------------------------------------- | -------------------- | -------------------- |
|
|
148
|
+
| `quick` | high-signal only | ~1–3 s | Every PR |
|
|
149
|
+
| `standard` | default | ~3–10 s | PRs + nightly |
|
|
150
|
+
| `deep` | adds cycles, duplication, complexity, mutation | ~10–30 s | Weekly / pre-release |
|
|
151
|
+
|
|
152
|
+
`deep` is where the judgement-heavy rules live; the extra cost is mostly I/O.
|
|
153
|
+
|
|
154
|
+
## Notes
|
|
155
|
+
|
|
156
|
+
- **No network at audit time.** USAT reads files and writes Markdown. It never uploads
|
|
157
|
+
anything, which is why it is safe on private repositories.
|
|
158
|
+
- **`--allow-commands` in CI.** Only if you trust the target repo — it shells out for
|
|
159
|
+
checks like `npm audit`. Off by default; those rules report ❓ NEEDS REVIEW instead.
|
|
160
|
+
- **Pin the version** in production pipelines (`@xenos1996/usat@1`, not `@latest`) so a
|
|
161
|
+
rule-pack change cannot fail your build without a commit.
|
|
162
|
+
- **Commit `.usat.yaml`.** Suppressions and overrides without a commit are invisible
|
|
163
|
+
decisions, and they are the first thing a reviewer asks about.
|
package/docs/concepts.md
ADDED
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
# Concepts
|
|
2
|
+
|
|
3
|
+
## Two axes: severity and status
|
|
4
|
+
|
|
5
|
+
Most audit checklists merge "how bad is it" and "what did we find" into one column.
|
|
6
|
+
That makes scoring impossible to reproduce — two auditors tag the same observation
|
|
7
|
+
differently, and the totals diverge. USAT keeps them apart.
|
|
8
|
+
|
|
9
|
+
### Severity — a property of the rule
|
|
10
|
+
|
|
11
|
+
Set by whoever wrote the rule. It says: _if this is violated, how bad is that?_
|
|
12
|
+
|
|
13
|
+
| Tag | Severity | Meaning | Default weight |
|
|
14
|
+
| --- | ---------- | ------------------------------------------------- | -------------- |
|
|
15
|
+
| 🔴 | `CRITICAL` | Breaks security, data integrity, or core function | 10 |
|
|
16
|
+
| 🟠 | `HIGH` | Major risk or substantial debt | 6 |
|
|
17
|
+
| 🟡 | `MEDIUM` | Important gap | 3 |
|
|
18
|
+
| 🟢 | `LOW` | Minor improvement | 1.5 |
|
|
19
|
+
| 🔵 | `FUTURE` | Not needed now, needed at scale | 0.5 |
|
|
20
|
+
|
|
21
|
+
### Status — a property of the observation
|
|
22
|
+
|
|
23
|
+
Produced by the engine (or by a reviewer, for judgement checks).
|
|
24
|
+
|
|
25
|
+
| Tag | Status | Score credit | When |
|
|
26
|
+
| --- | -------------- | ------------ | -------------------------------------------------- |
|
|
27
|
+
| ✅ | `GOOD` | 1.00 | Verified present and correct |
|
|
28
|
+
| 🧪 | `EXPERIMENTAL` | 0.50 | Present, unvalidated (`grep_experimental`) |
|
|
29
|
+
| 💀 | `DEPRECATED` | 0.40 | Present but EOL (`grep_deprecated`) |
|
|
30
|
+
| ⚠️ | `WRONG` | 0.15 | Present but implemented incorrectly (`grep_wrong`) |
|
|
31
|
+
| 🚫 | `MISSING` | 0.00 | Required and absent |
|
|
32
|
+
| ❓ | `NEEDS REVIEW` | _excluded_ | Judgement required; no evidence recorded |
|
|
33
|
+
| ➖ | `SKIPPED` | _excluded_ | Not applicable to this project |
|
|
34
|
+
|
|
35
|
+
**Why WRONG scores 0.15 and not 0.00.** Something exists, so there is partial
|
|
36
|
+
credit — the intent was right and the surface is smaller than a greenfield fix.
|
|
37
|
+
But a wrong implementation is _more dangerous_ than an absent one, because it looks
|
|
38
|
+
finished: nobody puts "add CSRF protection" on the roadmap when CSRF protection
|
|
39
|
+
appears to be there. Rule 9 of the agent behaviour rules exists for exactly this.
|
|
40
|
+
|
|
41
|
+
**Why NEEDS REVIEW is excluded rather than scored zero.** An unanswered question is
|
|
42
|
+
not a failure. It is a hole in the audit, and the place it shows up is
|
|
43
|
+
**confidence**, not the score.
|
|
44
|
+
|
|
45
|
+
---
|
|
46
|
+
|
|
47
|
+
## Scoring
|
|
48
|
+
|
|
49
|
+
### Per rule
|
|
50
|
+
|
|
51
|
+
```
|
|
52
|
+
creditᵢ = weightᵢ × CREDIT[statusᵢ]
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
### Per section
|
|
56
|
+
|
|
57
|
+
```
|
|
58
|
+
sectionScore = 10 × Σ creditᵢ / Σ weightᵢ (over resolved, applicable rules)
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
### Overall
|
|
62
|
+
|
|
63
|
+
```
|
|
64
|
+
overall = 100 × Σ (creditᵢ × sectionWeight) / Σ (weightᵢ × sectionWeight)
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
Rule-level weighting, rather than averaging section scores, means a section with
|
|
68
|
+
three rules cannot swing the total as hard as one with thirty.
|
|
69
|
+
|
|
70
|
+
### Section weights
|
|
71
|
+
|
|
72
|
+
Security, supply chain, and testing carry the heaviest multipliers — a repo that
|
|
73
|
+
fails an authorisation control is worse than one with a slightly untidy README, and
|
|
74
|
+
the number should say so.
|
|
75
|
+
|
|
76
|
+
| | Section | Weight |
|
|
77
|
+
| ------------------- | ------------------------------------------------- | ------- |
|
|
78
|
+
| **S2** | Security | **1.7** |
|
|
79
|
+
| **S3** | Supply Chain & Build Provenance | 1.3 |
|
|
80
|
+
| **S7** | Testing & Quality Assurance | 1.3 |
|
|
81
|
+
| **S10** | Dependencies & Third-Party | 1.2 |
|
|
82
|
+
| **S14** | AI / LLM-Era Risks | 1.1 |
|
|
83
|
+
| S4, S5, S6, S8, S15 | Architecture, Code Quality, Data, CI/CD, Platform | 1.0 |
|
|
84
|
+
| **S13** | Accessibility, i18n & Compliance | 0.9 |
|
|
85
|
+
| **S11** | Performance & Resilience | 0.8 |
|
|
86
|
+
| S1, S9, S12 | Repository, Release, Documentation | 0.7 |
|
|
87
|
+
| **S16** | Future Readiness | 0.4 |
|
|
88
|
+
|
|
89
|
+
Override any of them in `rules/sections.yaml`.
|
|
90
|
+
|
|
91
|
+
### Confidence
|
|
92
|
+
|
|
93
|
+
```
|
|
94
|
+
confidence = resolved / applicable
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
`resolved` counts rules the engine could settle (PASS, FAIL, WRONG, MISSING,
|
|
98
|
+
DEPRECATED, EXPERIMENTAL). `applicable` counts everything `applies_when` let through.
|
|
99
|
+
|
|
100
|
+
- Sections with **zero** resolved rules are reported as _"— not verified"_ and
|
|
101
|
+
excluded from the overall score. You cannot earn points for questions nobody answered.
|
|
102
|
+
- A high score at low confidence is the single most misleading thing an audit tool
|
|
103
|
+
can produce, which is why confidence sits next to every score in the report.
|
|
104
|
+
|
|
105
|
+
---
|
|
106
|
+
|
|
107
|
+
## Maturity dampening
|
|
108
|
+
|
|
109
|
+
Severity is reduced by the detected lifecycle stage, per rule class — but
|
|
110
|
+
**CRITICAL is never dampened, at any stage.**
|
|
111
|
+
|
|
112
|
+
| Stage | security | supply-chain | correctness | maintainability | operations | performance | compliance | documentation | style |
|
|
113
|
+
| ---------- | -------- | ------------ | ----------- | --------------- | ---------- | ----------- | ---------- | ------------- | ----- |
|
|
114
|
+
| Prototype | −1 | −1 | −1 | −2 | −2 | −2 | −2 | −2 | −2 |
|
|
115
|
+
| MVP | −0 | −1 | −1 | −1 | −1 | −1 | −1 | −2 | −2 |
|
|
116
|
+
| Beta | −0 | −0 | −0 | −1 | −1 | −1 | −1 | −1 | −1 |
|
|
117
|
+
| Production | −0 | −0 | −0 | −0 | −0 | −0 | −0 | −0 | −0 |
|
|
118
|
+
| Legacy | −0 | −0 | −0 | −1 | −0 | −1 | −0 | −1 | −1 |
|
|
119
|
+
|
|
120
|
+
Steps move down the ladder `FUTURE < LOW < MEDIUM < HIGH < CRITICAL`.
|
|
121
|
+
|
|
122
|
+
Every dampened finding says so in the report:
|
|
123
|
+
|
|
124
|
+
```
|
|
125
|
+
- 🚫 **LICENSE present** `REPO-005`
|
|
126
|
+
- 🪶 Downgraded HIGH → LOW by the Prototype / Spike profile
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
so a prototype reading knows which items will climb on their own as it grows up.
|
|
130
|
+
See [maturity-profiles.md](maturity-profiles.md).
|
|
131
|
+
|
|
132
|
+
---
|
|
133
|
+
|
|
134
|
+
## Applicability
|
|
135
|
+
|
|
136
|
+
A rule participates when **all** of these hold:
|
|
137
|
+
|
|
138
|
+
1. Its pack is not skipped (`skip_when` is false)
|
|
139
|
+
2. `applies_when` evaluates true against the detected facts
|
|
140
|
+
3. Its `depths` (if any) include the current `--depth`
|
|
141
|
+
4. It is not disabled in `.usat.yaml`
|
|
142
|
+
|
|
143
|
+
Two escape hatches, both loud: a force-included pack (`include:`) applies
|
|
144
|
+
all its rules regardless of 2 and 3 — the operator asked for it — and a
|
|
145
|
+
malformed `applies_when` (unknown key or operator, bad regex, non-mapping)
|
|
146
|
+
warns at load time and evaluates to false, so a typo can never silently
|
|
147
|
+
include rules. See ADR-0009.
|
|
148
|
+
|
|
149
|
+
```yaml
|
|
150
|
+
- id: DATA-007
|
|
151
|
+
applies_when: { fact: 'has:database' }
|
|
152
|
+
depths: [deep]
|
|
153
|
+
check: { kind: manual }
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Packs can contribute facts (`provides:`), and facts can make further packs apply.
|
|
157
|
+
The engine resolves this to a fixed point (max 3 passes) so a monorepo pack can pull
|
|
158
|
+
in boundary-enforcement rules without either pack knowing about the other.
|
|
159
|
+
|
|
160
|
+
---
|
|
161
|
+
|
|
162
|
+
## Suppressions
|
|
163
|
+
|
|
164
|
+
Anything suppressed needs a reason. That is not bureaucracy — it is the difference
|
|
165
|
+
between "we decided not to do this" and "we forgot".
|
|
166
|
+
|
|
167
|
+
```yaml
|
|
168
|
+
suppressions:
|
|
169
|
+
- rule: PERF-005
|
|
170
|
+
reason: 'Known N+1 in the admin panel; 40 rows max.'
|
|
171
|
+
until: '2026-12-31'
|
|
172
|
+
```
|
|
173
|
+
|
|
174
|
+
Suppressed findings are **excluded from the score, the severity tallies, and
|
|
175
|
+
the action sections, but still listed** under _Accepted Risk_. Anyone reading
|
|
176
|
+
the report can see what was waived and why — and the Findings Summary can
|
|
177
|
+
never contradict Immediate Action over a waived item.
|
|
178
|
+
|
|
179
|
+
Waivers **expire**: an `until` date in the past (or one that cannot be
|
|
180
|
+
parsed) is ignored with a warning, and the finding reports normally. See
|
|
181
|
+
ADR-0007.
|
|
182
|
+
|
|
183
|
+
---
|
|
184
|
+
|
|
185
|
+
## Reproducibility
|
|
186
|
+
|
|
187
|
+
The same tree, rules, depth, and profile always produce the same score. The only
|
|
188
|
+
inputs that can change a result between runs:
|
|
189
|
+
|
|
190
|
+
- file contents (obviously)
|
|
191
|
+
- the rule packs in `rules/`
|
|
192
|
+
- `--depth`, `--profile`, `--allow-commands`
|
|
193
|
+
- `.usat.yaml` suppressions and overrides
|
|
194
|
+
|
|
195
|
+
This is what makes `usat diff` meaningful, and it is why the report records the
|
|
196
|
+
commit SHA, the USAT version, and every option it ran with.
|