@harness-engineering/cli 2.6.1 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/agents/skills/claude-code/align-design-system/SKILL.md +174 -0
- package/dist/agents/skills/claude-code/align-design-system/skill.yaml +57 -0
- package/dist/agents/skills/claude-code/audit-brand-compliance/SKILL.md +189 -0
- package/dist/agents/skills/claude-code/audit-brand-compliance/skill.yaml +50 -0
- package/dist/agents/skills/claude-code/audit-component-anatomy/SKILL.md +181 -0
- package/dist/agents/skills/claude-code/audit-component-anatomy/skill.yaml +51 -0
- package/dist/agents/skills/claude-code/copy-craft/SKILL.md +192 -0
- package/dist/agents/skills/claude-code/copy-craft/skill.yaml +52 -0
- package/dist/agents/skills/claude-code/detect-design-drift/SKILL.md +139 -0
- package/dist/agents/skills/claude-code/detect-design-drift/skill.yaml +50 -0
- package/dist/agents/skills/claude-code/harness-accessibility/SKILL.md +10 -0
- package/dist/agents/skills/claude-code/harness-design-craft/SKILL.md +212 -0
- package/dist/agents/skills/claude-code/harness-design-craft/skill.yaml +57 -0
- package/dist/agents/skills/claude-code/harness-design-pipeline/SKILL.md +266 -0
- package/dist/agents/skills/claude-code/harness-design-pipeline/skill.yaml +70 -0
- package/dist/agents/skills/claude-code/knowledge-craft/SKILL.md +180 -0
- package/dist/agents/skills/claude-code/knowledge-craft/skill.yaml +49 -0
- package/dist/agents/skills/claude-code/naming-craft/SKILL.md +172 -0
- package/dist/agents/skills/claude-code/naming-craft/skill.yaml +51 -0
- package/dist/agents/skills/claude-code/security-craft/SKILL.md +205 -0
- package/dist/agents/skills/claude-code/security-craft/skill.yaml +58 -0
- package/dist/agents/skills/claude-code/spec-craft/SKILL.md +184 -0
- package/dist/agents/skills/claude-code/spec-craft/skill.yaml +55 -0
- package/dist/agents/skills/claude-code/test-craft/SKILL.md +212 -0
- package/dist/agents/skills/claude-code/test-craft/skill.yaml +58 -0
- package/dist/agents/skills/codex/align-design-system/SKILL.md +174 -0
- package/dist/agents/skills/codex/align-design-system/skill.yaml +57 -0
- package/dist/agents/skills/codex/audit-brand-compliance/SKILL.md +189 -0
- package/dist/agents/skills/codex/audit-brand-compliance/skill.yaml +50 -0
- package/dist/agents/skills/codex/audit-component-anatomy/SKILL.md +181 -0
- package/dist/agents/skills/codex/audit-component-anatomy/skill.yaml +51 -0
- package/dist/agents/skills/codex/copy-craft/SKILL.md +192 -0
- package/dist/agents/skills/codex/copy-craft/skill.yaml +52 -0
- package/dist/agents/skills/codex/detect-design-drift/SKILL.md +139 -0
- package/dist/agents/skills/codex/detect-design-drift/skill.yaml +50 -0
- package/dist/agents/skills/codex/harness-accessibility/SKILL.md +10 -0
- package/dist/agents/skills/codex/harness-design-craft/SKILL.md +212 -0
- package/dist/agents/skills/codex/harness-design-craft/skill.yaml +57 -0
- package/dist/agents/skills/codex/harness-design-pipeline/SKILL.md +266 -0
- package/dist/agents/skills/codex/harness-design-pipeline/skill.yaml +70 -0
- package/dist/agents/skills/codex/knowledge-craft/SKILL.md +180 -0
- package/dist/agents/skills/codex/knowledge-craft/skill.yaml +49 -0
- package/dist/agents/skills/codex/naming-craft/SKILL.md +172 -0
- package/dist/agents/skills/codex/naming-craft/skill.yaml +51 -0
- package/dist/agents/skills/codex/security-craft/SKILL.md +205 -0
- package/dist/agents/skills/codex/security-craft/skill.yaml +58 -0
- package/dist/agents/skills/codex/spec-craft/SKILL.md +184 -0
- package/dist/agents/skills/codex/spec-craft/skill.yaml +55 -0
- package/dist/agents/skills/codex/test-craft/SKILL.md +212 -0
- package/dist/agents/skills/codex/test-craft/skill.yaml +58 -0
- package/dist/agents/skills/cursor/align-design-system/SKILL.md +174 -0
- package/dist/agents/skills/cursor/align-design-system/skill.yaml +57 -0
- package/dist/agents/skills/cursor/audit-brand-compliance/SKILL.md +189 -0
- package/dist/agents/skills/cursor/audit-brand-compliance/skill.yaml +50 -0
- package/dist/agents/skills/cursor/audit-component-anatomy/SKILL.md +181 -0
- package/dist/agents/skills/cursor/audit-component-anatomy/skill.yaml +51 -0
- package/dist/agents/skills/cursor/copy-craft/SKILL.md +192 -0
- package/dist/agents/skills/cursor/copy-craft/skill.yaml +52 -0
- package/dist/agents/skills/cursor/detect-design-drift/SKILL.md +139 -0
- package/dist/agents/skills/cursor/detect-design-drift/skill.yaml +50 -0
- package/dist/agents/skills/cursor/harness-accessibility/SKILL.md +10 -0
- package/dist/agents/skills/cursor/harness-design-craft/SKILL.md +212 -0
- package/dist/agents/skills/cursor/harness-design-craft/skill.yaml +57 -0
- package/dist/agents/skills/cursor/harness-design-pipeline/SKILL.md +266 -0
- package/dist/agents/skills/cursor/harness-design-pipeline/skill.yaml +70 -0
- package/dist/agents/skills/cursor/knowledge-craft/SKILL.md +180 -0
- package/dist/agents/skills/cursor/knowledge-craft/skill.yaml +49 -0
- package/dist/agents/skills/cursor/naming-craft/SKILL.md +172 -0
- package/dist/agents/skills/cursor/naming-craft/skill.yaml +51 -0
- package/dist/agents/skills/cursor/security-craft/SKILL.md +205 -0
- package/dist/agents/skills/cursor/security-craft/skill.yaml +58 -0
- package/dist/agents/skills/cursor/spec-craft/SKILL.md +184 -0
- package/dist/agents/skills/cursor/spec-craft/skill.yaml +55 -0
- package/dist/agents/skills/cursor/test-craft/SKILL.md +212 -0
- package/dist/agents/skills/cursor/test-craft/skill.yaml +58 -0
- package/dist/agents/skills/gemini-cli/align-design-system/SKILL.md +174 -0
- package/dist/agents/skills/gemini-cli/align-design-system/skill.yaml +57 -0
- package/dist/agents/skills/gemini-cli/audit-brand-compliance/SKILL.md +189 -0
- package/dist/agents/skills/gemini-cli/audit-brand-compliance/skill.yaml +50 -0
- package/dist/agents/skills/gemini-cli/audit-component-anatomy/SKILL.md +181 -0
- package/dist/agents/skills/gemini-cli/audit-component-anatomy/skill.yaml +51 -0
- package/dist/agents/skills/gemini-cli/copy-craft/SKILL.md +192 -0
- package/dist/agents/skills/gemini-cli/copy-craft/skill.yaml +52 -0
- package/dist/agents/skills/gemini-cli/detect-design-drift/SKILL.md +139 -0
- package/dist/agents/skills/gemini-cli/detect-design-drift/skill.yaml +50 -0
- package/dist/agents/skills/gemini-cli/harness-accessibility/SKILL.md +10 -0
- package/dist/agents/skills/gemini-cli/harness-design-craft/SKILL.md +212 -0
- package/dist/agents/skills/gemini-cli/harness-design-craft/skill.yaml +57 -0
- package/dist/agents/skills/gemini-cli/harness-design-pipeline/SKILL.md +266 -0
- package/dist/agents/skills/gemini-cli/harness-design-pipeline/skill.yaml +70 -0
- package/dist/agents/skills/gemini-cli/knowledge-craft/SKILL.md +180 -0
- package/dist/agents/skills/gemini-cli/knowledge-craft/skill.yaml +49 -0
- package/dist/agents/skills/gemini-cli/naming-craft/SKILL.md +172 -0
- package/dist/agents/skills/gemini-cli/naming-craft/skill.yaml +51 -0
- package/dist/agents/skills/gemini-cli/security-craft/SKILL.md +205 -0
- package/dist/agents/skills/gemini-cli/security-craft/skill.yaml +58 -0
- package/dist/agents/skills/gemini-cli/spec-craft/SKILL.md +184 -0
- package/dist/agents/skills/gemini-cli/spec-craft/skill.yaml +55 -0
- package/dist/agents/skills/gemini-cli/test-craft/SKILL.md +212 -0
- package/dist/agents/skills/gemini-cli/test-craft/skill.yaml +58 -0
- package/dist/{agents-md-2JUQ4FE2.js → agents-md-267R2NUJ.js} +4 -4
- package/dist/{analyzer-VY6DJVOU-4AHIJLNS.js → analyzer-VY6DJVOU-WYKV3TTW.js} +2 -2
- package/dist/{architecture-CXAVNB5X.js → architecture-ZTEHGIHD.js} +6 -6
- package/dist/{assess-project-M626SSPN.js → assess-project-OKDW3YSK.js} +2 -1
- package/dist/bin/harness-mcp.js +17 -17
- package/dist/bin/harness.js +27 -27
- package/dist/{check-phase-gate-XMU6LLUQ.js → check-phase-gate-YJWFDW66.js} +5 -5
- package/dist/{chunk-372MGNTI.js → chunk-4EGPFB7V.js} +6185 -124
- package/dist/{chunk-HFOB2MPQ.js → chunk-67RP7MLR.js} +3 -3
- package/dist/{chunk-5JNSFYZU.js → chunk-6T5AYG5S.js} +6 -6
- package/dist/{chunk-5XLLIYYL.js → chunk-6UHDQP2N.js} +1 -1
- package/dist/{chunk-YP7I25SL.js → chunk-B3B64OPD.js} +1 -1
- package/dist/{chunk-VTTNIZJL.js → chunk-B4OLYXZ4.js} +93 -3
- package/dist/{chunk-6AX6R3LM.js → chunk-CXAHK5PJ.js} +9 -9
- package/dist/{chunk-BCFDGOMA.js → chunk-H2ZJOJ7A.js} +81 -1
- package/dist/{chunk-CQOMVZDC.js → chunk-HLPY3RHE.js} +1641 -292
- package/dist/{chunk-Q6ZR2L6Q.js → chunk-HMMSWRWR.js} +2 -2
- package/dist/{chunk-YYRQCQPZ.js → chunk-IY2TZLY5.js} +176 -107
- package/dist/{chunk-I6K43QQS.js → chunk-MQG7FHXX.js} +3 -3
- package/dist/{chunk-43M3RLFQ.js → chunk-OLTVDPA2.js} +2 -2
- package/dist/{chunk-JFZOWO7D.js → chunk-OQE6ME2Z.js} +1 -1
- package/dist/{chunk-GWE5LL3R.js → chunk-P7EO7USC.js} +10 -10
- package/dist/{chunk-24KCOJJG.js → chunk-PARV3G2F.js} +1 -1
- package/dist/{chunk-QOKMDDBJ.js → chunk-QUPC37UB.js} +6 -6
- package/dist/{chunk-EPUKTTJZ.js → chunk-RC3OI5NK.js} +5 -1
- package/dist/{chunk-L5BCZ5XS.js → chunk-S7GEONXL.js} +1 -1
- package/dist/{chunk-5FDBXUFW.js → chunk-TIH53QXY.js} +1 -1
- package/dist/{chunk-OA6SWTU4.js → chunk-TIZ3L6CU.js} +11 -8
- package/dist/{chunk-GWF2OILZ.js → chunk-VG6UK6G6.js} +1 -1
- package/dist/{chunk-SZEFU6XC.js → chunk-W3PDVZ4I.js} +1 -1
- package/dist/{chunk-VUCGB2KH.js → chunk-WOXR6I3R.js} +1 -1
- package/dist/{chunk-K246XQ2L.js → chunk-XTIPZUPR.js} +1 -1
- package/dist/{chunk-HMW3VKUF.js → chunk-Z2GUITAS.js} +10 -10
- package/dist/{chunk-PFZEADQM.js → chunk-ZKA3TGYS.js} +79 -2
- package/dist/{ci-workflow-3JNDZQYD.js → ci-workflow-YGTQTNVC.js} +4 -4
- package/dist/{dist-QPWXPCPL.js → dist-R3TOOPJ7.js} +1 -1
- package/dist/{dist-OHDQBTSO.js → dist-XIJWWQ5N.js} +3 -3
- package/dist/{docs-5RYMDHN3.js → docs-KCTLXK2G.js} +6 -6
- package/dist/{engine-GKGT5ULT.js → engine-J64WPH5K.js} +4 -4
- package/dist/{entropy-DQTAPSSE.js → entropy-M27JUL3V.js} +5 -5
- package/dist/{feedback-WH6XPQJS.js → feedback-W25QO3OL.js} +2 -2
- package/dist/{generate-agent-definitions-SZA4QIHN.js → generate-agent-definitions-RNMU5U5R.js} +5 -5
- package/dist/{glob-helper-XE2UGEOV.js → glob-helper-EX4YZKZO.js} +1 -1
- package/dist/{graph-loader-EQBOIHFZ.js → graph-loader-EIFEOUVI.js} +1 -1
- package/dist/index.d.ts +587 -6
- package/dist/index.js +27 -27
- package/dist/{loader-BNKGM23C.js → loader-HZKJ5ABV.js} +4 -4
- package/dist/{mcp-YKKPWUMH.js → mcp-NEAPIDTT.js} +17 -17
- package/dist/{performance-54TMJOEU.js → performance-3AVQUDRG.js} +6 -6
- package/dist/{review-pipeline-OTDDVPNY.js → review-pipeline-KA7SGUXJ.js} +4 -4
- package/dist/{runtime-W5THSNAL.js → runtime-IP6FVV5M.js} +4 -4
- package/dist/{scan-4PPP6ZXT.js → scan-LYKLM4EQ.js} +1 -1
- package/dist/{security-27HW7CYT.js → security-EAR4FJWI.js} +1 -1
- package/dist/{validate-QRVCD4GP.js → validate-EVH3UXIC.js} +5 -5
- package/dist/{validate-cross-check-BUAGBAPO.js → validate-cross-check-SNE3SQOB.js} +4 -4
- package/package.json +8 -7
|
@@ -0,0 +1,180 @@
|
|
|
1
|
+
# Knowledge Craft
|
|
2
|
+
|
|
3
|
+
> LLM-judgment critique of knowledge-entry quality. Critiques `docs/knowledge/` entries (EXCLUDING `decisions/` — that's spec-craft's territory) against a curated rubric catalog: does this state a load-bearing FACT or paraphrase the code? Would deleting it lose specific signal? Does it earn its place in the knowledge graph as `business_fact` / `business_rule` / `business_concept` / `business_decision`? Fifth non-design member of the craft-pipeline initiative (#9 of 10). Emits 3-axis findings (tier × impact × confidence per ADR 0019).
|
|
4
|
+
|
|
5
|
+
## When to Use
|
|
6
|
+
|
|
7
|
+
- During PR review on a new or substantially-rewritten knowledge entry
|
|
8
|
+
- After authoring a knowledge entry, before adding it to the index
|
|
9
|
+
- When onboarding a new contributor (audit entries they introduced)
|
|
10
|
+
- Periodically (per-sprint or per-release) to catch knowledge-entry rot
|
|
11
|
+
- As a quality gate before `harness-knowledge-pipeline` ingests an entry into the graph
|
|
12
|
+
- NOT for ADR / proposal critique (use `spec-craft` — `decisions/` is its territory)
|
|
13
|
+
- NOT for AGENTS.md critique (different shape: navigational manifest, not fact-bearing entry — v1.x)
|
|
14
|
+
- NOT for autofix / knowledge-entry rewriting (this is judgment-only; v1.x may add `align-knowledge` sibling)
|
|
15
|
+
- NOT for graph-membership checks (no graph reads at runtime — references the taxonomy in rubric prompts)
|
|
16
|
+
- NOT for source-code comment critique (use `code-craft` #4 or `docs-craft` #2)
|
|
17
|
+
|
|
18
|
+
## Process
|
|
19
|
+
|
|
20
|
+
### Phase 1: DISCOVER — Find knowledge entries
|
|
21
|
+
|
|
22
|
+
1. **Read project configuration.** Check `harness.config.json` for:
|
|
23
|
+
- `craft.knowledge.enabled` — gate (default `true`)
|
|
24
|
+
- `craft.knowledge.maxFiles` — entry count cap (default 50)
|
|
25
|
+
- `craft.knowledge.excludeDirs` — extra subdirs to skip
|
|
26
|
+
|
|
27
|
+
2. **Walk `docs/knowledge/` recursively:**
|
|
28
|
+
- Include `*.md` files (case-insensitive `README.md` excluded)
|
|
29
|
+
- EXCLUDE `decisions/` subdir entirely (spec-craft's territory)
|
|
30
|
+
- EXCLUDE any user-supplied extra dirs via `--exclude-dirs`
|
|
31
|
+
- Caller-supplied `--files` overrides discovery for explicit scoping
|
|
32
|
+
|
|
33
|
+
### Phase 2: CRITIQUE — Per (file, rubric) loop
|
|
34
|
+
|
|
35
|
+
7 seed rubrics:
|
|
36
|
+
|
|
37
|
+
| Rubric | Title |
|
|
38
|
+
| ----------- | ----------------------------------------------- |
|
|
39
|
+
| `KNOW-R001` | States a load-bearing fact (not paraphrase) |
|
|
40
|
+
| `KNOW-R002` | Truth a code reader could not derive |
|
|
41
|
+
| `KNOW-R003` | Earns a place in the knowledge graph taxonomy |
|
|
42
|
+
| `KNOW-R004` | Carries forward a decision that would erode |
|
|
43
|
+
| `KNOW-R005` | Deleting would lose specific knowledge |
|
|
44
|
+
| `KNOW-R006` | Concrete and operationally defined |
|
|
45
|
+
| `KNOW-R007` | A stranger could pick it up six months from now |
|
|
46
|
+
|
|
47
|
+
For each (file, rubric) pair:
|
|
48
|
+
|
|
49
|
+
1. Build prompt with rubric description + file path + relative-to-knowledge-root path + entry contents (truncated to 4000 chars for cost).
|
|
50
|
+
2. LLM returns fenced JSON: `null` (rubric doesn't apply / entry is fine) OR `{ tier, impact, confidence, message }`.
|
|
51
|
+
3. On non-null: emit a `KnowledgeFinding` with `cite.rubricId` populated for ADR 0020 traceability.
|
|
52
|
+
|
|
53
|
+
`KNOW-R003` is the rubric that references the graph taxonomy (`business_fact`, `business_rule`, `business_concept`, `business_decision`) inside its description — the LLM critiques against the taxonomy without knowledge-craft ever reading the graph.
|
|
54
|
+
|
|
55
|
+
### Phase 3: REPORT — Aggregate + cost telemetry
|
|
56
|
+
|
|
57
|
+
Emit `KnowledgeCraftOutput`:
|
|
58
|
+
|
|
59
|
+
```ts
|
|
60
|
+
{
|
|
61
|
+
findings: KnowledgeFinding[];
|
|
62
|
+
summary: {
|
|
63
|
+
phaseRun: ['critique'];
|
|
64
|
+
mode: 'fast';
|
|
65
|
+
durationMs: number;
|
|
66
|
+
llmCalls: { provider, model, count, costUsd };
|
|
67
|
+
catalog: { rubricsApplied: string[] };
|
|
68
|
+
counts: { filesScanned, filesSkipped };
|
|
69
|
+
runId: string;
|
|
70
|
+
}
|
|
71
|
+
}
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
## Harness Integration
|
|
75
|
+
|
|
76
|
+
- **`harness knowledge-craft`** — CLI entry. `--files <glob>` / `--exclude-dirs <dirs...>` / `--max-files <n>` / `--json` / `--verbose`.
|
|
77
|
+
- **`mcp__harness__knowledge_craft`** — MCP tool. Same input/output. Consumed by agents.
|
|
78
|
+
- **Cross-cutting API:** `critiqueKnowledgeFile(file, opts)` exported from `packages/cli/src/knowledge-craft/index.ts`. Future craft skills (or `harness-knowledge-pipeline`) can call this on a single entry without re-walking the project.
|
|
79
|
+
- **Shared craft infrastructure:** `LlmProvider`, `MockLlmProvider`, `derivePriority`, 3-axis types all live in `packages/cli/src/shared/craft/`.
|
|
80
|
+
|
|
81
|
+
## Success Criteria
|
|
82
|
+
|
|
83
|
+
See `docs/changes/craft-pipeline/knowledge-craft/proposal.md` for the full 25 success criteria. Highlights:
|
|
84
|
+
|
|
85
|
+
- 7 seed rubrics ship at `catalog/rubrics/<id>.ts` (file-per-rubric)
|
|
86
|
+
- 3-axis output preserved (tier × impact × confidence, never collapsed)
|
|
87
|
+
- `cite.rubricId` populated on every finding (ADR 0020)
|
|
88
|
+
- `decisions/` subdir is hard-excluded from discovery (spec-craft territory)
|
|
89
|
+
- `KNOW-R003` references graph node types in rubric prompt without graph imports at runtime
|
|
90
|
+
- `critiqueKnowledgeFile` cross-cutting API works on a single file without project walk
|
|
91
|
+
|
|
92
|
+
## Examples
|
|
93
|
+
|
|
94
|
+
### Example: Paraphrase entry
|
|
95
|
+
|
|
96
|
+
**Input:** `docs/knowledge/auth/email-validator.md`:
|
|
97
|
+
|
|
98
|
+
```
|
|
99
|
+
# Email Validator
|
|
100
|
+
|
|
101
|
+
The user service validates emails via the EmailValidator class, which
|
|
102
|
+
applies the standard regex pattern and rejects malformed addresses.
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
**Output (mock LLM):**
|
|
106
|
+
|
|
107
|
+
```
|
|
108
|
+
KNOW-R001 [foundational/large/medium] auth/email-validator.md
|
|
109
|
+
This entry restates what a reader would learn from opening
|
|
110
|
+
EmailValidator.ts. It states no load-bearing fact about the domain:
|
|
111
|
+
no upstream constraint, no historical reason for the choice, no business
|
|
112
|
+
rule that necessitated validation. Either rewrite to capture the WHY
|
|
113
|
+
(e.g., "emails must round-trip through Postmark within 30s for
|
|
114
|
+
deliverability tracking") or delete — the code already speaks.
|
|
115
|
+
KNOW-R005 [polish/medium/medium] auth/email-validator.md
|
|
116
|
+
Deleting this entry would lose nothing the code doesn't already convey.
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
### Example: Decision-bearing entry
|
|
120
|
+
|
|
121
|
+
**Input:** `docs/knowledge/storage/postgres-over-dynamo.md`:
|
|
122
|
+
|
|
123
|
+
```
|
|
124
|
+
# Why Postgres over DynamoDB
|
|
125
|
+
|
|
126
|
+
We chose Postgres over DynamoDB because our access patterns are
|
|
127
|
+
relational (frequent multi-table joins on tenant + user) and our team's
|
|
128
|
+
ops muscle is in SQL. DynamoDB's single-table design was rejected
|
|
129
|
+
because the modeling overhead of GSIs outweighs the latency win for
|
|
130
|
+
our request profile.
|
|
131
|
+
```
|
|
132
|
+
|
|
133
|
+
**Output:**
|
|
134
|
+
|
|
135
|
+
```
|
|
136
|
+
(no findings)
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
This entry carries forward a decision with the alternative AND the reason — KNOW-R004 passes, KNOW-R001 passes (load-bearing fact: the rejected option + the WHY), KNOW-R005 passes (deleting loses knowledge a reader couldn't reconstruct from the schema alone).
|
|
140
|
+
|
|
141
|
+
### Example: Empty project — no knowledge entries
|
|
142
|
+
|
|
143
|
+
**Input:** Project has no `docs/knowledge/` directory.
|
|
144
|
+
|
|
145
|
+
**Output:**
|
|
146
|
+
|
|
147
|
+
```
|
|
148
|
+
No knowledge-entry findings.
|
|
149
|
+
|
|
150
|
+
Summary: 0 findings across 0 entries (0 skipped, 7 rubrics, 0 LLM calls, $0.0000, 2ms)
|
|
151
|
+
```
|
|
152
|
+
|
|
153
|
+
## Gates
|
|
154
|
+
|
|
155
|
+
- **No autofix.** Sibling `align-knowledge` deferred until signal warrants safe-to-apply rewrites.
|
|
156
|
+
- **No ADR critique.** `decisions/` is spec-craft's territory; double-critique on the same files produces noise.
|
|
157
|
+
- **No AGENTS.md critique.** Navigational manifests need a different rubric vocabulary; v1.x.
|
|
158
|
+
- **No graph reads.** v1 references graph node types in rubric prompts so the LLM critiques against the taxonomy without a runtime graph dependency.
|
|
159
|
+
- **No graph persistence of findings.** Phase 1 MVP.
|
|
160
|
+
- **No per-section / per-claim mode.** v1 is per-file (knowledge entries are typically focused single-topic).
|
|
161
|
+
- **No `.mdx` support.** Different parsing concerns; v1.x.
|
|
162
|
+
- **No B' bootstrap.** Same posture as the rest of the craft family.
|
|
163
|
+
|
|
164
|
+
## Escalation
|
|
165
|
+
|
|
166
|
+
- **When LLM cost is too high:** drop `maxFiles` to 25, or scope explicitly with `--files`. Per-entry cost = rubrics × per-call; truncation already caps per-call cost at 4000 input chars.
|
|
167
|
+
- **When a rubric produces high false-positive rate:** v1 has no per-rubric disable; v1.x adds `craft.knowledge.disabledRubrics: ['KNOW-R007']`. Until then: filter findings by `cite.rubricId` in your consumer.
|
|
168
|
+
- **When an entry is intentionally scratchpad / in-progress:** low-confidence findings are de-emphasized per ADR 0019. v1.x adds per-entry opt-out via `<!-- knowledge-craft:skip -->` HTML comment.
|
|
169
|
+
- **When you want graph-aware critique (e.g., "this entry duplicates an existing business_fact node"):** v1.x opt-in mode will read the graph; v1 stays read-free.
|
|
170
|
+
- **When you want to critique an ADR:** use `harness spec-craft` — ADRs are its territory. Knowledge-craft will refuse to walk `decisions/`.
|
|
171
|
+
|
|
172
|
+
## Status
|
|
173
|
+
|
|
174
|
+
**v1 — in implementation.** See:
|
|
175
|
+
|
|
176
|
+
- Spec: `docs/changes/craft-pipeline/knowledge-craft/proposal.md`
|
|
177
|
+
- Roadmap entry: `craft-pipeline sub-project #9`
|
|
178
|
+
- Sibling craft skills: `naming-craft` (#1), `spec-craft` (#6), `copy-craft` (#5), `test-craft` (#3), `harness-design-craft` (design-pipeline #6)
|
|
179
|
+
- Shared infrastructure: `packages/cli/src/shared/craft/`
|
|
180
|
+
- Future: `align-knowledge` (FIX side), AGENTS.md / `.mdx` support, graph-aware mode, composition with `harness-knowledge-pipeline` at ingest time.
|
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
name: knowledge-craft
|
|
2
|
+
version: "0.1.0"
|
|
3
|
+
description: LLM-judgment critique of knowledge-entry quality (docs/knowledge/, excluding decisions/ which is spec-craft territory). Per-file critique against 7 seed rubrics that ask whether the entry states a load-bearing fact, earns a place in the graph taxonomy, carries forward a decision that would otherwise erode. Fifth non-design craft-pipeline ceiling skill.
|
|
4
|
+
stability: draft
|
|
5
|
+
cognitive_mode: constructive-architect
|
|
6
|
+
triggers:
|
|
7
|
+
- manual
|
|
8
|
+
- on_pr
|
|
9
|
+
- on_new_feature
|
|
10
|
+
platforms:
|
|
11
|
+
- claude-code
|
|
12
|
+
tools:
|
|
13
|
+
- Bash
|
|
14
|
+
- Read
|
|
15
|
+
- Glob
|
|
16
|
+
- Grep
|
|
17
|
+
cli:
|
|
18
|
+
command: harness knowledge-craft
|
|
19
|
+
args:
|
|
20
|
+
- name: path
|
|
21
|
+
description: Project root path
|
|
22
|
+
required: false
|
|
23
|
+
- name: files
|
|
24
|
+
description: Optional file scope (overrides discovery)
|
|
25
|
+
required: false
|
|
26
|
+
- name: exclude-dirs
|
|
27
|
+
description: Extra subdir names to skip under docs/knowledge/ (decisions is always excluded)
|
|
28
|
+
required: false
|
|
29
|
+
mcp:
|
|
30
|
+
tool: knowledge_craft
|
|
31
|
+
input:
|
|
32
|
+
path: string
|
|
33
|
+
type: rigid
|
|
34
|
+
tier: 2
|
|
35
|
+
phases:
|
|
36
|
+
- name: discover
|
|
37
|
+
description: Walk docs/knowledge/ recursively, exclude decisions/ + user-supplied dirs
|
|
38
|
+
required: true
|
|
39
|
+
- name: critique
|
|
40
|
+
description: LLM-rubric loop per (file, rubric) — 7 seed rubrics
|
|
41
|
+
required: true
|
|
42
|
+
- name: report
|
|
43
|
+
description: Aggregate findings + cost telemetry + entry counts
|
|
44
|
+
required: true
|
|
45
|
+
state:
|
|
46
|
+
persistent: false
|
|
47
|
+
depends_on:
|
|
48
|
+
- harness-soundness-review
|
|
49
|
+
- spec-craft
|
|
@@ -0,0 +1,172 @@
|
|
|
1
|
+
# Naming Craft
|
|
2
|
+
|
|
3
|
+
> LLM-judgment skill that critiques identifier names — variables, functions, types, and files — for clarity, concreteness, weight, and predictive power. First member of the craft-pipeline initiative (sub-project #1 of 10). Uses a curated rubric catalog seeded from Martin / Beck / Karlton. Emits 3-axis findings (tier × impact × confidence per ADR 0019).
|
|
4
|
+
|
|
5
|
+
## When to Use
|
|
6
|
+
|
|
7
|
+
- During PR review on code that adds or renames identifiers
|
|
8
|
+
- When onboarding a new contributor (audit names they introduced)
|
|
9
|
+
- When refactoring a module (verify renamed names earn their letters)
|
|
10
|
+
- As the cross-cutting naming critic for other craft skills (docs-craft, test-craft, code-craft will call into this)
|
|
11
|
+
- NOT for code-convention enforcement (use ESLint rules — this is ceiling, those are floor)
|
|
12
|
+
- NOT for autofix / rename codemod (this is judgment-only; the v2 sibling `align-naming` ships the fix path)
|
|
13
|
+
- NOT for module / branch / commit-subject naming (v1.x — different infrastructure)
|
|
14
|
+
- NOT for languages beyond TS/JS in v1 (Python/Go/Rust idiom catalogs are v1.x)
|
|
15
|
+
|
|
16
|
+
## Process
|
|
17
|
+
|
|
18
|
+
### Phase 1: EXTRACT — Identifier walk
|
|
19
|
+
|
|
20
|
+
1. **Read project configuration.** Check `harness.config.json` for:
|
|
21
|
+
- `craft.naming.enabled` — gate (default `true`)
|
|
22
|
+
- `craft.naming.maxFiles` — file count cap (default 100)
|
|
23
|
+
- `craft.naming.maxIdentifiersPerFile` — per-file sampling cap (default 15)
|
|
24
|
+
|
|
25
|
+
2. **Walk project files** (.ts / .tsx / .js / .jsx). Skip `node_modules`, `dist`, `build`, `coverage`, dotdirs.
|
|
26
|
+
|
|
27
|
+
3. **Extract identifiers per file** via TS Compiler API:
|
|
28
|
+
- **variable** — `const x =`, `let x =`, destructuring binders
|
|
29
|
+
- **function** — `function x()`, `const x = () =>`, class methods, arrow functions assigned to a name
|
|
30
|
+
- **type** — `type X`, `interface X`, `class X`
|
|
31
|
+
- For each: capture file, line, exported status, scope size (`short` = body ≤10 lines; `long` = otherwise), and ±2 context lines for LLM prompt construction.
|
|
32
|
+
|
|
33
|
+
### Phase 2: SAMPLE — Convention inference
|
|
34
|
+
|
|
35
|
+
For each identifier kind, sample up to N=500 identifiers across the project and infer the dominant convention via majority-rule:
|
|
36
|
+
|
|
37
|
+
- **variables / functions** — camelCase / snake_case / PascalCase
|
|
38
|
+
- **types** — PascalCase / camelCase
|
|
39
|
+
- **files** — kebab-case / camelCase / PascalCase (basenames sans extension)
|
|
40
|
+
|
|
41
|
+
`>50%` majority threshold per kind. Below threshold → `null` (no dominant convention) and the convention-conformance rubric silently skips.
|
|
42
|
+
|
|
43
|
+
### Phase 3: CRITIQUE — Per-rubric LLM loop
|
|
44
|
+
|
|
45
|
+
For each file:
|
|
46
|
+
|
|
47
|
+
1. **Sample identifiers** weighted by importance:
|
|
48
|
+
- Exported identifiers first
|
|
49
|
+
- Then long-scope (file-level, methods on long classes)
|
|
50
|
+
- Then short-scope random fill
|
|
51
|
+
- Cap at `maxIdentifiersPerFile` per file (default 15).
|
|
52
|
+
|
|
53
|
+
2. **For each (identifier, rubric)** in the cross-product:
|
|
54
|
+
- Build a prompt with rubric description + identifier + context lines + project convention.
|
|
55
|
+
- LLM returns fenced JSON: either `null` (rubric doesn't apply / name is fine) or `{ tier, impact, confidence, message }`.
|
|
56
|
+
- On non-null: emit a `NamingFinding` with `cite.rubricId` populated for ADR 0020 traceability.
|
|
57
|
+
|
|
58
|
+
3. **v1 rubric catalog (6 seed rubrics):**
|
|
59
|
+
- `NAME-R001` **predictive power** (Martin) — does the name predict the contract?
|
|
60
|
+
- `NAME-R002` **concreteness** (Martin / Beck) — concrete > vague
|
|
61
|
+
- `NAME-R003` **verb/noun honesty** (Beck) — verb for functions; noun for types; questions for booleans
|
|
62
|
+
- `NAME-R004` **convention conformance** (Karlton) — matches project convention
|
|
63
|
+
- `NAME-R005` **scope match** (Beck) — length proportional to scope
|
|
64
|
+
- `NAME-R006` **encoded measure** (Pragmatic Programmer) — silent units cause real bugs
|
|
65
|
+
|
|
66
|
+
### Phase 4: REPORT — Aggregate + cost telemetry
|
|
67
|
+
|
|
68
|
+
Emit `NamingCraftOutput`:
|
|
69
|
+
|
|
70
|
+
```ts
|
|
71
|
+
{
|
|
72
|
+
findings: NamingFinding[];
|
|
73
|
+
summary: {
|
|
74
|
+
phaseRun: ['critique'];
|
|
75
|
+
durationMs: number;
|
|
76
|
+
llmCalls: { provider, model, count, costUsd };
|
|
77
|
+
catalog: { rubricsApplied: string[] };
|
|
78
|
+
convention: { variables, functions, types, files };
|
|
79
|
+
runId: string;
|
|
80
|
+
}
|
|
81
|
+
}
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
## Harness Integration
|
|
85
|
+
|
|
86
|
+
- **`harness naming-craft`** — CLI entry. `--files <glob>` / `--kinds <variable|function|type|file>` / `--max-files <n>` / `--max-identifiers-per-file <n>` / `--json` / `--verbose`.
|
|
87
|
+
- **`mcp__harness__naming_craft`** — MCP tool. Same input/output. Consumed by agents.
|
|
88
|
+
- **Cross-cutting API:** `critiqueNamesInFile(file, opts)` exported from `packages/cli/src/naming-craft/index.ts`. Future craft skills (docs-craft, test-craft, code-craft) import and invoke this when they want naming critique on a file they're already processing — no project re-walk needed.
|
|
89
|
+
- **LLM provider reuse:** imports design-craft's `LlmProvider` + `MockLlmProvider` directly. v2 extracts to `packages/cli/src/shared/llm/` when a second non-design craft skill needs differences.
|
|
90
|
+
|
|
91
|
+
## Success Criteria
|
|
92
|
+
|
|
93
|
+
See `docs/changes/craft-pipeline/naming-craft/proposal.md` for the full 34 success criteria. Highlights:
|
|
94
|
+
|
|
95
|
+
- 6 seed rubrics ship in `catalog/rubrics/<id>.ts` (file-per-rubric matches design-craft pattern)
|
|
96
|
+
- 3-axis output preserved (tier × impact × confidence, never collapsed) per ADR 0019
|
|
97
|
+
- `cite.rubricId` populated on every finding per ADR 0020
|
|
98
|
+
- Convention sampler returns `null` when no dominant convention (>50% threshold)
|
|
99
|
+
- Cross-cutting `critiqueNamesInFile` API exported for future craft skills
|
|
100
|
+
- LlmProvider / MockLlmProvider IMPORTED from design-craft (no duplication)
|
|
101
|
+
- MCP tool count bumps (running total maintained by parallel PRs)
|
|
102
|
+
|
|
103
|
+
## Examples
|
|
104
|
+
|
|
105
|
+
### Example: Vague function name
|
|
106
|
+
|
|
107
|
+
**Input:** `src/orders/processor.ts`:
|
|
108
|
+
|
|
109
|
+
```ts
|
|
110
|
+
export function processData(orders: Order[]) { ... }
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
**Output (mock LLM):**
|
|
114
|
+
|
|
115
|
+
```
|
|
116
|
+
NAME-R002 [polish/medium/low] function processData:14
|
|
117
|
+
"processData" is a vague verb-pair where the operation and subject
|
|
118
|
+
are both unstated. Consider `applyDiscountsToOrders` or
|
|
119
|
+
`convertOrdersToInvoices` depending on the actual transform.
|
|
120
|
+
NAME-R001 [polish/medium/medium] function processData:14
|
|
121
|
+
The name predicts neither the input shape (orders) nor the operation.
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
(Real LLM responses vary; mock provider returns deterministic low-confidence findings for test determinism.)
|
|
125
|
+
|
|
126
|
+
### Example: Silent unit
|
|
127
|
+
|
|
128
|
+
**Input:**
|
|
129
|
+
|
|
130
|
+
```ts
|
|
131
|
+
const timeout = 5000;
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
**Output:**
|
|
135
|
+
|
|
136
|
+
```
|
|
137
|
+
NAME-R006 [foundational/medium/high] variable timeout:1
|
|
138
|
+
"timeout" implies a time measure but the unit is silent. Use
|
|
139
|
+
`timeoutMs` so the call site can't be misread as seconds.
|
|
140
|
+
```
|
|
141
|
+
|
|
142
|
+
### Example: Mixed-convention project — convention sampler returns null
|
|
143
|
+
|
|
144
|
+
**Input:** A project with 60% camelCase, 30% snake_case, 10% PascalCase variables. No >50% camelCase majority (60% IS >50%, so convention=camelCase). But with 45/40/15 split: no convention.
|
|
145
|
+
|
|
146
|
+
**Output:** convention-conformance rubric (NAME-R004) silently skips for the variables kind. Other rubrics still run.
|
|
147
|
+
|
|
148
|
+
## Gates
|
|
149
|
+
|
|
150
|
+
- **No autofix.** This is ceiling-judgment. v2's `align-naming` may add safe-rename codemods.
|
|
151
|
+
- **No NAMING.md authoring.** v1 derives convention from sampling.
|
|
152
|
+
- **No language support beyond TS/JS.** v1.x.
|
|
153
|
+
- **No modules / branches / commit subjects.** v1.x (and commit subjects go to copy-craft #5).
|
|
154
|
+
- **No graph persistence.** Phase 1 MVP posture (matches design-craft).
|
|
155
|
+
- **No deep/vision mode.** Naming is text-only.
|
|
156
|
+
|
|
157
|
+
## Escalation
|
|
158
|
+
|
|
159
|
+
- **When LLM cost is too high on a large project:** drop `maxIdentifiersPerFile` to 10 or `maxFiles` to 50. Cost = files × identifiers × rubrics × per-call cost.
|
|
160
|
+
- **When a rubric produces high false-positive rate:** v1 has no per-rubric disable; v1.x adds `craft.naming.disabledRubrics: ['NAME-R005']`. Until then: filter findings by `cite.rubricId` in your consumer.
|
|
161
|
+
- **When the convention sampler misidentifies a mid-migration project:** below 50% threshold returns null and convention rubric skips. Better silent skip than wrong findings. Wait until migration completes; until then disable NAME-R004 in v1.x or filter findings.
|
|
162
|
+
- **When you want naming critique for a single file (e.g. in CI on changed files):** use `--files <glob>` or call `critiqueNamesInFile()` via the cross-cutting API.
|
|
163
|
+
- **When you want module / branch / commit-subject naming today:** manual review. v1.x adds these surfaces.
|
|
164
|
+
|
|
165
|
+
## Status
|
|
166
|
+
|
|
167
|
+
**v1 — in implementation.** See:
|
|
168
|
+
|
|
169
|
+
- Spec: `docs/changes/craft-pipeline/naming-craft/proposal.md`
|
|
170
|
+
- Roadmap entry: `craft-pipeline sub-project #1` (the first member)
|
|
171
|
+
- Sibling: `harness-design-craft` (design-pipeline #6 — the LLM-judgment template this follows)
|
|
172
|
+
- Future cross-cutters: docs-craft (#2), test-craft (#3), code-craft (#4) will call into naming-craft's `critiqueNamesInFile()` for their domain-specific naming critique.
|
|
@@ -0,0 +1,51 @@
|
|
|
1
|
+
name: naming-craft
|
|
2
|
+
version: "0.1.0"
|
|
3
|
+
description: LLM-judgment skill that critiques identifier names (variables, functions, types, files) against a curated rubric catalog seeded from Martin / Beck / Karlton. First craft-pipeline ceiling skill; cross-cutting (other craft skills call into it for domain-specific naming).
|
|
4
|
+
stability: draft
|
|
5
|
+
cognitive_mode: constructive-architect
|
|
6
|
+
triggers:
|
|
7
|
+
- manual
|
|
8
|
+
- on_pr
|
|
9
|
+
- on_new_feature
|
|
10
|
+
platforms:
|
|
11
|
+
- claude-code
|
|
12
|
+
tools:
|
|
13
|
+
- Bash
|
|
14
|
+
- Read
|
|
15
|
+
- Glob
|
|
16
|
+
- Grep
|
|
17
|
+
cli:
|
|
18
|
+
command: harness naming-craft
|
|
19
|
+
args:
|
|
20
|
+
- name: path
|
|
21
|
+
description: Project root path
|
|
22
|
+
required: false
|
|
23
|
+
- name: files
|
|
24
|
+
description: Optional file/glob scope
|
|
25
|
+
required: false
|
|
26
|
+
- name: kinds
|
|
27
|
+
description: Restrict to variable / function / type / file
|
|
28
|
+
required: false
|
|
29
|
+
mcp:
|
|
30
|
+
tool: naming_craft
|
|
31
|
+
input:
|
|
32
|
+
path: string
|
|
33
|
+
type: rigid
|
|
34
|
+
tier: 2
|
|
35
|
+
phases:
|
|
36
|
+
- name: extract
|
|
37
|
+
description: TS Compiler API walk; gather identifiers + scope metadata + context lines
|
|
38
|
+
required: true
|
|
39
|
+
- name: sample
|
|
40
|
+
description: Derive project naming convention via majority-rule sampling
|
|
41
|
+
required: true
|
|
42
|
+
- name: critique
|
|
43
|
+
description: LLM-rubric loop per (identifier, rubric) pair; emit 3-axis findings
|
|
44
|
+
required: true
|
|
45
|
+
- name: report
|
|
46
|
+
description: Aggregate findings + LLM cost + convention summary
|
|
47
|
+
required: true
|
|
48
|
+
state:
|
|
49
|
+
persistent: false
|
|
50
|
+
depends_on:
|
|
51
|
+
- harness-design-craft
|
|
@@ -0,0 +1,205 @@
|
|
|
1
|
+
# Security Craft
|
|
2
|
+
|
|
3
|
+
> LLM-judgment critique of security posture for TS/JS source — the ceiling counterpart to `harness-security-scan` (CVE/OWASP rule-based floor) and `harness-security-reviewer` (procedural review). Threat-modeling-as-skill rather than pattern-matching. Critiques whether trust boundaries are respected, where implicit privilege escalation lurks, whether the code defends in depth or just at the gate, whether principle of least authority is honored. Sixth non-design member of the craft-pipeline initiative (#10 of 10; the final sub-project). Emits 3-axis findings (tier × impact × confidence per ADR 0019).
|
|
4
|
+
|
|
5
|
+
## When to Use
|
|
6
|
+
|
|
7
|
+
- During PR review on a substantively-changed handler / middleware / privileged op
|
|
8
|
+
- After authoring a new endpoint, before exposing it to traffic
|
|
9
|
+
- When onboarding a new contributor (audit security-relevant code they introduced)
|
|
10
|
+
- Periodically (per-sprint or per-release) to catch security-shape drift
|
|
11
|
+
- For threat-modeling shape questions a CVE scanner doesn't address (trust boundaries, fail-closed, authz ordering)
|
|
12
|
+
- NOT for CVE / dependency scanning (use `harness-security-scan` — rule-based floor)
|
|
13
|
+
- NOT for procedural review checklists (use `harness-security-reviewer`)
|
|
14
|
+
- NOT for IaC critique (v1.x — Dockerfile / k8s / Terraform have different rubric vocabulary)
|
|
15
|
+
- NOT for secret detection (floor concern; existing regex/entropy scanners cover this)
|
|
16
|
+
- NOT for autofix / security rewriting (this is judgment-only; v1.x may add `align-security` with aggressive safeguards)
|
|
17
|
+
- NOT for test files (v1 excludes — test security has a different shape; v1.x with dedicated rubrics)
|
|
18
|
+
|
|
19
|
+
## Process
|
|
20
|
+
|
|
21
|
+
### Phase 1: DISCOVER — Find source files
|
|
22
|
+
|
|
23
|
+
1. Walk `packages/*/src/` recursively.
|
|
24
|
+
2. Include `*.{ts,tsx,js,jsx,mjs,cjs}`; exclude test files (`*.test.*`, `*.spec.*`, `tests/`, `__tests__/`).
|
|
25
|
+
3. Exclude generated / build / coverage dirs (`node_modules`, `dist`, `build`, `coverage`, `.next`, `.turbo`, `__snapshots__`).
|
|
26
|
+
4. Honor `--packages` for explicit package scoping; `--files` overrides discovery.
|
|
27
|
+
|
|
28
|
+
### Phase 2: SIGNAL — AST-driven security construct detection
|
|
29
|
+
|
|
30
|
+
For each source file, the TS Compiler API walks the AST once and emits `SecuritySignal`s. Files with zero signals are **skipped entirely** — this is the FP-management strategy from the spec (no path-heuristic fallback).
|
|
31
|
+
|
|
32
|
+
Detected signal kinds:
|
|
33
|
+
|
|
34
|
+
| Kind | What it matches |
|
|
35
|
+
| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------- |
|
|
36
|
+
| `http-handler` | `(req, res)` / `(req, res, next)` shapes; `app.get/post/...`; `@Get/@Post/...` |
|
|
37
|
+
| `middleware` | `(req, res, next) =>` arrow / `(ctx, next)` shapes |
|
|
38
|
+
| `auth-api` | `jwt.{sign,verify}`, `bcrypt.{hash,compare}`, `argon2.*`, `passport.*`, `req.session.*`, `res.cookie` |
|
|
39
|
+
| `privileged-op` | `child_process.{exec,spawn,...}`, `eval`, `new Function`, `vm.runIn*`, `fs.{writeFile,unlink,chmod,...}` |
|
|
40
|
+
| `data-egress` | `fetch`, `axios.*`, `http.request`, `https.request`, `net.connect` |
|
|
41
|
+
| `raw-query` | `*.query(\`...${x}...\`)`, `*.raw(...)`, `$queryRaw`, `$executeRaw` with SQL-shaped argument |
|
|
42
|
+
| `secret-handling` | Secret-named variable (`token`, `password`, `apiKey`, …) flowing into `console.*`, `logger.*`, `JSON.stringify`, template-literal sink |
|
|
43
|
+
|
|
44
|
+
AST awareness (not regex) avoids common false positives: `exec` in a comment, `eval` as a variable name, `token` in a CSS property name.
|
|
45
|
+
|
|
46
|
+
### Phase 3: CRITIQUE — Per (file, signal, rubric) loop
|
|
47
|
+
|
|
48
|
+
8 seed rubrics, each declaring `appliesToSignals` so per-signal pre-filtering minimizes LLM cost:
|
|
49
|
+
|
|
50
|
+
| Rubric | Title | Applies to signals |
|
|
51
|
+
| ---------- | ----------------------------------------------- | --------------------------------------------------- |
|
|
52
|
+
| `SEC-R001` | Trust boundary respected | http-handler, middleware, raw-query, privileged-op |
|
|
53
|
+
| `SEC-R002` | Principle of least authority honored | auth-api, privileged-op, http-handler |
|
|
54
|
+
| `SEC-R003` | Defense in depth (not gate-only) | auth-api, http-handler |
|
|
55
|
+
| `SEC-R004` | Assumed adversary realistic for the deployment | http-handler, middleware, auth-api |
|
|
56
|
+
| `SEC-R005` | Data flow across trust boundaries is visible | http-handler, raw-query, data-egress, privileged-op |
|
|
57
|
+
| `SEC-R006` | Fail closed, not open | auth-api, middleware, http-handler |
|
|
58
|
+
| `SEC-R007` | Secrets carried in a shape that resists leakage | secret-handling |
|
|
59
|
+
| `SEC-R008` | Authorization check happens before the action | http-handler, privileged-op |
|
|
60
|
+
|
|
61
|
+
For each (signal, rubric) pair where the rubric applies:
|
|
62
|
+
|
|
63
|
+
1. Build a prompt with rubric description + file path + signal info + **1500-char window AROUND the signal line** (not the whole file — security-critical context is local).
|
|
64
|
+
2. **Conservative-confidence system prompt** biases the LLM toward `medium` confidence by default; `high` requires a specific, named anti-pattern or visible missing guard.
|
|
65
|
+
3. LLM returns fenced JSON: `null` (rubric doesn't apply / code is fine) OR `{ tier, impact, confidence, message }`.
|
|
66
|
+
4. On non-null: emit a `SecurityFinding` with `cite.rubricId` populated for ADR 0020 traceability.
|
|
67
|
+
|
|
68
|
+
### Phase 4: REPORT — Aggregate + cost telemetry
|
|
69
|
+
|
|
70
|
+
Emit `SecurityCraftOutput`:
|
|
71
|
+
|
|
72
|
+
```ts
|
|
73
|
+
{
|
|
74
|
+
findings: SecurityFinding[];
|
|
75
|
+
summary: {
|
|
76
|
+
phaseRun: ['critique'];
|
|
77
|
+
mode: 'fast';
|
|
78
|
+
durationMs: number;
|
|
79
|
+
llmCalls: { provider, model, count, costUsd };
|
|
80
|
+
catalog: { rubricsApplied: string[] };
|
|
81
|
+
counts: { filesScanned, filesSkippedNoSignal, signalsDetected };
|
|
82
|
+
runId: string;
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
`filesSkippedNoSignal` is tracked separately so report consumers can see how aggressively the AST pre-filter trimmed the corpus.
|
|
88
|
+
|
|
89
|
+
## Harness Integration
|
|
90
|
+
|
|
91
|
+
- **`harness security-craft`** — CLI entry. `--files <glob>` / `--packages <names>` / `--max-files <n>` / `--max-signals-per-file <n>` / `--json` / `--verbose`.
|
|
92
|
+
- **`mcp__harness__security_craft`** — MCP tool. Same input/output. Consumed by agents.
|
|
93
|
+
- **Cross-cutting API:** `critiqueSecurityInFile(file, opts)` exported from `packages/cli/src/security-craft/index.ts`. Returns `[]` for files with no security signals (consistent with the orchestrator's FP-management strategy).
|
|
94
|
+
- **Shared craft infrastructure:** `LlmProvider`, `MockLlmProvider`, `derivePriority`, 3-axis types all live in `packages/cli/src/shared/craft/`.
|
|
95
|
+
|
|
96
|
+
## Success Criteria
|
|
97
|
+
|
|
98
|
+
See `docs/changes/craft-pipeline/security-craft/proposal.md` for the full 29 success criteria. Highlights:
|
|
99
|
+
|
|
100
|
+
- 8 seed rubrics ship at `catalog/rubrics/<id>.ts` (file-per-rubric)
|
|
101
|
+
- AST detector emits signals for all 7 signal kinds; comment / string contents don't fire (AST-aware, not regex)
|
|
102
|
+
- Files with zero signals are skipped (`filesSkippedNoSignal` tracked)
|
|
103
|
+
- Per-rubric `appliesToSignals` pre-filter avoids irrelevant LLM calls
|
|
104
|
+
- 3-axis output preserved; confidence defaults to medium per the spec's Decision #3
|
|
105
|
+
- `cite.rubricId` populated on every finding (ADR 0020)
|
|
106
|
+
- `critiqueSecurityInFile` cross-cutting API works on a single file
|
|
107
|
+
|
|
108
|
+
## Examples
|
|
109
|
+
|
|
110
|
+
### Example: User input flowing into child_process
|
|
111
|
+
|
|
112
|
+
**Input:** `packages/api/src/handlers/run-script.ts`:
|
|
113
|
+
|
|
114
|
+
```ts
|
|
115
|
+
import { exec } from 'child_process';
|
|
116
|
+
import type { Request, Response } from 'express';
|
|
117
|
+
|
|
118
|
+
export function runScript(req: Request, res: Response): void {
|
|
119
|
+
const userScript = req.body.script;
|
|
120
|
+
exec(`bash -c "${userScript}"`, (err, stdout) => {
|
|
121
|
+
res.json({ output: stdout });
|
|
122
|
+
});
|
|
123
|
+
}
|
|
124
|
+
```
|
|
125
|
+
|
|
126
|
+
**Output (mock LLM):**
|
|
127
|
+
|
|
128
|
+
```
|
|
129
|
+
SEC-R001 [foundational/large/high] child_process.exec:5
|
|
130
|
+
User-controlled `req.body.script` flows directly into `bash -c "${userScript}"`.
|
|
131
|
+
This is a textbook command-injection sink. Either reject the entire pattern
|
|
132
|
+
(no user-supplied shell strings) or move to `execFile` with an allowlist of
|
|
133
|
+
binaries and pre-validated arg arrays. Never templated into a shell.
|
|
134
|
+
SEC-R005 [foundational/large/high] child_process.exec:5
|
|
135
|
+
Untrusted input (`req.body.script`) crosses the trust boundary into a
|
|
136
|
+
privileged sink without any visible validation or escaping step. The crossing
|
|
137
|
+
is invisible — the variable is named generically and goes straight to exec.
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
### Example: Auth check after action
|
|
141
|
+
|
|
142
|
+
**Input:** `packages/api/src/handlers/get-doc.ts`:
|
|
143
|
+
|
|
144
|
+
```ts
|
|
145
|
+
export async function getDoc(req: Request, res: Response) {
|
|
146
|
+
const doc = await db.docs.findOne({ id: req.params.id });
|
|
147
|
+
if (doc.ownerId !== req.user.id) return res.status(403).send();
|
|
148
|
+
return res.json(doc);
|
|
149
|
+
}
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
**Output:**
|
|
153
|
+
|
|
154
|
+
```
|
|
155
|
+
SEC-R008 [foundational/medium/medium] req,res:1
|
|
156
|
+
The document is loaded BEFORE the authorization check. Even though the
|
|
157
|
+
response is denied, the load has already executed — observable side effects
|
|
158
|
+
(audit logs, rate-limit counters, cache populations) leak existence
|
|
159
|
+
information about documents the caller can't access. Authorize against the
|
|
160
|
+
identifier first (`req.params.id` + `req.user.id`), then load.
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
### Example: File with no security signals
|
|
164
|
+
|
|
165
|
+
**Input:** A pure utility file with no http/auth/exec/fs/network constructs.
|
|
166
|
+
|
|
167
|
+
**Output:**
|
|
168
|
+
|
|
169
|
+
```
|
|
170
|
+
No security findings.
|
|
171
|
+
|
|
172
|
+
Summary: 0 findings across 0 files (12 skipped, 0 signals, 8 rubrics, 0 LLM calls, $0.0000, 4ms)
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
The 12 files were scanned for signals but skipped because none had security-relevant AST constructs — exactly the FP-management strategy at work.
|
|
176
|
+
|
|
177
|
+
## Gates
|
|
178
|
+
|
|
179
|
+
- **No autofix.** Sibling `align-security` deferred to v2 with aggressive FP safeguards (security rewrites have asymmetric downside).
|
|
180
|
+
- **No IaC critique.** Dockerfile / k8s / Terraform need different rubrics; v1.x.
|
|
181
|
+
- **No multi-file auth-flow tracing.** Cross-file privilege-escalation analysis (handler → middleware → service) needs a graph traversal layer; v1.x once cross-file critique pays for itself elsewhere.
|
|
182
|
+
- **No dependency / CVE scanning.** `harness-security-scan` is the floor.
|
|
183
|
+
- **No secret detection** (floor concern).
|
|
184
|
+
- **No test-file critique.** Test security has a different shape; v1.x.
|
|
185
|
+
- **No path-heuristic fallback.** If AST scan finds zero signals, the file is skipped. Tight scoping is part of the FP-management strategy.
|
|
186
|
+
- **No B' bootstrap.**
|
|
187
|
+
|
|
188
|
+
## Escalation
|
|
189
|
+
|
|
190
|
+
- **When LLM cost is too high:** drop `maxFiles` to 50 or `maxSignalsPerFile` to 5, or scope explicitly with `--packages <name>`. Per-file cost = (signals × applicable rubrics × per-call); typical handler fires ~3-5 rubrics, not 8.
|
|
191
|
+
- **When a specific rubric produces false positives:** v1 has no per-rubric disable; v1.x adds `craft.security.disabledRubrics: ['SEC-R004']`. Until then: filter findings by `cite.rubricId` downstream.
|
|
192
|
+
- **When the AST detector misses a framework you use** (tRPC, Convex, Cloudflare Workers, Hono RPC): v1 ships baseline coverage for Express / Hono / Fastify / Koa / NestJS-decorator. Adding a framework is a 1-line config in `signals.ts`. Track as v1.x.
|
|
193
|
+
- **When findings are too cautious (confidence floor):** the conservative-by-default is deliberate (FP management); v1.x adds `craft.security.confidenceFloor` to tighten further. Loosening below medium is intentionally not exposed.
|
|
194
|
+
- **When you want IaC critique:** v1.x. For v1, scope explicitly to source files and accept the gap.
|
|
195
|
+
- **When a finding is wrong:** dismiss it in your consumer; signal as a `suppressedAt` entry on the rubric for future catalog evolution.
|
|
196
|
+
|
|
197
|
+
## Status
|
|
198
|
+
|
|
199
|
+
**v1 — in implementation.** See:
|
|
200
|
+
|
|
201
|
+
- Spec: `docs/changes/craft-pipeline/security-craft/proposal.md`
|
|
202
|
+
- Roadmap entry: `craft-pipeline sub-project #10` (the final sub-project; the craft-pipeline initiative completes with this PR)
|
|
203
|
+
- Sibling craft skills: `naming-craft` (#1), `spec-craft` (#6), `copy-craft` (#5), `test-craft` (#3), `knowledge-craft` (#9), `harness-design-craft` (design-pipeline #6)
|
|
204
|
+
- Shared infrastructure: `packages/cli/src/shared/craft/`
|
|
205
|
+
- Future: `align-security` (FIX side; aggressive safeguards), IaC critique, multi-file auth-flow tracing, test-file security, framework expansions (tRPC / Convex / Cloudflare Workers / Hono RPC).
|