@waratahlabs/canopy 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +244 -195
- package/SKILL.md +13 -2
- package/demo/data/insaider-threat-model.tc.json +7800 -0
- package/demo/data/openclaw-demo.tc.json +80 -303
- package/demo/data/openclaw-nemoclaw-demo.tc.json +114 -389
- package/dist/aivss/scoring/heuristic-scorer.d.ts +1 -1
- package/dist/aivss/scoring/heuristic-scorer.js +1 -1
- package/dist/cli.js +24 -37
- package/dist/cli.js.map +1 -1
- package/dist/examples/insaider-assessment.d.ts +28 -0
- package/dist/examples/insaider-assessment.d.ts.map +1 -0
- package/dist/examples/insaider-assessment.js +301 -0
- package/dist/examples/insaider-assessment.js.map +1 -0
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +0 -9
- package/dist/index.js.map +1 -1
- package/dist/output/tc-assembler.d.ts +0 -10
- package/dist/output/tc-assembler.d.ts.map +1 -1
- package/dist/output/tc-assembler.js +4 -42
- package/dist/output/tc-assembler.js.map +1 -1
- package/dist/references/container-taxonomy.d.ts +41 -0
- package/dist/references/container-taxonomy.d.ts.map +1 -0
- package/dist/references/container-taxonomy.js +207 -0
- package/dist/references/container-taxonomy.js.map +1 -0
- package/dist/references/index.d.ts +0 -7
- package/dist/references/index.d.ts.map +1 -1
- package/dist/references/index.js +0 -13
- package/dist/references/index.js.map +1 -1
- package/dist/references/nist-ai-rmf.d.ts +1 -6
- package/dist/references/nist-ai-rmf.d.ts.map +1 -1
- package/dist/references/nist-ai-rmf.js +1 -10
- package/dist/references/nist-ai-rmf.js.map +1 -1
- package/dist/references/owasp-agentic-top10.d.ts +1 -6
- package/dist/references/owasp-agentic-top10.d.ts.map +1 -1
- package/dist/references/owasp-agentic-top10.js +0 -9
- package/dist/references/owasp-agentic-top10.js.map +1 -1
- package/package.json +4 -18
- package/demo/index.html +0 -556
- package/dist/aivss/scoring/a2a-corpus.test.d.ts +0 -28
- package/dist/aivss/scoring/a2a-corpus.test.d.ts.map +0 -1
- package/dist/aivss/scoring/a2a-corpus.test.js +0 -109
- package/dist/aivss/scoring/a2a-corpus.test.js.map +0 -1
- package/dist/aivss/scoring/determinism.test.d.ts +0 -16
- package/dist/aivss/scoring/determinism.test.d.ts.map +0 -1
- package/dist/aivss/scoring/determinism.test.js +0 -169
- package/dist/aivss/scoring/determinism.test.js.map +0 -1
- package/dist/aivss/scoring/scoring-mode.d.ts +0 -48
- package/dist/aivss/scoring/scoring-mode.d.ts.map +0 -1
- package/dist/aivss/scoring/scoring-mode.js +0 -64
- package/dist/aivss/scoring/scoring-mode.js.map +0 -1
- package/dist/references/registry-parity.test.d.ts +0 -17
- package/dist/references/registry-parity.test.d.ts.map +0 -1
- package/dist/references/registry-parity.test.js +0 -87
- package/dist/references/registry-parity.test.js.map +0 -1
package/README.md
CHANGED
|
@@ -1,153 +1,39 @@
|
|
|
1
|
-
# Canopy
|
|
1
|
+
# Canopy — `@waratahlabs/canopy`
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Multi-framework threat modelling for agentic AI systems. Point the library at an agent definition, an N8N workflow, or a live OpenClaw install. It assesses the system across OWASP AIVSS V4 (9 AI-specific metrics, 39 sub-categories), maps findings to MITRE ATLAS adversary TTPs, cross-references ASD/ACSC Five Eyes government guidance, and writes structured threat statements in AWS Threat Composer format (`.tc.json`).
|
|
4
4
|
|
|
5
|
-
|
|
5
|
+
## How the analysis works
|
|
6
6
|
|
|
7
|
-
|
|
7
|
+

|
|
8
8
|
|
|
9
|
-
|
|
10
|
-
npm install -g @waratahlabs/canopy
|
|
11
|
-
```
|
|
12
|
-
|
|
13
|
-
Or run it without installing:
|
|
9
|
+
The pipeline diagram is generated from [`docs/architecture.c4`](docs/architecture.c4) — a live LikeC4 model of the library itself. To explore it interactively or export updated PNGs:
|
|
14
10
|
|
|
15
11
|
```bash
|
|
16
|
-
|
|
12
|
+
bunx likec4 serve docs/architecture.c4
|
|
13
|
+
bunx likec4 export png -o docs/diagrams/ docs/
|
|
17
14
|
```
|
|
18
15
|
|
|
19
|
-
|
|
16
|
+
**Scoring layer** — the pluggable `BatchProvider` routes metric assessments to whichever inference backend you have available:
|
|
20
17
|
|
|
21
|
-
|
|
18
|
+

|
|
22
19
|
|
|
23
|
-
|
|
20
|
+
## Quick start
|
|
24
21
|
|
|
25
22
|
```bash
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
|
|
29
|
-
This prints the `AgentDefinition` JSON schema plus a worked example. One entry per agent or autonomous process in your system.
|
|
30
|
-
|
|
31
|
-
### 2. Describe your system
|
|
32
|
-
|
|
33
|
-
Create `my-agents.json`. Describe what your agent actually is — what model, what tools, what it's allowed to touch:
|
|
34
|
-
|
|
35
|
-
```json
|
|
36
|
-
{
|
|
37
|
-
"applicationName": "Sales Data Pipeline",
|
|
38
|
-
"agents": [
|
|
39
|
-
{
|
|
40
|
-
"id": "reporting-agent-01",
|
|
41
|
-
"name": "reporting-agent",
|
|
42
|
-
"model": "claude-sonnet-4-6",
|
|
43
|
-
"systemPrompt": "Retrieve sales data via the reporting API. Fall back to direct DB query only if the API is unavailable.",
|
|
44
|
-
"tools": [
|
|
45
|
-
{ "name": "call_reporting_api", "description": "Fetch sales data via the authorised HTTP API.", "dangerous": false },
|
|
46
|
-
{ "name": "query_database", "description": "Execute SQL directly against the sales database.", "dangerous": true }
|
|
47
|
-
],
|
|
48
|
-
"permissions": ["read:reporting_api", "read:postgres_salesdb", "write:audit_log"],
|
|
49
|
-
"deploymentContext": "FinancialServices",
|
|
50
|
-
"modelComplexity": "Simple",
|
|
51
|
-
"mitigationMultiplier": 1.0,
|
|
52
|
-
"finetuned": false,
|
|
53
|
-
"dependencies": ["reporting-api:8080", "postgres:5432"],
|
|
54
|
-
"secretManagement": "Environment variables, no rotation.",
|
|
55
|
-
"deployment": "Docker container",
|
|
56
|
-
"autonomyLevel": "semi-autonomous",
|
|
57
|
-
"objectives": ["Retrieve monthly sales data"],
|
|
58
|
-
"externalApiCount": 1
|
|
59
|
-
}
|
|
60
|
-
],
|
|
61
|
-
"handoffs": []
|
|
62
|
-
}
|
|
63
|
-
```
|
|
23
|
+
bun install
|
|
24
|
+
bun run build
|
|
64
25
|
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
```bash
|
|
70
|
-
npx @waratahlabs/canopy assess --input my-agents.json --deterministic --out threat-model.tc.json
|
|
71
|
-
```
|
|
26
|
+
# Run the demo (writes output/demo-threat-model.tc.json)
|
|
27
|
+
# Falls back to mock severity if no provider credentials are set
|
|
28
|
+
bun run threat-model
|
|
72
29
|
|
|
30
|
+
# Serve the web demo
|
|
31
|
+
bun run demo # http://localhost:3000
|
|
73
32
|
```
|
|
74
|
-
✓ 35 threats · 29 mitigations → threat-model.tc.json
|
|
75
|
-
```
|
|
76
|
-
|
|
77
|
-
No API key. No account. No network call. Open the result at [threat-composer.awslabs.dev](https://awslabs.github.io/threat-composer/).
|
|
78
|
-
|
|
79
|
-
### 4. Gate it in CI
|
|
80
|
-
|
|
81
|
-
```bash
|
|
82
|
-
canopy assess --input my-agents.json --deterministic --diff
|
|
83
|
-
```
|
|
84
|
-
|
|
85
|
-
Saves a baseline and diffs against the previous run. **Exits non-zero when severities regress**, so a change that makes your agent riskier fails the build.
|
|
86
|
-
|
|
87
|
-
---
|
|
88
|
-
|
|
89
|
-
## Running inside a coding agent
|
|
90
|
-
|
|
91
|
-
Canopy can use the LLM you're already talking to as its inference engine — no API key, no subprocess, no extra model call.
|
|
92
|
-
|
|
93
|
-
```bash
|
|
94
|
-
canopy assess --input my-agents.json --harness --out threat-model.tc.json
|
|
95
|
-
```
|
|
96
|
-
|
|
97
|
-
The first run writes the assessment prompts to `canopy-batch-requests.json` and exits with instructions. Your coding agent (Claude Code, Cursor, or anything driving a shell) reads that file, answers each prompt with its own reasoning, writes `canopy-batch-responses.json`, and you re-run. Two runs, one round trip, zero network calls Canopy has to authenticate.
|
|
98
|
-
|
|
99
|
-
`SKILL.md` in this package is a ready-made skill file for that workflow — point your agent at it.
|
|
100
|
-
|
|
101
|
-
---
|
|
102
|
-
|
|
103
|
-
## Scoring modes — know where your number came from
|
|
104
|
-
|
|
105
|
-
A score you cannot reproduce is not evidence. Canopy always records which mode produced a finding, stamped on every threat as `scoring-mode` and in the artifact's audit trail.
|
|
106
|
-
|
|
107
|
-
| Mode | Flag | Reproducible | Credentials | Use it for |
|
|
108
|
-
|---|---|---|---|---|
|
|
109
|
-
| **Deterministic** | `--deterministic` | **Yes** — byte-identical across runs and machines | None | CI gating, audit trails, anything you have to defend |
|
|
110
|
-
| **Harness** | `--harness` | No — depends on the driving model | None | Exploring a new system interactively |
|
|
111
|
-
| **Model** | *(default with `ANTHROPIC_API_KEY`)* | No — varies with model version | Provider key | Richer reasoning on threat prose |
|
|
112
33
|
|
|
113
|
-
|
|
34
|
+
Open `output/demo-threat-model.tc.json` in [AWS Threat Composer](https://github.com/awslabs/threat-composer).
|
|
114
35
|
|
|
115
|
-
|
|
116
|
-
|
|
117
|
-
---
|
|
118
|
-
|
|
119
|
-
## Registered frameworks
|
|
120
|
-
|
|
121
|
-
Assess once, map many — one assessment engine, findings annotated against every registered framework:
|
|
122
|
-
|
|
123
|
-
| Framework | Risks | Type |
|
|
124
|
-
|---|---|---|
|
|
125
|
-
| ASD/ACSC — Careful Adoption of Agentic AI Services (Five Eyes, 2025) | 15 | security |
|
|
126
|
-
| CSA AI Controls Matrix (AICM) | 32 | compliance |
|
|
127
|
-
| DSTG — A Method for Ethical AI in Defence (MEAID) | 21 | ethical / compliance |
|
|
128
|
-
| NIST AI Risk Management Framework 1.0 | 52 | compliance |
|
|
129
|
-
| OWASP Top 10 for Agentic Applications 2026 | 10 | security |
|
|
130
|
-
| NeMo Claw — Security Wrapper for OpenClaw | 5 | security |
|
|
131
|
-
|
|
132
|
-
Adding a framework is one file plus one registration line — see [CONTRIBUTING.md](CONTRIBUTING.md). This table is asserted against the live registry by `src/references/registry-parity.test.ts`: a framework advertised here but not registered fails the build.
|
|
133
|
-
|
|
134
|
-
---
|
|
135
|
-
|
|
136
|
-
## What you get
|
|
137
|
-
|
|
138
|
-
Every threat in the `.tc.json` carries:
|
|
139
|
-
|
|
140
|
-
- **Threat Composer slots** — `threatSource`, `prerequisites`, `threatAction`, `threatImpact`, `impactedGoal[]`, `impactedAssets[]`, `statement`
|
|
141
|
-
- **AIVSS metadata** — metric code (MR/DS/EI/DC/AD/AA/LL/GV/CS), sub-category, severity, score
|
|
142
|
-
- **ATLAS metadata** — technique ID (e.g. `AML.T0051`), technique name, tactic
|
|
143
|
-
- **Framework annotations** — risk ID, category, and risk type from each registered framework
|
|
144
|
-
- **Provenance** — `scoring-mode`, deployment context, and version stamps for Canopy, AIVSS, and both ATLAS version fields
|
|
145
|
-
|
|
146
|
-
Agent-to-agent handoffs are surfaced as first-class threats (`AML.T0080.000` — implicit trust grants), because the trust boundary between two agents is where the interesting failures live.
|
|
147
|
-
|
|
148
|
-
---
|
|
149
|
-
|
|
150
|
-
## Library usage
|
|
36
|
+
## Usage
|
|
151
37
|
|
|
152
38
|
### Assess agent definitions
|
|
153
39
|
|
|
@@ -180,35 +66,31 @@ const tcFormat = await assessWorkflow({
|
|
|
180
66
|
input: { type: "agents", agents },
|
|
181
67
|
applicationName: "Mortgage Platform",
|
|
182
68
|
});
|
|
183
|
-
```
|
|
184
69
|
|
|
185
|
-
|
|
70
|
+
await Bun.write("output/threat-model.tc.json", JSON.stringify(tcFormat, null, 2));
|
|
71
|
+
```
|
|
186
72
|
|
|
187
|
-
|
|
73
|
+
### Assess an N8N workflow
|
|
188
74
|
|
|
189
75
|
```typescript
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
|
|
193
|
-
// A2A agent cards — fetches /.well-known/agent.json and walks the delegation graph
|
|
194
|
-
import { walkA2AGraph } from "@waratahlabs/canopy";
|
|
195
|
-
const { agents, handoffs } = await walkA2AGraph({ urls: ["https://agent.example.com"], maxHops: 2 });
|
|
76
|
+
import { assessWorkflow } from "@waratahlabs/canopy";
|
|
77
|
+
import n8nWorkflow from "./my-workflow.json";
|
|
196
78
|
|
|
197
|
-
|
|
198
|
-
|
|
199
|
-
|
|
79
|
+
const tcFormat = await assessWorkflow({
|
|
80
|
+
input: { type: "n8n", workflow: n8nWorkflow },
|
|
81
|
+
applicationName: "My Agent Workflow",
|
|
82
|
+
});
|
|
200
83
|
```
|
|
201
84
|
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
### Langfuse traces
|
|
85
|
+
### Assess Langfuse traces (Tier 4 — runtime telemetry)
|
|
205
86
|
|
|
206
|
-
|
|
87
|
+
Fetch live agent traces from a Langfuse instance and assess them against AIVSS V4. Because traces don't carry system prompts, tool declarations, or permissions, you supply those via a config overlay — a YAML file that maps agent name patterns to the fields traces can't provide.
|
|
207
88
|
|
|
208
89
|
```typescript
|
|
209
90
|
import { assessWorkflow, parseLangfuseTraces, loadOverlaysFromYaml } from "@waratahlabs/canopy";
|
|
210
91
|
|
|
211
92
|
const overlays = loadOverlaysFromYaml("./langfuse-overlay.yaml");
|
|
93
|
+
|
|
212
94
|
const { agents, handoffs } = await parseLangfuseTraces(
|
|
213
95
|
{
|
|
214
96
|
host: process.env.LANGFUSE_HOST!,
|
|
@@ -218,44 +100,114 @@ const { agents, handoffs } = await parseLangfuseTraces(
|
|
|
218
100
|
},
|
|
219
101
|
overlays,
|
|
220
102
|
);
|
|
103
|
+
|
|
104
|
+
const tcFormat = await assessWorkflow({
|
|
105
|
+
input: { type: "agents", agents, handoffs },
|
|
106
|
+
applicationName: "My Agent System",
|
|
107
|
+
});
|
|
221
108
|
```
|
|
222
109
|
|
|
223
|
-
|
|
110
|
+
See [`src/examples/langfuse-overlay.example.yaml`](src/examples/langfuse-overlay.example.yaml) for a complete overlay template. The `agentNamePattern` field accepts a plain string (exact or substring match) or a `/regex/flags` literal.
|
|
111
|
+
|
|
112
|
+
No `langfuse` SDK required — the adapter uses `fetch` with HTTP Basic auth directly.
|
|
113
|
+
|
|
114
|
+
### Sample frontend (dev tool)
|
|
224
115
|
|
|
225
|
-
|
|
116
|
+
`sample-fe/` is a Bun server + Vite React app that lets you point a browser at your Langfuse instance and see per-agent AIVSS scores in a local tab. It exists so you can iterate quickly on overlay configuration and verify assessments are landing correctly.
|
|
117
|
+
|
|
118
|
+
> **It is not a deployment artefact.** Canopy is designed to integrate into existing observability stacks — Langfuse, Arize Phoenix, Datadog, or anything else — via the same adapter pattern, producing OWASP AIVSS-grounded findings wherever you already collect traces. The sample frontend is a debugging aid for validating that integration; your integration is the product.
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
cd sample-fe
|
|
122
|
+
bun install
|
|
123
|
+
cp .env.example .env # fill in Langfuse host + keys
|
|
124
|
+
bun run dev # server on :3001, Vite on :5173
|
|
125
|
+
```
|
|
226
126
|
|
|
227
|
-
|
|
127
|
+
See [`sample-fe/README.md`](sample-fe/README.md) for setup details.
|
|
228
128
|
|
|
229
|
-
|
|
129
|
+
### Assess a live OpenClaw installation
|
|
230
130
|
|
|
231
131
|
```typescript
|
|
232
|
-
import { assessWorkflow,
|
|
132
|
+
import { assessWorkflow, parseOpenClawConfig } from "@waratahlabs/canopy";
|
|
233
133
|
|
|
234
|
-
//
|
|
235
|
-
|
|
236
|
-
|
|
237
|
-
|
|
134
|
+
// Reads ~/.openclaw/openclaw.json and workspace/skills/
|
|
135
|
+
const { agents, handoffs, undeclaredSkills } = parseOpenClawConfig();
|
|
136
|
+
|
|
137
|
+
if (undeclaredSkills.length > 0) {
|
|
138
|
+
console.warn("Skills with unknown capabilities:", undeclaredSkills);
|
|
139
|
+
}
|
|
140
|
+
|
|
141
|
+
const tcFormat = await assessWorkflow({
|
|
142
|
+
input: { type: "agents", agents, handoffs },
|
|
143
|
+
applicationName: "My OpenClaw Assistant",
|
|
238
144
|
});
|
|
145
|
+
```
|
|
146
|
+
|
|
147
|
+
Config path resolution order: explicit option → `OPENCLAW_CONFIG_PATH` env → `OPENCLAW_STATE_DIR/openclaw.json` → `~/.openclaw/openclaw.json` → `~/.clawdbot/clawdbot.json`.
|
|
148
|
+
|
|
149
|
+
## Batch providers — real severity scoring
|
|
150
|
+
|
|
151
|
+
By default Canopy mocks every sub-category severity as Medium. Wire in a `BatchProvider` for real AI-assessed severity.
|
|
152
|
+
|
|
153
|
+
### Amazon Bedrock Batch Inference
|
|
154
|
+
|
|
155
|
+
```typescript
|
|
156
|
+
import { assessWorkflow, createBatchProvider } from "@waratahlabs/canopy";
|
|
239
157
|
|
|
240
|
-
|
|
241
|
-
await assessWorkflow({
|
|
158
|
+
const tcFormat = await assessWorkflow({
|
|
242
159
|
input: { type: "agents", agents },
|
|
243
160
|
batchProvider: createBatchProvider({
|
|
244
161
|
type: "bedrock",
|
|
245
162
|
region: "us-east-1",
|
|
163
|
+
// modelId defaults to "us.amazon.nova-2-lite-v1:0"
|
|
246
164
|
s3InputUri: "s3://my-bucket/canopy-input/",
|
|
247
165
|
s3OutputUri: "s3://my-bucket/canopy-output/",
|
|
248
166
|
}),
|
|
167
|
+
onBatchProgress: (status) => console.log(status),
|
|
249
168
|
});
|
|
169
|
+
```
|
|
250
170
|
|
|
251
|
-
|
|
252
|
-
|
|
171
|
+
### Anthropic Messages Batches
|
|
172
|
+
|
|
173
|
+
```typescript
|
|
174
|
+
const tcFormat = await assessWorkflow({
|
|
253
175
|
input: { type: "agents", agents },
|
|
254
|
-
batchProvider: createBatchProvider({
|
|
176
|
+
batchProvider: createBatchProvider({
|
|
177
|
+
type: "anthropic",
|
|
178
|
+
// apiKey defaults to ANTHROPIC_API_KEY env var
|
|
179
|
+
model: "claude-haiku-4-5-20251001",
|
|
180
|
+
}),
|
|
255
181
|
});
|
|
256
182
|
```
|
|
257
183
|
|
|
258
|
-
|
|
184
|
+
### Ollama (local or cloud)
|
|
185
|
+
|
|
186
|
+
```typescript
|
|
187
|
+
// Local
|
|
188
|
+
const tcFormat = await assessWorkflow({
|
|
189
|
+
input: { type: "agents", agents },
|
|
190
|
+
batchProvider: createBatchProvider({
|
|
191
|
+
type: "ollama",
|
|
192
|
+
model: "llama3.2",
|
|
193
|
+
// baseUrl defaults to http://localhost:11434
|
|
194
|
+
concurrency: 2,
|
|
195
|
+
}),
|
|
196
|
+
});
|
|
197
|
+
|
|
198
|
+
// Ollama cloud (ollama.com)
|
|
199
|
+
const tcFormat = await assessWorkflow({
|
|
200
|
+
input: { type: "agents", agents },
|
|
201
|
+
batchProvider: createBatchProvider({
|
|
202
|
+
type: "ollama",
|
|
203
|
+
model: "llama3.3:70b",
|
|
204
|
+
baseUrl: "https://ollama.com/api",
|
|
205
|
+
apiKey: process.env.OLLAMA_API_KEY,
|
|
206
|
+
}),
|
|
207
|
+
});
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
Custom providers implement one interface:
|
|
259
211
|
|
|
260
212
|
```typescript
|
|
261
213
|
import type { BatchProvider } from "@waratahlabs/canopy";
|
|
@@ -268,55 +220,152 @@ const myProvider: BatchProvider = {
|
|
|
268
220
|
};
|
|
269
221
|
```
|
|
270
222
|
|
|
271
|
-
|
|
223
|
+
## Model evaluation
|
|
224
|
+
|
|
225
|
+
`runModelEval()` runs N Ollama models against the same agent set and produces a comparison report — useful for deciding which open-source models generate meaningful AIVSS assessments before committing to a full batch run.
|
|
226
|
+
|
|
227
|
+
```typescript
|
|
228
|
+
import { runModelEval, createBatchProvider } from "@waratahlabs/canopy";
|
|
229
|
+
|
|
230
|
+
const report = await runModelEval({
|
|
231
|
+
agents,
|
|
232
|
+
models: [
|
|
233
|
+
createBatchProvider({ type: "ollama", model: "llama3.2" }),
|
|
234
|
+
createBatchProvider({ type: "ollama", model: "qwen2.5:14b" }),
|
|
235
|
+
createBatchProvider({ type: "ollama", model: "mistral-nemo" }),
|
|
236
|
+
],
|
|
237
|
+
});
|
|
238
|
+
|
|
239
|
+
console.log(report.summary);
|
|
240
|
+
// → "qwen2.5:14b: recommended (87% parse success, 0.76 pairwise agreement)"
|
|
241
|
+
// → "llama3.2: conditional (61% parse success)"
|
|
242
|
+
// → "mistral-nemo: not-recommended (low finding coverage on AA/AD metrics)"
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
Report includes: per-model parse success rate, finding coverage by metric, mean severity, pairwise agreement matrix, per-metric outlier detection, and written verdict with reasons.
|
|
272
246
|
|
|
273
|
-
|
|
247
|
+
## Web demo
|
|
274
248
|
|
|
275
|
-
|
|
249
|
+
```bash
|
|
250
|
+
bun run demo # http://localhost:3000
|
|
251
|
+
```
|
|
252
|
+
|
|
253
|
+
The demo explorer loads a pre-generated OpenClaw threat model with a comparison toggle for OpenClaw + NeMo Claw. Toggle between them to see per-threat severity deltas, NeMo Claw control annotations (what each isolation layer actually addresses), and a Jarkas layer breakdown of what the kernel-level sandbox handles — and what it doesn't.
|
|
254
|
+
|
|
255
|
+
NeMo Claw is NVIDIA's kernel-level security sandbox for OpenClaw (Landlock + seccomp + network namespaces). The comparison illustrates the Jarkas taxonomy in action:
|
|
256
|
+
|
|
257
|
+
| Jarkas layer | NeMo Claw impact |
|
|
258
|
+
|---|---|
|
|
259
|
+
| L4 Host | Largely mitigated — host FS read-only, `openclaw.json` inaccessible |
|
|
260
|
+
| L3 Engine | Largely mitigated — seccomp blocks dangerous syscalls |
|
|
261
|
+
| L2 Application | Partially mitigated — egress allowlist + Privacy Router reduce blast radius |
|
|
262
|
+
| L1 Orchestration | **Unchanged** — cron persistence and prompt injection operate inside the sandbox |
|
|
263
|
+
|
|
264
|
+
## Output format
|
|
265
|
+
|
|
266
|
+
Every threat in the `.tc.json` carries:
|
|
267
|
+
|
|
268
|
+
- **Threat Composer slots**: `threatSource`, `prerequisites`, `threatAction`, `threatImpact`, `impactedGoal[]`, `impactedAssets[]`, `statement`
|
|
269
|
+
- **AIVSS metadata**: metric code (MR/DS/EI/DC/AD/AA/LL/GV/CS), sub-category, severity
|
|
270
|
+
- **ATLAS metadata**: technique ID (e.g. `AML.T0051`), technique name, tactic
|
|
271
|
+
- **ASD/ACSC metadata**: risk family and risk ID from the Five Eyes joint advisory
|
|
272
|
+
- **Jarkas layer**: container security layer implicated (when applicable)
|
|
273
|
+
- **Deployment context**: AIVSS weight profile used for scoring
|
|
274
|
+
- **Version stamps**: `canopy-version`, `aivss-version`, `atlas-version`, `acsc-version`, `generated-at`
|
|
275
|
+
|
|
276
|
+
Agent-to-agent handoffs are automatically surfaced as first-class threats (`AML.T0080.000` — implicit trust grants).
|
|
277
|
+
|
|
278
|
+
## Project structure
|
|
276
279
|
|
|
277
280
|
```
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
|
|
281
|
-
|
|
282
|
-
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
|
|
286
|
-
|
|
287
|
-
|
|
288
|
-
|
|
289
|
-
|
|
290
|
-
|
|
291
|
-
|
|
292
|
-
|
|
281
|
+
src/
|
|
282
|
+
├── aivss/
|
|
283
|
+
│ ├── metrics/ — 9 metric prompt builders (MR DS EI DC AD AA LL GV CS)
|
|
284
|
+
│ ├── scoring/ — V4 formula, 7 deployment-context weight profiles, unit tests
|
|
285
|
+
│ ├── atlas/ — ATLAS loader, sub-category coverage matrix (39 entries)
|
|
286
|
+
│ ├── batch/ — Prompt → batch request builders
|
|
287
|
+
│ └── prompts/ — Base system prompt, prompt composer
|
|
288
|
+
├── batch/
|
|
289
|
+
│ ├── providers/ — BedrockBatchProvider, AnthropicBatchProvider, OllamaBatchProvider
|
|
290
|
+
│ │ └── index.ts — createBatchProvider() factory
|
|
291
|
+
│ ├── bedrock-batch-runner.ts
|
|
292
|
+
│ └── response-parser.ts
|
|
293
|
+
├── adapters/
|
|
294
|
+
│ ├── n8n.ts — N8N workflow → AgentDefinition[] + HandoffEdge[]
|
|
295
|
+
│ └── openclaw.ts — OpenClaw config → AgentDefinition[] + HandoffEdge[]
|
|
296
|
+
├── providers/ — InferenceProvider (slot filling): Bedrock, Anthropic
|
|
297
|
+
├── output/
|
|
298
|
+
│ ├── slot-filler.ts — fillThreatSlots() — per-threat AI inference
|
|
299
|
+
│ ├── tc-assembler.ts — assembleTCFormat() — DataExchangeFormat builder
|
|
300
|
+
│ ├── tc-types.ts — DataExchangeFormat type definitions
|
|
301
|
+
│ └── likec4-generator.ts — Architecture diagram generator
|
|
302
|
+
├── references/ — Framework adapters: ACSC, Jarkas, MEAID, NIST AI RMF,
|
|
303
|
+
│ │ OWASP Agentic Top 10, NeMo Claw
|
|
304
|
+
│ └── registry.ts — FrameworkAdapter / FrameworkRisk interfaces
|
|
305
|
+
├── tools/
|
|
306
|
+
│ └── model-eval.ts — runModelEval() — cross-model comparison
|
|
307
|
+
└── index.ts — assessWorkflow() public entry point
|
|
308
|
+
|
|
309
|
+
vendor/
|
|
310
|
+
├── owasp-aivss/ — OWASP AIVSS V4 reference (cloned)
|
|
311
|
+
├── atlas-data/ — MITRE ATLAS submodule (provenance/license; dist/ATLAS.yaml is deprecated format)
|
|
312
|
+
└── atlas-release/ — MITRE ATLAS format-6 data (ATLAS-2026.07.yaml, downloaded release asset — see its README)
|
|
313
|
+
|
|
314
|
+
demo/
|
|
315
|
+
├── index.html — Static threat model explorer (baseline + NeMo Claw comparison)
|
|
316
|
+
└── data/
|
|
317
|
+
├── openclaw-demo.tc.json — Pre-generated OpenClaw threat model
|
|
318
|
+
└── openclaw-nemoclaw-demo.tc.json — NeMo Claw comparison with severity deltas
|
|
293
319
|
```
|
|
294
320
|
|
|
295
|
-
|
|
321
|
+
## References
|
|
296
322
|
|
|
297
|
-
|
|
323
|
+
### OWASP AIVSS v0.8
|
|
324
|
+
Scoring methodology, 9 AI-specific metrics, 39 sub-categories, V4 formula, 7 deployment-context weight profiles.
|
|
325
|
+
- [aivss.owasp.org](https://aivss.owasp.org)
|
|
298
326
|
|
|
299
|
-
|
|
327
|
+
### MITRE ATLAS™ (format-6.0.0, content 2026.07)
|
|
328
|
+
Adversary tactics, techniques, and mitigations for ML systems. ATLAS techniques are injected into metric prompts as test specifications — each threat is grounded in documented attack patterns. 16 tactics, 178 techniques (101 parent + 77 sub-techniques), 37 mitigations, 68 case studies. MITRE ships monthly calendar-versioned content releases (`YYYY.MM.N`) decoupled from rarer semver schema (`format-version`) bumps.
|
|
329
|
+
- [atlas.mitre.org](https://atlas.mitre.org)
|
|
300
330
|
|
|
301
|
-
|
|
331
|
+
### ASD/ACSC: Careful Adoption of Agentic AI Services (2025)
|
|
332
|
+
Joint advisory from ASD/ACSC, CISA, NSA, Cyber Centre (CA), NCSC-NZ, and NCSC-UK. Five risk families cross-referenced to AIVSS metrics and ATLAS techniques.
|
|
333
|
+
- [cyber.gov.au](https://www.cyber.gov.au/business-government/secure-design/artificial-intelligence/careful-adoption-of-agentic-ai-services)
|
|
302
334
|
|
|
303
|
-
|
|
335
|
+
### CSA AI Controls Matrix (AICM) v1.1.1 (2026)
|
|
336
|
+
Cloud Security Alliance's AI-specific control extension to the Cloud Controls Matrix — 247 controls across 18 domains. Canopy maps the 31 controls CSA tags "AI-Specific" (agent boundaries, sandboxing, data poisoning, model hardening, explainability, human oversight) to AIVSS metrics and ATLAS techniques; the remaining 216 "Cloud & AI Related"/"Cloud-Specific" controls are general cloud-security hygiene outside this adapter's scope.
|
|
337
|
+
- [cloudsecurityalliance.org/artifacts/ai-controls-matrix](https://cloudsecurityalliance.org/artifacts/ai-controls-matrix)
|
|
338
|
+
- Machine-readable bundle (JSON/YAML/OSCAL, generated 2026-07-22) — [cloudsecurityalliance.org/artifacts/aicm-machine-readable-bundle-json-yaml-oscal](https://cloudsecurityalliance.org/artifacts/aicm-machine-readable-bundle-json-yaml-oscal)
|
|
304
339
|
|
|
305
|
-
|
|
340
|
+
### Jarkas et al. — Container Security Taxonomy (2025)
|
|
341
|
+
200+ container CVEs classified into 47 exploit types across 5 architectural layers (L1–L5). Applied when `deployment` indicates a container runtime.
|
|
342
|
+
- Jarkas, O., Ko, R. K. L., Dong, N., and Mahmud, R. *A Container Security Survey.* ACM Computing Surveys, vol. 57, no. 7, 2025. https://doi.org/10.1145/3715001
|
|
306
343
|
|
|
307
|
-
|
|
344
|
+
### NeMo Claw (alpha-2026)
|
|
345
|
+
NVIDIA's kernel-level security sandbox for OpenClaw. Canopy documents residual risks (L1/L2 Jarkas layers) and mitigated vectors (L3/L4) for baseline vs. sandboxed comparison.
|
|
346
|
+
- [secondtalent.com — NeMo Claw vs OpenClaw](https://www.secondtalent.com/resources/nemoclaw-vs-openclaw/)
|
|
308
347
|
|
|
309
|
-
|
|
348
|
+
### LikeC4 — Architecture as Code
|
|
349
|
+
Generated alongside every threat model — agents, tools, data stores, handoff relationships, dangerous tool highlighting.
|
|
350
|
+
```bash
|
|
351
|
+
npx likec4 serve output/my-system.c4
|
|
352
|
+
npx likec4 export png -o docs/diagrams/ output/my-system.c4
|
|
353
|
+
```
|
|
354
|
+
- [likec4.dev](https://likec4.dev)
|
|
310
355
|
|
|
311
|
-
|
|
356
|
+
### AWS Threat Composer
|
|
357
|
+
Output format. `.tc.json` files open directly in Threat Composer for collaborative modelling.
|
|
358
|
+
- [github.com/awslabs/threat-composer](https://github.com/awslabs/threat-composer)
|
|
312
359
|
|
|
313
|
-
|
|
360
|
+
## Contributing — versioning and changelogs
|
|
314
361
|
|
|
315
|
-
|
|
362
|
+
This repo uses [Changesets](https://github.com/changesets/changesets) for version bumps and `CHANGELOG.md`. Every PR that changes the package needs a changeset:
|
|
316
363
|
|
|
317
|
-
|
|
364
|
+
```bash
|
|
365
|
+
bunx changeset # describe your change and pick a bump type (patch/minor/major)
|
|
366
|
+
```
|
|
318
367
|
|
|
319
|
-
|
|
368
|
+
`changeset-check.yml` fails the PR if one is missing (`bunx changeset add --empty` for changes that don't need a release, e.g. docs-only). Merging to `main` opens/updates a "Version Packages" PR (`changesets-release.yml`) that bumps `package.json` and writes `CHANGELOG.md`; merging *that* PR auto-tags the new version, which fires `publish.yml` (test → build → publish) — no manual step required.
|
|
320
369
|
|
|
321
370
|
## License
|
|
322
371
|
|
package/SKILL.md
CHANGED
|
@@ -43,12 +43,12 @@ Read the agent source files (system prompts, tool definitions, environment confi
|
|
|
43
43
|
```bash
|
|
44
44
|
bunx @waratahlabs/canopy assess \
|
|
45
45
|
--input canopy-input.json \
|
|
46
|
-
--
|
|
46
|
+
--no-provider \
|
|
47
47
|
--diagram \
|
|
48
48
|
--out threat-model.tc.json
|
|
49
49
|
```
|
|
50
50
|
|
|
51
|
-
`--
|
|
51
|
+
`--no-provider` skips a separate LLM API call — the calling agent (you) is the intelligence. The AIVSS scoring and ATLAS technique mapping are deterministic; only the threat statement prose uses stub fills. The structural threat model is complete and accurate regardless.
|
|
52
52
|
|
|
53
53
|
### Step 4 — augment with your own reasoning (optional but recommended)
|
|
54
54
|
|
|
@@ -57,6 +57,7 @@ Read the generated `threat-model.tc.json` alongside the reference examples in `d
|
|
|
57
57
|
```
|
|
58
58
|
demo/data/openclaw-demo.tc.json — personal AI assistant threat model
|
|
59
59
|
demo/data/openclaw-nemoclaw-demo.tc.json — sandboxed vs baseline comparison
|
|
60
|
+
demo/data/insaider-threat-model.tc.json — insider threat / credential abuse scenario
|
|
60
61
|
```
|
|
61
62
|
|
|
62
63
|
Use these as few-shot references to understand threat statement quality and depth. Then augment the generated output with your own analysis:
|
|
@@ -92,6 +93,16 @@ Each threat is annotated with:
|
|
|
92
93
|
- `dstg-meaid-risk-id` — DSTG MEAID ethical/compliance risk (where applicable)
|
|
93
94
|
- `agent-name`, `deployment-context`, `priority`
|
|
94
95
|
|
|
96
|
+
## Worked example — insAIder threat workshop
|
|
97
|
+
|
|
98
|
+
The insAIder workshop has two agents: `operative-agent` (LangChain ReAct, `qwen3.5:4b`, direct DB access) and `supervisor-agent` (autonomous Redis pub/sub escalation trigger). The key threat is the supervisor injecting authorization-expansion language into the operative's task queue, causing it to use `query_database` instead of the authorised API path.
|
|
99
|
+
|
|
100
|
+
Input highlights:
|
|
101
|
+
- `operative.tools[query_database].dangerous = true` — drives MR and AA findings
|
|
102
|
+
- `operative.secretManagement = "env vars, no rotation"` — drives DS findings
|
|
103
|
+
- `handoffs: [{ from: supervisor, to: operative }]` — generates AML.T0080.000 context-injection threat
|
|
104
|
+
- `deploymentContext = "FinancialServices"` — applies the FinancialServices AIVSS weight profile
|
|
105
|
+
|
|
95
106
|
## Adding frameworks
|
|
96
107
|
|
|
97
108
|
New risk frameworks are added by contributing to `src/references/` — one file per framework implementing `FrameworkAdapter`, registered in `src/references/index.ts`. No changes to the CLI or assembler required.
|