agentverify 1.2.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +28 -0
- package/README.md +47 -4
- package/dist/chunk-XVA2M2HS.mjs +131 -0
- package/dist/cli.js +247 -23
- package/dist/cli.mjs +190 -22
- package/dist/sdk.d.mts +117 -9
- package/dist/sdk.d.ts +117 -9
- package/dist/sdk.js +59 -3
- package/dist/sdk.mjs +1 -1
- package/package.json +4 -3
- package/dist/chunk-YOD7H7MC.mjs +0 -75
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,28 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to the `agentverify` CLI/SDK package are documented here.
|
|
4
|
+
|
|
5
|
+
## [1.4.0] - Unreleased
|
|
6
|
+
|
|
7
|
+
### Changed — CI exit codes now distinguish a security failure from an execution error (behavior change)
|
|
8
|
+
|
|
9
|
+
Previously, `--ci` mode used exit code `1` for **both** a real `NOT_VERIFIED` verdict **and** any execution error (missing/invalid API key, network failure, missing file). That made it impossible for a pipeline to tell "the agent failed the security check" apart from "the scan itself couldn't run."
|
|
10
|
+
|
|
11
|
+
New exit codes:
|
|
12
|
+
|
|
13
|
+
| Code | Meaning | Previously |
|
|
14
|
+
| --- | --- | --- |
|
|
15
|
+
| `0` | All scanned files verified | `0` (unchanged) |
|
|
16
|
+
| `1` | `NOT_VERIFIED` — a real security/verification failure | `1` (unchanged) |
|
|
17
|
+
| `2` | `NOT_ASSESSED` — insufficient evidence for a verdict | `2` (unchanged) |
|
|
18
|
+
| `3` | **New.** Execution error — bad/missing API key, network failure, missing file, unknown command | was `1` |
|
|
19
|
+
|
|
20
|
+
**If your pipeline only checks `exit code != 0` to block a deployment, this is not a breaking change** — every previously-non-zero case is still non-zero. **If your pipeline branches on the exact code `1`**, an execution error will now surface as `3` instead — update any such logic to treat both `1` and `3` as failures, or to react differently to `3` (an infrastructure problem, not a security finding).
|
|
21
|
+
|
|
22
|
+
### Fixed
|
|
23
|
+
|
|
24
|
+
- Scanner: findings that reference a detected credential no longer include the live secret value in `evidence` — only a redacted preview (e.g. `api_key: "sk_l****" (redacted)`). Affects report evidence produced by any scan, including those made public via report sharing.
|
|
25
|
+
|
|
26
|
+
## [1.3.0] - 2026-07-13
|
|
27
|
+
|
|
28
|
+
Initial public release. See git history for prior changes.
|
package/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
**Execution trust analysis for AI agents — CLI and SDK**
|
|
4
4
|
|
|
5
|
-
Scan AI agent configurations for security issues, credential exposure, and governance gaps.
|
|
5
|
+
Scan AI agent configurations for security issues, credential exposure, and governance gaps. Saved CLI reports appear in your Agent Verify dashboard when API report save succeeds.
|
|
6
6
|
|
|
7
7
|
## Install
|
|
8
8
|
|
|
@@ -37,6 +37,31 @@ agentverify scan ./agents
|
|
|
37
37
|
agentverify scan . --key av_your_key --json
|
|
38
38
|
```
|
|
39
39
|
|
|
40
|
+
### Markdown summary
|
|
41
|
+
|
|
42
|
+
```bash
|
|
43
|
+
agentverify scan ./agents --markdown
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
### CI mode
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
agentverify scan ./agents --ci
|
|
50
|
+
```
|
|
51
|
+
|
|
52
|
+
Normal scans print findings without failing the shell just because issues exist. CI mode is stricter: it exits non-zero when an agent is `NOT_VERIFIED`, when a scan errors, or when submitted content cannot be assessed.
|
|
53
|
+
|
|
54
|
+
Exit codes distinguish a security/verification result from an application error, so a pipeline can tell "the agent failed the check" apart from "the scan itself couldn't run":
|
|
55
|
+
|
|
56
|
+
| Code | Meaning |
|
|
57
|
+
| --- | --- |
|
|
58
|
+
| `0` | All scanned files verified |
|
|
59
|
+
| `1` | `NOT_VERIFIED` — a security/verification check failed |
|
|
60
|
+
| `2` | `NOT_ASSESSED` — insufficient evidence for a verdict (see `--allow-not-assessed`) |
|
|
61
|
+
| `3` | Execution error — the scan itself could not run (bad key, network failure, missing file) |
|
|
62
|
+
|
|
63
|
+
All non-zero codes still fail a pipeline step by default, so an existing `if [ $? -ne 0 ]`-style check keeps working unchanged.
|
|
64
|
+
|
|
40
65
|
## SDK Usage
|
|
41
66
|
|
|
42
67
|
```ts
|
|
@@ -51,7 +76,7 @@ const result = await scan(
|
|
|
51
76
|
{ apiKey: process.env.AGENTVERIFY_API_KEY }
|
|
52
77
|
)
|
|
53
78
|
|
|
54
|
-
result.verdict // 'VERIFIED' | '
|
|
79
|
+
result.verdict // 'VERIFIED' | 'NOT_VERIFIED' | 'NOT_ASSESSED'
|
|
55
80
|
result.riskScore // 0-100
|
|
56
81
|
result.findings // Array of security findings
|
|
57
82
|
```
|
|
@@ -73,10 +98,28 @@ const results = await scanMany(
|
|
|
73
98
|
### GitHub Actions
|
|
74
99
|
|
|
75
100
|
```yaml
|
|
76
|
-
- name:
|
|
77
|
-
run: npx agentverify scan ./agents --
|
|
101
|
+
- name: Agent Verify
|
|
102
|
+
run: npx agentverify scan ./agents --ci
|
|
103
|
+
env:
|
|
104
|
+
AGENTVERIFY_API_KEY: ${{ secrets.AGENTVERIFY_API_KEY }}
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
Use this in pull requests to block merges when an agent is `NOT_VERIFIED`. The CLI prints score, verdict, risk level, top blocker, fix-first guidance, threat categories, A2SPA docs, and the canonical report URL:
|
|
108
|
+
|
|
109
|
+
```text
|
|
110
|
+
https://aimodularity.com/agentverify/report/?id=REPORT_ID
|
|
78
111
|
```
|
|
79
112
|
|
|
113
|
+
For PR comments, run with `--markdown` and post the output with your preferred GitHub Action. For machine-readable pipelines, run with `--json`.
|
|
114
|
+
|
|
115
|
+
The workflow requires a valid `AGENTVERIFY_API_KEY`. The local Worker code path fails closed for missing, malformed, invalid, disabled, or revoked keys before findings are returned; validate the deployed API path before relying on this as production enforcement. Wire the workflow into branch protection so the non-zero `--ci` exit code blocks merges when an agent is `NOT_VERIFIED`, not assessed, or the API key is unauthorized.
|
|
116
|
+
|
|
117
|
+
Saved CLI reports sync to the dashboard through the API `cliReports` save path. If the API cannot save the report, the CLI still prints local scan output and tells you dashboard sync did not complete.
|
|
118
|
+
|
|
119
|
+
Verified reports can expose a README badge from the report page. A packaged Agent Verify GitHub Action (`AI-Blockchain-Ventures/agentverify@v1`) already exists and produces text/Markdown/JSON output plus an optional workflow artifact — see below. Automatically posting that output as a PR comment (rather than just producing it for you to post) is still a planned enhancement, not yet built.
|
|
120
|
+
|
|
121
|
+
See the full GitHub Action guide at `docs/github-action.md` for composite action inputs, JSON/Markdown examples, branch protection, optional PR comments, and badge guidance.
|
|
122
|
+
|
|
80
123
|
## Get your API key
|
|
81
124
|
|
|
82
125
|
Sign up at https://aimodularity.com/agentverify/dashboard and go to Dashboard -> API
|
|
@@ -0,0 +1,131 @@
|
|
|
1
|
+
// src/sdk.ts
|
|
2
|
+
var DEFAULT_API_URL = "https://agentverify-api.agentverify.workers.dev/v1/scan";
|
|
3
|
+
var DEFAULT_TIMEOUT = 3e4;
|
|
4
|
+
function normalizeVerdict(value) {
|
|
5
|
+
if (value === "VERIFIED") return "VERIFIED";
|
|
6
|
+
if (value === "NOT_ASSESSED" || value === "NOT ASSESSED") return "NOT_ASSESSED";
|
|
7
|
+
return "NOT_VERIFIED";
|
|
8
|
+
}
|
|
9
|
+
function normalizeScanResult(data) {
|
|
10
|
+
const findings = Array.isArray(data.findings) ? data.findings : [];
|
|
11
|
+
const riskScore = typeof data.riskScore === "number" ? data.riskScore : 0;
|
|
12
|
+
return {
|
|
13
|
+
...data,
|
|
14
|
+
reportId: typeof data.reportId === "string" ? data.reportId : "",
|
|
15
|
+
verdict: normalizeVerdict(data.verdict),
|
|
16
|
+
riskScore,
|
|
17
|
+
riskLevel: data.riskLevel ?? "High Risk",
|
|
18
|
+
confidence: typeof data.confidence === "number" ? data.confidence : riskScore > 0 ? 80 : 0,
|
|
19
|
+
optimizationScore: typeof data.optimizationScore === "number" ? data.optimizationScore : 0,
|
|
20
|
+
reportInsights: data.reportInsights,
|
|
21
|
+
threatCategories: Array.isArray(data.threatCategories) ? data.threatCategories : [],
|
|
22
|
+
findings,
|
|
23
|
+
categoryScores: Array.isArray(data.categoryScores) ? data.categoryScores : [],
|
|
24
|
+
bom: data.bom ?? {
|
|
25
|
+
detectedLanguage: "Unknown",
|
|
26
|
+
detectedFramework: null,
|
|
27
|
+
detectedPlatform: null,
|
|
28
|
+
agentName: null,
|
|
29
|
+
toolAccessLevel: "Unknown",
|
|
30
|
+
credentialExposure: "Not Detected",
|
|
31
|
+
memoryPersistence: "Unknown",
|
|
32
|
+
auditLogging: "Unknown",
|
|
33
|
+
humanGates: "Unknown",
|
|
34
|
+
rateLimiting: "Unknown",
|
|
35
|
+
promptInjectionSurface: "Unknown",
|
|
36
|
+
delegationScope: "Unknown"
|
|
37
|
+
},
|
|
38
|
+
metadata: data.metadata ?? {
|
|
39
|
+
schemaVersion: "1.3.0",
|
|
40
|
+
scannerVersion: "unknown",
|
|
41
|
+
fileName: "unknown",
|
|
42
|
+
fileSize: 0,
|
|
43
|
+
scannedAt: (/* @__PURE__ */ new Date()).toISOString(),
|
|
44
|
+
detectedLanguage: "Unknown",
|
|
45
|
+
detectedFramework: null,
|
|
46
|
+
selectedPlatform: null,
|
|
47
|
+
agentName: null,
|
|
48
|
+
scanDuration: 0
|
|
49
|
+
},
|
|
50
|
+
saved: data.saved === true,
|
|
51
|
+
reportUrl: data.reportUrl ?? null,
|
|
52
|
+
artifactFingerprint: data.artifactFingerprint,
|
|
53
|
+
reportIntegrity: data.reportIntegrity,
|
|
54
|
+
policyProfile: data.policyProfile ?? null,
|
|
55
|
+
policyResult: data.policyResult ?? null,
|
|
56
|
+
attestation: data.attestation ?? null
|
|
57
|
+
};
|
|
58
|
+
}
|
|
59
|
+
async function scan(input, options) {
|
|
60
|
+
const { apiKey, apiUrl = DEFAULT_API_URL, timeout = DEFAULT_TIMEOUT } = options;
|
|
61
|
+
if (!apiKey) {
|
|
62
|
+
throw new Error(
|
|
63
|
+
"API key required. Generate one at https://aimodularity.com/agentverify/dashboard"
|
|
64
|
+
);
|
|
65
|
+
}
|
|
66
|
+
if (!input.content || input.content.trim().length === 0) {
|
|
67
|
+
throw new Error("content is required and cannot be empty");
|
|
68
|
+
}
|
|
69
|
+
const controller = new AbortController();
|
|
70
|
+
const timer = setTimeout(() => controller.abort(), timeout);
|
|
71
|
+
try {
|
|
72
|
+
const response = await fetch(apiUrl, {
|
|
73
|
+
method: "POST",
|
|
74
|
+
headers: {
|
|
75
|
+
"Authorization": `Bearer ${apiKey}`,
|
|
76
|
+
"Content-Type": "application/json",
|
|
77
|
+
"User-Agent": "agentverify-sdk/1.0.0"
|
|
78
|
+
},
|
|
79
|
+
body: JSON.stringify({
|
|
80
|
+
content: input.content,
|
|
81
|
+
fileName: input.fileName ?? "agent",
|
|
82
|
+
platform: input.platform,
|
|
83
|
+
policyId: input.policyId
|
|
84
|
+
}),
|
|
85
|
+
signal: controller.signal
|
|
86
|
+
});
|
|
87
|
+
if (response.status === 401) {
|
|
88
|
+
throw new Error(
|
|
89
|
+
"Invalid or unauthorized Agent Verify API key. Generate a valid key at https://aimodularity.com/agentverify/dashboard/"
|
|
90
|
+
);
|
|
91
|
+
}
|
|
92
|
+
if (response.status === 400) {
|
|
93
|
+
const err = await response.json().catch(() => ({}));
|
|
94
|
+
throw new Error(err.error ?? "Invalid request");
|
|
95
|
+
}
|
|
96
|
+
if (!response.ok) {
|
|
97
|
+
throw new Error(`API error: ${response.status} ${response.statusText}`);
|
|
98
|
+
}
|
|
99
|
+
const data = await response.json();
|
|
100
|
+
return normalizeScanResult(data);
|
|
101
|
+
} catch (err) {
|
|
102
|
+
if (err instanceof Error && err.name === "AbortError") {
|
|
103
|
+
throw new Error(`Request timed out after ${timeout}ms`);
|
|
104
|
+
}
|
|
105
|
+
throw err;
|
|
106
|
+
} finally {
|
|
107
|
+
clearTimeout(timer);
|
|
108
|
+
}
|
|
109
|
+
}
|
|
110
|
+
async function scanMany(inputs, options, concurrency = 3) {
|
|
111
|
+
const results = [];
|
|
112
|
+
for (let i = 0; i < inputs.length; i += concurrency) {
|
|
113
|
+
const batch = inputs.slice(i, i + concurrency);
|
|
114
|
+
const batchResults = await Promise.allSettled(
|
|
115
|
+
batch.map((input) => scan(input, options))
|
|
116
|
+
);
|
|
117
|
+
batchResults.forEach((res, j) => {
|
|
118
|
+
if (res.status === "fulfilled") {
|
|
119
|
+
results.push({ input: batch[j], result: res.value, error: null });
|
|
120
|
+
} else {
|
|
121
|
+
results.push({ input: batch[j], result: null, error: res.reason instanceof Error ? res.reason.message : String(res.reason) });
|
|
122
|
+
}
|
|
123
|
+
});
|
|
124
|
+
}
|
|
125
|
+
return results;
|
|
126
|
+
}
|
|
127
|
+
|
|
128
|
+
export {
|
|
129
|
+
scan,
|
|
130
|
+
scanMany
|
|
131
|
+
};
|
package/dist/cli.js
CHANGED
|
@@ -8,6 +8,61 @@ var import_path = require("path");
|
|
|
8
8
|
// src/sdk.ts
|
|
9
9
|
var DEFAULT_API_URL = "https://agentverify-api.agentverify.workers.dev/v1/scan";
|
|
10
10
|
var DEFAULT_TIMEOUT = 3e4;
|
|
11
|
+
function normalizeVerdict(value) {
|
|
12
|
+
if (value === "VERIFIED") return "VERIFIED";
|
|
13
|
+
if (value === "NOT_ASSESSED" || value === "NOT ASSESSED") return "NOT_ASSESSED";
|
|
14
|
+
return "NOT_VERIFIED";
|
|
15
|
+
}
|
|
16
|
+
function normalizeScanResult(data) {
|
|
17
|
+
const findings = Array.isArray(data.findings) ? data.findings : [];
|
|
18
|
+
const riskScore = typeof data.riskScore === "number" ? data.riskScore : 0;
|
|
19
|
+
return {
|
|
20
|
+
...data,
|
|
21
|
+
reportId: typeof data.reportId === "string" ? data.reportId : "",
|
|
22
|
+
verdict: normalizeVerdict(data.verdict),
|
|
23
|
+
riskScore,
|
|
24
|
+
riskLevel: data.riskLevel ?? "High Risk",
|
|
25
|
+
confidence: typeof data.confidence === "number" ? data.confidence : riskScore > 0 ? 80 : 0,
|
|
26
|
+
optimizationScore: typeof data.optimizationScore === "number" ? data.optimizationScore : 0,
|
|
27
|
+
reportInsights: data.reportInsights,
|
|
28
|
+
threatCategories: Array.isArray(data.threatCategories) ? data.threatCategories : [],
|
|
29
|
+
findings,
|
|
30
|
+
categoryScores: Array.isArray(data.categoryScores) ? data.categoryScores : [],
|
|
31
|
+
bom: data.bom ?? {
|
|
32
|
+
detectedLanguage: "Unknown",
|
|
33
|
+
detectedFramework: null,
|
|
34
|
+
detectedPlatform: null,
|
|
35
|
+
agentName: null,
|
|
36
|
+
toolAccessLevel: "Unknown",
|
|
37
|
+
credentialExposure: "Not Detected",
|
|
38
|
+
memoryPersistence: "Unknown",
|
|
39
|
+
auditLogging: "Unknown",
|
|
40
|
+
humanGates: "Unknown",
|
|
41
|
+
rateLimiting: "Unknown",
|
|
42
|
+
promptInjectionSurface: "Unknown",
|
|
43
|
+
delegationScope: "Unknown"
|
|
44
|
+
},
|
|
45
|
+
metadata: data.metadata ?? {
|
|
46
|
+
schemaVersion: "1.3.0",
|
|
47
|
+
scannerVersion: "unknown",
|
|
48
|
+
fileName: "unknown",
|
|
49
|
+
fileSize: 0,
|
|
50
|
+
scannedAt: (/* @__PURE__ */ new Date()).toISOString(),
|
|
51
|
+
detectedLanguage: "Unknown",
|
|
52
|
+
detectedFramework: null,
|
|
53
|
+
selectedPlatform: null,
|
|
54
|
+
agentName: null,
|
|
55
|
+
scanDuration: 0
|
|
56
|
+
},
|
|
57
|
+
saved: data.saved === true,
|
|
58
|
+
reportUrl: data.reportUrl ?? null,
|
|
59
|
+
artifactFingerprint: data.artifactFingerprint,
|
|
60
|
+
reportIntegrity: data.reportIntegrity,
|
|
61
|
+
policyProfile: data.policyProfile ?? null,
|
|
62
|
+
policyResult: data.policyResult ?? null,
|
|
63
|
+
attestation: data.attestation ?? null
|
|
64
|
+
};
|
|
65
|
+
}
|
|
11
66
|
async function scan(input, options) {
|
|
12
67
|
const { apiKey, apiUrl = DEFAULT_API_URL, timeout = DEFAULT_TIMEOUT } = options;
|
|
13
68
|
if (!apiKey) {
|
|
@@ -31,13 +86,14 @@ async function scan(input, options) {
|
|
|
31
86
|
body: JSON.stringify({
|
|
32
87
|
content: input.content,
|
|
33
88
|
fileName: input.fileName ?? "agent",
|
|
34
|
-
platform: input.platform
|
|
89
|
+
platform: input.platform,
|
|
90
|
+
policyId: input.policyId
|
|
35
91
|
}),
|
|
36
92
|
signal: controller.signal
|
|
37
93
|
});
|
|
38
94
|
if (response.status === 401) {
|
|
39
95
|
throw new Error(
|
|
40
|
-
"Invalid API key. Generate
|
|
96
|
+
"Invalid or unauthorized Agent Verify API key. Generate a valid key at https://aimodularity.com/agentverify/dashboard/"
|
|
41
97
|
);
|
|
42
98
|
}
|
|
43
99
|
if (response.status === 400) {
|
|
@@ -48,7 +104,7 @@ async function scan(input, options) {
|
|
|
48
104
|
throw new Error(`API error: ${response.status} ${response.statusText}`);
|
|
49
105
|
}
|
|
50
106
|
const data = await response.json();
|
|
51
|
-
return data;
|
|
107
|
+
return normalizeScanResult(data);
|
|
52
108
|
} catch (err) {
|
|
53
109
|
if (err instanceof Error && err.name === "AbortError") {
|
|
54
110
|
throw new Error(`Request timed out after ${timeout}ms`);
|
|
@@ -60,8 +116,13 @@ async function scan(input, options) {
|
|
|
60
116
|
}
|
|
61
117
|
|
|
62
118
|
// src/cli.ts
|
|
63
|
-
var VERSION = "1.
|
|
64
|
-
var API_URL = "https://agentverify-api.agentverify.workers.dev/v1/scan";
|
|
119
|
+
var VERSION = "1.4.0";
|
|
120
|
+
var API_URL = process.env.AGENTVERIFY_API_URL ?? "https://agentverify-api.agentverify.workers.dev/v1/scan";
|
|
121
|
+
var A2SPA_DOCS_URL = "https://aimodularity.com/A2SPA/docs";
|
|
122
|
+
var EXIT_OK = 0;
|
|
123
|
+
var EXIT_NOT_VERIFIED = 1;
|
|
124
|
+
var EXIT_NOT_ASSESSED = 2;
|
|
125
|
+
var EXIT_EXECUTION_ERROR = 3;
|
|
65
126
|
var c = {
|
|
66
127
|
reset: "\x1B[0m",
|
|
67
128
|
bold: "\x1B[1m",
|
|
@@ -76,6 +137,7 @@ var c = {
|
|
|
76
137
|
var write = (value = "") => process.stdout.write(`${value}
|
|
77
138
|
`);
|
|
78
139
|
var SUPPORTED_EXTENSIONS = /* @__PURE__ */ new Set([".js", ".ts", ".py", ".json", ".yaml", ".yml", ".md", ".mjs", ".cjs"]);
|
|
140
|
+
var MAX_SCAN_FILE_BYTES = 5 * 1024 * 1024;
|
|
79
141
|
var SKIP_DIRS = /* @__PURE__ */ new Set(["node_modules", ".git", "dist", "build", ".next", "vendor", "coverage", ".cache"]);
|
|
80
142
|
function collectFiles(dir) {
|
|
81
143
|
const files = [];
|
|
@@ -103,8 +165,13 @@ function parseArgs(argv) {
|
|
|
103
165
|
key: "",
|
|
104
166
|
file: "",
|
|
105
167
|
json: false,
|
|
168
|
+
markdown: false,
|
|
169
|
+
ci: false,
|
|
170
|
+
allowNotAssessed: false,
|
|
106
171
|
help: false,
|
|
107
|
-
version: false
|
|
172
|
+
version: false,
|
|
173
|
+
policy: "",
|
|
174
|
+
summaryFile: ""
|
|
108
175
|
};
|
|
109
176
|
if (result.command === "--help" || result.command === "-h") {
|
|
110
177
|
result.command = "";
|
|
@@ -119,6 +186,11 @@ function parseArgs(argv) {
|
|
|
119
186
|
if (arg === "--key" || arg === "-k") result.key = args[++i] ?? "";
|
|
120
187
|
else if (arg === "--file" || arg === "-f") result.file = args[++i] ?? "";
|
|
121
188
|
else if (arg === "--json") result.json = true;
|
|
189
|
+
else if (arg === "--markdown") result.markdown = true;
|
|
190
|
+
else if (arg === "--ci") result.ci = true;
|
|
191
|
+
else if (arg === "--allow-not-assessed") result.allowNotAssessed = true;
|
|
192
|
+
else if (arg === "--policy") result.policy = args[++i] ?? "";
|
|
193
|
+
else if (arg === "--summary-file") result.summaryFile = args[++i] ?? "";
|
|
122
194
|
else if (arg === "--help" || arg === "-h") result.help = true;
|
|
123
195
|
else if (arg === "--version" || arg === "-v") result.version = true;
|
|
124
196
|
else if (!arg.startsWith("-")) result.dir = arg;
|
|
@@ -142,6 +214,14 @@ ${c.bold}Options:${c.reset}
|
|
|
142
214
|
--key, -k API key (or set AGENTVERIFY_API_KEY env var)
|
|
143
215
|
--file, -f Scan a single file instead of a directory
|
|
144
216
|
--json Output results as JSON
|
|
217
|
+
--markdown Output a Markdown summary
|
|
218
|
+
--ci CI mode: concise output and non-zero exit for NOT VERIFIED
|
|
219
|
+
--allow-not-assessed
|
|
220
|
+
In CI mode, do not fail only because content is NOT_ASSESSED
|
|
221
|
+
--policy <id> Evaluate a built-in policy profile against this scan's evidence
|
|
222
|
+
in addition to the scanner's own verdict (e.g. financial-agent)
|
|
223
|
+
--summary-file <path>
|
|
224
|
+
Write results as JSON to this path (in addition to any other output)
|
|
145
225
|
--version, -v Print version
|
|
146
226
|
--help, -h Show this help message
|
|
147
227
|
|
|
@@ -158,17 +238,123 @@ ${c.bold}Examples:${c.reset}
|
|
|
158
238
|
${c.gray}# Output as JSON (useful for CI/CD)${c.reset}
|
|
159
239
|
agentverify scan . --key av_your_key --json
|
|
160
240
|
|
|
241
|
+
${c.gray}# Markdown summary for pull requests${c.reset}
|
|
242
|
+
agentverify scan . --key av_your_key --markdown
|
|
243
|
+
|
|
244
|
+
${c.bold}Exit codes (--ci):${c.reset}
|
|
245
|
+
0 All scanned files verified
|
|
246
|
+
1 NOT VERIFIED \u2014 a security/verification check failed
|
|
247
|
+
2 NOT ASSESSED \u2014 insufficient evidence for a verdict (see --allow-not-assessed)
|
|
248
|
+
3 Execution error \u2014 the scan itself could not run (bad key, network, missing file)
|
|
249
|
+
|
|
161
250
|
${c.bold}Get your API key:${c.reset}
|
|
162
251
|
https://aimodularity.com/agentverify/dashboard
|
|
163
252
|
|
|
164
253
|
${c.bold}Docs:${c.reset}
|
|
165
254
|
https://github.com/AI-Blockchain-Ventures/agentverify
|
|
255
|
+
|
|
256
|
+
${c.bold}A2SPA docs:${c.reset}
|
|
257
|
+
${A2SPA_DOCS_URL}
|
|
258
|
+
`);
|
|
259
|
+
}
|
|
260
|
+
function getTopBlocker(result) {
|
|
261
|
+
return result.reportInsights?.topBlocker ?? result.findings[0]?.title ?? "No blocking finding detected";
|
|
262
|
+
}
|
|
263
|
+
function getFixFirst(result) {
|
|
264
|
+
const fromInsights = result.reportInsights?.fixPriority?.filter((item) => item.priority === "fix_first").map((item) => item.title) ?? [];
|
|
265
|
+
const fallback = result.findings.slice(0, 3).map((item) => item.title);
|
|
266
|
+
return (fromInsights.length ? fromInsights : fallback).slice(0, 3);
|
|
267
|
+
}
|
|
268
|
+
function getRelevantThreatCategories(result) {
|
|
269
|
+
return (result.threatCategories ?? []).filter((item) => item.status !== "not_assessed").slice(0, 6).map((item) => `${item.label} (${item.status.replace("_", " ")})`);
|
|
270
|
+
}
|
|
271
|
+
function criticalCount(result) {
|
|
272
|
+
return result.findings.filter((f) => f.severity === "critical").length;
|
|
273
|
+
}
|
|
274
|
+
function highCount(result) {
|
|
275
|
+
return result.findings.filter((f) => f.severity === "high").length;
|
|
276
|
+
}
|
|
277
|
+
function fingerprintPreview(result) {
|
|
278
|
+
const hash = result.artifactFingerprint?.artifactHash;
|
|
279
|
+
return hash ? `${hash.slice(0, 12)}\u2026` : null;
|
|
280
|
+
}
|
|
281
|
+
function printDetailedSummary(result, fileName) {
|
|
282
|
+
const verified = result.verdict === "VERIFIED";
|
|
283
|
+
const verdictColor = verified ? c.green : result.verdict === "NOT_ASSESSED" ? c.yellow : c.red;
|
|
284
|
+
write(`
|
|
285
|
+
${c.bold}${c.cyan}Agent Verify${c.reset}`);
|
|
286
|
+
write(`${c.gray}${fileName}${c.reset}`);
|
|
287
|
+
write(`${c.bold}Score:${c.reset} ${result.riskScore}/100`);
|
|
288
|
+
write(`${c.bold}Verdict:${c.reset} ${verdictColor}${result.verdict === "NOT_VERIFIED" ? "NOT VERIFIED" : result.verdict.replace("_", " ")}${c.reset}`);
|
|
289
|
+
write(`${c.bold}Risk level:${c.reset} ${result.riskLevel}`);
|
|
290
|
+
write(`${c.bold}Confidence:${c.reset} ${result.confidence}/100`);
|
|
291
|
+
write(`${c.bold}Critical / High findings:${c.reset} ${criticalCount(result)} / ${highCount(result)}`);
|
|
292
|
+
if (result.metadata?.scannerVersion) write(`${c.bold}Ruleset version:${c.reset} ${result.metadata.scannerVersion}`);
|
|
293
|
+
if (fingerprintPreview(result)) write(`${c.bold}Artifact fingerprint:${c.reset} ${fingerprintPreview(result)}`);
|
|
294
|
+
if (result.policyProfile) write(`${c.bold}Policy (${result.policyProfile}):${c.reset} ${result.policyResult === "PASS" ? `${c.green}PASS${c.reset}` : `${c.red}FAIL${c.reset}`}`);
|
|
295
|
+
write(`${c.bold}Top blocker:${c.reset} ${getTopBlocker(result)}`);
|
|
296
|
+
write(`
|
|
297
|
+
${c.bold}Fix first:${c.reset}`);
|
|
298
|
+
const fixes = getFixFirst(result);
|
|
299
|
+
if (fixes.length) fixes.forEach((item, index) => write(` ${index + 1}. ${item}`));
|
|
300
|
+
else write(` No blocking fixes detected. Re-scan after any material change.`);
|
|
301
|
+
if (result.reportInsights?.nextAction) write(`
|
|
302
|
+
${c.bold}Next action:${c.reset} ${result.reportInsights.nextAction}`);
|
|
303
|
+
const threats = getRelevantThreatCategories(result);
|
|
304
|
+
if (threats.length) {
|
|
305
|
+
write(`
|
|
306
|
+
${c.bold}Threat categories:${c.reset}`);
|
|
307
|
+
threats.forEach((item) => write(` - ${item}`));
|
|
308
|
+
}
|
|
309
|
+
if (result.saved && result.reportId) write(`${c.bold}View report:${c.reset} https://aimodularity.com/agentverify/report/?id=${result.reportId}`);
|
|
310
|
+
write(`${c.bold}A2SPA docs:${c.reset} ${A2SPA_DOCS_URL}`);
|
|
311
|
+
write(`${c.gray}Re-scan:${c.reset} agentverify scan --file ${fileName} --key $AGENTVERIFY_API_KEY
|
|
166
312
|
`);
|
|
167
313
|
}
|
|
314
|
+
function markdownSummary(results, summary) {
|
|
315
|
+
const lines = ["# Agent Verify", "", `Files scanned: ${summary.total}`, `Verified: ${summary.verified}`, `Not verified: ${summary.notVerified}`, `Not assessed: ${summary.notAssessed}`, `Errors: ${summary.errors}`, ""];
|
|
316
|
+
for (const item of results) {
|
|
317
|
+
if (!item.result) {
|
|
318
|
+
lines.push(`## ${item.file}`, "", `Error: ${item.error ?? "Unknown error"}`, "");
|
|
319
|
+
continue;
|
|
320
|
+
}
|
|
321
|
+
lines.push(`## ${item.file}`, "", `Score: ${item.result.riskScore}/100`, `Verdict: ${item.result.verdict === "NOT_VERIFIED" ? "NOT VERIFIED" : item.result.verdict.replace("_", " ")}`, `Risk level: ${item.result.riskLevel}`, `Critical / High findings: ${criticalCount(item.result)} / ${highCount(item.result)}`);
|
|
322
|
+
if (item.result.metadata?.scannerVersion) lines.push(`Ruleset version: ${item.result.metadata.scannerVersion}`);
|
|
323
|
+
if (fingerprintPreview(item.result)) lines.push(`Artifact fingerprint: \`${fingerprintPreview(item.result)}\``);
|
|
324
|
+
if (item.result.policyProfile) lines.push(`Policy (${item.result.policyProfile}): ${item.result.policyResult}`);
|
|
325
|
+
lines.push(`Top blocker: ${getTopBlocker(item.result)}`, "");
|
|
326
|
+
const fixes = getFixFirst(item.result);
|
|
327
|
+
if (fixes.length) lines.push("Fix first:", ...fixes.map((fix, index) => `${index + 1}. ${fix}`), "");
|
|
328
|
+
const threats = getRelevantThreatCategories(item.result);
|
|
329
|
+
if (threats.length) lines.push("Threat categories:", ...threats.map((threat) => `- ${threat}`), "");
|
|
330
|
+
if (item.result.reportId) lines.push(`Report: https://aimodularity.com/agentverify/report/?id=${item.result.reportId}`, "");
|
|
331
|
+
}
|
|
332
|
+
lines.push(`A2SPA docs: ${A2SPA_DOCS_URL}`);
|
|
333
|
+
return lines.join("\n");
|
|
334
|
+
}
|
|
335
|
+
function writeGithubJobSummary(results, summary) {
|
|
336
|
+
const summaryPath = process.env.GITHUB_STEP_SUMMARY;
|
|
337
|
+
if (!summaryPath) return;
|
|
338
|
+
try {
|
|
339
|
+
(0, import_fs.appendFileSync)(summaryPath, `${markdownSummary(results, summary)}
|
|
340
|
+
`);
|
|
341
|
+
} catch (e) {
|
|
342
|
+
console.error(`Warning: could not write GitHub job summary: ${e instanceof Error ? e.message : e}`);
|
|
343
|
+
}
|
|
344
|
+
}
|
|
345
|
+
function writeSummaryFile(path, results, summary) {
|
|
346
|
+
if (!path) return;
|
|
347
|
+
try {
|
|
348
|
+
(0, import_fs.writeFileSync)(path, JSON.stringify({ results, summary }, null, 2));
|
|
349
|
+
} catch (e) {
|
|
350
|
+
console.error(`Warning: could not write summary file: ${e instanceof Error ? e.message : e}`);
|
|
351
|
+
}
|
|
352
|
+
}
|
|
168
353
|
function printVerdict(result, fileName) {
|
|
169
354
|
const verified = result.verdict === "VERIFIED";
|
|
170
|
-
const
|
|
171
|
-
const
|
|
355
|
+
const notAssessed = result.verdict === "NOT_ASSESSED";
|
|
356
|
+
const icon = verified ? `${c.green}\u2713${c.reset}` : notAssessed ? `${c.yellow}\u26A0${c.reset}` : `${c.red}\u2717${c.reset}`;
|
|
357
|
+
const verdict = verified ? `${c.green}VERIFIED${c.reset} ` : notAssessed ? `${c.yellow}NOT ASSESSED${c.reset}` : `${c.red}NOT VERIFIED${c.reset}`;
|
|
172
358
|
const score = verified ? `${c.green}${result.riskScore}/100${c.reset}` : result.riskScore >= 50 ? `${c.yellow}${result.riskScore}/100${c.reset}` : `${c.red}${result.riskScore}/100${c.reset}`;
|
|
173
359
|
const findings = result.findings.length > 0 ? `${c.gray}\u2014 ${result.findings.length} finding${result.findings.length === 1 ? "" : "s"}${c.reset}` : "";
|
|
174
360
|
const name = fileName.padEnd(30);
|
|
@@ -182,20 +368,23 @@ async function runScan(args) {
|
|
|
182
368
|
console.error(`
|
|
183
369
|
${c.red}Error:${c.reset} API key required
|
|
184
370
|
`);
|
|
185
|
-
console.error(` Set with --key av_your_key or AGENTVERIFY_API_KEY env var
|
|
186
|
-
console.error(` Get your key at https://aimodularity.com/agentverify/dashboard
|
|
371
|
+
console.error(` Set with --key av_your_key or AGENTVERIFY_API_KEY env var.`);
|
|
372
|
+
console.error(` Get your key at https://aimodularity.com/agentverify/dashboard/`);
|
|
373
|
+
console.error(` Example: AGENTVERIFY_API_KEY=av_your_key agentverify scan .
|
|
187
374
|
`);
|
|
188
|
-
process.exit(
|
|
375
|
+
process.exit(EXIT_EXECUTION_ERROR);
|
|
189
376
|
}
|
|
190
377
|
let files = [];
|
|
191
378
|
if (args.file) {
|
|
192
379
|
files = [args.file];
|
|
193
380
|
} else {
|
|
194
381
|
const dir = args.dir;
|
|
195
|
-
|
|
382
|
+
if (!args.markdown && !args.json) {
|
|
383
|
+
write(`
|
|
196
384
|
${c.bold}Agent Verify${c.reset} ${c.gray}v${VERSION}${c.reset}`);
|
|
197
|
-
|
|
385
|
+
write(`${c.gray}Scanning ${dir}...${c.reset}
|
|
198
386
|
`);
|
|
387
|
+
}
|
|
199
388
|
files = collectFiles(dir);
|
|
200
389
|
if (files.length === 0) {
|
|
201
390
|
write(`${c.yellow}No agent files found in ${dir}${c.reset}`);
|
|
@@ -207,16 +396,21 @@ ${c.bold}Agent Verify${c.reset} ${c.gray}v${VERSION}${c.reset}`);
|
|
|
207
396
|
const jsonResults = [];
|
|
208
397
|
let verified = 0;
|
|
209
398
|
let notVerified = 0;
|
|
399
|
+
let notAssessed = 0;
|
|
210
400
|
let errors = 0;
|
|
211
401
|
const CONCURRENCY = 3;
|
|
212
402
|
for (let i = 0; i < files.length; i += CONCURRENCY) {
|
|
213
403
|
const batch = files.slice(i, i + CONCURRENCY);
|
|
214
404
|
const batchResults = await Promise.allSettled(
|
|
215
405
|
batch.map(async (filePath) => {
|
|
406
|
+
const fileStat = (0, import_fs.statSync)(filePath);
|
|
407
|
+
if (fileStat.size > MAX_SCAN_FILE_BYTES) {
|
|
408
|
+
throw new Error(`File is ${(fileStat.size / 1024 / 1024).toFixed(1)}MB, over the ${MAX_SCAN_FILE_BYTES / 1024 / 1024}MB scan limit \u2014 skipped`);
|
|
409
|
+
}
|
|
216
410
|
const content = (0, import_fs.readFileSync)(filePath, "utf-8");
|
|
217
411
|
const relPath = (0, import_path.relative)(args.dir === "." ? process.cwd() : args.dir, filePath);
|
|
218
412
|
const result = await scan(
|
|
219
|
-
{ content, fileName: relPath },
|
|
413
|
+
{ content, fileName: relPath, policyId: args.policy || void 0 },
|
|
220
414
|
{ apiKey: args.key, apiUrl: API_URL }
|
|
221
415
|
);
|
|
222
416
|
return { filePath, relPath, result };
|
|
@@ -226,9 +420,13 @@ ${c.bold}Agent Verify${c.reset} ${c.gray}v${VERSION}${c.reset}`);
|
|
|
226
420
|
const outcome = batchResults[j];
|
|
227
421
|
if (outcome.status === "fulfilled") {
|
|
228
422
|
const { relPath, result } = outcome.value;
|
|
229
|
-
if (!args.json
|
|
423
|
+
if (!args.json && !args.markdown) {
|
|
424
|
+
if (args.file && !args.ci) printDetailedSummary(result, relPath);
|
|
425
|
+
else printVerdict(result, relPath);
|
|
426
|
+
}
|
|
230
427
|
jsonResults.push({ file: relPath, result, error: null });
|
|
231
428
|
if (result.verdict === "VERIFIED") verified++;
|
|
429
|
+
else if (result.verdict === "NOT_ASSESSED") notAssessed++;
|
|
232
430
|
else notVerified++;
|
|
233
431
|
} else {
|
|
234
432
|
const filePath = batch[j];
|
|
@@ -242,32 +440,58 @@ ${c.bold}Agent Verify${c.reset} ${c.gray}v${VERSION}${c.reset}`);
|
|
|
242
440
|
}
|
|
243
441
|
}
|
|
244
442
|
}
|
|
443
|
+
const summary = { total: files.length, verified, notVerified, notAssessed, errors };
|
|
444
|
+
writeGithubJobSummary(jsonResults, summary);
|
|
445
|
+
writeSummaryFile(args.summaryFile, jsonResults, summary);
|
|
245
446
|
if (args.json) {
|
|
246
|
-
write(JSON.stringify({ results: jsonResults, summary
|
|
447
|
+
write(JSON.stringify({ results: jsonResults, summary }, null, 2));
|
|
448
|
+
if (errors > 0) process.exit(EXIT_EXECUTION_ERROR);
|
|
449
|
+
if (args.ci) {
|
|
450
|
+
if (notVerified > 0) process.exit(EXIT_NOT_VERIFIED);
|
|
451
|
+
if (notAssessed > 0 && !args.allowNotAssessed) process.exit(EXIT_NOT_ASSESSED);
|
|
452
|
+
}
|
|
247
453
|
return;
|
|
248
454
|
}
|
|
455
|
+
if (args.markdown) {
|
|
456
|
+
write(markdownSummary(jsonResults, summary));
|
|
457
|
+
if (errors > 0) process.exit(EXIT_EXECUTION_ERROR);
|
|
458
|
+
if (args.ci) {
|
|
459
|
+
if (notVerified > 0) process.exit(EXIT_NOT_VERIFIED);
|
|
460
|
+
if (notAssessed > 0 && !args.allowNotAssessed) process.exit(EXIT_NOT_ASSESSED);
|
|
461
|
+
}
|
|
462
|
+
return;
|
|
463
|
+
}
|
|
464
|
+
const savedReports = jsonResults.filter((item) => item.result?.saved && item.result.reportId).length;
|
|
249
465
|
write(`
|
|
250
466
|
${c.gray}${"\u2500".repeat(60)}${c.reset}`);
|
|
251
467
|
write(`${c.bold}Agent Verify Scan Complete${c.reset}`);
|
|
252
468
|
write(`${c.gray}Files scanned:${c.reset} ${files.length}`);
|
|
253
469
|
write(`${c.green}Verified:${c.reset} ${verified}`);
|
|
254
470
|
if (notVerified > 0) write(`${c.red}Not Verified:${c.reset} ${notVerified}`);
|
|
471
|
+
if (notAssessed > 0) write(`${c.yellow}Not Assessed:${c.reset} ${notAssessed}`);
|
|
255
472
|
if (errors > 0) write(`${c.yellow}Errors:${c.reset} ${errors}`);
|
|
256
473
|
write(`${c.gray}${"\u2500".repeat(60)}${c.reset}`);
|
|
257
|
-
write(`
|
|
258
|
-
${c.gray}
|
|
474
|
+
if (savedReports > 0) write(`
|
|
475
|
+
${c.gray}Saved CLI reports appear in your dashboard: https://aimodularity.com/agentverify/dashboard/${c.reset}`);
|
|
476
|
+
else write(`
|
|
477
|
+
${c.gray}No dashboard report was saved. Check API key access and retry if you need dashboard sync.${c.reset}`);
|
|
478
|
+
write(`${c.gray}A2SPA docs: ${A2SPA_DOCS_URL}${c.reset}
|
|
259
479
|
`);
|
|
260
|
-
if (
|
|
480
|
+
if (errors > 0) process.exit(EXIT_EXECUTION_ERROR);
|
|
481
|
+
if (args.ci) {
|
|
482
|
+
if (notVerified > 0) process.exit(EXIT_NOT_VERIFIED);
|
|
483
|
+
if (notAssessed > 0 && !args.allowNotAssessed) process.exit(EXIT_NOT_ASSESSED);
|
|
484
|
+
}
|
|
261
485
|
}
|
|
262
486
|
async function main() {
|
|
263
487
|
const args = parseArgs(process.argv);
|
|
264
488
|
if (args.version) {
|
|
265
489
|
write(`agentverify v${VERSION}`);
|
|
266
|
-
process.exit(
|
|
490
|
+
process.exit(EXIT_OK);
|
|
267
491
|
}
|
|
268
492
|
if (args.help || !args.command) {
|
|
269
493
|
printHelp();
|
|
270
|
-
process.exit(
|
|
494
|
+
process.exit(EXIT_OK);
|
|
271
495
|
}
|
|
272
496
|
if (args.command === "scan") {
|
|
273
497
|
await runScan(args);
|
|
@@ -276,11 +500,11 @@ async function main() {
|
|
|
276
500
|
${c.red}Unknown command: ${args.command}${c.reset}`);
|
|
277
501
|
console.error(`Run ${c.cyan}agentverify --help${c.reset} for usage
|
|
278
502
|
`);
|
|
279
|
-
process.exit(
|
|
503
|
+
process.exit(EXIT_EXECUTION_ERROR);
|
|
280
504
|
}
|
|
281
505
|
}
|
|
282
506
|
main().catch((err) => {
|
|
283
507
|
console.error(`
|
|
284
508
|
${c.red}Fatal error:${c.reset}`, err instanceof Error ? err.message : err);
|
|
285
|
-
process.exit(
|
|
509
|
+
process.exit(EXIT_EXECUTION_ERROR);
|
|
286
510
|
});
|