@prism-d1/cli 1.0.79 → 1.0.81
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/assets/github-workflows/prism-ai-metrics.yml +83 -122
- package/dist/assets/github-workflows/prism-eval-gate-kiro.yml +66 -0
- package/dist/assets/github-workflows/prism-eval-gate.yml +1 -1
- package/dist/assets/gitlab-workflows/.gitlab-ci.yml +0 -3
- package/dist/assets/gitlab-workflows/prism-ai-metrics.yml +53 -56
- package/dist/assets/infra/bin/app.ts +1 -0
- package/dist/assets/infra/lib/constructs/otel-collector-construct.ts +4 -0
- package/dist/assets/infra/lib/dashboard-stack.ts +378 -1235
- package/dist/assets/infra/lib/lambda/event-schema.ts +144 -0
- package/dist/assets/infra/lib/lambda/metrics-processor.ts +124 -36
- package/dist/assets/infra/lib/lambda/otel-receiver.ts +249 -35
- package/dist/assets/infra/lib/lambda/security-remediation-tracker.ts +17 -2
- package/dist/assets/infra/lib/lambda/velocity-widget.ts +1483 -0
- package/dist/assets/infra/lib/metrics-pipeline-stack.ts +111 -119
- package/dist/assets/infra/package.json +2 -1
- package/dist/assets/infra/tsconfig.json +9 -2
- package/dist/src/commands/assessment/web.d.ts.map +1 -1
- package/dist/src/commands/assessment/web.js +25 -2
- package/dist/src/commands/assessment/web.js.map +1 -1
- package/dist/src/commands/bootstrapper/install-eval-harness.d.ts +8 -2
- package/dist/src/commands/bootstrapper/install-eval-harness.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/install-eval-harness.js +46 -36
- package/dist/src/commands/bootstrapper/install-eval-harness.js.map +1 -1
- package/dist/src/commands/bootstrapper/install-git-hooks.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/install-git-hooks.js +10 -14
- package/dist/src/commands/bootstrapper/install-git-hooks.js.map +1 -1
- package/dist/src/commands/bootstrapper/install-github-workflows.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/install-github-workflows.js +7 -2
- package/dist/src/commands/bootstrapper/install-github-workflows.js.map +1 -1
- package/dist/src/commands/bootstrapper/install-gitlab-workflows.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/install-gitlab-workflows.js +7 -1
- package/dist/src/commands/bootstrapper/install-gitlab-workflows.js.map +1 -1
- package/dist/src/commands/bootstrapper/setup-github-oidc.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/setup-github-oidc.js +50 -27
- package/dist/src/commands/bootstrapper/setup-github-oidc.js.map +1 -1
- package/dist/src/commands/bootstrapper/setup-gitlab-oidc.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/setup-gitlab-oidc.js +69 -25
- package/dist/src/commands/bootstrapper/setup-gitlab-oidc.js.map +1 -1
- package/dist/src/commands/bootstrapper/setup-otel-sync.d.ts.map +1 -1
- package/dist/src/commands/bootstrapper/setup-otel-sync.js +145 -53
- package/dist/src/commands/bootstrapper/setup-otel-sync.js.map +1 -1
- package/dist/src/commands/workshop/generate-demo-data.d.ts +41 -2
- package/dist/src/commands/workshop/generate-demo-data.d.ts.map +1 -1
- package/dist/src/commands/workshop/generate-demo-data.js +391 -167
- package/dist/src/commands/workshop/generate-demo-data.js.map +1 -1
- package/dist/src/commands/workshop/verify-setup.d.ts.map +1 -1
- package/dist/src/commands/workshop/verify-setup.js +75 -14
- package/dist/src/commands/workshop/verify-setup.js.map +1 -1
- package/dist/src/utils/exec.d.ts +56 -0
- package/dist/src/utils/exec.d.ts.map +1 -0
- package/dist/src/utils/exec.js +72 -0
- package/dist/src/utils/exec.js.map +1 -0
- package/dist/src/utils/region.d.ts +50 -0
- package/dist/src/utils/region.d.ts.map +1 -0
- package/dist/src/utils/region.js +62 -0
- package/dist/src/utils/region.js.map +1 -0
- package/dist/src/utils/tempfile.d.ts +19 -0
- package/dist/src/utils/tempfile.d.ts.map +1 -0
- package/dist/src/utils/tempfile.js +32 -0
- package/dist/src/utils/tempfile.js.map +1 -0
- package/dist/src/utils/validate.d.ts +54 -0
- package/dist/src/utils/validate.d.ts.map +1 -0
- package/dist/src/utils/validate.js +143 -0
- package/dist/src/utils/validate.js.map +1 -0
- package/package.json +2 -2
- package/dist/assets/eval-harness/README.md +0 -144
- package/dist/assets/github-workflows/README.md +0 -110
- package/dist/assets/github-workflows/prism-dora-weekly.yml +0 -334
- package/dist/assets/gitlab-workflows/prism-dora-weekly.yml +0 -181
- package/dist/assets/infra/lib/lambda/defect-correlator.ts +0 -142
- package/dist/assets/infra/lib/lambda/spec-to-code-calculator.ts +0 -123
- package/dist/assets/metric-hooks/README.md +0 -107
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"tempfile.d.ts","sourceRoot":"","sources":["../../../src/utils/tempfile.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAMH,wBAAgB,eAAe,CAAC,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,MAAM,EAAE,GAAG,EAAE,CAAC,IAAI,EAAE,MAAM,KAAK,CAAC,GAAG,CAAC,CAS7F"}
|
|
@@ -0,0 +1,32 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Private temporary files for handing payloads to external commands.
|
|
3
|
+
*
|
|
4
|
+
* Two reasons this exists rather than writing to a fixed path under /tmp:
|
|
5
|
+
*
|
|
6
|
+
* `writeFileSync` follows symlinks. A predictable name like
|
|
7
|
+
* `/tmp/prism-request.json` can be pre-created by any local user as a symlink
|
|
8
|
+
* to a file the caller owns, and the write lands there instead. The sticky bit
|
|
9
|
+
* does not help — the attacker owns the symlink and we follow it. A fresh
|
|
10
|
+
* `mkdtemp` directory has an unpredictable name and mode 0700, so nothing can
|
|
11
|
+
* be planted inside it.
|
|
12
|
+
*
|
|
13
|
+
* The callback shape exists so the directory is reclaimed on every path. The
|
|
14
|
+
* first version of this helper returned the path and left callers to unlink
|
|
15
|
+
* only the file, which leaked one empty directory per invocation, and unlinked
|
|
16
|
+
* before throwing on failure so error paths leaked the file as well.
|
|
17
|
+
*/
|
|
18
|
+
import { mkdtempSync, writeFileSync, rmSync } from 'node:fs';
|
|
19
|
+
import { join } from 'node:path';
|
|
20
|
+
import { tmpdir } from 'node:os';
|
|
21
|
+
export function withPrivateTemp(name, content, use) {
|
|
22
|
+
const dir = mkdtempSync(join(tmpdir(), 'prism-'));
|
|
23
|
+
try {
|
|
24
|
+
const file = join(dir, name);
|
|
25
|
+
writeFileSync(file, content, { mode: 0o600 });
|
|
26
|
+
return use(file);
|
|
27
|
+
}
|
|
28
|
+
finally {
|
|
29
|
+
rmSync(dir, { recursive: true, force: true });
|
|
30
|
+
}
|
|
31
|
+
}
|
|
32
|
+
//# sourceMappingURL=tempfile.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"tempfile.js","sourceRoot":"","sources":["../../../src/utils/tempfile.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;GAgBG;AAEH,OAAO,EAAE,WAAW,EAAE,aAAa,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AAC7D,OAAO,EAAE,IAAI,EAAE,MAAM,WAAW,CAAC;AACjC,OAAO,EAAE,MAAM,EAAE,MAAM,SAAS,CAAC;AAEjC,MAAM,UAAU,eAAe,CAAI,IAAY,EAAE,OAAe,EAAE,GAAwB;IACxF,MAAM,GAAG,GAAG,WAAW,CAAC,IAAI,CAAC,MAAM,EAAE,EAAE,QAAQ,CAAC,CAAC,CAAC;IAClD,IAAI,CAAC;QACH,MAAM,IAAI,GAAG,IAAI,CAAC,GAAG,EAAE,IAAI,CAAC,CAAC;QAC7B,aAAa,CAAC,IAAI,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,KAAK,EAAE,CAAC,CAAC;QAC9C,OAAO,GAAG,CAAC,IAAI,CAAC,CAAC;IACnB,CAAC;YAAS,CAAC;QACT,MAAM,CAAC,GAAG,EAAE,EAAE,SAAS,EAAE,IAAI,EAAE,KAAK,EAAE,IAAI,EAAE,CAAC,CAAC;IAChD,CAAC;AACH,CAAC"}
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Validation for provider identity values collected at the prompt.
|
|
3
|
+
*
|
|
4
|
+
* These are NOT what prevents shell injection -- run() in ./exec.ts does that by
|
|
5
|
+
* never invoking a shell. Validation is here for two other reasons:
|
|
6
|
+
*
|
|
7
|
+
* 1. IAM role names are built from these values and accept only [\w+=,.@-].
|
|
8
|
+
* A value containing a slash or a space otherwise reaches AWS and comes back
|
|
9
|
+
* as a ValidationException that says nothing about which input was wrong.
|
|
10
|
+
* 2. Several of these values land inside an IAM trust policy condition. A
|
|
11
|
+
* malformed one does not fail loudly -- it produces a policy whose sub claim
|
|
12
|
+
* never matches (nothing can assume the role) or, worse, matches more
|
|
13
|
+
* broadly than intended.
|
|
14
|
+
*
|
|
15
|
+
* Each helper exits the process with a message naming the offending value rather
|
|
16
|
+
* than throwing, matching how the bootstrapper commands report other input
|
|
17
|
+
* errors.
|
|
18
|
+
*/
|
|
19
|
+
export declare function validateGithubOwner(value: string, label?: string): string;
|
|
20
|
+
export declare function validateGithubRepo(value: string, label?: string): string;
|
|
21
|
+
/** A single GitLab group or username -- one segment, no slash. */
|
|
22
|
+
export declare function validateGitlabNamespace(value: string, label?: string): string;
|
|
23
|
+
/**
|
|
24
|
+
* A full GitLab project path, which may be nested: `group/subgroup/project`.
|
|
25
|
+
*
|
|
26
|
+
* Slashes are legal here, unlike a GitHub repo name, so this cannot reuse the
|
|
27
|
+
* GitHub validator. Every segment is checked individually; empty segments (from
|
|
28
|
+
* a leading, trailing, or doubled slash) are rejected because they silently
|
|
29
|
+
* change the project_path sub claim.
|
|
30
|
+
*/
|
|
31
|
+
export declare function validateGitlabProjectPath(value: string, label?: string): string;
|
|
32
|
+
/**
|
|
33
|
+
* Normalizes a GitLab instance URL and returns the URL and its bare hostname.
|
|
34
|
+
*
|
|
35
|
+
* The hostname is used two ways that both break silently on a malformed value:
|
|
36
|
+
* as the last element of the OIDC provider ARN, and as the prefix of the
|
|
37
|
+
* `<host>:aud` / `<host>:sub` trust policy condition keys. Deriving it with a
|
|
38
|
+
* bare `.replace(/^https?:\/\//, '')` leaves any trailing slash or path in
|
|
39
|
+
* place, so `https://gitlab.com/` yields the condition key `gitlab.com/:sub`,
|
|
40
|
+
* which no token will ever match -- the role simply becomes unassumable, with
|
|
41
|
+
* no error at creation time.
|
|
42
|
+
*/
|
|
43
|
+
export declare function normalizeGitlabUrl(value: string, label?: string): {
|
|
44
|
+
url: string;
|
|
45
|
+
host: string;
|
|
46
|
+
};
|
|
47
|
+
/**
|
|
48
|
+
* A provider-assigned numeric id (GitHub owner/repo id, GitLab project id).
|
|
49
|
+
*
|
|
50
|
+
* These reach an IAM trust policy sub claim. A non-numeric value cannot be a
|
|
51
|
+
* real id, and permitting one risks widening the claim rather than failing.
|
|
52
|
+
*/
|
|
53
|
+
export declare function validateNumericId(value: string, label?: string): string;
|
|
54
|
+
//# sourceMappingURL=validate.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"validate.d.ts","sourceRoot":"","sources":["../../../src/utils/validate.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AAuBH,wBAAgB,mBAAmB,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,SAAwB,GAAG,MAAM,CAQxF;AAED,wBAAgB,kBAAkB,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,SAAoB,GAAG,MAAM,CAQnF;AAgBD,kEAAkE;AAClE,wBAAgB,uBAAuB,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,SAA0B,GAAG,MAAM,CAS9F;AAED;;;;;;;GAOG;AACH,wBAAgB,yBAAyB,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,SAAwB,GAAG,MAAM,CAU9F;AAED;;;;;;;;;;GAUG;AACH,wBAAgB,kBAAkB,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,SAAwB,GAAG;IAAE,GAAG,EAAE,MAAM,CAAC;IAAC,IAAI,EAAE,MAAM,CAAA;CAAE,CAqB9G;AAID;;;;;GAKG;AACH,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,MAAM,EAAE,KAAK,SAAO,GAAG,MAAM,CAKrE"}
|
|
@@ -0,0 +1,143 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* Validation for provider identity values collected at the prompt.
|
|
3
|
+
*
|
|
4
|
+
* These are NOT what prevents shell injection -- run() in ./exec.ts does that by
|
|
5
|
+
* never invoking a shell. Validation is here for two other reasons:
|
|
6
|
+
*
|
|
7
|
+
* 1. IAM role names are built from these values and accept only [\w+=,.@-].
|
|
8
|
+
* A value containing a slash or a space otherwise reaches AWS and comes back
|
|
9
|
+
* as a ValidationException that says nothing about which input was wrong.
|
|
10
|
+
* 2. Several of these values land inside an IAM trust policy condition. A
|
|
11
|
+
* malformed one does not fail loudly -- it produces a policy whose sub claim
|
|
12
|
+
* never matches (nothing can assume the role) or, worse, matches more
|
|
13
|
+
* broadly than intended.
|
|
14
|
+
*
|
|
15
|
+
* Each helper exits the process with a message naming the offending value rather
|
|
16
|
+
* than throwing, matching how the bootstrapper commands report other input
|
|
17
|
+
* errors.
|
|
18
|
+
*/
|
|
19
|
+
function fail(lines) {
|
|
20
|
+
for (const line of lines)
|
|
21
|
+
console.error(line);
|
|
22
|
+
process.exit(1);
|
|
23
|
+
}
|
|
24
|
+
// ---- GitHub ----
|
|
25
|
+
/**
|
|
26
|
+
* GitHub owner (user or org): alphanumerics and hyphens, no leading or trailing
|
|
27
|
+
* hyphen, 39 characters max.
|
|
28
|
+
*
|
|
29
|
+
* Deliberately looser than GitHub's current signup rule, which forbids
|
|
30
|
+
* consecutive hyphens. That rule postdates a lot of accounts, so grandfathered
|
|
31
|
+
* owners containing `--` still exist and IAM accepts them fine. Rejecting one
|
|
32
|
+
* would block a legitimate user for no security gain.
|
|
33
|
+
*/
|
|
34
|
+
const GITHUB_OWNER = /^[A-Za-z0-9](?:[A-Za-z0-9-]{0,37}[A-Za-z0-9])?$/;
|
|
35
|
+
/** GitHub repository: alphanumerics plus `.`, `_`, `-`, 100 characters max. */
|
|
36
|
+
const GITHUB_REPO = /^[A-Za-z0-9._-]{1,100}$/;
|
|
37
|
+
export function validateGithubOwner(value, label = 'GitHub username/org') {
|
|
38
|
+
if (!GITHUB_OWNER.test(value)) {
|
|
39
|
+
fail([
|
|
40
|
+
`Error: ${label} "${value}" is not a valid GitHub owner name.`,
|
|
41
|
+
' Expected letters, digits, and hyphens (max 39 characters), not starting or ending with a hyphen.',
|
|
42
|
+
]);
|
|
43
|
+
}
|
|
44
|
+
return value;
|
|
45
|
+
}
|
|
46
|
+
export function validateGithubRepo(value, label = 'Repository name') {
|
|
47
|
+
if (!GITHUB_REPO.test(value) || value === '.' || value === '..') {
|
|
48
|
+
fail([
|
|
49
|
+
`Error: ${label} "${value}" is not a valid GitHub repository name.`,
|
|
50
|
+
' Expected letters, digits, dots, underscores, and hyphens (max 100 characters).',
|
|
51
|
+
]);
|
|
52
|
+
}
|
|
53
|
+
return value;
|
|
54
|
+
}
|
|
55
|
+
// ---- GitLab ----
|
|
56
|
+
/**
|
|
57
|
+
* One segment of a GitLab namespace or project path.
|
|
58
|
+
*
|
|
59
|
+
* GitLab permits letters, digits, `_`, `.`, `-`, requires the first character to
|
|
60
|
+
* be alphanumeric, and reserves the `.git` and `.atom` suffixes.
|
|
61
|
+
*/
|
|
62
|
+
const GITLAB_SEGMENT = /^[A-Za-z0-9][A-Za-z0-9._-]{0,254}$/;
|
|
63
|
+
function validGitlabSegment(seg) {
|
|
64
|
+
return GITLAB_SEGMENT.test(seg) && !/\.(git|atom)$/i.test(seg);
|
|
65
|
+
}
|
|
66
|
+
/** A single GitLab group or username -- one segment, no slash. */
|
|
67
|
+
export function validateGitlabNamespace(value, label = 'GitLab group/username') {
|
|
68
|
+
if (!validGitlabSegment(value)) {
|
|
69
|
+
fail([
|
|
70
|
+
`Error: ${label} "${value}" is not a valid GitLab namespace.`,
|
|
71
|
+
' Expected a single path segment of letters, digits, dots, underscores, and hyphens.',
|
|
72
|
+
' Use --global with just the group name, not a full project path.',
|
|
73
|
+
]);
|
|
74
|
+
}
|
|
75
|
+
return value;
|
|
76
|
+
}
|
|
77
|
+
/**
|
|
78
|
+
* A full GitLab project path, which may be nested: `group/subgroup/project`.
|
|
79
|
+
*
|
|
80
|
+
* Slashes are legal here, unlike a GitHub repo name, so this cannot reuse the
|
|
81
|
+
* GitHub validator. Every segment is checked individually; empty segments (from
|
|
82
|
+
* a leading, trailing, or doubled slash) are rejected because they silently
|
|
83
|
+
* change the project_path sub claim.
|
|
84
|
+
*/
|
|
85
|
+
export function validateGitlabProjectPath(value, label = 'GitLab project path') {
|
|
86
|
+
const segments = value.split('/');
|
|
87
|
+
if (segments.length < 2 || !segments.every(validGitlabSegment)) {
|
|
88
|
+
fail([
|
|
89
|
+
`Error: ${label} "${value}" is not a valid GitLab project path.`,
|
|
90
|
+
' Expected group/project, optionally nested as group/subgroup/project.',
|
|
91
|
+
' Each segment may contain letters, digits, dots, underscores, and hyphens.',
|
|
92
|
+
]);
|
|
93
|
+
}
|
|
94
|
+
return value;
|
|
95
|
+
}
|
|
96
|
+
/**
|
|
97
|
+
* Normalizes a GitLab instance URL and returns the URL and its bare hostname.
|
|
98
|
+
*
|
|
99
|
+
* The hostname is used two ways that both break silently on a malformed value:
|
|
100
|
+
* as the last element of the OIDC provider ARN, and as the prefix of the
|
|
101
|
+
* `<host>:aud` / `<host>:sub` trust policy condition keys. Deriving it with a
|
|
102
|
+
* bare `.replace(/^https?:\/\//, '')` leaves any trailing slash or path in
|
|
103
|
+
* place, so `https://gitlab.com/` yields the condition key `gitlab.com/:sub`,
|
|
104
|
+
* which no token will ever match -- the role simply becomes unassumable, with
|
|
105
|
+
* no error at creation time.
|
|
106
|
+
*/
|
|
107
|
+
export function normalizeGitlabUrl(value, label = 'GitLab instance URL') {
|
|
108
|
+
let parsed;
|
|
109
|
+
try {
|
|
110
|
+
parsed = new URL(value);
|
|
111
|
+
}
|
|
112
|
+
catch {
|
|
113
|
+
return fail([
|
|
114
|
+
`Error: ${label} "${value}" is not a valid URL.`,
|
|
115
|
+
' Expected something like https://gitlab.com or https://gitlab.example.com.',
|
|
116
|
+
]);
|
|
117
|
+
}
|
|
118
|
+
if (parsed.protocol !== 'https:' && parsed.protocol !== 'http:') {
|
|
119
|
+
fail([`Error: ${label} must use http or https, got "${parsed.protocol}".`]);
|
|
120
|
+
}
|
|
121
|
+
if (parsed.pathname !== '/' && parsed.pathname !== '') {
|
|
122
|
+
fail([
|
|
123
|
+
`Error: ${label} "${value}" must not include a path.`,
|
|
124
|
+
` Use the instance root, e.g. ${parsed.protocol}//${parsed.host}`,
|
|
125
|
+
]);
|
|
126
|
+
}
|
|
127
|
+
// GitLab's aud claim is the instance URL with no trailing slash.
|
|
128
|
+
return { url: `${parsed.protocol}//${parsed.host}`, host: parsed.host };
|
|
129
|
+
}
|
|
130
|
+
// ---- Shared ----
|
|
131
|
+
/**
|
|
132
|
+
* A provider-assigned numeric id (GitHub owner/repo id, GitLab project id).
|
|
133
|
+
*
|
|
134
|
+
* These reach an IAM trust policy sub claim. A non-numeric value cannot be a
|
|
135
|
+
* real id, and permitting one risks widening the claim rather than failing.
|
|
136
|
+
*/
|
|
137
|
+
export function validateNumericId(value, label = 'ID') {
|
|
138
|
+
if (!/^\d+$/.test(value)) {
|
|
139
|
+
fail([`Error: ${label} "${value}" is not numeric.`]);
|
|
140
|
+
}
|
|
141
|
+
return value;
|
|
142
|
+
}
|
|
143
|
+
//# sourceMappingURL=validate.js.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"validate.js","sourceRoot":"","sources":["../../../src/utils/validate.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;GAiBG;AAEH,SAAS,IAAI,CAAC,KAAe;IAC3B,KAAK,MAAM,IAAI,IAAI,KAAK;QAAE,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAC9C,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;AAClB,CAAC;AAED,mBAAmB;AAEnB;;;;;;;;GAQG;AACH,MAAM,YAAY,GAAG,iDAAiD,CAAC;AAEvE,+EAA+E;AAC/E,MAAM,WAAW,GAAG,yBAAyB,CAAC;AAE9C,MAAM,UAAU,mBAAmB,CAAC,KAAa,EAAE,KAAK,GAAG,qBAAqB;IAC9E,IAAI,CAAC,YAAY,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;QAC9B,IAAI,CAAC;YACH,UAAU,KAAK,KAAK,KAAK,qCAAqC;YAC9D,oGAAoG;SACrG,CAAC,CAAC;IACL,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,MAAM,UAAU,kBAAkB,CAAC,KAAa,EAAE,KAAK,GAAG,iBAAiB;IACzE,IAAI,CAAC,WAAW,CAAC,IAAI,CAAC,KAAK,CAAC,IAAI,KAAK,KAAK,GAAG,IAAI,KAAK,KAAK,IAAI,EAAE,CAAC;QAChE,IAAI,CAAC;YACH,UAAU,KAAK,KAAK,KAAK,0CAA0C;YACnE,kFAAkF;SACnF,CAAC,CAAC;IACL,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,mBAAmB;AAEnB;;;;;GAKG;AACH,MAAM,cAAc,GAAG,oCAAoC,CAAC;AAE5D,SAAS,kBAAkB,CAAC,GAAW;IACrC,OAAO,cAAc,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,gBAAgB,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;AACjE,CAAC;AAED,kEAAkE;AAClE,MAAM,UAAU,uBAAuB,CAAC,KAAa,EAAE,KAAK,GAAG,uBAAuB;IACpF,IAAI,CAAC,kBAAkB,CAAC,KAAK,CAAC,EAAE,CAAC;QAC/B,IAAI,CAAC;YACH,UAAU,KAAK,KAAK,KAAK,oCAAoC;YAC7D,sFAAsF;YACtF,mEAAmE;SACpE,CAAC,CAAC;IACL,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED;;;;;;;GAOG;AACH,MAAM,UAAU,yBAAyB,CAAC,KAAa,EAAE,KAAK,GAAG,qBAAqB;IACpF,MAAM,QAAQ,GAAG,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC;IAClC,IAAI,QAAQ,CAAC,MAAM,GAAG,CAAC,IAAI,CAAC,QAAQ,CAAC,KAAK,CAAC,kBAAkB,CAAC,EAAE,CAAC;QAC/D,IAAI,CAAC;YACH,UAAU,KAAK,KAAK,KAAK,uCAAuC;YAChE,wEAAwE;YACxE,6EAA6E;SAC9E,CAAC,CAAC;IACL,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED;;;;;;;;;;GAUG;AACH,MAAM,UAAU,kBAAkB,CAAC,KAAa,EAAE,KAAK,GAAG,qBAAqB;IAC7E,IAAI,MAAW,CAAC;IAChB,IAAI,CAAC;QACH,MAAM,GAAG,IAAI,GAAG,CAAC,KAAK,CAAC,CAAC;IAC1B,CAAC;IAAC,MAAM,CAAC;QACP,OAAO,IAAI,CAAC;YACV,UAAU,KAAK,KAAK,KAAK,uBAAuB;YAChD,6EAA6E;SAC9E,CAAC,CAAC;IACL,CAAC;IACD,IAAI,MAAM,CAAC,QAAQ,KAAK,QAAQ,IAAI,MAAM,CAAC,QAAQ,KAAK,OAAO,EAAE,CAAC;QAChE,IAAI,CAAC,CAAC,UAAU,KAAK,iCAAiC,MAAM,CAAC,QAAQ,IAAI,CAAC,CAAC,CAAC;IAC9E,CAAC;IACD,IAAI,MAAM,CAAC,QAAQ,KAAK,GAAG,IAAI,MAAM,CAAC,QAAQ,KAAK,EAAE,EAAE,CAAC;QACtD,IAAI,CAAC;YACH,UAAU,KAAK,KAAK,KAAK,4BAA4B;YACrD,iCAAiC,MAAM,CAAC,QAAQ,KAAK,MAAM,CAAC,IAAI,EAAE;SACnE,CAAC,CAAC;IACL,CAAC;IACD,iEAAiE;IACjE,OAAO,EAAE,GAAG,EAAE,GAAG,MAAM,CAAC,QAAQ,KAAK,MAAM,CAAC,IAAI,EAAE,EAAE,IAAI,EAAE,MAAM,CAAC,IAAI,EAAE,CAAC;AAC1E,CAAC;AAED,mBAAmB;AAEnB;;;;;GAKG;AACH,MAAM,UAAU,iBAAiB,CAAC,KAAa,EAAE,KAAK,GAAG,IAAI;IAC3D,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,KAAK,CAAC,EAAE,CAAC;QACzB,IAAI,CAAC,CAAC,UAAU,KAAK,KAAK,KAAK,mBAAmB,CAAC,CAAC,CAAC;IACvD,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@prism-d1/cli",
|
|
3
|
-
"version": "1.0.
|
|
3
|
+
"version": "1.0.81",
|
|
4
4
|
"description": "PRISM D1 Velocity CLI — AI development lifecycle assessment tool",
|
|
5
5
|
"author": "andklee",
|
|
6
6
|
"engines": {
|
|
@@ -22,7 +22,7 @@
|
|
|
22
22
|
"access": "public"
|
|
23
23
|
},
|
|
24
24
|
"scripts": {
|
|
25
|
-
"build": "npx tsc && mkdir -p dist/assets/infra && cp -r ../bootstrapper/metric-hooks ../bootstrapper/eval-harness ../bootstrapper/github-workflows ../bootstrapper/gitlab-workflows dist/assets/ && cp -r ../infra/bin ../infra/lib ../infra/package.json ../infra/tsconfig.json ../infra/cdk.json dist/assets/infra/",
|
|
25
|
+
"build": "npx tsc && rm -rf dist/assets && mkdir -p dist/assets/infra && cp -r ../bootstrapper/metric-hooks ../bootstrapper/eval-harness ../bootstrapper/github-workflows ../bootstrapper/gitlab-workflows dist/assets/ && cp -r ../infra/bin ../infra/lib ../infra/package.json ../infra/tsconfig.json ../infra/cdk.json dist/assets/infra/",
|
|
26
26
|
"prepublishOnly": "npm run build",
|
|
27
27
|
"start": "tsx bin/prism-cli.ts",
|
|
28
28
|
"dev": "tsx bin/prism-cli.ts"
|
|
@@ -1,144 +0,0 @@
|
|
|
1
|
-
# Eval Harness
|
|
2
|
-
|
|
3
|
-
Evaluate code quality in CI using either **kiro-cli headless mode** (recommended) or Amazon Bedrock rubrics.
|
|
4
|
-
|
|
5
|
-
## Modes
|
|
6
|
-
|
|
7
|
-
| Mode | Install Command | How It Works |
|
|
8
|
-
|---|---|---|
|
|
9
|
-
| **Kiro** (recommended) | `prism-cli bootstrapper install-eval-harness --mode kiro` | kiro-cli reads files, follows `.kiro/steering/code-review.md`, outputs structured JSON findings |
|
|
10
|
-
| **Bedrock** (legacy) | `prism-cli bootstrapper install-eval-harness --with-rubrics` | Per-file rubric scoring via Bedrock API, weighted average client-side |
|
|
11
|
-
|
|
12
|
-
## Kiro Mode
|
|
13
|
-
|
|
14
|
-
```bash
|
|
15
|
-
prism-cli bootstrapper install-eval-harness --mode kiro
|
|
16
|
-
```
|
|
17
|
-
|
|
18
|
-
This installs:
|
|
19
|
-
- `.kiro/steering/code-review.md` — review rules (plain English)
|
|
20
|
-
- `.github/workflows/prism-eval-gate.yml` — kiro-cli headless CI workflow
|
|
21
|
-
|
|
22
|
-
**Requirements:**
|
|
23
|
-
- `KIRO_API_KEY` repository secret (generate at https://app.kiro.dev → Settings → API Keys)
|
|
24
|
-
- (Optional) `PRISM_METRICS_ROLE_ARN` for EventBridge metrics + AWS Continuum security scanning
|
|
25
|
-
|
|
26
|
-
**How it works:**
|
|
27
|
-
1. PR opened → workflow triggers
|
|
28
|
-
2. kiro-cli reads changed files + the diff via `--trust-all-tools`
|
|
29
|
-
3. Outputs structured JSON: findings with file/line, severity, score
|
|
30
|
-
4. Gate fails if any high-severity finding or score < 0.82
|
|
31
|
-
5. PR comment posted with findings table
|
|
32
|
-
|
|
33
|
-
## Bedrock Mode (Legacy)
|
|
34
|
-
|
|
35
|
-
## Bedrock Mode (Legacy) — Install
|
|
36
|
-
|
|
37
|
-
```bash
|
|
38
|
-
# Workshop mode — empty rubrics, create your own
|
|
39
|
-
bash prism-cli bootstrapper install-eval-harness
|
|
40
|
-
|
|
41
|
-
# Production mode — includes all 5 rubrics
|
|
42
|
-
bash prism-cli bootstrapper install-eval-harness --with-rubrics
|
|
43
|
-
|
|
44
|
-
# Non-interactive
|
|
45
|
-
bash prism-cli bootstrapper install-eval-harness --model us.anthropic.claude-haiku-4-5-20251001-v1:0 --threshold 0.82 --with-rubrics
|
|
46
|
-
```
|
|
47
|
-
|
|
48
|
-
This installs into your repo:
|
|
49
|
-
- `.prism/.prism/eval-harness/run-eval.sh` — evaluation script
|
|
50
|
-
- `.prism/.prism/eval-harness/eval-config.json` — model, threshold, region
|
|
51
|
-
- `.prism/.prism/eval-harness/rubrics/` — rubric JSON files
|
|
52
|
-
- `.github/workflows/prism-eval-gate.yml` — CI workflow
|
|
53
|
-
|
|
54
|
-
## Usage
|
|
55
|
-
|
|
56
|
-
```bash
|
|
57
|
-
# Evaluate a single file
|
|
58
|
-
./.prism/.prism/eval-harness/run-eval.sh .prism/.prism/eval-harness/rubrics/code-quality.json src/handler.ts
|
|
59
|
-
|
|
60
|
-
# With a spec file (for spec-compliance rubric)
|
|
61
|
-
./.prism/.prism/eval-harness/run-eval.sh .prism/.prism/eval-harness/rubrics/spec-compliance.json src/api.ts --spec specs/api.md
|
|
62
|
-
```
|
|
63
|
-
|
|
64
|
-
### Output
|
|
65
|
-
|
|
66
|
-
```
|
|
67
|
-
correctness: 0.9 — Handles all inputs correctly including edge cases
|
|
68
|
-
readability: 0.85 — Clear naming, minor style inconsistency in helper
|
|
69
|
-
...
|
|
70
|
-
|
|
71
|
-
Score: 0.8720
|
|
72
|
-
Result: PASS
|
|
73
|
-
Hallucinations: 0
|
|
74
|
-
```
|
|
75
|
-
|
|
76
|
-
Exit codes: `0` = pass, `1` = fail, `2` = error.
|
|
77
|
-
|
|
78
|
-
## Configuration
|
|
79
|
-
|
|
80
|
-
`eval-config.json`:
|
|
81
|
-
|
|
82
|
-
| Field | Description | Default |
|
|
83
|
-
|---|---|---|
|
|
84
|
-
| `pass_threshold` | Minimum score to pass (0-1) | `0.82` |
|
|
85
|
-
| `eval_model_id` | Bedrock model for evaluation | `us.anthropic.claude-haiku-4-5-20251001-v1:0` |
|
|
86
|
-
| `aws_region` | AWS region | `us-west-2` |
|
|
87
|
-
| `event_bus` | EventBridge bus name | `prism-d1-metrics` |
|
|
88
|
-
| `emit_to_eventbridge` | Emit events (workflow handles this) | `true` |
|
|
89
|
-
|
|
90
|
-
## Rubrics
|
|
91
|
-
|
|
92
|
-
Five production rubrics are available:
|
|
93
|
-
|
|
94
|
-
| Rubric | Auto-selected when file path matches |
|
|
95
|
-
|---|---|
|
|
96
|
-
| `code-quality.json` | Default fallback |
|
|
97
|
-
| `api-response-quality.json` | `api`, `handler`, `route`, `controller` |
|
|
98
|
-
| `agent-quality.json` | `agent`, `assistant`, `orchestrat`, `workflow`, `chain` |
|
|
99
|
-
| `security-compliance.json` | `auth`, `security`, `guard`, `policy`, `iam`, `crypto` |
|
|
100
|
-
| `spec-compliance.json` | Used when commit has `Spec-Ref:` trailer |
|
|
101
|
-
|
|
102
|
-
### Creating a Custom Rubric
|
|
103
|
-
|
|
104
|
-
```json
|
|
105
|
-
{
|
|
106
|
-
"rubric_name": "my-rubric",
|
|
107
|
-
"criteria": [
|
|
108
|
-
{
|
|
109
|
-
"name": "criterion_name",
|
|
110
|
-
"weight": 0.30,
|
|
111
|
-
"description": "What this measures",
|
|
112
|
-
"scoring": "How to score 0.0-1.0"
|
|
113
|
-
}
|
|
114
|
-
]
|
|
115
|
-
}
|
|
116
|
-
```
|
|
117
|
-
|
|
118
|
-
Weights must sum to 1.0. The script calculates the weighted average client-side (does not trust the LLM to do math).
|
|
119
|
-
|
|
120
|
-
## CI Workflow
|
|
121
|
-
|
|
122
|
-
The `prism-eval-gate.yml` workflow:
|
|
123
|
-
|
|
124
|
-
1. Detects commits with `AI-Origin:` trailers
|
|
125
|
-
2. Identifies changed source files from those commits
|
|
126
|
-
3. Auto-selects a rubric per file based on path
|
|
127
|
-
4. Runs `run-eval.sh` per file
|
|
128
|
-
5. Posts a PR comment with per-file scores
|
|
129
|
-
6. Waits for AWS Security Agent review (if installed)
|
|
130
|
-
7. Emits `prism.d1.eval` event to EventBridge
|
|
131
|
-
8. Fails the check if any file scores below threshold or Security Agent finds issues
|
|
132
|
-
|
|
133
|
-
### Requirements
|
|
134
|
-
|
|
135
|
-
- OIDC provider configured for GitHub Actions
|
|
136
|
-
- IAM role with `bedrock:InvokeModel` + `events:PutEvents`
|
|
137
|
-
- Repository secret `PRISM_METRICS_ROLE_ARN`
|
|
138
|
-
- `.prism/config.json` with `team_id`
|
|
139
|
-
|
|
140
|
-
## Uninstall
|
|
141
|
-
|
|
142
|
-
```bash
|
|
143
|
-
bash prism-cli bootstrapper install-eval-harness --uninstall
|
|
144
|
-
```
|
|
@@ -1,110 +0,0 @@
|
|
|
1
|
-
# GitHub Actions Workflows
|
|
2
|
-
|
|
3
|
-
Reusable GitHub Actions workflows for PRISM D1 Velocity metric collection.
|
|
4
|
-
|
|
5
|
-
## Workflows
|
|
6
|
-
|
|
7
|
-
| Workflow | Trigger | Purpose |
|
|
8
|
-
|---|---|---|
|
|
9
|
-
| `prism-ai-metrics.yml` | PR merge to main | Calculates AI-to-merge ratio, token usage, lead time. Emits `prism.d1.pr` + `prism.d1.deploy` events |
|
|
10
|
-
| `prism-eval-gate.yml` | PR open/update | Evaluates AI-generated code per-file with auto-selected rubrics, waits for Security Agent, blocks merge on failure |
|
|
11
|
-
| `prism-agent-eval.yml` | PR modifying agent code | Runs agent in mock mode, evaluates output with agent-quality rubric |
|
|
12
|
-
| `prism-dora-weekly.yml` | Weekly (Monday 09:00 UTC) | Calculates DORA + AI-DORA metrics, emits to EventBridge + CloudWatch |
|
|
13
|
-
|
|
14
|
-
## Setup
|
|
15
|
-
|
|
16
|
-
### 1. Configure AWS OIDC
|
|
17
|
-
|
|
18
|
-
```bash
|
|
19
|
-
bash prism-cli bootstrapper setup-github-oidc
|
|
20
|
-
```
|
|
21
|
-
|
|
22
|
-
This interactively creates:
|
|
23
|
-
- OIDC identity provider for `token.actions.githubusercontent.com`
|
|
24
|
-
- IAM role `GitHubActions-<repo>` with trust policy scoped to your repo
|
|
25
|
-
- Inline policy with `events:PutEvents` and `bedrock:InvokeModel`
|
|
26
|
-
|
|
27
|
-
For the weekly workflow, manually add `cloudwatch:PutMetricData` to the role policy.
|
|
28
|
-
|
|
29
|
-
### 2. Set Repository Secret
|
|
30
|
-
|
|
31
|
-
In GitHub: Settings → Secrets and variables → Actions → New repository secret:
|
|
32
|
-
|
|
33
|
-
| Secret | Value |
|
|
34
|
-
|---|---|
|
|
35
|
-
| `PRISM_METRICS_ROLE_ARN` | ARN printed by `setup-github-oidc` |
|
|
36
|
-
|
|
37
|
-
### 3. Install Git Hooks + Config
|
|
38
|
-
|
|
39
|
-
```bash
|
|
40
|
-
bash prism-cli bootstrapper install-git-hooks
|
|
41
|
-
```
|
|
42
|
-
|
|
43
|
-
Creates `.prism/config.json` with your team ID (read by all workflows).
|
|
44
|
-
|
|
45
|
-
### 4. Install Eval Harness
|
|
46
|
-
|
|
47
|
-
```bash
|
|
48
|
-
# Workshop mode — bring your own rubric
|
|
49
|
-
bash prism-cli bootstrapper install-eval-harness
|
|
50
|
-
|
|
51
|
-
# Production mode — includes all 5 rubrics
|
|
52
|
-
bash prism-cli bootstrapper install-eval-harness --with-rubrics
|
|
53
|
-
```
|
|
54
|
-
|
|
55
|
-
This copies `.prism/eval-harness/`, `eval-config.json`, rubrics, and the `prism-eval-gate.yml` workflow.
|
|
56
|
-
|
|
57
|
-
### 5. Copy Remaining Workflows
|
|
58
|
-
|
|
59
|
-
```bash
|
|
60
|
-
mkdir -p .github/workflows
|
|
61
|
-
cp bootstrapper/github-workflows/prism-ai-metrics.yml .github/workflows/
|
|
62
|
-
cp bootstrapper/github-workflows/prism-dora-weekly.yml .github/workflows/
|
|
63
|
-
# Optional — only if you have agents with --mock support:
|
|
64
|
-
cp bootstrapper/github-workflows/prism-agent-eval.yml .github/workflows/
|
|
65
|
-
```
|
|
66
|
-
|
|
67
|
-
## IAM Permissions
|
|
68
|
-
|
|
69
|
-
The OIDC role needs:
|
|
70
|
-
|
|
71
|
-
| Permission | Used by |
|
|
72
|
-
|---|---|
|
|
73
|
-
| `events:PutEvents` | All workflows |
|
|
74
|
-
| `bedrock:InvokeModel` | eval-gate, agent-eval |
|
|
75
|
-
| `cloudwatch:PutMetricData` | dora-weekly |
|
|
76
|
-
|
|
77
|
-
## Customization
|
|
78
|
-
|
|
79
|
-
| Setting | How |
|
|
80
|
-
|---|---|
|
|
81
|
-
| Branch | Edit `branches` in each workflow |
|
|
82
|
-
| AWS region | Edit `aws-region` field + EventBridge commands |
|
|
83
|
-
| Eval threshold | Edit `.prism/.prism/eval-harness/eval-config.json` → `pass_threshold` |
|
|
84
|
-
| Eval model | Edit `.prism/.prism/eval-harness/eval-config.json` → `eval_model_id` |
|
|
85
|
-
| Weekly schedule | Edit cron in `prism-dora-weekly.yml` (default: `0 9 * * 1`) |
|
|
86
|
-
|
|
87
|
-
## Events Emitted
|
|
88
|
-
|
|
89
|
-
| Detail Type | Source Workflow | Destination |
|
|
90
|
-
|---|---|---|
|
|
91
|
-
| `prism.d1.pr` | ai-metrics | EventBridge |
|
|
92
|
-
| `prism.d1.deploy` | ai-metrics | EventBridge |
|
|
93
|
-
| `prism.d1.eval` | eval-gate | EventBridge |
|
|
94
|
-
| `prism.d1.agent.eval` | agent-eval | EventBridge |
|
|
95
|
-
| `prism.d1.assessment` | dora-weekly | EventBridge |
|
|
96
|
-
| `prism.d1.security.code_review` | eval-gate (Security Agent) | EventBridge |
|
|
97
|
-
| `AIAdoptionRate`, `SpecCoverage`, tool counts | dora-weekly | CloudWatch |
|
|
98
|
-
|
|
99
|
-
All EventBridge events use source `prism.d1.velocity` and bus `prism-d1-metrics`.
|
|
100
|
-
|
|
101
|
-
## Troubleshooting
|
|
102
|
-
|
|
103
|
-
| Issue | Solution |
|
|
104
|
-
|---|---|
|
|
105
|
-
| OIDC auth fails | Verify trust policy `sub` matches `repo:org/repo:*` |
|
|
106
|
-
| EventBridge put fails | Check `events:PutEvents` on bus ARN |
|
|
107
|
-
| Eval gate always skips | Ensure commits have `AI-Origin:` trailers (install git hooks) |
|
|
108
|
-
| Weekly not running | Workflow must exist on default branch; test with `workflow_dispatch` |
|
|
109
|
-
| Agent eval skips | No `agent/main.py` found — add `--mock` support to your agent |
|
|
110
|
-
| Security Agent timeout | Agent takes 2+ min to start; workflow waits up to 12 min total |
|