agent-orchestrator-kit 0.1.5 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +12 -0
- package/README.md +46 -1
- package/bin/agent-orchestrator.js +76 -3
- package/package.json +1 -1
- package/templates/.gitlab/spec-verify.yml +47 -0
- package/templates/.gitlab-ci.starter.yml.example +2 -0
- package/templates/AGENTS.md +2 -0
- package/templates/scripts/post-mr-verdict.sh +116 -0
- package/templates/scripts/verify-specs.sh +196 -0
package/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,18 @@
|
|
|
2
2
|
|
|
3
3
|
All notable changes to this project will be documented in this file.
|
|
4
4
|
|
|
5
|
+
## [0.1.6] - 2026-07-02
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
- **`init --spec-verify`** — opt-in AI Spec Verifier for GitLab consumers: on MRs changing `src/`, an Amp agent verifies code against `openspec/specs/`, posts PASS/BLOCKED to the MR, and fails the pipeline on BLOCKED
|
|
9
|
+
- **Templates** — `.gitlab/spec-verify.yml` (blocking job, commented Phase 1 `allow_failure` fallback), `scripts/verify-specs.sh` (stack-agnostic prompt with project context from `openspec/config.yaml`, graceful skips, secret-safe), `scripts/post-mr-verdict.sh` (GitLab MR comment)
|
|
10
|
+
- **Orchestrator gate** — `spec-verify-blocking` auto-added to `roles.verifier.gates` (idempotent)
|
|
11
|
+
- **OpenSpec spec** — `spec-verify-consumer`
|
|
12
|
+
|
|
13
|
+
### Changed
|
|
14
|
+
- **`update`** refreshes spec-verify files via `KIT_OPTIN_PATHS` — only in projects that already installed them
|
|
15
|
+
- **README / AGENTS.md template** — AI Spec Verifier documented (install, CI variables, verdict schema, Phase 1 rollout)
|
|
16
|
+
|
|
5
17
|
## [0.1.5] - 2026-06-27
|
|
6
18
|
|
|
7
19
|
### Added
|
package/README.md
CHANGED
|
@@ -100,6 +100,7 @@ your-project/
|
|
|
100
100
|
├── CLAUDE.md
|
|
101
101
|
├── .github/workflows/agent-verify.yml # CI (default --ci github)
|
|
102
102
|
├── .gitlab/agent-verify.yml # CI fragment (--ci gitlab)
|
|
103
|
+
├── .gitlab/spec-verify.yml # AI Spec Verifier (--spec-verify, opt-in)
|
|
103
104
|
├── .agents/
|
|
104
105
|
│ ├── orchestrator.yaml
|
|
105
106
|
│ ├── mcp.json.example # Cursor MCP template
|
|
@@ -115,7 +116,8 @@ your-project/
|
|
|
115
116
|
│ ├── openspec-archive-change/
|
|
116
117
|
│ ├── openspec-sync-specs/
|
|
117
118
|
│ └── spec-workflow-openspec/
|
|
118
|
-
|
|
119
|
+
├── scripts/sync-local-agent-skills.sh
|
|
120
|
+
└── scripts/verify-specs.sh + post-mr-verdict.sh # (--spec-verify, opt-in)
|
|
119
121
|
```
|
|
120
122
|
|
|
121
123
|
### Included in kit
|
|
@@ -126,6 +128,7 @@ your-project/
|
|
|
126
128
|
| OpenSpec skills | All 7 skills for `/opsx:*` workflow |
|
|
127
129
|
| IDE sync | Cursor + Claude Code sync script |
|
|
128
130
|
| CI | `agent-verify.yml` — GitHub (default) or GitLab fragment + `prebuild` hook |
|
|
131
|
+
| AI Spec Verifier | `spec-verify.yml` + verifier scripts — GitLab opt-in (`--spec-verify`) |
|
|
129
132
|
| MCP templates | Memory MCP for Cursor and Amp |
|
|
130
133
|
|
|
131
134
|
### Not included (install separately)
|
|
@@ -326,6 +329,41 @@ Optional: include `.gitlab/agent-verify.yml` in `.gitlab-ci.yml` for full lint/b
|
|
|
326
329
|
|
|
327
330
|
Blocks merge if any gate fails.
|
|
328
331
|
|
|
332
|
+
#### AI Spec Verifier (GitLab, opt-in)
|
|
333
|
+
|
|
334
|
+
```bash
|
|
335
|
+
npx agent-orchestrator-kit init --ci gitlab --spec-verify
|
|
336
|
+
```
|
|
337
|
+
|
|
338
|
+
Installs an AI verification layer on top of the deterministic gates: on every merge request that changes `src/`, an Amp agent reads `openspec/specs/`, checks the changed code against every relevant requirement, posts a **PASS / BLOCKED** comment to the MR, and **fails the pipeline on BLOCKED** — specs become an enforceable merge contract, not just documentation.
|
|
339
|
+
|
|
340
|
+
Installed files:
|
|
341
|
+
|
|
342
|
+
| File | Purpose |
|
|
343
|
+
|------|---------|
|
|
344
|
+
| `.gitlab/spec-verify.yml` | CI fragment — hidden `.spec-verify-base` + blocking `spec-verify` job (MR + `src/**/*` only) |
|
|
345
|
+
| `scripts/verify-specs.sh` | Collects changed files + specs, builds prompt (project context from `openspec/config.yaml`), calls `amp -x`, writes `artifacts/verdict.json` |
|
|
346
|
+
| `scripts/post-mr-verdict.sh` | Posts the verdict as an MR comment via GitLab API |
|
|
347
|
+
|
|
348
|
+
The flag also adds `spec-verify-blocking` to `roles.verifier.gates` in `.agents/orchestrator.yaml`.
|
|
349
|
+
|
|
350
|
+
Setup after install:
|
|
351
|
+
|
|
352
|
+
1. Include the fragment from `.gitlab-ci.yml`:
|
|
353
|
+
|
|
354
|
+
```yaml
|
|
355
|
+
include:
|
|
356
|
+
- local: '.gitlab/spec-verify.yml'
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
2. Add CI/CD variables (Settings → CI/CD → Variables, masked): `AMP_API_KEY`, `GITLAB_VERIFIER_TOKEN` (project access token with `api` scope).
|
|
360
|
+
|
|
361
|
+
Verdict schema (`artifacts/verdict.json`): `pass`, `score` (0–100), `summary`, `findings[]` with `severity` (`error` fails the job), `spec`, `requirement`, `message`, `file`. The script degrades gracefully — no `src/` changes, no specs, missing `amp` CLI, or missing `AMP_API_KEY` produce a skipped passing verdict and never block the pipeline. Secrets are never logged; `.env`/key/token files are excluded from prompts.
|
|
362
|
+
|
|
363
|
+
**Warning-only rollout (Phase 1):** uncomment `allow_failure: true` in `.gitlab/spec-verify.yml` to keep the pipeline green while the team builds trust in verdicts, then remove it to enforce blocking (Phase 2).
|
|
364
|
+
|
|
365
|
+
`update` refreshes the three spec-verify files only in projects that already installed them — the feature stays opt-in.
|
|
366
|
+
|
|
329
367
|
---
|
|
330
368
|
|
|
331
369
|
### Archive — `/opsx:archive`
|
|
@@ -460,6 +498,7 @@ npx agent-orchestrator-kit init [options]
|
|
|
460
498
|
--lang <code> Agent language: en | uk | ...
|
|
461
499
|
--name <name> Project name (default: directory name)
|
|
462
500
|
--ci <provider> CI provider: gitlab | github | none (default: github)
|
|
501
|
+
--spec-verify Install AI Spec Verifier blocking gate (GitLab only)
|
|
463
502
|
--force Overwrite existing files
|
|
464
503
|
|
|
465
504
|
npx agent-orchestrator-kit update
|
|
@@ -501,6 +540,12 @@ openspec/ # Committed — spec-driven workflow
|
|
|
501
540
|
|
|
502
541
|
## Changelog
|
|
503
542
|
|
|
543
|
+
### 0.1.6
|
|
544
|
+
- `init --ci gitlab --spec-verify` — opt-in AI Spec Verifier: blocking MR gate via Amp CLI
|
|
545
|
+
- Templates: `.gitlab/spec-verify.yml`, `scripts/verify-specs.sh`, `scripts/post-mr-verdict.sh`
|
|
546
|
+
- `spec-verify-blocking` gate auto-added to `roles.verifier.gates`
|
|
547
|
+
- `update` refreshes spec-verify files only where already installed
|
|
548
|
+
|
|
504
549
|
### 0.1.5
|
|
505
550
|
- `init --ci gitlab|github|none` — GitLab verify via prebuild hook + CI fragment
|
|
506
551
|
- PM-aware `verify:openspec` / `prebuild` injection for GitLab projects
|
|
@@ -32,6 +32,13 @@ const KIT_MANAGED_PATHS = [
|
|
|
32
32
|
'scripts/sync-local-agent-skills.sh',
|
|
33
33
|
];
|
|
34
34
|
|
|
35
|
+
// Opt-in files: refreshed by `update` only when already present in the project
|
|
36
|
+
const KIT_OPTIN_PATHS = [
|
|
37
|
+
'.gitlab/spec-verify.yml',
|
|
38
|
+
'scripts/verify-specs.sh',
|
|
39
|
+
'scripts/post-mr-verdict.sh',
|
|
40
|
+
];
|
|
41
|
+
|
|
35
42
|
const VALID_CI_PROVIDERS = ['gitlab', 'github', 'none'];
|
|
36
43
|
const VERIFY_OPENSPEC_SCRIPT = 'npx openspec validate --all --strict';
|
|
37
44
|
|
|
@@ -223,6 +230,41 @@ function installCi(projectDir, templateDir, ci, force) {
|
|
|
223
230
|
}
|
|
224
231
|
}
|
|
225
232
|
|
|
233
|
+
function installSpecVerify(projectDir, templateDir, force) {
|
|
234
|
+
for (const rel of KIT_OPTIN_PATHS) {
|
|
235
|
+
const src = join(templateDir, rel);
|
|
236
|
+
const dest = join(projectDir, rel);
|
|
237
|
+
if (!existsSync(src)) continue;
|
|
238
|
+
if (!force && existsSync(dest)) {
|
|
239
|
+
log.warn(`skip (exists): ${rel}`);
|
|
240
|
+
continue;
|
|
241
|
+
}
|
|
242
|
+
mkdirSync(dirname(dest), { recursive: true });
|
|
243
|
+
copyFileSync(src, dest);
|
|
244
|
+
log.ok(rel);
|
|
245
|
+
}
|
|
246
|
+
try {
|
|
247
|
+
execSync(`chmod +x ${join(projectDir, 'scripts', 'verify-specs.sh')} ${join(projectDir, 'scripts', 'post-mr-verdict.sh')}`);
|
|
248
|
+
} catch {}
|
|
249
|
+
}
|
|
250
|
+
|
|
251
|
+
function patchOrchestratorSpecVerify(projectDir) {
|
|
252
|
+
const orchPath = join(projectDir, '.agents', 'orchestrator.yaml');
|
|
253
|
+
if (!existsSync(orchPath)) return;
|
|
254
|
+
|
|
255
|
+
let content = readFileSync(orchPath, 'utf-8');
|
|
256
|
+
if (content.includes('spec-verify-blocking')) return;
|
|
257
|
+
|
|
258
|
+
const anchor = /^(\s*)- openspec-validate-strict\s*$/m;
|
|
259
|
+
if (!anchor.test(content)) {
|
|
260
|
+
log.warn('could not add spec-verify-blocking gate: openspec-validate-strict anchor not found in orchestrator.yaml');
|
|
261
|
+
return;
|
|
262
|
+
}
|
|
263
|
+
content = content.replace(anchor, '$1- openspec-validate-strict\n$1- spec-verify-blocking');
|
|
264
|
+
writeFileSync(orchPath, content);
|
|
265
|
+
log.ok('spec-verify-blocking gate added to orchestrator.yaml');
|
|
266
|
+
}
|
|
267
|
+
|
|
226
268
|
function patchOrchestratorVerifier(projectDir, pm) {
|
|
227
269
|
const orchPath = join(projectDir, '.agents', 'orchestrator.yaml');
|
|
228
270
|
if (!existsSync(orchPath)) return;
|
|
@@ -245,7 +287,7 @@ function patchOrchestratorVerifier(projectDir, pm) {
|
|
|
245
287
|
writeFileSync(orchPath, content);
|
|
246
288
|
}
|
|
247
289
|
|
|
248
|
-
function printNextSteps(profile, projectDir, ci = 'github') {
|
|
290
|
+
function printNextSteps(profile, projectDir, ci = 'github', specVerify = false) {
|
|
249
291
|
const pm = detectPackageManager(projectDir);
|
|
250
292
|
const openspecReady = hasOpenSpec(projectDir);
|
|
251
293
|
const lines = [`${pc.bold('Next steps:')}`];
|
|
@@ -288,6 +330,13 @@ function printNextSteps(profile, projectDir, ci = 'github') {
|
|
|
288
330
|
lines.push(` ${pc.dim('Optional dev CI: include local .gitlab/agent-verify.yml (see kit templates/.gitlab-ci.starter.yml.example)')}`);
|
|
289
331
|
}
|
|
290
332
|
|
|
333
|
+
if (specVerify) {
|
|
334
|
+
lines.push(` ${pc.bold('AI Spec Verifier:')}`);
|
|
335
|
+
lines.push(` - include ${pc.cyan(".gitlab/spec-verify.yml")} from your .gitlab-ci.yml`);
|
|
336
|
+
lines.push(` - add CI/CD variables: ${pc.cyan('AMP_API_KEY')}, ${pc.cyan('GITLAB_VERIFIER_TOKEN')} (masked)`);
|
|
337
|
+
lines.push(` - BLOCKED verdict fails the MR pipeline (uncomment allow_failure for warning-only rollout)`);
|
|
338
|
+
}
|
|
339
|
+
|
|
291
340
|
console.log('\n' + lines.join('\n') + '\n');
|
|
292
341
|
}
|
|
293
342
|
|
|
@@ -319,6 +368,7 @@ program
|
|
|
319
368
|
.option('--name <name>', 'Project name (defaults to directory name)')
|
|
320
369
|
.option('--force', 'Overwrite existing files', false)
|
|
321
370
|
.option('--ci <provider>', 'CI provider: gitlab | github | none', 'github')
|
|
371
|
+
.option('--spec-verify', 'Install AI Spec Verifier blocking gate (GitLab only)', false)
|
|
322
372
|
.action((opts) => {
|
|
323
373
|
const projectDir = process.cwd();
|
|
324
374
|
const projectName = opts.name || basename(projectDir);
|
|
@@ -344,7 +394,10 @@ program
|
|
|
344
394
|
}
|
|
345
395
|
|
|
346
396
|
log.title('Installing scripts/');
|
|
347
|
-
copyDir(join(templateDir, 'scripts'), join(projectDir, 'scripts'), {
|
|
397
|
+
copyDir(join(templateDir, 'scripts'), join(projectDir, 'scripts'), {
|
|
398
|
+
overwrite: opts.force,
|
|
399
|
+
skip: ['verify-specs.sh', 'post-mr-verdict.sh'],
|
|
400
|
+
});
|
|
348
401
|
try {
|
|
349
402
|
execSync(`chmod +x ${join(projectDir, 'scripts', 'sync-local-agent-skills.sh')}`);
|
|
350
403
|
} catch {}
|
|
@@ -355,6 +408,15 @@ program
|
|
|
355
408
|
injectVerifyScripts(projectDir, { pm });
|
|
356
409
|
}
|
|
357
410
|
|
|
411
|
+
const specVerify = Boolean(opts.specVerify) && ci === 'gitlab';
|
|
412
|
+
if (opts.specVerify && ci !== 'gitlab') {
|
|
413
|
+
log.warn('--spec-verify requires --ci gitlab — skipping AI Spec Verifier install');
|
|
414
|
+
}
|
|
415
|
+
if (specVerify) {
|
|
416
|
+
log.title('Installing AI Spec Verifier (opt-in)');
|
|
417
|
+
installSpecVerify(projectDir, templateDir, opts.force);
|
|
418
|
+
}
|
|
419
|
+
|
|
358
420
|
log.title('Installing root files');
|
|
359
421
|
for (const f of ['AGENTS.md', 'CLAUDE.md']) {
|
|
360
422
|
const src = resolveTemplate(f, profile);
|
|
@@ -380,6 +442,9 @@ program
|
|
|
380
442
|
patchOrchestratorVerifier(projectDir, pm);
|
|
381
443
|
log.ok('.agents/orchestrator.yaml');
|
|
382
444
|
}
|
|
445
|
+
if (specVerify) {
|
|
446
|
+
patchOrchestratorSpecVerify(projectDir);
|
|
447
|
+
}
|
|
383
448
|
|
|
384
449
|
log.title('OpenSpec config template');
|
|
385
450
|
installOpenspecConfigExample(projectDir, profile, vars, opts.force);
|
|
@@ -389,7 +454,7 @@ program
|
|
|
389
454
|
|
|
390
455
|
log.title('Done');
|
|
391
456
|
log.ok(`agent-orchestrator-kit v${KIT_VERSION} installed`);
|
|
392
|
-
printNextSteps(profile, projectDir, ci);
|
|
457
|
+
printNextSteps(profile, projectDir, ci, specVerify);
|
|
393
458
|
});
|
|
394
459
|
|
|
395
460
|
program
|
|
@@ -414,6 +479,14 @@ program
|
|
|
414
479
|
}
|
|
415
480
|
}
|
|
416
481
|
|
|
482
|
+
for (const rel of KIT_OPTIN_PATHS) {
|
|
483
|
+
const src = join(templateDir, rel);
|
|
484
|
+
const dest = join(projectDir, rel);
|
|
485
|
+
if (!existsSync(src) || !existsSync(dest)) continue;
|
|
486
|
+
copyFileSync(src, dest);
|
|
487
|
+
log.ok(`${rel} (opt-in)`);
|
|
488
|
+
}
|
|
489
|
+
|
|
417
490
|
log.ok(`Updated to v${KIT_VERSION}`);
|
|
418
491
|
log.info('Run ./scripts/sync-local-agent-skills.sh to sync to local IDE');
|
|
419
492
|
});
|
package/package.json
CHANGED
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# ──────────────────────────────────────────────────────────────
|
|
2
|
+
# AI Spec Verifier — blocking gate on merge requests.
|
|
3
|
+
# Verifies that changed src/ code complies with openspec/specs/
|
|
4
|
+
# via Amp CLI, posts a PASS / BLOCKED comment to the MR, and
|
|
5
|
+
# fails the pipeline on a BLOCKED verdict.
|
|
6
|
+
#
|
|
7
|
+
# Installed by agent-orchestrator-kit (init --ci gitlab --spec-verify).
|
|
8
|
+
# Include from .gitlab-ci.yml:
|
|
9
|
+
# include:
|
|
10
|
+
# - local: '.gitlab/spec-verify.yml'
|
|
11
|
+
#
|
|
12
|
+
# Required CI/CD variables (Settings → CI/CD → Variables):
|
|
13
|
+
# AMP_API_KEY — Amp API key (masked, protected)
|
|
14
|
+
# GITLAB_VERIFIER_TOKEN — Project token with api scope (masked)
|
|
15
|
+
# ──────────────────────────────────────────────────────────────
|
|
16
|
+
.spec-verify-base:
|
|
17
|
+
image: node:20
|
|
18
|
+
script:
|
|
19
|
+
- apt-get update -qq && apt-get install -y -qq python3 curl git > /dev/null
|
|
20
|
+
- npm install -g @sourcegraph/amp@latest
|
|
21
|
+
- chmod +x scripts/verify-specs.sh scripts/post-mr-verdict.sh
|
|
22
|
+
- bash scripts/verify-specs.sh
|
|
23
|
+
- bash scripts/post-mr-verdict.sh
|
|
24
|
+
# Evaluate verdict — exit 1 if verifier says fail
|
|
25
|
+
- |
|
|
26
|
+
PASS=$(python3 -c "import json; v=json.load(open('artifacts/verdict.json')); print(str(v.get('pass',True)).lower())")
|
|
27
|
+
if [ "$PASS" != "true" ]; then
|
|
28
|
+
echo "Spec verifier found errors. See MR comment for details."
|
|
29
|
+
exit 1
|
|
30
|
+
fi
|
|
31
|
+
echo "Spec verification passed."
|
|
32
|
+
artifacts:
|
|
33
|
+
paths:
|
|
34
|
+
- artifacts/verdict.json
|
|
35
|
+
- artifacts/verifier-prompt.md
|
|
36
|
+
expire_in: 7 days
|
|
37
|
+
when: always
|
|
38
|
+
|
|
39
|
+
spec-verify:
|
|
40
|
+
extends: .spec-verify-base
|
|
41
|
+
rules:
|
|
42
|
+
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
|
|
43
|
+
changes:
|
|
44
|
+
- "src/**/*"
|
|
45
|
+
# Phase 1 (warning-only rollout): uncomment the next line to let
|
|
46
|
+
# the pipeline stay green while the team builds trust in verdicts.
|
|
47
|
+
# allow_failure: true
|
package/templates/AGENTS.md
CHANGED
|
@@ -26,6 +26,8 @@ Never mix phases in one chat — this is the single most important rule.
|
|
|
26
26
|
|
|
27
27
|
Verifier runs on **GitHub Actions** (default) or **GitLab** via `prebuild` → `verify:openspec` when using `init --ci gitlab`. GitLab projects do not use `.github/workflows/`.
|
|
28
28
|
|
|
29
|
+
With `init --ci gitlab --spec-verify`, an **AI Spec Verifier** also runs on MRs changing `src/`: an Amp agent checks the changed code against `openspec/specs/` and a **BLOCKED verdict fails the pipeline** (gate `spec-verify-blocking` in `.agents/orchestrator.yaml`).
|
|
30
|
+
|
|
29
31
|
## Hard Rules
|
|
30
32
|
|
|
31
33
|
- **One active change per developer** at a time.
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# ──────────────────────────────────────────────────────────────
|
|
3
|
+
# Posts spec verifier verdict as a GitLab MR comment.
|
|
4
|
+
#
|
|
5
|
+
# Installed by agent-orchestrator-kit (init --ci gitlab --spec-verify).
|
|
6
|
+
#
|
|
7
|
+
# Usage: ./scripts/post-mr-verdict.sh
|
|
8
|
+
# Env: CI_API_V4_URL, CI_PROJECT_ID, CI_MERGE_REQUEST_IID,
|
|
9
|
+
# GITLAB_VERIFIER_TOKEN (CI/CD variable, masked)
|
|
10
|
+
#
|
|
11
|
+
# Security: GITLAB_VERIFIER_TOKEN is a project access token with
|
|
12
|
+
# api scope. It is NEVER logged or echoed.
|
|
13
|
+
# ──────────────────────────────────────────────────────────────
|
|
14
|
+
set -euo pipefail
|
|
15
|
+
|
|
16
|
+
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
17
|
+
VERDICT_FILE="$ROOT/artifacts/verdict.json"
|
|
18
|
+
|
|
19
|
+
if [ ! -f "$VERDICT_FILE" ]; then
|
|
20
|
+
echo "No verdict file found — skipping MR comment."
|
|
21
|
+
exit 0
|
|
22
|
+
fi
|
|
23
|
+
|
|
24
|
+
if [ -z "${CI_API_V4_URL:-}" ] || [ -z "${CI_PROJECT_ID:-}" ] || [ -z "${CI_MERGE_REQUEST_IID:-}" ]; then
|
|
25
|
+
echo "Not running in MR context — skipping MR comment."
|
|
26
|
+
exit 0
|
|
27
|
+
fi
|
|
28
|
+
|
|
29
|
+
if [ -z "${GITLAB_VERIFIER_TOKEN:-}" ]; then
|
|
30
|
+
echo "GITLAB_VERIFIER_TOKEN not set — skipping MR comment."
|
|
31
|
+
exit 0
|
|
32
|
+
fi
|
|
33
|
+
|
|
34
|
+
# Parse verdict
|
|
35
|
+
PASS=$(python3 -c "import json,sys; v=json.load(open('$VERDICT_FILE')); print(str(v.get('pass',True)).lower())")
|
|
36
|
+
SCORE=$(python3 -c "import json,sys; v=json.load(open('$VERDICT_FILE')); print(v.get('score',0))")
|
|
37
|
+
SUMMARY=$(python3 -c "import json,sys; v=json.load(open('$VERDICT_FILE')); print(v.get('summary',''))")
|
|
38
|
+
SKIPPED=$(python3 -c "import json,sys; v=json.load(open('$VERDICT_FILE')); print(str(v.get('skipped',False)).lower())")
|
|
39
|
+
|
|
40
|
+
if [ "$SKIPPED" = "true" ]; then
|
|
41
|
+
ICON="⏭️"
|
|
42
|
+
STATUS="SKIPPED"
|
|
43
|
+
elif [ "$PASS" = "true" ]; then
|
|
44
|
+
ICON="✅"
|
|
45
|
+
STATUS="PASS"
|
|
46
|
+
else
|
|
47
|
+
ICON="❌"
|
|
48
|
+
STATUS="BLOCKED"
|
|
49
|
+
fi
|
|
50
|
+
|
|
51
|
+
# Build findings table
|
|
52
|
+
FINDINGS_TABLE=$(python3 <<'PYEOF'
|
|
53
|
+
import json, sys
|
|
54
|
+
|
|
55
|
+
with open("artifacts/verdict.json") as f:
|
|
56
|
+
v = json.load(f)
|
|
57
|
+
|
|
58
|
+
findings = v.get("findings", [])
|
|
59
|
+
if not findings:
|
|
60
|
+
print("_No findings._")
|
|
61
|
+
sys.exit(0)
|
|
62
|
+
|
|
63
|
+
severity_icons = {"error": "🔴", "warning": "🟡", "info": "🔵"}
|
|
64
|
+
|
|
65
|
+
print("| | Severity | Spec | Requirement | Message | File |")
|
|
66
|
+
print("|---|----------|------|-------------|---------|------|")
|
|
67
|
+
for f in findings:
|
|
68
|
+
icon = severity_icons.get(f.get("severity", "info"), "⚪")
|
|
69
|
+
sev = f.get("severity", "—")
|
|
70
|
+
spec = f.get("spec", "—")
|
|
71
|
+
req = f.get("requirement", "—")
|
|
72
|
+
msg = f.get("message", "—")
|
|
73
|
+
file = f.get("file", "—")
|
|
74
|
+
line = f.get("line")
|
|
75
|
+
if line:
|
|
76
|
+
file = f"{file}:{line}"
|
|
77
|
+
print(f"| {icon} | {sev} | {spec} | {req} | {msg} | {file} |")
|
|
78
|
+
PYEOF
|
|
79
|
+
)
|
|
80
|
+
|
|
81
|
+
# Build comment body
|
|
82
|
+
COMMENT_BODY="## ${ICON} Spec Verifier — ${STATUS}
|
|
83
|
+
|
|
84
|
+
**Score:** ${SCORE}/100
|
|
85
|
+
**Verdict:** ${STATUS}
|
|
86
|
+
|
|
87
|
+
### Summary
|
|
88
|
+
${SUMMARY}
|
|
89
|
+
|
|
90
|
+
### Findings
|
|
91
|
+
${FINDINGS_TABLE}
|
|
92
|
+
|
|
93
|
+
---
|
|
94
|
+
_AI Spec Verifier • agent-orchestrator-kit • Pipeline: ${CI_PIPELINE_ID:-local}_"
|
|
95
|
+
|
|
96
|
+
# Escape for JSON
|
|
97
|
+
COMMENT_JSON=$(python3 -c "
|
|
98
|
+
import json, sys
|
|
99
|
+
body = sys.stdin.read()
|
|
100
|
+
print(json.dumps({'body': body}))
|
|
101
|
+
" <<< "$COMMENT_BODY")
|
|
102
|
+
|
|
103
|
+
# Post to GitLab API
|
|
104
|
+
# Security: token is passed via header, never logged
|
|
105
|
+
HTTP_STATUS=$(curl -s -o /dev/null -w "%{http_code}" \
|
|
106
|
+
--request POST \
|
|
107
|
+
--header "PRIVATE-TOKEN: ${GITLAB_VERIFIER_TOKEN}" \
|
|
108
|
+
--header "Content-Type: application/json" \
|
|
109
|
+
--data "$COMMENT_JSON" \
|
|
110
|
+
"${CI_API_V4_URL}/projects/${CI_PROJECT_ID}/merge_requests/${CI_MERGE_REQUEST_IID}/notes")
|
|
111
|
+
|
|
112
|
+
if [ "$HTTP_STATUS" -ge 200 ] && [ "$HTTP_STATUS" -lt 300 ]; then
|
|
113
|
+
echo "MR comment posted successfully (HTTP $HTTP_STATUS)"
|
|
114
|
+
else
|
|
115
|
+
echo "Failed to post MR comment (HTTP $HTTP_STATUS)"
|
|
116
|
+
fi
|
|
@@ -0,0 +1,196 @@
|
|
|
1
|
+
#!/usr/bin/env bash
|
|
2
|
+
# ──────────────────────────────────────────────────────────────
|
|
3
|
+
# AI Spec Verifier — collects changed src/ files, concatenates
|
|
4
|
+
# openspec/specs/, builds a prompt, calls Amp CLI, and produces
|
|
5
|
+
# artifacts/verdict.json.
|
|
6
|
+
#
|
|
7
|
+
# Installed by agent-orchestrator-kit (init --ci gitlab --spec-verify).
|
|
8
|
+
#
|
|
9
|
+
# Usage: ./scripts/verify-specs.sh
|
|
10
|
+
# Env: CI_MERGE_REQUEST_DIFF_BASE_SHA (GitLab CI provides it)
|
|
11
|
+
# AMP_API_KEY — Amp API key (CI/CD variable, masked)
|
|
12
|
+
# SRC_GLOB — source path filter (default: src/)
|
|
13
|
+
#
|
|
14
|
+
# Security: this script NEVER logs tokens, keys, or .env content.
|
|
15
|
+
# ──────────────────────────────────────────────────────────────
|
|
16
|
+
set -euo pipefail
|
|
17
|
+
|
|
18
|
+
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
|
19
|
+
ARTIFACTS_DIR="$ROOT/artifacts"
|
|
20
|
+
SPECS_DIR="$ROOT/openspec/specs"
|
|
21
|
+
PROJECT_CONFIG="$ROOT/openspec/config.yaml"
|
|
22
|
+
VERDICT_FILE="$ARTIFACTS_DIR/verdict.json"
|
|
23
|
+
PROMPT_FILE="$ARTIFACTS_DIR/verifier-prompt.md"
|
|
24
|
+
SRC_GLOB="${SRC_GLOB:-src/}"
|
|
25
|
+
|
|
26
|
+
mkdir -p "$ARTIFACTS_DIR"
|
|
27
|
+
|
|
28
|
+
write_skipped_verdict() {
|
|
29
|
+
local summary="$1"
|
|
30
|
+
cat > "$VERDICT_FILE" <<EOF
|
|
31
|
+
{
|
|
32
|
+
"pass": true,
|
|
33
|
+
"score": 100,
|
|
34
|
+
"skipped": true,
|
|
35
|
+
"summary": "${summary}",
|
|
36
|
+
"findings": []
|
|
37
|
+
}
|
|
38
|
+
EOF
|
|
39
|
+
}
|
|
40
|
+
|
|
41
|
+
# ── 1. Collect changed source files ──────────────────────────
|
|
42
|
+
BASE_SHA="${CI_MERGE_REQUEST_DIFF_BASE_SHA:-HEAD~1}"
|
|
43
|
+
|
|
44
|
+
CHANGED_FILES=$(git diff --name-only "$BASE_SHA"...HEAD -- "$SRC_GLOB" || true)
|
|
45
|
+
|
|
46
|
+
if [ -z "$CHANGED_FILES" ]; then
|
|
47
|
+
echo "No ${SRC_GLOB} files changed — skipping spec verification."
|
|
48
|
+
write_skipped_verdict "No ${SRC_GLOB} files changed — verification skipped."
|
|
49
|
+
exit 0
|
|
50
|
+
fi
|
|
51
|
+
|
|
52
|
+
echo "Changed files:"
|
|
53
|
+
echo "$CHANGED_FILES"
|
|
54
|
+
|
|
55
|
+
# ── 2. Collect all spec files ────────────────────────────────
|
|
56
|
+
SPEC_FILES=$(find "$SPECS_DIR" -name '*.md' -type f 2>/dev/null || true)
|
|
57
|
+
|
|
58
|
+
if [ -z "$SPEC_FILES" ]; then
|
|
59
|
+
echo "No spec files found in $SPECS_DIR — skipping."
|
|
60
|
+
write_skipped_verdict "No spec files found — verification skipped."
|
|
61
|
+
exit 0
|
|
62
|
+
fi
|
|
63
|
+
|
|
64
|
+
# ── 3. Build spec content block ──────────────────────────────
|
|
65
|
+
SPECS_CONTENT=""
|
|
66
|
+
for spec_file in $SPEC_FILES; do
|
|
67
|
+
rel_path="${spec_file#$ROOT/}"
|
|
68
|
+
SPECS_CONTENT+="
|
|
69
|
+
--- FILE: $rel_path ---
|
|
70
|
+
$(cat "$spec_file")
|
|
71
|
+
"
|
|
72
|
+
done
|
|
73
|
+
|
|
74
|
+
# ── 4. Build changed file content block ──────────────────────
|
|
75
|
+
# Security: skip .env, secrets, tokens, keys from content
|
|
76
|
+
CHANGED_CONTENT=""
|
|
77
|
+
for file in $CHANGED_FILES; do
|
|
78
|
+
full_path="$ROOT/$file"
|
|
79
|
+
if [ -f "$full_path" ]; then
|
|
80
|
+
case "$file" in
|
|
81
|
+
*.env*|*secret*|*token*|*key*|*.pem|*.p12) continue ;;
|
|
82
|
+
esac
|
|
83
|
+
CHANGED_CONTENT+="
|
|
84
|
+
--- FILE: $file ---
|
|
85
|
+
$(cat "$full_path")
|
|
86
|
+
"
|
|
87
|
+
fi
|
|
88
|
+
done
|
|
89
|
+
|
|
90
|
+
# ── 5. Project context from openspec/config.yaml ─────────────
|
|
91
|
+
PROJECT_CONTEXT=""
|
|
92
|
+
if [ -f "$PROJECT_CONFIG" ]; then
|
|
93
|
+
PROJECT_CONTEXT="
|
|
94
|
+
## Project Context (openspec/config.yaml)
|
|
95
|
+
|
|
96
|
+
$(cat "$PROJECT_CONFIG")
|
|
97
|
+
"
|
|
98
|
+
fi
|
|
99
|
+
|
|
100
|
+
# ── 6. Build verifier prompt ─────────────────────────────────
|
|
101
|
+
cat > "$PROMPT_FILE" <<PROMPT
|
|
102
|
+
You are a SPEC VERIFIER.
|
|
103
|
+
Your job: verify that changed source code complies with project specifications.
|
|
104
|
+
${PROJECT_CONTEXT}
|
|
105
|
+
## Instructions
|
|
106
|
+
1. Read ALL specs below carefully.
|
|
107
|
+
2. Read ALL changed source files below.
|
|
108
|
+
3. For each changed file, check if it relates to any spec requirement.
|
|
109
|
+
4. Verify that every relevant requirement/scenario from specs is satisfied.
|
|
110
|
+
5. Respect project conventions from the project context above, if provided.
|
|
111
|
+
|
|
112
|
+
## Output
|
|
113
|
+
Return ONLY valid JSON (no markdown fences, no extra text) with this structure:
|
|
114
|
+
{
|
|
115
|
+
"pass": true|false,
|
|
116
|
+
"score": 0-100,
|
|
117
|
+
"summary": "Brief summary",
|
|
118
|
+
"findings": [
|
|
119
|
+
{
|
|
120
|
+
"severity": "error"|"warning"|"info",
|
|
121
|
+
"spec": "spec file path",
|
|
122
|
+
"requirement": "requirement name from spec",
|
|
123
|
+
"message": "what is wrong or missing",
|
|
124
|
+
"file": "affected source file",
|
|
125
|
+
"line": null
|
|
126
|
+
}
|
|
127
|
+
]
|
|
128
|
+
}
|
|
129
|
+
|
|
130
|
+
Rules for verdict:
|
|
131
|
+
- "pass": false if ANY finding has severity "error"
|
|
132
|
+
- "pass": true if only "warning" or "info" findings, or no findings
|
|
133
|
+
- "score": 100 minus 10 per error, 3 per warning (minimum 0)
|
|
134
|
+
- If a changed file has no related spec, add an "info" finding noting it
|
|
135
|
+
|
|
136
|
+
## SPECIFICATIONS
|
|
137
|
+
|
|
138
|
+
${SPECS_CONTENT}
|
|
139
|
+
|
|
140
|
+
## CHANGED SOURCE FILES
|
|
141
|
+
|
|
142
|
+
${CHANGED_CONTENT}
|
|
143
|
+
PROMPT
|
|
144
|
+
|
|
145
|
+
echo "Verifier prompt built ($(wc -c < "$PROMPT_FILE") bytes)"
|
|
146
|
+
|
|
147
|
+
# ── 7. Call Amp CLI ──────────────────────────────────────────
|
|
148
|
+
if ! command -v amp &>/dev/null; then
|
|
149
|
+
echo "amp CLI not found — writing fallback verdict."
|
|
150
|
+
write_skipped_verdict "amp CLI not installed — verification skipped."
|
|
151
|
+
exit 0
|
|
152
|
+
fi
|
|
153
|
+
|
|
154
|
+
if [ -z "${AMP_API_KEY:-}" ]; then
|
|
155
|
+
echo "AMP_API_KEY not set — skipping Amp call."
|
|
156
|
+
write_skipped_verdict "AMP_API_KEY not set in CI — Amp verification skipped."
|
|
157
|
+
exit 0
|
|
158
|
+
fi
|
|
159
|
+
|
|
160
|
+
echo "Running Amp verifier agent..."
|
|
161
|
+
export AMP_API_KEY
|
|
162
|
+
AMP_RESPONSE=$(amp -x < "$PROMPT_FILE" 2>/dev/null || true)
|
|
163
|
+
|
|
164
|
+
# ── 8. Extract JSON from response ────────────────────────────
|
|
165
|
+
# The response might contain markdown fences — strip them
|
|
166
|
+
CLEAN_JSON=$(echo "$AMP_RESPONSE" | sed -n '/^{/,/^}/p' | head -200)
|
|
167
|
+
|
|
168
|
+
if echo "$CLEAN_JSON" | python3 -m json.tool > /dev/null 2>&1; then
|
|
169
|
+
echo "$CLEAN_JSON" > "$VERDICT_FILE"
|
|
170
|
+
else
|
|
171
|
+
echo "Failed to parse verifier response as JSON."
|
|
172
|
+
echo "Raw response (first 500 chars):"
|
|
173
|
+
echo "$AMP_RESPONSE" | head -c 500
|
|
174
|
+
# Write a cautious pass — don't block on verifier infrastructure failure
|
|
175
|
+
cat > "$VERDICT_FILE" <<'EOF'
|
|
176
|
+
{
|
|
177
|
+
"pass": true,
|
|
178
|
+
"score": 50,
|
|
179
|
+
"skipped": false,
|
|
180
|
+
"summary": "Verifier did not return valid JSON — result inconclusive.",
|
|
181
|
+
"findings": [
|
|
182
|
+
{
|
|
183
|
+
"severity": "warning",
|
|
184
|
+
"spec": "N/A",
|
|
185
|
+
"requirement": "N/A",
|
|
186
|
+
"message": "Verifier agent response was not valid JSON. Manual review recommended.",
|
|
187
|
+
"file": "N/A",
|
|
188
|
+
"line": null
|
|
189
|
+
}
|
|
190
|
+
]
|
|
191
|
+
}
|
|
192
|
+
EOF
|
|
193
|
+
fi
|
|
194
|
+
|
|
195
|
+
echo "Verdict written to $VERDICT_FILE"
|
|
196
|
+
cat "$VERDICT_FILE" | python3 -m json.tool 2>/dev/null || cat "$VERDICT_FILE"
|