@lifeaitools/rdc-skills 0.24.32 → 0.24.33
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/commands/help.md +7 -2
- package/commands/self-test.md +12 -6
- package/git-sha.json +1 -1
- package/package.json +1 -1
- package/scripts/acceptance.mjs +11 -4
- package/scripts/lib/runner.mjs +57 -21
- package/skills/help/SKILL.md +4 -2
- package/skills/self-test/SKILL.md +6 -4
- package/skills/tests/MATRIX.md +4 -4
- package/skills/tests/README.md +4 -3
- package/tests/acceptance.test.mjs +4 -3
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "rdc",
|
|
3
|
-
"version": "0.24.
|
|
3
|
+
"version": "0.24.33",
|
|
4
4
|
"description": "RDC typed-agent dispatch skill suite for Claude Code — plan, build, review, overnight unattended builds with work-item tracking and TDD enforcement.",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "LIFEAI",
|
package/commands/help.md
CHANGED
|
@@ -20,7 +20,9 @@ Resolve the manifest in this order and use the first existing file:
|
|
|
20
20
|
2. `C:/Dev/rdc-skills/.claude-plugin/plugin.json`
|
|
21
21
|
3. `~/.claude/plugins/cache/rdc-skills/rdc-skills/latest/.claude-plugin/plugin.json`
|
|
22
22
|
|
|
23
|
-
Read `skills_meta` from the manifest.
|
|
23
|
+
Read `skills_meta` from the manifest. Accept either the current object map shape
|
|
24
|
+
(`skills_meta.<name>`) or an older array shape (`skills_meta[]`); normalize to
|
|
25
|
+
entries before rendering. Group entries by `category` in this order:
|
|
24
26
|
`planning`, `build`, `deploy`, `release`, `dev-loop`, `reporting`, `tooling`,
|
|
25
27
|
`infra`.
|
|
26
28
|
|
|
@@ -38,7 +40,8 @@ static fallback table from `skills/help/SKILL.md`.
|
|
|
38
40
|
## Direct MCP / curl Access
|
|
39
41
|
|
|
40
42
|
Show this block after the command table so non-Claude callers have the same
|
|
41
|
-
entry point
|
|
43
|
+
entry point. Do not summarize, compress, relabel, or omit the curl examples;
|
|
44
|
+
this block is the public direct-caller contract for MCP/API/curl users.
|
|
42
45
|
|
|
43
46
|
```text
|
|
44
47
|
MCP endpoint: https://rdc-skills.regendevcorp.com/mcp
|
|
@@ -80,6 +83,8 @@ curl -s -X POST https://rdc-skills.regendevcorp.com/mcp \
|
|
|
80
83
|
-d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"rdc_skill_get","arguments":{"name":"rdc:build","variant":"cli","format":"json"}}}'
|
|
81
84
|
```
|
|
82
85
|
|
|
86
|
+
Required exact token for tests and direct callers: `"format":"json"`
|
|
87
|
+
|
|
83
88
|
## Hard Rules
|
|
84
89
|
|
|
85
90
|
- `cf: yes` skills must consult CodeFlow before acting.
|
package/commands/self-test.md
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: rdc:self-test
|
|
3
3
|
description: >-
|
|
4
|
-
Usage `rdc:self-test [--strict] [--skill <name>] [--json] [--fix] [--tier2] [--parallel <n>] [--quick]` — validates every rdc-*.md skill: frontmatter, Usage marker, name↔filename match, referenced guides/rules/hooks exist, output contract banner, plugin manifest, duplicate-name + collision checks. Tier 1 static lint; Tier 2 behavioral runs via headless
|
|
4
|
+
Usage `rdc:self-test [--strict] [--skill <name>] [--json] [--fix] [--tier2] [--parallel <n>] [--quick]` — validates every rdc-*.md skill: frontmatter, Usage marker, name↔filename match, referenced guides/rules/hooks exist, output contract banner, plugin manifest, duplicate-name + collision checks. Tier 1 static lint; Tier 2 behavioral runs via headless Claude or Codex in isolated worktrees + Supabase branch. Run before every release.
|
|
5
5
|
---
|
|
6
6
|
|
|
7
7
|
> **⚠️ OUTPUT CONTRACT (READ FIRST):** `guides/output-contract.md`
|
|
@@ -22,7 +22,7 @@ description: >-
|
|
|
22
22
|
| Tier | What it checks | Status |
|
|
23
23
|
|------|----------------|--------|
|
|
24
24
|
| Tier 1 | Static lint — frontmatter, Usage line, referenced files, name match | ✅ live |
|
|
25
|
-
| Tier 2 | Behavioral — headless Claude runs each skill in sandbox, asserts artifacts | ✅ live,
|
|
25
|
+
| Tier 2 | Behavioral — headless Claude or Codex runs each skill in sandbox, asserts artifacts | ✅ live — 29 manifests; acceptance harness records transcripts, tool calls, artifacts, and lessons learned |
|
|
26
26
|
| Tier 3 | Golden checklists — snapshot output format, regress on drift | 🔒 future |
|
|
27
27
|
|
|
28
28
|
## Procedure (Tier 1)
|
|
@@ -67,10 +67,11 @@ description: >-
|
|
|
67
67
|
|
|
68
68
|
## Procedure (Tier 2)
|
|
69
69
|
|
|
70
|
-
Tier 2 runs each skill end-to-end in an isolated sandbox and asserts on observed state (files touched, commits made, work items, exit code). Use it before shipping behavioral changes — Tier 1 alone can't catch runtime drift.
|
|
70
|
+
Tier 2 runs each skill end-to-end in an isolated sandbox and asserts on observed state (files touched, commits made, work items, exit code). Build acceptance additionally records all observable engine events/tool calls, assistant output, stdout/stderr artifacts, lessons learned, and next build optimizations. It supports `--engine claude` and `--engine codex`; both engines use the same manifests, worktree sandbox, JSONL evidence, and Markdown report. Use it before shipping behavioral changes — Tier 1 alone can't catch runtime drift.
|
|
71
71
|
|
|
72
72
|
1. **Prerequisites:**
|
|
73
|
-
- `claude` CLI on PATH (headless mode: `claude --print`)
|
|
73
|
+
- `claude` CLI on PATH for Claude runs (headless mode: `claude --print`)
|
|
74
|
+
- `codex` CLI on PATH for Codex runs (headless mode: `codex exec --json`)
|
|
74
75
|
- clauth daemon unlocked (`curl -s http://127.0.0.1:52437/ping`)
|
|
75
76
|
- Supabase MCP reachable (runner creates a throwaway test branch)
|
|
76
77
|
- Clean git tree in `rdc-skills` (worktrees are added under `.rdc/sandbox/<run-id>/`)
|
|
@@ -81,17 +82,21 @@ Tier 2 runs each skill end-to-end in an isolated sandbox and asserts on observed
|
|
|
81
82
|
node scripts/self-test.mjs --tier2 --skill rdc:build # single skill
|
|
82
83
|
node scripts/self-test.mjs --tier2 --parallel 3 # up to 3 skills in parallel
|
|
83
84
|
node scripts/self-test.mjs --tier2 --quick # skip long-running assertions
|
|
85
|
+
node scripts/acceptance.mjs --skill rdc:build # build acceptance with JSONL/tool-call evidence
|
|
86
|
+
node scripts/acceptance.mjs --engine codex --skill rdc:build
|
|
84
87
|
```
|
|
85
88
|
|
|
86
89
|
3. **What it does:**
|
|
87
90
|
- Runs Tier 1 as a pre-gate (fails fast if static lint fails)
|
|
88
91
|
- Creates one Supabase test branch for the run
|
|
89
|
-
- For each skill: `git worktree add` into `.rdc/sandbox/<run-id>/<skill>/`, sets `RDC_TEST=1`, invokes
|
|
92
|
+
- For each skill: `git worktree add` into `.rdc/sandbox/<run-id>/<skill>/`, sets `RDC_TEST=1`, invokes the selected headless engine with the skill prompt, waits for exit
|
|
90
93
|
- Asserts per the skill's manifest: exit code, files touched, commits made, stdout patterns
|
|
94
|
+
- Build acceptance writes JSONL evidence and extracts tool calls from the engine stream
|
|
91
95
|
- Cleans up worktrees + deletes the Supabase branch at the end (even on failure)
|
|
92
96
|
|
|
93
97
|
4. **Reports:**
|
|
94
98
|
- `.rdc/reports/self-test-tier2-<iso>.json` — full per-skill result, findings, timings
|
|
99
|
+
- `.rdc/reports/acceptance-*.jsonl` and `.rdc/reports/acceptance-*.md` — build acceptance evidence, transcript artifact pointers, lessons learned, and next build optimizations
|
|
95
100
|
- Exit codes: `0` pass, `1` fail (one or more skills failed assertions), `2` runner error (couldn't set up sandbox / branch)
|
|
96
101
|
|
|
97
102
|
5. **Adding a new manifest:**
|
|
@@ -104,4 +109,5 @@ Tier 2 runs each skill end-to-end in an isolated sandbox and asserts on observed
|
|
|
104
109
|
- Run Tier 1 **before every `rdc:release rdc-skills`** — it catches the backtick-drift class of bugs that break the skill menu silently.
|
|
105
110
|
- Use `--strict` in CI. Warnings matter in the release path.
|
|
106
111
|
- Do NOT skip findings by relaxing the linter. Fix the skill.
|
|
107
|
-
- Tier 2
|
|
112
|
+
- Run Tier 2 before tagging a release. Gate the tag if any manifested skill fails.
|
|
113
|
+
- Headless sessions must not steal focus. The runner uses hidden process launch options where supported; if windows flash or focus is grabbed, inspect `scripts/lib/runner.mjs` before changing skill behavior.
|
package/git-sha.json
CHANGED
package/package.json
CHANGED
package/scripts/acceptance.mjs
CHANGED
|
@@ -132,6 +132,16 @@ function claudeToolCalls(stdout) {
|
|
|
132
132
|
}
|
|
133
133
|
|
|
134
134
|
function assistantText(engine, stdout) {
|
|
135
|
+
if (engine === 'codex') {
|
|
136
|
+
const chunks = [];
|
|
137
|
+
for (const event of parseJsonLines(stdout)) {
|
|
138
|
+
if (event?.type === 'item.completed' && event.item?.type === 'agent_message' && typeof event.item.text === 'string') {
|
|
139
|
+
chunks.push(event.item.text);
|
|
140
|
+
}
|
|
141
|
+
if (event?.type === 'message' && typeof event.text === 'string') chunks.push(event.text);
|
|
142
|
+
}
|
|
143
|
+
return chunks.join('\n\n').trim() || String(stdout || '').trim();
|
|
144
|
+
}
|
|
135
145
|
if (engine !== 'claude') return String(stdout || '').trim();
|
|
136
146
|
const resultEvents = parseJsonLines(stdout).filter((event) => event.type === 'result' && typeof event.result === 'string');
|
|
137
147
|
if (resultEvents.length > 0) return resultEvents.at(-1).result.trim();
|
|
@@ -333,10 +343,6 @@ async function main() {
|
|
|
333
343
|
console.error(`unsupported --engine ${ENGINE}; expected claude or codex`);
|
|
334
344
|
process.exit(2);
|
|
335
345
|
}
|
|
336
|
-
if (ENGINE === 'codex') {
|
|
337
|
-
console.error('codex acceptance adapter can parse Codex JSONL, but live Codex agent spawning is not wired in this repo yet.');
|
|
338
|
-
process.exit(2);
|
|
339
|
-
}
|
|
340
346
|
|
|
341
347
|
mkdirSync(REPORTS_DIR, { recursive: true });
|
|
342
348
|
const startedAt = new Date().toISOString();
|
|
@@ -379,6 +385,7 @@ async function main() {
|
|
|
379
385
|
const result = await runManifest(manifest, {
|
|
380
386
|
runId: RUN_ID,
|
|
381
387
|
projectCwd: PROJECT_CWD,
|
|
388
|
+
engine: ENGINE,
|
|
382
389
|
});
|
|
383
390
|
const toolCalls = result.observed ? extractToolCalls(ENGINE, result.observed) : [];
|
|
384
391
|
const safeSkill = manifest.skill.replace(/[^a-z0-9-]+/gi, '-').replace(/^-+|-+$/g, '');
|
package/scripts/lib/runner.mjs
CHANGED
|
@@ -1,14 +1,16 @@
|
|
|
1
1
|
// runner.mjs — Tier 2 per-skill test orchestration.
|
|
2
2
|
//
|
|
3
3
|
// Public API:
|
|
4
|
-
// runManifest(manifest, { runId, supabaseBranchRef?, timeout?, claudeBin? })
|
|
4
|
+
// runManifest(manifest, { runId, supabaseBranchRef?, timeout?, engine?, claudeBin?, codexBin? })
|
|
5
5
|
// → { skill, pass, duration_ms, observed?, failures?, error? }
|
|
6
6
|
//
|
|
7
7
|
// Flow per manifest:
|
|
8
8
|
// 1. Create a worktree at .rdc-sandbox/<runId>/<skill-slug>/
|
|
9
9
|
// 2. Write precondition_files into the worktree
|
|
10
10
|
// 3. Snapshot HEAD sha + work_items rowcount (via Supabase branch if given)
|
|
11
|
-
// 4. Spawn
|
|
11
|
+
// 4. Spawn the selected engine in headless JSONL mode:
|
|
12
|
+
// - claude: `claude --print "<prompt>" --output-format stream-json`
|
|
13
|
+
// - codex: `codex exec --json "<prompt>"`
|
|
12
14
|
// - cwd = worktree
|
|
13
15
|
// - env = { ...process.env, ...fixture.env, RDC_TEST: "1" }
|
|
14
16
|
// - timeout kills the child on overrun
|
|
@@ -85,25 +87,19 @@ function writePreconditionFiles(worktreePath, files) {
|
|
|
85
87
|
}
|
|
86
88
|
}
|
|
87
89
|
|
|
88
|
-
|
|
89
|
-
function spawnClaude({ claudeBin, prompt, cwd, env, timeoutMs }) {
|
|
90
|
+
function spawnHiddenShell(command, args, { cwd, env, timeoutMs }) {
|
|
90
91
|
return new Promise((resolve) => {
|
|
91
92
|
let settled = false;
|
|
92
93
|
let stdout = "";
|
|
93
94
|
let stderr = "";
|
|
94
95
|
let child;
|
|
95
96
|
try {
|
|
96
|
-
// shell: true is required on Windows so `.cmd`/`.bat` shims (npm global
|
|
97
|
+
// shell: true is required on Windows so `.cmd`/`.bat`/`.ps1` shims (npm global
|
|
97
98
|
// wrappers like `claude`) resolve. On Unix shell: true routes through sh,
|
|
98
|
-
// which is harmless here.
|
|
99
|
-
//
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
child = spawn(
|
|
103
|
-
claudeBin,
|
|
104
|
-
["--print", safePrompt, "--output-format", "stream-json", "--verbose", "--dangerously-skip-permissions"],
|
|
105
|
-
{ cwd, env, shell: true, windowsHide: true, stdio: ["ignore", "pipe", "pipe"] },
|
|
106
|
-
);
|
|
99
|
+
// which is harmless here. Quote each arg so the shell preserves prompts
|
|
100
|
+
// with spaces/metacharacters as single argv tokens.
|
|
101
|
+
const safeArgs = args.map((arg) => JSON.stringify(String(arg)));
|
|
102
|
+
child = spawn(command, safeArgs, { cwd, env, shell: true, windowsHide: true, stdio: ["ignore", "pipe", "pipe"] });
|
|
107
103
|
} catch (e) {
|
|
108
104
|
resolve({ exit: -1, stdout: "", stderr: `spawn failed: ${e.message}`, timedOut: false });
|
|
109
105
|
return;
|
|
@@ -131,6 +127,35 @@ function spawnClaude({ claudeBin, prompt, cwd, env, timeoutMs }) {
|
|
|
131
127
|
});
|
|
132
128
|
}
|
|
133
129
|
|
|
130
|
+
/** Spawn `claude --print ...` with a hard timeout. Resolves to {exit, stdout, stderr, timedOut}. */
|
|
131
|
+
function spawnClaude({ claudeBin, prompt, cwd, env, timeoutMs }) {
|
|
132
|
+
return spawnHiddenShell(
|
|
133
|
+
claudeBin,
|
|
134
|
+
["--print", prompt ?? "", "--output-format", "stream-json", "--verbose", "--dangerously-skip-permissions"],
|
|
135
|
+
{ cwd, env, timeoutMs },
|
|
136
|
+
);
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
/** Spawn `codex exec --json ...` with a hard timeout. Resolves to {exit, stdout, stderr, timedOut}. */
|
|
140
|
+
function spawnCodex({ codexBin, prompt, cwd, env, timeoutMs }) {
|
|
141
|
+
return spawnHiddenShell(
|
|
142
|
+
codexBin,
|
|
143
|
+
[
|
|
144
|
+
"exec",
|
|
145
|
+
"--json",
|
|
146
|
+
"--cd",
|
|
147
|
+
cwd,
|
|
148
|
+
"--sandbox",
|
|
149
|
+
"danger-full-access",
|
|
150
|
+
"--dangerously-bypass-approvals-and-sandbox",
|
|
151
|
+
"--dangerously-bypass-hook-trust",
|
|
152
|
+
"--ephemeral",
|
|
153
|
+
prompt ?? "",
|
|
154
|
+
],
|
|
155
|
+
{ cwd, env, timeoutMs },
|
|
156
|
+
);
|
|
157
|
+
}
|
|
158
|
+
|
|
134
159
|
// ─── work_items fetch (live) ────────────────────────────────────────────────
|
|
135
160
|
//
|
|
136
161
|
// These helpers talk to the Supabase PostgREST endpoint on the branch created
|
|
@@ -237,7 +262,9 @@ export async function runManifest(manifest, opts = {}) {
|
|
|
237
262
|
runId,
|
|
238
263
|
supabaseBranchRef = null,
|
|
239
264
|
timeout = manifest?.timeout_ms || DEFAULT_TIMEOUT_MS,
|
|
265
|
+
engine = process.env.RDC_ACCEPTANCE_ENGINE || "claude",
|
|
240
266
|
claudeBin = process.env.CLAUDE_BIN || "claude",
|
|
267
|
+
codexBin = process.env.CODEX_BIN || "codex",
|
|
241
268
|
// projectCwd: the repo claude --print will commit to (regen-root).
|
|
242
269
|
// Worktrees are created FROM this repo so git state tracking is correct.
|
|
243
270
|
// claude runs inside the worktree (not projectCwd directly) so each skill
|
|
@@ -283,33 +310,41 @@ export async function runManifest(manifest, opts = {}) {
|
|
|
283
310
|
const headBefore = snapshotHead(worktree.path);
|
|
284
311
|
const wiCountBefore = await fetchWorkItemsCount(supabaseBranchRef);
|
|
285
312
|
|
|
286
|
-
|
|
287
|
-
|
|
313
|
+
const selectedEngine = String(engine || "claude").toLowerCase();
|
|
314
|
+
if (!["claude", "codex"].includes(selectedEngine)) {
|
|
315
|
+
return { skill, pass: false, duration_ms: nowMs() - started, error: `unsupported engine: ${selectedEngine}` };
|
|
316
|
+
}
|
|
317
|
+
|
|
318
|
+
// Start from process.env but scrub inherited engine session vars — the child
|
|
319
|
+
// process should not see parent session context. Then set RDC_TEST
|
|
288
320
|
// and overlay fixture env.
|
|
289
321
|
const env = { ...process.env };
|
|
290
322
|
for (const key of Object.keys(env)) {
|
|
291
323
|
if (key.startsWith("CLAUDE_")) delete env[key];
|
|
324
|
+
if (key.startsWith("CODEX_") && key !== "CODEX_HOME") delete env[key];
|
|
292
325
|
}
|
|
293
326
|
env.RDC_TEST = "1";
|
|
294
327
|
Object.assign(env, manifest.fixture?.env || {});
|
|
295
328
|
|
|
296
|
-
// Run
|
|
329
|
+
// Run the agent inside the worktree — commits land there, git tracking is correct,
|
|
297
330
|
// and parallel skills each get their own isolated branch with no file conflicts.
|
|
298
331
|
let spawnRes;
|
|
299
332
|
try {
|
|
300
|
-
|
|
301
|
-
claudeBin,
|
|
333
|
+
const spawnArgs = {
|
|
302
334
|
prompt: manifest.fixture?.prompt || "",
|
|
303
335
|
cwd: worktree.path,
|
|
304
336
|
env,
|
|
305
337
|
timeoutMs: timeout,
|
|
306
|
-
}
|
|
338
|
+
};
|
|
339
|
+
spawnRes = selectedEngine === "codex"
|
|
340
|
+
? await spawnCodex({ ...spawnArgs, codexBin })
|
|
341
|
+
: await spawnClaude({ ...spawnArgs, claudeBin });
|
|
307
342
|
} catch (e) {
|
|
308
343
|
return {
|
|
309
344
|
skill,
|
|
310
345
|
pass: false,
|
|
311
346
|
duration_ms: nowMs() - started,
|
|
312
|
-
error: `spawn
|
|
347
|
+
error: `spawn ${selectedEngine}: ${e.message}`,
|
|
313
348
|
};
|
|
314
349
|
}
|
|
315
350
|
|
|
@@ -321,6 +356,7 @@ export async function runManifest(manifest, opts = {}) {
|
|
|
321
356
|
exit_code: spawnRes.exit,
|
|
322
357
|
stdout: spawnRes.stdout,
|
|
323
358
|
stderr: spawnRes.stderr,
|
|
359
|
+
engine: selectedEngine,
|
|
324
360
|
files_modified,
|
|
325
361
|
commits,
|
|
326
362
|
work_items_delta,
|
package/skills/help/SKILL.md
CHANGED
|
@@ -17,11 +17,11 @@ description: "Usage `rdc:help` — Show all MCP skills with usage, requirements,
|
|
|
17
17
|
- `{PLUGIN_ROOT}/.claude-plugin/plugin.json` ← preferred when running from the published plugin
|
|
18
18
|
- `C:/Dev/rdc-skills/.claude-plugin/plugin.json` ← source-checkout fallback
|
|
19
19
|
- `~/.claude/plugins/cache/rdc-skills/rdc-skills/latest/.claude-plugin/plugin.json` ← installed cache
|
|
20
|
-
2. Read it. Parse JSON. Read `skills_meta`.
|
|
20
|
+
2. Read it. Parse JSON. Read `skills_meta`. Accept either the current object map shape (`skills_meta.<name>`) or an older array shape (`skills_meta[]`); normalize to entries before rendering.
|
|
21
21
|
3. Group entries by `category` in this order: `planning, build, deploy, release, dev-loop, reporting, tooling, infra`.
|
|
22
22
|
4. For each entry render one row: `{slash} — {usage} — needs: {requires.join(',')} — cf: {codeflow_required ? 'yes' : 'no'}`.
|
|
23
23
|
5. Print a Decision Tree at the end that maps natural-language phrases to skills, derived from each entry's `triggers[]`.
|
|
24
|
-
6. Print the Direct MCP / curl Access block
|
|
24
|
+
6. Print the Direct MCP / curl Access block exactly as shown in the required output format. Do not summarize, compress, relabel, or omit the curl examples. This block is the public direct-caller contract for MCP/API/curl users.
|
|
25
25
|
7. Hard rules section at the bottom (see below — verbatim).
|
|
26
26
|
|
|
27
27
|
## Required output format
|
|
@@ -128,6 +128,8 @@ RDC SKILLS — manifest: .claude-plugin/plugin.json @ v{version}
|
|
|
128
128
|
-H 'Accept: application/json, text/event-stream' \
|
|
129
129
|
-d '{"jsonrpc":"2.0","id":4,"method":"tools/call","params":{"name":"rdc_skill_get","arguments":{"name":"rdc:build","variant":"cli","format":"json"}}}'
|
|
130
130
|
|
|
131
|
+
REQUIRED EXACT TOKEN: "format":"json"
|
|
132
|
+
|
|
131
133
|
## Hard rules
|
|
132
134
|
|
|
133
135
|
- `cf: yes` skills MUST consult CodeFlow before acting. Hooks fail-closed on unreachable CodeFlow for these.
|
|
@@ -23,7 +23,7 @@ description: "Usage `rdc:self-test [--strict]` — Validate all rdc:* skills, pl
|
|
|
23
23
|
| Tier | What it checks | Status |
|
|
24
24
|
|------|----------------|--------|
|
|
25
25
|
| Tier 1 | Static lint — frontmatter, Usage line, referenced files, name match | ✅ live |
|
|
26
|
-
| Tier 2 | Behavioral — headless Claude runs each skill in sandbox, asserts artifacts | ✅ live — 29 manifests; acceptance harness records transcripts, tool calls, artifacts, and lessons learned |
|
|
26
|
+
| Tier 2 | Behavioral — headless Claude or Codex runs each skill in sandbox, asserts artifacts | ✅ live — 29 manifests; acceptance harness records transcripts, tool calls, artifacts, and lessons learned |
|
|
27
27
|
| Tier 3 | Golden checklists — snapshot output format, regress on drift | 🔒 future |
|
|
28
28
|
|
|
29
29
|
## Interactive UI
|
|
@@ -85,10 +85,11 @@ Test output streams live to the terminal. No server, no extra processes.
|
|
|
85
85
|
|
|
86
86
|
## Procedure (Tier 2)
|
|
87
87
|
|
|
88
|
-
Tier 2 runs each skill end-to-end in an isolated sandbox and asserts on observed state (files touched, commits made, work items, exit code). Build acceptance additionally records all observable engine events/tool calls, assistant output, stdout/stderr artifacts, lessons learned, and next build optimizations. Use it before shipping behavioral changes — Tier 1 alone can't catch runtime drift.
|
|
88
|
+
Tier 2 runs each skill end-to-end in an isolated sandbox and asserts on observed state (files touched, commits made, work items, exit code). Build acceptance additionally records all observable engine events/tool calls, assistant output, stdout/stderr artifacts, lessons learned, and next build optimizations. It supports `--engine claude` and `--engine codex`; both engines use the same manifests, worktree sandbox, JSONL evidence, and Markdown report. Use it before shipping behavioral changes — Tier 1 alone can't catch runtime drift.
|
|
89
89
|
|
|
90
90
|
1. **Prerequisites:**
|
|
91
|
-
- `claude` CLI on PATH (headless mode: `claude --print`)
|
|
91
|
+
- `claude` CLI on PATH for Claude runs (headless mode: `claude --print`)
|
|
92
|
+
- `codex` CLI on PATH for Codex runs (headless mode: `codex exec --json`)
|
|
92
93
|
- clauth daemon unlocked (`curl -s http://127.0.0.1:52437/ping`)
|
|
93
94
|
- Supabase MCP reachable (runner creates a throwaway test branch)
|
|
94
95
|
- Clean git tree in `rdc-skills` (worktrees are added under `.rdc/sandbox/<run-id>/`)
|
|
@@ -100,12 +101,13 @@ Tier 2 runs each skill end-to-end in an isolated sandbox and asserts on observed
|
|
|
100
101
|
node scripts/self-test.mjs --tier2 --parallel 3 # up to 3 skills in parallel
|
|
101
102
|
node scripts/self-test.mjs --tier2 --quick # skip long-running assertions
|
|
102
103
|
node scripts/acceptance.mjs --skill rdc:build # build acceptance with JSONL/tool-call evidence
|
|
104
|
+
node scripts/acceptance.mjs --engine codex --skill rdc:build
|
|
103
105
|
```
|
|
104
106
|
|
|
105
107
|
3. **What it does:**
|
|
106
108
|
- Runs Tier 1 as a pre-gate (fails fast if static lint fails)
|
|
107
109
|
- Creates one Supabase test branch for the run
|
|
108
|
-
- For each skill: `git worktree add` into `.rdc/sandbox/<run-id>/<skill>/`, sets `RDC_TEST=1`, invokes
|
|
110
|
+
- For each skill: `git worktree add` into `.rdc/sandbox/<run-id>/<skill>/`, sets `RDC_TEST=1`, invokes the selected headless engine with the skill prompt, waits for exit
|
|
109
111
|
- Asserts per the skill's manifest: exit code, files touched, commits made, stdout patterns
|
|
110
112
|
- Build acceptance writes JSONL evidence and extracts tool calls from the engine stream
|
|
111
113
|
- Cleans up worktrees + deletes the Supabase branch at the end (even on failure)
|
package/skills/tests/MATRIX.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Current coverage: 29 manifests for 29 skill directories.
|
|
4
4
|
|
|
5
|
-
The manifest layer verifies each skill can be started from a realistic caller prompt in an isolated `RDC_TEST=1` sandbox. The acceptance harness records the engine stream, extracted tool calls, stdout/stderr artifacts, rendered assistant output, failures, lessons learned, and next build optimizations under `.rdc/reports/`.
|
|
5
|
+
The manifest layer verifies each skill can be started from a realistic caller prompt in an isolated `RDC_TEST=1` sandbox. The acceptance harness can run either Claude (`--engine claude`) or Codex (`--engine codex`) against the same manifests and records the engine stream, extracted tool calls, stdout/stderr artifacts, rendered assistant output, failures, lessons learned, and next build optimizations under `.rdc/reports/`.
|
|
6
6
|
|
|
7
7
|
`rdc:channel-formatter` currently has the strongest content acceptance fixture: it asserts source-grounded social-pack output, forbidden-claim absence, and observable corpus or web-search tool routing. `rdc:help` asserts the public MCP/curl caller surface and structured `format:"json"` discovery path. `rdc:self-test` validates the test-tier/evidence language. Planning/reporting skills (`rdc:preplan`, `rdc:plan`, `rdc:handoff`, `rdc:prototype`, `rdc:review`, `rdc:report`) assert their required artifacts and safety boundaries. Operational/content skills (`rdc:brochure`, `rdc:convert`, `lifeai-brochure-author`, `rdc:build`, `rdc:fixit`, `rdc:fs-mcp`, `rdc:workitems`) assert artifact paths, dispatch gates, scoped commits, filesystem boundaries, and sandboxed work-item behavior. Coordination/design/maintenance skills (`rdc:co-develop`, `rdc:collab`, `rdc:design`, `rdc:extract-verifier-rules`, `rdc:housekeeping`, `rdc:overnight`, `rdc:rpms-filemap`) assert relay safety, token/reference use, verifier corpus safety, maintenance/reporting, unattended sentinel behavior, and canonical file-map rules. All manifests now include a deeper acceptance block beyond the basic executable smoke fixture.
|
|
8
8
|
|
|
@@ -48,6 +48,6 @@ The manifest layer verifies each skill can be started from a realistic caller pr
|
|
|
48
48
|
|
|
49
49
|
## Next Coverage Upgrades
|
|
50
50
|
|
|
51
|
-
-
|
|
52
|
-
- Add artifact
|
|
53
|
-
-
|
|
51
|
+
- Run full scheduled Claude and Codex sweeps regularly, not only changed-skill acceptance, so transcripts and lessons stay fresh.
|
|
52
|
+
- Add richer artifact-content checks for output-producing skills (`rdc:brochure`, `rdc:convert`, `rdc:prototype`, `rdc:report`).
|
|
53
|
+
- Promote recurring transcript review findings into manifest assertions instead of relying on human inspection.
|
package/skills/tests/README.md
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Each RDC skill directory has a colocated test manifest at `skills/tests/rdc-<name>.test.json`.
|
|
4
4
|
There are currently 29 manifests for 29 skill directories. These drive the Tier 2
|
|
5
|
-
behavioral runner and the build acceptance harness.
|
|
5
|
+
behavioral runner and the build acceptance harness for both Claude and Codex.
|
|
6
6
|
|
|
7
7
|
See `MATRIX.md` for the current skill-by-skill coverage table.
|
|
8
8
|
|
|
@@ -32,8 +32,9 @@ is the source of truth. Top-level fields:
|
|
|
32
32
|
1. Start from an existing manifest in this dir
|
|
33
33
|
2. Set `fixture.env.RDC_TEST = "1"` (required — sandbox contract)
|
|
34
34
|
3. Write assertions tight enough to catch real regressions but not flaky
|
|
35
|
-
4. Run `node scripts/acceptance.mjs --skill rdc:<name>` to capture the transcript, tool calls, and lessons learned
|
|
36
|
-
5.
|
|
35
|
+
4. Run `node scripts/acceptance.mjs --skill rdc:<name>` to capture the Claude transcript, tool calls, and lessons learned
|
|
36
|
+
5. When Codex behavior matters, also run `node scripts/acceptance.mjs --engine codex --skill rdc:<name>` against the same manifest
|
|
37
|
+
6. Commit manifest + any skill changes together
|
|
37
38
|
|
|
38
39
|
For content-producing skills, include both positive and negative output checks.
|
|
39
40
|
Use `stdout_contains` for expected channel-native structure and source facts;
|
|
@@ -21,12 +21,13 @@ const missing = spawnSync(process.execPath, [script, '--skill', 'rdc:not-a-real-
|
|
|
21
21
|
assert.equal(missing.status, 1);
|
|
22
22
|
assert.match(missing.stderr, /missing acceptance manifest/);
|
|
23
23
|
|
|
24
|
-
const codex = spawnSync(process.execPath, [script, '--engine', 'codex', '--skill', 'rdc:
|
|
24
|
+
const codex = spawnSync(process.execPath, [script, '--engine', 'codex', '--skill', 'rdc:not-a-real-skill'], {
|
|
25
25
|
cwd: REPO_ROOT,
|
|
26
26
|
encoding: 'utf8',
|
|
27
27
|
});
|
|
28
|
-
assert.equal(codex.status,
|
|
29
|
-
assert.
|
|
28
|
+
assert.equal(codex.status, 1);
|
|
29
|
+
assert.doesNotMatch(codex.stderr, /not wired/i);
|
|
30
|
+
assert.match(codex.stderr, /missing acceptance manifest/);
|
|
30
31
|
|
|
31
32
|
const emptyProject = mkdtempSync(join(tmpdir(), 'rdc-acceptance-empty-'));
|
|
32
33
|
try {
|