cyber-aced 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +18 -0
- package/.plugin/pins.json +1 -0
- package/.plugin/plugin.json +18 -0
- package/agents/aced-case-judge.md +165 -0
- package/agents/aced-impl-judge.md +55 -0
- package/agents/aced-scenario-writer.md +76 -0
- package/agents/aced-spec-validator.md +66 -0
- package/governances/README.md +52 -0
- package/governances/agent-tool-output.md +86 -0
- package/governances/cli-resolution.md +102 -0
- package/governances/skill-design.md +258 -0
- package/governances/skill-repo-structure.md +173 -0
- package/package.json +44 -0
- package/readme.md +124 -0
- package/skills/aced-architect-impl/README.md +9 -0
- package/skills/aced-architect-impl/SKILL.md +73 -0
- package/skills/aced-builder-impl/README.md +14 -0
- package/skills/aced-builder-impl/SKILL.md +46 -0
- package/skills/aced-builder-spec/README.md +15 -0
- package/skills/aced-builder-spec/SKILL.md +130 -0
- package/skills/aced-fit/README.md +21 -0
- package/skills/aced-fit/SKILL.md +105 -0
- package/skills/aced-impl-producer/SKILL.md +75 -0
- package/skills/add-scenario/README.md +25 -0
- package/skills/add-scenario/SKILL.md +75 -0
- package/skills/bench/README.md +64 -0
- package/skills/bench/SKILL.md +199 -0
- package/skills/bench/scripts/bench-claude-code.mts +187 -0
- package/skills/bench/scripts/bench-compare.mts +369 -0
- package/skills/bench/scripts/bench-record.mts +133 -0
- package/skills/bench/scripts/bench.mts +930 -0
- package/skills/check-freshness/README.md +57 -0
- package/skills/check-freshness/SKILL.md +76 -0
- package/skills/check-freshness/scripts/check-freshness.mts +399 -0
- package/skills/compare/README.md +25 -0
- package/skills/compare/SKILL.md +87 -0
- package/skills/contribute-skill/README.md +25 -0
- package/skills/contribute-skill/SKILL.md +241 -0
- package/skills/contribute-skill/scripts/source.mts +188 -0
- package/skills/define-agent/README.md +40 -0
- package/skills/define-agent/SKILL.md +279 -0
- package/skills/define-command/README.md +36 -0
- package/skills/define-command/SKILL.md +174 -0
- package/skills/define-governance/SKILL.md +151 -0
- package/skills/define-skill/README.md +37 -0
- package/skills/define-skill/SKILL.md +200 -0
- package/skills/extract-situation/README.md +32 -0
- package/skills/extract-situation/SKILL.md +54 -0
- package/skills/extract-situation/scripts/extract-situation.mts +287 -0
- package/skills/improve/README.md +25 -0
- package/skills/improve/SKILL.md +100 -0
- package/skills/improve-skill/README.md +38 -0
- package/skills/improve-skill/SKILL.md +169 -0
- package/skills/improve-skill/references/check-definitions.md +215 -0
- package/skills/improve-skill/scripts/validate.mts +1033 -0
- package/skills/init-aced/README.md +5 -0
- package/skills/init-aced/SKILL.md +76 -0
- package/skills/init-aced/scripts/ensure-results-ignored.mts +117 -0
- package/skills/list-skills/README.md +19 -0
- package/skills/list-skills/SKILL.md +55 -0
- package/skills/list-skills/scripts/list-skills.mts +210 -0
- package/skills/manage/README.md +35 -0
- package/skills/manage/SKILL.md +65 -0
- package/skills/manage-model-runners/README.md +25 -0
- package/skills/manage-model-runners/SKILL.md +113 -0
- package/skills/manage-skill-dirs/README.md +17 -0
- package/skills/manage-skill-dirs/SKILL.md +54 -0
- package/skills/manage-skill-dirs/scripts/manage-skill-dirs.mts +328 -0
- package/skills/repair-private-skills/README.md +17 -0
- package/skills/repair-private-skills/SKILL.md +59 -0
- package/skills/repair-private-skills/scripts/repair-private-skills.mts +279 -0
- package/skills/report/README.md +25 -0
- package/skills/report/SKILL.md +74 -0
- package/skills/run/README.md +25 -0
- package/skills/run/SKILL.md +220 -0
- package/skills/skillify/README.md +25 -0
- package/skills/skillify/SKILL.md +141 -0
|
@@ -0,0 +1,57 @@
|
|
|
1
|
+
# check-freshness
|
|
2
|
+
|
|
3
|
+
Internal, non-invokable ACED engine. It decides whether a recorded eval result still describes the
|
|
4
|
+
configuration on disk, and returns `current`, `stale`, `incomplete`, or `absent`.
|
|
5
|
+
|
|
6
|
+
## What it does
|
|
7
|
+
|
|
8
|
+
`run` writes an eval result to `.agents/aced/results/<target>/<timestamp>.json` carrying an
|
|
9
|
+
`evaluated` set — every input the run reports consuming, each as a repository path plus a SHA-256
|
|
10
|
+
hash. This engine resolves a node's target from its `eval.md`, selects the newest **readable** result
|
|
11
|
+
whose own `target` field is that subject, re-hashes each recorded path, and compares.
|
|
12
|
+
|
|
13
|
+
Two rules do the work:
|
|
14
|
+
|
|
15
|
+
- A **file** entry hashes the bytes read, so a content change is detectable and a timestamp never
|
|
16
|
+
stands in for one.
|
|
17
|
+
- A **directory** entry hashes the **names** the listing returned, so a file *added* to a directory
|
|
18
|
+
the subject loads from is detectable without re-resolving what the subject depends on now.
|
|
19
|
+
|
|
20
|
+
A directory the run expanded carries both its own entry and a `file` entry per file the listing
|
|
21
|
+
yielded — the directory hash covers names only, so without the per-file entries an edit inside that
|
|
22
|
+
directory would read `current` forever.
|
|
23
|
+
|
|
24
|
+
## Why the record and not the tree
|
|
25
|
+
|
|
26
|
+
The first attempt at this capability inferred the subject's dependencies from outside: it guessed
|
|
27
|
+
the file set from a slug, treated modification time as a content change, and read a judge's prose
|
|
28
|
+
with a regex. All three guesses trace to one missing upstream contract — the producer recorded
|
|
29
|
+
nothing about what it evaluated. Fixing the producer turns freshness into a comparison instead of
|
|
30
|
+
four stacked heuristics, which is why every decision here reads only what the record names.
|
|
31
|
+
|
|
32
|
+
## Selection is by what the result records
|
|
33
|
+
|
|
34
|
+
The newest result is the one whose **recorded timestamp** is greatest, not the one whose filename
|
|
35
|
+
sorts last, and a result belongs to the target its own `target` field names, not the directory it
|
|
36
|
+
happens to sit in. An unparseable result is skipped and named rather than thrown on, so one corrupt
|
|
37
|
+
record cannot blind the check to the readable ones beside it.
|
|
38
|
+
|
|
39
|
+
## Limits it states rather than hides
|
|
40
|
+
|
|
41
|
+
`current` means the run's account still holds — not that the run read everything it should have. An
|
|
42
|
+
under-reporting run records a shorter set whose entries all match, and no verdict here can find it,
|
|
43
|
+
because the only witness to what was read is the same self-report under test. See the trust boundary
|
|
44
|
+
in `SKILL.md`.
|
|
45
|
+
|
|
46
|
+
Nothing consults this engine yet: wiring `run` and `improve` to it is `#476`. Until that lands it is
|
|
47
|
+
specified, tested, and reachable only by a human running it directly — a recorded gap, not an
|
|
48
|
+
oversight.
|
|
49
|
+
|
|
50
|
+
## Tests
|
|
51
|
+
|
|
52
|
+
`scripts/check-freshness.test.mts` — one test per frozen scenario, with fixtures chosen so a
|
|
53
|
+
guessing implementation cannot pass: the result is filed under a deliberately wrong directory name,
|
|
54
|
+
alphabetical filename order is the reverse of recorded timestamp order, and one fixture rewrites a
|
|
55
|
+
file's modification time while leaving its bytes identical.
|
|
56
|
+
|
|
57
|
+
Spec: `.agents/specs/aced/eval-run/check-freshness/`.
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: check-freshness
|
|
3
|
+
description: "Partial Skill: invoke by name only — the ACED eval-result freshness check — decides whether a recorded result still describes the configuration on disk, not user-triggered."
|
|
4
|
+
user-invocable: false
|
|
5
|
+
metadata:
|
|
6
|
+
internal: true
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# check-freshness
|
|
10
|
+
|
|
11
|
+
The **freshness check** — the deterministic engine that reads the newest eval result `run` recorded
|
|
12
|
+
for a target, re-hashes the paths that result recorded, and returns one of four verdicts. It is
|
|
13
|
+
**read-only**: it never writes, repairs, or deletes a result.
|
|
14
|
+
|
|
15
|
+
It answers the question **from the record itself**. `run` records the inputs it reports consuming and
|
|
16
|
+
their hashes; this re-hashes those same paths in the working tree and compares. Nothing is inferred:
|
|
17
|
+
no modification times, no guessed file sets, no guessed directory names.
|
|
18
|
+
|
|
19
|
+
## Invocation
|
|
20
|
+
|
|
21
|
+
```
|
|
22
|
+
node "<skill>/scripts/check-freshness.mts" --node <node-dir>
|
|
23
|
+
```
|
|
24
|
+
|
|
25
|
+
`<node-dir>` holds the node's `eval.md` (whose `subject:` names the target) and its frozen
|
|
26
|
+
`.feature`. Exit status is **zero only for `current`**; every other verdict, and every fail-closed
|
|
27
|
+
path, exits non-zero.
|
|
28
|
+
|
|
29
|
+
It also exposes the hashing routine, so `run` records entries with the same implementation that
|
|
30
|
+
reads them back — never a second one that merely agrees today:
|
|
31
|
+
|
|
32
|
+
```
|
|
33
|
+
node "<skill>/scripts/check-freshness.mts" --hash-file <path>
|
|
34
|
+
node "<skill>/scripts/check-freshness.mts" --hash-dir <path>
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
## The four verdicts
|
|
38
|
+
|
|
39
|
+
| Verdict | Means |
|
|
40
|
+
|---|---|
|
|
41
|
+
| `current` | Every input the run recorded still hashes as recorded. |
|
|
42
|
+
| `stale` | At least one recorded **subject input** changed, is gone, or (a directory) now lists different entries — no individual score can be relied on without re-running. |
|
|
43
|
+
| `incomplete` | Every subject input matches, but the recorded **frozen suite** changed — the scores survive, the suite grew past them. |
|
|
44
|
+
| `absent` | There is nothing to compare: no results directory, no result for this target, none readable, no evaluated set, or an evaluated set that contradicts the record carrying it. |
|
|
45
|
+
|
|
46
|
+
A node with no `eval.md`, or an `eval.md` with no `subject:` key, **fails closed and emits no
|
|
47
|
+
verdict** — a guard that cannot decide must not manufacture one, and `absent` would read as
|
|
48
|
+
"checked, nothing recorded".
|
|
49
|
+
|
|
50
|
+
## Two boundaries, both deliberate
|
|
51
|
+
|
|
52
|
+
**Closed world.** It compares the inputs the result recorded and nothing else; it never re-resolves
|
|
53
|
+
what the subject depends on *now*. So `current` means "every input this result recorded still hashes
|
|
54
|
+
as recorded", never "the subject has not grown". Growth is caught at the producer — `run` records a
|
|
55
|
+
hash over the entry names of every directory it listed. Growth reaching the subject by a path `run`
|
|
56
|
+
never consumed stays invisible by construction: it was not an input, so the result stays truthful
|
|
57
|
+
about what it measured.
|
|
58
|
+
|
|
59
|
+
**Trust boundary.** `evaluated` is what the run **reports** consuming, not a verified trace. An
|
|
60
|
+
under-reporting run records a shorter set whose entries all match, and this answers `current` with
|
|
61
|
+
full confidence — a silent failure pointing the unsafe way (`#475`). What is checkable is
|
|
62
|
+
**coherence**: a set omitting the `.feature` whose scenarios the record scores, or the configuration
|
|
63
|
+
its own `target` names, contradicts the record it accompanies and reads `absent`. That catches only
|
|
64
|
+
an *inconsistent* under-reporter; a uniform one escapes.
|
|
65
|
+
|
|
66
|
+
Reading `current` as "the run's account still holds" is sound. Reading it as "the run read everything
|
|
67
|
+
it should have" is not, and this engine never claims the second.
|
|
68
|
+
|
|
69
|
+
## Boundaries
|
|
70
|
+
|
|
71
|
+
Scores nothing (`run`), compares no two versions (`compare`), rolls nothing up (`report`), and does
|
|
72
|
+
not judge whether a recorded *pass* was well-founded (a judge-protocol question, `#477`). It decides
|
|
73
|
+
a verdict and reports it; **what a caller does with that verdict is the caller's** — wiring `run` and
|
|
74
|
+
`improve` to consult it is `#476`.
|
|
75
|
+
|
|
76
|
+
Spec: `.agents/specs/aced/eval-run/check-freshness/`.
|
|
@@ -0,0 +1,399 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
// check-freshness — is a recorded eval result still current?
|
|
3
|
+
//
|
|
4
|
+
// Reads the newest result `run` recorded for a node's target, re-hashes the paths that result
|
|
5
|
+
// recorded, and returns current / stale / incomplete / absent. Nothing is inferred: no modification
|
|
6
|
+
// times, no guessed file sets, no guessed directory names. The rejected first attempt at this
|
|
7
|
+
// capability guessed all three, which is why every decision here reads only what the record names.
|
|
8
|
+
//
|
|
9
|
+
// TWO BOUNDARIES, both deliberate, both stated in .agents/specs/aced/eval-run/check-freshness/:
|
|
10
|
+
//
|
|
11
|
+
// Closed world — this compares the inputs the result recorded and NOTHING else. It never
|
|
12
|
+
// re-resolves what the subject depends on now. So `current` means "every input this result
|
|
13
|
+
// recorded still hashes as recorded", never "the subject has not grown". Growth is caught at the
|
|
14
|
+
// producer: `run` records a hash over the entries of every directory it listed, so a file added
|
|
15
|
+
// to a directory the subject loads from changes a recorded entry. Growth reaching the subject by
|
|
16
|
+
// a path `run` never consumed stays invisible BY CONSTRUCTION — it was not an input.
|
|
17
|
+
//
|
|
18
|
+
// Trust boundary — `evaluated` is what the run REPORTS consuming, not a verified trace. An
|
|
19
|
+
// under-reporting run records a shorter set whose every entry then matches, and this returns
|
|
20
|
+
// `current` with full confidence. That failure is silent and points the unsafe way; see #475.
|
|
21
|
+
// What IS checkable is coherence: a set omitting the .feature whose scenarios the record scores,
|
|
22
|
+
// or the configuration its own `target` names, contradicts the record it accompanies and reads
|
|
23
|
+
// `absent`. That catches only an INCONSISTENT under-reporter — a uniform one escapes.
|
|
24
|
+
//
|
|
25
|
+
// This engine owns the hashing implementation for BOTH sides of the comparison. `run` computes its
|
|
26
|
+
// entries by invoking `--hash-file` / `--hash-dir` here rather than reimplementing the rule: two
|
|
27
|
+
// schemes that merely agree today would surface their divergence as a permanent `stale`, never as
|
|
28
|
+
// an error. The RULE is `run`'s contract (eval-run/run/); this is its single implementation.
|
|
29
|
+
//
|
|
30
|
+
// Pure functions are exported for node:test; running the file directly drives the CLI. No
|
|
31
|
+
// dependencies — plain node strips the types.
|
|
32
|
+
|
|
33
|
+
import { createHash } from 'node:crypto'
|
|
34
|
+
import { existsSync, readdirSync, readFileSync, realpathSync } from 'node:fs'
|
|
35
|
+
import { join, relative, resolve } from 'node:path'
|
|
36
|
+
import { pathToFileURL } from 'node:url'
|
|
37
|
+
|
|
38
|
+
// ─── types ────────────────────────────────────────────────────────────────────
|
|
39
|
+
|
|
40
|
+
export type Verdict = 'current' | 'stale' | 'incomplete' | 'absent'
|
|
41
|
+
|
|
42
|
+
// `kind` is what makes the two hash rules distinguishable at read time. Without it a directory
|
|
43
|
+
// entry is indistinguishable from a file entry and would be re-hashed as bytes, which fails for
|
|
44
|
+
// every directory and would read as a permanent `stale`.
|
|
45
|
+
export interface EvaluatedEntry {
|
|
46
|
+
path: string
|
|
47
|
+
sha256: string
|
|
48
|
+
kind: 'file' | 'directory'
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
export interface ResultRecord {
|
|
52
|
+
timestamp?: string
|
|
53
|
+
target?: string
|
|
54
|
+
scenarios?: unknown[]
|
|
55
|
+
evaluated?: EvaluatedEntry[]
|
|
56
|
+
}
|
|
57
|
+
|
|
58
|
+
export interface LoadedResult {
|
|
59
|
+
file: string
|
|
60
|
+
record: ResultRecord
|
|
61
|
+
}
|
|
62
|
+
|
|
63
|
+
export interface Decision {
|
|
64
|
+
verdict: Verdict
|
|
65
|
+
// Why, in one line, for the reader. Never a second machine channel — the exit code is that.
|
|
66
|
+
reason: string
|
|
67
|
+
// Recorded inputs that no longer match. Populated for stale/incomplete only: `absent` has
|
|
68
|
+
// nothing to compare and `current` has nothing that failed.
|
|
69
|
+
mismatched: string[]
|
|
70
|
+
skipped: string[]
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
// ─── hashing — the single routine both sides use ──────────────────────────────
|
|
74
|
+
|
|
75
|
+
/** A file entry hashes the bytes read. */
|
|
76
|
+
export function hashFile(abs: string): string {
|
|
77
|
+
return createHash('sha256').update(readFileSync(abs)).digest('hex')
|
|
78
|
+
}
|
|
79
|
+
|
|
80
|
+
/**
|
|
81
|
+
* A directory entry hashes the NAMES the listing returned, never their contents — that is what
|
|
82
|
+
* makes a file later ADDED to the directory detectable without re-resolving the subject.
|
|
83
|
+
*
|
|
84
|
+
* The canonical form is load-bearing: entry names (not paths), sorted, joined by \n, no trailing
|
|
85
|
+
* newline. `readdirSync` order is filesystem-dependent, so an unsorted join would hash differently
|
|
86
|
+
* on two machines holding identical trees and report `stale` for nothing.
|
|
87
|
+
*/
|
|
88
|
+
export function hashDirListing(abs: string): string {
|
|
89
|
+
return createHash('sha256')
|
|
90
|
+
.update(canonicalListing(readdirSync(abs)))
|
|
91
|
+
.digest('hex')
|
|
92
|
+
}
|
|
93
|
+
|
|
94
|
+
/**
|
|
95
|
+
* The canonical form of a listing: sorted, `\n`-joined, no trailing newline.
|
|
96
|
+
*
|
|
97
|
+
* Split out as a pure function so the ordering guarantee is testable WITHOUT a filesystem. It is not
|
|
98
|
+
* otherwise falsifiable here: ext4 returns `readdirSync` already sorted, so deleting the sort keeps
|
|
99
|
+
* the whole suite green on Linux while handing every APFS and NTFS user a permanent spurious `stale`
|
|
100
|
+
* — the divergence would surface as a wrong verdict, never as an error.
|
|
101
|
+
*/
|
|
102
|
+
export function canonicalListing(names: string[]): string {
|
|
103
|
+
return [...names].sort().join('\n')
|
|
104
|
+
}
|
|
105
|
+
|
|
106
|
+
// ─── eval.md — resolve the target ─────────────────────────────────────────────
|
|
107
|
+
|
|
108
|
+
/**
|
|
109
|
+
* Pulls `subject:` out of an eval.md's frontmatter. Deliberately not a YAML parser: the one key
|
|
110
|
+
* that matters is a scalar path, and a dependency-free engine that reads one key cannot be broken
|
|
111
|
+
* by an unrelated key's syntax.
|
|
112
|
+
*/
|
|
113
|
+
export function readSubject(evalText: string): string | null {
|
|
114
|
+
const fm = /^---\r?\n([\s\S]*?)\r?\n---/.exec(evalText)
|
|
115
|
+
const body = fm ? fm[1] : evalText
|
|
116
|
+
for (const line of body.split(/\r?\n/)) {
|
|
117
|
+
const m = /^subject:\s*(.+?)\s*$/.exec(line)
|
|
118
|
+
if (!m) continue
|
|
119
|
+
const raw = m[1].replace(/^["']|["']$/g, '').trim()
|
|
120
|
+
return raw === '' ? null : raw
|
|
121
|
+
}
|
|
122
|
+
return null
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
/** The node's own frozen suite — one member of the evaluated set, handled apart from the rest. */
|
|
126
|
+
export function findFrozenFeature(nodeDir: string): string | null {
|
|
127
|
+
const hit = readdirSync(nodeDir).find((n) => n.endsWith('.feature'))
|
|
128
|
+
return hit ? join(nodeDir, hit) : null
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
// ─── result selection ─────────────────────────────────────────────────────────
|
|
132
|
+
|
|
133
|
+
/** Every `*.json` under the results tree, at any depth. */
|
|
134
|
+
export function collectResultFiles(resultsDir: string): string[] {
|
|
135
|
+
const out: string[] = []
|
|
136
|
+
const walk = (dir: string) => {
|
|
137
|
+
for (const e of readdirSync(dir, { withFileTypes: true })) {
|
|
138
|
+
const p = join(dir, e.name)
|
|
139
|
+
if (e.isDirectory()) walk(p)
|
|
140
|
+
else if (e.name.endsWith('.json')) out.push(p)
|
|
141
|
+
}
|
|
142
|
+
}
|
|
143
|
+
walk(resultsDir)
|
|
144
|
+
return out.sort()
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
export interface Scan {
|
|
148
|
+
forTarget: LoadedResult[]
|
|
149
|
+
unreadable: string[]
|
|
150
|
+
}
|
|
151
|
+
|
|
152
|
+
/**
|
|
153
|
+
* Partitions results by the target each one RECORDS — never by the directory it sits in. A result
|
|
154
|
+
* filed under a mis-slugged directory still belongs to the target its own `target` field names, and
|
|
155
|
+
* an implementation matching on directory name would silently miss it.
|
|
156
|
+
*
|
|
157
|
+
* An unparseable file is collected rather than thrown on: one corrupt record must not blind the
|
|
158
|
+
* check to the readable ones beside it.
|
|
159
|
+
*/
|
|
160
|
+
export function scanResults(files: string[], target: string, read: (f: string) => string): Scan {
|
|
161
|
+
const forTarget: LoadedResult[] = []
|
|
162
|
+
const unreadable: string[] = []
|
|
163
|
+
for (const file of files) {
|
|
164
|
+
let record: ResultRecord
|
|
165
|
+
try {
|
|
166
|
+
record = JSON.parse(read(file)) as ResultRecord
|
|
167
|
+
} catch {
|
|
168
|
+
unreadable.push(file)
|
|
169
|
+
continue
|
|
170
|
+
}
|
|
171
|
+
if (record && record.target === target) forTarget.push({ file, record })
|
|
172
|
+
}
|
|
173
|
+
return { forTarget, unreadable }
|
|
174
|
+
}
|
|
175
|
+
|
|
176
|
+
/**
|
|
177
|
+
* Newest by RECORDED timestamp, not by filename. Filenames sort lexicographically and a timestamp
|
|
178
|
+
* is the fact that matters; where the two disagree, trusting the name picks the wrong record.
|
|
179
|
+
*/
|
|
180
|
+
export function selectNewest(results: LoadedResult[]): LoadedResult | null {
|
|
181
|
+
let best: LoadedResult | null = null
|
|
182
|
+
for (const r of results) {
|
|
183
|
+
const t = r.record.timestamp ?? ''
|
|
184
|
+
if (!best || t > (best.record.timestamp ?? '')) best = r
|
|
185
|
+
}
|
|
186
|
+
return best
|
|
187
|
+
}
|
|
188
|
+
|
|
189
|
+
// ─── coherence — the conditional oracle ───────────────────────────────────────
|
|
190
|
+
|
|
191
|
+
/**
|
|
192
|
+
* Taking the record's OWN `scenarios` and `target` as given, does its evaluated set cover the
|
|
193
|
+
* inputs they imply were read? Scores cannot come from a `.feature` never read, and a configuration
|
|
194
|
+
* cannot be judged unopened.
|
|
195
|
+
*
|
|
196
|
+
* This is a CONDITIONAL relation, never a proof: `scenarios` and `target` are written by the same
|
|
197
|
+
* agent and self-reported exactly as `evaluated` is, so a fabricated-but-self-consistent record
|
|
198
|
+
* passes. It catches the inconsistent under-reporter only.
|
|
199
|
+
*/
|
|
200
|
+
export function incoherence(record: ResultRecord, featureRel: string, targetRel: string): string | null {
|
|
201
|
+
const paths = new Set((record.evaluated ?? []).map((e) => e.path))
|
|
202
|
+
if ((record.scenarios?.length ?? 0) > 0 && !paths.has(featureRel)) {
|
|
203
|
+
return `the recorded provenance contradicts the result it accompanies: it scores ${record.scenarios?.length} scenarios but carries no entry for ${featureRel}`
|
|
204
|
+
}
|
|
205
|
+
if (record.target && !paths.has(targetRel)) {
|
|
206
|
+
return `the recorded provenance omits the configuration the result names: no entry for ${targetRel}`
|
|
207
|
+
}
|
|
208
|
+
return null
|
|
209
|
+
}
|
|
210
|
+
|
|
211
|
+
// ─── the verdict ──────────────────────────────────────────────────────────────
|
|
212
|
+
|
|
213
|
+
/**
|
|
214
|
+
* Split the evaluated set into the frozen suite and the subject inputs, then compare.
|
|
215
|
+
*
|
|
216
|
+
* The ORDER of the two questions is the whole `incomplete` / `stale` distinction. A moved subject
|
|
217
|
+
* input means no individual score survives, so it wins outright. Only when every subject input is
|
|
218
|
+
* unmoved does a changed suite mean something narrower: the scores stand, the suite grew past them.
|
|
219
|
+
* Collapsing the two would throw away that surviving fact.
|
|
220
|
+
*/
|
|
221
|
+
export function decideVerdict(
|
|
222
|
+
record: ResultRecord,
|
|
223
|
+
repoRoot: string,
|
|
224
|
+
featureRel: string,
|
|
225
|
+
skipped: string[] = [],
|
|
226
|
+
): Decision {
|
|
227
|
+
const entries = record.evaluated ?? []
|
|
228
|
+
const subject = entries.filter((e) => e.path !== featureRel)
|
|
229
|
+
const suite = entries.filter((e) => e.path === featureRel)
|
|
230
|
+
|
|
231
|
+
const moved = (e: EvaluatedEntry): string | null => {
|
|
232
|
+
const abs = join(repoRoot, e.path)
|
|
233
|
+
if (!existsSync(abs)) return `${e.path} (missing from the tree)`
|
|
234
|
+
try {
|
|
235
|
+
const now = e.kind === 'directory' ? hashDirListing(abs) : hashFile(abs)
|
|
236
|
+
return now === e.sha256 ? null : `${e.path} (content changed)`
|
|
237
|
+
} catch {
|
|
238
|
+
return `${e.path} (unreadable)`
|
|
239
|
+
}
|
|
240
|
+
}
|
|
241
|
+
|
|
242
|
+
const subjectMoved = subject.map(moved).filter((m): m is string => m !== null)
|
|
243
|
+
if (subjectMoved.length > 0) {
|
|
244
|
+
return {
|
|
245
|
+
verdict: 'stale',
|
|
246
|
+
reason: 'an input the scores rest on has moved; no individual score can be relied on without re-running',
|
|
247
|
+
mismatched: subjectMoved,
|
|
248
|
+
skipped,
|
|
249
|
+
}
|
|
250
|
+
}
|
|
251
|
+
|
|
252
|
+
const suiteMoved = suite.map(moved).filter((m): m is string => m !== null)
|
|
253
|
+
if (suiteMoved.length > 0) {
|
|
254
|
+
return {
|
|
255
|
+
verdict: 'incomplete',
|
|
256
|
+
reason:
|
|
257
|
+
'every input the scores rest on is unmoved, but the suite changed past them, so the result no longer covers the whole suite',
|
|
258
|
+
mismatched: suiteMoved,
|
|
259
|
+
skipped,
|
|
260
|
+
}
|
|
261
|
+
}
|
|
262
|
+
|
|
263
|
+
return {
|
|
264
|
+
verdict: 'current',
|
|
265
|
+
reason: "every input the run recorded still hashes as recorded — the run's account still holds",
|
|
266
|
+
mismatched: [],
|
|
267
|
+
skipped,
|
|
268
|
+
}
|
|
269
|
+
}
|
|
270
|
+
|
|
271
|
+
// ─── repo root ────────────────────────────────────────────────────────────────
|
|
272
|
+
|
|
273
|
+
export function findRepoRoot(from: string): string | null {
|
|
274
|
+
let dir = resolve(from)
|
|
275
|
+
for (;;) {
|
|
276
|
+
if (existsSync(join(dir, 'pnpm-workspace.yaml')) || existsSync(join(dir, '.git'))) return dir
|
|
277
|
+
const up = resolve(dir, '..')
|
|
278
|
+
if (up === dir) return null
|
|
279
|
+
dir = up
|
|
280
|
+
}
|
|
281
|
+
}
|
|
282
|
+
|
|
283
|
+
export const RESULTS_DIR = join('.agents', 'aced', 'results')
|
|
284
|
+
|
|
285
|
+
// ─── CLI ──────────────────────────────────────────────────────────────────────
|
|
286
|
+
|
|
287
|
+
function report(d: Decision): void {
|
|
288
|
+
process.stdout.write(`verdict: ${d.verdict}\n`)
|
|
289
|
+
process.stdout.write(`${d.reason}\n`)
|
|
290
|
+
for (const s of d.skipped) process.stdout.write(`skipped (unreadable): ${s}\n`)
|
|
291
|
+
for (const m of d.mismatched) process.stdout.write(`no longer matching: ${m}\n`)
|
|
292
|
+
}
|
|
293
|
+
|
|
294
|
+
export function main(argv: string[]): number {
|
|
295
|
+
// Shared-hashing entry points. `run` calls these rather than reimplementing the rule.
|
|
296
|
+
const hashFileIdx = argv.indexOf('--hash-file')
|
|
297
|
+
if (hashFileIdx !== -1) {
|
|
298
|
+
const p = argv[hashFileIdx + 1]
|
|
299
|
+
if (!p) {
|
|
300
|
+
console.error('✗ --hash-file needs a path')
|
|
301
|
+
return 1
|
|
302
|
+
}
|
|
303
|
+
process.stdout.write(hashFile(resolve(p)) + '\n')
|
|
304
|
+
return 0
|
|
305
|
+
}
|
|
306
|
+
const hashDirIdx = argv.indexOf('--hash-dir')
|
|
307
|
+
if (hashDirIdx !== -1) {
|
|
308
|
+
const p = argv[hashDirIdx + 1]
|
|
309
|
+
if (!p) {
|
|
310
|
+
console.error('✗ --hash-dir needs a path')
|
|
311
|
+
return 1
|
|
312
|
+
}
|
|
313
|
+
process.stdout.write(hashDirListing(resolve(p)) + '\n')
|
|
314
|
+
return 0
|
|
315
|
+
}
|
|
316
|
+
|
|
317
|
+
const nodeIdx = argv.indexOf('--node')
|
|
318
|
+
const nodeArg = nodeIdx === -1 ? undefined : argv[nodeIdx + 1]
|
|
319
|
+
if (!nodeArg) {
|
|
320
|
+
console.error('✗ usage: check-freshness --node <node-dir>')
|
|
321
|
+
return 1
|
|
322
|
+
}
|
|
323
|
+
const nodeDir = resolve(nodeArg)
|
|
324
|
+
|
|
325
|
+
// Fail closed: no eval.md and no subject key each emit NO verdict. A guard that cannot decide
|
|
326
|
+
// must not manufacture one — `absent` would read as "checked, nothing recorded".
|
|
327
|
+
const evalPath = join(nodeDir, 'eval.md')
|
|
328
|
+
if (!existsSync(evalPath)) {
|
|
329
|
+
console.error(`✗ ${relative(process.cwd(), evalPath)}: no eval.md in the node directory; no verdict`)
|
|
330
|
+
return 1
|
|
331
|
+
}
|
|
332
|
+
const subject = readSubject(readFileSync(evalPath, 'utf8'))
|
|
333
|
+
if (!subject) {
|
|
334
|
+
console.error(`✗ ${relative(process.cwd(), evalPath)}: eval.md carries no subject key; no verdict`)
|
|
335
|
+
return 1
|
|
336
|
+
}
|
|
337
|
+
|
|
338
|
+
const repoRoot = findRepoRoot(nodeDir)
|
|
339
|
+
if (!repoRoot) {
|
|
340
|
+
console.error('✗ no repository root above the node directory; no verdict')
|
|
341
|
+
return 1
|
|
342
|
+
}
|
|
343
|
+
|
|
344
|
+
process.stdout.write(`target: ${subject}\n`)
|
|
345
|
+
|
|
346
|
+
const featureAbs = findFrozenFeature(nodeDir)
|
|
347
|
+
const featureRel = featureAbs ? relative(repoRoot, featureAbs) : ''
|
|
348
|
+
|
|
349
|
+
const resultsDir = join(repoRoot, RESULTS_DIR)
|
|
350
|
+
if (!existsSync(resultsDir)) {
|
|
351
|
+
report({ verdict: 'absent', reason: 'no result is recorded anywhere', mismatched: [], skipped: [] })
|
|
352
|
+
return 1
|
|
353
|
+
}
|
|
354
|
+
|
|
355
|
+
const scan = scanResults(collectResultFiles(resultsDir), subject, (f) => readFileSync(f, 'utf8'))
|
|
356
|
+
const skipped = scan.unreadable.map((f) => relative(repoRoot, f))
|
|
357
|
+
|
|
358
|
+
if (scan.forTarget.length === 0) {
|
|
359
|
+
report({
|
|
360
|
+
verdict: 'absent',
|
|
361
|
+
reason:
|
|
362
|
+
skipped.length > 0
|
|
363
|
+
? 'every result recorded for this target is unreadable'
|
|
364
|
+
: 'no result is recorded for this target',
|
|
365
|
+
mismatched: [],
|
|
366
|
+
skipped,
|
|
367
|
+
})
|
|
368
|
+
return 1
|
|
369
|
+
}
|
|
370
|
+
|
|
371
|
+
const newest = selectNewest(scan.forTarget)!
|
|
372
|
+
if (!newest.record.evaluated) {
|
|
373
|
+
report({
|
|
374
|
+
verdict: 'absent',
|
|
375
|
+
reason: 'the result carries no recorded provenance',
|
|
376
|
+
mismatched: [],
|
|
377
|
+
skipped,
|
|
378
|
+
})
|
|
379
|
+
return 1
|
|
380
|
+
}
|
|
381
|
+
|
|
382
|
+
const bad = incoherence(newest.record, featureRel, subject)
|
|
383
|
+
if (bad) {
|
|
384
|
+
report({ verdict: 'absent', reason: bad, mismatched: [], skipped })
|
|
385
|
+
return 1
|
|
386
|
+
}
|
|
387
|
+
|
|
388
|
+
const decision = decideVerdict(newest.record, repoRoot, featureRel, skipped)
|
|
389
|
+
report(decision)
|
|
390
|
+
return decision.verdict === 'current' ? 0 : 1
|
|
391
|
+
}
|
|
392
|
+
|
|
393
|
+
// `import.meta.main` is Node >=24.2 and `undefined` on this repo's >=22 floor, where the CLI would
|
|
394
|
+
// never run — printing nothing and exiting 0, which a caller keying on the exit code would read as
|
|
395
|
+
// `current`. `pathToFileURL` rather than a `file://` concat: `import.meta.url` percent-encodes, so
|
|
396
|
+
// the naive form mismatches on any install path holding a space and reproduces the same bug.
|
|
397
|
+
if (process.argv[1] && import.meta.url === pathToFileURL(realpathSync(process.argv[1])).href) {
|
|
398
|
+
process.exit(main(process.argv.slice(2)))
|
|
399
|
+
}
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# compare
|
|
2
|
+
|
|
3
|
+
Run the frozen `.feature` suite against two versions of a target agent configuration and diff the results — the regression check before committing a change.
|
|
4
|
+
|
|
5
|
+
## When to use
|
|
6
|
+
|
|
7
|
+
Use this skill when you've edited a skill, subagent, command, or AGENTS.md section and want to confirm the edit actually helped before committing it.
|
|
8
|
+
|
|
9
|
+
Good triggers include:
|
|
10
|
+
|
|
11
|
+
- "Did this change actually improve things?"
|
|
12
|
+
- "Check for regressions before I commit"
|
|
13
|
+
- Verifying an `improve`-proposed edit worked as intended
|
|
14
|
+
|
|
15
|
+
Defer to a sibling when you just want a single version's score (`run`), or you're diagnosing why a version fails rather than diffing two of them (`improve`).
|
|
16
|
+
|
|
17
|
+
## What it does
|
|
18
|
+
|
|
19
|
+
Scores the frozen `.feature` suite against a "before" version (default: previous git revision) and an "after" version (default: current working tree), computes per-scenario and aggregate deltas, and reports improved / regressed / unchanged scenarios with an explicit regression warning if any scenario got worse.
|
|
20
|
+
|
|
21
|
+
## Install
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
npx skills add cyberuni/cyberplace --skill aced/compare
|
|
25
|
+
```
|
|
@@ -0,0 +1,87 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: compare
|
|
3
|
+
description: Use this skill when comparing two versions of an agent configuration to detect regressions or confirm improvements before committing a change.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# ACED Compare
|
|
7
|
+
|
|
8
|
+
Run the frozen `.feature` suite against two versions of a target agent configuration and diff the results.
|
|
9
|
+
|
|
10
|
+
## Identify the two versions
|
|
11
|
+
|
|
12
|
+
Default: current working tree vs. previous git revision.
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
git show HEAD:<relative-path-to-agent-config> > /tmp/aced-compare-before.md
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The user may also provide:
|
|
19
|
+
- Two explicit file paths
|
|
20
|
+
- A git ref (branch, commit SHA, tag) for the "before" version
|
|
21
|
+
|
|
22
|
+
Read both versions in full before proceeding. **If the "before" version cannot be read (an unresolvable git ref or a missing path), report that the version cannot be resolved and stop — score neither side.**
|
|
23
|
+
|
|
24
|
+
## Run evals on both versions
|
|
25
|
+
|
|
26
|
+
For each version, run every scenario in the node's frozen `<node>.feature` in the project spec through `aced-case-judge` (same process as `run`) — passing the `.feature` path and the scenario name, never the scenario body. The judge blinds its own simulating context; handing it the `Then` or the rubric would defeat that.
|
|
27
|
+
|
|
28
|
+
Label results as **before** and **after**.
|
|
29
|
+
|
|
30
|
+
Do not write to `results/` during compare — this is a diff operation, not a recorded run. Only write if the user explicitly asks to record.
|
|
31
|
+
|
|
32
|
+
When you do write one, the record carries `scoring_model` on the same terms `run`'s does: the model
|
|
33
|
+
you dispatched `aced-case-judge` under, or `unknown` when you cannot name it. One value covers the
|
|
34
|
+
whole comparison — a diff scores both sides in a single invocation, so one model scored both — and
|
|
35
|
+
`unknown` is a recorded value, never an omitted field.
|
|
36
|
+
|
|
37
|
+
## Compute the diff
|
|
38
|
+
|
|
39
|
+
For each scenario, record:
|
|
40
|
+
- Before and after **per dimension** (`score`/`max`), plus each side's total against the same maximum
|
|
41
|
+
- Before pass/fail, after pass/fail
|
|
42
|
+
- Change type: `improved` | `regressed` | `unchanged` | `now-passing` | `now-failing`
|
|
43
|
+
|
|
44
|
+
Both sides are scored against the **same frozen scenario**, so their maxima match and their totals
|
|
45
|
+
are comparable to each other. Totals are **not** comparable across different scenarios — maxima
|
|
46
|
+
differ per scenario, so never average raw totals into a headline number. Aggregate instead:
|
|
47
|
+
- Net pass rate delta (e.g., +3 scenarios passing)
|
|
48
|
+
- Count by change type
|
|
49
|
+
- The per-dimension deltas, which are what say *where* the change landed
|
|
50
|
+
|
|
51
|
+
A dimension delta is the useful signal: a total that holds steady while one dimension drops and
|
|
52
|
+
another rises is a real change the total hides.
|
|
53
|
+
|
|
54
|
+
## Report
|
|
55
|
+
|
|
56
|
+
```
|
|
57
|
+
ACED Compare — <name>
|
|
58
|
+
──────────────────────────────
|
|
59
|
+
Before: 18/22 passing (82%)
|
|
60
|
+
After: 21/22 passing (95%)
|
|
61
|
+
|
|
62
|
+
Net change: +3 passing
|
|
63
|
+
|
|
64
|
+
IMPROVED (now passing, or a dimension gained):
|
|
65
|
+
✓ no trigger for an audit request invoked no → yes (expected yes) now passing
|
|
66
|
+
✓ stages only related files 3/5 → 4/5 correctness 2/3 → 3/3 now passing
|
|
67
|
+
✓ red tests block the commit 3/5 → 5/5 correctness 1/3 → 3/3, completeness 2/2 → 2/2 now passing
|
|
68
|
+
|
|
69
|
+
REGRESSED (now failing, or a dimension lost):
|
|
70
|
+
none
|
|
71
|
+
|
|
72
|
+
UNCHANGED: 18 scenarios
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
## Regression gate
|
|
76
|
+
|
|
77
|
+
If any scenario regressed (a dimension dropped, the total dropped, or it flipped from pass to fail), warn explicitly:
|
|
78
|
+
|
|
79
|
+
```
|
|
80
|
+
⚠ REGRESSION DETECTED
|
|
81
|
+
✗ trigger on skill creation invoked no → yes (expected no) (was passing, now failing)
|
|
82
|
+
|
|
83
|
+
Do not commit this change until the regression is resolved.
|
|
84
|
+
Run improve to address it, or revert and try a different edit.
|
|
85
|
+
```
|
|
86
|
+
|
|
87
|
+
If no regressions and net improvement: confirm the change is safe to commit.
|
|
@@ -0,0 +1,25 @@
|
|
|
1
|
+
# contribute-skill
|
|
2
|
+
|
|
3
|
+
Contribute local improvements to an installed skill back to its source repository via pull request.
|
|
4
|
+
|
|
5
|
+
## When to use
|
|
6
|
+
|
|
7
|
+
Use this skill when you improved a skill installed from another repo and want to send the change upstream.
|
|
8
|
+
|
|
9
|
+
Good triggers include:
|
|
10
|
+
|
|
11
|
+
- "Patch this skill upstream"
|
|
12
|
+
- "Contribute my skill fix back to the source repo"
|
|
13
|
+
- After editing a skill under `~/.agents/skills/` or `.agents/skills/`
|
|
14
|
+
|
|
15
|
+
Do not use for skills native to the current repo under `skills/<name>/` — that repo is already the source.
|
|
16
|
+
|
|
17
|
+
## What it does
|
|
18
|
+
|
|
19
|
+
The skill maps local changes to `skills/<skill-name>/` in the upstream repo, opens a branch, and creates a PR. It never includes `SKILL.local.md` or paths outside the canonical skill folder.
|
|
20
|
+
|
|
21
|
+
## Install
|
|
22
|
+
|
|
23
|
+
```bash
|
|
24
|
+
npx skills add cyberuni/cyberplace --skill contribute-skill
|
|
25
|
+
```
|