@agentsfleet/orly 0.10.5 → 0.10.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +5 -5
- package/README.md +26 -8
- package/audits/agents-md.md +7 -3
- package/audits/spec-template.sh +30 -113
- package/audits/spec-template.ts +193 -0
- package/core/operating-model.md +5 -5
- package/dispatch/lifecycle.md +98 -42
- package/dispatch/verify.md +30 -48
- package/dispatch/write_spec.md +56 -82
- package/docs/ORLY_ARCHITECTURE.md +13 -3
- package/docs/RULE_ENFORCEMENT.md +2 -2
- package/docs/TEMPLATE.md +82 -77
- package/docs/VERIFY_TIERS.md +20 -26
- package/llms.txt +17 -12
- package/package.json +1 -1
- package/registry.json +4 -0
- package/skills/orly-spec-new/SKILL.md +3 -1
- package/skills/orly-write-integration-test/SKILL.md +1 -1
- package/skills/orly-write-unit-test/SKILL.md +1 -1
- package/src/cli.ts +14 -16
- package/src/criteria.test.ts +47 -6
- package/src/criteria.ts +30 -49
- package/src/criteria_spec.ts +53 -6
- package/src/criteria_support.ts +5 -4
- package/src/gates.test.ts +2 -2
- package/src/gates_spec.test.ts +40 -1
- package/src/gates_test_support.ts +18 -4
- package/src/install.ts +16 -81
- package/src/render.test.ts +5 -7
- package/src/setup.test.ts +116 -0
- package/src/spec_baseline.test.ts +96 -0
- package/src/spec_template.test.ts +109 -0
- package/src/spec_template_io.test.ts +97 -0
- package/src/validation.ts +9 -0
package/AGENTS.md
CHANGED
|
@@ -168,7 +168,7 @@ Guards fire pre-hoc regardless of lifecycle stage. Override: `<GATE>: SKIPPED pe
|
|
|
168
168
|
| Begin implementation OR branch carries spec changes in `pending/` | CHORE(open): `pending/`→`active/`, `Status: IN_PROGRESS` + `Branch:`, create worktree, commit on feature branch. **No code until 4 steps committed.** |
|
|
169
169
|
| Every commit during implementation | Mark completed Dimensions/Sections `DONE` same commit as the code. |
|
|
170
170
|
| All work complete, before PR | CHORE(close). |
|
|
171
|
-
|
|
|
171
|
+
| Final implementation commit, or explicit request to park a spec | CHORE(close) is mandatory next action; Section commits continue the stream. |
|
|
172
172
|
|
|
173
173
|
---
|
|
174
174
|
|
|
@@ -182,13 +182,13 @@ Non-trivial (full lifecycle) if it: touches >1 file · new abstraction · data m
|
|
|
182
182
|
|
|
183
183
|
**Anchor invariant — `orly gate` proves the boundary.** No PR opens unless every `orly gate pr` criterion is green or carries an `Orly-Override` trailer the user recorded with a reason.
|
|
184
184
|
|
|
185
|
-
**
|
|
185
|
+
**Gate ownership.** `work` runs the declared `conform` command at commit. `verify` runs documentation checks and non-test `verify.*` commands at push; it permits unfinished Sections. `pr` checks branch shape, exact upstream revision, spec completeness, and every declared `verify.*` command itself. `orly gate pr` is the command CHORE(close) runs; a push is never proof that tests ran. Unit and integration baselines are measured before the Pull Request against the recorded comparison revision. Full unit and integration suites run at the PR boundary, not at opening or every push. Integration and memory suites skip only on branches with no code. The authoritative sequence is `dispatch/lifecycle.md`; no spec means only spec criteria skip. A user-surface change without a docs change is red until docs land or an override says why not.
|
|
186
186
|
|
|
187
187
|
**LAND (after merge, or when the user confirms it merged):** pull the default branch, prune the merged worktree + branch, `make down` where defined.
|
|
188
188
|
|
|
189
189
|
### CHORE (open)
|
|
190
190
|
|
|
191
|
-
Spec `pending/`→`active/`; `Status: IN_PROGRESS`; `Branch:` set; **`Test Baseline:`
|
|
191
|
+
Spec `pending/`→`active/`; `Status: IN_PROGRESS`; `Branch:` set; **`Test Baseline:` declared pending; `Baseline revision:` records the full comparison commit** (unit and integration counts plus evidence are due before the Pull Request); committed. Worktree created, CWD inside (verify `pwd` + `git worktree list`), carrying the base checkout's uncommitted work in with it. No code until the 4 steps are committed.
|
|
192
192
|
|
|
193
193
|
### PLAN
|
|
194
194
|
|
|
@@ -210,8 +210,8 @@ In `agentsfleet` this stage is `make harness-verify`; its output block and end-o
|
|
|
210
210
|
|
|
211
211
|
### VERIFY
|
|
212
212
|
|
|
213
|
-
**
|
|
214
|
-
The `agentsfleet` output block and both cadences live in `docs/VERIFY_TIERS.md`. **FIRST: `/orly-write-unit-test`** audits diff coverage. **LAST: the Test Delta row** compares against the
|
|
213
|
+
**Section proof and final verification.** Follow `dispatch/lifecycle.md` for command timing. Section checks prove the changed Dimensions; the final gate runs the full declared verification set. A package-scoped command never replaces a declared boundary command, and a Section result never satisfies a repository-wide "tests pass" claim.
|
|
214
|
+
The `agentsfleet` output block and both cadences live in `docs/VERIFY_TIERS.md`. **FIRST: `/orly-write-unit-test`** audits diff coverage. **LAST: the Test Delta row** compares against the recorded comparison revision, measured before the Pull Request; zero or negative unit growth on a code-adding diff needs justification or a return to EXECUTE. Paste memory-leak evidence into PR Session Notes or cite its Continuous Integration (CI) URL. After refactors, list newly dead code before removing it.
|
|
215
215
|
|
|
216
216
|
### REVIEW
|
|
217
217
|
|
package/README.md
CHANGED
|
@@ -61,15 +61,33 @@ Works with Claude Code, Codex, OpenCode, and Amp.
|
|
|
61
61
|
Run this inside the repository you want governed.
|
|
62
62
|
|
|
63
63
|
```bash
|
|
64
|
-
|
|
64
|
+
bun add -g @agentsfleet/orly
|
|
65
|
+
orly init
|
|
65
66
|
```
|
|
66
67
|
|
|
67
68
|
orly scans your source, detects your languages, and installs only the rules that apply.
|
|
68
69
|
|
|
69
70
|
> [!TIP]
|
|
70
|
-
> Try `
|
|
71
|
+
> Try `orly init --dry-run` first. It previews the generated rules and changes nothing.
|
|
71
72
|
|
|
72
|
-
|
|
73
|
+
The Git hooks use the installed `orly` executable. Keep it on `PATH`.
|
|
74
|
+
Complete any setup items printed by `init`, then run `orly doctor` and commit the generated files.
|
|
75
|
+
Teammates get the rules on clone and run `orly init` to install their hooks.
|
|
76
|
+
|
|
77
|
+
Declare `conform` and at least one `verify.*` command in `.oracle/orly.json`.
|
|
78
|
+
Orly runs your repository's commands; the verification name does not require a particular language or test runner.
|
|
79
|
+
For a documentation repository whose Makefile provides `test` and `lint`:
|
|
80
|
+
|
|
81
|
+
```json
|
|
82
|
+
"commands": {
|
|
83
|
+
"conform": [["make", "test"]],
|
|
84
|
+
"verify.docs": [["make", "lint"]]
|
|
85
|
+
}
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
Here `test` checks documentation during work, and `lint` validates the site and links before the pull request.
|
|
89
|
+
Markdown and Markdown JSX (MDX) repositories do not need application unit or integration suites.
|
|
90
|
+
`orly doctor` checks configuration and installed files; run the gates to verify the commands themselves.
|
|
73
91
|
|
|
74
92
|
---
|
|
75
93
|
|
|
@@ -120,8 +138,8 @@ Three checkpoints, all running your own declared commands:
|
|
|
120
138
|
| Checkpoint | When | Effect |
|
|
121
139
|
|---|---|---|
|
|
122
140
|
| `pre-commit` | every commit | stops at the first failing check |
|
|
123
|
-
| `pre-push` | every push |
|
|
124
|
-
| `orly gate pr` | opening the Pull Request |
|
|
141
|
+
| `pre-push` | every push | documentation and declared non-test verification checks |
|
|
142
|
+
| `orly gate pr` | opening the Pull Request | runs all declared verification commands and checks the branch and spec |
|
|
125
143
|
|
|
126
144
|
---
|
|
127
145
|
|
|
@@ -144,7 +162,7 @@ flowchart TB
|
|
|
144
162
|
subgraph active["⚙️ docs/v1/active/"]
|
|
145
163
|
direction TB
|
|
146
164
|
|
|
147
|
-
plan["PLAN<br/><small>understand spec ·
|
|
165
|
+
plan["PLAN<br/><small>understand spec · resolve decisions</small>"]
|
|
148
166
|
execute["EXECUTE<br/><small>make the change</small>"]
|
|
149
167
|
conform["CONFORM<br/><small>apply matching rule pages</small>"]
|
|
150
168
|
verify["VERIFY<br/><small>run deterministic gates</small>"]
|
|
@@ -171,7 +189,7 @@ flowchart TB
|
|
|
171
189
|
| Directory | Meaning |
|
|
172
190
|
|---|---|
|
|
173
191
|
| `docs/v1/pending/` | the spec is written and committed on `main` |
|
|
174
|
-
| `docs/v1/active/` | branch cut,
|
|
192
|
+
| `docs/v1/active/` | branch cut, comparison revision recorded, measurement pending; no code until it commits |
|
|
175
193
|
| `docs/v1/done/` | gates green, Pull Request opens |
|
|
176
194
|
|
|
177
195
|
Inside `active/`, work runs through the stages: **PLAN → EXECUTE → CONFORM → VERIFY → REVIEW → DOCUMENT → COMMIT**.
|
|
@@ -216,7 +234,7 @@ The last two files exist because installing a rules file is not the same as deli
|
|
|
216
234
|
| `orly update --with <pack>` | add an opt-in pack, recorded for every clone |
|
|
217
235
|
| `orly gate` | run your declared checks in order, stopping at the first failure |
|
|
218
236
|
| `orly override <criterion> --reason <why>` | record a gate exception as an empty commit that rides into the Pull Request |
|
|
219
|
-
| `orly doctor` |
|
|
237
|
+
| `orly doctor` | check installed files and required command configuration |
|
|
220
238
|
|
|
221
239
|
---
|
|
222
240
|
|
package/audits/agents-md.md
CHANGED
|
@@ -107,7 +107,7 @@ The questionnaire is organised by scenario. Each scenario corresponds to a momen
|
|
|
107
107
|
|---|---|---|
|
|
108
108
|
| 5.1 | Is the agent required to verify CWD is inside the active worktree before resuming (`pwd` + `git worktree list`)? | YES |
|
|
109
109
|
| 5.2 | Must the agent re-read RULES.md when sub-task shape changes (new layer/language/resume after break)? | YES |
|
|
110
|
-
| 5.3 | If the spec is in `active/`,
|
|
110
|
+
| 5.3 | If the spec is in `active/`, does CHORE(close) follow the final implementation commit or an explicit request to park, while Section commits continue implementation? | YES |
|
|
111
111
|
| 5.4 | If unexpected changes appear in files the agent is editing, must the agent stop and ask (not overwrite as stale)? | YES |
|
|
112
112
|
|
|
113
113
|
### Scenario 6 — Conformance and verification lifecycle
|
|
@@ -122,8 +122,11 @@ The questionnaire is organised by scenario. Each scenario corresponds to a momen
|
|
|
122
122
|
| 6.6 | Is at least one declared `verify.integration` run from clean state required per branch before ship-ready (where the repository declares one)? | YES |
|
|
123
123
|
| 6.7 | Are package-scoped runners (`bun run test`, `vitest <file>`, `zig build test` w/o integration) explicitly **not** verification? | YES |
|
|
124
124
|
| 6.8 | Must memleak evidence (last 3 lines verbatim) appear in PR Session Notes when touching `src/http/**` / `src/cmd/serve.zig` / allocator wiring? | YES |
|
|
125
|
-
| 6.9 | Does CHORE(open)
|
|
126
|
-
| 6.
|
|
125
|
+
| 6.9 | Does CHORE(open) DECLARE the `Test Baseline:` line in the spec header while measuring nothing, with the counts from the repository's declared `verify.*` commands landing before the Pull Request (a product pack may name a dedicated counter)? | YES |
|
|
126
|
+
| 6.9a | Does the baseline identify the full comparison commit, with matching run evidence or an isolated checkout, declared unit and integration counts, and a report reference? | YES |
|
|
127
|
+
| 6.9b | Does `orly gate pr` reject missing declared baseline counts, allow reasoned `n/a` only without code or baseline lanes, and run all declared verification commands independently of custom hooks? | YES |
|
|
128
|
+
| 6.9c | Does setup require `conform` and a named `verify.*` command, allowing `verify.docs` without application test suites while reporting incomplete configuration? | YES |
|
|
129
|
+
| 6.10 | Does VERIFY end with a Test Delta row (growth vs the branch point's baseline) plus a lacking-areas verdict, with zero/negative unit delta on a code-adding diff requiring justification or a return to EXECUTE? | YES |
|
|
127
130
|
| 6.11 | Do the lifecycle stage runbooks (CHORE(open/close) checklists, PLAN expansions, deferral quote format, pre-PR gates, LAND) resolve from `dispatch/lifecycle.md`, with AGENTS.md keeping each stage's binding essence? | YES |
|
|
128
131
|
|
|
129
132
|
### Scenario 7 — Review discipline before merge
|
|
@@ -233,6 +236,7 @@ The questionnaire is organised by scenario. Each scenario corresponds to a momen
|
|
|
233
236
|
| 18.1 | Must `gitleaks` pass before any `git commit` / `git push`? | YES |
|
|
234
237
|
| 18.2 | Is the rule "one worktree per active stream — no edits outside, no reads from siblings, merge only after REVIEW and the final commit" preserved? | YES |
|
|
235
238
|
| 18.3 | Are cross-worktree edits explicitly forbidden without explicit user approval? | YES |
|
|
239
|
+
| 18.4 | Does CHORE(open) MOVE the base checkout's uncommitted work into the new worktree — never copy it, naming what came across, and stopping on a conflicting `git stash pop` — so a stream is never cut from a dirty base that then reads as unowned drift? | YES |
|
|
236
240
|
|
|
237
241
|
### Scenario 19 — CONFORM combined audit
|
|
238
242
|
|
package/audits/spec-template.sh
CHANGED
|
@@ -8,12 +8,12 @@
|
|
|
8
8
|
# 2. REQUIRED-PRESENT + NO-PLACEHOLDER (positive space) — the determinism
|
|
9
9
|
# sections the agent-facing template mandates (PR Intent, Applicable Gates,
|
|
10
10
|
# Prior-Art, Metrics, Decomposition, tiered Test Spec, Product Clarity,
|
|
11
|
-
# Discovery, …) must EXIST and
|
|
11
|
+
# Discovery, …) must EXIST and carry content: no known unfilled slots and
|
|
12
12
|
# no surviving "tpl:" guidance comments (template fill grammar). A spec
|
|
13
13
|
# that omits them forces the executing agent to guess intent. This is the
|
|
14
14
|
# half that makes a spec "built for the agent".
|
|
15
15
|
# 2b. DECLARED-COMMAND PARITY — a pending/active spec's Acceptance Rubric
|
|
16
|
-
# must quote the repository's declared conform + verify
|
|
16
|
+
# must quote the repository's declared conform + applicable verify.* commands
|
|
17
17
|
# (.oracle/orly.json) verbatim: the rubric and `orly gate` grade one
|
|
18
18
|
# boundary. Staged/file scope only; done/ specs stay historical.
|
|
19
19
|
#
|
|
@@ -25,7 +25,7 @@
|
|
|
25
25
|
# --all (default) pending+active specs only — current/in-flight work
|
|
26
26
|
# --include-done adds done/ specs to the scan (one-time sweep tool)
|
|
27
27
|
#
|
|
28
|
-
# Family 2 runs
|
|
28
|
+
# Family 2 runs in --staged and --file scope — the spec being authored/edited right
|
|
29
29
|
# now, which is exactly the agent's own output. The bulk scans (--all /
|
|
30
30
|
# --include-done) run Family 1 only, so they behave identically over the whole
|
|
31
31
|
# corpus and never break an existing spec. No legacy carve-out, no heuristics:
|
|
@@ -37,6 +37,7 @@
|
|
|
37
37
|
set -euo pipefail
|
|
38
38
|
|
|
39
39
|
MODE="${1:-${SCOPE:-all}}"
|
|
40
|
+
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
40
41
|
ROOT="$(git rev-parse --show-toplevel)"
|
|
41
42
|
cd "$ROOT"
|
|
42
43
|
|
|
@@ -116,93 +117,8 @@ PATTERNS=(
|
|
|
116
117
|
# Carve-outs — lines that look prohibited but are legitimate. Skip any match.
|
|
117
118
|
SAFE_LINES_RE='^\*\*Date:\*\*|^\s*```|<!--|^\s*//|"[0-9]+\s*(h|hour|day|min)"'
|
|
118
119
|
|
|
119
|
-
# ---------------------------------------------------------------------------
|
|
120
|
-
# Family 2 — Required determinism sections (agent-facing docs/TEMPLATE.md).
|
|
121
|
-
# Each entry: <heading ERE>:<human description>
|
|
122
|
-
# ---------------------------------------------------------------------------
|
|
123
|
-
REQUIRED_SECTIONS=(
|
|
124
|
-
'^#+ .*PR Intent:PR Intent & comprehension handshake'
|
|
125
|
-
'^#+ .*Applicable Rules:Applicable Rules'
|
|
126
|
-
'^#+ .*Applicable Gates:Applicable Gates'
|
|
127
|
-
'^#+ .*Overview:Overview'
|
|
128
|
-
'^#+ .*(Prior-Art|Reference Implementation):Prior-Art / Reference Implementations'
|
|
129
|
-
'^#+ .*Files Changed:Files Changed (blast radius)'
|
|
130
|
-
'^#+ .*(Decomposition|Alternatives):Decomposition & alternatives (patch vs refactor)'
|
|
131
|
-
'^#+ .*Sections:Sections (implementation slices)'
|
|
132
|
-
'^#+ .*Metrics.*Observability:Metrics & Observability'
|
|
133
|
-
'^#+ .*Interfaces:Interfaces'
|
|
134
|
-
'^#+ .*Failure Modes:Failure Modes'
|
|
135
|
-
'^#+ .*Invariants:Invariants'
|
|
136
|
-
'^#+ .*Test Specification:Test Specification'
|
|
137
|
-
'^#+ .*Acceptance (Criteria|Rubric):Acceptance Rubric (legacy heading Acceptance Criteria accepted)'
|
|
138
|
-
'^#+ .*Product Clarity:Product Clarity (authoring record)'
|
|
139
|
-
'^#+ .*Discovery:Discovery (consult log)'
|
|
140
|
-
)
|
|
141
|
-
|
|
142
|
-
# Template residue — strings that exist ONLY in the unfilled template. Their
|
|
143
|
-
# survival in a pending/active spec means the section was never filled. The
|
|
144
|
-
# "<!-- tpl:" marker is the template's fill grammar: every guidance comment is
|
|
145
|
-
# deleted at authoring, so any survivor is residue by construction.
|
|
146
|
-
PLACEHOLDER_SENTINELS=(
|
|
147
|
-
'<!-- tpl:'
|
|
148
|
-
'Title — testable, not vague'
|
|
149
|
-
'one-line reason}'
|
|
150
|
-
'Slice title}'
|
|
151
|
-
'path/to/file.ext'
|
|
152
|
-
'test_<short_name>'
|
|
153
|
-
'why this is the right pattern to mirror'
|
|
154
|
-
'{outcome the user can observe}'
|
|
155
|
-
'{exit 0 / substring / 0 matches}'
|
|
156
|
-
'path/to/spec_or_doc.md'
|
|
157
|
-
'{path/to/old_file.ext}'
|
|
158
|
-
'{old_symbol}'
|
|
159
|
-
'{one sentence that could be a test name}'
|
|
160
|
-
'{smallest verifiable behaviour}'
|
|
161
|
-
'{one-line behavioural claim}'
|
|
162
|
-
'{conform command from .oracle/orly.json'
|
|
163
|
-
'{verify.unit command from .oracle/orly.json'
|
|
164
|
-
"{gate from this repository's AGENTS.md dispatch index}"
|
|
165
|
-
)
|
|
166
|
-
|
|
167
|
-
# ---------------------------------------------------------------------------
|
|
168
|
-
# Family 2b — declared-command parity (TEMPLATE.md "Command source rule").
|
|
169
|
-
# A pending/active spec's Acceptance Rubric must quote the repository's
|
|
170
|
-
# declared conform + verify.unit commands VERBATIM — the same set `orly gate`
|
|
171
|
-
# runs — so the spec's ship gate and the mechanical PR gate grade one
|
|
172
|
-
# boundary. Reads .oracle/orly.json with bun (orly's own runtime). Done specs
|
|
173
|
-
# are historical records and are never retrofitted. No config, no bun, or no
|
|
174
|
-
# declared commands → NOTE and skip; this check never guesses.
|
|
175
|
-
# ---------------------------------------------------------------------------
|
|
176
|
-
scan_commands() {
|
|
177
|
-
local spec="$1" cfg=".oracle/orly.json" cmds
|
|
178
|
-
case "$spec" in
|
|
179
|
-
*/pending/*|*/active/*) ;;
|
|
180
|
-
*) return 0 ;;
|
|
181
|
-
esac
|
|
182
|
-
[[ -f "$cfg" ]] || { note "$spec — no $cfg; declared-command parity skipped"; return 0; }
|
|
183
|
-
command -v bun >/dev/null 2>&1 || { note "$spec — bun unavailable; declared-command parity skipped"; return 0; }
|
|
184
|
-
cmds="$(bun -e '
|
|
185
|
-
const cfg = await Bun.file(".oracle/orly.json").json().catch(() => null);
|
|
186
|
-
const commands = cfg && typeof cfg.commands === "object" && cfg.commands !== null ? cfg.commands : {};
|
|
187
|
-
for (const key of ["conform", "verify.unit"]) {
|
|
188
|
-
for (const argv of Array.isArray(commands[key]) ? commands[key] : []) {
|
|
189
|
-
if (Array.isArray(argv) && argv.length > 0) console.log(argv.join(" "));
|
|
190
|
-
}
|
|
191
|
-
}' 2>/dev/null || true)"
|
|
192
|
-
[[ -n "$cmds" ]] || { note "$spec — no declared conform/verify.unit commands; parity skipped"; return 0; }
|
|
193
|
-
local miss=0 cmd
|
|
194
|
-
while IFS= read -r cmd; do
|
|
195
|
-
[[ -z "$cmd" ]] && continue
|
|
196
|
-
grep -qF "$cmd" "$spec" && continue
|
|
197
|
-
fail "$spec — rubric misses declared command verbatim: \"$cmd\" (.oracle/orly.json — the set orly gate runs)"
|
|
198
|
-
miss=$((miss + 1))
|
|
199
|
-
done <<<"$cmds"
|
|
200
|
-
if [[ $miss -eq 0 ]]; then ok "$spec — rubric quotes the declared conform/verify.unit commands"; fi
|
|
201
|
-
return 0
|
|
202
|
-
}
|
|
203
|
-
|
|
204
120
|
scan_spec() {
|
|
205
|
-
local spec="$1"
|
|
121
|
+
local spec="$1" content_file="$2"
|
|
206
122
|
local hits=0
|
|
207
123
|
local pattern desc severity line content
|
|
208
124
|
while IFS=: read -r severity desc pattern; do
|
|
@@ -221,36 +137,37 @@ scan_spec() {
|
|
|
221
137
|
note "$spec:$line — $desc"
|
|
222
138
|
fi
|
|
223
139
|
hits=$((hits + 1))
|
|
224
|
-
done < <(grep -nE "$pattern" "$
|
|
140
|
+
done < <(grep -nE "$pattern" "$content_file" 2>/dev/null || true)
|
|
225
141
|
done < <(printf '%s\n' "${PATTERNS[@]}")
|
|
226
142
|
if [[ $hits -eq 0 ]]; then ok "$spec — no prohibited patterns"; fi
|
|
227
143
|
return 0
|
|
228
144
|
}
|
|
229
145
|
|
|
230
|
-
#
|
|
231
|
-
|
|
232
|
-
|
|
233
|
-
local spec="$1"
|
|
234
|
-
local miss=0 entry sec desc sentinel
|
|
235
|
-
for entry in "${REQUIRED_SECTIONS[@]}"; do
|
|
236
|
-
sec="${entry%%:*}"; desc="${entry#*:}"
|
|
237
|
-
grep -qE "$sec" "$spec" && continue
|
|
238
|
-
fail "$spec — missing required section: $desc"
|
|
239
|
-
miss=$((miss + 1))
|
|
240
|
-
done
|
|
241
|
-
for sentinel in "${PLACEHOLDER_SENTINELS[@]}"; do
|
|
242
|
-
grep -qF "$sentinel" "$spec" || continue
|
|
243
|
-
fail "$spec — unfilled template placeholder: \"$sentinel\""
|
|
244
|
-
miss=$((miss + 1))
|
|
245
|
-
done
|
|
246
|
-
if [[ $miss -eq 0 ]]; then ok "$spec — required sections present, no placeholders"; fi
|
|
247
|
-
return 0
|
|
248
|
-
}
|
|
249
|
-
|
|
146
|
+
# Staged checks grade the index, including files removed only from the worktree.
|
|
147
|
+
CONTENT_DIR="$(mktemp -d)"
|
|
148
|
+
trap 'rm -rf "$CONTENT_DIR"' EXIT
|
|
250
149
|
for spec in "${SPECS[@]}"; do
|
|
251
|
-
|
|
252
|
-
|
|
253
|
-
|
|
150
|
+
content_file="$spec"
|
|
151
|
+
case "$MODE" in
|
|
152
|
+
--staged|staged)
|
|
153
|
+
content_file="$CONTENT_DIR/spec.md"
|
|
154
|
+
if ! git show ":$spec" > "$content_file"; then
|
|
155
|
+
fail "$spec — cannot read staged content"
|
|
156
|
+
continue
|
|
157
|
+
fi
|
|
158
|
+
;;
|
|
159
|
+
esac
|
|
160
|
+
[[ -f "$content_file" ]] || { fail "$spec — cannot read spec"; continue; }
|
|
161
|
+
scan_spec "$spec" "$content_file"
|
|
162
|
+
case "$MODE" in
|
|
163
|
+
--staged|staged|--file|file)
|
|
164
|
+
if ! command -v bun >/dev/null 2>&1; then
|
|
165
|
+
fail "$spec — Bun is required for spec readiness checks"
|
|
166
|
+
elif ! bun "$SCRIPT_DIR/spec-template.ts" "$spec" "$content_file" "$MODE"; then
|
|
167
|
+
FAIL=1
|
|
168
|
+
fi
|
|
169
|
+
;;
|
|
170
|
+
esac
|
|
254
171
|
done
|
|
255
172
|
|
|
256
173
|
if [[ $FAIL -ne 0 ]]; then
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
import { existsSync } from "node:fs";
|
|
2
|
+
import { isAbsolute, join, normalize, resolve } from "node:path";
|
|
3
|
+
|
|
4
|
+
type MarkdownElement = { type?: unknown; props: { children?: MarkdownNode[] } };
|
|
5
|
+
type MarkdownNode = string | MarkdownElement;
|
|
6
|
+
type Section = { title: string; nodes: MarkdownNode[] };
|
|
7
|
+
type Commands = Record<string, string[][]>;
|
|
8
|
+
|
|
9
|
+
const CONFIG_PATH = ".oracle/orly.json";
|
|
10
|
+
const SPEC_LINE_LIMIT = 320;
|
|
11
|
+
const BACKTICK = "`";
|
|
12
|
+
const EMPTY = "";
|
|
13
|
+
const PIPE_OUTPUT = "pipe";
|
|
14
|
+
const TABLE_ROW = "tr";
|
|
15
|
+
const TEST_HEADING = /^Test Specification/;
|
|
16
|
+
const RUBRIC_HEADING = /^Acceptance (Rubric|Criteria)/;
|
|
17
|
+
const REQUIRED = [
|
|
18
|
+
/^Overview/, /^PR Intent/, /^Implementing agent/, /^Files Changed/, /^Applicable Rules/,
|
|
19
|
+
/^Applicable Gates/, /^(Prior-Art|Reference Implementation)/, /^Sections/,
|
|
20
|
+
/^Interfaces/, /^Failure Modes/, /^Invariants/, /^Metrics.*Observability/,
|
|
21
|
+
TEST_HEADING, RUBRIC_HEADING, /^Dead Code Sweep/, /^Out of Scope/,
|
|
22
|
+
/^Product Clarity/, /^(Decomposition|Alternatives)/, /^Discovery/,
|
|
23
|
+
];
|
|
24
|
+
const RESIDUE = /<!--\s*tpl:|\{\{fill:|\{unresolved_[^}]+\}|path\/to\/file\.ext|path\/to\/spec_or_doc\.md|test_<short_name>|\{one-line reason\}|\{Slice title\}|\{smallest verifiable behaviour\}|\{one-line behavioural claim\}/;
|
|
25
|
+
const STAGED_MODES = new Set(["--staged", "staged"]);
|
|
26
|
+
const EXPECTED_COLUMN = "Expected";
|
|
27
|
+
|
|
28
|
+
function sectionsOf(text: string): Section[] {
|
|
29
|
+
const sections: Section[] = [];
|
|
30
|
+
const document = Bun.markdown.react(text) as MarkdownElement;
|
|
31
|
+
for (const node of children(document)) {
|
|
32
|
+
if (!isString(node) && node.type === "h2") sections.push({ title: textOf(node), nodes: [] });
|
|
33
|
+
else if (sections.length) sections[sections.length - 1]!.nodes.push(node);
|
|
34
|
+
}
|
|
35
|
+
return sections;
|
|
36
|
+
}
|
|
37
|
+
|
|
38
|
+
function section(sections: Section[], heading: RegExp): MarkdownNode[] {
|
|
39
|
+
return sections.find((entry) => heading.test(entry.title))?.nodes ?? [];
|
|
40
|
+
}
|
|
41
|
+
|
|
42
|
+
function children(node: MarkdownNode): MarkdownNode[] {
|
|
43
|
+
return isString(node) ? [] : node.props.children ?? [];
|
|
44
|
+
}
|
|
45
|
+
|
|
46
|
+
function elements(nodes: MarkdownNode[], type: string): MarkdownElement[] {
|
|
47
|
+
return nodes.filter((node): node is MarkdownElement => !isString(node) && node.type === type);
|
|
48
|
+
}
|
|
49
|
+
|
|
50
|
+
function textOf(node: MarkdownNode): string {
|
|
51
|
+
if (isString(node)) return node;
|
|
52
|
+
const text = children(node).map(textOf).join(EMPTY);
|
|
53
|
+
return node.type === "code" ? `${BACKTICK}${text}${BACKTICK}` : text;
|
|
54
|
+
}
|
|
55
|
+
|
|
56
|
+
function table(nodes: MarkdownNode[]): { columns: string[]; rows: string[][] } {
|
|
57
|
+
const groups = elements(nodes, "table").flatMap(children);
|
|
58
|
+
const headers = elements(groups, "thead").flatMap(children);
|
|
59
|
+
const bodies = elements(groups, "tbody").flatMap(children);
|
|
60
|
+
return {
|
|
61
|
+
columns: elements(headers, TABLE_ROW).flatMap(children).map((node) => textOf(node).trim()),
|
|
62
|
+
rows: elements(bodies, TABLE_ROW).map((row) => children(row).map((node) => textOf(node).trim())),
|
|
63
|
+
};
|
|
64
|
+
}
|
|
65
|
+
|
|
66
|
+
function testMappings(sections: Section[]): string[] {
|
|
67
|
+
const errors: string[] = [];
|
|
68
|
+
const items = elements(section(sections, /^Sections/), "ul").flatMap(children).map(textOf);
|
|
69
|
+
const declared = items.filter((item) => item.startsWith("Dimension "));
|
|
70
|
+
const dimensions = declared.flatMap((item) => [...item.matchAll(/^Dimension (\d+\.\d+)\b[^\n]*?→ Test `([^`]+)`/g)]);
|
|
71
|
+
if (!dimensions.length || dimensions.length !== declared.length) errors.push("every Dimension must name its proof with → Test `name`");
|
|
72
|
+
const ids = dimensions.map((match) => match[1]);
|
|
73
|
+
if (new Set(ids).size !== ids.length) errors.push("Dimension identifiers must be unique");
|
|
74
|
+
const tests = table(section(sections, TEST_HEADING));
|
|
75
|
+
const tier = tests.columns.indexOf("Tier");
|
|
76
|
+
const name = tests.columns.indexOf("Test");
|
|
77
|
+
const assertion = tests.columns.findIndex((column) => column.startsWith("Asserts"));
|
|
78
|
+
const id = tests.columns.indexOf("Dimension");
|
|
79
|
+
if ([tier, name, assertion, id].some((index) => index < 0)) return [...errors, "Test Specification needs Dimension, Tier, Test, and Asserts columns"];
|
|
80
|
+
for (const dimension of dimensions) {
|
|
81
|
+
if (!tests.rows.some((row) => row[id] === dimension[1] && row[name]?.replaceAll(BACKTICK, EMPTY) === dimension[2] && row[tier] && row[assertion])) {
|
|
82
|
+
errors.push(`Dimension ${dimension[1]} has no matching tiered test and assertion`);
|
|
83
|
+
}
|
|
84
|
+
}
|
|
85
|
+
for (const row of tests.rows) {
|
|
86
|
+
if (row[id] && !ids.includes(row[id])) errors.push(`test row references unknown Dimension ${row[id]}`);
|
|
87
|
+
}
|
|
88
|
+
return errors;
|
|
89
|
+
}
|
|
90
|
+
|
|
91
|
+
function readFirst(sections: Section[], exists: (path: string) => boolean): string[] {
|
|
92
|
+
const items = elements(section(sections, /^Implementing agent/), "ol").flatMap(children).map(textOf);
|
|
93
|
+
const references = items.flatMap((item) => [...item.matchAll(/^(?:`([^`]+)`|(https?:\/\/\S+))/g)]);
|
|
94
|
+
if (!references.length) return ["read-first section needs existing repository paths or URLs"];
|
|
95
|
+
const errors: string[] = [];
|
|
96
|
+
for (const reference of references) {
|
|
97
|
+
const path = reference[1] ?? reference[2] ?? EMPTY;
|
|
98
|
+
if (/^https?:\/\//.test(path)) continue;
|
|
99
|
+
const normalized = normalize(path.split("#")[0] ?? EMPTY);
|
|
100
|
+
if (isAbsolute(normalized) || normalized.startsWith("..") || normalized.includes("{")) {
|
|
101
|
+
errors.push(`read-first pointer must name a repository file: ${path}`);
|
|
102
|
+
} else if (!exists(normalized)) {
|
|
103
|
+
errors.push(`read-first pointer does not resolve: ${path}`);
|
|
104
|
+
}
|
|
105
|
+
}
|
|
106
|
+
return errors;
|
|
107
|
+
}
|
|
108
|
+
|
|
109
|
+
function git(argv: string[], root: string): { ok: boolean; text: string } {
|
|
110
|
+
const result = Bun.spawnSync(["git", ...argv], { cwd: root, stdout: PIPE_OUTPUT, stderr: PIPE_OUTPUT });
|
|
111
|
+
return { ok: result.exitCode === 0, text: result.stdout.toString() };
|
|
112
|
+
}
|
|
113
|
+
|
|
114
|
+
async function commands(staged: boolean, root: string): Promise<Commands> {
|
|
115
|
+
const text = staged ? git(["show", `:${CONFIG_PATH}`], root) : { ok: true, text: await Bun.file(join(root, CONFIG_PATH)).text() };
|
|
116
|
+
if (!text.ok) throw new Error(`cannot read ${CONFIG_PATH} from the index`);
|
|
117
|
+
const config: unknown = JSON.parse(text.text);
|
|
118
|
+
if (!object(config) || !object(config.commands)) throw new Error(`${CONFIG_PATH} needs declared commands`);
|
|
119
|
+
const result: Commands = {};
|
|
120
|
+
for (const [name, invocations] of Object.entries(config.commands)) {
|
|
121
|
+
if (!Array.isArray(invocations) || !invocations.length) throw new Error(`empty command group: ${name}`);
|
|
122
|
+
result[name] = invocations.map((argv: unknown) => {
|
|
123
|
+
if (!Array.isArray(argv) || !argv.length || !argv.every(isString)) throw new Error(`invalid arguments for ${name}`);
|
|
124
|
+
return argv;
|
|
125
|
+
});
|
|
126
|
+
}
|
|
127
|
+
if (!Object.keys(result).length) throw new Error(`${CONFIG_PATH} needs declared commands`);
|
|
128
|
+
return result;
|
|
129
|
+
}
|
|
130
|
+
|
|
131
|
+
function object(value: unknown): value is Record<string, unknown> {
|
|
132
|
+
return typeof value === "object" && value !== null && !Array.isArray(value);
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
function isString(value: unknown): value is string {
|
|
136
|
+
return typeof value === "string";
|
|
137
|
+
}
|
|
138
|
+
|
|
139
|
+
function rubricCommands(sections: Section[], declared: Commands): string[] {
|
|
140
|
+
const rubric = table(section(sections, RUBRIC_HEADING));
|
|
141
|
+
const verify = rubric.columns.findIndex((column) => column.startsWith("Verify"));
|
|
142
|
+
const expected = rubric.columns.indexOf(EXPECTED_COLUMN);
|
|
143
|
+
if (verify < 0 || expected < 0) return ["Acceptance Rubric needs Verify and Expected columns"];
|
|
144
|
+
const errors: string[] = [];
|
|
145
|
+
for (const [name, invocations] of Object.entries(declared)) {
|
|
146
|
+
if (name !== "conform" && !name.startsWith("verify.")) continue;
|
|
147
|
+
for (const argv of invocations) {
|
|
148
|
+
const command = argv.join(" ");
|
|
149
|
+
if (!rubric.rows.some((row) => row[verify] === `${BACKTICK}${command}${BACKTICK}` && row[expected])) {
|
|
150
|
+
errors.push(`rubric misses declared command verbatim: ${command}`);
|
|
151
|
+
}
|
|
152
|
+
}
|
|
153
|
+
}
|
|
154
|
+
return errors;
|
|
155
|
+
}
|
|
156
|
+
|
|
157
|
+
export function validateSpec(text: string, declared: Commands, exists: (path: string) => boolean): string[] {
|
|
158
|
+
const sections = sectionsOf(text);
|
|
159
|
+
const errors: string[] = [];
|
|
160
|
+
const count = text.trimEnd().split(/\r?\n/).length;
|
|
161
|
+
if (count > SPEC_LINE_LIMIT) errors.push(`spec has ${count} lines; cap ${SPEC_LINE_LIMIT}`);
|
|
162
|
+
for (const heading of REQUIRED) {
|
|
163
|
+
if (!section(sections, heading).map(textOf).join(EMPTY).trim()) errors.push(`missing or empty required section: ${heading.source}`);
|
|
164
|
+
}
|
|
165
|
+
if (RESIDUE.test(text)) errors.push("unfilled template placeholder or authoring guidance");
|
|
166
|
+
errors.push(...readFirst(sections, exists), ...testMappings(sections), ...rubricCommands(sections, declared));
|
|
167
|
+
return errors;
|
|
168
|
+
}
|
|
169
|
+
|
|
170
|
+
export async function auditSpec(args: string[], root: string): Promise<string[]> {
|
|
171
|
+
const [spec, path, mode] = args;
|
|
172
|
+
if (!spec || !path || !mode) throw new Error("expected spec path, content path, and audit mode");
|
|
173
|
+
const staged = STAGED_MODES.has(mode);
|
|
174
|
+
const exists = (reference: string) => staged ? git(["cat-file", "-e", `:${reference}`], root).ok : existsSync(join(root, reference));
|
|
175
|
+
return validateSpec(await Bun.file(resolve(root, path)).text(), await commands(staged, root), exists);
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
if (import.meta.main) {
|
|
179
|
+
try {
|
|
180
|
+
const errors = await auditSpec(Bun.argv.slice(2), process.cwd());
|
|
181
|
+
for (const error of errors) {
|
|
182
|
+
// logging: audit findings are the command-line interface.
|
|
183
|
+
console.error(`FAIL: ${Bun.argv[2]} — ${error}`);
|
|
184
|
+
}
|
|
185
|
+
if (errors.length) process.exitCode = 1;
|
|
186
|
+
// logging: report structural checks without claiming semantic readiness.
|
|
187
|
+
else console.log(`OK: ${Bun.argv[2]} — sections, references, test mappings, and declared commands checked`);
|
|
188
|
+
} catch (error) {
|
|
189
|
+
// logging: unavailable audit inputs must fail, never silently skip checks.
|
|
190
|
+
console.error(`FAIL: spec readiness check — ${error instanceof Error ? error.message : String(error)}`);
|
|
191
|
+
process.exitCode = 1;
|
|
192
|
+
}
|
|
193
|
+
}
|
package/core/operating-model.md
CHANGED
|
@@ -173,7 +173,7 @@ Guards fire pre-hoc regardless of lifecycle stage. Override: `<GATE>: SKIPPED pe
|
|
|
173
173
|
| Begin implementation OR branch carries spec changes in `pending/` | CHORE(open): `pending/`→`active/`, `Status: IN_PROGRESS` + `Branch:`, create worktree, commit on feature branch. **No code until 4 steps committed.** |
|
|
174
174
|
| Every commit during implementation | Mark completed Dimensions/Sections `DONE` same commit as the code. |
|
|
175
175
|
| All work complete, before PR | CHORE(close). |
|
|
176
|
-
|
|
|
176
|
+
| Final implementation commit, or explicit request to park a spec | CHORE(close) is mandatory next action; Section commits continue the stream. |
|
|
177
177
|
<!-- oracle-packs:end -->
|
|
178
178
|
|
|
179
179
|
---
|
|
@@ -188,14 +188,14 @@ Non-trivial (full lifecycle) if it: touches >1 file · new abstraction · data m
|
|
|
188
188
|
|
|
189
189
|
**Anchor invariant — `orly gate` proves the boundary.** No PR opens unless every `orly gate pr` criterion is green or carries an `Orly-Override` trailer the user recorded with a reason.
|
|
190
190
|
|
|
191
|
-
**
|
|
191
|
+
**Gate ownership.** `work` runs the declared `conform` command at commit. `verify` runs documentation checks and non-test `verify.*` commands at push; it permits unfinished Sections. `pr` checks branch shape, exact upstream revision, spec completeness, and every declared `verify.*` command itself. `orly gate pr` is the command CHORE(close) runs; a push is never proof that tests ran. Unit and integration baselines are measured before the Pull Request against the recorded comparison revision. Full unit and integration suites run at the PR boundary, not at opening or every push. Integration and memory suites skip only on branches with no code. The authoritative sequence is `dispatch/lifecycle.md`; no spec means only spec criteria skip. A user-surface change without a docs change is red until docs land or an override says why not.
|
|
192
192
|
|
|
193
193
|
**LAND (after merge, or when the user confirms it merged):** pull the default branch, prune the merged worktree + branch, `make down` where defined.
|
|
194
194
|
|
|
195
195
|
<!-- oracle-packs:start workflow.specifications -->
|
|
196
196
|
### CHORE (open)
|
|
197
197
|
|
|
198
|
-
Spec `pending/`→`active/`; `Status: IN_PROGRESS`; `Branch:` set; **`Test Baseline:`
|
|
198
|
+
Spec `pending/`→`active/`; `Status: IN_PROGRESS`; `Branch:` set; **`Test Baseline:` declared pending; `Baseline revision:` records the full comparison commit** (unit and integration counts plus evidence are due before the Pull Request); committed. Worktree created, CWD inside (verify `pwd` + `git worktree list`), carrying the base checkout's uncommitted work in with it. No code until the 4 steps are committed.
|
|
199
199
|
<!-- oracle-packs:end -->
|
|
200
200
|
|
|
201
201
|
### PLAN
|
|
@@ -222,9 +222,9 @@ In `agentsfleet` this stage is `make harness-verify`; its output block and end-o
|
|
|
222
222
|
|
|
223
223
|
### VERIFY
|
|
224
224
|
|
|
225
|
-
**
|
|
225
|
+
**Section proof and final verification.** Follow `dispatch/lifecycle.md` for command timing. Section checks prove the changed Dimensions; the final gate runs the full declared verification set. A package-scoped command never replaces a declared boundary command, and a Section result never satisfies a repository-wide "tests pass" claim.
|
|
226
226
|
<!-- oracle-packs:start product.agentsfleet -->
|
|
227
|
-
The `agentsfleet` output block and both cadences live in `docs/VERIFY_TIERS.md`. **FIRST: `/orly-write-unit-test`** audits diff coverage. **LAST: the Test Delta row** compares against the
|
|
227
|
+
The `agentsfleet` output block and both cadences live in `docs/VERIFY_TIERS.md`. **FIRST: `/orly-write-unit-test`** audits diff coverage. **LAST: the Test Delta row** compares against the recorded comparison revision, measured before the Pull Request; zero or negative unit growth on a code-adding diff needs justification or a return to EXECUTE. Paste memory-leak evidence into PR Session Notes or cite its Continuous Integration (CI) URL. After refactors, list newly dead code before removing it.
|
|
228
228
|
<!-- oracle-packs:end -->
|
|
229
229
|
|
|
230
230
|
### REVIEW
|