@agentsfleet/orly 0.10.4 → 0.10.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +5 -5
- package/README.md +26 -8
- package/audits/agents-md.md +7 -3
- package/audits/spec-template.sh +30 -113
- package/audits/spec-template.ts +193 -0
- package/core/operating-model.md +5 -5
- package/dispatch/lifecycle.md +98 -42
- package/dispatch/verify.md +30 -48
- package/dispatch/write_spec.md +56 -82
- package/docs/ORLY_ARCHITECTURE.md +13 -3
- package/docs/RULE_ENFORCEMENT.md +2 -2
- package/docs/TEMPLATE.md +93 -76
- package/docs/VERIFY_TIERS.md +20 -26
- package/llms.txt +17 -12
- package/package.json +1 -1
- package/registry.json +4 -0
- package/skills/orly-spec-new/SKILL.md +3 -1
- package/skills/orly-write-integration-test/SKILL.md +1 -1
- package/skills/orly-write-unit-test/SKILL.md +1 -1
- package/src/cli.ts +14 -16
- package/src/criteria.test.ts +47 -6
- package/src/criteria.ts +30 -49
- package/src/criteria_spec.ts +53 -6
- package/src/criteria_support.ts +5 -4
- package/src/gates.test.ts +2 -2
- package/src/gates_spec.test.ts +40 -1
- package/src/gates_test_support.ts +18 -4
- package/src/install.ts +16 -81
- package/src/render.test.ts +5 -7
- package/src/setup.test.ts +116 -0
- package/src/spec_baseline.test.ts +96 -0
- package/src/spec_template.test.ts +109 -0
- package/src/spec_template_io.test.ts +97 -0
- package/src/validation.ts +9 -0
package/docs/TEMPLATE.md
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
> **CANONICAL TEMPLATE — two agents consume this file, in different ways.**
|
|
4
4
|
> The **authoring agent** (via the `orly-spec-new` skill) copies the body below the divider into `docs/v{N}/pending/M{N}_{NNN}_….md` and fills it — authoring order lives in the skill, not in this file. The **executing agent** reads the filled spec **top-to-bottom**: the body is physically ordered by execution need (understand → prepare → build → prove → record).
|
|
5
5
|
> Enforced by `audits/spec-template.sh` (SPEC TEMPLATE GATE, façade `dispatch/write_spec.md`): required sections present, zero template residue, prohibited patterns absent. Lifecycle: `AGENTS.md → Specification Standards`.
|
|
6
|
-
>
|
|
6
|
+
> Read a recent completed instance for examples; current rules and this template govern any conflict.
|
|
7
7
|
|
|
8
8
|
---
|
|
9
9
|
|
|
@@ -13,7 +13,7 @@ Three markup classes appear below the divider. The gate tells them apart mechani
|
|
|
13
13
|
|
|
14
14
|
| Marker | Meaning | Fate in the filled spec |
|
|
15
15
|
|---|---|---|
|
|
16
|
-
| `{
|
|
16
|
+
| `{{fill:description}}` | Authoring slot — replace with instance content | Gone before staging. Runtime parameters such as `/items/{id}` remain when defined. |
|
|
17
17
|
| `tpl:` guidance comment | How to fill the section — written for the authoring agent only | **Deleted after filling.** The gate BLOCKs any survivor. |
|
|
18
18
|
| SPEC AUTHORING RULES banner comment | Standing constraints on the instance | The one comment that survives, verbatim. |
|
|
19
19
|
|
|
@@ -89,22 +89,24 @@ SPEC AUTHORING RULES (load-bearing — the one comment that survives):
|
|
|
89
89
|
sequencing signal. A section that contradicts these rules loses — delete it.
|
|
90
90
|
-->
|
|
91
91
|
|
|
92
|
-
# M{Milestone}_{Workstream}: {Title — testable, not vague}
|
|
92
|
+
# M{{fill:Milestone}}_{{fill:Workstream}}: {{fill:Title — testable, not vague}}
|
|
93
93
|
|
|
94
|
-
**Prototype:** v{major}.{minor}.{patch}
|
|
95
|
-
**Milestone:** M{Number}
|
|
96
|
-
**Workstream:** {001-009}
|
|
97
|
-
**Date:** {MMM DD, YYYY}
|
|
94
|
+
**Prototype:** v{{fill:major}}.{{fill:minor}}.{{fill:patch}}
|
|
95
|
+
**Milestone:** M{{fill:Number}}
|
|
96
|
+
**Workstream:** {{fill:001-009}}
|
|
97
|
+
**Date:** {{fill:MMM DD, YYYY}}
|
|
98
98
|
**Status:** PENDING | IN_PROGRESS | DONE
|
|
99
|
-
**Priority:** P0 | P1 | P2 | P3 — {one-line reason}
|
|
100
|
-
**Categories:** {API | CLI | UI | SKILL | DOCS | OBS | INFRA — alphabetised, one or more}
|
|
101
|
-
**Batch:** B{1-4} — {parallel execution context}
|
|
102
|
-
**Branch:**
|
|
99
|
+
**Priority:** P0 | P1 | P2 | P3 — {{fill:one-line reason}}
|
|
100
|
+
**Categories:** {{fill:API | CLI | UI | SKILL | DOCS | OBS | INFRA — alphabetised, one or more}}
|
|
101
|
+
**Batch:** B{{fill:1-4}} — {{fill:parallel execution context}}
|
|
102
|
+
**Branch:** pending — set at CHORE(open)
|
|
103
103
|
<!-- A deliberately folded workstream adds Folded-into metadata per dispatch/lifecycle.md. -->
|
|
104
|
-
**
|
|
105
|
-
**
|
|
106
|
-
**
|
|
107
|
-
**
|
|
104
|
+
**Baseline revision:** pending — record the full comparison commit at CHORE(open)
|
|
105
|
+
**Test Baseline:** pending — measure declared unit and integration lanes before the Pull Request
|
|
106
|
+
**Baseline evidence:** pending — report path or run URL with revision, commands, passed/failed/skipped counts, and environment
|
|
107
|
+
**Depends on:** {{fill:spec identifiers with one-line reasons, or none}}
|
|
108
|
+
**Provenance:** human-written | LLM-drafted ({{fill:model}}, {{fill:date}}) | agent-generated (pre-spec, {{fill:source doc}})
|
|
109
|
+
**Canonical architecture:** `docs/architecture/{{fill:relevant-doc}}.md` §{{fill:N}}
|
|
108
110
|
|
|
109
111
|
<!-- tpl: Provenance is load-bearing — the implementing agent calibrates trust by
|
|
110
112
|
who wrote the spec: LLM-drafted gets extra cross-checking against the codebase;
|
|
@@ -116,9 +118,9 @@ Update Gate); greenfield → point at the doc that defines the shape. -->
|
|
|
116
118
|
|
|
117
119
|
## Overview
|
|
118
120
|
|
|
119
|
-
**Goal (testable):** {one sentence that could be a test name}
|
|
120
|
-
**Problem:** {observable symptoms, in user-facing terms}
|
|
121
|
-
**Solution summary:** {one paragraph — what changes, at what layer, what the user-visible outcome is}
|
|
121
|
+
**Goal (testable):** {{fill:one sentence that could be a test name}}
|
|
122
|
+
**Problem:** {{fill:observable symptoms, in user-facing terms}}
|
|
123
|
+
**Solution summary:** {{fill:one paragraph — what changes, at what layer, what the user-visible outcome is}}
|
|
122
124
|
|
|
123
125
|
<!-- tpl: Goal — bad: "Implement streaming." Good: "Server-Sent Events (SSE)
|
|
124
126
|
handler streams Redis pubsub messages as text/event-stream with stable ordering
|
|
@@ -128,18 +130,18 @@ don't have a foo handler"). Implementation steps belong in Sections, not here. -
|
|
|
128
130
|
|
|
129
131
|
## PR Intent & comprehension handshake
|
|
130
132
|
|
|
131
|
-
- **PR title (eventual):** {imperative, ≤72 chars — what the merged Pull Request (PR) is called}
|
|
132
|
-
- **Intent (one sentence):** {why this PR exists, in user-facing-outcome terms}
|
|
133
|
-
- **Handshake** — the implementing agent
|
|
133
|
+
- **PR title (eventual):** {{fill:imperative, ≤72 chars — what the merged Pull Request (PR) is called}}
|
|
134
|
+
- **Intent (one sentence):** {{fill:why this PR exists, in user-facing-outcome terms}}
|
|
135
|
+
- **Handshake** — pending until the implementing agent performs PLAN, before EXECUTE: restate the Intent in its own words and list `ASSUMPTIONS I'M MAKING: …`. A mismatch between the restatement and the Intent above → STOP and reconcile before any edit.
|
|
134
136
|
|
|
135
137
|
## Implementing agent — read these first
|
|
136
138
|
|
|
137
|
-
1. `{path/to/file.ext}` — {why this is the right pattern to mirror}
|
|
138
|
-
2. `{path/to/spec_or_doc.md}` — {what canonical knowledge lives there}
|
|
139
|
-
3. {external doc URL, if relevant} — {what convention to follow}
|
|
139
|
+
1. `{{fill:path/to/file.ext}}` — {{fill:why this is the right pattern to mirror}}
|
|
140
|
+
2. `{{fill:path/to/spec_or_doc.md}}` — {{fill:what canonical knowledge lives there}}
|
|
141
|
+
3. {{fill:external doc URL, if relevant}} — {{fill:what convention to follow}}
|
|
140
142
|
|
|
141
|
-
<!-- tpl: 3–5 pointers
|
|
142
|
-
|
|
143
|
+
<!-- tpl: Prefer 3–5 useful pointers; include enough to resolve the existing
|
|
144
|
+
implementation and governing requirements. This is where judgment is
|
|
143
145
|
preserved without pseudocode: point at code/docs to read BEFORE touching any
|
|
144
146
|
file. Greenfield (no existing pattern)? Say so explicitly and point at the
|
|
145
147
|
docs/architecture/ doc that defines the shape. Every pointer must resolve
|
|
@@ -150,7 +152,7 @@ never selected materialised nothing here. -->
|
|
|
150
152
|
|
|
151
153
|
| File | Action | Why |
|
|
152
154
|
|------|--------|-----|
|
|
153
|
-
| `path/to/file.ext` | CREATE / EDIT / DELETE | {one line — what changes about this file's role} |
|
|
155
|
+
| `path/to/file.ext` | CREATE / EDIT / DELETE | {{fill:one line — what changes about this file's role}} |
|
|
154
156
|
|
|
155
157
|
<!-- tpl: Every file created, modified, or deleted. Scopes the length gates,
|
|
156
158
|
orphan sweeps, and review effort; per AGENTS.md the executing agent may only
|
|
@@ -161,8 +163,8 @@ no path filter (dispatch/write_spec.md, Authoring discipline). -->
|
|
|
161
163
|
|
|
162
164
|
## Applicable Rules
|
|
163
165
|
|
|
164
|
-
- **`docs/greptile-learnings/RULES.md`** — {specific rule IDs this diff trips: e.g. UFS, NDC, NLR, ORP, FLL}
|
|
165
|
-
- {per-surface rule file} — {why it applies to this diff}
|
|
166
|
+
- **`docs/greptile-learnings/RULES.md`** — {{fill:specific rule IDs this diff trips: e.g. UFS, NDC, NLR, ORP, FLL}}
|
|
167
|
+
- {{fill:per-surface rule file}} — {{fill:why it applies to this diff}}
|
|
166
168
|
|
|
167
169
|
<!-- tpl: The rule files the executing agent re-reads BEFORE EXECUTE and
|
|
168
170
|
re-checks during VERIFY. Name the exact rule IDs the diff will trip — generic
|
|
@@ -177,8 +179,8 @@ set only — docs/greptile-learnings/RULES.md; no other rule files apply." -->
|
|
|
177
179
|
|
|
178
180
|
| Gate | Fires? | Satisfaction strategy |
|
|
179
181
|
|------|--------|-----------------------|
|
|
180
|
-
| {gate from this repository's AGENTS.md dispatch index} | {yes/no — why} | {how the diff stays clean} |
|
|
181
|
-
| File & Function Length (≤350/≤50/≤70) | {yes/no} | {split plan if a file approaches the cap} |
|
|
182
|
+
| {{fill:gate from this repository's AGENTS.md dispatch index}} | {{fill:yes/no — why}} | {{fill:how the diff stays clean}} |
|
|
183
|
+
| File & Function Length (≤350/≤50/≤70) | {{fill:yes/no}} | {{fill:split plan if a file approaches the cap}} |
|
|
182
184
|
|
|
183
185
|
<!-- tpl: Which Action-Triggered Guards this Pull Request (PR) WILL trip, and
|
|
184
186
|
how each stays clean — pre-declared so the agent plans for them instead of
|
|
@@ -190,7 +192,7 @@ replace the table with "N/A — docs/markdown only." -->
|
|
|
190
192
|
|
|
191
193
|
## Prior-Art / Reference Implementations
|
|
192
194
|
|
|
193
|
-
- **Reference:** {path / codebase} — {one line on alignment, or the justified divergence}
|
|
195
|
+
- **Reference:** {{fill:path / codebase}} — {{fill:one line on alignment, or the justified divergence}}
|
|
194
196
|
|
|
195
197
|
<!-- tpl: SOUL.md rule — before proposing architecture, find the reference
|
|
196
198
|
codebase; there almost always is one. Name it so the agent mirrors a known-good
|
|
@@ -205,22 +207,25 @@ divergence. API → docs/REST_API_DESIGN_GUIDELINES.md + the closest existing
|
|
|
205
207
|
handler. Schema → the nearest migration + docs/SCHEMA_CONVENTIONS.md. UI →
|
|
206
208
|
design-system primitives + theme.css tokens. Cite each doc only where this
|
|
207
209
|
repository materialises it. Greenfield → "no prior art; shape
|
|
208
|
-
defined in docs/architecture/{doc}.md." -->
|
|
210
|
+
defined in docs/architecture/{{fill:doc}}.md." -->
|
|
209
211
|
|
|
210
212
|
## Sections (implementation slices)
|
|
211
213
|
|
|
212
|
-
### §1 — {Slice title}
|
|
214
|
+
### §1 — {{fill:Slice title}}
|
|
213
215
|
|
|
214
|
-
{
|
|
216
|
+
{{fill:what this slice delivers, why it exists, and what it unblocks}}. Non-obvious choice → **Implementation default:** `{{fill:choice}}` because `{{fill:reason}}`.
|
|
215
217
|
|
|
216
|
-
- **Dimension 1.1** — {smallest verifiable behaviour} → Test `test_…`
|
|
217
|
-
- **Dimension 1.2** — {
|
|
218
|
+
- **Dimension 1.1** — {{fill:smallest verifiable behaviour}} → Test `test_…`
|
|
219
|
+
- **Dimension 1.2** — {{fill:…}} → Test `test_…`
|
|
218
220
|
|
|
219
|
-
### §2 — {Slice title}
|
|
221
|
+
### §2 — {{fill:Slice title}}
|
|
220
222
|
|
|
221
|
-
{Same shape.}
|
|
223
|
+
{{fill:Same shape.}}
|
|
222
224
|
|
|
223
|
-
<!-- tpl:
|
|
225
|
+
<!-- tpl: State prerequisites and who resolves each decision: agent choice,
|
|
226
|
+
measured preparation output, or required human decision. A preparation Section
|
|
227
|
+
may establish budgets or fixtures needed by later Sections; name its completion
|
|
228
|
+
condition. Each Section: WHAT one slice delivers and WHY (not how); numbered
|
|
224
229
|
Dimensions map 1:1 to Tests and get marked DONE in the same commit as their
|
|
225
230
|
code. The agent picks each Implementation default unless it has evidence to
|
|
226
231
|
deviate. Good: "§3 — Replay idempotency. Receiver dedupes on the delivery id.
|
|
@@ -234,7 +239,7 @@ the existing dedupe pattern and writes the call. -->
|
|
|
234
239
|
## Interfaces
|
|
235
240
|
|
|
236
241
|
```
|
|
237
|
-
{HTTP endpoints, request/response shapes, internal signatures other code depends on}
|
|
242
|
+
{{fill:HTTP endpoints, request/response shapes, internal signatures other code depends on}}
|
|
238
243
|
```
|
|
239
244
|
|
|
240
245
|
<!-- tpl: Lock the interface — the surface the agent must NOT change without
|
|
@@ -246,7 +251,7 @@ implementation. -->
|
|
|
246
251
|
|
|
247
252
|
| Mode | Cause | Handling (system response + what the caller observes) |
|
|
248
253
|
|------|-------|--------------------------------------------------------|
|
|
249
|
-
| {short name} | {trigger} | {response + observable} |
|
|
254
|
+
| {{fill:short name}} | {{fill:trigger}} | {{fill:response + observable}} |
|
|
250
255
|
|
|
251
256
|
<!-- tpl: Every failure path the agent must handle; each row → a negative/
|
|
252
257
|
integration test in the Test Specification. Cover at minimum: timeout,
|
|
@@ -255,7 +260,7 @@ dependency unavailable. -->
|
|
|
255
260
|
|
|
256
261
|
## Invariants
|
|
257
262
|
|
|
258
|
-
1. {Invariant} — {how it's enforced}
|
|
263
|
+
1. {{fill:Invariant}} — {{fill:how it's enforced}}
|
|
259
264
|
|
|
260
265
|
<!-- tpl: Each MUST be enforceable by code (compiler, lint, comptime assertion,
|
|
261
266
|
runtime check) — NOT by review discipline. If a human can violate it silently,
|
|
@@ -265,7 +270,7 @@ it's not an invariant. None → "N/A — no invariants." -->
|
|
|
265
270
|
|
|
266
271
|
| Metric / event | Owner | Fires when | Properties allowed | Privacy guard | Test proof |
|
|
267
272
|
|----------------|-------|------------|--------------------|---------------|------------|
|
|
268
|
-
| `{event_name}` | {product / ops / not applicable} | {exact user or system action} | {coarse product context, resource id, duration, outcome} | {no raw email/password/token/One-Time Password (OTP)/Secure Shell (SSH) key material} | `{test_name}` |
|
|
273
|
+
| `{{fill:event_name}}` | {{fill:product / ops / not applicable}} | {{fill:exact user or system action}} | {{fill:coarse product context, resource id, duration, outcome}} | {{fill:no raw email/password/token/One-Time Password (OTP)/Secure Shell (SSH) key material}} | `{{fill:test_name}}` |
|
|
269
274
|
|
|
270
275
|
<!-- tpl: Every realized spec declares what product or operational signal it
|
|
271
276
|
adds, or explicitly why none: internal-only cleanup → the single row "not
|
|
@@ -284,7 +289,7 @@ are implementation gaps, not notes for later. -->
|
|
|
284
289
|
|
|
285
290
|
| Dimension | Tier | Test | Asserts (concrete inputs → expected output) |
|
|
286
291
|
|-----------|------|------|---------------------------------------------|
|
|
287
|
-
| 1.1 | {unit / integration / e2e} | `test_<short_name>` | {one-line behavioural claim} |
|
|
292
|
+
| 1.1 | {{fill:unit / integration / e2e / manual}} | `test_<short_name>` | {{fill:one-line behavioural claim}} |
|
|
288
293
|
|
|
289
294
|
<!-- tpl: Prose-and-assertions only — no test code. One row per Dimension;
|
|
290
295
|
bound to the /orly-write-unit-test skill; ≥50% negative paths; every Failure Mode
|
|
@@ -296,8 +301,10 @@ user-centric scenario via test-e2e* walking the real path (subprocess CLI /
|
|
|
296
301
|
real HTTP request / rendered UI); a unit test is not a substitute. Also
|
|
297
302
|
include regression rows (pre-existing behaviour that must not change — "N/A —
|
|
298
303
|
greenfield" if none) and idempotency/replay rows (any retry semantics).
|
|
304
|
+
Human acceptance uses tier manual, with its procedure, required person, and
|
|
305
|
+
durable evidence. An agent cannot manufacture human sign-off.
|
|
299
306
|
Non-self-evident input shape → point at a fixture
|
|
300
|
-
(samples/fixtures/m{N}-fixtures/{name}.json); don't inline JSON. Hard-to-
|
|
307
|
+
(samples/fixtures/m{{fill:N}}-fixtures/{{fill:name}}.json); don't inline JSON. Hard-to-
|
|
301
308
|
describe behaviour in prose ⇒ the Goal is fuzzy — fix the Goal, not this
|
|
302
309
|
table. -->
|
|
303
310
|
|
|
@@ -305,35 +312,45 @@ table. -->
|
|
|
305
312
|
|
|
306
313
|
| # | Criterion (observable outcome) | Verify (copy-paste) | Expected | Priority | Graded (VERIFY) |
|
|
307
314
|
|---|--------------------------------|---------------------|----------|----------|-----------------|
|
|
308
|
-
| R1 | {outcome the user can observe} (§1) | `{command}` | {exit 0 / substring / 0 matches} | P0 | |
|
|
315
|
+
| R1 | {{fill:outcome the user can observe}} (§1) | `{{fill:command}}` | {{fill:exit 0 / substring / 0 matches}} | P0 | |
|
|
309
316
|
| R2 | Diff stays inside Files Changed | `git diff --name-only origin/main...HEAD` | 0 paths missing from the Files Changed table | P0 | |
|
|
310
|
-
| S1 | Conform gates green | `{conform command from .oracle/orly.json, verbatim}` | exit 0 | P0 | |
|
|
311
|
-
| S2 | Unit tests pass | `{verify.unit command from .oracle/orly.json, verbatim}` | exit 0 | P0 | |
|
|
312
|
-
| S3 | Slow tier green (code-carrying branch) | `{one row per declared
|
|
317
|
+
| S1 | Conform gates green | `{{fill:conform command from .oracle/orly.json, verbatim}}` | exit 0 | P0 | |
|
|
318
|
+
| S2 | Unit tests pass | `{{fill:verify.unit command from .oracle/orly.json, verbatim}}` | exit 0 | P0 | |
|
|
319
|
+
| S3 | Slow tier green (code-carrying branch) | `{{fill:one row per remaining declared verify.* command, verbatim}}` | exit 0 | P0 | |
|
|
313
320
|
| S4 | No secrets | `gitleaks detect` | exit 0 | P0 | |
|
|
314
321
|
| S5 | No oversize source file | `git diff --name-only origin/main...HEAD \| grep -v '\.md$' \| xargs wc -l 2>/dev/null \| awk '$1>350 && $2!="total"'` | no output | P0 | |
|
|
315
322
|
| S6 | Orphan sweep | Dead Code Sweep greps | 0 matches | P0 | |
|
|
316
323
|
|
|
317
|
-
**Command source rule:** every
|
|
324
|
+
**Command source rule:** copy every declared `conform` and `verify.*` invocation from `.oracle/orly.json` into a Verify cell, verbatim, with an Expected value. Include conditional suites; the final gate decides applicability from the actual branch diff. Additional spec-specific commands, secret scans, and named manual checks are allowed. Missing configuration must be completed before authoring. See `dispatch/lifecycle.md` for command timing; baseline metadata is pending at opening and measured before the Pull Request.
|
|
318
325
|
|
|
319
|
-
**Grading protocol (VERIFY):** run
|
|
326
|
+
**Grading protocol (VERIFY):** run each spec-specific Verify command verbatim; Graded = ✅/❌ + one decisive output line. Repository-command rows point to the final `orly gate pr` results in Pull Request Session Notes, so recording those results does not require another code commit and suite run. **Ship gate:** every required check must pass before the Pull Request is ready; missing evidence or any ❌ returns to EXECUTE. A P1 ❌ requires an Indy-acked deferral quote in Discovery. A P0 may also be **MOVED** — see below.
|
|
327
|
+
|
|
328
|
+
**A P0 whose SCOPE moves is not a P0 shipped red.** Met and unmet are not the only two states a criterion has, and a gate that pretends otherwise forces an agent to invent a third. One did, twice in a day, before this clause existed.
|
|
329
|
+
|
|
330
|
+
A deferral and a transfer are different claims. A **deferral** leaves work unowned inside a closed spec, which is what the P0 gate exists to prevent — the P1 quote is as far as that goes. A **transfer** moves the criterion whole: its Dimensions, its verification and its rubric row land in a named successor spec that carries them as its own P0. Nothing is less owned afterwards; it is owned somewhere else.
|
|
331
|
+
|
|
332
|
+
Mark such a row `MOVED to M{{fill:N}}_{{fill:NNN}} R{{fill:n}}` and it is not ❌, on three conditions, all of which must hold:
|
|
333
|
+
|
|
334
|
+
1. The successor spec **exists** and carries the criterion as a rubric row of its own. A successor that does not carry the row is a deferral wearing a new word, and fails the gate as before.
|
|
335
|
+
2. Both specs record the mapping — the closing spec names where each Dimension went, the successor names what it inherited. One-sided assertion is not a transfer.
|
|
336
|
+
3. Discovery carries the **owner's verbatim quote** authorising it, in the deferral format. An agent-authored transfer is agent-authored scope reduction.
|
|
337
|
+
|
|
338
|
+
A MOVED row is never rendered ✅. The criterion has not been met; it has changed owner, and the rubric says which.
|
|
320
339
|
|
|
321
340
|
<!-- tpl: The single scoring surface — no other scoreboard. 5–12 rows after
|
|
322
341
|
pruning: one per Section outcome, failure class, or hygiene gate — never one
|
|
323
|
-
per Dimension (that ledger is the Test Specification).
|
|
324
|
-
|
|
325
|
-
diff's surface needs; delete rows whose command this repository does not
|
|
326
|
-
declare. Expected litmus: every Expected is
|
|
342
|
+
per Dimension (that ledger is the Test Specification). Repository suite commands come verbatim from `.oracle/orly.json`; expand S3
|
|
343
|
+
for every remaining declared lane. Additional spec-specific checks stay explicit. Expected litmus: every Expected is
|
|
327
344
|
mechanically checkable — an exit code, a literal substring, or a match count;
|
|
328
345
|
can't write it that way → the criterion is fuzzy — fix the criterion, not the
|
|
329
346
|
grading. Authoring fills every column except Graded; VERIFY fills Graded. -->
|
|
330
347
|
|
|
331
348
|
### Behaviour evals
|
|
332
349
|
|
|
333
|
-
- **Grounding rule:** {one sentence the output must never violate}
|
|
334
|
-
- **Golden set:** `samples/fixtures/{path}` — {N} cases across {3–5 coverage axes, incl. the nightmare case}. A failure found in the wild becomes a new case; the set only grows.
|
|
335
|
-
- **Ship threshold:** grounding 100% · task pass ≥{N}% · 0 critical failures on {nightmare case}. Each threshold is one rubric row with the command that computes it.
|
|
336
|
-
- **Fallback:** below threshold or low confidence → {named recoverable behaviour}; fabricated output is a P0 ❌.
|
|
350
|
+
- **Grounding rule:** {{fill:one sentence the output must never violate}}
|
|
351
|
+
- **Golden set:** `samples/fixtures/{{fill:path}}` — {{fill:N}} cases across {{fill:3–5 coverage axes, incl. the nightmare case}}. A failure found in the wild becomes a new case; the set only grows.
|
|
352
|
+
- **Ship threshold:** grounding 100% · task pass ≥{{fill:N}}% · 0 critical failures on {{fill:nightmare case}}. Each threshold is one rubric row with the command that computes it.
|
|
353
|
+
- **Fallback:** below threshold or low confidence → {{fill:named recoverable behaviour}}; fabricated output is a P0 ❌.
|
|
337
354
|
|
|
338
355
|
<!-- tpl: Delete this whole sub-section unless the diff changes prompt/model/
|
|
339
356
|
agent behaviour. Grounding-rule example: "responses cite only retrieved rows,
|
|
@@ -345,13 +362,13 @@ never invented identifiers." -->
|
|
|
345
362
|
|
|
346
363
|
| File to delete | Verify |
|
|
347
364
|
|----------------|--------|
|
|
348
|
-
| `{path/to/old_file.ext}` | `test ! -f {path/to/old_file.ext}` |
|
|
365
|
+
| `{{fill:path/to/old_file.ext}}` | `test ! -f {{fill:path/to/old_file.ext}}` |
|
|
349
366
|
|
|
350
367
|
**2. Orphaned references — zero remaining imports/uses.**
|
|
351
368
|
|
|
352
369
|
| Deleted symbol/import | Grep | Expected |
|
|
353
370
|
|-----------------------|------|----------|
|
|
354
|
-
| `{old_symbol}` | `grep -rn "{old_symbol}" src/ \| head` | 0 matches |
|
|
371
|
+
| `{{fill:old_symbol}}` | `grep -rn "{{fill:old_symbol}}" src/ \| head` | 0 matches |
|
|
355
372
|
|
|
356
373
|
<!-- tpl: Mandatory when the spec deletes or replaces files: for every deleted
|
|
357
374
|
file and removed/renamed public symbol, grep the repo; non-zero = stale. Use
|
|
@@ -360,7 +377,7 @@ both tables with "N/A — no files deleted." -->
|
|
|
360
377
|
|
|
361
378
|
## Out of Scope
|
|
362
379
|
|
|
363
|
-
- {Item explicitly not in this spec — points at a follow-up spec or "future work"}
|
|
380
|
+
- {{fill:Item explicitly not in this spec — points at a follow-up spec or "future work"}}
|
|
364
381
|
|
|
365
382
|
---
|
|
366
383
|
|
|
@@ -371,16 +388,16 @@ working sections stay front-loaded. -->
|
|
|
371
388
|
|
|
372
389
|
## Product Clarity (authoring record)
|
|
373
390
|
|
|
374
|
-
1. **Successful user moment** — {the single observable moment that proves this worked — a scene, not a metric}
|
|
375
|
-
2. **Preserved user behaviour** — {what users do today that keeps working unchanged; breaking any of it is a redesign}
|
|
376
|
-
3. **Optimal-way check** — {is this the most direct way to deliver moment #1? name the gap to the unconstrained-optimal shape and why it's acceptable now}
|
|
377
|
-
4. **Rebuild-vs-iterate** — {would a larger refactor serve better? verdict here; rationale in Decomposition below. A refactor that trades run-to-run determinism away is wrong by default}
|
|
378
|
-
5. **What we build** — {the shortest artifact list that delivers moment #1}
|
|
379
|
-
6. **What we do NOT build** — {adjacent scope rejected, one-line reason each — seeds Out of Scope}
|
|
380
|
-
7. **Fit with existing features** — {what this compounds with; the one feature it must not destabilize}
|
|
381
|
-
8. **Surface order** — {CLI-first (repo default), UI-first, or both; justify divergence}
|
|
382
|
-
9. **Dashboard restraint** — {what the UI must hide until the signal behind it is real: no controls before evidence, no quality claims before counters}
|
|
383
|
-
10. **Confused-user next step** — {the self-serve move (a command, an error message, a doc); "file a ticket" means a surface is missing from item 5}
|
|
391
|
+
1. **Successful user moment** — {{fill:the single observable moment that proves this worked — a scene, not a metric}}
|
|
392
|
+
2. **Preserved user behaviour** — {{fill:what users do today that keeps working unchanged; breaking any of it is a redesign}}
|
|
393
|
+
3. **Optimal-way check** — {{fill:is this the most direct way to deliver moment #1? name the gap to the unconstrained-optimal shape and why it's acceptable now}}
|
|
394
|
+
4. **Rebuild-vs-iterate** — {{fill:would a larger refactor serve better? verdict here; rationale in Decomposition below. A refactor that trades run-to-run determinism away is wrong by default}}
|
|
395
|
+
5. **What we build** — {{fill:the shortest artifact list that delivers moment #1}}
|
|
396
|
+
6. **What we do NOT build** — {{fill:adjacent scope rejected, one-line reason each — seeds Out of Scope}}
|
|
397
|
+
7. **Fit with existing features** — {{fill:what this compounds with; the one feature it must not destabilize}}
|
|
398
|
+
8. **Surface order** — {{fill:CLI-first (repo default), UI-first, or both; justify divergence}}
|
|
399
|
+
9. **Dashboard restraint** — {{fill:what the UI must hide until the signal behind it is real: no controls before evidence, no quality claims before counters}}
|
|
400
|
+
10. **Confused-user next step** — {{fill:the self-serve move (a command, an error message, a doc); "file a ticket" means a surface is missing from item 5}}
|
|
384
401
|
|
|
385
402
|
<!-- tpl: Indy's product questions, answered in order at authoring BEFORE the
|
|
386
403
|
implementation sections are written — so the authoring agent holds the product
|
|
@@ -391,9 +408,9 @@ no user surface" with the reason. -->
|
|
|
391
408
|
|
|
392
409
|
## Decomposition & alternatives (patch vs refactor)
|
|
393
410
|
|
|
394
|
-
- **Chosen shape:** {why this Section/Workstream split — the decomposition rationale}
|
|
395
|
-
- **Alternatives considered:** {
|
|
396
|
-
- **Patch-vs-refactor verdict:** this is a **{patch | refactor}** because {reason}. {If a larger refactor is the right long game but out of scope, name the follow-up spec rather than silently mud-patching.}
|
|
411
|
+
- **Chosen shape:** {{fill:why this Section/Workstream split — the decomposition rationale}}
|
|
412
|
+
- **Alternatives considered:** {{fill:≥1 — the larger refactor or the smaller patch — and why rejected for now}}
|
|
413
|
+
- **Patch-vs-refactor verdict:** this is a **{{fill:patch | refactor}}** because {{fill:reason}}. {{fill:If a larger refactor is the right long game but out of scope, name the follow-up spec rather than silently mud-patching.}}
|
|
397
414
|
|
|
398
415
|
<!-- tpl: Indy's rule — don't ship a mud-patch when the problem wants a
|
|
399
416
|
refactor, and don't refactor when a patch is right. Match solution-size to
|
package/docs/VERIFY_TIERS.md
CHANGED
|
@@ -14,35 +14,29 @@ page carries only what is specific to this repository.
|
|
|
14
14
|
| version | `make check-version` | Boundary. `VERSION` against `build.zig.zon`, `cli/package.json` and both `rustd/Cargo.toml` sites. |
|
|
15
15
|
| integration | `make test-integration-rustd` | Boundary. Live Postgres and Redis via docker compose, schemas reset per run. `orly gate pr` skips it on a branch carrying no code. |
|
|
16
16
|
|
|
17
|
-
These
|
|
18
|
-
`orly gate pr` runs
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
**Which gate runs which row.** `orly gate work` runs the `conform` row alone and
|
|
27
|
-
is what the pre-commit hook fires, so it costs seconds. `orly gate verify` runs
|
|
28
|
-
the lint, unit, and version rows and is what pre-push fires. `orly gate pr` runs
|
|
29
|
-
the integration row plus the whole-branch and spec criteria, and is what
|
|
30
|
-
CHORE(close) fires — it skips the fast rows because `git.pushed` proves HEAD is
|
|
31
|
-
the commit pre-push already graded.
|
|
32
|
-
|
|
33
|
-
One lane needs live datastores, and only one. `make/test-infra.mk` brings up
|
|
34
|
-
docker compose Postgres and Redis for `test-integration-rustd`; `make
|
|
35
|
-
test-unit-all` stays datastore-free, because every Rust test needing one is
|
|
36
|
-
`#[ignore]`d and runs only in that lane. `KEEP_TEST_STATE=1` skips the reset for
|
|
37
|
-
the inner loop; Continuous Integration (CI) never sets it.
|
|
17
|
+
These are the commands `.oracle/orly.json` declares. The commit hook runs
|
|
18
|
+
conformance; `orly gate pr` runs every verification row.
|
|
19
|
+
|
|
20
|
+
The authoritative timing and gate ownership live in
|
|
21
|
+
`dispatch/lifecycle.md` §What runs at each stage. At Section completion, run
|
|
22
|
+
`cargo fmt` from `rustd/`, `make lint-rustd`, conformance over staged changes,
|
|
23
|
+
and the relevant behavior tests. The existing `make test-unit-rustd` runs the
|
|
24
|
+
whole Rust workspace; it is not a single-package check.
|
|
38
25
|
|
|
39
26
|
## Test Baseline
|
|
40
27
|
|
|
41
|
-
`make test-unit-all`
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
28
|
+
Before the Pull Request, obtain `make test-unit-all` and
|
|
29
|
+
`make test-integration-rustd` results for the spec's full `Baseline revision`.
|
|
30
|
+
Record per-package unit counts and the integration lane's own summary in the
|
|
31
|
+
baseline report; put comparable totals in `Test Baseline` and its path or run
|
|
32
|
+
URL in `Baseline evidence`. The report includes passed, failed, and skipped
|
|
33
|
+
counts, commands, and environment. Never substitute the default checkout's
|
|
34
|
+
current revision for the recorded one. Keep integration datastores isolated.
|
|
35
|
+
|
|
36
|
+
The final PR gate runs all declared commands against the implementation.
|
|
37
|
+
Capture its output in Pull Request Session Notes, including the Test Delta.
|
|
38
|
+
Unit growth alone does not prove coverage; explain removals and changes in test
|
|
39
|
+
selection. Zero or negative growth on code-adding work requires justification.
|
|
46
40
|
|
|
47
41
|
## Wire fixtures
|
|
48
42
|
|
package/llms.txt
CHANGED
|
@@ -36,17 +36,19 @@ Run from the repository root. Execute these in order and stop at the first
|
|
|
36
36
|
non-zero exit.
|
|
37
37
|
|
|
38
38
|
```bash
|
|
39
|
-
# 1.
|
|
40
|
-
|
|
39
|
+
# 1. install the executable used by Git hooks
|
|
40
|
+
bun add -g @agentsfleet/orly
|
|
41
41
|
|
|
42
|
-
# 2.
|
|
43
|
-
|
|
42
|
+
# 2. preview, then initialize
|
|
43
|
+
orly init --dry-run
|
|
44
|
+
orly init
|
|
44
45
|
|
|
45
|
-
# 3.
|
|
46
|
-
|
|
46
|
+
# 3. complete any reported setup items, then check configuration and files
|
|
47
|
+
orly doctor
|
|
47
48
|
```
|
|
48
49
|
|
|
49
50
|
Then commit the result. Teammates receive the rules on clone.
|
|
51
|
+
Keep `orly` on `PATH`; hooks use the installed executable.
|
|
50
52
|
|
|
51
53
|
IMPORTANT: git does not clone hooks. Each checkout must run `orly init` once.
|
|
52
54
|
If a human reports that gates are not firing, this is the first thing to check.
|
|
@@ -67,8 +69,11 @@ If a human reports that gates are not firing, this is the first thing to check.
|
|
|
67
69
|
| `.oracle/orly.json` | packs, gate commands, install record | shared — edit the commands |
|
|
68
70
|
|
|
69
71
|
`orly init` seeds `.oracle/orly.json` with gate commands found in `Makefile` or
|
|
70
|
-
`package.json`.
|
|
71
|
-
|
|
72
|
+
`package.json`. Read the repository's build files to complete missing commands;
|
|
73
|
+
ask only when the repository does not define a check. Do not invent commands.
|
|
74
|
+
Declare `conform` and at least one named `verify.*` command. Documentation
|
|
75
|
+
repositories can use `verify.docs` without application unit or integration
|
|
76
|
+
suites. `init` reports missing configuration; `doctor` rejects it.
|
|
72
77
|
|
|
73
78
|
---
|
|
74
79
|
|
|
@@ -91,7 +96,7 @@ move it. `orly gate pr` reads where it ended up.
|
|
|
91
96
|
```
|
|
92
97
|
docs/v1/pending/ spec written, committed on main
|
|
93
98
|
│
|
|
94
|
-
│ CHORE(open) branch cut,
|
|
99
|
+
│ CHORE(open) branch cut, comparison revision recorded, baseline measurement pending
|
|
95
100
|
▼
|
|
96
101
|
docs/v1/active/ PLAN → EXECUTE → CONFORM → VERIFY → REVIEW → DOCUMENT → COMMIT
|
|
97
102
|
│
|
|
@@ -117,14 +122,14 @@ Rules for you:
|
|
|
117
122
|
| same gate fails twice on the same cause | stop, report the gate name and output to the user |
|
|
118
123
|
| `orly init` refuses a file | orly did not write that file. Report it. Do **not** pass `--force` on your own initiative |
|
|
119
124
|
| hooks not firing | run `git config core.hooksPath` and confirm `orly init` was run in this checkout |
|
|
120
|
-
| gate command missing from `.oracle/orly.json` |
|
|
125
|
+
| gate command missing from `.oracle/orly.json` | read the build files and declare the existing check; ask only if none exists |
|
|
121
126
|
| `orly gate pr` red and the failure is genuinely inapplicable | ask the user; only they authorise `orly override` |
|
|
122
127
|
| gstack `/review` unavailable | continue; orly records the skipped review in the PR notes |
|
|
123
128
|
|
|
124
129
|
Overrides are recorded as an empty commit that rides into the Pull Request:
|
|
125
130
|
|
|
126
131
|
```bash
|
|
127
|
-
|
|
132
|
+
orly override <criterion> --reason "<why>"
|
|
128
133
|
```
|
|
129
134
|
|
|
130
135
|
Never record an override without an explicit instruction from the user, and
|
|
@@ -155,7 +160,7 @@ never with a placeholder reason.
|
|
|
155
160
|
| `orly gate` | run declared checks in order, stopping at the first failure |
|
|
156
161
|
| `orly gate pr` | the Pull Request criteria check |
|
|
157
162
|
| `orly override <criterion> --reason <why>` | record a gate exception (user-authorised only) |
|
|
158
|
-
| `orly doctor` |
|
|
163
|
+
| `orly doctor` | check installed files and required command configuration |
|
|
159
164
|
|
|
160
165
|
---
|
|
161
166
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@agentsfleet/orly",
|
|
3
|
-
"version": "0.10.
|
|
3
|
+
"version": "0.10.6",
|
|
4
4
|
"description": "Repository-scoped engineering harness: renders agent rules, materialises the gates that enforce them, and proves the PR boundary \u2014 for any coding agent.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"type": "module",
|
package/registry.json
CHANGED
|
@@ -74,6 +74,8 @@ This is the step that prevents greptile findings — the spec becomes a pre-comm
|
|
|
74
74
|
- **Every Failure Mode → a negative test.** **Every Invariant → enforced by code** (compiler, lint, comptime assertion, runtime check) — never by review discipline.
|
|
75
75
|
- **Every Metrics row → event/test proof.** User-facing or operator-facing specs declare product/operator signals, privacy guards, and analytics/funnel playbook updates; internal-only cleanup explicitly says no signal changed.
|
|
76
76
|
- **Rubric = 5–12 outcome rows** — one per Section outcome, failure class, or hygiene gate (never per Dimension); each with a copy-paste Verify command and a mechanically checkable Expected (exit code / literal substring / match count). Standard rows come pre-filled in the template — prune to the touched surface.
|
|
77
|
+
- **Stage-owned values** — keep branch and baseline measurement pending at authoring. Opening records the full comparison commit; unit and integration measurement is due before the Pull Request. The lifecycle dispatch owns command timing and hydration.
|
|
78
|
+
- **Decision ownership** — identify agent choices, preparation outputs, and required human decisions. Human acceptance uses tier `manual` with a procedure and durable evidence; never invent the person's sign-off.
|
|
77
79
|
- **Reporting spine** — **Discovery (consult log)** carries consults, skill-chain outcomes, and Indy-acked deferral quotes; the **Acceptance Rubric's Graded column** carries the VERIFY verdicts (✅/❌ + one decisive output line each). Both empty at creation, populated as work proceeds.
|
|
78
80
|
|
|
79
81
|
→ Fills **Sections (+ Dimensions)**, **Metrics & Observability**, **Failure Modes**, **Invariants**, **Test Specification (tiered)**, **Acceptance Rubric**, **Discovery**.
|
|
@@ -90,7 +92,7 @@ cp docs/TEMPLATE.md docs/v{N}/pending/M{N}_{WS}_P{P}_{CATEGORIES}_{NAME}.md
|
|
|
90
92
|
```
|
|
91
93
|
|
|
92
94
|
|
|
93
|
-
**Fill grammar** — the template body is ordered for the *executing* agent (execution read order), not for authoring; fill sections in the order of Steps 1–3 above, not top-to-bottom. Replace every `{
|
|
95
|
+
**Fill grammar** — the template body is ordered for the *executing* agent (execution read order), not for authoring; fill sections in the order of Steps 1–3 above, not top-to-bottom. Replace every `{{fill:description}}` authoring slot with instance content, then **delete every `<!-- tpl: … -->` guidance comment** — the SPEC TEMPLATE GATE BLOCKs any survivor, unfilled slot sentinels, and missing required sections. The SPEC AUTHORING RULES banner is the one comment that stays.
|
|
94
96
|
|
|
95
97
|
**Inputs** — Milestone `M{N}` (next free, sortable) · Workstream `{WS}` zero-padded (`001`…) · Priority (**P0** blocking · **P1** customer/operator-facing · **P2** tooling · **P3** deferrable) · Category set alphabetised (`API` Zig/Go · `CLI` agentsfleet/Node · `UI` Next.js · `OBS` Grafana · `SKILL` SKILL.md · `INFRA` Terraform) · Name UPPER_SNAKE_CASE ≤6 words describing the outcome (`BUN_VENDOR_UTILITIES`, not `BUMP_BUN_DEPS`) · Prototype tag (`v1.0.0`, `v2.0.0` — drives `docs/v1/` vs `docs/v2/`).
|
|
96
98
|
|
|
@@ -69,7 +69,7 @@ Spec asserts "503 on Redis down", code returns 200 → test the spec, flag the c
|
|
|
69
69
|
1. **Read the spec** — Failure Modes, Error Contracts, Concurrency Contracts, Resource Limits, Streaming Contracts tables are authoritative.
|
|
70
70
|
2. **Read `docs/greptile-learnings/RULES.md`** — every rule maps to a regression test.
|
|
71
71
|
3. **Bring up real deps** — `make up` (or stack equivalent). Verify health before writing tests.
|
|
72
|
-
4. **
|
|
72
|
+
4. **Follow lifecycle timing** — record the comparison revision at opening; measure full unit and integration baselines before the Pull Request. Run focused reproduction and Section proofs while implementing; first application hydration occurs at Section verification. `dispatch/lifecycle.md` owns the sequence.
|
|
73
73
|
5. **Map the ordered request path** — list every dependency interaction in execution order, including repeated acquisitions of the same resource, plus every `catch`/`orelse`/`except`/`Err` in the chain. That list seeds T4 and its partial-completion matrix.
|
|
74
74
|
|
|
75
75
|
## Three execution modes
|
|
@@ -51,7 +51,7 @@ Spec contradicts runtime → surface the conflict before testing. Code contradic
|
|
|
51
51
|
1. **Read the spec, then the code** — spec defines what to test; code is what gets tested. Don't let code shape expectations.
|
|
52
52
|
2. **Read `docs/greptile-learnings/RULES.md`** — every rule is a regression test.
|
|
53
53
|
3. **Detect the stack** from project files (`pyproject.toml`, `Cargo.toml`, `build.zig`, `package.json` + `react`/`bin`, `*.sh`, `openapi.yaml`).
|
|
54
|
-
4. **
|
|
54
|
+
4. **Follow lifecycle timing** — record the comparison revision at opening; measure full unit and integration baselines before the Pull Request. Run focused reproduction and Section proofs while implementing; first application hydration occurs at Section verification. `dispatch/lifecycle.md` owns the sequence.
|
|
55
55
|
5. **Follow existing naming and fixture conventions.**
|
|
56
56
|
|
|
57
57
|
## Step 0 — the diff is the spec: build the required-test ledger first
|