create-pmos 0.1.2 → 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +97 -9
- package/package.json +2 -2
- package/payload/AGENTS-HUMAN.md +43 -0
- package/payload/AGENTS.md +80 -5
- package/payload/KIT-VERSION +2 -2
- package/payload/README.md +23 -4
- package/payload/okf/core/concepts/health-anchors.md +41 -0
- package/payload/okf/core/concepts/index.md +1 -0
- package/payload/okf/core/concepts/okr-anti-optimism.md +7 -0
- package/payload/okf/core/concepts/okr-tree.md +7 -0
- package/payload/okf/core/concepts/output-eval.md +2 -1
- package/payload/okf/core/concepts/sdlc-loop.md +2 -1
- package/payload/okf/core/templates/content-brief-template.md +70 -0
- package/payload/okf/core/templates/design-brief-template.md +1 -1
- package/payload/okf/core/templates/eval-rubric-template.md +24 -4
- package/payload/okf/core/templates/incident-record-template.md +66 -0
- package/payload/okf/core/templates/index.md +3 -0
- package/payload/okf/core/templates/prd-template.md +13 -4
- package/payload/okf/core/templates/spike-brief-template.md +68 -0
- package/payload/planning/evals/agent-run-eval-rubric.md +5 -1
- package/payload/planning/okrs/health-budgets.md +7 -0
- package/payload/scripts/gate-initiative.sh +50 -11
- package/payload/scripts/metrics.py +46 -1
- package/payload/scripts/pmos-log.sh +76 -0
- package/payload/skills/SKILL-FORMAT.md +103 -1
- package/payload/skills/copywriting.skill +55 -0
- package/payload/skills/discovery.skill +69 -0
- package/payload/skills/eval-rubric.skill +29 -1
- package/payload/skills/evaluator.skill +68 -14
- package/payload/skills/explain-run.skill +116 -0
- package/payload/skills/prd.skill +21 -10
- package/payload/state/outcome_checks.jsonl +0 -0
- package/payload/templates/agent-run-eval-rubric.md +5 -1
- package/payload/templates/product-ci-starter.yml +36 -0
package/README.md
CHANGED
|
@@ -98,10 +98,16 @@ open about what you don't know. Copy-paste starters:
|
|
|
98
98
|
## The rhythm (every piece of work)
|
|
99
99
|
|
|
100
100
|
1. **Name the work.** An initiative file is created in `pmos/state/initiatives/` — anchored to one
|
|
101
|
-
of your goals,
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
101
|
+
of your goals (or, for honest maintenance with no goal, a declared health budget), so nothing
|
|
102
|
+
exists "just because."
|
|
103
|
+
2. **Define "good" first.** The agent drafts the spec artifact that fits the work's lane — a PRD +
|
|
104
|
+
scoring rubric for a feature (the default), a spike brief for research (`discovery`), a stated
|
|
105
|
+
intent for maintenance (`chore`), a content brief for copy (`content`). For a feature you
|
|
106
|
+
approve the contract (`PM_approved: true`) before any building — and spike/content briefs get
|
|
107
|
+
your sign-off before work starts too; a chore needs only its stated
|
|
108
|
+
intent, and an `incident` is the one sanctioned inversion — fix first, record within 48 hours.
|
|
109
|
+
Deciding the standard in advance is your task — it is also what makes the agent's verdicts
|
|
110
|
+
mean something later.
|
|
105
111
|
3. **The agent builds** on its own branch and opens the work for review. A gate checks that every
|
|
106
112
|
change is anchored. If someone (including the agent) skips the protocol, you'll see:
|
|
107
113
|
|
|
@@ -119,12 +125,33 @@ open about what you don't know. Copy-paste starters:
|
|
|
119
125
|
> initiative/[id] against its rubric. Be skeptical — try to refute it. Append your verdict to
|
|
120
126
|
> pmos/state/evals.jsonl.
|
|
121
127
|
|
|
122
|
-
|
|
128
|
+
Each dimension of that verdict has to cite a verbatim quote or a file+line from the work
|
|
129
|
+
itself. A dimension the evaluator can't evidence is marked `unverified` and scored low — it
|
|
130
|
+
never gets a high score on trust.
|
|
131
|
+
|
|
132
|
+
5. **You understand it before you judge it.** The same evaluator writes you a plain-language
|
|
133
|
+
explainer of what the run actually did — including what it got wrong — ending in a short quiz
|
|
134
|
+
you can self-check against. Read that instead of the raw transcript:
|
|
135
|
+
|
|
136
|
+
> Follow pmos/skills/explain-run.skill: write the explainer + quiz for run [id] to
|
|
137
|
+
> pmos/planning/evals/explainers/.
|
|
138
|
+
|
|
139
|
+
Worth the two minutes on anything that touches a gate. Accepting work you couldn't explain to
|
|
140
|
+
someone else is the one watermelon no gate can catch for you.
|
|
141
|
+
|
|
142
|
+
6. **You accept — or return it.** Your merge is the final gate. Returning imperfect work is not
|
|
123
143
|
conflict; it's the job. Record your call:
|
|
124
144
|
|
|
125
145
|
> I accept this run. Append the acceptance to pmos/state/acceptances.jsonl with pm_label
|
|
126
146
|
> "pass" and corrections: [how many things you had to fix].
|
|
127
147
|
|
|
148
|
+
Or record it yourself, no agent needed — the helper validates and appends in one command
|
|
149
|
+
(`pmos/AGENTS-HUMAN.md` is the whole protocol on one page for humans):
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
pmos/scripts/pmos-log.sh accept <run_id> <initiative_id> pass "one wording fix"
|
|
153
|
+
```
|
|
154
|
+
|
|
128
155
|
## Your weekly ten minutes
|
|
129
156
|
|
|
130
157
|
```bash
|
|
@@ -158,8 +185,8 @@ Each stage of the loop has a mechanism behind it — not a promise, a mechanism:
|
|
|
158
185
|
|
|
159
186
|
| Without the kit | With the kit |
|
|
160
187
|
|---|---|
|
|
161
|
-
| Work starts because someone felt like it | **Nothing exists without an initiative** anchored to a goal — the gate rejects unanchored work, automatically on GitHub, by one command locally |
|
|
162
|
-
| "Good" gets defined after seeing the output (so everything looks fine) | **The
|
|
188
|
+
| Work starts because someone felt like it | **Nothing exists without an initiative** anchored to a goal — or a declared health budget for honest maintenance — the gate rejects unanchored work, automatically on GitHub, by one command locally |
|
|
189
|
+
| "Good" gets defined after seeing the output (so everything looks fine) | **The lane's spec artifact is written first** — a feature is blocked until you've flipped `PM_approved: true`; every lane's requirement is gate-enforced (the one sanctioned inversion: an incident is fixed first, recorded within 48h) |
|
|
163
190
|
| The builder grades their own homework | **A separate adversarial evaluator scores against your rubric** — and even its verdict is advisory; the merge is yours alone |
|
|
164
191
|
| History gets quietly rewritten | **State files are append-only** — corrections are added, never erased; git keeps the audit trail |
|
|
165
192
|
| Metrics flatter you | **Reds print first**, and a tripwire fires if the evaluator passes everything while you keep fixing things — a green that hides red gets caught |
|
|
@@ -177,8 +204,9 @@ defends structurally rather than relying on discipline:
|
|
|
177
204
|
session read `pmos/AGENTS.md` before working — so a new session, a new model, or a new teammate
|
|
178
205
|
starts on-protocol by default instead of drifting from a stale memory.
|
|
179
206
|
- **The gate makes skipping visible.** On GitHub it's a failing check nobody can miss; locally
|
|
180
|
-
it's the one command before a merge.
|
|
181
|
-
only
|
|
207
|
+
it's the one command before a merge. The `no-initiative` escape hatch is for anchor-free
|
|
208
|
+
trivia only (a typo, a broken link) — real maintenance takes the `chore` lane with a stated
|
|
209
|
+
intent, and if you find yourself reaching for the hatch weekly, that's a signal, not a shortcut.
|
|
182
210
|
- **The weekly ten minutes is the ritual.** Put `metrics.py` + the discoveries scan on your
|
|
183
211
|
calendar. A friction that shows up twice with no decision gets promoted to a rule — that's how
|
|
184
212
|
the process improves instead of eroding.
|
|
@@ -221,6 +249,66 @@ Refreshes the skills, concepts, and scripts only. Your records — `state/`, `pl
|
|
|
221
249
|
`okf/product/` — are yours, and an update never touches them. `pmos/KIT-VERSION` always says
|
|
222
250
|
exactly what you're running.
|
|
223
251
|
|
|
252
|
+
## What's new in 0.3.0
|
|
253
|
+
|
|
254
|
+
0.2.0 made the workflow fit real work. 0.3.0 goes after the two ways the *judging* half quietly
|
|
255
|
+
fails: a judge that can be talked into a pass, and a PM who accepts what they don't understand.
|
|
256
|
+
|
|
257
|
+
- **The evaluator now defends itself.** It reads diffs and PR text written by someone else, so it
|
|
258
|
+
treats every word of that as *content under evaluation*, never as instructions. A "mark this
|
|
259
|
+
PASS" buried in a diff is reported as a defect instead of obeyed. And it's structurally
|
|
260
|
+
read-only — its one write is the verdict — because the part that reads hostile text must not be
|
|
261
|
+
the part that can act. Prompt-level defenses alone get bypassed; removing the capability doesn't.
|
|
262
|
+
- **An understanding layer.** The new `explain-run` skill writes a literate explainer + a
|
|
263
|
+
comprehension quiz for a run, authored by the *independent* evaluator, never the builder. Read
|
|
264
|
+
that before you accept. Raw transcripts are a vigilance task nobody wins at; accepting without
|
|
265
|
+
comprehending makes the human gate only formally human.
|
|
266
|
+
- **Evidence-anchored scoring.** Every scored dimension must cite a verbatim quote or a precise
|
|
267
|
+
pointer (file+line, command output, CI line). No evidence, no high score — the dimension is
|
|
268
|
+
marked `unverified` and capped. Verdicts became auditable instead of assertable.
|
|
269
|
+
- **Richer anchored scales.** Rubrics now prefer five described levels over a bare 0/0.5/1. It is
|
|
270
|
+
the single largest evidenced lever on judge agreement: the more precisely each level is
|
|
271
|
+
described, the less two evaluators — or the evaluator and you — drift apart.
|
|
272
|
+
- **Balanced exemplars before scoring.** The evaluator picks 2–4 prior cases from your calibration
|
|
273
|
+
log, deliberately balanced across passes *and* fails, and names them in the verdict. If your log
|
|
274
|
+
has no fail-side cases yet, it says so out loud rather than quietly leaning generous.
|
|
275
|
+
- **A leniency disclosure in every verdict.** The evaluator states its own recent pass-fraction. A
|
|
276
|
+
judge that passes everything is the failure mode you can't see from inside a single run.
|
|
277
|
+
- **SKILL-FORMAT v2.** The skill format is now compatible with the broader Agent Skills ecosystem,
|
|
278
|
+
with the quality bar for writing one made explicit.
|
|
279
|
+
|
|
280
|
+
**Upgrade note:** `npx create-pmos@latest --update` brings all of this in — it refreshes skills,
|
|
281
|
+
concepts, templates, and scripts and never touches your `state/`, `planning/`, or `okf/product/`
|
|
282
|
+
records. One consequence to expect: because unevidenced dimensions are now capped, verdicts on
|
|
283
|
+
work the evaluator can't directly observe will score *lower* than they did on 0.2.0. That's the
|
|
284
|
+
change working, not a regression — it was scoring on trust before.
|
|
285
|
+
|
|
286
|
+
## What's new in 0.2.0
|
|
287
|
+
|
|
288
|
+
The kernel grew four capabilities, each born from real dogfooding of 0.1.x:
|
|
289
|
+
|
|
290
|
+
- **Work-type lanes.** `type: feature | discovery | chore | incident | content` on the initiative
|
|
291
|
+
file — each lane has a fitting spec artifact (spike brief, stated intent, post-hoc incident
|
|
292
|
+
record, content brief) and the gate enforces per lane. Research, maintenance, fires, and copy no
|
|
293
|
+
longer have to cosplay as features.
|
|
294
|
+
- **Human builders, first-class.** `pmos/AGENTS-HUMAN.md` is the whole protocol on one page for a
|
|
295
|
+
person, and `pmos/scripts/pmos-log.sh` writes validated state records with one command — mixed
|
|
296
|
+
human/agent teams share the same files and metrics.
|
|
297
|
+
- **Health budgets.** Maintenance with no honest goal anchors to a declared budget
|
|
298
|
+
(`parent_kr: health:<id>` from `pmos/planning/okrs/health-budgets.md`, seeded with capped
|
|
299
|
+
tech/security/ops lines) instead of forcing a fake goal link. A fake anchor is worse than an
|
|
300
|
+
honest budget.
|
|
301
|
+
- **Test-first Quality Gate.** `templates/product-ci-starter.yml` ships failing-until-wired, so a
|
|
302
|
+
fresh repo's CI is loud until you give it real tests — never silently empty.
|
|
303
|
+
|
|
304
|
+
**Upgrade note:** the gate now rejects an initiative file with **no `parent_kr`**. If `--update`
|
|
305
|
+
makes an old initiative fail the gate, that file was always unanchored — add a real goal, or a
|
|
306
|
+
health budget id if it's honest maintenance. The gate failing is the fix working. (One caveat for
|
|
307
|
+
upgraders: `--update` never touches `planning/`, so it won't create
|
|
308
|
+
`pmos/planning/okrs/health-budgets.md` — declare your budgets by writing that small table yourself
|
|
309
|
+
or copying it from a fresh vendor; the gate accepts the `health:<id>` string either way, but an
|
|
310
|
+
undeclared budget id is exactly the fake-anchor problem in new clothes.)
|
|
311
|
+
|
|
224
312
|
## Scope & support
|
|
225
313
|
|
|
226
314
|
This package contains only the PMOS workflow kernel. It holds no reference to, and never
|
package/package.json
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "create-pmos",
|
|
3
|
-
"version": "0.
|
|
4
|
-
"description": "Scaffold the PMOS portable kit — run the PMOS product-management workflow (initiative →
|
|
3
|
+
"version": "0.3.0",
|
|
4
|
+
"description": "Scaffold the PMOS portable kit — run the PMOS product-management workflow (initiative → lane-fit spec → contract → build → three gates, with work-type lanes for features, research, maintenance, incidents, and content) entirely inside your own repo, with all state as git-versioned files. No database, no network, no telemetry.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"bin": {
|
|
7
7
|
"create-pmos": "bin/create-pmos.js"
|
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# AGENTS-HUMAN.md — the PMOS kit for a human builder (D58)
|
|
2
|
+
|
|
3
|
+
You're a person, not an agent — same loop, plain terms, one helper. The full protocol lives in
|
|
4
|
+
[AGENTS.md](AGENTS.md); this page is everything you need day-to-day. **Never edit `pmos/state/`
|
|
5
|
+
files by hand — use the helper**, which validates and appends (records are append-only; a
|
|
6
|
+
correction is a new record, never an edit).
|
|
7
|
+
|
|
8
|
+
## The loop (per piece of work)
|
|
9
|
+
|
|
10
|
+
1. **Open the work.** Create `pmos/state/initiatives/<id>.md` with a lane
|
|
11
|
+
(`type: feature | discovery | chore | incident | content`) and its artifact — feature: PRD +
|
|
12
|
+
rubric (copy the templates from `pmos/okf/core/templates/`); discovery: a spike brief; chore:
|
|
13
|
+
an `intent:` line; incident: fix first, record within 48h; content: a content brief. Anchor to
|
|
14
|
+
a KR in `pmos/planning/okrs/` — or, for maintenance work with no honest KR, to a declared
|
|
15
|
+
health budget (`parent_kr: health:<id>` from `pmos/planning/okrs/health-budgets.md`). Always
|
|
16
|
+
one or the other; never unanchored.
|
|
17
|
+
2. **Branch `initiative/<id>`, do the work.** For feature work, write your tests first where you
|
|
18
|
+
can — the CI you wired via `templates/product-ci-starter.yml` is your Quality Gate.
|
|
19
|
+
3. **Record as you go** (each command prints exactly what it appended):
|
|
20
|
+
|
|
21
|
+
```sh
|
|
22
|
+
pmos/scripts/pmos-log.sh run <run_id> <initiative_id> [human_corrections]
|
|
23
|
+
pmos/scripts/pmos-log.sh eval <run_id> <initiative_id> <score 0..1> <PASS|PASS-WITH-FINDINGS|FAIL> <rubric_path>
|
|
24
|
+
pmos/scripts/pmos-log.sh accept <run_id> <initiative_id> <pass|fail> "note"
|
|
25
|
+
pmos/scripts/pmos-log.sh outcome <initiative_id> <validated|refuted|inconclusive> "evidence"
|
|
26
|
+
pmos/scripts/pmos-log.sh friction "what fought you"
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
The helper rejects bad values with a clear message (wrong verdict/disposition, a
|
|
30
|
+
validated/refuted outcome with no evidence, malformed score) — the same rules the hosted
|
|
31
|
+
backend enforces.
|
|
32
|
+
4. **Merge = the Acceptance Gate.** The **accepting human (the PM)** merges and records it
|
|
33
|
+
(`accept … pass`) — if you built it and someone else accepts, they merge, not you. A solo
|
|
34
|
+
operator is their own PM and merges with that hat on. If
|
|
35
|
+
you declared an `outcome_metric` on the initiative, you're not done until you dispose it at
|
|
36
|
+
its horizon (`outcome …`).
|
|
37
|
+
5. **Weekly (15 min):** run `python3 pmos/scripts/metrics.py` and read the red first; skim
|
|
38
|
+
`pmos/state/discoveries.md`.
|
|
39
|
+
|
|
40
|
+
## Working with an agent sometimes?
|
|
41
|
+
|
|
42
|
+
Mixed teams are fine: agents follow `AGENTS.md`, you follow this page — both write the same state
|
|
43
|
+
through the same conventions, and `metrics.py` doesn't care who built.
|
package/payload/AGENTS.md
CHANGED
|
@@ -18,7 +18,7 @@ yourself wanting an MCP tool, use the state adapter below instead.
|
|
|
18
18
|
|
|
19
19
|
| Hosted tool (as skills name them) | File-mode equivalent |
|
|
20
20
|
|---|---|
|
|
21
|
-
| `create_initiative` | write `pmos/state/initiatives/<id>.md` (frontmatter: id, title, parent_kr, stage, status, prd_path, rubric_path, created) |
|
|
21
|
+
| `create_initiative` | write `pmos/state/initiatives/<id>.md` (frontmatter: id, title, parent_kr, stage, status, **type** — the D56 lane, default `feature` — plus `intent` for the chore lane, prd_path, rubric_path, created; optionally `outcome_metric` + `outcome_horizon` — the D57 declaration for post-ship success criteria) |
|
|
22
22
|
| `save_spec` | the PRD file itself at `pmos/planning/prd/<id>-prd.md` (set `prd_path` on the initiative file) |
|
|
23
23
|
| `get_spec` / `get_planning_artifact` | read `pmos/planning/…` |
|
|
24
24
|
| `get_skill` | read `pmos/skills/<name>.skill` |
|
|
@@ -27,6 +27,7 @@ yourself wanting an MCP tool, use the state adapter below instead.
|
|
|
27
27
|
| `log_agent_run` | append one JSON line to `pmos/state/runs.jsonl` (`{run_id, initiative_id, pre_run_contract, self_eval, human_corrections, date}`) |
|
|
28
28
|
| `save_eval` | append one JSON line to `pmos/state/evals.jsonl` (`{run_id, initiative_id, rubric_path, score, verdict, dimensions, date}`) |
|
|
29
29
|
| `record_acceptance` | append one JSON line to `pmos/state/acceptances.jsonl` (`{run_id, initiative_id, accepted, pm_label, note, date}`) — **written only on the PM's explicit word** |
|
|
30
|
+
| `record_outcome_check` | append one JSON line to `pmos/state/outcome_checks.jsonl` (`{initiative_id, disposition, evidence, date}`) — the D57 post-acceptance outcome verdict (`validated | refuted | inconclusive`; evidence required for the first two). **PM's word only — never auto-dispose from metrics**; a re-judgment is a new line |
|
|
30
31
|
| `save_okf_doc` | write the doc under `pmos/okf/product/…` marked as a DRAFT; the PM approves it via PR review (the file-mode form of draft→approved) |
|
|
31
32
|
| `register_product_repo` | not applicable — this repo IS the product; skip the step |
|
|
32
33
|
| `log_friction` / `log_discovery` | append to `pmos/state/discoveries.md` (ISO date, `#friction` tag where applicable) |
|
|
@@ -36,6 +37,26 @@ State files are **append-only**: never rewrite or delete an existing line or ini
|
|
|
36
37
|
correct by appending. When a vendored skill names a hosted tool, apply this table — the skill
|
|
37
38
|
bodies are single-sourced from the PMOS kernel and are not forked for file mode.
|
|
38
39
|
|
|
40
|
+
## Work-type lanes (D56)
|
|
41
|
+
|
|
42
|
+
Every initiative carries a `type` — the lane picks what the gate requires **before** work; the
|
|
43
|
+
kernel (anchor — a KR **or** `health:<id>` from `planning/okrs/health-budgets.md` (D59) —
|
|
44
|
+
eval-before-work in lane-appropriate form, human Acceptance, append-only state)
|
|
45
|
+
holds in every lane. Unknown/missing type = `feature`, the strictest lane. The enum is closed —
|
|
46
|
+
never invent a lane; a work shape with no lane escalates to the PM.
|
|
47
|
+
|
|
48
|
+
| type | spec artifact the gate requires | eval form |
|
|
49
|
+
|---|---|---|
|
|
50
|
+
| `feature` (default) | PRD + eval rubric before build | rubric + evaluator skill (unchanged) |
|
|
51
|
+
| `discovery` | spike brief at `prd_path` (question + timebox — `skills/discovery.skill`, template in `okf/core/templates/`) | disposition: learned / promoted / discarded; rubric optional |
|
|
52
|
+
| `chore` | a one-paragraph `intent:` in the initiative file | repo CI + PM spot-check at merge |
|
|
53
|
+
| `incident` | incident record at `prd_path`, **post-hoc within 48h** (template in `okf/core/templates/`) — the one sanctioned inversion of eval-first | retro in the record; follow-ups become their own initiatives |
|
|
54
|
+
| `content` | content brief at `prd_path` (audience, message, claims + sources — `skills/copywriting.skill`, template in `okf/core/templates/`) | copy rubric recommended, not required (D58); PM judges at the gate otherwise |
|
|
55
|
+
|
|
56
|
+
`no-initiative` label = anchor-free trivia ONLY (a typo, a broken link). A chore, spike,
|
|
57
|
+
incident, or piece of content is real work and takes its lane — that is what keeps the record
|
|
58
|
+
honest.
|
|
59
|
+
|
|
39
60
|
## Agent run protocol (every run, in order)
|
|
40
61
|
|
|
41
62
|
1. Read this file.
|
|
@@ -43,9 +64,14 @@ bodies are single-sourced from the PMOS kernel and are not forked for file mode.
|
|
|
43
64
|
No skill fits → escalate to the PM; do not improvise a procedure.
|
|
44
65
|
3. Retrieve context: `pmos/okf/` for concepts, `pmos/okf/product/` for this product's knowledge,
|
|
45
66
|
`pmos/state/runs.jsonl` + `discoveries.md` for what prior runs tried (episodic memory).
|
|
46
|
-
4. Anchor: the initiative file must exist at stage `intake`/`spec` with
|
|
47
|
-
**before** build (eval-first, D13
|
|
48
|
-
|
|
67
|
+
4. Anchor: the initiative file must exist at stage `intake`/`spec` with its **lane's** spec/eval
|
|
68
|
+
artifacts authored **before** build (eval-first, D13, generalized per lane — see the lane table
|
|
69
|
+
above; `feature` = PRD + rubric; the incident lane alone records post-hoc). Contract per lane:
|
|
70
|
+
`feature` = the rubric's sprint-contract section with the PM's explicit `PM_approved: true`;
|
|
71
|
+
`discovery` = the PM-approved spike brief (question + timebox); `chore` = the stated intent;
|
|
72
|
+
`content` = the PM-approved content brief. No work before the lane's approval. Feature-lane
|
|
73
|
+
`test_method`s SHOULD be executable checks written before build (D58) — wire
|
|
74
|
+
`templates/product-ci-starter.yml` so tier 1 is never silently empty.
|
|
49
75
|
5. Branch `initiative/<id>`; execute at control altitude; one commit per artifact.
|
|
50
76
|
6. Self-evaluate against the contract — self-eval is unreliable; it is an input to the gates, never
|
|
51
77
|
a verdict.
|
|
@@ -55,6 +81,28 @@ bodies are single-sourced from the PMOS kernel and are not forked for file mode.
|
|
|
55
81
|
= the human PM merges and disposes — final and non-delegable. Never merge your own PR.
|
|
56
82
|
8. Log the run (`runs.jsonl`) and any friction (`discoveries.md`, `#friction`).
|
|
57
83
|
|
|
84
|
+
## Human builders & multi-repo initiatives (D58)
|
|
85
|
+
|
|
86
|
+
- **A human building instead of an agent?** Same loop, plain terms: [AGENTS-HUMAN.md](AGENTS-HUMAN.md).
|
|
87
|
+
State is written with `scripts/pmos-log.sh` (validated, append-only) — never hand-edited. Mixed
|
|
88
|
+
human+agent teams write the same state through the same conventions.
|
|
89
|
+
- **An initiative spanning multiple repos?** ONE repo is its **home** — the initiative record,
|
|
90
|
+
brief/PRD, rubric, and state live there. Every sibling repo's PR anchors by declaring
|
|
91
|
+
`Initiative: <id>` in the PR body (the gate's body-key path); records land in the home state
|
|
92
|
+
layer. There is no cross-repo registry — the body key IS the join. **Sibling repos either do
|
|
93
|
+
not install the pmos gate, or mirror the initiative file locally** — a vendored sibling gate
|
|
94
|
+
fails closed on a missing local record by design (S5 Review Gate finding).
|
|
95
|
+
|
|
96
|
+
## New agent paths — the lethal-trifecta audit
|
|
97
|
+
|
|
98
|
+
Before creating any new agent path or widening a tool grant, verify it does **not** hold all three
|
|
99
|
+
of: **private data access + untrusted-content exposure + external communication**. An agent holding
|
|
100
|
+
all three can be tricked by content it reads into exfiltrating what it knows — remove a leg
|
|
101
|
+
(preferably external communication) structurally; prompt-level defenses alone are bypassed by
|
|
102
|
+
adaptive attacks. If a path genuinely needs all three, its PRD/brief must name the compensating
|
|
103
|
+
gate; silence is not a pass. The evaluator is the standing example: it reads adversarial diffs, so
|
|
104
|
+
it stays read-only with its verdict append as the only write.
|
|
105
|
+
|
|
58
106
|
## Escalation (stop and ask the PM)
|
|
59
107
|
|
|
60
108
|
An architectural choice with no recorded decision; a wrong or unsatisfiable contract; scope growth;
|
|
@@ -73,7 +121,34 @@ and the Goodhart tripwire from `pmos/state/`.
|
|
|
73
121
|
|
|
74
122
|
Scan `discoveries.md`; a friction theme seen twice with no resolving decision must be promoted to a
|
|
75
123
|
decision. Read run records since last review; log overrides to `eval-calibration.md`. Run
|
|
76
|
-
`pmos/scripts/metrics.py` and read the red first.
|
|
124
|
+
`pmos/scripts/metrics.py` and read the red first. **Scan the lane mix (D56):** count initiatives
|
|
125
|
+
per `type` — a chore share climbing review over review, discovery spikes that never close their
|
|
126
|
+
disposition, or `no-initiative` PRs that should have been chores are drift signals; a lane misused
|
|
127
|
+
twice is promoted to a decision like any recurring friction. **Scan the anchor mix (D59):** feature-vs-health share of the period's initiatives; a budget over
|
|
128
|
+
its cap or health-shaped work with a KR anchor is a drift signal; quarterly, review the caps.
|
|
129
|
+
**Scan outcome checks (D57):**
|
|
130
|
+
metrics.py lists due outcomes (accepted, past horizon, undisposed) — dispose each
|
|
131
|
+
(`validated | refuted | inconclusive`, evidence for the first two) or consciously defer; a check
|
|
132
|
+
deferred twice, or a horizon moved twice, is promoted like recurring friction. An accepted
|
|
133
|
+
initiative with a declared outcome is NOT finished until disposed. **Read the run's explainer where
|
|
134
|
+
one exists** — the `explain-run` skill writes an independently-authored literate explainer + quiz to
|
|
135
|
+
`pmos/planning/evals/explainers/<run_id>-explainer.md`; start there instead of the raw transcript,
|
|
136
|
+
and for a high-risk run pass its quiz as a comprehension self-check before you accept. Recommended
|
|
137
|
+
discipline, not an enforced gate — accepting runs you cannot explain is the PM's own watermelon.
|
|
138
|
+
|
|
139
|
+
**Anti-atrophy forcing functions (complacency is structural, not a willpower problem):**
|
|
140
|
+
- **Name one hand-verified fact at every Acceptance** — the `record_acceptance` note states something
|
|
141
|
+
*you* checked directly (ran it, read the hunk, reproduced it), not the evaluator's verdict restated.
|
|
142
|
+
- **Watch the acceptance-friction meta-watermelon** — a run of zero-comment, fast acceptances is
|
|
143
|
+
itself a flag; mandated oversight decays into rubber-stamping that legitimizes the system.
|
|
144
|
+
- **Guard the four satisficing heuristics**: plan-as-proxy, tests-as-guarantee (CI-green ≠ correct —
|
|
145
|
+
the Quality Gate is not the Acceptance Gate), eyeballing-suffices, deference-in-unfamiliar-domains.
|
|
146
|
+
- **Adaptive autonomy** — drop an initiative to every-diff review on a failed eval, a reverted merge,
|
|
147
|
+
a gate-touching change, or a subsystem you haven't personally touched in ~3 months.
|
|
148
|
+
- **Monthly**, do one hands-on rep — personally diagnose one defect or re-derive one diff from its
|
|
149
|
+
spec before reading the agent's version (diagnosis-shaped; comprehension decays faster than typing).
|
|
150
|
+
**Quarterly**, ask *"what would I no longer know how to fix by hand?"* — supervision cost rises as
|
|
151
|
+
runs improve.
|
|
77
152
|
|
|
78
153
|
## Kit maintenance
|
|
79
154
|
|
package/payload/KIT-VERSION
CHANGED
|
@@ -1,3 +1,3 @@
|
|
|
1
|
-
kernel-source: pmos@
|
|
2
|
-
assembled: 2026-07-
|
|
1
|
+
kernel-source: pmos@25d32ad
|
|
2
|
+
assembled: 2026-07-20
|
|
3
3
|
mode: vendor
|
package/payload/README.md
CHANGED
|
@@ -10,6 +10,9 @@ personally-owned PMOS backend).
|
|
|
10
10
|
|
|
11
11
|
- `AGENTS.md` — the run protocol. **Start here; every agent session reads it first.** Contains the
|
|
12
12
|
state adapter: the table mapping each hosted PMOS tool to a file convention.
|
|
13
|
+
- `AGENTS-HUMAN.md` — the same loop for a **human** builder (D58): plain terms + the
|
|
14
|
+
`scripts/pmos-log.sh` helper that appends validated state records in one command. Mixed
|
|
15
|
+
human+agent teams share the same state.
|
|
13
16
|
- `skills/` — the PMOS skill kernel (prd, eval-rubric, okr, evaluator, …). Single-sourced from the
|
|
14
17
|
PMOS repo; where a skill names a hosted tool, apply the AGENTS.md state adapter.
|
|
15
18
|
- `okf/core/` — kernel concepts (the sdlc loop, the three-tier gate, watermelon flags, …).
|
|
@@ -47,11 +50,17 @@ anchored; your local `git merge` is the Acceptance Gate. If the repo later lands
|
|
|
47
50
|
|
|
48
51
|
## Operate (every piece of work)
|
|
49
52
|
|
|
50
|
-
1. Create `pmos/state/initiatives/<id>.md` (stage `intake`, anchored to a KR in `planning/okrs/`
|
|
51
|
-
|
|
52
|
-
|
|
53
|
+
1. Create `pmos/state/initiatives/<id>.md` (stage `intake`, anchored to a KR in `planning/okrs/` — or to a health budget,
|
|
54
|
+
`parent_kr: health:<id>` from `planning/okrs/health-budgets.md` (D59)),
|
|
55
|
+
with its **work-type lane**: `type: feature` (default) `| discovery | chore | incident | content`
|
|
56
|
+
(D56/D58 — the lane table in `AGENTS.md` says what each requires).
|
|
57
|
+
2. Author the lane's spec artifacts **before building** — feature: PRD + rubric (copy the
|
|
58
|
+
templates), sprint contract, PM flips `PM_approved: true`; discovery: a spike brief (question +
|
|
59
|
+
timebox) the PM approves; chore: a one-paragraph `intent:`; incident: fix first, record within
|
|
60
|
+
48h.
|
|
53
61
|
3. Branch `initiative/<id>`, build, open a PR — the `pmos-gate` CI check fails any PR that isn't
|
|
54
|
-
anchored (branch or `Initiative: <id>` in the PR body; `no-initiative` label for
|
|
62
|
+
anchored (branch or `Initiative: <id>` in the PR body; `no-initiative` label for anchor-free
|
|
63
|
+
trivia ONLY — a chore, spike, or incident takes its lane, D56).
|
|
55
64
|
4. Review Gate: run the evaluator skill against the rubric; append the verdict to
|
|
56
65
|
`state/evals.jsonl`. The verdict is advisory until locally calibrated.
|
|
57
66
|
5. Acceptance Gate: the **human PM** merges and disposes; append to `state/acceptances.jsonl`
|
|
@@ -69,6 +78,11 @@ Talk to your agent like a colleague; your task is direction and judgment, its ta
|
|
|
69
78
|
initiative — fast first win, not completeness."
|
|
70
79
|
- **Start a piece of work:** "Read pmos/AGENTS.md. I want: [outcome]. Create the initiative file,
|
|
71
80
|
then draft the PRD and eval rubric for my approval before you build anything."
|
|
81
|
+
- **Explore before committing (discovery lane):** "Read pmos/AGENTS.md. I don't know yet whether
|
|
82
|
+
[idea] is worth building. Open a discovery initiative per pmos/skills/discovery.skill: draft the
|
|
83
|
+
spike brief — one question, a timebox — for my approval. Prototype code is throwaway."
|
|
84
|
+
- **Quick maintenance (chore lane):** "Read pmos/AGENTS.md. Open a chore initiative: [one-sentence
|
|
85
|
+
intent]. No PRD or rubric — just state the intent in the initiative file and do it."
|
|
72
86
|
- **Review Gate:** "Follow pmos/skills/evaluator.skill: score the work on branch initiative/[id]
|
|
73
87
|
against its rubric. Be skeptical — try to refute it. Append the verdict to pmos/state/evals.jsonl."
|
|
74
88
|
- **Acceptance:** "I accept this run. Append the acceptance to pmos/state/acceptances.jsonl with
|
|
@@ -92,6 +106,11 @@ PMOS checkout) — refreshes ONLY the kernel (skills/okf-core/scripts/AGENTS.md/
|
|
|
92
106
|
stamps `KIT-VERSION`, never touches `state/`, `planning/`, or `okf/product/`. Updates flow
|
|
93
107
|
one-way, PMOS → kit; review the diff as a normal PR.
|
|
94
108
|
|
|
109
|
+
> Upgrade note (D59 kernel): the gate now rejects an initiative file with **no `parent_kr`**
|
|
110
|
+
> (hosted parity, D21). If a pre-D59 vendored file trips it, that file was always unanchored —
|
|
111
|
+
> add a real KR, or `parent_kr: health:<id>` from `planning/okrs/health-budgets.md` if it is
|
|
112
|
+
> honest maintenance. The gate failing is the fix working, not a regression.
|
|
113
|
+
|
|
95
114
|
## The containment guarantee
|
|
96
115
|
|
|
97
116
|
The assembly script greps its own output and fails on any reference to the hosted PMOS backend.
|
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Concept
|
|
3
|
+
title: Health Anchors
|
|
4
|
+
description: The second anchor class — work anchors to a KR or a declared health budget (a standing, capped, PM-reviewed allocation), so refactor/ops/security work anchors honestly instead of faking a KR or escaping the record.
|
|
5
|
+
tags: [anchoring, health, anti-optimism, budgets]
|
|
6
|
+
timestamp: 2026-07-18
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Health Anchors
|
|
10
|
+
|
|
11
|
+
**Situating context:** Added with [D59](/okf/products/pmos/adr/d59-health-anchors.md) out of the
|
|
12
|
+
SDLC stress test: KR-only anchoring forced legitimate KR-orthogonal work (refactors, ops/SLO
|
|
13
|
+
upkeep, security hardening) into fake anchors or the escape hatch — Goodhart on the anchoring rule
|
|
14
|
+
itself. This concept defines the second anchor class the [OKR tree](/okf/core/concepts/okr-tree.md)
|
|
15
|
+
rule now admits.
|
|
16
|
+
|
|
17
|
+
## The rule
|
|
18
|
+
|
|
19
|
+
Every initiative anchors ([D21]) — but to one of **two classes**:
|
|
20
|
+
|
|
21
|
+
- **A KR** — an outcome bet: the initiative moves a measurable result.
|
|
22
|
+
- **A health budget** — `health:<id>` from the closed registry
|
|
23
|
+
[`planning/okrs/health-budgets.md`](/planning/okrs/health-budgets.md): a standing, capped
|
|
24
|
+
allocation for work that keeps the system worth building on (tech health, security, ops).
|
|
25
|
+
|
|
26
|
+
The distinction is honest bookkeeping: **health is a budget, not an outcome bet.** Dressing a
|
|
27
|
+
refactor as a KR corrupts both — the KR's metric and the refactor's justification.
|
|
28
|
+
|
|
29
|
+
## Governance
|
|
30
|
+
|
|
31
|
+
The registry is closed (PM-edited, visible diffs). Caps are shares of a quarter's initiatives.
|
|
32
|
+
The weekly scan reads the **anchor mix** (feature vs health) beside the lane mix — a climbing
|
|
33
|
+
health share, a budget over cap, or health-shaped work still arriving with KR anchors are drift
|
|
34
|
+
signals. The quarterly cadence reviews caps and **promotes** budget work to a real KR when it
|
|
35
|
+
stops being upkeep and becomes a bet.
|
|
36
|
+
|
|
37
|
+
## Relationship to lanes
|
|
38
|
+
|
|
39
|
+
Orthogonal axes of one kernel rule: the [lane](/okf/products/pmos/adr/d56-initiative-lanes.md)
|
|
40
|
+
fixes WHAT the gate requires; the anchor fixes WHY the work exists. A `chore`-lane refactor
|
|
41
|
+
anchored `health:tech` is the canonical honest pairing.
|
|
@@ -7,6 +7,7 @@ to pick concepts, then fetch the file(s) you need.
|
|
|
7
7
|
- [control-plane](control-plane.md) — What it means for PMOS to be a control plane — it orchestrates, specs, evaluates, and routes agent work rather than generating production code itself.
|
|
8
8
|
- [design-loop](design-loop.md) — How PMOS makes UI/UX design a first-class, repeatable sub-flow of the Build stage — brief → design → reconcile → build → design-sync — gated by the Review Gate, presentation-only, and never a data decision.
|
|
9
9
|
- [eval-driven-pm](eval-driven-pm.md) — PMOS's core method — write the eval before the build. The rubric is the spec; the PM manages by defining what "good" means, not by inspecting work after the fact.
|
|
10
|
+
- [health-anchors](health-anchors.md) — The second anchor class (D59): work anchors to a KR or a declared, capped, PM-reviewed health budget, so refactor/ops/security work anchors honestly.
|
|
10
11
|
- [okf-governance](okf-governance.md) — How PMOS organizes its Open Knowledge Format bundle and the governance rules that keep it spec-conformant.
|
|
11
12
|
- [okr-anti-optimism](okr-anti-optimism.md) — Why PMOS biases status reporting toward pessimism — surfacing the red underneath green metrics — to keep KR self-reporting honest.
|
|
12
13
|
- [okr-tree](okr-tree.md) — How PMOS structures Objectives and Key Results as a tree that links strategy to initiatives and to the agent runs that move them.
|
|
@@ -52,3 +52,10 @@ The goal is *calibration*, not gloom. Over-reporting red is also a failure — i
|
|
|
52
52
|
ignore reports. The discipline is: report the honest reading, weight uncertainty toward the
|
|
53
53
|
downside, and always separate "where we are" from "how confident we are." A well-calibrated amber is
|
|
54
54
|
worth more than a confident green or a reflexive red.
|
|
55
|
+
|
|
56
|
+
## Amendment — the anchor mix (D59, 2026-07-18)
|
|
57
|
+
|
|
58
|
+
Anti-optimism reporting gains the **anchor mix**: the feature-vs-health share of a period's
|
|
59
|
+
initiatives, read beside the lane mix. Drift signals: a health budget over its cap, a climbing
|
|
60
|
+
health share crowding out outcome bets, or health-shaped work still arriving with KR anchors
|
|
61
|
+
(fake-anchor recidivism). See [health-anchors](/okf/core/concepts/health-anchors.md).
|
|
@@ -65,3 +65,10 @@ Individual KRs should ladder toward it, directly or indirectly.
|
|
|
65
65
|
- KRs are reviewed on the PM's regular cadence; objectives are reset each period.
|
|
66
66
|
- The OKR file is an **operational instance** — it lives under `/planning/`, not in OKF. This
|
|
67
67
|
concept is the timeless *structure*; the quarterly file is the dated *content*.
|
|
68
|
+
|
|
69
|
+
## Amendment — the second anchor class (D59, 2026-07-18)
|
|
70
|
+
|
|
71
|
+
Initiative anchoring generalizes: an initiative anchors to a KR **or** to a declared
|
|
72
|
+
[health budget](/okf/core/concepts/health-anchors.md) (`health:<id>` from the closed registry in
|
|
73
|
+
`/planning/okrs/health-budgets.md`). Health work is a budget, not an outcome bet — forcing it
|
|
74
|
+
under a KR corrupts both. The tree's KR discipline is unchanged for outcome work.
|
|
@@ -39,7 +39,8 @@ nature and a different owner.
|
|
|
39
39
|
|
|
40
40
|
- **Nature:** subjective, strategic.
|
|
41
41
|
- **Checks:** strategic fit. Is this what the product actually needs right now? Does it move the
|
|
42
|
-
parent KR
|
|
42
|
+
parent KR — or, for health-anchored work, honestly serve its declared budget (D59)? **This is
|
|
43
|
+
NOT a re-check of the rubric.**
|
|
43
44
|
- **Owner:** the **human PM only**. Non-delegable.
|
|
44
45
|
- The PM's role at the end of every agent run **is** the Acceptance Gate.
|
|
45
46
|
|
|
@@ -27,7 +27,8 @@ PMOS drives a closed lifecycle. Each turn of the loop moves one initiative throu
|
|
|
27
27
|
```
|
|
28
28
|
|
|
29
29
|
1. **Spec** — the initiative gets a PRD via the [prd skill](/skills/core/prd.skill). The PRD says
|
|
30
|
-
what to build and why, anchored to a KR in the [OKR tree](/okf/core/concepts/okr-tree.md)
|
|
30
|
+
what to build and why, anchored to a KR in the [OKR tree](/okf/core/concepts/okr-tree.md) —
|
|
31
|
+
or, for maintenance work, a declared [health budget](/okf/core/concepts/health-anchors.md) (D59).
|
|
31
32
|
2. **Contract** — generator and evaluator agree a **pre-run contract**: testable done-criteria,
|
|
32
33
|
before any work starts. The PM approves it. (P0: temp file + PM approval; P1+:
|
|
33
34
|
`agent_runs.pre_run_contract`.)
|
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
---
|
|
2
|
+
type: Template
|
|
3
|
+
title: Content Brief Template
|
|
4
|
+
description: The scaffold for a content-lane brief (D58) — audience, message, voice, deliverables, claim sources, and an optional weighted copy rubric. Copy into /planning/prd/ as <id>-content-brief.md and set it as the initiative's prd_path.
|
|
5
|
+
tags: [template, content, copywriting, lanes]
|
|
6
|
+
timestamp: 2026-07-18
|
|
7
|
+
---
|
|
8
|
+
|
|
9
|
+
# Content Brief Template
|
|
10
|
+
|
|
11
|
+
**Situating context:** Authored with [D58](/okf/products/pmos/adr/d58-builder-profiles.md) as the
|
|
12
|
+
content lane's spec artifact — the contract copy work is drafted and judged against. The
|
|
13
|
+
[copywriting skill](/skills/domains/copywriting.skill) instantiates it. A rubric is recommended,
|
|
14
|
+
not required (PM call at D58 ratification); the optional section below is the recommended shape.
|
|
15
|
+
|
|
16
|
+
> Usage: replace every `<…>` placeholder. An empty audience or an unsourced claim list means the
|
|
17
|
+
> brief is not ready. PM approval of this brief is the lane's pre-work contract.
|
|
18
|
+
|
|
19
|
+
---
|
|
20
|
+
|
|
21
|
+
```markdown
|
|
22
|
+
# Content Brief: <initiative title>
|
|
23
|
+
|
|
24
|
+
## Identity
|
|
25
|
+
- initiative_id: <id>
|
|
26
|
+
- type: content
|
|
27
|
+
- product_slug: <product_slug>
|
|
28
|
+
- pm_slug: <pm_slug>
|
|
29
|
+
- parent_KR: <link or health:<id> (D59)> — *"<KR text quoted>"; <why this copy advances it>* <!-- REQUIRED -->
|
|
30
|
+
- status: draft | approved
|
|
31
|
+
- date: <YYYY-MM-DD>
|
|
32
|
+
|
|
33
|
+
## Audience & placement
|
|
34
|
+
<!-- WHO reads this, WHERE it appears (page/screen/email), and what they know walking in. One
|
|
35
|
+
line per audience/placement pair; each pair gets its own variant in the draft. -->
|
|
36
|
+
- <audience> @ <placement>
|
|
37
|
+
|
|
38
|
+
## Message
|
|
39
|
+
<!-- The ONE thing each audience should take away, and the action it should prompt. -->
|
|
40
|
+
|
|
41
|
+
## Voice & constraints
|
|
42
|
+
<!-- Tone (e.g. plain, technical, playful), banned phrases, length limits, required terms,
|
|
43
|
+
localization notes. Link the product voice guide if one exists in OKF. -->
|
|
44
|
+
|
|
45
|
+
## Claims & sources
|
|
46
|
+
<!-- EVERY factual claim the copy will make, each with its source. Unsourced claims don't ship —
|
|
47
|
+
wrong public claims are incidents, not nits. -->
|
|
48
|
+
- <claim> — <source>
|
|
49
|
+
|
|
50
|
+
## Deliverables
|
|
51
|
+
- <artifact + path, e.g. web/landing hero copy, README intro>
|
|
52
|
+
|
|
53
|
+
## Copy rubric (RECOMMENDED, optional — delete if the PM judges at the gate instead)
|
|
54
|
+
| Dimension | Weight |
|
|
55
|
+
|---|---:|
|
|
56
|
+
| Audience-fit (right register + assumed knowledge per variant) | 0.30 |
|
|
57
|
+
| Clarity (message lands in one read; no filler) | 0.30 |
|
|
58
|
+
| Voice (constraints above followed, or deviation argued) | 0.15 |
|
|
59
|
+
| Claim accuracy (every claim matches its source) | 0.25 |
|
|
60
|
+
| **Total** | **1.00** |
|
|
61
|
+
pass_threshold: 0.85
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
---
|
|
65
|
+
|
|
66
|
+
## Authoring checklist
|
|
67
|
+
|
|
68
|
+
- [ ] `parent_KR` present with quoted text — content is anchored work, never the hatch.
|
|
69
|
+
- [ ] Every claim has a source; every audience/placement pair has a planned variant.
|
|
70
|
+
- [ ] PM approved the brief before drafting began.
|
|
@@ -28,7 +28,7 @@ instance and does not live in the OKF bundle.
|
|
|
28
28
|
- initiative_id: <id>
|
|
29
29
|
- product_slug: <product_slug>
|
|
30
30
|
- pm_slug: <pm_slug>
|
|
31
|
-
- parent_KR: <objective.kr reference>
|
|
31
|
+
- parent_KR: <objective.kr reference OR health:<id> (D59)>
|
|
32
32
|
- date: <YYYY-MM-DD>
|
|
33
33
|
|
|
34
34
|
## Point of view (so the design has one)
|
|
@@ -28,7 +28,7 @@ OKF bundle.
|
|
|
28
28
|
- initiative_id: <id>
|
|
29
29
|
- product_slug: <product_slug>
|
|
30
30
|
- pm_slug: <pm_slug>
|
|
31
|
-
- parent_KR: <reference>
|
|
31
|
+
- parent_KR: <reference OR health:<id> (D59)>
|
|
32
32
|
- pass_threshold: <0.00–1.00> <!-- pre-committed, before output exists -->
|
|
33
33
|
- date: <YYYY-MM-DD>
|
|
34
34
|
|
|
@@ -79,11 +79,28 @@ Rules:
|
|
|
79
79
|
- **Weights sum to 1.0.** If they do not, the rubric is malformed.
|
|
80
80
|
- **Partial credit is mandatory.** A dimension that can only score 0 or 1 hides quality gradients and
|
|
81
81
|
defeats weighted scoring — split it or define intermediate anchors.
|
|
82
|
-
- **Anchor the scale.** For each dimension,
|
|
83
|
-
is repeatable
|
|
84
|
-
[eval-calibration.md](/planning/evals/eval-calibration.md)
|
|
82
|
+
- **Anchor the scale — richer anchors lift agreement.** For each dimension, define what the
|
|
83
|
+
score levels look like so scoring is repeatable and calibratable against
|
|
84
|
+
[eval-calibration.md](/planning/evals/eval-calibration.md). The **minimum** is `0 / 0.5 / 1`;
|
|
85
|
+
**prefer a 5-level anchored scale** (`0 / 0.25 / 0.5 / 0.75 / 1`, or a 1–5 scale mapped onto it) with
|
|
86
|
+
a concrete one-line descriptor per level. Anchored multi-level scales are the single **largest
|
|
87
|
+
evidenced lever on judge agreement** — Cohen's κ rises materially versus a bare numeric or a
|
|
88
|
+
three-point scale ([llm-judge-calibration](/planning/research/llm-judge-calibration.md)): the more
|
|
89
|
+
precisely each level is described, the less two evaluators (or the evaluator vs the PM) diverge.
|
|
90
|
+
Spend the extra levels on the dimensions where the quality gradient actually matters, not every one.
|
|
85
91
|
- **Threshold is pre-committed.** Never tune the threshold after seeing the output.
|
|
92
|
+
- **Evidence per dimension.** A Review-Gate verdict must carry, for each scored dimension, a short
|
|
93
|
+
verbatim quote or precise pointer (file+line, query output, CI line) from the evaluated artifact;
|
|
94
|
+
a dimension without evidence is marked `unverified` and scored conservatively (≤ 0.5). Rubric
|
|
95
|
+
authors should write dimensions so that evidence is *citable* — observable outcomes, not
|
|
96
|
+
impressions ([llm-judge-calibration](/planning/research/llm-judge-calibration.md)).
|
|
86
97
|
- **Outcome-only.** Score the artifact produced, not the tool-call sequence or reasoning path.
|
|
98
|
+
- **Coherence criteria are semantic.** A "rule holds everywhere" criterion must enumerate its
|
|
99
|
+
live-doc set — every living mirror of a live surface (skills, templates, playbooks, concepts,
|
|
100
|
+
protocol docs, tool schemas + their reference/data-model mirrors, UI copy, intake instruments) —
|
|
101
|
+
and be tested by a semantic pass (reading for meaning); a grep sweep is evidence,
|
|
102
|
+
never the definition of done (health-anchors lesson —
|
|
103
|
+
[eval-calibration](/planning/evals/eval-calibration.md)).
|
|
87
104
|
|
|
88
105
|
## Authoring checklist
|
|
89
106
|
|
|
@@ -92,4 +109,7 @@ Rules:
|
|
|
92
109
|
- [ ] Each dimension supports partial credit with anchored 0/0.5/1.
|
|
93
110
|
- [ ] `pass_threshold` pre-committed.
|
|
94
111
|
- [ ] Gate assignment stated; Acceptance Gate marked PM-only, not graded.
|
|
112
|
+
- [ ] Dimensions written so a verdict can cite verbatim evidence per dimension (observable
|
|
113
|
+
outcomes, not impressions).
|
|
114
|
+
- [ ] Any rule-coherence criterion enumerates its live-doc set and tests by semantic pass.
|
|
95
115
|
- [ ] Sprint contract present with `PM_approved`.
|