create-pmos 0.27.0 → 0.29.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +26 -0
- package/package.json +1 -1
- package/payload/KIT-VERSION +2 -2
- package/payload/scripts/pmos-log.sh +29 -2
- package/payload/scripts/workflow-state.js +40 -6
- package/payload/skills/evaluator.skill +45 -12
- package/payload/skills/pmos-router.skill +1 -1
- package/payload/skills-native/evaluator/SKILL.md +45 -12
- package/payload/skills-native/pmos-router/SKILL.md +1 -1
- package/workspace/README.md +22 -0
- package/workspace/lib/drive-guard.js +421 -30
- package/workspace/lib/engine/workflow-state.js +40 -6
- package/workspace/package-lock.json +2 -2
- package/workspace/package.json +1 -1
- package/workspace/public/pm/data.js +17 -0
- package/workspace/public/pm/panel-initiatives.js +161 -3
- package/workspace/server.js +246 -6
package/README.md
CHANGED
|
@@ -333,6 +333,32 @@ Refreshes the skills, concepts, and scripts only. Your records — `state/`, `pl
|
|
|
333
333
|
`okf/product/` — are yours, and an update never touches them. `pmos/KIT-VERSION` always says
|
|
334
334
|
exactly what you're running.
|
|
335
335
|
|
|
336
|
+
## What's new in 0.28.0
|
|
337
|
+
|
|
338
|
+
**`search everything…` now searches every file in the folder, and any file it finds opens in a tab.**
|
|
339
|
+
Run `npx create-pmos@latest workspace`; the version bump means an existing install auto-upgrades.
|
|
340
|
+
|
|
341
|
+
- **The box searched three lists, and the folder was not one of them.** It matched initiative slugs,
|
|
342
|
+
paths *declared on* an initiative record, and session names — so a file PMOS had never been told
|
|
343
|
+
about could not be found by typing its exact name. Worse, the miss read *"nothing matches — a
|
|
344
|
+
successful read of nothing"*, which is the phrase for *we looked and there was nothing there*,
|
|
345
|
+
printed after a search that never opened the folder.
|
|
346
|
+
- **Type any part of a name or a path.** Every file under the workspace root now matches, at any
|
|
347
|
+
depth, whether or not any record mentions it — `README.md`, a script, a CSV you dropped in
|
|
348
|
+
yesterday. `node_modules/`, `dist/`, `.git/` and the other junk directories are skipped, exactly as
|
|
349
|
+
the folder lens already skips them, so search and the file tree agree about what is in there.
|
|
350
|
+
- **Type a phrase that is *inside* a file.** Text files are searched by content too, and the result
|
|
351
|
+
names the file **and the line**, so you can see the match before you open anything.
|
|
352
|
+
- **Click a result and it opens in a document tab** — the same reader you already use, with `view` ·
|
|
353
|
+
`raw` · `edit` · `diff`. A file the reader cannot show (an image, a binary) is still listed, greyed,
|
|
354
|
+
and says so, instead of opening a tab onto a failure.
|
|
355
|
+
- **It works in every mode.** The search box was rendered in all seven and did something in exactly
|
|
356
|
+
one — including doing nothing in the library lens, the one screen that lists files.
|
|
357
|
+
- **It tells you what it did not look at.** The walk is bounded so a huge folder can never hang the
|
|
358
|
+
box, and any bound that actually bit is stated in the heading. `11 of 1424 searched` is a
|
|
359
|
+
different sentence from `BOUNDED — files beyond the walk limit were not looked at`, and you get
|
|
360
|
+
whichever one is true. Nothing leaves your machine, as ever.
|
|
361
|
+
|
|
336
362
|
## What's new in 0.26.0
|
|
337
363
|
|
|
338
364
|
**Your sessions now know which initiative they belong to — whatever branch you are on — and each
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "create-pmos",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.29.0",
|
|
4
4
|
"description": "Scaffold the PMOS portable kit \u2014 run the PMOS product-management workflow (initiative \u2192 lane-fit spec \u2192 contract \u2192 build \u2192 three gates, with work-type lanes for features, research, maintenance, incidents, and content) entirely inside your own repo, with all state as git-versioned files. Includes `workflow-sweep`, which audits the state every board derives from and separates real defects from work you simply have not done. `create-pmos workspace` also installs + launches the local Terminal Workspace GUI. No database, no network, no telemetry.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"bin": {
|
package/payload/KIT-VERSION
CHANGED
|
@@ -1,3 +1,3 @@
|
|
|
1
|
-
kernel-source: pmos@
|
|
2
|
-
assembled: 2026-08-
|
|
1
|
+
kernel-source: pmos@4104fde
|
|
2
|
+
assembled: 2026-08-04
|
|
3
3
|
mode: vendor
|
|
@@ -45,10 +45,37 @@ case "$cmd" in
|
|
|
45
45
|
append runs.jsonl "{\"run_id\":$(js "$1"),\"initiative_id\":$(js "$2"),\"human_corrections\":$corr,\"date\":\"$today\"$model_field}"
|
|
46
46
|
;;
|
|
47
47
|
eval)
|
|
48
|
-
|
|
48
|
+
# SKIPPED is a verdict this writer must be able to record.
|
|
49
|
+
#
|
|
50
|
+
# The enum here was PASS|PASS-WITH-FINDINGS|FAIL, while AGENTS.md tells every adopter: "a gate
|
|
51
|
+
# that was skipped is recorded as skipped — never leave a missing eval to be read as a pass."
|
|
52
|
+
# An adopter following that instruction hit `fail` and had no way to comply except by hand-
|
|
53
|
+
# appending, which is the drift this writer exists to prevent. PMOS-self's own ledger carries 16
|
|
54
|
+
# SKIPPED rows; the kit could not have written one of them.
|
|
55
|
+
#
|
|
56
|
+
# A skip is NOT a soft pass: it takes no score (a skip has no score to give), it requires a
|
|
57
|
+
# stated reason — an unexplained skip is indistinguishable from a forgotten one — and it leaves
|
|
58
|
+
# the verification the run owes still owing.
|
|
59
|
+
if [ "${4:-}" = "SKIPPED" ] || [ "${3:-}" = "SKIPPED" ]; then
|
|
60
|
+
# tolerated both argument positions: `eval <run> <init> SKIPPED <reason>` is the natural form
|
|
61
|
+
# when there is no score to give, and refusing it on argument order would just push people
|
|
62
|
+
# back to hand-appending.
|
|
63
|
+
if [ "${3:-}" = "SKIPPED" ]; then skip_reason="${4:-}"; skip_rubric="${5:-}"; skip_model="${6:-}"
|
|
64
|
+
else skip_reason="${5:-}"; skip_rubric="${6:-}"; skip_model="${7:-}"; fi
|
|
65
|
+
nonempty run_id "$1"; nonempty initiative_id "$2"
|
|
66
|
+
[ -n "$skip_reason" ] || fail "SKIPPED needs a reason: eval <run_id> <initiative_id> SKIPPED <reason> [rubric_path] [evaluator_model]"
|
|
67
|
+
rubric_field=""
|
|
68
|
+
if [ -n "$skip_rubric" ]; then rubric_field=",\"rubric_path\":$(js "$skip_rubric")"; fi
|
|
69
|
+
emodel_field=""
|
|
70
|
+
if [ -n "$skip_model" ]; then emodel_field=",\"evaluator_model\":$(js "$skip_model")"; fi
|
|
71
|
+
append evals.jsonl "{\"run_id\":$(js "$1"),\"initiative_id\":$(js "$2"),\"score\":null,\"verdict\":\"SKIPPED\",\"skipped_reason\":$(js "$skip_reason")$rubric_field,\"date\":\"$today\"$emodel_field}"
|
|
72
|
+
exit 0
|
|
73
|
+
fi
|
|
74
|
+
[ $# -ge 5 ] || fail "usage: eval <run_id> <initiative_id> <score 0..1> <PASS|PASS-WITH-FINDINGS|FAIL> <rubric_path> [evaluator_model]
|
|
75
|
+
or: eval <run_id> <initiative_id> SKIPPED <reason> [rubric_path] [evaluator_model]"
|
|
49
76
|
nonempty run_id "$1"; nonempty initiative_id "$2"
|
|
50
77
|
printf '%s' "$3" | grep -qE '^(0?\.[0-9]+|0|1(\.0+)?)$' || fail "score must be 0..1 written as 0, 1, N.N or .N — got '$3'."
|
|
51
|
-
case "$4" in PASS|PASS-WITH-FINDINGS|FAIL) ;; *) fail "verdict must be PASS | PASS-WITH-FINDINGS | FAIL, got '$4'." ;; esac
|
|
78
|
+
case "$4" in PASS|PASS-WITH-FINDINGS|FAIL) ;; *) fail "verdict must be PASS | PASS-WITH-FINDINGS | FAIL | SKIPPED, got '$4'." ;; esac
|
|
52
79
|
score="$3"; case "$score" in .*) score="0$score" ;; esac # ".5" -> "0.5": bare .5 is not valid JSON
|
|
53
80
|
[ -f "$KIT/$5" ] || [ -f "$5" ] || fail "rubric_path '$5' not found (relative to the kit or the repo)."
|
|
54
81
|
# optional evaluator-model provenance (which model rendered the verdict). Calibration (κ) is a
|
|
@@ -859,14 +859,48 @@ function computeWorkflow(id, opts) {
|
|
|
859
859
|
// ── review ─ the Review Gate's verdict, where the lane has a rubric ──────────────────────────
|
|
860
860
|
const evals = readJsonl(path.join(L.runsDir, 'evals.jsonl')).filter((e) => e && e.initiative_id === id);
|
|
861
861
|
{
|
|
862
|
-
|
|
863
|
-
|
|
864
|
-
|
|
865
|
-
|
|
862
|
+
/* The rung reads the RUBRIC, not the lane.
|
|
863
|
+
*
|
|
864
|
+
* It keyed on `lane` alone, so a chore that deliberately carries a rubric could never surface
|
|
865
|
+
* its Review Gate — the ladder said "chore lane carries no rubric (D56)" while a SKIPPED verdict
|
|
866
|
+
* sat in evals.jsonl unread. That is how `guard-and-ledger-hardening`, a chore whose own record
|
|
867
|
+
* argues "blast radius overrides lane" and which carried a rubric on purpose, reached `accept`
|
|
868
|
+
* with the gate never run. The board could not show what the ledger already knew.
|
|
869
|
+
*
|
|
870
|
+
* A prototype is still never scored — D60 grills leg A as dialogue, and that is a property of
|
|
871
|
+
* the lane, not of whether someone attached a rubric. Every other lane defers to the artifact. */
|
|
872
|
+
const declaresRubric = !!String(fm.rubric_path || '').trim();
|
|
873
|
+
/* EVERY lane defers to the artifact — the first version said so in the comment and applied it
|
|
874
|
+
* only to `chore`, so a discovery or incident initiative with no rubric (legitimate under D56)
|
|
875
|
+
* showed a permanent `no Review-Gate verdict recorded`: a rung it could never satisfy. The lanes
|
|
876
|
+
* that carry no rubric by design are named once, here. */
|
|
877
|
+
const RUBRIC_OPTIONAL = ['chore', 'discovery', 'incident', 'content'];
|
|
878
|
+
if (effLane === 'prototype') {
|
|
879
|
+
steps.push(step('review', 'review', 'na', 'leg A is grilled as dialogue, never scored (D60)'));
|
|
880
|
+
} else if (RUBRIC_OPTIONAL.includes(effLane) && !declaresRubric && !evals.length) {
|
|
881
|
+
steps.push(step('review', 'review', 'na', `${effLane} lane carries no rubric (D56)`));
|
|
866
882
|
} else if (evals.length) {
|
|
867
883
|
const last = evals[evals.length - 1];
|
|
868
|
-
|
|
869
|
-
|
|
884
|
+
const verdict = String(last.verdict || '').toUpperCase();
|
|
885
|
+
const label = `${last.verdict}${last.score != null ? ` (${last.score})` : ''} — advisory until calibrated per evaluator model (D40)`;
|
|
886
|
+
/* A SKIPPED gate is NOT a satisfied gate, and a FAIL is emphatically not one.
|
|
887
|
+
*
|
|
888
|
+
* This rung rendered both as a green ✓, which is the watermelon this repo names as its own
|
|
889
|
+
* failure mode: the board said the Review Gate was done over a row that says judging never
|
|
890
|
+
* happened. Thirteen initiatives currently render `✓ review` over a SKIPPED verdict, and the
|
|
891
|
+
* fix that made a chore's gate visible ADDED one. The module's own rule, repeated in three
|
|
892
|
+
* files — "a skip is an honest RECORD that verification did not happen; it is not
|
|
893
|
+
* verification" — now applies to the surface that reports it. */
|
|
894
|
+
if (verdict === 'SKIPPED') {
|
|
895
|
+
steps.push(step('review', 'review', 'pending', label,
|
|
896
|
+
'the gate was skipped, so it is still owed — commission an independent evaluator and record a real verdict',
|
|
897
|
+
{ rule: 'R3' }));
|
|
898
|
+
} else if (verdict === 'FAIL') {
|
|
899
|
+
steps.push(step('review', 'review', 'blocked', label,
|
|
900
|
+
'the Review Gate FAILED — remediate the findings and record a new verdict', { rule: 'R3' }));
|
|
901
|
+
} else {
|
|
902
|
+
steps.push(step('review', 'review', 'done', label));
|
|
903
|
+
}
|
|
870
904
|
} else {
|
|
871
905
|
steps.push(step('review', 'review', 'pending', 'no Review-Gate verdict recorded',
|
|
872
906
|
'score the build against the rubric (build-skills/evaluator.skill), then append to evals.jsonl',
|
|
@@ -7,7 +7,7 @@ description: Adversarially evaluate an agent run's output against its rubric and
|
|
|
7
7
|
|
|
8
8
|
This skill is PMOS's **Review Gate** made a first-class, repeatable capability (P4). It formalizes the
|
|
9
9
|
adversarial independent evaluator run by hand through P1–P3: judge what an agent run produced against a
|
|
10
|
-
rubric, write a structured verdict
|
|
10
|
+
rubric, write a structured verdict through the validated writer (step 9), and surface it on the board. It runs in
|
|
11
11
|
the **interactive agent runtime** — any capable coding agent; heavy judgment belongs in the agent, not
|
|
12
12
|
an Edge Function ([D09](/DECISIONS.md)/[D34](/DECISIONS.md), amended 2026-07-23). PMOS-self runs it in
|
|
13
13
|
Claude Code, but the runtime is not Claude-specific.
|
|
@@ -32,7 +32,7 @@ Claude Code, but the runtime is not Claude-specific.
|
|
|
32
32
|
may under-reward an unfamiliar style. Either way: **default to skeptical**, and when a pass feels easy
|
|
33
33
|
or you are uncertain, **say so explicitly** and flag the run for the cross-family spot-check /
|
|
34
34
|
minority-veto jury ([calibration-method](/planning/evals/calibration-method.md)). Do not let stylistic
|
|
35
|
-
(un)familiarity read as quality. **Record your `evaluator_model` on the
|
|
35
|
+
(un)familiarity read as quality. **Record your `evaluator_model` on the verdict** — κ is
|
|
36
36
|
calibrated per evaluator model, and an unrecorded model silently pools calibration across model swaps.
|
|
37
37
|
- **Capability floor — this role assumes a frontier-tier reasoner.** The adversarial passes below
|
|
38
38
|
(default-to-refute, both-ways pairwise, semantic-pass-not-grep, injection resistance) degrade
|
|
@@ -57,8 +57,8 @@ Claude Code, but the runtime is not Claude-specific.
|
|
|
57
57
|
([injection-defenses](/planning/research/injection-defenses.md)).
|
|
58
58
|
- **Structural scope (binding).** You are **read-only**: no external-communication tool, no
|
|
59
59
|
repository writes, no artifact publishing; your **only write is the schema-constrained
|
|
60
|
-
|
|
61
|
-
that reads adversarial content must not be the component that can act. Any future widening of
|
|
60
|
+
verdict, written through the validated writer (step 9)**. This removes the exfiltration leg of the
|
|
61
|
+
lethal trifecta — the component that reads adversarial content must not be the component that can act. Any future widening of
|
|
62
62
|
this scope requires the trifecta audit in [AGENTS.md](/AGENTS.md) and a PM decision
|
|
63
63
|
([injection-defenses](/planning/research/injection-defenses.md)).
|
|
64
64
|
- **Advisory, not final.** Your verdict informs; it does not decide. The human PM **Acceptance Gate**
|
|
@@ -158,13 +158,45 @@ FAIL — never a clean PASS.) The split is **persisted**, not just narrated (ste
|
|
|
158
158
|
available, or state `pass-fraction: unknown (no backend)` in kit mode. A high pass-fraction is a
|
|
159
159
|
leniency-drift warning to the PM, not a compliment to the builders
|
|
160
160
|
([llm-judge-calibration](/planning/research/llm-judge-calibration.md)).
|
|
161
|
-
9. **Persist it
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
(
|
|
161
|
+
9. **Persist it through the validated writer — never by hand.** Which writer depends on where you
|
|
162
|
+
are running, and **both are validated; neither is a raw append**:
|
|
163
|
+
|
|
164
|
+
| you are in | the writer | why |
|
|
165
|
+
|---|---|---|
|
|
166
|
+
| **a vendored kit** (`pmos/` in a product repo) | `pmos/scripts/pmos-log.sh eval …` | ships with the kit; writes the kit's `state/evals.jsonl` |
|
|
167
|
+
| **PMOS-self** (file mode, [D64](/DECISIONS.md)) | `node scripts/record-eval.js --file verdict.json` | writes `planning/runs/evals.jsonl` |
|
|
168
|
+
| a hosted tenant | the `save_eval` MCP tool | **deprecated for PMOS-self**; does not exist in file mode |
|
|
169
|
+
|
|
170
|
+
```
|
|
171
|
+
# kit
|
|
172
|
+
pmos/scripts/pmos-log.sh eval <run_id> <initiative_id> <score> <PASS|PASS-WITH-FINDINGS|FAIL> <rubric_path> [evaluator_model]
|
|
173
|
+
pmos/scripts/pmos-log.sh eval <run_id> <initiative_id> SKIPPED "<reason>" [rubric_path] [evaluator_model]
|
|
174
|
+
|
|
175
|
+
# PMOS-self
|
|
176
|
+
node scripts/record-eval.js --file verdict.json # or pipe the JSON on stdin
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
Both **fail loud**: a rejected verdict exits non-zero with the specific reason and writes nothing,
|
|
180
|
+
so a malformed record can never light the gate green. `record-eval.js` additionally refuses any
|
|
181
|
+
sink but the eval ledger — resolved, not by name — which is what keeps a Review-Gate verdict out
|
|
182
|
+
of the PM's acceptance ledger.
|
|
183
|
+
|
|
184
|
+
If the writer your context calls for is not present, **say so in the verdict and stop** — do not
|
|
185
|
+
hand-append. A ledger written by hand is how a field ends up meaning three things.
|
|
186
|
+
|
|
187
|
+
The record carries `{ initiative_id, run_id, rubric_path, score, verdict, evaluator_model,
|
|
188
|
+
recorded_at, dimensions }`, where `dimensions` is `{ <name>: { score, weight, note, evidence,
|
|
189
|
+
axis } }` — `evidence` carries the quote/pointer from step 6 (or the string `unverified`), and
|
|
190
|
+
**`axis` is `"spec"` or `"standards"`** so the two-axis split is persisted, not just narrated. Put
|
|
191
|
+
the two axis pass/fail summaries in the top-level `note` and tag every dimension with its axis.
|
|
192
|
+
|
|
193
|
+
A `SKIPPED` verdict is **declared, not omitted** — it requires `skipped_reason` and carries
|
|
194
|
+
`score: null`. A skip is an honest record that judging did not happen; it does not close the
|
|
195
|
+
verification the run owes, and the queue checker (PMOS-self: `scripts/check-verify-queue.js`)
|
|
196
|
+
will keep reporting the debt.
|
|
197
|
+
|
|
198
|
+
This is what makes the gate self-hosting: the verdict lands in the eval ledger and surfaces on the
|
|
199
|
+
board.
|
|
168
200
|
10. **Hand to the PM.** The Acceptance Gate is theirs, and the PM records the ground-truth `pm_label` to
|
|
169
201
|
`acceptances` — paired against your verdict by `v_calibration` to compute the kappa that calibrates this
|
|
170
202
|
gate ([D40](/DECISIONS.md), [calibration-method](/planning/evals/calibration-method.md)). If the PM
|
|
@@ -177,5 +209,6 @@ A short report: **most important finding first**, then the **two axis verdicts (
|
|
|
177
209
|
one line; Standards: pass/fail — one line)** and the **overall VERDICT** (score vs threshold, under
|
|
178
210
|
the combine rule) with the pass-fraction disclosure, the calibration exemplars used (step 3), the
|
|
179
211
|
numbered defects, and the per-dimension scores **with their evidence quotes/pointers and `axis`
|
|
180
|
-
tag** — followed by the
|
|
212
|
+
tag** — followed by the verdict record, written through **the writer for your context (step 9)**:
|
|
213
|
+
`pmos/scripts/pmos-log.sh eval …` in a vendored kit, `node scripts/record-eval.js …` in PMOS-self. Be concrete; cite specifics; if it is genuinely solid, say
|
|
181
214
|
so — but only after a real refutation attempt.
|
|
@@ -19,7 +19,7 @@ lanes and gates that already exist; where it and an authority disagree, the auth
|
|
|
19
19
|
router is the bug (fix it via the weekly harness scan).
|
|
20
20
|
|
|
21
21
|
> **Format note:** three-level progressive disclosure ([SKILL-FORMAT](/skills/core/SKILL-FORMAT.md)).
|
|
22
|
-
> Sources of truth: the AGENTS.md lane list, `.github/scripts/initiative-gate.sh
|
|
22
|
+
> Sources of truth: the AGENTS.md lane list, the initiative gate (PMOS-self: `.github/scripts/initiative-gate.sh`; a vendored kit: `pmos/scripts/gate-initiative.sh`), D56/D58/D60.
|
|
23
23
|
|
|
24
24
|
## Step 0 — three checks before any route
|
|
25
25
|
|
|
@@ -8,7 +8,7 @@ license: MIT
|
|
|
8
8
|
|
|
9
9
|
This skill is PMOS's **Review Gate** made a first-class, repeatable capability (P4). It formalizes the
|
|
10
10
|
adversarial independent evaluator run by hand through P1–P3: judge what an agent run produced against a
|
|
11
|
-
rubric, write a structured verdict
|
|
11
|
+
rubric, write a structured verdict through the validated writer (step 9), and surface it on the board. It runs in
|
|
12
12
|
the **interactive agent runtime** — any capable coding agent; heavy judgment belongs in the agent, not
|
|
13
13
|
an Edge Function ([D09](/DECISIONS.md)/[D34](/DECISIONS.md), amended 2026-07-23). PMOS-self runs it in
|
|
14
14
|
Claude Code, but the runtime is not Claude-specific.
|
|
@@ -33,7 +33,7 @@ Claude Code, but the runtime is not Claude-specific.
|
|
|
33
33
|
may under-reward an unfamiliar style. Either way: **default to skeptical**, and when a pass feels easy
|
|
34
34
|
or you are uncertain, **say so explicitly** and flag the run for the cross-family spot-check /
|
|
35
35
|
minority-veto jury ([calibration-method](/planning/evals/calibration-method.md)). Do not let stylistic
|
|
36
|
-
(un)familiarity read as quality. **Record your `evaluator_model` on the
|
|
36
|
+
(un)familiarity read as quality. **Record your `evaluator_model` on the verdict** — κ is
|
|
37
37
|
calibrated per evaluator model, and an unrecorded model silently pools calibration across model swaps.
|
|
38
38
|
- **Capability floor — this role assumes a frontier-tier reasoner.** The adversarial passes below
|
|
39
39
|
(default-to-refute, both-ways pairwise, semantic-pass-not-grep, injection resistance) degrade
|
|
@@ -58,8 +58,8 @@ Claude Code, but the runtime is not Claude-specific.
|
|
|
58
58
|
([injection-defenses](/planning/research/injection-defenses.md)).
|
|
59
59
|
- **Structural scope (binding).** You are **read-only**: no external-communication tool, no
|
|
60
60
|
repository writes, no artifact publishing; your **only write is the schema-constrained
|
|
61
|
-
|
|
62
|
-
that reads adversarial content must not be the component that can act. Any future widening of
|
|
61
|
+
verdict, written through the validated writer (step 9)**. This removes the exfiltration leg of the
|
|
62
|
+
lethal trifecta — the component that reads adversarial content must not be the component that can act. Any future widening of
|
|
63
63
|
this scope requires the trifecta audit in [AGENTS.md](/AGENTS.md) and a PM decision
|
|
64
64
|
([injection-defenses](/planning/research/injection-defenses.md)).
|
|
65
65
|
- **Advisory, not final.** Your verdict informs; it does not decide. The human PM **Acceptance Gate**
|
|
@@ -159,13 +159,45 @@ FAIL — never a clean PASS.) The split is **persisted**, not just narrated (ste
|
|
|
159
159
|
available, or state `pass-fraction: unknown (no backend)` in kit mode. A high pass-fraction is a
|
|
160
160
|
leniency-drift warning to the PM, not a compliment to the builders
|
|
161
161
|
([llm-judge-calibration](/planning/research/llm-judge-calibration.md)).
|
|
162
|
-
9. **Persist it
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
166
|
-
|
|
167
|
-
|
|
168
|
-
(
|
|
162
|
+
9. **Persist it through the validated writer — never by hand.** Which writer depends on where you
|
|
163
|
+
are running, and **both are validated; neither is a raw append**:
|
|
164
|
+
|
|
165
|
+
| you are in | the writer | why |
|
|
166
|
+
|---|---|---|
|
|
167
|
+
| **a vendored kit** (`pmos/` in a product repo) | `pmos/scripts/pmos-log.sh eval …` | ships with the kit; writes the kit's `state/evals.jsonl` |
|
|
168
|
+
| **PMOS-self** (file mode, [D64](/DECISIONS.md)) | `node scripts/record-eval.js --file verdict.json` | writes `planning/runs/evals.jsonl` |
|
|
169
|
+
| a hosted tenant | the `save_eval` MCP tool | **deprecated for PMOS-self**; does not exist in file mode |
|
|
170
|
+
|
|
171
|
+
```
|
|
172
|
+
# kit
|
|
173
|
+
pmos/scripts/pmos-log.sh eval <run_id> <initiative_id> <score> <PASS|PASS-WITH-FINDINGS|FAIL> <rubric_path> [evaluator_model]
|
|
174
|
+
pmos/scripts/pmos-log.sh eval <run_id> <initiative_id> SKIPPED "<reason>" [rubric_path] [evaluator_model]
|
|
175
|
+
|
|
176
|
+
# PMOS-self
|
|
177
|
+
node scripts/record-eval.js --file verdict.json # or pipe the JSON on stdin
|
|
178
|
+
```
|
|
179
|
+
|
|
180
|
+
Both **fail loud**: a rejected verdict exits non-zero with the specific reason and writes nothing,
|
|
181
|
+
so a malformed record can never light the gate green. `record-eval.js` additionally refuses any
|
|
182
|
+
sink but the eval ledger — resolved, not by name — which is what keeps a Review-Gate verdict out
|
|
183
|
+
of the PM's acceptance ledger.
|
|
184
|
+
|
|
185
|
+
If the writer your context calls for is not present, **say so in the verdict and stop** — do not
|
|
186
|
+
hand-append. A ledger written by hand is how a field ends up meaning three things.
|
|
187
|
+
|
|
188
|
+
The record carries `{ initiative_id, run_id, rubric_path, score, verdict, evaluator_model,
|
|
189
|
+
recorded_at, dimensions }`, where `dimensions` is `{ <name>: { score, weight, note, evidence,
|
|
190
|
+
axis } }` — `evidence` carries the quote/pointer from step 6 (or the string `unverified`), and
|
|
191
|
+
**`axis` is `"spec"` or `"standards"`** so the two-axis split is persisted, not just narrated. Put
|
|
192
|
+
the two axis pass/fail summaries in the top-level `note` and tag every dimension with its axis.
|
|
193
|
+
|
|
194
|
+
A `SKIPPED` verdict is **declared, not omitted** — it requires `skipped_reason` and carries
|
|
195
|
+
`score: null`. A skip is an honest record that judging did not happen; it does not close the
|
|
196
|
+
verification the run owes, and the queue checker (PMOS-self: `scripts/check-verify-queue.js`)
|
|
197
|
+
will keep reporting the debt.
|
|
198
|
+
|
|
199
|
+
This is what makes the gate self-hosting: the verdict lands in the eval ledger and surfaces on the
|
|
200
|
+
board.
|
|
169
201
|
10. **Hand to the PM.** The Acceptance Gate is theirs, and the PM records the ground-truth `pm_label` to
|
|
170
202
|
`acceptances` — paired against your verdict by `v_calibration` to compute the kappa that calibrates this
|
|
171
203
|
gate ([D40](/DECISIONS.md), [calibration-method](/planning/evals/calibration-method.md)). If the PM
|
|
@@ -178,5 +210,6 @@ A short report: **most important finding first**, then the **two axis verdicts (
|
|
|
178
210
|
one line; Standards: pass/fail — one line)** and the **overall VERDICT** (score vs threshold, under
|
|
179
211
|
the combine rule) with the pass-fraction disclosure, the calibration exemplars used (step 3), the
|
|
180
212
|
numbered defects, and the per-dimension scores **with their evidence quotes/pointers and `axis`
|
|
181
|
-
tag** — followed by the
|
|
213
|
+
tag** — followed by the verdict record, written through **the writer for your context (step 9)**:
|
|
214
|
+
`pmos/scripts/pmos-log.sh eval …` in a vendored kit, `node scripts/record-eval.js …` in PMOS-self. Be concrete; cite specifics; if it is genuinely solid, say
|
|
182
215
|
so — but only after a real refutation attempt.
|
|
@@ -21,7 +21,7 @@ lanes and gates that already exist; where it and an authority disagree, the auth
|
|
|
21
21
|
router is the bug (fix it via the weekly harness scan).
|
|
22
22
|
|
|
23
23
|
> **Format note:** three-level progressive disclosure ([SKILL-FORMAT](/skills/core/SKILL-FORMAT.md)).
|
|
24
|
-
> Sources of truth: the AGENTS.md lane list, `.github/scripts/initiative-gate.sh
|
|
24
|
+
> Sources of truth: the AGENTS.md lane list, the initiative gate (PMOS-self: `.github/scripts/initiative-gate.sh`; a vendored kit: `pmos/scripts/gate-initiative.sh`), D56/D58/D60.
|
|
25
25
|
|
|
26
26
|
## Step 0 — three checks before any route
|
|
27
27
|
|
package/workspace/README.md
CHANGED
|
@@ -156,6 +156,28 @@ tokens, JWTs, `Authorization:` values, api keys): scrollback holds whatever the
|
|
|
156
156
|
so D08's never-log-headers discipline applies to what this hands back. Matching runs against
|
|
157
157
|
the *scrubbed* text, so the endpoint cannot be used to grep a secret out.
|
|
158
158
|
|
|
159
|
+
## `search everything…` searches the folder (the file lens)
|
|
160
|
+
The search box shipped matching three derived lists — initiative slugs, paths **declared on** an
|
|
161
|
+
initiative record, and session names — so a file PMOS had never been told about could not be found
|
|
162
|
+
by typing its exact name, and the miss printed *"nothing matches — a successful read of nothing"*
|
|
163
|
+
after a search that had never opened the folder.
|
|
164
|
+
|
|
165
|
+
`GET /api/search/files?q=…&ws=…&limit=…` is the folder corpus: every file under the workspace root,
|
|
166
|
+
matched by **path** and by **content**, with the matching line and its number. Clicking a hit opens
|
|
167
|
+
it in the ordinary document tab. It is a **different corpus** from `/api/search` above — that one is
|
|
168
|
+
scrollback, this one is disk — and they are deliberately separate endpoints, because their hits
|
|
169
|
+
share no field and one payload would have to lie about what it looked at.
|
|
170
|
+
|
|
171
|
+
Three guarantees. The walk uses the **folder lens's own rules** (`node_modules`, `dist`, `.git` and
|
|
172
|
+
the rest skipped, dotted names excluded), so search and the file tree can never disagree about what
|
|
173
|
+
is in the folder. There is **no path parameter** — the query is a needle, not a location — and the
|
|
174
|
+
walk starts at the root and never follows a symlink out of it, so nothing becomes reachable that
|
|
175
|
+
`/api/doc` would refuse; `openable` on each hit reports whether the reader can actually show it.
|
|
176
|
+
And it is **bounded, out loud**: `truncated` means files were never looked at, `content_truncated`
|
|
177
|
+
means every file was named but not every body was read, `scanned`/`content_scanned` give the counts.
|
|
178
|
+
Two caps, two flags — merged into one, the serious flag would light on any repository with a lock
|
|
179
|
+
file in it, which is the same as retiring it.
|
|
180
|
+
|
|
159
181
|
## Branches: a new one doesn't move the others
|
|
160
182
|
|
|
161
183
|
Starting a session, the branch control is **optional** — the default is "stay on current" and it
|