@onlooker-community/ecosystem 0.34.1 → 0.43.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (80) hide show
  1. package/.claude-plugin/plugin.json +1 -1
  2. package/.github/workflows/release.yml +14 -14
  3. package/.release-please-manifest.json +4 -4
  4. package/AGENTS.md +5 -0
  5. package/CHANGELOG.md +168 -0
  6. package/CLAUDE.md +5 -0
  7. package/docs/lesson-promotion-pipeline.md +6 -6
  8. package/docs/superpowers/plans/2026-08-10-lesson-confirmation.md +1273 -0
  9. package/docs/superpowers/plans/2026-08-11-lesson-judging.md +1232 -0
  10. package/docs/superpowers/plans/2026-08-11-lesson-unconfirm.md +458 -0
  11. package/docs/superpowers/plans/2026-08-12-author-key.md +567 -0
  12. package/docs/superpowers/plans/2026-08-13-approved-pool.md +747 -0
  13. package/docs/superpowers/plans/2026-08-13-librarian-cluster.md +392 -0
  14. package/docs/superpowers/plans/2026-08-14-criterion-scores-consumers.md +1374 -0
  15. package/docs/superpowers/plans/2026-08-14-criterion-scores-schema.md +277 -0
  16. package/docs/superpowers/plans/2026-08-16-cartographer-undocumented-entity.md +741 -0
  17. package/docs/superpowers/specs/2026-08-10-lesson-confirmation-design.md +213 -0
  18. package/docs/superpowers/specs/2026-08-11-lesson-judging-design.md +267 -0
  19. package/docs/superpowers/specs/2026-08-11-lesson-unconfirm-design.md +152 -0
  20. package/docs/superpowers/specs/2026-08-12-author-key-design.md +317 -0
  21. package/docs/superpowers/specs/2026-08-13-approved-pool-design.md +232 -0
  22. package/docs/superpowers/specs/2026-08-14-criterion-scores-design.md +254 -0
  23. package/docs/superpowers/specs/2026-08-16-cartographer-undocumented-entity-design.md +249 -0
  24. package/package.json +2 -2
  25. package/plugins/cartographer/.claude-plugin/plugin.json +1 -1
  26. package/plugins/cartographer/CHANGELOG.md +54 -0
  27. package/plugins/cartographer/README.md +56 -1
  28. package/plugins/cartographer/config.json +7 -1
  29. package/plugins/cartographer/scripts/hooks/cartographer-post-write.sh +4 -4
  30. package/plugins/cartographer/scripts/hooks/cartographer-session-start.sh +7 -4
  31. package/plugins/cartographer/scripts/lib/cartographer-collect.sh +8 -1
  32. package/plugins/cartographer/scripts/lib/cartographer-config.sh +26 -0
  33. package/plugins/cartographer/scripts/lib/cartographer-events.sh +90 -0
  34. package/plugins/cartographer/scripts/lib/cartographer-filter.sh +105 -0
  35. package/plugins/cartographer/scripts/lib/cartographer-omission.sh +130 -0
  36. package/plugins/cartographer/scripts/lib/cartographer-resolve.sh +132 -0
  37. package/plugins/cartographer/scripts/run-audit.sh +136 -53
  38. package/plugins/cartographer/skills/cartographer/SKILL.md +33 -9
  39. package/plugins/librarian/.claude-plugin/plugin.json +1 -1
  40. package/plugins/librarian/CHANGELOG.md +64 -0
  41. package/plugins/librarian/config.json +30 -1
  42. package/plugins/librarian/docs/adr/002-agent-definitions-are-shared-assets.md +82 -0
  43. package/plugins/librarian/scripts/hooks/librarian-session-end.sh +28 -0
  44. package/plugins/librarian/scripts/hooks/librarian-session-start.sh +42 -12
  45. package/plugins/librarian/scripts/lib/librarian-author-key.sh +279 -0
  46. package/plugins/librarian/scripts/lib/librarian-cli.sh +344 -1
  47. package/plugins/librarian/scripts/lib/librarian-lesson-judge.sh +349 -0
  48. package/plugins/librarian/scripts/lib/librarian-lesson-promote.sh +225 -0
  49. package/plugins/librarian/scripts/lib/librarian-lesson-review.sh +312 -0
  50. package/plugins/librarian/scripts/lib/librarian-lesson-rubric.sh +56 -0
  51. package/plugins/librarian/scripts/lib/librarian-lesson-storage.sh +82 -13
  52. package/plugins/librarian/scripts/lib/librarian-lesson-transform.sh +6 -1
  53. package/plugins/librarian/scripts/lib/librarian-lesson-validate.sh +131 -62
  54. package/plugins/librarian/skills/librarian/SKILL.md +98 -1
  55. package/plugins/tribunal/.claude-plugin/plugin.json +1 -1
  56. package/plugins/tribunal/CHANGELOG.md +56 -0
  57. package/plugins/tribunal/agents/tribunal-judge-adversarial.md +34 -0
  58. package/plugins/tribunal/agents/tribunal-judge-security.md +33 -0
  59. package/plugins/tribunal/agents/tribunal-judge-standard.md +12 -0
  60. package/plugins/tribunal/config.json +1 -0
  61. package/plugins/tribunal/scripts/lib/tribunal-aggregate.sh +154 -11
  62. package/plugins/tribunal/scripts/lib/tribunal-gate.sh +193 -15
  63. package/plugins/tribunal/skills/tribunal/SKILL.md +5 -3
  64. package/test/bats/cartographer-config.bats +43 -0
  65. package/test/bats/cartographer-events.bats +267 -0
  66. package/test/bats/cartographer-filter.bats +169 -0
  67. package/test/bats/cartographer-omission.bats +202 -0
  68. package/test/bats/cartographer-resolve.bats +287 -0
  69. package/test/bats/cartographer-run-audit.bats +325 -0
  70. package/test/bats/librarian-author-key.bats +477 -0
  71. package/test/bats/librarian-lesson-judge.bats +884 -0
  72. package/test/bats/librarian-lesson-promote.bats +552 -0
  73. package/test/bats/librarian-lesson-review.bats +900 -0
  74. package/test/bats/librarian-lesson-transform.bats +17 -3
  75. package/test/bats/librarian-session-end.bats +161 -0
  76. package/test/bats/librarian-session-start.bats +34 -0
  77. package/test/bats/tribunal-aggregate.bats +354 -1
  78. package/test/bats/tribunal-gate.bats +631 -0
  79. package/test/bats/tribunal-judge-agents.bats +119 -0
  80. package/test/node/lesson-validate-agreement.test.mjs +35 -6
@@ -0,0 +1,1374 @@
1
+ # Per-Criterion Scores — Consumer Implementation Plan
2
+
3
+ > **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
4
+
5
+ **Goal:** Make tribunal's and librarian's rubric `weight` and `min_pass` values actually change outcomes, so a criterion can act as a floor that a strong weighted mean cannot average away.
6
+
7
+ **Architecture:** Judges gain a `criterion_scores` map on their output contract, keyed by **rubric** criterion name. Tribunal's `tribunal_aggregate` computes a real weighted mean from those scores and its gate blocks on any criterion below its `min_pass`, emitting the new `criterion_floor` reason. Librarian's parallel implementations gain the same two capabilities through widened signatures, which lets the public lesson tier trade its inert `unanimous` stand-in for a real `disclosure` floor.
8
+
9
+ **Tech Stack:** bash 3.2-compatible shell, `jq`, `awk`, bats, `@onlooker-community/schema` 2.12.0.
10
+
11
+ ## Global Constraints
12
+
13
+ - **`@onlooker-community/schema` must be at `^2.12.0`.** 2.12.0 adds optional `criterion_scores` on `TribunalVerdictPayload` and the `criterion_floor` value in `tribunal.gate.blocked`'s `reason` enum plus optional `failed_criterion`. Task 1 is blocked until that version publishes.
14
+ - **Absent must never read as zero.** A judge that did not score a criterion must never be treated as one that scored `0.0`. Use `has()` or an explicit type test. **Specifically banned: defaulting a criterion score with `// 0`.** (Defaulting a missing *array* with `// []` is fine and appears in this plan — the ban is on scores.)
15
+ - **Look criterion names up as `.criterion_scores[$name]` with `--arg`/`$c.name`, never as a dotted path.** `path-traversal` and `edge-cases` already ship in agent contracts. `jq '.criterion_scores.path-traversal'` is a **compile error**: `jq` exits 3 with empty stdout, which `awk` then reads as `0`, failing toward blocking on a name the rubric author is entitled to write.
16
+ - **A floor on a criterion that no judge scored must be surfaced, never silently passed.** That is this design's own failure mode one layer down.
17
+ - **Tribunal and librarian each implement their own aggregate and gate. Do not unify them.** `plugins/librarian/docs/adr/002-agent-definitions-are-shared-assets.md` licenses reusing tribunal's *agent definitions* but rules out sourcing its bash. The near-duplicate jq between Tasks 2/3 and Task 4 is **plan-mandated**, not an oversight — a reviewer must not flag it as a DRY violation.
18
+ - **macOS bash 3.2:** a failing **non-final** `[[ ]]` does **not** fail a bats test. Use `[ ]` for assertions, or append `|| return 1`. A bats `[[ =~ ]]` regex must be an **unquoted variable**.
19
+ - **Use `run --separate-stderr`** whenever a test asserts empty stdout alongside a stderr message. Plain `run` merges them and the assertion becomes unsatisfiable.
20
+ - **Every test must be falsifiable.** Before committing a task, delete the guard under test and confirm a test fails. The recurring defect in this epic — seven-plus instances — is a *downstream* guard making an *upstream* guard's test pass whether or not the upstream guard exists.
21
+ - All hooks are bash; runtime artifacts live under `$ONLOOKER_DIR`, never a hardcoded `~/.onlooker`; ULIDs via each plugin's own `*-ulid.sh`; config defaults in `config.json` with overrides per ADR-004; the emitter fails open per ADR-005.
22
+ - Tests are bats under an isolated temp home per `.claude/skills/writing-tests`. Run `npm run test:ci` before the PR.
23
+ - **PR-only. Never push to `main`.** Branch from `feat/criterion-scores`, which already carries the corrected spec.
24
+
25
+ ---
26
+
27
+ ## Background the tasks assume
28
+
29
+ Verified against the repo on 2026-08-14. Trust this section over the spec's prose — the spec was corrected today precisely because its descriptions of code were wrong, and two of its claims were **still** wrong after that correction (noted below).
30
+
31
+ **Tribunal's default rubric** (`plugins/tribunal/config.json`, `.tribunal.rubric.builtins[0]`):
32
+
33
+ | criterion | weight | min_pass |
34
+ |---|---|---|
35
+ | correctness | 0.4 | 0.7 |
36
+ | completeness | 0.3 | 0.7 |
37
+ | safety | 0.2 | **0.8** |
38
+ | clarity | 0.1 | 0.5 |
39
+
40
+ `judge_types: ["standard","adversarial"]`, `gate_policy: "majority"`, `aggregation_method: "weighted_mean"`, `score_threshold: 0.75`.
41
+
42
+ **Librarian's rubrics** (`plugins/librarian/config.json`, `.librarian.lesson_judging.rubrics` — an **array**, looked up by `.id`):
43
+
44
+ - `lesson-promotion` — grounding 0.45/0.7, scope_accuracy 0.35/0.7, generality 0.20/0.6; `gate_policy: majority`
45
+ - `lesson-promotion-public` — grounding 0.32/0.7, scope_accuracy 0.24/0.7, generality 0.14/0.6, **disclosure 0.30/0.9**; `gate_policy: unanimous`
46
+
47
+ Both declare `judge_types: ["standard","adversarial"]`.
48
+
49
+ **The three judge agents report their own investigative lenses, not rubric criteria:**
50
+
51
+ | agent | `criteria_evaluated` example | has a rubric section? |
52
+ |---|---|---|
53
+ | `tribunal-judge-standard.md` | `["correctness","completeness","clarity"]` | yes |
54
+ | `tribunal-judge-adversarial.md` | `["edge-cases","concurrency","idempotency"]` | **no** |
55
+ | `tribunal-judge-security.md` | `["injection","secrets","path-traversal"]` | **no** |
56
+
57
+ **Two corrections to the corrected spec.** It says `-security` already receives the rubric — it does not; the gap is two agents. And it implies `criteria_evaluated` is rubric-derived — it is not, for any of the three. Task 5 fixes the spec text.
58
+
59
+ **Consequence:** on the default panel, `safety` — the criterion with the highest floor — appears in **no** judge's output. Its floor would silently never fire. The resolution this plan adopts:
60
+
61
+ > `criteria_evaluated` keeps its present meaning — the agent's own investigative lenses — and is left **unchanged**. `criterion_scores` is a **separate** map keyed by **rubric** criterion names. Every judge scores every rubric criterion it was given; its lens list is how it got there.
62
+
63
+ **Callers.** `tribunal_aggregate` and `tribunal_gate_decide` are called only from `plugins/tribunal/skills/tribunal/SKILL.md` (steps 5 and 7). `librarian_lesson_aggregate` and `librarian_lesson_gate` are called only from `librarian_lesson_judge` in the same file. Both new parameters are **optional and trailing**, so no caller breaks.
64
+
65
+ ---
66
+
67
+ ## File structure
68
+
69
+ | File | Responsibility | Task |
70
+ |---|---|---|
71
+ | `package.json` | schema dependency floor | 1 |
72
+ | `plugins/tribunal/agents/tribunal-judge-standard.md` | output contract gains `criterion_scores` | 1 |
73
+ | `plugins/tribunal/agents/tribunal-judge-adversarial.md` | gains a rubric section + `criterion_scores` | 1 |
74
+ | `plugins/tribunal/agents/tribunal-judge-security.md` | gains a rubric section + `criterion_scores` | 1 |
75
+ | `test/bats/tribunal-judge-agents.bats` | **new** — contract tests over the shipped agent files | 1 |
76
+ | `plugins/tribunal/scripts/lib/tribunal-aggregate.sh` | real `weighted_mean` | 2 |
77
+ | `test/bats/tribunal-aggregate.bats` | aggregation tests (10 existing) | 2 |
78
+ | `plugins/tribunal/scripts/lib/tribunal-gate.sh` | `min_pass` floor, `criterion_floor` reason | 3 |
79
+ | `test/bats/tribunal-gate.bats` | gate tests (10 existing) | 3 |
80
+ | `plugins/tribunal/skills/tribunal/SKILL.md` | pass the rubric; emit `failed_criterion` | 3 |
81
+ | `plugins/librarian/scripts/lib/librarian-lesson-judge.sh` | rubric-aware aggregate + gate | 4 |
82
+ | `test/bats/librarian-lesson-judge.bats` | judging tests (43 existing) | 4 |
83
+ | `plugins/librarian/config.json` | public tier `unanimous` → `majority` | 5 |
84
+ | `plugins/librarian/scripts/lib/librarian-lesson-rubric.sh` | the "inert" comment is no longer true | 5 |
85
+ | `plugins/librarian/skills/librarian/SKILL.md` | judges must return `criterion_scores` | 5 |
86
+ | `docs/superpowers/specs/2026-08-11-lesson-judging-design.md` | the `unanimous` stand-in never worked | 5 |
87
+ | `docs/superpowers/specs/2026-08-14-criterion-scores-design.md` | `-security` correction | 5 |
88
+
89
+ ---
90
+
91
+ ### Task 1: Judges emit `criterion_scores`
92
+
93
+ **Files:**
94
+ - Modify: `package.json` (the `@onlooker-community/schema` dependency)
95
+ - Modify: `plugins/tribunal/agents/tribunal-judge-standard.md:26-47`
96
+ - Modify: `plugins/tribunal/agents/tribunal-judge-adversarial.md:34-51`
97
+ - Modify: `plugins/tribunal/agents/tribunal-judge-security.md:30-47`
98
+ - Test: `test/bats/tribunal-judge-agents.bats` (new file)
99
+
100
+ **Interfaces:**
101
+ - Consumes: nothing from earlier tasks.
102
+ - Produces: the judge output contract every later task assumes —
103
+ `criterion_scores` is an **object** mapping a **rubric criterion name** to a number in `[0,1]`. It is **optional**: a judge that cannot score a criterion omits that key rather than sending `0`. `criteria_evaluated` is unchanged and remains the agent's own lens list.
104
+
105
+ **BLOCKED UNTIL** `@onlooker-community/schema` 2.12.0 is published to npm. Check with `npm view @onlooker-community/schema version`. If it still reports `2.11.0`, stop and report BLOCKED — do not hand-edit a version that cannot install.
106
+
107
+ - [ ] **Step 1: Write the failing contract test**
108
+
109
+ Create `test/bats/tribunal-judge-agents.bats`:
110
+
111
+ ```bash
112
+ #!/usr/bin/env bats
113
+
114
+ setup() {
115
+ source "${BATS_TEST_DIRNAME}/../helpers/setup.bash"
116
+ setup_test_env
117
+
118
+ AGENTS_DIR="${REPO_ROOT}/plugins/tribunal/agents"
119
+ }
120
+
121
+ # Extract the first fenced ```json block from an agent definition.
122
+ _agent_json() {
123
+ awk '/^```json/ { f = 1; next } /^```/ { if (f) exit } f' "$1"
124
+ }
125
+
126
+ @test "every judge agent's example verdict carries criterion_scores" {
127
+ local agent json
128
+ for agent in standard adversarial security; do
129
+ json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-${agent}.md")
130
+ [ -n "$json" ] || return 1
131
+ printf '%s' "$json" | jq -e 'has("criterion_scores")' >/dev/null || return 1
132
+ done
133
+ }
134
+
135
+ @test "criterion_scores is an object of numbers in [0,1], not an array" {
136
+ local agent json
137
+ for agent in standard adversarial security; do
138
+ json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-${agent}.md")
139
+ printf '%s' "$json" | jq -e '
140
+ (.criterion_scores | type) == "object"
141
+ and (.criterion_scores | length) > 0
142
+ and all(.criterion_scores[]; type == "number" and . >= 0 and . <= 1)
143
+ ' >/dev/null || return 1
144
+ done
145
+ }
146
+
147
+ @test "criterion_scores is keyed by rubric criteria, not by the agent's own lenses" {
148
+ # The default rubric's criteria are the only legal keys. This is the whole
149
+ # point: an agent keying by its own lens names (edge-cases, injection)
150
+ # produces scores no aggregator can ever match to a floor.
151
+ local rubric_names json agent
152
+ rubric_names=$(jq -c '[.tribunal.rubric.builtins[0].criteria[].name]' \
153
+ "${REPO_ROOT}/plugins/tribunal/config.json")
154
+
155
+ for agent in standard adversarial security; do
156
+ json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-${agent}.md")
157
+ printf '%s' "$json" | jq -e --argjson want "$rubric_names" '
158
+ [.criterion_scores | keys[]] | all(. as $k | $want | index($k) != null)
159
+ ' >/dev/null || return 1
160
+ done
161
+ }
162
+
163
+ @test "every judge agent scores safety, the criterion with the highest floor" {
164
+ # safety carries min_pass 0.8 and appeared in NO agent contract before this
165
+ # change, so its floor could never fire. Regression guard.
166
+ local agent json
167
+ for agent in standard adversarial security; do
168
+ json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-${agent}.md")
169
+ printf '%s' "$json" | jq -e '.criterion_scores | has("safety")' >/dev/null || return 1
170
+ done
171
+ }
172
+
173
+ @test "adversarial and security agents document the rubric they must score" {
174
+ local agent
175
+ for agent in adversarial security; do
176
+ grep -qi 'rubric' "${AGENTS_DIR}/tribunal-judge-${agent}.md" || return 1
177
+ done
178
+ }
179
+
180
+ @test "criteria_evaluated keeps each agent's own investigative lenses" {
181
+ # Deliberately NOT unified with criterion_scores. If a future edit collapses
182
+ # the two, the adversarial agent stops reporting what it actually probed.
183
+ local json
184
+ json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-adversarial.md")
185
+ printf '%s' "$json" | jq -e '
186
+ (.criteria_evaluated | index("edge-cases")) != null
187
+ ' >/dev/null
188
+ }
189
+
190
+ @test "the schema dependency admits criterion_scores" {
191
+ # criterion_scores and the criterion_floor reason land in 2.12.0. On ^2.11.0
192
+ # the runtime emitter rejects the payload wherever the package resolves.
193
+ local range
194
+ range=$(jq -r '.dependencies["@onlooker-community/schema"] // .devDependencies["@onlooker-community/schema"]' \
195
+ "${REPO_ROOT}/package.json")
196
+ [ "$range" = "^2.12.0" ]
197
+ }
198
+ ```
199
+
200
+ - [ ] **Step 2: Run the test to verify it fails**
201
+
202
+ Run: `bats test/bats/tribunal-judge-agents.bats`
203
+ Expected: FAIL — 6 of 7 tests fail. No agent has `criterion_scores` yet, and the dependency is `^2.11.0`. The `criteria_evaluated` test passes already, which is correct: it guards behavior that must **not** change.
204
+
205
+ - [ ] **Step 3: Bump the schema dependency**
206
+
207
+ ```bash
208
+ npm install @onlooker-community/schema@^2.12.0
209
+ ```
210
+
211
+ Confirm `package.json` reads `"@onlooker-community/schema": "^2.12.0"` and `package-lock.json` resolves `2.12.0`.
212
+
213
+ - [ ] **Step 4: Add `criterion_scores` to the standard judge**
214
+
215
+ In `plugins/tribunal/agents/tribunal-judge-standard.md`, inside the ```json block, add the field after `criteria_evaluated`:
216
+
217
+ ```json
218
+ "criteria_evaluated": ["correctness", "completeness", "clarity"],
219
+ "criterion_scores": {
220
+ "correctness": 0.9,
221
+ "completeness": 0.75,
222
+ "safety": 0.85,
223
+ "clarity": 0.8
224
+ },
225
+ ```
226
+
227
+ Then, immediately after the "Required fields" paragraph (currently line 43), add:
228
+
229
+ ```markdown
230
+ `criterion_scores` maps **each criterion name from the rubric you were given** to your score for it in `[0,1]`. This is separate from `criteria_evaluated`, which lists the dimensions *you* chose to investigate — the rubric's criteria are what the orchestrator weights and floors.
231
+
232
+ Score every rubric criterion you can judge. **Omit any criterion you genuinely cannot assess — do not send `0` for it.** A `0` means "I assessed this and it failed"; an omission means "I did not assess this." The orchestrator treats them very differently: a `0` on a criterion with a floor blocks the task outright, while an omission is reported as a coverage gap.
233
+ ```
234
+
235
+ - [ ] **Step 5: Add a rubric section and `criterion_scores` to the adversarial judge**
236
+
237
+ In `plugins/tribunal/agents/tribunal-judge-adversarial.md`, add to the ```json block after `criteria_evaluated`:
238
+
239
+ ```json
240
+ "criteria_evaluated": ["edge-cases", "concurrency", "idempotency"],
241
+ "criterion_scores": {
242
+ "correctness": 0.5,
243
+ "completeness": 0.6,
244
+ "safety": 0.55,
245
+ "clarity": 0.8
246
+ },
247
+ ```
248
+
249
+ Then add this section immediately before `## Output format`:
250
+
251
+ ```markdown
252
+ ## Scoring against the rubric
253
+
254
+ You are given a rubric with named criteria, each carrying a weight and a
255
+ `min_pass` floor. Your falsification work is how you form a judgment; the
256
+ rubric's criteria are how you report it.
257
+
258
+ Report a score in `[0,1]` for every rubric criterion in `criterion_scores`,
259
+ keyed by the rubric's own names. Your `criteria_evaluated` list stays what it
260
+ has always been — the dimensions you probed (edge cases, concurrency,
261
+ idempotency). The two lists are not expected to match.
262
+
263
+ `safety` in particular is a criterion you are well placed to score and no other
264
+ default judge covers. A crash on malformed input, a non-idempotent migration, a
265
+ race that corrupts state — those are safety findings, and this is where they
266
+ belong.
267
+
268
+ **Omit any criterion you cannot assess rather than scoring it `0`.** A `0` says
269
+ you assessed it and it failed, which on a criterion with a floor blocks the
270
+ task by itself.
271
+ ```
272
+
273
+ - [ ] **Step 6: Add a rubric section and `criterion_scores` to the security judge**
274
+
275
+ In `plugins/tribunal/agents/tribunal-judge-security.md`, add to the ```json block after `criteria_evaluated`:
276
+
277
+ ```json
278
+ "criteria_evaluated": ["injection", "secrets", "path-traversal"],
279
+ "criterion_scores": {
280
+ "correctness": 0.7,
281
+ "completeness": 0.6,
282
+ "safety": 0.2,
283
+ "clarity": 0.75
284
+ },
285
+ ```
286
+
287
+ Then add this section immediately before `## Output format`:
288
+
289
+ ```markdown
290
+ ## Scoring against the rubric
291
+
292
+ You are given a rubric with named criteria, each carrying a weight and a
293
+ `min_pass` floor. Your findings are how you form a judgment; the rubric's
294
+ criteria are how you report it.
295
+
296
+ Report a score in `[0,1]` for every rubric criterion in `criterion_scores`,
297
+ keyed by the rubric's own names. Your `criteria_evaluated` list stays what it
298
+ has always been — the dimensions you swept (injection, secrets, path
299
+ traversal). The two lists are not expected to match.
300
+
301
+ `safety` is where your findings land. It carries the highest floor in the
302
+ default rubric, and a single unresolved injection or leaked credential should
303
+ put your `safety` score below it.
304
+
305
+ **Omit any criterion you cannot assess rather than scoring it `0`.** A `0` says
306
+ you assessed it and it failed, which on a criterion with a floor blocks the
307
+ task by itself.
308
+ ```
309
+
310
+ - [ ] **Step 7: Run the tests to verify they pass**
311
+
312
+ Run: `bats test/bats/tribunal-judge-agents.bats`
313
+ Expected: PASS, 7/7.
314
+
315
+ - [ ] **Step 8: Prove the tests are falsifiable**
316
+
317
+ In a scratch (uncommitted) edit, delete the `"safety"` key from `tribunal-judge-adversarial.md`'s `criterion_scores`. Re-run: the "every judge agent scores safety" test must fail. Restore it. Then change one `criterion_scores` key in the security agent to `"injection"` and confirm the rubric-keys test fails. Restore. **Report both results in your report file.**
318
+
319
+ - [ ] **Step 9: Verify the whole suite still passes**
320
+
321
+ Run: `npm run test:ci`
322
+ Expected: exit 0. Read the exit code directly with `$?`, never through a pipe.
323
+
324
+ - [ ] **Step 10: Commit**
325
+
326
+ ```bash
327
+ git add package.json package-lock.json plugins/tribunal/agents/ test/bats/tribunal-judge-agents.bats
328
+ git commit -m "feat(tribunal): have judges score the rubric's criteria :straight_ruler:"
329
+ ```
330
+
331
+ ---
332
+
333
+ ### Task 2: `weighted_mean` becomes real
334
+
335
+ **Files:**
336
+ - Modify: `plugins/tribunal/scripts/lib/tribunal-aggregate.sh:1-59`
337
+ - Test: `test/bats/tribunal-aggregate.bats`
338
+
339
+ **Interfaces:**
340
+ - Consumes: the `criterion_scores` contract from Task 1.
341
+ - Produces: `tribunal_aggregate <method> <verdicts_json> [<rubric_json>]` — signature **unchanged**, third parameter now used instead of discarded. For `weighted_mean` with usable scores it echoes the weighted mean; otherwise it echoes the plain mean of `.score`, exactly as today.
342
+
343
+ - [ ] **Step 1: Write the failing tests**
344
+
345
+ Append to `test/bats/tribunal-aggregate.bats`:
346
+
347
+ ```bash
348
+ # Two criteria with deliberately unequal weights, so weighted_mean and mean
349
+ # cannot coincide. Judge A is strong on the heavy criterion, weak on the light
350
+ # one; judge B is the reverse.
351
+ RUBRIC_UNEQUAL='{"criteria":[{"name":"correctness","weight":0.9,"min_pass":0.7},{"name":"clarity","weight":0.1,"min_pass":0.5}]}'
352
+ SCORED='[
353
+ {"judge_id":"a","score":0.5,"criterion_scores":{"correctness":1.0,"clarity":0.0}},
354
+ {"judge_id":"b","score":0.5,"criterion_scores":{"correctness":1.0,"clarity":0.0}}
355
+ ]'
356
+
357
+ @test "weighted_mean differs from mean when weights are unequal" {
358
+ # mean of .score is 0.5 for both judges. The weighted mean is
359
+ # 0.9*1.0 + 0.1*0.0 = 0.9. If these come out equal, weights are still inert.
360
+ local w m
361
+ w=$(tribunal_aggregate "weighted_mean" "$SCORED" "$RUBRIC_UNEQUAL")
362
+ m=$(tribunal_aggregate "mean" "$SCORED" "$RUBRIC_UNEQUAL")
363
+ awk -v a="$w" -v b="$m" 'BEGIN { exit !(a != b) }' || return 1
364
+ awk -v a="$w" 'BEGIN { exit !(a > 0.89 && a < 0.91) }'
365
+ }
366
+
367
+ @test "weighted_mean averages judges within a criterion before weighting" {
368
+ local out
369
+ out=$(tribunal_aggregate "weighted_mean" '[
370
+ {"judge_id":"a","score":0.5,"criterion_scores":{"correctness":1.0,"clarity":1.0}},
371
+ {"judge_id":"b","score":0.5,"criterion_scores":{"correctness":0.0,"clarity":1.0}}
372
+ ]' "$RUBRIC_UNEQUAL")
373
+ # correctness mean 0.5, clarity mean 1.0 → 0.9*0.5 + 0.1*1.0 = 0.55
374
+ awk -v a="$out" 'BEGIN { exit !(a > 0.549 && a < 0.551) }'
375
+ }
376
+
377
+ @test "weighted_mean degrades to mean when no verdict carries criterion_scores" {
378
+ # Every verdict emitted before Task 1 shipped looks like this.
379
+ local out
380
+ out=$(tribunal_aggregate "weighted_mean" \
381
+ '[{"judge_id":"a","score":0.8},{"judge_id":"b","score":0.6}]' "$RUBRIC_UNEQUAL")
382
+ awk -v a="$out" 'BEGIN { exit !(a > 0.699 && a < 0.701) }'
383
+ }
384
+
385
+ @test "an absent criterion is skipped, not counted as zero" {
386
+ # clarity is absent everywhere. If absence read as 0 the answer would be
387
+ # 0.9*1.0 + 0.1*0.0 = 0.9. Skipping it renormalizes to 0.9/0.9 = 1.0.
388
+ local out
389
+ out=$(tribunal_aggregate "weighted_mean" \
390
+ '[{"judge_id":"a","score":0.5,"criterion_scores":{"correctness":1.0}}]' \
391
+ "$RUBRIC_UNEQUAL")
392
+ awk -v a="$out" 'BEGIN { exit !(a > 0.999 && a < 1.001) }'
393
+ }
394
+
395
+ @test "a criterion scored at zero is honored, not treated as absent" {
396
+ # The mirror of the previous test, and the one that catches a `// 0` fix
397
+ # that "passes" the absence test by accident.
398
+ local out
399
+ out=$(tribunal_aggregate "weighted_mean" \
400
+ '[{"judge_id":"a","score":0.5,"criterion_scores":{"correctness":0.0}}]' \
401
+ "$RUBRIC_UNEQUAL")
402
+ awk -v a="$out" 'BEGIN { exit !(a >= 0 && a < 0.001) }'
403
+ }
404
+
405
+ @test "weights that do not sum to 1.0 are normalized" {
406
+ # tribunal_rubric_validate rejects such a rubric, but librarian's loader
407
+ # validates nothing and hands its rubric straight through. Normalizing here
408
+ # means the two paths cannot disagree.
409
+ local out
410
+ out=$(tribunal_aggregate "weighted_mean" \
411
+ '[{"judge_id":"a","score":0.1,"criterion_scores":{"correctness":1.0,"clarity":0.0}}]' \
412
+ '{"criteria":[{"name":"correctness","weight":1.8,"min_pass":0.7},{"name":"clarity","weight":0.2,"min_pass":0.5}]}')
413
+ # 1.8*1.0 + 0.2*0.0 = 1.8, over a weight sum of 2.0 → 0.9
414
+ awk -v a="$out" 'BEGIN { exit !(a > 0.899 && a < 0.901) }'
415
+ }
416
+
417
+ @test "a hyphenated criterion name scores correctly" {
418
+ # A dotted jq path would be a COMPILE error here: exit 3, empty stdout,
419
+ # which awk reads as 0.
420
+ local out
421
+ out=$(tribunal_aggregate "weighted_mean" \
422
+ '[{"judge_id":"a","score":0.2,"criterion_scores":{"path-traversal":1.0}}]' \
423
+ '{"criteria":[{"name":"path-traversal","weight":1.0,"min_pass":0.5}]}')
424
+ awk -v a="$out" 'BEGIN { exit !(a > 0.999 && a < 1.001) }'
425
+ }
426
+
427
+ @test "weighted_mean falls back to mean when the rubric is absent" {
428
+ local out
429
+ out=$(tribunal_aggregate "weighted_mean" "$SCORED")
430
+ awk -v a="$out" 'BEGIN { exit !(a > 0.499 && a < 0.501) }'
431
+ }
432
+
433
+ @test "a non-number criterion score is ignored rather than poisoning the mean" {
434
+ local out
435
+ out=$(tribunal_aggregate "weighted_mean" \
436
+ '[{"judge_id":"a","score":0.5,"criterion_scores":{"correctness":1.0,"clarity":"n/a"}}]' \
437
+ "$RUBRIC_UNEQUAL")
438
+ awk -v a="$out" 'BEGIN { exit !(a > 0.999 && a < 1.001) }'
439
+ }
440
+ ```
441
+
442
+ - [ ] **Step 2: Run the tests to verify they fail**
443
+
444
+ Run: `bats test/bats/tribunal-aggregate.bats`
445
+ Expected: FAIL — the new weighted tests fail because `weighted_mean` currently shares the `mean` branch. The degrade and fallback tests pass already; that is correct, since they pin behavior that must survive.
446
+
447
+ - [ ] **Step 3: Implement the real weighted mean**
448
+
449
+ Replace the header comment block at `plugins/tribunal/scripts/lib/tribunal-aggregate.sh:18-23` with:
450
+
451
+ ```bash
452
+ # weighted_mean uses *rubric criterion weights*: average the judges' scores on
453
+ # each criterion, weight each criterion's mean, and normalize by the weights
454
+ # actually used. A criterion no judge scored contributes nothing and its weight
455
+ # is excluded from the denominator — absence is not a zero. When no criterion
456
+ # has any score (every verdict emitted before judges shipped criterion_scores),
457
+ # weighted_mean degrades to mean rather than collapsing to 0.
458
+ ```
459
+
460
+ Replace the function body's `_rubric` handling and the `mean|weighted_mean` branch:
461
+
462
+ ```bash
463
+ tribunal_aggregate() {
464
+ local method="${1:-mean}"
465
+ local verdicts="${2:-[]}"
466
+ local rubric="${3:-}"
467
+ [ -z "$rubric" ] && rubric='{}'
468
+
469
+ local count
470
+ count=$(printf '%s' "$verdicts" | jq 'length' 2>/dev/null) || count=0
471
+ [[ "$count" -eq 0 ]] && { printf '0'; return 0; }
472
+
473
+ case "$method" in
474
+ mean)
475
+ printf '%s' "$verdicts" | jq -r '[.[].score] | add / length'
476
+ ;;
477
+ weighted_mean)
478
+ local weighted
479
+ weighted=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
480
+ . as $v
481
+ | [ ($rubric.criteria // [])[]
482
+ | select((.name | type) == "string" and (.weight | type) == "number")
483
+ | . as $c
484
+ | ([ $v[]
485
+ | select((.criterion_scores | type) == "object")
486
+ | select(.criterion_scores | has($c.name))
487
+ | .criterion_scores[$c.name]
488
+ | select(type == "number") ]) as $scores
489
+ | select(($scores | length) > 0)
490
+ | { w: $c.weight, m: (($scores | add) / ($scores | length)) } ]
491
+ | (map(.w) | add) as $den
492
+ | if length == 0 or $den == null or $den <= 0 then empty
493
+ else (map(.w * .m) | add) / $den
494
+ end
495
+ ' 2>/dev/null)
496
+ if [ -n "$weighted" ]; then
497
+ printf '%s' "$weighted"
498
+ else
499
+ # No criterion carried a usable score — degrade to mean.
500
+ printf '%s' "$verdicts" | jq -r '[.[].score] | add / length'
501
+ fi
502
+ ;;
503
+ median)
504
+ ```
505
+
506
+ Leave `median`, `min`, and the unknown-method fallback exactly as they are.
507
+
508
+ - [ ] **Step 4: Run the tests to verify they pass**
509
+
510
+ Run: `bats test/bats/tribunal-aggregate.bats`
511
+ Expected: PASS, 19/19.
512
+
513
+ - [ ] **Step 5: Prove the guards are falsifiable**
514
+
515
+ Three scratch (uncommitted) edits, each reverted after checking:
516
+
517
+ 1. Change `select(.criterion_scores | has($c.name))` to `select(true)` and `.criterion_scores[$c.name]` to `(.criterion_scores[$c.name] // 0)`. The "absent criterion is skipped" test must fail. **This is the exact regression the `// 0` ban exists to prevent.**
518
+ 2. Change `.criterion_scores[$c.name]` to `.criterion_scores[$c.name] | tostring | tonumber` — the hyphenated-name test must still pass (proving the bracket lookup, not the name, is what matters). Then change the lookup to a dotted `.criterion_scores.correctness` and confirm the hyphenated test fails.
519
+ 3. Delete `/ $den` (dividing by nothing). The normalization test must fail.
520
+
521
+ **Report all three results in your report file.**
522
+
523
+ - [ ] **Step 6: Verify the whole suite**
524
+
525
+ Run: `npm run test:ci`
526
+ Expected: exit 0, read from `$?` directly.
527
+
528
+ - [ ] **Step 7: Commit**
529
+
530
+ ```bash
531
+ git add plugins/tribunal/scripts/lib/tribunal-aggregate.sh test/bats/tribunal-aggregate.bats
532
+ git commit -m "feat(tribunal): make weighted_mean weight something :straight_ruler:"
533
+ ```
534
+
535
+ ---
536
+
537
+ ### Task 3: `min_pass` becomes enforceable
538
+
539
+ **Files:**
540
+ - Modify: `plugins/tribunal/scripts/lib/tribunal-gate.sh:1-112`
541
+ - Modify: `plugins/tribunal/skills/tribunal/SKILL.md:117-124`
542
+ - Test: `test/bats/tribunal-gate.bats`
543
+
544
+ **Interfaces:**
545
+ - Consumes: the `criterion_scores` contract from Task 1.
546
+ - Produces: `tribunal_gate_decide <policy> <verdicts> <aggregated_score> <score_threshold> <meta> <dissent_score> <dissent_threshold> [<rubric_json>]` — an **eighth, optional, trailing** parameter. When a criterion's mean falls below its `min_pass`, it echoes `{"passed":false,"reason":"criterion_floor","failed_criterion":"<name>"}`. All existing reasons and their precedence are unchanged.
547
+
548
+ - [ ] **Step 1: Write the failing tests**
549
+
550
+ Append to `test/bats/tribunal-gate.bats`:
551
+
552
+ ```bash
553
+ RUBRIC_FLOOR='{"criteria":[{"name":"correctness","weight":0.5,"min_pass":0.7},{"name":"safety","weight":0.5,"min_pass":0.8}]}'
554
+
555
+ @test "a criterion below its floor blocks even when score and jury both pass" {
556
+ # This is the property that does not exist today: aggregate 0.82 clears the
557
+ # 0.75 threshold, both judges passed, and the gate blocks anyway.
558
+ local out
559
+ out=$(tribunal_gate_decide "majority" '[
560
+ {"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"correctness":0.9,"safety":0.3}},
561
+ {"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"correctness":0.9,"safety":0.3}}
562
+ ]' "0.82" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
563
+ printf '%s' "$out" | jq -e '.passed == false' >/dev/null || return 1
564
+ printf '%s' "$out" | jq -e '.reason == "criterion_floor"' >/dev/null || return 1
565
+ printf '%s' "$out" | jq -e '.failed_criterion == "safety"' >/dev/null
566
+ }
567
+
568
+ @test "a criterion at exactly its floor passes" {
569
+ local out
570
+ out=$(tribunal_gate_decide "majority" '[
571
+ {"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"correctness":0.7,"safety":0.8}},
572
+ {"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"correctness":0.7,"safety":0.8}}
573
+ ]' "0.82" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
574
+ printf '%s' "$out" | jq -e '.passed == true' >/dev/null
575
+ }
576
+
577
+ @test "absent criterion_scores never block" {
578
+ # Every verdict emitted before Task 1 shipped. Treating absence as violation
579
+ # would make every pre-upgrade judge fail every rubric carrying a floor.
580
+ local out
581
+ out=$(tribunal_gate_decide "majority" "$ALL_PASSED" "0.82" "0.75" \
582
+ "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
583
+ printf '%s' "$out" | jq -e '.passed == true' >/dev/null
584
+ }
585
+
586
+ @test "a criterion scored exactly zero does block" {
587
+ # The mirror of the previous test. A fix that conflates absent with zero
588
+ # passes one of these two and fails the other.
589
+ local out
590
+ out=$(tribunal_gate_decide "majority" '[
591
+ {"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"correctness":0.9,"safety":0.0}},
592
+ {"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"correctness":0.9,"safety":0.0}}
593
+ ]' "0.82" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
594
+ printf '%s' "$out" | jq -e '.reason == "criterion_floor"' >/dev/null || return 1
595
+ printf '%s' "$out" | jq -e '.failed_criterion == "safety"' >/dev/null
596
+ }
597
+
598
+ @test "a hyphenated criterion name gates correctly" {
599
+ local out
600
+ out=$(tribunal_gate_decide "majority" '[
601
+ {"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"path-traversal":0.1}},
602
+ {"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"path-traversal":0.1}}
603
+ ]' "0.82" "0.75" "$NO_META" "0.05" "0.25" \
604
+ '{"criteria":[{"name":"path-traversal","weight":1.0,"min_pass":0.5}]}')
605
+ printf '%s' "$out" | jq -e '.failed_criterion == "path-traversal"' >/dev/null
606
+ }
607
+
608
+ @test "low_score still wins over criterion_floor" {
609
+ # Precedence matters for the retry digest: if the aggregate missed the
610
+ # threshold, that is the more actionable thing to tell the Actor.
611
+ local out
612
+ out=$(tribunal_gate_decide "majority" '[
613
+ {"judge_id":"a","score":0.20,"passed":true,"criterion_scores":{"correctness":0.1,"safety":0.1}},
614
+ {"judge_id":"b","score":0.20,"passed":true,"criterion_scores":{"correctness":0.1,"safety":0.1}}
615
+ ]' "0.20" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
616
+ printf '%s' "$out" | jq -e '.reason == "low_score"' >/dev/null
617
+ }
618
+
619
+ @test "a floor on a criterion no judge scored is reported on stderr" {
620
+ # The adversarial-judge gap: safety carries the highest floor and appeared
621
+ # in no agent contract. Silently passing a floor nobody scored is this
622
+ # design's own failure mode one layer down.
623
+ run --separate-stderr tribunal_gate_decide "majority" '[
624
+ {"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"correctness":0.9}},
625
+ {"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"correctness":0.9}}
626
+ ]' "0.82" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR"
627
+ printf '%s' "$output" | jq -e '.passed == true' >/dev/null || return 1
628
+ local re='safety'
629
+ [[ "$stderr" =~ $re ]]
630
+ }
631
+
632
+ @test "no unscored-criterion warning when no judge scored anything" {
633
+ # The pre-upgrade fleet must not spew a warning on every single gate.
634
+ run --separate-stderr tribunal_gate_decide "majority" "$ALL_PASSED" "0.82" "0.75" \
635
+ "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR"
636
+ [ -z "$stderr" ]
637
+ }
638
+
639
+ @test "the gate still works with no rubric at all" {
640
+ local out
641
+ out=$(tribunal_gate_decide "majority" "$ALL_PASSED" "0.82" "0.75" "$NO_META" "0.05" "0.25")
642
+ printf '%s' "$out" | jq -e '.passed == true' >/dev/null
643
+ }
644
+ ```
645
+
646
+ - [ ] **Step 2: Run the tests to verify they fail**
647
+
648
+ Run: `bats test/bats/tribunal-gate.bats`
649
+ Expected: FAIL — the floor tests fail because no floor exists. The "absent never blocks", "no rubric" and "low_score wins" tests pass already, pinning behavior that must survive.
650
+
651
+ - [ ] **Step 3: Implement the floor**
652
+
653
+ In `plugins/tribunal/scripts/lib/tribunal-gate.sh`, extend the header comment's reason list (line 17) to:
654
+
655
+ ```bash
656
+ # reason is one of: low_score | meta_override | bias_detected | dissent_unresolved
657
+ # | criterion_floor (with failed_criterion naming the criterion)
658
+ ```
659
+
660
+ Add the eighth parameter after line 29:
661
+
662
+ ```bash
663
+ local dissent_threshold="${7:-0.25}"
664
+ local rubric="${8:-}"
665
+ [ -z "$rubric" ] && rubric='{}'
666
+ ```
667
+
668
+ Then insert this block immediately **after** the `score_ok` computation (currently line 93) and **before** the `if [[ "$jury_ok" -eq 0 && "$score_ok" -eq 0 ]]` check:
669
+
670
+ ```bash
671
+ # Per-criterion floors. A criterion whose mean across the judges that scored
672
+ # it falls below min_pass blocks regardless of the aggregate or the policy —
673
+ # that is the whole point of a floor.
674
+ #
675
+ # A criterion no judge scored is NOT a violation: absence is not a zero, and
676
+ # treating it as one would fail every verdict emitted before judges shipped
677
+ # criterion_scores. It is reported on stderr instead, because a floor nobody
678
+ # scores is a silent hole in the rubric.
679
+ local floor_failed unscored_floors any_scored
680
+ any_scored=$(printf '%s' "$verdicts" | jq -r '
681
+ [.[] | select((.criterion_scores | type) == "object")
682
+ | select((.criterion_scores | length) > 0)] | length > 0
683
+ ' 2>/dev/null) || any_scored="false"
684
+
685
+ floor_failed=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
686
+ . as $v
687
+ | [ ($rubric.criteria // [])[]
688
+ | select((.name | type) == "string" and (.min_pass | type) == "number")
689
+ | . as $c
690
+ | ([ $v[]
691
+ | select((.criterion_scores | type) == "object")
692
+ | select(.criterion_scores | has($c.name))
693
+ | .criterion_scores[$c.name]
694
+ | select(type == "number") ]) as $scores
695
+ | select(($scores | length) > 0)
696
+ | select((($scores | add) / ($scores | length)) < $c.min_pass)
697
+ | $c.name ]
698
+ | first // empty
699
+ ' 2>/dev/null) || floor_failed=""
700
+
701
+ if [[ "$any_scored" == "true" ]]; then
702
+ unscored_floors=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
703
+ . as $v
704
+ | [ ($rubric.criteria // [])[]
705
+ | select((.name | type) == "string" and (.min_pass | type) == "number")
706
+ | . as $c
707
+ | select([ $v[]
708
+ | select((.criterion_scores | type) == "object")
709
+ | select(.criterion_scores | has($c.name))
710
+ | .criterion_scores[$c.name]
711
+ | select(type == "number") ] | length == 0)
712
+ | $c.name ]
713
+ | join(", ")
714
+ ' 2>/dev/null) || unscored_floors=""
715
+ if [[ -n "$unscored_floors" ]]; then
716
+ printf 'tribunal-gate: no judge scored these criteria, so their min_pass floors did not apply: %s\n' \
717
+ "$unscored_floors" >&2
718
+ fi
719
+ fi
720
+ ```
721
+
722
+ Finally, change the blocking-reason selection (currently lines 100-111) so the floor sits between `low_score` and the jury reasons:
723
+
724
+ ```bash
725
+ # Pick the most informative blocking reason. low_score first: if the
726
+ # aggregate missed the threshold, that is the more actionable thing to tell
727
+ # the Actor than any single criterion.
728
+ if [[ "$score_ok" -ne 0 ]]; then
729
+ printf '{"passed":false,"reason":"low_score"}'
730
+ elif [[ -n "$floor_failed" ]]; then
731
+ printf '{"passed":false,"reason":"criterion_floor","failed_criterion":"%s"}' "$floor_failed"
732
+ elif [[ "$jury_ok" -ne 0 ]]; then
733
+ if [[ "$meta_override" == "reject" ]]; then
734
+ printf '{"passed":false,"reason":"meta_override"}'
735
+ else
736
+ printf '{"passed":false,"reason":"dissent_unresolved"}'
737
+ fi
738
+ else
739
+ printf '{"passed":true}'
740
+ fi
741
+ ```
742
+
743
+ and delete the now-redundant early `if [[ "$jury_ok" -eq 0 && "$score_ok" -eq 0 ]]` pass block at lines 95-98, since the `else` arm above covers it. **Both a floor failure and a jury failure now reach this chain, so the old two-condition early return would have let a floor failure through.**
744
+
745
+ - [ ] **Step 4: Run the tests to verify they pass**
746
+
747
+ Run: `bats test/bats/tribunal-gate.bats`
748
+ Expected: PASS, 19/19.
749
+
750
+ - [ ] **Step 5: Wire the reason through the orchestration walk**
751
+
752
+ In `plugins/tribunal/skills/tribunal/SKILL.md`, replace the step 7 code block (lines 118-121) with:
753
+
754
+ ```bash
755
+ policy=$(printf '%s' "$rubric" | jq -r '.gate_policy // "majority"')
756
+ gate=$(tribunal_gate_decide "$policy" "$verdicts" "$aggregated" "$threshold" "$meta" "$dissent" "$dissent_threshold" "$rubric")
757
+ ```
758
+
759
+ and replace the prose at line 122 with:
760
+
761
+ ```markdown
762
+ If `gate.passed == true`, emit `tribunal.gate.passed` with `final_score: aggregated` and break the loop with outcome `accepted`. Otherwise emit `tribunal.gate.blocked` with the `reason`, `will_retry: (iteration_number + 1 < max_iterations)`, and `retry_iteration_number` if retrying. **When `reason` is `criterion_floor`, copy `gate.failed_criterion` onto the event payload as `failed_criterion`** — without it the log records that a floor blocked but not which one. Persist `gate.json` either way.
763
+
764
+ A `criterion_floor` block means the aggregate *cleared* its threshold and one criterion still failed its floor. Say so when you report it: "blocked on `safety` (0.30 < 0.80) despite an overall 0.82" is actionable, "blocked" is not.
765
+ ```
766
+
767
+ - [ ] **Step 6: Prove the guards are falsifiable**
768
+
769
+ Two scratch (uncommitted) edits, each reverted after checking:
770
+
771
+ 1. Delete the `elif [[ -n "$floor_failed" ]]` arm. The three floor-blocking tests must fail.
772
+ 2. Change `select(($scores | length) > 0)` in the `floor_failed` query to `select(true)` — the "absent criterion_scores never block" test must fail, proving absence is genuinely excluded rather than accidentally passing.
773
+
774
+ **Report both results in your report file.**
775
+
776
+ - [ ] **Step 7: Verify the whole suite**
777
+
778
+ Run: `npm run test:ci`
779
+ Expected: exit 0, read from `$?` directly.
780
+
781
+ - [ ] **Step 8: Commit**
782
+
783
+ ```bash
784
+ git add plugins/tribunal/scripts/lib/tribunal-gate.sh plugins/tribunal/skills/tribunal/SKILL.md test/bats/tribunal-gate.bats
785
+ git commit -m "feat(tribunal): let a criterion floor block a passing score :octagonal_sign:"
786
+ ```
787
+
788
+ ---
789
+
790
+ ### Task 4: Librarian's aggregate and gate become rubric-aware
791
+
792
+ **Files:**
793
+ - Modify: `plugins/librarian/scripts/lib/librarian-lesson-judge.sh:12-81` and `:164-169`
794
+ - Test: `test/bats/librarian-lesson-judge.bats`
795
+
796
+ **Interfaces:**
797
+ - Consumes: the `criterion_scores` contract from Task 1.
798
+ - Produces:
799
+ - `librarian_lesson_aggregate <verdicts_json> [<rubric_json>]` — optional trailing rubric. With usable criterion scores it returns the weighted mean; otherwise the plain mean it returns today. Still returns 1 on an empty panel.
800
+ - `librarian_lesson_gate <gate_policy> <verdicts_json> <aggregate> <threshold> [<rubric_json>]` — optional trailing rubric. Adds the reason `criterion_floor` with a `failed_criterion` key to its echoed JSON. Existing reasons (`gate_passed`, `below_threshold`, `jury_not_majority`, `jury_not_unanimous`, `unknown_gate_policy`) are unchanged.
801
+
802
+ Librarian's reasons are written to on-disk proposal JSON, **not** to a schema-validated event, so this costs no schema version.
803
+
804
+ **Duplication with Tasks 2 and 3 is deliberate** — see Global Constraints and `plugins/librarian/docs/adr/002-agent-definitions-are-shared-assets.md`.
805
+
806
+ - [ ] **Step 1: Write the failing tests**
807
+
808
+ Append to `test/bats/librarian-lesson-judge.bats`:
809
+
810
+ ```bash
811
+ PUBLIC_RUBRIC='{"id":"lesson-promotion-public","criteria":[
812
+ {"name":"grounding","weight":0.32,"min_pass":0.7},
813
+ {"name":"scope_accuracy","weight":0.24,"min_pass":0.7},
814
+ {"name":"generality","weight":0.14,"min_pass":0.6},
815
+ {"name":"disclosure","weight":0.30,"min_pass":0.9}],
816
+ "score_threshold":0.75,"gate_policy":"majority"}'
817
+
818
+ @test "lesson aggregate weights criteria when scores are present" {
819
+ local w m verdicts
820
+ verdicts='[
821
+ {"judge_type":"standard","score":0.5,"passed":true,"criterion_scores":{"grounding":1.0,"scope_accuracy":1.0,"generality":1.0,"disclosure":0.0}},
822
+ {"judge_type":"adversarial","score":0.5,"passed":true,"criterion_scores":{"grounding":1.0,"scope_accuracy":1.0,"generality":1.0,"disclosure":0.0}}
823
+ ]'
824
+ w=$(librarian_lesson_aggregate "$verdicts" "$PUBLIC_RUBRIC")
825
+ m=$(librarian_lesson_aggregate "$verdicts")
826
+ # weighted: 0.32+0.24+0.14 = 0.70 over a weight sum of 1.0. Plain mean: 0.5.
827
+ awk -v a="$w" -v b="$m" 'BEGIN { exit !(a != b) }' || return 1
828
+ awk -v a="$w" 'BEGIN { exit !(a > 0.699 && a < 0.701) }'
829
+ }
830
+
831
+ @test "lesson aggregate degrades to the plain mean without criterion scores" {
832
+ local out
833
+ out=$(librarian_lesson_aggregate \
834
+ '[{"judge_type":"standard","score":0.8,"passed":true},
835
+ {"judge_type":"adversarial","score":0.6,"passed":true}]' "$PUBLIC_RUBRIC")
836
+ awk -v a="$out" 'BEGIN { exit !(a > 0.699 && a < 0.701) }'
837
+ }
838
+
839
+ @test "lesson aggregate still returns 1 on an empty panel" {
840
+ run librarian_lesson_aggregate '[]' "$PUBLIC_RUBRIC"
841
+ [ "$status" -eq 1 ]
842
+ }
843
+
844
+ @test "lesson aggregate normalizes weights that do not sum to 1.0" {
845
+ # librarian_lesson_rubric_get validates NOTHING, so a mis-summed rubric
846
+ # reaches this function where tribunal's validator would have refused it.
847
+ local out
848
+ out=$(librarian_lesson_aggregate \
849
+ '[{"judge_type":"standard","score":0.1,"passed":true,"criterion_scores":{"grounding":1.0,"disclosure":0.0}}]' \
850
+ '{"criteria":[{"name":"grounding","weight":1.8,"min_pass":0.7},{"name":"disclosure","weight":0.2,"min_pass":0.9}]}')
851
+ awk -v a="$out" 'BEGIN { exit !(a > 0.899 && a < 0.901) }'
852
+ }
853
+
854
+ @test "a low disclosure score blocks a public lesson under majority" {
855
+ # The reason this whole thread exists: disclosure's 0.9 floor blocks even
856
+ # though both judges passed and the aggregate clears 0.75.
857
+ local out
858
+ out=$(librarian_lesson_gate "majority" '[
859
+ {"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9,"disclosure":0.4}},
860
+ {"judge_type":"adversarial","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9,"disclosure":0.4}}
861
+ ]' "0.78" "0.75" "$PUBLIC_RUBRIC")
862
+ printf '%s' "$out" | jq -e '.passed == false' >/dev/null || return 1
863
+ printf '%s' "$out" | jq -e '.reason == "criterion_floor"' >/dev/null || return 1
864
+ printf '%s' "$out" | jq -e '.failed_criterion == "disclosure"' >/dev/null
865
+ }
866
+
867
+ @test "a low generality score no longer blocks a public lesson" {
868
+ # The behavioral difference from the unanimous stand-in. generality's floor
869
+ # is 0.6; 0.65 clears it, so a judge merely unhappy about generality does
870
+ # not veto a public lesson the way unanimous would have.
871
+ local out
872
+ out=$(librarian_lesson_gate "majority" '[
873
+ {"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.65,"disclosure":0.95}},
874
+ {"judge_type":"adversarial","score":0.85,"passed":true,"criterion_scores":{"grounding":0.9,"scope_accuracy":0.9,"generality":0.65,"disclosure":0.95}}
875
+ ]' "0.88" "0.75" "$PUBLIC_RUBRIC")
876
+ printf '%s' "$out" | jq -e '.passed == true' >/dev/null
877
+ }
878
+
879
+ @test "lesson gate: a verdict with no criterion_scores key at all never blocks" {
880
+ # Every verdict emitted before judges shipped criterion_scores. Note this
881
+ # case is caught by the OUTER type guard and never reaches has() — it does
882
+ # NOT pin the per-criterion absence guard. The next test does that.
883
+ local out
884
+ out=$(librarian_lesson_gate "majority" '[
885
+ {"judge_type":"standard","score":0.9,"passed":true},
886
+ {"judge_type":"adversarial","score":0.85,"passed":true}
887
+ ]' "0.88" "0.75" "$PUBLIC_RUBRIC")
888
+ printf '%s' "$out" | jq -e '.passed == true and .reason == "gate_passed"' >/dev/null
889
+ }
890
+
891
+ @test "lesson gate: scores present but one floored criterion omitted does not block" {
892
+ # THE test that pins the has() guard. These verdicts DO carry
893
+ # criterion_scores, so they survive the outer type guard and reach the
894
+ # per-criterion lookup — but `disclosure`, whose floor is 0.9, is absent.
895
+ # Substituting `// 0` for has() makes disclosure read as 0.0 and blocks.
896
+ #
897
+ # Written as its own test because the case above cannot fail when has() is
898
+ # deleted: its fixture is filtered one layer earlier. Two different absences
899
+ # sharing one test is how an outer guard silently stands in for an inner one
900
+ # — this project has hit that shape eight times.
901
+ local out
902
+ out=$(librarian_lesson_gate "majority" '[
903
+ {"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9}},
904
+ {"judge_type":"adversarial","score":0.85,"passed":true,"criterion_scores":{"grounding":0.9,"scope_accuracy":0.9,"generality":0.85}}
905
+ ]' "0.88" "0.75" "$PUBLIC_RUBRIC")
906
+ printf '%s' "$out" | jq -e '.passed == true' >/dev/null || return 1
907
+ printf '%s' "$out" | jq -e '.reason == "gate_passed"' >/dev/null
908
+ }
909
+
910
+ @test "lesson gate: a criterion scored zero does block" {
911
+ local out
912
+ out=$(librarian_lesson_gate "majority" '[
913
+ {"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"disclosure":0.0}},
914
+ {"judge_type":"adversarial","score":0.85,"passed":true,"criterion_scores":{"disclosure":0.0}}
915
+ ]' "0.88" "0.75" "$PUBLIC_RUBRIC")
916
+ printf '%s' "$out" | jq -e '.reason == "criterion_floor"' >/dev/null || return 1
917
+ printf '%s' "$out" | jq -e '.failed_criterion == "disclosure"' >/dev/null
918
+ }
919
+
920
+ @test "lesson gate: below_threshold still wins over criterion_floor" {
921
+ local out
922
+ out=$(librarian_lesson_gate "majority" '[
923
+ {"judge_type":"standard","score":0.2,"passed":true,"criterion_scores":{"disclosure":0.1}},
924
+ {"judge_type":"adversarial","score":0.2,"passed":true,"criterion_scores":{"disclosure":0.1}}
925
+ ]' "0.20" "0.75" "$PUBLIC_RUBRIC")
926
+ printf '%s' "$out" | jq -e '.reason == "below_threshold"' >/dev/null
927
+ }
928
+
929
+ @test "lesson gate: jury policy still wins over criterion_floor" {
930
+ local out
931
+ out=$(librarian_lesson_gate "majority" '[
932
+ {"judge_type":"standard","score":0.9,"passed":false,"criterion_scores":{"disclosure":0.1}},
933
+ {"judge_type":"adversarial","score":0.9,"passed":false,"criterion_scores":{"disclosure":0.1}}
934
+ ]' "0.90" "0.75" "$PUBLIC_RUBRIC")
935
+ printf '%s' "$out" | jq -e '.reason == "jury_not_majority"' >/dev/null
936
+ }
937
+
938
+ @test "lesson gate: works with no rubric at all" {
939
+ local out
940
+ out=$(librarian_lesson_gate "majority" '[
941
+ {"judge_type":"standard","score":0.9,"passed":true},
942
+ {"judge_type":"adversarial","score":0.85,"passed":true}
943
+ ]' "0.88" "0.75")
944
+ printf '%s' "$out" | jq -e '.passed == true' >/dev/null
945
+ }
946
+
947
+ @test "lesson gate: a hyphenated criterion name gates correctly" {
948
+ local out
949
+ out=$(librarian_lesson_gate "majority" '[
950
+ {"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"scope-accuracy":0.1}},
951
+ {"judge_type":"adversarial","score":0.9,"passed":true,"criterion_scores":{"scope-accuracy":0.1}}
952
+ ]' "0.90" "0.75" '{"criteria":[{"name":"scope-accuracy","weight":1.0,"min_pass":0.7}]}')
953
+ printf '%s' "$out" | jq -e '.failed_criterion == "scope-accuracy"' >/dev/null
954
+ }
955
+ ```
956
+
957
+ - [ ] **Step 2: Run the tests to verify they fail**
958
+
959
+ Run: `bats test/bats/librarian-lesson-judge.bats`
960
+ Expected: FAIL — the weighted and floor tests fail. The degrade, empty-panel, no-rubric, and precedence tests pass already, pinning behavior that must survive.
961
+
962
+ - [ ] **Step 3: Widen `librarian_lesson_aggregate`**
963
+
964
+ Replace `plugins/librarian/scripts/lib/librarian-lesson-judge.sh:17-29` with:
965
+
966
+ ```bash
967
+ # Aggregate the judges' scores. Returns 1 on an empty panel.
968
+ #
969
+ # With a rubric and per-criterion scores, this is a weighted mean: average the
970
+ # judges on each criterion, weight each criterion's mean, normalize by the
971
+ # weights actually used. Without them it is the plain mean it has always been.
972
+ #
973
+ # A criterion no judge scored contributes nothing and its weight leaves the
974
+ # denominator — absence is not a zero. Scoring it 0 instead would turn a judge
975
+ # that skipped a criterion into one that failed it.
976
+ #
977
+ # Usage: librarian_lesson_aggregate <verdicts_json> [<rubric_json>]
978
+ librarian_lesson_aggregate() {
979
+ local verdicts="${1:-[]}"
980
+ local rubric="${2:-}"
981
+ [ -z "$rubric" ] && rubric='{}'
982
+
983
+ local n
984
+ n=$(printf '%s' "$verdicts" | jq 'length' 2>/dev/null) || return 1
985
+ [[ -z "$n" || "$n" -eq 0 ]] && return 1
986
+
987
+ local weighted
988
+ weighted=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
989
+ . as $v
990
+ | [ ($rubric.criteria // [])[]
991
+ | select((.name | type) == "string" and (.weight | type) == "number")
992
+ | . as $c
993
+ | ([ $v[]
994
+ | select((.criterion_scores | type) == "object")
995
+ | select(.criterion_scores | has($c.name))
996
+ | .criterion_scores[$c.name]
997
+ | select(type == "number") ]) as $scores
998
+ | select(($scores | length) > 0)
999
+ | { w: $c.weight, m: (($scores | add) / ($scores | length)) } ]
1000
+ | (map(.w) | add) as $den
1001
+ | if length == 0 or $den == null or $den <= 0 then empty
1002
+ else (map(.w * .m) | add) / $den
1003
+ end
1004
+ ' 2>/dev/null)
1005
+ if [ -n "$weighted" ]; then
1006
+ printf '%s' "$weighted"
1007
+ return 0
1008
+ fi
1009
+
1010
+ printf '%s' "$verdicts" | jq -r '[.[].score] | add / length' 2>/dev/null || return 1
1011
+ }
1012
+ ```
1013
+
1014
+ - [ ] **Step 4: Widen `librarian_lesson_gate`**
1015
+
1016
+ Replace the doc comment and signature at `:31-41` with:
1017
+
1018
+ ```bash
1019
+ # Decide pass/block from the panel, the aggregate, and the rubric's floors.
1020
+ #
1021
+ # Echoes {"passed": bool, "reason": string} and, when a floor failed, a
1022
+ # "failed_criterion" naming it. Three conditions must hold: the jury clears its
1023
+ # policy, the aggregate clears the threshold, and no criterion sits below its
1024
+ # min_pass.
1025
+ #
1026
+ # The floor is what makes the public tier meaningfully stricter than org. Before
1027
+ # it existed, lesson-promotion-public declared gate_policy `unanimous` for that
1028
+ # purpose and it did nothing at all — see ecosystem-j74.
1029
+ #
1030
+ # Usage: librarian_lesson_gate <gate_policy> <verdicts_json> <aggregate> <threshold> [<rubric_json>]
1031
+ librarian_lesson_gate() {
1032
+ local policy="${1:-majority}"
1033
+ local verdicts="${2:-[]}"
1034
+ local aggregate="${3:-0}"
1035
+ local threshold="${4:-0.75}"
1036
+ local rubric="${5:-}"
1037
+ [ -z "$rubric" ] && rubric='{}'
1038
+ ```
1039
+
1040
+ Then replace the threshold check at `:74-80` with:
1041
+
1042
+ ```bash
1043
+ # awk for the float comparison: bash cannot compare decimals.
1044
+ if ! awk -v s="$aggregate" -v t="$threshold" 'BEGIN { exit !(s >= t) }'; then
1045
+ printf '{"passed":false,"reason":"below_threshold"}'
1046
+ return 0
1047
+ fi
1048
+
1049
+ # Per-criterion floors, checked last: the jury and the aggregate are both
1050
+ # more actionable to report, so they take precedence.
1051
+ local floor_failed
1052
+ floor_failed=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
1053
+ . as $v
1054
+ | [ ($rubric.criteria // [])[]
1055
+ | select((.name | type) == "string" and (.min_pass | type) == "number")
1056
+ | . as $c
1057
+ | ([ $v[]
1058
+ | select((.criterion_scores | type) == "object")
1059
+ | select(.criterion_scores | has($c.name))
1060
+ | .criterion_scores[$c.name]
1061
+ | select(type == "number") ]) as $scores
1062
+ | select(($scores | length) > 0)
1063
+ | select((($scores | add) / ($scores | length)) < $c.min_pass)
1064
+ | $c.name ]
1065
+ | first // empty
1066
+ ' 2>/dev/null) || floor_failed=""
1067
+
1068
+ if [[ -n "$floor_failed" ]]; then
1069
+ printf '{"passed":false,"reason":"criterion_floor","failed_criterion":"%s"}' "$floor_failed"
1070
+ return 0
1071
+ fi
1072
+
1073
+ printf '{"passed":true,"reason":"gate_passed"}'
1074
+ return 0
1075
+ }
1076
+ ```
1077
+
1078
+ - [ ] **Step 5: Thread the rubric through the caller**
1079
+
1080
+ In `librarian_lesson_judge`, replace lines 164-166 with:
1081
+
1082
+ ```bash
1083
+ local aggregate gate
1084
+ aggregate=$(librarian_lesson_aggregate "$verdicts" "$rubric") || return 2
1085
+ gate=$(librarian_lesson_gate "$policy" "$verdicts" "$aggregate" "$threshold" "$rubric") || {
1086
+ ```
1087
+
1088
+ - [ ] **Step 6: Run the tests to verify they pass**
1089
+
1090
+ Run: `bats test/bats/librarian-lesson-judge.bats`
1091
+ Expected: PASS, 55/55.
1092
+
1093
+ - [ ] **Step 7: Prove the guards are falsifiable**
1094
+
1095
+ Three scratch (uncommitted) edits, each reverted after checking:
1096
+
1097
+ 1. Delete the `if [[ -n "$floor_failed" ]]` block. The disclosure-floor tests must fail.
1098
+ 2. Change `.criterion_scores[$c.name]` to `(.criterion_scores[$c.name] // 0)` and drop the `has()` select. **`scores present but one floored criterion omitted` must fail, while `a criterion scored zero does block` still passes** — that pair is what discriminates absence from zero.
1099
+
1100
+ **Do not expect `a verdict with no criterion_scores key at all` to fail here.** It will not, and that is correct: its fixture is removed by the outer `select((.criterion_scores | type) == "object")` before the `has()` lookup is ever reached. Task 3 shipped with only that weaker test and the guard was consequently unpinned — verified, not hypothetical. If you find yourself editing the no-key test to make it fail, stop: you would be deleting the coverage of the pre-upgrade case to duplicate a test you already have.
1101
+ 3. Revert Step 5's rubric threading (pass no rubric from `librarian_lesson_judge`). Any end-to-end judging test that exercises a floor must fail. **If none does, that is a coverage gap — add one before finishing.**
1102
+
1103
+ **Report all three results in your report file.**
1104
+
1105
+ - [ ] **Step 8: Verify the whole suite**
1106
+
1107
+ Run: `npm run test:ci`
1108
+ Expected: exit 0, read from `$?` directly.
1109
+
1110
+ - [ ] **Step 9: Commit**
1111
+
1112
+ ```bash
1113
+ git add plugins/librarian/scripts/lib/librarian-lesson-judge.sh test/bats/librarian-lesson-judge.bats
1114
+ git commit -m "feat(librarian): weight and floor the lesson jury's criteria :straight_ruler:"
1115
+ ```
1116
+
1117
+ ---
1118
+
1119
+ ### Task 5: The public tier gets a real floor
1120
+
1121
+ **Files:**
1122
+ - Modify: `plugins/librarian/config.json` (`.librarian.lesson_judging.rubrics[1].gate_policy`)
1123
+ - Modify: `plugins/librarian/scripts/lib/librarian-lesson-rubric.sh:9-13`
1124
+ - Modify: `plugins/librarian/skills/librarian/SKILL.md:106-114`
1125
+ - Modify: `docs/superpowers/specs/2026-08-11-lesson-judging-design.md`
1126
+ - Modify: `docs/superpowers/specs/2026-08-14-criterion-scores-design.md`
1127
+ - Test: `test/bats/librarian-lesson-judge.bats`
1128
+
1129
+ **Interfaces:**
1130
+ - Consumes: `librarian_lesson_gate`'s rubric parameter and `criterion_floor` reason from Task 4.
1131
+ - Produces: nothing later tasks depend on. This is the terminal task.
1132
+
1133
+ Closes `ecosystem-j74`.
1134
+
1135
+ - [ ] **Step 1: Write the failing tests**
1136
+
1137
+ Append to `test/bats/librarian-lesson-judge.bats`:
1138
+
1139
+ ```bash
1140
+ @test "the public rubric no longer relies on unanimous" {
1141
+ # unanimous was a stand-in for the disclosure floor and never worked: at the
1142
+ # configured panel of 2, unanimous and majority are the same function for
1143
+ # every possible pass count. ecosystem-j74.
1144
+ local policy
1145
+ policy=$(jq -r '.librarian.lesson_judging.rubrics[]
1146
+ | select(.id == "lesson-promotion-public") | .gate_policy' \
1147
+ "${REPO_ROOT}/plugins/librarian/config.json")
1148
+ [ "$policy" = "majority" ]
1149
+ }
1150
+
1151
+ @test "the public rubric keeps disclosure's floor at 0.9" {
1152
+ # The floor is now the ONLY thing making public stricter than org. If this
1153
+ # drops, the public tier silently loses its protection entirely.
1154
+ local floor
1155
+ floor=$(jq -r '.librarian.lesson_judging.rubrics[]
1156
+ | select(.id == "lesson-promotion-public")
1157
+ | .criteria[] | select(.name == "disclosure") | .min_pass' \
1158
+ "${REPO_ROOT}/plugins/librarian/config.json")
1159
+ [ "$floor" = "0.9" ]
1160
+ }
1161
+
1162
+ @test "org and public rubrics differ by more than their gate policy" {
1163
+ # Both are `majority` now. If the criteria ever converge too, the two tiers
1164
+ # become indistinguishable and the public tier is inert again — the exact
1165
+ # shape of j74.
1166
+ local org_crit pub_crit
1167
+ org_crit=$(jq -c '[.librarian.lesson_judging.rubrics[]
1168
+ | select(.id == "lesson-promotion") | .criteria[].name] | sort' \
1169
+ "${REPO_ROOT}/plugins/librarian/config.json")
1170
+ pub_crit=$(jq -c '[.librarian.lesson_judging.rubrics[]
1171
+ | select(.id == "lesson-promotion-public") | .criteria[].name] | sort' \
1172
+ "${REPO_ROOT}/plugins/librarian/config.json")
1173
+ [ "$org_crit" != "$pub_crit" ]
1174
+ }
1175
+
1176
+ @test "the librarian walk tells judges to return criterion_scores" {
1177
+ # The rubric floors are unreachable unless the judges actually score them,
1178
+ # and this walk is where librarian's judges get their instructions.
1179
+ grep -q 'criterion_scores' "${REPO_ROOT}/plugins/librarian/skills/librarian/SKILL.md"
1180
+ }
1181
+ ```
1182
+
1183
+ - [ ] **Step 2: Run the tests to verify they fail**
1184
+
1185
+ Run: `bats test/bats/librarian-lesson-judge.bats`
1186
+ Expected: FAIL — the gate-policy test and the SKILL.md test fail. The floor and criteria-differ tests pass already, pinning what must not regress.
1187
+
1188
+ - [ ] **Step 2b: Carry `failed_criterion` into the on-disk verdict**
1189
+
1190
+ Found by Task 4's review. `librarian_lesson_gate` computes `failed_criterion`, but `librarian_lesson_judge` discards it — its verdict block extracts only `.passed` and `.reason`. So a rejected proposal records `"reason":"criterion_floor"` with no indication of *which* criterion, which is exactly the ambiguity `criterion_floor` was introduced to remove. Tribunal's walk copies the field onto its event for this reason; librarian had no counterpart.
1191
+
1192
+ First, the failing test — append to `test/bats/librarian-lesson-judge.bats`:
1193
+
1194
+ ```bash
1195
+ @test "a floor rejection records which criterion failed" {
1196
+ # "reason": "criterion_floor" without the criterion name is no more
1197
+ # actionable than "blocked" — the whole argument for a distinct reason
1198
+ # was naming the thing that failed.
1199
+ _seed_confirmed "floorname01" "public"
1200
+ local verdicts='[
1201
+ {"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9,"disclosure":0.4}},
1202
+ {"judge_type":"adversarial","score":0.85,"passed":true,"criterion_scores":{"grounding":0.9,"scope_accuracy":0.9,"generality":0.85,"disclosure":0.4}}
1203
+ ]'
1204
+ run librarian_lesson_judge "$PROJECT_KEY" "floorname01" "$verdicts"
1205
+ [ "$status" -eq 0 ] || return 1
1206
+ [ "$(_status_of floorname01)" = "rejected" ] || return 1
1207
+
1208
+ local path
1209
+ path="$(librarian_lessons_dir "$PROJECT_KEY")/proposals/floorname01.json"
1210
+ jq -e '.verdict.reason == "criterion_floor"' "$path" >/dev/null || return 1
1211
+ jq -e '.verdict.failed_criterion == "disclosure"' "$path" >/dev/null
1212
+ }
1213
+
1214
+ @test "an approved verdict carries no failed_criterion key" {
1215
+ # Absent rather than null: a key that is always present but usually empty
1216
+ # invites `// ""` at the read site, which is how this pipeline has
1217
+ # repeatedly lost the absent-vs-empty distinction.
1218
+ _seed_confirmed "floorname02" "public"
1219
+ local verdicts='[
1220
+ {"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9,"disclosure":0.95}},
1221
+ {"judge_type":"adversarial","score":0.9,"passed":true,"criterion_scores":{"grounding":0.9,"scope_accuracy":0.9,"generality":0.9,"disclosure":0.95}}
1222
+ ]'
1223
+ run librarian_lesson_judge "$PROJECT_KEY" "floorname02" "$verdicts"
1224
+ [ "$status" -eq 0 ] || return 1
1225
+
1226
+ local path
1227
+ path="$(librarian_lessons_dir "$PROJECT_KEY")/proposals/floorname02.json"
1228
+ jq -e '.verdict | has("failed_criterion") | not' "$path" >/dev/null
1229
+ }
1230
+ ```
1231
+
1232
+ These use the file's existing helpers — `_seed_confirmed <id> <visibility>` (line 138), `_status_of <id>` (line 154), and `$PROJECT_KEY` — verified against the neighboring end-to-end test at line 690. Do not invent new fixtures.
1233
+
1234
+ Run them: the first must fail on the missing `failed_criterion` key; the second passes already, since it pins behavior that must not regress.
1235
+
1236
+ Then, in `plugins/librarian/scripts/lib/librarian-lesson-judge.sh`, change the verdict construction inside `librarian_lesson_judge` from:
1237
+
1238
+ ```bash
1239
+ '{rubric_id: $r, gate_policy: $p, score_threshold: $th,
1240
+ aggregate_score: $ag, passed: $g.passed, reason: $g.reason,
1241
+ judges: $j}') || {
1242
+ ```
1243
+
1244
+ to:
1245
+
1246
+ ```bash
1247
+ '{rubric_id: $r, gate_policy: $p, score_threshold: $th,
1248
+ aggregate_score: $ag, passed: $g.passed, reason: $g.reason,
1249
+ judges: $j}
1250
+ + (if $g.failed_criterion then {failed_criterion: $g.failed_criterion} else {} end)') || {
1251
+ ```
1252
+
1253
+ The `+ (if … else {} end)` form keeps the key **absent** on a non-floor verdict rather than present-and-null. Re-run: both tests pass.
1254
+
1255
+ - [ ] **Step 3: Swap the public tier's gate policy**
1256
+
1257
+ In `plugins/librarian/config.json`, in the rubric with `"id": "lesson-promotion-public"`, change:
1258
+
1259
+ ```json
1260
+ "gate_policy": "unanimous",
1261
+ ```
1262
+
1263
+ to:
1264
+
1265
+ ```json
1266
+ "gate_policy": "majority",
1267
+ ```
1268
+
1269
+ Leave every criterion, weight, floor and `score_threshold` exactly as they are.
1270
+
1271
+ - [ ] **Step 4: Correct the rubric library's comment**
1272
+
1273
+ Replace `plugins/librarian/scripts/lib/librarian-lesson-rubric.sh:9-13` — which claims the weights are inert — with:
1274
+
1275
+ ```bash
1276
+ # The per-criterion weights and min_pass floors are LIVE as of ecosystem-pht:
1277
+ # librarian_lesson_aggregate weights them and librarian_lesson_gate blocks on
1278
+ # any criterion below its floor.
1279
+ #
1280
+ # `disclosure` at min_pass 0.9 is what makes the public tier stricter than org.
1281
+ # It replaces gate_policy `unanimous`, which was intended as a stand-in for
1282
+ # exactly this and turned out to be a no-op: at the configured two-judge panel,
1283
+ # `unanimous` and `majority` agree on every possible pass count. See
1284
+ # ecosystem-j74. Changing judge_types without re-reading that bead is how the
1285
+ # hole reopens.
1286
+ ```
1287
+
1288
+ - [ ] **Step 5: Tell librarian's judges to score the criteria**
1289
+
1290
+ In `plugins/librarian/skills/librarian/SKILL.md`, replace the bullet at lines 112-114 with:
1291
+
1292
+ ```markdown
1293
+ - Each judge returns a JSON object with `score`, `passed`, `judge_type`,
1294
+ `feedback_summary`, and `criterion_scores` — a map from **each rubric
1295
+ criterion name you gave it** to a score in `[0,1]`. Tell each judge
1296
+ explicitly to omit any criterion it cannot assess rather than scoring it
1297
+ `0`: a `0` on `disclosure` blocks a public lesson by itself, while an
1298
+ omission does not. Collect both verdicts into a JSON array **verbatim** —
1299
+ never summarize or reconstruct a judge's verdict.
1300
+ ```
1301
+
1302
+ - [ ] **Step 6: Correct the 4z8.3 spec**
1303
+
1304
+ In `docs/superpowers/specs/2026-08-11-lesson-judging-design.md`, find the passage describing `gate_policy: unanimous` as the public tier's protection and replace its claim with:
1305
+
1306
+ ```markdown
1307
+ **Correction (2026-08-14).** This section described `unanimous` as delivering
1308
+ "a single judge's objection cannot be outvoted." It never did. Both rubrics
1309
+ declare `judge_types: ["standard", "adversarial"]` — a panel of two — and at
1310
+ panel size two `unanimous` (`passed == count`) and `majority`
1311
+ (`passed * 2 > count`) return the same answer for all three possible pass
1312
+ counts. They diverge only at three judges or more, and `librarian_lesson_judge`
1313
+ refuses any panel whose judge-type multiset does not match the rubric's, so a
1314
+ third judge never reaches the gate.
1315
+
1316
+ The public tier was never stricter than the org tier. `ecosystem-pht` replaced
1317
+ the stand-in with `disclosure`'s real `min_pass` floor and the policy is now
1318
+ `majority`. Tracked as `ecosystem-j74`.
1319
+ ```
1320
+
1321
+ - [ ] **Step 7: Correct the criterion-scores spec**
1322
+
1323
+ In `docs/superpowers/specs/2026-08-14-criterion-scores-design.md`, the "Judges emit it" section says `tribunal-judge-standard` and `-security` already receive the rubric. Only `-standard` does. Replace that sentence with:
1324
+
1325
+ ```markdown
1326
+ `tribunal-judge-standard` already receives the rubric with its criteria and is
1327
+ told to "score each criterion in [0,1]". It has nowhere to put the result; its
1328
+ output contract gains `criterion_scores`.
1329
+
1330
+ **Neither `tribunal-judge-adversarial` nor `tribunal-judge-security` has a
1331
+ rubric section at all.** Both report `criteria_evaluated` lists drawn from their
1332
+ own investigative lenses — `edge-cases`, `concurrency`, `idempotency` for one;
1333
+ `injection`, `secrets`, `path-traversal` for the other — disjoint from every
1334
+ rubric in the repo.
1335
+ ```
1336
+
1337
+ - [ ] **Step 8: Run the tests to verify they pass**
1338
+
1339
+ Run: `bats test/bats/librarian-lesson-judge.bats`
1340
+ Expected: PASS, 59/59.
1341
+
1342
+ - [ ] **Step 9: Prove the swap actually changed behavior**
1343
+
1344
+ The gate-policy test only reads config. Confirm the swap changes an outcome: temporarily set `gate_policy` back to `unanimous` in a scratch edit and re-run the two public-tier gate tests from Task 4. **They must still pass** — because at panel 2 the policies are identical, which is the whole finding. Record that result in your report file as direct evidence that the floor, not the policy, is doing the work now.
1345
+
1346
+ - [ ] **Step 10: Verify the whole suite**
1347
+
1348
+ Run: `npm run test:ci`
1349
+ Expected: exit 0, read from `$?` directly.
1350
+
1351
+ - [ ] **Step 11: Commit**
1352
+
1353
+ ```bash
1354
+ git add plugins/librarian/config.json plugins/librarian/scripts/lib/librarian-lesson-rubric.sh plugins/librarian/skills/librarian/SKILL.md docs/superpowers/specs/2026-08-11-lesson-judging-design.md docs/superpowers/specs/2026-08-14-criterion-scores-design.md test/bats/librarian-lesson-judge.bats
1355
+ git commit -m "feat(librarian): give the public tier a floor that actually holds :closed_lock_with_key:"
1356
+ ```
1357
+
1358
+ ---
1359
+
1360
+ ## After the tasks
1361
+
1362
+ - Close `ecosystem-pht` and `ecosystem-j74`.
1363
+ - Open a PR into `main` per the repo's PR-only rule. The PR body should lead with the j74 finding — a reviewer who does not know the `unanimous` stand-in was inert will read the `majority` swap as a loosening rather than a tightening.
1364
+ - Not in scope, deliberately: retiring `criteria_evaluated`; any new judge type; widening `judge_types` to three (rejected — it buys a property this plan delivers anyway, at the cost of a third Opus judge per public candidate); and the `TODO(ONL-6)` hard-fail on type/schema divergence, which is the schema repo's own cleanup.
1365
+
1366
+ ## Self-review
1367
+
1368
+ **Spec coverage.** Payload consumption → Tasks 2, 3, 4. Judges emit it → Task 1. `weighted_mean` real → Task 2. Normalize by weight sum → Tasks 2 and 4. Degrade to mean → Tasks 2 and 4. `min_pass` enforceable → Tasks 3 and 4. Absent must not block → Tasks 3 and 4. Public tier's real floor → Task 5. Doc corrections → Task 5. Dependency bump → Task 1. Every test the spec calls for maps to a step, including the two it gained in correction (hyphenated names, unscored floors).
1369
+
1370
+ **One gap the spec left open, resolved here:** it never said whether judges score *rubric* criteria or their own lenses. They score rubric criteria; `criteria_evaluated` keeps its present meaning. Without that decision `criterion_scores` keys would never match a floor.
1371
+
1372
+ **Type consistency.** `criterion_scores` is an object of name → number everywhere. `failed_criterion` is a string, present only alongside `reason: "criterion_floor"`, in both plugins. Both new parameters are optional and trailing: `tribunal_aggregate` position 3 (existing, now used), `tribunal_gate_decide` position 8 (new), `librarian_lesson_aggregate` position 2 (new), `librarian_lesson_gate` position 5 (new).
1373
+
1374
+ **Precedence differs between the plugins, deliberately.** Tribunal: `low_score` → `criterion_floor` → jury reasons. Librarian: jury → `below_threshold` → `criterion_floor`. Each preserves its own existing order and appends the floor where that order allows; unifying them would change shipped behavior in one of the two.