@onlooker-community/ecosystem 0.34.1 → 0.43.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude-plugin/plugin.json +1 -1
- package/.github/workflows/release.yml +14 -14
- package/.release-please-manifest.json +4 -4
- package/AGENTS.md +5 -0
- package/CHANGELOG.md +168 -0
- package/CLAUDE.md +5 -0
- package/docs/lesson-promotion-pipeline.md +6 -6
- package/docs/superpowers/plans/2026-08-10-lesson-confirmation.md +1273 -0
- package/docs/superpowers/plans/2026-08-11-lesson-judging.md +1232 -0
- package/docs/superpowers/plans/2026-08-11-lesson-unconfirm.md +458 -0
- package/docs/superpowers/plans/2026-08-12-author-key.md +567 -0
- package/docs/superpowers/plans/2026-08-13-approved-pool.md +747 -0
- package/docs/superpowers/plans/2026-08-13-librarian-cluster.md +392 -0
- package/docs/superpowers/plans/2026-08-14-criterion-scores-consumers.md +1374 -0
- package/docs/superpowers/plans/2026-08-14-criterion-scores-schema.md +277 -0
- package/docs/superpowers/plans/2026-08-16-cartographer-undocumented-entity.md +741 -0
- package/docs/superpowers/specs/2026-08-10-lesson-confirmation-design.md +213 -0
- package/docs/superpowers/specs/2026-08-11-lesson-judging-design.md +267 -0
- package/docs/superpowers/specs/2026-08-11-lesson-unconfirm-design.md +152 -0
- package/docs/superpowers/specs/2026-08-12-author-key-design.md +317 -0
- package/docs/superpowers/specs/2026-08-13-approved-pool-design.md +232 -0
- package/docs/superpowers/specs/2026-08-14-criterion-scores-design.md +254 -0
- package/docs/superpowers/specs/2026-08-16-cartographer-undocumented-entity-design.md +249 -0
- package/package.json +2 -2
- package/plugins/cartographer/.claude-plugin/plugin.json +1 -1
- package/plugins/cartographer/CHANGELOG.md +54 -0
- package/plugins/cartographer/README.md +56 -1
- package/plugins/cartographer/config.json +7 -1
- package/plugins/cartographer/scripts/hooks/cartographer-post-write.sh +4 -4
- package/plugins/cartographer/scripts/hooks/cartographer-session-start.sh +7 -4
- package/plugins/cartographer/scripts/lib/cartographer-collect.sh +8 -1
- package/plugins/cartographer/scripts/lib/cartographer-config.sh +26 -0
- package/plugins/cartographer/scripts/lib/cartographer-events.sh +90 -0
- package/plugins/cartographer/scripts/lib/cartographer-filter.sh +105 -0
- package/plugins/cartographer/scripts/lib/cartographer-omission.sh +130 -0
- package/plugins/cartographer/scripts/lib/cartographer-resolve.sh +132 -0
- package/plugins/cartographer/scripts/run-audit.sh +136 -53
- package/plugins/cartographer/skills/cartographer/SKILL.md +33 -9
- package/plugins/librarian/.claude-plugin/plugin.json +1 -1
- package/plugins/librarian/CHANGELOG.md +64 -0
- package/plugins/librarian/config.json +30 -1
- package/plugins/librarian/docs/adr/002-agent-definitions-are-shared-assets.md +82 -0
- package/plugins/librarian/scripts/hooks/librarian-session-end.sh +28 -0
- package/plugins/librarian/scripts/hooks/librarian-session-start.sh +42 -12
- package/plugins/librarian/scripts/lib/librarian-author-key.sh +279 -0
- package/plugins/librarian/scripts/lib/librarian-cli.sh +344 -1
- package/plugins/librarian/scripts/lib/librarian-lesson-judge.sh +349 -0
- package/plugins/librarian/scripts/lib/librarian-lesson-promote.sh +225 -0
- package/plugins/librarian/scripts/lib/librarian-lesson-review.sh +312 -0
- package/plugins/librarian/scripts/lib/librarian-lesson-rubric.sh +56 -0
- package/plugins/librarian/scripts/lib/librarian-lesson-storage.sh +82 -13
- package/plugins/librarian/scripts/lib/librarian-lesson-transform.sh +6 -1
- package/plugins/librarian/scripts/lib/librarian-lesson-validate.sh +131 -62
- package/plugins/librarian/skills/librarian/SKILL.md +98 -1
- package/plugins/tribunal/.claude-plugin/plugin.json +1 -1
- package/plugins/tribunal/CHANGELOG.md +56 -0
- package/plugins/tribunal/agents/tribunal-judge-adversarial.md +34 -0
- package/plugins/tribunal/agents/tribunal-judge-security.md +33 -0
- package/plugins/tribunal/agents/tribunal-judge-standard.md +12 -0
- package/plugins/tribunal/config.json +1 -0
- package/plugins/tribunal/scripts/lib/tribunal-aggregate.sh +154 -11
- package/plugins/tribunal/scripts/lib/tribunal-gate.sh +193 -15
- package/plugins/tribunal/skills/tribunal/SKILL.md +5 -3
- package/test/bats/cartographer-config.bats +43 -0
- package/test/bats/cartographer-events.bats +267 -0
- package/test/bats/cartographer-filter.bats +169 -0
- package/test/bats/cartographer-omission.bats +202 -0
- package/test/bats/cartographer-resolve.bats +287 -0
- package/test/bats/cartographer-run-audit.bats +325 -0
- package/test/bats/librarian-author-key.bats +477 -0
- package/test/bats/librarian-lesson-judge.bats +884 -0
- package/test/bats/librarian-lesson-promote.bats +552 -0
- package/test/bats/librarian-lesson-review.bats +900 -0
- package/test/bats/librarian-lesson-transform.bats +17 -3
- package/test/bats/librarian-session-end.bats +161 -0
- package/test/bats/librarian-session-start.bats +34 -0
- package/test/bats/tribunal-aggregate.bats +354 -1
- package/test/bats/tribunal-gate.bats +631 -0
- package/test/bats/tribunal-judge-agents.bats +119 -0
- package/test/node/lesson-validate-agreement.test.mjs +35 -6
|
@@ -0,0 +1,1374 @@
|
|
|
1
|
+
# Per-Criterion Scores — Consumer Implementation Plan
|
|
2
|
+
|
|
3
|
+
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
|
4
|
+
|
|
5
|
+
**Goal:** Make tribunal's and librarian's rubric `weight` and `min_pass` values actually change outcomes, so a criterion can act as a floor that a strong weighted mean cannot average away.
|
|
6
|
+
|
|
7
|
+
**Architecture:** Judges gain a `criterion_scores` map on their output contract, keyed by **rubric** criterion name. Tribunal's `tribunal_aggregate` computes a real weighted mean from those scores and its gate blocks on any criterion below its `min_pass`, emitting the new `criterion_floor` reason. Librarian's parallel implementations gain the same two capabilities through widened signatures, which lets the public lesson tier trade its inert `unanimous` stand-in for a real `disclosure` floor.
|
|
8
|
+
|
|
9
|
+
**Tech Stack:** bash 3.2-compatible shell, `jq`, `awk`, bats, `@onlooker-community/schema` 2.12.0.
|
|
10
|
+
|
|
11
|
+
## Global Constraints
|
|
12
|
+
|
|
13
|
+
- **`@onlooker-community/schema` must be at `^2.12.0`.** 2.12.0 adds optional `criterion_scores` on `TribunalVerdictPayload` and the `criterion_floor` value in `tribunal.gate.blocked`'s `reason` enum plus optional `failed_criterion`. Task 1 is blocked until that version publishes.
|
|
14
|
+
- **Absent must never read as zero.** A judge that did not score a criterion must never be treated as one that scored `0.0`. Use `has()` or an explicit type test. **Specifically banned: defaulting a criterion score with `// 0`.** (Defaulting a missing *array* with `// []` is fine and appears in this plan — the ban is on scores.)
|
|
15
|
+
- **Look criterion names up as `.criterion_scores[$name]` with `--arg`/`$c.name`, never as a dotted path.** `path-traversal` and `edge-cases` already ship in agent contracts. `jq '.criterion_scores.path-traversal'` is a **compile error**: `jq` exits 3 with empty stdout, which `awk` then reads as `0`, failing toward blocking on a name the rubric author is entitled to write.
|
|
16
|
+
- **A floor on a criterion that no judge scored must be surfaced, never silently passed.** That is this design's own failure mode one layer down.
|
|
17
|
+
- **Tribunal and librarian each implement their own aggregate and gate. Do not unify them.** `plugins/librarian/docs/adr/002-agent-definitions-are-shared-assets.md` licenses reusing tribunal's *agent definitions* but rules out sourcing its bash. The near-duplicate jq between Tasks 2/3 and Task 4 is **plan-mandated**, not an oversight — a reviewer must not flag it as a DRY violation.
|
|
18
|
+
- **macOS bash 3.2:** a failing **non-final** `[[ ]]` does **not** fail a bats test. Use `[ ]` for assertions, or append `|| return 1`. A bats `[[ =~ ]]` regex must be an **unquoted variable**.
|
|
19
|
+
- **Use `run --separate-stderr`** whenever a test asserts empty stdout alongside a stderr message. Plain `run` merges them and the assertion becomes unsatisfiable.
|
|
20
|
+
- **Every test must be falsifiable.** Before committing a task, delete the guard under test and confirm a test fails. The recurring defect in this epic — seven-plus instances — is a *downstream* guard making an *upstream* guard's test pass whether or not the upstream guard exists.
|
|
21
|
+
- All hooks are bash; runtime artifacts live under `$ONLOOKER_DIR`, never a hardcoded `~/.onlooker`; ULIDs via each plugin's own `*-ulid.sh`; config defaults in `config.json` with overrides per ADR-004; the emitter fails open per ADR-005.
|
|
22
|
+
- Tests are bats under an isolated temp home per `.claude/skills/writing-tests`. Run `npm run test:ci` before the PR.
|
|
23
|
+
- **PR-only. Never push to `main`.** Branch from `feat/criterion-scores`, which already carries the corrected spec.
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## Background the tasks assume
|
|
28
|
+
|
|
29
|
+
Verified against the repo on 2026-08-14. Trust this section over the spec's prose — the spec was corrected today precisely because its descriptions of code were wrong, and two of its claims were **still** wrong after that correction (noted below).
|
|
30
|
+
|
|
31
|
+
**Tribunal's default rubric** (`plugins/tribunal/config.json`, `.tribunal.rubric.builtins[0]`):
|
|
32
|
+
|
|
33
|
+
| criterion | weight | min_pass |
|
|
34
|
+
|---|---|---|
|
|
35
|
+
| correctness | 0.4 | 0.7 |
|
|
36
|
+
| completeness | 0.3 | 0.7 |
|
|
37
|
+
| safety | 0.2 | **0.8** |
|
|
38
|
+
| clarity | 0.1 | 0.5 |
|
|
39
|
+
|
|
40
|
+
`judge_types: ["standard","adversarial"]`, `gate_policy: "majority"`, `aggregation_method: "weighted_mean"`, `score_threshold: 0.75`.
|
|
41
|
+
|
|
42
|
+
**Librarian's rubrics** (`plugins/librarian/config.json`, `.librarian.lesson_judging.rubrics` — an **array**, looked up by `.id`):
|
|
43
|
+
|
|
44
|
+
- `lesson-promotion` — grounding 0.45/0.7, scope_accuracy 0.35/0.7, generality 0.20/0.6; `gate_policy: majority`
|
|
45
|
+
- `lesson-promotion-public` — grounding 0.32/0.7, scope_accuracy 0.24/0.7, generality 0.14/0.6, **disclosure 0.30/0.9**; `gate_policy: unanimous`
|
|
46
|
+
|
|
47
|
+
Both declare `judge_types: ["standard","adversarial"]`.
|
|
48
|
+
|
|
49
|
+
**The three judge agents report their own investigative lenses, not rubric criteria:**
|
|
50
|
+
|
|
51
|
+
| agent | `criteria_evaluated` example | has a rubric section? |
|
|
52
|
+
|---|---|---|
|
|
53
|
+
| `tribunal-judge-standard.md` | `["correctness","completeness","clarity"]` | yes |
|
|
54
|
+
| `tribunal-judge-adversarial.md` | `["edge-cases","concurrency","idempotency"]` | **no** |
|
|
55
|
+
| `tribunal-judge-security.md` | `["injection","secrets","path-traversal"]` | **no** |
|
|
56
|
+
|
|
57
|
+
**Two corrections to the corrected spec.** It says `-security` already receives the rubric — it does not; the gap is two agents. And it implies `criteria_evaluated` is rubric-derived — it is not, for any of the three. Task 5 fixes the spec text.
|
|
58
|
+
|
|
59
|
+
**Consequence:** on the default panel, `safety` — the criterion with the highest floor — appears in **no** judge's output. Its floor would silently never fire. The resolution this plan adopts:
|
|
60
|
+
|
|
61
|
+
> `criteria_evaluated` keeps its present meaning — the agent's own investigative lenses — and is left **unchanged**. `criterion_scores` is a **separate** map keyed by **rubric** criterion names. Every judge scores every rubric criterion it was given; its lens list is how it got there.
|
|
62
|
+
|
|
63
|
+
**Callers.** `tribunal_aggregate` and `tribunal_gate_decide` are called only from `plugins/tribunal/skills/tribunal/SKILL.md` (steps 5 and 7). `librarian_lesson_aggregate` and `librarian_lesson_gate` are called only from `librarian_lesson_judge` in the same file. Both new parameters are **optional and trailing**, so no caller breaks.
|
|
64
|
+
|
|
65
|
+
---
|
|
66
|
+
|
|
67
|
+
## File structure
|
|
68
|
+
|
|
69
|
+
| File | Responsibility | Task |
|
|
70
|
+
|---|---|---|
|
|
71
|
+
| `package.json` | schema dependency floor | 1 |
|
|
72
|
+
| `plugins/tribunal/agents/tribunal-judge-standard.md` | output contract gains `criterion_scores` | 1 |
|
|
73
|
+
| `plugins/tribunal/agents/tribunal-judge-adversarial.md` | gains a rubric section + `criterion_scores` | 1 |
|
|
74
|
+
| `plugins/tribunal/agents/tribunal-judge-security.md` | gains a rubric section + `criterion_scores` | 1 |
|
|
75
|
+
| `test/bats/tribunal-judge-agents.bats` | **new** — contract tests over the shipped agent files | 1 |
|
|
76
|
+
| `plugins/tribunal/scripts/lib/tribunal-aggregate.sh` | real `weighted_mean` | 2 |
|
|
77
|
+
| `test/bats/tribunal-aggregate.bats` | aggregation tests (10 existing) | 2 |
|
|
78
|
+
| `plugins/tribunal/scripts/lib/tribunal-gate.sh` | `min_pass` floor, `criterion_floor` reason | 3 |
|
|
79
|
+
| `test/bats/tribunal-gate.bats` | gate tests (10 existing) | 3 |
|
|
80
|
+
| `plugins/tribunal/skills/tribunal/SKILL.md` | pass the rubric; emit `failed_criterion` | 3 |
|
|
81
|
+
| `plugins/librarian/scripts/lib/librarian-lesson-judge.sh` | rubric-aware aggregate + gate | 4 |
|
|
82
|
+
| `test/bats/librarian-lesson-judge.bats` | judging tests (43 existing) | 4 |
|
|
83
|
+
| `plugins/librarian/config.json` | public tier `unanimous` → `majority` | 5 |
|
|
84
|
+
| `plugins/librarian/scripts/lib/librarian-lesson-rubric.sh` | the "inert" comment is no longer true | 5 |
|
|
85
|
+
| `plugins/librarian/skills/librarian/SKILL.md` | judges must return `criterion_scores` | 5 |
|
|
86
|
+
| `docs/superpowers/specs/2026-08-11-lesson-judging-design.md` | the `unanimous` stand-in never worked | 5 |
|
|
87
|
+
| `docs/superpowers/specs/2026-08-14-criterion-scores-design.md` | `-security` correction | 5 |
|
|
88
|
+
|
|
89
|
+
---
|
|
90
|
+
|
|
91
|
+
### Task 1: Judges emit `criterion_scores`
|
|
92
|
+
|
|
93
|
+
**Files:**
|
|
94
|
+
- Modify: `package.json` (the `@onlooker-community/schema` dependency)
|
|
95
|
+
- Modify: `plugins/tribunal/agents/tribunal-judge-standard.md:26-47`
|
|
96
|
+
- Modify: `plugins/tribunal/agents/tribunal-judge-adversarial.md:34-51`
|
|
97
|
+
- Modify: `plugins/tribunal/agents/tribunal-judge-security.md:30-47`
|
|
98
|
+
- Test: `test/bats/tribunal-judge-agents.bats` (new file)
|
|
99
|
+
|
|
100
|
+
**Interfaces:**
|
|
101
|
+
- Consumes: nothing from earlier tasks.
|
|
102
|
+
- Produces: the judge output contract every later task assumes —
|
|
103
|
+
`criterion_scores` is an **object** mapping a **rubric criterion name** to a number in `[0,1]`. It is **optional**: a judge that cannot score a criterion omits that key rather than sending `0`. `criteria_evaluated` is unchanged and remains the agent's own lens list.
|
|
104
|
+
|
|
105
|
+
**BLOCKED UNTIL** `@onlooker-community/schema` 2.12.0 is published to npm. Check with `npm view @onlooker-community/schema version`. If it still reports `2.11.0`, stop and report BLOCKED — do not hand-edit a version that cannot install.
|
|
106
|
+
|
|
107
|
+
- [ ] **Step 1: Write the failing contract test**
|
|
108
|
+
|
|
109
|
+
Create `test/bats/tribunal-judge-agents.bats`:
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
#!/usr/bin/env bats
|
|
113
|
+
|
|
114
|
+
setup() {
|
|
115
|
+
source "${BATS_TEST_DIRNAME}/../helpers/setup.bash"
|
|
116
|
+
setup_test_env
|
|
117
|
+
|
|
118
|
+
AGENTS_DIR="${REPO_ROOT}/plugins/tribunal/agents"
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
# Extract the first fenced ```json block from an agent definition.
|
|
122
|
+
_agent_json() {
|
|
123
|
+
awk '/^```json/ { f = 1; next } /^```/ { if (f) exit } f' "$1"
|
|
124
|
+
}
|
|
125
|
+
|
|
126
|
+
@test "every judge agent's example verdict carries criterion_scores" {
|
|
127
|
+
local agent json
|
|
128
|
+
for agent in standard adversarial security; do
|
|
129
|
+
json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-${agent}.md")
|
|
130
|
+
[ -n "$json" ] || return 1
|
|
131
|
+
printf '%s' "$json" | jq -e 'has("criterion_scores")' >/dev/null || return 1
|
|
132
|
+
done
|
|
133
|
+
}
|
|
134
|
+
|
|
135
|
+
@test "criterion_scores is an object of numbers in [0,1], not an array" {
|
|
136
|
+
local agent json
|
|
137
|
+
for agent in standard adversarial security; do
|
|
138
|
+
json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-${agent}.md")
|
|
139
|
+
printf '%s' "$json" | jq -e '
|
|
140
|
+
(.criterion_scores | type) == "object"
|
|
141
|
+
and (.criterion_scores | length) > 0
|
|
142
|
+
and all(.criterion_scores[]; type == "number" and . >= 0 and . <= 1)
|
|
143
|
+
' >/dev/null || return 1
|
|
144
|
+
done
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
@test "criterion_scores is keyed by rubric criteria, not by the agent's own lenses" {
|
|
148
|
+
# The default rubric's criteria are the only legal keys. This is the whole
|
|
149
|
+
# point: an agent keying by its own lens names (edge-cases, injection)
|
|
150
|
+
# produces scores no aggregator can ever match to a floor.
|
|
151
|
+
local rubric_names json agent
|
|
152
|
+
rubric_names=$(jq -c '[.tribunal.rubric.builtins[0].criteria[].name]' \
|
|
153
|
+
"${REPO_ROOT}/plugins/tribunal/config.json")
|
|
154
|
+
|
|
155
|
+
for agent in standard adversarial security; do
|
|
156
|
+
json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-${agent}.md")
|
|
157
|
+
printf '%s' "$json" | jq -e --argjson want "$rubric_names" '
|
|
158
|
+
[.criterion_scores | keys[]] | all(. as $k | $want | index($k) != null)
|
|
159
|
+
' >/dev/null || return 1
|
|
160
|
+
done
|
|
161
|
+
}
|
|
162
|
+
|
|
163
|
+
@test "every judge agent scores safety, the criterion with the highest floor" {
|
|
164
|
+
# safety carries min_pass 0.8 and appeared in NO agent contract before this
|
|
165
|
+
# change, so its floor could never fire. Regression guard.
|
|
166
|
+
local agent json
|
|
167
|
+
for agent in standard adversarial security; do
|
|
168
|
+
json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-${agent}.md")
|
|
169
|
+
printf '%s' "$json" | jq -e '.criterion_scores | has("safety")' >/dev/null || return 1
|
|
170
|
+
done
|
|
171
|
+
}
|
|
172
|
+
|
|
173
|
+
@test "adversarial and security agents document the rubric they must score" {
|
|
174
|
+
local agent
|
|
175
|
+
for agent in adversarial security; do
|
|
176
|
+
grep -qi 'rubric' "${AGENTS_DIR}/tribunal-judge-${agent}.md" || return 1
|
|
177
|
+
done
|
|
178
|
+
}
|
|
179
|
+
|
|
180
|
+
@test "criteria_evaluated keeps each agent's own investigative lenses" {
|
|
181
|
+
# Deliberately NOT unified with criterion_scores. If a future edit collapses
|
|
182
|
+
# the two, the adversarial agent stops reporting what it actually probed.
|
|
183
|
+
local json
|
|
184
|
+
json=$(_agent_json "${AGENTS_DIR}/tribunal-judge-adversarial.md")
|
|
185
|
+
printf '%s' "$json" | jq -e '
|
|
186
|
+
(.criteria_evaluated | index("edge-cases")) != null
|
|
187
|
+
' >/dev/null
|
|
188
|
+
}
|
|
189
|
+
|
|
190
|
+
@test "the schema dependency admits criterion_scores" {
|
|
191
|
+
# criterion_scores and the criterion_floor reason land in 2.12.0. On ^2.11.0
|
|
192
|
+
# the runtime emitter rejects the payload wherever the package resolves.
|
|
193
|
+
local range
|
|
194
|
+
range=$(jq -r '.dependencies["@onlooker-community/schema"] // .devDependencies["@onlooker-community/schema"]' \
|
|
195
|
+
"${REPO_ROOT}/package.json")
|
|
196
|
+
[ "$range" = "^2.12.0" ]
|
|
197
|
+
}
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
- [ ] **Step 2: Run the test to verify it fails**
|
|
201
|
+
|
|
202
|
+
Run: `bats test/bats/tribunal-judge-agents.bats`
|
|
203
|
+
Expected: FAIL — 6 of 7 tests fail. No agent has `criterion_scores` yet, and the dependency is `^2.11.0`. The `criteria_evaluated` test passes already, which is correct: it guards behavior that must **not** change.
|
|
204
|
+
|
|
205
|
+
- [ ] **Step 3: Bump the schema dependency**
|
|
206
|
+
|
|
207
|
+
```bash
|
|
208
|
+
npm install @onlooker-community/schema@^2.12.0
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
Confirm `package.json` reads `"@onlooker-community/schema": "^2.12.0"` and `package-lock.json` resolves `2.12.0`.
|
|
212
|
+
|
|
213
|
+
- [ ] **Step 4: Add `criterion_scores` to the standard judge**
|
|
214
|
+
|
|
215
|
+
In `plugins/tribunal/agents/tribunal-judge-standard.md`, inside the ```json block, add the field after `criteria_evaluated`:
|
|
216
|
+
|
|
217
|
+
```json
|
|
218
|
+
"criteria_evaluated": ["correctness", "completeness", "clarity"],
|
|
219
|
+
"criterion_scores": {
|
|
220
|
+
"correctness": 0.9,
|
|
221
|
+
"completeness": 0.75,
|
|
222
|
+
"safety": 0.85,
|
|
223
|
+
"clarity": 0.8
|
|
224
|
+
},
|
|
225
|
+
```
|
|
226
|
+
|
|
227
|
+
Then, immediately after the "Required fields" paragraph (currently line 43), add:
|
|
228
|
+
|
|
229
|
+
```markdown
|
|
230
|
+
`criterion_scores` maps **each criterion name from the rubric you were given** to your score for it in `[0,1]`. This is separate from `criteria_evaluated`, which lists the dimensions *you* chose to investigate — the rubric's criteria are what the orchestrator weights and floors.
|
|
231
|
+
|
|
232
|
+
Score every rubric criterion you can judge. **Omit any criterion you genuinely cannot assess — do not send `0` for it.** A `0` means "I assessed this and it failed"; an omission means "I did not assess this." The orchestrator treats them very differently: a `0` on a criterion with a floor blocks the task outright, while an omission is reported as a coverage gap.
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
- [ ] **Step 5: Add a rubric section and `criterion_scores` to the adversarial judge**
|
|
236
|
+
|
|
237
|
+
In `plugins/tribunal/agents/tribunal-judge-adversarial.md`, add to the ```json block after `criteria_evaluated`:
|
|
238
|
+
|
|
239
|
+
```json
|
|
240
|
+
"criteria_evaluated": ["edge-cases", "concurrency", "idempotency"],
|
|
241
|
+
"criterion_scores": {
|
|
242
|
+
"correctness": 0.5,
|
|
243
|
+
"completeness": 0.6,
|
|
244
|
+
"safety": 0.55,
|
|
245
|
+
"clarity": 0.8
|
|
246
|
+
},
|
|
247
|
+
```
|
|
248
|
+
|
|
249
|
+
Then add this section immediately before `## Output format`:
|
|
250
|
+
|
|
251
|
+
```markdown
|
|
252
|
+
## Scoring against the rubric
|
|
253
|
+
|
|
254
|
+
You are given a rubric with named criteria, each carrying a weight and a
|
|
255
|
+
`min_pass` floor. Your falsification work is how you form a judgment; the
|
|
256
|
+
rubric's criteria are how you report it.
|
|
257
|
+
|
|
258
|
+
Report a score in `[0,1]` for every rubric criterion in `criterion_scores`,
|
|
259
|
+
keyed by the rubric's own names. Your `criteria_evaluated` list stays what it
|
|
260
|
+
has always been — the dimensions you probed (edge cases, concurrency,
|
|
261
|
+
idempotency). The two lists are not expected to match.
|
|
262
|
+
|
|
263
|
+
`safety` in particular is a criterion you are well placed to score and no other
|
|
264
|
+
default judge covers. A crash on malformed input, a non-idempotent migration, a
|
|
265
|
+
race that corrupts state — those are safety findings, and this is where they
|
|
266
|
+
belong.
|
|
267
|
+
|
|
268
|
+
**Omit any criterion you cannot assess rather than scoring it `0`.** A `0` says
|
|
269
|
+
you assessed it and it failed, which on a criterion with a floor blocks the
|
|
270
|
+
task by itself.
|
|
271
|
+
```
|
|
272
|
+
|
|
273
|
+
- [ ] **Step 6: Add a rubric section and `criterion_scores` to the security judge**
|
|
274
|
+
|
|
275
|
+
In `plugins/tribunal/agents/tribunal-judge-security.md`, add to the ```json block after `criteria_evaluated`:
|
|
276
|
+
|
|
277
|
+
```json
|
|
278
|
+
"criteria_evaluated": ["injection", "secrets", "path-traversal"],
|
|
279
|
+
"criterion_scores": {
|
|
280
|
+
"correctness": 0.7,
|
|
281
|
+
"completeness": 0.6,
|
|
282
|
+
"safety": 0.2,
|
|
283
|
+
"clarity": 0.75
|
|
284
|
+
},
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
Then add this section immediately before `## Output format`:
|
|
288
|
+
|
|
289
|
+
```markdown
|
|
290
|
+
## Scoring against the rubric
|
|
291
|
+
|
|
292
|
+
You are given a rubric with named criteria, each carrying a weight and a
|
|
293
|
+
`min_pass` floor. Your findings are how you form a judgment; the rubric's
|
|
294
|
+
criteria are how you report it.
|
|
295
|
+
|
|
296
|
+
Report a score in `[0,1]` for every rubric criterion in `criterion_scores`,
|
|
297
|
+
keyed by the rubric's own names. Your `criteria_evaluated` list stays what it
|
|
298
|
+
has always been — the dimensions you swept (injection, secrets, path
|
|
299
|
+
traversal). The two lists are not expected to match.
|
|
300
|
+
|
|
301
|
+
`safety` is where your findings land. It carries the highest floor in the
|
|
302
|
+
default rubric, and a single unresolved injection or leaked credential should
|
|
303
|
+
put your `safety` score below it.
|
|
304
|
+
|
|
305
|
+
**Omit any criterion you cannot assess rather than scoring it `0`.** A `0` says
|
|
306
|
+
you assessed it and it failed, which on a criterion with a floor blocks the
|
|
307
|
+
task by itself.
|
|
308
|
+
```
|
|
309
|
+
|
|
310
|
+
- [ ] **Step 7: Run the tests to verify they pass**
|
|
311
|
+
|
|
312
|
+
Run: `bats test/bats/tribunal-judge-agents.bats`
|
|
313
|
+
Expected: PASS, 7/7.
|
|
314
|
+
|
|
315
|
+
- [ ] **Step 8: Prove the tests are falsifiable**
|
|
316
|
+
|
|
317
|
+
In a scratch (uncommitted) edit, delete the `"safety"` key from `tribunal-judge-adversarial.md`'s `criterion_scores`. Re-run: the "every judge agent scores safety" test must fail. Restore it. Then change one `criterion_scores` key in the security agent to `"injection"` and confirm the rubric-keys test fails. Restore. **Report both results in your report file.**
|
|
318
|
+
|
|
319
|
+
- [ ] **Step 9: Verify the whole suite still passes**
|
|
320
|
+
|
|
321
|
+
Run: `npm run test:ci`
|
|
322
|
+
Expected: exit 0. Read the exit code directly with `$?`, never through a pipe.
|
|
323
|
+
|
|
324
|
+
- [ ] **Step 10: Commit**
|
|
325
|
+
|
|
326
|
+
```bash
|
|
327
|
+
git add package.json package-lock.json plugins/tribunal/agents/ test/bats/tribunal-judge-agents.bats
|
|
328
|
+
git commit -m "feat(tribunal): have judges score the rubric's criteria :straight_ruler:"
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
---
|
|
332
|
+
|
|
333
|
+
### Task 2: `weighted_mean` becomes real
|
|
334
|
+
|
|
335
|
+
**Files:**
|
|
336
|
+
- Modify: `plugins/tribunal/scripts/lib/tribunal-aggregate.sh:1-59`
|
|
337
|
+
- Test: `test/bats/tribunal-aggregate.bats`
|
|
338
|
+
|
|
339
|
+
**Interfaces:**
|
|
340
|
+
- Consumes: the `criterion_scores` contract from Task 1.
|
|
341
|
+
- Produces: `tribunal_aggregate <method> <verdicts_json> [<rubric_json>]` — signature **unchanged**, third parameter now used instead of discarded. For `weighted_mean` with usable scores it echoes the weighted mean; otherwise it echoes the plain mean of `.score`, exactly as today.
|
|
342
|
+
|
|
343
|
+
- [ ] **Step 1: Write the failing tests**
|
|
344
|
+
|
|
345
|
+
Append to `test/bats/tribunal-aggregate.bats`:
|
|
346
|
+
|
|
347
|
+
```bash
|
|
348
|
+
# Two criteria with deliberately unequal weights, so weighted_mean and mean
|
|
349
|
+
# cannot coincide. Judge A is strong on the heavy criterion, weak on the light
|
|
350
|
+
# one; judge B is the reverse.
|
|
351
|
+
RUBRIC_UNEQUAL='{"criteria":[{"name":"correctness","weight":0.9,"min_pass":0.7},{"name":"clarity","weight":0.1,"min_pass":0.5}]}'
|
|
352
|
+
SCORED='[
|
|
353
|
+
{"judge_id":"a","score":0.5,"criterion_scores":{"correctness":1.0,"clarity":0.0}},
|
|
354
|
+
{"judge_id":"b","score":0.5,"criterion_scores":{"correctness":1.0,"clarity":0.0}}
|
|
355
|
+
]'
|
|
356
|
+
|
|
357
|
+
@test "weighted_mean differs from mean when weights are unequal" {
|
|
358
|
+
# mean of .score is 0.5 for both judges. The weighted mean is
|
|
359
|
+
# 0.9*1.0 + 0.1*0.0 = 0.9. If these come out equal, weights are still inert.
|
|
360
|
+
local w m
|
|
361
|
+
w=$(tribunal_aggregate "weighted_mean" "$SCORED" "$RUBRIC_UNEQUAL")
|
|
362
|
+
m=$(tribunal_aggregate "mean" "$SCORED" "$RUBRIC_UNEQUAL")
|
|
363
|
+
awk -v a="$w" -v b="$m" 'BEGIN { exit !(a != b) }' || return 1
|
|
364
|
+
awk -v a="$w" 'BEGIN { exit !(a > 0.89 && a < 0.91) }'
|
|
365
|
+
}
|
|
366
|
+
|
|
367
|
+
@test "weighted_mean averages judges within a criterion before weighting" {
|
|
368
|
+
local out
|
|
369
|
+
out=$(tribunal_aggregate "weighted_mean" '[
|
|
370
|
+
{"judge_id":"a","score":0.5,"criterion_scores":{"correctness":1.0,"clarity":1.0}},
|
|
371
|
+
{"judge_id":"b","score":0.5,"criterion_scores":{"correctness":0.0,"clarity":1.0}}
|
|
372
|
+
]' "$RUBRIC_UNEQUAL")
|
|
373
|
+
# correctness mean 0.5, clarity mean 1.0 → 0.9*0.5 + 0.1*1.0 = 0.55
|
|
374
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.549 && a < 0.551) }'
|
|
375
|
+
}
|
|
376
|
+
|
|
377
|
+
@test "weighted_mean degrades to mean when no verdict carries criterion_scores" {
|
|
378
|
+
# Every verdict emitted before Task 1 shipped looks like this.
|
|
379
|
+
local out
|
|
380
|
+
out=$(tribunal_aggregate "weighted_mean" \
|
|
381
|
+
'[{"judge_id":"a","score":0.8},{"judge_id":"b","score":0.6}]' "$RUBRIC_UNEQUAL")
|
|
382
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.699 && a < 0.701) }'
|
|
383
|
+
}
|
|
384
|
+
|
|
385
|
+
@test "an absent criterion is skipped, not counted as zero" {
|
|
386
|
+
# clarity is absent everywhere. If absence read as 0 the answer would be
|
|
387
|
+
# 0.9*1.0 + 0.1*0.0 = 0.9. Skipping it renormalizes to 0.9/0.9 = 1.0.
|
|
388
|
+
local out
|
|
389
|
+
out=$(tribunal_aggregate "weighted_mean" \
|
|
390
|
+
'[{"judge_id":"a","score":0.5,"criterion_scores":{"correctness":1.0}}]' \
|
|
391
|
+
"$RUBRIC_UNEQUAL")
|
|
392
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.999 && a < 1.001) }'
|
|
393
|
+
}
|
|
394
|
+
|
|
395
|
+
@test "a criterion scored at zero is honored, not treated as absent" {
|
|
396
|
+
# The mirror of the previous test, and the one that catches a `// 0` fix
|
|
397
|
+
# that "passes" the absence test by accident.
|
|
398
|
+
local out
|
|
399
|
+
out=$(tribunal_aggregate "weighted_mean" \
|
|
400
|
+
'[{"judge_id":"a","score":0.5,"criterion_scores":{"correctness":0.0}}]' \
|
|
401
|
+
"$RUBRIC_UNEQUAL")
|
|
402
|
+
awk -v a="$out" 'BEGIN { exit !(a >= 0 && a < 0.001) }'
|
|
403
|
+
}
|
|
404
|
+
|
|
405
|
+
@test "weights that do not sum to 1.0 are normalized" {
|
|
406
|
+
# tribunal_rubric_validate rejects such a rubric, but librarian's loader
|
|
407
|
+
# validates nothing and hands its rubric straight through. Normalizing here
|
|
408
|
+
# means the two paths cannot disagree.
|
|
409
|
+
local out
|
|
410
|
+
out=$(tribunal_aggregate "weighted_mean" \
|
|
411
|
+
'[{"judge_id":"a","score":0.1,"criterion_scores":{"correctness":1.0,"clarity":0.0}}]' \
|
|
412
|
+
'{"criteria":[{"name":"correctness","weight":1.8,"min_pass":0.7},{"name":"clarity","weight":0.2,"min_pass":0.5}]}')
|
|
413
|
+
# 1.8*1.0 + 0.2*0.0 = 1.8, over a weight sum of 2.0 → 0.9
|
|
414
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.899 && a < 0.901) }'
|
|
415
|
+
}
|
|
416
|
+
|
|
417
|
+
@test "a hyphenated criterion name scores correctly" {
|
|
418
|
+
# A dotted jq path would be a COMPILE error here: exit 3, empty stdout,
|
|
419
|
+
# which awk reads as 0.
|
|
420
|
+
local out
|
|
421
|
+
out=$(tribunal_aggregate "weighted_mean" \
|
|
422
|
+
'[{"judge_id":"a","score":0.2,"criterion_scores":{"path-traversal":1.0}}]' \
|
|
423
|
+
'{"criteria":[{"name":"path-traversal","weight":1.0,"min_pass":0.5}]}')
|
|
424
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.999 && a < 1.001) }'
|
|
425
|
+
}
|
|
426
|
+
|
|
427
|
+
@test "weighted_mean falls back to mean when the rubric is absent" {
|
|
428
|
+
local out
|
|
429
|
+
out=$(tribunal_aggregate "weighted_mean" "$SCORED")
|
|
430
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.499 && a < 0.501) }'
|
|
431
|
+
}
|
|
432
|
+
|
|
433
|
+
@test "a non-number criterion score is ignored rather than poisoning the mean" {
|
|
434
|
+
local out
|
|
435
|
+
out=$(tribunal_aggregate "weighted_mean" \
|
|
436
|
+
'[{"judge_id":"a","score":0.5,"criterion_scores":{"correctness":1.0,"clarity":"n/a"}}]' \
|
|
437
|
+
"$RUBRIC_UNEQUAL")
|
|
438
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.999 && a < 1.001) }'
|
|
439
|
+
}
|
|
440
|
+
```
|
|
441
|
+
|
|
442
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
443
|
+
|
|
444
|
+
Run: `bats test/bats/tribunal-aggregate.bats`
|
|
445
|
+
Expected: FAIL — the new weighted tests fail because `weighted_mean` currently shares the `mean` branch. The degrade and fallback tests pass already; that is correct, since they pin behavior that must survive.
|
|
446
|
+
|
|
447
|
+
- [ ] **Step 3: Implement the real weighted mean**
|
|
448
|
+
|
|
449
|
+
Replace the header comment block at `plugins/tribunal/scripts/lib/tribunal-aggregate.sh:18-23` with:
|
|
450
|
+
|
|
451
|
+
```bash
|
|
452
|
+
# weighted_mean uses *rubric criterion weights*: average the judges' scores on
|
|
453
|
+
# each criterion, weight each criterion's mean, and normalize by the weights
|
|
454
|
+
# actually used. A criterion no judge scored contributes nothing and its weight
|
|
455
|
+
# is excluded from the denominator — absence is not a zero. When no criterion
|
|
456
|
+
# has any score (every verdict emitted before judges shipped criterion_scores),
|
|
457
|
+
# weighted_mean degrades to mean rather than collapsing to 0.
|
|
458
|
+
```
|
|
459
|
+
|
|
460
|
+
Replace the function body's `_rubric` handling and the `mean|weighted_mean` branch:
|
|
461
|
+
|
|
462
|
+
```bash
|
|
463
|
+
tribunal_aggregate() {
|
|
464
|
+
local method="${1:-mean}"
|
|
465
|
+
local verdicts="${2:-[]}"
|
|
466
|
+
local rubric="${3:-}"
|
|
467
|
+
[ -z "$rubric" ] && rubric='{}'
|
|
468
|
+
|
|
469
|
+
local count
|
|
470
|
+
count=$(printf '%s' "$verdicts" | jq 'length' 2>/dev/null) || count=0
|
|
471
|
+
[[ "$count" -eq 0 ]] && { printf '0'; return 0; }
|
|
472
|
+
|
|
473
|
+
case "$method" in
|
|
474
|
+
mean)
|
|
475
|
+
printf '%s' "$verdicts" | jq -r '[.[].score] | add / length'
|
|
476
|
+
;;
|
|
477
|
+
weighted_mean)
|
|
478
|
+
local weighted
|
|
479
|
+
weighted=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
|
|
480
|
+
. as $v
|
|
481
|
+
| [ ($rubric.criteria // [])[]
|
|
482
|
+
| select((.name | type) == "string" and (.weight | type) == "number")
|
|
483
|
+
| . as $c
|
|
484
|
+
| ([ $v[]
|
|
485
|
+
| select((.criterion_scores | type) == "object")
|
|
486
|
+
| select(.criterion_scores | has($c.name))
|
|
487
|
+
| .criterion_scores[$c.name]
|
|
488
|
+
| select(type == "number") ]) as $scores
|
|
489
|
+
| select(($scores | length) > 0)
|
|
490
|
+
| { w: $c.weight, m: (($scores | add) / ($scores | length)) } ]
|
|
491
|
+
| (map(.w) | add) as $den
|
|
492
|
+
| if length == 0 or $den == null or $den <= 0 then empty
|
|
493
|
+
else (map(.w * .m) | add) / $den
|
|
494
|
+
end
|
|
495
|
+
' 2>/dev/null)
|
|
496
|
+
if [ -n "$weighted" ]; then
|
|
497
|
+
printf '%s' "$weighted"
|
|
498
|
+
else
|
|
499
|
+
# No criterion carried a usable score — degrade to mean.
|
|
500
|
+
printf '%s' "$verdicts" | jq -r '[.[].score] | add / length'
|
|
501
|
+
fi
|
|
502
|
+
;;
|
|
503
|
+
median)
|
|
504
|
+
```
|
|
505
|
+
|
|
506
|
+
Leave `median`, `min`, and the unknown-method fallback exactly as they are.
|
|
507
|
+
|
|
508
|
+
- [ ] **Step 4: Run the tests to verify they pass**
|
|
509
|
+
|
|
510
|
+
Run: `bats test/bats/tribunal-aggregate.bats`
|
|
511
|
+
Expected: PASS, 19/19.
|
|
512
|
+
|
|
513
|
+
- [ ] **Step 5: Prove the guards are falsifiable**
|
|
514
|
+
|
|
515
|
+
Three scratch (uncommitted) edits, each reverted after checking:
|
|
516
|
+
|
|
517
|
+
1. Change `select(.criterion_scores | has($c.name))` to `select(true)` and `.criterion_scores[$c.name]` to `(.criterion_scores[$c.name] // 0)`. The "absent criterion is skipped" test must fail. **This is the exact regression the `// 0` ban exists to prevent.**
|
|
518
|
+
2. Change `.criterion_scores[$c.name]` to `.criterion_scores[$c.name] | tostring | tonumber` — the hyphenated-name test must still pass (proving the bracket lookup, not the name, is what matters). Then change the lookup to a dotted `.criterion_scores.correctness` and confirm the hyphenated test fails.
|
|
519
|
+
3. Delete `/ $den` (dividing by nothing). The normalization test must fail.
|
|
520
|
+
|
|
521
|
+
**Report all three results in your report file.**
|
|
522
|
+
|
|
523
|
+
- [ ] **Step 6: Verify the whole suite**
|
|
524
|
+
|
|
525
|
+
Run: `npm run test:ci`
|
|
526
|
+
Expected: exit 0, read from `$?` directly.
|
|
527
|
+
|
|
528
|
+
- [ ] **Step 7: Commit**
|
|
529
|
+
|
|
530
|
+
```bash
|
|
531
|
+
git add plugins/tribunal/scripts/lib/tribunal-aggregate.sh test/bats/tribunal-aggregate.bats
|
|
532
|
+
git commit -m "feat(tribunal): make weighted_mean weight something :straight_ruler:"
|
|
533
|
+
```
|
|
534
|
+
|
|
535
|
+
---
|
|
536
|
+
|
|
537
|
+
### Task 3: `min_pass` becomes enforceable
|
|
538
|
+
|
|
539
|
+
**Files:**
|
|
540
|
+
- Modify: `plugins/tribunal/scripts/lib/tribunal-gate.sh:1-112`
|
|
541
|
+
- Modify: `plugins/tribunal/skills/tribunal/SKILL.md:117-124`
|
|
542
|
+
- Test: `test/bats/tribunal-gate.bats`
|
|
543
|
+
|
|
544
|
+
**Interfaces:**
|
|
545
|
+
- Consumes: the `criterion_scores` contract from Task 1.
|
|
546
|
+
- Produces: `tribunal_gate_decide <policy> <verdicts> <aggregated_score> <score_threshold> <meta> <dissent_score> <dissent_threshold> [<rubric_json>]` — an **eighth, optional, trailing** parameter. When a criterion's mean falls below its `min_pass`, it echoes `{"passed":false,"reason":"criterion_floor","failed_criterion":"<name>"}`. All existing reasons and their precedence are unchanged.
|
|
547
|
+
|
|
548
|
+
- [ ] **Step 1: Write the failing tests**
|
|
549
|
+
|
|
550
|
+
Append to `test/bats/tribunal-gate.bats`:
|
|
551
|
+
|
|
552
|
+
```bash
|
|
553
|
+
RUBRIC_FLOOR='{"criteria":[{"name":"correctness","weight":0.5,"min_pass":0.7},{"name":"safety","weight":0.5,"min_pass":0.8}]}'
|
|
554
|
+
|
|
555
|
+
@test "a criterion below its floor blocks even when score and jury both pass" {
|
|
556
|
+
# This is the property that does not exist today: aggregate 0.82 clears the
|
|
557
|
+
# 0.75 threshold, both judges passed, and the gate blocks anyway.
|
|
558
|
+
local out
|
|
559
|
+
out=$(tribunal_gate_decide "majority" '[
|
|
560
|
+
{"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"correctness":0.9,"safety":0.3}},
|
|
561
|
+
{"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"correctness":0.9,"safety":0.3}}
|
|
562
|
+
]' "0.82" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
|
|
563
|
+
printf '%s' "$out" | jq -e '.passed == false' >/dev/null || return 1
|
|
564
|
+
printf '%s' "$out" | jq -e '.reason == "criterion_floor"' >/dev/null || return 1
|
|
565
|
+
printf '%s' "$out" | jq -e '.failed_criterion == "safety"' >/dev/null
|
|
566
|
+
}
|
|
567
|
+
|
|
568
|
+
@test "a criterion at exactly its floor passes" {
|
|
569
|
+
local out
|
|
570
|
+
out=$(tribunal_gate_decide "majority" '[
|
|
571
|
+
{"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"correctness":0.7,"safety":0.8}},
|
|
572
|
+
{"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"correctness":0.7,"safety":0.8}}
|
|
573
|
+
]' "0.82" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
|
|
574
|
+
printf '%s' "$out" | jq -e '.passed == true' >/dev/null
|
|
575
|
+
}
|
|
576
|
+
|
|
577
|
+
@test "absent criterion_scores never block" {
|
|
578
|
+
# Every verdict emitted before Task 1 shipped. Treating absence as violation
|
|
579
|
+
# would make every pre-upgrade judge fail every rubric carrying a floor.
|
|
580
|
+
local out
|
|
581
|
+
out=$(tribunal_gate_decide "majority" "$ALL_PASSED" "0.82" "0.75" \
|
|
582
|
+
"$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
|
|
583
|
+
printf '%s' "$out" | jq -e '.passed == true' >/dev/null
|
|
584
|
+
}
|
|
585
|
+
|
|
586
|
+
@test "a criterion scored exactly zero does block" {
|
|
587
|
+
# The mirror of the previous test. A fix that conflates absent with zero
|
|
588
|
+
# passes one of these two and fails the other.
|
|
589
|
+
local out
|
|
590
|
+
out=$(tribunal_gate_decide "majority" '[
|
|
591
|
+
{"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"correctness":0.9,"safety":0.0}},
|
|
592
|
+
{"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"correctness":0.9,"safety":0.0}}
|
|
593
|
+
]' "0.82" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
|
|
594
|
+
printf '%s' "$out" | jq -e '.reason == "criterion_floor"' >/dev/null || return 1
|
|
595
|
+
printf '%s' "$out" | jq -e '.failed_criterion == "safety"' >/dev/null
|
|
596
|
+
}
|
|
597
|
+
|
|
598
|
+
@test "a hyphenated criterion name gates correctly" {
|
|
599
|
+
local out
|
|
600
|
+
out=$(tribunal_gate_decide "majority" '[
|
|
601
|
+
{"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"path-traversal":0.1}},
|
|
602
|
+
{"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"path-traversal":0.1}}
|
|
603
|
+
]' "0.82" "0.75" "$NO_META" "0.05" "0.25" \
|
|
604
|
+
'{"criteria":[{"name":"path-traversal","weight":1.0,"min_pass":0.5}]}')
|
|
605
|
+
printf '%s' "$out" | jq -e '.failed_criterion == "path-traversal"' >/dev/null
|
|
606
|
+
}
|
|
607
|
+
|
|
608
|
+
@test "low_score still wins over criterion_floor" {
|
|
609
|
+
# Precedence matters for the retry digest: if the aggregate missed the
|
|
610
|
+
# threshold, that is the more actionable thing to tell the Actor.
|
|
611
|
+
local out
|
|
612
|
+
out=$(tribunal_gate_decide "majority" '[
|
|
613
|
+
{"judge_id":"a","score":0.20,"passed":true,"criterion_scores":{"correctness":0.1,"safety":0.1}},
|
|
614
|
+
{"judge_id":"b","score":0.20,"passed":true,"criterion_scores":{"correctness":0.1,"safety":0.1}}
|
|
615
|
+
]' "0.20" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR")
|
|
616
|
+
printf '%s' "$out" | jq -e '.reason == "low_score"' >/dev/null
|
|
617
|
+
}
|
|
618
|
+
|
|
619
|
+
@test "a floor on a criterion no judge scored is reported on stderr" {
|
|
620
|
+
# The adversarial-judge gap: safety carries the highest floor and appeared
|
|
621
|
+
# in no agent contract. Silently passing a floor nobody scored is this
|
|
622
|
+
# design's own failure mode one layer down.
|
|
623
|
+
run --separate-stderr tribunal_gate_decide "majority" '[
|
|
624
|
+
{"judge_id":"a","score":0.85,"passed":true,"criterion_scores":{"correctness":0.9}},
|
|
625
|
+
{"judge_id":"b","score":0.80,"passed":true,"criterion_scores":{"correctness":0.9}}
|
|
626
|
+
]' "0.82" "0.75" "$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR"
|
|
627
|
+
printf '%s' "$output" | jq -e '.passed == true' >/dev/null || return 1
|
|
628
|
+
local re='safety'
|
|
629
|
+
[[ "$stderr" =~ $re ]]
|
|
630
|
+
}
|
|
631
|
+
|
|
632
|
+
@test "no unscored-criterion warning when no judge scored anything" {
|
|
633
|
+
# The pre-upgrade fleet must not spew a warning on every single gate.
|
|
634
|
+
run --separate-stderr tribunal_gate_decide "majority" "$ALL_PASSED" "0.82" "0.75" \
|
|
635
|
+
"$NO_META" "0.05" "0.25" "$RUBRIC_FLOOR"
|
|
636
|
+
[ -z "$stderr" ]
|
|
637
|
+
}
|
|
638
|
+
|
|
639
|
+
@test "the gate still works with no rubric at all" {
|
|
640
|
+
local out
|
|
641
|
+
out=$(tribunal_gate_decide "majority" "$ALL_PASSED" "0.82" "0.75" "$NO_META" "0.05" "0.25")
|
|
642
|
+
printf '%s' "$out" | jq -e '.passed == true' >/dev/null
|
|
643
|
+
}
|
|
644
|
+
```
|
|
645
|
+
|
|
646
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
647
|
+
|
|
648
|
+
Run: `bats test/bats/tribunal-gate.bats`
|
|
649
|
+
Expected: FAIL — the floor tests fail because no floor exists. The "absent never blocks", "no rubric" and "low_score wins" tests pass already, pinning behavior that must survive.
|
|
650
|
+
|
|
651
|
+
- [ ] **Step 3: Implement the floor**
|
|
652
|
+
|
|
653
|
+
In `plugins/tribunal/scripts/lib/tribunal-gate.sh`, extend the header comment's reason list (line 17) to:
|
|
654
|
+
|
|
655
|
+
```bash
|
|
656
|
+
# reason is one of: low_score | meta_override | bias_detected | dissent_unresolved
|
|
657
|
+
# | criterion_floor (with failed_criterion naming the criterion)
|
|
658
|
+
```
|
|
659
|
+
|
|
660
|
+
Add the eighth parameter after line 29:
|
|
661
|
+
|
|
662
|
+
```bash
|
|
663
|
+
local dissent_threshold="${7:-0.25}"
|
|
664
|
+
local rubric="${8:-}"
|
|
665
|
+
[ -z "$rubric" ] && rubric='{}'
|
|
666
|
+
```
|
|
667
|
+
|
|
668
|
+
Then insert this block immediately **after** the `score_ok` computation (currently line 93) and **before** the `if [[ "$jury_ok" -eq 0 && "$score_ok" -eq 0 ]]` check:
|
|
669
|
+
|
|
670
|
+
```bash
|
|
671
|
+
# Per-criterion floors. A criterion whose mean across the judges that scored
|
|
672
|
+
# it falls below min_pass blocks regardless of the aggregate or the policy —
|
|
673
|
+
# that is the whole point of a floor.
|
|
674
|
+
#
|
|
675
|
+
# A criterion no judge scored is NOT a violation: absence is not a zero, and
|
|
676
|
+
# treating it as one would fail every verdict emitted before judges shipped
|
|
677
|
+
# criterion_scores. It is reported on stderr instead, because a floor nobody
|
|
678
|
+
# scores is a silent hole in the rubric.
|
|
679
|
+
local floor_failed unscored_floors any_scored
|
|
680
|
+
any_scored=$(printf '%s' "$verdicts" | jq -r '
|
|
681
|
+
[.[] | select((.criterion_scores | type) == "object")
|
|
682
|
+
| select((.criterion_scores | length) > 0)] | length > 0
|
|
683
|
+
' 2>/dev/null) || any_scored="false"
|
|
684
|
+
|
|
685
|
+
floor_failed=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
|
|
686
|
+
. as $v
|
|
687
|
+
| [ ($rubric.criteria // [])[]
|
|
688
|
+
| select((.name | type) == "string" and (.min_pass | type) == "number")
|
|
689
|
+
| . as $c
|
|
690
|
+
| ([ $v[]
|
|
691
|
+
| select((.criterion_scores | type) == "object")
|
|
692
|
+
| select(.criterion_scores | has($c.name))
|
|
693
|
+
| .criterion_scores[$c.name]
|
|
694
|
+
| select(type == "number") ]) as $scores
|
|
695
|
+
| select(($scores | length) > 0)
|
|
696
|
+
| select((($scores | add) / ($scores | length)) < $c.min_pass)
|
|
697
|
+
| $c.name ]
|
|
698
|
+
| first // empty
|
|
699
|
+
' 2>/dev/null) || floor_failed=""
|
|
700
|
+
|
|
701
|
+
if [[ "$any_scored" == "true" ]]; then
|
|
702
|
+
unscored_floors=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
|
|
703
|
+
. as $v
|
|
704
|
+
| [ ($rubric.criteria // [])[]
|
|
705
|
+
| select((.name | type) == "string" and (.min_pass | type) == "number")
|
|
706
|
+
| . as $c
|
|
707
|
+
| select([ $v[]
|
|
708
|
+
| select((.criterion_scores | type) == "object")
|
|
709
|
+
| select(.criterion_scores | has($c.name))
|
|
710
|
+
| .criterion_scores[$c.name]
|
|
711
|
+
| select(type == "number") ] | length == 0)
|
|
712
|
+
| $c.name ]
|
|
713
|
+
| join(", ")
|
|
714
|
+
' 2>/dev/null) || unscored_floors=""
|
|
715
|
+
if [[ -n "$unscored_floors" ]]; then
|
|
716
|
+
printf 'tribunal-gate: no judge scored these criteria, so their min_pass floors did not apply: %s\n' \
|
|
717
|
+
"$unscored_floors" >&2
|
|
718
|
+
fi
|
|
719
|
+
fi
|
|
720
|
+
```
|
|
721
|
+
|
|
722
|
+
Finally, change the blocking-reason selection (currently lines 100-111) so the floor sits between `low_score` and the jury reasons:
|
|
723
|
+
|
|
724
|
+
```bash
|
|
725
|
+
# Pick the most informative blocking reason. low_score first: if the
|
|
726
|
+
# aggregate missed the threshold, that is the more actionable thing to tell
|
|
727
|
+
# the Actor than any single criterion.
|
|
728
|
+
if [[ "$score_ok" -ne 0 ]]; then
|
|
729
|
+
printf '{"passed":false,"reason":"low_score"}'
|
|
730
|
+
elif [[ -n "$floor_failed" ]]; then
|
|
731
|
+
printf '{"passed":false,"reason":"criterion_floor","failed_criterion":"%s"}' "$floor_failed"
|
|
732
|
+
elif [[ "$jury_ok" -ne 0 ]]; then
|
|
733
|
+
if [[ "$meta_override" == "reject" ]]; then
|
|
734
|
+
printf '{"passed":false,"reason":"meta_override"}'
|
|
735
|
+
else
|
|
736
|
+
printf '{"passed":false,"reason":"dissent_unresolved"}'
|
|
737
|
+
fi
|
|
738
|
+
else
|
|
739
|
+
printf '{"passed":true}'
|
|
740
|
+
fi
|
|
741
|
+
```
|
|
742
|
+
|
|
743
|
+
and delete the now-redundant early `if [[ "$jury_ok" -eq 0 && "$score_ok" -eq 0 ]]` pass block at lines 95-98, since the `else` arm above covers it. **Both a floor failure and a jury failure now reach this chain, so the old two-condition early return would have let a floor failure through.**
|
|
744
|
+
|
|
745
|
+
- [ ] **Step 4: Run the tests to verify they pass**
|
|
746
|
+
|
|
747
|
+
Run: `bats test/bats/tribunal-gate.bats`
|
|
748
|
+
Expected: PASS, 19/19.
|
|
749
|
+
|
|
750
|
+
- [ ] **Step 5: Wire the reason through the orchestration walk**
|
|
751
|
+
|
|
752
|
+
In `plugins/tribunal/skills/tribunal/SKILL.md`, replace the step 7 code block (lines 118-121) with:
|
|
753
|
+
|
|
754
|
+
```bash
|
|
755
|
+
policy=$(printf '%s' "$rubric" | jq -r '.gate_policy // "majority"')
|
|
756
|
+
gate=$(tribunal_gate_decide "$policy" "$verdicts" "$aggregated" "$threshold" "$meta" "$dissent" "$dissent_threshold" "$rubric")
|
|
757
|
+
```
|
|
758
|
+
|
|
759
|
+
and replace the prose at line 122 with:
|
|
760
|
+
|
|
761
|
+
```markdown
|
|
762
|
+
If `gate.passed == true`, emit `tribunal.gate.passed` with `final_score: aggregated` and break the loop with outcome `accepted`. Otherwise emit `tribunal.gate.blocked` with the `reason`, `will_retry: (iteration_number + 1 < max_iterations)`, and `retry_iteration_number` if retrying. **When `reason` is `criterion_floor`, copy `gate.failed_criterion` onto the event payload as `failed_criterion`** — without it the log records that a floor blocked but not which one. Persist `gate.json` either way.
|
|
763
|
+
|
|
764
|
+
A `criterion_floor` block means the aggregate *cleared* its threshold and one criterion still failed its floor. Say so when you report it: "blocked on `safety` (0.30 < 0.80) despite an overall 0.82" is actionable, "blocked" is not.
|
|
765
|
+
```
|
|
766
|
+
|
|
767
|
+
- [ ] **Step 6: Prove the guards are falsifiable**
|
|
768
|
+
|
|
769
|
+
Two scratch (uncommitted) edits, each reverted after checking:
|
|
770
|
+
|
|
771
|
+
1. Delete the `elif [[ -n "$floor_failed" ]]` arm. The three floor-blocking tests must fail.
|
|
772
|
+
2. Change `select(($scores | length) > 0)` in the `floor_failed` query to `select(true)` — the "absent criterion_scores never block" test must fail, proving absence is genuinely excluded rather than accidentally passing.
|
|
773
|
+
|
|
774
|
+
**Report both results in your report file.**
|
|
775
|
+
|
|
776
|
+
- [ ] **Step 7: Verify the whole suite**
|
|
777
|
+
|
|
778
|
+
Run: `npm run test:ci`
|
|
779
|
+
Expected: exit 0, read from `$?` directly.
|
|
780
|
+
|
|
781
|
+
- [ ] **Step 8: Commit**
|
|
782
|
+
|
|
783
|
+
```bash
|
|
784
|
+
git add plugins/tribunal/scripts/lib/tribunal-gate.sh plugins/tribunal/skills/tribunal/SKILL.md test/bats/tribunal-gate.bats
|
|
785
|
+
git commit -m "feat(tribunal): let a criterion floor block a passing score :octagonal_sign:"
|
|
786
|
+
```
|
|
787
|
+
|
|
788
|
+
---
|
|
789
|
+
|
|
790
|
+
### Task 4: Librarian's aggregate and gate become rubric-aware
|
|
791
|
+
|
|
792
|
+
**Files:**
|
|
793
|
+
- Modify: `plugins/librarian/scripts/lib/librarian-lesson-judge.sh:12-81` and `:164-169`
|
|
794
|
+
- Test: `test/bats/librarian-lesson-judge.bats`
|
|
795
|
+
|
|
796
|
+
**Interfaces:**
|
|
797
|
+
- Consumes: the `criterion_scores` contract from Task 1.
|
|
798
|
+
- Produces:
|
|
799
|
+
- `librarian_lesson_aggregate <verdicts_json> [<rubric_json>]` — optional trailing rubric. With usable criterion scores it returns the weighted mean; otherwise the plain mean it returns today. Still returns 1 on an empty panel.
|
|
800
|
+
- `librarian_lesson_gate <gate_policy> <verdicts_json> <aggregate> <threshold> [<rubric_json>]` — optional trailing rubric. Adds the reason `criterion_floor` with a `failed_criterion` key to its echoed JSON. Existing reasons (`gate_passed`, `below_threshold`, `jury_not_majority`, `jury_not_unanimous`, `unknown_gate_policy`) are unchanged.
|
|
801
|
+
|
|
802
|
+
Librarian's reasons are written to on-disk proposal JSON, **not** to a schema-validated event, so this costs no schema version.
|
|
803
|
+
|
|
804
|
+
**Duplication with Tasks 2 and 3 is deliberate** — see Global Constraints and `plugins/librarian/docs/adr/002-agent-definitions-are-shared-assets.md`.
|
|
805
|
+
|
|
806
|
+
- [ ] **Step 1: Write the failing tests**
|
|
807
|
+
|
|
808
|
+
Append to `test/bats/librarian-lesson-judge.bats`:
|
|
809
|
+
|
|
810
|
+
```bash
|
|
811
|
+
PUBLIC_RUBRIC='{"id":"lesson-promotion-public","criteria":[
|
|
812
|
+
{"name":"grounding","weight":0.32,"min_pass":0.7},
|
|
813
|
+
{"name":"scope_accuracy","weight":0.24,"min_pass":0.7},
|
|
814
|
+
{"name":"generality","weight":0.14,"min_pass":0.6},
|
|
815
|
+
{"name":"disclosure","weight":0.30,"min_pass":0.9}],
|
|
816
|
+
"score_threshold":0.75,"gate_policy":"majority"}'
|
|
817
|
+
|
|
818
|
+
@test "lesson aggregate weights criteria when scores are present" {
|
|
819
|
+
local w m verdicts
|
|
820
|
+
verdicts='[
|
|
821
|
+
{"judge_type":"standard","score":0.5,"passed":true,"criterion_scores":{"grounding":1.0,"scope_accuracy":1.0,"generality":1.0,"disclosure":0.0}},
|
|
822
|
+
{"judge_type":"adversarial","score":0.5,"passed":true,"criterion_scores":{"grounding":1.0,"scope_accuracy":1.0,"generality":1.0,"disclosure":0.0}}
|
|
823
|
+
]'
|
|
824
|
+
w=$(librarian_lesson_aggregate "$verdicts" "$PUBLIC_RUBRIC")
|
|
825
|
+
m=$(librarian_lesson_aggregate "$verdicts")
|
|
826
|
+
# weighted: 0.32+0.24+0.14 = 0.70 over a weight sum of 1.0. Plain mean: 0.5.
|
|
827
|
+
awk -v a="$w" -v b="$m" 'BEGIN { exit !(a != b) }' || return 1
|
|
828
|
+
awk -v a="$w" 'BEGIN { exit !(a > 0.699 && a < 0.701) }'
|
|
829
|
+
}
|
|
830
|
+
|
|
831
|
+
@test "lesson aggregate degrades to the plain mean without criterion scores" {
|
|
832
|
+
local out
|
|
833
|
+
out=$(librarian_lesson_aggregate \
|
|
834
|
+
'[{"judge_type":"standard","score":0.8,"passed":true},
|
|
835
|
+
{"judge_type":"adversarial","score":0.6,"passed":true}]' "$PUBLIC_RUBRIC")
|
|
836
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.699 && a < 0.701) }'
|
|
837
|
+
}
|
|
838
|
+
|
|
839
|
+
@test "lesson aggregate still returns 1 on an empty panel" {
|
|
840
|
+
run librarian_lesson_aggregate '[]' "$PUBLIC_RUBRIC"
|
|
841
|
+
[ "$status" -eq 1 ]
|
|
842
|
+
}
|
|
843
|
+
|
|
844
|
+
@test "lesson aggregate normalizes weights that do not sum to 1.0" {
|
|
845
|
+
# librarian_lesson_rubric_get validates NOTHING, so a mis-summed rubric
|
|
846
|
+
# reaches this function where tribunal's validator would have refused it.
|
|
847
|
+
local out
|
|
848
|
+
out=$(librarian_lesson_aggregate \
|
|
849
|
+
'[{"judge_type":"standard","score":0.1,"passed":true,"criterion_scores":{"grounding":1.0,"disclosure":0.0}}]' \
|
|
850
|
+
'{"criteria":[{"name":"grounding","weight":1.8,"min_pass":0.7},{"name":"disclosure","weight":0.2,"min_pass":0.9}]}')
|
|
851
|
+
awk -v a="$out" 'BEGIN { exit !(a > 0.899 && a < 0.901) }'
|
|
852
|
+
}
|
|
853
|
+
|
|
854
|
+
@test "a low disclosure score blocks a public lesson under majority" {
|
|
855
|
+
# The reason this whole thread exists: disclosure's 0.9 floor blocks even
|
|
856
|
+
# though both judges passed and the aggregate clears 0.75.
|
|
857
|
+
local out
|
|
858
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
859
|
+
{"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9,"disclosure":0.4}},
|
|
860
|
+
{"judge_type":"adversarial","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9,"disclosure":0.4}}
|
|
861
|
+
]' "0.78" "0.75" "$PUBLIC_RUBRIC")
|
|
862
|
+
printf '%s' "$out" | jq -e '.passed == false' >/dev/null || return 1
|
|
863
|
+
printf '%s' "$out" | jq -e '.reason == "criterion_floor"' >/dev/null || return 1
|
|
864
|
+
printf '%s' "$out" | jq -e '.failed_criterion == "disclosure"' >/dev/null
|
|
865
|
+
}
|
|
866
|
+
|
|
867
|
+
@test "a low generality score no longer blocks a public lesson" {
|
|
868
|
+
# The behavioral difference from the unanimous stand-in. generality's floor
|
|
869
|
+
# is 0.6; 0.65 clears it, so a judge merely unhappy about generality does
|
|
870
|
+
# not veto a public lesson the way unanimous would have.
|
|
871
|
+
local out
|
|
872
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
873
|
+
{"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.65,"disclosure":0.95}},
|
|
874
|
+
{"judge_type":"adversarial","score":0.85,"passed":true,"criterion_scores":{"grounding":0.9,"scope_accuracy":0.9,"generality":0.65,"disclosure":0.95}}
|
|
875
|
+
]' "0.88" "0.75" "$PUBLIC_RUBRIC")
|
|
876
|
+
printf '%s' "$out" | jq -e '.passed == true' >/dev/null
|
|
877
|
+
}
|
|
878
|
+
|
|
879
|
+
@test "lesson gate: a verdict with no criterion_scores key at all never blocks" {
|
|
880
|
+
# Every verdict emitted before judges shipped criterion_scores. Note this
|
|
881
|
+
# case is caught by the OUTER type guard and never reaches has() — it does
|
|
882
|
+
# NOT pin the per-criterion absence guard. The next test does that.
|
|
883
|
+
local out
|
|
884
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
885
|
+
{"judge_type":"standard","score":0.9,"passed":true},
|
|
886
|
+
{"judge_type":"adversarial","score":0.85,"passed":true}
|
|
887
|
+
]' "0.88" "0.75" "$PUBLIC_RUBRIC")
|
|
888
|
+
printf '%s' "$out" | jq -e '.passed == true and .reason == "gate_passed"' >/dev/null
|
|
889
|
+
}
|
|
890
|
+
|
|
891
|
+
@test "lesson gate: scores present but one floored criterion omitted does not block" {
|
|
892
|
+
# THE test that pins the has() guard. These verdicts DO carry
|
|
893
|
+
# criterion_scores, so they survive the outer type guard and reach the
|
|
894
|
+
# per-criterion lookup — but `disclosure`, whose floor is 0.9, is absent.
|
|
895
|
+
# Substituting `// 0` for has() makes disclosure read as 0.0 and blocks.
|
|
896
|
+
#
|
|
897
|
+
# Written as its own test because the case above cannot fail when has() is
|
|
898
|
+
# deleted: its fixture is filtered one layer earlier. Two different absences
|
|
899
|
+
# sharing one test is how an outer guard silently stands in for an inner one
|
|
900
|
+
# — this project has hit that shape eight times.
|
|
901
|
+
local out
|
|
902
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
903
|
+
{"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9}},
|
|
904
|
+
{"judge_type":"adversarial","score":0.85,"passed":true,"criterion_scores":{"grounding":0.9,"scope_accuracy":0.9,"generality":0.85}}
|
|
905
|
+
]' "0.88" "0.75" "$PUBLIC_RUBRIC")
|
|
906
|
+
printf '%s' "$out" | jq -e '.passed == true' >/dev/null || return 1
|
|
907
|
+
printf '%s' "$out" | jq -e '.reason == "gate_passed"' >/dev/null
|
|
908
|
+
}
|
|
909
|
+
|
|
910
|
+
@test "lesson gate: a criterion scored zero does block" {
|
|
911
|
+
local out
|
|
912
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
913
|
+
{"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"disclosure":0.0}},
|
|
914
|
+
{"judge_type":"adversarial","score":0.85,"passed":true,"criterion_scores":{"disclosure":0.0}}
|
|
915
|
+
]' "0.88" "0.75" "$PUBLIC_RUBRIC")
|
|
916
|
+
printf '%s' "$out" | jq -e '.reason == "criterion_floor"' >/dev/null || return 1
|
|
917
|
+
printf '%s' "$out" | jq -e '.failed_criterion == "disclosure"' >/dev/null
|
|
918
|
+
}
|
|
919
|
+
|
|
920
|
+
@test "lesson gate: below_threshold still wins over criterion_floor" {
|
|
921
|
+
local out
|
|
922
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
923
|
+
{"judge_type":"standard","score":0.2,"passed":true,"criterion_scores":{"disclosure":0.1}},
|
|
924
|
+
{"judge_type":"adversarial","score":0.2,"passed":true,"criterion_scores":{"disclosure":0.1}}
|
|
925
|
+
]' "0.20" "0.75" "$PUBLIC_RUBRIC")
|
|
926
|
+
printf '%s' "$out" | jq -e '.reason == "below_threshold"' >/dev/null
|
|
927
|
+
}
|
|
928
|
+
|
|
929
|
+
@test "lesson gate: jury policy still wins over criterion_floor" {
|
|
930
|
+
local out
|
|
931
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
932
|
+
{"judge_type":"standard","score":0.9,"passed":false,"criterion_scores":{"disclosure":0.1}},
|
|
933
|
+
{"judge_type":"adversarial","score":0.9,"passed":false,"criterion_scores":{"disclosure":0.1}}
|
|
934
|
+
]' "0.90" "0.75" "$PUBLIC_RUBRIC")
|
|
935
|
+
printf '%s' "$out" | jq -e '.reason == "jury_not_majority"' >/dev/null
|
|
936
|
+
}
|
|
937
|
+
|
|
938
|
+
@test "lesson gate: works with no rubric at all" {
|
|
939
|
+
local out
|
|
940
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
941
|
+
{"judge_type":"standard","score":0.9,"passed":true},
|
|
942
|
+
{"judge_type":"adversarial","score":0.85,"passed":true}
|
|
943
|
+
]' "0.88" "0.75")
|
|
944
|
+
printf '%s' "$out" | jq -e '.passed == true' >/dev/null
|
|
945
|
+
}
|
|
946
|
+
|
|
947
|
+
@test "lesson gate: a hyphenated criterion name gates correctly" {
|
|
948
|
+
local out
|
|
949
|
+
out=$(librarian_lesson_gate "majority" '[
|
|
950
|
+
{"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"scope-accuracy":0.1}},
|
|
951
|
+
{"judge_type":"adversarial","score":0.9,"passed":true,"criterion_scores":{"scope-accuracy":0.1}}
|
|
952
|
+
]' "0.90" "0.75" '{"criteria":[{"name":"scope-accuracy","weight":1.0,"min_pass":0.7}]}')
|
|
953
|
+
printf '%s' "$out" | jq -e '.failed_criterion == "scope-accuracy"' >/dev/null
|
|
954
|
+
}
|
|
955
|
+
```
|
|
956
|
+
|
|
957
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
958
|
+
|
|
959
|
+
Run: `bats test/bats/librarian-lesson-judge.bats`
|
|
960
|
+
Expected: FAIL — the weighted and floor tests fail. The degrade, empty-panel, no-rubric, and precedence tests pass already, pinning behavior that must survive.
|
|
961
|
+
|
|
962
|
+
- [ ] **Step 3: Widen `librarian_lesson_aggregate`**
|
|
963
|
+
|
|
964
|
+
Replace `plugins/librarian/scripts/lib/librarian-lesson-judge.sh:17-29` with:
|
|
965
|
+
|
|
966
|
+
```bash
|
|
967
|
+
# Aggregate the judges' scores. Returns 1 on an empty panel.
|
|
968
|
+
#
|
|
969
|
+
# With a rubric and per-criterion scores, this is a weighted mean: average the
|
|
970
|
+
# judges on each criterion, weight each criterion's mean, normalize by the
|
|
971
|
+
# weights actually used. Without them it is the plain mean it has always been.
|
|
972
|
+
#
|
|
973
|
+
# A criterion no judge scored contributes nothing and its weight leaves the
|
|
974
|
+
# denominator — absence is not a zero. Scoring it 0 instead would turn a judge
|
|
975
|
+
# that skipped a criterion into one that failed it.
|
|
976
|
+
#
|
|
977
|
+
# Usage: librarian_lesson_aggregate <verdicts_json> [<rubric_json>]
|
|
978
|
+
librarian_lesson_aggregate() {
|
|
979
|
+
local verdicts="${1:-[]}"
|
|
980
|
+
local rubric="${2:-}"
|
|
981
|
+
[ -z "$rubric" ] && rubric='{}'
|
|
982
|
+
|
|
983
|
+
local n
|
|
984
|
+
n=$(printf '%s' "$verdicts" | jq 'length' 2>/dev/null) || return 1
|
|
985
|
+
[[ -z "$n" || "$n" -eq 0 ]] && return 1
|
|
986
|
+
|
|
987
|
+
local weighted
|
|
988
|
+
weighted=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
|
|
989
|
+
. as $v
|
|
990
|
+
| [ ($rubric.criteria // [])[]
|
|
991
|
+
| select((.name | type) == "string" and (.weight | type) == "number")
|
|
992
|
+
| . as $c
|
|
993
|
+
| ([ $v[]
|
|
994
|
+
| select((.criterion_scores | type) == "object")
|
|
995
|
+
| select(.criterion_scores | has($c.name))
|
|
996
|
+
| .criterion_scores[$c.name]
|
|
997
|
+
| select(type == "number") ]) as $scores
|
|
998
|
+
| select(($scores | length) > 0)
|
|
999
|
+
| { w: $c.weight, m: (($scores | add) / ($scores | length)) } ]
|
|
1000
|
+
| (map(.w) | add) as $den
|
|
1001
|
+
| if length == 0 or $den == null or $den <= 0 then empty
|
|
1002
|
+
else (map(.w * .m) | add) / $den
|
|
1003
|
+
end
|
|
1004
|
+
' 2>/dev/null)
|
|
1005
|
+
if [ -n "$weighted" ]; then
|
|
1006
|
+
printf '%s' "$weighted"
|
|
1007
|
+
return 0
|
|
1008
|
+
fi
|
|
1009
|
+
|
|
1010
|
+
printf '%s' "$verdicts" | jq -r '[.[].score] | add / length' 2>/dev/null || return 1
|
|
1011
|
+
}
|
|
1012
|
+
```
|
|
1013
|
+
|
|
1014
|
+
- [ ] **Step 4: Widen `librarian_lesson_gate`**
|
|
1015
|
+
|
|
1016
|
+
Replace the doc comment and signature at `:31-41` with:
|
|
1017
|
+
|
|
1018
|
+
```bash
|
|
1019
|
+
# Decide pass/block from the panel, the aggregate, and the rubric's floors.
|
|
1020
|
+
#
|
|
1021
|
+
# Echoes {"passed": bool, "reason": string} and, when a floor failed, a
|
|
1022
|
+
# "failed_criterion" naming it. Three conditions must hold: the jury clears its
|
|
1023
|
+
# policy, the aggregate clears the threshold, and no criterion sits below its
|
|
1024
|
+
# min_pass.
|
|
1025
|
+
#
|
|
1026
|
+
# The floor is what makes the public tier meaningfully stricter than org. Before
|
|
1027
|
+
# it existed, lesson-promotion-public declared gate_policy `unanimous` for that
|
|
1028
|
+
# purpose and it did nothing at all — see ecosystem-j74.
|
|
1029
|
+
#
|
|
1030
|
+
# Usage: librarian_lesson_gate <gate_policy> <verdicts_json> <aggregate> <threshold> [<rubric_json>]
|
|
1031
|
+
librarian_lesson_gate() {
|
|
1032
|
+
local policy="${1:-majority}"
|
|
1033
|
+
local verdicts="${2:-[]}"
|
|
1034
|
+
local aggregate="${3:-0}"
|
|
1035
|
+
local threshold="${4:-0.75}"
|
|
1036
|
+
local rubric="${5:-}"
|
|
1037
|
+
[ -z "$rubric" ] && rubric='{}'
|
|
1038
|
+
```
|
|
1039
|
+
|
|
1040
|
+
Then replace the threshold check at `:74-80` with:
|
|
1041
|
+
|
|
1042
|
+
```bash
|
|
1043
|
+
# awk for the float comparison: bash cannot compare decimals.
|
|
1044
|
+
if ! awk -v s="$aggregate" -v t="$threshold" 'BEGIN { exit !(s >= t) }'; then
|
|
1045
|
+
printf '{"passed":false,"reason":"below_threshold"}'
|
|
1046
|
+
return 0
|
|
1047
|
+
fi
|
|
1048
|
+
|
|
1049
|
+
# Per-criterion floors, checked last: the jury and the aggregate are both
|
|
1050
|
+
# more actionable to report, so they take precedence.
|
|
1051
|
+
local floor_failed
|
|
1052
|
+
floor_failed=$(printf '%s' "$verdicts" | jq -r --argjson rubric "$rubric" '
|
|
1053
|
+
. as $v
|
|
1054
|
+
| [ ($rubric.criteria // [])[]
|
|
1055
|
+
| select((.name | type) == "string" and (.min_pass | type) == "number")
|
|
1056
|
+
| . as $c
|
|
1057
|
+
| ([ $v[]
|
|
1058
|
+
| select((.criterion_scores | type) == "object")
|
|
1059
|
+
| select(.criterion_scores | has($c.name))
|
|
1060
|
+
| .criterion_scores[$c.name]
|
|
1061
|
+
| select(type == "number") ]) as $scores
|
|
1062
|
+
| select(($scores | length) > 0)
|
|
1063
|
+
| select((($scores | add) / ($scores | length)) < $c.min_pass)
|
|
1064
|
+
| $c.name ]
|
|
1065
|
+
| first // empty
|
|
1066
|
+
' 2>/dev/null) || floor_failed=""
|
|
1067
|
+
|
|
1068
|
+
if [[ -n "$floor_failed" ]]; then
|
|
1069
|
+
printf '{"passed":false,"reason":"criterion_floor","failed_criterion":"%s"}' "$floor_failed"
|
|
1070
|
+
return 0
|
|
1071
|
+
fi
|
|
1072
|
+
|
|
1073
|
+
printf '{"passed":true,"reason":"gate_passed"}'
|
|
1074
|
+
return 0
|
|
1075
|
+
}
|
|
1076
|
+
```
|
|
1077
|
+
|
|
1078
|
+
- [ ] **Step 5: Thread the rubric through the caller**
|
|
1079
|
+
|
|
1080
|
+
In `librarian_lesson_judge`, replace lines 164-166 with:
|
|
1081
|
+
|
|
1082
|
+
```bash
|
|
1083
|
+
local aggregate gate
|
|
1084
|
+
aggregate=$(librarian_lesson_aggregate "$verdicts" "$rubric") || return 2
|
|
1085
|
+
gate=$(librarian_lesson_gate "$policy" "$verdicts" "$aggregate" "$threshold" "$rubric") || {
|
|
1086
|
+
```
|
|
1087
|
+
|
|
1088
|
+
- [ ] **Step 6: Run the tests to verify they pass**
|
|
1089
|
+
|
|
1090
|
+
Run: `bats test/bats/librarian-lesson-judge.bats`
|
|
1091
|
+
Expected: PASS, 55/55.
|
|
1092
|
+
|
|
1093
|
+
- [ ] **Step 7: Prove the guards are falsifiable**
|
|
1094
|
+
|
|
1095
|
+
Three scratch (uncommitted) edits, each reverted after checking:
|
|
1096
|
+
|
|
1097
|
+
1. Delete the `if [[ -n "$floor_failed" ]]` block. The disclosure-floor tests must fail.
|
|
1098
|
+
2. Change `.criterion_scores[$c.name]` to `(.criterion_scores[$c.name] // 0)` and drop the `has()` select. **`scores present but one floored criterion omitted` must fail, while `a criterion scored zero does block` still passes** — that pair is what discriminates absence from zero.
|
|
1099
|
+
|
|
1100
|
+
**Do not expect `a verdict with no criterion_scores key at all` to fail here.** It will not, and that is correct: its fixture is removed by the outer `select((.criterion_scores | type) == "object")` before the `has()` lookup is ever reached. Task 3 shipped with only that weaker test and the guard was consequently unpinned — verified, not hypothetical. If you find yourself editing the no-key test to make it fail, stop: you would be deleting the coverage of the pre-upgrade case to duplicate a test you already have.
|
|
1101
|
+
3. Revert Step 5's rubric threading (pass no rubric from `librarian_lesson_judge`). Any end-to-end judging test that exercises a floor must fail. **If none does, that is a coverage gap — add one before finishing.**
|
|
1102
|
+
|
|
1103
|
+
**Report all three results in your report file.**
|
|
1104
|
+
|
|
1105
|
+
- [ ] **Step 8: Verify the whole suite**
|
|
1106
|
+
|
|
1107
|
+
Run: `npm run test:ci`
|
|
1108
|
+
Expected: exit 0, read from `$?` directly.
|
|
1109
|
+
|
|
1110
|
+
- [ ] **Step 9: Commit**
|
|
1111
|
+
|
|
1112
|
+
```bash
|
|
1113
|
+
git add plugins/librarian/scripts/lib/librarian-lesson-judge.sh test/bats/librarian-lesson-judge.bats
|
|
1114
|
+
git commit -m "feat(librarian): weight and floor the lesson jury's criteria :straight_ruler:"
|
|
1115
|
+
```
|
|
1116
|
+
|
|
1117
|
+
---
|
|
1118
|
+
|
|
1119
|
+
### Task 5: The public tier gets a real floor
|
|
1120
|
+
|
|
1121
|
+
**Files:**
|
|
1122
|
+
- Modify: `plugins/librarian/config.json` (`.librarian.lesson_judging.rubrics[1].gate_policy`)
|
|
1123
|
+
- Modify: `plugins/librarian/scripts/lib/librarian-lesson-rubric.sh:9-13`
|
|
1124
|
+
- Modify: `plugins/librarian/skills/librarian/SKILL.md:106-114`
|
|
1125
|
+
- Modify: `docs/superpowers/specs/2026-08-11-lesson-judging-design.md`
|
|
1126
|
+
- Modify: `docs/superpowers/specs/2026-08-14-criterion-scores-design.md`
|
|
1127
|
+
- Test: `test/bats/librarian-lesson-judge.bats`
|
|
1128
|
+
|
|
1129
|
+
**Interfaces:**
|
|
1130
|
+
- Consumes: `librarian_lesson_gate`'s rubric parameter and `criterion_floor` reason from Task 4.
|
|
1131
|
+
- Produces: nothing later tasks depend on. This is the terminal task.
|
|
1132
|
+
|
|
1133
|
+
Closes `ecosystem-j74`.
|
|
1134
|
+
|
|
1135
|
+
- [ ] **Step 1: Write the failing tests**
|
|
1136
|
+
|
|
1137
|
+
Append to `test/bats/librarian-lesson-judge.bats`:
|
|
1138
|
+
|
|
1139
|
+
```bash
|
|
1140
|
+
@test "the public rubric no longer relies on unanimous" {
|
|
1141
|
+
# unanimous was a stand-in for the disclosure floor and never worked: at the
|
|
1142
|
+
# configured panel of 2, unanimous and majority are the same function for
|
|
1143
|
+
# every possible pass count. ecosystem-j74.
|
|
1144
|
+
local policy
|
|
1145
|
+
policy=$(jq -r '.librarian.lesson_judging.rubrics[]
|
|
1146
|
+
| select(.id == "lesson-promotion-public") | .gate_policy' \
|
|
1147
|
+
"${REPO_ROOT}/plugins/librarian/config.json")
|
|
1148
|
+
[ "$policy" = "majority" ]
|
|
1149
|
+
}
|
|
1150
|
+
|
|
1151
|
+
@test "the public rubric keeps disclosure's floor at 0.9" {
|
|
1152
|
+
# The floor is now the ONLY thing making public stricter than org. If this
|
|
1153
|
+
# drops, the public tier silently loses its protection entirely.
|
|
1154
|
+
local floor
|
|
1155
|
+
floor=$(jq -r '.librarian.lesson_judging.rubrics[]
|
|
1156
|
+
| select(.id == "lesson-promotion-public")
|
|
1157
|
+
| .criteria[] | select(.name == "disclosure") | .min_pass' \
|
|
1158
|
+
"${REPO_ROOT}/plugins/librarian/config.json")
|
|
1159
|
+
[ "$floor" = "0.9" ]
|
|
1160
|
+
}
|
|
1161
|
+
|
|
1162
|
+
@test "org and public rubrics differ by more than their gate policy" {
|
|
1163
|
+
# Both are `majority` now. If the criteria ever converge too, the two tiers
|
|
1164
|
+
# become indistinguishable and the public tier is inert again — the exact
|
|
1165
|
+
# shape of j74.
|
|
1166
|
+
local org_crit pub_crit
|
|
1167
|
+
org_crit=$(jq -c '[.librarian.lesson_judging.rubrics[]
|
|
1168
|
+
| select(.id == "lesson-promotion") | .criteria[].name] | sort' \
|
|
1169
|
+
"${REPO_ROOT}/plugins/librarian/config.json")
|
|
1170
|
+
pub_crit=$(jq -c '[.librarian.lesson_judging.rubrics[]
|
|
1171
|
+
| select(.id == "lesson-promotion-public") | .criteria[].name] | sort' \
|
|
1172
|
+
"${REPO_ROOT}/plugins/librarian/config.json")
|
|
1173
|
+
[ "$org_crit" != "$pub_crit" ]
|
|
1174
|
+
}
|
|
1175
|
+
|
|
1176
|
+
@test "the librarian walk tells judges to return criterion_scores" {
|
|
1177
|
+
# The rubric floors are unreachable unless the judges actually score them,
|
|
1178
|
+
# and this walk is where librarian's judges get their instructions.
|
|
1179
|
+
grep -q 'criterion_scores' "${REPO_ROOT}/plugins/librarian/skills/librarian/SKILL.md"
|
|
1180
|
+
}
|
|
1181
|
+
```
|
|
1182
|
+
|
|
1183
|
+
- [ ] **Step 2: Run the tests to verify they fail**
|
|
1184
|
+
|
|
1185
|
+
Run: `bats test/bats/librarian-lesson-judge.bats`
|
|
1186
|
+
Expected: FAIL — the gate-policy test and the SKILL.md test fail. The floor and criteria-differ tests pass already, pinning what must not regress.
|
|
1187
|
+
|
|
1188
|
+
- [ ] **Step 2b: Carry `failed_criterion` into the on-disk verdict**
|
|
1189
|
+
|
|
1190
|
+
Found by Task 4's review. `librarian_lesson_gate` computes `failed_criterion`, but `librarian_lesson_judge` discards it — its verdict block extracts only `.passed` and `.reason`. So a rejected proposal records `"reason":"criterion_floor"` with no indication of *which* criterion, which is exactly the ambiguity `criterion_floor` was introduced to remove. Tribunal's walk copies the field onto its event for this reason; librarian had no counterpart.
|
|
1191
|
+
|
|
1192
|
+
First, the failing test — append to `test/bats/librarian-lesson-judge.bats`:
|
|
1193
|
+
|
|
1194
|
+
```bash
|
|
1195
|
+
@test "a floor rejection records which criterion failed" {
|
|
1196
|
+
# "reason": "criterion_floor" without the criterion name is no more
|
|
1197
|
+
# actionable than "blocked" — the whole argument for a distinct reason
|
|
1198
|
+
# was naming the thing that failed.
|
|
1199
|
+
_seed_confirmed "floorname01" "public"
|
|
1200
|
+
local verdicts='[
|
|
1201
|
+
{"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9,"disclosure":0.4}},
|
|
1202
|
+
{"judge_type":"adversarial","score":0.85,"passed":true,"criterion_scores":{"grounding":0.9,"scope_accuracy":0.9,"generality":0.85,"disclosure":0.4}}
|
|
1203
|
+
]'
|
|
1204
|
+
run librarian_lesson_judge "$PROJECT_KEY" "floorname01" "$verdicts"
|
|
1205
|
+
[ "$status" -eq 0 ] || return 1
|
|
1206
|
+
[ "$(_status_of floorname01)" = "rejected" ] || return 1
|
|
1207
|
+
|
|
1208
|
+
local path
|
|
1209
|
+
path="$(librarian_lessons_dir "$PROJECT_KEY")/proposals/floorname01.json"
|
|
1210
|
+
jq -e '.verdict.reason == "criterion_floor"' "$path" >/dev/null || return 1
|
|
1211
|
+
jq -e '.verdict.failed_criterion == "disclosure"' "$path" >/dev/null
|
|
1212
|
+
}
|
|
1213
|
+
|
|
1214
|
+
@test "an approved verdict carries no failed_criterion key" {
|
|
1215
|
+
# Absent rather than null: a key that is always present but usually empty
|
|
1216
|
+
# invites `// ""` at the read site, which is how this pipeline has
|
|
1217
|
+
# repeatedly lost the absent-vs-empty distinction.
|
|
1218
|
+
_seed_confirmed "floorname02" "public"
|
|
1219
|
+
local verdicts='[
|
|
1220
|
+
{"judge_type":"standard","score":0.9,"passed":true,"criterion_scores":{"grounding":0.95,"scope_accuracy":0.95,"generality":0.9,"disclosure":0.95}},
|
|
1221
|
+
{"judge_type":"adversarial","score":0.9,"passed":true,"criterion_scores":{"grounding":0.9,"scope_accuracy":0.9,"generality":0.9,"disclosure":0.95}}
|
|
1222
|
+
]'
|
|
1223
|
+
run librarian_lesson_judge "$PROJECT_KEY" "floorname02" "$verdicts"
|
|
1224
|
+
[ "$status" -eq 0 ] || return 1
|
|
1225
|
+
|
|
1226
|
+
local path
|
|
1227
|
+
path="$(librarian_lessons_dir "$PROJECT_KEY")/proposals/floorname02.json"
|
|
1228
|
+
jq -e '.verdict | has("failed_criterion") | not' "$path" >/dev/null
|
|
1229
|
+
}
|
|
1230
|
+
```
|
|
1231
|
+
|
|
1232
|
+
These use the file's existing helpers — `_seed_confirmed <id> <visibility>` (line 138), `_status_of <id>` (line 154), and `$PROJECT_KEY` — verified against the neighboring end-to-end test at line 690. Do not invent new fixtures.
|
|
1233
|
+
|
|
1234
|
+
Run them: the first must fail on the missing `failed_criterion` key; the second passes already, since it pins behavior that must not regress.
|
|
1235
|
+
|
|
1236
|
+
Then, in `plugins/librarian/scripts/lib/librarian-lesson-judge.sh`, change the verdict construction inside `librarian_lesson_judge` from:
|
|
1237
|
+
|
|
1238
|
+
```bash
|
|
1239
|
+
'{rubric_id: $r, gate_policy: $p, score_threshold: $th,
|
|
1240
|
+
aggregate_score: $ag, passed: $g.passed, reason: $g.reason,
|
|
1241
|
+
judges: $j}') || {
|
|
1242
|
+
```
|
|
1243
|
+
|
|
1244
|
+
to:
|
|
1245
|
+
|
|
1246
|
+
```bash
|
|
1247
|
+
'{rubric_id: $r, gate_policy: $p, score_threshold: $th,
|
|
1248
|
+
aggregate_score: $ag, passed: $g.passed, reason: $g.reason,
|
|
1249
|
+
judges: $j}
|
|
1250
|
+
+ (if $g.failed_criterion then {failed_criterion: $g.failed_criterion} else {} end)') || {
|
|
1251
|
+
```
|
|
1252
|
+
|
|
1253
|
+
The `+ (if … else {} end)` form keeps the key **absent** on a non-floor verdict rather than present-and-null. Re-run: both tests pass.
|
|
1254
|
+
|
|
1255
|
+
- [ ] **Step 3: Swap the public tier's gate policy**
|
|
1256
|
+
|
|
1257
|
+
In `plugins/librarian/config.json`, in the rubric with `"id": "lesson-promotion-public"`, change:
|
|
1258
|
+
|
|
1259
|
+
```json
|
|
1260
|
+
"gate_policy": "unanimous",
|
|
1261
|
+
```
|
|
1262
|
+
|
|
1263
|
+
to:
|
|
1264
|
+
|
|
1265
|
+
```json
|
|
1266
|
+
"gate_policy": "majority",
|
|
1267
|
+
```
|
|
1268
|
+
|
|
1269
|
+
Leave every criterion, weight, floor and `score_threshold` exactly as they are.
|
|
1270
|
+
|
|
1271
|
+
- [ ] **Step 4: Correct the rubric library's comment**
|
|
1272
|
+
|
|
1273
|
+
Replace `plugins/librarian/scripts/lib/librarian-lesson-rubric.sh:9-13` — which claims the weights are inert — with:
|
|
1274
|
+
|
|
1275
|
+
```bash
|
|
1276
|
+
# The per-criterion weights and min_pass floors are LIVE as of ecosystem-pht:
|
|
1277
|
+
# librarian_lesson_aggregate weights them and librarian_lesson_gate blocks on
|
|
1278
|
+
# any criterion below its floor.
|
|
1279
|
+
#
|
|
1280
|
+
# `disclosure` at min_pass 0.9 is what makes the public tier stricter than org.
|
|
1281
|
+
# It replaces gate_policy `unanimous`, which was intended as a stand-in for
|
|
1282
|
+
# exactly this and turned out to be a no-op: at the configured two-judge panel,
|
|
1283
|
+
# `unanimous` and `majority` agree on every possible pass count. See
|
|
1284
|
+
# ecosystem-j74. Changing judge_types without re-reading that bead is how the
|
|
1285
|
+
# hole reopens.
|
|
1286
|
+
```
|
|
1287
|
+
|
|
1288
|
+
- [ ] **Step 5: Tell librarian's judges to score the criteria**
|
|
1289
|
+
|
|
1290
|
+
In `plugins/librarian/skills/librarian/SKILL.md`, replace the bullet at lines 112-114 with:
|
|
1291
|
+
|
|
1292
|
+
```markdown
|
|
1293
|
+
- Each judge returns a JSON object with `score`, `passed`, `judge_type`,
|
|
1294
|
+
`feedback_summary`, and `criterion_scores` — a map from **each rubric
|
|
1295
|
+
criterion name you gave it** to a score in `[0,1]`. Tell each judge
|
|
1296
|
+
explicitly to omit any criterion it cannot assess rather than scoring it
|
|
1297
|
+
`0`: a `0` on `disclosure` blocks a public lesson by itself, while an
|
|
1298
|
+
omission does not. Collect both verdicts into a JSON array **verbatim** —
|
|
1299
|
+
never summarize or reconstruct a judge's verdict.
|
|
1300
|
+
```
|
|
1301
|
+
|
|
1302
|
+
- [ ] **Step 6: Correct the 4z8.3 spec**
|
|
1303
|
+
|
|
1304
|
+
In `docs/superpowers/specs/2026-08-11-lesson-judging-design.md`, find the passage describing `gate_policy: unanimous` as the public tier's protection and replace its claim with:
|
|
1305
|
+
|
|
1306
|
+
```markdown
|
|
1307
|
+
**Correction (2026-08-14).** This section described `unanimous` as delivering
|
|
1308
|
+
"a single judge's objection cannot be outvoted." It never did. Both rubrics
|
|
1309
|
+
declare `judge_types: ["standard", "adversarial"]` — a panel of two — and at
|
|
1310
|
+
panel size two `unanimous` (`passed == count`) and `majority`
|
|
1311
|
+
(`passed * 2 > count`) return the same answer for all three possible pass
|
|
1312
|
+
counts. They diverge only at three judges or more, and `librarian_lesson_judge`
|
|
1313
|
+
refuses any panel whose judge-type multiset does not match the rubric's, so a
|
|
1314
|
+
third judge never reaches the gate.
|
|
1315
|
+
|
|
1316
|
+
The public tier was never stricter than the org tier. `ecosystem-pht` replaced
|
|
1317
|
+
the stand-in with `disclosure`'s real `min_pass` floor and the policy is now
|
|
1318
|
+
`majority`. Tracked as `ecosystem-j74`.
|
|
1319
|
+
```
|
|
1320
|
+
|
|
1321
|
+
- [ ] **Step 7: Correct the criterion-scores spec**
|
|
1322
|
+
|
|
1323
|
+
In `docs/superpowers/specs/2026-08-14-criterion-scores-design.md`, the "Judges emit it" section says `tribunal-judge-standard` and `-security` already receive the rubric. Only `-standard` does. Replace that sentence with:
|
|
1324
|
+
|
|
1325
|
+
```markdown
|
|
1326
|
+
`tribunal-judge-standard` already receives the rubric with its criteria and is
|
|
1327
|
+
told to "score each criterion in [0,1]". It has nowhere to put the result; its
|
|
1328
|
+
output contract gains `criterion_scores`.
|
|
1329
|
+
|
|
1330
|
+
**Neither `tribunal-judge-adversarial` nor `tribunal-judge-security` has a
|
|
1331
|
+
rubric section at all.** Both report `criteria_evaluated` lists drawn from their
|
|
1332
|
+
own investigative lenses — `edge-cases`, `concurrency`, `idempotency` for one;
|
|
1333
|
+
`injection`, `secrets`, `path-traversal` for the other — disjoint from every
|
|
1334
|
+
rubric in the repo.
|
|
1335
|
+
```
|
|
1336
|
+
|
|
1337
|
+
- [ ] **Step 8: Run the tests to verify they pass**
|
|
1338
|
+
|
|
1339
|
+
Run: `bats test/bats/librarian-lesson-judge.bats`
|
|
1340
|
+
Expected: PASS, 59/59.
|
|
1341
|
+
|
|
1342
|
+
- [ ] **Step 9: Prove the swap actually changed behavior**
|
|
1343
|
+
|
|
1344
|
+
The gate-policy test only reads config. Confirm the swap changes an outcome: temporarily set `gate_policy` back to `unanimous` in a scratch edit and re-run the two public-tier gate tests from Task 4. **They must still pass** — because at panel 2 the policies are identical, which is the whole finding. Record that result in your report file as direct evidence that the floor, not the policy, is doing the work now.
|
|
1345
|
+
|
|
1346
|
+
- [ ] **Step 10: Verify the whole suite**
|
|
1347
|
+
|
|
1348
|
+
Run: `npm run test:ci`
|
|
1349
|
+
Expected: exit 0, read from `$?` directly.
|
|
1350
|
+
|
|
1351
|
+
- [ ] **Step 11: Commit**
|
|
1352
|
+
|
|
1353
|
+
```bash
|
|
1354
|
+
git add plugins/librarian/config.json plugins/librarian/scripts/lib/librarian-lesson-rubric.sh plugins/librarian/skills/librarian/SKILL.md docs/superpowers/specs/2026-08-11-lesson-judging-design.md docs/superpowers/specs/2026-08-14-criterion-scores-design.md test/bats/librarian-lesson-judge.bats
|
|
1355
|
+
git commit -m "feat(librarian): give the public tier a floor that actually holds :closed_lock_with_key:"
|
|
1356
|
+
```
|
|
1357
|
+
|
|
1358
|
+
---
|
|
1359
|
+
|
|
1360
|
+
## After the tasks
|
|
1361
|
+
|
|
1362
|
+
- Close `ecosystem-pht` and `ecosystem-j74`.
|
|
1363
|
+
- Open a PR into `main` per the repo's PR-only rule. The PR body should lead with the j74 finding — a reviewer who does not know the `unanimous` stand-in was inert will read the `majority` swap as a loosening rather than a tightening.
|
|
1364
|
+
- Not in scope, deliberately: retiring `criteria_evaluated`; any new judge type; widening `judge_types` to three (rejected — it buys a property this plan delivers anyway, at the cost of a third Opus judge per public candidate); and the `TODO(ONL-6)` hard-fail on type/schema divergence, which is the schema repo's own cleanup.
|
|
1365
|
+
|
|
1366
|
+
## Self-review
|
|
1367
|
+
|
|
1368
|
+
**Spec coverage.** Payload consumption → Tasks 2, 3, 4. Judges emit it → Task 1. `weighted_mean` real → Task 2. Normalize by weight sum → Tasks 2 and 4. Degrade to mean → Tasks 2 and 4. `min_pass` enforceable → Tasks 3 and 4. Absent must not block → Tasks 3 and 4. Public tier's real floor → Task 5. Doc corrections → Task 5. Dependency bump → Task 1. Every test the spec calls for maps to a step, including the two it gained in correction (hyphenated names, unscored floors).
|
|
1369
|
+
|
|
1370
|
+
**One gap the spec left open, resolved here:** it never said whether judges score *rubric* criteria or their own lenses. They score rubric criteria; `criteria_evaluated` keeps its present meaning. Without that decision `criterion_scores` keys would never match a floor.
|
|
1371
|
+
|
|
1372
|
+
**Type consistency.** `criterion_scores` is an object of name → number everywhere. `failed_criterion` is a string, present only alongside `reason: "criterion_floor"`, in both plugins. Both new parameters are optional and trailing: `tribunal_aggregate` position 3 (existing, now used), `tribunal_gate_decide` position 8 (new), `librarian_lesson_aggregate` position 2 (new), `librarian_lesson_gate` position 5 (new).
|
|
1373
|
+
|
|
1374
|
+
**Precedence differs between the plugins, deliberately.** Tribunal: `low_score` → `criterion_floor` → jury reasons. Librarian: jury → `below_threshold` → `criterion_floor`. Each preserves its own existing order and appends the floor where that order allows; unifying them would change shipped behavior in one of the two.
|