@onlooker-community/ecosystem 0.43.2 → 0.43.3

package/docs/superpowers/specs/2026-08-21-schema-emission-harness-design.md

@@ -0,0 +1,255 @@
+# Repo-Wide Schema Emission Harness — Design
+
+**Status:** Approved, not started.
+**Tracked by:** `ecosystem-u0t`.
+**Single repository.** Everything lands under `scripts/lib/`, `test/`, and
+`package.json`.
+
+---
+
+## What this is
+
+Three bugs now share one root cause. A hook built a payload that diverged from
+what the published schema admits; the emitter rejected it; the hook swallowed
+the rejection and exited 0; the on-disk artifacts kept working. Nothing looked
+broken.
+
+- `ecosystem-q4d` (P1, closed) — both cartographer payloads off-contract.
+- `ecosystem-1p1` (P2, closed) — librarian `scan.complete` emitted
+  `outcome:budget_exceeded`, which the enum rejects.
+- `ecosystem-ci0` (P3, open) — cartographer `finding_type:"unknown"`, admitted
+  by no schema.
+
+Each fix added its own bats test driving a real payload through
+`scripts/lib/onlooker-event.mjs`. That guarantee stops at the plugin being
+repaired. Nothing generalizes it, so the next divergence ships the same way:
+silently, with green tests and working artifacts. Every instance so far was
+found by hand, long after it landed, and each cost a two-repo release cycle.
+
+This builds one harness that makes the class detectable.
+
+## Why the obvious seam is the wrong one
+
+`ecosystem-u0t` proposes enumerating payload builders per plugin, noting that
+the `q4d` fix extracted cartographer's builders out of `run_emit` "precisely so
+tests and production share one construction — that pattern is the likely seam,
+but not every plugin has been refactored that way yet."
+
+The seam does not exist. Counting named `*_payload()` builders across every
+`plugins/*/scripts/lib/*-events.sh`:
+
+| Plugin | Builders |
+|--------|----------|
+| cartographer | 3 |
+| every other events lib (12) | 0 |
+
+Three plugins — curator, historian, librarian — have no events lib at all.
+Everywhere except cartographer, the payload is assembled inline at the call
+site and handed to `<plugin>_emit_event` as a finished JSON string.
+
+Choosing that seam means refactoring fifteen plugins before writing a single
+assertion, and the refactor would be driven by a test rather than by need.
+Rejected.
+
+## The failure signature
+
+The rejection is not a bad line on the bus. It is the **absence** of a line.
+
+Probing the emitter directly with a payload that violates
+`governor.gate.checked`:
+
+```
+EMITTER EXIT=1
+stderr: [ { "path": "/decision",
+            "message": "must be equal to one of the allowed values" }, ... ]
+bus:    NO LOG FILE — event was dropped
+```
+
+Per [ADR-005](../../adr/005-runtime-emitter-fails-open.md), validation is
+attempted through a lazy `await import('@onlooker-community/schema')`. Where
+the package resolves — dev, CI, tests — the emitter validates and **rejects**.
+Where it does not, it emits anyway. So in CI an invalid payload never reaches
+the log, and sweeping the log for invalid lines would find nothing, ever.
+
+Hooks fail soft and exit 0 by design, which discards the emitter's exit 1 and
+its stderr. That is the whole bug: the only two signals the emitter produces
+are both destroyed before anyone can see them. `ecosystem-1p1` established the
+usable property in the other direction — presence on the bus proves the payload
+validated.
+
+The harness therefore has to capture the rejection at the point it happens, in
+a place that outlives the hook.
+
+## Architecture
+
+```
+107 bats files -> real hooks -> real branches
+                                     |
+                      scripts/lib/onlooker-event.mjs
+                                     |
+                            tryValidate(event)
+                     +---------------+---------------+
+                  valid                          rejected
+                     |                               |
+               bus (per-test,                  dropped, exit 1,
+                ephemeral)                    stderr swallowed
+                     +---------------+---------------+
+                                     |
+             $ONLOOKER_TEST_REPORT_DIR/emissions.jsonl   <- durable
+                  { event_type, validated, valid, errors? }
+                                     |
+                      Gate A                    Gate B
+                 no valid:false line     manifest accounts for 125
+```
+
+## Why a durable report directory
+
+`test/helpers/setup.bash:15` points `ONLOOKER_DIR` at
+`${BATS_TEST_TMPDIR}/home/.onlooker`. Bats deletes that tree after **each
+test**, so a sidecar written under `$ONLOOKER_DIR` dies with the test that
+produced it and neither gate could read it afterward.
+
+A shared bats `teardown` was considered and rejected. Teardown is per-file: a
+definition in the shared helper would clobber the one bats file that already
+defines its own, and any future file defining a teardown would silently opt out
+of the gate — the same silent-opt-out failure mode that produced `u0t`.
+
+Instead the suite exports `ONLOOKER_TEST_REPORT_DIR` once, outside
+`BATS_TEST_TMPDIR`. One mechanism then feeds both gates, with no per-test
+opt-in to forget.
+
+## The emitter change
+
+In `scripts/lib/onlooker-event.mjs`, after `tryValidate`: when
+`ONLOOKER_TEST_REPORT_DIR` is set, append one JSON line recording the event
+type, whether validation actually ran, whether it passed, and any ajv errors.
+
+Constraints:
+
+- Guarded entirely on the environment variable. Production never sets it, so
+  production behavior, exit codes, and the fail-open contract are untouched.
+- One `appendFileSync` of one line, so the record is atomic under POSIX
+  `O_APPEND` for the line sizes involved.
+- Records **`validated`** separately from **`valid`**. This is not redundant;
+  see below.
+
+Roughly twelve lines. It is the only production file this design touches.
+
+## Gate A — no rejected emission
+
+Fails if any line in `emissions.jsonl` has `valid:false`, reporting the event
+type and the ajv errors that caused it.
+
+It additionally asserts that **at least one line recorded `validated:true`**.
+Without that assertion the gate is a trap. If `node_modules` is ever absent,
+`tryValidate` fails open, nothing validates, no line is ever `valid:false`, and
+Gate A passes green while checking nothing at all — reproducing the exact bug it
+exists to catch, one level up. Recording whether validation ran is what makes
+that state detectable rather than indistinguishable from success.
+
+## Gate B — every registered type accounted for
+
+A committed manifest, `test/bus-coverage.json`:
+
+```json
+{
+  "expected": ["session.start", "governor.gate.checked", "..."],
+  "excluded": {
+    "meridian.hint.generated": "meridian plugin lives in another repo",
+    "curator.finding.contradiction": "LLM contradiction sweep deferred",
+    "compass.check.overridden": "compass has no implementation yet"
+  }
+}
+```
+
+Gate B asserts three things:
+
+1. Every type in `expected` appears in `emissions.jsonl` with `valid:true`.
+2. `expected` and `excluded` together equal `ALL_EVENT_TYPES` exactly — no
+   gaps, no extras.
+3. Every exclusion carries a non-empty reason.
+
+Assertion 2 is the anti-drift property. A newly registered schema type belongs
+to neither list, so CI fails until someone triages it deliberately. This is the
+same shape as the `test:shellcheck` fix in #190: derive coverage from the
+authoritative set rather than from a hand-maintained list that rots.
+
+All three assertions are hard from the first commit: the manifest is authored
+during implementation precisely to make them true. What ratchets is the
+manifest itself, as types move from `excluded` to `expected` when the features
+that emit them land.
+
+## Where the gates run
+
+The gates become their own npm script, `test:bus`, sequenced after `test:bats`
+in `test:ci`. It **fails when the report is missing**.
+
+They deliberately do not live inside `test:schema`. A node test that skips when
+the report is absent would pass vacuously for anyone running `test:schema`
+standalone — silent skipping being, again, the failure mode under repair.
+
+`test:bats` clears and exports the report directory; the path is gitignored.
+
+## Bootstrapping the manifest
+
+Of 125 registered types, 75 are referenced in this repo's source and 50 are
+not. The 50 split three ways, and each group takes a different disposition:
+
+| Group | Count | Disposition |
+|-------|-------|-------------|
+| Foreign plugins — meridian 6, sentinel 3, oracle 2, relay 2 | 13 | `excluded`, "plugin lives in another repo" |
+| In-repo but unemitted — curator 11, tribunal 6, archivist 5, governor 4, historian 3, librarian 2, compass 2, and three singles | 36 | `excluded`, with the specific reason |
+| `onlooker.session.summary` | 1 | `excluded`, emitted by the agent, not this repo |
+
+Curator's eleven are deferred by design; compass has no implementation at all
+and is still in its design phase. Those move to `expected` as the features land,
+which is the ratchet working as intended.
+
+Note that source reference is not emission. The initial `expected` list is
+seeded from what the first instrumented run actually observes, not from the 75
+grep hits — a type named only in a comment or a doc would otherwise be demanded
+to have an example it can never produce.
+
+## Testing the harness itself
+
+Three checks, each broken on purpose before being trusted:
+
+1. Feed a known-bad payload through the emitter with the report directory set;
+   confirm a `valid:false` line appears and Gate A goes red.
+2. Remove a type from `expected` that the suite does emit; confirm assertion 2
+   fails on the resulting gap.
+3. Simulate the schema package being unresolvable; confirm Gate A fails on the
+   absence of any `validated:true` line rather than passing green.
+
+Check 3 is the one that matters most, because it is the only one that tests the
+harness against its own failure mode.
+
+## Out of scope
+
+- **Refactoring fifteen plugins to extract payload builders.** The integration
+  seam makes it unnecessary.
+- **Forcing coverage of the 36 unemitted in-repo types now.** They are
+  excluded with reasons and promoted as features land.
+- **Parallel bats.** The suite runs serially. Running `bats -j` would need
+  append-atomicity revisited; not addressed here.
+- **The `-S error` shellcheck severity question.** Unrelated, separately filed.
+
+## Acceptance
+
+- `npm run test:ci` runs `test:bus` after `test:bats` and fails when the report
+  is missing.
+- Gate A fails on a rejected emission, naming the type and the ajv errors.
+- Gate A fails when validation never ran, rather than passing vacuously.
+- Gate B fails when a registered type appears in neither manifest list.
+- `ecosystem-ci0`'s `finding_type:"unknown"` payload is caught by Gate A once
+  a test exercises that branch, without a bespoke test being written for it.
+- Production emission behavior is unchanged: no new dependency, no exit-code
+  change, and nothing written unless `ONLOOKER_TEST_REPORT_DIR` is set.
+
+## Documentation to update
+
+- `docs/architecture.md` — how the bus gates fit the event pipeline.
+- `CLAUDE.md` — the "Adding a new plugin" checklist gains a step: triage any
+  new event type into `test/bus-coverage.json`.
+- `.claude/skills/writing-tests/SKILL.md` — note that emissions are gated
+  suite-wide, so new plugins need no bespoke schema test.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@onlooker-community/ecosystem",
-  "version": "0.43.2",
+  "version": "0.43.3",
   "description": "Agents, skills, hooks, commands, rules, and MCP configurations that power [Onlooker](https://onlooker.dev)",
   "author": {
     "name": "Onlooker Community",
@@ -20,16 +20,18 @@
   },
   "scripts": {
     "test": "npm run test:bats && npm run test:schema",
-    "test:bats": "bats test/bats",
-    "test:schema": "node --test test/node/*.test.mjs",
+    "test:bats": "rm -rf \"$PWD/test/tmp-emission-report\" && ONLOOKER_TEST_REPORT_DIR=\"$PWD/test/tmp-emission-report\" bats test/bats",
+    "test:schema": "ONLOOKER_TEST_REPORT_DIR=\"$PWD/test/tmp-emission-report\" node --test test/node/*.test.mjs",
+    "test:bus": "node scripts/lint/check-bus-coverage.mjs",
     "test:shellcheck": "git ls-files -z '*.sh' | xargs -0 shellcheck -S error -x",
     "lint:references": "node scripts/lint/check-references.mjs",
     "lint:manifests": "node scripts/lint/check-manifests.mjs",
     "lint:lesson-schema": "node scripts/lint/check-lesson-schema-drift.mjs",
+    "lint:managed-blocks": "node scripts/lint/check-managed-blocks.mjs",
     "coverage:node": "node scripts/coverage/run-coverage.mjs",
     "coverage:bash": "node scripts/coverage/bash-coverage.mjs",
     "coverage": "npm run coverage:node && npm run coverage:bash",
-    "test:ci": "npm run test:shellcheck && npm run test:bats && npm run test:schema && npm run lint:check && npm run lint:manifests && npm run lint:references && npm run lint:lesson-schema",
+    "test:ci": "npm run test:shellcheck && npm run test:bats && npm run test:schema && npm run test:bus && npm run lint:managed-blocks && npm run lint:check && npm run lint:manifests && npm run lint:references && npm run lint:lesson-schema",
     "lint:check": "biome check . && markdownlint '**/*.md'",
     "lint": "biome lint --write && markdownlint --fix '**/*.md'",
     "format": "biome format --write && markdownlint --fix '**/*.md'",

package/scripts/lib/onlooker-event.mjs

@@ -4,7 +4,7 @@
  * Uses @onlooker-community/schema for envelope shape and validation.
  */
 import { randomUUID } from 'node:crypto';
-import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';
+import { appendFileSync, existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';
 import { join } from 'node:path';
 
 // Canonical event-type constants, inlined rather than imported from
@@ -105,6 +105,41 @@ async function tryValidate(event) {
   return { available: true, ...schema.validate(event) };
 }
 
+/**
+ * Test-only emission report.
+ *
+ * When ONLOOKER_TEST_REPORT_DIR is set, append one line per emission recording
+ * the event type, whether validation actually ran, and whether it passed.
+ *
+ * This exists because both signals the emitter produces on rejection — a
+ * non-zero exit and stderr — are destroyed by the hook's fail-soft exit 0,
+ * which leaves a dropped event indistinguishable from one that never fired.
+ * The report lives outside the per-test BATS_TEST_TMPDIR so the suite can gate
+ * on it after the fact.
+ *
+ * Production never sets the variable, so nothing is written there and the
+ * fail-open contract in ADR-005 is untouched. `validated` is recorded
+ * separately from `valid` so a run where the schema package never resolved is
+ * distinguishable from a run where everything passed — without it, a missing
+ * node_modules would make the gate pass while checking nothing.
+ */
+function recordEmission(event, check) {
+  const dir = process.env.ONLOOKER_TEST_REPORT_DIR;
+  if (!dir) return;
+  const record = {
+    event_type: event?.event_type ?? null,
+    validated: check.available === true,
+    valid: check.available === true ? check.valid === true : null,
+  };
+  if (check.available && !check.valid) record.errors = check.errors;
+  try {
+    mkdirSync(dir, { recursive: true });
+    appendFileSync(join(dir, 'emissions.jsonl'), `${JSON.stringify(record)}\n`);
+  } catch {
+    // A broken report must never break an emission.
+  }
+}
+
 function summarizeText(value, maxLen = 1000) {
   if (value == null) return undefined;
   const text = String(value).replace(/\s+/g, ' ').trim();
@@ -510,6 +545,7 @@ async function main() {
     // Best-effort: reject when a validator is present (dev/CI), fail open in
     // installed plugins so the event is still emitted.
     const check = await tryValidate(mapped.event);
+    recordEmission(mapped.event, check);
     if (check.available && !check.valid) {
       console.error(JSON.stringify(check.errors, null, 2));
       process.exit(1);
@@ -538,6 +574,7 @@ async function main() {
     // Best-effort: reject when a validator is present (dev/CI), fail open in
     // installed plugins so the event is still emitted.
     const check = await tryValidate(event);
+    recordEmission(event, check);
     if (check.available && !check.valid) {
       console.error(JSON.stringify(check.errors, null, 2));
       process.exit(1);

package/scripts/lint/check-bus-coverage.mjs

@@ -0,0 +1,153 @@
+#!/usr/bin/env node
+// Bus coverage gates.
+//
+// Gate A: no emission recorded during the suite was rejected by the schema.
+//
+// Reads the report the emitter writes when ONLOOKER_TEST_REPORT_DIR is set —
+// see recordEmission in scripts/lib/onlooker-event.mjs. A rejected emission is
+// invisible any other way: the emitter exits 1 and prints ajv errors, and the
+// hook's fail-soft exit 0 destroys both.
+//
+// Gate B: every registered event type is accounted for — either it produced
+// a validated emission during the suite, or the manifest excuses it with a
+// stated reason. See test/bus-coverage.json.
+//
+// Exit codes:
+//   0  ok
+//   1  gate A or gate B failure
+//   2  unknown argument
+//
+// Usage: check-bus-coverage.mjs [--report <dir>] [--manifest <path>]
+import { existsSync, readFileSync } from 'node:fs';
+import { dirname, join, resolve } from 'node:path';
+import { fileURLToPath } from 'node:url';
+
+const HERE = dirname(fileURLToPath(import.meta.url));
+const REPO_ROOT = resolve(HERE, '..', '..');
+
+function parseArgs(argv) {
+  const out = {
+    report: join(REPO_ROOT, 'test', 'tmp-emission-report'),
+    manifest: join(REPO_ROOT, 'test', 'bus-coverage.json'),
+  };
+  for (let i = 2; i < argv.length; i += 1) {
+    const a = argv[i];
+    if (a === '--report') out.report = argv[++i];
+    else if (a === '--manifest') out.manifest = argv[++i];
+    else if (a === '--help') {
+      process.stderr.write('Usage: check-bus-coverage.mjs [--report <dir>] [--manifest <path>]\n');
+      process.exit(0);
+    } else {
+      process.stderr.write(`check-bus-coverage: unknown argument: ${a}\n`);
+      process.exit(2);
+    }
+  }
+  return out;
+}
+
+function loadReport(dir) {
+  const p = join(dir, 'emissions.jsonl');
+  if (!existsSync(p)) return [];
+  return readFileSync(p, 'utf8')
+    .trim()
+    .split('\n')
+    .filter(Boolean)
+    .map((l) => JSON.parse(l));
+}
+
+function gateA(lines) {
+  const failures = [];
+  if (lines.length === 0) {
+    failures.push('no emissions recorded — run `npm run test:bats` with ONLOOKER_TEST_REPORT_DIR set');
+    return failures;
+  }
+  if (!lines.some((l) => l.validated === true)) {
+    failures.push(
+      'no emission was validated: @onlooker-community/schema did not resolve, so this gate ' +
+        'checked nothing. Run `npm ci` and try again.',
+    );
+  }
+  for (const l of lines.filter((x) => x.valid === false)) {
+    failures.push(`rejected emission: ${l.event_type} — ${JSON.stringify(l.errors)}`);
+  }
+  return failures;
+}
+
+/**
+ * Gate B: every registered event type is accounted for.
+ *
+ * `expected` types must have produced a validated emission during the suite.
+ * `excluded` types must carry a reason and must NOT have been emitted —
+ * otherwise coverage is silently under-claimed and the manifest can drift
+ * downward without CI noticing. Together `expected` and `excluded` must
+ * equal ALL_EVENT_TYPES exactly, so a newly registered type belongs to
+ * neither and fails here until someone triages it deliberately.
+ */
+async function gateB(lines, manifestPath) {
+  const failures = [];
+  let schema;
+  try {
+    schema = await import('@onlooker-community/schema');
+  } catch {
+    return ['@onlooker-community/schema is not installed; run `npm ci`'];
+  }
+  const registered = new Set(schema.ALL_EVENT_TYPES);
+  const manifest = JSON.parse(readFileSync(manifestPath, 'utf8'));
+  const expected = manifest.expected ?? [];
+  const excluded = manifest.excluded ?? {};
+
+  const emitted = new Set(lines.filter((l) => l.valid === true).map((l) => l.event_type));
+  for (const t of expected) {
+    if (!emitted.has(t)) failures.push(`expected type never emitted during the suite: ${t}`);
+  }
+
+  const accounted = new Set([...expected, ...Object.keys(excluded)]);
+  for (const t of registered) {
+    if (!accounted.has(t)) {
+      failures.push(`registered type is in neither list — triage it in the manifest: ${t}`);
+    }
+  }
+  for (const t of accounted) {
+    if (!registered.has(t)) {
+      failures.push(`manifest names a type the schema does not register: ${t}`);
+    }
+  }
+  for (const [t, reason] of Object.entries(excluded)) {
+    if (!reason || !String(reason).trim()) {
+      failures.push(`excluded type needs a reason: ${t}`);
+    }
+  }
+  // An excluded type that is actually emitted and valid is coverage silently
+  // under-claimed: the manifest says "nothing tests this" while the suite
+  // does. Catch it before the manifest can drift downward unnoticed.
+  for (const t of Object.keys(excluded)) {
+    if (emitted.has(t)) {
+      failures.push(`excluded type is actually emitted — move it to expected: ${t}`);
+    }
+  }
+  return failures;
+}
+
+async function main() {
+  const args = parseArgs(process.argv);
+  const lines = loadReport(args.report);
+  const failures = gateA(lines);
+  // Skip Gate B unless something was genuinely validated. A merely
+  // non-empty report where nothing validated (schema package never
+  // resolved) would otherwise bury the one real failure under a spurious
+  // "expected type never emitted" line for every expected type.
+  if (lines.some((l) => l.validated === true)) failures.push(...(await gateB(lines, args.manifest)));
+  if (failures.length) {
+    for (const f of failures) process.stderr.write(`check-bus-coverage: ${f}\n`);
+    process.exit(1);
+  }
+  process.stdout.write(`check-bus-coverage: ok (${lines.length} emission(s))\n`);
+}
+
+const isMain = process.argv[1]?.endsWith('check-bus-coverage.mjs') ?? false;
+if (isMain) {
+  main().catch((err) => {
+    process.stderr.write(`check-bus-coverage: ${err.message}\n`);
+    process.exit(1);
+  });
+}