@onlooker-community/ecosystem 0.29.0 → 0.29.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/.claude/skills/writing-tests/SKILL.md +198 -0
- package/.claude-plugin/plugin.json +1 -1
- package/.release-please-manifest.json +3 -2
- package/CHANGELOG.md +14 -0
- package/CLAUDE.md +1 -1
- package/README.md +114 -13
- package/docs/adr/002-centralized-jsonl-event-log.md +2 -1
- package/docs/adr/005-runtime-emitter-fails-open.md +46 -0
- package/docs/architecture.md +2 -2
- package/package.json +2 -4
- package/plugins/inspector/.claude-plugin/plugin.json +1 -1
- package/plugins/inspector/CHANGELOG.md +8 -0
- package/release-please-config.json +17 -1
- package/scripts/lib/onlooker-event.mjs +78 -50
- package/test/bats/librarian-session-end.bats +8 -1
- package/test/helpers/setup.bash +23 -0
- package/test/node/schema-published.test.mjs +92 -0
|
@@ -0,0 +1,198 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: writing-tests
|
|
3
|
+
description: How to write tests in the Onlooker ecosystem repo — bats integration tests for hooks and the node:test schema suite. Use when adding or changing anything under test/, when writing a new hook or plugin that needs tests, or when deciding which shared helper to reach for instead of hand-rolling setup, dates, CLI stubs, project keys, or event-log assertions. Steers toward test/helpers/setup.bash and each plugin's own libs so tests stay isolated and never become time bombs.
|
|
4
|
+
---
|
|
5
|
+
|
|
6
|
+
# Writing tests
|
|
7
|
+
|
|
8
|
+
This repo has two test suites. Use the right one for what you're testing, and lean on the shared helpers below — most fragility in this codebase has come from tests hand-rolling things a helper already does.
|
|
9
|
+
|
|
10
|
+
- **bats** (`test/bats/*.bats`) — bash hooks and end-to-end plugin behavior. This is where almost all tests live.
|
|
11
|
+
- **node:test** (`test/node/*.test.mjs`) — schema mapping, manifest, and reference validation. Pure data-in/data-out, no hooks.
|
|
12
|
+
|
|
13
|
+
## Run the suite
|
|
14
|
+
|
|
15
|
+
```bash
|
|
16
|
+
npm test # bats + node schema suite (what most changes need)
|
|
17
|
+
npm run test:bats # just bats (bats test/bats)
|
|
18
|
+
npm run test:schema # just node (node --test test/node/*.test.mjs)
|
|
19
|
+
npm run test:ci # shellcheck + bats + schema + biome + manifest/reference lint
|
|
20
|
+
```
|
|
21
|
+
|
|
22
|
+
Run a single bats file while iterating: `bats test/bats/<name>.bats`. `bats` comes from mise (it is on `PATH`), not `npx`.
|
|
23
|
+
|
|
24
|
+
## Golden rules
|
|
25
|
+
|
|
26
|
+
1. Every bats test starts by isolating the filesystem — source `setup.bash` and call `setup_test_env`. Never touch the real `$HOME` or `~/.onlooker`.
|
|
27
|
+
2. Always reference `$ONLOOKER_DIR` / `$ONLOOKER_EVENTS_LOG`, never a literal `~/.onlooker`.
|
|
28
|
+
3. Never hardcode a date that feeds a relative window. Use `relative_iso_days_ago`.
|
|
29
|
+
4. Resolve project keys, emit events, and generate ULIDs through the plugin's own libs — don't reimplement them.
|
|
30
|
+
5. Assert behavior through the **event log** and on-disk artifacts, the same surfaces real consumers read.
|
|
31
|
+
|
|
32
|
+
## bats tests
|
|
33
|
+
|
|
34
|
+
### Isolate the environment first
|
|
35
|
+
|
|
36
|
+
Every test file's `setup()` sources the shared helper and calls `setup_test_env`, which repoints `HOME`, `ONLOOKER_DIR`, `CLAUDE_HOME`, and `CLAUDE_PLUGIN_ROOT` into `$BATS_TEST_TMPDIR` and severs git from your global config:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
setup() {
|
|
40
|
+
source "${BATS_TEST_DIRNAME}/../helpers/setup.bash"
|
|
41
|
+
setup_test_env
|
|
42
|
+
|
|
43
|
+
PLUGIN_ROOT="${REPO_ROOT}/plugins/<plugin>"
|
|
44
|
+
export CLAUDE_PLUGIN_ROOT="$PLUGIN_ROOT"
|
|
45
|
+
export ONLOOKER_ECOSYSTEM_ROOT="$REPO_ROOT"
|
|
46
|
+
}
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
If your hook needs the resolved Onlooker paths (`$ONLOOKER_EVENTS_LOG`, the session/metrics dirs) materialized, call `load_validate_path` instead — it runs `setup_test_env`, sources `scripts/lib/validate-path.sh`, and `mkdir -p`s the standard directories.
|
|
50
|
+
|
|
51
|
+
`setup_test_env` exports (read `test/helpers/setup.bash` for the authoritative list):
|
|
52
|
+
|
|
53
|
+
| Variable | Points at |
|
|
54
|
+
|----------|-----------|
|
|
55
|
+
| `HOME` | `$BATS_TEST_TMPDIR/home` (isolated) |
|
|
56
|
+
| `ONLOOKER_DIR` | `$HOME/.onlooker` |
|
|
57
|
+
| `ONLOOKER_EVENTS_LOG` | `$ONLOOKER_DIR/logs/onlooker-events.jsonl` (after `validate-path.sh` is sourced) |
|
|
58
|
+
| `CLAUDE_HOME` | `$HOME/.claude` |
|
|
59
|
+
| `REPO_ROOT` | the repo root (set when `setup.bash` is sourced) |
|
|
60
|
+
|
|
61
|
+
### Dates: use `relative_iso_days_ago`, never a literal
|
|
62
|
+
|
|
63
|
+
A hardcoded ISO date is a **time bomb** when the code under test computes a window from "now" (for example librarian's "now − `bootstrap_lookback_days`" scan window). The fixture passes today and silently fails once wall-clock now drifts past the threshold. Date such fixtures relative to now:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
# yesterday, UTC — comfortably inside a 14-day lookback window
|
|
67
|
+
created_at=$(relative_iso_days_ago 1)
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
`relative_iso_days_ago N` lives in `test/helpers/setup.bash` and returns an ISO-8601 UTC timestamp N days in the past (0 = now, negative = future). It uses `python3` for portable date math, since `date -d` (GNU) and `date -v` (BSD/macOS) disagree. For a plain "now" timestamp, `date -u +%Y-%m-%dT%H:%M:%SZ` is fine and portable; only the *offset* math needs the helper.
|
|
71
|
+
|
|
72
|
+
### Resolve project keys through the plugin lib
|
|
73
|
+
|
|
74
|
+
Artifacts are partitioned by a project key. Don't recompute the SHA — source the plugin's `*-project-key.sh` and call its function (`<plugin>_project_key <repo-root>`):
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
source "${PLUGIN_ROOT}/scripts/lib/librarian-project-key.sh"
|
|
78
|
+
PROJECT_KEY=$(librarian_project_key "$PROJECT_REPO")
|
|
79
|
+
ARTIFACT_DIR="${ONLOOKER_DIR}/<plugin>/${PROJECT_KEY}"
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
For key resolution to succeed the test needs a git context — stand up a throwaway repo in `setup()`:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
PROJECT_REPO="${BATS_TEST_TMPDIR}/repo"
|
|
86
|
+
mkdir -p "$PROJECT_REPO"
|
|
87
|
+
git -C "$PROJECT_REPO" init -q
|
|
88
|
+
git -C "$PROJECT_REPO" config user.email t@example.com
|
|
89
|
+
git -C "$PROJECT_REPO" config user.name "Test"
|
|
90
|
+
git -C "$PROJECT_REPO" remote add origin git@github.com:org/fixture.git
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
### Stub the `claude` CLI on `PATH`
|
|
94
|
+
|
|
95
|
+
Hooks that classify or judge shell out to `claude`. Stub it deterministically — branch on the prompt so each case returns a known response — and prepend the stub dir to `PATH`:
|
|
96
|
+
|
|
97
|
+
```bash
|
|
98
|
+
STUB_BIN="${BATS_TEST_TMPDIR}/bin"
|
|
99
|
+
mkdir -p "$STUB_BIN"
|
|
100
|
+
cat > "${STUB_BIN}/claude" <<'STUB'
|
|
101
|
+
#!/usr/bin/env bash
|
|
102
|
+
prompt=$(cat)
|
|
103
|
+
if [[ "$prompt" == *"some-marker"* ]]; then
|
|
104
|
+
printf '%s' '{"type":"feedback","title":"...","body":"...","confidence":0.84}'
|
|
105
|
+
else
|
|
106
|
+
printf '%s' '{"type":null,"title":"","body":"","confidence":0.2}'
|
|
107
|
+
fi
|
|
108
|
+
STUB
|
|
109
|
+
chmod +x "${STUB_BIN}/claude"
|
|
110
|
+
export PATH="${STUB_BIN}:${PATH}"
|
|
111
|
+
```
|
|
112
|
+
|
|
113
|
+
Same pattern stubs any external CLI a hook depends on (`curl`, `git`, …). Keep the responses minimal but schema-valid.
|
|
114
|
+
|
|
115
|
+
### Build hook input with `jq`
|
|
116
|
+
|
|
117
|
+
Hooks read a JSON payload on stdin. Build it with `jq -cn` and feed it in — don't hand-concatenate JSON strings:
|
|
118
|
+
|
|
119
|
+
```bash
|
|
120
|
+
_hook_input() {
|
|
121
|
+
jq -cn --arg cwd "$PROJECT_REPO" --arg sid "sess-test" \
|
|
122
|
+
'{cwd: $cwd, session_id: $sid, hook_event_name: "SessionEnd"}'
|
|
123
|
+
}
|
|
124
|
+
|
|
125
|
+
run bash -c "printf '%s' '$(_hook_input)' | '$HOOK'"
|
|
126
|
+
[ "$status" -eq 0 ] # hooks must always exit 0 — they never block the session
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
### Assert against the event log
|
|
130
|
+
|
|
131
|
+
Plugins communicate through the canonical JSONL event bus, so that's where you assert. Grep for the event type, then check the payload with `jq -e`:
|
|
132
|
+
|
|
133
|
+
```bash
|
|
134
|
+
grep '"event_type":"<plugin>.scan.complete"' "$ONLOOKER_EVENTS_LOG" \
|
|
135
|
+
| jq -e '.payload.outcome == "ok" and .payload.candidates_proposed == 2' >/dev/null
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
To assert an emitted line is a **schema-valid** envelope (not just present), pipe it through the canonical validator rather than eyeballing fields:
|
|
139
|
+
|
|
140
|
+
```bash
|
|
141
|
+
tail -n 1 "$ONLOOKER_EVENTS_LOG" \
|
|
142
|
+
| ONLOOKER_DIR="$ONLOOKER_DIR" node "${REPO_ROOT}/scripts/lib/onlooker-event.mjs" validate >/dev/null
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
Production code must emit only through `scripts/lib/onlooker-event.mjs` (often via a plugin wrapper like `librarian_emit` / `assayer_emit_event`) — never by writing the log directly. New event types must be registered in `@onlooker-community/schema` before they validate.
|
|
146
|
+
|
|
147
|
+
### Seed fixtures and generate IDs
|
|
148
|
+
|
|
149
|
+
Write fixtures with `jq -n` into the plugin's project-keyed directory, and generate IDs with the plugin's `*-ulid.sh` (ULIDs, not UUIDs — a repo-wide convention). A typical artifact seeder:
|
|
150
|
+
|
|
151
|
+
```bash
|
|
152
|
+
_seed_artifact() {
|
|
153
|
+
local kind="$1" id="$2" summary="$3" detail="$4" created_at="${5:-$(relative_iso_days_ago 1)}"
|
|
154
|
+
mkdir -p "${ARTIFACT_DIR}/${kind}"
|
|
155
|
+
jq -n --arg id "$id" --arg summary "$summary" --arg detail "$detail" \
|
|
156
|
+
--arg created_at "$created_at" \
|
|
157
|
+
'{ id: $id, created_at: $created_at, updated_at: $created_at,
|
|
158
|
+
summary: $summary, detail: $detail }' \
|
|
159
|
+
> "${ARTIFACT_DIR}/${kind}/${id}.json"
|
|
160
|
+
}
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
## Anti-patterns
|
|
164
|
+
|
|
165
|
+
Don't hand-roll these — each has bitten the suite before:
|
|
166
|
+
|
|
167
|
+
- **Hardcoded dates in window-sensitive fixtures.** Use `relative_iso_days_ago`. A literal like `2026-06-01T12:00:00Z` aged out of librarian's lookback window and broke two tests the day wall-clock now crossed it.
|
|
168
|
+
- **Literal `~/.onlooker` or `$HOME`.** Always `$ONLOOKER_DIR` and the `validate-path.sh` exports, so the isolated temp home is honored.
|
|
169
|
+
- **Reimplementing the project key, ULID, or event envelope.** Source the plugin lib (`*-project-key.sh`, `*-ulid.sh`) and emit through `onlooker-event.mjs`.
|
|
170
|
+
- **Concatenating JSON by hand.** Build payloads and hook input with `jq -n` / `jq -cn --arg`.
|
|
171
|
+
- **Asserting on log lines with `grep` substring matches alone** when you mean "valid event" — validate the envelope with `onlooker-event.mjs validate`.
|
|
172
|
+
- **Expecting a hook to exit non-zero.** Hooks fail soft and exit 0; assert on emitted events or absent side effects, not exit codes.
|
|
173
|
+
|
|
174
|
+
## Node tests
|
|
175
|
+
|
|
176
|
+
`test/node/*.test.mjs` use the built-in `node:test` runner and `node:assert/strict` — no external framework. They cover schema mapping (`schema-events.test.mjs`), manifest validation (`check-manifests.test.mjs`), and cross-references (`check-references.test.mjs`), driven by fixtures under `test/fixtures/`.
|
|
177
|
+
|
|
178
|
+
```javascript
|
|
179
|
+
import test from 'node:test';
|
|
180
|
+
import assert from 'node:assert/strict';
|
|
181
|
+
|
|
182
|
+
test('maps PostToolUse Read to tool.file.read', () => {
|
|
183
|
+
const mapped = mapHookInputToCanonical(loadFixture('post-tool-use-read.json'), { plugin: 'onlooker' });
|
|
184
|
+
assert.equal(mapped.valid, true);
|
|
185
|
+
assert.equal(mapped.event.event_type, 'tool.file.read');
|
|
186
|
+
});
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
Prefer adding a fixture under `test/fixtures/` and asserting the mapping over inlining large JSON literals. Run with `npm run test:schema`.
|
|
190
|
+
|
|
191
|
+
## Adding tests for a new plugin
|
|
192
|
+
|
|
193
|
+
1. Create `test/bats/<plugin>-<surface>.bats` (e.g. `<plugin>-session-end.bats`).
|
|
194
|
+
2. `setup()`: source `setup.bash`, call `setup_test_env`, set `PLUGIN_ROOT` / `CLAUDE_PLUGIN_ROOT`, stand up a fixture git repo, resolve the project key via the plugin lib.
|
|
195
|
+
3. Stub `claude` (and any other CLI) on `PATH` if the hook shells out.
|
|
196
|
+
4. Drive the hook with `jq`-built input; assert on `$ONLOOKER_EVENTS_LOG` and on-disk artifacts.
|
|
197
|
+
5. Date any window-sensitive fixture with `relative_iso_days_ago`.
|
|
198
|
+
6. Run `npm run test:ci` before opening the PR — it adds shellcheck and the manifest/reference linters on top of the tests.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ecosystem",
|
|
3
|
-
"version": "0.29.
|
|
3
|
+
"version": "0.29.2",
|
|
4
4
|
"description": "Observability substrate for Claude Code. Provides the shared $ONLOOKER_DIR storage root (default $HOME/.onlooker), canonical schema-validated event emission, session and tool tracking hooks, and prompt rules. Required by all other Onlooker plugins.",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Onlooker Community",
|
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
{
|
|
2
|
-
".": "0.29.
|
|
2
|
+
".": "0.29.2",
|
|
3
3
|
"plugins/archivist": "0.1.0",
|
|
4
4
|
"plugins/tribunal": "1.0.1",
|
|
5
5
|
"plugins/echo": "0.2.0",
|
|
@@ -14,5 +14,6 @@
|
|
|
14
14
|
"plugins/historian": "0.2.0",
|
|
15
15
|
"plugins/assayer": "1.0.0",
|
|
16
16
|
"plugins/bursar": "0.1.0",
|
|
17
|
-
"plugins/lineage": "0.1.0"
|
|
17
|
+
"plugins/lineage": "0.1.0",
|
|
18
|
+
"plugins/inspector": "0.2.0"
|
|
18
19
|
}
|
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,19 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [0.29.2](https://github.com/onlooker-community/ecosystem/compare/ecosystem-v0.29.1...ecosystem-v0.29.2) (2026-06-21)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Bug Fixes
|
|
7
|
+
|
|
8
|
+
* **emitter:** make emission dependency-free and fail-open :relieved: ([#99](https://github.com/onlooker-community/ecosystem/issues/99)) ([0dda7f8](https://github.com/onlooker-community/ecosystem/commit/0dda7f803ed2dfa28561bd8d9e4193b2b18e5bbf))
|
|
9
|
+
|
|
10
|
+
## [0.29.1](https://github.com/onlooker-community/ecosystem/compare/ecosystem-v0.29.0...ecosystem-v0.29.1) (2026-06-15)
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
### Bug Fixes
|
|
14
|
+
|
|
15
|
+
* **cartographer:** correct typo in release-please bump-patch key :face_with_spiral_eyes: ([#91](https://github.com/onlooker-community/ecosystem/issues/91)) ([dfab160](https://github.com/onlooker-community/ecosystem/commit/dfab1602afda2b6255a72b0975ebab9289d75b8e))
|
|
16
|
+
|
|
3
17
|
## [0.29.0](https://github.com/onlooker-community/ecosystem/compare/ecosystem-v0.28.1...ecosystem-v0.29.0) (2026-06-15)
|
|
4
18
|
|
|
5
19
|
|
package/CLAUDE.md
CHANGED
|
@@ -73,7 +73,7 @@ See `plugins/compass/docs/adr/001-evaluate-prompts-in-context.md` for the full d
|
|
|
73
73
|
2. Use `scripts/lib/onlooker-event.mjs` for all event emission — never write directly to the JSONL log.
|
|
74
74
|
3. Store runtime artifacts under `${ONLOOKER_DIR:-$HOME/.onlooker}/<name>/<project-key>/`. Always use `$ONLOOKER_DIR` — never hardcode `~/.onlooker` — so the test suite's isolated temp home is respected.
|
|
75
75
|
4. Derive the project key via `tribunal_project_key` (or equivalent) — first 12 hex chars of SHA256(`remote:<origin-url>`), falling back to SHA256(`root:<repo-root>`) for repos without a remote. See `plugins/tribunal/scripts/lib/tribunal-project-key.sh`.
|
|
76
|
-
5. Register event types in `@onlooker-community/schema` before emitting them
|
|
76
|
+
5. Register event types in `@onlooker-community/schema` before emitting them. The runtime emitter is dependency-free and **fails open**: it validates against the schema package only when that package is resolvable (dev, CI, tests) and emits unconditionally otherwise, because installed marketplace plugins ship no `node_modules`. Schema drift is caught in CI against the published schemas at `schema.onlooker.dev`. See [ADR-005](docs/adr/005-runtime-emitter-fails-open.md).
|
|
77
77
|
6. Fail-soft when `~/.onlooker/` is absent — plugins must not block a session they were not invited to.
|
|
78
78
|
|
|
79
79
|
## Development
|
package/README.md
CHANGED
|
@@ -1,31 +1,101 @@
|
|
|
1
1
|
# Onlooker Ecosystem
|
|
2
2
|
|
|
3
3
|
[](https://github.com/onlooker-community/ecosystem/actions/workflows/test.yml)
|
|
4
|
+
[](https://github.com/onlooker-community/ecosystem/actions/workflows/coverage.yml)
|
|
5
|
+
[](./LICENSE)
|
|
6
|
+
[](#plugins)
|
|
7
|
+
[](https://www.conventionalcommits.org)
|
|
4
8
|
|
|
5
|
-
|
|
9
|
+
Composable observability, memory, and quality-gate plugins for [Claude Code](https://docs.claude.com/en/docs/claude-code) — all built on the [Onlooker](https://onlooker.dev) event substrate.
|
|
6
10
|
|
|
7
|
-
The ecosystem is a **Claude Code plugin marketplace
|
|
11
|
+
The ecosystem is a **Claude Code plugin marketplace**. Every plugin writes to a shared, schema-validated event log, derives a stable project key from your git remote, and stores artifacts under `~/.onlooker/` — so plugins compose without stepping on each other, and every event is queryable in one place.
|
|
12
|
+
|
|
13
|
+
---
|
|
14
|
+
|
|
15
|
+
## Core concepts
|
|
16
|
+
|
|
17
|
+
- **Shared event substrate.** The `ecosystem` plugin is always-on infrastructure. Every other plugin emits [canonical Onlooker events](https://github.com/onlooker-community/schema) through `scripts/lib/onlooker-event.mjs`; nothing writes the log directly. Event types follow `<plugin>.<noun>.<verb>` (e.g. `tribunal.gate.blocked`, `inspector.run.completed`).
|
|
18
|
+
- **Plugins compose, they don't couple.** Plugins coordinate only by reading and writing the JSONL event bus — no plugin calls another directly. Every plugin depends on `ecosystem`; none depends on another plugin. So `bursar` can roll up `governor.session.complete` events without importing governor, and degrade gracefully when it's absent.
|
|
19
|
+
- **Stable project keys.** Artifacts are partitioned by a project key — the first 12 hex chars of `SHA256(remote:<origin-url>)`, falling back to the repo root for remote-less checkouts — so history stays attached to a project across clones and worktrees.
|
|
20
|
+
- **One storage root.** Runtime artifacts live under `$ONLOOKER_DIR` (default `$HOME/.onlooker/`), namespaced per plugin and project. Plugins fail soft when it's absent — a plugin never blocks a session it wasn't invited to.
|
|
21
|
+
- **Opt-in by default.** Most plugins ship disabled and are enabled per-project or globally via `settings.json`. The substrate, and a couple of low-cost reporters, are the exceptions (see the [table](#plugins)).
|
|
22
|
+
|
|
23
|
+
For how these fit together, see [docs/architecture.md](docs/architecture.md) and the [ecosystem-level ADRs](docs/adr/).
|
|
8
24
|
|
|
9
25
|
---
|
|
10
26
|
|
|
11
27
|
## Plugins
|
|
12
28
|
|
|
13
|
-
|
|
29
|
+
Seventeen plugins, grouped by what they do. Each links to its own README and config.
|
|
30
|
+
|
|
31
|
+
### Substrate
|
|
32
|
+
|
|
33
|
+
| Plugin | Description | Default |
|
|
34
|
+
|--------|-------------|---------|
|
|
35
|
+
| [`ecosystem`](./) | Observability substrate: `$ONLOOKER_DIR` storage, canonical schema-validated event emission, session/tool tracking hooks, and prompt rules. Required by every other plugin. | Always on |
|
|
36
|
+
|
|
37
|
+
### Memory & context
|
|
38
|
+
|
|
39
|
+
| Plugin | Description | Default |
|
|
40
|
+
|--------|-------------|---------|
|
|
41
|
+
| [`archivist`](./plugins/archivist) | Structured session memory across context truncation. Extracts decisions, dead ends, and open questions on `PreCompact`; reinjects the most relevant items at the next `SessionStart`. | Opt-in |
|
|
42
|
+
| [`librarian`](./plugins/librarian) | Consolidation layer between archivist's per-session artifacts and your durable typed memory store. Detects which decisions deserve to persist, classifies them, and queues proposals for explicit confirmation. | Opt-in |
|
|
43
|
+
| [`curator`](./plugins/curator) | Maintenance layer for the typed auto-memory store. Runs cheap heuristic checks (date-decayed, broken paths, orphaned entries) within a wall-clock budget and points you at `/curator review`. Never edits memory directly. | Opt-in |
|
|
44
|
+
| [`historian`](./plugins/historian) | Episodic memory. Chunks and sanitizes the transcript at `SessionEnd`, then embeds each prompt on `UserPromptSubmit` to retrieve relevant past context. | Opt-in |
|
|
45
|
+
| [`scribe`](./plugins/scribe) | Intent documentation from agent activity. Captures *why* changes were made — problem context, decisions, tradeoffs — and distills them into readable artifacts at session end. | Enabled |
|
|
46
|
+
| [`cartographer`](./plugins/cartographer) | Proactive auditor of the instruction layer (`CLAUDE.md`, `AGENTS.md`, `.claude/rules/`). Maps relationships and surfaces contradictions, shadowing, gaps, and drift before they cause misbehavior. | Opt-in |
|
|
47
|
+
|
|
48
|
+
### Quality & verification
|
|
49
|
+
|
|
50
|
+
| Plugin | Description | Default |
|
|
51
|
+
|--------|-------------|---------|
|
|
52
|
+
| [`tribunal`](./plugins/tribunal) | Multi-agent quality gates. Wraps a task in an Actor → Jury → Meta-Judge → Gate loop and retries the Actor with critique until the gate passes or `max_iterations` is reached. Grounded in LLM-as-a-Judge (Zheng et al. 2023) and LLM-as-a-Meta-Judge (Wu et al. 2024). | Skill always on; Stop hook opt-in |
|
|
53
|
+
| [`echo`](./plugins/echo) | Prompt-change regression detection. When a watched agent file is modified, runs a single-judge quality pass and compares against a stored baseline to report improved, degraded, or neutral. | Opt-in |
|
|
54
|
+
| [`assayer`](./plugins/assayer) | Claim verification. Parses the agent's final message for testable claims ("tests pass", "build is green") and checks each against the actual command results in the transcript. Catches lying-without-malice. Advisory when on. | Opt-in |
|
|
55
|
+
| [`inspector`](./plugins/inspector) | Per-edit lint and typecheck gate. Runs the project's configured checks on just the touched file after every `Write`/`Edit`/`MultiEdit`, so the agent sees its own type errors before claiming success. Cheaper than a full project verify; complements assayer. | Opt-in |
|
|
56
|
+
|
|
57
|
+
### Safety & alignment
|
|
58
|
+
|
|
59
|
+
| Plugin | Description | Default |
|
|
14
60
|
|--------|-------------|---------|
|
|
15
|
-
| [`
|
|
16
|
-
| [`
|
|
17
|
-
| [`tribunal`](./plugins/tribunal) | Multi-agent quality gates. Wraps a task in an Actor → Jury → Meta-Judge → Gate loop; retries the Actor with critique until the gate passes or `max_iterations` is reached. | Yes — skill always available; Stop hook opt-in |
|
|
18
|
-
| [`echo`](./plugins/echo) | Prompt-change regression detection. When a watched agent file is modified, runs a quality pass and reports whether the change improved, degraded, or had no measurable effect. | Yes — disabled by default |
|
|
19
|
-
| [`cartographer`](./plugins/cartographer) | Proactive instruction-file auditor. Discovers all `CLAUDE.md`, `AGENTS.md`, and `.claude/rules/` files, maps their relationships, and surfaces contradictions, dead rules, stale references, and scope collisions before they cause agent misbehavior. | Yes — disabled by default |
|
|
61
|
+
| [`compass`](./plugins/compass) | Pre-write intent clarity gate. Intercepts write-class tool calls and requires a confidence threshold before allowing them, evaluating the pending write against the prior assistant turn to avoid false positives on question-answer turns. | Opt-in |
|
|
62
|
+
| [`warden`](./plugins/warden) | Untrusted-content gate. Scans `WebFetch`/`Read` content for prompt-injection patterns and, on a hit, closes a session-scoped gate that blocks `Write`/`Edit`/`Bash` until you clear it. Applies Meta's Agents Rule of Two by removing external actions while untrusted content is in play. | Opt-in |
|
|
20
63
|
|
|
21
|
-
|
|
64
|
+
### Cost & governance
|
|
65
|
+
|
|
66
|
+
| Plugin | Description | Default |
|
|
67
|
+
|--------|-------------|---------|
|
|
68
|
+
| [`governor`](./plugins/governor) | Resource governance and budget enforcement. Tracks per-session token and cost spend and gates `Task` spawns before they exceed a configurable ceiling. Named for the steam-engine governor that regulates output. | Opt-in |
|
|
69
|
+
| [`bursar`](./plugins/bursar) | Multi-session, per-project budget accounting. Rolls each session's spend into a per-project ledger on `SessionEnd` and surfaces "this project burned $X this week" at `SessionStart`. The cross-session rollup to governor's single-session view. | Opt-in |
|
|
70
|
+
|
|
71
|
+
### Observability & insight
|
|
72
|
+
|
|
73
|
+
| Plugin | Description | Default |
|
|
74
|
+
|--------|-------------|---------|
|
|
75
|
+
| [`lineage`](./plugins/lineage) | Per-change provenance — answers "why does this line exist?". Records session/turn metadata plus a redacted, size-capped snippet for every edit, then traces a file or line back to the change (and prompt) that introduced it. | Opt-in |
|
|
76
|
+
| [`counsel`](./plugins/counsel) | Weekly synthesis across the full observability stack. Reads every plugin's event log, produces a structured improvement brief, and injects it at session start when the last brief is stale. | Enabled |
|
|
22
77
|
|
|
23
78
|
---
|
|
24
79
|
|
|
25
80
|
## Quick start
|
|
26
81
|
|
|
82
|
+
### Use it in Claude Code
|
|
83
|
+
|
|
84
|
+
Add the marketplace, then install the plugins you want:
|
|
85
|
+
|
|
86
|
+
```text
|
|
87
|
+
/plugin marketplace add onlooker-community/ecosystem
|
|
88
|
+
/plugin install ecosystem@onlooker-community
|
|
89
|
+
/plugin install inspector@onlooker-community
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
`ecosystem` is the substrate every other plugin builds on — install it first. Most plugins are disabled by default; enable and tune them under their namespace key in `~/.claude/settings.json` (global) or `.claude/settings.json` (per-project). See [ADR-004](docs/adr/004-plugin-config-with-settings-overlay.md) for the configuration model.
|
|
93
|
+
|
|
94
|
+
### Onlooker CLI
|
|
95
|
+
|
|
96
|
+
The companion CLI reads the shared event log for cross-session reporting:
|
|
97
|
+
|
|
27
98
|
```bash
|
|
28
|
-
# Install the Onlooker CLI
|
|
29
99
|
brew tap onlooker-community/tap
|
|
30
100
|
brew install onlooker
|
|
31
101
|
|
|
@@ -35,6 +105,22 @@ onlooker setup
|
|
|
35
105
|
|
|
36
106
|
---
|
|
37
107
|
|
|
108
|
+
## Configuration
|
|
109
|
+
|
|
110
|
+
Each plugin ships defaults in its own `config.json`. Override them per-namespace in `settings.json`:
|
|
111
|
+
|
|
112
|
+
```jsonc
|
|
113
|
+
{
|
|
114
|
+
// ~/.claude/settings.json (global) or .claude/settings.json (per-project)
|
|
115
|
+
"inspector": { "enabled": true },
|
|
116
|
+
"tribunal": { "enabled": true, "stop_hook": { "enabled": true } }
|
|
117
|
+
}
|
|
118
|
+
```
|
|
119
|
+
|
|
120
|
+
Project-level settings override global by the plugin's namespace key.
|
|
121
|
+
|
|
122
|
+
---
|
|
123
|
+
|
|
38
124
|
## Development
|
|
39
125
|
|
|
40
126
|
Install tools with [mise](https://mise.jdx.dev/) (`mise install`), then install dependencies:
|
|
@@ -46,15 +132,24 @@ npm run test:shellcheck
|
|
|
46
132
|
npm run test:ci # shellcheck + bats + schema + lint
|
|
47
133
|
```
|
|
48
134
|
|
|
49
|
-
|
|
135
|
+
Tests live under `test/bats/` and `test/node/` and use an isolated temp home, so nothing writes to your real `~/.onlooker/`.
|
|
136
|
+
|
|
137
|
+
Hooks emit [canonical Onlooker events](https://github.com/onlooker-community/schema) via `scripts/lib/onlooker-event.mjs`; shared bash helpers live in `scripts/lib/`. New event types must be registered in `@onlooker-community/schema` before a plugin emits them — the emitter validates the envelope.
|
|
138
|
+
|
|
139
|
+
### Adding a plugin
|
|
50
140
|
|
|
51
|
-
|
|
141
|
+
1. Create `plugins/<name>/` with `.claude-plugin/plugin.json`, `config.json`, and `hooks/hooks.json`.
|
|
142
|
+
2. Emit only through `scripts/lib/onlooker-event.mjs`, and register your event types in `@onlooker-community/schema` first.
|
|
143
|
+
3. Store artifacts under `${ONLOOKER_DIR:-$HOME/.onlooker}/<name>/<project-key>/` — never hardcode the path.
|
|
144
|
+
4. Register the plugin in `.claude-plugin/marketplace.json`, `release-please-config.json`, and `.release-please-manifest.json`.
|
|
145
|
+
|
|
146
|
+
Commits follow [Conventional Commits](https://www.conventionalcommits.org); releases are automated with [release-please](https://github.com/googleapis/release-please) per plugin.
|
|
52
147
|
|
|
53
148
|
---
|
|
54
149
|
|
|
55
150
|
## Prompt rules
|
|
56
151
|
|
|
57
|
-
The ecosystem plugin ships a `UserPromptSubmit` hook that injects declarative guidance when a user prompt matches a regex. Rules fire deterministically on literal prompt-text patterns, filling the niche
|
|
152
|
+
The ecosystem plugin ships a `UserPromptSubmit` hook that injects declarative guidance when a user prompt matches a regex. Rules fire deterministically on literal prompt-text patterns, filling the niche skills can't: guidance that must fire regardless of whether the model would have chosen a skill.
|
|
58
153
|
|
|
59
154
|
Rules live in two files:
|
|
60
155
|
|
|
@@ -76,3 +171,9 @@ Rules live in two files:
|
|
|
76
171
|
```
|
|
77
172
|
|
|
78
173
|
`pattern` is POSIX ERE (`[[ =~ ]]` semantics). Run `/list-prompt-rules` to see active rules and per-session fire state.
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## License
|
|
178
|
+
|
|
179
|
+
[MIT](./LICENSE) © Onlooker Community
|
|
@@ -1,7 +1,8 @@
|
|
|
1
1
|
# ADR-002: Centralized JSONL Event Log with Schema Validation
|
|
2
2
|
|
|
3
3
|
**Status:** Accepted
|
|
4
|
-
**Date:** 2026-05-24
|
|
4
|
+
**Date:** 2026-05-24
|
|
5
|
+
**Amended by:** [ADR-005](005-runtime-emitter-fails-open.md) — runtime validation was moved off the hot path; the emitter now fails open. The "schema validation at write time" rationale below holds only for dev/CI, not installed plugins.
|
|
5
6
|
|
|
6
7
|
## Context
|
|
7
8
|
|
|
@@ -0,0 +1,46 @@
|
|
|
1
|
+
# ADR-005: The Runtime Emitter Is Dependency-Free and Fails Open
|
|
2
|
+
|
|
3
|
+
**Status:** Accepted
|
|
4
|
+
**Date:** 2026-06-20
|
|
5
|
+
**Amends:** [ADR-002](002-centralized-jsonl-event-log.md)
|
|
6
|
+
|
|
7
|
+
## Context
|
|
8
|
+
|
|
9
|
+
The canonical emitter `scripts/lib/onlooker-event.mjs` built each event envelope and **validated it against `@onlooker-community/schema` before printing it** (ADR-002: "schema validation at write time prevents silent corruption"). `@onlooker-community/schema` was a runtime `dependency`, and it pulls in `ajv` + `ajv-formats`. The package also reads its JSON schema files from disk at import time.
|
|
10
|
+
|
|
11
|
+
This assumed `node_modules` would be present wherever a hook runs. It is not. **Claude Code installs a marketplace plugin by cloning the marketplace repo; it never runs `npm install`.** `node_modules/` is git-ignored, so the installed plugin ships none. Every emission therefore ran:
|
|
12
|
+
|
|
13
|
+
```
|
|
14
|
+
node scripts/lib/onlooker-event.mjs emit
|
|
15
|
+
└─ import '@onlooker-community/schema' → ERR_MODULE_NOT_FOUND → exit 1
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
The bash side of each hook kept working (hook-health, per-session trackers need no node), so hooks looked healthy while **every event was silently dropped**. In one observed case `~/.onlooker/logs/onlooker-events.jsonl` received nothing for 18 days. Fail-closed validation, intended to prevent "silent corruption," instead produced total silent loss — a strictly worse failure than the one it guarded against.
|
|
19
|
+
|
|
20
|
+
Mechanisms considered to make the dependency available in the install: vendor a committed `node_modules`, bundle the emitter + `ajv` into one file with esbuild, or a self-healing `SessionStart` hook that runs `npm install`. Each ships or fetches a validator with the plugin and carries cost (repo bloat, a build step, or a network dependency on every fresh install).
|
|
21
|
+
|
|
22
|
+
## Decision
|
|
23
|
+
|
|
24
|
+
**The runtime emitter has zero external dependencies and fails open.** Validation is best-effort, not a gate:
|
|
25
|
+
|
|
26
|
+
- `createEvent` (envelope assembly) and the event-type constants are **inlined** into `onlooker-event.mjs` — pure `node:` stdlib, no package import.
|
|
27
|
+
- Validation is attempted via a lazy `await import('@onlooker-community/schema')`. When the package resolves (dev, CI, tests) the emitter validates and **rejects** invalid events. When it does not resolve (installed plugins) the emitter **emits anyway**.
|
|
28
|
+
- `@onlooker-community/schema` moves from `dependencies` to `devDependencies`.
|
|
29
|
+
- A CI test (`test/node/schema-published.test.mjs`) validates representative emitter output against the canonical schemas **published at `https://schema.onlooker.dev`** — the contract downstream consumers actually enforce. It skips when the endpoint is unreachable.
|
|
30
|
+
|
|
31
|
+
## Rationale
|
|
32
|
+
|
|
33
|
+
**Observability must not be fragile.** A telemetry pipeline that deletes everything when a dependency is missing — or when a payload drifts from the schema — is worse than one that records a slightly-off event. For an observability substrate, fail-open is the correct default: capture the signal, surface problems out-of-band.
|
|
34
|
+
|
|
35
|
+
**Best-effort keeps the dev-time guard.** The same lazy import that degrades gracefully in production still gates strictly wherever the validator is installed. The negative tests across the plugins ("emission fails loudly on a bogus event_type") keep passing because CI runs with dev dependencies present. We lose nothing in the loop that catches mistakes, and gain resilience where mistakes are unrecoverable.
|
|
36
|
+
|
|
37
|
+
**Validation belongs where it can act, not on the hot path.** A hook firing on every tool call cannot meaningfully respond to a validation failure except by dropping data. CI can: a red test blocks the release. Moving the authoritative check to CI — against the *published* schemas, not just the pinned npm copy — catches drift between what the emitter produces and what the deployed contract accepts.
|
|
38
|
+
|
|
39
|
+
**No artifact to ship or fetch.** Dependency-free is simpler than vendoring a `node_modules`, bundling with esbuild, or bootstrapping `npm install` at session start. There is nothing to keep in sync, nothing to build, and the install works offline.
|
|
40
|
+
|
|
41
|
+
## Consequences
|
|
42
|
+
|
|
43
|
+
- The JSONL log may, in an installed plugin, contain an event that violates the schema if a payload builder drifts. Downstream consumers (the onlooker agent, the backend) validate on ingest; CI catches drift before release. This is the accepted cost of fail-open.
|
|
44
|
+
- `node` is still required to emit (the emitter is JS), but **no npm packages are**. Hooks that shell out to `onlooker-event.mjs` work on a fresh clone with nothing installed.
|
|
45
|
+
- The `validate` CLI subcommand and `onlooker_validate_event` still require the dev dependency; they are used only by the test suite, never on a runtime hook path.
|
|
46
|
+
- `test:schema` now makes a network call to `schema.onlooker.dev`. It skips cleanly offline, so local and air-gapped runs are unaffected; CI with egress exercises the live contract.
|
package/docs/architecture.md
CHANGED
|
@@ -22,7 +22,7 @@ flowchart TB
|
|
|
22
22
|
echo --> emitter
|
|
23
23
|
cartographer --> emitter
|
|
24
24
|
|
|
25
|
-
emitter -->|"
|
|
25
|
+
emitter -->|"event envelope<br/>(validated in dev/CI)"| log
|
|
26
26
|
log -.->|"append-only JSONL"| log
|
|
27
27
|
end
|
|
28
28
|
```
|
|
@@ -34,7 +34,7 @@ The `ecosystem` plugin (repo root) is not optional — it provides the infrastru
|
|
|
34
34
|
| Component | What it does |
|
|
35
35
|
|-----------|-------------|
|
|
36
36
|
| `~/.onlooker/` directory | Shared storage root, created by the Onlooker installer. All plugins store artifacts here under their own sub-path. |
|
|
37
|
-
| `scripts/lib/onlooker-event.mjs` | Canonical event builder. Accepts a JSON payload on stdin
|
|
37
|
+
| `scripts/lib/onlooker-event.mjs` | Canonical event builder. Accepts a JSON payload on stdin and prints a canonical envelope to stdout; callers capture it and append to the JSONL log. **Dependency-free and fail-open**: it validates against `@onlooker-community/schema` only when that package is resolvable (dev/CI) and emits unconditionally otherwise, so a fresh marketplace install — which ships no `node_modules` — never loses telemetry. See [ADR-005](adr/005-runtime-emitter-fails-open.md). |
|
|
38
38
|
| `scripts/lib/onlooker-schema.sh` | Bash convenience wrapper around `onlooker-event.mjs`. Provides `onlooker_event_from_hook` (builds an envelope via `node`) and `onlooker_append_event` (appends a pre-built envelope to the log). Node is still required. |
|
|
39
39
|
| `scripts/lib/validate-path.sh` | Sets canonical `$ONLOOKER_*` environment variables (log path, tracker dirs, etc.) so every hook uses consistent paths. |
|
|
40
40
|
| Session trackers | `SessionStart`, `SessionEnd`, `PreCompact`, `PostCompact`, `PreToolUse`, `PostToolUse`, `UserPromptSubmit`, `TaskCreated`, `TaskCompleted`, `WorktreeCreate`, and `WorktreeRemove` hooks that emit `session.*`, `tool.*`, `turn.*` events for the observability layer. |
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@onlooker-community/ecosystem",
|
|
3
|
-
"version": "0.29.
|
|
3
|
+
"version": "0.29.2",
|
|
4
4
|
"description": "Agents, skills, hooks, commands, rules, and MCP configurations that power [Onlooker](https://onlooker.dev)",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Onlooker Community",
|
|
@@ -18,9 +18,6 @@
|
|
|
18
18
|
"bin": {
|
|
19
19
|
"onlooker-install": "install.sh"
|
|
20
20
|
},
|
|
21
|
-
"dependencies": {
|
|
22
|
-
"@onlooker-community/schema": "^2.9.0"
|
|
23
|
-
},
|
|
24
21
|
"scripts": {
|
|
25
22
|
"postinstall": "echo '\\n onlooker-ecosystem installed!\\n Run: npx onlooker-install typescript\\n Docs: https://github.com/onlooker-community/ecosystem\\n'",
|
|
26
23
|
"test": "npm run test:bats && npm run test:schema",
|
|
@@ -45,6 +42,7 @@
|
|
|
45
42
|
},
|
|
46
43
|
"devDependencies": {
|
|
47
44
|
"@biomejs/biome": "2.4.15",
|
|
45
|
+
"@onlooker-community/schema": "^2.9.0",
|
|
48
46
|
"globals": "^17.6.0",
|
|
49
47
|
"markdownlint-cli": "^0.48.0"
|
|
50
48
|
}
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "inspector",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.2.0",
|
|
4
4
|
"description": "Per-edit lint and typecheck gate. Runs the project's configured checks on just the touched file after every Write / Edit / MultiEdit, so the agent sees its own type errors before claiming success. Cheaper than proctor (which runs project-wide verification at Stop); complements assayer (which catches the agent lying about claims). Builds on the Onlooker ecosystem plugin.",
|
|
5
5
|
"author": {
|
|
6
6
|
"name": "Onlooker Community",
|
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
## [0.2.0](https://github.com/onlooker-community/ecosystem/compare/inspector-v0.1.0...inspector-v0.2.0) (2026-06-15)
|
|
4
|
+
|
|
5
|
+
|
|
6
|
+
### Features
|
|
7
|
+
|
|
8
|
+
* **inspector:** ship the per-edit lint/typecheck plugin ([#88](https://github.com/onlooker-community/ecosystem/issues/88)) ([2018243](https://github.com/onlooker-community/ecosystem/commit/201824384abd6a4fc5f4395266924aa413a2ffd1))
|
|
@@ -67,7 +67,7 @@
|
|
|
67
67
|
"changelog-path": "CHANGELOG.md",
|
|
68
68
|
"release-type": "simple",
|
|
69
69
|
"bump-minor-pre-major": true,
|
|
70
|
-
"bump-patch-for-
|
|
70
|
+
"bump-patch-for-minor-pre-major": false,
|
|
71
71
|
"component": "cartographer",
|
|
72
72
|
"draft": false,
|
|
73
73
|
"prerelease": false,
|
|
@@ -254,6 +254,22 @@
|
|
|
254
254
|
"jsonpath": "$.version"
|
|
255
255
|
}
|
|
256
256
|
]
|
|
257
|
+
},
|
|
258
|
+
"plugins/inspector": {
|
|
259
|
+
"changelog-path": "CHANGELOG.md",
|
|
260
|
+
"release-type": "simple",
|
|
261
|
+
"bump-minor-pre-major": true,
|
|
262
|
+
"bump-patch-for-minor-pre-major": false,
|
|
263
|
+
"component": "inspector",
|
|
264
|
+
"draft": false,
|
|
265
|
+
"prerelease": false,
|
|
266
|
+
"extra-files": [
|
|
267
|
+
{
|
|
268
|
+
"type": "json",
|
|
269
|
+
"path": ".claude-plugin/plugin.json",
|
|
270
|
+
"jsonpath": "$.version"
|
|
271
|
+
}
|
|
272
|
+
]
|
|
257
273
|
}
|
|
258
274
|
},
|
|
259
275
|
"$schema": "https://raw.githubusercontent.com/googleapis/release-please/main/schemas/config.json"
|
|
@@ -6,20 +6,24 @@
|
|
|
6
6
|
import { randomUUID } from 'node:crypto';
|
|
7
7
|
import { existsSync, mkdirSync, readFileSync, statSync, writeFileSync } from 'node:fs';
|
|
8
8
|
import { join } from 'node:path';
|
|
9
|
-
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
22
|
-
|
|
9
|
+
|
|
10
|
+
// Canonical event-type constants, inlined rather than imported from
|
|
11
|
+
// @onlooker-community/schema so this emitter has ZERO runtime dependencies.
|
|
12
|
+
// Claude Code installs plugins by cloning the marketplace repo and never runs
|
|
13
|
+
// `npm install`, so an installed plugin has no node_modules — a runtime import
|
|
14
|
+
// of the schema package fails with ERR_MODULE_NOT_FOUND and silently kills all
|
|
15
|
+
// telemetry. The full contract still lives in @onlooker-community/schema; it is
|
|
16
|
+
// now a devDependency used to validate emitter output in CI (see tryValidate).
|
|
17
|
+
const SKILL_INVOKED = 'skill.invoked';
|
|
18
|
+
const TASK_START = 'task.start';
|
|
19
|
+
const TASK_COMPLETE = 'task.complete';
|
|
20
|
+
const TOOL_AGENT_COMPLETE = 'tool.agent.complete';
|
|
21
|
+
const TOOL_AGENT_SPAWN = 'tool.agent.spawn';
|
|
22
|
+
const TOOL_FILE_EDIT = 'tool.file.edit';
|
|
23
|
+
const TOOL_FILE_READ = 'tool.file.read';
|
|
24
|
+
const TOOL_FILE_WRITE = 'tool.file.write';
|
|
25
|
+
const TOOL_SHELL_EXEC = 'tool.shell.exec';
|
|
26
|
+
const TOOL_WEB_FETCH = 'tool.web.fetch';
|
|
23
27
|
|
|
24
28
|
export function ensureMachineId(onlookerDir) {
|
|
25
29
|
const path = join(onlookerDir, 'machine_id');
|
|
@@ -55,21 +59,52 @@ export function buildCanonicalEvent({
|
|
|
55
59
|
cost_usd,
|
|
56
60
|
token_count,
|
|
57
61
|
}) {
|
|
58
|
-
|
|
62
|
+
// Envelope assembly inlined from @onlooker-community/schema's createEvent so
|
|
63
|
+
// emission needs no external package. The schema is the source of truth for
|
|
64
|
+
// this shape; CI validates against it (see test/node/schema-*.test.mjs).
|
|
65
|
+
const event = {
|
|
66
|
+
id: randomUUID(),
|
|
67
|
+
schema_version: '1.0',
|
|
59
68
|
runtime,
|
|
60
|
-
adapter_id,
|
|
61
69
|
plugin,
|
|
62
70
|
machine_id: ensureMachineId(onlookerDir),
|
|
71
|
+
timestamp: new Date().toISOString(),
|
|
63
72
|
session_id,
|
|
73
|
+
sequence: nextSequence(onlookerDir),
|
|
64
74
|
event_type,
|
|
65
75
|
payload,
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
event.
|
|
76
|
+
redacted: false,
|
|
77
|
+
};
|
|
78
|
+
if (adapter_id !== undefined) event.adapter_id = adapter_id;
|
|
79
|
+
if (cost_usd !== undefined) event.cost_usd = cost_usd;
|
|
80
|
+
if (token_count !== undefined) event.token_count = token_count;
|
|
70
81
|
return event;
|
|
71
82
|
}
|
|
72
83
|
|
|
84
|
+
/**
|
|
85
|
+
* Best-effort schema validation.
|
|
86
|
+
*
|
|
87
|
+
* @onlooker-community/schema is a devDependency, so it resolves in dev, CI, and
|
|
88
|
+
* tests — where we DO want emitter output gated against the contract (this is
|
|
89
|
+
* what the negative tests across the plugins rely on) — but is absent in an
|
|
90
|
+
* installed marketplace plugin. When it is absent we fail OPEN: build and emit
|
|
91
|
+
* anyway, so schema drift can never silently kill telemetry the way a hard
|
|
92
|
+
* runtime dependency did. Drift is caught in CI, which validates emitter output
|
|
93
|
+
* against the published schemas at https://schema.onlooker.dev.
|
|
94
|
+
*
|
|
95
|
+
* Returns { available: false } when no validator is installed, otherwise the
|
|
96
|
+
* { valid, errors? } result from the schema package.
|
|
97
|
+
*/
|
|
98
|
+
async function tryValidate(event) {
|
|
99
|
+
let schema;
|
|
100
|
+
try {
|
|
101
|
+
schema = await import('@onlooker-community/schema');
|
|
102
|
+
} catch {
|
|
103
|
+
return { available: false };
|
|
104
|
+
}
|
|
105
|
+
return { available: true, ...schema.validate(event) };
|
|
106
|
+
}
|
|
107
|
+
|
|
73
108
|
function summarizeText(value, maxLen = 1000) {
|
|
74
109
|
if (value == null) return undefined;
|
|
75
110
|
const text = String(value).replace(/\s+/g, ' ').trim();
|
|
@@ -219,11 +254,7 @@ export function mapSkillHookInput(hookInput, options) {
|
|
|
219
254
|
payload,
|
|
220
255
|
});
|
|
221
256
|
|
|
222
|
-
|
|
223
|
-
if (!result.valid) {
|
|
224
|
-
return { valid: false, errors: result.errors, event_type: SKILL_INVOKED };
|
|
225
|
-
}
|
|
226
|
-
return { valid: true, event: result.event };
|
|
257
|
+
return { valid: true, event };
|
|
227
258
|
}
|
|
228
259
|
|
|
229
260
|
/**
|
|
@@ -269,11 +300,7 @@ export function mapTaskHookInput(hookInput, options) {
|
|
|
269
300
|
payload,
|
|
270
301
|
});
|
|
271
302
|
|
|
272
|
-
|
|
273
|
-
if (!result.valid) {
|
|
274
|
-
return { valid: false, errors: result.errors, event_type: eventType };
|
|
275
|
-
}
|
|
276
|
-
return { valid: true, event: result.event };
|
|
303
|
+
return { valid: true, event };
|
|
277
304
|
}
|
|
278
305
|
|
|
279
306
|
/**
|
|
@@ -322,11 +349,7 @@ export function mapWorktreeHookInput(hookInput, options) {
|
|
|
322
349
|
payload,
|
|
323
350
|
});
|
|
324
351
|
|
|
325
|
-
|
|
326
|
-
if (!result.valid) {
|
|
327
|
-
return { valid: false, errors: result.errors, event_type: TOOL_SHELL_EXEC };
|
|
328
|
-
}
|
|
329
|
-
return { valid: true, event: result.event };
|
|
352
|
+
return { valid: true, event };
|
|
330
353
|
}
|
|
331
354
|
|
|
332
355
|
/**
|
|
@@ -444,11 +467,7 @@ export function mapHookInputToCanonical(hookInput, options) {
|
|
|
444
467
|
payload,
|
|
445
468
|
});
|
|
446
469
|
|
|
447
|
-
|
|
448
|
-
if (!result.valid) {
|
|
449
|
-
return { valid: false, errors: result.errors, event_type: eventType };
|
|
450
|
-
}
|
|
451
|
-
return { valid: true, event: result.event };
|
|
470
|
+
return { valid: true, event };
|
|
452
471
|
}
|
|
453
472
|
|
|
454
473
|
function readStdin() {
|
|
@@ -468,12 +487,16 @@ async function main() {
|
|
|
468
487
|
if (command === 'validate') {
|
|
469
488
|
const raw = await readStdin();
|
|
470
489
|
const parsed = JSON.parse(raw || '{}');
|
|
471
|
-
const
|
|
472
|
-
if (!
|
|
473
|
-
console.error(
|
|
490
|
+
const check = await tryValidate(parsed);
|
|
491
|
+
if (!check.available) {
|
|
492
|
+
console.error('@onlooker-community/schema is not installed; cannot validate');
|
|
493
|
+
process.exit(1);
|
|
494
|
+
}
|
|
495
|
+
if (!check.valid) {
|
|
496
|
+
console.error(JSON.stringify(check.errors, null, 2));
|
|
474
497
|
process.exit(1);
|
|
475
498
|
}
|
|
476
|
-
console.log(JSON.stringify(
|
|
499
|
+
console.log(JSON.stringify(parsed));
|
|
477
500
|
return;
|
|
478
501
|
}
|
|
479
502
|
|
|
@@ -484,8 +507,11 @@ async function main() {
|
|
|
484
507
|
if (!mapped) {
|
|
485
508
|
process.exit(0);
|
|
486
509
|
}
|
|
487
|
-
|
|
488
|
-
|
|
510
|
+
// Best-effort: reject when a validator is present (dev/CI), fail open in
|
|
511
|
+
// installed plugins so the event is still emitted.
|
|
512
|
+
const check = await tryValidate(mapped.event);
|
|
513
|
+
if (check.available && !check.valid) {
|
|
514
|
+
console.error(JSON.stringify(check.errors, null, 2));
|
|
489
515
|
process.exit(1);
|
|
490
516
|
}
|
|
491
517
|
console.log(JSON.stringify(mapped.event));
|
|
@@ -509,12 +535,14 @@ async function main() {
|
|
|
509
535
|
event_type: params.event_type,
|
|
510
536
|
payload: params.payload,
|
|
511
537
|
});
|
|
512
|
-
|
|
513
|
-
|
|
514
|
-
|
|
538
|
+
// Best-effort: reject when a validator is present (dev/CI), fail open in
|
|
539
|
+
// installed plugins so the event is still emitted.
|
|
540
|
+
const check = await tryValidate(event);
|
|
541
|
+
if (check.available && !check.valid) {
|
|
542
|
+
console.error(JSON.stringify(check.errors, null, 2));
|
|
515
543
|
process.exit(1);
|
|
516
544
|
}
|
|
517
|
-
console.log(JSON.stringify(
|
|
545
|
+
console.log(JSON.stringify(event));
|
|
518
546
|
return;
|
|
519
547
|
}
|
|
520
548
|
|
|
@@ -62,11 +62,18 @@ STUB
|
|
|
62
62
|
export PATH="${STUB_BIN}:${PATH}"
|
|
63
63
|
|
|
64
64
|
HOOK="${PLUGIN_ROOT}/scripts/hooks/librarian-session-end.sh"
|
|
65
|
+
|
|
66
|
+
# On its first scan the hook has no watermark and falls back to a relative
|
|
67
|
+
# "now - bootstrap_lookback_days" window (default 14 days). Fixtures must be
|
|
68
|
+
# dated inside that window or the scan sees nothing, so date them relative to
|
|
69
|
+
# now — one day back keeps a wide margin. See relative_iso_days_ago in
|
|
70
|
+
# test/helpers/setup.bash for why a hardcoded date is a time bomb here.
|
|
71
|
+
FIXTURE_CREATED_AT=$(relative_iso_days_ago 1)
|
|
65
72
|
}
|
|
66
73
|
|
|
67
74
|
# Helper: write an archivist artifact for the project.
|
|
68
75
|
_seed_artifact() {
|
|
69
|
-
local kind="$1" id="$2" summary="$3" detail="$4" created_at="${5
|
|
76
|
+
local kind="$1" id="$2" summary="$3" detail="$4" created_at="${5:-$FIXTURE_CREATED_AT}"
|
|
70
77
|
local dir="${ARCHIVIST_DIR}/${kind}"
|
|
71
78
|
mkdir -p "$dir"
|
|
72
79
|
jq -n \
|
package/test/helpers/setup.bash
CHANGED
|
@@ -31,6 +31,29 @@ setup_test_env() {
|
|
|
31
31
|
unset XDG_CONFIG_HOME
|
|
32
32
|
}
|
|
33
33
|
|
|
34
|
+
# Produce an ISO-8601 UTC timestamp offset from "now" by N days into the past.
|
|
35
|
+
# Positive N = N days ago, 0 = now, negative N = N days in the future.
|
|
36
|
+
#
|
|
37
|
+
# Use this for any fixture whose date must fall inside a relative window the
|
|
38
|
+
# code computes from "now" (e.g. a "now - lookback_days" scan window). A
|
|
39
|
+
# hardcoded ISO date silently ages out of such a window and turns the test
|
|
40
|
+
# into a time bomb that passes today and fails on some future date. Always
|
|
41
|
+
# date those fixtures relative to now.
|
|
42
|
+
#
|
|
43
|
+
# Uses python3 (already a hook dependency) for portable date math — `date -d`
|
|
44
|
+
# vs `date -v` diverges between GNU and BSD/macOS.
|
|
45
|
+
#
|
|
46
|
+
# Usage: created_at=$(relative_iso_days_ago 1) # yesterday, UTC
|
|
47
|
+
relative_iso_days_ago() {
|
|
48
|
+
local days="${1:-0}"
|
|
49
|
+
python3 -c '
|
|
50
|
+
import datetime, sys
|
|
51
|
+
delta = datetime.timedelta(days=int(sys.argv[1]))
|
|
52
|
+
now = datetime.datetime.now(datetime.timezone.utc)
|
|
53
|
+
print((now - delta).strftime("%Y-%m-%dT%H:%M:%SZ"))
|
|
54
|
+
' "$days"
|
|
55
|
+
}
|
|
56
|
+
|
|
34
57
|
# Source validate-path.sh with test env vars already set.
|
|
35
58
|
load_validate_path() {
|
|
36
59
|
setup_test_env
|
|
@@ -0,0 +1,92 @@
|
|
|
1
|
+
import assert from 'node:assert/strict';
|
|
2
|
+
import { mkdtempSync, rmSync } from 'node:fs';
|
|
3
|
+
import { tmpdir } from 'node:os';
|
|
4
|
+
import { join } from 'node:path';
|
|
5
|
+
import { test } from 'node:test';
|
|
6
|
+
import { Ajv2020 } from 'ajv/dist/2020.js';
|
|
7
|
+
import _addFormats from 'ajv-formats';
|
|
8
|
+
import { buildCanonicalEvent } from '../../scripts/lib/onlooker-event.mjs';
|
|
9
|
+
|
|
10
|
+
// The runtime emitter is dependency-free and fails open (it never drops events
|
|
11
|
+
// in an installed plugin — see scripts/lib/onlooker-event.mjs). This test is the
|
|
12
|
+
// other half of that contract: it proves the emitter's output still conforms to
|
|
13
|
+
// the canonical schemas PUBLISHED at https://schema.onlooker.dev — the contract
|
|
14
|
+
// downstream consumers (the onlooker agent, the backend) actually validate
|
|
15
|
+
// against. Schema drift that would silently pass at runtime is caught here.
|
|
16
|
+
//
|
|
17
|
+
// Network-resilient: when the published endpoint is unreachable (offline dev,
|
|
18
|
+
// CI without egress) the test skips rather than failing red.
|
|
19
|
+
|
|
20
|
+
const addFormats = _addFormats.default ?? _addFormats;
|
|
21
|
+
const BASE = 'https://schema.onlooker.dev/schemas';
|
|
22
|
+
|
|
23
|
+
async function fetchJson(url) {
|
|
24
|
+
const res = await fetch(url, { signal: AbortSignal.timeout(8000) });
|
|
25
|
+
if (!res.ok) throw new Error(`${url} -> HTTP ${res.status}`);
|
|
26
|
+
return res.json();
|
|
27
|
+
}
|
|
28
|
+
|
|
29
|
+
// Representative events the emitter produces, paired with the payload schema
|
|
30
|
+
// file each event family maps to.
|
|
31
|
+
const SAMPLES = [
|
|
32
|
+
{
|
|
33
|
+
event_type: 'session.start',
|
|
34
|
+
payloadFile: 'session.json',
|
|
35
|
+
payload: { working_directory: '/tmp', git_branch: 'main' },
|
|
36
|
+
},
|
|
37
|
+
{ event_type: 'session.prompt', payloadFile: 'session.json', payload: { turn_number: 2, input_summary: 'hello' } },
|
|
38
|
+
{ event_type: 'tool.shell.exec', payloadFile: 'tool.json', payload: { command: 'ls -la', exit_code: 0 } },
|
|
39
|
+
{
|
|
40
|
+
event_type: 'skill.invoked',
|
|
41
|
+
payloadFile: 'skill.json',
|
|
42
|
+
payload: { skill_name: 'commit', invocation_source: 'tool' },
|
|
43
|
+
},
|
|
44
|
+
{ event_type: 'task.start', payloadFile: 'task.json', payload: { task_summary: 'implement the thing' } },
|
|
45
|
+
];
|
|
46
|
+
|
|
47
|
+
test('emitter output conforms to the published schemas at schema.onlooker.dev', async (t) => {
|
|
48
|
+
let envelope;
|
|
49
|
+
const payloadDocs = new Map();
|
|
50
|
+
try {
|
|
51
|
+
envelope = await fetchJson(`${BASE}/event.v1.json`);
|
|
52
|
+
for (const file of new Set(SAMPLES.map((s) => s.payloadFile))) {
|
|
53
|
+
payloadDocs.set(file, await fetchJson(`${BASE}/payload/${file}`));
|
|
54
|
+
}
|
|
55
|
+
} catch (err) {
|
|
56
|
+
t.skip(`published schemas unreachable (${err.message}); skipping live-contract check`);
|
|
57
|
+
return;
|
|
58
|
+
}
|
|
59
|
+
|
|
60
|
+
const ajv = new Ajv2020({ allErrors: true, strict: false });
|
|
61
|
+
addFormats(ajv);
|
|
62
|
+
const validateEnvelope = ajv.compile(envelope);
|
|
63
|
+
|
|
64
|
+
const tmp = mkdtempSync(join(tmpdir(), 'onlooker-published-'));
|
|
65
|
+
try {
|
|
66
|
+
for (const sample of SAMPLES) {
|
|
67
|
+
const event = buildCanonicalEvent({
|
|
68
|
+
onlookerDir: tmp,
|
|
69
|
+
plugin: 'onlooker',
|
|
70
|
+
session_id: 'published-contract-test',
|
|
71
|
+
event_type: sample.event_type,
|
|
72
|
+
payload: sample.payload,
|
|
73
|
+
});
|
|
74
|
+
|
|
75
|
+
assert.ok(
|
|
76
|
+
validateEnvelope(event),
|
|
77
|
+
`published envelope rejected ${sample.event_type}: ${ajv.errorsText(validateEnvelope.errors)}`,
|
|
78
|
+
);
|
|
79
|
+
|
|
80
|
+
const def = payloadDocs.get(sample.payloadFile)?.$defs?.[sample.event_type];
|
|
81
|
+
assert.ok(def, `published payload/${sample.payloadFile} has no $defs entry for ${sample.event_type}`);
|
|
82
|
+
|
|
83
|
+
const validatePayload = ajv.compile(def);
|
|
84
|
+
assert.ok(
|
|
85
|
+
validatePayload(event.payload),
|
|
86
|
+
`published payload schema rejected ${sample.event_type}: ${ajv.errorsText(validatePayload.errors)}`,
|
|
87
|
+
);
|
|
88
|
+
}
|
|
89
|
+
} finally {
|
|
90
|
+
rmSync(tmp, { recursive: true, force: true });
|
|
91
|
+
}
|
|
92
|
+
});
|