@crewhaus/watchme-store 0.4.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md ADDED
@@ -0,0 +1,72 @@
1
+ # @crewhaus/watchme-store
2
+
3
+ Durable state for the **"watch me"** feature (`crewhaus watchme`) — the store
4
+ that outlives the 30-day transcript TTL. Three pieces: a per-harness store of
5
+ redacted session digests and judge verdicts, the global cross-harness
6
+ registry, and the pure quality→shadow-arm join behind
7
+ `watchme report --feed-routing`. Zero-dep by design: redaction is an injected
8
+ callback at every CLI append site — this package never imports PII detectors
9
+ and never inspects content.
10
+
11
+ ## Per-harness store
12
+
13
+ `openWatchmeStore(rootDir, opts?)` roots the store at `<rootDir>/watchme`
14
+ (rootDir is the harness `.crewhaus` directory, the scoreboard convention) and
15
+ returns `{ dir, appendObservation, readObservations, readAggregates,
16
+ appendJudgment, readJudgments, compact, state, setState, windowKey,
17
+ acquireLock }`. Files (all mode `0600`):
18
+
19
+ - `observations.jsonl` — append-only `WatchmeObservation` digests plus
20
+ `{agg:1}` Welford aggregate lines. **No TTL**: digests must stand alone
21
+ after the transcript sweep (sessionId is provenance only, never a pointer).
22
+ Appends are atomic small-line `O_APPEND` writes, so concurrent harness
23
+ processes never lose each other's lines; malformed/torn lines are skipped
24
+ on read. `compact()` folds raw lines into one Welford aggregate per
25
+ `<specName>|<target>` key (mean/M2 of turn counts and pooled quality,
26
+ summed token/cost/tool/feedback counters, per-intent counts) and lands
27
+ write-then-rename; aggregates parallel-combine (Chan et al.) with later
28
+ lines on the next compaction.
29
+ - `judgments.jsonl` — append-only `WatchmeJudgment` verdicts from the
30
+ budgeted phase-2 judge. Machine signal stays OUT of the human feedback
31
+ channel; the explicit `--emit-feedback` bridge converts, never this store.
32
+ - `state.json` — `WatchmeState` (watching flag, analysis watermark, consumed
33
+ report windows, last report time), tmp+rename atomic. A missing or torn
34
+ file reads as the default rather than wedging the schedule.
35
+ - `run.lock` — advisory single-writer lock in the dream-engine mold:
36
+ `acquireLock()` is a try-once `O_EXCL` create returning a release fn or
37
+ `undefined` on contention; a lock older than 30 s (holder crashed) is
38
+ stolen and re-raced.
39
+
40
+ `windowKey(nowMs, everyMs)` is the report-window idempotency key —
41
+ `watchme:<spec>:<floor(nowMs/everyMs)>`, fixed epoch-anchored flooring so
42
+ every process computes the same window for the same clock. Pass `specName`
43
+ in the options; without it the harness directory basename stands in.
44
+
45
+ ## Cross-harness registry
46
+
47
+ `openHarnessRegistry(globalRoot, opts?)` opens
48
+ `<globalRoot>/harnesses.json` (global root default `~/.crewhaus/watchme`;
49
+ the CLI overrides via `CREWHAUS_WATCHME_ROOT` or `--root`) and returns
50
+ `{ register, deregister, list }`. Entries are keyed by absolute harness dir;
51
+ `register` upserts (keeping `registeredAt`, refreshing `lastSeen`) and lands
52
+ tmp+rename atomic. `list()` prunes tolerantly: entries whose dir vanished are
53
+ dropped from the returned list and reported via `onWarn` — the file is only
54
+ rewritten by `register`/`deregister`, so reads never race writers.
55
+
56
+ ## Quality join
57
+
58
+ `joinQualityToArms(decisions, quality)` joins durable route decisions to
59
+ delayed quality scores per `(sessionId, turnNumber)`; the decision's model
60
+ names the arm. Emitted rows carry SHADOW routeKeys — `"q:" + routeKey`, a
61
+ namespace the runtime router never mints or reads — so recording them
62
+ observes routing quality without steering it. Scores clamp to `[0, 1]` and
63
+ multiple scores for one turn average. Rewards are computed by the caller via
64
+ routing-store's `computeReward` with `obs.quality` set; this module stays
65
+ reward-free and fs-free.
66
+
67
+ ## Exports
68
+
69
+ `openWatchmeStore`, `openHarnessRegistry`, `joinQualityToArms`, and the types
70
+ `WatchmeObservation`, `WatchmeAggregate`, `WatchmeJudgment`, `WatchmeState`,
71
+ `HarnessEntry`, `WatchmeStore`, `WatchmeStoreOptions`, `HarnessRegistry`,
72
+ `HarnessRegistryOptions`, `RouteDecision`, `TurnQuality`, `QualityArmRow`.
@@ -0,0 +1,14 @@
1
+ /**
2
+ * `@crewhaus/watchme-store` — durable "watch me" state, independent of the
3
+ * 30-day transcript TTL. Three pieces: a per-harness store of redacted
4
+ * session digests + judge verdicts (`openWatchmeStore`), the global
5
+ * cross-harness registry (`openHarnessRegistry`), and the pure
6
+ * quality→shadow-arm join (`joinQualityToArms`) behind
7
+ * `crewhaus watchme report --feed-routing`. Zero-dep by design; redaction is
8
+ * an injected callback at every CLI append site — this package never imports
9
+ * PII detectors.
10
+ */
11
+ export { joinQualityToArms, type QualityArmRow, type RouteDecision, type TurnQuality, } from "./join.js";
12
+ export { openHarnessRegistry, type HarnessRegistry, type HarnessRegistryOptions, } from "./registry.js";
13
+ export { openWatchmeStore, type WatchmeStore, type WatchmeStoreOptions } from "./store.js";
14
+ export type { HarnessEntry, WatchmeAggregate, WatchmeJudgment, WatchmeObservation, WatchmeState, } from "./types.js";
package/dist/index.js ADDED
@@ -0,0 +1,13 @@
1
+ /**
2
+ * `@crewhaus/watchme-store` — durable "watch me" state, independent of the
3
+ * 30-day transcript TTL. Three pieces: a per-harness store of redacted
4
+ * session digests + judge verdicts (`openWatchmeStore`), the global
5
+ * cross-harness registry (`openHarnessRegistry`), and the pure
6
+ * quality→shadow-arm join (`joinQualityToArms`) behind
7
+ * `crewhaus watchme report --feed-routing`. Zero-dep by design; redaction is
8
+ * an injected callback at every CLI append site — this package never imports
9
+ * PII detectors.
10
+ */
11
+ export { joinQualityToArms, } from "./join.js";
12
+ export { openHarnessRegistry, } from "./registry.js";
13
+ export { openWatchmeStore } from "./store.js";
package/dist/join.d.ts ADDED
@@ -0,0 +1,45 @@
1
+ /**
2
+ * The pure quality→routing join behind `crewhaus watchme report --feed-routing`.
3
+ *
4
+ * Route decisions (durable `model_route` lines) and delayed quality scores
5
+ * (phase-2 judgments or normalized ratings) meet per `(sessionId, turnNumber)`;
6
+ * the model on the decision names the arm. Emitted rows carry SHADOW routeKeys
7
+ * — `"q:" + originalRouteKey` — a namespace the runtime router never mints or
8
+ * reads, so recording them observes routing quality without steering it.
9
+ * Rewards are computed by the CALLER via routing-store's `computeReward` with
10
+ * `obs.quality` set; this module stays reward-free.
11
+ */
12
+ /** One durable route decision (a `model_route` line, turn-attributed). */
13
+ export type RouteDecision = {
14
+ sessionId: string;
15
+ turnNumber: number;
16
+ routeKey: string;
17
+ model: string;
18
+ latencyMs?: number;
19
+ costUsd?: number;
20
+ success: boolean;
21
+ };
22
+ /** One delayed quality score for a turn, in `[0, 1]` (clamped on join). */
23
+ export type TurnQuality = {
24
+ sessionId: string;
25
+ turnNumber: number;
26
+ score: number;
27
+ };
28
+ /** One shadow-arm row ready for `openScoreboard(...).record(...)`. */
29
+ export type QualityArmRow = {
30
+ routeKey: string;
31
+ model: string;
32
+ obs: {
33
+ success: boolean;
34
+ latencyMs: number;
35
+ costUsd?: number;
36
+ quality: number;
37
+ };
38
+ };
39
+ /**
40
+ * Join route decisions to quality scores per `(sessionId, turnNumber)`.
41
+ * Decisions without a quality score are dropped (nothing to learn from);
42
+ * multiple scores for the same turn (a rating AND a judgment) average.
43
+ * Row order follows `decisions` order.
44
+ */
45
+ export declare function joinQualityToArms(decisions: ReadonlyArray<RouteDecision>, quality: ReadonlyArray<TurnQuality>): ReadonlyArray<QualityArmRow>;
package/dist/join.js ADDED
@@ -0,0 +1,44 @@
1
+ /**
2
+ * The pure quality→routing join behind `crewhaus watchme report --feed-routing`.
3
+ *
4
+ * Route decisions (durable `model_route` lines) and delayed quality scores
5
+ * (phase-2 judgments or normalized ratings) meet per `(sessionId, turnNumber)`;
6
+ * the model on the decision names the arm. Emitted rows carry SHADOW routeKeys
7
+ * — `"q:" + originalRouteKey` — a namespace the runtime router never mints or
8
+ * reads, so recording them observes routing quality without steering it.
9
+ * Rewards are computed by the CALLER via routing-store's `computeReward` with
10
+ * `obs.quality` set; this module stays reward-free.
11
+ */
12
+ const SHADOW_PREFIX = "q:";
13
+ const clamp01 = (x) => Math.min(1, Math.max(0, x));
14
+ /**
15
+ * Join route decisions to quality scores per `(sessionId, turnNumber)`.
16
+ * Decisions without a quality score are dropped (nothing to learn from);
17
+ * multiple scores for the same turn (a rating AND a judgment) average.
18
+ * Row order follows `decisions` order.
19
+ */
20
+ export function joinQualityToArms(decisions, quality) {
21
+ const scores = new Map();
22
+ for (const q of quality) {
23
+ const key = `${q.sessionId}\u0000${q.turnNumber}`;
24
+ const acc = scores.get(key) ?? { sum: 0, n: 0 };
25
+ acc.sum += clamp01(q.score);
26
+ acc.n += 1;
27
+ scores.set(key, acc);
28
+ }
29
+ const rows = [];
30
+ for (const d of decisions) {
31
+ const acc = scores.get(`${d.sessionId}\u0000${d.turnNumber}`);
32
+ if (acc === undefined || acc.n === 0)
33
+ continue;
34
+ const obs = {
35
+ success: d.success,
36
+ latencyMs: Math.max(0, d.latencyMs ?? 0),
37
+ quality: clamp01(acc.sum / acc.n),
38
+ };
39
+ if (d.costUsd !== undefined)
40
+ obs.costUsd = d.costUsd;
41
+ rows.push({ routeKey: `${SHADOW_PREFIX}${d.routeKey}`, model: d.model, obs });
42
+ }
43
+ return rows;
44
+ }
@@ -0,0 +1,17 @@
1
+ import type { HarnessEntry } from "./types.js";
2
+ export type HarnessRegistry = {
3
+ /** Upsert by (absolute) dir: a re-register keeps `registeredAt` and
4
+ * refreshes `lastSeen` + the descriptive fields. tmp+rename atomic. */
5
+ register(entry: Omit<HarnessEntry, "registeredAt" | "lastSeen">): void;
6
+ deregister(dir: string): void;
7
+ /** Tolerant-pruning: entries whose dir vanished are dropped + reported. */
8
+ list(): HarnessEntry[];
9
+ };
10
+ export type HarnessRegistryOptions = {
11
+ /** Clock for `registeredAt`/`lastSeen`; injected so tests are deterministic. */
12
+ readonly now?: () => number;
13
+ /** Receives one line per pruned vanished-dir entry. Default: `console.error`. */
14
+ readonly onWarn?: (message: string) => void;
15
+ };
16
+ /** Open the registry rooted at `globalRoot` (created lazily on first write). */
17
+ export declare function openHarnessRegistry(globalRoot: string, opts?: HarnessRegistryOptions): HarnessRegistry;
@@ -0,0 +1,86 @@
1
+ /**
2
+ * The cross-harness registry at `<globalRoot>/harnesses.json` (global root
3
+ * default `~/.crewhaus/watchme`; the CLI overrides via `CREWHAUS_WATCHME_ROOT`
4
+ * or `--root`). One JSON document `{v:1, harnesses: HarnessEntry[]}`, absolute
5
+ * dirs, upsert-by-dir, tmp+rename atomic writes, tolerant reads.
6
+ *
7
+ * `list()` prunes tolerantly: an entry whose directory has vanished (harness
8
+ * deleted, volume unmounted) is dropped from the RETURNED list and reported
9
+ * via `onWarn` — the file itself is only rewritten by `register`/`deregister`,
10
+ * so a read never races a concurrent writer.
11
+ */
12
+ import { existsSync, mkdirSync, readFileSync, renameSync, writeFileSync } from "node:fs";
13
+ import { join, resolve } from "node:path";
14
+ const REGISTRY_FILENAME = "harnesses.json";
15
+ function isEntry(rec) {
16
+ if (typeof rec !== "object" || rec === null)
17
+ return false;
18
+ const e = rec;
19
+ return (typeof e["dir"] === "string" &&
20
+ typeof e["specName"] === "string" &&
21
+ typeof e["target"] === "string" &&
22
+ typeof e["registeredAt"] === "number" &&
23
+ typeof e["lastSeen"] === "number");
24
+ }
25
+ /** Open the registry rooted at `globalRoot` (created lazily on first write). */
26
+ export function openHarnessRegistry(globalRoot, opts = {}) {
27
+ const path = join(globalRoot, REGISTRY_FILENAME);
28
+ const now = opts.now ?? Date.now;
29
+ const onWarn = opts.onWarn ?? ((message) => console.error(message));
30
+ const readAll = () => {
31
+ if (!existsSync(path))
32
+ return [];
33
+ try {
34
+ const parsed = JSON.parse(readFileSync(path, "utf8"));
35
+ if (typeof parsed === "object" && parsed !== null) {
36
+ const harnesses = parsed["harnesses"];
37
+ if (Array.isArray(harnesses))
38
+ return harnesses.filter(isEntry);
39
+ }
40
+ }
41
+ catch {
42
+ // Unreadable → treated as empty; the next write rewrites it whole.
43
+ }
44
+ return [];
45
+ };
46
+ const writeAll = (entries) => {
47
+ if (!existsSync(globalRoot))
48
+ mkdirSync(globalRoot, { recursive: true, mode: 0o700 });
49
+ const sorted = [...entries].sort((a, b) => a.dir.localeCompare(b.dir));
50
+ const tmp = `${path}.tmp`;
51
+ writeFileSync(tmp, `${JSON.stringify({ v: 1, harnesses: sorted }, null, 2)}\n`, {
52
+ mode: 0o600,
53
+ });
54
+ renameSync(tmp, path);
55
+ };
56
+ return {
57
+ register(entry) {
58
+ const dir = resolve(entry.dir);
59
+ const entries = readAll();
60
+ const existing = entries.find((e) => e.dir === dir);
61
+ const next = {
62
+ ...entry,
63
+ dir,
64
+ registeredAt: existing?.registeredAt ?? now(),
65
+ lastSeen: now(),
66
+ };
67
+ writeAll([...entries.filter((e) => e.dir !== dir), next]);
68
+ },
69
+ deregister(dir) {
70
+ const absolute = resolve(dir);
71
+ writeAll(readAll().filter((e) => e.dir !== absolute));
72
+ },
73
+ list() {
74
+ const alive = [];
75
+ for (const entry of readAll()) {
76
+ if (existsSync(entry.dir)) {
77
+ alive.push(entry);
78
+ }
79
+ else {
80
+ onWarn(`watchme-store: registry entry for vanished dir ${entry.dir} dropped`);
81
+ }
82
+ }
83
+ return alive;
84
+ },
85
+ };
86
+ }
@@ -0,0 +1,40 @@
1
+ import type { WatchmeAggregate, WatchmeJudgment, WatchmeObservation, WatchmeState } from "./types.js";
2
+ export type WatchmeStoreOptions = {
3
+ /**
4
+ * Spec name used in {@link WatchmeStore.windowKey} (`watchme:<spec>:<idx>`).
5
+ * When omitted the harness directory basename (rootDir's parent) stands in —
6
+ * still collision-exact for every process on the same harness.
7
+ */
8
+ readonly specName?: string;
9
+ };
10
+ export type WatchmeStore = {
11
+ /** `<rootDir>/watchme` — where every store file lives. */
12
+ readonly dir: string;
13
+ /** Append one redacted digest (`O_APPEND`, mode 0600). */
14
+ appendObservation(obs: WatchmeObservation): void;
15
+ /** All raw (non-aggregate) observation lines; malformed lines are skipped. */
16
+ readObservations(): WatchmeObservation[];
17
+ /** All `{agg:1}` aggregate lines written by `compact()`. */
18
+ readAggregates(): WatchmeAggregate[];
19
+ /** Append one judge verdict to `judgments.jsonl`. */
20
+ appendJudgment(j: WatchmeJudgment): void;
21
+ readJudgments(): WatchmeJudgment[];
22
+ /** Fold raw observations (and prior aggregates) into one Welford aggregate
23
+ * line per key; write-then-rename. Run under `acquireLock()`. */
24
+ compact(): void;
25
+ /** Current state; a missing or torn `state.json` reads as the default. */
26
+ state(): WatchmeState;
27
+ /** Top-level shallow merge (nested objects are replaced wholesale); lands
28
+ * tmp+rename atomic, mode 0600. */
29
+ setState(patch: Partial<WatchmeState>): void;
30
+ /** `watchme:<spec>:<floor(nowMs/everyMs)>` — the report-window idempotency
31
+ * key (dream-engine window math: fixed epoch-anchored flooring). */
32
+ windowKey(nowMs: number, everyMs: number): string;
33
+ /** Try-once advisory `run.lock` acquire: a release fn on success,
34
+ * `undefined` on contention. The holder refreshes the lock mtime while it
35
+ * works, so only a lock idle past {@link LOCK_STALE_MS} (holder crashed) is
36
+ * stolen and re-raced, per the §7.6 policy. */
37
+ acquireLock(): (() => void) | undefined;
38
+ };
39
+ /** Open (or create) the watch-me store rooted at the harness `.crewhaus` dir. */
40
+ export declare function openWatchmeStore(rootDir: string, opts?: WatchmeStoreOptions): WatchmeStore;
package/dist/store.js ADDED
@@ -0,0 +1,362 @@
1
+ /**
2
+ * The per-harness "watch me" store at `<rootDir>/watchme` (rootDir is the
3
+ * harness `.crewhaus` directory, scoreboard convention).
4
+ *
5
+ * Files (all mode 0600):
6
+ * - `observations.jsonl` — append-only {@link WatchmeObservation} digests
7
+ * plus `{agg:1}` Welford aggregate lines written by `compact()`. NO TTL:
8
+ * this is the long-horizon store that outlives the 30-day transcript
9
+ * sweep, so digests must stand alone (no pointers into raw transcripts).
10
+ * - `judgments.jsonl` — append-only {@link WatchmeJudgment} verdicts from
11
+ * the budgeted phase-2 judge (machine signal, kept OUT of the human
12
+ * feedback channel).
13
+ * - `state.json` — {@link WatchmeState}, tmp+rename atomic.
14
+ * - `run.lock` — advisory single-writer lock in the dream-engine `run.lock`
15
+ * mold (O_EXCL create). The holder is heartbeated while held and only a
16
+ * lock idle past {@link LOCK_STALE_MS} (a model-phase-scale window) is
17
+ * stolen, so a legitimately long report/judge pass is never evicted.
18
+ *
19
+ * Concurrency model is the scoreboard's: appends are atomic small-line
20
+ * `O_APPEND` writes so concurrent harness processes never lose each other's
21
+ * lines; `compact()` and `setState()` are single-writer maintenance ops run
22
+ * under `acquireLock()` and land write-then-rename so a reader never
23
+ * observes a torn file.
24
+ */
25
+ import { appendFileSync, existsSync, mkdirSync, readFileSync, renameSync, statSync, unlinkSync, utimesSync, writeFileSync, } from "node:fs";
26
+ import { basename, dirname, join, resolve } from "node:path";
27
+ /**
28
+ * A lock whose file mtime is older than this is presumed abandoned (§7.6).
29
+ * A watchme report legitimately holds the lock across the whole phase-1 grader
30
+ * pass plus phase-2 judge model calls — minutes, not seconds — so the window
31
+ * is model-phase-scale, matching dream-engine's own `staleMs: 15 * 60_000`
32
+ * (packages/dream-engine/src/index.ts). The holder additionally heartbeats the
33
+ * lock mtime while it works, so even a run longer than this is never stolen.
34
+ */
35
+ const LOCK_STALE_MS = 15 * 60_000;
36
+ /** How often the holder refreshes the lock mtime (well under LOCK_STALE_MS). */
37
+ const LOCK_HEARTBEAT_MS = 60_000;
38
+ const DEFAULT_STATE = {
39
+ schemaVersion: 1,
40
+ watching: false,
41
+ watermark: { lastMtimeMs: 0 },
42
+ windows: {},
43
+ };
44
+ function emptyFold(key) {
45
+ return {
46
+ key,
47
+ n: 0,
48
+ meanTurns: 0,
49
+ m2Turns: 0,
50
+ qualityN: 0,
51
+ meanQuality: 0,
52
+ m2Quality: 0,
53
+ tokensIn: 0,
54
+ tokensOut: 0,
55
+ costUsdMicros: 0,
56
+ toolCalls: 0,
57
+ toolErrors: 0,
58
+ feedbackUp: 0,
59
+ feedbackDown: 0,
60
+ intents: {},
61
+ };
62
+ }
63
+ /** Welford update of one (mean, M2) pair. */
64
+ function foldSample(state, x) {
65
+ state.n += 1;
66
+ const delta = x - state.mean;
67
+ state.mean += delta / state.n;
68
+ state.m2 += delta * (x - state.mean);
69
+ }
70
+ /** Chan et al. parallel combine of aggregate B into A. */
71
+ function combinePair(a, bN, bMean, bM2) {
72
+ if (bN <= 0)
73
+ return;
74
+ const n = a.n + bN;
75
+ const delta = bMean - a.mean;
76
+ a.mean = a.mean + (delta * bN) / n;
77
+ a.m2 = a.m2 + bM2 + (delta * delta * a.n * bN) / n;
78
+ a.n = n;
79
+ }
80
+ /** The count-weighted pooled quality mean of one observation, if it has one. */
81
+ function qualitySample(obs) {
82
+ const q = obs.quality;
83
+ if (q === undefined)
84
+ return undefined;
85
+ let weight = 0;
86
+ let sum = 0;
87
+ if (q.meanRating !== undefined && q.ratings > 0) {
88
+ weight += q.ratings;
89
+ sum += q.ratings * q.meanRating;
90
+ }
91
+ if (q.meanJudge !== undefined && q.judged > 0) {
92
+ weight += q.judged;
93
+ sum += q.judged * q.meanJudge;
94
+ }
95
+ return weight > 0 ? sum / weight : undefined;
96
+ }
97
+ function foldObservation(fold, obs) {
98
+ const turns = { n: fold.n, mean: fold.meanTurns, m2: fold.m2Turns };
99
+ foldSample(turns, obs.turnCount);
100
+ fold.n = turns.n;
101
+ fold.meanTurns = turns.mean;
102
+ fold.m2Turns = turns.m2;
103
+ const q = qualitySample(obs);
104
+ if (q !== undefined) {
105
+ const quality = { n: fold.qualityN, mean: fold.meanQuality, m2: fold.m2Quality };
106
+ foldSample(quality, q);
107
+ fold.qualityN = quality.n;
108
+ fold.meanQuality = quality.mean;
109
+ fold.m2Quality = quality.m2;
110
+ }
111
+ for (const m of obs.models) {
112
+ fold.tokensIn += m.usage.in;
113
+ fold.tokensOut += m.usage.out;
114
+ fold.costUsdMicros += m.costUsdMicros ?? 0;
115
+ }
116
+ for (const t of obs.toolStats) {
117
+ fold.toolCalls += t.calls;
118
+ fold.toolErrors += t.errors;
119
+ }
120
+ fold.feedbackUp += obs.feedback?.up ?? 0;
121
+ fold.feedbackDown += obs.feedback?.down ?? 0;
122
+ for (const key of obs.intentKeys)
123
+ fold.intents[key] = (fold.intents[key] ?? 0) + 1;
124
+ }
125
+ function foldAggregate(fold, agg) {
126
+ const turns = { n: fold.n, mean: fold.meanTurns, m2: fold.m2Turns };
127
+ combinePair(turns, agg.n, agg.meanTurns, agg.m2Turns);
128
+ fold.n = turns.n;
129
+ fold.meanTurns = turns.mean;
130
+ fold.m2Turns = turns.m2;
131
+ const quality = { n: fold.qualityN, mean: fold.meanQuality, m2: fold.m2Quality };
132
+ combinePair(quality, agg.qualityN, agg.meanQuality, agg.m2Quality);
133
+ fold.qualityN = quality.n;
134
+ fold.meanQuality = quality.mean;
135
+ fold.m2Quality = quality.m2;
136
+ fold.tokensIn += agg.tokensIn;
137
+ fold.tokensOut += agg.tokensOut;
138
+ fold.costUsdMicros += agg.costUsdMicros;
139
+ fold.toolCalls += agg.toolCalls;
140
+ fold.toolErrors += agg.toolErrors;
141
+ fold.feedbackUp += agg.feedbackUp;
142
+ fold.feedbackDown += agg.feedbackDown;
143
+ for (const [key, count] of Object.entries(agg.intents)) {
144
+ fold.intents[key] = (fold.intents[key] ?? 0) + count;
145
+ }
146
+ }
147
+ function toAggregate(fold) {
148
+ return {
149
+ v: 1,
150
+ agg: 1,
151
+ key: fold.key,
152
+ n: fold.n,
153
+ meanTurns: fold.meanTurns,
154
+ m2Turns: fold.m2Turns,
155
+ meanQuality: fold.meanQuality,
156
+ m2Quality: fold.m2Quality,
157
+ qualityN: fold.qualityN,
158
+ tokensIn: fold.tokensIn,
159
+ tokensOut: fold.tokensOut,
160
+ costUsdMicros: fold.costUsdMicros,
161
+ toolCalls: fold.toolCalls,
162
+ toolErrors: fold.toolErrors,
163
+ feedbackUp: fold.feedbackUp,
164
+ feedbackDown: fold.feedbackDown,
165
+ intents: fold.intents,
166
+ };
167
+ }
168
+ /** Parse a JSONL file into records, skipping blank/torn/malformed lines. */
169
+ function readJsonl(path) {
170
+ if (!existsSync(path))
171
+ return [];
172
+ const records = [];
173
+ for (const line of readFileSync(path, "utf8").split("\n")) {
174
+ const trimmed = line.trim();
175
+ if (trimmed.length === 0)
176
+ continue;
177
+ try {
178
+ const parsed = JSON.parse(trimmed);
179
+ if (typeof parsed === "object" && parsed !== null) {
180
+ records.push(parsed);
181
+ }
182
+ }
183
+ catch {
184
+ // Tolerate a torn final line from a crashed writer.
185
+ }
186
+ }
187
+ return records;
188
+ }
189
+ const isObservation = (rec) => rec["agg"] !== 1 &&
190
+ typeof rec["sessionId"] === "string" &&
191
+ typeof rec["specName"] === "string" &&
192
+ typeof rec["turnCount"] === "number";
193
+ const isAggregate = (rec) => rec["agg"] === 1 && typeof rec["key"] === "string" && typeof rec["n"] === "number";
194
+ const isJudgment = (rec) => typeof rec["sessionId"] === "string" &&
195
+ typeof rec["turnNumber"] === "number" &&
196
+ typeof rec["score"] === "number";
197
+ /** Open (or create) the watch-me store rooted at the harness `.crewhaus` dir. */
198
+ export function openWatchmeStore(rootDir, opts = {}) {
199
+ const dir = join(rootDir, "watchme");
200
+ const observationsPath = join(dir, "observations.jsonl");
201
+ const judgmentsPath = join(dir, "judgments.jsonl");
202
+ const statePath = join(dir, "state.json");
203
+ const lockPath = join(dir, "run.lock");
204
+ const specToken = opts.specName ?? basename(dirname(resolve(rootDir)));
205
+ const ensureDir = () => {
206
+ if (!existsSync(dir))
207
+ mkdirSync(dir, { recursive: true, mode: 0o700 });
208
+ };
209
+ const readObservations = () => {
210
+ // Last-writer-wins per sessionId: when a session is re-analyzed after it
211
+ // grew, its later digest supersedes the earlier line, so a re-appended or
212
+ // grown session is counted ONCE with its latest turnCount/sums (no
213
+ // double-count). `n` therefore reflects DISTINCT sessions. The Map keeps
214
+ // each session at its first-seen position while carrying the last value,
215
+ // so append order over distinct sessions is preserved.
216
+ const all = readJsonl(observationsPath).filter(isObservation);
217
+ const byId = new Map();
218
+ for (const obs of all)
219
+ byId.set(obs.sessionId, obs);
220
+ return [...byId.values()];
221
+ };
222
+ const readAggregates = () => readJsonl(observationsPath).filter(isAggregate);
223
+ const state = () => {
224
+ if (!existsSync(statePath))
225
+ return DEFAULT_STATE;
226
+ try {
227
+ const parsed = JSON.parse(readFileSync(statePath, "utf8"));
228
+ if (typeof parsed === "object" &&
229
+ parsed !== null &&
230
+ parsed["schemaVersion"] === 1) {
231
+ return parsed;
232
+ }
233
+ }
234
+ catch {
235
+ // A torn or hand-mangled state file reads as never-ran rather than
236
+ // wedging the schedule (dream-engine semantics).
237
+ }
238
+ return DEFAULT_STATE;
239
+ };
240
+ return {
241
+ dir,
242
+ appendObservation(obs) {
243
+ ensureDir();
244
+ appendFileSync(observationsPath, `${JSON.stringify(obs)}\n`, { mode: 0o600 });
245
+ },
246
+ readObservations,
247
+ readAggregates,
248
+ appendJudgment(j) {
249
+ ensureDir();
250
+ appendFileSync(judgmentsPath, `${JSON.stringify(j)}\n`, { mode: 0o600 });
251
+ },
252
+ readJudgments() {
253
+ return readJsonl(judgmentsPath).filter(isJudgment);
254
+ },
255
+ compact() {
256
+ ensureDir();
257
+ const folds = new Map();
258
+ const foldFor = (key) => {
259
+ let fold = folds.get(key);
260
+ if (fold === undefined) {
261
+ fold = emptyFold(key);
262
+ folds.set(key, fold);
263
+ }
264
+ return fold;
265
+ };
266
+ // Prior aggregates first, then raw deltas — parallel-combine is
267
+ // order-insensitive, but this mirrors on-disk order for readability.
268
+ // `readObservations()` already dedups raw lines by sessionId (last
269
+ // wins), so a grown session's latest digest supersedes its earlier one
270
+ // BEFORE folding: `n` counts distinct sessions and the summed counters
271
+ // reflect the per-latest-digest values, never a double-count.
272
+ for (const agg of readAggregates())
273
+ foldAggregate(foldFor(agg.key), agg);
274
+ for (const obs of readObservations()) {
275
+ foldObservation(foldFor(`${obs.specName}|${obs.target}`), obs);
276
+ }
277
+ const lines = [...folds.values()]
278
+ .filter((f) => f.n > 0)
279
+ .sort((a, b) => a.key.localeCompare(b.key))
280
+ .map((f) => JSON.stringify(toAggregate(f)));
281
+ // Write-then-rename for an atomic swap: a concurrent reader sees either
282
+ // the old or the new file, never a half-written one.
283
+ const tmp = `${observationsPath}.tmp`;
284
+ writeFileSync(tmp, lines.length > 0 ? `${lines.join("\n")}\n` : "", { mode: 0o600 });
285
+ renameSync(tmp, observationsPath);
286
+ },
287
+ state,
288
+ setState(patch) {
289
+ ensureDir();
290
+ const next = { ...state(), ...patch, schemaVersion: 1 };
291
+ const tmp = `${statePath}.tmp`;
292
+ writeFileSync(tmp, `${JSON.stringify(next, null, 2)}\n`, { mode: 0o600 });
293
+ renameSync(tmp, statePath);
294
+ },
295
+ windowKey(nowMs, everyMs) {
296
+ return `watchme:${specToken}:${Math.floor(nowMs / everyMs)}`;
297
+ },
298
+ acquireLock() {
299
+ ensureDir();
300
+ for (;;) {
301
+ try {
302
+ const payload = { pid: process.pid, acquiredAt: new Date().toISOString() };
303
+ writeFileSync(lockPath, `${JSON.stringify(payload)}\n`, { flag: "wx", mode: 0o600 });
304
+ // Heartbeat the mtime so a live-but-slow holder is never presumed
305
+ // stale (unref'd: never keeps the process alive; fires during the
306
+ // judge phase's awaits). Best-effort — a torn touch just risks one
307
+ // early steal, which the stale window already tolerates.
308
+ const heartbeat = setInterval(() => {
309
+ try {
310
+ const now = new Date();
311
+ utimesSync(lockPath, now, now);
312
+ }
313
+ catch {
314
+ // Lock vanished (released/stolen) — nothing to refresh.
315
+ }
316
+ }, LOCK_HEARTBEAT_MS);
317
+ heartbeat.unref?.();
318
+ let released = false;
319
+ return () => {
320
+ if (released)
321
+ return; // Idempotent.
322
+ released = true;
323
+ clearInterval(heartbeat);
324
+ try {
325
+ unlinkSync(lockPath);
326
+ }
327
+ catch {
328
+ // Already gone — release is idempotent.
329
+ }
330
+ };
331
+ }
332
+ catch (err) {
333
+ if (err.code !== "EEXIST")
334
+ throw err;
335
+ }
336
+ let mtimeMs;
337
+ try {
338
+ mtimeMs = statSync(lockPath).mtimeMs;
339
+ }
340
+ catch {
341
+ continue; // The holder released between create and stat — re-race.
342
+ }
343
+ if (Date.now() - mtimeMs > LOCK_STALE_MS) {
344
+ try {
345
+ // Re-stat immediately before the destructive steal: only unlink if
346
+ // the lock is STILL the same stale instance we observed. This
347
+ // narrows the two-waiter race where both observe a dead lock and
348
+ // one unlinks the other's freshly-acquired lock.
349
+ if (statSync(lockPath).mtimeMs === mtimeMs) {
350
+ unlinkSync(lockPath);
351
+ }
352
+ }
353
+ catch {
354
+ // Another waiter stole it first, or the holder just released.
355
+ }
356
+ continue; // Re-race the create — another waiter may legitimately win.
357
+ }
358
+ return undefined;
359
+ }
360
+ },
361
+ };
362
+ }
@@ -0,0 +1,178 @@
1
+ /**
2
+ * Record schemas for the "watch me" durable store. Everything persisted by
3
+ * `@crewhaus/watchme-store` is one of these shapes, serialized as JSONL (the
4
+ * append-only files) or a single JSON document (state / registry).
5
+ *
6
+ * Redaction is NOT this package's job: every text-bearing field is redacted
7
+ * by the caller BEFORE append (the CLI injects its PII redactor). This
8
+ * package never imports detectors and never inspects content.
9
+ */
10
+ /**
11
+ * One redacted per-session digest line. NO transcript text except through the
12
+ * injected redact callback; NO pointers into raw transcripts (sessionId is
13
+ * provenance only — the digest must stand alone after TTL eviction).
14
+ */
15
+ export type WatchmeObservation = {
16
+ readonly v: 1;
17
+ readonly sessionId: string;
18
+ readonly specName: string;
19
+ readonly target: string;
20
+ readonly agentId?: string;
21
+ /** Analysis time, epoch ms. */
22
+ readonly ts: number;
23
+ /** deriveTurns count (the turnNumber contract). */
24
+ readonly turnCount: number;
25
+ /** How per-turn model attribution was derived: envelope turnNumber from the
26
+ * `.events.jsonl` sibling (`exact`) vs insertion-order correlation of the
27
+ * durable mirrors (`ordered` — aggregate at session level only). */
28
+ readonly joinConfidence: "exact" | "ordered";
29
+ readonly models: ReadonlyArray<{
30
+ readonly wire: string;
31
+ readonly spec?: string;
32
+ readonly provider: string;
33
+ readonly turns: number;
34
+ readonly usage: {
35
+ in: number;
36
+ out: number;
37
+ cacheRead: number;
38
+ cacheCreate: number;
39
+ };
40
+ readonly costUsdMicros?: number;
41
+ readonly unpriced?: boolean;
42
+ }>;
43
+ readonly toolStats: ReadonlyArray<{
44
+ readonly name: string;
45
+ readonly calls: number;
46
+ readonly errors: number;
47
+ }>;
48
+ readonly continuity?: Readonly<Record<string, {
49
+ score: number;
50
+ passed: boolean;
51
+ higherIsBetter: boolean;
52
+ }>>;
53
+ readonly factuality?: {
54
+ readonly claims: number;
55
+ readonly grounded: number;
56
+ readonly faithfulness?: number;
57
+ readonly hallucinationRate?: number;
58
+ readonly vacuous?: boolean;
59
+ };
60
+ readonly quality?: {
61
+ readonly ratings: number;
62
+ readonly meanRating?: number;
63
+ readonly judged: number;
64
+ readonly meanJudge?: number;
65
+ };
66
+ /** Cluster keys only (redacted upstream). */
67
+ readonly intentKeys: ReadonlyArray<string>;
68
+ readonly feedback?: {
69
+ readonly up: number;
70
+ readonly down: number;
71
+ };
72
+ };
73
+ /**
74
+ * Welford aggregate line: `{v:1, agg:1, key, n, mean..., m2...}` (the
75
+ * scoreboard.ts line grammar, cloned). `compact()` folds raw observation
76
+ * lines into one aggregate per key so `observations.jsonl` stays a bounded
77
+ * long-horizon store; aggregates parallel-combine (Chan et al.) with later
78
+ * aggregates on the next compaction.
79
+ */
80
+ export type WatchmeAggregate = {
81
+ readonly v: 1;
82
+ readonly agg: 1;
83
+ /** Aggregation key: `<specName>|<target>`. */
84
+ readonly key: string;
85
+ /** Observations folded into this aggregate. */
86
+ readonly n: number;
87
+ /** Welford mean/M2 of per-session `turnCount`. */
88
+ readonly meanTurns: number;
89
+ readonly m2Turns: number;
90
+ /** Welford mean/M2 of the per-session quality mean (ratings and judgments
91
+ * pooled, count-weighted); `qualityN` counts the sessions that carried one. */
92
+ readonly meanQuality: number;
93
+ readonly m2Quality: number;
94
+ readonly qualityN: number;
95
+ /** Summed counters across folded observations. */
96
+ readonly tokensIn: number;
97
+ readonly tokensOut: number;
98
+ readonly costUsdMicros: number;
99
+ readonly toolCalls: number;
100
+ readonly toolErrors: number;
101
+ readonly feedbackUp: number;
102
+ readonly feedbackDown: number;
103
+ /** Per-intent-cluster occurrence counts (cluster keys only, redacted upstream). */
104
+ readonly intents: Readonly<Record<string, number>>;
105
+ };
106
+ /** Phase-2 judge verdict — watchme's OWN store, deliberately NOT a FeedbackRecord. */
107
+ export type WatchmeJudgment = {
108
+ readonly v: 1;
109
+ readonly sessionId: string;
110
+ readonly turnNumber: number;
111
+ /** Model that PRODUCED the turn (spec string). */
112
+ readonly model: string;
113
+ readonly judgeModel: string;
114
+ /** Judge score in `[0, 1]`. */
115
+ readonly score: number;
116
+ /** Redacted before append. */
117
+ readonly rationale: string;
118
+ readonly ts: number;
119
+ };
120
+ /** The `watchme/state.json` document. */
121
+ export type WatchmeState = {
122
+ readonly schemaVersion: 1;
123
+ readonly watching: boolean;
124
+ readonly startedAt?: number;
125
+ /** Analysis high-water mark over the sessions directory — a cheap scalar
126
+ * pre-filter only. The authority for the skip/growth decision is the
127
+ * per-session `observed` cursor below; the watermark is consulted solely
128
+ * for sessions the cursor has never tracked (pre-cursor digests or entries
129
+ * the cursor cap evicted), so a grown session is never permanently skipped. */
130
+ readonly watermark: {
131
+ readonly lastMtimeMs: number;
132
+ readonly lastSessionId?: string;
133
+ };
134
+ /**
135
+ * Durable per-session analysis cursor: the last-digested `mtimeMs` +
136
+ * `turnCount` for each sessionId. It lives in `state.json`, which
137
+ * `compact()` never rewrites, so it survives compaction — unlike the raw
138
+ * observation lines, which fold away into `{agg:1}` aggregates. Enumeration
139
+ * skips a session only while its on-disk state is unchanged relative to this
140
+ * cursor; a session whose file grew (mtime advanced ⇒ turnCount may have
141
+ * grown) is re-analyzed, and the last-writer-wins observation log makes the
142
+ * re-digest idempotent. Bounded to the most-recent entries by `mtimeMs`
143
+ * (oldest evicted, cap in watchme-report) so `state.json` stays small.
144
+ */
145
+ readonly observed?: Readonly<Record<string, {
146
+ readonly mtimeMs: number;
147
+ readonly turnCount: number;
148
+ }>>;
149
+ /** Durable `sessionId#turnNumber` keys already recorded as shadow routing
150
+ * arms by `report --feed-routing`, so re-runs never double-record the same
151
+ * arm even after the sessions fall past the analysis watermark. Bounded
152
+ * (oldest-first eviction, cap in watchme-report). */
153
+ readonly fedRoutingKeys?: readonly string[];
154
+ /** Consumed report windows by `windowKey` → outcome. `ok` and
155
+ * `model_refused_unpriced` consume the window; a transient `model_failed`
156
+ * is recorded but the next tick retries (dream-engine semantics). */
157
+ readonly windows: Readonly<Record<string, "ok" | "model_refused_unpriced" | "model_failed">>;
158
+ readonly lastReportAt?: number;
159
+ };
160
+ /** One row of the global `~/.crewhaus/watchme/harnesses.json` registry. */
161
+ export type HarnessEntry = {
162
+ /** Absolute harness directory (the standalone-harness cwd). */
163
+ readonly dir: string;
164
+ readonly specName: string;
165
+ readonly target: string;
166
+ readonly agentId?: string;
167
+ /**
168
+ * Whether this harness opted into cross-harness co-learning (`watchme.share`
169
+ * from the spec), captured at registration time. `report --all` recalls a
170
+ * peer's local wiki articles ONLY when this is `true`. Absent on legacy
171
+ * entries → treated as not-shared (backward-compatible default). This is an
172
+ * intent flag on the same-machine registry, NOT a Thredz visibility signal
173
+ * (cross-machine sharing is a deferred seam — design/watch-me.md §10).
174
+ */
175
+ readonly share?: boolean;
176
+ readonly registeredAt: number;
177
+ readonly lastSeen: number;
178
+ };
package/dist/types.js ADDED
@@ -0,0 +1,10 @@
1
+ /**
2
+ * Record schemas for the "watch me" durable store. Everything persisted by
3
+ * `@crewhaus/watchme-store` is one of these shapes, serialized as JSONL (the
4
+ * append-only files) or a single JSON document (state / registry).
5
+ *
6
+ * Redaction is NOT this package's job: every text-bearing field is redacted
7
+ * by the caller BEFORE append (the CLI injects its PII redactor). This
8
+ * package never imports detectors and never inspects content.
9
+ */
10
+ export {};
package/package.json ADDED
@@ -0,0 +1,37 @@
1
+ {
2
+ "name": "@crewhaus/watchme-store",
3
+ "version": "0.4.2",
4
+ "type": "module",
5
+ "description": "Durable \"watch me\" observation store — per-harness session digests, judge verdicts, and the cross-harness registry behind crewhaus watchme",
6
+ "main": "dist/index.js",
7
+ "types": "dist/index.d.ts",
8
+ "exports": {
9
+ ".": {
10
+ "types": "./dist/index.d.ts",
11
+ "import": "./dist/index.js"
12
+ }
13
+ },
14
+ "scripts": {
15
+ "test": "bun test src"
16
+ },
17
+ "dependencies": {},
18
+ "license": "Apache-2.0",
19
+ "author": {
20
+ "name": "Max Meier",
21
+ "email": "max@crewhaus.ai",
22
+ "url": "https://crewhaus.ai"
23
+ },
24
+ "repository": {
25
+ "type": "git",
26
+ "url": "git+https://github.com/crewhaus/factory.git",
27
+ "directory": "packages/watchme-store"
28
+ },
29
+ "homepage": "https://github.com/crewhaus/factory/tree/main/packages/watchme-store#readme",
30
+ "bugs": {
31
+ "url": "https://github.com/crewhaus/factory/issues"
32
+ },
33
+ "publishConfig": {
34
+ "access": "public"
35
+ },
36
+ "files": ["dist", "README.md", "LICENSE", "NOTICE"]
37
+ }