@possumtech/rummy 0.5.0 → 2.0.0

package/FIDELITY_CONTRACT.md

@@ -1,172 +0,0 @@
-# Fidelity Contract — Observed State vs Intended
-
-## Observed Behavior (traced from test/mab/results/2026-04-14T15-13-55-950Z/last_run.txt, turn 24)
-
-### Flow
-
-```
-Model emits tool
-      ↓
-Tool handler stores body in known_entries.body (raw, as model wrote it)
-      ↓
-Next turn: TurnExecutor materializes context
-      ↓
-For each row: hooks.tools.view(scheme, entry) → plugin's view hook returns projected body
-      ↓
-Projected body stored in turn_context.body with fidelity-projected token count
-      ↓
-Assembly phase: section renderers (knowns, unknowns, previous, performed) pull from ctx.rows (which has projected body) and render tags
-      ↓
-Model sees the assembled <knowns>, <previous>, etc. sections in the system prompt
-```
-
-### Fidelity Values (from code)
-
-- **full**: fully shown
-- **summary**: "compact" shown — but WHAT "compact" means varies per plugin
-- **archive**: excluded by `v_model_context` SQL before reaching any renderer (clean)
-
-## Three Breaks in the Intended Contract
-
-### Break 1 — Plugins disagree on what summary means
-
-Every plugin that registers view hooks decides what body to project per fidelity. Observed:
-
-| Plugin | full() | summary() |
-|--------|--------|-----------|
-| known | `# known ${path}\n${body}` | **same as full** (wrong) |
-| prompt | `body` | **500-char truncation + marker** (correct) |
-| budget | `body` | `body` (ok — budget is naturally short) |
-| skill | `body` | `body` (inherited default) |
-| unknown | varies — needs audit | needs audit |
-| others | needs audit | needs audit |
-
-The `known` plugin's `summary()` returning the full body is a direct contract violation. The summary view should return a compact representation of the entry, not the same full body.
-
-### Break 2 — Renderers re-apply fidelity logic
-
-Two renderers currently re-check entry fidelity and override the plugin's projection:
-
-**`known.js` `renderKnownTag`** (lines 111-115):
-```js
-if (entry.fidelity === "archive") return "";
-if (entry.fidelity === "summary") {
-    return `<${tag} path="${entry.path}"...${summary}${fidelity}${tokens}${flag}/>`;
-}
-return `<${tag} path="${entry.path}"...${summary}${fidelity}${tokens}${flag}>${entry.body}</${tag}>`;
-```
-
-This ignores entry.body at summary fidelity and renders self-closing. It's a workaround for known.summary() returning the wrong content. Belt over broken suspenders.
-
-**`previous.js` `renderToolTag`** (my edit this session):
-```js
-if (entry.fidelity === "full") {
-    return `<${entry.scheme} ${attrs}>${body}</${entry.scheme}>`;
-}
-// summary: self-closing with summary attr
-```
-
-I added this fidelity re-check when I should have trusted the plugin's projected body. Same mistake as known, added today.
-
-### Break 3 — Model writes scheme headers into body
-
-Every known/update/unknown entry in the DB has a body that starts with `# known known://path\n`, `# update\n`, or `# unknown\n`. The model writes this because the examples in the system prompt render tags with the body prefixed by `# ${scheme} ${path}\n`.
-
-Then the plugin's `full()` hook prepends ANOTHER `# ${scheme} ${path}\n` when projecting. Result: duplicate headers in the rendered output.
-
-Observed in turn 16 update body: `"# update\n# update\nDocuments 20-22 indexed and archived."`
-
-And in unknown paths: the slug-generation for pathless unknowns takes the body including the `# unknown\n` prefix, resulting in URL-encoded paths like:
-```
-unknown://%23%20unknown%0ADocument%2023%20is%20missing%20from%20the%20prompt.
-```
-
-## The Intended Contract
-
-Based on the user's stated philosophy ("surface problems, don't solve them; plugin decides, renderer renders"):
-
-### Layer 1 — Plugin decides per fidelity
-
-Each plugin registers view hooks that return the body content for each fidelity value:
-
-```js
-core.hooks.tools.onView("known", (entry) => entry.body, "full");
-core.hooks.tools.onView("known", (entry) => "", "summary");
-```
-
-At archive, no view hook is called (v_model_context excludes them).
-
-### Layer 2 — Renderer shows the projected body
-
-Renderers take the projected body from `ctx.rows[].body`:
-- If non-empty, wrap in tag with body
-- If empty, render self-closing tag
-
-Renderers do NOT re-check entry.fidelity. They trust the plugin's projection.
-
-### Layer 3 — Tag attributes always present
-
-Tag attributes visible in both full and summary rendering:
-- `path` — always
-- `summary` — if present in entry.attributes.summary
-- `turn` — if source_turn is set
-- `status` — if status is set
-- `fidelity` — always (the value itself)
-- `tokens` — always (full-cost value, unchanged by fidelity per `set_fidelity` SQL)
-
-### Per-plugin view decisions (revised)
-
-| Plugin | Category | Full body | Summary body | Notes |
-|--------|----------|-----------|--------------|-------|
-| known | data | `entry.body` (no `# known` prefix) | `""` | Tag's summary attr carries the keywords |
-| unknown | unknown | `entry.body` | `""` | Same pattern as known/skill — summary attr carries the label |
-| prompt | prompt | `entry.body` | 500-char truncation with `[truncated...]` | Current behavior is correct |
-| budget | logging | `entry.body` | `entry.body` | Feedback signal — always full |
-| update | logging | `entry.body` | `entry.body` | Already 80-char capped |
-| summarize | logging | `entry.body` | `entry.body` | Already 80-char capped |
-| get | logging | result body | `""` | Just the action tag at summary |
-| set, rm, cp, mv | logging | result body | `""` | Just the action tag at summary |
-| env, sh | logging | output | `""` | Just the action tag at summary |
-| search | logging | results | `""` | Just the action tag at summary |
-| skill | data | `entry.body` | `""` | Same as known |
-| file | data | `entry.body` | `""` | Same as known |
-| http, https | data | — | — | **Move to rummy.web plugin** — not in core |
-
-## The Body-Header Problem
-
-Separate from fidelity: the model writes `# scheme path` into the body because examples show that shape. Plugin view hooks then prepend another header.
-
-**Rule**: `# scheme` prefix belongs only in **logging** scheme outputs (tool execution results where the prefix identifies the log entry type). Non-logging schemes (known, unknown, prompt, data entries) should have no body prefix — tag attributes identify the entry.
-
-**What to remove**:
-- `known.js` `full()`: remove `# known ${entry.path}\n` prefix — just return `entry.body`
-- `unknown.js` `full()`: remove any `# unknown\n` prefix
-- Tooldoc examples for known/unknown that show bodies starting with `# scheme path` — remove so model stops copying
-
-**What to keep**:
-- Logging plugins (update, summarize, budget, get, set, etc.) may keep `# scheme` prefixes if present — they're describing tool execution results.
-
-## Test Plan
-
-To enforce the contract:
-
-1. **Per-plugin unit tests**: Each plugin with fidelity-sensitive views tests `full(entry)` and `summary(entry)` return the expected content.
-2. **Renderer tests**: Each section renderer (knowns, previous, performed, unknowns) tests that it trusts `entry.body` without re-checking fidelity.
-3. **Integration test**: Load a DB with entries at each fidelity, assemble context, verify:
-   - Archive entries absent from any section
-   - Summary entries visible as compact tags
-   - Full entries visible with body
-   - No double headers in bodies
-4. **Contract lint**: Grep for `entry.fidelity ===` in renderer files — should have zero matches.
-
-## Deliverable Order
-
-Before touching code, this document should be reviewed. Once aligned, the fix order would be:
-
-1. Fix plugin view hooks to return correct body per fidelity
-2. Remove fidelity re-checks from renderers
-3. Remove the `# scheme path` header prepending (plugin-side) and examples (tooldoc-side)
-4. Write tests per the plan above
-5. Regenerate a sample context packet to confirm clean output
-
-No silent interventions. No belt-and-suspenders logic. Plugin projects, renderer renders, model sees honest representation.

package/src/agent/KnownStore.js

@@ -1,337 +0,0 @@
-import slugify from "../sql/functions/slugify.js";
-
-export default class KnownStore {
-	#db;
-	#onChanged;
-	#schemes = new Map();
-	#seq = 0;
-	#pendingResolutions = new Map();
-
-	constructor(db, { onChanged } = {}) {
-		this.#db = db;
-		this.#onChanged = onChanged || null;
-	}
-
-	async loadSchemes(db) {
-		const rows = await (db || this.#db).get_all_schemes.all();
-		this.#schemes.clear();
-		for (const row of rows) {
-			this.#schemes.set(row.name, row);
-		}
-	}
-
-	#emitChanged(runId, path, changeType) {
-		if (this.#onChanged) this.#onChanged({ runId, path, changeType });
-	}
-
-	static scheme(path) {
-		const idx = path.indexOf("://");
-		return idx > 0 ? path.slice(0, idx) : null;
-	}
-
-	static normalizePath(path) {
-		if (!path?.includes("://")) return path;
-		const sep = path.indexOf("://");
-		const scheme = path.slice(0, sep).toLowerCase();
-		const rest = path.slice(sep + 3);
-		try {
-			// Decode first (idempotent), then encode — but preserve slashes
-			const decoded = decodeURIComponent(rest);
-			return `${scheme}://${decoded.split("/").map(encodeURIComponent).join("/")}`;
-		} catch {
-			return `${scheme}://${rest.split("/").map(encodeURIComponent).join("/")}`;
-		}
-	}
-
-	async nextTurn(runId) {
-		const row = await this.#db.next_turn.get({ run_id: runId });
-		return row.turn;
-	}
-
-	async dedup(runId, scheme, target, turn) {
-		const encodedTarget = encodeURIComponent(target);
-		const turnPrefix = turn ? `turn_${turn}/` : "";
-		const candidate = `${scheme}://${turnPrefix}${encodedTarget}`;
-		const existing = await this.#db.get_entry_body.get({
-			run_id: runId,
-			path: candidate,
-		});
-		if (!existing) return candidate;
-		return `${candidate}_${++this.#seq}`;
-	}
-
-	async slugPath(runId, scheme, content, summary) {
-		const source = summary || content || "";
-		const base = slugify(source);
-		const prefix = `${scheme}://`;
-
-		if (!base) return `${prefix}${++this.#seq}`;
-
-		const candidate = `${prefix}${base}`;
-		const existing = await this.#db.get_entry_body.get({
-			run_id: runId,
-			path: candidate,
-		});
-		if (!existing) return candidate;
-
-		return `${prefix}${base}_${++this.#seq}`;
-	}
-
-	async upsert(
-		runId,
-		turn,
-		path,
-		body,
-		status,
-		{
-			fidelity = "promoted",
-			attributes = null,
-			hash = null,
-			updatedAt = null,
-			loopId = null,
-		} = {},
-	) {
-		const normalized = KnownStore.normalizePath(path);
-		await this.#db.upsert_known_entry.run({
-			run_id: runId,
-			loop_id: loopId,
-			turn,
-			path: normalized,
-			body,
-			status,
-			fidelity,
-			hash,
-			attributes: attributes ? JSON.stringify(attributes) : null,
-			updated_at: updatedAt,
-		});
-		this.#emitChanged(runId, normalized, "upsert");
-	}
-
-	async promote(runId, path, turn) {
-		const normalized = KnownStore.normalizePath(path);
-		await this.#db.promote_path.run({
-			run_id: runId,
-			path: normalized,
-			turn,
-		});
-		this.#emitChanged(runId, normalized, "promote");
-	}
-
-	async setFileFidelity(runId, pattern, fidelity) {
-		const result = await this.#db.set_file_fidelity.run({
-			run_id: runId,
-			pattern,
-			fidelity,
-		});
-		if (result.changes === 0) {
-			await this.upsert(runId, 0, pattern, "", 200, { fidelity });
-		}
-		this.#emitChanged(runId, pattern, "fidelity");
-	}
-
-	async setFidelity(runId, path, fidelity) {
-		const normalized = KnownStore.normalizePath(path);
-		await this.#db.set_fidelity.run({
-			run_id: runId,
-			path: normalized,
-			fidelity,
-		});
-		this.#emitChanged(runId, normalized, "fidelity");
-	}
-
-	async demote(runId, path) {
-		const normalized = KnownStore.normalizePath(path);
-		await this.#db.demote_path.run({
-			run_id: runId,
-			path: normalized,
-		});
-		this.#emitChanged(runId, normalized, "demote");
-	}
-
-	async remove(runId, path) {
-		const normalized = KnownStore.normalizePath(path);
-		await this.#db.delete_known_entry.run({
-			run_id: runId,
-			path: normalized,
-		});
-		this.#emitChanged(runId, normalized, "remove");
-	}
-
-	async removeFilesByPattern(runId, pattern) {
-		await this.#db.delete_file_entries_by_pattern.run({
-			run_id: runId,
-			pattern,
-		});
-		this.#emitChanged(runId, pattern, "remove");
-	}
-
-	static #bodyPattern(body) {
-		return body || null;
-	}
-
-	async promoteByPattern(runId, path, body, turn) {
-		await this.#db.promote_by_pattern.run({
-			run_id: runId,
-			path,
-			body: KnownStore.#bodyPattern(body),
-			turn,
-		});
-		this.#emitChanged(runId, path, "promote");
-	}
-
-	async demoteByPattern(runId, path, body) {
-		await this.#db.demote_by_pattern.run({
-			run_id: runId,
-			path,
-			body: KnownStore.#bodyPattern(body),
-		});
-		this.#emitChanged(runId, path, "demote");
-	}
-
-	async getEntriesByPattern(runId, path, body, { limit, offset } = {}) {
-		return this.#db.get_entries_by_pattern.all({
-			run_id: runId,
-			path,
-			body: KnownStore.#bodyPattern(body),
-			limit: limit ?? null,
-			offset: offset ?? null,
-		});
-	}
-
-	async deleteByPattern(runId, path, body) {
-		await this.#db.delete_entries_by_pattern.run({
-			run_id: runId,
-			path,
-			body: KnownStore.#bodyPattern(body),
-		});
-		this.#emitChanged(runId, path, "remove");
-	}
-
-	async updateBodyByPattern(runId, path, body, newBody) {
-		await this.#db.update_body_by_pattern.run({
-			run_id: runId,
-			path,
-			body: KnownStore.#bodyPattern(body),
-			new_body: newBody,
-		});
-		this.#emitChanged(runId, path, "body");
-	}
-
-	async resolve(runId, path, status, body) {
-		const normalized = KnownStore.normalizePath(path);
-		await this.#db.resolve_known_entry.run({
-			run_id: runId,
-			path: normalized,
-			status,
-			body,
-		});
-		this.#emitChanged(runId, normalized, "resolve");
-		const key = `${runId}:${normalized}`;
-		const resolver = this.#pendingResolutions.get(key);
-		if (resolver) {
-			this.#pendingResolutions.delete(key);
-			resolver();
-		}
-	}
-
-	waitForResolution(runId, path) {
-		const normalized = KnownStore.normalizePath(path);
-		const key = `${runId}:${normalized}`;
-		return new Promise((resolve) => {
-			this.#pendingResolutions.set(key, resolve);
-		});
-	}
-
-	async restoreSummarizedPrompts(runId) {
-		await this.#db.restore_summarized_prompts.run({ run_id: runId });
-		this.#emitChanged(runId, "prompt://batch", "fidelity");
-	}
-
-	async getLog(runId) {
-		return this.#db.get_results.all({ run_id: runId });
-	}
-
-	async getEntries(runId) {
-		return this.#db.get_known_entries.all({ run_id: runId });
-	}
-
-	async getFileEntries(runId) {
-		return this.#db.get_file_entries.all({ run_id: runId });
-	}
-
-	async getFileStatesByPattern(runId, pattern) {
-		return this.#db.get_file_states_by_pattern.all({ run_id: runId, pattern });
-	}
-
-	async hasRejections(runId, loopId) {
-		const row = await this.#db.has_rejections.get({
-			run_id: runId,
-			loop_id: loopId,
-		});
-		return row.count > 0;
-	}
-
-	async hasAcceptedActions(runId) {
-		const row = await this.#db.has_accepted_actions.get({ run_id: runId });
-		return row.count > 0;
-	}
-
-	async getUnresolved(runId) {
-		return this.#db.get_unresolved.all({ run_id: runId });
-	}
-
-	async countUnknowns(runId) {
-		const row = await this.#db.count_unknowns.get({ run_id: runId });
-		return row.count;
-	}
-
-	async getUnknownValues(runId) {
-		const rows = await this.#db.get_unknown_values.all({ run_id: runId });
-		return new Set(rows.map((r) => r.body));
-	}
-
-	async getBody(runId, path) {
-		const row = await this.#db.get_entry_body.get({
-			run_id: runId,
-			path: KnownStore.normalizePath(path),
-		});
-		return row?.body ?? null;
-	}
-
-	async setAttributes(runId, path, attrs) {
-		const normalized = KnownStore.normalizePath(path);
-		await this.#db.update_entry_attributes.run({
-			run_id: runId,
-			path: normalized,
-			attributes: JSON.stringify(attrs),
-		});
-		this.#emitChanged(runId, normalized, "attributes");
-	}
-
-	async getState(runId, path) {
-		return this.#db.get_entry_state.get({
-			run_id: runId,
-			path: KnownStore.normalizePath(path),
-		});
-	}
-
-	async getAttributes(runId, path) {
-		const row = await this.#db.get_entry_attributes.get({
-			run_id: runId,
-			path: KnownStore.normalizePath(path),
-		});
-		return row?.attributes ? JSON.parse(row.attributes) : null;
-	}
-
-	async getTurnAudit(runId, turn) {
-		return this.#db.get_turn_audit.all({ run_id: runId, turn });
-	}
-
-	static toolFromPath(path) {
-		return KnownStore.scheme(path);
-	}
-
-	static isSystemPath(path) {
-		return path.includes("://");
-	}
-}