@possumtech/rummy 2.1.0 → 2.2.1

package/src/plugins/set/setDoc.md

@@ -1,22 +1,52 @@
-## <set path="[path/to/file]">[content or edit]</set> - Create, edit, or update a file or entry
+## <set path="{path}" tags="{topical,searchable,folksonomic,internal,tags}">[content or edit]</set> - Create, edit, or update an entry or file
 
-Example: <set path="known://project/milestones" visibility="summarized" summary="milestone,deadline,2026"/>
-<!-- Visibility control first — most unique capability of set. -->
+* The <set/> command requires HEREDOC string literal syntax
+* The <set/> command's SEARCH/REPLACE string literal syntax uses HEREDOC instead of git conflict markers
+* The `{SEARCH|REPLACE|NEW|APPEND|PREPEND|DELETE} Operative Labels determine the type of edit
 
-Example: <set path="src/app.js">
-<<<<<<< SEARCH
-old text
-=======
-new text
->>>>>>> REPLACE
-</set>
-<!-- SEARCH/REPLACE block — primary edit pattern for existing files. -->
+YOU MAY add additional characters to the Operative Labels to avoid collisions
 
-Example: <set path="src/config.js">s/port = 3000/port = 8080/g;s/We're almost done/We're done./g;</set>
-<!-- Sed syntax: chained s/old/new/ patterns with semicolons. -->
+Example:
+	<set path="src/main.go" tags="go,source,unlinted"><<SEARCH
+	exact
+	text
+	to be
+	replaced
+	SEARCH<<REPLACE
+	new
+	replacement
+	text
+	REPLACE</set>
+<!-- SEARCH/REPLACE: surgical edit, fuzzy on whitespace. Multiple pairs in one body apply in order. -->
 
-Example: <set path="example.md">Full file content here</set>
-<!-- Create: body contents are entire file. -->
+Example:
+	<set path="src/main.go"><<NEW
+	package main
+	
+	func main() {}
+	NEW</set>
+<!-- NEW: create with body content. -->
 
-YOU MUST NOT use <sh></sh> or <env></env> to list, create, read, or edit files — use <get></get> and <set></set>
-<!-- Reinforces at the decision point — model reading setDoc for file ops sees the prohibition here, not just buried in shDoc/envDoc which it may not be reading. -->
+Example:
+	<set path="known://plan" tags="plan,project,todo"><<APPEND
+	- [ ] new task
+	APPEND</set>
+<!-- APPEND adds to the end; PREPEND to the start. -->
+
+Example:
+	<set path="known://plan" tags="docs"><<PREPEND0
+	Documenting the <<PREPEND label	
+	PREPEND0</set>
+<!-- APPEND adds to the end; PREPEND to the start. -->
+
+Example:
+	<set path="src/main.go"><<DELETE
+	deprecated_function()
+	DELETE</set>
+<!-- DELETE: remove a literal-matching region. -->
+
+Example:
+	<set path="docs/guide.md" tags="docs"><<GUIDE
+	The pair is <<SEARCH ... SEARCH<<REPLACE ... REPLACE.
+	GUIDE</set>
+<!-- Any IDENT brackets opaque body. Use a custom IDENT (GUIDE, EOF, DOC, file paths, etc.) for bodies that contain `<<` literally. -->

package/src/plugins/sh/README.md

@@ -7,7 +7,7 @@ after the proposal is accepted.
 ## Registration
 
 - **Tool**: `sh`
-- **Scheme**: `sh` — `category: "data"` (channels only; see below)
+- **Scheme**: `sh` — `category: "logging"` (channels are time-indexed activity, not state)
 - **Handler**: Upserts the proposal entry at status 202 (proposed). The
   client must approve execution.
 
@@ -22,10 +22,11 @@ record, one data payload:
   and finalized by `stream/completed` with exit code + duration. Renders
   inside the `<log>` block as `<sh>`.
 - **Data channels**: `sh://turn_N/{slug}_1` (stdout), `sh://turn_N/{slug}_2`
-  (stderr) — scheme=`sh`, category=`data`. Created at status=102 on
-  proposal acceptance, grow via the `stream` RPC, transition to 200/500
-  via `stream/completed`. Render inside `<visible>` as `<sh>` when
-  promoted; listed in `<summarized>` otherwise.
+  (stderr) — scheme=`sh`, category=`logging` (time-indexed activity).
+  Created at status=102 on proposal acceptance, grow via the `stream`
+  RPC, transition to 200/500 via `stream/completed`. Render inside
+  `<log>` adjacent to their parent `<sh>` action entry; visibility
+  controls whether the body is full or compact, not which block.
 
 The `sh` scheme exists **only** for the data channels. The proposal/log
 entry itself is in the unified `log://` namespace along with every

package/src/plugins/sh/sh.js

@@ -8,8 +8,11 @@ export default class Sh {
 
 	constructor(core) {
 		this.#core = core;
-		// data scheme = streamed stdout/stderr; audit lives in log://. SPEC #streaming_entries.
-		core.registerScheme({ category: "data" });
+		// Streaming stdout/stderr is time-indexed activity output, not
+		// topic-indexed state — category="logging" so it renders in <log>
+		// adjacent to its action entry, not in <summary>/<visible> next
+		// to knowns and files. SPEC #streaming_entries.
+		core.registerScheme({ category: "logging" });
 		core.on("handler", this.handler.bind(this));
 		core.on("visible", this.full.bind(this));
 		core.on("summarized", this.summary.bind(this));
@@ -25,7 +28,7 @@ export default class Sh {
 		if (m?.[1] !== "sh") return;
 		let command = "";
 		if (ctx.attrs?.command) command = ctx.attrs.command;
-		else if (ctx.attrs?.summary) command = ctx.attrs.summary;
+		else if (ctx.attrs?.tags) command = ctx.attrs.tags;
 		const turn = (await ctx.db.get_run_by_id.get({ id: ctx.runId })).next_turn;
 		const dataBase = logPathToDataBase(ctx.path);
 		for (const ch of [1, 2]) {
@@ -36,7 +39,7 @@ export default class Sh {
 				body: "",
 				state: "streaming",
 				visibility: "summarized",
-				attributes: { command, summary: command, channel: ch },
+				attributes: { command, tags: command, channel: ch },
 			});
 		}
 		await ctx.entries.set({
@@ -56,7 +59,7 @@ export default class Sh {
 			path: entry.resultPath,
 			body: "",
 			state: "proposed",
-			attributes: { ...entry.attributes, summary: entry.attributes.command },
+			attributes: { ...entry.attributes, tags: entry.attributes.command },
 			loopId,
 		});
 	}

package/src/plugins/sh/shDoc.md

@@ -1,13 +1,12 @@
 ## <sh>[command]</sh> - Run a shell command with side effects
 
-Example: <sh>npm install express</sh>
-<!-- Package install. Real side-effect command. -->
-
-Example: <sh>npm test</sh>
-<!-- Test execution. Another common side-effect action. -->
+Example:
+	<sh><<EOF
+	npm install express
+	npm test 2>&1 | tee npm.log
+	EOF</sh>
+Example: <get path="sh://turn_N/*" line="-50"/>
+<!-- Heredoc body is opaque — embed multi-line scripts, redirects, and special characters without escaping. Output is addressable: every <sh> result lives at sh://turn_N/<slug>. Slice with line/limit instead of re-running. -->
 
 YOU MUST NOT use <sh></sh> to read, create, or edit files — use <get></get> and <set></set>
-<!-- Forces file operations through the entry system. -->
-
 YOU MUST use <env></env> for commands without side effects
-<!-- Reinforces the env/sh split. Read = env, mutate = sh. -->

package/src/plugins/skill/README.md

@@ -1,23 +1,46 @@
 # skill {#skill_plugin}
 
-Runtime skill management. A skill is a markdown file that gets
-attached to a run as a `skill://<name>` entry. Models see skills
-like any other entry.
+Drop-in deep skills: a single markdown file, a folder of markdown files,
+or a `.zip` archive — local or URL — archived under `skill://<name>/...`
+for the run.
 
 ## Files
 
-- **skill.js** — RPC registration and skill file loading.
+- **skill.js** — `<skill>` tag handler + `skill://` scheme.
+- **skillDoc.md** — model-facing tooldoc.
 
-## Registration
+## Tag
 
-- **Scheme**: `skill` (category: `data`)
-- **Projections**: visible → body; summarized → empty.
+`<skill path="[path-or-url]"/>`
 
-## RPC Methods
+- Single `.md` file → archived at `skill://<basename>` (summarized).
+- Folder → walk `*.md`; index file (`index.md`) → `skill://<foldername>`
+  (summarized); rest → `skill://<foldername>/<relpath-without-.md>`
+  (archived). `index.md` segments collapse: `foo/index.md` becomes
+  `skill://<foldername>/foo`.
+- `.zip` → unpack `*.md`; same layout as folder. Top-level archive
+  folder is stripped (`example/index.md` inside `example.zip` ↦
+  `skill://example`).
+- URL → fetch. `.zip` extension or `Content-Type: application/zip`
+  triggers zip unpack; otherwise treated as a single markdown file.
 
-| Method | Params | Notes |
-|--------|--------|-------|
-| `skill/add` | `{ run, name }` | Load `${RUMMY_HOME}/skills/<name>.md` and attach it to the run as `skill://<name>`. |
-| `skill/remove` | `{ run, name }` | Remove the skill entry from the run. |
-| `getSkills` | `{ run }` | Skills active on a run. |
-| `listSkills` | — | Available skill files on disk. |
+Relative paths resolve against the project root. Absolute paths used
+as-is.
+
+## Authoring
+
+Skill files reference each other with absolute `skill://...` URIs:
+`[next](skill://playbook/next)`. No relative-link rewriting at archive
+time — the contract is explicit so navigation works the same regardless
+of how the skill was packaged.
+
+## Visibility
+
+- Index page → `summarized` (model sees a header in summary; pulls
+  full body via `<get>`).
+- All other pages → `archived` (out of context until promoted).
+
+## Re-emit
+
+Re-emitting `<skill path="..."/>` overwrites prior entries — source may
+have changed mid-run.

package/src/plugins/skill/skill.js

@@ -1,130 +1,229 @@
-import fs from "node:fs/promises";
-import { join } from "node:path";
+import {
+	mkdtemp,
+	readdir,
+	readFile,
+	rm,
+	stat,
+	writeFile,
+} from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { basename, extname, isAbsolute, join, relative } from "node:path";
+import { open as openZip } from "yauzl-promise";
+import docs from "./skillDoc.js";
 
 export default class Skill {
 	#core;
 
 	constructor(core) {
 		this.#core = core;
-		core.registerScheme({
-			name: "skill",
-			category: "data",
-		});
+		core.registerScheme({ name: "skill", category: "data" });
 		core.hooks.tools.onView("skill", (entry) => entry.body, "visible");
 		core.hooks.tools.onView("skill", () => "", "summarized");
 
-		const r = core.hooks.rpc.registry;
-
-		r.register("skill/add", {
-			handler: async (params, ctx) => {
-				if (!params.name) throw new Error("name is required");
-				if (!params.run) throw new Error("run is required");
-
-				const runRow = await ctx.db.get_run_by_alias.get({ alias: params.run });
-				if (!runRow) throw new Error(`Run not found: ${params.run}`);
+		core.on("handler", this.handler.bind(this));
+		core.filter("instructions.toolDocs", async (docsMap) => {
+			docsMap.skill = docs;
+			return docsMap;
+		});
+	}
 
-				const body = await loadFile("skills", params.name);
-				const store = ctx.projectAgent.entries;
+	async handler(entry, rummy) {
+		const {
+			entries: store,
+			sequence: turn,
+			runId,
+			loopId,
+			db,
+			projectId,
+		} = rummy;
+		const path = entry.attributes.path;
+		if (!path) {
+			await store.set({
+				runId,
+				turn,
+				loopId,
+				path: entry.resultPath,
+				body: 'Missing required "path" on <skill>.',
+				state: "failed",
+				outcome: "validation",
+			});
+			return;
+		}
+
+		const projectRoot = await projectRootFor(db, projectId);
+		let resolved;
+		try {
+			resolved = await resolveSource(path, projectRoot);
+		} catch (err) {
+			await store.set({
+				runId,
+				turn,
+				loopId,
+				path: entry.resultPath,
+				body: err.message,
+				state: "failed",
+				outcome: "not_found",
+				attributes: { path },
+			});
+			return;
+		}
+
+		const { name, files, cleanup } = resolved;
+		try {
+			let added = 0;
+			for (const { relPath, body } of files) {
+				const isRoot = relPath === "";
+				const skillPath = isRoot
+					? `skill://${name}`
+					: `skill://${name}/${relPath}`;
 				await store.set({
-					runId: runRow.id,
-					turn: 0,
-					path: `skill://${params.name}`,
+					runId,
+					turn,
+					path: skillPath,
 					body,
 					state: "resolved",
-					attributes: {
-						name: params.name,
-						source: filePath("skills", params.name),
-					},
+					visibility: isRoot ? "summarized" : "archived",
+					attributes: { source: path },
+					loopId,
 				});
+				added += 1;
+			}
+			await store.set({
+				runId,
+				turn,
+				loopId,
+				path: entry.resultPath,
+				body: `skill '${name}' added: ${added} entr${added === 1 ? "y" : "ies"} at skill://${name}${added > 1 ? "/*" : ""}`,
+				state: "resolved",
+				attributes: { path, name, count: added },
+			});
+		} finally {
+			if (cleanup) await cleanup();
+		}
+	}
+}
 
-				return { status: "ok", skill: params.name };
-			},
-			description:
-				"Add a skill to a run. Reads from RUMMY_HOME/skills/{name}.md.",
-			params: {
-				run: "string — run alias",
-				name: "string — skill name (filename without .md)",
-			},
-			requiresInit: true,
-		});
+async function projectRootFor(db, projectId) {
+	if (!projectId) return null;
+	const project = await db.get_project_by_id.get({ id: projectId });
+	return project.project_root;
+}
 
-		r.register("skill/remove", {
-			handler: async (params, ctx) => {
-				if (!params.name) throw new Error("name is required");
-				if (!params.run) throw new Error("run is required");
-
-				const runRow = await ctx.db.get_run_by_alias.get({ alias: params.run });
-				if (!runRow) throw new Error(`Run not found: ${params.run}`);
-
-				const store = ctx.projectAgent.entries;
-				await store.rm({ runId: runRow.id, path: `skill://${params.name}` });
-
-				return { status: "ok" };
-			},
-			description: "Remove a skill from a run.",
-			params: {
-				run: "string — run alias",
-				name: "string — skill name",
-			},
-			requiresInit: true,
-		});
+function isUrl(p) {
+	return /^https?:\/\//.test(p);
+}
 
-		r.register("getSkills", {
-			handler: async (params, ctx) => {
-				if (!params.run) throw new Error("run is required");
-
-				const runRow = await ctx.db.get_run_by_alias.get({ alias: params.run });
-				if (!runRow) throw new Error(`Run not found: ${params.run}`);
-
-				const store = ctx.projectAgent.entries;
-				const entries = await store.getEntriesByPattern(
-					runRow.id,
-					"skill://*",
-					null,
-				);
-				return entries.map((e) => ({
-					name: e.path.replace("skill://", ""),
-					status: e.status,
-				}));
-			},
-			description: "List skills active on a run. Returns [{ name, status }].",
-			params: { run: "string — run alias" },
-			requiresInit: true,
-		});
+async function resolveSource(rawPath, projectRoot) {
+	if (isUrl(rawPath)) return resolveUrl(rawPath);
+	const absPath = isAbsolute(rawPath)
+		? rawPath
+		: projectRoot
+			? join(projectRoot, rawPath)
+			: rawPath;
+	const st = await stat(absPath).catch(() => null);
+	if (!st) throw new Error(`skill source not found: ${rawPath}`);
+
+	if (st.isDirectory()) {
+		const name = basename(absPath);
+		const files = await walkFolder(absPath);
+		return { name, files, cleanup: null };
+	}
+	if (extname(absPath).toLowerCase() === ".zip") {
+		const name = basename(absPath, extname(absPath));
+		const files = await extractZipToFiles(absPath);
+		return { name, files, cleanup: null };
+	}
+	const name = basename(absPath, extname(absPath));
+	const body = await readFile(absPath, "utf8");
+	return { name, files: [{ relPath: "", body }], cleanup: null };
+}
 
-		r.register("listSkills", {
-			handler: async () => listAvailable("skills"),
-			description: "List available skill files. Returns [{ name, path }].",
-			requiresInit: true,
-		});
+async function resolveUrl(url) {
+	const u = new URL(url);
+	const pathBase = basename(u.pathname);
+	const ext = extname(pathBase).toLowerCase();
+	const res = await fetch(url);
+	if (!res.ok) throw new Error(`skill fetch failed (${res.status}): ${url}`);
+	const ctype = res.headers.get("content-type");
+	const isZip = ext === ".zip" || ctype?.includes("application/zip");
+	if (isZip) {
+		const buf = Buffer.from(await res.arrayBuffer());
+		const tmp = await mkdtemp(join(tmpdir(), "rummy-skill-"));
+		const zipPath = join(tmp, "src.zip");
+		await writeFile(zipPath, buf);
+		const files = await extractZipToFiles(zipPath);
+		const name = basename(pathBase, ext);
+		return {
+			name,
+			files,
+			cleanup: () => rm(tmp, { recursive: true, force: true }),
+		};
+	}
+	const body = await res.text();
+	const name = basename(pathBase, ext);
+	return { name, files: [{ relPath: "", body }], cleanup: null };
+}
+
+async function walkFolder(root) {
+	const out = [];
+	for await (const file of walk(root)) {
+		if (extname(file).toLowerCase() !== ".md") continue;
+		const body = await readFile(file, "utf8");
+		out.push({ relPath: relPathFor(root, file), body });
+	}
+	return out;
+}
+
+async function* walk(dir) {
+	const dirents = await readdir(dir, { withFileTypes: true });
+	for (const e of dirents) {
+		const full = join(dir, e.name);
+		if (e.isDirectory()) yield* walk(full);
+		else yield full;
+	}
+}
 
-		// Persona methods extracted to persona plugin.
+async function extractZipToFiles(zipPath) {
+	const zip = await openZip(zipPath);
+	const out = [];
+	try {
+		for await (const entry of zip) {
+			if (entry.filename.endsWith("/")) continue;
+			if (extname(entry.filename).toLowerCase() !== ".md") continue;
+			const stream = await entry.openReadStream();
+			const chunks = [];
+			for await (const chunk of stream) chunks.push(chunk);
+			const body = Buffer.concat(chunks).toString("utf8");
+			const stripped = stripTopFolder(entry.filename);
+			out.push({ relPath: relPathFromArchive(stripped), body });
+		}
+	} finally {
+		await zip.close();
 	}
+	return out;
 }
 
-function configDir(subfolder) {
-	const home = process.env.RUMMY_HOME;
-	if (home) return join(home, subfolder);
-	return null;
+function stripTopFolder(p) {
+	const idx = p.indexOf("/");
+	if (idx === -1) return p;
+	return p.slice(idx + 1);
 }
 
-function filePath(subfolder, name) {
-	const dir = configDir(subfolder);
-	if (!dir) return null;
-	return join(dir, `${name}.md`);
+function relPathFor(root, full) {
+	const rel = relative(root, full).replaceAll("\\", "/");
+	return mapToSkillRel(rel);
 }
 
-async function loadFile(subfolder, name) {
-	const path = filePath(subfolder, name);
-	if (!path) throw new Error("RUMMY_HOME not configured");
-	return fs.readFile(path, "utf8");
+function relPathFromArchive(rel) {
+	return mapToSkillRel(rel);
 }
 
-async function listAvailable(subfolder) {
-	const dir = configDir(subfolder);
-	if (!dir) return [];
-	const files = await fs.readdir(dir);
-	return files
-		.filter((f) => f.endsWith(".md"))
-		.map((f) => ({ name: f.replace(".md", ""), path: join(dir, f) }));
+// "index.md"        → ""           (root)
+// "foo.md"          → "foo"
+// "foo/index.md"    → "foo"
+// "foo/bar.md"      → "foo/bar"
+function mapToSkillRel(rel) {
+	const noExt = rel.replace(/\.md$/i, "");
+	if (noExt === "index") return "";
+	return noExt.replace(/\/index$/, "");
 }

package/src/plugins/skill/skillDoc.js

@@ -0,0 +1,3 @@
+import { loadDoc } from "../helpers.js";
+
+export default loadDoc(import.meta.url, "skillDoc.md");

package/src/plugins/skill/skillDoc.md

@@ -0,0 +1,9 @@
+## <skill path="[path-or-url]"/> - Drop in a deep skill
+
+Example: <skill path="docs/refactoring.md"/>
+<!-- Single-file skill: archived at skill://refactoring (summarized). -->
+Example: <skill path="docs/playbook/"/>
+<!-- Folder skill: index.md → skill://playbook (summarized). All other *.md → skill://playbook/<relpath> (archived). Navigate via <get skill://playbook/<page>>. -->
+Example: <skill path="https://example.com/team-skill.zip"/>
+<!-- URL skill: fetch + unpack. .zip or Content-Type: application/zip → multi-file deep skill. Otherwise single file. -->
+<!-- Inside a multi-file skill, link sister pages with absolute URIs: [next](skill://playbook/next). -->

package/src/plugins/stream/README.md

@@ -14,10 +14,11 @@ A streaming action lives in **two namespaces** by design:
   client resolves. Renders inside `<log>`.
 - **Data channels** (payload): `{action}://turn_N/{slug}_1`,
   `{action}://turn_N/{slug}_2`, ... — scheme=`{action}` (sh, env, ...),
-  category=`data`. Created at status=102 on proposal acceptance. Grow
-  via `stream`; terminal via `stream/completed` / `stream/aborted` /
-  `stream/cancel`. Render inside `<visible>` (or `<summarized>` if
-  demoted).
+  category=`logging` (time-indexed activity, not topic-indexed state).
+  Created at status=102 on proposal acceptance. Grow via `stream`;
+  terminal via `stream/completed` / `stream/aborted` / `stream/cancel`.
+  Render inside `<log>` adjacent to their parent action entry; visibility
+  controls body projection (full vs compact), not section assignment.
 
 The stream RPC `path` param is always the **log-entry path** (the
 `log://...` path the client discovers via `getEntries` after a
@@ -75,8 +76,8 @@ A streaming producer plugin:
    `TurnExecutor` builds the path via `logPath`; the producer's
    `handler` just persists it).
 2. On `proposal.accepted`, derives the data base
-   (`logPathToDataBase(ctx.path)`) and creates **data entries** at
-   `{dataBase}_1`, `{dataBase}_2`, etc. at status=102, category=data,
+   (`logPathToDataBase(ctx.path)`) and creates **channel entries** at
+   `{dataBase}_1`, `{dataBase}_2`, etc. at status=102, category=logging,
    visibility=summarized, empty body. Then rewrites the log entry body
    to reference the channel paths.
 3. Client or external producer calls the `stream` RPC with chunks as