@possumtech/rummy 0.2.1

package/SPEC.md

@@ -0,0 +1,524 @@
+# RUMMY: Architecture Specification
+
+The authoritative reference for Rummy's design. The sacred prompt
+The instructions plugin (`preamble.md` + tool docs) defines
+model-facing behavior. This document defines everything else.
+
+---
+
+## 0. Design Philosophy: Events & Filters
+
+Rummy is a hooks-and-filters system. Every structural seam in the
+pipeline is a hookable checkpoint. Plugins subscribe to events
+(fire-and-forget side effects) and filters (transformation chains
+that thread a value through subscribers in priority order).
+
+**Every `<tag>` the model sees is a plugin.** The `<known>` section
+of the system message is rendered by the known plugin. The `<progress>`
+section is rendered by the progress plugin. The `<ask>` tag is rendered
+by the prompt plugin. No monolithic assembler decides what goes where.
+Each plugin filters for its own data from the shared row set, renders
+its section, and returns.
+
+**Plugins compose, they don't coordinate.** A plugin subscribes to a
+filter at a priority. It receives the accumulator value, appends its
+contribution, and returns. It doesn't know what other plugins exist.
+Priority determines ordering. Lower numbers run first.
+
+**The core is a filter chain invocation.** The TurnExecutor computes
+`loopStartTurn` (one value from one row), then calls
+`assembly.system.filter(instructions, ctx)` and
+`assembly.user.filter("", ctx)`. Everything else is plugins.
+
+---
+
+## 1. The Known Store
+
+All model-facing state lives in `known_entries`. Files, knowledge, tool
+results, skills, audit — everything is a keyed entry with a URI scheme,
+body, attributes, and state.
+
+### 1.1 Schema
+
+```sql
+known_entries (
+    id, run_id, turn, path, body, scheme, state, hash,
+    attributes, tokens, tokens_full, refs, write_count,
+    created_at, updated_at
+)
+```
+
+| Column | Purpose |
+|--------|---------|
+| `path` | Entry identity. Bare paths (`src/app.js`) or URIs (`known://auth`) |
+| `body` | Tag body text. File content, tool output, skill docs. |
+| `attributes` | Tag attributes as JSON. Handler-private workspace. `CHECK (json_valid)` |
+| `scheme` | Generated from path via `schemeOf()`. Drives dispatch and view routing |
+| `state` | Lifecycle stage. Determines model visibility |
+| `hash` | SHA-256 for file change detection |
+| `tokens` | Context cost at current state |
+| `tokens_full` | Cost of raw body at full fidelity |
+| `turn` | Freshness — when was this entry last touched |
+
+### 1.2 Schemes & States
+
+Paths use URI scheme syntax. Bare paths (no `://`) are files.
+
+**Files** (`scheme IS NULL`):
+
+| State | Model sees |
+|-------|-----------|
+| `full` | File content in code fence |
+| `index` | Path listed in File Index |
+| `stored` | Invisible, retrievable via `<get>` |
+
+**Knowledge** (`known://`, `unknown://`):
+
+| State | Model sees |
+|-------|-----------|
+| `full` | Key — value in bullet list |
+| `stored` | Key listed, no value |
+
+**Tool results** (`set://`, `sh://`, `env://`, `rm://`, `ask_user://`,
+`mv://`, `cp://`, `search://`, `get://`, `store://`):
+
+All start at `full` state when recorded. Handlers set the final state:
+`proposed`, `pass`, `rejected`, `error`, `pattern`, `read`, `stored`, `info`.
+
+**Skills** (`skill://`): `full` or `stored`. Rendered in system message.
+
+**Tools** (`tool://`): `full`, `model_visible = 0`. Internal plugin metadata.
+
+**URLs** (`http://`, `https://`): `full`, `summary`, `stored`.
+
+**Structural** (`summarize://`, `update://`): Status signals.
+
+**Audit** (`system://`, `prompt://`, `ask://`, `act://`, `progress://`,
+`reasoning://`, `model://`, `error://`, `user://`, `assistant://`,
+`content://`): `info` state, `model_visible = 0` (hidden from model).
+
+### 1.3 State Validation
+
+The `schemes` table is a bootstrap registry — 30 rows of static config.
+INSERT/UPDATE triggers validate state against `schemes.valid_states`.
+Plugins cannot bypass this (circular dependency prevents schemes as entries).
+
+### 1.4 UPSERT Semantics
+
+INSERT OR REPLACE on `(run_id, path)`. Each write increments `write_count`.
+Blank body is valid. Deletion uses `<rm>`, which removes the row entirely.
+
+---
+
+## 2. Relational Tables
+
+The K/V store is the memory. Relational tables are the skeleton.
+
+```sql
+projects (id, name UNIQUE, project_root, config_path, created_at)
+models   (id, alias UNIQUE, actual, context_length, created_at)
+runs     (id, project_id, parent_run_id, model, alias UNIQUE, status,
+          temperature, persona, context_limit, next_turn, created_at)
+turns    (id, run_id, sequence, prompt_tokens, completion_tokens,
+          total_tokens, cost, created_at)
+
+file_constraints (id, project_id, pattern, visibility, created_at)
+prompt_queue     (id, run_id, mode, model, prompt, config, status, result)
+rpc_log          (id, project_id, method, rpc_id, params, result, error)
+```
+
+**No sessions.** Runs belong to projects. Any client that knows the project
+name can access any run. Temperature, persona, and context_limit are per-run.
+
+**Models** are bootstrapped from `RUMMY_MODEL_*` env vars at startup (upsert).
+Clients can add/remove models at runtime via RPC. No default model — the
+client picks for every run.
+
+### 2.1 Run State Machine
+
+```
+queued → running → proposed → running → completed
+                → completed
+                → failed → running
+                → aborted → running
+```
+
+All terminal states allow transition back to `running`. Runs are long-lived.
+
+### 2.2 Prompt Queue
+
+All prompts flow through `prompt_queue`. FIFO per run. One active at a time.
+Abort stops the current prompt; pending prompts survive.
+
+---
+
+## 3. Entry-Driven Dispatch
+
+### 3.1 Unified API
+
+Three callers, one interface. Each tier is a superset of the one below.
+
+| Tier | Transport | Invocation shape |
+|------|-----------|-----------------|
+| Model | XML tags | `{ name: "rm", path: "file.txt" }` |
+| Client | JSON-RPC | `{ method: "rm", params: { path: "file.txt" } }` |
+| Plugin | PluginContext | `rummy.rm({ path: "file.txt" })` |
+
+`name` (model) = `method` (client) = method name (plugin). The params
+object is the same shape at every tier.
+
+| Method | Model | Client | Plugin |
+|--------|-------|--------|--------|
+| `get`, `set`, `rm`, `mv`, `cp`, `sh`, `env`, `store` | ✓ | ✓ | ✓ |
+| `known`, `unknown`, `ask_user`, `summarize`, `update` | ✓ | ✓ | ✓ |
+| `ask`, `act`, `resolve`, `abort`, `startRun` | — | ✓ | ✓ |
+| `getRuns`, `getModels`, `getEntries` | — | ✓ | ✓ |
+| `on()`, `filter()`, db/store access | — | — | ✓ |
+
+Model tier restrictions enforced by mode (ask removes act-only tools).
+Client tier requires project init. Plugin tier has no restrictions.
+
+### 3.2 Dispatch Path
+
+All three tiers feed the same handler chain:
+
+```
+Model:  XmlParser → { name, path, ... } → #record() → dispatch(scheme, entry, rummy)
+Client: JSON-RPC  → { method, params }   → #record() → dispatch(scheme, entry, rummy)
+Plugin: rummy.rm({ path })               → #record() → dispatch(scheme, entry, rummy)
+```
+
+### 3.3 Plugin Convention
+
+A plugin is an instantiated class. The class name matches the file name.
+The constructor receives `core` (a PluginContext) — the plugin's
+complete interface with the system.
+
+```js
+export default class Rm {
+    #core;
+
+    constructor(core) {
+        this.#core = core;
+        core.on("handler", this.handler.bind(this));
+        core.on("full", this.full.bind(this));
+    }
+
+    async handler(entry, rummy) {
+        // rummy here is per-turn RummyContext (not the startup PluginContext)
+    }
+
+    full(entry) {
+        return `# rm ${entry.attributes.path}`;
+    }
+}
+```
+
+**Two objects:**
+- `this.#core` — PluginContext (startup). For registration: `on()`, `filter()`.
+- `rummy` argument — RummyContext (per-turn). For runtime: tool verbs, queries.
+
+**Plugin types:**
+- **Tool plugins**: register `handler` + `full`/`summary`. Model-invokable.
+- **Assembly plugins**: register `core.filter("assembly.system", ...)`. Own a packet tag.
+- **Infrastructure plugins**: register `core.on("turn", ...)`. Background work.
+
+A plugin can be multiple types. Known is a tool AND an assembly plugin.
+
+### 3.4 Mode Enforcement
+
+All tools are available by default. In ask mode, the core removes
+act-only tools (`sh`, file-scheme `set`) from the tool list. This is
+a core concern — plugins do not declare their modes.
+
+---
+
+## 4. Message Structure
+
+Two messages per turn. System = stable truth. User = active task.
+
+### 4.1 Packet Structure
+
+```
+[system]
+    [instructions]
+        [sacred_prompt/]
+        [toolDescriptions/]
+        [persona/]
+        [skills/]
+    [/instructions]
+    <knowledge>
+        ...entries sorted by fidelity (index, summary, full), then by scheme
+    </knowledge>
+    <previous>
+        (pre-loop user prompt, model responses, agent warnings, and tools used, in order)
+    </previous>
+    <unknowns></unknowns>
+[/system]
+[user]
+    <current>
+        (current loop model responses, agent warnings, and tools used, in order)
+    </current>
+    <progress>the above actions have been performed on this user prompt:</progress>
+    <ask tools="..." warn="...">user prompt</ask>
+    — OR —
+    <act tools="...">user prompt</act>
+[/user]
+```
+
+**System** contains everything the model needs to know.
+**User** contains everything the model needs to do.
+
+The `<ask>`/`<act>` tag is present on every turn — first turn and
+continuations alike. The model always sees its task. The active prompt
+is extracted from its chronological position and placed last for maximum
+recency. `<progress>` bridges the gap, narrating the causal relationship
+between `<current>` (the work) and the prompt (the cause).
+
+### 4.2 Loops, Previous, and Current
+
+A **loop** is one `ask` or `act` invocation and all its continuation
+turns until summarize, fail, or abort.
+
+**Previous** = all completed loops on this run. The user prompt, model
+responses, tool results, agent warnings — the full chronicle in order.
+Lives in the system message as established history. Omitted on the
+first turn of the first loop.
+
+**Current** = the active loop's work so far. Model responses, tool
+results, agent warnings — in order. Does NOT include the user prompt
+(one per loop, extracted to `<ask>`/`<act>`). Lives in the user
+message as immediate context. Empty on the first turn of a loop.
+
+When a new prompt arrives on an existing run, the prior loop's
+`<current>` content plus its prompt move to `<previous>`. When a loop
+continues (next turn), new results append to `<current>`.
+
+### 4.3 Key Entries
+
+| Path | Lifetime | Body | Attributes |
+|------|----------|------|-----------|
+| `instructions://system` | One per run (mutable) | Empty (projection builds from preamble + plugins) | `{ persona }` |
+| `system://N` | Audit, one per turn | Full assembled system message | — |
+| `user://N` | Audit, one per turn | Full assembled user message | — |
+| `assistant://N` | Audit, one per turn | Model's raw response | — |
+
+`instructions://system` is the only mutable entry in this group. The
+framework auto-populates `toolDescriptions` from tool registrations
+that include `docs`. The instructions projection assembles the final
+text from body + attributes.
+
+### 4.4 Materialization
+
+Each turn:
+
+1. Write `instructions://system` (empty body, attributes = { persona })
+2. Run plugin hooks (`onTurn`) — plugins modify entries before the model sees them
+3. Project `instructions://system` → instructions text
+4. Query `v_model_context` VIEW → visible entries
+5. Project each entry through its tool's `full`/`summary` projection
+6. Insert projected rows into `turn_context`
+7. Invoke `assembly.system` filter chain (instructions text as base):
+   - Known plugin (priority 100) → `<known>` section
+   - Previous plugin (priority 200) → `<previous>` section
+   - Unknown plugin (priority 300) → `<unknowns>` section
+8. Invoke `assembly.user` filter chain (empty string as base):
+   - Current plugin (priority 100) → `<current>` section
+   - Progress plugin (priority 200) → `<progress>` section
+   - Prompt plugin (priority 300) → `<ask>`/`<act>` section
+9. Store as `system://N` and `user://N` audit entries
+
+The VIEW determines visibility. State IS fidelity:
+- `full` → body visible
+- `summary` → body visible
+- `index` → path listed, no content
+- `stored` → invisible
+- `proposed` → invisible (pending client)
+- `model_visible = 0` → invisible (audit, tool, instructions)
+
+### 4.5 progress:// as Entry
+
+The continuation prompt is a `progress://N` entry. Plugins can modify its
+body before materialization.
+
+---
+
+## 5. RPC Protocol
+
+JSON-RPC 2.0 over WebSocket. `discover` returns the live catalog.
+
+### 5.1 Methods
+
+#### Protocol
+
+| Method | Params |
+|--------|--------|
+| `ping` | — |
+| `discover` | — |
+| `init` | `{ name, projectRoot, configPath? }` |
+
+#### Models
+
+| Method | Params |
+|--------|--------|
+| `getModels` | `{ limit?, offset? }` |
+| `addModel` | `{ alias, actual, contextLength? }` |
+| `removeModel` | `{ alias }` |
+
+#### Entry Operations (dispatched through handler chain)
+
+| Method | Params |
+|--------|--------|
+| `read` | `{ path, run?, persist?, readonly? }` |
+| `store` | `{ path, run?, persist?, ignore?, clear? }` |
+| `write` | `{ run, path, body?, state?, attributes? }` |
+| `delete` | `{ run, path }` |
+| `getEntries` | `{ pattern?, body?, run?, limit?, offset? }` |
+
+`persist` creates a project-level file constraint (operator privilege).
+Without `persist`, operations dispatch through the handler chain.
+
+#### Runs
+
+| Method | Params |
+|--------|--------|
+| `startRun` | `{ model, temperature?, persona?, contextLimit? }` |
+| `ask` | `{ prompt, model, run?, temperature?, persona?, contextLimit?, noContext?, fork? }` |
+| `act` | `{ prompt, model, run?, temperature?, persona?, contextLimit?, noContext?, fork? }` |
+| `run/resolve` | `{ run, resolution: { path, action, output? } }` |
+| `run/abort` | `{ run }` |
+| `run/rename` | `{ run, name }` |
+| `run/inject` | `{ run, message }` |
+| `run/config` | `{ run, temperature?, persona?, contextLimit?, model? }` |
+
+`model` is required on `ask`, `act`, and `startRun`. No default.
+
+#### Queries
+
+| Method | Params |
+|--------|--------|
+| `getRuns` | `{ limit?, offset? }` |
+| `getRun` | `{ run }` |
+
+#### Skills & Personas
+
+| Method | Params |
+|--------|--------|
+| `skill/add` | `{ run, name }` |
+| `skill/remove` | `{ run, name }` |
+| `getSkills` | `{ run }` |
+| `listSkills` | — |
+| `persona/set` | `{ run, name?, text? }` |
+| `listPersonas` | — |
+
+Skills loaded from `RUMMY_HOME/skills/{name}.md`. Personas from
+`RUMMY_HOME/personas/{name}.md`.
+
+### 5.2 Notifications
+
+| Notification | Scoped by |
+|-------------|-----------|
+| `run/state` | projectId |
+| `run/progress` | projectId |
+| `ui/render` | projectId |
+| `ui/notify` | projectId |
+
+### 5.3 Resolution
+
+| Resolution | Model signal | Outcome |
+|-----------|-------------|---------|
+| reject | any | `completed` — rejection stops the bus |
+| accept | `<update>` | `running` — model has more work |
+| accept | `<summarize>` | `completed` |
+| accept | neither | `running` — healer decides |
+| error | any | `running` — error state, model retries |
+
+---
+
+## 6. Plugin System
+
+See [PLUGINS.md](PLUGINS.md) for the full plugin development guide,
+including the RummyContext API, tool registration, handler chains,
+projections, events, filters, and hedberg pattern library.
+
+Each plugin has its own README at `src/plugins/{name}/README.md`.
+
+---
+
+## 7. Hedberg Editing Syntax
+
+The model picks its preferred edit format. The parser understands all of them:
+
+1. Git merge conflict: `<<<<<<< SEARCH ... ======= ... >>>>>>> REPLACE`
+2. Replace-only: `======= ... >>>>>>> REPLACE`
+3. Unified diff: `@@ -1,3 +1,3 @@` with `-`/`+` lines
+4. Sed syntax: `s/old/new/flags`
+5. Claude XML: `<old_text>old</old_text><new_text>new</new_text>`
+6. JSON body: `{"search": "old", "replace": "new"}` or `{search="old", replace="new"}`
+7. XML attributes: `<set search="old" replace="new"/>`
+8. Full replacement: anything else becomes the new content
+
+---
+
+## 8. Response Healing
+
+The server never throws on model output. Recovery order:
+
+1. Can we recover? Extract the data and continue.
+2. Can we warn? Log structured warnings.
+3. Did our structure cause this? Check formatting, prompts.
+4. Model drift is the LAST answer.
+
+Termination protocol:
+- `<summarize>` → run terminates
+- `<update>` → run continues
+- Both → summarize wins
+- Neither + tools → stall counter
+- Neither + plain text → healed to summarize
+- Repeated commands → loop detection
+
+---
+
+## 9. Testing
+
+| Tier | Location | LLM? |
+|------|----------|------|
+| Unit | `src/**/*.test.js` | No |
+| Integration | `test/integration/` | No |
+| Live | `test/live/` | Yes |
+| E2E | `test/e2e/` | Yes |
+
+E2E tests must NEVER mock the LLM. Environment cascade:
+`.env.example` → `.env` → `.env.test`. Always use `npm run test:*`.
+
+---
+
+## 10. SQL Functions
+
+| Function | Purpose |
+|----------|---------|
+| `schemeOf(path)` | Extract URI scheme |
+| `countTokens(text)` | Token count (tiktoken o200k_base, `ceil(len/4)` fallback) |
+| `hedmatch(pattern, string)` | Full-string pattern match (paths, equality) |
+| `hedsearch(pattern, string)` | Substring pattern search (content filtering) |
+| `hedreplace(pattern, replacement, string)` | Pattern-based replacement |
+| `slugify(text)` | URI-encoded slug, max 80 chars |
+
+See [PLUGINS.md](PLUGINS.md) for the hedberg pattern type reference.
+
+---
+
+## 11. Configuration
+
+```env
+RUMMY_HOME=~/.rummy
+RUMMY_MAX_TURNS=15
+RUMMY_MAX_STALLS=3
+RUMMY_MAX_REPETITIONS=3
+RUMMY_RETENTION_DAYS=31
+RUMMY_TEMPERATURE=0.7
+RUMMY_DEBUG=false
+```
+
+Model aliases: `RUMMY_MODEL_{alias}={provider/model}`. Seeded into
+`models` table at startup.

package/lang/en.json

@@ -0,0 +1,34 @@
+{
+	"error.session_not_found": "Session '{sessionId}' not found.",
+	"error.project_not_found": "Project '{projectId}' not found.",
+	"error.run_not_found": "Run '{runId}' not found.",
+	"error.not_initialized": "Project not initialized.",
+	"error.method_not_found": "Method '{method}' not found.",
+	"error.rpc_timeout": "RPC '{method}' timed out after {timeout}ms.",
+	"error.run_name_invalid": "Name must match [a-z_]+.",
+	"error.run_name_taken": "Name '{name}' is already taken.",
+	"error.model_alias_unknown": "Unknown model alias '{alias}'. Define RUMMY_MODEL_{alias} in your environment.",
+	"error.openai_base_url_missing": "openai/ model requested but neither OPENAI_BASE_URL nor OPENAI_API_BASE is set.",
+	"error.openrouter_api_key_missing": "OpenRouter API key is missing. Please set OPENROUTER_API_KEY in your environment.",
+	"error.openrouter_auth": "OpenRouter Authentication Error: {status}. Please check your OPENROUTER_API_KEY.",
+	"error.openrouter_api": "OpenRouter API error: {status}",
+	"error.openrouter_catalog": "OpenRouter /models failed: {status}. Check your OPENROUTER_API_KEY.",
+	"error.openrouter_model_not_found": "Model '{model}' not found in OpenRouter catalog.",
+	"error.openrouter_no_context_length": "OpenRouter reports no context_length for model '{model}'.",
+	"error.ollama_api": "Ollama API error: {status}",
+	"error.ollama_show_failed": "Ollama /api/show failed: {status}. Is Ollama running at {baseUrl}?",
+	"error.ollama_no_context_length": "Ollama /api/show returned no context_length for model '{model}'.",
+	"error.ollama_unreachable": "Ollama /api/show unreachable after 3 attempts. Is Ollama running at {baseUrl}?",
+	"error.openai_api": "OpenAI-compatible API error: {status}",
+	"error.openai_models_failed": "OpenAI-compatible /v1/models failed: {status}. Is the server running at {baseUrl}?",
+	"error.openai_no_context_length": "OpenAI-compatible /v1/models returned no context size.",
+	"error.missing_summary": "missing required summary",
+	"error.unresolved_proposed": "Blocked: run has {count} unresolved proposed entries.",
+	"error.tool_already_registered": "Tool '{name}' already registered.",
+	"error.rpc_already_registered": "RPC method '{name}' already registered.",
+	"error.resolution_invalid": "Invalid resolution action: {action}. Use 'accept' or 'reject'.",
+	"error.xai_base_url_missing": "x.ai/ model requested but XAI_BASE_URL is not set.",
+	"error.xai_api_key_missing": "x.ai/ model requested but XAI_API_KEY is not set.",
+	"error.xai_auth": "xAI Authentication Error: {status}. Please check your XAI_API_KEY.",
+	"error.xai_api": "xAI API error: {status}"
+}