durar-ai 2026.4.4

package/docs/concepts/context-engine.md

@@ -0,0 +1,274 @@
+---
+summary: "Context engine: pluggable context assembly, compaction, and subagent lifecycle"
+read_when:
+  - You want to understand how Durar assembles model context
+  - You are switching between the legacy engine and a plugin engine
+  - You are building a context engine plugin
+title: "Context Engine"
+---
+
+# Context Engine
+
+A **context engine** controls how Durar builds model context for each run.
+It decides which messages to include, how to summarize older history, and how
+to manage context across subagent boundaries.
+
+Durar ships with a built-in `legacy` engine. Plugins can register
+alternative engines that replace the active context-engine lifecycle.
+
+## Quick start
+
+Check which engine is active:
+
+```bash
+Durar doctor
+# or inspect config directly:
+cat ~/.Durar/Durar.json | jq '.plugins.slots.contextEngine'
+```
+
+### Installing a context engine plugin
+
+Context engine plugins are installed like any other Durar plugin. Install
+first, then select the engine in the slot:
+
+```bash
+# Install from npm
+Durar plugins install @martian-engineering/lossless-claw
+
+# Or install from a local path (for development)
+Durar plugins install -l ./my-context-engine
+```
+
+Then enable the plugin and select it as the active engine in your config:
+
+```json5
+// Durar.json
+{
+  plugins: {
+    slots: {
+      contextEngine: "lossless-claw", // must match the plugin's registered engine id
+    },
+    entries: {
+      "lossless-claw": {
+        enabled: true,
+        // Plugin-specific config goes here (see the plugin's docs)
+      },
+    },
+  },
+}
+```
+
+Restart the gateway after installing and configuring.
+
+To switch back to the built-in engine, set `contextEngine` to `"legacy"` (or
+remove the key entirely — `"legacy"` is the default).
+
+## How it works
+
+Every time Durar runs a model prompt, the context engine participates at
+four lifecycle points:
+
+1. **Ingest** — called when a new message is added to the session. The engine
+   can store or index the message in its own data store.
+2. **Assemble** — called before each model run. The engine returns an ordered
+   set of messages (and an optional `systemPromptAddition`) that fit within
+   the token budget.
+3. **Compact** — called when the context window is full, or when the user runs
+   `/compact`. The engine summarizes older history to free space.
+4. **After turn** — called after a run completes. The engine can persist state,
+   trigger background compaction, or update indexes.
+
+### Subagent lifecycle (optional)
+
+Durar currently calls one subagent lifecycle hook:
+
+- **onSubagentEnded** — clean up when a subagent session completes or is swept.
+
+The `prepareSubagentSpawn` hook is part of the interface for future use, but
+the runtime does not invoke it yet.
+
+### System prompt addition
+
+The `assemble` method can return a `systemPromptAddition` string. Durar
+prepends this to the system prompt for the run. This lets engines inject
+dynamic recall guidance, retrieval instructions, or context-aware hints
+without requiring static workspace files.
+
+## The legacy engine
+
+The built-in `legacy` engine preserves Durar's original behavior:
+
+- **Ingest**: no-op (the session manager handles message persistence directly).
+- **Assemble**: pass-through (the existing sanitize → validate → limit pipeline
+  in the runtime handles context assembly).
+- **Compact**: delegates to the built-in summarization compaction, which creates
+  a single summary of older messages and keeps recent messages intact.
+- **After turn**: no-op.
+
+The legacy engine does not register tools or provide a `systemPromptAddition`.
+
+When no `plugins.slots.contextEngine` is set (or it's set to `"legacy"`), this
+engine is used automatically.
+
+## Plugin engines
+
+A plugin can register a context engine using the plugin API:
+
+```ts
+export default function register(api) {
+  api.registerContextEngine("my-engine", () => ({
+    info: {
+      id: "my-engine",
+      name: "My Context Engine",
+      ownsCompaction: true,
+    },
+
+    async ingest({ sessionId, message, isHeartbeat }) {
+      // Store the message in your data store
+      return { ingested: true };
+    },
+
+    async assemble({ sessionId, messages, tokenBudget }) {
+      // Return messages that fit the budget
+      return {
+        messages: buildContext(messages, tokenBudget),
+        estimatedTokens: countTokens(messages),
+        systemPromptAddition: "Use lcm_grep to search history...",
+      };
+    },
+
+    async compact({ sessionId, force }) {
+      // Summarize older context
+      return { ok: true, compacted: true };
+    },
+  }));
+}
+```
+
+Then enable it in config:
+
+```json5
+{
+  plugins: {
+    slots: {
+      contextEngine: "my-engine",
+    },
+    entries: {
+      "my-engine": {
+        enabled: true,
+      },
+    },
+  },
+}
+```
+
+### The ContextEngine interface
+
+Required members:
+
+| Member             | Kind     | Purpose                                                  |
+| ------------------ | -------- | -------------------------------------------------------- |
+| `info`             | Property | Engine id, name, version, and whether it owns compaction |
+| `ingest(params)`   | Method   | Store a single message                                   |
+| `assemble(params)` | Method   | Build context for a model run (returns `AssembleResult`) |
+| `compact(params)`  | Method   | Summarize/reduce context                                 |
+
+`assemble` returns an `AssembleResult` with:
+
+- `messages` — the ordered messages to send to the model.
+- `estimatedTokens` (required, `number`) — the engine's estimate of total
+  tokens in the assembled context. Durar uses this for compaction threshold
+  decisions and diagnostic reporting.
+- `systemPromptAddition` (optional, `string`) — prepended to the system prompt.
+
+Optional members:
+
+| Member                         | Kind   | Purpose                                                                                                         |
+| ------------------------------ | ------ | --------------------------------------------------------------------------------------------------------------- |
+| `bootstrap(params)`            | Method | Initialize engine state for a session. Called once when the engine first sees a session (e.g., import history). |
+| `ingestBatch(params)`          | Method | Ingest a completed turn as a batch. Called after a run completes, with all messages from that turn at once.     |
+| `afterTurn(params)`            | Method | Post-run lifecycle work (persist state, trigger background compaction).                                         |
+| `prepareSubagentSpawn(params)` | Method | Set up shared state for a child session.                                                                        |
+| `onSubagentEnded(params)`      | Method | Clean up after a subagent ends.                                                                                 |
+| `dispose()`                    | Method | Release resources. Called during gateway shutdown or plugin reload — not per-session.                           |
+
+### ownsCompaction
+
+`ownsCompaction` controls whether Pi's built-in in-attempt auto-compaction stays
+enabled for the run:
+
+- `true` — the engine owns compaction behavior. Durar disables Pi's built-in
+  auto-compaction for that run, and the engine's `compact()` implementation is
+  responsible for `/compact`, overflow recovery compaction, and any proactive
+  compaction it wants to do in `afterTurn()`.
+- `false` or unset — Pi's built-in auto-compaction may still run during prompt
+  execution, but the active engine's `compact()` method is still called for
+  `/compact` and overflow recovery.
+
+`ownsCompaction: false` does **not** mean Durar automatically falls back to
+the legacy engine's compaction path.
+
+That means there are two valid plugin patterns:
+
+- **Owning mode** — implement your own compaction algorithm and set
+  `ownsCompaction: true`.
+- **Delegating mode** — set `ownsCompaction: false` and have `compact()` call
+  `delegateCompactionToRuntime(...)` from `Durar/plugin-sdk/core` to use
+  Durar's built-in compaction behavior.
+
+A no-op `compact()` is unsafe for an active non-owning engine because it
+disables the normal `/compact` and overflow-recovery compaction path for that
+engine slot.
+
+## Configuration reference
+
+```json5
+{
+  plugins: {
+    slots: {
+      // Select the active context engine. Default: "legacy".
+      // Set to a plugin id to use a plugin engine.
+      contextEngine: "legacy",
+    },
+  },
+}
+```
+
+The slot is exclusive at run time — only one registered context engine is
+resolved for a given run or compaction operation. Other enabled
+`kind: "context-engine"` plugins can still load and run their registration
+code; `plugins.slots.contextEngine` only selects which registered engine id
+Durar resolves when it needs a context engine.
+
+## Relationship to compaction and memory
+
+- **Compaction** is one responsibility of the context engine. The legacy engine
+  delegates to Durar's built-in summarization. Plugin engines can implement
+  any compaction strategy (DAG summaries, vector retrieval, etc.).
+- **Memory plugins** (`plugins.slots.memory`) are separate from context engines.
+  Memory plugins provide search/retrieval; context engines control what the
+  model sees. They can work together — a context engine might use memory
+  plugin data during assembly.
+- **Session pruning** (trimming old tool results in-memory) still runs
+  regardless of which context engine is active.
+
+## Tips
+
+- Use `Durar doctor` to verify your engine is loading correctly.
+- If switching engines, existing sessions continue with their current history.
+  The new engine takes over for future runs.
+- Engine errors are logged and surfaced in diagnostics. If a plugin engine
+  fails to register or the selected engine id cannot be resolved, Durar
+  does not fall back automatically; runs fail until you fix the plugin or
+  switch `plugins.slots.contextEngine` back to `"legacy"`.
+- For development, use `Durar plugins install -l ./my-engine` to link a
+  local plugin directory without copying.
+
+See also: [Compaction](/concepts/compaction), [Context](/concepts/context),
+[Plugins](/tools/plugin), [Plugin manifest](/plugins/manifest).
+
+## Related
+
+- [Context](/concepts/context) — how context is built for agent turns
+- [Plugin Architecture](/plugins/architecture) — registering context engine plugins
+- [Compaction](/concepts/compaction) — summarizing long conversations

package/docs/concepts/context.md

@@ -0,0 +1,179 @@
+---
+summary: "Context: what the model sees, how it is built, and how to inspect it"
+read_when:
+  - You want to understand what “context” means in Durar
+  - You are debugging why the model “knows” something (or forgot it)
+  - You want to reduce context overhead (/context, /status, /compact)
+title: "Context"
+---
+
+# Context
+
+“Context” is **everything Durar sends to the model for a run**. It is bounded by the model’s **context window** (token limit).
+
+Beginner mental model:
+
+- **System prompt** (Durar-built): rules, tools, skills list, time/runtime, and injected workspace files.
+- **Conversation history**: your messages + the assistant’s messages for this session.
+- **Tool calls/results + attachments**: command output, file reads, images/audio, etc.
+
+Context is _not the same thing_ as “memory”: memory can be stored on disk and reloaded later; context is what’s inside the model’s current window.
+
+## Quick start (inspect context)
+
+- `/status` → quick “how full is my window?” view + session settings.
+- `/context list` → what’s injected + rough sizes (per file + totals).
+- `/context detail` → deeper breakdown: per-file, per-tool schema sizes, per-skill entry sizes, and system prompt size.
+- `/usage tokens` → append per-reply usage footer to normal replies.
+- `/compact` → summarize older history into a compact entry to free window space.
+
+See also: [Slash commands](/tools/slash-commands), [Token use & costs](/reference/token-use), [Compaction](/concepts/compaction).
+
+## Example output
+
+Values vary by model, provider, tool policy, and what’s in your workspace.
+
+### `/context list`
+
+```
+🧠 Context breakdown
+Workspace: <workspaceDir>
+Bootstrap max/file: 20,000 chars
+Sandbox: mode=non-main sandboxed=false
+System prompt (run): 38,412 chars (~9,603 tok) (Project Context 23,901 chars (~5,976 tok))
+
+Injected workspace files:
+- AGENTS.md: OK | raw 1,742 chars (~436 tok) | injected 1,742 chars (~436 tok)
+- SOUL.md: OK | raw 912 chars (~228 tok) | injected 912 chars (~228 tok)
+- TOOLS.md: TRUNCATED | raw 54,210 chars (~13,553 tok) | injected 20,962 chars (~5,241 tok)
+- IDENTITY.md: OK | raw 211 chars (~53 tok) | injected 211 chars (~53 tok)
+- USER.md: OK | raw 388 chars (~97 tok) | injected 388 chars (~97 tok)
+- HEARTBEAT.md: MISSING | raw 0 | injected 0
+- BOOTSTRAP.md: OK | raw 0 chars (~0 tok) | injected 0 chars (~0 tok)
+
+Skills list (system prompt text): 2,184 chars (~546 tok) (12 skills)
+Tools: read, edit, write, exec, process, browser, message, sessions_send, …
+Tool list (system prompt text): 1,032 chars (~258 tok)
+Tool schemas (JSON): 31,988 chars (~7,997 tok) (counts toward context; not shown as text)
+Tools: (same as above)
+
+Session tokens (cached): 14,250 total / ctx=32,000
+```
+
+### `/context detail`
+
+```
+🧠 Context breakdown (detailed)
+…
+Top skills (prompt entry size):
+- frontend-design: 412 chars (~103 tok)
+- oracle: 401 chars (~101 tok)
+… (+10 more skills)
+
+Top tools (schema size):
+- browser: 9,812 chars (~2,453 tok)
+- exec: 6,240 chars (~1,560 tok)
+… (+N more tools)
+```
+
+## What counts toward the context window
+
+Everything the model receives counts, including:
+
+- System prompt (all sections).
+- Conversation history.
+- Tool calls + tool results.
+- Attachments/transcripts (images/audio/files).
+- Compaction summaries and pruning artifacts.
+- Provider “wrappers” or hidden headers (not visible, still counted).
+
+## How Durar builds the system prompt
+
+The system prompt is **Durar-owned** and rebuilt each run. It includes:
+
+- Tool list + short descriptions.
+- Skills list (metadata only; see below).
+- Workspace location.
+- Time (UTC + converted user time if configured).
+- Runtime metadata (host/OS/model/thinking).
+- Injected workspace bootstrap files under **Project Context**.
+
+Full breakdown: [System Prompt](/concepts/system-prompt).
+
+## Injected workspace files (Project Context)
+
+By default, Durar injects a fixed set of workspace files (if present):
+
+- `AGENTS.md`
+- `SOUL.md`
+- `TOOLS.md`
+- `IDENTITY.md`
+- `USER.md`
+- `HEARTBEAT.md`
+- `BOOTSTRAP.md` (first-run only)
+
+Large files are truncated per-file using `agents.defaults.bootstrapMaxChars` (default `20000` chars). Durar also enforces a total bootstrap injection cap across files with `agents.defaults.bootstrapTotalMaxChars` (default `150000` chars). `/context` shows **raw vs injected** sizes and whether truncation happened.
+
+When truncation occurs, the runtime can inject an in-prompt warning block under Project Context. Configure this with `agents.defaults.bootstrapPromptTruncationWarning` (`off`, `once`, `always`; default `once`).
+
+## Skills: injected vs loaded on-demand
+
+The system prompt includes a compact **skills list** (name + description + location). This list has real overhead.
+
+Skill instructions are _not_ included by default. The model is expected to `read` the skill’s `SKILL.md` **only when needed**.
+
+## Tools: there are two costs
+
+Tools affect context in two ways:
+
+1. **Tool list text** in the system prompt (what you see as “Tooling”).
+2. **Tool schemas** (JSON). These are sent to the model so it can call tools. They count toward context even though you don’t see them as plain text.
+
+`/context detail` breaks down the biggest tool schemas so you can see what dominates.
+
+## Commands, directives, and "inline shortcuts"
+
+Slash commands are handled by the Gateway. There are a few different behaviors:
+
+- **Standalone commands**: a message that is only `/...` runs as a command.
+- **Directives**: `/think`, `/verbose`, `/reasoning`, `/elevated`, `/model`, `/queue` are stripped before the model sees the message.
+  - Directive-only messages persist session settings.
+  - Inline directives in a normal message act as per-message hints.
+- **Inline shortcuts** (allowlisted senders only): certain `/...` tokens inside a normal message can run immediately (example: “hey /status”), and are stripped before the model sees the remaining text.
+
+Details: [Slash commands](/tools/slash-commands).
+
+## Sessions, compaction, and pruning (what persists)
+
+What persists across messages depends on the mechanism:
+
+- **Normal history** persists in the session transcript until compacted/pruned by policy.
+- **Compaction** persists a summary into the transcript and keeps recent messages intact.
+- **Pruning** removes old tool results from the _in-memory_ prompt for a run, but does not rewrite the transcript.
+
+Docs: [Session](/concepts/session), [Compaction](/concepts/compaction), [Session pruning](/concepts/session-pruning).
+
+By default, Durar uses the built-in `legacy` context engine for assembly and
+compaction. If you install a plugin that provides `kind: "context-engine"` and
+select it with `plugins.slots.contextEngine`, Durar delegates context
+assembly, `/compact`, and related subagent context lifecycle hooks to that
+engine instead. `ownsCompaction: false` does not auto-fallback to the legacy
+engine; the active engine must still implement `compact()` correctly. See
+[Context Engine](/concepts/context-engine) for the full
+pluggable interface, lifecycle hooks, and configuration.
+
+## What `/context` actually reports
+
+`/context` prefers the latest **run-built** system prompt report when available:
+
+- `System prompt (run)` = captured from the last embedded (tool-capable) run and persisted in the session store.
+- `System prompt (estimate)` = computed on the fly when no run report exists (or when running via a CLI backend that doesn’t generate the report).
+
+Either way, it reports sizes and top contributors; it does **not** dump the full system prompt or tool schemas.
+
+## Related
+
+- [Context Engine](/concepts/context-engine) — custom context injection via plugins
+- [Compaction](/concepts/compaction) — summarizing long conversations
+- [System Prompt](/concepts/system-prompt) — how the system prompt is built
+- [Agent Loop](/concepts/agent-loop) — the full agent execution cycle