fengming 0.3.9 → 0.3.11

package/docs/concepts/context-engine.md

@@ -1,347 +0,0 @@
----
-summary: "Context engine: pluggable context assembly, compaction, and subagent lifecycle"
-read_when:
-  - You want to understand how FengMing assembles model context
-  - You are switching between the legacy engine and a plugin engine
-  - You are building a context engine plugin
-title: "Context engine"
-sidebarTitle: "Context engine"
----
-
-A **context engine** controls how FengMing builds model context for each run: which messages to include, how to summarize older history, and how to manage context across subagent boundaries.
-
-FengMing ships with a built-in `legacy` engine and uses it by default - most users never need to change this. Install and select a plugin engine only when you want different assembly, compaction, or cross-session recall behavior.
-
-## Quick start
-
-<Steps>
-  <Step title="Check which engine is active">
-    ```bash
-    fengming doctor
-    # or inspect config directly:
-    cat ~/.fengming/fengming.json | jq '.plugins.slots.contextEngine'
-    ```
-  </Step>
-  <Step title="Install a plugin engine">
-    Context engine plugins are installed like any other FengMing plugin.
-
-    <Tabs>
-      <Tab title="From npm">
-        ```bash
-        fengming plugins install @martian-engineering/lossless-claw
-        ```
-      </Tab>
-      <Tab title="From a local path">
-        ```bash
-        fengming plugins install -l ./my-context-engine
-        ```
-      </Tab>
-    </Tabs>
-
-  </Step>
-  <Step title="Enable and select the engine">
-    ```json5
-    // fengming.json
-    {
-      plugins: {
-        slots: {
-          contextEngine: "lossless-claw", // must match the plugin's registered engine id
-        },
-        entries: {
-          "lossless-claw": {
-            enabled: true,
-            // Plugin-specific config goes here (see the plugin's docs)
-          },
-        },
-      },
-    }
-    ```
-
-    Restart the gateway after installing and configuring.
-
-  </Step>
-  <Step title="Switch back to legacy (optional)">
-    Set `contextEngine` to `"legacy"` (or remove the key entirely - `"legacy"` is the default).
-  </Step>
-</Steps>
-
-## How it works
-
-Every time FengMing runs a model prompt, the context engine participates at four lifecycle points:
-
-<AccordionGroup>
-  <Accordion title="1. Ingest">
-    Called when a new message is added to the session. The engine can store or index the message in its own data store.
-  </Accordion>
-  <Accordion title="2. Assemble">
-    Called before each model run. The engine returns an ordered set of messages (and an optional `systemPromptAddition`) that fit within the token budget.
-  </Accordion>
-  <Accordion title="3. Compact">
-    Called when the context window is full, or when the user runs `/compact`. The engine summarizes older history to free space.
-  </Accordion>
-  <Accordion title="4. After turn">
-    Called after a run completes. The engine can persist state, trigger background compaction, or update indexes.
-  </Accordion>
-</AccordionGroup>
-
-For the bundled non-ACP Codex harness, FengMing applies the same lifecycle by projecting assembled context into Codex developer instructions and the current turn prompt. Codex still owns its native thread history and native compactor.
-
-### Subagent lifecycle (optional)
-
-FengMing calls two optional subagent lifecycle hooks:
-
-<ParamField path="prepareSubagentSpawn" type="method">
-  Prepare shared context state before a child run starts. The hook receives parent/child session keys, `contextMode` (`isolated` or `fork`), available transcript ids/files, and optional TTL. If it returns a rollback handle, FengMing calls it when spawn fails after preparation succeeds.
-</ParamField>
-<ParamField path="onSubagentEnded" type="method">
-  Clean up when a subagent session completes or is swept.
-</ParamField>
-
-### System prompt addition
-
-The `assemble` method can return a `systemPromptAddition` string. FengMing prepends this to the system prompt for the run. This lets engines inject dynamic recall guidance, retrieval instructions, or context-aware hints without requiring static workspace files.
-
-## The legacy engine
-
-The built-in `legacy` engine preserves FengMing's original behavior:
-
-- **Ingest**: no-op (the session manager handles message persistence directly).
-- **Assemble**: pass-through (the existing sanitize → validate → limit pipeline in the runtime handles context assembly).
-- **Compact**: delegates to the built-in summarization compaction, which creates a single summary of older messages and keeps recent messages intact.
-- **After turn**: no-op.
-
-The legacy engine does not register tools or provide a `systemPromptAddition`.
-
-When no `plugins.slots.contextEngine` is set (or it's set to `"legacy"`), this engine is used automatically.
-
-## Plugin engines
-
-A plugin can register a context engine using the plugin API:
-
-```ts
-import { buildMemorySystemPromptAddition } from "fengming/plugin-sdk/core";
-
-export default function register(api) {
-  api.registerContextEngine("my-engine", (ctx) => ({
-    info: {
-      id: "my-engine",
-      name: "My Context Engine",
-      ownsCompaction: true,
-    },
-
-    async ingest({ sessionId, message, isHeartbeat }) {
-      // Store the message in your data store
-      return { ingested: true };
-    },
-
-    async assemble({ sessionId, messages, tokenBudget, availableTools, citationsMode }) {
-      // Return messages that fit the budget
-      return {
-        messages: buildContext(messages, tokenBudget),
-        estimatedTokens: countTokens(messages),
-        systemPromptAddition: buildMemorySystemPromptAddition({
-          availableTools: availableTools ?? new Set(),
-          citationsMode,
-        }),
-      };
-    },
-
-    async compact({ sessionId, force }) {
-      // Summarize older context
-      return { ok: true, compacted: true };
-    },
-  }));
-}
-```
-
-The factory `ctx` includes optional `config`, `agentDir`, and `workspaceDir`
-values so plugins can initialize per-agent or per-workspace state before the
-first lifecycle hook runs.
-
-Then enable it in config:
-
-```json5
-{
-  plugins: {
-    slots: {
-      contextEngine: "my-engine",
-    },
-    entries: {
-      "my-engine": {
-        enabled: true,
-      },
-    },
-  },
-}
-```
-
-### The ContextEngine interface
-
-Required members:
-
-| Member             | Kind     | Purpose                                                  |
-| ------------------ | -------- | -------------------------------------------------------- |
-| `info`             | Property | Engine id, name, version, and whether it owns compaction |
-| `ingest(params)`   | Method   | Store a single message                                   |
-| `assemble(params)` | Method   | Build context for a model run (returns `AssembleResult`) |
-| `compact(params)`  | Method   | Summarize/reduce context                                 |
-
-`assemble` returns an `AssembleResult` with:
-
-<ParamField path="messages" type="Message[]" required>
-  The ordered messages to send to the model.
-</ParamField>
-<ParamField path="estimatedTokens" type="number" required>
-  The engine's estimate of total tokens in the assembled context. FengMing uses this for compaction threshold decisions and diagnostic reporting.
-</ParamField>
-<ParamField path="systemPromptAddition" type="string">
-  Prepended to the system prompt.
-</ParamField>
-<ParamField path="promptAuthority" type='"assembled" | "preassembly_may_overflow"'>
-  Controls which token estimate the runner uses for preemptive overflow
-  prechecks. Defaults to `"assembled"`, which means only the assembled
-  prompt's estimate is checked - appropriate for engines that return a
-  windowed, self-contained context. Set to `"preassembly_may_overflow"` only
-  when your assembled view can hide overflow risk in the underlying
-  transcript; the runner then takes the maximum of the assembled estimate
-  and the pre-assembly (unwindowed) session-history estimate when deciding
-  whether to preemptively compact. Either way, the messages you return are
-  still what the model sees - `promptAuthority` only affects the precheck.
-</ParamField>
-
-`compact` returns a `CompactResult`. When compaction rotates the active
-transcript, `result.sessionId` and `result.sessionFile` identify the successor
-session that the next retry or turn must use.
-
-Optional members:
-
-| Member                         | Kind   | Purpose                                                                                                         |
-| ------------------------------ | ------ | --------------------------------------------------------------------------------------------------------------- |
-| `bootstrap(params)`            | Method | Initialize engine state for a session. Called once when the engine first sees a session (e.g., import history). |
-| `ingestBatch(params)`          | Method | Ingest a completed turn as a batch. Called after a run completes, with all messages from that turn at once.     |
-| `afterTurn(params)`            | Method | Post-run lifecycle work (persist state, trigger background compaction).                                         |
-| `prepareSubagentSpawn(params)` | Method | Set up shared state for a child session before it starts.                                                       |
-| `onSubagentEnded(params)`      | Method | Clean up after a subagent ends.                                                                                 |
-| `dispose()`                    | Method | Release resources. Called during gateway shutdown or plugin reload - not per-session.                           |
-
-### Host requirements
-
-Context engines can declare host capability requirements on `info.hostRequirements`.
-FengMing checks these requirements before starting the operation and fails closed
-with a descriptive error when the selected runtime cannot satisfy them.
-
-For agent runs, declare `assemble-before-prompt` when the engine must control the
-actual model prompt through `assemble()`:
-
-```ts
-info: {
-  id: "my-context-engine",
-  name: "My Context Engine",
-  hostRequirements: {
-    "agent-run": {
-      requiredCapabilities: ["assemble-before-prompt"],
-      unsupportedMessage:
-        "Use the native Codex or FengMing embedded runtime, or select the legacy context engine.",
-    },
-  },
-}
-```
-
-Native Codex and FengMing embedded agent runs satisfy `assemble-before-prompt`.
-Generic CLI backends do not, so engines that require it are rejected before the
-CLI process starts.
-
-### Failure isolation
-
-FengMing isolates the selected plugin engine from the core reply path. If a
-non-legacy engine is missing, fails contract validation, throws during factory
-creation, or throws from a lifecycle method, FengMing quarantines that engine
-for the current Gateway process and downgrades context-engine work to the
-built-in `legacy` engine. The error is logged with the failed operation so the
-operator can repair, update, or disable the plugin without the agent going
-silent.
-
-Host requirement failures are different: when an engine declares that a runtime
-lacks a required capability, FengMing fails closed before starting the run. That
-protects engines that would corrupt state if they ran in an unsupported host.
-
-### ownsCompaction
-
-`ownsCompaction` controls whether FengMing runtime's built-in in-attempt auto-compaction stays enabled for the run:
-
-<AccordionGroup>
-  <Accordion title="ownsCompaction: true">
-    The engine owns compaction behavior. FengMing disables FengMing runtime's built-in auto-compaction for that run, and the engine's `compact()` implementation is responsible for `/compact`, overflow recovery compaction, and any proactive compaction it wants to do in `afterTurn()`. FengMing may still run the pre-prompt overflow safeguard; when it predicts the full transcript will overflow, the recovery path calls the active engine's `compact()` before submitting another prompt.
-  </Accordion>
-  <Accordion title="ownsCompaction: false or unset">
-    FengMing runtime's built-in auto-compaction may still run during prompt execution, but the active engine's `compact()` method is still called for `/compact` and overflow recovery.
-  </Accordion>
-</AccordionGroup>
-
-<Warning>
-`ownsCompaction: false` does **not** mean FengMing automatically falls back to the legacy engine's compaction path.
-</Warning>
-
-That means there are two valid plugin patterns:
-
-<Tabs>
-  <Tab title="Owning mode">
-    Implement your own compaction algorithm and set `ownsCompaction: true`.
-  </Tab>
-  <Tab title="Delegating mode">
-    Set `ownsCompaction: false` and have `compact()` call `delegateCompactionToRuntime(...)` from `fengming/plugin-sdk/core` to use FengMing's built-in compaction behavior.
-  </Tab>
-</Tabs>
-
-A no-op `compact()` is unsafe for an active non-owning engine because it disables the normal `/compact` and overflow-recovery compaction path for that engine slot.
-
-## Configuration reference
-
-```json5
-{
-  plugins: {
-    slots: {
-      // Select the active context engine. Default: "legacy".
-      // Set to a plugin id to use a plugin engine.
-      contextEngine: "legacy",
-    },
-  },
-}
-```
-
-<Note>
-The slot is exclusive at run time - only one registered context engine is resolved for a given run or compaction operation. Other enabled `kind: "context-engine"` plugins can still load and run their registration code; `plugins.slots.contextEngine` only selects which registered engine id FengMing resolves when it needs a context engine.
-</Note>
-
-<Note>
-**Plugin uninstall:** when you uninstall the plugin currently selected as `plugins.slots.contextEngine`, FengMing resets the slot back to the default (`legacy`). The same reset behavior applies to `plugins.slots.memory`. No manual config edit is required.
-</Note>
-
-## Relationship to compaction and memory
-
-<AccordionGroup>
-  <Accordion title="Compaction">
-    Compaction is one responsibility of the context engine. The legacy engine delegates to FengMing's built-in summarization. Plugin engines can implement any compaction strategy (DAG summaries, vector retrieval, etc.).
-  </Accordion>
-  <Accordion title="Memory plugins">
-    Memory plugins (`plugins.slots.memory`) are separate from context engines. Memory plugins provide search/retrieval; context engines control what the model sees. They can work together - a context engine might use memory plugin data during assembly. Plugin engines that want the active memory prompt path should prefer `buildMemorySystemPromptAddition(...)` from `fengming/plugin-sdk/core`, which converts the active memory prompt sections into a ready-to-prepend `systemPromptAddition`. If an engine needs lower-level control, it can still pull raw lines from `fengming/plugin-sdk/memory-host-core` via `buildActiveMemoryPromptSection(...)`.
-  </Accordion>
-  <Accordion title="Session pruning">
-    Trimming old tool results in-memory still runs regardless of which context engine is active.
-  </Accordion>
-</AccordionGroup>
-
-## Tips
-
-- Use `fengming doctor` to verify your engine is loading correctly.
-- If switching engines, existing sessions continue with their current history. The new engine takes over for future runs.
-- Engine errors are logged and the selected plugin engine is quarantined for the current Gateway process. FengMing falls back to `legacy` for user turns so replies can continue, but you should still repair, update, disable, or uninstall the broken plugin.
-- For development, use `fengming plugins install -l ./my-engine` to link a local plugin directory without copying.
-
-## Related
-
-- [Compaction](/concepts/compaction) - summarizing long conversations
-- [Context](/concepts/context) - how context is built for agent turns
-- [Plugin Architecture](/plugins/architecture) - registering context engine plugins
-- [Plugin manifest](/plugins/manifest) - plugin manifest fields
-- [Plugins](/tools/plugin) - plugin overview

package/docs/concepts/context.md

@@ -1,199 +0,0 @@
----
-summary: "Context: what the model sees, how it is built, and how to inspect it"
-read_when:
-  - You want to understand what "context" means in FengMing
-  - You are debugging why the model "knows" something (or forgot it)
-  - You want to reduce context overhead (/context, /status, /compact)
-title: "Context"
----
-
-"Context" is **everything FengMing sends to the model for a run**. It is bounded by the model's **context window** (token limit).
-
-Beginner mental model:
-
-- **System prompt** (FengMing-built): rules, tools, skills list, time/runtime, and injected workspace files.
-- **Conversation history**: your messages + the assistant's messages for this session.
-- **Tool calls/results + attachments**: command output, file reads, images/audio, etc.
-
-Context is _not the same thing_ as "memory": memory can be stored on disk and reloaded later; context is what's inside the model's current window.
-
-## Quick start (inspect context)
-
-- `/status` → quick "how full is my window?" view + session settings.
-- `/context list` → what's injected + rough sizes (per file + totals).
-- `/context detail` → deeper breakdown: per-file, per-tool schema sizes, per-skill entry sizes, and system prompt size.
-- `/context map` → WinDirStat-style treemap image of the current session's tracked context contributors.
-- `/usage tokens` → append per-reply usage footer to normal replies.
-- `/compact` → summarize older history into a compact entry to free window space.
-
-See also: [Slash commands](/tools/slash-commands), [Token use & costs](/reference/token-use), [Compaction](/concepts/compaction).
-
-## Example output
-
-Values vary by model, provider, tool policy, and what's in your workspace.
-
-### `/context list`
-
-```
-🧠 Context breakdown
-Workspace: <workspaceDir>
-Bootstrap max/file: 12,000 chars
-Sandbox: mode=non-main sandboxed=false
-System prompt (run): 38,412 chars (~9,603 tok) (Project Context 23,901 chars (~5,976 tok))
-
-Injected workspace files:
-- AGENTS.md: OK | raw 1,742 chars (~436 tok) | injected 1,742 chars (~436 tok)
-- SOUL.md: OK | raw 912 chars (~228 tok) | injected 912 chars (~228 tok)
-- TOOLS.md: TRUNCATED | raw 54,210 chars (~13,553 tok) | injected 20,962 chars (~5,241 tok)
-- IDENTITY.md: OK | raw 211 chars (~53 tok) | injected 211 chars (~53 tok)
-- USER.md: OK | raw 388 chars (~97 tok) | injected 388 chars (~97 tok)
-- HEARTBEAT.md: MISSING | raw 0 | injected 0
-- BOOTSTRAP.md: OK | raw 0 chars (~0 tok) | injected 0 chars (~0 tok)
-
-Skills list (system prompt text): 2,184 chars (~546 tok) (12 skills)
-Tools: read, edit, write, exec, process, browser, message, sessions_send, …
-Tool list (system prompt text): 1,032 chars (~258 tok)
-Tool schemas (JSON): 31,988 chars (~7,997 tok) (counts toward context; not shown as text)
-Tools: (same as above)
-
-Session tokens (cached): 14,250 total / ctx=32,000
-```
-
-### `/context detail`
-
-```
-🧠 Context breakdown (detailed)
-…
-Top skills (prompt entry size):
-- frontend-design: 412 chars (~103 tok)
-- oracle: 401 chars (~101 tok)
-… (+10 more skills)
-
-Top tools (schema size):
-- browser: 9,812 chars (~2,453 tok)
-- exec: 6,240 chars (~1,560 tok)
-… (+N more tools)
-```
-
-### `/context map`
-
-Sends an image generated from the latest cached run report. Before a normal message has produced a run report in the session, `/context map` returns an unavailable message instead of rendering an estimate. Rectangle area is proportional to tracked prompt characters:
-
-- injected workspace files
-- base system prompt text
-- skill prompt entries
-- tool JSON schemas
-
-`/context list`, `/context detail`, and `/context json` can still inspect an on-demand estimate when no run report is cached.
-
-## What counts toward the context window
-
-Everything the model receives counts, including:
-
-- System prompt (all sections).
-- Conversation history.
-- Tool calls + tool results.
-- Attachments/transcripts (images/audio/files).
-- Compaction summaries and pruning artifacts.
-- Provider "wrappers" or hidden headers (not visible, still counted).
-
-## How FengMing builds the system prompt
-
-The system prompt is **FengMing-owned** and rebuilt each run. It includes:
-
-- Tool list + short descriptions.
-- Skills list (metadata only; see below).
-- Workspace location.
-- Time (UTC + converted user time if configured).
-- Runtime metadata (host/OS/model/thinking).
-- Injected workspace bootstrap files under **Project Context**.
-
-Full breakdown: [System Prompt](/concepts/system-prompt).
-
-## Injected workspace files (Project Context)
-
-By default, FengMing injects a fixed set of workspace files (if present):
-
-- `AGENTS.md`
-- `SOUL.md`
-- `TOOLS.md`
-- `IDENTITY.md`
-- `USER.md`
-- `HEARTBEAT.md`
-- `BOOTSTRAP.md` (first-run only)
-
-Large files are truncated per-file using `agents.defaults.bootstrapMaxChars` (default `12000` chars). FengMing also enforces a total bootstrap injection cap across files with `agents.defaults.bootstrapTotalMaxChars` (default `60000` chars). `/context` shows **raw vs injected** sizes and whether truncation happened.
-
-When truncation occurs, the runtime can inject an in-prompt warning block under Project Context. Configure this with `agents.defaults.bootstrapPromptTruncationWarning` (`off`, `once`, `always`; default `always`).
-
-## Skills: injected vs loaded on-demand
-
-The system prompt includes a compact **skills list** (name + description + location). This list has real overhead.
-
-Skill instructions are _not_ included by default. The model is expected to `read` the skill's `SKILL.md` **only when needed**.
-
-## Tools: there are two costs
-
-Tools affect context in two ways:
-
-1. **Tool list text** in the system prompt (what you see as "Tooling").
-2. **Tool schemas** (JSON). These are sent to the model so it can call tools. They count toward context even though you don't see them as plain text.
-
-`/context detail` breaks down the biggest tool schemas so you can see what dominates.
-
-## Commands, directives, and "inline shortcuts"
-
-Slash commands are handled by the Gateway. There are a few different behaviors:
-
-- **Standalone commands**: a message that is only `/...` runs as a command.
-- **Directives**: `/think`, `/verbose`, `/trace`, `/reasoning`, `/elevated`, `/model`, `/queue` are stripped before the model sees the message.
-  - Directive-only messages persist session settings.
-  - Inline directives in a normal message act as per-message hints.
-- **Inline shortcuts** (allowlisted senders only): certain `/...` tokens inside a normal message can run immediately (example: "hey /status"), and are stripped before the model sees the remaining text.
-
-Details: [Slash commands](/tools/slash-commands).
-
-## Sessions, compaction, and pruning (what persists)
-
-What persists across messages depends on the mechanism:
-
-- **Normal history** persists in the session transcript until compacted/pruned by policy.
-- **Compaction** persists a summary into the transcript and keeps recent messages intact.
-- **Pruning** drops old tool results from the _in-memory_ prompt to free context-window space, but does not rewrite the session transcript - the full history is still inspectable on disk.
-
-Docs: [Session](/concepts/session), [Compaction](/concepts/compaction), [Session pruning](/concepts/session-pruning).
-
-By default, FengMing uses the built-in `legacy` context engine for assembly and
-compaction. If you install a plugin that provides `kind: "context-engine"` and
-select it with `plugins.slots.contextEngine`, FengMing delegates context
-assembly, `/compact`, and related subagent context lifecycle hooks to that
-engine instead. `ownsCompaction: false` does not auto-fallback to the legacy
-engine; the active engine must still implement `compact()` correctly. See
-[Context Engine](/concepts/context-engine) for the full
-pluggable interface, lifecycle hooks, and configuration.
-
-## What `/context` actually reports
-
-`/context` prefers the latest **run-built** system prompt report when available:
-
-- `System prompt (run)` = captured from the last embedded (tool-capable) run and persisted in the session store.
-- `System prompt (estimate)` = computed on the fly when no run report exists (or when running via a CLI backend that doesn't generate the report).
-
-Either way, it reports sizes and top contributors; it does **not** dump the full system prompt or tool schemas.
-
-## Related
-
-<CardGroup cols={2}>
-  <Card title="Context engine" href="/concepts/context-engine" icon="puzzle-piece">
-    Custom context injection via plugins.
-  </Card>
-  <Card title="Compaction" href="/concepts/compaction" icon="compress">
-    Summarizing long conversations to keep them inside the model window.
-  </Card>
-  <Card title="System prompt" href="/concepts/system-prompt" icon="message-lines">
-    How the system prompt is built and what it injects each turn.
-  </Card>
-  <Card title="Agent loop" href="/concepts/agent-loop" icon="arrows-rotate">
-    The full agent execution cycle from inbound message to final reply.
-  </Card>
-</CardGroup>