agentus 0.0.0-stage → 0.2.0

package/docs/transcript-folding.md

@@ -0,0 +1,139 @@
+# The transcript: one message, one row, one bubble
+
+Why this file exists: the operator looked at the `hello` session and found **one reply rendered as five
+messages with a markdown table cut in half**, and "show earlier messages" producing a few unrelated
+fragments. Both were real, and both came from the same three assumptions being wrong at once. This is
+what they were, what was done about them, and how it is verified.
+
+## What was actually wrong
+
+Measured on the operator's own session (`hello`, 8 prompts):
+
+| Fact | Number |
+|---|---|
+| rows in `messages` for that one session | **3804** |
+| rows carrying a `messageId` (real streamed chunks) | 3702, **average 3.8 characters per row** |
+| rows with **no** `messageId` (whole messages the agent re-sent) | 57, average 341 characters |
+| rows for the reply the operator was reading | **1259** |
+| client page size when opening a session / on "load earlier" | 500 rows → ~1900 chars, then 200 rows each |
+
+So three things were true:
+
+1. **A row was a token, and paging was by row.** Opening the session fetched the newest 500 rows — a
+   third of ONE reply; each "load earlier" added another 200 rows, i.e. another arbitrary byte slice.
+   The client built bubbles from *adjacent same-kind rows*, so every page boundary became a message
+   boundary. Measured: the table sat at rows 2875–3142 and the page boundary was row **3105** — the
+   table was split exactly there, and nothing in either half was valid markdown any more.
+2. **The client merged by adjacency, not by identity**, so the same logical message could not be
+   recognised across a page boundary (or across a reconnect replay).
+3. **The agent re-sends text it has already sent** (a recap of a message, sometimes with different
+   whitespace, sometimes as one anonymous block). Persisted naively, the same answer landed in the
+   transcript up to six times, each copy its own bubble.
+
+## What the two sibling projects do
+
+- **hermes-studio** (`packages/server/src/services/agent-runner/coding-agent-run-manager.ts`,
+  `run-chat/response-stream.ts`): the streaming accumulator is `appendedTextDelta(existing, next)` — if
+  the incoming frame starts with what we hold it is a snapshot (take the tail), otherwise strip the
+  longest ≥16-character overlap, otherwise it is a genuine delta. That is the re-send problem solved on
+  the server, which is where the row is written.
+- **AionUi** (`packages/desktop/src/renderer/pages/conversation/Messages/hooks.ts`): each message has a
+  stable key (`type:msg_id`, namespaced so a thought block and a text block sharing one id cannot
+  collide), a `msgIdIndex` map so a frame finds **its own** entry instead of "the last one", a
+  `replace` flag for snapshot content, and `mergeLoadedPageWithCurrent` / `prependHistoryMessages`
+  dedupe a page against what is on screen **by that key** — so a message straddling a page boundary
+  stays one entry. Their comment is explicit that a tool/thinking interruption is allowed to keep the
+  message boundary.
+
+## What Agentus does now
+
+**1. The store writes one row per message** (`Store.appendTextChunk`, `store/store.ts`):
+
+- chunks are folded into the row that already holds this `messageId` (`messages.block_key`, indexed);
+- `appendedTextDelta` decides what is new, so a snapshot adds only its tail and an exact re-send adds
+  nothing;
+- an **id-less** frame is folded into a recent same-kind row when `reemissionTarget` says it is the
+  same block: the normalized text must be equal, or contained with the shorter covering ≥60% of the
+  longer (either direction, so "the same block, extended" appends). Below 24 characters it is never
+  folded — "好的" twice is two messages, and a short string is contained in almost every long block;
+- the surviving row keeps its original `seq` (paging anchors stay valid, nothing is renumbered).
+
+**2. Paging is bounded by rows AND bytes** (`messagesBefore`): 120 rows, 256 KB, always at least one row.
+A page can no longer be "a third of a message", and one 30 KB message cannot overflow a phone fetch.
+
+**3. The client folds by key** (`packages/web/src/transcript.ts`, used by `state.ts`):
+
+- a live frame carries `delta` (wire addition on `{ t: "message" }`) and appends to the block with that
+  key — even if a tool card arrived in between, because one `messageId` is one message;
+- a **read** (history, replay) carries no `delta` and *replaces* that block's text with the row's, so
+  re-reading can never duplicate;
+- `loadEarlier` folds the older page into a scratch list first and then merges **by key** into what is on
+  screen (longer text wins), then re-indexes;
+- a frame with no key falls back to the old adjacent-merge, which is correct for the agent's anonymous
+  recaps.
+
+## The rule that was missing: a growing row's version is its LENGTH, not its seq
+
+The first cut of all this shipped a bug, and it is worth writing down because the same trap is one line
+away in any streaming UI: **folding a message into one row means that row keeps its seq.** The client
+had a guard from before — "skip a frame whose seq I have already seen" — which was correct when every
+chunk was its own row, and silently wrong the moment chunks started folding. Every chunk after the
+first was dropped. On the phone the reply read `不是` (literally the first chunk) and the next frame
+landed in an empty bubble (its first chunk had been whitespace).
+
+The fix has two halves, and both are needed:
+
+- the server sends `n`, the block's **total length after this frame**, and on a growth ships only the new
+  part (so a long reply is not re-sent whole on every token);
+- the client guards a growing frame by that length, never by `seq`: `missing = n - held.length`, and if
+  it is `≤ 0` the frame is a duplicate, otherwise it appends the missing tail (`part.slice(-missing)`,
+  which also covers a frame whose head we already hold).
+
+A read (no `delta`) needs none of this: the stored row IS the block, so it either replaces the text or
+is a no-op when it matches.
+
+`npm run live-stream` is the guard for it: it sends a real prompt through the mock backend, feeds every
+frame the server emits into the **real cockpit store** (`apply`, the same entry point the WebSocket
+calls), and asserts that the transcript built from the live frames equals the rows the store persisted —
+same bubble count, exactly the same text, no empty bubble. Reverting the guard to `seq`-only makes that
+check fail on the text comparison, which is how we know the test has teeth rather than just passing.
+
+## Where the re-send problem is *not* solved
+
+A short id-less message that the agent re-sends (measured: a 12-character "不客气，随时找我干活 👋" three
+times) still appears more than once. Folding it needs information beyond the text — a two-character
+"好的" repeated is a legitimate second message — so the guard stays at 24 characters and the leftover is
+accepted rather than guessed away.
+
+## Invariants to keep
+
+1. **One `messageId` = one row = one bubble.** If a new event kind streams text, it must go through
+   `appendTextChunk` with its id, or it re-opens the whole problem.
+2. **The row is the truth on reads; `delta` exists only so a live turn does not re-send the whole text
+   every frame.** Anything that reads the transcript (`hotwords.ts` context, the phone's notifications)
+   reads rows, so it sees accumulated text for free — it used to see 40 chunks of 3.8 characters.
+3. **Never fold on a guess.** Same id, or whitespace-insensitive equality, or ≥60% coverage. Everything
+   else stays as it is, and the migration is logged (`[store] 历史分片已折叠：3804 行 → …`) with a
+   one-time `VACUUM INTO` backup beside the DB (`agentus.sqlite.pre-fold.bak`).
+4. **A page boundary must not be able to cut a message.** Page sizes are row- and byte-bounded, and the
+   store guarantees a message is one row; if either changes, re-run the checks below.
+
+## Verify it
+
+```bash
+npm run transcript-fold   # 17 checks: folding rules on their own (page boundary, snapshot, no-id)
+npm run store-fold        # 27 checks: accumulation, re-sends, byte budget, the legacy migration
+npm run smoke             # a real (mock) turn through the live server: 2 rows, not 17
+```
+
+And against real data — the one-off used when this landed, on the operator's own session:
+
+```bash
+# rows → the CLIENT's fold, over the operator API
+curl -s -H "Authorization: Bearer $(cat "${AGENTUS_DATA:-$HOME/.agentus}/auth.token")" \
+  'http://127.0.0.1:8787/api/sessions/<id>/messages?tail=1'
+```
+
+Result for `hello`: **3804 rows → 70** (10 agent messages, 15 thought blocks, 37 tool cards, 8 prompts),
+`hasOlder=false`, both markdown tables reported complete (the 2623-character reply holds its 8-row table
+in one block), and one duplicate left — the 12-character re-send described above.

package/package.json

@@ -1,6 +1,77 @@
 {
-  "name": "agentus",
-  "version": "0.0.0-stage",
-  "stub": true,
-  "description": "Temporary package placeholder for staged publishing"
-}
+ "name": "agentus",
+ "version": "0.2.0",
+ "description": "Multi-session web cockpit for ACP-speaking coding agents (Qoder CLI / Hermes). UI has no intelligence — the agent lives in spawned CLI subprocesses.",
+ "license": "Apache-2.0",
+ "repository": {
+  "type": "git",
+  "url": "git+https://github.com/agentus-org/agentus.git"
+ },
+ "homepage": "https://github.com/agentus-org/agentus#readme",
+ "bugs": {
+  "url": "https://github.com/agentus-org/agentus/issues"
+ },
+ "bin": {
+  "agentus": "bin/agentus.mjs"
+ },
+ "files": [
+  "bin/",
+  "packages/server/src/",
+  "packages/server/mock/",
+  "packages/server/mcp/",
+  "packages/server/package.json",
+  "packages/shared/src/",
+  "packages/shared/package.json",
+  "packages/web/dist/",
+  "scripts/make-cert.sh",
+  "docs/",
+  "NOTICE"
+ ],
+ "dependencies": {
+  "@agentclientprotocol/sdk": "1.5.1",
+  "tsx": "^4.19.0",
+  "ws": "^8.22.0"
+ },
+ "workspaces": [
+  "packages/*"
+ ],
+ "scripts": {
+  "dev": "npm run dev -w @agentus/server & npm run dev -w @agentus/web & wait",
+  "dev:server": "npm run dev -w @agentus/server",
+  "dev:web": "npm run dev -w @agentus/web",
+  "build": "npm run build -ws --if-present",
+  "typecheck": "npm run typecheck -ws --if-present",
+  "lint": "npm run lint -ws --if-present",
+  "test": "npm run test -ws --if-present",
+  "start": "bash scripts/start.sh",
+  "agentus": "node bin/agentus.mjs",
+  "prepack": "npm run build -w @agentus/web",
+  "package-smoke": "node scripts/qa/package-install.mjs",
+  "smoke": "node scripts/smoke.mjs",
+  "backend-smoke": "node scripts/backend-registry-smoke.mjs",
+  "auth-smoke": "node scripts/auth-smoke.mjs",
+  "workspace-smoke": "node scripts/workspace-smoke.mjs",
+  "voice-smoke": "node scripts/voice-smoke.mjs",
+  "sentence-units": "tsx scripts/qa/sentence-units.mts",
+  "plan-service-sweep": "node scripts/qa/plan-service-sweep.mjs",
+  "plan-mcp-sweep": "node scripts/qa/plan-mcp-sweep.mjs",
+  "plan-v2-sweep": "tsx scripts/qa/plan-v2-sweep.mts",
+  "tls-smoke": "node scripts/tls-smoke.mjs",
+  "notify-testbed": "tsx scripts/notify-testbed.mts",
+  "notify-smoke": "tsx scripts/notify-smoke.mts",
+  "apk": "bash scripts/build-apk.sh",
+  "transcript-fold": "tsx scripts/qa/transcript-fold.mts",
+  "store-fold": "tsx scripts/qa/store-fold.mts",
+  "live-stream": "tsx scripts/qa/live-stream.mts",
+  "open-target": "tsx scripts/qa/open-target.mts",
+ "rail-recent": "tsx scripts/qa/rail-recent.mts",
+ "turn-pulse": "tsx scripts/qa/turn-pulse.mts",
+ "resume-tail": "tsx scripts/qa/resume-tail.mts",
+ "render-lag-sweep": "node scripts/qa/render-lag-sweep.mjs",
+ "replay-window": "node scripts/qa/replay-window.mjs",
+ "think-cadence": "node scripts/qa/think-cadence.mjs"
+ },
+ "engines": {
+  "node": ">=22.5"
+ }
+}

package/packages/server/mcp/plan-server.mjs

@@ -0,0 +1,217 @@
+#!/usr/bin/env node
+/**
+ * `agentus-plan` — the cockpit's plan tool, as a stdio MCP server.
+ *
+ * WHY THIS EXISTS
+ * The cockpit owns a session's plan (design-plan-service.md §2-3) and an agent's own todo list
+ * dies with its process. So the plan the card renders is the cockpit's OBJECT, and this tool is
+ * how the agent keeps it current: every backend is handed it at handshake time (a row can still
+ * opt back into rendering its own ACP frames — `nativePlanSource: "acp"`). An agent needs **zero
+ * adapter code**: it just has to speak MCP.
+ *
+ * HOW IT IS REACHED (and why that is safe)
+ * Nothing registers this server in an agent's own config. The cockpit only ever hands it over as
+ * part of the ACP handshake (`session/new` / `session/load` / `session/resume` → `mcpServers`),
+ * which is a per-session parameter, so an agent started any other way (CLI, cron, Studio) does
+ * not have it — see design-plan-service.md §4. The same handshake carries the three variables
+ * below, including a token minted for THAT ONE session: even someone who copies the command out
+ * of `ps` can only write the plan of the session the token was minted for.
+ *
+ * PROTOCOL NOTES
+ * Transport is newline-delimited JSON-RPC 2.0 (what the MCP stdio transport mandates). stdout
+ * carries protocol frames ONLY — anything printed there that is not a JSON-RPC message breaks the
+ * client, so all diagnostics go to stderr.
+ */
+
+const ENDPOINT = (process.env.AGENTUS_PLAN_ENDPOINT ?? "").replace(/\/+$/, "");
+const TOKEN = process.env.AGENTUS_PLAN_TOKEN ?? "";
+const SESSION = process.env.AGENTUS_PLAN_SESSION ?? "";
+
+/** MCP revision this server implements. Cheap to keep current: we only use tools/*. */
+const PROTOCOL_VERSION = "2024-11-05";
+
+const STATUSES = ["pending", "in_progress", "completed", "cancelled"];
+
+const TOOLS = [
+  {
+    name: "update_plan",
+    description:
+      "Publish this session's plan — the step list the operator watches in the Agentus cockpit, one " +
+      "progress dot per step. PREFER THIS over any checklist/todo tool of your own: this list is the " +
+      "one the cockpit DISPLAYS, and it survives your restarts, while a built-in task list dies with " +
+      "your process (the cockpit does not read it and will not show it). Call it when you START a " +
+      "multi-step task, and again each time a step's status changes — a plan that is not updated is a " +
+      "plan that lies. `items` is the WHOLE list as it stands now (a snapshot, not a delta): include " +
+      "every step, finished or not, in order. Statuses: pending, in_progress (at most one at a time), " +
+      "completed, cancelled.",
+    inputSchema: {
+      type: "object",
+      properties: {
+        items: {
+          type: "array",
+          description: "The full plan, in order. Replaces the previous list.",
+          items: {
+            type: "object",
+            properties: {
+              content: { type: "string", description: "What the step is, in the operator's language." },
+              status: { type: "string", enum: STATUSES, description: "Where this step stands now." },
+              priority: { type: "string", enum: ["high", "medium", "low"], description: "Optional." },
+            },
+            required: ["content", "status"],
+          },
+        },
+        explanation: {
+          type: "string",
+          description:
+            "Optional one-line remark shown under the list: why the plan changed, or how far the " +
+            "scope reaches. Max 1000 characters.",
+        },
+      },
+      required: ["items"],
+    },
+  },
+  {
+    name: "read_plan",
+    description:
+      "Read the plan the cockpit currently holds for this session. Use it after a restart, or when " +
+      "the operator asks what is still outstanding, instead of trusting your own memory of it.",
+    inputSchema: { type: "object", properties: {} },
+  },
+];
+
+/** POST one operation to the cockpit, which owns the plan object. */
+async function callCockpit(op, payload) {
+  if (!ENDPOINT || !TOKEN) {
+    throw new Error(
+      "this plan tool was started without AGENTUS_PLAN_ENDPOINT/AGENTUS_PLAN_TOKEN — it only works " +
+        "inside an Agentus session; keep your own todo list instead",
+    );
+  }
+  const res = await fetch(`${ENDPOINT}/api/plan/mcp`, {
+    method: "POST",
+    headers: { "content-type": "application/json", "x-agentus-plan-token": TOKEN },
+    body: JSON.stringify({ session: SESSION, op, ...payload }),
+  });
+  const body = await res.json().catch(() => null);
+  if (!res.ok) throw new Error(body?.error ?? `the cockpit answered HTTP ${res.status}`);
+  return body;
+}
+
+/** Render the plan for the agent: compact, and honest about what is not there. */
+function renderPlan(plan) {
+  if (!plan || !Array.isArray(plan.items) || plan.items.length === 0) {
+    return "This session has no plan yet. Publish one with update_plan when the work has steps.";
+  }
+  const lines = plan.items.map((i) => `- [${i.status}] ${i.content}`);
+  const head = `Plan (revision ${plan.revision ?? 0}${plan.source ? `, source ${plan.source}` : ""}${plan.terminal ? `, ${plan.terminal}` : ""}):`;
+  return [head, ...lines, plan.explanation ? `\nRemark: ${plan.explanation}` : ""].filter(Boolean).join("\n");
+}
+
+async function callTool(name, args) {
+  if (name === "read_plan") {
+    const body = await callCockpit("read", {});
+    return renderPlan(body?.plan);
+  }
+  if (name === "update_plan") {
+    const items = Array.isArray(args?.items) ? args.items : null;
+    if (!items) throw new Error("`items` must be an array of steps");
+    const body = await callCockpit("update", {
+      items,
+      ...(args?.explanation !== undefined ? { explanation: args.explanation } : {}),
+    });
+    const plan = body?.plan;
+    const done = plan?.items?.filter((i) => i.status === "completed").length ?? 0;
+    const total = plan?.items?.length ?? 0;
+    // Say something the model can act on, including the case where the cockpit IGNORED the write:
+    // a session already fed by native ACP frames has one writer, and this tool is not it. Silently
+    // succeeding there would leave the model believing it owns a list it does not.
+    if (body?.accepted === false) {
+      return `${body.reason ?? "ignored: this session's plan comes from the agent's own frames"}. The plan on screen is unchanged (${done}/${total} done).`;
+    }
+    return `Plan saved (${done}/${total} done${plan?.explanation ? `; remark kept` : ""}). It survives your restarts.`;
+  }
+  throw new Error(`unknown tool: ${name}`);
+}
+
+// ---- the stdio loop -----------------------------------------------------------------------
+
+function send(msg) {
+  process.stdout.write(`${JSON.stringify(msg)}\n`);
+}
+
+function reply(id, result) {
+  send({ jsonrpc: "2.0", id, result });
+}
+
+function replyError(id, code, message) {
+  send({ jsonrpc: "2.0", id, error: { code, message } });
+}
+
+async function handle(msg) {
+  const { id, method, params } = msg;
+  // Notifications carry no id and must never be answered.
+  const isNotification = id === undefined || id === null;
+  switch (method) {
+    case "initialize":
+      reply(id, {
+        protocolVersion: PROTOCOL_VERSION,
+        capabilities: { tools: {} },
+        serverInfo: { name: "agentus-plan", version: "1.0.0" },
+      });
+      return;
+    case "notifications/initialized":
+    case "notifications/cancelled":
+      return;
+    case "ping":
+      reply(id, {});
+      return;
+    case "tools/list":
+      reply(id, { tools: TOOLS });
+      return;
+    case "tools/call": {
+      const name = String(params?.name ?? "");
+      try {
+        const text = await callTool(name, params?.arguments ?? {});
+        reply(id, { content: [{ type: "text", text }] });
+      } catch (e) {
+        // isError (not a JSON-RPC error): the model should see the failure as tool output and
+        // react — retrying a transport failure as if it were a protocol error helps nobody.
+        reply(id, {
+          content: [{ type: "text", text: e instanceof Error ? e.message : String(e) }],
+          isError: true,
+        });
+      }
+      return;
+    }
+    default:
+      if (!isNotification) replyError(id, -32601, `method not found: ${String(method)}`);
+  }
+}
+
+let buffer = "";
+process.stdin.setEncoding("utf8");
+process.stdin.on("data", (chunk) => {
+  buffer += chunk;
+  for (;;) {
+    const nl = buffer.indexOf("\n");
+    if (nl < 0) break;
+    const line = buffer.slice(0, nl).trim();
+    buffer = buffer.slice(nl + 1);
+    if (!line) continue;
+    let msg;
+    try {
+      msg = JSON.parse(line);
+    } catch {
+      process.stderr.write(`agentus-plan: ignoring a line that is not JSON\n`);
+      continue;
+    }
+    handle(msg).catch((e) => {
+      process.stderr.write(`agentus-plan: ${e instanceof Error ? e.message : String(e)}\n`);
+      if (msg?.id !== undefined && msg?.id !== null) replyError(msg.id, -32603, "internal error");
+    });
+  }
+});
+process.stdin.on("end", () => process.exit(0));
+process.stderr.write(
+  `agentus-plan ready (session ${SESSION || "?"}, endpoint ${ENDPOINT || "unset"})\n`,
+);