@xingular/harness 0.1.7 → 1.0.0

package/docs/architecture/05-refinement.md

@@ -0,0 +1,107 @@
+# 05 Refinement
+
+## Purpose
+
+Refinement improves a variant from evidence of how it ran, with model weights fixed
+(Harness²). On one task it runs the current variant, lets improver sessions edit it, probes each
+candidate, composes the best behaviours, and picks one final output without ever seeing a score.
+A person adopts the result. Adopted variants and their lessons go to an experience library that the
+next task's improvers read.
+
+## Components
+
+| Type / function | Module | Role |
+|---|---|---|
+| `Refinements` | `refine/service.ts` | `start`, `wait`, `get`, `list`, `cancel`, `candidates`, `adopt`, `library`, `libraryEntry`, `forget`. |
+| `runRefinement` | `refine/runner.ts` | One refinement end to end; writes `summary.json`. |
+| `runMatrix` | `refine/suite.ts` | Evaluates variant versions on a task set without refining: one probe execution per version × task, recorded as `matrix.json` with comparability keys. Reads with `readMatrix`. |
+| recursion | `refine/recursion.ts` | Parallel (`agg`) and sequential (`seq`) modes; stages `INV`, `DOM`, `AGG`, `COMPOSE`. |
+| `Ledger` | `refine/ledger.ts` | Budget identity: R = 4k rollouts, P = 3k improver sessions. |
+| evidence | `refine/evidence.ts` | Builds the evidence folder and guards it against grader output. |
+| selection | `refine/selection.ts` | Blind choice among anonymised outputs, then reflection. |
+| `Library` | `refine/library.ts` | The experience library per scope and area. |
+| `PiExecutor`, `PiImprover` | `refine/pi.ts` | Executions and improvers as ordinary harness runs, triggers `probe` and `improver`. |
+| `RefineHost` | `refine/host.ts` | The application's part (page 01, surface 7). |
+| prompts | `refine/prompts.ts` | Improver missions. |
+
+## Method
+
+```text
+base variant ──► k base executions (probe runs, isolated workspaces)
+          ──► evidence folder (task, source, trajectories, deliverables, variant, library, pool)
+          ──► improvers: k INV (how the agent works) + k DOM (what a practitioner knows)
+          ──► probe each candidate ──► k compositions from behavioural contrasts ──► k finals
+          ──► blind selection among anonymised outputs ──► reflection ──► candidate for adoption
+```
+
+Parallel mode explores wider; sequential mode refines deeper. Both cost R = 4k executions and
+P = 3k improver sessions. An invalid proposal gets one retry with the validator's message
+(`STAGE_ATTEMPTS = 2`), then inherits the current variant.
+
+An **evaluation matrix** is the measurement half on its own: `runMatrix` runs every
+`{variant version × task}` pair through the same `Executor` contract — no improver, no selection,
+no adoption, and an empty or failed execution is an honest row. Every row records
+deliverability (`ok`, `delivered`) and cost, plus its comparability keys (task-prompt hash,
+executor name, package version, variant fingerprint), so matrices taken across variant versions or
+package commits are diffable directly. An evaluation never moves a head, and it inherits the
+answer-free rule: a variant or record carrying grader output is refused before any execution
+(`INV-REF-1`).
+
+## Contracts
+
+- `start({ task: { id, scope, prompt, ... }, k?, mode?: 'parallel' | 'sequential', baseVersion?, improver?, executor?, retainWorkspaces? })`
+  returns `{ id }` at once; progress arrives as `refine.changed` and `refine.event` notices.
+- With `retainWorkspaces: true` (default `false`), each execution's workspace — base, candidate
+  and final — is copied whole to `rollouts/<label>/workspace/` after its run settles and before
+  the host's `cleanup`. Binary and hidden files are kept, and symbolic links are copied as links.
+  A failed copy is recorded as `workspaceError` in `rollout.json` and never fails the execution.
+  `candidates(id)` reports each execution's `workspace` path, or `null` (PROP-0002).
+- `executor: { model?, effort?, accountId?, poolId? }` names how executions run. For each value
+  the host's `session(...)` wins, then `executor`, then the account default. `summary.json`
+  records `settings: { executor, improver }` as started; each probe run's record holds what it
+  resolved (PROP-0003).
+- `adopt(id, { label? })` commits the chosen (or named) candidate as the scope's head with source
+  `refinement`. It refuses while any plugin code in it is untrusted.
+- Executions run as `sessions.run` with `trigger: 'probe'`, `automatic: true`, in the host's
+  workspace. Improvers run with `trigger: 'improver'`, `automatic: true`, file tools only, no
+  providers, no connected apps, no harness tools, no variant, in a temporary copy of the evidence.
+
+## Invariants
+
+- **INV-REF-1** Refinement is answer-free. No rubric, score, grader output or reviewer verdict
+  reaches an improver, the selector or the library. `refine/evidence.ts` guards every evidence
+  folder (`GRADER_VERDICT_MARKERS` as a tripwire) and `assertTreeHasNoGraderVerdicts` guards the
+  library. Retained workspaces are output for measurement after the fact; evidence, selection,
+  reflection and the library never read them.
+- **INV-REF-2** The selector sees only the request, the source and the outputs, shuffled under
+  arbitrary ids. It never sees a harness, a trajectory or a verdict.
+- **INV-REF-3** The budget identity R = 4k, P = 3k holds in both modes (`Ledger.assertIdentity`).
+  A change to the method changes both modes and their tests together.
+- **INV-REF-4** Refinement never acts in the world. Executions are unattended `probe` runs in a
+  folder the `RefineHost` prepared; anything that needs a person is refused at the gate, never
+  queued.
+- **INV-REF-5** Improvers have file tools only and work on a copy of the evidence.
+- **INV-REF-6** Refinement never moves a head by itself. Only `adopt`, a person's action, does, and
+  only with all plugin code trusted (`INV-VAR-5`).
+- **INV-REF-7** The library carries whole adopted variants with an evidence-backed lesson, never a
+  summary in place of the variant.
+- **INV-REF-8** Per-observation truncation of trajectories is never below `MIN_OBS_TRUNC`
+  (10,000 characters); lower values fabricate defects.
+
+## Data
+
+```text
+<data root>/refine/<id>/      summary.json, task.json, variants/<label>/, rollouts/<label>/,
+                              sessions/<label>/, context/, selection/, reflection/, improver/
+<data root>/refine/<id>/rollouts/<label>/workspace/    only with retainWorkspaces
+<data root>/refine/<id>/      matrix.json, suite.json   (evaluation matrices; see above)
+<data root>/library/<scope>/<area>/INDEX.md
+<data root>/library/<scope>/<area>/variants/<task>/{variant/, EXPERIENCE.md}
+```
+
+## Source map
+
+`src/refine/*.ts`. Method origin: Harness² (`harness2/core/{recursion,ledger,selection,library,context}.py`,
+Apache-2.0). The evaluation matrix is this harness's addition (`refine/suite.ts`), shaped by the
+same executor contract and demo executor. Try it with no model: `npm run build && node dist/cli.js refine demo --k 2`,
+or evaluate a matrix with `node dist/cli.js refine matrix --demo [--data <root>] [--out <file.json>]`.

package/docs/architecture/06-plugins-and-capabilities.md

@@ -0,0 +1,93 @@
+# 06 Plugins and capabilities
+
+## Purpose
+
+A *plugin* is a bundle with a title, a description and contributions: instructions, skills,
+commands, tools, connections, code (a Pi extension) and themes. A Pi package is a plugin as it is;
+so are the harness's own parts, the application's parts, a person's registry, the folder every
+account shares, and what a working folder brings. This part decides what a session loads, who has
+what switched on, which code may run, and what reaches one caller. A *capability set* is the
+callable result for one run.
+
+## Components
+
+| Type / function | Module | Role |
+|---|---|---|
+| `Plugin`, `Contribution`, `PluginDescriptor`, `CONTRIBUTION_KINDS`, `PLUGIN_SOURCES` | `plugins/types.ts` | The model of a plugin and what it brings. |
+| `Plugins` (`loading`, `loaded`) | `plugins/plugins.ts` | Chooses the paths a session or terminal loads; records what each extension registered. |
+| discovery | `plugins/discover.ts` | Finds items with Pi's own resolver, which runs no extension code. |
+| attachments (`load`, `save`, `attached`, `decided`, `set`) | `plugins/attachments.ts` | The one store of what is switched on, per scope. |
+| `HARNESS_PLUGINS` | `plugins/builtin.ts` | Descriptors of the harness's own parts: `pi`, `harness`, `subagents`, `wiki`, `history`, `ask-user`, `checklist`, `connected-apps`. |
+| `FIRST_PARTY_PLUGINS` | `plugins/firstParty.ts` | `output-trim`, `budget-guard`: inline extensions shipped switched off. `lessons`: switched off; its contribution turns on reflection after a run (page 08, PROP-0008) and loads no Pi code. |
+| `Hooks` | `plugins/hooks.ts` | Runs a package's hooks file after a write. |
+| install / trust | `plugins/install.ts`, `plugins/trust.ts` | Install through Pi's package manager with scripts off; trust code by hash. |
+| inventory / api | `plugins/inventory.ts`, `plugins/api.ts` | `plugins.list`, `plugins.inventory`, attach, trust. |
+| capability api | `capabilities/api.ts`, `capabilities/types.ts` | `list`, `resolve`, `toolIntersection`, `inventory`, `attach`. |
+| run snapshot | `capabilities/runSnapshot.ts` | `freezeRun`, `loadRun`, `capabilitySet`. |
+
+## Contracts
+
+- **Loading.** `Plugins.loading({ cwd, agentDir, caller, contributions, install? })` returns
+  `extensionPaths`, `skillPaths`, `promptPaths`, `themePaths`, `builtins` (terminal only), `hooks`,
+  `found`, `detached`, `projectCode`, `packageCode`. A session opens Pi with its own discovery off
+  and only these paths; a terminal receives them as `--extension`, `--skill`,
+  `--prompt-template` and `--theme` after `--no-extensions --no-skills --no-prompt-templates
+  --no-themes`. Pi still reads `AGENTS.md` and `SYSTEM.md` itself.
+- **Attachments.** `config/plugin-attachments.json` holds `global`, `agents`, `projects` (each
+  `{ on, off }`) and `settings` per plugin. The narrowest scope that says something wins: agent,
+  then project, then everywhere. With no entry an item keeps its default.
+- **Hooks.** A package declares `"harness": { "extends": [...], "hooks": "./hooks.mjs" }`. Events:
+  `wiki.page.saved`, `history.run.ended`. A hook has 30 seconds; its errors are kept in
+  `plugin-hooks.json`. `ctx.call(connection, tool, args)` reaches only a connected app that is on
+  for everyone, and only a tool marked to run without asking.
+- **Capability set** (`CapabilitySet`): `{ version: 1, tools, connectors, contributions }`.
+  `capabilities.resolve` returns the set available to a caller; `capabilities.intersect` /
+  `toolIntersection` narrows it for a child.
+
+## Invariants
+
+- **INV-PLG-1** The harness is the only loader. A session or terminal loads exactly what
+  `Plugins.loading` chose, with Pi's discovery switched off. Never pass a session a loader that
+  discovers on its own.
+- **INV-PLG-2** Code waits for trust. A working folder's `.pi/extensions` and the packages its
+  settings name are trusted together; each installed package is trusted on its own. Trust is a
+  hash of the files; a changed file asks again. A loose file a person put in the shared
+  `extensions/` folder needs no trust. Text contributions always load.
+- **INV-PLG-3** Installing runs no code (`npm_config_ignore_scripts`). A package arrives switched
+  off and, if it has code, untrusted. Installing grants no attachment, trust, credential or
+  permission.
+- **INV-PLG-4** The approval gate is the last extension to see a tool call (page 07, `INV-GATE-2`).
+  A plugin that changes a call's input does so before the gate sees it.
+- **INV-PLG-5** Everything model-visible belongs to a plugin. A new prompt section, tool group or
+  tool is declared by a `PluginDescriptor` (`plugins/builtin.ts` for the harness's own), or
+  `plugins.inventory` reports it as `unattributed`.
+- **INV-PLG-6** One store says what is switched on: `plugins/attachments.ts`. A new kind of
+  switchable thing gets a default and is read with `attached`; no second enablement file.
+- **INV-PLG-7** A hook is told, never asked. Hooks run after the write, in the background; a write
+  never waits for a hook and a hook's failure never reaches it. A new write point that should be
+  heard calls `noteChange` (`core/changes.ts`). No hook may stop or change a write.
+- **INV-PLG-8** A hook runs only while its plugin is on for everyone and trusted, checked before
+  each delivery. A working folder's package cannot bring a hook.
+- **INV-PLG-9** Unattended runs load no extension found on disk (`INV-RUN-5`). The harness's own
+  parts are required and refuse to be switched off; switching something on grants no permission.
+- **INV-CAP-1** A capability set only narrows. `intersect` never adds access; a run's set is
+  frozen when it starts (`INV-RUN-2`).
+- **INV-CAP-2** The capability contract is generic. It names no application task, role or UI
+  concept.
+- **INV-CAP-3** A capability set says what may be called, not whether a call may run. The gate
+  still decides each call.
+
+## Data
+
+```text
+<data root>/config/plugin-attachments.json     (registry-enabled.json read until the first change)
+<data root>/plugins/<name>/                     packages kept with the data; removal moves to trash/plugins/
+<state dir>/plugin-trust.json                   trusted code, by hash
+<state dir>/plugin-registered.json              what each extension registered last time
+<state dir>/plugin-hooks.json                   hooks and their last run
+<state dir>/accounts/_shared/pi/                settings and packages shared by every account
+```
+
+## Source map
+
+`src/plugins/*.ts`, `src/capabilities/*.ts`, `src/core/changes.ts`.

package/docs/architecture/07-tools-and-gate.md

@@ -0,0 +1,88 @@
+# 07 Tools and the gate
+
+## Purpose
+
+A run's tools come from four places: the harness's own tools, the connected-apps broker,
+subagent tools, and the application's tool providers. Every call, whatever its source, passes one
+per-call gate that asks the application's policy, then standing approvals, then the person.
+
+## Components
+
+| Type / function | Module | Role |
+|---|---|---|
+| `HarnessToolDefinition`, `RECALL_TOOLS`, `TRAJECTORY_TOOLS`, `WIKI_READ_TOOLS`, `WIKI_WRITE_TOOLS`, `SKILL_TOOLS`, `DEFAULT_TOOL_NAMES` | `tools/definitions.ts` | The harness's own tools, defined once. |
+| `dispatch` | `tools/dispatch.ts` | One entry point for every History, Wiki and skill call, from a Pi tool, the server or a terminal. |
+| `harnessTools` | `tools/harnessTools.ts` | The definitions as `ToolSpec`s for one session. |
+| `askUserTool` (`ask_user`) | `tools/askUser.ts` | Stops the run and asks a question with choices. |
+| `checklistTool` (`checklist_write`) | `tools/checklist.ts` | Writes the run's whole checklist as a `checklist.updated` event (PROP-0006). Read-only for the gate, so plan mode allows it. |
+| `subagentTools` | `tools/spawnSubagent.ts` | `spawn_subagent`, `wait_subagents`. |
+| `webTools` | `web/tools.ts` | `web_search` (Brave, key from the secret store), `web_fetch` (checked public reads). An application provides them as a provider. |
+| gate | `sessions/sessions.ts::gate`, `sessions/runner.ts` | `ToolGate { ask, check }`, loaded as the last extension (`xagent-approvals`). |
+| `Approvals`, `decide`, `grant`, `approvalKey` | `sessions/approvals.ts` | Standing approvals. |
+
+Harness tools: `history_search`, `history_read`, `trajectory_search`, `trajectory_read`,
+`wiki_search`, `wiki_read`, `wiki_list`, `wiki_write` (not in the default set), `skill_load`,
+`ask_user`, `checklist_write`. Broker: `mcp_list_tools`, `mcp_call_tool`. Native Pi tools: `read`, `grep`, `find`,
+`ls`, `edit`, `write`, `bash`.
+
+## The gate, per call
+
+```text
+child run and tool outside its snapshot ─► refuse
+plan mode and tool not research-safe    ─► refuse (before approvals)
+policy.scope(call)
+  not allowed and (automatic or no approval possible) ─► refuse with reason
+policy.authorize(call) throws           ─► refuse
+allowed, not humanOnly, (automatic or not gated) ─► run
+trigger probe or improver               ─► refuse (nobody to ask)
+run.requestApproval(...)                ─► standing approvals may answer, else the person
+  denied                                ─► refuse
+policy.authorize(call) again            ─► run or refuse
+```
+
+Standing approvals (`approvals.json`): an always-approve switch per key (`chat`, or the run's
+`harness`), tools allowed across the state directory per key, and targets remembered per
+session. `humanOnly` calls are never answered by remembered tools, and by the switch only when
+`allowGlobalApproval`. `ApprovalFacts.unattended` disables standing answers; `readOnly` disables
+the switch and global tools.
+
+## Invariants
+
+- **INV-GATE-1** Every tool call of every run passes the gate. No tool source bypasses it,
+  including extension-registered tools (which ask before every call) and the initial Bash command.
+- **INV-GATE-2** The gate is the last extension loaded, after disk, variant and inline extensions, so
+  it sees the final input of every call.
+- **INV-GATE-3** Order is fixed: capability and plan-mode refusals, then the policy's scope and
+  authorize, then standing approvals, then the person, then authorize again.
+- **INV-GATE-4** Refinement runs (`probe`, `improver`) never ask a person; what would need one is
+  refused, never queued (`INV-REF-4`).
+- **INV-GATE-5** A tool is registered only if its name is in the run's frozen capability set, in
+  that order (`INV-CAP-1`).
+- **INV-GATE-6** Tool attachment bytes never enter the run-event stream; only references are
+  recorded.
+- **INV-GATE-7** Tool descriptions name the product with `{product}` (filled by `describeTool`),
+  never an application's name (`INV-BND-2`).
+
+## Data
+
+`<state dir>/approvals.json`.
+
+## Source map
+
+`src/tools/*.ts`, `src/web/*.ts`, `src/sessions/sessions.ts`, `src/sessions/runner.ts`,
+`src/sessions/approvals.ts`, `src/sessions/policy.ts`.
+
+## Session naming (PROP-0010)
+
+`session_name` (`tools/sessionName.ts`) binds to the current native session. It maintains
+only Pi session metadata and is explicitly plan-safe, while still subject to capability
+selection and the host policy. The terminal registers the same definition through its Pi
+bridge; standalone policy permits naming without approval. See page 02 for validation and
+persistence. It is a foundation tool, not mutable variant data.
+
+## Diagnostic probes
+
+`/doctor` is a user command, never a model tool. Live diagnostics create separate real runs,
+intersect current capabilities, and execute tool calls through the existing gate. Calls that
+require an additional per-call approval report blocked; batch diagnostic consent does not grant
+standing permission. No raw tool inputs, results, or screenshots enter diagnostic findings.

package/docs/architecture/08-memory-and-history.md

@@ -0,0 +1,94 @@
+# 08 Memory and history
+
+## Purpose
+
+Two kinds of memory sit outside the variant. The **Wiki** holds project notes: linked
+Markdown that people and agents read and write as reference material, never as instructions.
+**History** is the record of earlier runs: recall over what was said, and bounded reads of the
+recorded trajectory. Both are served through harness tools (page 07) and kept searchable in a
+rebuildable SQLite index.
+
+## Components
+
+| Type / function | Module | Role |
+|---|---|---|
+| `paths`, `list`, `read`, `write`, `remove`, `versions`, `version`, `search`, `graph`, `ensureLayout`, `moveLegacy` | `wiki/notes.ts` | The Wiki. Logical module (functions), not a class. |
+| `noteChange` | `core/changes.ts` | Announces a completed write to listeners (hooks). |
+| `indexRun`, `sync`, `search`, `read` | `history/recall.ts` | Recall passages from runs, scoped to the caller. |
+| trajectory search and read | `history/trajectory.ts` | Paged, redacted reads of canonical run events. |
+| `Lessons`, `keptLessons`, `lessonMission` | `history/lessons.ts`, `history/lessonPrompt.ts` | Reflection on finished runs when the `lessons` plugin is on; keeps general lessons as Wiki notes (PROP-0008). |
+| `withIndex`, `migrate`, `MIGRATIONS` | `core/index.ts` | The shared `index.sqlite` connection and its migrations. |
+
+## Contracts
+
+- **Note paths.** A note's path is its file path under `wiki/` without `.md`. Links are
+  `[[path]]`, `[[path|label]]` or a relative Markdown link. Notes live in `home.md`, `library/`
+  (no project) and `projects/<slug>/` (a project the application named in `projects.json`).
+- **Writes.** `write(root, path, body, { expectedUpdatedAt?, actor?, reason? })` refuses text over
+  400,000 characters, text that looks like a credential (`secret_refused`), an unknown project
+  slug, and a stale `expectedUpdatedAt` (`conflict`). The previous text is kept in `wiki-history/`.
+- **Recall scope.** A `Code` caller (a session in a project) reads only session runs of its project,
+  or its own session. A `Chat` caller reads its agent's conversations. `recallExcludeAgents` hides
+  named agents' conversations. A caller never reads a run outside its scope.
+- **Trajectory reads** return pages of canonical execution events, including active runs,
+  bounded and marked when truncated. They omit account identifiers, credential fields,
+  attachment bytes, native adapter payloads and provider reasoning text. They report evidence,
+  never a score.
+- **Lessons** (PROP-0008, [ADR-0010](../decisions/ADR-0010-lessons-in-the-wiki.md)). When the
+  first-party `lessons` plugin is on for a run's caller (its contribution is in the run's
+  `capabilities.json`), a **completed, top-level** run that made at least `minToolCalls` tool calls
+  is queued after it ends. Subagent runs and runs with trigger `probe`, `improver` or `reflection`
+  are never queued. One reflection runs at a time, in the background, and the queue is not kept
+  across a restart.
+  - The reflection is a session with trigger `reflection` on the run's own account and model, file
+    tools `read`, `grep`, `find`, `ls` only, no providers, connections, harness tools or subagents,
+    `automatic: true`; anything needing a person is refused at the gate. Its evidence is a temporary
+    `context/` with `run.md` (the request and the rendered trajectory, observations kept to at least
+    10,000 characters) and `lessons.md` (the kept lessons' paths and cues), checked with the grader
+    tripwire.
+  - It replies `NO_ENTRY`, `SAME <path>`, or one `<experience>` block (`cue`, `lesson`, `evidence`,
+    `confidence`): refinement's lesson shape, which never carries the run's own facts.
+  - A new lesson is written with `notes.write` to `library/lessons/<slug of cue>` with actor
+    `reflection`, frontmatter `title`, `cue`, `confidence`, `sources: [runId]`. `SAME` adds the run
+    id to that note's `sources`. Nothing is written when the lesson is longer than `maxChars`,
+    looks like a credential, carries a grader marker, or the Wiki already keeps `maxLessons`
+    lessons; nothing is evicted.
+  - The `wiki` pointer section says how many lessons there are when there are any.
+- **Index.** Every finished run is indexed when it ends; `open` and `rebuildIndex()` sync from
+  `runs/`.
+
+## Invariants
+
+- **INV-MEM-1** The Wiki is reference material, not instructions. Only the variant shapes an agent
+  (`INV-VAR-1`); the Wiki reaches a prompt only as the dynamic `wiki` pointer and selected
+  `references` (`INV-PRM-3`).
+- **INV-MEM-2** The harness is the only writer of the Wiki and of run records. An application
+  reads them and writes notes only through harness methods (`wiki.*`). Inside the harness, the
+  Wiki is written by `wiki_write`, by `wiki.*` methods, and by `Lessons` (actor `reflection`, only
+  under `library/lessons/`, only while the `lessons` plugin is on for the run's caller).
+- **INV-MEM-3** Every write point calls `noteChange` after the file is in place, and never waits
+  for listeners (`INV-PLG-7`). (Removal does not yet: drift D-7.)
+- **INV-MEM-4** A note write keeps the previous text as a version. Nothing deletes history except
+  an explicit person's action.
+- **INV-MEM-5** Files are the source of truth. `index.sqlite` is derived and can be deleted and
+  rebuilt (`harness.rebuildIndex()`).
+- **INV-MEM-6** Index migrations are append-only once released. Each runs in one transaction
+  after the file is backed up; a failure restores the previous index.
+- **INV-MEM-7** Recall and trajectory reads respect the caller's scope and redact secrets. They
+  never return another scope's runs.
+- **INV-MEM-8** Wiki code is synchronous (`INV-BND-4`).
+
+## Data
+
+```text
+<data root>/wiki/            home.md, library/, projects/<slug>/
+<data root>/wiki-history/    earlier versions of notes
+<data root>/legacy/          an earlier Wiki's memory blocks, facts, revisions, Inbox (moved once)
+<data root>/index.sqlite     derived: pages, links, runs, recall passages (FTS5)
+<data root>/projects.json    read only: the application owns it
+```
+
+## Source map
+
+`src/wiki/notes.ts`, `src/history/recall.ts`, `src/history/trajectory.ts`, `src/history/lessons.ts`, `src/core/index.ts`,
+`src/core/changes.ts`, `src/core/projects.ts`, `src/core/secrets.ts`.

package/docs/architecture/09-accounts-and-connectors.md

@@ -0,0 +1,90 @@
+# 09 Accounts and connectors
+
+## Purpose
+
+**Accounts** are isolated Pi identities: each has its own agent folder, credentials, sessions and
+history, so one person can use several providers and subscriptions without them mixing.
+**Connectors** (connected apps) are MCP servers a session reaches through a broker, with scoped
+access, approval of non-read calls, and redaction. The **Hub** installs exact catalog versions of
+skills, prompts, rules and Pi packages.
+
+## Components
+
+| Type / function | Module | Role |
+|---|---|---|
+| `Accounts` | `accounts/accounts.ts` | Accounts and pools: `resolve`, `context`, `runtime`, `models`, `status`, sign-in (`loginStart`, `loginStatus`, `loginRespond`, `loginCancel`), `setApiKey`, `signOut`, terminal accounting. |
+| `isolateProcess`, `ensureSharedDir`, `sharedDir` | `accounts/env.ts` | Isolated launch environments. |
+| account store, audit | `accounts/store.ts` | The registry and audit log. |
+| `SecretStore`, `secretKey`, `fileSecretStore`, `memorySecretStore` | `accounts/secrets.ts` | Where credentials live (page 01). |
+| `ConnectedApps` | `mcp/service.ts` | The broker: `grant`, `session`, `expose`, `tools`, `revokeRun`, `callUnattended`, traces. |
+| `Connections` | `mcp/connections.ts` | Saved connections, scopes, secrets, `redactEvent`, `scrub`. |
+| `McpManager` | `mcp/manager.ts` | MCP clients. |
+| OAuth | `mcp/oauth.ts` | Sign-in; an application's own registered clients (`AppOAuthClients`). |
+| `Hub` | `hub/hub.ts`, `hub/schema.ts` | Catalog, preview, install, uninstall, receipts. |
+
+## Contracts
+
+- **Account folder.** Every Pi session gets its account's own agent folder
+  (`PI_CODING_AGENT_DIR`) under `<state dir>/accounts/`. Settings, skills, prompts and
+  instructions are shared through links to `_shared/pi/`; credentials, sessions and history stay
+  in the account's own folder.
+- **Resolution.** A new session may name an account, a pool, or neither (the default). Once
+  started, it stays on that account.
+- **Sign-in progress** lives in the running process only. `accounts.loginStatus` recovers the
+  current code or prompt with a revision; it is never written to account files.
+- **Broker tools.** `mcp_list_tools` and `mcp_call_tool`. A connection reaches a session only if
+  its scope allows the caller and the policy's `connections` keeps it. Calls that are not reads
+  are approved through the gate. Each call's redacted result is traced in `mcp-context.json`.
+- **`apps.expose(tools)`** offers the application's own tools through the broker, which is how a
+  terminal session reaches them.
+- **Hub.** `hub.list`, `get`, `preview`, `install`, `uninstall`, `readDocument`. Pi packages need
+  an exact npm version or a full Git commit. Documents are saved as inert JSON.
+
+## Invariants
+
+- **INV-ACC-1** Accounts are isolated. A session runs in its account's own Pi folder, never in
+  `~/.pi`. The process drops inherited credential variables (`*_API_KEY`, `*_AUTH_TOKEN`,
+  `*_OAUTH_TOKEN`, `*_ACCESS_TOKEN`, `CODEX_HOME`, `CLAUDE_CONFIG_DIR`, `PI_CODING_AGENT_DIR`)
+  before any session starts.
+- **INV-ACC-2** An account is never exchanged for another. A pinned session that cannot use its
+  account fails. Changing account for a terminal is an explicit person's action that restarts
+  the child.
+- **INV-ACC-3** The harness never writes a secret to its own files; it asks the `SecretStore`.
+  The default file store is owner-only (`0600`).
+- **INV-ACC-4** A terminal's `/accounts` goes through the session socket's `accounts.*` calls, and
+  every one needs the session's control key. A command the model runs in the shell cannot make
+  an account change.
+- **INV-MCP-1** Connector secrets live in the `SecretStore`; `connections.json` holds no secret.
+  Run events and traces are redacted before they are stored or shown.
+- **INV-MCP-2** A session reaches only the connections its scope and the policy allow, and loses
+  them when its run ends (`revokeRun`).
+- **INV-MCP-3** An application's OAuth client is usable only by a catalog entry it lists, and
+  nothing of it is stored.
+- **INV-MCP-4** Hub installation grants no attachment, trust, credential or permission. Writes
+  are serialized; existing registry files are never overwritten; uninstall removes only the exact
+  receipt's resources and revokes its attachment choices.
+
+## Data
+
+```text
+<state dir>/accounts/         registry, per-account Pi agent folders, _shared/pi/, audit log
+<state dir>/secrets.json      default SecretStore (owner-only)
+<state dir>/connections.json  connected apps, no secrets
+<state dir>/hub/catalog.json  catalog cache
+<data root>/hub/              installed documents and provenance
+<data root>/runs/<id>/mcp-context.json
+```
+
+## Source map
+
+`src/accounts/*.ts`, `src/mcp/*.ts`, `src/hub/*.ts`. Decision:
+[ADR-0005](../decisions/ADR-0005-app-provided-oauth-clients.md).
+
+## Account and connector diagnostics
+
+`Accounts.inspect` reads credentials and a non-refreshing Pi model catalogue through a read-only
+credential store. It never refreshes OAuth tokens or verifies a key over the network. `/doctor`
+labels this local readiness. A selected live model probe sends one synthetic request, caps output
+at 64 tokens, and records usage through the normal run writer. It never substitutes an account.
+MCP live diagnostics connect and list tools on the selected scoped connector, without calling
+connector tools or starting sign-in. Disposable grants and clients are closed afterwards.

package/docs/architecture/10-server-protocol.md

@@ -0,0 +1,84 @@
+# 10 Server protocol
+
+## Purpose
+
+A host that is not a Node program embeds the harness as a child process and talks to it over
+newline-delimited JSON on stdin and stdout. The protocol exposes the same in-process API a Node
+application uses, run events as they are saved, state-change notices, and calls from the harness
+to the host for the extension points the host supplies.
+
+## Components
+
+| Type / function | Module | Role |
+|---|---|---|
+| `RpcPeer`, `Transport`, `stdioTransport`, `PROTOCOL_VERSION` | `server/rpc.ts` | Framing, requests, replies, host calls. |
+| `methods(harness)` | `server/methods.ts` | The method table. |
+| `serve`, `attach` | `server/serve.ts` | Registers the table, forwards events and notices. |
+| `remoteSecrets`, remote policy, directory, tools | `server/remote.ts` | Extension points supplied by the host over the protocol. |
+| CLI `serve` | `src/cli.ts` | `serve --state <dir> --data <dir> --name <product> [--builtins <dir>] [--host-secrets] [--host-policy] [--host-tools] [--host-directory]`. |
+
+## Contracts
+
+```text
+host → harness   { v: 1, id, method, params }        harness → host   { v: 1, id, result | error }
+harness → host   { v: 1, event }                     a run event, validated, after it is saved
+harness → host   { v: 1, notice }                    state changed: read it again
+harness → host   { v: 1, id, call, params }          host → harness   { v: 1, id, result | error }
+```
+
+- stdout carries only protocol lines; diagnostics go to stderr.
+- `harness.methods` lists every method. Groups in the table: `accounts`, `harnesses`, `runs`,
+  `connections`, `plugins`, `wiki`, `refine`, `capabilities`, `hub`, `terminal`, `approvals`,
+  `harness`, `subagents`, `sessions`, `library`, `recall`, `context`.
+- Host calls (only with the matching `--host-*` flag): `secrets.get|set|delete`,
+  `policy.scope|authorize|approval|connections`, `tools.call`. The host registers tools with
+  `tools.register` and names with `directory.set`, and describes its parts with
+  `plugins.describe`.
+- `refine.start` takes `{ task, k?, mode?, improver?, executor?, baseVersion?, retainWorkspaces?, id? }`;
+  `refine.candidates` returns each execution with `workspace: string | null` (page 05, PROP-0002, PROP-0003).
+- `runs.checklist { runId }` returns the run's latest checklist, `[]` when it wrote none (PROP-0006).
+- `runs.continuation { runId, sessionDir? }` returns a `Continuation`, and `sessions.run` accepts
+  `continueFrom` (page 02, PROP-0007). `harness.status` reports `recovery.interruptedRunIds`.
+- Errors carry a stable `code`, a message and an optional hint (`core/errors.ts`).
+
+## Invariants
+
+- **INV-SRV-1** `src/server/methods.ts` is the contract for every non-Node host. Adding a method
+  to an existing group is an additive change (update this page). Renaming, removing, or changing
+  the params or result shape of a method is an architecture change.
+- **INV-SRV-2** A method handler calls the same in-process API a Node application uses. The server
+  holds no logic of its own beyond parameter checking.
+- **INV-SRV-3** Events reach the host only after they are persisted and redacted (`INV-RUN-3`).
+- **INV-SRV-4** Changing the framing or `PROTOCOL_VERSION` is an architecture change.
+- **INV-SRV-5** The harness works with none of the host calls supplied (`INV-EXT-2`).
+- **INV-SRV-6** After a change to the server, the CLI or startup, `npm run prove` must pass: it
+  starts the server on a throwaway data root with no application present and exercises accounts,
+  variants, the Wiki, the prompt, connected apps, a recorded session, a plugin hook and the
+  refinement demo. The App's sidecar tests parse every reply with its schemas and must pass too.
+
+## Source map
+
+`src/server/*.ts`, `src/cli.ts`, `src/core/errors.ts`, `scripts/prove-standalone.mjs`.
+
+## Doctor jobs (PROP-0011)
+
+| Method | Input | Reply |
+|---|---|---|
+| `doctor.start` | `{ context: { caller, cwd, accountId, model, tools, variant?, capabilities? }, mode?: "local" \| "live", checks?: string[], confirmed?: boolean }` | `DoctorReport` |
+| `doctor.read` | `{ id }` | `DoctorReport` |
+| `doctor.cancel` | `{ id }` | `DoctorReport` |
+
+A trusted host attests confirmation for the exact selected live checks. Live mode requires both
+`confirmed: true` and non-empty `checks`. Terminal equivalents are `terminal.doctor.*`; every call
+requires the control key and each job belongs to its originating terminal. The terminal supplies
+its account and pinned variant; the controlled frontend supplies its current model and tool list.
+
+Reports carry job state (`running`, `completed`, `cancelled`), progress, timestamps, selected
+account/model, findings, and available live-check notices. Findings distinguish `local` from
+`live` and statuses `pass`, `warning`, `fail`, `disabled`, `blocked`, `unverified`. Schemas and
+Node types are exported. Jobs are in-memory until shutdown; real live-run records remain.
+
+Remote `tools.register` may include `diagnostics: [{ id, category, title, tools, probe?: { notice,
+tool, input, contains } }]`. The host answers `tools.diagnose({ id, context })` with `{ status,
+evidence, remedy? }` for local inspection. A probe is a declared registered tool call through
+the gate, with an expected response marker; it cannot supply arbitrary executable code.

package/docs/architecture/11-terminal.md

@@ -0,0 +1,81 @@
+# 11 Terminal
+
+## Purpose
+
+The harness owns a generic interactive terminal application (`xharness`) over a
+persistent Pi session on a variant. It is the standalone way to use the harness, and an
+embedding application can host it in its own PTY. Detail, screen layout and verification evidence
+are in [`../terminal-tui.md`](../terminal-tui.md); this page states the binding parts.
+
+## Components
+
+| Type / module | Role |
+|---|---|
+| `Terminals` (`terminal/terminals.ts`) | Parent side: `prepare` (account, compiled context, trusted resources, run writer) returns a launch specification; `send`, `state`, `input`, `dispose`. |
+| `src/terminal/opentui/` (`app.ts`, `root.tsx`) | The OpenTUI/Solid frontend: lifecycle, submission, commands, decisions, composition, transcript, composer, dialogs. |
+| `src/terminal/opentui/attachments.ts`, `composerMarks.ts` | Composer attachments: pasted images and collapsed large pastes, their atomic labels, and expansion into the prompt text and Pi image content on send. |
+| `src/terminal/tui/runtime.ts` | `AgentSessionRuntime` over `createAgentSession`, `ModelRuntime`, `SessionManager`, `DefaultResourceLoader`. |
+| `src/terminal/socket.ts`, `client.ts`, `control.ts`, `controller.ts` | The session socket, the private control key, and the run controller. |
+| `src/terminal/piExtension.ts` | The bridge extension, loaded last as the trusted inline gate. |
+| `src/terminal/cli.ts` | Wiki and connected-apps commands for the session's shell. |
+
+```text
+Harness parent: account + compiled context + trusted resources + run writer
+       | launch specification / session socket
+Harness terminal child (Bun): OpenTUI application + persistent Pi AgentSessionRuntime
+       | typed native events / plugin UI callbacks
+Screen: transcript above the composer, centred page dialogs
+```
+
+## Contracts
+
+- The published executable is `xharness`; with no command it starts the interactive terminal. `terminal` remains an explicit alias, and `serve` remains the host-facing RPC path.
+- `TerminalAssets` is `{ node, cli, piExtension, openTui }`. The command is Bun with
+  `--conditions=browser` and `dist/terminal-opentui.js`; no global `pi` is needed.
+- The `piExtension` path is the final prepared extension argument; the runtime loads it as the
+  final trusted inline factory.
+- The terminal compiles its prompt once (`compile` + `render`) and keeps that pack; each run
+  records a copy (page 04).
+- Standalone defaults: `~/.xingular-harness/{data,state}`; `npm run dev` uses
+  `~/.xingular-harness-dev/{data,state}`.
+
+## Invariants
+
+- **INV-TUI-1** OpenTUI is the only terminal frontend. Pi's `InteractiveMode` is never constructed
+  ([ADR-0006](../decisions/ADR-0006-opentui-only-terminal.md)).
+- **INV-TUI-2** The harness owns screen composition, input, scrolling, command routing and plugin
+  UI. Pi owns the execution runtime, native history and rich content components. A host owns the
+  PTY and process supervision ([ADR-0001](../decisions/ADR-0001-owned-terminal.md)).
+- **INV-TUI-3** Execution, account isolation, trust, authorization and run recording stay behind
+  their existing layers. The terminal adds no second path for any of them; its runs are written
+  through the controller and the run store.
+- **INV-TUI-4** Changes made through the session socket (`plugins.attach|detach|trust|untrust|configure`,
+  `accounts.*`, Hub installs) require the control key the trusted terminal process claimed once
+  (`control.claim`). The session's shell can read the session token, so the token alone is never
+  enough.
+- **INV-TUI-5** Every resource reload revalidates the original trusted selection; changed or
+  revoked code needs review and a fresh terminal.
+- **INV-TUI-6** `sessions.run` stays the embedding and unattended path. The terminal keeps a
+  persistent child instead of opening a session per turn.
+
+## Source map
+
+`src/terminal/**`. Decisions: ADR-0001, [ADR-0004](../decisions/ADR-0004-fullscreen-terminal-layout.md)
+(superseded), ADR-0006. Experiments: EXP-0001, EXP-0002 in `../experiments/`.
+
+## Session name in the working line (PROP-0010)
+
+The bridge registers `session_name` against Pi's `getSessionName` / `setSessionName` APIs.
+The native `session_info_changed` event invalidates the frontend's cached snapshot and
+refreshes the working line. Both the active and retained completion line append
+` – <name>` after duration and token rate, for example
+`Working (2m, 50 tok/s) – Add web browser integrations`. Unnamed sessions omit the suffix.
+Names use the existing terminal text sanitization, remain outside the verb shimmer, and
+truncate at the right edge. Manual `/name` and existing long names remain supported.
+
+## Doctor command
+
+`/doctor` opens a command page and runs local readiness checks. Refresh and retry are local;
+Live checks opens an explicit selection followed by account/model, destination, cost and isolation
+confirmation. The page polls structured reports, retains findings after cancellation, and cancels
+on close. No command text is sent to the conversation model. See [Doctor](../doctor.md).