pentesting 0.100.5 → 0.100.10

package/ARCHITECTURE.md

@@ -1,447 +1,523 @@
-# Architecture
-
-This document reflects the current `main` branch architecture. It is the single,
-detailed reference for how the runtime is structured; the [README](README.md)
-stays intentionally short and usage-focused.
-
-> **Engine vs product.** The orchestration engine is **`builder`** — a
-> general-purpose, local-first Rust agent runtime. **`pentesting`** is the
-> published, security-focused distribution of that same engine: the npm package
-> and the command you run. They share one binary; `pentesting` is `builder` with
-> a security skill set and banner. The model proposes actions, while the Rust
-> runtime owns routing, tool policy, evidence capture, verification, and
-> completion adjudication.
-
-## Design Pillars
-
-- **Runtime-Owned Adjudication** — Model output is a *candidate* until a runtime
-  acceptance-gate lattice verifies and closes the task (outcome: complete /
-  replan / blocked).
-- **Local-First State** — Runtime state and a markdown knowledge graph live on
-  disk. No external storage service.
-- **Weak-Model Hardening** — Rust-enforced classifiers, validators, and
-  self-reviews guard against hallucination.
-- **Lineage & Horizons** — Run lineage, objective IDs, and horizon-aware memory
-  reuse, all in local state.
-- **Policy-Gated Tools** — `Allow/Deny/Confirm` gates shell, file, and network
-  access (policy-based today; OS sandboxing is roadmap).
-- **Intelligent Queuing** — Queued inputs are deduplicated and prioritized live;
-  obsolete tasks are preempted, not FIFO-replayed.
-- **Prompt Ownership** — Prompt mode owns Enter submission. Buffered replay may
-  preserve typed-ahead input, but live terminal control responses, redraws, and
-  resize noise are not allowed to turn a normal Enter into a pasted newline.
-- **Lab Session Multiplexing** — `shell-listener` runs multiple authorized TCP
-  sessions with per-session routing, identity tags, verified PTY upgrade state,
-  transcripts, command ledgers, replay, and orphaned evidence recovery.
-- **Dynamic Agent Profiles** — No fixed persona: the runtime *derives* a profile
-  per request from the prompt and can overlay a named autonomy profile, driving
-  tool scope, phase, and memory weighting.
-- **Ebbinghaus-Inspired Memory** — Memories carry a *strength* that fades like
-  human memory and reinforces on recall. Faded notes are de-referenced, never
-  destroyed.
-- **Git-Backed Rewind** — Working-tree checkpoint/restore keeps autonomous edits
-  reversible.
-
-## Runtime Flow
-
-```mermaid
-flowchart TD
-    U[User request] --> CLI[builder_main CLI / TUI]
-    CLI --> API[builder_api facade]
-    API --> APP[builder_app coordination]
-    APP --> CLS[Intent classifier]
-    CLS --> STR[Dynamic profile + execution strategy]
-    STR --> AG[Active agent]
-    AG --> DEC{Delegate?}
-    DEC -- no --> TOOLS[Policy-gated tools]
-    DEC -- yes --> SUB[Subagent task packet]
-    SUB --> TOOLS
-    TOOLS --> EV[Evidence + state cards]
-    EV --> VER[Verification gates]
-    VER --> DONE[Completion adjudication]
-    DONE --> STORE[Local runtime state]
-    STORE --> CTX[Next-turn context]
-    CTX --> APP
-```
-
-Plain view:
-
-```text
-user input
-  -> CLI / interactive prompt loop
-  -> terminal-session prompt ownership + buffered replay boundary
-  -> BuilderAPI facade
-  -> BuilderApp classification and strategy
-  -> active agent or delegated subagent
-  -> policy-gated tool execution
-  -> evidence, workflow state, and verification records
-  -> completion adjudication
-  -> local state and next-turn context
-```
-
-Builder is a **domain-neutral** runtime with a security focus. Development,
-pentesting, CTF, audit, and release work layer in via skills — never baked into
-the core.
-
-## Dynamic Agent Profiles
-
-There is no fixed persona. The runtime derives a profile per request, optionally
-overlays a named autonomy profile, then reuses tool scope, phase, and memory
-weighting.
-
-CTF frontier exploration keeps the global `[exploration]` default off, then
-enables the effective CTF profile when `engagement.kind = "ctf"` or a
-`flag_format` is configured. That profile selects `ctf-competition` autonomy,
-turns on exploration, requires verified `flag_check` evidence before completion,
-and auto-continues the TUI loop after `/goal`. See `docs/config/ctf.toml`,
-`docs/plans/2026/07/08/PLAN_CtfFrontierExploration_2026-07-08.md`, and the
-decision record beside it.
-
-The shipped design is advisor-guided, not scheduler-owned: `decide_next` is a
-pure traversal recommendation over `EvidenceKind::Lead` events, while dispatch
-uses the existing delegated-task fan-out. There is no detached task registry or
-second runtime scheduler in the default implementation. Phase 10 is pinned by
-`orch_spec::exploration_e2e`, which exercises the flag-off path and the
-breadth/depth/pivot/reseed/merge flow without launching real subagents.
-
-Reliability guards sit below the profile layer. Provider protocol capabilities
-default native tool support for OpenAI/Anthropic-compatible responses on
-registry misses, unparsed tool-shaped markup becomes retryable instead of a
-false completion, raw control markup is scrubbed from the stream, and the
-markdown renderer bounds very large no-newline buffers.
-
-```mermaid
-flowchart LR
-    R[Request] --> GEN["Derive<br/>dynamic profile"]
-    GEN --> REC["Overlay<br/>named autonomy profile"]
-    REC --> USE["Reuse<br/>tool scope · phase · memory weight"]
-```
-
-Request to closure:
-
-```mermaid
-flowchart TD
-    U[User request] --> C[Intent classifier]
-    C --> P[Dynamic profile<br/>task shape + tool scope + rigor]
-    P --> R[Runtime router]
-    R --> A[Active agent]
-    A --> T{Need delegation?}
-    T -- no --> O[Tool execution]
-    T -- yes --> D[Agent tool]
-    D --> CO[coordinator]
-    D --> I[investigator]
-    D --> OP[operator]
-    D --> RV[reviewer]
-    D --> V[verifier]
-    D --> W[report-writer]
-    CO --> O
-    I --> O
-    OP --> O
-    RV --> O
-    V --> O
-    W --> O
-    O --> G[Completion gates]
-    G --> M[Memory + artifacts]
-    M --> U
-```
-
-## Agent Team
-
-The core agent team is domain-neutral. Each agent ships as a markdown persona
-definition under `crates/builder_repo/src/agents/`.
-
-| Agent | Default role | Runtime profile |
-| --- | --- | --- |
-| `builder` | Hands-on implementation, refactoring, local file changes, tests | Implement + broad local write |
-| `planner` | Implementation plans and risk breakdowns | Plan + read-only |
-| `researcher` | Read-only codebase and reference research | Investigate + read-only |
-| `coordinator` | Splits broad work into owned packets and consolidates results | Coordinate + broad local write |
-| `investigator` | Evidence gathering across code, logs, commands, APIs, and behavior | Investigate + shell diagnostics |
-| `operator` | Builds, tests, packaging, service startup, and runtime workflows | Implement + broad local write |
-| `reviewer` | Findings-first technical review | Review + strict read-only |
-| `verifier` | Reproduction, build/test proof, and completion claim checks | Verify + strict read-only |
-| `report-writer` | Reports, handoffs, release notes, and reproducibility records | Implement + bounded write |
-
-> Distinct tool-scope is runtime-enforced for `builder`, `planner`, and
-> `researcher`; the remaining personas share a task-shaped generic profile
-> (read-only when the task shape is review/verify, broad local write otherwise)
-> and differ by their prompt instructions.
-
-## Named Autonomy Profiles
-
-Optionally pin the top-level profile (`autonomy_profile = "ctf-competition"`);
-leave unset for classifier-driven defaults. CTF engagement config auto-selects
-the CTF profile when no explicit profile is set. Delegated subagent contracts
-stay intact. Defined in `crates/builder_domain/src/autonomy_profile.rs`.
-
-| Profile | Purpose |
-| --- | --- |
-| `general-agent` | Broad autonomous local orchestration for mixed tasks |
-| `local-builder` | Hands-on implementation with fresh evidence retrieval |
-| `ctf-competition` | Competition/lab workflow backed by `ctf-competition` and `pentesting-methodology` skills |
-| `enterprise-review` | Strict, review-heavy profile with read-only dynamic scope |
-
-## Engagement Metadata
-
-Runs can attach typed engagement context — scope, phase, tags, flag format,
-flag-evidence requirement, and standard refs (PTES, MITRE ATT&CK, OWASP,
-CWE/CAPEC, NIST CSF, CIS Controls) — to workflow metadata. `/workflow report`
-exports it as a Markdown handoff.
-
-## Memory & Knowledge
-
-Local-first and split across two complementary layers, no vector DB, no cloud:
-
-- **Conversation memories** — facts/preferences/gotchas distilled from sessions,
-  persisted in local runtime state and recalled into each prompt by hybrid
-  retrieval. These carry a *strength* that fades like human memory, so the store
-  stays sharp instead of rotting.
-- **Knowledge vault** — an optional Obsidian-style set of markdown notes under
-  `.builder/knowledge` (wiki-links, backlinks, tags). The agent reads it on every
-  turn and grows it with the `write` tool; it is never auto-deleted.
-
-Both layers feed the same strength-weighted hybrid retrieval.
-
-**Storage** — Ebbinghaus lifecycle (decay · reinforce · floor, never deleted):
-
-```mermaid
-flowchart TD
-    N[New memory] --> S["strength = quality × recall × e^-λ·age"]
-    S --> U{recalled?}
-    U -- yes --> RE[reinforce ↑ · reset age]
-    U -- no --> D[decay over time]
-    RE --> S
-    D --> F{below floor?}
-    F -- no --> S
-    F -- yes --> AR[de-reference: archive / tombstone]
-    AR -. recoverable on disk .-> N
-```
-
-Kinds fade at different speeds (procedural outlives episodic); bi-temporal
-`event_time` vs `ingestion_time` lets newer facts supersede stale ones.
-
-**Retrieval** — hybrid fuse, strength-weighted, read-only:
-
-```mermaid
-flowchart LR
-    Q[Query] --> L[Lexical]
-    Q --> SE[Semantic]
-    Q --> G[Graph]
-    L --> RRF[RRF fuse]
-    SE --> RRF
-    G --> RRF
-    RRF --> RR[rerank: phase · recency · task]
-    RR --> W[weight by strength]
-    W --> P[Prompt context]
-```
-
-Faded, private, or unsafe memories are held back from the prompt. Lookups never
-write — reinforce/archive/supersede are explicit, never search side effects.
-
-## Storage Strategy
-
-Builder uses one operational storage strategy: local on-disk runtime state plus
-markdown-native knowledge artifacts.
-
-```mermaid
-flowchart LR
-    W[Workspace files] --> AD[Source adapters]
-    S[Skills] --> AD
-    M[Conversation memories] --> AD
-    N[Markdown notes] --> AD
-    AD --> IDX[Local lexical + semantic + graph index]
-    IDX --> RR[Contextual reranker]
-    RR --> PC[Prompt context]
-    PC --> RUN[Agent run]
-    RUN --> LS[.builder local state]
-    RUN --> ART[Local artifacts and reports]
-```
-
-Key invariants:
-
-- Runtime state is local-first and does not require an external data service.
-- The markdown knowledge graph is derived from local facts: notes, skills,
-  conversation memory, wiki-links, backlinks, and run artifacts.
-- Graph data is a derived retrieval view, not a second source of truth.
-- Workflow runs, evidence, state cards, memories, and large-output artifacts
-  persist through local repositories.
-- Domain methods such as development, CTF practice, pentesting methodology,
-  audits, and release work live in skills and task instructions; they do not
-  change the core runtime identity.
-
-## Crate Map
-
-| Layer | Crates | Responsibility |
-| --- | --- | --- |
-| Interaction | `builder_main`, `builder_ratatui`, `builder_select`, `builder_markdown_stream` | CLI, TUI, prompt flow, rendering, selector fallback |
-| Facade | `builder_api` | Thin application boundary used by entry points |
-| Coordination | `builder_app` | classification, routing, execution strategy, orchestration, completion gates |
-| Contracts | `builder_domain` | typed IDs, tools, workflow records, engagement metadata, policies |
-| Capabilities | `builder_services` | tool services, auth, provider support, file/search/shell/fetch operations |
-| Persistence | `builder_repo`, `builder_provider_repo` | local repositories, provider catalog, chat repository |
-| Knowledge | `builder_knowledge`, `builder_workspace_index`, `builder_embed` | markdown note adapters, graph parsing, hybrid retrieval, local semantic scoring |
-| Infrastructure | `builder_config`, `builder_infra`, `builder_fs`, `builder_walker` | config, environment, filesystem, process/runtime helpers |
-| Support | `builder_display`, `builder_stream`, `builder_template`, `builder_json_repair`, `builder_snaps`, `builder_brand`, `builder_tracker`, `builder_test_kit`, `builder_tool_macros` | formatting, streaming, templates, JSON repair, snapshots, branding/theming, version tracking, tests, tool macros |
-
-## Tool Surface
-
-The runtime tool catalog is file, shell, network-fetch, planning, skill, todo,
-and delegation oriented. Core tools:
-
-- `read`, `write`, `patch`, `multi_patch`, `undo`, `remove`
-- `fs_search`, `sem_search`
-- `shell`, `fetch`
-- `plan`, `skill`, `todo_read`, `todo_write`
-- `task` and dynamically registered agent tools
-
-The full catalog (`builder_domain/src/tools/catalog.rs`) also includes web
-search, memory, verification primitives (flag check, best-of-N verify, finding),
-session/process control, and compaction tools. Direct raw storage-query tooling
-is not part of the active runtime surface.
-
-## Verification And Completion
-
-Completion is runtime-owned:
-
-```text
-candidate answer
-  -> evidence check
-  -> verification record
-  -> acceptance-gate check
-  -> adjudication
-  -> complete / replan / blocked
-```
-
-The key rule is that a model response is not considered done until the runtime
-has enough recorded evidence to close the task.
-
-## Distribution — single source, two surfaces
-
-Pentesting and Builder share the **same Rust runtime binary**. The `pentesting`
-npm package is a thin distribution facade:
-
-```text
-npm install -g pentesting
-         │
-         ▼
-  pentesting CLI (Node.js shim)
-         │  resolves or downloads the matching Builder release asset
-         ▼
-  Builder binary (Rust)  ← single runtime engine
-         │  PENTESTING_PRODUCT_NAME=pentesting
-         ▼
-  Interactive TUI with "pentesting" banner
-```
-
-- The npm package installs a launcher, **not** a second agent runtime.
-- It resolves or downloads the correct release asset from
-  `agnusdei1207/pentesting-public`.
-- It forwards arguments directly into the Rust binary — no command translation
-  or compatibility shims.
-- If a change would add orchestration, memory, or prompt logic into the npm
-  layer, that change belongs upstream in the Rust runtime.
-
-## Supported Runtime Targets
-
-The npm launcher currently resolves a managed native release asset for Linux
-x64. Other operating systems, including Apple Silicon Macs, Linux arm64,
-Windows, and Android, can run the multi-architecture Docker image or use
-`PENTESTING_BIN` to point at a locally built compatible binary.
-
-| OS | CPU | Release asset |
-| --- | --- | --- |
-| Linux | x64 | `pentesting-x86_64-unknown-linux-musl` |
-
-`/update` (and the npm postinstall) resolves the matching asset for the current
-supported OS/CPU target from `agnusdei1207/pentesting-public`.
-
-## Configuration
-
-Pentesting reads a user-global config from `~/.pentesting/.pentesting.toml`,
-overridden by project-local `.pentesting.toml` files discovered by walking up
-from the working directory. Local storage is the default; normal setup only
-needs the public model endpoint environment.
-
-The same shape is represented in `builder.schema.json`, `.pentesting.toml`, and
-the interactive `/config` UI. Set `PENTESTING_CONFIG` to override the global
-config directory.
-
-| Variable | Description |
-| --- | --- |
-| `OPENAI_API_KEY` | API key or gateway token for OpenAI-compatible providers. |
-| `OPENAI_BASE_URL` | OpenAI-compatible base URL, for example `https://api.openai.com/v1` or a self-hosted gateway. |
-| `OPENAI_MODEL` | Default model id. |
-| `OPENAI_MAX_TOKENS` | Optional output-token override; accepts values like `4096`, `32k`, or `1M`. |
-| `PENTESTING_BIN` | Use an already-installed Builder binary instead of the managed download. |
-| `PENTESTING_PRODUCT_NAME` | Runtime banner label. The `pentesting` launcher sets this to `pentesting` automatically. |
-| `PENTESTING_REPO` | Override the public release repo used for binary downloads. Defaults to `agnusdei1207/pentesting-public`. |
-| `PENTESTING_SKIP_DOWNLOAD` | Skip the postinstall binary download. Useful in CI or when `PENTESTING_BIN` will be provided later. |
-| `PENTESTING_CONFIG` | Override the global config directory. |
-
-### Backward compatibility — `builder` ↔ `pentesting`
-
-The runtime engine is still `builder` under the hood, so legacy names keep
-working. Use whichever you like; the `PENTESTING_*` form wins when both are set.
-
-| Surface | Canonical | Legacy (still accepted) |
-| :--- | :--- | :--- |
-| Env vars | `PENTESTING_*` | `BUILDER_*` |
-| Config-dir override | `PENTESTING_CONFIG` | `BUILDER_CONFIG` |
-| Global config file | `~/.pentesting/.pentesting.toml` | `~/.builder/.builder.toml`, `~/builder/.builder.toml` |
-| Project config file | `.pentesting.toml` | `.builder.toml` |
-
-## Interactive Commands
-
-Inside an interactive session, these commands inspect and drive runtime state
-(`/help` lists the full set):
-
-```text
-/status        Show the current run phase, active tasks, gates, hooks, and budget signals
-/workflow      Show the current focus and recent workflow steps for the active conversation
-/workflow report
-               Export the active run, engagement metadata, evidence, and large outputs to Markdown
-/context       Show recent context-budget snapshots for the current conversation
-/memory        Show stored conversation memories for the current conversation
-/tools         List the currently available tools and schemas
-/agent         Switch the active agent
-/conversation  Browse conversations for the active workspace
-/goal <task>   Set the active goal
-/auto          Toggle autonomous mode for the current goal
-/update        Download and apply the latest public release asset for supported native targets
-/help          Show all commands
-/exit          Quit
-```
-
-## Security Domain Skills
-
-The `ctf-competition` and `pentesting-methodology` skills map authorized
-assessments to standard frameworks:
-
-- **PTES** (Penetration Testing Execution Standard)
-- **MITRE ATT&CK** tactics and techniques
-- **OWASP** Top 10 and Testing Guide
-- **CWE/CAPEC** weakness and attack pattern catalogs
-- **NIST CSF** and **CIS Controls**
-
-### Shell listener for authorized labs
-
-```bash
-pentesting shell-listener --bind 127.0.0.1 --port 4444
-```
-
-Manages multiple accepted TCP sessions with per-session routing, buffered
-output, raw byte logging, full-duplex transcripts, command history, attach /
-detach events, replay, and evidence manifests. `pty-upgrade` sends the helper
-only; `upgrade` runs the helper plus a probe and records `pty_state` as verified
-or failed. The upgrade helper is drawn from a named technique catalog
-(`python3`, `python`, `python2`, `script`, `expect`) selectable per request —
-there is no default, so the request is rejected unless the caller picks a
-technique or supplies a full custom command, forcing the agent to recon the
-target and match what it actually has instead of assuming one interpreter; a
-missing tool is recorded as `pty_state=failed` with reason `<technique> helper
-unavailable`. Raw logs and transcripts are capped per session and closed sessions
-are archived automatically so a noisy peer cannot grow memory or disk without
-bound. Secret redaction is applied before new raw/transcript/display records are
-written; it is not retroactive for output that arrived before the secret value
-was registered. Bound to loopback by default; `--allow-remote` is an explicit
-opt-in gate.
+# Architecture
+
+This document reflects the current `main` branch architecture. It is the single,
+detailed reference for how the runtime is structured; the [README](README.md)
+stays intentionally short and usage-focused.
+
+> **Engine vs product.** The orchestration engine is **`builder`** — a
+> general-purpose, local-first Rust agent runtime. **`pentesting`** is the
+> published, security-focused distribution of that same engine: the npm package
+> and the command you run. They share one binary. The model owns task reasoning;
+> the Rust runtime owns protocol normalization, authorization, bounded execution,
+> observations, and verified outcome provenance.
+
+## 1. Architecture at a glance
+
+```text
+user input
+  → deterministic turn queue
+  → model/provider adapter
+  → one tool gateway
+  → authorized, bounded execution
+  → append observation to the turn
+  → repeat until TurnExit
+  → emit TurnCompleted
+  → emit TaskComplete only for a verified MissionState
+```
+
+The core is deliberately narrower than the complete product. Its append-only
+`TurnEvent` trajectory contains model terminals, actions, observations, and
+verification provenance. The complete trajectory is stored in the terminal
+`RunOutcome`; the request-time `TurnSnapshot` is its bounded O(1) projection.
+Memory, MCP,
+delegation, CTF exploration, and security sessions use the same tool boundary.
+Engagement context changes instructions, not tool reachability; resource caps
+bound expensive operations.
+
+## 2. Design pillars
+
+- **Model-Driven Work** — The model chooses plans, probes, commands, and ordinary
+  recovery. The runtime does not duplicate that reasoning with hot-path LLM judges.
+- **Hard Boundaries, Not Hard Opinions** — Schema, scope, permission, secrets,
+  cancellation, time, output, memory, and process limits are deterministic.
+- **Typed Turn Lifecycle** — Provider terminal state, `TurnExit`, and
+  `MissionState` are separate. Stop is never proof of success.
+- **One Tool Entry Boundary** — Built-in, delegated, and MCP tools share
+  allowlist, provenance, registered-contract, and schema validation. Hooks are
+  orchestrator-owned; permission checks apply only when a tool produces a
+  policy operation. Ordinary/MCP tools use configured timeouts, while
+  delegation uses its own runtime budget.
+- **Append-Oriented Observation** — Trajectory and evidence are local-first;
+  prior-run information is linked by reference rather than copied.
+- **Deterministic Queuing** — FIFO revisions support amend-next and explicit
+  replace without auxiliary LLM prioritization.
+- **One Security Surface** — Session, process, finding, flag verification, and
+  frontier operations are ordinary tools. Engagement context, permission,
+  resource budget, and proof remain separate responsibilities.
+- **Prompt Ownership** — Prompt mode owns Enter submission. Buffered replay may
+  preserve typed-ahead input, but live terminal control responses, redraws, and
+  resize noise are not allowed to turn a normal Enter into a pasted newline.
+  Modal return restores the transcript scroll region and its pre-modal saved
+  cursor before the fixed bottom UI is redrawn.
+- **Lab Session Multiplexing** — `shell-listener` runs multiple authorized TCP,
+  local PTY, and local pipe sessions. A single actor exclusively owns each
+  session's descriptors, command queue, transcript, raw log, and child
+  lifecycle; the registry keeps only a bounded sender and projection.
+- **Ebbinghaus-Inspired Memory** — Memories carry a *strength* that fades like
+  human memory and reinforces on recall. Faded notes are de-referenced, never
+  destroyed.
+- **Git-Backed Rewind** — Working-tree checkpoint/restore keeps autonomous edits
+  reversible.
+
+## 3. Runtime flow, one level deeper
+
+```mermaid
+flowchart TD
+    U[User request] --> CLI[builder_main CLI / TUI]
+    CLI --> API[builder_api facade]
+    API --> APP[builder_app coordination]
+    APP --> TURN[Turn kernel]
+    TURN --> MODEL[Provider adapter]
+    MODEL --> GW[Tool entry boundary]
+    GW --> POLICY[Schema + applicable policy/resource boundary]
+    POLICY --> EXEC[Tool / environment execution]
+    EXEC --> OBS[Observation + artifact refs]
+    OBS --> TURN
+    TURN --> EXIT[Typed TurnExit]
+    EXIT --> VERIFY[Outcome verifier]
+    VERIFY --> OUT[TurnCompleted / verified TaskComplete]
+```
+
+Plain view:
+
+```text
+user input
+  -> CLI / interactive prompt loop
+  -> terminal-session prompt ownership + buffered replay boundary
+  -> BuilderAPI facade
+  -> BuilderApp turn kernel and provider adapter
+  -> registered tool contract and common entry boundary
+  -> policy-gated, resource-bounded execution
+  -> observation and artifact references
+  -> typed turn exit and optional verified mission outcome
+```
+
+Builder is a domain-neutral runtime with a security-focused distribution.
+Development, pentesting, CTF, audit, and release behavior layers in through
+context, tools, and skills rather than new core strategy kinds.
+
+### Runtime persistence ports
+
+Hot-path consumers depend on ten cohesive persistence ports:
+
+```text
+run/objective/evidence/task/execution/outcome
+audit/observation/lineage/conversation-memory
+                         |
+          local persistence composition root
+```
+
+Each producer/consumer depends on the smallest relevant port. The methodless
+`ControlPlaneRepository` marker composes those ports only at construction and
+dynamic storage boundaries. Implementations and test fakes implement the
+physical ports directly; there are no blanket forwarding adapters, fallback
+methods, alternate schemas, or second runtime state.
+
+## 4. Engagement context and bounded CTF exploration
+
+There is no model-backed intent classifier or weak/standard model mode in the
+request hot path. Explicit event metadata is routed deterministically; absent
+metadata receives one general strategy. Named profiles may add context, but
+coarse labels do not authorize or reject tool calls.
+
+CTF engagement metadata may select `ctf-competition` prompt context, and
+`flag_check` remains a verification tool and evidence source rather than a
+default completion gate. `explore` is always available; `[exploration]` only
+sets node and fan-out caps. Installed users edit the relevant engagement, flag,
+and frontier fields through `/config`.
+
+The shipped exploration tool does not replace the core execution strategy.
+It provides a bounded `EvidenceKind::Lead` ledger and explicit dispatch. Stable
+generated identity is separate from semantic deduplication. Dead probes suppress
+an execution branch without rejecting its knowledge hypothesis, and fresh
+evidence reopens scheduling. There is no passive `/frontier` command, detached
+task registry, or second scheduler. `orch_spec::exploration_e2e` pins the off
+path and the explicit breadth/depth/pivot/reseed/merge behavior.
+
+Reliability guards sit below the profile layer. Provider protocol capabilities
+default native tool support for OpenAI/Anthropic-compatible responses on
+registry misses, unparsed tool-shaped markup becomes retryable instead of a
+false completion, raw control markup is scrubbed from the stream, and the
+markdown renderer flushes long no-newline prose at soft boundaries while still
+bounding very large buffers.
+
+```mermaid
+flowchart LR
+    R[Request] --> GEN["Resolve<br/>task-shaped defaults"]
+    GEN --> REC["Optional<br/>named profile / pack"]
+    REC --> USE["Bound<br/>tool scope and budget"]
+```
+
+Request to closure:
+
+```mermaid
+flowchart TD
+    U[User request] --> A[Agent turn]
+    A --> M[Model output]
+    M --> T{Registered action?}
+    T -- no --> X[Typed turn exit]
+    T -- yes --> G[Common tool gateway]
+    G --> O[Observation]
+    O --> A
+    X --> V{Verified artifact?}
+    V -- no --> TC[TurnCompleted]
+    V -- yes --> MC[TaskComplete + provenance]
+```
+
+## Agent Team
+
+The core agent team is domain-neutral. Each agent ships as a markdown persona
+definition under `crates/builder_repo/src/agents/`.
+
+Subagents are explicit and bounded. A `task` call selects one agent; its tasks
+run concurrently up to four per call and delegation stops at depth two.
+Separate top-level tool calls are sequenced. Built-ins include `crypto`, `rev`,
+and `pwn`.
+
+`explore dispatch` is a separate path. It runs selected leads as `builder`
+clones with category overlays, defaults to breadth one, is hard-capped at four,
+and waits for the batch. It does not auto-route leads to specialist personas.
+Every child uses the same tool/policy gateway, budget, lineage, and evidence
+merge contracts.
+
+| Agent | Default role | Runtime profile |
+| --- | --- | --- |
+| `builder` | Hands-on implementation, refactoring, local file changes, tests | Implement + broad local write |
+| `planner` | Implementation plans and risk breakdowns | Plan + read-only |
+| `researcher` | Read-only codebase and reference research | Investigate + read-only |
+| `coordinator` | Splits broad work into owned packets and consolidates results | Coordinate + broad local write |
+| `investigator` | Evidence gathering across code, logs, commands, APIs, and behavior | Investigate + shell diagnostics |
+| `operator` | Builds, tests, packaging, service startup, and runtime workflows | Implement + broad local write |
+| `reviewer` | Findings-first technical review | Review + strict read-only |
+| `verifier` | Reproduction, build/test proof, and completion claim checks | Verify + strict read-only |
+| `report-writer` | Reports, handoffs, release notes, and reproducibility records | Implement + bounded write |
+
+> Distinct tool-scope is runtime-enforced for `builder`, `planner`, and
+> `researcher`; the remaining personas share a task-shaped generic profile
+> (read-only when the task shape is review/verify, broad local write otherwise)
+> and differ by their prompt instructions.
+
+## Named Autonomy Profiles
+
+Optionally pin the top-level profile (`autonomy_profile = "ctf-competition"`);
+leave unset for the deterministic general default. CTF engagement config may
+select CTF prompt context when no explicit profile is set, but it never changes
+tool reachability. Delegated subagent contracts stay intact. Defined in
+`crates/builder_domain/src/autonomy_profile.rs`.
+
+| Profile | Purpose |
+| --- | --- |
+| `general-agent` | Broad autonomous local orchestration for mixed tasks |
+| `local-builder` | Hands-on implementation with fresh evidence retrieval |
+| `ctf-competition` | Competition/lab workflow backed by `ctf-competition` and `pentesting-methodology` skills |
+| `enterprise-review` | Strict, review-heavy profile with read-only dynamic scope |
+
+## Engagement Metadata
+
+Runs can attach typed engagement context — scope, phase, tags, flag format,
+flag-evidence requirement, and standard refs (PTES, MITRE ATT&CK, OWASP,
+CWE/CAPEC, NIST CSF, CIS Controls) — to workflow metadata. `/workflow report`
+exports it as a Markdown handoff.
+
+## Memory & Knowledge
+
+Local-first and split across two complementary layers, no vector DB, no cloud:
+
+- **Conversation memories** — facts/preferences/gotchas distilled from sessions,
+  persisted in local runtime state and recalled into each prompt by hybrid
+  retrieval. These carry a *strength* that fades like human memory, so the store
+  stays sharp instead of rotting.
+- **Knowledge vault** — an optional Obsidian-style set of markdown notes under
+  `.pentesting/knowledge` (wiki-links, backlinks, tags). The agent reads it on every
+  turn and grows it with the `write` tool; it is never auto-deleted.
+
+Both layers feed the same strength-weighted hybrid retrieval.
+
+**Storage** — Ebbinghaus lifecycle (decay · reinforce · floor, never deleted):
+
+```mermaid
+flowchart TD
+    N[New memory] --> S["strength = quality × recall × e^-λ·age"]
+    S --> U{recalled?}
+    U -- yes --> RE[reinforce ↑ · reset age]
+    U -- no --> D[decay over time]
+    RE --> S
+    D --> F{below floor?}
+    F -- no --> S
+    F -- yes --> AR[de-reference: archive / tombstone]
+    AR -. recoverable on disk .-> N
+```
+
+Kinds fade at different speeds (procedural outlives episodic); bi-temporal
+`event_time` vs `ingestion_time` lets newer facts supersede stale ones.
+
+**Retrieval** — hybrid fuse, strength-weighted, read-only:
+
+```mermaid
+flowchart LR
+    Q[Query] --> L[Lexical]
+    Q --> SE[Semantic]
+    Q --> G[Graph]
+    L --> RRF[RRF fuse]
+    SE --> RRF
+    G --> RRF
+    RRF --> RR[rerank: phase · recency · task]
+    RR --> W[weight by strength]
+    W --> P[Prompt context]
+```
+
+Faded, private, or unsafe memories are held back from the prompt. Lookups never
+write — reinforce/archive/supersede are explicit, never search side effects.
+
+## Storage Strategy
+
+Builder uses one operational storage strategy: local on-disk runtime state plus
+markdown-native knowledge artifacts.
+
+```mermaid
+flowchart LR
+    W[Workspace files] --> AD[Source adapters]
+    S[Skills] --> AD
+    M[Conversation memories] --> AD
+    N[Markdown notes] --> AD
+    AD --> IDX[Local lexical + semantic + graph index]
+    IDX --> RR[Contextual reranker]
+    RR --> PC[Prompt context]
+    PC --> RUN[Agent run]
+    RUN --> LS[.pentesting local state]
+    RUN --> ART[Local artifacts and reports]
+```
+
+Key invariants:
+
+- Runtime state is local-first and does not require an external data service.
+- The markdown knowledge graph is derived from local facts: notes, skills,
+  conversation memory, wiki-links, backlinks, and run artifacts.
+- Graph data is a derived retrieval view, not a second source of truth.
+- Evidence, workflow runs, state cards, and verified artifacts persist through
+  local repositories. Removed semantic-verdict, recovery-route, completion
+  bundle, replay-archive, and curation projections cannot redirect the active
+  model loop because they no longer exist in the runtime model.
+- Domain methods such as development, CTF practice, pentesting methodology,
+  audits, and release work live in skills and task instructions; they do not
+  change the core runtime identity.
+
+## Crate Map
+
+| Layer | Crates | Responsibility |
+| --- | --- | --- |
+| Interaction | `builder_main`, `builder_ratatui`, `builder_select`, `builder_markdown_stream` | CLI, TUI, prompt flow, rendering, selector fallback |
+| Facade | `builder_api` | Thin application boundary used by entry points |
+| Coordination | `builder_app` | turn loop, provider adaptation, common tool gateway, observations, verified outcome |
+| Contracts | `builder_domain` | typed IDs, tools, workflow records, engagement metadata, policies |
+| Capabilities | `builder_services` | tool services, auth, provider support, file/search/shell/fetch operations |
+| Persistence | `builder_repo`, `builder_provider_repo` | local repositories, provider catalog, chat repository |
+| Knowledge | `builder_knowledge`, `builder_workspace_index`, `builder_embed` | markdown note adapters, graph parsing, hybrid retrieval, local semantic scoring |
+| Infrastructure | `builder_config`, `builder_infra`, `builder_fs`, `builder_walker` | config, environment, filesystem, process/runtime helpers |
+| Support | `builder_display`, `builder_stream`, `builder_template`, `builder_json_repair`, `builder_snaps`, `builder_brand`, `builder_tracker`, `builder_test_kit`, `builder_tool_macros` | formatting, streaming, templates, JSON repair, snapshots, branding/theming, version tracking, tests, tool macros |
+
+## Tool Surface
+
+The runtime tool catalog is file, shell, network-fetch, planning, skill, todo,
+and delegation oriented. Core tools:
+
+- `read`, `write`, `patch`, `multi_patch`, `undo`, `remove`
+- `fs_search`, `sem_search`
+- `shell`, `fetch`
+- `plan`, `skill`, `todo_read`, `todo_write`
+- `task` and dynamically registered agent tools
+
+The full catalog (`builder_domain/src/tools/catalog.rs`) also includes web
+search, memory, verification primitives, session/process control, and
+compaction tools. Security-session primitives, `flag_check`, and `explore` are
+all present on the same surface. Engagement metadata supplies context;
+permission and resource budgets constrain execution. Direct raw storage-query
+tooling is not part of the active runtime surface.
+
+## Verification And Completion
+
+Turn closure and mission closure are separate:
+
+```text
+provider terminal
+  -> TurnExit
+  -> TurnCompleted
+
+candidate outcome + trusted artifact
+  -> bind objective + target + verifier + artifact + freshness
+  -> MissionState::Verified
+  -> TaskComplete
+```
+
+The key rule is that a model response may finish a turn but cannot prove the
+mission. The former completion-adjudicator, semantic self-review, consensus
+panel, recovery synthesis, and hot-path replay export have been removed. A
+trusted verifier with explicit provenance is the only mission-promotion path;
+`flag_check` is the direct CTF fast path.
+
+## Diagnostic UI boundary
+
+The default TUI renders the response stream and deterministic input queue. It
+does not maintain a second workflow engine. `/status` consumes one immutable
+`RunStatusProjection`; `/workflow report` is the explicit offline export. The
+former bare `/workflow` alias has been removed. The live status row deliberately
+shows only actionable progress/wait state; frontier JSON is not parsed into
+hidden chrome state, and full run/tool detail remains in explicit output and
+`/status`.
+
+## Distribution — single source, two surfaces
+
+Pentesting and Builder share the **same Rust runtime binary**. The `pentesting`
+npm package is a thin distribution facade:
+
+```text
+npm install -g pentesting
+         │
+         ▼
+  pentesting CLI (Node.js shim)
+         │  resolves or downloads the matching Builder release asset
+         ▼
+  Builder binary (Rust)  ← single runtime engine
+         │  PENTESTING_PRODUCT_NAME=pentesting
+         ▼
+  Interactive TUI with "pentesting" banner
+```
+
+- The npm package installs a launcher, **not** a second agent runtime.
+- It resolves or downloads the correct release asset from
+  `agnusdei1207/pentesting-public`.
+- It forwards arguments directly into the Rust binary — no command translation
+  or compatibility shims.
+- If a change would add orchestration, memory, or prompt logic into the npm
+  layer, that change belongs upstream in the Rust runtime.
+
+## Supported Runtime Targets
+
+The npm launcher currently resolves a managed native release asset for Linux
+x64. Docker is the recommended runtime on Windows and macOS. Published Docker
+images support `linux/amd64` only; Apple Silicon Macs use Docker Desktop's amd64
+emulation. Native ARM64 images and binaries are not supported. Advanced users
+may set `PENTESTING_BIN` to a locally supplied compatible binary.
+
+| OS | CPU | Release asset |
+| --- | --- | --- |
+| Linux | x64 | `pentesting-x86_64-unknown-linux-musl` |
+
+`/update` (and the npm postinstall) resolves the matching asset for the current
+supported OS/CPU target from `agnusdei1207/pentesting-public`.
+
+## Build containment boundary
+
+Compilation is outside the application runtime boundary and is always executed
+through resource-capped Docker wrappers. Host Cargo is not an emergency
+fallback. `dbuild.sh` owns individual Cargo commands, `dverify.sh` owns
+sequential gate profiles, and `dbuild-image.sh` owns BuildKit. Each path shares
+storage admission, effective-limit inspection, a machine-wide lock, watchdogs,
+and structured failure reports. Automated CI/CD is disabled; maintainers invoke
+these wrappers explicitly. The
+normative contract is [Docker-only build safety](https://github.com/agnusdei1207/pentesting/blob/main/docs/BUILD_SAFETY.md).
+
+The measured next-stage simplification and deletion plan is indexed at
+[Minimal Core plan](https://github.com/agnusdei1207/pentesting/blob/main/docs/plans/2026/07/10/minimal-core/README.md).
+The concrete PTY actor, unified security-tool surface, turn-trajectory, and dead-path reduction is
+recorded in [the 2026-07-10 implementation report](docs/plans/2026/07/10/IMPLEMENTATION_LightweightCoreRefactor_2026-07-10.md).
+
+## Configuration
+
+Pentesting reads a user-global config from `~/.pentesting/.pentesting.toml`,
+overridden by project-local `.pentesting.toml` files discovered by walking up
+from the working directory. Local storage is the default; normal setup only
+needs the public model endpoint environment.
+
+The full file shape is represented in `builder.schema.json`; common runtime and
+CTF fields are editable through `/config`. Set `PENTESTING_CONFIG` to override
+the global config directory.
+
+| Variable | Description |
+| --- | --- |
+| `OPENAI_API_KEY` | API key or gateway token for OpenAI-compatible providers. |
+| `OPENAI_BASE_URL` | OpenAI-compatible base URL, for example `https://api.openai.com/v1` or a self-hosted gateway. |
+| `OPENAI_MODEL` | Default model id. |
+| `OPENAI_MAX_TOKENS` | Optional output-token override; accepts values like `4096`, `32k`, or `1M`. |
+| `PENTESTING_BIN` | Use an already-installed Builder binary instead of the managed download. |
+| `PENTESTING_PRODUCT_NAME` | Runtime banner label. The `pentesting` launcher sets this to `pentesting` automatically. |
+| `PENTESTING_REPO` | Override the public release repo used for binary downloads. Defaults to `agnusdei1207/pentesting-public`. |
+| `PENTESTING_SKIP_DOWNLOAD` | Skip the postinstall binary download. Useful in CI or when `PENTESTING_BIN` will be provided later. |
+| `PENTESTING_CONFIG` | Override the global config directory. |
+
+### Canonical configuration contract
+
+The next-generation runtime accepts one product namespace. Configuration uses
+`PENTESTING_*`, `~/.pentesting/.pentesting.toml`, and project-local
+`.pentesting.toml`. `BUILDER_*`, `.builder.toml`, `.builder/`, JSON config
+fallbacks, and runtime alias translation have been removed. Existing users must
+move data explicitly before upgrading; the runtime does not carry a permanent
+old/new branch.
+
+## Interactive Commands
+
+Inside an interactive session, these commands inspect and drive runtime state
+(`/help` lists the full set):
+
+```text
+/status        Show the current run phase, active tasks, hooks, and budget signals
+/workflow report
+               Export the active run, engagement metadata, evidence, and large outputs to Markdown
+/context       Show recent context-budget snapshots for the current conversation
+/memory        Show stored conversation memories for the current conversation
+/tools         List the currently available tools and schemas
+/agent         Switch the active agent
+/conversation  Browse conversations for the active workspace
+/goal <task>   Set the active goal without enabling autonomous mode
+/auto          Toggle autonomous mode for the current goal (default: off; Esc stops it)
+/update        Download and apply the latest public release asset for supported native targets
+/help          Show all commands
+/exit          Quit
+```
+
+## Security Domain Skills
+
+The `ctf-competition` and `pentesting-methodology` skills map authorized
+assessments to standard frameworks:
+
+- **PTES** (Penetration Testing Execution Standard)
+- **MITRE ATT&CK** tactics and techniques
+- **OWASP** Top 10 and Testing Guide
+- **CWE/CAPEC** weakness and attack pattern catalogs
+- **NIST CSF** and **CIS Controls**
+
+### Shell listener for authorized labs
+
+```bash
+pentesting shell-listener --bind 127.0.0.1 --port 4444
+```
+
+Manages multiple accepted TCP sessions with per-session routing, buffered
+output, raw byte logging, full-duplex transcripts, command history, attach /
+detach events, replay, and evidence manifests. `pty-upgrade` sends one concrete
+helper only; `upgrade` runs a helper plus a probe and records `pty_state` as
+verified or failed. Explicit `technique=auto` adds a bounded listener-side state
+machine: framed capability recon, allowlisted `bash`/`sh` selection, ordered
+helper attempts (`python3`, `python`, `python2`, `script`, `expect`), generic
+prompt observation, and phase-local probe verification. Only unavailable,
+early-returned, or completed-but-unverified attempts advance to the next
+candidate; timeout, disconnect, or writer failure stops without blind input.
+The entire operation keeps one command id, deadline, session lease, ledger, and
+final response. There is still no silent default: omission is rejected, manual
+techniques keep their single-attempt behavior, and custom text remains the
+escape hatch for unusual targets. Raw logs and transcripts are capped per session and closed sessions
+are archived automatically so a noisy peer cannot grow memory or disk without
+bound. Secret redaction is applied before new raw/transcript/display records are
+written; it is not retroactive for output that arrived before the secret value
+was registered. Bound to loopback by default; `--allow-remote` is an explicit
+opt-in gate.