@andorra-labs/code 0.0.0-stage → 0.1.5

package/tests/graphiti-test-plan.md

@@ -0,0 +1,359 @@
+# QA test plan: Graphiti persistent memory
+
+Goal: verify that Andorra's persistent memory (Graphiti) behaves as designed.
+Follow the steps in order; each scenario lists **steps** and **expected
+behavior**. Record any deviation as a bug.
+
+Unit/integration coverage lives in `test/memory-*.test.ts` (run `npm test`).
+This plan is for end-to-end behavior with a running harness.
+
+## 0. Setup
+
+1. Build: `npm run build`.
+2. Start the mock Graphiti service (no Python needed):
+   ```sh
+   node tests/mock-graphiti.mjs 8000
+   ```
+   Alternatively run the real Graphiti service and adjust the endpoint.
+3. Use a scratch config home and a scratch git workspace:
+   ```sh
+   export ANDORRACODE_HOME=/tmp/andorra-qa-home
+   mkdir -p /tmp/andorra-qa-repo && cd /tmp/andorra-qa-repo && git init && git remote add origin https://github.com/qa/demo-repo
+   ```
+   A configured model provider is required for runs (`andorra setup`).
+4. Enable memory in `$ANDORRACODE_HOME/config.json`:
+   ```json
+   "memory": { "enabled": true, "provider": "graphiti", "graphiti": { "endpoint": "http://localhost:8000" } }
+   ```
+5. Compute the scratch repo's memory group id. Repository ids are a readable
+   slug plus a short hash of the canonical remote, so distinct remotes can
+   never collide. For the remote above:
+   ```sh
+   node -e "console.log('repo:github.com-qa-demo-repo-' + require('crypto').createHash('sha256').update('github.com/qa/demo-repo').digest('hex').slice(0, 8))"
+   ```
+   The rest of this plan calls this value `<group-id>`. You can also read it
+   from any `memory.retrieved` run-log event (`data.repository` is the id
+   without the `repo:` prefix).
+
+Run artifacts land in `$ANDORRACODE_HOME/runs/<runId>.jsonl`; memory activity
+is visible as `memory.*` events there and as `[Memory]` lines in the
+transcript.
+
+---
+
+## 1. Default behavior (memory not configured)
+
+**Steps:** remove the `memory` section from config; run a task:
+`andorra --cwd /tmp/andorra-qa-repo -p "Add a hello.txt file" --yolo`.
+
+**Expected:**
+- The run completes normally; memory is silently off.
+- No `memory_search` tool is advertised to the agent; no "Relevant persistent
+  memory" block in the task; no `memory.retrieved`/`memory.distilled` events
+  in the run log.
+- The run log starts with a replayed `memory.health` event (status
+  `unconfigured` or `disabled`) so the log explains why memory was absent.
+- Nothing is written to the mock service (its stdout shows no requests).
+
+## 2. Graphiti unavailable → graceful degradation
+
+**Steps:** enable memory (setup step 4) but stop the mock service. Run a task.
+
+**Expected:**
+- Stderr shows one warning like `Memory: Graphiti ... unavailable` (or the
+  health detail), then the run proceeds and completes normally.
+- No crash, no failed run status caused by memory.
+- Run log contains a replayed `memory.health` event with
+  `status: "unavailable"` (attached to the first run even though the check
+  happened at startup).
+- `memory_search` is not offered (provider degraded to NoOp).
+
+## 3. Configuration validation
+
+**Steps:** for each bad config below, start the CLI and observe the error;
+restore a good config afterwards.
+
+| Config change | Expected |
+|---|---|
+| `"provider": "bogus"` | startup error: `memory.provider must be graphiti or none` |
+| `"graphiti": {"endpoint": "http://8.8.8.8:8000"}` | error: public Graphiti endpoints require HTTPS |
+| `"graphiti": {"endpoint": "http://user:pass@localhost:8000"}` | error: credentials in URL not allowed |
+| `"graphiti": {"endpoint": "https://graphiti.example.com"}` | accepted (HTTPS public) |
+| `"maxRetrievedMemories": 99` | error: must be 1–20 |
+| `"memoryContextChars": 100` | error: must be 500–16000 |
+
+**Env overrides:** `ANDORRA_GRAPHITI_URL=http://localhost:8000 andorra -p ...`
+with no config section → memory active. `ANDORRA_MEMORY_PROVIDER=none` →
+memory off even with a config section. Neither writes back to config.json.
+
+## 4. Automatic retrieval
+
+**Steps:**
+1. Seed memory directly through the mock (or by completing scenario 6 first),
+   substituting the computed `<group-id>`:
+   ```sh
+   curl -X POST http://localhost:8000/messages -H 'content-type: application/json' -d '{
+     "group_id": "<group-id>",
+     "messages": [{"uuid":"seed-1","name":"andorra:architecture_decision","role":"andorra-memory","role_type":"system",
+       "content":"Sessions use PostgreSQL\nSessions moved from Redis to PostgreSQL for durable revocation.",
+       "timestamp":"2026-01-01T00:00:00Z",
+       "source_description":"{\"andorra\":1,\"type\":\"architecture_decision\",\"title\":\"Sessions use PostgreSQL\",\"verification\":\"user\"}"}]}'
+   ```
+2. Run: `andorra -p "Fix the authentication issue in session handling" --yolo`.
+
+**Expected:**
+- Transcript shows a `[Memory]` block: `1 relevant memory for orchestrator (repo: github.com-qa-demo-repo-<hash>, N chars)` listing `seed-1 architecture_decision CURRENT`.
+- Run log has `memory.retrieved` with the memory id, type, `temporalStatus: "current"`, verification, and injected char count.
+- The agent's task contains "Relevant persistent memory" with the seeded fact
+  and the advisory preamble (current repository is authoritative).
+- Retrieval happens per agent node without the model asking for it.
+
+## 5. Explicit memory_search tool
+
+**Steps:** with the seed from scenario 4, run
+`andorra -p "Use the memory_search tool to answer: what backend do sessions use, and why?" --yolo`.
+
+**Expected:**
+- The agent calls `memory_search`; tool output is concise JSON with `id`,
+  `type`, `temporalStatus`, `verification`, provenance (`commit`, `tests`).
+- Default search returns current facts only.
+- `memory_search({query:"redis sessions", includeHistorical:true})` also
+  returns superseded facts labeled `temporalStatus: "historical"` with
+  `validUntil` set.
+- The persisted `node.tool.completed` event for `memory_search` in the run
+  log contains result metadata (id/type/status) but **not** the memory
+  `content` bodies — memory text goes to the model only.
+- Invalid args produce tool errors: missing/empty `query`, `limit` outside
+  1–20.
+- With memory disabled/unavailable, the tool answers with a plain
+  "unavailable" message instead of throwing.
+
+## 6. Distillation on completion
+
+**Steps:** run a fix task end-to-end, e.g. create a tiny bug in the scratch
+repo and run `andorra -p "Fix the bug in hello.js where ..." --yolo` so the
+run completes with passing tests.
+
+**Expected:**
+- Run log ends with `memory.distilled` (candidates/persisted counts).
+- The mock received `POST /messages` with `group_id: "<group-id>"`,
+  `name: "andorra:bug_fix"`, and `source_description` JSON containing type,
+  title, verification, and provenance (`commitSha`, `files`, `tests`,
+  `branch`).
+- Verification is `"test"` only when the run's test entries read as passing
+  (e.g. `"... passed"`). A completion whose tests say "Not run" or report
+  failures must not produce `test`-verified memory (at most
+  `agent_verified`, and only when the candidate is otherwise durable and
+  verified).
+- `curl 'http://localhost:8000/episodes/<group-id>?last_n=10'`
+  shows the episode.
+- The distillation model call's tokens are included in the run's reported
+  usage (run result `.json` `usage` fields exceed the node-only totals).
+- A follow-up run for a related task now retrieves this memory automatically
+  (scenario 4 behavior).
+- A run that fails/is cancelled produces **no** `memory.distilled` and no new
+  episodes. A run that technically succeeds but ends in a negative terminal
+  outcome (a QA node answering `fail`/`blocked`/`rejected`) also persists
+  nothing.
+
+## 7. Temporal supersession (Redis → PostgreSQL)
+
+**Steps:**
+1. Complete scenario 6 so a fact like "Sessions use Redis" exists (or seed it
+   via curl).
+2. Complete a later run whose outcome is "Sessions now use PostgreSQL"
+   (distillation writes a new episode superseding the old one). Note the
+   title wording may change; supersession must still happen when the new
+   memory has the same type and substantially the same subject.
+
+**Expected:**
+- Default retrieval/search returns only the PostgreSQL fact (current).
+- `includeHistorical: true` returns both; the Redis fact is marked
+  `historical` with `validUntil`, never presented as current.
+- Both episodes remain visible via `/episodes/...` (history preserved, not
+  deleted).
+
+## 8. Repository isolation
+
+**Steps:** create a second scratch repo (`git init`, different remote URL),
+enable the same memory config, run a task there and search memory.
+
+**Expected:**
+- The second repo's group is `repo:<its-own-id>`; searches never return the
+  first repo's memories.
+- No unscoped search requests reach the service (every `/search` body has
+  exactly one `group_ids` entry).
+- Two remotes whose slugs would collide (`https://github.com/acme/a-b` and
+  `https://github.com/acme-a/b`) still get different group ids because of
+  the canonical-remote hash suffix. Verify by computing both ids with the
+  setup one-liner.
+
+## 9. Staleness flagging
+
+**Steps:** persist a memory whose provenance names a file and commit (via a
+real fix run). Then modify that file. Run a related task **before**
+committing, then again after committing.
+
+**Expected:** in both cases the retrieved memory is flagged
+`REQUIRES REVALIDATION` in the injected block and `requiresRevalidation: true`
+in the `memory.retrieved` event; it is ranked below current verified
+memories. (Uncommitted/staged edits count, not just new commits.)
+
+## 10. Security: secrets never persisted
+
+**Steps:** craft a task whose completion/candidate would contain a credential
+(e.g. have the agent fix a config containing `api_key: 9f8e7d6c5b4a3210`, or
+a test command like `curl -H "Authorization: Bearer <token>" ...`).
+Inspect the episode written to the mock.
+
+**Expected:** no episode content contains the raw key — the candidate is
+rejected (`memory.rejected` event) or the span is `[REDACTED]`. Titles with
+credential-like material are always rejected. The same screening applies to
+provenance: redacted or dropped inside `source.tests`, `source.files`, and
+`entities` in `source_description`.
+
+---
+
+## Edge cases / potential bugs to probe
+
+1. **Remote URL formats**: repeat setup with `git@github.com:Org/Repo.git`,
+   `https://github.com/Org/Repo/`, and no remote at all. Expect the same repo
+   id for equivalent remotes (`github.com-org-repo-<hash8>`) and a stable
+   `local-<hash>` id across runs without a remote.
+2. **Service dies mid-run**: start the mock, start a run, kill the mock
+   before retrieval. The run must still complete; a `memory.error` event is
+   logged **with the run id** and rendered in the transcript, never an
+   exception.
+3. **Slow service**: add latency to the mock (e.g. 30s). Requests must time
+   out per `requestTimeoutMs` (default 10s; health ≤5s) without hanging the
+   run; consider whether startup health adds noticeable latency to `-p` runs.
+4. **Empty/noisy memory**: a repo with zero memories must inject no block
+   (empty section must not appear). A repo with 50 memories must inject at
+   most `maxRetrievedMemories` (default 8) within `memoryContextChars`.
+5. **Duplicate suppression**: run the same fix twice; the second distillation
+   must not create a duplicate episode for identical content.
+6. **Changed content or changed title, same subject**: expect supersession
+   (old → historical), not two competing current facts. An unrelated memory
+   of the same type (different subject) must NOT be superseded — in
+   particular, titles sharing only generic words or a single entity are not
+   enough: "Use Redis for caching" must not supersede "Use Redis for
+   sessions". Supersession requires a shared entity plus a significant token,
+   or at least two shared significant tokens with majority overlap.
+7. **Historical labeling**: verify historical facts are never injected as
+   current context and are explicitly labeled when explicitly requested.
+8. **Concurrent runs**: two simultaneous runs in the same repo; both can
+   read/write without cross-run corruption (Graphiti serializes writes).
+9. **Config/env precedence**: env override + config section set together →
+   env wins; config file unchanged afterwards.
+10. **Budget guard**: a pathological task where every memory is huge must
+    still cap the injected block at `memoryContextChars`; an oversized
+    top-ranked memory is clipped (with `…`) and must not starve smaller
+    relevant memories out of the block.
+11. **No writes from agents**: the agent has no tool that writes memory
+    directly; verify only distillation writes (mock logs show `/messages`
+    only after `memory.distilled`).
+12. **`memory_search` in restricted graphs**: a custom graph node whose
+    `tools` array omits `memory_search` must not see the tool; a node
+    listing it (or having no `tools` allowlist) must.
+13. **Headless vs interactive**: retrieval and distillation work in both
+    `--prompt` headless runs and interactive TUI runs; degradation warnings
+    go to stderr and never corrupt `--json` output.
+14. **Non-git workspace**: run in a plain directory; memory still works under
+    a `local-<hash>` scope, no git errors surface.
+15. **Redirecting endpoint**: point the endpoint at a server answering
+    301/307/308 to another host (e.g. `http://127.0.0.1:9`). Expected: the
+    client treats it as service failure (degrades to NoOp) and never sends
+    queries or episode bodies to the redirect target.
+16. **Evidence grounding**: if the distiller's candidates claim files/tests
+    the run never touched (or an invented commit SHA), the persisted episode's
+    provenance must be narrowed to the run's actual evidence and the real
+    HEAD SHA. A hallucinated "tests passed" claim must never earn `test`
+    verification.
+17. **Cancellation during distillation**: abort (Ctrl-C / SIGTERM) while the
+    distiller's model call is in flight. Expected: no `memory.distilled`
+    event, no episodes written, and no deterministic-fallback candidates
+    persisted after the abort.
+18. **Token budget exhaustion**: a run that exhausts its token limit at the
+    final node must skip distillation entirely (no `memory.distilled`, no
+    extra model call beyond the budget).
+19. **Memory block vs node context budget**: with a small
+    `limits.maxContextChars`, the injected block is clipped into the room
+    actually remaining after system prompt/history/tool schemas; with no
+    meaningful room, the block is omitted and the run still succeeds rather
+    than exhausting its context. Repeat with a model whose discovered context
+    window is smaller than the graph default (`resolveModel` contextWindow):
+    the block must be sized against the finalized limit, not the default.
+20. **Orchestrator renamed**: use a custom graph whose orchestrator node has
+    `agentRole: "orchestrator"` under a different id (e.g. `intake`).
+    User-confirmed requirements/decisions must still be distilled; silently
+    missing them is a bug.
+21. **Old repositories**: a repo whose memory has >200 episodes; search for a
+    still-current fact sourced from an old episode. Expected: type, title and
+    provenance still resolve (metadata window widens automatically) instead
+    of degrading to `historical_fact`/`unverified`. Also probe a long-lived
+    session at the widest (5000-episode) window where another session writes
+    a new episode: the next search must refresh once and still resolve its
+    metadata.
+22. **Failed-outcome runs**: a graph routing a QA `fail` outcome to End still
+    reports run status `succeeded`, but must persist nothing. Probe that no
+    `bug_fix` memory appears after such a run.
+23. **Cancellation against a stalled service**: add latency to the mock
+    (e.g. 30s on `/search`), start a run, then Ctrl-C while retrieval is in
+    flight. Expected: the request aborts promptly (caller signal is combined
+    with the timeout), the run cancels without waiting out
+    `requestTimeoutMs`, and a `memory.error` event is logged.
+24. **Staleness on explicit search**: persist a memory with file/commit
+    provenance, modify the file, then ask the agent a question that makes it
+    call `memory_search`. Expected: the tool result carries
+    `requiresRevalidation: true` — same as automatic retrieval.
+25. **Many-commit provenance**: seed memories referencing more than three
+    distinct commits, then change a file tied to the fourth. Expected: that
+    memory is still flagged `requiresRevalidation` — every referenced commit
+    is diffed, not just the first few.
+26. **Malformed episode metadata**: hand-edit an episode's
+    `source_description` in the mock so `provenance.tests` is an object (or
+    `entities` contains non-strings). Expected: the fact degrades to
+    `historical_fact`/`unverified`; retrieval must not error out.
+27. **Partial write failure**: make the mock fail the second `POST /messages`
+    of a two-candidate distillation. Expected: the run record reports the
+    first id as persisted (`memory.distilled` persisted: 1) plus a
+    `memory.error` for the failed write — not zero persisted memories.
+28. **Duplicate rerun**: run the identical fix task twice. Expected: the
+    second run's distillation skips the write as a duplicate (no new episode,
+    no supersession of the existing current fact), even though search returns
+    the episode-body shape (title line + content + decorations).
+
+## Gaps and limitations
+
+- End-to-end scenarios above are designed to run against
+  `tests/mock-graphiti.mjs`; the mock implements Graphiti's REST routes
+  (`/healthcheck`, `/messages`, `/search`, `/episodes/:group`, `/group/:group`
+  DELETE) but not its semantic extraction, embeddings, or automatic temporal
+  invalidation. Scenario 7's CURRENT/HISTORICAL transition depends on the
+  supersession marker in the episode body plus the mock's expiry handling;
+  validating Graphiti's own temporal inference requires the real service.
+- Model-driven distillation quality (candidate wording, entity extraction) is
+  not asserted end-to-end; only structure and policy outcomes are.
+- Branch/worktree-scoped memory is stored as metadata (`branch`,
+  `workspaceId`) but retrieval does not filter by branch yet.
+
+## Validation evidence
+
+Automated (this PR, Windows 11 + Node 22):
+
+- `npx tsc --noEmit` — clean.
+- `node --test dist/test/memory-*.test.js dist/test/graphiti-provider.test.js
+  dist/test/config-memory.test.js` — 85/85 pass (provider, policy, retrieval,
+  distillation, isolation, security, supersession, degradation).
+
+Manual/proposed: scenarios 1–10 and edge cases 1–28 above are pending
+execution against the mock service; none have been run by a human yet.
+
+## Known pre-existing issues (not part of this feature)
+
+- On Windows, `test/cli.test.ts` "SIGTERM cancels an active shell..." fails
+  and can hang the runner, and `test/tools.test.ts` "shell timeouts and
+  cancellation terminate running command trees" times out; both reproduce at
+  git HEAD without the memory changes. Run `node --test
+  dist/test/memory-*.test.js dist/test/graphiti-provider.test.js
+  dist/test/config-memory.test.js` directly for the memory suite.

package/tests/harness-extensions-test-plan.md

@@ -0,0 +1,110 @@
+# Harness extension managers QA plan
+
+## Scope and prerequisites
+
+Verify `/skills`, `/plugins`, `/mcp`, `/logout`, persistence and next-task runtime behavior on Node 22+, including Windows PowerShell. Network access is required for GitHub installation. Use a dedicated worktree of this feature branch; keep the installed CLI checkout on `main`. Run `npm ci`, `npm run check`, `npm run build`, then `node dist/src/cli.js --cwd <fixture-workspace>` in an interactive terminal. Configure a test provider in `/providers` or place its provider config in the isolated home. Never use production credentials for fixture MCP servers. Live Codex sign-out requires the official Codex CLI and a disposable signed-in account.
+
+PowerShell isolation:
+
+```powershell
+$env:ANDORRACODE_HOME = Join-Path $env:TEMP ('andorra-extension-qa-' + [guid]::NewGuid())
+New-Item -ItemType Directory -Path $env:ANDORRACODE_HOME
+node dist/src/cli.js --help
+```
+
+Use placeholders for auth values. Create a local `example/SKILL.md` matching `docs/extensions.md` and `example/references/style.md` with a distinctive marker. Create a plugin with `andorra-plugin.json`, `skills/writing/SKILL.md`, and a known MCP fixture server. Keep originals outside the isolated home. Reset by exiting the CLI, verifying the resolved QA home is the isolated directory, deleting only that directory, and removing the session's `ANDORRACODE_HOME` environment override.
+
+## How to use the feature
+
+1. Start the built CLI, type `/`, select `/skills`, and choose List skills. Install the fixture directory, confirm, then inspect its row. Select it and choose Disable, return, then Enable.
+2. Submit a task using the fixture skill. Complete ordinary Orchestrator plan/research/clarification. Inspect tool details with Ctrl+O: `read_skill_resource` reads `SKILL.md` and its reference marker. Changes apply only on the next task.
+3. Open `/mcp`, choose Add, name it `fixture`, choose stdio, command `node`, and arguments `["<absolute-fixture-server-path>"]`. Enter `{}` environment. Submit a task, approve connection and then the external tool call. Observe the namespaced tool output and connection notice.
+4. Select the server to disable, edit or delete it. For HTTP choose the exact fixture `/mcp` endpoint and header object. Enter `${QA_TOKEN}` placeholders rather than tokens.
+5. Install the plugin directory with `/plugins`. Inspect its included skill/server availability in the next task, then disable the plugin and repeat. Uninstall to clean up copied content.
+6. With a disposable Codex provider configured, run `/logout codex`. Confirm sign-out in a subsequent official Codex account read, and use `/providers` to restore the test sign-in. Verify config fields remain.
+
+## Test cases and expected results
+
+| ID | Preconditions / steps | Observable expected result |
+| --- | --- | --- |
+| EXT-01 | Type `/`, browse all four commands, then `/help` | All commands appear and execute; Escape returns from management screens without mutations. |
+| EXT-02 | Install local example skill, list, toggle, restart CLI | Copied `skills/example/SKILL.md`; row and persisted disabled id reflect state; original files intact. |
+| EXT-03 | Enabled skill task reads SKILL.md and reference, then disable and start next task | Catalog and bounded reader expose enabled skill; disabled skill is absent and cannot be read. |
+| EXT-04 | Root Orchestrator tries reader before update_todos; then publishes plan and reads | First call rejected; second read allowed; skill read alone cannot satisfy repository research or user confirmation. |
+| EXT-05 | Install same id twice; malformed source; symlink; cancelled Git install | Existing installation unchanged; actionable failure; no visible partial install; cancellation returns to manager. |
+| EXT-06 | Install GitHub root and `/tree/main/skills/example` sources | Selected directory only installed; `.git` excluded; no package installer executed. |
+| EXT-07 | Read `../outside`, absolute outside, symlink escape or >128 KiB resource | Rejected with bounded failure; outside bytes never returned. |
+| EXT-08 | Add/edit/toggle/delete stdio and HTTP servers, restart | Config retains other fields; ids unique; JSON input validated; enabled state retained. |
+| EXT-09 | Approve actual stdio or HTTP fixture connection and tool; deny a subsequent call | Approved call produces fixture output; denied call does not reach fixture; tools names do not collide. |
+| EXT-10 | Server missing, malformed protocol, missing auth env or unreachable URL | Bounded unavailable notice, remaining task/servers can continue; no credential echoed. |
+| EXT-11 | Cancel active task/tool, close task; inspect fixture PID and HTTP sessions | Clients close and stdio process exits; tool responds cancelled; no persistent session leak. |
+| EXT-12 | Default engineer, read-only agent, restricted child and shell-enabled child with inherited tools | Engineer may call MCP; intake/read-only agents omit external tools. Every child omits task-owned MCP tools, including inherited shell-enabled roles, to preserve workspace isolation. |
+| EXT-13 | Plugin skill plus `${PLUGIN_ROOT}` MCP script; direct server shares same id | Both namespaced servers coexist; plugin assets resolve; disabling plugin removes only its contributions. |
+| EXT-14 | Corrupt installed bundle then list/start task/uninstall | Bundle shows unavailable, other tasks remain usable, uninstall repairs state. |
+| EXT-15 | Deny config write (controlled fake persistence test), try install/remove/toggle | No success reported; installed bundle rolled back; removed directory restored; old selected state retained. |
+| EXT-16 | `/logout codex` with a disposable Codex provider; mock transport failure; `/logout codex` with no Codex provider configured | Success exits, config retained; failure stays and reports; with no Codex provider it says so, API-key credentials are unchanged and the session stays. |
+| EXT-17 | Existing model/provider/graph/usage commands and CLI main checkout | Prior menus work; installation checkout remains main; no unrelated changes discarded. |
+| EXT-18 | Pause with Ctrl+C during MCP connection approval/discovery; issue `/status`, then submit new direction | Pending startup aborts and closes clients; slash commands do not resume; new direction retries startup with a fresh signal. Stop before startup completes creates no graph run. |
+| EXT-19 | Install skills/plugins with long Unicode names/descriptions and more than 16 skills; start task | Names/descriptions are bounded to 128/300 characters and 256/512 UTF-8 bytes; at most 16 skills and 12,000 catalog characters advertised. Omitted skills produce a notice and cannot be read through this task's reader. Disable unused skills to make room next task. |
+| EXT-20 | Swap a resource file or parent directory to an outside link during read | No outside bytes returned; no-follow open where supported plus opened-file identity and path rechecks reject changed resources. |
+| EXT-21 | MCP fixture returns large schemas/descriptions across pages and multiple servers | Advertised metadata stays within 16,000 UTF-8 bytes and 500 tools per task. Oversized server is rejected atomically with a notice and closed; earlier accepted servers and later small servers remain usable. |
+| EXT-22 | During an active task submit `/auth logout` and `/auth status`, then a filesystem path such as `/authentic/file needs a test` | Auth commands produce the Ctrl+C command hint without reaching the model; the path remains ordinary steering. |
+| EXT-23 | In graph editor configure a restricted filesystem-readable child role; select `read_skill_resource`, save and reopen. Repeat with an existing reader selection, no parent/child filesystem access, and a parent allowlist excluding the reader | Reader checkbox is available and retained for permitted roles; no unavailable warning for a valid saved reader. Missing filesystem capability or parent permission excludes the reader; graph validation agrees. |
+| EXT-24 | Enable ten empty-catalog MCP fixtures and insert a disabled entry first; start a task then close it. Repeat cancellation during connection approval | Only the first eight enabled servers are approved/started; a skip notice identifies the limit. Disabled entries do not consume attempts. Every started process exits on close; granting approval after cancellation starts no process. Failed/unreachable servers also count toward the eight-attempt limit. |
+| EXT-25 | After successful MCP discovery, start a waiting tool call and press Ctrl+C; submit new direction and invoke another tool, then cancel the task | Waiting call cancels; same stdio PID serves the resumed call without reconnecting. Task cancellation closes the process even without explicit cleanup. |
+| EXT-26 | Install GitHub repository root, `#tag`, and `/tree/ref/subdirectory`; include an executable plugin script | Root/ref/subdirectory content and ids match the selected source, no `.git` appears, executable permissions retained on POSIX. Only selected files are staged. |
+| EXT-27 | Supply archives over 64 MiB compressed, over 80 MiB expanded, over 64 MiB selected file content or over 10,000 selected paths (including implicit directories) | Download stream cancelled at transfer limit; compressed bombs rejected before extraction; entry/content limits validated before staging. Failure leaves no install or hidden staging directory. Repository-wide archive caps also apply to subtree installs; a local checkout remains available. |
+| EXT-28 | Supply a malformed tar checksum, traversal path, selected symlink, duplicate file, or cancel during download | Actionable rejection/cancellation, closed response stream, no published bundle or staging residue; outside files unchanged. |
+
+| EXT-29 | Populate each extension directory with 140 visible bundles, then list and start a task | At most 128 entries per directory validated in name order, at most four concurrent validations; manager and runtime report 12 omitted entries per directory. Runtime skill catalog remains limited to 16. Remove unused folders to expose omitted entries next scan. Cancellation stops queued validations. |
+| EXT-30 | Set fake unrelated and source variables in the CLI; map only the source using a distinct configured stdio env key | Real stdio fixture sees neither unrelated nor source variable, but sees the mapped value. Server still starts with minimal platform environment; notices contain no fake values. Do not use real credentials for this case. |
+| EXT-31 | Select the configured home or an ancestor as a local install source; repeat through a directory alias | Immediate ancestor-source error before recursive content reads; no publication or staging residue. An unrelated local source still installs. |
+| EXT-32 | Configure a shell-enabled child with `workspace: "worktree"` and `toolInheritance: "inherit"`, then inspect its tool catalog | Parent retains MCP tool; child catalog omits it. Child cannot call the parent-scoped server or mutate the parent via that MCP client. Built-in workspace tools retain their normal capability rules. |
+| EXT-33 | In an isolated home containing literal `$&`, `$\`` and `$'` path components, install a plugin whose stdio command, repeated args and environment use `${PLUGIN_ROOT}` | Each occurrence expands to the exact installed path, including all literal dollar metasequences. No `${PLUGIN_ROOT}` fragment remains from replacement expansion. |
+| EXT-34 | Use a plugin manifest with 5,000 copies/aliases of one skill path, then with 17 unique paths. Cancel listing, snapshot revalidation and installation during the first nested document read | Repeated resolved paths validate once per pass and advertise one skill. More than 16 unique directories fails before reading any skill document; installation publishes nothing and manager shows unavailable. Cancellation rejects immediately after the in-flight filesystem operation, closes its handle, stops subsequent skill reads and removes installation staging. |
+
+## Edge cases and potential bugs
+
+| Case | Steps | Expected result |
+| --- | --- | --- |
+| EXT-36 | Install a plugin with eight unique MCP declarations (one disabled); repeat with nine and 3,000 declarations, invalid server entries and a missing skill path | Eight declarations are accepted and seven enabled servers enter the snapshot. Excessive declarations reject before individual server validation or nested skill reads; manager shows unavailable, runtime contributes no servers, failed installation leaves no staging residue. Disabled server declarations count toward the cap. |
+| EXT-37 | Create 128 disabled skills and 128 disabled malformed plugins before an enabled skill/plugin in name order; start task, then open each manager | Runtime opens only enabled bundle documents, advertises the enabled skill/server, and emits no disabled-bundle unavailable or omission notice. Disabled bundles consume no runtime discovery slots. Managers still open and validate the first 128 visible bundles, showing disabled skills and unavailable disabled plugins with omission notice for the final enabled entry. |
+| EXT-38 | Configure a local HTTP fixture endpoint with `?api_key=fake-query-secret&tenant=fixture`; approve connection, invoke tool, close task | Approval shows endpoint without query; notices contain no fake query/header credentials. Initialization, discovery, invocation and DELETE cleanup retain the full query on the wire. Use only fake credentials. |
+| EXT-39 | Set isolated home to a literal `home-${ANDORRA_MCP_ROOT_COMPONENT}` path. Install a plugin with stdio env value `${PLUGIN_ROOT}|${QA_TOKEN}`. Run with the component variable unset, then set to a different value; use a fake QA token | Both connections succeed. The child receives the exact installed path including the literal component, followed by the expanded fake QA token. Inserted paths are never expanded again. Command/argument root replacement and explicit environment isolation remain intact. |
+| EXT-40 | Install a GitHub repository or subtree named `bad.skill`, then a local directory with an invalid basename (including over 100 characters) | ID validation rejects before any archive request or recursive scan/copy. No bundle/config mutation or staging residue remains. Existing valid local/GitHub installs still succeed. Automated fixtures count download, traversal and copy calls for invalid names. |
+| EXT-41 | On POSIX create malformed entries `:install`, `:list`, and `:bundle:normal` in each extension directory. Open the manager and select each row | Each row opens its own bundle actions, including Uninstall; ordinary Install/List actions remain distinct. Uninstall removes the selected entry transactionally. Windows regression injects these directory names to verify menu routing; actual POSIX removal remains proposed QA work. |
+| EXT-42 | With YOLO off and an enabled MCP server, start a task. At connection approval enter `/yolo on` without choosing Approve | The pending chooser settles immediately, connection proceeds and the task reaches inference. Later `/yolo on` does not resolve the old chooser. Denial, cancellation and pause/retry remain covered by existing tests. The automated CLI fixture mocks SDK connection; real terminal keyboard interaction remains proposed QA work. |
+
+Duplicate ids fail instead of replacing bundles. Git tree refs containing slashes require a local checkout; refs beginning with `-` fail. Installer bounds prevent large bundles; link/special-file rejection is intentional. Cancellation during staging rolls back before persistence; uninstall cleanup failure reports a hidden staging directory remaining. Configuration rename is atomic but does not coordinate two concurrent CLI writers; last successful writer wins, so avoid concurrent manager edits. Headers/env placeholders need the variable present in the CLI's environment. HTTP auth redirects are rejected. MCP tools are sequential and receive existing permission/cancellation gates; external server behavior is outside the filesystem sandbox. Read-only server annotations cannot grant additional capabilities. Custom graph explicit tool lists exclude dynamic MCP tools, and restricted research subagents do not inherit them by default.
+
+## Gaps and limitations
+
+Live authenticated third-party MCP services, real GitHub download/cancellation, physical terminal walkthrough and real account sign-out are proposed manual checks unless explicitly listed below. Automated fixtures validate the protocol/runtime without signing out the user's account. Legacy SSE transports, MCP resources/prompts/OAuth login UI, marketplace discovery, arbitrary plugin entry points and Codex plugin manifest compatibility are unsupported. Source changes affect subsequent tasks. No known reproduced defects remain after automated checks; future manual results must be added here.
+
+## Validation evidence
+
+2026-10-02 fifth follow-up review: `npm run check` and `npm run build` passed on Windows PowerShell / Node v24.16.0. `node scripts/run-tests.mjs dist/test/extensions.test.js dist/test/extensions-publication.test.js dist/test/extensions-github.test.js dist/test/extensions-runtime.test.js dist/test/mcp.test.js` passed 44/44 with no exclusions. New regressions cover eight accepted versus nine/3,000 rejected MCP declarations, disabled server counting, rejection before invalid entry/missing skill validation, 256 disabled bundles skipped without document opens during runtime but fully validated in managers, and query redaction with full URL preservation across real HTTP initialization/discovery/invocation/cleanup. Full suite and manual terminal/account/third-party MCP checks were not rerun; prior baseline failures remain outside this remediation.
+
+2026-10-02 fourth follow-up review: `npm run check` and `npm run build` passed on Windows PowerShell / Node v24.16.0. The affected suites passed 81/81 with no exclusions using `node scripts/run-tests.mjs dist/test/extensions.test.js dist/test/extensions-publication.test.js dist/test/extensions-github.test.js dist/test/extensions-runtime.test.js dist/test/mcp.test.js dist/test/command-screens.test.js dist/test/agent-execution.test.js`. New regressions track document handles across 130 skills and 130 plugins (128 reads per directory, at most four concurrent open documents), verify omission notices, drain in-flight reads on cancellation, reject canonical ancestors before scanning, inspect fake environment variables in a real stdio process, and verify parent MCP tools are absent from a shell-enabled inherited child catalog. The inherited-child runtime fixture uses a read-only role under the worktree policy; built-in writing-child worktree isolation also passed in the existing agent-execution suite. Full suite and physical terminal/account workflows were not rerun for these narrow fixes; previous baseline failures remain documented below.
+
+Environment: Windows PowerShell, Node v24.16.0; isolated feature worktree and temporary ANDORRACODE_HOME. `npm run check` and `npm run build` passed. Focused changed-feature/regression suite: 167/167 passed (menus, auth, TUI, commands, graph revisions/schema, timeout behavior, skill resources and actual MCP transports). Two additional committed hardened-runtime regressions passed: pre-plan skill read rejected with zero underlying reads, post-plan read allowed, and default engineer external-tool execution. Latest extension/runtime/transport rerun: 12/12 passed, including real stdio and SDK HTTP sessions, paginated discovery, approval side effects, cancellation/PID exit, explicit HTTP DELETE cleanup, auth environment references and file-boundary checks. A live isolated GitHub install of `https://github.com/openai/skills/tree/main/skills/.system/skill-creator` succeeded, then read SKILL.md through the bounded tool; fixture home was removed.
+
+Full `npm test` initially ran 963 cases: 916 passed, 42 failed, 5 skipped. Pristine `main` at `9fe7f5d` independently ran 944: 897 passed, 42 failed, 5 skipped. Forty failure titles matched, largely existing Windows memory/CodeGraph assumptions and CLI/process timeout behavior. One new revision-3 assertion was updated for revision 4 and its focused checks passed. The feature-only headless Orchestrator timeout passed an isolated rerun (14.8 seconds); pristine baseline also passed it in isolation (15.4 seconds). Baseline-only CLI selector/provider cases varied under concurrent load. Existing two metadata fallback failures reproduced on pristine main in isolation. The complete suite is not clean; no claim is made that all failures are fixed. Remaining manual cases are pending as described above.
+
+After merging current `main` at `ed43e60`, `npm run check` and `npm run build` passed. Final focused suite passed 121/121, including all five Codex review regressions (bounded empty-page pagination, editor discovery, subagent skill permissions, publication cancellation, and invalid-entry removal), plus account/provider logout separation. The updated slash-completion regression also passed separately. The merged full attempt ran 1,926 cases: 1,840 passed, 79 failed, 1 cancelled, 6 skipped; two stalled test workers (`cli-context-persistence` and `unbounded-runs`) were stopped after pristine main independently reproduced both hangs. A pristine `ed43e60` run excluding those two files ran 1,892: 1,809 passed, 76 failed, 1 cancelled, 6 skipped. Every other failure title matched, except the new slash test's `/lo` ambiguity with upstream `/login`; that test now uses `/logou` and passed. The full suite remains blocked by upstream failures and hangs; it was not rerun after the final narrow review fixes.
+
+2026-10-02 follow-up review: serialized `npm run check` and `npm run build` passed. Extension, publication, runtime and MCP suites passed 21/21; CLI preflight cancellation passed 3/3, including the new pause/abort/retry MCP approval case. Deterministic filesystem testing swapped a parent junction at open and restored it before verification; the outside handle was rejected. Large Unicode metadata and multi-server paginated schemas exercised the catalog budgets. Broader CLI live-control/run-control checks ran 11 cases: 9 passed, 2 failed (usage-counter assertion and nested-child tool-selection assertion); both identical failures reproduced in the existing pristine `ed43e60` baseline. Initial parallel generation attempts collided in ignored engine runtime files; serialized checks passed afterward. One follow-up rerun timed out at the new test's original 15-second limit during unrelated startup; the fixture now disables CodeGraph and allows 60 seconds while retaining the same approval assertions. Full suite and manual device/terminal/account workflows were not rerun for this narrow follow-up.
+
+2026-10-02 second follow-up review: `npm run check` and `npm run build` passed on Windows PowerShell / Node v24.16.0. The affected extension, MCP, graph-editor/server/validation and terminal suites passed 221/221 after excluding two independently reproduced baseline tests: expanding one prompt releases capture with another collapsed prompt visible, and collapsed prompt disclosure during resize. Both failed with identical assertions on the existing pristine main worktree at ed43e60. New coverage verifies active-run multiword auth commands never steer the model, the restricted child skill-reader checkbox respects parent and child permissions, eight enabled MCP startup attempts including empty catalogs, process cleanup, and cancellation while awaiting approval. Full suite and manual terminal/account workflows were not rerun for this narrow follow-up.
+
+2026-10-02 bounded GitHub download follow-up: compiled extension, publication and new GitHub archive suites passed 17/17 on Windows / Node v24.16.0. Fixtures exercise root/default-ref, explicit tag, selected subtree, compressed-transfer cancellation (both declared length and chunked body), an 81 MiB gzip bomb, selected-content overflow, excessive implicit directories, traversal, symlinks, duplicate paths, PAX long filenames, corrupt checksums and cancellation cleanup. A live isolated `https://github.com/openai/skills/tree/main/skills/.system/skill-creator` archive install succeeded (18,664-byte SKILL.md, catalog available, no `.git` or staging residue). Live root `https://github.com/openai/skills` downloaded and parsed, then correctly failed because that root lacks SKILL.md; its staging directory was removed. Temporary homes were removed. Successful live root installation is covered by the archive fixture, not a live root bundle. POSIX executable-mode preservation has a committed assertion but was not executed on Windows. Private repositories require an authenticated local checkout; archive download sends no credentials. Repository-wide 64 MiB compressed/80 MiB expanded limits apply before extraction, including unselected content, and archive paths are capped at 32 KiB.
+
+2026-10-02 third follow-up review: `npm run check` and `npm run build` passed on Windows PowerShell / Node v24.16.0. The affected extension/publication/runtime, MCP, CLI preflight cancellation and run-control suites passed 36/36 using `node scripts/run-tests.mjs '--test-skip-pattern=nested child keeps' dist/test/extensions.test.js dist/test/extensions-publication.test.js dist/test/extensions-runtime.test.js dist/test/mcp.test.js dist/test/cli-preflight-cancellation.test.js dist/test/run-control.test.js`. The excluded nested-child tool-selection test failed identically on the existing pristine ed43e60 baseline (`Tool error: Subagent tool selection exceeds its parent`); it remains outside this fix. A real stdio server verified pause cancellation, same-process resumed tool execution, and task-abort cleanup. Full suite and physical terminal/account workflows were not rerun.
+
+## Transport response regression (EXT-35)
+
+Use a local stdio fixture that writes over 1 MiB without a newline, then HTTP fixtures returning an oversized declared body, chunked JSON, and an unfinished SSE data event. Discover a tool normally first, then invoke it. Expected: pending call rejects, stdio process exits, HTTP producer is cancelled before completing its 64 MiB payload, and no oversized document is parsed or included in the transcript. A normal bounded JSON/SSE response still succeeds. The cap applies cumulatively to long-lived HTTP streams; reconnect by starting a new task. Automated coverage: test/mcp.test.ts. Live third-party servers and long-running SSE compatibility remain proposed QA work.
+
+2026-10-02 fourth follow-up review: npm run check and npm run build passed on Windows PowerShell / Node v24.16.0. Compiled extension, publication, GitHub archive, runtime and MCP suites passed 42/42 with no exclusions, including seven new regressions for literal plugin roots, deduplicated/capped skills, nested cancellation and transport response limits. Real subprocess and HTTP fixtures verified producer/process cleanup. Full suite, live third-party MCP services and manual terminal/account workflows were not rerun; previously documented baseline failures remain outside this remediation.
+
+2026-10-02 fifth follow-up review: npm run check, npm run build and git diff --check passed on Windows PowerShell / Node v24.16.0. The affected extension, publication, GitHub archive, runtime, MCP, manager and CLI preflight suites passed 81/81 with no exclusions. New regressions cover literal placeholder-bearing plugin roots at real stdio connection, early invalid ID rejection before download/traversal/copy, collision-free malformed bundle menu routing, and live /yolo on settling startup approval with stale callback cleanup. EXT-39 through EXT-42 document expected behavior and manual gaps. An additional unchanged cli-live-control test failed its request-count assertion identically on the clean existing ed43e60e6d33897ff6c334214cc77744a20f2eaf baseline (expected 0,1,1,1,3; actual 1,2,3,3,6). It remains outside remediation. Full suite, actual POSIX malformed-entry deletion, physical terminal input, third-party services and account sign-out were not rerun.