@voybio/ace-swarm 2.4.4 → 2.4.6

package/docs/state-model.md

@@ -0,0 +1,113 @@
+# ACEPACK State Model
+
+> Companion to the [ACEPACK summary in the README](../README.md#acepack-state-model). This document describes the implemented v2 format, its authority boundary, and safe maintenance behavior.
+
+ACE's canonical generated workflow state is stored in `agent-state/ace-state.ace`. Older workspaces may still contain `.agents/ACE/ace-state.ace`; that path is a read fallback, not a second authority. If both files exist, `ace fsck` reports a dual-store conflict so the operator can migrate deliberately instead of guessing which file to delete.
+
+One intentional exception remains: active goal records and plans under `agent-state/goals/` are authoritative loose files so detached goal processes can coordinate without rewriting the packed store. A complete backup therefore includes both `agent-state/ace-state.ace` and `agent-state/goals/` when that directory exists.
+
+## On-disk regions
+
+ACEPACK v2 has four regions:
+
+```text
+ace-state.ace
+├── Header (128 bytes, big-endian)
+│     magic · version · flags
+│     kv_index_offset · kv_index_length · kv_chunk_end
+│     evt_offset · evt_length · evt_count · evt_base_id
+├── KV Chunk Region
+│     knowledge/agents/{name}/{file}  → agent instruction text
+│     knowledge/skills/{name}/{file}  → skill content
+│     topology/{kind}                 → swarm registry, route tables
+│     state/{...}                     → handoffs, todo, ledger, scheduler
+│     meta/{...}                      → schema version, project name
+│     each chunk: [uint32 length][bytes]
+├── Columnar Event Section (rewritten on commit)
+│     uint32    event count
+│     int64[N]  timestamps      (epoch ms, big-endian)
+│     uint8[N]  kinds           (EntityKind enum)
+│     uint8[N]  sources         (ContentSource enum)
+│     uint8[N]  flags           (0x01 = deleted)
+│     padding to 4-byte alignment
+│     uint32[N] payload offsets (relative to payload pool)
+│     uint32[N] payload lengths
+│     uint32    pool length
+│     bytes[M]  payload pool    (UTF-8 JSON blobs)
+└── KV Index (JSON, rewritten on every store write)
+      key → { offset, length, crc? }
+```
+
+The header field definitions live in `src/store/types.ts`; serialization and bounds checks live in `src/store/ace-packed-store.ts` and `src/store/fsck.ts`.
+
+## KV reads and writes
+
+The KV index makes a single-key read an indexed seek plus the selected chunk read: approximately `O(chunk_length)` after the index is loaded. CRC-bearing entries are verified on read, and `ace fsck` also verifies region ordering, length prefixes, bounds, and CRCs.
+
+The writer is append-structured, but the file is **not appended in place**:
+
+- Synchronous operational writes retain old KV chunks, append replacement chunks in the new KV region, copy the current event section, write a new index, and atomically rename a complete temporary file.
+- Async `AcePackedStore.commit()` rebuilds the live KV region, serializes the complete hot event set, writes a complete temporary file, and atomically renames it.
+- `compact()` similarly rewrites the complete container with only live KV chunks.
+
+Consequently, filesystem write cost is proportional to the retained KV bytes plus hot event bytes and the index—not `O(1)`. Atomic rename prevents readers from observing a partially promoted file on supported local filesystems, but it also changes the inode. Writers are serialized through the store write coordinator; bypassing that coordinator is unsupported.
+
+## Columnar events
+
+The physical event layout separates fixed-width columns from JSON payloads and uses about 19 fixed bytes per hot event plus payload bytes. The current JavaScript reader eagerly loads and parses hot-event payloads when opening/scanning a store, so the layout's theoretical timestamp/kind-only scan is not yet exposed by the production API. Do not size current scans as 9 bytes per event.
+
+Events survive `compact()`. `archive/events/*` may contain older contiguous event batches compressed with BGZF and indexed by `archive/events/__index`; `AcePackedStore.archiveColdEvents()` implements this tier, while the normal CLI does not invoke it automatically.
+
+## Authority and projections
+
+The packed store is authoritative for generated operational artifacts, agent instructions, skills, topology, schemas, scheduler state, handoffs, todos, run ledger, and event history. Host-facing files are materialized only through explicit projection allowlists.
+
+Exceptions and boundaries:
+
+- `agent-state/goals/*.json`, plan JSON, and detached goal logs are authoritative loose goal state.
+- Source code and user-authored project files remain ordinary workspace files.
+- `.agents/ACE/ace-state.ace` is legacy fallback state and should not coexist indefinitely with the canonical file.
+- `ACE_STORE_FIRST=0` is a legacy diagnostic escape hatch; it can expose stale disk projections and is not the normal precedence.
+
+## Maintenance and recovery
+
+### Inspect before changing
+
+```bash
+ace fsck --json
+ace ls agent-state
+ace read agent-state/STATUS.md --source
+```
+
+`ace fsck` diagnoses binary structure, CRCs, stale write leases, orphan temporary files, and dual-store conflicts. `--repair` removes only repairable stale/orphan coordination artifacts; it does not reconstruct corrupt application data.
+
+### Compact
+
+```bash
+ace compact --no-sweep  # store rewrite only
+ace compact             # store rewrite plus loose generated-artifact sweep
+```
+
+The default command runs a sweep after compaction: eligible loose generated ACE artifacts are ingested into the store and removed from disk. Use `--no-sweep` when only the container rewrite is intended. Back up the store and active goal directory before maintenance on a critical workspace.
+
+### Repair
+
+`ace repair` re-bakes packaged assets, repairs known store-backed projections, and restores host files. It is not a general corrupt-ACEPACK byte repair tool. Review every warning and rerun `ace fsck`; recover unrecoverable corruption from a known-good backup or a quarantined pre-write copy.
+
+When a write refuses corrupt source bytes, ACE leaves the original in place and writes a timestamped `.corrupt-*` quarantine copy. Do not delete either copy until the cause and recovery source are established.
+
+### Migrate
+
+`ace migrate [store-path]` imports the supported legacy ACE state families: handoffs, todos, run-ledger entries, status events, and Vericify posts. It is not an arbitrary workspace-file importer. Verify the resulting canonical store with `ace fsck`, then archive the legacy store rather than leaving both authorities active.
+
+## Backup contract
+
+For a recoverable snapshot:
+
+1. Stop active ACE writers and detached goal workers.
+2. Copy `agent-state/ace-state.ace`.
+3. Copy `agent-state/goals/` if present.
+4. Record `ace fsck --json` output and the store checksum.
+5. Preserve any `.corrupt-*` quarantine until recovery is verified.
+
+Restoring only the `.ace` file while omitting active loose goal records is not a complete workspace-state restore.

package/docs/troubleshooting.md

@@ -0,0 +1,95 @@
+# Troubleshooting
+
+Start with the smallest command that distinguishes configuration, packed-state, provider, and model failures:
+
+```bash
+ace paths
+ace fsck
+ace doctor --scan
+```
+
+## Store and bootstrap
+
+### No ACE store found
+
+Run `ace init` or `ace turnkey` in the intended workspace. Confirm `ace paths` reports that directory as `WORKSPACE_ROOT`.
+
+### Canonical and legacy stores both exist
+
+`ace fsck` reports `store:legacy-dual-store`. Do not delete either file reflexively. Back up both, inspect their logical state with `ace ls`/`ace read`, migrate deliberately, verify the canonical store, then archive the legacy copy.
+
+### Store/disk copies differ
+
+Use full values, not abbreviations:
+
+```bash
+ace read agent-state/STATUS.md --source
+ace read agent-state/STATUS.md --prefer store
+ace read agent-state/STATUS.md --prefer disk
+```
+
+Store-first is normal. `ACE_STORE_FIRST=0` is a legacy diagnostic escape hatch.
+
+### Corruption or quarantine
+
+Run `ace fsck --json`. A `.corrupt-*` file is a preserved copy, not disposable litter. `ace repair` re-bakes known assets and projections; it does not reconstruct arbitrary corrupt bytes. Restore from a verified backup when structural checks cannot be repaired safely.
+
+## Providers
+
+### Ollama unreachable or model missing
+
+```bash
+ollama serve
+ollama pull <model>
+ace doctor --llm ollama --model <model> --base-url http://127.0.0.1:11434
+```
+
+`--repair-ollama` pulls a missing model and re-probes availability. Treat a nonzero doctor result as unresolved.
+
+### llama.cpp endpoint mismatch
+
+Use the OpenAI-compatible base, normally `http://127.0.0.1:8080/v1`, and check its model list before starting ACE.
+
+### Hosted provider is discovered but chat fails
+
+Discovery does not prove credentials. Confirm the provider's API key/base URL and run a provider-specific `ace doctor` command.
+
+## Hermes
+
+### Missing Python or Hermes checkout
+
+Set `ACE_HERMES_ROOT` and either `HERMES_PYTHON`, `ACE_HERMES_UV_PROJECT`, or the advanced `ACE_HERMES_COMMAND_JSON`. Run `ace doctor ... --hermes` before the TUI or live driver.
+
+### Live driver writes a report but no artifact
+
+Inspect each report row's `status`, `job_done`, `protocol_done`, and `tool_write_done`. Any failed selected row makes the driver exit nonzero. A report path alone is not success.
+
+### Hung turn
+
+Interrupt the TUI with `Ctrl+C`. Hermes-local has an executor-owned turn timeout and child termination path; if the process survives, capture diagnostics and stop the session rather than launching a second concurrent mutation.
+
+## Goals
+
+### Goal remains planning or reports `contract_uncheckable`
+
+Declare a required artifact or verification command:
+
+```bash
+ace goal start "Implement the change" --verify "npm test"
+```
+
+### Detached goal failure
+
+Use `ace goal status <goal_id>` and inspect `agent-state/goals/<goal_id>.log`. Active goal files are authoritative loose state and must be included in backups.
+
+## Evaluation
+
+The packaged battery is mutating. `ace eval` creates isolated per-run workspaces and records provenance; do not point an explicit unsafe-workspace option at a valuable checkout. See [Evaluation Safety and Reproducibility](evaluation-safety.md).
+
+## Package and development
+
+If `npm pack` fails under `~/.npm/_cacache` with ownership errors, use a writable temporary npm cache for diagnosis. Do not change ownership or delete a shared cache without understanding the host policy.
+
+## Escalation evidence
+
+Include the exact command, exit code, relevant doctor/fsck checks, provider/model/base URL with secrets removed, and the logical artifact paths retrievable with `ace read`.

package/docs/tui-operator-guide.md

@@ -0,0 +1,75 @@
+# TUI Operator Guide
+
+The ACE terminal UI is an interactive operator surface over the same workspace state used by the MCP server.
+
+## Start and preflight
+
+Run doctor first so provider failures are separated from display or session failures:
+
+```bash
+ace doctor --scan
+ace tui
+```
+
+Pin a provider when auto-discovery is ambiguous:
+
+```bash
+ace tui --provider ollama --model qwen3:8b --base-url http://127.0.0.1:11434
+ace tui --provider llama.cpp --model local-model --base-url http://127.0.0.1:8080/v1
+```
+
+For Codex, Claude, Gemini, and Copilot credentials, endpoints, default models, and provider-specific doctor commands, follow the [hosted provider quick setup](provider-security.md#hosted-provider-quick-setup) before starting a chat.
+
+Hermes-local is opt-in and has additional prerequisites; see [Hermes Companion Engine](hermes.md).
+
+## Input modes
+
+- In normal mode, typing a printable character enters chat mode and preserves that character.
+- `/` enters command mode.
+- `Esc` returns to normal mode.
+- In chat mode, a leading `/command` is routed to the command registry. Prefix with `//` to send a literal slash to the model.
+- `Tab` and `Shift+Tab` cycle tabs.
+- `Alt+1` through `Alt+9` switch directly to a tab.
+
+Dashboard controls use Up/Down to select, Left/Right to change a value, and Enter to apply.
+
+## Global shortcuts
+
+| Shortcut | Action |
+|---|---|
+| `Ctrl+C` | Interrupt the active stream; it does not quit an idle TUI |
+| `Ctrl+D` | Quit |
+| `Ctrl+N` | Open a chat tab |
+| `Ctrl+R` | Refresh |
+| `Ctrl+L` | Clear the current view |
+| `Ctrl+W` | Close the current tab |
+| `Ctrl+Z` | Suspend; resume with the shell's foreground command |
+
+## Commands
+
+Run `/help` for the authoritative command list. Current command families include:
+
+- Provider/model: `/provider`, `/providers`, `/model`, `/models`, `/pull`, `/ollama`
+- Agents/tasks: `/agent`, `/agents`, `/kill`, `/task`, `/tasks`, `/handoff`
+- Approval/config: `/approve`, `/reject`, `/trust`, `/config`, `/theme`
+- Skills: `/skill`, `/skills`
+- Tabs/display: `/chat`, `/logs`, `/close`, `/archive`, `/clear`
+- Lifecycle: `/status`, `/help`, `/quit`
+
+Command completion is available while typing a slash command.
+
+## Provider switching
+
+`/provider <name>` changes the active provider and `/model <name>` changes the active model. A provider shown by discovery is not necessarily authenticated or reachable. Run `ace doctor` for the selected provider after changing credentials or endpoints.
+
+## Failure and recovery
+
+- If chat is rejected before a request, read the on-screen configuration diagnosis and run its suggested `ace doctor` command.
+- If a direct stream stalls, `Ctrl+C` requests interruption. Confirm that the UI leaves streaming state before sending another task.
+- Hermes-local owns a subprocess and uses its own timeout/abort path; a terminal result must contain a valid final frame and pass completion evidence checks.
+- If the UI display is stale, use `Ctrl+R`; if the underlying state is suspect, exit and run `ace fsck`.
+- Session state survives in ACEPACK, but an unverified model response is not executable authority.
+
+## Non-interactive alternative
+
+Use the CLI or MCP tools for automation. The TUI is interactive and should not be used as a CI success signal.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@voybio/ace-swarm",
-  "version": "2.4.4",
+  "version": "2.4.6",
   "description": "ACE Framework MCP server and CLI — single-file ACEPACK state, local-model serving, agent orchestration, and host compliance enforcement.",
   "type": "module",
   "main": "dist/index.js",
@@ -22,7 +22,9 @@
     "dist/**/*.js",
     "dist/**/*.d.ts",
     "scripts/hermes_bridge_worker.py",
+    "scripts/dev/run_hermes_crispr_live.mjs",
     "assets",
+    "docs",
     "README.md",
     "CHANGELOG.md"
   ],

package/scripts/dev/run_hermes_crispr_live.mjs

@@ -0,0 +1,212 @@
+#!/usr/bin/env node
+
+import { existsSync, mkdirSync, readFileSync, writeFileSync } from "node:fs";
+import { resolve } from "node:path";
+import { pathToFileURL } from "node:url";
+
+const workspaceRoot = process.env.ACE_WORKSPACE_ROOT || resolve(process.cwd(), "..");
+const packageRoot = process.cwd();
+const distRoot = resolve(packageRoot, "dist");
+const reportDir = resolve(workspaceRoot, "agent-state", "hermes-crispr-live");
+mkdirSync(reportDir, { recursive: true });
+
+const timestamp = new Date().toISOString().replace(/[-:]/g, "").replace(/\..+$/, "Z");
+const reportPath = resolve(reportDir, `${timestamp}.json`);
+const ollamaArtifact = "agent-state/hermes-crispr-live/crispr-ollama.html";
+const llamaArtifact = "agent-state/hermes-crispr-live/crispr-llama-cpp.html";
+const pythonCommand =
+  process.env.HERMES_PYTHON;
+if (!pythonCommand) {
+  console.error("HERMES_PYTHON is required; point it at the Python environment containing the operator-supplied Hermes checkout.");
+  process.exit(2);
+}
+
+const { runLocalModelTask } = await import(pathToFileURL(resolve(distRoot, "local-model-runtime.js")).href);
+const { HermesSubprocessExecutor } = await import(pathToFileURL(resolve(distRoot, "hermes", "session-manager.js")).href);
+
+function htmlTask({ provider, model, artifact }) {
+  if (process.env.ACE_LIVE_OUTPUT_ONLY === "1") {
+    return [
+      "Output only a complete self-contained HTML document about CRISPR.",
+      "Start with <!DOCTYPE html> and end with </html>.",
+      "No markdown fences. No explanation. No tool code.",
+      `Runtime label to include visibly: ${provider} / ${model}`,
+      "Include inline CSS in <style> and inline JavaScript in <script>.",
+      "Include short sections: mechanism, applications, risks/ethics.",
+      "Include a small interactive quiz or toggle implemented in JavaScript.",
+      "No external assets, CDNs, or network URLs.",
+      "Keep it concise.",
+      `ACE will save your output to ${artifact}.`,
+    ].join("\n");
+  }
+
+  return [
+    "You are running as an ACE coder in Hermes-local mode.",
+    `Provider: ${provider}`,
+    `Model: ${model}`,
+    "",
+    "Create a concise, complete, self-contained HTML file about CRISPR.",
+    `Use the ACE MCP tool write_workspace_file to write exactly this workspace path: ${artifact}`,
+    "",
+    "Hard requirements for the file:",
+    "- It must be a single .html document with inline CSS and inline JavaScript.",
+    "- It must include sections on mechanism, applications, risks/ethics, and a short interactive quiz or toggle implemented in JavaScript.",
+    "- It must visibly identify which runtime produced it.",
+    "- It must not reference external assets, CDNs, or network resources.",
+    "- Keep it under 180 lines.",
+    "- Return completed only after write_workspace_file succeeds, and cite the tool evidence.",
+    "- If the tool is unavailable, output only the full HTML document starting with <!DOCTYPE html> and ending with </html> so ACE can materialize it for inspection.",
+  ].join("\n");
+}
+
+function artifactStats(relativePath) {
+  const absolutePath = resolve(workspaceRoot, relativePath);
+  if (!existsSync(absolutePath)) {
+    return { path: relativePath, exists: false };
+  }
+  const content = readFileSync(absolutePath, "utf-8");
+  return {
+    path: relativePath,
+    exists: true,
+    bytes: Buffer.byteLength(content),
+    hasHtml: /<html[\s>]/i.test(content),
+    hasCss: /<style[\s>]/i.test(content),
+    hasJs: /<script[\s>]/i.test(content),
+    mentionsCrispr: /crispr/i.test(content),
+    externalRefs: [...content.matchAll(/\b(?:https?:)?\/\//gi)].length,
+  };
+}
+
+function extractHtml(text) {
+  if (typeof text !== "string") return "";
+  const start = text.search(/<!doctype html|<html[\s>]/i);
+  const end = text.search(/<\/html>/i);
+  if (start === -1 || end === -1 || end <= start) return "";
+  return text.slice(start, end + "</html>".length);
+}
+
+function materializeHermesHtmlIfNeeded(row, relativePath) {
+  const stats = artifactStats(relativePath);
+  if (stats.exists && !row.artifact_preexisting) return stats;
+  const html = extractHtml(row.summary);
+  if (!html) return { ...stats, stale: Boolean(row.artifact_preexisting) };
+  writeFileSync(resolve(workspaceRoot, relativePath), html, "utf-8");
+  row.materialized_from_hermes_final = true;
+  row.materialization_note =
+    "Hermes/local model produced a complete HTML payload but did not execute the ACE write tool; the live driver materialized the final payload so the job artifact can be inspected separately from tool-call success.";
+  return artifactStats(relativePath);
+}
+
+const selectedProviders = String(process.env.ACE_LIVE_PROVIDERS || "ollama,llama.cpp")
+  .split(",")
+  .map((entry) => entry.trim())
+  .filter(Boolean);
+
+const runs = [
+  {
+    label: "ollama",
+    provider: "ollama",
+    model:
+      process.env.ACE_LIVE_OLLAMA_MODEL ||
+      "hf.co/Qwen/Qwen2.5-Coder-1.5B-Instruct-GGUF:Q2_K",
+    baseUrl: process.env.OLLAMA_HOST || "http://127.0.0.1:11434",
+    artifact: ollamaArtifact,
+  },
+  {
+    label: "llama.cpp",
+    provider: "llama.cpp",
+    model: process.env.ACE_LIVE_LLAMA_CPP_MODEL || "gemma-4-E2B-it-UD-Q2_K_XL.gguf",
+    baseUrl: process.env.ACE_LIVE_LLAMA_CPP_BASE_URL || "http://127.0.0.1:8080/v1",
+    artifact: llamaArtifact,
+  },
+].filter((run) => selectedProviders.includes(run.provider) || selectedProviders.includes(run.label));
+
+const report = {
+  schema: "ace-hermes-crispr-live@1",
+  generated_at: new Date().toISOString(),
+  workspace_root: workspaceRoot,
+  package_root: packageRoot,
+  python_command: pythonCommand,
+  rows: [],
+};
+
+function persist() {
+  writeFileSync(reportPath, JSON.stringify(report, null, 2), "utf-8");
+}
+
+for (const run of runs) {
+  const started = Date.now();
+  const row = {
+    ...run,
+    engine: "hermes_local",
+    status: "not_run",
+    started_at: new Date(started).toISOString(),
+    artifact_preexisting: artifactStats(run.artifact).exists,
+  };
+  report.rows.push(row);
+  persist();
+
+  try {
+    const result = await runLocalModelTask({
+      workspaceRoot,
+      provider: run.provider,
+      model: run.model,
+      baseUrl: run.baseUrl,
+      engine: "hermes_local",
+      modelClass: "small_local",
+      role: "coders",
+      maxTurns: Number(process.env.ACE_LIVE_HERMES_MAX_TURNS || 8),
+      tier: "brief",
+      toolScope: ["write_workspace_file"],
+      task: htmlTask(run),
+      hermesExecutor: new HermesSubprocessExecutor({ pythonCommand }),
+    });
+    row.status = result.result.status;
+    row.summary = result.result.summary;
+    row.turns = result.result.turns;
+    row.tool_calls = result.result.tool_calls;
+    row.evidence_refs = result.result.evidence_refs || [];
+    row.policy = result.policy;
+    row.runtime = result.runtime;
+    row.hermes = {
+      bridge_protocol_version: result.hermes?.bridge_protocol_version,
+      hermes_session_id: result.hermes?.hermes_session_id,
+      shadow_mcp_session_id: result.hermes?.shadow_mcp_session_id,
+      execution_engine: result.hermes?.execution_engine,
+      provider: result.hermes?.provider,
+      model: result.hermes?.model,
+      event_count: result.hermes?.events?.length || 0,
+      event_types: [...new Set((result.hermes?.events || []).map((event) => event.type))],
+    };
+  } catch (error) {
+    row.status = "error";
+    row.summary = error instanceof Error ? error.stack || error.message : String(error);
+  } finally {
+    row.duration_ms = Date.now() - started;
+    row.finished_at = new Date().toISOString();
+    row.artifact = materializeHermesHtmlIfNeeded(row, run.artifact);
+    row.job_done = Boolean(
+      row.artifact?.exists &&
+        !row.artifact?.stale &&
+        row.artifact.hasHtml &&
+        row.artifact.hasCss &&
+        row.artifact.hasJs &&
+        row.artifact.mentionsCrispr &&
+        row.artifact.externalRefs === 0
+    );
+    row.protocol_done = Boolean(
+      row.hermes?.event_count > 0 &&
+        row.hermes.event_types?.includes("session_open") &&
+        row.hermes.event_types?.includes("session_close")
+    );
+    row.tool_write_done = Boolean(
+      row.tool_calls?.some((tool) => tool.tool === "write_workspace_file" && tool.ok)
+    );
+    persist();
+  }
+}
+
+console.log(reportPath);
+if (report.rows.length === 0 || report.rows.some((row) => row.status === "error" || !row.job_done || !row.protocol_done)) {
+  process.exitCode = 1;
+}