@voybio/ace-swarm 0.1.0 → 0.2.0

package/README.md

@@ -2,7 +2,7 @@
 
 Autonomous Coding Entity (ACE) is a local MCP server and CLI for agent-assisted coding.
 
-`ace-swarm` gives Claude Code, Codex, Cursor, VS Code, and local-model workflows a shared runtime: durable workspace state, typed handoffs, scheduler primitives, change intelligence, and operator tooling. It is built around a Zarrita-backed `.agents/ACE/ace-state.zarr` store, so the workspace keeps one source of truth instead of a forest of generated state files.
+`ace-swarm` gives Claude Code, Codex, Cursor, VS Code, and local-model workflows a shared runtime: durable workspace state, typed handoffs, scheduler primitives, change intelligence, and operator tooling. All state lives in a single binary file — `.agents/ACE/ace-state.ace` — so the workspace keeps one source of truth instead of a forest of generated state files.
 
 Modern coding agents can already read files, write code, run commands, and call tools. The missing layer is coordination that survives restarts, model switches, and multi-agent work. ACE fills that gap by keeping the workflow in the store and projecting only the few files external clients still need. For local models, ACE provides the eyes and ears they usually lack: task context, tool surfaces, status events, evidence trails, and resumable state.
 
@@ -12,7 +12,7 @@ Modern coding agents can already read files, write code, run commands, and call
 - Handoffs should be typed, evidence-linked, and validated.
 - Shared workflow state should live in the workspace, not only in chat history.
 - Local models should get the same structure and observability as hosted models.
-- Bootstrap should be contained and reversible: ACE writes into `.agents/ACE/ace-state.zarr`.
+- Bootstrap should be contained and reversible: ACE writes into `.agents/ACE/ace-state.ace`.
 
 ### Serving local models
 
@@ -33,21 +33,62 @@ Frontier models are getting more capable and more restricted at the same time. P
 
 ACE sits underneath that stack. It does not compete with the host you already use. It gives that host a durable local runtime.
 
-## Zarr State Model
+## ACEPACK State Model
 
-ACE uses `@zarrita/core` and `@zarrita/storage` to present `.agents/ACE/ace-state.zarr` as a hierarchical store.
-Zarrita.js supplies the Zarr v3 groups, arrays, and typed views; ACE supplies `AcePackedStore`, the single-file local backend that keeps the store portable and atomic.
+ACE uses a custom binary format called ACEPACK to present `.agents/ACE/ace-state.ace` as a structured single-file store. The file has three regions:
+
+```
+ace-state.ace
+├── Header (128 bytes)
+│     magic · version · flags
+│     kv_index_offset · kv_index_length · kv_chunk_end
+│     evt_offset · evt_length · evt_count · evt_base_id
+├── KV Chunk Region  (append-only, random-access)
+│     knowledge/agents/{name}/{file}  →  agent instruction text
+│     knowledge/skills/{name}/{file}  →  skill content
+│     topology/{kind}                 →  swarm registry, route tables
+│     state/{...}                     →  handoffs, todo, ledger, scheduler
+│     meta/{...}                      →  schema version, project name
+│     Each chunk: [uint32 length][bytes]
+└── Columnar Event Section  (rewritten on commit)
+      uint32    event count
+      int64[N]  timestamps      (epoch ms, big-endian)
+      uint8[N]  kinds           (EntityKind enum)
+      uint8[N]  sources         (ContentSource enum)
+      uint8[N]  flags           (0x01 = deleted)
+      uint32[N] payload offsets (→ payload pool)
+      uint32[N] payload lengths
+      uint32    pool length
+      bytes[M]  payload pool    (UTF-8 JSON blobs)
+```
+
+### KV region — random-access blob cache
+
+All persistent state — agent instructions, skills, topology, task indexes, scheduler queues — lives in the KV region. Each key maps to an immutable chunk appended at write time. `commit()` rewrites the KV index (a small JSON map at the file tail) to reflect the live keys. `compact()` rewrites the KV region to remove dead space from overwritten keys, atomically via tmp-rename.
+
+Write cost: O(1) append + O(index_size) index rewrite.
+Read cost: O(1) seek + O(chunk_length) read.
+
+### Columnar event section — typed handoff log
+
+Every `appendEntry()` call — handoffs, status events, ledger entries, session events — lands in an in-memory buffer and is flushed to the columnar section on `commit()`. The columnar layout stores fixed-width fields (timestamp, kind, source, flags) separately from payloads. This means:
+
+- Scanning "all handoffs in the last hour" reads only `timestamps[]` + `kinds[]` — 9 bytes per event, never touching the payload pool.
+- Per-event overhead is ~19 bytes fixed vs ~150 bytes for equivalent JSON.
+- At 100,000 events, the fixed columns are ~1.9 MB; a full JSON log would be ~15 MB.
+
+Events accumulate for the workspace lifetime and survive `compact()`. Full historical replay is always available. See `HOT_COLD_EVENT_TIERING.md` for the future branch that adds compressed batch archiving for very long-lived workspaces.
 
 ### Why one state file
 
-| One `.zarr` authority | What it avoids |
+| One `.ace` authority | What it avoids |
 |---|---|
-| One read/write source of truth | Hundreds of generated state files drifting apart |
-| Zarr groups and arrays | Hand-rolled file sprawl for runtime state, instructions, and indexes |
-| Typed hot fields | Repeated JSON parsing for counters and scheduler state |
+| Single append-only file | Hundreds of generated state files drifting apart |
+| KV region with typed keys | Hand-rolled file sprawl for runtime state, instructions, indexes |
+| Columnar event log | Repeated full-JSON parsing for timestamp and kind scanning |
 | Store-backed projections | External tools reading the same truth through different paths |
 
-The store holds runtime state, agent instructions, skills, topology, schemas, and indexes. Materialized files exist only when an external client still needs a path on disk.
+The store holds runtime state, agent instructions, skills, topology, schemas, and the complete event history. Materialized files exist only when an external client still needs a path on disk.
 
 ## Quick Start
 
@@ -87,9 +128,9 @@ If you already have a local runtime running, `ace doctor --scan` will probe comm
 
 ## What Bootstrap Writes
 
-ACE bootstrap is intentionally contained. The store is authoritative at `.agents/ACE/ace-state.zarr`; only a few host-facing files land in the workspace when requested.
+ACE bootstrap is intentionally contained. The store is authoritative at `.agents/ACE/ace-state.ace`; only a few host-facing files land in the workspace when requested.
 
-- `.agents/ACE/ace-state.zarr` — runtime state, agent instructions, skills, topology, schemas, and indexes
+- `.agents/ACE/ace-state.ace` — runtime state, agent instructions, skills, topology, schemas, and the event log
 - `.agents/ACE/ace-hook-context.json` — compact hook snapshot for external clients
 - `.agents/ACE/tasks/todo.md` — human-facing todo surface
 - Optional workspace-root stubs — `AGENTS.md`, `CLAUDE.md`, `.cursorrules`, `.github/copilot-instructions.md`, and `.vscode/mcp.json`
@@ -115,7 +156,7 @@ Developer or MCP host
   |
   +-- .agents/ACE/
   |       |
-  |       +-- ace-state.zarr
+  |       +-- ace-state.ace
   |       +-- ace-hook-context.json
   |       +-- tasks/todo.md
   |       `-- host config bundles
@@ -137,13 +178,15 @@ Developer or MCP host
 | Runtime roles        | 17 runtime roles: 4 swarm roles + 13 composable roles        |
 | Skills               | 15 packaged skills                                           |
 | MCP server           | stdio server for tools, prompts, resources, and workflows    |
-| Durable state        | single `.zarr` authority for handoffs, status events, todo state, run ledger, job queue, and discovery |
+| Durable state        | single .ace authority for handoffs, status events, todo      |
+|                      | state, run ledger, job queue, and discovery                  |
 | Projections          | hook context, todo surface, and host configs when needed      |
 | Coordination         | priority dispatch, dependency gates, leases, resource locks  |
 | Change intelligence  | delta scan, semantic snapshots, drift reports, rewrite hints |
 | Terminal UI          | provider-aware TUI with chat, tabs, telemetry, agent tabs    |
-| Local models         | Ollama and llama.cpp bootstrap, doctor checks, model bridge, tool loops |
-| Verification         | 208 tests passing in the current suite                       |
+| Local models         | Ollama and llama.cpp bootstrap, doctor checks, model bridge, |
+|                      | tool loops                                                   |
+| Verification         | 242 tests passing in the current suite                       |
 +----------------------+--------------------------------------------------------------+
 ```
 
@@ -162,7 +205,7 @@ In that setup, the host remains the interface. ACE becomes the state contract un
 
 Local models are good at generating text. They usually need help seeing the workspace, hearing the handoff trail, and remembering what changed two turns ago. ACE closes that gap.
 
-- `ace init --llm ollama` or `ace init --llm llama.cpp` seeds the profile inside `.agents/ACE/ace-state.zarr`.
+- `ace init --llm ollama` or `ace init --llm llama.cpp` seeds the profile inside `.agents/ACE/ace-state.ace`.
 - `ace doctor` verifies that the selected runtime is reachable and configured.
 - `ace tui` can talk to either local runtime and surface the same workspace state the MCP server sees.
 - The ACE model bridge gives local runs tool selection, execution flow, and persisted state instead of a fragile one-shot chat loop.
@@ -199,7 +242,6 @@ If you want local agents with memory, evidence, and handoff discipline, this is
 +---------------------------------------------+---------------------------------------------------+
 ```
 
-
 ## Development
 
 ```bash
@@ -216,10 +258,8 @@ See [CHANGELOG.md](./CHANGELOG.md) for release history.
 
 - [Claude Code overview](https://docs.anthropic.com/en/docs/claude-code/overview)
 - [OpenAI connectors and MCP servers](https://platform.openai.com/docs/guides/tools-remote-mcp?lang=python)
-- [Introducing GPT-5.2-Codex](https://openai.com/index/introducing-gpt-5-2-codex/)
 - [Ollama tool calling and agent loops](https://docs.ollama.com/capabilities/tool-calling)
 - [llama.cpp](https://github.com/ggerganov/llama.cpp)
-- [Zarrita.js](https://zarrita.dev/)
 - [Model Context Protocol SDKs](https://modelcontextprotocol.io/docs/sdk)
 
 ## License

package/assets/scripts/ace-hook-dispatch.mjs

@@ -206,7 +206,7 @@ function buildRoleToolHint(role) {
  * Try to read the store-materialized hook context snapshot.
  * Returns the parsed snapshot or undefined if not present / unreadable.
  * When present this is the single authoritative source; the dispatcher
- * never needs to open the .zarr store directly.
+ * never needs to open the .ace store directly.
  */
 function tryReadHookContextSnapshot(workspaceRoot) {
   const snapshotPath = resolve(workspaceRoot, ACE_ROOT, "ace-hook-context.json");

package/dist/ace-server-instructions.js

@@ -187,7 +187,7 @@ export function buildHostInstructionText(host, workspaceRoot) {
         `# ACE Bootstrap For ${hostLabel}`,
         "",
         "This is a thin ACE bootstrap stub.",
-        "Full instructions, skills, task packs, and runtime state live inside `.agents/ACE/ace-state.zarr` and should be accessed through the ACE MCP server.",
+        "Full instructions, skills, task packs, and runtime state live inside `.agents/ACE/ace-state.ace` and should be accessed through the ACE MCP server.",
         "",
         "Session start protocol:",
         "1. Call `recall_context` to load task, scope, and status from ACE state.",
@@ -195,7 +195,7 @@ export function buildHostInstructionText(host, workspaceRoot) {
         "3. Prefer ACE prompts/resources (`ace-orchestrator`, `get_task_pack`, `get_skill_instructions`) over relying on this file.",
         "",
         "Ground truth:",
-        "- `.agents/ACE/ace-state.zarr` is authoritative.",
+        "- `.agents/ACE/ace-state.ace` is authoritative.",
         "- `.agents/ACE/tasks/todo.md` is the human-facing todo surface.",
         "- `.agents/ACE/ace-hook-context.json` is the compact hook snapshot.",
         "",

package/dist/cli.js

@@ -33,7 +33,7 @@ Options for tui:
   --ollama-url <url>           Legacy alias for --base-url
 
 Options for init:
-  --project <name>             Project name stored in ${ACE_ROOT_REL}/ace-state.zarr metadata
+  --project <name>             Project name stored in ${ACE_ROOT_REL}/ace-state.ace metadata
   --force                      Overwrite scaffolded files if they already exist
   --mcp-config                 Also write .vscode/mcp.json workspace bridge
   --client-config-bundle       Also write minimal workspace host stubs (AGENTS.md, CLAUDE.md, .cursorrules, .github/copilot-instructions.md)
@@ -43,7 +43,7 @@ Options for init:
   --ollama-url <url>           Legacy alias for --base-url
 
 Options for doctor:
-  --llm <provider>             ollama|llama.cpp (default: auto from ${ACE_ROOT_REL}/ace-state.zarr)
+  --llm <provider>             ollama|llama.cpp (default: auto from ${ACE_ROOT_REL}/ace-state.ace)
   --model <name>               Model name override
   --base-url <url>             Local runtime base URL override
   --ollama-url <url>           Legacy alias for --base-url
@@ -224,7 +224,7 @@ async function runInit(args, mode = "init") {
             message: `ace ${mode} completed: ${storeResult.materialized.length} bootstrap files materialized`,
             artifacts: [
                 `${ACE_TASKS_ROOT_REL}/`,
-                `${ACE_ROOT_REL}/ace-state.zarr`,
+                `${ACE_ROOT_REL}/ace-state.ace`,
                 delta.index_path,
                 astIndex.output_json_path,
             ],
@@ -302,7 +302,7 @@ async function runDoctor(args) {
         detail: hasMcpConfig
             ? hasWorkspaceMcpConfig
                 ? "Found one or more workspace MCP bridge files."
-                : "Store-backed MCP snippets are baked into ace-state.zarr. Run `ace mcp-config --all` for global install."
+                : "Store-backed MCP snippets are baked into ace-state.ace. Run `ace mcp-config --all` for global install."
             : "No MCP bridge or baked snippets found. Run `ace init` first.",
     });
     const profilePath = `${getWorkspaceStorePath(WORKSPACE_ROOT)}#state/runtime/llm_profile`;
@@ -312,7 +312,7 @@ async function runDoctor(args) {
         ok: hasProfile,
         detail: hasProfile
             ? profilePath
-            : `Missing ${ACE_ROOT_REL}/ace-state.zarr#state/runtime/llm_profile. Run \`ace init --llm ollama\` or \`ace doctor --scan\`.`,
+            : `Missing ${ACE_ROOT_REL}/ace-state.ace#state/runtime/llm_profile. Run \`ace init --llm ollama\` or \`ace doctor --scan\`.`,
     });
     let provider = llm.llmProvider;
     let model = llm.llmModel;
@@ -354,7 +354,7 @@ async function runDoctor(args) {
         }
     }
     if (!provider) {
-        throw new Error(`No local runtime provider configured. Use --llm <provider>, bootstrap one into ${ACE_ROOT_REL}/ace-state.zarr#state/runtime/llm_profile, or run \`ace doctor --scan\`.`);
+        throw new Error(`No local runtime provider configured. Use --llm <provider>, bootstrap one into ${ACE_ROOT_REL}/ace-state.ace#state/runtime/llm_profile, or run \`ace doctor --scan\`.`);
     }
     if (!model) {
         model = provider === "llama.cpp" ? DEFAULT_LLAMA_CPP_MODEL : DEFAULT_OLLAMA_MODEL;
@@ -540,7 +540,7 @@ async function main() {
     }
     if (command === "migrate") {
         const { importFromFileWorkspace } = await import("./store/importer.js");
-        const storePath = args[1] ?? `${WORKSPACE_ROOT}/.agents/ACE/ace-state.zarr`;
+        const storePath = args[1] ?? `${WORKSPACE_ROOT}/.agents/ACE/ace-state.ace`;
         console.log(`Migrating file workspace at ${WORKSPACE_ROOT} to store at ${storePath}...`);
         const result = await importFromFileWorkspace(WORKSPACE_ROOT, storePath);
         console.log(`Migration complete:`);

package/dist/helpers.js

@@ -530,7 +530,7 @@ export function classifyPathSource(path) {
     if (!path)
         return "missing";
     const root = currentWorkspaceRoot();
-    if (path.includes(".zarr#"))
+    if (path.includes(".ace#"))
         return "store";
     if (isInside(root, path))
         return "workspace";

package/dist/store/ace-packed-store.d.ts

@@ -1,42 +1,81 @@
 /**
- * AcePackedStore
+ * AcePackedStore — ACEPACK v2
  *
- * Custom single-file store that implements Zarrita's AsyncReadable + AsyncWritable
- * interfaces over a packed binary file at `.agents/ACE/ace-state.zarr`.
+ * Single-file binary store. One file, no subdirectories, no companion files.
+ * Events grow in the columnar section indefinitely (Option A). compact() only
+ * reclaims dead space in the KV chunk region — it does NOT reset or archive
+ * events. Historical event replay across all past runs is always available.
  *
- * File format:
- *   [ Header  64 bytes ]   magic(8) version(4) flags(4) index_offset(8) index_length(8) reserved(32)
- *   [ Chunk region     ]   chunk_0: [uint32 length][bytes] ... chunk_N: [uint32 length][bytes]
- *   [ Index region     ]   UTF-8 JSON: { key → { offset, length } }
+ * File layout:
  *
- * Write protocol:
- *   1. Append new chunk to end of chunk region.
- *   2. Update in-memory index.
- *   3. On commit(): rewrite index region → update header → fsync.
+ *   ┌─────────────────────────────────────────────────────────┐
+ *   │ Header (128 bytes, big-endian)                           │
+ *   │   magic "ACEPACK\0" · version uint32 · flags uint32     │
+ *   │   kv_index_offset uint64 · kv_index_length uint64       │
+ *   │   kv_chunk_end uint64                                    │
+ *   │   evt_offset uint64 · evt_length uint64                  │
+ *   │   evt_count uint32 · evt_base_id uint32                  │
+ *   │   reserved (zeros)                                       │
+ *   ├─────────────────────────────────────────────────────────┤
+ *   │ KV Chunk Region (append-only, random-access)            │
+ *   │   knowledge/agents/{name}/{file} → instruction text     │
+ *   │   knowledge/skills/{name}/{file} → skill content        │
+ *   │   topology/{kind}                → JSON array           │
+ *   │   state/{...}                    → runtime state blobs  │
+ *   │   meta/{...}                     → schema version etc.  │
+ *   │   Each chunk: [uint32 length BE][bytes]                 │
+ *   ├─────────────────────────────────────────────────────────┤
+ *   │ Columnar Event Section (rewritten on every commit)      │
+ *   │   uint32    count                                        │
+ *   │   int64[N]  timestamps  (epoch ms, big-endian)          │
+ *   │   uint8[N]  kinds       (EntityKind enum)               │
+ *   │   uint8[N]  sources     (ContentSource enum)            │
+ *   │   uint8[N]  flags       (0x01=deleted)                  │
+ *   │   [pad to 4-byte alignment]                             │
+ *   │   uint32[N] pay_offsets (relative to pool start)        │
+ *   │   uint32[N] pay_lengths                                 │
+ *   │   uint32    pool_length                                  │
+ *   │   bytes[M]  payload pool (UTF-8 JSON: {key,payload,...})│
+ *   ├─────────────────────────────────────────────────────────┤
+ *   │ KV Index (JSON, rewritten on commit)                    │
+ *   │   { "knowledge/agents/ace-ops/AGENT.md": {offset, length}, ... }
+ *   └─────────────────────────────────────────────────────────┘
  *
- * Compaction:
- *   Write live chunks to temp file → validate → atomic rename.
+ * Event overhead: ~19 bytes fixed + payload (vs ~150 bytes full JSON).
+ * Timestamp/kind scanning reads only the fixed-width columns, never the pool.
  *
- * Locking:
- *   Advisory flock-style via a `.lock` sidecar file (PID + timestamp).
- *   Stale detection: lock held >30s and PID not alive → break lock.
+ * compact() removes dead KV space (overwritten keys) via atomic tmp-rename.
+ * Events are preserved across compact() — they accumulate for the workspace
+ * lifetime. See HOT_COLD_EVENT_TIERING.md for a future branch that adds
+ * BGZF batch archiving for very long-lived workspaces.
+ *
+ * Backward compat: v1 files (64-byte header) are migrated on first commit().
+ * v1 events live in core/log/ KV blobs and are pulled into the columnar section.
  */
 import { IAcePackedStore, UnifiedEntry, EntryFilter, TopologyEntry, ContentSourceCode } from "./types.js";
 export declare class AcePackedStore implements IAcePackedStore {
     private storePath;
-    private lockPath;
     private readOnly;
-    private index;
-    private chunkEnd;
     private fh;
-    private dirty;
-    private entryCounter;
+    private kvIndex;
+    private kvChunkEnd;
+    private committed;
+    private pending;
+    private evtBaseId;
     open(path: string, opts?: {
         readOnly?: boolean;
     }): Promise<void>;
-    private _initNewStore;
-    private _loadExistingStore;
+    private _initNew;
+    private _loadExisting;
+    /** Read a KV blob directly from a loaded file buffer (used during migration). */
+    private _readKvBlobDirect;
     commit(): Promise<void>;
+    /**
+     * Compacts the KV chunk region — removes dead space left by overwritten keys.
+     * Events are NOT reset or archived; they accumulate in the columnar section
+     * for the workspace lifetime (Option A — grow forever).
+     * See HOT_COLD_EVENT_TIERING.md for the future branch that adds BGZF archiving.
+     */
     compact(): Promise<void>;
     close(): Promise<void>;
     get(key: string): Promise<Uint8Array | undefined>;
@@ -49,7 +88,7 @@ export declare class AcePackedStore implements IAcePackedStore {
     setBlob(key: string, content: string): Promise<void>;
     appendEntry(entry: Omit<UnifiedEntry, "id" | "ts">): Promise<UnifiedEntry>;
     getEntries(filter?: EntryFilter): Promise<UnifiedEntry[]>;
-    getEntry(key: string): Promise<UnifiedEntry | undefined>;
+    getEntry(_key: string): Promise<UnifiedEntry | undefined>;
     getTopology(kind: string): Promise<TopologyEntry[]>;
     setTopology(kind: string, entries: TopologyEntry[]): Promise<void>;
     getAgentInstruction(agent: string, file: string): Promise<string | undefined>;
@@ -58,9 +97,9 @@ export declare class AcePackedStore implements IAcePackedStore {
     getSkillContent(skill: string, file: string): Promise<string | undefined>;
     setSkillContent(skill: string, file: string, content: string, source?: ContentSourceCode): Promise<void>;
     listSkills(): Promise<string[]>;
-    /** Dead space ratio — used to decide if compaction is needed */
+    /** Dead space ratio in the KV chunk region. Used to decide if compaction is needed. */
     get deadSpaceRatio(): number;
-    /** Number of entries in the unified log */
+    /** Total events in log (committed + pending). */
     get entryCount(): number;
 }
 export declare function openStore(storePath: string, opts?: {