memory-lancedb-pro 1.0.4 → 1.0.5

package/CHANGELOG.md

@@ -1,5 +1,10 @@
 # Changelog
 
+## 1.0.5
+
+- Add: optional JSONL session distillation workflow (incremental cursor + batch format) via `scripts/jsonl_distill.py`.
+- Docs: document the JSONL distiller setup in README (EN) and README_CN (ZH).
+
 ## 1.0.4
 
 - Fix: `embedding.dimensions` is now parsed robustly (number / numeric string / env-var string), so it properly overrides hardcoded model dims (fixes Ollama `nomic-embed-text` dimension mismatch).

package/README.md

@@ -374,6 +374,136 @@ Cross-encoder reranking supports multiple providers via `rerankProvider`:
 
 ---
 
+## Optional: JSONL Session Distillation (Auto-memories from chat logs)
+
+OpenClaw already persists **full session transcripts** as JSONL files:
+
+- `~/.openclaw/agents/<agentId>/sessions/*.jsonl`
+
+This plugin focuses on **high-quality long-term memory**. If you dump raw transcripts into LanceDB, retrieval quality quickly degrades.
+
+Instead, you can run an **hourly distiller** that:
+
+1) Incrementally reads only the **newly appended tail** of each session JSONL (byte-offset cursor)
+2) Filters noise (tool output, injected `<relevant-memories>`, logs, boilerplate)
+3) Uses a dedicated agent to **distill** reusable lessons / rules / preferences into short atomic memories
+4) Stores them via `memory_store` into the right **scope** (`global` or `agent:<agentId>`)
+
+### What you get
+
+- ✅ Fully automatic (cron)
+- ✅ Multi-agent support (main + bots)
+- ✅ No re-reading: cursor ensures the next run only processes new lines
+- ✅ Memory hygiene: quality gate + dedupe + per-run caps
+
+### Script
+
+This repo includes the extractor script:
+
+- `scripts/jsonl_distill.py`
+
+It produces a small **batch JSON** file under:
+
+- `~/.openclaw/state/jsonl-distill/batches/`
+
+and keeps a cursor here:
+
+- `~/.openclaw/state/jsonl-distill/cursor.json`
+
+The script is **safe**: it never modifies session logs.
+
+By default it skips historical reset snapshots (`*.reset.*`) and excludes the distiller agent itself (`memory-distiller`) to prevent self-ingestion loops.
+
+### Recommended setup (dedicated distiller agent)
+
+#### 1) Create a dedicated agent
+
+```bash
+openclaw agents add memory-distiller \
+  --non-interactive \
+  --workspace ~/.openclaw/workspace-memory-distiller \
+  --model openai-codex/gpt-5.2
+```
+
+#### 2) Initialize cursor (Mode A: start from now)
+
+This marks all existing JSONL files as "already read" by setting offsets to EOF.
+
+```bash
+# Set PLUGIN_DIR to where this plugin is installed.
+# - If you cloned into your OpenClaw workspace (recommended):
+#   PLUGIN_DIR="$HOME/.openclaw/workspace/plugins/memory-lancedb-pro"
+# - Otherwise, check: `openclaw plugins info memory-lancedb-pro` and locate the directory.
+PLUGIN_DIR="/path/to/memory-lancedb-pro"
+
+python3 "$PLUGIN_DIR/scripts/jsonl_distill.py" init
+```
+
+#### 3) Create an hourly cron job (Asia/Shanghai)
+
+Tip: start the message with `run ...` so `memory-lancedb-pro`'s adaptive retrieval will skip auto-recall injection (saves tokens).
+
+```bash
+# IMPORTANT: replace <PLUGIN_DIR> in the template below with your actual plugin path.
+MSG=$(cat <<'EOF'
+run jsonl memory distill
+
+Goal: distill NEW chat content from OpenClaw session JSONL files into high-quality LanceDB memories using memory_store.
+
+Hard rules:
+- Incremental only: call the extractor script; do NOT scan full history.
+- Store only reusable memories; skip routine chatter.
+- English memory text + final line: Keywords (zh): ...
+- < 500 chars, atomic.
+- <= 3 memories per agent per run; <= 3 global per run.
+- Scope: global for broadly reusable; otherwise agent:<agentId>.
+
+Workflow:
+1) exec: python3 <PLUGIN_DIR>/scripts/jsonl_distill.py run
+2) If noop: stop.
+3) Read batchFile (created/pending)
+4) memory_store(...) for selected memories
+5) exec: python3 <PLUGIN_DIR>/scripts/jsonl_distill.py commit --batch-file <batchFile>
+EOF
+)
+
+openclaw cron add \
+  --agent memory-distiller \
+  --name "jsonl-memory-distill (hourly)" \
+  --cron "0 * * * *" \
+  --tz "Asia/Shanghai" \
+  --session isolated \
+  --wake now \
+  --timeout-seconds 420 \
+  --stagger 5m \
+  --no-deliver \
+  --message "$MSG"
+```
+
+#### 4) Debug run
+
+```bash
+openclaw cron run <jobId> --expect-final --timeout 180000
+openclaw cron runs --id <jobId> --limit 5
+```
+
+### Scope strategy (recommended)
+
+When distilling **all agents**, always set `scope` explicitly when calling `memory_store`:
+
+- Broadly reusable → `scope=global`
+- Agent-specific → `scope=agent:<agentId>`
+
+This prevents cross-bot memory pollution.
+
+### Rollback
+
+- Disable/remove cron job: `openclaw cron disable <jobId>` / `openclaw cron rm <jobId>`
+- Delete agent: `openclaw agents delete memory-distiller`
+- Remove cursor state: `rm -rf ~/.openclaw/state/jsonl-distill/`
+
+---
+
 ## CLI Commands
 
 ```bash

package/README_CN.md

@@ -325,7 +325,117 @@ openclaw config get plugins.slots.memory
 | **Jina**（推荐） | `jina-embeddings-v5-text-small` | `https://api.jina.ai/v1` | 1024 |
 | **OpenAI** | `text-embedding-3-small` | `https://api.openai.com/v1` | 1536 |
 | **Google Gemini** | `gemini-embedding-001` | `https://generativelanguage.googleapis.com/v1beta/openai/` | 3072 |
-| **Ollama**（本地） | `nomic-embed-text` | `http://localhost:11434/v1` | 768 |
+| **Ollama**（本地） | `nomic-embed-text` | `http://localhost:11434/v1` | _与本地模型输出一致_（建议显式设置 `embedding.dimensions`） |
+
+---
+
+## （可选）从 Session JSONL 自动蒸馏记忆（全自动）
+
+OpenClaw 会把每个 Agent 的完整会话自动落盘为 JSONL：
+
+- `~/.openclaw/agents/<agentId>/sessions/*.jsonl`
+
+但 JSONL 含大量噪声（tool 输出、系统块、重复回调等），**不建议直接把原文塞进 LanceDB**。
+
+本插件提供一个安全的 extractor 脚本 `scripts/jsonl_distill.py`，配合 OpenClaw 的 `cron` + 独立 distiller agent，实现“增量蒸馏 → 高质量记忆入库”：
+
+- 只读取每个 JSONL 文件**新增尾巴**（byte offset cursor），避免重复和 token 浪费
+- 生成一个小型 batch JSON
+- 由 distiller agent 把 batch 蒸馏成短、原子、可复用的记忆，再用 `memory_store` 写入
+
+### 你会得到什么
+
+- ✅ 全自动（每小时）
+- ✅ 多 Agent 支持（main + 各 bot）
+- ✅ 只处理新增内容（不回读）
+- ✅ 防自我吞噬：默认排除 `memory-distiller` 自己的 session
+
+### 脚本输出位置
+
+- Cursor：`~/.openclaw/state/jsonl-distill/cursor.json`
+- Batches：`~/.openclaw/state/jsonl-distill/batches/`
+
+> 脚本只读 session JSONL，不会修改原始日志。
+
+### 推荐部署（独立 distiller agent）
+
+#### 1）创建 distiller agent（示例用 gpt-5.2）
+
+```bash
+openclaw agents add memory-distiller \
+  --non-interactive \
+  --workspace ~/.openclaw/workspace-memory-distiller \
+  --model openai-codex/gpt-5.2
+```
+
+#### 2）初始化 cursor（模式 A：从现在开始，不回溯历史）
+
+先确定插件目录（PLUGIN_DIR）：
+
+```bash
+# 如果你按推荐方式 clone 到 workspace：
+#   PLUGIN_DIR="$HOME/.openclaw/workspace/plugins/memory-lancedb-pro"
+PLUGIN_DIR="/path/to/memory-lancedb-pro"
+
+python3 "$PLUGIN_DIR/scripts/jsonl_distill.py" init
+```
+
+#### 3）创建每小时 Cron（Asia/Shanghai）
+
+建议 cron message 以 `run ...` 开头，这样本插件的自适应检索会跳过自动 recall 注入（节省 token）。
+
+```bash
+MSG=$(cat <<'EOF'
+run jsonl memory distill
+
+Goal: Distill ONLY new content from OpenClaw session JSONL tails into high-quality LanceDB memories.
+
+Hard rules:
+- Incremental only: exec the extractor. Do NOT scan full history.
+- If extractor returns action=noop: stop immediately.
+- Store only reusable memories (rules, pitfalls, decisions, preferences, stable facts). Skip routine chatter.
+- Each memory: idiomatic English + final line `Keywords (zh): ...` (3-8 short phrases).
+- Keep each memory < 500 chars and atomic.
+- Caps: <= 3 memories per agent per run; <= 3 global per run.
+- Scope:
+  - broadly reusable -> global
+  - agent-specific -> agent:<agentId>
+
+Workflow:
+1) exec: python3 <PLUGIN_DIR>/scripts/jsonl_distill.py run
+2) Determine batch file (created/pending)
+3) memory_store(...) for selected memories
+4) exec: python3 <PLUGIN_DIR>/scripts/jsonl_distill.py commit --batch-file <batchFile>
+EOF
+)
+
+openclaw cron add \
+  --agent memory-distiller \
+  --name "jsonl-memory-distill (hourly)" \
+  --cron "0 * * * *" \
+  --tz "Asia/Shanghai" \
+  --session isolated \
+  --wake now \
+  --timeout-seconds 420 \
+  --stagger 5m \
+  --no-deliver \
+  --message "$MSG"
+```
+
+### scope 策略（非常重要）
+
+当蒸馏“所有 agents”时，务必显式设置 scope：
+
+- 跨 agent 通用规则/偏好/坑 → `scope=global`
+- agent 私有 → `scope=agent:<agentId>`
+
+否则不同 bot 的记忆会相互污染。
+
+### 回滚
+
+- 禁用/删除 cron：`openclaw cron disable <jobId>` / `openclaw cron rm <jobId>`
+- 删除 distiller agent：`openclaw agents delete memory-distiller`
+- 删除 cursor 状态：`rm -rf ~/.openclaw/state/jsonl-distill/`
 
 ---
 

package/openclaw.plugin.json

@@ -2,7 +2,7 @@
   "id": "memory-lancedb-pro",
   "name": "Memory (LanceDB Pro)",
   "description": "Enhanced LanceDB-backed long-term memory with hybrid retrieval, multi-scope isolation, and management CLI",
-  "version": "1.0.4",
+  "version": "1.0.5",
   "kind": "memory",
   "configSchema": {
     "type": "object",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "memory-lancedb-pro",
-  "version": "1.0.4",
+  "version": "1.0.5",
   "description": "OpenClaw enhanced LanceDB memory plugin with hybrid retrieval (Vector + BM25), cross-encoder rerank, multi-scope isolation, and management CLI",
   "type": "module",
   "main": "index.ts",

package/scripts/jsonl_distill.py

@@ -0,0 +1,451 @@
+#!/usr/bin/env python3
+"""jsonl_distill.py
+
+Incrementally extract new chat messages from OpenClaw session JSONL files and
+write a compact batch file for a distiller agent to turn into LanceDB memories.
+
+Design goals:
+- Read only the newly-appended tail of each session file (byte-offset cursor).
+- Avoid token waste: if there is no new content, produce no batch.
+- Safety: never delete/modify session logs.
+- Robustness: handle file rotation/truncation using inode+size checks.
+
+This script does NOT call any LLM or write to LanceDB. It only prepares data
+for the distiller agent.
+"""
+
+from __future__ import annotations
+
+import argparse
+import hashlib
+import json
+import os
+import re
+import sys
+import time
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Any, Dict, List, Optional, Tuple
+
+
+DEFAULT_STATE_DIR = Path.home() / ".openclaw" / "state" / "jsonl-distill"
+DEFAULT_AGENTS_DIR = Path.home() / ".openclaw" / "agents"
+
+# Prevent self-ingestion loops: the distiller agent itself should never be a source.
+EXCLUDED_AGENT_IDS = {
+    "memory-distiller",
+}
+
+
+NOISE_PREFIXES = (
+    "✅ New session started",
+    "NO_REPLY",
+)
+
+
+def _now_ms() -> int:
+    return int(time.time() * 1000)
+
+
+def _sha256(s: str) -> str:
+    return hashlib.sha256(s.encode("utf-8", errors="ignore")).hexdigest()
+
+
+def _read_jsonl_lines(path: Path, start_offset: int, max_bytes: int) -> Tuple[List[str], int]:
+    """Read up to max_bytes from path starting at start_offset. Returns (lines, end_offset)."""
+    lines: List[str] = []
+    with path.open("rb") as f:
+        f.seek(start_offset)
+        data = f.read(max_bytes)
+        end_offset = f.tell()
+
+    if not data:
+        return [], end_offset
+
+    # Ensure we end on a newline boundary to avoid partial JSON lines.
+    if not data.endswith(b"\n"):
+        last_nl = data.rfind(b"\n")
+        if last_nl == -1:
+            # No complete line in this chunk.
+            return [], start_offset
+        data = data[: last_nl + 1]
+        end_offset = start_offset + len(data)
+
+    text = data.decode("utf-8", errors="replace")
+    for line in text.splitlines():
+        line = line.strip()
+        if line:
+            lines.append(line)
+    return lines, end_offset
+
+
+def _extract_text_blocks(content: Any) -> str:
+    if content is None:
+        return ""
+    if isinstance(content, str):
+        return content
+    if isinstance(content, list):
+        parts: List[str] = []
+        for block in content:
+            if isinstance(block, dict) and block.get("type") == "text":
+                t = block.get("text")
+                if isinstance(t, str) and t:
+                    parts.append(t)
+        return "\n".join(parts)
+    return ""
+
+
+def _clean_text(s: str) -> str:
+    s = s.strip()
+    if not s:
+        return ""
+
+    # Drop injected memory blocks entirely.
+    if "<relevant-memories>" in s:
+        s = re.sub(r"<relevant-memories>[\s\S]*?</relevant-memories>", "", s)
+
+    # Drop embedded JSON blocks (often metadata) to reduce token waste.
+    s = re.sub(r"```json[\s\S]*?```", "", s)
+
+    # Collapse whitespace.
+    s = re.sub(r"\n{3,}", "\n\n", s)
+    return s.strip()
+
+
+def _is_noise(s: str) -> bool:
+    if not s:
+        return True
+    for p in NOISE_PREFIXES:
+        if s.startswith(p):
+            return True
+    # Skip overly long blocks (logs / dumps). The distiller can still capture the essence later.
+    if len(s) > 2000:
+        return True
+    # Skip pure code fences (usually tool output).
+    if s.strip().startswith("```") and s.strip().endswith("```"):
+        return True
+    return False
+
+
+@dataclass
+class CursorEntry:
+    inode: int
+    committed: int
+    pending: Optional[int] = None
+    pending_batch: Optional[str] = None
+    last_size: Optional[int] = None
+
+
+def _load_cursor(cursor_path: Path) -> Dict[str, Any]:
+    if not cursor_path.exists():
+        return {"version": 1, "files": {}, "createdAtMs": _now_ms(), "updatedAtMs": _now_ms()}
+    return json.loads(cursor_path.read_text("utf-8"))
+
+
+def _save_cursor(cursor_path: Path, cursor: Dict[str, Any]) -> None:
+    cursor["updatedAtMs"] = _now_ms()
+    cursor_path.parent.mkdir(parents=True, exist_ok=True)
+    tmp = cursor_path.with_suffix(".tmp")
+    tmp.write_text(json.dumps(cursor, ensure_ascii=False, indent=2) + "\n", "utf-8")
+    tmp.replace(cursor_path)
+
+
+def _list_session_files(agents_dir: Path) -> List[Tuple[str, Path]]:
+    results: List[Tuple[str, Path]] = []
+    if not agents_dir.exists():
+        return results
+
+    for agent_dir in sorted(agents_dir.iterdir()):
+        if not agent_dir.is_dir():
+            continue
+        agent_id = agent_dir.name
+        if agent_id in EXCLUDED_AGENT_IDS:
+            continue
+        sessions_dir = agent_dir / "sessions"
+        if not sessions_dir.exists():
+            continue
+
+        for f in sorted(sessions_dir.iterdir()):
+            name = f.name
+            if not f.is_file():
+                continue
+            if not name.endswith(".jsonl"):
+                continue
+            if ".reset." in name:
+                # Reset snapshots are historical; we start from now and focus on live session tails.
+                continue
+            if name.endswith(".lock") or ".deleted." in name:
+                continue
+            results.append((agent_id, f))
+
+    return results
+
+
+def init_from_now(state_dir: Path, agents_dir: Path) -> Dict[str, Any]:
+    cursor_path = state_dir / "cursor.json"
+    cursor = _load_cursor(cursor_path)
+    files = cursor.setdefault("files", {})
+
+    for agent_id, f in _list_session_files(agents_dir):
+        st = f.stat()
+        key = str(f)
+        files[key] = {
+            "agentId": agent_id,
+            "inode": int(st.st_ino),
+            "committed": int(st.st_size),
+            "pending": None,
+            "pendingBatch": None,
+            "lastSize": int(st.st_size),
+            "updatedAtMs": _now_ms(),
+        }
+
+    _save_cursor(cursor_path, cursor)
+    return {
+        "ok": True,
+        "action": "init",
+        "cursorPath": str(cursor_path),
+        "trackedFiles": len(files),
+    }
+
+
+def run_extract(state_dir: Path, agents_dir: Path, max_bytes_per_file: int, max_messages_per_agent: int) -> Dict[str, Any]:
+    cursor_path = state_dir / "cursor.json"
+    cursor = _load_cursor(cursor_path)
+    files: Dict[str, Any] = cursor.setdefault("files", {})
+
+    # If there is a pending batch, return it and do not read new data.
+    pending_batches = sorted({v.get("pendingBatch") for v in files.values() if v.get("pendingBatch")})
+    pending_batches = [b for b in pending_batches if b]
+    if pending_batches:
+        return {
+            "ok": True,
+            "action": "pending",
+            "batchFiles": pending_batches,
+            "cursorPath": str(cursor_path),
+        }
+
+    # Collect new messages.
+    per_agent_msgs: Dict[str, List[Dict[str, Any]]] = {}
+    touched_files: List[Dict[str, Any]] = []
+
+    for agent_id, f in _list_session_files(agents_dir):
+        key = str(f)
+        st = f.stat()
+        inode = int(st.st_ino)
+        size = int(st.st_size)
+
+        entry = files.get(key)
+        committed = 0
+        if entry and entry.get("inode") == inode:
+            committed = int(entry.get("committed") or 0)
+            # Handle truncation.
+            if size < committed:
+                committed = 0
+        else:
+            # New file not tracked yet: start from EOF (A-mode behavior).
+            committed = size
+
+        if size <= committed:
+            # Nothing new.
+            files[key] = {
+                "agentId": agent_id,
+                "inode": inode,
+                "committed": committed,
+                "pending": None,
+                "pendingBatch": None,
+                "lastSize": size,
+                "updatedAtMs": _now_ms(),
+            }
+            continue
+
+        lines, end_offset = _read_jsonl_lines(f, committed, max_bytes_per_file)
+        if not lines:
+            # Might have hit partial line boundary; do not advance.
+            continue
+
+        extracted: List[Dict[str, Any]] = []
+        for line in lines:
+            try:
+                obj = json.loads(line)
+            except Exception:
+                continue
+            if obj.get("type") != "message":
+                continue
+            msg = obj.get("message")
+            if not isinstance(msg, dict):
+                continue
+            role = msg.get("role")
+            if role not in ("user", "assistant"):
+                continue
+
+            text = _extract_text_blocks(msg.get("content"))
+            text = _clean_text(text)
+            if _is_noise(text):
+                continue
+
+            extracted.append({
+                "ts": obj.get("timestamp") or msg.get("timestamp"),
+                "role": role,
+                "text": text,
+            })
+
+        if not extracted:
+            # Advance committed to end_offset anyway to avoid re-reading pure noise.
+            files[key] = {
+                "agentId": agent_id,
+                "inode": inode,
+                "committed": end_offset,
+                "pending": None,
+                "pendingBatch": None,
+                "lastSize": size,
+                "updatedAtMs": _now_ms(),
+            }
+            continue
+
+        per_agent_msgs.setdefault(agent_id, []).extend(extracted)
+        touched_files.append({
+            "path": key,
+            "agentId": agent_id,
+            "inode": inode,
+            "committed": committed,
+            "pending": end_offset,
+            "size": size,
+        })
+
+    # Cap messages per agent to keep token usage stable.
+    for agent_id, msgs in per_agent_msgs.items():
+        if len(msgs) > max_messages_per_agent:
+            per_agent_msgs[agent_id] = msgs[-max_messages_per_agent:]
+
+    if not per_agent_msgs:
+        _save_cursor(cursor_path, cursor)
+        return {
+            "ok": True,
+            "action": "noop",
+            "cursorPath": str(cursor_path),
+        }
+
+    batches_dir = state_dir / "batches"
+    batches_dir.mkdir(parents=True, exist_ok=True)
+    batch_id = time.strftime("%Y%m%d-%H%M%S")
+    batch_path = batches_dir / f"batch-{batch_id}.json"
+
+    batch_obj = {
+        "version": 1,
+        "createdAtMs": _now_ms(),
+        "agents": [
+            {
+                "agentId": agent_id,
+                "messages": per_agent_msgs.get(agent_id, []),
+            }
+            for agent_id in sorted(per_agent_msgs.keys())
+        ],
+        "touchedFiles": touched_files,
+    }
+
+    batch_path.write_text(json.dumps(batch_obj, ensure_ascii=False, indent=2) + "\n", "utf-8")
+
+    # Write pending offsets.
+    for tf in touched_files:
+        key = tf["path"]
+        files[key] = {
+            "agentId": tf["agentId"],
+            "inode": tf["inode"],
+            "committed": tf["committed"],
+            "pending": tf["pending"],
+            "pendingBatch": str(batch_path),
+            "lastSize": tf["size"],
+            "updatedAtMs": _now_ms(),
+        }
+
+    _save_cursor(cursor_path, cursor)
+
+    return {
+        "ok": True,
+        "action": "created",
+        "batchFile": str(batch_path),
+        "agents": len(per_agent_msgs),
+        "cursorPath": str(cursor_path),
+    }
+
+
+def commit_batch(state_dir: Path, batch_file: Path) -> Dict[str, Any]:
+    cursor_path = state_dir / "cursor.json"
+    cursor = _load_cursor(cursor_path)
+    files: Dict[str, Any] = cursor.setdefault("files", {})
+
+    committed_files = 0
+    for key, v in list(files.items()):
+        if v.get("pendingBatch") != str(batch_file):
+            continue
+        pending = v.get("pending")
+        if pending is None:
+            continue
+        v["committed"] = int(pending)
+        v["pending"] = None
+        v["pendingBatch"] = None
+        v["updatedAtMs"] = _now_ms()
+        files[key] = v
+        committed_files += 1
+
+    _save_cursor(cursor_path, cursor)
+    try:
+        batch_file.unlink()
+    except Exception:
+        pass
+
+    return {
+        "ok": True,
+        "action": "committed",
+        "cursorPath": str(cursor_path),
+        "committedFiles": committed_files,
+        "batchFile": str(batch_file),
+    }
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser()
+    ap.add_argument("--state-dir", default=str(DEFAULT_STATE_DIR))
+    ap.add_argument("--agents-dir", default=str(DEFAULT_AGENTS_DIR))
+
+    sub = ap.add_subparsers(dest="cmd", required=True)
+
+    s_init = sub.add_parser("init", help="Initialize cursor to EOF for all current session files")
+
+    s_run = sub.add_parser("run", help="Extract incremental message tail and create a batch file")
+    s_run.add_argument("--max-bytes-per-file", type=int, default=256_000)
+    s_run.add_argument("--max-messages-per-agent", type=int, default=30)
+
+    s_commit = sub.add_parser("commit", help="Commit a processed batch (advance committed offsets)")
+    s_commit.add_argument("--batch-file", required=True)
+
+    args = ap.parse_args()
+
+    state_dir = Path(args.state_dir).expanduser().resolve()
+    agents_dir = Path(args.agents_dir).expanduser().resolve()
+
+    if args.cmd == "init":
+        out = init_from_now(state_dir, agents_dir)
+        print(json.dumps(out, ensure_ascii=False))
+        return 0
+
+    if args.cmd == "run":
+        out = run_extract(
+            state_dir,
+            agents_dir,
+            max_bytes_per_file=int(args.max_bytes_per_file),
+            max_messages_per_agent=int(args.max_messages_per_agent),
+        )
+        print(json.dumps(out, ensure_ascii=False))
+        return 0
+
+    if args.cmd == "commit":
+        out = commit_batch(state_dir, Path(args.batch_file).expanduser().resolve())
+        print(json.dumps(out, ensure_ascii=False))
+        return 0
+
+    raise RuntimeError("unreachable")
+
+
+if __name__ == "__main__":
+    raise SystemExit(main())