clawmem 0.14.0 → 0.15.0

package/docs/guides/systemd-services.md

@@ -0,0 +1,332 @@
+# Systemd services for ClawMem
+
+Keep ClawMem's AI agent memory services running automatically with systemd user units. This is important for GPU setups — if a llama-server crashes, ClawMem silently falls back to in-process inference via `node-llama-cpp` (Metal on Apple Silicon, Vulkan where available, CPU as last resort). With GPU acceleration (Metal/Vulkan) the fallback is fast; on CPU-only systems it is significantly slower. Systemd's `Restart=on-failure` ensures servers come back up automatically. To disable silent fallback entirely, set `CLAWMEM_NO_LOCAL_MODELS=true`.
+
+## Watcher service
+
+Monitors collections for file changes and re-indexes automatically:
+
+```bash
+cat > ~/.config/systemd/user/clawmem-watcher.service << 'EOF'
+[Unit]
+Description=ClawMem file watcher — auto-indexes on .md changes
+After=default.target
+
+[Service]
+Type=simple
+ExecStart=%h/clawmem/bin/clawmem watch
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=default.target
+EOF
+```
+
+## Embed timer
+
+Daily embedding sweep at 04:00 UTC:
+
+```bash
+cat > ~/.config/systemd/user/clawmem-embed.service << 'EOF'
+[Unit]
+Description=ClawMem embedding sweep
+
+[Service]
+Type=oneshot
+ExecStart=%h/clawmem/bin/clawmem embed
+EOF
+
+cat > ~/.config/systemd/user/clawmem-embed.timer << 'EOF'
+[Unit]
+Description=ClawMem daily embedding sweep
+
+[Timer]
+OnCalendar=*-*-* 04:00:00
+Persistent=true
+RandomizedDelaySec=300
+
+[Install]
+WantedBy=timers.target
+EOF
+```
+
+## Enable and start
+
+```bash
+mkdir -p ~/.config/systemd/user
+systemctl --user daemon-reload
+systemctl --user enable --now clawmem-watcher.service clawmem-embed.timer
+
+# Persist across reboots (start without login)
+loginctl enable-linger $(whoami)
+```
+
+## Verify
+
+```bash
+systemctl --user status clawmem-watcher.service
+systemctl --user status clawmem-embed.timer
+```
+
+## Background maintenance workers (v0.8.2)
+
+ClawMem ships two background workers that improve retrieval quality over time:
+
+- **Light consolidation lane** — every 5-10 min, enriches documents missing A-MEM metadata (Phase 1), merges near-duplicate observations (Phase 2), and synthesizes deductive observations from related decisions (Phase 3). Off by default.
+- **Heavy maintenance lane** — runs on a longer interval *only inside a configurable hour window*, batches stale-first work (least-recently-recalled documents first), and journals every attempt. Designed for overnight maintenance on large vaults. Off by default.
+
+As of v0.8.2 the canonical host for both lanes is the long-lived `clawmem-watcher.service` you just set up, so the heavy lane's quiet window actually sees a live worker every night regardless of whether any agent session is open. The per-session stdio MCP host (`clawmem mcp`) retains the same env-var gates as a fallback for non-watcher deployments, but emits a warning when heavy lane is enabled there.
+
+Both lanes share DB-backed `worker_leases` exclusivity, so running multiple host processes against the same vault is safe — only one worker can hold the lease for each lane at a time.
+
+### Enable via systemd drop-in
+
+Recommended approach: `systemctl --user edit clawmem-watcher.service` and paste the following. This creates `~/.config/systemd/user/clawmem-watcher.service.d/override.conf` without editing the main unit file, so a future `clawmem` upgrade that touches the unit file does not clobber your tuning.
+
+```ini
+[Service]
+# Light consolidation lane — drains enrichment backlog and runs Phase 2/3
+# consolidation. Hosted inside the long-lived watcher process, so it ticks
+# whenever the watcher is up. v0.8.2+ wraps every tick in a worker_leases
+# row so multi-host deployments are safe.
+Environment=CLAWMEM_ENABLE_CONSOLIDATION=true
+Environment=CLAWMEM_CONSOLIDATION_INTERVAL=600000
+
+# Heavy maintenance lane — quiet-hours stale-first batch consolidation +
+# deductive synthesis. Off-hours window in LOCAL time (not UTC).
+Environment=CLAWMEM_HEAVY_LANE=true
+Environment=CLAWMEM_HEAVY_LANE_INTERVAL=1800000
+Environment=CLAWMEM_HEAVY_LANE_WINDOW_START=2
+Environment=CLAWMEM_HEAVY_LANE_WINDOW_END=6
+```
+
+Then reload + restart:
+
+```bash
+systemctl --user daemon-reload
+systemctl --user restart clawmem-watcher.service
+journalctl --user -u clawmem-watcher.service -n 30 --no-pager | grep -E "Starting (consolidation|heavy)"
+# Expected:
+#   [watch] Starting consolidation worker (light lane, interval=600000ms)
+#   [consolidation] Worker started
+#   [watch] Starting heavy maintenance lane worker
+#   [heavy-lane] Starting worker (interval=1800000ms, window=2-6, ...)
+```
+
+### Tuning for your usage pattern
+
+The defaults (`CLAWMEM_CONSOLIDATION_INTERVAL=600000` / 10 min; heavy lane `CLAWMEM_HEAVY_LANE_INTERVAL=1800000` / 30 min, window 02:00-06:00 local) suit a single-developer workstation with overnight idle time. Adjust based on your actual usage:
+
+| Pattern | Light interval | Heavy window | Notes |
+|---|---|---|---|
+| **Single workstation, overnight idle** | 600000 (10 min) | 02:00-06:00 local | Default. Fine for vaults with up to a few thousand docs. |
+| **Always-on workstation, no idle window** | 900000 (15 min) | unset (no window — runs every interval) | Heavy lane will fire every 30 min regardless of hour. The query-rate gate (`CLAWMEM_HEAVY_LANE_MAX_USAGES=30`) skips ticks during active sessions, so it self-throttles. |
+| **Shared multi-user server** | 1200000 (20 min) | 03:00-05:00 local | Conservative — minimize background CPU/GPU contention. |
+| **Large vault (>10k docs), heavy ingestion** | 600000 (10 min) | 01:00-07:00 local | Wider quiet window to give Phase 2 + Phase 3 time to drain stale-first batches. Bump `CLAWMEM_HEAVY_LANE_OBS_LIMIT=200` and `CLAWMEM_HEAVY_LANE_DED_LIMIT=80` to process more per tick. |
+| **Laptop / ephemeral host** | unset (off) | unset (off) | Workers do not survive sleep/suspend cleanly. Run consolidation manually via `clawmem consolidate` instead, or accept the per-session-MCP fallback. |
+
+**Minimum intervals are clamped in code:** light lane is forced to ≥15 s (`CLAWMEM_CONSOLIDATION_INTERVAL=15000`), heavy lane to ≥30 s (`CLAWMEM_HEAVY_LANE_INTERVAL=30000`). Lower values are silently bumped up.
+
+**Quiet window semantics:** start/end hours are in **local time** (`new Date().getHours()`). Both bounds inclusive at start, exclusive at end. Supports midnight wrap (`START=22 END=6` = 22:00-06:00 across midnight). Either bound unset → no window (always run).
+
+### What to expect after the first restart
+
+Assuming you set both env vars and restarted the watcher around midday:
+
+- **+ a few seconds:** watcher logs `[consolidation] Worker started` and `[heavy-lane] Starting worker (...)`. Workers are scheduled but no tick has fired yet — `setInterval` waits one full interval before the first call.
+- **+ light interval (e.g. ~10 min):** first light-lane tick fires. Phase 1 enriches up to 3 unenriched docs per tick, so a backlog of N unenriched docs takes roughly `N/3 × interval` to drain. A 100-doc backlog at 10-min interval drains in ~5.5 hours.
+- **+ heavy interval (e.g. ~30 min):** first heavy-lane tick fires. If you are *outside* the quiet window, the lane journals `(phase=gate, status=skipped, reason=outside_window)` to `maintenance_runs` and waits for the next interval. This is normal and expected during the day.
+- **First night inside the quiet window:** heavy lane runs in earnest. Phase 2 (consolidation) and Phase 3 (deductive synthesis) each get their own `maintenance_runs` row per tick with `status=completed` and per-phase metrics.
+
+### Monitoring
+
+```bash
+# Light-lane drainage progress: how much enrichment backlog is left?
+sqlite3 -readonly ~/.cache/clawmem/index.sqlite \
+  "SELECT COUNT(*) AS total,
+          SUM(CASE WHEN amem_keywords IS NULL OR amem_keywords='' THEN 1 ELSE 0 END) AS unenriched
+     FROM documents WHERE active=1"
+
+# Heavy-lane journal: every scheduled attempt, including skipped ones
+sqlite3 -readonly ~/.cache/clawmem/index.sqlite \
+  "SELECT lane, phase, status, reason, started_at, finished_at
+     FROM maintenance_runs
+     WHERE lane = 'heavy'
+     ORDER BY id DESC LIMIT 10"
+
+# Why did the heavy lane skip its most recent tick?
+sqlite3 -readonly ~/.cache/clawmem/index.sqlite \
+  "SELECT status, reason, started_at FROM maintenance_runs
+     WHERE lane = 'heavy' AND phase = 'gate' AND status = 'skipped'
+     ORDER BY id DESC LIMIT 5"
+
+# Phase 2 / Phase 3 throughput over the last 7 days
+sqlite3 -readonly ~/.cache/clawmem/index.sqlite \
+  "SELECT phase, status, COUNT(*), SUM(processed_count), SUM(created_count)
+     FROM maintenance_runs
+     WHERE lane = 'heavy' AND started_at > datetime('now', '-7 days')
+     GROUP BY phase, status"
+
+# Are any leases stuck? (Should be empty unless a worker is mid-tick right now)
+sqlite3 -readonly ~/.cache/clawmem/index.sqlite \
+  "SELECT worker_name, acquired_at, expires_at FROM worker_leases"
+
+# Live worker activity in the watcher service
+journalctl --user -u clawmem-watcher.service -n 50 --no-pager | \
+  grep -E "(consolidation|heavy-lane|Worker)"
+```
+
+### Rollback
+
+To temporarily disable the workers without removing the drop-in, set both env vars to anything other than `"true"`:
+
+```bash
+systemctl --user edit clawmem-watcher.service
+# Change the values to "false" then save
+systemctl --user daemon-reload
+systemctl --user restart clawmem-watcher.service
+```
+
+To remove the drop-in entirely:
+
+```bash
+rm ~/.config/systemd/user/clawmem-watcher.service.d/override.conf
+systemctl --user daemon-reload
+systemctl --user restart clawmem-watcher.service
+```
+
+The vault state (existing `consolidated_observations`, deductive documents, `maintenance_runs` history) is preserved. Disabling the workers only stops further background work — it does not roll back any consolidation that already happened.
+
+For the architectural deep dive on how the workers operate, see [docs/concepts/architecture.md](../concepts/architecture.md) sections "Heavy maintenance lane (v0.8.0)" and "Dual-host worker architecture (v0.8.2)".
+
+## Remote GPU
+
+If GPU services run on a different machine, add environment overrides to both services:
+
+```ini
+[Service]
+Environment=CLAWMEM_EMBED_URL=http://gpu-host:8088
+Environment=CLAWMEM_LLM_URL=http://gpu-host:8089
+Environment=CLAWMEM_LLM_MODEL=qwen3
+Environment=CLAWMEM_RERANK_URL=http://gpu-host:8090
+```
+
+Or create a drop-in override:
+
+```bash
+mkdir -p ~/.config/systemd/user/clawmem-watcher.service.d
+cat > ~/.config/systemd/user/clawmem-watcher.service.d/gpu.conf << 'EOF'
+[Service]
+Environment=CLAWMEM_EMBED_URL=http://gpu-host:8088
+Environment=CLAWMEM_LLM_URL=http://gpu-host:8089
+Environment=CLAWMEM_LLM_MODEL=qwen3
+Environment=CLAWMEM_RERANK_URL=http://gpu-host:8090
+EOF
+systemctl --user daemon-reload
+systemctl --user restart clawmem-watcher.service
+```
+
+## REST API service (for OpenClaw)
+
+Required for OpenClaw agent tools and remote access. Optional for local MCP clients like Claude Code (which use MCP stdio directly).
+
+```bash
+cat > ~/.config/systemd/user/clawmem-serve.service << 'EOF'
+[Unit]
+Description=ClawMem REST API server
+After=default.target
+
+[Service]
+Type=simple
+ExecStart=%h/clawmem/bin/clawmem serve --port 7438
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=default.target
+EOF
+```
+
+For authenticated access, add `Environment=CLAWMEM_API_TOKEN=your-secret` to the `[Service]` section.
+
+Enable alongside the other services:
+
+```bash
+systemctl --user enable --now clawmem-serve.service
+```
+
+See the [REST API reference](../reference/rest-api.md) for endpoints and usage.
+
+## GPU service units
+
+The default stack's three inference servers can also run as systemd services:
+
+```bash
+# Example: embedding server
+cat > ~/.config/systemd/user/clawmem-embed-server.service << 'EOF'
+[Unit]
+Description=ClawMem embedding server (zembed-1)
+After=default.target
+
+[Service]
+Type=simple
+ExecStart=/usr/local/bin/llama-server \
+  -m %h/models/zembed-1-Q4_K_M.gguf \
+  --embeddings --port 8088 --host 0.0.0.0 -ngl 99 -c 8192 -b 2048 -ub 2048
+Restart=on-failure
+RestartSec=5
+
+[Install]
+WantedBy=default.target
+EOF
+```
+
+Repeat for the LLM (port 8089) and the **default** reranker (port 8090 — `qwen3-reranker-0.6B` via `--reranking`) with their respective models and flags.
+
+> **The SOTA reranker is not a `llama-server` unit.** The zerank-2 SOTA reranker runs as a transformers **sidecar** (a small container behind the same `/v1/rerank` contract), not a systemd `llama-server` instance — see [`extras/rerankers/zerank-2-seq/`](../../extras/rerankers/zerank-2-seq/). The old `zerank-2-Q4_K_M` GGUF served via `--reranking` is deprecated (llama.cpp drops zerank's score head → near-zero, uninformative scores).
+
+## Reranker health check (scheduled)
+
+`clawmem rerank-health` probes the reranker for **discrimination** (not just liveness) and exits non-zero when it is degenerate — catching the failure mode where a mis-converted reranker (e.g. the deprecated zerank-2 GGUF, whose llama.cpp conversion drops the score head) returns HTTP 200 + valid JSON but near-zero, non-discriminating scores, silently collapsing the final ranking to RRF. The same probe runs inside `clawmem doctor`; this scheduled unit alerts proactively without query traffic. Schedule it when the reranker is a remote sidecar that could be redeployed/reverted out from under you.
+
+```bash
+# clawmem-rerank-health.service — oneshot probe; exits 1 if the reranker is degenerate
+cat > ~/.config/systemd/user/clawmem-rerank-health.service << 'EOF'
+[Unit]
+Description=ClawMem reranker discrimination health check
+
+[Service]
+Type=oneshot
+# Hard outer bound — a hung reranker must not hang the check (the probe also times out per-request).
+TimeoutStartSec=120
+ExecStart=%h/clawmem/bin/clawmem rerank-health
+# Alert on failure: point this at your notifier unit (ntfy, mail, etc.).
+OnFailure=clawmem-rerank-health-alert@%n.service
+EOF
+
+# clawmem-rerank-health.timer — every 6h (the failure it guards is rare + infra-driven)
+cat > ~/.config/systemd/user/clawmem-rerank-health.timer << 'EOF'
+[Unit]
+Description=ClawMem reranker health check (every 6h)
+
+[Timer]
+OnCalendar=*-*-* 00/6:00:00
+Persistent=true
+RandomizedDelaySec=300
+
+[Install]
+WantedBy=timers.target
+EOF
+
+systemctl --user daemon-reload
+systemctl --user enable --now clawmem-rerank-health.timer
+```
+
+For remote GPU setups, add `Environment=CLAWMEM_RERANK_URL=http://host:8090` (+ embed/LLM) to the `.service`. `OnFailure=` is the primary alert path; the `curator-nudge` SessionStart hook is a secondary "you missed the page" surface. Run `clawmem rerank-health` (or `--json`) manually any time to check on demand.
+
+## Notes
+
+- `%h` in systemd units expands to the user's home directory
+- The watcher does NOT embed — it only indexes. The embed timer handles embeddings separately.
+- If clawmem is installed elsewhere, update `ExecStart` paths accordingly