better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/session-storage.md

@@ -0,0 +1,523 @@
+# Session Storage
+
+Hermes Agent uses a SQLite database (`~/.hermes/state.db`) to persist session
+metadata, full message history, and model configuration across CLI and gateway
+sessions. This replaces the earlier per-session JSONL file approach.
+
+Source files: `hermes_state.py` (facade) plus the `hermes_state_*.py` siblings (schema, fts, search, compression, portability, gateway, ...)
+
+## Hermes home and profile isolation
+
+`get_hermes_home()` is the authoritative filesystem resolver for state and
+configuration. It uses a context-local override first, then the `HERMES_HOME`
+environment variable, and finally the platform default (`~/.hermes` on macOS
+and Linux; `%LOCALAPPDATA%/hermes` on Windows). Consequently, the default
+database is always `get_hermes_home() / "state.db"`, not a path that callers
+should hard-code as `~/.hermes/state.db`.
+
+Named profiles are isolated directories: a profile named `coder`, for example,
+uses `<default Hermes root>/profiles/coder/` and therefore has its own
+`state.db`, configuration, logs, and other profile-scoped state. A process that
+creates a database, reads configuration, or starts a child process for a
+profile must retain or pass that profile's `HERMES_HOME`; falling back to the
+default root mixes the wrong profile's state into the operation.
+
+The CLI bootstrap calls `_apply_profile_override()` before importing the rest
+of Hermes. An explicit `--profile`/`-p` resolves that profile and writes the
+resolved directory to `HERMES_HOME`. Without an explicit selector, a
+profile-specific `HERMES_HOME` is preserved; otherwise the bootstrap can use
+the default root's active-profile selection. `HOME` only determines the
+platform default used when no context override or `HERMES_HOME` is available.
+Changing `HOME` is not a safe way to select a named profile. In particular, a
+subprocess that drops `HERMES_HOME` can fall back to the default profile even
+when another profile is active, so subprocess spawners should pass
+`HERMES_HOME` explicitly.
+
+Use `display_hermes_home()` only for user-facing text. It formats the resolved
+home relative to the user's home directory when possible (for example,
+`~/.hermes/profiles/coder`); it does not provide a separate resolution rule.
+
+### Test isolation guard
+
+Tests must use a temporary `HERMES_HOME` or an explicit temporary database
+path. The live-system guard raises before a test-context process opens a
+production `state.db` under the real default Hermes root or a real named
+profile, preventing fixture data or SQLite side effects from reaching a live
+installation.
+
+`HERMES_STATE_DB_GUARD_BYPASS=1` is a test-only escape hatch for a spawned
+child process that genuinely must access the live database. The equivalent
+in-process escape hatch is `@pytest.mark.live_system_guard_bypass`. Do not set
+either bypass in normal Hermes commands, development shells, or application
+configuration: it disables the guard (a hard `RuntimeError`) that protects live
+session history, and a shell that exports it hands the bypass to every later
+pytest run.
+
+### Desktop profile isolation and compaction generations
+
+Each named profile stores its transcript in its own `$HERMES_HOME/state.db`,
+including when one `hermes serve` process serves several profiles. In-session
+agent rebuilds (Bot Chat capability refresh and `tools.configure`) must retain
+that session's database handle and bind its profile home during construction.
+Releasing the outgoing agent must not close the handle inherited by its replacement.
+`tools.configure` resolves configuration from the live session's `profile_home`,
+even when the client supplies only `session_id`. Rebuilds prepare model configuration
+before allocating a replacement, then install the agent and transfer ownership
+together; preparation failure leaves the existing agent responsible for teardown.
+Explicit profiles that cannot be resolved or whose directory has disappeared fail
+before accessing launch configuration or history. A stale `tools.configure`
+session ID likewise returns `session not found` without changing configuration;
+omitting the session ID still supports the global settings operation.
+
+In-place compaction archives old rows with `active=0` and inserts the retained
+context as `active=1` rows. A protected message can therefore legitimately appear
+in both generations with identical content and timestamp. Do not delete these
+archive rows as duplicates. Diagnose duplicate *live* writes using `active=1`,
+and check the database's profile as well as the session ID when investigating
+history that appears to revert.
+
+
+
+## Codex app-server input ownership
+
+The agent persists an accepted user input before starting its Codex turn. Codex
+then projects that input as a leading `userMessage` notification. At the runtime
+splice boundary, Hermes excludes only that leading item when it exactly matches
+the text serialized into `turn/start`, including rich-input coercion. Later or
+nonmatching user events remain intact, as do separately accepted identical turns.
+This also applies to synthetic/keyless input; it does not depend on a platform
+message ID. Existing historical duplicates are not rewritten. The gateway skips
+its transcript write when the agent reports that it owns persistence.
+
+## Gateway exception-path input ownership
+
+A gateway exception can occur before agent construction or after its input reaches
+SQLite. The gateway gives the accepted input an owner marker in the existing
+`display_metadata` sidecar and passes it through the agent's normal persistence
+path. Provider messages never contain this metadata. Platform markers namespace
+the inbound message ID by platform, profile, scope, chat, and thread; the original
+`platform_message_id` remains unchanged for quote/reply resolution. Keyless turns
+receive a fresh marker, even for identical text and timestamps.
+
+The exception writer probes only for that marker, following the published reroute
+and canonical live compression successor, then compression ancestors. Active rows
+and compaction archives count; undone rows, observed input, and unrelated writers
+do not. An unrelated process writing the same session cannot suppress this turn.
+No whole-history baseline or archived message-body allocation is needed. Failed
+ownership reads do not authorize a speculative append; ordinary history-read
+failures retain the existing history-unavailable response.
+
+Normal agent-owned persistence is unchanged. This is failure-writer arbitration,
+not universal exactly-once delivery, content deduplication, or a schema migration.
+Historical rows are not rewritten; unmarked historical inputs cannot establish
+ownership for a redelivered event.
+
+## Architecture Overview
+
+```
+~/.hermes/state.db (SQLite, WAL mode)
+├── sessions              — Session metadata, token counts, billing
+├── messages              — Full message history per session
+├── session_model_usage   — Per-model/per-task usage attribution rows
+├── messages_fts          — FTS5 virtual table (content + tool_name + tool_calls)
+├── messages_fts_trigram  — FTS5 virtual table with trigram tokenizer (CJK / substring search)
+├── messages_fts_cjk      — FTS5 virtual table with cjk_unicode61 tokenizer
+├── state_meta            — Key/value metadata table
+├── gateway_routing       — Gateway routing metadata
+├── compression_locks     — Cross-process compression locking
+├── async_delegations     — Async delegation bookkeeping
+├── delivery_obligations  — Gateway outbox (owed replies); created lazily by gateway/delivery_ledger.py
+└── schema_version        — Single-row table tracking migration state
+```
+
+`hermes sessions recover` copies the row-bearing tables above into the
+recovered database (FTS indexes and `schema_version` are regenerated), including
+the lazily-created `delivery_obligations` ledger when the source has one — its
+row count is verified like `sessions`/`messages`.
+
+Key design decisions:
+- **WAL mode** for concurrent readers + one writer (gateway multi-platform)
+- **FTS5 virtual table** for fast text search across all session messages
+- **Session lineage** via `parent_session_id` chains (compression-triggered splits)
+- **Source tagging** (`cli`, `telegram`, `discord`, etc.) for platform filtering
+- Batch runner and RL trajectories are NOT stored here (separate systems)
+
+
+## SQLite Schema
+
+### Sessions Table
+
+Abridged — see `SCHEMA_SQL` in `hermes_state_common.py` (applied by `hermes_state_schema.py`) for the full current column list
+(which also includes gateway routing metadata such as `session_key`, `chat_id`,
+`chat_type`, `thread_id`, `display_name`, `origin_json`, `expiry_finalized`,
+workspace fields `cwd` / `git_branch` / `git_repo_root`, handoff and
+compression-failure fields, `profile_name`, `rewind_count`, `archived`, and
+`pinned`):
+
+```sql
+CREATE TABLE IF NOT EXISTS sessions (
+    id TEXT PRIMARY KEY,
+    source TEXT NOT NULL,
+    user_id TEXT,
+    model TEXT,
+    model_config TEXT,
+    system_prompt TEXT,
+    parent_session_id TEXT,
+    started_at REAL NOT NULL,
+    ended_at REAL,
+    end_reason TEXT,
+    message_count INTEGER DEFAULT 0,
+    tool_call_count INTEGER DEFAULT 0,
+    input_tokens INTEGER DEFAULT 0,
+    output_tokens INTEGER DEFAULT 0,
+    cache_read_tokens INTEGER DEFAULT 0,
+    cache_write_tokens INTEGER DEFAULT 0,
+    reasoning_tokens INTEGER DEFAULT 0,
+    billing_provider TEXT,
+    billing_base_url TEXT,
+    billing_mode TEXT,
+    estimated_cost_usd REAL,
+    actual_cost_usd REAL,
+    cost_status TEXT,
+    cost_source TEXT,
+    pricing_version TEXT,
+    title TEXT,
+    api_call_count INTEGER DEFAULT 0,
+    -- ... additional gateway/workspace/handoff/compression columns ...
+    FOREIGN KEY (parent_session_id) REFERENCES sessions(id)
+);
+
+CREATE INDEX IF NOT EXISTS idx_sessions_source ON sessions(source);
+CREATE INDEX IF NOT EXISTS idx_sessions_parent ON sessions(parent_session_id);
+CREATE INDEX IF NOT EXISTS idx_sessions_started ON sessions(started_at DESC);
+CREATE UNIQUE INDEX IF NOT EXISTS idx_sessions_title_unique
+    ON sessions(title) WHERE title IS NOT NULL;
+```
+
+### Messages Table
+
+Abridged — the full schema also includes `effect_disposition`,
+`platform_message_id`, `observed`, `active`, `compacted`, `api_content`,
+`display_kind`, and `display_metadata`:
+
+```sql
+CREATE TABLE IF NOT EXISTS messages (
+    id INTEGER PRIMARY KEY AUTOINCREMENT,
+    session_id TEXT NOT NULL REFERENCES sessions(id),
+    role TEXT NOT NULL,
+    content TEXT,
+    tool_call_id TEXT,
+    tool_calls TEXT,
+    tool_name TEXT,
+    timestamp REAL NOT NULL,
+    token_count INTEGER,
+    finish_reason TEXT,
+    reasoning TEXT,
+    reasoning_content TEXT,
+    reasoning_details TEXT,
+    codex_reasoning_items TEXT,
+    codex_message_items TEXT
+    -- ... additional display/compaction columns ...
+);
+
+CREATE INDEX IF NOT EXISTS idx_messages_session ON messages(session_id, timestamp);
+CREATE INDEX IF NOT EXISTS idx_messages_session_id ON messages(session_id, id);
+```
+
+Notes:
+- `tool_calls` is stored as a JSON string (serialized list of tool call objects)
+- `reasoning_details`, `codex_reasoning_items`, and `codex_message_items` are stored as JSON strings
+- Desktop history hydration retains assistant sidecars in both REST and JSON-RPC (`session.resume`, `session.activate`, `session.history`) projections, including rows with reasoning and tool calls. REST may return the SQLite JSON string while RPC returns decoded items; Desktop accepts both. A final Responses reply may live only in `codex_message_items` while `content` is empty. Canonical content still takes precedence, and analysis/commentary items are not promoted to reply text.
+- `reasoning` stores the raw reasoning text for providers that expose it
+- `api_content` is a byte-fidelity sidecar: the exact content string sent to the API for this message when it differs from `content` (ephemeral memory/plugin injections, persist overrides). It preserves the wire bytes for prompt-cache-stable replay — stored as sent, except lone surrogates, which sqlite3 cannot bind and which the conversation loop scrubs from every outgoing payload anyway. `NULL` means `content` was sent verbatim.
+- Timestamps are Unix epoch floats (`time.time()`)
+
+### FTS5 Full-Text Search
+
+```sql
+CREATE VIRTUAL TABLE IF NOT EXISTS messages_fts USING fts5(
+    content,
+    tool_name,
+    tool_calls,
+    content='messages',
+    content_rowid='id'
+);
+```
+
+The FTS5 table is kept in sync via three triggers that fire on INSERT, UPDATE,
+and DELETE of the `messages` table. The current triggers are gated on the
+`fts_rebuild_high_water` / `fts_rebuild_progress` markers in `state_meta` (so a
+background FTS rebuild can proceed without double-indexing) and cover all three
+indexed columns — see `SCHEMA_SQL` in `hermes_state_common.py` for the exact SQL.
+
+
+## Schema Version and Migrations
+
+Current schema version: **23**
+
+The `schema_version` table stores a single integer. Simple column additions are handled declaratively by `_reconcile_columns()` (which diffs live columns against `SCHEMA_SQL` and ADDs any missing ones). The version-gated chain is reserved for data migrations and index/FTS changes that can't be expressed declaratively:
+
+| Version | Change |
+|---------|--------|
+| 1 | Initial schema (sessions, messages, FTS5) |
+| 2 | Add `finish_reason` column to messages |
+| 3 | Add `title` column to sessions |
+| 4 | Add unique index on `title` (NULLs allowed, non-NULL must be unique) |
+| 5 | Add billing columns: `cache_read_tokens`, `cache_write_tokens`, `reasoning_tokens`, `billing_provider`, `billing_base_url`, `billing_mode`, `estimated_cost_usd`, `actual_cost_usd`, `cost_status`, `cost_source`, `pricing_version` |
+| 6 | Add reasoning columns to messages: `reasoning`, `reasoning_details`, `codex_reasoning_items` |
+| 7 | Add `reasoning_content` column to messages |
+| 8 | Add `api_call_count` column to sessions |
+| 9 | Add `codex_message_items` column to messages for Codex Responses message id/phase replay |
+| 10 | Add `messages_fts_trigram` virtual table (trigram tokenizer for CJK / substring search) and backfill existing rows |
+| 11 | Re-index `messages_fts` and `messages_fts_trigram` to cover `tool_name` + `tool_calls` and switch from external-content to inline mode; drop old triggers and backfill every message row |
+| 16 | Tag delegate subagent rows in `model_config` (`$._delegate_from`) so session pickers stay clean after parent deletes orphan them |
+| 18 | Gateway metadata consolidation — backfill `display_name` / `origin_json` / `expiry_finalized` from `sessions.json` |
+| 20 | Per-model usage attribution — seed `session_model_usage` rows from historical per-session aggregate totals |
+| 22 | Task-dimension usage attribution — rebuild `session_model_usage` so the `task` column participates in the PRIMARY KEY |
+| 23 | FTS storage redesign — external-content FTS tables replacing the v11 inline-mode copies (opt-in transition for existing DBs) |
+| 29 | Cron sessions leave the trigram (substring/CJK) index; `messages_fts_trigram_src` view + triggers filter on `sessions.source`, one-time rebuild purges historical rows |
+| 30 | Delegate-child (subagent) sessions leave the trigram index too — `source='subagent'` or the `$._delegate_from` marker (`FTS_TRIGRAM_SESSION_SQL`). Rows stay in `messages` and the standard `messages_fts` word index, so `session_search` still finds them; only the ~2.6× trigram shadow tables shrink. Same one-time rebuild as v29 |
+
+Versions not listed above were declarative column additions handled by `_reconcile_columns()` (version bump only, no data migration).
+
+Declarative column adds use `ALTER TABLE ADD COLUMN` wrapped in try/except to handle the column-already-exists case (idempotent). The version number is bumped after each successful migration block.
+
+
+## Write Contention Handling
+
+Multiple hermes processes (gateway + CLI sessions + worktree agents) share one
+`state.db`. The `SessionDB` class handles write contention with:
+
+- **Short SQLite timeout** (1 second) instead of the default 30s
+- **Application-level retry** with random jitter (20-150ms, up to 15 retries)
+- **BEGIN IMMEDIATE** transactions to surface lock contention at transaction start
+- **Periodic WAL checkpoints** every 50 successful writes (PASSIVE mode)
+
+This avoids the "convoy effect" where SQLite's deterministic internal backoff
+causes all competing writers to retry at the same intervals.
+
+```
+_WRITE_MAX_RETRIES = 15
+_WRITE_RETRY_MIN_S = 0.020   # 20ms
+_WRITE_RETRY_MAX_S = 0.150   # 150ms
+_CHECKPOINT_EVERY_N_WRITES = 50
+```
+
+
+## Common Operations
+
+### Initialize
+
+```python
+from hermes_state import SessionDB
+
+db = SessionDB()                           # Default: ~/.hermes/state.db
+db = SessionDB(db_path=Path("/tmp/test.db"))  # Custom path
+```
+
+### Create and Manage Sessions
+
+```python
+# Create a new session
+db.create_session(
+    session_id="sess_abc123",
+    source="cli",
+    model="anthropic/claude-sonnet-4.6",
+    user_id="user_1",
+    parent_session_id=None,  # or previous session ID for lineage
+)
+
+# End a session
+db.end_session("sess_abc123", end_reason="user_exit")
+
+# Reopen a session (clear ended_at/end_reason)
+db.reopen_session("sess_abc123")
+```
+
+### Store Messages
+
+```python
+msg_id = db.append_message(
+    session_id="sess_abc123",
+    role="assistant",
+    content="Here's the answer...",
+    tool_calls=[{"id": "call_1", "function": {"name": "terminal", "arguments": "{}"}}],
+    token_count=150,
+    finish_reason="stop",
+    reasoning="Let me think about this...",
+)
+```
+
+### Retrieve Messages
+
+```python
+# Raw messages with all metadata
+messages = db.get_messages("sess_abc123")
+
+# OpenAI conversation format (for API replay)
+conversation = db.get_messages_as_conversation("sess_abc123")
+# Returns: [{"role": "user", "content": "..."}, {"role": "assistant", ...}]
+```
+
+### Session Titles
+
+```python
+# Set a title (must be unique among non-NULL titles)
+db.set_session_title("sess_abc123", "Fix Docker Build")
+
+# Resolve by title (returns most recent in lineage)
+session_id = db.resolve_session_by_title("Fix Docker Build")
+
+# Auto-generate next title in lineage
+next_title = db.get_next_title_in_lineage("Fix Docker Build")
+# Returns: "Fix Docker Build #2"
+```
+
+
+## Full-Text Search
+
+The `search_messages()` method supports FTS5 query syntax with automatic
+sanitization of user input.
+
+### Basic Search
+
+```python
+results = db.search_messages("docker deployment")
+```
+
+### FTS5 Query Syntax
+
+| Syntax | Example | Meaning |
+|--------|---------|---------|
+| Keywords | `docker deployment` | Both terms (implicit AND) |
+| Quoted phrase | `"exact phrase"` | Exact phrase match |
+| Boolean OR | `docker OR kubernetes` | Either term |
+| Boolean NOT | `python NOT java` | Exclude term |
+| Prefix | `deploy*` | Prefix match |
+
+### Filtered Search
+
+```python
+# Search only CLI sessions
+results = db.search_messages("error", source_filter=["cli"])
+
+# Exclude gateway sessions
+results = db.search_messages("bug", exclude_sources=["telegram", "discord"])
+
+# Search only user messages
+results = db.search_messages("help", role_filter=["user"])
+```
+
+### Search Results Format
+
+Each result includes:
+- `id`, `session_id`, `role`, `timestamp`
+- `snippet` — FTS5-generated snippet with `>>>match<<<` markers
+- `context` — 1 message before and after the match (content truncated to 200 chars)
+- `source`, `model`, `session_started` — from the parent session
+
+The `_sanitize_fts5_query()` method handles edge cases:
+- Strips unmatched quotes and special characters
+- Wraps hyphenated terms in quotes (`chat-send` → `"chat-send"`)
+- Removes dangling boolean operators (`hello AND` → `hello`)
+
+
+## Session Lineage
+
+Sessions can form chains via `parent_session_id`. This happens when context
+compression triggers a session split in the gateway.
+
+### Query: Find Session Lineage
+
+```sql
+-- Find all ancestors of a session
+WITH RECURSIVE lineage AS (
+    SELECT * FROM sessions WHERE id = ?
+    UNION ALL
+    SELECT s.* FROM sessions s
+    JOIN lineage l ON s.id = l.parent_session_id
+)
+SELECT id, title, started_at, parent_session_id FROM lineage;
+
+-- Find all descendants of a session
+WITH RECURSIVE descendants AS (
+    SELECT * FROM sessions WHERE id = ?
+    UNION ALL
+    SELECT s.* FROM sessions s
+    JOIN descendants d ON s.parent_session_id = d.id
+)
+SELECT id, title, started_at FROM descendants;
+```
+
+### Query: Recent Sessions with Preview
+
+```sql
+SELECT s.*,
+    COALESCE(
+        (SELECT SUBSTR(m.content, 1, 63)
+         FROM messages m
+         WHERE m.session_id = s.id AND m.role = 'user' AND m.content IS NOT NULL
+         ORDER BY m.timestamp, m.id LIMIT 1),
+        ''
+    ) AS preview,
+    COALESCE(
+        (SELECT MAX(m2.timestamp) FROM messages m2 WHERE m2.session_id = s.id),
+        s.started_at
+    ) AS last_active
+FROM sessions s
+ORDER BY s.started_at DESC
+LIMIT 20;
+```
+
+### Query: Token Usage Statistics
+
+```sql
+-- Total tokens by model
+SELECT model,
+       COUNT(*) as session_count,
+       SUM(input_tokens) as total_input,
+       SUM(output_tokens) as total_output,
+       SUM(estimated_cost_usd) as total_cost
+FROM sessions
+WHERE model IS NOT NULL
+GROUP BY model
+ORDER BY total_cost DESC;
+
+-- Sessions with highest token usage
+SELECT id, title, model, input_tokens + output_tokens AS total_tokens,
+       estimated_cost_usd
+FROM sessions
+ORDER BY total_tokens DESC
+LIMIT 10;
+```
+
+
+## Export and Cleanup
+
+```python
+# Export a single session with messages
+data = db.export_session("sess_abc123")
+
+# Export all sessions (with messages) as list of dicts
+all_data = db.export_all(source="cli")
+
+# Delete old sessions (only ended sessions)
+deleted_count = db.prune_sessions(older_than_days=90)
+deleted_count = db.prune_sessions(older_than_days=30, source="telegram")
+
+# Clear messages but keep the session record
+db.clear_messages("sess_abc123")
+
+# Delete session and all messages
+db.delete_session("sess_abc123")
+```
+
+
+## Database Location
+
+Default path: `get_hermes_home() / "state.db"` — `~/.hermes/state.db` for the
+default profile, `~/.hermes/profiles/<name>/state.db` for a named profile, or
+wherever `HERMES_HOME` points (see [Hermes home and profile isolation](#hermes-home-and-profile-isolation)).
+
+The database file, WAL file (`state.db-wal`), and shared-memory file
+(`state.db-shm`) are all created in the same directory.

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/state-db-recovery.md

@@ -0,0 +1,107 @@
+---
+title: "State DB Recovery"
+description: "How Hermes recovers state.db when the FTS index or the file itself is corrupt"
+---
+
+# State database and FTS recovery
+
+`state.db` stores two different data classes:
+
+- `sessions` and `messages` are the canonical transcript.
+- `messages_fts*` tables and their sync triggers are derived search indexes.
+
+The derived indexes may be detached temporarily. They must not turn a live
+message write or search into an unbounded full-transcript rebuild.
+
+## Live behavior when FTS is corrupt
+
+If an FTS write or search reports the corruption error class, `SessionDB`:
+
+1. records the durable `fts_stale` marker;
+2. removes the FTS sync triggers in the same transaction;
+3. retries canonical writes without the derived-index sinks; and
+4. serves searches from canonical rows through the `LIKE` fallback.
+
+The failing live operation never runs `FTS5('rebuild')`. Existing recovery
+ownership remains unchanged: a later `SessionDB` open may rebuild under the
+cross-process admission lock and foreign-holder guard. If that guarded rebuild
+cannot run, FTS remains detached, canonical writes stay available, and
+`hermes doctor` reports the explicit repair command.
+
+## Live behavior when the file itself is corrupt
+
+If a live write reports bare `SQLITE_CORRUPT` / `SQLITE_NOTADB` (`database
+disk image is malformed`, `file is not a database`) with no FTS provenance,
+the damage is in a canonical B-tree, the schema, or the freelist. `SessionDB`
+then quarantines that handle (`StateDbCorruptError`):
+
+1. the failing write propagates the typed error and nothing is retried;
+2. later writes on the handle fail immediately without touching the file;
+3. the handle never reopens its connection after `close()`; and
+4. `close()` skips its explicit WAL checkpoint.
+
+Stopping the writes is the protection. In the field, a handle that kept
+writing for ~50 minutes after the first structural error checkpointed 15
+pages under the wrong page numbers on shutdown (page 1 received a
+`messages_fts_trigram_data` leaf) and turned a damaged-but-readable file into
+one that no longer opened at all. Skipping the explicit checkpoint is the
+second line of defence; on Python 3.12+ the quarantine also disables
+SQLite's own last-connection checkpoint (`SQLITE_DBCONFIG_NO_CKPT_ON_CLOSE`),
+so the `-wal` sidecar survives `close()` for forensics. On Python 3.11 that
+switch is unavailable and SQLite may still checkpoint once on close, so copy
+`state.db`, `state.db-wal` and `state.db-shm` together before restarting
+anything.
+
+The gateway and the agent flush path treat the quarantine like a replaced
+file: pending transcripts go to `sessions/<id>.jsonl` and the gateway
+`pending_messages/` spool instead of the retry queue, and the FTS one-shot
+rebuild never runs on the damaged file. The quarantine is per process — the
+shared handle stays poisoned for every holder until the process restarts on a
+repaired or restored file. Do not run `hermes doctor --fix` while the gateway
+is still up. Next steps:
+
+```bash
+hermes gateway stop
+HERMES_HOME="$HOME/.hermes" hermes sessions recover --source "$HOME/.hermes/state.db" --inspect-only
+# if recoverable:
+HERMES_HOME="$HOME/.hermes" hermes sessions recover --source "$HOME/.hermes/state.db" --output "$HOME/recovered-state.db"
+```
+
+or restore the newest snapshot from `state-snapshots/`.
+
+## Explicit repair
+
+Stop every process that can open the profile database before repairing it.
+Keep them stopped for the complete repair and verification window.
+
+```bash
+hermes gateway stop
+HERMES_HOME="$HOME/.hermes" hermes sessions repair --check-only
+HERMES_HOME="$HOME/.hermes" hermes sessions repair
+```
+
+`sessions repair` creates a SQLite backup by default and performs structural
+work through the repository's guarded snapshot-and-promotion path. Do not copy
+`state.db`, `state.db-wal`, and `state.db-shm` independently with `cp`; those
+files are one live SQLite image.
+
+After repair, verify the health probe, stale marker, trigger set, and canonical
+row counts before restarting the gateway:
+
+```bash
+HERMES_HOME="$HOME/.hermes" hermes sessions repair --check-only
+sqlite3 "$HOME/.hermes/state.db" \
+  "SELECT key, value FROM state_meta WHERE key = 'fts_stale';"
+sqlite3 "$HOME/.hermes/state.db" \
+  "SELECT type, name FROM sqlite_master WHERE name IN
+   ('messages_fts_insert','messages_fts_update','messages_fts_delete')
+   ORDER BY name;"
+sqlite3 "$HOME/.hermes/state.db" \
+  "SELECT 'sessions', COUNT(*) FROM sessions
+   UNION ALL SELECT 'messages', COUNT(*) FROM messages;"
+```
+
+The marker query should return no row, the expected FTS triggers should be
+present, and canonical row counts must not decrease. If repair fails, preserve
+both the live database and the reported backup; never delete canonical rows to
+make a derived-index error disappear.