better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/gateway-session-lifecycle.md

@@ -0,0 +1,621 @@
+---
+title: "Gateway Session Lifecycle"
+description: "SessionSource, SessionEntry, SessionStore, session-key rules and multi-user isolation in the gateway"
+---
+
+# Session Lifecycle
+
+> **Audience:** Gateway developers and maintainers
+> **Source files:** `gateway/session.py` (~1200 lines + `session_*.py` siblings), `gateway/run.py` (~5500 lines facade + `run_*.py` phases), `gateway/config.py`
+> **Last updated:** 2026-06-16
+
+## Overview
+
+A **session** represents a continuous conversation between the agent and one or more users on a
+messaging platform. The session lifecycle governs when conversations persist, when they reset,
+how they survive gateway restarts, and how messages queue during concurrent operations.
+
+The session system lives primarily in two modules:
+
+- `gateway/session.py` — Data model (`SessionSource`, `SessionEntry`, `SessionContext`),
+  key generation (`build_session_key`), and the main store (`SessionStore`).
+- `gateway/run.py` — Gateway runner (`GatewayRunner`) facade that wires sessions into the message
+  processing pipeline; the phases live in `run_*.py` siblings: session housekeeping
+  (`run_watchers.py`), agent caching (`run_agent_cache.py`), restart recovery
+  (`session_recovery.py`), and message queuing (`run_busy.py`).
+
+---
+
+## 1. SessionSource — Message Origin Descriptor
+
+`SessionSource` is a frozen record of *where a message came from*. It is attached to every
+incoming `MessageEvent` and used for routing, isolation, and context injection.
+
+### Fields
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `platform` | `Platform` | *(required)* | Enum identifying the messaging platform (telegram, discord, slack, signal, whatsapp, matrix, local, etc.). |
+| `chat_id` | `str` | *(required)* | Platform-level chat/group/channel identifier. Routed through the adapter's `chat_id_key` transform. |
+| `chat_name` | `Optional[str]` | `None` | Human-readable name of the chat or group. |
+| `chat_type` | `str` | `"dm"` | One of `"dm"`, `"group"`, `"channel"`, `"thread"`. Controls session key generation and isolation. |
+| `user_id` | `Optional[str]` | `None` | Platform-specific user identifier. Used for authorization and per-user session isolation. |
+| `user_name` | `Optional[str]` | `None` | Display name of the message author. Injected into system prompt. |
+| `thread_id` | `Optional[str]` | `None` | Forum topic / Discord thread / Slack thread identifier. Differentiates threaded conversations. |
+| `chat_topic` | `Optional[str]` | `None` | Channel topic or description (Discord channel topic, Slack channel purpose). |
+| `user_id_alt` | `Optional[str]` | `None` | Platform-specific stable alternative ID (Signal UUID, Feishu union_id). Used when `user_id` is ephemeral. |
+| `chat_id_alt` | `Optional[str]` | `None` | Signal group internal ID — maps a Signal group V2 identifier to its canonical form. |
+| `is_bot` | `bool` | `False` | True when the message author is a bot or webhook (Discord bots). |
+| `guild_id` | `Optional[str]` | `None` | Discord guild / Slack workspace / Matrix server scope identifier. |
+| `parent_chat_id` | `Optional[str]` | `None` | Parent channel when `chat_id` refers to a thread. |
+| `message_id` | `Optional[str]` | `None` | ID of the triggering message. Used for pin/reply/react operations and Discord ID injection. |
+| `role_authorized` | `bool` | `False` | True when adapter granted access via a platform role (not individual user ID). |
+
+### Key Methods
+
+- **`description`** (property: `str`) — Human-readable summary e.g. `"DM with Alice"`,
+  `"group: My Group, thread: 12345"`.
+- **`to_dict()` / `from_dict()`** — Serialization round-trip for persistence in `sessions.json`.
+
+---
+
+## 2. SessionEntry — Active Session Record
+
+`SessionEntry` is the per-session metadata record stored in memory and persisted to
+`{sessions_dir}/sessions.json`. Each entry maps a `session_key` to its current `session_id`.
+
+### Fields
+
+| Field | Type | Default | Description |
+|---|---|---|---|
+| `session_key` | `str` | *(required)* | Deterministic key identifying the conversation lane (see §4). |
+| `session_id` | `str` | *(required)* | Unique identifier for this specific conversation incarnation. Format: `YYYYMMDD_HHMMSS_<8hex>`. |
+| `created_at` | `datetime` | *(required)* | When this session incarnation was created. |
+| `updated_at` | `datetime` | *(required)* | Last activity timestamp used for resource housekeeping. |
+| `origin` | `Optional[SessionSource]` | `None` | The source that created this session, used for delivery routing. |
+| `display_name` | `Optional[str]` | `None` | Chat display name (sourced from `SessionSource.chat_name`). |
+| `platform` | `Optional[Platform]` | `None` | Platform enum persisted for routing across restarts. |
+| `chat_type` | `str` | `"dm"` | Chat type, also persisted for policy lookup. |
+| `input_tokens` | `int` | `0` | Cumulative LLM input (prompt) tokens consumed. |
+| `output_tokens` | `int` | `0` | Cumulative LLM output (completion) tokens consumed. |
+| `cache_read_tokens` | `int` | `0` | Cumulative prompt cache read tokens. |
+| `cache_write_tokens` | `int` | `0` | Cumulative prompt cache write tokens. |
+| `total_tokens` | `int` | `0` | Total token count across all turns. |
+| `estimated_cost_usd` | `float` | `0.0` | Estimated cumulative USD cost. |
+| `cost_status` | `str` | `"unknown"` | Cost tracking status label. |
+| `last_prompt_tokens` | `int` | `0` | Last API-reported prompt token count. Used for accurate compression pre-check. |
+
+### Boolean Flags (State Machine)
+
+SessionEntry has several boolean flags that form a simple state machine governing session
+behavior on the next access.
+
+| Flag | Type | Default | Description |
+|---|---|---|---|
+| `was_auto_reset` | `bool` | `False` | Set when explicit suspension causes a replacement session. Also retained for historical records. |
+| `auto_reset_reason` | `Optional[str]` | `None` | `"suspended"` for explicit suspension; older rows may retain historical reset reasons. |
+| `reset_had_activity` | `bool` | `False` | Whether the replaced session had prior activity. |
+| `is_fresh_reset` | `bool` | `False` | Set by explicit `/new` or `/reset`. Triggers topic/channel skill re-injection on first message. Distinguished from `was_auto_reset` to avoid misleading "session expired" notices. |
+| `expiry_finalized` | `bool` | `False` | Historical finalization fence retained for recovery; no timer writes it. |
+| `suspended` | `bool` | `False` | Hard force-wipe signal. Set by `/stop` or stuck-loop escalation (3+ consecutive restart failures). On next `get_or_create_session()`, forces a new `session_id` regardless of `resume_pending`. |
+| `resume_pending` | `bool` | `False` | Soft recovery marker. Set by `suspend_recently_active()` (crash recovery) or drain timeout. On next access, preserves the existing `session_id` — the user continues on the same transcript. Cleared after the next successful turn completes. |
+| `resume_reason` | `Optional[str]` | `None` | Why resume was marked: `"restart_timeout"`, `"shutdown_timeout"`, `"restart_interrupted"`. |
+| `last_resume_marked_at` | `Optional[datetime]` | `None` | Timestamp of the last resume-pending marking. |
+
+### State Transition Logic (get_or_create_session)
+
+```
+                    ┌──────────┐
+                    │  Incoming │
+                    │  Message  │
+                    └────┬─────┘
+                         │
+                         ▼
+              ┌──────────────────────┐
+              │  session_key exists  │──── No ──► Create fresh SessionEntry
+              │  AND !force_new      │
+              └──────────┬───────────┘
+                         │ Yes
+                         ▼
+              ┌──────────────────────┐
+              │  entry.suspended?    │──── Yes ──► Auto-reset: new session_id
+              └──────────┬───────────┘           (reason="suspended")
+                         │ No
+                         ▼
+              ┌──────────────────────┐
+              │ entry.resume_pending?│──── Yes ──► Return existing entry
+              └──────────┬───────────┘           (preserve session_id)
+                         │ No                     Clear flag on next successful turn
+                         ▼
+              ┌──────────────────────┐
+              │   Policy says reset? │──── Yes ──► Auto-reset: new session_id
+              └──────────┬───────────┘           (reason="idle"/"daily")
+                         │ No
+                         ▼
+              ┌──────────────────────┐
+              │  Return existing     │
+              │  entry, bump         │
+              │  updated_at          │
+              └──────────────────────┘
+```
+
+**Priority order in `get_or_create_session()`:**
+1. `suspended=True` → always force-reset (hard wipe)
+2. `resume_pending=True` → preserve session_id (soft recovery)
+3. No trigger → return existing entry (bump `updated_at`)
+
+---
+
+## 3. SessionStore — Storage and Operations
+
+`SessionStore` is the main storage layer. It maintains an in-memory dict (`_entries`) persisted
+to `sessions.json`, with SQLite (`SessionDB`) as the canonical store for session metadata and
+message transcripts.
+
+### Constructor
+
+```python
+SessionStore(sessions_dir: Path, config: GatewayConfig, has_active_processes_fn=None)
+```
+
+- `sessions_dir` — Directory where `sessions.json` lives.
+- `config` — `GatewayConfig` instance for routing and housekeeping settings.
+- `has_active_processes_fn` — Optional callback keyed by `session_key` to check for running
+  background processes. Sessions with active processes are protected from routing-entry pruning.
+
+### Operations (Methods)
+
+| Method | Description |
+|---|---|
+| `get_or_create_session(source, force_new=False)` | Core entry point. Returns existing or creates new `SessionEntry`. Evaluates explicit suspension and restart recovery state. Creates/ends SQLite records. |
+| `update_session(session_key, last_prompt_tokens=None)` | Lightweight metadata update after an interaction. Bumps `updated_at`, optionally records `last_prompt_tokens`. |
+| `reset_session(session_key, display_name=None)` | Explicit reset (from `/new` or `/reset`). Creates new `session_id`, sets `is_fresh_reset=True`. Ends old SQLite session, creates new one. |
+| `switch_session(session_key, target_session_id)` | Switch to a different existing session ID (from `/resume`). Ends current SQLite session, reopens target. |
+| `suspend_session(session_key)` | Mark session as `suspended=True` (from `/stop`). Forces auto-reset on next access. |
+| `mark_resume_pending(session_key, reason)` | Mark session as `resume_pending=True` (from drain timeout). Preserves session_id on next access. Will NOT override `suspended=True`. |
+| `clear_resume_pending(session_key)` | Clear `resume_pending` after a successful resumed turn. Called from gateway after `run_conversation()` returns. |
+| `suspend_recently_active(max_age_seconds=120)` | Crash recovery: mark recently-active sessions as `resume_pending=True`. Skips already-pending and already-suspended entries. Called on startup after unclean shutdown. |
+| `prune_old_entries(max_age_days)` | Drop entries older than `max_age_days` (based on `updated_at`). Skips `suspended` entries and sessions with active processes. |
+| `list_sessions(active_minutes=None)` | Return all sessions, optionally filtered by recent activity. Sorted by `updated_at` descending. |
+| `lookup_by_session_id(session_id)` | Find the active `SessionEntry` for a persisted session ID. |
+| `has_any_sessions()` | Check if any sessions have ever been created (uses SQLite for history, not just in-memory dict). |
+| `append_to_transcript(session_id, message, skip_db=False)` | Append a message to SQLite transcript. `skip_db=True` prevents duplicate writes when the agent already persisted. |
+| `rewrite_transcript(session_id, messages)` | Full replacement of session transcript (used by `/retry`, `/undo`, `/compress`). |
+| `load_transcript(session_id)` | Load all messages from a session's SQLite transcript. |
+| `rewind_session(session_id, n=1)` | Back up `n` user turns via soft-delete (keeps audit trail); thin wrapper over `SessionDB.rewind_user_turn` (`hermes_state_rewind.py`), the one rewind shared with CLI `/undo`/`/retry` and the TUI. Returns `{rewound_count, turns_undone, target_text}`. |
+
+### Internal Helpers
+
+- `_ensure_loaded()` / `_ensure_loaded_locked()` — Load `sessions.json` into `_entries` dict.
+- `_save()` — Atomic write to `sessions.json` via temp file + `atomic_replace`.
+- `_generate_session_key(source)` — Delegates to `build_session_key()` with config params.
+
+### Storage Layout
+
+```
+{sessions_dir}/
+  sessions.json          # In-memory _entries dict, persisted as JSON
+                           Maps session_key → SessionEntry (metadata only)
+  {session_id}.jsonl     # (Legacy, removed in spec 002)
+```
+
+The canonical transcript store is SQLite via `SessionDB` (from `hermes_state`). The
+`sessions.json` file persists the `session_key → session_id` mapping and entry metadata
+(flags, timestamps, token counts). If SQLite is unavailable, the store falls back to
+JSONL, but this is a degradation path.
+
+---
+
+## 4. SessionKey Generation Rules
+
+Session keys are deterministic strings that identify a conversation lane. They are generated
+by `build_session_key(source, group_sessions_per_user, thread_sessions_per_user)`.
+
+### Key Format
+
+```
+agent:main:{platform}:{chat_type}[:{chat_id}][:{thread_id}][:{participant_id}]
+```
+
+### DM Rules
+
+| Scenario | Key |
+|---|---|
+| DM with chat_id | `agent:main:telegram:dm:12345` |
+| DM with chat_id + thread | `agent:main:telegram:dm:12345:thread_678` |
+| DM without chat_id, with participant_id | `agent:main:signal:dm:user_abc` |
+| DM without chat_id or participant_id | `agent:main:telegram:dm` |
+| WhatsApp DM (canonicalized) | `agent:main:whatsapp:dm:{canonical_number}` |
+
+- DMs always include `chat_id` when present, isolating each private conversation.
+- `thread_id` further differentiates threaded DMs within the same DM chat.
+- Without `chat_id`, falls back to `user_id_alt` or `user_id` as participant_id.
+- Without any identifier, all DMs on that platform collapse to one shared session.
+
+### Group/Channel Rules
+
+| Scenario | Key |
+|---|---|
+| Group chat | `agent:main:telegram:group:-10012345` |
+| Group chat, per-user isolation | `agent:main:telegram:group:-10012345:user_abc` |
+| Thread in group, shared | `agent:main:discord:group:12345:thread_678` |
+| Thread in group, per-user | `agent:main:discord:group:12345:thread_678:user_abc` |
+| Channel | `agent:main:slack:channel:C12345` |
+| WhatsApp group (canonicalized) | `agent:main:whatsapp:group:{canonical_id}:{participant}` |
+
+- `chat_id` identifies the parent group/channel.
+- `thread_id` differentiates threads within that parent.
+- **Per-user isolation** (append `participant_id`) is controlled by:
+  - `group_sessions_per_user` (default: `True`) — group/channel sessions are isolated.
+  - `thread_sessions_per_user` (default: `False`) — threads are **shared** by default
+    (Telegram forum topics, Discord threads, Slack threads all share one session per thread).
+- `participant_id` = `user_id_alt` or `user_id` (in that priority).
+- WhatsApp identifiers are canonicalized to handle JID/LID alias flips.
+
+### Special Case: WhatApp
+
+WhatsApp phone numbers go through `canonical_whatsapp_identifier()` which strips the
+`@s.whatsapp.net` suffix and normalizes to E.164 format. This prevents session fragmentation
+when the bridge returns different alias forms of the same phone number.
+
+---
+
+## 5. Multi-User Isolation Strategy
+
+Multi-user isolation determines whether multiple users in the same chat share a conversation
+or each get their own private session.
+
+### Decision Logic (`is_shared_multi_user_session`)
+
+```python
+def is_shared_multi_user_session(source, *, group_sessions_per_user, thread_sessions_per_user):
+    if source.chat_type == "dm":
+        return False  # DMs are always private
+    if source.thread_id:
+        return not thread_sessions_per_user  # Threads: shared unless per-user
+    return not group_sessions_per_user       # Groups: isolated unless shared
+```
+
+### Summary
+
+| Chat Type | Default | Config Control |
+|---|---|---|
+| DM | Private (never shared) | N/A |
+| Group/Channel | Per-user isolation | `group_sessions_per_user` (default: True) |
+| Thread (forum, discord) | Shared (all participants see same context) | `thread_sessions_per_user` (default: False) |
+
+### Impact on System Prompt
+
+When `shared_multi_user_session=True`, the system prompt omits a fixed user name and instead
+states: *"Multi-user \{thread|session\} — messages are prefixed with [sender name]. Multiple
+users may participate."* Individual sender names are prefixed on each user message by the
+gateway at runtime, preserving prompt caching (the system prompt doesn't change per-turn).
+
+---
+
+## 6. Explicit Conversation Boundaries
+
+Inactivity and wall-clock time never rotate a conversation. `/new` and `/reset`
+create an explicit boundary; context compression continues to manage long histories.
+Legacy timer configuration is ignored. The existing `SessionResetPolicy` datatype
+is inert compatibility data, not a runtime policy.
+
+Explicit suspension still creates a boundary on the next inbound turn. Recovery
+respects explicit and historical finalized boundaries rather than reopening them.
+Resource-only eviction and WebSocket orphan reaping leave conversations resumable.
+
+---
+
+## 7. Restart Recovery Flow
+
+The restart recovery system ensures that in-flight sessions are preserved across gateway
+restarts, crashes, and drain timeouts. It is the solution to issue #7536.
+
+### Startup Recovery Sequence
+
+```
+Gateway starts
+       │
+       ▼
+┌───────────────────────────────┐
+│ Check for .clean_shutdown     │── Exists? ──► Skip suspension (clean exit)
+│ marker                        │
+└───────────────────────────────┘
+       │ Missing
+       ▼
+┌───────────────────────────────┐
+│ session_store                 │── Marks sessions updated within
+│ .suspend_recently_active()    │   last 120 seconds as resume_pending
+└───────────────────────────────┘
+       │
+       ▼
+┌───────────────────────────────┐
+│ _suspend_stuck_loop_sessions()│── Suspends sessions that have been
+│                               │   active across 3+ restarts
+└───────────────────────────────┘
+       │
+       ▼
+┌───────────────────────────────┐
+│ Queue inbound messages while  │
+│ startup restore runs          │
+│ (_startup_restore_in_progress)│
+└───────────────────────────────┘
+       │
+       ▼
+┌───────────────────────────────┐
+│ For each adapter, find        │
+│ resume_pending sessions →     │
+│ synthesize MessageEvent and   │
+│ run _handle_message to let    │
+│ the agent auto-continue       │
+└───────────────────────────────┘
+```
+
+### suspend_recently_active(max_age_seconds=120)
+
+Called on gateway startup when no `.clean_shutdown` marker exists (indicating a crash or
+unexpected exit). For each session updated within the last 120 seconds:
+
+- Sets `resume_pending=True`, `resume_reason="restart_interrupted"`,
+  `last_resume_marked_at=now`.
+- Skips entries already `resume_pending=True` (no double-mark).
+- Skips entries explicitly `suspended=True` (hard wipe should stay).
+
+### Stuck-Loop Detection (`_suspend_stuck_loop_sessions`)
+
+Counts consecutive restarts via a JSON file (`{HERMES_HOME}/restart_counts.json`). If a
+session has been active across 3+ consecutive restarts, it's auto-suspended so the user
+gets a clean slate.
+
+### Drain-Timeout Marking
+
+On graceful shutdown/restart, the drain system calls `mark_resume_pending()` for any
+session that was mid-turn when the drain timeout fired. Reasons:
+
+- `"restart_timeout"` — killed during restart drain
+- `"shutdown_timeout"` — killed during shutdown drain
+- `"restart_interrupted"` — crash recovery (from `suspend_recently_active`)
+
+All three reasons are in `_AUTO_RESUME_REASONS` and eligible for startup auto-resume.
+
+### Auto-Resume on Next Access
+
+When `get_or_create_session()` encounters `resume_pending=True`:
+
+1. It returns the existing entry **without** creating a new `session_id`.
+2. The existing transcript is loaded intact.
+3. The marking is not cleared here — it survives until the next successful turn
+   completes (`clear_resume_pending()` is called from the gateway after
+   `run_conversation()` returns a real response).
+4. If the resumed turn is interrupted again, the `resume_pending` flag remains set,
+   and the next restart will retry. The stuck-loop counter handles terminal escalation
+   (3 retries → suspended).
+
+### Clean Shutdown Marker (`.clean_shutdown`)
+
+Written at the end of a graceful shutdown. On next startup:
+
+- If present: skip `suspend_recently_active()` entirely. Active agents were already
+  drained, so no sessions are stuck.
+- Then delete the marker.
+
+This prevents unwanted auto-resets after `hermes update`, `hermes gateway restart`,
+or `/restart`.
+
+---
+
+## 8. Message Queuing Flow
+
+The message queuing system handles two scenarios:
+
+1. **Interrupt follow-ups** — When a user sends multiple messages while the agent is
+   processing, subsequent messages are queued as single-slot pending messages.
+2. **`/queue` FIFO** — Explicit `/queue` commands that must each produce their own full
+   agent turn, in order, without merging.
+
+### Data Structures
+
+```
+adapter._pending_messages: Dict[session_key, MessageEvent]
+    └── Single "next-up" slot per session. Overwritten on repeat sends
+        (burst collapse). Shared with photo-burst follow-ups.
+
+self._queued_events: Dict[session_key, List[MessageEvent]]
+    └── Overflow buffer. Each /queue invocation appends here when the
+        slot is occupied. Promoted one-at-a-time after each drain.
+```
+
+### Enqueue (`_enqueue_fifo`)
+
+```
+_enqueue_fifo(session_key, event, adapter)
+       │
+       ▼
+┌───────────────────────────────────────┐
+│ Is slot free?                         │
+│ (session_key NOT in _pending_messages)│── Yes ──► Place event in slot
+└───────────────────────────────────────┘
+       │ No
+       ▼
+Append to _queued_events[session_key] (overflow tail)
+```
+
+### Dequeue / Promotion (`_promote_queued_event`)
+
+Called at the drain site after the slot was consumed. If there's an overflow item:
+
+- When `pending_event is None` (slot was empty), return overflow head as the new event.
+- When `pending_event` exists, stage overflow head in the slot for the next recursion.
+- If no adapter available, push back to `_queued_events` (don't silently drop).
+
+### Queue Depth
+
+`_queue_depth(session_key, adapter)` returns `len(overflow) + (1 if slot occupied else 0)`.
+
+### Clearing
+
+Queued events for a session are cleared on `/new` and `/reset` (via `_handle_reset_command`).
+
+### FIFO Invariant
+
+Each `/queue` invocation produces exactly one full agent turn, in FIFO order, with no
+merging. The single-slot `_pending_messages` + overflow `_queued_events` design ensures
+that repeated sends during an active turn don't cause out-of-order processing.
+
+---
+
+## 9. Session Context Injection
+
+`SessionContext` is built from a `SessionSource` and `GatewayConfig` and injected into the
+agent's system prompt. It tells the agent:
+
+- Where the current message came from
+- What platforms are connected
+- Where it can deliver scheduled task outputs
+- Whether this is a shared multi-user session
+
+### Construction (`build_session_context`)
+
+```python
+def build_session_context(source, config, session_entry=None) -> SessionContext
+```
+
+1. Collects connected platforms from config.
+2. Collects home channels for each platform.
+3. Determines `shared_multi_user_session` via `is_shared_multi_user_session()`.
+4. Attaches session metadata (key, id, timestamps) if `session_entry` is provided.
+
+### PII Redaction (`build_session_context_prompt`)
+
+The dynamic system prompt section (`## Current Session Context`) can optionally redact
+personally identifiable information before sending to the LLM:
+
+- User IDs → `user_<12hex>` (SHA-256 prefix)
+- Chat IDs → `<platform>:<12hex>` or just `<12hex>`
+- Platforms excluded from redaction: Discord (needs raw IDs for `@mentions`),
+  and any plugin-registered platform not marked `pii_safe`.
+
+Redaction applies only to the system prompt text. Routing, session keys, and adapter
+operations always use the original values.
+
+---
+
+## 10. Background Housekeeping
+
+The `_session_housekeeping_watcher` periodically sweeps idle cached agents, sheds
+cache entries under memory pressure, and prunes old routing entries hourly.
+It never ends a transcript because of inactivity or the time of day.
+
+TTL, LRU and pressure eviction commit the live transcript to memory providers before
+soft-releasing clients. Active turns remain protected; terminal, browser and background
+process resources survive soft release. Routing-entry pruning preserves the canonical
+SQLite transcript, and live processes protect their routing entries from pruning.
+Historical `expiry_finalized` flags remain recovery fences but are no longer written
+by a timer watcher.
+
+---
+
+## 11. Agent Cache
+
+The gateway maintains an LRU cache of `AIAgent` instances keyed by `session_key` to
+preserve prompt caching across turns.
+
+### Cache Properties
+
+- **Max size:** 128 entries (`agent.agent_cache.max_size`, default `_AGENT_CACHE_MAX_SIZE`).
+- **Eviction policy:** Least-recently-used (LRU via `OrderedDict`).
+- **Idle TTL:** 3600s (1h) — `agent.agent_cache.idle_ttl_secs`, enforced by
+  `_session_housekeeping_watcher`.
+- **Memory budget:** `agent.agent_cache.memory_high_mb` (default `auto`) — see below.
+- **Lock:** `_agent_cache_lock` (threading) for thread safety.
+
+### Memory-Pressure Eviction
+
+A cached agent pins `_session_messages`, the full live transcript including tool
+outputs — tens of MB on a session with 100+ tool calls. The entry cap and the idle
+TTL are both blind to that: a gateway serving many chats keeps every warm transcript
+resident (agents that took a turn within the TTL are never idle-swept), so RSS climbs until
+the cgroup throttles and SIGTERM can no longer flush inside systemd's stop timeout
+(#80764).
+
+`_sweep_agent_cache_under_pressure()` is the valve. Each watcher tick it compares anonymous
+memory against `memory_high_mb` — the cgroup's own `memory.stat` `anon` when the gateway runs
+under a cgroup limit (the scope the budget is charged against, so same-unit children such as
+`execute_code` kernels count; #110549), otherwise the process's own anonymous RSS; over budget, it evicts LRU agents
+through the same soft path the cap enforcer uses (`_commit_then_release_soft`), then
+runs `malloc_trim` so the freed arenas actually return to the OS. Evicted sessions
+rebuild their transcript from the persisted session on the next turn.
+
+Three classes of session are never shed:
+
+- agents currently mid-turn (their clients and sandboxes are in use);
+- the `protect_recent` most-recently-used sessions (their prompt cache is worth the
+  most);
+- any session whose live transcript has not finished reaching disk —
+  `transcript_persistence_caught_up()` compares `_last_flushed_db_idx` against
+  `len(_session_messages)`, the same divergence the FTS write-corruption guard reacts
+  to when it preserves live history over a lagging transcript.
+
+`memory_high_mb: auto` derives the budget from the cgroup limit the gateway runs under
+(`memory.high`, then `memory.max`, then cgroup v1), falling back to total RAM when
+uncapped. Set a number to pin it, or `0`/`off` to disable the pass entirely. Helpers
+live in `gateway/agent_cache_pressure.py`.
+
+### Cache Lifecycle
+
+```
+Message arrives
+    │
+    ▼
+get_or_create_session()  →  session_key obtained
+    │
+    ▼
+Lookup _agent_cache[session_key]
+    │
+    ├── Hit → move_to_end(), reuse AIAgent (preserves prompt cache)
+    │
+    └── Miss → create new AIAgent, store in cache
+                (if at capacity, popitem(last=False) evicts LRU entry)
+    │
+    ▼
+run_conversation()  →  agent processes message
+    │
+    ▼
+Housekeeping soft-releases idle agents without ending transcripts
+```
+
+### Cleanup Flow
+
+Resource eviction removes the cached agent and commits memory before soft-releasing
+clients. Full `_cleanup_agent_resources(agent)` teardown is reserved for actual
+conversation boundaries and shutdown.
+
+---
+
+## Appendix: Key Configuration
+
+| Config Key | Type | Default | Description |
+|---|---|---|---|
+| `group_sessions_per_user` | `bool` | `true` | Isolate group/channel sessions per user |
+| `thread_sessions_per_user` | `bool` | `false` | Isolate thread sessions per user |
+| `session_store_max_age_days` | `int` | `0` | Prune sessions older than N days (0=disabled) |
+| `agent.gateway_auto_continue_freshness` | `int` | `3600` | Seconds for resume freshness window |
+| `agent.gateway_timeout` | `int` | `1800` | Agent turn timeout (30 min default) |
+| `agent.agent_cache.max_size` | `int` | `128` | LRU entry cap on cached AIAgents |
+| `agent.agent_cache.idle_ttl_secs` | `int` | `3600` | Evict agents idle this long |
+| `agent.agent_cache.memory_high_mb` | `int`/`str` | `auto` | Anon-RSS budget above which LRU transcripts are shed |
+| `agent.agent_cache.max_evictions_per_pass` | `int` | `16` | Cap on sessions shed per pressure pass |
+| `agent.agent_cache.protect_recent` | `int` | `8` | MRU sessions the pressure pass never touches |
+
+## State database and FTS recovery
+
+The canonical transcript lives in the `sessions` and `messages` tables. FTS5
+tables and their sync triggers are derived indexes that can be detached and
+rebuilt without deleting canonical messages. See
+[State DB recovery](state-db-recovery.md) for the bounded live failure
+mode and the explicit repair procedure.
+
+### Conversation lifetime
+
+No idle or daily reset settings are supported. Explicit `/new` and `/reset`,
+compaction, suspension and crash recovery retain their separate lifecycle roles.