agenr 1.2.0 → 1.4.0

package/CHANGELOG.md

@@ -1,5 +1,59 @@
 # Changelog
 
+## [1.4.0] - 2026-03-30
+
+Configurable summary models, surgeon personal knowledge protection, and documentation overhaul.
+
+### Added
+
+- **Configurable continuity and episode summary models.** New `continuityModel` and `episodeModel` fields in the OpenClaw plugin config (`plugins.entries.agenr.config`) allow overriding the model used for continuity and episode summary generation independently. Falls back to the agent's primary model when unset. Use a fast model like `openai/gpt-5.4-mini` for these structured extraction tasks instead of burning Opus tokens.
+- **Personal knowledge protection in surgeon.** The surgeon retirement pass now has explicit guidance that personal facts (family, pets, hardware, contacts, identity, physical environment) are durable by nature. Only retires personal entries when contradicted or clearly duplicated — not for low recall or moderate importance.
+- **Corpus age awareness in surgeon.** The `_meta` table now tracks `last_bulk_ingest_at`, surfaced via `get_health_stats`. The surgeon heavily discounts `recall_count = 0` when the corpus was rebuilt within 30 days, preventing mass retirements of freshly ingested entries.
+- **New documentation: `docs/EPISODES.md`.** Comprehensive episodic memory docs covering lifecycle, CLI usage, recall modes, temporal window parser, search modes, embeddings, session discovery, and architecture.
+- **New documentation: `docs/SURGEON.md`.** Comprehensive surgeon docs covering tools, CLI commands, dry-run vs apply, budget governance, configuration, protection thresholds, and audit history.
+
+### Changed
+
+- **Continuity summary timeout increased.** Inner timeout bumped from 15s to 30s, read-time wrapper from 20s to 35s. Prevents timeout failures when using slower models for continuity summaries.
+- **Updated `docs/RECALL.md`.** Added unified recall mode routing (`auto`/`entries`/`episodes`), auto-routing rules, temporal window parser reference, and episode search pipeline documentation.
+- **Updated `docs/INGEST.md`.** Added episode ingest section with full flag documentation, behavior differences from entry ingest, session discovery, surface reconstruction, and practical examples.
+- **Updated `README.md`.** Added episodic memory and surgeon to features list, CLI commands table, and new "How Episodes Work" and "How the Surgeon Works" sections with doc links.
+
+## [1.3.0] - 2026-03-30
+
+Episodic memory — session-level temporal recall for the brain.
+
+### Added
+
+- **Episodic memory system (Phases 0–6).** A new `episodes` table stores narrative summaries of what happened during each session, queryable by time range and semantic similarity. Gives the brain temporal awareness — "what happened yesterday", "what were we working on last week", "sessions about schema changes" all work.
+- **Episode ingest CLI.** `agenr ingest episodes <path>` scans OpenClaw session transcripts (including rotated `.reset.*` and `.deleted.*` files), generates episodic summaries, and writes episodes to the database. Supports `--recent`, `--regenerate`, `--dry-run`, `--concurrency`, `--embed-only`, and `--no-embed` flags.
+- **Session discovery via `sessions.json`.** Uses OpenClaw's session registry for authoritative metadata (surface, agentId, chatType) on active sessions. Falls back to transcript-based surface reconstruction for rotated files.
+- **Surface reconstruction from transcripts.** Detects session surface (webchat, telegram, signal, tui, subagent, heartbeat, cron) from Sender metadata blocks, Conversation info blocks, inbound_meta, and content heuristics. Integrated into the transcript parser to avoid double file reads.
+- **Agent ID derivation from directory path.** Falls back to parsing the OpenClaw directory structure (`agents/{agentId}/sessions/`) when registry metadata is unavailable.
+- **Unified recall with mode routing.** `agenr_recall` gains a `mode` parameter (`auto`, `entries`, `episodes`). Auto-routing uses three-band rules: temporal narrative → episodes, factual → entries, mixed → both. Results returned in separate sections with routing metadata.
+- **Calendar-aware temporal window parser.** Parses "today", "yesterday", "this/last week", "this/last month", "N days/weeks/months ago", "in March", "March 15th", "last Friday", and ISO dates into precise calendar intervals for episode search.
+- **Episode recall pipeline.** Pure temporal search via interval overlap scoring. No embedding dependency for basic temporal queries.
+- **Hybrid semantic episode search (Phase 6).** Episode embeddings stored at write time, vector index (`idx_episodes_embedding`) for cosine similarity search. Three modes: pure temporal, pure semantic, and hybrid (hard temporal filter + semantic rerank).
+- **Episode embedding backfill.** `agenr ingest episodes --embed-only` backfills embeddings for episodes missing them — no LLM calls, just embedding API.
+- **Episode model configuration.** New `episodeModel` config field in `agenr setup` / `agenr init` for episode summary generation model override, following the existing extraction/dedup/surgeon pattern.
+- **Episode writing at session start.** Predecessor episodes generated via `before_prompt_build` hook, best-effort with timeout. Backfill CLI is the canonical repair path.
+- **Parallel preflight parsing.** Episode ingest Stage 1 parses transcript files concurrently to handle large session directories (600+ files) without hanging.
+
+### Changed
+
+- **Continuity domain extraction.** Session handoff logic reorganized under `src/adapters/openclaw/session/continuity/` as a proper subdomain with clean public API. Continuity summaries (for session handoff) are now explicitly separate from episodic summaries (for temporal recall).
+- **Renamed session summaries to continuity summaries.** Clarifies the distinction between handoff artifacts and episodic memory artifacts throughout the codebase.
+- **Removed `before_reset` hook.** Does not fire for OpenClaw plugins — all episode and continuity logic uses `before_prompt_build` only.
+- **Semantic memory cleanup (Ref #7).** Removed `reflection` entry type, migrated `todo` to `tasks` table, renamed `event` to `milestone`, broadened type descriptions.
+- **Transcript file discovery.** Fixed regex to match rotated files with ISO timestamp dots (`.628Z`). Discovery now finds all 600+ files instead of only active `.jsonl` files.
+
+### Fixed
+
+- **Episode vector search ambiguous column.** Prefixed select columns with table alias in `episodeVectorSearch()` to resolve `id` ambiguity with `vector_top_k` join.
+- **Temporal parser ordinal dates.** "March 15th", "January 1st", "February 2nd" now parse correctly.
+- **Subagent/cron surface classification.** Session registry detects `:subagent:` and `:cron:` in session keys for correct surface tagging.
+- **Internal session skipping.** Agenr prompt pipeline sessions are filtered from episode generation.
+
 ## [1.2.0] - 2026-03-29
 
 The surgeon retirement pass — an autonomous agent that evaluates and retires stale knowledge entries.

package/README.md

@@ -14,18 +14,20 @@ Local-first, durable memory infrastructure for AI agents.
 
 ## What is agenr?
 
-agenr gives agents a persistent brain: a local SQLite database of durable knowledge that survives across sessions, tools, and agent restarts. Instead of relying on fragile prompt state or file-based scratch memory, agents can ingest transcripts, extract decisions and lessons, store them as typed entries, and recall them later with semantic search and memory-aware ranking.
+agenr gives agents a persistent brain: a local SQLite database of durable knowledge that survives across sessions, tools, and agent restarts. Instead of relying on fragile prompt state or file-based scratch memory, agents can ingest transcripts, extract decisions and lessons, store them as typed entries, generate episodic summaries of what happened, and recall them later with semantic search and memory-aware ranking.
 
-It exists because most agent runtimes forget everything important between sessions. Even when a tool has a built-in memory feature, it is often lossy, file-based, or tightly coupled to one surface. agenr keeps memory structured and queryable: facts, decisions, preferences, lessons, todos, events, and relationships live in one local store instead of getting flattened into prompt text.
+It exists because most agent runtimes forget everything important between sessions. Even when a tool has a built-in memory feature, it is often lossy, file-based, or tightly coupled to one surface. agenr keeps memory structured and queryable: facts, decisions, preferences, lessons, tasks, milestones, relationships, and session-level episodes live in one local store instead of getting flattened into prompt text.
 
-What makes agenr different is the combination of local-first storage, semantic embeddings, hybrid recall, and adapter-friendly architecture. The core is hexagonal, so multiple agent systems can share the same brain over time. Today the production adapter is the OpenClaw memory plugin, published separately as `@agenr/openclaw-plugin`, and the CLI provides offline ingest and recall against that same database.
+What makes agenr different is the combination of local-first storage, semantic embeddings, hybrid recall, episodic temporal memory, and adapter-friendly architecture. The core is hexagonal, so multiple agent systems can share the same brain over time. Today the production adapter is the OpenClaw memory plugin, published separately as `@agenr/openclaw-plugin`, and the CLI provides offline ingest, recall, and maintenance against that same database.
 
 ## Features
 
-- Hybrid recall: vector similarity, lexical FTS, temporal awareness, recency decay, and importance weighting.
+- Hybrid recall for durable knowledge: vector similarity, lexical FTS, temporal awareness, recency decay, and importance weighting.
+- Episodic memory: session-level summaries with temporal filtering and optional semantic episode search for questions like "what happened yesterday?"
 - LLM-powered knowledge extraction from conversation transcripts.
 - Semantic deduplication using exact hashes, normalized hashes, embeddings, and within-run clustering.
-- Session continuity with predecessor resolution, recent transcript tails, and LLM-generated session summaries.
+- Session continuity with predecessor resolution, recent transcript tails, and LLM-generated continuity summaries.
+- Surgeon retirement pass for corpus maintenance: inspect stale candidates, simulate recall impact, and retire semantically obsolete knowledge with audit history.
 - Agent tools for `store`, `recall`, `retire`, `update`, and `trace` through the OpenClaw plugin.
 - Native OpenClaw memory plugin that replaces OpenClaw's built-in memory slot.
 - Local-first storage with SQLite/libSQL. Memory stays on your machine; only model and embedding calls leave it.
@@ -46,7 +48,7 @@ It walks through:
 - model selection filtered by the auth method you chose
 - OpenAI embedding key setup
 - OpenClaw detection and optional plugin installation
-- session scanning and optional bulk ingestion of existing transcripts
+- session scanning and optional bulk ingestion of existing transcripts into durable entries and episodic summaries
 
 Run `agenr init` again any time you want to re-run onboarding, reinstall the plugin, or ingest another batch of existing sessions.
 
@@ -109,10 +111,10 @@ Key config fields:
 
 | Field                            | What it does                                                                                                                                                                      |
 | -------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `auth`                           | Authentication method: `openai-api-key`, `openai-subscription`, `anthropic-api-key`, `anthropic-oauth`, or `anthropic-token`.                                                     |
+| `auth`                           | Authentication method: `openai-api-key`, `openai-subscription`, `anthropic-api-key`, `anthropic-oauth`, or `anthropic-token`.                                                 |
 | `provider` / `model`             | Default LLM provider and model used for extraction tasks unless overridden.                                                                                                       |
 | `credentials`                    | Stored manual credentials. Today that can include `openaiApiKey`, `anthropicApiKey`, and `anthropicOauthToken`. The config file is written with locked-down permissions (`0600`). |
-| `credentials.openaiApiKey`       | OpenAI key used for embeddings, and also for extraction when `auth` is `openai-api-key`. Older configs may still rely on legacy `embeddingApiKey` or `apiKey` fallback fields.    |
+| `credentials.openaiApiKey`       | OpenAI key used for embeddings, and also for extraction when `auth` is `openai-api-key`. Older configs may still rely on legacy `embeddingApiKey` or `apiKey` fallback fields. |
 | `embeddingModel`                 | Embedding model. Defaults to `text-embedding-3-small`.                                                                                                                            |
 | `extractionModel` / `dedupModel` | Optional per-pipeline overrides so extraction and dedup can use different provider/model pairs.                                                                                   |
 | `extractionContext`              | Optional user context injected into extraction prompts to help the model decide what is worth remembering.                                                                        |
@@ -128,15 +130,21 @@ Important: when agenr is running as an OpenClaw plugin, session summaries use Op
 
 ## CLI Commands
 
-The current CLI surface is intentionally small. Today the `db` group only exposes `reset`.
-
-| Command                | What it does                                                                                                             |
-| ---------------------- | ------------------------------------------------------------------------------------------------------------------------ |
-| `agenr init`           | Interactive first-run wizard: auth, model selection, OpenClaw detection, plugin install, and optional initial ingestion. |
-| `agenr setup`          | Configure auth, model defaults, embeddings, and the agenr database path.                                                 |
-| `agenr recall <query>` | Run the hybrid recall pipeline with optional temporal and type/tag filters.                                              |
-| `agenr ingest <path>`  | Bulk-ingest one file or directory of OpenClaw transcript files.                                                          |
-| `agenr db reset`       | Delete and recreate the knowledge database.                                                                              |
+The CLI surface is still intentionally compact, but it now covers setup, recall, ingest, and corpus maintenance.
+
+| Command                        | What it does                                                                                                             |
+| ------------------------------ | ------------------------------------------------------------------------------------------------------------------------ |
+| `agenr init`                   | Interactive first-run wizard: auth, model selection, OpenClaw detection, plugin install, and optional initial ingestion. |
+| `agenr setup`                  | Configure auth, model defaults, embeddings, and the agenr database path.                                                 |
+| `agenr recall <query>`         | Run the hybrid recall pipeline with optional temporal and type/tag filters.                                              |
+| `agenr ingest <path>`          | Default durable-entry ingest shorthand. Equivalent to `agenr ingest entries <path>`.                                     |
+| `agenr ingest entries <path>`  | Bulk-ingest one file or directory of OpenClaw transcript files into durable knowledge entries.                           |
+| `agenr ingest episodes <path>` | Backfill episodic summaries from OpenClaw session transcripts, including rotated `.reset.*` and `.deleted.*` files.      |
+| `agenr surgeon run`            | Execute the surgeon retirement pass. Dry-run by default; add `--apply` to mutate the corpus.                            |
+| `agenr surgeon status`         | Show corpus health plus the latest surgeon run summary.                                                                  |
+| `agenr surgeon history`        | Show recent surgeon runs.                                                                                                |
+| `agenr surgeon actions <run>`  | Show the audit trail for one surgeon run.                                                                                |
+| `agenr db reset`               | Delete and recreate the knowledge database.                                                                              |
 
 The OpenClaw plugin also gives the agent five tools directly inside the runtime: `agenr_store`, `agenr_recall`, `agenr_retire`, `agenr_update`, and `agenr_trace`.
 
@@ -146,9 +154,15 @@ Examples:
 # Recall knowledge
 agenr recall "what decisions did we make about the API?"
 
-# Ingest transcripts
+# Ingest transcripts into durable entries
 agenr ingest ~/.openclaw/agents/main/sessions/
 
+# Backfill episodic summaries
+agenr ingest episodes ~/.openclaw/agents/main/sessions/ --recent 30d
+
+# Run the surgeon retirement pass (dry-run by default)
+agenr surgeon run --budget 2.00
+
 # Reset the database
 agenr db reset
 ```
@@ -167,7 +181,20 @@ Recall is a hybrid pipeline. Agenr embeds the query, retrieves candidates throug
 
 ## How Ingestion Works
 
-Ingestion is transcript-to-memory: parse OpenClaw JSONL transcripts, normalize them, choose whole-file or chunked extraction, run LLM extraction, validate the structured entries, deduplicate them within the ingest run, generate embeddings, and then persist surviving entries through the store pipeline. The current CLI ingest path is OpenClaw-transcript-specific. Details: [docs/INGEST.md](./docs/INGEST.md) and [docs/STORE.md](./docs/STORE.md).
+Agenr has two ingest pipelines over the same transcript corpus:
+
+- `agenr ingest entries <path>` extracts durable typed knowledge such as facts, decisions, preferences, lessons, milestones, and relationships.
+- `agenr ingest episodes <path>` generates one narrative summary per session so the brain can answer temporal questions like "what happened last week?"
+
+Both paths parse OpenClaw transcripts first, but they optimize for different outputs: entry ingest distills durable knowledge and runs semantic dedup across the whole ingest batch, while episode ingest does a session-by-session preflight pass, uses `sessions.json` metadata when available, reconstructs missing surface metadata for rotated files, and writes episodic summaries. Details: [docs/INGEST.md](./docs/INGEST.md) and [docs/STORE.md](./docs/STORE.md).
+
+## How Episodes Work
+
+Episodes are session-level memory artifacts stored separately from durable entries. They preserve temporal narrative: what happened in a session, when it happened, which agent/session it belonged to, and optionally an embedding for semantic episode search. Recall can route narrative or time-bounded questions toward episodes automatically, or combine episode and entry results in mixed mode. For implementation details and the episode recall model, see [docs/EPISODES.md](./docs/EPISODES.md).
+
+## How the Surgeon Works
+
+The surgeon is a maintenance pass for the durable-memory corpus. It evaluates retirement candidates, inspects related context, can simulate recall impact before mutation, and records runs plus per-action audit history in the database. `agenr surgeon run` is safe by default because it starts in dry-run mode; `--apply` is the explicit mutation switch. For runtime details, governance, and audit behavior, see [docs/SURGEON.md](./docs/SURGEON.md).
 
 ## Development
 
@@ -183,10 +210,10 @@ pnpm check        # format + lint + typecheck + test
 | Problem                                                           | What to check                                                                                                                                                                                                                        |
 | ----------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
 | `agenr init` cannot complete setup                                | Re-run `agenr setup` and verify the selected auth method is actually available. Subscription flows depend on the relevant external login being present, and non-OpenAI extraction setups still need a separate OpenAI embedding key. |
-| `openclaw plugins install @agenr/openclaw-plugin` fails           | Make sure the `openclaw` CLI is installed and on `PATH`. For local development, run `pnpm build` and use `plugins.load.paths` instead.                                                                                               |
-| `agenr recall` or `agenr_recall` fails with embedding/auth errors | Embeddings always use OpenAI. Confirm `credentials.openaiApiKey` is configured, or re-run `agenr setup` to set the embedding key explicitly.                                                                                         |
-| SQLite says the database is locked                                | Avoid running multiple writers against the same DB at once. Stop overlapping ingest/reset runs, restart the OpenClaw gateway if needed, then retry.                                                                                  |
-| OpenClaw does not pick up the plugin                              | Restart the gateway, confirm `plugins.slots.memory` is `agenr`, confirm `plugins.allow` contains `agenr`, and for dev installs confirm `plugins.load.paths` points at the built `packages/openclaw-plugin` directory.                |
+| `openclaw plugins install @agenr/openclaw-plugin` fails           | Make sure the `openclaw` CLI is installed and on `PATH`. For local development, run `pnpm build` and use `plugins.load.paths` instead.                                                                                            |
+| `agenr recall` or `agenr_recall` fails with embedding/auth errors | Embeddings always use OpenAI. Confirm `credentials.openaiApiKey` is configured, or re-run `agenr setup` to set the embedding key explicitly.                                                                                        |
+| SQLite says the database is locked                                | Avoid running multiple writers against the same DB at once. Stop overlapping ingest/reset runs, restart the OpenClaw gateway if needed, then retry.                                                                                 |
+| OpenClaw does not pick up the plugin                              | Restart the gateway, confirm `plugins.slots.memory` is `agenr`, confirm `plugins.allow` contains `agenr`, and for dev installs confirm `plugins.load.paths` points at the built `packages/openclaw-plugin` directory.               |
 
 ## License