lemura 1.6.0 → 1.8.0

package/CHANGELOG.md

@@ -5,6 +5,34 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.8.0] - 2026-06-16
+
+### Added
+
+- **`LemurAgent` — preferred alias for `SessionManager`**: New code should construct `LemurAgent`; `SessionManager` remains exported unchanged for backward compatibility. Both are the exact same class (so `instanceof SessionManager` still holds for `new LemurAgent(...)`), exported in both value and type position from the root entry point.
+- **Long-Term Memory subsystem** (`lemura/memory`, `SessionConfig.memory`): A persistent, cross-session, ranked, decaying, **token-budget-aware** memory — the capability missing between STM (large blobs), Scratchpad (per-session notes), and RAG (passive document store). Opt-in: when `config.memory` is absent, behavior is identical to 1.7.0. **Embedding-free by default** — the only place embeddings can enter is a pluggable scorer.
+- **`MemoryManager`** (`src/memory/MemoryManager.ts`): Orchestrates `remember` (de-dup + reinforce), `recall` (composite-score ranking with reinforcement), `reflect` (autonomous fact extraction), `consolidate` (merge near-duplicates), and `forget`. Composite retrieval score `wR·relevance + wT·recency + wI·importance + wF·frequency` with exponential recency decay (generative-agents formula). Only the `relevance` term is pluggable; the rest is pure arithmetic.
+- **`MemoryInjectionStrategy`** (`IContextStrategy`): Budget-aware recall. Runs inside `ContextManager.prepare()` exactly like `SummaryInjectionStrategy`, so the recalled `<lemura:memory>` block is counted against `maxTokens` — memory can never overflow the window. Greedily fills `recallTokenBudget`; injected as a single synthetic system turn (`turnIndex = -2`) updated in place across iterations (idempotent).
+- **Pluggable scorers** (`IMemoryScorer`): `LexicalScorer` (default, embedding-free BM25-ish, offline), `EmbeddingScorer` (opt-in, requires the new optional `adapter.embed?()`; falls back to lexical otherwise), `LLMReRankScorer` (opt-in, paraphrase-robust via one cheap `complete()` call, no vector store).
+- **`StorageMemoryStore`** (`IMemoryStore`): Default store over any `IStorageAdapter` ("Any storage" — in-memory, IndexedDB, SQLite, Redis, file store), with a lightweight index record for filtered `list()` without full scans.
+- **Built-in `remember`/`recall`/`forget` tools** (`src/tools/builtin/memory.ts`): Auto-registered when `config.memory` is present, categorized `memory` for the router. `remember`/`recall` are auto-trusted by the firewall (local-only side effects); `forget` (a delete) stays firewall-gated.
+- **`IProviderAdapter.embed?()`** (optional, additive): Returns an embedding vector; enables `EmbeddingScorer`. Existing adapters compile and run unchanged.
+- **`ToolContext.memory?`** (additive): The `MemoryManager` is exposed to tool `execute()`.
+- **`SessionConfig.memory.autoReflect`**: When true, durable facts are extracted from the conversation once at the end of each `run()` (one cheap LLM call; off by default, best-effort).
+- **`TraceEvent.type: 'memory'`**: New trace type — `memory_write`, `memory_recall`, `memory_reflect`, `memory_consolidate`, `memory_forget`.
+- **Tests**: `tests/unit/memory/Memory.test.ts` (store, scorer, manager ranking/dedup/reflect/forget, injection budget+idempotency) and `tests/unit/memory/SessionMemory.test.ts` (tool registration, cross-session recall, autoReflect, no-overflow).
+
+## [1.7.0] - 2026-06-13
+
+### Added
+
+- **Progressive skills — model-driven skill selection** (`strategy: 'progressive'`): A third skill strategy alongside `fixed` and `dynamic`. The model sees a lightweight catalog (`name: description`) of progressive skills in its system prompt and decides which to pull in by calling a built-in `load_skill` tool; only the chosen skill's full content is then injected. This is the "progressive disclosure" pattern — the host wires nothing, the agent selects its own skills. Fully backward-compatible (default strategy remains `fixed`).
+- **Built-in `load_skill` tool** (`src/tools/builtin/load_skill.ts`, `createLoadSkillTool`): Auto-registered by `SessionManager` whenever a `progressive` skill is present, and auto-trusted by the tool firewall (it only toggles skill injection, no external side effects). The `name` parameter is constrained to the available progressive skills; unknown names and `maxConcurrent` violations return soft errors that keep the ReAct loop going.
+- **`SkillSelectionConfig`** (`SessionConfig.skillSelection`): Tunes progressive selection — `persistence` (`'per_turn'` default, or `'session'`), `maxConcurrent` (cap on simultaneously loaded skills), and `catalogHeader` (override the catalog preamble). Ignored when no progressive skills are present.
+- **`SkillInjector.buildCatalog(header?)`**: Builds the `name: description` catalog block for progressive skills. Plus `resetProgressiveSkills()`, `getProgressiveSkills()`, and `countEnabledProgressive()` helpers. `enableSkill`/`disableSkill`/`enableByTags`/`disableByTags` now operate on both `dynamic` and `progressive` skills.
+- **Full skill-lifecycle tracing**: `skill_load` now fires for **every registered skill** (not just active ones) and carries an `enabled` flag, so inactive `dynamic`/`progressive` skills are visible from session init. New `skill_enable` trace (`{ name, source: 'load_skill' }`) records the model's selection decision when it loads a progressive skill, and new `skill_reset` trace (`{ skills, reason: 'per_turn' }`) records per-turn clearing. `session_init` reports a `progressive` skill count; `skill_inject` fires for both dynamic and progressive injections.
+- **Tests**: `tests/unit/skills/ProgressiveSkills.test.ts` (catalog, activation, `load_skill` tool, `maxConcurrent`) and new `SessionManager` cases (auto-registration, per-turn vs session persistence).
+
 ## [1.6.0] - 2026-06-05
 
 ### Added

package/README.md

@@ -25,6 +25,7 @@
 ### ✨ Key Features
 
 - **🧠 Dynamic Skill Market**: Switch skills on/off at runtime via tags, names, or tool dependencies.
+- **📖 Progressive Skills**: Let the agent pick its own skills — it reads a catalog and loads only what's relevant via the built-in `load_skill` tool (`strategy: 'progressive'`).
 - **🔌 Native MCP Support**: Connect to any Model Context Protocol server with custom header support (Auth).
 - **🛡️ Tool Firewall**: Fully integrated ask/accept/deny policy layer for secure tool execution — fail-safe by design.
 - **🎯 Goal Maintenance & Verification**: LLM-powered sub-goal decomposition, progress reconciliation, and post-run goal verification that re-enters the loop with full tool access to finish incomplete answers.

package/dist/adapters/index.d.mts

@@ -1,4 +1,4 @@
-import { a as IProviderAdapter, l as CompletionRequest, m as CompletionResponse, k as CompletionChunk, M as ModelInfo, d as TranscriptionRequest, e as TranscriptionResponse, f as SynthesisRequest, A as AudioChunk, V as VisionRequest, g as VisionResponse, h as ImageGenRequest, i as ImageGenResponse } from '../adapters-BN6und82.mjs';
+import { a as IProviderAdapter, m as CompletionRequest, n as CompletionResponse, l as CompletionChunk, j as ModelInfo, d as TranscriptionRequest, e as TranscriptionResponse, f as SynthesisRequest, A as AudioChunk, V as VisionRequest, g as VisionResponse, h as ImageGenRequest, i as ImageGenResponse } from '../adapters-BQA6PxCL.mjs';
 import '../rag-La_Bo-J8.mjs';
 import '../logger-DxvKliuk.mjs';
 

package/dist/adapters/index.d.ts

@@ -1,4 +1,4 @@
-import { a as IProviderAdapter, l as CompletionRequest, m as CompletionResponse, k as CompletionChunk, M as ModelInfo, d as TranscriptionRequest, e as TranscriptionResponse, f as SynthesisRequest, A as AudioChunk, V as VisionRequest, g as VisionResponse, h as ImageGenRequest, i as ImageGenResponse } from '../adapters-CnsgefYR.js';
+import { a as IProviderAdapter, m as CompletionRequest, n as CompletionResponse, l as CompletionChunk, j as ModelInfo, d as TranscriptionRequest, e as TranscriptionResponse, f as SynthesisRequest, A as AudioChunk, V as VisionRequest, g as VisionResponse, h as ImageGenRequest, i as ImageGenResponse } from '../adapters-nzsMKcf4.js';
 import '../rag-La_Bo-J8.js';
 import '../logger-DxvKliuk.js';
 

package/dist/{adapters-BN6und82.d.mts → adapters-BQA6PxCL.d.mts}

@@ -135,32 +135,146 @@ declare class ShortTermMemoryRegistry {
     delete(id: string): Promise<void>;
 }
 
-interface ToolContext {
-    sessionId: string;
-    turnIndex: number;
-    logger: ILogger;
-    adapter?: IProviderAdapter;
-    ragAdapter?: IRAGAdapter;
-    stmRegistry?: ShortTermMemoryRegistry;
-    scratchpad?: string;
-    scratchpadAdapter?: IScratchpadAdapter;
+/**
+ * Long-Term Memory types — persistent, cross-session, ranked, decaying memory.
+ *
+ * Distinct from STM (large blobs), Scratchpad (per-session notes), and RAG (passive
+ * document store): long-term memory is written autonomously, ranked by a composite
+ * score (relevance + recency + importance + frequency), decays over time, and is
+ * recalled into the context window under a token budget.
+ *
+ * Embedding-free by default: the only place embeddings can enter is the pluggable
+ * {@link IMemoryScorer}. The default `LexicalScorer` needs no embeddings.
+ *
+ * @since 1.8.0
+ */
+type MemoryKind = 'fact' | 'preference' | 'episode' | 'entity' | 'summary';
+/** A single durable memory record. */
+interface MemoryRecord {
+    /** Stable unique id. */
+    id: string;
+    /** The fact / observation text. */
+    content: string;
+    /** Category of memory. */
+    kind: MemoryKind;
+    /** Salience 1–10, rated by the LLM at write time (default 5). Higher = recalled more readily. */
+    importance: number;
+    /** Epoch ms of creation. */
+    createdAt: number;
+    /** Epoch ms of last recall — drives the recency decay term. */
+    lastAccessedAt: number;
+    /** Number of times this record has been recalled — drives the frequency term. */
+    accessCount: number;
+    /** Free-form tags for filtering. */
+    tags?: string[];
+    /** People/places/things mentioned — keys for graph recall and pre-filtering. */
+    entities?: string[];
+    /** Ids of related memories (graph edges). */
+    links?: string[];
+    /** Optional embedding vector — present only when an embedding scorer is used. */
+    embedding?: number[];
+    /** Provenance of the write. */
+    source?: 'auto' | 'user' | 'tool';
+    /** Partition key (e.g. userId) for multi-tenant stores. */
+    scope?: string;
+    /** Arbitrary extra metadata. */
+    metadata?: Record<string, unknown>;
 }
-interface IToolDefinition {
-    name: string;
-    description: string;
-    parameters: Record<string, unknown>;
-    /** Per-call timeout in milliseconds. Falls back to ToolRegistry.defaultTimeoutMs when omitted. */
-    timeoutMs?: number;
+/**
+ * Pluggable relevance term. The ONLY place embeddings could enter the subsystem.
+ * Embedding-free implementations (the default) ignore `record.embedding`.
+ */
+interface IMemoryScorer {
+    readonly name: string;
     /**
-     * Optional category used by the router (see `SessionConfig.enableRouting`) to
-     * narrow which tools are exposed to the model on a given turn. Open string —
-     * lemura does not own a fixed catalog. Tools left uncategorized are never
-     * filtered out (treated as always available).
+     * Relevance of a record to the query, normalized 0..1.
      *
-     * @since 1.6.0
+     * @param query - The current user message (or an explicit recall query).
+     * @param record - The candidate memory record.
+     * @param corpus - All candidate records, supplied so lexical scorers can compute
+     *   inverse-document-frequency weighting. May be ignored.
      */
-    category?: string;
-    execute(params: unknown, context: ToolContext): Promise<unknown>;
+    relevance(query: string, record: MemoryRecord, corpus?: MemoryRecord[]): number | Promise<number>;
+    /**
+     * Optional hook to enrich a record at write time (e.g. attach an embedding).
+     * No-op for lexical scorers. Returns the (possibly mutated) record.
+     */
+    prepare?(record: MemoryRecord): Promise<MemoryRecord>;
+}
+/** Filter passed to {@link IMemoryStore.list} to narrow the candidate set. */
+interface MemoryFilter {
+    scope?: string;
+    kinds?: MemoryKind[];
+    tags?: string[];
+    entities?: string[];
+    /** Hard cap on candidates returned to the scorer. Default 500. */
+    limit?: number;
+}
+/**
+ * Persistence contract for memory records. The default implementation
+ * ({@link StorageMemoryStore}) rides on any {@link IStorageAdapter}, so the same
+ * memory works against in-memory, IndexedDB, SQLite, Redis, or a file store.
+ */
+interface IMemoryStore {
+    put(record: MemoryRecord): Promise<void>;
+    get(id: string): Promise<MemoryRecord | undefined>;
+    /** Returns candidate records to score. May pre-filter by scope/tags/entities. */
+    list(filter?: MemoryFilter): Promise<MemoryRecord[]>;
+    delete(id: string): Promise<void>;
+    healthCheck?(): Promise<boolean>;
+}
+/** Relative weights for the composite retrieval score. */
+interface MemoryScoreWeights {
+    /** Weight of the (pluggable) relevance term. Default 1.0. */
+    relevance?: number;
+    /** Weight of the recency decay term. Default 0.5. */
+    recency?: number;
+    /** Weight of the importance term. Default 0.3. */
+    importance?: number;
+    /** Weight of the access-frequency term. Default 0.1. */
+    frequency?: number;
+}
+/** A scored memory record returned by recall. */
+interface RankedMemory {
+    record: MemoryRecord;
+    score: number;
+    /** Per-term breakdown, for tracing/debugging. */
+    terms: {
+        relevance: number;
+        recency: number;
+        importance: number;
+        frequency: number;
+    };
+}
+/**
+ * Configuration for the long-term memory subsystem. Attach to
+ * `SessionConfig.memory`. Absent ⇒ no memory wiring (identical to 1.7.0).
+ */
+interface MemoryConfig {
+    /** Persistence backend. Required. */
+    store: IMemoryStore;
+    /** Relevance scorer. Default: `LexicalScorer` (embedding-free). */
+    scorer?: IMemoryScorer;
+    /** Partition key applied to all writes/reads for this session. */
+    scope?: string;
+    /** Composite-score weights. */
+    weights?: MemoryScoreWeights;
+    /** Recency half-life in ms. Default 7 days. */
+    recencyHalfLifeMs?: number;
+    /** Token budget for the recalled-memory block injected each turn. Default 800. */
+    recallTokenBudget?: number;
+    /** Max records considered for injection regardless of budget. Default 8. */
+    recallTopK?: number;
+    /** Minimum composite score to be eligible for injection/recall. Default 0.15. */
+    minScore?: number;
+    /** Run reflection (autonomous write) automatically at session end. Default false. */
+    autoReflect?: boolean;
+    /** Run consolidation every N writes (0 = never). Default 0. */
+    consolidateEvery?: number;
+    /** Priority of the MemoryInjectionStrategy. Default 2 (after summary injection at 1). */
+    injectionPriority?: number;
+    /** Header label for the injected block. Default 'Relevant memories'. */
+    injectionLabel?: string;
 }
 
 interface ContentBlock {
@@ -198,6 +312,114 @@ interface IContextStrategy {
     apply(ctx: ContextWindow): Promise<ContextWindow>;
 }
 
+interface MemoryManagerConfig {
+    store: IMemoryStore;
+    scorer?: IMemoryScorer;
+    adapter?: IProviderAdapter;
+    model?: string;
+    logger?: ILogger;
+    scope?: string;
+    weights?: MemoryScoreWeights;
+    recencyHalfLifeMs?: number;
+    recallTopK?: number;
+    minScore?: number;
+    consolidateEvery?: number;
+    /** Called for trace emission. */
+    onTrace?: (name: string, metadata: Record<string, unknown>) => void;
+}
+/**
+ * Orchestrates long-term memory: autonomous write (reflection), explicit write,
+ * ranked recall (composite score with decay), consolidation, and forgetting.
+ *
+ * Embedding-free by default — the relevance term is delegated to a pluggable
+ * {@link IMemoryScorer}; everything else (recency, importance, frequency, storage)
+ * is provider- and embedding-independent.
+ *
+ * @since 1.8.0
+ */
+declare class MemoryManager {
+    private cfg;
+    private store;
+    private scorer;
+    private weights;
+    private halfLifeMs;
+    private recallTopK;
+    private minScore;
+    private consolidateEvery;
+    private writesSinceConsolidate;
+    constructor(cfg: MemoryManagerConfig);
+    private trace;
+    /**
+     * Persist a memory. De-duplicates against existing records: if an existing
+     * record is near-identical (relevance > 0.9) it is reinforced (importance bumped,
+     * lastAccessedAt refreshed) instead of inserting a duplicate.
+     */
+    remember(input: {
+        content: string;
+        kind?: MemoryKind;
+        importance?: number;
+        tags?: string[];
+        entities?: string[];
+        source?: MemoryRecord['source'];
+    }): Promise<MemoryRecord>;
+    /**
+     * Rank stored memories against a query using the composite score
+     * `wR*relevance + wT*recency + wI*importance + wF*frequency`. Returns the top
+     * records above `minScore`. Recalled records are reinforced (lastAccessedAt /
+     * accessCount bumped) so frequently-useful memories stay salient.
+     */
+    recall(query: string, topK?: number): Promise<RankedMemory[]>;
+    /**
+     * Extract durable facts/preferences from recent turns and persist them, each
+     * rated for importance. One temperature-0 LLM call. Requires an adapter; no-op
+     * if none was supplied. This is the generative-agents "reflection" step.
+     */
+    reflect(turns: Turn[]): Promise<MemoryRecord[]>;
+    /**
+     * Merge near-duplicate records into one, keeping the store bounded. Conservative:
+     * only collapses records whose mutual relevance exceeds 0.9 and that share a kind.
+     * The survivor inherits the max importance and summed access count.
+     */
+    consolidate(): Promise<{
+        merged: number;
+    }>;
+    /** Delete a memory by id, or the single best match for a query. */
+    forget(input: {
+        id?: string;
+        query?: string;
+    }): Promise<boolean>;
+}
+
+interface ToolContext {
+    sessionId: string;
+    turnIndex: number;
+    logger: ILogger;
+    adapter?: IProviderAdapter;
+    ragAdapter?: IRAGAdapter;
+    stmRegistry?: ShortTermMemoryRegistry;
+    scratchpad?: string;
+    scratchpadAdapter?: IScratchpadAdapter;
+    /** Long-term memory orchestrator. Present when `SessionConfig.memory` is set. @since 1.8.0 */
+    memory?: MemoryManager;
+}
+interface IToolDefinition {
+    name: string;
+    description: string;
+    parameters: Record<string, unknown>;
+    /** Per-call timeout in milliseconds. Falls back to ToolRegistry.defaultTimeoutMs when omitted. */
+    timeoutMs?: number;
+    /**
+     * Optional category used by the router (see `SessionConfig.enableRouting`) to
+     * narrow which tools are exposed to the model on a given turn. Open string —
+     * lemura does not own a fixed catalog. Tools left uncategorized are never
+     * filtered out (treated as always available).
+     *
+     * @since 1.6.0
+     */
+    category?: string;
+    execute(params: unknown, context: ToolContext): Promise<unknown>;
+}
+
 /** Represents a single message in the provider format */
 interface NormalizedMessage {
     /** The role of the message sender */
@@ -353,6 +575,14 @@ interface IProviderAdapter {
     synthesize(request: SynthesisRequest): AsyncIterable<AudioChunk>;
     describeImage(request: VisionRequest): Promise<VisionResponse>;
     generateImage(request: ImageGenRequest): Promise<ImageGenResponse>;
+    /**
+     * Optional. Returns an embedding vector for the given text. Enables the
+     * long-term-memory `EmbeddingScorer`; when absent, memory falls back to the
+     * embedding-free `LexicalScorer`. Existing adapters need not implement this.
+     *
+     * @since 1.8.0
+     */
+    embed?(text: string, model?: string): Promise<number[]>;
     /**
      * Estimates token count for standard text.
      *
@@ -374,4 +604,4 @@ interface IProviderAdapter {
     healthCheck(): Promise<boolean>;
 }
 
-export { type AudioChunk as A, type ContextWindow as C, type IToolDefinition as I, type ModelInfo as M, type NormalizedMessage as N, ShortTermMemoryRegistry as S, type Turn as T, type VisionRequest as V, type IProviderAdapter as a, type IContextStrategy as b, type IScratchpadAdapter as c, type TranscriptionRequest as d, type TranscriptionResponse as e, type SynthesisRequest as f, type VisionResponse as g, type ImageGenRequest as h, type ImageGenResponse as i, type ContentBlock as j, type CompletionChunk as k, type CompletionRequest as l, type CompletionResponse as m, type IStorageAdapter as n, type STMItem as o, type STMRegistryConfig as p, type TokenUsage as q, type ToolCall as r, type ToolContext as s, type ToolResult as t };
+export { type AudioChunk as A, type ToolCall as B, type ContextWindow as C, type ToolContext as D, type ToolResult as E, type IToolDefinition as I, type MemoryConfig as M, type NormalizedMessage as N, type RankedMemory as R, ShortTermMemoryRegistry as S, type Turn as T, type VisionRequest as V, type IProviderAdapter as a, type IContextStrategy as b, type IScratchpadAdapter as c, type TranscriptionRequest as d, type TranscriptionResponse as e, type SynthesisRequest as f, type VisionResponse as g, type ImageGenRequest as h, type ImageGenResponse as i, type ModelInfo as j, type ContentBlock as k, type CompletionChunk as l, type CompletionRequest as m, type CompletionResponse as n, type IMemoryScorer as o, type IMemoryStore as p, type IStorageAdapter as q, type MemoryFilter as r, type MemoryKind as s, MemoryManager as t, type MemoryManagerConfig as u, type MemoryRecord as v, type MemoryScoreWeights as w, type STMItem as x, type STMRegistryConfig as y, type TokenUsage as z };