@ppagent/memory 0.4.4 → 0.4.6

package/llms.txt

@@ -1,6 +1,6 @@
 # @ppagent/memory
 
-> `@ppagent/memory` is an independent TypeScript/Node.js long-term memory package for AI agents. It stores complete raw conversation messages in a pluggable vector store (LanceDB or SQLite/sqlite-vec, auto-detected per platform), builds importance-aware Topic summaries, returns dynamically budgeted Topics plus recent uncompressed messages as one context window, extracts and searches entity relationships with Grafeo, supports cached user/chat facts, and ingests Markdown documents as a searchable knowledge base.
+> `@ppagent/memory` is an independent TypeScript/Node.js long-term memory package for AI agents. It stores semantic raw conversation history in a pluggable vector store (LanceDB or SQLite/sqlite-vec, auto-detected per platform), strips historical image payloads, builds importance-aware Topic summaries, returns dynamically budgeted Topics plus recent uncompressed messages as one context window, extracts and searches entity relationships with Grafeo, supports cached user/chat facts, and ingests Markdown documents as a searchable knowledge base.
 
 This file is written for AI coding agents and assistants. Use it as the primary package guide when generating code that depends on `@ppagent/memory`.
 
@@ -154,8 +154,10 @@ const memory = new MemoryManager({
   topicCompactionSyncRatio: 1.25,
   contextUsageRatio: 0.75,
   precompressionRatio: 0.75,
-  compressionBatchRatio: 0.5,
+  compressionBatchRatio: 0.7,
   compressionBatchTokenLimit: 0,
+  compressionMaxRetries: 2,
+  compressionRetryBaseDelayMs: 500,
   topicSummaryMaxTokens: 2048,
   defaultModelContextTokens: 256 * 1024,
   maxHistoryAgeMs: 0,
@@ -327,7 +329,7 @@ Embedding fallback behavior:
 HTTP and retry fields:
 
 - `httpTimeoutMs?: number` - timeout per LLM or embedding HTTP attempt. Default: `60000`.
-- `httpMaxRetries?: number` - retry count after the first attempt for network errors, timeout/abort, HTTP `429`, and HTTP `5xx`. Default: `2`.
+- `httpMaxRetries?: number` - retry count after the first attempt for network errors, timeout/abort, HTTP `429`, HTTP `5xx`, and invalid JSON in a successful HTTP response. Default: `2`.
 - `embeddingBatchSize?: number` - max texts per embedding request. Default: `20`.
 - `embeddingConcurrency?: number` - concurrent embedding batches. Default: `2`.
 
@@ -337,14 +339,22 @@ Conversation memory fields:
 - `compressedContextRatio?: number` - maximum fraction of the usable model-history window assigned to Topics. Default: `0.10`. The effective Topic budget is the smaller of this ratio and `compressedContextTokenLimit`.
 - `topicCompactionSyncRatio?: number` - a cold-start Topic overage blocks `getHistoryWindow` only after it exceeds this multiple of the effective Topic budget, unless Topic plus raw history already exceeds the usable window. Default: `1.25`.
 - `contextUsageRatio?: number` - fraction of the current model context available to conversation history. Default: `0.75`.
-- `precompressionRatio?: number` - start background raw-message compression when raw usage reaches this fraction of its dynamic budget. Default: `0.75`.
-- `compressionBatchRatio?: number` - target fraction of the raw-message budget compressed in one batch. Default: `0.5`.
+- `precompressionRatio?: number` - start background raw-message compression when raw usage reaches this fraction of its dynamic budget. Default: `0.60`, leaving enough headroom for normal runs while compression finishes.
+- `compressionBatchRatio?: number` - target fraction of the current uncompressed raw-message tokens compressed from the oldest edge in one batch. Default: `0.7`, leaving the newest approximately `0.3` verbatim; complete conversation-group boundaries are atomic.
 - `compressionBatchTokenLimit?: number` - optional hard limit for one compression batch. Default: `0` (ratio only).
+- `compressionMaxRetries?: number` - result-level retry count after the first compression-model attempt when the returned content is malformed, missing a summary, or empty. Default: `2`. Transport failures do not multiply this count because the HTTP layer already owns bounded retries.
+- `compressionRetryBaseDelayMs?: number` - exponential-backoff base for result-level compression retries. Default: `500`; set `0` to retry immediately.
+- `backgroundCompressionMaxRetries?: number` - whole-task retries after a background compression still fails (including exhausted HTTP attempts). Default: `2`, so at most three task waves run.
+- `backgroundCompressionRetryBaseDelayMs?: number` - exponential-backoff base for whole-task background retries. Default: `1000`; set `0` to retry immediately.
+- `onCompressionEvent?: (event) => void | Promise<void>` - receives contained raw/Topic compression `start`, `retry`, `success`, and `failure` lifecycle events with mode, attempts, token counts, latency, and error text. Callback failures never fail memory work.
 - `topicSummaryMaxTokens?: number` - hard maximum for one Topic summary. The LLM uses less for low-value chat and more for facts, experience, decisions, preferences, constraints, and reusable knowledge. Default: `2048`.
 - `defaultModelContextTokens?: number` - used when `getHistoryWindow` receives no model size; a warning is logged. Default: `262144`.
 - `maxHistoryAgeMs?: number` - maximum Topic and raw-message age restored during lazy cold start. Default: `0` (all history).
 - `sessionIdleTtlMs?: number` - idle time before a session cache is released. Persistent data is not deleted. Default: `1800000`.
 - `sessionSweepIntervalMs?: number` - idle cache sweep interval. Default: `60000`.
+- `autoMigrateLegacyDataOnInit?: boolean` - run the idempotent 0.4 storage migration in the background once per data store, including full historical-image stripping. Default: `true`; a sidecar marker prevents a full scan on every restart.
+- `autoResumeCompressionOnInit?: boolean` - scan persisted sessions in the background and resume raw windows that were eligible but not compressed before shutdown/failure. Default: `true`.
+- `restoreConcurrency?: number` - bounded startup migration/recovery scan concurrency. Default: `8`.
 - `maxConcurrentCompressions?: number` - global semaphore limit for concurrent compression and knowledge ingestion tasks. Default: `3`.
 - `entitySimilarityThreshold?: number` - graph entity similarity threshold. Default: `0.92`.
 - `defaultSearchLimit?: number` - default `search` result limit. Default: `10`.
@@ -374,12 +384,15 @@ The package intentionally separates durable base writes from expensive graph con
 `updateChat(messages, opts)`:
 
 - Registers its write synchronously, so a caller may intentionally fire-and-forget it and a following `getHistoryWindow` will still wait for that write.
-- Calculates tokens over `content`, `parts`, `payload`, `contextPayload`, and `metadata`, embeds only the searchable text, and upserts raw messages by stable `messageId` before resolving.
-- `contextPayload` is a replay-only host payload: it is stored and returned unchanged and counts toward the raw budget, but is excluded from embedding, FTS, Topic-summary input, and conversation graph extraction.
-- Raw-message token usage always counts the complete stored `metadata`. When a raw batch is summarized, common tool-call/tool-result fields are removed only from the temporary LLM compression input; persisted metadata and uncompressed `recentMessages` remain unchanged.
+- Recursively strips known historical image blocks (`image*` content parts and `file` parts with `image/*` media types) from `parts`, `payload`, `contextPayload`, and `metadata`, while preserving adjacent text, tool protocol, and non-image files. Legacy image rows are lazily repaired and written back when a session is first hydrated; the manual migration command performs the same repair.
+- Persists optional `compressionGroupId` / `compressionRole` hints in a reserved internal metadata namespace and removes that namespace again on read. Host metadata remains unchanged.
+- Calculates tokens over the sanitized `content`, `parts`, `payload`, `contextPayload`, and `metadata`, embeds only the searchable text, and upserts raw messages by stable `messageId` before resolving. If image removal changes a host payload, any host-supplied usage is recalculated against the stored form.
+- `contextPayload` is a replay-only host payload: its non-image structure is stored and returned and counts toward the raw budget, but it is excluded from embedding, FTS, Topic-summary input, and conversation graph extraction.
+- Raw-message token usage counts the complete sanitized stored `metadata`. When a raw batch is summarized, common tool-call/tool-result fields are removed only from the temporary LLM compression input; persisted non-image metadata and uncompressed `recentMessages` remain unchanged.
 - Creates or updates the session record.
 - Updates the in-memory session cache.
-- Once `getHistoryWindow` has supplied the current model size, reaching the precompression threshold starts background compression.
+- Reaching the dynamic precompression threshold starts background compression. A failed task is retried with finite exponential backoff, and an unfinished eligible tail is discovered again during the next startup scan.
+- Compression selects an oldest prefix of complete conversation groups. Explicit group ids are authoritative; older rows fall back to `user`/`assistant`/`tool`/`system` role inference. The newest complete group is kept verbatim whenever another group exists.
 - Topic storage and cache replacement finish before conversation graph extraction; graph work remains asynchronous.
 
 `flushChat(sessionId?, opts?)`:
@@ -394,7 +407,7 @@ The package intentionally separates durable base writes from expensive graph con
 
 - Lazily hydrates the session from persistent Topics plus raw messages after the newest Topic boundary.
 - Waits for registered message writes. If the hard raw budget is exceeded, it also waits for or starts compression until the returned context fits.
-- Returns `{ compressionRevision, compressedContext, recentMessages, usage }`. `recentMessages` preserves the `RawMessage` input shape, including `messageId`, `parts`, `payload`, `contextPayload`, `metadata`, and `createdAt`. `compressionRevision` changes when the returned Topic/raw compression boundary changes and is intended for host observability rather than optimistic locking.
+- Returns `{ compressionRevision, compressedContext, recentMessages, usage }`. `recentMessages` preserves the sanitized `RawMessage` input shape, including `messageId`, non-image `parts`, `payload`, `contextPayload`, `metadata`, and `createdAt`. `compressionRevision` changes when the returned Topic/raw compression boundary changes and is intended for host observability rather than optimistic locking.
 - The effective Topic budget is `min(compressedContextTokenLimit, usableContextTokens * compressedContextRatio)`; the remaining usable history budget is reserved for raw messages.
 - Cold hydration always restores all persisted Topics and never silently truncates them. Each rollup summarizes the oldest approximately half of the current Topic tokens; a single oversized Topic is re-summarized by itself.
 - A small Topic-only overage returns immediately and schedules a transient background rollup for the next read. It blocks only when the overage exceeds `topicCompactionSyncRatio` or Topic plus raw history cannot fit the usable window.
@@ -424,9 +437,11 @@ Use `wait: true` when the next line of code must immediately call `searchKnowled
 - `sessionId?: string` - session id.
 - `type?: "text" | "image" | "file"` - default: `"text"`.
 - `content: string` - required plain text used for embedding and retrieval.
-- `parts?: ContentPart[]` - optional multimodal content parts; stored serialized.
-- `payload?: unknown` - host-framework message payload stored and returned without interpretation.
-- `contextPayload?: unknown` - replay-only host context stored and returned without interpretation. It counts toward the raw history budget but is excluded from retrieval, compression summaries, and graph extraction.
+- `parts?: ContentPart[]` - optional multimodal content parts; non-image parts are stored serialized.
+- `payload?: unknown` - host-framework message payload stored and returned after recursive historical-image stripping.
+- `contextPayload?: unknown` - replay-only host context stored and returned after recursive historical-image stripping. Its remaining content counts toward the raw history budget but is excluded from retrieval, compression summaries, and graph extraction.
+- `compressionGroupId?: string` - stable id shared by a user message and all assistant/tool messages in the same complete turn. Compression never splits this group.
+- `compressionRole?: "user" | "assistant" | "tool" | "system"` - explicit role used for compatible grouping when no group id is supplied.
 - `usage?: number` - token count; estimated with `tiktoken` if omitted.
 - `metadata?: Record<string, unknown>` - custom metadata stored as JSON.
 - `createdAt?: number` - Unix milliseconds; default is current time.
@@ -523,7 +538,7 @@ Creates the manager and resolves defaults. It does not connect to storage until
 
 ### `init(): Promise<void>`
 
-Initializes tiktoken, the vector store (backend auto-detection happens here), Grafeo, and the facts cache. Conversation sessions are hydrated lazily on first access and evicted after the configured idle timeout. Always call before using read/write APIs.
+Initializes tiktoken, the vector store (backend auto-detection happens here), Grafeo, and the facts cache. It then starts non-blocking startup maintenance: a marker-guarded full legacy/image migration and a bounded scan that resumes eligible unfinished compression. Conversation sessions remain available immediately and are evicted after the configured idle timeout. Always call before using read/write APIs.
 
 ### `updateChat(messages, opts?): Promise<void>`
 
@@ -622,7 +637,7 @@ Returns the complete model-history window:
 
 - `compressionRevision?: string` - lightweight identifier for the current Topic/raw compression boundary. Background or blocking compression changes it; ordinary raw appends are not guaranteed to do so.
 - `compressedContext: string` - chronological Topic summaries for the current model-size budget. A soft cold-start overage may be returned once while its transient rollup runs in the background.
-- `recentMessages: MemoryRawMessage[]` - all currently uncompressed raw messages in chronological order, preserving the host payload, replay-only context payload, and metadata.
+- `recentMessages: MemoryRawMessage[]` - all currently uncompressed raw messages in chronological order, preserving sanitized non-image host payload, replay-only context payload, and metadata.
 - `usage` - model size, usable history size, compressed usage, and dynamic raw-message budget/usage.
 
 Pass the active model's context size on every read. Omitting it uses `defaultModelContextTokens` (256K by default) and logs a warning.
@@ -732,7 +747,7 @@ Lists documents, optionally filtered by:
 
 ### `destroy(): Promise<void>`
 
-Waits for registered writes, compression, graph work, and storage maintenance, then closes Grafeo and the vector store. It does not delete data.
+Waits for registered writes, startup recovery, finite background compression retries, graph work, and storage maintenance, then closes Grafeo and the vector store. It does not delete data.
 
 ## Common Recipes
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ppagent/memory",
-  "version": "0.4.4",
+  "version": "0.4.6",
   "description": "独立记忆系统模块，向量存储支持 LanceDB / SQLite(sqlite-vec) 双后端自动切换 + Grafeo 知识图谱",
   "type": "module",
   "main": "dist/index.js",