@pyxmate/memory 0.20.5 → 0.21.1

package/skills/pyx-memory/reference/advanced.md

@@ -1,274 +0,0 @@
-# Advanced Features
-
-## RAG Strategies
-
-| Strategy | How It Works | Requirements |
-|----------|-------------|--------------|
-| `'naive'` | Embed query → vector similarity → top-K | (default, always available) |
-| `'graph'` | Extract entities → graph traversal → context expansion | `graphStore` in MemoryOptions |
-| `'hybrid'` | BM25 + vector + graph + community summaries → RRF fusion (k=60) → reranking | `graphStore` in MemoryOptions |
-| `'agentic'` | LLM decides strategy → iterative refinement (3 rounds) | `reasoningProvider` in MemoryOptions |
-
-```typescript
-// Naive (default)
-await memory.search({ query: 'user preferences', limit: 10 });
-
-// Graph
-await memory.search({ query: 'who works at Acme', strategy: 'graph', limit: 10 });
-
-// Hybrid (recommended for best quality)
-await memory.search({ query: 'deployment config', strategy: 'hybrid', limit: 10 });
-
-// With query transformation (HyDE generates hypothetical answer, embeds that instead)
-// EMBEDDED ONLY — enableHyDE not forwarded by HTTP API
-await memory.search({ query: 'deployment config', strategy: 'hybrid', enableHyDE: true });
-
-// With reranking (cross-encoder scores each result for precision)
-// EMBEDDED ONLY — enableRerank not forwarded by HTTP API
-await memory.search({ query: 'deployment config', strategy: 'hybrid', enableRerank: true });
-```
-
-### Retrieval Pipeline (Hybrid Strategy)
-
-```
-Query → Stage 0: Transform (decompose, HyDE, multi-query)
-      → Stage 1: Parallel retrieval (BM25/FTS5 + dense vector + graph traverse + community summaries)
-      → Stage 2: RRF fusion (k=60) + dedup
-      → Stage 3: Cross-encoder reranking → top-N
-```
-
----
-
-## Bi-Temporal Model
-
-Every entry tracks two timestamps for temporal reasoning:
-
-- **`eventTime`**: When the fact/event actually occurred
-- **`ingestTime`**: When it was stored in memory (auto-set)
-
-**Sidecar note**: All `StoreInput` fields (including `eventTime`, `id`, `parentId`, `ingestTime`) are forwarded by `MemoryClient.store()`. Temporal search filters (`eventTimeRange`, `asOf`) are forwarded by `MemoryClient.search()`. However, `filters` (source, importanceMin, parentId, contentType), `enableHyDE`, and `enableRerank` are still not forwarded by the search endpoint.
-
-```typescript
-// Store with explicit event time (works in both embedded and sidecar)
-await memory.store({
-  content: 'User changed address to 123 Main St',
-  type: 'long-term',
-  metadata: {},
-  eventTime: '2026-01-15T00:00:00Z',  // when it happened
-});
-
-// Query as-of (available via MemoryClient.queryAsOf() or HTTP endpoint)
-const snapshot = await memory.queryAsOf('2026-01-20T00:00:00Z', { type: 'long-term' });
-
-// Query by event time range (available via MemoryClient.queryByEventTime() or HTTP endpoint)
-const events = await memory.queryByEventTime('2026-01-01T00:00:00Z', '2026-02-01T00:00:00Z');
-
-// Filter by event time range in search (EMBEDDED ONLY via filters param)
-await memory.search({
-  query: 'address',
-  filters: { eventTimeRange: ['2026-01-01', '2026-02-01'] },
-});
-```
-
----
-
-## Consolidation Pipeline
-
-When `consolidate()` runs, it executes a 7-step pipeline:
-
-1. **Extract facts** — LLM or regex extraction of factual statements
-2. **Deduplicate** — Content hash + vector similarity (>0.90) → LLM classifies ADD/UPDATE/DELETE/NOOP
-3. **Resolve conflicts** — Detect contradictions, resolve by recency and source trust
-4. **Score importance** — LLM rates 1-10 (or heuristic: recency + access count + entity density)
-5. **Enrich graph** — Extract entities and relationships, merge with existing graph
-6. **Summarize** — Rolling session summaries, memory compaction
-7. **Decay** — Archive entries below importance threshold: `importance * 0.995^hours * (1 + 0.02 * min(accessCount, 20)) * eventAgeFactor`
-
-All steps have **non-LLM fallbacks** — consolidation works without an LLM, just less intelligently.
-
-### Pinned entries (opt out of consolidation)
-
-Entries stored with `pinned: true` are exempt from both the dedup step and the decay-archive step. `Memory.runDecay()` skips them via the same filter. Use for anchor rows whose ID is referenced by external systems — e.g. a file-catalog entry whose ID is stored as a foreign key in a downstream service's database:
-
-```typescript
-await memory.store({
-  content: '[File: "report.pdf" (application/pdf)]',
-  type: MemoryType.LONG_TERM,
-  metadata: { source: 'file-ingestion', filename: 'report.pdf' },
-  pinned: true, // external FK points here — must not be merged or archived
-});
-```
-
-Without `pinned`, short near-duplicate anchor content can trip the 0.95 cosine dedup threshold and get hard-deleted, orphaning the external reference. Pinned entries' exemption is SQL-side (`WHERE pinned = 0` filter on consolidation's query) and backed by a partial index `idx_memory_pinned WHERE pinned = 1`. Callers can query with `QueryFilters.excludePinned: true` to reproduce the same exclusion.
-
----
-
-## Community Detection
-
-When a `graphStore` is configured, the system can detect communities of related entities using the Louvain algorithm:
-
-```typescript
-import { CommunityDetector } from '@pyx-memory/core';
-
-const detector = new CommunityDetector(graphStore, llm);
-const communities = await detector.detect();
-// Each community: { id, nodeIds, summary? }
-// Summaries are used by hybrid RAG for corpus-level queries
-```
-
-Communities are leveraged by the hybrid RAG strategy to answer broad "what are the themes" queries.
-
----
-
-## Multi-Tenant Isolation
-
-pyx-memory supports multi-tenant data isolation via three scoping fields: `tenantId`, `userId`, and `teamId`.
-
-### Embedded mode
-
-```typescript
-// Instance-level scoping — auto-scopes ALL operations
-const memory = new Memory({
-  dataDir: './data',
-  tenantId: 'tenant-abc',
-});
-
-// Or per-operation scoping
-await memory.store({
-  content: 'Fact for this tenant.',
-  tenantId: 'tenant-abc',
-  userId: 'user-123',
-  teamId: 'team-eng',
-});
-
-// Search is auto-scoped to the instance tenantId
-const results = await memory.search({ query: 'facts', limit: 5 });
-
-// get/delete/clearSession/stats accept TenantScopeOptions
-const entry = await memory.get('entry-id', { tenantId: 'tenant-abc' });
-```
-
-### Sidecar mode
-
-```typescript
-// Set tenant headers via MemoryClientOptions
-const memory = new MemoryClient('http://localhost:7822', {
-  apiKey: process.env.MEMORY_API_KEY,
-  defaultHeaders: { 'X-Tenant-Id': 'tenant-abc', 'X-User-Id': 'user-123' },
-});
-
-// Or pass tenantId per-operation in the request body
-await memory.store({
-  content: 'Tenant-scoped memory.',
-  tenantId: 'tenant-abc',
-  userId: 'user-123',
-});
-```
-
-### Server enforcement
-
-Set `TENANT_MODE=multi` to require `X-Tenant-Id` on all operations. Requests without it are rejected (HTTP 400). When `TENANT_MODE=single` (default), tenant fields are stored but not enforced — backward compatible.
-
-Content hash dedup is scoped by `tenantId` to prevent cross-tenant collisions.
-
----
-
-## Sensitivity Classification & Encryption
-
-pyx-memory auto-classifies content sensitivity and optionally encrypts sensitive data at rest.
-
-### How it works
-
-1. Every `store()` scans content for credentials and PII
-2. Auto-classifies as `public`, `internal` (PII), or `secret` (credentials)
-3. Behavior depends on `SENSITIVITY_POLICY` env var (or embedded config):
-   - `flag`: classify only, no content modification (default)
-   - `redact`: replace detected credentials with `[REDACTED]`
-   - `block`: reject ingestion (HTTP 400)
-   - `encrypt`: encrypt `secret` entries at rest with AES-256-GCM
-
-### Embedded encryption
-
-```typescript
-const memory = new Memory({
-  dataDir: './data',
-  encryptionKey: Buffer.from(process.env.ENCRYPTION_KEY!, 'hex'), // 32 bytes
-});
-
-// Entries classified as 'secret' are automatically encrypted before storage
-// Decryption is automatic on get() when the encryption key is available
-```
-
-### Search access control
-
-```typescript
-// Embedded: filter by max sensitivity
-const results = await memory.search({
-  query: 'config',
-  maxSensitivity: 'internal', // secret entries are redacted in results
-});
-
-// Sidecar: via header
-const memory = new MemoryClient('http://localhost:7822', {
-  apiKey: 'key',
-  defaultHeaders: { 'X-Caller-Access-Level': 'internal' },
-});
-```
-
----
-
-## Confidence / Abstention
-
-Search results can include a confidence score that enables "I don't know" responses when retrieval quality is low.
-
-```typescript
-const results = await memory.search({
-  query: 'user preferences',
-  strategy: 'hybrid',
-  abstentionThreshold: 0.5, // 0-1, default 0.3
-});
-
-if (results.confidence?.shouldAbstain) {
-  // Confidence is below threshold — don't use these results
-  return "I don't have enough information to answer that.";
-}
-```
-
-The confidence score is computed from four signals: top score magnitude, score gap (top vs second), score spread (stdDev), and score separation (top vs mean). It's automatically included when using hybrid, graph, or agentic strategies.
-
----
-
-## Automatic Behaviors
-
-These happen automatically — no configuration needed:
-
-- **PII detection**: Every `store()` scans content and sets `metadata.piiDetected` + `metadata.piiTypes` if found
-- **Sensitivity classification**: Every `store()` auto-classifies content as `public`, `internal`, or `secret`
-- **Content hashing**: Every `store()` computes SHA-256 `contentHash`
-- **Access tracking**: Every `search()` increments `accessCount` and updates `lastAccessed` on returned entries
-- **FTS5 sync**: SQLite triggers keep full-text search index in sync with memory_entries
-- **Graph storage**: When `targets` includes `'graph'`, agent-provided `entities` are stored to the graph (best-effort — failures don't block store)
-- **Auto-registration on import**: Importing `@pyx-memory/core` registers StubEmbeddingProvider, LanceDBProvider, and NaiveRAGEngine
-- **Graph/Agentic RAG registration**: Memory constructor auto-registers GraphRAGEngine and AgenticRAGEngine when you pass `graphStore` or `reasoningProvider`
-- **Agent scoping**: If `agentId` is set in MemoryOptions, all operations auto-filter by that agent
-- **Tenant scoping**: If `tenantId` is set in MemoryOptions, all operations auto-filter by that tenant
-
----
-
-## Initialization Sequence
-
-```
-1. (Optional) Create and await graphStore.initialize()
-2. Construct: new Memory({ graphStore?, llm?, ... })   ← embedding is internal (BGE-M3)
-3. Await: memory.initialize()    ← creates SQLite DB + LanceDB vector store
-4. Use: store(), search(), etc.
-5. Cleanup: await memory.shutdown()
-6. (Optional) await graphStore.shutdown()
-```
-
-**Memory.initialize()** creates:
-- SQLite database at `{dataDir}/memory/memory.db` (with FTS5 + migrations)
-- LanceDB vector store at `{dataDir}/vectors/`
-- Both directories are created automatically (mkdirSync recursive)
-
-**Memory does NOT** call `graphStore.initialize()` — you must do this yourself before or after constructing Memory, but before calling `memory.initialize()`.