@pyxmate/memory 0.1.0-beta → 0.1.2-beta

package/skills/pyx-memory-integration/patterns/embedded.md

@@ -0,0 +1,223 @@
+# Embedded Integration Patterns
+
+For projects using pyx-memory directly in-process with full feature access.
+
+## Contents
+- [Pattern 1: Testing / Development](#pattern-1-testing--development)
+- [Pattern 2: Production](#pattern-2-production)
+- [Pattern 3: Production with Store Targets](#pattern-3-production-with-store-targets)
+- [Pattern 4: With Knowledge Graph](#pattern-4-with-knowledge-graph)
+- [Pattern 5: With LLM Lifecycle](#pattern-5-with-llm-lifecycle)
+- [Pattern 6: File Ingestion](#pattern-6-file-ingestion)
+- [Pattern 7: Factory with Auto-Mode Switching](#pattern-7-factory-with-auto-mode-switching)
+- [Adding as Git Submodule](#adding-as-git-submodule-recommended-for-embedded-mode)
+- [MemoryOptions Quick Reference](#memoryoptions-quick-reference)
+
+---
+
+## Pattern 1: Testing / Development
+
+```typescript
+import { Memory } from '@pyx-memory/core';
+
+const memory = new Memory({ dataDir: ':memory:' });
+await memory.initialize();
+// Memory internally creates a LocalEmbeddingProvider (BGE-M3, 1024d)
+// No embedder needed — embedding is fully managed
+```
+
+## Pattern 2: Production
+
+```typescript
+import { Memory } from '@pyx-memory/core';
+
+const memory = new Memory({ dataDir: './data' });
+await memory.initialize();
+// Embedding is handled internally by LocalEmbeddingProvider (BGE-M3 via @huggingface/transformers, 1024d)
+// No external embedding provider needed
+```
+
+## Pattern 3: Production with Store Targets
+
+```typescript
+import { Memory } from '@pyx-memory/core';
+
+const memory = new Memory({ dataDir: './data' });
+await memory.initialize();
+
+// Default: stores to sqlite + vector
+await memory.store({
+  content: 'User prefers dark mode',
+  type: 'long-term',
+  metadata: { source: 'settings' },
+});
+
+// Explicit targets: sqlite only (skip vector indexing)
+await memory.store({
+  content: 'Temporary note',
+  type: 'working',
+  metadata: {},
+  targets: ['sqlite'],
+});
+```
+
+## Pattern 4: With Knowledge Graph
+
+```typescript
+import { Memory, createGraphStore } from '@pyx-memory/core';
+import type { StoreTarget, IngestEntity, IngestRelationship } from '@pyx-memory/shared';
+
+// 1. Create and initialize graph store BEFORE Memory
+const graphStore = createGraphStore({}); // returns SQLiteGraphStore (default)
+await graphStore.initialize({});         // REQUIRED — Memory does NOT init this for you
+
+// For Neo4j instead: createGraphStore({ neo4jUrl: 'bolt://localhost:7687' })
+
+const memory = new Memory({
+  dataDir: './data',
+  graphStore, // enables graph RAG search
+});
+await memory.initialize();
+
+// Graph storage is agent-driven — YOU provide entities and relationships explicitly
+await memory.store({
+  content: 'Alice works at Acme Corp as a senior engineer',
+  type: 'long-term',
+  metadata: {},
+  targets: ['sqlite', 'vector', 'graph'],
+  entities: [
+    { name: 'Alice', type: 'PERSON', properties: { role: 'senior engineer' } },
+    { name: 'Acme Corp', type: 'ORGANIZATION' },
+  ],
+  relationships: [
+    { source: 'Alice', target: 'Acme Corp', type: 'WORKS_AT' },
+  ],
+});
+
+// Graph-aware search
+const results = await memory.search({ query: 'Alice employer', strategy: 'graph' });
+
+// Cleanup both
+await memory.shutdown();
+await graphStore.shutdown();
+```
+
+## Pattern 5: With LLM Lifecycle
+
+```typescript
+import { Memory } from '@pyx-memory/core';
+import type { LLMCallback } from '@pyx-memory/core';
+
+// LLMCallback: any function that takes a prompt string and returns a completion string
+const llm: LLMCallback = async (prompt) => {
+  const res = await fetch('https://api.anthropic.com/v1/messages', {
+    method: 'POST',
+    headers: {
+      'x-api-key': process.env.ANTHROPIC_API_KEY!,
+      'content-type': 'application/json',
+      'anthropic-version': '2023-06-01',
+    },
+    body: JSON.stringify({
+      model: 'claude-sonnet-4-20250514',
+      max_tokens: 1024,
+      messages: [{ role: 'user', content: prompt }],
+    }),
+  });
+  const data = await res.json() as any;
+  return data.content[0].text;
+};
+
+const memory = new Memory({
+  dataDir: './data',
+  llm, // enables LLM-powered lifecycle
+});
+await memory.initialize();
+
+// Now lifecycle methods use LLM intelligence
+await memory.consolidate();                    // LLM scoring + dedup + conflict resolution
+await memory.summarizeSession('session-123');   // LLM summarization
+await memory.runDecay();                        // importance-based archival
+```
+
+**Without LLM**: Lifecycle still works using heuristic fallbacks (regex extraction, embedding-distance dedup, formula-based scoring). LLM makes it smarter, not mandatory.
+
+## Pattern 6: File Ingestion
+
+```typescript
+import { IngestionAgent, Memory } from '@pyx-memory/core';
+
+const memory = new Memory({ dataDir: './data' });
+await memory.initialize();
+
+const agent = new IngestionAgent({
+  llm: myLlmCallback,                          // optional: smart classification + enrichment
+  embedder: (texts) => myEmbedder.embed(texts), // optional: semantic chunking (separate from Memory's internal embedder)
+  useSemanticChunking: true,
+  useStructuralChunking: false,
+  enableEnrichment: true,
+  enableMetadata: true,
+  enableHierarchical: false,                    // requires LLM
+});
+
+// Supported: .txt, .md, .csv, .pdf, .docx, .json, .html
+const buffer = Buffer.from(await Bun.file('report.pdf').arrayBuffer());
+const result = await agent.ingest(buffer, 'report.pdf', memory);
+// result: { filename, fileType, chunks, entryIds, totalCharacters }
+```
+
+**Note**: `IngestionAgent` may accept its own `embedder` for semantic chunking. This is separate from Memory's internal embedding — Memory handles its own embedding automatically.
+
+## Pattern 7: Factory with Auto-Mode Switching
+
+```typescript
+import { createMemory } from '@pyx-memory/core';
+
+// Embedded mode (default)
+const memory = createMemory({
+  dataDir: './data',
+});
+
+// Sidecar mode (when MEMORY_URL is set)
+const remote = createMemory({
+  memoryUrl: process.env.MEMORY_URL, // e.g., 'http://localhost:7822'
+  apiKey: process.env.MEMORY_API_KEY,
+});
+
+await memory.initialize();
+
+// WARNING: createMemory() returns MemoryInterface, NOT ExtendedMemoryInterface.
+// If you need lifecycle methods, cast:
+// const extended = memory as ExtendedMemoryInterface;
+// Or prefer `new Memory()` / `new MemoryClient()` directly.
+```
+
+---
+
+## Adding as Git Submodule (Recommended for Embedded Mode)
+
+```bash
+git submodule add https://github.com/fysoul17/pyx-memory-v1.git vendor/pyx-memory
+```
+
+Add to your `package.json` workspaces:
+
+```json
+{
+  "workspaces": [
+    "packages/*",
+    "vendor/pyx-memory/packages/shared",
+    "vendor/pyx-memory/packages/client",
+    "vendor/pyx-memory/packages/core"
+  ]
+}
+```
+
+Then: `bun install`
+
+---
+
+## MemoryOptions Quick Reference
+
+`embedder` has been removed from MemoryOptions. Memory internally creates a `LocalEmbeddingProvider` using BGE-M3 (1024 dimensions) via `@huggingface/transformers`. You never need to provide an embedding function.
+
+See [reference/types.md](../reference/types.md#memoryoptions-reference) for the full MemoryOptions table.

package/skills/pyx-memory-integration/reference/advanced.md

@@ -0,0 +1,139 @@
+# Advanced Features
+
+## RAG Strategies
+
+| Strategy | How It Works | Requirements |
+|----------|-------------|--------------|
+| `'naive'` | Embed query → vector similarity → top-K | (default, always available) |
+| `'graph'` | Extract entities → graph traversal → context expansion | `graphStore` in MemoryOptions |
+| `'hybrid'` | BM25 + vector + graph + community summaries → RRF fusion (k=60) → reranking | `graphStore` in MemoryOptions |
+| `'agentic'` | LLM decides strategy → iterative refinement (3 rounds) | `reasoningProvider` in MemoryOptions |
+
+```typescript
+// Naive (default)
+await memory.search({ query: 'user preferences', limit: 10 });
+
+// Graph
+await memory.search({ query: 'who works at Acme', strategy: 'graph', limit: 10 });
+
+// Hybrid (recommended for best quality)
+await memory.search({ query: 'deployment config', strategy: 'hybrid', limit: 10 });
+
+// With query transformation (HyDE generates hypothetical answer, embeds that instead)
+// EMBEDDED ONLY — enableHyDE not forwarded by HTTP API
+await memory.search({ query: 'deployment config', strategy: 'hybrid', enableHyDE: true });
+
+// With reranking (cross-encoder scores each result for precision)
+// EMBEDDED ONLY — enableRerank not forwarded by HTTP API
+await memory.search({ query: 'deployment config', strategy: 'hybrid', enableRerank: true });
+```
+
+### Retrieval Pipeline (Hybrid Strategy)
+
+```
+Query → Stage 0: Transform (decompose, HyDE, multi-query)
+      → Stage 1: Parallel retrieval (BM25/FTS5 + dense vector + graph traverse + community summaries)
+      → Stage 2: RRF fusion (k=60) + dedup
+      → Stage 3: Cross-encoder reranking → top-N
+```
+
+---
+
+## Bi-Temporal Model
+
+Every entry tracks two timestamps for temporal reasoning:
+
+- **`eventTime`**: When the fact/event actually occurred
+- **`ingestTime`**: When it was stored in memory (auto-set)
+
+**Sidecar note**: All `StoreInput` fields (including `eventTime`, `id`, `parentId`, `ingestTime`) are forwarded by `MemoryClient.store()`. Temporal search filters (`eventTimeRange`, `asOf`) are forwarded by `MemoryClient.search()`. However, `filters` (source, importanceMin, parentId, contentType), `enableHyDE`, and `enableRerank` are still not forwarded by the search endpoint.
+
+```typescript
+// Store with explicit event time (works in both embedded and sidecar)
+await memory.store({
+  content: 'User changed address to 123 Main St',
+  type: 'long-term',
+  metadata: {},
+  eventTime: '2026-01-15T00:00:00Z',  // when it happened
+});
+
+// Query as-of (available via MemoryClient.queryAsOf() or HTTP endpoint)
+const snapshot = await memory.queryAsOf('2026-01-20T00:00:00Z', { type: 'long-term' });
+
+// Query by event time range (available via MemoryClient.queryByEventTime() or HTTP endpoint)
+const events = await memory.queryByEventTime('2026-01-01T00:00:00Z', '2026-02-01T00:00:00Z');
+
+// Filter by event time range in search (EMBEDDED ONLY via filters param)
+await memory.search({
+  query: 'address',
+  filters: { eventTimeRange: ['2026-01-01', '2026-02-01'] },
+});
+```
+
+---
+
+## Consolidation Pipeline
+
+When `consolidate()` runs, it executes a 7-step pipeline:
+
+1. **Extract facts** — LLM or regex extraction of factual statements
+2. **Deduplicate** — Content hash + vector similarity (>0.90) → LLM classifies ADD/UPDATE/DELETE/NOOP
+3. **Resolve conflicts** — Detect contradictions, resolve by recency and source trust
+4. **Score importance** — LLM rates 1-10 (or heuristic: recency + access count + entity density)
+5. **Enrich graph** — Extract entities and relationships, merge with existing graph
+6. **Summarize** — Rolling session summaries, memory compaction
+7. **Decay** — Archive entries below importance threshold: `importance * 0.995^hours * (1 + 0.02 * min(accessCount, 20)) * eventAgeFactor`
+
+All steps have **non-LLM fallbacks** — consolidation works without an LLM, just less intelligently.
+
+---
+
+## Community Detection
+
+When a `graphStore` is configured, the system can detect communities of related entities using the Louvain algorithm:
+
+```typescript
+import { CommunityDetector } from '@pyx-memory/core';
+
+const detector = new CommunityDetector(graphStore, llm);
+const communities = await detector.detect();
+// Each community: { id, nodeIds, summary? }
+// Summaries are used by hybrid RAG for corpus-level queries
+```
+
+Communities are leveraged by the hybrid RAG strategy to answer broad "what are the themes" queries.
+
+---
+
+## Automatic Behaviors
+
+These happen automatically — no configuration needed:
+
+- **PII detection**: Every `store()` scans content and sets `metadata.piiDetected` + `metadata.piiTypes` if found
+- **Content hashing**: Every `store()` computes SHA-256 `contentHash`
+- **Access tracking**: Every `search()` increments `accessCount` and updates `lastAccessed` on returned entries
+- **FTS5 sync**: SQLite triggers keep full-text search index in sync with memory_entries
+- **Graph storage**: When `targets` includes `'graph'`, agent-provided `entities` are stored to the graph (best-effort — failures don't block store)
+- **Auto-registration on import**: Importing `@pyx-memory/core` registers StubEmbeddingProvider, LanceDBProvider, and NaiveRAGEngine
+- **Graph/Agentic RAG registration**: Memory constructor auto-registers GraphRAGEngine and AgenticRAGEngine when you pass `graphStore` or `reasoningProvider`
+- **Agent scoping**: If `agentId` is set in MemoryOptions, all operations auto-filter by that agent
+
+---
+
+## Initialization Sequence
+
+```
+1. (Optional) Create and await graphStore.initialize()
+2. Construct: new Memory({ graphStore?, llm?, ... })   ← embedding is internal (BGE-M3)
+3. Await: memory.initialize()    ← creates SQLite DB + LanceDB vector store
+4. Use: store(), search(), etc.
+5. Cleanup: await memory.shutdown()
+6. (Optional) await graphStore.shutdown()
+```
+
+**Memory.initialize()** creates:
+- SQLite database at `{dataDir}/memory/memory.db` (with FTS5 + migrations)
+- LanceDB vector store at `{dataDir}/vectors/`
+- Both directories are created automatically (mkdirSync recursive)
+
+**Memory does NOT** call `graphStore.initialize()` — you must do this yourself before or after constructing Memory, but before calling `memory.initialize()`.

package/skills/pyx-memory-integration/reference/http-api.md

@@ -0,0 +1,82 @@
+# pyx-memory HTTP API Reference
+
+## Authentication
+
+When the server has `API_KEY` configured, all requests (except `/health`) require one of:
+
+```
+Authorization: Bearer <your-api-key>
+X-API-Key: <your-api-key>
+```
+
+Destructive operations (DELETE, forget, decay, consolidate, reindex) require `ADMIN_API_KEY` when configured (falls back to `API_KEY`).
+
+`MemoryClient` handles this automatically when `apiKey` is passed to the constructor:
+```typescript
+const client = new MemoryClient('http://localhost:7822', process.env.MEMORY_API_KEY);
+```
+
+## Core (10 endpoints)
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/health` | Public health check (status only — no internals exposed) |
+| GET | `/admin/health` | Admin health check (version, uptime, embedding provider, memory stats) |
+| POST | `/api/memory/ingest` | Store a memory (JSON: `{ content, type, metadata, agentId?, sessionId?, targets?, entities?, relationships?, importance?, source?, eventTime?, id?, parentId?, ingestTime? }`) |
+| POST | `/api/memory/ingest/file` | Upload file (multipart, 50MB limit) |
+| GET | `/api/memory/search?query=...&strategy=...&limit=...` | Search memories — **does NOT support** filters, enableHyDE, enableRerank |
+| GET | `/api/memory/stats` | Memory statistics |
+| GET | `/api/memory/entries?page=...&limit=...` | List entries (paginated) |
+| GET | `/api/memory/entries/:id` | Get entry by ID |
+| DELETE | `/api/memory/entries/:id` | Delete entry |
+| DELETE | `/api/memory/sessions/:sessionId` | Clear session |
+
+## Graph (4 endpoints)
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| GET | `/api/memory/graph/nodes?name=...&type=...` | Find graph nodes |
+| GET | `/api/memory/graph/edges` | Graph stats |
+| GET | `/api/memory/graph/relationships` | List relationships |
+| POST | `/api/memory/graph/query` | Traverse (JSON: `{ nodeId, depth? }`) |
+
+## Lifecycle (9 endpoints)
+
+| Method | Endpoint | Description |
+|--------|----------|-------------|
+| POST | `/api/memory/consolidate` | Run consolidation pipeline |
+| POST | `/api/memory/forget/:id` | Soft-delete (JSON: `{ reason? }`) |
+| POST | `/api/memory/sessions/:sid/summarize` | Summarize session |
+| POST | `/api/memory/decay` | Run decay pass |
+| POST | `/api/memory/reindex` | Rebuild FTS5 + vector indices |
+| DELETE | `/api/memory/source/:source` | Delete by source |
+| GET | `/api/memory/consolidation-log` | Audit trail |
+| GET | `/api/memory/query-as-of?asOf=...` | Bi-temporal point-in-time query (asOf, type, agentId, source, limit) |
+| GET | `/api/memory/query-by-event-time?startTime=...&endTime=...` | Bi-temporal event time range query (startTime, endTime, type, agentId, source, limit) |
+
+## Response Format
+
+All responses follow: `{ success: boolean, data?: T, error?: string }`
+
+---
+
+## Server Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `MEMORY_SERVER_PORT` | `7822` | HTTP server port |
+| `DATA_DIR` | `./data` | Storage directory |
+| ~~`EMBEDDING_PROVIDER`~~ | — | **Removed** — embedding is now internal (BGE-M3 via LocalEmbeddingProvider) |
+| ~~`EMBEDDING_API_KEY`~~ | — | **Removed** — no external embedding provider needed |
+| ~~`EMBEDDING_MODEL`~~ | — | **Removed** — model is always BGE-M3 (ONNX int8 quantized) |
+| `EMBEDDING_DIMENSIONS` | `1024` | Dimension override for the internal LocalEmbeddingProvider (default: 1024) |
+| `NEO4J_URL` | — | Neo4j bolt URL (enables Neo4j graph store) |
+| `NEO4J_USERNAME` | `neo4j` | Neo4j username |
+| `NEO4J_PASSWORD` | — | Neo4j password (never logged) |
+| `API_KEY` | — | API key for authenticating requests. Unset = open access |
+| `ADMIN_API_KEY` | — | Separate admin key for destructive ops (DELETE, forget, decay, consolidate, reindex). Falls back to `API_KEY` |
+| `CORS_ORIGIN` | `*` | CORS allowed origin. Set to specific domain in production |
+| `MAX_REQUEST_BODY_MB` | `10` | Maximum request body size in MB |
+| `NODE_ENV` | `development` | Set to `production` to mask 5xx error details and enable HSTS |
+| `PII_POLICY` | `flag` | PII handling: `flag` (detect + tag), `redact` (replace with [REDACTED]), `block` (reject 400) |
+| `RATE_LIMIT_RPM` | `0` | Requests per minute per IP. 0 = disabled |

package/skills/pyx-memory-integration/reference/parity.md

@@ -0,0 +1,63 @@
+# Feature Parity: Embedded vs Sidecar
+
+**Most features are available in both modes.** The HTTP API and MemoryClient forward all core `StoreInput` fields. A few advanced search parameters remain embedded-only.
+
+## store() Field Parity
+
+| Field | Embedded `Memory.store()` | Sidecar `MemoryClient.store()` |
+|-------|--------------------------|-------------------------------|
+| content | yes | yes |
+| type | yes | yes |
+| metadata | yes | yes |
+| agentId | yes | yes |
+| sessionId | yes | yes |
+| targets | yes | yes |
+| entities | yes | yes |
+| relationships | yes | yes |
+| importance | yes | yes |
+| source | yes | yes |
+| eventTime | yes | yes |
+| id (custom) | yes | yes |
+| parentId | yes | yes |
+| ingestTime | yes | yes |
+
+**All StoreInput fields are forwarded.** Full parity.
+
+## search() Param Parity
+
+| Param | Embedded `Memory.search()` | Sidecar `MemoryClient.search()` |
+|-------|---------------------------|--------------------------------|
+| query, limit, type, agentId, strategy | yes | yes |
+| eventTimeRange (bi-temporal search) | yes | yes |
+| asOf (point-in-time search) | yes | yes |
+| **filters** (source, importanceMin, parentId, contentType) | yes | **NO** — not forwarded |
+| **enableHyDE** | yes | **NO** — not forwarded |
+| **enableRerank** | yes | **NO** — not forwarded |
+
+**Impact**: Sidecar consumers cannot use advanced search filters, HyDE query expansion, or reranking. Temporal search filters (eventTimeRange, asOf) ARE supported.
+
+## Endpoint Coverage by Client
+
+| Server Endpoint | MemoryClient | DashboardClient |
+|----------------|-------------|-----------------|
+| All core (9) | yes | yes (inherited) |
+| Graph nodes/edges/query (3) | yes (concrete methods) | yes (inherited) |
+| **Graph relationships** | **NO** | yes (`graphRelationships()`) |
+| All lifecycle (7) | yes | yes (inherited) |
+| **Consolidation log** | **NO** | yes (`consolidationLog()`) |
+| Query as-of (bi-temporal) | yes (`queryAsOf()`) | yes (inherited) |
+| Query by event time | yes (`queryByEventTime()`) | yes (inherited) |
+
+## Security Features (Server-side)
+
+| Feature | Configuration |
+|---------|--------------|
+| API key auth | `API_KEY` env var (unset = open access) |
+| Admin key for destructive ops | `ADMIN_API_KEY` env var |
+| Rate limiting | `RATE_LIMIT_RPM` env var (0 = disabled) |
+| CORS | `CORS_ORIGIN` env var (default: `*`) |
+| Security headers | Always on (CSP, X-Frame-Options, nosniff) |
+| HSTS | Auto-enabled when `NODE_ENV=production` |
+| PII policy | `PII_POLICY` env var (`flag` / `redact` / `block`) |
+| Body size limit | `MAX_REQUEST_BODY_MB` env var (default: 10) |
+| Error masking | 5xx details hidden when `NODE_ENV=production` |