@pyxmate/memory 0.20.4 → 0.21.1

package/skills/pyx-memory/patterns/consumer.md

@@ -1,129 +0,0 @@
-# Consumer Integration Patterns
-
-For downstream projects that **consume** pyx-memory (e.g., agent-forge, custom AI agents).
-
-**Rule: Only depend on `@pyx-memory/client` + `@pyx-memory/shared`. Never import `@pyx-memory/core`.**
-
-`@pyx-memory/core` is pyx-memory's internal implementation (SQLiteStore, LanceDB, embedding providers,
-RAG engines, graph stores). Importing it creates tight coupling — any internal change can break your project.
-
----
-
-## Pattern 8: Consumer (Sidecar-Only)
-
-```typescript
-import { MemoryClient, MemoryServerError } from '@pyx-memory/client';
-import type { MemoryInterface, ExtendedMemoryInterface } from '@pyx-memory/client';
-import type { MemoryEntry, MemorySearchResult } from '@pyx-memory/shared';
-
-const MEMORY_URL = process.env.MEMORY_URL; // e.g., 'http://localhost:7822'
-
-let memory: MemoryClient | null = null;
-
-if (MEMORY_URL) {
-  // Simple: API key only
-  memory = new MemoryClient(MEMORY_URL, process.env.MEMORY_API_KEY);
-
-  // Multi-tenant: API key + tenant headers
-  // memory = new MemoryClient(MEMORY_URL, {
-  //   apiKey: process.env.MEMORY_API_KEY,
-  //   defaultHeaders: { 'X-Tenant-Id': 'tenant-abc', 'X-User-Id': 'user-123' },
-  // });
-
-  await memory.initialize(); // verifies connectivity
-}
-
-// Use memory if available
-if (memory) {
-  await memory.store({ content: 'fact', type: 'long-term', metadata: {} });
-  const results = await memory.search({ query: 'fact', limit: 5 });
-
-  // Graph queries — available on MemoryClient directly (no @pyx-memory/core needed)
-  const nodes = await memory.graphNodes();
-  const traversal = await memory.graphQuery({ nodeId: 'node-1', depth: 2 });
-
-  // File ingestion — native NDJSON streaming via ingestFileEvents()
-  const file = new File([buffer], 'report.pdf', { type: 'application/pdf' });
-  for await (const event of memory.ingestFileEvents(file)) {
-    if (event.type === 'progress') console.log(`[${event.stage}] ${event.message ?? ''}`);
-    if (event.type === 'error') throw new Error(event.message ?? event.error);
-    // event.type === 'result' carries the terminal { filename, chunks, entryIds, ... }
-  }
-
-  // Image ingestion with AI description
-  // pyx-memory saves the original image to {DATA_DIR}/files/ and indexes the
-  // description in vector + SQLite for semantic search. Without a description,
-  // images get a minimal placeholder ("[Image] filename (size KB)").
-  const image = new File([imgBuffer], 'photo.png', { type: 'image/png' });
-  for await (const _ of memory.ingestFileEvents(image, {
-    description: 'A screenshot of the login page showing an error dialog',
-  })) {
-    /* drain to terminal */
-  }
-
-  // Lifecycle
-  await memory.consolidate();
-  await memory.runDecay();
-}
-```
-
----
-
-## Pattern 9: Graceful Degradation (DisabledMemory)
-
-When your project should work with or without pyx-memory, implement a no-op wrapper.
-See [examples/disabled-memory.ts](../examples/disabled-memory.ts) for a copy-paste ready implementation.
-
-```typescript
-import { MemoryClient } from '@pyx-memory/client';
-
-const memory: MemoryInterface = process.env.MEMORY_URL
-  ? new MemoryClient(process.env.MEMORY_URL, process.env.MEMORY_API_KEY)
-  : new DisabledMemory();
-
-await memory.initialize();
-```
-
----
-
-## Health Endpoint Pattern
-
-Report memory status in your app's health check:
-
-```typescript
-// In your health route
-const memoryStatus = process.env.MEMORY_URL
-  ? { status: 'connected', url: process.env.MEMORY_URL }
-  : { status: 'disabled' };
-```
-
----
-
-## Adding as Sidecar (Docker)
-
-```yaml
-# docker-compose.yaml
-services:
-  memory:
-    build:
-      context: ./vendor/pyx-memory
-      dockerfile: docker/Dockerfile
-    ports:
-      - "7822:7822"
-    volumes:
-      - memory-data:/data
-    environment:
-      - DATA_DIR=/data
-      - API_KEY=${MEMORY_API_KEY}              # auth for all requests
-      # - TENANT_MODE=multi                    # require X-Tenant-Id on all ops
-      # - SENSITIVITY_POLICY=encrypt           # encrypt secret entries at rest
-      # - ENCRYPTION_KEY=${ENCRYPTION_KEY}     # 32 bytes as 64 hex chars
-      # Embedding is internal (BGE-M3, 1024d) — no API keys needed
-
-  your-app:
-    environment:
-      - MEMORY_URL=http://memory:7822
-
-volumes:
-  memory-data:
-```

package/skills/pyx-memory/patterns/embedded.md

@@ -1,249 +0,0 @@
-# Embedded Integration Patterns
-
-For projects using pyx-memory directly in-process with full feature access.
-
-## Contents
-- [Pattern 1: Testing / Development](#pattern-1-testing--development)
-- [Pattern 2: Production](#pattern-2-production)
-- [Pattern 3: Production with Store Targets](#pattern-3-production-with-store-targets)
-- [Pattern 4: With Knowledge Graph](#pattern-4-with-knowledge-graph)
-- [Pattern 5: With LLM Lifecycle](#pattern-5-with-llm-lifecycle)
-- [Pattern 6: File Ingestion](#pattern-6-file-ingestion)
-- [Pattern 7: Factory with Auto-Mode Switching](#pattern-7-factory-with-auto-mode-switching)
-- [Adding as Git Submodule](#adding-as-git-submodule-recommended-for-embedded-mode)
-- [MemoryOptions Quick Reference](#memoryoptions-quick-reference)
-
----
-
-## Pattern 1: Testing / Development
-
-```typescript
-import { Memory } from '@pyx-memory/core';
-
-const memory = new Memory({ dataDir: ':memory:' });
-await memory.initialize();
-// Memory internally creates a LocalEmbeddingProvider (BGE-M3, 1024d)
-// No embedder needed — embedding is fully managed
-```
-
-## Pattern 2: Production
-
-```typescript
-import { Memory } from '@pyx-memory/core';
-
-const memory = new Memory({ dataDir: './data' });
-await memory.initialize();
-// Embedding is handled internally by LocalEmbeddingProvider (BGE-M3 via @huggingface/transformers, 1024d)
-// No external embedding provider needed
-```
-
-## Pattern 2b: Production with Multi-Tenant + Encryption
-
-```typescript
-import { Memory } from '@pyx-memory/core';
-
-const memory = new Memory({
-  dataDir: './data',
-  tenantId: 'tenant-abc',       // Auto-scope all operations to this tenant
-  encryptionKey: Buffer.from(process.env.ENCRYPTION_KEY!, 'hex'), // 32 bytes for AES-256-GCM
-});
-await memory.initialize();
-
-// All store/search/get/delete/list operations are automatically tenant-scoped
-await memory.store({
-  content: 'Secret API key: sk-abc123',  // auto-classified as 'secret', encrypted at rest
-  type: 'long-term',
-  metadata: { source: 'config' },
-});
-
-// Search respects tenant isolation + sensitivity filtering
-const results = await memory.search({
-  query: 'API key',
-  maxSensitivity: 'internal', // 'secret' entries are redacted in results
-});
-```
-
-## Pattern 3: Production with Store Targets
-
-```typescript
-import { Memory } from '@pyx-memory/core';
-
-const memory = new Memory({ dataDir: './data' });
-await memory.initialize();
-
-// Default: stores to sqlite + vector
-await memory.store({
-  content: 'User prefers dark mode',
-  type: 'long-term',
-  metadata: { source: 'settings' },
-});
-
-// Explicit targets: sqlite only (skip vector indexing)
-await memory.store({
-  content: 'Temporary note',
-  type: 'working',
-  metadata: {},
-  targets: ['sqlite'],
-});
-```
-
-## Pattern 4: With Knowledge Graph
-
-```typescript
-import { Memory, createGraphStore } from '@pyx-memory/core';
-import type { StoreTarget, IngestEntity, IngestRelationship } from '@pyx-memory/shared';
-
-// 1. Create and initialize graph store BEFORE Memory
-const graphStore = createGraphStore({}); // returns SQLiteGraphStore (default)
-await graphStore.initialize({});         // REQUIRED — Memory does NOT init this for you
-
-// For Neo4j instead: createGraphStore({ neo4jUrl: 'bolt://localhost:7687' })
-
-const memory = new Memory({
-  dataDir: './data',
-  graphStore, // enables graph RAG search
-});
-await memory.initialize();
-
-// Graph storage is agent-driven — YOU provide entities and relationships explicitly
-await memory.store({
-  content: 'Alice works at Acme Corp as a senior engineer',
-  type: 'long-term',
-  metadata: {},
-  targets: ['sqlite', 'vector', 'graph'],
-  entities: [
-    { name: 'Alice', type: 'PERSON', properties: { role: 'senior engineer' } },
-    { name: 'Acme Corp', type: 'ORGANIZATION' },
-  ],
-  relationships: [
-    { source: 'Alice', target: 'Acme Corp', type: 'WORKS_AT' },
-  ],
-});
-
-// Graph-aware search
-const results = await memory.search({ query: 'Alice employer', strategy: 'graph' });
-
-// Cleanup both
-await memory.shutdown();
-await graphStore.shutdown();
-```
-
-## Pattern 5: With LLM Lifecycle
-
-```typescript
-import { Memory } from '@pyx-memory/core';
-import type { LLMCallback } from '@pyx-memory/core';
-
-// LLMCallback: any function that takes a prompt string and returns a completion string
-const llm: LLMCallback = async (prompt) => {
-  const res = await fetch('https://api.anthropic.com/v1/messages', {
-    method: 'POST',
-    headers: {
-      'x-api-key': process.env.ANTHROPIC_API_KEY!,
-      'content-type': 'application/json',
-      'anthropic-version': '2023-06-01',
-    },
-    body: JSON.stringify({
-      model: 'claude-sonnet-4-20250514',
-      max_tokens: 1024,
-      messages: [{ role: 'user', content: prompt }],
-    }),
-  });
-  const data = await res.json() as any;
-  return data.content[0].text;
-};
-
-const memory = new Memory({
-  dataDir: './data',
-  llm, // enables LLM-powered lifecycle
-});
-await memory.initialize();
-
-// Now lifecycle methods use LLM intelligence
-await memory.consolidate();                    // LLM scoring + dedup + conflict resolution
-await memory.summarizeSession('session-123');   // LLM summarization
-await memory.runDecay();                        // importance-based archival
-```
-
-**Without LLM**: Lifecycle still works using heuristic fallbacks (regex extraction, embedding-distance dedup, formula-based scoring). LLM makes it smarter, not mandatory.
-
-## Pattern 6: File Ingestion
-
-```typescript
-import { IngestionAgent, Memory } from '@pyx-memory/core';
-
-const memory = new Memory({ dataDir: './data' });
-await memory.initialize();
-
-const agent = new IngestionAgent({
-  llm: myLlmCallback,                          // optional: smart classification + enrichment
-  embedder: (texts) => myEmbedder.embed(texts), // optional: semantic chunking (separate from Memory's internal embedder)
-  useSemanticChunking: true,
-  useStructuralChunking: false,
-  enableEnrichment: true,
-  enableMetadata: true,
-  enableHierarchical: false,                    // requires LLM
-});
-
-// Supported: .txt, .md, .csv, .pdf, .docx, .json, .html
-const buffer = Buffer.from(await Bun.file('report.pdf').arrayBuffer());
-const result = await agent.ingest(buffer, 'report.pdf', memory);
-// result: { filename, fileType, chunks, entryIds, totalCharacters }
-```
-
-**Note**: `IngestionAgent` may accept its own `embedder` for semantic chunking. This is separate from Memory's internal embedding — Memory handles its own embedding automatically.
-
-## Pattern 7: Factory with Auto-Mode Switching
-
-```typescript
-import { createMemory } from '@pyx-memory/core';
-
-// Embedded mode (default)
-const memory = createMemory({
-  dataDir: './data',
-});
-
-// Sidecar mode (when MEMORY_URL is set)
-const remote = createMemory({
-  memoryUrl: process.env.MEMORY_URL, // e.g., 'http://localhost:7822'
-  apiKey: process.env.MEMORY_API_KEY,
-});
-
-await memory.initialize();
-
-// WARNING: createMemory() returns MemoryInterface, NOT ExtendedMemoryInterface.
-// If you need lifecycle methods, cast:
-// const extended = memory as ExtendedMemoryInterface;
-// Or prefer `new Memory()` / `new MemoryClient()` directly.
-```
-
----
-
-## Adding as Git Submodule (Recommended for Embedded Mode)
-
-```bash
-git submodule add https://github.com/fysoul17/pyx-memory-v1.git vendor/pyx-memory
-```
-
-Add to your `package.json` workspaces:
-
-```json
-{
-  "workspaces": [
-    "packages/*",
-    "vendor/pyx-memory/packages/shared",
-    "vendor/pyx-memory/packages/client",
-    "vendor/pyx-memory/packages/core"
-  ]
-}
-```
-
-Then: `bun install`
-
----
-
-## MemoryOptions Quick Reference
-
-`embedder` has been removed from MemoryOptions. Memory internally creates a `LocalEmbeddingProvider` using BGE-M3 (1024 dimensions) via `@huggingface/transformers`. You never need to provide an embedding function.
-
-See [reference/types.md](../reference/types.md#memoryoptions-reference) for the full MemoryOptions table.

package/skills/pyx-memory/patterns/file-uploads.md

@@ -1,78 +0,0 @@
-# Pattern: File Uploads
-
-This is consumer-side guidance: how to decide whether to forward a file
-straight to `ingestFileEvents()` or to pre-extract its text upstream first.
-
-## TL;DR
-
-| Format | Default action |
-|---|---|
-| `txt`, `md`, `csv`, `tsv`, `log`, `json`, `jsonl`, `html`, `htm` | Forward raw — pyx-memory streams. |
-| `pdf` | Forward raw — pyx-memory streams via poppler. Install `poppler-utils` on the server image. |
-| Images (`png`, `jpg`, `jpeg`, `webp`, `gif`, `bmp`, `tiff`, `svg`) | Forward raw with a `description` (use vision capability). |
-| `docx` ≤ 10 MB | Forward raw. |
-| `docx` > 10 MB | **Pre-extract upstream** as `.txt` or `.md`. The server returns a `MemoryError` if you don't. |
-| `xlsx` (large or shared-string-heavy) | **Pre-extract upstream** as `.xxx.xlsx.txt` for deterministic UX. |
-| `pptx` (production UX) | **Pre-extract upstream** as `.xxx.pptx.txt` for deterministic UX. |
-
-"Production UX" here means: you can't afford a single hung upload to wedge
-the user-facing layer for 30+ seconds, you need actionable error messages
-on every failure, and you have your own copy of the original file
-(separate from pyx-memory's internal storage).
-
-## Why pre-extract pptx and large xlsx
-
-The server's pptx parser decompresses the full ZIP in memory (~3× file
-size peak). The xlsx parser streams rows but caches shared strings for
-the entire workbook (`ExcelJS.WorkbookReader { sharedStrings: 'cache' }`).
-Both are bounded by the 100 MB file / 200 MB decompressed caps, but
-"bounded" is not "constant" — pathological files (huge shared-string
-tables, dense cell formulas, embedded media) can push peak memory and
-parse time well past what naive callers expect.
-
-If you control the upload boundary (e.g. you operate a runtime/proxy
-service that fronts pyx-memory), upstream pre-extraction lets you:
-
-1. **Catch parse failures at your boundary**, where you can return an
-   actionable error to the user (`"Excel formula evaluation failed at
-   sheet 'Q3 Revenue', row 412"`) instead of a generic upstream 5xx.
-2. **Bound the wire payload to pyx-memory** — text/plain only — so the
-   memory server's parser is never the bottleneck.
-3. **Keep the original binary in your own storage**, so users can still
-   download the file. pyx-memory's catalog only holds the indexed text.
-
-## Reference implementation
-
-[ai-rag-hub](https://github.com/fysoul17/one-query-v1) (a consumer of
-pyx-memory) implements this pattern in its runtime:
-
-- `packages/server/src/text-extractors.ts` — local extractors for `pptx`
-  and `xlsx`, both wrapped in the same OOXML safety envelope (zip-bomb
-  defense, path-traversal check, macro reject, decompressed-size cap,
-  char limit).
-- `packages/server/src/routes/memory.ts` — `prepareFileForIngest`
-  dispatches via `getTextExtractor(mimeType)`; matched formats are
-  re-uploaded as `<original>.txt` with `text/plain`. Catalog metadata
-  flags `downloadableFromMemory: false` so the consumer's own
-  `/api/team/documents/[id]/download` route serves the original
-  binary instead.
-
-## When NOT to pre-extract
-
-Single-tenant lab usage, internal tools, batch jobs where a 30-second
-parse latency is acceptable, or any case where you don't have your own
-copy of the file and need pyx-memory's `GET /api/memory/files/download/:filename`
-to return the original binary. In those cases, the native pyx-memory
-parsers are exactly what you want.
-
-## What about other formats
-
-- **HTML**: pyx-memory strips `<script>` and `<style>` during parse
-  (`parsers/html.ts`). No upstream sanitizer needed for indexing.
-- **PDF with images**: use the SDK's two-phase enrichment via
-  `EnrichmentCallbacks` — see `reference/sdk-guide.md`. Don't pre-extract
-  the PDF as text and lose image enrichment.
-- **SVG**: currently classified as an image but pyx-memory's image
-  parser only stores it as a placeholder; if you need SVG text indexed,
-  pre-extract the `<text>` and `<desc>` content yourself or convert to
-  raster + describeImage.