cozo-memory 1.0.4 → 1.0.6

package/README.md

@@ -28,6 +28,8 @@
 - [Development](#development)
 - [User Preference Profiling](#user-preference-profiling-mem0-style)
 - [Troubleshooting](#troubleshooting)
+- [Roadmap](#roadmap)
+- [Contributing](#contributing)
 - [License](#license)
 
 ## Quick Start
@@ -81,6 +83,10 @@ Now you can add the server to your MCP client (e.g. Claude Desktop).
 
 📦 **Export/Import (since v1.8)** - Export to JSON, Markdown, or Obsidian-ready ZIP; import from Mem0, MemGPT, Markdown, or native format
 
+📄 **PDF Support (since v1.9)** - Direct PDF ingestion with text extraction via pdfjs-dist; supports file path and content parameters
+
+🕐 **Dual Timestamp Format (since v1.9)** - All timestamps returned in both Unix microseconds and ISO 8601 format for maximum flexibility
+
 ### Detailed Features
 - **Hybrid Search (v0.7 Optimized)**: Combination of semantic search (HNSW), **Full-Text Search (FTS)**, and graph signals, merged via Reciprocal Rank Fusion (RRF).
 - **Full-Text Search (FTS)**: Native CozoDB v0.7 FTS indices with stemming, stopword filtering, and robust query sanitizing (cleaning of `+ - * / \ ( ) ? .`) for maximum stability.
@@ -224,7 +230,11 @@ graph LR
 
 ### Prerequisites
 - Node.js 20+ (recommended)
-- CozoDB native dependency is installed via `cozo-node`.
+- **RAM: 1.7 GB minimum** (for default bge-m3 model)
+  - Model download: ~600 MB
+  - Runtime memory: ~1.1 GB
+  - For lower-spec machines, see [Embedding Model Options](#embedding-model-options) below
+- CozoDB native dependency is installed via `cozo-node`
 
 ### Via npm (Easiest)
 
@@ -257,6 +267,62 @@ Notes:
 - On first start, `@xenova/transformers` downloads the embedding model (may take time).
 - Embeddings are processed on the CPU.
 
+### Embedding Model Options
+
+CozoDB Memory supports multiple embedding models via the `EMBEDDING_MODEL` environment variable:
+
+| Model | Size | RAM | Dimensions | Best For |
+|-------|------|-----|------------|----------|
+| `Xenova/bge-m3` (default) | ~600 MB | ~1.7 GB | 1024 | High accuracy, production use |
+| `Xenova/all-MiniLM-L6-v2` | ~80 MB | ~400 MB | 384 | Low-spec machines, development |
+| `Xenova/bge-small-en-v1.5` | ~130 MB | ~600 MB | 384 | Balanced performance |
+
+**Configuration Options:**
+
+**Option 1: Using `.env` file (Easiest for beginners)**
+
+```bash
+# Copy the example file
+cp .env.example .env
+
+# Edit .env and set your preferred model
+EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2
+```
+
+**Option 2: MCP Server Config (For Claude Desktop / Kiro)**
+
+```json
+{
+  "mcpServers": {
+    "cozo-memory": {
+      "command": "npx",
+      "args": ["cozo-memory"],
+      "env": {
+        "EMBEDDING_MODEL": "Xenova/all-MiniLM-L6-v2"
+      }
+    }
+  }
+}
+```
+
+**Option 3: Command Line**
+
+```bash
+# Use lightweight model for development
+EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2 npm run start
+```
+
+**Download Model First (Recommended):**
+
+```bash
+# Set model in .env or via command line, then:
+EMBEDDING_MODEL=Xenova/all-MiniLM-L6-v2 npm run download-model
+```
+}
+```
+
+**Note:** Changing models requires re-embedding existing data. The model is downloaded once on first use.
+
 ## Start / Integration
 
 ### MCP Server (stdio)
@@ -269,6 +335,67 @@ npm run start
 
 Default database path: `memory_db.cozo.db` in project root (created automatically).
 
+### CLI Tool
+
+CozoDB Memory includes a full-featured CLI for all operations:
+
+```bash
+# System operations
+cozo-memory system health
+cozo-memory system metrics
+
+# Entity operations
+cozo-memory entity create -n "MyEntity" -t "person" -m '{"age": 30}'
+cozo-memory entity get -i <entity-id>
+cozo-memory entity delete -i <entity-id>
+
+# Observations
+cozo-memory observation add -i <entity-id> -t "Some note"
+
+# Relations
+cozo-memory relation create --from <id1> --to <id2> --type "knows" -s 0.8
+
+# Search
+cozo-memory search query -q "search term" -l 10
+cozo-memory search context -q "context query"
+
+# Graph operations
+cozo-memory graph explore -s <entity-id> -h 3
+cozo-memory graph pagerank
+cozo-memory graph communities
+
+# Export/Import
+cozo-memory export json -o backup.json --include-metadata --include-relationships --include-observations
+cozo-memory export markdown -o notes.md
+cozo-memory export obsidian -o vault.zip
+cozo-memory import file -i data.json -f cozo
+
+# All commands support -f json or -f pretty for output formatting
+```
+
+### TUI (Terminal User Interface)
+
+Interactive TUI with mouse support powered by Python Textual:
+
+```bash
+# Install Python dependencies (one-time)
+pip install textual
+
+# Launch TUI
+npm run tui
+# or directly:
+cozo-memory-tui
+```
+
+**TUI Features:**
+- 🖱️ Full mouse support (click buttons, scroll, select inputs)
+- ⌨️ Keyboard shortcuts (q=quit, h=help, r=refresh)
+- 📊 Interactive menus for all operations
+- 🎨 Rich terminal UI with colors and animations
+- 📋 Real-time results display
+- 🔍 Forms for entity creation, search, graph operations
+- 📤 Export/Import wizards
+
 ### Claude Desktop Integration
 
 #### Using npx (Recommended)
@@ -335,6 +462,14 @@ DB_ENGINE=rocksdb npm run dev
 | **RocksDB** | Prepared & Tested | For high-performance or very large datasets. |
 | **MDBX** | Not supported | Requires manual build of `cozo-node` from source. |
 
+### Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `DB_ENGINE` | `sqlite` | Database backend: `sqlite` or `rocksdb` |
+| `EMBEDDING_MODEL` | `Xenova/bge-m3` | Embedding model (see [Embedding Model Options](#embedding-model-options)) |
+| `PORT` | `3001` | HTTP API bridge port (if using `npm run bridge`) |
+
 ---
 
 ## Data Model
@@ -367,7 +502,10 @@ Actions:
 - `create_relation`: `{ from_id, to_id, relation_type, strength?, metadata? }`
 - `run_transaction`: `{ operations: Array<{ action, params }> }` **(New v1.2)**: Executes multiple operations atomically.
 - `add_inference_rule`: `{ name, datalog }`
-- `ingest_file`: `{ format, content, entity_id?, entity_name?, entity_type?, chunking?, metadata?, observation_metadata?, deduplicate?, max_observations? }`
+- `ingest_file`: `{ format, file_path?, content?, entity_id?, entity_name?, entity_type?, chunking?, metadata?, observation_metadata?, deduplicate?, max_observations? }`
+  - `format` options: `"markdown"`, `"json"`, `"pdf"` **(New v1.9)**
+  - `file_path`: Optional path to file on disk (alternative to `content` parameter)
+  - `content`: File content as string (required if `file_path` not provided)
   - `chunking` options: `"none"`, `"paragraphs"` (future: `"semantic"`)
 
 Important Details:
@@ -419,7 +557,7 @@ Example (Transitive Manager ⇒ Upper Manager):
 }
 ```
 
-Bulk Ingestion (Markdown/JSON):
+Bulk Ingestion (Markdown/JSON/PDF):
 
 ```json
 {
@@ -433,6 +571,19 @@ Bulk Ingestion (Markdown/JSON):
 }
 ```
 
+PDF Ingestion via File Path:
+
+```json
+{
+  "action": "ingest_file",
+  "entity_name": "Research Paper",
+  "format": "pdf",
+  "file_path": "/path/to/document.pdf",
+  "chunking": "paragraphs",
+  "deduplicate": true
+}
+```
+
 ### query_memory (Read)
 
 Actions:
@@ -660,6 +811,24 @@ Returns deletion statistics showing exactly what was removed.
 
 ## Technical Highlights
 
+### Dual Timestamp Format (v1.9)
+
+All write operations (`create_entity`, `add_observation`, `create_relation`) return timestamps in both formats:
+- `created_at`: Unix microseconds (CozoDB native format, precise for calculations)
+- `created_at_iso`: ISO 8601 string (human-readable, e.g., `"2026-02-28T17:21:19.343Z"`)
+
+This dual format provides maximum flexibility - use Unix timestamps for time calculations and comparisons, or ISO strings for display and logging.
+
+Example response:
+```json
+{
+  "id": "...",
+  "created_at": 1772299279343000,
+  "created_at_iso": "2026-02-28T17:21:19.343Z",
+  "status": "Entity created"
+}
+```
+
 ### Local ONNX Embeddings (Transformers)
 
 Default Model: `Xenova/bge-m3` (1024 dimensions).
@@ -784,6 +953,45 @@ npx ts-node test-user-pref.ts
 - Use `health` action to check cache hit rates
 - Consider RocksDB backend for datasets > 100k entities
 
+## Roadmap
+
+CozoDB Memory is actively developed. Here's what's planned:
+
+### Near-Term (v1.x)
+
+- **GPU Acceleration** - CUDA support for embedding generation (10-50x faster)
+- **Streaming Ingestion** - Real-time data ingestion from logs, APIs, webhooks
+- **Advanced Chunking** - Semantic chunking for `ingest_file` (paragraph-aware splitting)
+- **Query Optimization** - Automatic query plan optimization for complex graph traversals
+- **Additional Export Formats** - Notion, Roam Research, Logseq compatibility
+
+### Mid-Term (v2.x)
+
+- **Multi-Modal Embeddings** - Image and audio embedding support via CLIP/Whisper
+- **Distributed Mode** - Multi-node deployment with CozoDB clustering
+- **Real-Time Sync** - WebSocket-based live updates for collaborative use cases
+- **Advanced Inference** - Causal reasoning, temporal pattern detection
+- **Web UI** - Optional web interface for memory exploration and visualization
+
+### Long-Term
+
+- **Federated Learning** - Privacy-preserving model updates across instances
+- **Custom Embedding Models** - Fine-tune embeddings on domain-specific data
+- **Plugin System** - Extensible architecture for custom tools and integrations
+
+### Community Requests
+
+Have a feature idea? Open an issue with the `enhancement` label or check [Low-Hanging-Fruit.md](Low-Hanging-Fruit.md) for quick wins you can contribute.
+
+## Contributing
+
+Contributions are welcome! Please read [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines on:
+
+- Setting up the development environment
+- Coding standards and best practices
+- Testing and documentation requirements
+- Pull request process
+
 ## License
 
 This project is licensed under the Apache-2.0 License. See the [LICENSE](LICENSE) file for details.

package/dist/api_bridge.js

@@ -35,11 +35,13 @@ app.post("/api/entities", async (req, res) => {
     try {
         // We use the same logic as in create_entity tool
         const id = (0, uuid_1.v4)();
-        const embedding = await memoryServer.embeddingService.embed(name + " " + type);
+        const content = name + " " + type;
+        const embedding = await memoryServer.embeddingService.embed(content);
+        const nameEmbedding = await memoryServer.embeddingService.embed(name);
         await memoryServer.db.run(`
-        ?[id, created_at, name, type, embedding, metadata] <- [
-          [$id, "ASSERT", $name, $type, [${embedding.join(",")}], $metadata]
-        ] :put entity {id, created_at => name, type, embedding, metadata}
+        ?[id, created_at, name, type, embedding, name_embedding, metadata] <- [
+          [$id, "ASSERT", $name, $type, [${embedding.join(",")}], [${nameEmbedding.join(",")}], $metadata]
+        ] :put entity {id, created_at => name, type, embedding, name_embedding, metadata}
       `, { id, name, type, metadata: metadata || {} });
         res.status(201).json({ id, name, type, metadata, status: "Entity created" });
     }

package/dist/cli-commands.js

@@ -0,0 +1,204 @@
+"use strict";
+/**
+ * Shared CLI command logic for both pure CLI and TUI
+ * Calls MemoryServer public methods directly
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.CLICommands = void 0;
+const index_js_1 = require("./index.js");
+class CLICommands {
+    server;
+    initialized = false;
+    constructor() {
+        this.server = new index_js_1.MemoryServer();
+    }
+    async init() {
+        if (!this.initialized) {
+            await this.server.initPromise;
+            this.initialized = true;
+        }
+    }
+    async close() {
+        // CozoDB handles cleanup automatically
+    }
+    // Entity operations - use db directly
+    async createEntity(name, type, metadata) {
+        const { v4: uuidv4 } = await import('uuid');
+        const id = uuidv4();
+        const content = name + " " + type;
+        const embedding = await this.server.embeddingService.embed(content);
+        const nameEmbedding = await this.server.embeddingService.embed(name);
+        const now = Date.now() * 1000; // microseconds
+        const nowIso = new Date().toISOString();
+        await this.server.db.run(`
+        ?[id, created_at, name, type, embedding, name_embedding, metadata] <- [
+          [$id, "ASSERT", $name, $type, [${embedding.join(",")}], [${nameEmbedding.join(",")}], $metadata]
+        ] :put entity {id, created_at => name, type, embedding, name_embedding, metadata}
+      `, { id, name, type, metadata: metadata || {} });
+        return { id, name, type, metadata, created_at: now, created_at_iso: nowIso, status: "Entity created" };
+    }
+    async getEntity(entityId) {
+        const entityRes = await this.server.db.run('?[id, name, type, metadata, ts] := *entity{id, name, type, metadata, created_at, @ "NOW"}, id = $id, ts = to_int(created_at)', { id: entityId });
+        if (entityRes.rows.length === 0) {
+            throw new Error("Entity not found");
+        }
+        const obsRes = await this.server.db.run('?[id, text, metadata, ts] := *observation{id, entity_id, text, metadata, created_at, @ "NOW"}, entity_id = $id, ts = to_int(created_at)', { id: entityId });
+        const relRes = await this.server.db.run(`
+        ?[target_id, type, strength, metadata, direction] := *relationship{from_id, to_id, relation_type: type, strength, metadata, @ "NOW"}, from_id = $id, target_id = to_id, direction = 'outgoing'
+        ?[target_id, type, strength, metadata, direction] := *relationship{from_id, to_id, relation_type: type, strength, metadata, @ "NOW"}, to_id = $id, target_id = from_id, direction = 'incoming'
+      `, { id: entityId });
+        return {
+            entity: {
+                id: entityRes.rows[0][0],
+                name: entityRes.rows[0][1],
+                type: entityRes.rows[0][2],
+                metadata: entityRes.rows[0][3],
+                created_at: entityRes.rows[0][4]
+            },
+            observations: obsRes.rows.map((r) => ({ id: r[0], text: r[1], metadata: r[2], created_at: r[3] })),
+            relations: relRes.rows.map((r) => ({ target_id: r[0], type: r[1], strength: r[2], metadata: r[3], direction: r[4] }))
+        };
+    }
+    async deleteEntity(entityId) {
+        await this.server.db.run(`
+        { ?[id, created_at] := *observation{id, entity_id, created_at}, entity_id = $target_id :rm observation {id, created_at} }
+        { ?[from_id, to_id, relation_type, created_at] := *relationship{from_id, to_id, relation_type, created_at}, from_id = $target_id :rm relationship {from_id, to_id, relation_type, created_at} }
+        { ?[from_id, to_id, relation_type, created_at] := *relationship{from_id, to_id, relation_type, created_at}, to_id = $target_id :rm relationship {from_id, to_id, relation_type, created_at} }
+        { ?[id, created_at] := *entity{id, created_at}, id = $target_id :rm entity {id, created_at} }
+      `, { target_id: entityId });
+        return { status: "Entity and related data deleted" };
+    }
+    // Observation operations
+    async addObservation(entityId, text, metadata) {
+        const { v4: uuidv4 } = await import('uuid');
+        const id = uuidv4();
+        const embedding = await this.server.embeddingService.embed(text);
+        const now = Date.now() * 1000;
+        const nowIso = new Date().toISOString();
+        await this.server.db.run(`
+        ?[id, created_at, entity_id, text, embedding, metadata] <- [
+          [$id, "ASSERT", $entity_id, $text, [${embedding.join(",")}], $metadata]
+        ] :put observation {id, created_at => entity_id, text, embedding, metadata}
+      `, { id, entity_id: entityId, text, metadata: metadata || {} });
+        return { id, entity_id: entityId, text, metadata, created_at: now, created_at_iso: nowIso, status: "Observation added" };
+    }
+    // Relation operations
+    async createRelation(fromId, toId, relationType, strength, metadata) {
+        const str = strength !== undefined ? strength : 1.0;
+        const now = Date.now() * 1000;
+        const nowIso = new Date().toISOString();
+        await this.server.db.run(`
+        ?[from_id, to_id, relation_type, created_at, strength, metadata] <- [
+          [$from_id, $to_id, $relation_type, "ASSERT", $strength, $metadata]
+        ] :put relationship {from_id, to_id, relation_type, created_at => strength, metadata}
+      `, { from_id: fromId, to_id: toId, relation_type: relationType, strength: str, metadata: metadata || {} });
+        return { from_id: fromId, to_id: toId, relation_type: relationType, strength: str, metadata, created_at: now, created_at_iso: nowIso, status: "Relation created" };
+    }
+    // Search operations - use the MCP tool directly
+    async search(query, limit = 10, entityTypes, includeEntities = true, includeObservations = true) {
+        // Call the search method from the server's query_memory tool
+        const result = await this.server.hybridSearch.search({
+            query,
+            limit,
+            entityTypes,
+            includeEntities,
+            includeObservations
+        });
+        // If result is empty or has issues, return it as-is
+        return result;
+    }
+    async advancedSearch(params) {
+        return await this.server.advancedSearch(params);
+    }
+    async context(query, contextWindow, timeRangeHours) {
+        // Use advancedSearch with appropriate parameters
+        return await this.server.advancedSearch({
+            query,
+            limit: contextWindow || 10,
+            timeRangeHours
+        });
+    }
+    // Graph operations
+    async explore(startEntity, endEntity, maxHops, relationTypes) {
+        // Use graph_walking or advancedSearch
+        if (endEntity) {
+            // Path finding
+            return await this.server.computeShortestPath({
+                start_entity: startEntity,
+                end_entity: endEntity
+            });
+        }
+        else {
+            // Graph exploration - use advancedSearch with graph constraints
+            return await this.server.advancedSearch({
+                query: '',
+                graphConstraints: {
+                    maxDepth: maxHops || 3,
+                    requiredRelations: relationTypes,
+                    targetEntityIds: [startEntity]
+                }
+            });
+        }
+    }
+    async pagerank() {
+        return await this.server.recomputePageRank();
+    }
+    async communities() {
+        return await this.server.recomputeCommunities();
+    }
+    // System operations
+    async health() {
+        const entityCount = await this.server.db.run('?[count(id)] := *entity{id, @ "NOW"}');
+        const obsCount = await this.server.db.run('?[count(id)] := *observation{id, @ "NOW"}');
+        const relCount = await this.server.db.run('?[count(from_id)] := *relationship{from_id, @ "NOW"}');
+        return {
+            status: "healthy",
+            entities: entityCount.rows[0][0],
+            observations: obsCount.rows[0][0],
+            relationships: relCount.rows[0][0]
+        };
+    }
+    async metrics() {
+        // Access private metrics via type assertion
+        return this.server.metrics;
+    }
+    async exportMemory(format, options) {
+        const { ExportImportService } = await import('./export-import-service.js');
+        // Create a simple wrapper that implements DbService interface
+        const dbService = {
+            run: async (query, params) => {
+                return await this.server.db.run(query, params);
+            }
+        };
+        const exportService = new ExportImportService(dbService);
+        return await exportService.exportMemory({
+            format,
+            includeMetadata: options?.includeMetadata,
+            includeRelationships: options?.includeRelationships,
+            includeObservations: options?.includeObservations,
+            entityTypes: options?.entityTypes,
+            since: options?.since
+        });
+    }
+    async importMemory(data, sourceFormat, options) {
+        const { ExportImportService } = await import('./export-import-service.js');
+        // Create a simple wrapper that implements DbService interface
+        const dbService = {
+            run: async (query, params) => {
+                return await this.server.db.run(query, params);
+            }
+        };
+        const exportService = new ExportImportService(dbService);
+        return await exportService.importMemory(data, {
+            sourceFormat: sourceFormat,
+            mergeStrategy: options?.mergeStrategy || 'skip',
+            defaultEntityType: options?.defaultEntityType
+        });
+    }
+    async ingestFile(entityId, format, filePath, content, options) {
+        // This would need to be implemented similar to the MCP tool
+        // For now, return a placeholder
+        return { status: "not_implemented", message: "Use MCP server for file ingestion" };
+    }
+}
+exports.CLICommands = CLICommands;