@mmmbuto/nexuscli 0.8.9 → 0.9.0

package/lib/server/docs/DATABASE_SCHEMA.md

@@ -1,783 +0,0 @@
-# NexusCLI Database Schema - SQLite
-
-**Version**: 0.1.0
-**Created**: 2025-11-17
-**Database**: SQLite (better-sqlite3)
-**Philosophy**: Minimal, portable, chat-friendly
-
----
-
-## 🎯 Why SQLite?
-
-### Alignment with Three Pillars
-
-**1. Lightweight** ✅
-- Single dependency: `better-sqlite3` (~500KB native)
-- No server process (embedded)
-- Single file database (< 10MB for 1000s of messages)
-
-**2. Portable (Linux + Termux)** ✅
-- Works on x86_64 and aarch64 (Termux)
-- Pre-compiled binaries for Android
-- Single `.db` file (easy backup)
-
-**3. ChatGPT UI** ✅
-- Perfect for conversation history
-- Fast full-text search (`FTS5`)
-- Efficient pagination
-
-### Alternatives Considered
-
-| Option | Pros | Cons | Verdict |
-|--------|------|------|---------|
-| **SQLite** | Lightweight, portable, SQL | Native dependency | ✅ **CHOSEN** |
-| JSON files | Zero deps, simple | Slow search, no relations | ❌ Not scalable |
-| MongoDB | Powerful | Heavy, not Termux-friendly | ❌ Too heavy |
-| LevelDB | Fast K-V store | No SQL, harder queries | ❌ Not chat-friendly |
-
----
-
-## 📦 Installation
-
-### Dependency
-
-```json
-// backend/package.json
-{
-  "dependencies": {
-    "express": "^4.18.2",
-    "cors": "^2.8.5",
-    "uuid": "^9.0.0",
-    "better-sqlite3": "^9.2.2"  // ← Only new dependency
-  }
-}
-```
-
-**Total dependencies**: 4 packages (still < 10 ✅)
-
-### Setup (Linux)
-
-```bash
-cd backend
-npm install better-sqlite3
-# Native compilation (uses node-gyp)
-```
-
-### Setup (Termux)
-
-```bash
-# Termux requires build tools
-pkg install clang make python
-
-# Install better-sqlite3 (compiles for aarch64)
-npm install better-sqlite3
-```
-
-**Note**: Pre-built binaries usually available, compilation is fallback.
-
----
-
-## 🗄️ Schema Design
-
-### Database File Location
-
-**Linux**:
-```
-/var/lib/nexuscli/nexuscli.db
-```
-
-**Termux**:
-```
-/data/data/com.termux/files/home/.nexuscli/nexuscli.db
-```
-
-**Code**:
-```javascript
-// config.js
-const isTermux = process.env.PREFIX?.includes('com.termux');
-
-const DB_PATH = isTermux
-  ? `${process.env.HOME}/.nexuscli/nexuscli.db`
-  : '/var/lib/nexuscli/nexuscli.db';
-```
-
----
-
-## 📊 Tables
-
-### 1. `conversations`
-
-Tracks chat sessions (like ChatGPT conversations).
-
-```sql
-CREATE TABLE conversations (
-  id TEXT PRIMARY KEY,              -- UUID
-  title TEXT NOT NULL,              -- "Deploy to Production"
-  created_at INTEGER NOT NULL,      -- Unix timestamp (ms)
-  updated_at INTEGER NOT NULL,      -- Unix timestamp (ms)
-  metadata TEXT,                    -- JSON: {"tags": ["prod"], "archived": false}
-
-  INDEX idx_conversations_updated_at ON conversations(updated_at DESC)
-);
-```
-
-**Example Row**:
-```json
-{
-  "id": "conv-abc123",
-  "title": "Deploy NexusCLI to Production",
-  "created_at": 1700000000000,
-  "updated_at": 1700001000000,
-  "metadata": "{\"tags\":[\"prod\",\"deploy\"],\"archived\":false}"
-}
-```
-
----
-
-### 2. `messages`
-
-Individual messages in conversations (user prompts + assistant responses).
-
-```sql
-CREATE TABLE messages (
-  id TEXT PRIMARY KEY,              -- UUID
-  conversation_id TEXT NOT NULL,    -- FK to conversations.id
-  role TEXT NOT NULL,               -- 'user' | 'assistant' | 'system'
-  content TEXT NOT NULL,            -- Markdown text
-  created_at INTEGER NOT NULL,      -- Unix timestamp (ms)
-  metadata TEXT,                    -- JSON: {"streaming": true, "tokens": 123}
-
-  FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE CASCADE,
-  INDEX idx_messages_conversation_id ON messages(conversation_id),
-  INDEX idx_messages_created_at ON messages(created_at ASC)
-);
-```
-
-**Example Rows**:
-```json
-// User message
-{
-  "id": "msg-001",
-  "conversation_id": "conv-abc123",
-  "role": "user",
-  "content": "Deploy to all production nodes",
-  "created_at": 1700000000000,
-  "metadata": null
-}
-
-// Assistant message
-{
-  "id": "msg-002",
-  "conversation_id": "conv-abc123",
-  "role": "assistant",
-  "content": "Deploying to 3 nodes:\n- prod-001 ✅\n- prod-002 ✅\n- prod-003 ⚠️",
-  "created_at": 1700000001000,
-  "metadata": "{\"tokens\":45,\"duration\":1234}"
-}
-```
-
----
-
-### 3. `jobs`
-
-Job execution records (linked to messages).
-
-```sql
-CREATE TABLE jobs (
-  id TEXT PRIMARY KEY,              -- Job ID (UUID)
-  conversation_id TEXT,             -- FK to conversations.id (optional)
-  message_id TEXT,                  -- FK to messages.id (optional)
-  node_id TEXT NOT NULL,            -- Target node
-  tool TEXT NOT NULL,               -- 'bash' | 'git' | 'docker'
-  command TEXT NOT NULL,            -- Command executed
-  status TEXT NOT NULL,             -- 'queued' | 'executing' | 'completed' | 'failed'
-  exit_code INTEGER,                -- Exit code (NULL if not completed)
-  stdout TEXT,                      -- Command stdout
-  stderr TEXT,                      -- Command stderr
-  duration INTEGER,                 -- Execution time (ms)
-  created_at INTEGER NOT NULL,      -- Unix timestamp (ms)
-  started_at INTEGER,               -- Unix timestamp (ms)
-  completed_at INTEGER,             -- Unix timestamp (ms)
-
-  FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE SET NULL,
-  FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE SET NULL,
-  INDEX idx_jobs_conversation_id ON jobs(conversation_id),
-  INDEX idx_jobs_status ON jobs(status),
-  INDEX idx_jobs_created_at ON jobs(created_at DESC)
-);
-```
-
-**Example Row**:
-```json
-{
-  "id": "job-xyz",
-  "conversation_id": "conv-abc123",
-  "message_id": "msg-002",
-  "node_id": "prod-001",
-  "tool": "bash",
-  "command": "systemctl restart nginx",
-  "status": "completed",
-  "exit_code": 0,
-  "stdout": "nginx.service restarted",
-  "stderr": "",
-  "duration": 1234,
-  "created_at": 1700000000000,
-  "started_at": 1700000000100,
-  "completed_at": 1700000001334
-}
-```
-
----
-
-### 4. `nodes` (Optional - for multi-node setups)
-
-Registered nodes.
-
-```sql
-CREATE TABLE nodes (
-  id TEXT PRIMARY KEY,              -- Node ID
-  hostname TEXT NOT NULL,           -- Node hostname
-  ip_address TEXT,                  -- Node IP
-  status TEXT NOT NULL,             -- 'online' | 'offline'
-  capabilities TEXT,                -- JSON: {"tools": ["bash", "git"]}
-  last_heartbeat INTEGER,           -- Unix timestamp (ms)
-  created_at INTEGER NOT NULL,      -- Unix timestamp (ms)
-
-  INDEX idx_nodes_status ON nodes(status)
-);
-```
-
----
-
-## 🔍 Full-Text Search (Optional Enhancement)
-
-For searching chat history.
-
-```sql
--- Virtual FTS5 table for messages
-CREATE VIRTUAL TABLE messages_fts USING fts5(
-  content,
-  conversation_id UNINDEXED,
-  content='messages',
-  content_rowid='rowid'
-);
-
--- Triggers to keep FTS index synced
-CREATE TRIGGER messages_fts_insert AFTER INSERT ON messages BEGIN
-  INSERT INTO messages_fts(rowid, content, conversation_id)
-  VALUES (new.rowid, new.content, new.conversation_id);
-END;
-
-CREATE TRIGGER messages_fts_delete AFTER DELETE ON messages BEGIN
-  DELETE FROM messages_fts WHERE rowid = old.rowid;
-END;
-
-CREATE TRIGGER messages_fts_update AFTER UPDATE ON messages BEGIN
-  UPDATE messages_fts SET content = new.content WHERE rowid = new.rowid;
-END;
-```
-
-**Usage**:
-```sql
--- Search messages for "nginx"
-SELECT m.* FROM messages m
-JOIN messages_fts fts ON fts.rowid = m.rowid
-WHERE messages_fts MATCH 'nginx'
-ORDER BY m.created_at DESC
-LIMIT 20;
-```
-
----
-
-## 💾 Database Wrapper
-
-### Initialization
-
-```javascript
-// backend/db.js
-const Database = require('better-sqlite3');
-const path = require('path');
-const fs = require('fs');
-
-const isTermux = process.env.PREFIX?.includes('com.termux');
-const dbDir = isTermux
-  ? path.join(process.env.HOME, '.nexuscli')
-  : '/var/lib/nexuscli';
-
-const dbPath = path.join(dbDir, 'nexuscli.db');
-
-// Ensure directory exists
-if (!fs.existsSync(dbDir)) {
-  fs.mkdirSync(dbDir, { recursive: true });
-}
-
-// Open database (creates if doesn't exist)
-const db = new Database(dbPath);
-
-// Enable WAL mode for better concurrency
-db.pragma('journal_mode = WAL');
-
-// Enable foreign keys
-db.pragma('foreign_keys = ON');
-
-// Initialize schema
-function initSchema() {
-  db.exec(`
-    CREATE TABLE IF NOT EXISTS conversations (
-      id TEXT PRIMARY KEY,
-      title TEXT NOT NULL,
-      created_at INTEGER NOT NULL,
-      updated_at INTEGER NOT NULL,
-      metadata TEXT
-    );
-
-    CREATE INDEX IF NOT EXISTS idx_conversations_updated_at
-    ON conversations(updated_at DESC);
-
-    CREATE TABLE IF NOT EXISTS messages (
-      id TEXT PRIMARY KEY,
-      conversation_id TEXT NOT NULL,
-      role TEXT NOT NULL,
-      content TEXT NOT NULL,
-      created_at INTEGER NOT NULL,
-      metadata TEXT,
-      FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE CASCADE
-    );
-
-    CREATE INDEX IF NOT EXISTS idx_messages_conversation_id
-    ON messages(conversation_id);
-
-    CREATE INDEX IF NOT EXISTS idx_messages_created_at
-    ON messages(created_at ASC);
-
-    CREATE TABLE IF NOT EXISTS jobs (
-      id TEXT PRIMARY KEY,
-      conversation_id TEXT,
-      message_id TEXT,
-      node_id TEXT NOT NULL,
-      tool TEXT NOT NULL,
-      command TEXT NOT NULL,
-      status TEXT NOT NULL,
-      exit_code INTEGER,
-      stdout TEXT,
-      stderr TEXT,
-      duration INTEGER,
-      created_at INTEGER NOT NULL,
-      started_at INTEGER,
-      completed_at INTEGER,
-      FOREIGN KEY (conversation_id) REFERENCES conversations(id) ON DELETE SET NULL,
-      FOREIGN KEY (message_id) REFERENCES messages(id) ON DELETE SET NULL
-    );
-
-    CREATE INDEX IF NOT EXISTS idx_jobs_conversation_id
-    ON jobs(conversation_id);
-
-    CREATE INDEX IF NOT EXISTS idx_jobs_status
-    ON jobs(status);
-
-    CREATE INDEX IF NOT EXISTS idx_jobs_created_at
-    ON jobs(created_at DESC);
-  `);
-
-  console.log(`✅ Database initialized: ${dbPath}`);
-}
-
-initSchema();
-
-module.exports = db;
-```
-
----
-
-### CRUD Operations
-
-```javascript
-// backend/models/conversation.js
-const db = require('../db');
-const { v4: uuidv4 } = require('uuid');
-
-class Conversation {
-  // Create new conversation
-  static create(title) {
-    const id = uuidv4();
-    const now = Date.now();
-
-    const stmt = db.prepare(`
-      INSERT INTO conversations (id, title, created_at, updated_at)
-      VALUES (?, ?, ?, ?)
-    `);
-
-    stmt.run(id, title, now, now);
-
-    return { id, title, created_at: now, updated_at: now };
-  }
-
-  // Get conversation by ID
-  static getById(id) {
-    const stmt = db.prepare('SELECT * FROM conversations WHERE id = ?');
-    return stmt.get(id);
-  }
-
-  // List recent conversations
-  static listRecent(limit = 20) {
-    const stmt = db.prepare(`
-      SELECT * FROM conversations
-      ORDER BY updated_at DESC
-      LIMIT ?
-    `);
-    return stmt.all(limit);
-  }
-
-  // Update conversation title
-  static updateTitle(id, title) {
-    const stmt = db.prepare(`
-      UPDATE conversations
-      SET title = ?, updated_at = ?
-      WHERE id = ?
-    `);
-    stmt.run(title, Date.now(), id);
-  }
-
-  // Delete conversation (cascade deletes messages)
-  static delete(id) {
-    const stmt = db.prepare('DELETE FROM conversations WHERE id = ?');
-    stmt.run(id);
-  }
-}
-
-module.exports = Conversation;
-```
-
-```javascript
-// backend/models/message.js
-const db = require('../db');
-const { v4: uuidv4 } = require('uuid');
-
-class Message {
-  // Add message to conversation
-  static create(conversationId, role, content, metadata = null) {
-    const id = uuidv4();
-    const now = Date.now();
-
-    const stmt = db.prepare(`
-      INSERT INTO messages (id, conversation_id, role, content, created_at, metadata)
-      VALUES (?, ?, ?, ?, ?, ?)
-    `);
-
-    stmt.run(id, conversationId, role, content, now,
-             metadata ? JSON.stringify(metadata) : null);
-
-    // Update conversation updated_at
-    const updateConv = db.prepare(`
-      UPDATE conversations SET updated_at = ? WHERE id = ?
-    `);
-    updateConv.run(now, conversationId);
-
-    return { id, conversation_id: conversationId, role, content, created_at: now };
-  }
-
-  // Get messages for conversation
-  static getByConversation(conversationId, limit = 100) {
-    const stmt = db.prepare(`
-      SELECT * FROM messages
-      WHERE conversation_id = ?
-      ORDER BY created_at ASC
-      LIMIT ?
-    `);
-    return stmt.all(conversationId, limit);
-  }
-
-  // Search messages (if FTS enabled)
-  static search(query, limit = 20) {
-    const stmt = db.prepare(`
-      SELECT m.* FROM messages m
-      JOIN messages_fts fts ON fts.rowid = m.rowid
-      WHERE messages_fts MATCH ?
-      ORDER BY m.created_at DESC
-      LIMIT ?
-    `);
-    return stmt.all(query, limit);
-  }
-}
-
-module.exports = Message;
-```
-
----
-
-## 🔄 Migration Strategy
-
-### Initial Setup (v0.1.0)
-
-```javascript
-// backend/migrations/001-initial.js
-module.exports = {
-  up(db) {
-    db.exec(`
-      CREATE TABLE conversations (...);
-      CREATE TABLE messages (...);
-      CREATE TABLE jobs (...);
-    `);
-  },
-
-  down(db) {
-    db.exec(`
-      DROP TABLE IF EXISTS jobs;
-      DROP TABLE IF EXISTS messages;
-      DROP TABLE IF EXISTS conversations;
-    `);
-  }
-};
-```
-
-### Migration Runner
-
-```javascript
-// backend/migrate.js
-const db = require('./db');
-const fs = require('fs');
-const path = require('path');
-
-function getCurrentVersion() {
-  try {
-    const row = db.prepare('SELECT value FROM schema_version WHERE key = "version"').get();
-    return parseInt(row.value);
-  } catch {
-    // First run
-    db.exec('CREATE TABLE IF NOT EXISTS schema_version (key TEXT PRIMARY KEY, value TEXT)');
-    db.prepare('INSERT INTO schema_version (key, value) VALUES ("version", "0")').run();
-    return 0;
-  }
-}
-
-function runMigrations() {
-  const currentVersion = getCurrentVersion();
-  const migrationsDir = path.join(__dirname, 'migrations');
-
-  const migrations = fs.readdirSync(migrationsDir)
-    .filter(f => f.endsWith('.js'))
-    .sort();
-
-  for (const file of migrations) {
-    const migration = require(path.join(migrationsDir, file));
-    const version = parseInt(file.split('-')[0]);
-
-    if (version > currentVersion) {
-      console.log(`Running migration ${file}...`);
-      migration.up(db);
-      db.prepare('UPDATE schema_version SET value = ? WHERE key = "version"').run(version);
-    }
-  }
-
-  console.log('✅ Migrations complete');
-}
-
-runMigrations();
-```
-
----
-
-## 📈 Performance Optimization
-
-### WAL Mode (Write-Ahead Logging)
-
-```javascript
-// Already enabled in db.js
-db.pragma('journal_mode = WAL');
-```
-
-**Benefits**:
-- Concurrent reads while writing
-- Faster writes
-- Better crash recovery
-
-### Prepared Statements
-
-```javascript
-// ✅ GOOD: Reuse prepared statement
-const stmt = db.prepare('SELECT * FROM messages WHERE conversation_id = ?');
-const messages = stmt.all(conversationId);
-
-// ❌ BAD: Parse SQL every time
-db.prepare('SELECT * FROM messages WHERE conversation_id = ?').all(conversationId);
-```
-
-### Indexes
-
-All critical queries have indexes:
-- `conversations.updated_at` (recent list)
-- `messages.conversation_id` (get messages)
-- `messages.created_at` (chronological order)
-- `jobs.status` (filter by status)
-
----
-
-## 💾 Backup & Recovery
-
-### Backup Script
-
-```bash
-#!/bin/bash
-# backup-db.sh
-
-DB_PATH="/var/lib/nexuscli/nexuscli.db"
-BACKUP_DIR="/var/backups/nexuscli"
-TIMESTAMP=$(date +%Y%m%d_%H%M%S)
-
-mkdir -p "$BACKUP_DIR"
-
-# SQLite backup (safe even with active connections)
-sqlite3 "$DB_PATH" ".backup $BACKUP_DIR/nexuscli-$TIMESTAMP.db"
-
-# Keep last 30 days
-find "$BACKUP_DIR" -name "nexuscli-*.db" -mtime +30 -delete
-
-echo "✅ Backup complete: $BACKUP_DIR/nexuscli-$TIMESTAMP.db"
-```
-
-### Restore
-
-```bash
-# Stop NexusCLI
-systemctl stop nexuscli
-
-# Restore backup
-cp /var/backups/nexuscli/nexuscli-20251117.db /var/lib/nexuscli/nexuscli.db
-
-# Start NexusCLI
-systemctl start nexuscli
-```
-
----
-
-## 🎯 API Integration
-
-### REST Endpoints
-
-```javascript
-// backend/routes/conversations.js
-const express = require('express');
-const Conversation = require('../models/conversation');
-const Message = require('../models/message');
-
-const router = express.Router();
-
-// GET /api/v1/conversations - List recent
-router.get('/', (req, res) => {
-  const conversations = Conversation.listRecent(20);
-  res.json({ conversations });
-});
-
-// POST /api/v1/conversations - Create new
-router.post('/', (req, res) => {
-  const { title } = req.body;
-  const conversation = Conversation.create(title || 'New Conversation');
-  res.status(201).json(conversation);
-});
-
-// GET /api/v1/conversations/:id - Get conversation with messages
-router.get('/:id', (req, res) => {
-  const conversation = Conversation.getById(req.params.id);
-  if (!conversation) {
-    return res.status(404).json({ error: 'Conversation not found' });
-  }
-
-  const messages = Message.getByConversation(req.params.id);
-  res.json({ ...conversation, messages });
-});
-
-// POST /api/v1/conversations/:id/messages - Add message
-router.post('/:id/messages', (req, res) => {
-  const { role, content } = req.body;
-  const message = Message.create(req.params.id, role, content);
-  res.status(201).json(message);
-});
-
-// DELETE /api/v1/conversations/:id - Delete conversation
-router.delete('/:id', (req, res) => {
-  Conversation.delete(req.params.id);
-  res.json({ success: true });
-});
-
-module.exports = router;
-```
-
----
-
-## 🧪 Testing
-
-### Unit Tests (SQLite in-memory)
-
-```javascript
-// backend/tests/conversation.test.js
-const Database = require('better-sqlite3');
-const Conversation = require('../models/conversation');
-
-// Use in-memory DB for tests
-const testDb = new Database(':memory:');
-
-beforeEach(() => {
-  // Initialize schema
-  testDb.exec(`CREATE TABLE conversations (...)`);
-});
-
-test('create conversation', () => {
-  const conv = Conversation.create('Test Conversation');
-  expect(conv.title).toBe('Test Conversation');
-  expect(conv.id).toBeTruthy();
-});
-
-test('list recent conversations', () => {
-  Conversation.create('Conv 1');
-  Conversation.create('Conv 2');
-
-  const list = Conversation.listRecent();
-  expect(list.length).toBe(2);
-  expect(list[0].title).toBe('Conv 2'); // Most recent first
-});
-```
-
----
-
-## 📊 Size Estimates
-
-| Data | Rows | Size per Row | Total Size |
-|------|------|--------------|------------|
-| Conversations | 100 | ~200 bytes | ~20 KB |
-| Messages | 10,000 | ~500 bytes | ~5 MB |
-| Jobs | 5,000 | ~1 KB | ~5 MB |
-| **Total** | - | - | **~10 MB** |
-
-**Conclusion**: Even with heavy usage, DB stays < 50MB (lightweight ✅)
-
----
-
-## 🎯 Alignment with Three Pillars
-
-### 1. Lightweight ✅
-- Single dependency: `better-sqlite3`
-- Embedded DB (no server)
-- WAL mode (< 10MB typical size)
-
-### 2. Portable ✅
-- Works on Linux x86_64
-- Works on Termux aarch64
-- Single `.db` file (easy sync)
-
-### 3. ChatGPT UI ✅
-- Perfect for conversation history
-- Fast message retrieval
-- Full-text search support
-
----
-
-## 📚 Related Documents
-
-- [Design Principles](./DESIGN_PRINCIPLES.md) - Three pillars
-- [Architecture Overview](./ARCHITECTURE.md) - System design
-- [API Contract](./API_WRAPPER_CONTRACT.md) - REST API spec
-
----
-
-_Database design for NexusCLI - Lightweight, Portable, Chat-Friendly_
-_Generated by Claude Code (Sonnet 4.5) - 2025-11-17_