@joystick.js/db-canary 0.0.0-canary.2213 → 0.0.0-canary.2217

package/prompts/22-build-script-integration.md

@@ -1,129 +0,0 @@
-# Task 22: Build Script Integration for CLI and Node Packages
-
-## Objective
-Integrate JoystickDB components into the CLI and Node packages by modifying their build scripts to copy the necessary database code before building, and updating their package.json files with required dependencies.
-
-## Current State
-- JoystickDB is fully implemented in `/Users/rglover/projects/cheatcode/joystick/db`
-- CLI package needs server components for development database functionality
-- Node package needs client components for database connectivity
-- Both packages use esbuild with `bundle: false` configuration
-- Current build scripts do not include database code integration
-
-## Requirements
-
-### 1. CLI Package Integration
-**Target**: `/Users/rglover/projects/cheatcode/joystick/cli`
-
-**Build Script Updates** (`cli/.build/index.js`):
-- Copy `db/dist/server/` → `cli/src/lib/joystickdb/` before building CLI files
-- Ensure copying happens before the existing build process
-- Use proper error handling for copy operations
-- Create target directories if they don't exist
-
-**Package.json Updates** (`cli/package.json`):
-- Add JoystickDB server dependencies to `dependencies`:
-  - `"lmdb": "^3.4.2"`
-  - `"msgpackr": "^1.11.5"`
-  - `"bcrypt": "^6.0.0"`
-  - `"winston": "^3.17.0"`
-  - `"winston-daily-rotate-file": "^5.0.0"`
-  - `"@aws-sdk/client-s3": "^3.879.0"`
-
-### 2. Node Package Integration
-**Target**: `/Users/rglover/projects/cheatcode/joystick/node`
-
-**Build Script Updates** (`node/.build/index.js`):
-- Copy `db/dist/client/` → `node/src/lib/joystickdb/` before building Node files
-- Ensure copying happens before the existing build process
-- Use proper error handling for copy operations
-- Create target directories if they don't exist
-
-**Package.json Updates** (`node/package.json`):
-- Add JoystickDB client dependencies to `dependencies`:
-  - `"msgpackr": "^1.11.5"`
-
-## Implementation Details
-
-### Copy Operation Requirements
-- Use Node.js `fs` module with recursive directory copying
-- Preserve file structure and permissions
-- Handle existing directories (clean and recreate)
-- Provide clear error messages if source directories don't exist
-- Log copy operations for debugging
-
-### Build Process Flow
-**CLI Build Process**:
-1. Clean existing `cli/src/lib/joystickdb/` directory
-2. Copy `db/dist/server/` → `cli/src/lib/joystickdb/`
-3. Proceed with existing CLI build process (getFilesToBuild, buildFiles, copyFiles)
-
-**Node Build Process**:
-1. Clean existing `node/src/lib/joystickdb/` directory
-2. Copy `db/dist/client/` → `node/src/lib/joystickdb/`
-3. Proceed with existing Node build process (getFilesToBuild, buildFiles, copyFiles)
-
-### Error Handling
-- Graceful handling of missing source directories
-- Clear error messages indicating which copy operation failed
-- Build process should fail if database code copying fails
-- Provide guidance on running `npm run build` in db package first
-
-### Import Path Consistency
-After integration, code can import database functionality using:
-- **CLI**: `import dbServer from './lib/joystickdb/index.js'`
-- **Node**: `import dbClient from './lib/joystickdb/index.js'`
-
-## Code Standards
-- Follow existing build script patterns in both packages
-- Use ESM syntax (`import`/`export`)
-- Follow snake_case for variables and functions
-- Two-space indentation
-- Minimal comments (only when necessary for clarity)
-- Maintain compatibility with existing build processes
-
-## Dependencies Management
-- Only add dependencies that are actually used by the copied code
-- CLI gets full server dependencies (including binary modules)
-- Node gets lightweight client dependencies only
-- Maintain version consistency with db package
-- Update package-lock.json files after dependency changes
-
-## Testing Requirements
-- Verify build scripts run without errors after modifications
-- Confirm copied files are present in expected locations
-- Test that built packages include database functionality
-- Ensure existing CLI and Node functionality remains intact
-- Verify dependency installation works correctly
-
-## Branch Management
-- Create feature branch: `feature/build-script-integration`
-- All work must be done in this branch
-- No direct commits to master or canary branch
-- Merge only when all build processes work correctly
-
-## Success Criteria
-- ✅ CLI build script copies database server code before building
-- ✅ Node build script copies database client code before building
-- ✅ CLI package.json includes all necessary database server dependencies
-- ✅ Node package.json includes necessary database client dependencies
-- ✅ Build processes complete without errors
-- ✅ Copied database code is available at expected import paths
-- ✅ Existing CLI and Node functionality remains unaffected
-- ✅ Package-lock.json files are properly updated
-
-## Implementation Notes
-- The db package must be built first (`npm run build` in db folder) to generate the dist files
-- Copy operations should use the built/dist versions, not the source files
-- This approach maintains the db folder as the "source of truth" while providing embedded functionality
-- CLI will be heavier due to binary dependencies, but this is acceptable for a development tool
-- Node package remains lightweight with only client functionality
-
-## Future Considerations
-- This integration enables CLI to start database servers in development
-- Node package can connect to databases using embedded client code
-- Standalone db package remains available for production database server deployments
-- Version synchronization across packages will be handled by existing release processes
-
-## Context for Implementation
-This task implements the build-time copying strategy discussed for integrating JoystickDB components into the CLI and Node packages. The approach maintains clean separation while providing embedded functionality without runtime path complexity.

package/prompts/23-cli-integration.md

@@ -1,268 +0,0 @@
-# Task 23: CLI Integration for Development Database
-
-## Objective
-Integrate JoystickDB server into the existing CLI package development server infrastructure, enabling JoystickDB to be used alongside MongoDB, PostgreSQL, and Redis with identical developer experience.
-
-## Current State
-- JoystickDB server code successfully copied to `cli/src/lib/development_server/databases/joystickdb/` during build process
-- CLI package includes all required server dependencies (lmdb, msgpackr, bcrypt, winston, etc.)
-- Existing database infrastructure supports MongoDB, PostgreSQL, and Redis via binary execution
-- Database provider map, installer, and connection patterns established
-- JoystickDB runs as standalone Node.js server via `create_server()` function
-
-## Requirements
-
-### 1. JoystickDB Provider Integration
-**Target**: `cli/src/lib/development_server/databases/`
-
-**Create JoystickDB Module** (`joystickdb/index.js`):
-- Import JoystickDB server from `./joystickdb/index.js` (copied database code in same directory)
-- Implement `start_joystickdb(port)` function using child_process.spawn()
-- Start JoystickDB as Node.js child process, not binary execution
-- Use `node ./lib/development_server/databases/joystickdb/index.js` pattern (file runs directly)
-- Handle process lifecycle: startup, shutdown, error handling
-- Return process ID for consistency with other databases
-
-**Create Connection Module** (`joystickdb/connect.js`):
-- Follow exact pattern of `mongodb/connect.js`, `postgresql/connect.js`, `redis/connect.js`
-- Support both external connections (when `settings.connection` provided) and local development server
-- Return `{ pid, connection }` object matching other database providers
-- Default connection: `{ hostname: "127.0.0.1", port: <port>, database: "app" }`
-
-**Create Connection Check** (`joystickdb/check_connection.js`):
-- Implement TCP connection validation to JoystickDB server
-- Follow pattern of other database connection checkers
-- Test basic connectivity and authentication if configured
-
-### 2. Provider Map Registration
-**Target**: `cli/src/lib/development_server/databases/provider_map.js`
-
-**Add JoystickDB Entry**:
-```javascript
-import connect_joystickdb from './joystickdb/connect.js';
-
-const provider_map = {
-  // ... existing providers
-  joystickdb: {
-    name: 'JoystickDB',
-    connect: connect_joystickdb,
-  },
-};
-```
-
-### 3. Installer Integration
-**Target**: `cli/src/lib/development_server/databases/installer.js`
-
-**Skip Binary Installation for JoystickDB**:
-- Modify `install_database()` function to recognize `joystickdb` as special case
-- JoystickDB doesn't require binary download/installation (code already embedded)
-- Return early for `joystickdb` database type without attempting download
-- Add `joystickdb` to supported database list but bypass installation logic
-
-### 4. Child Process Management
-**Implementation Details**:
-
-**Process Spawning**:
-- Use `child_process.spawn('node', ['./lib/development_server/databases/joystickdb/index.js'])` (file runs directly)
-- Set `cwd` to CLI package root directory for proper import resolution
-- Configure stdio: `{ stdio: ['pipe', 'pipe', 'pipe'] }` for output capture
-- Handle process startup confirmation via stdout monitoring
-- Set required environment variables (JOYSTICK_DB_SETTINGS) for proper startup
-
-**Process Lifecycle**:
-- Monitor stdout for "JoystickDB server started" or similar confirmation message
-- Implement graceful shutdown via process.kill() or SIGTERM
-- Handle process crashes and restart logic if needed
-- Track process ID for port management consistency
-
-**Error Handling**:
-- Catch spawn errors (Node.js not found, import failures, etc.)
-- Handle JoystickDB server startup errors (port conflicts, permission issues)
-- Provide clear error messages matching other database error patterns
-- Fallback gracefully if JoystickDB server fails to start
-
-### 5. Port Management Integration
-**Requirements**:
-- Use existing `kill_port_process()` and `get_process_id_from_port()` utilities
-- Default JoystickDB port: `1983` (matching standalone server default)
-- Support custom port configuration via settings
-- Ensure port cleanup on development server shutdown
-
-### 6. Data Directory Management
-**Setup Data Directory**:
-- Create `.joystick/data/joystickdb_<port>/` directory structure
-- Pass data directory path to JoystickDB server via environment variables or settings
-- Follow pattern of MongoDB data directory management
-- Support multiple JoystickDB instances on different ports
-
-### 7. Settings Integration
-**Configuration Support**:
-- Accept JoystickDB settings in same format as other databases
-- Support external connection strings for remote JoystickDB servers
-- Handle authentication settings (password, API keys)
-- Pass settings to JoystickDB server process via environment or config file
-
-### 8. Automated Setup for Development
-**Transparent Setup Process**:
-- Automatically read API_KEY from `./lib/development_server/databases/joystickdb/API_KEY` file
-- Create default admin user automatically on localhost without developer intervention
-- Use HTTP API to create admin user: `POST /api/users` with API key authentication
-- Default admin credentials: `username: 'admin'`, `password: 'password'`
-- Provide clear logging of setup process for debugging
-- Ensure setup happens after server startup confirmation but before returning connection
-
-## Implementation Patterns
-
-### Child Process Pattern
-```javascript
-const start_joystickdb_process = (joystickdb_port = 1983) => {
-  return new Promise((resolve, reject) => {
-    const database_process = child_process.spawn('node', [
-      './lib/development_server/databases/joystickdb/index.js'
-    ], {
-      cwd: process.cwd(),
-      stdio: ['pipe', 'pipe', 'pipe'],
-      env: {
-        ...process.env,
-        JOYSTICK_DB_SETTINGS: JSON.stringify({
-          port: joystickdb_port,
-          data_path: `./.joystick/data/joystickdb_${joystickdb_port}`,
-          authentication: {}
-        })
-      }
-    });
-
-    database_process.on('error', reject);
-    
-    database_process.stdout.on('data', (data) => {
-      if (data.toString().includes('Starting JoystickDB server')) {
-        resolve(database_process.pid);
-      }
-    });
-  });
-};
-```
-
-### Automated Setup Pattern
-```javascript
-const setup_joystickdb_admin = async (port = 1983) => {
-  try {
-    // Read API key from copied database files
-    const api_key_path = path.resolve('./lib/development_server/databases/joystickdb/API_KEY');
-    const api_key = fs.readFileSync(api_key_path, 'utf8').trim();
-    
-    // Create admin user via HTTP API
-    const response = await fetch(`http://localhost:${port + 1}/api/users`, {
-      method: 'POST',
-      headers: {
-        'Content-Type': 'application/json',
-        'x-joystick-db-api-key': api_key
-      },
-      body: JSON.stringify({
-        username: 'admin',
-        password: 'password',
-        role: 'read_write'
-      })
-    });
-    
-    const result = await response.json();
-    if (result.success) {
-      console.log('✅ JoystickDB admin user created automatically');
-      return true;
-    } else {
-      // User might already exist, which is fine
-      if (result.error && result.error.includes('already exists')) {
-        console.log('ℹ️ JoystickDB admin user already exists');
-        return true;
-      }
-      throw new Error(result.error || 'Failed to create admin user');
-    }
-  } catch (error) {
-    console.warn('⚠️ JoystickDB admin setup failed:', error.message);
-    return false;
-  }
-};
-```
-
-### Connection Pattern
-```javascript
-const connect = async (settings = {}, port = 1983) => {
-  const has_connection = settings.connection && Object.keys(settings.connection).length > 0;
-
-  if (has_connection) {
-    await check_connection(settings.connection, settings.options);
-  } else {
-    // Start local development server
-    const pid = await start_joystickdb(port);
-    
-    // Wait a moment for HTTP server to be ready
-    await new Promise(resolve => setTimeout(resolve, 2000));
-    
-    // Automatically setup admin user for development
-    await setup_joystickdb_admin(port);
-  }
-
-  return {
-    pid: !has_connection ? await start_joystickdb(port) : null,
-    connection: has_connection ? settings.connection : {
-      hostname: "127.0.0.1",
-      port,
-      database: "app",
-      username: "admin",
-      password: "password"
-    },
-  };
-};
-```
-
-## Code Standards
-- Follow existing CLI database integration patterns exactly
-- Use ESM syntax (`import`/`export`)
-- Follow snake_case for variables and functions
-- Two-space indentation
-- Minimal comments (only when necessary for clarity)
-- Error handling consistent with other database providers
-
-## Testing Requirements
-- Verify JoystickDB starts as child process successfully
-- Test port management and cleanup
-- Confirm data directory creation and usage
-- Test connection validation and error handling
-- Ensure graceful shutdown of JoystickDB processes
-- Verify integration with existing development server workflow
-
-## Success Criteria
-- ✅ JoystickDB appears in provider map alongside MongoDB, PostgreSQL, Redis
-- ✅ JoystickDB starts as Node.js child process (not binary)
-- ✅ Connection management follows exact pattern of other databases
-- ✅ Port management and cleanup works correctly
-- ✅ Data directory setup and management implemented
-- ✅ Error handling and process lifecycle management robust
-- ✅ No binary installation required (installer skips JoystickDB)
-- ✅ Developer experience identical to other databases
-- ✅ Existing CLI functionality remains unaffected
-- ✅ Automated admin user setup works transparently on localhost
-- ✅ API key reading and HTTP user creation handled automatically
-- ✅ Default admin credentials provided in connection object
-
-## Integration Notes
-- JoystickDB is unique in being a Node.js process rather than external binary
-- Leverage embedded database code from build script integration at `src/lib/development_server/databases/joystickdb/`
-- Maintain consistency with existing database provider patterns
-- This enables seamless database switching in development environments
-- JoystickDB becomes first-class citizen alongside traditional databases
-
-## Future Considerations
-- This integration enables JoystickDB development database functionality
-- Developers can use JoystickDB in development with same ease as MongoDB/PostgreSQL
-- Production deployments still use standalone JoystickDB server package
-- CLI integration provides embedded development database experience
-- Foundation for potential JoystickDB-specific development tools and utilities
-
-## Context for Implementation
-This task completes the JoystickDB integration trilogy:
-1. ✅ Build script integration (database code copying)
-2. ✅ Package dependency management
-3. 🎯 **CLI development server integration** (this task)
-
-The result is seamless JoystickDB development experience matching existing database providers while leveraging the unique advantage of embedded Node.js execution rather than binary dependencies.

package/prompts/FULL_TEXT_SEARCH.md

@@ -1,293 +0,0 @@
-# Full Text Search Implementation Prompt
-
-## Overview
-
-Implement full text search capabilities for the JoystickDB LMDB-based database server. This feature should provide fast, scalable text search across document fields while maintaining the existing clustered architecture and performance characteristics.
-
-## Requirements
-
-### Core Functionality
-- Support full text search across specified document fields
-- Implement inverted index strategy for optimal search performance
-- Provide boolean query operators (AND, OR, NOT)
-- Support phrase matching with quotes
-- Include fuzzy matching capabilities for typos/variations
-- Case-insensitive search by default with case-sensitive option
-
-### Text Processing
-- Tokenization: Split text into searchable terms
-- Normalization: Convert to lowercase, handle Unicode
-- Stop word filtering: Remove common words (the, and, or, etc.)
-- Stemming: Reduce words to root forms (running → run)
-- Support for multiple languages (start with English)
-
-### Index Management
-- Automatic index creation for designated searchable fields
-- Atomic index updates during document modifications
-- Index rebuilding capabilities for schema changes
-- Configurable index settings per field
-
-### Query Interface
-- Add new operation type: `text_search`
-- Query syntax: `{ "query": "search terms", "fields": ["title", "content"], "options": {...} }`
-- Support for field-specific searches: `title:"exact phrase" content:keyword`
-- Result ranking based on relevance scores
-- Pagination support for large result sets
-
-## Technical Implementation
-
-### LMDB Storage Strategy
-```
-Document Storage:
-- Key: `doc:{collection}:{id}`
-- Value: Original document data
-
-Inverted Index Storage:
-- Key: `idx:{collection}:{field}:{term}`
-- Value: Array of document IDs with positions
-
-Term Frequency Storage:
-- Key: `tf:{collection}:{field}:{doc_id}`
-- Value: Term frequency map for relevance scoring
-
-Metadata Storage:
-- Key: `meta:{collection}:search_config`
-- Value: Search configuration and field mappings
-```
-
-### Architecture Integration
-
-#### Master Process Responsibilities
-- Handle index updates during write operations
-- Coordinate index rebuilding across workers
-- Manage search configuration changes
-- Ensure index consistency during bulk operations
-
-#### Worker Process Responsibilities
-- Execute search queries against local LMDB instances
-- Perform text processing and tokenization
-- Calculate relevance scores
-- Handle search result pagination
-
-### New Operation Types
-Add to `op_types.js`:
-```javascript
-"text_search",
-"create_text_index",
-"drop_text_index",
-"rebuild_text_index"
-```
-
-### Search Query Structure
-```javascript
-{
-  "op": "text_search",
-  "data": {
-    "collection": "documents",
-    "query": "machine learning algorithms",
-    "fields": ["title", "content", "tags"],
-    "options": {
-      "fuzzy": true,
-      "case_sensitive": false,
-      "max_results": 50,
-      "offset": 0,
-      "highlight": true,
-      "min_score": 0.1
-    }
-  }
-}
-```
-
-### Response Format
-```javascript
-{
-  "success": true,
-  "data": {
-    "results": [
-      {
-        "document": { /* original document */ },
-        "score": 0.85,
-        "highlights": {
-          "title": ["<mark>machine learning</mark> basics"],
-          "content": ["advanced <mark>algorithms</mark> for..."]
-        }
-      }
-    ],
-    "total_count": 127,
-    "search_time_ms": 23,
-    "query_info": {
-      "parsed_terms": ["machine", "learning", "algorithms"],
-      "fields_searched": ["title", "content", "tags"]
-    }
-  }
-}
-```
-
-## Implementation Steps
-
-### Phase 1: Core Infrastructure
-1. Create text processing utilities (tokenization, normalization, stemming)
-2. Implement inverted index data structures
-3. Add LMDB storage patterns for search indexes
-4. Create search configuration management
-
-### Phase 2: Basic Search
-1. Implement simple keyword search
-2. Add boolean operators (AND, OR, NOT)
-3. Create relevance scoring algorithm
-4. Add result ranking and pagination
-
-### Phase 3: Advanced Features
-1. Implement phrase matching
-2. Add fuzzy search capabilities
-3. Create result highlighting
-4. Add field-specific search syntax
-
-### Phase 4: Integration
-1. Integrate with existing cluster architecture
-2. Add search operations to worker/master communication
-3. Implement atomic index updates
-4. Add search-specific error handling
-
-### Phase 5: Optimization
-1. Performance tuning for large datasets
-2. Memory usage optimization
-3. Index compression techniques
-4. Query optimization strategies
-
-## File Structure
-
-```
-src/server/search/
-├── index.js                 # Main search module exports
-├── text_processor.js        # Tokenization, stemming, normalization
-├── inverted_index.js        # Index creation and management
-├── query_parser.js          # Parse search query syntax
-├── relevance_scorer.js      # Calculate document relevance scores
-├── search_executor.js       # Execute search queries
-├── index_manager.js         # Manage index lifecycle
-└── search_config.js         # Search configuration utilities
-
-src/server/lib/
-├── search_operations.js     # Search-specific database operations
-└── search_utils.js          # Shared search utilities
-
-test/server/search/
-├── text_processor.test.js
-├── inverted_index.test.js
-├── query_parser.test.js
-├── relevance_scorer.test.js
-└── search_integration.test.js
-```
-
-## Configuration Options
-
-### Database Settings
-```javascript
-{
-  "search": {
-    "default_language": "en",
-    "stop_words_enabled": true,
-    "stemming_enabled": true,
-    "fuzzy_threshold": 0.8,
-    "max_index_memory_mb": 512,
-    "index_update_batch_size": 1000,
-    "search_timeout_ms": 5000
-  }
-}
-```
-
-### Field-Level Configuration
-```javascript
-{
-  "collections": {
-    "documents": {
-      "search_fields": {
-        "title": {
-          "weight": 2.0,
-          "stemming": true,
-          "fuzzy": true
-        },
-        "content": {
-          "weight": 1.0,
-          "stemming": true,
-          "fuzzy": false
-        },
-        "tags": {
-          "weight": 1.5,
-          "stemming": false,
-          "fuzzy": true
-        }
-      }
-    }
-  }
-}
-```
-
-## Performance Considerations
-
-### Memory Management
-- Use LMDB's memory-mapped files for large indexes
-- Implement index compression for frequently accessed terms
-- Cache frequently used search results
-- Limit concurrent search operations per worker
-
-### Scalability
-- Distribute index updates across multiple workers
-- Implement index sharding for very large collections
-- Use background processes for index maintenance
-- Monitor index size and performance metrics
-
-### Query Optimization
-- Implement query plan optimization
-- Use term frequency for query reordering
-- Cache parsed queries for repeated searches
-- Implement search result caching
-
-## Testing Strategy
-
-### Unit Tests
-- Text processing functions
-- Index creation and updates
-- Query parsing logic
-- Relevance scoring algorithms
-
-### Integration Tests
-- End-to-end search operations
-- Index consistency during concurrent writes
-- Search performance under load
-- Error handling and recovery
-
-### Performance Tests
-- Search latency benchmarks
-- Index update performance
-- Memory usage patterns
-- Concurrent search handling
-
-## Success Criteria
-
-1. **Functionality**: All search operations work correctly with accurate results
-2. **Performance**: Search queries complete within 100ms for typical datasets
-3. **Scalability**: System handles 1000+ concurrent searches without degradation
-4. **Reliability**: Index consistency maintained during all write operations
-5. **Usability**: Intuitive query syntax with comprehensive error messages
-
-## Dependencies
-
-### New Package Dependencies
-```javascript
-{
-  "natural": "^6.0.0",        // Text processing and stemming
-  "stopword": "^2.0.0",       // Stop word removal
-  "fuse.js": "^7.0.0"         // Fuzzy search capabilities
-}
-```
-
-### Development Dependencies
-```javascript
-{
-  "benchmark": "^2.1.4"       // Performance testing
-}
-```
-
-This implementation will provide a robust, scalable full text search solution that integrates seamlessly with the existing JoystickDB architecture while maintaining high performance and reliability.