@joystick.js/db-canary 0.0.0-canary.2209

package/prompts/01-core-infrastructure.md

@@ -0,0 +1,56 @@
+# Task 1: Core Infrastructure Upgrades
+
+## Objective
+Upgrade the existing JoystickDB implementation with core production-grade infrastructure features.
+
+## Current State
+- Basic LMDB query engine with CRUD operations
+- Simple TCP server using Node.js net module on port 1983
+- MessagePack serialization with msgpackr
+- Basic authentication and settings management
+- 119 passing tests
+
+## Requirements
+
+### 1. Auto-calculating LMDB map_size
+- Detect total disk size using `df -k .` command
+- Set `map_size = disk_size * 0.8` (80% of total disk)
+- Example: 3 TB disk → map_size = ~2.4 TB
+- Implement auto-growth: when data approaches 80% of current map_size, double it (capped at 80% of total disk)
+- Add monitoring and logging for map_size changes
+
+### 2. Proper TCP Protocol with Length Prefixing
+- Replace current direct MessagePack protocol
+- Implement: 4-byte length prefix + MessagePack payload
+- Each message format: `[4-byte length][MessagePack data]`
+- Update both server message handling for proper framing
+
+### 3. Enhanced Server Configuration
+- Keep port 1983 as default
+- Add connection limits (~1000 max connections)
+- Implement idle connection timeout (10 minutes)
+- Add request timeout handling (5 seconds max per request)
+
+### 4. Dependencies Update
+- Install missing dependencies: `@aws-sdk/client-s3`, `crypto`
+- Keep msgpackr (no change needed)
+- Update package.json with new dependencies
+
+### 5. Enhanced Error Handling and Logging
+- Add structured logging with: `{ ts, client_id, op, collection, duration_ms, status }`
+- Implement proper connection management and cleanup
+- Add memory and performance monitoring
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Maintain all existing tests (119 tests must continue passing)
+- Add comprehensive tests for new features
+- Ensure graceful degradation if disk size detection fails
+
+## Success Criteria
+- All existing tests pass
+- New tests for map_size calculation and auto-growth
+- Proper TCP framing protocol implemented
+- Connection limits and timeouts working
+- Enhanced logging in place
+- Performance monitoring active

package/prompts/02-secondary-indexing.md

@@ -0,0 +1,65 @@
+# Task 2: Secondary Indexing System
+
+## Objective
+Implement a comprehensive secondary indexing system for JoystickDB to optimize query performance.
+
+## Current State
+- Basic LMDB storage with collection:document_id keys
+- MongoDB-compatible query operators implemented
+- No indexing - all queries scan entire collections
+- Query engine supports filters but without optimization
+
+## Requirements
+
+### 1. Index Storage Architecture
+- Create sub-databases for indexed fields in LMDB
+- Index key format: `index:<collection>:<field>:<value> → [document_ids...]`
+- Support multiple document IDs per index entry (array of IDs)
+- Maintain separate index databases for different field types
+
+### 2. Index Management Operations
+- `create_index(collection, field, options)` - Create new index
+- `drop_index(collection, field)` - Remove existing index
+- `get_indexes(collection)` - List all indexes for collection
+- `list_collections()` - Enhanced to show index information
+
+### 3. Index Maintenance
+- Auto-update indexes on `insert_one`, `update_one`, `delete_one`
+- Handle `bulk_write` operations with proper index updates
+- Rollback index changes if document operations fail
+- Support for compound field updates (nested object fields)
+
+### 4. Query Optimization
+- Detect when queries can use indexes
+- Automatically use indexes for supported operators:
+  - `$eq`, `$ne`, `$in`, `$nin`
+  - `$gt`, `$gte`, `$lt`, `$lte` (range queries)
+  - `$exists`
+- Fall back to collection scan when no suitable index exists
+- Add query planning and execution statistics
+
+### 5. Index Types Support
+- Single field indexes
+- Compound indexes (multiple fields)
+- Sparse indexes (skip null/undefined values)
+- Unique indexes (enforce uniqueness constraint)
+
+### 6. Admin Integration
+- Extend admin operation to include index statistics
+- Add index-specific admin actions:
+  - `create_index`, `drop_index`, `get_indexes`
+  - Index usage statistics and performance metrics
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Maintain transactional consistency between documents and indexes
+- Add comprehensive error handling for index corruption
+- Implement index rebuilding capabilities for recovery
+
+## Success Criteria
+- All existing tests continue to pass
+- New comprehensive test suite for indexing operations
+- Query performance significantly improved for indexed fields
+- Index management operations working correctly
+- Proper error handling and recovery mechanisms
+- Admin operations enhanced with index information

package/prompts/03-write-serialization.md

@@ -0,0 +1,63 @@
+# Task 3: Write Serialization and Queuing
+
+## Objective
+Implement proper write serialization to ensure only one LMDB write transaction occurs at a time, with proper queuing for concurrent writes.
+
+## Current State
+- Multiple concurrent writes can cause LMDB conflicts
+- No write queuing system in place
+- Bulk operations work but lack proper serialization
+- No write performance monitoring
+
+## Requirements
+
+### 1. Write Queue System
+- Implement a single write queue for all write operations
+- Queue operations: `insert_one`, `update_one`, `delete_one`, `bulk_write`
+- Process writes sequentially in FIFO order
+- Maintain operation context and client information
+
+### 2. Transaction Management
+- Ensure only one LMDB write transaction at a time
+- Implement proper transaction rollback on failures
+- Handle transaction timeouts and deadlocks
+- Add transaction retry logic with exponential backoff
+
+### 3. Write Operation Wrapper
+- Create unified write operation handler
+- Wrap all write operations in queue system
+- Maintain operation ordering and atomicity
+- Add write operation logging and metrics
+
+### 4. Concurrent Read Support
+- Allow concurrent read operations during writes
+- Implement read-write lock semantics
+- Ensure read consistency during write operations
+- Optimize read performance while writes are queued
+
+### 5. Performance Monitoring
+- Track write queue depth and wait times
+- Monitor write operation throughput
+- Add write latency metrics
+- Implement write performance alerts
+
+### 6. Error Handling and Recovery
+- Handle write queue overflow scenarios
+- Implement graceful degradation under high load
+- Add write operation timeout handling
+- Ensure proper cleanup on server shutdown
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Maintain all existing functionality and tests
+- Add comprehensive error handling
+- Implement proper resource cleanup
+
+## Success Criteria
+- All existing tests continue to pass
+- Write operations properly serialized
+- No LMDB write conflicts under concurrent load
+- Write queue functioning correctly
+- Performance monitoring in place
+- Proper error handling and recovery
+- New tests for write serialization scenarios

package/prompts/04-enhanced-authentication.md

@@ -0,0 +1,75 @@
+# Task 4: Enhanced Authentication System
+
+## Objective
+Implement a production-grade authentication system with proper password management, rate limiting, and protocol versioning.
+
+## Current State
+- Basic bcrypt authentication with simple password checking
+- Authentication stored in settings, not dedicated auth file
+- No rate limiting for failed attempts
+- No protocol versioning
+- Simple socket-based session management
+
+## Requirements
+
+### 1. Setup Command for Password Bootstrap
+- Add new `setup` operation to op_types
+- Generate 32-character random password on `setup` command
+- Store bcrypt hash in dedicated `./authentication.json` file
+- Return the generated password to client (one-time only)
+- Only allow setup if no authentication.json exists (prevent overwrite)
+
+### 2. Authentication File Management
+- Create `./authentication.json` with structure:
+  ```json
+  {
+    "password_hash": "bcrypt_hash_here",
+    "created_at": "2025-08-30T19:00:00.000Z",
+    "last_updated": "2025-08-30T19:00:00.000Z",
+    "failed_attempts": {},
+    "rate_limits": {}
+  }
+  ```
+- Secure file permissions (600)
+- Atomic file updates for password changes
+
+### 3. Rate Limiting System
+- Implement 5 failed attempts per minute per IP address
+- Track failed attempts in memory and persist to authentication.json
+- Exponential backoff for repeated failures
+- IP-based blocking with automatic expiration
+- Whitelist for local/trusted IPs
+
+### 4. Protocol Versioning
+- Return version information after successful authentication
+- Response format: `{ ok: 1, version: "1.0.0" }`
+- Maintain API compatibility promise
+- Add version checking for future client compatibility
+
+### 5. Enhanced Session Management
+- Improve socket-based session tracking
+- Add session timeouts and cleanup
+- Track authentication timestamps
+- Support session invalidation
+
+### 6. Security Enhancements
+- Add timing attack protection for password verification
+- Implement secure password generation utilities
+- Add authentication audit logging
+- Support for future multi-factor authentication
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Update op_types.js to include "setup" operation
+- Maintain backward compatibility with existing tests
+- Add comprehensive security testing
+- Implement proper error handling without information leakage
+
+## Success Criteria
+- All existing tests continue to pass
+- Setup command generates and returns password
+- Rate limiting prevents brute force attacks
+- Protocol versioning implemented
+- Secure authentication.json file management
+- New comprehensive authentication test suite
+- Security audit logging in place

package/prompts/05-comprehensive-admin-operations.md

@@ -0,0 +1,75 @@
+# Task 5: Comprehensive Admin Operations
+
+## Objective
+Expand the basic admin operation into a comprehensive administrative interface with metadata management, data operations, and system monitoring.
+
+## Current State
+- Basic admin operation returns server info, database stats, and authentication status
+- No granular admin actions or data management capabilities
+- Limited administrative functionality
+
+## Requirements
+
+### 1. Admin Action Framework
+- Extend admin operation to support `admin_action` parameter
+- Route different admin actions to specialized handlers
+- Maintain authentication requirements for all admin operations
+- Add proper error handling and validation
+
+### 2. Metadata Operations
+- `stats` - Enhanced server and database statistics
+  - Memory usage, disk usage, ops/sec metrics
+  - Database size, document counts, map_size usage %
+  - Index statistics and performance metrics
+- `list_collections` - List all collections with metadata
+  - Document counts per collection
+  - Index information per collection
+  - Collection size and statistics
+- Index management (if indexing implemented):
+  - `create_index` - Create new indexes
+  - `drop_index` - Remove existing indexes
+  - `get_indexes` - List indexes for collection
+
+### 3. Data Management Operations
+- `list_documents` - List documents in collection with pagination
+- `get_document` - Retrieve specific document by ID
+- `query_documents` - Execute ad-hoc queries with admin privileges
+- `insert_document` - Insert document with admin privileges
+- `update_document` - Update document with admin privileges
+- `delete_document` - Delete document with admin privileges
+
+### 4. System Monitoring
+- Real-time performance metrics
+- Connection statistics and client information
+- Query performance analytics
+- Error rate monitoring and alerting
+- Resource usage tracking (CPU, memory, disk I/O)
+
+### 5. Enhanced Statistics
+- Operations per second (read/write breakdown)
+- Average query response times
+- Connection pool statistics
+- Cache hit rates (if applicable)
+- Database health indicators
+
+### 6. Admin Security
+- Separate admin authentication level (if needed)
+- Admin operation audit logging
+- Rate limiting for admin operations
+- Admin session management
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Extend current admin operation rather than replacing it
+- Add comprehensive error handling and validation
+- Implement proper pagination for large result sets
+- Add performance monitoring for admin operations themselves
+
+## Success Criteria
+- All existing tests continue to pass
+- Admin action routing working correctly
+- All metadata operations implemented and tested
+- Data management operations secure and functional
+- System monitoring providing accurate metrics
+- Comprehensive admin test suite
+- Proper error handling and security measures

package/prompts/06-backup-and-restore-system.md

@@ -0,0 +1,106 @@
+# Task 6: Backup and Restore System
+
+## Objective
+Implement a comprehensive backup and restore system with S3 integration, automatic scheduling, and one-shot restore capabilities.
+
+## Current State
+- No backup or restore functionality
+- No S3 integration
+- No automated backup scheduling
+- No backup verification or integrity checking
+
+## Requirements
+
+### 1. S3 Integration Setup
+- Install and configure `@aws-sdk/client-s3`
+- Support S3-compatible endpoints (AWS, MinIO, etc.)
+- Implement S3 credential management from settings.json
+- Add S3 connection testing and validation
+
+### 2. Backup Operations
+- `backup_now` admin action for manual backups
+- Create compressed sparse backups using `tar --sparse`
+- Generate SHA256 checksums for backup verification
+- Stream backups directly to S3 (minimal local storage)
+- Keep only current backup locally as temporary file
+- Backup filename format: `joystickdb-backup-YYYY-MM-DDTHH:mm:ss.tar.gz`
+
+### 3. Automatic Backup Scheduling
+- Implement hourly backup schedule if S3 configured
+- Use settings.json `backup_schedule` parameter (default: "hourly")
+- Support different schedules: hourly, daily, weekly
+- Add backup job management and monitoring
+- Ensure backups don't interfere with database operations
+
+### 4. Backup Retention Management (S3 Only)
+- Keep 24 hourly backups in S3
+- Keep 30 daily backups (retain one per day after 24 hours)
+- Automatic cleanup of old backups in S3
+- Configurable retention policies
+- Safe deletion with verification
+- No long-term local backup storage
+
+### 5. Restore System
+- `restore_backup` admin action for manual restore
+- One-shot restore via settings.json `restore_from` parameter
+- Download backup from S3 to temporary local file
+- Verify SHA256 checksum before restore
+- Replace `./data/data.mdb` with restored data
+- Clean up temporary restore files immediately
+- Handle restore failures gracefully (boot fresh LMDB)
+- Auto-remove `restore_from` from settings.json on success
+
+### 6. Backup Management Operations
+- `list_backups` admin action to show available S3 backups
+- Show backup metadata: size, date, checksum, S3 location
+- Support filtering and sorting of backup lists
+- Backup integrity verification commands
+- Backup deletion and cleanup operations in S3
+
+### 7. Settings.json Schema Support
+```json
+{
+  "s3": {
+    "bucket": "joystickdb-backups",
+    "region": "us-east-1",
+    "access_key": "AKIA...",
+    "secret_key": "abcd...",
+    "endpoint": "https://s3.amazonaws.com"
+  },
+  "backup_schedule": "hourly",
+  "restore_from": "joystickdb-backup-2025-08-28T12:00:00.tar.gz"
+}
+```
+
+### 8. Local Storage Management
+- Use temporary directory for backup creation
+- Stream backups to S3 during creation
+- Clean up temporary files immediately after S3 upload
+- Monitor local disk usage during backup operations
+- Fail backup if insufficient local disk space
+
+### 9. Error Handling and Recovery
+- Robust error handling for S3 operations
+- Network failure recovery for uploads/downloads
+- Immediate cleanup of temporary files on failures
+- Backup corruption detection and handling
+- Comprehensive logging for backup operations
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Add backup operations to admin action routing
+- Implement proper file locking during backups
+- Use streaming for large backup files to minimize disk usage
+- Add comprehensive error handling and logging
+- Prioritize S3 storage over local storage
+
+## Success Criteria
+- All existing tests continue to pass
+- Manual backup operations working correctly
+- Automatic backup scheduling functional
+- S3 integration working with proper error handling
+- Restore operations reliable and safe
+- Backup retention policies enforced in S3
+- Minimal local disk usage for backups
+- Comprehensive backup test suite
+- Settings.json schema properly supported

package/prompts/07-production-safety-features.md

@@ -0,0 +1,107 @@
+# Task 7: Production Safety Features
+
+## Objective
+Implement comprehensive production safety features including health monitoring, timeouts, connection management, and enhanced logging.
+
+## Current State
+- Basic server functionality without production safety measures
+- Limited connection management
+- Basic logging without structured monitoring
+- No timeout handling or resource limits
+
+## Requirements
+
+### 1. Health Monitoring System
+- Enhanced `ping` operation always returns `{ ok: 1 }`
+- Comprehensive `stats` in admin operations including:
+  - Memory usage and trends
+  - Disk usage and available space
+  - Operations per second (read/write breakdown)
+  - Document count and database size
+  - Map_size usage percentage
+  - Connection statistics
+  - Error rates and response times
+
+### 2. Request Timeout Management
+- Implement 5-second maximum timeout per request
+- Add timeout handling for all operations
+- Graceful timeout responses to clients
+- Timeout monitoring and logging
+- Configurable timeout values per operation type
+
+### 3. Connection Management
+- Maximum ~1000 concurrent connections
+- Connection pooling and resource management
+- Drop idle connections after 10 minutes
+- Connection rate limiting and throttling
+- Client connection tracking and monitoring
+- Graceful connection cleanup on server shutdown
+
+### 4. Enhanced Structured Logging
+- Implement comprehensive logging format:
+  ```json
+  {
+    "ts": "2025-08-30T19:00:00.000Z",
+    "client_id": "socket_12345",
+    "op": "find_one",
+    "collection": "users",
+    "duration_ms": 45,
+    "status": "success|error",
+    "error": "error_message_if_any",
+    "request_size": 1024,
+    "response_size": 2048
+  }
+  ```
+- Log all operations with performance metrics
+- Error tracking and categorization
+- Request/response size monitoring
+
+### 5. Resource Monitoring and Alerts
+- Memory usage monitoring with thresholds
+- Disk space monitoring and alerts
+- CPU usage tracking
+- Database performance metrics
+- Connection pool health monitoring
+- Automatic resource cleanup
+
+### 6. Error Handling and Recovery
+- Comprehensive error categorization
+- Graceful degradation under high load
+- Circuit breaker patterns for failing operations
+- Automatic recovery mechanisms
+- Error rate monitoring and alerting
+- Client error response standardization
+
+### 7. Performance Optimization
+- Query performance monitoring
+- Slow query detection and logging
+- Connection reuse optimization
+- Memory usage optimization
+- Database operation batching where appropriate
+- Resource usage profiling
+
+### 8. Security and Rate Limiting
+- Request rate limiting per client
+- Resource usage limits per connection
+- DDoS protection mechanisms
+- Malformed request handling
+- Security event logging
+- IP-based connection limits
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Integrate with existing logging infrastructure
+- Add comprehensive monitoring without impacting performance
+- Implement graceful degradation strategies
+- Add extensive testing for edge cases and failure scenarios
+
+## Success Criteria
+- All existing tests continue to pass
+- Request timeouts working correctly
+- Connection limits enforced properly
+- Enhanced logging providing detailed insights
+- Health monitoring accurate and responsive
+- Resource usage optimized and monitored
+- Error handling robust and informative
+- Performance metrics available and actionable
+- New comprehensive test suite for safety features

package/prompts/08-tcp-client-library.md

@@ -0,0 +1,121 @@
+# Task 8: JoystickDB TCP Client Library
+
+## Objective
+Create a comprehensive Node.js client library for connecting to JoystickDB with automatic reconnection, connection pooling, and full operation support.
+
+## Current State
+- No client library exists
+- Server uses TCP with 4-byte length prefix + MessagePack protocol
+- Server supports all CRUD operations, admin actions, and authentication
+
+## Requirements
+
+### 1. Client Connection Management
+- Export main `joystickdb` object with `client(options)` method
+- Automatic connection establishment and authentication
+- Connection pooling and reuse
+- Auto-reconnection with exponential backoff
+- Connection health monitoring and heartbeat
+
+### 2. Authentication Handling
+- Automatic authentication on connect using provided password
+- Handle authentication failures gracefully
+- Support for setup operation (password generation)
+- Session management and re-authentication on reconnect
+- Protocol version handling
+
+### 3. CRUD Operations
+- `insert_one(collection, document, options)` 
+- `find_one(collection, filter, options)`
+- `find(collection, filter, options)`
+- `update_one(collection, filter, update, options)`
+- `delete_one(collection, filter, options)`
+- `bulk_write(collection, operations, options)`
+
+### 4. Admin Operations
+#### Metadata Operations
+- `get_stats()` - Server and database statistics
+- `list_collections()` - List all collections
+- `create_index(collection, field, options)` - Create index
+- `drop_index(collection, field)` - Drop index  
+- `get_indexes(collection)` - List collection indexes
+
+#### Data Management Operations
+- `list_documents(collection, options)` - List documents with pagination
+- `get_document(collection, document_id)` - Get specific document
+- `query_documents(collection, filter, options)` - Ad-hoc queries
+- `insert_document(collection, document)` - Admin insert
+- `update_document(collection, document_id, update)` - Admin update
+- `delete_document(collection, document_id)` - Admin delete
+
+#### Backup Operations
+- `backup_now()` - Trigger manual backup
+- `list_backups()` - List available backups
+- `restore_backup(backup_name)` - Restore from backup
+
+### 5. Health and Utility Operations
+- `ping()` - Health check
+- `reload()` - Reload server configuration
+
+### 6. Protocol Implementation
+- Implement 4-byte length prefix + MessagePack protocol
+- Handle message framing correctly
+- Support for large message handling
+- Proper error handling for malformed messages
+
+### 7. Request Management
+- Per-request timeout (5 seconds default, configurable)
+- Request queuing during disconnection
+- Request retry logic with backoff
+- Concurrent request handling
+- Request/response correlation
+
+### 8. Error Handling and Recovery
+- Comprehensive error handling for all operations
+- Network error recovery and reconnection
+- Timeout handling and cleanup
+- Connection failure notifications
+- Graceful degradation strategies
+
+### 9. Client Configuration and Usage
+```javascript
+import joystickdb from 'joystickdb';
+
+const client = joystickdb.client({
+  host: 'localhost',
+  port: 1983,
+  authentication: {
+    password: 'generated_password_from_setup'
+  },
+  timeout: 5000,
+  reconnect: true,
+  max_reconnect_attempts: 10,
+  reconnect_delay: 1000
+});
+```
+
+### 10. Event System
+- Connection events: 'connect', 'disconnect', 'error', 'reconnect'
+- Operation events for monitoring and debugging
+- Error event handling and logging
+- Connection state change notifications
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Use msgpackr for MessagePack serialization (consistent with server)
+- Implement proper resource cleanup and connection management
+- Add comprehensive error handling and logging
+- Support both callback and Promise-based APIs
+- Password should be obtained from setup operation and stored separately
+- Main export should be `joystickdb` object with `client()` method
+
+## Success Criteria
+- Client can connect and authenticate successfully
+- All CRUD operations working correctly
+- All admin operations implemented and functional
+- Auto-reconnection working reliably
+- Request timeout and retry logic functional
+- Comprehensive error handling in place
+- Full test suite covering all client functionality
+- Documentation and usage examples provided
+- Performance optimized for high-throughput scenarios