@joystick.js/db-canary 0.0.0-canary.2209

package/prompts/17-emergency-password-change.md

@@ -0,0 +1,108 @@
+# Task 17: Emergency Password Change System
+
+## Objective
+Implement a secure emergency password change system that allows administrators to quickly change the database password in case of security breaches or emergencies, with minimal downtime and maximum security.
+
+## Current State
+- Authentication system with bcrypt password hashing implemented
+- Password stored in authentication.json file
+- Current setup only allows initial password generation
+- 423 passing tests from previous tasks
+
+## Requirements
+
+### 1. Emergency Recovery Token Generation
+- Add CLI flag `--generate-recovery-token` to server startup
+- When flag is used, server should:
+  - Generate cryptographically secure recovery token (UUID v4 or similar)
+  - Print recovery URL to console
+  - Exit immediately (do not start normal server operations)
+- Example usage: `node server.js --generate-recovery-token`
+- Example output:
+  ```
+  Emergency Recovery Token Generated
+  Visit: http://localhost:1984/recovery?token=abc123def456
+  Token expires in 10 minutes
+  ```
+
+### 2. Recovery Token Storage
+- Store generated token temporarily with expiration timestamp
+- Use in-memory storage or temporary file that auto-expires
+- Token valid for 10 minutes from generation
+- Only one recovery token can be active at a time
+- Token automatically invalidated after first successful use
+
+### 3. Recovery Password Change Endpoint
+- Add HTTP endpoint `/recovery` requiring valid recovery token
+- Validate token exists and has not expired
+- Present simple form to enter new password
+- Require password confirmation (enter twice)
+- Implement password strength validation (minimum 12 characters)
+
+### 4. Password Change Process
+- Hash new password using existing bcrypt implementation
+- Atomically update authentication.json file
+- Invalidate all existing client connections (force re-authentication)
+- Generate new authentication salt/keys if applicable
+- Log successful password change with timestamp
+
+### 5. Minimal Downtime Strategy
+- Password change should not require server restart
+- Existing connections gracefully terminated with re-auth message
+- New connections immediately use new password
+- Process should complete in under 30 seconds
+- Recovery token generation does not interfere with running server
+
+### 6. Security Measures
+- Recovery tokens can only be generated by server administrator (CLI access)
+- Rate limiting: only one recovery token active at a time
+- All recovery attempts logged with IP address and timestamp
+- Token brute force protection (lock after 3 failed attempts)
+- Recovery token generation requires server file system access
+
+### 7. Audit Trail
+- Log recovery token generation attempts
+- Log all recovery password change attempts (success/failure)
+- Log client disconnections due to password change
+- Maintain audit log separate from regular application logs
+
+### 8. Recovery Instructions
+- Provide clear instructions in response:
+  ```
+  Password changed successfully!
+  
+  New password is now active.
+  All existing connections have been terminated.
+  
+  Next steps:
+  1. Update your app's environment settings so the client can connect using your new password
+  2. Restart or redeploy your applications to use new password
+  3. Monitor logs for any unauthorized access attempts
+  
+  Recovery completed at: [timestamp]
+  ```
+
+### 9. CLI Integration
+- Add command line argument parsing for `--generate-recovery-token`
+- Ensure flag works with existing server startup process
+- Provide help text for recovery token generation
+- Handle edge cases like server already running
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Maintain all existing tests (423 tests must continue passing)
+- Add comprehensive tests for recovery token generation and password change
+- Ensure no interference with normal server operations
+- Use existing bcrypt and authentication utilities
+
+## Success Criteria
+- All existing tests pass (423 tests)
+- `--generate-recovery-token` flag generates token and exits
+- Recovery endpoint works with valid tokens at `/recovery`
+- Password changes take effect immediately without server restart
+- All existing connections terminated and require re-authentication
+- Comprehensive audit logging of all recovery operations
+- Token expiration and rate limiting work correctly
+- Clear instructions provided for post-change steps
+- No security vulnerabilities in recovery process
+- CLI flag does not interfere with normal server startup

package/prompts/18-joystick-framework-integration.md

@@ -0,0 +1,116 @@
+# Task 18: Joystick Framework Integration - Phase 1: Migration and Setup
+
+## Objective
+Migrate the complete JoystickDB wrapper implementation from the standalone repository into the main Joystick framework, establishing it as an integrated database solution alongside existing database drivers.
+
+## Current State
+- Complete JoystickDB implementation with 474 passing tests
+- Standalone Node.js wrapper around LMDB with TCP server/client architecture
+- Full production-grade features: authentication, clustering, replication, backup/restore
+- Located at: `/Users/rglover/projects/cheatcode/joystick-db`
+- Target integration: `/Users/rglover/projects/cheatcode/joystick` framework
+
+## Phase 1 Requirements: Migration and Structure Setup
+
+### 1. Repository Migration
+- **Source**: `/Users/rglover/projects/cheatcode/joystick-db`
+- **Target**: `/Users/rglover/projects/cheatcode/joystick/db`
+- Migrate entire source tree maintaining directory structure
+- Preserve all existing functionality and tests
+- Maintain git history where possible
+
+### 2. Framework Integration Structure
+- Study existing database packages in `/Users/rglover/projects/cheatcode/joystick`
+- Mimic established patterns for:
+  - Package structure and organization
+  - Build and release processes
+  - Canary release behavior
+  - Configuration management
+  - Integration points with CLI and node back-end
+
+### 3. Package Configuration
+- Update `package.json` to match framework conventions
+- Establish proper dependencies and peer dependencies
+- Configure build scripts and testing workflows
+- Set up canary release pipeline integration
+
+### 4. Branch Management
+- Create feature branch: `feature/joystick-db-integration`
+- All work must be done in this branch
+- No direct commits to master or canary branch
+
+## Architecture Changes for Integration
+
+### 1. Database Server Process Management
+- **Current**: Direct LMDB wrapper execution
+- **Target**: Child process management for database server
+- Server startup via `src/server/index.js` as separate process
+- Process lifecycle management (start, stop, restart, health checks)
+- Integration with framework's process management system
+
+### 2. Client Driver Implementation
+- Implement JoystickDB as a database driver in the node portion
+- Follow existing driver patterns and interfaces
+- Provide seamless integration with framework's ORM/query builder
+- Maintain compatibility with existing JoystickDB client API
+- Add support for accounts and queues to the JoystickDB integration (just like MongoDB and PostgreSQL).
+
+### 3. CLI Integration Points
+- Database management commands
+- Server lifecycle controls (start, stop, status)
+- Migration and setup utilities
+- Configuration management through CLI
+
+## Implementation Guidelines
+
+### Code Standards
+- Follow Joystick framework conventions
+- Maintain existing JoystickDB code patterns where compatible
+- Use ESM syntax (`import`/`export`)
+- Follow snake_case for variables, functions, and keys
+- Two-space indentation
+- Minimal comments (only when necessary for clarity)
+
+### Testing Requirements
+- All existing JoystickDB tests (474) must continue to pass
+- Add integration tests for framework compatibility
+- Test child process management and lifecycle
+- Verify CLI integration points
+- Retain usage of Ava.js for authoring tests (do not modify the existing tests beyond ensuring they test the correct code within the joystick/db folder).
+
+### Dependencies Management
+- Integrate with framework's dependency management
+- Maintain JoystickDB-specific dependencies: `lmdb`, `msgpackr`, `bcrypt`, `@aws-sdk/client-s3`
+- Resolve any conflicts with framework dependencies
+- Update peer dependencies as needed
+
+## Success Criteria - Phase 1
+- ✅ Complete source migration to `/Users/rglover/projects/cheatcode/joystick/db`
+- ✅ All 474 existing tests continue to pass in new location
+- ✅ Package structure matches framework conventions
+- ✅ Canary release pipeline configured
+- ✅ Build and development workflows operational
+- ✅ Branch created and all work isolated properly
+
+## Future Phases (Separate Tasks/Chats)
+
+### Phase 2: Process Management Integration
+- Implement child process management for database server
+- CLI commands for server lifecycle management
+- Health monitoring and automatic restart capabilities
+
+### Phase 3: Driver Implementation
+- Create JoystickDB driver for framework's node back-end
+- Integrate with existing ORM/query patterns
+- Provide migration utilities and setup commands
+
+## Notes
+- This is Phase 1 of a multi-phase integration
+- Focus on migration and structural setup only
+- Preserve all existing functionality during migration
+- Each phase will require a new prompt and potentially new chat session
+- Integration must not break existing framework functionality
+- JoystickDB should be additive to the framework's capabilities
+
+## Context for Next Phase
+After Phase 1 completion, the next phase will focus on implementing child process management to replace the current binary download pattern used by other database drivers. This will require careful integration with the framework's process management system and CLI tooling.

package/prompts/19-api-key-authentication-system.md

@@ -0,0 +1,137 @@
+# Task 19: API Key Authentication System
+
+## Objective
+Implement an API key-based authentication system that eliminates auto-generated passwords, provides secure database initialization with API keys, and enables user management through HTTP APIs.
+
+## Current State
+- Enhanced authentication system with password management
+- HTTP server setup UI with token-based security
+- TCP server with MessagePack protocol
+- HTTP server for setup operations
+- 423 passing tests from previous tasks
+
+## Requirements
+
+### 1. API Key File Generation
+- Remove all auto-password generation functionality
+- On server startup, check for existing `API_KEY` file in root directory
+- If no `API_KEY` file exists, generate random API key (32-character alphanumeric)
+- Create `API_KEY` file with generated key
+- Display clear startup message:
+```
+JoystickDB Setup
+
+To finish setting up your database and enable operations for authenticated users, you will need to create an admin user via the database's admin API using the API key displayed below.
+
+=== STORE THIS KEY SECURELY -- IT WILL NOT BE DISPLAYED AGAIN ===
+[api_key_here]
+===
+
+To create a user, send a POST request to the server:
+
+<example POST request to http://localhost:<database_port>/api/users via fetch() with the API key stored in the x-joystick-db-api-key header>
+
+==!==
+Store this key securely. It will not be displayed again.
+==!==
+
+API key saved to: ./API_KEY
+```
+
+### 2. Pre-Setup Authentication
+- All HTTP requests must include `x-joystick-db-api-key` header
+- Validate header against contents of `API_KEY` file
+- Return 403 Forbidden for requests without valid API key:
+  ```json
+  {
+    "error": "Database setup incomplete. A valid API key must be passed until an admin user has been created."
+  }
+  ```
+- Only allow requests with valid API key until admin user exists
+
+### 3. User Management HTTP API
+- Remove existing setup UI in favor of HTTP API endpoints
+- Add `/api/users` endpoint for user management operations
+- Support user creation with username, password, and role
+- Available roles: `read`, `write`, `read_write`
+- All endpoints require valid `x-joystick-db-api-key` header
+
+### 4. User Creation Endpoint
+- `POST /api/users` - Create new user
+- Request body:
+  ```json
+  {
+    "username": "admin",
+    "password": "secure_password",
+    "role": "read_write"
+  }
+  ```
+- Response format:
+  ```json
+  {
+    "ok": 1,
+    "user": {
+      "username": "admin",
+      "role": "read_write",
+      "created_at": "2025-08-30T19:00:00.000Z"
+    }
+  }
+  ```
+
+### 5. User Management Operations
+- `GET /api/users` - List all users (admin only)
+- `PUT /api/users/:username` - Update user role/password
+- `DELETE /api/users/:username` - Delete user
+- All operations require valid API key header
+- Implement proper role-based access control
+
+### 6. Admin User Detection
+- Track when first admin user (role: `read_write`) is created
+- After admin user creation, API key requirement can be relaxed for authenticated operations (API key is still required for the /api/<resource> routes but normal TCP-based database operations with valid auth can pass through).
+- Transition to standard user authentication for subsequent requests
+- Maintain API key requirement for user management operations
+
+### 7. Security Enhancements
+- Store user data in the database itself in a _users collection.
+- Hash passwords using bcrypt with proper salt rounds
+- Implement rate limiting for user creation attempts
+- Add audit logging for all user management operations
+- Secure file permissions for API_KEY
+
+### 8. Error Handling and Validation
+- Validate username format (alphanumeric, 3-50 characters)
+- Enforce password complexity requirements
+- Validate role values against allowed options
+- Return descriptive error messages for validation failures
+- Handle duplicate username attempts gracefully
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Write small, single-purpose functions with max 4 arguments (combine into objects if needed)
+- Use clear, full names for maintainability - no shorthand or clever naming
+- Always use two-space indentation
+- Always use arrow functions unless function() required for scope
+- Avoid usage of this unless required (inside classes) or technically necessary
+- Always use snake_case for function names, argument names, and variable names
+- Always use closing semicolons
+- Always use ESM JavaScript (import instead of require)
+- Use default exports for single functions/modules, named exports for multiple
+- DO NOT write unnecessary comments
+- When comments are needed, use proper grammar and prefix with `// NOTE:`
+- Maintain all existing tests (423 tests must continue passing)
+- Add comprehensive tests for API key validation and user management
+- Remove password auto-generation from all existing code paths
+- Ensure atomic file operations for user data persistence
+
+## Success Criteria
+- All existing tests pass (423 tests)
+- API_KEY file generated on first startup
+- Clear startup message displays generated API key
+- HTTP requests require valid x-joystick-db-api-key header
+- 403 Forbidden returned for invalid/missing API keys
+- User creation endpoint accepts username, password, and role
+- User management operations work through HTTP API
+- No auto-generated passwords in any code path
+- Secure file handling for API keys and user data
+- Proper role-based access control implemented
+- Admin user detection triggers authentication transition

package/prompts/20-configurable-server-port.md

@@ -0,0 +1,105 @@
+# Task 20: Configurable Server Port
+
+## Objective
+Make the server port configurable through the `process.env.JOYSTICK_DB_SETTINGS` environment variable instead of being hardcoded, allowing flexible deployment configurations.
+
+## Current State
+- TCP server currently hardcoded to port 1983
+- HTTP server currently hardcoded to port 1984
+- Port configuration handled through command line arguments and environment variables
+- 423 passing tests from previous tasks
+- API key authentication system with HTTP endpoints
+
+## Requirements
+
+### 1. Settings-Based Port Configuration
+- Remove hardcoded port values (1983 for TCP, 1984 for HTTP)
+- Read port configuration from `process.env.JOYSTICK_DB_SETTINGS` JSON object
+- Support single `port` setting for TCP server
+- HTTP server port automatically calculated as `port + 1`
+- Maintain backward compatibility with existing `PORT` environment variable
+- Default value: TCP port 1983 (HTTP port 1984) if not specified
+
+### 2. Settings Schema Extension
+- Extend settings schema to include:
+  ```json
+  {
+    "port": 1983,
+    "authentication": { ... },
+    "s3": { ... },
+    "replication": { ... }
+  }
+  ```
+- Validate port number (1024-65534 range to allow for HTTP port + 1)
+- Ensure HTTP port (port + 1) doesn't exceed 65535
+
+### 3. Port Resolution Priority
+- Priority order for TCP port:
+  1. `JOYSTICK_DB_SETTINGS.port`
+  2. Default: 1983
+- HTTP port always calculated as: `tcp_port + 1`
+
+### 4. Server Startup Updates
+- Update `create_server()` function to read port from settings
+- Update cluster startup to use configured port
+- Update HTTP server initialization to use `port + 1`
+- Ensure proper error handling for port conflicts
+
+### 5. Logging and Diagnostics
+- Log configured ports during server startup
+- Include both TCP and HTTP port information in server status messages
+- Update startup messages to show actual ports being used
+- Add port validation warnings for invalid configurations
+
+### 6. Settings Validation
+- Validate port number is integer within valid range (1024-65534)
+- Ensure HTTP port (port + 1) doesn't exceed maximum port number (65535)
+- Provide clear error messages for invalid port configurations
+- Handle edge cases like ports already in use
+
+### 7. Backward Compatibility
+- Maintain support for existing `PORT` environment variable
+- Ensure existing deployment scripts continue working
+- Keep command line argument support for port specification
+- Preserve existing test configurations
+
+### 8. Configuration Examples
+- Document settings format in startup messages:
+  ```bash
+  export JOYSTICK_DB_SETTINGS='{"port": 3000}'
+  # TCP server: 3000, HTTP server: 3001
+  ```
+- Support common deployment scenarios:
+  - Development: default port 1983 (HTTP: 1984)
+  - Production: custom port via settings
+  - Docker: environment-based configuration
+  - Kubernetes: configmap-based settings
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Write small, single-purpose functions with max 4 arguments (combine into objects if needed)
+- Use clear, full names for maintainability - no shorthand or clever naming
+- Always use two-space indentation
+- Always use arrow functions unless function() required for scope
+- Avoid usage of this unless required (inside classes) or technically necessary
+- Always use snake_case for function names, argument names, and variable names
+- Always use closing semicolons
+- Always use ESM JavaScript (import instead of require)
+- Use default exports for single functions/modules, named exports for multiple
+- DO NOT write unnecessary comments
+- When comments are needed, use proper grammar and prefix with `// NOTE:`
+- Maintain all existing tests (423 tests must continue passing)
+- Add comprehensive tests for port configuration scenarios
+- Update existing port-related tests to use configurable ports
+- Ensure atomic configuration loading and validation
+
+## Success Criteria
+- All existing tests pass (423 tests)
+- TCP server port configurable via `JOYSTICK_DB_SETTINGS.port`
+- HTTP server port automatically set to `port + 1`
+- Port validation prevents invalid configurations (including HTTP port overflow)
+- Clear startup logging shows both TCP and HTTP ports
+- Settings reload updates port configuration appropriately
+- Comprehensive test coverage for all port configuration scenarios
+- Documentation updated with configuration examples
+- Error handling for port conflicts and invalid values

package/prompts/21-multi-database-support.md

@@ -0,0 +1,161 @@
+# Task 21: Multi-Database Support
+
+## Objective
+Implement multi-database support that allows multiple databases to run on a single server instance, providing a MongoDB-like chaining API for database and collection selection: `client.db('database_name').collection('collection_name').find()`.
+
+## Current State
+- Single database per server instance
+- Collection-based chaining API: `client.collection('name').find()`
+- TCP server on port 1983 with MessagePack protocol
+- HTTP server on port 1984 for setup operations
+- API key authentication system
+- All existing CRUD, indexing, admin, backup, and replication operations
+- 423+ passing tests from previous tasks
+
+## Requirements
+
+### 1. Multi-Database Architecture
+- Enable multiple databases to coexist on a single server instance
+- Each database maintains its own collections, indexes, and data isolation
+- Database names follow MongoDB naming conventions (alphanumeric, underscores, hyphens)
+- Maximum database name length of 64 characters
+- Reserved database names: `admin`, `config`, `local`
+
+### 2. Enhanced Client API
+- Replace current API with database selection via chaining
+- New API pattern: `client.db('database_name').collection('collection_name').find()`
+- Database selection returns a Database interface for further chaining
+- Collection selection from Database interface returns Collection interface
+- Remove direct `client.collection()` method - all operations must specify database
+
+### 3. Database Interface Implementation
+```javascript
+// New chaining API examples:
+const client = joystickdb.client({ password: 'password' });
+
+// Multi-database operations
+const results = await client.db('analytics').collection('events').find();
+const user = await client.db('users').collection('profiles').find_one({ email: 'user@example.com' });
+
+// Database-level operations
+const db = client.db('analytics');
+const collections = await db.list_collections();
+const stats = await db.get_stats();
+```
+
+### 4. Database Class Implementation
+- Create Database class similar to existing Collection class
+- Database class methods:
+  - `collection(name)` - Returns Collection interface for the database
+  - `list_collections()` - Lists collections in the database
+  - `get_stats()` - Gets database-specific statistics
+  - `drop_database()` - Drops the entire database (admin operation)
+  - `create_collection(name, options)` - Explicitly creates a collection
+
+### 5. Updated Collection Class
+- Collection class now requires database context
+- All existing Collection methods remain the same
+- Collection operations are scoped to the parent database
+- Constructor: `new Collection(client, database_name, collection_name)`
+
+### 6. Server-Side Database Management
+- Modify server to handle database context in all operations
+- Update message protocol to include database name in requests
+- Database creation on-demand (similar to MongoDB behavior)
+- Database isolation in file system storage
+- Update all existing operations to be database-aware
+
+### 7. Protocol Updates
+- Extend MessagePack protocol to include `database` field in requests
+- Request format:
+  ```javascript
+  {
+    op: 'find',
+    database: 'analytics', // Required field
+    data: {
+      collection: 'events',
+      filter: {},
+      options: {}
+    }
+  }
+  ```
+- All requests must include database field
+
+### 8. Storage Layer Updates
+- Organize data files by database: `data/{database_name}/{collection_name}.json`
+- Organize index files by database: `indexes/{database_name}/{collection_name}/`
+- Organize backup files by database: `backups/{database_name}/`
+- Atomic operations across database boundaries when needed
+
+### 9. Admin Operations Enhancement
+- Update admin operations to be database-aware
+- New admin operations:
+  - `list_databases` - Lists all databases on server
+  - `get_database_stats` - Gets statistics for specific database
+  - `drop_database` - Removes entire database and all collections
+  - `copy_database` - Copies database to new name
+- All admin operations require database context
+
+### 10. Authentication and Authorization
+- Maintain existing authentication system
+- Database access controlled by existing user roles
+- All authenticated users can access all databases
+- Future enhancement: database-specific permissions (not in this task)
+
+### 11. Backup and Restore Updates
+- Backup operations can target specific databases
+- Full server backup includes all databases
+- Restore operations can target specific databases
+- Backup file naming: `backup_{database_name}_{timestamp}.json`
+- Full backup naming: `backup_full_{timestamp}.json`
+
+### 12. Replication Updates
+- Replication system replicates all databases
+- Database-specific replication filtering (future enhancement)
+- Update replication protocol with database context
+
+### 13. Error Handling and Validation
+- Validate database names against naming rules
+- Handle database not found scenarios gracefully
+- Add database context to error messages where relevant
+- Return clear errors when database field is missing from requests
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Write small, single-purpose functions with max 4 arguments (combine into objects if needed)
+- Use clear, full names for maintainability - no shorthand or clever naming
+- Always use two-space indentation
+- Always use arrow functions unless function() required for scope
+- Avoid usage of this unless required (inside classes) or technically necessary
+- Always use snake_case for function names, argument names, and variable names
+- Always use closing semicolons
+- Always use ESM JavaScript (import instead of require)
+- Use default exports for single functions/modules, named exports for multiple
+- DO NOT write unnecessary comments
+- When comments are needed, use proper grammar and prefix with `// NOTE:`
+- Update all existing tests to use new API pattern
+- Add comprehensive tests for multi-database functionality
+- Database operations should be atomic and consistent
+
+## API Changes Required
+- Remove `client.collection(name)` method from JoystickDBClient class
+- Add `client.db(name)` method that returns Database instance
+- Update all client methods to require database context
+- Update Collection class constructor to accept database_name parameter
+- All operations must specify both database and collection
+
+## Success Criteria
+- All tests updated and passing with new API pattern
+- New Database class provides database-level operations
+- Client supports `client.db('name').collection('name')` chaining exclusively
+- Multiple databases can coexist on single server
+- Database isolation: operations in one database don't affect others
+- On-demand database creation (no explicit database creation required)
+- Admin operations support database-specific and server-wide operations
+- Backup and restore operations support database-specific targeting
+- Replication system handles multiple databases
+- File system organization supports multiple databases
+- Protocol requires database field in all requests
+- Comprehensive test coverage for multi-database scenarios
+- Clear error messages with database context
+- All existing functionality preserved within new API structure