@joystick.js/db-canary 0.0.0-canary.2209

package/prompts/12-tcp-replication-system.md

@@ -0,0 +1,333 @@
+# Task 12: TCP Replication System
+
+## Overview
+Implement TCP-based replication to read-only secondary nodes to provide high availability and read scaling. The system should track secondary nodes in settings, replicate data changes over secure TCP connections, and forward write requests from secondaries to the primary.
+
+## Current State
+- Single primary node handles all reads and writes
+- No replication or high availability features
+- No read scaling capabilities
+- All data exists only on primary node
+
+## Target State
+- Primary node replicates data to multiple read-only secondary nodes
+- Secondary nodes configured in settings.json with connection details
+- Secure TCP connections using private keys for authentication
+- Write requests to secondaries are forwarded to primary
+- Read operations can be served by secondary nodes
+- Automatic failover detection and reconnection
+
+## Requirements
+
+### 1. Secondary Node Configuration
+- Track secondary nodes in settings.json configuration
+- Support multiple secondary nodes with IP, port, and private key
+- Validate secondary node configurations on startup
+- Support dynamic addition/removal of secondary nodes
+
+### 2. Replication Protocol
+- Replicate all write operations from primary to secondary nodes
+- Ensure data consistency across all nodes
+- Handle replication failures gracefully
+- Support batch replication for performance
+
+### 3. Write Forwarding
+- Secondary nodes forward write requests to primary
+- Maintain client connection while forwarding request
+- Return primary's response to original client
+- Handle forwarding failures and timeouts
+
+### 4. Security and Authentication
+- Use private key authentication for secondary connections
+- Encrypt replication traffic
+- Validate secondary node identity
+- Secure write forwarding between secondary and primary
+
+### 5. Connection Management
+- Maintain persistent connections to secondary nodes
+- Automatic reconnection on connection failures
+- Health monitoring of secondary nodes
+- Connection pooling for multiple secondaries
+
+## Technical Specifications
+
+### Settings Configuration
+```json
+{
+  "replication": {
+    "enabled": true,
+    "mode": "async",
+    "timeout_ms": 5000,
+    "retry_attempts": 3,
+    "batch_size": 100,
+    "secondaries": [
+      {
+        "id": "secondary-1",
+        "ip": "192.168.1.100",
+        "port": 1984,
+        "private_key": "<base64-encoded-private-key>",
+        "enabled": true
+      },
+      {
+        "id": "secondary-2", 
+        "ip": "192.168.1.101",
+        "port": 1984,
+        "private_key": "<base64-encoded-private-key>",
+        "enabled": true
+      }
+    ]
+  }
+}
+```
+
+### Replication Message Protocol
+```javascript
+const replication_message = {
+  type: 'replication',
+  timestamp: Date.now(),
+  sequence_number: 12345,
+  operations: [
+    {
+      operation: 'insert_one',
+      collection: 'users',
+      data: { document: { name: 'John' } },
+      transaction_id: 'tx_123'
+    },
+    {
+      operation: 'update_one',
+      collection: 'users', 
+      data: { filter: { name: 'John' }, update: { $set: { age: 30 } } },
+      transaction_id: 'tx_124'
+    }
+  ]
+};
+```
+
+### Write Forwarding Protocol
+```javascript
+const forward_request = {
+  type: 'forward_write',
+  original_client_id: 'client_123',
+  operation: 'insert_one',
+  collection: 'users',
+  data: { document: { name: 'John' } },
+  forwarded_by: 'secondary-1',
+  timestamp: Date.now()
+};
+```
+
+### Secondary Node Authentication
+```javascript
+const auth_message = {
+  type: 'replication_auth',
+  node_id: 'secondary-1',
+  timestamp: Date.now(),
+  signature: '<signed-with-private-key>'
+};
+```
+
+## Implementation Details
+
+### 1. Primary Node Components
+
+#### Replication Manager
+- Manage connections to all secondary nodes
+- Queue and batch replication operations
+- Handle replication failures and retries
+- Monitor secondary node health
+
+#### Write Operation Interceptor
+- Intercept all write operations on primary
+- Queue operations for replication
+- Ensure operations are replicated before confirming success
+- Handle replication acknowledgments
+
+### 2. Secondary Node Components
+
+#### Write Forwarder
+- Detect write operations from clients
+- Forward write requests to primary node
+- Maintain client connection during forwarding
+- Return primary's response to original client
+
+#### Replication Receiver
+- Accept replication messages from primary
+- Apply replicated operations to local database
+- Maintain data consistency with primary
+- Handle replication sequence ordering
+
+#### Read Handler
+- Serve read-only operations locally
+- Ensure read consistency
+- Handle read preference routing
+
+### 3. Connection Management
+- Maintain persistent TCP connections between nodes
+- Handle connection lifecycle (connect, disconnect, reconnect)
+- Implement connection timeouts and health checks
+- Load balance connections across multiple secondaries
+
+### 4. Security Layer
+- Implement private key authentication
+- Encrypt replication and forwarding messages
+- Validate message integrity
+- Prevent replay attacks
+
+## Testing Requirements
+
+### Unit Tests
+- Test replication manager functionality
+- Verify write forwarding logic
+- Test connection pool management
+- Test operation interception and queuing
+- Validate security and authentication
+
+### Integration Tests
+- Test end-to-end replication flow
+- Test write forwarding from secondary to primary
+- Verify data consistency across nodes
+- Test connection failure scenarios
+- Validate security protocols
+
+### Performance Tests
+- Measure replication latency
+- Test write forwarding performance
+- Test with high write volumes
+- Verify batch replication performance
+- Test connection pool efficiency
+
+## Implementation Steps
+
+1. **Configuration System**
+   - Add replication configuration to settings
+   - Implement configuration validation
+   - Add runtime configuration updates
+
+2. **Connection Management**
+   - Implement TCP connection pool for secondaries
+   - Add connection health monitoring
+   - Implement automatic reconnection logic
+
+3. **Security Layer**
+   - Implement private key authentication
+   - Add message encryption/decryption
+   - Implement message signing and verification
+
+4. **Primary Node Replication**
+   - Create replication operation queue
+   - Implement batch replication logic
+   - Add failure handling and retries
+   - Intercept write operations
+
+5. **Secondary Node Implementation**
+   - Implement secondary node server mode
+   - Add write forwarding to primary
+   - Implement replication message processing
+   - Add read-only operation handling
+
+6. **Write Forwarding System**
+   - Detect write operations on secondary
+   - Forward requests to primary
+   - Handle forwarding responses
+   - Maintain client connections during forwarding
+
+## API Extensions
+
+### Admin Operations
+```javascript
+// Get replication status
+await client.get_replication_status();
+
+// Add secondary node
+await client.add_secondary({
+  id: 'secondary-3',
+  ip: '192.168.1.102', 
+  port: 1984,
+  private_key: '<key>'
+});
+
+// Remove secondary node
+await client.remove_secondary('secondary-3');
+
+// Force replication sync
+await client.sync_secondaries();
+
+// Get secondary node health
+await client.get_secondary_health();
+```
+
+### Configuration Updates
+```javascript
+// Enable/disable replication
+await client.update_settings({
+  replication: {
+    enabled: true,
+    mode: 'sync'
+  }
+});
+```
+
+## Secondary Node Setup
+
+### Secondary Node Configuration
+```json
+{
+  "mode": "secondary",
+  "primary": {
+    "ip": "192.168.1.10",
+    "port": 1983,
+    "public_key": "<primary-public-key>"
+  },
+  "replication": {
+    "port": 1984,
+    "private_key": "<secondary-private-key>",
+    "read_only": true,
+    "forward_writes": true
+  }
+}
+```
+
+### Secondary Node Operations
+- Accept client connections for read and write operations
+- Serve read operations locally
+- Forward write operations to primary
+- Accept replication connections from primary
+- Apply replicated operations to local database
+- Report health status to primary
+
+## Write Forwarding Flow
+
+1. **Client connects to secondary** and sends write request
+2. **Secondary detects write operation** and maintains client connection
+3. **Secondary forwards request to primary** with forwarding metadata
+4. **Primary processes write operation** and replicates to all secondaries
+5. **Primary sends response back to forwarding secondary**
+6. **Secondary returns response to original client**
+7. **Secondary receives replicated operation** and applies it locally
+
+## Success Criteria
+
+- [ ] Secondary nodes can be configured in settings
+- [ ] Primary replicates write operations to secondaries
+- [ ] Secondaries forward write requests to primary
+- [ ] Write forwarding maintains client connections
+- [ ] Secure authentication using private keys works
+- [ ] Connection management handles failures gracefully
+- [ ] Data consistency is maintained across nodes
+- [ ] Read operations work on secondary nodes
+- [ ] Admin interface provides replication monitoring
+- [ ] Performance impact on primary is minimal
+- [ ] All existing tests continue to pass
+- [ ] Comprehensive test coverage for replication features
+
+## Notes
+
+- Write forwarding allows clients to connect to any node
+- Secondary nodes appear as full database servers to clients
+- Replication ensures all nodes have consistent data
+- Start with asynchronous replication for better performance
+- Monitor replication lag and provide alerts
+- Implement proper logging for replication and forwarding events
+- Consider compression for replication messages over slow networks
+- Plan for future features like automatic failover
+- Ensure forwarding doesn't create circular request loops

package/prompts/13-master-read-write-operations.md

@@ -0,0 +1,57 @@
+# Task 13: Master Node Read and Write Operations
+
+## Objective
+Ensure that master nodes in the cluster can handle both read and write operations, not just writes. Currently, the master may be configured to only accept writes, but it should handle all operations.
+
+## Current State
+- Cluster system with master/worker architecture implemented
+- Master nodes may be restricted to write operations only
+- Secondary/worker nodes are read-only
+- 423 passing tests from previous tasks
+
+## Requirements
+
+### 1. Master Node Operation Handling
+- Verify master nodes can handle all operations:
+  - **Read operations**: `find`, `find_one`, `get_indexes`
+  - **Write operations**: `insert_one`, `update_one`, `delete_one`, `bulk_write`
+  - **Index operations**: `create_index`, `drop_index`
+  - **Admin operations**: All admin commands
+- Remove any restrictions that prevent masters from handling reads
+
+### 2. Operation Routing Review
+- Review `src/server/cluster/master.js` for operation handling
+- Ensure master processes both read and write operations locally
+- Verify no artificial restrictions on read operations for master nodes
+- Maintain existing write forwarding from workers to master
+
+### 3. Load Balancing Considerations
+- Masters should handle operations directly when they are the target node
+- Maintain proper load distribution between master and worker nodes
+- Ensure read operations can be served by both master and workers appropriately
+
+### 4. Configuration Validation
+- Review cluster configuration to ensure masters are not write-only
+- Verify operation dispatcher routes all operation types to master correctly
+- Check that master nodes have full database access permissions
+
+### 5. Testing and Validation
+- Add tests to verify master nodes handle read operations
+- Test mixed read/write workloads on master nodes
+- Ensure no performance degradation from handling both operation types
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Maintain all existing tests (423 tests must continue passing)
+- Add tests specifically for master node read operations
+- Ensure no breaking changes to existing cluster functionality
+- Secondary nodes remain read-only as designed
+
+## Success Criteria
+- All existing tests pass (423 tests)
+- Master nodes handle both read and write operations
+- No artificial restrictions on master read operations
+- New tests verify master read operation handling
+- Cluster load balancing works correctly
+- No performance regression for mixed workloads
+- Secondary nodes remain properly read-only

package/prompts/14-index-upsert-operations.md

@@ -0,0 +1,68 @@
+# Task 14: Index Upsert Operations
+
+## Objective
+Implement index upsert functionality where indexes are created if they don't exist, or updated if they do exist, providing a seamless "create or update" operation for index management.
+
+## Current State
+- Secondary indexing system implemented with `create_index` and `drop_index` operations
+- Current `create_index` fails if index already exists
+- Index management requires manual checking of existing indexes
+- 423 passing tests from previous tasks
+
+## Requirements
+
+### 1. Upsert Index Operation
+- Modify `create_index` operation to support upsert behavior
+- Add optional `upsert` parameter (default: false for backward compatibility)
+- When `upsert: true`:
+  - Create index if it doesn't exist
+  - Update index definition if it exists with different configuration
+  - Return success in both cases
+- When `upsert: false` (default): maintain current behavior (fail if exists)
+
+### 2. Index Comparison Logic
+- Implement index definition comparison to detect changes
+- Compare index fields, types, and configuration options
+- Determine if existing index needs to be updated or can be left as-is
+- Handle cases where index structure is incompatible (require drop/recreate)
+
+### 3. Update Index Implementation
+- For compatible changes: update index metadata without rebuilding
+- For incompatible changes: drop existing index and create new one
+- Maintain data integrity during index updates
+- Ensure atomic operations (all or nothing)
+
+### 4. API Enhancement
+- Update `src/server/lib/operations/create_index.js` with upsert logic
+- Add validation for upsert parameter
+- Enhance error handling for upsert scenarios
+- Maintain backward compatibility with existing create_index calls
+
+### 5. Client Library Integration
+- Update client library to support upsert parameter
+- Add convenience method `upsert_index()` for explicit upsert operations
+- Update TypeScript definitions for new functionality
+- Provide clear documentation for upsert behavior
+
+### 6. Performance Considerations
+- Optimize index existence checking
+- Minimize index rebuilding when possible
+- Add logging for upsert operations (created vs updated)
+- Monitor performance impact of upsert operations
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Maintain all existing tests (423 tests must continue passing)
+- Add comprehensive tests for upsert functionality
+- Ensure backward compatibility with existing index operations
+- Handle edge cases like concurrent index operations
+
+## Success Criteria
+- All existing tests pass (423 tests)
+- `create_index` with `upsert: true` creates or updates indexes
+- Backward compatibility maintained for existing `create_index` calls
+- New tests for all upsert scenarios (create, update, no-change)
+- Proper error handling for incompatible index changes
+- Client library supports upsert operations
+- Performance monitoring shows minimal impact
+- Atomic operations ensure data integrity

package/prompts/15-client-api-return-types.md

@@ -0,0 +1,81 @@
+# Task 15: Client API Return Types Enhancement
+
+## Objective
+Update the client API so that `find()` returns an array of matching documents and `find_one()` returns a single document (or null), providing more intuitive return types for developers.
+
+## Current State
+- Client library implemented with method chaining API
+- Current `find()` and `find_one()` may return wrapped responses
+- API method chaining allows `client.collection('name').find()`
+- 423 passing tests from previous tasks
+
+## Requirements
+
+### 1. find() Method Enhancement
+- Update `find()` to return an array of actual documents
+- Remove any response wrapping for document data
+- Return empty array `[]` when no documents match
+- Maintain query filtering and options support
+- Example expected behavior:
+  ```javascript
+  const client = joystickdb.client();
+  const db = client.collection('example');
+  
+  const docs = db.find({ status: 'active' });
+  // docs === [{ _id: '1', status: 'active', name: 'John' }, { _id: '2', status: 'active', name: 'Jane' }]
+  ```
+
+### 2. find_one() Method Enhancement
+- Update `find_one()` to return a single document object or null
+- Return the actual document, not wrapped in response object
+- Return `null` when no document matches the query
+- Maintain query filtering and options support
+- Example expected behavior:
+  ```javascript
+  const client = joystickdb.client();
+  const db = client.collection('example');
+  
+  const doc = db.find_one({ _id: '123' });
+  // doc === { _id: '123', name: 'John', status: 'active' } or null
+  ```
+
+### 3. Response Processing
+- Update `src/client/index.js` to process server responses appropriately
+- Extract document data from server response format
+- Handle error cases and maintain proper error propagation
+- Ensure consistent behavior across different query scenarios
+
+### 4. Error Handling
+- Maintain proper error handling for network issues
+- Distinguish between "no results" and actual errors
+- Preserve error messages and codes from server
+- Ensure errors are thrown appropriately, not returned as values
+
+### 5. Backward Compatibility Considerations
+- Document any breaking changes in API behavior
+- Consider if existing code might depend on current return format
+- Provide migration guidance if needed
+- Update TypeScript definitions to reflect new return types
+
+### 6. Performance Optimization
+- Minimize response processing overhead
+- Ensure efficient array/object creation for results
+- Maintain streaming capabilities where applicable
+- Optimize memory usage for large result sets
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Maintain all existing tests (423 tests must continue passing)
+- Update tests to expect new return formats
+- Ensure client-server protocol remains compatible
+- Update TypeScript definitions for new return types
+
+## Success Criteria
+- All existing tests pass with updated expectations (423 tests)
+- `find()` returns array of actual documents
+- `find_one()` returns single document or null
+- No response wrapping for document data
+- Proper error handling maintained
+- TypeScript definitions updated
+- Performance remains optimal
+- Clear documentation of API changes

package/prompts/16-server-setup-ui.md

@@ -0,0 +1,97 @@
+# Task 16: Server Setup UI
+
+## Objective
+Implement a minimalist web UI hosted by the server that provides a secure setup interface using a randomly generated token, eliminating the need to connect and disconnect a separate client for initial setup while preventing unauthorized access.
+
+## Current State
+- Server runs on TCP port 1983 with MessagePack protocol
+- Setup command exists for password generation and initial configuration
+- Currently requires separate client connection to run setup
+- 423 passing tests from previous tasks
+
+## Requirements
+
+### 1. HTTP Server Integration
+- Add HTTP server capability to existing TCP server
+- Host HTTP server on port 1984 (TCP + 1) by default
+- Keep existing TCP server on port 1983 unchanged
+- Ensure HTTP server doesn't interfere with TCP operations
+
+### 2. Setup Token Security
+- On server startup, check if setup has been completed
+- If not setup, generate random setup token (UUID or similar)
+- Log clear setup message to console:
+  ```
+  JoystickDB Setup Required
+  Visit: http://localhost:1984/setup?token=abc123def456
+  ```
+- Store token temporarily in memory (not persisted to disk)
+- Token expires after first successful setup or server restart
+
+### 3. Setup Endpoint with Token Validation
+- Add HTTP endpoint `/setup` that requires valid token parameter
+- Validate token matches the generated setup token
+- Only allow setup when valid token is provided
+- Call existing setup command after token validation
+- Return JSON response with setup results and configuration details
+
+### 4. Minimalist Setup UI
+- Create simple HTML page served at `/setup?token=<valid_token>`
+- Plain text styling - no fancy CSS or JavaScript frameworks
+- Display setup form or confirmation button
+- Show setup results directly in the browser after execution
+- Include clear instructions about next steps
+
+### 5. Response Format
+- Display JSON response in readable format in browser
+- Include instructions like:
+  ```
+  Setup completed successfully!
+  
+  Generated password: [password_here]
+  
+  Next steps:
+  1. Set environment variable: JOYSTICKDB_PASSWORD=[password_here]
+  2. Restart your application
+  3. Connect using the TCP client on port 1983
+  
+  Configuration saved to: [path_to_auth_file]
+  ```
+
+### 6. Security Considerations
+- Only allow setup when no authentication is configured
+- Require valid setup token for all setup operations
+- Token prevents unauthorized bots from hijacking setup
+- Add basic rate limiting to prevent token brute force attempts
+- Log all setup attempts for security monitoring
+
+### 7. Error Handling
+- Handle cases where setup has already been completed
+- Provide clear error messages for invalid or missing tokens
+- Display helpful troubleshooting information
+- Ensure graceful degradation if HTTP server fails to start
+
+### 8. Configuration Options
+- Allow HTTP port configuration via settings
+- Option to disable HTTP server entirely if not needed
+- Maintain backward compatibility with existing configurations
+- Document new HTTP server settings
+
+## Implementation Notes
+- Follow existing code patterns: ESM syntax, snake_case, arrow functions
+- Maintain all existing tests (423 tests must continue passing)
+- Add tests for HTTP server, token generation, and setup endpoint
+- Keep HTTP server lightweight and minimal
+- Ensure no dependencies on external web frameworks
+
+## Success Criteria
+- All existing tests pass (423 tests)
+- HTTP server runs on port 1984 alongside TCP server
+- Setup token generated and logged on startup when needed
+- Setup UI accessible only with valid token
+- Setup endpoint successfully calls existing setup command
+- JSON response displayed clearly in browser
+- Clear instructions provided for environment variable setup
+- Security token prevents unauthorized setup attempts
+- HTTP server can be disabled via configuration
+- No impact on existing TCP server functionality