@joystick.js/db-canary 0.0.0-canary.2209

package/prompts/09-api-method-chaining.md

@@ -0,0 +1,134 @@
+# Task 9: API Method Chaining Enhancement
+
+## Overview
+Update the client API to chain methods off of `collection()` instead of passing collection names as the first argument. This will provide a more intuitive and fluent API interface.
+
+## Current State
+The current API requires passing collection names as the first argument:
+```javascript
+await client.insert_one('users', { name: 'John' });
+await client.find_one('users', { name: 'John' });
+await client.update_one('users', { name: 'John' }, { $set: { age: 30 } });
+```
+
+## Target State
+The new API should allow method chaining off a collection reference:
+```javascript
+await client.collection('users').insert_one({ name: 'John' });
+await client.collection('users').find_one({ name: 'John' });
+await client.collection('users').update_one({ name: 'John' }, { $set: { age: 30 } });
+```
+
+## Requirements
+
+### 1. Collection Method
+- Add `collection(name)` method to client that returns a collection interface
+- Collection interface should provide all CRUD operations
+- Maintain backward compatibility with existing API
+
+### 2. Collection Interface Methods
+The collection interface should provide:
+- `insert_one(document, options)`
+- `find_one(filter, options)`
+- `find(filter, options)`
+- `update_one(filter, update, options)`
+- `delete_one(filter, options)`
+- `bulk_write(operations, options)`
+- `create_index(field, options)`
+- `drop_index(field)`
+- `get_indexes()`
+
+### 3. Backward Compatibility
+- Keep existing methods working (don't break existing code)
+- Both APIs should work simultaneously
+- No deprecation warnings initially
+
+### 4. Implementation Details
+- Collection interface should reuse existing client methods internally
+- No duplication of protocol logic
+- Maintain same error handling and response formats
+- Support all existing options and parameters
+
+## Technical Specifications
+
+### Collection Class Structure
+```javascript
+class Collection {
+  constructor(client, collection_name) {
+    this.client = client;
+    this.collection_name = collection_name;
+  }
+
+  async insert_one(document, options = {}) {
+    return this.client.insert_one(this.collection_name, document, options);
+  }
+
+  // ... other methods
+}
+```
+
+### Client Integration
+```javascript
+// In client class
+collection(name) {
+  return new Collection(this, name);
+}
+```
+
+## Testing Requirements
+
+### Unit Tests
+- Test collection method returns proper interface
+- Test all collection methods delegate correctly
+- Test error handling propagation
+- Test options passing
+
+### Integration Tests
+- Test both old and new API styles work
+- Test complex operations with method chaining
+- Test error scenarios with chained methods
+- Test performance impact (should be minimal)
+
+### Backward Compatibility Tests
+- Ensure all existing tests continue to pass
+- Test mixed usage of both API styles
+- Verify no breaking changes
+
+## Implementation Steps
+
+1. **Create Collection Class**
+   - Implement collection interface class
+   - Add all required methods that delegate to client
+
+2. **Update Client Class**
+   - Add `collection(name)` method
+   - Ensure proper instantiation of Collection class
+
+3. **Update Type Definitions**
+   - Add TypeScript definitions for Collection class
+   - Update client interface to include collection method
+
+4. **Add Tests**
+   - Comprehensive test suite for new API
+   - Backward compatibility verification tests
+
+5. **Update Documentation**
+   - Update README.md with new API examples
+   - Add migration guide for users wanting to switch
+
+## Success Criteria
+
+- [ ] Collection method returns functional interface
+- [ ] All CRUD operations work through collection interface
+- [ ] Existing API continues to work unchanged
+- [ ] All existing tests pass
+- [ ] New comprehensive test suite passes
+- [ ] TypeScript definitions updated
+- [ ] Documentation updated with examples
+
+## Notes
+
+- This change improves developer experience without breaking existing code
+- The fluent interface is more intuitive and follows modern API design patterns
+- Implementation should be lightweight with minimal performance overhead
+- Consider adding IDE autocomplete benefits through proper TypeScript definitions

package/prompts/10-automatic-index-creation.md

@@ -0,0 +1,223 @@
+# Task 10: Automatic Index Creation
+
+## Overview
+Implement automatic index creation based on query patterns to optimize database performance without requiring manual index management. The system should intelligently create indexes for frequently queried fields.
+
+## Current State
+- Indexes must be created manually using `create_index()`
+- No automatic optimization based on query patterns
+- Users must anticipate which fields need indexing
+- Performance degrades on unindexed queries
+
+## Target State
+- Automatic index creation based on query frequency and patterns
+- Configurable thresholds for automatic index creation
+- Smart index recommendations and creation
+- Minimal performance impact from monitoring
+
+## Requirements
+
+### 1. Query Pattern Monitoring
+- Track query patterns and field usage frequency
+- Monitor query performance metrics
+- Identify slow queries that would benefit from indexes
+- Store query statistics in memory with configurable retention
+
+### 2. Automatic Index Creation Logic
+- Create indexes automatically when query frequency exceeds threshold
+- Consider query performance impact (slow queries get priority)
+- Avoid creating redundant or unnecessary indexes
+- Respect maximum index limits per collection
+
+### 3. Configuration Options
+- Enable/disable automatic indexing globally
+- Configure frequency thresholds for index creation
+- Set performance thresholds (query time) for index creation
+- Configure maximum automatic indexes per collection
+- Whitelist/blacklist fields for automatic indexing
+
+### 4. Index Management
+- Track automatically created indexes separately from manual ones
+- Provide ability to remove automatic indexes
+- Monitor index usage and remove unused automatic indexes
+- Prevent automatic removal of manually created indexes
+
+## Technical Specifications
+
+### Query Monitoring Structure
+```javascript
+const query_stats = {
+  collection_name: {
+    field_name: {
+      query_count: 0,
+      total_time_ms: 0,
+      avg_time_ms: 0,
+      last_queried: Date,
+      slow_query_count: 0
+    }
+  }
+};
+```
+
+### Configuration Schema
+```json
+{
+  "auto_indexing": {
+    "enabled": true,
+    "frequency_threshold": 100,
+    "performance_threshold_ms": 50,
+    "max_auto_indexes_per_collection": 10,
+    "monitoring_window_hours": 24,
+    "cleanup_unused_after_hours": 168,
+    "excluded_fields": ["_id", "created_at"],
+    "included_collections": ["*"],
+    "excluded_collections": []
+  }
+}
+```
+
+### Auto Index Metadata
+```javascript
+const auto_index_metadata = {
+  collection_name: {
+    field_name: {
+      created_at: Date,
+      query_count_at_creation: 0,
+      avg_performance_improvement_ms: 0,
+      last_used: Date,
+      usage_count: 0,
+      auto_created: true
+    }
+  }
+};
+```
+
+## Implementation Details
+
+### 1. Query Interceptor
+- Intercept all find operations to collect statistics
+- Measure query execution time
+- Track field usage in query filters
+- Update query statistics in real-time
+
+### 2. Index Decision Engine
+- Analyze query statistics periodically
+- Apply thresholds to determine index candidates
+- Check for existing indexes to avoid duplicates
+- Create indexes asynchronously to avoid blocking queries
+
+### 3. Index Lifecycle Management
+- Track index usage after creation
+- Monitor performance improvements
+- Remove unused automatic indexes after configured period
+- Provide admin interface for index management
+
+### 4. Performance Considerations
+- Minimal overhead for query monitoring
+- Asynchronous index creation
+- Configurable monitoring granularity
+- Memory-efficient statistics storage
+
+## Testing Requirements
+
+### Unit Tests
+- Query statistics collection accuracy
+- Index decision logic with various thresholds
+- Configuration validation and defaults
+- Index metadata management
+
+### Integration Tests
+- End-to-end automatic index creation flow
+- Performance improvement verification
+- Index cleanup and removal
+- Configuration changes during runtime
+
+### Performance Tests
+- Monitoring overhead measurement
+- Index creation impact on concurrent operations
+- Memory usage with large query statistics
+- Cleanup performance with many automatic indexes
+
+## Implementation Steps
+
+1. **Query Monitoring Infrastructure**
+   - Add query interceptor to track field usage
+   - Implement statistics collection and storage
+   - Add performance timing measurements
+
+2. **Configuration System**
+   - Add auto-indexing configuration to settings
+   - Implement configuration validation
+   - Add runtime configuration updates
+
+3. **Index Decision Engine**
+   - Implement threshold-based decision logic
+   - Add index candidate evaluation
+   - Create asynchronous index creation system
+
+4. **Index Lifecycle Management**
+   - Track automatic vs manual indexes
+   - Implement usage monitoring
+   - Add cleanup for unused indexes
+
+5. **Admin Interface**
+   - Add endpoints for viewing auto-index status
+   - Provide manual control over automatic indexes
+   - Add query statistics viewing
+
+6. **Testing and Optimization**
+   - Comprehensive test suite
+   - Performance optimization
+   - Memory usage optimization
+
+## API Extensions
+
+### New Admin Operations
+```javascript
+// Get auto-indexing statistics
+await client.get_auto_index_stats();
+
+// Get query statistics
+await client.get_query_stats(collection_name);
+
+// Force index evaluation
+await client.evaluate_auto_indexes(collection_name);
+
+// Remove automatic indexes
+await client.remove_auto_indexes(collection_name, field_names);
+
+// Configure auto-indexing
+await client.configure_auto_indexing(config);
+```
+
+### Configuration Updates
+```javascript
+// Enable/disable auto-indexing
+await client.update_settings({
+  auto_indexing: {
+    enabled: true,
+    frequency_threshold: 50
+  }
+});
+```
+
+## Success Criteria
+
+- [ ] Query patterns are accurately monitored
+- [ ] Indexes are automatically created based on thresholds
+- [ ] Performance improvements are measurable
+- [ ] Configuration system works correctly
+- [ ] Unused automatic indexes are cleaned up
+- [ ] Admin interface provides visibility and control
+- [ ] Minimal performance overhead from monitoring
+- [ ] All existing tests continue to pass
+- [ ] Comprehensive test coverage for new features
+
+## Notes
+
+- Start with conservative thresholds to avoid over-indexing
+- Consider field cardinality when creating indexes (avoid low-cardinality fields)
+- Monitor memory usage of query statistics
+- Provide clear documentation on configuration options
+- Consider adding machine learning for more intelligent index decisions in future versions
+- Ensure automatic indexes don't interfere with manual index management

package/prompts/11-operation-naming-consistency.md

@@ -0,0 +1,268 @@
+# Task 11: Operation Naming Consistency
+
+## Overview
+Standardize the operation object structure to use snake_case naming (like `insert_one`) instead of camelCase (like `insertOne`) for consistency across the entire codebase and API. This task assumes Task 9 has been completed and only the chained API exists.
+
+## Current State (After Task 9)
+- Only chained API exists: `client.collection('users').insert_one()`
+- Traditional API has been removed/deprecated
+- Mixed naming conventions still exist in operation objects
+- Bulk operations use camelCase (`insertOne`, `updateOne`, `deleteOne`)
+- Client methods use snake_case (`insert_one`, `update_one`, `delete_one`)
+- Inconsistency creates confusion and maintenance issues
+
+## Target State
+- Consistent snake_case naming throughout the entire system
+- All operation objects use snake_case field names
+- Protocol messages use snake_case consistently
+- Chained API uses consistent naming in all operations
+
+## Requirements
+
+### 1. Operation Object Structure Updates
+- Change all camelCase operation names to snake_case
+- Update bulk_write operation structures for chained API
+- Maintain backward compatibility during transition
+- Update all internal operation handling
+
+### 2. Protocol Message Consistency
+- Ensure all TCP protocol messages use snake_case
+- Update message serialization/deserialization
+- Maintain protocol version compatibility
+- Update operation type constants
+
+### 3. Chained API Consistency
+- Ensure collection interface uses snake_case consistently
+- Update bulk operation structures for chained API
+- Align all collection methods with naming convention
+- Update TypeScript definitions for collection interface
+
+## Technical Specifications
+
+### Current Chained API (After Task 9, Mixed Naming)
+```javascript
+// Collection methods (consistent snake_case)
+await client.collection('users').insert_one(document);
+await client.collection('users').update_one(filter, update);
+await client.collection('users').delete_one(filter);
+
+// Bulk operations (inconsistent camelCase)
+await client.collection('users').bulk_write([
+  { insertOne: { document: { name: 'John' } } },  // camelCase - inconsistent
+  { updateOne: { filter: {}, update: {} } },      // camelCase - inconsistent
+  { deleteOne: { filter: {} } }                   // camelCase - inconsistent
+]);
+```
+
+### Target Chained API (Consistent snake_case)
+```javascript
+// Collection methods (consistent snake_case)
+await client.collection('users').insert_one(document);
+await client.collection('users').update_one(filter, update);
+await client.collection('users').delete_one(filter);
+
+// Bulk operations (consistent snake_case)
+await client.collection('users').bulk_write([
+  { insert_one: { document: { name: 'John' } } },  // snake_case - consistent
+  { update_one: { filter: {}, update: {} } },      // snake_case - consistent
+  { delete_one: { filter: {} } }                   // snake_case - consistent
+]);
+```
+
+### Collection Interface (From Task 9)
+```javascript
+class Collection {
+  constructor(client, collection_name) {
+    this.client = client;
+    this.collection_name = collection_name;
+  }
+
+  // CRUD methods (already snake_case)
+  async insert_one(document, options = {}) {
+    return this.client.insert_one(this.collection_name, document, options);
+  }
+
+  async find_one(filter, options = {}) {
+    return this.client.find_one(this.collection_name, filter, options);
+  }
+
+  async find(filter, options = {}) {
+    return this.client.find(this.collection_name, filter, options);
+  }
+
+  async update_one(filter, update, options = {}) {
+    return this.client.update_one(this.collection_name, filter, update, options);
+  }
+
+  async delete_one(filter, options = {}) {
+    return this.client.delete_one(this.collection_name, filter, options);
+  }
+
+  // Bulk operations
+  async bulk_write(operations, options = {}) {
+    return this.client.bulk_write(this.collection_name, operations, options);
+  }
+
+  // Index methods (already snake_case)
+  async create_index(field, options = {}) {
+    return this.client.create_index(this.collection_name, field, options);
+  }
+
+  async drop_index(field) {
+    return this.client.drop_index(this.collection_name, field);
+  }
+
+  async get_indexes() {
+    return this.client.get_indexes(this.collection_name);
+  }
+}
+```
+
+### Operation Type Constants
+```javascript
+// Current (mixed)
+const OP_TYPES = {
+  INSERT_ONE: 'insert_one',
+  BULK_WRITE: 'bulk_write',
+  // But bulk operations use: insertOne, updateOne, deleteOne
+};
+
+// Target (consistent)
+const OP_TYPES = {
+  INSERT_ONE: 'insert_one',
+  UPDATE_ONE: 'update_one',
+  DELETE_ONE: 'delete_one',
+  BULK_WRITE: 'bulk_write'
+};
+```
+
+## Implementation Details
+
+### 1. Operation Handler Updates
+- Update bulk_write operation parsing for chained API
+- Modify operation type recognition to handle snake_case
+- Update all operation processors
+- Ensure error messages use consistent naming
+
+### 2. Protocol Message Updates
+- Update TCP protocol message structures
+- Modify serialization/deserialization logic
+- Update protocol documentation
+- Maintain version compatibility
+
+### 3. Backward Compatibility Layer
+- Support both naming conventions temporarily
+- Add deprecation warnings for camelCase operations
+- Plan migration timeline for complete snake_case adoption
+
+## Testing Requirements
+
+### Unit Tests
+- Test operation name recognition with both conventions
+- Verify bulk operation parsing with new naming
+- Test protocol message serialization/deserialization
+- Validate error handling with consistent naming
+
+### Integration Tests
+- Test bulk operations with new naming through collection interface
+- Verify client-server communication with updated protocol
+- Test backward compatibility with old naming
+- Test all collection methods with consistent naming
+
+### Collection Interface Tests
+- Test bulk_write method with new operation naming
+- Verify error handling in collection interface
+- Validate TypeScript definitions for collection interface
+
+## Implementation Steps
+
+1. **Update Operation Constants**
+   - Standardize all operation type constants to snake_case
+   - Update operation type mappings
+   - Add backward compatibility mappings
+
+2. **Update Bulk Operation Parsing**
+   - Modify bulk_write operation parser
+   - Add support for both naming conventions
+   - Add deprecation warnings for camelCase
+
+3. **Update Protocol Messages**
+   - Ensure all protocol messages use snake_case
+   - Update message validation
+   - Maintain protocol version compatibility
+
+4. **Update TypeScript Definitions**
+   - Update Collection interface definitions
+   - Ensure consistent naming in type definitions
+
+5. **Add Migration Support**
+   - Create operation name translation utilities
+   - Add configuration for naming convention preference
+   - Implement gradual migration path
+
+## API Examples
+
+### Basic Collection Operations (Already Consistent)
+```javascript
+const users = client.collection('users');
+
+await users.insert_one({ name: 'John', age: 30 });
+const user = await users.find_one({ name: 'John' });
+await users.update_one({ name: 'John' }, { $set: { age: 31 } });
+await users.delete_one({ name: 'John' });
+```
+
+### Bulk Operations (Updated for Consistency)
+```javascript
+const users = client.collection('users');
+
+// Before (inconsistent)
+await users.bulk_write([
+  { insertOne: { document: { name: 'John' } } },
+  { updateOne: { filter: { name: 'Jane' }, update: { $set: { age: 25 } } } },
+  { deleteOne: { filter: { name: 'Bob' } } }
+]);
+
+// After (consistent)
+await users.bulk_write([
+  { insert_one: { document: { name: 'John' } } },
+  { update_one: { filter: { name: 'Jane' }, update: { $set: { age: 25 } } } },
+  { delete_one: { filter: { name: 'Bob' } } }
+]);
+```
+
+### Index Operations (Already Consistent)
+```javascript
+const users = client.collection('users');
+
+await users.create_index('email', { unique: true });
+const indexes = await users.get_indexes();
+await users.drop_index('email');
+```
+
+## Success Criteria
+
+- [ ] All operation objects use consistent snake_case naming
+- [ ] Protocol messages use snake_case throughout
+- [ ] Collection interface maintains snake_case consistency
+- [ ] Backward compatibility works during transition
+- [ ] Deprecation warnings are properly displayed
+- [ ] All existing tests pass with new naming
+- [ ] New tests cover both naming conventions
+- [ ] TypeScript definitions are updated for collection interface
+- [ ] Documentation reflects consistent naming
+- [ ] Migration path is clear and functional
+
+## Dependencies
+
+- **Task 9**: API Method Chaining Enhancement must be completed first
+- This task assumes only the chained API exists after Task 9
+
+## Notes
+
+- This change improves code consistency and maintainability
+- The transition should be gradual to avoid breaking existing applications
+- Clear communication about the change is essential
+- Consider providing automated migration tools for applications using old bulk operation naming
+- Monitor community feedback during the transition period
+- Ensure all team members understand the new naming convention