@joystick.js/db-canary 0.0.0-canary.2212 → 0.0.0-canary.2216

package/prompts/PROMPTS.md

@@ -1,158 +0,0 @@
-# Prompt 1: Build JoystickDB TCP Server (Production-Grade, Secure, Reliable)
-
-You are building a Node.js app that acts as the **JoystickDB TCP server**.
-
-Requirements:
-- Use ESM syntax (`import`).
-- Follow coding conventions: snake_case for variables/functions/keys, arrow functions preferred.
-- Install dependencies: `lmdb`, `msgpack-lite`, `bcrypt`, `@aws-sdk/client-s3`, `crypto`, plus standard Node libs.
-- The server should:
-
-### Core
-1. **Auto-calculate map_size at bootstrap**:  
-   - Detect total disk size (e.g. via `df -k .`).  
-   - Set `map_size = disk_size * 0.8`.  
-   - Example: 3 TB disk → map_size = ~2.4 TB.  
-   - This ensures JoystickDB never consumes 100% of disk, leaving margin for OS, swap, logs, and backups.  
-   - Map_size must be **grown automatically** if data approaches 80% of current map_size (double it, capped at 80% of total disk).  
-2. Open an LMDB environment at `./data` with that map_size.  
-3. Start a TCP server on port `6000` (Node `net` module).  
-4. Each message = 4-byte length prefix + MessagePack payload.  
-5. Decode MessagePack payload into JSON-like object:  
-   ```json
-   {
-     "op": "auth" | "find_one" | "find" | "insert_one" | "update_one" | "delete_one" | "bulk_write" | "admin" | "ping",
-     "collection": "string",
-     "filter": { ... },
-     "update": { ... },
-     "options": { ... },
-     "operations": [ ... ],
-     "admin_action": "stats" | "list_collections" | "create_index" | "drop_index" | "get_indexes" | "list_documents" | "get_document" | "query_documents" | "insert_document" | "update_document" | "delete_document" | "backup_now" | "restore_backup" | "list_backups",
-     "password": "string"
-   }
-   ```
-
-### Query Language
-6. Implement a **MongoDB-style query language** (1:1 with MongoDB except no aggregation pipeline).  
-7. **Secondary indexing**:  
-   - Maintain sub-databases for indexed fields.  
-   - Keys: `index:<collection>:<field>:<value> → [document_ids...]`.  
-   - Update indexes on insert/update.  
-   - Use indexes for queries when filters match indexed fields.  
-
-### Writes
-8. **Bulk writes**:  
-   - Accept array of insert/update/delete ops.  
-   - Run in a single LMDB write transaction (atomic).  
-   - Roll back entire batch if any fail.  
-9. **Write serialization**:  
-   - Only one LMDB write transaction at a time.  
-   - Queue concurrent writes.  
-
-### Authentication
-10. **Auth flow**:  
-    - Bootstrap: provision 32-char password via SSH.  
-    - Store `bcrypt.hash(password)` in `./auth.json`.  
-    - First op must be `{ op: "auth", password: "..." }`.  
-    - Rate-limit failures (5/min per IP).  
-
-### Admin Operations
-11. **Admin actions**:  
-    - Metadata: `stats`, `list_collections`, `create_index`, `drop_index`, `get_indexes`.  
-    - Data: `list_documents`, `get_document`, `query_documents`, `insert_document`, `update_document`, `delete_document`.  
-    - Backups: `backup_now`, `list_backups`, `restore_backup`.  
-
-### Backups & Restore
-12. **Automatic backups**:  
-    - If S3 configured in `settings.json`, run `backup_now` hourly.  
-    - Store compressed sparse backup (`tar --sparse`) in `./backups` and S3.  
-    - Retention: keep 24 hourly + 30 daily, rotate older.  
-    - Backup includes SHA256 checksum file.  
-13. **restore_from one-shot**:  
-    - If `settings.json` has `"restore_from": "<backup_file>"`:  
-      - Download from S3 (or local).  
-      - Verify SHA256 checksum.  
-      - Replace `./data/data.mdb`.  
-      - If restore fails: log + boot fresh LMDB.  
-      - If restore succeeds: rewrite `settings.json` without `restore_from`.  
-
-### Safety
-14. **Health**:  
-    - `ping` always returns `{ ok: 1 }`.  
-    - `stats` includes: memory, disk, ops/sec, doc count, map_size usage %.  
-15. **Timeouts**:  
-    - Max 5s per request.  
-16. **Connections**:  
-    - Max ~1000 connections.  
-    - Drop idle after 10m.  
-17. **Logging**:  
-    - Structured logs: `{ ts, client_id, op, collection, duration_ms, status }`.  
-18. **Protocol versioning**:  
-    - After auth: `{ ok: 1, version: "1.0.0" }`.  
-    - API never changes; future changes must remain backward-compatible.  
-
----
-
-# Prompt 2: Build JoystickDB TCP Client
-
-- ESM + snake_case.  
-- Install `msgpack-lite`.  
-- Export `create_client({ host, port, password })`.  
-- On connect: send `{ op: "auth", password: "..." }`.  
-- Methods:  
-  - CRUD: `insert_one`, `find_one`, `find`, `update_one`, `delete_one`.  
-  - Bulk: `bulk_write`.  
-  - Admin Metadata: `get_stats`, `list_collections`, `create_index`, `drop_index`, `get_indexes`.  
-  - Admin Data: `list_documents`, `get_document`, `query_documents`, `insert_document`, `update_document`, `delete_document`.  
-  - Backups: `backup_now`, `list_backups`, `restore_backup`.  
-  - Health: `ping`.  
-- Per-request timeout: 5s.  
-- Auto reconnect + queueing.  
-
----
-
-# Prompt 3: NGINX TLS Termination
-
-```nginx
-stream {
-  upstream joystickdb_backend {
-    server 127.0.0.1:6000;
-  }
-
-  server {
-    listen 443 ssl;
-    proxy_pass joystickdb_backend;
-
-    ssl_certificate     /etc/letsencrypt/live/db.cheatcode.co/fullchain.pem;
-    ssl_certificate_key /etc/letsencrypt/live/db.cheatcode.co/privkey.pem;
-
-    ssl_protocols TLSv1.3;
-    ssl_ciphers HIGH:!aNULL:!MD5;
-    ssl_prefer_server_ciphers on;
-    ssl_session_cache shared:SSL:10m;
-    ssl_session_timeout 10m;
-  }
-}
-```
-
----
-
-# Prompt 4: settings.json Schema
-
-```json
-{
-  "s3": {
-    "bucket": "joystickdb-backups",
-    "region": "us-east-1",
-    "access_key": "AKIA...",
-    "secret_key": "abcd...",
-    "endpoint": "https://s3.amazonaws.com"
-  },
-  "backup_schedule": "hourly",
-  "restore_from": "joystickdb-backup-2025-08-28T12:00:00.mdb"
-}
-```
-
-- `s3`: optional.  
-- `backup_schedule`: default hourly if omitted.  
-- `restore_from`: one-shot restore, auto-cleared after success.  

package/prompts/README.md

@@ -1,221 +0,0 @@
-# JoystickDB Implementation Tasks
-
-This directory contains detailed prompts for implementing the complete production-grade JoystickDB TCP server. Each task builds upon the previous ones and should be completed in order.
-
-## Current Status
-✅ **COMPLETED**: Basic query engine with CRUD operations, authentication, and admin/reload operations (119 tests passing)
-✅ **COMPLETED**: Task 1 - Core Infrastructure Upgrades (153 tests passing)
-✅ **COMPLETED**: Task 2 - Secondary Indexing System (196 tests passing)
-✅ **COMPLETED**: Task 3 - Write Serialization and Queuing (214 tests passing)
-✅ **COMPLETED**: Task 4 - Enhanced Authentication System (251 tests passing)
-✅ **COMPLETED**: Task 5 - Comprehensive Admin Operations (266 tests passing)
-✅ **COMPLETED**: Task 6 - Backup and Restore System (296 tests passing)
-✅ **COMPLETED**: Task 7 - Production Safety Features (316 tests passing)
-✅ **COMPLETED**: Task 8 - TCP Client Library (All original tasks complete)
-
-## Next Phase: API and Infrastructure Enhancements
-✅ **COMPLETED**: Task 9 - API Method Chaining Enhancement (347 tests passing)
-✅ **COMPLETED**: Task 10 - Automatic Index Creation (381 tests passing)
-✅ **COMPLETED**: Task 11 - Operation Naming Consistency (383 tests passing)
-✅ **COMPLETED**: Task 12 - TCP Replication System (423 tests passing)
-
-## Phase 3: Core Improvements and User Experience
-✅ **COMPLETED**: Task 13 - Master Node Read and Write Operations (420 tests passing)
-✅ **COMPLETED**: Task 14 - Index Upsert Operations (428 tests passing)
-✅ **COMPLETED**: Task 15 - Client API Return Types Enhancement (428 tests passing)
-✅ **COMPLETED**: Task 16 - Server Setup UI (451 tests passing - Test Suite Stabilized)
-✅ **COMPLETED**: Task 17 - Emergency Password Change System (474 tests passing)
-
-## All Core Tasks Complete
-🎉 **ALL TASKS COMPLETED**: JoystickDB production-grade implementation is complete with comprehensive emergency password recovery system.
-
-## Task Overview
-
-### 1. Core Infrastructure Upgrades
-**File**: `01-core-infrastructure.md`
-**Priority**: High
-**Dependencies**: None
-**Description**: Implement auto-calculating LMDB map_size, proper TCP protocol with length prefixing, connection limits, timeouts, and enhanced logging.
-
-### 2. Secondary Indexing System
-**File**: `02-secondary-indexing.md`
-**Priority**: High
-**Dependencies**: Task 1
-**Description**: Build comprehensive indexing system for query optimization with index management operations and automatic index maintenance.
-
-### 3. Write Serialization and Queuing
-**File**: `03-write-serialization.md`
-**Priority**: High
-**Dependencies**: Task 1
-**Description**: Implement write queue system to ensure only one LMDB write transaction at a time with proper concurrent read support.
-
-### 4. Enhanced Authentication System
-**File**: `04-enhanced-authentication.md`
-**Priority**: Medium
-**Dependencies**: Task 1
-**Description**: Add setup command for password generation, rate limiting, protocol versioning, and secure authentication.json file management.
-
-### 5. Comprehensive Admin Operations
-**File**: `05-comprehensive-admin-operations.md`
-**Priority**: Medium
-**Dependencies**: Tasks 1, 2
-**Description**: Expand admin operations with metadata management, data operations, and system monitoring capabilities.
-
-### 6. Backup and Restore System
-**File**: `06-backup-and-restore-system.md`
-**Priority**: Medium
-**Dependencies**: Tasks 1, 5
-**Description**: Implement S3-integrated backup system with automatic scheduling, retention management, and one-shot restore capabilities.
-
-### 7. Production Safety Features
-**File**: `07-production-safety-features.md`
-**Priority**: High
-**Dependencies**: Tasks 1, 3
-**Description**: Add comprehensive health monitoring, request timeouts, connection management, and enhanced structured logging.
-
-### 8. TCP Client Library
-**File**: `08-tcp-client-library.md`
-**Priority**: Medium
-**Dependencies**: Tasks 1, 4, 5
-**Description**: Create Node.js client library with auto-reconnection, full operation support, and comprehensive error handling.
-
-### 9. API Method Chaining Enhancement
-**File**: `09-api-method-chaining.md`
-**Priority**: High
-**Dependencies**: Task 8
-**Description**: Update the client API to chain methods off of `collection()` instead of passing collection names as the first argument, providing a more intuitive and fluent API interface.
-
-### 10. Automatic Index Creation
-**File**: `10-automatic-index-creation.md`
-**Priority**: Medium
-**Dependencies**: Task 2
-**Description**: Implement automatic index creation based on query patterns to optimize database performance without requiring manual index management.
-
-### 11. Operation Naming Consistency
-**File**: `11-operation-naming-consistency.md`
-**Priority**: Medium
-**Dependencies**: Task 9
-**Description**: Standardize the operation object structure to use snake_case naming consistently across the entire codebase and API.
-
-### 12. TCP Replication System
-**File**: `12-tcp-replication-system.md`
-**Priority**: High
-**Dependencies**: Tasks 1, 8
-**Description**: Implement TCP-based replication to read-only secondary nodes with write forwarding, providing high availability and read scaling capabilities.
-
-### 13. Master Node Read and Write Operations
-**File**: `13-master-read-write-operations.md`
-**Priority**: Medium
-**Dependencies**: Task 12
-**Description**: Ensure that master nodes in the cluster can handle both read and write operations, not just writes, improving load distribution and eliminating artificial restrictions.
-
-### 14. Index Upsert Operations
-**File**: `14-index-upsert-operations.md`
-**Priority**: Medium
-**Dependencies**: Task 2
-**Description**: Implement index upsert functionality where indexes are created if they don't exist or updated if they do exist, providing seamless "create or update" operations for index management.
-
-### 15. Client API Return Types Enhancement
-**File**: `15-client-api-return-types.md`
-**Priority**: High
-**Dependencies**: Task 9
-**Description**: Update the client API so that `find()` returns an array of matching documents and `find_one()` returns a single document (or null), providing more intuitive return types for developers.
-
-### 16. Server Setup UI
-**File**: `16-server-setup-ui.md`
-**Priority**: Medium
-**Dependencies**: Task 4
-**Description**: Implement a minimalist web UI hosted by the server that provides a secure setup interface using randomly generated tokens, eliminating the need to connect a separate client for initial setup.
-
-### 17. Emergency Password Change System
-**File**: `17-emergency-password-change.md`
-**Priority**: High
-**Dependencies**: Tasks 4, 16
-**Description**: Implement a secure emergency password change system using CLI-generated recovery tokens, allowing administrators to quickly change passwords in case of security breaches with minimal downtime.
-
-## Implementation Guidelines
-
-### Code Standards
-- Use ESM syntax (`import`/`export`)
-- Follow snake_case for variables, functions, and keys
-- Prefer arrow functions
-- Two-space indentation
-- Minimal comments (only when necessary for clarity)
-
-### Testing Requirements
-- All existing tests must continue to pass (currently 119 tests)
-- Add comprehensive test suites for each new feature
-- Use AVA test runner with `--serial` flag
-- Include both unit and integration tests
-
-### Dependencies
-- Keep existing: `lmdb`, `msgpackr`, `bcrypt`
-- Add as needed: `@aws-sdk/client-s3`, `crypto`
-- Maintain compatibility with existing codebase
-
-## Getting Started
-
-1. Choose a task based on priority and dependencies
-2. Read the detailed prompt in the corresponding `.md` file
-3. Create a new branch for the task (see Git Workflow below)
-4. Implement the features following the specifications
-5. Ensure all existing tests continue to pass
-6. Add comprehensive tests for new functionality
-7. Debug and fix any failing tests (no shortcuts or hacks allowed)
-8. Merge branch to main only when all tests pass
-9. Move to the next task
-
-## Git Workflow
-
-Each task must be implemented in a separate branch to maintain code quality and enable proper review:
-
-### Branch Management
-- **Branch Naming**: Use format `task-N-description` (e.g., `task-1-core-infrastructure`)
-- **Create Branch**: `git checkout -b task-N-description` from main
-- **Work Isolation**: All development for a task happens in its dedicated branch
-- **No Partial Merges**: Only merge when task is 100% complete with all tests passing
-
-### Development Process
-1. **Start Task**: Create new branch from main
-2. **Implement Features**: Follow task specifications exactly
-3. **Write Tests**: Comprehensive test coverage for all new functionality
-4. **Debug Thoroughly**: Fix all failing tests without shortcuts or hacks
-5. **Verify Quality**: Run full test suite (`npm test`) - all 119+ tests must pass
-6. **Merge to Main**: Only when everything works perfectly
-
-### Quality Gates
-- ✅ All existing tests continue to pass (119 baseline tests)
-- ✅ New comprehensive tests for implemented features
-- ✅ No failing tests (debug and fix, don't skip or hack)
-- ✅ Code follows established patterns and standards
-- ✅ No temporary workarounds or incomplete implementations
-
-### Merge Requirements
-```bash
-# Before merging, verify:
-npm test                    # All tests pass
-git status                  # Clean working directory
-git checkout main           # Switch to main
-git merge task-N-description # Merge completed task
-git branch -d task-N-description # Clean up branch
-```
-
-## Cline Chat Management
-
-This project is being developed using Cline, an AI coding assistant. To manage context window limits:
-
-- **Context Threshold**: When context usage exceeds 50K tokens during development
-- **Task Completion Rule**: Always complete the current task before starting a new chat
-- **New Chat Process**: Once a task is complete and context > 50K tokens:
-  1. Update this README.md with task completion status
-  2. Generate a detailed prompt for the next task
-  3. Use Cline's "new task" feature to start fresh with the next task prompt
-  4. Include full context about completed work and current project state
-
-## Notes
-
-- Tasks 1, 3, and 7 are critical for production readiness
-- Task 2 (indexing) significantly improves query performance
-- Tasks 4, 5, 6, and 8 add important functionality but are less critical
-- Each task is designed to be completed independently while building on previous work
-- Context management ensures efficient development without losing important project context

package/prompts/TYPESCRIPT_GENERATION.md

@@ -1,179 +0,0 @@
-# TypeScript Definition Generation from JSDoc
-
-This document explains how to generate TypeScript definition files (.d.ts) from the JSDoc comments in the JoystickDB project.
-
-## Prerequisites
-
-Make sure you have TypeScript installed as a dev dependency:
-
-```bash
-npm install --save-dev typescript
-```
-
-## Configuration Files
-
-### tsconfig.json
-The TypeScript configuration file is set up to:
-- Allow JavaScript files (`allowJs: true`)
-- Generate only declaration files (`emitDeclarationOnly: true`)
-- Output to the `./types` directory
-- Include all files in `src/**/*.js`
-- Exclude test files and node_modules
-
-### package.json Scripts
-Three npm scripts are available for type generation:
-
-```json
-{
-  "build:types": "tsc --declaration --emitDeclarationOnly --allowJs --outDir types src/**/*.js",
-  "build:types:client": "tsc --declaration --emitDeclarationOnly --allowJs --outDir types/client src/client/*.js",
-  "build:types:server": "tsc --declaration --emitDeclarationOnly --allowJs --outDir types/server src/server/**/*.js"
-}
-```
-
-## Generating TypeScript Definitions
-
-### Generate All Types
-```bash
-npm run build:types
-```
-
-### Generate Client Types Only
-```bash
-npm run build:types:client
-```
-
-### Generate Server Types Only
-```bash
-npm run build:types:server
-```
-
-### Manual Generation
-You can also run TypeScript compiler directly:
-
-```bash
-# Generate all types
-npx tsc
-
-# Generate specific files
-npx tsc --declaration --emitDeclarationOnly --allowJs --outDir types src/client/index.js
-```
-
-## Output Structure
-
-After running the type generation, you'll find the TypeScript definition files in the `types/` directory:
-
-```
-types/
-├── client/
-│   └── index.d.ts          # Client library types
-└── server/
-    ├── index.d.ts          # Main server types
-    ├── cluster/
-    │   ├── index.d.ts      # Cluster management types
-    │   └── master.d.ts     # Cluster master types
-    └── lib/
-        ├── logger.d.ts     # Logger types
-        ├── tcp_protocol.d.ts # TCP protocol types
-        └── operations/
-            ├── insert_one.d.ts  # Insert operation types
-            ├── find_one.d.ts    # Find operation types
-            └── ...              # Other operation types
-```
-
-## Using the Generated Types
-
-### In TypeScript Projects
-```typescript
-import { JoystickDBClient, ClientOptions } from './types/client/index.js';
-
-const client: JoystickDBClient = joystickdb.client({
-  host: 'localhost',
-  port: 1983,
-  authentication: {
-    password: 'your-password'
-  }
-});
-```
-
-### In JavaScript Projects with JSDoc
-```javascript
-/**
- * @typedef {import('./types/client/index.js').ClientOptions} ClientOptions
- * @typedef {import('./types/client/index.js').JoystickDBClient} JoystickDBClient
- */
-
-/** @type {ClientOptions} */
-const options = {
-  host: 'localhost',
-  port: 1983
-};
-```
-
-## Troubleshooting
-
-### Common Issues
-
-1. **"No inputs were found" error**: Make sure your JavaScript files have proper JSDoc comments and are in the correct directories.
-
-2. **Import/Export issues**: Ensure your JavaScript files use proper ES module syntax (`import`/`export`).
-
-3. **Missing types**: Check that your JSDoc comments use proper syntax:
-   - `@param {Type} name - description`
-   - `@returns {Type} description`
-   - `@typedef {Object} TypeName`
-
-### Validating Generated Types
-
-You can validate the generated types by creating a test TypeScript file:
-
-```typescript
-// test-types.ts
-import { JoystickDBClient } from './types/client/index.js';
-
-// This should compile without errors if types are correct
-const client = new JoystickDBClient({
-  host: 'localhost',
-  port: 1983
-});
-```
-
-Then compile it:
-```bash
-npx tsc test-types.ts --noEmit
-```
-
-## Continuous Integration
-
-Add type generation to your CI/CD pipeline:
-
-```yaml
-# .github/workflows/ci.yml
-- name: Generate TypeScript definitions
-  run: npm run build:types
-
-- name: Validate generated types
-  run: npx tsc --noEmit types/**/*.d.ts
-```
-
-## Updating Types
-
-When you modify JSDoc comments in your source files:
-
-1. Run the type generation script again
-2. Commit the updated `.d.ts` files to version control
-3. Update version numbers if publishing to npm
-
-## Publishing Types
-
-If publishing to npm, include the types in your package.json:
-
-```json
-{
-  "main": "src/server/index.js",
-  "types": "types/server/index.d.ts",
-  "files": [
-    "src/",
-    "types/"
-  ]
-}