bun-sqlite-for-rxdb 1.0.1

package/.serena/project.yml

@@ -0,0 +1,84 @@
+# list of languages for which language servers are started; choose from:
+#   al               bash             clojure          cpp              csharp           csharp_omnisharp
+#   dart             elixir           elm              erlang           fortran          go
+#   haskell          java             julia            kotlin           lua              markdown
+#   nix              perl             php              python           python_jedi      r
+#   rego             ruby             ruby_solargraph  rust             scala            swift
+#   terraform        typescript       typescript_vts   yaml             zig
+# Note:
+#   - For C, use cpp
+#   - For JavaScript, use typescript
+# Special requirements:
+#   - csharp: Requires the presence of a .sln file in the project folder.
+# When using multiple languages, the first language server that supports a given file will be used for that file.
+# The first language is the default language and the respective language server will be used as a fallback.
+# Note that when using the JetBrains backend, language servers are not used and this list is correspondingly ignored.
+languages:
+- typescript
+
+# the encoding used by text files in the project
+# For a list of possible encodings, see https://docs.python.org/3.11/library/codecs.html#standard-encodings
+encoding: "utf-8"
+
+# whether to use the project's gitignore file to ignore files
+# Added on 2025-04-07
+ignore_all_files_in_gitignore: true
+
+# list of additional paths to ignore
+# same syntax as gitignore, so you can use * and **
+# Was previously called `ignored_dirs`, please update your config if you are using that.
+# Added (renamed) on 2025-04-07
+ignored_paths: []
+
+# whether the project is in read-only mode
+# If set to true, all editing tools will be disabled and attempts to use them will result in an error
+# Added on 2025-04-18
+read_only: false
+
+# list of tool names to exclude. We recommend not excluding any tools, see the readme for more details.
+# Below is the complete list of tools for convenience.
+# To make sure you have the latest list of tools, and to view their descriptions, 
+# execute `uv run scripts/print_tool_overview.py`.
+#
+#  * `activate_project`: Activates a project by name.
+#  * `check_onboarding_performed`: Checks whether project onboarding was already performed.
+#  * `create_text_file`: Creates/overwrites a file in the project directory.
+#  * `delete_lines`: Deletes a range of lines within a file.
+#  * `delete_memory`: Deletes a memory from Serena's project-specific memory store.
+#  * `execute_shell_command`: Executes a shell command.
+#  * `find_referencing_code_snippets`: Finds code snippets in which the symbol at the given location is referenced.
+#  * `find_referencing_symbols`: Finds symbols that reference the symbol at the given location (optionally filtered by type).
+#  * `find_symbol`: Performs a global (or local) search for symbols with/containing a given name/substring (optionally filtered by type).
+#  * `get_current_config`: Prints the current configuration of the agent, including the active and available projects, tools, contexts, and modes.
+#  * `get_symbols_overview`: Gets an overview of the top-level symbols defined in a given file.
+#  * `initial_instructions`: Gets the initial instructions for the current project.
+#     Should only be used in settings where the system prompt cannot be set,
+#     e.g. in clients you have no control over, like Claude Desktop.
+#  * `insert_after_symbol`: Inserts content after the end of the definition of a given symbol.
+#  * `insert_at_line`: Inserts content at a given line in a file.
+#  * `insert_before_symbol`: Inserts content before the beginning of the definition of a given symbol.
+#  * `list_dir`: Lists files and directories in the given directory (optionally with recursion).
+#  * `list_memories`: Lists memories in Serena's project-specific memory store.
+#  * `onboarding`: Performs onboarding (identifying the project structure and essential tasks, e.g. for testing or building).
+#  * `prepare_for_new_conversation`: Provides instructions for preparing for a new conversation (in order to continue with the necessary context).
+#  * `read_file`: Reads a file within the project directory.
+#  * `read_memory`: Reads the memory with the given name from Serena's project-specific memory store.
+#  * `remove_project`: Removes a project from the Serena configuration.
+#  * `replace_lines`: Replaces a range of lines within a file with new content.
+#  * `replace_symbol_body`: Replaces the full definition of a symbol.
+#  * `restart_language_server`: Restarts the language server, may be necessary when edits not through Serena happen.
+#  * `search_for_pattern`: Performs a search for a pattern in the project.
+#  * `summarize_changes`: Provides instructions for summarizing the changes made to the codebase.
+#  * `switch_modes`: Activates modes by providing a list of their names
+#  * `think_about_collected_information`: Thinking tool for pondering the completeness of collected information.
+#  * `think_about_task_adherence`: Thinking tool for determining whether the agent is still on track with the current task.
+#  * `think_about_whether_you_are_done`: Thinking tool for determining whether the task is truly completed.
+#  * `write_memory`: Writes a named memory (for future reference) to Serena's project-specific memory store.
+excluded_tools: []
+
+# initial prompt for the project. It will always be given to the LLM upon activating the project
+# (contrary to the memories, which are loaded on demand).
+initial_prompt: ""
+
+project_name: "bun-sqlite-for-rxdb"
+included_optional_tools: []

package/CHANGELOG.md

@@ -0,0 +1,300 @@
+# Changelog
+
+## [1.0.1] - 2026-02-23
+
+### Added
+- **EXPLAIN QUERY PLAN Debug Mode** (Development Tool)
+  - Activated via `DEBUG_QUERIES=1` environment variable
+  - Logs query plans, SQL, and args to console
+  - Helps verify query builder generates optimal SQL
+  - Validates SQLite index usage
+  - Zero production overhead (env var check only)
+
+### Usage
+```bash
+DEBUG_QUERIES=1 bun test
+```
+
+---
+
+## [1.0.0] - 2026-02-23
+
+### Added
+- **Attachments Support** (Phase 4)
+  - Storage-level implementation with 4 comprehensive tests
+  - `getAttachmentData()` returns base64 strings with digest validation
+  - Preserves `_attachments` metadata in documents
+  - Attachments table with composite keys (documentId||attachmentId)
+  - 122/122 official RxDB tests passing (includes 5 attachment tests)
+- **RxDB Helper Functions** (Phase 4)
+  - `categorizeBulkWriteRows()` - Battle-tested conflict detection + attachment extraction
+  - `stripAttachmentsDataFromDocument()` - Remove attachment .data field, keep metadata
+  - `stripAttachmentsDataFromRow()` - Strip attachments from bulk write rows
+  - `attachmentWriteDataToNormalData()` - Convert attachment write format to storage format
+  - `getAttachmentSize()` - Calculate attachment size from base64
+
+### Changed
+- **bulkWrite Refactored** - Now uses `categorizeBulkWriteRows()` helper
+  - Cleaner architecture with proper conflict handling
+  - Automatic attachment extraction
+  - Matches official adapter patterns (Dexie, MongoDB, SQLite)
+
+### Test Results
+- **Local tests: 138/138 pass (100%)** ✅
+- **Official RxDB tests: 122/122 pass (100%)** ✅
+- **Total: 260/260 tests pass (100%)** 🎉
+
+### Performance
+- Database operations: 1.06-1.68x faster than better-sqlite3
+- Query builder cache: 5.2-57.9x speedup for cached queries
+- All optimizations from v0.4.0 included
+
+### Documentation
+- Updated ROADMAP.md - Phase 4 marked COMPLETE
+- Removed redundant Phase 4 TDD implementation details
+- All helper functions documented with line numbers
+
+---
+
+## [0.4.0] - 2026-02-23
+
+### Added
+- **Query Builder LRU Cache** (Phase 2.5)
+  - 5.2-57.9x speedup for repeated queries
+  - High-frequency: 505K-808K queries/sec
+  - Bounded at 500 entries (no memory leak)
+  - True LRU eviction with canonical key generation (fast-stable-stringify)
+  - Zero dependencies except fast-stable-stringify (5KB)
+- **RxDB Official Test Suite Integration** (Phase 3.1)
+  - 112/112 official RxDB tests passing (100%)
+  - StatementManager abstraction for automatic statement lifecycle
+  - Connection pooling with reference counting for multi-instance support
+  - Official `addRxStorageMultiInstanceSupport()` integration
+  - Composite primary key support
+  - Bun test suite compatibility (Mocha through Bun)
+- **Performance Benchmarks** (Phase 3.2)
+  - Bun:sqlite 1.06-1.68x faster than better-sqlite3
+  - Benchmarked at 1M documents with WAL + PRAGMA synchronous = 1
+  - Query builder cache benchmarks
+  - Raw database comparison benchmarks
+- **New Query Operators** (8 operators added)
+  - `$exists` - Field existence check with IS NULL/IS NOT NULL
+  - `$regex` - Pattern matching with LIKE translation and Mingo fallback
+  - `$elemMatch` - Array element matching (Mingo fallback)
+  - `$not` - Negation operator
+  - `$nor` - Logical NOR
+  - `$type` - Type checking with typeof()
+  - `$size` - Array size with json_array_length()
+  - `$mod` - Modulo operations
+
+### Changed
+- **BREAKING**: Statement lifecycle management
+  - Static SQL uses db.query() (cached, max 20)
+  - Dynamic SQL uses db.prepare() + finalize() (no cache pollution)
+  - StatementManager abstraction eliminates manual try-finally boilerplate
+- Connection pooling now mandatory for multi-instance support
+- Switched from custom multi-instance to RxDB's official implementation
+
+### Fixed
+- Statement resource leaks (7 locations in instance.ts)
+- Collection isolation bug (events leaked across collections)
+- Composite primary key handling (string vs object format)
+- EventBulk.id generation (empty string → unique timestamp + random)
+- Multi-instance event propagation via BroadcastChannel
+- Bun test suite compatibility (node:sqlite import, test globals)
+
+### Performance
+- Query builder cache: 5.2-57.9x speedup for cached queries
+- Database operations: 1.06-1.68x faster than better-sqlite3
+- No OOM errors (proper statement finalization)
+- Tests complete in 12.91s (no hangs)
+
+### Test Results
+- **Local tests: 134/134 pass (100%)**
+- **Official RxDB tests: 112/112 pass (100%)**
+- **Total: 246/246 tests pass (100%)** 🎉
+
+### Documentation
+- Added `docs/id1-testsuite-journey.md` - Complete debugging journey (15 iterations)
+- Added `docs/official-test-suite-setup.md` - Guide for running RxDB tests with Bun
+- Updated `docs/architectural-patterns.md` - Added patterns 15-24
+- Updated `ROADMAP.md` - Phase 2.5 and 3.2 marked COMPLETE, Phase 4 marked READY
+
+### Technical Debt Resolved
+- Statement lifecycle properly managed (no leaks)
+- Connection pooling with reference counting
+- Test architecture at correct level (RxDatabase for integration, storage for low-level)
+- Proper separation of concerns (we handle DB pooling, RxDB handles event coordination)
+
+---
+
+## [0.3.1] - 2026-02-22
+
+### Added
+- **JSONB Storage**: Implemented SQLite's native binary JSON format as default storage
+  - 1.57x faster complex queries (657ms → 418ms at 1M docs)
+  - 1.20x faster read + parse operations
+  - 1.04x faster simple queries
+  - Uses `jsonb()` on INSERT, `json()` on SELECT
+- **Smart Regex Optimization**: Convert simple regex patterns to SQL operators
+  - 2.03x faster exact matches (^text$ → = operator)
+  - 1.23x faster case-insensitive (COLLATE NOCASE vs LOWER())
+  - Overall 1.24x speedup across all patterns
+- **Regression Tests**: Added tests for % and _ escaping edge cases
+
+### Fixed
+- Critical bug: Missing % and _ escaping in case-insensitive exact match regex patterns
+
+### Investigated
+- **FTS5 Trigram Indexes**: Benchmarked at 100k and 1M scales
+  - Result: 1.5-1.79x SLOWER at our scale
+  - Decision: NOT implemented (only beneficial at 10M+ rows)
+  - Documented findings in architectural-patterns.md
+
+### Performance
+- Complex queries: 1.57x faster (JSONB)
+- Exact match regex: 2.03x faster (smart optimization)
+- Read operations: 1.20x faster (JSONB)
+
+---
+
+## [0.3.0] - 2026-02-22
+
+### Added
+- **Advanced Query Operators** (4 new operators)
+  - `$in` - Value in array (80% usage in production)
+  - `$nin` - Value not in array
+  - `$or` - Logical OR with proper parentheses handling
+  - `$and` - Explicit logical AND
+- NULL handling for `$in` / `$nin` operators
+- Recursive query builder with `logicalDepth` tracking
+- 13 new tests for advanced operators
+
+### Performance
+- Benchmark results (10k documents):
+  - Average query time: 27.39ms
+  - Supports complex nested queries: `{$or: [{$and: [...]}, {field: {$in: [...]}}]}`
+
+### Technical
+- DRY architecture: Pure operator functions, recursive builder
+- Type-safe: 0 `any` types, proper TypeScript throughout
+- Test coverage: 44/44 tests passing
+
+## [0.2.0] - 2026-02-22
+
+### Added
+- Conflict detection for concurrent writes
+  - Catches UNIQUE constraint violations
+  - Returns 409 status with existing document
+  - Enables proper RxDB replication conflict handling
+
+### Changed
+- **BREAKING**: `bulkWrite` now uses individual INSERT statements instead of INSERT OR REPLACE
+  - Conflicts are now detected and returned as errors
+  - Enables proper RxDB replication conflict resolution
+
+### Performance
+- Benchmark results (10k documents, 10-run average):
+  - **JSON + TEXT: 23.40ms** (WINNER)
+  - Tested alternatives: MessagePack (137ms), bun:jsc (37ms)
+  - **Verdict: Bun's SIMD-accelerated JSON is fastest**
+
+### Research Notes
+- Extensively tested binary serialization formats (MessagePack, bun:jsc)
+- MessagePack: 5.6x slower than JSON (pure JS implementation)
+- bun:jsc (optimized): 1.58x slower than JSON (Structured Clone overhead)
+- **Conclusion: JSON + TEXT is optimal for Bun's architecture**
+
+## [0.1.2] - 2026-02-22
+
+### Added
+- Conflict detection for concurrent writes
+  - Catches UNIQUE constraint violations
+  - Returns 409 status with existing document
+  - Enables proper RxDB replication conflict handling
+
+### Changed
+- **BREAKING**: `bulkWrite` now uses individual INSERT statements instead of INSERT OR REPLACE
+  - Conflicts are now detected and returned as errors
+  - Enables proper RxDB replication conflict resolution
+
+### Performance
+- Benchmark results (10k documents):
+  - Simple equality: 15.40ms
+  - Greater than: 22.05ms
+  - Multiple conditions: 23.51ms
+  - Range query: 24.89ms
+  - Average: 21.46ms
+
+## [0.1.2] - 2026-02-22
+
+### Added
+- WAL mode for 3-6x write speedup and better concurrency
+  - Automatically enabled for file-based databases
+  - Skipped for in-memory databases (not supported by SQLite)
+- Proper checkpoint implementation in changeStream
+  - Checkpoint structure: `{ id: documentId, lwt: timestamp }`
+  - Enables efficient replication tracking
+
+### Removed
+- **BREAKING**: Removed `conflictResultionTasks()` method (doesn't exist in RxDB interface)
+- **BREAKING**: Removed `resolveConflictResultionTask()` method (doesn't exist in RxDB interface)
+  - Conflict resolution happens at RxDB replication level, not storage level
+
+### Changed
+- WAL mode test now uses file-based database instead of in-memory
+
+### Performance
+- Write operations: 3-6x faster with WAL mode
+- Concurrent reads during writes: No blocking with WAL mode
+
+## [0.1.1] - 2026-02-22
+
+### Added
+- SQL query builder for WHERE clause generation (10-100x speedup)
+  - 6 Mango operators: `$eq`, `$ne`, `$gt`, `$gte`, `$lt`, `$lte`
+  - Functional architecture (pure functions, no state)
+  - Column mapping for RxDB internal fields
+  - Fallback to in-memory filtering if WHERE fails
+- Benchmark script for performance testing
+- Comprehensive test suite (27 tests total)
+  - 19 storage tests
+  - 6 operator tests
+  - 10 query builder tests
+
+### Changed
+- **BREAKING**: Removed per-document error handling in `bulkWrite`
+  - Now uses atomic transactions (all or nothing)
+  - Entire batch fails if any document fails
+- Removed ALL `any` types (32 instances → 0)
+- Proper TypeScript types throughout
+  - `RxStorage<BunSQLiteInternals, BunSQLiteStorageSettings>`
+  - `RxStorageInstance` with proper type parameters
+  - `PreparedQuery<RxDocType>` for queries
+  - `RxStorageWriteError<RxDocType>` for errors
+
+### Fixed
+- Type safety: Zero `any` types in entire codebase
+- Test isolation: Reference folder excluded from test runs
+- Schema types: Proper RxDB internal fields in all schemas
+
+### Performance
+- Query performance: ~20ms average for 10k documents
+- Uses SQL WHERE clauses with indexed columns
+- Expected 10-100x speedup vs in-memory filtering
+
+## [0.1.0] - 2026-02-20
+
+### Added
+- Initial RxDB storage adapter for Bun's native SQLite
+- Basic CRUD operations (bulkWrite, query, findById, count)
+- Atomic transactions via `bun:sqlite`
+- In-memory Mango query filtering
+- Streak tracking support
+- Change stream for reactivity
+- 8 passing tests
+
+### Features
+- SQLite via `bun:sqlite` (no external dependencies)
+- RxDB v16 and v17 beta support
+- MIT licensed

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 adam2am
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,87 @@
+# bun-sqlite-for-rxdb
+
+> Unofficial RxDB storage adapter for Bun's native SQLite (`bun:sqlite`)
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+## Features
+
+- ✅ Uses Bun's native SQLite
+- ✅ **260/260 tests passing** (138 local + 122 official RxDB tests)
+- ✅ Full RxDB storage interface implementation
+- ✅ **Attachments support** (base64 storage with digest validation)
+- ✅ **Query builder LRU cache** (5.2-57.9x speedup for repeated queries)
+- ✅ **18 Mango operators** ($eq, $ne, $gt, $gte, $lt, $lte, $in, $nin, $or, $and, $exists, $regex, $elemMatch, $not, $nor, $type, $size, $mod)
+- ✅ **Multi-instance support** with connection pooling
+- ✅ **1.06-1.68x faster than better-sqlite3** (with WAL mode)
+- ✅ MIT licensed
+
+## Performance
+
+Benchmarked against better-sqlite3 (1M documents, WAL mode + PRAGMA synchronous = 1):
+
+| Operation | Bun SQLite | better-sqlite3 | Speedup |
+|-----------|------------|----------------|---------|
+| Bulk INSERT (1M docs) | 7.42s | 7.90s | **1.06x faster** |
+| SELECT by ID (10K lookups) | 170ms | 170ms | Equal |
+| Complex WHERE query | 484ms | 814ms | **1.68x faster** |
+
+**Requirements for optimal performance:**
+```typescript
+db.run("PRAGMA journal_mode = WAL");
+db.run("PRAGMA synchronous = 1");
+``` 
+
+## Installation
+
+```bash
+bun add bun-sqlite-for-rxdb
+```
+
+## Usage
+
+```typescript
+import { createRxDatabase } from 'rxdb';
+import { getRxStorageBunSQLite } from 'bun-sqlite-for-rxdb';
+
+const db = await createRxDatabase({
+  name: 'mydb',
+  storage: getRxStorageBunSQLite()
+});
+```
+
+## Development
+
+```bash
+# Install dependencies
+bun install
+
+# Build
+bun run build
+
+# Test
+bun test
+
+# Type check
+bun run typecheck
+```
+
+## Architecture
+
+**Key components:**
+- `RxStorage` factory
+- `RxStorageInstance` implementation (11 required methods)
+- Mango query → SQL translator
+- Change stream observables
+
+## License
+
+MIT © adam2am
+
+## Contributing
+
+Contributions welcome! This is a community project.
+
+---
+
+**Not affiliated with RxDB or Bun. Community-maintained adapter.**