resplite 1.2.6 → 1.2.8

package/spec/05-scan-admin-implementation.md

@@ -0,0 +1,379 @@
+# RESPLite Specification v1 — SCAN, Admin Commands, Project Structure, Testing
+
+## 17. SCAN Behavior
+
+SCAN is part of the v1 scope.
+
+### 17.1 Minimum supported form
+
+Required in v1:
+
+- `SCAN cursor`
+
+### 17.2 Response shape
+
+The response should follow Redis-like shape:
+
+- next cursor
+- array of keys
+
+### 17.3 Implementation note
+
+A simple initial strategy is acceptable, such as lexicographic traversal over `redis_keys.key`.
+
+### 17.4 Future extensions
+
+Possible later additions:
+
+- `MATCH pattern`
+- `COUNT n`
+
+These are not required in the first implementation.
+
+---
+
+## 18. Administrative Commands
+
+### 18.1 SQLITE.INFO
+
+This command should expose useful operational information such as:
+
+- database path
+- SQLite version
+- counts by type
+- total logical key count
+- WAL mode status if available
+
+### 18.2 CACHE.INFO
+
+This command should expose cache information such as:
+
+- cache enabled or disabled
+- entry count
+- estimated memory use
+- hit count
+- miss count
+- hit ratio if available
+
+These commands are intended for observability and debugging.
+
+---
+
+## 19. Future Search Scope
+
+Search is not part of the base v1 implementation, but the architecture must remain compatible with a near-term v1.1 search layer.
+
+### 19.1 Search vision
+
+The future search subsystem should provide RediSearch-inspired commands powered by SQLite FTS5 with BM25 ranking.
+
+Candidate commands:
+
+- `FT.CREATE`
+- `FT.SEARCH`
+- `FT.INFO`
+- `FT.DROPINDEX`
+
+### 19.2 Recommended document model
+
+The preferred future approach is to index hashes as documents.
+Example logical model:
+
+- `article:1` stored as a hash
+- an index with prefix `article:`
+- fields such as `title`, `body`, and `category`
+
+### 19.3 Backend strategy
+
+- SQLite FTS5 virtual tables
+- BM25 ranking
+- auxiliary metadata tables for filtering and sorting
+
+The project should not attempt to fully clone RediSearch grammar or behavior at the start.
+A clean, reduced, Redis-like search surface is preferable.
+
+---
+
+## 20. Project Structure
+
+Recommended Node.js JavaScript ESM project layout:
+
+```text
+resplite/
+  package.json
+  README.md
+  SPEC.md
+  src/
+    index.js
+    server/
+      tcp-server.js
+      connection.js
+    resp/
+      parser.js
+      encoder.js
+      types.js
+    commands/
+      registry.js
+      ping.js
+      echo.js
+      quit.js
+      get.js
+      set.js
+      del.js
+      exists.js
+      expire.js
+      pexpire.js
+      ttl.js
+      pttl.js
+      persist.js
+      incr.js
+      decr.js
+      incrby.js
+      decrby.js
+      type.js
+      scan.js
+      hset.js
+      hget.js
+      hmget.js
+      hgetall.js
+      hdel.js
+      hexists.js
+      hincrby.js
+      sadd.js
+      srem.js
+      smembers.js
+      sismember.js
+      scard.js
+      sqlite-info.js
+      cache-info.js
+    engine/
+      engine.js
+      errors.js
+      expiration.js
+      validate.js
+    storage/
+      sqlite/
+        db.js
+        schema.js
+        pragmas.js
+        keys.js
+        strings.js
+        hashes.js
+        sets.js
+        tx.js
+    cache/
+      lru.js
+      cache.js
+    util/
+      buffers.js
+      patterns.js
+      clock.js
+      logger.js
+  test/
+    helpers/
+      server.js
+      client.js
+      tmp.js
+      clock.js
+      fixtures.js
+    unit/
+      ...
+    integration/
+      ...
+    contract/
+      ...
+    stress/
+      ...
+```
+
+---
+
+## 21. Implementation Stack
+
+Recommended initial stack:
+
+- Node.js
+- JavaScript ESM
+- `better-sqlite3`
+- built-in `node:test`
+- `assert/strict`
+- `redis` npm package for contract tests
+
+The v1 codebase should prioritize correctness, testability, and operational simplicity over TypeScript or extensive framework usage.
+
+---
+
+## 22. Testing Strategy
+
+Testing is a first-class requirement.
+A command is not considered implemented until it has meaningful coverage.
+
+### 22.1 Testing categories
+
+The project must include the following categories of tests:
+
+- unit tests
+- integration tests
+- contract tests
+- persistence tests
+- expiration tests
+- consistency tests
+- concurrency tests
+- protocol framing tests
+- binary-safety tests
+
+### 22.2 Per-command testing rule
+
+Each supported command should have tests for:
+
+- normal behavior
+- missing-key behavior where applicable
+- wrong-type behavior where applicable
+- TTL interaction where applicable
+- persistence across restart where applicable
+- binary-safe behavior where applicable
+
+### 22.3 RESP protocol tests
+
+The RESP layer must be tested for:
+
+- valid parsing of RESP2 types
+- partial frames split across TCP chunks
+- multiple commands arriving in a single TCP chunk
+- correct encoding of all response types
+
+### 22.4 Contract tests with real clients
+
+v1 contract tests should use:
+
+- `redis-cli`
+- the official `redis` npm client
+
+### 22.5 Persistence tests
+
+The project must verify:
+
+- data survives server restart
+- TTL metadata survives restart
+- expired keys behave as missing after restart
+- mixed key types remain valid after restart
+
+### 22.6 Consistency tests
+
+The project must verify internal invariants such as:
+
+- rows in type tables must correspond to a valid row in `redis_keys`
+- logical type must match actual storage table placement
+- expired keys must not appear through command results
+- empty hashes and sets are removed logically
+
+### 22.7 Concurrency tests
+
+The project must verify behavior under multiple clients, including:
+
+- many concurrent INCR operations on the same key
+- reads and writes overlapping across connections
+- expiration sweeps occurring during active traffic
+- restart safety under recent write activity
+
+### 22.8 Performance sanity tests
+
+The project does not need benchmark-grade tests in v1, but it should include sanity checks such as:
+
+- repeated SET and GET operations at modest scale
+- many TTL expirations without protocol failure
+- SCAN on a non-trivial keyspace
+
+---
+
+## 23. Package Scripts
+
+Suggested package scripts:
+
+```json
+{
+  "type": "module",
+  "scripts": {
+    "test": "node --test",
+    "test:unit": "node --test test/unit",
+    "test:integration": "node --test test/integration",
+    "test:contract": "node --test test/contract",
+    "test:stress": "node --test test/stress",
+    "test:all": "node --test test"
+  }
+}
+```
+
+---
+
+## 24. Acceptance Criteria for Initial Milestone
+
+The first meaningful milestone is achieved when the server can be exercised successfully through `redis-cli` and passes automated tests for the following sequence:
+
+```text
+PING
+SET foo bar
+GET foo
+EXPIRE foo 10
+TTL foo
+DEL foo
+HSET user:1 name Martin age 42
+HGET user:1 name
+HGETALL user:1
+SADD tags a b c
+SMEMBERS tags
+TYPE foo
+SCAN 0
+```
+
+In addition, the system must satisfy these conditions:
+
+- persistence survives restart
+- wrong-type errors are returned correctly
+- expired keys behave as missing keys
+- binary-safe values do not break command semantics
+- multiple clients can connect and issue commands without corrupting state
+
+---
+
+## 25. Implementation Order
+
+Recommended phased implementation order:
+
+### Phase 1
+
+Infrastructure: TCP server, RESP parser/encoder, command registry, SQLite init, error model, protocol tests.
+
+### Phase 2
+
+Strings and core: PING, ECHO, GET, SET, DEL, EXISTS, TYPE.
+
+### Phase 3
+
+Expiration: EXPIRE, PEXPIRE, TTL, PTTL, PERSIST, lazy expiration, active sweeper.
+
+### Phase 4
+
+Numeric string operations: INCR, DECR, INCRBY, DECRBY.
+
+### Phase 5
+
+Hashes: HSET, HGET, HMGET, HGETALL, HDEL, HEXISTS, HINCRBY.
+
+### Phase 6
+
+Sets: SADD, SREM, SMEMBERS, SISMEMBER, SCARD.
+
+### Phase 7
+
+Introspection and observability: SCAN, SQLITE.INFO, CACHE.INFO.
+
+### Phase 8
+
+Migration tooling: programmatic Redis migration API using SCAN, import verification and reporting.
+
+### Phase 9
+
+Search extension: FT.CREATE, FT.SEARCH, FT.INFO, FT.DROPINDEX (SQLite FTS5 with BM25).
+
+Search should only begin after the core command, TTL, persistence, and concurrency suites are stable.

package/spec/06-migration-strategy-core.md

@@ -0,0 +1,79 @@
+# RESPLite Specification v1 — Migration Strategy (Core)
+
+## 26. Migration Strategy from Redis
+
+Migration is not part of the RESP protocol surface in v1.
+It should be implemented as a programmatic module outside the RESP command surface.
+
+### 26.1 Initial migration method
+
+Recommended initial approach:
+
+- connect to a real Redis instance
+- iterate keys using `SCAN`
+- discover key type with `TYPE`
+- fetch values according to type
+- fetch expiration using `PTTL`
+- write translated data into SQLite
+
+### 26.2 Initial migratable subset
+
+The initial migration module should support:
+
+- strings
+- hashes
+- sets
+- TTL metadata
+
+### 26.3 Not required initially
+
+- RDB parsing
+- AOF parsing
+- mirror mode
+- dual-write migration
+
+These can be considered later if adoption demands them.
+
+---
+
+## 27. Compatibility Matrix Guidance
+
+The README should include a compatibility matrix with at least three groups:
+
+### Supported
+
+All commands implemented in v1 (and any extensions such as lists, zsets, FT.*, blocking, migration API).
+
+### Planned
+
+Future near-term commands or features as documented in the respective spec files.
+
+### Not Supported
+
+Features explicitly excluded from the roadmap for now, such as:
+
+- Pub/Sub
+- Streams
+- Lua
+- Cluster
+- Replication
+- Blocking commands (unless added per blocking spec)
+
+This matrix is essential for setting correct user expectations.
+
+---
+
+## 28. Final Design Rule
+
+A command should only be added if its implementation is:
+
+- natural on SQLite
+- semantically clear
+- testable
+- operationally safe
+- maintainable without excessive special handling
+
+If a command requires too much implementation gymnastics to preserve Redis-like behavior, it should not be part of the supported surface until a clean design exists.
+
+This rule is central to the project.
+It protects correctness, keeps the scope honest, and preserves the identity of RESPLite as a practical Redis-compatible server built on top of SQLite rather than a fragile imitation.

package/spec/07-type-lists.md

@@ -0,0 +1,202 @@
+# RESPLite — Type: Lists (Appendix B)
+
+## B.1 Goals
+
+* Provide Redis-compatible LIST commands over RESP2, backed by SQLite persistence.
+* Avoid O(n) reindexing for push/pop operations.
+* Support binary-safe values (`Buffer`) and keys as `BLOB`.
+* Maintain correct Redis semantics for empty lists and wrong-type errors.
+
+## B.2 Supported Commands (vNext)
+
+* `LPUSH key value [value ...]`
+* `RPUSH key value [value ...]`
+* `LPOP key [count]`
+* `RPOP key [count]`
+* `LLEN key`
+* `LRANGE key start stop`
+* `LINDEX key index` (optional, recommended)
+* `LSET key index value` (optional, later)
+* `LTRIM key start stop` (optional, later)
+
+**Not supported initially**
+
+* Blocking commands: `BLPOP`, `BRPOP`, `BRPOPLPUSH`
+* `RPOPLPUSH`, `LMOVE` (later if needed)
+
+## B.3 Redis Semantics
+
+* **Wrong type:** If `key` exists and is not a list, return:
+
+  * `WRONGTYPE Operation against a key holding the wrong kind of value`
+* **Non-existent key:**
+
+  * `LLEN` returns `0`
+  * `LRANGE` returns empty array
+  * `LPOP/RPOP` returns `nil` (or empty array for count > 1)
+* **Empty list after removals:** When a list becomes empty, delete the logical key:
+
+  * delete metadata from `redis_keys`
+  * delete list meta/items rows
+
+## B.4 Data Model (SQLite)
+
+Lists are stored using a monotonic sequence strategy per key to avoid shifting indices.
+
+### B.4.1 Metadata table
+
+```sql
+CREATE TABLE redis_list_meta (
+  key BLOB PRIMARY KEY,
+  head_seq INTEGER NOT NULL,
+  tail_seq INTEGER NOT NULL,
+  FOREIGN KEY(key) REFERENCES redis_keys(key) ON DELETE CASCADE
+);
+```
+
+**Invariant:**
+
+* Empty list should not exist (no meta row). If it exists, it must satisfy `head_seq <= tail_seq`.
+
+### B.4.2 Items table
+
+```sql
+CREATE TABLE redis_list_items (
+  key BLOB NOT NULL,
+  seq INTEGER NOT NULL,
+  value BLOB NOT NULL,
+  PRIMARY KEY (key, seq),
+  FOREIGN KEY(key) REFERENCES redis_keys(key) ON DELETE CASCADE
+);
+
+CREATE INDEX redis_list_items_key_seq_idx ON redis_list_items(key, seq);
+```
+
+## B.5 Sequence Strategy
+
+* Each list key maintains:
+
+  * `head_seq`: the smallest sequence currently used (front)
+  * `tail_seq`: the largest sequence currently used (back)
+* For a new list:
+
+  * initialize `head_seq = 0`, `tail_seq = -1` (or similar empty sentinel)
+  * but we recommend **not creating meta** until first push.
+
+### Push operations
+
+* `LPUSH`:
+
+  * decrement `head_seq` and insert new items at `seq = head_seq - 1` per pushed element
+* `RPUSH`:
+
+  * increment `tail_seq` and insert new items at `seq = tail_seq + 1`
+
+Maintain ordering:
+
+* front is smaller `seq`, back is larger `seq`.
+
+## B.6 Command Behavior
+
+### B.6.1 LPUSH
+
+**Request:** `LPUSH key v1 v2 ...`
+**Response:** Integer = new list length
+
+Implementation notes:
+
+* If key does not exist: create `redis_keys` type list + create `redis_list_meta`.
+* Insert values in left-push order consistent with Redis:
+
+  * `LPUSH mylist a b c` results in list `[c, b, a]` at the head.
+* Use a single transaction:
+
+  * ensure type
+  * upsert meta
+  * insert items
+  * bump `redis_keys.version`, update `updated_at`
+
+### B.6.2 RPUSH
+
+Same structure, response is new length.
+
+### B.6.3 LLEN
+
+Return length:
+
+* If key not exist: `0`
+* Else length = `tail_seq - head_seq + 1` (derived from meta)
+* Do not count rows for performance.
+
+### B.6.4 LPOP / RPOP
+
+* Without `count`: return bulk string or `nil`
+* With `count`:
+
+  * Redis returns array of popped elements
+  * If list has fewer than count: return all available
+  * If key does not exist: return `nil` for no-count, or empty array for count
+* After popping last element: delete key and meta.
+
+**Implementation:**
+
+* In a transaction:
+
+  * read meta
+  * compute seq range to remove
+  * select values for response
+  * delete those rows
+  * update meta head/tail accordingly
+  * if empty -> delete key meta entirely
+
+### B.6.5 LRANGE
+
+`LRANGE key start stop` returns array of elements.
+
+Index rules (Redis):
+
+* `start` and `stop` are inclusive.
+* Negative indices count from end (`-1` is last element).
+* Clamp indices to valid bounds.
+* If start > stop after normalization: empty array.
+
+**Mapping indices to sequences:**
+
+* Let `len = tail_seq - head_seq + 1`
+* Normalize start/stop into `[0..len-1]`
+* Convert to seq:
+
+  * `seq_start = head_seq + start`
+  * `seq_stop  = head_seq + stop`
+* SQL:
+
+  * `SELECT value FROM redis_list_items WHERE key=? AND seq BETWEEN ? AND ? ORDER BY seq ASC`
+
+## B.7 Expiration and Cache
+
+* TTL is stored in `redis_keys.expires_at`, same as other types.
+* Lazy expiration must delete list meta/items like other types.
+* Cache strategy (recommended):
+
+  * Cache small lists (len <= configured threshold) optionally.
+  * Otherwise cache only meta (`head_seq`, `tail_seq`) if beneficial.
+  * Always invalidate on list writes.
+
+## B.8 Complexity Targets
+
+* `LPUSH/RPUSH`: O(k) inserts, no reindexing
+* `LPOP/RPOP`: O(k) deletes + selects for return
+* `LLEN`: O(1)
+* `LRANGE`: O(n) in returned range
+
+## B.9 Tests (Required)
+
+For each command above:
+
+* Happy path
+* Non-existent key behavior
+* Wrong-type behavior
+* TTL interaction (expires then acts as missing)
+* Persistence across restart
+* Binary safety for values
+* Concurrency sanity (multiple clients pushing/popping does not corrupt meta)