resplite 1.2.6 → 1.2.8

package/spec/SPEC_B.md

@@ -1,426 +0,0 @@
-
-## Appendix B: Lists (Type = list)
-
-### B.1 Goals
-
-* Provide Redis-compatible LIST commands over RESP2, backed by SQLite persistence.
-* Avoid O(n) reindexing for push/pop operations.
-* Support binary-safe values (`Buffer`) and keys as `BLOB`.
-* Maintain correct Redis semantics for empty lists and wrong-type errors.
-
-### B.2 Supported Commands (vNext)
-
-* `LPUSH key value [value ...]`
-* `RPUSH key value [value ...]`
-* `LPOP key [count]`
-* `RPOP key [count]`
-* `LLEN key`
-* `LRANGE key start stop`
-* `LINDEX key index` (optional, recommended)
-* `LSET key index value` (optional, later)
-* `LTRIM key start stop` (optional, later)
-
-**Not supported initially**
-
-* Blocking commands: `BLPOP`, `BRPOP`, `BRPOPLPUSH`
-* `RPOPLPUSH`, `LMOVE` (later if needed)
-
-### B.3 Redis Semantics
-
-* **Wrong type:** If `key` exists and is not a list, return:
-
-  * `WRONGTYPE Operation against a key holding the wrong kind of value`
-* **Non-existent key:**
-
-  * `LLEN` returns `0`
-  * `LRANGE` returns empty array
-  * `LPOP/RPOP` returns `nil` (or empty array for count > 1)
-* **Empty list after removals:** When a list becomes empty, delete the logical key:
-
-  * delete metadata from `redis_keys`
-  * delete list meta/items rows
-
-### B.4 Data Model (SQLite)
-
-Lists are stored using a monotonic sequence strategy per key to avoid shifting indices.
-
-#### B.4.1 Metadata table
-
-```sql
-CREATE TABLE redis_list_meta (
-  key BLOB PRIMARY KEY,
-  head_seq INTEGER NOT NULL,
-  tail_seq INTEGER NOT NULL,
-  FOREIGN KEY(key) REFERENCES redis_keys(key) ON DELETE CASCADE
-);
-```
-
-**Invariant:**
-
-* Empty list should not exist (no meta row). If it exists, it must satisfy `head_seq <= tail_seq`.
-
-#### B.4.2 Items table
-
-```sql
-CREATE TABLE redis_list_items (
-  key BLOB NOT NULL,
-  seq INTEGER NOT NULL,
-  value BLOB NOT NULL,
-  PRIMARY KEY (key, seq),
-  FOREIGN KEY(key) REFERENCES redis_keys(key) ON DELETE CASCADE
-);
-
-CREATE INDEX redis_list_items_key_seq_idx ON redis_list_items(key, seq);
-```
-
-### B.5 Sequence Strategy
-
-* Each list key maintains:
-
-  * `head_seq`: the smallest sequence currently used (front)
-  * `tail_seq`: the largest sequence currently used (back)
-* For a new list:
-
-  * initialize `head_seq = 0`, `tail_seq = -1` (or similar empty sentinel)
-  * but we recommend **not creating meta** until first push.
-
-#### Push operations
-
-* `LPUSH`:
-
-  * decrement `head_seq` and insert new items at `seq = head_seq - 1` per pushed element
-* `RPUSH`:
-
-  * increment `tail_seq` and insert new items at `seq = tail_seq + 1`
-
-Maintain ordering:
-
-* front is smaller `seq`, back is larger `seq`.
-
-### B.6 Command Behavior
-
-#### B.6.1 LPUSH
-
-**Request:** `LPUSH key v1 v2 ...`
-**Response:** Integer = new list length
-
-Implementation notes:
-
-* If key does not exist: create `redis_keys` type list + create `redis_list_meta`.
-* Insert values in left-push order consistent with Redis:
-
-  * `LPUSH mylist a b c` results in list `[c, b, a]` at the head.
-* Use a single transaction:
-
-  * ensure type
-  * upsert meta
-  * insert items
-  * bump `redis_keys.version`, update `updated_at`
-
-#### B.6.2 RPUSH
-
-Same structure, response is new length.
-
-#### B.6.3 LLEN
-
-Return length:
-
-* If key not exist: `0`
-* Else length = `tail_seq - head_seq + 1` (derived from meta)
-* Do not count rows for performance.
-
-#### B.6.4 LPOP / RPOP
-
-* Without `count`: return bulk string or `nil`
-* With `count`:
-
-  * Redis returns array of popped elements
-  * If list has fewer than count: return all available
-  * If key does not exist: return `nil` for no-count, or empty array for count
-* After popping last element: delete key and meta.
-
-**Implementation:**
-
-* In a transaction:
-
-  * read meta
-  * compute seq range to remove
-  * select values for response
-  * delete those rows
-  * update meta head/tail accordingly
-  * if empty -> delete key meta entirely
-
-#### B.6.5 LRANGE
-
-`LRANGE key start stop` returns array of elements.
-
-Index rules (Redis):
-
-* `start` and `stop` are inclusive.
-* Negative indices count from end (`-1` is last element).
-* Clamp indices to valid bounds.
-* If start > stop after normalization: empty array.
-
-**Mapping indices to sequences:**
-
-* Let `len = tail_seq - head_seq + 1`
-* Normalize start/stop into `[0..len-1]`
-* Convert to seq:
-
-  * `seq_start = head_seq + start`
-  * `seq_stop  = head_seq + stop`
-* SQL:
-
-  * `SELECT value FROM redis_list_items WHERE key=? AND seq BETWEEN ? AND ? ORDER BY seq ASC`
-
-### B.7 Expiration and Cache
-
-* TTL is stored in `redis_keys.expires_at`, same as other types.
-* Lazy expiration must delete list meta/items like other types.
-* Cache strategy (recommended):
-
-  * Cache small lists (len <= configured threshold) optionally.
-  * Otherwise cache only meta (`head_seq`, `tail_seq`) if beneficial.
-  * Always invalidate on list writes.
-
-### B.8 Complexity Targets
-
-* `LPUSH/RPUSH`: O(k) inserts, no reindexing
-* `LPOP/RPOP`: O(k) deletes + selects for return
-* `LLEN`: O(1)
-* `LRANGE`: O(n) in returned range
-
-### B.9 Tests (Required)
-
-For each command above:
-
-* Happy path
-* Non-existent key behavior
-* Wrong-type behavior
-* TTL interaction (expires then acts as missing)
-* Persistence across restart
-* Binary safety for values
-* Concurrency sanity (multiple clients pushing/popping does not corrupt meta)
-
----
-
-## Appendix C: Sorted Sets / ZSET (Type = zset)
-
-### C.1 Goals
-
-* Provide a Redis-compatible subset of ZSET commands with efficient range and score queries.
-* Persist in SQLite with appropriate indexing.
-* Keep semantics close to Redis for ordering, score ties, and missing elements.
-
-### C.2 Supported Commands (vNext)
-
-Recommended minimal set:
-
-* `ZADD key [NX|XX] [CH] [INCR] score member [score member ...]` (start with a reduced subset)
-* `ZREM key member [member ...]`
-* `ZCARD key`
-* `ZSCORE key member`
-* `ZRANGE key start stop [WITHSCORES]`
-* `ZREVRANGE key start stop [WITHSCORES]` (optional but useful)
-* `ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]`
-* `ZREMRANGEBYSCORE key min max` (optional, later)
-* `ZSCAN key cursor [MATCH pattern] [COUNT n]` (later)
-
-**Initial simplification for v1 of ZSET:**
-
-* Support `ZADD key score member [score member ...]` (no flags) returning number of new elements.
-* Add flags later.
-
-### C.3 Redis Semantics
-
-* Sorted set is ordered by:
-
-  1. score ascending
-  2. member lexicographically ascending as tie-breaker (Redis behavior)
-* Wrong type errors same pattern as other types.
-* Non-existent key:
-
-  * `ZCARD` => `0`
-  * `ZRANGE` => empty array
-  * `ZSCORE` => `nil`
-
-### C.4 Data Model (SQLite)
-
-```sql
-CREATE TABLE redis_zsets (
-  key BLOB NOT NULL,
-  member BLOB NOT NULL,
-  score REAL NOT NULL,
-  PRIMARY KEY (key, member),
-  FOREIGN KEY(key) REFERENCES redis_keys(key) ON DELETE CASCADE
-);
-
-CREATE INDEX redis_zsets_key_score_member_idx
-  ON redis_zsets(key, score, member);
-```
-
-Notes:
-
-* `PRIMARY KEY (key, member)` allows upsert of member score.
-* Secondary index supports score range scans and stable ordering by `(score, member)`.
-
-### C.5 Command Behavior
-
-#### C.5.1 ZADD
-
-**Minimal v1 behavior:**
-
-* `ZADD key score member [score member ...]`
-* Response: integer count of **new** members added (not updated).
-* If key does not exist: create metadata type zset.
-* For existing member: update score (does not increment return count).
-* Use one transaction:
-
-  * ensure type
-  * upsert all pairs
-  * bump key version
-
-**Later flags (optional):**
-
-* `NX`: only add new
-* `XX`: only update existing
-* `CH`: count changed elements
-* `INCR`: single member increment
-
-#### C.5.2 ZREM
-
-* Remove one or more members.
-* Response: integer number removed.
-* If zset becomes empty: delete key metadata.
-
-#### C.5.3 ZCARD
-
-* Return cardinality:
-
-  * Prefer `SELECT COUNT(*)` (acceptable).
-  * If performance needs: maintain count in a meta table (not needed initially).
-
-#### C.5.4 ZSCORE
-
-* Return bulk string representing score (Redis returns string form), or `nil`.
-* Store as `REAL`, but serialize consistently:
-
-  * Use a stable conversion (avoid scientific notation surprises if possible).
-  * Accept that exact formatting may differ from Redis; document if needed.
-
-#### C.5.5 ZRANGE (by rank)
-
-**Request:** `ZRANGE key start stop [WITHSCORES]`
-
-Rank rules like LRANGE:
-
-* start/stop inclusive
-* negative indexes from end
-* clamp
-
-Implementation:
-
-* let `len = ZCARD`
-* normalize range
-* SQL for ordering:
-
-  ```sql
-  SELECT member, score
-  FROM redis_zsets
-  WHERE key=?
-  ORDER BY score ASC, member ASC
-  LIMIT ? OFFSET ?;
-  ```
-* Response:
-
-  * without WITHSCORES: array of members
-  * with WITHSCORES: array `[member1, score1, member2, score2, ...]`
-
-#### C.5.6 ZRANGEBYSCORE
-
-**Request:** `ZRANGEBYSCORE key min max [WITHSCORES] [LIMIT offset count]`
-
-Score bounds rules:
-
-* Support numeric `min/max`.
-* Optional later: `(` exclusive bounds, `-inf`, `+inf`.
-
-Implementation:
-
-```sql
-SELECT member, score
-FROM redis_zsets
-WHERE key=? AND score >= ? AND score <= ?
-ORDER BY score ASC, member ASC
-LIMIT ? OFFSET ?;
-```
-
-Return format same as `ZRANGE`.
-
-### C.6 Expiration and Cache
-
-* TTL is in `redis_keys.expires_at`.
-* Lazy expiration removes zset rows via cascade.
-* Cache:
-
-  * Cache `ZSCORE` lookups optionally (key+member) if beneficial.
-  * Avoid caching full zsets early.
-  * Always invalidate on `ZADD/ZREM`.
-
-### C.7 Complexity Targets
-
-* `ZADD`: O(k log n) effectively via index maintenance, practical for SQLite
-* `ZRANGE`: O(m) over returned slice with index support
-* `ZRANGEBYSCORE`: O(m) over match range with index support
-* `ZSCORE`: O(log n) via PK on (key, member)
-
-### C.8 Tests (Required)
-
-* Correct ordering:
-
-  * by score, then by member for ties
-* Rank normalization:
-
-  * negative indices
-  * out of range
-  * start > stop -> empty
-* Score range:
-
-  * boundaries inclusive
-  * LIMIT behavior
-* Wrong type behavior
-* TTL interaction and lazy deletion
-* Persistence across restart
-* Binary member support
-* Concurrency sanity:
-
-  * multiple clients doing `ZADD` on same key does not corrupt ordering or counts
-
----
-
-## Small integration notes (for both LIST and ZSET)
-
-### Type constants
-
-Extend `redis_keys.type` enum:
-
-* `4 = list`
-* `5 = zset`
-
-### Wrong-type enforcement
-
-Any command on a key must:
-
-1. run lazy-expire check
-2. read `redis_keys.type`
-3. if mismatch, return WRONGTYPE
-
-### Key deletion when empty
-
-For list and zset:
-
-* if becomes empty, delete metadata key row (and meta/items rows if any)
-
-### SCAN behavior
-
-* SCAN should include list/zset keys automatically (it reads from `redis_keys`).

package/src/cli/import-from-redis.js

@@ -1,194 +0,0 @@
-/**
- * Import data from Redis into RESPlite SQLite DB (SPEC §26).
- * External CLI: connect to Redis, SCAN keys, TYPE, fetch by type, PTTL, write to SQLite.
- * Usage: node src/cli/import-from-redis.js --redis-url redis://127.0.0.1:6379 --db ./data.db
- */
-
-import { createClient } from 'redis';
-import { fileURLToPath } from 'node:url';
-import { openDb } from '../storage/sqlite/db.js';
-import { createKeysStorage } from '../storage/sqlite/keys.js';
-import { createStringsStorage } from '../storage/sqlite/strings.js';
-import { createHashesStorage } from '../storage/sqlite/hashes.js';
-import { createSetsStorage } from '../storage/sqlite/sets.js';
-import { createListsStorage } from '../storage/sqlite/lists.js';
-import { createZsetsStorage } from '../storage/sqlite/zsets.js';
-import { asKey, asValue } from '../util/buffers.js';
-
-const SUPPORTED_TYPES = new Set(['string', 'hash', 'set', 'list', 'zset']);
-
-function parseArgs() {
-  const args = process.argv.slice(2);
-  let redisUrl = null;
-  let host = '127.0.0.1';
-  let port = 6379;
-  let dbPath = null;
-  let pragmaTemplate = 'default';
-
-  for (let i = 0; i < args.length; i++) {
-    if (args[i] === '--redis-url' && args[i + 1]) {
-      redisUrl = args[++i];
-    } else if (args[i] === '--host' && args[i + 1]) {
-      host = args[++i];
-    } else if (args[i] === '--port' && args[i + 1]) {
-      port = parseInt(args[++i], 10);
-    } else if (args[i] === '--db' && args[i + 1]) {
-      dbPath = args[++i];
-    } else if (args[i] === '--pragma-template' && args[i + 1]) {
-      pragmaTemplate = args[++i];
-    }
-  }
-
-  if (!dbPath) {
-    console.error('Usage: node import-from-redis.js --db <path> [--redis-url <url> | --host <host> --port <port>] [--pragma-template <name>]');
-    process.exit(1);
-  }
-
-  return {
-    redisUrl: redisUrl || `redis://${host}:${port}`,
-    dbPath,
-    pragmaTemplate,
-  };
-}
-
-function toBuffer(value) {
-  if (value == null) return null;
-  if (Buffer.isBuffer(value)) return value;
-  if (typeof value === 'string') return Buffer.from(value, 'utf8');
-  return Buffer.from(String(value), 'utf8');
-}
-
-/**
- * Normalize scan result: node-redis can return { cursor, keys } or [cursor, keys].
- */
-function parseScanResult(result) {
-  if (Array.isArray(result)) {
-    return { cursor: parseInt(result[0], 10), keys: result[1] || [] };
-  }
-  if (result && typeof result === 'object') {
-    const cursor = typeof result.cursor === 'number' ? result.cursor : parseInt(String(result.cursor), 10);
-    const keys = result.keys || [];
-    return { cursor, keys };
-  }
-  return { cursor: 0, keys: [] };
-}
-
-async function importFromRedis(redisClient, dbPath, options = {}) {
-  const { pragmaTemplate = 'default' } = options;
-  const db = openDb(dbPath, { pragmaTemplate });
-  const keys = createKeysStorage(db);
-  const strings = createStringsStorage(db, keys);
-  const hashes = createHashesStorage(db, keys);
-  const sets = createSetsStorage(db, keys);
-  const lists = createListsStorage(db, keys);
-  const zsets = createZsetsStorage(db, keys);
-
-  const now = Date.now();
-  const stats = { string: 0, hash: 0, set: 0, list: 0, zset: 0, skipped: 0, errors: 0 };
-
-  let cursor = 0;
-  do {
-    const result = await redisClient.scan(cursor);
-    const parsed = parseScanResult(result);
-    cursor = parsed.cursor;
-    const keyList = parsed.keys || [];
-
-    for (const keyName of keyList) {
-      try {
-        const type = (await redisClient.type(keyName)).toLowerCase();
-        if (!SUPPORTED_TYPES.has(type)) {
-          stats.skipped++;
-          continue;
-        }
-
-        let pttl = await redisClient.pTTL(keyName);
-        if (pttl === -2) pttl = -1;
-        const expiresAt = pttl > 0 ? now + pttl : null;
-        const keyBuf = asKey(keyName);
-
-        if (type === 'string') {
-          const value = await redisClient.get(keyName);
-          if (value !== undefined && value !== null) {
-            strings.set(keyBuf, asValue(value), { expiresAt, updatedAt: now });
-            stats.string++;
-          }
-        } else if (type === 'hash') {
-          const obj = await redisClient.hGetAll(keyName);
-          if (obj && typeof obj === 'object') {
-            const pairs = [];
-            for (const [f, v] of Object.entries(obj)) {
-              pairs.push(toBuffer(f), toBuffer(v));
-            }
-            if (pairs.length) {
-              hashes.setMultiple(keyBuf, pairs, { updatedAt: now });
-              keys.setExpires(keyBuf, expiresAt, now);
-              stats.hash++;
-            }
-          }
-        } else if (type === 'set') {
-          const members = await redisClient.sMembers(keyName);
-          if (members && members.length) {
-            const memberBuffers = members.map((m) => toBuffer(m));
-            sets.add(keyBuf, memberBuffers, { updatedAt: now });
-            keys.setExpires(keyBuf, expiresAt, now);
-            stats.set++;
-          }
-        } else if (type === 'list') {
-          const elements = await redisClient.lRange(keyName, 0, -1);
-          if (elements && elements.length) {
-            const valueBuffers = elements.map((e) => toBuffer(e));
-            lists.rpush(keyBuf, valueBuffers, { updatedAt: now });
-            keys.setExpires(keyBuf, expiresAt, now);
-            stats.list++;
-          }
-        } else if (type === 'zset') {
-          const withScores = await redisClient.zRangeWithScores(keyName, 0, -1);
-          if (withScores && withScores.length) {
-            const pairs = withScores.map((item) => ({
-              member: toBuffer(item.value),
-              score: Number(item.score),
-            }));
-            zsets.add(keyBuf, pairs, { updatedAt: now });
-            keys.setExpires(keyBuf, expiresAt, now);
-            stats.zset++;
-          }
-        }
-      } catch (err) {
-        stats.errors++;
-        console.error(`Error importing key "${keyName}":`, err.message);
-      }
-    }
-  } while (cursor !== 0);
-
-  return { db, stats };
-}
-
-async function main() {
-  const { redisUrl, dbPath, pragmaTemplate } = parseArgs();
-
-  const client = createClient({ url: redisUrl });
-  client.on('error', (err) => console.error('Redis client error:', err.message));
-
-  await client.connect();
-
-  try {
-    console.log(`Importing from ${redisUrl} into ${dbPath} ...`);
-    const { stats } = await importFromRedis(client, dbPath, { pragmaTemplate });
-    console.log('Import complete.');
-    console.log(`  strings: ${stats.string}, hashes: ${stats.hash}, sets: ${stats.set}, lists: ${stats.list}, zsets: ${stats.zset}`);
-    if (stats.skipped) console.log(`  skipped (unsupported type): ${stats.skipped}`);
-    if (stats.errors) console.log(`  errors: ${stats.errors}`);
-  } finally {
-    await client.quit();
-  }
-}
-
-const isMain = process.argv[1] === fileURLToPath(import.meta.url);
-if (isMain) {
-  main().catch((err) => {
-    console.error(err);
-    process.exit(1);
-  });
-}
-
-export { importFromRedis, parseScanResult, parseArgs };

package/src/cli/resplite-dirty-tracker.js

@@ -1,92 +0,0 @@
-#!/usr/bin/env node
-/**
- * resplite-dirty-tracker (SPEC_F §F.6): subscribe to Redis keyspace notifications, record dirty keys in SQLite.
- * Usage: resplite-dirty-tracker start|stop [options]
- */
-
-import { fileURLToPath } from 'node:url';
-import { openDb } from '../storage/sqlite/db.js';
-import { getRun, setRunStatus, RUN_STATUS } from '../migration/registry.js';
-import { startDirtyTracker } from '../migration/tracker.js';
-
-function parseArgs(argv = process.argv.slice(2)) {
-  const args = { _: [] };
-  for (let i = 0; i < argv.length; i++) {
-    const arg = argv[i];
-    if (arg === '--from' && argv[i + 1]) args.from = argv[++i];
-    else if (arg === '--to' && argv[i + 1]) args.to = argv[++i];
-    else if (arg === '--run-id' && argv[i + 1]) args.runId = argv[++i];
-    else if (arg === '--channels' && argv[i + 1]) args.channels = argv[++i];
-    else if (arg === '--pragma-template' && argv[i + 1]) args.pragmaTemplate = argv[++i];
-    else if (arg === '--config-command' && argv[i + 1]) args.configCommand = argv[++i];
-    else if (!arg.startsWith('--')) args._.push(arg);
-  }
-  return args;
-}
-
-async function startTrackerCli(args) {
-  const redisUrl = args.from || process.env.RESPLITE_IMPORT_FROM || 'redis://127.0.0.1:6379';
-  const dbPath = args.to;
-  const runId = args.runId || process.env.RESPLITE_RUN_ID;
-  if (!dbPath || !runId) {
-    console.error('Usage: resplite-dirty-tracker start --run-id <id> --to <db-path> [--from <redis-url>] [--config-command <name>]');
-    process.exit(1);
-  }
-
-  const tracker = await startDirtyTracker({
-    from: redisUrl,
-    to: dbPath,
-    runId,
-    pragmaTemplate: args.pragmaTemplate || 'default',
-    configCommand: args.configCommand || 'CONFIG',
-  });
-
-  console.log('Subscribed to __keyevent@0__:* — dirty tracker running. Ctrl+C to stop.');
-
-  const shutdown = async () => {
-    console.log('Stopping dirty tracker...');
-    await tracker.stop();
-    process.exit(0);
-  };
-
-  process.on('SIGINT', shutdown);
-  process.on('SIGTERM', shutdown);
-}
-
-async function stopTracker(args) {
-  const dbPath = args.to;
-  const runId = args.runId || process.env.RESPLITE_RUN_ID;
-  if (!dbPath || !runId) {
-    console.error('Usage: resplite-dirty-tracker stop --run-id <id> --to <db-path>');
-    process.exit(1);
-  }
-  const db = openDb(dbPath, { pragmaTemplate: args.pragmaTemplate || 'default' });
-  const run = getRun(db, runId);
-  if (run && run.status === RUN_STATUS.RUNNING) {
-    setRunStatus(db, runId, RUN_STATUS.PAUSED);
-    console.log('Run', runId, 'status set to paused. Tracker process must be stopped with Ctrl+C if still running.');
-  } else {
-    console.log('Run', runId, 'status:', run?.status ?? 'not found');
-  }
-}
-
-async function main() {
-  const args = parseArgs();
-  const sub = args._[0];
-  if (sub === 'start') await startTrackerCli(args);
-  else if (sub === 'stop') await stopTracker(args);
-  else {
-    console.error('Usage: resplite-dirty-tracker <start|stop> --run-id <id> --to <db-path> [--from <redis-url>]');
-    process.exit(1);
-  }
-}
-
-const isMain = process.argv[1] === fileURLToPath(import.meta.url) || process.argv[1]?.endsWith('resplite-dirty-tracker.js');
-if (isMain) {
-  main().catch((e) => {
-    console.error(e);
-    process.exit(1);
-  });
-}
-
-export { parseArgs, stopTracker };