resplite 1.0.2 → 1.0.6

package/README.md

@@ -35,6 +35,24 @@ OK
 "bar"
 ```
 
+### Standalone server script (fixed port)
+
+Run this as a persistent background process (`node server.js`). RESPLite will listen on port 6380 and stay up until the process is killed.
+
+```javascript
+// server.js
+import { createRESPlite } from 'resplite/embed';
+
+const srv = await createRESPlite({ port: 6380, db: './data.db' });
+console.log(`RESPLite listening on ${srv.host}:${srv.port}`);
+```
+
+Then connect from any other script or process:
+
+```bash
+redis-cli -p 6380 PING
+```
+
 ### Environment variables
 
 | Variable | Default | Description |
@@ -141,6 +159,23 @@ console.log(await client.lPop('queue'));           // → "a"
 console.log(await client.rPop('queue'));           // → "e"
 ```
 
+### Blocking list commands (BLPOP / BRPOP)
+
+`BLPOP` and `BRPOP` block until an element is available or a timeout (seconds) is reached. Use them for simple queues or coordination between producers and consumers.
+
+```javascript
+// Consumer: block up to 10 seconds for an element from "tasks" or "fallback"
+const result = await client.blPop(['tasks', 'fallback'], 10);
+// result is { key: 'tasks', element: 'item1' } or null on timeout
+
+// Producer (e.g. another client or process)
+await client.rPush('tasks', 'item1');
+```
+
+- **Timeout**: `0` = block indefinitely; `> 0` = block up to that many seconds.
+- **Return**: `{ key, element }` on success, or `null` on timeout.
+- **Multi-key**: Keys are checked in order; the first key that has an element wins. One push wakes at most one blocked client (FIFO per key).
+
 ### Sorted sets
 
 ```javascript
@@ -239,7 +274,7 @@ await srv2.close();
 | **TTL** | EXPIRE, PEXPIRE, TTL, PTTL, PERSIST |
 | **Hashes** | HSET, HGET, HMGET, HGETALL, HDEL, HEXISTS, HINCRBY |
 | **Sets** | SADD, SREM, SMEMBERS, SISMEMBER, SCARD |
-| **Lists** | LPUSH, RPUSH, LLEN, LRANGE, LINDEX, LPOP, RPOP |
+| **Lists** | LPUSH, RPUSH, LLEN, LRANGE, LINDEX, LPOP, RPOP, BLPOP, BRPOP |
 | **Sorted sets** | ZADD, ZREM, ZCARD, ZSCORE, ZRANGE, ZRANGEBYSCORE |
 | **Search (FT.\*)** | FT.CREATE, FT.INFO, FT.ADD, FT.DEL, FT.SEARCH, FT.SUGADD, FT.SUGGET, FT.SUGDEL |
 | **Introspection** | TYPE, SCAN |
@@ -252,14 +287,18 @@ await srv2.close();
 - Streams (XADD, XRANGE, etc.)
 - Lua (EVAL, EVALSHA)
 - Transactions (MULTI, EXEC, WATCH)
-- Blocking commands (BLPOP, etc.)
+- BRPOPLPUSH, BLMOVE (blocking list moves)
 - SELECT (multiple logical DBs)
 
 Unsupported commands return: `ERR command not supported yet`.
 
 ## Migration from Redis
 
-An external CLI imports data from a running Redis instance into a RESPlite SQLite DB. Supports strings, hashes, sets, lists, zsets, and TTL. Requires a local or remote Redis and the `redis` npm package (dev dependency).
+Migration supports two modes:
+
+### Simple one-shot import (legacy)
+
+For small datasets or when downtime is acceptable:
 
 ```bash
 # Default: redis://127.0.0.1:6379 → ./data.db
@@ -275,7 +314,143 @@ npm run import-from-redis -- --db ./migrated.db --host 127.0.0.1 --port 6379
 npm run import-from-redis -- --db ./migrated.db --pragma-template performance
 ```
 
-Then start RESPLite with the migrated DB: `RESPLITE_DB=./migrated.db npm start`.
+### Redis with authentication
+
+Migration supports Redis instances protected by a password. Use a Redis URL that includes the password (or username and password for Redis 6+ ACL):
+
+- **Password only:** `redis://:PASSWORD@host:port`
+- **Username and password:** `redis://username:PASSWORD@host:port`
+
+Examples:
+
+```bash
+# One-shot import from authenticated Redis
+npm run import-from-redis -- --db ./migrated.db --redis-url "redis://:mysecret@127.0.0.1:6379"
+
+# SPEC_F flow: use --from with the full URL (or set RESPLITE_IMPORT_FROM)
+npx resplite-import preflight --from "redis://:mysecret@10.0.0.10:6379" --to ./resplite.db
+npx resplite-dirty-tracker start --run-id run_001 --from "redis://:mysecret@10.0.0.10:6379" --to ./resplite.db
+```
+
+For one-shot import, authentication is only available when using `--redis-url`; the `--host` / `--port` options do not support a password.
+
+### Minimal-downtime migration (SPEC_F)
+
+For large datasets (~30 GB), use the Dirty Key Registry flow so the bulk of the migration runs online and only a short cutover is needed.
+
+**Enable keyspace notifications in Redis** (required for the dirty-key tracker). Either run at runtime:
+
+```bash
+redis-cli CONFIG SET notify-keyspace-events Kgxe
+```
+
+Or add to `redis.conf` and restart Redis:
+
+```
+notify-keyspace-events Kgxe
+```
+
+(`K` = keyspace, `g` = generic, `x` = expired; `e` = keyevent. This lets the tracker see key changes and expirations.)
+
+1. **Preflight** – Check Redis, key count, type distribution, and that keyspace notifications are enabled:
+   ```bash
+   npx resplite-import preflight --from redis://10.0.0.10:6379 --to ./resplite.db
+   ```
+
+2. **Start dirty-key tracker** – Captures keys modified during bulk (requires `notify-keyspace-events` in Redis):
+   ```bash
+   npx resplite-dirty-tracker start --run-id run_001 --from redis://10.0.0.10:6379 --to ./resplite.db
+   ```
+
+3. **Bulk import** – SCAN and copy all keys; progress is checkpointed and resumable:
+   ```bash
+   npx resplite-import bulk --run-id run_001 --from redis://10.0.0.10:6379 --to ./resplite.db \
+     --scan-count 1000 --max-rps 2000 --batch-keys 200 --batch-bytes 64MB --resume
+   ```
+
+4. **Monitor** – Check run and dirty-key counts:
+   ```bash
+   npx resplite-import status --run-id run_001 --to ./resplite.db
+   ```
+
+5. **Cutover** – Freeze app writes to Redis, then apply remaining dirty keys:
+   ```bash
+   npx resplite-import apply-dirty --run-id run_001 --from redis://10.0.0.10:6379 --to ./resplite.db
+   ```
+
+6. **Stop tracker and switch** – Stop the tracker and point clients to RespLite:
+   ```bash
+   npx resplite-dirty-tracker stop --run-id run_001 --to ./resplite.db
+   ```
+
+7. **Verify** – Optional sampling check between Redis and destination:
+   ```bash
+   npx resplite-import verify --run-id run_001 --from redis://10.0.0.10:6379 --to ./resplite.db --sample 0.5%
+   ```
+
+Then start RespLite with the migrated DB: `RESPLITE_DB=./resplite.db npm start`.
+
+### Programmatic migration API
+
+As an alternative to the CLI, the full migration flow is available as a JavaScript API via `resplite/migration`. Useful for embedding the migration inside your own scripts or automation pipelines.
+
+```javascript
+import { createMigration } from 'resplite/migration';
+
+const m = createMigration({
+  from:  'redis://127.0.0.1:6379',  // source Redis URL (default)
+  to:    './resplite.db',           // destination SQLite DB path (required)
+  runId: 'my-migration-1',          // unique run ID (required for bulk/status/applyDirty)
+
+  // optional — same defaults as the CLI:
+  scanCount:      1000,
+  batchKeys:      200,
+  batchBytes:     64 * 1024 * 1024,  // 64 MB
+  maxRps:         0,                  // 0 = unlimited
+  pragmaTemplate: 'default',
+});
+
+// Step 0 — Preflight: inspect Redis before starting
+const info = await m.preflight();
+console.log('keys (estimate):', info.keyCountEstimate);
+console.log('type distribution:', info.typeDistribution);
+console.log('recommended params:', info.recommended);
+
+// Step 1 — Bulk import (checkpointed, resumable)
+await m.bulk({
+  resume: false,                            // true to resume a previous run
+  onProgress: (r) => console.log(
+    `scanned=${r.scanned_keys} migrated=${r.migrated_keys} errors=${r.error_keys}`
+  ),
+});
+
+// Check status at any point (synchronous, no Redis needed)
+const { run, dirty } = m.status();
+console.log('bulk status:', run.status, '— dirty counts:', dirty);
+
+// Step 2 — Apply dirty keys that changed in Redis during bulk
+await m.applyDirty();
+
+// Step 3 — Verify a sample of keys match between Redis and the destination
+const result = await m.verify({ samplePct: 0.5, maxSample: 10000 });
+console.log(`verified ${result.sampled} keys — mismatches: ${result.mismatches.length}`);
+
+// Disconnect Redis when done
+await m.close();
+```
+
+The dirty-key tracker (to capture writes during bulk) still runs as a separate process via `npx resplite-dirty-tracker`. The API above handles everything else in a single script.
+
+#### Low-level re-exports
+
+If you need more control, the individual functions and registry helpers are also exported:
+
+```javascript
+import {
+  runPreflight, runBulkImport, runApplyDirty, runVerify,
+  getRun, getDirtyCounts, createRun, setRunStatus, logError,
+} from 'resplite/migration';
+```
 
 ## Benchmark (Redis vs RESPLite)
 
@@ -303,7 +478,9 @@ npm run benchmark -- --iterations 10000 --redis-port 6379 --resplite-port 6380
 | `npm run test:contract` | Contract tests (redis client) |
 | `npm run test:stress` | Stress tests |
 | `npm run benchmark` | Comparative benchmark Redis vs RESPLite |
-| `npm run import-from-redis` | Import from Redis into a SQLite DB |
+| `npm run import-from-redis` | One-shot import from Redis into a SQLite DB |
+| `npx resplite-import` (preflight, bulk, status, apply-dirty, verify) | Migration CLI (SPEC_F minimal-downtime flow) |
+| `npx resplite-dirty-tracker <start\|stop>` | Dirty-key tracker for migration cutover |
 
 ## Specification
 

package/package.json

@@ -1,12 +1,17 @@
 {
   "name": "resplite",
-  "version": "1.0.2",
+  "version": "1.0.6",
   "description": "A RESP2 server with practical Redis compatibility, backed by SQLite",
   "type": "module",
   "main": "src/index.js",
+  "bin": {
+    "resplite-import": "src/cli/resplite-import.js",
+    "resplite-dirty-tracker": "src/cli/resplite-dirty-tracker.js"
+  },
   "exports": {
     ".": "./src/index.js",
-    "./embed": "./src/embed.js"
+    "./embed": "./src/embed.js",
+    "./migration": "./src/migration/index.js"
   },
   "scripts": {
     "start": "node src/index.js",