resplite 1.0.6 → 1.1.4

package/README.md

@@ -1,15 +1,63 @@
 # RESPLite
 
-A RESP2 server with practical Redis compatibility, backed by SQLite for persistent single-node workloads.
+A RESP server backed by SQLite. Compatible with `redis` clients and `redis-cli`, persistent by default, zero external daemons, and minimal memory footprint.
 
 ## Overview
 
-RESPLite is not a full Redis clone. It is a RESP server with Redis-like semantics for a carefully selected subset of commands that map naturally to SQLite. Ideal for small to medium applications, persistent caches, local development, and low-ops deployments.
+RESPLite speaks **RESP** (the Redis Serialization Protocol), so your existing `redis` npm client and `redis-cli` work without changes. The storage layer is **SQLite**: WAL mode, FTS5 for full-text search, and a single `.db` file that survives restarts without snapshots or AOF.
 
-- **Zero external services** — just Node.js and a SQLite file.
+It is not a Redis clone. It covers a practical subset of commands that map naturally to SQLite, suited for single-node workloads where Redis' in-memory latency is not a hard requirement.
+
+- **Zero external services** — just Node.js and a `.db` file.
 - **Drop-in compatible** — works with the official `redis` npm client and `redis-cli`.
-- **Persistent by default** — data survives restarts without snapshots or AOF.
-- **Embeddable** — start the server and connect from the same script (see examples below).
+- **Persistent by default** — no snapshots, no AOF, no config.
+- **Embeddable** — start the server and connect from the same script.
+- **Full-text search** — FT.\* commands via SQLite FTS5.
+- **Simple queues** — lists with BLPOP/BRPOP.
+
+### When RESPLite beats Redis in Docker
+
+Building this project surfaced a clear finding: **Redis running inside Docker** on the same host often has **worse latency** than **RESPLite running locally**. Docker's virtual network adds overhead that disappears when the server runs in the same process/host. For single-node workloads this makes RESPLite the faster, simpler option.
+
+The strongest use case is **migrating a non-replicated Redis instance that has grown large** (tens of GB). You don't need to manage replicas, AOF, or RDB. Once migrated, you get a single SQLite file and latency that is good enough for most workloads. The built-in migration tooling (see [Migration from Redis](#migration-from-redis)) handles datasets of that size with minimal downtime.
+
+## Benchmark (Redis vs RESPLite)
+
+A typical comparison is **Redis (e.g. in Docker)** on one side and **RESPLite locally** on the other. In that setup, RESPLite often shows **better latency** because it avoids Docker networking and runs in the same process/host. The benchmark below uses RESPLite with the **default** PRAGMA template only.
+
+**Example results (Redis vs RESPLite, default pragma, 10k iterations):**
+
+| Suite           | Redis (Docker) | RESPLite (default) |
+|-----------------|----------------|--------------------|
+| PING            | 9.17K/s        | 27.59K/s           |
+| SET+GET         | 4.65K/s        | 9.53K/s            |
+| MSET+MGET(10)   | 4.47K/s        | 5.35K/s            |
+| INCR            | 9.84K/s        | 15.56K/s           |
+| HSET+HGET       | 4.63K/s        | 10.95K/s           |
+| HGETALL(50)     | 9.19K/s        | 8.19K/s            |
+| SADD+SMEMBERS   | 9.16K/s        | 15.54K/s           |
+| LPUSH+LRANGE    | 7.97K/s        | 10.24K/s           |
+| ZADD+ZRANGE     | 8.08K/s        | 2.89K/s            |
+| SET+DEL         | 4.60K/s        | 6.62K/s            |
+| FT.SEARCH       | 8.20K/s        | 7.20K/s            |
+
+*Run `npm run benchmark -- --template default` to reproduce. Numbers depend on host and whether Redis is native or in Docker.*
+
+How to run:
+
+```bash
+# Terminal 1: Redis on 6379 (e.g. docker run -p 6379:6379 redis). Terminal 2: RESPLite on 6380
+RESPLITE_PORT=6380 npm start
+
+# Terminal 3: run benchmark (Redis=6379, RESPLite=6380 by default)
+npm run benchmark
+
+# Only RESPLite with default pragma
+npm run benchmark -- --template default
+
+# Custom iterations and ports
+npm run benchmark -- --iterations 10000 --redis-port 6379 --resplite-port 6380
+```
 
 ## Install
 
@@ -265,6 +313,8 @@ await srv2.close();
 
 ## Compatibility matrix
 
+RESPLite implements **47 core Redis commands** (~19% of the ~246 commands in Redis 7). The uncovered 81% is mostly entire subsystems that are out of scope by design: pub/sub (~10 commands), Streams (~20), cluster/replication (~30), Lua scripting (~5), server admin (~40), and extended variants of data-structure commands. For typical single-node application workloads — strings, hashes, sets, lists, sorted sets, key TTLs — coverage is close to the commands developers reach for daily.
+
 ### Supported (v1)
 
 | Category | Commands |
@@ -294,6 +344,8 @@ Unsupported commands return: `ERR command not supported yet`.
 
 ## Migration from Redis
 
+RESPLite is a good fit for migrating **non-replicated Redis** instances that have **grown large** (e.g. tens of GB) and where RESPLite’s latency is acceptable. The SPEC_F flow (dirty-key tracker, bulk import, cutover) is designed for that scenario with minimal downtime.
+
 Migration supports two modes:
 
 ### Simple one-shot import (legacy)
@@ -341,16 +393,18 @@ For large datasets (~30 GB), use the Dirty Key Registry flow so the bulk of the
 **Enable keyspace notifications in Redis** (required for the dirty-key tracker). Either run at runtime:
 
 ```bash
-redis-cli CONFIG SET notify-keyspace-events Kgxe
+redis-cli CONFIG SET notify-keyspace-events KEA
 ```
 
 Or add to `redis.conf` and restart Redis:
 
 ```
-notify-keyspace-events Kgxe
+notify-keyspace-events KEA
 ```
 
-(`K` = keyspace, `g` = generic, `x` = expired; `e` = keyevent. This lets the tracker see key changes and expirations.)
+(`K` = keyspace prefix, `E` = keyevent prefix, `A` = all event types — lets the tracker see every key change and expiration.)
+
+> **Renamed CONFIG command?** Some Redis deployments rename `CONFIG` for security. Pass `--config-command <name>` to the CLI tools, or the `configCommand` option to the JS API — see below.
 
 1. **Preflight** – Check Redis, key count, type distribution, and that keyspace notifications are enabled:
    ```bash
@@ -360,6 +414,8 @@ notify-keyspace-events Kgxe
 2. **Start dirty-key tracker** – Captures keys modified during bulk (requires `notify-keyspace-events` in Redis):
    ```bash
    npx resplite-dirty-tracker start --run-id run_001 --from redis://10.0.0.10:6379 --to ./resplite.db
+   # If CONFIG was renamed:
+   npx resplite-dirty-tracker start --run-id run_001 --from redis://10.0.0.10:6379 --to ./resplite.db --config-command MYCONFIG
    ```
 
 3. **Bulk import** – SCAN and copy all keys; progress is checkpointed and resumable:
@@ -408,14 +464,25 @@ const m = createMigration({
   batchBytes:     64 * 1024 * 1024,  // 64 MB
   maxRps:         0,                  // 0 = unlimited
   pragmaTemplate: 'default',
+
+  // If your Redis deployment renamed CONFIG for security:
+  // configCommand: 'MYCONFIG',
 });
 
 // Step 0 — Preflight: inspect Redis before starting
 const info = await m.preflight();
 console.log('keys (estimate):', info.keyCountEstimate);
 console.log('type distribution:', info.typeDistribution);
+console.log('notify-keyspace-events:', info.notifyKeyspaceEvents);
+console.log('CONFIG available:', info.configCommandAvailable);  // false if renamed
 console.log('recommended params:', info.recommended);
 
+// Step 0b — Enable keyspace notifications (required for dirty-key tracking)
+// Reads the current value and merges the new flags — existing flags are preserved.
+const ks = await m.enableKeyspaceNotifications();
+// → { ok: true, previous: '', applied: 'KEA' }
+// If CONFIG is renamed and configCommand was not set, ok=false and error explains how to fix it.
+
 // Step 1 — Bulk import (checkpointed, resumable)
 await m.bulk({
   resume: false,                            // true to resume a previous run
@@ -441,30 +508,42 @@ await m.close();
 
 The dirty-key tracker (to capture writes during bulk) still runs as a separate process via `npx resplite-dirty-tracker`. The API above handles everything else in a single script.
 
-#### Low-level re-exports
+#### Renamed CONFIG command
 
-If you need more control, the individual functions and registry helpers are also exported:
+If your Redis instance has the `CONFIG` command renamed (a common hardening practice), pass the new name to `createMigration`:
 
 ```javascript
-import {
-  runPreflight, runBulkImport, runApplyDirty, runVerify,
-  getRun, getDirtyCounts, createRun, setRunStatus, logError,
-} from 'resplite/migration';
-```
+const m = createMigration({
+  from: 'redis://10.0.0.10:6379',
+  to:   './resplite.db',
+  runId: 'run_001',
+  configCommand: 'MYCONFIG',  // the renamed command
+});
 
-## Benchmark (Redis vs RESPLite)
+// preflight will use MYCONFIG GET notify-keyspace-events
+const info = await m.preflight();
+// info.configCommandAvailable → false if the name is wrong
+
+// enableKeyspaceNotifications will use MYCONFIG SET notify-keyspace-events KEA
+const result = await m.enableKeyspaceNotifications({ value: 'KEA' });
+```
 
-Compare throughput of local Redis and RESPLite with the same workload (PING, SET/GET, hashes, sets, lists, zsets, etc.):
+The same flag is available in the CLI:
 
 ```bash
-# Terminal 1: Redis on 6379 (default). Terminal 2: RESPLite on 6380
-RESPLITE_PORT=6380 npm start
+npx resplite-dirty-tracker start --run-id run_001 --to ./resplite.db \
+  --from redis://10.0.0.10:6379 --config-command MYCONFIG
+```
 
-# Terminal 3: run benchmark (Redis=6379, RESPLite=6380 by default)
-npm run benchmark
+#### Low-level re-exports
 
-# Optional: custom iterations and ports
-npm run benchmark -- --iterations 10000 --redis-port 6379 --resplite-port 6380
+If you need more control, the individual functions and registry helpers are also exported:
+
+```javascript
+import {
+  runPreflight, runBulkImport, runApplyDirty, runVerify,
+  getRun, getDirtyCounts, createRun, setRunStatus, logError,
+} from 'resplite/migration';
 ```
 
 ## Scripts

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "resplite",
-  "version": "1.0.6",
+  "version": "1.1.4",
   "description": "A RESP2 server with practical Redis compatibility, backed by SQLite",
   "type": "module",
   "main": "src/index.js",
@@ -24,6 +24,25 @@
     "benchmark": "node scripts/benchmark-redis-vs-resplite.js",
     "test:all": "node --test 'test/**/*.test.js'"
   },
+  "keywords": [
+    "redis",
+    "sqlite",
+    "resp",
+    "emulator",
+    "sdd",
+    "persist",
+    "cache",
+    "key-value",
+    "store",
+    "embedded",
+    "server",
+    "valkey",
+    "compatible",
+    "dragonfly",
+    "keydb",
+    "garnet",
+    "clasen"
+  ],
   "dependencies": {
     "better-sqlite3": "^11.6.0"
   },

package/scripts/benchmark-redis-vs-resplite.js

@@ -1,15 +1,15 @@
 #!/usr/bin/env node
 /**
- * Comparative benchmark: Redis (local) vs RESPlite (all PRAGMA templates).
+ * Comparative benchmark: Redis (local) vs RESPlite (all or one PRAGMA template).
  *
  * Prerequisites:
  *   - Redis running on port 6379 (default)
  *
- * The script spawns one RESPlite process per PRAGMA template (default, performance, safety, minimal)
- * on consecutive ports (6380, 6381, 6382, 6383 by default) and runs the same workload against each.
+ * By default the script spawns one RESPlite process per PRAGMA template (default, performance, safety, minimal)
+ * on consecutive ports. Use --template <name> to run only one template (e.g. default).
  *
  * Usage:
- *   node scripts/benchmark-redis-vs-resplite.js [--iterations N] [--redis-port P] [--resplite-port P]
+ *   node scripts/benchmark-redis-vs-resplite.js [--iterations N] [--redis-port P] [--resplite-port P] [--template NAME]
  */
 
 import { createClient } from 'redis';
@@ -26,8 +26,11 @@ const DEFAULTS = {
   iterations: 10000,
   redisPort: 6379,
   resplitePort: 6380,
+  template: null, // null = all templates (except none); or 'default' | 'performance' | 'safety' | 'minimal'
 };
 
+const VALID_TEMPLATES = ['default', 'performance', 'safety', 'minimal'];
+
 function parseArgs() {
   const out = { ...DEFAULTS };
   const args = process.argv.slice(2);
@@ -38,6 +41,13 @@ function parseArgs() {
       out.redisPort = parseInt(args[++i], 10);
     } else if (args[i] === '--resplite-port' && args[i + 1]) {
       out.resplitePort = parseInt(args[++i], 10);
+    } else if (args[i] === '--template' && args[i + 1]) {
+      const name = args[++i];
+      if (!VALID_TEMPLATES.includes(name)) {
+        console.error(`Invalid --template "${name}". Must be one of: ${VALID_TEMPLATES.join(', ')}`);
+        process.exit(1);
+      }
+      out.template = name;
     }
   }
   return out;
@@ -204,6 +214,13 @@ async function benchHgetall(client, n) {
   for (let i = 0; i < n; i++) await client.hGetAll(key);
 }
 
+async function benchHlen(client, n) {
+  const key = 'bm:hash:hlen';
+  await client.del(key);
+  await client.hSet(key, Object.fromEntries(Array.from({ length: 50 }, (_, i) => [`f${i}`, `v${i}`])));
+  for (let i = 0; i < n; i++) await client.hLen(key);
+}
+
 async function benchSaddSmembers(client, n) {
   const key = 'bm:set';
   for (let i = 0; i < n; i++) {
@@ -221,6 +238,18 @@ async function benchLpushLrange(client, n) {
   }
 }
 
+async function benchLrem(client, n) {
+  const key = 'bm:list:lrem';
+  await client.del(key);
+  // Pre-populate with a mix of values so LREM always finds something to do.
+  // Each iteration pushes one 'target' element and removes it — net-zero list size.
+  await client.rPush(key, Array.from({ length: 20 }, (_, i) => `item-${i}`));
+  for (let i = 0; i < n; i++) {
+    await client.rPush(key, 'target');
+    await client.lRem(key, 1, 'target');
+  }
+}
+
 async function benchZaddZrange(client, n) {
   const key = 'bm:zset';
   for (let i = 0; i < n; i++) {
@@ -277,8 +306,10 @@ const SUITES = [
   { name: 'INCR', fn: benchIncr, iterScale: 1 },
   { name: 'HSET+HGET', fn: benchHsetHget, iterScale: 1 },
   { name: 'HGETALL(50)', fn: benchHgetall, iterScale: 1 },
+  { name: 'HLEN(50)', fn: benchHlen, iterScale: 1 },
   { name: 'SADD+SMEMBERS', fn: benchSaddSmembers, iterScale: 1 },
   { name: 'LPUSH+LRANGE', fn: benchLpushLrange, iterScale: 1 },
+  { name: 'LREM', fn: benchLrem, iterScale: 1 },
   { name: 'ZADD+ZRANGE', fn: benchZaddZrange, iterScale: 1 },
   { name: 'SET+DEL', fn: benchDel, iterScale: 1 },
   { name: 'FT.SEARCH', fn: benchFtSearch, iterScale: 1 },
@@ -304,15 +335,17 @@ async function runSuite(redis, respliteClients, suite, iterations) {
 }
 
 async function main() {
-  const { iterations, redisPort, resplitePort } = parseArgs();
-  const templateNames = getPragmaTemplateNames().filter((t) => t !== 'none');
+  const { iterations, redisPort, resplitePort, template } = parseArgs();
+  const templateNames = template
+    ? [template]
+    : getPragmaTemplateNames().filter((t) => t !== 'none');
 
   const benchTmpDir = path.join(PROJECT_ROOT, 'tmp', 'bench');
   fs.mkdirSync(benchTmpDir, { recursive: true });
 
-  console.log('Benchmark: Redis vs RESPlite (all PRAGMA templates)');
+  console.log(template ? `Benchmark: Redis vs RESPlite (template: ${template})` : 'Benchmark: Redis vs RESPlite (all PRAGMA templates)');
   console.log(`  Redis:    127.0.0.1:${redisPort}`);
-  console.log(`  RESPlite: one process per template on ports ${resplitePort}..${resplitePort + templateNames.length - 1}`);
+  console.log(`  RESPlite: ${templateNames.length} process(es) on port(s) ${resplitePort}${templateNames.length > 1 ? `..${resplitePort + templateNames.length - 1}` : ''}`);
   console.log(`  Templates: ${templateNames.join(', ')}`);
   console.log(`  Iterations per suite: ${iterations}`);
   console.log('');

package/src/cli/resplite-dirty-tracker.js

@@ -4,10 +4,10 @@
  * Usage: resplite-dirty-tracker start|stop [options]
  */
 
-import { createClient } from 'redis';
 import { fileURLToPath } from 'node:url';
 import { openDb } from '../storage/sqlite/db.js';
-import { createRun, getRun, setRunStatus, upsertDirtyKey, logError, RUN_STATUS } from '../migration/registry.js';
+import { getRun, setRunStatus, RUN_STATUS } from '../migration/registry.js';
+import { startDirtyTracker } from '../migration/tracker.js';
 
 function parseArgs(argv = process.argv.slice(2)) {
   const args = { _: [] };
@@ -18,71 +18,39 @@ function parseArgs(argv = process.argv.slice(2)) {
     else if (arg === '--run-id' && argv[i + 1]) args.runId = argv[++i];
     else if (arg === '--channels' && argv[i + 1]) args.channels = argv[++i];
     else if (arg === '--pragma-template' && argv[i + 1]) args.pragmaTemplate = argv[++i];
+    else if (arg === '--config-command' && argv[i + 1]) args.configCommand = argv[++i];
     else if (!arg.startsWith('--')) args._.push(arg);
   }
   return args;
 }
 
-async function checkKeyspaceNotifications(client) {
-  let val = null;
-  try {
-    const config = await client.configGet('notify-keyspace-events');
-    val = config && typeof config === 'object' ? config['notify-keyspace-events'] : null;
-  } catch (_) {}
-  if (!val || val === '') {
-    throw new Error('Redis notify-keyspace-events is not set. Enable it (e.g. CONFIG SET notify-keyspace-events Kgxe) for the dirty-key tracker.');
-  }
-  return val;
-}
-
-async function startTracker(args) {
+async function startTrackerCli(args) {
   const redisUrl = args.from || process.env.RESPLITE_IMPORT_FROM || 'redis://127.0.0.1:6379';
   const dbPath = args.to;
   const runId = args.runId || process.env.RESPLITE_RUN_ID;
   if (!dbPath || !runId) {
-    console.error('Usage: resplite-dirty-tracker start --run-id <id> --to <db-path> [--from <redis-url>] [--channels keyevent]');
+    console.error('Usage: resplite-dirty-tracker start --run-id <id> --to <db-path> [--from <redis-url>] [--config-command <name>]');
     process.exit(1);
   }
 
-  const db = openDb(dbPath, { pragmaTemplate: args.pragmaTemplate || 'default' });
-  createRun(db, runId, redisUrl);
-
-  const mainClient = createClient({ url: redisUrl });
-  mainClient.on('error', (e) => console.error('Redis (main):', e.message));
-  await mainClient.connect();
-  await checkKeyspaceNotifications(mainClient);
-
-  const subClient = mainClient.duplicate();
-  subClient.on('error', (e) => {
-    console.error('Redis (sub):', e.message);
-    logError(db, runId, 'dirty_apply', 'Tracker disconnect: ' + e.message, null);
+  const tracker = await startDirtyTracker({
+    from: redisUrl,
+    to: dbPath,
+    runId,
+    pragmaTemplate: args.pragmaTemplate || 'default',
+    configCommand: args.configCommand || 'CONFIG',
   });
-  await subClient.connect();
 
-  const pattern = '__keyevent@0__:*';
-  console.log('Subscribing to', pattern, '...');
-
-  await subClient.pSubscribe(pattern, (message, channel) => {
-    const event = typeof channel === 'string' ? channel.split(':').pop() : (channel && channel.toString?.())?.split(':').pop() || 'unknown';
-    const key = message;
-    try {
-      upsertDirtyKey(db, runId, key, event);
-    } catch (err) {
-      logError(db, runId, 'dirty_apply', err.message, key);
-    }
-  });
+  console.log('Subscribed to __keyevent@0__:* — dirty tracker running. Ctrl+C to stop.');
 
   const shutdown = async () => {
     console.log('Stopping dirty tracker...');
-    await subClient.pUnsubscribe(pattern);
-    await subClient.quit();
-    await mainClient.quit();
+    await tracker.stop();
     process.exit(0);
   };
 
   process.on('SIGINT', shutdown);
   process.on('SIGTERM', shutdown);
-  console.log('Dirty tracker running. Ctrl+C to stop.');
 }
 
 async function stopTracker(args) {
@@ -105,7 +73,7 @@ async function stopTracker(args) {
 async function main() {
   const args = parseArgs();
   const sub = args._[0];
-  if (sub === 'start') await startTracker(args);
+  if (sub === 'start') await startTrackerCli(args);
   else if (sub === 'stop') await stopTracker(args);
   else {
     console.error('Usage: resplite-dirty-tracker <start|stop> --run-id <id> --to <db-path> [--from <redis-url>]');
@@ -121,4 +89,4 @@ if (isMain) {
   });
 }
 
-export { parseArgs, startTracker, stopTracker };
+export { parseArgs, stopTracker };

package/src/commands/hlen.js

@@ -0,0 +1,15 @@
+/**
+ * HLEN key
+ */
+
+export function handleHlen(engine, args) {
+  if (!args || args.length < 1) {
+    return { error: "ERR wrong number of arguments for 'HLEN' command" };
+  }
+  try {
+    return engine.hlen(args[0]);
+  } catch (e) {
+    const msg = e && e.message ? e.message : String(e);
+    return { error: msg.startsWith('ERR ') ? msg : msg.startsWith('WRONGTYPE') ? msg : 'ERR ' + msg };
+  }
+}

package/src/commands/lrem.js

@@ -0,0 +1,15 @@
+/**
+ * LREM key count element
+ */
+
+export function handleLrem(engine, args) {
+  if (!args || args.length < 3) {
+    return { error: "ERR wrong number of arguments for 'LREM' command" };
+  }
+  try {
+    return engine.lrem(args[0], args[1], args[2]);
+  } catch (e) {
+    const msg = e && e.message ? e.message : String(e);
+    return { error: msg.startsWith('ERR ') ? msg : msg.startsWith('WRONGTYPE') ? msg : 'ERR ' + msg };
+  }
+}

package/src/commands/registry.js

@@ -27,6 +27,7 @@ import * as hget from './hget.js';
 import * as hmget from './hmget.js';
 import * as hgetall from './hgetall.js';
 import * as hdel from './hdel.js';
+import * as hlen from './hlen.js';
 import * as hexists from './hexists.js';
 import * as hincrby from './hincrby.js';
 import * as sadd from './sadd.js';
@@ -41,6 +42,7 @@ import * as lrange from './lrange.js';
 import * as lindex from './lindex.js';
 import * as lpop from './lpop.js';
 import * as rpop from './rpop.js';
+import * as lrem from './lrem.js';
 import * as blpop from './blpop.js';
 import * as brpop from './brpop.js';
 import * as scan from './scan.js';
@@ -87,6 +89,7 @@ const HANDLERS = new Map([
   ['HMGET', (e, a) => hmget.handleHmget(e, a)],
   ['HGETALL', (e, a) => hgetall.handleHgetall(e, a)],
   ['HDEL', (e, a) => hdel.handleHdel(e, a)],
+  ['HLEN', (e, a) => hlen.handleHlen(e, a)],
   ['HEXISTS', (e, a) => hexists.handleHexists(e, a)],
   ['HINCRBY', (e, a) => hincrby.handleHincrby(e, a)],
   ['SADD', (e, a) => sadd.handleSadd(e, a)],
@@ -101,6 +104,7 @@ const HANDLERS = new Map([
   ['LINDEX', (e, a) => lindex.handleLindex(e, a)],
   ['LPOP', (e, a, ctx) => lpop.handleLpop(e, a)],
   ['RPOP', (e, a, ctx) => rpop.handleRpop(e, a)],
+  ['LREM', (e, a) => lrem.handleLrem(e, a)],
   ['BLPOP', (e, a, ctx) => blpop.handleBlpop(e, a, ctx)],
   ['BRPOP', (e, a, ctx) => brpop.handleBrpop(e, a, ctx)],
   ['SCAN', (e, a) => scan.handleScan(e, a)],

package/src/engine/engine.js

@@ -209,6 +209,14 @@ export function createEngine(opts = {}) {
       return hashes.delete(k, fields.map((f) => asKey(f)));
     },
 
+    hlen(key) {
+      const k = asKey(key);
+      const meta = getKeyMeta(key);
+      if (!meta) return 0;
+      expectHash(meta);
+      return hashes.count(k);
+    },
+
     hexists(key, field) {
       const v = this.hget(key, field);
       return v != null ? 1 : 0;
@@ -322,6 +330,17 @@ export function createEngine(opts = {}) {
       return lists.rpop(k, count);
     },
 
+    lrem(key, count, element) {
+      const k = asKey(key);
+      const meta = getKeyMeta(key);
+      if (!meta) return 0;
+      expectList(meta);
+      const c = parseInt(Buffer.isBuffer(count) ? count.toString() : String(count), 10);
+      if (Number.isNaN(c)) throw new Error('ERR value is not an integer or out of range');
+      const elem = Buffer.isBuffer(element) ? element : asValue(element);
+      return lists.lrem(k, c, elem);
+    },
+
     zadd(key, scoreMemberPairs) {
       const k = asKey(key);
       getKeyMeta(key);

package/src/migration/apply-dirty.js

@@ -52,10 +52,12 @@ export async function runApplyDirty(redisClient, dbPath, runId, options = {}) {
       r = getRun(db, runId);
     }
 
-    const batch = getDirtyBatch(db, runId, 'dirty', batch_keys);
-    if (batch.length === 0) break;
+    const dirtyBatch   = getDirtyBatch(db, runId, 'dirty',   batch_keys);
+    const deletedBatch = getDirtyBatch(db, runId, 'deleted', batch_keys);
+    if (dirtyBatch.length === 0 && deletedBatch.length === 0) break;
 
-    for (const { key: keyBuf } of batch) {
+    // ── Re-import (or remove) keys that changed while bulk was running ──
+    for (const { key: keyBuf } of dirtyBatch) {
       r = getRun(db, runId);
       if (r && r.status === RUN_STATUS.ABORTED) break;
       while (r && r.status === RUN_STATUS.PAUSED) {
@@ -91,6 +93,35 @@ export async function runApplyDirty(redisClient, dbPath, runId, options = {}) {
         markDirtyState(db, runId, keyBuf, 'error');
       }
     }
+
+    // ── Apply deletions recorded by the tracker (del / expired events) ──
+    // The tracker already determined these keys are gone; delete from destination.
+    // Marked as 'deleted' in the run counter; state changed away from 'deleted'
+    // so the next getDirtyBatch call won't return them again (avoiding infinite loop).
+    for (const { key: keyBuf } of deletedBatch) {
+      r = getRun(db, runId);
+      if (r && r.status === RUN_STATUS.ABORTED) break;
+      while (r && r.status === RUN_STATUS.PAUSED) {
+        await sleep(2000);
+        r = getRun(db, runId);
+      }
+
+      try {
+        keys.delete(keyBuf);
+        // Increment dirty_keys_deleted counter and transition state out of 'deleted'
+        // so this key is not re-processed in the next batch iteration.
+        const now = Date.now();
+        db.prepare(
+          `UPDATE migration_dirty_keys SET state = 'applied', last_seen_at = ? WHERE run_id = ? AND key = ?`
+        ).run(now, runId, keyBuf);
+        db.prepare(
+          `UPDATE migration_runs SET dirty_keys_deleted = dirty_keys_deleted + 1, updated_at = ? WHERE run_id = ?`
+        ).run(now, runId);
+      } catch (err) {
+        logError(db, runId, 'dirty_apply', err.message, keyBuf);
+        markDirtyState(db, runId, keyBuf, 'error');
+      }
+    }
   }
 
   return getRun(db, runId);