resplite 1.0.4 → 1.0.6

package/README.md

@@ -35,6 +35,24 @@ OK
 "bar"
 ```
 
+### Standalone server script (fixed port)
+
+Run this as a persistent background process (`node server.js`). RESPLite will listen on port 6380 and stay up until the process is killed.
+
+```javascript
+// server.js
+import { createRESPlite } from 'resplite/embed';
+
+const srv = await createRESPlite({ port: 6380, db: './data.db' });
+console.log(`RESPLite listening on ${srv.host}:${srv.port}`);
+```
+
+Then connect from any other script or process:
+
+```bash
+redis-cli -p 6380 PING
+```
+
 ### Environment variables
 
 | Variable | Default | Description |
@@ -296,9 +314,43 @@ npm run import-from-redis -- --db ./migrated.db --host 127.0.0.1 --port 6379
 npm run import-from-redis -- --db ./migrated.db --pragma-template performance
 ```
 
+### Redis with authentication
+
+Migration supports Redis instances protected by a password. Use a Redis URL that includes the password (or username and password for Redis 6+ ACL):
+
+- **Password only:** `redis://:PASSWORD@host:port`
+- **Username and password:** `redis://username:PASSWORD@host:port`
+
+Examples:
+
+```bash
+# One-shot import from authenticated Redis
+npm run import-from-redis -- --db ./migrated.db --redis-url "redis://:mysecret@127.0.0.1:6379"
+
+# SPEC_F flow: use --from with the full URL (or set RESPLITE_IMPORT_FROM)
+npx resplite-import preflight --from "redis://:mysecret@10.0.0.10:6379" --to ./resplite.db
+npx resplite-dirty-tracker start --run-id run_001 --from "redis://:mysecret@10.0.0.10:6379" --to ./resplite.db
+```
+
+For one-shot import, authentication is only available when using `--redis-url`; the `--host` / `--port` options do not support a password.
+
 ### Minimal-downtime migration (SPEC_F)
 
-For large datasets (~30 GB), use the Dirty Key Registry flow so the bulk of the migration runs online and only a short cutover is needed:
+For large datasets (~30 GB), use the Dirty Key Registry flow so the bulk of the migration runs online and only a short cutover is needed.
+
+**Enable keyspace notifications in Redis** (required for the dirty-key tracker). Either run at runtime:
+
+```bash
+redis-cli CONFIG SET notify-keyspace-events Kgxe
+```
+
+Or add to `redis.conf` and restart Redis:
+
+```
+notify-keyspace-events Kgxe
+```
+
+(`K` = keyspace, `g` = generic, `x` = expired; `e` = keyevent. This lets the tracker see key changes and expirations.)
 
 1. **Preflight** – Check Redis, key count, type distribution, and that keyspace notifications are enabled:
    ```bash
@@ -338,6 +390,68 @@ For large datasets (~30 GB), use the Dirty Key Registry flow so the bulk of the
 
 Then start RespLite with the migrated DB: `RESPLITE_DB=./resplite.db npm start`.
 
+### Programmatic migration API
+
+As an alternative to the CLI, the full migration flow is available as a JavaScript API via `resplite/migration`. Useful for embedding the migration inside your own scripts or automation pipelines.
+
+```javascript
+import { createMigration } from 'resplite/migration';
+
+const m = createMigration({
+  from:  'redis://127.0.0.1:6379',  // source Redis URL (default)
+  to:    './resplite.db',           // destination SQLite DB path (required)
+  runId: 'my-migration-1',          // unique run ID (required for bulk/status/applyDirty)
+
+  // optional — same defaults as the CLI:
+  scanCount:      1000,
+  batchKeys:      200,
+  batchBytes:     64 * 1024 * 1024,  // 64 MB
+  maxRps:         0,                  // 0 = unlimited
+  pragmaTemplate: 'default',
+});
+
+// Step 0 — Preflight: inspect Redis before starting
+const info = await m.preflight();
+console.log('keys (estimate):', info.keyCountEstimate);
+console.log('type distribution:', info.typeDistribution);
+console.log('recommended params:', info.recommended);
+
+// Step 1 — Bulk import (checkpointed, resumable)
+await m.bulk({
+  resume: false,                            // true to resume a previous run
+  onProgress: (r) => console.log(
+    `scanned=${r.scanned_keys} migrated=${r.migrated_keys} errors=${r.error_keys}`
+  ),
+});
+
+// Check status at any point (synchronous, no Redis needed)
+const { run, dirty } = m.status();
+console.log('bulk status:', run.status, '— dirty counts:', dirty);
+
+// Step 2 — Apply dirty keys that changed in Redis during bulk
+await m.applyDirty();
+
+// Step 3 — Verify a sample of keys match between Redis and the destination
+const result = await m.verify({ samplePct: 0.5, maxSample: 10000 });
+console.log(`verified ${result.sampled} keys — mismatches: ${result.mismatches.length}`);
+
+// Disconnect Redis when done
+await m.close();
+```
+
+The dirty-key tracker (to capture writes during bulk) still runs as a separate process via `npx resplite-dirty-tracker`. The API above handles everything else in a single script.
+
+#### Low-level re-exports
+
+If you need more control, the individual functions and registry helpers are also exported:
+
+```javascript
+import {
+  runPreflight, runBulkImport, runApplyDirty, runVerify,
+  getRun, getDirtyCounts, createRun, setRunStatus, logError,
+} from 'resplite/migration';
+```
+
 ## Benchmark (Redis vs RESPLite)
 
 Compare throughput of local Redis and RESPLite with the same workload (PING, SET/GET, hashes, sets, lists, zsets, etc.):

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "resplite",
-  "version": "1.0.4",
+  "version": "1.0.6",
   "description": "A RESP2 server with practical Redis compatibility, backed by SQLite",
   "type": "module",
   "main": "src/index.js",
@@ -10,7 +10,8 @@
   },
   "exports": {
     ".": "./src/index.js",
-    "./embed": "./src/embed.js"
+    "./embed": "./src/embed.js",
+    "./migration": "./src/migration/index.js"
   },
   "scripts": {
     "start": "node src/index.js",

package/src/migration/index.js

@@ -0,0 +1,170 @@
+/**
+ * Programmatic migration API (SPEC_F §F.9).
+ *
+ * Usage:
+ *   import { createMigration } from 'resplite/migration';
+ *
+ *   const m = createMigration({
+ *     from: 'redis://127.0.0.1:6379',
+ *     to:   './data.db',
+ *     runId: 'my-migration-1',
+ *   });
+ *
+ *   const info   = await m.preflight();
+ *   await m.bulk({ onProgress: console.log });
+ *   const status = m.status();
+ *   await m.applyDirty();
+ *   const result = await m.verify();
+ *   await m.close();
+ */
+
+import { createClient } from 'redis';
+import { openDb } from '../storage/sqlite/db.js';
+import { runPreflight } from './preflight.js';
+import { runBulkImport } from './bulk.js';
+import { runApplyDirty } from './apply-dirty.js';
+import { runVerify } from './verify.js';
+import { getRun, getDirtyCounts } from './registry.js';
+
+/**
+ * @typedef {object} MigrationOptions
+ * @property {string} [from='redis://127.0.0.1:6379'] - Source Redis URL.
+ * @property {string} to                              - Destination SQLite DB path.
+ * @property {string} [runId]                         - Unique run identifier (required for bulk/status/applyDirty).
+ * @property {string} [pragmaTemplate='default']      - PRAGMA preset.
+ * @property {number} [scanCount=1000]
+ * @property {number} [maxRps=0]                      - Max requests/s (0 = unlimited).
+ * @property {number} [batchKeys=200]
+ * @property {number} [batchBytes=67108864]           - 64 MB default.
+ */
+
+/**
+ * Create a migration controller bound to a source Redis and destination DB.
+ *
+ * @param {MigrationOptions} options
+ * @returns {{
+ *   preflight(): Promise<import('./preflight.js').PreflightResult>,
+ *   bulk(opts?: { resume?: boolean, onProgress?: function }): Promise<object>,
+ *   status(): { run: object, dirty: object } | null,
+ *   applyDirty(opts?: { batchKeys?: number, maxRps?: number }): Promise<object>,
+ *   verify(opts?: { samplePct?: number, maxSample?: number }): Promise<object>,
+ *   close(): Promise<void>,
+ * }}
+ */
+export function createMigration({
+  from = 'redis://127.0.0.1:6379',
+  to,
+  runId,
+  pragmaTemplate = 'default',
+  scanCount = 1000,
+  maxRps = 0,
+  batchKeys = 200,
+  batchBytes = 64 * 1024 * 1024,
+} = {}) {
+  if (!to) throw new Error('createMigration: "to" (db path) is required');
+
+  let _client = null;
+
+  async function getClient() {
+    if (_client) return _client;
+    _client = createClient({ url: from });
+    _client.on('error', (err) => {
+      /* non-fatal connection errors; callers surface them on next await */
+      void err;
+    });
+    await _client.connect();
+    return _client;
+  }
+
+  function requireRunId() {
+    if (!runId) throw new Error('createMigration: "runId" is required for this operation');
+    return runId;
+  }
+
+  return {
+    /**
+     * Step 0 — Preflight: inspect the source Redis instance.
+     * Returns key count estimate, type distribution, keyspace-events config,
+     * and recommended import parameters.
+     */
+    async preflight() {
+      const client = await getClient();
+      return runPreflight(client);
+    },
+
+    /**
+     * Step 1 — Bulk import: SCAN all keys from Redis into the destination DB.
+     * Supports resume (checkpoint-based) and optional progress callback.
+     *
+     * @param {{ resume?: boolean, onProgress?: (run: object) => void }} [opts]
+     */
+    async bulk({ resume = false, onProgress } = {}) {
+      const id = requireRunId();
+      const client = await getClient();
+      return runBulkImport(client, to, id, {
+        sourceUri: from,
+        pragmaTemplate,
+        scan_count: scanCount,
+        max_rps: maxRps,
+        batch_keys: batchKeys,
+        batch_bytes: batchBytes,
+        resume,
+        onProgress,
+      });
+    },
+
+    /**
+     * Step 2 — Status: read run metadata and dirty-key counts from the DB.
+     * Synchronous — no Redis connection needed.
+     *
+     * @returns {{ run: object, dirty: object } | null}
+     */
+    status() {
+      const id = requireRunId();
+      const db = openDb(to, { pragmaTemplate });
+      const run = getRun(db, id);
+      if (!run) return null;
+      const dirty = getDirtyCounts(db, id);
+      return { run, dirty };
+    },
+
+    /**
+     * Step 3 — Apply dirty: reconcile keys that changed in Redis during bulk import.
+     *
+     * @param {{ batchKeys?: number, maxRps?: number }} [opts]
+     */
+    async applyDirty({ batchKeys: bk = batchKeys, maxRps: rps = maxRps } = {}) {
+      const id = requireRunId();
+      const client = await getClient();
+      return runApplyDirty(client, to, id, {
+        pragmaTemplate,
+        batch_keys: bk,
+        max_rps: rps,
+      });
+    },
+
+    /**
+     * Step 4 — Verify: sample keys from Redis and compare with the destination DB.
+     *
+     * @param {{ samplePct?: number, maxSample?: number }} [opts]
+     * @returns {Promise<{ sampled: number, matched: number, mismatches: Array<{ key: string, reason: string }> }>}
+     */
+    async verify({ samplePct = 0.5, maxSample = 10000 } = {}) {
+      const client = await getClient();
+      return runVerify(client, to, { pragmaTemplate, samplePct, maxSample });
+    },
+
+    /**
+     * Disconnect from Redis. Call when done with all migration operations.
+     */
+    async close() {
+      if (_client) {
+        await _client.quit().catch(() => {});
+        _client = null;
+      }
+    },
+  };
+}
+
+export { runPreflight, runBulkImport, runApplyDirty, runVerify };
+export { getRun, getDirtyCounts, createRun, setRunStatus, logError } from './registry.js';