spindb 0.37.2 → 0.38.0

package/engines/valkey/README.md

@@ -1,219 +0,0 @@
-# Valkey Engine Implementation
-
-## Overview
-
-Valkey is a Redis-compatible fork maintained by the Linux Foundation. SpinDB downloads Valkey binaries from hostdb and manages them identically to Redis.
-
-## Platform Support
-
-| Platform | Architecture | Status | Notes |
-|----------|--------------|--------|-------|
-| darwin | x64 | Supported | Uses hostdb binaries |
-| darwin | arm64 | Supported | Uses hostdb binaries (Apple Silicon) |
-| linux | x64 | Supported | Uses hostdb binaries |
-| linux | arm64 | Supported | Uses hostdb binaries |
-| win32 | x64 | Supported | **Custom-built binary** (see below) |
-
-### Windows Binary - Custom Build
-
-**There is no official Windows binary for Valkey.** The hostdb Windows binary was **manually built in a Windows VM** specifically for SpinDB. This required:
-
-1. Setting up a Windows development environment
-2. Building Valkey from source with Cygwin runtime
-3. Packaging the resulting binaries with Cygwin DLLs
-4. Uploading to hostdb
-
-This is one of the more unusual workarounds in SpinDB's engine support.
-
-### Cygwin Runtime on Windows
-
-The Windows Valkey binary is built with Cygwin, which means:
-- Paths must be converted to Cygwin format
-- The binary expects `/cygdrive/c/...` style paths
-
-```typescript
-// Path conversion function for Windows
-function toCygwinPath(windowsPath: string): string {
-  // C:\Users\foo\config.conf -> /cygdrive/c/Users/foo/config.conf
-  const driveMatch = windowsPath.match(/^([A-Za-z]):[/\\]/)
-  if (!driveMatch) return windowsPath.replace(/\\/g, '/')
-
-  const driveLetter = driveMatch[1].toLowerCase()
-  const restOfPath = windowsPath.slice(3).replace(/\\/g, '/')
-  return `/cygdrive/${driveLetter}/${restOfPath}`
-}
-```
-
-## Binary Packaging
-
-### Archive Format
-- **Unix (macOS/Linux)**: `tar.gz`
-- **Windows**: `zip` (contains Cygwin-built binaries)
-
-### Archive Structure
-```
-valkey/
-└── bin/
-    ├── valkey-server    # Server binary
-    ├── valkey-cli       # Client CLI
-    ├── valkey-benchmark # Benchmarking tool
-    └── cygwin1.dll      # Cygwin runtime (Windows only)
-```
-
-### Version Map Sync
-
-```typescript
-export const VALKEY_VERSION_MAP: Record<string, string> = {
-  '8': '8.0.6',
-  '9': '9.x.x',
-}
-```
-
-## Implementation Details
-
-### Binary Manager
-
-Valkey uses `BaseBinaryManager` with configuration identical to Redis.
-
-### Version Parsing
-
-- **Version output format**: `Valkey server v=8.0.6 sha=00000000:0 malloc=jemalloc-5.3.0...`
-- **Parse pattern**: `/v=(\d+\.\d+\.\d+)/` then fallback to `/(\d+\.\d+\.\d+)/`
-
-### Redis Compatibility
-
-Valkey uses `redis://` connection scheme for client compatibility:
-
-```
-redis://127.0.0.1:{port}/{database}
-```
-
-Not `valkey://` - this ensures compatibility with existing Redis clients and tools.
-
-### Default Configuration
-
-- **Default Port**: 6379 (same as Redis, auto-increments on conflict)
-- **Databases**: 16 numbered databases (0-15)
-- **PID File**: `valkey.pid` in container directory
-- **Persistence**: RDB snapshots enabled
-
-### Generated Configuration
-
-SpinDB generates `valkey.conf` with Redis-compatible settings:
-```
-port {port}
-bind 127.0.0.1
-dir {dataDir}
-daemonize yes     # Unix only
-logfile {logFile}
-pidfile {pidFile}
-
-# Persistence
-save 900 1
-save 300 10
-save 60 10000
-dbfilename dump.rdb
-appendonly no
-```
-
-### Windows Detached Spawn
-
-Like Redis, Valkey on Windows uses detached spawn:
-```typescript
-const spawnOpts: SpawnOptions = {
-  stdio: ['ignore', 'pipe', 'pipe'],
-  detached: true,
-  windowsHide: true,
-}
-// Convert path to Cygwin format
-const cygwinConfigPath = toCygwinPath(configPath)
-spawn(valkeyServer, [cygwinConfigPath], spawnOpts)
-```
-
-## Backup & Restore
-
-### Backup Formats
-
-| Format | Extension | Tool | Notes |
-|--------|-----------|------|-------|
-| text | `.valkey` | Custom RESP | Human-readable commands |
-| rdb | `.rdb` | Native RDB | Binary snapshot |
-
-Identical to Redis backup formats, with `.valkey` extension for text format.
-
-## Integration Test Notes
-
-### Port Allocation
-
-Valkey tests should avoid conflicting with Redis tests since they share default port 6379.
-
-### Test Fixtures
-
-Located in `tests/fixtures/valkey/seeds/`:
-- Key-value pairs for testing
-
-## Docker E2E Test Notes
-
-Valkey Docker E2E tests verify Redis-compatible operations:
-- Server lifecycle
-- Key-value operations
-- Backup/restore
-- Database switching
-
-## Known Issues & Gotchas
-
-### 1. Windows Cygwin Paths
-
-**Critical**: The Windows binary expects Cygwin-style paths. Passing Windows paths directly will fail:
-```
-# Wrong: C:\Users\spindb\valkey.conf
-# Right: /cygdrive/c/Users/spindb/valkey.conf
-```
-
-### 2. No Official Windows Binary
-
-The Windows support required manual compilation. If issues arise with the Windows binary, it may need to be rebuilt from source in a Windows VM.
-
-### 3. Port Conflict with Redis
-
-Valkey and Redis share default port 6379. SpinDB prevents conflicts via auto-increment.
-
-### 4. iredis Compatibility
-
-The `iredis` enhanced CLI (Python tool) works with Valkey since it's protocol-compatible. The engine detects and supports `iredis` as an alternative to `valkey-cli`.
-
-### 5. Connection Scheme
-
-Despite being Valkey, uses `redis://` scheme:
-- `redis://` (plain)
-- `rediss://` (TLS)
-- `valkey://` / `valkeys://` are also accepted and normalized
-
-## CI/CD Notes
-
-### GitHub Actions Cache Step
-
-```yaml
-- name: Cache Valkey binaries
-  uses: actions/cache@v4
-  with:
-    path: ~/.spindb/bin/valkey-*
-    key: valkey-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('engines/valkey/version-maps.ts') }}
-```
-
-### Windows CI Considerations
-
-Windows CI tests for Valkey verify the Cygwin-built binary works correctly with path conversion.
-
-## Building Windows Binaries
-
-If you need to rebuild the Windows binary:
-
-1. Set up a Windows VM with Visual Studio Build Tools
-2. Install Cygwin with development packages
-3. Clone Valkey source
-4. Build with: `make`
-5. Package `valkey-server.exe`, `valkey-cli.exe`, and required Cygwin DLLs
-6. Create zip and upload to hostdb
-
-This is a manual process not automated in CI.

package/engines/valkey/backup.ts

@@ -1,389 +0,0 @@
-/**
- * Valkey backup module
- * Supports two formats:
- * - RDB: Binary snapshot via BGSAVE (default, compact, fast restore)
- * - Text: Human-readable Valkey commands (.valkey file)
- */
-
-import { spawn } from 'child_process'
-import { copyFile, stat, mkdir, writeFile } from 'fs/promises'
-import { existsSync } from 'fs'
-import { join, dirname } from 'path'
-import { logDebug, logWarning } from '../../core/error-handler'
-import { paths } from '../../config/paths'
-import { getValkeyCliPath, VALKEY_CLI_NOT_FOUND_ERROR } from './cli-utils'
-import type { ContainerConfig, BackupOptions, BackupResult } from '../../types'
-
-/**
- * Execute a Valkey command via stdin piping
- * This is safer than shell command construction as it handles
- * keys/values with spaces and special characters correctly
- */
-async function execValkeyCommand(
-  valkeyCli: string,
-  port: number,
-  command: string,
-): Promise<string> {
-  return new Promise((resolve, reject) => {
-    const proc = spawn(valkeyCli, ['-h', '127.0.0.1', '-p', String(port)], {
-      stdio: ['pipe', 'pipe', 'pipe'],
-    })
-
-    let stdout = ''
-    let stderr = ''
-
-    proc.stdout.on('data', (data: Buffer) => {
-      stdout += data.toString()
-    })
-    proc.stderr.on('data', (data: Buffer) => {
-      stderr += data.toString()
-    })
-
-    proc.on('error', reject)
-
-    proc.on('close', (code) => {
-      if (code === 0 || code === null) {
-        resolve(stdout)
-      } else {
-        reject(new Error(stderr || `valkey-cli exited with code ${code}`))
-      }
-    })
-
-    // Write command to stdin and close
-    proc.stdin.write(command + '\n')
-    proc.stdin.end()
-  })
-}
-
-/**
- * Escape a string value for Valkey command output
- * Wraps in single quotes and escapes backslashes and single quotes
- */
-function escapeValkeyValue(value: string): string {
-  // Escape backslashes first, then single quotes (order matters!)
-  // e.g., "test\'value" → "test\\\'value" (backslash and quote both escaped)
-  return `'${value.replace(/\\/g, '\\\\').replace(/'/g, "\\'")}'`
-}
-
-/**
- * Create a text-based backup (.valkey file)
- * Exports all keys as Valkey commands that can be replayed
- */
-async function createTextBackup(
-  container: ContainerConfig,
-  outputPath: string,
-): Promise<BackupResult> {
-  const { port } = container
-
-  const valkeyCli = await getValkeyCliPath()
-  if (!valkeyCli) {
-    throw new Error(VALKEY_CLI_NOT_FOUND_ERROR)
-  }
-
-  const commands: string[] = []
-  commands.push('# Valkey backup generated by SpinDB')
-  commands.push(`# Date: ${new Date().toISOString()}`)
-  commands.push('')
-
-  // Get all keys using KEYS * (for small datasets this is fine)
-  // For production with millions of keys, SCAN would be better
-  const keysOutput = await execValkeyCommand(valkeyCli, port, 'KEYS *')
-  // Use /\r?\n/ to handle both Unix (\n) and Windows (\r\n) line endings
-  const keys = keysOutput
-    .trim()
-    .split(/\r?\n/)
-    .map((k) => k.trim())
-    .filter((k) => k)
-
-  logDebug(`Found ${keys.length} keys to backup`)
-
-  for (const key of keys) {
-    // Get key type - quote key to handle spaces/special chars
-    const typeOutput = await execValkeyCommand(
-      valkeyCli,
-      port,
-      `TYPE "${key.replace(/"/g, '\\"')}"`,
-    )
-    const keyType = typeOutput.trim()
-
-    // Get TTL
-    const ttlOutput = await execValkeyCommand(
-      valkeyCli,
-      port,
-      `TTL "${key.replace(/"/g, '\\"')}"`,
-    )
-    const ttl = parseInt(ttlOutput.trim(), 10)
-
-    // Quote the key for output commands
-    const quotedKey =
-      key.includes(' ') || /[*?[\]{}$`"'\\!<>|;&()]/.test(key)
-        ? `"${key.replace(/"/g, '\\"')}"`
-        : key
-
-    switch (keyType) {
-      case 'string': {
-        const value = await execValkeyCommand(
-          valkeyCli,
-          port,
-          `GET "${key.replace(/"/g, '\\"')}"`,
-        )
-        commands.push(`SET ${quotedKey} ${escapeValkeyValue(value.trim())}`)
-        break
-      }
-      case 'hash': {
-        const hashData = await execValkeyCommand(
-          valkeyCli,
-          port,
-          `HGETALL "${key.replace(/"/g, '\\"')}"`,
-        )
-        const lines = hashData
-          .trim()
-          .split(/\r?\n/)
-          .map((l) => l.trim())
-          .filter((l) => l)
-        if (lines.length >= 2) {
-          const pairs: string[] = []
-          for (let i = 0; i < lines.length; i += 2) {
-            const field = lines[i].trim()
-            const value = lines[i + 1]?.trim() || ''
-            // Quote field if it contains special chars
-            const quotedField =
-              field.includes(' ') || /[*?[\]{}$`"'\\!<>|;&()]/.test(field)
-                ? `"${field.replace(/"/g, '\\"')}"`
-                : field
-            pairs.push(`${quotedField} ${escapeValkeyValue(value)}`)
-          }
-          commands.push(`HSET ${quotedKey} ${pairs.join(' ')}`)
-        }
-        break
-      }
-      case 'list': {
-        const listData = await execValkeyCommand(
-          valkeyCli,
-          port,
-          `LRANGE "${key.replace(/"/g, '\\"')}" 0 -1`,
-        )
-        const items = listData
-          .trim()
-          .split(/\r?\n/)
-          .map((l) => l.trim())
-          .filter((l) => l)
-        if (items.length > 0) {
-          const escapedItems = items.map((item) =>
-            escapeValkeyValue(item.trim()),
-          )
-          commands.push(`RPUSH ${quotedKey} ${escapedItems.join(' ')}`)
-        }
-        break
-      }
-      case 'set': {
-        const setData = await execValkeyCommand(
-          valkeyCli,
-          port,
-          `SMEMBERS "${key.replace(/"/g, '\\"')}"`,
-        )
-        const members = setData
-          .trim()
-          .split(/\r?\n/)
-          .map((l) => l.trim())
-          .filter((l) => l)
-        if (members.length > 0) {
-          const escapedMembers = members.map((m) => escapeValkeyValue(m.trim()))
-          commands.push(`SADD ${quotedKey} ${escapedMembers.join(' ')}`)
-        }
-        break
-      }
-      case 'zset': {
-        const zsetData = await execValkeyCommand(
-          valkeyCli,
-          port,
-          `ZRANGE "${key.replace(/"/g, '\\"')}" 0 -1 WITHSCORES`,
-        )
-        const lines = zsetData
-          .trim()
-          .split(/\r?\n/)
-          .map((l) => l.trim())
-          .filter((l) => l)
-        if (lines.length >= 2) {
-          const pairs: string[] = []
-          for (let i = 0; i < lines.length; i += 2) {
-            const member = lines[i].trim()
-            const score = lines[i + 1]?.trim() || '0'
-            pairs.push(`${score} ${escapeValkeyValue(member)}`)
-          }
-          commands.push(`ZADD ${quotedKey} ${pairs.join(' ')}`)
-        }
-        break
-      }
-      default:
-        logWarning(`Skipping key ${key} with unsupported type: ${keyType}`)
-    }
-
-    // Add EXPIRE if key has TTL
-    if (ttl > 0) {
-      commands.push(`EXPIRE ${quotedKey} ${ttl}`)
-    }
-  }
-
-  // Ensure output directory exists
-  const outputDir = dirname(outputPath)
-  if (!existsSync(outputDir)) {
-    await mkdir(outputDir, { recursive: true })
-  }
-
-  // Write commands to file
-  const content = commands.join('\n') + '\n'
-  await writeFile(outputPath, content, 'utf-8')
-
-  const stats = await stat(outputPath)
-
-  return {
-    path: outputPath,
-    format: 'text',
-    size: stats.size,
-  }
-}
-
-/**
- * Create a backup using BGSAVE (RDB format)
- *
- * Valkey backups are RDB files (binary snapshots).
- * The backup process:
- * 1. Trigger BGSAVE command
- * 2. Poll LASTSAVE until timestamp changes (backup complete)
- * 3. Copy dump.rdb to output path
- */
-async function createRdbBackup(
-  container: ContainerConfig,
-  outputPath: string,
-): Promise<BackupResult> {
-  const { port, name } = container
-
-  const valkeyCli = await getValkeyCliPath()
-  if (!valkeyCli) {
-    throw new Error(VALKEY_CLI_NOT_FOUND_ERROR)
-  }
-
-  // Trigger background save
-  let bgsaveResponse: string
-  try {
-    bgsaveResponse = await execValkeyCommand(valkeyCli, port, 'BGSAVE')
-    bgsaveResponse = bgsaveResponse.trim()
-  } catch (error) {
-    const execError = error as Error
-    throw new Error(`BGSAVE command failed: ${execError.message}`)
-  }
-
-  logDebug(`BGSAVE response: ${bgsaveResponse}`)
-
-  // Check for Valkey error responses in stdout
-  // Valkey returns errors like "ERR ..." or "(error) ..." in stdout
-  if (
-    bgsaveResponse.startsWith('ERR') ||
-    bgsaveResponse.startsWith('(error)')
-  ) {
-    // Special case: if a save is already in progress, we can wait for it
-    if (bgsaveResponse.includes('Background save already in progress')) {
-      logDebug('BGSAVE already in progress, waiting for it to complete')
-    } else {
-      throw new Error(`BGSAVE failed: ${bgsaveResponse}`)
-    }
-  } else if (
-    !bgsaveResponse.includes('Background saving started') &&
-    !bgsaveResponse.includes('Background save already in progress')
-  ) {
-    // Unexpected response - warn but continue (might be a different Valkey version)
-    logDebug(
-      `Unexpected BGSAVE response (continuing anyway): ${bgsaveResponse}`,
-    )
-  }
-
-  // Wait for save to complete by checking rdb_bgsave_in_progress
-  // This is more reliable than LASTSAVE timestamp which has 1-second resolution
-  const startTime = Date.now()
-  const timeout = 60000 // 1 minute timeout
-
-  while (Date.now() - startTime < timeout) {
-    const infoOutput = await execValkeyCommand(
-      valkeyCli,
-      port,
-      'INFO persistence',
-    )
-
-    // Check if BGSAVE is still in progress
-    const inProgress = infoOutput.includes('rdb_bgsave_in_progress:1')
-    if (!inProgress) {
-      // Also check for errors
-      const statusMatch = infoOutput.match(/rdb_last_bgsave_status:(\w+)/)
-      const status = statusMatch?.[1]
-      if (status === 'err') {
-        throw new Error('BGSAVE failed. Check Valkey logs for details.')
-      }
-      logDebug('BGSAVE completed successfully')
-      break
-    }
-
-    await new Promise((r) => setTimeout(r, 100))
-  }
-
-  if (Date.now() - startTime >= timeout) {
-    throw new Error('BGSAVE timed out after 60 seconds')
-  }
-
-  // Get the RDB file path from Valkey data directory
-  const dataDir = paths.getContainerDataPath(name, { engine: 'valkey' })
-  const rdbPath = join(dataDir, 'dump.rdb')
-
-  if (!existsSync(rdbPath)) {
-    throw new Error(
-      `RDB file not found at ${rdbPath} after BGSAVE. Check Valkey configuration.`,
-    )
-  }
-
-  // Ensure output directory exists
-  const outputDir = dirname(outputPath)
-  if (!existsSync(outputDir)) {
-    await mkdir(outputDir, { recursive: true })
-  }
-
-  // Copy RDB file to output path
-  await copyFile(rdbPath, outputPath)
-
-  const stats = await stat(outputPath)
-
-  return {
-    path: outputPath,
-    format: 'rdb',
-    size: stats.size,
-  }
-}
-
-/**
- * Create a backup
- * Supports two formats:
- * - 'text': Human-readable Valkey commands (.valkey file)
- * - 'rdb' (default): Binary RDB snapshot
- */
-export async function createBackup(
-  container: ContainerConfig,
-  outputPath: string,
-  options: BackupOptions,
-): Promise<BackupResult> {
-  // 'text' format means text-based commands for Valkey
-  if (options.format === 'text') {
-    return createTextBackup(container, outputPath)
-  }
-  // Default to RDB format
-  return createRdbBackup(container, outputPath)
-}
-
-/**
- * Create a backup for cloning purposes
- * Always uses RDB format for speed and reliability
- */
-export async function createCloneBackup(
-  container: ContainerConfig,
-  outputPath: string,
-): Promise<BackupResult> {
-  return createRdbBackup(container, outputPath)
-}

package/engines/valkey/binary-manager.ts

@@ -1,45 +0,0 @@
-/**
- * Valkey Binary Manager
- *
- * Handles downloading, extracting, and managing Valkey binaries from hostdb.
- * Extends BaseBinaryManager for shared download/extraction logic.
- */
-
-import {
-  BaseBinaryManager,
-  type BinaryManagerConfig,
-} from '../../core/base-binary-manager'
-import { getBinaryUrl } from './binary-urls'
-import { normalizeVersion } from './version-maps'
-import { Engine, type Platform, type Arch } from '../../types'
-
-class ValkeyBinaryManager extends BaseBinaryManager {
-  protected readonly config: BinaryManagerConfig = {
-    engine: Engine.Valkey,
-    engineName: 'valkey',
-    displayName: 'Valkey',
-    serverBinary: 'valkey-server',
-  }
-
-  protected getBinaryUrlFromModule(
-    version: string,
-    platform: Platform,
-    arch: Arch,
-  ): string {
-    return getBinaryUrl(version, platform, arch)
-  }
-
-  protected normalizeVersionFromModule(version: string): string {
-    return normalizeVersion(version)
-  }
-
-  protected parseVersionFromOutput(stdout: string): string | null {
-    // Extract version from output like "Valkey server v=8.0.6 sha=00000000:0 malloc=jemalloc-5.3.0 bits=64 build=..."
-    // or "v=8.0.6" pattern
-    const match = stdout.match(/v=(\d+\.\d+\.\d+)/)
-    const altMatch = !match ? stdout.match(/(\d+\.\d+\.\d+)/) : null
-    return match?.[1] ?? altMatch?.[1] ?? null
-  }
-}
-
-export const valkeyBinaryManager = new ValkeyBinaryManager()