spindb 0.37.2 → 0.38.0

package/engines/meilisearch/README.md

@@ -1,255 +0,0 @@
-# Meilisearch Engine Implementation
-
-## Overview
-
-Meilisearch is a fast full-text search engine with a REST API. Like Qdrant, it uses HTTP for all operations instead of a CLI shell.
-
-## Platform Support
-
-| Platform | Architecture | Status | Notes |
-|----------|--------------|--------|-------|
-| darwin | x64 | Supported | Uses hostdb binaries |
-| darwin | arm64 | Supported | Uses hostdb binaries (Apple Silicon) |
-| linux | x64 | Supported | Uses hostdb binaries |
-| linux | arm64 | Supported | Uses hostdb binaries |
-| win32 | x64 | Supported | Backup/restore has issues (see below) |
-
-### Windows Backup/Restore Limitation
-
-Windows backup/restore **fails** due to an upstream Meilisearch bug:
-- Snapshot creation fails with page size alignment error
-- This is a Meilisearch issue, not SpinDB
-
-Server operations (start, stop, indexing) work correctly on Windows.
-
-## Binary Packaging
-
-### Archive Format
-- **Unix (macOS/Linux)**: `tar.gz`
-- **Windows**: `zip`
-
-### Archive Structure
-```
-meilisearch/
-└── bin/
-    └── meilisearch      # Server binary
-```
-
-### Version Map Sync
-
-```typescript
-export const MEILISEARCH_VERSION_MAP: Record<string, string> = {
-  '1': '1.33.1',
-}
-```
-
-## Implementation Details
-
-### Binary Manager
-
-Meilisearch uses `BaseBinaryManager` with standard configuration.
-
-### Version Parsing
-
-- **Version output format**: `meilisearch 1.33.1` or `v1.33.1`
-- **Parse pattern**: `/(?:meilisearch\s+)?v?(\d+\.\d+\.\d+)/`
-
-### REST API Engine
-
-Meilisearch is a **REST API engine**:
-- `spindb run` is **NOT applicable**
-- `spindb connect` opens the web dashboard in browser
-- All operations use HTTP REST API
-
-### Single Port
-
-Unlike Qdrant (dual ports), Meilisearch only uses HTTP:
-- **HTTP Port** (default 7700): REST API and dashboard
-
-### Default Configuration
-
-- **Default Port**: 7700 (auto-increments on conflict)
-- **Health Endpoint**: `/health` (returns `{"status":"available"}`)
-- **Dashboard**: `/` (root path, NOT `/dashboard`)
-- **PID File**: `meilisearch.pid` in container directory
-
-### Index Naming
-
-Meilisearch uses "indexes" instead of "databases":
-- Index UIDs only allow **alphanumeric characters and underscores**
-- Container names with dashes are auto-converted: `my-app` -> index `my_app`
-
-### Connection String Format
-
-```
-http://127.0.0.1:{port}
-```
-
-### Web Dashboard
-
-Meilisearch dashboard is at the root path:
-```
-http://localhost:{port}/
-```
-
-(NOT `/dashboard` like Qdrant)
-
-## Backup & Restore
-
-### Backup Formats
-
-| Format | Extension | Tool | Notes |
-|--------|-----------|------|-------|
-| snapshot | `.snapshot` | REST API | Meilisearch native snapshot |
-
-### Snapshot Directory Placement - CRITICAL
-
-**MUST be a sibling of data directory, NOT inside it.**
-
-```
-container/
-├── data/           # Meilisearch data (--db-path)
-└── snapshots/      # Snapshot directory (--snapshot-dir)
-```
-
-**NOT:**
-```
-container/
-└── data/
-    └── snapshots/  # WRONG - causes "failed to infer version" error
-```
-
-If `--snapshot-dir` points inside `--db-path`, Meilisearch fails with:
-> "failed to infer the version of the database"
-
-### Windows Backup Failure
-
-On Windows, snapshot creation fails with page size alignment errors. This is an upstream Meilisearch bug.
-
-## Integration Test Notes
-
-### REST API Testing
-
-Integration tests use `fetch()` or `curl`.
-
-### Test Fixtures
-
-Located in `tests/fixtures/meilisearch/seeds/`:
-- `README.md` documenting the API-based approach
-
-## Docker E2E Test Notes
-
-Meilisearch Docker E2E uses `curl`:
-
-```bash
-# Health check
-curl http://localhost:7700/health
-
-# Create index
-curl -X POST http://localhost:7700/indexes \
-  -H 'Content-Type: application/json' \
-  -d '{"uid":"movies","primaryKey":"id"}'
-
-# Add documents
-curl -X POST http://localhost:7700/indexes/movies/documents \
-  -H 'Content-Type: application/json' \
-  -d '[{"id":1,"title":"Batman"}]'
-```
-
-### Backup/Restore Skipped in Docker E2E
-
-Meilisearch backup/restore tests are skipped in Docker E2E (covered by integration tests).
-
-## Known Issues & Gotchas
-
-### 1. No CLI Shell
-
-`spindb run` does nothing for Meilisearch. Use the REST API or web dashboard.
-
-### 2. Dashboard at Root
-
-The dashboard is at `/` (root), not `/dashboard`. Different from Qdrant.
-
-### 3. Snapshot Directory Location
-
-Snapshot directory **must be sibling** of data directory. This is the most common Meilisearch configuration issue.
-
-### 4. Index UID Constraints
-
-Index UIDs only allow `[a-zA-Z0-9_]`. Dashes cause API errors:
-```
-# Invalid: my-movies
-# Valid: my_movies
-```
-
-### 5. Windows Backup Broken
-
-Snapshot creation fails on Windows due to page size alignment bug. This is a Meilisearch upstream issue.
-
-### 6. Health Endpoint
-
-Use `/health` (not `/healthz` like Qdrant) for health checks.
-
-## CI/CD Notes
-
-### curl-Based Testing
-
-CI tests use `curl` commands.
-
-### Windows Backup Tests Skipped
-
-Backup/restore tests are skipped on Windows due to upstream bug.
-
-### GitHub Actions Cache Step
-
-```yaml
-- name: Cache Meilisearch binaries
-  uses: actions/cache@v4
-  with:
-    path: ~/.spindb/bin/meilisearch-*
-    key: meilisearch-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('engines/meilisearch/version-maps.ts') }}
-```
-
-## REST API Quick Reference
-
-### Indexes
-```bash
-# List indexes
-GET /indexes
-
-# Create index
-POST /indexes
-{"uid": "movies", "primaryKey": "id"}
-
-# Get index
-GET /indexes/{uid}
-
-# Delete index
-DELETE /indexes/{uid}
-```
-
-### Documents
-```bash
-# Add documents
-POST /indexes/{uid}/documents
-
-# Get document
-GET /indexes/{uid}/documents/{id}
-
-# Search
-POST /indexes/{uid}/search
-{"q": "search query"}
-```
-
-### Snapshots
-```bash
-# Create snapshot
-POST /snapshots
-```
-
-### Health
-```bash
-# Health check
-GET /health
-# Returns: {"status": "available"}
-```

package/engines/meilisearch/api-client.ts

@@ -1,61 +0,0 @@
-/**
- * Shared Meilisearch REST API client utilities
- */
-
-/**
- * Make an HTTP request to Meilisearch REST API
- *
- * @param port - The HTTP port Meilisearch is listening on
- * @param method - HTTP method (GET, POST, PUT, DELETE)
- * @param path - API path (e.g., '/indexes', '/health')
- * @param body - Optional JSON body for POST/PUT requests
- * @param timeoutMs - Request timeout in milliseconds (default: 30s)
- */
-export async function meilisearchApiRequest(
-  port: number,
-  method: string,
-  path: string,
-  body?: Record<string, unknown>,
-  timeoutMs = 30000,
-): Promise<{ status: number; data: unknown }> {
-  const url = `http://127.0.0.1:${port}${path}`
-
-  const controller = new AbortController()
-  const timeoutId = setTimeout(() => controller.abort(), timeoutMs)
-
-  const options: RequestInit = {
-    method,
-    headers: {
-      'Content-Type': 'application/json',
-    },
-    signal: controller.signal,
-  }
-
-  if (body) {
-    options.body = JSON.stringify(body)
-  }
-
-  try {
-    const response = await fetch(url, options)
-
-    // Try to parse as JSON, fall back to text for endpoints like /health
-    let data: unknown
-    const contentType = response.headers.get('content-type') || ''
-    if (contentType.includes('application/json')) {
-      data = await response.json()
-    } else {
-      data = await response.text()
-    }
-
-    return { status: response.status, data }
-  } catch (error) {
-    if (error instanceof Error && error.name === 'AbortError') {
-      throw new Error(
-        `Meilisearch API request timed out after ${timeoutMs / 1000}s: ${method} ${path}`,
-      )
-    }
-    throw error
-  } finally {
-    clearTimeout(timeoutId)
-  }
-}

package/engines/meilisearch/backup.ts

@@ -1,233 +0,0 @@
-/**
- * Meilisearch backup module
- * Supports snapshot-based backup using Meilisearch's REST API
- */
-
-import { mkdir, stat, copyFile, readdir } from 'fs/promises'
-import { existsSync } from 'fs'
-import { join, dirname } from 'path'
-import { logDebug } from '../../core/error-handler'
-import { paths } from '../../config/paths'
-import { meilisearchApiRequest } from './api-client'
-import type { ContainerConfig, BackupOptions, BackupResult } from '../../types'
-
-// Backup operations may take longer than the default timeout
-const BACKUP_TIMEOUT_MS = 600000 // 10 minutes
-
-/**
- * Create a snapshot backup using Meilisearch's REST API
- * This creates a full snapshot of the entire Meilisearch instance
- */
-export async function createBackup(
-  container: ContainerConfig,
-  outputPath: string,
-  _options: BackupOptions,
-): Promise<BackupResult> {
-  const { port, name } = container
-
-  // Ensure output directory exists
-  const outputDir = dirname(outputPath)
-  if (!existsSync(outputDir)) {
-    await mkdir(outputDir, { recursive: true })
-  }
-
-  // Trigger snapshot creation via REST API
-  logDebug(`Creating Meilisearch snapshot via REST API on port ${port}`)
-
-  const response = await meilisearchApiRequest(
-    port,
-    'POST',
-    '/snapshots',
-    undefined,
-    BACKUP_TIMEOUT_MS,
-  )
-
-  // Meilisearch returns 202 Accepted for snapshot creation
-  if (response.status !== 202 && response.status !== 200) {
-    throw new Error(
-      `Failed to create Meilisearch snapshot: ${JSON.stringify(response.data)}`,
-    )
-  }
-
-  // Meilisearch creates snapshots asynchronously and returns a task
-  // We need to wait for the task to complete
-  const taskData = response.data as { taskUid?: number }
-  const taskUid = taskData?.taskUid
-
-  if (taskUid !== undefined) {
-    logDebug(`Meilisearch snapshot task created: ${taskUid}`)
-    // Wait for task to complete
-    await waitForTask(port, taskUid, BACKUP_TIMEOUT_MS)
-  }
-
-  // The snapshot is stored in the snapshots folder (sibling of data, not inside it)
-  // IMPORTANT: Meilisearch fails if --snapshot-dir is inside --db-path
-  const containerDir = paths.getContainerPath(name, { engine: 'meilisearch' })
-  const snapshotsDir = join(containerDir, 'snapshots')
-
-  // Wait for the snapshot file to appear
-  const maxWait = 60000 // 60 seconds
-  const startTime = Date.now()
-  let snapshotPath: string | null = null
-
-  while (Date.now() - startTime < maxWait) {
-    if (existsSync(snapshotsDir)) {
-      const files = await readdir(snapshotsDir)
-      const snapshotFiles = files.filter((f) => f.endsWith('.snapshot'))
-      if (snapshotFiles.length > 0) {
-        // Get the most recent snapshot by mtime (not lexicographic sort)
-        let newestFile: string | null = null
-        let newestMtime = 0
-        for (const file of snapshotFiles) {
-          const filePath = join(snapshotsDir, file)
-          try {
-            const fileStat = await stat(filePath)
-            if (fileStat.mtimeMs > newestMtime) {
-              newestMtime = fileStat.mtimeMs
-              newestFile = file
-            }
-          } catch {
-            // Skip files that can't be stat'd (may be deleted during iteration)
-          }
-        }
-        if (newestFile) {
-          snapshotPath = join(snapshotsDir, newestFile)
-          break
-        }
-      }
-    }
-    await new Promise((r) => setTimeout(r, 500))
-  }
-
-  if (!snapshotPath || !existsSync(snapshotPath)) {
-    throw new Error(
-      `Meilisearch snapshot file not found in ${snapshotsDir} after timeout`,
-    )
-  }
-
-  // Wait for file to be fully written (size stabilizes)
-  let lastSize = -1
-  let stabilized = false
-  const writeWaitStart = Date.now()
-  while (Date.now() - writeWaitStart < 30000) {
-    const currentStats = await stat(snapshotPath)
-    if (currentStats.size > 0 && currentStats.size === lastSize) {
-      stabilized = true
-      break
-    }
-    lastSize = currentStats.size
-    await new Promise((r) => setTimeout(r, 500))
-  }
-
-  if (!stabilized) {
-    throw new Error(
-      `Meilisearch snapshot did not stabilize within 30 seconds. ` +
-        `File may still be writing: ${snapshotPath}`,
-    )
-  }
-
-  // Copy snapshot to output path
-  await copyFile(snapshotPath, outputPath)
-
-  const stats = await stat(outputPath)
-
-  return {
-    path: outputPath,
-    format: 'snapshot',
-    size: stats.size,
-  }
-}
-
-/**
- * Wait for a Meilisearch task to complete
- */
-async function waitForTask(
-  port: number,
-  taskUid: number,
-  timeoutMs: number,
-): Promise<void> {
-  const startTime = Date.now()
-  const checkInterval = 500
-
-  while (Date.now() - startTime < timeoutMs) {
-    const response = await meilisearchApiRequest(
-      port,
-      'GET',
-      `/tasks/${taskUid}`,
-    )
-
-    if (response.status === 200) {
-      const task = response.data as { status?: string }
-      if (task.status === 'succeeded') {
-        logDebug(`Meilisearch task ${taskUid} succeeded`)
-        return
-      }
-      if (task.status === 'failed') {
-        throw new Error(
-          `Meilisearch task ${taskUid} failed: ${JSON.stringify(task)}`,
-        )
-      }
-    }
-
-    await new Promise((r) => setTimeout(r, checkInterval))
-  }
-
-  throw new Error(`Meilisearch task ${taskUid} did not complete within timeout`)
-}
-
-/**
- * Create a backup for cloning purposes
- * Uses snapshot format for reliable data transfer
- */
-export async function createCloneBackup(
-  container: ContainerConfig,
-  outputPath: string,
-): Promise<BackupResult> {
-  return createBackup(container, outputPath, { database: 'default' })
-}
-
-/**
- * List available snapshots for a container
- */
-export async function listSnapshots(container: ContainerConfig): Promise<
-  Array<{
-    name: string
-    createdAt: string
-    size: number
-  }>
-> {
-  const { name } = container
-  // Snapshots directory is sibling of data, not inside it
-  const containerDir = paths.getContainerPath(name, { engine: 'meilisearch' })
-  const snapshotsDir = join(containerDir, 'snapshots')
-
-  if (!existsSync(snapshotsDir)) {
-    return []
-  }
-
-  const files = await readdir(snapshotsDir)
-  const snapshotFiles = files.filter((file) => file.endsWith('.snapshot'))
-
-  // Stat all snapshot files in parallel for better performance
-  // Filter out any files that fail to stat (e.g., deleted during iteration)
-  const statsResults = await Promise.all(
-    snapshotFiles.map(async (file) => {
-      const filePath = join(snapshotsDir, file)
-      try {
-        const stats = await stat(filePath)
-        return {
-          name: file,
-          createdAt: stats.mtime.toISOString(),
-          size: stats.size,
-        }
-      } catch {
-        // File may have been deleted between readdir and stat
-        return null
-      }
-    }),
-  )
-
-  return statsResults.filter(
-    (result): result is NonNullable<typeof result> => result !== null,
-  )
-}

package/engines/meilisearch/binary-manager.ts

@@ -1,43 +0,0 @@
-/**
- * Meilisearch Binary Manager
- *
- * Handles downloading, extracting, and managing Meilisearch binaries from hostdb.
- * Extends BaseBinaryManager for shared download/extraction logic.
- */
-
-import {
-  BaseBinaryManager,
-  type BinaryManagerConfig,
-} from '../../core/base-binary-manager'
-import { getBinaryUrl } from './binary-urls'
-import { normalizeVersion } from './version-maps'
-import { Engine, type Platform, type Arch } from '../../types'
-
-class MeilisearchBinaryManager extends BaseBinaryManager {
-  protected readonly config: BinaryManagerConfig = {
-    engine: Engine.Meilisearch,
-    engineName: 'meilisearch',
-    displayName: 'Meilisearch',
-    serverBinary: 'meilisearch',
-  }
-
-  protected getBinaryUrlFromModule(
-    version: string,
-    platform: Platform,
-    arch: Arch,
-  ): string {
-    return getBinaryUrl(version, platform, arch)
-  }
-
-  protected normalizeVersionFromModule(version: string): string {
-    return normalizeVersion(version)
-  }
-
-  protected parseVersionFromOutput(stdout: string): string | null {
-    // Extract version from output like "meilisearch 1.33.1" or "v1.33.1"
-    const match = stdout.match(/(?:meilisearch\s+)?v?(\d+\.\d+\.\d+)/)
-    return match?.[1] ?? null
-  }
-}
-
-export const meilisearchBinaryManager = new MeilisearchBinaryManager()

package/engines/meilisearch/binary-urls.ts

@@ -1,69 +0,0 @@
-import { normalizeVersion } from './version-maps'
-import { buildHostdbUrl } from '../../core/hostdb-client'
-import { Engine, Platform, type Arch } from '../../types'
-
-/**
- * Supported platform identifiers for hostdb downloads.
- * hostdb uses standard Node.js platform naming - this set validates
- * that a platform/arch combination is supported, not transforms it.
- */
-const SUPPORTED_PLATFORMS = new Set([
-  'darwin-arm64',
-  'darwin-x64',
-  'linux-arm64',
-  'linux-x64',
-  'win32-x64',
-])
-
-/**
- * Get the hostdb platform identifier
- *
- * hostdb uses standard platform naming that matches Node.js identifiers directly.
- * This function validates the platform/arch combination is supported.
- *
- * @param platform - Node.js platform (e.g., 'darwin', 'linux', 'win32')
- * @param arch - Node.js architecture (e.g., 'arm64', 'x64')
- * @returns hostdb platform identifier or null if unsupported
- */
-export function getHostdbPlatform(
-  platform: Platform,
-  arch: Arch,
-): string | null {
-  const key = `${platform}-${arch}`
-  return SUPPORTED_PLATFORMS.has(key) ? key : null
-}
-
-/**
- * Build the download URL for Meilisearch binaries from hostdb
- *
- * Format: https://registry.layerbase.host/meilisearch-{version}/meilisearch-{version}-{platform}-{arch}.{ext}
- *
- * @param version - Meilisearch version (e.g., '1', '1.33.1')
- * @param platform - Platform identifier (e.g., 'darwin', 'linux', 'win32')
- * @param arch - Architecture identifier (e.g., 'arm64', 'x64')
- * @returns Download URL for the binary
- */
-export function getBinaryUrl(
-  version: string,
-  platform: Platform,
-  arch: Arch,
-): string {
-  const platformKey = `${platform}-${arch}`
-  const hostdbPlatform = getHostdbPlatform(platform, arch)
-  if (!hostdbPlatform) {
-    const supported = Array.from(SUPPORTED_PLATFORMS).join(', ')
-    throw new Error(
-      `Unsupported platform: ${platformKey}. Supported platforms: ${supported}`,
-    )
-  }
-
-  // Normalize version (handles major version lookup and X.Y -> X.Y.Z conversion)
-  const fullVersion = normalizeVersion(version)
-  const ext = platform === Platform.Win32 ? 'zip' : 'tar.gz'
-
-  return buildHostdbUrl(Engine.Meilisearch, {
-    version: fullVersion,
-    hostdbPlatform,
-    extension: ext,
-  })
-}

package/engines/meilisearch/hostdb-releases.ts

@@ -1,26 +0,0 @@
-/**
- * hostdb Releases Module for Meilisearch
- *
- * Fetches Meilisearch binary information from the hostdb repository at
- * https://github.com/robertjbass/hostdb
- */
-
-import { createHostdbReleases } from '../../core/hostdb-releases-factory'
-import {
-  MEILISEARCH_VERSION_MAP,
-  SUPPORTED_MAJOR_VERSIONS,
-} from './version-maps'
-import { meilisearchBinaryManager } from './binary-manager'
-import { Engine } from '../../types'
-
-const hostdbReleases = createHostdbReleases({
-  engine: Engine.Meilisearch,
-  displayName: 'Meilisearch',
-  versionMap: MEILISEARCH_VERSION_MAP,
-  supportedMajorVersions: SUPPORTED_MAJOR_VERSIONS,
-  groupingStrategy: 'single-digit',
-  listInstalled: () => meilisearchBinaryManager.listInstalled(),
-})
-
-export const fetchAvailableVersions = hostdbReleases.fetchAvailableVersions
-export const getLatestVersion = hostdbReleases.getLatestVersion