spindb 0.35.4 → 0.36.1

package/engines/weaviate/README.md

@@ -0,0 +1,302 @@
+# Weaviate Engine Implementation
+
+## Overview
+
+Weaviate is an AI-native vector database with REST and gRPC APIs. Like Qdrant and Meilisearch, it uses HTTP for all operations. Uses classes/collections instead of traditional databases.
+
+## Platform Support
+
+| Platform | Architecture | Status | Notes |
+|----------|--------------|--------|-------|
+| darwin | x64 | Supported | Uses hostdb binaries |
+| darwin | arm64 | Supported | Uses hostdb binaries (Apple Silicon) |
+| linux | x64 | Supported | Uses hostdb binaries |
+| linux | arm64 | Supported | Uses hostdb binaries |
+| win32 | x64 | Supported | Uses hostdb binaries |
+
+## Binary Packaging
+
+### Archive Format
+- **Unix (macOS/Linux)**: `tar.gz`
+- **Windows**: `zip`
+
+### Archive Structure
+```text
+weaviate/
+└── bin/
+    └── weaviate           # Server binary
+```
+
+### Version Map Sync
+
+```typescript
+export const WEAVIATE_VERSION_MAP: Record<string, string> = {
+  '1': '1.35.7',
+}
+```
+
+## Implementation Details
+
+### Binary Manager
+
+Weaviate uses `BaseBinaryManager` with a custom `verify()` override:
+
+```typescript
+// Weaviate doesn't support --version (as of v1.35.x)
+// Verification just checks binary existence
+async verify(): Promise<boolean> {
+  return existsSync(binaryPath)
+}
+```
+
+See: https://github.com/weaviate/weaviate/issues/6571
+
+### Version Parsing
+
+Not applicable for current version (no `--version` flag). The `parseVersionFromOutput` method is implemented for forward compatibility when the flag is added:
+- **Parse pattern**: `/(?:weaviate\s+)?v?(\d+\.\d+\.\d+)/`
+
+### REST API Engine
+
+Weaviate is a **REST API engine** - it doesn't have a CLI shell:
+- `spindb run` is **NOT applicable**
+- `spindb connect` opens the web dashboard in browser
+- All data operations use HTTP REST API
+
+### Dual Ports
+
+Weaviate uses two ports:
+- **HTTP Port** (default 8080): REST API
+- **gRPC Port** (default 8081): gRPC API (typically HTTP + 1)
+
+### Default Configuration
+
+- **Default HTTP Port**: 8080 (auto-increments on conflict)
+- **gRPC Port**: HTTP port + 1
+- **Health Endpoint**: `/v1/.well-known/ready`
+- **Schema Endpoint**: `/v1/schema`
+- **PID File**: `weaviate.pid` in container directory
+
+### Environment Variable Configuration
+
+Weaviate uses environment variables (not a config file):
+
+```bash
+PERSISTENCE_DATA_PATH=/path/to/data
+BACKUP_FILESYSTEM_PATH=/path/to/data/backups
+QUERY_DEFAULTS_LIMIT=25
+AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true
+DEFAULT_VECTORIZER_MODULE=none
+ENABLE_MODULES=backup-filesystem
+GRPC_PORT=8081
+CLUSTER_HOSTNAME=node-{port}        # Must be unique per container
+CLUSTER_GOSSIP_BIND_PORT={port+100}  # Memberlist gossip (default 7946)
+CLUSTER_DATA_BIND_PORT={port+101}    # Memberlist data (default 7947)
+RAFT_PORT={port+200}                 # Raft consensus (default 8300)
+RAFT_INTERNAL_RPC_PORT={port+201}    # Raft internal RPC (default 8301)
+```
+
+### Internal Cluster Ports
+
+Weaviate uses 4 internal cluster ports in addition to HTTP and gRPC. These **must be unique per container** or Weaviate will fail to start (or silently conflict with other instances):
+
+| Port | Default | SpinDB Formula | Purpose |
+|------|---------|----------------|---------|
+| HTTP | 8080 | `{port}` | REST API |
+| gRPC | 8081 | `{port}+1` | gRPC API |
+| Gossip | 7946 | `{port}+100` | Memberlist gossip |
+| Data | 7947 | `{port}+101` | Memberlist data |
+| Raft | 8300 | `{port}+200` | Raft consensus |
+| Raft RPC | 8301 | `{port}+201` | Raft internal RPC |
+
+### Connection String Format
+
+```text
+http://127.0.0.1:{port}
+```
+
+### Web Dashboard
+
+The `connect` command opens the root URL in the default browser:
+```text
+http://localhost:{port}/
+```
+
+## Backup & Restore
+
+### Backup Formats
+
+| Format | Extension | Tool | Notes |
+|--------|-----------|------|-------|
+| snapshot | `.snapshot` | REST API | Weaviate filesystem backup |
+
+### Backup API
+
+Backup and restore use Weaviate's filesystem backup endpoints:
+- `POST /v1/backups/filesystem` - Create backup (with status polling)
+- `GET /v1/backups/filesystem/{id}` - Check backup status
+- `POST /v1/backups/filesystem/{id}/restore` - Restore backup
+
+### Backup Flow
+
+1. `BACKUP_FILESYSTEM_PATH` env var points to `{dataDir}/backups`
+2. `ENABLE_MODULES=backup-filesystem` must be set (or backup API returns 404)
+3. Trigger backup via `POST /v1/backups/filesystem` with `{ id: "spindb-backup-{ts}" }`
+4. Poll status via `GET /v1/backups/filesystem/{id}` until `SUCCESS`
+5. Copy backup **directory** (not a single file) from `{backupsDir}/{id}/` to output path
+
+### Restore Flow
+
+1. Copy backup directory into target container's `{backupsDir}/{backupId}/`
+2. **The directory name MUST match the internal backup ID** stored in `backup_config.json` inside the backup. Weaviate validates this and returns 422 on mismatch.
+3. Start Weaviate
+4. Trigger restore via `POST /v1/backups/filesystem/{backupId}/restore`
+5. If restoring to a container with a different `CLUSTER_HOSTNAME`, pass `node_mapping` in the request body:
+   ```json
+   { "node_mapping": { "node-8080": "node-9090" } }
+   ```
+6. Poll restore status until `SUCCESS`
+
+## Integration Test Notes
+
+### REST API Testing
+
+Integration tests use `fetch()` for operations, not CLI tools.
+
+### Test Ports
+
+```typescript
+weaviate: { base: 8090, clone: 8092, renamed: 8091 }
+```
+
+## Docker E2E Test Notes
+
+Weaviate Docker E2E uses `curl` for all operations:
+
+```bash
+# Health check
+curl http://localhost:8080/v1/.well-known/ready
+
+# Create class
+curl -X POST http://localhost:8080/v1/schema \
+  -H 'Content-Type: application/json' \
+  -d '{"class":"TestVectors","vectorizer":"none","properties":[...]}'
+
+# Insert objects (batch)
+curl -X POST http://localhost:8080/v1/batch/objects \
+  -H 'Content-Type: application/json' \
+  -d '{"objects":[...]}'
+```
+
+## Known Issues & Gotchas
+
+### 1. No --version Flag
+
+Weaviate binary doesn't support `--version` as of v1.35.x. Tracked in [weaviate/weaviate#6571](https://github.com/weaviate/weaviate/issues/6571). Binary verification only checks file existence. Same pattern as CouchDB.
+
+### 2. No CLI Shell
+
+`spindb run` does nothing for Weaviate. Use the REST API or web dashboard.
+
+### 3. Vector Database Semantics
+
+Weaviate uses "classes" (or "collections") instead of "databases". Operations are vector-centric:
+- Create classes with property schemas
+- Insert objects with optional vectors
+- Search by vector similarity or filters
+
+### 4. Internal Cluster Ports Must Be Unique
+
+Weaviate binds 4 internal ports (gossip 7946, data 7947, raft 8300, raft RPC 8301) by default. Running multiple Weaviate containers without unique ports causes silent conflicts or startup failures. SpinDB derives unique ports from the HTTP port (see "Internal Cluster Ports" above).
+
+### 5. ENABLE_MODULES Required for Backup
+
+`ENABLE_MODULES=backup-filesystem` must be set at startup or the backup/restore API endpoints return 404.
+
+### 6. Backup Directory Name Must Match Internal ID
+
+Weaviate backups are directories (not single files). The backup directory name **must match** the internal backup ID in `backup_config.json`. When copying a backup to a new location, `restore.ts` reads `backup_config.json` to discover the real ID and names the target directory accordingly.
+
+### 7. Node Mapping for Cross-Container Restore
+
+When restoring a backup to a container with a different `CLUSTER_HOSTNAME`, the Weaviate restore API requires a `node_mapping` parameter. Without it, restore fails with "cannot resolve hostname" (422).
+
+### 8. Windows Backup Fails (LSM File Locking)
+
+Weaviate on Windows holds exclusive locks on LSM storage files, preventing `fsync` during backup while the server is running. The backup API returns "Access is denied" errors. Integration tests skip the backup/restore clone test on Windows. Same pattern as Meilisearch.
+
+### 9. gRPC Port
+
+The gRPC port is separate from HTTP (HTTP + 1). Ensure both ports are available if using gRPC clients.
+
+### 10. Snapshot Format
+
+Snapshots are Weaviate's native backup format and are not compatible with other databases.
+
+### 11. Health Check Endpoint
+
+Use `/v1/.well-known/ready` for health checks (returns 200 when ready).
+
+### 12. Class/Collection Naming
+
+Weaviate class names must start with an uppercase letter (PascalCase). Container names with dashes are auto-converted (e.g., `my-app` becomes class `My_app`).
+
+## CI/CD Notes
+
+### curl-Based Testing
+
+CI tests use `curl` commands rather than database CLI tools.
+
+### GitHub Actions Cache Step
+
+```yaml
+- name: Cache Weaviate binaries
+  uses: actions/cache@v4
+  with:
+    path: ~/.spindb/bin/weaviate-*
+    key: weaviate-${{ runner.os }}-${{ runner.arch }}-${{ hashFiles('engines/weaviate/version-maps.ts') }}
+```
+
+## REST API Quick Reference
+
+### Schema (Classes)
+```bash
+# List all classes
+GET /v1/schema
+
+# Get class info
+GET /v1/schema/{class}
+
+# Create class
+POST /v1/schema
+
+# Delete class
+DELETE /v1/schema/{class}
+```
+
+### Objects
+```bash
+# Batch insert objects
+POST /v1/batch/objects
+
+# Get object
+GET /v1/objects/{class}/{id}
+
+# Delete object
+DELETE /v1/objects/{class}/{id}
+```
+
+### Search
+```bash
+# GraphQL query
+POST /v1/graphql
+```
+
+### Meta
+```bash
+# Server meta info (includes version)
+GET /v1/meta
+
+# Health/ready check
+GET /v1/.well-known/ready
+```

package/engines/weaviate/api-client.ts

@@ -0,0 +1,61 @@
+/**
+ * Shared Weaviate REST API client utilities
+ */
+
+/**
+ * Make an HTTP request to Weaviate REST API
+ *
+ * @param port - The HTTP port Weaviate is listening on
+ * @param method - HTTP method (GET, POST, PUT, DELETE)
+ * @param path - API path (e.g., '/v1/schema', '/v1/.well-known/ready')
+ * @param body - Optional JSON body for POST/PUT requests
+ * @param timeoutMs - Request timeout in milliseconds (default: 30s)
+ */
+export async function weaviateApiRequest(
+  port: number,
+  method: string,
+  path: string,
+  body?: Record<string, unknown>,
+  timeoutMs = 30000,
+): Promise<{ status: number; data: unknown }> {
+  const url = `http://127.0.0.1:${port}${path}`
+
+  const controller = new AbortController()
+  const timeoutId = setTimeout(() => controller.abort(), timeoutMs)
+
+  const options: RequestInit = {
+    method,
+    headers: {
+      'Content-Type': 'application/json',
+    },
+    signal: controller.signal,
+  }
+
+  if (body) {
+    options.body = JSON.stringify(body)
+  }
+
+  try {
+    const response = await fetch(url, options)
+
+    // Try to parse as JSON, fall back to text for endpoints like /v1/.well-known/ready
+    let data: unknown
+    const contentType = response.headers.get('content-type') || ''
+    if (contentType.includes('application/json')) {
+      data = await response.json()
+    } else {
+      data = await response.text()
+    }
+
+    return { status: response.status, data }
+  } catch (error) {
+    if (error instanceof Error && error.name === 'AbortError') {
+      throw new Error(
+        `Weaviate API request timed out after ${timeoutMs / 1000}s: ${method} ${path}`,
+      )
+    }
+    throw error
+  } finally {
+    clearTimeout(timeoutId)
+  }
+}

package/engines/weaviate/backup.ts

@@ -0,0 +1,145 @@
+/**
+ * Weaviate backup module
+ * Supports snapshot-based backup using Weaviate's filesystem backup API.
+ *
+ * Weaviate's filesystem backup creates a directory at BACKUP_FILESYSTEM_PATH/<id>/
+ * containing backup metadata and class data. We copy this entire directory
+ * as the "snapshot" for backup/restore.
+ */
+
+import { mkdir, stat, cp, readdir } from 'fs/promises'
+import { existsSync } from 'fs'
+import { join, dirname } from 'path'
+import { logDebug } from '../../core/error-handler'
+import { paths } from '../../config/paths'
+import { weaviateApiRequest } from './api-client'
+import type { ContainerConfig, BackupOptions, BackupResult } from '../../types'
+
+/**
+ * Create a snapshot backup using Weaviate's REST API.
+ * Triggers a filesystem backup, polls for completion, then copies the
+ * backup directory to the output path.
+ */
+export async function createBackup(
+  container: ContainerConfig,
+  outputPath: string,
+  _options: BackupOptions,
+): Promise<BackupResult> {
+  const { port, name } = container
+
+  // Ensure output parent directory exists
+  const outputDir = dirname(outputPath)
+  if (!existsSync(outputDir)) {
+    await mkdir(outputDir, { recursive: true })
+  }
+
+  // Generate a unique backup ID
+  const backupId = `spindb-backup-${Date.now()}`
+
+  // Trigger backup creation via REST API
+  logDebug(
+    `Creating Weaviate backup '${backupId}' via REST API on port ${port}`,
+  )
+
+  const response = await weaviateApiRequest(
+    port,
+    'POST',
+    '/v1/backups/filesystem',
+    { id: backupId },
+    600000, // 10 minute timeout
+  )
+
+  if (response.status !== 200) {
+    throw new Error(
+      `Failed to create Weaviate backup: ${JSON.stringify(response.data)}`,
+    )
+  }
+
+  logDebug(`Weaviate backup initiated: ${backupId}`)
+
+  // Poll for backup completion
+  const maxWait = 300000 // 5 minutes
+  const startTime = Date.now()
+
+  let backupCompleted = false
+
+  while (Date.now() - startTime < maxWait) {
+    const statusResponse = await weaviateApiRequest(
+      port,
+      'GET',
+      `/v1/backups/filesystem/${backupId}`,
+    )
+
+    if (statusResponse.status === 200) {
+      const statusData = statusResponse.data as {
+        status?: string
+        path?: string
+      }
+
+      if (statusData.status === 'SUCCESS') {
+        logDebug(`Weaviate backup completed: ${backupId}`)
+        backupCompleted = true
+        break
+      }
+
+      if (statusData.status === 'FAILED') {
+        throw new Error(`Weaviate backup failed: ${JSON.stringify(statusData)}`)
+      }
+
+      logDebug(`Backup status: ${statusData.status}, waiting...`)
+    }
+
+    await new Promise((r) => setTimeout(r, 1000))
+  }
+
+  if (!backupCompleted) {
+    throw new Error(
+      `Weaviate backup '${backupId}' timed out after ${maxWait / 1000}s without completing`,
+    )
+  }
+
+  // Weaviate stores backup at BACKUP_FILESYSTEM_PATH/<backupId>/
+  // BACKUP_FILESYSTEM_PATH is set to <dataDir>/backups in start()
+  const dataDir = paths.getContainerDataPath(name, { engine: 'weaviate' })
+  const backupDir = join(dataDir, 'backups', backupId)
+
+  if (!existsSync(backupDir)) {
+    throw new Error(
+      `Weaviate backup directory not found at ${backupDir} after completion`,
+    )
+  }
+
+  // Copy entire backup directory to output path
+  await cp(backupDir, outputPath, { recursive: true })
+
+  // Get total size of backup directory
+  const files = await readdir(backupDir, { recursive: true })
+  let totalSize = 0
+  for (const file of files) {
+    try {
+      const filePath = join(backupDir, String(file))
+      const stats = await stat(filePath)
+      if (stats.isFile()) {
+        totalSize += stats.size
+      }
+    } catch {
+      // Skip inaccessible files
+    }
+  }
+
+  return {
+    path: outputPath,
+    format: 'snapshot',
+    size: totalSize,
+  }
+}
+
+/**
+ * Create a backup for cloning purposes
+ */
+export async function createCloneBackup(
+  container: ContainerConfig,
+  outputPath: string,
+): Promise<BackupResult> {
+  return createBackup(container, outputPath, { database: 'default' })
+}

package/engines/weaviate/binary-manager.ts

@@ -0,0 +1,80 @@
+/**
+ * Weaviate Binary Manager
+ *
+ * Handles downloading, extracting, and managing Weaviate binaries from hostdb.
+ * Extends BaseBinaryManager for shared download/extraction logic.
+ *
+ * Note: Weaviate doesn't support --version flag (as of v1.35.x). We override
+ * verify() to just check binary existence instead of running it.
+ */
+
+import { existsSync } from 'fs'
+import { join } from 'path'
+import {
+  BaseBinaryManager,
+  type BinaryManagerConfig,
+} from '../../core/base-binary-manager'
+import { getBinaryUrl } from './binary-urls'
+import { normalizeVersion } from './version-maps'
+import { Engine, Platform, type Arch } from '../../types'
+import { paths } from '../../config/paths'
+
+class WeaviateBinaryManager extends BaseBinaryManager {
+  protected readonly config: BinaryManagerConfig = {
+    engine: Engine.Weaviate,
+    engineName: 'weaviate',
+    displayName: 'Weaviate',
+    serverBinary: 'weaviate',
+  }
+
+  protected getBinaryUrlFromModule(
+    version: string,
+    platform: Platform,
+    arch: Arch,
+  ): string {
+    return getBinaryUrl(version, platform, arch)
+  }
+
+  protected normalizeVersionFromModule(version: string): string {
+    return normalizeVersion(version)
+  }
+
+  protected parseVersionFromOutput(stdout: string): string | null {
+    // Extract version from output like "weaviate v1.35.7" or "1.35.7"
+    const match = stdout.match(/(?:weaviate\s+)?v?(\d+\.\d+\.\d+)/)
+    return match?.[1] ?? null
+  }
+
+  /**
+   * Override verify to just check binary existence.
+   * Weaviate doesn't support --version flag (as of v1.35.x).
+   * See: https://github.com/weaviate/weaviate/issues/6571
+   */
+  async verify(
+    version: string,
+    platform: Platform,
+    arch: Arch,
+  ): Promise<boolean> {
+    const fullVersion = this.getFullVersion(version)
+    const binPath = paths.getBinaryPath({
+      engine: this.config.engineName,
+      version: fullVersion,
+      platform,
+      arch,
+    })
+
+    const ext = platform === Platform.Win32 ? '.exe' : ''
+    const serverPath = join(binPath, 'bin', `${this.config.serverBinary}${ext}`)
+
+    if (!existsSync(serverPath)) {
+      throw new Error(
+        `${this.config.displayName} binary not found at ${binPath}/bin/`,
+      )
+    }
+
+    // Just verify the binary exists - we can't run --version on Weaviate
+    return true
+  }
+}
+
+export const weaviateBinaryManager = new WeaviateBinaryManager()

package/engines/weaviate/binary-urls.ts

@@ -0,0 +1,115 @@
+import { WEAVIATE_VERSION_MAP } from './version-maps'
+import { buildHostdbUrl } from '../../core/hostdb-client'
+import { logDebug } from '../../core/error-handler'
+import { Engine, Platform, type Arch } from '../../types'
+
+/**
+ * Supported platform identifiers for hostdb downloads.
+ * hostdb uses standard Node.js platform naming - this set validates
+ * that a platform/arch combination is supported, not transforms it.
+ */
+const SUPPORTED_PLATFORMS = new Set([
+  'darwin-arm64',
+  'darwin-x64',
+  'linux-arm64',
+  'linux-x64',
+  'win32-x64',
+])
+
+/**
+ * Get the hostdb platform identifier
+ *
+ * hostdb uses standard platform naming that matches Node.js identifiers directly.
+ * This function validates the platform/arch combination is supported.
+ *
+ * @param platform - Node.js platform (e.g., 'darwin', 'linux', 'win32')
+ * @param arch - Node.js architecture (e.g., 'arm64', 'x64')
+ * @returns hostdb platform identifier or null if unsupported
+ */
+export function getHostdbPlatform(
+  platform: Platform,
+  arch: Arch,
+): string | null {
+  const key = `${platform}-${arch}`
+  return SUPPORTED_PLATFORMS.has(key) ? key : null
+}
+
+/**
+ * Build the download URL for Weaviate binaries from hostdb
+ *
+ * Format: https://registry.layerbase.host/weaviate-{version}/weaviate-{version}-{platform}-{arch}.{ext}
+ *
+ * @param version - Weaviate version (e.g., '1', '1.35.7')
+ * @param platform - Platform identifier (e.g., 'darwin', 'linux', 'win32')
+ * @param arch - Architecture identifier (e.g., 'arm64', 'x64')
+ * @returns Download URL for the binary
+ */
+export function getBinaryUrl(
+  version: string,
+  platform: Platform,
+  arch: Arch,
+): string {
+  const platformKey = `${platform}-${arch}`
+  const hostdbPlatform = getHostdbPlatform(platform, arch)
+  if (!hostdbPlatform) {
+    const supported = Array.from(SUPPORTED_PLATFORMS).join(', ')
+    throw new Error(
+      `Unsupported platform: ${platformKey}. Supported platforms: ${supported}`,
+    )
+  }
+
+  // Normalize version (handles major version lookup and X.Y -> X.Y.Z conversion)
+  const fullVersion = normalizeVersion(version, WEAVIATE_VERSION_MAP)
+  const ext = platform === Platform.Win32 ? 'zip' : 'tar.gz'
+
+  return buildHostdbUrl(Engine.Weaviate, {
+    version: fullVersion,
+    hostdbPlatform,
+    extension: ext,
+  })
+}
+
+/**
+ * Normalize version string to X.Y.Z format
+ *
+ * @param version - Version string (e.g., '1', '1.35', '1.35.7')
+ * @param versionMap - Optional version map for major version lookup
+ * @returns Normalized version (e.g., '1.35.7')
+ */
+function normalizeVersion(
+  version: string,
+  versionMap: Record<string, string> = WEAVIATE_VERSION_MAP,
+): string {
+  // Check if it's an exact key in the map (handles "1", "1.35", etc.)
+  if (versionMap[version]) {
+    return versionMap[version]
+  }
+
+  const parts = version.split('.')
+
+  // If it's already a full version (X.Y.Z), return as-is
+  if (parts.length === 3) {
+    return version
+  }
+
+  // For two-part versions (e.g., "1.35"), first try exact two-part key, then fall back to major
+  if (parts.length === 2) {
+    const twoPart = `${parts[0]}.${parts[1]}`
+    if (versionMap[twoPart]) {
+      return versionMap[twoPart]
+    }
+    // Fall back to major version for latest patch
+    const major = parts[0]
+    const mapped = versionMap[major]
+    if (mapped) {
+      return mapped
+    }
+  }
+
+  // Unknown version format - log and return as-is
+  // This may cause download failures if the version doesn't exist in hostdb
+  logDebug(
+    `Weaviate version '${version}' not in version map, may not be available in hostdb`,
+  )
+  return version
+}