@strav/search 0.3.20 → 0.3.22

package/README.md

@@ -1,6 +1,6 @@
 # @strav/search
 
-Full-text search for the [Strav](https://www.npmjs.com/package/@strav/core) framework. Unified API for Meilisearch, Typesense, and Algolia with automatic indexing via model events.
+Full-text search for the [Strav](https://www.npmjs.com/package/@strav/core) framework. Unified API across several engines — including a built-in `embedded` driver that runs in-process with no external service to deploy.
 
 ## Install
 
@@ -51,16 +51,135 @@ await search.delete('posts', ['1'])
 
 ## Drivers
 
+- **Embedded** — in-process SQLite FTS5, zero deps, recommended for self-host / SMB (~50k–500k docs)
+- **Postgres FTS** — tsvector + GIN + pg_trgm, drop-in upgrade for higher volume (1M–100M docs)
 - **Meilisearch** — fast, typo-tolerant, self-hosted
 - **Typesense** — open-source, instant search
 - **Algolia** — hosted search-as-a-service
 - **Null** — no-op driver for testing
 
+### Embedded driver
+
+Runs entirely inside your app process using `bun:sqlite`'s FTS5 engine — no Meilisearch/Typesense container to run. Each index is a single `.sqlite` file in the configured data directory.
+
+Features:
+
+- BM25 ranking with per-field weights (via `searchableAttributes` ordering)
+- Prefix (`type*`), phrase (`"quick brown fox"`), negation (`-foo`), required (`+foo`)
+- Porter stemmer for English morphology
+- Typo tolerance (Levenshtein-1) on the fly, configurable
+- Highlighted snippets with `<mark>` tags
+- Object-form filters with equality, `in`, and comparison operators
+
+Limitations for v1:
+
+- English stemming only (other languages are tokenised but not stemmed)
+- One writer at a time per index file (SQLite WAL — concurrent reads are fine)
+- Object-form filters only; raw SQL filter strings are rejected
+- Index settings changes require recreating the index
+
+Configuration:
+
+```ts
+// config/search.ts
+embedded: {
+  driver: 'embedded',
+  path: env('SEARCH_PATH', './storage/search'),   // directory of .sqlite files
+  synchronous: 'NORMAL',                            // 'OFF' | 'NORMAL' | 'FULL'
+  typoTolerance: 'auto',                            // 'off' | 'auto' | { minTokenLength, maxDistance }
+}
+```
+
+Select it as the default with `SEARCH_DRIVER=embedded`.
+
+### Postgres FTS driver
+
+Higher-volume tier (1M–100M docs per index) backed by your existing Postgres. Same `SearchEngine` interface as the embedded driver — drop-in swap by changing one config line.
+
+Features:
+
+- BM25-shaped ranking via `ts_rank_cd(fts, q, 1 | 32)` with per-field weights (`A`/`B`/`C`/`D`)
+- `websearch_to_tsquery` Google-style queries plus prefix (`type*`)
+- Multi-language stemming via Postgres text-search configurations (`english`, `french`, ...) — set per index
+- Levenshtein-near typo tolerance via `pg_trgm` + optional `fuzzystrmatch`
+- `<mark>`-highlighted snippets via `ts_headline`, computed only on the top-K to keep latency bounded
+- Object-form filters with `eq`/`neq`/`gt`/`gte`/`lt`/`lte`/`in`/`nin` against generated typed columns
+- One table per index in a dedicated `strav_search` schema (auto-created)
+
+Requirements:
+
+- Postgres ≥ 15
+- `pg_trgm` extension (auto-`CREATE EXTENSION IF NOT EXISTS` on first use; superuser or owner privilege)
+- `fuzzystrmatch` is optional — if present, typo expansion re-ranks trigram candidates with bounded Levenshtein for higher precision
+
+Configuration:
+
+```ts
+postgres: {
+  driver: 'postgres-fts',
+  // Optional: pass a Bun SQL handle. Falls back to @strav/database's Database.raw.
+  // connection: db.sql,
+  schema: env('SEARCH_PG_SCHEMA', 'strav_search'),
+  language: env('SEARCH_PG_LANGUAGE', 'english'),
+  typoTolerance: env('SEARCH_TYPO_TOLERANCE', 'auto'),
+  workMem: env('SEARCH_PG_WORK_MEM', '64MB'),
+  gin: { fastupdate: false },  // better tail latency
+}
+```
+
+Select it with `SEARCH_DRIVER=postgres`.
+
+Limitations for v1:
+
+- Settings change (e.g. add a new searchable attribute) requires `bun strav search:rebuild <model>`. Tier picked by row count: in-place UPDATE under 100k, batched UPDATE up to 10M, dual-table swap deferred to v1.1 with a clear error above 10M.
+- Adding a new `filterableAttribute` on an existing large table currently rewrites the whole heap (`ALTER TABLE ADD COLUMN ... GENERATED ... STORED`). Plan an offline window for big tables in v1.
+- One language per index — mixed-locale indexes deferred.
+- Object-form filters only; raw SQL filter strings rejected.
+
+Ranking note: `ts_rank_cd` is BM25-*shaped* (length normalisation + bounded mapping), not strict BM25. For the size and shape of corpora the driver targets, the difference is small in practice; the embedded driver remains the answer when strict BM25 matters and the corpus fits.
+
+Model example with per-field weights (column order determines BM25 weight — title first = highest):
+
+```ts
+class Ticket extends searchable(BaseModel) {
+  static searchableSettings() {
+    return {
+      searchableAttributes: ['subject', 'body'],
+      filterableAttributes: ['status', 'priority'],
+      sortableAttributes: ['priority', 'created_at'],
+    }
+  }
+}
+```
+
+#### Replacing Postgres `tsvector`
+
+If you've been using raw `tsvector` columns, the embedded driver gives you better ranking, typo tolerance, and highlighted snippets without adding a network service. The migration is roughly:
+
+```ts
+// Before: hand-rolled tsvector query
+const rows = await db.sql`
+  SELECT id, subject, ts_rank_cd(fts, q) AS rank
+  FROM tickets, websearch_to_tsquery('english', ${q}) q
+  WHERE fts @@ q ORDER BY rank DESC LIMIT 20
+`
+
+// After: searchable() + embedded driver
+const results = await Ticket.search(q, {
+  perPage: 20,
+  attributesToHighlight: ['subject', 'body'],
+})
+```
+
+You run `bun strav search:import Ticket` once to populate the index, then model events keep it up to date.
+
 ## CLI
 
 ```bash
-bun strav search:import    # Import all searchable models
-bun strav search:flush     # Flush all indexes
+bun strav search:import <model>      # Import all records for a model
+bun strav search:flush <model>       # Flush all documents from an index
+bun strav search:optimize <model>    # (embedded) Merge FTS5 segments; run periodically
+bun strav search:rebuild <model>     # (postgres) Recompute fts after settings change
 ```
 
 ## Documentation

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@strav/search",
-  "version": "0.3.20",
+  "version": "0.3.22",
   "type": "module",
   "description": "Full-text search for the Strav framework",
   "license": "MIT",
@@ -18,9 +18,9 @@
     "tsconfig.json"
   ],
   "peerDependencies": {
-    "@strav/kernel": "0.3.20",
-    "@strav/database": "0.3.20",
-    "@strav/cli": "0.3.20"
+    "@strav/kernel": "0.3.22",
+    "@strav/database": "0.3.22",
+    "@strav/cli": "0.3.22"
   },
   "scripts": {
     "test": "bun test tests/",

package/src/commands/search_optimize.ts

@@ -0,0 +1,52 @@
+import type { Command } from 'commander'
+import chalk from 'chalk'
+import { bootstrap, shutdown } from '@strav/cli'
+import { BaseModel } from '@strav/database'
+import SearchManager from '../search_manager.ts'
+import { EmbeddedDriver } from '../drivers/embedded/index.ts'
+
+export function register(program: Command): void {
+  program
+    .command('search:optimize <model>')
+    .description("Merge FTS5 segments for a model's index (embedded driver only)")
+    .action(async (modelPath: string) => {
+      let db
+      try {
+        const { db: database, config } = await bootstrap()
+        db = database
+
+        new BaseModel(db)
+        new SearchManager(config)
+
+        const resolved = require.resolve(`${process.cwd()}/${modelPath}`)
+        const module = await import(resolved)
+        const ModelClass = module.default ?? (Object.values(module)[0] as any)
+
+        if (typeof ModelClass?.searchableAs !== 'function') {
+          console.error(chalk.red(`Model "${modelPath}" does not use the searchable() mixin.`))
+          process.exit(1)
+        }
+
+        const indexName = SearchManager.indexName(ModelClass.searchableAs())
+        const engine = SearchManager.engine()
+
+        if (!(engine instanceof EmbeddedDriver)) {
+          console.error(
+            chalk.red(
+              `search:optimize is only meaningful for the embedded driver (current: ${engine.name}).`
+            )
+          )
+          process.exit(1)
+        }
+
+        console.log(chalk.dim(`Optimizing "${indexName}"...`))
+        engine.optimize(indexName)
+        console.log(chalk.green(`Optimized "${indexName}".`))
+      } catch (err) {
+        console.error(chalk.red(`Error: ${err instanceof Error ? err.message : err}`))
+        process.exit(1)
+      } finally {
+        if (db) await shutdown(db)
+      }
+    })
+}

package/src/commands/search_rebuild.ts

@@ -0,0 +1,73 @@
+import type { Command } from 'commander'
+import chalk from 'chalk'
+import { bootstrap, shutdown } from '@strav/cli'
+import { BaseModel } from '@strav/database'
+import SearchManager from '../search_manager.ts'
+import { PostgresFtsDriver } from '../drivers/postgres/index.ts'
+
+export function register(program: Command): void {
+  program
+    .command('search:rebuild <model>')
+    .description("Recompute a model's fts column in place (postgres-fts driver only)")
+    .option('--no-reindex', "Skip the GIN REINDEX after the rebuild")
+    .option('--pause <ms>', 'Pause between batches in tier-2 mode (default 50)', '50')
+    .action(async (modelPath: string, options: { reindex: boolean; pause: string }) => {
+      let db
+      try {
+        const { db: database, config } = await bootstrap()
+        db = database
+
+        new BaseModel(db)
+        new SearchManager(config)
+
+        const resolved = require.resolve(`${process.cwd()}/${modelPath}`)
+        const module = await import(resolved)
+        const ModelClass = module.default ?? (Object.values(module)[0] as any)
+
+        if (typeof ModelClass?.searchableAs !== 'function') {
+          console.error(chalk.red(`Model "${modelPath}" does not use the searchable() mixin.`))
+          process.exit(1)
+        }
+
+        const indexName = SearchManager.indexName(ModelClass.searchableAs())
+        const engine = SearchManager.engine()
+
+        if (!(engine instanceof PostgresFtsDriver)) {
+          console.error(
+            chalk.red(
+              `search:rebuild is only meaningful for the postgres-fts driver (current: ${engine.name}).`
+            )
+          )
+          process.exit(1)
+        }
+
+        // Make sure the engine knows about the model's settings (so rebuild
+        // computes fts with the right weights/language).
+        const settings = (ModelClass.searchableSettings?.() ?? undefined) as any
+        if (settings) await engine.createIndex(indexName, settings)
+
+        console.log(chalk.dim(`Rebuilding "${indexName}"...`))
+        const result = await engine.rebuild(indexName, {
+          reindex: options.reindex !== false,
+          pauseMs: Number(options.pause),
+          onProgress: (done, total) => {
+            const pct = total > 0 ? Math.round((done / total) * 100) : 100
+            process.stdout.write(`\r  ${done}/${total} rows (${pct}%) `)
+          },
+        })
+        if (result.tier === 2) process.stdout.write('\n')
+
+        console.log(
+          chalk.green(
+            `Rebuilt ${result.rows} row(s) in "${indexName}" using tier-${result.tier} ` +
+              `strategy (${result.elapsedMs}ms).`
+          )
+        )
+      } catch (err) {
+        console.error(chalk.red(`Error: ${err instanceof Error ? err.message : err}`))
+        process.exit(1)
+      } finally {
+        if (db) await shutdown(db)
+      }
+    })
+}

package/src/drivers/embedded/embedded_driver.ts

@@ -0,0 +1,136 @@
+import type { SearchEngine } from '../../search_engine.ts'
+import type {
+  SearchDocument,
+  SearchOptions,
+  SearchResult,
+  IndexSettings,
+  DriverConfig,
+} from '../../types.ts'
+import { SqliteEngine } from './engine/sqlite_engine.ts'
+import { resolveTypoTolerance } from './engine/typo_expander.ts'
+import { resolveIndexPath, MEMORY_PATH } from './storage/paths.ts'
+import type { EmbeddedConfig, ResolvedTypoTolerance } from './types.ts'
+
+/**
+ * In-process full-text search driver backed by SQLite FTS5.
+ *
+ * Each index lives in its own SQLite file (or `:memory:` for tests). The
+ * driver maintains a `Map<indexName, SqliteEngine>` and creates engines
+ * lazily on first reference. This means a fresh `upsert()` against a
+ * never-created index will auto-create a default schema (single `_text`
+ * column). Callers that want per-field weights call `createIndex()` first
+ * with their `IndexSettings`.
+ */
+export class EmbeddedDriver implements SearchEngine {
+  readonly name = 'embedded'
+
+  private readonly config: EmbeddedConfig
+  private readonly synchronous: 'OFF' | 'NORMAL' | 'FULL'
+  private readonly typo: ResolvedTypoTolerance
+  private readonly engines = new Map<string, SqliteEngine>()
+  /** Pending settings for indexes that haven't been opened yet. */
+  private readonly pendingSettings = new Map<string, IndexSettings>()
+
+  constructor(config: DriverConfig) {
+    this.config = (config ?? {}) as EmbeddedConfig
+    this.synchronous = this.config.synchronous ?? 'NORMAL'
+    this.typo = resolveTypoTolerance(this.config.typoTolerance)
+  }
+
+  // ── Document operations ──────────────────────────────────────────────────
+
+  async upsert(
+    index: string,
+    id: string | number,
+    document: Record<string, unknown>
+  ): Promise<void> {
+    this.engineFor(index).upsert(id, document)
+  }
+
+  async upsertMany(index: string, documents: SearchDocument[]): Promise<void> {
+    this.engineFor(index).upsertMany(documents)
+  }
+
+  async delete(index: string, id: string | number): Promise<void> {
+    this.engineFor(index).delete(id)
+  }
+
+  async deleteMany(index: string, ids: Array<string | number>): Promise<void> {
+    this.engineFor(index).deleteMany(ids)
+  }
+
+  // ── Index operations ─────────────────────────────────────────────────────
+
+  async flush(index: string): Promise<void> {
+    this.engineFor(index).flush()
+  }
+
+  async deleteIndex(index: string): Promise<void> {
+    const engine = this.engines.get(index)
+    if (engine) {
+      engine.close()
+      this.engines.delete(index)
+    }
+    this.pendingSettings.delete(index)
+
+    const path = resolveIndexPath(this.config, index)
+    if (path === MEMORY_PATH) return
+
+    const fs = await import('node:fs/promises')
+    for (const suffix of ['', '-wal', '-shm']) {
+      try {
+        await fs.unlink(`${path}${suffix}`)
+      } catch (err: any) {
+        if (err?.code !== 'ENOENT') throw err
+      }
+    }
+  }
+
+  async createIndex(index: string, options?: IndexSettings): Promise<void> {
+    if (options) this.pendingSettings.set(index, options)
+    // Force engine instantiation so the schema exists on disk.
+    this.engineFor(index)
+  }
+
+  // ── Search ───────────────────────────────────────────────────────────────
+
+  async search(index: string, query: string, options?: SearchOptions): Promise<SearchResult> {
+    return this.engineFor(index).search(query, options)
+  }
+
+  // ── Lifecycle ────────────────────────────────────────────────────────────
+
+  /** Close all open engines. Call from app shutdown. */
+  close(): void {
+    for (const engine of this.engines.values()) engine.close()
+    this.engines.clear()
+  }
+
+  /** Run FTS5 segment merge on every open index. Use from CLI for periodic ops. */
+  optimize(index?: string): void {
+    if (index) {
+      this.engineFor(index).optimize()
+      return
+    }
+    for (const engine of this.engines.values()) engine.optimize()
+  }
+
+  // ── Internals ────────────────────────────────────────────────────────────
+
+  private engineFor(index: string): SqliteEngine {
+    let engine = this.engines.get(index)
+    if (engine) return engine
+
+    const settings = this.pendingSettings.get(index)
+    engine = new SqliteEngine({
+      path: resolveIndexPath(this.config, index),
+      synchronous: this.synchronous,
+      typoTolerance: this.typo,
+      indexName: index,
+      settings,
+    })
+    this.engines.set(index, engine)
+    this.pendingSettings.delete(index)
+    return engine
+  }
+}

package/src/drivers/embedded/engine/field_registry.ts

@@ -0,0 +1,97 @@
+import type { IndexSettings } from '../../../types.ts'
+
+/** The default searchable column name used when no `searchableAttributes` are configured. */
+export const DEFAULT_TEXT_COLUMN = '_text'
+
+/**
+ * The schema layout for one index: which document attributes feed which FTS5
+ * column and which typed `documents` columns exist for filtering / sorting.
+ *
+ * When a caller doesn't declare `searchableAttributes`, we fall back to a
+ * single `_text` column that concatenates every string-valued field at
+ * indexing time. Users who want per-field weights opt in by passing
+ * `IndexSettings`.
+ */
+export class FieldRegistry {
+  /** FTS5 columns in declaration order — also the order BM25 weights apply in. */
+  readonly searchable: string[]
+  /** Filterable attributes — materialized as typed columns on `documents`. */
+  readonly filterable: string[]
+  /** Sortable attributes — materialized as typed columns on `documents`. */
+  readonly sortable: string[]
+  /** Union of filterable + sortable, deduplicated. */
+  readonly typedColumns: string[]
+  /** Primary key field name — defaults to 'id'. */
+  readonly primaryKey: string
+
+  constructor(settings?: IndexSettings) {
+    this.primaryKey = settings?.primaryKey ?? 'id'
+    this.searchable =
+      settings?.searchableAttributes && settings.searchableAttributes.length > 0
+        ? [...settings.searchableAttributes]
+        : [DEFAULT_TEXT_COLUMN]
+    this.filterable = settings?.filterableAttributes ?? []
+    this.sortable = settings?.sortableAttributes ?? []
+    this.typedColumns = Array.from(new Set([...this.filterable, ...this.sortable]))
+  }
+
+  /** Whether this registry uses the synthesised `_text` column. */
+  get usesDefaultTextColumn(): boolean {
+    return this.searchable.length === 1 && this.searchable[0] === DEFAULT_TEXT_COLUMN
+  }
+
+  /**
+   * Project a document into the values that go into the FTS5 row.
+   * For default mode, concatenate every string-valued field.
+   * For declared mode, pick each named attribute (coerced to string).
+   */
+  projectFtsValues(document: Record<string, unknown>): string[] {
+    if (this.usesDefaultTextColumn) {
+      const parts: string[] = []
+      for (const value of Object.values(document)) {
+        if (typeof value === 'string' && value.length > 0) parts.push(value)
+        else if (Array.isArray(value)) {
+          for (const item of value) {
+            if (typeof item === 'string' && item.length > 0) parts.push(item)
+          }
+        }
+      }
+      return [parts.join(' ')]
+    }
+
+    return this.searchable.map(attr => coerceText(document[attr]))
+  }
+
+  /**
+   * Project a document into the typed-column values stored on `documents`.
+   * Returned in the same order as `typedColumns`.
+   */
+  projectTypedValues(document: Record<string, unknown>): unknown[] {
+    return this.typedColumns.map(attr => coerceTyped(document[attr]))
+  }
+
+  /**
+   * Concatenate every searchable attribute into one long string suitable for
+   * tokenization (used for terms-dictionary maintenance).
+   */
+  concatSearchableText(document: Record<string, unknown>): string {
+    return this.projectFtsValues(document).join(' ')
+  }
+}
+
+function coerceText(value: unknown): string {
+  if (value === null || value === undefined) return ''
+  if (typeof value === 'string') return value
+  if (Array.isArray(value)) {
+    return value.map(v => coerceText(v)).filter(Boolean).join(' ')
+  }
+  if (typeof value === 'number' || typeof value === 'boolean') return String(value)
+  return ''
+}
+
+function coerceTyped(value: unknown): unknown {
+  if (value === null || value === undefined) return null
+  if (Array.isArray(value)) return JSON.stringify(value)
+  if (typeof value === 'object') return JSON.stringify(value)
+  return value
+}