@strav/search 0.4.31 → 1.0.0-alpha.33

package/README.md

@@ -1,191 +0,0 @@
-# @strav/search
-
-Full-text search for the [Strav](https://www.npmjs.com/package/@strav/core) framework. Unified API across several engines — including a built-in `embedded` driver that runs in-process with no external service to deploy.
-
-## Install
-
-```bash
-bun add @strav/search
-bun strav install search
-```
-
-Requires `@strav/core` as a peer dependency.
-
-## Setup
-
-```ts
-import { SearchProvider } from '@strav/search'
-
-app.use(new SearchProvider())
-```
-
-## Searchable Models
-
-```ts
-import { searchable } from '@strav/search'
-
-class Post extends searchable(BaseModel) {
-  static searchableAs = 'posts'
-
-  toSearchableDocument() {
-    return { id: this.id, title: this.title, body: this.body }
-  }
-}
-```
-
-## Usage
-
-```ts
-import { search } from '@strav/search'
-
-// Search
-const results = await search.query('posts', 'hello world', {
-  filters: 'status = published',
-  limit: 20,
-})
-
-// Manual indexing
-await search.index('posts', [{ id: 1, title: 'Hello' }])
-await search.delete('posts', ['1'])
-```
-
-## Drivers
-
-- **Embedded** — in-process SQLite FTS5, zero deps, recommended for self-host / SMB (~50k–500k docs)
-- **Postgres FTS** — tsvector + GIN + pg_trgm, drop-in upgrade for higher volume (1M–100M docs)
-- **Meilisearch** — fast, typo-tolerant, self-hosted
-- **Typesense** — open-source, instant search
-- **Algolia** — hosted search-as-a-service
-- **Null** — no-op driver for testing
-
-### Embedded driver
-
-Runs entirely inside your app process using `bun:sqlite`'s FTS5 engine — no Meilisearch/Typesense container to run. Each index is a single `.sqlite` file in the configured data directory.
-
-Features:
-
-- BM25 ranking with per-field weights (via `searchableAttributes` ordering)
-- Prefix (`type*`), phrase (`"quick brown fox"`), negation (`-foo`), required (`+foo`)
-- Porter stemmer for English morphology
-- Typo tolerance (Levenshtein-1) on the fly, configurable
-- Highlighted snippets with `<mark>` tags
-- Object-form filters with equality, `in`, and comparison operators
-
-Limitations for v1:
-
-- English stemming only (other languages are tokenised but not stemmed)
-- One writer at a time per index file (SQLite WAL — concurrent reads are fine)
-- Object-form filters only; raw SQL filter strings are rejected
-- Index settings changes require recreating the index
-
-Configuration:
-
-```ts
-// config/search.ts
-embedded: {
-  driver: 'embedded',
-  path: env('SEARCH_PATH', './storage/search'),   // directory of .sqlite files
-  synchronous: 'NORMAL',                            // 'OFF' | 'NORMAL' | 'FULL'
-  typoTolerance: 'auto',                            // 'off' | 'auto' | { minTokenLength, maxDistance }
-}
-```
-
-Select it as the default with `SEARCH_DRIVER=embedded`.
-
-### Postgres FTS driver
-
-Higher-volume tier (1M–100M docs per index) backed by your existing Postgres. Same `SearchEngine` interface as the embedded driver — drop-in swap by changing one config line.
-
-Features:
-
-- BM25-shaped ranking via `ts_rank_cd(fts, q, 1 | 32)` with per-field weights (`A`/`B`/`C`/`D`)
-- `websearch_to_tsquery` Google-style queries plus prefix (`type*`)
-- Multi-language stemming via Postgres text-search configurations (`english`, `french`, ...) — set per index
-- Levenshtein-near typo tolerance via `pg_trgm` + optional `fuzzystrmatch`
-- `<mark>`-highlighted snippets via `ts_headline`, computed only on the top-K to keep latency bounded
-- Object-form filters with `eq`/`neq`/`gt`/`gte`/`lt`/`lte`/`in`/`nin` against generated typed columns
-- One table per index in a dedicated `strav_search` schema (auto-created)
-
-Requirements:
-
-- Postgres ≥ 15
-- `pg_trgm` extension (auto-`CREATE EXTENSION IF NOT EXISTS` on first use; superuser or owner privilege)
-- `fuzzystrmatch` is optional — if present, typo expansion re-ranks trigram candidates with bounded Levenshtein for higher precision
-
-Configuration:
-
-```ts
-postgres: {
-  driver: 'postgres-fts',
-  // Optional: pass a Bun SQL handle. Falls back to @strav/database's Database.raw.
-  // connection: db.sql,
-  schema: env('SEARCH_PG_SCHEMA', 'strav_search'),
-  language: env('SEARCH_PG_LANGUAGE', 'english'),
-  typoTolerance: env('SEARCH_TYPO_TOLERANCE', 'auto'),
-  workMem: env('SEARCH_PG_WORK_MEM', '64MB'),
-  gin: { fastupdate: false },  // better tail latency
-}
-```
-
-Select it with `SEARCH_DRIVER=postgres`.
-
-Limitations for v1:
-
-- Settings change (e.g. add a new searchable attribute) requires `bun strav search:rebuild <model>`. Tier picked by row count: in-place UPDATE under 100k, batched UPDATE up to 10M, dual-table swap deferred to v1.1 with a clear error above 10M.
-- Adding a new `filterableAttribute` on an existing large table currently rewrites the whole heap (`ALTER TABLE ADD COLUMN ... GENERATED ... STORED`). Plan an offline window for big tables in v1.
-- One language per index — mixed-locale indexes deferred.
-- Object-form filters only; raw SQL filter strings rejected.
-
-Ranking note: `ts_rank_cd` is BM25-*shaped* (length normalisation + bounded mapping), not strict BM25. For the size and shape of corpora the driver targets, the difference is small in practice; the embedded driver remains the answer when strict BM25 matters and the corpus fits.
-
-Model example with per-field weights (column order determines BM25 weight — title first = highest):
-
-```ts
-class Ticket extends searchable(BaseModel) {
-  static searchableSettings() {
-    return {
-      searchableAttributes: ['subject', 'body'],
-      filterableAttributes: ['status', 'priority'],
-      sortableAttributes: ['priority', 'created_at'],
-    }
-  }
-}
-```
-
-#### Replacing Postgres `tsvector`
-
-If you've been using raw `tsvector` columns, the embedded driver gives you better ranking, typo tolerance, and highlighted snippets without adding a network service. The migration is roughly:
-
-```ts
-// Before: hand-rolled tsvector query
-const rows = await db.sql`
-  SELECT id, subject, ts_rank_cd(fts, q) AS rank
-  FROM tickets, websearch_to_tsquery('english', ${q}) q
-  WHERE fts @@ q ORDER BY rank DESC LIMIT 20
-`
-
-// After: searchable() + embedded driver
-const results = await Ticket.search(q, {
-  perPage: 20,
-  attributesToHighlight: ['subject', 'body'],
-})
-```
-
-You run `bun strav search:import Ticket` once to populate the index, then model events keep it up to date.
-
-## CLI
-
-```bash
-bun strav search:import <model>      # Import all records for a model
-bun strav search:flush <model>       # Flush all documents from an index
-bun strav search:optimize <model>    # (embedded) Merge FTS5 segments; run periodically
-bun strav search:rebuild <model>     # (postgres) Recompute fts after settings change
-```
-
-## Documentation
-
-See the full [Search guide](../../guides/search.md).
-
-## License
-
-MIT

package/src/commands/search_flush.ts

@@ -1,41 +0,0 @@
-import type { Command } from 'commander'
-import chalk from 'chalk'
-import { bootstrap, shutdown } from '@strav/cli'
-import { BaseModel } from '@strav/database'
-import SearchManager from '../search_manager.ts'
-
-export function register(program: Command): void {
-  program
-    .command('search:flush <model>')
-    .description("Flush all documents from a model's search index")
-    .action(async (modelPath: string) => {
-      let db
-      try {
-        const { db: database, config } = await bootstrap()
-        db = database
-
-        new BaseModel(db)
-        new SearchManager(config)
-
-        const resolved = require.resolve(`${process.cwd()}/${modelPath}`)
-        const module = await import(resolved)
-        const ModelClass = module.default ?? (Object.values(module)[0] as any)
-
-        if (typeof ModelClass?.flushIndex !== 'function') {
-          console.error(chalk.red(`Model "${modelPath}" does not use the searchable() mixin.`))
-          process.exit(1)
-        }
-
-        const indexName = ModelClass.searchableAs()
-        console.log(chalk.dim(`Flushing "${indexName}"...`))
-
-        await ModelClass.flushIndex()
-        console.log(chalk.green(`Flushed all documents from "${indexName}".`))
-      } catch (err) {
-        console.error(chalk.red(`Error: ${err instanceof Error ? err.message : err}`))
-        process.exit(1)
-      } finally {
-        if (db) await shutdown(db)
-      }
-    })
-}

package/src/commands/search_import.ts

@@ -1,43 +0,0 @@
-import type { Command } from 'commander'
-import chalk from 'chalk'
-import { bootstrap, shutdown } from '@strav/cli'
-import { BaseModel } from '@strav/database'
-import SearchManager from '../search_manager.ts'
-
-export function register(program: Command): void {
-  program
-    .command('search:import <model>')
-    .description('Import all records for a model into the search index')
-    .option('--chunk <size>', 'Records per batch', '500')
-    .action(async (modelPath: string, options: { chunk: string }) => {
-      let db
-      try {
-        const { db: database, config } = await bootstrap()
-        db = database
-
-        new BaseModel(db)
-        new SearchManager(config)
-
-        const resolved = require.resolve(`${process.cwd()}/${modelPath}`)
-        const module = await import(resolved)
-        const ModelClass = module.default ?? (Object.values(module)[0] as any)
-
-        if (typeof ModelClass?.importAll !== 'function') {
-          console.error(chalk.red(`Model "${modelPath}" does not use the searchable() mixin.`))
-          process.exit(1)
-        }
-
-        const chunkSize = parseInt(options.chunk, 10)
-        const indexName = ModelClass.searchableAs()
-        console.log(chalk.dim(`Importing ${ModelClass.name} into "${indexName}"...`))
-
-        const count = await ModelClass.importAll(chunkSize)
-        console.log(chalk.green(`Imported ${count} record(s) into "${indexName}".`))
-      } catch (err) {
-        console.error(chalk.red(`Error: ${err instanceof Error ? err.message : err}`))
-        process.exit(1)
-      } finally {
-        if (db) await shutdown(db)
-      }
-    })
-}

package/src/commands/search_optimize.ts

@@ -1,52 +0,0 @@
-import type { Command } from 'commander'
-import chalk from 'chalk'
-import { bootstrap, shutdown } from '@strav/cli'
-import { BaseModel } from '@strav/database'
-import SearchManager from '../search_manager.ts'
-import { EmbeddedDriver } from '../drivers/embedded/index.ts'
-
-export function register(program: Command): void {
-  program
-    .command('search:optimize <model>')
-    .description("Merge FTS5 segments for a model's index (embedded driver only)")
-    .action(async (modelPath: string) => {
-      let db
-      try {
-        const { db: database, config } = await bootstrap()
-        db = database
-
-        new BaseModel(db)
-        new SearchManager(config)
-
-        const resolved = require.resolve(`${process.cwd()}/${modelPath}`)
-        const module = await import(resolved)
-        const ModelClass = module.default ?? (Object.values(module)[0] as any)
-
-        if (typeof ModelClass?.searchableAs !== 'function') {
-          console.error(chalk.red(`Model "${modelPath}" does not use the searchable() mixin.`))
-          process.exit(1)
-        }
-
-        const indexName = SearchManager.indexName(ModelClass.searchableAs())
-        const engine = SearchManager.engine()
-
-        if (!(engine instanceof EmbeddedDriver)) {
-          console.error(
-            chalk.red(
-              `search:optimize is only meaningful for the embedded driver (current: ${engine.name}).`
-            )
-          )
-          process.exit(1)
-        }
-
-        console.log(chalk.dim(`Optimizing "${indexName}"...`))
-        engine.optimize(indexName)
-        console.log(chalk.green(`Optimized "${indexName}".`))
-      } catch (err) {
-        console.error(chalk.red(`Error: ${err instanceof Error ? err.message : err}`))
-        process.exit(1)
-      } finally {
-        if (db) await shutdown(db)
-      }
-    })
-}

package/src/commands/search_rebuild.ts

@@ -1,73 +0,0 @@
-import type { Command } from 'commander'
-import chalk from 'chalk'
-import { bootstrap, shutdown } from '@strav/cli'
-import { BaseModel } from '@strav/database'
-import SearchManager from '../search_manager.ts'
-import { PostgresFtsDriver } from '../drivers/postgres/index.ts'
-
-export function register(program: Command): void {
-  program
-    .command('search:rebuild <model>')
-    .description("Recompute a model's fts column in place (postgres-fts driver only)")
-    .option('--no-reindex', "Skip the GIN REINDEX after the rebuild")
-    .option('--pause <ms>', 'Pause between batches in tier-2 mode (default 50)', '50')
-    .action(async (modelPath: string, options: { reindex: boolean; pause: string }) => {
-      let db
-      try {
-        const { db: database, config } = await bootstrap()
-        db = database
-
-        new BaseModel(db)
-        new SearchManager(config)
-
-        const resolved = require.resolve(`${process.cwd()}/${modelPath}`)
-        const module = await import(resolved)
-        const ModelClass = module.default ?? (Object.values(module)[0] as any)
-
-        if (typeof ModelClass?.searchableAs !== 'function') {
-          console.error(chalk.red(`Model "${modelPath}" does not use the searchable() mixin.`))
-          process.exit(1)
-        }
-
-        const indexName = SearchManager.indexName(ModelClass.searchableAs())
-        const engine = SearchManager.engine()
-
-        if (!(engine instanceof PostgresFtsDriver)) {
-          console.error(
-            chalk.red(
-              `search:rebuild is only meaningful for the postgres-fts driver (current: ${engine.name}).`
-            )
-          )
-          process.exit(1)
-        }
-
-        // Make sure the engine knows about the model's settings (so rebuild
-        // computes fts with the right weights/language).
-        const settings = (ModelClass.searchableSettings?.() ?? undefined) as any
-        if (settings) await engine.createIndex(indexName, settings)
-
-        console.log(chalk.dim(`Rebuilding "${indexName}"...`))
-        const result = await engine.rebuild(indexName, {
-          reindex: options.reindex !== false,
-          pauseMs: Number(options.pause),
-          onProgress: (done, total) => {
-            const pct = total > 0 ? Math.round((done / total) * 100) : 100
-            process.stdout.write(`\r  ${done}/${total} rows (${pct}%) `)
-          },
-        })
-        if (result.tier === 2) process.stdout.write('\n')
-
-        console.log(
-          chalk.green(
-            `Rebuilt ${result.rows} row(s) in "${indexName}" using tier-${result.tier} ` +
-              `strategy (${result.elapsedMs}ms).`
-          )
-        )
-      } catch (err) {
-        console.error(chalk.red(`Error: ${err instanceof Error ? err.message : err}`))
-        process.exit(1)
-      } finally {
-        if (db) await shutdown(db)
-      }
-    })
-}

package/src/drivers/algolia_driver.ts

@@ -1,170 +0,0 @@
-import { ExternalServiceError } from '@strav/kernel'
-import type { SearchEngine } from '../search_engine.ts'
-import type {
-  SearchDocument,
-  SearchOptions,
-  SearchResult,
-  SearchHit,
-  IndexSettings,
-  DriverConfig,
-} from '../types.ts'
-
-/**
- * Algolia driver — communicates with the Algolia REST API via raw `fetch()`.
- *
- * @see https://www.algolia.com/doc/rest-api/search/
- */
-export class AlgoliaDriver implements SearchEngine {
-  readonly name = 'algolia'
-  private appId: string
-  private apiKey: string
-  private baseUrl: string
-
-  constructor(config: DriverConfig) {
-    this.appId = (config.appId as string) ?? ''
-    this.apiKey = (config.apiKey as string) ?? ''
-    this.baseUrl = `https://${this.appId}.algolia.net`
-  }
-
-  // ── Interface ────────────────────────────────────────────────────────────
-
-  async upsert(
-    index: string,
-    id: string | number,
-    document: Record<string, unknown>
-  ): Promise<void> {
-    await this.request(
-      'PUT',
-      `/1/indexes/${encodeURIComponent(index)}/${encodeURIComponent(String(id))}`,
-      document
-    )
-  }
-
-  async upsertMany(index: string, documents: SearchDocument[]): Promise<void> {
-    const requests = documents.map(doc => ({
-      action: 'updateObject',
-      body: { objectID: String(doc.id), ...doc },
-    }))
-    await this.request('POST', `/1/indexes/${encodeURIComponent(index)}/batch`, { requests })
-  }
-
-  async delete(index: string, id: string | number): Promise<void> {
-    await this.request(
-      'DELETE',
-      `/1/indexes/${encodeURIComponent(index)}/${encodeURIComponent(String(id))}`
-    )
-  }
-
-  async deleteMany(index: string, ids: Array<string | number>): Promise<void> {
-    const requests = ids.map(id => ({
-      action: 'deleteObject',
-      body: { objectID: String(id) },
-    }))
-    await this.request('POST', `/1/indexes/${encodeURIComponent(index)}/batch`, { requests })
-  }
-
-  async flush(index: string): Promise<void> {
-    await this.request('POST', `/1/indexes/${encodeURIComponent(index)}/clear`)
-  }
-
-  async deleteIndex(index: string): Promise<void> {
-    await this.request('DELETE', `/1/indexes/${encodeURIComponent(index)}`)
-  }
-
-  async createIndex(index: string, options?: IndexSettings): Promise<void> {
-    // Algolia creates indexes implicitly on first write.
-    // If settings are provided, configure them.
-    if (options) {
-      const settings: Record<string, unknown> = {}
-      if (options.searchableAttributes) settings.searchableAttributes = options.searchableAttributes
-      if (options.displayedAttributes) settings.attributesToRetrieve = options.displayedAttributes
-      if (options.filterableAttributes) {
-        settings.attributesForFaceting = options.filterableAttributes.map(
-          attr => `filterOnly(${attr})`
-        )
-      }
-      if (options.sortableAttributes) settings.ranking = options.sortableAttributes
-
-      if (Object.keys(settings).length > 0) {
-        await this.request('PUT', `/1/indexes/${encodeURIComponent(index)}/settings`, settings)
-      }
-    }
-  }
-
-  async search(index: string, query: string, options?: SearchOptions): Promise<SearchResult> {
-    const perPage = options?.perPage ?? 20
-    const page = options?.page ?? 1
-
-    const body: Record<string, unknown> = {
-      query,
-      hitsPerPage: perPage,
-      page: page - 1, // Algolia uses 0-based pages
-    }
-
-    if (options?.filter) {
-      body.filters =
-        typeof options.filter === 'string' ? options.filter : this.buildFilter(options.filter)
-    }
-    if (options?.attributesToRetrieve) body.attributesToRetrieve = options.attributesToRetrieve
-    if (options?.attributesToHighlight) body.attributesToHighlight = options.attributesToHighlight
-
-    const data = await this.request('POST', `/1/indexes/${encodeURIComponent(index)}/query`, body)
-
-    return {
-      hits: (data.hits ?? []).map(
-        (hit: any): SearchHit => ({
-          document: hit,
-          highlights: hit._highlightResult
-            ? Object.fromEntries(
-                Object.entries(hit._highlightResult).map(([key, val]: [string, any]) => [
-                  key,
-                  val.value ?? '',
-                ])
-              )
-            : undefined,
-        })
-      ),
-      totalHits: data.nbHits ?? 0,
-      page,
-      perPage,
-      processingTimeMs: data.processingTimeMS,
-    }
-  }
-
-  // ── Private ──────────────────────────────────────────────────────────────
-
-  private headers(): Record<string, string> {
-    return {
-      'content-type': 'application/json',
-      'x-algolia-application-id': this.appId,
-      'x-algolia-api-key': this.apiKey,
-    }
-  }
-
-  private async request(method: string, path: string, body?: unknown): Promise<any> {
-    const response = await fetch(`${this.baseUrl}${path}`, {
-      method,
-      headers: this.headers(),
-      body: body !== undefined ? JSON.stringify(body) : undefined,
-    })
-
-    if (!response.ok) {
-      const text = await response.text()
-      throw new ExternalServiceError('Algolia', response.status, text)
-    }
-
-    if (response.status === 204 || response.headers.get('content-length') === '0') return null
-    return response.json()
-  }
-
-  private buildFilter(filter: Record<string, unknown>): string {
-    return Object.entries(filter)
-      .map(([key, value]) => {
-        if (Array.isArray(value)) {
-          return value.map(v => `${key}:${JSON.stringify(v)}`).join(' OR ')
-        }
-        return `${key}:${JSON.stringify(value)}`
-      })
-      .join(' AND ')
-  }
-}