@absolutejs/sync 0.4.0 → 0.5.0

package/README.md

@@ -33,9 +33,9 @@ top-N ordering are maintained incrementally through a composable operator graph
 > write-behind cache), Tier 2 (Drizzle + Prisma topic adapters, `createLiveQuery`),
 > and Tier 3 (sync engine: collections, WebSocket diff transport, optimistic
 > mutations + offline queue, a local-first client cache, declarative row-level
-> permissions, CDC for Postgres/MySQL/SQLite, incremental aggregations + joins,
-> and a declarative operator graph) are in place.
-> Everything ships as subpaths of this one package.
+> permissions, live full-text + vector search, CDC for Postgres/MySQL/SQLite,
+> incremental aggregations + joins, and a declarative operator graph) are in
+> place. Everything ships as subpaths of this one package.
 
 ## Install
 
@@ -249,6 +249,37 @@ await orders.mutate({
     });
     ```
 
+- **Live search.** A `defineSearchCollection` is a full-text or vector index kept
+  live from a table's change feed. The subscription's `params` are the query (a
+  string for keyword search, an embedding for similarity); the ranked top-K stream
+  back as an ordinary collection and re-rank as rows change. Read permissions on
+  the source table still scope a caller's hits. Standalone, `createTextIndex` and
+  `createVectorIndex` are reusable (e.g. RAG retrieval with `@absolutejs/rag`).
+
+    ```ts
+    // server
+    engine.registerSearch(
+    	defineSearchCollection<Doc>({
+    		name: 'docSearch',
+    		table: 'docs',
+    		index: () =>
+    			createTextIndex({
+    				key: (d) => d.id,
+    				fields: ['title', 'body']
+    			}),
+    		source: () => db.select().from(docs), // the corpus to index
+    		key: (d) => d.id
+    	})
+    );
+
+    // client — params are the query; each result row carries `_score`
+    const results = createSyncCollection<Doc>({
+    	url,
+    	collection: 'docSearch',
+    	params: 'quick brown fox' // a vector for createVectorIndex
+    });
+    ```
+
 ## Write-behind cache — keep a remote store off your hot path
 
 ```ts
@@ -328,21 +359,24 @@ mutate({
 
 ### `@absolutejs/sync/engine`
 
-| Export                                                                            | What it is                                                                                                                                                                                                             |
-| --------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `createSyncEngine()`                                                              | Registry + view syncer: `register`, `subscribe`, `applyChange`, `connectSource`, `registerMutation`, `registerWriter`, `runMutation`.                                                                                  |
-| `defineCollection({ name, hydrate, key?, match?, authorize?, tables? })`          | Define a syncable collection.                                                                                                                                                                                          |
-| `defineMutation({ name, handler, authorize? })`                                   | Define a server mutation. Its `handler` gets `actions.insert/update/delete` (write through a registered `TableWriter` → persists + emits in one step) plus `actions.change` (escape hatch). Changes commit atomically. |
-| `registerWriter(table, { insert, update, delete })`                               | Teach the engine how to persist a table (any ORM), so writes auto-emit — you can't write without going live.                                                                                                           |
-| `createAggregate({ key, groupBy?, value? })`                                      | Incremental count/sum/avg/min/max by group.                                                                                                                                                                            |
-| `createMaterializedView({ key, match, equals? })`                                 | The predicate-matching IVM primitive (`apply`/`reset` → diffs).                                                                                                                                                        |
-| `createPollingChangeSource({ poll, intervalMs?, startSeq?, onProcessed? })`       | DB-agnostic CDC `ChangeSource` that tails a changelog (outbox) table.                                                                                                                                                  |
-| `engine.connectCluster(bus)` + `createInMemoryClusterBus()`                       | Horizontal scale: fan changes across server instances over a `ClusterBus` (BYO Redis/Postgres; in-memory bus for dev).                                                                                                 |
-| `createPresenceHub()` + `syncSocket({ engine, presence })`                        | Ephemeral room-scoped presence (online / typing / cursors) over the same socket — not persisted, auto-cleaned on disconnect.                                                                                           |
-| `query(source).filter().map().join().leftJoin().groupBy().orderBy()`              | Declarative incremental query builder (the operator graph).                                                                                                                                                            |
-| `defineGraphCollection({ name, query, key, authorize? })`                         | Run a `query` as a live collection.                                                                                                                                                                                    |
-| `defineReactiveQuery({ name, run, key })` + `registerReactive` / `registerReader` | Read-set-tracked query: `run(ctx)` reads via `ctx.db` (`all`/`get`/`where`) and re-runs only when the rows/ranges it read change — no `match`, no manual emit.                                                         |
-| `definePermissions({ [table]: { read?, insert?, update?, delete?, write? } })`    | Declarative row-level access control. Pass as `createSyncEngine({ permissions })` or `registerPermissions(table, rules)`. Read rules filter every row emitted; write rules gate `actions.insert/update/delete`.        |
+| Export                                                                                   | What it is                                                                                                                                                                                                             |
+| ---------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `createSyncEngine()`                                                                     | Registry + view syncer: `register`, `subscribe`, `applyChange`, `connectSource`, `registerMutation`, `registerWriter`, `runMutation`.                                                                                  |
+| `defineCollection({ name, hydrate, key?, match?, authorize?, tables? })`                 | Define a syncable collection.                                                                                                                                                                                          |
+| `defineMutation({ name, handler, authorize? })`                                          | Define a server mutation. Its `handler` gets `actions.insert/update/delete` (write through a registered `TableWriter` → persists + emits in one step) plus `actions.change` (escape hatch). Changes commit atomically. |
+| `registerWriter(table, { insert, update, delete })`                                      | Teach the engine how to persist a table (any ORM), so writes auto-emit — you can't write without going live.                                                                                                           |
+| `createAggregate({ key, groupBy?, value? })`                                             | Incremental count/sum/avg/min/max by group.                                                                                                                                                                            |
+| `createMaterializedView({ key, match, equals? })`                                        | The predicate-matching IVM primitive (`apply`/`reset` → diffs).                                                                                                                                                        |
+| `createPollingChangeSource({ poll, intervalMs?, startSeq?, onProcessed? })`              | DB-agnostic CDC `ChangeSource` that tails a changelog (outbox) table.                                                                                                                                                  |
+| `engine.connectCluster(bus)` + `createInMemoryClusterBus()`                              | Horizontal scale: fan changes across server instances over a `ClusterBus` (BYO Redis/Postgres; in-memory bus for dev).                                                                                                 |
+| `createPresenceHub()` + `syncSocket({ engine, presence })`                               | Ephemeral room-scoped presence (online / typing / cursors) over the same socket — not persisted, auto-cleaned on disconnect.                                                                                           |
+| `query(source).filter().map().join().leftJoin().groupBy().orderBy()`                     | Declarative incremental query builder (the operator graph).                                                                                                                                                            |
+| `defineGraphCollection({ name, query, key, authorize? })`                                | Run a `query` as a live collection.                                                                                                                                                                                    |
+| `defineReactiveQuery({ name, run, key })` + `registerReactive` / `registerReader`        | Read-set-tracked query: `run(ctx)` reads via `ctx.db` (`all`/`get`/`where`) and re-runs only when the rows/ranges it read change — no `match`, no manual emit.                                                         |
+| `definePermissions({ [table]: { read?, insert?, update?, delete?, write? } })`           | Declarative row-level access control. Pass as `createSyncEngine({ permissions })` or `registerPermissions(table, rules)`. Read rules filter every row emitted; write rules gate `actions.insert/update/delete`.        |
+| `defineSearchCollection({ name, table, index, source, key, limit? })` + `registerSearch` | Live search collection: the subscription's `params` are the query (string/vector), the ranked top-K stream back as a normal collection, re-ranked as rows change. Each row carries its score under `_score`.           |
+| `createTextIndex({ key, fields, tokenize?, stopwords?, k1?, b? })`                       | Incremental BM25 full-text index (keyword search). Implements `SearchIndex`; usable standalone or inside a search collection.                                                                                          |
+| `createVectorIndex({ key, embedding, metric? })`                                         | Incremental vector index (cosine/dot/euclidean exact k-NN) for semantic search — pairs with `@absolutejs/ai` / `@absolutejs/rag` for RAG retrieval on your own data.                                                   |
 
 ### `@absolutejs/sync/postgres`
 

package/dist/engine/index.d.ts

@@ -32,6 +32,12 @@ export { defineGraphCollection, query } from './graph';
 export type { GraphCollectionDefinition, GraphInstance, GraphSource, GroupByOptions, JoinOptions, OrderByQueryOptions, Query } from './graph';
 export { definePermissions } from './permissions';
 export type { PermissionsDefinition, ReadRule, TablePermissions, WriteRule } from './permissions';
+export { defineSearchCollection, SEARCH_SCORE_FIELD } from './search';
+export type { SearchCollectionDefinition, SearchHit, SearchIndex } from './search';
+export { createTextIndex } from './textIndex';
+export type { TextIndexOptions } from './textIndex';
+export { createVectorIndex } from './vectorIndex';
+export type { VectorIndexOptions, VectorMetric } from './vectorIndex';
 export { defineMutation } from './mutation';
 export type { MutationActions, MutationDefinition, MutationHandler, TableWriter, TransactionRunner } from './mutation';
 export { createSyncEngine, UnauthorizedError } from './syncEngine';

package/dist/engine/index.js

@@ -885,6 +885,156 @@ var query = (source) => makeQuery(source, []);
 var defineGraphCollection = (definition) => ({ ...definition, kind: "graph" });
 // src/engine/permissions.ts
 var definePermissions = (permissions) => permissions;
+// src/engine/search.ts
+var SEARCH_SCORE_FIELD = "_score";
+var defineSearchCollection = (definition) => ({
+  ...definition,
+  kind: "search"
+});
+// src/engine/textIndex.ts
+var defaultTokenize = (text) => text.toLowerCase().match(/[a-z0-9]+/g) ?? [];
+var createTextIndex = (options) => {
+  const { key, fields } = options;
+  const tokenize = options.tokenize ?? defaultTokenize;
+  const stopwords = new Set(options.stopwords ?? []);
+  const k1 = options.k1 ?? 1.5;
+  const b = options.b ?? 0.75;
+  const docs = new Map;
+  const postings = new Map;
+  let totalLen = 0;
+  const termsOf = (row) => {
+    const text = fields.map((field) => {
+      const value = row[field];
+      return value === undefined || value === null ? "" : String(value);
+    }).join(" ");
+    return tokenize(text).filter((term) => !stopwords.has(term));
+  };
+  const remove = (rowKey) => {
+    const doc = docs.get(rowKey);
+    if (doc === undefined) {
+      return;
+    }
+    for (const term of doc.tf.keys()) {
+      const set = postings.get(term);
+      if (set !== undefined) {
+        set.delete(rowKey);
+        if (set.size === 0) {
+          postings.delete(term);
+        }
+      }
+    }
+    totalLen -= doc.len;
+    docs.delete(rowKey);
+  };
+  const add = (row) => {
+    const rowKey = key(row);
+    remove(rowKey);
+    const terms = termsOf(row);
+    const tf = new Map;
+    for (const term of terms) {
+      tf.set(term, (tf.get(term) ?? 0) + 1);
+    }
+    for (const term of tf.keys()) {
+      let set = postings.get(term);
+      if (set === undefined) {
+        set = new Set;
+        postings.set(term, set);
+      }
+      set.add(rowKey);
+    }
+    docs.set(rowKey, { row, len: terms.length, tf });
+    totalLen += terms.length;
+  };
+  const search = (query2, limit) => {
+    const total = docs.size;
+    if (total === 0) {
+      return [];
+    }
+    const avgdl = totalLen / total;
+    const queryTerms = new Set(tokenize(query2).filter((term) => !stopwords.has(term)));
+    const scores = new Map;
+    for (const term of queryTerms) {
+      const set = postings.get(term);
+      if (set === undefined) {
+        continue;
+      }
+      const df = set.size;
+      const idf = Math.log(1 + (total - df + 0.5) / (df + 0.5));
+      for (const rowKey of set) {
+        const doc = docs.get(rowKey);
+        const freq = doc.tf.get(term) ?? 0;
+        const norm = freq * (k1 + 1) / (freq + k1 * (1 - b + b * doc.len / avgdl));
+        scores.set(rowKey, (scores.get(rowKey) ?? 0) + idf * norm);
+      }
+    }
+    return [...scores.entries()].map(([rowKey, score]) => ({ row: docs.get(rowKey).row, score })).sort((first, second) => second.score - first.score).slice(0, limit);
+  };
+  return {
+    add,
+    remove,
+    search,
+    size: () => docs.size,
+    clear: () => {
+      docs.clear();
+      postings.clear();
+      totalLen = 0;
+    }
+  };
+};
+// src/engine/vectorIndex.ts
+var dot = (first, second) => {
+  const length = Math.min(first.length, second.length);
+  let sum = 0;
+  for (let index = 0;index < length; index += 1) {
+    sum += first[index] * second[index];
+  }
+  return sum;
+};
+var normOf = (vec) => Math.sqrt(dot(vec, vec));
+var euclidean = (first, second) => {
+  const length = Math.max(first.length, second.length);
+  let sum = 0;
+  for (let index = 0;index < length; index += 1) {
+    const delta = (first[index] ?? 0) - (second[index] ?? 0);
+    sum += delta * delta;
+  }
+  return Math.sqrt(sum);
+};
+var createVectorIndex = (options) => {
+  const { key, embedding } = options;
+  const metric = options.metric ?? "cosine";
+  const entries = new Map;
+  const score = (query2, queryNorm, entry) => {
+    if (metric === "dot") {
+      return dot(query2, entry.vec);
+    }
+    if (metric === "euclidean") {
+      return -euclidean(query2, entry.vec);
+    }
+    const denominator = queryNorm * entry.norm;
+    return denominator === 0 ? 0 : dot(query2, entry.vec) / denominator;
+  };
+  return {
+    add: (row) => {
+      const vec = embedding(row);
+      entries.set(key(row), { row, vec, norm: normOf(vec) });
+    },
+    remove: (rowKey) => {
+      entries.delete(rowKey);
+    },
+    search: (query2, limit) => {
+      const queryNorm = normOf(query2);
+      return [...entries.values()].map((entry) => ({
+        row: entry.row,
+        score: score(query2, queryNorm, entry)
+      })).sort((first, second) => second.score - first.score).slice(0, limit);
+    },
+    size: () => entries.size,
+    clear: () => {
+      entries.clear();
+    }
+  };
+};
 // src/engine/mutation.ts
 var defineMutation = (definition) => definition;
 // src/engine/syncEngine.ts
@@ -906,6 +1056,15 @@ var shallowEqual4 = (a, b) => {
   const bKeys = Object.keys(b);
   return aKeys.length === bKeys.length && aKeys.every((k) => a[k] === b[k]);
 };
+var equalsIgnoringScore = (a, b) => {
+  if (typeof a !== "object" || typeof b !== "object" || a === null || b === null) {
+    return a === b;
+  }
+  const strip = (value) => Object.keys(value).filter((k) => k !== SEARCH_SCORE_FIELD);
+  const aKeys = strip(a);
+  const bKeys = strip(b);
+  return aKeys.length === bKeys.length && aKeys.every((k) => a[k] === b[k]);
+};
 var createSyncEngine = (options = {}) => {
   const registry = new Map;
   const mutations = new Map;
@@ -921,6 +1080,8 @@ var createSyncEngine = (options = {}) => {
     return rules?.[op] ?? rules?.write;
   };
   const reactiveSubs = new Set;
+  const searchSubs = new Set;
+  const searchIndexes = new Map;
   const active = new Map;
   const tableIndex = new Map;
   const changeLogSize = options.changeLogSize ?? 1024;
@@ -973,6 +1134,9 @@ var createSyncEngine = (options = {}) => {
     if (subscription.kind === "reactive") {
       return EMPTY_DIFF;
     }
+    if (subscription.kind === "search") {
+      return EMPTY_DIFF;
+    }
     if (subscription.incremental) {
       try {
         return subscription.view.apply(change);
@@ -1085,7 +1249,7 @@ var createSyncEngine = (options = {}) => {
       }
     };
   };
-  const diffRerun = (sub, rows) => {
+  const diffRerun = (sub, rows, equals = shallowEqual4) => {
     const next = new Map;
     for (const row of rows) {
       next.set(sub.key(row), row);
@@ -1097,7 +1261,7 @@ var createSyncEngine = (options = {}) => {
       const previous = sub.current.get(rowKey);
       if (previous === undefined) {
         added.push(row);
-      } else if (!shallowEqual4(previous, row)) {
+      } else if (!equals(previous, row)) {
         changed.push(row);
       }
     }
@@ -1128,6 +1292,47 @@ var createSyncEngine = (options = {}) => {
     }
     return pairs;
   };
+  const ensureSearchIndex = async (definition) => {
+    let entry = searchIndexes.get(definition.name);
+    if (entry === undefined) {
+      entry = { index: definition.index(), definition, hydrated: false };
+      searchIndexes.set(definition.name, entry);
+    }
+    if (!entry.hydrated) {
+      for (const row of await definition.source()) {
+        entry.index.add(row);
+      }
+      entry.hydrated = true;
+    }
+    return entry;
+  };
+  const searchPairs = (changes) => {
+    const touched = new Set;
+    for (const { table, change } of changes) {
+      for (const entry of searchIndexes.values()) {
+        if (!entry.hydrated || entry.definition.table !== table) {
+          continue;
+        }
+        if (change.op === "delete") {
+          entry.index.remove(entry.definition.key(change.row));
+        } else {
+          entry.index.add(change.row);
+        }
+        touched.add(entry.definition.name);
+      }
+    }
+    const pairs = [];
+    for (const sub of searchSubs) {
+      if (!touched.has(sub.collection)) {
+        continue;
+      }
+      const diff = diffRerun(sub, sub.rerun(), equalsIgnoringScore);
+      if (!isEmptyViewDiff(diff)) {
+        pairs.push([sub, diff]);
+      }
+    }
+    return pairs;
+  };
   const logChange = (changeVersion, entry) => {
     changeLog.push(entry);
     if (changeLog.length > changeLogSize) {
@@ -1148,6 +1353,7 @@ var createSyncEngine = (options = {}) => {
     emissions.push(...await reactivePairs([
       { table, key: changedKeyFor(table, change), row: change.row }
     ]));
+    emissions.push(...searchPairs([{ table, change }]));
     for (const [subscription, diff] of emissions) {
       subscription.onDiff(diff, changeVersion);
     }
@@ -1188,6 +1394,7 @@ var createSyncEngine = (options = {}) => {
       }
     }
     emissions.push(...await reactivePairs(reactiveChanges));
+    emissions.push(...searchPairs(changes));
     for (const [subscription, diff] of emissions) {
       subscription.onDiff(diff, batchVersion);
     }
@@ -1330,6 +1537,50 @@ var createSyncEngine = (options = {}) => {
       }
     };
   };
+  const subscribeSearch = async (collection, definition, params, ctx, onDiff, set) => {
+    const query2 = params;
+    if (definition.authorize !== undefined) {
+      const allowed = await definition.authorize(query2, ctx);
+      if (!allowed) {
+        throw new UnauthorizedError(`subscribe to collection "${collection}"`);
+      }
+    }
+    const entry = await ensureSearchIndex(definition);
+    const limit = definition.limit ?? 20;
+    const readRule = readRuleFor(definition.table);
+    const rerun = () => {
+      const candidates = entry.index.search(query2, readRule ? limit * 5 : limit);
+      const visible = readRule ? candidates.filter((hit) => readRule(ctx, hit.row)) : candidates;
+      return visible.slice(0, limit).map((hit) => ({
+        ...hit.row,
+        [SEARCH_SCORE_FIELD]: hit.score
+      }));
+    };
+    const initial = rerun();
+    const current = new Map;
+    for (const row of initial) {
+      current.set(definition.key(row), row);
+    }
+    const atVersion = version;
+    const subscription = {
+      kind: "search",
+      collection,
+      key: definition.key,
+      rerun,
+      current,
+      onDiff
+    };
+    set.add(subscription);
+    searchSubs.add(subscription);
+    return {
+      initial,
+      version: atVersion,
+      unsubscribe: () => {
+        set.delete(subscription);
+        searchSubs.delete(subscription);
+      }
+    };
+  };
   return {
     register: (collection) => {
       registry.set(collection.name, collection);
@@ -1348,6 +1599,9 @@ var createSyncEngine = (options = {}) => {
         addTableIndex(table, collection.name);
       }
     },
+    registerSearch: (collection) => {
+      registry.set(collection.name, collection);
+    },
     subscribe: async ({ collection, params, ctx, onDiff, since }) => {
       const registered = registry.get(collection);
       if (registered === undefined) {
@@ -1368,6 +1622,10 @@ var createSyncEngine = (options = {}) => {
         const reactived = await subscribeReactive(collection, registered, params, ctx, typedOnDiff, subscribeSet);
         return reactived;
       }
+      if (registeredKind === "search") {
+        const searched = await subscribeSearch(collection, registered, params, ctx, typedOnDiff, subscribeSet);
+        return searched;
+      }
       const definition = registered;
       if (definition.authorize !== undefined) {
         const allowed = await definition.authorize(params, ctx);
@@ -1801,12 +2059,15 @@ export {
   hydrateRoute,
   fromRowChange,
   filterOp,
+  defineSearchCollection,
   defineReactiveQuery,
   definePermissions,
   defineMutation,
   defineJoinCollection,
   defineGraphCollection,
   defineCollection,
+  createVectorIndex,
+  createTextIndex,
   createSyncEngine,
   createSyncConnection,
   createPresenceHub,
@@ -1817,8 +2078,9 @@ export {
   createAggregate,
   chain,
   aggregateOp,
-  UnauthorizedError
+  UnauthorizedError,
+  SEARCH_SCORE_FIELD
 };
 
-//# debugId=04C93A0AB37B69AD64756E2164756E21
+//# debugId=28E08370F43FEBAE64756E2164756E21
 //# sourceMappingURL=index.js.map