@duckedup/nidus 0.1.1 → 0.53.0

package/README.md

@@ -12,6 +12,10 @@ browser.
 npm install @duckedup/nidus
 ```
 
+This package is versioned in lockstep with nidus itself: a given `@duckedup/nidus`
+version is the client for the identically-numbered nidus release. Match the two and the
+wire contract lines up.
+
 ## Connecting
 
 "Local vs remote" is just the base URL — point the client at a local `nidus serve`
@@ -32,9 +36,9 @@ const db = new NidusClient({
 
 ## Upserting and searching
 
-`attrs` accept plain JS values — strings, integers, booleans, string arrays, and
-`null` — and are normalized to nidus's typed values for you. (For an explicit type,
-use the `v.*` helpers.)
+`attrs` accept plain JS values — strings, numbers, booleans, string arrays, `Date`s,
+and `null` — and are normalized to nidus's typed values for you. (For an explicit
+type, use the `v.*` helpers.)
 
 ```ts
 await db.createCollection("docs");
@@ -52,10 +56,40 @@ for (const hit of hits) {
 }
 ```
 
+nidus has separate `Int` and `Float` attribute types and compares them same-type only,
+but JS has one `number` and `1.0 === 1` — so a plain number becomes an `Int` when
+`Number.isInteger` says so and a `Float` otherwise. That means a whole-numbered
+measurement lands as an `Int` in whichever records it came out round, and a `Float`
+range filter then skips exactly those. Pin such a field with `v.float`:
+
+```ts
+import { v } from "@duckedup/nidus";
+
+await db.upsert("docs", [
+  {
+    id: "d",
+    attrs: {
+      score: v.float(1), // a Float even though the value is whole
+      ratio: 0.75, // already a Float — not an integer
+      year: 2024, // an Int
+      seen: new Date(), // a DateTime: a UTC instant, epoch milliseconds
+    },
+  },
+]);
+```
+
+A `DateTime` carries no timezone and has millisecond resolution; it decodes back to a
+`Date`, so a decoded `attrs` map re-encodes to what it came from. `NaN` and `Infinity`
+throw — JSON has no spelling for them. The Go and Python SDKs have the numeric types JS
+lacks and decide from those instead, so a Python `2.0` or a Go `float64(2)` is a
+`Float` where a bare `2` here is an `Int`.
+
 ## Filtering
 
 Build an AND-filter with the `f.*` helpers. Each predicate is a positive assertion
-about a present attribute (an absent key matches nothing).
+about a present attribute (an absent key matches nothing). Comparisons are same-type
+only, so an operand must encode to the attribute's type — `f.ge("score", v.float(2))`,
+not `f.ge("score", 2)`, for a `Float` attribute.
 
 ```ts
 import { f } from "@duckedup/nidus";
@@ -72,10 +106,25 @@ const hits = await db.search({
 });
 ```
 
+Beyond the comparisons there are text predicates — approximate, token-wise, and
+regular-expression matching over a plain attribute (no full-text index required):
+
+```ts
+f.fuzzy("title", "vecter store", 2);      // within 2 Levenshtein edits (max 8)
+f.containsAllTokens("body", "vector store"); // both tokens, any order
+f.containsAnyToken("body", "vector store");
+f.containsTokenSequence("body", "vector store"); // as a phrase, in order
+f.regex("path", "src/.*\\.rs");           // anchored at both ends, like f.glob
+```
+
+`f.regex` uses Rust's `regex` syntax, not JavaScript's: no backreferences and no
+lookaround. Prefix it with `(?i)` for case-insensitive matching.
+
 ## Full-text and hybrid search
 
 ```ts
 await db.setFtsSchema("docs", ["body"]);
+// Per-field tuning: await db.setFtsSchema("docs", [{ field: "body", k1: 1.5 }]);
 
 // BM25 text search
 const text = await db.textSearch({ field: "body", query: "vector store", topK: 10 });
@@ -89,6 +138,106 @@ const hybrid = await db.hybridSearch({
 });
 ```
 
+A query can search several fields at once, each with its own text, by sending `clauses`
+instead of the single field — folded by `combine`, `"Sum"` (a doc hitting title *and*
+body outranks one hitting either) or `"Max"` (a long body cannot out-accumulate a
+precise title match). Weight the two hybrid legs with `vectorWeight`/`textWeight`.
+
+```ts
+const hits = await db.textSearch({
+  clauses: [
+    { field: "title", query: "rust" },
+    { field: "body", query: "async runtime" },
+  ],
+  combine: "Max",
+  topK: 10,
+});
+```
+
+## Explaining and highlighting a hit
+
+`explain` reports what each leg and each matched clause contributed; `highlight` returns
+excerpts of the stored text (so it works even on a field the projection dropped). Both
+land on `hit.annotations`, which is absent unless you asked for one of them.
+
+```ts
+const hits = await db.hybridSearch({
+  vector: [0.1, 0.2, 0.3],
+  field: "body",
+  text: "vector store",
+  explain: true,
+  highlight: true, // or { maxFragments: 3, fragmentChars: 120 }
+});
+
+for (const { field, fragments } of hits[0]?.annotations?.highlights ?? []) {
+  for (const fragment of fragments) {
+    for (const span of fragment.spans) {
+      console.log(field, fragment.text.slice(...span));
+    }
+  }
+}
+```
+
+nidus reports a span as a **UTF-8 byte** range, but a JS string is indexed in UTF-16
+code units — so a raw span slices the wrong text out of any non-ASCII excerpt. This SDK
+converts them for you: `fragment.spans` are JS string indices, and `fragment.text.slice`
+is the matched term. If you compare them against the raw HTTP response, expect the
+numbers to differ wherever the excerpt is not ASCII.
+
+## Ranking, grouping, ordering, and aggregating
+
+```ts
+// Prefer recent hits: subtract a penalty that halves every `scale` ms of age.
+const recent = await db.search({
+  query: [0.1, 0.2, 0.3],
+  rankBy: {
+    decay: { field: "updated_at", origin: Date.now(), scale: 7 * 86_400_000 },
+  },
+  // …and keep at most 2 hits from any one file
+  limitPer: { field: "path", max: 2 },
+});
+
+// Sort a listing by an attribute instead of storage order
+await db.list({ orderBy: { field: "updated_at", descending: true } });
+
+// Count matches and sum attributes, without reading a single vector
+const { count, sums } = await db.aggregate({
+  filter: f.and(f.eq("lang", "rust")),
+  sum: ["bytes"],
+});
+```
+
+Ages are measured back from `origin`, never the wall clock, so the same query against an
+unchanged store ranks the same way twice. The penalty is *subtracted* from the score, so
+it stays meaningful for a metric whose scores are negative or unbounded, and a record
+with no usable timestamp is not penalized at all (`missing` defaults to `1`).
+
+## Remembering and recalling (text-native)
+
+When the server is started with an embedder (`nidus serve --embed-provider …`), you
+can send **text** and let the server embed it — no need to compute vectors client-side.
+`remember` embeds and upserts; `recall` embeds the query and vector-searches.
+
+```ts
+// Embed "the quick brown fox" and store it under id "a"
+await db.remember("notes", "a", "the quick brown fox", { attrs: { tag: "x" } });
+
+// Summarize first, then embed the summary (server also needs --summarize-provider).
+// The stored record additionally carries `nidus.summary` and `nidus.source` attrs.
+await db.remember("notes", "b", longArticle, { mode: "summarize" });
+
+// Embed the query text and search, best-first (attrs decoded to plain JS values)
+const hits = await db.recall("notes", "quick fox", {
+  topK: 5,
+  minScore: 0.2,
+  filter: f.and(f.eq("tag", "x")),
+});
+```
+
+Both throw a `NidusError` with status `400` if the server has no embedder configured
+(the message names `--embed-provider`); `mode: "summarize"` without a summarizer is
+likewise a `400`.
+
 ## Everything else
 
 ```ts

package/dist/index.cjs

@@ -31,6 +31,49 @@ __export(index_exports, {
 });
 module.exports = __toCommonJS(index_exports);
 
+// src/annotations.ts
+function decodeAnnotations(a) {
+  const out = {};
+  if (a.vector) out.vector = a.vector;
+  if (a.text) out.text = a.text;
+  if (a.clauses) out.clauses = a.clauses;
+  if (a.highlights) out.highlights = a.highlights.map(decodeHighlight);
+  return out;
+}
+function decodeHighlight(h) {
+  return { field: h.field, fragments: h.fragments.map(decodeFragment) };
+}
+function decodeFragment(fr) {
+  return { text: fr.text, spans: toStringIndices(fr.text, fr.spans) };
+}
+function toStringIndices(text, spans) {
+  if (spans.length === 0 || isAscii(text)) return spans;
+  const index = byteToUnit(text);
+  const at = (b) => index[Math.min(Math.max(b, 0), index.length - 1)];
+  return spans.map(([start, end]) => [at(start), at(end)]);
+}
+function byteToUnit(text) {
+  const index = [];
+  let unit = 0;
+  for (const ch of text) {
+    for (let n = utf8Len(ch.codePointAt(0)); n > 0; n--) index.push(unit);
+    unit += ch.length;
+  }
+  index.push(unit);
+  return index;
+}
+function utf8Len(codePoint) {
+  if (codePoint < 128) return 1;
+  if (codePoint < 2048) return 2;
+  return codePoint < 65536 ? 3 : 4;
+}
+function isAscii(text) {
+  for (let i = 0; i < text.length; i++) {
+    if (text.charCodeAt(i) > 127) return false;
+  }
+  return true;
+}
+
 // src/errors.ts
 var NidusError = class extends Error {
   /** The HTTP status code, or `0` for a transport/timeout failure (no response). */
@@ -59,6 +102,7 @@ var NidusError = class extends Error {
 };
 
 // src/values.ts
+var TAGS = ["Str", "Int", "Bool", "List", "Float", "DateTime"];
 var v = {
   str: (s) => ({ Str: s }),
   int: (n) => {
@@ -67,15 +111,34 @@ var v = {
     }
     return { Int: n };
   },
+  float: (n) => {
+    if (typeof n !== "number" || !Number.isFinite(n)) {
+      throw new TypeError(`v.float expects a finite number, got ${n}`);
+    }
+    return { Float: n };
+  },
   bool: (b) => ({ Bool: b }),
   list: (items) => ({ List: items }),
+  /**
+   * A UTC instant, from a `Date` or a raw epoch-millisecond count. Milliseconds is
+   * the wire type, so there is no sub-millisecond precision and no timezone.
+   */
+  datetime: (when) => {
+    const ms = when instanceof Date ? when.getTime() : when;
+    if (!Number.isSafeInteger(ms)) {
+      throw new TypeError(
+        `v.datetime expects a valid Date or epoch ms, got ${when}`
+      );
+    }
+    return { DateTime: ms };
+  },
   /** The explicit `Null` value — set-but-empty, distinct from an absent key. */
   nil: () => "Null"
 };
 function isValue(x) {
   if (x === "Null") return true;
   if (typeof x !== "object" || x === null) return false;
-  return "Str" in x || "Int" in x || "Bool" in x || "List" in x;
+  return TAGS.some((tag) => tag in x);
 }
 function encodeValue(input) {
   if (isValue(input)) return input;
@@ -86,8 +149,9 @@ function encodeValue(input) {
     case "boolean":
       return { Bool: input };
     case "number":
-      return v.int(input);
+      return Number.isInteger(input) ? v.int(input) : v.float(input);
     case "object":
+      if (input instanceof Date) return v.datetime(input);
       if (Array.isArray(input)) {
         if (!input.every((e) => typeof e === "string")) {
           throw new TypeError("a List attribute must contain only strings");
@@ -112,6 +176,8 @@ function decodeValue(value) {
   if ("Int" in value) return value.Int;
   if ("Bool" in value) return value.Bool;
   if ("List" in value) return value.List;
+  if ("Float" in value) return value.Float;
+  if ("DateTime" in value) return new Date(value.DateTime);
   return value;
 }
 function decodeAttrs(attrs) {
@@ -231,10 +297,14 @@ var NidusClient = class {
       attrs: decodeAttrs(r.attrs)
     }));
   }
-  /** Declare the full-text-indexed attribute fields for a collection. */
+  /**
+   * Declare the full-text-indexed attribute fields for a collection. A bare string
+   * takes the server's BM25/analyzer defaults; an {@link FtsField} object tunes `k1`,
+   * `b`, and the analyzer for that field alone.
+   */
   async setFtsSchema(name, fields) {
     await this.request("POST", `/collections/${enc(name)}/fts-schema`, {
-      fields
+      fields: fields.map(encodeFtsField)
     });
   }
   // ── Search ──────────────────────────────────────────────────────────────
@@ -244,32 +314,54 @@ var NidusClient = class {
       query: opts.query,
       scope: opts.scope ?? [],
       top_k: opts.topK,
+      offset: opts.offset,
       min_score: opts.minScore,
-      filter: opts.filter ?? []
+      filter: opts.filter ?? [],
+      exact: opts.exact,
+      include_attributes: opts.includeAttributes,
+      exclude_attributes: opts.excludeAttributes,
+      rank_by: encodeRankBy(opts.rankBy),
+      limit_per: opts.limitPer
     });
   }
-  /** BM25 full-text search over one indexed field. */
+  /**
+   * BM25 full-text search over one indexed field, or over a `clauses` list folded by
+   * `combine` (`"Sum"` unless said otherwise). Naming the fields both ways is a `400`.
+   */
   textSearch(opts) {
     return this.searchRequest("/text-search", {
-      field: opts.field,
-      query: opts.query,
+      ...opts.clauses ? { clauses: opts.clauses, combine: opts.combine } : { field: opts.field, query: opts.query },
       scope: opts.scope ?? [],
       top_k: opts.topK,
+      offset: opts.offset,
       min_score: opts.minScore,
-      filter: opts.filter ?? []
+      filter: opts.filter ?? [],
+      explain: opts.explain,
+      highlight: encodeHighlight(opts.highlight),
+      include_attributes: opts.includeAttributes,
+      exclude_attributes: opts.excludeAttributes,
+      rank_by: encodeRankBy(opts.rankBy),
+      limit_per: opts.limitPer
     });
   }
-  /** Hybrid search: fuse a vector query and a BM25 text query via RRF. */
+  /**
+   * Hybrid search: fuse a vector query and a BM25 text query via RRF. The text leg takes
+   * the same single-field / `clauses` choice as {@link NidusClient.textSearch}.
+   */
   hybridSearch(opts) {
     return this.searchRequest("/hybrid-search", {
       vector: opts.vector,
-      field: opts.field,
-      text: opts.text,
+      ...opts.clauses ? { clauses: opts.clauses, combine: opts.combine } : { field: opts.field, text: opts.text },
       scope: opts.scope ?? [],
       top_k: opts.topK,
+      offset: opts.offset,
       filter: opts.filter ?? [],
       rrf_k: opts.rrfK,
-      candidates: opts.candidates
+      candidates: opts.candidates,
+      explain: opts.explain,
+      highlight: encodeHighlight(opts.highlight),
+      vector_weight: opts.vectorWeight,
+      text_weight: opts.textWeight
     });
   }
   /** Metadata-only listing (no vector), paginated by `offset`/`limit`. */
@@ -278,6 +370,113 @@ var NidusClient = class {
       scope: opts.scope ?? [],
       offset: opts.offset,
       limit: opts.limit,
+      filter: opts.filter ?? [],
+      include_attributes: opts.includeAttributes,
+      exclude_attributes: opts.excludeAttributes,
+      order_by: opts.orderBy
+    });
+  }
+  /**
+   * Count the records matching a filter and sum the named attributes. Answered from the
+   * in-RAM index alone — no record is built and no vector is read.
+   */
+  async aggregate(opts = {}) {
+    const res = await this.request(
+      "POST",
+      "/aggregate",
+      prune({
+        scope: opts.scope ?? [],
+        filter: opts.filter ?? [],
+        sum: opts.sum ?? [],
+        group_by: opts.groupBy
+      })
+    );
+    return {
+      count: res.count,
+      sums: decodeAttrs(res.sums),
+      // Kept absent, not `undefined`, so an ungrouped answer is the shape it always was.
+      ...res.groups ? {
+        groups: res.groups.map((g) => ({
+          value: g.value === null ? null : decodeValue(g.value),
+          count: g.count,
+          sums: decodeAttrs(g.sums)
+        }))
+      } : {},
+      ...res.groups_truncated ? { groupsTruncated: true } : {}
+    };
+  }
+  /**
+   * Answer several vector queries in one round-trip (16 max). Returns one ranking per
+   * query in request order, or — with `opts.fuse` — a single array holding the one fused
+   * ranking, so the return shape is uniform either way.
+   *
+   * The server validates the whole batch before running any leg, so a malformed query
+   * fails the call rather than returning a partial answer that cannot be told apart.
+   */
+  async batchSearch(opts) {
+    const body = prune({
+      queries: opts.queries.map((q) => ({
+        query: q.query,
+        scope: q.scope ?? [],
+        top_k: q.topK,
+        offset: q.offset,
+        min_score: q.minScore,
+        filter: q.filter ?? [],
+        exact: q.exact,
+        include_attributes: q.includeAttributes,
+        exclude_attributes: q.excludeAttributes,
+        rank_by: encodeRankBy(q.rankBy),
+        limit_per: q.limitPer
+      })),
+      fuse: opts.fuse ? prune({
+        rrf_k: opts.fuse.rrfK,
+        weights: opts.fuse.weights,
+        top_k: opts.fuse.topK
+      }) : void 0
+    });
+    const res = await this.request(
+      "POST",
+      "/search/batch",
+      body
+    );
+    return (res.fused ? [res.fused] : res.results ?? []).map(
+      (hits) => hits.map((h) => this.decodeHit(h))
+    );
+  }
+  // ── Memory (text-native) ──────────────────────────────────────────────────
+  //
+  // Available only when `nidus serve` was started with an embedder
+  // (`--embed-provider …`); otherwise these answer `400`. The server embeds the
+  // text/query — the client only sends strings.
+  /**
+   * Embed `text` and upsert it under `id` in `collection` (idempotent on `id`).
+   * With `opts.mode === "summarize"` the server summarizes first, embeds the
+   * summary, and stamps `nidus.summary`/`nidus.source` attrs (requires the
+   * server to have a summarizer). `opts.attrs` accept plain JS values or `v.*`
+   * helpers; they are normalized for you.
+   */
+  async remember(collection, id, text, opts = {}) {
+    await this.request(
+      "POST",
+      `/collections/${enc(collection)}/remember`,
+      prune({
+        id,
+        text,
+        mode: opts.mode,
+        attrs: opts.attrs ? encodeAttrs(opts.attrs) : void 0
+      })
+    );
+  }
+  /**
+   * Embed `query` and vector-search `collection`, best-first (attrs decoded to
+   * plain JS values). Refused with a cross-model guard if the collection was
+   * written with a different embedder than the server's.
+   */
+  recall(collection, query, opts = {}) {
+    return this.searchRequest(`/collections/${enc(collection)}/recall`, {
+      query,
+      top_k: opts.topK,
+      min_score: opts.minScore,
       filter: opts.filter ?? []
     });
   }
@@ -294,12 +493,18 @@ var NidusClient = class {
   /** Run a search-family request and decode the resulting hits' attrs. */
   async searchRequest(path, body) {
     const hits = await this.request("POST", path, prune(body));
-    return hits.map((h) => ({
+    return hits.map((h) => this.decodeHit(h));
+  }
+  /** One wire hit into a {@link Hit}. Shared so every search surface decodes identically. */
+  decodeHit(h) {
+    return {
       collection: h.collection,
       id: h.id,
       score: h.score,
-      attrs: decodeAttrs(h.attrs)
-    }));
+      attrs: decodeAttrs(h.attrs),
+      // Kept absent, not `undefined`, so an unannotated hit is the shape it always was.
+      ...h.annotations ? { annotations: decodeAnnotations(h.annotations) } : {}
+    };
   }
   /** Issue a request and parse a JSON body, mapping a non-2xx to {@link NidusError}. */
   async request(method, path, body) {
@@ -336,9 +541,42 @@ var NidusClient = class {
     }
   }
 };
+function encodeRankBy(rank) {
+  if (!rank) return void 0;
+  const d = rank.decay;
+  return {
+    Decay: prune({
+      field: d.field,
+      origin: d.origin instanceof Date ? d.origin.getTime() : d.origin,
+      scale: d.scale,
+      decay: d.decay,
+      lambda: d.lambda,
+      missing: d.missing
+    })
+  };
+}
+function encodeHighlight(h) {
+  if (h === void 0 || h === false) return void 0;
+  if (h === true) return {};
+  return prune({
+    max_fragments: h.maxFragments,
+    fragment_chars: h.fragmentChars
+  });
+}
 function enc(name) {
   return encodeURIComponent(name);
 }
+function encodeFtsField(f2) {
+  if (typeof f2 === "string") return f2;
+  return prune({
+    field: f2.field,
+    k1: f2.k1,
+    b: f2.b,
+    language: f2.language,
+    ascii_folding: f2.asciiFolding,
+    max_token_len: f2.maxTokenLen
+  });
+}
 function prune(body) {
   const out = {};
   for (const [k, val] of Object.entries(body)) {
@@ -367,6 +605,13 @@ var f = {
   }),
   /** `attrs[key]` is a `Str` matching the glob pattern (`*`, `?`, `[..]`). */
   glob: (key, pattern) => ({ Glob: [key, pattern] }),
+  /**
+   * {@link f.glob}, ignoring **ASCII** case on both sides — `"Src/*"` matches
+   * `"src/main.rs"`. Non-ASCII is not folded (`É` does not match `é`).
+   */
+  iglob: (key, pattern) => ({
+    IGlob: [key, pattern]
+  }),
   /** `attrs[key]` equals one of `values`. */
   in: (key, values) => ({
     In: [key, values.map(encodeValue)]
@@ -391,6 +636,56 @@ var f = {
   ge: (key, value) => ({
     Ge: [key, encodeValue(value)]
   }),
+  /** `attrs[key]` is a `List` containing `value` (whole-element, not substring). */
+  contains: (key, value) => ({
+    Contains: [key, encodeValue(value)]
+  }),
+  /** `attrs[key]` is a present `List` not containing `value`. */
+  notContains: (key, value) => ({
+    NotContains: [key, encodeValue(value)]
+  }),
+  /** `attrs[key]` is a `List` sharing at least one element with `values`. */
+  containsAny: (key, values) => ({
+    ContainsAny: [key, values.map(encodeValue)]
+  }),
+  /** Every sub-predicate holds. `all()` is `true`. */
+  all: (...preds) => ({ All: preds }),
+  /** At least one sub-predicate holds. `any()` is `false`. */
+  any: (...preds) => ({ Any: preds }),
+  /**
+   * The sub-predicate does not hold. Differs from {@link f.ne} on an absent key:
+   * `not(eq(k, v))` matches a record with no `k`, `ne(k, v)` does not.
+   */
+  not: (pred) => ({ Not: pred }),
+  /**
+   * `attrs[key]` is within `maxEdits` Levenshtein edits of `text`, ASCII-case-folded on
+   * both sides; a `List` matches if any element does. The only three-element predicate.
+   * A `maxEdits` above 8 is refused by the server, not clamped.
+   */
+  fuzzy: (key, text, maxEdits) => ({
+    Fuzzy: [key, text, maxEdits]
+  }),
+  /**
+   * Every token of `text` appears among `attrs[key]`'s tokens, in any order. Tokens are
+   * ASCII-case-folded runs of alphanumerics; a `List` matches if any single element does.
+   */
+  containsAllTokens: (key, text) => ({
+    ContainsAllTokens: [key, text]
+  }),
+  /** At least one token of `text` appears among `attrs[key]`'s tokens. Empty never matches. */
+  containsAnyToken: (key, text) => ({
+    ContainsAnyToken: [key, text]
+  }),
+  /** `text`'s tokens appear consecutively and in order — a phrase match. */
+  containsTokenSequence: (key, text) => ({
+    ContainsTokenSequence: [key, text]
+  }),
+  /**
+   * `attrs[key]` matches the regular expression, **anchored at both ends** like
+   * {@link f.glob} — `.*` opts back into a substring search, and `(?i)` into case folding.
+   * The syntax is Rust's `regex`, not JS's: no backreferences and no lookaround.
+   */
+  regex: (key, pattern) => ({ Regex: [key, pattern] }),
   /** Collect predicates into a {@link Filter} (purely sugar — they already AND). */
   and: (...preds) => preds
 };
@@ -405,6 +700,7 @@ var f = {
   f,
   v
 });
+//! Decoding a hit's optional annotations — the opt-in "why did this match".
 //! Error type carrying the HTTP status the server reported.
 //! Ergonomic constructors and decoders for the externally-tagged `Value` wire type.
 //! `NidusClient` — a remote client over the `nidus serve` HTTP API.