@lde/search-typesense 0.1.1 → 0.3.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,22 +1,51 @@
1
1
  # @lde/search-typesense
2
2
 
3
- [Typesense](https://typesense.org/) engine adapter for RDF-backed search
4
- pipelines. Engine-specific (Typesense) but domain-agnostic – the caller supplies
5
- the collection schema and documents.
3
+ [Typesense](https://typesense.org/) engine adapter for the engine- and
4
+ domain-agnostic [`@lde/search`](../search) core. **Engine-specific (Typesense) but
5
+ domain-agnostic** – you supply a `SearchType`; this package never names your
6
+ domain. It is the Typesense implementation of the `SearchEngine` port: it derives
7
+ a collection schema from the field model, compiles the neutral `SearchQuery` into
8
+ Typesense search params, runs it, reconstructs the engine-neutral `SearchResult`,
9
+ and manages the search index lifecycle (blue/green rebuild).
6
10
 
7
- The engine-agnostic half of the pipeline – framing `CONSTRUCT` quads into a
8
- JSON-LD IR and projecting that IR into flat documents from a declarative field
9
- spec – lives in [`@lde/search`](../search). This package consumes those
10
- documents and writes them to Typesense.
11
+ ## Collection schema and engine
12
+
13
+ `buildCollectionSchema(searchType, { name, defaultSortingField, … })` derives a
14
+ Typesense collection from the unified `SearchField` model — the Typesense field
15
+ type comes from each field’s `kind`, and the physical fanout (per-locale
16
+ search/sort keys) matches what the projection writes, via
17
+ `@lde/search`’s `physicalFields`, so the index and the documents cannot drift.
18
+
19
+ `createTypesenseSearchEngine(client, { collection, labelsCollection })` is the
20
+ `SearchEngine` implementation. Each search:
21
+
22
+ - validates the query against the search type (the port contract — a
23
+ structurally invalid query is rejected, never sent);
24
+ - compiles it into Typesense search params (`buildSearchParams`);
25
+ - runs the search;
26
+ - resolves reference (and reference-facet) labels from the sidecar `labels`
27
+ collection in a single lookup;
28
+ - reconstructs the logical `SearchResult` (`parseSearchResponse`) — language
29
+ maps, labelled references, labelled facet buckets.
30
+
31
+ `searchFacets` – the port’s batch entry point – answers a whole batch of
32
+ facet-only queries (e.g. a faceted sidebar’s skip-own-filter variants) as a
33
+ **single `multi_search` round-trip**, with one bundled label lookup shared by
34
+ every facet result in the batch. A failed entry is reported in place as a
35
+ per-query outcome, so its siblings’ facets survive.
36
+
37
+ The pure halves `buildSearchParams` and `parseSearchResponse` are exported for
38
+ direct use and testing.
11
39
 
12
40
  ## Indexing
13
41
 
14
- `rebuild` blue/green-rebuilds a search index in one call: it creates a fresh
15
- versioned collection (`${schema.name}_<timestamp>`), streams the documents into
16
- it in batches, atomically repoints the `schema.name` alias to it, then drops the
17
- collection it superseded. The caller passes only the logical index name (as
18
- `schema.name`) and a stream of documents; the versioned collection and the alias
19
- are managed for them.
42
+ `rebuild` blue/green-rebuilds a search index in one call, straight from the
43
+ declaration: it derives the collection schema from your `SearchType` (via
44
+ `buildCollectionSchema`), creates a fresh versioned collection
45
+ (`${name}_<timestamp>`), streams the documents into it in batches, atomically
46
+ repoints the `name` alias to it, then drops the collection it superseded. The
47
+ caller passes the `SearchType`, the logical index `name` and a stream of
48
+ documents; the versioned collection and the alias are managed for them.
20
49
 
21
50
  ```ts
22
51
  import { Client } from 'typesense';
@@ -30,9 +59,13 @@ const client = new Client({
30
59
  // `documents` is an async iterable (e.g. a streaming projection); only one
31
60
  // batch is held in memory at a time. `rebuild` returns the live collection name
32
61
  // and the imported count (or `null` if another rebuild was already running).
33
- const result = await rebuild(client, schema, documents);
62
+ const result = await rebuild(client, documents, DATASET, { name: 'datasets' });
34
63
  ```
35
64
 
65
+ The options accept everything `buildCollectionSchema` does (`defaultLocale`,
66
+ `defaultSortingField`, `synonymSets`) plus the rebuild knobs (`batchSize`,
67
+ `lockTtlMs`).
68
+
36
69
  `rebuild` takes a `Client` the caller owns (and reuses for queries), so this
37
70
  package adds no connection or document type of its own – any object with an `id`
38
71
  is a valid document, including the `SearchDocument`s `@lde/search` produces.
package/dist/adapter.d.ts CHANGED
@@ -1,8 +1,21 @@
1
- import type { Client, CollectionCreateSchema } from 'typesense';
1
+ import type { Client } from 'typesense';
2
+ import type { SearchType } from '@lde/search';
3
+ import { type CollectionSchemaOptions } from './collection-schema.js';
4
+ /** {@link rebuild} options: the collection-schema options (`name` is the
5
+ * logical index name the alias is kept on) plus the rebuild tuning knobs. */
6
+ export interface RebuildOptions extends CollectionSchemaOptions {
7
+ /** Documents imported per Typesense request (default 1000). */
8
+ readonly batchSize?: number;
9
+ /** A held lock older than this (ms) is reclaimed (default 10 minutes). */
10
+ readonly lockTtlMs?: number;
11
+ }
2
12
  /**
3
- * Blue/green-rebuild the search index `name`.
13
+ * Blue/green-rebuild the search index `options.name` from one declarative
14
+ * source: the collection schema is derived from `searchType`
15
+ * ({@link buildCollectionSchema}) and the documents are streamed in — one call
16
+ * from declaration to live index.
4
17
  *
5
- * 1. create a fresh versioned collection (`${name}_<timestamp>`) from `schema`
18
+ * 1. create a fresh versioned collection (`${name}_<timestamp>`)
6
19
  * 2. stream `documents` into it in batches
7
20
  * 3. atomically repoint the `name` alias to the new collection, then
8
21
  * drop the collection it superseded. The caller passes only the logical
@@ -36,12 +49,7 @@ import type { Client, CollectionCreateSchema } from 'typesense';
36
49
  */
37
50
  export declare function rebuild<Document extends {
38
51
  id: string;
39
- }>(client: Client, schema: CollectionCreateSchema, documents: AsyncIterable<Document>, options?: {
40
- /** Documents imported per Typesense request (default 1000). */
41
- batchSize?: number;
42
- /** A held lock older than this (ms) is reclaimed (default 10 minutes). */
43
- lockTtlMs?: number;
44
- }): Promise<{
52
+ }>(client: Client, documents: AsyncIterable<Document>, searchType: SearchType, options: RebuildOptions): Promise<{
45
53
  collection: string;
46
54
  imported: number;
47
55
  } | null>;
@@ -1 +1 @@
1
- {"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../src/adapter.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAE,sBAAsB,EAAkB,MAAM,WAAW,CAAC;AAKhF;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkCG;AACH,wBAAsB,OAAO,CAAC,QAAQ,SAAS;IAAE,EAAE,EAAE,MAAM,CAAA;CAAE,EAC3D,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,sBAAsB,EAC9B,SAAS,EAAE,aAAa,CAAC,QAAQ,CAAC,EAClC,OAAO,GAAE;IACP,+DAA+D;IAC/D,SAAS,CAAC,EAAE,MAAM,CAAC;IACnB,0EAA0E;IAC1E,SAAS,CAAC,EAAE,MAAM,CAAC;CACf,GACL,OAAO,CAAC;IAAE,UAAU,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,MAAM,CAAA;CAAE,GAAG,IAAI,CAAC,CAmC1D"}
1
+ {"version":3,"file":"adapter.d.ts","sourceRoot":"","sources":["../src/adapter.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAkB,MAAM,WAAW,CAAC;AACxD,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,aAAa,CAAC;AAC9C,OAAO,EAEL,KAAK,uBAAuB,EAC7B,MAAM,wBAAwB,CAAC;AAKhC;8EAC8E;AAC9E,MAAM,WAAW,cAAe,SAAQ,uBAAuB;IAC7D,+DAA+D;IAC/D,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,CAAC;IAC5B,0EAA0E;IAC1E,QAAQ,CAAC,SAAS,CAAC,EAAE,MAAM,CAAC;CAC7B;AAED;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAqCG;AACH,wBAAsB,OAAO,CAAC,QAAQ,SAAS;IAAE,EAAE,EAAE,MAAM,CAAA;CAAE,EAC3D,MAAM,EAAE,MAAM,EACd,SAAS,EAAE,aAAa,CAAC,QAAQ,CAAC,EAClC,UAAU,EAAE,UAAU,EACtB,OAAO,EAAE,cAAc,GACtB,OAAO,CAAC;IAAE,UAAU,EAAE,MAAM,CAAC;IAAC,QAAQ,EAAE,MAAM,CAAA;CAAE,GAAG,IAAI,CAAC,CAwC1D"}
package/dist/adapter.js CHANGED
@@ -1,9 +1,13 @@
1
+ import { buildCollectionSchema, } from './collection-schema.js';
1
2
  const LOCK_COLLECTION = 'rebuild_locks';
2
3
  const DEFAULT_LOCK_TTL_MS = 10 * 60 * 1000;
3
4
  /**
4
- * Blue/green-rebuild the search index `name`.
5
+ * Blue/green-rebuild the search index `options.name` from one declarative
6
+ * source: the collection schema is derived from `searchType`
7
+ * ({@link buildCollectionSchema}) and the documents are streamed in — one call
8
+ * from declaration to live index.
5
9
  *
6
- * 1. create a fresh versioned collection (`${name}_<timestamp>`) from `schema`
10
+ * 1. create a fresh versioned collection (`${name}_<timestamp>`)
7
11
  * 2. stream `documents` into it in batches
8
12
  * 3. atomically repoint the `name` alias to the new collection, then
9
13
  * drop the collection it superseded. The caller passes only the logical
@@ -35,9 +39,10 @@ const DEFAULT_LOCK_TTL_MS = 10 * 60 * 1000;
35
39
  * minutes) can be reclaimed by another caller and run concurrently; size the
36
40
  * TTL above your longest rebuild.
37
41
  */
38
- export async function rebuild(client, schema, documents, options = {}) {
39
- const { batchSize = 1000, lockTtlMs = DEFAULT_LOCK_TTL_MS } = options;
40
- const name = schema.name;
42
+ export async function rebuild(client, documents, searchType, options) {
43
+ const { batchSize = 1000, lockTtlMs = DEFAULT_LOCK_TTL_MS, ...schemaOptions } = options;
44
+ const schema = buildCollectionSchema(searchType, schemaOptions);
45
+ const name = schemaOptions.name;
41
46
  if (!(await acquireLock(client, name, lockTtlMs))) {
42
47
  return null;
43
48
  }
@@ -0,0 +1,30 @@
1
+ import type { CollectionCreateSchema } from 'typesense';
2
+ import { type SearchType } from '@lde/search';
3
+ /** Deployment-specific options the generic field model does not carry. */
4
+ export interface CollectionSchemaOptions {
5
+ /** The Typesense collection (or alias) name. */
6
+ readonly name: string;
7
+ /** Snowball stemming locale for non-localized searchable fields (e.g. `en`).
8
+ * Unset, those fields are not stemmed — folding still applies — so no
9
+ * language is ever assumed. Localized text search fields always stem in
10
+ * their own locale. */
11
+ readonly defaultLocale?: string;
12
+ /** The field Typesense sorts by when a query imposes no order. */
13
+ readonly defaultSortingField?: string;
14
+ /** Synonym sets the collection references (synced separately). */
15
+ readonly synonymSets?: readonly string[];
16
+ }
17
+ /**
18
+ * Build a Typesense collection schema from the unified {@link SearchType}, so
19
+ * the index and the projection are driven by one declarative source and cannot
20
+ * drift. Each field fans out into the same physical fields the projection writes
21
+ * ({@link physicalFields}); the Typesense field type is derived from the field
22
+ * `kind`, never re-declared.
23
+ *
24
+ * Text stems each folded `*_search_${locale}` field in its own language; the
25
+ * untagged `und` locale (and any searchable keyword/reference companion)
26
+ * stems in `defaultLocale` when one is set, and is left unstemmed (folded
27
+ * only) otherwise.
28
+ */
29
+ export declare function buildCollectionSchema(searchType: SearchType, options: CollectionSchemaOptions): CollectionCreateSchema;
30
+ //# sourceMappingURL=collection-schema.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"collection-schema.d.ts","sourceRoot":"","sources":["../src/collection-schema.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,sBAAsB,EAAE,MAAM,WAAW,CAAC;AAExD,OAAO,EAAoB,KAAK,UAAU,EAAE,MAAM,aAAa,CAAC;AAGhE,0EAA0E;AAC1E,MAAM,WAAW,uBAAuB;IACtC,gDAAgD;IAChD,QAAQ,CAAC,IAAI,EAAE,MAAM,CAAC;IACtB;;;4BAGwB;IACxB,QAAQ,CAAC,aAAa,CAAC,EAAE,MAAM,CAAC;IAChC,kEAAkE;IAClE,QAAQ,CAAC,mBAAmB,CAAC,EAAE,MAAM,CAAC;IACtC,kEAAkE;IAClE,QAAQ,CAAC,WAAW,CAAC,EAAE,SAAS,MAAM,EAAE,CAAC;CAC1C;AAED;;;;;;;;;;;GAWG;AACH,wBAAgB,qBAAqB,CACnC,UAAU,EAAE,UAAU,EACtB,OAAO,EAAE,uBAAuB,GAC/B,sBAAsB,CAexB"}
@@ -0,0 +1,105 @@
1
+ import { physicalFields } from '@lde/search/adapter';
2
+ /**
3
+ * Build a Typesense collection schema from the unified {@link SearchType}, so
4
+ * the index and the projection are driven by one declarative source and cannot
5
+ * drift. Each field fans out into the same physical fields the projection writes
6
+ * ({@link physicalFields}); the Typesense field type is derived from the field
7
+ * `kind`, never re-declared.
8
+ *
9
+ * Text stems each folded `*_search_${locale}` field in its own language; the
10
+ * untagged `und` locale (and any searchable keyword/reference companion)
11
+ * stems in `defaultLocale` when one is set, and is left unstemmed (folded
12
+ * only) otherwise.
13
+ */
14
+ export function buildCollectionSchema(searchType, options) {
15
+ const { defaultLocale } = options;
16
+ const collection = {
17
+ name: options.name,
18
+ fields: searchType.fields.flatMap((field) => typesenseFields(field, defaultLocale, options.defaultSortingField)),
19
+ };
20
+ if (options.defaultSortingField !== undefined) {
21
+ collection.default_sorting_field = options.defaultSortingField;
22
+ }
23
+ if (options.synonymSets !== undefined) {
24
+ collection.synonym_sets = [...options.synonymSets];
25
+ }
26
+ return collection;
27
+ }
28
+ /** The physical Typesense fields one declaration produces. */
29
+ function typesenseFields(field, defaultLocale, defaultSortingField) {
30
+ const names = physicalFields(field);
31
+ if (field.kind === 'text') {
32
+ const locales = field.locales;
33
+ return [
34
+ // Display labels: stored, not indexed for search (search uses the folded
35
+ // companions), accents preserved.
36
+ ...names.display.map((name) => ({
37
+ name,
38
+ type: 'string',
39
+ index: false,
40
+ optional: true,
41
+ })),
42
+ // One folded search field per locale, each stemmed in its own
43
+ // language; the untagged `und` locale is folded but unstemmed unless
44
+ // the deployment opts in via `defaultLocale`.
45
+ ...names.search.map((name, index) => {
46
+ const locale = locales[index] === 'und' ? defaultLocale : locales[index];
47
+ return {
48
+ name,
49
+ type: 'string',
50
+ optional: true,
51
+ ...(locale !== undefined && { stem: true, locale }),
52
+ };
53
+ }),
54
+ ...names.sort.map((name) => ({
55
+ name,
56
+ type: 'string',
57
+ sort: true,
58
+ optional: true,
59
+ })),
60
+ ];
61
+ }
62
+ const valueType = typesenseValueType(field);
63
+ const fields = [
64
+ {
65
+ name: field.name,
66
+ type: valueType,
67
+ facet: field.facetable ?? false,
68
+ sort: field.sortable ?? false,
69
+ // A `required` field is non-optional; so is the `default_sorting_field`,
70
+ // which Typesense requires to be present. Everything else may be absent.
71
+ optional: field.required !== true && field.name !== defaultSortingField,
72
+ },
73
+ ];
74
+ // `names.search` is non-empty exactly when the field projects a folded
75
+ // search companion — physicalFields owns that rule.
76
+ for (const name of names.search) {
77
+ fields.push({
78
+ name,
79
+ type: valueType,
80
+ optional: true,
81
+ ...(defaultLocale !== undefined && {
82
+ stem: true,
83
+ locale: defaultLocale,
84
+ }),
85
+ });
86
+ }
87
+ return fields;
88
+ }
89
+ /** The Typesense field type for a non-localized field, from its `kind`. 64-bit
90
+ * integers (and dates, stored as Unix seconds) so large counts never overflow. */
91
+ function typesenseValueType(field) {
92
+ switch (field.kind) {
93
+ case 'integer':
94
+ case 'date':
95
+ return 'int64';
96
+ case 'number':
97
+ return 'float';
98
+ case 'boolean':
99
+ return 'bool';
100
+ case 'keyword':
101
+ case 'reference':
102
+ case 'text':
103
+ return field.array === true ? 'string[]' : 'string';
104
+ }
105
+ }
package/dist/index.d.ts CHANGED
@@ -1,2 +1,9 @@
1
1
  export { rebuild } from './adapter.js';
2
+ export type { RebuildOptions } from './adapter.js';
3
+ export { buildCollectionSchema } from './collection-schema.js';
4
+ export type { CollectionSchemaOptions } from './collection-schema.js';
5
+ export { buildSearchParams } from './query-compiler.js';
6
+ export type { BuildSearchParamsOptions } from './query-compiler.js';
7
+ export { createTypesenseSearchEngine, parseSearchResponse } from './search.js';
8
+ export type { TypesenseSearchEngineOptions, TypesenseSearchResponse, } from './search.js';
2
9
  //# sourceMappingURL=index.d.ts.map
@@ -1 +1 @@
1
- {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC"}
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,OAAO,EAAE,MAAM,cAAc,CAAC;AACvC,YAAY,EAAE,cAAc,EAAE,MAAM,cAAc,CAAC;AACnD,OAAO,EAAE,qBAAqB,EAAE,MAAM,wBAAwB,CAAC;AAC/D,YAAY,EAAE,uBAAuB,EAAE,MAAM,wBAAwB,CAAC;AACtE,OAAO,EAAE,iBAAiB,EAAE,MAAM,qBAAqB,CAAC;AACxD,YAAY,EAAE,wBAAwB,EAAE,MAAM,qBAAqB,CAAC;AACpE,OAAO,EAAE,2BAA2B,EAAE,mBAAmB,EAAE,MAAM,aAAa,CAAC;AAC/E,YAAY,EACV,4BAA4B,EAC5B,uBAAuB,GACxB,MAAM,aAAa,CAAC"}
package/dist/index.js CHANGED
@@ -1 +1,4 @@
1
1
  export { rebuild } from './adapter.js';
2
+ export { buildCollectionSchema } from './collection-schema.js';
3
+ export { buildSearchParams } from './query-compiler.js';
4
+ export { createTypesenseSearchEngine, parseSearchResponse } from './search.js';
@@ -0,0 +1,42 @@
1
+ import type { SearchParams } from 'typesense/lib/Typesense/Documents.js';
2
+ import { type Filter, type SearchQuery, type SearchType } from '@lde/search';
3
+ /**
4
+ * Options for {@link buildSearchParams} — the query half of the engine
5
+ * adapter. {@link TypesenseSearchEngineOptions} extends this, so each knob is
6
+ * declared once and the engine forwards its options wholesale.
7
+ */
8
+ export interface BuildSearchParamsOptions {
9
+ /**
10
+ * Cap on the number of buckets returned per facet (`max_facet_values`). Left
11
+ * unset, Typesense defaults to 10 — too few for high-cardinality facets
12
+ * (publisher, keyword), so a deployment with such facets must raise it. Range
13
+ * facets return one bucket per declared range regardless, but a value > the
14
+ * range count is still safe.
15
+ */
16
+ readonly maxFacetValues?: number;
17
+ /**
18
+ * Called for each `where` clause that compiles to nothing and is therefore
19
+ * skipped: an unknown field, an operator that does not match the field’s
20
+ * kind ({@link filterOperatorFor}), an empty `in` list, or a `range` with no
21
+ * usable bound. Skipping keeps a malformed clause from reaching the engine
22
+ * as garbage; supply this to log it instead of losing it silently. Through
23
+ * the engine, a structurally invalid query throws up front
24
+ * (`assertValidQuery`), so there only the vacuous clauses reach this.
25
+ */
26
+ readonly onIgnoredFilter?: (filter: Filter) => void;
27
+ }
28
+ /**
29
+ * Compile the engine-neutral {@link SearchQuery} into Typesense search
30
+ * parameters — the query half of the engine adapter. Pure (no client, no env),
31
+ * so the mapping is asserted directly in unit tests. Field names come from
32
+ * {@link physicalFields}, the same convention the projection and the collection
33
+ * schema use, so a query can never reference a field the index does not carry.
34
+ */
35
+ export declare function buildSearchParams(query: SearchQuery, searchType: SearchType, options?: BuildSearchParamsOptions): SearchParams<object>;
36
+ /**
37
+ * Backtick-wrap a filter value so reserved characters in IRIs and media types
38
+ * (`:`, `/`, `&`, `,`, …) are taken literally instead of parsed as filter syntax.
39
+ * An embedded backtick is escaped.
40
+ */
41
+ export declare function escapeFilterValue(value: string): string;
42
+ //# sourceMappingURL=query-compiler.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"query-compiler.d.ts","sourceRoot":"","sources":["../src/query-compiler.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,YAAY,EAAE,MAAM,sCAAsC,CAAC;AAEzE,OAAO,EAEL,KAAK,MAAM,EAEX,KAAK,WAAW,EAChB,KAAK,UAAU,EAEhB,MAAM,aAAa,CAAC;AAYrB;;;;GAIG;AACH,MAAM,WAAW,wBAAwB;IACvC;;;;;;OAMG;IACH,QAAQ,CAAC,cAAc,CAAC,EAAE,MAAM,CAAC;IACjC;;;;;;;;OAQG;IACH,QAAQ,CAAC,eAAe,CAAC,EAAE,CAAC,MAAM,EAAE,MAAM,KAAK,IAAI,CAAC;CACrD;AAED;;;;;;GAMG;AACH,wBAAgB,iBAAiB,CAC/B,KAAK,EAAE,WAAW,EAClB,UAAU,EAAE,UAAU,EACtB,OAAO,GAAE,wBAA6B,GACrC,YAAY,CAAC,MAAM,CAAC,CAkCtB;AA2LD;;;;GAIG;AACH,wBAAgB,iBAAiB,CAAC,KAAK,EAAE,MAAM,GAAG,MAAM,CAEvD"}
@@ -0,0 +1,193 @@
1
+ import { fold } from '@lde/text-normalization';
2
+ import { fieldNamed, filterOperator, filterOperatorFor, isoToUnixSeconds, isRangeFacet, pageForOffset, physicalFields, searchableFields, } from '@lde/search/adapter';
3
+ /**
4
+ * Compile the engine-neutral {@link SearchQuery} into Typesense search
5
+ * parameters — the query half of the engine adapter. Pure (no client, no env),
6
+ * so the mapping is asserted directly in unit tests. Field names come from
7
+ * {@link physicalFields}, the same convention the projection and the collection
8
+ * schema use, so a query can never reference a field the index does not carry.
9
+ */
10
+ export function buildSearchParams(query, searchType, options = {}) {
11
+ const folded = query.text !== undefined && query.text.length > 0
12
+ ? fold(query.text)
13
+ : undefined;
14
+ const { names, weights } = queryFields(searchType, query.locale);
15
+ const filterBy = compileFilterBy(query.where, searchType, options.onIgnoredFilter);
16
+ const sortBy = query.orderBy
17
+ .map((sort) => compileSort(sort, searchType, query.locale))
18
+ .join(',');
19
+ const params = {
20
+ q: folded ?? '*',
21
+ query_by: names.join(','),
22
+ query_by_weights: weights.join(','),
23
+ per_page: query.limit,
24
+ page: pageForOffset(query.offset, query.limit),
25
+ };
26
+ if (filterBy.length > 0) {
27
+ params.filter_by = filterBy;
28
+ }
29
+ if (sortBy.length > 0) {
30
+ params.sort_by = sortBy;
31
+ }
32
+ if (query.facets.length > 0) {
33
+ params.facet_by = compileFacetBy(query.facets, searchType);
34
+ if (options.maxFacetValues !== undefined) {
35
+ params.max_facet_values = options.maxFacetValues;
36
+ }
37
+ }
38
+ return params;
39
+ }
40
+ /**
41
+ * The `facet_by` clause. A facet on a numeric field that declares
42
+ * {@link SearchField.facetRanges} faceted into those fixed half-open `[min, max)`
43
+ * bins (a histogram); every other facet is a plain per-value facet on its field
44
+ * name. Typesense range syntax is already start-inclusive/end-exclusive, so the
45
+ * declared bounds pass straight through with no boundary fix-up.
46
+ */
47
+ function compileFacetBy(facets, searchType) {
48
+ return facets
49
+ .map((name) => {
50
+ const field = fieldNamed(searchType, name);
51
+ return field !== undefined && isRangeFacet(field)
52
+ ? compileRangeFacet(field.name, field.facetRanges)
53
+ : name;
54
+ })
55
+ .join(',');
56
+ }
57
+ /** `name(key:[min, max], …)`; a blank bound is open-ended (Typesense `[75, ]`). */
58
+ function compileRangeFacet(name, ranges) {
59
+ const bins = ranges
60
+ .map((range) => `${range.key}:[${range.min ?? ''}, ${range.max ?? ''}]`)
61
+ .join(', ');
62
+ return `${name}(${bins})`;
63
+ }
64
+ /**
65
+ * The `query_by` fields and aligned weights. Each searchable field expands to its
66
+ * folded `*_search` companion(s); a localized field’s active-locale companion
67
+ * keeps its full weight while the other locale is gently demoted (−1, floored at
68
+ * 1), so a match in the user’s language ranks higher while cross-language matches
69
+ * still surface.
70
+ */
71
+ function queryFields(searchType, locale) {
72
+ const names = [];
73
+ const weights = [];
74
+ for (const field of searchableFields(searchType)) {
75
+ const search = physicalFields(field).search;
76
+ const baseWeight = field.searchable.weight;
77
+ if (field.kind === 'text') {
78
+ const locales = field.locales;
79
+ search.forEach((name, index) => {
80
+ names.push(name);
81
+ // The active locale keeps full weight; `und` is language-neutral, so
82
+ // it is never demoted (an untagged-only field would otherwise always
83
+ // rank below its declared weight).
84
+ weights.push(locales[index] === locale || locales[index] === 'und'
85
+ ? baseWeight
86
+ : Math.max(1, baseWeight - 1));
87
+ });
88
+ }
89
+ else {
90
+ for (const name of search) {
91
+ names.push(name);
92
+ weights.push(baseWeight);
93
+ }
94
+ }
95
+ }
96
+ return { names, weights };
97
+ }
98
+ /** AND-join the compiled `where` clauses; a clause that compiles to nothing is
99
+ * skipped and reported to `onIgnoredFilter`. */
100
+ function compileFilterBy(where, searchType, onIgnoredFilter) {
101
+ return where
102
+ .map((filter) => {
103
+ const clause = compileFilter(filter, searchType);
104
+ if (clause === undefined) {
105
+ onIgnoredFilter?.(filter);
106
+ }
107
+ return clause;
108
+ })
109
+ .filter((clause) => clause !== undefined)
110
+ .join(' && ');
111
+ }
112
+ function compileFilter(filter, searchType) {
113
+ const field = fieldNamed(searchType, filter.field);
114
+ if (field === undefined) {
115
+ return undefined;
116
+ }
117
+ // A clause whose operator does not match the field's kind (e.g. `range` on a
118
+ // keyword) would reach the engine as garbage syntax — skip it instead.
119
+ if (filterOperatorFor(field.kind) !== filterOperator(filter)) {
120
+ return undefined;
121
+ }
122
+ if ('in' in filter) {
123
+ return filter.in.length > 0
124
+ ? compileMembership(field, filter.in)
125
+ : undefined;
126
+ }
127
+ if ('range' in filter) {
128
+ return compileRange(field, filter.range);
129
+ }
130
+ return `${field.name}:=${filter.is}`;
131
+ }
132
+ /**
133
+ * A membership clause. A non-facet (tokenized) field uses the exact `:=`
134
+ * operator so an IRI cannot partial-match on a shared path segment.
135
+ */
136
+ function compileMembership(field, values) {
137
+ const list = `[${values.map(escapeFilterValue).join(',')}]`;
138
+ return field.facetable !== true
139
+ ? `${field.name}:=${list}`
140
+ : `${field.name}:${list}`;
141
+ }
142
+ /** An inclusive Typesense range clause, or `undefined` when neither bound is set. */
143
+ function compileRange(field, range) {
144
+ const name = field.name;
145
+ const min = storedBound(field, range.min);
146
+ const max = storedBound(field, range.max);
147
+ if (min !== undefined && max !== undefined) {
148
+ return `${name}:[${min}..${max}]`;
149
+ }
150
+ if (min !== undefined) {
151
+ return `${name}:>=${min}`;
152
+ }
153
+ if (max !== undefined) {
154
+ return `${name}:<=${max}`;
155
+ }
156
+ return undefined;
157
+ }
158
+ /** A range bound as stored: a `date` field’s ISO 8601 bound becomes the indexed
159
+ * Unix seconds ({@link isoToUnixSeconds}); an unparseable bound is dropped. */
160
+ function storedBound(field, bound) {
161
+ return field.kind === 'date' && typeof bound === 'string'
162
+ ? isoToUnixSeconds(bound)
163
+ : bound;
164
+ }
165
+ /**
166
+ * One `sort_by` term. `relevance` maps to Typesense’s `_text_match`; a localized
167
+ * text field sorts on its active-locale folded key; any other field (including a
168
+ * deployment tie-break like `status_rank`) sorts on its own name.
169
+ */
170
+ function compileSort(sort, searchType, locale) {
171
+ if (sort.field === 'relevance') {
172
+ return `_text_match:${sort.direction}`;
173
+ }
174
+ const field = fieldNamed(searchType, sort.field);
175
+ if (field !== undefined && field.kind === 'text') {
176
+ // Sort on the active locale's key, falling back to the first declared
177
+ // locale (e.g. an und-only field sorted by a user in any language).
178
+ const index = field.locales.indexOf(locale);
179
+ const sortName = physicalFields(field).sort[index === -1 ? 0 : index];
180
+ if (sortName !== undefined) {
181
+ return `${sortName}:${sort.direction}`;
182
+ }
183
+ }
184
+ return `${sort.field}:${sort.direction}`;
185
+ }
186
+ /**
187
+ * Backtick-wrap a filter value so reserved characters in IRIs and media types
188
+ * (`:`, `/`, `&`, `,`, …) are taken literally instead of parsed as filter syntax.
189
+ * An embedded backtick is escaped.
190
+ */
191
+ export function escapeFilterValue(value) {
192
+ return `\`${value.replace(/`/g, '\\`')}\``;
193
+ }
@@ -0,0 +1,86 @@
1
+ import type { Client } from 'typesense';
2
+ import { type LocalizedValue, type SearchEngine, type SearchResult, type SearchSchema, type SearchType } from '@lde/search';
3
+ import { type BuildSearchParamsOptions } from './query-compiler.js';
4
+ /** Where the engine reads documents and (optionally) reference labels — plus
5
+ * every query-compiler knob ({@link BuildSearchParamsOptions}), declared once
6
+ * there and forwarded wholesale into each search. `TypeName` is the union of
7
+ * the schema’s type names, so a literal schema makes the `collections` map
8
+ * exhaustive at compile time. */
9
+ export interface TypesenseSearchEngineOptions<TypeName extends string = string> extends BuildSearchParamsOptions {
10
+ /** The collection (or alias) per search type, keyed by the type’s `name` —
11
+ * with a literal schema, omitting a type is a compile error. */
12
+ readonly collections: Readonly<Record<TypeName, string>>;
13
+ /** The sidecar `labels` collection (IRI → label); omit for id-only references. */
14
+ readonly labelsCollection?: string;
15
+ /**
16
+ * Called when reference-label resolution fails; the search then degrades to
17
+ * id-only references rather than failing. Optional — omit to swallow silently.
18
+ */
19
+ readonly onLabelError?: (error: unknown) => void;
20
+ /**
21
+ * Opt-in in-memory label cache. When set (and {@link labelsCollection} is
22
+ * set), the FULL sidecar `labels` collection is loaded once via the documents
23
+ * export endpoint and held in a process-lifetime cache for this many
24
+ * milliseconds; each `search` then resolves its reference labels by in-memory
25
+ * lookup instead of a per-search `multi_search` round-trip. Omit to keep the
26
+ * per-search {@link fetchLabels} behaviour unchanged.
27
+ */
28
+ readonly labelCacheTtlMs?: number;
29
+ }
30
+ /**
31
+ * A Typesense-backed {@link SearchEngine}, bound to the whole
32
+ * {@link SearchSchema} at construction — like every other schema consumer.
33
+ * Each type’s collection comes from `options.collections`; the label sidecar
34
+ * and its cache are shared across all types. `search` compiles the query
35
+ * ({@link buildSearchParams}), runs it against the type’s collection,
36
+ * resolves the reference labels for the page of hits from the sidecar
37
+ * `labels` collection in one lookup, and reconstructs the engine-neutral
38
+ * {@link SearchResult} ({@link parseSearchResponse}). `searchFacets` answers
39
+ * a whole facet batch (e.g. a sidebar’s skip-own-filter query variants) as a
40
+ * single `multi_search` round-trip with one shared label lookup; a failed
41
+ * entry is reported in place as a per-query outcome, so it never discards
42
+ * its siblings’ facets. Every engine specific stays here; consumers see only
43
+ * logical documents.
44
+ *
45
+ * With a schema built by `searchSchema` over `defineSearchType` literals,
46
+ * `search()` accepts only the deployment’s own types and returns typo-safe
47
+ * facet/document keys per call — no caller-side generics.
48
+ */
49
+ export declare function createTypesenseSearchEngine<const Types extends readonly SearchType[]>(client: Client, schema: SearchSchema<Types>, options: TypesenseSearchEngineOptions<Types[number]['name']>): SearchEngine<Types>;
50
+ /**
51
+ * Resolve labels for `iris` from the sidecar `labels` collection. Each
52
+ * `label_${locale}` becomes a language-map entry; the default `label` is the
53
+ * untagged (`und`) fallback when no locale variant exists.
54
+ *
55
+ * Sent as one `multi_search` (POST) call, the id-list split over per-search
56
+ * batches: the id-list of a page or facet carrying many references — e.g. a
57
+ * dataset with dozens of classes — would overflow Typesense’s GET query-string
58
+ * limit (4000 chars, and IRIs URL-encode to several times their length) if it
59
+ * travelled in the URL. POST puts it in the body; each batch stays under
60
+ * Typesense’s `per_page` cap, and bundling the batches keeps it one round-trip
61
+ * regardless of IRI count. Exported for unit testing against a fake client.
62
+ */
63
+ export declare function fetchLabels(client: Pick<Client, 'multiSearch'>, collection: string, iris: readonly string[]): Promise<Map<string, LocalizedValue>>;
64
+ /** The subset of a Typesense search response this adapter reads. */
65
+ export interface TypesenseSearchResponse {
66
+ readonly found: number;
67
+ readonly hits?: readonly {
68
+ readonly document: Record<string, unknown>;
69
+ }[];
70
+ readonly facet_counts?: readonly {
71
+ readonly field_name: string;
72
+ readonly counts: readonly {
73
+ readonly value: string;
74
+ readonly count: number;
75
+ }[];
76
+ }[];
77
+ }
78
+ /**
79
+ * Reconstruct a Typesense response into the engine-neutral {@link SearchResult}:
80
+ * the flat, fanned-out document is turned back into a logical one (per-locale
81
+ * display fields → a language map, reference IRIs → labelled references via the
82
+ * sidecar `labels` lookup, scalars passed through). `labels` maps a reference IRI
83
+ * to its resolved label; an IRI absent from it yields an id-only reference.
84
+ */
85
+ export declare function parseSearchResponse(response: TypesenseSearchResponse, searchType: SearchType, labels: ReadonlyMap<string, LocalizedValue>): SearchResult;
86
+ //# sourceMappingURL=search.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"search.d.ts","sourceRoot":"","sources":["../src/search.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,MAAM,EAAE,MAAM,WAAW,CAAC;AACxC,OAAO,EAGL,KAAK,cAAc,EAMnB,KAAK,YAAY,EAEjB,KAAK,YAAY,EACjB,KAAK,YAAY,EACjB,KAAK,UAAU,EAGhB,MAAM,aAAa,CAAC;AAUrB,OAAO,EAGL,KAAK,wBAAwB,EAC9B,MAAM,qBAAqB,CAAC;AAE7B;;;;kCAIkC;AAClC,MAAM,WAAW,4BAA4B,CAC3C,QAAQ,SAAS,MAAM,GAAG,MAAM,CAChC,SAAQ,wBAAwB;IAChC;qEACiE;IACjE,QAAQ,CAAC,WAAW,EAAE,QAAQ,CAAC,MAAM,CAAC,QAAQ,EAAE,MAAM,CAAC,CAAC,CAAC;IACzD,kFAAkF;IAClF,QAAQ,CAAC,gBAAgB,CAAC,EAAE,MAAM,CAAC;IACnC;;;OAGG;IACH,QAAQ,CAAC,YAAY,CAAC,EAAE,CAAC,KAAK,EAAE,OAAO,KAAK,IAAI,CAAC;IACjD;;;;;;;OAOG;IACH,QAAQ,CAAC,eAAe,CAAC,EAAE,MAAM,CAAC;CACnC;AAED;;;;;;;;;;;;;;;;;;GAkBG;AACH,wBAAgB,2BAA2B,CACzC,KAAK,CAAC,KAAK,SAAS,SAAS,UAAU,EAAE,EAEzC,MAAM,EAAE,MAAM,EACd,MAAM,EAAE,YAAY,CAAC,KAAK,CAAC,EAC3B,OAAO,EAAE,4BAA4B,CAAC,KAAK,CAAC,MAAM,CAAC,CAAC,MAAM,CAAC,CAAC,GAC3D,YAAY,CAAC,KAAK,CAAC,CAwKrB;AA+DD;;;;;;;;;;;;GAYG;AACH,wBAAsB,WAAW,CAC/B,MAAM,EAAE,IAAI,CAAC,MAAM,EAAE,aAAa,CAAC,EACnC,UAAU,EAAE,MAAM,EAClB,IAAI,EAAE,SAAS,MAAM,EAAE,GACtB,OAAO,CAAC,GAAG,CAAC,MAAM,EAAE,cAAc,CAAC,CAAC,CAmCtC;AA6BD,oEAAoE;AACpE,MAAM,WAAW,uBAAuB;IACtC,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;IACvB,QAAQ,CAAC,IAAI,CAAC,EAAE,SAAS;QAAE,QAAQ,CAAC,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAA;KAAE,EAAE,CAAC;IAC1E,QAAQ,CAAC,YAAY,CAAC,EAAE,SAAS;QAC/B,QAAQ,CAAC,UAAU,EAAE,MAAM,CAAC;QAC5B,QAAQ,CAAC,MAAM,EAAE,SAAS;YACxB,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;YACvB,QAAQ,CAAC,KAAK,EAAE,MAAM,CAAC;SACxB,EAAE,CAAC;KACL,EAAE,CAAC;CACL;AAED;;;;;;GAMG;AACH,wBAAgB,mBAAmB,CACjC,QAAQ,EAAE,uBAAuB,EACjC,UAAU,EAAE,UAAU,EACtB,MAAM,EAAE,WAAW,CAAC,MAAM,EAAE,cAAc,CAAC,GAC1C,YAAY,CAiCd"}
package/dist/search.js ADDED
@@ -0,0 +1,355 @@
1
+ import { assertTypeInSchema, assertValidQuery, fieldNamed, isRangeFacet, outputFields, physicalFields, referenceFields, } from '@lde/search/adapter';
2
+ import { buildSearchParams, escapeFilterValue, } from './query-compiler.js';
3
+ /**
4
+ * A Typesense-backed {@link SearchEngine}, bound to the whole
5
+ * {@link SearchSchema} at construction — like every other schema consumer.
6
+ * Each type’s collection comes from `options.collections`; the label sidecar
7
+ * and its cache are shared across all types. `search` compiles the query
8
+ * ({@link buildSearchParams}), runs it against the type’s collection,
9
+ * resolves the reference labels for the page of hits from the sidecar
10
+ * `labels` collection in one lookup, and reconstructs the engine-neutral
11
+ * {@link SearchResult} ({@link parseSearchResponse}). `searchFacets` answers
12
+ * a whole facet batch (e.g. a sidebar’s skip-own-filter query variants) as a
13
+ * single `multi_search` round-trip with one shared label lookup; a failed
14
+ * entry is reported in place as a per-query outcome, so it never discards
15
+ * its siblings’ facets. Every engine specific stays here; consumers see only
16
+ * logical documents.
17
+ *
18
+ * With a schema built by `searchSchema` over `defineSearchType` literals,
19
+ * `search()` accepts only the deployment’s own types and returns typo-safe
20
+ * facet/document keys per call — no caller-side generics.
21
+ */
22
+ export function createTypesenseSearchEngine(client, schema, options) {
23
+ // Resolve every type's collection ONCE at construction, so a
24
+ // misconfiguration fails at startup — never on the first search that
25
+ // happens to hit the unwired type.
26
+ const collections = new Map([...schema.values()].map((searchType) => {
27
+ const collection = options.collections[searchType.name];
28
+ if (collection === undefined) {
29
+ throw new Error(`No collection configured for search type “${searchType.name}”; set options.collections.${searchType.name}.`);
30
+ }
31
+ return [searchType.type, collection];
32
+ }));
33
+ // Process-lifetime cache for the FULL `labels` collection, held in the engine
34
+ // closure. Populated lazily on the first cached search; `loadAll` is the
35
+ // single-flight in-flight promise so concurrent first-loads share one export.
36
+ let cachedLabels;
37
+ let cacheExpiresAt = 0;
38
+ let inFlightLoad;
39
+ function cachedAllLabels(labelsCollection, ttlMs) {
40
+ if (cachedLabels !== undefined && Date.now() < cacheExpiresAt) {
41
+ return Promise.resolve(cachedLabels);
42
+ }
43
+ // Single-flight: a load already running serves every concurrent caller.
44
+ inFlightLoad ??= loadAllLabels(client, labelsCollection)
45
+ .then((loaded) => {
46
+ cachedLabels = loaded;
47
+ cacheExpiresAt = Date.now() + ttlMs;
48
+ return loaded;
49
+ })
50
+ // A failed load degrades to id-only references and is NOT cached, so the
51
+ // next search retries rather than serving an empty map for the whole TTL.
52
+ .catch((error) => {
53
+ options.onLabelError?.(error);
54
+ return new Map();
55
+ })
56
+ .finally(() => {
57
+ inFlightLoad = undefined;
58
+ });
59
+ return inFlightLoad;
60
+ }
61
+ // Cached path: the once-loaded full collection serves labels by in-memory
62
+ // lookup (no per-search round-trip). The load does not depend on the
63
+ // response, so it is started BEFORE awaiting the search and runs alongside
64
+ // it; it never rejects (a failed load degrades to an empty map), so it
65
+ // cannot leave an unhandled rejection behind if the search itself fails.
66
+ // `undefined` when the cache (or the sidecar) is not configured.
67
+ function startCachedLabels() {
68
+ return options.labelsCollection !== undefined &&
69
+ options.labelCacheTtlMs !== undefined
70
+ ? cachedAllLabels(options.labelsCollection, options.labelCacheTtlMs)
71
+ : undefined;
72
+ }
73
+ // Labels are supplementary: a failed lookup (e.g. the sidecar collection
74
+ // mid-rebuild) degrades to id-only references rather than failing the whole
75
+ // search, so the listing still renders with bare IRIs. `iris` is a thunk so
76
+ // the cached and no-sidecar paths never pay for collecting them.
77
+ async function resolveLabels(cachedLabelsPromise, iris) {
78
+ if (cachedLabelsPromise !== undefined) {
79
+ return cachedLabelsPromise;
80
+ }
81
+ if (options.labelsCollection === undefined) {
82
+ return new Map();
83
+ }
84
+ try {
85
+ return await fetchLabels(client, options.labelsCollection, iris());
86
+ }
87
+ catch (error) {
88
+ options.onLabelError?.(error);
89
+ return new Map();
90
+ }
91
+ }
92
+ const engine = {
93
+ schema,
94
+ async search(searchType, query) {
95
+ // The port contract: a type outside the bound schema and a structurally
96
+ // invalid query (unknown field, wrong operator, unknown facet) are both
97
+ // rejected up front, for EVERY caller.
98
+ assertTypeInSchema(schema, searchType);
99
+ assertValidQuery(query, searchType);
100
+ const params = buildSearchParams(query, searchType, options);
101
+ const cachedLabelsPromise = startCachedLabels();
102
+ const response = (await client
103
+ .collections(collections.get(searchType.type))
104
+ .documents()
105
+ .search(params));
106
+ const labels = await resolveLabels(cachedLabelsPromise, () => referenceIris(response, searchType));
107
+ return parseSearchResponse(response, searchType, labels);
108
+ },
109
+ async searchFacets(searchType, queries) {
110
+ assertTypeInSchema(schema, searchType);
111
+ for (const query of queries) {
112
+ assertValidQuery(query, searchType);
113
+ }
114
+ if (queries.length === 0) {
115
+ return [];
116
+ }
117
+ const collection = collections.get(searchType.type);
118
+ const cachedLabelsPromise = startCachedLabels();
119
+ // The whole batch travels as ONE multi_search round-trip. Each query
120
+ // compiles as facet-only regardless of what it carries: no hits
121
+ // (per_page 0) and no ordering – nothing is transferred or sorted that
122
+ // this method cannot return.
123
+ const { results } = (await client.multiSearch.perform({
124
+ searches: queries.map((query) => ({
125
+ collection,
126
+ ...buildSearchParams({ ...query, orderBy: [], limit: 0, offset: 0 }, searchType, options),
127
+ })),
128
+ }));
129
+ // One label lookup serves every successful facet result in the batch.
130
+ const responses = results.filter((result) => !('error' in result));
131
+ const labels = await resolveLabels(cachedLabelsPromise, () => [
132
+ ...new Set(responses.flatMap((response) => referenceIris(response, searchType))),
133
+ ]);
134
+ // multi_search reports a failed entry inline instead of rejecting the
135
+ // call; pass that through as a per-query outcome – naming the facets,
136
+ // not the position, since the batch order is the caller's internal –
137
+ // so one failed query never discards its siblings' facets.
138
+ return results.map((result, index) => 'error' in result
139
+ ? {
140
+ error: new Error(`Typesense facet search for “${queries[index].facets.join('”, “')}” failed${result.code !== undefined ? ` (${result.code})` : ''}: ${result.error}`),
141
+ }
142
+ : { facets: parseSearchResponse(result, searchType, labels).facets });
143
+ },
144
+ };
145
+ // The runtime object is string-keyed; the literal-schema typing narrows it.
146
+ return engine;
147
+ }
148
+ /**
149
+ * Load the FULL `labels` collection into a label map via the documents export
150
+ * endpoint, which streams every document as JSONL (one JSON object per line).
151
+ * Each line is reconstructed by {@link labelToLocalizedValue}, exactly as the
152
+ * per-search {@link fetchLabels} path does for its `multi_search` hits.
153
+ */
154
+ async function loadAllLabels(client, collection) {
155
+ const jsonl = await client.collections(collection).documents().export();
156
+ const labels = new Map();
157
+ for (const line of jsonl.split('\n')) {
158
+ if (line.length === 0) {
159
+ continue;
160
+ }
161
+ const document = JSON.parse(line);
162
+ labels.set(String(document.id), labelToLocalizedValue(document));
163
+ }
164
+ return labels;
165
+ }
166
+ /** Every distinct reference IRI whose label the result will actually use. */
167
+ function referenceIris(response, searchType) {
168
+ const referenceFieldSet = new Set(referenceFields(searchType).map((field) => field.name));
169
+ // Hits only carry labels for OUTPUT reference fields: reconstructDocument skips
170
+ // non-output fields, so resolving a non-output reference's hit labels (e.g. a
171
+ // facet-only `class` with dozens of IRIs per hit) is pure waste.
172
+ const outputReferenceFields = referenceFields(searchType)
173
+ .filter((field) => field.output === true)
174
+ .map((field) => field.name);
175
+ const iris = new Set();
176
+ for (const hit of response.hits ?? []) {
177
+ for (const name of outputReferenceFields) {
178
+ const raw = hit.document[name];
179
+ if (Array.isArray(raw)) {
180
+ for (const value of raw) {
181
+ iris.add(String(value));
182
+ }
183
+ }
184
+ else if (typeof raw === 'string') {
185
+ iris.add(raw);
186
+ }
187
+ }
188
+ }
189
+ // Reference-facet bucket values are IRIs too (incl. facet-only references like
190
+ // `class`); resolve them in the same lookup.
191
+ for (const facet of response.facet_counts ?? []) {
192
+ if (referenceFieldSet.has(facet.field_name)) {
193
+ for (const bucket of facet.counts) {
194
+ iris.add(bucket.value);
195
+ }
196
+ }
197
+ }
198
+ return [...iris];
199
+ }
200
+ /**
201
+ * Resolve labels for `iris` from the sidecar `labels` collection. Each
202
+ * `label_${locale}` becomes a language-map entry; the default `label` is the
203
+ * untagged (`und`) fallback when no locale variant exists.
204
+ *
205
+ * Sent as one `multi_search` (POST) call, the id-list split over per-search
206
+ * batches: the id-list of a page or facet carrying many references — e.g. a
207
+ * dataset with dozens of classes — would overflow Typesense’s GET query-string
208
+ * limit (4000 chars, and IRIs URL-encode to several times their length) if it
209
+ * travelled in the URL. POST puts it in the body; each batch stays under
210
+ * Typesense’s `per_page` cap, and bundling the batches keeps it one round-trip
211
+ * regardless of IRI count. Exported for unit testing against a fake client.
212
+ */
213
+ export async function fetchLabels(client, collection, iris) {
214
+ const labels = new Map();
215
+ if (iris.length === 0) {
216
+ return labels;
217
+ }
218
+ const searches = [];
219
+ for (let start = 0; start < iris.length; start += LABEL_BATCH_SIZE) {
220
+ const batch = iris.slice(start, start + LABEL_BATCH_SIZE);
221
+ searches.push({
222
+ collection,
223
+ q: '*',
224
+ query_by: 'label',
225
+ filter_by: `id:[${batch.map(escapeFilterValue).join(',')}]`,
226
+ per_page: batch.length,
227
+ });
228
+ }
229
+ const { results } = (await client.multiSearch.perform({ searches }));
230
+ for (const result of results) {
231
+ // multi_search reports a failed entry inline instead of rejecting; throw
232
+ // so the caller's degradation path (onLabelError, id-only references)
233
+ // engages rather than mistaking the failure for absent labels.
234
+ if ('error' in result) {
235
+ throw new Error(`Typesense label lookup failed${result.code !== undefined ? ` (${result.code})` : ''}: ${result.error}`);
236
+ }
237
+ for (const hit of result.hits ?? []) {
238
+ labels.set(String(hit.document.id), labelToLocalizedValue(hit.document));
239
+ }
240
+ }
241
+ return labels;
242
+ }
243
+ /** Typesense caps `per_page` at 250; the multi_search POST body holds the
244
+ * id-list comfortably, so resolve references in batches of this size. */
245
+ const LABEL_BATCH_SIZE = 200;
246
+ /** Turn a `labels` document into a language map (`label_${locale}` → locale). */
247
+ function labelToLocalizedValue(document) {
248
+ const map = {};
249
+ for (const [key, value] of Object.entries(document)) {
250
+ if (key.startsWith('label_') && typeof value === 'string') {
251
+ map[key.slice('label_'.length)] = [value];
252
+ }
253
+ }
254
+ if (Object.keys(map).length === 0 && typeof document.label === 'string') {
255
+ map.und = [document.label];
256
+ }
257
+ return map;
258
+ }
259
+ /**
260
+ * Reconstruct a Typesense response into the engine-neutral {@link SearchResult}:
261
+ * the flat, fanned-out document is turned back into a logical one (per-locale
262
+ * display fields → a language map, reference IRIs → labelled references via the
263
+ * sidecar `labels` lookup, scalars passed through). `labels` maps a reference IRI
264
+ * to its resolved label; an IRI absent from it yields an id-only reference.
265
+ */
266
+ export function parseSearchResponse(response, searchType, labels) {
267
+ const hits = (response.hits ?? []).map((hit) => ({
268
+ id: String(hit.document.id),
269
+ document: reconstructDocument(hit.document, searchType, labels),
270
+ }));
271
+ // Reference facets are IRI-keyed; their buckets carry a resolved data label.
272
+ // Plain facets (tokens, free strings) carry no label — the consumer owns display.
273
+ const referenceFacets = new Set(referenceFields(searchType).map((field) => field.name));
274
+ const facets = {};
275
+ for (const facet of response.facet_counts ?? []) {
276
+ const labelled = referenceFacets.has(facet.field_name);
277
+ // A range facet echoes the declared range key as the bucket value; look the
278
+ // bin's half-open bounds back up by key so the bucket is self-describing.
279
+ const field = fieldNamed(searchType, facet.field_name);
280
+ const rangesByKey = field !== undefined && isRangeFacet(field)
281
+ ? new Map(field.facetRanges.map((range) => [range.key, range]))
282
+ : undefined;
283
+ facets[facet.field_name] = facet.counts.map((bucket) => {
284
+ const label = labelled ? labels.get(bucket.value) : undefined;
285
+ const range = rangesByKey?.get(bucket.value);
286
+ return {
287
+ value: bucket.value,
288
+ count: bucket.count,
289
+ ...(label !== undefined ? { label } : {}),
290
+ ...(range?.min !== undefined ? { min: range.min } : {}),
291
+ ...(range?.max !== undefined ? { max: range.max } : {}),
292
+ };
293
+ });
294
+ }
295
+ return { hits, total: response.found, facets };
296
+ }
297
+ /** Rebuild one logical document from a flat Typesense document. */
298
+ function reconstructDocument(flat, searchType, labels) {
299
+ const document = {};
300
+ for (const field of outputFields(searchType)) {
301
+ const value = logicalValue(flat, field, labels);
302
+ if (value !== undefined) {
303
+ document[field.name] = value;
304
+ }
305
+ }
306
+ return document;
307
+ }
308
+ function logicalValue(flat, field, labels) {
309
+ switch (field.kind) {
310
+ case 'text':
311
+ return localizedValue(flat, field);
312
+ case 'reference':
313
+ return referenceValue(flat, field, labels);
314
+ case 'keyword': {
315
+ const value = flat[field.name];
316
+ return Array.isArray(value) || typeof value === 'string'
317
+ ? value
318
+ : undefined;
319
+ }
320
+ case 'integer':
321
+ case 'number':
322
+ case 'date': {
323
+ const value = flat[field.name];
324
+ return typeof value === 'number' ? value : undefined;
325
+ }
326
+ case 'boolean':
327
+ // A boolean is always present; an absent value means false.
328
+ return flat[field.name] === true;
329
+ }
330
+ }
331
+ /** Gather the per-locale display fields back into a language map. */
332
+ function localizedValue(flat, field) {
333
+ const map = {};
334
+ const display = physicalFields(field).display;
335
+ field.locales.forEach((locale, index) => {
336
+ const value = flat[display[index]];
337
+ if (typeof value === 'string') {
338
+ map[locale] = [value];
339
+ }
340
+ });
341
+ return Object.keys(map).length > 0 ? map : undefined;
342
+ }
343
+ /** Map stored reference IRIs to labelled references; id-only when no label. */
344
+ function referenceValue(flat, field, labels) {
345
+ const raw = flat[field.name];
346
+ if (raw === undefined) {
347
+ return undefined;
348
+ }
349
+ const iris = Array.isArray(raw) ? raw : [String(raw)];
350
+ const references = iris.map((iri) => {
351
+ const label = labels.get(iri);
352
+ return label === undefined ? { id: iri } : { id: iri, label };
353
+ });
354
+ return field.array === true ? references : references[0];
355
+ }
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "@lde/search-typesense",
3
- "version": "0.1.1",
4
- "description": "Generic Typesense engine adapter for RDF-backed search pipelines: collection lifecycle, bulk upsert and blue/green alias swap",
3
+ "version": "0.3.0",
4
+ "description": "Typesense implementation of the @lde/search SearchEngine port: collection-schema builder, query compiler, label-resolving result reconstruction, and blue/green index lifecycle. Engine-specific (Typesense) but domain-agnostic.",
5
5
  "repository": {
6
6
  "url": "git+https://github.com/ldelements/lde.git",
7
7
  "directory": "packages/search-typesense"
@@ -25,6 +25,8 @@
25
25
  "!**/*.tsbuildinfo"
26
26
  ],
27
27
  "dependencies": {
28
+ "@lde/search": "^0.3.0",
29
+ "@lde/text-normalization": "^0.1.1",
28
30
  "tslib": "^2.3.0",
29
31
  "typesense": "^3.0.6"
30
32
  },