@leu2m/semantic-search 0.2.0-beta.1

package/docs/integration.md

@@ -0,0 +1,204 @@
+# Host integration
+
+## Install as a versioned dependency
+
+Use a pinned Git commit or the tarball produced by `npm pack` until a registry release exists. Git installs run `prepare` to build declarations and JavaScript. Do not import another checkout's `src/` files or use product-specific aliases inside this package. A future npm release uses the same package name and exports.
+
+Both inspected consumers can use ESM: Vector bundles dependencies with esbuild, while Nexus's ESM packages/daemon import the core directly. This work does not modify either consumer. Current package-install checks exercise native ESM and installed declarations for all three public exports; core also executes in a browser-like realm and, in Alpha.5, an actual Chrome page/module worker from the installed tarball. The initial foundation separately verified an esbuild CommonJS consumer bundle.
+
+## Obsidian vault (host-owned adapter)
+
+The following is a conceptual host-side adapter. The host must apply its own hidden/protected-path policy on **both** list and read, and resolve the active editor versus saved content according to product semantics.
+
+```ts
+import type { SearchSource } from '@leu2m/semantic-search';
+import { SemanticSearchEngine, isSupportedFile } from '@leu2m/semantic-search';
+
+const vaultSource: SearchSource = {
+  id: configuredVaultId,
+  type: 'obsidian',
+  async list() {
+    return app.vault.getFiles()
+      .filter(file => isAllowed(file.path) && isSupportedFile(file.path))
+      .map(file => ({
+        id: file.path, sourceId: configuredVaultId,
+        uri: makeObsidianUri(file.path), path: file.path, name: file.name,
+        modifiedAt: file.stat.mtime,
+        metadata: { tags: hostTagsFor(file) },
+      }));
+  },
+  async read(id) {
+    if (!isAllowed(id)) throw new Error('Source access denied');
+    const file = app.vault.getFileByPath(id);
+    if (!file) throw new Error('Document unavailable');
+    return { text: await app.vault.read(file) };
+  },
+};
+const engine = new SemanticSearchEngine();
+engine.addSource(vaultSource);
+await engine.initialize();
+```
+
+On vault create/modify/rename/delete events, debounce and serialize `refreshSource(configuredVaultId)`. Dispose host listeners when the plugin unloads. This foundation rescans; event-level incremental scheduling is a later slice. A failed source scan retains previous results; a host that cannot permit stale evidence should remove that source or refuse searches until refresh succeeds. `SearchResult` has enough line/path/URI information for a host to re-read exact evidence before risky edits.
+
+## Filesystem folders (desktop or daemon)
+
+```ts
+import { FileSystemFolderSource } from '@leu2m/semantic-search/filesystem';
+engine.addSource(new FileSystemFolderSource({
+  id: 'approved-project', root: configuration.approvedAbsoluteRoot,
+  maxFileBytes: 512 * 1024,
+}));
+await engine.refreshSource('approved-project');
+const evidence = await engine.search({ query: 'validateSession', sources: ['approved-project'] });
+```
+
+The root must be explicitly approved/configured by the host. Do not turn an agent's arbitrary path argument directly into a registered source. Symlinks are not followed. No filesystem watcher is created and no external folder is inferred. Mobile/browser hosts should use their own source APIs and not import the Node adapter.
+
+## Custom sources and parsers
+
+Implement `SearchSource` for any explicitly configured document collection. IDs are unique within a source and every listed document must carry that source's ID. Add a `DocumentParser` to a `ParserRegistry`; for extra text formats, pass an extended `FileTypeRegistry` to both the parser registry and the filesystem adapter. Use `new SemanticSearchEngine({ parsers })`. The shared registry provides parser selection and admission policy rather than competing extension arrays. Unknown files are rejected unless explicitly registered.
+
+## Optional storage and embedding
+
+```ts
+import { SemanticSearchEngine, type Embedder, type IndexStorage } from '@leu2m/semantic-search';
+
+// Host-provided implementations; alternatively use the optional MiniLMEmbedder below.
+const embedder: Embedder = myWorkerEmbedder;
+const storage: IndexStorage = myTransactionalStorage;
+const engine = new SemanticSearchEngine({ embedder, storage });
+engine.addSource(mySource);
+await engine.initialize({ signal: abortController.signal });
+```
+
+A browser app can supply a browser source and IndexedDB-backed storage; a daemon can supply filesystem sources and its own store. The core imports neither. Omit either option independently. Storage alone persists lexical records. An embedder alone reuses vectors in the current engine. Both enable compatible restart reuse. With an embedder, omitted mode now means hybrid retrieval; without one it means lexical. Storage is independent of that default. Explicit lexical mode requires no query inference and remains usable after a model has been disposed, using the last good index. Semantic/hybrid require a working embedder and fail on model errors rather than silently falling back.
+
+```ts
+const exact = await engine.search({ query: 'validateSession', mode: 'lexical' });
+const conceptual = await engine.search({ query: 'verify the caller identity', mode: 'semantic', sources: ['approved-project'] });
+const combined = await engine.search({ query: 'verify the caller identity', mode: 'hybrid', filters: { extensions: ['ts'] } });
+```
+
+This is a deliberate Alpha.3-to-Alpha.4 default change. Set `mode: 'lexical'` where existing consumers require lexical results or zero query-model latency. Keep the embedder alive through semantic/hybrid searches; the engine never disposes a shared host adapter. Any finite Float32 embedder dimensionality is supported, but query and candidate vectors must have nonzero norm for cosine. All candidate vectors must exist and match the engine configuration; inconsistency surfaces as an error.
+
+Source/filter restrictions apply before both channels, and no eligible chunks means no inference. Results retain full provenance and one-based ranks, not confidence. RRF can retain noisy lexical matches; use evaluation to choose a mode for a workload. Exact search scales linearly with vector count/dimensions plus sorting; curate sources and filter scope before assuming a large-corpus latency budget.
+
+Use `createSearchSession(engine)` per run to cache completed equivalent searches. It resolves default mode and includes mode/filters/sources/limit in keys. Semantic query text is case/whitespace-sensitive in the cache. Calls overlapping refresh see their captured old index and are not cached under the new revision. Equivalent unordered source/tag/extension sets share cache entries. `clear()` prevents earlier pending calls from repopulating the cache without cancelling their callers. Re-read cited material before mutations when current source content matters.
+
+For Node/CLI/desktop consumers, a concrete atomic store is available from the existing Node-only export:
+
+```ts
+import { FileIndexStorage, FileSystemFolderSource } from '@leu2m/semantic-search/filesystem';
+const engine = new SemanticSearchEngine({
+  storage: new FileIndexStorage(configuration.absoluteIndexFilename),
+  // embedder: myEmbedder, // optional, including MiniLMEmbedder from /minilm
+});
+engine.addSource(new FileSystemFolderSource({ id: 'folder', root: configuration.approvedAbsoluteRoot }));
+await engine.initialize();
+```
+
+The host creates the index parent directory and chooses an index location outside its indexed source roots. Use one writer per store. Save failures surface and preserve the last good live source state. Missing files mean an empty store; corrupt JSON/compatible snapshots produce errors. Hosts choose recovery by repairing/removing that store or selecting a fresh one. The package never chooses a product-specific cache path.
+
+Loaded snapshots are reconciled with current sources before becoming searchable. Unchanged exact content—including renames—reuses document vectors. Changed/new content alone requires embedding. Deletes leave live and persisted state on successful refresh. Initialization is per-source atomic; an early successful save during multi-source startup includes only reconciled live sources. A later failure keeps earlier successes and may leave other sources needing re-indexing after restart.
+
+Removing a source clears it from memory immediately; the next successful source refresh saves that removal. This synchronous API does not promise an immediate durable write. To persist a document deletion, refresh its still-registered source, including when its listing is empty.
+
+### Version custom parsers and embedding configurations
+
+```ts
+import { ParserRegistry } from '@leu2m/semantic-search';
+const parsers = new ParserRegistry(myParsers, myFileTypes, 'my-parser-behavior-v1');
+const engine = new SemanticSearchEngine({ parsers, storage, embedder });
+```
+
+The version describes all custom parser behavior. Increment it when that behavior changes. Without an explicit version, custom registries still work, but persisted reuse is disabled. Registry definitions, chunker version and effective chunking options are also checked automatically. Incompatible snapshots rebuild wholesale. Keep parser and embedder configurations immutable; a changed embedder identity/dimension requires a new engine. Its ID must include model revision and vector-affecting settings, not just a friendly model name.
+
+`IndexStorage.save` must atomically publish and reject failure/abort before publication. Once it resolves successfully, late cancellation cannot prevent the matching live commit. See [architecture](architecture.md) for the exact failure and cancellation policy. MiniLM is an optional adapter; hosts control its execution and caches.
+
+## Optional MiniLM: online or cached Hub loading
+
+Install `@huggingface/transformers@4.2.0` alongside the package only when using `/minilm`. Built tarball installs for core-only usage omit the optional peer. Git/source builds install development dependencies for compilation, including Transformers.js.
+
+```ts
+import { MiniLMEmbedder } from '@leu2m/semantic-search/minilm';
+const embedder = new MiniLMEmbedder({
+  device: 'cpu', dtype: 'fp32', batchSize: 4,
+  cacheDir: configuration.modelCacheDirectory,
+  onProgress: event => showModelLoadProgress(event),
+});
+const engine = new SemanticSearchEngine({ embedder, storage });
+engine.addSource(mySource);
+await engine.initialize();
+// Later, after every engine/consumer has finished with this adapter:
+await embedder.dispose();
+```
+
+No runtime/model load occurs on import or construction. The first nonempty call loads once; subsequent calls reuse the pipeline. The host controls the batch size and execution location. Empty document batches return no vectors without loading. Disposal is terminal; later calls reject. Concurrent calls serialize; cancellation is checked around expensive stages but does not interrupt native work already running.
+
+The default model is `Xenova/all-MiniLM-L6-v2` at `751bff37182d3f1213fa05d7196b954e230abad9`, using 384-dimensional normalized mean-pooled embeddings. Custom model IDs/paths require an explicit revision and compatible 384-dimensional output. Model/revision, dtype/device, runtime/adapter behavior, pooling/normalization and tokenizer policy participate in identity. Batch size, cache location and `localFilesOnly` do not. Mutable revisions such as `main` weaken reproducibility. The default tokenizer truncates beyond 512 tokens.
+
+`cacheDir` is a cache location, not an offline promise. `localFilesOnly` forwards `local_files_only`; Transformers.js 4.2.0 tokenizer metadata discovery does not forward that option, revision or cache location on every request. Thus a Hub ID may still trigger remote metadata discovery even with populated cached files and `localFilesOnly: true`. Do not use that flag alone to promise zero network access.
+
+## Strict offline: host-staged assets
+
+The host provisions model assets before entering offline mode and ensures their bytes match the declared immutable revision. For the default CPU/fp32 deployment, stage this layout beneath a host-selected base directory:
+
+```text
+<localModelPath>/
+└── Xenova/all-MiniLM-L6-v2/
+    ├── config.json
+    ├── tokenizer.json
+    ├── tokenizer_config.json
+    └── onnx/model.onnx
+```
+
+Other dtype choices require their corresponding ONNX assets. The host may copy the verified pinned cache assets into this deployment layout as a provisioning step; the adapter never copies or downloads staged files itself. A populated revision-specific Hub cache has a different role and is not sufficient by itself for strict offline Hub-ID resolution in 4.2.0.
+
+```ts
+// HOST bootstrap, before constructing/using any models in this runtime realm.
+import { env } from '@huggingface/transformers';
+import { MiniLMEmbedder, MINILM_DEFAULT_REVISION } from '@leu2m/semantic-search/minilm';
+
+env.localModelPath = configuration.stagedModelBaseWithTrailingSlash;
+env.allowLocalModels = true;
+env.allowRemoteModels = false;
+
+const embedder = new MiniLMEmbedder({
+  revision: MINILM_DEFAULT_REVISION,
+  device: 'cpu', dtype: 'fp32', localFilesOnly: true,
+});
+const engine = new SemanticSearchEngine({ embedder, storage });
+engine.addSource(mySource);
+await engine.initialize();
+```
+
+These global settings belong to the host and affect other Transformers.js users in the same realm. The adapter reads the selected runtime but never changes its environment. Configure globals before concurrent users start; use separate host-managed workers/processes when different policies need isolation. Local asset resolution does not enforce the Hub revision: hosts must bind staged bytes to the declared revision and change identity when those bytes change. Keeping the same model/revision and settings with the exact same staged bytes preserves vector compatibility with Hub loading.
+
+The CPU acceptance run disables model caches and remote loading and traps fetch calls; staged inference succeeds with zero network attempts. Missing staged assets must fail instead of downloading replacements.
+
+## Browser and Web Worker hosts
+
+Inside a host-created worker, import only core and `/minilm`, install/bundle the optional runtime, and choose `device: 'wasm'` or supported `webgpu` explicitly. The default `cpu` device targets Node. The adapter creates no worker, worker thread or process. The host supplies sources and storage adapters and transfers results as appropriate.
+
+For browser offline deployment, the host stages and serves/caches model files, the JavaScript runtime and required WASM/backend assets. Configure `env.localModelPath`, `env.allowRemoteModels`, and `env.backends.onnx.wasm.wasmPaths` as required by that deployment. Local browser URLs may still involve fetch and require a functioning local/offline serving or service-worker strategy. The Node zero-fetch acceptance does not certify a browser asset-serving arrangement; real browser/backend inference must be validated by the host. Core and the adapter source remain free of Node filesystem/process imports.
+
+The accepted [loading ADR](decisions/0001-minilm-loading.md) and [evaluation matrix](evaluation.md) distinguish supported inference paths from the known Hub-cache-only limitation.
+
+## Tested standalone examples and deployment limits
+
+Start with [maintained examples](../examples/README.md): `core.mjs` exercises lexical-only usage and custom source/storage/embedder wiring; `filesystem.mjs` uses an explicit approved root and store path; `minilm.mjs` owns model disposal. These are public-export consumer code, not imports of internals. [API responsibilities](api.md) documents failures, mutability, source registration, storage single-writer rules and worker placement for each contract.
+
+Alpha.5's explicit browser harness bundles the installed core example into a Chrome page and module worker, with no Node globals or Transformers peer. Its storage is memory-backed to prove the contract; it is not a durable IndexedDB implementation. The separate MiniLM worker test uses `device: 'wasm'`, fp32, `env.backends.onnx.wasm.numThreads = 1`, host-served backend assets and local model URLs, with remote models disabled and external requests blocked. It validates embeddings, semantic retrieval and disposal. All these settings are host test bootstrap, never adapter/core side effects. No WebGPU or cross-browser certification is claimed.
+
+The browser test serves assets over loopback HTTP. Zero external requests differs from zero browser fetches: local URLs still need a server or host-managed offline cache. Node staged-local acceptance retains its stronger zero-fetch observation. Hub cache-only offline remains unsupported under Transformers.js 4.2.0; `localFilesOnly` is not a guarantee.
+
+Before broader deployment, choose a retrieval mode for the workload and treat candidates as unverified evidence. Alpha.5 found high-similarity near misses and noisy lexical interference; unchanged hybrid defaults are a compromise. Exact wide-scope search becomes expensive at tens of thousands of candidates on the measured laptop. Filters and host-controlled workers help scope cost; no threshold, ANN, mode router or new host adapter is included.
+
+## Consuming evidence without inventing an answerability claim
+
+The Alpha.6 [host example](../examples/evidence.mjs) returns candidate retrieval observations with `answerability: 'not-assessed'`. Copy/adapt it as consumer code; it is not another package export. `retrieved` and `no-results` describe only the returned list. No-result cause is not inferred, and unavailable models, malformed vectors or aborts remain rejected operations rather than empty success. A request spanning refresh may return a coherent older view; the example reports revision overlap but cannot detect source edits that were never indexed.
+
+Hosts deciding to act on a passage must establish whether it contains the requested fact/task evidence, whether multiple passages are required, and whether it is current and authoritative. A procedure for collecting a measurement cannot substitute for that measurement. High rank, similarity or lexical/semantic agreement is not an assessment. The library does not require a particular human/rules/model verification method or define an evaluator ABI. If a host adds assessment, failures/unavailable/partial/stale evaluation must remain distinct from a justified finding that the *assessed set* is insufficient. Do not store those decisions in the retrieval cache as though they were candidate results.
+
+Runtime/API compatibility is unchanged. Keep current mode choices, generic embedder/storage/source adapters and optional MiniLM settings. See [the decision and evidence](alpha6-answerability.md); independent Alpha.6C will review this boundary before a release-gate recommendation.

package/docs/next-slice.md

@@ -0,0 +1,11 @@
+# Next: independent Alpha.6C adversarial review
+
+Alpha.6A+B is complete. It selects retrieval-only core, clarifies the public contract, adds a tested unassessed-evidence host example and development chunk/set evaluation. Production ranking/API remains unchanged and package version stays `0.1.0-alpha.5`. No Beta promotion is made.
+
+The next step is **Alpha.6C by an independent model**, not further implementation by the current author. Start with [the design record](alpha6-answerability.md), [ADR 0003](decisions/0003-retrieval-evidence-boundary.md), [traces](alpha6-evidence-traces.md), [evaluation](evaluation.md) and the ignored `local/alpha6-c-handoff.md` in the working checkout. The handoff is intentionally not part of Git or the package.
+
+C must challenge the relevance/sufficiency judgments and task interpretations, construct/review held-out adversarial cases, inspect the tiny B source/comment/example diff, and decide whether this boundary is useful to unrelated consumers. Do not trust labels just because they are documented. Prioritize topic-versus-fact near misses, dates/measurements, contradictory evidence, multi-chunk requirements, incomplete code, filters and stale sources. Explicitly revisit subjective locator/ambiguous interpretations and whether publication/backup requirements were added too broadly.
+
+The development run preserves every frozen Alpha.5 metric. Sufficient-set hit@5 is .6286/.8857/.8857 (lexical/semantic/hybrid) on 35 supported tasks; ten unsupported queries still receive results, and five are underspecified. These figures validate neither calibrated answerability nor independent retrieval quality. RRF/defaults remain unchanged. C owns the independent release gate and may recommend keeping Alpha; Beta is not preauthorized by A+B.
+
+Do not start ANN, reranking, automatic routing, a score/evaluator API, consumer integration, Vector/Nexus/Obsidian work or another campaign as part of C without a separate decision and authorization.

package/examples/README.md

@@ -0,0 +1,38 @@
+# Maintained examples
+
+These files use public package exports. They are included in the tarball as readable examples, not new package subpath exports. Copy/adapt them into a consumer or run the checkout after `npm run build`.
+
+```sh
+node --input-type=module -e "import { exerciseCore } from './examples/core.mjs'; console.log(await exerciseCore());"
+```
+
+- `evidence.mjs`: `retrieveUnassessedEvidence(engine, request)` preserves candidate evidence and reports `retrieved`/`no-results`, always with `answerability: 'not-assessed'`. Errors/abort reject; `indexChangedDuringSearch` reports only observed index revision overlap, never source freshness. This envelope is an example-local choice, not a public package contract.
+- `core.mjs`: lexical-only indexing; custom SearchSource, memory-only IndexStorage, toy injected Embedder; semantic/hybrid retrieval, cache and restart checks. The two-topic vectors demonstrate plumbing and are not a production semantic model.
+- `filesystem.mjs`: `searchFolder(approvedRoot, indexFilename, query)` uses explicit host paths and atomic Node storage. Create the index parent first; keep it outside indexed roots.
+- `minilm.mjs`: `searchWithMiniLM(source, storage, query, options)` manages one concrete model lifetime. First inference may download files. Strict local deployments configure Transformers.js globals in host bootstrap before using this function; see integration docs.
+
+The installed-tarball Node/browser test executes `exerciseCore` and `searchFolder`. The ordinary package test loads the MiniLM example with an empty source without loading a runtime; real Node and browser MiniLM acceptance separately prove actual inference. The browser harness bundles the installed core example into a page and a module worker. Production worker topology remains host-owned.
+
+Browser commands (explicit, separate from offline `npm test`):
+
+```sh
+npx playwright install chromium
+npm run test:browser
+npm run test:browser:minilm
+```
+
+`BROWSER_EXECUTABLE` optionally selects a host browser executable. Browser MiniLM requires provisioned pinned fp32 assets (`MINILM_LOCAL_MODEL_DIR` or the documented `.cache/minilm` fixture), serves them and backend files from a temporary loopback server, selects WASM with one thread, and forbids external requests. It downloads no model itself. A successful local-server smoke is not certification of every browser, WebGPU, service-worker cache or offline PWA deployment.
+
+## Evidence interpretation example
+
+```js
+import { retrieveUnassessedEvidence } from './evidence.mjs'; // Copy/adapt the maintained host example.
+const observation = await retrieveUnassessedEvidence(engine, {
+  query: 'What was the measured duration of the latest recovery drill?',
+  mode: 'hybrid',
+});
+// observation.candidates are ranked passages, possibly just a drill procedure.
+// observation.answerability is always 'not-assessed', including when candidates is [].
+```
+
+Do not convert `no-results` to “the corpus cannot answer”: lexical mismatches, empty queries or filters can also produce it. Do not convert nonempty results to “supported”: instructions to measure are not a measured value. A host choosing to verify support must separately define the task, eligible scope, evidence completeness, source currency/authority and treatment of contradictions. A failed assessment of top-K evidence does not prove absence elsewhere. There is no evaluator interface or assessment cache in this example. The core example and installed browser/worker harness exercise this helper; focused tests cover cancellation, model errors and concurrent refresh.

package/examples/core.mjs

@@ -0,0 +1,53 @@
+import { SemanticSearchEngine, createSearchSession } from '@leu2m/semantic-search';
+import { retrieveUnassessedEvidence } from './evidence.mjs';
+/** A minimal host source. Replace its read/list methods with your authorized content APIs. */
+export function memorySource(files, id = 'example') {
+    return { id, type: 'memory',
+        async list(options = {}) { options.signal?.throwIfAborted(); return [...files.keys()].map(path => ({ id: path, sourceId: id, name: path, path, uri: `memory://${id}/${path}` })); },
+        async read(id, options = {}) { options.signal?.throwIfAborted(); if (!files.has(id))
+            throw new Error('Document unavailable'); return { text: files.get(id) }; },
+    };
+}
+/** Test/example storage only: no browser database or durable production adapter is implied. */
+export function memoryStorage() {
+    let snapshot = null;
+    return { async load(options = {}) { options.signal?.throwIfAborted(); return structuredClone(snapshot); },
+        async save(candidate, options = {}) { options.signal?.throwIfAborted(); const next = structuredClone(candidate); snapshot = next; } };
+}
+/** Toy vectors demonstrate injection only. Substitute your real text embedder in applications. */
+export function demonstrationEmbedder() {
+    const stats = { documents: 0, queries: 0 };
+    const encode = text => new Float32Array(/password|credential|login/i.test(text) ? [2, 0] : [0, 3]);
+    return { id: 'example-two-topics-v1', dimensions: 2, stats,
+        async embedDocuments(texts, options = {}) { options.signal?.throwIfAborted(); stats.documents += texts.length; return texts.map(encode); },
+        async embedQuery(text, options = {}) { options.signal?.throwIfAborted(); stats.queries++; return encode(text); } };
+}
+export async function exerciseCore() {
+    const source = memorySource(new Map([['account.txt', 'Recover a lost password'], ['garden.txt', 'Water plants at dawn']]));
+    const lexical = new SemanticSearchEngine();
+    lexical.addSource(source);
+    await lexical.initialize();
+    const evidence = await retrieveUnassessedEvidence(lexical, { query: 'password' });
+    if (evidence.answerability !== 'not-assessed' || evidence.retrieval !== 'retrieved' || evidence.candidates[0]?.path !== 'account.txt')
+        throw new Error('Lexical example failed');
+    const storage = memoryStorage();
+    const embedder = demonstrationEmbedder();
+    for (let restart = 0; restart < 2; restart++) {
+        const engine = new SemanticSearchEngine({ storage, embedder });
+        engine.addSource(source);
+        await engine.initialize();
+        for (const mode of ['semantic', 'hybrid']) {
+            const run = createSearchSession(engine);
+            const request = { query: 'credentials', mode };
+            const hit = (await run.search(request))[0];
+            if (hit?.path !== 'account.txt' || hit.semanticRank !== 1 || !hit.uri || !hit.chunkId)
+                throw new Error('Rank/provenance example failed');
+            await run.search(request);
+            if (run.stats.hits !== 1)
+                throw new Error('Cache example failed');
+        }
+    }
+    if (embedder.stats.documents !== 2 || embedder.stats.queries !== 4)
+        throw new Error('Restart/query reuse example failed');
+    return { lexical: true, semantic: true, hybrid: true, restart: true, ...embedder.stats };
+}

package/examples/evidence.mjs

@@ -0,0 +1,20 @@
+/**
+ * Host-side example, not a new package API or an answerability evaluator.
+ * Empty and nonempty searches both leave support for the task unassessed.
+ * @param {import('@leu2m/semantic-search').SemanticSearchEngine} engine
+ * @param {import('@leu2m/semantic-search').SearchRequest} request
+ */
+export async function retrieveUnassessedEvidence(engine, request) {
+  request.signal?.throwIfAborted();
+  const startedRevision = engine.revision;
+  // Do not convert failure/cancellation into an apparently successful empty search.
+  const candidates = await engine.search(request);
+  request.signal?.throwIfAborted();
+  return {
+    retrieval: candidates.length ? 'retrieved' : 'no-results',
+    answerability: 'not-assessed',
+    candidates,
+    // False is NOT a freshness guarantee: sources can change without an index refresh.
+    indexChangedDuringSearch: engine.revision !== startedRevision,
+  };
+}

package/examples/filesystem.mjs

@@ -0,0 +1,9 @@
+import { SemanticSearchEngine } from '@leu2m/semantic-search';
+import { FileSystemFolderSource, FileIndexStorage } from '@leu2m/semantic-search/filesystem';
+/** Host chooses an approved absolute root and an index filename outside that root. */
+export async function searchFolder(root, indexFilename, query) {
+    const engine = new SemanticSearchEngine({ storage: new FileIndexStorage(indexFilename) });
+    engine.addSource(new FileSystemFolderSource({ id: 'folder', root }));
+    await engine.initialize();
+    return engine.search({ query, mode: 'lexical' });
+}

package/examples/minilm.mjs

@@ -0,0 +1,15 @@
+import { SemanticSearchEngine } from '@leu2m/semantic-search';
+import { MiniLMEmbedder } from '@leu2m/semantic-search/minilm';
+/** Run in the host's chosen process/worker. First inference may download assets. */
+export async function searchWithMiniLM(source, storage, query, options = {}) {
+    const embedder = new MiniLMEmbedder(options);
+    try {
+        const engine = new SemanticSearchEngine({ embedder, storage });
+        engine.addSource(source);
+        await engine.initialize();
+        return await engine.search({ query, mode: 'hybrid' });
+    }
+    finally {
+        await embedder.dispose();
+    }
+}

package/package.json

@@ -0,0 +1,73 @@
+{
+  "name": "@leu2m/semantic-search",
+  "version": "0.2.0-beta.1",
+  "description": "Source-oriented retrieval infrastructure with parsing, chunking and provenance.",
+  "type": "module",
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    },
+    "./filesystem": {
+      "types": "./dist/adapters/filesystem.d.ts",
+      "import": "./dist/adapters/filesystem.js"
+    },
+    "./minilm": {
+      "types": "./dist/adapters/minilm.d.ts",
+      "import": "./dist/adapters/minilm.js"
+    }
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "LICENSE",
+    "docs",
+    "examples"
+  ],
+  "sideEffects": false,
+  "engines": {
+    "node": ">=20"
+  },
+  "license": "MIT",
+  "author": "leu2m",
+  "repository": {
+    "type": "git",
+    "url": "git+ssh://git@github.com/leu2m/semantic-search.git"
+  },
+  "scripts": {
+    "clean": "node scripts/clean.mjs",
+    "build": "npm run clean && tsc -p tsconfig.json",
+    "typecheck": "tsc -p tsconfig.json --noEmit",
+    "test": "npm run build && node tests/run.mjs",
+    "docs:file-types": "npm run build && node scripts/file-types.mjs",
+    "benchmark": "npm run build && node scripts/benchmark.mjs && node scripts/lifecycle-benchmark.mjs && node scripts/retrieval-benchmark.mjs",
+    "prepack": "npm run build && node scripts/file-types.mjs --check",
+    "prepare": "npm run build",
+    "test:minilm": "npm run build && node tests/minilm.acceptance.mjs",
+    "benchmark:minilm": "npm run build && node scripts/minilm-benchmark.mjs",
+    "test:browser": "npm run build && node tests/browser.real.mjs",
+    "test:browser:minilm": "npm run build && node tests/browser.real.mjs --minilm",
+    "evaluate": "npm run build && node scripts/representative-evaluation.mjs",
+    "evaluate:minilm": "npm run build && node scripts/representative-evaluation.mjs --minilm",
+    "benchmark:scale": "npm run build && node scripts/scale-benchmark.mjs",
+    "evaluate:evidence": "npm run build && node scripts/evidence-evaluation.mjs",
+    "evaluate:evidence:minilm": "npm run build && node scripts/evidence-evaluation.mjs --minilm"
+  },
+  "devDependencies": {
+    "@huggingface/transformers": "4.2.0",
+    "@types/node": "^22.15.17",
+    "esbuild": "0.25.5",
+    "playwright": "1.62.1",
+    "typescript": "^5.8.3"
+  },
+  "peerDependencies": {
+    "@huggingface/transformers": "4.2.0"
+  },
+  "peerDependenciesMeta": {
+    "@huggingface/transformers": {
+      "optional": true
+    }
+  }
+}