@graphorin/embedder-transformersjs 0.5.0

package/CHANGELOG.md

@@ -0,0 +1,9 @@
+# @graphorin/embedder-transformersjs
+
+## 0.1.0
+
+### Minor Changes
+
+- Initial release. Default in-process embedder for the Graphorin
+  framework — wraps `@huggingface/transformers` for the multilingual
+  `Xenova/multilingual-e5-base` model.

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Oleksiy Stepurenko
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,84 @@
+# @graphorin/embedder-transformersjs
+
+> Default in-process embedder for the Graphorin framework.
+
+`@graphorin/embedder-transformersjs` wraps `@huggingface/transformers@^4.1.0`
+to produce dense embeddings inside the Graphorin process — no external
+service, no network call after the first model download. The default
+model is `Xenova/multilingual-e5-base` (768-dim, multilingual). Other
+multilingual variants from the same family (`multilingual-e5-small`,
+`multilingual-e5-large`, `bge-m3`) are first-class through the same
+factory.
+
+The package implements the `EmbedderProvider` contract from
+`@graphorin/core/contracts`:
+
+- `id()` — canonical id, e.g.
+  `'transformersjs:Xenova/multilingual-e5-base@768'`.
+- `dim()` — output vector dimensionality (resolved at first `embed`).
+- `configHash()` — deterministic hex hash over canonical-JSON config.
+- `embed(texts)` — batched, lazy. Returns one `Float32Array` per text.
+
+## Install
+
+```bash
+pnpm add @graphorin/embedder-transformersjs @huggingface/transformers
+```
+
+## Quick start
+
+```ts
+import { createTransformersJsEmbedder } from '@graphorin/embedder-transformersjs';
+
+const embedder = createTransformersJsEmbedder({
+  model: 'Xenova/multilingual-e5-base',
+  pooling: 'mean',
+  normalize: true,
+});
+
+const [vec] = await embedder.embed(['Loves espresso.']);
+console.log(embedder.id(), embedder.dim(), vec.length);
+```
+
+## E5 asymmetric prefixes
+
+E5-family models (the default, and any model id carrying an `e5` token) require
+asymmetric `query:` / `passage:` prefixes — omitting them measurably degrades
+retrieval. The embedder applies them automatically: pass `taskType: 'query'`
+when embedding a search query and `taskType: 'passage'` (the default) for stored
+content. The Graphorin memory tiers thread this for you (`query` on search,
+`passage` on `remember`).
+
+> **Migration:** enabling the prefixes (the new default for E5) changes the
+> embedder's `configHash`, and therefore its `id`. Under the default
+> `lock-on-first` policy an existing index built before this change reports an
+> embedder mismatch — run `graphorin memory migrate` to re-embed it, or pass
+> `disableTaskPrefix: true` to keep the old (unprefixed) behaviour and id.
+
+## Cache directory
+
+The Hugging Face model cache lives under `os.homedir()/.cache/...` by
+default. Override with the standard env var or pass it explicitly:
+
+```bash
+GRAPHORIN_CACHE_DIR=/var/lib/graphorin/models pnpm start
+```
+
+## Network behaviour
+
+The embedder is **local-first**: the only outbound call is the first
+model download (cached after success). The
+`scripts/check-no-network.mjs` static guard explicitly allow-lists
+`packages/embedder-*/` for this reason.
+
+If the model cannot be downloaded (offline / corporate firewall), the
+embedder throws an `EmbedderModelLoadError` with an actionable message:
+the Phase 17 docs cover the offline-install procedure.
+
+## License
+
+MIT © 2026 Oleksiy Stepurenko.
+
+---
+
+**Project Graphorin** · v0.5.0 · MIT License · © 2026 Oleksiy Stepurenko · <https://github.com/o-stepper/graphorin>

package/dist/index.d.ts

@@ -0,0 +1,141 @@
+import { EmbedOptions, EmbedderProvider } from "@graphorin/core/contracts";
+
+//#region src/index.d.ts
+
+/** Canonical version constant. Mirrors the `package.json` version. */
+declare const VERSION = "0.5.0";
+/**
+ * Pooling strategy. Defaults to `'mean'` per the multilingual-e5 model
+ * card.
+ *
+ * @stable
+ */
+type Pooling = 'mean' | 'cls' | 'first_token' | 'eos' | 'last_token' | 'none';
+/**
+ * Configuration accepted by {@link createTransformersJsEmbedder}.
+ *
+ * @stable
+ */
+interface TransformersJsEmbedderOptions {
+  /** Default `'Xenova/multilingual-e5-base'` (768-dim). */
+  readonly model?: string;
+  /** Default `'mean'`. */
+  readonly pooling?: Pooling;
+  /** Default `true`. */
+  readonly normalize?: boolean;
+  /** Optional model revision pin (`'main'` if unset). */
+  readonly revision?: string;
+  /**
+   * Optional cache directory. When unset, the embedder honours
+   * `process.env.GRAPHORIN_CACHE_DIR`, otherwise falls back to the
+   * Hugging Face default (`os.homedir()/.cache/huggingface/hub`).
+   */
+  readonly cacheDir?: string;
+  /**
+   * Optional dtype hint (`'fp32' | 'fp16' | 'q8' | 'q4'`). When unset,
+   * the runtime picks the model's recommended default.
+   */
+  readonly dtype?: string;
+  /** Override device (`'cpu'`, `'webgpu'`, …). Default `'cpu'`. */
+  readonly device?: string;
+  /**
+   * Override the underlying `pipeline` factory — used by the test
+   * suite to inject a stub. Production callers should leave this
+   * unset so the package lazily loads `@huggingface/transformers`.
+   */
+  readonly pipelineFactory?: PipelineFactory;
+  /**
+   * Optional dimensionality hint. When the caller knows the output
+   * dimension up-front, it is included in the canonical id without
+   * waiting for the first `embed()` call.
+   */
+  readonly dim?: number;
+  /**
+   * Disable the automatic E5 `query:` / `passage:` prefixing (PS-10). The
+   * prefixes are applied by default for E5-family models (the multilingual-e5
+   * default and any model whose id carries an `e5` token), because the E5 model
+   * card requires them and omitting them measurably degrades retrieval. Set
+   * this to `true` only if your inputs are already prefixed or you use a
+   * non-standard E5 export. Toggling it changes the canonical `configHash`
+   * (and thus the embedder id), which triggers a re-embedding migration.
+   */
+  readonly disableTaskPrefix?: boolean;
+}
+/**
+ * Tiny structural shape of `@huggingface/transformers`' feature-
+ * extraction pipeline used by this package. Declared inline so the
+ * embedder does not import the heavy peer at build time.
+ *
+ * @internal
+ */
+type FeatureExtractor = (texts: string | readonly string[], options?: {
+  pooling?: Pooling;
+  normalize?: boolean;
+  signal?: AbortSignal;
+}) => Promise<{
+  data: Float32Array;
+  dims: readonly number[];
+  tolist?(): unknown;
+}>;
+/**
+ * Pipeline-factory shape used for dependency injection in tests.
+ *
+ * @internal
+ */
+type PipelineFactory = (task: 'feature-extraction', model: string, opts: {
+  revision?: string;
+  cache_dir?: string;
+  dtype?: string;
+  device?: string;
+}) => Promise<FeatureExtractor>;
+/**
+ * Raised when the underlying transformer model cannot be loaded
+ * (offline / corporate firewall / wrong cache dir).
+ *
+ * @stable
+ */
+declare class EmbedderModelLoadError extends Error {
+  readonly name = "EmbedderModelLoadError";
+}
+/**
+ * Build a `TransformersJsEmbedder` instance. Lazy: the underlying
+ * pipeline is constructed on the first `embed()` call so packaging
+ * the embedder does not pay the model-load cost.
+ *
+ * @stable
+ */
+declare function createTransformersJsEmbedder(options?: TransformersJsEmbedderOptions): TransformersJsEmbedder;
+/**
+ * `EmbedderProvider` implementation backed by `@huggingface/transformers`.
+ *
+ * @stable
+ */
+declare class TransformersJsEmbedder implements EmbedderProvider {
+  #private;
+  constructor(options: TransformersJsEmbedderOptions);
+  id(): string;
+  dim(): number;
+  configHash(): string;
+  embed(texts: ReadonlyArray<string>, opts?: EmbedOptions): Promise<ReadonlyArray<Float32Array>>;
+}
+/** Test-only cache reset. */
+declare function _resetPipelineFactoryCacheForTesting(): void;
+/**
+ * True when a model id belongs to the E5 family, which requires asymmetric
+ * `query:` / `passage:` prefixes (PS-10). Matches an `e5` token bounded by a
+ * path / dash / underscore so it covers `multilingual-e5-base`, `e5-large`,
+ * `intfloat/e5-mistral`, etc. without false-matching unrelated names.
+ */
+declare function isE5Model(model: string): boolean;
+/**
+ * Canonical-JSON deterministic hash of an embedder configuration.
+ * Object keys are sorted lexicographically; primitives flow through as
+ * `JSON.stringify` would render them. Used by the multi-table per-
+ * embedder vec0 layout to tell drift apart from a true model swap.
+ *
+ * @stable
+ */
+declare function canonicalConfigHash(config: unknown): string;
+//#endregion
+export { EmbedderModelLoadError, FeatureExtractor, PipelineFactory, Pooling, TransformersJsEmbedder, TransformersJsEmbedderOptions, VERSION, _resetPipelineFactoryCacheForTesting, canonicalConfigHash, createTransformersJsEmbedder, isE5Model };
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","names":[],"sources":["../src/index.ts"],"sourcesContent":[],"mappings":";;;;;AAiGY,cAlFC,OAAA,GAkFc,OAId;AAWb;AAWA;AAWA;;;;AAyD2B,KAxKf,OAAA,GAwKe,MAAA,GAAA,KAAA,GAAA,aAAA,GAAA,KAAA,GAAA,YAAA,GAAA,MAAA;;;;;AAyE3B;AAsBgB,UAhQC,6BAAA,CAgQQ;EAYT;;;qBAxQK;;;;;;;;;;;;;;;;;;;;;;;6BAuBQ;;;;;;;;;;;;;;;;;;;;;;;;;KA0BjB,gBAAA;YAEY;;WAAuC;MAC1D;QACG;;;;;;;;;KAUI,eAAA;;;;;MAIP,QAAQ;;;;;;;cAWA,sBAAA,SAA+B,KAAA;;;;;;;;;;iBAW5B,4BAAA,WACL,gCACR;;;;;;cASU,sBAAA,YAAkC;;uBAexB;;;;eAwCZ,8BACD,eACL,QAAQ,cAAc;;;iBAyEX,oCAAA,CAAA;;;;;;;iBAsBA,SAAA;;;;;;;;;iBAYA,mBAAA"}

package/dist/index.js

@@ -0,0 +1,185 @@
+import { createHash } from "node:crypto";
+
+//#region src/index.ts
+/**
+* @graphorin/embedder-transformersjs — default in-process embedder.
+*
+* Wraps `@huggingface/transformers@^4.1.0` to produce dense embeddings
+* inside the Graphorin process. Default model
+* `Xenova/multilingual-e5-base` (768-dim, multilingual; DEC-130 /
+* ADR-025).
+*
+* @packageDocumentation
+*/
+/** Canonical version constant. Mirrors the `package.json` version. */
+const VERSION = "0.5.0";
+const DEFAULT_MODEL = "Xenova/multilingual-e5-base";
+const DEFAULT_DIM = 768;
+/**
+* Raised when the underlying transformer model cannot be loaded
+* (offline / corporate firewall / wrong cache dir).
+*
+* @stable
+*/
+var EmbedderModelLoadError = class extends Error {
+	name = "EmbedderModelLoadError";
+};
+/**
+* Build a `TransformersJsEmbedder` instance. Lazy: the underlying
+* pipeline is constructed on the first `embed()` call so packaging
+* the embedder does not pay the model-load cost.
+*
+* @stable
+*/
+function createTransformersJsEmbedder(options = {}) {
+	return new TransformersJsEmbedder(options);
+}
+/**
+* `EmbedderProvider` implementation backed by `@huggingface/transformers`.
+*
+* @stable
+*/
+var TransformersJsEmbedder = class {
+	#model;
+	#pooling;
+	#normalize;
+	#revision;
+	#cacheDir;
+	#dtype;
+	#device;
+	#pipelineFactory;
+	/** Whether to apply E5 `query:` / `passage:` prefixes (PS-10). */
+	#taskPrefix;
+	#extractor = null;
+	#loading = null;
+	#resolvedDim;
+	constructor(options) {
+		this.#model = options.model ?? DEFAULT_MODEL;
+		this.#pooling = options.pooling ?? "mean";
+		this.#normalize = options.normalize ?? true;
+		this.#revision = options.revision;
+		this.#cacheDir = options.cacheDir ?? process.env.GRAPHORIN_CACHE_DIR;
+		this.#dtype = options.dtype;
+		this.#device = options.device;
+		this.#pipelineFactory = options.pipelineFactory;
+		this.#resolvedDim = options.dim ?? guessDefaultDim(this.#model);
+		this.#taskPrefix = isE5Model(this.#model) && options.disableTaskPrefix !== true;
+	}
+	id() {
+		const dim = this.#resolvedDim ?? DEFAULT_DIM;
+		return `transformersjs:${this.#model}@${dim}`;
+	}
+	dim() {
+		if (this.#resolvedDim !== null) return this.#resolvedDim;
+		return DEFAULT_DIM;
+	}
+	configHash() {
+		return canonicalConfigHash({
+			adapter: "transformersjs",
+			model: this.#model,
+			pooling: this.#pooling,
+			normalize: this.#normalize,
+			revision: this.#revision ?? null,
+			dtype: this.#dtype ?? null,
+			device: this.#device ?? null,
+			...this.#taskPrefix ? { taskPrefix: "e5" } : {}
+		});
+	}
+	async embed(texts, opts = {}) {
+		if (texts.length === 0) return [];
+		const result = await (await this.#getExtractor())(this.#taskPrefix ? texts.map((t) => `${opts.taskType ?? "passage"}: ${t}`) : [...texts], {
+			pooling: this.#pooling,
+			normalize: this.#normalize,
+			...opts.signal !== void 0 ? { signal: opts.signal } : {}
+		});
+		const lastDim = result.dims[result.dims.length - 1] ?? this.#resolvedDim ?? DEFAULT_DIM;
+		if (this.#resolvedDim === null) this.#resolvedDim = lastDim;
+		const dim = lastDim;
+		const out = [];
+		for (let i = 0; i < texts.length; i++) {
+			const start = i * dim;
+			out.push(result.data.slice(start, start + dim));
+		}
+		return out;
+	}
+	async #getExtractor() {
+		if (this.#extractor !== null) return this.#extractor;
+		if (this.#loading !== null) return this.#loading;
+		this.#loading = (this.#pipelineFactory ?? await loadDefaultPipelineFactory())("feature-extraction", this.#model, {
+			...this.#revision !== void 0 ? { revision: this.#revision } : {},
+			...this.#cacheDir !== void 0 ? { cache_dir: this.#cacheDir } : {},
+			...this.#dtype !== void 0 ? { dtype: this.#dtype } : {},
+			...this.#device !== void 0 ? { device: this.#device } : {}
+		}).then((pipe) => {
+			this.#extractor = pipe;
+			this.#loading = null;
+			return pipe;
+		}).catch((err) => {
+			this.#loading = null;
+			throw new EmbedderModelLoadError(`[graphorin/embedder-transformersjs] failed to load model '${this.#model}'. See the offline-install guide for instructions on pre-downloading the model.`, { cause: err });
+		});
+		return this.#loading;
+	}
+};
+/** @internal */
+let CACHED_PIPELINE_FACTORY = null;
+async function loadDefaultPipelineFactory() {
+	if (CACHED_PIPELINE_FACTORY !== null) return CACHED_PIPELINE_FACTORY;
+	try {
+		CACHED_PIPELINE_FACTORY = (await import("@huggingface/transformers")).pipeline;
+		return CACHED_PIPELINE_FACTORY;
+	} catch (err) {
+		throw new EmbedderModelLoadError("[graphorin/embedder-transformersjs] required peer '@huggingface/transformers' is not installed.", { cause: err });
+	}
+}
+/** Test-only cache reset. */
+function _resetPipelineFactoryCacheForTesting() {
+	CACHED_PIPELINE_FACTORY = null;
+}
+const KNOWN_DIMS = new Map([
+	["Xenova/multilingual-e5-small", 384],
+	["Xenova/multilingual-e5-base", 768],
+	["Xenova/multilingual-e5-large", 1024],
+	["Xenova/bge-m3", 1024],
+	["onnx-community/all-MiniLM-L6-v2-ONNX", 384]
+]);
+function guessDefaultDim(model) {
+	return KNOWN_DIMS.get(model) ?? null;
+}
+/**
+* True when a model id belongs to the E5 family, which requires asymmetric
+* `query:` / `passage:` prefixes (PS-10). Matches an `e5` token bounded by a
+* path / dash / underscore so it covers `multilingual-e5-base`, `e5-large`,
+* `intfloat/e5-mistral`, etc. without false-matching unrelated names.
+*/
+function isE5Model(model) {
+	return /(?:^|[/_-])e5(?:[/_-]|$)/i.test(model);
+}
+/**
+* Canonical-JSON deterministic hash of an embedder configuration.
+* Object keys are sorted lexicographically; primitives flow through as
+* `JSON.stringify` would render them. Used by the multi-table per-
+* embedder vec0 layout to tell drift apart from a true model swap.
+*
+* @stable
+*/
+function canonicalConfigHash(config) {
+	const canonical = canonicalize(config);
+	return createHash("sha256").update(canonical, "utf8").digest("hex");
+}
+function canonicalize(value) {
+	if (value === null) return "null";
+	if (typeof value === "boolean") return value ? "true" : "false";
+	if (typeof value === "number") return Number.isFinite(value) ? value.toString() : "null";
+	if (typeof value === "string") return JSON.stringify(value);
+	if (Array.isArray(value)) return `[${value.map(canonicalize).join(",")}]`;
+	if (typeof value === "object") {
+		const obj = value;
+		return `{${Object.keys(obj).sort().map((k) => `${JSON.stringify(k)}:${canonicalize(obj[k])}`).join(",")}}`;
+	}
+	return "null";
+}
+
+//#endregion
+export { EmbedderModelLoadError, TransformersJsEmbedder, VERSION, _resetPipelineFactoryCacheForTesting, canonicalConfigHash, createTransformersJsEmbedder, isE5Model };
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","names":["#model","#pooling","#normalize","#revision","#cacheDir","#dtype","#device","#pipelineFactory","#taskPrefix","#resolvedDim","#getExtractor","out: Float32Array[]","#extractor","#loading","CACHED_PIPELINE_FACTORY: PipelineFactory | null","KNOWN_DIMS: ReadonlyMap<string, number>"],"sources":["../src/index.ts"],"sourcesContent":["/**\n * @graphorin/embedder-transformersjs — default in-process embedder.\n *\n * Wraps `@huggingface/transformers@^4.1.0` to produce dense embeddings\n * inside the Graphorin process. Default model\n * `Xenova/multilingual-e5-base` (768-dim, multilingual; DEC-130 /\n * ADR-025).\n *\n * @packageDocumentation\n */\n\nimport { createHash } from 'node:crypto';\nimport type { EmbedderProvider, EmbedOptions } from '@graphorin/core/contracts';\n\n/** Canonical version constant. Mirrors the `package.json` version. */\nexport const VERSION = '0.5.0';\n\n/**\n * Pooling strategy. Defaults to `'mean'` per the multilingual-e5 model\n * card.\n *\n * @stable\n */\nexport type Pooling = 'mean' | 'cls' | 'first_token' | 'eos' | 'last_token' | 'none';\n\n/**\n * Configuration accepted by {@link createTransformersJsEmbedder}.\n *\n * @stable\n */\nexport interface TransformersJsEmbedderOptions {\n  /** Default `'Xenova/multilingual-e5-base'` (768-dim). */\n  readonly model?: string;\n  /** Default `'mean'`. */\n  readonly pooling?: Pooling;\n  /** Default `true`. */\n  readonly normalize?: boolean;\n  /** Optional model revision pin (`'main'` if unset). */\n  readonly revision?: string;\n  /**\n   * Optional cache directory. When unset, the embedder honours\n   * `process.env.GRAPHORIN_CACHE_DIR`, otherwise falls back to the\n   * Hugging Face default (`os.homedir()/.cache/huggingface/hub`).\n   */\n  readonly cacheDir?: string;\n  /**\n   * Optional dtype hint (`'fp32' | 'fp16' | 'q8' | 'q4'`). When unset,\n   * the runtime picks the model's recommended default.\n   */\n  readonly dtype?: string;\n  /** Override device (`'cpu'`, `'webgpu'`, …). Default `'cpu'`. */\n  readonly device?: string;\n  /**\n   * Override the underlying `pipeline` factory — used by the test\n   * suite to inject a stub. Production callers should leave this\n   * unset so the package lazily loads `@huggingface/transformers`.\n   */\n  readonly pipelineFactory?: PipelineFactory;\n  /**\n   * Optional dimensionality hint. When the caller knows the output\n   * dimension up-front, it is included in the canonical id without\n   * waiting for the first `embed()` call.\n   */\n  readonly dim?: number;\n  /**\n   * Disable the automatic E5 `query:` / `passage:` prefixing (PS-10). The\n   * prefixes are applied by default for E5-family models (the multilingual-e5\n   * default and any model whose id carries an `e5` token), because the E5 model\n   * card requires them and omitting them measurably degrades retrieval. Set\n   * this to `true` only if your inputs are already prefixed or you use a\n   * non-standard E5 export. Toggling it changes the canonical `configHash`\n   * (and thus the embedder id), which triggers a re-embedding migration.\n   */\n  readonly disableTaskPrefix?: boolean;\n}\n\n/**\n * Tiny structural shape of `@huggingface/transformers`' feature-\n * extraction pipeline used by this package. Declared inline so the\n * embedder does not import the heavy peer at build time.\n *\n * @internal\n */\nexport type FeatureExtractor = (\n  texts: string | readonly string[],\n  options?: { pooling?: Pooling; normalize?: boolean; signal?: AbortSignal },\n) => Promise<{\n  data: Float32Array;\n  dims: readonly number[];\n  tolist?(): unknown;\n}>;\n\n/**\n * Pipeline-factory shape used for dependency injection in tests.\n *\n * @internal\n */\nexport type PipelineFactory = (\n  task: 'feature-extraction',\n  model: string,\n  opts: { revision?: string; cache_dir?: string; dtype?: string; device?: string },\n) => Promise<FeatureExtractor>;\n\nconst DEFAULT_MODEL = 'Xenova/multilingual-e5-base';\nconst DEFAULT_DIM = 768;\n\n/**\n * Raised when the underlying transformer model cannot be loaded\n * (offline / corporate firewall / wrong cache dir).\n *\n * @stable\n */\nexport class EmbedderModelLoadError extends Error {\n  override readonly name = 'EmbedderModelLoadError';\n}\n\n/**\n * Build a `TransformersJsEmbedder` instance. Lazy: the underlying\n * pipeline is constructed on the first `embed()` call so packaging\n * the embedder does not pay the model-load cost.\n *\n * @stable\n */\nexport function createTransformersJsEmbedder(\n  options: TransformersJsEmbedderOptions = {},\n): TransformersJsEmbedder {\n  return new TransformersJsEmbedder(options);\n}\n\n/**\n * `EmbedderProvider` implementation backed by `@huggingface/transformers`.\n *\n * @stable\n */\nexport class TransformersJsEmbedder implements EmbedderProvider {\n  readonly #model: string;\n  readonly #pooling: Pooling;\n  readonly #normalize: boolean;\n  readonly #revision: string | undefined;\n  readonly #cacheDir: string | undefined;\n  readonly #dtype: string | undefined;\n  readonly #device: string | undefined;\n  readonly #pipelineFactory: PipelineFactory | undefined;\n  /** Whether to apply E5 `query:` / `passage:` prefixes (PS-10). */\n  readonly #taskPrefix: boolean;\n  #extractor: FeatureExtractor | null = null;\n  #loading: Promise<FeatureExtractor> | null = null;\n  #resolvedDim: number | null;\n\n  constructor(options: TransformersJsEmbedderOptions) {\n    this.#model = options.model ?? DEFAULT_MODEL;\n    this.#pooling = options.pooling ?? 'mean';\n    this.#normalize = options.normalize ?? true;\n    this.#revision = options.revision;\n    this.#cacheDir = options.cacheDir ?? process.env.GRAPHORIN_CACHE_DIR;\n    this.#dtype = options.dtype;\n    this.#device = options.device;\n    this.#pipelineFactory = options.pipelineFactory;\n    this.#resolvedDim = options.dim ?? guessDefaultDim(this.#model);\n    this.#taskPrefix = isE5Model(this.#model) && options.disableTaskPrefix !== true;\n  }\n\n  id(): string {\n    const dim = this.#resolvedDim ?? DEFAULT_DIM;\n    return `transformersjs:${this.#model}@${dim}`;\n  }\n\n  dim(): number {\n    if (this.#resolvedDim !== null) return this.#resolvedDim;\n    return DEFAULT_DIM;\n  }\n\n  configHash(): string {\n    return canonicalConfigHash({\n      adapter: 'transformersjs',\n      model: this.#model,\n      pooling: this.#pooling,\n      normalize: this.#normalize,\n      revision: this.#revision ?? null,\n      dtype: this.#dtype ?? null,\n      device: this.#device ?? null,\n      // PS-10: the prefix policy changes the embeddings, so it must change the\n      // id. Only added when active (E5 + not disabled) so non-E5 ids — and the\n      // historical hash of an E5 model with prefixing turned off — are stable.\n      ...(this.#taskPrefix ? { taskPrefix: 'e5' as const } : {}),\n    });\n  }\n\n  async embed(\n    texts: ReadonlyArray<string>,\n    opts: EmbedOptions = {},\n  ): Promise<ReadonlyArray<Float32Array>> {\n    if (texts.length === 0) return [];\n    const extractor = await this.#getExtractor();\n    // PS-10: E5 models require an asymmetric `query:` / `passage:` prefix.\n    // Default to `passage` (the indexing role) when the caller doesn't specify.\n    const inputs = this.#taskPrefix\n      ? texts.map((t) => `${opts.taskType ?? 'passage'}: ${t}`)\n      : [...texts];\n    const result = await extractor(inputs, {\n      pooling: this.#pooling,\n      normalize: this.#normalize,\n      ...(opts.signal !== undefined ? { signal: opts.signal } : {}),\n    });\n    const lastDim = result.dims[result.dims.length - 1] ?? this.#resolvedDim ?? DEFAULT_DIM;\n    if (this.#resolvedDim === null) {\n      this.#resolvedDim = lastDim;\n    }\n    const dim = lastDim;\n    const out: Float32Array[] = [];\n    for (let i = 0; i < texts.length; i++) {\n      const start = i * dim;\n      out.push(result.data.slice(start, start + dim));\n    }\n    return out;\n  }\n\n  async #getExtractor(): Promise<FeatureExtractor> {\n    if (this.#extractor !== null) return this.#extractor;\n    if (this.#loading !== null) return this.#loading;\n    const loader = this.#pipelineFactory ?? (await loadDefaultPipelineFactory());\n    this.#loading = loader('feature-extraction', this.#model, {\n      ...(this.#revision !== undefined ? { revision: this.#revision } : {}),\n      ...(this.#cacheDir !== undefined ? { cache_dir: this.#cacheDir } : {}),\n      ...(this.#dtype !== undefined ? { dtype: this.#dtype } : {}),\n      ...(this.#device !== undefined ? { device: this.#device } : {}),\n    })\n      .then((pipe) => {\n        this.#extractor = pipe;\n        this.#loading = null;\n        return pipe;\n      })\n      .catch((err) => {\n        this.#loading = null;\n        throw new EmbedderModelLoadError(\n          `[graphorin/embedder-transformersjs] failed to load model '${this.#model}'. ` +\n            'See the offline-install guide for instructions on pre-downloading the model.',\n          { cause: err },\n        );\n      });\n    return this.#loading;\n  }\n}\n\n/** @internal */\nlet CACHED_PIPELINE_FACTORY: PipelineFactory | null = null;\n\nasync function loadDefaultPipelineFactory(): Promise<PipelineFactory> {\n  if (CACHED_PIPELINE_FACTORY !== null) return CACHED_PIPELINE_FACTORY;\n  try {\n    const mod = (await import('@huggingface/transformers')) as unknown as {\n      pipeline: PipelineFactory;\n    };\n    CACHED_PIPELINE_FACTORY = mod.pipeline;\n    return CACHED_PIPELINE_FACTORY;\n  } catch (err) {\n    throw new EmbedderModelLoadError(\n      \"[graphorin/embedder-transformersjs] required peer '@huggingface/transformers' is not installed.\",\n      { cause: err },\n    );\n  }\n}\n\n/** Test-only cache reset. */\nexport function _resetPipelineFactoryCacheForTesting(): void {\n  CACHED_PIPELINE_FACTORY = null;\n}\n\nconst KNOWN_DIMS: ReadonlyMap<string, number> = new Map([\n  ['Xenova/multilingual-e5-small', 384],\n  ['Xenova/multilingual-e5-base', 768],\n  ['Xenova/multilingual-e5-large', 1024],\n  ['Xenova/bge-m3', 1024],\n  ['onnx-community/all-MiniLM-L6-v2-ONNX', 384],\n]);\n\nfunction guessDefaultDim(model: string): number | null {\n  return KNOWN_DIMS.get(model) ?? null;\n}\n\n/**\n * True when a model id belongs to the E5 family, which requires asymmetric\n * `query:` / `passage:` prefixes (PS-10). Matches an `e5` token bounded by a\n * path / dash / underscore so it covers `multilingual-e5-base`, `e5-large`,\n * `intfloat/e5-mistral`, etc. without false-matching unrelated names.\n */\nexport function isE5Model(model: string): boolean {\n  return /(?:^|[/_-])e5(?:[/_-]|$)/i.test(model);\n}\n\n/**\n * Canonical-JSON deterministic hash of an embedder configuration.\n * Object keys are sorted lexicographically; primitives flow through as\n * `JSON.stringify` would render them. Used by the multi-table per-\n * embedder vec0 layout to tell drift apart from a true model swap.\n *\n * @stable\n */\nexport function canonicalConfigHash(config: unknown): string {\n  const canonical = canonicalize(config);\n  return createHash('sha256').update(canonical, 'utf8').digest('hex');\n}\n\nfunction canonicalize(value: unknown): string {\n  if (value === null) return 'null';\n  if (typeof value === 'boolean') return value ? 'true' : 'false';\n  if (typeof value === 'number') return Number.isFinite(value) ? value.toString() : 'null';\n  if (typeof value === 'string') return JSON.stringify(value);\n  if (Array.isArray(value)) {\n    return `[${value.map(canonicalize).join(',')}]`;\n  }\n  if (typeof value === 'object') {\n    const obj = value as Record<string, unknown>;\n    const keys = Object.keys(obj).sort();\n    return `{${keys.map((k) => `${JSON.stringify(k)}:${canonicalize(obj[k])}`).join(',')}}`;\n  }\n  return 'null';\n}\n"],"mappings":";;;;;;;;;;;;;;AAeA,MAAa,UAAU;AAwFvB,MAAM,gBAAgB;AACtB,MAAM,cAAc;;;;;;;AAQpB,IAAa,yBAAb,cAA4C,MAAM;CAChD,AAAkB,OAAO;;;;;;;;;AAU3B,SAAgB,6BACd,UAAyC,EAAE,EACnB;AACxB,QAAO,IAAI,uBAAuB,QAAQ;;;;;;;AAQ5C,IAAa,yBAAb,MAAgE;CAC9D,CAASA;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;;CAET,CAASC;CACT,aAAsC;CACtC,WAA6C;CAC7C;CAEA,YAAY,SAAwC;AAClD,QAAKR,QAAS,QAAQ,SAAS;AAC/B,QAAKC,UAAW,QAAQ,WAAW;AACnC,QAAKC,YAAa,QAAQ,aAAa;AACvC,QAAKC,WAAY,QAAQ;AACzB,QAAKC,WAAY,QAAQ,YAAY,QAAQ,IAAI;AACjD,QAAKC,QAAS,QAAQ;AACtB,QAAKC,SAAU,QAAQ;AACvB,QAAKC,kBAAmB,QAAQ;AAChC,QAAKE,cAAe,QAAQ,OAAO,gBAAgB,MAAKT,MAAO;AAC/D,QAAKQ,aAAc,UAAU,MAAKR,MAAO,IAAI,QAAQ,sBAAsB;;CAG7E,KAAa;EACX,MAAM,MAAM,MAAKS,eAAgB;AACjC,SAAO,kBAAkB,MAAKT,MAAO,GAAG;;CAG1C,MAAc;AACZ,MAAI,MAAKS,gBAAiB,KAAM,QAAO,MAAKA;AAC5C,SAAO;;CAGT,aAAqB;AACnB,SAAO,oBAAoB;GACzB,SAAS;GACT,OAAO,MAAKT;GACZ,SAAS,MAAKC;GACd,WAAW,MAAKC;GAChB,UAAU,MAAKC,YAAa;GAC5B,OAAO,MAAKE,SAAU;GACtB,QAAQ,MAAKC,UAAW;GAIxB,GAAI,MAAKE,aAAc,EAAE,YAAY,MAAe,GAAG,EAAE;GAC1D,CAAC;;CAGJ,MAAM,MACJ,OACA,OAAqB,EAAE,EACe;AACtC,MAAI,MAAM,WAAW,EAAG,QAAO,EAAE;EAOjC,MAAM,SAAS,OANG,MAAM,MAAKE,cAAe,EAG7B,MAAKF,aAChB,MAAM,KAAK,MAAM,GAAG,KAAK,YAAY,UAAU,IAAI,IAAI,GACvD,CAAC,GAAG,MAAM,EACyB;GACrC,SAAS,MAAKP;GACd,WAAW,MAAKC;GAChB,GAAI,KAAK,WAAW,SAAY,EAAE,QAAQ,KAAK,QAAQ,GAAG,EAAE;GAC7D,CAAC;EACF,MAAM,UAAU,OAAO,KAAK,OAAO,KAAK,SAAS,MAAM,MAAKO,eAAgB;AAC5E,MAAI,MAAKA,gBAAiB,KACxB,OAAKA,cAAe;EAEtB,MAAM,MAAM;EACZ,MAAME,MAAsB,EAAE;AAC9B,OAAK,IAAI,IAAI,GAAG,IAAI,MAAM,QAAQ,KAAK;GACrC,MAAM,QAAQ,IAAI;AAClB,OAAI,KAAK,OAAO,KAAK,MAAM,OAAO,QAAQ,IAAI,CAAC;;AAEjD,SAAO;;CAGT,OAAMD,eAA2C;AAC/C,MAAI,MAAKE,cAAe,KAAM,QAAO,MAAKA;AAC1C,MAAI,MAAKC,YAAa,KAAM,QAAO,MAAKA;AAExC,QAAKA,WADU,MAAKN,mBAAqB,MAAM,4BAA4B,EACpD,sBAAsB,MAAKP,OAAQ;GACxD,GAAI,MAAKG,aAAc,SAAY,EAAE,UAAU,MAAKA,UAAW,GAAG,EAAE;GACpE,GAAI,MAAKC,aAAc,SAAY,EAAE,WAAW,MAAKA,UAAW,GAAG,EAAE;GACrE,GAAI,MAAKC,UAAW,SAAY,EAAE,OAAO,MAAKA,OAAQ,GAAG,EAAE;GAC3D,GAAI,MAAKC,WAAY,SAAY,EAAE,QAAQ,MAAKA,QAAS,GAAG,EAAE;GAC/D,CAAC,CACC,MAAM,SAAS;AACd,SAAKM,YAAa;AAClB,SAAKC,UAAW;AAChB,UAAO;IACP,CACD,OAAO,QAAQ;AACd,SAAKA,UAAW;AAChB,SAAM,IAAI,uBACR,6DAA6D,MAAKb,MAAO,kFAEzE,EAAE,OAAO,KAAK,CACf;IACD;AACJ,SAAO,MAAKa;;;;AAKhB,IAAIC,0BAAkD;AAEtD,eAAe,6BAAuD;AACpE,KAAI,4BAA4B,KAAM,QAAO;AAC7C,KAAI;AAIF,6BAHa,MAAM,OAAO,8BAGI;AAC9B,SAAO;UACA,KAAK;AACZ,QAAM,IAAI,uBACR,mGACA,EAAE,OAAO,KAAK,CACf;;;;AAKL,SAAgB,uCAA6C;AAC3D,2BAA0B;;AAG5B,MAAMC,aAA0C,IAAI,IAAI;CACtD,CAAC,gCAAgC,IAAI;CACrC,CAAC,+BAA+B,IAAI;CACpC,CAAC,gCAAgC,KAAK;CACtC,CAAC,iBAAiB,KAAK;CACvB,CAAC,wCAAwC,IAAI;CAC9C,CAAC;AAEF,SAAS,gBAAgB,OAA8B;AACrD,QAAO,WAAW,IAAI,MAAM,IAAI;;;;;;;;AASlC,SAAgB,UAAU,OAAwB;AAChD,QAAO,4BAA4B,KAAK,MAAM;;;;;;;;;;AAWhD,SAAgB,oBAAoB,QAAyB;CAC3D,MAAM,YAAY,aAAa,OAAO;AACtC,QAAO,WAAW,SAAS,CAAC,OAAO,WAAW,OAAO,CAAC,OAAO,MAAM;;AAGrE,SAAS,aAAa,OAAwB;AAC5C,KAAI,UAAU,KAAM,QAAO;AAC3B,KAAI,OAAO,UAAU,UAAW,QAAO,QAAQ,SAAS;AACxD,KAAI,OAAO,UAAU,SAAU,QAAO,OAAO,SAAS,MAAM,GAAG,MAAM,UAAU,GAAG;AAClF,KAAI,OAAO,UAAU,SAAU,QAAO,KAAK,UAAU,MAAM;AAC3D,KAAI,MAAM,QAAQ,MAAM,CACtB,QAAO,IAAI,MAAM,IAAI,aAAa,CAAC,KAAK,IAAI,CAAC;AAE/C,KAAI,OAAO,UAAU,UAAU;EAC7B,MAAM,MAAM;AAEZ,SAAO,IADM,OAAO,KAAK,IAAI,CAAC,MAAM,CACpB,KAAK,MAAM,GAAG,KAAK,UAAU,EAAE,CAAC,GAAG,aAAa,IAAI,GAAG,GAAG,CAAC,KAAK,IAAI,CAAC;;AAEvF,QAAO"}

package/package.json

@@ -0,0 +1,71 @@
+{
+  "name": "@graphorin/embedder-transformersjs",
+  "version": "0.5.0",
+  "description": "Default in-process embedder for the Graphorin framework. Wraps `@huggingface/transformers@^4.1.0` to produce dense vectors with the multilingual `Xenova/multilingual-e5-base` model (768-dim) by default. Implements the `EmbedderProvider` contract from `@graphorin/core/contracts` with deterministic `configHash`, batched embed, lazy model download (cache honours `GRAPHORIN_CACHE_DIR`), and an actionable error if the model cannot be fetched.",
+  "license": "MIT",
+  "author": "Oleksiy Stepurenko",
+  "homepage": "https://github.com/o-stepper/graphorin/tree/main/packages/embedder-transformersjs",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/o-stepper/graphorin.git",
+    "directory": "packages/embedder-transformersjs"
+  },
+  "bugs": {
+    "url": "https://github.com/o-stepper/graphorin/issues"
+  },
+  "keywords": [
+    "graphorin",
+    "ai",
+    "agents",
+    "framework",
+    "embeddings",
+    "transformersjs",
+    "huggingface",
+    "multilingual"
+  ],
+  "type": "module",
+  "engines": {
+    "node": ">=22.0.0"
+  },
+  "main": "./dist/index.js",
+  "module": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    },
+    "./package.json": "./package.json"
+  },
+  "files": [
+    "dist",
+    "README.md",
+    "CHANGELOG.md",
+    "LICENSE"
+  ],
+  "dependencies": {
+    "@graphorin/core": "0.5.0"
+  },
+  "peerDependencies": {
+    "@huggingface/transformers": "^4.1.0"
+  },
+  "peerDependenciesMeta": {
+    "@huggingface/transformers": {
+      "optional": false
+    }
+  },
+  "publishConfig": {
+    "access": "public",
+    "provenance": true
+  },
+  "devDependencies": {
+    "@huggingface/transformers": "^4.1.0"
+  },
+  "scripts": {
+    "build": "tsdown",
+    "typecheck": "tsc --noEmit && tsc -p tsconfig.tests.json",
+    "test": "vitest run",
+    "lint": "biome check .",
+    "clean": "rimraf dist .turbo *.tsbuildinfo"
+  }
+}