@graphorin/embedder-transformersjs 0.5.0 → 0.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +30 -1
- package/README.md +8 -8
- package/dist/index.d.ts +12 -2
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +15 -5
- package/dist/index.js.map +1 -1
- package/package.json +2 -2
package/CHANGELOG.md
CHANGED
|
@@ -1,9 +1,38 @@
|
|
|
1
1
|
# @graphorin/embedder-transformersjs
|
|
2
2
|
|
|
3
|
+
## 0.6.0
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- [#135](https://github.com/o-stepper/graphorin/pull/135) [`4f850d9`](https://github.com/o-stepper/graphorin/commit/4f850d9bc0a05d6256c59c5117b010336fcb41d3) Thanks [@o-stepper](https://github.com/o-stepper)! - Server / client / periphery correctness (audit 2026-07-04 Wave B, cluster B6).
|
|
8
|
+
|
|
9
|
+
- periphery-01: `POST /v1/workflows/:id/resume` actually resumes - it background-iterates `workflow.resume(threadId, {resume})` and emits every event on the `workflow:<id>/runs/<runId>/events` subject (mirroring execute); the fictional unmounted SSE URL is gone and the run completes. `POST /v1/workflows/:id/fork` returns an honest 501 instead of a 202 that forked nothing.
|
|
10
|
+
- periphery-02: `standalone-server.md` now states plainly that `graphorin start` serves health/metrics/tokens/tickets/WS-SSE only and that the domain routes mount only when adapters are composed programmatically via `createServer({...})`.
|
|
11
|
+
- periphery-03: the client SSE fallback survives reconnects - `#reconnect` no longer attempts the RPC resubscribe on the read-only SSE transport (which threw and permanently killed the subscription); the reconnect carries the subscription's cursor as a `Last-Event-ID` header so the server replays only missed events; a closed bound subscription is recreated on the next `subscribe()`.
|
|
12
|
+
- periphery-08: the idempotency middleware tracks in-flight keyed executions - a concurrent duplicate gets `409 idempotency-in-flight` + `Retry-After` instead of double-executing (per draft-ietf-httpapi-idempotency-key-header).
|
|
13
|
+
- periphery-09: `subscribe({ target: 'run' })` without `sessionId` throws a clear client-side error instead of building a `run:` subject the server grammar can never accept.
|
|
14
|
+
- periphery-10: session-stream subjects gate on `sessions:read:<sessionId>` (read-only streams, sessionId resource slot) instead of `agents:invoke:<sessionId>`, aligning with the SSE route's requirement.
|
|
15
|
+
- periphery-11: interval triggers under `catchupPolicy: 'none'` advance to the next FUTURE boundary after downtime instead of firing immediately on restart; `recordActivity()` no longer arms idle timers on a stopped scheduler (P-14).
|
|
16
|
+
- periphery-05: the TransformersJS embedder throws for an unknown model with no `dim` hint (the PS-11 fix ported from the Ollama embedder) instead of assuming 768 - a wrong assumed width baked a wrong-width id + vec0 table and the id changed after the first embed; a width drift against a bound dim now throws too.
|
|
17
|
+
- P-05: the WS upgrade bearer verification passes the client IP so the per-IP failure lockout engages for upgrade attempts (previously a lockout-free brute-force surface).
|
|
18
|
+
- Docs: the fictional "disconnect policy" section replaced with the real reconnect-and-replay behaviour; the WS ticket endpoint path corrected to `/v1/session/ws-ticket`.
|
|
19
|
+
|
|
20
|
+
- Updated dependencies [[`32f20c1`](https://github.com/o-stepper/graphorin/commit/32f20c110f184f8cef7eec85bf39f5f07c886cb6), [`4f850d9`](https://github.com/o-stepper/graphorin/commit/4f850d9bc0a05d6256c59c5117b010336fcb41d3), [`4f850d9`](https://github.com/o-stepper/graphorin/commit/4f850d9bc0a05d6256c59c5117b010336fcb41d3), [`17a2d30`](https://github.com/o-stepper/graphorin/commit/17a2d30564154ca2ab87473335cdef43a5089c84), [`17a2d30`](https://github.com/o-stepper/graphorin/commit/17a2d30564154ca2ab87473335cdef43a5089c84), [`17a2d30`](https://github.com/o-stepper/graphorin/commit/17a2d30564154ca2ab87473335cdef43a5089c84), [`17a2d30`](https://github.com/o-stepper/graphorin/commit/17a2d30564154ca2ab87473335cdef43a5089c84), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627), [`ce06b47`](https://github.com/o-stepper/graphorin/commit/ce06b472af9e30ac5d0792f7a8b6f42170a94627)]:
|
|
21
|
+
- @graphorin/core@0.6.0
|
|
22
|
+
|
|
23
|
+
## 0.5.0
|
|
24
|
+
|
|
25
|
+
First version published to the npm registry (with Sigstore build
|
|
26
|
+
provenance). The 0.2.0, 0.3.0, and 0.4.0 versions were internal lockstep
|
|
27
|
+
milestones and were never published. All `@graphorin/*` packages release
|
|
28
|
+
lockstep at the same version; the full release notes for 0.2.0-0.5.0 live
|
|
29
|
+
in the repository-level
|
|
30
|
+
[CHANGELOG](https://github.com/o-stepper/graphorin/blob/main/CHANGELOG.md).
|
|
31
|
+
|
|
3
32
|
## 0.1.0
|
|
4
33
|
|
|
5
34
|
### Minor Changes
|
|
6
35
|
|
|
7
36
|
- Initial release. Default in-process embedder for the Graphorin
|
|
8
|
-
framework
|
|
37
|
+
framework - wraps `@huggingface/transformers` for the multilingual
|
|
9
38
|
`Xenova/multilingual-e5-base` model.
|
package/README.md
CHANGED
|
@@ -3,7 +3,7 @@
|
|
|
3
3
|
> Default in-process embedder for the Graphorin framework.
|
|
4
4
|
|
|
5
5
|
`@graphorin/embedder-transformersjs` wraps `@huggingface/transformers@^4.1.0`
|
|
6
|
-
to produce dense embeddings inside the Graphorin process
|
|
6
|
+
to produce dense embeddings inside the Graphorin process - no external
|
|
7
7
|
service, no network call after the first model download. The default
|
|
8
8
|
model is `Xenova/multilingual-e5-base` (768-dim, multilingual). Other
|
|
9
9
|
multilingual variants from the same family (`multilingual-e5-small`,
|
|
@@ -13,11 +13,11 @@ factory.
|
|
|
13
13
|
The package implements the `EmbedderProvider` contract from
|
|
14
14
|
`@graphorin/core/contracts`:
|
|
15
15
|
|
|
16
|
-
- `id()`
|
|
16
|
+
- `id()` - canonical id, e.g.
|
|
17
17
|
`'transformersjs:Xenova/multilingual-e5-base@768'`.
|
|
18
|
-
- `dim()`
|
|
19
|
-
- `configHash()`
|
|
20
|
-
- `embed(texts)`
|
|
18
|
+
- `dim()` - output vector dimensionality (resolved at first `embed`).
|
|
19
|
+
- `configHash()` - deterministic hex hash over canonical-JSON config.
|
|
20
|
+
- `embed(texts)` - batched, lazy. Returns one `Float32Array` per text.
|
|
21
21
|
|
|
22
22
|
## Install
|
|
23
23
|
|
|
@@ -43,7 +43,7 @@ console.log(embedder.id(), embedder.dim(), vec.length);
|
|
|
43
43
|
## E5 asymmetric prefixes
|
|
44
44
|
|
|
45
45
|
E5-family models (the default, and any model id carrying an `e5` token) require
|
|
46
|
-
asymmetric `query:` / `passage:` prefixes
|
|
46
|
+
asymmetric `query:` / `passage:` prefixes - omitting them measurably degrades
|
|
47
47
|
retrieval. The embedder applies them automatically: pass `taskType: 'query'`
|
|
48
48
|
when embedding a search query and `taskType: 'passage'` (the default) for stored
|
|
49
49
|
content. The Graphorin memory tiers thread this for you (`query` on search,
|
|
@@ -52,7 +52,7 @@ content. The Graphorin memory tiers thread this for you (`query` on search,
|
|
|
52
52
|
> **Migration:** enabling the prefixes (the new default for E5) changes the
|
|
53
53
|
> embedder's `configHash`, and therefore its `id`. Under the default
|
|
54
54
|
> `lock-on-first` policy an existing index built before this change reports an
|
|
55
|
-
> embedder mismatch
|
|
55
|
+
> embedder mismatch - run `graphorin memory migrate` to re-embed it, or pass
|
|
56
56
|
> `disableTaskPrefix: true` to keep the old (unprefixed) behaviour and id.
|
|
57
57
|
|
|
58
58
|
## Cache directory
|
|
@@ -81,4 +81,4 @@ MIT © 2026 Oleksiy Stepurenko.
|
|
|
81
81
|
|
|
82
82
|
---
|
|
83
83
|
|
|
84
|
-
**Project Graphorin** · v0.
|
|
84
|
+
**Project Graphorin** · v0.6.0 · MIT License · © 2026 Oleksiy Stepurenko · <https://github.com/o-stepper/graphorin>
|
package/dist/index.d.ts
CHANGED
|
@@ -3,7 +3,7 @@ import { EmbedOptions, EmbedderProvider } from "@graphorin/core/contracts";
|
|
|
3
3
|
//#region src/index.d.ts
|
|
4
4
|
|
|
5
5
|
/** Canonical version constant. Mirrors the `package.json` version. */
|
|
6
|
-
declare const VERSION = "0.
|
|
6
|
+
declare const VERSION = "0.6.0";
|
|
7
7
|
/**
|
|
8
8
|
* Pooling strategy. Defaults to `'mean'` per the multilingual-e5 model
|
|
9
9
|
* card.
|
|
@@ -39,7 +39,7 @@ interface TransformersJsEmbedderOptions {
|
|
|
39
39
|
/** Override device (`'cpu'`, `'webgpu'`, …). Default `'cpu'`. */
|
|
40
40
|
readonly device?: string;
|
|
41
41
|
/**
|
|
42
|
-
* Override the underlying `pipeline` factory
|
|
42
|
+
* Override the underlying `pipeline` factory - used by the test
|
|
43
43
|
* suite to inject a stub. Production callers should leave this
|
|
44
44
|
* unset so the package lazily loads `@huggingface/transformers`.
|
|
45
45
|
*/
|
|
@@ -114,6 +114,16 @@ declare class TransformersJsEmbedder implements EmbedderProvider {
|
|
|
114
114
|
#private;
|
|
115
115
|
constructor(options: TransformersJsEmbedderOptions);
|
|
116
116
|
id(): string;
|
|
117
|
+
/**
|
|
118
|
+
* Output dimension - the explicit `dim` option, a known-model
|
|
119
|
+
* default, or the width resolved from the first `embed()`.
|
|
120
|
+
* periphery-05 (the PS-11 fix ported from the Ollama embedder):
|
|
121
|
+
* throws for an unknown model with no `dim` hint instead of silently
|
|
122
|
+
* assuming 768 - a wrong assumed width bakes a wrong-width id AND a
|
|
123
|
+
* wrong-width vec0 table, and the id then CHANGES after the first
|
|
124
|
+
* `embed()` resolves the truth, which `lock-on-first` reads as an
|
|
125
|
+
* embedder swap.
|
|
126
|
+
*/
|
|
117
127
|
dim(): number;
|
|
118
128
|
configHash(): string;
|
|
119
129
|
embed(texts: ReadonlyArray<string>, opts?: EmbedOptions): Promise<ReadonlyArray<Float32Array>>;
|
package/dist/index.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.d.ts","names":[],"sources":["../src/index.ts"],"sourcesContent":[],"mappings":";;;;;AAiGY,cAlFC,OAAA,GAkFc,OAId;AAWb;AAWA;AAWA;;;;
|
|
1
|
+
{"version":3,"file":"index.d.ts","names":[],"sources":["../src/index.ts"],"sourcesContent":[],"mappings":";;;;;AAiGY,cAlFC,OAAA,GAkFc,OAId;AAWb;AAWA;AAWA;;;;AAsE2B,KArLf,OAAA,GAqLe,MAAA,GAAA,KAAA,GAAA,aAAA,GAAA,KAAA,GAAA,YAAA,GAAA,MAAA;;;;;AAkF3B;AAsBgB,UAtRC,6BAAA,CAsRQ;EAYT;;;qBA9RK;;;;;;;;;;;;;;;;;;;;;;;6BAuBQ;;;;;;;;;;;;;;;;;;;;;;;;;KA0BjB,gBAAA;YAEY;;WAAuC;MAC1D;QACG;;;;;;;;;KAUI,eAAA;;;;;MAIP,QAAQ;;;;;;;cAWA,sBAAA,SAA+B,KAAA;;;;;;;;;;iBAW5B,4BAAA,WACL,gCACR;;;;;;cASU,sBAAA,YAAkC;;uBAexB;;;;;;;;;;;;;;eAqDZ,8BACD,eACL,QAAQ,cAAc;;;iBAkFX,oCAAA,CAAA;;;;;;;iBAsBA,SAAA;;;;;;;;;iBAYA,mBAAA"}
|
package/dist/index.js
CHANGED
|
@@ -2,7 +2,7 @@ import { createHash } from "node:crypto";
|
|
|
2
2
|
|
|
3
3
|
//#region src/index.ts
|
|
4
4
|
/**
|
|
5
|
-
* @graphorin/embedder-transformersjs
|
|
5
|
+
* @graphorin/embedder-transformersjs - default in-process embedder.
|
|
6
6
|
*
|
|
7
7
|
* Wraps `@huggingface/transformers@^4.1.0` to produce dense embeddings
|
|
8
8
|
* inside the Graphorin process. Default model
|
|
@@ -12,7 +12,7 @@ import { createHash } from "node:crypto";
|
|
|
12
12
|
* @packageDocumentation
|
|
13
13
|
*/
|
|
14
14
|
/** Canonical version constant. Mirrors the `package.json` version. */
|
|
15
|
-
const VERSION = "0.
|
|
15
|
+
const VERSION = "0.6.0";
|
|
16
16
|
const DEFAULT_MODEL = "Xenova/multilingual-e5-base";
|
|
17
17
|
const DEFAULT_DIM = 768;
|
|
18
18
|
/**
|
|
@@ -66,12 +66,21 @@ var TransformersJsEmbedder = class {
|
|
|
66
66
|
this.#taskPrefix = isE5Model(this.#model) && options.disableTaskPrefix !== true;
|
|
67
67
|
}
|
|
68
68
|
id() {
|
|
69
|
-
|
|
70
|
-
return `transformersjs:${this.#model}@${dim}`;
|
|
69
|
+
return `transformersjs:${this.#model}@${this.dim()}`;
|
|
71
70
|
}
|
|
71
|
+
/**
|
|
72
|
+
* Output dimension - the explicit `dim` option, a known-model
|
|
73
|
+
* default, or the width resolved from the first `embed()`.
|
|
74
|
+
* periphery-05 (the PS-11 fix ported from the Ollama embedder):
|
|
75
|
+
* throws for an unknown model with no `dim` hint instead of silently
|
|
76
|
+
* assuming 768 - a wrong assumed width bakes a wrong-width id AND a
|
|
77
|
+
* wrong-width vec0 table, and the id then CHANGES after the first
|
|
78
|
+
* `embed()` resolves the truth, which `lock-on-first` reads as an
|
|
79
|
+
* embedder swap.
|
|
80
|
+
*/
|
|
72
81
|
dim() {
|
|
73
82
|
if (this.#resolvedDim !== null) return this.#resolvedDim;
|
|
74
|
-
|
|
83
|
+
throw new Error(`[graphorin/embedder-transformersjs] Unknown embedding width for model '${this.#model}'. Pass { dim: <width> } to createTransformersJsEmbedder(...) (or call embed() once before registration so the width is resolved from the model output).`);
|
|
75
84
|
}
|
|
76
85
|
configHash() {
|
|
77
86
|
return canonicalConfigHash({
|
|
@@ -93,6 +102,7 @@ var TransformersJsEmbedder = class {
|
|
|
93
102
|
...opts.signal !== void 0 ? { signal: opts.signal } : {}
|
|
94
103
|
});
|
|
95
104
|
const lastDim = result.dims[result.dims.length - 1] ?? this.#resolvedDim ?? DEFAULT_DIM;
|
|
105
|
+
if (this.#resolvedDim !== null && lastDim !== this.#resolvedDim) throw new Error(`[graphorin/embedder-transformersjs] Model '${this.#model}' produced ${lastDim}-dim vectors but the embedder is bound to ${this.#resolvedDim} - check the 'dim' option.`);
|
|
96
106
|
if (this.#resolvedDim === null) this.#resolvedDim = lastDim;
|
|
97
107
|
const dim = lastDim;
|
|
98
108
|
const out = [];
|
package/dist/index.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"index.js","names":["#model","#pooling","#normalize","#revision","#cacheDir","#dtype","#device","#pipelineFactory","#taskPrefix","#resolvedDim","#getExtractor","out: Float32Array[]","#extractor","#loading","CACHED_PIPELINE_FACTORY: PipelineFactory | null","KNOWN_DIMS: ReadonlyMap<string, number>"],"sources":["../src/index.ts"],"sourcesContent":["/**\n * @graphorin/embedder-transformersjs — default in-process embedder.\n *\n * Wraps `@huggingface/transformers@^4.1.0` to produce dense embeddings\n * inside the Graphorin process. Default model\n * `Xenova/multilingual-e5-base` (768-dim, multilingual; DEC-130 /\n * ADR-025).\n *\n * @packageDocumentation\n */\n\nimport { createHash } from 'node:crypto';\nimport type { EmbedderProvider, EmbedOptions } from '@graphorin/core/contracts';\n\n/** Canonical version constant. Mirrors the `package.json` version. */\nexport const VERSION = '0.5.0';\n\n/**\n * Pooling strategy. Defaults to `'mean'` per the multilingual-e5 model\n * card.\n *\n * @stable\n */\nexport type Pooling = 'mean' | 'cls' | 'first_token' | 'eos' | 'last_token' | 'none';\n\n/**\n * Configuration accepted by {@link createTransformersJsEmbedder}.\n *\n * @stable\n */\nexport interface TransformersJsEmbedderOptions {\n /** Default `'Xenova/multilingual-e5-base'` (768-dim). */\n readonly model?: string;\n /** Default `'mean'`. */\n readonly pooling?: Pooling;\n /** Default `true`. */\n readonly normalize?: boolean;\n /** Optional model revision pin (`'main'` if unset). */\n readonly revision?: string;\n /**\n * Optional cache directory. When unset, the embedder honours\n * `process.env.GRAPHORIN_CACHE_DIR`, otherwise falls back to the\n * Hugging Face default (`os.homedir()/.cache/huggingface/hub`).\n */\n readonly cacheDir?: string;\n /**\n * Optional dtype hint (`'fp32' | 'fp16' | 'q8' | 'q4'`). When unset,\n * the runtime picks the model's recommended default.\n */\n readonly dtype?: string;\n /** Override device (`'cpu'`, `'webgpu'`, …). Default `'cpu'`. */\n readonly device?: string;\n /**\n * Override the underlying `pipeline` factory — used by the test\n * suite to inject a stub. Production callers should leave this\n * unset so the package lazily loads `@huggingface/transformers`.\n */\n readonly pipelineFactory?: PipelineFactory;\n /**\n * Optional dimensionality hint. When the caller knows the output\n * dimension up-front, it is included in the canonical id without\n * waiting for the first `embed()` call.\n */\n readonly dim?: number;\n /**\n * Disable the automatic E5 `query:` / `passage:` prefixing (PS-10). The\n * prefixes are applied by default for E5-family models (the multilingual-e5\n * default and any model whose id carries an `e5` token), because the E5 model\n * card requires them and omitting them measurably degrades retrieval. Set\n * this to `true` only if your inputs are already prefixed or you use a\n * non-standard E5 export. Toggling it changes the canonical `configHash`\n * (and thus the embedder id), which triggers a re-embedding migration.\n */\n readonly disableTaskPrefix?: boolean;\n}\n\n/**\n * Tiny structural shape of `@huggingface/transformers`' feature-\n * extraction pipeline used by this package. Declared inline so the\n * embedder does not import the heavy peer at build time.\n *\n * @internal\n */\nexport type FeatureExtractor = (\n texts: string | readonly string[],\n options?: { pooling?: Pooling; normalize?: boolean; signal?: AbortSignal },\n) => Promise<{\n data: Float32Array;\n dims: readonly number[];\n tolist?(): unknown;\n}>;\n\n/**\n * Pipeline-factory shape used for dependency injection in tests.\n *\n * @internal\n */\nexport type PipelineFactory = (\n task: 'feature-extraction',\n model: string,\n opts: { revision?: string; cache_dir?: string; dtype?: string; device?: string },\n) => Promise<FeatureExtractor>;\n\nconst DEFAULT_MODEL = 'Xenova/multilingual-e5-base';\nconst DEFAULT_DIM = 768;\n\n/**\n * Raised when the underlying transformer model cannot be loaded\n * (offline / corporate firewall / wrong cache dir).\n *\n * @stable\n */\nexport class EmbedderModelLoadError extends Error {\n override readonly name = 'EmbedderModelLoadError';\n}\n\n/**\n * Build a `TransformersJsEmbedder` instance. Lazy: the underlying\n * pipeline is constructed on the first `embed()` call so packaging\n * the embedder does not pay the model-load cost.\n *\n * @stable\n */\nexport function createTransformersJsEmbedder(\n options: TransformersJsEmbedderOptions = {},\n): TransformersJsEmbedder {\n return new TransformersJsEmbedder(options);\n}\n\n/**\n * `EmbedderProvider` implementation backed by `@huggingface/transformers`.\n *\n * @stable\n */\nexport class TransformersJsEmbedder implements EmbedderProvider {\n readonly #model: string;\n readonly #pooling: Pooling;\n readonly #normalize: boolean;\n readonly #revision: string | undefined;\n readonly #cacheDir: string | undefined;\n readonly #dtype: string | undefined;\n readonly #device: string | undefined;\n readonly #pipelineFactory: PipelineFactory | undefined;\n /** Whether to apply E5 `query:` / `passage:` prefixes (PS-10). */\n readonly #taskPrefix: boolean;\n #extractor: FeatureExtractor | null = null;\n #loading: Promise<FeatureExtractor> | null = null;\n #resolvedDim: number | null;\n\n constructor(options: TransformersJsEmbedderOptions) {\n this.#model = options.model ?? DEFAULT_MODEL;\n this.#pooling = options.pooling ?? 'mean';\n this.#normalize = options.normalize ?? true;\n this.#revision = options.revision;\n this.#cacheDir = options.cacheDir ?? process.env.GRAPHORIN_CACHE_DIR;\n this.#dtype = options.dtype;\n this.#device = options.device;\n this.#pipelineFactory = options.pipelineFactory;\n this.#resolvedDim = options.dim ?? guessDefaultDim(this.#model);\n this.#taskPrefix = isE5Model(this.#model) && options.disableTaskPrefix !== true;\n }\n\n id(): string {\n const dim = this.#resolvedDim ?? DEFAULT_DIM;\n return `transformersjs:${this.#model}@${dim}`;\n }\n\n dim(): number {\n if (this.#resolvedDim !== null) return this.#resolvedDim;\n return DEFAULT_DIM;\n }\n\n configHash(): string {\n return canonicalConfigHash({\n adapter: 'transformersjs',\n model: this.#model,\n pooling: this.#pooling,\n normalize: this.#normalize,\n revision: this.#revision ?? null,\n dtype: this.#dtype ?? null,\n device: this.#device ?? null,\n // PS-10: the prefix policy changes the embeddings, so it must change the\n // id. Only added when active (E5 + not disabled) so non-E5 ids — and the\n // historical hash of an E5 model with prefixing turned off — are stable.\n ...(this.#taskPrefix ? { taskPrefix: 'e5' as const } : {}),\n });\n }\n\n async embed(\n texts: ReadonlyArray<string>,\n opts: EmbedOptions = {},\n ): Promise<ReadonlyArray<Float32Array>> {\n if (texts.length === 0) return [];\n const extractor = await this.#getExtractor();\n // PS-10: E5 models require an asymmetric `query:` / `passage:` prefix.\n // Default to `passage` (the indexing role) when the caller doesn't specify.\n const inputs = this.#taskPrefix\n ? texts.map((t) => `${opts.taskType ?? 'passage'}: ${t}`)\n : [...texts];\n const result = await extractor(inputs, {\n pooling: this.#pooling,\n normalize: this.#normalize,\n ...(opts.signal !== undefined ? { signal: opts.signal } : {}),\n });\n const lastDim = result.dims[result.dims.length - 1] ?? this.#resolvedDim ?? DEFAULT_DIM;\n if (this.#resolvedDim === null) {\n this.#resolvedDim = lastDim;\n }\n const dim = lastDim;\n const out: Float32Array[] = [];\n for (let i = 0; i < texts.length; i++) {\n const start = i * dim;\n out.push(result.data.slice(start, start + dim));\n }\n return out;\n }\n\n async #getExtractor(): Promise<FeatureExtractor> {\n if (this.#extractor !== null) return this.#extractor;\n if (this.#loading !== null) return this.#loading;\n const loader = this.#pipelineFactory ?? (await loadDefaultPipelineFactory());\n this.#loading = loader('feature-extraction', this.#model, {\n ...(this.#revision !== undefined ? { revision: this.#revision } : {}),\n ...(this.#cacheDir !== undefined ? { cache_dir: this.#cacheDir } : {}),\n ...(this.#dtype !== undefined ? { dtype: this.#dtype } : {}),\n ...(this.#device !== undefined ? { device: this.#device } : {}),\n })\n .then((pipe) => {\n this.#extractor = pipe;\n this.#loading = null;\n return pipe;\n })\n .catch((err) => {\n this.#loading = null;\n throw new EmbedderModelLoadError(\n `[graphorin/embedder-transformersjs] failed to load model '${this.#model}'. ` +\n 'See the offline-install guide for instructions on pre-downloading the model.',\n { cause: err },\n );\n });\n return this.#loading;\n }\n}\n\n/** @internal */\nlet CACHED_PIPELINE_FACTORY: PipelineFactory | null = null;\n\nasync function loadDefaultPipelineFactory(): Promise<PipelineFactory> {\n if (CACHED_PIPELINE_FACTORY !== null) return CACHED_PIPELINE_FACTORY;\n try {\n const mod = (await import('@huggingface/transformers')) as unknown as {\n pipeline: PipelineFactory;\n };\n CACHED_PIPELINE_FACTORY = mod.pipeline;\n return CACHED_PIPELINE_FACTORY;\n } catch (err) {\n throw new EmbedderModelLoadError(\n \"[graphorin/embedder-transformersjs] required peer '@huggingface/transformers' is not installed.\",\n { cause: err },\n );\n }\n}\n\n/** Test-only cache reset. */\nexport function _resetPipelineFactoryCacheForTesting(): void {\n CACHED_PIPELINE_FACTORY = null;\n}\n\nconst KNOWN_DIMS: ReadonlyMap<string, number> = new Map([\n ['Xenova/multilingual-e5-small', 384],\n ['Xenova/multilingual-e5-base', 768],\n ['Xenova/multilingual-e5-large', 1024],\n ['Xenova/bge-m3', 1024],\n ['onnx-community/all-MiniLM-L6-v2-ONNX', 384],\n]);\n\nfunction guessDefaultDim(model: string): number | null {\n return KNOWN_DIMS.get(model) ?? null;\n}\n\n/**\n * True when a model id belongs to the E5 family, which requires asymmetric\n * `query:` / `passage:` prefixes (PS-10). Matches an `e5` token bounded by a\n * path / dash / underscore so it covers `multilingual-e5-base`, `e5-large`,\n * `intfloat/e5-mistral`, etc. without false-matching unrelated names.\n */\nexport function isE5Model(model: string): boolean {\n return /(?:^|[/_-])e5(?:[/_-]|$)/i.test(model);\n}\n\n/**\n * Canonical-JSON deterministic hash of an embedder configuration.\n * Object keys are sorted lexicographically; primitives flow through as\n * `JSON.stringify` would render them. Used by the multi-table per-\n * embedder vec0 layout to tell drift apart from a true model swap.\n *\n * @stable\n */\nexport function canonicalConfigHash(config: unknown): string {\n const canonical = canonicalize(config);\n return createHash('sha256').update(canonical, 'utf8').digest('hex');\n}\n\nfunction canonicalize(value: unknown): string {\n if (value === null) return 'null';\n if (typeof value === 'boolean') return value ? 'true' : 'false';\n if (typeof value === 'number') return Number.isFinite(value) ? value.toString() : 'null';\n if (typeof value === 'string') return JSON.stringify(value);\n if (Array.isArray(value)) {\n return `[${value.map(canonicalize).join(',')}]`;\n }\n if (typeof value === 'object') {\n const obj = value as Record<string, unknown>;\n const keys = Object.keys(obj).sort();\n return `{${keys.map((k) => `${JSON.stringify(k)}:${canonicalize(obj[k])}`).join(',')}}`;\n }\n return 'null';\n}\n"],"mappings":";;;;;;;;;;;;;;AAeA,MAAa,UAAU;AAwFvB,MAAM,gBAAgB;AACtB,MAAM,cAAc;;;;;;;AAQpB,IAAa,yBAAb,cAA4C,MAAM;CAChD,AAAkB,OAAO;;;;;;;;;AAU3B,SAAgB,6BACd,UAAyC,EAAE,EACnB;AACxB,QAAO,IAAI,uBAAuB,QAAQ;;;;;;;AAQ5C,IAAa,yBAAb,MAAgE;CAC9D,CAASA;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;;CAET,CAASC;CACT,aAAsC;CACtC,WAA6C;CAC7C;CAEA,YAAY,SAAwC;AAClD,QAAKR,QAAS,QAAQ,SAAS;AAC/B,QAAKC,UAAW,QAAQ,WAAW;AACnC,QAAKC,YAAa,QAAQ,aAAa;AACvC,QAAKC,WAAY,QAAQ;AACzB,QAAKC,WAAY,QAAQ,YAAY,QAAQ,IAAI;AACjD,QAAKC,QAAS,QAAQ;AACtB,QAAKC,SAAU,QAAQ;AACvB,QAAKC,kBAAmB,QAAQ;AAChC,QAAKE,cAAe,QAAQ,OAAO,gBAAgB,MAAKT,MAAO;AAC/D,QAAKQ,aAAc,UAAU,MAAKR,MAAO,IAAI,QAAQ,sBAAsB;;CAG7E,KAAa;EACX,MAAM,MAAM,MAAKS,eAAgB;AACjC,SAAO,kBAAkB,MAAKT,MAAO,GAAG;;CAG1C,MAAc;AACZ,MAAI,MAAKS,gBAAiB,KAAM,QAAO,MAAKA;AAC5C,SAAO;;CAGT,aAAqB;AACnB,SAAO,oBAAoB;GACzB,SAAS;GACT,OAAO,MAAKT;GACZ,SAAS,MAAKC;GACd,WAAW,MAAKC;GAChB,UAAU,MAAKC,YAAa;GAC5B,OAAO,MAAKE,SAAU;GACtB,QAAQ,MAAKC,UAAW;GAIxB,GAAI,MAAKE,aAAc,EAAE,YAAY,MAAe,GAAG,EAAE;GAC1D,CAAC;;CAGJ,MAAM,MACJ,OACA,OAAqB,EAAE,EACe;AACtC,MAAI,MAAM,WAAW,EAAG,QAAO,EAAE;EAOjC,MAAM,SAAS,OANG,MAAM,MAAKE,cAAe,EAG7B,MAAKF,aAChB,MAAM,KAAK,MAAM,GAAG,KAAK,YAAY,UAAU,IAAI,IAAI,GACvD,CAAC,GAAG,MAAM,EACyB;GACrC,SAAS,MAAKP;GACd,WAAW,MAAKC;GAChB,GAAI,KAAK,WAAW,SAAY,EAAE,QAAQ,KAAK,QAAQ,GAAG,EAAE;GAC7D,CAAC;EACF,MAAM,UAAU,OAAO,KAAK,OAAO,KAAK,SAAS,MAAM,MAAKO,eAAgB;AAC5E,MAAI,MAAKA,gBAAiB,KACxB,OAAKA,cAAe;EAEtB,MAAM,MAAM;EACZ,MAAME,MAAsB,EAAE;AAC9B,OAAK,IAAI,IAAI,GAAG,IAAI,MAAM,QAAQ,KAAK;GACrC,MAAM,QAAQ,IAAI;AAClB,OAAI,KAAK,OAAO,KAAK,MAAM,OAAO,QAAQ,IAAI,CAAC;;AAEjD,SAAO;;CAGT,OAAMD,eAA2C;AAC/C,MAAI,MAAKE,cAAe,KAAM,QAAO,MAAKA;AAC1C,MAAI,MAAKC,YAAa,KAAM,QAAO,MAAKA;AAExC,QAAKA,WADU,MAAKN,mBAAqB,MAAM,4BAA4B,EACpD,sBAAsB,MAAKP,OAAQ;GACxD,GAAI,MAAKG,aAAc,SAAY,EAAE,UAAU,MAAKA,UAAW,GAAG,EAAE;GACpE,GAAI,MAAKC,aAAc,SAAY,EAAE,WAAW,MAAKA,UAAW,GAAG,EAAE;GACrE,GAAI,MAAKC,UAAW,SAAY,EAAE,OAAO,MAAKA,OAAQ,GAAG,EAAE;GAC3D,GAAI,MAAKC,WAAY,SAAY,EAAE,QAAQ,MAAKA,QAAS,GAAG,EAAE;GAC/D,CAAC,CACC,MAAM,SAAS;AACd,SAAKM,YAAa;AAClB,SAAKC,UAAW;AAChB,UAAO;IACP,CACD,OAAO,QAAQ;AACd,SAAKA,UAAW;AAChB,SAAM,IAAI,uBACR,6DAA6D,MAAKb,MAAO,kFAEzE,EAAE,OAAO,KAAK,CACf;IACD;AACJ,SAAO,MAAKa;;;;AAKhB,IAAIC,0BAAkD;AAEtD,eAAe,6BAAuD;AACpE,KAAI,4BAA4B,KAAM,QAAO;AAC7C,KAAI;AAIF,6BAHa,MAAM,OAAO,8BAGI;AAC9B,SAAO;UACA,KAAK;AACZ,QAAM,IAAI,uBACR,mGACA,EAAE,OAAO,KAAK,CACf;;;;AAKL,SAAgB,uCAA6C;AAC3D,2BAA0B;;AAG5B,MAAMC,aAA0C,IAAI,IAAI;CACtD,CAAC,gCAAgC,IAAI;CACrC,CAAC,+BAA+B,IAAI;CACpC,CAAC,gCAAgC,KAAK;CACtC,CAAC,iBAAiB,KAAK;CACvB,CAAC,wCAAwC,IAAI;CAC9C,CAAC;AAEF,SAAS,gBAAgB,OAA8B;AACrD,QAAO,WAAW,IAAI,MAAM,IAAI;;;;;;;;AASlC,SAAgB,UAAU,OAAwB;AAChD,QAAO,4BAA4B,KAAK,MAAM;;;;;;;;;;AAWhD,SAAgB,oBAAoB,QAAyB;CAC3D,MAAM,YAAY,aAAa,OAAO;AACtC,QAAO,WAAW,SAAS,CAAC,OAAO,WAAW,OAAO,CAAC,OAAO,MAAM;;AAGrE,SAAS,aAAa,OAAwB;AAC5C,KAAI,UAAU,KAAM,QAAO;AAC3B,KAAI,OAAO,UAAU,UAAW,QAAO,QAAQ,SAAS;AACxD,KAAI,OAAO,UAAU,SAAU,QAAO,OAAO,SAAS,MAAM,GAAG,MAAM,UAAU,GAAG;AAClF,KAAI,OAAO,UAAU,SAAU,QAAO,KAAK,UAAU,MAAM;AAC3D,KAAI,MAAM,QAAQ,MAAM,CACtB,QAAO,IAAI,MAAM,IAAI,aAAa,CAAC,KAAK,IAAI,CAAC;AAE/C,KAAI,OAAO,UAAU,UAAU;EAC7B,MAAM,MAAM;AAEZ,SAAO,IADM,OAAO,KAAK,IAAI,CAAC,MAAM,CACpB,KAAK,MAAM,GAAG,KAAK,UAAU,EAAE,CAAC,GAAG,aAAa,IAAI,GAAG,GAAG,CAAC,KAAK,IAAI,CAAC;;AAEvF,QAAO"}
|
|
1
|
+
{"version":3,"file":"index.js","names":["#model","#pooling","#normalize","#revision","#cacheDir","#dtype","#device","#pipelineFactory","#taskPrefix","#resolvedDim","#getExtractor","out: Float32Array[]","#extractor","#loading","CACHED_PIPELINE_FACTORY: PipelineFactory | null","KNOWN_DIMS: ReadonlyMap<string, number>"],"sources":["../src/index.ts"],"sourcesContent":["/**\n * @graphorin/embedder-transformersjs - default in-process embedder.\n *\n * Wraps `@huggingface/transformers@^4.1.0` to produce dense embeddings\n * inside the Graphorin process. Default model\n * `Xenova/multilingual-e5-base` (768-dim, multilingual; DEC-130 /\n * ADR-025).\n *\n * @packageDocumentation\n */\n\nimport { createHash } from 'node:crypto';\nimport type { EmbedderProvider, EmbedOptions } from '@graphorin/core/contracts';\n\n/** Canonical version constant. Mirrors the `package.json` version. */\nexport const VERSION = '0.6.0';\n\n/**\n * Pooling strategy. Defaults to `'mean'` per the multilingual-e5 model\n * card.\n *\n * @stable\n */\nexport type Pooling = 'mean' | 'cls' | 'first_token' | 'eos' | 'last_token' | 'none';\n\n/**\n * Configuration accepted by {@link createTransformersJsEmbedder}.\n *\n * @stable\n */\nexport interface TransformersJsEmbedderOptions {\n /** Default `'Xenova/multilingual-e5-base'` (768-dim). */\n readonly model?: string;\n /** Default `'mean'`. */\n readonly pooling?: Pooling;\n /** Default `true`. */\n readonly normalize?: boolean;\n /** Optional model revision pin (`'main'` if unset). */\n readonly revision?: string;\n /**\n * Optional cache directory. When unset, the embedder honours\n * `process.env.GRAPHORIN_CACHE_DIR`, otherwise falls back to the\n * Hugging Face default (`os.homedir()/.cache/huggingface/hub`).\n */\n readonly cacheDir?: string;\n /**\n * Optional dtype hint (`'fp32' | 'fp16' | 'q8' | 'q4'`). When unset,\n * the runtime picks the model's recommended default.\n */\n readonly dtype?: string;\n /** Override device (`'cpu'`, `'webgpu'`, …). Default `'cpu'`. */\n readonly device?: string;\n /**\n * Override the underlying `pipeline` factory - used by the test\n * suite to inject a stub. Production callers should leave this\n * unset so the package lazily loads `@huggingface/transformers`.\n */\n readonly pipelineFactory?: PipelineFactory;\n /**\n * Optional dimensionality hint. When the caller knows the output\n * dimension up-front, it is included in the canonical id without\n * waiting for the first `embed()` call.\n */\n readonly dim?: number;\n /**\n * Disable the automatic E5 `query:` / `passage:` prefixing (PS-10). The\n * prefixes are applied by default for E5-family models (the multilingual-e5\n * default and any model whose id carries an `e5` token), because the E5 model\n * card requires them and omitting them measurably degrades retrieval. Set\n * this to `true` only if your inputs are already prefixed or you use a\n * non-standard E5 export. Toggling it changes the canonical `configHash`\n * (and thus the embedder id), which triggers a re-embedding migration.\n */\n readonly disableTaskPrefix?: boolean;\n}\n\n/**\n * Tiny structural shape of `@huggingface/transformers`' feature-\n * extraction pipeline used by this package. Declared inline so the\n * embedder does not import the heavy peer at build time.\n *\n * @internal\n */\nexport type FeatureExtractor = (\n texts: string | readonly string[],\n options?: { pooling?: Pooling; normalize?: boolean; signal?: AbortSignal },\n) => Promise<{\n data: Float32Array;\n dims: readonly number[];\n tolist?(): unknown;\n}>;\n\n/**\n * Pipeline-factory shape used for dependency injection in tests.\n *\n * @internal\n */\nexport type PipelineFactory = (\n task: 'feature-extraction',\n model: string,\n opts: { revision?: string; cache_dir?: string; dtype?: string; device?: string },\n) => Promise<FeatureExtractor>;\n\nconst DEFAULT_MODEL = 'Xenova/multilingual-e5-base';\nconst DEFAULT_DIM = 768;\n\n/**\n * Raised when the underlying transformer model cannot be loaded\n * (offline / corporate firewall / wrong cache dir).\n *\n * @stable\n */\nexport class EmbedderModelLoadError extends Error {\n override readonly name = 'EmbedderModelLoadError';\n}\n\n/**\n * Build a `TransformersJsEmbedder` instance. Lazy: the underlying\n * pipeline is constructed on the first `embed()` call so packaging\n * the embedder does not pay the model-load cost.\n *\n * @stable\n */\nexport function createTransformersJsEmbedder(\n options: TransformersJsEmbedderOptions = {},\n): TransformersJsEmbedder {\n return new TransformersJsEmbedder(options);\n}\n\n/**\n * `EmbedderProvider` implementation backed by `@huggingface/transformers`.\n *\n * @stable\n */\nexport class TransformersJsEmbedder implements EmbedderProvider {\n readonly #model: string;\n readonly #pooling: Pooling;\n readonly #normalize: boolean;\n readonly #revision: string | undefined;\n readonly #cacheDir: string | undefined;\n readonly #dtype: string | undefined;\n readonly #device: string | undefined;\n readonly #pipelineFactory: PipelineFactory | undefined;\n /** Whether to apply E5 `query:` / `passage:` prefixes (PS-10). */\n readonly #taskPrefix: boolean;\n #extractor: FeatureExtractor | null = null;\n #loading: Promise<FeatureExtractor> | null = null;\n #resolvedDim: number | null;\n\n constructor(options: TransformersJsEmbedderOptions) {\n this.#model = options.model ?? DEFAULT_MODEL;\n this.#pooling = options.pooling ?? 'mean';\n this.#normalize = options.normalize ?? true;\n this.#revision = options.revision;\n this.#cacheDir = options.cacheDir ?? process.env.GRAPHORIN_CACHE_DIR;\n this.#dtype = options.dtype;\n this.#device = options.device;\n this.#pipelineFactory = options.pipelineFactory;\n this.#resolvedDim = options.dim ?? guessDefaultDim(this.#model);\n this.#taskPrefix = isE5Model(this.#model) && options.disableTaskPrefix !== true;\n }\n\n id(): string {\n return `transformersjs:${this.#model}@${this.dim()}`;\n }\n\n /**\n * Output dimension - the explicit `dim` option, a known-model\n * default, or the width resolved from the first `embed()`.\n * periphery-05 (the PS-11 fix ported from the Ollama embedder):\n * throws for an unknown model with no `dim` hint instead of silently\n * assuming 768 - a wrong assumed width bakes a wrong-width id AND a\n * wrong-width vec0 table, and the id then CHANGES after the first\n * `embed()` resolves the truth, which `lock-on-first` reads as an\n * embedder swap.\n */\n dim(): number {\n if (this.#resolvedDim !== null) return this.#resolvedDim;\n throw new Error(\n `[graphorin/embedder-transformersjs] Unknown embedding width for model '${this.#model}'. ` +\n `Pass { dim: <width> } to createTransformersJsEmbedder(...) (or call embed() once before ` +\n `registration so the width is resolved from the model output).`,\n );\n }\n\n configHash(): string {\n return canonicalConfigHash({\n adapter: 'transformersjs',\n model: this.#model,\n pooling: this.#pooling,\n normalize: this.#normalize,\n revision: this.#revision ?? null,\n dtype: this.#dtype ?? null,\n device: this.#device ?? null,\n // PS-10: the prefix policy changes the embeddings, so it must change the\n // id. Only added when active (E5 + not disabled) so non-E5 ids - and the\n // historical hash of an E5 model with prefixing turned off - are stable.\n ...(this.#taskPrefix ? { taskPrefix: 'e5' as const } : {}),\n });\n }\n\n async embed(\n texts: ReadonlyArray<string>,\n opts: EmbedOptions = {},\n ): Promise<ReadonlyArray<Float32Array>> {\n if (texts.length === 0) return [];\n const extractor = await this.#getExtractor();\n // PS-10: E5 models require an asymmetric `query:` / `passage:` prefix.\n // Default to `passage` (the indexing role) when the caller doesn't specify.\n const inputs = this.#taskPrefix\n ? texts.map((t) => `${opts.taskType ?? 'passage'}: ${t}`)\n : [...texts];\n const result = await extractor(inputs, {\n pooling: this.#pooling,\n normalize: this.#normalize,\n ...(opts.signal !== undefined ? { signal: opts.signal } : {}),\n });\n const lastDim = result.dims[result.dims.length - 1] ?? this.#resolvedDim ?? DEFAULT_DIM;\n // periphery-05: a width already published (via `dim` option or a\n // prior embed) must not silently drift - the vec0 table and the\n // canonical id were derived from it.\n if (this.#resolvedDim !== null && lastDim !== this.#resolvedDim) {\n throw new Error(\n `[graphorin/embedder-transformersjs] Model '${this.#model}' produced ${lastDim}-dim ` +\n `vectors but the embedder is bound to ${this.#resolvedDim} - check the 'dim' option.`,\n );\n }\n if (this.#resolvedDim === null) {\n this.#resolvedDim = lastDim;\n }\n const dim = lastDim;\n const out: Float32Array[] = [];\n for (let i = 0; i < texts.length; i++) {\n const start = i * dim;\n out.push(result.data.slice(start, start + dim));\n }\n return out;\n }\n\n async #getExtractor(): Promise<FeatureExtractor> {\n if (this.#extractor !== null) return this.#extractor;\n if (this.#loading !== null) return this.#loading;\n const loader = this.#pipelineFactory ?? (await loadDefaultPipelineFactory());\n this.#loading = loader('feature-extraction', this.#model, {\n ...(this.#revision !== undefined ? { revision: this.#revision } : {}),\n ...(this.#cacheDir !== undefined ? { cache_dir: this.#cacheDir } : {}),\n ...(this.#dtype !== undefined ? { dtype: this.#dtype } : {}),\n ...(this.#device !== undefined ? { device: this.#device } : {}),\n })\n .then((pipe) => {\n this.#extractor = pipe;\n this.#loading = null;\n return pipe;\n })\n .catch((err) => {\n this.#loading = null;\n throw new EmbedderModelLoadError(\n `[graphorin/embedder-transformersjs] failed to load model '${this.#model}'. ` +\n 'See the offline-install guide for instructions on pre-downloading the model.',\n { cause: err },\n );\n });\n return this.#loading;\n }\n}\n\n/** @internal */\nlet CACHED_PIPELINE_FACTORY: PipelineFactory | null = null;\n\nasync function loadDefaultPipelineFactory(): Promise<PipelineFactory> {\n if (CACHED_PIPELINE_FACTORY !== null) return CACHED_PIPELINE_FACTORY;\n try {\n const mod = (await import('@huggingface/transformers')) as unknown as {\n pipeline: PipelineFactory;\n };\n CACHED_PIPELINE_FACTORY = mod.pipeline;\n return CACHED_PIPELINE_FACTORY;\n } catch (err) {\n throw new EmbedderModelLoadError(\n \"[graphorin/embedder-transformersjs] required peer '@huggingface/transformers' is not installed.\",\n { cause: err },\n );\n }\n}\n\n/** Test-only cache reset. */\nexport function _resetPipelineFactoryCacheForTesting(): void {\n CACHED_PIPELINE_FACTORY = null;\n}\n\nconst KNOWN_DIMS: ReadonlyMap<string, number> = new Map([\n ['Xenova/multilingual-e5-small', 384],\n ['Xenova/multilingual-e5-base', 768],\n ['Xenova/multilingual-e5-large', 1024],\n ['Xenova/bge-m3', 1024],\n ['onnx-community/all-MiniLM-L6-v2-ONNX', 384],\n]);\n\nfunction guessDefaultDim(model: string): number | null {\n return KNOWN_DIMS.get(model) ?? null;\n}\n\n/**\n * True when a model id belongs to the E5 family, which requires asymmetric\n * `query:` / `passage:` prefixes (PS-10). Matches an `e5` token bounded by a\n * path / dash / underscore so it covers `multilingual-e5-base`, `e5-large`,\n * `intfloat/e5-mistral`, etc. without false-matching unrelated names.\n */\nexport function isE5Model(model: string): boolean {\n return /(?:^|[/_-])e5(?:[/_-]|$)/i.test(model);\n}\n\n/**\n * Canonical-JSON deterministic hash of an embedder configuration.\n * Object keys are sorted lexicographically; primitives flow through as\n * `JSON.stringify` would render them. Used by the multi-table per-\n * embedder vec0 layout to tell drift apart from a true model swap.\n *\n * @stable\n */\nexport function canonicalConfigHash(config: unknown): string {\n const canonical = canonicalize(config);\n return createHash('sha256').update(canonical, 'utf8').digest('hex');\n}\n\nfunction canonicalize(value: unknown): string {\n if (value === null) return 'null';\n if (typeof value === 'boolean') return value ? 'true' : 'false';\n if (typeof value === 'number') return Number.isFinite(value) ? value.toString() : 'null';\n if (typeof value === 'string') return JSON.stringify(value);\n if (Array.isArray(value)) {\n return `[${value.map(canonicalize).join(',')}]`;\n }\n if (typeof value === 'object') {\n const obj = value as Record<string, unknown>;\n const keys = Object.keys(obj).sort();\n return `{${keys.map((k) => `${JSON.stringify(k)}:${canonicalize(obj[k])}`).join(',')}}`;\n }\n return 'null';\n}\n"],"mappings":";;;;;;;;;;;;;;AAeA,MAAa,UAAU;AAwFvB,MAAM,gBAAgB;AACtB,MAAM,cAAc;;;;;;;AAQpB,IAAa,yBAAb,cAA4C,MAAM;CAChD,AAAkB,OAAO;;;;;;;;;AAU3B,SAAgB,6BACd,UAAyC,EAAE,EACnB;AACxB,QAAO,IAAI,uBAAuB,QAAQ;;;;;;;AAQ5C,IAAa,yBAAb,MAAgE;CAC9D,CAASA;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;CACT,CAASC;;CAET,CAASC;CACT,aAAsC;CACtC,WAA6C;CAC7C;CAEA,YAAY,SAAwC;AAClD,QAAKR,QAAS,QAAQ,SAAS;AAC/B,QAAKC,UAAW,QAAQ,WAAW;AACnC,QAAKC,YAAa,QAAQ,aAAa;AACvC,QAAKC,WAAY,QAAQ;AACzB,QAAKC,WAAY,QAAQ,YAAY,QAAQ,IAAI;AACjD,QAAKC,QAAS,QAAQ;AACtB,QAAKC,SAAU,QAAQ;AACvB,QAAKC,kBAAmB,QAAQ;AAChC,QAAKE,cAAe,QAAQ,OAAO,gBAAgB,MAAKT,MAAO;AAC/D,QAAKQ,aAAc,UAAU,MAAKR,MAAO,IAAI,QAAQ,sBAAsB;;CAG7E,KAAa;AACX,SAAO,kBAAkB,MAAKA,MAAO,GAAG,KAAK,KAAK;;;;;;;;;;;;CAapD,MAAc;AACZ,MAAI,MAAKS,gBAAiB,KAAM,QAAO,MAAKA;AAC5C,QAAM,IAAI,MACR,0EAA0E,MAAKT,MAAO,0JAGvF;;CAGH,aAAqB;AACnB,SAAO,oBAAoB;GACzB,SAAS;GACT,OAAO,MAAKA;GACZ,SAAS,MAAKC;GACd,WAAW,MAAKC;GAChB,UAAU,MAAKC,YAAa;GAC5B,OAAO,MAAKE,SAAU;GACtB,QAAQ,MAAKC,UAAW;GAIxB,GAAI,MAAKE,aAAc,EAAE,YAAY,MAAe,GAAG,EAAE;GAC1D,CAAC;;CAGJ,MAAM,MACJ,OACA,OAAqB,EAAE,EACe;AACtC,MAAI,MAAM,WAAW,EAAG,QAAO,EAAE;EAOjC,MAAM,SAAS,OANG,MAAM,MAAKE,cAAe,EAG7B,MAAKF,aAChB,MAAM,KAAK,MAAM,GAAG,KAAK,YAAY,UAAU,IAAI,IAAI,GACvD,CAAC,GAAG,MAAM,EACyB;GACrC,SAAS,MAAKP;GACd,WAAW,MAAKC;GAChB,GAAI,KAAK,WAAW,SAAY,EAAE,QAAQ,KAAK,QAAQ,GAAG,EAAE;GAC7D,CAAC;EACF,MAAM,UAAU,OAAO,KAAK,OAAO,KAAK,SAAS,MAAM,MAAKO,eAAgB;AAI5E,MAAI,MAAKA,gBAAiB,QAAQ,YAAY,MAAKA,YACjD,OAAM,IAAI,MACR,8CAA8C,MAAKT,MAAO,aAAa,QAAQ,4CACrC,MAAKS,YAAa,4BAC7D;AAEH,MAAI,MAAKA,gBAAiB,KACxB,OAAKA,cAAe;EAEtB,MAAM,MAAM;EACZ,MAAME,MAAsB,EAAE;AAC9B,OAAK,IAAI,IAAI,GAAG,IAAI,MAAM,QAAQ,KAAK;GACrC,MAAM,QAAQ,IAAI;AAClB,OAAI,KAAK,OAAO,KAAK,MAAM,OAAO,QAAQ,IAAI,CAAC;;AAEjD,SAAO;;CAGT,OAAMD,eAA2C;AAC/C,MAAI,MAAKE,cAAe,KAAM,QAAO,MAAKA;AAC1C,MAAI,MAAKC,YAAa,KAAM,QAAO,MAAKA;AAExC,QAAKA,WADU,MAAKN,mBAAqB,MAAM,4BAA4B,EACpD,sBAAsB,MAAKP,OAAQ;GACxD,GAAI,MAAKG,aAAc,SAAY,EAAE,UAAU,MAAKA,UAAW,GAAG,EAAE;GACpE,GAAI,MAAKC,aAAc,SAAY,EAAE,WAAW,MAAKA,UAAW,GAAG,EAAE;GACrE,GAAI,MAAKC,UAAW,SAAY,EAAE,OAAO,MAAKA,OAAQ,GAAG,EAAE;GAC3D,GAAI,MAAKC,WAAY,SAAY,EAAE,QAAQ,MAAKA,QAAS,GAAG,EAAE;GAC/D,CAAC,CACC,MAAM,SAAS;AACd,SAAKM,YAAa;AAClB,SAAKC,UAAW;AAChB,UAAO;IACP,CACD,OAAO,QAAQ;AACd,SAAKA,UAAW;AAChB,SAAM,IAAI,uBACR,6DAA6D,MAAKb,MAAO,kFAEzE,EAAE,OAAO,KAAK,CACf;IACD;AACJ,SAAO,MAAKa;;;;AAKhB,IAAIC,0BAAkD;AAEtD,eAAe,6BAAuD;AACpE,KAAI,4BAA4B,KAAM,QAAO;AAC7C,KAAI;AAIF,6BAHa,MAAM,OAAO,8BAGI;AAC9B,SAAO;UACA,KAAK;AACZ,QAAM,IAAI,uBACR,mGACA,EAAE,OAAO,KAAK,CACf;;;;AAKL,SAAgB,uCAA6C;AAC3D,2BAA0B;;AAG5B,MAAMC,aAA0C,IAAI,IAAI;CACtD,CAAC,gCAAgC,IAAI;CACrC,CAAC,+BAA+B,IAAI;CACpC,CAAC,gCAAgC,KAAK;CACtC,CAAC,iBAAiB,KAAK;CACvB,CAAC,wCAAwC,IAAI;CAC9C,CAAC;AAEF,SAAS,gBAAgB,OAA8B;AACrD,QAAO,WAAW,IAAI,MAAM,IAAI;;;;;;;;AASlC,SAAgB,UAAU,OAAwB;AAChD,QAAO,4BAA4B,KAAK,MAAM;;;;;;;;;;AAWhD,SAAgB,oBAAoB,QAAyB;CAC3D,MAAM,YAAY,aAAa,OAAO;AACtC,QAAO,WAAW,SAAS,CAAC,OAAO,WAAW,OAAO,CAAC,OAAO,MAAM;;AAGrE,SAAS,aAAa,OAAwB;AAC5C,KAAI,UAAU,KAAM,QAAO;AAC3B,KAAI,OAAO,UAAU,UAAW,QAAO,QAAQ,SAAS;AACxD,KAAI,OAAO,UAAU,SAAU,QAAO,OAAO,SAAS,MAAM,GAAG,MAAM,UAAU,GAAG;AAClF,KAAI,OAAO,UAAU,SAAU,QAAO,KAAK,UAAU,MAAM;AAC3D,KAAI,MAAM,QAAQ,MAAM,CACtB,QAAO,IAAI,MAAM,IAAI,aAAa,CAAC,KAAK,IAAI,CAAC;AAE/C,KAAI,OAAO,UAAU,UAAU;EAC7B,MAAM,MAAM;AAEZ,SAAO,IADM,OAAO,KAAK,IAAI,CAAC,MAAM,CACpB,KAAK,MAAM,GAAG,KAAK,UAAU,EAAE,CAAC,GAAG,aAAa,IAAI,GAAG,GAAG,CAAC,KAAK,IAAI,CAAC;;AAEvF,QAAO"}
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@graphorin/embedder-transformersjs",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.6.0",
|
|
4
4
|
"description": "Default in-process embedder for the Graphorin framework. Wraps `@huggingface/transformers@^4.1.0` to produce dense vectors with the multilingual `Xenova/multilingual-e5-base` model (768-dim) by default. Implements the `EmbedderProvider` contract from `@graphorin/core/contracts` with deterministic `configHash`, batched embed, lazy model download (cache honours `GRAPHORIN_CACHE_DIR`), and an actionable error if the model cannot be fetched.",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Oleksiy Stepurenko",
|
|
@@ -44,7 +44,7 @@
|
|
|
44
44
|
"LICENSE"
|
|
45
45
|
],
|
|
46
46
|
"dependencies": {
|
|
47
|
-
"@graphorin/core": "0.
|
|
47
|
+
"@graphorin/core": "0.6.0"
|
|
48
48
|
},
|
|
49
49
|
"peerDependencies": {
|
|
50
50
|
"@huggingface/transformers": "^4.1.0"
|