libsql-search 0.7.0 → 0.7.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +26 -48
- package/docs/API.md +114 -107
- package/docs/INDEXING.md +12 -25
- package/docs/INTEGRATIONS.md +220 -40
- package/docs/MIGRATIONS.md +174 -0
- package/docs/PROVIDERS.md +86 -245
- package/docs/README.md +7 -9
- package/docs/TESTING.md +138 -0
- package/package.json +1 -1
package/docs/PROVIDERS.md
CHANGED
|
@@ -1,6 +1,8 @@
|
|
|
1
1
|
# Embedding Providers
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Use this page to choose an embedding provider, confirm the table width it needs, and understand what crosses a network boundary.
|
|
4
|
+
|
|
5
|
+
`libsql-search` supports these provider values:
|
|
4
6
|
|
|
5
7
|
- `local`
|
|
6
8
|
- `cloudflare`
|
|
@@ -9,11 +11,9 @@
|
|
|
9
11
|
- `openai`
|
|
10
12
|
- `openai-compatible`
|
|
11
13
|
|
|
12
|
-
|
|
13
|
-
between stored vectors and query vectors will break search quality or fail at
|
|
14
|
-
query time.
|
|
14
|
+
## Shared Behavior
|
|
15
15
|
|
|
16
|
-
|
|
16
|
+
All providers share the same `EmbeddingOptions` surface:
|
|
17
17
|
|
|
18
18
|
```ts
|
|
19
19
|
interface EmbeddingOptions {
|
|
@@ -38,88 +38,37 @@ interface EmbeddingOptions {
|
|
|
38
38
|
}
|
|
39
39
|
```
|
|
40
40
|
|
|
41
|
-
|
|
42
|
-
- `dimensions` defaults to `384` for the library's default local provider.
|
|
43
|
-
Provider-specific defaults can differ; Cloudflare and Mistral use `1024`,
|
|
44
|
-
and Gemini defaults to `3072`.
|
|
45
|
-
- `maxLength` defaults to `8000`
|
|
46
|
-
- `intent` can be `"document"` or `"query"`; indexing defaults to
|
|
47
|
-
`"document"` and search defaults to `"query"` unless explicitly set
|
|
48
|
-
- `timeoutMs` defaults to `30000`
|
|
49
|
-
- `apiKey` is used by Mistral, Gemini, and OpenAI and falls back to
|
|
50
|
-
`MISTRAL_API_KEY`, `GEMINI_API_KEY`, or `OPENAI_API_KEY`. For
|
|
51
|
-
`openai-compatible`, `apiKey` is optional and never falls back to
|
|
52
|
-
`OPENAI_API_KEY`.
|
|
53
|
-
- `accountId` and `apiToken` are used by Cloudflare and fall back to
|
|
54
|
-
`CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN`
|
|
55
|
-
- `baseUrl`, `model`, and `batchSize` are used by `openai-compatible`
|
|
56
|
-
|
|
57
|
-
## Provider Contract
|
|
41
|
+
Shared defaults and rules:
|
|
58
42
|
|
|
59
|
-
|
|
60
|
-
|
|
61
|
-
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
| "mistral"
|
|
67
|
-
| "gemini"
|
|
68
|
-
| "openai"
|
|
69
|
-
| "openai-compatible";
|
|
70
|
-
model: string;
|
|
71
|
-
dimensions: number;
|
|
72
|
-
batch: {
|
|
73
|
-
mode: "native" | "sequential";
|
|
74
|
-
maxSize?: number;
|
|
75
|
-
};
|
|
76
|
-
}
|
|
77
|
-
```
|
|
78
|
-
|
|
79
|
-
Use `getEmbeddingProviderMetadata(options)` or
|
|
80
|
-
`createEmbeddingProvider(options).metadata` to inspect the effective model,
|
|
81
|
-
dimensions, and batch behavior. Metadata inspection does not require hosted
|
|
82
|
-
provider credentials.
|
|
83
|
-
|
|
84
|
-
Batch modes:
|
|
85
|
-
|
|
86
|
-
- `"native"` means the upstream provider accepts the batch in one request
|
|
87
|
-
- `"sequential"` means the library accepts a batch and processes items one at a
|
|
88
|
-
time
|
|
89
|
-
- when `maxSize` is present, it is a hard maximum enforced before provider or
|
|
90
|
-
network work
|
|
91
|
-
|
|
92
|
-
`generateEmbeddings(texts, options)` returns vectors in the same order as the
|
|
93
|
-
input texts. Provider responses are validated before database writes:
|
|
94
|
-
|
|
95
|
-
- result count must match input count
|
|
96
|
-
- each vector must match the provider's effective dimensions
|
|
97
|
-
- every vector value must be a finite number
|
|
98
|
-
- indexed batch responses must contain unique contiguous indices and are
|
|
99
|
-
reordered before being returned
|
|
43
|
+
- `provider` defaults to `local`
|
|
44
|
+
- `maxLength` defaults to `8000` code units
|
|
45
|
+
- `timeoutMs` defaults to `30000`
|
|
46
|
+
- `indexContent()` defaults to `intent: "document"`
|
|
47
|
+
- `search()` defaults to `intent: "query"`
|
|
48
|
+
- `getEmbeddingProviderMetadata()` reports the effective provider, model, dimensions, and batch mode without making a hosted call
|
|
49
|
+
- use one embedding space per table: if provider, dimensions, model selection, endpoint, or formatting contract changes, build a new table and reindex
|
|
100
50
|
|
|
101
|
-
|
|
102
|
-
or making network calls.
|
|
51
|
+
Intent behavior is intentionally narrow today:
|
|
103
52
|
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
`generateEmbedding()` and `generateEmbeddings()` return only arrays.
|
|
53
|
+
- only the current Gemini adapter changes the formatted payload for `"document"` versus `"query"`
|
|
54
|
+
- all other providers still carry `intent` metadata through the API, but they embed the same text string either way
|
|
107
55
|
|
|
108
|
-
|
|
109
|
-
options. They are not cached globally across different credentials or
|
|
110
|
-
configurations. The local Hugging Face Transformers pipeline is loaded lazily
|
|
111
|
-
and cached by model name.
|
|
56
|
+
The `model` option is only used by `openai-compatible`.
|
|
112
57
|
|
|
113
|
-
|
|
114
|
-
context and without raw upstream bodies, credentials, Authorization headers, or
|
|
115
|
-
full URLs with query strings.
|
|
58
|
+
## Provider Matrix
|
|
116
59
|
|
|
117
|
-
|
|
60
|
+
| Provider | Literal | Upstream model used by this adapter | Dimensions | Credentials | Batching | Network and privacy boundary | Cost and table planning |
|
|
61
|
+
| --- | --- | --- | --- | --- | --- | --- | --- |
|
|
62
|
+
| Local | `local` | `Xenova/all-MiniLM-L6-v2` | Fixed `384` | None | Sequential in-process | No hosted API call. First use may download model artifacts and cache them locally. | No hosted API bill. Table must be `F32_BLOB(384)`. |
|
|
63
|
+
| Cloudflare Workers AI | `cloudflare` | `@cf/baai/bge-m3` | Fixed `1024` | `accountId` and `apiToken`, or `CLOUDFLARE_ACCOUNT_ID` and `CLOUDFLARE_API_TOKEN` | Native batch in one request | Indexed and queried text is sent to Cloudflare. | Check Cloudflare pricing before large rebuilds. Table must be `F32_BLOB(1024)`. |
|
|
64
|
+
| Mistral | `mistral` | `mistral-embed` | Fixed `1024` | `apiKey`, or `MISTRAL_API_KEY` | Native batch in one request | Indexed and queried text is sent to Mistral. | Check Mistral pricing before rebuilds. Table must be `F32_BLOB(1024)`. |
|
|
65
|
+
| Gemini | `gemini` | `gemini-embedding-2` | Default `3072`; allowed integers `128-3072` | `apiKey`, or `GEMINI_API_KEY` | Sequential SDK request per input | Indexed and queried text is sent to Google. The adapter currently rewrites payload text by intent. | Check Gemini pricing before rebuilds. Table width must match the chosen dimension count exactly. |
|
|
66
|
+
| OpenAI | `openai` | `text-embedding-3-small` when `dimensions <= 1536`, otherwise `text-embedding-3-large` | Default `768`; any positive integer accepted locally and forwarded as `dimensions` | `apiKey`, or `OPENAI_API_KEY` | Native batch in one request, max `2048` inputs | Indexed and queried text is sent to OpenAI. | Check OpenAI pricing before rebuilds. Table width must match the chosen dimension count exactly. |
|
|
67
|
+
| OpenAI-compatible | `openai-compatible` | Your configured `model` | Required positive integer; no default | `baseUrl`, `model`, and `dimensions` are required. `apiKey` is optional and never falls back to env. | Metadata reports `native`; outbound requests are chunked sequentially at `batchSize`, default `32` | Boundary depends on the operator behind `baseUrl`. Treat `baseUrl` as a trusted server-side setting and review HTTPS, SSRF, logging, and retention controls yourself. | Table width must match the configured `dimensions`. Any endpoint or model change should use a new table plus full reindex. |
|
|
118
68
|
|
|
119
|
-
Provider
|
|
69
|
+
## Provider Notes
|
|
120
70
|
|
|
121
|
-
|
|
122
|
-
`@huggingface/transformers`.
|
|
71
|
+
### Local
|
|
123
72
|
|
|
124
73
|
```ts
|
|
125
74
|
embeddingOptions: {
|
|
@@ -127,26 +76,16 @@ embeddingOptions: {
|
|
|
127
76
|
}
|
|
128
77
|
```
|
|
129
78
|
|
|
130
|
-
|
|
79
|
+
- fixed at `384` dimensions
|
|
80
|
+
- rejects any other `dimensions` value before loading the runtime
|
|
81
|
+
- uses `@huggingface/transformers` lazily and caches the local pipeline by model name
|
|
131
82
|
|
|
132
|
-
|
|
133
|
-
384 finite numbers
|
|
134
|
-
- `dimensions: 384` is accepted explicitly; any other local dimension is
|
|
135
|
-
rejected before the runtime is imported or loaded
|
|
136
|
-
- metadata reports 384 dimensions
|
|
137
|
-
- batch metadata is `{ mode: "sequential" }`
|
|
138
|
-
- the first run downloads and caches the model and can take longer on a fresh
|
|
139
|
-
machine
|
|
140
|
-
- no API key is required
|
|
141
|
-
- this remains the default provider for offline use
|
|
83
|
+
References:
|
|
142
84
|
|
|
143
|
-
|
|
85
|
+
- [Transformers.js in Node.js](https://huggingface.co/docs/transformers.js/en/tutorials/node)
|
|
86
|
+
- [Transformers.js environment and cache controls](https://huggingface.co/docs/transformers.js/en/api/env)
|
|
144
87
|
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
Cloudflare is the recommended hosted provider for low-cost Markdown search.
|
|
148
|
-
It uses Workers AI `@cf/baai/bge-m3` through Cloudflare's OpenAI-compatible
|
|
149
|
-
embeddings endpoint.
|
|
88
|
+
### Cloudflare Workers AI
|
|
150
89
|
|
|
151
90
|
```ts
|
|
152
91
|
embeddingOptions: {
|
|
@@ -156,25 +95,16 @@ embeddingOptions: {
|
|
|
156
95
|
}
|
|
157
96
|
```
|
|
158
97
|
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
-
|
|
162
|
-
- if `apiToken` is omitted, the library reads `CLOUDFLARE_API_TOKEN`
|
|
163
|
-
- blank Cloudflare credentials are treated as missing
|
|
164
|
-
- `@cf/baai/bge-m3` returns 1024 dimensions
|
|
165
|
-
- metadata reports `@cf/baai/bge-m3` and 1024 dimensions without requiring
|
|
166
|
-
credentials
|
|
167
|
-
- batch metadata is `{ mode: "native" }`
|
|
168
|
-
- response items are reordered by provider-supplied index before being returned
|
|
169
|
-
- Cloudflare does not accept custom dimensions in this provider; use
|
|
170
|
-
`createTable(client, "articles", 1024)` for Cloudflare-backed indexes
|
|
98
|
+
- fixed at `1024` dimensions
|
|
99
|
+
- uses the account-scoped Workers AI embeddings endpoint
|
|
100
|
+
- blank credentials are treated as missing
|
|
171
101
|
|
|
172
|
-
|
|
102
|
+
References:
|
|
173
103
|
|
|
174
|
-
|
|
104
|
+
- [Cloudflare `@cf/baai/bge-m3`](https://developers.cloudflare.com/workers-ai/models/bge-m3/)
|
|
105
|
+
- [Cloudflare Workers AI pricing](https://developers.cloudflare.com/workers-ai/platform/pricing/)
|
|
175
106
|
|
|
176
|
-
Mistral
|
|
177
|
-
`https://api.mistral.ai/v1/embeddings`.
|
|
107
|
+
### Mistral
|
|
178
108
|
|
|
179
109
|
```ts
|
|
180
110
|
embeddingOptions: {
|
|
@@ -183,24 +113,17 @@ embeddingOptions: {
|
|
|
183
113
|
}
|
|
184
114
|
```
|
|
185
115
|
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
-
|
|
189
|
-
- blank Mistral credentials are treated as missing
|
|
190
|
-
- `mistral-embed` returns 1024 dimensions
|
|
191
|
-
- metadata reports `mistral-embed` and 1024 dimensions without requiring
|
|
192
|
-
credentials
|
|
193
|
-
- batch metadata is `{ mode: "native" }`
|
|
194
|
-
- request bodies send `encoding_format: "float"`
|
|
195
|
-
- response items are reordered by provider-supplied index before being returned
|
|
196
|
-
- Mistral does not accept custom dimensions in this provider; use
|
|
197
|
-
`createTable(client, "articles", 1024)` for Mistral-backed indexes
|
|
116
|
+
- fixed at `1024` dimensions
|
|
117
|
+
- sends `encoding_format: "float"`
|
|
118
|
+
- expects indexed upstream responses
|
|
198
119
|
|
|
199
|
-
|
|
120
|
+
References:
|
|
200
121
|
|
|
201
|
-
|
|
122
|
+
- [Mistral embeddings guide](https://docs.mistral.ai/studio/knowledge-rag/embeddings/text_embeddings)
|
|
123
|
+
- [Mistral embeddings API reference](https://docs.mistral.ai/api/endpoint/embeddings)
|
|
124
|
+
- [Mistral model overview for `mistral-embed`](https://docs.mistral.ai/models/mistral-embed-23-12)
|
|
202
125
|
|
|
203
|
-
Gemini
|
|
126
|
+
### Gemini
|
|
204
127
|
|
|
205
128
|
```ts
|
|
206
129
|
embeddingOptions: {
|
|
@@ -210,26 +133,18 @@ embeddingOptions: {
|
|
|
210
133
|
}
|
|
211
134
|
```
|
|
212
135
|
|
|
213
|
-
|
|
136
|
+
- defaults to `3072` dimensions
|
|
137
|
+
- accepts only integer dimensions from `128` through `3072`
|
|
138
|
+
- currently formats document inputs as `title: none | text: ...`
|
|
139
|
+
- currently formats query inputs as `task: search result | query: ...`
|
|
140
|
+
- sends one SDK request per input, even when you call `generateEmbeddings()`
|
|
214
141
|
|
|
215
|
-
|
|
216
|
-
- blank Gemini credentials are treated as missing
|
|
217
|
-
- Gemini defaults to 3072 dimensions
|
|
218
|
-
- explicit Gemini dimensions must be integers from 128 through 3072
|
|
219
|
-
- 768, 1536, and 3072 are recommended practical sizes
|
|
220
|
-
- metadata reports `gemini-embedding-2` and the effective dimensions
|
|
221
|
-
- batch metadata is `{ mode: "sequential" }`
|
|
222
|
-
- the library sends one SDK request per input and verifies one vector per input
|
|
223
|
-
- document inputs are formatted as `title: none | text: ...`
|
|
224
|
-
- query inputs are formatted as `task: search result | query: ...`
|
|
225
|
-
- the current implementation does not expose custom model selection
|
|
142
|
+
References:
|
|
226
143
|
|
|
227
|
-
|
|
144
|
+
- [Gemini embeddings guide](https://ai.google.dev/gemini-api/docs/embeddings)
|
|
145
|
+
- [Gemini pricing](https://ai.google.dev/gemini-api/docs/pricing)
|
|
228
146
|
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
OpenAI uses `text-embedding-3-small` when `dimensions <= 1536` and
|
|
232
|
-
`text-embedding-3-large` when `dimensions > 1536`.
|
|
147
|
+
### OpenAI
|
|
233
148
|
|
|
234
149
|
```ts
|
|
235
150
|
embeddingOptions: {
|
|
@@ -239,119 +154,45 @@ embeddingOptions: {
|
|
|
239
154
|
}
|
|
240
155
|
```
|
|
241
156
|
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
-
|
|
245
|
-
-
|
|
246
|
-
- metadata reports `text-embedding-3-small` when `dimensions <= 1536` and
|
|
247
|
-
`text-embedding-3-large` when `dimensions > 1536`
|
|
248
|
-
- batch metadata is `{ mode: "native", maxSize: 2048 }`
|
|
249
|
-
- use the same dimension count in `createTable()`
|
|
157
|
+
- defaults to `768` dimensions
|
|
158
|
+
- uses `text-embedding-3-small` through `1536`
|
|
159
|
+
- uses `text-embedding-3-large` above `1536`
|
|
160
|
+
- rejects batches above `2048` inputs before any network request
|
|
250
161
|
|
|
251
|
-
|
|
162
|
+
References:
|
|
252
163
|
|
|
253
|
-
|
|
164
|
+
- [OpenAI embeddings guide](https://developers.openai.com/api/docs/guides/embeddings)
|
|
165
|
+
- [OpenAI embeddings API reference](https://developers.openai.com/api/reference/resources/embeddings/methods/create/)
|
|
166
|
+
- [OpenAI data controls](https://developers.openai.com/api/docs/guides/your-data#default-usage-policies-by-endpoint)
|
|
167
|
+
- [OpenAI models overview](https://developers.openai.com/api/docs/models)
|
|
168
|
+
- [OpenAI API pricing](https://openai.com/api/pricing/)
|
|
254
169
|
|
|
255
|
-
|
|
256
|
-
Hugging Face Text Embeddings Inference (TEI) or an internal gateway. This is an
|
|
257
|
-
optional escape hatch; `local` remains the default offline provider, and the
|
|
258
|
-
named hosted providers above are still preferred when their fixed adapters fit.
|
|
170
|
+
### OpenAI-compatible
|
|
259
171
|
|
|
260
172
|
```ts
|
|
261
173
|
embeddingOptions: {
|
|
262
174
|
provider: "openai-compatible",
|
|
263
|
-
baseUrl:
|
|
264
|
-
model:
|
|
265
|
-
dimensions:
|
|
175
|
+
baseUrl: process.env.EMBEDDING_BASE_URL,
|
|
176
|
+
model: process.env.EMBEDDING_MODEL,
|
|
177
|
+
dimensions: Number(process.env.EMBEDDING_DIMENSIONS),
|
|
178
|
+
apiKey: process.env.EMBEDDING_API_KEY,
|
|
266
179
|
batchSize: 32,
|
|
267
180
|
}
|
|
268
181
|
```
|
|
269
182
|
|
|
270
|
-
|
|
271
|
-
|
|
272
|
-
|
|
273
|
-
|
|
274
|
-
|
|
275
|
-
baseUrl: "https://embeddings.example.com/v1",
|
|
276
|
-
apiKey: process.env.EMBEDDINGS_API_KEY,
|
|
277
|
-
model: "BAAI/bge-large-en-v1.5",
|
|
278
|
-
dimensions: 1024,
|
|
279
|
-
}
|
|
280
|
-
```
|
|
183
|
+
- `baseUrl`, `model`, and `dimensions` are required
|
|
184
|
+
- `baseUrl` must be an absolute `http` or `https` URL without URL credentials, query strings, or fragments
|
|
185
|
+
- the library normalizes `baseUrl` to an `/embeddings` endpoint
|
|
186
|
+
- `batchSize` defaults to `32` and controls outbound chunking
|
|
187
|
+
- `apiKey` is optional and never falls back to `OPENAI_API_KEY`
|
|
281
188
|
|
|
282
|
-
|
|
189
|
+
References:
|
|
283
190
|
|
|
284
|
-
-
|
|
285
|
-
-
|
|
286
|
-
sends requests to `/embeddings` below that base
|
|
287
|
-
- a base URL that already ends in `/embeddings` is used as-is
|
|
288
|
-
- only absolute `http` and `https` URLs are accepted
|
|
289
|
-
- URL usernames, passwords, query strings, and fragments are rejected
|
|
290
|
-
- `apiKey` is optional; blank keys are ignored and no `Authorization` header is
|
|
291
|
-
sent
|
|
292
|
-
- `OPENAI_API_KEY` is never read for this provider
|
|
293
|
-
- `batchSize` defaults to `32`, matching TEI's conservative client batch size;
|
|
294
|
-
larger input arrays are split into sequential outbound requests and returned
|
|
295
|
-
in the original input order
|
|
296
|
-
- requests send `{ input, model, dimensions, encoding_format: "float" }`
|
|
297
|
-
- responses must use the standard OpenAI embeddings shape with indexed
|
|
298
|
-
`data[]` items; each response chunk must include unique contiguous indices
|
|
299
|
-
- batch metadata is `{ mode: "native" }`; the internal outbound chunk size is
|
|
300
|
-
not reported as `batch.maxSize`
|
|
301
|
-
|
|
302
|
-
Security boundary:
|
|
303
|
-
|
|
304
|
-
- treat `baseUrl` as trusted server-side configuration only
|
|
305
|
-
- never pass user-supplied request values directly into `baseUrl`
|
|
306
|
-
- use HTTPS for remote endpoints
|
|
307
|
-
- credentials are not sent across redirects
|
|
308
|
-
- non-2xx response bodies are not read, and errors avoid echoing API keys or
|
|
309
|
-
configured endpoint URLs
|
|
310
|
-
|
|
311
|
-
TEI exposes an OpenAI-compatible base at `/v1`, so a local TEI server usually
|
|
312
|
-
uses:
|
|
191
|
+
- [Text Embeddings Inference quick tour](https://huggingface.co/docs/text-embeddings-inference/quick_tour)
|
|
192
|
+
- [Text Embeddings Inference CLI arguments](https://huggingface.co/docs/text-embeddings-inference/cli_arguments)
|
|
313
193
|
|
|
314
|
-
|
|
315
|
-
embeddingOptions: {
|
|
316
|
-
provider: "openai-compatible",
|
|
317
|
-
baseUrl: "http://localhost:8080/v1",
|
|
318
|
-
model: "BAAI/bge-large-en-v1.5",
|
|
319
|
-
dimensions: 1024,
|
|
320
|
-
}
|
|
321
|
-
```
|
|
194
|
+
## Next Steps
|
|
322
195
|
|
|
323
|
-
|
|
324
|
-
|
|
325
|
-
|
|
326
|
-
|
|
327
|
-
- `local` is fixed at `384`
|
|
328
|
-
- `cloudflare` is fixed at `1024`
|
|
329
|
-
- `mistral` is fixed at `1024`
|
|
330
|
-
- Gemini defaults to `3072` and accepts explicit dimensions from `128` through
|
|
331
|
-
`3072`; use `768`, `1536`, or `3072` unless you have a specific reason
|
|
332
|
-
- OpenAI can be used at 1536 or 3072, or another supported OpenAI dimension
|
|
333
|
-
value you explicitly set
|
|
334
|
-
- `openai-compatible` requires you to set the dimension count that your
|
|
335
|
-
endpoint/model actually returns
|
|
336
|
-
|
|
337
|
-
If you switch provider, endpoint, model, or dimensions for an existing table,
|
|
338
|
-
recreate the table or rebuild the index into a separate table so stored vectors
|
|
339
|
-
stay consistent.
|
|
340
|
-
|
|
341
|
-
Existing Gemini indexes created with `text-embedding-004` must be fully
|
|
342
|
-
re-embedded for `gemini-embedding-2`, even if you keep `dimensions: 768`,
|
|
343
|
-
because both the model and query/document input formatting changed. If you move
|
|
344
|
-
to the new 3072-dimensional default, create a new table or recreate the vector
|
|
345
|
-
table first; `indexContent()` clears rows but does not change the `F32_BLOB`
|
|
346
|
-
width. A separate table is safer because rebuilds are not transactional.
|
|
347
|
-
|
|
348
|
-
Existing local indexes created with the older padded-local behavior usually have
|
|
349
|
-
`F32_BLOB(768)` rows containing the 384 model values followed by zero padding.
|
|
350
|
-
The current local contract stores the native 384-dimensional model output. To
|
|
351
|
-
migrate, create or recreate a `F32_BLOB(384)` table and run a full re-index
|
|
352
|
-
before querying it. Using the same model ID avoids an intentional model-space
|
|
353
|
-
change, but bit-identical vectors are not promised across runtime, model
|
|
354
|
-
revision, dtype, pooling, or normalization changes; validate search quality and
|
|
355
|
-
re-index when those details change.
|
|
356
|
-
|
|
357
|
-
Routine unit tests mock the local runtime and do not download the model.
|
|
196
|
+
- [Integration examples](./INTEGRATIONS.md) for end-to-end configuration flows
|
|
197
|
+
- [Migration guide](./MIGRATIONS.md) before changing widths, models, or endpoints
|
|
198
|
+
- [Testing guide](./TESTING.md) for CI-safe provider coverage
|
package/docs/README.md
CHANGED
|
@@ -1,16 +1,14 @@
|
|
|
1
1
|
# Documentation
|
|
2
2
|
|
|
3
3
|
This directory holds the longer-form reference material for `libsql-search`.
|
|
4
|
-
Start with the page that matches the job you are doing:
|
|
5
4
|
|
|
6
|
-
- [Provider
|
|
7
|
-
|
|
5
|
+
- [Provider selection and configuration](./PROVIDERS.md): compare local, hosted, and custom embedding providers before you build an index
|
|
6
|
+
- [Integration examples](./INTEGRATIONS.md): reusable provider flow plus Astro and Next.js examples
|
|
7
|
+
- [Migration and reindexing guide](./MIGRATIONS.md): table-width changes, provider/model swaps, and safe cutovers
|
|
8
8
|
- [API reference](./API.md): exported functions, option shapes, and result data
|
|
9
|
-
- [
|
|
10
|
-
- [
|
|
11
|
-
|
|
12
|
-
- [Troubleshooting](./TROUBLESHOOTING.md): known install/runtime issues
|
|
9
|
+
- [Indexing and operations](./INDEXING.md): content layout, rebuild behavior, and search quality notes
|
|
10
|
+
- [Testing guidance](./TESTING.md): CI-safe mocks and no-live-call policy
|
|
11
|
+
- [Troubleshooting](./TROUBLESHOOTING.md): known install and runtime issues
|
|
13
12
|
- [Releasing](./RELEASING.md): maintainer release workflow
|
|
14
13
|
|
|
15
|
-
For the shortest first-use path, go back to the repository
|
|
16
|
-
[README](../README.md).
|
|
14
|
+
For the shortest first-use path, go back to the repository [README](../README.md).
|
package/docs/TESTING.md
ADDED
|
@@ -0,0 +1,138 @@
|
|
|
1
|
+
# Testing Guidance
|
|
2
|
+
|
|
3
|
+
Routine unit tests and CI should not make live embedding-provider calls and should not require real provider credentials.
|
|
4
|
+
|
|
5
|
+
## Repository Policy
|
|
6
|
+
|
|
7
|
+
- no live provider keys in unit tests or CI
|
|
8
|
+
- no routine network calls to hosted embedding providers in CI
|
|
9
|
+
- validate option handling and response parsing with mocks first
|
|
10
|
+
- assert failures happen before network calls when configuration is invalid
|
|
11
|
+
|
|
12
|
+
The current test suite follows that pattern in `tests/embeddings.test.ts` and [`tests/huggingface-transformers.mock.ts`](../tests/huggingface-transformers.mock.ts).
|
|
13
|
+
|
|
14
|
+
## Local Provider Mocks
|
|
15
|
+
|
|
16
|
+
The local provider should use a lightweight Transformers.js mock instead of downloading the real model during routine tests.
|
|
17
|
+
|
|
18
|
+
```ts
|
|
19
|
+
import {
|
|
20
|
+
huggingFaceTransformersMock,
|
|
21
|
+
resetHuggingFaceTransformersMock,
|
|
22
|
+
} from "./huggingface-transformers.mock.js";
|
|
23
|
+
|
|
24
|
+
beforeEach(() => {
|
|
25
|
+
resetHuggingFaceTransformersMock();
|
|
26
|
+
});
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
The repository source file is `huggingface-transformers.mock.ts`. The example keeps the `.js` import suffix because this repo's ESM TypeScript source uses explicit `.js` relative imports that resolve after compilation.
|
|
30
|
+
|
|
31
|
+
Test the contract you care about:
|
|
32
|
+
|
|
33
|
+
- the library requests `Xenova/all-MiniLM-L6-v2`
|
|
34
|
+
- the call uses `pooling: "mean"` and `normalize: true`
|
|
35
|
+
- non-`384` local dimensions fail before runtime loading
|
|
36
|
+
|
|
37
|
+
## HTTP Provider Mocks
|
|
38
|
+
|
|
39
|
+
Cloudflare, Mistral, OpenAI, and `openai-compatible` should use `fetch` mocks.
|
|
40
|
+
|
|
41
|
+
```ts
|
|
42
|
+
import { vi } from "vitest";
|
|
43
|
+
|
|
44
|
+
const fetchMock = vi.fn().mockResolvedValue({
|
|
45
|
+
ok: true,
|
|
46
|
+
headers: new Headers(),
|
|
47
|
+
json: async () => ({
|
|
48
|
+
data: [
|
|
49
|
+
{ index: 0, embedding: [1, 2] },
|
|
50
|
+
{ index: 1, embedding: [3, 4] },
|
|
51
|
+
],
|
|
52
|
+
}),
|
|
53
|
+
});
|
|
54
|
+
|
|
55
|
+
vi.stubGlobal("fetch", fetchMock);
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
Useful assertions:
|
|
59
|
+
|
|
60
|
+
- request body includes the expected `model`
|
|
61
|
+
- OpenAI includes `dimensions`
|
|
62
|
+
- Mistral and `openai-compatible` include `encoding_format: "float"`
|
|
63
|
+
- `openai-compatible` chunking honors `batchSize`
|
|
64
|
+
- Cloudflare and Mistral reorder indexed responses correctly
|
|
65
|
+
- invalid config fails before `fetch` is called
|
|
66
|
+
|
|
67
|
+
## Gemini SDK Mocks
|
|
68
|
+
|
|
69
|
+
Gemini uses the `@google/genai` SDK, so routine tests should mock the SDK client rather than calling Google.
|
|
70
|
+
|
|
71
|
+
```ts
|
|
72
|
+
vi.mock("@google/genai", () => ({
|
|
73
|
+
GoogleGenAI: class {
|
|
74
|
+
readonly models = {
|
|
75
|
+
embedContent: vi.fn(async () => ({
|
|
76
|
+
embeddings: [{ values: [1, 2, 3] }],
|
|
77
|
+
})),
|
|
78
|
+
};
|
|
79
|
+
},
|
|
80
|
+
}));
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
Useful assertions:
|
|
84
|
+
|
|
85
|
+
- the adapter sends `gemini-embedding-2`
|
|
86
|
+
- document intent formats text as `title: none | text: ...`
|
|
87
|
+
- query intent formats text as `task: search result | query: ...`
|
|
88
|
+
- explicit dimensions are forwarded as `outputDimensionality`
|
|
89
|
+
- invalid dimensions fail before SDK work
|
|
90
|
+
|
|
91
|
+
## Environment Cleanup
|
|
92
|
+
|
|
93
|
+
Tests that touch provider credentials should save and restore environment variables so one case cannot leak into another:
|
|
94
|
+
|
|
95
|
+
```ts
|
|
96
|
+
let originalOpenAIKey: string | undefined;
|
|
97
|
+
|
|
98
|
+
beforeEach(() => {
|
|
99
|
+
originalOpenAIKey = process.env.OPENAI_API_KEY;
|
|
100
|
+
delete process.env.OPENAI_API_KEY;
|
|
101
|
+
});
|
|
102
|
+
|
|
103
|
+
afterEach(() => {
|
|
104
|
+
if (originalOpenAIKey === undefined) {
|
|
105
|
+
delete process.env.OPENAI_API_KEY;
|
|
106
|
+
} else {
|
|
107
|
+
process.env.OPENAI_API_KEY = originalOpenAIKey;
|
|
108
|
+
}
|
|
109
|
+
});
|
|
110
|
+
```
|
|
111
|
+
|
|
112
|
+
Apply the same pattern to `GEMINI_API_KEY`, `MISTRAL_API_KEY`, `CLOUDFLARE_ACCOUNT_ID`, and `CLOUDFLARE_API_TOKEN`.
|
|
113
|
+
|
|
114
|
+
## Validation-before-network Coverage
|
|
115
|
+
|
|
116
|
+
Prefer tests that prove bad inputs fail locally:
|
|
117
|
+
|
|
118
|
+
- unknown provider
|
|
119
|
+
- missing provider credentials
|
|
120
|
+
- blank credentials where trimming is expected
|
|
121
|
+
- invalid local dimensions
|
|
122
|
+
- invalid Gemini dimensions
|
|
123
|
+
- invalid `openai-compatible` `baseUrl`
|
|
124
|
+
- invalid `openai-compatible` `batchSize`
|
|
125
|
+
- OpenAI batches above `2048`
|
|
126
|
+
|
|
127
|
+
These checks keep CI fast and prove the library rejects bad inputs before it ships them to a provider.
|
|
128
|
+
|
|
129
|
+
## Optional Live Smoke Tests
|
|
130
|
+
|
|
131
|
+
If you want live provider smoke coverage, keep it outside routine CI:
|
|
132
|
+
|
|
133
|
+
- run it only when a developer explicitly opts in
|
|
134
|
+
- use dedicated throwaway credentials and test content
|
|
135
|
+
- isolate it from unit-test jobs
|
|
136
|
+
- expect provider cost and external data transfer
|
|
137
|
+
|
|
138
|
+
This repository does not require or expect live provider smoke tests for normal pull-request validation.
|