local-embed 0.2.3__tar.gz → 0.2.4__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Potentially problematic release.
This version of local-embed might be problematic. Click here for more details.
- {local_embed-0.2.3 → local_embed-0.2.4}/.gitignore +1 -1
- {local_embed-0.2.3 → local_embed-0.2.4}/PKG-INFO +103 -90
- {local_embed-0.2.3 → local_embed-0.2.4}/README.md +101 -88
- local_embed-0.2.4/install.ps1 +85 -0
- local_embed-0.2.4/install.sh +73 -0
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/cli.py +23 -43
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/cmd_set.py +15 -10
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/config.py +42 -18
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/engine.py +154 -107
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/logging.py +1 -1
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/server.py +179 -97
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/server_utils.py +0 -50
- {local_embed-0.2.3 → local_embed-0.2.4}/pyproject.toml +2 -2
- {local_embed-0.2.3 → local_embed-0.2.4}/tests/test_client.py +11 -6
- {local_embed-0.2.3 → local_embed-0.2.4}/tests/test_config.py +59 -0
- {local_embed-0.2.3 → local_embed-0.2.4}/tests/test_engine.py +198 -69
- {local_embed-0.2.3 → local_embed-0.2.4}/tests/test_server.py +126 -21
- {local_embed-0.2.3 → local_embed-0.2.4}/tests/test_set.py +26 -47
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/__init__.py +0 -0
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/__main__.py +0 -0
- {local_embed-0.2.3 → local_embed-0.2.4}/local_embed/threads.py +0 -0
- {local_embed-0.2.3 → local_embed-0.2.4}/tests/test_logging.py +0 -0
|
@@ -1,5 +1,5 @@
|
|
|
1
1
|
# Runtime config — personal per-user settings. The three initial configs
|
|
2
|
-
# (slife.json5 / local_embed.json5 /
|
|
2
|
+
# (slife.json5 / local_embed.json5 / tools.json5) are git-tracked so the
|
|
3
3
|
# one-click installers can seed them as out-of-the-box defaults. Copies in a
|
|
4
4
|
# user's data dir (~/.slife/) remain personal and are NOT tracked.
|
|
5
5
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.3
|
|
2
2
|
Name: local-embed
|
|
3
|
-
Version: 0.2.
|
|
3
|
+
Version: 0.2.4
|
|
4
4
|
Summary: Standalone local embedding server — GGUF / transformer models as an OpenAI-compatible /v1/embeddings service
|
|
5
5
|
Project-URL: Homepage, https://github.com/juzcn/slife
|
|
6
6
|
Project-URL: Repository, https://github.com/juzcn/slife
|
|
@@ -16,7 +16,7 @@ Classifier: Programming Language :: Python :: 3.13
|
|
|
16
16
|
Classifier: Topic :: Terminals
|
|
17
17
|
Requires-Python: >=3.13
|
|
18
18
|
Requires-Dist: fastmcp>=2.0.0
|
|
19
|
-
Requires-Dist:
|
|
19
|
+
Requires-Dist: json-five>=1.1.2
|
|
20
20
|
Requires-Dist: starlette>=0.37.0
|
|
21
21
|
Provides-Extra: gguf
|
|
22
22
|
Requires-Dist: llama-cpp-python==0.3.34; extra == 'gguf'
|
|
@@ -51,15 +51,18 @@ them ([Model weights](#model-weights)).
|
|
|
51
51
|
vector DB / RAG pipeline ──────┘ ▼
|
|
52
52
|
┌───────────────────────┐
|
|
53
53
|
slife (plugin host) ─── /mcp ────────────> │ local-embed │
|
|
54
|
-
│
|
|
54
|
+
│ models load on demand │
|
|
55
55
|
└───────────────────────┘
|
|
56
56
|
```
|
|
57
57
|
|
|
58
58
|
## Features
|
|
59
59
|
|
|
60
60
|
- **OpenAI-compatible** `/v1/embeddings` + Models API + `/health`.
|
|
61
|
-
- Two backends (GGUF / transformer), **many models
|
|
62
|
-
|
|
61
|
+
- Two backends (GGUF / transformer), **many models as peers — each request
|
|
62
|
+
names the one it wants** (standard OpenAI semantics, no "active model").
|
|
63
|
+
- **Lazy load** — startup is fast; each model materialises on the first
|
|
64
|
+
request that names it (per-model `autoload: true` preloads a specific
|
|
65
|
+
model at startup).
|
|
63
66
|
- **Real dimension** reported after load (a guessed width is never served).
|
|
64
67
|
- Runs standalone, and can also be **loaded as a slife plugin** (see
|
|
65
68
|
[Loading as a slife plugin](#loading-as-a-slife-plugin)).
|
|
@@ -68,18 +71,36 @@ Requires Python ≥ 3.13.
|
|
|
68
71
|
|
|
69
72
|
## Install
|
|
70
73
|
|
|
71
|
-
The core package is `fastmcp` + `starlette` + `
|
|
74
|
+
The core package is `fastmcp` + `starlette` + `json-five` (the JSON5 parser,
|
|
75
|
+
imported as `json5`); the model backends are
|
|
72
76
|
optional extras.
|
|
73
77
|
|
|
78
|
+
One-click installers (install `uv` if needed, then `uv tool install --force
|
|
79
|
+
local-embed` — the backend is **not** included, see below):
|
|
80
|
+
|
|
81
|
+
```bash
|
|
82
|
+
# macOS / Linux / WSL
|
|
83
|
+
curl -fsSL https://raw.githubusercontent.com/juzcn/slife/main/local-embed/install.sh | bash
|
|
84
|
+
|
|
85
|
+
# Windows PowerShell
|
|
86
|
+
powershell -ExecutionPolicy Bypass -Command "irm https://raw.githubusercontent.com/juzcn/slife/main/local-embed/install.ps1 | iex"
|
|
87
|
+
```
|
|
88
|
+
|
|
89
|
+
Manual installs are equivalent:
|
|
90
|
+
|
|
74
91
|
```bash
|
|
75
92
|
# the app + CLI (standalone tool)
|
|
76
|
-
uv tool install local-embed
|
|
93
|
+
uv tool install --force local-embed
|
|
77
94
|
|
|
78
95
|
# with a backend:
|
|
79
|
-
uv tool install "local-embed[transformer]"
|
|
80
|
-
uv tool install "local-embed[gguf]"
|
|
96
|
+
uv tool install --force "local-embed[transformer]"
|
|
97
|
+
uv tool install --force "local-embed[gguf]"
|
|
81
98
|
```
|
|
82
99
|
|
|
100
|
+
`--force` (used by the installers too) is **idempotent** — the first run
|
|
101
|
+
installs, re-running **updates** to the latest PyPI release. A plain
|
|
102
|
+
`uv tool install local-embed` no-ops when the tool is already installed.
|
|
103
|
+
|
|
83
104
|
If you run local-embed inside an *existing* environment (alongside another
|
|
84
105
|
tool), install the backend into **that** environment's interpreter instead of
|
|
85
106
|
creating a separate one:
|
|
@@ -140,15 +161,14 @@ Installing does **not** fetch a model — get the weights first.
|
|
|
140
161
|
|
|
141
162
|
## Configuration: `local_embed.json5`
|
|
142
163
|
|
|
143
|
-
Everything — host, port, models,
|
|
164
|
+
Everything — host, port, models, backend — comes from
|
|
144
165
|
`local_embed.json5`, written by the CLI helpers or by hand. Path resolution:
|
|
145
166
|
`$LOCAL_EMBED_FILE` > slife project root (dev) > `~/.local-embed/local_embed.json5`.
|
|
146
167
|
|
|
147
168
|
```json5
|
|
148
169
|
{
|
|
149
|
-
active_model: "bge-m3", // model served by default
|
|
150
170
|
models: {
|
|
151
|
-
"bge-m3": { backend: "gguf", gguf_path: "…", device: "" },
|
|
171
|
+
"bge-m3": { backend: "gguf", gguf_path: "…", device: "", autoload: false },
|
|
152
172
|
"bge-m3-transformer": { backend: "transformer", model: "BAAI/bge-m3", device: "" }
|
|
153
173
|
},
|
|
154
174
|
env: { // injected into this process before any model loads
|
|
@@ -160,8 +180,14 @@ Everything — host, port, models, active model, backend — comes from
|
|
|
160
180
|
}
|
|
161
181
|
```
|
|
162
182
|
|
|
163
|
-
- `models` — map of name → `{backend, gguf_path | model, device, max_tokens}`.
|
|
164
|
-
|
|
183
|
+
- `models` — map of name → `{backend, gguf_path | model, device, max_tokens, autoload}`.
|
|
184
|
+
Every configured model is a peer; there is **no `active_model`** — each
|
|
185
|
+
request names the model it wants (standard OpenAI semantics).
|
|
186
|
+
- `autoload` (per model, default `false`) — a model's weights are large and
|
|
187
|
+
memory-hungry, so nothing loads until a request names it (**lazy**).
|
|
188
|
+
`autoload: true` on one model eager-loads just that model in the
|
|
189
|
+
background at startup (memory paid up front for a warm first embed); every
|
|
190
|
+
unflagged model stays lazy.
|
|
165
191
|
- `env:` — injected before any backend loads; an existing shell env var wins.
|
|
166
192
|
Without `HF_HUB_CACHE`, transformer repos resolve against the default cache
|
|
167
193
|
and a model downloaded elsewhere is silently re-fetched.
|
|
@@ -232,8 +258,8 @@ curl http://127.0.0.1:17347/v1/embeddings \
|
|
|
232
258
|
-d '{"model": "bge-m3", "input": ["hello world", "another text"]}'
|
|
233
259
|
```
|
|
234
260
|
|
|
235
|
-
`input` is a string or a list of strings; `model`
|
|
236
|
-
|
|
261
|
+
`input` is a string or a list of strings; `model` is **required** and names
|
|
262
|
+
any configured model. Returns the standard shape:
|
|
237
263
|
|
|
238
264
|
```json
|
|
239
265
|
{
|
|
@@ -247,27 +273,43 @@ curl http://127.0.0.1:17347/v1/embeddings \
|
|
|
247
273
|
}
|
|
248
274
|
```
|
|
249
275
|
|
|
250
|
-
|
|
251
|
-
row
|
|
276
|
+
OpenAI forbids empty-string input, so an empty/whitespace string in the
|
|
277
|
+
batch is rejected with a strict `400` — there is no zero-vector row
|
|
278
|
+
alignment on the wire; filter blanks before batching.
|
|
279
|
+
|
|
280
|
+
#### Errors — the OpenAI contract
|
|
281
|
+
|
|
282
|
+
Every error uses the standard envelope
|
|
283
|
+
`{"error": {"message", "type", "param", "code"}}` (`param` / `code` are
|
|
284
|
+
`null` when not applicable):
|
|
285
|
+
|
|
286
|
+
| Status | `type` | When | `param` / `code` |
|
|
287
|
+
|---|---|---|---|
|
|
288
|
+
| `400` | `invalid_request_error` | unparseable JSON body; `input` not a string/array of strings; missing `model`; empty / whitespace-only input | `input` / `model` |
|
|
289
|
+
| `400` | `invalid_request_error` | input exceeds the model's context length (`max_tokens` — never silently truncated) | `input` / `context_length_exceeded` |
|
|
290
|
+
| `404` | `invalid_request_error` | unknown `model` ("does not exist or you do not have access to it") | — / `model_not_found` |
|
|
291
|
+
| `503` | `server_error` | the model's engine is **still loading** (is_loading — retry shortly) or unavailable (backend dependency missing, load failed) | — |
|
|
292
|
+
| `500` | `server_error` | unexpected internal failure | — |
|
|
293
|
+
|
|
294
|
+
Split long documents into pieces of at most `max_tokens` tokens client-side
|
|
295
|
+
(slife's semantic drainer does this automatically).
|
|
252
296
|
|
|
253
297
|
### `GET /v1/models`
|
|
254
298
|
|
|
255
299
|
Every configured model, each with its real embedding dimension
|
|
256
|
-
(`dimension` / `dimension_known`), backend, load state
|
|
300
|
+
(`dimension` / `dimension_known`), backend, and load state. The listing is
|
|
301
|
+
standard — no `active` marker, all models are peers.
|
|
257
302
|
|
|
258
303
|
### `GET /v1/models/{id}`
|
|
259
304
|
|
|
260
305
|
One model's detail (the OpenAI `retrieve` endpoint); 404 with the standard
|
|
261
306
|
error envelope when the id is unknown.
|
|
262
307
|
|
|
263
|
-
### `POST /v1/models/{id}/activate`
|
|
264
|
-
|
|
265
|
-
Switch the active model (loads it on demand). A local-embed extension, not an
|
|
266
|
-
OpenAI endpoint.
|
|
267
|
-
|
|
268
308
|
### `GET /health`
|
|
269
309
|
|
|
270
|
-
`{status,
|
|
310
|
+
Liveness + engine state: `{status, models: [{name, backend, model,
|
|
311
|
+
dimension, dimension_known, loaded, available, max_tokens}]}` — `status`
|
|
312
|
+
is `ok` when any configured model's backend is usable.
|
|
271
313
|
|
|
272
314
|
### Any OpenAI client
|
|
273
315
|
|
|
@@ -300,10 +342,11 @@ config.
|
|
|
300
342
|
|
|
301
343
|
### `local-embed set` / `set-gguf` — write the config
|
|
302
344
|
|
|
303
|
-
`set` (transformer) and `set-gguf` (gguf) upsert a model in the config
|
|
304
|
-
|
|
305
|
-
|
|
306
|
-
untouched.
|
|
345
|
+
`set` (transformer) and `set-gguf` (gguf) upsert a model in the config and
|
|
346
|
+
pin port (and, for `set`, the HF cache + offline flag). Both are
|
|
347
|
+
**idempotent** — re-running yields the same config — and leave other models
|
|
348
|
+
untouched. There is no `active_model`: every configured model is a peer
|
|
349
|
+
and requests name the one they want.
|
|
307
350
|
|
|
308
351
|
| | `set` (transformer) | `set-gguf` |
|
|
309
352
|
|---|---|---|
|
|
@@ -373,84 +416,54 @@ claude mcp add --transport http local-embed http://127.0.0.1:17347/mcp
|
|
|
373
416
|
> reachable from the desktop app.
|
|
374
417
|
|
|
375
418
|
**Over MCP you get status, not embeddings.** The only MCP tool is the internal
|
|
376
|
-
`__check` status probe (
|
|
419
|
+
`__check` status probe (model list, dimensions, load state) — local-embed is
|
|
377
420
|
a *service-provider* MCP server, not a tool provider. To embed text, point
|
|
378
421
|
your consumer at the OpenAI-compatible `/v1/*` API instead. A client that only
|
|
379
422
|
speaks stdio can reach it through a stdio→HTTP bridge (e.g. `mcp-remote`).
|
|
380
423
|
|
|
381
|
-
##
|
|
382
|
-
|
|
383
|
-
local-embed is a
|
|
384
|
-
|
|
385
|
-
|
|
386
|
-
|
|
387
|
-
|
|
388
|
-
|
|
389
|
-
|
|
390
|
-
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
396
|
-
|
|
397
|
-
How it conforms:
|
|
398
|
-
|
|
399
|
-
- **Spawn target** — `local_embed/server.py` exposes `main()`, spawnable via
|
|
400
|
-
`python -m local_embed.server` (the contract's `sys.executable -m <module>`
|
|
401
|
-
shape).
|
|
402
|
-
- **Port signal** — emits `{"port": N}` on stdout (then closes stdout) only
|
|
403
|
-
after the FastMCP app is ready to serve (its lifespan completed), so the
|
|
404
|
-
host's first `initialize` always lands on a ready server.
|
|
405
|
-
- **Readiness = MCP `initialize` handshake** — startup is handshake-fast:
|
|
406
|
-
backend imports are resolved lazily and the model warms up *after* the first
|
|
407
|
-
`tools/list` (`warm_after_handshake`), never in the lifespan.
|
|
408
|
-
- **Internal `__check` tool** — `__`-prefixed, so the host's tool registration
|
|
409
|
-
filters it out of the LLM registry and `system_health` probes it
|
|
410
|
-
programmatically.
|
|
411
|
-
- **Non-MCP routes on the same port** — the OpenAI endpoints are
|
|
412
|
-
`@mcp.custom_route` handlers on the same uvicorn app (the "one port, two
|
|
413
|
-
protocols" pattern).
|
|
414
|
-
- **Logging adoption** — when spawned by slife it adopts `SLIFE_LOG_DIR` /
|
|
415
|
-
`SLIFE_AGENT_NAME` / `SLIFE_SESSION_ID` / `SLIFE_PLUGIN_NAME`, so the session
|
|
416
|
-
log lands next to the main log; an uncaught startup exception prints one
|
|
417
|
-
clean stderr line (the host relays it) with the full traceback in the file.
|
|
418
|
-
Standalone falls back to `~/.local-embed/logs`.
|
|
419
|
-
- **Config path** — honors `$LOCAL_EMBED_FILE` (slife exports it as
|
|
420
|
-
`<dir of slife.json5>/local_embed.json5`), else the standalone default.
|
|
421
|
-
- **Stable port** — binds the configured `17347` rather than an OS-assigned
|
|
422
|
-
port, so slife's `base_url` is fixed; a taken port is a hard error, not a
|
|
423
|
-
silent fallback.
|
|
424
|
-
|
|
425
|
-
### Registering with slife
|
|
426
|
-
|
|
427
|
-
Register local-embed as an external plugin and point slife's embedding config
|
|
428
|
-
at it (the unified OpenAI-compatible format slife uses for every embedding
|
|
429
|
-
model):
|
|
424
|
+
## Running as a slife embedding backend
|
|
425
|
+
|
|
426
|
+
local-embed is a **plugin** — one row in slife's central plugin spec, so
|
|
427
|
+
slife spawns it, watches it, and reports it like every other child plugin. It
|
|
428
|
+
is also runnable standalone (`local-embed` on PATH), which is what the CLI is
|
|
429
|
+
for; slife still consumes the MODEL SERVICE purely as an OpenAI-compatible
|
|
430
|
+
HTTP endpoint (never through MCP tools).
|
|
431
|
+
|
|
432
|
+
Because its config pins a port that a static embeddings `base_url` points at,
|
|
433
|
+
it is the one plugin with a **fixed** port: it binds that port itself and a
|
|
434
|
+
taken port is a hard error (no fallback — a second instance would double the
|
|
435
|
+
model in memory and serve nothing new). The host then reports the plugin as
|
|
436
|
+
failed to load, with the reason, and the running instance keeps serving.
|
|
437
|
+
|
|
438
|
+
Point slife's embedding config at the daemon's stable port:
|
|
430
439
|
|
|
431
440
|
```json5
|
|
432
|
-
plugins: {
|
|
433
|
-
external: [
|
|
434
|
-
{ name: "local-embed", module: "local_embed.server" }
|
|
435
|
-
]
|
|
436
|
-
},
|
|
437
441
|
embeddings: {
|
|
438
442
|
providers: {
|
|
439
|
-
|
|
443
|
+
local: {
|
|
440
444
|
base_url: "http://127.0.0.1:17347/v1", // stable port from local_embed.json5
|
|
441
445
|
api_key: "local",
|
|
446
|
+
model: "bge-m3", // id POSTed on /v1/embeddings
|
|
442
447
|
}
|
|
443
448
|
},
|
|
444
|
-
active_model: "local
|
|
449
|
+
active_model: "local", // provider-id only, or "local/<model>"
|
|
445
450
|
enabled: true
|
|
446
451
|
}
|
|
447
452
|
```
|
|
448
453
|
|
|
449
454
|
slife treats every embedding model as a remote OpenAI-compatible endpoint —
|
|
450
|
-
local-embed is one such endpoint. The model is
|
|
451
|
-
|
|
452
|
-
`
|
|
453
|
-
gracefully to keyword search
|
|
455
|
+
local-embed is one such endpoint. The model a request hits is the one the
|
|
456
|
+
caller names; when the config names no model, slife discovers it from
|
|
457
|
+
`GET /v1/models` (the first entry) on load. When the daemon is unreachable,
|
|
458
|
+
slife degrades gracefully to keyword search (`check_embeddings` in
|
|
459
|
+
`system_health` probes the **active** provider's HTTP endpoint and reports it
|
|
460
|
+
down — it never assumes that provider is local-embed).
|
|
461
|
+
|
|
462
|
+
The MCP surface it still serves is the internal **`__check`** (engine status:
|
|
463
|
+
model list, dimensions, load state) — a service-provider facade
|
|
464
|
+
for direct probing, not a slife plugin contract. It also serves plain OpenAPI
|
|
465
|
+
routes (`/v1/embeddings`, `/v1/models`, `/health`) on the same port via
|
|
466
|
+
`@mcp.custom_route` — one port, two protocols, no slife involvement.
|
|
454
467
|
|
|
455
468
|
## License
|
|
456
469
|
|
|
@@ -25,15 +25,18 @@ them ([Model weights](#model-weights)).
|
|
|
25
25
|
vector DB / RAG pipeline ──────┘ ▼
|
|
26
26
|
┌───────────────────────┐
|
|
27
27
|
slife (plugin host) ─── /mcp ────────────> │ local-embed │
|
|
28
|
-
│
|
|
28
|
+
│ models load on demand │
|
|
29
29
|
└───────────────────────┘
|
|
30
30
|
```
|
|
31
31
|
|
|
32
32
|
## Features
|
|
33
33
|
|
|
34
34
|
- **OpenAI-compatible** `/v1/embeddings` + Models API + `/health`.
|
|
35
|
-
- Two backends (GGUF / transformer), **many models
|
|
36
|
-
|
|
35
|
+
- Two backends (GGUF / transformer), **many models as peers — each request
|
|
36
|
+
names the one it wants** (standard OpenAI semantics, no "active model").
|
|
37
|
+
- **Lazy load** — startup is fast; each model materialises on the first
|
|
38
|
+
request that names it (per-model `autoload: true` preloads a specific
|
|
39
|
+
model at startup).
|
|
37
40
|
- **Real dimension** reported after load (a guessed width is never served).
|
|
38
41
|
- Runs standalone, and can also be **loaded as a slife plugin** (see
|
|
39
42
|
[Loading as a slife plugin](#loading-as-a-slife-plugin)).
|
|
@@ -42,18 +45,36 @@ Requires Python ≥ 3.13.
|
|
|
42
45
|
|
|
43
46
|
## Install
|
|
44
47
|
|
|
45
|
-
The core package is `fastmcp` + `starlette` + `
|
|
48
|
+
The core package is `fastmcp` + `starlette` + `json-five` (the JSON5 parser,
|
|
49
|
+
imported as `json5`); the model backends are
|
|
46
50
|
optional extras.
|
|
47
51
|
|
|
52
|
+
One-click installers (install `uv` if needed, then `uv tool install --force
|
|
53
|
+
local-embed` — the backend is **not** included, see below):
|
|
54
|
+
|
|
55
|
+
```bash
|
|
56
|
+
# macOS / Linux / WSL
|
|
57
|
+
curl -fsSL https://raw.githubusercontent.com/juzcn/slife/main/local-embed/install.sh | bash
|
|
58
|
+
|
|
59
|
+
# Windows PowerShell
|
|
60
|
+
powershell -ExecutionPolicy Bypass -Command "irm https://raw.githubusercontent.com/juzcn/slife/main/local-embed/install.ps1 | iex"
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
Manual installs are equivalent:
|
|
64
|
+
|
|
48
65
|
```bash
|
|
49
66
|
# the app + CLI (standalone tool)
|
|
50
|
-
uv tool install local-embed
|
|
67
|
+
uv tool install --force local-embed
|
|
51
68
|
|
|
52
69
|
# with a backend:
|
|
53
|
-
uv tool install "local-embed[transformer]"
|
|
54
|
-
uv tool install "local-embed[gguf]"
|
|
70
|
+
uv tool install --force "local-embed[transformer]"
|
|
71
|
+
uv tool install --force "local-embed[gguf]"
|
|
55
72
|
```
|
|
56
73
|
|
|
74
|
+
`--force` (used by the installers too) is **idempotent** — the first run
|
|
75
|
+
installs, re-running **updates** to the latest PyPI release. A plain
|
|
76
|
+
`uv tool install local-embed` no-ops when the tool is already installed.
|
|
77
|
+
|
|
57
78
|
If you run local-embed inside an *existing* environment (alongside another
|
|
58
79
|
tool), install the backend into **that** environment's interpreter instead of
|
|
59
80
|
creating a separate one:
|
|
@@ -114,15 +135,14 @@ Installing does **not** fetch a model — get the weights first.
|
|
|
114
135
|
|
|
115
136
|
## Configuration: `local_embed.json5`
|
|
116
137
|
|
|
117
|
-
Everything — host, port, models,
|
|
138
|
+
Everything — host, port, models, backend — comes from
|
|
118
139
|
`local_embed.json5`, written by the CLI helpers or by hand. Path resolution:
|
|
119
140
|
`$LOCAL_EMBED_FILE` > slife project root (dev) > `~/.local-embed/local_embed.json5`.
|
|
120
141
|
|
|
121
142
|
```json5
|
|
122
143
|
{
|
|
123
|
-
active_model: "bge-m3", // model served by default
|
|
124
144
|
models: {
|
|
125
|
-
"bge-m3": { backend: "gguf", gguf_path: "…", device: "" },
|
|
145
|
+
"bge-m3": { backend: "gguf", gguf_path: "…", device: "", autoload: false },
|
|
126
146
|
"bge-m3-transformer": { backend: "transformer", model: "BAAI/bge-m3", device: "" }
|
|
127
147
|
},
|
|
128
148
|
env: { // injected into this process before any model loads
|
|
@@ -134,8 +154,14 @@ Everything — host, port, models, active model, backend — comes from
|
|
|
134
154
|
}
|
|
135
155
|
```
|
|
136
156
|
|
|
137
|
-
- `models` — map of name → `{backend, gguf_path | model, device, max_tokens}`.
|
|
138
|
-
|
|
157
|
+
- `models` — map of name → `{backend, gguf_path | model, device, max_tokens, autoload}`.
|
|
158
|
+
Every configured model is a peer; there is **no `active_model`** — each
|
|
159
|
+
request names the model it wants (standard OpenAI semantics).
|
|
160
|
+
- `autoload` (per model, default `false`) — a model's weights are large and
|
|
161
|
+
memory-hungry, so nothing loads until a request names it (**lazy**).
|
|
162
|
+
`autoload: true` on one model eager-loads just that model in the
|
|
163
|
+
background at startup (memory paid up front for a warm first embed); every
|
|
164
|
+
unflagged model stays lazy.
|
|
139
165
|
- `env:` — injected before any backend loads; an existing shell env var wins.
|
|
140
166
|
Without `HF_HUB_CACHE`, transformer repos resolve against the default cache
|
|
141
167
|
and a model downloaded elsewhere is silently re-fetched.
|
|
@@ -206,8 +232,8 @@ curl http://127.0.0.1:17347/v1/embeddings \
|
|
|
206
232
|
-d '{"model": "bge-m3", "input": ["hello world", "another text"]}'
|
|
207
233
|
```
|
|
208
234
|
|
|
209
|
-
`input` is a string or a list of strings; `model`
|
|
210
|
-
|
|
235
|
+
`input` is a string or a list of strings; `model` is **required** and names
|
|
236
|
+
any configured model. Returns the standard shape:
|
|
211
237
|
|
|
212
238
|
```json
|
|
213
239
|
{
|
|
@@ -221,27 +247,43 @@ curl http://127.0.0.1:17347/v1/embeddings \
|
|
|
221
247
|
}
|
|
222
248
|
```
|
|
223
249
|
|
|
224
|
-
|
|
225
|
-
row
|
|
250
|
+
OpenAI forbids empty-string input, so an empty/whitespace string in the
|
|
251
|
+
batch is rejected with a strict `400` — there is no zero-vector row
|
|
252
|
+
alignment on the wire; filter blanks before batching.
|
|
253
|
+
|
|
254
|
+
#### Errors — the OpenAI contract
|
|
255
|
+
|
|
256
|
+
Every error uses the standard envelope
|
|
257
|
+
`{"error": {"message", "type", "param", "code"}}` (`param` / `code` are
|
|
258
|
+
`null` when not applicable):
|
|
259
|
+
|
|
260
|
+
| Status | `type` | When | `param` / `code` |
|
|
261
|
+
|---|---|---|---|
|
|
262
|
+
| `400` | `invalid_request_error` | unparseable JSON body; `input` not a string/array of strings; missing `model`; empty / whitespace-only input | `input` / `model` |
|
|
263
|
+
| `400` | `invalid_request_error` | input exceeds the model's context length (`max_tokens` — never silently truncated) | `input` / `context_length_exceeded` |
|
|
264
|
+
| `404` | `invalid_request_error` | unknown `model` ("does not exist or you do not have access to it") | — / `model_not_found` |
|
|
265
|
+
| `503` | `server_error` | the model's engine is **still loading** (is_loading — retry shortly) or unavailable (backend dependency missing, load failed) | — |
|
|
266
|
+
| `500` | `server_error` | unexpected internal failure | — |
|
|
267
|
+
|
|
268
|
+
Split long documents into pieces of at most `max_tokens` tokens client-side
|
|
269
|
+
(slife's semantic drainer does this automatically).
|
|
226
270
|
|
|
227
271
|
### `GET /v1/models`
|
|
228
272
|
|
|
229
273
|
Every configured model, each with its real embedding dimension
|
|
230
|
-
(`dimension` / `dimension_known`), backend, load state
|
|
274
|
+
(`dimension` / `dimension_known`), backend, and load state. The listing is
|
|
275
|
+
standard — no `active` marker, all models are peers.
|
|
231
276
|
|
|
232
277
|
### `GET /v1/models/{id}`
|
|
233
278
|
|
|
234
279
|
One model's detail (the OpenAI `retrieve` endpoint); 404 with the standard
|
|
235
280
|
error envelope when the id is unknown.
|
|
236
281
|
|
|
237
|
-
### `POST /v1/models/{id}/activate`
|
|
238
|
-
|
|
239
|
-
Switch the active model (loads it on demand). A local-embed extension, not an
|
|
240
|
-
OpenAI endpoint.
|
|
241
|
-
|
|
242
282
|
### `GET /health`
|
|
243
283
|
|
|
244
|
-
`{status,
|
|
284
|
+
Liveness + engine state: `{status, models: [{name, backend, model,
|
|
285
|
+
dimension, dimension_known, loaded, available, max_tokens}]}` — `status`
|
|
286
|
+
is `ok` when any configured model's backend is usable.
|
|
245
287
|
|
|
246
288
|
### Any OpenAI client
|
|
247
289
|
|
|
@@ -274,10 +316,11 @@ config.
|
|
|
274
316
|
|
|
275
317
|
### `local-embed set` / `set-gguf` — write the config
|
|
276
318
|
|
|
277
|
-
`set` (transformer) and `set-gguf` (gguf) upsert a model in the config
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
untouched.
|
|
319
|
+
`set` (transformer) and `set-gguf` (gguf) upsert a model in the config and
|
|
320
|
+
pin port (and, for `set`, the HF cache + offline flag). Both are
|
|
321
|
+
**idempotent** — re-running yields the same config — and leave other models
|
|
322
|
+
untouched. There is no `active_model`: every configured model is a peer
|
|
323
|
+
and requests name the one they want.
|
|
281
324
|
|
|
282
325
|
| | `set` (transformer) | `set-gguf` |
|
|
283
326
|
|---|---|---|
|
|
@@ -347,84 +390,54 @@ claude mcp add --transport http local-embed http://127.0.0.1:17347/mcp
|
|
|
347
390
|
> reachable from the desktop app.
|
|
348
391
|
|
|
349
392
|
**Over MCP you get status, not embeddings.** The only MCP tool is the internal
|
|
350
|
-
`__check` status probe (
|
|
393
|
+
`__check` status probe (model list, dimensions, load state) — local-embed is
|
|
351
394
|
a *service-provider* MCP server, not a tool provider. To embed text, point
|
|
352
395
|
your consumer at the OpenAI-compatible `/v1/*` API instead. A client that only
|
|
353
396
|
speaks stdio can reach it through a stdio→HTTP bridge (e.g. `mcp-remote`).
|
|
354
397
|
|
|
355
|
-
##
|
|
356
|
-
|
|
357
|
-
local-embed is a
|
|
358
|
-
|
|
359
|
-
|
|
360
|
-
|
|
361
|
-
|
|
362
|
-
|
|
363
|
-
|
|
364
|
-
|
|
365
|
-
|
|
366
|
-
|
|
367
|
-
|
|
368
|
-
|
|
369
|
-
|
|
370
|
-
|
|
371
|
-
How it conforms:
|
|
372
|
-
|
|
373
|
-
- **Spawn target** — `local_embed/server.py` exposes `main()`, spawnable via
|
|
374
|
-
`python -m local_embed.server` (the contract's `sys.executable -m <module>`
|
|
375
|
-
shape).
|
|
376
|
-
- **Port signal** — emits `{"port": N}` on stdout (then closes stdout) only
|
|
377
|
-
after the FastMCP app is ready to serve (its lifespan completed), so the
|
|
378
|
-
host's first `initialize` always lands on a ready server.
|
|
379
|
-
- **Readiness = MCP `initialize` handshake** — startup is handshake-fast:
|
|
380
|
-
backend imports are resolved lazily and the model warms up *after* the first
|
|
381
|
-
`tools/list` (`warm_after_handshake`), never in the lifespan.
|
|
382
|
-
- **Internal `__check` tool** — `__`-prefixed, so the host's tool registration
|
|
383
|
-
filters it out of the LLM registry and `system_health` probes it
|
|
384
|
-
programmatically.
|
|
385
|
-
- **Non-MCP routes on the same port** — the OpenAI endpoints are
|
|
386
|
-
`@mcp.custom_route` handlers on the same uvicorn app (the "one port, two
|
|
387
|
-
protocols" pattern).
|
|
388
|
-
- **Logging adoption** — when spawned by slife it adopts `SLIFE_LOG_DIR` /
|
|
389
|
-
`SLIFE_AGENT_NAME` / `SLIFE_SESSION_ID` / `SLIFE_PLUGIN_NAME`, so the session
|
|
390
|
-
log lands next to the main log; an uncaught startup exception prints one
|
|
391
|
-
clean stderr line (the host relays it) with the full traceback in the file.
|
|
392
|
-
Standalone falls back to `~/.local-embed/logs`.
|
|
393
|
-
- **Config path** — honors `$LOCAL_EMBED_FILE` (slife exports it as
|
|
394
|
-
`<dir of slife.json5>/local_embed.json5`), else the standalone default.
|
|
395
|
-
- **Stable port** — binds the configured `17347` rather than an OS-assigned
|
|
396
|
-
port, so slife's `base_url` is fixed; a taken port is a hard error, not a
|
|
397
|
-
silent fallback.
|
|
398
|
-
|
|
399
|
-
### Registering with slife
|
|
400
|
-
|
|
401
|
-
Register local-embed as an external plugin and point slife's embedding config
|
|
402
|
-
at it (the unified OpenAI-compatible format slife uses for every embedding
|
|
403
|
-
model):
|
|
398
|
+
## Running as a slife embedding backend
|
|
399
|
+
|
|
400
|
+
local-embed is a **plugin** — one row in slife's central plugin spec, so
|
|
401
|
+
slife spawns it, watches it, and reports it like every other child plugin. It
|
|
402
|
+
is also runnable standalone (`local-embed` on PATH), which is what the CLI is
|
|
403
|
+
for; slife still consumes the MODEL SERVICE purely as an OpenAI-compatible
|
|
404
|
+
HTTP endpoint (never through MCP tools).
|
|
405
|
+
|
|
406
|
+
Because its config pins a port that a static embeddings `base_url` points at,
|
|
407
|
+
it is the one plugin with a **fixed** port: it binds that port itself and a
|
|
408
|
+
taken port is a hard error (no fallback — a second instance would double the
|
|
409
|
+
model in memory and serve nothing new). The host then reports the plugin as
|
|
410
|
+
failed to load, with the reason, and the running instance keeps serving.
|
|
411
|
+
|
|
412
|
+
Point slife's embedding config at the daemon's stable port:
|
|
404
413
|
|
|
405
414
|
```json5
|
|
406
|
-
plugins: {
|
|
407
|
-
external: [
|
|
408
|
-
{ name: "local-embed", module: "local_embed.server" }
|
|
409
|
-
]
|
|
410
|
-
},
|
|
411
415
|
embeddings: {
|
|
412
416
|
providers: {
|
|
413
|
-
|
|
417
|
+
local: {
|
|
414
418
|
base_url: "http://127.0.0.1:17347/v1", // stable port from local_embed.json5
|
|
415
419
|
api_key: "local",
|
|
420
|
+
model: "bge-m3", // id POSTed on /v1/embeddings
|
|
416
421
|
}
|
|
417
422
|
},
|
|
418
|
-
active_model: "local
|
|
423
|
+
active_model: "local", // provider-id only, or "local/<model>"
|
|
419
424
|
enabled: true
|
|
420
425
|
}
|
|
421
426
|
```
|
|
422
427
|
|
|
423
428
|
slife treats every embedding model as a remote OpenAI-compatible endpoint —
|
|
424
|
-
local-embed is one such endpoint. The model is
|
|
425
|
-
|
|
426
|
-
`
|
|
427
|
-
gracefully to keyword search
|
|
429
|
+
local-embed is one such endpoint. The model a request hits is the one the
|
|
430
|
+
caller names; when the config names no model, slife discovers it from
|
|
431
|
+
`GET /v1/models` (the first entry) on load. When the daemon is unreachable,
|
|
432
|
+
slife degrades gracefully to keyword search (`check_embeddings` in
|
|
433
|
+
`system_health` probes the **active** provider's HTTP endpoint and reports it
|
|
434
|
+
down — it never assumes that provider is local-embed).
|
|
435
|
+
|
|
436
|
+
The MCP surface it still serves is the internal **`__check`** (engine status:
|
|
437
|
+
model list, dimensions, load state) — a service-provider facade
|
|
438
|
+
for direct probing, not a slife plugin contract. It also serves plain OpenAPI
|
|
439
|
+
routes (`/v1/embeddings`, `/v1/models`, `/health`) on the same port via
|
|
440
|
+
`@mcp.custom_route` — one port, two protocols, no slife involvement.
|
|
428
441
|
|
|
429
442
|
## License
|
|
430
443
|
|