@bendyline/gezel-inference 1.0.0 → 1.0.2

package/README.md

@@ -1,14 +1,96 @@
 # @bendyline/gezel-inference
 
-The shared on-device runtime used by Gezel's full service and by embedded applications. It owns native engine lifecycle and capacity admission, model preparation, OpenAI-compatible chat, knowledge retrieval, worker entry points, and native release verification. The full service supplies product state, tools, cloud providers, UI and background work. There is one implementation of the shared runtime.
+The on-device inference runtime shared by [Gezel](https://github.com/bendyline/gezel)
+and by applications that embed it. It runs local AI models on the user's own
+computer and answers them through OpenAI-compatible routes:
 
-Use the application SDK with `host: { mode: 'in-process', inferenceOnly: true }`. Install this package alongside `@bendyline/gezel-app-sdk`. Startup opens no HTTP listener and returns an authenticated Fetch transport serving `/v1/chat`, `/v1/responses`, `/v1/embeddings`, `/v1/models`, `/v1/models/ensure`, `/v1/knowledge` and the engine routes; `model: "gezel:embeddings"` runs the on-device text embedder. It does not download models. Model preparation requires an explicit request. This is the only implementation of the inference-only profile: a child-hosted `gezeld` serves the same host behind the service's loopback listener.
+- **Native engines** — starting, sizing, and stopping llama.cpp, MLX, and the
+  other on-device engines, with admission control so models fit in memory.
+- **Model management** — listing, downloading, verifying, and preparing local
+  chat models.
+- **Embeddings and knowledge** — the on-device text embedder
+  (`model: "gezel:embeddings"`) and search over installed knowledge catalogs,
+  each in its own worker.
+- **OpenAI-compatible routes** — `/v1/chat`, `/v1/responses`,
+  `/v1/embeddings`, `/v1/models`, `/v1/models/ensure`, `/v1/knowledge`, and
+  the engine routes.
 
-- `/packaging` stages the pinned native release for an Electron host.
-- `/native-release` exposes the release pins without starting inference.
-- `/speech` exposes the shared Kokoro frontend and dictionary resolver.
-- `/internal/*` is the coordinated workspace interface for Gezel's full service; external applications should use the SDK.
+It holds no product state: no gezels, projects, chats, tools, cloud providers,
+or UI. Those belong to the full Gezel service, `@bendyline/gezel-service`,
+which uses this same runtime.
 
-The build copies only chat-model and knowledge-catalog metadata from the exact Gilde pin owned by the catalog package. This is generated content, never a separately edited catalog. `@bendyline/gezel-catalog` loads it through its ordinary source API. The full service separately installs all of Gilde.
+## Install
 
-Build with `pnpm --filter @bendyline/gezel-inference build` under the repository dependency lease. Unit and integration coverage for the moved runtime remains in the service suite; this package additionally tests the isolated host boundary.
+```bash
+npm install @bendyline/gezel-app-sdk @bendyline/gezel-inference
+```
+
+Node 24 or newer. On-device text embeddings and the optional relevance model
+also need the `@huggingface/transformers` peer:
+
+```bash
+npm install @huggingface/transformers@^4.3.1
+```
+
+## Use it through the app SDK
+
+Most applications should not import this package directly. Install it beside
+[`@bendyline/gezel-app-sdk`](https://www.npmjs.com/package/@bendyline/gezel-app-sdk)
+and ask the SDK for an inference-only host. The SDK loads this runtime inside
+your process, so there is no second process to supervise, no separate Node
+executable to ship, and no loopback network connection:
+
+```ts
+import { connectOrHost } from '@bendyline/gezel-app-sdk/host';
+
+const gezel = await connectOrHost({
+  appId: 'acme.editor',
+  appName: 'Acme Editor',
+  scopes: ['openai'],
+  host: { mode: 'in-process', inferenceOnly: true },
+});
+
+const { chatModel } = await gezel.ensureModel({ model: 'qwen3.5-0.8b-q4' });
+
+const reply = await gezel.openai.chat({
+  model: chatModel, // 'mlx:qwen3.5-0.8b-q4' on Apple Silicon, 'llama-cpp:…' elsewhere
+  messages: [{ role: 'user', content: 'Suggest a title for these notes: …' }],
+});
+console.log(reply.choices[0].message.content);
+
+await gezel.close();
+```
+
+The runtime never downloads a model on its own: weights are fetched only when
+your app asks for them, for example with `ensureModel`. See the
+[app SDK README](https://www.npmjs.com/package/@bendyline/gezel-app-sdk) for
+consent, hosting options, shipping engines and models with your app, and
+knowledge catalogs.
+
+## Entry points
+
+| Import | What it provides |
+|---|---|
+| `@bendyline/gezel-inference` | `startService()`, the in-process host the app SDK calls, and `verifyNativeBinaries()` |
+| `@bendyline/gezel-inference/native-release` | The pinned native engine release and its archive digests, as data. Importing it starts nothing |
+| `@bendyline/gezel-inference/packaging` | Build-time helpers that stage the pinned native engines for an Electron app |
+| `@bendyline/gezel-inference/speech` | The Kokoro text-to-speech frontend and its pronunciation-dictionary lookup |
+
+The package also includes chat-model and knowledge-catalog metadata generated
+from the pinned [`@bendyline/gilde`](https://www.npmjs.com/package/@bendyline/gilde)
+catalog. The pronunciation dictionaries' license is at
+`dist/kokoro-lexicon/LICENSE`.
+
+## Stability
+
+The entry points above are public API under semver.
+`@bendyline/gezel-inference/internal/*` is not: it is how the Gezel service
+reaches the runtime, always installed at exactly the same version, and it can
+change in any release. Do not import it.
+
+## Documentation
+
+- [Repository and full documentation](https://github.com/bendyline/gezel)
+- [Embedding guide](https://github.com/bendyline/gezel/blob/main/docs/embedding-sdk.md)
+
+MIT © Bendyline

package/dist/cache/adapter.d.ts

@@ -45,6 +45,13 @@ export interface EngineCacheAdapter {
      * gone for accounting purposes either way.
      */
     evict(sessionIds: readonly string[]): Promise<void>;
+    /**
+     * This session will never be resumed (a one-shot `gezel run`): keep only
+     * what helps other sessions, such as a shared prompt-prefix snapshot, and
+     * never a resumable snapshot of its own. Optional — an engine that keeps
+     * nothing on disk can ignore it. Must not do I/O.
+     */
+    markEphemeral?(sessionId: string): void;
     /**
      * Snapshot of what the engine currently has cached. Polled by the
      * controller every few seconds for telemetry; also informs the

package/dist/cache/controller.d.ts

@@ -97,6 +97,12 @@ export declare class SessionCacheController {
      * `setBudget` after construction.
      */
     private readonly pendingBudgets;
+    /**
+     * One-shot sessions, remembered so an adapter registered after the session
+     * was created (engines start lazily, on the first turn) still learns them.
+     * Bounded: only a session's own turns need the mark.
+     */
+    private readonly ephemeralSessions;
     private readonly now;
     private readonly logger;
     private reconcileTimer?;
@@ -109,6 +115,12 @@ export declare class SessionCacheController {
      * sets a per-engine RAM-aware default via `setBudget`.
      */
     registerAdapter(adapter: EngineCacheAdapter): void;
+    /**
+     * A session that will never be resumed keeps no resumable cache of its own
+     * on disk. Every one-shot `gezel run` left a ~300 MB llama slot snapshot
+     * that nothing ever read (2026-10-15 npm audit).
+     */
+    markEphemeral(sessionId: string): void;
     /**
      * Set the memory budget for a provider. Operator-tunable; Phase 3
      * exposes this in the Settings UI. Eviction kicks in immediately if

package/dist/cache/controller.js

@@ -43,6 +43,7 @@ function emptyHitsBySource() {
     return { memory: 0, disk: 0, prefix: 0, fresh: 0 };
 }
 const RECENT_TURN_WINDOW = 50;
+const MAX_REMEMBERED_EPHEMERAL_SESSIONS = 1024;
 export class SessionCacheController {
     providers = new Map();
     /**
@@ -55,6 +56,12 @@ export class SessionCacheController {
      * `setBudget` after construction.
      */
     pendingBudgets = new Map();
+    /**
+     * One-shot sessions, remembered so an adapter registered after the session
+     * was created (engines start lazily, on the first turn) still learns them.
+     * Bounded: only a session's own turns need the mark.
+     */
+    ephemeralSessions = new Set();
     now;
     logger;
     reconcileTimer;
@@ -93,6 +100,23 @@ export class SessionCacheController {
             gezelOutcomes: new Map(),
             recentOutcomes: [],
         });
+        for (const sessionId of this.ephemeralSessions)
+            adapter.markEphemeral?.(sessionId);
+    }
+    /**
+     * A session that will never be resumed keeps no resumable cache of its own
+     * on disk. Every one-shot `gezel run` left a ~300 MB llama slot snapshot
+     * that nothing ever read (2026-10-15 npm audit).
+     */
+    markEphemeral(sessionId) {
+        this.ephemeralSessions.add(sessionId);
+        if (this.ephemeralSessions.size > MAX_REMEMBERED_EPHEMERAL_SESSIONS) {
+            const oldest = this.ephemeralSessions.values().next().value;
+            if (oldest !== undefined)
+                this.ephemeralSessions.delete(oldest);
+        }
+        for (const state of this.providers.values())
+            state.adapter.markEphemeral?.(sessionId);
     }
     /**
      * Set the memory budget for a provider. Operator-tunable; Phase 3

package/dist/catalog/chat-models/bt/btl4-35b-q4/manifest.json

@@ -37,6 +37,7 @@
     "toolCallFormat": "function-call"
   },
   "behaviors": [
+    "provider.merge-system-messages",
     "reasoning.strip-think-tags",
     "prompt.private-reasoning-guidance",
     "prompt.tool-cookbook-condensed",

package/dist/catalog/chat-models/gl/glm-5.2-754b-q2/manifest.json

@@ -26,6 +26,7 @@
   "version": "1.0.0",
   "updatedAt": "2026-07-25T00:00:00Z",
   "license": "MIT",
+  "licenseClass": "open",
   "parameterSize": "754B",
   "approxSizeBytes": 211075856448,
   "supportsTools": true,