@nemesis-oss/ollama-sdk 1.13.0 → 1.14.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +24 -0
- package/README.md +110 -4
- package/dist/{anthropic-B9AFyzXI.d.ts → anthropic-DrDUfbfQ.d.ts} +1 -1
- package/dist/{anthropic-CYy_ECam.d.cts → anthropic-UeaVjLeP.d.cts} +1 -1
- package/dist/{compose-6UOPEknN.d.ts → compose-CeFbHJeO.d.ts} +1 -1
- package/dist/{compose-DhvoUrWa.d.cts → compose-DFkmtUOP.d.cts} +1 -1
- package/dist/generated-api.d.cts +5 -5
- package/dist/generated-api.d.ts +5 -5
- package/dist/generated-runtime.d.cts +2 -2
- package/dist/generated-runtime.d.ts +2 -2
- package/dist/index.cjs +34 -2
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +40 -11
- package/dist/index.d.ts +40 -11
- package/dist/index.js +34 -2
- package/dist/index.js.map +1 -1
- package/dist/mcp-generated.d.cts +2 -2
- package/dist/mcp-generated.d.ts +2 -2
- package/dist/{runtime-CIQYCRL3.d.ts → runtime-BIjr5ME4.d.ts} +1 -1
- package/dist/{runtime-DmI1cOEq.d.cts → runtime-C4RRHqOs.d.cts} +1 -1
- package/dist/skills.d.cts +3 -3
- package/dist/skills.d.ts +3 -3
- package/dist/testing.cjs +281 -0
- package/dist/testing.cjs.map +1 -0
- package/dist/testing.d.cts +336 -0
- package/dist/testing.d.ts +336 -0
- package/dist/testing.js +275 -0
- package/dist/testing.js.map +1 -0
- package/dist/{types-B57ZZfp3.d.cts → types-DbPnfVh-.d.cts} +20 -3
- package/dist/{types-B57ZZfp3.d.ts → types-DbPnfVh-.d.ts} +20 -3
- package/package.json +14 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,29 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## [1.14.0] — Speculative-decoding model assembly & the public testing subpath: `draftGguf`, `draft_num_predict` docs, `@nemesis-oss/ollama-sdk/testing`
|
|
4
|
+
|
|
5
|
+
The October 12 digest ("Speculative Decoding & Draft Model Creation") reviewed the official model-creation surfaces — `draft_files` + `draft_quantize` on `POST /api/create`, the `draft_num_predict` Modelfile parameter and runtime option — and asked for an exportable zero-network mock server. Audited against the codebase first (see [ADR 0029](./docs/adr/0029-public-testing-subpath-speculative-draft-conveniences.md)): the digest's headline typing claims were stale — `draft_files`/`draft_quantize` already shipped on both the generated `CreateRequest` and the hand-written `CreateRequestOptions` (official OpenAPI, 2026-10-07 pin), and `draft_num_predict` was already typed + documented on the runtime `ModelOptions`; the "DRAFT `<model>`" Modelfile instruction the digest describes is not in the current official docs (only the parameter is). The real gaps — a path-based draft upload convenience, wire tests, discoverable documentation, and the not-exported mock server — are closed here. All additive, no breaking changes. 932 pre-existing tests stay green, 28 new tests added.
|
|
6
|
+
|
|
7
|
+
### Speculative model assembly (`createModelFromGguf` → `draftGguf`)
|
|
8
|
+
|
|
9
|
+
- **`CreateModelFromGgufOptions.draftGguf`**: pass the draft network's GGUF path(s) alongside the primary shards — each is uploaded as a blob and assembled into `draft_files` under its basename, the same protocol as primary shards. Split draft GGUFs, explicit `draft_files` digest maps (merged; path uploads win on basename collision), and `draft_quantize` passthrough all work; no draft → no `draft_files` key on the wire at all.
|
|
10
|
+
- **Discoverable speculative-decoding docs**: `CreateRequestOptions.parameters` now documents `draft_num_predict` (separate draft models default to 4; `0` disables drafting; embedded MTP tensors require setting it explicitly — official Modelfile reference), and `draft_files`/`draft_quantize` carry the speculative-decoding context with the verification-semantics note that makes aggressive draft compression safe.
|
|
11
|
+
- Wire tests pin the whole contract: `createModel` round-trips `draft_files` + `draft_quantize` + `parameters.draft_num_predict`; `createModelFromGguf` uploads draft blobs separately (HEAD + POST per shard) and never leaks `draftGguf` or empty maps into the request body.
|
|
12
|
+
|
|
13
|
+
### Public testing subpath (`@nemesis-oss/ollama-sdk/testing`)
|
|
14
|
+
|
|
15
|
+
- The SDK's internal mock engine is **promoted, not forked** (ADR 0029, same discipline as ADRs 0027/0028): `test/mocks/ollama-mock-server.ts` is now a re-export shim, so the SDK's own suite exercises exactly the code consumers import.
|
|
16
|
+
- **`createMockOllamaServer(options?)`** — the recommended factory. Built-in **default routes** answer the official liveness probe (`HEAD /` → 200, `GET /` → the plain-text body "Ollama is running" byte-for-byte) and `GET /api/version` (configurable version, default `0.14.0`), so `client.ping()` works against a fresh mock with zero setup. `defaultRoutes: false` restores bare 404s; `register()` always wins over the defaults.
|
|
17
|
+
- **Request recording** — `server.requests` captures every request (method, path, parsed body, timestamp), including default-route hits and 404s; `clearRequests()` for `beforeEach` hooks. Wire-level assertions without per-route callbacks.
|
|
18
|
+
- **In-band mid-stream error injection** — `injectErrorFrame: { message, after? }` emits an `{"error": "..."}` NDJSON frame while HTTP stays 200, exactly how a real daemon reports GPU OOM/driver crashes mid-generation. The default injection point is `chunks.length - 1`: every content chunk, then the error frame, with the final `done` frame never emitted — an error frame after `done` would be unread (the stream has already completed cleanly). `after: 0` errors before any content; explicit larger values move the position. The SDK's stream readers raise `OllamaStreamError` with the partial content.
|
|
19
|
+
- **`rawBody`** route option for exact non-NDJSON bodies (the liveness body uses it — no trailing newline).
|
|
20
|
+
- **Chunk builders** producing correctly-shaped `/api/chat` frames: `mockChatChunk` (one frame, counter defaults on the done frame for telemetry paths), `mockTextChunks` (chunked text + done frame), and `mockToolCallChunks` — parallel tool calls in one frame, the documented native shape, followed by the done frame.
|
|
21
|
+
- **Packaging**: dedicated `./testing` entry in the exports map (paired `.d.ts`/`.d.cts`), a new tsup entry, and the consumer-type verification now compiles a real CJS (node16) **and** ESM (nodenext) consumer importing the subpath. Node.js-only by design (`node:http`) — never imported from the Edge-safe main entry, so `verify:edge-runtime` stays clean.
|
|
22
|
+
|
|
23
|
+
### Lab
|
|
24
|
+
|
|
25
|
+
- `lab/15-speculative-draft/measure-draft-speedup.ts`: live harness comparing `draft_num_predict` 0/4/8/16 — warm-up per setting, 5 samples each, median `tokensPerSecond` via `formatTelemetry`, and a speedup table against drafting-disabled. Makes "the server has no draft network, the option is being ignored" obvious at a glance.
|
|
26
|
+
|
|
3
27
|
## [1.13.0] — docs.ollama.com re-audit: parity verifier reads checkbox state, strict compat types, web-search agent tools
|
|
4
28
|
|
|
5
29
|
A full re-read of docs.ollama.com (API reference, capability guides, integrations, OpenAPI spec) against the SDK. The native API was already at parity except for two upstream changes; the real defect was in the compatibility-parity tooling, which had been reporting fields Ollama documents as unsupported as "supported".
|
package/README.md
CHANGED
|
@@ -33,7 +33,8 @@ Nexum’s model harness (2026-10-09) exercised this SDK via `gemma4:cloud` and `
|
|
|
33
33
|
- 🌉 **OpenAI & Anthropic Compatibility Bridges**: Built-in clients for `/v1/chat/completions`, `/v1/responses`, `/v1/models`, and `/v1/messages`, including `reasoning_effort`/`reasoning.effort` for thinking models — plus an ergonomic dual-mode `client.responses.create()` bridge that prefers native `/v1/responses` and transparently re-issues via `/api/chat` on older servers (pre-v0.13.3).
|
|
34
34
|
- 💬 **KV-Cache-Aware Conversation Sessions**: `client.session(model, systemPrompt)` keeps an append-only, prefix-stable history that maximizes Ollama's KV-cache reuse across turns, surfaces per-turn + cumulative cache hit rates (`prompt_eval_cached_count`) so cache degradation is visible instead of silent, and offers a one-call `compact()` sliding-window escape hatch when a session outgrows its window.
|
|
35
35
|
- 🛑 **Context-Window Safety**: Heuristic client-side token estimation warns (or throws, `onContextOverflow: 'throw'`) before a request is sent when the prompt approaches the effective `num_ctx` window — Ollama's default behavior is to _silently truncate_ oversized prompts — and `defaultContextLength` makes the window explicit on every request.
|
|
36
|
-
- 📦 **Blob & Custom Model Publishing**: Content-addressed blob management per Ollama's documented protocol — `computeBlobDigest()` (SHA-256 → `sha256:<hex>`), `createBlobFromData()`/`createBlobFromFile()` (HEAD-check + upload, skipping existing blobs), and one-shot `createModelFromGguf()` (upload GGUF shard blobs, then `/api/create` with `files: {name: digest}`).
|
|
36
|
+
- 📦 **Blob & Custom Model Publishing**: Content-addressed blob management per Ollama's documented protocol — `computeBlobDigest()` (SHA-256 → `sha256:<hex>`), `createBlobFromData()`/`createBlobFromFile()` (HEAD-check + upload, skipping existing blobs), and one-shot `createModelFromGguf()` (upload GGUF shard blobs, then `/api/create` with `files: {name: digest}`) — with **speculative-decoding assembly**: `draftGguf` uploads the draft network's shards into `draft_files`, `draft_quantize` compresses them independently, and `parameters.draft_num_predict` tunes the per-step draft length.
|
|
37
|
+
- 🧪 **In-Memory Mock Server (`@nemesis-oss/ollama-sdk/testing`)**: The SDK's own deterministic test double, promoted public — `createMockOllamaServer()` with scriptable NDJSON routes, built-in liveness/version defaults (so `ping()` works out of the box), request recording for wire-level assertions, mid-stream **in-band error injection** (`{"error": "..."}` frames at HTTP 200, exactly how a real daemon reports GPU OOM), connection drops, and correctly-shaped chunk builders for streaming text and **parallel tool calls** — zero network, zero daemon, CI-deterministic.
|
|
37
38
|
- 🧵 **Batched Embeddings with Backpressure**: `client.embedBatch()` splits corpora of any size into bounded `/api/embed` batches (default 32 inputs × 3 in flight) — order-preserving, fail-fast, progress-reporting — so RAG/vector ingestion can't OOM a local daemon or saturate `OLLAMA_MAX_QUEUE` the way `Promise.all` floods do, with per-string context-window pre-flight against silent truncation.
|
|
38
39
|
- 🧹 **Client Teardown (`destroy()`)**: One call aborts every in-flight request, active stream, and queued capacity waiter with `OllamaAbortError` — the clean-exit path for `worker_threads`, CLI runners, and short-lived scripts where dangling fetch bodies keep the event loop alive.
|
|
39
40
|
- 🧠 **Dynamic Context-Window Discovery**: `client.models.getContextLength({ model })` resolves the real window from the server — the allocated window of the running instance (`/api/ps`), the Modelfile `num_ctx` default, and the native GGUF maximum (`model_info`), in precedence order — so applications stop hardcoding 2048/4096 guesses.
|
|
@@ -497,6 +498,45 @@ await client.models.createModelFromGguf('my-model', [
|
|
|
497
498
|
]);
|
|
498
499
|
```
|
|
499
500
|
|
|
501
|
+
### Speculative Decoding (Draft Models)
|
|
502
|
+
|
|
503
|
+
Ollama packages speculative decoding as a small **draft network** whose token
|
|
504
|
+
guesses are batch-verified against the primary weights in a single forward
|
|
505
|
+
pass — the draft runs ahead, the primary model confirms. On the wire this is
|
|
506
|
+
`draft_files` (draft GGUF digests) + `draft_quantize` on `/api/create`, the
|
|
507
|
+
`draft_num_predict` Modelfile parameter, and the `draft_num_predict` runtime
|
|
508
|
+
option at generation time. The SDK covers the full flow:
|
|
509
|
+
|
|
510
|
+
```typescript
|
|
511
|
+
// 1. Assemble: primary + draft shards, draft weights quantized independently
|
|
512
|
+
// (draft tokens are verified, never returned — aggressive compression is
|
|
513
|
+
// usually safe). Separate draft models default draft_num_predict to 4;
|
|
514
|
+
// 0 disables drafting; models with embedded MTP tensors require setting
|
|
515
|
+
// it explicitly.
|
|
516
|
+
await client.models.createModelFromGguf('my-model', '/models/model.gguf', {
|
|
517
|
+
draftGguf: '/models/draft.gguf',
|
|
518
|
+
draftQuantize: 'q8_0',
|
|
519
|
+
parameters: { draft_num_predict: 8 },
|
|
520
|
+
});
|
|
521
|
+
|
|
522
|
+
// 2. Generate with drafting (also available ad hoc via options at runtime)
|
|
523
|
+
const response = await client.chat({
|
|
524
|
+
model: 'my-model',
|
|
525
|
+
messages: [{ role: 'user', content: 'Describe a lighthouse in a storm.' }],
|
|
526
|
+
options: { draft_num_predict: 8 },
|
|
527
|
+
});
|
|
528
|
+
|
|
529
|
+
// 3. Measure the speedup with the standardized telemetry
|
|
530
|
+
const t = formatTelemetry(response);
|
|
531
|
+
console.log(`${t.tokensPerSecond} tok/s (load ${t.modelLoadMs}ms)`);
|
|
532
|
+
```
|
|
533
|
+
|
|
534
|
+
Already have the draft blobs uploaded? Pass the digest map directly —
|
|
535
|
+
`createModel({ draft_files: { 'draft.gguf': digest } })` — or merge both
|
|
536
|
+
(path uploads win on basename collision). A live measurement harness
|
|
537
|
+
comparing `draft_num_predict` settings lives at
|
|
538
|
+
`lab/15-speculative-draft/measure-draft-speedup.ts`.
|
|
539
|
+
|
|
500
540
|
### Token Log Probabilities (`logprobs`)
|
|
501
541
|
|
|
502
542
|
Set `logprobs: true` (optionally with `top_logprobs`) on `chat()`/`generate()` to get per-token log probabilities back — useful for confidence scoring, speculative decoding, or agent routing decisions.
|
|
@@ -955,9 +995,19 @@ rejects `x-api-key`-only auth; the SDK always sends the bearer header plus
|
|
|
955
995
|
`anthropic-version: 2023-06-01`):
|
|
956
996
|
|
|
957
997
|
```typescript
|
|
958
|
-
const cloud = new OllamaClient({
|
|
959
|
-
|
|
960
|
-
|
|
998
|
+
const cloud = new OllamaClient({
|
|
999
|
+
baseUrl: 'https://ollama.com',
|
|
1000
|
+
apiKey: process.env.OLLAMA_API_KEY,
|
|
1001
|
+
});
|
|
1002
|
+
await cloud.openai.chatCompletions({
|
|
1003
|
+
model: 'gemma4:31b',
|
|
1004
|
+
messages: [{ role: 'user', content: 'Hi' }],
|
|
1005
|
+
});
|
|
1006
|
+
await cloud.anthropic.messages({
|
|
1007
|
+
model: 'gemma4:31b',
|
|
1008
|
+
max_tokens: 1024,
|
|
1009
|
+
messages: [{ role: 'user', content: 'Hi' }],
|
|
1010
|
+
});
|
|
961
1011
|
```
|
|
962
1012
|
|
|
963
1013
|
The cloud API does not support stateful Responses, built-in web search through
|
|
@@ -1454,6 +1504,62 @@ windows: [...] })` directly with your own `windowMs`/`maxTokens`/`maxRequests` p
|
|
|
1454
1504
|
|
|
1455
1505
|
---
|
|
1456
1506
|
|
|
1507
|
+
### Testing with the In-Memory Mock Server (`/testing`)
|
|
1508
|
+
|
|
1509
|
+
The SDK ships its own deterministic test double as a dedicated subpath —
|
|
1510
|
+
`@nemesis-oss/ollama-sdk/testing` — the same engine the SDK's own test suite
|
|
1511
|
+
runs against (ADR 0029: promote, don't fork). No daemon, no network, no GPU:
|
|
1512
|
+
scriptable NDJSON routes on a real `node:http` server, built-in defaults for
|
|
1513
|
+
the official liveness probe and `/api/version`, wire-level request recording,
|
|
1514
|
+
and failure injection that covers what static cassettes can't.
|
|
1515
|
+
|
|
1516
|
+
```typescript
|
|
1517
|
+
import {
|
|
1518
|
+
createMockOllamaServer,
|
|
1519
|
+
mockTextChunks,
|
|
1520
|
+
mockToolCallChunks,
|
|
1521
|
+
} from '@nemesis-oss/ollama-sdk/testing';
|
|
1522
|
+
import { OllamaClient } from '@nemesis-oss/ollama-sdk';
|
|
1523
|
+
|
|
1524
|
+
const server = createMockOllamaServer(); // ephemeral port, defaults on
|
|
1525
|
+
await server.start();
|
|
1526
|
+
const client = new OllamaClient({ baseUrl: server.baseUrl });
|
|
1527
|
+
|
|
1528
|
+
// Defaults answer out of the box: HEAD / (ping) and GET /api/version.
|
|
1529
|
+
// Register everything else — register() always wins over the defaults.
|
|
1530
|
+
server.register('/api/chat', {
|
|
1531
|
+
chunks: mockToolCallChunks([
|
|
1532
|
+
// Parallel tool calls: one frame, complete calls — the native shape.
|
|
1533
|
+
{ name: 'get_weather', arguments: { city: 'Tokyo' } },
|
|
1534
|
+
{ name: 'get_time', arguments: { tz: 'Asia/Tokyo' } },
|
|
1535
|
+
]),
|
|
1536
|
+
chunkDelayMs: 10, // inter-chunk backpressure testing
|
|
1537
|
+
});
|
|
1538
|
+
|
|
1539
|
+
// Mid-stream failure, GPU-OOM style: HTTP stays 200, the done frame is
|
|
1540
|
+
// replaced by an {"error": "..."} frame — the SDK raises OllamaStreamError.
|
|
1541
|
+
server.register('/api/generate', {
|
|
1542
|
+
chunks: mockTextChunks('partial answer', { chunkSize: 8 }),
|
|
1543
|
+
injectErrorFrame: { message: 'CUDA out of memory' }, // after: N to move it
|
|
1544
|
+
});
|
|
1545
|
+
|
|
1546
|
+
// ... run your code against the client ...
|
|
1547
|
+
|
|
1548
|
+
// Wire-level assertions: every request was recorded.
|
|
1549
|
+
expect(server.requests[0]?.body).toMatchObject({ model: 'llama3.2' });
|
|
1550
|
+
server.clearRequests(); // and server.clear() between tests
|
|
1551
|
+
|
|
1552
|
+
await server.stop(); // idempotent — safe in afterEach
|
|
1553
|
+
```
|
|
1554
|
+
|
|
1555
|
+
Per-route controls: `status` (503/502/429 bodies), `chunkDelayMs`,
|
|
1556
|
+
`dropConnectionMidStream` (connection reset mid-response), `onRequest`
|
|
1557
|
+
(per-route body capture), and `injectErrorFrame.after` (error position).
|
|
1558
|
+
Chunk builders: `mockChatChunk` (one frame with counter defaults),
|
|
1559
|
+
`mockTextChunks` (chunked text + done frame), `mockToolCallChunks` (parallel
|
|
1560
|
+
tool calls + done frame). Node.js only — the subpath builds on `node:http`
|
|
1561
|
+
and is never imported from the Edge-safe main entry.
|
|
1562
|
+
|
|
1457
1563
|
### Edge Runtime Compatibility
|
|
1458
1564
|
|
|
1459
1565
|
The core client (`OllamaClient`, `Agent`, `ToolRegistry`, and everything exported from
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { O as OllamaRuntime, H as HttpClient, A as AbortableAsyncIterable, af as SseEvent, y as OllamaClientError } from './runtime-
|
|
1
|
+
import { O as OllamaRuntime, H as HttpClient, A as AbortableAsyncIterable, af as SseEvent, y as OllamaClientError } from './runtime-BIjr5ME4.js';
|
|
2
2
|
|
|
3
3
|
/**
|
|
4
4
|
* AUTO-GENERATED by `npm run contract:generate`. DO NOT EDIT.
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
import { O as OllamaRuntime, H as HttpClient, A as AbortableAsyncIterable, af as SseEvent, y as OllamaClientError } from './runtime-
|
|
1
|
+
import { O as OllamaRuntime, H as HttpClient, A as AbortableAsyncIterable, af as SseEvent, y as OllamaClientError } from './runtime-C4RRHqOs.cjs';
|
|
2
2
|
|
|
3
3
|
/**
|
|
4
4
|
* AUTO-GENERATED by `npm run contract:generate`. DO NOT EDIT.
|
package/dist/generated-api.d.cts
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
import { O as OpenAIChatCompletionRequest, a as OpenAIChatCompletionResponse, b as OpenAIChatCompletionChunk, c as OpenAICompletionRequest, d as OpenAICompletionResponse, e as OpenAICompletionChunk, f as OpenAIEmbeddingRequest, g as OpenAIEmbeddingResponse, h as OpenAIListModelsResponse, i as OpenAIModelItem, j as OpenAIResponsesRequest, k as OpenAIResponsesResponse, l as OpenAIResponsesStreamEvent, A as AnthropicMessagesRequest, m as AnthropicMessagesResponse, n as AnthropicMessageStreamEvent } from './anthropic-
|
|
2
|
-
export { N as NativeApi } from './anthropic-
|
|
3
|
-
import { O as OllamaRuntime } from './runtime-
|
|
4
|
-
export { a as OperationDefinition } from './runtime-
|
|
5
|
-
import './types-
|
|
1
|
+
import { O as OpenAIChatCompletionRequest, a as OpenAIChatCompletionResponse, b as OpenAIChatCompletionChunk, c as OpenAICompletionRequest, d as OpenAICompletionResponse, e as OpenAICompletionChunk, f as OpenAIEmbeddingRequest, g as OpenAIEmbeddingResponse, h as OpenAIListModelsResponse, i as OpenAIModelItem, j as OpenAIResponsesRequest, k as OpenAIResponsesResponse, l as OpenAIResponsesStreamEvent, A as AnthropicMessagesRequest, m as AnthropicMessagesResponse, n as AnthropicMessageStreamEvent } from './anthropic-UeaVjLeP.cjs';
|
|
2
|
+
export { N as NativeApi } from './anthropic-UeaVjLeP.cjs';
|
|
3
|
+
import { O as OllamaRuntime } from './runtime-C4RRHqOs.cjs';
|
|
4
|
+
export { a as OperationDefinition } from './runtime-C4RRHqOs.cjs';
|
|
5
|
+
import './types-DbPnfVh-.cjs';
|
|
6
6
|
import 'zod';
|
|
7
7
|
|
|
8
8
|
/**
|
package/dist/generated-api.d.ts
CHANGED
|
@@ -1,8 +1,8 @@
|
|
|
1
|
-
import { O as OpenAIChatCompletionRequest, a as OpenAIChatCompletionResponse, b as OpenAIChatCompletionChunk, c as OpenAICompletionRequest, d as OpenAICompletionResponse, e as OpenAICompletionChunk, f as OpenAIEmbeddingRequest, g as OpenAIEmbeddingResponse, h as OpenAIListModelsResponse, i as OpenAIModelItem, j as OpenAIResponsesRequest, k as OpenAIResponsesResponse, l as OpenAIResponsesStreamEvent, A as AnthropicMessagesRequest, m as AnthropicMessagesResponse, n as AnthropicMessageStreamEvent } from './anthropic-
|
|
2
|
-
export { N as NativeApi } from './anthropic-
|
|
3
|
-
import { O as OllamaRuntime } from './runtime-
|
|
4
|
-
export { a as OperationDefinition } from './runtime-
|
|
5
|
-
import './types-
|
|
1
|
+
import { O as OpenAIChatCompletionRequest, a as OpenAIChatCompletionResponse, b as OpenAIChatCompletionChunk, c as OpenAICompletionRequest, d as OpenAICompletionResponse, e as OpenAICompletionChunk, f as OpenAIEmbeddingRequest, g as OpenAIEmbeddingResponse, h as OpenAIListModelsResponse, i as OpenAIModelItem, j as OpenAIResponsesRequest, k as OpenAIResponsesResponse, l as OpenAIResponsesStreamEvent, A as AnthropicMessagesRequest, m as AnthropicMessagesResponse, n as AnthropicMessageStreamEvent } from './anthropic-DrDUfbfQ.js';
|
|
2
|
+
export { N as NativeApi } from './anthropic-DrDUfbfQ.js';
|
|
3
|
+
import { O as OllamaRuntime } from './runtime-BIjr5ME4.js';
|
|
4
|
+
export { a as OperationDefinition } from './runtime-BIjr5ME4.js';
|
|
5
|
+
import './types-DbPnfVh-.js';
|
|
6
6
|
import 'zod';
|
|
7
7
|
|
|
8
8
|
/**
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
export { I as InvokeRequest, O as OllamaRuntime, b as OllamaRuntimeOptions, a as OperationDefinition, S as SchemaRef } from './runtime-
|
|
1
|
+
export { I as InvokeRequest, O as OllamaRuntime, b as OllamaRuntimeOptions, a as OperationDefinition, S as SchemaRef } from './runtime-C4RRHqOs.cjs';
|
|
2
2
|
import { z } from 'zod';
|
|
3
|
-
import './types-
|
|
3
|
+
import './types-DbPnfVh-.cjs';
|
|
4
4
|
|
|
5
5
|
/**
|
|
6
6
|
* AUTO-GENERATED by `npm run contract:generate`. DO NOT EDIT.
|
|
@@ -1,6 +1,6 @@
|
|
|
1
|
-
export { I as InvokeRequest, O as OllamaRuntime, b as OllamaRuntimeOptions, a as OperationDefinition, S as SchemaRef } from './runtime-
|
|
1
|
+
export { I as InvokeRequest, O as OllamaRuntime, b as OllamaRuntimeOptions, a as OperationDefinition, S as SchemaRef } from './runtime-BIjr5ME4.js';
|
|
2
2
|
import { z } from 'zod';
|
|
3
|
-
import './types-
|
|
3
|
+
import './types-DbPnfVh-.js';
|
|
4
4
|
|
|
5
5
|
/**
|
|
6
6
|
* AUTO-GENERATED by `npm run contract:generate`. DO NOT EDIT.
|
package/dist/index.cjs
CHANGED
|
@@ -2817,12 +2817,29 @@ var ModelsClient = class {
|
|
|
2817
2817
|
* await client.models.createModelFromGguf('my-model', '/path/to/model.gguf', {
|
|
2818
2818
|
* template: '{{ .Prompt }}',
|
|
2819
2819
|
* });
|
|
2820
|
+
*
|
|
2821
|
+
* // Speculative decoding: primary + draft shards + tuning:
|
|
2822
|
+
* await client.models.createModelFromGguf('my-model', '/path/to/model.gguf', {
|
|
2823
|
+
* draftGguf: '/path/to/draft.gguf',
|
|
2824
|
+
* draftQuantize: 'q8_0',
|
|
2825
|
+
* parameters: { draft_num_predict: 8 },
|
|
2826
|
+
* });
|
|
2820
2827
|
* ```
|
|
2821
2828
|
*
|
|
2829
|
+
* Speculative-decoding models assemble the same way: pass the draft
|
|
2830
|
+
* network's GGUF(s) as `draftGguf` (uploaded into `draft_files`), optionally
|
|
2831
|
+
* set `draft_quantize` to compress the draft weights independently of the
|
|
2832
|
+
* primary model, and set `parameters.draft_num_predict` to tune how many
|
|
2833
|
+
* tokens the draft network predicts per step (Ollama defaults separate
|
|
2834
|
+
* draft models to 4; `0` disables drafting; models with embedded MTP
|
|
2835
|
+
* tensors require setting it explicitly).
|
|
2836
|
+
*
|
|
2822
2837
|
* @param model Name for the new model (e.g. `'my-model:latest'`).
|
|
2823
2838
|
* @param gguf Path(s) to GGUF file(s) on the local filesystem.
|
|
2824
2839
|
* @param opts Everything else `/api/create` accepts (`template`, `system`,
|
|
2825
|
-
* `parameters`, `license`, `quantize`, …) except `files
|
|
2840
|
+
* `parameters`, `license`, `quantize`, `draftGguf`, …) except `files`,
|
|
2841
|
+
* `draft_files` (use `draftGguf` for path-based drafts; an explicit digest
|
|
2842
|
+
* map is still accepted and merged), `from`, and `stream`.
|
|
2826
2843
|
*/
|
|
2827
2844
|
async createModelFromGguf(model, gguf, opts) {
|
|
2828
2845
|
const paths = typeof gguf === "string" ? [gguf] : gguf;
|
|
@@ -2836,14 +2853,29 @@ var ModelsClient = class {
|
|
|
2836
2853
|
const uploaded = await this.createBlobFromFile(path, { signal: opts?.signal });
|
|
2837
2854
|
files[uploaded.fileName] = uploaded.digest;
|
|
2838
2855
|
}
|
|
2856
|
+
const draftOpt = opts?.draftGguf;
|
|
2857
|
+
const draftPaths = typeof draftOpt === "string" ? [draftOpt] : draftOpt ?? [];
|
|
2839
2858
|
const {
|
|
2840
2859
|
model: _model,
|
|
2841
2860
|
files: _files,
|
|
2842
2861
|
from: _from,
|
|
2843
2862
|
stream: _stream,
|
|
2863
|
+
draftGguf: _draftGguf,
|
|
2864
|
+
draft_files: _draftFiles,
|
|
2844
2865
|
...rest
|
|
2845
2866
|
} = opts ?? {};
|
|
2846
|
-
|
|
2867
|
+
const draftFiles = { ..._draftFiles ?? {} };
|
|
2868
|
+
for (const path of draftPaths) {
|
|
2869
|
+
const uploaded = await this.createBlobFromFile(path, { signal: opts?.signal });
|
|
2870
|
+
draftFiles[uploaded.fileName] = uploaded.digest;
|
|
2871
|
+
}
|
|
2872
|
+
return this.create({
|
|
2873
|
+
...rest,
|
|
2874
|
+
model,
|
|
2875
|
+
files,
|
|
2876
|
+
...Object.keys(draftFiles).length > 0 ? { draft_files: draftFiles } : {},
|
|
2877
|
+
stream: false
|
|
2878
|
+
});
|
|
2847
2879
|
}
|
|
2848
2880
|
};
|
|
2849
2881
|
|