@nemesis-oss/ollama-sdk 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +71 -0
- package/LICENSE +21 -0
- package/README.md +378 -0
- package/dist/compose-DC11aDJn.d.cts +430 -0
- package/dist/compose-DC11aDJn.d.ts +430 -0
- package/dist/index.cjs +2049 -0
- package/dist/index.cjs.map +1 -0
- package/dist/index.d.cts +1044 -0
- package/dist/index.d.ts +1044 -0
- package/dist/index.js +1969 -0
- package/dist/index.js.map +1 -0
- package/dist/skills.cjs +185 -0
- package/dist/skills.cjs.map +1 -0
- package/dist/skills.d.cts +19 -0
- package/dist/skills.d.ts +19 -0
- package/dist/skills.js +181 -0
- package/dist/skills.js.map +1 -0
- package/package.json +137 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,71 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to `@nemesis-oss/ollama-sdk` will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [1.0.0] - 2026-08-17
|
|
9
|
+
|
|
10
|
+
First public release. The version was `0.1.0` throughout development but was never
|
|
11
|
+
published to npm, so the entire feature set below ships as the initial `1.0.0`.
|
|
12
|
+
|
|
13
|
+
`1.0.0` marks the public API surface (`OllamaClient`, `Agent`, `ToolRegistry`, the
|
|
14
|
+
streaming adapters, the error hierarchy, and the compatibility bridges) as stable under
|
|
15
|
+
[Semantic Versioning](https://semver.org/spec/v2.0.0.html) — breaking changes to it now
|
|
16
|
+
require a major version bump.
|
|
17
|
+
|
|
18
|
+
### Added
|
|
19
|
+
|
|
20
|
+
- **Core Ollama REST API Client:**
|
|
21
|
+
- Full support for `chat`, `generate`, `embed`, `embeddings`, `ps`, and `version`.
|
|
22
|
+
- Zero-runtime-dependency HTTP transport with native `fetch`, streaming NDJSON parsing, and binary body blob uploads.
|
|
23
|
+
- Multi-endpoint high availability registry with circuit breaker failover, priority routing, and health checks.
|
|
24
|
+
- Configurable exponential backoff with full jitter, retryable error predicates, and timeout signal propagation.
|
|
25
|
+
- **Model Lifecycle & Blob Management:**
|
|
26
|
+
- `createModel`, `pullModel`, `pushModel`, `copyModel`, `deleteModel`, `listModels`, `showModel`.
|
|
27
|
+
- Dedicated blob management endpoints: `createBlob` (`POST /api/blobs/:digest`) and `checkBlob` (`HEAD /api/blobs/:digest`).
|
|
28
|
+
- **Structured Outputs & Schema Validation:**
|
|
29
|
+
- Seamless Zod schema conversion to JSON Schema, supporting both Zod v4 (native `z.toJSONSchema`) and Zod v3 (structural fallback).
|
|
30
|
+
- Structured output parsing with resilient markdown code fence JSON extraction and strict validation errors (`OllamaToolValidationError`).
|
|
31
|
+
- `zod` is a peer dependency (`^3.22.0 || ^4.0.0`) rather than a bundled dependency, so consumers don't end up with a duplicate copy in `node_modules`.
|
|
32
|
+
- **Reasoning & Thinking Tokens:**
|
|
33
|
+
- Native parsing of reasoning traces (`<think>` tags and `message.thinking`) with dual stream events (`thinking` vs `token`).
|
|
34
|
+
- **Agentic Workflow & Tool Calling:**
|
|
35
|
+
- Autonomous multi-turn agent execution loop (`Agent`).
|
|
36
|
+
- Tool definition helper (`defineTool`) with Zod parameter schemas.
|
|
37
|
+
- Tool registry with duplicate detection and execution error recovery.
|
|
38
|
+
- Model Context Protocol (MCP) server integration (`createMcpToolSet`, `registerMcpTools`).
|
|
39
|
+
- Opt-in tool execution sandboxing: per-tool/registry `timeoutMs` (cooperative cancellation via `AbortSignal`, surfaced as `OllamaToolTimeoutError`), `maxConcurrency` bounding parallel tool calls, and `maxOutputChars` truncating oversized tool output before it re-enters the conversation history. See [ADR 0004](./docs/adr/0004-tool-execution-sandboxing.md).
|
|
40
|
+
- Synthetic, client-generated `tool_call_id` correlation: `ToolCall.id`, `Message.tool_call_id`, and `ToolExecutionResult.toolCallId` — Ollama's native protocol has no call ID, so the SDK synthesizes a stable one (`crypto.randomUUID()`, the Web Standard global, not `node:crypto`, to stay Edge-safe) the first time it sees a call without one, and reuses it consistently across the streamed `tool_call` event, the final aggregated message, the tool execution result, and the `role: 'tool'` history entry `Agent` appends. Execution/result correlation by array order still works exactly as before; `id` is an additive convenience. See [ADR 0007](./docs/adr/0007-synthetic-tool-call-ids.md).
|
|
41
|
+
- **Architecture Decision Records:** `docs/adr/` documents the rationale behind the circuit breaker failure model, the dual ESM/CJS packaging strategy, Zod v3/v4 dual support, the tool execution sandboxing model, OpenTelemetry instrumentation, Edge runtime CI verification, synthetic tool-call IDs, endpoint failover scope, and registry parameter variance.
|
|
42
|
+
- **OpenTelemetry Instrumentation:** Automatic spans (`@opentelemetry/api` is an optional peer dependency, a no-op when absent or unconfigured) for HTTP requests, endpoint failover attempts, non-streaming `chat`/`generate` calls (using the Gen AI semantic conventions, including token usage), and `Agent` runs (`invoke_agent` → `ollama.agent.turn` → `execute_tool`). See [ADR 0005](./docs/adr/0005-opentelemetry-instrumentation.md).
|
|
43
|
+
- **Edge Runtime CI Verification:** `npm run verify:edge-runtime` bundles `dist/index.js` for a browser/edge platform with `esbuild` (failing on any `node:*` import, matching Cloudflare Workers/Vercel Edge Runtime's own bundlers) and runs a full `OllamaClient` + `Agent` + tool-calling round trip inside `@edge-runtime/vm`'s sandboxed Edge Runtime — a real V8 context exposing only Web Standard globals. Wired into CI as its own job and into `verify`/`prepublishOnly`. See [ADR 0006](./docs/adr/0006-edge-runtime-ci-and-benchmarks.md).
|
|
44
|
+
- **Benchmarks:** `npm run bench` (via `vitest bench`, no new dependency) covers NDJSON stream parsing, Zod schema conversion/structured output parsing, `ToolRegistry` dispatch overhead, and `OllamaClient.chat`'s end-to-end request pipeline overhead. Wired into CI as its own job.
|
|
45
|
+
- **Protocol Compatibility Bridges:**
|
|
46
|
+
- OpenAI compatibility bridge (`/v1/chat/completions`, `/v1/models`), including `stream_options.include_usage` and `tools` (function calling) typing — both confirmed supported by Ollama's OpenAI-compat layer; `tool_choice`/`parallel_tool_calls` are deliberately not typed, since Ollama documents `tool_choice` as explicitly unsupported. `@remarks` JSDoc on `OpenAICompatClient` scopes it as a documented subset of the OpenAI API, not the full Responses API surface.
|
|
47
|
+
- Anthropic compatibility bridge (`/v1/messages`), including `cache_control` typing on content blocks for prompt caching. `@remarks` JSDoc on `AnthropicCompatClient` scopes it as a subset of the Messages API (no tool use, extended thinking, citations, files, or Batches API).
|
|
48
|
+
- **Capability Detection Fixes:** `ModelCapabilities.supportsStructuredOutputRequest` is no longer hardcoded `true` — it's inferred `false` for endpoints classified as `cloud` by `inferRuntimeMode` (Ollama Cloud does not currently support structured outputs) and documented as a best-effort heuristic, not a guarantee, since Ollama doesn't expose this as a queryable capability. Added `supportsThinking`, derived from the model's reported `thinking` capability.
|
|
49
|
+
- **Fail-Fast `OllamaUnsupportedCapabilityError`:** `chat`/`chatStream`/`chatWithSchema` and `generate`/`generateStream`/`generateWithSchema` now throw this error _before making any network call_ when `format` is set against an endpoint inferred as Ollama Cloud, instead of sending a request Ollama Cloud is known to reject. `unsupported_capability` is included in `DEFAULT_FAILOVER_CODES`, so a multi-endpoint setup tries the next candidate (e.g. a local fallback) before the error ever reaches the caller — verified with a test asserting the rejected cloud endpoint's URL is never actually fetched.
|
|
50
|
+
- **`dimensions` on `embed()`:** `EmbedRequestOptions.dimensions` lets callers request truncated embedding vectors, matching Ollama's `/api/embed` parameter.
|
|
51
|
+
- **Environment Variable Fallbacks:** `OllamaClient`'s default single-endpoint `baseUrl`/`apiKey` fall back to `OLLAMA_HOST`/`OLLAMA_API_KEY` (the same variables the official `ollama` CLI and client libraries read) when not passed explicitly, matching the convention of other major LLM SDKs. Guarded to remain a no-op (not a `ReferenceError`) on Edge runtimes where `process` doesn't exist. Explicit `config.baseUrl`/`config.apiKey`, and any use of `config.endpoints`, always take precedence.
|
|
52
|
+
- **Web Standard Stream Adapters:**
|
|
53
|
+
- `toTextStream`, `toDataStream`, and `toResponse` for direct integration with Next.js, Vercel AI SDK, and Web standard streams.
|
|
54
|
+
- **Skills System:**
|
|
55
|
+
- Frontmatter parser for `SKILL.md` documents.
|
|
56
|
+
- Skill composition and prompt injection into system messages (`applySkill`).
|
|
57
|
+
- **Documentation Hygiene:** `OllamaClient.embeddings()` is now marked `@deprecated` (Ollama's `/api/embeddings` was superseded by `/api/embed`, exposed as `embed()`). `ToolCall`, `Agent`, and `ToolRegistry.executeToolCalls` document that Ollama's native tool-calling protocol has no wire-level call ID, dispatch/results are still ordered/concurrency-bounded rather than ID-driven, and that `id`/`tool_call_id` (see above) is a client-synthesized convenience layered on top, not a protocol guarantee.
|
|
58
|
+
- **Testing & Quality Assurance:**
|
|
59
|
+
- New cancellation tests prove `AbortSignal` genuinely aborts the underlying `fetch` call (not just racing a promise) and propagates from `Agent.run` into `ToolExecutionContext.signal`, including the early-abort race where the signal is already aborted before the request starts.
|
|
60
|
+
- 4-tier test architecture: Unit, Integration, Functional, and Behavioral testing (50 tests).
|
|
61
|
+
- VCR record and replay harness with real cassettes generated against `qwen3.5:2b` and `nomic-embed-text:latest`.
|
|
62
|
+
- Multi-node CI/CD workflow (Node 18, 20, 22) and automated npm release with provenance.
|
|
63
|
+
|
|
64
|
+
### Fixed
|
|
65
|
+
|
|
66
|
+
- **`ToolRegistry` rejected every concretely-typed tool.** Its constructor, `register`, and `registerMany` accepted `Tool<never, unknown>`; because `Tool` is invariant in `TParams` (it appears in both `schema: z.ZodType<TParams>` and `execute`), _nothing_ was assignable to it — so `new ToolRegistry([myTool])`, `new ToolRegistry({ tools: [myTool] })`, and `.register(myTool)` all failed to compile for consumers, including the exact example in this README. Replaced with a new exported `AnyTool` alias. Tool authoring keeps full type safety (`defineTool` still infers `TParams` from the Zod schema and type-checks `execute` against it); only the registry's storage type is widened. Found by installing the packed tarball into a throwaway TypeScript project during pre-publish verification. See [ADR 0009](./docs/adr/0009-anytool-registry-variance.md).
|
|
67
|
+
- **Tests are now type-checked.** `tsconfig.json` scoped checking to `src/`, and Vitest's esbuild transform strips types without verifying them — so all 20 test files were unchecked, and had masked the bug above with `as never` casts. `npm run typecheck` now uses `tsconfig.typecheck.json`, covering `src/`, `test/`, `bench/`, and `scripts/`. The casts (and a matching `as unknown as z.ZodType<never>` in `mcp-tools.ts`) are gone.
|
|
68
|
+
|
|
69
|
+
### Changed
|
|
70
|
+
|
|
71
|
+
- **Endpoint Failover Scope:** `ModelsClient` operations (`list`, `show`, `pull`, `push`, `create`, `delete`, `copy`, `ps`, `version`, `createBlob`, `checkBlob`) and `OllamaClient.capabilities()` no longer participate in cross-endpoint failover — each now targets only the single best candidate endpoint (same-endpoint retry via backoff still applies). These operations act on a specific server's local model catalog/blob store, which isn't an interchangeable resource across endpoints the way `chat`/`generate`/`embed` inference is; failing `deleteModel`/`listModels`/etc. over to a different endpoint was silently operating on the wrong server's state, not retrying "the same" request. `chat`/`generate`/`embed`/`embeddings`/`webSearch`/`webFetch` failover behavior is unchanged. See [ADR 0008](./docs/adr/0008-endpoint-failover-scope.md).
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Shubham Taywade
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,378 @@
|
|
|
1
|
+
# @nemesis-oss/ollama-sdk
|
|
2
|
+
|
|
3
|
+
[](https://github.com/shubhamtaywade82/ollama-client-ts/actions/workflows/ci.yml)
|
|
4
|
+
[](https://www.npmjs.com/package/@nemesis-oss/ollama-sdk)
|
|
5
|
+
[](https://opensource.org/licenses/MIT)
|
|
6
|
+
|
|
7
|
+
> Production-grade TypeScript SDK for Ollama. Built with native fetch, high availability failover, multi-turn tool calling, structured outputs with Zod, reasoning stream tokens, OpenAI & Anthropic compatibility bridges, MCP integration, and Web Stream adapters.
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Key Features
|
|
12
|
+
|
|
13
|
+
- 🚀 **Native Web Standards**: Built on native `fetch` and Web Streams. Zero external HTTP dependencies.
|
|
14
|
+
- 🧠 **Reasoning & Thinking Tokens**: First-class support for reasoning models (`qwen3.5:2b`, `deepseek-r1`) with discrete `thinking` and `token` streaming events.
|
|
15
|
+
- 🎯 **Zod-Powered Structured Outputs**: Strictly typed schema enforcement via `chatWithSchema` and `generateWithSchema` with resilient markdown JSON parsing.
|
|
16
|
+
- 🛠️ **Autonomous Agent & Tool Calling**: Multi-turn agent loop (`Agent`) with automated tool execution, parameter validation, and self-correcting error recovery.
|
|
17
|
+
- 🌐 **High Availability & Failover**: Multi-endpoint registry with priority routing, circuit breaker failover, and active health checks.
|
|
18
|
+
- 🔌 **Model Context Protocol (MCP)**: Native adapters to convert MCP tools into Ollama-compatible function schemas.
|
|
19
|
+
- 🌉 **OpenAI & Anthropic Compatibility Bridges**: Built-in clients for `/v1/chat/completions`, `/v1/models`, and `/v1/messages`.
|
|
20
|
+
- 🌊 **Web Stream Adapters**: Drop-in adapters (`toTextStream`, `toDataStream`, `toResponse`) for Next.js Route Handlers and Vercel AI SDK.
|
|
21
|
+
- 📈 **OpenTelemetry Instrumentation**: Automatic spans for HTTP requests, endpoint failover, chat/generate calls, and agent runs — zero-cost when OpenTelemetry isn't installed.
|
|
22
|
+
- ⚡ **Edge Runtime Verified**: CI bundles and runs the client in a real Edge Runtime sandbox (Cloudflare Workers/Vercel Edge-compatible) with zero Node.js APIs.
|
|
23
|
+
- 📦 **Dual ESM & CJS Build**: Full module support with clean TypeScript `.d.ts` declaration maps.
|
|
24
|
+
|
|
25
|
+
---
|
|
26
|
+
|
|
27
|
+
## Installation
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
npm install @nemesis-oss/ollama-sdk zod
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
`zod` is a peer dependency (`^3.22.0 || ^4.0.0`) — install whichever major version your project already uses instead of getting a second copy bundled in.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Quick Start
|
|
38
|
+
|
|
39
|
+
### Basic Chat & Completion
|
|
40
|
+
|
|
41
|
+
```typescript
|
|
42
|
+
import { OllamaClient } from '@nemesis-oss/ollama-sdk';
|
|
43
|
+
|
|
44
|
+
const client = new OllamaClient();
|
|
45
|
+
|
|
46
|
+
// Text helper
|
|
47
|
+
const answer = await client.chatText({
|
|
48
|
+
model: 'qwen3.5:2b',
|
|
49
|
+
messages: [{ role: 'user', content: 'Explain quantum computing in one sentence.' }],
|
|
50
|
+
});
|
|
51
|
+
console.log(answer);
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
### Thinking & Reasoning Token Streams
|
|
55
|
+
|
|
56
|
+
```typescript
|
|
57
|
+
const stream = await client.chatStream({
|
|
58
|
+
model: 'qwen3.5:2b',
|
|
59
|
+
messages: [{ role: 'user', content: 'What is 18 * 4?' }],
|
|
60
|
+
options: { temperature: 0 },
|
|
61
|
+
});
|
|
62
|
+
|
|
63
|
+
for await (const event of stream) {
|
|
64
|
+
if (event.type === 'thinking') {
|
|
65
|
+
process.stdout.write(`\x1b[33m${event.data.delta}\x1b[0m`); // Thinking trace
|
|
66
|
+
} else if (event.type === 'token') {
|
|
67
|
+
process.stdout.write(event.data.delta); // Final answer token
|
|
68
|
+
}
|
|
69
|
+
}
|
|
70
|
+
|
|
71
|
+
const final = await stream.finalResult;
|
|
72
|
+
console.log(`\nEval tokens/sec: ${final.usage?.tokensPerSecond}`);
|
|
73
|
+
```
|
|
74
|
+
|
|
75
|
+
### Structured Outputs with Zod
|
|
76
|
+
|
|
77
|
+
```typescript
|
|
78
|
+
import { z } from 'zod';
|
|
79
|
+
|
|
80
|
+
const ProductSchema = z.object({
|
|
81
|
+
name: z.string(),
|
|
82
|
+
category: z.enum(['electronics', 'books', 'apparel']),
|
|
83
|
+
price: z.number(),
|
|
84
|
+
tags: z.array(z.string()),
|
|
85
|
+
});
|
|
86
|
+
|
|
87
|
+
const product = await client.chatWithSchema(
|
|
88
|
+
{
|
|
89
|
+
model: 'qwen3.5:2b',
|
|
90
|
+
messages: [{ role: 'user', content: 'Generate a gaming keyboard item.' }],
|
|
91
|
+
},
|
|
92
|
+
ProductSchema,
|
|
93
|
+
);
|
|
94
|
+
|
|
95
|
+
console.log(product.name, product.price);
|
|
96
|
+
```
|
|
97
|
+
|
|
98
|
+
### Vector Embeddings & Similarity
|
|
99
|
+
|
|
100
|
+
```typescript
|
|
101
|
+
const res = await client.embed({
|
|
102
|
+
model: 'nomic-embed-text:latest',
|
|
103
|
+
input: [
|
|
104
|
+
'Machine learning and neural networks',
|
|
105
|
+
'Artificial intelligence algorithms',
|
|
106
|
+
'Baking traditional French sourdough bread',
|
|
107
|
+
],
|
|
108
|
+
});
|
|
109
|
+
|
|
110
|
+
console.log(
|
|
111
|
+
`Generated ${res.embeddings.length} vectors with dimension ${res.embeddings[0].length}`,
|
|
112
|
+
);
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
### Autonomous Agent & Tool Calling
|
|
116
|
+
|
|
117
|
+
```typescript
|
|
118
|
+
import { Agent, defineTool, ToolRegistry, OllamaClient } from '@nemesis-oss/ollama-sdk';
|
|
119
|
+
import { z } from 'zod';
|
|
120
|
+
|
|
121
|
+
const client = new OllamaClient();
|
|
122
|
+
|
|
123
|
+
const weatherTool = defineTool({
|
|
124
|
+
name: 'get_weather',
|
|
125
|
+
description: 'Get the current weather for a city',
|
|
126
|
+
schema: z.object({ city: z.string() }),
|
|
127
|
+
execute: async ({ city }) => ({ city, temperature: '22°C', condition: 'Sunny' }),
|
|
128
|
+
});
|
|
129
|
+
|
|
130
|
+
const registry = new ToolRegistry([weatherTool]);
|
|
131
|
+
const agent = new Agent(client, { tools: registry, maxIterations: 5 });
|
|
132
|
+
|
|
133
|
+
const response = await agent.run({
|
|
134
|
+
model: 'qwen3.5:2b',
|
|
135
|
+
messages: [{ role: 'user', content: 'What is the weather in Tokyo?' }],
|
|
136
|
+
});
|
|
137
|
+
|
|
138
|
+
console.log(response.finalMessage.content);
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
Ollama's native tool-calling protocol has no OpenAI-style call ID, but the SDK
|
|
142
|
+
synthesizes a stable one so results are still correlatable without tracking array
|
|
143
|
+
position by hand: `response.turns[0].toolCalls[0].id` matches
|
|
144
|
+
`response.turns[0].toolResults[0].toolCallId`, and the `role: 'tool'` message `Agent`
|
|
145
|
+
appends to history carries the same value as `tool_call_id`. See
|
|
146
|
+
[ADR 0007](./docs/adr/0007-synthetic-tool-call-ids.md).
|
|
147
|
+
|
|
148
|
+
### Tool Execution Safety & Sandboxing
|
|
149
|
+
|
|
150
|
+
Tool arguments and, indirectly, which tools get called at all are driven by model
|
|
151
|
+
output — treat them as untrusted input. `ToolRegistry` supports three defensive
|
|
152
|
+
controls, all opt-in (disabled by default, matching prior behavior) so existing agents
|
|
153
|
+
aren't affected until you turn them on:
|
|
154
|
+
|
|
155
|
+
```typescript
|
|
156
|
+
const registry = new ToolRegistry({
|
|
157
|
+
tools: [weatherTool],
|
|
158
|
+
// Fail a call that runs longer than this instead of stalling the agent loop forever.
|
|
159
|
+
// Override per-tool via `defineTool({ ..., timeoutMs: 2_000 })`.
|
|
160
|
+
timeoutMs: 10_000,
|
|
161
|
+
// Cap how many tool calls run in parallel when the model requests several at once.
|
|
162
|
+
maxConcurrency: 4,
|
|
163
|
+
// Truncate oversized tool output before it re-enters the conversation history.
|
|
164
|
+
maxOutputChars: 20_000,
|
|
165
|
+
});
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
- **`timeoutMs`** races the tool call against a timer and rejects with
|
|
169
|
+
`OllamaToolTimeoutError` on expiry. Enforcement is cooperative: it stops the _agent_
|
|
170
|
+
from waiting indefinitely, but genuinely halting a tool's in-flight work still
|
|
171
|
+
requires the tool itself to check `ToolExecutionContext.signal` (which the registry
|
|
172
|
+
aborts on timeout) — plain synchronous or non-abort-aware async code cannot be
|
|
173
|
+
force-killed from the same thread. See [ADR 0004](./docs/adr/0004-tool-execution-sandboxing.md)
|
|
174
|
+
for the full rationale and what a stronger guarantee would require.
|
|
175
|
+
- **`maxConcurrency`** bounds parallel execution instead of the previous unconditional
|
|
176
|
+
`Promise.all`, so a model requesting dozens of simultaneous tool calls can't exhaust
|
|
177
|
+
connection pools, rate limits, or memory all at once.
|
|
178
|
+
- **`maxOutputChars`** truncates `outputString` (what gets fed back into the
|
|
179
|
+
conversation) while leaving the untruncated value on `result.result` for callers who
|
|
180
|
+
need it — bounding how much a single tool call can inflate context size or memory.
|
|
181
|
+
- Zod's `safeParse` already validates every tool call's arguments against its schema
|
|
182
|
+
before `execute` runs (`OllamaToolValidationError` on mismatch). By default, Zod
|
|
183
|
+
objects silently strip unrecognized keys rather than rejecting them; call `.strict()`
|
|
184
|
+
on a tool's schema if you need to reject unexpected extra arguments outright.
|
|
185
|
+
|
|
186
|
+
### Web Standard Streams & Next.js Integration
|
|
187
|
+
|
|
188
|
+
```typescript
|
|
189
|
+
import { toResponse } from '@nemesis-oss/ollama-sdk';
|
|
190
|
+
|
|
191
|
+
export async function POST(req: Request) {
|
|
192
|
+
const { messages } = await req.json();
|
|
193
|
+
const stream = await client.chatStream({
|
|
194
|
+
model: 'qwen3.5:2b',
|
|
195
|
+
messages,
|
|
196
|
+
});
|
|
197
|
+
|
|
198
|
+
return toResponse(stream);
|
|
199
|
+
}
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
### OpenAI & Anthropic Compatibility Bridges
|
|
203
|
+
|
|
204
|
+
```typescript
|
|
205
|
+
// OpenAI compatibility endpoint (/v1/chat/completions)
|
|
206
|
+
const openAIRes = await client.openai.chatCompletions({
|
|
207
|
+
model: 'llama3.2',
|
|
208
|
+
messages: [{ role: 'user', content: 'Hello via OpenAI bridge' }],
|
|
209
|
+
});
|
|
210
|
+
|
|
211
|
+
// Anthropic compatibility endpoint (/v1/messages)
|
|
212
|
+
const anthropicRes = await client.anthropic.messages({
|
|
213
|
+
model: 'llama3.2',
|
|
214
|
+
messages: [{ role: 'user', content: 'Hello via Anthropic bridge' }],
|
|
215
|
+
});
|
|
216
|
+
```
|
|
217
|
+
|
|
218
|
+
### Multi-Endpoint High Availability Failover
|
|
219
|
+
|
|
220
|
+
```typescript
|
|
221
|
+
const client = new OllamaClient({
|
|
222
|
+
endpoints: [
|
|
223
|
+
{ name: 'local-gpu', baseUrl: 'http://localhost:11434', priority: 10 },
|
|
224
|
+
{
|
|
225
|
+
name: 'cloud-replica',
|
|
226
|
+
baseUrl: 'https://ollama.internal.net',
|
|
227
|
+
apiKey: 'secret',
|
|
228
|
+
priority: 5,
|
|
229
|
+
},
|
|
230
|
+
],
|
|
231
|
+
timeoutMs: 30_000,
|
|
232
|
+
retries: 3,
|
|
233
|
+
});
|
|
234
|
+
|
|
235
|
+
// Active health check probe
|
|
236
|
+
const health = await client.healthCheck();
|
|
237
|
+
console.log(health);
|
|
238
|
+
```
|
|
239
|
+
|
|
240
|
+
Failover applies to inference calls (`chat`, `generate`, `embed`, `embeddings`,
|
|
241
|
+
`webSearch`, `webFetch`) — a different endpoint serving the same model is a genuine
|
|
242
|
+
substitute for those. Model/blob management (`listModels`, `pullModel`, `deleteModel`,
|
|
243
|
+
etc.) and `capabilities()` target one specific endpoint's local state and deliberately do
|
|
244
|
+
**not** fail over to a different candidate: retrying `deleteModel` against a different
|
|
245
|
+
server doesn't retry the same operation, it silently acts on a different model catalog.
|
|
246
|
+
See [ADR 0008](./docs/adr/0008-endpoint-failover-scope.md).
|
|
247
|
+
|
|
248
|
+
---
|
|
249
|
+
|
|
250
|
+
### Observability with OpenTelemetry
|
|
251
|
+
|
|
252
|
+
The client automatically emits [OpenTelemetry](https://opentelemetry.io/) spans for HTTP
|
|
253
|
+
requests, endpoint failover attempts, `chat`/`generate` calls (using the
|
|
254
|
+
[Gen AI semantic conventions](https://opentelemetry.io/docs/specs/semconv/gen-ai/)), and
|
|
255
|
+
`Agent` runs (`invoke_agent` → `ollama.agent.turn` → `execute_tool`) — no client
|
|
256
|
+
configuration required. `@opentelemetry/api` is an **optional peer dependency**: if it
|
|
257
|
+
isn't installed, or if your process hasn't registered a `TracerProvider`, tracing is a
|
|
258
|
+
no-op and costs nothing beyond a single cached import attempt.
|
|
259
|
+
|
|
260
|
+
```bash
|
|
261
|
+
npm install @opentelemetry/api @opentelemetry/sdk-node @opentelemetry/auto-instrumentations-node
|
|
262
|
+
```
|
|
263
|
+
|
|
264
|
+
```typescript
|
|
265
|
+
// instrumentation.ts — run before importing the rest of your app
|
|
266
|
+
import { NodeSDK } from '@opentelemetry/sdk-node';
|
|
267
|
+
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
|
|
268
|
+
|
|
269
|
+
const sdk = new NodeSDK({
|
|
270
|
+
instrumentations: [getNodeAutoInstrumentations()],
|
|
271
|
+
});
|
|
272
|
+
sdk.start();
|
|
273
|
+
```
|
|
274
|
+
|
|
275
|
+
Once a `TracerProvider` is registered, every `OllamaClient`/`Agent` call in your process
|
|
276
|
+
produces spans automatically. See [ADR 0005](./docs/adr/0005-opentelemetry-instrumentation.md)
|
|
277
|
+
for exactly which spans and attributes are emitted, and the tradeoffs behind that design.
|
|
278
|
+
|
|
279
|
+
---
|
|
280
|
+
|
|
281
|
+
### Edge Runtime Compatibility
|
|
282
|
+
|
|
283
|
+
The core client (`OllamaClient`, `Agent`, `ToolRegistry`, and everything exported from
|
|
284
|
+
the package root) is built entirely on native `fetch` and Web Streams, so it runs
|
|
285
|
+
unmodified on Cloudflare Workers, Vercel Edge Runtime, and Next.js Edge middleware/route
|
|
286
|
+
handlers — no Node.js APIs required. The only Node-specific code (`SkillRegistry`, which
|
|
287
|
+
reads `SKILL.md` files from disk) lives behind the separate `@nemesis-oss/ollama-sdk/skills`
|
|
288
|
+
subpath export and is never pulled into the main bundle.
|
|
289
|
+
|
|
290
|
+
This is enforced in CI, not just asserted: `npm run verify:edge-runtime` bundles
|
|
291
|
+
`dist/index.js` with `esbuild` targeting a browser/edge platform (which hard-fails on any
|
|
292
|
+
`node:*` import, the same way Cloudflare's and Vercel's own bundlers do) and then runs a
|
|
293
|
+
full `OllamaClient` + `Agent` + tool-calling round trip inside `@edge-runtime/vm` — a
|
|
294
|
+
real Edge Runtime sandbox exposing only Web Standard globals. See
|
|
295
|
+
[ADR 0006](./docs/adr/0006-edge-runtime-ci-and-benchmarks.md) for the full rationale.
|
|
296
|
+
|
|
297
|
+
---
|
|
298
|
+
|
|
299
|
+
## Error Handling
|
|
300
|
+
|
|
301
|
+
Every failure thrown by the client is an `OllamaClientError` subclass, so you can catch the base
|
|
302
|
+
class or narrow to a specific `code`:
|
|
303
|
+
|
|
304
|
+
| Class | `code` | `retryable` | Thrown when |
|
|
305
|
+
| ---------------------------------- | ------------------------------- | ----------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
|
306
|
+
| `OllamaNetworkError` | `network_error` | `true` | The request failed before a response was received (DNS, connection refused, etc). |
|
|
307
|
+
| `OllamaTimeoutError` | `timeout` | `true` | The request exceeded `timeoutMs`. |
|
|
308
|
+
| `OllamaAuthError` | `auth_error` | `false` | The endpoint returned `401`/`403`. |
|
|
309
|
+
| `OllamaNotFoundError` | `not_found` | `false` | The endpoint returned `404` (e.g. unknown model). |
|
|
310
|
+
| `OllamaRateLimitError` | `rate_limited` | `true` | The endpoint returned `429`. |
|
|
311
|
+
| `OllamaServerError` | `server_error` | `true` | The endpoint returned `5xx`. |
|
|
312
|
+
| `OllamaAbortError` | `aborted` | `false` | The request was cancelled via `AbortSignal`. |
|
|
313
|
+
| `OllamaToolValidationError` | `tool_validation_error` | `false` | A tool call's arguments, or a `chatWithSchema`/`generateWithSchema` result, failed Zod validation. |
|
|
314
|
+
| `OllamaUnsupportedCapabilityError` | `unsupported_capability` | `false` | A `format` (structured output) request was made against an endpoint inferred as Ollama Cloud, which doesn't currently support it. Thrown before any network call; in `DEFAULT_FAILOVER_CODES`, so a multi-endpoint setup tries the next candidate first. |
|
|
315
|
+
| `OllamaAgentMaxIterationsError` | `agent_max_iterations_exceeded` | `false` | An `Agent` run exceeded `maxTurns` without producing a final answer. |
|
|
316
|
+
| `OllamaMcpError` | `mcp_error` | varies | An MCP `listTools`/`callTool` call failed. |
|
|
317
|
+
| `OllamaSkillNotFoundError` | `skill_not_found` | `false` | `applySkill` referenced a skill that isn't registered. |
|
|
318
|
+
| `OllamaSkillInvalidError` | `skill_invalid` | `false` | A skill's frontmatter or contents failed to parse. |
|
|
319
|
+
| `OllamaGenericClientError` | `client_error` | `false` | Any other non-2xx response not covered above. |
|
|
320
|
+
|
|
321
|
+
All subclasses carry `status`, `retryable`, and optional `request`/`response` context, and preserve
|
|
322
|
+
the original error via the standard `cause` property:
|
|
323
|
+
|
|
324
|
+
```typescript
|
|
325
|
+
import { OllamaClientError, OllamaRateLimitError } from '@nemesis-oss/ollama-sdk';
|
|
326
|
+
|
|
327
|
+
try {
|
|
328
|
+
await client.chatText({ model: 'qwen3.5:2b', messages: [...] });
|
|
329
|
+
} catch (err) {
|
|
330
|
+
if (err instanceof OllamaRateLimitError) {
|
|
331
|
+
console.warn(`Rate limited, retry after ${err.retryAfterMs}ms`);
|
|
332
|
+
} else if (err instanceof OllamaClientError) {
|
|
333
|
+
console.error(`[${err.code}] ${err.message}`, { retryable: err.retryable, cause: err.cause });
|
|
334
|
+
} else {
|
|
335
|
+
throw err;
|
|
336
|
+
}
|
|
337
|
+
}
|
|
338
|
+
```
|
|
339
|
+
|
|
340
|
+
Multi-endpoint failover (`endpoints: [...]`) fails open rather than throwing a dedicated
|
|
341
|
+
"circuit open" error: once an endpoint's failure count crosses `failureThreshold`, it's skipped in
|
|
342
|
+
favor of healthy endpoints for `cooldownMs`, and only used again — sorted soonest-to-recover — if
|
|
343
|
+
every endpoint is cooling down. Call `client.healthCheck()` or inspect the registry's `status()` to
|
|
344
|
+
observe per-endpoint circuit state directly.
|
|
345
|
+
|
|
346
|
+
---
|
|
347
|
+
|
|
348
|
+
## Testing
|
|
349
|
+
|
|
350
|
+
The test suite contains 105 automated tests across 4 testing tiers:
|
|
351
|
+
|
|
352
|
+
```bash
|
|
353
|
+
# Run unit, integration, and functional test suite
|
|
354
|
+
npm test
|
|
355
|
+
|
|
356
|
+
# Run typechecker
|
|
357
|
+
npm run typecheck
|
|
358
|
+
|
|
359
|
+
# Run linter
|
|
360
|
+
npm run lint
|
|
361
|
+
|
|
362
|
+
# Verify the built package runs correctly in a real Edge Runtime sandbox with zero
|
|
363
|
+
# Node.js APIs (see "Edge Runtime Compatibility" above) — requires `npm run build` first
|
|
364
|
+
npm run verify:edge-runtime
|
|
365
|
+
|
|
366
|
+
# Run the benchmark suite (NDJSON streaming, schema conversion, tool dispatch, the
|
|
367
|
+
# request pipeline)
|
|
368
|
+
npm run bench
|
|
369
|
+
|
|
370
|
+
# Run full CI verification pipeline (typecheck, lint, test, build, edge runtime check)
|
|
371
|
+
npm run verify
|
|
372
|
+
```
|
|
373
|
+
|
|
374
|
+
---
|
|
375
|
+
|
|
376
|
+
## License
|
|
377
|
+
|
|
378
|
+
MIT © [Shubham Taywade](https://github.com/shubhamtaywade82)
|