@molecule/api-ai-local 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE ADDED
@@ -0,0 +1,115 @@
1
+ Apache License
2
+ Version 2.0, January 2004
3
+ http://www.apache.org/licenses/
4
+
5
+ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
6
+
7
+ 1. Definitions.
8
+
9
+ "License" shall mean the terms and conditions for use, reproduction,
10
+ and distribution as defined by Sections 1 through 9 of this document.
11
+
12
+ "Licensor" shall mean the copyright owner or entity authorized by
13
+ the copyright owner that is granting the License.
14
+
15
+ "Legal Entity" shall mean the union of the acting entity and all
16
+ other entities that control, are controlled by, or are under common
17
+ control with that entity. For the purposes of this definition,
18
+ "control" means (i) the power, direct or indirect, to cause the
19
+ direction or management of such entity, whether by contract or
20
+ otherwise, or (ii) ownership of fifty percent (50%) or more of the
21
+ outstanding shares, or (iii) beneficial ownership of such entity.
22
+
23
+ "You" (or "Your") shall mean an individual or Legal Entity
24
+ exercising permissions granted by this License.
25
+
26
+ "Source" form shall mean the preferred form for making modifications,
27
+ including but not limited to software source code, documentation
28
+ source, and configuration files.
29
+
30
+ "Object" form shall mean any form resulting from mechanical
31
+ transformation or translation of a Source form, including but
32
+ not limited to compiled object code, generated documentation,
33
+ and conversions to other media types.
34
+
35
+ "Work" shall mean the work of authorship, whether in Source or
36
+ Object form, made available under the License, as indicated by a
37
+ copyright notice that is included in or attached to the work.
38
+
39
+ "Derivative Works" shall mean any work, whether in Source or Object
40
+ form, that is based on (or derived from) the Work and for which the
41
+ editorial revisions, annotations, elaborations, or other modifications
42
+ represent, as a whole, an original work of authorship.
43
+
44
+ "Contribution" shall mean any work of authorship, including the
45
+ original version of the Work and any modifications or additions
46
+ to that Work, that is intentionally submitted to the Licensor for
47
+ inclusion in the Work by the copyright owner or by an individual or
48
+ Legal Entity authorized to submit on behalf of the copyright owner.
49
+
50
+ "Contributor" shall mean Licensor and any individual or Legal Entity
51
+ on behalf of whom a Contribution has been received by the Licensor and
52
+ subsequently incorporated within the Work.
53
+
54
+ 2. Grant of Copyright License. Subject to the terms and conditions of
55
+ this License, each Contributor hereby grants to You a perpetual,
56
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
57
+ copyright license to reproduce, prepare Derivative Works of,
58
+ publicly display, publicly perform, sublicense, and distribute the
59
+ Work and such Derivative Works in Source or Object form.
60
+
61
+ 3. Grant of Patent License. Subject to the terms and conditions of
62
+ this License, each Contributor hereby grants to You a perpetual,
63
+ worldwide, non-exclusive, no-charge, royalty-free, irrevocable
64
+ patent license to make, have made, use, offer to sell, sell, import,
65
+ and otherwise transfer the Work.
66
+
67
+ 4. Redistribution. You may reproduce and distribute copies of the
68
+ Work or Derivative Works thereof in any medium, with or without
69
+ modifications, and in Source or Object form, provided that You
70
+ meet the following conditions:
71
+
72
+ (a) You must give any other recipients of the Work or
73
+ Derivative Works a copy of this License; and
74
+
75
+ (b) You must cause any modified files to carry prominent notices
76
+ stating that You changed the files; and
77
+
78
+ (c) You must retain, in the Source form of any Derivative Works
79
+ that You distribute, all copyright, patent, trademark, and
80
+ attribution notices from the Source form of the Work,
81
+ excluding those notices that do not pertain to any part of
82
+ the Derivative Works; and
83
+
84
+ (d) If the Work includes a "NOTICE" text file as part of its
85
+ distribution, then any Derivative Works that You distribute must
86
+ include a readable copy of the attribution notices contained
87
+ within such NOTICE file.
88
+
89
+ 5. Submission of Contributions.
90
+
91
+ 6. Trademarks. This License does not grant permission to use the trade
92
+ names, trademarks, service marks, or product names of the Licensor.
93
+
94
+ 7. Disclaimer of Warranty. Unless required by applicable law or
95
+ agreed to in writing, Licensor provides the Work on an "AS IS" BASIS,
96
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND.
97
+
98
+ 8. Limitation of Liability. In no event and under no legal theory shall
99
+ any Contributor be liable to You for damages.
100
+
101
+ 9. Accepting Warranty or Additional Liability.
102
+
103
+ Copyright 2026 Molecule Dev, Inc.
104
+
105
+ Licensed under the Apache License, Version 2.0 (the "License");
106
+ you may not use this file except in compliance with the License.
107
+ You may obtain a copy of the License at
108
+
109
+ http://www.apache.org/licenses/LICENSE-2.0
110
+
111
+ Unless required by applicable law or agreed to in writing, software
112
+ distributed under the License is distributed on an "AS IS" BASIS,
113
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
114
+ See the License for the specific language governing permissions and
115
+ limitations under the License.
@@ -0,0 +1,2 @@
1
+ export {};
2
+ //# sourceMappingURL=browser-guard.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"browser-guard.d.ts","sourceRoot":"","sources":["../src/browser-guard.ts"],"names":[],"mappings":"AAuBA,OAAO,EAAE,CAAA"}
@@ -0,0 +1,18 @@
1
+ /**
2
+ * Browser guard — `@molecule/api-ai-local` is SERVER-ONLY.
3
+ *
4
+ * Generated by scripts/gen-browser-guards.mjs (workspace root) — edit THAT, not this.
5
+ * Evaluating a server package in a browser bundle is always an import-graph mistake
6
+ * (node APIs, secrets); without this guard it surfaces as a cryptic downstream crash
7
+ * ("Buffer is not defined") far from the culprit. Throwing here names the package and
8
+ * the fix at the exact moment the client bundle evaluates it. jsdom tests and SSR are
9
+ * unaffected: the throw requires browser globals AND the absence of a node runtime.
10
+ */
11
+ const g = globalThis;
12
+ if (g.window !== undefined && g.document !== undefined && !g.process?.versions?.node) {
13
+ throw new Error('@molecule/api-ai-local is SERVER-ONLY: it was bundled into browser/client code. Import it only ' +
14
+ 'from server code (a server route/function or your API), or dynamic-import it inside ' +
15
+ 'the server handler — never from components or shared client modules, and never ' +
16
+ 'polyfill Buffer/process to silence this.');
17
+ }
18
+ export {};
@@ -0,0 +1,22 @@
1
+ /**
2
+ * Local (OpenAI-compatible) ai-local provider for molecule.dev.
3
+ *
4
+ * Streams chat completions from any local inference server that speaks the
5
+ * OpenAI `chat/completions` protocol (Ollama, LM Studio, llama.cpp, vLLM),
6
+ * keyless by default.
7
+ *
8
+ * @remarks
9
+ * **Error message disambiguation**: a plain 400 that ISN'T a context-length error (bad param,
10
+ * malformed tool schema) gets its own non-retryable message distinct from the generic
11
+ * "AI service error. Please try again." used for retryable failures.
12
+ *
13
+ * @module
14
+ */
15
+ export * from './browser-guard.js';
16
+ export { createProvider, LocalAIProvider } from './provider.js';
17
+ export * from './secrets.js';
18
+ export * from './types.js';
19
+ import type { AIProvider } from '@molecule/api-ai';
20
+ /** The provider implementation. Constructs keyless — no secret required. */
21
+ export declare const provider: AIProvider;
22
+ //# sourceMappingURL=index.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;GAaG;AAEH,cAAc,oBAAoB,CAAA;AAClC,OAAO,EAAE,cAAc,EAAE,eAAe,EAAE,MAAM,eAAe,CAAA;AAC/D,cAAc,cAAc,CAAA;AAC5B,cAAc,YAAY,CAAA;AAE1B,OAAO,KAAK,EAAE,UAAU,EAAE,MAAM,kBAAkB,CAAA;AAMlD,4EAA4E;AAC5E,eAAO,MAAM,QAAQ,EAAE,UAUrB,CAAA"}
package/dist/index.js ADDED
@@ -0,0 +1,35 @@
1
+ /**
2
+ * Local (OpenAI-compatible) ai-local provider for molecule.dev.
3
+ *
4
+ * Streams chat completions from any local inference server that speaks the
5
+ * OpenAI `chat/completions` protocol (Ollama, LM Studio, llama.cpp, vLLM),
6
+ * keyless by default.
7
+ *
8
+ * @remarks
9
+ * **Error message disambiguation**: a plain 400 that ISN'T a context-length error (bad param,
10
+ * malformed tool schema) gets its own non-retryable message distinct from the generic
11
+ * "AI service error. Please try again." used for retryable failures.
12
+ *
13
+ * @module
14
+ */
15
+ export * from './browser-guard.js';
16
+ export { createProvider, LocalAIProvider } from './provider.js';
17
+ export * from './secrets.js';
18
+ export * from './types.js';
19
+ import { createProvider } from './provider.js';
20
+ /** Lazily-initialized provider singleton. Defers creation until first use so env vars / config are resolved. */
21
+ let _provider = null;
22
+ /** The provider implementation. Constructs keyless — no secret required. */
23
+ export const provider = new Proxy({}, {
24
+ get(_, prop, receiver) {
25
+ if (!_provider)
26
+ _provider = createProvider();
27
+ return Reflect.get(_provider, prop, receiver);
28
+ },
29
+ // set trap: methods run with `this` bound to the proxy — without it, instance-state writes land on the dummy target and are lost (see api-push-notifications-web-push)
30
+ set(_, prop, value) {
31
+ if (!_provider)
32
+ _provider = createProvider();
33
+ return Reflect.set(_provider, prop, value);
34
+ },
35
+ });
@@ -0,0 +1,89 @@
1
+ /**
2
+ * Local implementation of AIProvider.
3
+ *
4
+ * Streams chat completions from any OpenAI-compatible LOCAL inference server
5
+ * (Ollama, LM Studio, llama.cpp, vLLM) over `POST {baseUrl}/chat/completions`
6
+ * using raw `fetch` — no SDK. It is the OpenAI provider with (a) a configurable
7
+ * `baseUrl`, (b) an OPTIONAL/absent API key (the `Authorization` header is sent
8
+ * only when a key is present), and (c) no "API key required" throw at
9
+ * construction (local runs are keyless).
10
+ *
11
+ * @module
12
+ */
13
+ import './secrets.js';
14
+ import type { AIProvider, ChatEvent, ChatParams } from '@molecule/api-ai';
15
+ import type { LocalConfig } from './types.js';
16
+ /**
17
+ * Local (OpenAI-compatible) chat provider implementing the `AIProvider`
18
+ * interface. Mirrors `@molecule/api-ai-openai` so the same handler code can
19
+ * dispatch to a local endpoint.
20
+ */
21
+ export declare class LocalAIProvider implements AIProvider {
22
+ readonly name = "local";
23
+ private apiKey;
24
+ private defaultModel;
25
+ private maxTokens;
26
+ private baseUrl;
27
+ private onRateLimit?;
28
+ constructor(config?: LocalConfig);
29
+ /**
30
+ * Send a chat request and yield streamed `ChatEvent`s.
31
+ *
32
+ * @param params - Chat parameters.
33
+ * @yields {ChatEvent}
34
+ */
35
+ chat(params: ChatParams): AsyncIterable<ChatEvent>;
36
+ /**
37
+ * Convert internal `ChatMessage` objects to the OpenAI message format.
38
+ * The system prompt becomes a leading `{role:'system'}` message.
39
+ */
40
+ private formatMessages;
41
+ /** Map a generic `ContentBlock` to OpenAI's content-part shape. */
42
+ private formatBlock;
43
+ /** Convert internal `AITool` definitions to OpenAI's `tools` array. */
44
+ private formatTools;
45
+ /**
46
+ * Parse a non-streaming `/chat/completions` response.
47
+ *
48
+ * @param data - The parsed JSON response body.
49
+ * @yields {ChatEvent} Text, tool-use, and done events extracted from the response.
50
+ */
51
+ private parseNonStreamingResponse;
52
+ /**
53
+ * Parse a streaming SSE response from `/chat/completions`.
54
+ *
55
+ * @param response - The fetch Response whose body is an SSE stream.
56
+ * @yields {ChatEvent} Incremental text, tool-use progress, keep-alive, and done events.
57
+ */
58
+ private parseStreamingResponse;
59
+ /**
60
+ * Process SSE lines from an OpenAI-compatible `chat/completions` stream.
61
+ *
62
+ * @param lines - Individual SSE lines to parse.
63
+ * @param state - Mutable stream state tracking tokens and pending tool calls.
64
+ * @yields {ChatEvent} Text, tool-use progress, and usage events parsed from the SSE data lines.
65
+ */
66
+ private processSSELines;
67
+ /**
68
+ * Accumulate a streamed `delta.tool_calls` chunk into `state`, emitting a
69
+ * `tool_use_start` the first time a tool call is identified and a
70
+ * `tool_input_delta` for each argument chunk — real progress that re-arms the
71
+ * consumer's stream timeout instead of a silent `keep_alive` while a large
72
+ * tool input streams in. The completed `tool_use` is flushed at stream end.
73
+ *
74
+ * @param toolCalls - The `delta.tool_calls` array from one SSE chunk.
75
+ * @param state - Mutable stream state holding pending tool calls.
76
+ * @yields {ChatEvent} `tool_use_start` and `tool_input_delta` progress events.
77
+ */
78
+ private processToolCallDeltas;
79
+ }
80
+ /**
81
+ * Create a local (OpenAI-compatible) AI provider instance.
82
+ *
83
+ * Constructs WITHOUT requiring any secret — local endpoints run keyless.
84
+ *
85
+ * @param config - Local provider configuration.
86
+ * @returns An `AIProvider` backed by an OpenAI-compatible local endpoint.
87
+ */
88
+ export declare function createProvider(config?: LocalConfig): AIProvider;
89
+ //# sourceMappingURL=provider.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"provider.d.ts","sourceRoot":"","sources":["../src/provider.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;GAWG;AAKH,OAAO,cAAc,CAAA;AAErB,OAAO,KAAK,EACV,UAAU,EAGV,SAAS,EAET,UAAU,EAEX,MAAM,kBAAkB,CAAA;AAGzB,OAAO,KAAK,EAAE,WAAW,EAAE,MAAM,YAAY,CAAA;AAqB7C;;;;GAIG;AACH,qBAAa,eAAgB,YAAW,UAAU;IAChD,QAAQ,CAAC,IAAI,WAAU;IACvB,OAAO,CAAC,MAAM,CAAQ;IACtB,OAAO,CAAC,YAAY,CAAQ;IAC5B,OAAO,CAAC,SAAS,CAAQ;IACzB,OAAO,CAAC,OAAO,CAAQ;IACvB,OAAO,CAAC,WAAW,CAAC,CAAqB;gBAE7B,MAAM,GAAE,WAAgB;IAepC;;;;;OAKG;IACI,IAAI,CAAC,MAAM,EAAE,UAAU,GAAG,aAAa,CAAC,SAAS,CAAC;IAoLzD;;;OAGG;IACH,OAAO,CAAC,cAAc;IAoBtB,mEAAmE;IACnE,OAAO,CAAC,WAAW;IAiCnB,uEAAuE;IACvE,OAAO,CAAC,WAAW;IAWnB;;;;;OAKG;IACH,OAAO,CAAE,yBAAyB;IA8ClC;;;;;OAKG;YACY,sBAAsB;IA2ErC;;;;;;OAMG;IACH,OAAO,CAAE,eAAe;IA+DxB;;;;;;;;;;OAUG;IACH,OAAO,CAAE,qBAAqB;CAqC/B;AAED;;;;;;;GAOG;AACH,wBAAgB,cAAc,CAAC,MAAM,CAAC,EAAE,WAAW,GAAG,UAAU,CAE/D"}
@@ -0,0 +1,540 @@
1
+ /**
2
+ * Local implementation of AIProvider.
3
+ *
4
+ * Streams chat completions from any OpenAI-compatible LOCAL inference server
5
+ * (Ollama, LM Studio, llama.cpp, vLLM) over `POST {baseUrl}/chat/completions`
6
+ * using raw `fetch` — no SDK. It is the OpenAI provider with (a) a configurable
7
+ * `baseUrl`, (b) an OPTIONAL/absent API key (the `Authorization` header is sent
8
+ * only when a key is present), and (c) no "API key required" throw at
9
+ * construction (local runs are keyless).
10
+ *
11
+ * @module
12
+ */
13
+ // Side-effect import: registers this bond's OPTIONAL secret definitions so the
14
+ // runtime registry is populated even when provider.js is imported directly
15
+ // (not through the package barrel).
16
+ import './secrets.js';
17
+ import { getLogger } from '@molecule/api-bond';
18
+ const logger = getLogger();
19
+ /** Ollama's OpenAI-compatible endpoint (base URL includes the `/v1` segment). */
20
+ const DEFAULT_BASE_URL = 'http://localhost:11434/v1';
21
+ /** Default model when none is configured or requested. */
22
+ const DEFAULT_MODEL = 'llama3.1';
23
+ /** Default max output tokens. */
24
+ const DEFAULT_MAX_TOKENS = 4096;
25
+ /**
26
+ * Local (OpenAI-compatible) chat provider implementing the `AIProvider`
27
+ * interface. Mirrors `@molecule/api-ai-openai` so the same handler code can
28
+ * dispatch to a local endpoint.
29
+ */
30
+ export class LocalAIProvider {
31
+ name = 'local';
32
+ apiKey;
33
+ defaultModel;
34
+ maxTokens;
35
+ baseUrl;
36
+ onRateLimit;
37
+ constructor(config = {}) {
38
+ // Keyless by default — many local servers ignore auth. Empty string means
39
+ // "no key", so no Authorization header is sent. NEVER throws for a missing
40
+ // key at construction.
41
+ this.apiKey = config.apiKey ?? process.env.LOCAL_AI_API_KEY ?? '';
42
+ this.defaultModel = config.model ?? process.env.LOCAL_AI_MODEL ?? DEFAULT_MODEL;
43
+ this.maxTokens = DEFAULT_MAX_TOKENS;
44
+ this.onRateLimit = config.onRateLimit;
45
+ this.baseUrl =
46
+ config.baseUrl ??
47
+ process.env.LOCAL_AI_BASE_URL ??
48
+ process.env.OLLAMA_BASE_URL ??
49
+ DEFAULT_BASE_URL;
50
+ }
51
+ /**
52
+ * Send a chat request and yield streamed `ChatEvent`s.
53
+ *
54
+ * @param params - Chat parameters.
55
+ * @yields {ChatEvent}
56
+ */
57
+ async *chat(params) {
58
+ const model = params.model ?? this.defaultModel;
59
+ const maxTokens = params.maxTokens ?? this.maxTokens;
60
+ const messages = this.formatMessages(params.messages, params.system);
61
+ const body = {
62
+ // Abuse attribution (see ChatParams.endUserId): OpenAI-compatible APIs
63
+ // scope enforcement by `user`, so an abusive tenant can be actioned
64
+ // individually instead of the org key that serves everyone. The value is
65
+ // the caller's opaque hash — never derived or logged here.
66
+ ...(params.endUserId ? { user: params.endUserId } : {}),
67
+ model,
68
+ messages,
69
+ max_tokens: maxTokens,
70
+ };
71
+ if (params.temperature !== undefined)
72
+ body.temperature = params.temperature;
73
+ // Reasoning depth: the caller resolves the model's native reasoning_effort
74
+ // value from the model catalog and passes it as `thinking.effort`. Local
75
+ // servers that don't support it typically ignore an unknown field.
76
+ if (params.thinking?.effort) {
77
+ body.reasoning_effort = params.thinking.effort;
78
+ }
79
+ const tools = params.tools?.length ? this.formatTools(params.tools) : null;
80
+ if (tools) {
81
+ body.tools = tools;
82
+ if (params.toolChoice === 'required') {
83
+ body.tool_choice = 'required';
84
+ }
85
+ else if (typeof params.toolChoice === 'object' && params.toolChoice?.type === 'tool') {
86
+ body.tool_choice = { type: 'function', function: { name: params.toolChoice.name } };
87
+ }
88
+ }
89
+ const useStream = params.stream !== false;
90
+ if (useStream) {
91
+ body.stream = true;
92
+ body.stream_options = { include_usage: true };
93
+ }
94
+ const headers = {
95
+ 'Content-Type': 'application/json',
96
+ };
97
+ // Only authenticate when a key is configured — local servers are keyless.
98
+ if (this.apiKey)
99
+ headers.Authorization = `Bearer ${this.apiKey}`;
100
+ const DEFAULT_TIMEOUT_MS = 5 * 60 * 1000;
101
+ const signal = params.signal ?? AbortSignal.timeout(DEFAULT_TIMEOUT_MS);
102
+ const url = `${this.baseUrl}/chat/completions`;
103
+ try {
104
+ const MAX_RETRIES = 3;
105
+ let response = null;
106
+ for (let attempt = 0; attempt <= MAX_RETRIES; attempt++) {
107
+ response = await fetch(url, {
108
+ method: 'POST',
109
+ headers,
110
+ body: JSON.stringify(body),
111
+ signal,
112
+ });
113
+ if (response.status === 429 || response.status === 503) {
114
+ const willRetry = attempt < MAX_RETRIES;
115
+ // Retry-After is equally-valid as delta-seconds or an HTTP-date;
116
+ // parseInt on the date form yields NaN, degrading the backoff to
117
+ // a ~0ms retry against an already-busy local endpoint. Guard it.
118
+ const retryAfter = response.headers.get('retry-after');
119
+ const parsedRetryAfterSeconds = retryAfter ? parseInt(retryAfter, 10) : NaN;
120
+ const hasRetryAfter = Number.isFinite(parsedRetryAfterSeconds) && parsedRetryAfterSeconds >= 0;
121
+ // Additive jitter (<500ms): concurrent callers rejected together must
122
+ // not sleep identical delays and re-arrive as the same burst.
123
+ const delayMs = willRetry
124
+ ? (hasRetryAfter
125
+ ? Math.min(parsedRetryAfterSeconds * 1000, 60_000)
126
+ : Math.min(1000 * 2 ** attempt, 30_000)) + Math.floor(Math.random() * 500)
127
+ : 0;
128
+ try {
129
+ this.onRateLimit?.({
130
+ provider: this.name,
131
+ model,
132
+ status: response.status,
133
+ attempt: attempt + 1,
134
+ willRetry,
135
+ retryInMs: delayMs,
136
+ ...(hasRetryAfter ? { retryAfterSeconds: parsedRetryAfterSeconds } : {}),
137
+ });
138
+ }
139
+ catch (error) {
140
+ // Host-app telemetry; its failure must never break the request.
141
+ logger.warn('onRateLimit callback threw', { error });
142
+ }
143
+ if (willRetry) {
144
+ logger.warn('Local AI endpoint busy, retrying', {
145
+ status: response.status,
146
+ attempt: attempt + 1,
147
+ delayMs,
148
+ });
149
+ await new Promise((resolve) => {
150
+ const timer = setTimeout(resolve, delayMs);
151
+ signal.addEventListener('abort', () => {
152
+ clearTimeout(timer);
153
+ resolve();
154
+ }, { once: true });
155
+ });
156
+ if (signal.aborted)
157
+ break;
158
+ continue;
159
+ }
160
+ }
161
+ break;
162
+ }
163
+ if (!response.ok) {
164
+ const errorBody = await response.text();
165
+ let detail = `HTTP ${response.status}`;
166
+ try {
167
+ const parsed = JSON.parse(errorBody);
168
+ if (parsed.error?.message)
169
+ detail = parsed.error.message;
170
+ }
171
+ catch (_error) {
172
+ // Best-effort: detail already has an HTTP-status fallback; malformed body just stays as-is.
173
+ if (errorBody.length > 0 && errorBody.length < 200)
174
+ detail = errorBody;
175
+ }
176
+ logger.error('Local AI endpoint error', { status: response.status, url, detail });
177
+ const clientMessage = response.status === 429
178
+ ? 'AI rate limit exceeded. Please try again shortly.'
179
+ : response.status === 401 || response.status === 403
180
+ ? 'AI service configuration error.'
181
+ : response.status === 404
182
+ ? 'Local AI endpoint or model not found. Is the server running and the model pulled?'
183
+ : response.status === 400 &&
184
+ /context.*length|maximum context|too many tokens|prompt is too long/i.test(detail)
185
+ ? "Conversation too long for the model's context window."
186
+ : response.status === 400
187
+ ? 'AI request was invalid — check the model and request parameters.'
188
+ : response.status === 503 || response.status === 502
189
+ ? 'AI service is temporarily overloaded. Please try again in a moment.'
190
+ : 'AI service error. Please try again.';
191
+ // Non-OK is a soft failure surfaced as an in-band error EVENT then a
192
+ // graceful return — matching the 8 sibling providers. molecule-dev's
193
+ // chat-handler consumes this via its `case 'error':` branch, which is a
194
+ // different path from its outer try/catch; throwing here would break
195
+ // that consumer contract.
196
+ yield { type: 'error', message: clientMessage, errorKey: 'ai.error.apiError' };
197
+ return;
198
+ }
199
+ if (!useStream) {
200
+ const data = (await response.json());
201
+ yield* this.parseNonStreamingResponse(data);
202
+ return;
203
+ }
204
+ yield* this.parseStreamingResponse(response);
205
+ }
206
+ catch (error) {
207
+ // A deliberate abort — client disconnect via `params.signal`, or the
208
+ // internal stream timeout — must PROPAGATE as a rejection with NO error
209
+ // event: the metering-snapshot contract books the usage of a cut stream,
210
+ // and this matches the sibling providers. Every other failure (a network
211
+ // error like connection-refused — common for a local server that isn't
212
+ // running — or a stream/parse error) becomes a sanitized error event +
213
+ // graceful return, never a throw.
214
+ if (signal.aborted)
215
+ throw error;
216
+ logger.error('Local AI endpoint request failed', { url, error });
217
+ yield {
218
+ type: 'error',
219
+ message: 'AI service error. Please try again.',
220
+ errorKey: 'ai.error.apiError',
221
+ };
222
+ return;
223
+ }
224
+ }
225
+ /**
226
+ * Convert internal `ChatMessage` objects to the OpenAI message format.
227
+ * The system prompt becomes a leading `{role:'system'}` message.
228
+ */
229
+ formatMessages(messages, systemPrompt) {
230
+ const out = [];
231
+ if (systemPrompt)
232
+ out.push({ role: 'system', content: systemPrompt });
233
+ for (const m of messages) {
234
+ if (m.role === 'system') {
235
+ out.push({ role: 'system', content: typeof m.content === 'string' ? m.content : '' });
236
+ continue;
237
+ }
238
+ const content = typeof m.content === 'string'
239
+ ? m.content
240
+ : m.content.map((b) => this.formatBlock(b));
241
+ out.push({ role: m.role, content });
242
+ }
243
+ return out;
244
+ }
245
+ /** Map a generic `ContentBlock` to OpenAI's content-part shape. */
246
+ formatBlock(block) {
247
+ switch (block.type) {
248
+ case 'text':
249
+ return { type: 'text', text: block.text };
250
+ case 'image':
251
+ return {
252
+ type: 'image_url',
253
+ image_url: { url: `data:${block.mediaType};base64,${block.data}` },
254
+ };
255
+ case 'document':
256
+ return {
257
+ type: 'text',
258
+ text: `[Document attachment (${block.mediaType}) — not supported by this provider]`,
259
+ };
260
+ case 'audio':
261
+ return {
262
+ type: 'text',
263
+ text: `[Audio attachment (${block.mediaType}) — not supported by this provider]`,
264
+ };
265
+ case 'video':
266
+ return {
267
+ type: 'text',
268
+ text: `[Video attachment (${block.mediaType}) — not supported by this provider]`,
269
+ };
270
+ case 'tool_use':
271
+ return { type: 'text', text: `[tool_use ${block.name}]` };
272
+ case 'tool_result':
273
+ return { type: 'text', text: `[tool_result for ${block.tool_use_id}]` };
274
+ default:
275
+ return block;
276
+ }
277
+ }
278
+ /** Convert internal `AITool` definitions to OpenAI's `tools` array. */
279
+ formatTools(tools) {
280
+ return tools.map((tool) => ({
281
+ type: 'function',
282
+ function: {
283
+ name: tool.name,
284
+ description: tool.description,
285
+ parameters: tool.parameters,
286
+ },
287
+ }));
288
+ }
289
+ /**
290
+ * Parse a non-streaming `/chat/completions` response.
291
+ *
292
+ * @param data - The parsed JSON response body.
293
+ * @yields {ChatEvent} Text, tool-use, and done events extracted from the response.
294
+ */
295
+ *parseNonStreamingResponse(data) {
296
+ const choices = data.choices;
297
+ const message = choices?.[0]?.message;
298
+ const content = message?.content;
299
+ if (typeof content === 'string' && content.length > 0) {
300
+ yield { type: 'text', content };
301
+ }
302
+ const toolCalls = message?.tool_calls;
303
+ if (toolCalls) {
304
+ for (const call of toolCalls) {
305
+ const fn = call.function;
306
+ let input = {};
307
+ try {
308
+ input = fn?.arguments ? JSON.parse(fn.arguments) : {};
309
+ }
310
+ catch (_error) {
311
+ // Best-effort: malformed tool-call arguments are unusable; fall back to empty object.
312
+ }
313
+ yield {
314
+ type: 'tool_use',
315
+ id: String(call.id ?? ''),
316
+ name: String(fn?.name ?? ''),
317
+ input,
318
+ };
319
+ }
320
+ }
321
+ const usage = data.usage;
322
+ const cached = usage?.prompt_tokens_details?.cached_tokens ?? 0;
323
+ yield {
324
+ type: 'done',
325
+ usage: {
326
+ // Report uncached input + cached separately (Anthropic semantics) so the
327
+ // consumer's context/cost math doesn't double-count the cached subset.
328
+ inputTokens: Math.max(0, (usage?.prompt_tokens ?? 0) - cached),
329
+ outputTokens: usage?.completion_tokens ?? 0,
330
+ cacheReadInputTokens: cached,
331
+ },
332
+ };
333
+ }
334
+ /**
335
+ * Parse a streaming SSE response from `/chat/completions`.
336
+ *
337
+ * @param response - The fetch Response whose body is an SSE stream.
338
+ * @yields {ChatEvent} Incremental text, tool-use progress, keep-alive, and done events.
339
+ */
340
+ async *parseStreamingResponse(response) {
341
+ const reader = response.body?.getReader();
342
+ if (!reader) {
343
+ yield {
344
+ type: 'error',
345
+ message: 'No response body from the local AI endpoint.',
346
+ errorKey: 'ai.error.noResponseBody',
347
+ };
348
+ return;
349
+ }
350
+ const decoder = new TextDecoder();
351
+ let buffer = '';
352
+ const state = {
353
+ inputTokens: 0,
354
+ outputTokens: 0,
355
+ cachedTokens: 0,
356
+ pendingTools: new Map(),
357
+ };
358
+ try {
359
+ while (true) {
360
+ const { done, value } = await reader.read();
361
+ if (done)
362
+ break;
363
+ buffer += decoder.decode(value, { stream: true });
364
+ const lines = buffer.split('\n');
365
+ buffer = lines.pop() ?? '';
366
+ let yielded = false;
367
+ for (const event of this.processSSELines(lines, state)) {
368
+ yielded = true;
369
+ yield event;
370
+ // A mid-stream error (rate-limit/overload during high load, sent as a
371
+ // `data: {"error": {...}}` chunk) means the turn was truncated. Stop
372
+ // here and do NOT fall through to the trailing `done`, which would
373
+ // report the truncated turn as a successful completion and skip the
374
+ // consumer's overload/retry recovery.
375
+ if (event.type === 'error')
376
+ return;
377
+ }
378
+ // Received data but produced no ChatEvent (a keepalive comment, an empty
379
+ // delta, a ping). The model is still alive — signal liveness so the
380
+ // consumer's inter-event stream timeout doesn't false-fire mid-generation.
381
+ if (!yielded)
382
+ yield { type: 'keep_alive' };
383
+ }
384
+ if (buffer.trim()) {
385
+ for (const event of this.processSSELines(buffer.split('\n'), state)) {
386
+ yield event;
387
+ if (event.type === 'error')
388
+ return;
389
+ }
390
+ }
391
+ // Flush any pending tool calls with their accumulated arguments.
392
+ for (const tool of state.pendingTools.values()) {
393
+ let input = {};
394
+ try {
395
+ input = tool.argsJson ? JSON.parse(tool.argsJson) : {};
396
+ }
397
+ catch (_error) {
398
+ // Best-effort: streaming tool arguments arrived malformed; fall back to empty object.
399
+ }
400
+ yield { type: 'tool_use', id: tool.id, name: tool.name, input };
401
+ }
402
+ state.pendingTools.clear();
403
+ }
404
+ finally {
405
+ reader.releaseLock();
406
+ }
407
+ yield {
408
+ type: 'done',
409
+ usage: {
410
+ inputTokens: Math.max(0, state.inputTokens - state.cachedTokens),
411
+ outputTokens: state.outputTokens,
412
+ cacheReadInputTokens: state.cachedTokens,
413
+ },
414
+ };
415
+ }
416
+ /**
417
+ * Process SSE lines from an OpenAI-compatible `chat/completions` stream.
418
+ *
419
+ * @param lines - Individual SSE lines to parse.
420
+ * @param state - Mutable stream state tracking tokens and pending tool calls.
421
+ * @yields {ChatEvent} Text, tool-use progress, and usage events parsed from the SSE data lines.
422
+ */
423
+ *processSSELines(lines, state) {
424
+ for (const line of lines) {
425
+ if (!line.startsWith('data: '))
426
+ continue;
427
+ const json = line.slice(6).trim();
428
+ if (json === '[DONE]')
429
+ continue;
430
+ try {
431
+ const event = JSON.parse(json);
432
+ // A mid-stream error arrives as a `data: {"error": {...}}` chunk (no
433
+ // `choices`) — most importantly rate-limit/overload when a shared local
434
+ // endpoint is under load. Without this it matches no branch below and is
435
+ // silently dropped, so the stream falls through to a misleading `done`
436
+ // and the truncated turn reads as a successful completion. Surface it as
437
+ // a real error event (also satisfies the no-silent-swallow rule).
438
+ const streamError = event.error;
439
+ if (streamError) {
440
+ const detail = `${streamError.code ?? streamError.type ?? ''} ${streamError.message ?? ''}`;
441
+ logger.error('Local AI streaming error event', {
442
+ type: streamError.type,
443
+ code: streamError.code,
444
+ message: streamError.message,
445
+ });
446
+ const clientMessage = /overload|capacity|503|529/i.test(detail)
447
+ ? 'AI service is temporarily overloaded. Please try again in a moment.'
448
+ : /rate.?limit|429|quota/i.test(detail)
449
+ ? 'AI rate limit exceeded. Please try again shortly.'
450
+ : 'AI service error. Please try again.';
451
+ yield { type: 'error', message: clientMessage, errorKey: 'ai.error.apiError' };
452
+ continue;
453
+ }
454
+ const choices = event.choices;
455
+ if (choices) {
456
+ for (const choice of choices) {
457
+ const delta = choice.delta;
458
+ if (typeof delta?.content === 'string' && delta.content.length > 0) {
459
+ yield { type: 'text', content: delta.content };
460
+ }
461
+ const toolCalls = delta?.tool_calls;
462
+ if (toolCalls) {
463
+ yield* this.processToolCallDeltas(toolCalls, state);
464
+ }
465
+ }
466
+ }
467
+ const usage = event.usage;
468
+ if (usage) {
469
+ if (typeof usage.prompt_tokens === 'number')
470
+ state.inputTokens = usage.prompt_tokens;
471
+ if (typeof usage.completion_tokens === 'number')
472
+ state.outputTokens = usage.completion_tokens;
473
+ if (typeof usage.prompt_tokens_details?.cached_tokens === 'number')
474
+ state.cachedTokens = usage.prompt_tokens_details.cached_tokens;
475
+ }
476
+ }
477
+ catch (error) {
478
+ logger.debug('Skipping malformed local AI SSE JSON line', { json, error });
479
+ }
480
+ }
481
+ }
482
+ /**
483
+ * Accumulate a streamed `delta.tool_calls` chunk into `state`, emitting a
484
+ * `tool_use_start` the first time a tool call is identified and a
485
+ * `tool_input_delta` for each argument chunk — real progress that re-arms the
486
+ * consumer's stream timeout instead of a silent `keep_alive` while a large
487
+ * tool input streams in. The completed `tool_use` is flushed at stream end.
488
+ *
489
+ * @param toolCalls - The `delta.tool_calls` array from one SSE chunk.
490
+ * @param state - Mutable stream state holding pending tool calls.
491
+ * @yields {ChatEvent} `tool_use_start` and `tool_input_delta` progress events.
492
+ */
493
+ *processToolCallDeltas(toolCalls, state) {
494
+ for (const call of toolCalls) {
495
+ const idx = Number(call.index ?? 0);
496
+ const fn = call.function;
497
+ const existing = state.pendingTools.get(idx) ?? {
498
+ id: String(call.id ?? ''),
499
+ name: '',
500
+ argsJson: '',
501
+ started: false,
502
+ };
503
+ if (call.id)
504
+ existing.id = String(call.id);
505
+ if (fn?.name)
506
+ existing.name = String(fn.name);
507
+ // Surface the tool start (id + name) as soon as both are known so the
508
+ // client shows activity before the full input finishes streaming.
509
+ if (!existing.started && existing.id && existing.name) {
510
+ existing.started = true;
511
+ state.pendingTools.set(idx, existing);
512
+ yield { type: 'tool_use_start', id: existing.id, name: existing.name };
513
+ }
514
+ if (typeof fn?.arguments === 'string' && fn.arguments.length > 0) {
515
+ existing.argsJson += fn.arguments;
516
+ state.pendingTools.set(idx, existing);
517
+ // Emit the chunk's char count (+ raw partial-JSON chunk, server-side
518
+ // only) as real progress rather than a silent keep_alive.
519
+ yield {
520
+ type: 'tool_input_delta',
521
+ id: existing.id,
522
+ chars: fn.arguments.length,
523
+ text: fn.arguments,
524
+ };
525
+ }
526
+ state.pendingTools.set(idx, existing);
527
+ }
528
+ }
529
+ }
530
+ /**
531
+ * Create a local (OpenAI-compatible) AI provider instance.
532
+ *
533
+ * Constructs WITHOUT requiring any secret — local endpoints run keyless.
534
+ *
535
+ * @param config - Local provider configuration.
536
+ * @returns An `AIProvider` backed by an OpenAI-compatible local endpoint.
537
+ */
538
+ export function createProvider(config) {
539
+ return new LocalAIProvider(config);
540
+ }
@@ -0,0 +1,15 @@
1
+ /**
2
+ * Local AI bond secret definitions — self-registered at import time so the
3
+ * runtime secrets registry (`@molecule/api-secrets`) can surface them in
4
+ * boot-time configuration reports.
5
+ *
6
+ * Both entries are OPTIONAL (`required: false`): a local endpoint runs keyless
7
+ * and defaults to Ollama's `http://localhost:11434/v1`, so nothing here can
8
+ * block boot. They exist only to document the overrides in the config report.
9
+ *
10
+ * @module
11
+ */
12
+ import type { SecretDefinition } from '@molecule/api-secrets';
13
+ /** Optional secret/config overrides recognised by the local AI bond. */
14
+ export declare const aiLocalSecretDefinitions: SecretDefinition[];
15
+ //# sourceMappingURL=secrets.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"secrets.d.ts","sourceRoot":"","sources":["../src/secrets.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;GAUG;AAEH,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,uBAAuB,CAAA;AAG7D,wEAAwE;AACxE,eAAO,MAAM,wBAAwB,EAAE,gBAAgB,EAetD,CAAA"}
@@ -0,0 +1,28 @@
1
+ /**
2
+ * Local AI bond secret definitions — self-registered at import time so the
3
+ * runtime secrets registry (`@molecule/api-secrets`) can surface them in
4
+ * boot-time configuration reports.
5
+ *
6
+ * Both entries are OPTIONAL (`required: false`): a local endpoint runs keyless
7
+ * and defaults to Ollama's `http://localhost:11434/v1`, so nothing here can
8
+ * block boot. They exist only to document the overrides in the config report.
9
+ *
10
+ * @module
11
+ */
12
+ import { registerSecrets } from '@molecule/api-secrets';
13
+ /** Optional secret/config overrides recognised by the local AI bond. */
14
+ export const aiLocalSecretDefinitions = [
15
+ {
16
+ key: 'LOCAL_AI_BASE_URL',
17
+ description: 'Base URL of the OpenAI-compatible local endpoint, including the version segment (e.g. http://localhost:11434/v1 for Ollama).',
18
+ required: false,
19
+ default: 'http://localhost:11434/v1',
20
+ example: 'http://localhost:11434/v1',
21
+ },
22
+ {
23
+ key: 'LOCAL_AI_API_KEY',
24
+ description: 'Optional API key for the local endpoint. Most local servers ignore auth; leave unset to run keyless.',
25
+ required: false,
26
+ },
27
+ ];
28
+ registerSecrets(aiLocalSecretDefinitions);
@@ -0,0 +1,44 @@
1
+ /**
2
+ * Local AI provider configuration.
3
+ *
4
+ * The local provider talks to any **OpenAI-compatible** local inference server
5
+ * — Ollama, LM Studio, llama.cpp's server, vLLM, etc. — all of which expose
6
+ * `POST {baseUrl}/chat/completions` with SSE streaming. It is effectively the
7
+ * OpenAI provider with a configurable `baseUrl` and an OPTIONAL API key (most
8
+ * local servers ignore authentication entirely).
9
+ *
10
+ * Resolution order + defaults:
11
+ * - `baseUrl`: `config.baseUrl` → `LOCAL_AI_BASE_URL` → `OLLAMA_BASE_URL` →
12
+ * `http://localhost:11434/v1` (Ollama's OpenAI-compatible endpoint). Must
13
+ * include the API version segment (e.g. `/v1`); the request path is
14
+ * `${baseUrl}/chat/completions`.
15
+ * - `apiKey`: `config.apiKey` → `LOCAL_AI_API_KEY` → unset. When unset, NO
16
+ * `Authorization` header is sent — local runs are keyless by default.
17
+ * - `model`: `params.model` (per call) → `config.model` → `LOCAL_AI_MODEL` →
18
+ * `llama3.1`.
19
+ *
20
+ * @module
21
+ */
22
+ import type { AiRateLimitCallback } from '@molecule/api-ai';
23
+ /** Local (OpenAI-compatible) provider configuration. */
24
+ export interface LocalConfig {
25
+ /** Called on each rate-limited/busy upstream response, before any retry sleep. */
26
+ onRateLimit?: AiRateLimitCallback;
27
+ /**
28
+ * Base URL of the OpenAI-compatible endpoint, INCLUDING the version segment
29
+ * (e.g. `http://localhost:11434/v1`). Overrides `LOCAL_AI_BASE_URL` /
30
+ * `OLLAMA_BASE_URL`. Defaults to Ollama's `http://localhost:11434/v1`.
31
+ */
32
+ baseUrl?: string;
33
+ /**
34
+ * Optional API key. Most local servers ignore auth; when omitted (and no
35
+ * `LOCAL_AI_API_KEY` env var is set) no `Authorization` header is sent.
36
+ */
37
+ apiKey?: string;
38
+ /**
39
+ * Default model when a call doesn't specify one. Overrides `LOCAL_AI_MODEL`.
40
+ * Defaults to `llama3.1`.
41
+ */
42
+ model?: string;
43
+ }
44
+ //# sourceMappingURL=types.d.ts.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;GAoBG;AAEH,OAAO,KAAK,EAAE,mBAAmB,EAAE,MAAM,kBAAkB,CAAA;AAE3D,wDAAwD;AACxD,MAAM,WAAW,WAAW;IAC1B,kFAAkF;IAClF,WAAW,CAAC,EAAE,mBAAmB,CAAA;IACjC;;;;OAIG;IACH,OAAO,CAAC,EAAE,MAAM,CAAA;IAChB;;;OAGG;IACH,MAAM,CAAC,EAAE,MAAM,CAAA;IACf;;;OAGG;IACH,KAAK,CAAC,EAAE,MAAM,CAAA;CACf"}
package/dist/types.js ADDED
@@ -0,0 +1,22 @@
1
+ /**
2
+ * Local AI provider configuration.
3
+ *
4
+ * The local provider talks to any **OpenAI-compatible** local inference server
5
+ * — Ollama, LM Studio, llama.cpp's server, vLLM, etc. — all of which expose
6
+ * `POST {baseUrl}/chat/completions` with SSE streaming. It is effectively the
7
+ * OpenAI provider with a configurable `baseUrl` and an OPTIONAL API key (most
8
+ * local servers ignore authentication entirely).
9
+ *
10
+ * Resolution order + defaults:
11
+ * - `baseUrl`: `config.baseUrl` → `LOCAL_AI_BASE_URL` → `OLLAMA_BASE_URL` →
12
+ * `http://localhost:11434/v1` (Ollama's OpenAI-compatible endpoint). Must
13
+ * include the API version segment (e.g. `/v1`); the request path is
14
+ * `${baseUrl}/chat/completions`.
15
+ * - `apiKey`: `config.apiKey` → `LOCAL_AI_API_KEY` → unset. When unset, NO
16
+ * `Authorization` header is sent — local runs are keyless by default.
17
+ * - `model`: `params.model` (per call) → `config.model` → `LOCAL_AI_MODEL` →
18
+ * `llama3.1`.
19
+ *
20
+ * @module
21
+ */
22
+ export {};
package/package.json ADDED
@@ -0,0 +1,47 @@
1
+ {
2
+ "name": "@molecule/api-ai-local",
3
+ "version": "1.0.0",
4
+ "description": "AIProvider bond for any OpenAI-compatible local inference server (Ollama, LM Studio, llama.cpp, vLLM) — configurable base URL, keyless by default, streamed via raw fetch.",
5
+ "type": "module",
6
+ "main": "dist/index.js",
7
+ "types": "dist/index.d.ts",
8
+ "scripts": {
9
+ "build": "tsc",
10
+ "test": "vitest run",
11
+ "test:watch": "vitest"
12
+ },
13
+ "exports": {
14
+ ".": {
15
+ "types": "./dist/index.d.ts",
16
+ "import": "./dist/index.js"
17
+ }
18
+ },
19
+ "files": [
20
+ "dist"
21
+ ],
22
+ "keywords": [
23
+ "molecule",
24
+ "ai",
25
+ "local"
26
+ ],
27
+ "license": "Apache-2.0",
28
+ "devDependencies": {
29
+ "@molecule/api-ai": "1.0.0",
30
+ "@molecule/api-bond": "1.0.0",
31
+ "@molecule/api-secrets": "1.0.0",
32
+ "@types/node": "26.1.2",
33
+ "typescript": "6.0.3",
34
+ "vitest": "4.1.10"
35
+ },
36
+ "peerDependencies": {
37
+ "@molecule/api-ai": "^1.0.0",
38
+ "@molecule/api-bond": "^1.0.0",
39
+ "@molecule/api-i18n": "^1.0.0",
40
+ "@molecule/api-secrets": "^1.0.0"
41
+ },
42
+ "peerDependenciesMeta": {
43
+ "@molecule/api-ai": {
44
+ "optional": true
45
+ }
46
+ }
47
+ }