llm-stream-assemble 1.2.0 → 1.3.5
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +256 -24
- package/dist/adapters/openai-compatible.cjs +79 -19
- package/dist/adapters/openai-compatible.cjs.map +1 -1
- package/dist/adapters/openai-compatible.d.cts +36 -2
- package/dist/adapters/openai-compatible.d.ts +36 -2
- package/dist/adapters/openai-compatible.js +68 -20
- package/dist/adapters/openai-compatible.js.map +1 -1
- package/dist/index.cjs +78 -19
- package/dist/index.cjs.map +1 -1
- package/dist/index.d.cts +1 -1
- package/dist/index.d.ts +1 -1
- package/dist/index.js +68 -20
- package/dist/index.js.map +1 -1
- package/package.json +12 -4
package/README.md
CHANGED
|
@@ -1,28 +1,36 @@
|
|
|
1
1
|
# llm-stream-assemble
|
|
2
2
|
|
|
3
|
-

|
|
4
4
|

|
|
5
5
|

|
|
6
|
-

|
|
7
7
|
[](https://github.com/01laky/llm-stream-assemble/actions/workflows/ci.yml)
|
|
8
|
-

|
|
9
9
|
|
|
10
10
|
**One typed event model for every LLM stream** — text, tool calls, reasoning, JSON, usage, refusals, errors, and non-streaming responses.
|
|
11
11
|
|
|
12
12
|
> A zero-dependency TypeScript layer for assembling **OpenAI**, **Anthropic**, **Google Gemini**, and **OpenAI-compatible** LLM streams into unified events — so you can stop hand-rolling provider parsers and keep one clean, typed event model across chat UIs, agents, proxies, and backends.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
Turn provider SSE fragments into typed events — **not another `+=` loop**.
|
|
15
|
+
|
|
16
|
+
**Status:** Stable `1.3.5`. Five built-in adapters, thirteen OpenAI-compatible host presets (including **Azure OpenAI** and **Cloudflare Workers AI**), transforms, replay helpers, and examples are production-ready. Pin semver ranges as usual and review [CHANGELOG.md](./CHANGELOG.md) before major upgrades.
|
|
15
17
|
|
|
16
18
|
---
|
|
17
19
|
|
|
18
20
|
## Contents
|
|
19
21
|
|
|
22
|
+
- [Why not just concatenate?](#why-not-just-concatenate)
|
|
23
|
+
- [Edge-case showcase](#edge-case-showcase)
|
|
20
24
|
- [Why use this](#why-use-this)
|
|
21
25
|
- [Architecture](#architecture)
|
|
22
26
|
- [Providers at a glance](#providers-at-a-glance)
|
|
23
27
|
- [Install](#install)
|
|
28
|
+
- [First success in 30 seconds](#first-success-in-30-seconds)
|
|
24
29
|
- [Quickstart](#quickstart)
|
|
30
|
+
- [Quick decision guide](#quick-decision-guide)
|
|
25
31
|
- [Documentation](#documentation)
|
|
32
|
+
- [How this compares](#how-this-compares)
|
|
33
|
+
- [Examples](#examples)
|
|
26
34
|
- [Usage guides](#usage-guides)
|
|
27
35
|
- [Transforms & replay](#transforms--replay)
|
|
28
36
|
- [Examples & proxy safety](#examples--proxy-safety)
|
|
@@ -31,13 +39,67 @@
|
|
|
31
39
|
|
|
32
40
|
---
|
|
33
41
|
|
|
42
|
+
## Why not just concatenate?
|
|
43
|
+
|
|
44
|
+
Raw LLM streams look like text, but **simple string concatenation or naive `JSON.parse` per chunk fails** in production. Providers emit **protocol events**, not finished messages.
|
|
45
|
+
|
|
46
|
+
1. **SSE mid-line splits** — TCP chunks can break `data: {"choices":[...]}\n` across reads; you need a line buffer (`parse-sse.ts`, fixtures **LSA-C**).
|
|
47
|
+
2. **Tool argument fragmentation** — function parameters arrive as partial JSON across dozens of deltas; only assembly produces valid `tool_call.done` args.
|
|
48
|
+
3. **Anthropic id/index ordering** — `tool_use` blocks may stream `index` before `id`; fine-grained `input_json_delta` is invalid JSON until the block ends.
|
|
49
|
+
4. **Reasoning vs user text** — DeepSeek R1, Claude thinking, and OpenAI reasoning models interleave hidden reasoning that must map to `reasoning.*`, not `text.*`.
|
|
50
|
+
5. **JSON mode streaming** — structured output streams as deltas; you do not receive a parsed object until completion (`json.delta` / `json.done`).
|
|
51
|
+
6. **Stream lifecycle** — `[DONE]` markers, usage-only tail chunks, and incomplete streams without explicit finish need consistent terminal handling.
|
|
52
|
+
7. **Mid-stream errors** — provider error payloads must not leak raw internals to browsers; use `sanitizeErrors` when proxying (**LSA-X23**).
|
|
53
|
+
8. **Dual code paths** — the same `StreamEvent` union should work for `stream: true` SSE and non-stream JSON (`assembleStream` vs `assembleResponse`).
|
|
54
|
+
|
|
55
|
+
This library is the **assembly layer** between those raw bytes and your UI, agent, or proxy.
|
|
56
|
+
|
|
57
|
+
### Why not `text += chunk`?
|
|
58
|
+
|
|
59
|
+
The first reaction is often: “Why not `message += chunk`?” Provider streams are **protocol events**, not finished message strings.
|
|
60
|
+
|
|
61
|
+
| Failure mode | What breaks with `+=` / naive parse | This library |
|
|
62
|
+
| --------------------------------- | --------------------------------------------------- | ------------------------------------------------- |
|
|
63
|
+
| **Chunk boundaries** | SSE `data:` line split mid-payload across TCP reads | Line buffer — `parse-sse.ts` |
|
|
64
|
+
| **Incomplete structures** | One SSE payload ≠ one complete JSON message | Adapter per payload; assembler until `.done` |
|
|
65
|
+
| **State management** | Parallel tools, reasoning vs text channels | `EventAssembler` per stream |
|
|
66
|
+
| **Parser invalidity mid-stream** | Anthropic `input_json_delta`, partial tool args | Partial preview; valid at `.done` |
|
|
67
|
+
| **JSON partials** | Structured output streams as fragments | `json.*`, `tool_call.args.delta` |
|
|
68
|
+
| **Markdown fences in model text** | ` ```json ` split across **text tokens** | **Out of scope** — render `text.delta` in your UI |
|
|
69
|
+
|
|
70
|
+
See [Edge-case showcase](#edge-case-showcase) for concrete chunk examples.
|
|
71
|
+
|
|
72
|
+
---
|
|
73
|
+
|
|
74
|
+
## Edge-case showcase
|
|
75
|
+
|
|
76
|
+
Raw streams break in predictable ways. Three layers — **SSE framing**, **tool/JSON assembly**, **UI text** — fail differently:
|
|
77
|
+
|
|
78
|
+

|
|
79
|
+
|
|
80
|
+
- **SSE mid-line split** — TCP reads break `data: {...}\n` across buffers; line parser required.
|
|
81
|
+
- **Tool JSON partials** — args stream as `{`, `"city":`, `"Paris"}` before `tool_call.done`.
|
|
82
|
+
- **JSON mode** — structured output arrives as `json.delta` strings, not a parsed object.
|
|
83
|
+
|
|
84
|
+
**[Full edge-case walkthrough →](./docs/edge-cases.md)** — DIY vs `assembleStream`, fixture replay, test IDs (**LSA-C04**, **LSA-C52**, golden fixtures).
|
|
85
|
+
|
|
86
|
+
---
|
|
87
|
+
|
|
34
88
|
## Why use this
|
|
35
89
|
|
|
36
90
|
- **Zero runtime dependencies** — thin adapters + core assembly, no provider SDKs.
|
|
37
91
|
- **Stream and non-stream parity** — same `StreamEvent` union from SSE chunks or JSON bodies.
|
|
38
|
-
- **Provider presets, not forks** — Groq, Azure, Perplexity, xAI, and others reuse one compatible parser with dialect options.
|
|
92
|
+
- **Provider presets, not forks** — Groq, Azure, Cloudflare, Perplexity, xAI, and others reuse one compatible parser with dialect options.
|
|
39
93
|
- **Proxy-ready transforms** — `toSSE({ sanitizeErrors: true })`, `tapEvents`, `collectStream`, fixture replay.
|
|
40
94
|
|
|
95
|
+
### Performance at a glance
|
|
96
|
+
|
|
97
|
+
- **Zero runtime dependencies** — verified in CI (`pnpm verify:deps`)
|
|
98
|
+
- **Incremental SSE parsing** — line buffer; no full-stream re-parse
|
|
99
|
+
- **Single-pass O(n) assembly** — **LSA-C52** smoke test on 10k chunks
|
|
100
|
+
- **Bounded buffers** — `maxBufferBytes` for untrusted streams
|
|
101
|
+
- **Local repro:** `pnpm bench:smoke` — see [performance](./docs/performance.md)
|
|
102
|
+
|
|
41
103
|
---
|
|
42
104
|
|
|
43
105
|
## Architecture
|
|
@@ -58,19 +120,30 @@ Every adapter maps provider-specific fragments into the same **`StreamEvent`** u
|
|
|
58
120
|
|
|
59
121
|
**Design constraints:** adapters never accumulate cross-chunk state beyond id/index reconciliation; assembly, buffering, and `.done` emission live in core. No HTTP client, no tool execution, no UI — just the stream layer.
|
|
60
122
|
|
|
123
|
+
### Lifecycle & concurrency
|
|
124
|
+
|
|
125
|
+
- **`EventAssembler` is stateful per stream** — it buffers text, reasoning, JSON, refusals, and open tool calls until `.done` / `finish`.
|
|
126
|
+
- **Public APIs create a new assembler per call** — `assembleStream`, `assembleFromPayloads`, `assembleResponse`, and `createAssemblyTransform` each construct their own instance.
|
|
127
|
+
- **One assembler = one stream/response** — do not share an instance across concurrent requests.
|
|
128
|
+
- **`EventAssembler.reset()`** clears state for tests or explicit reuse after a stream completes.
|
|
129
|
+
- **Adapters are thin** — one payload in, `RawChunk[]` out; create **one adapter instance per request/stream** (minimal id/index map only).
|
|
130
|
+
- **Transforms are stateless** — `tapEvents`, `toSSE`, and `collectStream` operate on the unified event stream.
|
|
131
|
+
|
|
132
|
+

|
|
133
|
+
|
|
61
134
|
Diagram sources: [`docs/img/`](./docs/img/) (Mermaid `.mmd` + committed SVG). Regenerate with `pnpm diagrams:build`.
|
|
62
135
|
|
|
63
136
|
---
|
|
64
137
|
|
|
65
138
|
## Providers at a glance
|
|
66
139
|
|
|
67
|
-
| Adapter | Provider / API
|
|
68
|
-
| --------------------------------------- |
|
|
69
|
-
| `openaiChatAdapter()` | OpenAI Chat Completions
|
|
70
|
-
| `openaiCompatibleAdapter({ provider })` | Groq, DeepSeek, Mistral, Ollama, LM Studio, Together, Fireworks, OpenRouter, Perplexity, xAI, **Azure OpenAI**, generic | `llm-stream-assemble` |
|
|
71
|
-
| `anthropicAdapter()` | Anthropic Messages
|
|
72
|
-
| `openaiResponsesAdapter()` | OpenAI Responses API
|
|
73
|
-
| `geminiAdapter()` | Google AI Gemini
|
|
140
|
+
| Adapter | Provider / API | Import |
|
|
141
|
+
| --------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------- |
|
|
142
|
+
| `openaiChatAdapter()` | OpenAI Chat Completions | `llm-stream-assemble` |
|
|
143
|
+
| `openaiCompatibleAdapter({ provider })` | Groq, DeepSeek, Mistral, Ollama, LM Studio, Together, Fireworks, OpenRouter, Perplexity, xAI, **Azure OpenAI**, **Cloudflare Workers AI**, generic | `llm-stream-assemble` |
|
|
144
|
+
| `anthropicAdapter()` | Anthropic Messages | `llm-stream-assemble` |
|
|
145
|
+
| `openaiResponsesAdapter()` | OpenAI Responses API | `llm-stream-assemble` |
|
|
146
|
+
| `geminiAdapter()` | Google AI Gemini | `llm-stream-assemble` or `/adapters/gemini` |
|
|
74
147
|
|
|
75
148
|
Full feature flags and quirks: [compatibility matrix](./docs/compatibility.md).
|
|
76
149
|
|
|
@@ -87,6 +160,23 @@ pnpm add llm-stream-assemble
|
|
|
87
160
|
|
|
88
161
|
---
|
|
89
162
|
|
|
163
|
+
## First success in 30 seconds
|
|
164
|
+
|
|
165
|
+
Minimal loop once you have a streaming `response.body` — see [Quickstart](#quickstart) for full `fetch` setup:
|
|
166
|
+
|
|
167
|
+
```ts
|
|
168
|
+
import { assembleStream, openaiChatAdapter } from "llm-stream-assemble";
|
|
169
|
+
|
|
170
|
+
for await (const event of assembleStream(response.body!, openaiChatAdapter())) {
|
|
171
|
+
if (event.type === "text.delta") process.stdout.write(event.text);
|
|
172
|
+
if (event.type === "text.done") console.log("\n--- done:", event.text);
|
|
173
|
+
}
|
|
174
|
+
```
|
|
175
|
+
|
|
176
|
+
Swap `openaiChatAdapter()` for `anthropicAdapter()`, `geminiAdapter()`, or `openaiCompatibleAdapter({ provider: "ollama" })` — [Quick decision guide](#quick-decision-guide).
|
|
177
|
+
|
|
178
|
+
---
|
|
179
|
+
|
|
90
180
|
## Quickstart
|
|
91
181
|
|
|
92
182
|
```ts
|
|
@@ -99,10 +189,31 @@ for await (const event of assembleStream(response.body!, openaiChatAdapter())) {
|
|
|
99
189
|
|
|
100
190
|
---
|
|
101
191
|
|
|
192
|
+
## Quick decision guide
|
|
193
|
+
|
|
194
|
+
Pick an adapter in ~30 seconds:
|
|
195
|
+
|
|
196
|
+

|
|
197
|
+
|
|
198
|
+
- **OpenAI Chat Completions SSE** → `openaiChatAdapter()`
|
|
199
|
+
- **OpenAI Responses API** → `openaiResponsesAdapter()`
|
|
200
|
+
- **Anthropic Messages** → `anthropicAdapter()`
|
|
201
|
+
- **Google Gemini** → `geminiAdapter()`
|
|
202
|
+
- **Groq, Ollama, Azure, Cloudflare, OpenRouter, …** → `openaiCompatibleAdapter({ provider })`
|
|
203
|
+
- **Non-streaming JSON body** → `assembleResponse(body, adapter)`
|
|
204
|
+
- **React chat UI / full agent framework** → not this package — see [comparison](./docs/comparison.md)
|
|
205
|
+
- **XML/markdown tag parsing from model text** → out of scope — see [Non-goals](#non-goals)
|
|
206
|
+
|
|
207
|
+
---
|
|
208
|
+
|
|
102
209
|
## Documentation
|
|
103
210
|
|
|
104
211
|
- [Provider compatibility matrix](./docs/compatibility.md)
|
|
105
212
|
- [Adapter author guide](./docs/adapter-guide.md)
|
|
213
|
+
- [Performance & runtime behavior](./docs/performance.md)
|
|
214
|
+
- [Edge-case showcase](./docs/edge-cases.md)
|
|
215
|
+
- [How this compares](./docs/comparison.md)
|
|
216
|
+
- [FAQ](./docs/faq.md)
|
|
106
217
|
- [Architecture diagrams](./docs/img/README.md)
|
|
107
218
|
- [Live smoke checklist (maintainers)](./docs/live-smoke.md)
|
|
108
219
|
- [Post-1.0 provider roadmap](./docs/post-1.0-provider-roadmap.md)
|
|
@@ -110,6 +221,86 @@ for await (const event of assembleStream(response.body!, openaiChatAdapter())) {
|
|
|
110
221
|
|
|
111
222
|
---
|
|
112
223
|
|
|
224
|
+
## How this compares
|
|
225
|
+
|
|
226
|
+
| | llm-stream-assemble | Full-stack AI SDK | Provider SDK | DIY concat |
|
|
227
|
+
| ------------ | --------------------- | ------------------ | -------------- | ------------ |
|
|
228
|
+
| Scope | Stream assembly only | HTTP + UI + agents | Vendor RPC | Manual parse |
|
|
229
|
+
| Events | Unified `StreamEvent` | Framework types | Vendor types | Ad hoc |
|
|
230
|
+
| Dependencies | Zero runtime | Many | Vendor package | None |
|
|
231
|
+
|
|
232
|
+
Full matrix, when-not-to-use, and alternatives: **[docs/comparison.md](./docs/comparison.md)**.
|
|
233
|
+
|
|
234
|
+
---
|
|
235
|
+
|
|
236
|
+
## Examples
|
|
237
|
+
|
|
238
|
+
Curated index — full snippets live in [Usage guides](#usage-guides) and [`examples/`](./examples/README.md).
|
|
239
|
+
|
|
240
|
+
### OpenAI Chat
|
|
241
|
+
|
|
242
|
+
```ts
|
|
243
|
+
import { assembleStream, openaiChatAdapter } from "llm-stream-assemble";
|
|
244
|
+
// fetch(..., { stream: true }) then:
|
|
245
|
+
for await (const event of assembleStream(response.body!, openaiChatAdapter())) {
|
|
246
|
+
if (event.type === "text.delta") process.stdout.write(event.text);
|
|
247
|
+
}
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
→ [`examples/node-fetch/openai-chat.ts`](./examples/node-fetch/openai-chat.ts)
|
|
251
|
+
|
|
252
|
+
### Ollama (local)
|
|
253
|
+
|
|
254
|
+
```ts
|
|
255
|
+
import { assembleStream, openaiCompatibleAdapter } from "llm-stream-assemble";
|
|
256
|
+
const adapter = openaiCompatibleAdapter({ provider: "ollama" });
|
|
257
|
+
for await (const event of assembleStream(response.body!, adapter)) {
|
|
258
|
+
if (event.type === "text.delta") process.stdout.write(event.text);
|
|
259
|
+
}
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
→ [`examples/node-fetch/openai-compatible.ts`](./examples/node-fetch/openai-compatible.ts) · Usage: [OpenAI-Compatible](#openai-compatible-usage)
|
|
263
|
+
|
|
264
|
+
### Anthropic Messages
|
|
265
|
+
|
|
266
|
+
→ [`examples/node-fetch/anthropic.ts`](./examples/node-fetch/anthropic.ts) · Usage: [Anthropic Messages](#anthropic-messages-usage)
|
|
267
|
+
|
|
268
|
+
### Google Gemini
|
|
269
|
+
|
|
270
|
+
→ [`examples/node-fetch/gemini.ts`](./examples/node-fetch/gemini.ts) · Usage: [Gemini](#gemini-usage)
|
|
271
|
+
|
|
272
|
+
### Streaming JSON (structured output)
|
|
273
|
+
|
|
274
|
+
```ts
|
|
275
|
+
for await (const event of assembleStream(response.body!, openaiChatAdapter({ jsonMode: true }))) {
|
|
276
|
+
if (event.type === "json.delta") process.stdout.write(event.delta);
|
|
277
|
+
if (event.type === "json.done") console.log(event.json);
|
|
278
|
+
}
|
|
279
|
+
```
|
|
280
|
+
|
|
281
|
+
### Tool calling
|
|
282
|
+
|
|
283
|
+
```ts
|
|
284
|
+
for await (const event of assembleStream(response.body!, openaiChatAdapter())) {
|
|
285
|
+
if (event.type === "tool_call.args.delta") process.stdout.write(event.delta);
|
|
286
|
+
if (event.type === "tool_call.done") console.log(event.name, event.args);
|
|
287
|
+
}
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
### Chat UI / markdown rendering
|
|
291
|
+
|
|
292
|
+
Stream `text.delta` into your renderer — this library does **not** parse markdown/XML tags from model output (see [Non-goals](#non-goals)).
|
|
293
|
+
|
|
294
|
+
### SSE proxy to browser
|
|
295
|
+
|
|
296
|
+
→ [`examples/proxy-safety/`](./examples/proxy-safety/) — `toSSE(events, { sanitizeErrors: true })`
|
|
297
|
+
|
|
298
|
+
### Fixture replay
|
|
299
|
+
|
|
300
|
+
→ [`examples/node-fetch/replay-fixture.ts`](./examples/node-fetch/replay-fixture.ts)
|
|
301
|
+
|
|
302
|
+
---
|
|
303
|
+
|
|
113
304
|
## Usage guides
|
|
114
305
|
|
|
115
306
|
### Core Usage
|
|
@@ -193,8 +384,9 @@ Provider presets:
|
|
|
193
384
|
| `perplexity` | Perplexity API | Search-grounded answers; citations in `metadata.raw` |
|
|
194
385
|
| `xai` | xAI Grok API | OpenAI-compatible; `reasoning_content` mapped when present |
|
|
195
386
|
| `azure` | Azure OpenAI Chat Completions | Stricter preset; deployment URL + `api-key` auth; content filter metadata in `metadata.raw` |
|
|
387
|
+
| `cloudflare` | Cloudflare Workers AI REST | OpenAI-compatible `/v1/chat/completions`; Bearer + account id; loose preset like Groq |
|
|
196
388
|
|
|
197
|
-
Base URL examples: Groq `https://api.groq.com/openai/v1`, DeepSeek `https://api.deepseek.com`, Mistral `https://api.mistral.ai/v1`, Ollama `http://localhost:11434/v1`, LM Studio `http://localhost:1234/v1`, Together `https://api.together.xyz/v1`, Fireworks `https://api.fireworks.ai/inference/v1`, OpenRouter `https://openrouter.ai/api/v1`, Perplexity `https://api.perplexity.ai`, xAI `https://api.x.ai/v1`, Azure OpenAI `https://{resource}.openai.azure.com/openai/deployments/{deployment}/chat/completions?api-version={version}`.
|
|
389
|
+
Base URL examples: Groq `https://api.groq.com/openai/v1`, DeepSeek `https://api.deepseek.com`, Mistral `https://api.mistral.ai/v1`, Ollama `http://localhost:11434/v1`, LM Studio `http://localhost:1234/v1`, Together `https://api.together.xyz/v1`, Fireworks `https://api.fireworks.ai/inference/v1`, OpenRouter `https://openrouter.ai/api/v1`, Perplexity `https://api.perplexity.ai`, xAI `https://api.x.ai/v1`, Azure OpenAI `https://{resource}.openai.azure.com/openai/deployments/{deployment}/chat/completions?api-version={version}`, Cloudflare Workers AI `https://api.cloudflare.com/client/v4/accounts/{account_id}/ai/v1/chat/completions`.
|
|
198
390
|
|
|
199
391
|
Strict vs loose configuration:
|
|
200
392
|
|
|
@@ -256,6 +448,44 @@ Use `openaiCompatibleAdapter({ provider: "azure", jsonMode: true })` when struct
|
|
|
256
448
|
|
|
257
449
|
See `examples/node-fetch/azure-openai.ts` for a URL builder helper and `examples/proxy-safety/README.md` for server-side proxy notes.
|
|
258
450
|
|
|
451
|
+
### Cloudflare Workers AI Usage
|
|
452
|
+
|
|
453
|
+
Cloudflare Workers AI exposes an OpenAI-compatible REST endpoint at `/v1/chat/completions` under your account. Use the **`cloudflare`** preset — not `generic` — when you want fixture-tested defaults for Workers AI REST (loose metadata tolerance like Groq).
|
|
454
|
+
|
|
455
|
+
```ts
|
|
456
|
+
import { assembleStream, openaiCompatibleAdapter } from "llm-stream-assemble";
|
|
457
|
+
|
|
458
|
+
const accountId = process.env.CLOUDFLARE_ACCOUNT_ID!;
|
|
459
|
+
const url = `https://api.cloudflare.com/client/v4/accounts/${accountId}/ai/v1/chat/completions`;
|
|
460
|
+
|
|
461
|
+
const response = await fetch(url, {
|
|
462
|
+
method: "POST",
|
|
463
|
+
headers: {
|
|
464
|
+
Authorization: `Bearer ${process.env.CLOUDFLARE_API_TOKEN!}`,
|
|
465
|
+
"Content-Type": "application/json",
|
|
466
|
+
},
|
|
467
|
+
body: JSON.stringify({
|
|
468
|
+
model: "@cf/meta/llama-3.1-8b-instruct",
|
|
469
|
+
messages: [{ role: "user", content: "Hello" }],
|
|
470
|
+
stream: true,
|
|
471
|
+
stream_options: { include_usage: true },
|
|
472
|
+
}),
|
|
473
|
+
});
|
|
474
|
+
|
|
475
|
+
for await (const event of assembleStream(
|
|
476
|
+
response.body!,
|
|
477
|
+
openaiCompatibleAdapter({ provider: "cloudflare" }),
|
|
478
|
+
)) {
|
|
479
|
+
if (event.type === "text.delta") process.stdout.write(event.text);
|
|
480
|
+
}
|
|
481
|
+
```
|
|
482
|
+
|
|
483
|
+
Streaming usage requires `stream_options: { include_usage: true }` on the request. Use `openaiCompatibleAdapter({ provider: "cloudflare", jsonMode: true })` when JSON output should map to `json.*` events.
|
|
484
|
+
|
|
485
|
+
The **`env.AI.run(model, { stream: true })`** Worker binding can return SSE bytes compatible with `assembleStream` when the model streams Chat Completions-shaped payloads — account binding and auth stay in your Worker; this library only parses the bytes.
|
|
486
|
+
|
|
487
|
+
See `examples/workers-ai/rest-chat-completions.ts` and `examples/proxy-safety/README.md` (Bearer token + account id must never reach the browser).
|
|
488
|
+
|
|
259
489
|
### Anthropic Messages Usage
|
|
260
490
|
|
|
261
491
|
`anthropicAdapter()` parses Anthropic Messages streaming events and non-streaming responses. Create one adapter instance per request/stream.
|
|
@@ -377,17 +607,18 @@ for await (const event of assembleFromFile(
|
|
|
377
607
|
|
|
378
608
|
## Examples & proxy safety
|
|
379
609
|
|
|
380
|
-
| Example
|
|
381
|
-
|
|
|
382
|
-
| [`examples/node-fetch/openai-chat.ts`](./examples/node-fetch/openai-chat.ts)
|
|
383
|
-
| [`examples/node-fetch/openai-compatible.ts`](./examples/node-fetch/openai-compatible.ts)
|
|
384
|
-
| [`examples/node-fetch/azure-openai.ts`](./examples/node-fetch/azure-openai.ts)
|
|
385
|
-
| [`examples/
|
|
386
|
-
| [`examples/node-fetch/
|
|
387
|
-
| [`examples/node-fetch/
|
|
388
|
-
| [`examples/node-fetch/
|
|
389
|
-
| [`examples/node-fetch/
|
|
390
|
-
| [`examples/
|
|
610
|
+
| Example | Description |
|
|
611
|
+
| ------------------------------------------------------------------------------------------------ | ------------------------------------------------ |
|
|
612
|
+
| [`examples/node-fetch/openai-chat.ts`](./examples/node-fetch/openai-chat.ts) | OpenAI Chat Completions streaming |
|
|
613
|
+
| [`examples/node-fetch/openai-compatible.ts`](./examples/node-fetch/openai-compatible.ts) | OpenAI-compatible presets |
|
|
614
|
+
| [`examples/node-fetch/azure-openai.ts`](./examples/node-fetch/azure-openai.ts) | Azure OpenAI deployment URL + `api-key` |
|
|
615
|
+
| [`examples/workers-ai/rest-chat-completions.ts`](./examples/workers-ai/rest-chat-completions.ts) | Cloudflare Workers AI REST + `cloudflare` preset |
|
|
616
|
+
| [`examples/node-fetch/perplexity.ts`](./examples/node-fetch/perplexity.ts) | Perplexity streaming |
|
|
617
|
+
| [`examples/node-fetch/xai.ts`](./examples/node-fetch/xai.ts) | xAI Grok streaming |
|
|
618
|
+
| [`examples/node-fetch/anthropic.ts`](./examples/node-fetch/anthropic.ts) | Anthropic Messages |
|
|
619
|
+
| [`examples/node-fetch/gemini.ts`](./examples/node-fetch/gemini.ts) | Google Gemini SSE |
|
|
620
|
+
| [`examples/node-fetch/replay-fixture.ts`](./examples/node-fetch/replay-fixture.ts) | Local fixture replay |
|
|
621
|
+
| [`examples/proxy-safety/`](./examples/proxy-safety/) | Proxy + browser client patterns |
|
|
391
622
|
|
|
392
623
|
Proxy safety:
|
|
393
624
|
|
|
@@ -420,6 +651,7 @@ pnpm verify
|
|
|
420
651
|
| `pnpm verify:deps` | fail if runtime dependencies are added |
|
|
421
652
|
| `pnpm release:prep` | pre-tag checks (version, CHANGELOG, dist, npm pack) |
|
|
422
653
|
| `pnpm diagrams:build` | regenerate README SVGs from Mermaid sources |
|
|
654
|
+
| `pnpm bench:smoke` | local LSA-C52 timing script (requires build first) |
|
|
423
655
|
| `pnpm test` | Vitest smoke tests |
|
|
424
656
|
| `pnpm build` | tsup → ESM + CJS + declarations |
|
|
425
657
|
|
|
@@ -484,7 +484,34 @@ function createOpenAIChatLikeAdapter(options) {
|
|
|
484
484
|
});
|
|
485
485
|
}
|
|
486
486
|
|
|
487
|
-
// src/adapters/openai-compatible.ts
|
|
487
|
+
// src/adapters/openai-compatible-presets.ts
|
|
488
|
+
var OPENAI_COMPATIBLE_PROVIDERS = [
|
|
489
|
+
"generic",
|
|
490
|
+
"openrouter",
|
|
491
|
+
"groq",
|
|
492
|
+
"deepseek",
|
|
493
|
+
"mistral",
|
|
494
|
+
"ollama",
|
|
495
|
+
"lmstudio",
|
|
496
|
+
"together",
|
|
497
|
+
"fireworks",
|
|
498
|
+
"perplexity",
|
|
499
|
+
"xai",
|
|
500
|
+
"azure",
|
|
501
|
+
"cloudflare"
|
|
502
|
+
];
|
|
503
|
+
var HOST_COMPATIBLE_PRESETS = OPENAI_COMPATIBLE_PROVIDERS.filter(
|
|
504
|
+
(p) => p !== "generic"
|
|
505
|
+
);
|
|
506
|
+
var STRICT_COMPATIBLE_PRESETS = [
|
|
507
|
+
"azure"
|
|
508
|
+
];
|
|
509
|
+
function isStrictCompatiblePreset(provider) {
|
|
510
|
+
return STRICT_COMPATIBLE_PRESETS.includes(provider);
|
|
511
|
+
}
|
|
512
|
+
var LOOSE_HOST_PRESETS = HOST_COMPATIBLE_PRESETS.filter(
|
|
513
|
+
(p) => !isStrictCompatiblePreset(p)
|
|
514
|
+
);
|
|
488
515
|
var DEFAULT_PRESET = {
|
|
489
516
|
looseErrorShape: true,
|
|
490
517
|
allowMissingMetadata: true,
|
|
@@ -502,33 +529,66 @@ var PRESET_OVERRIDES = {
|
|
|
502
529
|
reasoningFieldAliases: []
|
|
503
530
|
}
|
|
504
531
|
};
|
|
505
|
-
|
|
532
|
+
var PRESET_OVERRIDE_KEYS = Object.keys(PRESET_OVERRIDES);
|
|
533
|
+
function hasPresetOverride(provider) {
|
|
534
|
+
return provider in PRESET_OVERRIDES;
|
|
535
|
+
}
|
|
536
|
+
function providerPreset(provider) {
|
|
537
|
+
return {
|
|
538
|
+
...DEFAULT_PRESET,
|
|
539
|
+
...PRESET_OVERRIDES[provider]
|
|
540
|
+
};
|
|
541
|
+
}
|
|
542
|
+
|
|
543
|
+
// src/adapters/openai-compatible-resolve.ts
|
|
544
|
+
function resolveCompatibleAdapterConfig(options = {}) {
|
|
506
545
|
const preset = providerPreset(options.provider ?? "generic");
|
|
507
|
-
const
|
|
508
|
-
const
|
|
509
|
-
const
|
|
546
|
+
const allowMissingMetadata = options.allowMissingMetadata ?? preset.allowMissingMetadata ?? DEFAULT_PRESET.allowMissingMetadata;
|
|
547
|
+
const looseErrorShape = options.looseErrorShape ?? preset.looseErrorShape ?? DEFAULT_PRESET.looseErrorShape;
|
|
548
|
+
const useChoicePositionFallback = options.useChoicePositionFallback ?? preset.useChoicePositionFallback ?? DEFAULT_PRESET.useChoicePositionFallback;
|
|
549
|
+
return {
|
|
550
|
+
looseErrorShape,
|
|
551
|
+
allowMissingMetadata,
|
|
552
|
+
useChoicePositionFallback,
|
|
553
|
+
rejectUnrecognizedPayloads: allowMissingMetadata === false,
|
|
554
|
+
reasoningFieldAliases: [
|
|
555
|
+
...preset.reasoningFieldAliases ?? [],
|
|
556
|
+
...options.reasoningFieldAliases ?? []
|
|
557
|
+
]
|
|
558
|
+
};
|
|
559
|
+
}
|
|
560
|
+
function compatibleProviderLabel(provider) {
|
|
561
|
+
return provider ?? "generic";
|
|
562
|
+
}
|
|
563
|
+
|
|
564
|
+
// src/adapters/openai-compatible.ts
|
|
565
|
+
function openaiCompatibleAdapter(options = {}) {
|
|
566
|
+
const resolved = resolveCompatibleAdapterConfig(options);
|
|
510
567
|
return createOpenAIChatLikeAdapter({
|
|
511
568
|
...options,
|
|
512
|
-
looseErrorShape:
|
|
513
|
-
allowMissingMetadata:
|
|
514
|
-
useChoicePositionFallback:
|
|
569
|
+
looseErrorShape: resolved.looseErrorShape,
|
|
570
|
+
allowMissingMetadata: resolved.allowMissingMetadata,
|
|
571
|
+
useChoicePositionFallback: resolved.useChoicePositionFallback,
|
|
515
572
|
errorPrefix: "openaiCompatibleAdapter",
|
|
516
573
|
usageInputTokenFields: ["prompt_tokens", "input_tokens"],
|
|
517
574
|
usageOutputTokenFields: ["completion_tokens", "output_tokens"],
|
|
518
|
-
rejectUnrecognizedPayloads:
|
|
519
|
-
reasoningFieldAliases:
|
|
520
|
-
...preset.reasoningFieldAliases ?? [],
|
|
521
|
-
...options.reasoningFieldAliases ?? []
|
|
522
|
-
]
|
|
575
|
+
rejectUnrecognizedPayloads: resolved.rejectUnrecognizedPayloads,
|
|
576
|
+
reasoningFieldAliases: resolved.reasoningFieldAliases
|
|
523
577
|
});
|
|
524
578
|
}
|
|
525
|
-
function providerPreset(provider) {
|
|
526
|
-
return {
|
|
527
|
-
...DEFAULT_PRESET,
|
|
528
|
-
...PRESET_OVERRIDES[provider]
|
|
529
|
-
};
|
|
530
|
-
}
|
|
531
579
|
|
|
580
|
+
exports.DEFAULT_PRESET = DEFAULT_PRESET;
|
|
581
|
+
exports.HOST_COMPATIBLE_PRESETS = HOST_COMPATIBLE_PRESETS;
|
|
582
|
+
exports.LOOSE_HOST_PRESETS = LOOSE_HOST_PRESETS;
|
|
583
|
+
exports.OPENAI_COMPATIBLE_PROVIDERS = OPENAI_COMPATIBLE_PROVIDERS;
|
|
584
|
+
exports.PRESET_OVERRIDES = PRESET_OVERRIDES;
|
|
585
|
+
exports.PRESET_OVERRIDE_KEYS = PRESET_OVERRIDE_KEYS;
|
|
586
|
+
exports.STRICT_COMPATIBLE_PRESETS = STRICT_COMPATIBLE_PRESETS;
|
|
587
|
+
exports.compatibleProviderLabel = compatibleProviderLabel;
|
|
588
|
+
exports.hasPresetOverride = hasPresetOverride;
|
|
589
|
+
exports.isStrictCompatiblePreset = isStrictCompatiblePreset;
|
|
532
590
|
exports.openaiCompatibleAdapter = openaiCompatibleAdapter;
|
|
591
|
+
exports.providerPreset = providerPreset;
|
|
592
|
+
exports.resolveCompatibleAdapterConfig = resolveCompatibleAdapterConfig;
|
|
533
593
|
//# sourceMappingURL=openai-compatible.cjs.map
|
|
534
594
|
//# sourceMappingURL=openai-compatible.cjs.map
|