pi-lemonade-link 1.1.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +54 -21
- package/docs/README.md +23 -0
- package/docs/anthropic.md +183 -6
- package/docs/internal.md +1080 -0
- package/docs/lemonade.md +2453 -1665
- package/docs/llamacpp.md +385 -252
- package/docs/mcp.md +232 -128
- package/docs/ollama.md +967 -15
- package/docs/openai.md +1734 -634
- package/docs/router.md +201 -0
- package/index.ts +1543 -227
- package/lemonade.example.json +16 -2
- package/package.json +3 -3
package/docs/openai.md
CHANGED
|
@@ -1,46 +1,79 @@
|
|
|
1
1
|
# OpenAI-Compatible API
|
|
2
2
|
|
|
3
|
-
This
|
|
3
|
+
This page documents Lemonade's implementation of the [OpenAI API](https://developers.openai.com/api/docs).
|
|
4
4
|
|
|
5
|
+
<!-- BEGIN GENERATED: openai.summary -->
|
|
5
6
|
| Method | Endpoint | Description | Modality |
|
|
6
7
|
|--------|----------|-------------|----------|
|
|
7
|
-
| `POST` | [`/v1/chat/completions`](#post-v1chatcompletions) |
|
|
8
|
-
| `POST` | [`/v1/completions`](#post-v1completions) |
|
|
9
|
-
| `POST` | [`/v1/embeddings`](#post-v1embeddings) |
|
|
10
|
-
| `POST` | [`/v1/responses`](#post-v1responses) | Responses
|
|
11
|
-
| `POST` | [`/v1/audio/transcriptions`](#post-v1audiotranscriptions) |
|
|
12
|
-
| `POST` | [`/v1/
|
|
8
|
+
| `POST` | [`/v1/chat/completions`](#post-v1chatcompletions) | Generate the next message in a conversation | messages -> completion |
|
|
9
|
+
| `POST` | [`/v1/completions`](#post-v1completions) | Continue a text prompt | prompt -> completion |
|
|
10
|
+
| `POST` | [`/v1/embeddings`](#post-v1embeddings) | Embed text as vectors | text -> vector representations |
|
|
11
|
+
| `POST` | [`/v1/responses`](#post-v1responses) | Generate a response in OpenAI's Responses format | prompt/messages -> event |
|
|
12
|
+
| `POST` | [`/v1/audio/transcriptions`](#post-v1audiotranscriptions) | Transcribe an audio file to text | audio file -> text |
|
|
13
|
+
| `POST` | [`/v1/images/generations`](#post-v1imagesgenerations) | Generate an image from a text prompt | prompt -> image |
|
|
14
|
+
| `POST` | [`/v1/images/edits`](#post-v1imagesedits) | Edit images as a prompt describes | image + prompt -> edited image |
|
|
15
|
+
| `POST` | [`/v1/images/variations`](#post-v1imagesvariations) | Generate a variation of an image | image -> varied image |
|
|
16
|
+
| `POST` | [`/v1/images/upscale`](#post-v1imagesupscale) | Upscale an image | image + ESRGAN model -> upscaled image |
|
|
17
|
+
| `POST` | [`/v1/audio/speech`](#post-v1audiospeech) | Generate speech from text | text -> audio |
|
|
13
18
|
| `WS` | [`/realtime`](#ws-realtime) | Realtime Audio Transcription, OpenAI SDK compatible | streaming audio -> text |
|
|
14
|
-
| `
|
|
15
|
-
| `
|
|
16
|
-
| `
|
|
17
|
-
| `
|
|
18
|
-
| `
|
|
19
|
-
| `
|
|
20
|
-
|
|
19
|
+
| `GET` | [`/v1/models`](#get-v1models) | List the server's models | n/a |
|
|
20
|
+
| `GET` | [`/v1/models/{id}`](#get-v1modelsid) | Read one model | n/a |
|
|
21
|
+
| `GET` | [`/v1/models/{id}/files`](#get-v1modelsidfiles) | List one model's local files | n/a |
|
|
22
|
+
| `GET` | [`/v1/models/{id}/options`](#get-v1modelsidoptions) | Read a model's recipe options by layer | n/a |
|
|
23
|
+
| `POST` | [`/v1/models/{id}/options`](#post-v1modelsidoptions) | Save a model's recipe options | n/a |
|
|
24
|
+
| `DELETE` | [`/v1/models/{id}/options`](#delete-v1modelsidoptions) | Reset a model's recipe options | n/a |
|
|
25
|
+
<!-- END GENERATED: openai.summary -->
|
|
26
|
+
|
|
27
|
+
<!-- BEGIN GENERATED: openai.chat_completions -->
|
|
21
28
|
## `POST /v1/chat/completions`
|
|
22
|
-
<sub></sub>
|
|
30
|
+
|
|
31
|
+
Generates the next assistant message for a conversation, downloading and loading the model if it is not loaded.
|
|
23
32
|
|
|
24
|
-
|
|
33
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
25
34
|
|
|
26
35
|
### Parameters
|
|
27
36
|
|
|
28
37
|
| Parameter | Required | Description | Status |
|
|
29
38
|
|-----------|----------|-------------|--------|
|
|
30
|
-
| `messages` | Yes |
|
|
31
|
-
| `model` | Yes | The model to
|
|
32
|
-
| `stream` | No |
|
|
33
|
-
| `stop` | No |
|
|
34
|
-
| `logprobs` | No |
|
|
35
|
-
| `temperature` | No |
|
|
36
|
-
| `repeat_penalty` | No |
|
|
37
|
-
| `
|
|
38
|
-
| `
|
|
39
|
-
| `
|
|
40
|
-
| `
|
|
41
|
-
| `
|
|
42
|
-
|
|
43
|
-
|
|
39
|
+
| `messages` | Yes | The conversation so far, in OpenAI's message format: each message has a `role` and `content`. | <sub></sub> |
|
|
40
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
41
|
+
| `stream` | No | `true` sends the response as server-sent events as it progresses; `false` sends one response once complete. Defaults to `false`. | <sub></sub> |
|
|
42
|
+
| `stop` | No | Sequences that stop generation, as a string or an array of strings. The returned text leaves them out. | <sub></sub> |
|
|
43
|
+
| `logprobs` | No | `true` returns the log probability of each output token. Defaults to `false`. | <sub></sub> |
|
|
44
|
+
| `temperature` | No | Sampling temperature. | <sub></sub> |
|
|
45
|
+
| `repeat_penalty` | No | From `1.0` to `2.0`: `1.0` applies no penalty, and higher values discourage repetition. | <sub></sub> |
|
|
46
|
+
| `frequency_penalty` | No | From `-2.0` to `2.0`: positive values discourage tokens in proportion to how often they have appeared. | <sub></sub> |
|
|
47
|
+
| `seed` | No | Random seed for reproducible sampling. | <sub></sub> |
|
|
48
|
+
| `top_k` | No | Number of top tokens to consider during sampling. | <sub></sub> |
|
|
49
|
+
| `top_p` | No | Cumulative probability of the top tokens to consider during nucleus sampling, from `0.0` to `1.0`. | <sub></sub> |
|
|
50
|
+
| `tools` | No | A list of tools the model may call. | <sub></sub> |
|
|
51
|
+
| `tool_choice` | No | Which tool the model must call, in OpenAI's format: `auto`, `none` or `required`. vLLM and cloud models also take an object naming a function, which llama.cpp models treat as `auto`. | <sub></sub> |
|
|
52
|
+
| `response_format` | No | The format the model must output, in OpenAI's format, such as `{"type": "json_object"}`. | <sub></sub> |
|
|
53
|
+
| `max_tokens` | No | Upper bound on generated tokens. | <sub></sub> |
|
|
54
|
+
| `max_completion_tokens` | No | OpenAI's name for `max_tokens`. Send one of the two: when both are sent, the backend picks, and llama.cpp uses `max_tokens` while vLLM uses `max_completion_tokens`. | <sub></sub> |
|
|
55
|
+
| `enable_thinking` | No | Lemonade extension: `false` turns off reasoning on models that think. `thinking: false` does the same, and `enable_thinking` wins when both are sent. | <sub></sub> |
|
|
56
|
+
| `route_trace` | No | Lemonade extension for `collection.router` models: `true` adds the per-condition `trace` to `x_lemonade_route`. Defaults to `false`. | <sub></sub> |
|
|
57
|
+
| `ctx_size` | No | Lemonade extension: context size to load the model with, when this request loads it. Ignored when the model is already loaded. | <sub></sub> |
|
|
58
|
+
|
|
59
|
+
Naming a `collection.router` model routes the request to one of its candidates; see [Router API](./router.md).
|
|
60
|
+
|
|
61
|
+
### Server-Side Tools
|
|
62
|
+
|
|
63
|
+
Omni collection orchestration is a Lemonade extension to the Chat Completions API; requests to other models are unaffected.
|
|
64
|
+
|
|
65
|
+
When `model` names an Omni **collection** model (`recipe: "collection.omni"`, e.g., `LMX-Omni-52B-Halo`), the route runs an internal tool-calling loop instead of a plain completion. The server injects the reference system prompt and tools, routes to the collection's chat component, executes the omni tools (image generation/editing, text-to-speech) against the matching components, and returns one OpenAI-compatible response. Generated media is embedded in the assistant `content`:
|
|
66
|
+
|
|
67
|
+
- **images** → markdown ``
|
|
68
|
+
- **speech** → `<audio>data:audio/mpeg;base64,…</audio>`
|
|
69
|
+
|
|
70
|
+
Both non-streaming and `stream: true` are supported. In streaming mode the media arrives in `content` deltas of at most 16,000 characters each, the moment its tool finishes, and an error arrives as `Error: <message>` text in `content` followed by `finish_reason: "stop"`.
|
|
71
|
+
|
|
72
|
+
**Merge semantics.** The built-in omni system prompt goes before a client-provided one. Client-provided `tools` are merged with the built-in omni tools. The server resolves omni tool calls internally; calls to client-provided tools are returned in a `finish_reason: "tool_calls"` response for the client to execute and resume. Targeting a collection name invokes the server-side loop; targeting a component LLM name bypasses it and returns a plain completion. See [Lemonade Omni Models](../dev/lemonade-omni.md) for details.
|
|
73
|
+
|
|
74
|
+
### Examples
|
|
75
|
+
|
|
76
|
+
#### Non-streaming
|
|
44
77
|
|
|
45
78
|
=== "PowerShell"
|
|
46
79
|
|
|
@@ -50,243 +83,581 @@ Chat Completions API. You provide a list of messages and receive a completion. T
|
|
|
50
83
|
-Method POST `
|
|
51
84
|
-Headers @{ "Content-Type" = "application/json" } `
|
|
52
85
|
-Body '{
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
"content": "What is the population of Paris?"
|
|
58
|
-
}
|
|
59
|
-
],
|
|
60
|
-
"stream": false
|
|
61
|
-
}'
|
|
86
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
87
|
+
"messages": [{"role": "user", "content": "What is the population of Paris?"}],
|
|
88
|
+
"stream": false
|
|
89
|
+
}'
|
|
62
90
|
```
|
|
91
|
+
|
|
63
92
|
=== "Bash"
|
|
64
93
|
|
|
65
94
|
```bash
|
|
66
|
-
curl
|
|
95
|
+
curl http://localhost:13305/v1/chat/completions \
|
|
67
96
|
-H "Content-Type: application/json" \
|
|
68
97
|
-d '{
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
"stream": false
|
|
74
|
-
}'
|
|
98
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
99
|
+
"messages": [{"role": "user", "content": "What is the population of Paris?"}],
|
|
100
|
+
"stream": false
|
|
101
|
+
}'
|
|
75
102
|
```
|
|
76
103
|
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
=== "Non-streaming responses"
|
|
104
|
+
**Response:** `200 OK`
|
|
80
105
|
|
|
81
|
-
|
|
106
|
+
```json
|
|
107
|
+
{
|
|
108
|
+
"id": "chatcmpl-cJjl2vW81HgtjvknoAdqxCGKeEtNOG4u",
|
|
109
|
+
"object": "chat.completion",
|
|
110
|
+
"created": 1793827645,
|
|
111
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
112
|
+
"system_fingerprint": "b12109-13ec8f614",
|
|
113
|
+
"choices": [
|
|
82
114
|
{
|
|
83
|
-
"
|
|
84
|
-
"
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
"role": "assistant",
|
|
91
|
-
"content": "Paris has a population of approximately 2.2 million people in the city proper."
|
|
92
|
-
},
|
|
93
|
-
"finish_reason": "stop"
|
|
94
|
-
}]
|
|
115
|
+
"index": 0,
|
|
116
|
+
"message": {
|
|
117
|
+
"role": "assistant",
|
|
118
|
+
"content": "The population of Paris, France, is approximately **1,215,000** as of the latest available data. This number reflects the city's significant size and historical importance as the capital of France.",
|
|
119
|
+
"reasoning_content": "Okay, so I need to find out the population of Paris. Let me think... I remember ... (1,176 characters)"
|
|
120
|
+
},
|
|
121
|
+
"finish_reason": "stop"
|
|
95
122
|
}
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
123
|
+
],
|
|
124
|
+
"usage": {
|
|
125
|
+
"prompt_tokens": 15,
|
|
126
|
+
"completion_tokens": 364,
|
|
127
|
+
"total_tokens": 379,
|
|
128
|
+
"prompt_tokens_details": {"cached_tokens": 0}
|
|
129
|
+
},
|
|
130
|
+
"timings": {
|
|
131
|
+
"cache_n": 0,
|
|
132
|
+
"predicted_ms": 3151.928,
|
|
133
|
+
"predicted_n": 364,
|
|
134
|
+
"predicted_per_second": 115.16760535139127,
|
|
135
|
+
"predicted_per_token_ms": 8.682997245179063,
|
|
136
|
+
"prompt_ms": 220.395,
|
|
137
|
+
"prompt_n": 15,
|
|
138
|
+
"prompt_per_second": 68.05962022731913,
|
|
139
|
+
"prompt_per_token_ms": 14.693000000000001
|
|
140
|
+
}
|
|
141
|
+
}
|
|
142
|
+
```
|
|
99
143
|
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
"
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
"
|
|
111
|
-
|
|
112
|
-
|
|
113
|
-
|
|
144
|
+
#### Streaming
|
|
145
|
+
|
|
146
|
+
=== "PowerShell"
|
|
147
|
+
|
|
148
|
+
```powershell
|
|
149
|
+
Invoke-WebRequest `
|
|
150
|
+
-Uri "http://localhost:13305/v1/chat/completions" `
|
|
151
|
+
-Method POST `
|
|
152
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
153
|
+
-Body '{
|
|
154
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
155
|
+
"messages": [{"role": "user", "content": "What is the population of Paris?"}],
|
|
156
|
+
"stream": true
|
|
157
|
+
}'
|
|
114
158
|
```
|
|
115
159
|
|
|
116
|
-
|
|
160
|
+
=== "Bash"
|
|
117
161
|
|
|
118
|
-
|
|
162
|
+
```bash
|
|
163
|
+
curl http://localhost:13305/v1/chat/completions \
|
|
164
|
+
-H "Content-Type: application/json" \
|
|
165
|
+
-d '{
|
|
166
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
167
|
+
"messages": [{"role": "user", "content": "What is the population of Paris?"}],
|
|
168
|
+
"stream": true
|
|
169
|
+
}'
|
|
170
|
+
```
|
|
119
171
|
|
|
120
|
-
|
|
172
|
+
**Response:** `200 OK`
|
|
121
173
|
|
|
122
|
-
```
|
|
123
|
-
|
|
124
|
-
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
"role": "user",
|
|
130
|
-
"content": [
|
|
131
|
-
{"type": "text", "text": "What is in this image?"},
|
|
132
|
-
{"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD..."}}
|
|
133
|
-
]
|
|
134
|
-
}
|
|
135
|
-
],
|
|
136
|
-
"stream": false
|
|
137
|
-
}'
|
|
174
|
+
```text
|
|
175
|
+
data: {"id": "chatcmpl-NoEWRv1s5czlHNYL4Vp5FhTw91Ou8XM1", "object": "chat.completion.chunk", "created": 1793827645, "model": "~/.cache/huggingface/hub/models--unsloth--Qwen3-0.6B-GGUF/snapshots/50968a4468ef4233ed78cd7c3de230dd1d61a56b/Qwen3-0.6B-Q4_0.gguf", "system_fingerprint": "b12109-13ec8f614", "choices": [{"index": 0, "delta": {"role": "assistant", "content": null}, "finish_reason": null}]}
|
|
176
|
+
data: {"id": "chatcmpl-NoEWRv1s5czlHNYL4Vp5FhTw91Ou8XM1", "object": "chat.completion.chunk", "created": 1793827645, "model": "~/.cache/huggingface/hub/models--unsloth--Qwen3-0.6B-GGUF/snapshots/50968a4468ef4233ed78cd7c3de230dd1d61a56b/Qwen3-0.6B-Q4_0.gguf", "system_fingerprint": "b12109-13ec8f614", "choices": [{"index": 0, "delta": {"reasoning_content": "Okay"}, "finish_reason": null}]}
|
|
177
|
+
data: {"id": "chatcmpl-NoEWRv1s5czlHNYL4Vp5FhTw91Ou8XM1", "object": "chat.completion.chunk", "created": 1793827645, "model": "~/.cache/huggingface/hub/models--unsloth--Qwen3-0.6B-GGUF/snapshots/50968a4468ef4233ed78cd7c3de230dd1d61a56b/Qwen3-0.6B-Q4_0.gguf", "system_fingerprint": "b12109-13ec8f614", "choices": [{"index": 0, "delta": {"reasoning_content": ","}, "finish_reason": null}]}
|
|
178
|
+
...
|
|
179
|
+
data: {"id": "chatcmpl-NoEWRv1s5czlHNYL4Vp5FhTw91Ou8XM1", "object": "chat.completion.chunk", "created": 1793827647, "model": "~/.cache/huggingface/hub/models--unsloth--Qwen3-0.6B-GGUF/snapshots/50968a4468ef4233ed78cd7c3de230dd1d61a56b/Qwen3-0.6B-Q4_0.gguf", "system_fingerprint": "b12109-13ec8f614", "choices": [{"index": 0, "delta": {}, "finish_reason": "stop"}], "timings": {"cache_n": 14, "prompt_n": 1, "prompt_ms": 13.334, "prompt_per_token_ms": 13.334, "prompt_per_second": 74.99625018749063, "predicted_n": 179, "predicted_ms": 1827.578, "predicted_per_token_ms": 10.267292134831461, "predicted_per_second": 97.39666378124491}}
|
|
180
|
+
data: [DONE]
|
|
138
181
|
```
|
|
139
182
|
|
|
140
|
-
####
|
|
183
|
+
#### Image input
|
|
184
|
+
|
|
185
|
+
Asks a vision model what `image.png` shows. A `messages[*].content` array mixes `text` and `image_url` items, with each image as a base64 data URL, such as one from `FileReader.readAsDataURL(...)` in a web app.
|
|
186
|
+
|
|
187
|
+
=== "PowerShell"
|
|
188
|
+
|
|
189
|
+
```powershell
|
|
190
|
+
$image = [Convert]::ToBase64String([IO.File]::ReadAllBytes("image.png"))
|
|
191
|
+
$body = @"
|
|
192
|
+
{
|
|
193
|
+
"model": "Qwen3.5-0.8B-GGUF",
|
|
194
|
+
"messages": [
|
|
195
|
+
{
|
|
196
|
+
"role": "user",
|
|
197
|
+
"content": [
|
|
198
|
+
{"type": "text", "text": "What is in this image?"},
|
|
199
|
+
{
|
|
200
|
+
"type": "image_url",
|
|
201
|
+
"image_url": {"url": "data:image/png;base64,$image"}
|
|
202
|
+
}
|
|
203
|
+
]
|
|
204
|
+
}
|
|
205
|
+
],
|
|
206
|
+
"stream": false
|
|
207
|
+
}
|
|
208
|
+
"@
|
|
209
|
+
Invoke-WebRequest `
|
|
210
|
+
-Uri "http://localhost:13305/v1/chat/completions" `
|
|
211
|
+
-Method POST `
|
|
212
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
213
|
+
-Body $body
|
|
214
|
+
```
|
|
215
|
+
|
|
216
|
+
=== "Bash"
|
|
217
|
+
|
|
218
|
+
```bash
|
|
219
|
+
curl http://localhost:13305/v1/chat/completions \
|
|
220
|
+
-H "Content-Type: application/json" \
|
|
221
|
+
-d "{
|
|
222
|
+
\"model\": \"Qwen3.5-0.8B-GGUF\",
|
|
223
|
+
\"messages\": [
|
|
224
|
+
{
|
|
225
|
+
\"role\": \"user\",
|
|
226
|
+
\"content\": [
|
|
227
|
+
{\"type\": \"text\", \"text\": \"What is in this image?\"},
|
|
228
|
+
{
|
|
229
|
+
\"type\": \"image_url\",
|
|
230
|
+
\"image_url\": {\"url\": \"data:image/png;base64,$(base64 -w0 image.png)\"}
|
|
231
|
+
}
|
|
232
|
+
]
|
|
233
|
+
}
|
|
234
|
+
],
|
|
235
|
+
\"stream\": false
|
|
236
|
+
}"
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
**Response:** `200 OK`
|
|
141
240
|
|
|
142
241
|
```json
|
|
143
242
|
{
|
|
144
|
-
"id": "
|
|
243
|
+
"id": "chatcmpl-bykV6FdzHGGdiC5ePf9PBFOxRu0AFTBi",
|
|
145
244
|
"object": "chat.completion",
|
|
146
|
-
"created":
|
|
147
|
-
"model": "
|
|
148
|
-
"
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
"
|
|
152
|
-
"
|
|
153
|
-
|
|
154
|
-
|
|
155
|
-
|
|
245
|
+
"created": 1793827657,
|
|
246
|
+
"model": "Qwen3.5-0.8B-GGUF",
|
|
247
|
+
"system_fingerprint": "b12109-13ec8f614",
|
|
248
|
+
"choices": [
|
|
249
|
+
{
|
|
250
|
+
"index": 0,
|
|
251
|
+
"message": {
|
|
252
|
+
"role": "assistant",
|
|
253
|
+
"content": "This is a logo image featuring a solid red circle centered on a gradient background. The background transitions from a teal and green mix on the left to a grey on the right side.",
|
|
254
|
+
"reasoning_content": "The user wants me to identify the visual content of the provided image.\n\n1. **A... (1,535 characters)"
|
|
255
|
+
},
|
|
256
|
+
"finish_reason": "stop"
|
|
257
|
+
}
|
|
258
|
+
],
|
|
259
|
+
"usage": {
|
|
260
|
+
"prompt_tokens": 82,
|
|
261
|
+
"completion_tokens": 405,
|
|
262
|
+
"total_tokens": 487,
|
|
263
|
+
"prompt_tokens_details": {"cached_tokens": 0}
|
|
264
|
+
},
|
|
265
|
+
"timings": {
|
|
266
|
+
"cache_n": 0,
|
|
267
|
+
"predicted_ms": 1989.104,
|
|
268
|
+
"predicted_n": 405,
|
|
269
|
+
"predicted_per_second": 203.10652434462952,
|
|
270
|
+
"predicted_per_token_ms": 4.923524752475248,
|
|
271
|
+
"prompt_ms": 465.51,
|
|
272
|
+
"prompt_n": 82,
|
|
273
|
+
"prompt_per_second": 176.15088827307684,
|
|
274
|
+
"prompt_per_token_ms": 5.676951219512195
|
|
275
|
+
}
|
|
156
276
|
}
|
|
157
277
|
```
|
|
158
278
|
|
|
159
|
-
|
|
160
|
-
|
|
161
|
-
> **Note:** Omni collection orchestration is a Lemonade-specific extension to the Chat Completions API; it is not part of the OpenAI specification. Requests that target ordinary (non-collection) models are unaffected and are fully OpenAI-compatible.
|
|
162
|
-
|
|
163
|
-
When `model` names an Omni **collection** model (`recipe: "collection.omni"`, e.g., `LMX-Omni-52B-Halo`), this endpoint runs an internal tool-calling loop instead of a plain completion. The server injects the reference system prompt and tools, routes to the collection's chat component, executes the omni tools (image generation/editing, text-to-speech) against the matching components, and returns one OpenAI-compatible response. Generated media is embedded in the assistant `content`:
|
|
164
|
-
|
|
165
|
-
- **images** → markdown ``
|
|
166
|
-
- **speech** → `<audio>data:audio/mpeg;base64,…</audio>`
|
|
279
|
+
#### Server-side tools
|
|
167
280
|
|
|
168
|
-
|
|
281
|
+
Runs the [server-side tools](#server-side-tools) loop on `LMX-Omni-5.5B-Lite`, whose image tool draws the requested picture.
|
|
169
282
|
|
|
170
|
-
|
|
283
|
+
=== "PowerShell"
|
|
171
284
|
|
|
172
|
-
|
|
285
|
+
```powershell
|
|
286
|
+
Invoke-WebRequest `
|
|
287
|
+
-Uri "http://localhost:13305/v1/chat/completions" `
|
|
288
|
+
-Method POST `
|
|
289
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
290
|
+
-Body '{
|
|
291
|
+
"model": "LMX-Omni-5.5B-Lite",
|
|
292
|
+
"messages": [{"role": "user", "content": "Draw a red apple on a table."}],
|
|
293
|
+
"stream": false
|
|
294
|
+
}'
|
|
295
|
+
```
|
|
173
296
|
|
|
174
|
-
|
|
175
|
-
curl -X POST http://localhost:13305/v1/chat/completions \
|
|
176
|
-
-H "Content-Type: application/json" \
|
|
177
|
-
-d '{
|
|
178
|
-
"model": "LMX-Omni-52B-Halo",
|
|
179
|
-
"messages": [{"role": "user", "content": "Draw a red apple on a table."}],
|
|
180
|
-
"stream": false
|
|
181
|
-
}'
|
|
182
|
-
```
|
|
297
|
+
=== "Bash"
|
|
183
298
|
|
|
184
|
-
|
|
299
|
+
```bash
|
|
300
|
+
curl http://localhost:13305/v1/chat/completions \
|
|
301
|
+
-H "Content-Type: application/json" \
|
|
302
|
+
-d '{
|
|
303
|
+
"model": "LMX-Omni-5.5B-Lite",
|
|
304
|
+
"messages": [{"role": "user", "content": "Draw a red apple on a table."}],
|
|
305
|
+
"stream": false
|
|
306
|
+
}'
|
|
307
|
+
```
|
|
185
308
|
|
|
186
|
-
|
|
309
|
+
**Response:** `200 OK`
|
|
187
310
|
|
|
188
311
|
```json
|
|
189
312
|
{
|
|
190
|
-
"id": "
|
|
313
|
+
"id": "chatcmpl-i37OtXcuKiFJ82bZjOEJmIDgbbrSLIWk",
|
|
191
314
|
"object": "chat.completion",
|
|
192
|
-
"created":
|
|
193
|
-
"model": "LMX-Omni-
|
|
194
|
-
"
|
|
195
|
-
|
|
196
|
-
|
|
197
|
-
"
|
|
198
|
-
"
|
|
199
|
-
|
|
200
|
-
|
|
201
|
-
|
|
315
|
+
"created": 1792076833,
|
|
316
|
+
"model": "LMX-Omni-5.5B-Lite",
|
|
317
|
+
"system_fingerprint": "b11393-dbe4c3ed4",
|
|
318
|
+
"choices": [
|
|
319
|
+
{
|
|
320
|
+
"index": 0,
|
|
321
|
+
"message": {
|
|
322
|
+
"role": "assistant",
|
|
323
|
+
"content": "I've generated an image of a red apple on a table for you! The image shows a cla... (393,258 characters)"
|
|
324
|
+
},
|
|
325
|
+
"finish_reason": "stop"
|
|
326
|
+
}
|
|
327
|
+
],
|
|
328
|
+
"usage": {
|
|
329
|
+
"prompt_tokens": 1102,
|
|
330
|
+
"completion_tokens": 63,
|
|
331
|
+
"total_tokens": 1165,
|
|
332
|
+
"prompt_tokens_details": {"cached_tokens": 1082}
|
|
333
|
+
},
|
|
334
|
+
"timings": {
|
|
335
|
+
"cache_n": 1082,
|
|
336
|
+
"draft_n": 69,
|
|
337
|
+
"draft_n_accepted": 39,
|
|
338
|
+
"predicted_ms": 706.57,
|
|
339
|
+
"predicted_n": 63,
|
|
340
|
+
"predicted_per_second": 87.74785230055055,
|
|
341
|
+
"predicted_per_token_ms": 11.396290322580645,
|
|
342
|
+
"prompt_ms": 97.681,
|
|
343
|
+
"prompt_n": 20,
|
|
344
|
+
"prompt_per_second": 204.74810863934647,
|
|
345
|
+
"prompt_per_token_ms": 4.88405
|
|
346
|
+
}
|
|
202
347
|
}
|
|
203
348
|
```
|
|
204
349
|
|
|
350
|
+
### Response format
|
|
205
351
|
|
|
352
|
+
#### JSON
|
|
353
|
+
|
|
354
|
+
- `id` - Identifier of the completion.
|
|
355
|
+
- `object` - Always `"chat.completion"`.
|
|
356
|
+
- `created` - Unix timestamp of when the completion was created.
|
|
357
|
+
- `model` - The model that generated the response.
|
|
358
|
+
- `system_fingerprint` - Build identifier of the backend, when it reports one.
|
|
359
|
+
- `choices` - The generated messages, one per choice.
|
|
360
|
+
- `index` - Position of the choice in `choices`.
|
|
361
|
+
- `message` - The generated assistant message.
|
|
362
|
+
- `role` - Always `"assistant"`.
|
|
363
|
+
- `content` - The message text.
|
|
364
|
+
- `reasoning_content` - Reasoning models only: the model's thinking.
|
|
365
|
+
- `tool_calls` - Calls to the request's tools.
|
|
366
|
+
- `finish_reason` - Why generation stopped, such as `stop`, `length` or `tool_calls`.
|
|
367
|
+
- `usage` - Token counts for the request.
|
|
368
|
+
- `prompt_tokens` - Tokens in the prompt.
|
|
369
|
+
- `completion_tokens` - Tokens generated.
|
|
370
|
+
- `total_tokens` - Sum of `prompt_tokens` and `completion_tokens`.
|
|
371
|
+
- `prompt_tokens_details` - Breakdown of the prompt tokens, such as `cached_tokens`, when the backend reports it.
|
|
372
|
+
- `timings` - Prompt processing and generation timings, passed through from backends that report them, such as llama.cpp.
|
|
373
|
+
- `x_lemonade_route` - The routing decision, when `model` names a `collection.router` model. See [Routing](./router.md#routing-collectionrouter) for the decision object.
|
|
374
|
+
|
|
375
|
+
#### Event stream
|
|
376
|
+
|
|
377
|
+
One event per `data:` line. The stream ends with `data: [DONE]`.
|
|
378
|
+
|
|
379
|
+
##### Chunk
|
|
380
|
+
|
|
381
|
+
A chunk of the assistant's message.
|
|
382
|
+
|
|
383
|
+
- `id` - Identifier of the completion, the same on every chunk.
|
|
384
|
+
- `object` - Always `"chat.completion.chunk"`.
|
|
385
|
+
- `created` - Unix timestamp of when the completion was created.
|
|
386
|
+
- `model` - The backend's name for the model, such as the GGUF file path for llama.cpp models.
|
|
387
|
+
- `system_fingerprint` - Build identifier of the backend, when it reports one.
|
|
388
|
+
- `choices` - The next part of each choice's message.
|
|
389
|
+
- `index` - Position of the choice in `choices`.
|
|
390
|
+
- `delta` - The part of the assistant message generated since the previous chunk.
|
|
391
|
+
- `role` - First chunk only: always `"assistant"`.
|
|
392
|
+
- `content` - The next message text.
|
|
393
|
+
- `reasoning_content` - Reasoning models only: the next part of the model's thinking.
|
|
394
|
+
- `tool_calls` - Parts of calls to the request's tools, matched by `index`: a call's first part carries its `id` and `function.name`, and later parts carry the next piece of `function.arguments`.
|
|
395
|
+
- `finish_reason` - Why generation stopped, such as `stop`, `length` or `tool_calls`, on the choice's last chunk; `null` before it.
|
|
396
|
+
- `usage` - Token counts for the request, on an extra last chunk with an empty `choices`. Sent when the request sets `stream_options.include_usage`, and always by vLLM models.
|
|
397
|
+
- `timings` - Final chunk only: prompt processing and generation timings, passed through from backends that report them, such as llama.cpp.
|
|
398
|
+
- `x_lemonade_route` - The routing decision, when `model` names a `collection.router` model. See [Routing](./router.md#routing-collectionrouter) for the decision object.
|
|
399
|
+
|
|
400
|
+
##### Error event
|
|
401
|
+
|
|
402
|
+
Sent in place of the rest of the stream when the backend fails after the stream has started. No `[DONE]` follows it.
|
|
403
|
+
|
|
404
|
+
- `error` - The backend's error, as an object or, when the backend sent one, a string.
|
|
405
|
+
- `message` - What failed.
|
|
406
|
+
- `type` - The error's type, such as `backend_error`.
|
|
407
|
+
- `status_code` - The status the backend answered with, or `502` when it closed the stream without a response.
|
|
408
|
+
|
|
409
|
+
### Errors
|
|
410
|
+
|
|
411
|
+
| Status | Code | When |
|
|
412
|
+
|--------|------|------|
|
|
413
|
+
| `400` | `invalid_json` | The body is not a JSON object |
|
|
414
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
415
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
416
|
+
| `400` | | `model` is not an LLM, which is checked after it loads |
|
|
417
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
418
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
419
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
420
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
421
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
422
|
+
| `500` | | The backend reports an error; the backend's own status replaces this one when it sends one |
|
|
423
|
+
| `500` | | The request fails unexpectedly |
|
|
424
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
425
|
+
<!-- END GENERATED: openai.chat_completions -->
|
|
426
|
+
|
|
427
|
+
<!-- BEGIN GENERATED: openai.completions -->
|
|
206
428
|
## `POST /v1/completions`
|
|
207
|
-
<sub></sub>
|
|
430
|
+
|
|
431
|
+
Continues a text prompt, downloading and loading the model if it is not loaded.
|
|
208
432
|
|
|
209
|
-
|
|
433
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
210
434
|
|
|
211
435
|
### Parameters
|
|
212
436
|
|
|
213
437
|
| Parameter | Required | Description | Status |
|
|
214
438
|
|-----------|----------|-------------|--------|
|
|
215
|
-
| `prompt` | Yes | The
|
|
216
|
-
| `model` | Yes | The model to
|
|
217
|
-
| `stream` | No |
|
|
218
|
-
| `stop` | No |
|
|
219
|
-
| `echo` | No |
|
|
220
|
-
| `logprobs` | No |
|
|
221
|
-
| `temperature` | No |
|
|
222
|
-
| `repeat_penalty` | No |
|
|
223
|
-
| `
|
|
224
|
-
| `
|
|
225
|
-
| `
|
|
226
|
-
|
|
227
|
-
|
|
439
|
+
| `prompt` | Yes | The text to continue, as a string or an array of strings. | <sub></sub> |
|
|
440
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
441
|
+
| `stream` | No | `true` sends the response as server-sent events as it progresses; `false` sends one response once complete. Defaults to `false`. | <sub></sub> |
|
|
442
|
+
| `stop` | No | Sequences that stop generation, as a string or an array of strings. The returned text leaves them out. | <sub></sub> |
|
|
443
|
+
| `echo` | No | `true` asks the backend to return the prompt before the completion. llama.cpp models ignore it. | <sub></sub> |
|
|
444
|
+
| `logprobs` | No | `true`, or a number of top alternatives per token, returns the log probability of each output token. Defaults to `false`. | <sub></sub> |
|
|
445
|
+
| `temperature` | No | Sampling temperature. | <sub></sub> |
|
|
446
|
+
| `repeat_penalty` | No | From `1.0` to `2.0`: `1.0` applies no penalty, and higher values discourage repetition. | <sub></sub> |
|
|
447
|
+
| `frequency_penalty` | No | From `-2.0` to `2.0`: positive values discourage tokens in proportion to how often they have appeared. | <sub></sub> |
|
|
448
|
+
| `seed` | No | Random seed for reproducible sampling. | <sub></sub> |
|
|
449
|
+
| `top_k` | No | Number of top tokens to consider during sampling. | <sub></sub> |
|
|
450
|
+
| `top_p` | No | Cumulative probability of the top tokens to consider during nucleus sampling, from `0.0` to `1.0`. | <sub></sub> |
|
|
451
|
+
| `max_tokens` | No | Upper bound on generated tokens. | <sub></sub> |
|
|
452
|
+
| `route_trace` | No | Lemonade extension for `collection.router` models: `true` adds the per-condition `trace` to `x_lemonade_route`. Defaults to `false`. | <sub></sub> |
|
|
453
|
+
| `ctx_size` | No | Lemonade extension: context size to load the model with, when this request loads it. Ignored when the model is already loaded. | <sub></sub> |
|
|
454
|
+
|
|
455
|
+
Naming a `collection.router` model routes the request to one of its candidates; see [Router API](./router.md).
|
|
456
|
+
|
|
457
|
+
### Examples
|
|
458
|
+
|
|
459
|
+
#### Non-streaming
|
|
228
460
|
|
|
229
461
|
=== "PowerShell"
|
|
230
462
|
|
|
231
463
|
```powershell
|
|
232
|
-
Invoke-WebRequest
|
|
464
|
+
Invoke-WebRequest `
|
|
465
|
+
-Uri "http://localhost:13305/v1/completions" `
|
|
233
466
|
-Method POST `
|
|
234
467
|
-Headers @{ "Content-Type" = "application/json" } `
|
|
235
468
|
-Body '{
|
|
236
|
-
|
|
237
|
-
|
|
238
|
-
|
|
239
|
-
|
|
469
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
470
|
+
"prompt": "What is the population of Paris?",
|
|
471
|
+
"stream": false,
|
|
472
|
+
"max_tokens": 64
|
|
473
|
+
}'
|
|
240
474
|
```
|
|
241
475
|
|
|
242
476
|
=== "Bash"
|
|
243
477
|
|
|
244
478
|
```bash
|
|
245
|
-
curl
|
|
479
|
+
curl http://localhost:13305/v1/completions \
|
|
246
480
|
-H "Content-Type: application/json" \
|
|
247
481
|
-d '{
|
|
248
|
-
|
|
249
|
-
|
|
250
|
-
|
|
251
|
-
|
|
482
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
483
|
+
"prompt": "What is the population of Paris?",
|
|
484
|
+
"stream": false,
|
|
485
|
+
"max_tokens": 64
|
|
486
|
+
}'
|
|
252
487
|
```
|
|
253
488
|
|
|
254
|
-
|
|
255
|
-
|
|
256
|
-
The following format is used for both streaming and non-streaming responses:
|
|
489
|
+
**Response:** `200 OK`
|
|
257
490
|
|
|
258
491
|
```json
|
|
259
492
|
{
|
|
260
|
-
"id": "
|
|
493
|
+
"id": "chatcmpl-yRFRUAekD8zEgLPHmtMKr50iJa1iKKUU",
|
|
261
494
|
"object": "text_completion",
|
|
262
|
-
"created":
|
|
495
|
+
"created": 1793827667,
|
|
263
496
|
"model": "Qwen3-0.6B-GGUF",
|
|
264
|
-
"
|
|
265
|
-
|
|
266
|
-
|
|
267
|
-
|
|
268
|
-
|
|
497
|
+
"system_fingerprint": "b12109-13ec8f614",
|
|
498
|
+
"choices": [
|
|
499
|
+
{
|
|
500
|
+
"index": 0,
|
|
501
|
+
"text": " And what are the characteristics of Paris? Also, what is the capital of France? What is the capital of Italy? What is the capital of Spain? What is the capital of Nigeria? And what is the capital of India?\n\nOkay, let's tackle these questions. First, the population of Paris. I remember that",
|
|
502
|
+
"logprobs": null,
|
|
503
|
+
"finish_reason": "length"
|
|
504
|
+
}
|
|
505
|
+
],
|
|
506
|
+
"usage": {
|
|
507
|
+
"prompt_tokens": 7,
|
|
508
|
+
"completion_tokens": 64,
|
|
509
|
+
"total_tokens": 71,
|
|
510
|
+
"prompt_tokens_details": {"cached_tokens": 0}
|
|
511
|
+
},
|
|
512
|
+
"timings": {
|
|
513
|
+
"cache_n": 0,
|
|
514
|
+
"predicted_ms": 3637.764,
|
|
515
|
+
"predicted_n": 64,
|
|
516
|
+
"predicted_per_second": 17.318330710843252,
|
|
517
|
+
"predicted_per_token_ms": 57.742285714285714,
|
|
518
|
+
"prompt_ms": 133.468,
|
|
519
|
+
"prompt_n": 7,
|
|
520
|
+
"prompt_per_second": 52.44702850121378,
|
|
521
|
+
"prompt_per_token_ms": 19.066857142857142
|
|
522
|
+
}
|
|
269
523
|
}
|
|
270
524
|
```
|
|
271
525
|
|
|
526
|
+
#### Streaming
|
|
527
|
+
|
|
528
|
+
=== "PowerShell"
|
|
529
|
+
|
|
530
|
+
```powershell
|
|
531
|
+
Invoke-WebRequest `
|
|
532
|
+
-Uri "http://localhost:13305/v1/completions" `
|
|
533
|
+
-Method POST `
|
|
534
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
535
|
+
-Body '{
|
|
536
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
537
|
+
"prompt": "What is the population of Paris?",
|
|
538
|
+
"stream": true,
|
|
539
|
+
"max_tokens": 64
|
|
540
|
+
}'
|
|
541
|
+
```
|
|
542
|
+
|
|
543
|
+
=== "Bash"
|
|
544
|
+
|
|
545
|
+
```bash
|
|
546
|
+
curl http://localhost:13305/v1/completions \
|
|
547
|
+
-H "Content-Type: application/json" \
|
|
548
|
+
-d '{
|
|
549
|
+
"model": "Qwen3-0.6B-GGUF",
|
|
550
|
+
"prompt": "What is the population of Paris?",
|
|
551
|
+
"stream": true,
|
|
552
|
+
"max_tokens": 64
|
|
553
|
+
}'
|
|
554
|
+
```
|
|
555
|
+
|
|
556
|
+
**Response:** `200 OK`
|
|
557
|
+
|
|
558
|
+
```text
|
|
559
|
+
data: {"id": "chatcmpl-NnGAa2fcrBou11qX4PgyeQCgXNJB9W6j", "object": "text_completion", "created": 1793827667, "model": "~/.cache/huggingface/hub/models--unsloth--Qwen3-0.6B-GGUF/snapshots/50968a4468ef4233ed78cd7c3de230dd1d61a56b/Qwen3-0.6B-Q4_0.gguf", "system_fingerprint": "b12109-13ec8f614", "choices": [{"index": 0, "text": " And", "logprobs": null, "finish_reason": null}]}
|
|
560
|
+
data: {"id": "chatcmpl-NnGAa2fcrBou11qX4PgyeQCgXNJB9W6j", "object": "text_completion", "created": 1793827667, "model": "~/.cache/huggingface/hub/models--unsloth--Qwen3-0.6B-GGUF/snapshots/50968a4468ef4233ed78cd7c3de230dd1d61a56b/Qwen3-0.6B-Q4_0.gguf", "system_fingerprint": "b12109-13ec8f614", "choices": [{"index": 0, "text": " how", "logprobs": null, "finish_reason": null}]}
|
|
561
|
+
data: {"id": "chatcmpl-NnGAa2fcrBou11qX4PgyeQCgXNJB9W6j", "object": "text_completion", "created": 1793827667, "model": "~/.cache/huggingface/hub/models--unsloth--Qwen3-0.6B-GGUF/snapshots/50968a4468ef4233ed78cd7c3de230dd1d61a56b/Qwen3-0.6B-Q4_0.gguf", "system_fingerprint": "b12109-13ec8f614", "choices": [{"index": 0, "text": " is", "logprobs": null, "finish_reason": null}]}
|
|
562
|
+
...
|
|
563
|
+
data: {"id": "chatcmpl-NnGAa2fcrBou11qX4PgyeQCgXNJB9W6j", "object": "text_completion", "created": 1793827670, "model": "~/.cache/huggingface/hub/models--unsloth--Qwen3-0.6B-GGUF/snapshots/50968a4468ef4233ed78cd7c3de230dd1d61a56b/Qwen3-0.6B-Q4_0.gguf", "system_fingerprint": "b12109-13ec8f614", "choices": [{"index": 0, "text": "", "logprobs": null, "finish_reason": "length"}], "usage": {"prompt_tokens": 7, "completion_tokens": 64, "total_tokens": 71, "prompt_tokens_details": {"cached_tokens": 6}}, "timings": {"cache_n": 6, "prompt_n": 1, "prompt_ms": 5.813, "prompt_per_token_ms": 5.813, "prompt_per_second": 172.0282126268708, "predicted_n": 64, "predicted_ms": 3611.246, "predicted_per_token_ms": 57.32136507936508, "predicted_per_second": 17.445502189548982}}
|
|
564
|
+
data: [DONE]
|
|
565
|
+
```
|
|
272
566
|
|
|
567
|
+
### Response format
|
|
273
568
|
|
|
569
|
+
#### JSON
|
|
570
|
+
|
|
571
|
+
- `id` - Identifier of the completion.
|
|
572
|
+
- `object` - Always `"text_completion"`.
|
|
573
|
+
- `created` - Unix timestamp of when the completion was created.
|
|
574
|
+
- `model` - The model that generated the response.
|
|
575
|
+
- `system_fingerprint` - Build identifier of the backend, when it reports one.
|
|
576
|
+
- `choices` - The generated text, one entry per choice.
|
|
577
|
+
- `index` - Position of the choice in `choices`.
|
|
578
|
+
- `text` - The generated text.
|
|
579
|
+
- `logprobs` - Log probabilities of the output tokens when the request sets `logprobs`, else `null`.
|
|
580
|
+
- `finish_reason` - Why generation stopped, such as `stop` or `length`.
|
|
581
|
+
- `usage` - Token counts for the request.
|
|
582
|
+
- `prompt_tokens` - Tokens in the prompt.
|
|
583
|
+
- `completion_tokens` - Tokens generated.
|
|
584
|
+
- `total_tokens` - Sum of `prompt_tokens` and `completion_tokens`.
|
|
585
|
+
- `prompt_tokens_details` - Breakdown of the prompt tokens, such as `cached_tokens`, when the backend reports it.
|
|
586
|
+
- `timings` - Prompt processing and generation timings, passed through from backends that report them, such as llama.cpp.
|
|
587
|
+
- `x_lemonade_route` - The routing decision, when `model` names a `collection.router` model. See [Routing](./router.md#routing-collectionrouter) for the decision object.
|
|
588
|
+
|
|
589
|
+
#### Event stream
|
|
590
|
+
|
|
591
|
+
One event per `data:` line. The stream ends with `data: [DONE]`.
|
|
592
|
+
|
|
593
|
+
##### Chunk
|
|
594
|
+
|
|
595
|
+
A chunk of the completion text.
|
|
596
|
+
|
|
597
|
+
- `id` - Identifier of the completion, the same on every chunk.
|
|
598
|
+
- `object` - Always `"text_completion"`.
|
|
599
|
+
- `created` - Unix timestamp of when the completion was created.
|
|
600
|
+
- `model` - The backend's name for the model, such as the GGUF file path for llama.cpp models.
|
|
601
|
+
- `system_fingerprint` - Build identifier of the backend, when it reports one.
|
|
602
|
+
- `choices` - The next part of each choice's text.
|
|
603
|
+
- `index` - Position of the choice in `choices`.
|
|
604
|
+
- `text` - The text generated since the previous chunk.
|
|
605
|
+
- `logprobs` - Log probabilities of the output tokens when the request sets `logprobs`, else `null`.
|
|
606
|
+
- `finish_reason` - Why generation stopped, such as `stop` or `length`, on the choice's last chunk; `null` before it.
|
|
607
|
+
- `usage` - Final chunk only: token counts for the request.
|
|
608
|
+
- `prompt_tokens` - Tokens in the prompt.
|
|
609
|
+
- `completion_tokens` - Tokens generated.
|
|
610
|
+
- `total_tokens` - Sum of `prompt_tokens` and `completion_tokens`.
|
|
611
|
+
- `prompt_tokens_details` - Breakdown of the prompt tokens, such as `cached_tokens`, when the backend reports it.
|
|
612
|
+
- `timings` - Final chunk only: prompt processing and generation timings, passed through from backends that report them, such as llama.cpp.
|
|
613
|
+
- `x_lemonade_route` - The routing decision, when `model` names a `collection.router` model. See [Routing](./router.md#routing-collectionrouter) for the decision object.
|
|
614
|
+
|
|
615
|
+
##### Error event
|
|
616
|
+
|
|
617
|
+
Sent in place of the rest of the stream when the backend fails after the stream has started. No `[DONE]` follows it.
|
|
618
|
+
|
|
619
|
+
- `error` - The backend's error, as an object or, when the backend sent one, a string.
|
|
620
|
+
- `message` - What failed.
|
|
621
|
+
- `type` - The error's type, such as `backend_error`.
|
|
622
|
+
- `status_code` - The status the backend answered with, or `502` when it closed the stream without a response.
|
|
623
|
+
|
|
624
|
+
### Errors
|
|
625
|
+
|
|
626
|
+
| Status | Code | When |
|
|
627
|
+
|--------|------|------|
|
|
628
|
+
| `400` | `invalid_json` | The body is not a JSON object |
|
|
629
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
630
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
631
|
+
| `400` | | `model` is not an LLM, which is checked after it loads |
|
|
632
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
633
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
634
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
635
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
636
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
637
|
+
| `500` | | The backend reports an error; the backend's own status replaces this one when it sends one |
|
|
638
|
+
| `500` | | The backend's response lacks a field the route returns, such as `choices` or `labels` |
|
|
639
|
+
| `500` | | The request fails unexpectedly |
|
|
640
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
641
|
+
<!-- END GENERATED: openai.completions -->
|
|
642
|
+
|
|
643
|
+
<!-- BEGIN GENERATED: openai.embeddings -->
|
|
274
644
|
## `POST /v1/embeddings`
|
|
275
645
|
<sub></sub>
|
|
276
646
|
|
|
277
|
-
|
|
647
|
+
Returns vector representations (embeddings) of the input text for semantic search, clustering and similarity comparisons, downloading and loading the model if it is not loaded. Embedding models use the `llamacpp` or `flm` recipe.
|
|
278
648
|
|
|
279
|
-
|
|
649
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
280
650
|
|
|
281
651
|
### Parameters
|
|
282
652
|
|
|
283
653
|
| Parameter | Required | Description | Status |
|
|
284
654
|
|-----------|----------|-------------|--------|
|
|
285
|
-
| `input` | Yes | The
|
|
286
|
-
| `model` | Yes | The model to
|
|
287
|
-
| `encoding_format` | No |
|
|
655
|
+
| `input` | Yes | The text to embed, as a string or an array of strings. | <sub></sub> |
|
|
656
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
657
|
+
| `encoding_format` | No | `float` or `base64`. Defaults to `float`. | <sub></sub> |
|
|
658
|
+
| `ctx_size` | No | Lemonade extension: context size to load the model with, when this request loads it. Ignored when the model is already loaded. | <sub></sub> |
|
|
288
659
|
|
|
289
|
-
### Example
|
|
660
|
+
### Example
|
|
290
661
|
|
|
291
662
|
=== "PowerShell"
|
|
292
663
|
|
|
@@ -296,25 +667,25 @@ Embeddings API. You provide input text and receive vector representations (embed
|
|
|
296
667
|
-Method POST `
|
|
297
668
|
-Headers @{ "Content-Type" = "application/json" } `
|
|
298
669
|
-Body '{
|
|
299
|
-
|
|
300
|
-
|
|
301
|
-
|
|
302
|
-
|
|
670
|
+
"model": "nomic-embed-text-v1-GGUF",
|
|
671
|
+
"input": ["Hello, world!", "How are you?"],
|
|
672
|
+
"encoding_format": "float"
|
|
673
|
+
}'
|
|
303
674
|
```
|
|
304
675
|
|
|
305
676
|
=== "Bash"
|
|
306
677
|
|
|
307
678
|
```bash
|
|
308
|
-
curl
|
|
679
|
+
curl http://localhost:13305/v1/embeddings \
|
|
309
680
|
-H "Content-Type: application/json" \
|
|
310
681
|
-d '{
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
682
|
+
"model": "nomic-embed-text-v1-GGUF",
|
|
683
|
+
"input": ["Hello, world!", "How are you?"],
|
|
684
|
+
"encoding_format": "float"
|
|
685
|
+
}'
|
|
315
686
|
```
|
|
316
687
|
|
|
317
|
-
|
|
688
|
+
**Response:** `200 OK`
|
|
318
689
|
|
|
319
690
|
```json
|
|
320
691
|
{
|
|
@@ -323,184 +694,355 @@ Embeddings API. You provide input text and receive vector representations (embed
|
|
|
323
694
|
{
|
|
324
695
|
"object": "embedding",
|
|
325
696
|
"index": 0,
|
|
326
|
-
"embedding": [0.
|
|
697
|
+
"embedding": [0.03605353832244873, 0.0006784393335692585, 0.004262526985257864, -0.02508561499416828, -0.028299998492002487, 0.030086658895015717, -0.012136111967265606, 0.01537393033504486, ... 760 more]
|
|
327
698
|
},
|
|
328
|
-
|
|
329
|
-
"object": "embedding",
|
|
330
|
-
"index": 1,
|
|
331
|
-
"embedding": [0.0456, -0.0678, 0.1234, ...]
|
|
332
|
-
}
|
|
699
|
+
... 1 more
|
|
333
700
|
],
|
|
334
701
|
"model": "nomic-embed-text-v1-GGUF",
|
|
335
|
-
"usage": {
|
|
336
|
-
"prompt_tokens": 12,
|
|
337
|
-
"total_tokens": 12
|
|
338
|
-
}
|
|
702
|
+
"usage": {"prompt_tokens": 12, "total_tokens": 12}
|
|
339
703
|
}
|
|
340
704
|
```
|
|
341
705
|
|
|
342
|
-
|
|
343
|
-
|
|
344
|
-
- `object` - Type of response object, always `"list"`
|
|
345
|
-
- `data` - Array of embedding objects
|
|
346
|
-
- `object` - Type of embedding object, always `"embedding"`
|
|
347
|
-
- `index` - Index position of the input text in the request
|
|
348
|
-
- `embedding` - Vector representation as an array of floats
|
|
349
|
-
- `model` - Model identifier used to generate the embeddings
|
|
350
|
-
- `usage` - Token usage statistics
|
|
351
|
-
- `prompt_tokens` - Number of tokens in the input
|
|
352
|
-
- `total_tokens` - Total tokens processed
|
|
706
|
+
### Response format
|
|
353
707
|
|
|
708
|
+
- `object` - Always `"list"`.
|
|
709
|
+
- `data` - One embedding per input.
|
|
710
|
+
- `object` - Always `"embedding"`.
|
|
711
|
+
- `index` - Position of the input text in the request.
|
|
712
|
+
- `embedding` - The vector, as an array of floats, or a base64 string when `encoding_format` is `base64`.
|
|
713
|
+
- `model` - The model that produced the embeddings.
|
|
714
|
+
- `usage` - Token usage.
|
|
715
|
+
- `prompt_tokens` - Tokens in the input.
|
|
716
|
+
- `total_tokens` - Tokens processed in total.
|
|
717
|
+
|
|
718
|
+
### Errors
|
|
719
|
+
|
|
720
|
+
| Status | Code | When |
|
|
721
|
+
|--------|------|------|
|
|
722
|
+
| `400` | `invalid_json` | The body is not a JSON object |
|
|
723
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
724
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
725
|
+
| `400` | | `model`'s backend cannot embed, such as a text-to-speech model |
|
|
726
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
727
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
728
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
729
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
730
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
731
|
+
| `500` | | The backend reports an error; the backend's own status replaces this one when it sends one |
|
|
732
|
+
| `500` | | The request fails unexpectedly |
|
|
733
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
734
|
+
| `501` | | `model` is a llama.cpp model that is not an embedding model |
|
|
735
|
+
<!-- END GENERATED: openai.embeddings -->
|
|
736
|
+
|
|
737
|
+
<!-- BEGIN GENERATED: openai.responses -->
|
|
354
738
|
## `POST /v1/responses`
|
|
355
|
-
<sub></sub>
|
|
356
740
|
|
|
357
|
-
|
|
741
|
+
Generates a model response to an input, in OpenAI's Responses format, downloading and loading the model if it is not loaded.
|
|
742
|
+
|
|
743
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
358
744
|
|
|
359
745
|
### Parameters
|
|
360
746
|
|
|
361
747
|
| Parameter | Required | Description | Status |
|
|
362
748
|
|-----------|----------|-------------|--------|
|
|
363
|
-
| `input` | Yes |
|
|
364
|
-
| `model` | Yes | The model to
|
|
365
|
-
| `max_output_tokens` | No |
|
|
366
|
-
| `temperature` | No |
|
|
367
|
-
| `repeat_penalty` | No |
|
|
368
|
-
| `top_k` | No |
|
|
369
|
-
| `top_p` | No |
|
|
370
|
-
| `stream` | No |
|
|
371
|
-
|
|
749
|
+
| `input` | Yes | The input, as a string or an array of input items. | <sub></sub> |
|
|
750
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
751
|
+
| `max_output_tokens` | No | Upper bound on generated tokens. | <sub></sub> |
|
|
752
|
+
| `temperature` | No | Sampling temperature. | <sub></sub> |
|
|
753
|
+
| `repeat_penalty` | No | From `1.0` to `2.0`: `1.0` applies no penalty, and higher values discourage repetition. | <sub></sub> |
|
|
754
|
+
| `top_k` | No | Number of top tokens to consider during sampling. | <sub></sub> |
|
|
755
|
+
| `top_p` | No | Cumulative probability of the top tokens to consider during nucleus sampling, from `0.0` to `1.0`. | <sub></sub> |
|
|
756
|
+
| `stream` | No | `true` sends the response as server-sent events as it progresses; `false` sends one response once complete. Defaults to `false`. | <sub></sub> |
|
|
757
|
+
| `route_trace` | No | Lemonade extension for `collection.router` models: `true` adds the per-condition `trace` to `x_lemonade_route`. Defaults to `false`. | <sub></sub> |
|
|
758
|
+
| `ctx_size` | No | Lemonade extension: context size to load the model with, when this request loads it. Ignored when the model is already loaded. | <sub></sub> |
|
|
372
759
|
|
|
373
|
-
|
|
760
|
+
Naming a `collection.router` model routes the request to one of its candidates; see [Router API](./router.md).
|
|
374
761
|
|
|
375
|
-
|
|
762
|
+
### Examples
|
|
376
763
|
|
|
377
|
-
-
|
|
378
|
-
- `response.output_text.delta`
|
|
379
|
-
- `response.completed`
|
|
380
|
-
|
|
381
|
-
For a full list of event types, see the [API reference for streaming](https://platform.openai.com/docs/api-reference/responses-streaming).
|
|
382
|
-
|
|
383
|
-
### Example request
|
|
764
|
+
#### Non-streaming
|
|
384
765
|
|
|
385
766
|
=== "PowerShell"
|
|
386
767
|
|
|
387
768
|
```powershell
|
|
388
|
-
Invoke-WebRequest
|
|
769
|
+
Invoke-WebRequest `
|
|
770
|
+
-Uri "http://localhost:13305/v1/responses" `
|
|
389
771
|
-Method POST `
|
|
390
772
|
-Headers @{ "Content-Type" = "application/json" } `
|
|
391
773
|
-Body '{
|
|
392
|
-
|
|
393
|
-
|
|
394
|
-
|
|
395
|
-
|
|
774
|
+
"model": "Llama-3.2-1B-Instruct-GGUF",
|
|
775
|
+
"input": "What is the population of Paris?",
|
|
776
|
+
"stream": false
|
|
777
|
+
}'
|
|
396
778
|
```
|
|
397
779
|
|
|
398
780
|
=== "Bash"
|
|
399
781
|
|
|
400
782
|
```bash
|
|
401
|
-
curl
|
|
783
|
+
curl http://localhost:13305/v1/responses \
|
|
402
784
|
-H "Content-Type: application/json" \
|
|
403
785
|
-d '{
|
|
404
|
-
|
|
405
|
-
|
|
406
|
-
|
|
407
|
-
|
|
786
|
+
"model": "Llama-3.2-1B-Instruct-GGUF",
|
|
787
|
+
"input": "What is the population of Paris?",
|
|
788
|
+
"stream": false
|
|
789
|
+
}'
|
|
408
790
|
```
|
|
409
791
|
|
|
792
|
+
**Response:** `200 OK`
|
|
410
793
|
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
794
|
+
```json
|
|
795
|
+
{
|
|
796
|
+
"id": "resp_Cxow6uUoxWRLPnhhXU7QEvXjMrKaBGYi",
|
|
797
|
+
"object": "response",
|
|
798
|
+
"created_at": 1793827678,
|
|
799
|
+
"completed_at": 1793827678,
|
|
800
|
+
"model": "Llama-3.2-1B-Instruct-GGUF",
|
|
801
|
+
"status": "completed",
|
|
802
|
+
"output": [
|
|
416
803
|
{
|
|
417
|
-
"
|
|
418
|
-
|
|
419
|
-
"model": "Llama-3.2-1B-Instruct-Hybrid",
|
|
420
|
-
"object": "response",
|
|
421
|
-
"output": [{
|
|
422
|
-
"id": "0",
|
|
423
|
-
"content": [{
|
|
804
|
+
"content": [
|
|
805
|
+
{
|
|
424
806
|
"annotations": [],
|
|
425
|
-
"
|
|
426
|
-
|
|
427
|
-
|
|
807
|
+
"logprobs": [],
|
|
808
|
+
"text": "As of the latest available data from the French National Institute for Statistics and Economic Studies (Institut National de la Statistique et des Études Économiques, INSEE), the population of Paris is approximately 2.1 million people within the city limits and over 12 million people in the metropolitan area, which includes the surrounding suburbs and regions.",
|
|
809
|
+
"type": "output_text"
|
|
810
|
+
}
|
|
811
|
+
],
|
|
812
|
+
"id": "msg_zkaZoMfka6ivKETrMiMf7f8NzXFj7FCQ",
|
|
813
|
+
"role": "assistant",
|
|
814
|
+
"status": "completed",
|
|
815
|
+
"type": "message"
|
|
428
816
|
}
|
|
817
|
+
],
|
|
818
|
+
"usage": {
|
|
819
|
+
"input_tokens": 42,
|
|
820
|
+
"output_tokens": 73,
|
|
821
|
+
"total_tokens": 115,
|
|
822
|
+
"input_tokens_details": {"cached_tokens": 0}
|
|
823
|
+
}
|
|
824
|
+
}
|
|
825
|
+
```
|
|
826
|
+
|
|
827
|
+
#### Streaming
|
|
828
|
+
|
|
829
|
+
=== "PowerShell"
|
|
830
|
+
|
|
831
|
+
```powershell
|
|
832
|
+
Invoke-WebRequest `
|
|
833
|
+
-Uri "http://localhost:13305/v1/responses" `
|
|
834
|
+
-Method POST `
|
|
835
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
836
|
+
-Body '{
|
|
837
|
+
"model": "Llama-3.2-1B-Instruct-GGUF",
|
|
838
|
+
"input": "What is the population of Paris?",
|
|
839
|
+
"stream": true
|
|
840
|
+
}'
|
|
429
841
|
```
|
|
430
842
|
|
|
431
|
-
=== "
|
|
432
|
-
|
|
843
|
+
=== "Bash"
|
|
844
|
+
|
|
845
|
+
```bash
|
|
846
|
+
curl http://localhost:13305/v1/responses \
|
|
847
|
+
-H "Content-Type: application/json" \
|
|
848
|
+
-d '{
|
|
849
|
+
"model": "Llama-3.2-1B-Instruct-GGUF",
|
|
850
|
+
"input": "What is the population of Paris?",
|
|
851
|
+
"stream": true
|
|
852
|
+
}'
|
|
853
|
+
```
|
|
433
854
|
|
|
855
|
+
**Response:** `200 OK`
|
|
856
|
+
|
|
857
|
+
```text
|
|
858
|
+
event: response.created
|
|
859
|
+
data: {"type": "response.created", "response": {"id": "resp_L7uzVj7alzqP0uYc4HYBpvdp60tAsd1Z", "object": "response", "status": "in_progress"}}
|
|
860
|
+
event: response.in_progress
|
|
861
|
+
data: {"type": "response.in_progress", "response": {"id": "resp_L7uzVj7alzqP0uYc4HYBpvdp60tAsd1Z", "object": "response", "status": "in_progress"}}
|
|
862
|
+
event: response.output_item.added
|
|
863
|
+
data: {"type": "response.output_item.added", "item": {"content": [], "id": "msg_depHpGR0ZqLNdgRXk7wGBHbhXKjkiC0M", "role": "assistant", "status": "in_progress", "type": "message"}, "output_index": 0}
|
|
864
|
+
...
|
|
865
|
+
event: response.completed
|
|
866
|
+
data: {"type": "response.completed", "response": {"id": "resp_L7uzVj7alzqP0uYc4HYBpvdp60tAsd1Z", "object": "response", "created_at": 1793827678, "status": "completed", "model": "~/.cache/huggingface/hub/models--unsloth--Llama-3.2-1B-Instruct-GGUF/snapshots/b69aef112e9f895e6f98d7ae0949f72ff09aa401/Llama-3.2-1B-Instruct-UD-Q4_K_XL.gguf", "output": [{"type": "message", "status": "completed", "id": "msg_depHpGR0ZqLNdgRXk7wGBHbhXKjkiC0M", "content": [{"type": "output_text", "annotations": [], "logprobs": [], "text": "As of my knowledge cutoff in December 2023, the population of Paris, France is approximately 2.1 million people within the city limits and 12.2 million people in the larger metropolitan area, according to official statistics from the French National Institute for Statistics and Economic Studies (INSEE)."}], "role": "assistant"}], "usage": {"input_tokens": 42, "output_tokens": 61, "total_tokens": 103, "input_tokens_details": {"cached_tokens": 41}}}, "timings": {"cache_n": 41, "prompt_n": 1, "prompt_ms": 16.128, "prompt_per_token_ms": 16.128, "prompt_per_second": 62.00396825396825, "predicted_n": 61, "predicted_ms": 375.356, "predicted_per_token_ms": 6.255933333333333, "predicted_per_second": 159.84825072730953}}
|
|
867
|
+
data: [DONE]
|
|
868
|
+
```
|
|
434
869
|
|
|
870
|
+
### Response format
|
|
435
871
|
|
|
872
|
+
#### JSON
|
|
873
|
+
|
|
874
|
+
- `id` - Identifier of the response.
|
|
875
|
+
- `object` - Always `"response"`.
|
|
876
|
+
- `created_at` - Unix timestamp of when the response was created.
|
|
877
|
+
- `completed_at` - Unix timestamp of when the response completed, when the backend reports it.
|
|
878
|
+
- `model` - The model that generated the response.
|
|
879
|
+
- `status` - State of the response, such as `completed`.
|
|
880
|
+
- `output` - Output items; a message item carries its text in `content[].text`.
|
|
881
|
+
- `usage` - Token counts for the request.
|
|
882
|
+
- `input_tokens` - Tokens in the input.
|
|
883
|
+
- `output_tokens` - Tokens generated.
|
|
884
|
+
- `total_tokens` - Sum of `input_tokens` and `output_tokens`.
|
|
885
|
+
- `input_tokens_details` - Breakdown of the input tokens, such as `cached_tokens`, when the backend reports it.
|
|
886
|
+
- `x_lemonade_route` - The routing decision, when `model` names a `collection.router` model. See [Routing](./router.md#routing-collectionrouter) for the decision object.
|
|
887
|
+
|
|
888
|
+
#### Event stream
|
|
889
|
+
|
|
890
|
+
One event per `data:` line. The stream ends with `data: [DONE]`.
|
|
891
|
+
|
|
892
|
+
##### Response event
|
|
893
|
+
|
|
894
|
+
An event of the response, passed through from the backend.
|
|
895
|
+
|
|
896
|
+
- `type` - Event type, passed through from the backend, such as `response.created`, `response.output_text.delta` or `response.completed`; see OpenAI's [streaming reference](https://platform.openai.com/docs/api-reference/responses-streaming).
|
|
897
|
+
- `response` - Lifecycle events, such as `response.created`, `response.in_progress` and `response.completed`, only: the response so far.
|
|
898
|
+
- `item` - Output item events, such as `response.output_item.added`, only: the output item.
|
|
899
|
+
- `output_index` - Zero-based index of the output item in the response's `output` array, on output-item and delta events.
|
|
900
|
+
- `item_id` - Delta events only: the `id` of the output item the text belongs to.
|
|
901
|
+
- `delta` - Delta events, such as `response.output_text.delta` and `response.reasoning_text.delta`, only: the next text.
|
|
902
|
+
- `timings` - `response.completed` only: prompt processing and generation timings, passed through from backends that report them, such as llama.cpp.
|
|
903
|
+
- `x_lemonade_route` - The routing decision, when `model` names a `collection.router` model. See [Routing](./router.md#routing-collectionrouter) for the decision object.
|
|
904
|
+
|
|
905
|
+
##### Error event
|
|
906
|
+
|
|
907
|
+
Sent in place of the rest of the stream when the backend fails after the stream has started. No `[DONE]` follows it.
|
|
908
|
+
|
|
909
|
+
- `error` - The backend's error, as an object or, when the backend sent one, a string.
|
|
910
|
+
- `message` - What failed.
|
|
911
|
+
- `type` - The error's type, such as `backend_error`.
|
|
912
|
+
- `status_code` - The status the backend answered with, or `502` when it closed the stream without a response.
|
|
913
|
+
|
|
914
|
+
### Errors
|
|
915
|
+
|
|
916
|
+
| Status | Code | When |
|
|
917
|
+
|--------|------|------|
|
|
918
|
+
| `400` | `invalid_json` | The body is not a JSON object |
|
|
919
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
920
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
921
|
+
| `400` | | The request does not stream and `model` is a FastFlowLM or cloud model, which has no Responses API |
|
|
922
|
+
| `400` | `model_not_applicable` | The request does not stream and `model` is not an LLM |
|
|
923
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
924
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
925
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
926
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
927
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
928
|
+
| `500` | | The backend reports an error; the backend's own status replaces this one when it sends one |
|
|
929
|
+
| `500` | | The request fails unexpectedly |
|
|
930
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
931
|
+
<!-- END GENERATED: openai.responses -->
|
|
932
|
+
|
|
933
|
+
<!-- BEGIN GENERATED: openai.audio_transcriptions -->
|
|
436
934
|
## `POST /v1/audio/transcriptions`
|
|
437
|
-
<sub></sub>
|
|
438
936
|
|
|
439
|
-
|
|
937
|
+
Transcribes an audio file to text, loading the model on first use. Whisper (whisper.cpp), Moonshine and FastFlowLM models serve this endpoint.
|
|
440
938
|
|
|
441
|
-
|
|
442
|
-
>
|
|
443
|
-
> **Limitations:** Only `wav` audio input is currently supported. The `response_format` field supports `json`, `verbose_json`, `text`, `srt`, and `vtt`. On the FastFlowLM (FLM) backend, `srt` and `vtt` are rejected with a `400` because FLM returns no segment timestamps, and `verbose_json` returns the compact shape without a `segments` field.
|
|
939
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
444
940
|
|
|
445
941
|
### Parameters
|
|
446
942
|
|
|
447
943
|
| Parameter | Required | Description | Status |
|
|
448
944
|
|-----------|----------|-------------|--------|
|
|
449
|
-
| `file` | Yes | The audio file to transcribe
|
|
450
|
-
| `model` | Yes | The
|
|
451
|
-
| `language` | No | The language of the audio (ISO 639-1 code, e.g., `en`, `es`, `fr`). Defaults to `auto`, which
|
|
452
|
-
| `response_format` | No |
|
|
945
|
+
| `file` | Yes | The audio file to transcribe, as WAV. | <sub></sub> |
|
|
946
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
947
|
+
| `language` | No | The language of the audio (ISO 639-1 code, e.g., `en`, `es`, `fr`). Defaults to `auto`, which detects it, for whisper.cpp models, and to the backend's own for the others. | <sub></sub> |
|
|
948
|
+
| `response_format` | No | `json`, `verbose_json`, `text`, `srt` or `vtt`. Defaults to `json`. | <sub></sub> |
|
|
949
|
+
| `prompt` | No | Text to guide the transcription's style or continue a previous segment. | <sub></sub> |
|
|
950
|
+
| `temperature` | No | Sampling temperature. Defaults to `0` for whisper.cpp models, and to the backend's own for FastFlowLM models. Moonshine models ignore it. | <sub></sub> |
|
|
453
951
|
|
|
454
|
-
###
|
|
952
|
+
### Examples
|
|
455
953
|
|
|
456
|
-
|
|
954
|
+
#### Transcript
|
|
457
955
|
|
|
458
|
-
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
956
|
+
=== "PowerShell"
|
|
957
|
+
|
|
958
|
+
```powershell
|
|
959
|
+
curl.exe http://localhost:13305/v1/audio/transcriptions `
|
|
960
|
+
-F "model=Whisper-Tiny" `
|
|
961
|
+
-F file=@speech.wav
|
|
462
962
|
```
|
|
463
963
|
|
|
464
|
-
=== "
|
|
964
|
+
=== "Bash"
|
|
465
965
|
|
|
466
966
|
```bash
|
|
467
|
-
curl
|
|
468
|
-
-F "
|
|
469
|
-
-F
|
|
967
|
+
curl http://localhost:13305/v1/audio/transcriptions \
|
|
968
|
+
-F "model=Whisper-Tiny" \
|
|
969
|
+
-F file=@speech.wav
|
|
470
970
|
```
|
|
471
971
|
|
|
472
|
-
|
|
972
|
+
**Response:** `200 OK`
|
|
473
973
|
|
|
474
974
|
```json
|
|
475
|
-
{
|
|
476
|
-
"text": "Hello, this is a sample transcription of the audio file."
|
|
477
|
-
}
|
|
975
|
+
{"text": " Just seeing if this is working.\n"}
|
|
478
976
|
```
|
|
479
977
|
|
|
480
|
-
|
|
978
|
+
#### Plain text
|
|
979
|
+
|
|
980
|
+
=== "PowerShell"
|
|
981
|
+
|
|
982
|
+
```powershell
|
|
983
|
+
curl.exe http://localhost:13305/v1/audio/transcriptions `
|
|
984
|
+
-F "model=Whisper-Tiny" `
|
|
985
|
+
-F "response_format=text" `
|
|
986
|
+
-F file=@speech.wav
|
|
987
|
+
```
|
|
988
|
+
|
|
989
|
+
=== "Bash"
|
|
990
|
+
|
|
991
|
+
```bash
|
|
992
|
+
curl http://localhost:13305/v1/audio/transcriptions \
|
|
993
|
+
-F "model=Whisper-Tiny" \
|
|
994
|
+
-F "response_format=text" \
|
|
995
|
+
-F file=@speech.wav
|
|
996
|
+
```
|
|
997
|
+
|
|
998
|
+
**Response:** `200 OK`
|
|
481
999
|
|
|
482
|
-
|
|
1000
|
+
```text
|
|
1001
|
+
Just seeing if this is working.
|
|
1002
|
+
```
|
|
483
1003
|
|
|
1004
|
+
### Response format
|
|
484
1005
|
|
|
1006
|
+
- `text` - The transcribed text.
|
|
1007
|
+
- `segments` - `verbose_json` only: timestamped segments.
|
|
1008
|
+
|
|
1009
|
+
### Errors
|
|
1010
|
+
|
|
1011
|
+
| Status | Code | When |
|
|
1012
|
+
|--------|------|------|
|
|
1013
|
+
| `400` | | The request is not `multipart/form-data` |
|
|
1014
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
1015
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
1016
|
+
| `400` | | `model` cannot transcribe, which is found after it loads; the error's `type` is `unsupported_operation` |
|
|
1017
|
+
| `400` | | `response_format` is `srt` or `vtt` and `model` is a FastFlowLM model |
|
|
1018
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
1019
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
1020
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
1021
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
1022
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
1023
|
+
| `500` | | The backend reports an error; the backend's own status replaces this one when it sends one |
|
|
1024
|
+
| `500` | | The request fails unexpectedly |
|
|
1025
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
1026
|
+
<!-- END GENERATED: openai.audio_transcriptions -->
|
|
485
1027
|
|
|
486
1028
|
## `WS /realtime`
|
|
487
1029
|
<sub></sub>
|
|
488
1030
|
|
|
489
1031
|
Realtime Audio Transcription API via WebSocket (OpenAI SDK compatible). Stream audio from a microphone and receive transcriptions in real-time with Voice Activity Detection (VAD).
|
|
490
1032
|
|
|
491
|
-
|
|
1033
|
+
Models with the `realtime-transcription` [label](#model-labels) serve it: the Whisper models and the Moonshine streaming models. Audio must be 16 kHz mono PCM16; downsample higher rates on the client.
|
|
492
1034
|
|
|
493
1035
|
### Connection
|
|
494
1036
|
|
|
495
|
-
|
|
1037
|
+
Connect on the main HTTP port (default 13305), the same port as the REST endpoints, naming the model:
|
|
496
1038
|
|
|
497
1039
|
```
|
|
498
1040
|
ws://localhost:13305/v1/realtime?model=Whisper-Tiny
|
|
499
1041
|
```
|
|
500
1042
|
|
|
501
|
-
|
|
1043
|
+
OpenAI Realtime SDK clients connect to `/v1/realtime` as-is. When `LEMONADE_API_KEY` is set, pass `?api_key=KEY` as a query parameter.
|
|
502
1044
|
|
|
503
|
-
A dedicated WebSocket port also remains for backward compatibility
|
|
1045
|
+
A dedicated WebSocket port also remains for backward compatibility. It is OS-assigned and reported as `websocket_port` by [`/v1/health`](./lemonade.md#get-v1health), and it speaks the identical protocol, so moving a client to the main port changes only its URL. The main port also leaves one port to expose remotely and works through reverse proxies that pass `Upgrade: websocket`:
|
|
504
1046
|
|
|
505
1047
|
```
|
|
506
1048
|
ws://localhost:<websocket_port>/realtime?model=Whisper-Tiny
|
|
@@ -551,12 +1093,7 @@ Upon connection, the server sends a `session.created` message with a session ID.
|
|
|
551
1093
|
}
|
|
552
1094
|
```
|
|
553
1095
|
|
|
554
|
-
|
|
555
|
-
- 16kHz sample rate
|
|
556
|
-
- Mono (single channel)
|
|
557
|
-
- 16-bit signed integer (PCM16)
|
|
558
|
-
- Base64 encoded
|
|
559
|
-
- Sent in chunks (~85ms recommended)
|
|
1096
|
+
Send base64-encoded PCM16 audio in chunks of about 85 to 256 ms.
|
|
560
1097
|
|
|
561
1098
|
### Example: Transcription Result
|
|
562
1099
|
|
|
@@ -591,7 +1128,7 @@ VAD settings can be configured via `session.update`:
|
|
|
591
1128
|
| `silence_duration_ms` | 800 | Silence duration to trigger speech end |
|
|
592
1129
|
| `prefix_padding_ms` | 250 | Minimum speech duration before triggering |
|
|
593
1130
|
|
|
594
|
-
Set `turn_detection` to `null` to disable server-side VAD and use explicit commits instead
|
|
1131
|
+
Set `turn_detection` to `null` to disable server-side VAD and use explicit commits instead. The server then buffers audio and emits no VAD or interim transcription events until `input_audio_buffer.commit`:
|
|
595
1132
|
|
|
596
1133
|
```json
|
|
597
1134
|
{
|
|
@@ -614,101 +1151,134 @@ A complete, runnable example:
|
|
|
614
1151
|
python examples/realtime_transcription.py --model Whisper-Tiny
|
|
615
1152
|
```
|
|
616
1153
|
|
|
617
|
-
|
|
618
|
-
|
|
619
|
-
- **Audio Format**: Server expects 16kHz mono PCM16. Higher sample rates must be downsampled client-side.
|
|
620
|
-
- **Chunk Size**: Send audio in ~85-256ms chunks for optimal latency/efficiency.
|
|
621
|
-
- **VAD Behavior**: Server automatically detects speech boundaries and triggers transcription on speech end.
|
|
622
|
-
- **Manual Commit**: Set `turn_detection` to `null`, then use `input_audio_buffer.commit` to force transcription. In this mode the server buffers audio but does not emit VAD or interim transcription events.
|
|
623
|
-
- **Clear Buffer**: Use `input_audio_buffer.clear` to discard audio without transcribing.
|
|
624
|
-
- **Chunking**: We are still tuning the chunking to balance latency vs. accuracy.
|
|
625
|
-
- **Migrating off the dedicated port**: Clients that discover `websocket_port` via `/v1/health` and connect there can switch to `ws://HOST:13305/v1/realtime?model=...` — the protocol (events, audio format, auth) is identical on both ports, so it is a URL change only. This also simplifies remote setups (one port to expose) and works through reverse proxies that pass `Upgrade: websocket`. Keep the `websocket_port` fallback only if you must support servers older than this release.
|
|
626
|
-
|
|
627
|
-
|
|
1154
|
+
<!-- BEGIN GENERATED: openai.images_generations -->
|
|
628
1155
|
## `POST /v1/images/generations`
|
|
629
|
-
<sub></sub>
|
|
630
1157
|
|
|
631
|
-
|
|
1158
|
+
Generates an image from a text prompt. CPU inference takes minutes per image; a GPU backend is much faster.
|
|
632
1159
|
|
|
633
|
-
|
|
1160
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
634
1161
|
|
|
635
1162
|
### Parameters
|
|
636
1163
|
|
|
637
1164
|
| Parameter | Required | Description | Status |
|
|
638
1165
|
|-----------|----------|-------------|--------|
|
|
639
1166
|
| `prompt` | Yes | The text description of the image to generate. | <sub></sub> |
|
|
640
|
-
| `model` | Yes | The
|
|
641
|
-
| `size` | No |
|
|
642
|
-
| `n` | No | Number of images to generate.
|
|
643
|
-
| `response_format` | No |
|
|
644
|
-
| `steps` | No | Number of inference steps
|
|
645
|
-
| `cfg_scale` | No | Classifier-free guidance scale.
|
|
646
|
-
| `seed` | No | Random seed for
|
|
1167
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
1168
|
+
| `size` | No | Image size as `WIDTHxHEIGHT`, such as `256x256`. Defaults to the model's own, else `512x512`. | <sub></sub> |
|
|
1169
|
+
| `n` | No | Number of images to generate. Defaults to `1`. | <sub></sub> |
|
|
1170
|
+
| `response_format` | No | Only `b64_json`, a base64-encoded image, is supported. | <sub></sub> |
|
|
1171
|
+
| `steps` | No | Number of inference steps; lower is faster, higher can improve quality. Defaults to the model's own setting, else the backend's. | <sub></sub> |
|
|
1172
|
+
| `cfg_scale` | No | Classifier-free guidance scale. Defaults to the model's own, else `7.0`. | <sub></sub> |
|
|
1173
|
+
| `seed` | No | Random seed for reproducible output; a negative value picks a random seed. Defaults to the backend's fixed seed. | <sub></sub> |
|
|
1174
|
+
|
|
1175
|
+
### Example
|
|
1176
|
+
|
|
1177
|
+
=== "PowerShell"
|
|
647
1178
|
|
|
648
|
-
|
|
1179
|
+
```powershell
|
|
1180
|
+
Invoke-WebRequest `
|
|
1181
|
+
-Uri "http://localhost:13305/v1/images/generations" `
|
|
1182
|
+
-Method POST `
|
|
1183
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
1184
|
+
-Body '{
|
|
1185
|
+
"model": "SD-Turbo",
|
|
1186
|
+
"prompt": "A serene mountain landscape at sunset",
|
|
1187
|
+
"size": "512x512",
|
|
1188
|
+
"steps": 4,
|
|
1189
|
+
"response_format": "b64_json"
|
|
1190
|
+
}'
|
|
1191
|
+
```
|
|
649
1192
|
|
|
650
1193
|
=== "Bash"
|
|
651
1194
|
|
|
652
1195
|
```bash
|
|
653
|
-
curl
|
|
1196
|
+
curl http://localhost:13305/v1/images/generations \
|
|
654
1197
|
-H "Content-Type: application/json" \
|
|
655
1198
|
-d '{
|
|
656
|
-
|
|
657
|
-
|
|
658
|
-
|
|
659
|
-
|
|
660
|
-
|
|
661
|
-
|
|
1199
|
+
"model": "SD-Turbo",
|
|
1200
|
+
"prompt": "A serene mountain landscape at sunset",
|
|
1201
|
+
"size": "512x512",
|
|
1202
|
+
"steps": 4,
|
|
1203
|
+
"response_format": "b64_json"
|
|
1204
|
+
}'
|
|
662
1205
|
```
|
|
663
1206
|
|
|
1207
|
+
**Response:** `200 OK`
|
|
1208
|
+
|
|
1209
|
+
```json
|
|
1210
|
+
{
|
|
1211
|
+
"created": 1791943145,
|
|
1212
|
+
"data": [
|
|
1213
|
+
{
|
|
1214
|
+
"b64_json": "iVBORw0KGgoAAAANSUhEUgAAAgAAAAIACAIAAAB7GkOtAAAEgnRFWHRwYXJhbWV0ZXJzAEEgc2VyZW5l... (886,104 characters)"
|
|
1215
|
+
}
|
|
1216
|
+
],
|
|
1217
|
+
"output_format": "png"
|
|
1218
|
+
}
|
|
1219
|
+
```
|
|
1220
|
+
|
|
1221
|
+
### Response format
|
|
1222
|
+
|
|
1223
|
+
- `created` - Unix timestamp of when the image was generated.
|
|
1224
|
+
- `data` - The generated images, one entry per image.
|
|
1225
|
+
- `b64_json` - Base64-encoded PNG.
|
|
1226
|
+
- `generation_info` - Optional generation parameters and model metadata reported by the image backend.
|
|
1227
|
+
- `output_format` - File format of the images, such as `png`, when the backend reports it.
|
|
1228
|
+
|
|
1229
|
+
### Errors
|
|
1230
|
+
|
|
1231
|
+
| Status | Code | When |
|
|
1232
|
+
|--------|------|------|
|
|
1233
|
+
| `400` | `invalid_json` | The body is not a JSON object |
|
|
1234
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
1235
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
1236
|
+
| `400` | | The backend's reply does not parse, or a request field not listed above has the wrong JSON type |
|
|
1237
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
1238
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
1239
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
1240
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
1241
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
1242
|
+
| `500` | | The backend reports an error, for example when `model` cannot serve this route, an error whose `type` is `unsupported_operation` |
|
|
1243
|
+
| `500` | | The request fails unexpectedly |
|
|
1244
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
1245
|
+
<!-- END GENERATED: openai.images_generations -->
|
|
1246
|
+
|
|
1247
|
+
<!-- BEGIN GENERATED: openai.images_edits -->
|
|
664
1248
|
## `POST /v1/images/edits`
|
|
665
|
-
<sub></sub>
|
|
666
1250
|
|
|
667
|
-
|
|
1251
|
+
Edits one or more reference images as a prompt describes. The request is `multipart/form-data`. CPU inference takes minutes per image; a GPU backend is much faster.
|
|
668
1252
|
|
|
669
|
-
|
|
670
|
-
>
|
|
671
|
-
> **Performance:** CPU inference takes several minutes per image. GPU (ROCm) is significantly faster.
|
|
1253
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
672
1254
|
|
|
673
1255
|
### Parameters
|
|
674
1256
|
|
|
675
1257
|
| Parameter | Required | Description | Status |
|
|
676
1258
|
|-----------|----------|-------------|--------|
|
|
677
|
-
| `model` | Yes | The
|
|
678
|
-
| `image`
|
|
1259
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
1260
|
+
| `image` or `image[]` | Yes | PNG or JPEG image upload. For multiple references, repeat `image[]` in order. Files are forwarded without re-encoding. | <sub></sub> |
|
|
679
1261
|
| `prompt` | Yes | A text description of the desired edit. | <sub></sub> |
|
|
680
|
-
| `mask` | No |
|
|
681
|
-
| `size` | No |
|
|
682
|
-
| `n` | No | Number of images to generate
|
|
683
|
-
| `response_format` | No |
|
|
684
|
-
| `steps` | No | Number of inference steps.
|
|
685
|
-
| `cfg_scale` | No | Classifier-free guidance scale.
|
|
686
|
-
| `seed` | No | Random seed for
|
|
687
|
-
| `user` | No |
|
|
688
|
-
| `background` | No |
|
|
689
|
-
| `quality` | No |
|
|
690
|
-
| `input_fidelity` | No |
|
|
691
|
-
| `output_compression` | No | OpenAI
|
|
692
|
-
|
|
693
|
-
###
|
|
694
|
-
|
|
695
|
-
=== "Bash"
|
|
696
|
-
|
|
697
|
-
```bash
|
|
698
|
-
curl -X POST http://localhost:13305/v1/images/edits \
|
|
699
|
-
-F "model=Flux-2-Klein-4B" \
|
|
700
|
-
-F "prompt=Add a red barn and mountains in the background, photorealistic" \
|
|
701
|
-
-F "size=512x512" \
|
|
702
|
-
-F "n=1" \
|
|
703
|
-
-F "response_format=b64_json" \
|
|
704
|
-
-F "image=@/path/to/source_image.png"
|
|
705
|
-
```
|
|
1262
|
+
| `mask` | No | A PNG mask: white areas are edited and black areas are kept. | <sub></sub> |
|
|
1263
|
+
| `size` | No | Output size as `WIDTHxHEIGHT`, such as `512x512`. Defaults to the first reference image's size, scaled down to fit the model's `soft_pixel_cap` option (2048x2048 pixels, about 4 MP, unless the model sets one). An explicit size skips that cap; backend limits still apply. | <sub></sub> |
|
|
1264
|
+
| `n` | No | Number of images to generate, from `1` to `10`. Defaults to `1`. | <sub></sub> |
|
|
1265
|
+
| `response_format` | No | Only `b64_json`, a base64-encoded image, is supported. | <sub></sub> |
|
|
1266
|
+
| `steps` | No | Number of inference steps. Defaults to the model's own. | <sub></sub> |
|
|
1267
|
+
| `cfg_scale` | No | Classifier-free guidance scale. Defaults to the model's own. | <sub></sub> |
|
|
1268
|
+
| `seed` | No | Random seed for reproducible output. | <sub></sub> |
|
|
1269
|
+
| `user` | No | Accepted for OpenAI compatibility and ignored. | <sub></sub> |
|
|
1270
|
+
| `background` | No | Accepted for OpenAI compatibility and ignored. | <sub></sub> |
|
|
1271
|
+
| `quality` | No | Accepted for OpenAI compatibility and ignored. | <sub></sub> |
|
|
1272
|
+
| `input_fidelity` | No | Accepted for OpenAI compatibility and ignored. | <sub></sub> |
|
|
1273
|
+
| `output_compression` | No | Accepted for OpenAI compatibility and ignored. | <sub></sub> |
|
|
1274
|
+
|
|
1275
|
+
### Edit with the OpenAI Python Client
|
|
706
1276
|
|
|
707
1277
|
=== "Python (OpenAI client)"
|
|
708
1278
|
|
|
709
1279
|
```python
|
|
710
1280
|
from openai import OpenAI
|
|
711
|
-
client = OpenAI(base_url="http://localhost:13305/
|
|
1281
|
+
client = OpenAI(base_url="http://localhost:13305/v1", api_key="not-needed")
|
|
712
1282
|
with open("source_image.png", "rb") as image_file:
|
|
713
1283
|
response = client.images.edit(
|
|
714
1284
|
model="Flux-2-Klein-4B",
|
|
@@ -721,124 +1291,207 @@ Image Editing API. You provide a source image and a text prompt describing the d
|
|
|
721
1291
|
open("edited_image.png", "wb").write(image_data)
|
|
722
1292
|
```
|
|
723
1293
|
|
|
724
|
-
|
|
725
|
-
|
|
726
|
-
## `POST /v1/images/variations`
|
|
727
|
-
<sub></sub>
|
|
728
|
-
|
|
729
|
-
Image Variations API. You provide a source image and receive a variation of it.
|
|
730
|
-
|
|
731
|
-
> **Note:** This endpoint accepts `multipart/form-data` requests (not JSON). Unlike `/images/edits`, a `prompt` parameter is not supported and will be ignored — the model generates a variation based solely on the input image.
|
|
732
|
-
>
|
|
733
|
-
> **Performance:** CPU inference takes several minutes per image. GPU (ROCm) is significantly faster.
|
|
1294
|
+
### Example
|
|
734
1295
|
|
|
735
|
-
|
|
736
|
-
|
|
737
|
-
| Parameter | Required | Description | Status |
|
|
738
|
-
|-----------|----------|-------------|--------|
|
|
739
|
-
| `model` | Yes | The Stable Diffusion model to use (e.g., `Flux-2-Klein-4B`, `SD-Turbo`). | <sub></sub> |
|
|
740
|
-
| `image` | Yes | The source image file (PNG). Sent as a file in multipart/form-data. | <sub></sub> |
|
|
741
|
-
| `size` | No | The size of the output image. Format: `WIDTHxHEIGHT` (e.g., `512x512`). Default: `512x512`. | <sub></sub> |
|
|
742
|
-
| `n` | No | Number of variations to generate. Integer between 1 and 10 inclusive. Default: `1`. Values outside this range result in a 400 Bad Request error. | <sub></sub> |
|
|
743
|
-
| `response_format` | No | Format of the response. Only `b64_json` (base64-encoded image) is supported. | <sub></sub> |
|
|
744
|
-
| `user` | No | OpenAI API compatibility field. Accepted but not forwarded to the backend. | <sub></sub> |
|
|
1296
|
+
=== "PowerShell"
|
|
745
1297
|
|
|
746
|
-
|
|
1298
|
+
```powershell
|
|
1299
|
+
curl.exe http://localhost:13305/v1/images/edits `
|
|
1300
|
+
-F "model=Flux-2-Klein-4B" `
|
|
1301
|
+
-F "prompt=Add a red barn and mountains in the background, photorealistic" `
|
|
1302
|
+
-F "size=512x512" `
|
|
1303
|
+
-F "n=1" `
|
|
1304
|
+
-F "response_format=b64_json" `
|
|
1305
|
+
-F image=@image.png
|
|
1306
|
+
```
|
|
747
1307
|
|
|
748
1308
|
=== "Bash"
|
|
749
1309
|
|
|
750
1310
|
```bash
|
|
751
|
-
curl
|
|
1311
|
+
curl http://localhost:13305/v1/images/edits \
|
|
752
1312
|
-F "model=Flux-2-Klein-4B" \
|
|
1313
|
+
-F "prompt=Add a red barn and mountains in the background, photorealistic" \
|
|
753
1314
|
-F "size=512x512" \
|
|
754
1315
|
-F "n=1" \
|
|
755
1316
|
-F "response_format=b64_json" \
|
|
756
|
-
-F
|
|
1317
|
+
-F image=@image.png
|
|
757
1318
|
```
|
|
758
1319
|
|
|
759
|
-
|
|
1320
|
+
**Response:** `200 OK`
|
|
760
1321
|
|
|
761
|
-
|
|
762
|
-
|
|
763
|
-
|
|
764
|
-
|
|
765
|
-
|
|
766
|
-
|
|
767
|
-
|
|
768
|
-
|
|
769
|
-
|
|
770
|
-
|
|
771
|
-
|
|
772
|
-
|
|
1322
|
+
```json
|
|
1323
|
+
{
|
|
1324
|
+
"created": 1791943169,
|
|
1325
|
+
"data": [
|
|
1326
|
+
{
|
|
1327
|
+
"b64_json": "iVBORw0KGgoAAAANSUhEUgAAAgAAAAIACAIAAAB7GkOtAAAFR3RFWHRwYXJhbWV0ZXJzAEFkZCBhIHJl... (689,496 characters)"
|
|
1328
|
+
}
|
|
1329
|
+
],
|
|
1330
|
+
"output_format": "png"
|
|
1331
|
+
}
|
|
1332
|
+
```
|
|
1333
|
+
|
|
1334
|
+
### Response format
|
|
1335
|
+
|
|
1336
|
+
- `created` - Unix timestamp of when the image was generated.
|
|
1337
|
+
- `data` - The generated images, one entry per image.
|
|
1338
|
+
- `b64_json` - Base64-encoded PNG.
|
|
1339
|
+
- `generation_info` - Optional generation parameters and model metadata reported by the image backend.
|
|
1340
|
+
- `output_format` - File format of the images, such as `png`, when the backend reports it.
|
|
1341
|
+
|
|
1342
|
+
### Errors
|
|
1343
|
+
|
|
1344
|
+
| Status | Code | When |
|
|
1345
|
+
|--------|------|------|
|
|
1346
|
+
| `400` | | The request is not `multipart/form-data` |
|
|
1347
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
1348
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
1349
|
+
| `400` | | The form has files under both `image` and `image[]` |
|
|
1350
|
+
| `400` | | The backend's reply does not parse, or a request field not listed above has the wrong JSON type |
|
|
1351
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
1352
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
1353
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
1354
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
1355
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
1356
|
+
| `500` | | The backend reports an error, for example when `model` cannot serve this route, an error whose `type` is `unsupported_operation` |
|
|
1357
|
+
| `500` | | The request fails unexpectedly |
|
|
1358
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
1359
|
+
<!-- END GENERATED: openai.images_edits -->
|
|
1360
|
+
|
|
1361
|
+
<!-- BEGIN GENERATED: openai.images_variations -->
|
|
1362
|
+
## `POST /v1/images/variations`
|
|
1363
|
+
<sub></sub>
|
|
1364
|
+
|
|
1365
|
+
Generates a variation of a source image, sent as `multipart/form-data`. CPU inference takes minutes per image; a GPU backend is much faster.
|
|
1366
|
+
|
|
1367
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
1368
|
+
|
|
1369
|
+
### Parameters
|
|
1370
|
+
|
|
1371
|
+
| Parameter | Required | Description | Status |
|
|
1372
|
+
|-----------|----------|-------------|--------|
|
|
1373
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
1374
|
+
| `image` or `image[]` | Yes | The source image (PNG). | <sub></sub> |
|
|
1375
|
+
| `prompt` | No | Ignored; the variation depends only on the image. | <sub></sub> |
|
|
1376
|
+
| `size` | No | Output size as `WIDTHxHEIGHT`. Defaults to the model's own, else `512x512`. | <sub></sub> |
|
|
1377
|
+
| `n` | No | Number of images to generate, from `1` to `10`. Defaults to `1`. | <sub></sub> |
|
|
1378
|
+
| `response_format` | No | Only `b64_json`, a base64-encoded image, is supported. | <sub></sub> |
|
|
1379
|
+
| `user` | No | Accepted for OpenAI compatibility and ignored. | <sub></sub> |
|
|
1380
|
+
|
|
1381
|
+
### Create Variations with the OpenAI Python Client
|
|
1382
|
+
|
|
1383
|
+
=== "Python (OpenAI client)"
|
|
1384
|
+
|
|
1385
|
+
```python
|
|
1386
|
+
from openai import OpenAI
|
|
1387
|
+
client = OpenAI(base_url="http://localhost:13305/v1", api_key="not-needed")
|
|
1388
|
+
with open("source_image.png", "rb") as image_file:
|
|
1389
|
+
response = client.images.create_variation(
|
|
1390
|
+
model="Flux-2-Klein-4B",
|
|
1391
|
+
image=image_file,
|
|
1392
|
+
size="512x512",
|
|
1393
|
+
n=1,
|
|
1394
|
+
)
|
|
1395
|
+
import base64
|
|
1396
|
+
image_data = base64.b64decode(response.data[0].b64_json)
|
|
773
1397
|
open("variation.png", "wb").write(image_data)
|
|
774
1398
|
```
|
|
775
1399
|
|
|
776
|
-
|
|
1400
|
+
### Example
|
|
1401
|
+
|
|
1402
|
+
=== "PowerShell"
|
|
777
1403
|
|
|
1404
|
+
```powershell
|
|
1405
|
+
curl.exe http://localhost:13305/v1/images/variations `
|
|
1406
|
+
-F "model=Flux-2-Klein-4B" `
|
|
1407
|
+
-F "size=512x512" `
|
|
1408
|
+
-F "n=1" `
|
|
1409
|
+
-F "response_format=b64_json" `
|
|
1410
|
+
-F image=@image.png
|
|
1411
|
+
```
|
|
1412
|
+
|
|
1413
|
+
=== "Bash"
|
|
1414
|
+
|
|
1415
|
+
```bash
|
|
1416
|
+
curl http://localhost:13305/v1/images/variations \
|
|
1417
|
+
-F "model=Flux-2-Klein-4B" \
|
|
1418
|
+
-F "size=512x512" \
|
|
1419
|
+
-F "n=1" \
|
|
1420
|
+
-F "response_format=b64_json" \
|
|
1421
|
+
-F image=@image.png
|
|
1422
|
+
```
|
|
1423
|
+
|
|
1424
|
+
**Response:** `200 OK`
|
|
1425
|
+
|
|
1426
|
+
```json
|
|
1427
|
+
{
|
|
1428
|
+
"created": 1791943188,
|
|
1429
|
+
"data": [
|
|
1430
|
+
{
|
|
1431
|
+
"b64_json": "iVBORw0KGgoAAAANSUhEUgAAAgAAAAIACAIAAAB7GkOtAAAE3XRFWHRwYXJhbWV0ZXJzAHZhcmlhdGlv... (204,076 characters)"
|
|
1432
|
+
}
|
|
1433
|
+
],
|
|
1434
|
+
"output_format": "png"
|
|
1435
|
+
}
|
|
1436
|
+
```
|
|
1437
|
+
|
|
1438
|
+
### Response format
|
|
1439
|
+
|
|
1440
|
+
- `created` - Unix timestamp of when the image was generated.
|
|
1441
|
+
- `data` - The generated images, one entry per image.
|
|
1442
|
+
- `b64_json` - Base64-encoded PNG.
|
|
1443
|
+
- `generation_info` - Optional generation parameters and model metadata reported by the image backend.
|
|
1444
|
+
- `output_format` - File format of the images, such as `png`, when the backend reports it.
|
|
1445
|
+
|
|
1446
|
+
### Errors
|
|
1447
|
+
|
|
1448
|
+
| Status | Code | When |
|
|
1449
|
+
|--------|------|------|
|
|
1450
|
+
| `400` | | The request is not `multipart/form-data` |
|
|
1451
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
1452
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
1453
|
+
| `400` | | The backend's reply does not parse, or a request field not listed above has the wrong JSON type |
|
|
1454
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
1455
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
1456
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
1457
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
1458
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
1459
|
+
| `500` | | The backend reports an error, for example when `model` cannot serve this route, an error whose `type` is `unsupported_operation` |
|
|
1460
|
+
| `500` | | The request fails unexpectedly |
|
|
1461
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
1462
|
+
<!-- END GENERATED: openai.images_variations -->
|
|
1463
|
+
|
|
1464
|
+
<!-- BEGIN GENERATED: openai.images_upscale -->
|
|
778
1465
|
## `POST /v1/images/upscale`
|
|
779
1466
|
<sub></sub>
|
|
780
1467
|
|
|
781
|
-
|
|
1468
|
+
Upscales an image with an upscaling model such as `RealESRGAN-x4plus`. The model sets the default upscale factor; TheNoise models also accept an optional `upscale_factor`.
|
|
782
1469
|
|
|
783
|
-
|
|
1470
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
784
1471
|
|
|
785
1472
|
### Parameters
|
|
786
1473
|
|
|
787
1474
|
| Parameter | Required | Description | Status |
|
|
788
1475
|
|-----------|----------|-------------|--------|
|
|
789
1476
|
| `image` | Yes | Base64-encoded PNG image to upscale. | <sub></sub> |
|
|
790
|
-
| `model` | Yes | The
|
|
1477
|
+
| `model` | Yes | The upscaling model to use, such as `RealESRGAN-x4plus` or `Remacri-4x-TheNoise`. | <sub></sub> |
|
|
1478
|
+
| `upscale_factor` | No | Optional scale factor for TheNoise (up to the upscaler's native scale). | <sub></sub> |
|
|
791
1479
|
|
|
792
|
-
|
|
1480
|
+
The model is downloaded on first use.
|
|
793
1481
|
|
|
794
|
-
|
|
1482
|
+
### Upscale a Generated Image
|
|
795
1483
|
|
|
796
|
-
|
|
1484
|
+
To upscale an image from [`POST /v1/images/generations`](#post-v1imagesgenerations), pass its `b64_json` as `image`.
|
|
797
1485
|
|
|
798
|
-
|
|
799
|
-
# Step 1: Generate an image and save the base64 response
|
|
800
|
-
RESPONSE=$(curl -s -X POST http://localhost:13305/v1/images/generations \
|
|
801
|
-
-H "Content-Type: application/json" \
|
|
802
|
-
-d '{
|
|
803
|
-
"model": "SD-Turbo",
|
|
804
|
-
"prompt": "A serene mountain landscape at sunset",
|
|
805
|
-
"size": "512x512",
|
|
806
|
-
"steps": 4,
|
|
807
|
-
"response_format": "b64_json"
|
|
808
|
-
}')
|
|
809
|
-
|
|
810
|
-
# Step 2: Build the upscale JSON payload and pipe it to curl via stdin
|
|
811
|
-
# (base64 images are too large for command-line interpolation)
|
|
812
|
-
echo "$RESPONSE" | python3 -c "
|
|
813
|
-
import sys, json
|
|
814
|
-
b64 = json.load(sys.stdin)['data'][0]['b64_json']
|
|
815
|
-
print(json.dumps({'image': b64, 'model': 'RealESRGAN-x4plus'}))
|
|
816
|
-
" | curl -X POST http://localhost:13305/v1/images/upscale \
|
|
817
|
-
-H "Content-Type: application/json" \
|
|
818
|
-
-d @-
|
|
819
|
-
```
|
|
1486
|
+
### Example
|
|
820
1487
|
|
|
821
1488
|
=== "PowerShell"
|
|
822
1489
|
|
|
823
1490
|
```powershell
|
|
824
|
-
|
|
825
|
-
$
|
|
826
|
-
|
|
827
|
-
|
|
828
|
-
-Headers @{ "Content-Type" = "application/json" } `
|
|
829
|
-
-Body '{
|
|
830
|
-
"model": "SD-Turbo",
|
|
831
|
-
"prompt": "A serene mountain landscape at sunset",
|
|
832
|
-
"size": "512x512",
|
|
833
|
-
"steps": 4,
|
|
834
|
-
"response_format": "b64_json"
|
|
835
|
-
}'
|
|
836
|
-
|
|
837
|
-
# Step 2: Extract the base64 image
|
|
838
|
-
$imageB64 = ($genResponse.Content | ConvertFrom-Json).data[0].b64_json
|
|
839
|
-
|
|
840
|
-
# Step 3: Upscale the image with Real-ESRGAN
|
|
841
|
-
$body = @{ image = $imageB64; model = "RealESRGAN-x4plus" } | ConvertTo-Json
|
|
1491
|
+
$image = [Convert]::ToBase64String([IO.File]::ReadAllBytes("image.png"))
|
|
1492
|
+
$body = @"
|
|
1493
|
+
{"image": "$image", "model": "RealESRGAN-x4plus"}
|
|
1494
|
+
"@
|
|
842
1495
|
Invoke-WebRequest `
|
|
843
1496
|
-Uri "http://localhost:13305/v1/images/upscale" `
|
|
844
1497
|
-Method POST `
|
|
@@ -846,230 +1499,284 @@ A typical workflow is to generate an image first, then upscale it:
|
|
|
846
1499
|
-Body $body
|
|
847
1500
|
```
|
|
848
1501
|
|
|
849
|
-
=== "
|
|
850
|
-
|
|
851
|
-
```python
|
|
852
|
-
import requests
|
|
853
|
-
import base64
|
|
1502
|
+
=== "Bash"
|
|
854
1503
|
|
|
855
|
-
|
|
856
|
-
|
|
857
|
-
|
|
858
|
-
|
|
859
|
-
"model": "SD-Turbo",
|
|
860
|
-
"prompt": "A serene mountain landscape at sunset",
|
|
861
|
-
"size": "512x512",
|
|
862
|
-
"steps": 4,
|
|
863
|
-
"response_format": "b64_json",
|
|
864
|
-
})
|
|
865
|
-
image_b64 = gen_response.json()["data"][0]["b64_json"]
|
|
866
|
-
|
|
867
|
-
# Step 2: Upscale the image with Real-ESRGAN (512x512 -> 2048x2048)
|
|
868
|
-
upscale_response = requests.post(f"{BASE_URL}/images/upscale", json={
|
|
869
|
-
"image": image_b64,
|
|
870
|
-
"model": "RealESRGAN-x4plus",
|
|
871
|
-
})
|
|
872
|
-
|
|
873
|
-
# Step 3: Save the upscaled image to a file
|
|
874
|
-
upscaled_b64 = upscale_response.json()["data"][0]["b64_json"]
|
|
875
|
-
with open("upscaled.png", "wb") as f:
|
|
876
|
-
f.write(base64.b64decode(upscaled_b64))
|
|
1504
|
+
```bash
|
|
1505
|
+
curl http://localhost:13305/v1/images/upscale \
|
|
1506
|
+
-H "Content-Type: application/json" \
|
|
1507
|
+
-d "{\"image\": \"$(base64 -w0 image.png)\", \"model\": \"RealESRGAN-x4plus\"}"
|
|
877
1508
|
```
|
|
878
1509
|
|
|
879
|
-
|
|
1510
|
+
**Response:** `200 OK`
|
|
880
1511
|
|
|
881
1512
|
```json
|
|
882
1513
|
{
|
|
883
|
-
"created":
|
|
1514
|
+
"created": 1791943191,
|
|
884
1515
|
"data": [
|
|
885
1516
|
{
|
|
886
|
-
"b64_json": "
|
|
1517
|
+
"b64_json": "iVBORw0KGgoAAAANSUhEUgAABAAAAAQACAIAAADwf7zUAAADZHRFWHRwYXJhbWV0ZXJzAApTdGVwczog... (1,818,596 characters)"
|
|
887
1518
|
}
|
|
888
1519
|
]
|
|
889
1520
|
}
|
|
890
1521
|
```
|
|
891
1522
|
|
|
892
|
-
|
|
893
|
-
|
|
894
|
-
- `created` - Unix timestamp of when the upscaled image was generated
|
|
895
|
-
- `data` - Array containing the upscaled image
|
|
896
|
-
- `b64_json` - Base64-encoded PNG of the upscaled image
|
|
897
|
-
|
|
898
|
-
### Error responses
|
|
899
|
-
|
|
900
|
-
| Status Code | Condition | Example |
|
|
901
|
-
|-------------|-----------|---------|
|
|
902
|
-
| 400 | Missing `image` field | `{"error": {"message": "Missing 'image' field (base64 encoded)", "type": "invalid_request_error"}}` |
|
|
903
|
-
| 400 | Missing `model` field | `{"error": {"message": "Missing 'model' field", "type": "invalid_request_error"}}` |
|
|
904
|
-
| 404 | Unknown model name | `{"error": {"message": "Upscale model not found: bad-model", "type": "invalid_request_error"}}` |
|
|
905
|
-
| 500 | Upscale failed | `{"error": {"message": "ESRGAN upscale failed", "type": "server_error"}}` |
|
|
906
|
-
|
|
907
|
-
---
|
|
1523
|
+
### Response format
|
|
908
1524
|
|
|
1525
|
+
- `created` - Unix timestamp of when the upscaled image was generated.
|
|
1526
|
+
- `data` - The upscaled image.
|
|
1527
|
+
- `b64_json` - Base64-encoded PNG of the upscaled image.
|
|
1528
|
+
- `output_format` - File format of the images, such as `png`, when the backend reports it.
|
|
1529
|
+
|
|
1530
|
+
### Errors
|
|
1531
|
+
|
|
1532
|
+
| Status | Code | When |
|
|
1533
|
+
|--------|------|------|
|
|
1534
|
+
| `400` | `invalid_json` | The body is not a JSON object |
|
|
1535
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
1536
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
1537
|
+
| `400` | | `model` has no `upscaling` label |
|
|
1538
|
+
| `400` | | `model`'s recipe cannot upscale |
|
|
1539
|
+
| `400` | | The backend's reply does not parse, or a request field not listed above has the wrong JSON type |
|
|
1540
|
+
| `404` | | `model` names no model in the registry, or downloading it fails |
|
|
1541
|
+
| `500` | | The backend produces no upscaled image |
|
|
1542
|
+
| `500` | | The request fails unexpectedly |
|
|
1543
|
+
<!-- END GENERATED: openai.images_upscale -->
|
|
1544
|
+
|
|
1545
|
+
<!-- BEGIN GENERATED: openai.audio_speech -->
|
|
909
1546
|
## `POST /v1/audio/speech`
|
|
910
|
-
<sub></sub>
|
|
911
1548
|
|
|
912
|
-
|
|
1549
|
+
Generates speech audio from text. The model selects the engine: `kokoro-v1` speaks with fixed voices ([Kokoros](https://github.com/lucasjinreal/Kokoros)), while `OpenMOSS-TTS` and `MOSS-TTS-Local` can also clone a voice or design one from a description. `MOSS-VoiceGen` is kept for compatibility.
|
|
913
1550
|
|
|
914
|
-
|
|
915
|
-
>
|
|
916
|
-
> **Limitations:** Which `response_format` values are accepted depends on the model's backend: `kokoro-v1` encodes `mp3`, `wav`, `opus`, and `pcm`; OpenMOSS v0.3 encodes buffered `wav` or `pcm`. Native streaming is narrower for both backends and uses `pcm` only, so an explicit non-PCM `response_format` on a streaming request is rejected rather than mislabeled or silently transcoded. OpenMOSS raw PCM is returned as `audio/pcm` with `X-MOSS-Sample-Rate` and `X-MOSS-Channels`, because its native format is model-dependent (24 kHz mono for OpenMOSS-TTS and 48 kHz stereo for MOSS-TTS-Local).
|
|
1551
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
917
1552
|
|
|
918
1553
|
### Parameters
|
|
919
1554
|
|
|
920
1555
|
| Parameter | Required | Description | Status |
|
|
921
1556
|
|-----------|----------|-------------|--------|
|
|
922
1557
|
| `input` | Yes | The text to speak. | <sub></sub> |
|
|
923
|
-
| `model` | Yes | The model to
|
|
924
|
-
| `speed` | No | Speaking speed.
|
|
925
|
-
| `voice` | No |
|
|
926
|
-
| `voice` (OpenMOSS) | No | For OpenMOSS models the field is a free-text voice/style instruction instead of a fixed voice name (e.g. `a calm, deep male narrator voice`). | <sub></sub> |
|
|
1558
|
+
| `model` | Yes | The model to run; loaded on first use. | <sub></sub> |
|
|
1559
|
+
| `speed` | No | Speaking speed. Defaults to `1.0`. | <sub></sub> |
|
|
1560
|
+
| `voice` | No | For `kokoro-v1`, a voice name: OpenAI's voices (`alloy`, `ash`, ...) or Kokoro's own (`af_sky`, `am_echo`, ...), defaulting to `shimmer`. For OpenMOSS models, a free-text voice or style instruction, e.g. `a calm, deep male narrator voice`. | <sub></sub> |
|
|
927
1561
|
| `reference_wav_b64` | No | Lemonade extension (OpenMOSS voice cloning): base64-encoded WAV sample of the voice to clone. | <sub></sub> |
|
|
928
|
-
| `voice_design_description` | No | Lemonade extension (OpenMOSS voice design): a description of the voice to invent, e.g. `a warm low female voice with a British accent`. Lemonade renders a short sample in that voice and uses it as the reference, so the effect is the same as supplying `reference_wav_b64` yourself. Ignored when `reference_wav_b64` is also present.
|
|
929
|
-
| `response_format` | No | Container for the returned audio.
|
|
930
|
-
| `stream_format` | No |
|
|
1562
|
+
| `voice_design_description` | No | Lemonade extension (OpenMOSS voice design): a description of the voice to invent, e.g. `a warm low female voice with a British accent`. Lemonade renders a short sample in that voice and uses it as the reference, so the effect is the same as supplying `reference_wav_b64` yourself. Ignored when `reference_wav_b64` is also present. Only this field triggers design; `voice` never does. | <sub></sub> |
|
|
1563
|
+
| `response_format` | No | Container for the returned audio. Defaults to `mp3` when buffered and `pcm` when streaming, or to the backend's first supported format when it cannot encode those. | <sub></sub> |
|
|
1564
|
+
| `stream_format` | No | `audio` streams the response. It selects the transport only: the container still comes from `response_format`. | <sub></sub> |
|
|
1565
|
+
| `stream` | No | `true` streams the response, like `stream_format: audio`. | <sub></sub> |
|
|
1566
|
+
|
|
1567
|
+
Accepted `response_format` values depend on the backend: `kokoro-v1` encodes `mp3`, `wav`, `opus` and `pcm`, and OpenMOSS encodes `wav` or `pcm`. Streaming produces `pcm` only. OpenMOSS PCM is returned as `audio/pcm` with `X-MOSS-Sample-Rate` and `X-MOSS-Channels` headers, since its format depends on the model (24 kHz mono for `OpenMOSS-TTS`, 48 kHz stereo for `MOSS-TTS-Local`).
|
|
1568
|
+
|
|
1569
|
+
### Examples
|
|
931
1570
|
|
|
932
|
-
|
|
1571
|
+
#### Non-streaming
|
|
1572
|
+
|
|
1573
|
+
=== "PowerShell"
|
|
1574
|
+
|
|
1575
|
+
```powershell
|
|
1576
|
+
Invoke-WebRequest `
|
|
1577
|
+
-Uri "http://localhost:13305/v1/audio/speech" `
|
|
1578
|
+
-Method POST `
|
|
1579
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
1580
|
+
-Body '{
|
|
1581
|
+
"model": "kokoro-v1",
|
|
1582
|
+
"input": "Lemonade can speak!",
|
|
1583
|
+
"speed": 1.0,
|
|
1584
|
+
"response_format": "mp3"
|
|
1585
|
+
}' `
|
|
1586
|
+
-OutFile output.mp3
|
|
1587
|
+
```
|
|
933
1588
|
|
|
934
1589
|
=== "Bash"
|
|
935
1590
|
|
|
936
1591
|
```bash
|
|
937
|
-
curl
|
|
1592
|
+
curl http://localhost:13305/v1/audio/speech \
|
|
938
1593
|
-H "Content-Type: application/json" \
|
|
939
1594
|
-d '{
|
|
940
|
-
|
|
941
|
-
|
|
942
|
-
|
|
943
|
-
|
|
944
|
-
|
|
945
|
-
|
|
1595
|
+
"model": "kokoro-v1",
|
|
1596
|
+
"input": "Lemonade can speak!",
|
|
1597
|
+
"speed": 1.0,
|
|
1598
|
+
"response_format": "mp3"
|
|
1599
|
+
}' \
|
|
1600
|
+
--output output.mp3
|
|
946
1601
|
```
|
|
947
1602
|
|
|
948
|
-
|
|
1603
|
+
**Response:** `200 OK`, `audio/mpeg`, 40,516 bytes
|
|
1604
|
+
|
|
1605
|
+
#### Streaming
|
|
1606
|
+
|
|
1607
|
+
=== "PowerShell"
|
|
1608
|
+
|
|
1609
|
+
```powershell
|
|
1610
|
+
Invoke-WebRequest `
|
|
1611
|
+
-Uri "http://localhost:13305/v1/audio/speech" `
|
|
1612
|
+
-Method POST `
|
|
1613
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
1614
|
+
-Body '{
|
|
1615
|
+
"model": "kokoro-v1",
|
|
1616
|
+
"input": "Lemonade can speak!",
|
|
1617
|
+
"speed": 1.0,
|
|
1618
|
+
"stream_format": "audio"
|
|
1619
|
+
}' `
|
|
1620
|
+
-OutFile output.l16
|
|
1621
|
+
```
|
|
949
1622
|
|
|
950
|
-
|
|
1623
|
+
=== "Bash"
|
|
951
1624
|
|
|
1625
|
+
```bash
|
|
1626
|
+
curl http://localhost:13305/v1/audio/speech \
|
|
1627
|
+
-H "Content-Type: application/json" \
|
|
1628
|
+
-d '{
|
|
1629
|
+
"model": "kokoro-v1",
|
|
1630
|
+
"input": "Lemonade can speak!",
|
|
1631
|
+
"speed": 1.0,
|
|
1632
|
+
"stream_format": "audio"
|
|
1633
|
+
}' \
|
|
1634
|
+
--output output.l16
|
|
1635
|
+
```
|
|
952
1636
|
|
|
1637
|
+
**Response:** `200 OK`, `audio/l16;rate=24000;endianness=little-endian`, 97,200 bytes
|
|
1638
|
+
|
|
1639
|
+
### Errors
|
|
1640
|
+
|
|
1641
|
+
| Status | Code | When |
|
|
1642
|
+
|--------|------|------|
|
|
1643
|
+
| `400` | `invalid_json` | The body is not a JSON object |
|
|
1644
|
+
| `400` | `missing_argument` | A required argument is missing |
|
|
1645
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
1646
|
+
| `400` | `model_not_applicable` | `model` is not a text-to-speech model, which is checked before it loads |
|
|
1647
|
+
| `400` | | `response_format` is not a format `model`'s backend produces, or, when streaming, one it streams |
|
|
1648
|
+
| `400` | `model_load_refused` | The backend refuses the requested model as it is registered, before it starts anything, such as a decision model whose GGUF type the bundled llama.cpp does not support |
|
|
1649
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
1650
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
1651
|
+
| `409` | `router_residency_conflict` | Loading the requested model would evict a model that a `collection.router` model needs resident on the same NPU |
|
|
1652
|
+
| `409` | `slots_pinned_error` | Every loaded model of the requested model's type is pinned, so none can be unloaded to make room |
|
|
1653
|
+
| `500` | | The request fails unexpectedly |
|
|
1654
|
+
| `500` | | The backend reports an error; the backend's own status replaces this one when it sends one |
|
|
1655
|
+
| `500` | `model_load_error` | Loading the requested model fails for another reason, such as a failed download or a backend that does not start |
|
|
1656
|
+
| `502` | | The backend produces no output, for example after it crashes or runs out of GPU memory |
|
|
1657
|
+
<!-- END GENERATED: openai.audio_speech -->
|
|
1658
|
+
|
|
1659
|
+
<!-- BEGIN GENERATED: openai.models -->
|
|
953
1660
|
## `GET /v1/models`
|
|
954
1661
|
<sub></sub>
|
|
955
1662
|
|
|
956
|
-
|
|
957
|
-
|
|
958
|
-
By default, only models available locally (downloaded) are shown, matching OpenAI API behavior.
|
|
1663
|
+
Lists the models available on the server, in OpenAI's format extended with the fields described below.
|
|
959
1664
|
|
|
960
|
-
|
|
1665
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
961
1666
|
|
|
962
1667
|
### Parameters
|
|
963
1668
|
|
|
964
|
-
| Parameter | Required | Description |
|
|
965
|
-
|
|
966
|
-
| `show_all` | No |
|
|
1669
|
+
| Parameter | Required | Description | Status |
|
|
1670
|
+
|-----------|----------|-------------|--------|
|
|
1671
|
+
| `show_all` (query) | No | `true` also lists catalog models that are not downloaded. Defaults to `false`, which lists downloaded models only. | <sub></sub> |
|
|
1672
|
+
| `include_paths` (query) | No | `true` adds each present checkpoint's absolute `path` to its `checkpoint_status` entry, which can reveal local usernames and the cache layout. Defaults to `false`. The files a model is made of have their own list, at [`GET /v1/models/{id}/files`](./openai.md#get-v1modelsidfiles). | <sub></sub> |
|
|
967
1673
|
|
|
968
|
-
|
|
1674
|
+
A model alias whose target is listed appears too, as a copy of the target with the alias as its `id`.
|
|
969
1675
|
|
|
970
|
-
|
|
971
|
-
# Show only downloaded models (OpenAI-compatible)
|
|
972
|
-
curl http://localhost:13305/v1/models
|
|
1676
|
+
With cloud providers configured, their models are listed too, with `recipe: "cloud"`, a `cloud_provider` field and a provider-prefixed id such as `fireworks.kimi-k2p5`. Models of `openai`-wire providers accept chat completions and completions requests; `anthropic`-wire providers are served through [`POST /v1/messages`](./anthropic.md#post-v1messages). See [Cloud Offload](../guide/configuration/cloud.md).
|
|
973
1677
|
|
|
974
|
-
|
|
975
|
-
curl http://localhost:13305/v1/models?show_all=true
|
|
976
|
-
```
|
|
1678
|
+
`HEAD` returns `200 OK` with an empty body.
|
|
977
1679
|
|
|
978
|
-
###
|
|
1680
|
+
### Example
|
|
1681
|
+
|
|
1682
|
+
=== "PowerShell"
|
|
1683
|
+
|
|
1684
|
+
```powershell
|
|
1685
|
+
Invoke-WebRequest -Uri "http://localhost:13305/v1/models"
|
|
1686
|
+
```
|
|
1687
|
+
|
|
1688
|
+
=== "Bash"
|
|
1689
|
+
|
|
1690
|
+
```bash
|
|
1691
|
+
curl http://localhost:13305/v1/models
|
|
1692
|
+
```
|
|
1693
|
+
|
|
1694
|
+
**Response:** `200 OK`
|
|
979
1695
|
|
|
980
1696
|
```json
|
|
981
1697
|
{
|
|
982
1698
|
"object": "list",
|
|
983
1699
|
"data": [
|
|
984
1700
|
{
|
|
985
|
-
"id": "
|
|
986
|
-
"created":
|
|
1701
|
+
"id": "Phishing-Email-Detection-ONNX",
|
|
1702
|
+
"created": 1234567890,
|
|
987
1703
|
"object": "model",
|
|
988
1704
|
"owned_by": "lemonade",
|
|
989
|
-
"checkpoint": "
|
|
990
|
-
"recipe": "
|
|
991
|
-
"size": 0.
|
|
992
|
-
"max_context_window": 40960,
|
|
993
|
-
"context_length": 8192,
|
|
1705
|
+
"checkpoint": "lemonade-sdk/phishing-email-detection-distilbert-ONNX",
|
|
1706
|
+
"recipe": "onnxruntime",
|
|
1707
|
+
"size": 0.25,
|
|
994
1708
|
"downloaded": true,
|
|
995
|
-
"suggested": true,
|
|
996
1709
|
"update_available": false,
|
|
997
|
-
"
|
|
1710
|
+
"suggested": false,
|
|
1711
|
+
"labels": ["classification"],
|
|
1712
|
+
"components": [],
|
|
1713
|
+
"checkpoints": {"main": "lemonade-sdk/phishing-email-detection-distilbert-ONNX"},
|
|
1714
|
+
"checkpoint_status": [
|
|
1715
|
+
{"type": "main", "present": true, "required": true, "registry": "huggingface"}
|
|
1716
|
+
],
|
|
1717
|
+
"source": "huggingface",
|
|
1718
|
+
"registry_source": "huggingface",
|
|
1719
|
+
"recipe_options": {}
|
|
998
1720
|
},
|
|
999
|
-
|
|
1000
|
-
"id": "Gemma-3-4b-it-GGUF",
|
|
1001
|
-
"created": 1744173590,
|
|
1002
|
-
"object": "model",
|
|
1003
|
-
"owned_by": "lemonade",
|
|
1004
|
-
"checkpoint": "ggml-org/gemma-3-4b-it-GGUF:Q4_K_M",
|
|
1005
|
-
"recipe": "llamacpp",
|
|
1006
|
-
"size": 3.61,
|
|
1007
|
-
"downloaded": true,
|
|
1008
|
-
"suggested": true,
|
|
1009
|
-
"labels": ["hot", "vision"]
|
|
1010
|
-
},
|
|
1011
|
-
{
|
|
1012
|
-
"id": "SD-Turbo",
|
|
1013
|
-
"created": 1744173590,
|
|
1014
|
-
"object": "model",
|
|
1015
|
-
"owned_by": "lemonade",
|
|
1016
|
-
"checkpoint": "stabilityai/sd-turbo:sd_turbo.safetensors",
|
|
1017
|
-
"recipe": "sd-cpp",
|
|
1018
|
-
"size": 5.2,
|
|
1019
|
-
"downloaded": true,
|
|
1020
|
-
"suggested": true,
|
|
1021
|
-
"labels": ["image"],
|
|
1022
|
-
"image_defaults": {
|
|
1023
|
-
"steps": 4,
|
|
1024
|
-
"cfg_scale": 1.0,
|
|
1025
|
-
"width": 512,
|
|
1026
|
-
"height": 512
|
|
1027
|
-
}
|
|
1028
|
-
}
|
|
1721
|
+
... 10 more
|
|
1029
1722
|
]
|
|
1030
1723
|
}
|
|
1031
1724
|
```
|
|
1032
1725
|
|
|
1033
|
-
|
|
1034
|
-
|
|
1035
|
-
- `object` - Type of response object, always `"list"`
|
|
1036
|
-
- `data` - Array of model objects with the following fields:
|
|
1037
|
-
- `id` - Model identifier (used for loading and inference requests)
|
|
1038
|
-
- `created` - Unix timestamp of when the model entry was created
|
|
1039
|
-
- `object` - Type of object, always `"model"`
|
|
1040
|
-
- `owned_by` - Owner of the model, always `"lemonade"`
|
|
1041
|
-
- `checkpoint` - Full checkpoint identifier on Hugging Face
|
|
1042
|
-
- `recipe` - Backend/device recipe used to load the model (e.g., `"ryzenai-llm"`, `"llamacpp"`, `"flm"`)
|
|
1043
|
-
- `size` - Model size in GB (omitted for models without size information)
|
|
1044
|
-
- `max_context_window` - Optional integer indicating the maximum model-supported text context discovered from local static metadata. Currently populated for downloaded GGUF/llama.cpp models and installed FLM text-context models.
|
|
1045
|
-
- `context_length` - Number of tokens the model can handle in one request. Uses the loaded value when the model is running and the configured `ctx_size` otherwise (omitted when neither is known).
|
|
1046
|
-
- `downloaded` - Boolean indicating if the model is downloaded and available locally
|
|
1047
|
-
- `update_available` - Boolean indicating a newer commit exists on HuggingFace for this model. Only set for downloaded HF-backed models. `false` otherwise.
|
|
1048
|
-
- `suggested` - Boolean indicating if the model is recommended for general use
|
|
1049
|
-
- `labels` - Array of tags describing the model's capabilities and characteristics. See [Model Labels](#model-labels) for the full list.
|
|
1050
|
-
- `image_defaults` - (Image models only) Default generation parameters for the model:
|
|
1051
|
-
- `steps` - Number of inference steps (e.g., 4 for turbo models, 20 for standard models)
|
|
1052
|
-
- `cfg_scale` - Classifier-free guidance scale (e.g., 1.0 for turbo models, 7.5 for standard models)
|
|
1053
|
-
- `width` - Default image width in pixels
|
|
1054
|
-
- `height` - Default image height in pixels
|
|
1055
|
-
- `components` - (Omni collections only, `recipe: "collection.omni"`) Ordered array of the component model names that make up the collection
|
|
1056
|
-
- `models` - (Omni collections only) Ordered array embedding each component's full model object (same shape as the entries in this list), parallel to `components`. This makes a collection's `/v1/models/{model_id}` response self-contained — exporting it produces a file that can be imported elsewhere via [`/v1/pull`](./lemonade.md#post-v1pull)
|
|
1726
|
+
### Response format
|
|
1057
1727
|
|
|
1728
|
+
- `object` - Always `"list"`.
|
|
1729
|
+
- `data` - The model objects.
|
|
1730
|
+
- `id` - Model identifier for loading and inference requests.
|
|
1731
|
+
- `created` - Always `1234567890`, for OpenAI compatibility.
|
|
1732
|
+
- `object` - Always `"model"`.
|
|
1733
|
+
- `owned_by` - Always `"lemonade"`.
|
|
1734
|
+
- `checkpoint` - The main checkpoint in the model's registry; see `registry_source`.
|
|
1735
|
+
- `recipe` - Recipe used to load the model, e.g. `llamacpp` or `flm`.
|
|
1736
|
+
- `size` - Size in GB; omitted when unknown.
|
|
1737
|
+
- `max_context_window` - Largest text context the model supports, read from local metadata: downloaded GGUF models and installed FLM text models.
|
|
1738
|
+
- `context_length` - Tokens the model handles in one request: the loaded `ctx_size` while it runs, else the configured `ctx_size`, else `max_context_window`; omitted when none is known.
|
|
1739
|
+
- `max_output_tokens` - Cloud models only: the most tokens one completion can generate, when the provider reports it.
|
|
1740
|
+
- `max_completion_tokens` - Cloud models only: `max_output_tokens` under its OpenAI name.
|
|
1741
|
+
- `downloaded` - Whether the model is downloaded.
|
|
1742
|
+
- `update_available` - Whether the model's registry has a newer commit; `false` except for downloaded registry models.
|
|
1743
|
+
- `suggested` - Whether the model is recommended for general use.
|
|
1744
|
+
- `labels` - The model's capabilities and characteristics; see [Model Labels](./openai.md#model-labels).
|
|
1745
|
+
- `components` - The names of a collection's component models, in order; empty for other models.
|
|
1746
|
+
- `models` - Omni collections only: each component's full model object, in the order of `components`, so an exported collection can be imported elsewhere through [`/v1/pull`](./lemonade.md#post-v1pull).
|
|
1747
|
+
- `checkpoints` - Every checkpoint by role, such as `main`, `mmproj`, `draft`, `text_encoder` or `vae`.
|
|
1748
|
+
- The checkpoint that fills the role named by its key.
|
|
1749
|
+
- `checkpoint_status` - One entry per checkpoint of the model, answering whether its bytes are there.
|
|
1750
|
+
- `type` - Checkpoint slot, e.g. `main`, `mmproj`, `vae`.
|
|
1751
|
+
- `present` - `true` only when the checkpoint's bytes are complete and readable.
|
|
1752
|
+
- `required` - A missing required checkpoint keeps the model undownloaded; an optional one, loaded lazily, does not.
|
|
1753
|
+
- `registry` - Registry hosting the checkpoint; empty for local files and directories.
|
|
1754
|
+
- `path` - Absolute path to the checkpoint file; present only with `include_paths=true` and when `present` is `true`.
|
|
1755
|
+
- `source` - Where the model came from: a registry name, `local_upload`, `local_path` or `extra_models_dir`.
|
|
1756
|
+
- `registry_source` - Remote registry the checkpoint downloads from: `huggingface` or `modelscope`.
|
|
1757
|
+
- `recipe_options` - The model's own options: those its registry row authors, with what was saved in `recipe_options.json` on top.
|
|
1758
|
+
- `cloud_provider` - Cloud models only: the provider that serves the model.
|
|
1759
|
+
- `cost_input_per_million` - Cloud models only: USD per million input tokens, when the provider reports it.
|
|
1760
|
+
- `cost_output_per_million` - Cloud models only: USD per million output tokens, when the provider reports it.
|
|
1761
|
+
- `system_prompt` - Omni collections only: the collection's system prompt override.
|
|
1762
|
+
- `version` - Router collections only: the policy document's version.
|
|
1763
|
+
- `routing` - Router collections only: the routing policy.
|
|
1764
|
+
<!-- END GENERATED: openai.models -->
|
|
1058
1765
|
|
|
1059
1766
|
### Model Labels
|
|
1060
1767
|
|
|
1061
1768
|
Labels describe what a model can do. A model may carry multiple labels.
|
|
1062
1769
|
|
|
1063
|
-
**Deployment labels
|
|
1770
|
+
**Deployment labels**: determine which backend endpoint the model is routed to.
|
|
1064
1771
|
Every model names exactly one deployment mode, and a model is never given two
|
|
1065
1772
|
labels that name different modes:
|
|
1066
1773
|
|
|
1067
1774
|
| Label | Endpoint | Description |
|
|
1068
1775
|
|-------|----------|-------------|
|
|
1069
|
-
| `chat` | `/chat/completions`, `/completions`, `/responses` | Text-generating LLM. This label is what makes a model an LLM
|
|
1070
|
-
| `transcription` | `/audio/transcriptions` | Speech-to-text transcription model (e.g. Whisper). An omni LLM that accepts audio in a chat turn is not one of these
|
|
1776
|
+
| `chat` | `/chat/completions`, `/completions`, `/responses` | Text-generating LLM. This label is what makes a model an LLM; it is not inferred from `reasoning`/`vision`/`tool-calling`/`chat-transcription`, which are characteristics rather than deployment modes. |
|
|
1777
|
+
| `transcription` | `/audio/transcriptions` | Speech-to-text transcription model (e.g. Whisper). An omni LLM that accepts audio in a chat turn is not one of these: it carries `chat` and the `chat-transcription` capability below. |
|
|
1071
1778
|
| `embeddings` | `/embeddings` | Produces text embedding vectors. Also accepted as `embedding`. |
|
|
1072
|
-
| `reranking` | `/rerank` | Scores and reranks a list of passages given a query.
|
|
1779
|
+
| `reranking` | `/rerank` | Scores and reranks a list of passages given a query. |
|
|
1073
1780
|
| `image` | `/images/generations`, `/images/edits`, `/images/variations` | Text-to-image generation model. |
|
|
1074
1781
|
| `tts` | `/audio/speech` | Text-to-speech synthesis model. |
|
|
1075
1782
|
| `audio-generation` | `/audio/generations` | Text-to-audio generation model (e.g. music, sound effects). |
|
|
@@ -1077,16 +1784,17 @@ labels that name different modes:
|
|
|
1077
1784
|
| `3d` | `/3d/generations` | Text- or image-to-3D mesh generation model. |
|
|
1078
1785
|
|
|
1079
1786
|
When a model declares no deployment label at all, it inherits its recipe's
|
|
1080
|
-
default
|
|
1787
|
+
default: `chat` for `llamacpp`, `flm`, `ryzenai-llm`, `vllm` and `cloud`,
|
|
1081
1788
|
`transcription` for `whispercpp`, `image` for `sd-cpp`, `tts` for `kokoro`, and
|
|
1082
1789
|
so on.
|
|
1083
1790
|
|
|
1084
1791
|
Two label sets describe a model that cannot exist, and are refused rather than
|
|
1085
1792
|
repaired:
|
|
1086
1793
|
|
|
1087
|
-
- **A mode the recipe's backend does not serve.** `/classify` is served
|
|
1088
|
-
`onnxruntime`,
|
|
1089
|
-
|
|
1794
|
+
- **A mode the recipe's backend does not serve.** `/classify` is served by
|
|
1795
|
+
`onnxruntime`, and by `llamacpp` only for a model that also has the
|
|
1796
|
+
`systemone` label. `labels: ["classification"]` on a `llamacpp` model without
|
|
1797
|
+
`systemone` is an error; register it as the chat model it is.
|
|
1090
1798
|
- **Two different modes.** `labels: ["chat", "embeddings"]` on a `llamacpp` model
|
|
1091
1799
|
is an error even though llama.cpp serves both: the subprocess is launched for
|
|
1092
1800
|
one mode, so the second would name an endpoint it was never configured to
|
|
@@ -1094,37 +1802,48 @@ repaired:
|
|
|
1094
1802
|
booleans count as mode claims here, exactly as the labels do.
|
|
1095
1803
|
|
|
1096
1804
|
[`POST /v1/pull`](./lemonade.md#post-v1pull) answers `400` and registers nothing.
|
|
1097
|
-
|
|
1098
|
-
|
|
1099
|
-
can be corrected by hand.
|
|
1805
|
+
A stored `user_models.json` entry that breaks these rules is skipped at startup with
|
|
1806
|
+
an error naming it; the file is left for you to correct.
|
|
1100
1807
|
|
|
1101
|
-
**Input-modality labels
|
|
1808
|
+
**Input-modality labels**: the model accepts additional input types in `/chat/completions`:
|
|
1102
1809
|
|
|
1103
1810
|
| Label | Description |
|
|
1104
1811
|
|-------|-------------|
|
|
1105
1812
|
| `vision` | Accepts image attachments in chat messages. |
|
|
1106
|
-
| `chat-transcription` | Accepts audio attachments in chat messages and transcribes them as part of its answer (e.g. Qwen2.5-Omni). Like `vision`, this is something a chat model can do, not a deployment mode of its own
|
|
1813
|
+
| `chat-transcription` | Accepts audio attachments in chat messages and transcribes them as part of its answer (e.g. Qwen2.5-Omni). Like `vision`, this is something a chat model can do, not a deployment mode of its own; a model carrying it also carries `chat`. It is distinct from `transcription`, which deploys a dedicated ASR model on `/audio/transcriptions`. |
|
|
1107
1814
|
|
|
1108
|
-
**
|
|
1815
|
+
**Classification capability labels:** carried alongside `classification`:
|
|
1816
|
+
|
|
1817
|
+
| Label | Description |
|
|
1818
|
+
|-------|-------------|
|
|
1819
|
+
| `zero-shot` | Takes its labels from each `/classify` request instead of a fixed label set (e.g. `LFM2.5-Encoder-350M-Prompt-Router-ONNX`). A router policy must use such a model in a `zero_shot` classifier, and a fixed-label model in a `classifier`; registration refuses either mismatch. |
|
|
1820
|
+
|
|
1821
|
+
**Streaming labels:** capability flags for real-time features:
|
|
1109
1822
|
|
|
1110
1823
|
| Label | Description |
|
|
1111
1824
|
|-------|-------------|
|
|
1112
1825
|
| `realtime-transcription` | Supports the WebSocket `/realtime` endpoint for live microphone transcription. |
|
|
1113
1826
|
|
|
1114
|
-
**Runtime labels
|
|
1827
|
+
**Runtime labels:** affect backend launch defaults:
|
|
1115
1828
|
|
|
1116
1829
|
| Label | Description |
|
|
1117
1830
|
|-------|-------------|
|
|
1118
1831
|
| `mtp` | Enables llama.cpp MTP draft decoding defaults (`--spec-type draft-mtp --spec-draft-n-max 3 --spec-draft-p-min 0.75`); users can override these with `llamacpp_args`. |
|
|
1119
1832
|
|
|
1120
|
-
**
|
|
1833
|
+
**Execution labels:** control how a model is invoked:
|
|
1834
|
+
|
|
1835
|
+
| Label | Description |
|
|
1836
|
+
|-------|-------------|
|
|
1837
|
+
| `on-demand` | Runs through a task-specific endpoint without a resident model server, so `/v1/load` is not supported. Built-in upscalers use `/v1/images/upscale`. |
|
|
1838
|
+
|
|
1839
|
+
**Image capability labels:** carried alongside `image`; they refine what the model is offered for without changing its deployment mode:
|
|
1121
1840
|
|
|
1122
1841
|
| Label | Description |
|
|
1123
1842
|
|-------|-------------|
|
|
1124
1843
|
| `edit` | Tuned for editing an input image (`/images/edits`). Also selects the model for the `edit_image` role in an omni collection. |
|
|
1125
1844
|
| `upscaling` | Image upscaling model (e.g. Real-ESRGAN, `/images/upscale`). Used as a component in image pipelines rather than offered on its own. |
|
|
1126
1845
|
|
|
1127
|
-
**Characteristic labels
|
|
1846
|
+
**Characteristic labels:** informational, do not affect routing:
|
|
1128
1847
|
|
|
1129
1848
|
| Label | Description |
|
|
1130
1849
|
|-------|-------------|
|
|
@@ -1132,61 +1851,442 @@ can be corrected by hand.
|
|
|
1132
1851
|
| `reasoning` | Uses extended chain-of-thought reasoning (e.g. DeepSeek, Qwen3). |
|
|
1133
1852
|
| `tool-calling` | Supports function/tool calling in chat completions. |
|
|
1134
1853
|
| `coding` | Tuned for code generation and software tasks. |
|
|
1135
|
-
| `experimental` | Not yet validated for production use. |
|
|
1136
|
-
|
|
1137
1854
|
|
|
1138
|
-
|
|
1855
|
+
<!-- BEGIN GENERATED: openai.models_id -->
|
|
1856
|
+
## `GET /v1/models/{id}`
|
|
1139
1857
|
<sub></sub>
|
|
1140
1858
|
|
|
1141
|
-
|
|
1859
|
+
Returns one model, as the same model object [`GET /v1/models`](#get-v1models) lists.
|
|
1860
|
+
|
|
1861
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
1142
1862
|
|
|
1143
1863
|
### Parameters
|
|
1144
1864
|
|
|
1145
|
-
| Parameter | Required | Description |
|
|
1146
|
-
|
|
1147
|
-
| `
|
|
1865
|
+
| Parameter | Required | Description | Status |
|
|
1866
|
+
|-----------|----------|-------------|--------|
|
|
1867
|
+
| `id` (path) | Yes | Model id, as listed by [`GET /v1/models`](#get-v1models), or an alias. A trailing `:latest`, the tag Ollama clients append, names the bare id when nothing is registered under the tagged name. | <sub></sub> |
|
|
1868
|
+
| `include_paths` (query) | No | `true` adds each present checkpoint's absolute `path` to its `checkpoint_status` entry, which can reveal local usernames and the cache layout. Defaults to `false`. The files a model is made of have their own list, at [`GET /v1/models/{id}/files`](./openai.md#get-v1modelsidfiles). | <sub></sub> |
|
|
1148
1869
|
|
|
1149
|
-
### Example
|
|
1870
|
+
### Example
|
|
1150
1871
|
|
|
1151
|
-
|
|
1152
|
-
curl http://localhost:13305/v1/models/Qwen3-0.6B-GGUF
|
|
1153
|
-
```
|
|
1872
|
+
=== "PowerShell"
|
|
1154
1873
|
|
|
1155
|
-
|
|
1874
|
+
```powershell
|
|
1875
|
+
Invoke-WebRequest -Uri "http://localhost:13305/v1/models/Qwen3-0.6B-GGUF"
|
|
1876
|
+
```
|
|
1877
|
+
|
|
1878
|
+
=== "Bash"
|
|
1156
1879
|
|
|
1157
|
-
|
|
1880
|
+
```bash
|
|
1881
|
+
curl http://localhost:13305/v1/models/Qwen3-0.6B-GGUF
|
|
1882
|
+
```
|
|
1883
|
+
|
|
1884
|
+
**Response:** `200 OK`
|
|
1158
1885
|
|
|
1159
1886
|
```json
|
|
1160
1887
|
{
|
|
1161
1888
|
"id": "Qwen3-0.6B-GGUF",
|
|
1162
|
-
"created":
|
|
1889
|
+
"created": 1234567890,
|
|
1163
1890
|
"object": "model",
|
|
1164
1891
|
"owned_by": "lemonade",
|
|
1165
1892
|
"checkpoint": "unsloth/Qwen3-0.6B-GGUF:Q4_0",
|
|
1166
1893
|
"recipe": "llamacpp",
|
|
1167
|
-
"size": 0.
|
|
1894
|
+
"size": 0.356,
|
|
1168
1895
|
"max_context_window": 40960,
|
|
1169
|
-
"context_length":
|
|
1896
|
+
"context_length": 40960,
|
|
1170
1897
|
"downloaded": true,
|
|
1898
|
+
"update_available": false,
|
|
1171
1899
|
"suggested": true,
|
|
1172
|
-
"labels": ["reasoning"],
|
|
1173
|
-
"
|
|
1900
|
+
"labels": ["chat", "reasoning", "tool-calling"],
|
|
1901
|
+
"components": [],
|
|
1902
|
+
"checkpoints": {"main": "unsloth/Qwen3-0.6B-GGUF:Q4_0"},
|
|
1903
|
+
"checkpoint_status": [
|
|
1904
|
+
{"type": "main", "present": true, "required": true, "registry": "huggingface"}
|
|
1905
|
+
],
|
|
1906
|
+
"source": "huggingface",
|
|
1907
|
+
"registry_source": "huggingface",
|
|
1908
|
+
"recipe_options": {}
|
|
1909
|
+
}
|
|
1910
|
+
```
|
|
1911
|
+
|
|
1912
|
+
### Response format
|
|
1913
|
+
|
|
1914
|
+
- `id` - Model identifier for loading and inference requests.
|
|
1915
|
+
- `created` - Always `1234567890`, for OpenAI compatibility.
|
|
1916
|
+
- `object` - Always `"model"`.
|
|
1917
|
+
- `owned_by` - Always `"lemonade"`.
|
|
1918
|
+
- `checkpoint` - The main checkpoint in the model's registry; see `registry_source`.
|
|
1919
|
+
- `recipe` - Recipe used to load the model, e.g. `llamacpp` or `flm`.
|
|
1920
|
+
- `size` - Size in GB; omitted when unknown.
|
|
1921
|
+
- `max_context_window` - Largest text context the model supports, read from local metadata: downloaded GGUF models and installed FLM text models.
|
|
1922
|
+
- `context_length` - Tokens the model handles in one request: the loaded `ctx_size` while it runs, else the configured `ctx_size`, else `max_context_window`; omitted when none is known.
|
|
1923
|
+
- `max_output_tokens` - Cloud models only: the most tokens one completion can generate, when the provider reports it.
|
|
1924
|
+
- `max_completion_tokens` - Cloud models only: `max_output_tokens` under its OpenAI name.
|
|
1925
|
+
- `downloaded` - Whether the model is downloaded.
|
|
1926
|
+
- `update_available` - Whether the model's registry has a newer commit; `false` except for downloaded registry models.
|
|
1927
|
+
- `suggested` - Whether the model is recommended for general use.
|
|
1928
|
+
- `labels` - The model's capabilities and characteristics; see [Model Labels](./openai.md#model-labels).
|
|
1929
|
+
- `components` - The names of a collection's component models, in order; empty for other models.
|
|
1930
|
+
- `models` - Omni collections only: each component's full model object, in the order of `components`, so an exported collection can be imported elsewhere through [`/v1/pull`](./lemonade.md#post-v1pull).
|
|
1931
|
+
- `checkpoints` - Every checkpoint by role, such as `main`, `mmproj`, `draft`, `text_encoder` or `vae`.
|
|
1932
|
+
- The checkpoint that fills the role named by its key.
|
|
1933
|
+
- `checkpoint_status` - One entry per checkpoint of the model, answering whether its bytes are there.
|
|
1934
|
+
- `type` - Checkpoint slot, e.g. `main`, `mmproj`, `vae`.
|
|
1935
|
+
- `present` - `true` only when the checkpoint's bytes are complete and readable.
|
|
1936
|
+
- `required` - A missing required checkpoint keeps the model undownloaded; an optional one, loaded lazily, does not.
|
|
1937
|
+
- `registry` - Registry hosting the checkpoint; empty for local files and directories.
|
|
1938
|
+
- `path` - Absolute path to the checkpoint file; present only with `include_paths=true` and when `present` is `true`.
|
|
1939
|
+
- `source` - Where the model came from: a registry name, `local_upload`, `local_path` or `extra_models_dir`.
|
|
1940
|
+
- `registry_source` - Remote registry the checkpoint downloads from: `huggingface` or `modelscope`.
|
|
1941
|
+
- `recipe_options` - The model's own options: those its registry row authors, with what was saved in `recipe_options.json` on top.
|
|
1942
|
+
- `cloud_provider` - Cloud models only: the provider that serves the model.
|
|
1943
|
+
- `cost_input_per_million` - Cloud models only: USD per million input tokens, when the provider reports it.
|
|
1944
|
+
- `cost_output_per_million` - Cloud models only: USD per million output tokens, when the provider reports it.
|
|
1945
|
+
- `system_prompt` - Omni collections only: the collection's system prompt override.
|
|
1946
|
+
- `version` - Router collections only: the policy document's version.
|
|
1947
|
+
- `routing` - Router collections only: the routing policy.
|
|
1948
|
+
|
|
1949
|
+
### Errors
|
|
1950
|
+
|
|
1951
|
+
| Status | Code | When |
|
|
1952
|
+
|--------|------|------|
|
|
1953
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
1954
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
1955
|
+
<!-- END GENERATED: openai.models_id -->
|
|
1956
|
+
|
|
1957
|
+
<!-- BEGIN GENERATED: openai.models_id_files -->
|
|
1958
|
+
## `GET /v1/models/{id}/files`
|
|
1959
|
+
<sub></sub>
|
|
1960
|
+
|
|
1961
|
+
Lists the resolved local files of one model, for model-detail views such as the app's Files tab. It inventories one model, not drive storage.
|
|
1962
|
+
|
|
1963
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
1964
|
+
|
|
1965
|
+
### Parameters
|
|
1966
|
+
|
|
1967
|
+
| Parameter | Required | Description | Status |
|
|
1968
|
+
|-----------|----------|-------------|--------|
|
|
1969
|
+
| `id` (path) | Yes | Model id, as listed by [`GET /v1/models`](#get-v1models). An alias's id is not accepted here; use its target. | <sub></sub> |
|
|
1970
|
+
| `include_paths` (query) | No | `true` adds each file's absolute `path`, which can reveal local usernames and the cache layout. Defaults to `false`. | <sub></sub> |
|
|
1971
|
+
|
|
1972
|
+
### Example
|
|
1973
|
+
|
|
1974
|
+
=== "PowerShell"
|
|
1975
|
+
|
|
1976
|
+
```powershell
|
|
1977
|
+
Invoke-WebRequest -Uri "http://localhost:13305/v1/models/Qwen3-0.6B-GGUF/files"
|
|
1978
|
+
```
|
|
1979
|
+
|
|
1980
|
+
=== "Bash"
|
|
1981
|
+
|
|
1982
|
+
```bash
|
|
1983
|
+
curl http://localhost:13305/v1/models/Qwen3-0.6B-GGUF/files
|
|
1984
|
+
```
|
|
1985
|
+
|
|
1986
|
+
**Response:** `200 OK`
|
|
1987
|
+
|
|
1988
|
+
```json
|
|
1989
|
+
{
|
|
1990
|
+
"model_id": "Qwen3-0.6B-GGUF",
|
|
1991
|
+
"files": [
|
|
1992
|
+
{"name": "Qwen3-0.6B-Q4_0.gguf", "role": "main", "size_bytes": 382156480, "exists": true}
|
|
1993
|
+
]
|
|
1994
|
+
}
|
|
1995
|
+
```
|
|
1996
|
+
|
|
1997
|
+
### Response format
|
|
1998
|
+
|
|
1999
|
+
- `model_id` - The model's public id.
|
|
2000
|
+
- `files` - The model's resolved files.
|
|
2001
|
+
- `name` - Base filename from the resolved path.
|
|
2002
|
+
- `path` - Absolute path; present only with `include_paths=true`.
|
|
2003
|
+
- `role` - Checkpoint role, for example `main`, `mmproj`, or another recipe-specific role.
|
|
2004
|
+
- `size_bytes` - File size in bytes. Directories are summed recursively. Missing files report `0`.
|
|
2005
|
+
- `exists` - Whether the resolved path currently exists on disk.
|
|
2006
|
+
|
|
2007
|
+
### Errors
|
|
2008
|
+
|
|
2009
|
+
| Status | Code | When |
|
|
2010
|
+
|--------|------|------|
|
|
2011
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
2012
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
2013
|
+
| `404` | `model_load_error` | Resolving or listing `id`'s files fails |
|
|
2014
|
+
<!-- END GENERATED: openai.models_id_files -->
|
|
2015
|
+
|
|
2016
|
+
<!-- BEGIN GENERATED: openai.models_id_options_get -->
|
|
2017
|
+
## `GET /v1/models/{id}/options`
|
|
2018
|
+
<sub></sub>
|
|
2019
|
+
|
|
2020
|
+
Reads a model's recipe options, separated by layer, without loading it. With `POST` and `DELETE` on the same path, this manages per-model options independently of [`/v1/load`](./lemonade.md#post-v1load).
|
|
2021
|
+
|
|
2022
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
2023
|
+
|
|
2024
|
+
### Parameters
|
|
2025
|
+
|
|
2026
|
+
| Parameter | Required | Description | Status |
|
|
2027
|
+
|-----------|----------|-------------|--------|
|
|
2028
|
+
| `id` (path) | Yes | Model id, as listed by [`GET /v1/models`](#get-v1models), or an alias. A trailing `:latest`, the tag Ollama clients append, names the bare id when nothing is registered under the tagged name. | <sub></sub> |
|
|
2029
|
+
|
|
2030
|
+
Per-architecture defaults are chosen by the architecture in the model's GGUF metadata, so a model that has not been downloaded yet shows every key with the value it has before those defaults apply.
|
|
2031
|
+
|
|
2032
|
+
### Example
|
|
2033
|
+
|
|
2034
|
+
=== "PowerShell"
|
|
2035
|
+
|
|
2036
|
+
```powershell
|
|
2037
|
+
Invoke-WebRequest -Uri "http://localhost:13305/v1/models/Qwen3-0.6B-GGUF/options"
|
|
2038
|
+
```
|
|
2039
|
+
|
|
2040
|
+
=== "Bash"
|
|
2041
|
+
|
|
2042
|
+
```bash
|
|
2043
|
+
curl http://localhost:13305/v1/models/Qwen3-0.6B-GGUF/options
|
|
2044
|
+
```
|
|
2045
|
+
|
|
2046
|
+
**Response:** `200 OK`
|
|
2047
|
+
|
|
2048
|
+
```json
|
|
2049
|
+
{
|
|
2050
|
+
"model_name": "Qwen3-0.6B-GGUF",
|
|
2051
|
+
"recipe": "llamacpp",
|
|
2052
|
+
"saved": {},
|
|
2053
|
+
"effective": {
|
|
2054
|
+
"auto_evict": null,
|
|
2055
|
+
"auto_update": null,
|
|
2056
|
+
"ctx_size": -1,
|
|
2057
|
+
"downsize_idle_timeout": 60,
|
|
2058
|
+
"evict_idle_timeout": 300,
|
|
2059
|
+
"evict_weight_factor": 1.0,
|
|
2060
|
+
"llamacpp_args": "--temp 0.6 --top-p 0.85 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 --parallel 1",
|
|
2061
|
+
"llamacpp_backend": "vulkan",
|
|
2062
|
+
"llamacpp_device": "",
|
|
2063
|
+
"merge_args": true,
|
|
2064
|
+
"model_name": "Qwen3-0.6B-GGUF"
|
|
2065
|
+
},
|
|
2066
|
+
"defaults": {
|
|
2067
|
+
"auto_evict": null,
|
|
2068
|
+
"auto_update": null,
|
|
2069
|
+
"ctx_size": -1,
|
|
2070
|
+
"downsize_idle_timeout": 60,
|
|
2071
|
+
"evict_idle_timeout": 300,
|
|
2072
|
+
"evict_weight_factor": 1.0,
|
|
2073
|
+
"llamacpp_args": "--temp 0.6 --top-p 0.85 --top-k 20 --min-p 0.0 --repeat-penalty 1.0 --parallel 1",
|
|
2074
|
+
"llamacpp_backend": "vulkan",
|
|
2075
|
+
"llamacpp_device": "",
|
|
2076
|
+
"merge_args": true,
|
|
2077
|
+
"model_name": "Qwen3-0.6B-GGUF"
|
|
2078
|
+
},
|
|
2079
|
+
"resolved_ctx_size": 40960
|
|
2080
|
+
}
|
|
2081
|
+
```
|
|
2082
|
+
|
|
2083
|
+
### Response format
|
|
2084
|
+
|
|
2085
|
+
- `model_name` - The id from the URL. It appears again inside `effective` and `defaults` so that each one is a complete `/v1/load` body.
|
|
2086
|
+
- `recipe` - The recipe the option names belong to.
|
|
2087
|
+
- `saved` - The model's own entry in `recipe_options.json`: only what was explicitly saved, or `{}` when nothing is. It can also hold keys this endpoint does not accept, such as `pinned` written by `/v1/load`, so replay `effective` rather than `saved`.
|
|
2088
|
+
- `effective` - The body a [`POST /v1/load`](./lemonade.md#post-v1load) for this model uses now: every option the recipe accepts, resolved through the full priority chain. Posting it back whole saves every resolved value as an override, so send only the options that changed.
|
|
2089
|
+
- `defaults` - What `effective` becomes if `saved` is erased, in the same shape. A `ctx_size` of `-1` means the server picks the context size automatically.
|
|
2090
|
+
- `resolved_ctx_size` - The context size a load right now would use: the effective `ctx_size`, or the automatically computed size when that is `-1`.
|
|
2091
|
+
|
|
2092
|
+
### Errors
|
|
2093
|
+
|
|
2094
|
+
| Status | Code | When |
|
|
2095
|
+
|--------|------|------|
|
|
2096
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
2097
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
2098
|
+
| `500` | `internal_error` | Reading or saving the model's options fails, for example on an unwritable options file |
|
|
2099
|
+
<!-- END GENERATED: openai.models_id_options_get -->
|
|
2100
|
+
|
|
2101
|
+
<!-- BEGIN GENERATED: openai.models_id_options_post -->
|
|
2102
|
+
## `POST /v1/models/{id}/options`
|
|
2103
|
+
<sub></sub>
|
|
2104
|
+
|
|
2105
|
+
Saves recipe options for a model without loading it. The body is a flat object of the recipe options [`/v1/load`](./lemonade.md#post-v1load) accepts.
|
|
2106
|
+
|
|
2107
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
2108
|
+
|
|
2109
|
+
### Parameters
|
|
2110
|
+
|
|
2111
|
+
| Parameter | Required | Description | Status |
|
|
2112
|
+
|-----------|----------|-------------|--------|
|
|
2113
|
+
| `id` (path) | Yes | Model id, as listed by [`GET /v1/models`](#get-v1models), or an alias. A trailing `:latest`, the tag Ollama clients append, names the bare id when nothing is registered under the tagged name. | <sub></sub> |
|
|
2114
|
+
| `ctx_size` | No | Context size: a positive whole number, or `-1` to size it automatically even when the server-wide `ctx_size` is a specific number. Only recipes that take a context size use it, such as `llamacpp`, `flm`, `ryzenai-llm` and `vllm`. | <sub></sub> |
|
|
2115
|
+
| `dry_run` | No | `true` reports what the request would do without doing it. Defaults to `false`. | <sub></sub> |
|
|
2116
|
+
| `model_name` | No | Ignored; the URL names the model. Accepted so `effective` can be posted back as is. | <sub></sub> |
|
|
2117
|
+
|
|
2118
|
+
The request merges into the model's saved entry, so keys it leaves out keep their values. `null` removes an option, as do `-1`, `""` and `"auto"` for any option but `ctx_size`, and the model falls back to the next layer of the [priority chain](./lemonade.md#post-v1load). [`DELETE`](#delete-v1modelsidoptions) removes every saved option at once.
|
|
2119
|
+
|
|
2120
|
+
A request that fails validation saves nothing.
|
|
2121
|
+
|
|
2122
|
+
With `dry_run`, the request is validated and resolved as a save would be, but nothing is persisted: `effective` and `resolved_ctx_size` describe the state the save would produce, while `saved` keeps reporting the entry on disk.
|
|
2123
|
+
|
|
2124
|
+
A running model keeps its current options until it is next loaded.
|
|
2125
|
+
|
|
2126
|
+
`pinned` is omitted from `effective` and `defaults`; it belongs to [`/v1/load`](./lemonade.md#post-v1load) and [`/internal/pin`](./internal.md#post-internalpin).
|
|
2127
|
+
|
|
2128
|
+
### Example
|
|
2129
|
+
|
|
2130
|
+
=== "PowerShell"
|
|
2131
|
+
|
|
2132
|
+
```powershell
|
|
2133
|
+
Invoke-WebRequest `
|
|
2134
|
+
-Uri "http://localhost:13305/v1/models/Qwen3-4B-GGUF/options" `
|
|
2135
|
+
-Method POST `
|
|
2136
|
+
-Headers @{ "Content-Type" = "application/json" } `
|
|
2137
|
+
-Body '{"ctx_size": 8192}'
|
|
2138
|
+
```
|
|
2139
|
+
|
|
2140
|
+
=== "Bash"
|
|
2141
|
+
|
|
2142
|
+
```bash
|
|
2143
|
+
curl http://localhost:13305/v1/models/Qwen3-4B-GGUF/options \
|
|
2144
|
+
-H "Content-Type: application/json" \
|
|
2145
|
+
-d '{"ctx_size": 8192}'
|
|
2146
|
+
```
|
|
2147
|
+
|
|
2148
|
+
**Response:** `200 OK`
|
|
2149
|
+
|
|
2150
|
+
```json
|
|
2151
|
+
{
|
|
2152
|
+
"model_name": "Qwen3-4B-GGUF",
|
|
2153
|
+
"recipe": "llamacpp",
|
|
2154
|
+
"saved": {"ctx_size": 8192},
|
|
2155
|
+
"effective": {
|
|
2156
|
+
"auto_evict": null,
|
|
2157
|
+
"auto_update": null,
|
|
1174
2158
|
"ctx_size": 8192,
|
|
1175
|
-
"
|
|
1176
|
-
"
|
|
1177
|
-
|
|
2159
|
+
"downsize_idle_timeout": 60,
|
|
2160
|
+
"evict_idle_timeout": 300,
|
|
2161
|
+
"evict_weight_factor": 1.0,
|
|
2162
|
+
"llamacpp_args": "--parallel 1",
|
|
2163
|
+
"llamacpp_backend": "vulkan",
|
|
2164
|
+
"llamacpp_device": "",
|
|
2165
|
+
"merge_args": true,
|
|
2166
|
+
"model_name": "Qwen3-4B-GGUF"
|
|
2167
|
+
},
|
|
2168
|
+
"defaults": {
|
|
2169
|
+
"auto_evict": null,
|
|
2170
|
+
"auto_update": null,
|
|
2171
|
+
"ctx_size": -1,
|
|
2172
|
+
"downsize_idle_timeout": 60,
|
|
2173
|
+
"evict_idle_timeout": 300,
|
|
2174
|
+
"evict_weight_factor": 1.0,
|
|
2175
|
+
"llamacpp_args": "--parallel 1",
|
|
2176
|
+
"llamacpp_backend": "vulkan",
|
|
2177
|
+
"llamacpp_device": "",
|
|
2178
|
+
"merge_args": true,
|
|
2179
|
+
"model_name": "Qwen3-4B-GGUF"
|
|
2180
|
+
},
|
|
2181
|
+
"resolved_ctx_size": 8192
|
|
1178
2182
|
}
|
|
1179
2183
|
```
|
|
1180
2184
|
|
|
1181
|
-
###
|
|
2185
|
+
### Response format
|
|
2186
|
+
|
|
2187
|
+
- `model_name` - The id from the URL. It appears again inside `effective` and `defaults` so that each one is a complete `/v1/load` body.
|
|
2188
|
+
- `recipe` - The recipe the option names belong to.
|
|
2189
|
+
- `saved` - The model's own entry in `recipe_options.json`: only what was explicitly saved, or `{}` when nothing is. It can also hold keys this endpoint does not accept, such as `pinned` written by `/v1/load`, so replay `effective` rather than `saved`.
|
|
2190
|
+
- `effective` - The body a [`POST /v1/load`](./lemonade.md#post-v1load) for this model uses now: every option the recipe accepts, resolved through the full priority chain. Posting it back whole saves every resolved value as an override, so send only the options that changed.
|
|
2191
|
+
- `defaults` - What `effective` becomes if `saved` is erased, in the same shape. A `ctx_size` of `-1` means the server picks the context size automatically.
|
|
2192
|
+
- `resolved_ctx_size` - The context size a load right now would use: the effective `ctx_size`, or the automatically computed size when that is `-1`.
|
|
2193
|
+
|
|
2194
|
+
### Errors
|
|
2195
|
+
|
|
2196
|
+
| Status | Code | When |
|
|
2197
|
+
|--------|------|------|
|
|
2198
|
+
| `400` | `invalid_json` | The body is not a JSON object |
|
|
2199
|
+
| `400` | `invalid_argument` | An argument or a value nested in it has the wrong JSON type, or a value its schema does not list |
|
|
2200
|
+
| `400` | | The body names an option that the model's recipe does not accept |
|
|
2201
|
+
| `400` | | The body sets `pinned`, which belongs to `/v1/load` and `/internal/pin` |
|
|
2202
|
+
| `400` | | A field not listed above has the wrong JSON type |
|
|
2203
|
+
| `400` | | `ctx_size` is not a positive whole number or `-1`, or a whole-number option has a fractional value |
|
|
2204
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
2205
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
2206
|
+
| `500` | `internal_error` | Reading or saving the model's options fails, for example on an unwritable options file |
|
|
2207
|
+
<!-- END GENERATED: openai.models_id_options_post -->
|
|
2208
|
+
|
|
2209
|
+
<!-- BEGIN GENERATED: openai.models_id_options_delete -->
|
|
2210
|
+
## `DELETE /v1/models/{id}/options`
|
|
2211
|
+
<sub></sub>
|
|
2212
|
+
|
|
2213
|
+
Resets a model to its defaults by erasing its `recipe_options.json` entry. Defaults from its registry entry and the server's configuration still apply.
|
|
2214
|
+
|
|
2215
|
+
Requires `LEMONADE_API_KEY` when it is set.
|
|
2216
|
+
|
|
2217
|
+
### Parameters
|
|
2218
|
+
|
|
2219
|
+
| Parameter | Required | Description | Status |
|
|
2220
|
+
|-----------|----------|-------------|--------|
|
|
2221
|
+
| `id` (path) | Yes | Model id, as listed by [`GET /v1/models`](#get-v1models), or an alias. A trailing `:latest`, the tag Ollama clients append, names the bare id when nothing is registered under the tagged name. | <sub></sub> |
|
|
2222
|
+
|
|
2223
|
+
### Example
|
|
2224
|
+
|
|
2225
|
+
=== "PowerShell"
|
|
2226
|
+
|
|
2227
|
+
```powershell
|
|
2228
|
+
Invoke-WebRequest `
|
|
2229
|
+
-Uri "http://localhost:13305/v1/models/Qwen3-4B-GGUF/options" `
|
|
2230
|
+
-Method DELETE
|
|
2231
|
+
```
|
|
2232
|
+
|
|
2233
|
+
=== "Bash"
|
|
1182
2234
|
|
|
1183
|
-
|
|
2235
|
+
```bash
|
|
2236
|
+
curl -X DELETE http://localhost:13305/v1/models/Qwen3-4B-GGUF/options
|
|
2237
|
+
```
|
|
2238
|
+
|
|
2239
|
+
**Response:** `200 OK`
|
|
1184
2240
|
|
|
1185
2241
|
```json
|
|
1186
2242
|
{
|
|
1187
|
-
"
|
|
1188
|
-
|
|
1189
|
-
|
|
1190
|
-
|
|
2243
|
+
"model_name": "Qwen3-4B-GGUF",
|
|
2244
|
+
"recipe": "llamacpp",
|
|
2245
|
+
"saved": {},
|
|
2246
|
+
"effective": {
|
|
2247
|
+
"auto_evict": null,
|
|
2248
|
+
"auto_update": null,
|
|
2249
|
+
"ctx_size": -1,
|
|
2250
|
+
"downsize_idle_timeout": 60,
|
|
2251
|
+
"evict_idle_timeout": 300,
|
|
2252
|
+
"evict_weight_factor": 1.0,
|
|
2253
|
+
"llamacpp_args": "--parallel 1",
|
|
2254
|
+
"llamacpp_backend": "vulkan",
|
|
2255
|
+
"llamacpp_device": "",
|
|
2256
|
+
"merge_args": true,
|
|
2257
|
+
"model_name": "Qwen3-4B-GGUF"
|
|
2258
|
+
},
|
|
2259
|
+
"defaults": {
|
|
2260
|
+
"auto_evict": null,
|
|
2261
|
+
"auto_update": null,
|
|
2262
|
+
"ctx_size": -1,
|
|
2263
|
+
"downsize_idle_timeout": 60,
|
|
2264
|
+
"evict_idle_timeout": 300,
|
|
2265
|
+
"evict_weight_factor": 1.0,
|
|
2266
|
+
"llamacpp_args": "--parallel 1",
|
|
2267
|
+
"llamacpp_backend": "vulkan",
|
|
2268
|
+
"llamacpp_device": "",
|
|
2269
|
+
"merge_args": true,
|
|
2270
|
+
"model_name": "Qwen3-4B-GGUF"
|
|
2271
|
+
},
|
|
2272
|
+
"resolved_ctx_size": 32768
|
|
1191
2273
|
}
|
|
1192
2274
|
```
|
|
2275
|
+
|
|
2276
|
+
### Response format
|
|
2277
|
+
|
|
2278
|
+
- `model_name` - The id from the URL. It appears again inside `effective` and `defaults` so that each one is a complete `/v1/load` body.
|
|
2279
|
+
- `recipe` - The recipe the option names belong to.
|
|
2280
|
+
- `saved` - The model's own entry in `recipe_options.json`: only what was explicitly saved, or `{}` when nothing is. It can also hold keys this endpoint does not accept, such as `pinned` written by `/v1/load`, so replay `effective` rather than `saved`.
|
|
2281
|
+
- `effective` - The body a [`POST /v1/load`](./lemonade.md#post-v1load) for this model uses now: every option the recipe accepts, resolved through the full priority chain. Posting it back whole saves every resolved value as an override, so send only the options that changed.
|
|
2282
|
+
- `defaults` - What `effective` becomes if `saved` is erased, in the same shape. A `ctx_size` of `-1` means the server picks the context size automatically.
|
|
2283
|
+
- `resolved_ctx_size` - The context size a load right now would use: the effective `ctx_size`, or the automatically computed size when that is `-1`.
|
|
2284
|
+
|
|
2285
|
+
### Errors
|
|
2286
|
+
|
|
2287
|
+
| Status | Code | When |
|
|
2288
|
+
|--------|------|------|
|
|
2289
|
+
| `404` | `model_not_supported` | The requested model is filtered out on this system, such as an NPU model without an NPU |
|
|
2290
|
+
| `404` | `model_not_found` | The requested model is not in the registry |
|
|
2291
|
+
| `500` | `internal_error` | Reading or saving the model's options fails, for example on an unwritable options file |
|
|
2292
|
+
<!-- END GENERATED: openai.models_id_options_delete -->
|