turbollm 0.1.0 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,93 +1,313 @@
1
1
  <p align="center">
2
- <img src="https://raw.githubusercontent.com/mohitsoni48/Turbo-LLM/main/turbollm/web/public/brand/turbollm-icon-512.jpeg" width="96" height="96" alt="TurboLLM" />
2
+ <img src="https://raw.githubusercontent.com/mohitsoni48/Turbo-LLM/main/turbollm/web/public/brand/turbollm-icon-512.jpeg?v=2" width="92" height="92" alt="TurboLLM" />
3
3
  </p>
4
4
 
5
5
  <h1 align="center">TurboLLM</h1>
6
6
 
7
7
  <p align="center">
8
- <strong>Run any local LLM, auto-tuned to your GPU — with a polished web UI and an
9
- OpenAI/Anthropic-compatible API.</strong><br/>
10
- Point Claude Code at your own machine in one command. Fully offline, no cloud key.
8
+ <strong>Run <em>any</em> local LLM engine, auto-tuned to your GPU — with a polished web UI
9
+ and an OpenAI/Anthropic-compatible API.</strong><br/>
10
+ Bring your own llama.cpp fork. No compiling. No Electron. No Python. Point Claude Code at
11
+ your own machine in one command — fully offline.
12
+ </p>
13
+
14
+ <p align="center">
15
+ <a href="https://www.npmjs.com/package/turbollm"><img src="https://img.shields.io/npm/v/turbollm.svg?color=e2552e" alt="npm version" /></a>
16
+ <a href="https://www.npmjs.com/package/turbollm"><img src="https://img.shields.io/npm/dm/turbollm.svg?color=e2552e" alt="npm downloads" /></a>
17
+ <img src="https://img.shields.io/badge/node-%E2%89%A522-3c873a.svg" alt="node >= 22" />
18
+ <img src="https://img.shields.io/badge/license-FSL--1.1--ALv2-blue.svg" alt="license" />
19
+ <img src="https://img.shields.io/badge/platform-Windows%20%C2%B7%20macOS%20%C2%B7%20Linux-555.svg" alt="platforms" />
11
20
  </p>
12
21
 
13
22
  <!-- Brand: shipped app icon web/public/brand/turbollm-icon-512.jpeg · high-res masters web/brand-assets/ (unshipped) · in-app mark web/src/components/Logo.tsx · favicon web/public/favicon.svg -->
14
23
 
24
+ ```bash
25
+ npx turbollm
26
+ ```
27
+
28
+ That one command starts a local daemon, opens a browser UI, and serves your models over an
29
+ API any tool can talk to. TurboLLM is the **performance & bleeding-edge layer for local
30
+ LLMs** — built for people who today hand-compile forks and hunt forums for the right flags.
31
+
32
+ <p align="center">
33
+ <img src="https://raw.githubusercontent.com/mohitsoni48/Turbo-LLM/main/assets/how-it-works.svg?v=2" width="860" alt="How TurboLLM works: clients -> one lightweight daemon -> any engine on your GPU" />
34
+ </p>
35
+
15
36
  ---
16
37
 
17
- TurboLLM is a single command — `npx turbollm` — that starts a local daemon, opens a
18
- browser UI, and serves your models over an API any tool can talk to. It manages **any
19
- bring-your-own inference engine** (stock `llama.cpp`, community forks, or a default it
20
- provisions for you), **auto-tunes the launch flags to your exact hardware**, and shows you
21
- **real measured tokens/sec**.
38
+ ## Contents
39
+
40
+ - [Why TurboLLM](#why-turbollm)
41
+ - [Features](#features)
42
+ - [Quick start](#quick-start)
43
+ - [⭐ Bring any engine — the headline feature](#-bring-any-engine--the-headline-feature)
44
+ - [Models — bring your own, or browse Hugging Face](#models)
45
+ - [Auto-tuning & performance](#auto-tuning--performance)
46
+ - [Chat](#chat)
47
+ - [APIs & integrations](#apis--integrations)
48
+ - [Run Claude Code on your own GPU](#run-claude-code-on-your-own-gpu)
49
+ - [Use it from any device on your network](#use-it-from-any-device-on-your-network)
50
+ - [Share the GPU with ComfyUI](#share-the-gpu-with-comfyui)
51
+ - [Command-line reference](#command-line-reference)
52
+ - [Configuration & data](#configuration--data)
53
+ - [Requirements](#requirements)
54
+ - [Privacy](#privacy)
55
+ - [How TurboLLM compares](#how-turbollm-compares)
56
+ - [Troubleshooting](#troubleshooting)
57
+ - [Develop from source](#develop-from-source)
58
+ - [License](#license)
22
59
 
23
- It's built for the prosumer/indie-dev who today hand-compiles forks and hunts forums for
24
- the right flags — not as "another easy chat app," but as the **performance & bleeding-edge
25
- layer for local LLMs**.
60
+ ---
26
61
 
27
62
  ## Why TurboLLM
28
63
 
29
- - **Any engine, including forks.** Most tools lock you to one blessed runtime. TurboLLM
30
- makes any `llama-server`-compatible binary a first-class choice — point it at a build you
31
- compiled, or let it auto-provision the right prebuilt for your GPU (CUDA / ROCm / Metal /
32
- SYCL / Vulkan, picked by detected vendor).
33
- - **Auto-tuned to your hardware.** It benchmarks on load and derives fast defaults (flash
34
- attention, speculative decoding / NextN, context, offload, KV-cache type, threads) with a
35
- VRAM-fit verdict *before* you load no more guessing flags.
36
- - **Real tokens/sec, never fake.** Speed shown in the model list is measured on your
37
- machine from actual generation, not a synthetic estimate.
38
- - **A real chat UI.** Streaming with live t/s, prefill %, TTFT and full stats; markdown +
39
- code highlighting; collapsible thinking blocks; edit / regenerate / delete / copy;
40
- persistent searchable conversations; image input for vision models; per-chat system
41
- prompt and sampling.
42
- - **Drop-in APIs.** OpenAI **and** Anthropic-compatible endpoints, so existing tools and
43
- agentic CLIs work unchanged.
44
- - **Usable from any device.** The web UI runs in the browser and can be shared across your
45
- LAN (with optional API-key auth), not locked to the machine it runs on.
46
- - **Bring your own models.** Point it at folders you already have (no re-download), or
47
- browse and download GGUFs from Hugging Face or any direct URL inside the app.
48
- - **Offline-first & private.** Core local use needs no account, no backend, no internet.
49
- No analytics are collected.
64
+ Local-LLM tools make two choices for you, and both cost you performance:
65
+
66
+ 1. **They pick the engine.** LM Studio ships one blessed runtime; Ollama hides the engine
67
+ entirely. The fastest community innovations new quant formats, speculative decoding,
68
+ low-bit KV cache land in **forks** first, and you can't use them without compiling.
69
+ 2. **They don't tell you what speed to expect**, and they don't tune the dozens of launch
70
+ flags (`-c`, `-ngl`, `--n-cpu-moe`, KV type, threads, flash-attn, draft models) that make
71
+ the difference between 20 and 80 tokens/sec on the *same* hardware.
72
+
73
+ TurboLLM does the opposite:
74
+
75
+ - **🔌 Any engine, including forks.** Point it at any `llama-server`-compatible binary — a
76
+ build you compiled, a community fork, or the one it auto-provisions for your GPU. It probes
77
+ the binary's real capabilities and adapts the UI to them. **This is the whole point.**
78
+ - **⚡ Auto-tuned to your hardware.** It benchmarks on load, derives fast defaults, and shows
79
+ a **VRAM-fit verdict before you load** no more flag guessing.
80
+ - **📊 Real tokens/sec, never faked.** Speed in the model list is *measured on your machine*
81
+ from actual generation live while you chat, and remembered per model.
82
+ - **🪶 Lightweight.** A ~0.3 MB npm package on Node **no Electron, no bundled Chromium, no
83
+ Python**. It downloads only the engine your GPU actually needs (Vulkan 38 MB).
84
+ - **🔌 Drop-in APIs.** OpenAI **and** Anthropic-compatible — so Claude Code and every existing
85
+ tool work unchanged.
86
+ - **🔒 Offline-first & private.** No account, no backend, no internet, **no telemetry.**
50
87
 
51
- ## Requirements
88
+ ---
52
89
 
53
- - **Node.js 22 or newer** — the daemon enforces this at startup and exits with a clear
54
- message if the version is too old. Download: <https://nodejs.org>
55
- - A GPU is recommended but not required (a CPU build is provisioned as a fallback).
90
+ ## Features
91
+
92
+ The current capability list (kept in sync with [CHANGELOG.md](CHANGELOG.md)):
93
+
94
+ **Engines**
95
+ - Bring any `llama-server`-compatible engine — stock builds or community forks — with real capability probing
96
+ - Auto-provision a GPU-matched `llama-server` build on first run (CUDA / ROCm / Metal / SYCL / Vulkan, CPU fallback)
97
+ - **vLLM** and **MLX** backends in addition to llama.cpp
98
+ - One-click backend install + switch from the Engines screen
99
+
100
+ **Models**
101
+ - Use your own local GGUF / safetensors, or browse & download from Hugging Face in-app
102
+ - Per-model load profiles (context, GPU offload, KV-cache quant, flash-attn, draft models)
103
+ - Auto-tune on load with a **VRAM-fit verdict before you load**
104
+ - Measured tokens/sec per model — never faked — live while you chat and remembered
105
+
106
+ **Chat**
107
+ - Streaming chat with live t/s, TTFT, context meter, and reasoning/thinking support
108
+ - Image and document attachments — including **send an image or file with no text**
109
+
110
+ **Integrations**
111
+ - OpenAI- **and** Anthropic-compatible APIs — run Claude Code on your own GPU
112
+ - LAN sharing with optional API-key auth
113
+ - **Share the GPU with ComfyUI** — auto-unload the model while ComfyUI renders, reload when it's done
114
+
115
+ **Platform**
116
+ - ~0.3 MB npm package on Node — no Electron, no Chromium, no Python
117
+ - Offline-first, no account, no telemetry
118
+
119
+ ---
56
120
 
57
121
  ## Quick start
58
122
 
59
123
  ```bash
60
- # run directly without installing (recommended for first try)
124
+ # run without installing (recommended for first try)
61
125
  npx turbollm
62
126
 
63
- # or install globally, then run
127
+ # or install globally
64
128
  npm install -g turbollm
65
129
  turbollm
66
130
  ```
67
131
 
68
- The daemon starts, prints the local URL (default <http://127.0.0.1:6996>), and opens your
69
- browser. On first run with no engine configured, it downloads a suitable prebuilt
70
- `llama-server` for your hardware automatically. Stop the daemon with **Ctrl+C**.
132
+ **On first run** the daemon:
133
+
134
+ 1. Detects your GPU and **downloads a matching `llama-server` build** (CUDA for NVIDIA, ROCm
135
+ for AMD, Metal for Apple, SYCL for Intel, Vulkan otherwise — with a CPU fallback).
136
+ 2. Starts on <http://127.0.0.1:6996> and opens your browser.
137
+ 3. Drops you on the **Chat** screen, ready to load a model.
138
+
139
+ Then open **Models**, download or pick a GGUF, click **Load**, and start chatting. Stop the
140
+ daemon any time with **Ctrl+C**.
141
+
142
+ <!--
143
+ 📸 SCREENSHOTS — drop PNGs into assets/screenshots/ and uncomment. Suggested shots:
144
+ - chat.png : a chat mid-stream showing the live t/s + context meter
145
+ - models.png : the Models › Library with measured t/s per model
146
+ - engines.png : the Engines screen + backend picker (the USP)
147
+ - tuning.png : the model load-params panel (ctx/ngl/NextN/VRAM verdict)
148
+ <p align="center"><img src="https://raw.githubusercontent.com/mohitsoni48/Turbo-LLM/main/assets/screenshots/chat.png" width="860" alt="TurboLLM chat" /></p>
149
+ -->
150
+
151
+ ---
152
+
153
+ ## ⭐ Bring any engine — the headline feature
154
+
155
+ No other local-LLM app lets you run **whatever inference engine you want**. TurboLLM treats
156
+ the engine as a swappable component.
157
+
158
+ **Add a custom engine** (Engines screen → **Add engine**):
159
+
160
+ 1. Compile or download any `llama-server`-compatible binary — stock
161
+ [llama.cpp](https://github.com/ggml-org/llama.cpp), a community fork, or your own build.
162
+ 2. Point TurboLLM at the binary. It runs a **capability probe** and learns exactly which
163
+ flags and features that build supports.
164
+ 3. Activate it. The load-parameter UI **adapts to that engine** — features the build doesn't
165
+ support are hidden; ones it adds (e.g. low-bit KV cache, NextN) light up.
166
+
167
+ **Auto-provisioned default.** Don't want to fetch anything? On first run TurboLLM downloads
168
+ the right upstream prebuilt for your GPU automatically — and a **backend picker** lets you
169
+ switch between CUDA / ROCm / Metal / SYCL / Vulkan / CPU at any time (it downloads the variant
170
+ you choose, LM Studio-style).
171
+
172
+ **Engine types.** Both **llama.cpp / GGUF** and **MLX** (on macOS) are first-class engine
173
+ kinds — pick the right one per model.
174
+
175
+ **Fully supervised.** Every engine runs under a real state machine: health-gated readiness,
176
+ graceful stop, an **idle auto-stop** watchdog, and **live logs + clear error surfacing** in
177
+ the UI when something fails to load.
178
+
179
+ > Why it matters: fork-exclusive features — **speculative decoding (NextN / MTP / draft)**,
180
+ > low-bit KV cache, new quant formats — are usable on day 0, with **zero compiler knowledge**
181
+ > on your part beyond producing the binary (and often not even that).
182
+
183
+ ---
184
+
185
+ ## Models
186
+
187
+ - **Use the folders you already have.** Point TurboLLM at any directory of GGUFs — your
188
+ existing LM Studio / Ollama / manual downloads — **no re-downloading.** It parses GGUF
189
+ metadata (arch, params, quant, context, vision) for every file.
190
+ - **Browse & download from Hugging Face**, in-app: search, see the file tree, pick a quant,
191
+ and download with **resume + SHA-256 verification**. Gated models (Llama, Gemma) work via
192
+ your own HF token, which **never leaves your machine**.
193
+ - **Import from any URL** — not just Hugging Face. Paste a direct `.gguf` link (model-author
194
+ sites, mirrors, private servers); it disk-space-checks and downloads through the same
195
+ manager.
196
+ - **Quant recommendation per GPU** and a **VRAM-fit verdict** so you pick a quant that
197
+ actually fits before you commit.
198
+ - **Primary download folder**, real-time **measured t/s per model**, and **delete-from-disk**
199
+ — full library management.
200
+
201
+ ---
202
+
203
+ ## Auto-tuning & performance
204
+
205
+ - **Auto-benchmark on load** derives fast defaults for your exact GPU.
206
+ - **Real measured tokens/sec** in the model list — **live** while a model is generating,
207
+ **last-session** when it's idle (never a synthetic estimate).
208
+ - **Full load-parameter UI**, a superset of what other tools expose:
209
+ context length, GPU offload (`-ngl`), **MoE CPU-offload (`--n-cpu-moe`)**, parallel slots,
210
+ **KV-cache quant type** (incl. low-bit on supporting forks), CPU threads, flash attention,
211
+ and **speculative decoding (NextN / MTP / draft)**.
212
+ - **Fast by default:** flash attention on, NextN self-speculative decoding on for models that
213
+ carry a draft head, threads auto — best speed out of the box, safely gated to what your
214
+ engine actually accepts.
215
+ - **Saved per-model profiles** — tune once, and it loads that way every time.
216
+
217
+ ---
218
+
219
+ ## Chat
220
+
221
+ A genuinely good chat UI, not an afterthought:
222
+
223
+ - **Streaming** with a **stop** button, **live tokens/sec**, **prompt-processing %** and
224
+ **prefill t/s**, **time-to-first-token**, **total time**, exact **token counts**, and a
225
+ **context-usage meter** (filled / max) on every reply.
226
+ - **Thinking control** — toggle reasoning **off** to get a direct answer (saves time and
227
+ tokens), or leave it **on** with collapsible, timed "thought for N s" blocks.
228
+ - **Markdown + syntax-highlighted code** with one-click copy.
229
+ - **Edit, regenerate, delete, copy** any message; **persistent, searchable conversations**
230
+ with rename, delete, and **auto-generated titles**.
231
+ - **Per-chat system prompt** and **per-chat sampling** overrides (temperature, top-p/k, min-p).
232
+ - **Image input** for vision models.
233
+ - **TurboLLM Expert** — a built-in assistant that knows the app and your hardware, for
234
+ onboarding and troubleshooting without leaving the UI.
235
+
236
+ ---
237
+
238
+ ## APIs & integrations
239
+
240
+ With a model loaded, TurboLLM serves two compatible APIs on the same port:
241
+
242
+ ```bash
243
+ # OpenAI-compatible
244
+ curl http://127.0.0.1:6996/v1/chat/completions \
245
+ -H "Content-Type: application/json" \
246
+ -d '{"model":"local","messages":[{"role":"user","content":"hello"}]}'
247
+ ```
248
+
249
+ - **OpenAI-compatible** `/v1/chat/completions`, `/v1/embeddings`, … — point any OpenAI client
250
+ or tool at it.
251
+ - **Anthropic-compatible** `/v1/messages` — including **tool use and streaming** — which is
252
+ what powers Claude Code below. No other local host offers this.
253
+ - **API-key auth** you can require when sharing over a LAN (Settings → Network).
71
254
 
72
- Then, in the UI: open **Models**, download or pick a GGUF, and load it.
255
+ ---
73
256
 
74
- ## Use your local model with Claude Code
257
+ ## Run Claude Code on your own GPU
75
258
 
76
- TurboLLM serves an Anthropic-compatible API, so coding CLIs like
77
- [Claude Code](https://www.npmjs.com/package/@anthropic-ai/claude-code) can run against
78
- whatever model you have loaded — no cloud key, fully offline. One command wires it up:
259
+ TurboLLM's Anthropic-compatible endpoint means [Claude
260
+ Code](https://www.npmjs.com/package/@anthropic-ai/claude-code) can run against whatever model
261
+ you've loaded — no cloud key, fully offline. One command wires it up:
79
262
 
80
263
  ```bash
81
264
  turbollm launch claude # opens Claude Code on your loaded model
82
265
  ```
83
266
 
84
- This requires the daemon running with a model loaded. It points Claude Code's
85
- `ANTHROPIC_BASE_URL` / `ANTHROPIC_MODEL` at TurboLLM and execs `claude`; extra args are
86
- forwarded (`turbollm launch claude --help`). If `claude` isn't installed, the command tells
87
- you how (`npm install -g @anthropic-ai/claude-code`). The in-app **Developer** screen also
88
- shows manual env-var snippets for any OpenAI- or Anthropic-compatible tool.
267
+ It sets Claude Code's `ANTHROPIC_BASE_URL` / `ANTHROPIC_MODEL` at TurboLLM and execs `claude`;
268
+ extra args are forwarded. If `claude` isn't installed, it tells you how. The in-app
269
+ **Developer** screen also shows copy-paste env snippets for any OpenAI- or Anthropic-compatible
270
+ tool (Open WebUI, Kilo Code, opencode, …).
271
+
272
+ ---
273
+
274
+ ## Use it from any device on your network
275
+
276
+ The UI runs in the browser, so any phone, tablet, or laptop on your LAN can use the model on
277
+ your GPU box:
278
+
279
+ ```bash
280
+ turbollm --addr 0.0.0.0:6996 # bind all interfaces, then open http://<your-ip>:6996
281
+ ```
282
+
283
+ Turn on **Require API key** in Settings → Network when you expose it.
89
284
 
90
- ## Command-line usage
285
+ ---
286
+
287
+ ## Share the GPU with ComfyUI
288
+
289
+ If you run **ComfyUI** on the same GPU, an LLM holding VRAM while ComfyUI renders means both
290
+ fight for memory (and one usually OOMs). TurboLLM can hand the GPU over automatically:
291
+
292
+ - The instant ComfyUI starts a render, TurboLLM **unloads its model and pauses new loads**.
293
+ - When ComfyUI's queue drains, TurboLLM **reloads the exact model it unloaded**.
294
+
295
+ It's **push-based, not polling** — ComfyUI signals TurboLLM the moment a job starts/ends, so
296
+ the handoff is immediate and deterministic (the model is gone *before* ComfyUI executes).
297
+
298
+ **One-time setup** (Settings → ComfyUI):
299
+
300
+ 1. Turn on **Pause for ComfyUI** and **Save**.
301
+ 2. Enter your ComfyUI folder (the one containing `custom_nodes`) and click **Install gate**.
302
+ TurboLLM writes a small custom node into ComfyUI, wired to this daemon.
303
+ 3. **Restart ComfyUI** once so it loads the node.
304
+
305
+ The Settings panel shows a live indicator (rendering / idle / connected). To undo it, click
306
+ **Remove** in the same panel.
307
+
308
+ ---
309
+
310
+ ## Command-line reference
91
311
 
92
312
  ```bash
93
313
  turbollm # start on :6996, open browser
@@ -99,38 +319,82 @@ turbollm launch claude # start Claude Code against the loaded model
99
319
 
100
320
  | Flag | Description |
101
321
  |------|-------------|
102
- | `--port <n>` | Listen on a specific port (default: 6996) |
322
+ | `--port <n>` | Listen on a specific port (default: `6996`) |
103
323
  | `--addr <host:port>` | Full host:port override, e.g. `0.0.0.0:6996` for LAN sharing |
104
324
  | `--no-open` | Start without opening a browser window |
105
325
  | `--config <file>` | Path to a custom config file |
106
326
  | `--help`, `-h` | Show usage and exit |
107
327
 
108
- State (config, database, downloaded engines and models) lives under **`~/.turbollm/`**.
328
+ ---
109
329
 
110
- ## API
330
+ ## Configuration & data
111
331
 
112
- With a model loaded, TurboLLM serves two compatible APIs on the same port:
332
+ Everything lives under **`~/.turbollm/`** on every OS `config.json`, the SQLite chat
333
+ database, downloaded engines, models cache, and logs. Back it up or delete it to reset.
334
+ Use `--config <file>` to point at an alternate config (its directory becomes the data dir).
113
335
 
114
- ```bash
115
- # OpenAI-compatible
116
- curl http://127.0.0.1:6996/v1/chat/completions \
117
- -H "Content-Type: application/json" \
118
- -d '{"model":"local","messages":[{"role":"user","content":"hello"}]}'
119
- ```
336
+ ---
337
+
338
+ ## Requirements
339
+
340
+ - **Node.js 22 or newer** — enforced at startup with a clear message. <https://nodejs.org>
341
+ - **Windows, macOS, or Linux.**
342
+ - A GPU is recommended but **not required** — a CPU build is provisioned as a fallback.
343
+ - On Windows, the first time the auto-downloaded `llama-server` runs, SmartScreen/Defender may
344
+ prompt (it's an upstream binary). Allow it once.
345
+
346
+ ---
120
347
 
121
- The Anthropic-compatible endpoint (`/v1/messages`, including tool use and streaming) powers
122
- the Claude Code integration above. When sharing over a LAN you can require an API key —
123
- enable it in **Settings Network**.
348
+ ## Privacy
349
+
350
+ TurboLLM is **offline-first**: core local use needs no account, no backend, and no internet.
351
+ **No analytics or telemetry are collected.** Your prompts, chats, files, and keys never leave
352
+ your machine.
353
+
354
+ ---
124
355
 
125
- ## Develop & run from source
356
+ ## How TurboLLM compares
357
+
358
+ Focused on the differences that matter — all four are good tools.
359
+
360
+ | | **TurboLLM** | LM Studio | Ollama | Open WebUI |
361
+ |---|:---:|:---:|:---:|:---:|
362
+ | Run **any engine / community forks** | ✅ | ❌ one runtime | ❌ hidden | ❌ |
363
+ | **Auto-tune** launch flags to your GPU | ✅ | ❌ | ❌ | ❌ |
364
+ | **Measured** t/s in the model list | ✅ | ◐ | ◐ | ❌ |
365
+ | **Anthropic** API (tool use) → Claude Code | ✅ | ❌ | ❌ | ❌ |
366
+ | OpenAI-compatible API | ✅ | ✅ | ✅ | ◐ proxy |
367
+ | Use existing model folders (no re-download) | ✅ | ◐ | ❌ | ❌ |
368
+ | Speculative decoding (NextN / MTP / draft) | ✅ | ◐ draft | ❌ | ❌ |
369
+ | Web UI from any LAN device | ✅ | ❌ | ❌ | ✅ |
370
+ | **Lightweight** (no Electron / no Python) | ✅ npm | ❌ Electron | ✅ Go | ❌ Python |
371
+ | Offline-first · no telemetry | ✅ | ◐ | ✅ | ✅ |
372
+
373
+ Prefer Open WebUI's chat breadth? It works great pointed at TurboLLM's OpenAI endpoint.
374
+
375
+ ---
376
+
377
+ ## Troubleshooting
378
+
379
+ - **`TurboLLM requires Node.js 22 or newer`** — upgrade Node: <https://nodejs.org>.
380
+ - **Model won't load / OOM** — pick a smaller quant (the VRAM verdict warns you), lower GPU
381
+ offload, or close other GPU apps. Failures surface in the Engines screen with the engine log.
382
+ - **Windows Defender / SmartScreen prompt** — that's the upstream `llama-server` binary on
383
+ first run; allow it once.
384
+ - **Port already in use** — `turbollm --port 9000`.
385
+ - **Slow generation** — open the model's load params; ensure GPU offload is high and flash
386
+ attention / NextN are on for supported models.
387
+
388
+ ---
389
+
390
+ ## Develop from source
126
391
 
127
392
  ```bash
128
393
  npm install # daemon deps
129
394
  cd web && npm install && cd ..
130
395
 
131
396
  npm run build:web # build the React UI -> src/webdist
132
- npm run start # run the daemon in dev (hot TS via tsx)
133
- # open http://127.0.0.1:6996 · curl http://127.0.0.1:6996/api/v1/status
397
+ npm run start # run the daemon in dev (hot TS via tsx) -> :6996
134
398
 
135
399
  npm run build # production bundle -> dist/cli.js (web assets included)
136
400
  node dist/cli.js --port 6996
@@ -139,25 +403,29 @@ node dist/cli.js --port 6996
139
403
  Frontend hot-reload: `cd web && npm run dev` (proxies `/api` and `/v1` to the daemon on
140
404
  :6996).
141
405
 
142
- ### Layout
406
+ **Stack:** Node ≥22 · TypeScript · Hono · `node:sqlite` · tsup — and a React 19 + Tailwind v4 +
407
+ shadcn/ui frontend. One TypeScript codebase, shipped as an npm package.
143
408
 
144
409
  ```
145
410
  turbollm/
146
- package.json npm package; bin "turbollm" -> bin/turbollm.mjs -> dist/cli.js
411
+ bin/turbollm.mjs launcher shim (Node guard) -> dist/cli.js
147
412
  src/
148
413
  cli.ts entrypoint: wiring + graceful shutdown
149
414
  server.ts Hono app: CORS, API, gateway, embedded SPA
150
- config/ config schema + load/save/migrate
151
415
  engines/ provisioning, probe, registry, lifecycle state machine
152
416
  api/routes.ts /api/v1/* handlers
153
417
  gateway/ /v1/* OpenAI + Anthropic gateway
154
- webdist/ built web UI (generated; served by the daemon)
155
- web/ React 19 + TS + Tailwind v4 + shadcn frontend (own package.json)
418
+ models/ · chat/ · hf/ · bench/ · downloads/
419
+ web/ React + TS + Tailwind + shadcn frontend (own package.json)
156
420
  ```
157
421
 
422
+ ---
423
+
158
424
  ## License
159
425
 
160
- Source-available under the **Functional Source License 1.1 (Apache 2.0 future grant)** —
161
- SPDX `FSL-1.1-ALv2`. Free for personal use, internal business use, education, and research;
162
- the only restriction is shipping a competing product. Each release converts to Apache-2.0
163
- two years after it's published. Full text in [LICENSE.md](LICENSE.md).
426
+ Source-available under the **Functional Source License 1.1 (Apache-2.0 future grant)** — SPDX
427
+ **`FSL-1.1-ALv2`**. Free for personal use, internal business use, education, and research; the
428
+ only restriction is shipping a competing product. Each release converts to Apache-2.0 two
429
+ years after it's published. Full text: [LICENSE.md](LICENSE.md).
430
+
431
+ <p align="center"><sub>Built for people who refuse to wait for the mainstream to bless the fast path. ⚡</sub></p>