nodmix 2026.5.25

package/docs/providers/litellm.md

@@ -0,0 +1,234 @@
+---
+summary: "Run Nodmix through LiteLLM Proxy for unified model access and cost tracking"
+title: "LiteLLM"
+read_when:
+  - You want to route Nodmix through a LiteLLM proxy
+  - You need cost tracking, logging, or model routing through LiteLLM
+---
+
+[LiteLLM](https://litellm.ai) is an open-source LLM gateway that provides a unified API to 100+ model providers. Route Nodmix through LiteLLM to get centralized cost tracking, logging, and the flexibility to switch backends without changing your Nodmix config.
+
+<Tip>
+**Why use LiteLLM with Nodmix?**
+
+- **Cost tracking** — See exactly what Nodmix spends across all models
+- **Model routing** — Switch between Claude, GPT-4, Gemini, Bedrock without config changes
+- **Virtual keys** — Create keys with spend limits for Nodmix
+- **Logging** — Full request/response logs for debugging
+- **Fallbacks** — Automatic failover if your primary provider is down
+
+</Tip>
+
+## Quick start
+
+<Tabs>
+  <Tab title="Onboarding (recommended)">
+    **Best for:** fastest path to a working LiteLLM setup.
+
+    <Steps>
+      <Step title="Run onboarding">
+        ```bash
+        nodmix onboard --auth-choice litellm-api-key
+        ```
+
+        For non-interactive setup against a remote proxy, pass the proxy URL explicitly:
+
+        ```bash
+        nodmix onboard --non-interactive --auth-choice litellm-api-key --litellm-api-key "$LITELLM_API_KEY" --custom-base-url "https://litellm.example/v1"
+        ```
+      </Step>
+    </Steps>
+
+  </Tab>
+
+  <Tab title="Manual setup">
+    **Best for:** full control over installation and config.
+
+    <Steps>
+      <Step title="Start LiteLLM Proxy">
+        ```bash
+        pip install 'litellm[proxy]'
+        litellm --model claude-opus-4-6
+        ```
+      </Step>
+      <Step title="Point Nodmix to LiteLLM">
+        ```bash
+        export LITELLM_API_KEY="your-litellm-key"
+
+        nodmix
+        ```
+
+        That's it. Nodmix now routes through LiteLLM.
+      </Step>
+    </Steps>
+
+  </Tab>
+</Tabs>
+
+## Configuration
+
+### Environment variables
+
+```bash
+export LITELLM_API_KEY="sk-litellm-key"
+```
+
+### Config file
+
+```json5
+{
+  models: {
+    providers: {
+      litellm: {
+        baseUrl: "http://localhost:4000",
+        apiKey: "${LITELLM_API_KEY}",
+        api: "openai-completions",
+        models: [
+          {
+            id: "claude-opus-4-6",
+            name: "Claude Opus 4.6",
+            reasoning: true,
+            input: ["text", "image"],
+            contextWindow: 200000,
+            maxTokens: 64000,
+          },
+          {
+            id: "gpt-4o",
+            name: "GPT-4o",
+            reasoning: false,
+            input: ["text", "image"],
+            contextWindow: 128000,
+            maxTokens: 8192,
+          },
+        ],
+      },
+    },
+  },
+  agents: {
+    defaults: {
+      model: { primary: "litellm/claude-opus-4-6" },
+    },
+  },
+}
+```
+
+## Advanced configuration
+
+### Image generation
+
+LiteLLM can also back the `image_generate` tool through OpenAI-compatible
+`/images/generations` and `/images/edits` routes. Configure a LiteLLM image
+model under `agents.defaults.imageGenerationModel`:
+
+```json5
+{
+  models: {
+    providers: {
+      litellm: {
+        baseUrl: "http://localhost:4000",
+        apiKey: "${LITELLM_API_KEY}",
+      },
+    },
+  },
+  agents: {
+    defaults: {
+      imageGenerationModel: {
+        primary: "litellm/gpt-image-2",
+        timeoutMs: 180_000,
+      },
+    },
+  },
+}
+```
+
+Loopback LiteLLM URLs such as `http://localhost:4000` work without a global
+private-network override. For a LAN-hosted proxy, set
+`models.providers.litellm.request.allowPrivateNetwork: true` because the API key
+will be sent to the configured proxy host.
+
+<AccordionGroup>
+  <Accordion title="Virtual keys">
+    Create a dedicated key for Nodmix with spend limits:
+
+    ```bash
+    curl -X POST "http://localhost:4000/key/generate" \
+      -H "Authorization: Bearer $LITELLM_MASTER_KEY" \
+      -H "Content-Type: application/json" \
+      -d '{
+        "key_alias": "nodmix",
+        "max_budget": 50.00,
+        "budget_duration": "monthly"
+      }'
+    ```
+
+    Use the generated key as `LITELLM_API_KEY`.
+
+  </Accordion>
+
+  <Accordion title="Model routing">
+    LiteLLM can route model requests to different backends. Configure in your LiteLLM `config.yaml`:
+
+    ```yaml
+    model_list:
+      - model_name: claude-opus-4-6
+        litellm_params:
+          model: claude-opus-4-6
+          api_key: os.environ/ANTHROPIC_API_KEY
+
+      - model_name: gpt-4o
+        litellm_params:
+          model: gpt-4o
+          api_key: os.environ/OPENAI_API_KEY
+    ```
+
+    Nodmix keeps requesting `claude-opus-4-6` — LiteLLM handles the routing.
+
+  </Accordion>
+
+  <Accordion title="Viewing usage">
+    Check LiteLLM's dashboard or API:
+
+    ```bash
+    # Key info
+    curl "http://localhost:4000/key/info" \
+      -H "Authorization: Bearer sk-litellm-key"
+
+    # Spend logs
+    curl "http://localhost:4000/spend/logs" \
+      -H "Authorization: Bearer $LITELLM_MASTER_KEY"
+    ```
+
+  </Accordion>
+
+  <Accordion title="Proxy behavior notes">
+    - LiteLLM runs on `http://localhost:4000` by default
+    - Nodmix connects through LiteLLM's proxy-style OpenAI-compatible `/v1`
+      endpoint
+    - Native OpenAI-only request shaping does not apply through LiteLLM:
+      no `service_tier`, no Responses `store`, no prompt-cache hints, and no
+      OpenAI reasoning-compat payload shaping
+    - Hidden Nodmix attribution headers (`originator`, `version`, `User-Agent`)
+      are not injected on custom LiteLLM base URLs
+  </Accordion>
+</AccordionGroup>
+
+<Note>
+For general provider configuration and failover behavior, see [Model Providers](/concepts/model-providers).
+</Note>
+
+## Related
+
+<CardGroup cols={2}>
+  <Card title="LiteLLM Docs" href="https://docs.litellm.ai" icon="book">
+    Official LiteLLM documentation and API reference.
+  </Card>
+  <Card title="Model selection" href="/concepts/model-providers" icon="layers">
+    Overview of all providers, model refs, and failover behavior.
+  </Card>
+  <Card title="Configuration" href="/gateway/configuration" icon="gear">
+    Full config reference.
+  </Card>
+  <Card title="Model selection" href="/concepts/models" icon="brain">
+    How to choose and configure models.
+  </Card>
+</CardGroup>

package/docs/providers/lmstudio.md

@@ -0,0 +1,224 @@
+---
+summary: "Run Nodmix with LM Studio"
+read_when:
+  - You want to run Nodmix with open source models via LM Studio
+  - You want to set up and configure LM Studio
+title: "LM Studio"
+---
+
+LM Studio is a friendly yet powerful app for running open-weight models on your own hardware. It lets you run llama.cpp (GGUF) or MLX models (Apple Silicon). Comes in a GUI package or headless daemon (`llmster`). For product and setup docs, see [lmstudio.ai](https://lmstudio.ai/).
+
+## Quick start
+
+1. Install LM Studio (desktop) or `llmster` (headless), then start the local server:
+
+```bash
+curl -fsSL https://lmstudio.ai/install.sh | bash
+```
+
+2. Start the server
+
+Make sure you either start the desktop app or run the daemon using the following command:
+
+```bash
+lms daemon up
+```
+
+```bash
+lms server start --port 1234
+```
+
+If you are using the app, make sure you have JIT enabled for a smooth experience. Learn more in the [LM Studio JIT and TTL guide](https://lmstudio.ai/docs/developer/core/ttl-and-auto-evict).
+
+3. If LM Studio authentication is enabled, set `LM_API_TOKEN`:
+
+```bash
+export LM_API_TOKEN="your-lm-studio-api-token"
+```
+
+If LM Studio authentication is disabled, you can leave the API key blank during interactive Nodmix setup.
+
+For LM Studio auth setup details, see [LM Studio Authentication](https://lmstudio.ai/docs/developer/core/authentication).
+
+4. Run onboarding and choose `LM Studio`:
+
+```bash
+nodmix onboard
+```
+
+5. In onboarding, use the `Default model` prompt to pick your LM Studio model.
+
+You can also set or change it later:
+
+```bash
+nodmix models set lmstudio/qwen/qwen3.5-9b
+```
+
+LM Studio model keys follow a `author/model-name` format (e.g. `qwen/qwen3.5-9b`). Nodmix
+model refs prepend the provider name: `lmstudio/qwen/qwen3.5-9b`. You can find the exact key for
+a model by running `curl http://localhost:1234/api/v1/models` and looking at the `key` field.
+
+## Non-interactive onboarding
+
+Use non-interactive onboarding when you want to script setup (CI, provisioning, remote bootstrap):
+
+```bash
+nodmix onboard \
+  --non-interactive \
+  --accept-risk \
+  --auth-choice lmstudio
+```
+
+Or specify the base URL, model, and optional API key:
+
+```bash
+nodmix onboard \
+  --non-interactive \
+  --accept-risk \
+  --auth-choice lmstudio \
+  --custom-base-url http://localhost:1234/v1 \
+  --lmstudio-api-key "$LM_API_TOKEN" \
+  --custom-model-id qwen/qwen3.5-9b
+```
+
+`--custom-model-id` takes the model key as returned by LM Studio (e.g. `qwen/qwen3.5-9b`), without
+the `lmstudio/` provider prefix.
+
+For authenticated LM Studio servers, pass `--lmstudio-api-key` or set `LM_API_TOKEN`.
+For unauthenticated LM Studio servers, omit the key; Nodmix stores a local non-secret marker.
+
+`--custom-api-key` remains supported for compatibility, but `--lmstudio-api-key` is preferred for LM Studio.
+
+This writes `models.providers.lmstudio` and sets the default model to
+`lmstudio/<custom-model-id>`. When you provide an API key, setup also writes the
+`lmstudio:default` auth profile.
+
+Interactive setup can prompt for an optional preferred load context length and applies it across the discovered LM Studio models it saves into config.
+LM Studio plugin config trusts the configured LM Studio endpoint for model requests, including loopback, LAN, and tailnet hosts. Metadata/link-local origins still require explicit opt-in. You can opt out by setting `models.providers.lmstudio.request.allowPrivateNetwork: false`.
+
+## Configuration
+
+### Streaming usage compatibility
+
+LM Studio is streaming-usage compatible. When it does not emit an OpenAI-shaped
+`usage` object, Nodmix recovers token counts from llama.cpp-style
+`timings.prompt_n` / `timings.predicted_n` metadata instead.
+
+Same streaming usage behavior applies to these OpenAI-compatible local backends:
+
+- vLLM
+- SGLang
+- llama.cpp
+- LocalAI
+- Jan
+- TabbyAPI
+- text-generation-webui
+
+### Thinking compatibility
+
+When LM Studio's `/api/v1/models` discovery reports model-specific reasoning
+options, Nodmix exposes the matching OpenAI-compatible `reasoning_effort`
+values in model compat metadata. Current LM Studio builds can advertise binary
+UI options such as `allowed_options: ["off", "on"]` while rejecting those values
+on `/v1/chat/completions`; Nodmix normalizes that binary discovery shape to
+`none`, `minimal`, `low`, `medium`, `high`, and `xhigh` before sending requests.
+Older saved LM Studio config that contains `off`/`on` reasoning maps is
+normalized the same way when the catalog is loaded.
+
+### Explicit configuration
+
+```json5
+{
+  models: {
+    providers: {
+      lmstudio: {
+        baseUrl: "http://localhost:1234/v1",
+        apiKey: "${LM_API_TOKEN}",
+        api: "openai-completions",
+        models: [
+          {
+            id: "qwen/qwen3-coder-next",
+            name: "Qwen 3 Coder Next",
+            reasoning: false,
+            input: ["text"],
+            cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 },
+            contextWindow: 128000,
+            maxTokens: 8192,
+          },
+        ],
+      },
+    },
+  },
+}
+```
+
+## Troubleshooting
+
+### LM Studio not detected
+
+Make sure LM Studio is running. If authentication is enabled, also set `LM_API_TOKEN`:
+
+```bash
+# Start via desktop app, or headless:
+lms server start --port 1234
+```
+
+Verify the API is accessible:
+
+```bash
+curl http://localhost:1234/api/v1/models
+```
+
+### Authentication errors (HTTP 401)
+
+If setup reports HTTP 401, verify your API key:
+
+- Check that `LM_API_TOKEN` matches the key configured in LM Studio.
+- For LM Studio auth setup details, see [LM Studio Authentication](https://lmstudio.ai/docs/developer/core/authentication).
+- If your server does not require authentication, leave the key blank during setup.
+
+### Just-in-time model loading
+
+LM Studio supports just-in-time (JIT) model loading, where models are loaded on first request. Nodmix preloads models through LM Studio's native load endpoint by default, which helps when JIT is disabled. To let LM Studio's JIT, idle TTL, and auto-evict behavior own model lifecycle, disable Nodmix's preload step:
+
+```json5
+{
+  models: {
+    providers: {
+      lmstudio: {
+        baseUrl: "http://localhost:1234/v1",
+        api: "openai-completions",
+        params: { preload: false },
+        models: [{ id: "qwen/qwen3.5-9b" }],
+      },
+    },
+  },
+}
+```
+
+### LAN or tailnet LM Studio host
+
+Use the LM Studio host's reachable address, keep `/v1`, and make sure LM Studio is bound beyond loopback on that machine:
+
+```json5
+{
+  models: {
+    providers: {
+      lmstudio: {
+        baseUrl: "http://gpu-box.local:1234/v1",
+        apiKey: "lmstudio",
+        api: "openai-completions",
+        models: [{ id: "qwen/qwen3.5-9b" }],
+      },
+    },
+  },
+}
+```
+
+`lmstudio` automatically trusts its configured local/private endpoint for guarded model requests. Custom/local OpenAI-compatible provider entries also trust their exact configured `baseUrl` origin, except metadata/link-local origins; requests to different private ports or destinations still require `models.providers.<id>.request.allowPrivateNetwork: true`. Set `models.providers.<id>.request.allowPrivateNetwork: false` to opt out of exact-origin trust.
+
+## Related
+
+- [Model selection](/concepts/model-providers)
+- [Ollama](/providers/ollama)
+- [Local models](/gateway/local-models)