nemo-platform-plugin 0.1.0__py3-none-any.whl

nemo_platform/skills/__init__.py

@@ -0,0 +1,28 @@
+# SPDX-FileCopyrightText: Copyright (c) 2025-2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+"""Bundled NeMo Platform skills, exposed via the ``nemo.skills`` entry point.
+
+This package serves a dual role:
+
+* As a Python package, it exports :func:`skills_dir`, the entry-point callable
+  registered under ``nemo.skills`` in the package's ``pyproject.toml``. The
+  skills registry uses the same discovery mechanism for both built-in and
+  plugin-provided skills, so the platform appears as just one provider among
+  many.
+
+* As a data directory, it holds the bundled ``<skill-name>/SKILL.md`` files
+  shipped with NeMo Platform. The vendor tool copies the entire directory into
+  the SDK package, so the SDK ships these alongside the generated client code.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+__all__ = ["skills_dir"]
+
+
+def skills_dir() -> Path:
+    """Return the directory containing this package's bundled skills."""
+    return Path(__file__).parent

nemo_platform/skills/inference/SKILL.md

@@ -0,0 +1,564 @@
+---
+name: inference
+description: >
+  End-to-end reference for inference on the NeMo platform — registering LLM
+  backends as ModelProviders, wiring them to VirtualModels with Switchyard
+  middleware (random routing, translate) and `nemo-guardrails`
+  content-safety rails, and hitting them via the nemo CLI. Use when the task
+  involves registering inference providers, discovering served models,
+  creating VirtualModels, configuring switchyard middleware, layering
+  guardrails alongside translate (correct middleware ordering for
+  OpenAI/Anthropic cross-format setups), making inference calls through IGW,
+  or debugging routing and translation failures locally. For platform startup,
+  Switchyard install, and DB-reset prerequisites, see the `nemo-setup` skill
+  (repo-level `SETUP.md`, mirrored at `.agents/skills/nemo-setup/SKILL.md`).
+user-invocable: true
+allowed-tools: Bash, Read, Grep
+---
+
+# Inference Reference — ModelProvider + VirtualModel + Switchyard
+
+## Prerequisites
+
+This skill assumes the platform is already running and any required plugins
+(`nemo-switchyard`, `nemo-guardrails`) are loaded. For local-platform startup,
+Switchyard install, and DB-reset choices, follow the **`nemo-setup` skill**
+(`SETUP.md` at the repo root, also at `.agents/skills/nemo-setup/SKILL.md`)
+first and then return here.
+
+## API key environment variables
+
+Applies to both branches. Before creating secrets or running notebook/test
+harness flows, confirm API key environment variables:
+
+1. Check whether `INFERENCE_NVIDIA_API_KEY` and/or `NVIDIA_API_KEY` are already
+   set in the user's shell.
+2. Treat `INFERENCE_NVIDIA_API_KEY` as the source of truth for the
+   `https://inference-api.nvidia.com/v1` provider secret and notebook/test
+   harness flows.
+3. If `INFERENCE_NVIDIA_API_KEY` is not set, ask whether the user has the same
+   token in another environment variable. Common alias:
+   `NVIDIA_INFERENCE_API_KEY`.
+4. If the user identifies an alias variable, export it into
+   `INFERENCE_NVIDIA_API_KEY` for the current shell/session before continuing.
+5. If `NVIDIA_API_KEY` is required for a Build/NIM-style provider or seeding
+   path such as `https://integrate.api.nvidia.com`, and is not set, ask
+   whether another env var already contains that token and export it into
+   `NVIDIA_API_KEY` if the user confirms.
+
+Example alias normalization:
+
+```bash
+export INFERENCE_NVIDIA_API_KEY="$NVIDIA_INFERENCE_API_KEY"
+```
+
+Use `INFERENCE_NVIDIA_API_KEY` for inference-api secret creation:
+
+```bash
+printf '%s' "$INFERENCE_NVIDIA_API_KEY" | nemo secrets create nvidia-inference-key \
+  --from-file - --workspace my-workspace
+```
+
+## Environment
+
+- **Port**: `8080` (CLI default — do NOT pass a custom `--base-url`)
+- **`export NMP_BASE_URL=http://localhost:8080` — required when targeting a local platform.** If your `~/.config/nmp/config.yaml` already points at a remote cluster, the CLI uses that base URL and ignores the local platform entirely. Setting this env var overrides the config file for the current shell session.
+- Workspace: `my-workspace` (substitute as needed; `default` also works)
+- Backend: `https://inference-api.nvidia.com/v1`
+
+For platform startup (`nemo services run`), Switchyard install, and state reset, see the `nemo-setup` skill (`SETUP.md` at the repo root).
+
+---
+
+## CLI gotchas (real failures observed)
+
+- **`nemo secrets create`** uses `--from-file` (pipe key in). No `--value` flag.
+- **`nemo inference providers create`** takes `<name>` as a **positional** arg.
+  Same for `update-status`, `get`, `delete`.
+- **`nemo virtual-models create`** is a **top-level** command (not under `nemo inference`) and takes `<name>` as positional.
+- **There is no `nemo inference chat completions create` command.** Use
+  `nemo inference gateway model post <path> <vm-name> --workspace <ws> --body '<json>'`.
+- **`example` is not a valid `--services` arg.** Valid services: `audit`,
+  `auth`, `customization`, `data-designer`, `entities`, `evaluation`, `files`,
+  `guardrails`, `hello-world`, `inference-gateway`, `intake`, `jobs`, `models`,
+  `safe-synthesizer`, `secrets`, `studio`.
+- **The reconciler re-syncs `served_models` from backend discovery every few
+  seconds and drops manually-registered entries.** Always point VM `models` at
+  auto-discovered entity IDs (e.g. `my-workspace/aws-anthropic-claude-opus-4-5`)
+  — these survive reconciler cycles. If a VM 404s shortly after working, the
+  reconciler overwrote your entry. Re-run `update-status` with the **full list**
+  (it replaces, not appends) or switch to auto-discovered IDs.
+- **`update-status` replaces the entire `served_models` list** — include all
+  entries, not just the new one.
+
+---
+
+## Step 1 — Workspace + Secret + Provider
+
+`my-workspace` is not created automatically. Create it first, then the secret
+and provider.
+
+```bash
+nemo workspaces create my-workspace
+
+printf '%s' "$INFERENCE_NVIDIA_API_KEY" | nemo secrets create nvidia-inference-key \
+  --from-file - --workspace my-workspace
+
+nemo inference providers create nvidia-inference \
+  --workspace my-workspace \
+  --host-url "https://inference-api.nvidia.com/v1" \
+  --api-key-secret-name "nvidia-inference-key"
+
+nemo wait inference provider nvidia-inference --workspace my-workspace
+```
+
+Auto-discovery runs within ~3s.
+
+For NVIDIA Build / NIM via `https://integrate.api.nvidia.com`, use
+`NVIDIA_API_KEY`:
+
+```bash
+printf '%s' "$NVIDIA_API_KEY" | nemo secrets create nvidia-build-key \
+  --from-file - --workspace my-workspace
+
+nemo inference providers create nvidia-build \
+  --workspace my-workspace \
+  --host-url "https://integrate.api.nvidia.com" \
+  --api-key-secret-name "nvidia-build-key"
+
+nemo wait inference provider nvidia-build --workspace my-workspace
+```
+
+For Anthropic direct (needs auth header rewrite):
+
+```bash
+printf '%s' "$ANTHROPIC_API_KEY" | nemo secrets create anthropic-api-key \
+  --from-file - --workspace my-workspace
+
+nemo inference providers create anthropic \
+  --workspace my-workspace \
+  --host-url "https://api.anthropic.com" \
+  --api-key-secret-name "anthropic-api-key" \
+  --auth-header-format "X-Api-Key: {{ auth_secret }}" \
+  --default-extra-headers '{"anthropic-version": "2023-06-01"}'
+```
+
+`{{ auth_secret }}` is Jinja2 — substituted at request time. Without it the
+gateway defaults to `Authorization: Bearer` which Anthropic rejects.
+
+---
+
+## Step 2 — Discover available models
+
+After the provider is ready, `served_models` contains auto-discovered entries.
+**Always use these entity IDs** in VM `--models` — they survive the reconciler.
+
+```bash
+# All entity IDs (use these in --models and inference body)
+nemo inference providers get nvidia-inference --workspace my-workspace \
+  --output-format json | jq -r '.served_models[].model_entity_id'
+
+# Side-by-side: served_model_name → model_entity_id
+nemo inference providers get nvidia-inference --workspace my-workspace \
+  --output-format json \
+  | jq -r '.served_models[] | "\(.served_model_name)  →  \(.model_entity_id)"'
+
+# Filter by vendor prefix
+nemo inference providers get nvidia-inference --workspace my-workspace \
+  --output-format json \
+  | jq -r '.served_models[] | select(.served_model_name | startswith("aws/anthropic")) | .model_entity_id'
+
+# Count
+nemo inference providers get nvidia-inference --workspace my-workspace \
+  --output-format json | jq '.served_models | length'
+```
+
+Entity ID normalization: slashes/dots → dashes, workspace prefix added.
+`aws/anthropic/claude-opus-4-5` → `my-workspace/aws-anthropic-claude-opus-4-5`
+
+---
+
+## Step 3 — Body + model entity convention
+
+`body["model"]` must be a **real backend entity ID** that IGW can resolve — not
+the VM name. The VM name is resolved via the URL path
+(`gateway model post v1/chat/completions <vm-name>`), not the body.
+
+Exception: VMs that **rewrite** `body["model"]` (random_routing,
+`ModelFormatLookupProcessor` in translate) — here the initial body model can be
+the VM name since the rewrite resolves it to the real entity.
+
+For VMs without a rewriting middleware: always send the real auto-discovered
+entity in the body.
+
+### Common failure: HTTP 422 from chat completion
+
+If `nemo agents invoke …`, an `openai`-SDK client, or a `langchain-openai`
+client returns HTTP 422 in well under a second with no provider call made,
+the gateway rejected the request's `model` field as malformed. The most
+common cause is sending the **upstream catalog name** (e.g.
+`meta/llama-3.3-70b-instruct` — vendor slash, dotted version) instead of
+the **hyphenated entity ID** (`meta-llama-3-3-70b-instruct`).
+
+The entity ID is what `nemo models list` returns and what every gateway
+input expects. The slash-with-dots form is `served_model_name` — a
+human-display alias from the upstream provider's catalog. It appears in
+provider metadata and error messages, but is **never** a valid request
+field. (A `workspace/entity-id` prefix is fine — that's a workspace
+qualifier, not the upstream alias. The rejected form is specifically
+`vendor/upstream.dotted.name`.)
+
+Diagnose and recover:
+
+```bash
+# 1. What entity IDs actually exist?
+nemo inference providers get nvidia-build --workspace default \
+  --output-format json \
+  | jq -r '.served_models[] | "\(.served_model_name)  →  \(.model_entity_id)"'
+
+# 2. If you set NEMO_DEFAULT_MODEL, confirm it's the hyphenated entity ID.
+echo "$NEMO_DEFAULT_MODEL"
+
+# 3. If a NAT-style deployed agent's resolved config carries the upstream
+#    slash form in model_name, re-deploy with NEMO_DEFAULT_MODEL set to the
+#    entity ID. The .config.llms.agent.model_name path is specific to NAT
+#    workflows (e.g. the calculator-agent example); other workflow types store
+#    the model elsewhere.
+nemo agents deployments list --workspace default \
+  | jq -r '.data[] | "\(.name)  model=\(.config.llms.agent.model_name // "n/a")"'
+```
+
+For an OpenAI-SDK or LangChain client, pass the entity ID (with or without
+the `workspace/` prefix) as `model=`:
+
+```python
+client.chat.completions.create(
+    model="default/meta-llama-3-3-70b-instruct",   # OR "meta-llama-3-3-70b-instruct"
+    messages=[...],
+)
+# NOT: model="meta/llama-3.3-70b-instruct"  → 422
+```
+
+This is intentional gateway behavior: accepting arbitrary slash-prefixed
+names would force IGW to guess provider attribution for every request
+instead of resolving it from a registered entity.
+
+---
+
+## Step 4 — VirtualModel patterns
+
+### Random routing — same format (deterministic test: `strong_probability=1.0`)
+
+```bash
+nemo virtual-models create vm-random-strong --workspace my-workspace \
+  --models '[
+    {"model":"my-workspace/nvidia-mistralai-mixtral-8x22b-instruct-v01","backend_format":"OPENAI_CHAT"},
+    {"model":"my-workspace/nvidia-qwen-qwen3-32b","backend_format":"OPENAI_CHAT"}
+  ]' \
+  --request-middleware '[{"name":"nemo-switchyard","config_type":"random_routing","config":{
+    "strong":{"model":"my-workspace/nvidia-mistralai-mixtral-8x22b-instruct-v01"},
+    "weak":{"model":"my-workspace/nvidia-qwen-qwen3-32b"},
+    "strong_probability":1.0,
+    "rng_seed":42,
+    "enable_stats":false
+  }}]'
+```
+
+### Random routing — cross format (chain routing + translate)
+
+`random_routing` picks the backend; `translate` rewrites the request format.
+Order matters: routing first, translate second. Use `response_middleware` too
+for full round-trip translation back to the client's format.
+
+```bash
+nemo virtual-models create vm-random-cross --workspace my-workspace \
+  --models '[
+    {"model":"my-workspace/aws-anthropic-claude-opus-4-5","backend_format":"ANTHROPIC_MESSAGES"},
+    {"model":"my-workspace/nvidia-nvidia-nemotron-nano-31b-v3","backend_format":"OPENAI_CHAT"}
+  ]' \
+  --request-middleware '[
+    {"name":"nemo-switchyard","config_type":"random_routing","config":{
+      "strong":{"model":"my-workspace/aws-anthropic-claude-opus-4-5"},
+      "weak":{"model":"my-workspace/nvidia-nvidia-nemotron-nano-31b-v3"},
+      "strong_probability":0.5,
+      "enable_stats":false
+    }},
+    {"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"auto","enable_stats":false}}
+  ]' \
+  --response-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"auto","enable_stats":false}}]'
+```
+
+### Translate — cross format, full round-trip ✓
+
+Client sends OpenAI shape, backend is Anthropic, response comes back as OpenAI.
+**Must list translate in BOTH `request_middleware` and `response_middleware`.**
+
+```bash
+nemo virtual-models create vm-translate-cross --workspace my-workspace \
+  --models '[{"model":"my-workspace/aws-anthropic-claude-opus-4-5","backend_format":"ANTHROPIC_MESSAGES"}]' \
+  --request-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}}]' \
+  --response-middleware '[{"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}}]'
+```
+
+### Guardrails — adding content safety to a VirtualModel
+
+Guardrail rails are attached as a `nemo-guardrails` MiddlewareCall. Configs are
+created separately with `nemo guardrail configs create` — see the
+**nemo-guardrails** skill for rails config JSON, prompt templates, task LLMs,
+and streaming output rails. This section only covers the VirtualModel wiring.
+
+The same call must appear in `--request-middleware` (input rails),
+`--response-middleware` (output rails), or both (full coverage). A call only on
+the request side fires input rails only; only on the response side fires output
+rails only.
+
+**Output rails only** — block bad bot responses (most common):
+
+```bash
+nemo virtual-models create vm-guarded --workspace my-workspace \
+  --models '[{"model":"my-workspace/<backend-entity-id>","backend_format":"OPENAI_CHAT"}]' \
+  --response-middleware '[{
+    "name":"nemo-guardrails",
+    "config_type":"guardrail_config",
+    "config_id":"my-workspace/content-safety"
+  }]'
+```
+
+**Input + output rails** — full coverage. Include the call in **both** lists:
+
+```bash
+nemo virtual-models create vm-guarded-full --workspace my-workspace \
+  --models '[{"model":"my-workspace/<backend-entity-id>","backend_format":"OPENAI_CHAT"}]' \
+  --request-middleware '[{"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"my-workspace/content-safety"}]' \
+  --response-middleware '[{"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"my-workspace/content-safety"}]'
+```
+
+The guardrails plugin doesn't route — the VirtualModel's `--models` array
+decides the upstream backend. Inline configs (no stored entity) are also
+supported via `"config":{...}` instead of `"config_id"`.
+
+#### Guardrails + Switchyard `translate` — ordering matters
+
+The `nemo-guardrails` plugin only understands **OpenAI chat completions**
+shape. When the backend is in a different format (e.g. `ANTHROPIC_MESSAGES`)
+and a `translate` middleware is also in the chain, guardrails must see the
+OpenAI form on both sides:
+
+- **Request stack:** `guardrails` BEFORE `translate`. Guardrails inspects the
+  OpenAI request from the client, then translate rewrites it to the backend
+  format.
+- **Response stack:** `guardrails` AFTER `translate`. Translate converts the
+  backend response back to OpenAI first, then guardrails runs output rails on
+  the OpenAI shape.
+- **Inference calls:** keep the request body in OpenAI shape regardless of
+  the backend's `backend_format`. The translate middleware handles the
+  conversion to/from backend format; sending Anthropic-shaped bodies bypasses
+  guardrails' input rails because the plugin can't parse them.
+
+```bash
+nemo virtual-models create vm-guarded-translate --workspace my-workspace \
+  --models '[{"model":"my-workspace/aws-anthropic-claude-opus-4-5","backend_format":"ANTHROPIC_MESSAGES"}]' \
+  --request-middleware '[
+    {"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"my-workspace/content-safety"},
+    {"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}}
+  ]' \
+  --response-middleware '[
+    {"name":"nemo-switchyard","config_type":"translate","config":{"target_format":"anthropic","enable_stats":false}},
+    {"name":"nemo-guardrails","config_type":"guardrail_config","config_id":"my-workspace/content-safety"}
+  ]'
+```
+
+Same principle applies when `random_routing` mixes OpenAI and Anthropic
+backends behind one VirtualModel: put `guardrails` first in the request chain
+(before any routing/translate decisions), and put `guardrails` last in the
+response chain (after translate has normalized everything back to OpenAI).
+
+If a rails config declares a task LLM via `models[]` (e.g. `content_safety`,
+`topic_control`), the task LLM must itself be addressable as OpenAI chat
+completions — point `models[].model` at an OpenAI-format entity ID, not at an
+Anthropic-format backend. The guardrails plugin doesn't translate task-LLM
+calls.
+
+---
+
+## Step 5 — Making inference calls
+
+**Preferred (nemo CLI):**
+
+```bash
+nemo inference gateway model post v1/chat/completions <vm-name> \
+  --workspace my-workspace \
+  --body '{"model":"my-workspace/<entity-id>","messages":[{"role":"user","content":"hi"}],"max_tokens":15}'
+```
+
+**Verify routing by running multiple times** (model field in response flips
+between backends at the configured probability). Parse with Python — see the
+parsing-pitfalls subsection below for why `jq` is the wrong tool here:
+
+```bash
+for i in $(seq 1 10); do
+  nemo inference gateway model post v1/chat/completions vm-random-cross \
+    --workspace my-workspace \
+    --body '{"model":"my-workspace/vm-random-cross","messages":[{"role":"user","content":"hi"}],"max_tokens":400}' \
+    | python3 -c 'import json,sys; d=json.loads(sys.stdin.read(), strict=False); print(d.get("model"))'
+done
+```
+
+For reasoning models (e.g. `nemotron-nano-*`), use `max_tokens` ≥ ~200; the
+model spends most of its budget on a hidden reasoning trace and produces an
+empty `content` if cut off mid-trace (`finish_reason: length`). That is not
+an inference error, just a budget shortfall.
+
+**curl (streaming or when you need raw output):**
+
+```bash
+curl -s -X POST \
+  "http://localhost:8080/apis/inference-gateway/v2/workspaces/my-workspace/openai/-/v1/chat/completions" \
+  -H "Content-Type: application/json" \
+  -d '{"model":"my-workspace/<entity-id>","messages":[{"role":"user","content":"Hello"}],"max_tokens":64,"stream":true}'
+```
+
+### Parsing pitfalls — `jq` and reasoning-model responses
+
+Reasoning models (`nemotron-nano-*` and similar) sometimes emit responses
+that contain raw control characters (literal U+0000–U+001F bytes — typically
+unescaped newlines or tabs) inside the `reasoning_content` field. This is
+technically invalid JSON. Python's `json.loads` rejects it by default but can
+be configured with `strict=False` to accept it. **`jq` does not** — it bails
+out with:
+
+```
+jq: parse error: Invalid string: control characters from U+0000 through
+U+001F must be escaped at line N, column M
+```
+
+In a routing-verification loop driven by `jq -r '.model'`, every nemotron
+response will look like a failure (empty `model` field) while every opus
+response parses fine — making the split look catastrophically broken when
+inference is actually working. **Always parse these responses with Python
+using `strict=False`** (or a similarly lenient parser) instead of `jq`:
+
+```bash
+echo "$resp" | python3 -c '
+import json, sys
+d = json.loads(sys.stdin.read(), strict=False)
+print(d.get("model"), d["choices"][0].get("finish_reason"))
+'
+```
+
+If you must stay in shell, `jq --slurp -R 'fromjson?'` will tolerate most of
+these payloads but is brittle and not worth the effort. Python is the right
+tool.
+
+---
+
+## Verifying routing decisions (DEBUG logs)
+
+Start with `LOG_LEVEL=DEBUG`. Filter service output:
+
+```bash
+grep -E "RandomRoutingRequestProcessor|picked tier|FormatTranslate|StampOriginalFormat" \
+  <service-output-file>
+```
+
+Expected output:
+```
+debug RandomRoutingRequestProcessor: picked tier=strong model=my-workspace/aws-anthropic-claude-opus-4-5
+debug RandomRoutingRequestProcessor: picked tier=weak   model=my-workspace/nvidia-nvidia-nemotron-nano-31b-v3
+```
+
+`tier=` and `model=` in DEBUG logs match `response.model` returned to the client.
+
+---
+
+## Failure cases
+
+### ❌ Translate in `request_middleware` only — response shape mismatch
+
+The most common mistake. Request translates fine, backend call succeeds, but the
+client receives the backend's **native** response shape instead of OpenAI:
+
+| Field | ✓ translate in both | ✗ request only |
+|---|---|---|
+| `id` prefix | `chatcmpl-` (OpenAI) | `msg_` (Anthropic) |
+| Shape | `choices[0].message.content` | `content[0].text` |
+| Extra keys | — | `type`, `stop_reason`, `stop_sequence` |
+
+Fix: always add `response_middleware` with the same translate config.
+
+### ❌ Path mismatch without translate → 502
+
+Sending `/v1/messages` traffic to a VM backed by an OpenAI-format model with no
+translate fails because IGW forwards the inbound path unchanged. Switchyard's
+`PathUpdateProcessor` only rewrites `request.path` when translate is configured.
+
+### ❌ Reconciler-induced 404
+
+VM works briefly then returns "Model entity not found". The reconciler overwrote
+`served_models` from upstream auto-discovery. Fix: use auto-discovered entity IDs
+in VM `--models` (they survive reconciler cycles) instead of manually-registered
+aliases.
+
+### ❌ NVIDIA hub silently accepts format mismatches
+
+OpenAI body at `/v1/chat/completions` routed to an Anthropic backend **without**
+translate does not fail at the upstream — NVIDIA hub accepts OpenAI-shaped bodies
+for Anthropic models and returns the Anthropic-native response. The visible
+failure is the response shape mismatch above (case 1), not a 4xx/5xx.
+
+### ❌ Guardrails after translate (request) or before translate (response)
+
+`nemo-guardrails` parses **OpenAI** chat shape only. With an Anthropic backend
+and `request_middleware=[translate, guardrails]`, guardrails sees the
+already-translated Anthropic body and either skips its rails or errors on the
+unexpected shape. Symmetrically, `response_middleware=[guardrails, translate]`
+runs guardrails against the raw Anthropic response. Fix: keep guardrails on the
+OpenAI side of translate in both chains — `request=[guardrails, translate]`,
+`response=[translate, guardrails]`. Same applies to clients sending
+Anthropic-shaped request bodies: input rails won't fire. Send OpenAI shape and
+let translate do the conversion.
+
+---
+
+## Troubleshooting
+
+**DB disk I/O error on startup** — orphaned WAL journal files. Delete all three:
+```bash
+rm -rf ~/.local/share/nemo
+```
+
+**`nemo-switchyard` fails to load at startup** — `switchyard.lib` not importable.
+Run `uv sync` from the repo root with default groups enabled to install the
+plugin and its vendored `switchyard` library
+(`plugins/nemo-switchyard/vendor/switchyard/`), then restart services.
+
+**401 from Anthropic** — missing or wrong `--auth-header-format`. Verify:
+```bash
+nemo inference providers get anthropic --workspace my-workspace --output-format json \
+  | jq '.auth_header_format'
+```
+
+**No served models after provider creation** — wait ~10s for the first reconciler
+cycle, then check:
+```bash
+nemo inference providers get nvidia-inference --workspace my-workspace \
+  --output-format json | jq '.served_models | length'
+```
+
+
+
+---
+
+## Cleanup
+
+```bash
+# Delete all switchyard test VMs
+for vm in $(nemo virtual-models list --workspace my-workspace --output-format json \
+  | jq -r '.data[].name' | grep vm-); do
+  nemo virtual-models delete "$vm" --workspace my-workspace
+done
+
+nemo inference providers delete nvidia-inference --workspace my-workspace
+nemo secrets delete nvidia-inference-key --workspace my-workspace
+nemo workspaces delete my-workspace
+```