infer-stack 0.7.0__tar.gz → 0.7.2__tar.gz

infer_stack-0.7.2/PKG-INFO

@@ -0,0 +1,780 @@
+Metadata-Version: 2.4
+Name: infer-stack
+Version: 0.7.2
+Summary: Profile-driven compose and KubeAI deployment compiler for inference stacks
+Author-email: "jon.crall" <jon.crall@kitware.com>
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/AIQ-Kitware/infer_stack
+Classifier: Development Status :: 1 - Planning
+Classifier: Intended Audience :: Developers
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Programming Language :: Python :: 3.15
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Utilities
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: jinja2>=3.1
+Requires-Dist: kwconf>=0.11.0
+Requires-Dist: loguru>=0.7
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: requests>=2.31
+Requires-Dist: rich>=13.0
+Requires-Dist: ubelt>=1.3
+Provides-Extra: tests
+Requires-Dist: pytest>=7.0; extra == "tests"
+Requires-Dist: pytest-codeblocks>=0.17; extra == "tests"
+Requires-Dist: pytest-cov>=3.0; extra == "tests"
+Requires-Dist: xdoctest>=1.1.5; extra == "tests"
+Provides-Extra: tui
+Requires-Dist: textual>=0.50; extra == "tui"
+Dynamic: license-file
+
+# Infer Stack
+
+[![PyPI version](https://img.shields.io/pypi/v/infer-stack.svg)](https://pypi.org/project/infer-stack/)
+[![Python versions](https://img.shields.io/pypi/pyversions/infer-stack.svg)](https://pypi.org/project/infer-stack/)
+[![License](https://img.shields.io/pypi/l/infer-stack.svg)](https://github.com/AIQ-Kitware/infer_stack/blob/main/LICENSE)
+
+Declare models in a catalog (`infer-stack catalog …`) and `acquire` or `run`
+endpoints on demand. `infer-stack help tree` prints the whole command surface;
+[docs/source/manual/](docs/source/manual/) has the Ollama + Open WebUI
+tutorial and the leasing demo.
+
+## Primary leasing workflow
+
+The normal user path has three steps and no separate publication phase:
+
+```bash
+infer-stack config init
+infer-stack catalog suggest --apply
+infer-stack acquire <endpoint>
+```
+
+No GPU here? `infer-stack catalog suggest --simulator --apply` adds
+`mock-smol`, a simulator that answers like vLLM with random text, so the same
+three steps run on any host with Docker ([docs/mock-endpoints.md](docs/mock-endpoints.md)).
+
+`settings.yaml` (`infer-stack config …`) and `catalog.yaml` are the user
+configuration. Leasing keeps an
+internal frozen recovery snapshot so a crash cannot re-render committed state
+with different settings, but ordinary `acquire` advances that snapshot
+automatically. Compatible catalog additions can be acquired while other models
+are live; conflicting redefinitions or global setting changes take effect once
+the affected stack is quiescent. `infer-stack config publish` is an advanced
+pre-seeding/preview tool for multi-catalog operators, not a required fourth
+step. See [ADR 0001](docs/adr/0001-user-config-is-authoritative.md).
+
+## Related work
+
+- [HyperQwen](https://github.com/syv-ai/HyperQwen) is a specialized model-preparation
+  and serving stack for Qwen3.8-27B. infer-stack delegates requantization,
+  patched-vLLM launchers, speculative decoding, and GPU-level tuning to its
+  image, while hardware discovery and suggestion choose explicit catalog
+  profiles by GPU *class* (compute capability + VRAM) rather than product-name
+  strings. Memory-tight 24-32 GiB Ampere-or-newer cards get the reference
+  fast/long/huge choices. High-VRAM Turing cards (sm75-sm79, >=48 GiB) get the
+  measured full-262K FP16/Triton profiles, with the prepared `-fast` checkpoint
+  preferred. Roomy Ampere-or-newer cards (>=48 GiB) get the fast baseline plus
+  a conservative/provisional full-262K ordinary-KV prefab; use
+  `dev/profile_qwen38_hyperqwen.sh` to refine that class on new hardware such as
+  Blackwell. Exact GPU-name gates remain available for future exceptions backed
+  by model-specific measurements. The model identity is
+  `qwen3.8-27b-dbirks-hyperqwen` because it starts from the
+  `dbirks/Qwen3.8-27B-W4A16-AutoRound` derivative; the unsuffixed
+  `qwen3.8-27b` identity is left available for the official checkpoint.
+
+## Supported platform
+
+`infer-stack` supports **Linux hosts only**. Its process locking, container
+runtime integration, GPU discovery, and deployment workflows are intentionally
+Linux-oriented. Windows is not a supported or tested execution platform.
+
+Operational and security constraints that are accepted during the current
+planning-stage release are tracked in
+[docs/planning/known-limitations.md](docs/planning/known-limitations.md).
+
+`infer-stack` serves the endpoints declared in a catalog. `acquire` takes a
+lease on an endpoint; the controller places its engine on free GPUs and
+reconciles the backend to run it:
+
+* **engines**: vLLM (one container per deployment) and Ollama (one daemon per
+  `runtime_hosts` entry, serving many tags);
+* **LiteLLM gateway**: one OpenAI base URL, `http://127.0.0.1:14042/v1`, in
+  front of every endpoint alias. On by default; `config set litellm false`
+  drops it;
+* **Open WebUI**: on by default at `http://127.0.0.1:13000`;
+  `config set ui false` or `acquire --no-ui` drops it;
+* **reverse proxy**: an optional single-port nginx in front of both.
+
+Two backends run this: **Compose** (single host; vLLM and Ollama) and
+**KubeAI** (a Kubernetes cluster; vLLM only).
+
+## Main commands
+
+```bash
+infer-stack config init               # data dir + default backend -> settings.yaml
+infer-stack catalog suggest --apply   # seed catalog.yaml from this host's GPUs
+infer-stack catalog show              # what can be acquired
+infer-stack kube inventory           # Kubernetes/GPU/KubeAI facts, before or after install
+infer-stack kube doctor              # detailed readiness + actionable fixes
+infer-stack kube node test NODE --expected-gpus=1  # plan worker GPU + real generation acceptance
+infer-stack kube node detach NODE     # preview handing a cluster GPU host to Compose
+infer-stack acquire <endpoint>        # lease, render, bring up, wait for a real generation
+infer-stack access <endpoint> --env-file e.env  # reach it, managed or external; leases only what runs here
+infer-stack test <endpoint>           # one generation through the gateway
+infer-stack leases                    # desired vs running, per deployment
+infer-stack status                    # paths, backend and a lease summary
+infer-stack release --all             # drop every lease
+infer-stack paths                     # where settings, catalog, ledger and caches live
+infer-stack version
+infer-stack help tree                 # the whole command surface
+```
+
+The CLI is built on [`kwconf`](https://github.com/Erotemic/kwconf),
+so every subcommand is also importable as a Python class:
+
+```python
+from infer_stack.cli import AcquireCLI, TestCLI
+
+AcquireCLI.main(argv=False, names=['smol135-1'], yes=True)
+TestCLI.main(argv=False, name='smol135-1')
+```
+
+`manage.py` and `infer-stack` are aliases for the same entry point;
+shell examples below use `infer-stack`.
+
+## Operating the rendered Compose stack
+
+`ps` and `logs` read the backend directly; `stack` wraps `docker compose` on
+the rendered project, so you never `cd` into it or repeat `-f`/`--env-file`.
+
+```bash
+infer-stack ps                            # engines, gateway, UI
+infer-stack ps -a                         # include exited instances
+infer-stack logs -f <endpoint>            # follow whatever serves an endpoint
+infer-stack logs --tail 200 litellm       # the gateway's backlog
+infer-stack logs -f --raw litellm         # full LiteLLM tracebacks
+infer-stack stack restart open-webui      # docker compose restart
+infer-stack stack stop                    # stop everything (no remove)
+infer-stack stack start                   # start it back up
+infer-stack stack pull                    # refresh images
+infer-stack stack compose -- ps --format json   # any other docker compose command
+```
+
+`logs` accepts a service or pod name, a container id prefix, a deployment id
+or an endpoint alias. Interactive `infer-stack logs -f` compacts only
+explicitly registered, known-noisy LiteLLM traceback shapes; unknown
+tracebacks pass through unchanged. Redirected or piped output stays raw, and
+`--raw` disables compaction in an interactive follow. `--no-color` drops the
+name-prefix colors. The TUI uses the same compactor.
+
+Ollama tags are pulled into the daemon on the first `acquire` of an endpoint
+that serves them. `stack compose` reaches the daemon for anything else, e.g.
+`infer-stack stack compose -- exec ollama-local-ollama ollama list`.
+
+On the KubeAI backend `ps` and `logs` read pods, and the `stack` Compose verbs
+act on the gateway's Compose project on this host.
+
+## Inspect an endpoint before running it
+
+```bash
+infer-stack catalog show <endpoint>
+infer-stack acquire <endpoint> --no-apply   # declare + write the compose project; start nothing
+infer-stack paths leasing                   # where docker-compose.yml landed
+infer-stack apply                           # start it (or `release --all` to discard)
+```
+
+## Catalog model
+
+`catalog.yaml` is the one user-edited description of what can run. Its
+sections:
+
+* `models`: weight sources (`hf://org/name`, with optional `revision`,
+  `quantization`, `dtype`);
+* `endpoints`: served API names. Each picks an `engine` (`vllm` or `ollama`),
+  a `model` (a `models` key, or an Ollama tag), and optionally `runtime`
+  (vLLM settings), `protocol`, `placement`, `sharing` and `reclaim`;
+* `runtime_hosts`: Ollama daemons, each with its GPUs and daemon settings;
+* `bundles`: named lists of endpoints to acquire together.
+
+An endpoint can instead name a server that already runs elsewhere, with
+`external: {api_base, model, api_key_env}` in place of `engine`/`model`.
+It has no lease: `infer-stack access` (or `run`) publishes its route on the
+LiteLLM front door, and the client asks for the alias through the same base
+URL as any other endpoint. Moving an alias between a managed runtime and an
+external server changes neither the alias nor the workflow:
+
+```bash
+infer-stack env REMOTE_QWEN_KEY=sk-...          # the upstream's key, by name
+infer-stack access qwen-remote --env-file qwen.env
+source qwen.env                                 # OPENAI_BASE_URL, OPENAI_API_KEY, ...
+```
+
+`api_base` is resolved from inside the gateway's container or pod:
+`localhost` there is the gateway itself, so a server on the same host is
+reached through the Docker bridge address (e.g. `http://172.17.0.1:8000/v1`).
+See [docs/planning/external-endpoints.md](docs/planning/external-endpoints.md).
+
+```yaml
+models:
+  smol135:
+    source: hf://HuggingFaceTB/SmolLM2-135M-Instruct
+
+endpoints:
+  smol135-1:
+    engine: vllm
+    model: smol135
+    runtime: {max_model_len: 8192}
+  chat:
+    engine: ollama
+    host: local-ollama
+    model: qwen3.5:4b
+
+runtime_hosts:
+  local-ollama:
+    engine: ollama
+    placement: {gpu_indices: [0]}
+    settings: {keep_alive: 30m}
+
+bundles:
+  both: [smol135-1, chat]
+```
+
+Edit it with `infer-stack catalog model|endpoint|host|bundle add …`, or by
+hand with `infer-stack catalog edit`; `infer-stack catalog validate` checks it.
+The schema reference is the docstring of `infer_stack/leasing/catalog.py`.
+
+Shapes the stack renders:
+
+```text
+Open WebUI -> LiteLLM -> vLLM / Ollama     # the default
+                     \-> an external OpenAI-compatible server (external:)
+Open WebUI -> vLLM / Ollama                # config set litellm false
+```
+
+The named stack profiles of earlier releases (`setup`, `switch`,
+`--profile`) are gone; see
+[docs/stack-graph-profiles.md](docs/stack-graph-profiles.md).
+
+## Where config and rendered artifacts live
+
+`infer-stack` follows XDG basedir conventions, so the directory you invoke it
+from never changes which config it reads or where it writes. There are two
+roots:
+
+| What | Default location | How to relocate |
+| --- | --- | --- |
+| `settings.yaml`, `catalog.yaml` | `~/.config/infer_stack/` (resp. `$XDG_CONFIG_HOME`) | `--config-dir` or `INFER_STACK_CONFIG_DIR` |
+| **Everything generated**: `leasing/` (the ledger, the compose project, its `.env`) and the bind-mounted state (`hf-cache/`, `vllm-cache/`, `open-webui/`, `ollama/`, …) | `~/.local/share/infer_stack/` (resp. `$XDG_DATA_HOME`) | `config set data_dir <path>`, `--data-dir` or `INFER_STACK_DATA_DIR` |
+
+`infer-stack paths` prints every resolved path and whether it exists.
+
+The data dir **relocates the one infer-stack installation controlling a
+host/backend; it does not create an isolated second installation**. Do not run
+controllers from multiple config/data roots against the same Docker host or
+Kubernetes namespace. See the
+[single-owner limitation](docs/planning/known-limitations.md#one-control-plane-per-host-or-backend-namespace).
+
+```bash
+# Persist it once; later commands read it from settings.yaml.
+infer-stack config set data_dir /data/service/docker/infer-stack
+
+# Or keep both roots in a checkout for an ad-hoc experiment.
+export INFER_STACK_CONFIG_DIR=$PWD/cfg INFER_STACK_DATA_DIR=$PWD/stack
+infer-stack paths
+```
+
+`--config-dir` / `--data-dir` are accepted by every subcommand, after the
+subcommand name.
+
+## Constraining placement to specific GPUs
+
+```bash
+# Only place onto GPU 1 (e.g. GPU 0 is running a display).
+infer-stack acquire <endpoint> --allowed-gpus 1
+
+# Or confine a TP=2 endpoint to physical GPUs 1 and 3.
+infer-stack acquire <tp2-endpoint> --allowed-gpus 1,3
+```
+
+`--allowed-gpus` (or `INFER_STACK_ALLOWED_GPUS=1,3`) filters the detected
+inventory before placement for that call only. Real indices are preserved, so
+the rendered compose stack pins `device_ids` to those exact GPUs. The durable
+forms live in data:
+
+* `placement: {gpu_indices: [1]}` on a vLLM endpoint pins it exactly (the list
+  length must equal tp×pp×dp); Ollama daemons pin through their
+  `runtime_hosts` entry;
+* `placement: {min_vram_gib: 24}` makes smaller GPUs ineligible;
+* `config set skip_display_gpus true` (or `--skip-display-gpus`) leaves the
+  GPU driving a monitor free.
+
+## Demos / integration recipes
+
+The user manual under [docs/source/manual/](docs/source/manual/) has two
+walkthroughs on the current CLI:
+[the Ollama + Open WebUI tutorial](docs/source/manual/ollama-openwebui-tutorial.md)
+and [the leasing demo](docs/source/manual/leasing-demo.md) (standing service,
+Open WebUI, several models side by side).
+
+---
+
+## Backend 1: Compose
+
+Use Compose for single-host serving. It runs vLLM and Ollama engines, mixed
+freely, behind the optional gateway and UI.
+
+### Getting started
+
+Prerequisite: Docker and the `docker compose` plugin.
+
+```bash
+infer-stack config init --backend compose
+infer-stack doctor --gpu
+infer-stack catalog init
+
+# A vLLM endpoint.
+infer-stack catalog model add smol135 --source hf://HuggingFaceTB/SmolLM2-135M-Instruct
+infer-stack catalog endpoint add --model smol135    # -> smol135-1
+infer-stack acquire smol135-1
+
+# An Ollama endpoint, on a daemon pinned to GPU 0.
+infer-stack catalog host add local-ollama --engine ollama --gpu 0
+infer-stack catalog endpoint add chat --engine ollama --host local-ollama --model qwen3.5:4b
+infer-stack acquire chat
+```
+
+`infer-stack catalog suggest --apply` fills the catalog with endpoints sized
+for the detected GPUs instead.
+
+### Test that it is responding
+
+With LiteLLM enabled, every endpoint is reachable by its alias at:
+
+```text
+http://127.0.0.1:14042/v1
+```
+
+`acquire` already blocks until the endpoint returns a real generation through
+that front door, which is stronger than Docker's container health. After
+`acquire --no-wait`, block separately; to check again later, send one request:
+
+```bash
+infer-stack wait smol135-1
+infer-stack test smol135-1
+infer-stack test chat --prompt "Name three colors." --max-tokens 32
+```
+
+Clients read the front door and the managed key from the env file:
+
+```bash
+export OPENAI_BASE_URL=$(infer-stack env OPENAI_BASE_URL)
+export OPENAI_API_KEY=$(infer-stack env LITELLM_MASTER_KEY)
+```
+
+or get both, plus per-endpoint names, from
+`infer-stack acquire <endpoint> --env-file lease.env`.
+
+To replace the gateway's master key (refused while leases are active; the
+gateway restarts, and clients must fetch the key again):
+
+```bash
+infer-stack secrets rotate
+```
+
+### Stop it
+
+```bash
+infer-stack release --all            # drop every lease
+infer-stack release --all --evict    # ...and stop the engines now
+infer-stack clean -f                 # no leases, nothing on a GPU; the gateway stays
+infer-stack stack down               # docker compose down, bypassing the ledger
+```
+
+After a plain `release`, `keep-warm` endpoints (the default `reclaim` policy)
+stay loaded until another lease needs their GPUs or `infer-stack evict` stops
+them. `stack down` releases no lease, so the next `apply` or `acquire` brings
+leased models back.
+
+All state is bind-mounted from the data dir, so none of these delete it,
+including `stack down --volumes`. For a destructive reset, remove the
+directories `infer-stack paths` lists.
+
+### Open WebUI authentication
+
+Open WebUI runs with `WEBUI_AUTH=False`: no login screen, and anyone who can
+reach port 13000 gets the UI. No setting changes that. Keep the host on a
+trusted network, or run without the UI (`config set ui false`).
+
+### Reverse proxy
+
+An optional nginx service publishes one HTTP port with the UI at `/` and the
+API at `/v1`. It needs the LiteLLM gateway.
+
+```bash
+infer-stack config set reverse_proxy true                        # port 80
+infer-stack config set reverse_proxy '{enabled: true, port: 8080}'
+```
+
+`acquire --reverse-proxy` turns it on for one call. Add `config_path:
+/path/to/nginx.conf` to the block to mount your own config at
+`/etc/nginx/conf.d/default.conf` instead of the generated one.
+
+It does no TLS and no authentication. Only the one port is published and no
+certificate is mounted, so terminate TLS in a proxy in front of it. The TLS
+and LDAP settings of the pre-leasing profiles no longer exist.
+
+### Persistent state and database layout
+
+Everything lives under the data dir (`infer-stack paths`):
+
+* `open-webui/`: Open WebUI's data directory (accounts, chats, settings),
+  mounted at `/app/backend/data`;
+* `postgres-litellm/`: LiteLLM's route store, rendered only with
+  `config set dynamic_routing true`;
+* `ollama/`: the Ollama model store, mounted at `/root/.ollama`;
+* `hf-cache/`, `vllm-cache/`, `torch-cache/`, `triton-cache/`, `cuda-cache/`:
+  vLLM weights and compile caches (see
+  [docs/persistent-caches-and-warm-restarts.md](docs/persistent-caches-and-warm-restarts.md));
+* `runtime/`: directories an endpoint's `runtime.mounts` asks for;
+* `leasing/`: the ledger and the rendered compose project.
+
+Open WebUI chat history is not tied to the models currently served, so old
+chats may name aliases the gateway no longer advertises. That is expected.
+
+### Custom .env values are preserved
+
+The compose project's `.env` holds the managed secrets (`LITELLM_MASTER_KEY`,
+`HF_TOKEN`, …). `infer-stack env KEY=VALUE` merges a value into it, and keys
+you add are kept across renders. Compose uses the file for interpolation, so a
+key reaches a container only when the rendered service references it (as
+`HF_TOKEN` does for vLLM). Set `HF_TOKEN` before the first `acquire` of a
+gated model.
+
+### Switching models
+
+There is no single active model to switch. `acquire` another endpoint and it
+runs beside the first; release the first when you are done:
+
+```bash
+infer-stack acquire smol135-1
+infer-stack acquire chat          # now both are served
+infer-stack leases                # find smol135-1's lease id
+infer-stack release <lease-id>
+```
+
+The gateway carries a route for every catalog endpoint, so adding or removing
+a model does not recreate LiteLLM, and Open WebUI stays up. When GPUs are
+short, `acquire` fails fast; `--queue` waits for a GPU instead, and idle
+`keep-warm` deployments are evicted to make room. vLLM containers are named
+after the served alias (`vllm-<alias>`), so `docker ps` and
+`infer-stack logs vllm-<alias>` identify them.
+
+### Protocol modes for base vs. instruct models
+
+An endpoint's `protocol` is `chat` (the default) or `completions`. It decides
+which surface the readiness probe and `infer-stack test` use, so a base model
+without a chat template must declare `completions` or its `acquire` never sees
+a ready generation:
+
+```bash
+infer-stack catalog model add pythia-160m --source hf://EleutherAI/pythia-160m
+infer-stack catalog endpoint add --model pythia-160m --protocol completions
+infer-stack acquire pythia-160m-1
+infer-stack test pythia-160m-1              # hits /v1/completions
+```
+
+The gateway forwards `/v1/completions` unchanged, so evaluation clients that
+need exact prompt control should call it directly. Open WebUI is a chat UI and
+sends chat requests, which a base model cannot answer.
+
+### Images with their own launcher
+
+Some images wrap vLLM in their own launcher and are configured through
+environment variables rather than `vllm serve` flags. Describe that in the
+endpoint's `runtime`; infer-stack has no model-specific code for it:
+
+```yaml
+runtime:
+  image: example.org/my-vllm-launcher:1.0
+  max_model_len: 65536
+  gpu_memory_utilization: 0.93
+  command: [single]              # replaces `vllm serve MODEL <flags>`
+  env:                           # container environment
+    PORT: '{port}'
+    MAX_LEN: '{max_model_len}'   # filled from the field above
+    GPU_UTIL: '{gpu_memory_utilization}'
+    EXTRA_ARGS: '--served-model-name={served_model_name}'
+  mounts:                        # persisted under the runtime data dir
+    /app/models: my-launcher/models
+```
+
+Changing the launcher's mode is a data edit, in the catalog or the TUI's
+endpoint editor. The HyperQwen suggestion (see "Related work") is a worked
+example: `catalog suggest` uses compute capability and VRAM to emit the serving
+profiles appropriate to the detected hardware class (and exact name gates only
+for measured exceptions). The hardware check happens only while suggesting;
+the resulting catalog holds ordinary explicit runtime data, so `apply`/`acquire`
+never retunes an endpoint after the fact.
+
+- `{max_model_len}`, `{gpu_memory_utilization}`, `{served_model_name}` and
+  `{port}` are filled in from the endpoint, so a launcher that takes them
+  through its own variables stays in step when the fields change.
+- Env values are written as strings (`true`/`false` for booleans), and `$`
+  is literal. `HF_TOKEN`, `VLLM_ATTENTION_BACKEND`, `CUDA_VISIBLE_DEVICES` and
+  `NVIDIA_VISIBLE_DEVICES` are infer-stack's and are refused.
+- `extra_args` stay what they were: flags appended to the stock `vllm serve`
+  command, after infer-stack's own, so vLLM keeps the extra value for a
+  repeated flag. Repeating a flag infer-stack acts on (served name, parallel
+  sizes, `--max-model-len`) is refused; with `command`, pass flags through the
+  launcher instead.
+- All of these are deployment identity: endpoints that launch differently
+  never share a process.
+- `env` works on both backends (on KubeAI it becomes the Model's `spec.env`).
+  `command` and `mounts` are Compose only; KubeAI refuses an endpoint with
+  either rather than serve stock vLLM in its place.
+
+### Reasoning / thinking models
+
+vLLM separates a reasoning trace from the answer when it is started with a
+reasoning parser. Pass the flag through `runtime.extra_args`:
+
+```yaml
+endpoints:
+  qwen3-think:
+    engine: vllm
+    model: qwen3-0.6b
+    runtime:
+      extra_args: [--reasoning-parser=qwen3]
+```
+
+The parser name depends on the model family and the vLLM version (`vllm serve
+--help` lists them). To test end to end:
+
+```bash
+infer-stack test qwen3-think --prompt "Think step by step: 17*23" --max-tokens 512
+
+# Streaming, through the gateway:
+curl -N "$(infer-stack env OPENAI_BASE_URL)/chat/completions" \
+  -H "Authorization: Bearer $(infer-stack env LITELLM_MASTER_KEY)" \
+  -H 'Content-Type: application/json' \
+  -d '{"model":"qwen3-think","stream":true,
+       "messages":[{"role":"user","content":"Think step by step: 17*23"}]}'
+```
+
+In Open WebUI, reasoning shows up best with streaming enabled in the
+chat settings.
+
+---
+
+## Backend 2: KubeAI
+
+`--backend kubeai` runs the same leasing verbs against a Kubernetes cluster
+running [KubeAI](https://www.kubeai.org): the same catalog, ledger, TTLs,
+env file and TUI, with `Model` custom resources in place of compose
+services and the cluster scheduler in place of the local GPU planner. The
+LiteLLM gateway still fronts everything, so a card sees one `OPENAI_BASE_URL`,
+the managed key and the endpoint alias on either backend.
+
+* Creating/joining a cluster and the distribution boundary:
+  [docs/cluster-setup.md](docs/cluster-setup.md).
+* KubeAI setup, settings and semantics:
+  [docs/kubeai-backend.md](docs/kubeai-backend.md).
+* What matches Compose, what does not yet, and what is deliberate:
+  [docs/backend-parity.md](docs/backend-parity.md); the plan to close the
+  rest: [docs/planning/backend-parity-roadmap.md](docs/planning/backend-parity-roadmap.md).
+
+The short version:
+
+```bash
+infer-stack kube inventory
+infer-stack kube doctor
+# New local cluster / NVIDIA plugin + GFD prerequisites (review before applying):
+infer-stack kube bootstrap --provider=k3s
+sudo -v
+infer-stack kube bootstrap --provider=k3s --apply
+# Automatic profiles from Kubernetes node labels; no temporary values file needed:
+infer-stack kube install
+infer-stack kube install --apply
+infer-stack kube doctor
+infer-stack catalog suggest --backend kubeai
+infer-stack config set backend kubeai
+infer-stack config set kubeai_gateway cluster
+infer-stack doctor --backend kubeai
+infer-stack acquire <endpoint> --ttl 2h --env-file lease.env --yes
+```
+
+If the workstation already has a Compose ledger/configuration that you plan to
+return to, keep that authority intact rather than switching backend kinds in
+the same ledger. The cluster setup guide documents the temporary test pattern:
+use `INFER_STACK_BACKEND=kubeai` with a separate KubeAI data root, and use
+`infer-stack kube node detach/attach` to hand each physical GPU host between
+Kubernetes scheduling and direct Compose use without uninstalling or rejoining
+the node. See [docs/cluster-setup.md](docs/cluster-setup.md).
+
+### Switching an existing recovery ledger to KubeAI
+
+Stopping the Compose realization retains its recovery backend and historical
+ledger. Backend kinds require separate recovery epochs. After releasing every
+old lease and stopping the old runtime:
+
+```bash
+infer-stack release --all --backend compose
+infer-stack stack down --backend compose
+infer-stack config set backend kubeai
+infer-stack status                     # configured kubeai / active recovery compose
+infer-stack ledger rotate              # verify quiescence and preview the archive
+infer-stack ledger rotate --yes        # archive old history; initialize KubeAI epoch
+infer-stack ledger archives            # archive paths + history inspection commands
+infer-stack acquire <endpoint> --yes
+```
+
+Rotation refuses active leases, remaining old runtime objects, and unknown
+runtime state. It does not change configuration/catalogs or silently tear down
+workloads. Old leases, deployments and the recovery profile remain in SQLite
+archives under `<ledger-directory>/archives/`. Rotation is transactional and
+safe to retry; a matching recovery backend makes the command a no-op. `gc`
+reports backend mismatches with transition instructions; `gc --forget` remains
+a history-only operation.
+
+### Kubernetes cluster setup and KubeAI integration
+
+[docs/cluster-setup.md](docs/cluster-setup.md) describes server/worker setup.
+`kube inventory` and `kube status` report structured facts; `--json` includes
+probe errors without discarding other discoveries. Inventory and catalog
+suggestions work before the KubeAI chart or namespace exists.
+
+`kube doctor` checks Kubernetes, NVIDIA scheduling/discovery, and KubeAI in
+dependency order. Top-level `doctor --backend kubeai` remains the operational
+preflight for acquiring work. If Helm installation succeeds but the configured
+KubeAI API URL is unavailable, doctor reports that remaining routing issue.
+Set `kubeai_base_url` to a reachable OpenAI URL for the installed service.
+
+`kube bootstrap` defaults to a read-only plan; `--apply` (or `--yes`) authorizes
+host/cluster changes. K3s is the implemented provisioning provider; inventory,
+installation and the backend work with any Kubernetes distribution. Bootstrap
+always provisions/reconciles the local K3s server, preserves unrelated selected
+contexts and kubeconfigs, installs Helm if missing,
+and reconciles NVIDIA device plugin **0.17.1** with GPU Feature Discovery.
+Install the host NVIDIA driver and container toolkit first. K3s discovers the
+installed runtime on startup; bootstrap restarts local K3s only when runtime
+discovery needs repair. It waits for Ready nodes, GPU allocation and GFD labels, and verifies unknown
+GPU-node runtime handlers with small node-specific canary pods. Root admin
+credentials stay `0600`; a private `0600` copy lives at
+`~/.kube/infer-stack-k3s.yaml`. Existing default configs are preserved. Select the
+local server explicitly with `export KUBECONFIG=~/.kube/infer-stack-k3s.yaml`
+before inventory/install; rerun bootstrap to refresh copied certificates.
+
+`kube install` shows inferred Helm values; `--apply` uses Helm upgrade/install
+and checks readiness afterward. `--dry-run`/`--plan` forces read-only behavior.
+`--namespace`, `--release`, `--chart`, `--version` and `--values` allow overrides.
+Existing named profiles and custom values are preserved; `HF_TOKEN` overrides
+the chart token using a temporary protected file. Generated public values live
+at `<data>/generated/kube/kubeai-values.yaml`.
+
+`kube setup` is a deprecated alias of `kube install`; it no longer owns a
+separate prerequisite workflow. `kube k3s bootstrap` provisions only the local
+server, `join` verifies requested worker membership, and `status` reports local
+membership independently of any stale selected admin context. Setup scripts are compatibility wrappers around the
+package commands.
+
+### Debugging serving failures
+
+`infer-stack kube status` summarizes live Kubernetes state. `infer-stack acquire`
+reports pod failures such as `ImagePullBackOff`, `Unschedulable`, and engine
+crashes. Use `infer-stack ps` and `infer-stack logs -f <endpoint>` to inspect
+model serving. If `/models` works but completions return 404, check that clients
+use the gateway alias; direct KubeAI requests use the Model name from the env
+file (`INFER_STACK_ENDPOINT_*`). Missing `libcuda.so.1` usually indicates that
+the model profile did not request GPUs or select the correct runtime.
+
+---
+
+## Which backend should I start with?
+
+**Compose** when one workstation is enough: it is the fastest path to a
+working server, everything it renders is a file you can read, and it needs
+only Docker.
+
+**KubeAI** when the models must run on more than one machine, or a cluster
+already exists. It is Compose plus a scheduler: the same catalog, verbs and
+env file, with a cluster and `resourceProfiles` supplied. Expect more
+first-request overhead (pod creation, image pull, model load).
+
+A catalog written for Compose runs on KubeAI, with two exceptions: ollama
+endpoints and custom container launches (`runtime.command` / `mounts`),
+which stay Compose-only. [docs/backend-parity.md](docs/backend-parity.md)
+has the full matrix.
+
+## vLLM startup caches
+
+Generated Compose mounts persist Hugging Face, vLLM, PyTorch/TorchInductor,
+Triton, and CUDA JIT caches. Warm starts avoid redownloading and redoing many
+compile/JIT steps, but a vLLM model swap still creates a new engine process and
+must reload weights into GPU memory.
+
+### Diagnosing readiness
+
+`docker compose` health only means a container-level healthcheck passed. It
+is not "the routed model answers a request through the front door", which is
+what `acquire` and `wait` check: a model swap starts a new engine process,
+and LiteLLM stays up while returning upstream connection errors until vLLM
+has loaded the weights.
+
+```bash
+infer-stack acquire <endpoint> --yes     # waits for a real generation
+infer-stack wait <endpoint>              # after acquire --no-wait
+infer-stack test <endpoint>              # one generation through the gateway
+infer-stack status                       # desired vs running, per deployment
+infer-stack logs <service> --tail 80     # the engine or gateway log
+```
+
+A crash-looping engine fails the acquire at once with its error quoted, so
+the timeout is only for a model that is loading. Reading Docker's own
+signals: `litellm exited with code 137` is a SIGKILL (an OOM kill or a forced
+replacement), whereas LiteLLM returning HTTP 500 with `Cannot connect to host
+vllm-*` means LiteLLM is running and its upstream vLLM is not ready yet.
+
+### Join and accept GPU workers
+
+`infer-stack kube k3s onboard namek --server=... --token-file=... --kubeconfig=...`
+plans a local GPU worker join and node-specific acceptance; add `--apply` to
+execute it. Host NVIDIA drivers/toolkit are prerequisites. It infers the server
+version, verifies every locally detected device, then checks actual one-GPU
+KubeAI placement and generation on this worker. Admin credentials are explicit
+private files, separate from any selected EKS context.
+
+For an already joined worker, run `infer-stack kube node test namek
+--expected-gpus=1 --namespace=default --apply` from the control plane. The same
+surface accepts two-device `yardrat` and four-device `aiq-gpu2`; mixed products
+are reported separately. Tests preserve leases and unrelated Models, remove
+their temporary resources, and retain a reusable node profile. See the
+[worker onboarding instructions](docs/cluster-setup.md#join-and-test-a-gpu-worker-in-one-reviewed-operation)
+for plan/apply, credentials, interrupted cleanup and heterogeneous-node limits.
+
+### Kubernetes dashboard
+
+`infer-stack tui` provides a **Cluster** tab in the expanded runtime pane on the
+KubeAI backend. It shows Ready/scheduling state, GPU allocation and scheduled
+requests across namespaces, GFD labels, runtime evidence, managed Models and
+KubeAI/NVIDIA pod startup/restart state. GPU requests are scheduling facts, not
+live GPU utilization; the system pane explicitly describes the local host.
+
+Cluster monitoring uses three batched API reads, cached for at least 15 seconds
+and polled only while the tab is visible. Slow probes cannot overlap or block
+keyboard input; **Refresh cluster** forces a sample, and **Doctor** runs the
+shared detailed checks on demand. Pod samples also feed the Instances view.
+
+Select a node for **Detach node** or **Attach node**. Each previews its workload
+and scheduling state and requires confirmation; detach drains workloads and
+emptyDir data, while attach confirms local Compose workloads have stopped.
+These reuse `kube node detach/attach` and preserve cluster membership. The
+Control tab provides **Apply** through the leasing controller and confirmed
+**Down** for managed Models and the gateway. Leases remain after Down. The
+KubeAI endpoint editor selects resource profiles; Kubernetes chooses node/GPU
+placement.