oumigo 0.1.0__tar.gz

oumigo-0.1.0/.env.example

@@ -0,0 +1,58 @@
+# Copy to `.env` and fill in. Your real `.env` is gitignored — never commit it.
+# In production, supply these via systemd EnvironmentFile= or the orchestrator.
+
+
+#----------------------------------------------------------------------------
+# oumigo manager — development / sandbox environment.
+#----------------------------------------------------------------------------
+
+# Shared bearer token workers must present. If unset, worker authentication is
+# DISABLED (v1 dev behavior) and the manager warns at startup.
+# Generate:  python -c "import secrets; print(secrets.token_urlsafe(32))"
+OUMIGO_MANAGER_TOKEN=
+
+# Optional: manager config file (else the search path is used). Overridden by
+# --config-file on the command line.
+# OUMIGO_CONFIG_FILE=./manager.yaml
+
+
+#----------------------------------------------------------------------------
+# oumigo worker  — development / sandbox environment.
+#----------------------------------------------------------------------------
+
+# Negotiable model config: these override what the manager hands the worker. Set
+# MODEL_NAME to serve a different model than the manager's node_spec (or to serve
+# at all when the manager has no model configured). MAX_MODEL_LEN caps context.
+# Applies to both --backend=vllm and --backend=transformer.
+#
+# MODEL_NAME=google/gemma-4-E2B-it
+# MAX_MODEL_LEN=8192
+
+# Optional: helps with HF download rate limits (in the case model is not gated).
+HF_TOKEN=
+
+# Hugging Face cache. Point at a SHARED cache to avoid re-downloading the GBs models.
+# Must be set BEFORE transformers is imported — the app loads this .env first.
+# HF_HOME=~/.hf_cache
+
+# Optional: vLLM's own cache (torch.compile / CUDA-graph artifacts, NOT model
+# weights — those live under HF_HOME). Defaults to ~/.cache/vllm if unset. Point
+# at a shared/persistent volume to avoid recompiling on every start.
+#
+# VLLM_CACHE_ROOT=~/.vllm_cache
+
+# --- FlashInfer JIT compile failures (EngineCore fails to start) ------------
+# vLLM JIT-compiles some FlashInfer CUDA kernels at startup, which is fragile if
+# the box's CUDA toolchain is inconsistent. Two levers, most reliable first:
+#
+# 1) Reliable: skip FlashInfer's JIT sampler. vLLM uses its native top-k/top-p
+#    path; if attention already uses Triton, FlashInfer isn't otherwise needed.
+# VLLM_USE_FLASHINFER_SAMPLER=0
+#
+# 2) Root-cause (only if the compiler, not the toolkit, is wrong): point CUDA_HOME
+#    at a *self-consistent* CUDA toolkit so nvcc + headers agree. The pip CUDA
+#    wheels ship one in the venv (find .venv -type d -path '*/nvidia/cu13') — but
+#    note those wheels can be minor-skewed (e.g. nvcc 13.2 vs headers 13.0), which
+#    FlashInfer's CCCL guard rejects; then only lever 1 helps.
+#
+# CUDA_HOME=/abs/path/to/.venv/lib/pythonX.Y/site-packages/nvidia/cu13

oumigo-0.1.0/.gitignore

@@ -0,0 +1,22 @@
+# Python
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+*.egg
+
+# Virtual envs / tooling
+.venv/
+venv/
+.env
+.uv/
+.ruff_cache/
+.mypy_cache/
+.pytest_cache/
+
+# Editors / OS
+.vscode/
+.idea/
+.DS_Store

oumigo-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 GotoAI Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

oumigo-0.1.0/PKG-INFO

@@ -0,0 +1,91 @@
+Metadata-Version: 2.4
+Name: oumigo
+Version: 0.1.0
+Summary: Vertical-integration toolkit for running and managing GPU fleets with vLLM and Transformer backends.
+Project-URL: Homepage, https://github.com/gotoai/oumigo
+Project-URL: Repository, https://github.com/gotoai/oumigo
+Project-URL: Issues, https://github.com/gotoai/oumigo/issues
+Author: GOTOAI
+License-Expression: MIT
+License-File: LICENSE
+Keywords: cluster,inference,llm,serving,vllm
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: POSIX :: Linux
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Requires-Dist: fastapi>=0.110
+Requires-Dist: httpx>=0.27
+Requires-Dist: pydantic-settings>=2
+Requires-Dist: pydantic>=2
+Requires-Dist: pyyaml>=6
+Requires-Dist: typer>=0.12
+Requires-Dist: uvicorn>=0.29
+Requires-Dist: zeroconf>=0.130
+Provides-Extra: conoha
+Provides-Extra: dev
+Requires-Dist: mypy>=1.10; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.23; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.5; extra == 'dev'
+Provides-Extra: manager
+Provides-Extra: worker
+Requires-Dist: accelerate; extra == 'worker'
+Requires-Dist: bitsandbytes; extra == 'worker'
+Requires-Dist: pillow; extra == 'worker'
+Requires-Dist: timm; extra == 'worker'
+Requires-Dist: transformers>=5.10.1; extra == 'worker'
+Requires-Dist: vllm>=0.5; extra == 'worker'
+Description-Content-Type: text/markdown
+
+# oumigo
+
+A vertical-integration toolkit for running and managing **vLLM replica fleets** — a
+manager/driver node coordinating many independent vLLM worker replicas behind a
+health-aware router, with a pluggable cloud-provisioning layer.
+
+> Status: early development. The CLI, manager console, config resolution, and the
+> LAN provider exist; core worker↔manager registration, the router, and vLLM
+> supervision are not built yet.
+
+## Architecture
+
+Two roles:
+
+- **Worker** (`oumigo.service.worker`): a long-lived *coordinator* supervises a vLLM server
+  as a child process, monitors health, executes start/stop/restart from the manager,
+  and owns the node state machine + restart-with-give-up policy. Workers self-register
+  with the manager and heartbeat.
+- **Manager** (`oumigo.service.manager`): coordinates the fleet, split into sub-layers:
+  - **control plane** (`manager.control`): tracks worker registrations and state,
+    drives worker lifecycle, reconciles desired vs. actual. Low-frequency,
+    correctness-critical.
+  - **data plane / router** (`manager.router`): forwards client inference calls to
+    healthy workers — on the hot path of every request.
+  - **provisioning** (`oumigo.providers`): how workers come into existence — a
+    minimal, lifecycle-shaped `Provider` protocol used by the control plane. Ships
+    with `StaticProvider` (LAN: workers are hand-started and self-register, no
+    provisioning); cloud backends (e.g. ConoHa, OpenStack-based) are future
+    implementations of the same protocol.
+  - **dashboard** (`manager.dashboard`): performance & diagnostics — later.
+
+Shared foundations: `oumigo.config` (typed settings + precedence resolution) and
+`oumigo.protocol` (the wire contract both roles import so it can't drift).
+
+## Development
+
+```bash
+# from oumigo/
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[worker,dev]"     # on a GPU worker box
+pip install -e ".[manager,dev]"    # on the manager box
+oumigo version
+```
+
+To consume this in-development package from a sibling project, editable-install it
+into that project's environment (e.g. `pip install -e ../oumigo`).

oumigo-0.1.0/README.md

@@ -0,0 +1,47 @@
+# oumigo
+
+A vertical-integration toolkit for running and managing **vLLM replica fleets** — a
+manager/driver node coordinating many independent vLLM worker replicas behind a
+health-aware router, with a pluggable cloud-provisioning layer.
+
+> Status: early development. The CLI, manager console, config resolution, and the
+> LAN provider exist; core worker↔manager registration, the router, and vLLM
+> supervision are not built yet.
+
+## Architecture
+
+Two roles:
+
+- **Worker** (`oumigo.service.worker`): a long-lived *coordinator* supervises a vLLM server
+  as a child process, monitors health, executes start/stop/restart from the manager,
+  and owns the node state machine + restart-with-give-up policy. Workers self-register
+  with the manager and heartbeat.
+- **Manager** (`oumigo.service.manager`): coordinates the fleet, split into sub-layers:
+  - **control plane** (`manager.control`): tracks worker registrations and state,
+    drives worker lifecycle, reconciles desired vs. actual. Low-frequency,
+    correctness-critical.
+  - **data plane / router** (`manager.router`): forwards client inference calls to
+    healthy workers — on the hot path of every request.
+  - **provisioning** (`oumigo.providers`): how workers come into existence — a
+    minimal, lifecycle-shaped `Provider` protocol used by the control plane. Ships
+    with `StaticProvider` (LAN: workers are hand-started and self-register, no
+    provisioning); cloud backends (e.g. ConoHa, OpenStack-based) are future
+    implementations of the same protocol.
+  - **dashboard** (`manager.dashboard`): performance & diagnostics — later.
+
+Shared foundations: `oumigo.config` (typed settings + precedence resolution) and
+`oumigo.protocol` (the wire contract both roles import so it can't drift).
+
+## Development
+
+```bash
+# from oumigo/
+python -m venv .venv
+source .venv/bin/activate
+pip install -e ".[worker,dev]"     # on a GPU worker box
+pip install -e ".[manager,dev]"    # on the manager box
+oumigo version
+```
+
+To consume this in-development package from a sibling project, editable-install it
+into that project's environment (e.g. `pip install -e ../oumigo`).

oumigo-0.1.0/docs/api.md

@@ -0,0 +1,372 @@
+# oumigo Python API — Reference
+
+> Status: implemented. The client library lives under `oumigo.api` (the manager/worker
+> *handles*, the spawn/attach functions, and the `agent` inference layer). The *services*
+> those handles talk to live under `oumigo.service`. Everything in this document is
+> importable from the top-level `oumigo` package.
+
+The API has two layers:
+
+1. **Fleet control** — spawn or attach a manager and workers from Python:
+   `oumigo_get_or_create_manager`, `oumigo_create_worker`, and the `OumigoManager` /
+   `OumigoWorker` handles.
+2. **Inference** — run chats and tools against the fleet's OpenAI-compatible data plane:
+   `OumigoManager.create_agent(...)` → `OumigoAgent` → `OumigoChat` → `OumigoResponse`,
+   plus the `@tool` decorator.
+
+```python
+from oumigo import (
+    oumigo_get_or_create_manager, oumigo_create_worker,   # fleet control
+    OumigoManager, OumigoWorker,                          # handles
+    tool, Tool, ToolDefinitionError,                      # tools
+    OumigoAgent, OumigoChat, OumigoResponse,              # inference
+)
+```
+
+---
+
+## Quickstart
+
+```python
+from oumigo import oumigo_get_or_create_manager, tool
+
+# 1. Attach to a LAN manager, or spawn one (dies with this process).
+manager = oumigo_get_or_create_manager(config_file="./manager.yaml")
+
+# 2. Declare a tool — its JSON schema is inferred from the signature + docstring.
+@tool
+def get_current_date(utc_offset_hours: int = 9) -> str:
+    """Get the current date at a given UTC offset.
+
+    Args:
+        utc_offset_hours: Timezone offset from UTC, in hours (e.g. 9 for JST).
+    """
+    from datetime import datetime, timedelta, timezone
+    return datetime.now(timezone(timedelta(hours=utc_offset_hours))).strftime("%Y-%m-%d")
+
+# 3. Make an agent (tools + sampling defaults) and a stateful chat.
+agent = manager.create_agent(tools=[get_current_date])
+chat = agent.create_chat(system="You are concise.")
+
+# 4. Ask. The tool loop runs automatically.
+print(chat.request("What's the date?").text)
+
+# Streaming:
+for piece in chat.request("And tomorrow?", stream=True):
+    print(piece, end="")
+```
+
+---
+
+## Fleet control
+
+### `oumigo_get_or_create_manager(...) -> OumigoManager`
+
+Return a manager handle, **reusing** a live manager discovered on the LAN (mDNS) or
+**spawning** one as a child of this process (armed with `PR_SET_PDEATHSIG`, so it exits
+when this process does). Blocks until the control plane is healthy.
+
+```python
+oumigo_get_or_create_manager(
+    bearer_token=None, config_file=None,
+    data_host=..., data_port=..., control_host=..., control_port=...,
+    provider=..., model=...,
+    *, discover_timeout=3.0, startup_timeout=20.0,
+) -> OumigoManager
+```
+
+Settings resolve **explicit arg > `config_file` (YAML) > built-in default**. A missing or
+unparseable `config_file` is ignored (a warning is logged).
+
+> **Discovery caveat:** if a manager is already advertising on the LAN, it is reused and
+> your `config_file` is **ignored** (the returned handle has `owned=False`). To guarantee
+> your config is used, stop any other manager first, or edit the config of the manager
+> that is actually running.
+
+### `oumigo_create_worker(...) -> OumigoWorker`
+
+Spawn a worker child on this host and block until its vLLM/HF replica is `SERVING`
+(includes the model load/download — potentially many minutes).
+
+```python
+oumigo_create_worker(
+    bearer_token=None, hf_home="~/.hf_cache", vllm_cache_root="~/.vllm_cache",
+    hf_token=None, model_name=None,
+    *, manager=None, backend="vllm", manager_url=None,
+    discover_timeout=10.0, serving_timeout=None, poll_interval=2.0,
+) -> OumigoWorker
+```
+
+The manager is resolved as: `manager` handle > `manager_url` > the last manager created in
+this process > mDNS. `serving_timeout=None` waits **indefinitely** (you own the give-up
+decision); the wait still ends early on a definitive failure (child exits or node reaches
+`FAILED`).
+
+### `OumigoManager`
+
+A handle to a running manager (spawned or discovered). Context manager; `stop()` is a
+no-op for a discovered (`owned=False`) manager.
+
+| Member | Description |
+|---|---|
+| `control_url` / `data_url` | Control-plane and data-plane base URLs. |
+| `token` | Shared bearer token (or `None`). |
+| `owned` | `True` only if this process spawned the child. |
+| `is_healthy(timeout_s=2.0) -> bool` | `True` once `/healthz` answers 200. |
+| `workers(timeout_s=5.0) -> list[dict]` | Current worker registry records. |
+| `metrics(*, since=None, prefixes=None, timeout_s=5.0) -> list[dict]` | Latest slot per node, or raw historical points when `since` is set. |
+| `create_agent(...) -> OumigoAgent` | Mint an inference agent (see below). |
+| `stop()` | Terminate the spawned control-plane child (owned only). |
+
+### `OumigoWorker`
+
+A handle to a worker child this process spawned. Context manager.
+
+| Member | Description |
+|---|---|
+| `address` / `port` / `model` / `backend` / `node_id` | Replica identity. |
+| `state() -> str \| None` | Node state the manager last saw (e.g. `"serving"`). |
+| `is_serving() -> bool` | `True` when the replica is `SERVING`. |
+| `is_alive() -> bool` | `True` while the child process runs. |
+| `stop()` | Drain and shut down the replica (35 s grace before SIGKILL). |
+
+---
+
+## Inference
+
+### `OumigoManager.create_agent(...) -> OumigoAgent`
+
+```python
+manager.create_agent(
+    tools=None,
+    *, max_iterations=5,
+    temperature=None, max_tokens=None, top_p=None, stop=None,
+) -> OumigoAgent
+```
+
+An **agent** is a capability bundle bound to the manager's data plane: the tools and
+sampling defaults shared by every chat it spawns.
+
+- `tools` — a sequence of `Tool` (or plain functions, which are wrapped via the strict
+  `@tool` validator). Duplicate tool names raise `ValueError`.
+- `max_iterations` — cap on model round-trips per request (the runaway tool-loop guard;
+  default 5). On the cap, the response's `finish_reason` is `"max_iterations"`.
+- Sampling defaults apply to every chat/request; override individually later as needed.
+
+The system prompt and tools are **server-owned** — they come from your code here, never
+from a client. (See [Security & the trust boundary](#security--the-trust-boundary).)
+
+### `OumigoAgent.create_chat(...) -> OumigoChat`
+
+```python
+agent.create_chat(system=None, max_history_turns=3, history=None) -> OumigoChat
+```
+
+Start a **stateful** conversation.
+
+- `system` — system-role content, prepended to every request in this chat.
+- `max_history_turns` — how many prior `(user, assistant)` exchanges to carry into each
+  request. `0` disables memory. Default 3.
+- `history` — prior conversation to seed (for stateless servers; see
+  [Stateless servers](#stateless-servers-history-rehydration)). Only `user`/`assistant`
+  turns are accepted.
+
+> `OumigoChat` is **not thread-safe** — use one chat per session.
+
+### `OumigoChat`
+
+| Member | Description |
+|---|---|
+| `request(contents: str, stream=False) -> OumigoResponse` | Run one user turn to completion (executes the tool loop). |
+| `history -> list[dict]` | A copy of the carried `{"role","content"}` turns — persist this to rehydrate later. Never includes system/tool/reasoning. |
+
+`request()` assembles `[system?] + recent history + {"role":"user","content":contents}`,
+posts to the data plane, and runs the **agent loop**: while the model asks for tools,
+execute the matching Python callbacks, feed their results back, and repeat — until the
+model returns prose or `max_iterations` is reached. The `(user, final-answer)` exchange is
+appended to `history`.
+
+### `OumigoResponse`
+
+The result of one `request()` — the same type whether or not you streamed.
+
+| Attribute | Description |
+|---|---|
+| `text: str` | The final answer (complete immediately for non-stream; complete after iteration for stream). |
+| `reasoning: str` | The model's thinking (`reasoning_content`), kept **out of** `text`. `""` unless the worker runs a `--reasoning-parser`. Output-only. |
+| `finish_reason: str \| None` | `"stop"`, `"length"`, or `"max_iterations"`. |
+| `tool_calls_made: list[dict]` | One `{"name","arguments","result"}` per executed tool, in order. |
+| `raw: dict \| None` | The last raw completion payload (escape hatch). |
+
+**Iteration** (streaming):
+
+```python
+resp = chat.request("hi", stream=True)
+
+for piece in resp:            # answer text deltas (str) — the default
+    ...
+
+for piece in resp.stream():   # identical to `for piece in resp`
+    ...
+
+for text, reasoning in resp.stream(get_reasoning=True):   # dual channel
+    if reasoning:
+        ...  # live "thinking" (reasoning_content deltas)
+    if text:
+        ...  # the answer (content deltas)
+```
+
+In `stream(get_reasoning=True)`, exactly one side of each pair is non-empty (the other is
+`""`, never `None`). Iterating a response drives its one-shot request, so use `for piece
+in resp` **or** one `stream(...)` call — not both. Either way `text` and `reasoning`
+accumulate; after consumption, re-iterating yields the accumulated value once.
+
+---
+
+## Tools
+
+Decorate a plain function with `@tool`; its JSON schema is inferred from the signature
+(types → required/optional) and the Google-style docstring (descriptions).
+
+```python
+from typing import Literal
+from oumigo import tool
+
+@tool
+def get_weather(city: str, units: Literal["celsius", "fahrenheit"] = "celsius") -> str:
+    """Get the current weather for a city.
+
+    Args:
+        city: City name, e.g. Tokyo.
+        units: Temperature unit.
+    """
+    ...
+```
+
+A decorated function is a `Tool` that **remains callable** (`get_weather("Tokyo")` still
+works) and carries `.name`, `.description`, `.parameters`, plus `.to_openai()` and
+`.invoke(**kwargs)`.
+
+### Strict validation (fail fast at import)
+
+`@tool` validates the declaration **at decoration time** and raises a single, aggregated
+`ToolDefinitionError` (listing every problem, with `file:line`) if anything is wrong.
+Hard errors include:
+
+- a parameter with no type annotation, or an unsupported type (`Any`, bare `list`/`dict`,
+  multi-type unions, arbitrary classes);
+- supported types are `str`, `int`, `float`, `bool`, `list[T]`, `Literal[...]` (→ enum),
+  and `Optional[X]` / `X | None`;
+- `*args` / `**kwargs` or positional-only parameters (the loop calls tools by keyword);
+- a default whose type doesn't match its annotation;
+- a missing docstring, an **undocumented** parameter, or a docstring `Args:` that drifts
+  from the signature;
+- a missing (or non-serializable) return annotation.
+
+**Escape hatches:**
+- `@tool(strict=False)` demotes *documentation* checks (docstring/param descriptions) to
+  warnings; structural checks stay hard errors.
+- `@tool(parameters={...})` supplies an explicit JSON Schema instead of inferring — still
+  validated against the signature (property names must match; each needs a type +
+  description).
+
+### Runtime tool errors
+
+If a tool raises when the model calls it (bad args, exception, unknown tool), the loop
+**catches it and feeds the error back to the model** as the tool result (`"Error: ..."`),
+so the model can recover — `request()` never crashes on a tool failure.
+
+---
+
+## Stateless servers (history rehydration)
+
+The XBCOM app (and any web service) is stateless per request, while `OumigoChat` is
+stateful. The rule: **store the history (data), not the Chat (object).** Persist a
+session's `chat.history` in your own store (dict / Redis / DB), and rehydrate a fresh,
+ephemeral chat per request:
+
+```python
+SESSIONS: dict[str, list] = {}                 # your session store
+AGENT = manager.create_agent(tools=[...])      # server-owned tools
+SYSTEM = "You are the XBCOM recommender."
+
+def handle(session_id: str, user_msg: str) -> str:
+    chat = AGENT.create_chat(system=SYSTEM, history=SESSIONS.get(session_id))  # rehydrate
+    resp = chat.request(user_msg)                                             # run one turn
+    SESSIONS[session_id] = chat.history                                       # persist
+    return resp.text
+```
+
+This is stateless-scalable (any process can serve any session), restart-safe, and
+sidesteps `OumigoChat`'s single-thread constraint (each request gets its own chat).
+
+> `max_history_turns` currently bounds **both** what the model sees and what `chat.history`
+> retains (the most recent N exchanges). Set it to how much context you want to keep.
+
+---
+
+## Security & the trust boundary
+
+Rehydrating history from a client-held or stored blob is a hardening point. `create_chat(
+history=...)` enforces a **trust boundary** (`_normalize_history`):
+
+- **Only `user`/`assistant` turns are accepted.** A `system` or `tool` role raises
+  `ValueError` — so a blob cannot inject a fake system prompt or a forged tool result.
+- **Smuggled keys are stripped** (`tool_calls`, `reasoning_content`, …); each turn is
+  rebuilt as a clean `{"role","content"}` dict — so spoofed tool calls can't sneak in.
+- **System prompt and tools stay server-owned** — supplied by your code via `create_agent`
+  / `create_chat(system=...)`, never from the request payload.
+
+Two caveats:
+
+1. This protects the *history*, not the *current user turn*, which an attacker always
+   controls. Prompt injection via the live message ("ignore your instructions…") is a
+   separate concern — screen it with a guardrail layer, and never let model output
+   directly authorize a privileged action (tool calls touching real systems must be
+   independently authorized server-side).
+2. `reasoning_content` is **output-only**: surface it for display/debugging, but it is
+   never stored in history or fed back to the model.
+
+---
+
+## Fleet configuration notes (vLLM)
+
+Behavior of the inference layer depends on how each worker's vLLM was launched. Pass extra
+flags through the `model.extra_args` list in `manager.yaml`; they are appended verbatim to
+`vllm serve`.
+
+**Tool calling** requires vLLM to be started with an auto tool-choice parser, or any
+request carrying `tools` fails with HTTP 400:
+
+```yaml
+model:
+  name: google/gemma-4-12B-it-qat-w4a16-ct
+  extra_args:
+    - --enable-auto-tool-choice
+    - --tool-call-parser
+    - gemma4            # pick the parser matching your model
+```
+
+**Reasoning** must be separated by a matching parser, or the model's channel/thinking
+control tokens leak into `content`. Add:
+
+```yaml
+    - --reasoning-parser
+    - gemma4
+```
+
+With the reasoning parser on, `resp.text` is the clean answer and `resp.reasoning` holds
+the thinking; without it, `resp.reasoning` is `""`.
+
+> `extra_args` flows `manager.yaml` → `NodeSpec` → the worker's `vllm serve`. The worker
+> fetches its spec from the manager at startup, so a change requires **restarting the
+> manager and the worker**.
+
+---
+
+## Notes & limits
+
+- The inference layer is synchronous/blocking; streaming is a synchronous generator.
+- Every model call funnels through one internal seam (`OumigoChat._payload`), reserved for
+  a future guardrail interceptor chain — not yet wired.
+- Sampling defaults currently cover `temperature` / `max_tokens` / `top_p` / `stop`.