legion-llm 0.9.54 → 0.10.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: '09bf7eb9fe4c93ccba0bb574864c66a518225327399b47a30dd08aa148ac3b74'
-  data.tar.gz: a54854addf081e387d94ed9fd3ba67f826df9b64ec3b042b2767a7e5a441d715
+  metadata.gz: 94a90f69b3940eb805b6b7bf0b15eb30aa8c616e66562560df01d3f0d5b24ca2
+  data.tar.gz: 4a90c910e19cec4b3da0e5e44a00d1076495d41d653ad4f7161b3b38892f8d8b
 SHA512:
-  metadata.gz: 69fa173952297d7da6410c101c9b9e13548514db85075188fdef153d0f9340f50c314e3f633432e5481c5a4d979468ea84dc3b24edff84d4d728a6d7df7f94c0
-  data.tar.gz: 52852f542515ec121bf1d9851506c3866e70cae5a551f034777f15d9c68fc7886888eb6d9c5cb9a9f58e18016d84d92d4dd010a58cc7c312b2c683bfb477815d
+  metadata.gz: e5f444b505e4b75937223475300cd4bae1e2f292f93148fbb3f0a06446ae6b90b0bf482a7d308a3e580e55219ddc4eb6f0e9274f6479c6e96619d54946cc0ef6
+  data.tar.gz: 6a9dfedae0e30d61f1bf8c4dc825764e4e9174829cc2d1ea6ee65a00ab7dea2cec5299e5988117aacd24adba93d956970124e00c11a1360bacfab04aa9cf6044

data/CHANGELOG.md

@@ -1,5 +1,71 @@
 # Legion LLM Changelog
 
+
+## [0.10.1] - 2026-05-29
+
+### Fixed
+- **Anthropic message format translation** — `AnthropicRequest` translator now properly converts `tool_use` content blocks to `tool_calls` arrays and `tool_result` blocks to `role: :tool` messages. This was the root cause of vLLM outputting JSON blobs instead of calling tools — it was receiving malformed messages with raw Anthropic content block hashes in the content field.
+- **Streaming tool_use event ordering** — Rewrote namespace streaming to emit tool_use content blocks inline after stream completes. Text `content_block_start` is deferred until actual text arrives, matching real Anthropic API behavior for tool-only responses.
+- **Curator current-turn preservation** — `curate_turn` now only curates messages older than the current turn. `apply_curation_pipeline` preserves recent turns by counting user messages (`preserve_recent_turns` setting, default 2). Prevents the model from losing context of its recent work.
+- **LexLLMAdapter TypeError flood** — Guarded `text_part_content` and `defined_method_access` against Array inputs. Flattened nested content arrays before iterating.
+- **Rescue log levels** — All rescue blocks in inference steps, executor, dispatch, and adapter now use `:warn` or `:error` (never `:debug`). Caught exceptions should be visible.
+- **Thinking override removed** — No longer forces `thinking: { enabled: false }` for vLLM with tools. Lets the model use its thinking template properly.
+- **Model routing hardcode removed** — Anthropic namespace no longer hardcodes `routing: { model: 'legionio' }`. Uses daemon default_provider/default_model from settings.
+
+### Changed
+- **RAG defaults** — `min_confidence` 0.85 → 0.92, `full_limit` 10 → 5, `compact_limit` 5 → 3. Tighter relevance threshold, fewer context injections.
+- **AnthropicResponse translator** — `extract_tool_calls`, `format_stop_reason`, `token_count` are now public methods (needed by streaming handler).
+- **Message.build debug log removed** — Fired dozens of times per request with no diagnostic value.
+- **format_chunk log** — Now includes actual text content and index for debugging stream flow.
+- **Curator debug logs** — Show full before/after content (no truncation).
+- **Request info log** — Namespace handler logs message count, char count, estimated tokens, and tool count at request start.
+- **Stream completion log** — Logs tool_calls count, stop_reason, and text length after streaming.
+
+## [0.10.0] - 2026-05-29
+
+### Added
+- **Sinatra Namespace API** — complete API refactor using `sinatra-contrib` namespaces with thin route blocks. Enabled via `settings[:llm][:api][:use_namespaces] = true`.
+- **OpenAI-compatible endpoints (full surface):**
+  - `POST /v1/responses` — Responses API with typed SSE streaming (Codex CLI drop-in)
+  - `GET/DELETE /v1/responses/:id`, `POST /:id/cancel`, `GET /:id/input_items`
+  - `POST /v1/chat/completions` — streaming (`data: [DONE]`) and sync
+  - `GET/POST/DELETE /v1/chat/completions/:id`, `GET /v1/chat/completions/:id/messages`
+  - `GET /v1/models`, `GET /v1/models/:id` — with Anthropic format branching via `detect_client`
+  - `POST /v1/embeddings`, `POST /v1/completions` (legacy)
+  - `POST /v1/images/generations`, `/edits`, `/variations`
+  - `POST /v1/audio/transcriptions`, `/translations`, `/speech`
+  - `POST /v1/moderations`
+  - `POST/GET /v1/conversations`, CRUD + items
+  - `POST/GET /v1/batches`, cancel
+  - `POST/GET /v1/files`, content download, delete
+  - `POST /v1/uploads`, parts, cancel, complete
+  - `POST/GET /v1/vector_stores`, search, files, file batches
+- **Anthropic-compatible endpoints (full surface):**
+  - `POST /v1/messages` — Messages API with correct SSE event ordering (Claude Code drop-in)
+  - `POST /v1/messages/count_tokens` — token estimation
+  - `POST/GET/DELETE /v1/messages/batches` — full batch CRUD + JSONL results
+  - `GET /v1/models` — Anthropic format via detect_client header branching
+  - Anthropic files namespace (standalone deployment mode)
+- **Native namespace endpoints** — all `/api/llm/*` routes ported to namespace pattern
+- **SharedHelpers module** (`lib/legion/llm/api/shared_helpers.rb`) — extracted from `native/helpers.rb`, shared by both legacy and namespace routes
+- **TokenEstimation module** (`lib/legion/llm/token_estimation.rb`) — character-based token counting for `count_tokens` endpoint
+- **Client detection** — `detect_client(env)` determines OpenAI vs Anthropic client from headers (anthropic-version, x-api-key)
+- **Auth middleware** — now returns format-appropriate error shapes (OpenAI vs Anthropic) based on detected client
+- **VectorStore::Storage** — table DDL, cosine similarity, chunk_text utilities for vector store endpoints
+- **Codex CLI conformance tests** — verifies exact streaming event order for drop-in compatibility
+- **Claude Code conformance tests** — verifies exact SSE event order, tool use, system prompt handling
+
+### Changed
+- `sinatra-contrib` added as runtime dependency (>= 2.0)
+- `rack-test` added to development dependencies
+- API registration now conditional: `use_namespaces: true` uses `Namespaces::Registration`, `false` (default) uses legacy flat routes
+- All namespace inference routes go through full 18-step `Inference::Executor` pipeline (Gaia, RAG, metering, audit, escalation, RBAC, tools, etc.)
+
+### Fixed
+- Auth before filter returns Anthropic-shaped error for Anthropic clients (was always OpenAI-shaped)
+- Anthropic streaming event order: `message_start` now emitted before any `content_block_delta` events
+- `/v1/models` path conflict resolved — single handler with client detection branching
+
 ## [0.9.54] - 2026-05-29
 
 ### Fixed
@@ -13,6 +79,12 @@
 
 ## [0.9.52] - 2026-05-27
 
+### Added
+- API: `/v1/chat/completions` now has full pipeline feature parity with the native `/api/llm/inference` endpoint — routing, escalation, RAG context injection, Gaia advisory, knowledge capture, tool discovery, sticky runners, confidence scoring, metering, and debate all activate when the relevant fields are provided.
+- API: `/v1/chat/completions` accepts extended fields via request body (`conversation_id`, `provider`, `tier`, `instance`, `cwd`, `requested_tools`, `client_tool_passthrough`, `caller`) or via `X-Legion-*` headers (`X-Legion-Conversation-Id`, `X-Legion-Provider`, `X-Legion-Tier`, `X-Legion-Instance`, `X-Legion-Cwd`, `X-Legion-Client-Tool-Passthrough`). Headers take precedence for scalar values.
+- API: `/v1/chat/completions` now performs pre-pipeline Gaia ingest (mirrors native endpoint awareness).
+- API: `/v1/chat/completions` reasoning/thinking token streaming via `include_reasoning: true` (or `include_thinking: true`). Emits `reasoning_content` delta chunks in OpenAI format. Non-streaming responses include `reasoning_content` in the message body.
+
 ### Fixed
 - Discovery: `verify_embedding` now checks `model_available?` for Ollama instead of blindly returning true — prevents `can_embed?` from reporting true when the embedding model (e.g. `mxbai-embed-large`) hasn't been pulled on the local node
 - Discovery: `detect_embedding_from_registry` now calls `verify_embedding` before setting `@can_embed = true`, closing a gap where registry-declared capability metadata was trusted without verifying the model exists locally

data/Gemfile

@@ -30,6 +30,7 @@ group :test do
     gem provider_gem, path: provider_path if Dir.exist?(provider_path)
   end
 
+  gem 'rack-test', '~> 2.0'
   gem 'rake'
   gem 'rspec'
   gem 'rspec_junit_formatter'

data/bin/h200-setup-postreboot.sh

@@ -0,0 +1,266 @@
+#!/bin/bash
+set -euo pipefail
+
+# DGX/HGX H200 SXM 8-GPU Setup Script (Post-Reboot)
+# Run as root after reboot: su - then bash h200-setup-postreboot.sh
+# Requires: h200-setup-prereboot.sh completed + reboot done
+
+echo "=== Verifying root ==="
+if [ "$(id -u)" -ne 0 ]; then
+  echo "ERROR: Must run as root (su -)"
+  exit 1
+fi
+
+echo "=== Verifying H200 PCI BAR recovery settings ==="
+if ! grep -qw 'pci=realloc=off' /proc/cmdline; then
+  echo "ERROR: pci=realloc=off is not active. Refusing to continue because Linux may zero H200 BARs."
+  echo "Current cmdline: $(cat /proc/cmdline)"
+  echo "Expected grub setting: GRUB_CMDLINE_LINUX_DEFAULT=\"quiet pci=realloc=off\""
+  exit 1
+fi
+
+for bdf in 04 14 64 77 84 94 e9 f6; do
+  resource="/sys/bus/pci/devices/0000:${bdf}:00.0/resource"
+  if [ ! -r "$resource" ]; then
+    echo "ERROR: GPU 0000:${bdf}:00.0 is missing from sysfs."
+    exit 1
+  fi
+
+  if awk 'NR==1||NR==3||NR==5 { if ($1 == "0x0000000000000000") bad = 1 } END { exit bad }' "$resource"; then
+    :
+  else
+    echo "ERROR: GPU 0000:${bdf}:00.0 has an unassigned BAR."
+    awk 'NR==1||NR==3||NR==5{print}' "$resource"
+    exit 1
+  fi
+done
+
+echo "=== Verifying NVMe mounts ==="
+if ! mountpoint -q /data; then
+  echo "ERROR: /data is not mounted. Check fstab."
+  exit 1
+fi
+if ! mountpoint -q /srv/legion; then
+  echo "ERROR: /srv/legion is not mounted. Check fstab."
+  exit 1
+fi
+
+echo "=== Verifying NVIDIA driver ==="
+if ! timeout 60 nvidia-smi; then
+  echo "ERROR: nvidia-smi failed. Driver not loaded."
+  echo "Check: dmesg | grep -i nvidia"
+  echo "Try: apt install -y nvidia-open && reboot"
+  exit 1
+fi
+
+GPU_COUNT=$(timeout 60 nvidia-smi --query-gpu=name --format=csv,noheader | wc -l)
+echo "Detected $GPU_COUNT GPUs"
+if [ "$GPU_COUNT" -ne 8 ]; then
+  echo "ERROR: Expected 8 H200 GPUs, found $GPU_COUNT"
+  echo "Check: lspci | grep -i nvidia"
+  exit 1
+fi
+
+echo "=== Starting nvidia-fabricmanager (NVSwitch required) ==="
+systemctl start nvidia-fabricmanager
+if ! systemctl status nvidia-fabricmanager --no-pager -l; then
+  echo "ERROR: fabricmanager failed to start. NVSwitch won't work without it."
+  echo "Check: journalctl -u nvidia-fabricmanager"
+  exit 1
+fi
+
+echo "=== GPU persistence mode ==="
+nvidia-smi -pm 1
+
+echo "=== Verifying NVSwitch topology ==="
+timeout 60 nvidia-smi topo -m
+echo ""
+echo "All GPUs should show NV18 (NVSwitch full mesh) connections."
+echo ""
+timeout 60 nvidia-smi nvlink -s
+
+echo "=== CUDA library path ==="
+CUDA_PATH=$(find /usr/local -maxdepth 1 -name "cuda-12*" -type d | sort -V | tail -1)
+if [ -z "$CUDA_PATH" ]; then
+  echo "WARNING: CUDA toolkit directory not found under /usr/local"
+  CUDA_PATH="/usr/local/cuda-12.8"
+fi
+echo "$CUDA_PATH/lib64" > /etc/ld.so.conf.d/cuda.conf
+ldconfig
+ln -sf "$CUDA_PATH" /usr/local/cuda
+
+echo "=== CPU performance governor ==="
+for gov in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
+  echo performance > "$gov" 2>/dev/null || true
+done
+echo never > /sys/kernel/mm/transparent_hugepage/defrag
+echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
+
+echo "=== NUMA topology check ==="
+numactl --hardware
+echo ""
+echo "For optimal GPU-CPU affinity, vLLM will use all NUMA nodes."
+
+echo "=== Setting up Ollama ownership ==="
+chown -R ollama:ollama /usr/share/ollama/.ollama
+chown -R ollama:llm /srv/legion/ollama
+chmod -R 775 /srv/legion/ollama
+
+echo "=== Restarting Ollama ==="
+systemctl restart ollama
+sleep 2
+systemctl status ollama --no-pager
+
+echo "=== Installing vLLM ==="
+if [ ! -d "/data/vllm/env" ]; then
+  python3 -m venv /data/vllm/env
+fi
+
+mkdir -p /data/tmp /data/cache/pip
+source /data/vllm/env/bin/activate
+TMPDIR=/data/tmp PIP_CACHE_DIR=/data/cache/pip pip install vllm xxhash
+deactivate
+
+echo "=== Creating vLLM config ==="
+cat > /data/vllm/config.yaml << 'EOF'
+# Model (rsync from Apollo: rsync -avP root@10.11.164.93:/data/models/qwen3.6-27b/ /data/models/qwen3.6-27b/)
+model: /data/models/qwen3.6-27b
+dtype: bfloat16
+tensor-parallel-size: 8
+served-model-name: qwen3.6-27b
+
+# Context & Memory — H200 141GB per card, 1.1TB total
+max-model-len: 131072
+gpu-memory-utilization: 0.95
+enable-prefix-caching: true
+prefix-caching-hash-algo: xxhash
+
+# Scheduling & Batching — aggressive for 8x H200
+max-num-seqs: 128
+max-num-batched-tokens: 131072
+enable-chunked-prefill: true
+scheduling-policy: fcfs
+performance-mode: throughput
+async-scheduling: true
+
+# Model Loading
+load-format: auto
+safetensors-load-strategy: mmap
+
+# Streaming
+stream-interval: 1
+
+# Thinking/Reasoning
+reasoning-parser: deepseek_v3
+
+# Tool calling
+enable-auto-tool-choice: true
+tool-call-parser: hermes
+
+# Server
+host: 0.0.0.0
+port: 8000
+
+# Monitoring
+enable-server-load-tracking: true
+enable-prompt-tokens-details: true
+enable-log-requests: true
+kv-cache-metrics: true
+enable-mfu-metrics: true
+enable-logging-iteration-details: true
+EOF
+
+echo "=== Creating vLLM systemd service ==="
+cat > /etc/systemd/system/vllm.service << 'EOF'
+[Unit]
+Description=vLLM Inference Server (8x H200 SXM)
+After=network.target nvidia-persistenced.service nvidia-fabricmanager.service
+Wants=nvidia-persistenced.service nvidia-fabricmanager.service
+
+[Service]
+Type=simple
+User=vllm
+Group=llm
+Environment="PATH=/data/vllm/env/bin:/usr/local/bin:/usr/bin:/bin"
+Environment="TMPDIR=/data/tmp"
+Environment="TORCHINDUCTOR_CACHE_DIR=/data/cache/torchinductor"
+Environment="TRITON_CACHE_DIR=/data/cache/triton"
+Environment="HF_HOME=/data/cache/huggingface"
+Environment="LD_LIBRARY_PATH=/usr/local/cuda/lib64"
+ExecStart=/data/vllm/env/bin/python -m vllm.entrypoints.openai.api_server --config /data/vllm/config.yaml
+Restart=on-failure
+RestartSec=10
+LimitNOFILE=1048576
+LimitMEMLOCK=infinity
+StandardOutput=append:/srv/legion/logs/vllm/vllm.log
+StandardError=append:/srv/legion/logs/vllm/vllm-error.log
+
+[Install]
+WantedBy=multi-user.target
+EOF
+
+echo "=== Setting ownership ==="
+chown -R vllm:llm /data/vllm /data/models /data/cache /data/tmp /srv/legion/logs/vllm
+
+systemctl daemon-reload
+systemctl enable vllm
+
+echo "=== Checking for model files ==="
+if [ -d "/data/models/qwen3.6-27b" ] && [ "$(find /data/models/qwen3.6-27b -name '*.safetensors' 2>/dev/null | wc -l)" -gt 0 ]; then
+  echo "Model found at /data/models/qwen3.6-27b"
+  echo "=== Starting vLLM ==="
+  rm -f /dev/shm/psm_* /dev/shm/sem.mp-* /dev/shm/VLLM_*
+  systemctl start vllm
+  echo ""
+  echo "vLLM is starting. First startup compiles CUDA graphs (~5-8 min on H200)."
+  echo "Subsequent restarts load from cache (~20 sec)."
+  echo "Monitor: journalctl -u vllm -f"
+  echo "Logs: tail -f /srv/legion/logs/vllm/vllm.log"
+else
+  echo ""
+  echo "WARNING: No model found at /data/models/qwen3.6-27b"
+  echo ""
+  echo "Rsync from Apollo V100 node:"
+  echo "  rsync -aHAXx --numeric-ids --info=progress2 --partial --append-verify root@10.11.164.93:/data/models/qwen3.6-27b/ /data/models/qwen3.6-27b/"
+  echo "  chown -R vllm:llm /data/models"
+  echo "  systemctl start vllm"
+fi
+
+echo ""
+echo "==========================================="
+echo "  POST-REBOOT SETUP COMPLETE"
+echo "==========================================="
+echo ""
+echo "  Hardware: 8x H200 SXM 141GB, NVSwitch full-mesh"
+echo "  VRAM:     1.128 TB total"
+echo "  RAM:      2.2 TiB"
+echo ""
+echo "  Services:"
+echo "    Ollama:          http://0.0.0.0:11434 (embeddings)"
+echo "    vLLM:            http://0.0.0.0:8000  (inference, TP=8, 131K context)"
+echo "    fabricmanager:   active (NVSwitch)"
+echo ""
+echo "  vLLM config highlights:"
+echo "    - TP=8 (all GPUs via NVSwitch, no DP needed for 27B)"
+echo "    - 131K context (H200 has room to spare with 27B)"
+echo "    - max-num-seqs=128, batched-tokens=131K"
+echo "    - bfloat16 (native H200 dtype)"
+echo ""
+echo "  Test:"
+echo "    curl http://localhost:8000/v1/models"
+echo "    curl http://localhost:8000/v1/chat/completions \\"
+echo "      -H 'Content-Type: application/json' \\"
+echo "      -d '{\"model\":\"qwen3.6-27b\",\"messages\":[{\"role\":\"user\",\"content\":\"hello\"}],\"max_tokens\":50,\"chat_template_kwargs\":{\"enable_thinking\":false}}'"
+echo ""
+echo "  For 235B later:"
+echo "    - Update /data/vllm/config.yaml: model path, max-model-len, and batching limits"
+echo "    - 235B at bf16 = ~470GB, fits easily in 1.1TB with room for KV cache"
+echo ""
+echo "  Useful commands:"
+echo "    nvtop                                    # GPU monitoring"
+echo "    nvidia-smi topo -m                       # NVSwitch topology"
+echo "    tail -f /srv/legion/logs/vllm/vllm.log   # vLLM logs"
+echo "    systemctl status nvidia-fabricmanager     # NVSwitch status"
+echo "    systemctl status vllm                    # vLLM status"
+echo "    systemctl status ollama                  # Ollama status"
+echo "==========================================="

data/bin/h200-setup-prereboot.sh

@@ -0,0 +1,228 @@
+#!/bin/bash
+set -euo pipefail
+
+# DGX/HGX H200 SXM 8-GPU Setup Script (Pre-Reboot)
+# Hardware: 8x H200 SXM 141GB, 4x NVSwitch, 2x AMD EPYC 9535, 2.2TiB RAM
+# Host: mn011-6labhz1-h200-0001
+# OS: Debian 13 (trixie), kernel 6.12
+# Run as root: su - then bash h200-setup-prereboot.sh
+# After completion: reboot, then run h200-setup-postreboot.sh
+#
+# NOTE: Corporate firewall does TLS inspection. All external HTTPS downloads
+# use --no-check-certificate / -k and apt repos use trusted=yes to bypass
+# certificate verification failures from the MITM proxy.
+
+echo "=== Verifying root ==="
+if [ "$(id -u)" -ne 0 ]; then
+  echo "ERROR: Must run as root (su -)"
+  exit 1
+fi
+
+if [ "${H200_ALLOW_DESTRUCTIVE_PREREBOOT:-0}" != "1" ]; then
+  echo "ERROR: This legacy prereboot script formats NVMe devices and is no longer the supported rebuild path."
+  echo "Use scripts/h200/h200-rebuild.sh for full automation, or set H200_ALLOW_DESTRUCTIVE_PREREBOOT=1 only if you intend to wipe the scripted NVMe devices."
+  exit 1
+fi
+
+echo "=== Fixing apt sources (remove cdrom if present) ==="
+sed -i '/cdrom/d' /etc/apt/sources.list 2>/dev/null || true
+
+echo "=== Installing base packages ==="
+apt update
+apt install -y \
+  sudo curl wget gnupg2 lsb-release ca-certificates \
+  build-essential linux-headers-$(uname -r) \
+  git vim htop tmux nvtop \
+  numactl hwloc \
+  pciutils dkms \
+  apt-transport-https \
+  rsync parted jq iperf3 \
+  python3-pip python3-venv \
+  mdadm lvm2
+
+echo "=== Setting up sudo for miverso2 ==="
+usermod -aG sudo miverso2
+
+echo "=== Bypassing TLS inspection for external repos ==="
+cat > /etc/apt/apt.conf.d/99bypass-tls-inspection << 'EOF'
+Acquire::https::developer.download.nvidia.com::Verify-Peer "false";
+Acquire::https::nvidia.github.io::Verify-Peer "false";
+Acquire::https::apt.releases.hashicorp.com::Verify-Peer "false";
+EOF
+
+echo "=== Setting up NVIDIA CUDA repo (debian12 — compatible with trixie) ==="
+wget -qO /usr/share/keyrings/cuda-archive-keyring.gpg --no-check-certificate \
+  https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/cuda-archive-keyring.gpg
+echo "deb [trusted=yes] https://developer.download.nvidia.com/compute/cuda/repos/debian12/x86_64/ /" \
+  > /etc/apt/sources.list.d/cuda-debian12.list
+apt update
+
+echo "=== Installing NVIDIA open kernel module + driver ==="
+apt install -y nvidia-kernel-open-dkms
+
+echo "=== Installing CUDA toolkit 12.8 ==="
+apt install -y cuda-toolkit-12-8
+
+echo "=== Installing nvidia-fabricmanager (REQUIRED for NVSwitch) ==="
+apt install -y nvidia-fabricmanager
+systemctl enable nvidia-fabricmanager
+
+echo "=== Installing Docker ==="
+apt install -y docker.io
+systemctl enable docker
+
+echo "=== Installing NVIDIA Container Toolkit ==="
+curl -fsSLk https://nvidia.github.io/libnvidia-container/gpgkey | \
+  gpg --dearmor > /etc/apt/trusted.gpg.d/nvidia-container-toolkit.gpg
+echo "deb [trusted=yes] https://nvidia.github.io/libnvidia-container/stable/deb/amd64 /" \
+  > /etc/apt/sources.list.d/nvidia-container-toolkit.list
+apt update
+apt install -y nvidia-container-toolkit
+nvidia-ctk runtime configure --runtime=docker
+
+echo "=== Installing HashiCorp Enterprise ==="
+wget -qO- --no-check-certificate https://apt.releases.hashicorp.com/gpg | \
+  gpg --dearmor > /etc/apt/trusted.gpg.d/hashicorp.gpg
+echo "deb [signed-by=/etc/apt/trusted.gpg.d/hashicorp.gpg trusted=yes] https://apt.releases.hashicorp.com bookworm main" \
+  > /etc/apt/sources.list.d/hashicorp.list
+apt update
+apt install -y nomad-enterprise consul-enterprise
+systemctl enable nomad
+systemctl enable consul
+
+echo "=== Installing Ollama ==="
+curl -fsSLk https://ollama.com/install.sh | sh
+
+echo "=== Setting up NVMe drives ==="
+echo "  nvme1n1 (7TB) -> /data (models, vllm cache, vllm env)"
+echo "  nvme2n1 (7TB) -> /srv/legion (ollama, docker, nomad, consul, logs)"
+echo "  nvme3n1 (7TB) -> /reserve (future use)"
+
+echo "=== Formatting NVMe drives ==="
+mkfs.ext4 -L data -m 0 /dev/nvme1n1
+mkfs.ext4 -L srv -m 0 /dev/nvme2n1
+mkfs.ext4 -L reserve -m 0 /dev/nvme3n1
+
+echo "=== Creating mount points ==="
+mkdir -p /data /srv/legion /reserve
+
+echo "=== Adding fstab entries ==="
+cat >> /etc/fstab << 'EOF'
+/dev/nvme1n1 /data ext4 defaults,noatime,discard 0 2
+/dev/nvme2n1 /srv/legion ext4 defaults,noatime,discard 0 2
+/dev/nvme3n1 /reserve ext4 defaults,noatime,discard 0 2
+EOF
+
+echo "=== Mounting all ==="
+mount -a
+
+echo "=== Creating /data directory structure ==="
+mkdir -p /data/{models,cache,vllm,tmp}
+mkdir -p /data/cache/{pip,triton,torchinductor,huggingface}
+
+echo "=== Creating /srv/legion directory structure ==="
+mkdir -p /srv/legion/{ollama,docker,nomad,consul,rabbitmq}
+mkdir -p /srv/legion/logs/{vllm,ollama,nomad}
+
+echo "=== Moving docker storage to NVMe ==="
+systemctl stop docker 2>/dev/null || true
+if [ -d "/var/lib/docker" ] && [ ! -L "/var/lib/docker" ]; then
+  rsync -a /var/lib/docker/ /srv/legion/docker/ 2>/dev/null || true
+  rm -rf /var/lib/docker
+  ln -sf /srv/legion/docker /var/lib/docker
+fi
+
+echo "=== Setting up Ollama storage on NVMe ==="
+mkdir -p /usr/share/ollama/.ollama
+echo "/srv/legion/ollama /usr/share/ollama/.ollama none bind 0 0" >> /etc/fstab
+mount -a
+
+echo "=== Configuring Ollama ==="
+mkdir -p /etc/systemd/system/ollama.service.d
+cat > /etc/systemd/system/ollama.service.d/override.conf << 'OLLEOF'
+[Service]
+Environment="OLLAMA_HOST=0.0.0.0:11434"
+OLLEOF
+
+echo "=== Creating users and groups ==="
+groupadd -f llm
+useradd -r -s /bin/false -d /data/vllm vllm 2>/dev/null || true
+usermod -aG llm miverso2
+usermod -aG llm ollama 2>/dev/null || true
+usermod -aG llm vllm
+usermod -aG video vllm
+usermod -aG render vllm
+
+echo "=== Setting ownership ==="
+chown -R ollama:ollama /usr/share/ollama/.ollama
+chown -R ollama:llm /srv/legion/ollama
+chmod -R 775 /srv/legion/ollama
+chown -R vllm:llm /data/vllm /data/models /data/cache /data/tmp
+chown -R vllm:llm /srv/legion/logs/vllm
+
+echo "=== Network tuning (EPYC + ConnectX-7) ==="
+cat > /etc/sysctl.d/99-h200.conf << 'EOF'
+net.ipv4.tcp_window_scaling = 1
+net.core.rmem_max = 67108864
+net.core.wmem_max = 67108864
+net.ipv4.tcp_rmem = 4096 87380 67108864
+net.ipv4.tcp_wmem = 4096 65536 67108864
+net.ipv4.tcp_congestion_control = bbr
+net.core.somaxconn = 3240000
+net.core.busy_poll = 50
+net.core.busy_read = 50
+net.core.netdev_max_backlog = 65536
+net.ipv4.tcp_max_syn_backlog = 3240000
+net.ipv4.ip_local_port_range = 10000 65535
+net.ipv4.tcp_fin_timeout = 15
+net.ipv4.tcp_max_tw_buckets = 65536
+kernel.numa_balancing = 0
+vm.nr_hugepages = 4096
+vm.swappiness = 1
+EOF
+sysctl -p /etc/sysctl.d/99-h200.conf
+
+echo "=== File descriptor limits ==="
+cat >> /etc/security/limits.conf << 'EOF'
+vllm soft nofile 1048576
+vllm hard nofile 1048576
+ollama soft nofile 65536
+ollama hard nofile 65536
+* soft memlock unlimited
+* hard memlock unlimited
+EOF
+
+echo "=== THP + CPU governor persistence ==="
+cat > /etc/rc.local << 'RCEOF'
+#!/bin/bash
+echo never > /sys/kernel/mm/transparent_hugepage/defrag
+echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
+for gov in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor; do
+  echo performance > $gov 2>/dev/null
+done
+exit 0
+RCEOF
+chmod +x /etc/rc.local
+
+echo "=== Applying THP settings now ==="
+echo never > /sys/kernel/mm/transparent_hugepage/defrag
+echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
+
+systemctl daemon-reload
+
+echo ""
+echo "==========================================="
+echo "  PRE-REBOOT SETUP COMPLETE"
+echo "==========================================="
+echo ""
+echo "  Hardware: 8x H200 SXM 141GB + 4x NVSwitch"
+echo "  NVMe:     nvme1n1 -> /data (models/vllm)"
+echo "            nvme2n1 -> /srv/legion (services/logs)"
+echo "            nvme3n1 -> /reserve (future)"
+echo ""
+echo "  Next steps:"
+echo "  1. reboot"
+echo "  2. After reboot, run: bash h200-setup-postreboot.sh"
+echo ""
+echo "  The reboot is required for the NVIDIA driver to load."
+echo "==========================================="