better-dsh 0.2.5 → 0.3.0-c

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/bundled/mlops/mlops-inference-llama-cpp.md

@@ -1,267 +0,0 @@
----
-title: "Llama Cpp — llama.cpp local GGUF inference + HF Hub model discovery"
-sidebar_label: "Llama Cpp"
-description: "llama.cpp local GGUF inference + HF Hub model discovery"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Llama Cpp
-
-llama.cpp local GGUF inference + HF Hub model discovery.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Bundled (installed by default) |
-| Path | `skills/mlops/inference/llama-cpp` |
-| Version | `2.1.2` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `llama-cpp-python>=0.2.0` |
-| Platforms | linux, macos, windows |
-| Tags | `llama.cpp`, `GGUF`, `Quantization`, `Hugging Face Hub`, `CPU Inference`, `Apple Silicon`, `Edge Deployment`, `AMD GPUs`, `Intel GPUs`, `NVIDIA`, `URL-first` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# llama.cpp + GGUF
-
-Use this skill for local GGUF inference, quant selection, or Hugging Face repo discovery for llama.cpp.
-
-## When to use
-
-- Run local models on CPU, Apple Silicon, CUDA, ROCm, or Intel GPUs
-- Find the right GGUF for a specific Hugging Face repo
-- Build a `llama-server` or `llama-cli` command from the Hub
-- Search the Hub for models that already support llama.cpp
-- Enumerate available `.gguf` files and sizes for a repo
-- Decide between Q4/Q5/Q6/IQ variants for the user's RAM or VRAM
-
-## Model Discovery workflow
-
-Prefer URL workflows before asking for `hf`, Python, or custom scripts.
-
-1. Search for candidate repos on the Hub:
-   - Base: `https://huggingface.co/models?apps=llama.cpp&sort=trending`
-   - Add `search=<term>` for a model family
-   - Add `num_parameters=min:0,max:24B` or similar when the user has size constraints
-2. Open the repo with the llama.cpp local-app view:
-   - `https://huggingface.co/<repo>?local-app=llama.cpp`
-3. Treat the local-app snippet as the source of truth when it is visible:
-   - copy the exact `llama-server` or `llama-cli` command
-   - report the recommended quant exactly as HF shows it
-4. Read the same `?local-app=llama.cpp` URL as page text or HTML and extract the section under `Hardware compatibility`:
-   - prefer its exact quant labels and sizes over generic tables
-   - keep repo-specific labels such as `UD-Q4_K_M` or `IQ4_NL_XL`
-   - if that section is not visible in the fetched page source, say so and fall back to the tree API plus generic quant guidance
-5. Query the tree API to confirm what actually exists:
-   - `https://huggingface.co/api/models/<repo>/tree/main?recursive=true`
-   - keep entries where `type` is `file` and `path` ends with `.gguf`
-   - use `path` and `size` as the source of truth for filenames and byte sizes
-   - separate quantized checkpoints from `mmproj-*.gguf` projector files and `BF16/` shard files
-   - use `https://huggingface.co/<repo>/tree/main` only as a human fallback
-6. If the local-app snippet is not text-visible, reconstruct the command from the repo plus the chosen quant:
-   - shorthand quant selection: `llama-server -hf <repo>:<QUANT>`
-   - exact-file fallback: `llama-server --hf-repo <repo> --hf-file <filename.gguf>`
-7. Only suggest conversion from Transformers weights if the repo does not already expose GGUF files.
-
-## Quick start
-
-### Install llama.cpp
-
-```bash
-# macOS / Linux (simplest)
-brew install llama.cpp
-```
-
-```bash
-winget install llama.cpp
-```
-
-```bash
-git clone https://github.com/ggml-org/llama.cpp
-cd llama.cpp
-cmake -B build
-cmake --build build --config Release
-```
-
-### Run directly from the Hugging Face Hub
-
-```bash
-llama-cli -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
-```
-
-```bash
-llama-server -hf bartowski/Llama-3.2-3B-Instruct-GGUF:Q8_0
-```
-
-### Run an exact GGUF file from the Hub
-
-Use this when the tree API shows custom file naming or the exact HF snippet is missing.
-
-```bash
-llama-server \
-    --hf-repo microsoft/Phi-3-mini-4k-instruct-gguf \
-    --hf-file Phi-3-mini-4k-instruct-q4.gguf \
-    -c 4096
-```
-
-### OpenAI-compatible server check
-
-```bash
-curl http://localhost:8080/v1/chat/completions \
-  -H "Content-Type: application/json" \
-  -d '{
-    "messages": [
-      {"role": "user", "content": "Write a limerick about Python exceptions"}
-    ]
-  }'
-```
-
-## Python bindings (llama-cpp-python)
-
-`pip install llama-cpp-python` (CUDA: `CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python --force-reinstall --no-cache-dir`; Metal: `CMAKE_ARGS="-DGGML_METAL=on" ...`).
-
-### Basic generation
-
-```python
-from llama_cpp import Llama
-
-llm = Llama(
-    model_path="./model-q4_k_m.gguf",
-    n_ctx=4096,
-    n_gpu_layers=35,     # 0 for CPU, 99 to offload everything
-    n_threads=8,
-)
-
-out = llm("What is machine learning?", max_tokens=256, temperature=0.7)
-print(out["choices"][0]["text"])
-```
-
-### Chat + streaming
-
-```python
-llm = Llama(
-    model_path="./model-q4_k_m.gguf",
-    n_ctx=4096,
-    n_gpu_layers=35,
-    chat_format="llama-3",   # or "chatml", "mistral", etc.
-)
-
-resp = llm.create_chat_completion(
-    messages=[
-        {"role": "system", "content": "You are a helpful assistant."},
-        {"role": "user", "content": "What is Python?"},
-    ],
-    max_tokens=256,
-)
-print(resp["choices"][0]["message"]["content"])
-
-# Streaming
-for chunk in llm("Explain quantum computing:", max_tokens=256, stream=True):
-    print(chunk["choices"][0]["text"], end="", flush=True)
-```
-
-### Embeddings
-
-```python
-llm = Llama(model_path="./model-q4_k_m.gguf", embedding=True, n_gpu_layers=35)
-vec = llm.embed("This is a test sentence.")
-print(f"Embedding dimension: {len(vec)}")
-```
-
-You can also load a GGUF straight from the Hub:
-
-```python
-llm = Llama.from_pretrained(
-    repo_id="bartowski/Llama-3.2-3B-Instruct-GGUF",
-    filename="*Q4_K_M.gguf",
-    n_gpu_layers=35,
-)
-```
-
-## Choosing a quant
-
-Use the Hub page first, generic heuristics second.
-
-- Prefer the exact quant that HF marks as compatible for the user's hardware profile.
-- For general chat, start with `Q4_K_M`.
-- For code or technical work, prefer `Q5_K_M` or `Q6_K` if memory allows.
-- For very tight RAM budgets, consider `Q3_K_M`, `IQ` variants, or `Q2` variants only if the user explicitly prioritizes fit over quality.
-- For multimodal repos, mention `mmproj-*.gguf` separately. The projector is not the main model file.
-- Do not normalize repo-native labels. If the page says `UD-Q4_K_M`, report `UD-Q4_K_M`.
-
-## Extracting available GGUFs from a repo
-
-When the user asks what GGUFs exist, return:
-
-- filename
-- file size
-- quant label
-- whether it is a main model or an auxiliary projector
-
-Ignore unless requested:
-
-- README
-- BF16 shard files
-- imatrix blobs or calibration artifacts
-
-Use the tree API for this step:
-
-- `https://huggingface.co/api/models/<repo>/tree/main?recursive=true`
-
-For a repo like `unsloth/Qwen3.6-35B-A3B-GGUF`, the local-app page can show quant chips such as `UD-Q4_K_M`, `UD-Q5_K_M`, `UD-Q6_K`, and `Q8_0`, while the tree API exposes exact file paths such as `Qwen3.6-35B-A3B-UD-Q4_K_M.gguf` and `Qwen3.6-35B-A3B-Q8_0.gguf` with byte sizes. Use the tree API to turn a quant label into an exact filename.
-
-## Search patterns
-
-Use these URL shapes directly:
-
-```text
-https://huggingface.co/models?apps=llama.cpp&sort=trending
-https://huggingface.co/models?search=<term>&apps=llama.cpp&sort=trending
-https://huggingface.co/models?search=<term>&apps=llama.cpp&num_parameters=min:0,max:24B&sort=trending
-https://huggingface.co/<repo>?local-app=llama.cpp
-https://huggingface.co/api/models/<repo>/tree/main?recursive=true
-https://huggingface.co/<repo>/tree/main
-```
-
-## Output format
-
-When answering discovery requests, prefer a compact structured result like:
-
-```text
-Repo: <repo>
-Recommended quant from HF: <label> (<size>)
-llama-server: <command>
-Other GGUFs:
-- <filename> - <size>
-- <filename> - <size>
-Source URLs:
-- <local-app URL>
-- <tree API URL>
-```
-
-## References
-
-- **[hub-discovery.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/llama-cpp/references/hub-discovery.md)** - URL-only Hugging Face workflows, search patterns, GGUF extraction, and command reconstruction
-- **[advanced-usage.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/llama-cpp/references/advanced-usage.md)** — speculative decoding, batched inference, grammar-constrained generation, LoRA, multi-GPU, custom builds, benchmark scripts
-- **[quantization.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/llama-cpp/references/quantization.md)** — quant quality tradeoffs, when to use Q4/Q5/Q6/IQ, model size scaling, imatrix
-- **[server.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/llama-cpp/references/server.md)** — direct-from-Hub server launch, OpenAI API endpoints, Docker deployment, NGINX load balancing, monitoring
-- **[optimization.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/llama-cpp/references/optimization.md)** — CPU threading, BLAS, GPU offload heuristics, batch tuning, benchmarks
-- **[troubleshooting.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/llama-cpp/references/troubleshooting.md)** — install/convert/quantize/inference/server issues, Apple Silicon, debugging
-
-## Resources
-
-- **GitHub**: https://github.com/ggml-org/llama.cpp
-- **Hugging Face GGUF + llama.cpp docs**: https://huggingface.co/docs/hub/gguf-llamacpp
-- **Hugging Face Local Apps docs**: https://huggingface.co/docs/hub/main/local-apps
-- **Hugging Face Local Agents docs**: https://huggingface.co/docs/hub/agents-local
-- **Example local-app page**: https://huggingface.co/unsloth/Qwen3.6-35B-A3B-GGUF?local-app=llama.cpp
-- **Example tree API**: https://huggingface.co/api/models/unsloth/Qwen3.6-35B-A3B-GGUF/tree/main?recursive=true
-- **Example llama.cpp search**: https://huggingface.co/models?num_parameters=min:0,max:24B&apps=llama.cpp&sort=trending
-- **License**: MIT

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/bundled/mlops/mlops-inference-serving-llms-vllm.md

@@ -1,387 +0,0 @@
----
-title: "Serving Llms Vllm — vLLM: high-throughput LLM serving, OpenAI API, quantization"
-sidebar_label: "Serving Llms Vllm"
-description: "vLLM: high-throughput LLM serving, OpenAI API, quantization"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Serving Llms Vllm
-
-vLLM: high-throughput LLM serving, OpenAI API, quantization.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Bundled (installed by default) |
-| Path | `skills/mlops/inference/serving-llms-vllm` |
-| Version | `1.0.1` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `vllm`, `torch`, `transformers` |
-| Platforms | linux, macos |
-| Tags | `vLLM`, `Inference Serving`, `PagedAttention`, `Continuous Batching`, `High Throughput`, `Production`, `OpenAI API`, `Quantization`, `Tensor Parallelism` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# vLLM - High-Performance LLM Serving
-
-## When to use
-
-Use when deploying production LLM APIs, optimizing inference latency/throughput, or serving models with limited GPU memory. Supports OpenAI-compatible endpoints, quantization (GPTQ/AWQ/FP8), and tensor parallelism.
-
-## Quick start
-
-vLLM achieves 24x higher throughput than standard transformers through PagedAttention (block-based KV cache) and continuous batching (mixing prefill/decode requests).
-
-**Installation**:
-```bash
-pip install vllm
-```
-
-**Basic offline inference**:
-```python
-from vllm import LLM, SamplingParams
-
-llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct")
-sampling = SamplingParams(temperature=0.7, max_tokens=256)
-
-outputs = llm.generate(["Explain quantum computing"], sampling)
-print(outputs[0].outputs[0].text)
-```
-
-**OpenAI-compatible server**:
-```bash
-vllm serve meta-llama/Meta-Llama-3-8B-Instruct
-
-# Query with OpenAI SDK
-python -c "
-from openai import OpenAI
-client = OpenAI(base_url='http://localhost:8000/v1', api_key='EMPTY')
-print(client.chat.completions.create(
-    model='meta-llama/Meta-Llama-3-8B-Instruct',
-    messages=[{'role': 'user', 'content': 'Hello!'}]
-).choices[0].message.content)
-"
-```
-
-## Common workflows
-
-### Workflow 1: Production API deployment
-
-Copy this checklist and track progress:
-
-```
-Deployment Progress:
-- [ ] Step 1: Configure server settings
-- [ ] Step 2: Test with limited traffic
-- [ ] Step 3: Enable monitoring
-- [ ] Step 4: Deploy to production
-- [ ] Step 5: Verify performance metrics
-```
-
-**Step 1: Configure server settings**
-
-Choose configuration based on your model size:
-
-```bash
-# For 7B-13B models on single GPU
-vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
-  --gpu-memory-utilization 0.9 \
-  --max-model-len 8192 \
-  --port 8000
-
-# For 30B-70B models with tensor parallelism
-vllm serve meta-llama/Meta-Llama-3-70B-Instruct \
-  --tensor-parallel-size 4 \
-  --gpu-memory-utilization 0.9 \
-  --quantization awq \
-  --port 8000
-
-# For production with caching (Prometheus metrics are exposed
-# automatically at /metrics on the API port)
-vllm serve meta-llama/Meta-Llama-3-8B-Instruct \
-  --gpu-memory-utilization 0.9 \
-  --enable-prefix-caching \
-  --port 8000 \
-  --host 0.0.0.0
-```
-
-**Step 2: Test with limited traffic**
-
-Run load test before production:
-
-```bash
-# Install load testing tool
-pip install locust
-
-# Create test_load.py with sample requests
-# Run: locust -f test_load.py --host http://localhost:8000
-```
-
-Verify TTFT (time to first token) < 500ms and throughput > 100 req/sec.
-
-**Step 3: Enable monitoring**
-
-vLLM exposes Prometheus metrics at `/metrics` on the API port (default 8000):
-
-```bash
-curl http://localhost:8000/metrics | grep vllm
-```
-
-Key metrics to monitor:
-- `vllm:time_to_first_token_seconds` - Latency
-- `vllm:num_requests_running` - Active requests
-- `vllm:gpu_cache_usage_perc` - KV cache utilization
-
-**Step 4: Deploy to production**
-
-Use Docker for consistent deployment:
-
-```bash
-# Run vLLM in Docker
-docker run --gpus all -p 8000:8000 \
-  vllm/vllm-openai:latest \
-  --model meta-llama/Meta-Llama-3-8B-Instruct \
-  --gpu-memory-utilization 0.9 \
-  --enable-prefix-caching
-```
-
-**Step 5: Verify performance metrics**
-
-Check that deployment meets targets:
-- TTFT < 500ms (for short prompts)
-- Throughput > target req/sec
-- GPU utilization > 80%
-- No OOM errors in logs
-
-### Workflow 2: Offline batch inference
-
-For processing large datasets without server overhead.
-
-Copy this checklist:
-
-```
-Batch Processing:
-- [ ] Step 1: Prepare input data
-- [ ] Step 2: Configure LLM engine
-- [ ] Step 3: Run batch inference
-- [ ] Step 4: Process results
-```
-
-**Step 1: Prepare input data**
-
-```python
-# Load prompts from file
-prompts = []
-with open("prompts.txt") as f:
-    prompts = [line.strip() for line in f]
-
-print(f"Loaded {len(prompts)} prompts")
-```
-
-**Step 2: Configure LLM engine**
-
-```python
-from vllm import LLM, SamplingParams
-
-llm = LLM(
-    model="meta-llama/Meta-Llama-3-8B-Instruct",
-    tensor_parallel_size=2,  # Use 2 GPUs
-    gpu_memory_utilization=0.9,
-    max_model_len=4096
-)
-
-sampling = SamplingParams(
-    temperature=0.7,
-    top_p=0.95,
-    max_tokens=512,
-    stop=["</s>", "\n\n"]
-)
-```
-
-**Step 3: Run batch inference**
-
-vLLM automatically batches requests for efficiency:
-
-```python
-# Process all prompts in one call
-outputs = llm.generate(prompts, sampling)
-
-# vLLM handles batching internally
-# No need to manually chunk prompts
-```
-
-**Step 4: Process results**
-
-```python
-# Extract generated text
-results = []
-for output in outputs:
-    prompt = output.prompt
-    generated = output.outputs[0].text
-    results.append({
-        "prompt": prompt,
-        "generated": generated,
-        "tokens": len(output.outputs[0].token_ids)
-    })
-
-# Save to file
-import json
-with open("results.jsonl", "w") as f:
-    for result in results:
-        f.write(json.dumps(result) + "\n")
-
-print(f"Processed {len(results)} prompts")
-```
-
-### Workflow 3: Quantized model serving
-
-Fit large models in limited GPU memory.
-
-```
-Quantization Setup:
-- [ ] Step 1: Choose quantization method
-- [ ] Step 2: Find or create quantized model
-- [ ] Step 3: Launch with quantization flag
-- [ ] Step 4: Verify accuracy
-```
-
-**Step 1: Choose quantization method**
-
-- **AWQ**: Best for 70B models, minimal accuracy loss
-- **GPTQ**: Wide model support, good compression
-- **FP8**: Fastest on H100 GPUs
-
-**Step 2: Find or create quantized model**
-
-Use pre-quantized models from HuggingFace:
-
-```bash
-# Search for AWQ models
-# Example: TheBloke/Llama-2-70B-AWQ
-```
-
-**Step 3: Launch with quantization flag**
-
-```bash
-# Using pre-quantized model
-vllm serve TheBloke/Llama-2-70B-AWQ \
-  --quantization awq \
-  --tensor-parallel-size 1 \
-  --gpu-memory-utilization 0.95
-
-# Results: 70B model in ~40GB VRAM
-```
-
-**Step 4: Verify accuracy**
-
-Test outputs match expected quality:
-
-```python
-# Compare quantized vs non-quantized responses
-# Verify task-specific performance unchanged
-```
-
-## When to use vs alternatives
-
-**Use vLLM when:**
-- Deploying production LLM APIs (100+ req/sec)
-- Serving OpenAI-compatible endpoints
-- Limited GPU memory but need large models
-- Multi-user applications (chatbots, assistants)
-- Need low latency with high throughput
-
-**Use alternatives instead:**
-- **llama.cpp**: CPU/edge inference, single-user
-- **HuggingFace transformers**: Research, prototyping, one-off generation
-- **TensorRT-LLM**: NVIDIA-only, need absolute maximum performance
-- **Text-Generation-Inference**: Already in HuggingFace ecosystem
-
-## Common issues
-
-**Issue: Out of memory during model loading**
-
-Reduce memory usage:
-```bash
-vllm serve MODEL \
-  --gpu-memory-utilization 0.7 \
-  --max-model-len 4096
-```
-
-Or use quantization:
-```bash
-vllm serve MODEL --quantization awq
-```
-
-**Issue: Slow first token (TTFT > 1 second)**
-
-Enable prefix caching for repeated prompts:
-```bash
-vllm serve MODEL --enable-prefix-caching
-```
-
-For long prompts, enable chunked prefill:
-```bash
-vllm serve MODEL --enable-chunked-prefill
-```
-
-**Issue: Model not found error**
-
-Use `--trust-remote-code` for custom models:
-```bash
-vllm serve MODEL --trust-remote-code
-```
-
-**Issue: Low throughput (&lt;50 req/sec)**
-
-Increase concurrent sequences:
-```bash
-vllm serve MODEL --max-num-seqs 512
-```
-
-Check GPU utilization with `nvidia-smi` - should be >80%.
-
-**Issue: Inference slower than expected**
-
-Verify tensor parallelism uses power of 2 GPUs:
-```bash
-vllm serve MODEL --tensor-parallel-size 4  # Not 3
-```
-
-Enable speculative decoding for faster generation (pass config as JSON;
-`--speculative-model` was removed in favor of `--speculative-config`):
-```bash
-vllm serve MODEL \
-  --speculative-config '{"model": "DRAFT_MODEL", "num_speculative_tokens": 5, "method": "draft_model"}'
-```
-
-## Advanced topics
-
-**Server deployment patterns**: See [references/server-deployment.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/serving-llms-vllm/references/server-deployment.md) for Docker, Kubernetes, and load balancing configurations.
-
-**Performance optimization**: See [references/optimization.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/serving-llms-vllm/references/optimization.md) for PagedAttention tuning, continuous batching details, and benchmark results.
-
-**Quantization guide**: See [references/quantization.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/serving-llms-vllm/references/quantization.md) for AWQ/GPTQ/FP8 setup, model preparation, and accuracy comparisons.
-
-**Troubleshooting**: See [references/troubleshooting.md](https://github.com/NousResearch/hermes-agent/blob/main/skills/mlops/inference/serving-llms-vllm/references/troubleshooting.md) for detailed error messages, debugging steps, and performance diagnostics.
-
-## Hardware requirements
-
-- **Small models (7B-13B)**: 1x A10 (24GB) or A100 (40GB)
-- **Medium models (30B-40B)**: 2x A100 (40GB) with tensor parallelism
-- **Large models (70B+)**: 4x A100 (40GB) or 2x A100 (80GB), use AWQ/GPTQ
-
-Supported platforms: NVIDIA (primary), AMD ROCm, Intel GPUs, TPUs
-
-## Resources
-
-- Official docs: https://docs.vllm.ai
-- GitHub: https://github.com/vllm-project/vllm
-- Paper: "Efficient Memory Management for Large Language Model Serving with PagedAttention" (SOSP 2023)
-- Community: https://discuss.vllm.ai