better-dsh 0.2.5-c → 0.2.5-e

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-llava.md

@@ -1,323 +0,0 @@
----
-title: "Llava — Vision-language chat: VQA, captioning, image dialogue"
-sidebar_label: "Llava"
-description: "Vision-language chat: VQA, captioning, image dialogue"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Llava
-
-Vision-language chat: VQA, captioning, image dialogue.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/llava` |
-| Path | `optional-skills/mlops\llava` |
-| Version | `1.0.0` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `transformers`, `torch`, `pillow` |
-| Platforms | linux, macos, windows |
-| Tags | `LLaVA`, `Vision-Language`, `Multimodal`, `Visual Question Answering`, `Image Chat`, `CLIP`, `Vicuna`, `Conversational AI`, `Instruction Tuning`, `VQA` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# LLaVA - Large Language and Vision Assistant
-
-Open-source vision-language model for conversational image understanding.
-
-## When to use LLaVA
-
-**Use when:**
-- Building vision-language chatbots
-- Visual question answering (VQA)
-- Image description and captioning
-- Multi-turn image conversations
-- Visual instruction following
-- Document understanding with images
-
-**Metrics**:
-- **23,000+ GitHub stars**
-- GPT-4V level capabilities (targeted)
-- Apache 2.0 License
-- Multiple model sizes (7B-34B params)
-
-**Use alternatives instead**:
-- **GPT-4V**: Highest quality, API-based
-- **CLIP**: Simple zero-shot classification
-- **BLIP-2**: Better for captioning only
-- **Flamingo**: Research, not open-source
-
-## Quick start
-
-### Installation
-
-```bash
-# Clone repository
-git clone https://github.com/haotian-liu/LLaVA
-cd LLaVA
-
-# Install
-pip install -e .
-```
-
-### Basic usage
-
-```python
-from llava.model.builder import load_pretrained_model
-from llava.mm_utils import get_model_name_from_path, process_images, tokenizer_image_token
-from llava.constants import IMAGE_TOKEN_INDEX, DEFAULT_IMAGE_TOKEN
-from llava.conversation import conv_templates
-from PIL import Image
-import torch
-
-# Load model
-model_path = "liuhaotian/llava-v1.5-7b"
-tokenizer, model, image_processor, context_len = load_pretrained_model(
-    model_path=model_path,
-    model_base=None,
-    model_name=get_model_name_from_path(model_path)
-)
-
-# Load image
-image = Image.open("image.jpg")
-image_tensor = process_images([image], image_processor, model.config)
-image_tensor = image_tensor.to(model.device, dtype=torch.float16)
-
-# Create conversation
-conv = conv_templates["llava_v1"].copy()
-conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")
-conv.append_message(conv.roles[1], None)
-prompt = conv.get_prompt()
-
-# Generate response
-input_ids = tokenizer_image_token(prompt, tokenizer, IMAGE_TOKEN_INDEX, return_tensors='pt').unsqueeze(0).to(model.device)
-
-with torch.inference_mode():
-    output_ids = model.generate(
-        input_ids,
-        images=image_tensor,
-        do_sample=True,
-        temperature=0.2,
-        max_new_tokens=512
-    )
-
-response = tokenizer.decode(output_ids[0], skip_special_tokens=True).strip()
-print(response)
-```
-
-## Available models
-
-| Model | Parameters | VRAM | Quality |
-|-------|------------|------|---------|
-| LLaVA-v1.5-7B | 7B | ~14 GB | Good |
-| LLaVA-v1.5-13B | 13B | ~28 GB | Better |
-| LLaVA-v1.6-34B | 34B | ~70 GB | Best |
-
-```python
-# Load different models
-model_7b = "liuhaotian/llava-v1.5-7b"
-model_13b = "liuhaotian/llava-v1.5-13b"
-model_34b = "liuhaotian/llava-v1.6-34b"
-
-# 4-bit quantization for lower VRAM
-load_4bit = True  # Reduces VRAM by ~4×
-```
-
-## CLI usage
-
-```bash
-# Single image query
-python -m llava.serve.cli \
-    --model-path liuhaotian/llava-v1.5-7b \
-    --image-file image.jpg \
-    --query "What is in this image?"
-
-# Multi-turn conversation
-python -m llava.serve.cli \
-    --model-path liuhaotian/llava-v1.5-7b \
-    --image-file image.jpg
-# Then type questions interactively
-```
-
-## Web UI (Gradio)
-
-```bash
-# Launch Gradio interface
-python -m llava.serve.gradio_web_server \
-    --model-path liuhaotian/llava-v1.5-7b \
-    --load-4bit  # Optional: reduce VRAM
-
-# Access at http://localhost:7860
-```
-
-## Multi-turn conversations
-
-```python
-# Initialize conversation
-conv = conv_templates["llava_v1"].copy()
-
-# Turn 1
-conv.append_message(conv.roles[0], DEFAULT_IMAGE_TOKEN + "\nWhat is in this image?")
-conv.append_message(conv.roles[1], None)
-response1 = generate(conv, model, image)  # "A dog playing in a park"
-
-# Turn 2
-conv.messages[-1][1] = response1  # Add previous response
-conv.append_message(conv.roles[0], "What breed is the dog?")
-conv.append_message(conv.roles[1], None)
-response2 = generate(conv, model, image)  # "Golden Retriever"
-
-# Turn 3
-conv.messages[-1][1] = response2
-conv.append_message(conv.roles[0], "What time of day is it?")
-conv.append_message(conv.roles[1], None)
-response3 = generate(conv, model, image)
-```
-
-## Common tasks
-
-### Image captioning
-
-```python
-question = "Describe this image in detail."
-response = ask(model, image, question)
-```
-
-### Visual question answering
-
-```python
-question = "How many people are in the image?"
-response = ask(model, image, question)
-```
-
-### Object detection (textual)
-
-```python
-question = "List all the objects you can see in this image."
-response = ask(model, image, question)
-```
-
-### Scene understanding
-
-```python
-question = "What is happening in this scene?"
-response = ask(model, image, question)
-```
-
-### Document understanding
-
-```python
-question = "What is the main topic of this document?"
-response = ask(model, document_image, question)
-```
-
-## Training custom model
-
-```bash
-# Stage 1: Feature alignment (558K image-caption pairs)
-bash scripts/v1_5/pretrain.sh
-
-# Stage 2: Visual instruction tuning (150K instruction data)
-bash scripts/v1_5/finetune.sh
-```
-
-## Quantization (reduce VRAM)
-
-```python
-# 4-bit quantization
-tokenizer, model, image_processor, context_len = load_pretrained_model(
-    model_path="liuhaotian/llava-v1.5-13b",
-    model_base=None,
-    model_name=get_model_name_from_path("liuhaotian/llava-v1.5-13b"),
-    load_4bit=True  # Reduces VRAM ~4×
-)
-
-# 8-bit quantization
-load_8bit=True  # Reduces VRAM ~2×
-```
-
-## Best practices
-
-1. **Start with 7B model** - Good quality, manageable VRAM
-2. **Use 4-bit quantization** - Reduces VRAM significantly
-3. **GPU required** - CPU inference extremely slow
-4. **Clear prompts** - Specific questions get better answers
-5. **Multi-turn conversations** - Maintain conversation context
-6. **Temperature 0.2-0.7** - Balance creativity/consistency
-7. **max_new_tokens 512-1024** - For detailed responses
-8. **Batch processing** - Process multiple images sequentially
-
-## Performance
-
-| Model | VRAM (FP16) | VRAM (4-bit) | Speed (tokens/s) |
-|-------|-------------|--------------|------------------|
-| 7B | ~14 GB | ~4 GB | ~20 |
-| 13B | ~28 GB | ~8 GB | ~12 |
-| 34B | ~70 GB | ~18 GB | ~5 |
-
-*On A100 GPU*
-
-## Benchmarks
-
-LLaVA achieves competitive scores on:
-- **VQAv2**: 78.5%
-- **GQA**: 62.0%
-- **MM-Vet**: 35.4%
-- **MMBench**: 64.3%
-
-## Limitations
-
-1. **Hallucinations** - May describe things not in image
-2. **Spatial reasoning** - Struggles with precise locations
-3. **Small text** - Difficulty reading fine print
-4. **Object counting** - Imprecise for many objects
-5. **VRAM requirements** - Need powerful GPU
-6. **Inference speed** - Slower than CLIP
-
-## Integration with frameworks
-
-### LangChain
-
-```python
-from langchain.llms.base import LLM
-
-class LLaVALLM(LLM):
-    def _call(self, prompt, stop=None):
-        # Custom LLaVA inference
-        return response
-
-llm = LLaVALLM()
-```
-
-### Gradio App
-
-```python
-import gradio as gr
-
-def chat(image, text, history):
-    response = ask_llava(model, image, text)
-    return response
-
-demo = gr.ChatInterface(
-    chat,
-    additional_inputs=[gr.Image(type="pil")],
-    title="LLaVA Chat"
-)
-demo.launch()
-```
-
-## Resources
-
-- **GitHub**: https://github.com/haotian-liu/LLaVA ⭐ 23,000+
-- **Paper**: https://arxiv.org/abs/2304.08485
-- **Demo**: https://llava.hliu.cc
-- **Models**: https://huggingface.co/liuhaotian
-- **License**: Apache 2.0

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/user-guide/skills/optional/mlops/mlops-modal.md

@@ -1,368 +0,0 @@
----
-title: "Modal — Serverless GPU cloud for ML jobs and model APIs"
-sidebar_label: "Modal"
-description: "Serverless GPU cloud for ML jobs and model APIs"
----
-
-{/* This page is auto-generated from the skill's SKILL.md by website/scripts/generate-skill-docs.py. Edit the source SKILL.md, not this page. */}
-
-# Modal
-
-Serverless GPU cloud for ML jobs and model APIs.
-
-## Skill metadata
-
-| | |
-|---|---|
-| Source | Optional — install with `hermes skills install official/mlops/modal` |
-| Path | `optional-skills/mlops\modal` |
-| Version | `1.0.1` |
-| Author | Orchestra Research |
-| License | MIT |
-| Dependencies | `modal>=1.0` |
-| Platforms | linux, macos, windows |
-| Tags | `Infrastructure`, `Serverless`, `GPU`, `Cloud`, `Deployment`, `Modal` |
-
-## Reference: full SKILL.md
-
-:::info
-The following is the complete skill definition that Hermes loads when this skill is triggered. This is what the agent sees as instructions when the skill is active.
-:::
-
-# Modal Serverless GPU
-
-Guide to running ML workloads on Modal's serverless GPU cloud platform.
-
-## When to use Modal
-
-**Use Modal when:**
-- Running GPU-intensive ML workloads without managing infrastructure
-- Deploying ML models as auto-scaling APIs
-- Running batch processing jobs (training, inference, data processing)
-- Need pay-per-second GPU pricing without idle costs
-- Prototyping ML applications quickly
-- Running scheduled jobs (cron-like workloads)
-
-**Key features:**
-- **Serverless GPUs**: T4, L4, A10G, L40S, A100, H100, H200, B200 on-demand
-- **Python-native**: Define infrastructure in Python code, no YAML
-- **Auto-scaling**: Scale to zero, scale to 100+ GPUs instantly
-- **Sub-second cold starts**: Rust-based infrastructure for fast container launches
-- **Container caching**: Image layers cached for rapid iteration
-- **Web endpoints**: Deploy functions as REST APIs with zero-downtime updates
-
-**Use alternatives instead:**
-- **RunPod**: For longer-running pods with persistent state
-- **Lambda Labs**: For reserved GPU instances
-- **SkyPilot**: For multi-cloud orchestration and cost optimization
-- **Kubernetes**: For complex multi-service architectures
-
-## Quick start
-
-### Installation
-
-```bash
-pip install modal
-modal setup  # Opens browser for authentication
-```
-
-### Hello World with GPU
-
-```python
-import modal
-
-app = modal.App("hello-gpu")
-
-@app.function(gpu="T4")
-def gpu_info():
-    import subprocess
-    return subprocess.run(["nvidia-smi"], capture_output=True, text=True).stdout
-
-@app.local_entrypoint()
-def main():
-    print(gpu_info.remote())
-```
-
-Run: `modal run hello_gpu.py`
-
-### Basic inference endpoint
-
-```python
-import modal
-
-app = modal.App("text-generation")
-image = modal.Image.debian_slim().pip_install("transformers", "torch", "accelerate")
-
-@app.cls(gpu="A10G", image=image)
-class TextGenerator:
-    @modal.enter()
-    def load_model(self):
-        from transformers import pipeline
-        self.pipe = pipeline("text-generation", model="gpt2", device=0)
-
-    @modal.method()
-    def generate(self, prompt: str) -> str:
-        return self.pipe(prompt, max_length=100)[0]["generated_text"]
-
-@app.local_entrypoint()
-def main():
-    print(TextGenerator().generate.remote("Hello, world"))
-```
-
-## Core concepts
-
-### Key components
-
-| Component | Purpose |
-|-----------|---------|
-| `App` | Container for functions and resources |
-| `Function` | Serverless function with compute specs |
-| `Cls` | Class-based functions with lifecycle hooks |
-| `Image` | Container image definition |
-| `Volume` | Persistent storage for models/data |
-| `Secret` | Secure credential storage |
-
-### Execution modes
-
-| Command | Description |
-|---------|-------------|
-| `modal run script.py` | Execute and exit |
-| `modal serve script.py` | Development with live reload |
-| `modal deploy script.py` | Persistent cloud deployment |
-
-## GPU configuration
-
-### Available GPUs
-
-| GPU | VRAM | Best For |
-|-----|------|----------|
-| `T4` | 16GB | Budget inference, small models |
-| `L4` | 24GB | Inference, Ada Lovelace arch |
-| `A10G` | 24GB | Training/inference, 3.3x faster than T4 |
-| `L40S` | 48GB | Recommended for inference (best cost/perf) |
-| `A100-40GB` | 40GB | Large model training |
-| `A100-80GB` | 80GB | Very large models |
-| `H100` | 80GB | Fastest, FP8 + Transformer Engine |
-| `H200` | 141GB | Auto-upgrade from H100, 4.8TB/s bandwidth |
-| `B200` | Latest | Blackwell architecture |
-
-### GPU specification patterns
-
-```python
-# Single GPU
-@app.function(gpu="A100")
-
-# Specific memory variant
-@app.function(gpu="A100-80GB")
-
-# Multiple GPUs (up to 8)
-@app.function(gpu="H100:4")
-
-# GPU with fallbacks
-@app.function(gpu=["H100", "A100", "L40S"])
-
-# Any available GPU
-@app.function(gpu="any")
-```
-
-## Container images
-
-```python
-# Basic image with pip
-image = modal.Image.debian_slim(python_version="3.11").pip_install(
-    "torch==2.1.0", "transformers==4.36.0", "accelerate"
-)
-
-# From CUDA base
-image = modal.Image.from_registry(
-    "nvidia/cuda:12.1.0-cudnn8-devel-ubuntu22.04",
-    add_python="3.11"
-).pip_install("torch", "transformers")
-
-# With system packages
-image = modal.Image.debian_slim().apt_install("git", "ffmpeg").pip_install("whisper")
-```
-
-## Persistent storage
-
-```python
-volume = modal.Volume.from_name("model-cache", create_if_missing=True)
-
-@app.function(gpu="A10G", volumes={"/models": volume})
-def load_model():
-    import os
-    model_path = "/models/llama-7b"
-    if not os.path.exists(model_path):
-        model = download_model()
-        model.save_pretrained(model_path)
-        volume.commit()  # Persist changes
-    return load_from_path(model_path)
-```
-
-## Web endpoints
-
-### FastAPI endpoint decorator
-
-```python
-@app.function()
-@modal.fastapi_endpoint(method="POST")
-def predict(text: str) -> dict:
-    return {"result": model.predict(text)}
-```
-
-### Full ASGI app
-
-```python
-from fastapi import FastAPI
-web_app = FastAPI()
-
-@web_app.post("/predict")
-async def predict(text: str):
-    return {"result": await model.predict.remote.aio(text)}
-
-@app.function()
-@modal.asgi_app()
-def fastapi_app():
-    return web_app
-```
-
-### Web endpoint types
-
-| Decorator | Use Case |
-|-----------|----------|
-| `@modal.fastapi_endpoint()` | Simple function → API |
-| `@modal.asgi_app()` | Full FastAPI/Starlette apps |
-| `@modal.wsgi_app()` | Django/Flask apps |
-| `@modal.web_server(port)` | Arbitrary HTTP servers |
-
-## Dynamic batching
-
-```python
-@app.function()
-@modal.batched(max_batch_size=32, wait_ms=100)
-async def batch_predict(inputs: list[str]) -> list[dict]:
-    # Inputs automatically batched
-    return model.batch_predict(inputs)
-```
-## Secrets management
-
-```bash
-# Create secret
-modal secret create huggingface HF_TOKEN=hf_xxx
-```
-
-```python
-@app.function(secrets=[modal.Secret.from_name("huggingface")])
-def download_model():
-    import os
-    token = os.environ["HF_TOKEN"]
-```
-
-## Scheduling
-
-```python
-@app.function(schedule=modal.Cron("0 0 * * *"))  # Daily midnight
-def daily_job():
-    pass
-
-@app.function(schedule=modal.Period(hours=1))
-def hourly_job():
-    pass
-```
-
-## Performance optimization
-
-### Cold start mitigation
-
-```python
-# Modal 1.0 autoscaler params: scaledown_window (was container_idle_timeout).
-# Input concurrency moved to the @modal.concurrent decorator.
-@app.function(scaledown_window=300)  # Keep warm 5 min
-@modal.concurrent(max_inputs=10)     # Handle concurrent requests per container
-def inference():
-    pass
-```
-
-### Model loading best practices
-
-```python
-@app.cls(gpu="A100")
-class Model:
-    @modal.enter()  # Run once at container start
-    def load(self):
-        self.model = load_model()  # Load during warm-up
-
-    @modal.method()
-    def predict(self, x):
-        return self.model(x)
-```
-
-## Parallel processing
-
-```python
-@app.function()
-def process_item(item):
-    return expensive_computation(item)
-
-@app.function()
-def run_parallel():
-    items = list(range(1000))
-    # Fan out to parallel containers
-    results = list(process_item.map(items))
-    return results
-```
-
-## Common configuration
-
-```python
-@app.function(
-    gpu="A100",
-    memory=32768,              # 32GB RAM
-    cpu=4,                     # 4 CPU cores
-    timeout=3600,              # 1 hour max
-    scaledown_window=120,      # Keep warm 2 min (was container_idle_timeout)
-    retries=3,                 # Retry on failure
-    max_containers=10,         # Max concurrent containers (was concurrency_limit)
-    min_containers=1,          # Keep N containers warm (was keep_warm)
-)
-def my_function():
-    pass
-```
-
-> **Modal 1.0 autoscaler renames** (see the [migration guide](https://modal.com/docs/guide/modal-1-0-migration)):
-> - `container_idle_timeout` → `scaledown_window`
-> - `concurrency_limit` → `max_containers`
-> - `keep_warm` → `min_containers`
-> - `allow_concurrent_inputs=N` → the `@modal.concurrent(max_inputs=N)` decorator
-
-## Debugging
-
-```python
-# Test locally
-if __name__ == "__main__":
-    result = my_function.local()
-
-# View logs
-# modal app logs my-app
-```
-
-## Common issues
-
-| Issue | Solution |
-|-------|----------|
-| Cold start latency | Increase `scaledown_window`, use `@modal.enter()` |
-| GPU OOM | Use larger GPU (`A100-80GB`), enable gradient checkpointing |
-| Image build fails | Pin dependency versions, check CUDA compatibility |
-| Timeout errors | Increase `timeout`, add checkpointing |
-
-## References
-
-- **[Advanced Usage](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\modal/references/advanced-usage.md)** - Multi-GPU, distributed training, cost optimization
-- **[Troubleshooting](https://github.com/NousResearch/hermes-agent/blob/main/optional-skills/mlops\modal/references/troubleshooting.md)** - Common issues and solutions
-
-## Resources
-
-- **Documentation**: https://modal.com/docs
-- **Examples**: https://github.com/modal-labs/modal-examples
-- **Pricing**: https://modal.com/pricing
-- **Discord**: https://discord.gg/modal