better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/guides/google-vertex.md

@@ -0,0 +1,148 @@
+---
+sidebar_position: 15
+title: "Google Vertex AI"
+description: "Use Hermes Agent with Gemini on Google Cloud Vertex AI — OAuth2 service account or ADC, GCP billing and quotas, no static API key"
+---
+
+# Google Vertex AI
+
+Hermes Agent supports **Gemini models on Google Cloud Vertex AI** through Vertex's OpenAI-compatible endpoint. Unlike the [Google AI Studio provider](/guides/google-gemini) (which uses a static API key against `generativelanguage.googleapis.com`), Vertex gives you **enterprise-grade rate limits and GCP billing/credits**, and is the right choice when you want Gemini usage to draw on your Google Cloud account rather than an AI Studio key.
+
+:::info Vertex authenticates with OAuth2, not an API key
+Vertex has **no static API key** for the standard endpoint. Every request needs a short-lived **OAuth2 access token** (≈1 hour TTL) minted from either a service-account JSON or Application Default Credentials (ADC). Hermes mints and **auto-refreshes** these tokens for you — you never paste a token by hand. This is why pasting a temporary token into a custom provider's `api_key` field does not work: it expires mid-session.
+:::
+
+## Prerequisites
+
+- **A Google Cloud project** with the **Vertex AI API enabled** and billing active.
+- **Credentials**, one of:
+  - a **service-account JSON** key file with the `roles/aiplatform.user` role, or
+  - **Application Default Credentials** via `gcloud auth application-default login` (or the metadata server when running on a GCP VM).
+- **`google-auth`** — installed automatically the first time you select Vertex (lazy install). Run `hermes setup` to repair a managed install if that fails.
+
+## Quick Start
+
+```bash
+# Option A — service account JSON (recommended for servers / gateways)
+echo "VERTEX_CREDENTIALS_PATH=/path/to/service-account.json" >> ~/.hermes/.env
+
+# Option B — Application Default Credentials (good for local dev)
+gcloud auth application-default login
+
+# Select Vertex as your provider
+hermes model
+# → Choose "More providers..." → "Google Vertex AI"
+# → Enter your GCP project ID (or leave blank to use the one in your credentials)
+# → Choose a region (default: global)
+# → Select a Gemini model
+
+# Start chatting
+hermes chat
+```
+
+## Configuration
+
+Vertex splits its settings by sensitivity:
+
+- The **credential path** is a pointer to a secret and lives in `~/.hermes/.env`.
+- **Project ID and region** are non-secret routing settings and live in `~/.hermes/config.yaml`.
+
+`~/.hermes/.env`:
+
+```bash
+# One of these (checked in this order); omit both to use ADC:
+VERTEX_CREDENTIALS_PATH=/path/to/service-account.json
+GOOGLE_APPLICATION_CREDENTIALS=/path/to/service-account.json
+```
+
+`~/.hermes/config.yaml`:
+
+```yaml
+model:
+  default: google/gemini-3-flash-preview
+  provider: vertex
+
+vertex:
+  project_id: my-gcp-project   # blank → use the project embedded in the credentials
+  region: global               # "global" is required for the Gemini 3.x previews
+```
+
+:::tip Environment variables win over config.yaml
+`VERTEX_PROJECT_ID` and `VERTEX_REGION` override the `vertex.project_id` / `vertex.region` values in `config.yaml`. Use them for per-shell overrides; keep the durable settings in `config.yaml`.
+:::
+
+### How authentication works
+
+1. Hermes resolves credentials in this order: `VERTEX_CREDENTIALS_PATH` → `GOOGLE_APPLICATION_CREDENTIALS` → ADC.
+2. It mints an OAuth2 access token (`cloud-platform` scope) and caches it, refreshing when the token is within 5 minutes of expiry.
+3. The token is handed to a standard OpenAI client pointed at the Vertex endpoint:
+   ```text
+   https://aiplatform.googleapis.com/v1beta1/projects/{project}/locations/{region}/endpoints/openapi
+   ```
+   Regional locations use a `{region}-aiplatform.googleapis.com` host instead.
+4. If a session runs longer than the token lifetime and a request returns `401`, Hermes re-mints the token and retries automatically. On a long-running gateway, if ADC's refresh token has itself expired, Hermes falls back to the service-account JSON when one is configured.
+
+## Available Models
+
+Vertex requires the `google/` vendor prefix on model IDs. The `hermes model` picker offers:
+
+| Model | ID |
+|-------|----|
+| Gemini 3.8 Flash | `google/gemini-3.8-flash` |
+| Gemini 3.7 Flash | `google/gemini-3.7-flash` |
+| Gemini 3.1 Pro Preview | `google/gemini-3.1-pro-preview` |
+| Gemini 3 Pro Preview | `google/gemini-3-pro-preview` |
+| Gemini 3 Flash Preview | `google/gemini-3-flash-preview` |
+| Gemini 3.1 Flash Lite Preview | `google/gemini-3.1-flash-lite-preview` |
+| Gemini 2.5 Pro | `google/gemini-2.5-pro` |
+| Gemini 2.5 Flash | `google/gemini-2.5-flash` |
+
+:::note `global` region for Gemini 3.x
+The Gemini 3.x preview models are served through the `global` endpoint. Regional endpoints (`us-central1`, etc.) may 404 them. Leave `region: global` unless you have a specific reason to pin a region.
+:::
+
+## Switching Models Mid-Session
+
+```text
+/model google/gemini-3-pro-preview
+/model google/gemini-3-flash-preview
+```
+
+`/model` switches among already-configured providers and models; it does not collect new credentials. Configure Vertex with `hermes model` first.
+
+## Reasoning / Thinking
+
+Vertex exposes Gemini's thinking budget through the OpenAI-compatible surface. Hermes maps its reasoning-effort setting onto `extra_body.google.thinking_config` automatically, so `reasoning_effort` works the same way it does on other Gemini surfaces.
+
+## Diagnostics
+
+```bash
+hermes doctor
+```
+
+The doctor reports whether Vertex credentials can be resolved (service-account path or ADC) and whether the provider is configured.
+
+## Troubleshooting
+
+### "Vertex AI credentials could not be resolved"
+
+Hermes found neither a service-account JSON nor working ADC. Either set `VERTEX_CREDENTIALS_PATH` in `~/.hermes/.env`, or run `gcloud auth application-default login`. If your project isn't embedded in the credentials, set `vertex.project_id` in `config.yaml`.
+
+### `google-auth` not installed
+
+Hermes lazy-installs it the first time you select the Vertex provider. If that fails, run `hermes setup` to repair the managed install.
+
+### 404 on Gemini 3.x models
+
+You are probably on a regional endpoint. Set `region: global` in the `vertex:` section of `config.yaml` (or unset `VERTEX_REGION`).
+
+### 403 / permission denied
+
+The service account (or your ADC identity) needs the `roles/aiplatform.user` role on the project, and the Vertex AI API must be enabled for that project.
+
+## Related
+
+- [Google Gemini (AI Studio)](/guides/google-gemini) — static-API-key Gemini without GCP
+- [AWS Bedrock](/guides/aws-bedrock) — another native cloud-provider integration
+- [AI Providers](/integrations/providers)
+- [Configuration](/user-guide/configuration)

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/guides/local-llm-on-mac.md

@@ -0,0 +1,252 @@
+---
+sidebar_position: 2
+title: "Run Local LLMs on Mac"
+description: "Set up a local OpenAI-compatible LLM server on macOS with llama.cpp or MLX, including model selection, memory optimization, and real benchmarks on Apple Silicon"
+---
+
+# Run Local LLMs on Mac
+
+:::tip Desktop users: there's a one-click path
+On the Hermes desktop app, **Settings → Providers → Local Models** installs
+and manages a local llama.cpp server for you — model downloads, memory
+fitting, and context sizing included. See [Local Models](/user-guide/local-models).
+This guide is for manual setup: MLX, custom builds, or servers you want to
+run yourself.
+:::
+
+This guide walks you through running a local LLM server on macOS with an OpenAI-compatible API. You get full privacy, zero API costs, and surprisingly good performance on Apple Silicon.
+
+We cover two backends:
+
+| Backend | Install | Best at | Format |
+|---------|---------|---------|--------|
+| **llama.cpp** | `brew install llama.cpp` | Fastest time-to-first-token, quantized KV cache for low memory | GGUF |
+| **omlx** | [omlx.ai](https://omlx.ai) | Fastest token generation, native Metal optimization | MLX (safetensors) |
+
+Both expose an OpenAI-compatible `/v1/chat/completions` endpoint. Hermes works with either one — just point it at `http://localhost:8080` or `http://localhost:8000`.
+
+:::info Apple Silicon only
+This guide targets Macs with Apple Silicon (M1 and later). Intel Macs will work with llama.cpp but without GPU acceleration — expect significantly slower performance.
+:::
+
+---
+
+## Choosing a model
+
+For getting started, we recommend **Qwen3.5-9B** — it's a strong reasoning model that fits comfortably in 8GB+ of unified memory with quantization.
+
+| Variant | Size on disk | RAM needed (128K context) | Backend |
+|---------|-------------|---------------------------|---------|
+| Qwen3.5-9B-Q4_K_M (GGUF) | 5.3 GB | ~10 GB with quantized KV cache | llama.cpp |
+| Qwen3.5-9B-mlx-lm-mxfp4 (MLX) | ~5 GB | ~12 GB | omlx |
+
+**Memory rule of thumb:** model size + KV cache. A 9B Q4 model is ~5 GB. The KV cache at 128K context with Q4 quantization adds ~4-5 GB. With default (f16) KV cache, that balloons to ~16 GB. The quantized KV cache flags in llama.cpp are the key trick for memory-constrained systems.
+
+For larger models (27B, 35B), you'll need 32 GB+ of unified memory. The 9B is the sweet spot for 8-16 GB machines.
+
+---
+
+## Option A: llama.cpp
+
+llama.cpp is the most portable local LLM runtime. On macOS it uses Metal for GPU acceleration out of the box.
+
+### Install
+
+```bash
+brew install llama.cpp
+```
+
+This gives you the `llama-server` command globally.
+
+### Download the model
+
+You need a GGUF-format model. The easiest source is Hugging Face via the `huggingface-cli`:
+
+```bash
+brew install huggingface-cli
+```
+
+Then download:
+
+```bash
+huggingface-cli download unsloth/Qwen3.5-9B-GGUF Qwen3.5-9B-Q4_K_M.gguf --local-dir ~/models
+```
+
+:::tip Gated models
+Some models on Hugging Face require authentication. Run `huggingface-cli login` first if you get a 401 or 404 error.
+:::
+
+### Start the server
+
+```bash
+llama-server -m ~/models/Qwen3.5-9B-Q4_K_M.gguf \
+  -ngl 99 \
+  -c 131072 \
+  -np 1 \
+  -fa on \
+  --cache-type-k q4_0 \
+  --cache-type-v q4_0 \
+  --host 0.0.0.0
+```
+
+Here's what each flag does:
+
+| Flag | Purpose |
+|------|---------|
+| `-ngl 99` | Offload all layers to GPU (Metal). Use a high number to ensure nothing stays on CPU. |
+| `-c 131072` | Context window size (128K tokens). Reduce this if you're low on memory. |
+| `-np 1` | Number of parallel slots. Keep at 1 for single-user use — more slots split your memory budget. |
+| `-fa on` | Flash attention. Reduces memory usage and speeds up long-context inference. |
+| `--cache-type-k q4_0` | Quantize the key cache to 4-bit. **This is the big memory saver.** |
+| `--cache-type-v q4_0` | Quantize the value cache to 4-bit. Together with the above, this cuts KV cache memory by ~75% vs f16. |
+| `--host 0.0.0.0` | Listen on all interfaces. Use `127.0.0.1` if you don't need network access. |
+
+The server is ready when you see:
+
+```
+main: server is listening on http://0.0.0.0:8080
+srv  update_slots: all slots are idle
+```
+
+### Memory optimization for constrained systems
+
+The `--cache-type-k q4_0 --cache-type-v q4_0` flags are the most important optimization for systems with limited memory. Here's the impact at 128K context:
+
+| KV cache type | KV cache memory (128K ctx, 9B model) |
+|---------------|--------------------------------------|
+| f16 (default) | ~16 GB |
+| q8_0 | ~8 GB |
+| **q4_0** | **~4 GB** |
+
+On an 8 GB Mac, use `q4_0` KV cache and choose a smaller model that can still fit Hermes' 64K minimum context. On 16 GB, you can comfortably do 128K context. On 32 GB+, you can run larger models or multiple parallel slots.
+
+If you're still running out of memory, reduce context only while staying at or above Hermes' 64K minimum; otherwise switch to a smaller model or smaller quantization (Q3_K_M instead of Q4_K_M).
+
+### Test it
+
+```bash
+curl -s http://localhost:8080/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen3.5-9B-Q4_K_M.gguf",
+    "messages": [{"role": "user", "content": "Hello!"}],
+    "max_tokens": 50
+  }' | jq .choices[0].message.content
+```
+
+### Get the model name
+
+If you forget the model name, query the models endpoint:
+
+```bash
+curl -s http://localhost:8080/v1/models | jq '.data[].id'
+```
+
+---
+
+## Option B: MLX via omlx
+
+[omlx](https://omlx.ai) is a macOS-native app that manages and serves MLX models. MLX is Apple's own machine learning framework, optimized specifically for Apple Silicon's unified memory architecture.
+
+### Install
+
+Download and install from [omlx.ai](https://omlx.ai). It provides a GUI for model management and a built-in server.
+
+### Download the model
+
+Use the omlx app to browse and download models. Search for `Qwen3.5-9B-mlx-lm-mxfp4` and download it. Models are stored locally (typically in `~/.omlx/models/`).
+
+### Start the server
+
+omlx serves models on `http://127.0.0.1:8000` by default. Start serving from the app UI, or use the CLI if available.
+
+### Test it
+
+```bash
+curl -s http://127.0.0.1:8000/v1/chat/completions \
+  -H "Content-Type: application/json" \
+  -d '{
+    "model": "Qwen3.5-9B-mlx-lm-mxfp4",
+    "messages": [{"role": "user", "content": "Hello!"}],
+    "max_tokens": 50
+  }' | jq .choices[0].message.content
+```
+
+### List available models
+
+omlx can serve multiple models simultaneously:
+
+```bash
+curl -s http://127.0.0.1:8000/v1/models | jq '.data[].id'
+```
+
+---
+
+## Benchmarks: llama.cpp vs MLX
+
+Both backends tested on the same machine (Apple M5 Max, 128 GB unified memory) running the same model (Qwen3.5-9B) at comparable quantization levels (Q4_K_M for GGUF, mxfp4 for MLX). Five diverse prompts, three runs each, backends tested sequentially to avoid resource contention.
+
+### Results
+
+| Metric | llama.cpp (Q4_K_M) | MLX (mxfp4) | Winner |
+|--------|-------------------|-------------|--------|
+| **TTFT (avg)** | **67 ms** | 289 ms | llama.cpp (4.3x faster) |
+| **TTFT (p50)** | **66 ms** | 286 ms | llama.cpp (4.3x faster) |
+| **Generation (avg)** | 70 tok/s | **96 tok/s** | MLX (37% faster) |
+| **Generation (p50)** | 70 tok/s | **96 tok/s** | MLX (37% faster) |
+| **Total time (512 tokens)** | 7.3s | **5.5s** | MLX (25% faster) |
+
+### What this means
+
+- **llama.cpp** excels at prompt processing — its flash attention + quantized KV cache pipeline gets you the first token in ~66ms. If you're building interactive applications where perceived responsiveness matters (chatbots, autocomplete), this is a meaningful advantage.
+
+- **MLX** generates tokens ~37% faster once it gets going. For batch workloads, long-form generation, or any task where total completion time matters more than initial latency, MLX finishes sooner.
+
+- Both backends are **extremely consistent** — variance across runs was negligible. You can rely on these numbers.
+
+### Which one should you pick?
+
+| Use case | Recommendation |
+|----------|---------------|
+| Interactive chat, low-latency tools | llama.cpp |
+| Long-form generation, bulk processing | MLX (omlx) |
+| Memory-constrained (8-16 GB) | llama.cpp (quantized KV cache is unmatched) |
+| Serving multiple models simultaneously | omlx (built-in multi-model support) |
+| Maximum compatibility (Linux too) | llama.cpp |
+
+---
+
+## Connect to Hermes
+
+Once your local server is running:
+
+```bash
+hermes model
+```
+
+Select **Custom endpoint** and follow the prompts. It will ask for the base URL and model name — use the values from whichever backend you set up above.
+
+---
+
+## Timeouts
+
+Hermes automatically detects local endpoints (localhost, LAN IPs) and relaxes its streaming timeouts. No configuration needed for most setups.
+
+If you still hit timeout errors (e.g. very large contexts on slow hardware), you can override the streaming read timeout:
+
+```bash
+# In your .env — raise from the 120s default to 30 minutes
+HERMES_STREAM_READ_TIMEOUT=1800
+```
+
+| Timeout | Default | Local auto-adjustment | Env var override |
+|---------|---------|----------------------|------------------|
+| Stream read (socket-level) | 120s | Raised to 1800s | `HERMES_STREAM_READ_TIMEOUT` |
+| Stale stream detection | 180s | Disabled entirely | `HERMES_STREAM_STALE_TIMEOUT` |
+| API call (non-streaming) | 1800s | No change needed | `HERMES_API_TIMEOUT` |
+
+The stream read timeout is the one most likely to cause issues — it's the socket-level deadline for receiving the next chunk of data. During prefill on large contexts, local models may produce no output for minutes while processing the prompt. The auto-detection handles this transparently.
+
+:::tip A silent first turn is usually prefill, not a hang
+Hermes sends its system prompt and tool schemas on every call, so on slower hardware the first turn can involve minutes of silence while the model processes that prompt before generating anything. That's prefill at work, not a stalled session. See [Slow first response (prefill)](./local-ollama-setup.md#slow-first-response-prefill) in the Ollama guide for mitigations like keeping the model loaded and trimming the fixed prompt with `hermes prompt-size`.
+:::