better-dsh 0.2.5-c → 0.2.5-e

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/trajectory-format.md

@@ -1,232 +0,0 @@
-# Trajectory Format
-
-Hermes Agent saves conversation trajectories in ShareGPT-compatible JSONL format
-for use as training data, debugging artifacts, and reinforcement learning datasets.
-
-Source files: `agent/trajectory.py`, `agent/session_persistence.py` (search for `_save_trajectory`), `batch_runner.py`
-
-
-## File Naming Convention
-
-Trajectories are written to files in the current working directory:
-
-| File | When |
-|------|------|
-| `trajectory_samples.jsonl` | Conversations that completed successfully (`completed=True`) |
-| `failed_trajectories.jsonl` | Conversations that failed or were interrupted (`completed=False`) |
-
-The batch runner (`batch_runner.py`) writes to a custom output file per batch
-(e.g., `batch_001_output.jsonl`) with additional metadata fields.
-
-You can override the filename via the `filename` parameter in `save_trajectory()`.
-
-
-## JSONL Entry Format
-
-Each line in the file is a self-contained JSON object. There are two variants:
-
-### CLI/Interactive Format (from `_save_trajectory`)
-
-```json
-{
-  "conversations": [ ... ],
-  "timestamp": "2026-03-30T14:22:31.456789",
-  "model": "anthropic/claude-sonnet-4.6",
-  "completed": true
-}
-```
-
-### Batch Runner Format (from `batch_runner.py`)
-
-```json
-{
-  "prompt_index": 42,
-  "conversations": [ ... ],
-  "metadata": { "prompt_source": "gsm8k", "difficulty": "hard" },
-  "completed": true,
-  "partial": false,
-  "api_calls": 7,
-  "toolsets_used": ["code_tools", "file_tools"],
-  "tool_stats": {
-    "terminal": {"count": 3, "success": 3, "failure": 0},
-    "read_file": {"count": 2, "success": 2, "failure": 0},
-    "write_file": {"count": 0, "success": 0, "failure": 0}
-  },
-  "tool_error_counts": {
-    "terminal": 0,
-    "read_file": 0,
-    "write_file": 0
-  }
-}
-```
-
-The `tool_stats` and `tool_error_counts` dictionaries are normalized to include
-ALL possible tools (from `model_tools.TOOL_TO_TOOLSET_MAP`) with zero defaults,
-ensuring consistent schema across entries for HuggingFace dataset loading.
-
-
-## Conversations Array (ShareGPT Format)
-
-The `conversations` array uses ShareGPT role conventions:
-
-| API Role | ShareGPT `from` |
-|----------|-----------------|
-| system | `"system"` |
-| user | `"human"` |
-| assistant | `"gpt"` |
-| tool | `"tool"` |
-
-### Complete Example
-
-```json
-{
-  "conversations": [
-    {
-      "from": "system",
-      "value": "You are a function calling AI model. You are provided with function signatures within <tools> </tools> XML tags. You may call one or more functions to assist with the user query. If available tools are not relevant in assisting with user query, just respond in natural conversational language. Don't make assumptions about what values to plug into functions. After calling & executing the functions, you will be provided with function results within <tool_response> </tool_response> XML tags. Here are the available tools:\n<tools>\n[{\"name\": \"terminal\", \"description\": \"Execute shell commands\", \"parameters\": {\"type\": \"object\", \"properties\": {\"command\": {\"type\": \"string\"}}}, \"required\": null}]\n</tools>\nFor each function call return a JSON object, with the following pydantic model json schema for each:\n{'title': 'FunctionCall', 'type': 'object', 'properties': {'name': {'title': 'Name', 'type': 'string'}, 'arguments': {'title': 'Arguments', 'type': 'object'}}, 'required': ['name', 'arguments']}\nEach function call should be enclosed within <tool_call> </tool_call> XML tags.\nExample:\n<tool_call>\n{'name': <function-name>,'arguments': <args-dict>}\n</tool_call>"
-    },
-    {
-      "from": "human",
-      "value": "What Python version is installed?"
-    },
-    {
-      "from": "gpt",
-      "value": "<think>\nThe user wants to know the Python version. I should run python3 --version.\n</think>\n<tool_call>\n{\"name\": \"terminal\", \"arguments\": {\"command\": \"python3 --version\"}}\n</tool_call>"
-    },
-    {
-      "from": "tool",
-      "value": "<tool_response>\n{\"tool_call_id\": \"call_abc123\", \"name\": \"terminal\", \"content\": \"Python 3.11.6\"}\n</tool_response>"
-    },
-    {
-      "from": "gpt",
-      "value": "<think>\nGot the version. I can now answer the user.\n</think>\nPython 3.11.6 is installed on this system."
-    }
-  ],
-  "timestamp": "2026-03-30T14:22:31.456789",
-  "model": "anthropic/claude-sonnet-4.6",
-  "completed": true
-}
-```
-
-
-## Normalization Rules
-
-### Reasoning Content Markup
-
-The trajectory converter normalizes ALL reasoning into `<think>` tags, regardless
-of how the model originally produced it:
-
-1. **Native thinking tokens** (`msg["reasoning"]` field from providers like
-   Anthropic, OpenAI o-series): Wrapped as `<think>\n{reasoning}\n</think>\n`
-   and prepended before the content.
-
-2. **REASONING_SCRATCHPAD XML** (when native thinking is disabled and the model
-   reasons via system-prompt-instructed XML): `<REASONING_SCRATCHPAD>` tags are
-   converted to `<think>` via `convert_scratchpad_to_think()`.
-
-3. **Empty think blocks**: Every `gpt` turn is guaranteed to have a `<think>`
-   block. If no reasoning was produced, an empty block is inserted:
-   `<think>\n</think>\n` — this ensures consistent format for training data.
-
-### Tool Call Normalization
-
-Tool calls from the API format (with `tool_call_id`, function name, arguments as
-JSON string) are converted to XML-wrapped JSON:
-
-```
-<tool_call>
-{"name": "terminal", "arguments": {"command": "ls -la"}}
-</tool_call>
-```
-
-- Arguments are parsed from JSON strings back to objects (not double-encoded)
-- If JSON parsing fails (shouldn't happen — validated during conversation),
-  an empty `{}` is used with a warning logged
-- Multiple tool calls in one assistant turn produce multiple `<tool_call>` blocks
-  in a single `gpt` message
-
-### Tool Response Normalization
-
-All tool results following an assistant message are grouped into a single `tool`
-turn with XML-wrapped JSON responses:
-
-```
-<tool_response>
-{"tool_call_id": "call_abc123", "name": "terminal", "content": "output here"}
-</tool_response>
-```
-
-- If tool content looks like JSON (starts with `{` or `[`), it's parsed so the
-  content field contains a JSON object/array rather than a string
-- Multiple tool results are joined with newlines in one message
-- The tool name is matched by position against the parent assistant's `tool_calls`
-  array
-
-### System Message
-
-The system message is generated at save time (not taken from the conversation).
-It follows the Hermes function-calling prompt template with:
-
-- Preamble explaining the function-calling protocol
-- `<tools>` XML block containing the JSON tool definitions
-- Schema reference for `FunctionCall` objects
-- `<tool_call>` example
-
-Tool definitions include `name`, `description`, `parameters`, and `required`
-(set to `null` to match the canonical format).
-
-
-## Loading Trajectories
-
-Trajectories are standard JSONL — load with any JSON-lines reader:
-
-```python
-import json
-
-def load_trajectories(path: str):
-    """Load trajectory entries from a JSONL file."""
-    entries = []
-    with open(path, "r", encoding="utf-8") as f:
-        for line in f:
-            line = line.strip()
-            if line:
-                entries.append(json.loads(line))
-    return entries
-
-# Filter to successful completions only
-successful = [e for e in load_trajectories("trajectory_samples.jsonl")
-              if e.get("completed")]
-
-# Extract just the conversations for training
-training_data = [e["conversations"] for e in successful]
-```
-
-### Loading for HuggingFace Datasets
-
-```python
-from datasets import load_dataset
-
-ds = load_dataset("json", data_files="trajectory_samples.jsonl")
-```
-
-The normalized `tool_stats` schema ensures all entries have the same columns,
-preventing Arrow schema mismatch errors during dataset loading.
-
-
-## Controlling Trajectory Saving
-
-Trajectory saving is a `run_agent.py` / library-level switch — the `hermes` CLI
-does not expose a config key or flag for it:
-
-```bash
-python run_agent.py --save_trajectories --query='your question here'
-```
-
-Or programmatically: `AIAgent(..., save_trajectories=True)` /
-`initialize_agent(..., save_trajectories=True)`. When enabled, the
-`_save_trajectory()` method is called at the end of each conversation turn.
-
-The batch runner always saves trajectories (that's its primary purpose).
-
-Samples with zero reasoning across all turns are automatically discarded by the
-batch runner to avoid polluting training data with non-reasoning examples.

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/video-gen-provider-plugin.md

@@ -1,231 +0,0 @@
----
-sidebar_position: 12
-title: "Video Generation Provider Plugins"
-description: "How to build a video-generation backend plugin for Hermes Agent"
----
-
-# Building a Video Generation Provider Plugin
-
-Video-gen provider plugins register a backend that services every `video_generate` tool call. Built-in providers (xAI, FAL, OpenRouter, DeepInfra) ship as plugins. Add a new one, or override a bundled one, by dropping a directory into `plugins/video_gen/<name>/`.
-
-:::tip
-Video-gen mirrors [Image Generation Provider Plugins](/developer-guide/image-gen-provider-plugin) almost line-for-line — if you've built an image-gen backend, you already know the shape. The main differences: a `capabilities()` method advertising modalities/aspect-ratios/durations, and a routing convention (pass `image_url` to use image-to-video, omit it to use text-to-video — the provider picks the right endpoint internally).
-:::
-
-## The unified surface (one tool, two modalities)
-
-The `video_generate` tool exposes two modalities through one parameter:
-
-- **Text-to-video** — call with `prompt` only. The provider routes to its text-to-video endpoint.
-- **Image-to-video** — call with `prompt` + `image_url`. The provider routes to its image-to-video endpoint.
-
-Edit and extend are intentionally out of scope. Most backends don't support them and the inconsistency would force per-backend prose into the agent's tool description.
-
-## How discovery works
-
-Hermes scans for video-gen backends in three places:
-
-1. **Bundled** — `<repo>/plugins/video_gen/<name>/` (auto-loaded with `kind: backend`)
-2. **User** — `~/.hermes/plugins/video_gen/<name>/` (opt-in via `plugins.enabled`)
-3. **Pip** — packages declaring a `hermes_agent.plugins` entry point
-
-Each plugin's `register(ctx)` function calls `ctx.register_video_gen_provider(...)`. The active provider is picked by `video_gen.provider` in `config.yaml`; `hermes tools` → Video Generation walks users through selection. Unlike `image_generate`, there is no in-tree legacy backend — every provider is a plugin.
-
-## Directory structure
-
-```
-plugins/video_gen/my-backend/
-├── __init__.py      # VideoGenProvider subclass + register()
-└── plugin.yaml      # Manifest with kind: backend
-```
-
-## The VideoGenProvider ABC
-
-Subclass `agent.video_gen_provider.VideoGenProvider`. Required: `name` property and `generate()` method.
-
-```python
-# plugins/video_gen/my-backend/__init__.py
-from typing import Any, Dict, List, Optional
-import os
-
-from agent.video_gen_provider import (
-    VideoGenProvider,
-    error_response,
-    success_response,
-)
-
-
-class MyVideoGenProvider(VideoGenProvider):
-    @property
-    def name(self) -> str:
-        return "my-backend"
-
-    @property
-    def display_name(self) -> str:
-        return "My Backend"
-
-    def is_available(self) -> bool:
-        return bool(os.environ.get("MY_API_KEY"))
-
-    def list_models(self) -> List[Dict[str, Any]]:
-        # Each entry is a model FAMILY — a name the user picks once.
-        # Your provider's generate() routes within the family based on
-        # whether image_url was passed.
-        return [
-            {
-                "id": "fast",
-                "display": "Fast",
-                "speed": "~30s",
-                "strengths": "Cheapest tier",
-                "price": "$0.05/s",
-                "modalities": ["text", "image"],  # advisory
-            },
-        ]
-
-    def default_model(self) -> Optional[str]:
-        return "fast"
-
-    def capabilities(self) -> Dict[str, Any]:
-        return {
-            "modalities": ["text", "image"],
-            "aspect_ratios": ["16:9", "9:16"],
-            "resolutions": ["720p", "1080p"],
-            "min_duration": 1,
-            "max_duration": 10,
-            "supports_audio": False,
-            "supports_negative_prompt": True,
-            "max_reference_images": 0,
-        }
-
-    def get_setup_schema(self) -> Dict[str, Any]:
-        return {
-            "name": "My Backend",
-            "badge": "paid",
-            "tag": "Short description shown in `hermes tools`",
-            "env_vars": [
-                {
-                    "key": "MY_API_KEY",
-                    "prompt": "My Backend API key",
-                    "url": "https://mybackend.example.com/keys",
-                },
-            ],
-        }
-
-    def generate(
-        self,
-        prompt: str,
-        *,
-        model: Optional[str] = None,
-        image_url: Optional[str] = None,
-        reference_image_urls: Optional[List[str]] = None,
-        duration: Optional[int] = None,
-        aspect_ratio: str = "16:9",
-        resolution: str = "720p",
-        negative_prompt: Optional[str] = None,
-        audio: Optional[bool] = None,
-        seed: Optional[int] = None,
-        **kwargs: Any,  # always ignore unknown kwargs for forward-compat
-    ) -> Dict[str, Any]:
-        # ROUTE: image_url presence picks the endpoint.
-        if image_url:
-            endpoint = "my-backend/image-to-video"
-            modality_used = "image"
-        else:
-            endpoint = "my-backend/text-to-video"
-            modality_used = "text"
-
-        # ... call your API ...
-
-        return success_response(
-            video="https://your-cdn/output.mp4",
-            model=model or "fast",
-            prompt=prompt,
-            modality=modality_used,
-            aspect_ratio=aspect_ratio,
-            duration=duration or 5,
-            provider=self.name,
-        )
-
-
-def register(ctx) -> None:
-    ctx.register_video_gen_provider(MyVideoGenProvider())
-```
-
-## The plugin manifest
-
-```yaml
-# plugins/video_gen/my-backend/plugin.yaml
-name: my-backend
-version: 1.0.0
-description: "My video generation backend"
-author: Your Name
-kind: backend
-requires_env:
-  - MY_API_KEY
-```
-
-## The `video_generate` schema
-
-The tool exposes one schema across every backend. Providers ignore parameters they don't support.
-
-| Parameter | What it does |
-|---|---|
-| `prompt` | Text instruction (required) |
-| `image_url` | When set → image-to-video; when omitted → text-to-video |
-| `reference_image_urls` | Style/character refs (provider-dependent) |
-| `duration` | Seconds — provider clamps |
-| `aspect_ratio` | `"16:9"`, `"9:16"`, `"1:1"`, ... — provider clamps |
-| `resolution` | `"480p"` / `"540p"` / `"720p"` / `"1080p"` — provider clamps |
-| `negative_prompt` | Content to avoid (Pixverse/Kling only) |
-| `audio` | Native audio (Veo3 / Pixverse pricing tier) |
-| `seed` | Reproducibility |
-| `model` | Override the active model/family |
-
-The provider's `capabilities()` advertises which of these are honored. The agent sees the active backend's capabilities in the tool description, dynamically rebuilt when the user changes backend via `hermes tools`.
-
-## Model families and endpoint routing (the FAL pattern)
-
-When your backend has multiple endpoints per "model" — like FAL, where every family (Veo 3.1, Pixverse v6, Kling O3) has both a `/text-to-video` and an `/image-to-video` URL — represent each **family** as one catalog entry. Your `generate()` picks the right endpoint based on whether `image_url` was passed:
-
-```python
-FAMILIES = {
-    "veo3.1": {
-        "text_endpoint": "fal-ai/veo3.1",
-        "image_endpoint": "fal-ai/veo3.1/image-to-video",
-        # ... family-specific capability flags ...
-    },
-}
-
-def generate(self, prompt, *, image_url=None, model=None, **kwargs):
-    family_id, family = _resolve_family(model)
-    endpoint = family["image_endpoint"] if image_url else family["text_endpoint"]
-    # ... build payload from family's declared capability flags, call endpoint ...
-```
-
-The user picks `veo3.1` once in `hermes tools`. The agent never thinks about endpoints — it just passes (or doesn't pass) `image_url`.
-
-## Selection precedence
-
-For per-instance model knobs (see `plugins/video_gen/fal/__init__.py`):
-
-1. `model=` keyword from the tool call
-2. `<PROVIDER>_VIDEO_MODEL` env var
-3. `video_gen.<provider>.model` in `config.yaml`
-4. `video_gen.model` in `config.yaml` (when it's one of your IDs)
-5. Provider's `default_model()`
-
-## Response shape
-
-`success_response()` and `error_response()` produce the dict shape every backend returns. Use them — don't hand-roll the dict.
-
-Success keys: `success`, `video` (URL or absolute path), `model`, `prompt`, `modality` (`"text"` or `"image"`), `aspect_ratio`, `duration`, `provider`, plus `extra`.
-
-Error keys: `success`, `video` (None), `error`, `error_type`, `model`, `prompt`, `aspect_ratio`, `provider`.
-
-## Where to save artifacts
-
-If your backend returns base64, use `save_b64_video()` to write under `$HERMES_HOME/cache/videos/`. For raw bytes from a follow-up HTTP fetch, use `save_bytes_video()`. Otherwise return the upstream URL directly — the gateway resolves remote URLs on delivery.
-
-## Testing
-
-Drop a smoke test under `tests/plugins/video_gen/test_<name>_plugin.py`. The xAI and FAL tests show the pattern — register, verify catalog, exercise routing both with and without `image_url`, assert clean error responses on missing auth.