better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/plugin-llm-access.md

@@ -0,0 +1,506 @@
+---
+sidebar_position: 11
+title: "Plugin LLM Access"
+description: "Run any LLM call from inside a plugin via ctx.llm — chat or structured, sync or async. Host-owned auth, fail-closed trust gate, optional JSON Schema validation."
+---
+
+# Plugin LLM Access
+
+`ctx.llm` is the supported way for a plugin to make an LLM call.
+Chat completion, structured extraction, sync, async, with or without
+images — same surface, same trust gate, same host-owned credentials.
+
+Plugins reach for this when they need to do something that involves
+the model but isn't part of the agent's conversation. A hook that
+rewrites a tool error into something a non-engineer can read. A
+gateway adapter that translates an inbound message before queuing
+it. A slash command that summarises a long paste. A scheduled job
+that scores yesterday's activity and writes one line to a status
+board. A pre-filter that decides whether a message is worth waking
+the agent up for at all.
+
+These are jobs the agent shouldn't be in the loop on. They want one
+LLM call, a typed answer, and to be done.
+
+## The smallest possible call
+
+```python
+result = ctx.llm.complete(messages=[{"role": "user", "content": "ping"}])
+return result.text
+```
+
+That's the whole API in one line. No keys, no provider config, no
+SDK initialisation. The plugin runs against whatever provider and
+model the user is currently using — when they switch providers, the
+plugin follows them automatically.
+
+## A more complete chat example
+
+```python
+result = ctx.llm.complete(
+    messages=[
+        {"role": "system", "content": "Rewrite errors as one short sentence a non-engineer can act on."},
+        {"role": "user",   "content": traceback_text},
+    ],
+    max_tokens=64,
+    purpose="hooks.error-rewrite",
+)
+return result.text
+```
+
+`purpose` is a free-form audit string — it shows up in `agent.log`
+and in `result.audit` so operators can see which plugin made which
+call. Optional but recommended for anything that fires often.
+
+## Structured output
+
+When the plugin needs a typed answer, switch to the structured lane:
+
+```python
+result = ctx.llm.complete_structured(
+    instructions="Score this support reply for urgency (0–1) and pick a category.",
+    input=[{"type": "text", "text": message_body}],
+    json_schema=TRIAGE_SCHEMA,
+    purpose="support.triage",
+    temperature=0.0,
+    max_tokens=128,
+)
+
+if result.parsed["urgency"] > 0.8:
+    await dispatch_to_oncall(result.parsed["category"], message_body)
+```
+
+The host requests JSON output from the provider, parses it locally
+as a fallback, validates against your schema if `jsonschema` is
+installed, and hands back a Python object on `result.parsed`. If the
+model couldn't produce valid JSON, `result.parsed` is `None` and
+`result.text` carries the raw response.
+
+## What this lane gives you
+
+* **One call, four shapes.** `complete()` for chat,
+  `complete_structured()` for typed JSON, `acomplete()` and
+  `acomplete_structured()` for asyncio. Same arguments, same result
+  objects.
+* **Host-owned credentials.** OAuth tokens, refresh flows, the
+  credential pool, per-task aux overrides — every credential
+  concept Hermes already has applies. The plugin never sees a
+  token; the host attributes the call back through `result.audit`.
+* **Bounded.** Single sync or async call. No streaming, no tool
+  loops, no conversation state to manage. State the input, get the
+  result, return.
+* **Fail-closed trust.** A plugin you've never configured cannot
+  pick its own provider, model, agent, or stored credential. The
+  default posture is "use what the user is using." Operators opt in
+  to specific overrides, per plugin, in `config.yaml`.
+
+## Quick start
+
+Two complete plugins below — one chat, one structured. Both ship
+inside a single `register(ctx)` function and need zero outside
+configuration to run against whatever model the user has active.
+
+### Chat completion — `/tldr`
+
+```python
+def register(ctx):
+    ctx.register_command(
+        name="tldr",
+        handler=lambda raw: _tldr(ctx, raw),
+        description="Summarise the supplied text in one paragraph.",
+        args_hint="<text>",
+    )
+
+
+def _tldr(ctx, raw_args: str) -> str:
+    text = raw_args.strip()
+    if not text:
+        return "Usage: /tldr <text to summarise>"
+    result = ctx.llm.complete(
+        messages=[
+            {"role": "system",
+             "content": "Summarise the user's text in one tight paragraph. No preamble."},
+            {"role": "user", "content": text},
+        ],
+        max_tokens=256,
+        temperature=0.3,
+        purpose="tldr",
+    )
+    return result.text
+```
+
+`result.text` is the model's response; `result.usage` carries token
+counts; `result.provider` and `result.model` carry attribution.
+
+### Structured extraction — `/paste-to-tasks`
+
+```python
+def register(ctx):
+    ctx.register_command(
+        name="paste-to-tasks",
+        handler=lambda raw: _paste_to_tasks(ctx, raw),
+        description="Turn freeform meeting notes into structured tasks.",
+        args_hint="<text>",
+    )
+
+
+_TASKS_SCHEMA = {
+    "type": "object",
+    "properties": {
+        "tasks": {
+            "type": "array",
+            "items": {
+                "type": "object",
+                "properties": {
+                    "owner":  {"type": "string"},
+                    "action": {"type": "string"},
+                    "due":    {"type": "string", "description": "ISO date or empty"},
+                },
+                "required": ["action"],
+            },
+        },
+    },
+    "required": ["tasks"],
+}
+
+
+def _paste_to_tasks(ctx, raw_args: str) -> str:
+    if not raw_args.strip():
+        return "Usage: /paste-to-tasks <meeting notes>"
+    result = ctx.llm.complete_structured(
+        instructions=(
+            "Extract concrete action items from these meeting notes. "
+            "One task per actionable line. If no owner is named, leave 'owner' blank."
+        ),
+        input=[{"type": "text", "text": raw_args}],
+        json_schema=_TASKS_SCHEMA,
+        schema_name="meeting.tasks",
+        purpose="paste-to-tasks",
+        temperature=0.0,
+        max_tokens=512,
+    )
+    if result.parsed is None:
+        return f"Couldn't parse a response. Raw output:\n{result.text}"
+    lines = [f"- [{t.get('owner') or '?'}] {t['action']}" for t in result.parsed["tasks"]]
+    return "\n".join(lines) or "(no tasks found)"
+```
+
+A third worked example, this time with image input, lives in the
+[`hermes-example-plugins`](https://github.com/NousResearch/hermes-example-plugins/tree/main/plugin-llm-example)
+repo (companion repo for reference plugins — not bundled with
+hermes-agent itself). For the async surface (`acomplete()` /
+`acomplete_structured()` with `asyncio.gather()`), see
+[`plugin-llm-async-example`](https://github.com/NousResearch/hermes-example-plugins/tree/main/plugin-llm-async-example)
+in the same repo.
+
+## When to use which
+
+| You want… | Reach for |
+|---|---|
+| A free-form text response (translation, summary, rewrite, generation) | `complete()` |
+| A multi-turn prompt (system + few-shot examples + user) | `complete()` |
+| A typed dict back, validated against a schema | `complete_structured()` |
+| Image-or-text input with a typed dict back | `complete_structured()` |
+| The same call from async code (gateway adapters, async hooks) | `acomplete()` / `acomplete_structured()` |
+
+Everything else — provider selection, model resolution, auth, fallback,
+timeout, vision routing — is the same across all four.
+
+## API surface
+
+`ctx.llm` is an instance of `agent.plugin_llm.PluginLlm`.
+
+### `complete()`
+
+```python
+result = ctx.llm.complete(
+    messages=[{"role": "user", "content": "Hi"}],
+    provider=None,         # optional, gated — Hermes provider id (e.g. "openrouter")
+    model=None,            # optional, gated — whatever string that provider expects
+    temperature=None,
+    max_tokens=None,
+    timeout=None,          # seconds
+    agent_id=None,         # optional, gated
+    profile=None,          # optional, gated — explicit auth-profile name
+    purpose="optional-audit-string",
+    task=None,             # optional — a plugin-registered auxiliary slot
+)
+# → PluginLlmCompleteResult(text, provider, model, agent_id, usage, audit)
+```
+
+Plain chat completion. `messages` is the standard OpenAI shape — a
+list of `{"role": "...", "content": "..."}` dicts. Multi-turn
+prompts (system + few-shot user/assistant pairs + final user) work
+exactly as they would with the OpenAI SDK.
+
+`provider=` and `model=` are independent and follow the same shape
+as the host's main config (`model.provider` + `model.model`). Set
+just `model=` to use the user's active provider with a different
+model on it. Set both to switch providers entirely. Either argument
+without operator opt-in raises `PluginLlmTrustError`.
+
+### `complete_structured()`
+
+```python
+result = ctx.llm.complete_structured(
+    instructions="What you want extracted.",
+    input=[
+        {"type": "text",  "text": "..."},
+        {"type": "image", "data": b"...", "mime_type": "image/png"},
+        {"type": "image", "url":  "https://..."},
+    ],
+    json_schema={...},     # optional — triggers parsed result + validation
+    json_mode=False,       # set True without a schema to ask for JSON anyway
+    schema_name=None,      # optional human-readable schema name
+    system_prompt=None,
+    provider=None,         # optional, gated
+    model=None,            # optional, gated
+    temperature=None,
+    max_tokens=None,
+    timeout=None,
+    agent_id=None,
+    profile=None,
+    purpose=None,
+    task=None,             # optional — a plugin-registered auxiliary slot
+)
+# → PluginLlmStructuredResult(text, provider, model, agent_id,
+#                             usage, parsed, content_type, audit)
+```
+
+Inputs are typed text or image blocks (raw bytes get base64 encoded
+as a `data:` URL automatically). When `json_schema` or
+`json_mode=True` is supplied, the host requests JSON output via
+`response_format`, parses it locally as a fallback, and validates
+against your schema if `jsonschema` is installed.
+
+* `result.content_type == "json"` — `result.parsed` is a Python
+  object that matches your schema.
+* `result.content_type == "text"` — parsing or validation failed;
+  inspect `result.text` for the raw model response.
+
+### Async
+
+```python
+result = await ctx.llm.acomplete(messages=..., task="classifier")
+result = await ctx.llm.acomplete_structured(
+    instructions=..., input=..., task="classifier"
+)
+```
+
+Same arguments and result types as their sync counterparts. Use
+these from gateway adapters, async hooks, or any plugin code
+already running on an asyncio loop.
+
+### Task-routed auxiliary calls
+
+Pass `task=` to any of the four call shapes when a plugin needs its
+own configured auxiliary route. Register that task during plugin setup;
+plugin defaults apply until the operator overrides its provider and model
+in `auxiliary.<task>`:
+
+```python
+def register(ctx):
+    ctx.register_auxiliary_task(
+        "classifier", display_name="Classifier", description="Classify input."
+    )
+
+
+result = ctx.llm.complete(messages=[...], task="classifier")
+result = ctx.llm.complete_structured(instructions=..., input=..., task="classifier")
+```
+
+```yaml
+auxiliary:
+  classifier:
+    provider: openrouter
+    model: vendor/model-id
+```
+
+Plugins may supply provider/model registration defaults for their own
+tasks. Operator configuration in `auxiliary.<task>` overrides those
+defaults and controls the deployment choice. A plugin can only use a task
+it registered itself; unknown or foreign task names fail before provider
+invocation. `allow_task_override: true` is an explicit operator grant for
+using Hermes built-in auxiliary tasks; it does not permit another plugin's
+tasks. Omit `task=` (or use `"auto"`) to keep the active main provider/model.
+
+### Result attributes
+
+```python
+@dataclass
+class PluginLlmCompleteResult:
+    text: str                    # the assistant's response
+    provider: str                # e.g. "openrouter", "anthropic"
+    model: str                   # whatever the provider returned for this call
+    agent_id: str                # whose model/auth was used
+    usage: PluginLlmUsage        # tokens + cache + cost estimate
+    audit: Dict[str, Any]        # plugin_id, purpose, profile
+
+@dataclass
+class PluginLlmStructuredResult:
+    # same fields as PluginLlmCompleteResult, plus:
+    parsed: Optional[Any]        # JSON object when content_type == "json"
+    content_type: str            # "json" or "text"
+    # audit also carries schema_name when supplied
+```
+
+`usage` carries `input_tokens`, `output_tokens`, `total_tokens`,
+`cache_read_tokens`, `cache_write_tokens`, and `cost_usd` when the
+provider returns those fields.
+
+## Trust gate
+
+The default behaviour is fail-closed. With no `plugins.entries`
+config block, a plugin can:
+
+* run any of the four methods against the user's active provider
+  and model,
+* set request-shaping arguments (`temperature`, `max_tokens`,
+  `timeout`, `system_prompt`, `purpose`, `messages`, `instructions`,
+  `input`, `json_schema`),
+
+…and that's it. `provider=`, `model=`, `agent_id=`, and `profile=`
+arguments raise `PluginLlmTrustError` until the operator opts in.
+Likewise, `task=` can use only the plugin's registered auxiliary task
+unless the operator grants `allow_task_override` for a built-in task.
+
+**Most plugins never need this section.** A plugin that just calls
+`ctx.llm.complete(messages=...)` with no overrides runs against
+whatever the user has active and works zero-config. The block below
+is only relevant when a plugin specifically wants to pin to a
+different model or provider than the user.
+
+```yaml
+plugins:
+  entries:
+    my-plugin:
+      llm:
+        # Allow this plugin to choose a different Hermes provider
+        # (must be one Hermes already knows about — same names as
+        # `hermes model` and config.yaml model.provider).
+        allow_provider_override: true
+
+        # Optionally restrict which providers. Use ["*"] for any.
+        allowed_providers:
+          - openrouter
+          - anthropic
+
+        # Allow this plugin to ask for a specific model.
+        allow_model_override: true
+
+        # Optionally restrict which models. Use ["*"] for any.
+        # Models are matched literally against whatever string the
+        # plugin sends — Hermes does not look anything up.
+        allowed_models:
+          - openai/gpt-4o-mini
+          - anthropic/claude-3-5-haiku
+
+        # Allow cross-agent calls (rare).
+        allow_agent_id_override: false
+
+        # Allow the plugin to request a specific stored auth profile
+        # (e.g. a different OAuth account on the same provider).
+        allow_profile_override: false
+```
+
+The plugin id is the manifest `name:` field for flat plugins, or the
+path-derived key for nested plugins (`image_gen/openai`,
+`memory/honcho`, etc.).
+
+### What the gate enforces
+
+| Override        | Default | Config key                       |
+| --------------- | ------- | -------------------------------- |
+| `provider=`     | denied  | `allow_provider_override: true`  |
+| ↳ allowlist     | —       | `allowed_providers: [...]`       |
+| `model=`        | denied  | `allow_model_override: true`     |
+| ↳ allowlist     | —       | `allowed_models: [...]`          |
+| `agent_id=`     | denied  | `allow_agent_id_override: true`  |
+| `profile=`      | denied  | `allow_profile_override: true`   |
+| built-in `task=` | denied  | `allow_task_override: true`      |
+
+Each override is independently gated. Granting `allow_model_override`
+does **not** also grant `allow_provider_override` — a plugin trusted
+to pick a model is still pinned to the user's active provider unless
+it gets the provider gate as well.
+
+### What the gate does NOT need to enforce
+
+* Request-shaping arguments — `temperature`, `max_tokens`,
+  `timeout`, `system_prompt`, `purpose`, `messages`, `instructions`,
+  `input`, `json_schema`, `schema_name`, `json_mode` — are always
+  allowed; they don't pick credentials or routes.
+* The default deny posture means an unconfigured plugin can still do
+  useful work — it just runs against the active provider and model.
+  Operators only need to think about `plugins.entries` for plugins
+  that want finer routing.
+
+## What the host owns
+
+A complete list of the things `ctx.llm` does for the plugin so you
+don't have to:
+
+* **Provider resolution.** Reads `model.provider` + `model.model`
+  from the user's config (or the explicit overrides when trusted).
+* **Auth.** Pulls API keys, OAuth tokens, or refresh tokens from
+  `~/.hermes/auth.json` / env, including the credential pool when
+  one is configured. The plugin never sees them.
+* **Vision routing.** When image input is supplied and the user's
+  active text model is text-only, the host falls back to the
+  configured vision model automatically.
+* **Fallback chain.** If the user's primary provider 5xxs or 429s,
+  the request goes through Hermes' usual aggregator-aware fallback
+  before it returns an error to the plugin.
+* **Timeout.** Honours your `timeout=` argument, falling back to
+  `auxiliary.<task>.timeout` config or the global aux default.
+* **JSON shaping.** Sends `response_format` to the provider when
+  you ask for JSON, then re-parses locally from a code-fenced
+  response if the provider returned one.
+* **Schema validation.** Validates against your `json_schema` when
+  `jsonschema` is installed; logs a debug line and skips strict
+  validation otherwise.
+* **Audit log.** Each call writes one INFO line to `agent.log` with
+  the plugin id, provider/model, purpose, and token totals.
+
+## What the plugin owns
+
+* **Request shape.** `messages` for chat, `instructions` + `input`
+  for structured. The plugin builds the prompt; the host runs it.
+* **Schema.** Whatever shape you want back. The host doesn't infer
+  it for you.
+* **Error handling.** `complete_structured()` raises `ValueError` on
+  empty inputs and on schema-validation failure. `PluginLlmTrustError`
+  fires when the trust gate denies an override. Anything else
+  (provider 5xx, no credentials configured, timeout) raises whatever
+  `auxiliary_client.call_llm()` raises.
+* **Cost.** Every call runs against the user's paid provider. Don't
+  loop on `complete()` for every gateway message without thinking
+  about token spend.
+
+## Where this fits in the plugin surface
+
+Existing `ctx.*` methods extend an existing Hermes subsystem:
+
+| `ctx.register_tool` | adds a tool the agent can call |
+| `ctx.register_platform` | wires a new gateway adapter |
+| `ctx.register_image_gen_provider` | replaces an image-gen backend |
+| `ctx.register_memory_provider` | replaces the memory backend |
+| `ctx.register_context_engine` | replaces the context compressor |
+| `ctx.register_hook` | observes a lifecycle event |
+
+`ctx.llm` is the first surface that lets a plugin run the same
+model the user is talking to, *out of band*, without any of the
+above. That's its only job. If your plugin needs to register a
+tool the agent invokes, use `register_tool`. If it needs to react
+to a lifecycle event, use `register_hook`. If it needs to make its
+own model call — for any reason, structured or not — `ctx.llm`.
+
+## Reference
+
+* Implementation: [`agent/plugin_llm.py`](https://github.com/NousResearch/hermes-agent/blob/main/agent/plugin_llm.py)
+* Tests: [`tests/agent/test_plugin_llm.py`](https://github.com/NousResearch/hermes-agent/blob/main/tests/agent/test_plugin_llm.py)
+* Reference plugins (companion repo):
+  * [`plugin-llm-example`](https://github.com/NousResearch/hermes-example-plugins/tree/main/plugin-llm-example) — sync structured extraction with image input
+  * [`plugin-llm-async-example`](https://github.com/NousResearch/hermes-example-plugins/tree/main/plugin-llm-async-example) — async with `asyncio.gather()`
+* Auxiliary client (the engine under the hood): see
+  [Provider Runtime](/developer-guide/provider-runtime).