@clawpump/claw-agent 0.1.5 → 0.1.6

package/agent/docs/middleware/README.md

@@ -0,0 +1,260 @@
+# Hermes Middleware
+
+Hermes middleware is the behavior-changing companion to observer hooks.
+Observer hooks report what happened. Middleware can change what happens by
+rewriting a request before execution or by wrapping the execution callback
+itself.
+
+This contract is intentionally backend-neutral. A plugin can use it for local
+policy, request shaping, tracing, adaptive routing, cache control, sandbox
+selection, or handoff to runtimes such as NeMo Relay without changing Hermes'
+planner, model provider adapters, tool registry, memory, or CLI UX.
+
+With middleware enabled, plugins can:
+
+- Rewrite LLM provider request kwargs before Hermes calls the provider.
+- Rewrite tool arguments before guardrails, approval checks, hooks, and tool
+  execution see them.
+- Wrap the actual LLM execution callback while preserving Hermes retry,
+  streaming, interrupt, and hook behavior.
+- Wrap the actual tool execution callback while preserving Hermes guardrails,
+  approval, post-tool hooks, and tool-result transformation.
+
+## Contract
+
+Plugins register middleware from `register(ctx)`:
+
+```python
+def register(ctx):
+    ctx.register_middleware("llm_request", on_llm_request)
+    ctx.register_middleware("llm_execution", on_llm_execution)
+    ctx.register_middleware("tool_request", on_tool_request)
+    ctx.register_middleware("tool_execution", on_tool_execution)
+```
+
+Every middleware callback receives:
+
+- `telemetry_schema_version`: currently `hermes.observer.v1`
+- `middleware_schema_version`: currently `hermes.middleware.v1`
+- Runtime context such as `session_id`, `task_id`, `turn_id`,
+  `api_request_id`, `provider`, `model`, `api_mode`, `tool_name`, and
+  `tool_call_id` when applicable.
+
+Supported middleware kinds:
+
+| Kind | Payload | Return shape | Purpose |
+| --- | --- | --- | --- |
+| `llm_request` | `request`, `original_request` | `{"request": {...}}` | Replace effective provider kwargs before provider execution. |
+| `tool_request` | `tool_name`, `args`, `original_args` | `{"args": {...}}` | Replace effective tool args before hooks, guardrails, approvals, and execution. |
+| `llm_execution` | `request`, `original_request`, `next_call` | Any provider response | Wrap or replace the actual provider call. |
+| `tool_execution` | `tool_name`, `args`, `original_args`, `next_call` | Any tool result | Wrap or replace the actual tool call. |
+
+Request middleware can return optional trace fields:
+
+```python
+return {
+    "request": updated_request,
+    "source": "my-plugin",
+    "reason": "selected fallback model",
+}
+```
+
+Hermes stores those trace entries in later observer hook payloads as
+`middleware_trace`.
+
+Execution middleware receives a `next_call` callback. Call it to continue the
+chain:
+
+```python
+def on_tool_execution(**kwargs):
+    result = kwargs["next_call"](kwargs["args"])
+    return result
+```
+
+If multiple plugins register the same execution middleware kind, Hermes runs
+them as a nested chain in registration order. Middleware failures are fail-open:
+Hermes logs a warning and continues with the next middleware or the base
+runtime path.
+
+## Execution Order
+
+### LLM Calls
+
+For each provider request, Hermes applies middleware in this order:
+
+1. Build provider kwargs from the current conversation.
+2. Apply `llm_request` middleware.
+3. Emit `pre_api_request` observer hooks with the effective request.
+4. Run provider execution through `llm_execution` middleware.
+5. Emit `post_api_request` or `api_request_error` observer hooks.
+
+Request middleware sees the full provider kwargs, including `messages` or
+Responses API `input`, model settings, tool definitions, stream options, and
+provider-specific options. Execution middleware receives the same effective
+request plus `next_call`.
+
+### Tool Calls
+
+For each tool call, Hermes applies middleware in this order:
+
+1. Parse and coerce model-provided tool arguments.
+2. Apply `tool_request` middleware.
+3. Run the normal Hermes pre-execution path against the effective arguments:
+   tool availability checks, observer block directives, guardrails, and
+   approval checks.
+4. Run tool execution through `tool_execution` middleware.
+5. Emit `post_tool_call` observer hooks.
+6. Apply `transform_tool_result` hooks before the result is appended back into
+   conversation context.
+
+Tool request middleware runs before approval checks. Use it carefully: a
+rewritten path, command, or URL is the value downstream policy will evaluate.
+
+## Enablement
+
+Middleware only runs for enabled plugins. For a bundled plugin:
+
+```bash
+hermes plugins enable <plugin-name>
+```
+
+For isolated local testing, use one `HERMES_HOME` for plugin enablement and the
+agent run:
+
+```bash
+export HERMES_HOME=/tmp/hermes-middleware-test
+mkdir -p "$HERMES_HOME"
+hermes plugins enable <plugin-name>
+hermes chat --query 'Reply exactly ok'
+```
+
+For source checkouts, prefer the source command so the runtime sees plugins and
+middleware from the working tree:
+
+```bash
+uv sync
+uv run hermes plugins enable <plugin-name>
+uv run hermes chat --query 'Reply exactly ok'
+```
+
+## Generic Plugin Examples
+
+The examples below are intentionally small. They show the middleware contract
+shape without depending on NeMo Relay.
+
+### LLM Request Middleware
+
+This plugin tags provider requests and records a middleware trace entry:
+
+```python
+def register(ctx):
+    ctx.register_middleware("llm_request", tag_llm_request)
+
+
+def tag_llm_request(**kwargs):
+    request = dict(kwargs["request"])
+    extra_body = dict(request.get("extra_body") or {})
+    extra_body.setdefault("metadata", {})["hermes_middleware_demo"] = True
+    request["extra_body"] = extra_body
+    return {
+        "request": request,
+        "source": "middleware-demo",
+        "reason": "tagged provider request",
+    }
+```
+
+The effective request is passed to `pre_api_request`, provider execution, and
+`post_api_request`.
+
+### Tool Request Middleware
+
+This plugin constrains `terminal` calls to a known working directory:
+
+```python
+def register(ctx):
+    ctx.register_middleware("tool_request", normalize_terminal_workdir)
+
+
+def normalize_terminal_workdir(**kwargs):
+    if kwargs.get("tool_name") != "terminal":
+        return None
+    args = dict(kwargs["args"])
+    args.setdefault("workdir", "/tmp/hermes-middleware-demo")
+    return {
+        "args": args,
+        "source": "middleware-demo",
+        "reason": "defaulted terminal workdir",
+    }
+```
+
+Because this runs before hooks and approvals, downstream telemetry and policy
+observe the rewritten `workdir`.
+
+### LLM Execution Middleware
+
+This plugin wraps the provider call and preserves the raw provider response:
+
+```python
+import time
+
+
+def register(ctx):
+    ctx.register_middleware("llm_execution", time_llm_execution)
+
+
+def time_llm_execution(**kwargs):
+    started = time.monotonic()
+    response = kwargs["next_call"](kwargs["request"])
+    elapsed_ms = int((time.monotonic() - started) * 1000)
+    print(f"llm_execution elapsed_ms={elapsed_ms}")
+    return response
+```
+
+Return the same response shape Hermes expects from the provider adapter. Do not
+wrap the response in a plugin-specific envelope unless the rest of the runtime
+expects that envelope.
+
+### Tool Execution Middleware
+
+This plugin wraps tool execution while preserving the tool result:
+
+```python
+def register(ctx):
+    ctx.register_middleware("tool_execution", annotate_tool_execution)
+
+
+def annotate_tool_execution(**kwargs):
+    result = kwargs["next_call"](kwargs["args"])
+    # Metrics, logging, or external routing can happen here.
+    return result
+```
+
+Execution middleware may call `next_call(modified_args)` to pass a changed
+payload to later middleware and the base tool dispatcher.
+
+Plugin-specific examples should live with the plugin that owns the behavior.
+For NeMo Relay adaptive execution middleware, see
+[`plugins/observability/nemo_relay/README.md`](../../plugins/observability/nemo_relay/README.md).
+
+## Safety Notes
+
+- Middleware should be deterministic for the same input unless it is explicitly
+  routing to a dynamic external system.
+- Request middleware should return complete replacement payloads, not partial
+  patches.
+- Execution middleware should call `next_call(...)` exactly once unless it is
+  intentionally short-circuiting execution.
+- If execution middleware raises before calling `next_call(...)`, Hermes treats
+  that as middleware failure and continues with the remaining middleware chain
+  and base execution.
+- If execution middleware calls `next_call(...)` successfully and then raises
+  during post-processing, Hermes preserves the downstream result and does not
+  run the provider or tool a second time.
+- If downstream provider or tool execution fails, middleware may let that error
+  propagate or translate it deliberately. Hermes does not convert downstream
+  failure into a successful `None` result.
+- Tool request middleware runs before approvals. If it mutates file paths,
+  commands, URLs, or arguments, the mutated values are what guardrails and
+  approvals evaluate.
+- Observer hooks remain the right place for read-only telemetry. Use middleware
+  only when a plugin needs to alter or wrap behavior.

package/agent/docs/observability/README.md

@@ -0,0 +1,316 @@
+# Hermes Observer Hooks
+
+Hermes observer hooks are the read-only telemetry contract for plugins that
+need to reconstruct agent execution without changing runtime behavior. This
+contract supports trace, metrics, audit, replay, and export integrations such
+as Langfuse, OpenTelemetry-style collectors, and NeMo Relay.
+
+Observer hooks are intentionally backend-neutral. They expose stable lifecycle
+events, correlation IDs, sanitized payloads, timing, status, and error fields.
+They do not replace Hermes' planner, model providers, memory, tool registry,
+approval UX, CLI, gateway behavior, or execution semantics.
+
+Behavior-changing request or execution wrappers are outside this observer
+contract. Observer hooks should report what happened; they should not replace
+provider requests, tool arguments, or execution callbacks.
+
+## Contract
+
+Plugins register observer callbacks from `register(ctx)`:
+
+```python
+def register(ctx):
+    ctx.register_hook("pre_api_request", on_pre_api_request)
+    ctx.register_hook("post_api_request", on_post_api_request)
+    ctx.register_hook("pre_tool_call", on_pre_tool_call)
+    ctx.register_hook("post_tool_call", on_post_tool_call)
+```
+
+Every hook callback receives keyword arguments. Plugins should accept
+`**kwargs` so additive fields remain backward-compatible:
+
+```python
+def on_post_tool_call(**kwargs):
+    tool_name = kwargs.get("tool_name")
+    status = kwargs.get("status")
+    result = kwargs.get("result")
+```
+
+The plugin manager injects this field into every hook payload:
+
+```text
+telemetry_schema_version = "hermes.observer.v1"
+```
+
+Hook callbacks are fail-open. Hermes catches callback exceptions, logs a
+warning, and keeps the agent loop running.
+
+Most observer hook return values are ignored. The exceptions are older
+behavior-affecting hooks:
+
+| Hook | Return behavior |
+| --- | --- |
+| `pre_llm_call` | May return a string or `{"context": "..."}` to inject ephemeral context into the current user message. |
+| `pre_tool_call` | May return `{"action": "block", "message": "..."}` to block a tool before execution. |
+| `transform_tool_result` | May return a replacement tool result string after `post_tool_call`. |
+| `transform_llm_output` | May return a replacement final assistant text string. |
+
+Telemetry plugins should treat these behavior-affecting returns as optional
+compatibility features, not as observability requirements.
+
+## Correlation IDs
+
+Observer payloads use stable IDs so plugins can join events without relying on
+callback order alone.
+
+| Field | Meaning |
+| --- | --- |
+| `session_id` | Conversation/session identity. |
+| `task_id` | Task identity, especially useful for subagents and isolated execution. |
+| `turn_id` | User-turn identity shared by API attempts and tool calls in a turn. |
+| `api_request_id` | Opaque provider-attempt identity. Do not parse its string format. |
+| `api_call_count` | Numeric API attempt count within the agent loop. |
+| `tool_call_id` | Provider-supplied tool call ID when available. |
+| `parent_session_id` / `child_session_id` | Session link for delegated subagents. |
+| `parent_subagent_id` / `child_subagent_id` | Subagent link when available. |
+| `parent_turn_id` | Parent turn that spawned delegated work. |
+
+Consumers should prefer explicit fields over parsing compound IDs. In
+particular, `api_request_id` is an opaque correlation value.
+
+## Event Families
+
+### Session Lifecycle
+
+Session hooks describe conversation boundaries and resets:
+
+| Hook | When it fires |
+| --- | --- |
+| `on_session_start` | A brand-new session starts after the system prompt is built. |
+| `on_session_end` | A `run_conversation` call ends, including interrupted or incomplete turns. |
+| `on_session_finalize` | CLI or gateway tears down an active session identity. |
+| `on_session_reset` | CLI or gateway moves from an old session identity to a new one. |
+
+Common fields include `session_id`, `completed`, `interrupted`, `reason`,
+`old_session_id`, and `new_session_id` where available.
+
+`on_session_end` is turn/run scoped. It is not necessarily the final lifetime
+boundary for a chat identity. Use `on_session_finalize` and `on_session_reset`
+for lifecycle cleanup that must happen once per session identity.
+
+### Turn-Scoped LLM Hooks
+
+These hooks frame the user turn, not individual provider API attempts:
+
+| Hook | When it fires |
+| --- | --- |
+| `pre_llm_call` | Before the tool loop begins for a user turn. |
+| `post_llm_call` | After the turn completes with final assistant output. |
+
+Common `pre_llm_call` fields include `session_id`, `turn_id`,
+`user_message`, `conversation_history`, `is_first_turn`, `model`, `platform`,
+and `sender_id`.
+
+Common `post_llm_call` fields include `session_id`, `turn_id`,
+`user_message`, `assistant_response`, `conversation_history`, `model`, and
+`platform`.
+
+Use request-scoped API hooks for LLM span telemetry. Use `pre_llm_call` and
+`post_llm_call` for turn-level context, compatibility, and final turn summary.
+
+### Request-Scoped API Hooks
+
+API hooks describe provider attempts inside the agent loop:
+
+| Hook | When it fires |
+| --- | --- |
+| `pre_api_request` | Immediately before a provider API request. |
+| `post_api_request` | After a successful provider response. |
+| `api_request_error` | After a failed provider request or retryable error path. |
+
+`pre_api_request` includes:
+
+- identity: `session_id`, `task_id`, `turn_id`, `api_request_id`
+- runtime: `platform`, `model`, `provider`, `base_url`, `api_mode`
+- attempt metadata: `api_call_count`, `message_count`, `tool_count`,
+  `approx_input_tokens`, `request_char_count`, `max_tokens`
+- timing: `started_at`
+- sanitized request payload: `request`
+
+`post_api_request` includes the same identity/runtime fields plus:
+
+- `api_duration`, `started_at`, `ended_at`
+- `finish_reason`, `message_count`, `response_model`
+- `usage`
+- `assistant_content_chars`, `assistant_tool_call_count`
+- sanitized response payload: `response`
+- compatibility object: `assistant_message`
+
+`api_request_error` includes the same identity/runtime fields plus:
+
+- `api_duration`, `started_at`, `ended_at`
+- `status_code`, `retry_count`, `max_retries`, `retryable`, `reason`
+- structured `error = {"type": ..., "message": ...}`
+- sanitized failed request payload: `request`
+
+The sanitized `request`, `response`, and `error` fields are the canonical
+observer inputs for new consumers.
+
+### Tool Lifecycle
+
+Tool hooks describe individual tool calls:
+
+| Hook | When it fires |
+| --- | --- |
+| `pre_tool_call` | Before guardrail-approved tool dispatch. |
+| `post_tool_call` | After tool dispatch, cancellation, block, or error completion. |
+| `transform_tool_result` | After `post_tool_call`, before the result is appended to model context. |
+
+`pre_tool_call` includes `tool_name`, `args`, `task_id`, `session_id`,
+`tool_call_id`, `turn_id`, and `api_request_id`.
+
+`post_tool_call` includes the same identity fields plus `result`,
+`duration_ms`, `status`, `error_type`, and `error_message`.
+
+`status` is the observer-grade lifecycle outcome. Common values include:
+
+| Status | Meaning |
+| --- | --- |
+| `ok` | Tool completed normally. |
+| `error` | Tool ran and returned or raised an error outcome. |
+| `blocked` | A `pre_tool_call` hook blocked execution. |
+| `cancelled` | Execution was cancelled before normal completion. |
+
+`post_tool_call` is emitted for blocked and cancelled paths so telemetry
+plugins can close spans cleanly.
+
+### Approval Lifecycle
+
+Approval hooks describe dangerous-command approval prompts:
+
+| Hook | When it fires |
+| --- | --- |
+| `pre_approval_request` | Before the approval request is shown or sent. |
+| `post_approval_response` | After the user responds or the request times out. |
+
+Common fields include `command`, `description`, `pattern_key`,
+`pattern_keys`, `session_key`, and `surface`.
+
+`post_approval_response` also includes `choice`, with values such as `once`,
+`session`, `always`, `deny`, and `timeout`.
+
+Approval hooks are observer-only. Plugins cannot pre-answer or veto approvals
+from these hooks. To prevent a tool from reaching approval, use
+`pre_tool_call` blocking.
+
+### Subagent Lifecycle
+
+Subagent hooks describe delegated child-agent work:
+
+| Hook | When it fires |
+| --- | --- |
+| `subagent_start` | A delegated child agent is created. |
+| `subagent_stop` | A delegated child agent returns or fails. |
+
+`subagent_start` fields include `parent_session_id`, `parent_turn_id`,
+`parent_subagent_id`, `child_session_id`, `child_subagent_id`, `child_role`,
+and `child_goal`.
+
+`subagent_stop` fields include parent/child session IDs, role/status fields,
+`child_summary`, and `duration_ms`.
+
+Observers can use these hooks to model nested trajectories while keeping child
+agent execution linked to the parent turn that spawned it.
+
+## Payload Safety
+
+Observer payloads are designed for telemetry consumers, not raw object access.
+New consumers should use the sanitized API payloads:
+
+- `pre_api_request.request`
+- `post_api_request.response`
+- `api_request_error.request`
+- `api_request_error.error`
+
+Sanitization converts provider objects to JSON-compatible structures, bounds
+large payloads, redacts sensitive keys, and avoids exposing raw response
+objects in sanitized fields.
+
+Legacy compatibility fields such as `request_messages`, `conversation_history`,
+and `assistant_message` may still be present for existing plugins. New
+observability consumers should prefer the sanitized payloads.
+
+## Performance
+
+The default uninstrumented path should stay cheap. Expensive request/response
+payload construction is gated behind `has_hook(...)`, so Hermes only builds
+sanitized API telemetry payloads when at least one plugin registered the
+relevant hook.
+
+Plugin authors should preserve this property:
+
+- Register only hooks the plugin actually consumes.
+- Avoid deep-copying or re-sanitizing already sanitized payloads.
+- Keep hook callbacks fast and fail-open.
+- Offload network export or batch writes when practical.
+
+## Writing An Observer Plugin
+
+Minimal observer plugin:
+
+```python
+def register(ctx):
+    ctx.register_hook("pre_api_request", on_pre_api_request)
+    ctx.register_hook("post_api_request", on_post_api_request)
+    ctx.register_hook("pre_tool_call", on_pre_tool_call)
+    ctx.register_hook("post_tool_call", on_post_tool_call)
+
+
+def on_pre_api_request(**kwargs):
+    start_llm_span(
+        request_id=kwargs.get("api_request_id"),
+        turn_id=kwargs.get("turn_id"),
+        request=kwargs.get("request"),
+        model=kwargs.get("model"),
+    )
+
+
+def on_post_api_request(**kwargs):
+    finish_llm_span(
+        request_id=kwargs.get("api_request_id"),
+        response=kwargs.get("response"),
+        usage=kwargs.get("usage"),
+        duration=kwargs.get("api_duration"),
+    )
+
+
+def on_pre_tool_call(**kwargs):
+    start_tool_span(
+        call_id=kwargs.get("tool_call_id"),
+        name=kwargs.get("tool_name"),
+        args=kwargs.get("args"),
+    )
+
+
+def on_post_tool_call(**kwargs):
+    finish_tool_span(
+        call_id=kwargs.get("tool_call_id"),
+        result=kwargs.get("result"),
+        status=kwargs.get("status"),
+        duration_ms=kwargs.get("duration_ms"),
+    )
+```
+
+Use `session_id`, `turn_id`, `api_request_id`, and `tool_call_id` for span
+correlation. Use subagent and approval hooks when the export format supports
+nested agent work or security lifecycle events.
+
+## Existing Consumers
+
+The bundled Langfuse plugin demonstrates direct hook-based observability for
+turns, provider requests, and tool calls.
+
+The bundled NeMo Relay plugin maps the same generic observer contract to NeMo
+Relay scopes, LLM spans, tool spans, marks, ATOF streams, and ATIF exports.
+NeMo Relay-specific configuration and examples live in
+[`plugins/observability/nemo_relay/README.md`](../../plugins/observability/nemo_relay/README.md).