better-dsh 0.2.4 → 0.2.5

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/micro-compaction.md

@@ -0,0 +1,406 @@
+---
+title: "Micro-compaction"
+description: "How micro-compaction amortizes context compression cost: triggers, tail sizing, the cached-prefix trade-off"
+---
+
+# Micro-compaction
+
+**A way to amortize the cost of compression.**
+
+Long conversations eventually outgrow the model's context window, and something
+has to be thrown away or summarized. Hermes has always done this in one batch:
+when the transcript crosses a threshold, the session stops, a large chunk of the
+middle is summarized in a single call, and the conversation resumes. That works,
+but the whole bill comes due at once — one visible pause, one big summarization
+request, at whatever moment you happened to cross the line.
+
+Micro-compaction pays the same bill in instalments. After each completed turn,
+Hermes folds the single oldest un-absorbed exchange into a running summary. The
+work is the same work; it just happens continuously, a piece at a time, instead
+of all at once in the middle of your session.
+
+It is not free and it is not a magic bullet, and it is **off by default** —
+`compression.micro_compact: true` turns it on. Each pass is a real call to the
+compression model, and it runs at the end of a turn — your answer has already
+streamed, but the turn does not close until the pass finishes. Each pass also
+rewrites already-sent history, which breaks the provider prompt-cache prefix
+every turn; read [Prompt caching](#prompt-caching--the-cost-you-are-opting-into)
+before enabling it, because for some setups that cost exceeds the benefit.
+
+What the feature gives you is a **tuning option**: you choose how the
+compression cost is distributed, and which model pays it. See
+[Choosing a compression model](#choosing-a-compression-model), because that
+choice matters more than anything else here.
+
+**The tradeoff is that knowledge gets a little earlier than you may be used to.**
+Because compaction is always running, older parts of the conversation become
+summaries sooner than they would under batch compaction — which leaves
+everything verbatim until the window actually fills. Detail from earlier in the
+session turns second-hand faster. You trade some of that fidelity for never
+eating one long stall, and for a context window that stays consistently smaller
+rather than sawtoothing up to the threshold and back.
+
+---
+
+## What it does
+
+After every turn that finishes normally, `finalize_turn` asks the context
+compressor to absorb **one** exchange:
+
+1. Find the oldest exchange that hasn't been summarized yet.
+2. Send just that exchange, plus the current running summary, to the auxiliary
+   summarization model.
+3. Replace those messages in the transcript with a single summary marker
+   carrying the updated running summary.
+
+One exchange per turn. The per-turn cost stays bounded no matter how long the
+conversation gets.
+
+An **exchange** is one full agent turn: an assistant message together with its
+tool results and any follow-up assistant iterations, up to the next user
+message. In tool-heavy work that's where the bulk of the tokens live — a
+file read or a command's output dwarfs the surrounding prose — which is why
+absorbing one exchange at a time is worth doing at all. Taking the whole turn
+(rather than a single assistant+tools group) also keeps the transcript's role
+alternation strictly valid: the summary marker is an assistant-role message,
+and a full turn is always bounded by user messages on both sides.
+
+## Your messages are never compacted
+
+An exchange deliberately starts at the *assistant* message. Micro-compaction
+walks straight past user messages to get there, so **what you typed is never
+summarized** — your prompts stay verbatim for the entire session, no matter how
+long it runs or how many times compaction fires.
+
+This is the most useful property of the whole design, and it's worth being
+explicit about why. What the assistant produces is largely an account of what it
+did: it read this file, it ran that command, it got this result. That kind of
+narration survives summarising with very little loss — "it did it this way" is
+about as informative compressed as it was in full. Your instructions are a
+different kind of thing. They're the intent everything else is derived from, and
+they cannot be reconstructed from the work that followed. Paraphrasing "use the
+existing retry helper, don't add a new one" into a summary is exactly how an
+agent ends up confidently doing the thing you told it not to, six turns later.
+
+So the asymmetry is on purpose: compact the derived material, keep the source of
+truth. The cost is a floor on how small the middle can get, since user turns
+accumulate and are never absorbed. In practice that floor is low — a prompt is
+normally a tiny fraction of what a single tool result costs — but it is a real
+floor. If you routinely paste 10–20K-token prompts, that weight stays in context
+by design.
+
+## What it never touches
+
+Two more regions are protected and stay verbatim:
+
+- **The head** — the system prompt and the opening messages, so the session's
+  founding instructions are never paraphrased.
+- **The tail** — a token-budgeted window of the most recent messages, so
+  everything that's immediately relevant is still there in full.
+
+Micro-compaction only ever works in the middle, between those two.
+
+## How it works
+
+### The cursor
+
+The compressor keeps a cursor: the index of the first message not yet absorbed.
+Each successful pass advances it past the exchange it just summarized.
+
+If that in-memory cursor is missing or out of range — a fresh process, a resumed
+session — it's recovered by scanning the transcript for the last summary marker
+and resuming just after it. The transcript itself is the source of truth, so
+resuming a session doesn't re-summarize work already done.
+
+### The rolling summary
+
+Rather than keeping a pile of per-exchange summaries, there is exactly one
+running summary that each new exchange is merged into. The summarizer is asked
+to fold in the new material's decisions, requirements, file paths and open
+questions, drop details that are no longer relevant, and preserve the existing
+structure. It's also explicitly instructed to replace any credentials it
+encounters with `[REDACTED]`.
+
+Because that summary is cumulative, only the newest marker is kept in the
+transcript. Earlier markers are strictly redundant — the current summary already
+contains everything they held — so they're dropped as they're superseded. This
+matters more than it sounds: leaving them in place stacks near-duplicate copies
+of the same text, each with its own heading and end-marker scaffolding, and the
+transcript grows on every turn instead of shrinking.
+
+### Defrag
+
+Merge into a summary often enough and it gets baggy — repetitive, and larger
+than the material justifies. When the running summary crosses a token threshold
+(2000 by default), the next pass **defrags**: one auxiliary call re-summarizes
+the running summary *itself* into a fresh compact version, and the summary
+marker in the transcript is rewritten in place.
+
+Defrag never touches the transcript's structure — no messages are absorbed or
+spliced, the cursor doesn't move, and user turns are untouched. It processes
+only the accumulated summary text, never conversation messages, so the
+"your messages are never compacted" guarantee holds through it.
+
+### Staying in step with the session database
+
+The in-memory splice alone isn't enough. Hermes's normal session flush is
+append-only, so the original rows would stay marked active and a resume would
+load *both* the summary and the messages it replaced — putting the session
+straight over the context limit.
+
+So each pass also calls `archive_and_compact`, which atomically soft-archives the
+active rows and inserts the compacted set. The messages are then stamped as
+already-persisted so the append-only flush that follows skips them. If that
+database step fails, it's logged and the session continues; the resume would
+double-load until the next batch compression cleans up.
+
+### When the summarizer fails
+
+A summarization call can fail — the auxiliary model is unreachable, out of quota,
+or the exchange itself is somehow unsummarizable. The transcript is left
+untouched and the failure is counted.
+
+If the *same* exchange fails three times in a row, the cursor is advanced past it
+anyway. Without that, one bad exchange would be retried on every single turn
+forever. Those skipped messages stay in the transcript and get picked up by the
+next batch compaction.
+
+## Interaction with batch compaction
+
+Micro-compaction doesn't replace batch compaction — it defers it. Threshold-based
+compaction is still there and still fires if the window fills anyway, and its
+summary markers are the same format, so the two interoperate. In practice
+micro-compaction keeps the transcript far enough below the threshold that the
+batch path fires much less often.
+
+## Configuration
+
+Micro-compaction is **off by default**. Turn it on explicitly:
+
+```yaml
+compression:
+  micro_compact: true             # default: false
+  micro_compact_every_n_turns: 1  # cadence — how often a pass runs
+  micro_compact_defrag_threshold_tokens: 2000
+```
+
+With `micro_compact` unset or `false` Hermes behaves exactly as it always has:
+batch-only compaction. Everything else about compression is unchanged.
+
+`micro_compact_every_n_turns` is the knob that matters most after the on/off
+switch, because it sets how often you pay the cache break described below. At
+`1` a pass runs after every completed turn: the most aggressive reclaim, and
+one broken prefix per turn. At `5` you get a fifth of the breaks and a fifth of
+the reclaim rate, which is the right direction if your sessions are long-lived
+and your provider's cache discount is deep. Values below `1` are clamped to `1`
+rather than silently disabling the feature. The counter advances per turn, not
+per committed pass, so a turn with nothing to absorb still moves the cadence
+along and cannot wedge it.
+
+`micro_compact_defrag_threshold_tokens` is when the rolling summary gets
+re-summarized instead of growing forever — see [Defrag](#defrag).
+
+It ships opt-in rather than on because of the prompt-cache cost described in
+the next section — that cost is real, it is not universally worth paying, and
+it should be a decision you make rather than one you inherit.
+
+## Prompt caching — the cost you are opting into
+
+Read this before enabling the feature. It is the strongest argument against it.
+
+A long-lived conversation reuses a cached prompt prefix every turn, and cached
+input tokens are billed at a fraction of uncached ones. That discount survives
+only as long as the prefix does not change. **A micro-compaction pass rewrites
+already-sent history**, which invalidates the prefix from the rewrite point
+onward — so with micro-compaction on, you break the cache *every turn* instead
+of once per batch compaction.
+
+This is the same cost the proactive prune deliberately avoids. That path gates
+itself behind `compression.proactive_prune_min_reclaim_tokens` (4096 by
+default) precisely so its rewrites stay, in the words of the config comment,
+"one big episodic break instead of a tiny break every tool iteration."
+
+Micro-compaction has no equivalent *reclaim-size* gate — a pass commits
+whatever the one absorbed exchange happened to save, large or small. What it
+has instead is a *frequency* dial, `micro_compact_every_n_turns`. Raising it
+makes the breaks rarer and more episodic, which is the same end the prune's
+gate serves by a different route, though it gets there by absorbing less rather
+than by waiting for a bigger win. If you want the prune's exact semantics here,
+a reclaim threshold on micro-compaction is the obvious follow-up and does not
+exist yet.
+
+So the honest framing is a trade of one cost for another, not a saving:
+
+| | Batch-only (default) | Micro-compaction on |
+|---|---|---|
+| Compression stalls | One long stall at the threshold | Spread across turns |
+| Context occupancy | Sawtooths up to the threshold | Stays low and flat |
+| Cache prefix | Intact between compactions | Broken every turn |
+
+Which side wins depends on numbers specific to you: how much your provider
+discounts cached input, how large your prefix is, how long your sessions run,
+and how much a mid-session stall actually costs you. On a provider with a deep
+cache discount and a big prefix, the per-turn invalidation can plausibly cost
+more than the stall it removes. Measure your own sessions — see
+[Measuring it](#measuring-it) — rather than assuming.
+
+## Choosing a compression model
+
+Micro-compaction uses the `auxiliary.compression` model:
+
+```yaml
+auxiliary:
+  compression:
+    provider: openai-api
+    model: <your choice>
+    base_url: <endpoint>
+```
+
+This is the single most important knob, and there is no universally right
+answer — it depends on your hardware and what you are willing to trade.
+
+Each pass sends the running summary plus one exchange, so the prompt is small
+(a few thousand tokens) but the call happens **every turn**, at the end of the
+turn. Two properties matter:
+
+- **Latency dominates.** Because a pass runs per turn, its wall-clock cost is
+  felt repeatedly. A model that takes 30 seconds turns every turn into a turn
+  plus 30 seconds.
+- **Reasoning models are a poor fit.** Merging one exchange into a summary is
+  mechanical work. A thinking model will spend reasoning tokens on it and be
+  substantially slower than a plain instruct model of similar size, for no
+  benefit to the output.
+
+Some measured points, on one particular setup — treat them as illustrations of
+the shape, not as recommendations:
+
+| model | observed |
+|---|---|
+| 7B 4-bit instruct, local (MLX, Apple Silicon) | ~31s per pass; box also serving other work |
+| large MoE reasoning model, remote GPU | noticeably slower still — thinking tokens on a summarisation task |
+
+The pattern is that a small, fast, non-reasoning instruct model is usually the
+right shape, and that a bigger or "smarter" model is often worse here rather
+than better. Where that lands for you depends on what you have to run it on.
+
+If passes feel too slow, your options in rough order of effect are: pick a
+faster or smaller compression model; give it a less contended host; or turn
+micro-compaction off and go back to batch compaction.
+
+## Measuring it
+
+Micro-compaction is not primarily a token-saving or time-saving optimisation,
+and judging it on tokens saved will undersell it. The two things it actually
+buys you are:
+
+1. **The long pause is amortized.** The same summarization work happens, but as
+   small increments after turns instead of one stall in the middle of a session.
+2. **Your context lasts longer.** Because the middle is continuously reclaimed,
+   occupancy stays low instead of sawtoothing up to the threshold. A session
+   runs much further — often indefinitely — before it needs a hard compaction
+   at all.
+
+So the number that matters is **occupancy**: how full the window is being kept,
+as a percentage of the compaction threshold. A session that holds steady around
+40% has headroom to keep going; one climbing through 90% is about to stall. The
+second number is **how many batch compactions actually fired** — ideally none.
+
+A session can save nothing on paper and still be a clear win on both counts.
+
+Every pass emits one content-free JSON line, in the same style as the batch
+compaction telemetry:
+
+```
+micro compaction telemetry: {"event":"micro_compaction","outcome":"absorbed",
+"tokens_before":12739,"tokens_after":12060,"tokens_delta":-679,
+"occupancy_pct":38.4,"threshold_tokens":34816,"context_limit":40960,
+"exchange_tokens":868,"rolling_summary_tokens":31,"passes_total":1,
+"tokens_saved_total":679,"duration_ms":14,...}
+```
+
+`occupancy_pct` is `tokens_after` as a share of the compaction threshold -- the
+headroom figure. It is null when the model's window has not been resolved yet:
+the telemetry reads only the cached value, because resolving it can issue a
+synchronous `/models` probe and telemetry must never be what blocks a turn.
+
+`tokens_delta` is negative when the pass shrank the transcript.
+`tokens_saved_total` and `passes_total` accumulate across the session, so a whole
+run can be summarised from its last line. No transcript content appears in the
+payload — only counts.
+
+To turn a log into an answer:
+
+```
+python scripts/micro_compaction_report.py [--per-session] [LOGFILE ...]
+```
+
+Defaults to `$HERMES_HOME/logs/agent.log`. It reports passes, outcome mix, net
+tokens saved, mean absorbed-exchange size and pass durations.
+
+### What it looks like when it is working
+
+One real session — a 3.5 hour whole-project code review, ~75K tokens of
+transcript, 400K window, compaction threshold at 320K:
+
+| pass | messages | tokens | delta | occupancy | duration |
+|---|---|---|---|---|---|
+| 1 | 40 -> 39 | 27,479 -> 27,778 | +299 | 8.7% | 2.2s |
+| 2 | 61 -> 59 | 48,676 -> 48,128 | -548 | 15.0% | 4.5s |
+| 3 | 70 -> 67 | 58,309 -> 55,915 | -2,394 | 17.5% | 9.1s |
+| 4 | 84 -> 80 | 75,251 -> 69,818 | -5,433 | 21.8% | 36.2s |
+| 5 | 84 -> 80 | 74,659 -> 70,264 | -4,395 | 22.0% | 31.2s |
+
+Three things to read off it.
+
+**Occupancy flattened.** It climbed to about 22% and stopped. The last two
+passes are identical (84 -> 80 messages); between them the conversation added
+4,841 tokens and micro-compaction reclaimed 4,395. That is equilibrium: the
+window holds steady instead of marching toward the threshold.
+
+**No batch compaction fired.** Across the whole session the long pause never
+happened.
+
+**Reclamation only ramps after the tail budget.** The first passes recovered
+almost nothing, because below the tail budget (here 64,000 tokens, 16% of the
+window) nearly the whole transcript is protected tail and there is very little
+that may be touched. Early sessions legitimately show no passes at all.
+
+And the cost, stated plainly: passes ran 2 to 37 seconds, median around 31, on
+a small local model that was also serving other work. Roughly two minutes of
+summarisation spread across three and a half hours. Against one batch
+compaction of a 75K-token middle that is still the better trade, but a
+37-second increment is not a rounding error. See
+[Choosing a compression model](#choosing-a-compression-model).
+
+### Reading the numbers honestly
+
+**The first pass in a session usually costs tokens rather than saving them.**
+Inserting the summary marker carries a fixed ~400 tokens of scaffolding — the
+compaction preamble, the historical heading, the end marker — and on pass one
+that is paid against a single absorbed exchange. A first pass showing
+`tokens_delta: +330` is not a malfunction.
+
+From the second pass on, the marker is *replaced* rather than added, so the
+scaffolding is already paid for and each absorbed exchange is close to pure
+saving. The break-even is normally the second or third pass. This is why the
+per-session view matters more than any single line: judge the feature on a
+session's trajectory, not on one turn.
+
+The plainer human-readable lines are still there too:
+
+```
+Micro-compaction: 37 -> 36 messages
+Micro-compaction defrag: rolling summary re-summarized (1843 chars)
+Micro-compaction: skipping exchange at cursor 12 after 3 consecutive failures
+```
+
+Message counts move by small amounts — that's expected. The token count is where
+the effect shows: absorbing one tool-heavy exchange can drop hundreds of tokens
+while changing the message count by one or two.
+
+## Failure behaviour
+
+Micro-compaction is best-effort throughout. The call in `finalize_turn` is wrapped
+so that any exception is logged and swallowed — a failure returns the conversation
+unchanged and the turn completes normally. It can degrade, but it shouldn't be
+able to break a session.

package/docs/80_upstream-docs-and-skills/hermes-dev-skill/src/docs/developer-guide/middleware.md

@@ -0,0 +1,266 @@
+---
+title: "Middleware"
+description: "Behavior-changing plugin middleware for LLM and tool calls: contract, execution order, examples"
+---
+
+# Hermes Middleware
+
+Hermes middleware is the behavior-changing companion to observer hooks.
+Observer hooks report what happened. Middleware can change what happens by
+rewriting a request before execution or by wrapping the execution callback
+itself.
+
+This contract is intentionally backend-neutral. A plugin can use it for local
+policy, request shaping, tracing, adaptive routing, cache control, sandbox
+selection, or handoff to runtimes such as NeMo Relay without changing Hermes'
+planner, model provider adapters, tool registry, memory, or CLI UX.
+
+With middleware enabled, plugins can:
+
+- Rewrite LLM provider request kwargs before Hermes calls the provider.
+- Rewrite tool arguments before guardrails, approval checks, hooks, and tool
+  execution see them.
+- Wrap the actual LLM execution callback while preserving Hermes retry,
+  streaming, interrupt, and hook behavior.
+- Wrap the actual tool execution callback while preserving Hermes guardrails,
+  approval, post-tool hooks, and tool-result transformation.
+
+## Contract
+
+Plugins register middleware from `register(ctx)`:
+
+```python
+def register(ctx):
+    ctx.register_middleware("llm_request", on_llm_request)
+    ctx.register_middleware("llm_execution", on_llm_execution)
+    ctx.register_middleware("tool_request", on_tool_request)
+    ctx.register_middleware("tool_execution", on_tool_execution)
+```
+
+Every middleware callback receives:
+
+- `telemetry_schema_version`: currently `hermes.observer.v1`
+- `middleware_schema_version`: currently `hermes.middleware.v1`
+- Runtime context such as `session_id`, `task_id`, `turn_id`,
+  `api_request_id`, `provider`, `model`, `api_mode`, `tool_name`, and
+  `tool_call_id` when applicable.
+
+Supported middleware kinds:
+
+| Kind | Payload | Return shape | Purpose |
+| --- | --- | --- | --- |
+| `llm_request` | `request`, `original_request` | `{"request": {...}}` | Replace effective provider kwargs before provider execution. |
+| `tool_request` | `tool_name`, `args`, `original_args` | `{"args": {...}}` | Replace effective tool args before hooks, guardrails, approvals, and execution. |
+| `llm_execution` | `request`, `original_request`, `next_call` | Any provider response | Wrap or replace the actual provider call. |
+| `tool_execution` | `tool_name`, `args`, `original_args`, `next_call` | Any tool result | Wrap or replace the actual tool call. |
+
+Request middleware can return optional trace fields:
+
+```python
+return {
+    "request": updated_request,
+    "source": "my-plugin",
+    "reason": "selected fallback model",
+}
+```
+
+Hermes stores those trace entries in later observer hook payloads as
+`middleware_trace`.
+
+Execution middleware receives a `next_call` callback. Call it to continue the
+chain:
+
+```python
+def on_tool_execution(**kwargs):
+    result = kwargs["next_call"](kwargs["args"])
+    return result
+```
+
+If multiple plugins register the same execution middleware kind, Hermes runs
+them as a nested chain in registration order. Middleware failures are fail-open:
+Hermes logs a warning and continues with the next middleware or the base
+runtime path.
+
+## Execution Order
+
+### LLM Calls
+
+For each provider request, Hermes applies middleware in this order:
+
+1. Build provider kwargs from the current conversation.
+2. Apply `llm_request` middleware.
+3. Emit `pre_api_request` observer hooks with the effective request.
+4. Run provider execution through `llm_execution` middleware.
+5. Emit `post_api_request` or `api_request_error` observer hooks.
+
+Request middleware sees the full provider kwargs, including `messages` or
+Responses API `input`, model settings, tool definitions, stream options, and
+provider-specific options. Execution middleware receives the same effective
+request plus `next_call`.
+
+### Tool Calls
+
+For each tool call, Hermes applies middleware in this order:
+
+1. Parse and coerce model-provided tool arguments.
+2. Apply `tool_request` middleware.
+3. Run the normal Hermes pre-execution path against the effective arguments:
+   tool availability checks, observer block directives, guardrails, and
+   approval checks.
+4. Run tool execution through `tool_execution` middleware.
+5. Emit `post_tool_call` observer hooks.
+6. Apply `transform_tool_result` hooks before the result is appended back into
+   conversation context.
+
+Tool request middleware runs before approval checks. Use it carefully: a
+rewritten path, command, or URL is the value downstream policy will evaluate.
+
+## Enablement
+
+Middleware only runs for enabled plugins. For a bundled plugin:
+
+```bash
+hermes plugins enable <plugin-name>
+```
+
+For isolated local testing, use one `HERMES_HOME` for plugin enablement and the
+agent run:
+
+```bash
+export HERMES_HOME=/tmp/hermes-middleware-test
+mkdir -p "$HERMES_HOME"
+hermes plugins enable <plugin-name>
+hermes chat --query 'Reply exactly ok'
+```
+
+For source checkouts, prefer the source command so the runtime sees plugins and
+middleware from the working tree:
+
+```bash
+uv sync
+uv run hermes plugins enable <plugin-name>
+uv run hermes chat --query 'Reply exactly ok'
+```
+
+## Generic Plugin Examples
+
+The examples below are intentionally small. They show the middleware contract
+shape without depending on NeMo Relay.
+
+### LLM Request Middleware
+
+This plugin tags provider requests and records a middleware trace entry:
+
+```python
+def register(ctx):
+    ctx.register_middleware("llm_request", tag_llm_request)
+
+
+def tag_llm_request(**kwargs):
+    request = dict(kwargs["request"])
+    extra_body = dict(request.get("extra_body") or {})
+    extra_body.setdefault("metadata", {})["hermes_middleware_demo"] = True
+    request["extra_body"] = extra_body
+    return {
+        "request": request,
+        "source": "middleware-demo",
+        "reason": "tagged provider request",
+    }
+```
+
+The effective request is passed to `pre_api_request`, provider execution, and
+`post_api_request`.
+
+### Tool Request Middleware
+
+This plugin constrains `terminal` calls to a known working directory:
+
+```python
+def register(ctx):
+    ctx.register_middleware("tool_request", normalize_terminal_workdir)
+
+
+def normalize_terminal_workdir(**kwargs):
+    if kwargs.get("tool_name") != "terminal":
+        return None
+    args = dict(kwargs["args"])
+    args.setdefault("workdir", "/tmp/hermes-middleware-demo")
+    return {
+        "args": args,
+        "source": "middleware-demo",
+        "reason": "defaulted terminal workdir",
+    }
+```
+
+Because this runs before hooks and approvals, downstream telemetry and policy
+observe the rewritten `workdir`.
+
+### LLM Execution Middleware
+
+This plugin wraps the provider call and preserves the raw provider response:
+
+```python
+import time
+
+
+def register(ctx):
+    ctx.register_middleware("llm_execution", time_llm_execution)
+
+
+def time_llm_execution(**kwargs):
+    started = time.monotonic()
+    response = kwargs["next_call"](kwargs["request"])
+    elapsed_ms = int((time.monotonic() - started) * 1000)
+    print(f"llm_execution elapsed_ms={elapsed_ms}")
+    return response
+```
+
+Return the same response shape Hermes expects from the provider adapter. Do not
+wrap the response in a plugin-specific envelope unless the rest of the runtime
+expects that envelope.
+
+### Tool Execution Middleware
+
+This plugin wraps tool execution while preserving the tool result:
+
+```python
+def register(ctx):
+    ctx.register_middleware("tool_execution", annotate_tool_execution)
+
+
+def annotate_tool_execution(**kwargs):
+    result = kwargs["next_call"](kwargs["args"])
+    # Metrics, logging, or external routing can happen here.
+    return result
+```
+
+Execution middleware may call `next_call(modified_args)` to pass a changed
+payload to later middleware and the base tool dispatcher.
+
+Plugin-specific examples should live with the plugin that owns the behavior.
+NeMo Relay execution middleware is installed through an explicitly selected
+Relay `plugins.toml`; see
+[Relay shared metrics](relay-shared-metrics.md).
+
+## Safety Notes
+
+- Middleware should be deterministic for the same input unless it is explicitly
+  routing to a dynamic external system.
+- Request middleware should return complete replacement payloads, not partial
+  patches.
+- Execution middleware should call `next_call(...)` exactly once unless it is
+  intentionally short-circuiting execution.
+- If execution middleware raises before calling `next_call(...)`, Hermes treats
+  that as middleware failure and continues with the remaining middleware chain
+  and base execution.
+- If execution middleware calls `next_call(...)` successfully and then raises
+  during post-processing, Hermes preserves the downstream result and does not
+  run the provider or tool a second time.
+- If downstream provider or tool execution fails, middleware may let that error
+  propagate or translate it deliberately. Hermes does not convert downstream
+  failure into a successful `None` result.
+- Tool request middleware runs before approvals. If it mutates file paths,
+  commands, URLs, or arguments, the mutated values are what guardrails and
+  approvals evaluate.
+- Observer hooks remain the right place for read-only telemetry. Use middleware
+  only when a plugin needs to alter or wrap behavior.