fengming 0.3.10 → 0.3.12

package/docs/gateway/opentelemetry.md

@@ -1,433 +0,0 @@
----
-summary: "Export FengMing diagnostics to any OpenTelemetry collector via the diagnostics-otel plugin (OTLP/HTTP)"
-title: "OpenTelemetry export"
-read_when:
-  - You want to send FengMing model usage, message flow, or session metrics to an OpenTelemetry collector
-  - You are wiring traces, metrics, or logs into Grafana, Datadog, Honeycomb, New Relic, Tempo, or another OTLP backend
-  - You need the exact metric names, span names, or attribute shapes to build dashboards or alerts
----
-
-FengMing exports diagnostics through the official `diagnostics-otel` plugin
-using **OTLP/HTTP (protobuf)**. Any collector or backend that accepts OTLP/HTTP
-works without code changes. For local file logs and how to read them, see
-[Logging](/logging).
-
-## How it fits together
-
-- **Diagnostics events** are structured, in-process records emitted by the
-  Gateway and bundled plugins for model runs, message flow, sessions, queues,
-  and exec.
-- **`diagnostics-otel` plugin** subscribes to those events and exports them as
-  OpenTelemetry **metrics**, **traces**, and **logs** over OTLP/HTTP.
-- **Provider calls** receive a W3C `traceparent` header from FengMing's
-  trusted model-call span context when the provider transport accepts custom
-  headers. Plugin-emitted trace context is not propagated.
-- Exporters only attach when both the diagnostics surface and the plugin are
-  enabled, so the in-process cost stays near zero by default.
-
-## Quick start
-
-For packaged installs, install the plugin first:
-
-```bash
-fengming plugins install clawhub:@fengming/diagnostics-otel
-```
-
-```json5
-{
-  plugins: {
-    allow: ["diagnostics-otel"],
-    entries: {
-      "diagnostics-otel": { enabled: true },
-    },
-  },
-  diagnostics: {
-    enabled: true,
-    otel: {
-      enabled: true,
-      endpoint: "http://otel-collector:4318",
-      protocol: "http/protobuf",
-      serviceName: "fengming-gateway",
-      traces: true,
-      metrics: true,
-      logs: true,
-      sampleRate: 0.2,
-      flushIntervalMs: 60000,
-    },
-  },
-}
-```
-
-You can also enable the plugin from the CLI:
-
-```bash
-fengming plugins enable diagnostics-otel
-```
-
-<Note>
-`protocol` currently supports `http/protobuf` only. `grpc` is ignored.
-</Note>
-
-## Signals exported
-
-| Signal      | What goes in it                                                                                                                                                                                                    |
-| ----------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
-| **Metrics** | Counters and histograms for token usage, cost, run duration, failover, skill usage, message flow, Talk events, queue lanes, session state/recovery, tool execution, oversized payloads, exec, and memory pressure. |
-| **Traces**  | Spans for model usage, model calls, harness lifecycle, skill usage, tool execution, exec, webhook/message processing, context assembly, and tool loops.                                                            |
-| **Logs**    | Structured `logging.file` records exported over OTLP when `diagnostics.otel.logs` is enabled; log bodies are withheld unless content capture is explicitly enabled.                                                |
-
-Toggle `traces`, `metrics`, and `logs` independently. Traces and metrics
-default to on when `diagnostics.otel.enabled` is true. Logs default to off and
-are exported only when `diagnostics.otel.logs` is explicitly `true`.
-
-## Configuration reference
-
-```json5
-{
-  diagnostics: {
-    enabled: true,
-    otel: {
-      enabled: true,
-      endpoint: "http://otel-collector:4318",
-      tracesEndpoint: "http://otel-collector:4318/v1/traces",
-      metricsEndpoint: "http://otel-collector:4318/v1/metrics",
-      logsEndpoint: "http://otel-collector:4318/v1/logs",
-      protocol: "http/protobuf", // grpc is ignored
-      serviceName: "fengming-gateway",
-      headers: { "x-collector-token": "..." },
-      traces: true,
-      metrics: true,
-      logs: true,
-      sampleRate: 0.2, // root-span sampler, 0.0..1.0
-      flushIntervalMs: 60000, // metric export interval (min 1000ms)
-      captureContent: {
-        enabled: false,
-        inputMessages: false,
-        outputMessages: false,
-        toolInputs: false,
-        toolOutputs: false,
-        systemPrompt: false,
-        toolDefinitions: false,
-      },
-    },
-  },
-}
-```
-
-### Environment variables
-
-| Variable                                                                                                          | Purpose                                                                                                                                                                                                                                                                                                                                        |
-| ----------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `OTEL_EXPORTER_OTLP_ENDPOINT`                                                                                     | Override `diagnostics.otel.endpoint`. If the value already contains `/v1/traces`, `/v1/metrics`, or `/v1/logs`, it is used as-is.                                                                                                                                                                                                              |
-| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` / `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` / `OTEL_EXPORTER_OTLP_LOGS_ENDPOINT` | Signal-specific endpoint overrides used when the matching `diagnostics.otel.*Endpoint` config key is unset. Signal-specific config wins over signal-specific env, which wins over the shared endpoint.                                                                                                                                         |
-| `OTEL_SERVICE_NAME`                                                                                               | Override `diagnostics.otel.serviceName`.                                                                                                                                                                                                                                                                                                       |
-| `OTEL_EXPORTER_OTLP_PROTOCOL`                                                                                     | Override the wire protocol (only `http/protobuf` is honored today).                                                                                                                                                                                                                                                                            |
-| `OTEL_SEMCONV_STABILITY_OPT_IN`                                                                                   | Set to `gen_ai_latest_experimental` to emit the latest experimental GenAI inference span shape, including `{gen_ai.operation.name} {gen_ai.request.model}` span names, `CLIENT` span kind, and `gen_ai.provider.name` instead of the legacy `gen_ai.system`. GenAI metrics always use bounded, low-cardinality semantic attributes regardless. |
-| `FENGMING_OTEL_PRELOADED`                                                                                         | Set to `1` when another preload or host process already registered the global OpenTelemetry SDK. The plugin then skips its own NodeSDK lifecycle but still wires diagnostic listeners and honors `traces`/`metrics`/`logs`.                                                                                                                    |
-
-## Privacy and content capture
-
-Raw model/tool content is **not** exported by default. Spans carry bounded
-identifiers (channel, provider, model, error category, hash-only request ids,
-tool source, tool owner, and skill name/source) and never include prompt text,
-response text, tool inputs, tool outputs, skill file paths, or session keys.
-OTLP log records keep severity, logger, code location, trusted trace context,
-and sanitized attributes by default, but the raw log message body is exported
-only when `diagnostics.otel.captureContent` is set to boolean `true`. Granular
-`captureContent.*` subkeys do not enable log bodies. Labels that look like
-scoped agent session keys are replaced with `unknown`.
-Talk metrics export only bounded event metadata such as mode, transport,
-provider, and event type. They do not include transcripts, audio payloads,
-session ids, turn ids, call ids, room ids, or handoff tokens.
-
-Outbound model requests may include a W3C `traceparent` header. That header is
-generated only from FengMing-owned diagnostic trace context for the active model
-call. Existing caller-supplied `traceparent` headers are replaced, so plugins or
-custom provider options cannot spoof cross-service trace ancestry.
-
-Set `diagnostics.otel.captureContent.*` to `true` only when your collector and
-retention policy are approved for prompt, response, tool, or system-prompt
-text. Each subkey is opt-in independently:
-
-- `inputMessages` - user prompt content.
-- `outputMessages` - model response content.
-- `toolInputs` - tool argument payloads.
-- `toolOutputs` - tool result payloads.
-- `systemPrompt` - assembled system/developer prompt.
-- `toolDefinitions` - model tool names, descriptions, and schemas.
-
-When any subkey is enabled, model and tool spans get bounded, redacted
-`fengming.content.*` attributes for that class only. Use boolean
-`captureContent: true` only for broad diagnostics captures where OTLP log
-message bodies are also approved for export.
-
-## Sampling and flushing
-
-- **Traces:** `diagnostics.otel.sampleRate` (root-span only, `0.0` drops all,
-  `1.0` keeps all).
-- **Metrics:** `diagnostics.otel.flushIntervalMs` (minimum `1000`).
-- **Logs:** OTLP logs respect `logging.level` (file log level). They use the
-  diagnostic log-record redaction path, not console formatting. High-volume
-  installs should prefer OTLP collector sampling/filtering over local sampling.
-- **File-log correlation:** JSONL file logs include top-level `traceId`,
-  `spanId`, `parentSpanId`, and `traceFlags` when the log call carries a valid
-  diagnostic trace context, which lets log processors join local log lines with
-  exported spans.
-- **Request correlation:** Gateway HTTP requests and WebSocket frames create an
-  internal request trace scope. Logs and diagnostic events inside that scope
-  inherit the request trace by default, while agent run and model-call spans are
-  created as children so provider `traceparent` headers stay on the same trace.
-
-## Exported metrics
-
-### Model usage
-
-- `fengming.tokens` (counter, attrs: `fengming.token`, `fengming.channel`, `fengming.provider`, `fengming.model`, `fengming.agent`)
-- `fengming.cost.usd` (counter, attrs: `fengming.channel`, `fengming.provider`, `fengming.model`)
-- `fengming.run.duration_ms` (histogram, attrs: `fengming.channel`, `fengming.provider`, `fengming.model`)
-- `fengming.context.tokens` (histogram, attrs: `fengming.context`, `fengming.channel`, `fengming.provider`, `fengming.model`)
-- `gen_ai.client.token.usage` (histogram, GenAI semantic-conventions metric, attrs: `gen_ai.token.type` = `input`/`output`, `gen_ai.provider.name`, `gen_ai.operation.name`, `gen_ai.request.model`)
-- `gen_ai.client.operation.duration` (histogram, seconds, GenAI semantic-conventions metric, attrs: `gen_ai.provider.name`, `gen_ai.operation.name`, `gen_ai.request.model`, optional `error.type`)
-- `fengming.model_call.duration_ms` (histogram, attrs: `fengming.provider`, `fengming.model`, `fengming.api`, `fengming.transport`, plus `fengming.errorCategory` and `fengming.failureKind` on classified errors)
-- `fengming.model_call.request_bytes` (histogram, UTF-8 byte size of the final model request payload; no raw payload content)
-- `fengming.model_call.response_bytes` (histogram, UTF-8 byte size of streamed model response events; no raw response content)
-- `fengming.model_call.time_to_first_byte_ms` (histogram, elapsed time before the first streamed response event)
-- `fengming.model.failover` (counter, attrs: `fengming.provider`, `fengming.model`, `fengming.failover.to_provider`, `fengming.failover.to_model`, `fengming.failover.reason`, `fengming.failover.suspended`, `fengming.lane`)
-- `fengming.skill.used` (counter, attrs: `fengming.skill.name`, `fengming.skill.source`, `fengming.skill.activation`, optional `fengming.agent`, optional `fengming.toolName`)
-
-### Message flow
-
-- `fengming.webhook.received` (counter, attrs: `fengming.channel`, `fengming.webhook`)
-- `fengming.webhook.error` (counter, attrs: `fengming.channel`, `fengming.webhook`)
-- `fengming.webhook.duration_ms` (histogram, attrs: `fengming.channel`, `fengming.webhook`)
-- `fengming.message.queued` (counter, attrs: `fengming.channel`, `fengming.source`)
-- `fengming.message.received` (counter, attrs: `fengming.channel`, `fengming.source`)
-- `fengming.message.dispatch.started` (counter, attrs: `fengming.channel`, `fengming.source`)
-- `fengming.message.dispatch.completed` (counter, attrs: `fengming.channel`, `fengming.outcome`, `fengming.reason`, `fengming.source`)
-- `fengming.message.dispatch.duration_ms` (histogram, attrs: `fengming.channel`, `fengming.outcome`, `fengming.reason`, `fengming.source`)
-- `fengming.message.processed` (counter, attrs: `fengming.channel`, `fengming.outcome`)
-- `fengming.message.duration_ms` (histogram, attrs: `fengming.channel`, `fengming.outcome`)
-- `fengming.message.delivery.started` (counter, attrs: `fengming.channel`, `fengming.delivery.kind`)
-- `fengming.message.delivery.duration_ms` (histogram, attrs: `fengming.channel`, `fengming.delivery.kind`, `fengming.outcome`, `fengming.errorCategory`)
-
-### Talk
-
-- `fengming.talk.event` (counter, attrs: `fengming.talk.event_type`, `fengming.talk.mode`, `fengming.talk.transport`, `fengming.talk.brain`, `fengming.talk.provider`)
-- `fengming.talk.event.duration_ms` (histogram, attrs: same as `fengming.talk.event`; emitted when a Talk event reports duration)
-- `fengming.talk.audio.bytes` (histogram, attrs: same as `fengming.talk.event`; emitted for Talk audio frame events that report byte length)
-
-### Queues and sessions
-
-- `fengming.queue.lane.enqueue` (counter, attrs: `fengming.lane`)
-- `fengming.queue.lane.dequeue` (counter, attrs: `fengming.lane`)
-- `fengming.queue.depth` (histogram, attrs: `fengming.lane` or `fengming.channel=heartbeat`)
-- `fengming.queue.wait_ms` (histogram, attrs: `fengming.lane`)
-- `fengming.session.state` (counter, attrs: `fengming.state`, `fengming.reason`)
-- `fengming.session.stuck` (counter, attrs: `fengming.state`; emitted for recoverable stale session bookkeeping)
-- `fengming.session.stuck_age_ms` (histogram, attrs: `fengming.state`; emitted for recoverable stale session bookkeeping)
-- `fengming.session.turn.created` (counter, attrs: `fengming.agent`, `fengming.channel`, `fengming.trigger`)
-- `fengming.session.recovery.requested` (counter, attrs: `fengming.state`, `fengming.action`, `fengming.active_work_kind`, `fengming.reason`)
-- `fengming.session.recovery.completed` (counter, attrs: `fengming.state`, `fengming.action`, `fengming.status`, `fengming.active_work_kind`, `fengming.reason`)
-- `fengming.session.recovery.age_ms` (histogram, attrs: same as the matching recovery counter)
-- `fengming.run.attempt` (counter, attrs: `fengming.attempt`)
-
-### Session liveness telemetry
-
-`diagnostics.stuckSessionWarnMs` is the no-progress age threshold for session
-liveness diagnostics. A `processing` session does not age toward this threshold
-while FengMing observes reply, tool, status, block, or ACP runtime progress.
-Typing keepalives are not counted as progress, so a silent model or harness can
-still be detected.
-
-FengMing classifies sessions by the work it can still observe:
-
-- `session.long_running`: active embedded work, model calls, or tool calls are
-  still making progress.
-- `session.stalled`: active work exists, but the active run has not reported
-  recent progress. Stalled embedded runs stay observe-only at first, then
-  abort-drain after `diagnostics.stuckSessionAbortMs` with no progress so queued
-  turns behind the lane can resume. When unset, the abort threshold defaults to
-  the safer extended window of at least 5 minutes and 3x
-  `diagnostics.stuckSessionWarnMs`.
-- `session.stuck`: stale session bookkeeping with no active work, or an idle
-  queued session with stale ownerless model/tool activity. This releases the
-  affected session lane immediately after recovery gates pass.
-
-Recovery emits structured `session.recovery.requested` and
-`session.recovery.completed` events. Diagnostic session state is marked idle
-only after a mutating recovery outcome (`aborted` or `released`) and only if the
-same processing generation is still current.
-
-Only `session.stuck` emits the `fengming.session.stuck` counter, the
-`fengming.session.stuck_age_ms` histogram, and the `fengming.session.stuck`
-span. Repeated `session.stuck` diagnostics back off while the session remains
-unchanged, so dashboards should alert on sustained increases rather than every
-heartbeat tick. For the config knob and defaults, see
-[Configuration reference](/gateway/configuration-reference#diagnostics).
-
-Liveness warnings also emit:
-
-- `fengming.liveness.warning` (counter, attrs: `fengming.liveness.reason`)
-- `fengming.liveness.event_loop_delay_p99_ms` (histogram, attrs: `fengming.liveness.reason`)
-- `fengming.liveness.event_loop_delay_max_ms` (histogram, attrs: `fengming.liveness.reason`)
-- `fengming.liveness.event_loop_utilization` (histogram, attrs: `fengming.liveness.reason`)
-- `fengming.liveness.cpu_core_ratio` (histogram, attrs: `fengming.liveness.reason`)
-
-### Harness lifecycle
-
-- `fengming.harness.duration_ms` (histogram, attrs: `fengming.harness.id`, `fengming.harness.plugin`, `fengming.outcome`, `fengming.harness.phase` on errors)
-
-### Tool execution
-
-- `fengming.tool.execution.duration_ms` (histogram, attrs: `gen_ai.tool.name`, `fengming.toolName`, `fengming.tool.source`, `fengming.tool.owner`, `fengming.tool.params.kind`, plus `fengming.errorCategory` on errors)
-- `fengming.tool.execution.blocked` (counter, attrs: `gen_ai.tool.name`, `fengming.toolName`, `fengming.tool.source`, `fengming.tool.owner`, `fengming.tool.params.kind`, `fengming.deniedReason`)
-
-### Exec
-
-- `fengming.exec.duration_ms` (histogram, attrs: `fengming.exec.target`, `fengming.exec.mode`, `fengming.outcome`, `fengming.failureKind`)
-
-### Diagnostics internals (memory and tool loop)
-
-- `fengming.payload.large` (counter, attrs: `fengming.payload.surface`, `fengming.payload.action`, `fengming.channel`, `fengming.plugin`, `fengming.reason`)
-- `fengming.payload.large_bytes` (histogram, attrs: same as `fengming.payload.large`)
-- `fengming.memory.heap_used_bytes` (histogram, attrs: `fengming.memory.kind`)
-- `fengming.memory.rss_bytes` (histogram)
-- `fengming.memory.pressure` (counter, attrs: `fengming.memory.level`)
-- `fengming.tool.loop.iterations` (counter, attrs: `fengming.toolName`, `fengming.outcome`)
-- `fengming.tool.loop.duration_ms` (histogram, attrs: `fengming.toolName`, `fengming.outcome`)
-
-## Exported spans
-
-- `fengming.model.usage`
-  - `fengming.channel`, `fengming.provider`, `fengming.model`
-  - `fengming.tokens.*` (input/output/cache_read/cache_write/total)
-  - `gen_ai.system` by default, or `gen_ai.provider.name` when the latest GenAI semantic conventions are opted in
-  - `gen_ai.request.model`, `gen_ai.operation.name`, `gen_ai.usage.*`
-- `fengming.run`
-  - `fengming.outcome`, `fengming.channel`, `fengming.provider`, `fengming.model`, `fengming.errorCategory`
-- `fengming.model.call`
-  - `gen_ai.system` by default, or `gen_ai.provider.name` when the latest GenAI semantic conventions are opted in
-  - `gen_ai.request.model`, `gen_ai.operation.name`, `fengming.provider`, `fengming.model`, `fengming.api`, `fengming.transport`
-  - `fengming.errorCategory` and optional `fengming.failureKind` on errors
-  - `fengming.model_call.request_bytes`, `fengming.model_call.response_bytes`, `fengming.model_call.time_to_first_byte_ms`
-  - `fengming.provider.request_id_hash` (bounded SHA-based hash of the upstream provider request id; raw ids are not exported)
-  - With `OTEL_SEMCONV_STABILITY_OPT_IN=gen_ai_latest_experimental`, model-call spans use the latest GenAI inference span name `{gen_ai.operation.name} {gen_ai.request.model}` and `CLIENT` span kind instead of `fengming.model.call`.
-- `fengming.harness.run`
-  - `fengming.harness.id`, `fengming.harness.plugin`, `fengming.outcome`, `fengming.provider`, `fengming.model`, `fengming.channel`
-  - On completion: `fengming.harness.result_classification`, `fengming.harness.yield_detected`, `fengming.harness.items.started`, `fengming.harness.items.completed`, `fengming.harness.items.active`
-  - On error: `fengming.harness.phase`, `fengming.errorCategory`, optional `fengming.harness.cleanup_failed`
-- `fengming.tool.execution`
-  - `gen_ai.tool.name`, `fengming.toolName`, `fengming.errorCategory`, `fengming.tool.params.*`
-- `fengming.exec`
-  - `fengming.exec.target`, `fengming.exec.mode`, `fengming.outcome`, `fengming.failureKind`, `fengming.exec.command_length`, `fengming.exec.exit_code`, `fengming.exec.timed_out`
-- `fengming.webhook.processed`
-  - `fengming.channel`, `fengming.webhook`
-- `fengming.webhook.error`
-  - `fengming.channel`, `fengming.webhook`, `fengming.error`
-- `fengming.message.processed`
-  - `fengming.channel`, `fengming.outcome`, `fengming.reason`
-- `fengming.message.delivery`
-  - `fengming.channel`, `fengming.delivery.kind`, `fengming.outcome`, `fengming.errorCategory`, `fengming.delivery.result_count`
-- `fengming.session.stuck`
-  - `fengming.state`, `fengming.ageMs`, `fengming.queueDepth`
-- `fengming.context.assembled`
-  - `fengming.prompt.size`, `fengming.history.size`, `fengming.context.tokens`, `fengming.errorCategory` (no prompt, history, response, or session-key content)
-- `fengming.tool.loop`
-  - `fengming.toolName`, `fengming.outcome`, `fengming.iterations`, `fengming.errorCategory` (no loop messages, params, or tool output)
-- `fengming.memory.pressure`
-  - `fengming.memory.level`, `fengming.memory.heap_used_bytes`, `fengming.memory.rss_bytes`
-
-When content capture is explicitly enabled, model and tool spans can also
-include bounded, redacted `fengming.content.*` attributes for the specific
-content classes you opted into.
-
-## Diagnostic event catalog
-
-The events below back the metrics and spans above. Plugins can also subscribe
-to them directly without OTLP export.
-
-**Model usage**
-
-- `model.usage` - tokens, cost, duration, context, provider/model/channel,
-  session ids. `usage` is provider/turn accounting for cost and telemetry;
-  `context.used` is the current prompt/context snapshot and can be lower than
-  provider `usage.total` when cached input or tool-loop calls are involved.
-
-**Message flow**
-
-- `webhook.received` / `webhook.processed` / `webhook.error`
-- `message.queued` / `message.processed`
-- `message.delivery.started` / `message.delivery.completed` / `message.delivery.error`
-
-**Queue and session**
-
-- `queue.lane.enqueue` / `queue.lane.dequeue`
-- `session.state` / `session.long_running` / `session.stalled` / `session.stuck`
-- `run.attempt` / `run.progress`
-- `diagnostic.heartbeat` (aggregate counters: webhooks/queue/session)
-
-**Harness lifecycle**
-
-- `harness.run.started` / `harness.run.completed` / `harness.run.error` -
-  per-run lifecycle for the agent harness. Includes `harnessId`, optional
-  `pluginId`, provider/model/channel, and run id. Completion adds
-  `durationMs`, `outcome`, optional `resultClassification`, `yieldDetected`,
-  and `itemLifecycle` counts. Errors add `phase`
-  (`prepare`/`start`/`send`/`resolve`/`cleanup`), `errorCategory`, and
-  optional `cleanupFailed`.
-
-**Exec**
-
-- `exec.process.completed` - terminal outcome, duration, target, mode, exit
-  code, and failure kind. Command text and working directories are not
-  included.
-
-## Without an exporter
-
-You can keep diagnostics events available to plugins or custom sinks without
-running `diagnostics-otel`:
-
-```json5
-{
-  diagnostics: { enabled: true },
-}
-```
-
-For targeted debug output without raising `logging.level`, use diagnostics
-flags. Flags are case-insensitive and support wildcards (e.g. `telegram.*` or
-`*`):
-
-```json5
-{
-  diagnostics: { flags: ["telegram.http"] },
-}
-```
-
-Or as a one-off env override:
-
-```bash
-FENGMING_DIAGNOSTICS=telegram.http,telegram.payload fengming gateway
-```
-
-Flag output goes to the standard log file (`logging.file`) and is still
-redacted by `logging.redactSensitive`. Full guide:
-[Diagnostics flags](/diagnostics/flags).
-
-## Disable
-
-```json5
-{
-  diagnostics: { otel: { enabled: false } },
-}
-```
-
-You can also leave `diagnostics-otel` out of `plugins.allow`, or run
-`fengming plugins disable diagnostics-otel`.
-
-## Related
-
-- [Logging](/logging) - file logs, console output, CLI tailing, and the Control UI Logs tab
-- [Gateway logging internals](/gateway/logging) - WS log styles, subsystem prefixes, and console capture
-- [Diagnostics flags](/diagnostics/flags) - targeted debug-log flags
-- [Diagnostics export](/gateway/diagnostics) - operator support-bundle tool (separate from OTEL export)
-- [Configuration reference](/gateway/configuration-reference#diagnostics) - full `diagnostics.*` field reference

package/docs/gateway/operator-scopes.md

@@ -1,119 +0,0 @@
----
-summary: "Operator roles, scopes, and approval-time checks for Gateway clients"
-read_when:
-  - Debugging missing operator scope errors
-  - Reviewing device or node pairing approvals
-  - Adding or classifying Gateway RPC methods
-title: "Operator scopes"
----
-
-Operator scopes define what a Gateway client may do after it authenticates.
-They are a control-plane guardrail inside one trusted Gateway operator domain,
-not hostile multi-tenant isolation. If you need strong separation between
-people, teams, or machines, run separate Gateways under separate OS users or
-hosts.
-
-Related: [Security](/gateway/security), [Gateway protocol](/gateway/protocol),
-[Gateway pairing](/gateway/pairing), [Devices CLI](/cli/devices).
-
-## Roles
-
-Gateway WebSocket clients connect with one role:
-
-- `operator`: control-plane clients such as CLI, Control UI, automation, and
-  trusted helper processes.
-- `node`: capability hosts such as macOS, iOS, Android, or headless nodes that
-  expose commands through `node.invoke`.
-
-Operator RPC methods require the `operator` role. Node-originated methods
-require the `node` role.
-
-## Scope levels
-
-| Scope                   | Meaning                                                                                                                                                                               |
-| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| `operator.read`         | Read-only status, lists, catalog, logs, session reads, and other non-mutating control-plane calls.                                                                                    |
-| `operator.write`        | Normal mutating operator actions such as sending messages, invoking tools, updating talk/voice settings, and node command relay. Also satisfies `operator.read`.                      |
-| `operator.admin`        | Administrative control-plane access. Satisfies every `operator.*` scope. Required for config mutation, updates, native hooks, sensitive reserved namespaces, and high-risk approvals. |
-| `operator.pairing`      | Device and node pairing management, including listing, approving, rejecting, removing, rotating, and revoking pairing records or device tokens.                                       |
-| `operator.approvals`    | Exec and plugin approval APIs.                                                                                                                                                        |
-| `operator.talk.secrets` | Reading Talk configuration with secrets included.                                                                                                                                     |
-
-Unknown future `operator.*` scopes require an exact match unless the caller has
-`operator.admin`.
-
-## Method scope is only the first gate
-
-Each Gateway RPC has a least-privilege method scope. That method scope decides
-whether the request can reach the handler. Some handlers then apply stricter
-approval-time checks based on the concrete thing being approved or mutated.
-
-Examples:
-
-- `device.pair.approve` is reachable with `operator.pairing`, but approving an
-  operator device can only mint or preserve scopes the caller already holds.
-- `node.pair.approve` is reachable with `operator.pairing`, then derives extra
-  approval scopes from the pending node command list.
-- `chat.send` is normally a write-scoped method, but persistent `/config set`
-  and `/config unset` require `operator.admin` at command level.
-
-This lets lower-scope operators perform low-risk pairing actions without making
-all pairing approval admin-only.
-
-## Device pairing approvals
-
-Device pairing records are the durable source of approved roles and scopes.
-Already paired devices do not get broader access silently: reconnects that ask
-for a broader role or broader scopes create a new pending upgrade request.
-
-When approving a device request:
-
-- A request with no operator role does not need operator token scope approval.
-- A request for a non-operator device role, such as `node`, requires
-  `operator.admin`, even when `device.pair.approve` is reachable with
-  `operator.pairing`.
-- A request for `operator.read`, `operator.write`, `operator.approvals`,
-  `operator.pairing`, or `operator.talk.secrets` requires the caller to hold
-  those scopes, or `operator.admin`.
-- A request for `operator.admin` requires `operator.admin`.
-- A repair request with no explicit scopes can inherit the existing operator
-  token scopes. If that existing token is admin-scoped, approval still requires
-  `operator.admin`.
-
-Non-admin shared-secret and trusted-proxy sessions can approve operator-device
-requests only inside their own declared operator scopes. Approving non-operator
-roles is admin-only even when those sessions can otherwise use
-`operator.pairing`.
-
-For paired-device token sessions, management is also self-scoped unless the
-caller has `operator.admin`: non-admin callers see only their own pairing
-entries, can approve or reject only their own pending request, and can rotate,
-revoke, or remove only their own device entry.
-
-## Node pairing approvals
-
-Legacy `node.pair.*` uses a separate Gateway-owned node pairing store. WS nodes
-use device pairing with `role: node`, but the same approval-level vocabulary
-applies.
-
-`node.pair.approve` uses the pending request command list to derive additional
-required scopes:
-
-- Commandless request: `operator.pairing`
-- Non-exec node commands: `operator.pairing` + `operator.write`
-- `system.run`, `system.run.prepare`, or `system.which`:
-  `operator.pairing` + `operator.admin`
-
-Node pairing establishes identity and trust. It does not replace the node's
-own `system.run` exec approval policy.
-
-## Shared-secret auth
-
-Shared gateway token/password auth is treated as trusted operator access for
-that Gateway. OpenAI-compatible HTTP surfaces, `/tools/invoke`, and HTTP session
-history endpoints restore the normal full operator default scope set for
-shared-secret bearer auth, even if a caller sends narrower declared scopes.
-
-Identity-bearing modes, such as trusted proxy auth or private-ingress `none`,
-can still honor explicit declared scopes. Use separate Gateways for real trust
-boundary separation.