@probelabs/visor 0.1.173 → 0.1.174-ee

package/dist/docs/telemetry-reference.md

@@ -0,0 +1,387 @@
+# Telemetry Reference — Spans, Metrics & Events
+
+This document is the complete reference for all OpenTelemetry signals emitted by Visor. Use it to build Grafana dashboards, set up alerts, or debug execution flows.
+
+For setup instructions, see [Telemetry Setup Guide](./telemetry-setup.md).
+
+---
+
+## Metrics
+
+All metrics use the `visor` meter and are exported when the OTLP sink is configured with `@opentelemetry/exporter-metrics-otlp-http`.
+
+> **Prometheus naming**: OTel metric names use dots (e.g., `visor.run.total`) but Prometheus converts them to underscores (e.g., `visor_run_total`). The table below shows both forms.
+
+### Run Metrics
+
+| Metric | Prometheus Name | Type | Unit | Description |
+|--------|----------------|------|------|-------------|
+| `visor.run.total` | `visor_run_total` | Counter | 1 | Total number of visor runs (workflow executions) |
+| `visor.run.duration_ms` | `visor_run_duration_ms` | Histogram | ms | Duration of a complete visor run |
+| `visor.run.active_checks` | `visor_run_active_checks` | UpDownCounter | 1 | Number of checks actively running (concurrent gauge) |
+| `visor.run.ai_calls` | `visor_run_ai_calls` | Histogram | 1 | Number of AI calls per visor run |
+
+**Labels for `visor.run.total`:**
+
+| Label | Description | Example |
+|-------|-------------|---------|
+| `visor.run.source` | Entry point that started the run | `cli`, `slack`, `tui`, `tui-rerun` |
+| `visor.run.user_id` | User identifier (Slack user ID, etc.) | `U01ABC123` |
+| `visor.run.user_name` | User display name | `alice` |
+| `visor.run.workflow` | Comma-separated check IDs | `security,performance` |
+| `visor.instance_id` | Unique instance identifier | `abc123` |
+
+**Labels for `visor.run.duration_ms`:**
+
+| Label | Description | Example |
+|-------|-------------|---------|
+| `visor.run.source` | Entry point | `cli`, `slack`, `tui` |
+| `visor.run.user_id` | User identifier | `U01ABC123` |
+| `visor.run.workflow` | Check IDs | `security,performance` |
+| `visor.run.success` | Whether the run succeeded | `true`, `false` |
+
+### Check Metrics
+
+| Metric | Prometheus Name | Type | Unit | Description |
+|--------|----------------|------|------|-------------|
+| `visor.check.duration_ms` | `visor_check_duration_ms` | Histogram | ms | Duration of a single check execution |
+| `visor.check.issues` | `visor_check_issues_total` | Counter | 1 | Number of issues produced by checks |
+
+**Labels:**
+
+| Label | Applies To | Description |
+|-------|-----------|-------------|
+| `visor.check.id` | Both | Check identifier (e.g., `security`, `overview`) |
+| `visor.check.group` | `duration_ms` | Check group (default: `default`) |
+| `severity` | `issues` | Issue severity level |
+
+### Provider Metrics
+
+| Metric | Prometheus Name | Type | Unit | Description |
+|--------|----------------|------|------|-------------|
+| `visor.provider.duration_ms` | `visor_provider_duration_ms` | Histogram | ms | Duration of provider execution |
+| `visor.foreach.item.duration_ms` | `visor_foreach_item_duration_ms` | Histogram | ms | Duration of a single forEach item |
+
+**Labels:**
+
+| Label | Applies To | Description |
+|-------|-----------|-------------|
+| `visor.check.id` | Both | Check identifier |
+| `visor.provider.type` | `provider.duration_ms` | Provider type (`ai`, `command`, `http`, etc.) |
+| `visor.foreach.index` | `foreach.item.duration_ms` | Iteration index |
+| `visor.foreach.total` | `foreach.item.duration_ms` | Total items in forEach |
+
+### AI Metrics
+
+| Metric | Prometheus Name | Type | Unit | Description |
+|--------|----------------|------|------|-------------|
+| `visor.ai_call.total` | `visor_ai_call_total` | Counter | 1 | Total number of AI provider calls |
+
+**Labels:**
+
+| Label | Description | Example |
+|-------|-------------|---------|
+| `visor.check.id` | Check that triggered the AI call | `security` |
+| `visor.ai.model` | AI model name | `gemini-2.5-flash`, `claude-sonnet-4-5-20250514` |
+| `visor.run.source` | Entry point | `cli`, `slack` |
+
+### Other Metrics
+
+| Metric | Prometheus Name | Type | Unit | Description |
+|--------|----------------|------|------|-------------|
+| `visor.fail_if.triggered` | `visor_fail_if_triggered_total` | Counter | 1 | Times a fail_if condition triggered |
+| `visor.diagram.blocks` | `visor_diagram_blocks_total` | Counter | 1 | Mermaid diagram blocks emitted |
+
+---
+
+## Spans
+
+### Root Span
+
+| Span Name | Description |
+|-----------|-------------|
+| `visor.run` | Root span wrapping an entire workflow execution. One per CLI invocation, Slack message, or TUI interaction. |
+
+**Attributes on `visor.run`:**
+
+| Attribute | Description |
+|-----------|-------------|
+| `visor.version` | Visor version (from package.json or `VISOR_VERSION` env) |
+| `visor.commit` | Git commit short SHA |
+| `visor.commit.sha` | Git commit full SHA |
+| `visor.instance_id` | Unique instance identifier |
+| `visor.run.source` | Entry point: `cli`, `slack`, `tui`, `tui-rerun`, `slack_message_trigger` |
+| `visor.run.ai_calls` | Total AI calls made during this run |
+| `visor.run.duration_ms` | Run duration in milliseconds |
+
+Additional attributes for Slack message triggers:
+
+| Attribute | Description |
+|-----------|-------------|
+| `visor.trigger.id` | Trigger configuration ID |
+| `visor.trigger.workflow` | Triggered workflow name |
+| `slack.channel_id` | Slack channel ID |
+| `slack.thread_ts` | Slack thread timestamp |
+| `slack.user_id` | Slack user ID |
+
+### State Machine Spans
+
+| Span Name | Description |
+|-----------|-------------|
+| `engine.state.init` | Engine initialization |
+| `engine.state.planready` | Plan is ready for execution |
+| `engine.state.waveplanning` | Planning which checks to execute in the next wave |
+| `engine.state.leveldispatch` | Dispatching checks for execution |
+| `engine.state.checkrunning` | Checks are actively running |
+| `engine.state.completed` | Engine has completed |
+| `engine.state.error` | Engine encountered an error |
+
+**Attributes on state spans:**
+
+| Attribute | Description |
+|-----------|-------------|
+| `wave` | Current execution wave number |
+| `wave_kind` | Wave type |
+| `session_id` | Session identifier |
+| `level_size` | Number of checks planned in this wave |
+| `level_checks_preview` | Preview of check IDs in this wave |
+
+### Check Spans
+
+| Span Name | Description |
+|-----------|-------------|
+| `visor.check.<checkId>` | Execution of a single check (e.g., `visor.check.security`) |
+
+**Attributes on check spans:**
+
+| Attribute | Description |
+|-----------|-------------|
+| `visor.check.id` | Check identifier |
+| `visor.check.type` | Provider type (`ai`, `command`, `http`, `mcp`, `claude-code`, `noop`, `logger`) |
+| `visor.foreach.index` | Iteration index (if inside a forEach) |
+| `session_id` | Session identifier |
+| `wave` | Wave number |
+
+### AI Provider Spans
+
+| Span Name | Description |
+|-----------|-------------|
+| `visor.ai_check` | Fresh AI agent call (new session) |
+| `visor.ai_check_reuse` | AI agent call reusing an existing session |
+
+**Attributes:**
+
+| Attribute | Description |
+|-----------|-------------|
+| `check.name` | Check name |
+| `check.session_id` | AI session ID (fresh calls only) |
+| `check.mode` | `session_reuse` (reuse calls only) |
+| `prompt.length` | Length of the prompt in characters |
+| `schema.type` | Output schema type |
+
+### Sandbox Spans
+
+| Span Name | Description |
+|-----------|-------------|
+| `visor.sandbox.start` | Starting a sandbox container |
+| `visor.sandbox.exec` | Executing a command in a sandbox |
+| `visor.sandbox.stopAll` | Stopping all sandboxes |
+| `visor.sandbox.build` | Building a sandbox Docker image |
+| `visor.sandbox.runCheck` | Running a check inside a sandbox |
+| `visor.sandbox.child.<type>` | Sandbox child process execution |
+
+### Agent Protocol Spans
+
+| Span Name | Description |
+|-----------|-------------|
+| `agent.task` | A2A agent task execution |
+| `agent.queue.execute` | Task queue worker execution |
+
+---
+
+## Span Events
+
+Events are attached to the currently active span and provide fine-grained execution details.
+
+### Routing Events
+
+| Event Name | Description |
+|------------|-------------|
+| `visor.routing` | A routing decision was made after a check completed |
+
+**Attributes:** `check_id`, `trigger`, `action` (`retry`/`goto`/`run`), `target`, `source`, `scope`, `goto_event`
+
+### Failure Condition Events
+
+| Event Name | Description |
+|------------|-------------|
+| `fail_if.evaluated` | A fail_if condition was evaluated |
+| `fail_if.triggered` | A fail_if condition was triggered (failed) |
+
+**Attributes:** `visor.check.id`, `scope` (`global`/`check`), `expression`, `result`, `severity`, `name`
+
+### Tool Setup Events (AI Provider)
+
+| Event Name | Description |
+|------------|-------------|
+| `tool_setup.mcp_servers_js` | MCP servers loaded from dynamic JS expression |
+| `tool_setup.mcp_servers_js_error` | Error evaluating MCP server expression |
+| `tool_setup.mcp_servers_js_skipped` | MCP server setup skipped |
+| `tool_setup.resolution` | Tool name resolution completed |
+| `tool_setup.sse_server_error` | SSE tool server connection failed |
+| `tool_setup.final` | Final tool setup summary |
+
+### Sandbox Events
+
+| Event Name | Description |
+|------------|-------------|
+| `visor.sandbox.container.started` | Docker container started |
+| `visor.sandbox.container.stopped` | Docker container stopped |
+| `visor.sandbox.bwrap.exec` | Bubblewrap sandbox execution |
+| `visor.sandbox.seatbelt.exec` | Seatbelt sandbox execution |
+| `visor.sandbox.stopped` | Sandbox stopped |
+
+### Diagram Events
+
+| Event Name | Description |
+|------------|-------------|
+| `diagram.block` | A Mermaid diagram block was emitted |
+
+**Attributes:** `check`, `origin` (`content`/`issue`), `code`
+
+---
+
+## State Capture Attributes
+
+When `VISOR_TELEMETRY_FULL_CAPTURE=true`, additional detailed attributes are captured on check spans:
+
+### Check Input Context
+
+| Attribute | Description | Max Size |
+|-----------|-------------|----------|
+| `visor.check.input.keys` | Comma-separated context variable names | — |
+| `visor.check.input.count` | Number of context keys | — |
+| `visor.check.input.context` | Full template context (JSON, sanitized) | 10 KB |
+| `visor.check.input.pr` | PR information subset | 1 KB |
+| `visor.check.input.outputs` | Previous check outputs | 5 KB |
+| `visor.check.input.env_keys` | Environment variable keys available | — |
+
+### Check Output
+
+| Attribute | Description | Max Size |
+|-----------|-------------|----------|
+| `visor.check.output.type` | Output type (`object`, `array`, `string`) | — |
+| `visor.check.output.length` | Array length (if output is array) | — |
+| `visor.check.output.preview` | Preview of first 10 items | 2 KB |
+| `visor.check.output` | Full serialized output | 10 KB |
+
+### Provider Call Details
+
+| Attribute | Description | Max Size |
+|-----------|-------------|----------|
+| `visor.provider.type` | Provider type | — |
+| `visor.provider.request.model` | AI model name | — |
+| `visor.provider.request.prompt.length` | Prompt character count | — |
+| `visor.provider.request.prompt.preview` | Prompt preview | 500 B |
+| `visor.provider.request.prompt` | Full prompt (full capture only) | 10 KB |
+| `visor.provider.response.length` | Response character count | — |
+| `visor.provider.response.preview` | Response preview | 500 B |
+| `visor.provider.response.content` | Full response (full capture only) | 10 KB |
+| `visor.provider.response.tokens` | Token count (if available) | — |
+
+### Liquid Template Evaluation
+
+| Attribute | Description | Max Size |
+|-----------|-------------|----------|
+| `visor.liquid.template` | Template source | 1 KB |
+| `visor.liquid.result` | Rendered result | 2 KB |
+| `visor.liquid.context` | Template context | 3 KB |
+
+### Transform JS
+
+| Attribute | Description | Max Size |
+|-----------|-------------|----------|
+| `visor.transform.code` | JavaScript code | 2 KB |
+| `visor.transform.input` | Input data | 2 KB |
+| `visor.transform.output` | Output data | 2 KB |
+
+### Conditional Evaluation
+
+| Attribute | Description | Max Size |
+|-----------|-------------|----------|
+| `visor.condition.expression` | Condition expression | 500 B |
+| `visor.condition.result` | Boolean evaluation result | — |
+| `visor.condition.context` | Evaluation context | 2 KB |
+
+### Routing Decision
+
+| Attribute | Description |
+|-----------|-------------|
+| `visor.routing.action` | Action: `retry`, `goto`, `run` |
+| `visor.routing.target` | Target check(s) or event |
+| `visor.routing.condition` | Routing condition expression |
+
+---
+
+## Resource Attributes
+
+The OTel resource is configured with:
+
+| Attribute | Value |
+|-----------|-------|
+| `service.name` | `visor` |
+| `service.version` | Visor version or `dev` |
+| `deployment.environment` | `github-actions`, `ci`, or `local` |
+
+---
+
+## Example Queries
+
+### Prometheus / Grafana
+
+```promql
+# Total runs per user (last 24h)
+sum by (visor_run_user_id) (increase(visor_run_total[24h]))
+
+# Average run duration by workflow
+avg by (visor_run_workflow) (visor_run_duration_ms_sum / visor_run_duration_ms_count)
+
+# AI calls per run (average)
+avg(visor_run_ai_calls_sum / visor_run_ai_calls_count)
+
+# AI calls by model (rate)
+sum by (visor_ai_model) (rate(visor_ai_call_total[5m]))
+
+# P95 check duration
+histogram_quantile(0.95, sum(rate(visor_check_duration_ms_bucket[5m])) by (le, visor_check_id))
+
+# Failed runs rate
+sum(rate(visor_run_duration_ms_count{visor_run_success="false"}[5m]))
+
+# Active concurrent checks
+visor_run_active_checks
+```
+
+### Tempo (TraceQL)
+
+```
+# Find all runs from Slack
+{ resource.service.name = "visor" && name = "visor.run" && span.visor.run.source = "slack" }
+
+# Find slow checks (>30s)
+{ resource.service.name = "visor" && name =~ "visor.check.*" } | duration > 30s
+
+# Find runs with many AI calls
+{ resource.service.name = "visor" && name = "visor.run" && span.visor.run.ai_calls > 10 }
+
+# Find failed routing decisions
+{ resource.service.name = "visor" } >> { name = "visor.routing" && span.action = "retry" }
+```
+
+---
+
+## Related Documentation
+
+- [Telemetry Setup Guide](./telemetry-setup.md) — How to enable and configure telemetry
+- [Debugging Guide](./debugging.md) — Debugging techniques including tracing
+- [Grafana Dashboards](./dashboards/) — Pre-built dashboards for Visor telemetry

package/dist/docs/telemetry-setup.md

@@ -262,6 +262,8 @@ For file-based tracing in CI (useful for artifact upload):
 
 ## Related Documentation
 
+- [Telemetry Reference](./telemetry-reference.md) — Complete reference of all spans, metrics, and events
+- [Grafana Dashboards](./dashboards/) — Pre-built dashboards for Visor telemetry
 - [Debugging Guide](./debugging.md) — Comprehensive debugging techniques
 - [Debug Visualizer](./debug-visualizer.md) — Live execution visualization with `--debug-server`
 - [Telemetry RFC](./telemetry-tracing-rfc.md) — Design rationale and architecture