@probelabs/visor 0.1.124 → 0.1.126

package/dist/docs/observability.md

@@ -1,12 +1,233 @@
-## 👀 Observability
+# Observability
 
-- Machine-readable output: `--output json` or `--output sarif`.
-- Prefer the built‑in `--output-file <path>` to save results without touching stdout.
-- Status/progress logs are written to stderr; control verbosity via `-q`, `-v`, or `--debug`.
+This document provides an overview of Visor's observability features, including output formats, logging verbosity, telemetry, debugging tools, and execution statistics.
 
-Examples:
+## Output Formats
 
-```
+Visor supports four output formats for different use cases:
+
+| Format | Flag | Description |
+|--------|------|-------------|
+| **table** | `--output table` | Default terminal summary with colored output (when TTY) |
+| **json** | `--output json` | Machine-readable JSON for pipelines and automation |
+| **markdown** | `--output markdown` | Render results as markdown (useful for PR comments) |
+| **sarif** | `--output sarif` | SARIF 2.1.0 format for code scanning integrations |
+
+### Saving Output to Files
+
+Use `--output-file <path>` to write formatted results directly to a file without mixing with logs:
+
+```bash
 visor --check all --output json --output-file results.json
 visor --check security --output sarif --output-file visor-results.sarif
+visor --check architecture --output markdown --output-file report.md
+```
+
+All status logs are sent to stderr; stdout contains only the formatted result when not using `--output-file`.
+
+For more details, see [Output Formats](./output-formats.md).
+
+## Verbosity Control
+
+Visor supports multiple log levels, from silent to debug:
+
+| Level | Priority | Description |
+|-------|----------|-------------|
+| `silent` | 0 | No output |
+| `error` | 10 | Errors only |
+| `warn` | 20 | Warnings and errors |
+| `info` | 30 | Default level - informational messages |
+| `verbose` | 40 | Additional detail without full debug |
+| `debug` | 50 | Full debug output including AI interactions |
+
+### CLI Flags
+
+| Flag | Effect |
+|------|--------|
+| `-q`, `--quiet` | Reduce verbosity to warnings and errors |
+| `-v`, `--verbose` | Increase verbosity without full debug |
+| `--debug` | Enable debug mode for detailed output |
+
+### Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `VISOR_LOG_LEVEL` | Set log level (`silent`, `error`, `warn`, `info`, `verbose`, `debug`, or `quiet`) |
+| `VISOR_DEBUG` | Set to `true` to enable debug mode |
+
+### JSON/SARIF Output Behavior
+
+When using `--output json` or `--output sarif`, Visor automatically suppresses info and warning logs to keep stdout clean for machine-readable output. This can be overridden by explicitly setting `--verbose` or `--debug`.
+
+## Telemetry and Tracing
+
+Visor supports OpenTelemetry-based telemetry for tracing and metrics.
+
+### Environment Variables
+
+| Variable | Description |
+|----------|-------------|
+| `VISOR_TELEMETRY_ENABLED` | Set to `true` to enable telemetry |
+| `VISOR_TELEMETRY_SINK` | Sink type: `file` (NDJSON), `otlp` (HTTP), or `console` |
+| `VISOR_TRACE_DIR` | Directory for trace output (default: `output/traces`) |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP endpoint URL for traces |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | OTLP endpoint specifically for traces |
+| `OTEL_EXPORTER_OTLP_HEADERS` | Headers for OTLP authentication |
+
+### CLI Flags
+
+| Flag | Description |
+|------|-------------|
+| `--telemetry` | Enable telemetry (overrides config) |
+| `--telemetry-sink <sink>` | Sink selection: `otlp`, `file`, or `console` |
+| `--telemetry-endpoint <url>` | OTLP endpoint (HTTP) for traces/metrics |
+| `--trace-report` | Write a static HTML trace report to output/traces |
+| `--auto-instrumentations` | Enable OpenTelemetry auto-instrumentations |
+
+### NDJSON Trace Output (Serverless Mode)
+
+When using file sink, Visor writes simplified NDJSON spans to `output/traces/run-<timestamp>.ndjson`. These can be ingested with an OTel Collector `filelog` receiver.
+
+```bash
+VISOR_TELEMETRY_ENABLED=true VISOR_TELEMETRY_SINK=file visor --check all
+ls output/traces/*.ndjson
+```
+
+### OpenTelemetry OTLP Export (Connected Mode)
+
+For real-time trace export to collectors like Jaeger or Tempo:
+
+```bash
+VISOR_TELEMETRY_ENABLED=true \
+VISOR_TELEMETRY_SINK=otlp \
+OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces \
+visor --check all
+```
+
+### Configuration File
+
+```yaml
+version: "1.0"
+telemetry:
+  enabled: true
+  sink: file       # otlp|file|console
+  otlp:
+    protocol: http
+    endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
+    headers: ${OTEL_EXPORTER_OTLP_HEADERS}
+  file:
+    dir: output/traces
+    ndjson: true
+  tracing:
+    auto_instrumentations: true
+    trace_report:
+      enabled: true
 ```
+
+For comprehensive setup instructions, see [Telemetry Setup](./telemetry-setup.md).
+
+## Debug Features
+
+### Debug Visualizer Server
+
+The Debug Visualizer is a lightweight HTTP server that streams OpenTelemetry spans during execution and exposes control endpoints for pause, resume, stop, and reset.
+
+| Flag | Description |
+|------|-------------|
+| `--debug-server` | Start debug visualizer server for live execution visualization |
+| `--debug-port <port>` | Port for debug server (default: 3456) |
+
+```bash
+visor --config .visor.yaml --debug-server --debug-port 3456
+```
+
+The server provides endpoints:
+- `GET /api/status` - Execution state and readiness
+- `GET /api/spans` - Live span stream
+- `POST /api/start` - Begin execution
+- `POST /api/pause` - Pause execution
+- `POST /api/resume` - Resume execution
+- `POST /api/stop` - Stop execution
+
+Set `VISOR_NOBROWSER=true` for headless/CI environments.
+
+For full details, see [Debug Visualizer](./debug-visualizer.md).
+
+### TUI Mode
+
+Enable interactive terminal UI with chat and logs tabs:
+
+```bash
+visor --tui --check all
+```
+
+### Debug Mode
+
+Enable `--debug` for detailed output including:
+- AI provider interactions
+- Template rendering details
+- Expression evaluation results
+- Dependency resolution paths
+- Error stack traces
+
+For comprehensive debugging techniques, see [Debugging Guide](./debugging.md).
+
+## Execution Statistics
+
+Visor tracks and displays detailed execution statistics for each run.
+
+### Summary Information
+
+After execution, Visor displays:
+- Total checks configured vs. total executions (including forEach iterations)
+- Success, failure, and skip counts
+- Total execution duration
+- Issue counts by severity
+
+### Table Output Example
+
+```
+Execution Complete (45.3s)
+Checks: 8 configured -> 23 executions
+Status: 20 success | 2 failed | 1 skipped
+Issues: 15 total (3 critical, 12 warnings)
+
+Check Details:
+| Check           | Duration | Status   | Details         |
+|-----------------|----------|----------|-----------------|
+| list-files      | 0.5s     | success  | 5 outputs       |
+| validate-file   | 12.3s    | success  | 5 iterations    |
+| security-scan   | 8.2s     | success  | 3 critical      |
+| notify-slack    | 2.1s     | failed   | HTTP 500        |
+| production-only | -        | skipped  | if: branch==... |
+```
+
+### JSON Output
+
+When using `--output json`, full `executionStatistics` object is included with:
+- Per-check statistics (runs, duration, issues)
+- Per-iteration timings for forEach checks
+- Skip condition evaluations
+- forEach item previews
+- Complete error messages
+
+### Tracked Metrics
+
+| Metric | Description |
+|--------|-------------|
+| `totalChecksConfigured` | Number of checks defined in config |
+| `totalExecutions` | Sum of all runs including forEach iterations |
+| `successfulExecutions` | Checks that completed without error |
+| `failedExecutions` | Checks that failed |
+| `skippedChecks` | Checks skipped due to conditions |
+| `totalDuration` | Total execution time in milliseconds |
+| Issue counts | By severity: critical, error, warning, info |
+
+## Related Documentation
+
+- [Output Formats](./output-formats.md) - Detailed format specifications
+- [Telemetry Setup](./telemetry-setup.md) - Complete telemetry configuration
+- [Debug Visualizer](./debug-visualizer.md) - Live execution visualization
+- [Debugging Guide](./debugging.md) - Debugging techniques and tips
+- [Troubleshooting](./troubleshooting.md) - Common issues and solutions
+- [Configuration](./configuration.md) - Full configuration reference

package/dist/docs/output-formats.md

@@ -1,20 +1,399 @@
-## 📊 Output Formats
+## Output Formats
 
-- Table: default terminal summary
-- JSON: `--output json` for pipelines
-- Markdown: render comments as markdown
-- SARIF 2.1.0: `--output sarif` for code scanning
+Visor supports multiple output formats to suit different use cases, from terminal-friendly tables to machine-readable formats for CI/CD pipelines.
 
-### Saving outputs reliably
+### Command-line Options
 
-- Use the built‑in `--output-file <path>` to write the formatted result directly to a file without mixing with logs.
-- All status logs are sent to stderr; stdout contains only the formatted result when not using `--output-file`.
+```bash
+# Short form
+visor -o json
 
-Examples:
+# Long form
+visor --output json
 
+# Write to file instead of stdout
+visor --output sarif --output-file results.sarif
 ```
+
+Supported formats: `table` (default), `json`, `markdown`, `sarif`
+
+---
+
+## Table Format (Default)
+
+The table format provides a colored, human-readable summary optimized for terminal output.
+
+### Structure
+
+```
+Analysis Summary
+┌─────────────────────────┬──────────────────────────────┐
+│ Metric                  │ Value                        │
+├─────────────────────────┼──────────────────────────────┤
+│ Total Issues            │ 5                            │
+│ Critical Issues         │ 1                            │
+│ Files Analyzed          │ 12                           │
+│ Total Additions         │ 245                          │
+│ Total Deletions         │ 89                           │
+│ Execution Time          │ 3421ms                       │
+│ Checks Executed         │ security, performance        │
+└─────────────────────────┴──────────────────────────────┘
+
+SECURITY Issues (2)
+┌─────────────────────────┬────────┬───────────────┬────────────────────────────────────────────────────────────┐
+│ File                    │ Line   │ Severity      │ Message                                                    │
+├─────────────────────────┼────────┼───────────────┼────────────────────────────────────────────────────────────┤
+│ src/auth.ts             │ 42     │ CRITICAL      │ SQL injection vulnerability in user query                  │
+│ src/utils.ts            │ 15     │ WARNING       │ Potential XSS via unsanitized input                        │
+└─────────────────────────┴────────┴───────────────┴────────────────────────────────────────────────────────────┘
+```
+
+### Columns
+
+| Column   | Width | Description                                    |
+|----------|-------|------------------------------------------------|
+| File     | 25    | Filename (truncated with `...` if too long)    |
+| Line     | 8     | Line number of the issue                       |
+| Severity | 15    | INFO, WARNING, ERROR, or CRITICAL              |
+| Message  | 60    | Issue description with suggestions/code fixes  |
+
+### Categories
+
+Issues are grouped by category when `groupByCategory: true` (default):
+- **Security** - Input validation, injection, authentication issues
+- **Performance** - Optimization opportunities, resource management
+- **Style** - Naming conventions, formatting inconsistencies
+- **Logic** - Complexity, potential bugs, control flow issues
+- **Documentation** - Missing or incomplete documentation
+
+### Colored Output
+
+When output is a TTY, severity levels are colorized:
+- `critical` / `error` - Red
+- `warning` - Yellow
+- `info` - Cyan
+
+### Truncation Behavior
+
+Table cells are truncated to prevent rendering slowdowns:
+- Maximum cell characters: 4000 (configurable via `VISOR_MAX_TABLE_CELL`)
+- Maximum code block lines: 120 (configurable via `VISOR_MAX_TABLE_CODE_LINES`)
+- Truncated content shows: `... [truncated]`
+
+See [Output Formatting](./output-formatting.md) for detailed truncation controls.
+
+---
+
+## JSON Format
+
+Machine-readable format ideal for programmatic processing and CI/CD pipelines.
+
+```bash
+visor -o json --output-file results.json
+```
+
+### Schema
+
+```json
+{
+  "summary": {
+    "totalIssues": 5,
+    "criticalIssues": 1,
+    "executionTime": 3421,
+    "timestamp": "2024-01-15T10:30:00.000Z",
+    "checksExecuted": ["security", "performance"]
+  },
+  "repository": {
+    "title": "Fix authentication flow",
+    "author": "developer",
+    "base": "main",
+    "head": "feature/auth-fix",
+    "isGitRepository": true,
+    "workingDirectory": "/home/user/project",
+    "filesChanged": 12,
+    "totalAdditions": 245,
+    "totalDeletions": 89
+  },
+  "issues": {
+    "security": [
+      {
+        "file": "src/auth.ts",
+        "line": 42,
+        "message": "SQL injection vulnerability in user query",
+        "severity": "critical",
+        "category": "security"
+      }
+    ],
+    "performance": [],
+    "style": [],
+    "logic": [],
+    "documentation": []
+  },
+  "files": [
+    {
+      "filename": "src/auth.ts",
+      "status": "modified",
+      "additions": 45,
+      "deletions": 12
+    }
+  ],
+  "debug": {
+    "provider": "openai",
+    "model": "gpt-4",
+    "processingTime": 2100,
+    "parallelExecution": true
+  },
+  "failureConditions": []
+}
+```
+
+### Fields
+
+| Field               | Type     | Description                                      |
+|---------------------|----------|--------------------------------------------------|
+| `summary`           | object   | Execution metrics and totals                     |
+| `repository`        | object   | Repository and PR metadata                       |
+| `issues`            | object   | Issues grouped by category                       |
+| `files`             | array    | Changed files with diff statistics (optional)    |
+| `debug`             | object   | Debug info when `--debug` enabled                |
+| `failureConditions` | array    | Results of `fail_if` conditions                  |
+
+---
+
+## Markdown Format
+
+GitHub-flavored markdown for embedding in PRs, wikis, or documentation.
+
+```bash
+visor -o markdown --output-file report.md
+```
+
+### Structure
+
+```markdown
+# Visor Analysis Results
+<!-- analysis results -->
+
+## Summary
+
+| Metric | Value |
+|--------|-------|
+| Total Issues | 5 |
+| Critical Issues | 1 |
+| Files Analyzed | 12 |
+| Execution Time | 3421ms |
+| Checks Executed | security, performance |
+
+## Repository Information
+
+- **Title**: Fix authentication flow
+- **Author**: developer
+- **Branch**: feature/auth-fix <- main
+- **Working Directory**: `/home/user/project`
+- **Changes**: +245/-89
+
+## Security Issues (2 found)
+
+### `src/auth.ts:42`
+**Severity**: CRITICAL
+**Message**: SQL injection vulnerability in user query
+**Suggestion**: Use parameterized queries instead of string concatenation
+
+**Suggested Fix**:
+```typescript
+const result = await db.query('SELECT * FROM users WHERE id = ?', [userId]);
+```
+
+<details>
+<summary>Show 3 more issues...</summary>
+<!-- Additional issues collapsed by default -->
+</details>
+```
+
+### Sections
+
+1. **Summary** - Metrics table with issue counts and execution stats
+2. **Repository Information** - PR/branch context
+3. **Issues by Category** - Grouped with severity, message, suggestions, and code fixes
+4. **Files Changed** - Optional diff statistics table
+
+---
+
+## SARIF Format
+
+[SARIF 2.1.0](https://sarifweb.azurewebsites.net/) format for integration with GitHub Code Scanning, VS Code, and other security tools.
+
+```bash
+visor -o sarif --output-file visor-results.sarif
+```
+
+### Schema
+
+```json
+{
+  "$schema": "https://json.schemastore.org/sarif-2.1.0.json",
+  "version": "2.1.0",
+  "runs": [
+    {
+      "tool": {
+        "driver": {
+          "name": "Visor",
+          "version": "1.0.0",
+          "informationUri": "https://github.com/your-org/visor",
+          "rules": [
+            {
+              "id": "visor-security-input-validation",
+              "shortDescription": { "text": "Input validation required" },
+              "fullDescription": { "text": "Input validation and sanitization should be implemented..." },
+              "helpUri": "https://owasp.org/www-project-top-ten/2017/A1_2017-Injection"
+            }
+          ]
+        }
+      },
+      "results": [
+        {
+          "ruleId": "visor-security-input-validation",
+          "ruleIndex": 0,
+          "level": "error",
+          "message": { "text": "SQL injection vulnerability in user query" },
+          "locations": [
+            {
+              "physicalLocation": {
+                "artifactLocation": {
+                  "uri": "src/auth.ts",
+                  "uriBaseId": "%SRCROOT%"
+                },
+                "region": {
+                  "startLine": 42,
+                  "startColumn": 1
+                }
+              }
+            }
+          ]
+        }
+      ]
+    }
+  ]
+}
+```
+
+### Severity Mapping
+
+| Visor Severity | SARIF Level |
+|----------------|-------------|
+| `critical`     | `error`     |
+| `error`        | `error`     |
+| `warning`      | `warning`   |
+| `info`         | `note`      |
+
+### Rule IDs
+
+| Category      | Rule ID                          |
+|---------------|----------------------------------|
+| security      | `visor-security-input-validation`|
+| performance   | `visor-performance-optimization` |
+| style         | `visor-style-consistency`        |
+| logic         | `visor-logic-complexity`         |
+| documentation | `visor-documentation-missing`    |
+
+---
+
+## Environment Variables
+
+| Variable                    | Default | Description                                          |
+|-----------------------------|---------|------------------------------------------------------|
+| `VISOR_OUTPUT_FORMAT`       | `table` | Default output format (overridden by `-o`/`--output`)|
+| `VISOR_MAX_TABLE_CELL`      | `4000`  | Maximum characters per table cell before truncation  |
+| `VISOR_MAX_TABLE_CODE_LINES`| `120`   | Maximum code block lines in table cells              |
+
+Example:
+```bash
+# Set default format via environment
+export VISOR_OUTPUT_FORMAT=json
+visor --config .visor.yaml  # Uses JSON format
+
+# Override environment with CLI flag
+visor -o table --config .visor.yaml  # Uses table format
+```
+
+---
+
+## Use-Case Guidance
+
+### When to Use Each Format
+
+| Format   | Best For                                                    |
+|----------|-------------------------------------------------------------|
+| Table    | Interactive terminal use, quick human review                |
+| JSON     | CI/CD pipelines, programmatic processing, API responses     |
+| Markdown | PR comments, documentation, reports for stakeholders        |
+| SARIF    | GitHub Code Scanning, IDE integrations, security dashboards |
+
+### CI/CD Integration Patterns
+
+**Basic CI Pipeline (JSON for parsing)**:
+```yaml
+- name: Run Visor
+  run: |
+    visor -o json --output-file results.json
+    CRITICAL=$(jq '.summary.criticalIssues' results.json)
+    if [ "$CRITICAL" -gt 0 ]; then
+      echo "Critical issues found!"
+      exit 1
+    fi
+```
+
+**GitHub Actions with SARIF Upload**:
+```yaml
+jobs:
+  visor:
+    runs-on: ubuntu-latest
+    permissions:
+      security-events: write
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Run Visor
+        run: npx -y @probelabs/visor@latest -o sarif --output-file visor-results.sarif
+        env:
+          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
+
+      - name: Upload SARIF
+        uses: github/codeql-action/upload-sarif@v3
+        with:
+          sarif_file: visor-results.sarif
+          category: visor
+```
+
+**PR Comment with Markdown**:
+```yaml
+- name: Run Visor
+  run: visor -o markdown --output-file report.md
+
+- name: Comment on PR
+  uses: marocchino/sticky-pull-request-comment@v2
+  with:
+    path: report.md
+```
+
+### Saving Output Reliably
+
+Use `--output-file` to write results directly to a file without mixing with logs:
+
+```bash
 visor --check all --output json --output-file results.json
 visor --check security --output sarif --output-file visor-results.sarif
 visor --check architecture --output markdown --output-file report.md
 visor --check style --output table --output-file summary.txt
 ```
+
+All status logs are sent to stderr; stdout contains only the formatted result when not using `--output-file`.
+
+---
+
+## Related Documentation
+
+- [Output Formatting](./output-formatting.md) - Table safety limits and truncation controls
+- [CI/CLI Mode](./ci-cli-mode.md) - Running Visor in CI environments
+- [Configuration](./configuration.md) - Full configuration options
+- [Failure Conditions](./fail-if.md) - Exit codes and `fail_if` conditions