@probelabs/visor 0.1.106 → 0.1.111

package/dist/docs/custom-tools.md

@@ -0,0 +1,424 @@
+# Custom Tools in YAML Configuration
+
+## Overview
+
+Custom tools allow you to define reusable command-line tools directly in your YAML configuration. These tools can then be used in MCP (Model Context Protocol) blocks throughout your configuration, making it easy to integrate any command-line tool or script into your workflow.
+
+## Features
+
+- **Define tools in YAML**: No need to create separate scripts or programs
+- **Input validation**: Define JSON Schema for tool parameters
+- **Template support**: Use Liquid templates for dynamic command generation
+- **Transform outputs**: Process tool output with Liquid templates or JavaScript
+- **Reusable**: Define once, use multiple times across your configuration
+- **Importable**: Share tools across projects using the `extends` mechanism
+- **Type-safe**: Full TypeScript support with input/output schemas
+- **MCP-compatible**: Tools follow the Model Context Protocol specification
+
+## Basic Tool Definition
+
+```yaml
+tools:
+  my-tool:
+    name: my-tool
+    description: Description of what the tool does
+    exec: 'echo "Hello World"'
+```
+
+## Complete Tool Schema
+
+```yaml
+tools:
+  tool-name:
+    # MCP-compatible fields (these map directly to MCP tool interface)
+    name: tool-name                    # Required: Tool identifier (MCP: name)
+    description: Tool description       # Recommended: Human-readable description (MCP: description)
+
+    # Input schema (JSON Schema format) - MCP: inputSchema
+    # This follows the JSON Schema specification and is used for:
+    # 1. Validating tool inputs before execution
+    # 2. Providing type information to AI models
+    # 3. Auto-generating documentation
+    inputSchema:
+      type: object
+      properties:
+        param1:
+          type: string
+          description: Parameter description  # Describe each parameter for AI models
+        param2:
+          type: number
+          description: Optional parameter
+      required: [param1]                     # List required parameters
+      additionalProperties: false            # Strict mode: reject unknown parameters
+
+    # Custom tool execution fields
+    exec: 'command {{ args.param1 }}'  # Required: Command to execute (supports Liquid)
+    stdin: '{{ args.param2 }}'         # Optional: Data to pipe to stdin (supports Liquid)
+
+    cwd: /path/to/directory            # Optional: Working directory
+    env:                                # Optional: Environment variables
+      MY_VAR: value
+
+    timeout: 30000                      # Optional: Timeout in milliseconds (default: 30000)
+    parseJson: true                     # Optional: Parse output as JSON
+
+    # Transform output with Liquid template
+    transform: '{ "result": {{ output | json }} }'
+
+    # OR transform with JavaScript
+    transform_js: |
+      return {
+        processed: output.trim().toUpperCase()
+      };
+
+    # Output schema for validation (optional) - MCP: outputSchema
+    # Not currently enforced but useful for documentation
+    outputSchema:
+      type: object
+      properties:
+        result:
+          type: string
+          description: The processed result
+```
+
+## MCP Compatibility
+
+Custom tools are designed to be fully compatible with the Model Context Protocol (MCP) specification. When you define a custom tool, it automatically becomes available as an MCP tool with the following mapping:
+
+| Custom Tool Field | MCP Tool Field | Purpose |
+|------------------|----------------|---------|
+| `name` | `name` | Unique identifier for the tool |
+| `description` | `description` | Human-readable description for AI models and documentation |
+| `inputSchema` | `inputSchema` | JSON Schema defining expected parameters |
+| `outputSchema` | `outputSchema` | JSON Schema for output validation (informational) |
+
+### Why MCP Compatibility Matters
+
+1. **AI Model Integration**: Tools with proper descriptions and schemas can be automatically understood and used by AI models
+2. **Type Safety**: Input schemas provide runtime validation and type checking
+3. **Documentation**: Schemas serve as self-documenting interfaces
+4. **Interoperability**: Tools can potentially be used with other MCP-compatible systems
+
+### Best Practices for MCP Compatibility
+
+1. **Always provide descriptions**: Help AI models understand what your tool does
+   ```yaml
+   tools:
+     analyze-code:
+       name: analyze-code
+       description: "Analyzes source code for complexity metrics and potential issues"
+   ```
+
+2. **Use detailed input schemas**: Include descriptions for each parameter
+   ```yaml
+   inputSchema:
+     type: object
+     properties:
+       file:
+         type: string
+         description: "Path to the source code file to analyze"
+       metrics:
+         type: array
+         description: "List of metrics to calculate"
+         items:
+           type: string
+           enum: ["complexity", "lines", "dependencies"]
+     required: ["file"]
+   ```
+
+3. **Consider output schemas**: While not enforced, they document expected outputs
+   ```yaml
+   outputSchema:
+     type: object
+     properties:
+       complexity:
+         type: number
+         description: "Cyclomatic complexity score"
+       issues:
+         type: array
+         description: "List of detected issues"
+   ```
+
+## Using Custom Tools
+
+Once defined, custom tools can be used in any MCP block by setting `transport: custom`:
+
+```yaml
+steps:
+  my-check:
+    type: mcp
+    transport: custom              # Use custom transport
+    method: my-tool                 # Tool name
+    methodArgs:                     # Tool arguments
+      param1: "value1"
+      param2: 42
+```
+
+## Template Context
+
+Tools have access to a rich template context through Liquid templates:
+
+### In `exec` and `stdin`:
+- `{{ args }}` - The arguments passed to the tool
+- `{{ pr }}` - Pull request information (number, title, author, etc.)
+- `{{ files }}` - List of files in the PR
+- `{{ outputs }}` - Outputs from previous checks
+- `{{ env }}` - Environment variables
+
+### In `transform` and `transform_js`:
+- All of the above, plus:
+- `{{ output }}` - The raw command output
+- `{{ stdout }}` - Standard output
+- `{{ stderr }}` - Standard error
+- `{{ exitCode }}` - Command exit code
+
+## Examples
+
+### 1. Simple Grep Tool
+
+```yaml
+tools:
+  grep-todos:
+    name: grep-todos
+    description: Find TODO comments in code
+    inputSchema:
+      type: object
+      properties:
+        pattern:
+          type: string
+        files:
+          type: array
+          items:
+            type: string
+    exec: 'grep -n "{{ args.pattern }}" {{ args.files | join: " " }}'
+```
+
+### 2. JSON Processing Tool
+
+```yaml
+tools:
+  analyze-package:
+    name: analyze-package
+    description: Analyze package.json dependencies
+    inputSchema:
+      type: object
+      properties:
+        file:
+          type: string
+    exec: 'cat {{ args.file }}'
+    parseJson: true
+    transform_js: |
+      const deps = Object.keys(output.dependencies || {});
+      const devDeps = Object.keys(output.devDependencies || {});
+      return {
+        totalDeps: deps.length + devDeps.length,
+        prodDeps: deps.length,
+        devDeps: devDeps.length
+      };
+```
+
+### 3. Multi-Step Tool with Error Handling
+
+```yaml
+tools:
+  build-and-test:
+    name: build-and-test
+    description: Build project and run tests
+    exec: |
+      npm run build && npm test
+    timeout: 300000  # 5 minutes
+    transform_js: |
+      if (exitCode !== 0) {
+        return {
+          success: false,
+          error: stderr || 'Build or tests failed'
+        };
+      }
+      return {
+        success: true,
+        output: output
+      };
+```
+
+### 4. Tool with Dynamic Command Generation
+
+```yaml
+tools:
+  flexible-linter:
+    name: flexible-linter
+    description: Run appropriate linter based on file type
+    inputSchema:
+      type: object
+      properties:
+        file:
+          type: string
+    exec: |
+      {% assign ext = args.file | split: "." | last %}
+      {% case ext %}
+        {% when "js", "ts" %}
+          eslint {{ args.file }}
+        {% when "py" %}
+          pylint {{ args.file }}
+        {% when "go" %}
+          golint {{ args.file }}
+        {% else %}
+          echo "No linter for .{{ ext }} files"
+      {% endcase %}
+```
+
+## Tool Libraries and Extends
+
+### Creating a Tool Library
+
+Create a file with just tool definitions:
+
+```yaml
+# tools-library.yaml
+version: "1.0"
+
+tools:
+  tool1:
+    name: tool1
+    exec: 'command1'
+
+  tool2:
+    name: tool2
+    exec: 'command2'
+```
+
+### Importing Tools
+
+Use the `extends` mechanism to import tools:
+
+```yaml
+version: "1.0"
+extends: ./tools-library.yaml
+
+# Additional tools can be defined here
+tools:
+  local-tool:
+    name: local-tool
+    exec: 'local-command'
+
+# Use both imported and local tools
+steps:
+  check1:
+    type: mcp
+    transport: custom
+    method: tool1  # From tools-library.yaml
+
+  check2:
+    type: mcp
+    transport: custom
+    method: local-tool  # Defined locally
+```
+
+### Multiple Extends
+
+You can import from multiple sources:
+
+```yaml
+extends:
+  - ./base-tools.yaml
+  - ./security-tools.yaml
+  - https://example.com/shared-tools.yaml
+```
+
+Tools are merged with later sources overriding earlier ones.
+
+## Integration with Other Features
+
+### Using with forEach
+
+```yaml
+steps:
+  lint-all-files:
+    type: mcp
+    transport: custom
+    method: my-linter
+    forEach: "{{ files }}"
+    methodArgs:
+      file: "{{ item.filename }}"
+```
+
+### Conditional Execution
+
+```yaml
+steps:
+  optional-check:
+    type: mcp
+    transport: custom
+    method: my-tool
+    if: "files.some(f => f.filename.endsWith('.js'))"
+    methodArgs:
+      target: "src/"
+```
+
+### Chaining with on_success/on_failure
+
+```yaml
+steps:
+  main-check:
+    type: mcp
+    transport: custom
+    method: build-tool
+    on_success:
+      - type: mcp
+        transport: custom
+        method: test-tool
+    on_failure:
+      - type: mcp
+        transport: custom
+        method: cleanup-tool
+```
+
+## Best Practices
+
+1. **Use Input Schemas**: Always define `inputSchema` to validate tool inputs
+2. **Handle Errors**: Use `transform_js` to check exit codes and handle errors
+3. **Set Timeouts**: Configure appropriate timeouts for long-running commands
+4. **Parse JSON**: Use `parseJson: true` for tools that output JSON
+5. **Document Tools**: Provide clear descriptions for each tool
+6. **Create Libraries**: Group related tools in separate YAML files
+7. **Version Control**: Store tool libraries in version control for sharing
+8. **Test Tools**: Test tools independently before using in complex workflows
+
+## Security Considerations
+
+- Tools execute with the same permissions as the Visor process
+- Be cautious with user input in tool commands
+- Use input validation to prevent command injection
+- Avoid exposing sensitive data in tool outputs
+- Consider using environment variables for secrets
+
+## Troubleshooting
+
+### Tool Not Found
+
+If you get "Tool not found" errors:
+1. Ensure the tool is defined in the `tools` section
+2. Check that the tool name matches exactly
+3. Verify extends paths are correct
+
+### Command Failures
+
+For command execution issues:
+1. Test the command manually first
+2. Check working directory (`cwd`) settings
+3. Verify required binaries are installed
+4. Check timeout settings for long operations
+
+### Template Errors
+
+For Liquid template problems:
+1. Validate template syntax
+2. Check that variables exist in context
+3. Use filters correctly (e.g., `| json`, `| join`)
+
+### Transform Errors
+
+For JavaScript transform issues:
+1. Ensure valid JavaScript syntax
+2. Always return a value
+3. Handle undefined/null cases
+4. Use try-catch for error handling

package/dist/docs/dashboards/README.md

@@ -0,0 +1,23 @@
+Dashboards for Visor Telemetry
+==============================
+
+This folder contains example Grafana dashboards to visualize Visor traces and metrics exported via OpenTelemetry (Tempo + Prometheus).
+
+What’s included
+- grafana-visor-overview.json — Run/Check overview: durations, issue counts by severity, recent runs.
+- grafana-visor-diagrams.json — Diagram telemetry (syntax pass rate, average score) if you enable diagram attributes.
+
+Setup
+1) Deploy Grafana + Tempo + Prometheus (or Grafana Cloud).
+2) Configure your OTel Collector to receive OTLP traces/metrics and forward to Tempo/Prometheus.
+3) Enable telemetry in Visor CI:
+   - VISOR_TELEMETRY_ENABLED=true
+   - VISOR_TELEMETRY_SINK=otlp
+   - OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example.com
+   - OTEL_EXPORTER_OTLP_HEADERS=Authorization=Bearer <token>
+4) Import the JSON dashboards into Grafana and set the Tempo + Prometheus data sources.
+
+Notes
+- Spans appear in Tempo’s Explore/Trace view (service.name=visor).
+- Metrics are emitted when OTLP metrics exporter is configured; see docs/telemetry-tracing-rfc.md for details.
+

package/dist/docs/dashboards/grafana-visor-diagrams.json

@@ -0,0 +1,20 @@
+{
+  "__inputs": [],
+  "title": "Visor Diagrams",
+  "timezone": "browser",
+  "panels": [
+    {
+      "type": "bargauge",
+      "title": "Diagram Blocks by Origin (5m rate)",
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "PROM_DS" },
+          "expr": "sum(rate(visor_diagram_blocks_total[5m])) by (origin)"
+        }
+      ],
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }
+    }
+  ],
+  "version": 1,
+  "editable": true
+}

package/dist/docs/dashboards/grafana-visor-overview.json

@@ -0,0 +1,33 @@
+{
+  "__inputs": [],
+  "title": "Visor Overview",
+  "timezone": "browser",
+  "panels": [
+    {
+      "type": "timeseries",
+      "title": "Check Duration (ms)",
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "PROM_DS" },
+          "expr": "histogram_quantile(0.95, sum(rate(visor_check_duration_ms_bucket[5m])) by (le))"
+        }
+      ],
+      "fieldConfig": { "defaults": { "unit": "ms" }, "overrides": [] },
+      "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 }
+    },
+    {
+      "type": "stat",
+      "title": "Issues by Severity (rate)",
+      "targets": [
+        {
+          "datasource": { "type": "prometheus", "uid": "PROM_DS" },
+          "expr": "sum(rate(visor_check_issues_total[5m])) by (severity)"
+        }
+      ],
+      "gridPos": { "h": 8, "w": 12, "x": 12, "y": 0 }
+    }
+  ],
+  "version": 1,
+  "editable": true
+}
+