@probelabs/visor 0.1.106 → 0.1.111

package/dist/docs/event-triggers.md

@@ -0,0 +1,292 @@
+# Visor Event Triggers
+
+This document explains what events trigger Visor and how to identify the context from logs.
+
+## Trigger Modes
+
+Visor can be triggered in two main modes:
+
+### 1. Manual CLI Mode
+- **When**: Running `visor` command locally
+- **Log indicator**: `🖥️  Mode: Manual CLI`
+- **Context**: Local git repository analysis
+- **Event simulation**: Use `--event` flag to simulate GitHub events (see [CLI Event Simulation](#cli-event-simulation))
+
+### 2. GitHub Action Mode
+- **When**: Triggered by GitHub webhook events
+- **Log indicator**: `🤖 Mode: GitHub Action`
+- **Context**: Based on GitHub event type
+
+## GitHub Events
+
+When running as a GitHub Action, Visor responds to these events:
+
+### Pull Request Events (`pull_request`)
+- **Actions**: `opened`, `synchronize`, `edited`
+- **Trigger mapping**:
+  - `opened` → `pr_opened`
+  - `synchronize` / `edited` → `pr_updated`
+- **Log format**:
+  ```
+  🤖 Mode: GitHub Action
+  📂 Repository: owner/repo
+  📋 Event: pull_request (action: opened)
+  🎯 Trigger: pr_opened
+  🔀 Context: Pull Request #123
+  ```
+
+### Issue Events (`issues`)
+- **Actions**: `opened`, `edited`, etc.
+- **Trigger mapping**: `issue_opened`
+- **Log format**:
+  ```
+  🤖 Mode: GitHub Action
+  📂 Repository: owner/repo
+  📋 Event: issues (action: opened)
+  🎯 Trigger: issue_opened
+  🎫 Context: Issue #456
+  ```
+
+### Issue Comment Events (`issue_comment`)
+- **Actions**: `created`, `edited`
+- **Trigger mapping**: `issue_comment`
+- **Context**: Can be on either a Pull Request or an Issue
+- **Log format** (on PR):
+  ```
+  🤖 Mode: GitHub Action
+  📂 Repository: owner/repo
+  📋 Event: issue_comment (action: created)
+  🎯 Trigger: issue_comment
+  💬 Context: Comment on Pull Request #123
+  ```
+- **Log format** (on Issue):
+  ```
+  🤖 Mode: GitHub Action
+  📂 Repository: owner/repo
+  📋 Event: issue_comment (action: created)
+  🎯 Trigger: issue_comment
+  💬 Context: Comment on Issue #456
+  ```
+
+## Example Logs
+
+### Before (old logging):
+```
+Event: issue_comment, Owner: TykTechnologies, Repo: tyk
+```
+
+### After (improved logging):
+```
+🤖 Mode: GitHub Action
+📂 Repository: TykTechnologies/tyk
+📋 Event: issue_comment (action: created)
+🎯 Trigger: issue_comment
+💬 Context: Comment on Pull Request #123
+📚 Total checks in loaded config: 10
+📚 Available checks: release-notes, overview, security, performance, quality, style, review-all, issue-assistant, dependency, connectivity
+🔧 Checks to run for issue_comment: review-all, issue-assistant
+```
+
+## Understanding the Log Output
+
+1. **Mode**: Tells you if this is a manual CLI run or GitHub Action
+2. **Repository**: The owner/repo being analyzed
+3. **Event**: The raw GitHub event name and action
+4. **Trigger**: The internal event type Visor uses for configuration
+5. **Context**: Specific context (PR number, Issue number, etc.)
+6. **Available checks**: All checks defined in configuration
+7. **Checks to run**: Which checks will execute for this event
+
+## Recursion Prevention
+
+When Visor posts a comment on an issue or PR, GitHub triggers a new `issue_comment` event. To prevent infinite loops, Visor automatically skips processing its own comments by detecting:
+
+1. **Bot username**: `visor[bot]`, `github-actions[bot]`, `probelabs[bot]`, or any user with `type: Bot`
+2. **Visor markers**: Comments containing `<!-- visor-comment-id:` or `*Powered by [Visor](...)*`
+
+**Example log (expected behavior):**
+```
+✓ Skipping bot's own comment to prevent recursion. Author: probelabs[bot], Type: Bot, Has markers: true
+```
+
+This is **normal and correct** - it means:
+- Issue opened → Bot posts comment → Comment triggers event → Bot detects its own comment → Skips ✅
+
+## Event Flow Example
+
+**When an issue is opened:**
+
+1. **Event 1**: `issues` (action: opened)
+   ```
+   🤖 Mode: GitHub Action
+   📋 Event: issues (action: opened)
+   🎯 Trigger: issue_opened
+   🎫 Context: Issue #456
+   🔧 Checks to run: issue-assistant
+   ✅ Posted issue assistant results to issue #456
+   ```
+
+2. **Event 2**: `issue_comment` (action: created) - Bot's own comment
+   ```
+   🤖 Mode: GitHub Action
+   📋 Event: issue_comment (action: created)
+   🎯 Trigger: issue_comment
+   💬 Context: Comment on Issue #456
+   ✓ Skipping bot's own comment to prevent recursion. Author: probelabs[bot], Type: Bot, Has markers: true
+   ```
+
+## Configuration
+
+Checks are configured to run on specific triggers using the `on` field:
+
+```yaml
+steps:
+  security-check:
+    on: [pr_opened, pr_updated]  # Only runs on PR events
+    type: ai
+    # ... rest of config
+
+  issue-assistant:
+    on: [issue_comment, issue_opened]  # Runs on issue events
+    type: ai
+    # ... rest of config
+```
+
+If `on` is not specified, the check can run on any event type.
+
+### Using goto_event to simulate a different trigger for a jump
+
+Sometimes you want to “re-run” an ancestor step as if a different event happened (e.g., from an `issue_comment` flow you want to re-execute a PR step under `pr_updated`). Use `goto_event` together with `goto`:
+
+```yaml
+steps:
+  overview:
+    type: ai
+    on: [pr_opened, pr_updated]
+
+  security:
+    type: ai
+    depends_on: [overview]
+    on: [pr_opened, pr_updated]
+    on_success:
+      goto: overview
+      goto_event: pr_updated  # simulate PR update for the inline jump
+```
+
+During that inline execution, event filtering and `if:` expressions see the simulated event. The current step is then re-run once. For a full PR re-run from an issue comment, synthesize a PR event and re-invoke the action entrypoint (see failure-routing docs for details).
+
+## CLI Event Simulation
+
+When running Visor locally via CLI, you can simulate GitHub events to test event-based check filtering. This is useful for testing your configuration locally before deploying to GitHub Actions.
+
+### Usage
+
+```bash
+# Simulate a PR update event
+visor --event pr_updated
+
+# Simulate a PR opened event
+visor --event pr_opened
+
+# Simulate an issue comment event
+visor --event issue_comment
+
+# Simulate an issue opened event
+visor --event issue_opened
+
+# Run all checks regardless of event filters (default)
+visor --event all
+```
+
+### Auto-Detection
+
+Visor automatically detects the appropriate event based on the checks being run:
+
+- **Code-review schemas** → Defaults to `pr_updated`
+- **Other schemas** → Defaults to `all` (no filtering)
+
+```bash
+# These two are equivalent when running checks with code-review schema
+visor --check security
+visor --check security --event pr_updated
+```
+
+### Event Filtering Behavior
+
+| Flag | Behavior | Use Case |
+|------|----------|----------|
+| `--event pr_updated` | Only runs checks with `on: [pr_updated]` | Test PR update scenario locally |
+| `--event pr_opened` | Only runs checks with `on: [pr_opened]` | Test PR opened scenario |
+| `--event issue_comment` | Only runs checks with `on: [issue_comment]` | Test comment assistant |
+| `--event manual` | Only runs checks with `on: [manual]` | Test manual-triggered checks |
+| `--event all` | Runs all checks (except manual-only) | Default behavior, ignores `on:` filters |
+| (no flag) | Auto-detects based on schema | Smart default |
+
+### Example Workflow
+
+**Testing PR review locally before pushing:**
+
+```bash
+# 1. Make changes on a feature branch
+git checkout -b feature/new-feature
+
+# 2. Simulate PR update event to see what checks would run
+visor --event pr_updated --analyze-branch-diff
+
+# 3. Review the output and fix any issues
+
+# 4. Push and open PR (GitHub Actions will run the same checks)
+git push origin feature/new-feature
+```
+
+### Combining with Other Flags
+
+Event simulation works alongside other CLI flags:
+
+```bash
+# Analyze branch diff with PR update event
+visor --event pr_updated --analyze-branch-diff
+
+# Run specific checks with event filtering
+visor --check security --check style --event pr_opened
+
+# Debug mode with event simulation
+visor --event pr_updated --debug
+
+# JSON output with event filtering
+visor --event issue_comment --output json
+```
+
+### Log Output
+
+When using event simulation, you'll see:
+
+```
+🎯 Simulating event: pr_updated
+▶ Running check: overview [1/4]
+▶ Running check: security [2/4]
+```
+
+When using `--event all`:
+
+```
+🎯 Event filtering: DISABLED (running all checks regardless of event triggers)
+```
+
+### Common Patterns
+
+**Test all PR checks locally:**
+```bash
+visor --event pr_updated --analyze-branch-diff
+```
+
+**Test issue assistant:**
+```bash
+# Issue assistants typically run on issue_comment or issue_opened
+visor --event issue_comment
+```
+
+**Run everything (useful for local development):**
+```bash
+visor --event all  # or just: visor
+```

package/dist/docs/execution-statistics-rfc.md

@@ -0,0 +1,290 @@
+# RFC: Enhanced Execution Logging & Statistics
+
+**Status:** Implemented
+**Date:** 2025-10-04
+**Version:** 1.0
+
+## Summary
+
+This RFC describes the implementation of comprehensive execution statistics tracking and enhanced logging for Visor checks, providing users with clear visibility into forEach iterations, skip reasons, and overall execution metrics.
+
+## Motivation
+
+Users running complex Visor workflows with features like `forEach`, `if` conditions, and `fail_if` lacked visibility into:
+- How many times each check actually ran (forEach iterations)
+- Why checks were skipped and what conditions failed
+- Overall execution metrics (total runs vs. configured checks)
+- Performance bottlenecks (time per check/iteration)
+- Issue distribution across checks
+
+This made debugging and understanding workflow execution difficult.
+
+## Design
+
+### Architecture
+
+The solution adds execution statistics tracking throughout the check execution lifecycle:
+
+```
+CheckExecutionEngine
+├── executionStats: Map<string, CheckExecutionStats>  // Tracks all stats
+├── initializeCheckStats()                             // Initialize stats for each check
+├── recordIterationStart/Complete()                    // Track forEach iterations
+├── recordSkip()                                       // Track skipped checks
+├── recordError()                                      // Track failures
+├── buildExecutionStatistics()                         // Build final stats object
+└── logExecutionSummary()                              // Display symbol-rich table
+```
+
+### Data Structures
+
+#### CheckExecutionStats
+```typescript
+interface CheckExecutionStats {
+  checkName: string;
+  totalRuns: number;          // How many times executed (1 or forEach iterations)
+  successfulRuns: number;
+  failedRuns: number;
+  skipped: boolean;
+  skipReason?: 'if_condition' | 'fail_fast' | 'dependency_failed';
+  skipCondition?: string;     // The actual if condition text
+  totalDuration: number;      // Total duration in milliseconds
+  perIterationDuration?: number[];  // Duration for each iteration
+  issuesFound: number;
+  issuesBySeverity: {
+    critical: number;
+    error: number;
+    warning: number;
+    info: number;
+  };
+  outputsProduced?: number;   // For forEach checks
+  errorMessage?: string;      // If failed
+  forEachPreview?: string[];  // Preview of forEach items
+}
+```
+
+#### ExecutionStatistics
+```typescript
+interface ExecutionStatistics {
+  totalChecksConfigured: number;
+  totalExecutions: number;     // Sum of all runs including forEach
+  successfulExecutions: number;
+  failedExecutions: number;
+  skippedChecks: number;
+  totalDuration: number;
+  checks: CheckExecutionStats[];
+}
+```
+
+### Tracking Points
+
+1. **Initialization** (`executeDependencyAwareChecks` start)
+   - Initialize stats for all checks
+
+2. **forEach Execution** (when forEach items detected)
+   - Record forEach preview items
+   - Track each iteration start/complete
+   - Log iteration progress: `✔ 3/5 (2.1s)`
+
+3. **Normal Execution** (single runs)
+   - Track start/complete
+   - Record outputs produced
+
+4. **Skip Conditions** (when `if` evaluates to false)
+   - Record skip reason and condition
+   - Log: `⏭ Skipped (if: branch == "main")`
+
+5. **Errors** (on exception)
+   - Record error message
+   - Track failed iteration
+
+6. **Completion** (`executeChecks` end)
+   - Build final statistics
+   - Display summary table
+
+### Output Formats
+
+#### Console (Table) - Default
+```
+┌───────────────────────────────────────┐
+│ Execution Complete (45.3s)            │
+├───────────────────────────────────────┤
+│ Checks: 8 configured → 23 executions  │
+│ Status: 20 ✔ │ 2 ✖ │ 1 ⏭             │
+│ Issues: 15 total (3 🔴 12 ⚠️)         │
+└───────────────────────────────────────┘
+
+Check Details:
+┌─────────────────────┬──────────┬──────────┬─────────────────────┐
+│ Check               │ Duration │ Status   │ Details             │
+├─────────────────────┼──────────┼──────────┼─────────────────────┤
+│ list-files          │ 0.5s     │ ✔        │ →5                  │
+│ validate-file       │ 12.3s    │ ✔ ×5     │ 12⚠️                │
+│ security-scan       │ 8.2s     │ ✔        │ 3🔴                 │
+│ transform-data      │ 6.1s     │ ✔ ×3     │ →3                  │
+│ notify-slack        │ 2.1s     │ ✖ ×3     │ HTTP 500            │
+│ production-only     │ -        │ ⏭ if     │ branch=="main"      │
+│ archive-results     │ 1.3s     │ ✔        │                     │
+│ final-report        │ 14.8s    │ ✔        │ →1                  │
+└─────────────────────┴──────────┴──────────┴─────────────────────┘
+
+Legend: ✔=success │ ✖=failed │ ⏭=skipped │ ×N=iterations │ →N=outputs │ N🔴=critical │ N⚠️=warnings
+```
+
+#### JSON Output
+Full `executionStatistics` object included in JSON output with all tracked data:
+- Per-iteration timings
+- Skip condition evaluations
+- forEach item previews
+- Complete error messages
+
+### Symbol Reference
+
+| Symbol | Meaning | Example |
+|--------|---------|---------|
+| `✔` | Success (1 run) | `✔` |
+| `✔ ×5` | Success (5 forEach iterations) | `✔ ×5` |
+| `✖ ×3` | Failed (3 iterations) | `✖ ×3` |
+| `✔/✖ 2/3` | Partial success (2 of 3 succeeded) | `✔/✖ 2/3` |
+| `⏭ if` | Skipped (if condition) | `⏭ if` |
+| `⏭ ff` | Skipped (fail-fast) | `⏭ ff` |
+| `⏭ dep` | Skipped (dependency failed) | `⏭ dep` |
+| `→N` | Produced N outputs | `→5` |
+| `N🔴` | N critical issues | `3🔴` |
+| `N⚠️` | N warnings | `12⚠️` |
+| `N💡` | N info messages | `5💡` |
+
+## Implementation
+
+### Files Modified
+
+1. **src/check-execution-engine.ts** (~300 lines added)
+   - Added `CheckExecutionStats` and `ExecutionStatistics` interfaces
+   - Added `executionStats` property to class
+   - Implemented tracking methods:
+     - `initializeCheckStats()`
+     - `recordIterationStart/Complete()`
+     - `recordSkip()`
+     - `recordError()`
+     - `recordForEachPreview()`
+     - `buildExecutionStatistics()`
+   - Implemented display methods:
+     - `logExecutionSummary()`
+     - `formatStatusColumn()`
+     - `formatDetailsColumn()`
+     - `truncate()`
+   - Updated `executeDependencyAwareChecks()` to track stats
+   - Updated `executeChecks()` to return statistics
+
+2. **src/output-formatters.ts** (~5 lines added)
+   - Added `executionStatistics?` field to `AnalysisResult` interface
+
+### Backward Compatibility
+
+✅ **No Breaking Changes**
+- All changes are additive
+- `executionStatistics` is optional in `AnalysisResult`
+- Existing output formats continue to work
+- Statistics appear automatically in table output (default)
+- JSON/SARIF include statistics when available
+
+## Examples
+
+### Scenario 1: forEach with Multiple Iterations
+
+**Config:**
+```yaml
+steps:
+  list-files:
+    type: command
+    exec: echo '["a.json", "b.json", "c.json"]'
+    forEach: true
+
+  validate-file:
+    type: command
+    exec: jsonlint {{ outputs.list-files }}
+    depends_on: [list-files]
+```
+
+**Output:**
+```
+▶ Running check: list-files [1/2]
+✔ Check complete: list-files (0.3s) - 3 items
+
+▶ Running check: validate-file [2/2]
+  Processing 3 items...
+  ✔ 1/3 (1.2s)
+  ✔ 2/3 (1.1s)
+  ✔ 3/3 (1.3s)
+✔ Check complete: validate-file (3.6s) - 3 runs
+
+┌───────────────────────────────────────┐
+│ Execution Complete (3.9s)             │
+├───────────────────────────────────────┤
+│ Checks: 2 configured → 4 executions   │
+│ Status: 4 ✔ │ 0 ✖ │ 0 ⏭              │
+└───────────────────────────────────────┘
+
+Check Details:
+┌─────────────────────┬──────────┬──────────┬─────────────────────┐
+│ Check               │ Duration │ Status   │ Details             │
+├─────────────────────┼──────────┼──────────┼─────────────────────┤
+│ list-files          │ 0.3s     │ ✔        │ →3                  │
+│ validate-file       │ 3.6s     │ ✔ ×3     │                     │
+└─────────────────────┴──────────┴──────────┴─────────────────────┘
+```
+
+### Scenario 2: Conditional Skip
+
+**Config:**
+```yaml
+steps:
+  production-deploy:
+    type: command
+    exec: ./deploy.sh
+    if: 'branch == "main"'
+```
+
+**Output (on feature branch):**
+```
+▶ Running check: production-deploy [1/1]
+⏭  Skipped (if: branch == "main")
+
+┌───────────────────────────────────────┐
+│ Execution Complete (0.0s)             │
+├───────────────────────────────────────┤
+│ Checks: 1 configured → 0 executions   │
+│ Status: 0 ✔ │ 0 ✖ │ 1 ⏭              │
+└───────────────────────────────────────┘
+
+Check Details:
+┌─────────────────────┬──────────┬──────────┬─────────────────────┐
+│ Check               │ Duration │ Status   │ Details             │
+├─────────────────────┼──────────┼──────────┼─────────────────────┤
+│ production-deploy   │ -        │ ⏭ if     │ branch == "main"    │
+└─────────────────────┴──────────┴──────────┴─────────────────────┘
+```
+
+## Benefits
+
+1. **🎯 Clarity**: Users immediately see forEach iterations and skip reasons
+2. **📊 Metrics**: Clear stats on total work done vs. configured checks
+3. **⚡ Performance**: Easy to spot slow checks and iterations
+4. **🐛 Debugging**: Skip conditions shown inline for quick troubleshooting
+5. **📈 Automation**: Full stats in JSON for CI/CD analytics
+6. **✅ Professional**: Clean, symbol-rich output matches modern CLIs
+
+## Future Enhancements
+
+Potential future improvements:
+- Color-coded output (green/red/yellow) when terminal supports it
+- Configurable verbosity levels for forEach iteration logging
+- Export statistics to external monitoring systems
+- Historical statistics comparison across runs
+- Performance regression detection
+
+## References
+
+- Implementation PR: [Link to PR]
+- Related Issues: Enhanced logging visibility (#XX)
+- Documentation: Updated in CLAUDE.md and README.md