@probelabs/visor 0.1.106 → 0.1.111

package/dist/docs/dependencies.md

@@ -0,0 +1,369 @@
+## 📊 Step Dependencies & Intelligent Execution
+
+Visor supports defining dependencies between checks using `depends_on`. This enables:
+
+- Sequential execution: dependents wait for prerequisites to finish
+- Parallel optimization: independent checks run simultaneously
+- Smart scheduling: automatic topological ordering
+
+### Configuration Example
+
+```yaml
+version: "1.0"
+steps:
+  security:
+    type: ai
+    group: code-review
+    schema: code-review
+    prompt: "Comprehensive security analysis..."
+    tags:
+      - security
+      - critical
+      - comprehensive
+    on:
+      - pr_opened
+      - pr_updated
+    # No dependencies - runs first
+
+  performance:
+    type: ai
+    group: code-review
+    schema: code-review
+    prompt: "Performance analysis..."
+    tags:
+      - performance
+      - fast
+      - local
+      - remote
+    on:
+      - pr_opened
+      - pr_updated
+    # No dependencies - runs parallel with security
+
+  style:
+    type: ai
+    group: code-review
+    schema: code-review
+    prompt: "Style analysis based on security findings..."
+    tags:
+      - style
+      - fast
+      - local
+    on:
+      - pr_opened
+    depends_on:
+      - security  # Waits for security to complete
+
+  architecture:
+    type: ai
+    group: code-review
+    schema: code-review
+    prompt: "Architecture analysis building on previous checks..."
+    on:
+      - pr_opened
+      - pr_updated
+    depends_on:
+      - security
+      - performance
+```
+
+### Execution Flow
+
+1. Level 0: `security` and `performance` run in parallel
+2. Level 1: `style` runs after `security`
+3. Level 2: `architecture` runs after both
+
+### Advanced Patterns
+
+#### Diamond Dependency
+```yaml
+steps:
+  foundation: { type: ai, group: base, schema: code-review, prompt: "Base analysis" }
+  branch_a:   { type: ai, group: code-review, schema: code-review, depends_on: [foundation] }
+  branch_b:   { type: ai, group: code-review, schema: code-review, depends_on: [foundation] }
+  final:      { type: ai, group: summary, schema: markdown, depends_on: [branch_a, branch_b] }
+```
+
+#### Multiple Independent Chains
+```yaml
+steps:
+  security_basic:     { type: ai, group: security,    schema: code-review }
+  security_advanced:  { type: ai, group: security,    schema: code-review, depends_on: [security_basic] }
+  performance_basic:  { type: ai, group: performance, schema: code-review }
+  performance_advanced:{ type: ai, group: performance, schema: code-review, depends_on: [performance_basic] }
+  integration:        { type: ai, group: summary,     schema: markdown, depends_on: [security_advanced, performance_advanced] }
+```
+
+#### Any‑of (OR) Dependency Groups
+
+Sometimes a check can proceed when any one of several upstream steps has completed successfully. Visor supports this with pipe‑separated tokens inside `depends_on`.
+
+## Criticality and Gating
+
+`continue_on_failure` controls whether dependents may run after a failure — it is a gating knob, not the definition of criticality. Classify steps by criticality (external | internal | policy | info) and derive defaults:
+
+- Critical: `continue_on_failure: false`, require `assume`/`guarantee`, tighter loop budgets, retries only for transient faults.
+- Non‑critical: may allow `continue_on_failure: true` to keep non‑critical branches moving.
+
+Example — non‑critical branch that can proceed after a soft failure:
+```yaml
+steps:
+  summarize:
+    type: ai
+    tags:
+      - info
+    continue_on_failure: true
+    fail_if: "(output.errors || []).length > 0"
+```
+
+```yaml
+checks:
+  parse-issue:   { type: noop }
+  parse-comment: { type: noop }
+  triage:        { type: noop, depends_on: ["parse-issue|parse-comment"] }
+```
+
+Rules:
+- Each string containing `|` denotes an ANY‑OF group. In the example above, either `parse-issue` or `parse-comment` satisfies the dependency for `triage`.
+- You may combine ALL‑OF and ANY‑OF: `depends_on: ["a|b", "c"]` means “(a or b) and c”.
+- Event gating still applies: a dependency only counts if it is applicable to the current event (has compatible `on` or no `on`).
+- Failure/skip semantics: a member that is skipped or fails fatally does not satisfy the group; at least one member must complete without a fatal error for the group to be satisfied.
+- Session reuse: if `reuse_ai_session: true` and `depends_on` contains a pipe group, the session parent is selected from the first satisfied member at runtime.
+
+Tip: When targeting a leaf in ad‑hoc runs (e.g., `visor --check final`), include one member of each pipe group explicitly (e.g., `--check a --check final`) to make intent unambiguous. In normal runs Visor computes the plan automatically from your config.
+
+### Error Handling
+
+- Cycle detection and missing dependency validation
+- Failed checks don't block independent branches
+- Dependency results are available to dependents via `outputs`
+
+## forEach Dependency Propagation with on_finish
+
+When a check has `forEach: true`, it outputs an array and all its dependent checks run once per array item. After **all** dependents complete **all** iterations, the `on_finish` hook on the forEach check triggers to aggregate results and optionally route to a different check.
+
+### Basic Flow
+
+```yaml
+checks:
+  extract-items:
+    type: ai
+    forEach: true
+    # Outputs: [item1, item2, item3]
+
+  process-item:
+    depends_on: [extract-items]
+    # Runs 3 times (once per item)
+```
+
+**Execution order:**
+1. `extract-items` runs once → outputs `[item1, item2, item3]`
+2. `process-item` runs 3 times (once for each item)
+3. All 3 iterations complete
+4. Downstream checks that depend on `process-item` can now run
+
+### on_finish Hook for Aggregation
+
+The `on_finish` hook runs **once** after all dependent checks complete all their iterations, making it perfect for aggregating results and making routing decisions:
+
+```yaml
+checks:
+  extract-facts:
+    type: ai
+    forEach: true
+    # Outputs: [fact1, fact2, fact3]
+
+    on_finish:
+      # Run aggregation check
+      run: [aggregate-validations]
+
+      # Then decide whether to retry
+      goto_js: |
+        const allValid = memory.get('all_valid', 'validation');
+        return allValid ? null : 'retry-assistant';
+
+  validate-fact:
+    depends_on: [extract-facts]
+    # Runs 3 times (once per fact)
+
+  aggregate-validations:
+    type: script
+    content: |
+      // Access ALL validation results
+      const results = outputs.history['validate-fact'];
+      const allValid = results.every(r => r.is_valid);
+      memory.set('all_valid', allValid, 'validation');
+      return { total: results.length, valid: allValid };
+```
+
+**Execution order:**
+1. `extract-facts` runs once → outputs array of facts
+2. `validate-fact` runs N times (once per fact)
+3. **on_finish triggers:**
+   - First: `aggregate-validations` runs
+   - Then: `goto_js` evaluates
+   - If goto returns a check name, jump to that ancestor
+4. Downstream checks continue
+
+### When on_finish Triggers
+
+- **Only** on checks with `forEach: true`
+- **After** ALL dependent checks complete ALL iterations
+- **Does not** trigger if forEach array is empty
+- **Before** any downstream checks that don't depend on the forEach check
+
+### Accessing forEach Results
+
+Inside `on_finish` hooks, you have access to all iteration results via `outputs.history`:
+
+```javascript
+// In on_finish.goto_js or on_finish.run_js
+{
+  outputs: {
+    'extract-facts': [...],  // The forEach array
+    'validate-fact': [...],  // Latest results (for compatibility)
+  },
+  outputs.history: {
+    'validate-fact': [[...], ...], // ALL results from ALL iterations
+  },
+  forEach: {
+    total: 3,       // Total forEach items
+    successful: 3,  // Number of successful iterations
+    failed: 0,      // Number of failed iterations
+    items: [...]    // The forEach items array
+  }
+}
+```
+
+### Complete Example: Multi-Dependent Aggregation
+
+The real power of `on_finish` is aggregating results from **multiple** dependent checks:
+
+```yaml
+checks:
+  # Step 1: Extract claims from AI response
+  extract-claims:
+    type: ai
+    forEach: true
+    prompt: "Extract all factual claims from: {{ outputs.ai-response }}"
+    transform_js: JSON.parse(output).claims
+    depends_on: [ai-response]
+
+    # Step 4: After ALL validations complete
+    on_finish:
+      run: [aggregate-all-validations]
+      goto_js: |
+        const securityOk = memory.get('security_valid', 'validation');
+        const technicalOk = memory.get('technical_valid', 'validation');
+        const attempt = memory.get('attempt', 'validation') || 0;
+
+        if (securityOk && technicalOk) {
+          return null;  // All good, proceed
+        }
+
+        if (attempt >= 2) {
+          return null;  // Max attempts, give up
+        }
+
+        memory.increment('attempt', 1, 'validation');
+        return 'ai-response';  // Retry with validation context
+
+  # Step 2: Validate security aspects (runs N times)
+  validate-security:
+    type: ai
+    depends_on: [extract-claims]
+    prompt: |
+      Validate security implications of: {{ outputs['extract-claims'].claim }}
+
+  # Step 3: Validate technical accuracy (runs N times)
+  validate-technical:
+    type: ai
+    depends_on: [extract-claims]
+    prompt: |
+      Validate technical accuracy of: {{ outputs['extract-claims'].claim }}
+
+  # Step 4a: Aggregate ALL results
+  aggregate-all-validations:
+    type: script
+    content: |
+      // Get results from BOTH dependent checks
+      const securityResults = outputs.history['validate-security'];
+      const technicalResults = outputs.history['validate-technical'];
+
+      const securityValid = securityResults.every(r => r.is_valid);
+      const technicalValid = technicalResults.every(r => r.is_valid);
+
+      memory.set('security_valid', securityValid, 'validation');
+      memory.set('technical_valid', technicalValid, 'validation');
+
+      // Store issues for retry context
+      if (!securityValid || !technicalValid) {
+        const issues = [
+          ...securityResults.filter(r => !r.is_valid),
+          ...technicalResults.filter(r => !r.is_valid)
+        ];
+        memory.set('validation_issues', issues, 'validation');
+      }
+
+      return {
+        security: { total: securityResults.length, valid: securityValid },
+        technical: { total: technicalResults.length, valid: technicalValid }
+      };
+
+  # Step 5: Post if validation passed
+  post-response:
+    type: github
+    depends_on: [extract-claims]
+    if: "memory.get('security_valid', 'validation') && memory.get('technical_valid', 'validation')"
+    op: comment.create
+    value: "{{ outputs['ai-response'] }}"
+```
+
+This is the **only way** to aggregate across multiple dependent checks in a forEach scenario. Without `on_finish`, there would be no single point where all results are available together.
+
+### Best Practices
+
+1. **Use outputs.history**: Access all forEach iteration results with `outputs.history['check-name']`
+2. **Store in Memory**: Use memory to pass aggregated state to `goto_js` and downstream checks
+3. **Handle Empty Arrays**: Check `forEach.total` or array length before processing
+4. **Limit Loops**: Use attempt counters in memory to prevent infinite retry loops
+5. **Multiple Dependents**: `on_finish` is perfect when you have multiple checks depending on the same forEach check
+6. **Event Preservation**: Use `goto_event` when jumping back to maintain correct event context
+
+### Comparison: on_finish vs Regular Dependent
+
+| Approach | When It Runs | Access to Results | Use Case |
+|----------|-------------|-------------------|----------|
+| Regular dependent check | After forEach parent completes | Only parent's array items | Process individual items |
+| `on_finish` hook | After **all** dependents complete **all** iterations | All iteration results via `outputs.history` | Aggregate, validate, route |
+
+**Example showing the difference:**
+
+```yaml
+checks:
+  extract-items:
+    type: command
+    forEach: true
+    exec: echo '[1, 2, 3]'
+    on_finish:
+      run: [summarize-all]
+
+  process-item:
+    depends_on: [extract-items]
+    # Runs 3 times, once per item
+    # Has access to: outputs['extract-items'] (current item)
+
+  summarize-all:
+    type: script
+    # Runs ONCE after all 3 process-item iterations
+    # Has access to: outputs.history['process-item'] (all 3 results)
+    content: |
+      const allResults = outputs.history['process-item'];
+      return { processed: allResults.length };
+```
+
+### See Also
+
+- [Failure Routing](./failure-routing.md) - Complete `on_finish` reference
+- [forEach Dependency Propagation](./foreach-dependency-propagation.md) - Detailed forEach mechanics
+- [Output History](./output-history.md) - Accessing historical outputs

package/dist/docs/dev-playbook.md

@@ -0,0 +1,9 @@
+## 🧭 Developer Experience Playbook
+
+- Start with defaults: copy `defaults/.visor.yaml` or an example; run `npx -y @probelabs/visor@latest --check all --debug`.
+- Treat config as code: review `.visor.yaml` and templates; pin providers/models for reproducibility.
+- Roll out gradually: gate heavier checks with tags (`local`, `fast`, `critical`).
+- Secure credentials: prefer GitHub App in production; scope/rotate API keys.
+- Make feedback actionable: group related checks; use `/review --check ...` triggers; enable `reuse_ai_session` for follow-ups.
+- Keep suppressions intentional: annotate context; audit `visor-disable-file` periodically.
+- Validate locally: `npx -y @probelabs/visor@latest --check security --output markdown`; run tests; `--fail-fast` for fast lanes.

package/dist/docs/engine-pause-resume-rfc.md

@@ -0,0 +1,192 @@
+# RFC: Proper Pause/Resume for State-Machine Engine (Event‑Bus + Snapshots)
+
+Status: draft
+
+Owner: visor engine
+
+Last updated: 2025-11-20
+
+## Summary
+
+We will add first‑class pause/resume to the state‑machine engine so long‑running or interactive workflows (e.g., human‑input in Slack) can suspend execution and later continue exactly where they left off, without re‑running completed work. The design builds on our event‑bus architecture and the existing ExecutionJournal. It introduces JSON snapshots of the engine’s RunState and journal, and a resume entrypoint that hydrates a new runner from a snapshot. Slack and other frontends trigger resume when the awaited user event arrives.
+
+## Goals
+
+- Pause a workflow at deterministic points (e.g., when HumanInputRequested is emitted) and resume later.
+- Persist minimal, safe state; do not leak secrets.
+- Avoid re‑executing completed checks; preserve outputs/history/routing.
+- Keep the event‑driven integration pattern (no long‑lived in‑memory runs).
+- Be robust to process restarts (snapshots on disk); work in CI and serverless.
+
+## Non‑Goals (initial)
+
+- Arbitrary mid‑provider checkpointing (we pause at well‑defined integration points).
+- Time‑travel debugging; only last consistent snapshot is kept by default.
+
+## Current State (as of this RFC)
+
+- We already end the run on human‑input and rely on PromptState + a new event to re‑enter the workflow. This re‑entry currently performs a cold start and plans from scratch, which is acceptable but can be inefficient and can re‑visit guards.
+- Engine has ExecutionJournal and experimental `saveSnapshotToFile()` that serializes RunState and journal; there is no hydrate/resume path.
+
+## High‑Level Design
+
+1) Snapshots (JSON)
+- When the engine encounters a pause point (e.g., HumanInputRequested), it writes a snapshot JSON file containing:
+  - `version`: number
+  - `sessionId`, `event` (trigger), `wave`
+  - `state`: serialized RunState (via a new `serializeRunState()` + `deserializeRunState()` pair)
+  - `journal`: visible `JournalEntry[]` up to the snapshot
+  - `requestedChecks`: string[]
+  - `meta`: optional { checkId, channel, threadTs, threadKey, promptTs }
+
+2) Pause Points
+- Initial scope: provider‑level pause during HumanInputRequested (event‑bus). Other future pause points can hook the same API.
+
+3) Resume Entry Point
+- New `engine.resumeFromSnapshot(snapshot, overrides)` that:
+  - Rebuilds `EngineContext` (config, requestedChecks, sessionId, event, journal)
+  - Creates a new `StateMachineRunner`, calls `runner.setState(deserializedRunState)` (new API), and continues the main loop.
+  - Applies overrides such as `webhookContext` to allow providers to consume the awaited input.
+
+4) Frontends & PromptState
+- Frontends (Slack) maintain the human prompt UI and store a pointer to the snapshot path in PromptState. On the awaited reply, frontends look up the snapshot and call `resumeFromSnapshot()` (via the socket/webhook path).
+
+5) Storage & Retention
+- Default snapshot directory: `${VISOR_SNAPSHOT_DIR || '.visor/snapshots'}`.
+- File naming: `${threadKey}-${checkId}.json` where `threadKey = "${channel}:${threadTs}"`.
+- Retention: delete on successful resume; background TTL cleanup (e.g., 24h) to reap orphans.
+
+## Detailed Design
+
+### A. RunState serialization/hydration
+
+- Today we have `serializeRunState(state)` for JSON. We will:
+  - Add `deserializeRunState(obj): RunState` (recreates Sets/Maps and ensures invariants).
+  - Add `StateMachineRunner.setState(state: RunState)`; only valid before `run()`; asserts state consistency.
+  - Ensure we never serialize provider internals or secrets; RunState contains only orchestration fields.
+
+### B. Engine APIs
+
+- `StateMachineExecutionEngine.saveSnapshotToFile(filePath)` already exists.
+- Add `resumeFromSnapshot(snapshot: SnapshotJson, opts?: { webhookContext?: ..., debug?: boolean }): Promise<ExecutionResult>`
+  - Recreate `EngineContext` using config and `snapshot.sessionId`.
+  - Rehydrate journal into a fresh `ExecutionJournal` (push `snapshot.journal`).
+  - Hydrate runner with `deserializeRunState(snapshot.state)` then continue.
+  - Wire `eventBus` and frontends like a normal run so integrations keep working.
+
+### C. Snapshot triggers & lifecycle
+
+- Human‑input path:
+  - Provider emits `HumanInputRequested(checkId, prompt, channel, threadTs, threadKey)` and returns.
+  - Engine listens for `HumanInputRequested` during the run and immediately calls `saveSnapshotToFile()` to `${SNAP_DIR}/${threadKey}-${checkId}.json`.
+  - Slack Frontend posts the prompt, also sets PromptState for `${threadKey}` with `snapshotPath`.
+  - The run completes (no blocking).
+  - On Slack reply in the same thread, the socket path looks up `${snapshotPath}` and calls `resumeFromSnapshot()` with the current `webhookContext`.
+- After a successful resume (terminal StateTransition), engine deletes the snapshot file and clears PromptState.
+
+### D. Idempotency & Safety
+
+- Completed checks are recorded in RunState + journal; resuming does not re‑run them.
+- We treat new inbound input as a new event; routing/guards continue from the hydrated state.
+- If snapshot is missing or corrupted, we gracefully fall back to a cold run (today’s behavior).
+
+### E. Security/Privacy
+
+- Snapshots omit secrets and environment variables. Only engine orchestration and committed results are stored.
+- Snapshot directory is local by default; users can relocate via `VISOR_SNAPSHOT_DIR`.
+
+## File Layout
+
+```
+.visor/
+  snapshots/
+    C123:1700.55-ask.json   # example: threadKey+checkId
+```
+
+## Snapshot JSON (v1) — Example
+
+```json
+{
+  "version": 1,
+  "sessionId": "a1b2c3",
+  "event": "issue_comment",
+  "wave": 2,
+  "state": { "currentState": "Routing", "wave": 2, "activeDispatches": [], "completedChecks": ["lint"], "stats": [], "historyLog": [], "forwardRunGuards": [], "currentLevelChecks": [], "pendingRunScopes": [] },
+  "journal": [
+    { "commitId": 1, "sessionId": "a1b2c3", "scope": [], "checkId": "lint", "event": "issue_comment", "result": { "issues": [] } }
+  ],
+  "requestedChecks": ["ask", "refine", "run-commands"],
+  "meta": { "checkId": "ask", "channel": "C123", "threadTs": "1700.55", "threadKey": "C123:1700.55", "promptTs": "1700.66" }
+}
+```
+
+## Slack Integration Flow (pause/resume)
+
+1) Run emits `HumanInputRequested` → engine writes snapshot to `${threadKey}-${checkId}.json`.
+2) Slack Frontend posts prompt and sets PromptState with `snapshotPath`.
+3) User replies in same thread; socket receives the envelope, finds PromptState/snapshot.
+4) Engine `resumeFromSnapshot(snapshot, { webhookContext })` continues the workflow.
+5) On terminal state, snapshot is deleted and PromptState cleared.
+
+## CLI/Config
+
+- Env:
+  - `VISOR_SNAPSHOT_DIR` — optional base directory for snapshots.
+- Config (optional):
+  - `limits.max_workflow_depth` continues to apply; pause/resume doesn’t alter nesting rules.
+  - Future: `snapshots.enabled` (default true in Slack/webhook contexts), `snapshots.retentionHours`.
+
+## Failure Modes & Recovery
+
+- Missing snapshot: fall back to cold run.
+- Corrupt snapshot: log, fall back to cold run.
+- Multiple prompts in the same thread: last snapshot wins; older ones are overwritten.
+- Process restart: snapshots survive; PromptState TTL means we rely on snapshot presence to resume.
+
+## Testing Plan
+
+1) Unit
+- `deserializeRunState(serializeRunState(state))` round‑trip equals for non‑object identity fields.
+- `resumeFromSnapshot` continues and does not re‑execute completed checks (assert via journal size).
+
+2) Integration (Slack)
+- First run → emits HumanInputRequested, snapshot written, prompt posted, run completes.
+- Second run (reply) → loads snapshot, resumes, consumes reply, deletes snapshot.
+
+3) Crash/Restart Simulation
+- Save snapshot, reset in‑memory state, then `resumeFromSnapshot` from file.
+
+## Rollout
+
+- Phase 1 (behind feature switch in code): add hydrate APIs and write snapshots on HumanInputRequested; continue to cold‑run on reply but verify snapshot creation.
+- Phase 2: wire socket path to call `resumeFromSnapshot`; delete snapshot on success.
+- Phase 3: expand pause points if needed; add retention cleanup task.
+
+## Work Items
+
+- Runner
+  - [ ] Add `deserializeRunState()`
+  - [ ] Add `runner.setState()`
+- Engine
+  - [ ] Add `resumeFromSnapshot()`
+  - [ ] On HumanInputRequested → `saveSnapshotToFile()` (path via threadKey+checkId)
+- Slack
+  - [ ] PromptState stores `snapshotPath` alongside prompt metadata
+  - [ ] Socket runner loads snapshot and calls `resumeFromSnapshot()` on reply
+- Tests
+  - [ ] Unit: serialize/deserialize round‑trip
+  - [ ] Integration: pause/resume end‑to‑end (Slack fixture), snapshot deletion
+- Docs
+  - [ ] Update Slack integration docs and human‑input provider docs
+
+## Alternatives Considered
+
+- Keeping the runner alive across Slack replies → fragile in CI/serverless and ties up resources.
+- Persisting only high‑level outputs and re‑planning on resume → simpler but can re‑emit side‑effects and re‑evaluate guards unexpectedly.
+
+## Open Questions
+
+- Should we also emit a `RunPaused` event for analytics/observability?
+- Do we want a structured `output.awaiting = true` signal at pause for downstream guards?
+- Snapshot encryption at rest (out of scope for now; directory is local/trusted).
+