@probelabs/visor 0.1.106 → 0.1.111

package/dist/docs/testing/flows.md

@@ -0,0 +1,92 @@
+# Flow Tests
+
+> Model realistic user journeys across multiple external events in one case.
+
+A flow case defines a `flow:` array of stages. Each stage has its own `event`, `fixture`, optional `env` and `mocks`, plus `expect`.
+
+```yaml
+- name: pr-review-e2e-flow
+  strict: true
+  flow:
+    - name: pr-open
+      event: pr_opened
+      fixture: gh.pr_open.minimal
+      mocks: { overview: { text: "Overview body", tags: { label: feature, review-effort: 2 } } }
+      expect:
+        calls:
+          - step: overview
+            exactly: 1
+          - step: apply-overview-labels
+            exactly: 1
+
+    - name: visor-retrigger
+      event: issue_comment
+      fixture: gh.issue_comment.visor_regenerate
+      mocks:
+        comment-assistant: { text: "Regenerating.", intent: comment_retrigger }
+        overview: { text: "Overview (regenerated)", tags: { label: feature, review-effort: 2 } }
+      expect:
+        calls:
+          - step: comment-assistant
+            exactly: 1
+          - step: overview
+            exactly: 1
+```
+
+## Stage selection and deltas
+
+- Run a single stage: `--only case#stage` (name substring) or `--only case#N` (1‑based index).
+- Coverage, prompts, outputs, and provider calls are computed per‑stage as deltas from the previous stage.
+- The same engine instance is reused across stages, so memory and output history carry over.
+
+## Ordering and `on_finish`
+
+- Flow execution honors dependencies and `on_success`/`on_fail` routing.
+- For forEach parents with `on_finish.run`, the runner defers static targets from the initial set so they execute after per-item processing.
+- Dynamic `on_finish.run_js` is executed and counted like regular steps.
+
+## Strict mode across stages
+
+- If any step executes in a stage and lacks a corresponding `expect.calls` entry for that stage, the stage fails under strict mode.
+- Use `no_calls` to assert absence (e.g., a standard comment should not trigger a reply or fact validation).
+
+## Example: Fact Validation Loop (pattern)
+
+Note: This is not a built‑in feature, just a concrete example of how to model a multi‑step workflow with your own step names.
+
+- Per-item validation (example): a step named `validate-fact` depends on `extract-facts` (which outputs an array) and runs once per item.
+- Aggregation (example): a step named `aggregate-validations` (type: `memory`) summarizes the latest validation wave and, when not all facts are valid, schedules a correction comment via `on_finish.run_js`.
+- In tests: provide array mocks for `extract-facts` and per‑call list mocks for `validate-fact[]`. Assert that only invalid facts appear in the correction prompt using `prompts.contains`/`not_contains`.
+
+Inline example:
+
+```yaml
+flow:
+  - name: facts-invalid
+    event: issue_comment
+    fixture: gh.issue_comment.visor_help
+    env: { ENABLE_FACT_VALIDATION: "true" }
+    mocks:
+      extract-facts:
+        - { id: f1, claim: "max_parallelism defaults to 4" }
+      validate-fact[]:
+        - { fact_id: f1, is_valid: false, correction: "max_parallelism defaults to 3" }
+    expect:
+      calls:
+        - step: validate-fact
+          exactly: 1
+      prompts:
+        - step: comment-assistant
+          index: last
+          contains: ["<previous_response>", "Correction:"]
+```
+
+## Stage-local mocks and env
+
+- Stage mocks override flow-level defaults: the runner merges `{...flow.mocks, ...stage.mocks}`.
+- `env:` applies only for the stage and is restored afterward.
+
+## Debugging flows
+
+- Set `VISOR_DEBUG=true` to print stage headers, selected checks, and internal debug lines from the engine.
+- To reduce noise, limit the run to a stage: `VISOR_DEBUG=true visor test --only pr-review-e2e-flow#facts-invalid`.

package/dist/docs/testing/getting-started.md

@@ -0,0 +1,93 @@
+# Visor Tests — Getting Started
+
+This is the developer-facing guide for writing and running integration tests for your Visor config. It focuses on minimal setup, helpful errors, and clear output.
+
+## TL;DR
+
+- Put your tests in `defaults/.visor.tests.yaml`.
+- Reference your base config with `extends: ".visor.yaml"`.
+- Use built-in GitHub fixtures like `gh.pr_open.minimal`.
+- Run with `visor test --config defaults/.visor.tests.yaml`.
+- Validate only with `visor test --validate`.
+
+```yaml
+version: "1.0"
+extends: ".visor.yaml"
+
+tests:
+  defaults:
+    strict: true           # every executed step must be asserted
+    ai_provider: mock      # offline by default
+  cases:
+    - name: label-flow
+      event: pr_opened
+      fixture: gh.pr_open.minimal
+      mocks:
+        overview:
+          text: "Overview body"
+          tags: { label: feature, review-effort: 2 }
+      expect:
+        calls:
+          - step: overview
+            exactly: 1
+          - step: apply-overview-labels
+            exactly: 1
+          - provider: github
+            op: labels.add
+            at_least: 1
+            args: { contains: [feature] }
+```
+
+## Why integration tests in YAML?
+
+- You test the same thing you ship: events → checks → outputs → effects.
+- No network required; GitHub calls are recorded, AI is mocked.
+- Flows let you simulate real user journeys across multiple events.
+
+## Strict by default
+
+If a step runs and you didn’t assert it under `expect.calls`, the case fails. This prevents silent regressions and “accidental” work.
+
+Turn off per-case via `strict: false` if you need to iterate.
+
+## CLI recipes
+
+- List cases: `visor test --list`
+- Run a subset: `visor test --only pr-review`
+- Stop on first failure: `--bail`
+- Validate tests file only: `--validate`
+- Parallelize cases: `--max-parallel 4`
+- Throttle prompt capture: `--prompt-max-chars 16000`
+
+## Coverage output
+
+After each case/stage, a compact table shows expected vs actual step calls:
+
+```
+Coverage (label-flow):
+  • overview                 want =1     got  1  ok
+  • apply-overview-labels    want =1     got  1  ok
+```
+
+Unexpected executed steps are listed under `unexpected:` to help you add missing assertions quickly.
+
+## Helpful validation errors
+
+Run `visor test --validate` to get precise YAML-path errors and suggestions:
+
+```
+❌ Tests file has 2 error(s):
+   • tests.cases[0].expext: must NOT have additional properties (Did you mean "expect"?)
+   • tests.cases[3].event: must be equal to one of the allowed values (allowed: manual, pr_opened, pr_updated, pr_closed, issue_opened, issue_comment)
+```
+
+Next steps:
+- Core reference: `docs/testing/dsl-reference.md`
+- Flows: `docs/testing/flows.md`
+- Mocks & fixtures: `docs/testing/fixtures-and-mocks.md`
+- Assertions: `docs/testing/assertions.md`
+- Cookbook: `docs/testing/cookbook.md`
+- CLI & reporters: `docs/testing/cli.md`
+- CI integration: `docs/testing/ci.md`
+- Troubleshooting: `docs/testing/troubleshooting.md`
+- Browse `defaults/.visor.tests.yaml` for full examples.

package/dist/docs/testing/troubleshooting.md

@@ -0,0 +1,55 @@
+# Test Runner Troubleshooting
+
+Common issues and how to fix them quickly.
+
+## “Unmocked AI step executed” warnings
+
+- The runner forces `ai_provider=mock` by default, but if a step overrides the provider to a real one, it can execute with keys from your env.
+- Add a mock under `mocks:` for that step to silence the warning, or keep `ai_provider: mock` and avoid overriding providers in tests.
+
+## “No captured prompt for step … at index …”
+
+- The step may not have executed in that stage (strict mode would also flag it).
+- Your `index` might be out of bounds (e.g., `last` when there are no prompts). Drop the `index` or use `where` to select by content.
+- Ensure the step is an AI provider (prompts are only captured for AI steps).
+
+## GitHub op not recorded (e.g., `labels.add`)
+
+- Check that your AI mock provides the fields your provider expects to do the op (e.g., `overview.tags.label`).
+- Use `VISOR_DEBUG=true` to print provider debug. Confirm the step ran and the recorder printed the intended op.
+
+## Strict mode failure (“Step executed without expect”)
+
+- Add the missing step to `expect.calls` with a count. If the step should not run, add it to `expect.no_calls` and fix your config to avoid running it.
+- Temporarily set `strict: false` at the case or stage level to iterate quickly (not recommended long term).
+
+## Fact validation behaves unexpectedly
+
+- Provide array mocks for `extract-facts` and per‑call list mocks for `validate-fact[]`.
+- Remember that aggregation uses the latest validation wave (size inferred from the last `extract-facts` output).
+
+## Prompts are huge or contain entire diffs
+
+- Use `--prompt-max-chars` (or `tests.defaults.prompt_max_chars`) to truncate captured text for assertions.
+- In config, you can keep AI focused with `skip_code_context` if available, or provide narrower mocks in tests.
+
+## How do I see more logs?
+
+- Set `VISOR_DEBUG=true`.
+- To isolate the problem, run a single stage: `visor test --only case#stage`.
+
+## Pause at “on_finish: no result found … — skip”
+
+Symptom: a short pause appears before the runner prints the coverage table, often preceded by a line like:
+
+```
+⏭ on_finish: no result found for "extract-facts" — skip
+```
+
+What it was: the engine always entered the `on_finish` scan, even when none of the forEach parents had produced any results in the current run. Internally it waited for a (now redundant) scan window to complete.
+
+What we changed: we added an early return in `handleOnFinishHooks` when there are zero forEach parents with results in the current grouped run. This preserves behavior (no hooks to run) and removes the delay entirely.
+
+Downsides: none functionally. The only trade‑off is that debug visibility is slightly reduced in that specific “no parents ran” case; enable `VISOR_DEBUG=true` if you need to trace the discovery step regardless.
+
+Guarantee: if a check executed and defines `on_finish`, `on_finish` still executes for that check once its forEach finishes. The early return only triggers when no eligible parent produced results in the run.

package/dist/docs/timeouts.md

@@ -0,0 +1,50 @@
+# Timeouts: configuration and behavior
+
+Checks can specify a per‑check timeout. When a timeout is reached, the provider is aborted and the engine records a timeout error and skips direct dependents.
+
+## Configuration
+
+```yaml
+steps:
+  fetch-tickets:
+    type: command
+    timeout: 300  # seconds (command)
+    exec: node scripts/jira/batch-fetch.js "$JQL" --limit $LIMIT
+```
+
+Provider units and defaults:
+
+- `command`:
+  - Units: seconds.
+  - Default: 60s if not specified.
+  - Effect on timeout: issue `command/timeout` with message “Command execution timed out after <N> seconds”. Direct dependents are skipped.
+- `http_client`:
+  - Units: milliseconds.
+  - Default: provider default (see provider docs/test for specifics).
+- `ai`:
+  - Use `ai.timeout` (milliseconds). CLI `--timeout` also maps to AI timeout.
+
+The engine propagates the per‑check timeout into providers in dependency‑aware execution so behavior is consistent even in multi‑check workflows.
+
+## Examples
+
+```yaml
+steps:
+  fetch:
+    type: command
+    timeout: 120
+    exec: curl -s https://example.com/data
+
+  process:
+    type: command
+    depends_on: [fetch]
+    exec: echo "{{ outputs.fetch }}" | jq '.items | length'
+```
+
+If `fetch` exceeds 120s, `process` is skipped with reason `dependency_failed`.
+
+## Notes
+
+- Be mindful of units when switching providers (seconds vs milliseconds).
+- Consider adding `fail_if` conditions for additional guarding (e.g., `fail_if: !output || length(output.items) === 0`).
+

package/dist/docs/troubleshooting.md

@@ -0,0 +1,7 @@
+## 🛠️ Troubleshooting
+
+- Increase logging with `--debug` or action `debug: true`.
+- Validate config locally: `npx -y @probelabs/visor@latest --check all`.
+- Ensure required permissions in Actions: contents/read, pull-requests/write, issues/write, checks/write.
+- If annotations don’t appear, confirm schema is `code-review` and issues have file/line.
+- For remote extends, set `--allowed-remote-patterns` or disable via `--no-remote-extends`.

package/dist/docs/visor-sdk-rfc.md

@@ -0,0 +1,186 @@
+# Visor SDK for Node.js (ESM/CJS) — RFC
+
+Status: Draft
+Owner: Probe Labs
+Last updated: 2025-10-03
+
+## 1) Motivation
+
+Visor is currently distributed primarily as a CLI and GitHub Action. Users increasingly want to:
+- Call Visor programmatically from build scripts, test runners, or IDE extensions.
+- Orchestrate checks dynamically (e.g., select checks at runtime, react to results, implement custom routing).
+- Integrate Visor output into custom dashboards and CI pipelines without shelling out.
+
+Delivering a first‑class SDK with proper TypeScript types and dual ESM/CJS builds unlocks these scenarios while keeping the CLI intact.
+
+### Principles (very thin shim)
+- SDK is a façade over the existing Visor engine. No new execution logic.
+- No bespoke sandboxing, schedulers, or provider behavior in the SDK.
+- The SDK only wires inputs/outputs, surfaces types/events, and forwards options to the core.
+- All safety and behavior live in the core engine; SDK remains lightweight and stable.
+
+## 2) Goals (MVP)
+- Dual module support: native ESM and CommonJS with a single npm package `@probelabs/visor`.
+- First‑class TypeScript: bundled `.d.ts` with JSDoc, no `any` in public API.
+- Stable public surface for: loading config, resolving checks, executing checks, streaming events, and collecting results.
+- Programmatic access to routing (on_fail/on_success) including retry/backoff and ancestor‑only `goto`.
+- Reasonable defaults that mirror CLI behavior; no breaking changes to CLI or Action.
+- Small, tree‑shakable entry for SDK; CLI remains bundled separately.
+
+### Non‑Goals (for MVP)
+- Browser/runtime‑agnostic build. Node 18+ only for MVP.
+- Public provider/engine SPI for third‑party engines (possible in a later phase).
+
+## 3) Public API (proposed)
+Top‑level entry: `@probelabs/visor/sdk` (via package `exports`).
+
+```ts
+// ESM
+import {
+  createVisor,
+  runChecks,
+  loadConfig,
+  resolveChecks,
+  type Visor,
+  type VisorOptions,
+  type RunOptions,
+  type RunSummary,
+  type VisorConfig,
+  type CheckResult,
+  type Issue,
+  type RoutingTrace,
+} from '@probelabs/visor/sdk';
+
+// CJS
+const {
+  createVisor,
+  runChecks,
+  loadConfig,
+  resolveChecks,
+} = require('@probelabs/visor/sdk');
+```
+
+### 3.1 Core types
+- `VisorOptions`
+  - `cwd?: string`
+  - `logger?: LoggerLike` (info/debug/warn/error)
+  - `ai?: { provider?: ProviderId; model?: string }`
+  - `parallelism?: number`
+  - `env?: NodeJS.ProcessEnv`
+- `RunOptions`
+  - `config: VisorConfig` or `configPath: string`
+  - `checks?: string[]` (IDs or `['all']`)
+  - `tags?: { include?: string[]; exclude?: string[] }`
+  - `inputs?: Record<string, unknown>` (templating context)
+  - `output?: { format?: 'table'|'markdown'|'json'|'sarif'; groupBy?: 'check'|'file' }`
+- `RunSummary`
+  - `checks: CheckResult[]`
+  - `issues: Issue[]` (all aggregated)
+  - `stats: { durationMs: number; successCount: number; failureCount: number }`
+  - `routing?: RoutingTrace[]`
+
+Public types will be exported from `src/types/*` and surfaced by the SDK bundle (`.d.ts`).
+
+### 3.2 Instances and helpers
+- `createVisor(opts?: VisorOptions): Visor`
+  - Returns an instance exposing:
+    - `events`: `EventEmitter` (typed)
+      - `check:start`, `check:success`, `check:fail`
+      - `routing:action` (run/goto/retry), `routing:loop`
+      - `log` (debug/info/warn/error)
+      - `done`
+    - `run(opts: RunOptions): Promise<RunSummary>`
+    - `stream(opts: RunOptions): AsyncIterable<VisorEvent>` (optional sugar)
+- `loadConfig(path: string): Promise<VisorConfig>`
+- `resolveChecks(checkIds: string[], config: VisorConfig): string[]` (dependency expansion)
+- `runChecks(runOpts: RunOptions & { visor?: VisorOptions }): Promise<RunSummary>` (one‑shot convenience)
+
+### 3.3 Routing access in SDK
+- The SDK honors `on_fail`/`on_success` from the provided config.
+- `RunSummary.routing` returns a concise, machine‑readable trace of routing actions (retry, goto, run_js, goto_js) including loop counters and decision metadata.
+- Per‑check attempt counters exposed in trace; forEach items are isolated.
+
+## 4) Packaging & Build
+- Keep CLI bundle via `@vercel/ncc` at `dist/index.js` for `bin` and GitHub Action.
+- Add SDK build via `tsup` (preferred) or Rollup:
+  - Inputs: `src/sdk.ts`
+  - Outputs: `dist/sdk/index.mjs` (ESM), `dist/sdk/index.cjs` (CJS), `dist/sdk/index.d.ts` (types)
+  - `sideEffects: false` for tree‑shaking
+  - Target: Node `>=18`
+  - Sourcemaps enabled
+- `package.json` changes (illustrative):
+  - `exports`:
+    ```json
+    {
+      ".": {
+        "import": "./dist/sdk/index.mjs",
+        "require": "./dist/sdk/index.cjs",
+        "types": "./dist/sdk/index.d.ts"
+      },
+      "./sdk": {
+        "import": "./dist/sdk/index.mjs",
+        "require": "./dist/sdk/index.cjs",
+        "types": "./dist/sdk/index.d.ts"
+      },
+      "./cli": {
+        "require": "./dist/index.js"
+      }
+    }
+    ```
+  - `types`: `./dist/sdk/index.d.ts`
+  - `files`: include `dist/` (SDK + CLI) and `defaults/`
+  - Scripts:
+    - `build:cli` (unchanged `ncc`)
+    - `build:sdk` (tsup for dual build + d.ts)
+    - `build` runs both
+
+## 5) Types & Stability
+- All exported types live in `src/types/` and are re‑exported from `src/sdk.ts`.
+- No `any` in public API; use discriminated unions for events and results.
+- Introduce `VisorError` with stable `code` (e.g., `CONFIG_NOT_FOUND`, `ROUTING_MAX_LOOPS_EXCEEDED`, `CHECK_FAILED`).
+- Semver policy: public types are part of the contract; breaking changes require a major bump.
+
+## 6) JS Sandbox Safety (run_js/goto_js)
+Out of scope for the SDK. The SDK does not introduce or maintain any sandbox. Safety is enforced by the core engine. The SDK will:
+- Pass through any routing/sandbox options to the core.
+- Surface resulting events and traces unchanged.
+
+If/when the core provides stricter sandbox controls (e.g., worker‑thread limits), the SDK will forward those knobs without adding behavior.
+
+## 7) Examples (to add alongside implementation)
+- `examples/sdk-basic.mjs` — load config, run checks, print table/markdown.
+- `examples/sdk-cjs.cjs` — CommonJS require + run.
+- `examples/sdk-types.ts` — TypeScript usage with typed events.
+- `examples/sdk-routing.ts` — demonstrates routing events and traces.
+
+## 8) Testing & CI
+- Unit: type tests for public API (tsd or TypeScript compile fixtures).
+- Integration:
+  - ESM import works on Node 18+.
+  - CJS require works on Node 18+.
+  - Event stream delivers expected sequence for simple config.
+  - Routing trace emitted for retry/goto; loop caps enforced deterministically.
+- Workflows:
+  - Add a matrix job `sdk-consume` to compile and run the three examples with `npm ci`.
+
+## 9) Migration & Backwards Compatibility
+- CLI remains at `dist/index.js` with existing `bin` and Action behavior.
+- No changes required for current users. New consumers import `@probelabs/visor/sdk`.
+- README will gain an “SDK Usage” section after implementation; `docs/NPM_USAGE.md` will link to it.
+
+## 10) Phased Plan
+1. RFC sign‑off (this doc).
+2. Create `src/sdk.ts` façade and re‑export stable types; wire through existing engine (`CheckExecutionEngine`).
+3. Add `tsup` and dual build; update `package.json` exports; keep `ncc` for CLI.
+4. (Core) Harden JS sandbox in engine as a separate track; SDK simply forwards options.
+5. Write examples and tests; add CI matrix for ESM/CJS/TS.
+6. Docs: README section and `docs/sdk.md` (guide). Release minor version.
+
+## 11) Open Questions
+- Do we want a browser/WebWorker build later? If yes, we’ll need to abstract fs/git and remove Node‑only deps from the core.
+- Should SDK expose a plugin registration API now (providers/check types), or defer until a stable SPI is designed?
+- Should `run()` also return an async iterator for live consumption, or keep `.events` as the primary streaming method?
+
+---
+
+If accepted, implementation will proceed under branch `feat/sdk` with a draft PR for incremental review.