@probelabs/visor 0.1.106 → 0.1.111

package/dist/docs/testing/assertions.md

@@ -0,0 +1,85 @@
+# Writing Assertions
+
+Assertions live under `expect:` and cover three surfaces:
+
+- `calls`: step counts and provider effects (GitHub ops)
+- `prompts`: final AI prompts (post templating/context)
+- `outputs`: step outputs with history and selectors
+
+## Calls
+
+```yaml
+expect:
+  calls:
+    - step: overview
+      exactly: 1
+    - provider: github
+      op: labels.add
+      at_least: 1
+      args:
+        contains: [feature, "review/effort:2"]
+```
+
+Counts are consistent everywhere: `exactly`, `at_least`, `at_most`.
+
+## Prompts
+
+```yaml
+expect:
+  prompts:
+    - step: overview
+      contains: ["feat: add user search", "diff --git a/src/search.ts"]
+    - step: comment-assistant
+      matches: "(?i)\\/visor\\s+help"
+    - step: overview
+      # Select the prompt that mentions a specific file
+      where:
+        contains: ["src/search.ts"]
+      contains: ["diff --git a/src/search.ts"]
+```
+
+- `contains`: required substrings
+- `not_contains`: forbidden substrings
+- `matches`: regex (prefix `(?i)` for case-insensitive)
+- `index`: `first` | `last` | N (default: last)
+- `where`: selector to choose a prompt from history using `contains`/`not_contains`/`matches` before applying the assertion
+
+Tip: enable `--prompt-max-chars` or `tests.defaults.prompt_max_chars` to cap stored prompt size for large diffs.
+
+## Outputs
+
+Use `path` with dot/bracket syntax. You can select by index or by a `where` probe over the same output history.
+
+```yaml
+expect:
+  outputs:
+    - step: validate-fact
+      index: 0
+      path: fact_id
+      equals: f1
+    - step: validate-fact
+      where: { path: fact_id, equals: f2 }
+      path: confidence
+      equals: high
+    - step: aggregate-validations
+      path: all_valid
+      equals: true
+```
+
+Supported comparators:
+- `equals` (primitive)
+- `equalsDeep` (structural)
+- `matches` (regex)
+- `contains_unordered` (array membership ignoring order)
+
+## Strict mode and “no calls”
+
+Strict mode (default) fails any executed step without a corresponding `expect.calls` entry. You can also assert absence explicitly:
+
+```yaml
+expect:
+  no_calls:
+    - provider: github
+      op: issues.createComment
+    - step: extract-facts
+```

package/dist/docs/testing/ci.md

@@ -0,0 +1,44 @@
+# CI Integration for Tests
+
+Run your in‑YAML integration tests in CI using the Visor CLI. Below is a GitHub Actions example. Adapt for other CIs similarly.
+
+```yaml
+name: Visor Tests
+on: [pull_request]
+
+jobs:
+  tests:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-node@v4
+        with: { node-version: '20' }
+      - run: npm ci
+      - run: npm run build --ignore-scripts
+
+      - name: Run integration tests (defaults)
+        run: |
+          mkdir -p output
+          node ./dist/index.js test \
+            --config defaults/.visor.tests.yaml \
+            --json output/visor-tests.json \
+            --report junit:output/visor-tests.xml \
+            --summary md:output/visor-tests.md
+
+      - name: Upload test artifacts
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: visor-test-results
+          path: |
+            output/visor-tests.json
+            output/visor-tests.xml
+            output/visor-tests.md
+```
+
+Tips
+- Keep `ai_provider: mock` in `tests.defaults` for fast, deterministic runs.
+- Set `--max-parallel` to speed up large suites (flows still run sequentially per case).
+- Use `--bail` for faster feedback on PRs; run full suite on main.
+- Collect artifacts so you can inspect failures without re-running.
+

package/dist/docs/testing/cli.md

@@ -0,0 +1,41 @@
+# Visor Test CLI
+
+Run integration tests for your Visor config using the built-in `test` subcommand.
+
+## Commands
+
+- Discover tests file and list cases
+  - `visor test --list [--config defaults/.visor.tests.yaml]`
+- Run cases
+  - `visor test [--config defaults/.visor.tests.yaml] [--only <substring>] [--bail]`
+- Validate tests YAML without running
+  - `visor test --validate [--config defaults/.visor.tests.yaml]`
+
+## Flags
+
+- `--config <path>`: Path to `.visor.tests.yaml` (auto-discovers `.visor.tests.yaml` or `defaults/.visor.tests.yaml`).
+- `--only <filter>`: Run cases whose `name` contains the substring (case-insensitive).
+ - `--only <filter>`: Run cases whose `name` contains the substring (case-insensitive).
+   - Stage filter: append `#<stage>` to run only a flow stage.
+     - Examples: `--only pr-review-e2e-flow#facts-invalid`, `--only pr-review-e2e-flow#3` (1‑based index)
+- `--bail`: Stop on first failure.
+- `--json <path|->`: Write a minimal JSON summary.
+- `--report junit:<path>`: Write a minimal JUnit XML.
+- `--summary md:<path>`: Write a minimal Markdown summary.
+- `--progress compact|detailed`: Progress verbosity (parsing supported; detailed view evolves over time).
+- `--max-parallel <N>`: Run up to N cases concurrently.
+- `--prompt-max-chars <N>`: Truncate captured prompt text to N characters.
+
+## Output
+
+- Per-case PASS/FAIL lines
+- Coverage table (expected vs actual step runs)
+- Summary totals
+
+## Tips
+
+- Use `--validate` when iterating on tests to catch typos early.
+- Keep `strict: true` in `tests.defaults` to surface missing `expect` quickly.
+- For large suites, increase `--max-parallel` to improve throughput.
+- Enable debug logs by setting `VISOR_DEBUG=true`.
+  - Example: `VISOR_DEBUG=true visor test --config defaults/.visor.tests.yaml --only pr-review-e2e-flow#facts-invalid`

package/dist/docs/testing/cookbook.md

@@ -0,0 +1,172 @@
+# Test Cookbook
+
+Copy‑pasteable recipes for common scenarios.
+
+## 1) Label PR on open
+
+```yaml
+- name: label-flow
+  event: pr_opened
+  fixture: gh.pr_open.minimal
+  mocks:
+    overview:
+      text: "Overview body"
+      tags: { label: feature, review-effort: 2 }
+  expect:
+    calls:
+      - step: overview
+        exactly: 1
+      - step: apply-overview-labels
+        exactly: 1
+      - provider: github
+        op: labels.add
+        at_least: 1
+        args: { contains: [feature, "review/effort:2"] }
+```
+
+## 2) Ignore normal comment
+
+```yaml
+- name: standard-comment
+  event: issue_comment
+  fixture: gh.issue_comment.standard
+  mocks:
+    comment-assistant: { text: "", intent: comment_reply }  # empty text → no reply
+  expect:
+    no_calls:
+      - provider: github
+        op: issues.createComment
+    calls:
+      - step: comment-assistant
+        exactly: 1
+```
+
+## 3) `/visor help` reply and prompt check
+
+```yaml
+- name: visor-plain
+  event: issue_comment
+  fixture: gh.issue_comment.visor_help
+  mocks:
+    comment-assistant: { text: "Sure, here’s how I can help.", intent: comment_reply }
+  expect:
+    calls:
+      - step: comment-assistant
+        exactly: 1
+      - provider: github
+        op: issues.createComment
+        exactly: 1
+    prompts:
+      - step: comment-assistant
+        matches: "(?i)\\/visor\\s+help"
+```
+
+## 4) Regenerate reviews on command
+
+```yaml
+- name: visor-retrigger
+  event: issue_comment
+  fixture: gh.issue_comment.visor_regenerate
+  mocks:
+    comment-assistant: { text: "Regenerating.", intent: comment_retrigger }
+    overview: { text: "Overview (regenerated)", tags: { label: feature, review-effort: 2 } }
+  expect:
+    calls:
+      - step: comment-assistant
+        exactly: 1
+      - step: overview
+        exactly: 1
+```
+
+## 5) Facts enabled (one fact)
+
+```yaml
+- name: facts-enabled
+  event: issue_comment
+  fixture: gh.issue_comment.visor_help
+  env: { ENABLE_FACT_VALIDATION: "true" }
+  mocks:
+    comment-assistant: { text: "We rely on defaults/.visor.yaml line 11 for max_parallelism=4.", intent: comment_reply }
+    extract-facts:
+      - { id: f1, category: Configuration, claim: "max_parallelism defaults to 4", verifiable: true }
+    validate-fact[]:
+      - { fact_id: f1, is_valid: true, confidence: high, evidence: "defaults/.visor.yaml:11" }
+  expect:
+    calls:
+      - step: extract-facts
+        exactly: 1
+      - step: validate-fact
+        at_least: 1
+      - step: aggregate-validations
+        exactly: 1
+```
+
+## 6) Facts invalid (correction reply)
+
+```yaml
+- name: facts-invalid
+  event: issue_comment
+  fixture: gh.issue_comment.visor_help
+  env: { ENABLE_FACT_VALIDATION: "true" }
+  mocks:
+    comment-assistant: { text: "We rely on defaults/.visor.yaml line 11 for max_parallelism=4.", intent: comment_reply }
+    extract-facts:
+      - { id: f1, category: Configuration, claim: "max_parallelism defaults to 4", verifiable: true }
+    validate-fact[]:
+      - { fact_id: f1, is_valid: false, confidence: high, evidence: "defaults/.visor.yaml:11", correction: "max_parallelism defaults to 3" }
+  expect:
+    calls:
+      - step: comment-assistant
+        exactly: 2
+      - step: aggregate-validations
+        exactly: 1
+    prompts:
+      - step: comment-assistant
+        index: last
+        contains: ["<previous_response>", "Correction: max_parallelism defaults to 3"]
+```
+
+## 7) Two facts (one invalid)
+
+```yaml
+- name: facts-two-items
+  event: issue_comment
+  fixture: gh.issue_comment.visor_help
+  env: { ENABLE_FACT_VALIDATION: "true" }
+  mocks:
+    comment-assistant: { text: "We rely on defaults/.visor.yaml for concurrency defaults.", intent: comment_reply }
+    extract-facts:
+      - { id: f1, category: Configuration, claim: "max_parallelism defaults to 4", verifiable: true }
+      - { id: f2, category: Feature,       claim: "Fast mode is enabled by default", verifiable: true }
+    validate-fact[]:
+      - { fact_id: f1, is_valid: false, confidence: high, evidence: "defaults/.visor.yaml:11", correction: "max_parallelism defaults to 3" }
+      - { fact_id: f2, is_valid: true,  confidence: high, evidence: "src/config.ts:FAST_MODE=true" }
+  expect:
+    calls:
+      - step: validate-fact
+        exactly: 2
+    prompts:
+      - step: comment-assistant
+        index: last
+        contains: ["max_parallelism defaults to 4", "Correction: max_parallelism defaults to 3"]
+        not_contains: ["Fast mode is enabled by default"]
+```
+
+## 8) GitHub negative mode
+
+```yaml
+- name: github-negative-mode
+  event: pr_opened
+  fixture: gh.pr_open.minimal
+  github_recorder: { error_code: 429 }
+  mocks: { overview: { text: "Overview body", tags: { label: feature, review-effort: 2 } } }
+  expect:
+    calls:
+      - step: overview
+        exactly: 1
+      - step: apply-overview-labels
+        exactly: 1
+    fail:
+      message_contains: "github/op_failed"
+```
+

package/dist/docs/testing/dsl-reference.md

@@ -0,0 +1,199 @@
+# Test DSL Reference
+
+This page documents the `.visor.tests.yaml` schema used by the Visor test runner.
+
+```yaml
+version: "1.0"
+extends: ".visor.yaml"   # required; base config to run under tests
+
+tests:
+  defaults:
+    strict: true                # default strict mode
+    ai_provider: mock           # force AI provider to mock
+    prompt_max_chars: 16000     # truncate captured prompts (optional)
+    github_recorder:            # optional negative modes
+      error_code: 0             # e.g., 429
+      timeout_ms: 0             # e.g., 1000
+    # Optional: include/exclude checks by tags (same semantics as main CLI)
+    tags: "local,fast"         # or [local, fast]
+    exclude_tags: "experimental,slow"  # or [experimental, slow]
+
+  fixtures: []                  # (optional) suite-level custom fixtures
+
+  cases:
+    - name: <string>
+      description: <markdown>
+      skip: false|true
+
+      # Single-event case
+      event: pr_opened | pr_updated | pr_closed | issue_opened | issue_comment | manual
+      fixture: <builtin|{ builtin, overrides }>
+      env: { <KEY>: <VALUE>, ... }
+      mocks: { <step>: <value>, <step>[]: [<value>...] }
+      expect: <expect-block>
+      strict: true|false         # overrides defaults.strict
+      tags: "security,fast"     # optional per-case include filter
+      exclude_tags: "slow"      # optional per-case exclude filter
+
+      # OR flow case
+      flow:
+        - name: <string>
+          event: ...             # per-stage event and fixture
+          fixture: ...
+          env: ...
+          mocks: ...             # merged with flow-level mocks
+          expect: <expect-block>
+          strict: true|false     # per-stage fallback to case/defaults
+          tags: "security"       # optional per-stage include filter
+          exclude_tags: "slow"   # optional per-stage exclude filter
+```
+
+## Fixtures
+
+- Built-in GitHub fixtures: `gh.pr_open.minimal`, `gh.pr_sync.minimal`, `gh.issue_open.minimal`, `gh.issue_comment.standard`, `gh.issue_comment.visor_help`, `gh.issue_comment.visor_regenerate`.
+- Use `overrides` to tweak titles, numbers, payload slices.
+
+## Mocks
+
+- Keys are step names; for forEach children use `step[]` (e.g., `validate-fact[]`).
+- AI mocks may be structured JSON if a schema is configured for the step; otherwise use `text` and optional fields used by templates.
+- Command/HTTP mocks emulate provider shape (`stdout`, `exit_code`, or HTTP body/status headers) and bypass real execution.
+
+Inline example (AI with schema + list mocks):
+
+```yaml
+mocks:
+  overview:
+    text: "Overview body"
+    tags: { label: feature, review-effort: 2 }
+  extract-facts:
+    - { id: f1, claim: "max_parallelism defaults to 4" }
+    - { id: f2, claim: "Fast mode is enabled by default" }
+  validate-fact[]:
+    - { fact_id: f1, is_valid: false, correction: "max_parallelism defaults to 3" }
+    - { fact_id: f2, is_valid: true }
+```
+
+## Expect block
+
+```yaml
+expect:
+  calls:
+    - step: <name> | provider: github + op: <rest.op>
+      exactly|at_least|at_most: <number>
+      args: { contains: [..], not_contains: [..] }   # provider args matching
+
+  no_calls:
+    - step: <name> | provider: github + op: <rest.op>
+
+  prompts:
+    - step: <name>
+      index: first|last|<N>     # default: last
+      where:                    # select a prompt from history, then assert
+        contains: [..] | not_contains: [..] | matches: <regex>
+      contains: [..]
+      not_contains: [..]
+      matches: <regex>
+
+  outputs:
+    - step: <name>
+      index: first|last|<N>     # or
+      where: { path: <expr>, equals|matches: <v> }
+      path: <expr>              # dot/bracket, e.g. tags['review-effort']
+      equals: <primitive>
+      equalsDeep: <object>
+      matches: <regex>
+      contains_unordered: [..]
+
+  fail:
+    message_contains: <string>  # assert overall case failure message
+
+  strict_violation:             # assert strict failure for a missing expect on a step
+    for_step: <name>
+    message_contains: <string>
+
+Inline example (calls + prompts + outputs):
+
+```yaml
+expect:
+  calls:
+    - step: overview
+      exactly: 1
+    - provider: github
+      op: labels.add
+      at_least: 1
+      args: { contains: [feature] }
+  prompts:
+    - step: overview
+      contains: ["feat:", "diff --git a/"]
+  outputs:
+    - step: overview
+      path: "tags['review-effort']"
+      equals: 2
+```
+```
+
+Note on dependencies: test execution honors your base config routing, including `depends_on`. You can express ANY‑OF groups using pipe syntax in the base config (e.g., `depends_on: ["issue-assistant|comment-assistant"]`). The runner mixes these with normal ALL‑OF deps.
+
+## Strict mode semantics
+
+- When `strict: true` (default), any executed step must appear in `expect.calls` with a matching count; otherwise the case/stage fails.
+- Use `no_calls` for explicit absence checks.
+
+## Selectors and paths
+
+- `index`: `first`, `last`, or 0‑based integer.
+- `where`: evaluates against the same prompt/output history and selects a single item by content.
+- `path`: dot/bracket (supports quoted keys: `tags['review-effort']`).
+
+## CLI shortcuts
+
+- Validate only: `visor test --validate --config <path>`
+- Run one case: `visor test --only label-flow`
+- Run one stage: `visor test --only pr-review-e2e-flow#facts-invalid`
+- JSON/JUnit/Markdown reporters: `--json`, `--report junit:<path>`, `--summary md:<path>`
+## JavaScript in Tests and Routing (run_js, goto_js, value_js, transform_js)
+
+### Tags default semantics in tests
+
+- The test runner passes tags to the engine using the same rules as the main CLI.
+- If no tags/exclude_tags are specified anywhere (suite defaults, case, or stage), only untagged checks run by default; tagged checks are skipped. This keeps tests deterministic and fast unless you explicitly opt into groups (for example, `github`).
+- To run GitHub‑tagged checks in tests, add:
+
+```yaml
+tests:
+  defaults:
+    tags: "github"
+```
+
+Visor evaluates your `run_js`, `goto_js`, `value_js` and `transform_js` snippets inside a hardened JavaScript sandbox. The goal is to provide a great developer experience with modern JS, while keeping the engine safe and deterministic.
+
+What you can use by default (Node 24, ES2023)
+- Language features: `const/let`, arrow functions, template strings, destructuring, spread, async/await, `Array.prototype.at`, `findLast`/`findLastIndex`.
+- Arrays: iteration helpers (`map`, `filter`, `some`, `every`, `reduce`, `keys/values/entries`, `forEach`), non‑mutating helpers (`toReversed`, `toSorted`, `toSpliced`, `with`), and `flat/flatMap`.
+- Strings: `replaceAll`, `matchAll`, `trimStart/End`, `at`, `repeat`, `normalize`.
+- Maps/Sets: `get/set/has/delete/keys/values/entries/forEach`.
+- Date/RegExp: `toISOString`, `getTime`, `test`, `exec`.
+
+What remains intentionally restricted
+- Prototype mutation and reflective escape hatches (e.g., `Object.defineProperty`, `__proto__`, `setPrototypeOf`) are not exposed to sandboxed code.
+- `if:` and `fail_if:` conditions are parsed by a small expression DSL (not full JS). Keep them simple (no optional chaining or nullish coalescing in those), or move complex logic to `run_js`/`goto_js`.
+
+Tips
+- Prefer non‑mutating array helpers (`toReversed`, `toSorted`, `with`) when deriving new arrays for clarity and correctness.
+- Use `Array.prototype.at(-1)` to read the last item. Example: `const last = (outputs_history['validate-fact'] || []).at(-1) || [];`.
+- For reshaping small maps, `Object.entries` + `Object.fromEntries` is concise and readable.
+
+Example: wave‑scoped correction gate
+```
+run_js: |
+  const facts = (outputs_history['extract-facts'] || []).at(-1) || [];
+  const ids = facts.map(f => String(f.id || '')).filter(Boolean);
+  const vf = outputs_history['validate-fact'] || [];
+  const lastItems = vf.filter(v => ids.includes(String((v && v.fact_id) || '')));
+  const hasProblems = lastItems.some(v => v.is_valid !== true || v.confidence !== 'high');
+  if (!hasProblems) return [];
+  return (event && event.name) === 'issue_opened' ? ['issue-assistant'] : ['comment-assistant'];
+```
+
+This evaluates the last `extract-facts` wave, finds the corresponding `validate-fact` results, and schedules a single correction pass when any item is invalid or low‑confidence.

package/dist/docs/testing/fixtures-and-mocks.md

@@ -0,0 +1,91 @@
+# Fixtures and Mocks
+
+Integration tests simulate outside world inputs and provider outputs.
+
+## Built-in GitHub fixtures (gh.*)
+
+Use via `fixture: gh.<name>` or `fixture: { builtin: gh.<name>, overrides: {...} }`.
+
+- `gh.pr_open.minimal` — pull_request opened with a tiny diff and `src/search.ts` file.
+- `gh.pr_sync.minimal` — pull_request synchronize with a small follow-up diff.
+- `gh.pr_closed.minimal` — pull_request closed event.
+- `gh.issue_open.minimal` — issues opened (short title/body).
+- `gh.issue_comment.standard` — normal human comment on a PR/issue.
+- `gh.issue_comment.visor_help` — comment containing `/visor help`.
+- `gh.issue_comment.visor_regenerate` — `/visor Regenerate reviews`.
+
+Overrides allow tailored inputs:
+
+```yaml
+fixture:
+  builtin: gh.pr_open.minimal
+  overrides:
+    pr.title: "feat: custom title"
+    webhook.payload.pull_request.number: 42
+```
+
+## GitHub recorder
+
+The test runner injects a recording Octokit by default:
+
+- Captures every GitHub op+args for assertions (`expect.calls` with `provider: github`).
+- Returns stable stub shapes so flows can continue without network.
+- Negative modes are available globally via `tests.defaults.github_recorder`:
+
+```yaml
+tests:
+  defaults:
+    github_recorder:
+      error_code: 429      # simulate API error
+      timeout_ms: 1000     # simulate request timeout
+```
+
+## Mocks
+
+Mocks are keyed by step name under `mocks`.
+
+Note: step names in examples (e.g., `extract-facts`, `validate-fact`) are not built‑ins — they are ordinary user‑defined checks in your `.visor.yaml`.
+
+Examples:
+
+```yaml
+mocks:
+  # AI with structured schema
+  overview:
+    text: "High-level PR summary."
+    tags: { label: feature, review-effort: 2 }
+
+  # AI plain text schema
+  comment-assistant:
+    text: "Sure, here’s how I can help."
+    intent: comment_reply
+
+  # Array outputs (e.g., extract-facts)
+  extract-facts:
+    - { id: f1, category: Configuration, claim: "max_parallelism defaults to 4", verifiable: true }
+
+  # Command provider
+  unit-tests:
+    stdout: '{"passed": 128, "failed": 0}'
+    exit_code: 0
+
+### Per‑call list mocks (for forEach children)
+
+When a step runs once per item (e.g., `validate-fact` depends on `extract-facts`), provide a list under the `[]` suffix:
+
+```yaml
+mocks:
+  extract-facts:
+    - { id: f1, category: Configuration, claim: "max_parallelism defaults to 4", verifiable: true }
+    - { id: f2, category: Feature,       claim: "Fast mode is enabled by default", verifiable: true }
+  validate-fact[]:
+    - { fact_id: f1, is_valid: false, confidence: high, evidence: "defaults/.visor.yaml:11", correction: "max_parallelism defaults to 3" }
+    - { fact_id: f2, is_valid: true,  confidence: high, evidence: "src/config.ts:FAST_MODE=true" }
+```
+
+The runner distributes items in order; if the list is shorter than invocations, the last entry is reused. This fits any forEach‑style step you define (naming is up to you).
+```
+
+Notes:
+- No `returns:` key; provide values directly.
+- For HTTP/Command providers, mocks bypass real execution and are recorded for assertions.