@probelabs/visor 0.1.124 → 0.1.126

package/dist/docs/telemetry-setup.md

@@ -4,34 +4,66 @@ This guide shows how to enable Visor telemetry and tracing with OpenTelemetry, e
 
 ## Quick Start (CLI)
 
-- Enable telemetry to serverless NDJSON traces:
-  - `VISOR_TELEMETRY_ENABLED=true`
-  - `VISOR_TELEMETRY_SINK=file`
-  - (optional) `VISOR_TRACE_DIR=output/traces`
-- Run:
-  - `visor --config ./.visor.yaml --output json`
-- Inspect traces:
-  - `ls output/traces/*.ndjson`
-
-## CLI Flags
-
-- `--telemetry` — enable telemetry (overrides config)
-- `--telemetry-sink <otlp|file|console>` — sink selection
-- `--telemetry-endpoint <url>` — OTLP endpoint (HTTP) for traces/metrics
-- `--trace-report` — write a static HTML trace report to output/traces
-- `--auto-instrumentations` — enable OpenTelemetry auto‑instrumentations
+Enable telemetry to serverless NDJSON traces:
+
+```bash
+export VISOR_TELEMETRY_ENABLED=true
+export VISOR_TELEMETRY_SINK=file
+export VISOR_TRACE_DIR=output/traces  # optional, defaults to output/traces
+
+visor --config ./.visor.yaml --output json
+
+# Inspect traces
+ls output/traces/*.ndjson
+```
+
+## Environment Variables
+
+Telemetry is configured via environment variables (highest precedence):
+
+| Variable | Description | Default |
+|----------|-------------|---------|
+| `VISOR_TELEMETRY_ENABLED` | Enable telemetry (`true`/`false`) | `false` |
+| `VISOR_TELEMETRY_SINK` | Sink type: `otlp`, `file`, or `console` | `file` |
+| `VISOR_TRACE_DIR` | Directory for trace files | `output/traces` |
+| `VISOR_TRACE_REPORT` | Generate static HTML trace report (`true`/`false`) | `false` |
+| `VISOR_TELEMETRY_AUTO_INSTRUMENTATIONS` | Enable auto‑instrumentations (`true`/`false`) | `false` |
+| `VISOR_TELEMETRY_FULL_CAPTURE` | Capture full AI prompts/responses in spans | `false` |
+| `VISOR_FALLBACK_TRACE_FILE` | Explicit path for NDJSON trace file | auto-generated |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP endpoint URL (for both traces and metrics) | - |
+| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | OTLP endpoint for traces (overrides above) | - |
+| `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` | OTLP endpoint for metrics (overrides above) | - |
+| `OTEL_EXPORTER_OTLP_HEADERS` | Headers for OTLP requests (e.g., auth tokens) | - |
 
 Examples:
-- `visor --config ./.visor.yaml --telemetry --telemetry-sink otlp --telemetry-endpoint https://otel.example.com`
-- `visor --config ./.visor.yaml --telemetry --trace-report --auto-instrumentations`
+
+```bash
+# File sink (serverless mode)
+VISOR_TELEMETRY_ENABLED=true \
+VISOR_TELEMETRY_SINK=file \
+visor --config ./.visor.yaml
+
+# OTLP sink with Jaeger
+VISOR_TELEMETRY_ENABLED=true \
+VISOR_TELEMETRY_SINK=otlp \
+OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:4318/v1/traces \
+visor --config ./.visor.yaml
+
+# With static HTML trace report
+VISOR_TELEMETRY_ENABLED=true \
+VISOR_TRACE_REPORT=true \
+visor --config ./.visor.yaml
+```
 
 ## Config (visor.yaml)
 
+Telemetry can also be configured via the `telemetry` section in your config file:
+
 ```yaml
 version: "1.0"
 telemetry:
   enabled: true
-  sink: file       # otlp|file|console
+  sink: file       # otlp | file | console
   otlp:
     protocol: http
     endpoint: ${OTEL_EXPORTER_OTLP_ENDPOINT}
@@ -45,18 +77,16 @@ telemetry:
       enabled: true
 ```
 
-ENV overrides (highest precedence):
-- `VISOR_TELEMETRY_ENABLED`, `VISOR_TELEMETRY_SINK`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`, `VISOR_TRACE_DIR`
-- `VISOR_TELEMETRY_AUTO_INSTRUMENTATIONS=true`
-- `VISOR_TRACE_REPORT=true`
+> **Note:** Environment variables take precedence over config file settings.
 
 ## Serverless Mode (NDJSON)
 
-- Visor writes NDJSON simplified spans to `output/traces/run-<id>-<ts>.ndjson`.
-- Ingest with OTel Collector `filelog` receiver + transform to OTLP.
+When using `VISOR_TELEMETRY_SINK=file` (the default), Visor writes NDJSON simplified spans to `output/traces/run-<timestamp>.ndjson`. This is ideal for serverless/CI environments where you can't run a persistent collector.
+
+You can then ingest these files using the OTel Collector `filelog` receiver:
 
-OTel Collector (example):
 ```yaml
+# otel-collector-config.yaml
 receivers:
   filelog:
     include: [ "/work/output/traces/*.ndjson" ]
@@ -75,45 +105,126 @@ service:
 
 ## Connected Mode (OTLP HTTP)
 
-- Set `VISOR_TELEMETRY_SINK=otlp` and `OTEL_EXPORTER_OTLP_ENDPOINT=https://collector.example.com`.
-- Metrics exporter is enabled automatically (optional dependency) — histograms/counters for checks, providers, foreach items, fail_if triggers, and diagram blocks.
+For real-time trace streaming to a collector:
+
+```bash
+export VISOR_TELEMETRY_ENABLED=true
+export VISOR_TELEMETRY_SINK=otlp
+export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=https://collector.example.com/v1/traces
+# Optional: authentication headers
+export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer your-token"
+```
+
+When using OTLP sink, the metrics exporter is automatically enabled if the required dependencies are installed (`@opentelemetry/exporter-metrics-otlp-http`, `@opentelemetry/sdk-metrics`). Metrics include histograms and counters for checks, providers, forEach items, and fail_if triggers.
 
 ## Auto‑Instrumentations
 
-- Enable with `--auto-instrumentations` or `telemetry.tracing.auto_instrumentations: true`.
-- Adds `@opentelemetry/auto-instrumentations-node` (http/undici/child_process/etc.) and correlates with Visor spans via context.
-- Optional dependency; if not installed, Visor skips auto‑instrumentation gracefully.
+Enable with `VISOR_TELEMETRY_AUTO_INSTRUMENTATIONS=true` or in config:
+
+```yaml
+telemetry:
+  tracing:
+    auto_instrumentations: true
+```
+
+This activates `@opentelemetry/auto-instrumentations-node` (http/undici/child_process/etc.) and correlates external calls with Visor spans via context propagation.
+
+> **Note:** Auto-instrumentations require `@opentelemetry/auto-instrumentations-node` as an optional dependency. If not installed, Visor skips auto‑instrumentation gracefully.
 
 ## Static Trace Report
 
-- Enable `--trace-report` or `telemetry.tracing.trace_report.enabled: true`.
-- Outputs two files per run:
-  - `*.trace.json` — simplified span JSON
-  - `*.report.html` — self‑contained HTML timeline (open locally)
+Enable with `VISOR_TRACE_REPORT=true` or in config:
 
-## Mermaid Telemetry
+```yaml
+telemetry:
+  tracing:
+    trace_report:
+      enabled: true
+```
+
+This outputs two files per run to your trace directory:
+- `*.trace.json` — simplified span JSON
+- `*.report.html` — self‑contained HTML timeline (open locally in your browser)
+
+## Span Attributes and Events
 
-- Visor emits full `diagram.block` events with Mermaid code from outputs and issue messages.
-- Metric: `visor.diagram.blocks{origin}` increments per diagram block.
+Visor emits spans with detailed attributes for debugging:
+
+### Check Spans (`visor.check.<checkId>`)
+- `visor.check.id` — Check identifier
+- `visor.check.type` — Provider type (ai, command, etc.)
+- `visor.check.input.context` — Liquid template context (sanitized)
+- `visor.check.output` — Check result (truncated if large)
+- `visor.foreach.index` — Index for forEach iterations
+
+### State Spans (`engine.state.*`)
+- `wave` — Current execution wave number
+- `wave_kind` — Wave type
+- `session_id` — Session identifier
+- `level_size` — Number of checks in wave
+- `level_checks_preview` — Preview of checks in wave
+
+### Routing Events (`visor.routing`)
+- `trigger` — What triggered the routing decision
+- `action` — Routing action (retry, goto, run)
+- `source` — Source check
+- `target` — Target check(s)
+- `scope` — Execution scope
+- `goto_event` — Event override for goto
 
 ## Security & Redaction
 
-- Diagram events are sent verbatim by default (as requested). You can later opt‑in to redaction via `telemetry.redaction` (not enforced by default).
+By default, sensitive environment variables (containing `api_key`, `secret`, `token`, `password`, `auth`, `credential`, `private_key`) are automatically redacted in span attributes.
+
+To capture full AI prompts and responses (for debugging), set:
+```bash
+export VISOR_TELEMETRY_FULL_CAPTURE=true
+```
+
+> **Warning:** Full capture may include sensitive data. Use only in secure debugging environments.
 
 ## GitHub Actions
 
-- Visor wraps the Action run in a single root span (`visor.run`). Publish the `trace_id` in logs/checks for linking.
-- Example step:
+Visor wraps each execution in a root span (`visor.run`). You can correlate traces with GitHub workflow runs by publishing the `trace_id` in logs or checks.
+
+Example workflow step:
+
 ```yaml
-- name: Visor
+- name: Run Visor with tracing
   run: |
     export VISOR_TELEMETRY_ENABLED=true
     export VISOR_TELEMETRY_SINK=otlp
-    export OTEL_EXPORTER_OTLP_ENDPOINT=${{ secrets.OTEL_ENDPOINT }}
+    export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=${{ secrets.OTEL_ENDPOINT }}
     export OTEL_EXPORTER_OTLP_HEADERS="Authorization=Bearer ${{ secrets.OTEL_TOKEN }}"
     npx -y @probelabs/visor@latest --config ./.visor.yaml --output json
 ```
 
-Troubleshooting:
-- No spans? Check `VISOR_TELEMETRY_ENABLED`, `VISOR_TELEMETRY_SINK`, and that optional deps resolved in the environment.
-- Huge mermaid outputs? Consider adding a soft length cap in Visor or pre-truncating in templates.
+For file-based tracing in CI (useful for artifact upload):
+
+```yaml
+- name: Run Visor with file traces
+  run: |
+    export VISOR_TELEMETRY_ENABLED=true
+    export VISOR_TELEMETRY_SINK=file
+    export VISOR_TRACE_DIR=./traces
+    npx -y @probelabs/visor@latest --config ./.visor.yaml
+
+- name: Upload traces
+  uses: actions/upload-artifact@v4
+  with:
+    name: visor-traces
+    path: ./traces/*.ndjson
+```
+
+## Troubleshooting
+
+- **No spans?** Verify `VISOR_TELEMETRY_ENABLED=true` and check that OpenTelemetry packages are installed.
+- **Missing metrics?** Install `@opentelemetry/exporter-metrics-otlp-http` and `@opentelemetry/sdk-metrics`.
+- **Auto-instrumentations not working?** Install `@opentelemetry/auto-instrumentations-node`.
+- **Large span attributes?** Visor truncates attributes at 10,000 characters. For full capture, use `VISOR_TELEMETRY_FULL_CAPTURE=true`.
+
+## Related Documentation
+
+- [Debugging Guide](./debugging.md) — Comprehensive debugging techniques
+- [Debug Visualizer](./debug-visualizer.md) — Live execution visualization with `--debug-server`
+- [Telemetry RFC](./telemetry-tracing-rfc.md) — Design rationale and architecture

package/dist/docs/test-framework-rfc.md

@@ -1,7 +1,8 @@
 # Visor Integration Test Framework (RFC)
 
-Status: In Progress
+Status: Implemented
 Date: 2025-10-27
+Last Updated: 2026-01-28
 Owners: @probelabs/visor
 
 ## Summary
@@ -17,15 +18,15 @@ Key ideas:
   (Docs use a “fact validation” workflow as an example pattern only; it is not a built‑in feature.)
 
 Developer Guides
-- Getting started: docs/testing/getting-started.md
-- DSL reference: docs/testing/dsl-reference.md
-- Flows: docs/testing/flows.md
-- Fixtures & mocks: docs/testing/fixtures-and-mocks.md
-- Assertions: docs/testing/assertions.md
-- Cookbook: docs/testing/cookbook.md
-- CLI & reporters: docs/testing/cli.md
-- CI integration: docs/testing/ci.md
-- Troubleshooting: docs/testing/troubleshooting.md
+- Getting started: [testing/getting-started.md](testing/getting-started.md)
+- DSL reference: [testing/dsl-reference.md](testing/dsl-reference.md)
+- Flows: [testing/flows.md](testing/flows.md)
+- Fixtures & mocks: [testing/fixtures-and-mocks.md](testing/fixtures-and-mocks.md)
+- Assertions: [testing/assertions.md](testing/assertions.md)
+- Cookbook: [testing/cookbook.md](testing/cookbook.md)
+- CLI & reporters: [testing/cli.md](testing/cli.md)
+- CI integration: [testing/ci.md](testing/ci.md)
+- Troubleshooting: [testing/troubleshooting.md](testing/troubleshooting.md)
 
 ## Progress Update (Oct 29, 2025)
 
@@ -64,7 +65,7 @@ Next steps (milestones excerpt)
 ## File Layout
 
 - Base config (unchanged): `defaults/.visor.yaml` (regular steps live here).
-- Test suite (new): `defaults/.visor.tests.yaml`
+- Test suite (new): `defaults/visor.tests.yaml`
   - `extends: ".visor.yaml"` to inherit the base checks.
   - Contains `tests.defaults`, `tests.fixtures`, `tests.cases`.
 
@@ -226,16 +227,16 @@ tests:
 ## CLI Usage
 
 - Discover tests:
-  - `node dist/index.js test --config defaults/.visor.tests.yaml --list`
+  - `node dist/index.js test --config defaults/visor.tests.yaml --list`
 - Validate test file shape (schema):
-  - `node dist/index.js test --config defaults/.visor.tests.yaml --validate`
+  - `node dist/index.js test --config defaults/visor.tests.yaml --validate`
 - Run all tests with compact progress (default):
-  - `node dist/index.js test --config defaults/.visor.tests.yaml`
+  - `node dist/index.js test --config defaults/visor.tests.yaml`
 - Run a single case:
-  - `node dist/index.js test --config defaults/.visor.tests.yaml --only label-flow`
+  - `node dist/index.js test --config defaults/visor.tests.yaml --only label-flow`
 - Run a single stage in a flow (by name or 1‑based index):
-  - `node dist/index.js test --config defaults/.visor.tests.yaml --only pr-review-e2e-flow#facts-invalid`
-  - `node dist/index.js test --config defaults/.visor.tests.yaml --only pr-review-e2e-flow#3`
+  - `node dist/index.js test --config defaults/visor.tests.yaml --only pr-review-e2e-flow#facts-invalid`
+  - `node dist/index.js test --config defaults/visor.tests.yaml --only pr-review-e2e-flow#3`
 - Emit artifacts:
   - JSON: `--json output/visor-tests.json`
   - JUnit: `--report junit:output/visor-tests.xml`
@@ -433,9 +434,9 @@ expect:
 ## CLI
 
 ```
-visor test --config defaults/.visor.tests.yaml         # run all cases
-visor test --config defaults/.visor.tests.yaml --only pr-review-e2e-flow
-visor test --config defaults/.visor.tests.yaml --list  # list case names
+visor test --config defaults/visor.tests.yaml         # run all cases
+visor test --config defaults/visor.tests.yaml --only pr-review-e2e-flow
+visor test --config defaults/visor.tests.yaml --list  # list case names
 ```
 
 Exit codes:
@@ -459,7 +460,7 @@ The runner prints a concise, human‑friendly summary optimized for scanning:
   - First mismatch shows an inline diff (expected vs actual substring/regex or value), with a clear hint to fix.
 - Flow cases show each stage nested under the parent with roll‑up status.
 - Summary footer with pass/fail counts, slowest cases, and a hint to rerun focused:
-  - e.g., visor test --config defaults/.visor.tests.yaml --only security-fail-if
+  - e.g., visor test --config defaults/visor.tests.yaml --only security-fail-if
 
 Color, symbols, and truncation rules mirror our main CLI:
 - Green checks for passes, red crosses for failures, yellow for skipped.
@@ -496,7 +497,7 @@ Progress Tracker
 - Milestone 7 — CLI reporters/UX polish — DONE (2025-10-27)
 - Milestone 8 — Validation and helpful errors — DONE (2025-10-27)
 - Milestone 9 — Coverage and perf — DONE (2025-10-27)
-- Milestone 10 — Docs, examples, migration — PENDING
+- Milestone 10 — Docs, examples, migration — DONE (2026-01-28)
 
 Progress Update — 2025-10-29
 - FlowStage refactor: each stage now recomputes prompts/output‑history deltas and execution statistics after any fallback run that executes “missing” expected steps. Coverage tables reflect the final state of the stage.
@@ -513,7 +514,7 @@ Progress Update — 2025-10-28
 
 Milestone 0 — DSL freeze and scaffolding (0.5 week) — DONE 2025-10-27
 - Finalize DSL keys: tests.defaults, fixtures, cases, flow, fixture, mocks, expect.{calls,prompts,outputs,fail,strict_violation}. ✅
-- Rename use_fixture → fixture across examples (done in this RFC and defaults/.visor.tests.yaml). ✅
+- Rename use_fixture → fixture across examples (done in this RFC and defaults/visor.tests.yaml). ✅
 - Create module skeletons: ✅
   - src/test-runner/index.ts (entry + orchestration)
   - src/test-runner/fixture-loader.ts (builtin + overrides)
@@ -526,7 +527,7 @@ Milestone 0 — DSL freeze and scaffolding (0.5 week) — DONE 2025-10-27
 Progress Notes
 - Discovery works against any .visor.tests.yaml (general-purpose, not tied to defaults).
 - Recording Octokit records arbitrary rest ops without hardcoding method lists.
-- defaults/.visor.tests.yaml updated to consistent count grammar and fixed indentation issues.
+- defaults/visor.tests.yaml updated to consistent count grammar and fixed indentation issues.
 
 Milestone 1 — MVP runner and single‑event cases (1 week) — DONE 2025-10-27 (non‑flow)
 - CLI: add visor test [--config path] [--only name] [--bail] [--list]. ✅
@@ -543,12 +544,12 @@ Notes
 
 Verification
 - Build CLI + SDK: npm run build — success.
-- Discovery: visor test --config defaults/.visor.tests.yaml --list — lists suite and cases.
+- Discovery: visor test --config defaults/visor.tests.yaml --list — lists suite and cases.
 - Run single cases:
-  - visor test --config defaults/.visor.tests.yaml --only label-flow — PASS
-  - visor test --config defaults/.visor.tests.yaml --only issue-triage — PASS
-  - visor test --config defaults/.visor.tests.yaml --only security-fail-if — PASS
-  - visor test --config defaults/.visor.tests.yaml --only strict-mode-example — PASS
+  - visor test --config defaults/visor.tests.yaml --only label-flow — PASS
+  - visor test --config defaults/visor.tests.yaml --only issue-triage — PASS
+  - visor test --config defaults/visor.tests.yaml --only security-fail-if — PASS
+  - visor test --config defaults/visor.tests.yaml --only strict-mode-example — PASS
 - Behavior observed:
   - Strict mode enforced (steps executed but not asserted would fail). 
   - GitHub ops recorded by default with dynamic recorder, no network calls.
@@ -606,7 +607,7 @@ Milestone 8 — Validation and helpful errors (0.5 week) — DONE 2025-10-27
 Usage:
 
 ```
-visor test --validate --config defaults/.visor.tests.yaml
+visor test --validate --config defaults/visor.tests.yaml
 ```
 
 Example error output:
@@ -626,12 +627,12 @@ Milestone 9 — Coverage and perf (0.5 week) — DONE 2025-10-27
 Usage examples:
 
 ```
-visor test --config defaults/.visor.tests.yaml --max-parallel 4
-visor test --config defaults/.visor.tests.yaml --prompt-max-chars 16000
+visor test --config defaults/visor.tests.yaml --max-parallel 4
+visor test --config defaults/visor.tests.yaml --prompt-max-chars 16000
 ```
 
-Milestone 10 — Docs, examples, and migration (0.5 week) — IN PROGRESS 2025-10-31
-- Update README to link the RFC and defaults/.visor.tests.yaml.
+Milestone 10 — Docs, examples, and migration (0.5 week) — DONE 2026-01-28
+- Update README to link the RFC and defaults/visor.tests.yaml.
 - Document built-in fixtures catalog and examples.
 - Migration note: how to move from embedded tests and from `returns` to new mocks.
 - Document `depends_on` ANY‑OF (pipe) groups with examples (done).
@@ -650,7 +651,7 @@ Success Metrics
 
 ## Compatibility & Migration
 
-- Tests moved from `defaults/.visor.yaml` into `defaults/.visor.tests.yaml` with `extends: ".visor.yaml"`.
+- Tests moved from `defaults/.visor.yaml` into `defaults/visor.tests.yaml` with `extends: ".visor.yaml"`.
 - Old `mocks.*.returns` is replaced by direct values (object/array/string).
 - You no longer need `run: steps` in tests; cases are integration‑driven by `event + fixture`.
 - `no_other_calls` is unnecessary with strict mode; it’s implied and enforced.
@@ -671,7 +672,7 @@ Success Metrics
 
 ## Appendix: Example Suite
 
-See `defaults/.visor.tests.yaml` in the repo for a complete, multi‑event example covering:
+See `defaults/visor.tests.yaml` in the repo for a complete, multi‑event example covering:
 - PR opened → overview + labels
 - Standard PR comment → no action
 - `/visor` comment → reply

package/dist/docs/testing/assertions.md

@@ -1,10 +1,15 @@
 # Writing Assertions
 
-Assertions live under `expect:` and cover three surfaces:
+Assertions live under `expect:` and cover several surfaces:
 
-- `calls`: step counts and provider effects (GitHub ops)
+- `calls`: step counts and provider effects (GitHub/Slack ops)
 - `prompts`: final AI prompts (post templating/context)
 - `outputs`: step outputs with history and selectors
+- `workflow_output`: workflow-level outputs (for workflow testing)
+- `no_calls`: assert that specific steps or provider ops were NOT called
+- `fail`: assert that the case failed with a specific message
+- `strict_violation`: assert strict mode failure for a missing expect on a step
+- `use`: reference reusable macros defined in `tests.defaults.macros`
 
 ## Calls
 
@@ -18,10 +23,22 @@ expect:
       at_least: 1
       args:
         contains: [feature, "review/effort:2"]
+    - provider: slack
+      op: chat.postMessage
+      at_least: 1
+      args:
+        contains: ["Review complete"]
 ```
 
 Counts are consistent everywhere: `exactly`, `at_least`, `at_most`.
 
+Supported providers:
+- `github`: GitHub API operations (e.g., `labels.add`, `issues.createComment`, `pulls.createReview`, `checks.create`)
+- `slack`: Slack API operations (e.g., `chat.postMessage`)
+
+The `args` field supports:
+- `contains`: array of values that must be present (for labels) or substrings (for Slack text)
+
 ## Prompts
 
 ```yaml
@@ -44,7 +61,7 @@ expect:
 - `index`: `first` | `last` | N (default: last)
 - `where`: selector to choose a prompt from history using `contains`/`not_contains`/`matches` before applying the assertion
 
-Tip: enable `--prompt-max-chars` or `tests.defaults.prompt_max_chars` to cap stored prompt size for large diffs.
+Tip: Enable `--prompt-max-chars` CLI flag or `tests.defaults.prompt_max_chars` config setting to cap stored prompt size for large diffs.
 
 ## Outputs
 
@@ -72,7 +89,31 @@ Supported comparators:
 - `matches` (regex)
 - `contains_unordered` (array membership ignoring order)
 
-## Strict mode and “no calls”
+## Workflow Outputs
+
+For workflow testing, use `workflow_output` to assert on workflow-level outputs (defined in the workflow's `outputs:` section):
+
+```yaml
+expect:
+  workflow_output:
+    - path: summary
+      contains: "Review completed"
+    - path: issues_found
+      equals: 3
+    - path: categories
+      contains_unordered: ["security", "performance"]
+```
+
+Supported comparators for workflow outputs:
+- `equals` (primitive)
+- `equalsDeep` (structural)
+- `matches` (regex)
+- `contains` (substring check, can be string or array)
+- `not_contains` (forbidden substrings)
+- `contains_unordered` (array membership ignoring order)
+- `where` (selector with `path` + `equals`/`matches`)
+
+## Strict mode and "no calls"
 
 Strict mode (default) fails any executed step without a corresponding `expect.calls` entry. You can also assert absence explicitly:
 
@@ -81,5 +122,52 @@ expect:
   no_calls:
     - provider: github
       op: issues.createComment
+    - provider: slack
+      op: chat.postMessage
     - step: extract-facts
 ```
+
+## Failure Assertions
+
+Assert that a test case fails with a specific error message:
+
+```yaml
+expect:
+  fail:
+    message_contains: "validation failed"
+```
+
+Assert that strict mode caught an unexpected step execution:
+
+```yaml
+expect:
+  strict_violation:
+    for_step: unexpected-step
+    message_contains: "Step executed without expect"
+```
+
+## Reusable Macros
+
+Define reusable assertion blocks in `tests.defaults.macros` and reference them with `use`:
+
+```yaml
+tests:
+  defaults:
+    macros:
+      basic-github-check:
+        calls:
+          - provider: github
+            op: checks.create
+            at_least: 1
+
+  cases:
+    - name: my-test
+      event: pr_opened
+      expect:
+        use: [basic-github-check]
+        calls:
+          - step: overview
+            exactly: 1
+```
+
+Macros are merged with inline expectations, allowing you to compose reusable assertion patterns.

package/dist/docs/testing/ci.md

@@ -1,6 +1,8 @@
 # CI Integration for Tests
 
-Run your in‑YAML integration tests in CI using the Visor CLI. Below is a GitHub Actions example. Adapt for other CIs similarly.
+Run your in-YAML integration tests in CI using the Visor CLI. Below is a GitHub Actions example. Adapt for other CIs similarly.
+
+## Basic GitHub Actions Example
 
 ```yaml
 name: Visor Tests
@@ -14,13 +16,13 @@ jobs:
       - uses: actions/setup-node@v4
         with: { node-version: '20' }
       - run: npm ci
-      - run: npm run build --ignore-scripts
+      - run: npm run build
 
-      - name: Run integration tests (defaults)
+      - name: Run integration tests
         run: |
           mkdir -p output
-          node ./dist/index.js test \
-            --config defaults/.visor.tests.yaml \
+          npx visor test \
+            --config defaults/visor.tests.yaml \
             --json output/visor-tests.json \
             --report junit:output/visor-tests.xml \
             --summary md:output/visor-tests.md
@@ -36,9 +38,56 @@ jobs:
             output/visor-tests.md
 ```
 
-Tips
+## Multi-Suite Discovery
+
+When you have multiple test files, Visor can discover and run them all:
+
+```yaml
+- name: Run all test suites
+  run: |
+    mkdir -p output
+    npx visor test tests/ \
+      --max-suites 4 \
+      --max-parallel 2 \
+      --json output/visor-tests.json \
+      --report junit:output/visor-tests.xml
+```
+
+The test runner automatically discovers:
+- Files ending with `.tests.yaml` or `.tests.yml`
+- YAML files containing a top-level `tests:` key with a `cases` array
+
+## Validation-Only Step
+
+Add a fast validation step before running tests to catch YAML syntax errors early:
+
+```yaml
+- name: Validate test files
+  run: npx visor test --validate --config defaults/visor.tests.yaml
+```
+
+## Environment Variables
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `VISOR_DEBUG` | `false` | Enable debug logging |
+| `VISOR_TEST_PROMPT_MAX_CHARS` | `4000` (CI) / `8000` | Truncate captured prompts |
+| `VISOR_TEST_HISTORY_LIMIT` | `200` (CI) / `500` | Limit output history entries |
+| `CI` | - | Automatically detected; adjusts defaults |
+
+## Tips
+
 - Keep `ai_provider: mock` in `tests.defaults` for fast, deterministic runs.
-- Set `--max-parallel` to speed up large suites (flows still run sequentially per case).
+- Set `--max-parallel` to speed up case execution within a suite.
+- Set `--max-suites` to run multiple test files in parallel.
 - Use `--bail` for faster feedback on PRs; run full suite on main.
 - Collect artifacts so you can inspect failures without re-running.
+- Use `--validate` in a separate step for faster feedback on syntax errors.
+
+## See Also
+
+- [CLI Reference](cli.md) - Full list of test command flags
+- [Getting Started](getting-started.md) - Writing your first tests
+- [Fixtures and Mocks](fixtures-and-mocks.md) - Mock AI providers for CI
+- [Troubleshooting](troubleshooting.md) - Common CI issues