@probelabs/visor 0.1.124 → 0.1.126

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (195) hide show
  1. package/dist/config.d.ts.map +1 -1
  2. package/dist/docs/DEPLOYMENT.md +117 -11
  3. package/dist/docs/GITHUB_CHECKS.md +18 -4
  4. package/dist/docs/NPM_USAGE.md +112 -39
  5. package/dist/docs/action-reference.md +63 -9
  6. package/dist/docs/advanced-ai.md +58 -51
  7. package/dist/docs/ai-configuration.md +99 -11
  8. package/dist/docs/ai-custom-tools-usage.md +70 -33
  9. package/dist/docs/ai-custom-tools.md +50 -27
  10. package/dist/docs/architecture.md +1232 -0
  11. package/dist/docs/bot-transports-rfc.md +13 -3
  12. package/dist/docs/ci-cli-mode.md +116 -8
  13. package/dist/docs/claude-code.md +111 -41
  14. package/dist/docs/command-provider.md +37 -15
  15. package/dist/docs/commands.md +252 -6
  16. package/dist/docs/configuration.md +138 -4
  17. package/dist/docs/contributing.md +737 -0
  18. package/dist/docs/custom-tools.md +39 -8
  19. package/dist/docs/dashboards/README.md +33 -19
  20. package/dist/docs/debug-visualizer-progress.md +14 -13
  21. package/dist/docs/debug-visualizer-rfc.md +14 -13
  22. package/dist/docs/debug-visualizer.md +30 -5
  23. package/dist/docs/debugging.md +73 -8
  24. package/dist/docs/default-output-schema.md +24 -20
  25. package/dist/docs/dependencies.md +75 -21
  26. package/dist/docs/dev-playbook.md +85 -9
  27. package/dist/docs/engine-pause-resume-rfc.md +11 -11
  28. package/dist/docs/engine-state-machine-plan.md +10 -3
  29. package/dist/docs/event-driven-github-integration-rfc.md +20 -11
  30. package/dist/docs/event-triggers.md +95 -6
  31. package/dist/docs/execution-statistics-rfc.md +16 -4
  32. package/dist/docs/fact-validator-gap-analysis.md +12 -1
  33. package/dist/docs/fact-validator-implementation-plan.md +19 -11
  34. package/dist/docs/fail-if.md +116 -11
  35. package/dist/docs/failure-conditions-implementation.md +40 -6
  36. package/dist/docs/failure-conditions-schema.md +243 -87
  37. package/dist/docs/failure-routing-rfc.md +43 -18
  38. package/dist/docs/failure-routing.md +80 -23
  39. package/dist/docs/faq.md +836 -0
  40. package/dist/docs/foreach-dependency-propagation.md +32 -15
  41. package/dist/docs/github-ops.md +6 -5
  42. package/dist/docs/glossary.md +322 -0
  43. package/dist/docs/goto-forward-run-plan.md +23 -10
  44. package/dist/docs/guides/criticality-modes.md +15 -13
  45. package/dist/docs/guides/fault-management-and-contracts.md +8 -5
  46. package/dist/docs/guides/workflow-style-guide.md +17 -8
  47. package/dist/docs/http.md +102 -3
  48. package/dist/docs/human-input-provider.md +20 -36
  49. package/dist/docs/index.md +206 -0
  50. package/dist/docs/lifecycle-hooks.md +322 -2
  51. package/dist/docs/limits.md +20 -5
  52. package/dist/docs/liquid-templates.md +86 -14
  53. package/dist/docs/loop-routing-refactor.md +4 -2
  54. package/dist/docs/mcp-provider.md +53 -19
  55. package/dist/docs/mcp.md +27 -1
  56. package/dist/docs/memory.md +7 -2
  57. package/dist/docs/migration.md +596 -0
  58. package/dist/docs/observability.md +227 -6
  59. package/dist/docs/output-formats.md +388 -9
  60. package/dist/docs/output-history.md +36 -6
  61. package/dist/docs/performance.md +510 -4
  62. package/dist/docs/pluggable.md +95 -4
  63. package/dist/docs/proposals/snapshot-scope-execution.md +6 -5
  64. package/dist/docs/providers/git-checkout.md +16 -14
  65. package/dist/docs/providers/noop.md +696 -0
  66. package/dist/docs/recipes.md +8 -9
  67. package/dist/docs/rfc/git-checkout-step.md +3 -1
  68. package/dist/docs/rfc/on_init-hook.md +18 -5
  69. package/dist/docs/rfc/workspace-isolation.md +16 -0
  70. package/dist/docs/roadmap/criticality-implementation-tasks.md +27 -27
  71. package/dist/docs/router-patterns.md +155 -43
  72. package/dist/docs/schema-templates.md +51 -15
  73. package/dist/docs/script.md +162 -13
  74. package/dist/docs/sdk.md +46 -12
  75. package/dist/docs/security.md +464 -5
  76. package/dist/docs/slack-integration.md +481 -0
  77. package/dist/docs/tag-filtering.md +60 -20
  78. package/dist/docs/telemetry-setup.md +157 -46
  79. package/dist/docs/test-framework-rfc.md +37 -36
  80. package/dist/docs/testing/assertions.md +92 -4
  81. package/dist/docs/testing/ci.md +56 -7
  82. package/dist/docs/testing/cli.md +57 -15
  83. package/dist/docs/testing/cookbook.md +53 -20
  84. package/dist/docs/testing/dsl-reference.md +110 -9
  85. package/dist/docs/testing/fixtures-and-mocks.md +28 -3
  86. package/dist/docs/testing/flows.md +59 -4
  87. package/dist/docs/testing/getting-started.md +14 -13
  88. package/dist/docs/testing/troubleshooting.md +39 -2
  89. package/dist/docs/timeouts.md +174 -18
  90. package/dist/docs/troubleshooting.md +176 -6
  91. package/dist/docs/workflow-creation-guide.md +101 -3
  92. package/dist/docs/workflows.md +138 -41
  93. package/dist/examples/README.md +169 -4
  94. package/dist/examples/ai-custom-tools-simple.yaml +2 -3
  95. package/dist/examples/cron-webhook-config.yaml +15 -0
  96. package/dist/examples/forEach-example.yaml +6 -0
  97. package/dist/examples/git-checkout-basic.yaml +4 -0
  98. package/dist/examples/git-checkout-compare.yaml +6 -0
  99. package/dist/examples/git-checkout-cross-repo.yaml +7 -0
  100. package/dist/examples/http-integration-config.yaml +30 -0
  101. package/dist/examples/https-server-config.yaml +15 -0
  102. package/dist/examples/mcp-provider-example.yaml +10 -10
  103. package/dist/examples/transform-example.yaml +3 -0
  104. package/dist/examples/webhook-pipeline-config.yaml +18 -0
  105. package/dist/examples/workflows/workflow-composition-example.yaml +4 -0
  106. package/dist/frontends/slack-frontend.d.ts +2 -0
  107. package/dist/frontends/slack-frontend.d.ts.map +1 -1
  108. package/dist/generated/config-schema.d.ts +11 -7
  109. package/dist/generated/config-schema.d.ts.map +1 -1
  110. package/dist/generated/config-schema.json +11 -7
  111. package/dist/index.js +3127 -974
  112. package/dist/output/traces/{run-2026-01-28T16-15-24-569Z.ndjson → run-2026-01-31T16-37-22-321Z.ndjson} +84 -84
  113. package/dist/output/traces/{run-2026-01-28T16-16-09-757Z.ndjson → run-2026-01-31T16-38-06-031Z.ndjson} +1013 -1013
  114. package/dist/providers/ai-check-provider.d.ts +9 -2
  115. package/dist/providers/ai-check-provider.d.ts.map +1 -1
  116. package/dist/providers/command-check-provider.d.ts.map +1 -1
  117. package/dist/providers/mcp-custom-sse-server.d.ts +17 -1
  118. package/dist/providers/mcp-custom-sse-server.d.ts.map +1 -1
  119. package/dist/providers/workflow-check-provider.d.ts.map +1 -1
  120. package/dist/providers/workflow-tool-executor.d.ts +68 -0
  121. package/dist/providers/workflow-tool-executor.d.ts.map +1 -0
  122. package/dist/sdk/{check-provider-registry-AQ3JETBG.mjs → check-provider-registry-3KI5RKXT.mjs} +6 -5
  123. package/dist/sdk/check-provider-registry-IYILYY35.mjs +28 -0
  124. package/dist/sdk/chunk-2CPMMNIX.mjs +1459 -0
  125. package/dist/sdk/chunk-2CPMMNIX.mjs.map +1 -0
  126. package/dist/sdk/chunk-5LI6T4O3.mjs +3600 -0
  127. package/dist/sdk/chunk-5LI6T4O3.mjs.map +1 -0
  128. package/dist/sdk/{chunk-YLQ4UN62.mjs → chunk-A4PGHURG.mjs} +6838 -6257
  129. package/dist/sdk/chunk-A4PGHURG.mjs.map +1 -0
  130. package/dist/sdk/chunk-EXFGO4FX.mjs +147 -0
  131. package/dist/sdk/chunk-EXFGO4FX.mjs.map +1 -0
  132. package/dist/sdk/chunk-PJ7K5UFC.mjs +17732 -0
  133. package/dist/sdk/chunk-PJ7K5UFC.mjs.map +1 -0
  134. package/dist/sdk/{chunk-BHZ4CKUS.mjs → chunk-PXFIALUH.mjs} +77 -8
  135. package/dist/sdk/chunk-PXFIALUH.mjs.map +1 -0
  136. package/dist/sdk/{chunk-PVITVJ6J.mjs → chunk-RTKJXNZS.mjs} +32 -9
  137. package/dist/sdk/chunk-RTKJXNZS.mjs.map +1 -0
  138. package/dist/sdk/chunk-VW2GBXQT.mjs +606 -0
  139. package/dist/sdk/chunk-VW2GBXQT.mjs.map +1 -0
  140. package/dist/sdk/{config-RQQPMLRD.mjs → config-5AUYQFHE.mjs} +2 -2
  141. package/dist/sdk/config-6CUVEH7H.mjs +16 -0
  142. package/dist/sdk/config-6CUVEH7H.mjs.map +1 -0
  143. package/dist/sdk/{github-frontend-6Q4BISZX.mjs → github-frontend-BZ4N3BFZ.mjs} +7 -3
  144. package/dist/sdk/github-frontend-BZ4N3BFZ.mjs.map +1 -0
  145. package/dist/sdk/host-4MT3EW2I.mjs +52 -0
  146. package/dist/sdk/{host-P5NQICP7.mjs → host-NYWXLIFC.mjs} +2 -2
  147. package/dist/sdk/host-NYWXLIFC.mjs.map +1 -0
  148. package/dist/sdk/{routing-DEY2AIXM.mjs → routing-6R42GXUO.mjs} +2 -2
  149. package/dist/sdk/routing-6R42GXUO.mjs.map +1 -0
  150. package/dist/sdk/routing-7FXPULTO.mjs +24 -0
  151. package/dist/sdk/routing-7FXPULTO.mjs.map +1 -0
  152. package/dist/sdk/sdk.d.mts +3 -1
  153. package/dist/sdk/sdk.d.ts +3 -1
  154. package/dist/sdk/sdk.js +12163 -11204
  155. package/dist/sdk/sdk.js.map +1 -1
  156. package/dist/sdk/sdk.mjs +14 -10
  157. package/dist/sdk/sdk.mjs.map +1 -1
  158. package/dist/sdk/slack-frontend-JUT3TYVC.mjs +821 -0
  159. package/dist/sdk/slack-frontend-JUT3TYVC.mjs.map +1 -0
  160. package/dist/sdk/workflow-check-provider-H3CUOLUD.mjs +28 -0
  161. package/dist/sdk/workflow-check-provider-H3CUOLUD.mjs.map +1 -0
  162. package/dist/sdk/workflow-check-provider-YUNNF4KC.mjs +28 -0
  163. package/dist/sdk/workflow-check-provider-YUNNF4KC.mjs.map +1 -0
  164. package/dist/sdk/workflow-registry-KFWSDSLM.mjs +12 -0
  165. package/dist/sdk/workflow-registry-KFWSDSLM.mjs.map +1 -0
  166. package/dist/slack/socket-runner.d.ts +2 -0
  167. package/dist/slack/socket-runner.d.ts.map +1 -1
  168. package/dist/state-machine/context/workflow-inputs.d.ts +20 -0
  169. package/dist/state-machine/context/workflow-inputs.d.ts.map +1 -0
  170. package/dist/state-machine/dispatch/execution-invoker.d.ts.map +1 -1
  171. package/dist/state-machine/dispatch/foreach-processor.d.ts.map +1 -1
  172. package/dist/state-machine/dispatch/stats-manager.d.ts.map +1 -1
  173. package/dist/state-machine/states/level-dispatch.d.ts.map +1 -1
  174. package/dist/state-machine/states/routing.d.ts +2 -1
  175. package/dist/state-machine/states/routing.d.ts.map +1 -1
  176. package/dist/traces/{run-2026-01-28T16-15-24-569Z.ndjson → run-2026-01-31T16-37-22-321Z.ndjson} +84 -84
  177. package/dist/traces/{run-2026-01-28T16-16-09-757Z.ndjson → run-2026-01-31T16-38-06-031Z.ndjson} +1013 -1013
  178. package/dist/types/config.d.ts +3 -1
  179. package/dist/types/config.d.ts.map +1 -1
  180. package/dist/utils/human-id.d.ts +12 -0
  181. package/dist/utils/human-id.d.ts.map +1 -0
  182. package/dist/utils/worktree-manager.d.ts +3 -0
  183. package/dist/utils/worktree-manager.d.ts.map +1 -1
  184. package/dist/workflow-executor.d.ts.map +1 -1
  185. package/dist/workflow-registry.d.ts +1 -0
  186. package/dist/workflow-registry.d.ts.map +1 -1
  187. package/package.json +2 -2
  188. package/dist/sdk/chunk-BHZ4CKUS.mjs.map +0 -1
  189. package/dist/sdk/chunk-PVITVJ6J.mjs.map +0 -1
  190. package/dist/sdk/chunk-YLQ4UN62.mjs.map +0 -1
  191. package/dist/sdk/github-frontend-6Q4BISZX.mjs.map +0 -1
  192. /package/dist/sdk/{check-provider-registry-AQ3JETBG.mjs.map → check-provider-registry-3KI5RKXT.mjs.map} +0 -0
  193. /package/dist/sdk/{config-RQQPMLRD.mjs.map → check-provider-registry-IYILYY35.mjs.map} +0 -0
  194. /package/dist/sdk/{routing-DEY2AIXM.mjs.map → config-5AUYQFHE.mjs.map} +0 -0
  195. /package/dist/sdk/{host-P5NQICP7.mjs.map → host-4MT3EW2I.mjs.map} +0 -0
@@ -5,37 +5,79 @@ Run integration tests for your Visor config using the built-in `test` subcommand
5
5
  ## Commands
6
6
 
7
7
  - Discover tests file and list cases
8
- - `visor test --list [--config defaults/.visor.tests.yaml]`
8
+ - `visor test --list [--config defaults/visor.tests.yaml]`
9
9
  - Run cases
10
- - `visor test [--config defaults/.visor.tests.yaml] [--only <substring>] [--bail]`
10
+ - `visor test [--config defaults/visor.tests.yaml] [--only <substring>] [--bail]`
11
11
  - Validate tests YAML without running
12
- - `visor test --validate [--config defaults/.visor.tests.yaml]`
12
+ - `visor test --validate [--config defaults/visor.tests.yaml]`
13
+
14
+ ## Auto-Discovery
15
+
16
+ When no `--config` is provided, the test runner searches for test files in the following order:
17
+
18
+ 1. `defaults/visor.tests.yaml` or `defaults/visor.tests.yml`
19
+ 2. `.visor.tests.yaml` or `.visor.tests.yml` in the project root
20
+
21
+ You can also pass a directory or glob pattern as a positional argument to discover multiple test suites:
22
+
23
+ ```bash
24
+ visor test defaults/ # Run all suites in defaults/
25
+ visor test "**/*.tests.yaml" # Run all matching suites
26
+ ```
13
27
 
14
28
  ## Flags
15
29
 
16
- - `--config <path>`: Path to `.visor.tests.yaml` (auto-discovers `.visor.tests.yaml` or `defaults/.visor.tests.yaml`).
30
+ ### Core Flags
31
+
32
+ - `--config <path>`: Path to `.visor.tests.yaml` (auto-discovers if not specified).
17
33
  - `--only <filter>`: Run cases whose `name` contains the substring (case-insensitive).
18
- - `--only <filter>`: Run cases whose `name` contains the substring (case-insensitive).
19
- - Stage filter: append `#<stage>` to run only a flow stage.
20
- - Examples: `--only pr-review-e2e-flow#facts-invalid`, `--only pr-review-e2e-flow#3` (1‑based index)
34
+ - Stage filter: append `#<stage>` to run only a flow stage.
35
+ - Examples: `--only pr-review-e2e-flow#facts-invalid`, `--only pr-review-e2e-flow#3` (1-based index)
21
36
  - `--bail`: Stop on first failure.
22
- - `--json <path|->`: Write a minimal JSON summary.
23
- - `--report junit:<path>`: Write a minimal JUnit XML.
24
- - `--summary md:<path>`: Write a minimal Markdown summary.
25
- - `--progress compact|detailed`: Progress verbosity (parsing supported; detailed view evolves over time).
26
- - `--max-parallel <N>`: Run up to N cases concurrently.
37
+ - `--list`: List discovered test cases without running them.
38
+ - `--validate`: Validate tests YAML syntax without running.
39
+
40
+ ### Parallelism
41
+
42
+ - `--max-parallel <N>`: Run up to N test cases concurrently within a suite (default: 1).
43
+ - `--max-suites <N>`: Run up to N test suites concurrently when discovering multiple files (default: number of CPUs).
44
+
45
+ ### Output & Reporting
46
+
47
+ - `--json <path|->`: Write a minimal JSON summary (`-` for stdout).
48
+ - `--report junit:<path>`: Write a JUnit XML report.
49
+ - `--summary md:<path>`: Write a Markdown summary.
50
+ - `--progress compact|detailed`: Progress verbosity (default: compact).
27
51
  - `--prompt-max-chars <N>`: Truncate captured prompt text to N characters.
28
52
 
53
+ ### Debugging
54
+
55
+ - `--debug`: Enable debug mode for verbose output (equivalent to `VISOR_DEBUG=true`).
56
+ - `--no-mocks`: Run tests without mock injection. Real providers execute and outputs are printed as suggested mocks.
57
+
29
58
  ## Output
30
59
 
31
60
  - Per-case PASS/FAIL lines
32
61
  - Coverage table (expected vs actual step runs)
33
- - Summary totals
62
+ - Summary totals (Jest-style format)
34
63
 
35
64
  ## Tips
36
65
 
37
66
  - Use `--validate` when iterating on tests to catch typos early.
38
67
  - Keep `strict: true` in `tests.defaults` to surface missing `expect` quickly.
39
68
  - For large suites, increase `--max-parallel` to improve throughput.
40
- - Enable debug logs by setting `VISOR_DEBUG=true`.
41
- - Example: `VISOR_DEBUG=true visor test --config defaults/.visor.tests.yaml --only pr-review-e2e-flow#facts-invalid`
69
+ - Use `--no-mocks` to capture real provider outputs, then copy the suggested mocks into your test case.
70
+ - Enable debug logs with `--debug` or `VISOR_DEBUG=true`:
71
+ ```bash
72
+ visor test --debug --config defaults/visor.tests.yaml --only pr-review-e2e-flow#facts-invalid
73
+ ```
74
+
75
+ ## Related Documentation
76
+
77
+ - [Getting Started](./getting-started.md) - Introduction to the test framework
78
+ - [DSL Reference](./dsl-reference.md) - Complete test YAML schema
79
+ - [Assertions](./assertions.md) - Available assertion types
80
+ - [Fixtures and Mocks](./fixtures-and-mocks.md) - Managing test data
81
+ - [Flows](./flows.md) - Multi-stage test flows
82
+ - [CI Integration](./ci.md) - Running tests in CI pipelines
83
+ - [Troubleshooting](./troubleshooting.md) - Common issues and solutions
@@ -86,70 +86,93 @@ Copy‑pasteable recipes for common scenarios.
86
86
  fixture: gh.issue_comment.visor_help
87
87
  env: { ENABLE_FACT_VALIDATION: "true" }
88
88
  mocks:
89
- comment-assistant: { text: "We rely on defaults/.visor.yaml line 11 for max_parallelism=4.", intent: comment_reply }
89
+ comment-assistant: { text: "We rely on defaults/visor.yaml line 11 for max_parallelism=4.", intent: comment_reply }
90
90
  extract-facts:
91
91
  - { id: f1, category: Configuration, claim: "max_parallelism defaults to 4", verifiable: true }
92
92
  validate-fact[]:
93
- - { fact_id: f1, is_valid: true, confidence: high, evidence: "defaults/.visor.yaml:11" }
93
+ - { fact_id: f1, claim: "max_parallelism defaults to 4", is_valid: true, confidence: high, evidence: "defaults/visor.yaml:11" }
94
94
  expect:
95
95
  calls:
96
+ - step: comment-assistant
97
+ exactly: 1
96
98
  - step: extract-facts
97
99
  exactly: 1
98
100
  - step: validate-fact
99
101
  at_least: 1
100
- - step: aggregate-validations
101
- exactly: 1
102
102
  ```
103
103
 
104
104
  ## 6) Facts invalid (correction reply)
105
105
 
106
+ When a fact is invalid, the correction flow triggers a re-run. Due to goto forward-running dependents, `extract-facts` and `validate-fact` also run again.
107
+
106
108
  ```yaml
107
109
  - name: facts-invalid
108
110
  event: issue_comment
109
111
  fixture: gh.issue_comment.visor_help
110
112
  env: { ENABLE_FACT_VALIDATION: "true" }
113
+ routing:
114
+ max_loops: 1
111
115
  mocks:
112
- comment-assistant: { text: "We rely on defaults/.visor.yaml line 11 for max_parallelism=4.", intent: comment_reply }
116
+ comment-assistant: { text: "We rely on defaults/visor.yaml line 11 for max_parallelism=4.", intent: comment_reply }
113
117
  extract-facts:
114
118
  - { id: f1, category: Configuration, claim: "max_parallelism defaults to 4", verifiable: true }
115
119
  validate-fact[]:
116
- - { fact_id: f1, is_valid: false, confidence: high, evidence: "defaults/.visor.yaml:11", correction: "max_parallelism defaults to 3" }
120
+ - { fact_id: f1, claim: "max_parallelism defaults to 4", is_valid: false, confidence: high, evidence: "defaults/visor.yaml:11 does not set 4", correction: "max_parallelism defaults to 3" }
117
121
  expect:
118
122
  calls:
119
123
  - step: comment-assistant
120
124
  exactly: 2
121
- - step: aggregate-validations
125
+ - step: extract-facts
126
+ exactly: 2
127
+ - step: validate-fact
128
+ exactly: 2
129
+ - step: aggregate
122
130
  exactly: 1
123
- prompts:
124
- - step: comment-assistant
125
- index: last
126
- contains: ["<previous_response>", "Correction: max_parallelism defaults to 3"]
131
+ outputs:
132
+ - step: validate-fact
133
+ where: { path: fact_id, equals: f1 }
134
+ path: correction
135
+ equals: "max_parallelism defaults to 3"
127
136
  ```
128
137
 
129
138
  ## 7) Two facts (one invalid)
130
139
 
140
+ With two facts extracted where only one is invalid, the correction pass runs for the invalid fact. Due to goto forward-running dependents, `extract-facts` and `validate-fact` run again on retry.
141
+
131
142
  ```yaml
132
143
  - name: facts-two-items
133
144
  event: issue_comment
134
145
  fixture: gh.issue_comment.visor_help
135
146
  env: { ENABLE_FACT_VALIDATION: "true" }
147
+ routing:
148
+ max_loops: 1
136
149
  mocks:
137
- comment-assistant: { text: "We rely on defaults/.visor.yaml for concurrency defaults.", intent: comment_reply }
150
+ comment-assistant: { text: "We rely on defaults/visor.yaml for concurrency defaults.", intent: comment_reply }
138
151
  extract-facts:
139
152
  - { id: f1, category: Configuration, claim: "max_parallelism defaults to 4", verifiable: true }
140
153
  - { id: f2, category: Feature, claim: "Fast mode is enabled by default", verifiable: true }
141
154
  validate-fact[]:
142
- - { fact_id: f1, is_valid: false, confidence: high, evidence: "defaults/.visor.yaml:11", correction: "max_parallelism defaults to 3" }
143
- - { fact_id: f2, is_valid: true, confidence: high, evidence: "src/config.ts:FAST_MODE=true" }
155
+ - { fact_id: f1, claim: "max_parallelism defaults to 4", is_valid: false, confidence: high, evidence: "defaults/visor.yaml:11", correction: "max_parallelism defaults to 3" }
156
+ - { fact_id: f2, claim: "Fast mode is enabled by default", is_valid: true, confidence: high, evidence: "src/config.ts:FAST_MODE=true" }
144
157
  expect:
145
158
  calls:
146
- - step: validate-fact
147
- exactly: 2
148
- prompts:
149
159
  - step: comment-assistant
150
- index: last
151
- contains: ["max_parallelism defaults to 4", "Correction: max_parallelism defaults to 3"]
152
- not_contains: ["Fast mode is enabled by default"]
160
+ exactly: 2
161
+ - step: extract-facts
162
+ exactly: 2
163
+ - step: validate-fact
164
+ exactly: 4
165
+ - step: aggregate
166
+ exactly: 1
167
+ outputs:
168
+ - step: validate-fact
169
+ where: { path: fact_id, equals: f1 }
170
+ path: is_valid
171
+ equals: false
172
+ - step: validate-fact
173
+ where: { path: fact_id, equals: f2 }
174
+ path: is_valid
175
+ equals: true
153
176
  ```
154
177
 
155
178
  ## 8) GitHub negative mode
@@ -170,3 +193,13 @@ Copy‑pasteable recipes for common scenarios.
170
193
  message_contains: "github/op_failed"
171
194
  ```
172
195
 
196
+ ## Related Documentation
197
+
198
+ - [Getting Started](./getting-started.md) - Introduction to the test framework
199
+ - [DSL Reference](./dsl-reference.md) - Complete test YAML schema
200
+ - [Assertions](./assertions.md) - Available assertion types
201
+ - [Fixtures and Mocks](./fixtures-and-mocks.md) - Managing test data
202
+ - [Flows](./flows.md) - Multi-stage test flows
203
+ - [CLI](./cli.md) - Test runner command line options
204
+ - [CI Integration](./ci.md) - Running tests in CI pipelines
205
+ - [Troubleshooting](./troubleshooting.md) - Common issues and solutions
@@ -11,9 +11,17 @@ tests:
11
11
  strict: true # default strict mode
12
12
  ai_provider: mock # force AI provider to mock
13
13
  prompt_max_chars: 16000 # truncate captured prompts (optional)
14
+ ai_include_code_context: false # include PR diff/context in AI prompts (default: false)
15
+ fail_on_unexpected_calls: false # fail if unexpected provider calls occur
16
+ frontends: ["github"] # enable specific frontends during tests
14
17
  github_recorder: # optional negative modes
15
18
  error_code: 0 # e.g., 429
16
19
  timeout_ms: 0 # e.g., 1000
20
+ macros: # reusable expect blocks (see Reusable Macros)
21
+ basic-check:
22
+ calls:
23
+ - step: overview
24
+ at_least: 1
17
25
  # Optional: include/exclude checks by tags (same semantics as main CLI)
18
26
  tags: "local,fast" # or [local, fast]
19
27
  exclude_tags: "experimental,slow" # or [experimental, slow]
@@ -24,16 +32,20 @@ tests:
24
32
  - name: <string>
25
33
  description: <markdown>
26
34
  skip: false|true
35
+ ai_include_code_context: false # per-case override
27
36
 
28
37
  # Single-event case
29
38
  event: pr_opened | pr_updated | pr_closed | issue_opened | issue_comment | manual
30
39
  fixture: <builtin|{ builtin, overrides }>
31
40
  env: { <KEY>: <VALUE>, ... }
32
41
  mocks: { <step>: <value>, <step>[]: [<value>...] }
42
+ workflow_input: { <key>: <value>, ... } # inputs for workflow testing
33
43
  expect: <expect-block>
34
44
  strict: true|false # overrides defaults.strict
35
45
  tags: "security,fast" # optional per-case include filter
36
46
  exclude_tags: "slow" # optional per-case exclude filter
47
+ github_recorder: # per-case recorder overrides
48
+ error_code: 429
37
49
 
38
50
  # OR flow case
39
51
  flow:
@@ -42,23 +54,31 @@ tests:
42
54
  fixture: ...
43
55
  env: ...
44
56
  mocks: ... # merged with flow-level mocks
57
+ routing: # per-stage routing overrides
58
+ max_loops: 10
45
59
  expect: <expect-block>
46
60
  strict: true|false # per-stage fallback to case/defaults
47
61
  tags: "security" # optional per-stage include filter
48
62
  exclude_tags: "slow" # optional per-stage exclude filter
63
+ github_recorder: # per-stage recorder overrides
64
+ error_code: 500
49
65
  ```
50
66
 
51
67
  ## Fixtures
52
68
 
53
- - Built-in GitHub fixtures: `gh.pr_open.minimal`, `gh.pr_sync.minimal`, `gh.issue_open.minimal`, `gh.issue_comment.standard`, `gh.issue_comment.visor_help`, `gh.issue_comment.visor_regenerate`.
69
+ - Built-in GitHub fixtures: `gh.pr_open.minimal`, `gh.pr_sync.minimal`, `gh.pr_closed.minimal`, `gh.issue_open.minimal`, `gh.issue_comment.standard`, `gh.issue_comment.visor_help`, `gh.issue_comment.visor_regenerate`.
54
70
  - Use `overrides` to tweak titles, numbers, payload slices.
55
71
 
72
+ See [Fixtures and Mocks](./fixtures-and-mocks.md) for details.
73
+
56
74
  ## Mocks
57
75
 
58
76
  - Keys are step names; for forEach children use `step[]` (e.g., `validate-fact[]`).
59
77
  - AI mocks may be structured JSON if a schema is configured for the step; otherwise use `text` and optional fields used by templates.
60
78
  - Command/HTTP mocks emulate provider shape (`stdout`, `exit_code`, or HTTP body/status headers) and bypass real execution.
61
79
 
80
+ See [Fixtures and Mocks](./fixtures-and-mocks.md) for detailed mock examples.
81
+
62
82
  Inline example (AI with schema + list mocks):
63
83
 
64
84
  ```yaml
@@ -78,13 +98,20 @@ mocks:
78
98
 
79
99
  ```yaml
80
100
  expect:
101
+ use: [macro-name] # reference macros from tests.defaults.macros
102
+
81
103
  calls:
82
- - step: <name> | provider: github + op: <rest.op>
104
+ - step: <name>
105
+ exactly|at_least|at_most: <number>
106
+ - provider: github|slack # provider-level calls
107
+ op: <rest.op> # e.g., labels.add, chat.postMessage
83
108
  exactly|at_least|at_most: <number>
84
- args: { contains: [..], not_contains: [..] } # provider args matching
109
+ args: { contains: [..] } # provider args matching
85
110
 
86
111
  no_calls:
87
- - step: <name> | provider: github + op: <rest.op>
112
+ - step: <name>
113
+ - provider: github|slack
114
+ op: <rest.op>
88
115
 
89
116
  prompts:
90
117
  - step: <name>
@@ -97,7 +124,7 @@ expect:
97
124
 
98
125
  outputs:
99
126
  - step: <name>
100
- index: first|last|<N> # or
127
+ index: first|last|<N>
101
128
  where: { path: <expr>, equals|matches: <v> }
102
129
  path: <expr> # dot/bracket, e.g. tags['review-effort']
103
130
  equals: <primitive>
@@ -105,12 +132,29 @@ expect:
105
132
  matches: <regex>
106
133
  contains_unordered: [..]
107
134
 
135
+ workflow_output: # assert on workflow-level outputs (for workflow testing)
136
+ - path: <output-name> # path into workflow outputs object
137
+ equals: <primitive>
138
+ equalsDeep: <object>
139
+ matches: <regex>
140
+ contains: <string|[..]> # substring check
141
+ not_contains: <string|[..]>
142
+ contains_unordered: [..]
143
+ where: { path: <expr>, equals|matches: <v> }
144
+
108
145
  fail:
109
146
  message_contains: <string> # assert overall case failure message
110
147
 
111
148
  strict_violation: # assert strict failure for a missing expect on a step
112
149
  for_step: <name>
113
150
  message_contains: <string>
151
+ ```
152
+
153
+ **Supported providers for `calls` and `no_calls`:**
154
+ - `github`: GitHub API operations (`labels.add`, `issues.createComment`, `pulls.createReview`, `checks.create`, `checks.update`)
155
+ - `slack`: Slack API operations (`chat.postMessage`)
156
+
157
+ See [Assertions](./assertions.md) for detailed assertion syntax and examples.
114
158
 
115
159
  Inline example (calls + prompts + outputs):
116
160
 
@@ -131,7 +175,6 @@ expect:
131
175
  path: "tags['review-effort']"
132
176
  equals: 2
133
177
  ```
134
- ```
135
178
 
136
179
  Note on dependencies: test execution honors your base config routing, including `depends_on`. You can express ANY‑OF groups using pipe syntax in the base config (e.g., `depends_on: ["issue-assistant|comment-assistant"]`). The runner mixes these with normal ALL‑OF deps.
137
180
 
@@ -152,6 +195,63 @@ Note on dependencies: test execution honors your base config routing, including
152
195
  - Run one case: `visor test --only label-flow`
153
196
  - Run one stage: `visor test --only pr-review-e2e-flow#facts-invalid`
154
197
  - JSON/JUnit/Markdown reporters: `--json`, `--report junit:<path>`, `--summary md:<path>`
198
+
199
+ See [CLI Reference](./cli.md) for all available options.
200
+
201
+ ## Reusable Macros
202
+
203
+ Define reusable assertion blocks in `tests.defaults.macros` and reference them with `use`:
204
+
205
+ ```yaml
206
+ tests:
207
+ defaults:
208
+ macros:
209
+ basic-github-check:
210
+ calls:
211
+ - provider: github
212
+ op: checks.create
213
+ at_least: 1
214
+ overview-ran:
215
+ calls:
216
+ - step: overview
217
+ exactly: 1
218
+
219
+ cases:
220
+ - name: my-test
221
+ event: pr_opened
222
+ expect:
223
+ use: [basic-github-check, overview-ran]
224
+ calls:
225
+ - step: extra-step
226
+ exactly: 1
227
+ ```
228
+
229
+ Macros are merged with inline expectations, allowing you to compose reusable assertion patterns.
230
+
231
+ ## Workflow Testing
232
+
233
+ Test standalone workflows by providing `workflow_input` and asserting on `workflow_output`:
234
+
235
+ ```yaml
236
+ tests:
237
+ cases:
238
+ - name: test-workflow
239
+ event: manual
240
+ workflow_input:
241
+ repo_url: "https://github.com/example/repo"
242
+ branch: "main"
243
+ mocks:
244
+ fetch-data:
245
+ status: 200
246
+ data: { items: [1, 2, 3] }
247
+ expect:
248
+ workflow_output:
249
+ - path: summary
250
+ contains: "completed"
251
+ - path: items_count
252
+ equals: 3
253
+ ```
254
+
155
255
  ## JavaScript in Tests and Routing (run_js, goto_js, value_js, transform_js)
156
256
 
157
257
  ### Tags default semantics in tests
@@ -184,8 +284,9 @@ Tips
184
284
  - Use `Array.prototype.at(-1)` to read the last item. Example: `const last = (outputs_history['validate-fact'] || []).at(-1) || [];`.
185
285
  - For reshaping small maps, `Object.entries` + `Object.fromEntries` is concise and readable.
186
286
 
187
- Example: wavescoped correction gate
188
- ```
287
+ Example: wave-scoped correction gate
288
+
289
+ ```yaml
189
290
  run_js: |
190
291
  const facts = (outputs_history['extract-facts'] || []).at(-1) || [];
191
292
  const ids = facts.map(f => String(f.id || '')).filter(Boolean);
@@ -196,4 +297,4 @@ run_js: |
196
297
  return (event && event.name) === 'issue_opened' ? ['issue-assistant'] : ['comment-assistant'];
197
298
  ```
198
299
 
199
- This evaluates the last `extract-facts` wave, finds the corresponding `validate-fact` results, and schedules a single correction pass when any item is invalid or lowconfidence.
300
+ This evaluates the last `extract-facts` wave, finds the corresponding `validate-fact` results, and schedules a single correction pass when any item is invalid or low-confidence.
@@ -13,6 +13,7 @@ Use via `fixture: gh.<name>` or `fixture: { builtin: gh.<name>, overrides: {...}
13
13
  - `gh.issue_comment.standard` — normal human comment on a PR/issue.
14
14
  - `gh.issue_comment.visor_help` — comment containing `/visor help`.
15
15
  - `gh.issue_comment.visor_regenerate` — `/visor Regenerate reviews`.
16
+ - `gh.issue_comment.edited` — issue_comment edited action.
16
17
 
17
18
  Overrides allow tailored inputs:
18
19
 
@@ -57,7 +58,7 @@ mocks:
57
58
 
58
59
  # AI plain text schema
59
60
  comment-assistant:
60
- text: "Sure, heres how I can help."
61
+ text: "Sure, here's how I can help."
61
62
  intent: comment_reply
62
63
 
63
64
  # Array outputs (e.g., extract-facts)
@@ -68,8 +69,9 @@ mocks:
68
69
  unit-tests:
69
70
  stdout: '{"passed": 128, "failed": 0}'
70
71
  exit_code: 0
72
+ ```
71
73
 
72
- ### Percall list mocks (for forEach children)
74
+ ### Per-call list mocks (for forEach children)
73
75
 
74
76
  When a step runs once per item (e.g., `validate-fact` depends on `extract-facts`), provide a list under the `[]` suffix:
75
77
 
@@ -83,9 +85,32 @@ mocks:
83
85
  - { fact_id: f2, is_valid: true, confidence: high, evidence: "src/config.ts:FAST_MODE=true" }
84
86
  ```
85
87
 
86
- The runner distributes items in order; if the list is shorter than invocations, the last entry is reused. This fits any forEachstyle step you define (naming is up to you).
88
+ The runner distributes items in order; if the list is shorter than invocations, the last entry is reused. This fits any forEach-style step you define (naming is up to you).
89
+
90
+ ### HTTP client mocks
91
+
92
+ For `http-client` provider steps, mocks can specify HTTP response fields:
93
+
94
+ ```yaml
95
+ mocks:
96
+ fetch-api-data:
97
+ status: 200
98
+ body: '{"items": [1, 2, 3]}'
99
+ headers:
100
+ content-type: application/json
87
101
  ```
88
102
 
89
103
  Notes:
90
104
  - No `returns:` key; provide values directly.
91
105
  - For HTTP/Command providers, mocks bypass real execution and are recorded for assertions.
106
+
107
+ ## Related Documentation
108
+
109
+ - [Getting Started](./getting-started.md) - Introduction to the test framework
110
+ - [DSL Reference](./dsl-reference.md) - Complete test YAML schema
111
+ - [Assertions](./assertions.md) - Available assertion types
112
+ - [Flows](./flows.md) - Multi-stage test flows
113
+ - [Cookbook](./cookbook.md) - Copy-pasteable test recipes
114
+ - [CLI](./cli.md) - Test runner command line options
115
+ - [CI Integration](./ci.md) - Running tests in CI pipelines
116
+ - [Troubleshooting](./troubleshooting.md) - Common issues and solutions
@@ -2,7 +2,7 @@
2
2
 
3
3
  > Model realistic user journeys across multiple external events in one case.
4
4
 
5
- A flow case defines a `flow:` array of stages. Each stage has its own `event`, `fixture`, optional `env` and `mocks`, plus `expect`.
5
+ A flow case defines a `flow:` array of stages. Each stage has its own `event`, `fixture`, and optional settings like `env`, `mocks`, `routing`, `tags`, `github_recorder`, plus `expect`.
6
6
 
7
7
  ```yaml
8
8
  - name: pr-review-e2e-flow
@@ -35,8 +35,9 @@ A flow case defines a `flow:` array of stages. Each stage has its own `event`, `
35
35
 
36
36
  ## Stage selection and deltas
37
37
 
38
- - Run a single stage: `--only case#stage` (name substring) or `--only case#N` (1based index).
39
- - Coverage, prompts, outputs, and provider calls are computed per‑stage as deltas from the previous stage.
38
+ - Run a single stage: `--only case#stage` (name substring match, case-insensitive) or `--only case#N` (1-based index).
39
+ - Examples: `--only pr-review-e2e-flow#facts-invalid`, `--only pr-review-e2e-flow#3`
40
+ - Coverage, prompts, outputs, and provider calls are computed per-stage as deltas from the previous stage.
40
41
  - The same engine instance is reused across stages, so memory and output history carry over.
41
42
 
42
43
  ## Ordering and `on_finish`
@@ -81,12 +82,66 @@ flow:
81
82
  contains: ["<previous_response>", "Correction:"]
82
83
  ```
83
84
 
84
- ## Stage-local mocks and env
85
+ ## Stage-local configuration
86
+
87
+ ### Mocks and env
85
88
 
86
89
  - Stage mocks override flow-level defaults: the runner merges `{...flow.mocks, ...stage.mocks}`.
87
90
  - `env:` applies only for the stage and is restored afterward.
88
91
 
92
+ ### Routing overrides
93
+
94
+ Per-stage routing settings override the base config for that stage only:
95
+
96
+ ```yaml
97
+ flow:
98
+ - name: correction-loop
99
+ event: issue_comment
100
+ routing:
101
+ max_loops: 10 # allow more iterations for this stage
102
+ # ...
103
+ ```
104
+
105
+ ### Tag filtering
106
+
107
+ Tags can be specified at flow-level and/or per-stage. They are merged with suite defaults:
108
+
109
+ ```yaml
110
+ - name: my-flow
111
+ tags: "github" # flow-level include filter
112
+ exclude_tags: "slow" # flow-level exclude filter
113
+ flow:
114
+ - name: stage-one
115
+ tags: "security" # additional per-stage filter
116
+ # ...
117
+ ```
118
+
119
+ ### GitHub recorder overrides
120
+
121
+ Simulate GitHub API errors or timeouts per-stage:
122
+
123
+ ```yaml
124
+ flow:
125
+ - name: api-error-stage
126
+ event: pr_opened
127
+ github_recorder:
128
+ error_code: 429 # simulate rate limit
129
+ # ...
130
+ ```
131
+
89
132
  ## Debugging flows
90
133
 
91
134
  - Set `VISOR_DEBUG=true` to print stage headers, selected checks, and internal debug lines from the engine.
92
135
  - To reduce noise, limit the run to a stage: `VISOR_DEBUG=true visor test --only pr-review-e2e-flow#facts-invalid`.
136
+ - Use the CLI `--debug` flag as a shorthand: `visor test --debug --only case#stage`.
137
+
138
+ ## Related Documentation
139
+
140
+ - [Getting Started](./getting-started.md) - Introduction to the test framework
141
+ - [DSL Reference](./dsl-reference.md) - Complete test YAML schema
142
+ - [Assertions](./assertions.md) - Available assertion types
143
+ - [Fixtures and Mocks](./fixtures-and-mocks.md) - Managing test data
144
+ - [Cookbook](./cookbook.md) - Copy-pasteable test recipes
145
+ - [CLI](./cli.md) - Test runner command line options
146
+ - [CI Integration](./ci.md) - Running tests in CI pipelines
147
+ - [Troubleshooting](./troubleshooting.md) - Common issues and solutions
@@ -4,10 +4,10 @@ This is the developer-facing guide for writing and running integration tests for
4
4
 
5
5
  ## TL;DR
6
6
 
7
- - Put your tests in `defaults/.visor.tests.yaml`.
7
+ - Put your tests in `defaults/visor.tests.yaml`.
8
8
  - Reference your base config with `extends: ".visor.yaml"`.
9
9
  - Use built-in GitHub fixtures like `gh.pr_open.minimal`.
10
- - Run with `visor test --config defaults/.visor.tests.yaml`.
10
+ - Run with `visor test --config defaults/visor.tests.yaml`.
11
11
  - Validate only with `visor test --validate`.
12
12
 
13
13
  ```yaml
@@ -78,16 +78,17 @@ Run `visor test --validate` to get precise YAML-path errors and suggestions:
78
78
  ```
79
79
  ❌ Tests file has 2 error(s):
80
80
  • tests.cases[0].expext: must NOT have additional properties (Did you mean "expect"?)
81
- • tests.cases[3].event: must be equal to one of the allowed values (allowed: manual, pr_opened, pr_updated, pr_closed, issue_opened, issue_comment)
81
+ • tests.cases[3].event: must be equal to one of the allowed values (allowed: manual, pr_opened, pr_updated, pr_closed, issue_opened, issue_comment, schedule, webhook_received)
82
82
  ```
83
83
 
84
- Next steps:
85
- - Core reference: `docs/testing/dsl-reference.md`
86
- - Flows: `docs/testing/flows.md`
87
- - Mocks & fixtures: `docs/testing/fixtures-and-mocks.md`
88
- - Assertions: `docs/testing/assertions.md`
89
- - Cookbook: `docs/testing/cookbook.md`
90
- - CLI & reporters: `docs/testing/cli.md`
91
- - CI integration: `docs/testing/ci.md`
92
- - Troubleshooting: `docs/testing/troubleshooting.md`
93
- - Browse `defaults/.visor.tests.yaml` for full examples.
84
+ ## Next Steps
85
+
86
+ - [DSL Reference](./dsl-reference.md) - Complete test YAML schema
87
+ - [Flows](./flows.md) - Multi-stage test flows
88
+ - [Fixtures and Mocks](./fixtures-and-mocks.md) - Managing test data
89
+ - [Assertions](./assertions.md) - Available assertion types
90
+ - [Cookbook](./cookbook.md) - Common patterns and recipes
91
+ - [CLI and Reporters](./cli.md) - Command-line options and output formats
92
+ - [CI Integration](./ci.md) - Running tests in CI pipelines
93
+ - [Troubleshooting](./troubleshooting.md) - Common issues and solutions
94
+ - Browse `defaults/visor.tests.yaml` for full examples.