@jmoak/vibrissa 0.1.0 → 2.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -21
- package/README.md +268 -254
- package/bin/vib-test.js +1 -1
- package/dist/chunk-AMDRQ2RV.js +2329 -0
- package/dist/chunk-AMDRQ2RV.js.map +1 -0
- package/dist/chunk-TZK5KIGQ.js +10706 -0
- package/dist/chunk-TZK5KIGQ.js.map +1 -0
- package/dist/cli.d.ts +57 -0
- package/dist/cli.js +5896 -613
- package/dist/cli.js.map +1 -1
- package/dist/index.d.ts +837 -58
- package/dist/index.js +211 -10
- package/dist/index.js.map +1 -1
- package/dist/runner-J_V7pgax.d.ts +1194 -0
- package/dist/vitest.d.ts +44 -0
- package/dist/vitest.js +240 -0
- package/dist/vitest.js.map +1 -0
- package/package.json +27 -9
- package/schema/case.schema.json +405 -14
- package/schema/vibrissa.schema.json +528 -12
- package/dist/chunk-FTVETYUK.js +0 -656
- package/dist/chunk-FTVETYUK.js.map +0 -1
package/LICENSE
CHANGED
|
@@ -1,21 +1,21 @@
|
|
|
1
|
-
MIT License
|
|
2
|
-
|
|
3
|
-
Copyright (c) 2025 Jordan Moak
|
|
4
|
-
|
|
5
|
-
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
-
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
-
in the Software without restriction, including without limitation the rights
|
|
8
|
-
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
-
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
-
furnished to do so, subject to the following conditions:
|
|
11
|
-
|
|
12
|
-
The above copyright notice and this permission notice shall be included in all
|
|
13
|
-
copies or substantial portions of the Software.
|
|
14
|
-
|
|
15
|
-
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
-
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
-
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
-
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
-
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
-
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
-
SOFTWARE.
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Jordan Moak
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
CHANGED
|
@@ -1,315 +1,329 @@
|
|
|
1
|
-
|
|
1
|
+
# vibrissa
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
> *vibrissa* (n.): a whisker; a sensory organ for probing the world.
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
- JSON cases: simple, diff‑friendly, easy to generate
|
|
8
|
-
- Powerful matching: exact, partial, regex, wildcard, unordered arrays
|
|
9
|
-
- Flexible: CLI (`vib-test`) or library API
|
|
5
|
+
**Contract testing for MCP servers.** Record real responses as JSON, replay them in CI,
|
|
6
|
+
lock your tool manifest, fuzz your schemas. No test code: the whole suite is data.
|
|
10
7
|
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
8
|
+
[](https://www.npmjs.com/package/@jmoak/vibrissa)
|
|
9
|
+
[](https://github.com/JMoak/vibrissa/actions/workflows/ci.yml)
|
|
10
|
+
[](https://nodejs.org)
|
|
11
|
+
[](./LICENSE)
|
|
14
12
|
|
|
15
|
-
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
13
|
+
Vibrissa drives your server over stdio or Streamable HTTP, speaks both MCP 2026-07-28
|
|
14
|
+
and the 2025 revisions, and diffs every tool, resource, and prompt result against
|
|
15
|
+
committed JSON. It also snapshots the server's whole surface into a lock file, so a
|
|
16
|
+
changed schema or a quietly rewritten tool description fails the PR instead of reaching
|
|
17
|
+
a client.
|
|
19
18
|
|
|
20
|
-
### Quickstart
|
|
21
19
|
```bash
|
|
22
20
|
npm i -D @jmoak/vibrissa
|
|
23
|
-
vib init
|
|
24
|
-
|
|
25
|
-
vib tools
|
|
26
|
-
vib call echo --arg text=hello
|
|
27
|
-
vib record echo --arg text=hello --
|
|
28
|
-
vib
|
|
21
|
+
npx vib init # vibrissa.json + a real sample case from your server
|
|
22
|
+
npx vib discover # protocol era, capabilities, identity
|
|
23
|
+
npx vib tools # what the server exposes
|
|
24
|
+
npx vib call echo --arg text=hello
|
|
25
|
+
npx vib record echo --arg text=hello --name "echo hello"
|
|
26
|
+
npx vib manifest record # lock the surface
|
|
27
|
+
npx vib run # replay every case: the CI gate
|
|
28
|
+
npx vib check # spec sanity + description lint
|
|
29
|
+
npx vib fuzz echo --runs 200 # schema-driven fuzzing
|
|
29
30
|
```
|
|
30
31
|
|
|
31
|
-
|
|
32
|
+
Releases ship through npm [trusted publishing](https://docs.npmjs.com/trusted-publishers/)
|
|
33
|
+
with provenance: each version links to the commit and workflow that built it.
|
|
32
34
|
|
|
33
|
-
|
|
34
|
-
|
|
35
|
-
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Why
|
|
38
|
+
|
|
39
|
+
MCP testing has sorted itself into layers:
|
|
40
|
+
|
|
41
|
+
| Layer | Question | Tools |
|
|
42
|
+
|---|---|---|
|
|
43
|
+
| 0: Conformance | Does the server speak the spec? | official conformance suite; `vib check` covers the deterministic subset |
|
|
44
|
+
| **1: Contract** | **Given these args, does this server return this?** | **Vibrissa** |
|
|
45
|
+
| 2: Robustness | Fuzzing, poisoning, rug pulls | `vib fuzz`, `vib check`, manifest lock (deterministic subset) |
|
|
46
|
+
| 3: Model evals | Does an LLM pick the right tool? | eval platforms; out of core by design |
|
|
47
|
+
|
|
48
|
+
Vibrissa owns Layer 1 and absorbs the deterministic halves of 0 and 2. No LLM in the
|
|
49
|
+
critical path, no accounts, no hosted anything: CI gates are reproducible and the loop is
|
|
50
|
+
sub-second.
|
|
51
|
+
|
|
52
|
+
The 2026-07-28 revision rewrote the core. Each change lands on what a contract tester
|
|
53
|
+
must do:
|
|
54
|
+
|
|
55
|
+
| 2026-07-28 change | Vibrissa |
|
|
56
|
+
|---|---|
|
|
57
|
+
| `initialize` replaced by `server/discover` | `protocol: auto` negotiates either era |
|
|
58
|
+
| `resultType` + `_meta` serverInfo on every result | normalized away; old goldens stay green |
|
|
59
|
+
| Multi Round-Trip Requests (`input_required`) | `respond` scripts the answers |
|
|
60
|
+
| No protocol sessions; state in server-minted handles | flows with `capture` and `{{name}}` |
|
|
61
|
+
| Deterministic `tools/list` with cache hints | manifest lock; `vib check` |
|
|
62
|
+
| JSON Schema 2020-12 `outputSchema` | enforced on every call |
|
|
63
|
+
| Tasks extension with `tasks/get` polling | `await`, `expectTask` |
|
|
64
|
+
| Renumbered error codes | stable code names on both eras |
|
|
65
|
+
|
|
66
|
+
## The loop
|
|
67
|
+
|
|
68
|
+
| Step | Command | What it does |
|
|
69
|
+
|---|---|---|
|
|
70
|
+
| explore | `vib discover`, `vib tools`, `vib resources`, `vib prompts` | see the surface |
|
|
71
|
+
| probe | `vib call <tool>`, `vib read <uri>`, `vib get-prompt <name>` | one live request, printed |
|
|
72
|
+
| capture | `vib record ...`, `vib tap -- <server>` | real responses become golden cases |
|
|
73
|
+
| gate | `vib run` | replay everything, diff the manifest; non-zero on any mismatch |
|
|
74
|
+
| harden | `vib check`, `vib fuzz <tool>` | spec sanity, poisoning lint, generated edge cases |
|
|
75
|
+
|
|
76
|
+
Every command except `tap` takes `--json`. [MCP Inspector](https://modelcontextprotocol.io/docs/tools/inspector)
|
|
77
|
+
is still the place to click around: `vib inspect` opens it with your config.
|
|
78
|
+
|
|
79
|
+
## Anatomy of a case
|
|
80
|
+
|
|
81
|
+
The v1 shape is still the core shape. `tests/integration/echo.basic.json`:
|
|
41
82
|
|
|
42
|
-
2) Create a case: `tests/integration/echo.basic.json`
|
|
43
83
|
```json
|
|
44
84
|
{
|
|
45
85
|
"name": "echo basic",
|
|
46
86
|
"tool": "echo",
|
|
47
|
-
"args": {
|
|
48
|
-
|
|
49
|
-
"uppercase": true
|
|
50
|
-
},
|
|
51
|
-
"expect": {
|
|
52
|
-
"content": [
|
|
53
|
-
{
|
|
54
|
-
"type": "text",
|
|
55
|
-
"text": "HELLO"
|
|
56
|
-
}
|
|
57
|
-
]
|
|
58
|
-
}
|
|
87
|
+
"args": { "text": "hello" },
|
|
88
|
+
"expect": { "content": [{ "type": "text", "text": "hello" }] }
|
|
59
89
|
}
|
|
60
90
|
```
|
|
61
91
|
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
vib-test run
|
|
65
|
-
```
|
|
92
|
+
`expect` is exact deep equality over the normalized result unless a matcher says
|
|
93
|
+
otherwise. `resource`, `resourceTemplate`, and `prompt` work the same way as `tool`.
|
|
66
94
|
|
|
67
|
-
|
|
68
|
-
```json
|
|
69
|
-
{
|
|
70
|
-
"scripts": {
|
|
71
|
-
"test:integration": "vib-test run"
|
|
72
|
-
}
|
|
73
|
-
}
|
|
74
|
-
```
|
|
95
|
+
**A flow** chains steps on one session and passes values forward:
|
|
75
96
|
|
|
76
|
-
### Test Case Semantics
|
|
77
|
-
- Default is exact deep equality.
|
|
78
|
-
- Partial
|
|
79
|
-
```json
|
|
80
|
-
{
|
|
81
|
-
"expect": {
|
|
82
|
-
"$partial": true,
|
|
83
|
-
"content": [
|
|
84
|
-
{
|
|
85
|
-
"type": "text"
|
|
86
|
-
}
|
|
87
|
-
]
|
|
88
|
-
}
|
|
89
|
-
}
|
|
90
|
-
```
|
|
91
|
-
- Regex
|
|
92
|
-
```json
|
|
93
|
-
{
|
|
94
|
-
"expect": {
|
|
95
|
-
"content": [
|
|
96
|
-
{
|
|
97
|
-
"type": "text",
|
|
98
|
-
"text": {
|
|
99
|
-
"$regex": "^hel+o$",
|
|
100
|
-
"flags": "i"
|
|
101
|
-
}
|
|
102
|
-
}
|
|
103
|
-
]
|
|
104
|
-
}
|
|
105
|
-
}
|
|
106
|
-
```
|
|
107
|
-
- Wildcard (glob‑like)
|
|
108
|
-
```json
|
|
109
|
-
{
|
|
110
|
-
"expect": {
|
|
111
|
-
"content": [
|
|
112
|
-
{
|
|
113
|
-
"type": "text",
|
|
114
|
-
"text": {
|
|
115
|
-
"$wildcard": "hel*o"
|
|
116
|
-
}
|
|
117
|
-
}
|
|
118
|
-
]
|
|
119
|
-
}
|
|
120
|
-
}
|
|
121
|
-
```
|
|
122
|
-
- Unordered arrays
|
|
123
97
|
```json
|
|
124
98
|
{
|
|
125
|
-
"
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
]
|
|
99
|
+
"name": "sum flows into echo",
|
|
100
|
+
"steps": [
|
|
101
|
+
{ "tool": "typed", "args": { "a": 2, "b": 3 }, "capture": { "sum": "structuredContent.sum" } },
|
|
102
|
+
{
|
|
103
|
+
"tool": "echo",
|
|
104
|
+
"args": { "text": "sum is {{sum}}" },
|
|
105
|
+
"expect": { "content": [{ "type": "text", "text": "sum is 5" }] }
|
|
133
106
|
}
|
|
134
|
-
|
|
107
|
+
]
|
|
135
108
|
}
|
|
136
109
|
```
|
|
137
|
-
|
|
110
|
+
|
|
111
|
+
**An MRTR case** answers the server's question and asserts it was asked:
|
|
112
|
+
|
|
138
113
|
```json
|
|
139
114
|
{
|
|
140
|
-
"
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
}
|
|
115
|
+
"name": "confirm deploy",
|
|
116
|
+
"tool": "confirm",
|
|
117
|
+
"args": { "action": "deploy" },
|
|
118
|
+
"respond": { "elicitation/create": { "action": "accept", "content": { "confirm": true } } },
|
|
119
|
+
"expectInputRequired": { "$partial": true, "elicitation/create": { "message": "Proceed with deploy?" } },
|
|
120
|
+
"expect": { "content": [{ "type": "text", "text": "confirmed: deploy" }] }
|
|
146
121
|
}
|
|
147
122
|
```
|
|
148
123
|
|
|
149
|
-
|
|
150
|
-
|
|
151
|
-
|
|
152
|
-
|
|
153
|
-
|
|
124
|
+
Also: `setup` / `teardown`, `await` for long-running tasks, `budgetMs`, `tags`, `skip`,
|
|
125
|
+
`flaky`, per-case `server`, `$ref` fragments. Every key: [docs/CASES.md](./docs/CASES.md).
|
|
126
|
+
|
|
127
|
+
## Matchers
|
|
128
|
+
|
|
129
|
+
Matchers nest anywhere in an expectation. Several in one object must all pass.
|
|
130
|
+
|
|
131
|
+
| Matcher | Example | Asserts |
|
|
132
|
+
|---|---|---|
|
|
133
|
+
| `$partial` | `{"$partial": true, "isError": false}` | only the fields you name |
|
|
134
|
+
| `$regex` | `{"$regex": "^hel+o$", "flags": "i"}` | pattern match |
|
|
135
|
+
| `$wildcard` | `{"$wildcard": "req_*"}` | glob match |
|
|
136
|
+
| `$unordered` | `{"$unordered": true, "value": [1, 2]}` | array in any order |
|
|
137
|
+
| `$any` / `$ignore` | `{"$any": true}` | present / don't care |
|
|
138
|
+
| `$type` | `{"$type": ["string", "null"]}` | JSON type |
|
|
139
|
+
| `$schema` | `{"$schema": {"type": "string", "format": "date-time"}}` | inline JSON Schema |
|
|
140
|
+
| `$oneOf` | `{"$oneOf": ["queued", "running"]}` | any alternative |
|
|
141
|
+
| `$contains` | `{"$contains": {"type": "image"}, "count": 1}` | some element matches |
|
|
142
|
+
| `$length` / `$number` | `{"$length": {"min": 1}}` | sizes and ranges |
|
|
143
|
+
| `$ref` | `{"$ref": "./shared/user.json"}` | reuse a fragment |
|
|
144
|
+
| `$literal` | `{"$literal": {"$ref": "#/a"}}` | data that looks like a matcher |
|
|
145
|
+
|
|
146
|
+
`expectError` asserts that a call fails correctly:
|
|
147
|
+
`{"code": "InvalidParams", "message": {"$regex": "text"}}`. Full reference:
|
|
148
|
+
[docs/MATCHERS.md](./docs/MATCHERS.md).
|
|
149
|
+
|
|
150
|
+
## Manifest lock
|
|
151
|
+
|
|
152
|
+
`vib manifest record` snapshots `server/discover`, `tools/list`, `resources/list`,
|
|
153
|
+
`resources/templates/list`, and `prompts/list` into
|
|
154
|
+
`tests/integration/__manifest__.json`. From then on `vib run` diffs the live surface
|
|
155
|
+
against it and fails on added or removed tools, changed input or output schemas, changed
|
|
156
|
+
annotations, changed capabilities, and **changed descriptions**.
|
|
157
|
+
|
|
158
|
+
It is `package-lock.json` for your MCP surface. It also catches rug pulls: a tool
|
|
159
|
+
description that changes after review, the move behind tool-poisoning attacks, shows up
|
|
160
|
+
as a reviewed diff in git instead of a silent change. Each change is marked breaking or
|
|
161
|
+
not (a removed property or a newly required one is; a reworded description is drift, but
|
|
162
|
+
not breaking), with a word diff for prose. Accept intended changes with
|
|
163
|
+
`vib run --update-manifest` and commit the lock with the PR that caused them. One lock
|
|
164
|
+
serves both protocol eras; `requireManifest: true` makes a missing lock fail CI.
|
|
165
|
+
|
|
166
|
+
## Checks and fuzz
|
|
167
|
+
|
|
168
|
+
**`vib check`** runs offline checks over the manifest: `spec/*` sanity (known protocol
|
|
169
|
+
version, deterministic `tools/list`, cache hints, valid 2020-12 schemas, resolvable
|
|
170
|
+
`$ref`s) and `desc/*` lint for hidden instructions, invisible Unicode, tool shadowing,
|
|
171
|
+
credential paths, and exfiltration URLs in model-visible text. It is a smoke detector,
|
|
172
|
+
not a scanner. `vib run --check` folds it into CI. Ids and allow-listing:
|
|
173
|
+
[docs/CHECKS.md](./docs/CHECKS.md).
|
|
174
|
+
|
|
175
|
+
**`vib fuzz <tool>`** generates arguments from the tool's `inputSchema`: valid,
|
|
176
|
+
boundary, invalid, and (with `--inject`) prompt-injection payloads. It checks invariants
|
|
177
|
+
that need no golden: valid args never cause a protocol error, invalid args get
|
|
178
|
+
`InvalidParams`, the server never crashes. Seeded and reproducible;
|
|
179
|
+
`--record-failures` writes each counterexample as a case. See [docs/FUZZ.md](./docs/FUZZ.md).
|
|
180
|
+
|
|
181
|
+
## Reporters and CI
|
|
182
|
+
|
|
183
|
+
Reporters: `console` (default), `json` (`reports/results.json`), `junit`
|
|
184
|
+
(`reports/junit.xml`), `github` (annotations + job summary), `tap`. Use several at once.
|
|
185
|
+
|
|
186
|
+
```yaml
|
|
187
|
+
# .github/workflows/vibrissa.yml (vib init --ci writes this)
|
|
188
|
+
name: vibrissa
|
|
189
|
+
on:
|
|
190
|
+
push:
|
|
191
|
+
branches: [main]
|
|
192
|
+
pull_request:
|
|
193
|
+
jobs:
|
|
194
|
+
contract:
|
|
195
|
+
runs-on: ubuntu-latest
|
|
196
|
+
steps:
|
|
197
|
+
- uses: actions/checkout@v4
|
|
198
|
+
- uses: actions/setup-node@v4
|
|
199
|
+
with: { node-version: 22, cache: npm }
|
|
200
|
+
- run: npm ci
|
|
201
|
+
- run: npm run build --if-present
|
|
202
|
+
- run: npx vib run --profile ci
|
|
203
|
+
- uses: actions/upload-artifact@v4
|
|
204
|
+
if: failure()
|
|
205
|
+
with: { name: vibrissa-reports, path: reports/ }
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
**Delta gating:** `vib run --baseline reports/last-green.json` fails only on regressions
|
|
209
|
+
against a previous green run, so a known-broken case does not block unrelated PRs.
|
|
210
|
+
`--repeat 5` measures flake rates, `--shard 2/4` splits the suite across jobs, and every
|
|
211
|
+
failing case gets a JSON-RPC transcript under `reports/traces/`. Details:
|
|
212
|
+
[docs/CI.md](./docs/CI.md).
|
|
154
213
|
|
|
155
|
-
|
|
214
|
+
## Agents write great cases
|
|
156
215
|
|
|
157
|
-
|
|
216
|
+
JSON cases are the most generatable test format there is, and `record` keeps agents
|
|
217
|
+
honest. Start the agent with **`vib help cases`**: the whole case format, every matcher,
|
|
218
|
+
flows, MRTR, and tasks on one screen. Then the same loop you use:
|
|
158
219
|
|
|
159
|
-
|
|
160
|
-
vib
|
|
161
|
-
vib
|
|
162
|
-
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
vib
|
|
166
|
-
|
|
220
|
+
1. `vib tools --json`: the agent reads the contract, schemas included.
|
|
221
|
+
2. `vib call <tool> --args '{...}' --json`: it probes behavior and edge cases.
|
|
222
|
+
3. `vib record ...`: real responses become goldens. No hallucinated envelopes, and
|
|
223
|
+
timestamps, UUIDs, and values that change between calls are loosened to matchers
|
|
224
|
+
automatically. (The stable check calls the tool twice; pass `--no-stable-check` for
|
|
225
|
+
tools with side effects. Tools annotated destructive are called once.)
|
|
226
|
+
4. `vib tap -- node dist/index.js`: or record real client sessions and pick calls with
|
|
227
|
+
`vib record --from-tap`.
|
|
228
|
+
5. `vib run --json --brief`: the agent verifies its own suite from one JSON report on
|
|
229
|
+
stdout, listing only what failed.
|
|
167
230
|
|
|
168
|
-
|
|
169
|
-
|
|
170
|
-
|
|
171
|
-
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
|
|
175
|
-
|
|
176
|
-
|
|
177
|
-
|
|
178
|
-
|
|
179
|
-
Shared options:
|
|
180
|
-
- **--config**: path to your config file (otherwise auto-discovered from the current directory)
|
|
181
|
-
- **--server**: override server command (quoted argv ok), e.g. `--server "node dist/index.js"`
|
|
182
|
-
- **--server-cwd**: working directory for starting the server (overrides `server.cwd`; relative paths resolve from the shell cwd)
|
|
183
|
-
- **--env KEY=VALUE**: extra environment (repeatable)
|
|
184
|
-
- **--timeout**: per-call / spawn timeout in ms
|
|
185
|
-
- **--json**: machine-readable output for `tools` / `call` / `record`
|
|
186
|
-
|
|
187
|
-
`run` options:
|
|
188
|
-
- **--cases**: glob of JSON cases (relative to the shell cwd when passed on the CLI)
|
|
189
|
-
- **--allow-empty**: exit 0 when no cases match (by default, zero matches is a failure)
|
|
190
|
-
- **--fail-fast**, **--pretty** / **--no-pretty**
|
|
191
|
-
|
|
192
|
-
`record` options: `--out`, `--name`, `--partial`, `--force`
|
|
193
|
-
|
|
194
|
-
Notes:
|
|
195
|
-
- Globs in `vibrissa.json` resolve relative to the config file's directory.
|
|
196
|
-
- Relative `server.cwd` values in config also resolve from that directory.
|
|
197
|
-
- Matching zero case files exits `1` unless `--allow-empty` / `allowEmpty: true` is set.
|
|
198
|
-
- Recorded cases include `$schema` pointing at `schema/case.schema.json` for editor validation.
|
|
199
|
-
|
|
200
|
-
### Case Layout (recommended)
|
|
201
|
-
```
|
|
202
|
-
tests/
|
|
203
|
-
integration/
|
|
204
|
-
echo.basic.json
|
|
205
|
-
tools/
|
|
206
|
-
invalid-args.json
|
|
207
|
-
flow.multistep.json
|
|
208
|
-
```
|
|
231
|
+
Everything is built for that loop: `--json` always prints one object with `ok` (errors
|
|
232
|
+
too, as `{ "ok": false, "error": { "type", "code", "message", "hint" }, "next" }`), unknown
|
|
233
|
+
flags, tools, keys, matchers, and error codes all fail with a did-you-mean, and every
|
|
234
|
+
error message says what to do next.
|
|
235
|
+
|
|
236
|
+
The worst an agent can do is record a case you don't want, and that is a one-file JSON
|
|
237
|
+
review. `vib explain` paraphrases what each case asserts and flags weak assertions (a
|
|
238
|
+
bare `$partial`, `expectError: {}`, no expectation), for faster PR review.
|
|
239
|
+
|
|
240
|
+
Cases can also be emitted from a [`tdd-dsl`](https://github.com/Rosavera-I/tdd-dsl)
|
|
241
|
+
contract: `tdd-dsl emit --target vibrissa --out-dir tests/integration contract.tdd`.
|
|
209
242
|
|
|
210
|
-
|
|
211
|
-
|
|
243
|
+
## Configuration
|
|
244
|
+
|
|
245
|
+
`vibrissa.json` at the repo root (`vib init` creates it):
|
|
212
246
|
|
|
213
247
|
```json
|
|
214
248
|
{
|
|
215
249
|
"$schema": "./node_modules/@jmoak/vibrissa/schema/vibrissa.schema.json",
|
|
216
|
-
"
|
|
217
|
-
"cmd": "node",
|
|
218
|
-
"
|
|
219
|
-
|
|
220
|
-
|
|
221
|
-
"LOG_LEVEL": "info"
|
|
250
|
+
"servers": {
|
|
251
|
+
"default": { "cmd": "node", "args": ["dist/index.js"] },
|
|
252
|
+
"staging": {
|
|
253
|
+
"url": "https://mcp.example.com/mcp",
|
|
254
|
+
"headers": { "Authorization": "Bearer ${MCP_TOKEN}" }
|
|
222
255
|
}
|
|
223
256
|
},
|
|
224
|
-
"
|
|
225
|
-
|
|
226
|
-
],
|
|
257
|
+
"defaultServer": "default",
|
|
258
|
+
"globs": ["tests/integration/**/*.json"],
|
|
227
259
|
"timeoutMs": 15000,
|
|
228
|
-
"
|
|
260
|
+
"profiles": {
|
|
261
|
+
"ci": { "reporters": ["console", "github", "junit", "json"], "check": true, "requireManifest": true },
|
|
262
|
+
"legacy": { "servers": { "default": { "protocol": "legacy" } } }
|
|
263
|
+
}
|
|
229
264
|
}
|
|
230
265
|
```
|
|
231
266
|
|
|
232
|
-
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
|
|
267
|
+
Precedence is defaults, then the file, then `--profile`, then CLI flags. Paths resolve
|
|
268
|
+
from the config file's directory. Zero matching cases fails the run unless `allowEmpty`.
|
|
269
|
+
Every key: [docs/CONFIG.md](./docs/CONFIG.md).
|
|
270
|
+
|
|
271
|
+
## Compatibility
|
|
272
|
+
|
|
273
|
+
v1 case files and v1 `vibrissa.json` (with `server`) load unchanged, and goldens
|
|
274
|
+
recorded against 2025 servers keep passing against 2026 servers because normalization
|
|
275
|
+
strips the new envelope fields. `vib migrate` rewrites old error-code names and the v1
|
|
276
|
+
`$schema` paths. See [docs/MIGRATION.md](./docs/MIGRATION.md).
|
|
236
277
|
|
|
237
|
-
|
|
238
|
-
- `$schema` gives editor validation/autocomplete. Using the local path works immediately after install and offline.
|
|
239
|
-
- CLI flags always override config values.
|
|
278
|
+
## Library API
|
|
240
279
|
|
|
241
|
-
### Library API
|
|
242
280
|
```ts
|
|
243
|
-
import { runCases } from
|
|
244
|
-
|
|
245
|
-
await runCases({
|
|
246
|
-
server: { cmd: process.execPath, args: ["dist/index.js"], env: {} },
|
|
247
|
-
globs: ["tests/integration/**/*.json"],
|
|
248
|
-
failFast: true,
|
|
249
|
-
timeoutMs: 15000
|
|
250
|
-
});
|
|
251
|
-
```
|
|
281
|
+
import { loadConfig, runCases } from '@jmoak/vibrissa'
|
|
252
282
|
|
|
253
|
-
|
|
254
|
-
|
|
255
|
-
name: integration
|
|
256
|
-
on: [push, pull_request]
|
|
257
|
-
jobs:
|
|
258
|
-
test:
|
|
259
|
-
runs-on: ubuntu-latest
|
|
260
|
-
steps:
|
|
261
|
-
- uses: actions/checkout@v4
|
|
262
|
-
- uses: actions/setup-node@v4
|
|
263
|
-
with:
|
|
264
|
-
node-version: 20
|
|
265
|
-
cache: npm
|
|
266
|
-
- run: npm ci
|
|
267
|
-
- run: npm run build
|
|
268
|
-
- run: npx vib-test --config vibrissa.json
|
|
269
|
-
- uses: actions/upload-artifact@v4
|
|
270
|
-
if: always()
|
|
271
|
-
with:
|
|
272
|
-
name: junit
|
|
273
|
-
path: reports/junit/junit.xml
|
|
283
|
+
const { options } = loadConfig(process.cwd(), { profile: 'ci' })
|
|
284
|
+
process.exitCode = await runCases(options) // 0 ok, 1 failures, 2 config error
|
|
274
285
|
```
|
|
275
286
|
|
|
276
|
-
|
|
277
|
-
|
|
278
|
-
|
|
279
|
-
|
|
280
|
-
- Streaming/long ops: use partial/regex for stability
|
|
281
|
-
- Optional performance thresholds per case
|
|
287
|
+
The v1 call shape (`runCases({ server: { cmd, args }, globs })`) still works.
|
|
288
|
+
`new Runner(options).execute()` returns the full result instead of an exit code.
|
|
289
|
+
|
|
290
|
+
### Vitest bridge
|
|
282
291
|
|
|
283
|
-
|
|
284
|
-
|
|
285
|
-
- Flaky text: prefer `$partial` or `$regex`
|
|
286
|
-
- Globs not matching: run with quotes and verify working directory
|
|
292
|
+
Teams with an existing Vitest suite can run the contract suite inside it, gated on the same
|
|
293
|
+
things as `vib run`:
|
|
287
294
|
|
|
288
|
-
|
|
289
|
-
|
|
295
|
+
```ts
|
|
296
|
+
// tests/mcp.contract.test.ts
|
|
297
|
+
import { describeCases } from '@jmoak/vibrissa/vitest'
|
|
290
298
|
|
|
291
|
-
|
|
299
|
+
await describeCases({ profile: 'ci', checks: true })
|
|
300
|
+
```
|
|
292
301
|
|
|
293
|
-
|
|
302
|
+
It registers a manifest-lock test per server, a checks test (with `checks`), and one `it`
|
|
303
|
+
per case. A string argument is a case glob. Options (`config`, `profile`, `globs`,
|
|
304
|
+
`checks`, `lockManifest`, `title`, `api`, runner options): [docs/CI.md](./docs/CI.md#inside-a-vitest-suite).
|
|
294
305
|
|
|
295
|
-
|
|
296
|
-
|
|
297
|
-
|
|
306
|
+
Also exported: `McpSession`, `loadCases`, `executeCase`, `matchValue`,
|
|
307
|
+
`normalizeResult`, `diffManifest`, `createReporters`, `registerReporter`, `runChecks`,
|
|
308
|
+
`generateArgs`.
|
|
298
309
|
|
|
299
|
-
|
|
300
|
-
given input:
|
|
301
|
-
{"text": "hello"}
|
|
302
|
-
when call "echo"
|
|
303
|
-
then equals:
|
|
304
|
-
{"content": [{"type": "text", "text": "hello"}]}
|
|
305
|
-
```
|
|
310
|
+
## Docs
|
|
306
311
|
|
|
307
|
-
|
|
308
|
-
|
|
309
|
-
|
|
310
|
-
|
|
312
|
+
| Doc | Covers |
|
|
313
|
+
|---|---|
|
|
314
|
+
| [CASES.md](./docs/CASES.md) | every case-file key, flows, MRTR, tasks, recipes |
|
|
315
|
+
| [MATCHERS.md](./docs/MATCHERS.md) | the assertion language |
|
|
316
|
+
| [CONFIG.md](./docs/CONFIG.md) | `vibrissa.json`, servers, profiles |
|
|
317
|
+
| [CLI.md](./docs/CLI.md) | every command and flag |
|
|
318
|
+
| [CI.md](./docs/CI.md) | GitHub Actions, JUnit, shards, baselines |
|
|
319
|
+
| [CHECKS.md](./docs/CHECKS.md) | `vib check` ids |
|
|
320
|
+
| [FUZZ.md](./docs/FUZZ.md) | `vib fuzz` generation and invariants |
|
|
321
|
+
| [MIGRATION.md](./docs/MIGRATION.md) | v1 to v2 |
|
|
322
|
+
| [FUTURE.md](./docs/FUTURE.md) | after 2.0 |
|
|
311
323
|
|
|
312
|
-
|
|
324
|
+
Contributing (humans and agents): [AGENTS.md](./AGENTS.md). Releasing:
|
|
325
|
+
[RELEASING.md](./RELEASING.md).
|
|
313
326
|
|
|
314
|
-
|
|
327
|
+
## License
|
|
315
328
|
|
|
329
|
+
[MIT](./LICENSE) © Jordan Moak
|
package/bin/vib-test.js
CHANGED
|
@@ -1,2 +1,2 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
|
-
import('../dist/cli.js')
|
|
2
|
+
import('../dist/cli.js').then((cli) => cli.runCli(process.argv.slice(2)))
|