@verica-app/cli 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +77 -0
- package/dist/cli.js +4590 -0
- package/package.json +51 -0
package/README.md
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# @verica-app/cli
|
|
2
|
+
|
|
3
|
+
Run a [Verica](https://verica.app) eval from CI and **gate the merge on the result**.
|
|
4
|
+
|
|
5
|
+
The eval — criteria, golden set, graders, and the pass condition — lives in Verica.
|
|
6
|
+
The CLI just triggers a run, waits for the verdict, writes a JUnit report, and exits
|
|
7
|
+
`0`/`1` so your pipeline can block the merge. It's a thin HTTP client: the only secret
|
|
8
|
+
it needs is a workspace token (provider keys stay in Verica via BYOK).
|
|
9
|
+
|
|
10
|
+
## Install
|
|
11
|
+
|
|
12
|
+
No install needed — run it with `npx`:
|
|
13
|
+
|
|
14
|
+
```bash
|
|
15
|
+
npx @verica-app/cli run --eval <eval-id> --prompt prompts/agent.txt --wait --junit report.xml
|
|
16
|
+
```
|
|
17
|
+
|
|
18
|
+
Or add it as a dev dependency:
|
|
19
|
+
|
|
20
|
+
```bash
|
|
21
|
+
npm i -D @verica-app/cli
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
## Usage
|
|
25
|
+
|
|
26
|
+
```bash
|
|
27
|
+
verica run \
|
|
28
|
+
--eval eval_8x2k9d \
|
|
29
|
+
--prompt prompts/support-agent.txt \
|
|
30
|
+
--model gpt-4.1-mini \
|
|
31
|
+
--wait \
|
|
32
|
+
--junit verica-results.xml \
|
|
33
|
+
--json
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Multi-prompt via a manifest:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
verica run --manifest .verica.yml --wait --junit report.xml
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
```yaml
|
|
43
|
+
# .verica.yml
|
|
44
|
+
evals:
|
|
45
|
+
- id: eval_8x2k9d
|
|
46
|
+
prompt: prompts/support-agent.txt
|
|
47
|
+
sampling: { temperature: 0.2, maxTokens: 512 }
|
|
48
|
+
model: gpt-4.1-mini
|
|
49
|
+
- id: eval_3p1m7q
|
|
50
|
+
prompt: prompts/triage.txt
|
|
51
|
+
model: claude-sonnet-4-6
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
## Environment
|
|
55
|
+
|
|
56
|
+
| Var | Required | Notes |
|
|
57
|
+
| ----------------- | -------- | -------------------------------------------- |
|
|
58
|
+
| `VERICA_TOKEN` | yes | Workspace API token (Settings → API tokens). |
|
|
59
|
+
| `VERICA_BASE_URL` | yes\* | Your Verica base URL (or pass `--base-url`). |
|
|
60
|
+
|
|
61
|
+
## Key flags
|
|
62
|
+
|
|
63
|
+
- `--eval <id>` / `--manifest <file>` — what to run.
|
|
64
|
+
- `--prompt <file>` / `--system-prompt <file>` — prompt content to push (versioned by content).
|
|
65
|
+
- `--model <model>` · `--sampling <file.json>` — execution config.
|
|
66
|
+
- `--wait` — poll to completion; the exit code reflects the gate.
|
|
67
|
+
- `--junit <file>` · `--junit-mode rows|gate` — JUnit report (default `rows`).
|
|
68
|
+
- `--json` — machine-readable results on stdout.
|
|
69
|
+
- `--threshold <0..1>` · `--baseline-ref <ref>` · `--baseline-run <id>` — override the gate per branch.
|
|
70
|
+
- `--git-sha` / `--git-ref` — provenance (auto-detected from CI env otherwise).
|
|
71
|
+
|
|
72
|
+
## Exit codes
|
|
73
|
+
|
|
74
|
+
`0` passed · `1` gate failed · `2` validation/transport error.
|
|
75
|
+
|
|
76
|
+
MIT licensed. There's no IP in the client — the engine, graders, gate, and crypto all
|
|
77
|
+
run server-side behind the token API.
|