clef-mcp 0.1.6 → 0.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +168 -143
- package/assets/demo.svg +70 -0
- package/dist/cli.js +987 -280
- package/dist/cli.js.map +1 -1
- package/dist/index.d.ts +10 -8
- package/dist/index.js +534 -52
- package/dist/index.js.map +1 -1
- package/package.json +2 -1
package/README.md
CHANGED
|
@@ -1,28 +1,22 @@
|
|
|
1
|
+
<div align="center">
|
|
2
|
+
|
|
3
|
+
<img src="./assets/demo.svg" alt="clef_decide — probability distributions for a production incident" width="880"/>
|
|
4
|
+
|
|
1
5
|
# clef-mcp
|
|
2
6
|
|
|
3
7
|
[](https://www.npmjs.com/package/clef-mcp)
|
|
8
|
+
[](https://www.npmjs.com/package/clef-mcp)
|
|
4
9
|
[](https://github.com/HighlyLoadedEgo/ClefMCP/actions/workflows/ci.yml)
|
|
10
|
+
[](https://registry.modelcontextprotocol.io)
|
|
5
11
|
[](LICENSE)
|
|
6
12
|
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
```text
|
|
10
|
-
AI Agent
|
|
11
|
-
↓ MCP (stdio)
|
|
12
|
-
clef-mcp
|
|
13
|
-
↓ POST /v1/systemone
|
|
14
|
-
llama.cpp (llama-server, local)
|
|
15
|
-
↓
|
|
16
|
-
Clef-Flash → probability distribution over your answer options
|
|
17
|
-
```
|
|
13
|
+
**A "reflex" for AI coding agents: structured decisions with probabilities — not prose.**
|
|
18
14
|
|
|
19
|
-
|
|
15
|
+
Local [MCP](https://modelcontextprotocol.io) server that gives agents (Claude Code, Codex, Cursor, ZCode, …) access to the **Clef-Flash** decision model (9B, Apache-2.0 by Cloudflare) through a single tool: `clef_decide`. Pass a `state` and typed questions, get a **probability distribution over your options** in one forward pass. Fully local, offline, no tokens burned.
|
|
20
16
|
|
|
21
|
-
|
|
17
|
+
</div>
|
|
22
18
|
|
|
23
|
-
|
|
24
|
-
- Node.js ≥ 20
|
|
25
|
-
- ≥ 16 GB of memory (Clef-Flash 9B @ 4-bit); more headroom enables higher precision
|
|
19
|
+
---
|
|
26
20
|
|
|
27
21
|
## Quick start
|
|
28
22
|
|
|
@@ -37,65 +31,28 @@ npx clef-mcp setup
|
|
|
37
31
|
npx clef-mcp
|
|
38
32
|
```
|
|
39
33
|
|
|
40
|
-
`clef-mcp` (step 3) never downloads anything. If the model is missing, tool calls return a structured `MODEL_NOT_INSTALLED` error with a hint. Steps 1 and 2
|
|
34
|
+
`clef-mcp` (step 3) never downloads anything. If the model is missing, tool calls return a structured `MODEL_NOT_INSTALLED` error with a hint. Steps 1 and 2 combine: `npx clef-mcp install --setup`.
|
|
41
35
|
|
|
42
|
-
|
|
36
|
+
## Why not just ask the LLM?
|
|
43
37
|
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
38
|
+
| | Chat LLM | `clef_decide` |
|
|
39
|
+
|---|---|---|
|
|
40
|
+
| Output | prose, you parse it | strict JSON: probability per option |
|
|
41
|
+
| Determinism | varies per run | single forward pass, no sampling |
|
|
42
|
+
| Latency (1 decision) | seconds of generation | ~0.5 s local |
|
|
43
|
+
| Context cost | grows with every decision | fixed, small schema |
|
|
44
|
+
| Privacy | depends on provider | 100% on-device, works offline |
|
|
45
|
+
| Calibration | vibes | softmax over trained option scores |
|
|
51
46
|
|
|
52
|
-
**
|
|
53
|
-
|
|
54
|
-
```toml
|
|
55
|
-
[mcp_servers.clef-mcp]
|
|
56
|
-
command = "clef-mcp"
|
|
57
|
-
args = []
|
|
58
|
-
```
|
|
59
|
-
|
|
60
|
-
**Cursor** — `.cursor/mcp.json`:
|
|
61
|
-
|
|
62
|
-
```json
|
|
63
|
-
{
|
|
64
|
-
"mcpServers": {
|
|
65
|
-
"clef-mcp": {
|
|
66
|
-
"command": "clef-mcp",
|
|
67
|
-
"args": []
|
|
68
|
-
}
|
|
69
|
-
}
|
|
70
|
-
}
|
|
71
|
-
```
|
|
72
|
-
|
|
73
|
-
**ZCode** — `~/.zcode/cli/config.json` (user scope; servers auto-connect in new sessions):
|
|
74
|
-
|
|
75
|
-
```json
|
|
76
|
-
{
|
|
77
|
-
"mcp": {
|
|
78
|
-
"servers": {
|
|
79
|
-
"clef-mcp": { "command": "clef-mcp", "args": [], "type": "stdio" }
|
|
80
|
-
}
|
|
81
|
-
}
|
|
82
|
-
}
|
|
83
|
-
```
|
|
84
|
-
|
|
85
|
-
**Generic MCP client** — any client that speaks MCP over stdio: run `clef-mcp` as the server command.
|
|
86
|
-
|
|
87
|
-
Ready-made snippets live in [`examples/`](examples/).
|
|
47
|
+
Sweet spot: **decision points inside agent loops** — next action, routing, classification, severity, yes/no judgment — asked dozens of times per task.
|
|
88
48
|
|
|
89
49
|
## The `clef_decide` tool
|
|
90
50
|
|
|
91
|
-
### Input
|
|
92
|
-
|
|
93
51
|
```json
|
|
94
52
|
{
|
|
95
53
|
"state": {
|
|
96
54
|
"task": "Fix failing tests",
|
|
97
|
-
"error": "TypeError: Cannot read properties of undefined"
|
|
98
|
-
"git_diff": "..."
|
|
55
|
+
"error": "TypeError: Cannot read properties of undefined"
|
|
99
56
|
},
|
|
100
57
|
"questions": {
|
|
101
58
|
"next_action": {
|
|
@@ -113,59 +70,141 @@ Ready-made snippets live in [`examples/`](examples/).
|
|
|
113
70
|
"instructions": "How confident are you in this decision?",
|
|
114
71
|
"criteria": ["very_low", "low", "medium", "high", "very_high"]
|
|
115
72
|
},
|
|
116
|
-
"is_outage": {
|
|
117
|
-
"type": "noul",
|
|
118
|
-
"instructions": "Is a service down?"
|
|
119
|
-
}
|
|
73
|
+
"is_outage": { "type": "noul", "instructions": "Is a service down?" }
|
|
120
74
|
}
|
|
121
75
|
}
|
|
122
76
|
```
|
|
123
77
|
|
|
124
|
-
| Type | Criteria |
|
|
125
|
-
|
|
126
|
-
| `choice` | map
|
|
127
|
-
| `score` | ordered list
|
|
128
|
-
| `noul` | optional `{"true": "...", "false": "..."}` |
|
|
78
|
+
| Type | Criteria | Answer |
|
|
79
|
+
|------|----------|--------|
|
|
80
|
+
| `choice` | map `option id → description`, or a plain list | probability per option |
|
|
81
|
+
| `score` | ordered list (index = score) | probability per level |
|
|
82
|
+
| `noul` | optional `{"true": "...", "false": "..."}` | `{"true": p, "false": 1-p}` |
|
|
129
83
|
|
|
130
|
-
|
|
131
|
-
|
|
132
|
-
Strictly structured — the server never turns the result into prose:
|
|
84
|
+
Response — strictly structured, never prose:
|
|
133
85
|
|
|
134
86
|
```json
|
|
135
87
|
{
|
|
136
88
|
"model": "clef-flash",
|
|
137
89
|
"decisions": {
|
|
138
|
-
"next_action": {
|
|
139
|
-
|
|
140
|
-
}
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
90
|
+
"next_action": { "answer": { "inspect": 0.72, "modify": 0.12, "test": 0.14, "ask_user": 0.02 } },
|
|
91
|
+
"confidence": { "answer": { "very_low": 0.01, "low": 0.04, "medium": 0.18, "high": 0.61, "very_high": 0.16 } },
|
|
92
|
+
"is_outage": { "answer": { "true": 0.9, "false": 0.1 } }
|
|
93
|
+
}
|
|
94
|
+
}
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
Batch up to **64 questions per call** — they are scored in one forward pass. `state` is treated strictly as **data**: never executed, never interpreted as instructions for the server.
|
|
98
|
+
|
|
99
|
+
## Prompts & resources
|
|
100
|
+
|
|
101
|
+
The server ships four MCP **prompts** (canned, decision-shaped asks — your client lists them via `prompts/list`):
|
|
102
|
+
|
|
103
|
+
| Prompt | Purpose |
|
|
104
|
+
|---|---|
|
|
105
|
+
| `incident-triage` | action + severity + user-impact questions for a production incident |
|
|
106
|
+
| `next-action` | what the coding agent should do next + confidence |
|
|
107
|
+
| `ticket-routing` | classify a message into a team + urgency |
|
|
108
|
+
| `security-review` | vulnerability yes/no, risk scale, first mitigation |
|
|
109
|
+
|
|
110
|
+
And three **resources** (read-only, no model needed):
|
|
111
|
+
|
|
112
|
+
| URI | Contents |
|
|
113
|
+
|---|---|
|
|
114
|
+
| `clef-mcp://capabilities` | live JSON: model, runtime, limits, error codes |
|
|
115
|
+
| `clef-mcp://evals/schema` | how to write eval cases |
|
|
116
|
+
| `clef-mcp://evals/dataset` | the bundled 30-case dataset |
|
|
117
|
+
|
|
118
|
+
## Measured, not marketed
|
|
119
|
+
|
|
120
|
+
Apple M4 Pro, Clef-Flash Q4_K_M (6 GB), single request through the full MCP stdio path:
|
|
121
|
+
|
|
122
|
+
| Scenario | Latency |
|
|
123
|
+
|---|---|
|
|
124
|
+
| Cold start (incl. model load, once per session) | ~4.4 s |
|
|
125
|
+
| 1 question | ~0.5 s |
|
|
126
|
+
| 10 questions, one call | ~2.9 s |
|
|
127
|
+
| 64 questions, one call | ~18.6 s |
|
|
128
|
+
|
|
129
|
+
Quality gate: a 30-case evaluation dataset (coding / security / classification / routing / yes-no) — **86.7% pass** on the live model. Run it yourself: `clef-mcp evals`.
|
|
130
|
+
|
|
131
|
+
## Register with your MCP client
|
|
132
|
+
|
|
133
|
+
<details open>
|
|
134
|
+
<summary><b>Claude Code</b></summary>
|
|
135
|
+
|
|
136
|
+
```bash
|
|
137
|
+
claude mcp add clef-mcp -- clef-mcp
|
|
138
|
+
# or, without a global install:
|
|
139
|
+
claude mcp add clef-mcp -- npx -y clef-mcp
|
|
140
|
+
```
|
|
141
|
+
</details>
|
|
142
|
+
|
|
143
|
+
<details>
|
|
144
|
+
<summary><b>Codex</b> — <code>~/.codex/config.toml</code></summary>
|
|
145
|
+
|
|
146
|
+
```toml
|
|
147
|
+
[mcp_servers.clef-mcp]
|
|
148
|
+
command = "clef-mcp"
|
|
149
|
+
args = []
|
|
150
|
+
```
|
|
151
|
+
</details>
|
|
152
|
+
|
|
153
|
+
<details>
|
|
154
|
+
<summary><b>Cursor</b> — <code>.cursor/mcp.json</code></summary>
|
|
155
|
+
|
|
156
|
+
```json
|
|
157
|
+
{
|
|
158
|
+
"mcpServers": {
|
|
159
|
+
"clef-mcp": { "command": "clef-mcp", "args": [] }
|
|
160
|
+
}
|
|
161
|
+
}
|
|
162
|
+
```
|
|
163
|
+
</details>
|
|
164
|
+
|
|
165
|
+
<details>
|
|
166
|
+
<summary><b>ZCode</b> — <code>~/.zcode/cli/config.json</code> (user scope, auto-connect)</summary>
|
|
167
|
+
|
|
168
|
+
```json
|
|
169
|
+
{
|
|
170
|
+
"mcp": {
|
|
171
|
+
"servers": {
|
|
172
|
+
"clef-mcp": { "command": "clef-mcp", "args": [], "type": "stdio" }
|
|
146
173
|
}
|
|
147
174
|
}
|
|
148
175
|
}
|
|
149
176
|
```
|
|
177
|
+
</details>
|
|
150
178
|
|
|
151
|
-
|
|
179
|
+
Ready-made snippets: [`examples/`](examples/).
|
|
152
180
|
|
|
153
181
|
## Teach your agent (skill)
|
|
154
182
|
|
|
155
|
-
The
|
|
156
|
-
|
|
157
|
-
Install it for your agent(s):
|
|
183
|
+
The schema tells the client *what* `clef_decide` accepts; agents also need to know *when* to reach for it and *how* to frame decisions. The bundled [`clef-decisions`](skills/clef-decisions/SKILL.md) skill covers decision patterns, batching, criteria writing, distribution interpretation and error recovery:
|
|
158
184
|
|
|
159
185
|
```bash
|
|
160
|
-
# automatic
|
|
161
|
-
clef-
|
|
186
|
+
clef-mcp setup # automatic
|
|
187
|
+
cp -r skills/clef-decisions ~/.agents/skills/ # manual, from repo
|
|
188
|
+
cp -r "$(npm root -g)/clef-mcp/skills/clef-decisions" ~/.agents/skills/ # from npm package
|
|
189
|
+
```
|
|
162
190
|
|
|
163
|
-
|
|
164
|
-
|
|
165
|
-
|
|
191
|
+
## Architecture
|
|
192
|
+
|
|
193
|
+
```mermaid
|
|
194
|
+
flowchart LR
|
|
195
|
+
subgraph clients [MCP clients]
|
|
196
|
+
CC[Claude Code]
|
|
197
|
+
CX[Codex]
|
|
198
|
+
CU[Cursor]
|
|
199
|
+
ZC[ZCode]
|
|
200
|
+
end
|
|
201
|
+
clients -- MCP stdio --> S[clef-mcp<br/>validation · limits · structured errors]
|
|
202
|
+
S -- SystemOne adapter --> R[ClefRuntime<br/>llama.cpp subprocess<br/>127.0.0.1]
|
|
203
|
+
R -- single forward pass --> M[("Clef-Flash<br/>9B · GGUF · local")]
|
|
204
|
+
M -. probabilities .-> S -. strict JSON .-> clients
|
|
166
205
|
```
|
|
167
206
|
|
|
168
|
-
|
|
207
|
+
The `ClefRuntime` interface (`load / decide / unload / health`) isolates the engine: MLX or remote runtimes plug in without changing the MCP API. The wire format is `POST /v1/systemone` — the same contract across llama.cpp and other Clef runtimes.
|
|
169
208
|
|
|
170
209
|
## CLI
|
|
171
210
|
|
|
@@ -180,44 +219,44 @@ clef-mcp uninstall # remove the model (and optionally the managed runtime)
|
|
|
180
219
|
clef-mcp evals # run the evaluation dataset against the installed model
|
|
181
220
|
```
|
|
182
221
|
|
|
183
|
-
|
|
222
|
+
Flags: `install --quant Q8_0 --yes --skip-probe`, `install --setup`, `setup --clients zcode,cursor --no-skill`, `doctor --deep` (re-hash the model file), `uninstall --runtime --yes`.
|
|
223
|
+
|
|
224
|
+
## Runtimes: llama.cpp and MLX
|
|
184
225
|
|
|
185
|
-
|
|
226
|
+
Two local runtimes behind the same `ClefRuntime` interface:
|
|
227
|
+
|
|
228
|
+
| | `llama-cpp` (default) | `mlx` |
|
|
229
|
+
|---|---|---|
|
|
230
|
+
| Platforms | macOS, Linux, Windows | macOS / Apple Silicon only |
|
|
231
|
+
| Model | GGUF from `ggml-org/Clef-Flash-GGUF` | MLX 4-bit from `mlx-community/clef-flash-4bit` |
|
|
232
|
+
| Extras | none | [uv](https://docs.astral.sh/uv/) on PATH (managed Python env) |
|
|
233
|
+
| Install | `clef-mcp install` | `clef-mcp install --runtime mlx` |
|
|
234
|
+
|
|
235
|
+
Switch at runtime with `CLEF_RUNTIME=mlx` (must be set for the MCP server process — e.g. in the client's `env` block). Both speak the same `POST /v1/systemone` contract. The MLX snapshot is fetched into `CLEF_HOME` via a uv-managed `huggingface_hub` (no global Python state) at a pinned revision.
|
|
236
|
+
|
|
237
|
+
## Configuration
|
|
186
238
|
|
|
187
239
|
| Variable | Default | Meaning |
|
|
188
240
|
|----------|---------|---------|
|
|
189
|
-
| `CLEF_MODEL` | `clef-flash` | Model id (
|
|
241
|
+
| `CLEF_MODEL` | `clef-flash` | Model id (per-call `model` also accepted) |
|
|
190
242
|
| `CLEF_HOME` | `~/.cache/clef-mcp` | Cache/model home |
|
|
191
|
-
| `CLEF_RUNTIME` | `llama-cpp` |
|
|
192
|
-
| `CLEF_LOG_LEVEL` | `error` | `error` \| `warn` \| `info` \| `debug` (
|
|
193
|
-
| `CLEF_LLAMA_BIN` | – | Explicit
|
|
194
|
-
| `CLEF_LLAMA_RELEASE_TAG` | latest nightly | Pin the managed llama.cpp build
|
|
243
|
+
| `CLEF_RUNTIME` | `llama-cpp` | `llama-cpp` \| `mlx` |
|
|
244
|
+
| `CLEF_LOG_LEVEL` | `error` | `error` \| `warn` \| `info` \| `debug` (stderr only) |
|
|
245
|
+
| `CLEF_LLAMA_BIN` | – | Explicit `llama-server` binary path (llama-cpp runtime) |
|
|
246
|
+
| `CLEF_LLAMA_RELEASE_TAG` | latest nightly | Pin the managed llama.cpp build |
|
|
247
|
+
| `CLEF_LLAMA_BATCH` | `8192` | llama.cpp physical batch (multi-question requests) |
|
|
248
|
+
| `CLEF_MLX_UV` | `uv` on PATH | Explicit uv binary (MLX runtime) |
|
|
195
249
|
| `CLEF_MAX_QUESTIONS` | `64` | Max questions per call |
|
|
196
250
|
| `CLEF_MAX_STATE_BYTES` | `1048576` | Max serialized `state` size |
|
|
197
251
|
| `CLEF_MAX_INSTRUCTION_CHARS` | `10000` | Max chars per question instructions |
|
|
198
252
|
|
|
199
|
-
Runtime
|
|
200
|
-
|
|
201
|
-
## Storage layout
|
|
202
|
-
|
|
203
|
-
```text
|
|
204
|
-
~/.cache/clef-mcp/ # or $CLEF_HOME
|
|
205
|
-
├── models/
|
|
206
|
-
│ └── clef-flash/
|
|
207
|
-
│ ├── 4bit/ # Q4_K_M
|
|
208
|
-
│ │ ├── Clef-Flash-Q4_K_M.gguf
|
|
209
|
-
│ │ └── manifest.json # repo, revision, sha256, license
|
|
210
|
-
│ ├── 8bit/ # Q8_0
|
|
211
|
-
│ └── bf16/ # BF16
|
|
212
|
-
└── runtime/
|
|
213
|
-
└── llama.cpp/darwin-arm64/llama-server
|
|
214
|
-
```
|
|
253
|
+
Runtime resolution: `CLEF_LLAMA_BIN` → managed binary in `CLEF_HOME/runtime` → `llama-server` on `PATH`.
|
|
215
254
|
|
|
216
|
-
|
|
255
|
+
Storage: `CLEF_HOME/models/<model>/<quant>/` (model + `manifest.json` with repo/revision/sha256/license) and `CLEF_HOME/runtime/llama.cpp/`.
|
|
217
256
|
|
|
218
|
-
|
|
257
|
+
The model is downloaded from the pinned official GGUF conversion ([ggml-org/Clef-Flash-GGUF](https://huggingface.co/ggml-org/Clef-Flash-GGUF)) and **sha256-verified** against Hugging Face's content hash. It is never repackaged by clef-mcp. Also listed in the [official MCP Registry](https://registry.modelcontextprotocol.io) as `io.github.HighlyLoadedEgo/clef-mcp`.
|
|
219
258
|
|
|
220
|
-
|
|
259
|
+
## Error handling
|
|
221
260
|
|
|
222
261
|
```json
|
|
223
262
|
{
|
|
@@ -229,42 +268,28 @@ Tool errors are structured JSON with a machine-readable code:
|
|
|
229
268
|
}
|
|
230
269
|
```
|
|
231
270
|
|
|
232
|
-
Codes: `MODEL_NOT_INSTALLED`, `MODEL_LOAD_FAILED`, `RUNTIME_NOT_FOUND`, `RUNTIME_INIT_FAILED`, `RUNTIME_NOT_SUPPORTED`, `INVALID_INPUT`, `CLEF_INFERENCE_FAILED`, `UNSUPPORTED_PLATFORM`, `OUT_OF_MEMORY`, `CHECKSUM_MISMATCH`, `DOWNLOAD_FAILED`.
|
|
233
|
-
|
|
234
|
-
Input exceeding the model's 16k-token context is rejected by the runtime with a hint to reduce the state.
|
|
235
|
-
|
|
236
|
-
## Architecture
|
|
237
|
-
|
|
238
|
-
```text
|
|
239
|
-
MCP interface (stdio) → validation, structured errors
|
|
240
|
-
↓
|
|
241
|
-
Clef adapter → clef_decide ⇄ SystemOne wire format (request/response mapping)
|
|
242
|
-
↓
|
|
243
|
-
ClefRuntime (interface)
|
|
244
|
-
↓
|
|
245
|
-
LocalClefRuntime → llama.cpp llama-server subprocess
|
|
246
|
-
```
|
|
247
|
-
|
|
248
|
-
The `ClefRuntime` interface (`load / decide / unload / health`) isolates the engine: future MLX or remote runtimes plug in without changing the MCP API.
|
|
271
|
+
Codes: `MODEL_NOT_INSTALLED`, `MODEL_LOAD_FAILED`, `RUNTIME_NOT_FOUND`, `RUNTIME_INIT_FAILED`, `RUNTIME_NOT_SUPPORTED`, `INVALID_INPUT`, `CLEF_INFERENCE_FAILED`, `UNSUPPORTED_PLATFORM`, `OUT_OF_MEMORY`, `CHECKSUM_MISMATCH`, `DOWNLOAD_FAILED`. Input exceeding the 16k-token model context is rejected with a hint to reduce the state.
|
|
249
272
|
|
|
250
273
|
## Security & data handling
|
|
251
274
|
|
|
252
275
|
- No network servers, no telemetry, no accounts; everything runs locally.
|
|
253
|
-
- The model downloads only on an explicit `install`, over HTTPS,
|
|
276
|
+
- The model downloads only on an explicit `install`, over HTTPS, checksum-verified.
|
|
254
277
|
- `state` content is passed to the model as data; the server never executes or instruction-interprets it.
|
|
255
278
|
- Filesystem access is limited to `CLEF_HOME` (plus reading standard MCP client config paths in `doctor`).
|
|
256
|
-
- The managed runtime is the official llama.cpp
|
|
279
|
+
- The managed runtime is the official llama.cpp build; pin it with `CLEF_LLAMA_RELEASE_TAG`.
|
|
280
|
+
|
|
281
|
+
See [SECURITY.md](SECURITY.md) for the full policy.
|
|
257
282
|
|
|
258
283
|
## Development
|
|
259
284
|
|
|
260
285
|
```bash
|
|
261
286
|
npm install
|
|
262
287
|
npm run build
|
|
263
|
-
npm test # unit + integration (
|
|
264
|
-
npm run evals # needs an installed model;
|
|
288
|
+
npm test # unit + integration (fake llama-server, no model needed)
|
|
289
|
+
npm run evals # needs an installed model; exit code reflects pass rate
|
|
265
290
|
```
|
|
266
291
|
|
|
267
|
-
See [CONTRIBUTING.md](CONTRIBUTING.md) and [`tests/evals/dataset.jsonl`](tests/evals/dataset.jsonl)
|
|
292
|
+
See [CONTRIBUTING.md](CONTRIBUTING.md) and [`tests/evals/dataset.jsonl`](tests/evals/dataset.jsonl).
|
|
268
293
|
|
|
269
294
|
## License
|
|
270
295
|
|
package/assets/demo.svg
ADDED
|
@@ -0,0 +1,70 @@
|
|
|
1
|
+
<svg xmlns="http://www.w3.org/2000/svg" width="880" height="420" viewBox="0 0 880 420" font-family="ui-monospace, SFMono-Regular, Menlo, Consolas, monospace">
|
|
2
|
+
<defs>
|
|
3
|
+
<style>
|
|
4
|
+
.bg { fill: #0d1117; }
|
|
5
|
+
.chrome { fill: #161b22; }
|
|
6
|
+
.title { fill: #8b949e; font-size: 12px; }
|
|
7
|
+
.lbl { fill: #c9d1d9; font-size: 13px; }
|
|
8
|
+
.dim { fill: #8b949e; font-size: 12px; }
|
|
9
|
+
.val { font-size: 13px; font-weight: bold; }
|
|
10
|
+
.json { fill: #7ee787; font-size: 12px; }
|
|
11
|
+
.key { fill: #79c0ff; }
|
|
12
|
+
</style>
|
|
13
|
+
</defs>
|
|
14
|
+
|
|
15
|
+
<rect class="bg" width="880" height="420" rx="10"/>
|
|
16
|
+
<rect class="chrome" width="880" height="36" rx="10"/>
|
|
17
|
+
<rect class="chrome" y="18" width="880" height="18"/>
|
|
18
|
+
<circle cx="22" cy="18" r="6" fill="#ff5f57"/>
|
|
19
|
+
<circle cx="42" cy="18" r="6" fill="#febc2e"/>
|
|
20
|
+
<circle cx="62" cy="18" r="6" fill="#28c840"/>
|
|
21
|
+
<text class="title" x="440" y="23" text-anchor="middle">clef_decide — production incident</text>
|
|
22
|
+
|
|
23
|
+
<!-- request -->
|
|
24
|
+
<text class="dim" x="28" y="66">state: "prod returns 500 for 40% of requests after a deploy"</text>
|
|
25
|
+
<text class="dim" x="28" y="84">questions: action (choice) · urgent (noul)</text>
|
|
26
|
+
|
|
27
|
+
<!-- question 1: action -->
|
|
28
|
+
<text class="key" x="28" y="116">action</text>
|
|
29
|
+
<text class="dim" x="92" y="116">"What should the coding agent do next?" — choice</text>
|
|
30
|
+
|
|
31
|
+
<text class="lbl" x="28" y="142">rollback</text>
|
|
32
|
+
<rect x="120" y="132" width="527" height="12" rx="3" fill="#3fb950" opacity="0.25"/>
|
|
33
|
+
<rect x="120" y="132" width="497" height="12" rx="3" fill="#3fb950"/>
|
|
34
|
+
<text class="val" x="655" y="143" fill="#3fb950">93.9%</text>
|
|
35
|
+
|
|
36
|
+
<text class="lbl" x="28" y="170">hotfix</text>
|
|
37
|
+
<rect x="120" y="160" width="527" height="12" rx="3" fill="#d29922" opacity="0.25"/>
|
|
38
|
+
<rect x="120" y="160" width="17" height="12" rx="3" fill="#d29922"/>
|
|
39
|
+
<text class="val" x="655" y="171" fill="#d29922">3.2%</text>
|
|
40
|
+
|
|
41
|
+
<text class="lbl" x="28" y="198">investigate</text>
|
|
42
|
+
<rect x="120" y="188" width="527" height="12" rx="3" fill="#58a6ff" opacity="0.25"/>
|
|
43
|
+
<rect x="120" y="188" width="9" height="12" rx="3" fill="#58a6ff"/>
|
|
44
|
+
<text class="val" x="655" y="199" fill="#58a6ff">1.8%</text>
|
|
45
|
+
|
|
46
|
+
<text class="lbl" x="28" y="226">wait</text>
|
|
47
|
+
<rect x="120" y="216" width="527" height="12" rx="3" fill="#8b949e" opacity="0.25"/>
|
|
48
|
+
<rect x="120" y="216" width="6" height="12" rx="3" fill="#8b949e"/>
|
|
49
|
+
<text class="val" x="655" y="227" fill="#8b949e">1.2%</text>
|
|
50
|
+
|
|
51
|
+
<!-- question 2: urgent -->
|
|
52
|
+
<text class="key" x="28" y="262">urgent</text>
|
|
53
|
+
<text class="dim" x="92" y="262">"Roll back right now?" — noul</text>
|
|
54
|
+
|
|
55
|
+
<text class="lbl" x="28" y="288">true</text>
|
|
56
|
+
<rect x="120" y="278" width="527" height="12" rx="3" fill="#f85149" opacity="0.25"/>
|
|
57
|
+
<rect x="120" y="278" width="416" height="12" rx="3" fill="#f85149"/>
|
|
58
|
+
<text class="val" x="655" y="289" fill="#f85149">78.7%</text>
|
|
59
|
+
|
|
60
|
+
<text class="lbl" x="28" y="316">false</text>
|
|
61
|
+
<rect x="120" y="306" width="527" height="12" rx="3" fill="#8b949e" opacity="0.25"/>
|
|
62
|
+
<rect x="120" y="306" width="112" height="12" rx="3" fill="#8b949e"/>
|
|
63
|
+
<text class="val" x="655" y="317" fill="#8b949e">21.3%</text>
|
|
64
|
+
|
|
65
|
+
<!-- raw json note -->
|
|
66
|
+
<text class="json" x="28" y="356">{ "model": "clef-flash",</text>
|
|
67
|
+
<text class="json" x="28" y="374"> "<tspan class="key">decisions</tspan>": { "action": { "answer": { "rollback": 0.939, ... } },</text>
|
|
68
|
+
<text class="json" x="28" y="392"> "urgent": { "answer": { "true": 0.787, "false": 0.213 } } } }</text>
|
|
69
|
+
<text class="dim" x="28" y="410">one forward pass · no prose · no sampling · offline</text>
|
|
70
|
+
</svg>
|