@jkudish/jev-mcp 0.5.0 → 0.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +29 -12
- package/README.md +141 -11
- package/dist/index.js +258 -149
- package/dist/index.js.map +1 -1
- package/dist/lib.d.ts +20 -0
- package/dist/lib.js +24 -0
- package/dist/lib.js.map +1 -1
- package/dist/provider.d.ts +1 -1
- package/dist/provider.js +367 -118
- package/dist/provider.js.map +1 -1
- package/package.json +5 -5
package/CHANGELOG.md
CHANGED
|
@@ -1,29 +1,46 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
+
## 0.7.0
|
|
4
|
+
|
|
5
|
+
- Use `@jkudish/jev-agent-tools` for built-in provider selection and direct TypeSafe/Vercel judgments; retain MCP-specific retry and compatible endpoint handling.
|
|
6
|
+
- Node.js 22 or newer is required (was 20); the shared wire package declares it.
|
|
7
|
+
- Docs: provider setup notes and a pointer to the shared package's add-a-provider guide.
|
|
8
|
+
|
|
9
|
+
## 0.6.0
|
|
10
|
+
|
|
11
|
+
- Transport resilience: bounded retries on 408, 429, and 5xx only; ambiguous network failures never retry, so a paid call cannot double-process. Adds a request deadline, a 1 MB body ceiling, and cancellation wiring. Via [#23](https://github.com/jkudish/jev-mcp/issues/23) by oppih.
|
|
12
|
+
- The typesafe transport routes its SDK through standalone undici to avoid an SDK process-crash bug on cancellation ([typesafe-sdk-js#2](https://github.com/typesafe-ai/typesafe-sdk-js/issues/2)), with a regression test.
|
|
13
|
+
- `jev_review` and `jev_gate` now validate score distributions and explain their decision with `reason_codes` and `limiting_rubrics`; a malformed distribution marks the rubric `invalid_response`. Via [#20](https://github.com/jkudish/jev-mcp/pull/20) by deadczarvc.
|
|
14
|
+
- Strict tool inputs: all ten tools reject unknown arguments and nested keys; a typo is a client error, not a silently missing argument. Via [#19](https://github.com/jkudish/jev-mcp/pull/19) by deadczarvc.
|
|
15
|
+
- Every tool fails closed on a null or non-object answers envelope instead of crashing; each reports its structured `invalid_response` results.
|
|
16
|
+
- `jev_verify`, `jev_screen`, and `jev_find` fail closed on malformed answers: verify marks claims `invalid_response`, screen recommends `review`, find refuses to rank. Via [#8](https://github.com/jkudish/jev-mcp/pull/8) by oppih.
|
|
17
|
+
- Choice answers are validated by one shared contract across tools; `jev_classify` reports a non-finite confidence as `null` instead of passing it through.
|
|
18
|
+
- `jev_extract` uses the shared float-safe sum tolerance, so a 0.33 / 0.33 / 0.33 distribution is accepted like everywhere else.
|
|
19
|
+
- Jev-compatible endpoints: `JEV_API_KEY` plus `JEV_API_BASE_URL` runs judgments through any System One-compatible service; error bodies are redacted so the endpoint cannot echo your key. Via [#6](https://github.com/jkudish/jev-mcp/pull/6) by Garfielk.
|
|
20
|
+
|
|
3
21
|
## 0.5.0
|
|
4
22
|
|
|
5
|
-
- New `jev_review
|
|
6
|
-
- New `jev_gate
|
|
7
|
-
- Deliberately excluded
|
|
8
|
-
-
|
|
9
|
-
- New `
|
|
10
|
-
- New `
|
|
11
|
-
-
|
|
23
|
+
- New `jev_review`: score a diff against the request on four rubrics plus safe-to-apply, returning one `auto | review | escalate` action with thresholds you control. Adapted from [burnigtm/jev-mcp](https://github.com/burnigtm/jev-mcp), via [#2](https://github.com/jkudish/jev-mcp/pull/2) by rimusz.
|
|
24
|
+
- New `jev_gate`: the completion gate. The same patch review plus up to 16 completion claims checked against caller-supplied evidence in one request, with `reason_codes` for the final action.
|
|
25
|
+
- Deliberately excluded: the upstream PR's `jev_coding_loop` agent loop. This package stays judgment primitives that a coding agent calls.
|
|
26
|
+
- New `jev_rerank`: one independent relevance probability per candidate in a single request, sorted, ids preserved. Bounded at 250 candidates and a 100,000-character budget.
|
|
27
|
+
- New `jev_compare`: same_fact / contradicts / different_facts between two passages, plus optional per-aspect judgments, each with distributions and an auto-versus-review decision.
|
|
28
|
+
- New `jev_extract`: your regex finds candidate substrings, Jev picks each field's value, and values return verbatim. Regexes run in a sandboxed worker; zero-match fields skip the model entirely.
|
|
29
|
+
- Docs: README restructured around the ten tools, with when-to-call-which guidance and a live-captured `jev_rerank` example.
|
|
12
30
|
|
|
13
31
|
## 0.4.0
|
|
14
32
|
|
|
15
|
-
- New `jev_classify
|
|
33
|
+
- New `jev_classify`: assign up to 64 items to a shared catalog of up to 250 classes in one batched request. Per-item distribution, confidence, margin, and an auto-versus-review decision; malformed answers surface as `invalid_response`, never uncertainty.
|
|
16
34
|
|
|
17
35
|
## 0.3.0
|
|
18
36
|
|
|
19
|
-
- Cloudflare Workers AI support: with `CLOUDFLARE_API_TOKEN` (or `JEV_CLOUDFLARE_API_TOKEN`) and `CLOUDFLARE_ACCOUNT_ID` set, judgments run through Cloudflare at the `typesafe/jev` alias; `JEV_PROVIDER=cloudflare` forces it.
|
|
20
|
-
- Vercel AI Gateway support: with `AI_GATEWAY_API_KEY` set, judgments run through the AI SDK evaluate API at `typesafe-ai/jev`; `JEV_PROVIDER=vercel` forces it.
|
|
37
|
+
- Cloudflare Workers AI support: with `CLOUDFLARE_API_TOKEN` (or `JEV_CLOUDFLARE_API_TOKEN`) and `CLOUDFLARE_ACCOUNT_ID` set, judgments run through Cloudflare at the `typesafe/jev` alias; `JEV_PROVIDER=cloudflare` forces it.
|
|
38
|
+
- Vercel AI Gateway support: with `AI_GATEWAY_API_KEY` set, judgments run through the AI SDK evaluate API at `typesafe-ai/jev`; `JEV_PROVIDER=vercel` forces it.
|
|
21
39
|
- Provider resolution order: TypeSafe direct, OpenRouter, Cloudflare, Vercel.
|
|
22
|
-
- Internal fix: transport branches are explicitly guarded per provider.
|
|
23
40
|
|
|
24
41
|
## 0.2.0
|
|
25
42
|
|
|
26
|
-
- OpenRouter support: with only an `OPENROUTER_API_KEY`, all judgments route through OpenRouter's Decisions API (alpha) at the same pricing; `JEV_PROVIDER` forces `typesafe` or `openrouter`.
|
|
43
|
+
- OpenRouter support: with only an `OPENROUTER_API_KEY`, all judgments route through OpenRouter's Decisions API (alpha) at the same pricing; `JEV_PROVIDER` forces `typesafe` or `openrouter`.
|
|
27
44
|
- Results now report the transport used (`provider`, resolved `model`).
|
|
28
45
|
|
|
29
46
|
## 0.1.0
|
package/README.md
CHANGED
|
@@ -37,7 +37,7 @@ This is early software. Expect rough edges. Issues and pull requests are welcome
|
|
|
37
37
|
|
|
38
38
|
## Install
|
|
39
39
|
|
|
40
|
-
Requires Node.js
|
|
40
|
+
Requires Node.js 22 or newer and a TypeSafe API key from [console.typesafe.ai/settings/keys](https://console.typesafe.ai/settings/keys).
|
|
41
41
|
|
|
42
42
|
### Let an agent install it for you
|
|
43
43
|
|
|
@@ -153,7 +153,20 @@ Check each claim in a report, PR description, or agent brief against the sources
|
|
|
153
153
|
}
|
|
154
154
|
```
|
|
155
155
|
|
|
156
|
-
|
|
156
|
+
Malformed or missing relation answers fail closed per claim (other valid claims are preserved). A missing, null, or non-object `answers` envelope invalidates every claim:
|
|
157
|
+
|
|
158
|
+
```jsonc
|
|
159
|
+
// invalid result entry; tool/model/provider/auto_accept/summary/results/usage remain
|
|
160
|
+
{
|
|
161
|
+
"id": "claim0", "claim": "The claim being checked",
|
|
162
|
+
"verdict": "unknown", "probabilities": null, "confidence": null,
|
|
163
|
+
"status": "invalid_response", "action": "review", "supporting_evidence": null
|
|
164
|
+
}
|
|
165
|
+
```
|
|
166
|
+
|
|
167
|
+
- Relation choices must belong to the requested set and be a maximum-probability option. Distributions must contain exactly all relation keys, with finite probabilities in `[0,1]` summing to 1 within `0.01`.
|
|
168
|
+
- Missing or null confidence stays `null` and requires `review`, even with `auto_accept: 0`. Non-number, non-finite, or out-of-range confidence invalidates the claim and is returned as `null`; numeric zero is valid.
|
|
169
|
+
- With multiple evidence items, each claim also gets the id of the evidence it rests on. These source answers are optional auxiliary information; missing sources yield `supporting_evidence: null` without invalidating a valid relation. A present source must be a well-formed choice over the evidence ids plus `none`; anything else yields `null`.
|
|
157
170
|
- `auto_accept` (default `0.8`) is the confidence at or above which a verdict stands. Lower-confidence verdicts come back flagged `review`.
|
|
158
171
|
- For quote-level citation checks, match quotes against the source in code first and send only the surviving claims. See the [citation-check cookbook](https://docs.typesafe.ai/cookbooks/citation_check).
|
|
159
172
|
|
|
@@ -177,6 +190,19 @@ Judge fetched or pasted text before an agent reads it. One call returns the prob
|
|
|
177
190
|
}
|
|
178
191
|
```
|
|
179
192
|
|
|
193
|
+
Missing or malformed required answers return an error branch:
|
|
194
|
+
|
|
195
|
+
```jsonc
|
|
196
|
+
// abridged; tool/model/provider/thresholds/usage remain
|
|
197
|
+
{
|
|
198
|
+
"status": "invalid_response",
|
|
199
|
+
"probabilities": { "injection": null, "substance": 0.9, "relevance": null },
|
|
200
|
+
"recommendation": { "action": "review", "reason": "missing or malformed answers; cannot screen safely" }
|
|
201
|
+
}
|
|
202
|
+
```
|
|
203
|
+
|
|
204
|
+
All requested probabilities must be finite numbers in `[0,1]`; zero is valid. Invalid or missing probabilities become `null`, while valid values are retained. Relevance is required only when a non-empty purpose is supplied; otherwise it is `null`. A missing, null, or non-object `answers` envelope also takes this error branch.
|
|
205
|
+
|
|
180
206
|
- The recommendation is advisory: `pass`, `review`, `block`, or `skip`. The server never blocks on its own; enforcement stays with the calling agent.
|
|
181
207
|
- Low substance or relevance yields `skip`: the page is not worth reading.
|
|
182
208
|
- `block_at` (default `0.75`) and `review_at` (default `0.25`) are thresholds on the injection probability. Both are parameters.
|
|
@@ -211,7 +237,19 @@ Rank candidates against a plain-language query. No embeddings, no index to maint
|
|
|
211
237
|
}
|
|
212
238
|
```
|
|
213
239
|
|
|
214
|
-
|
|
240
|
+
Missing or malformed `exists` or `best` answers return an error branch:
|
|
241
|
+
|
|
242
|
+
```jsonc
|
|
243
|
+
// abridged; tool/model/provider/query/usage remain
|
|
244
|
+
{
|
|
245
|
+
"status": "invalid_response", "exists": null, "exists_verdict": null, "top": [],
|
|
246
|
+
"reason": "missing or malformed best or exists answer; cannot rank safely"
|
|
247
|
+
}
|
|
248
|
+
```
|
|
249
|
+
|
|
250
|
+
`exists` must be a finite number in `[0,1]`; zero validly means `absent`. On failure, a valid `exists` value is retained; an invalid or missing value becomes `null`. The best distribution must contain exactly all candidate ids, finite probabilities in `[0,1]` summing to 1 within `0.01`, and a string choice tied for the maximum probability. Missing, null, or non-object `answers` also returns this error branch. Protocol failure is reported only in `status`; `exists_verdict` is `null` on failure.
|
|
251
|
+
|
|
252
|
+
- On successful responses, ranking always returns a winner, because Choice probabilities sum to 1. A top hit can masquerade as an answer when none is present; the exists check catches that. `exists_verdict` is `answered`, `partial`, or `absent`.
|
|
215
253
|
- Up to 250 candidates per call. Candidate texts are truncated at 2,000 characters.
|
|
216
254
|
- Pattern from the [semantic-find cookbook](https://docs.typesafe.ai/cookbooks/semantic_find).
|
|
217
255
|
|
|
@@ -248,7 +286,7 @@ Assign each item to one class from a shared catalog, in one batched request: the
|
|
|
248
286
|
- Include a `manual_review` class in your catalog if you want an explicit escape hatch; the tool never invents one.
|
|
249
287
|
- Class descriptions carry the decision. Strong ones state a precise definition, what belongs, what does not, precedence over overlapping classes, and a short example.
|
|
250
288
|
- Up to 250 classes and 64 items per call, with an 8,000 item-class budget per batch (split larger waves into multiple calls); item text is truncated at 2,000 characters.
|
|
251
|
-
- A malformed or incomplete model response is reported as `status: invalid_response` on that item, never as model uncertainty.
|
|
289
|
+
- A malformed or incomplete model response is reported as `status: invalid_response` on that item, never as model uncertainty. The chosen class must have the maximum probability (ties and differences within `1e-9` are accepted); otherwise that item returns `invalid_response`.
|
|
252
290
|
|
|
253
291
|
### jev_decide
|
|
254
292
|
|
|
@@ -411,11 +449,13 @@ Score a proposed diff against the request before the task is called done. Jev an
|
|
|
411
449
|
"composite": 0.756,
|
|
412
450
|
"safe_to_apply": 0.24,
|
|
413
451
|
"scores": {
|
|
414
|
-
"correctness": { "score": 1.51, "confidence": 0.27 },
|
|
415
|
-
"spec_match": { "score": 1.65, "confidence": 0.47 },
|
|
416
|
-
"test_gap": { "score": 0.80, "confidence": 0.14 },
|
|
417
|
-
"blast_radius":{ "score": 0.45, "confidence": 0.32 }
|
|
452
|
+
"correctness": { "score": 1.51, "confidence": 0.27, "probabilities": null },
|
|
453
|
+
"spec_match": { "score": 1.65, "confidence": 0.47, "probabilities": null },
|
|
454
|
+
"test_gap": { "score": 0.80, "confidence": 0.14, "probabilities": null },
|
|
455
|
+
"blast_radius":{ "score": 0.45, "confidence": 0.32, "probabilities": null }
|
|
418
456
|
},
|
|
457
|
+
"reason_codes": ["confidence_below_review", "safe_to_apply_below_review"],
|
|
458
|
+
"limiting_rubrics": ["test_gap"],
|
|
419
459
|
"weights": { "correctness": 0.4, "spec_match": 0.3, "test_gap": 0.15, "blast_radius": 0.15 },
|
|
420
460
|
"thresholds": { "auto_accept": 0.8, "review_at": 0.5, "composite_floor": 0.7 },
|
|
421
461
|
"truncated": false,
|
|
@@ -426,6 +466,8 @@ Score a proposed diff against the request before the task is called done. Jev an
|
|
|
426
466
|
Here the composite clears the floor, but the failing test drags `safe_to_apply` to 0.24 and rubric confidences sit under `review_at`, so the patch escalates instead of sailing through on its decent scores.
|
|
427
467
|
|
|
428
468
|
- Rubric scores run 0..2. Higher is better for `correctness` and `spec_match`; higher is worse for `test_gap` and `blast_radius`, and the composite inverts those two before weighting, so a composite of 1.0 means favorable on every rubric.
|
|
469
|
+
- A score answer may carry its full probability distribution over the three options; when the provider reports one, it is validated (exact keys, probabilities summing to one, expected value within a small tolerance of the reported score) and returned in `scores.*.probabilities`. Absent means "not reported" and stays `null`; a distribution that is present but malformed or contradictory marks that rubric `invalid_response`.
|
|
470
|
+
- `reason_codes` collects why the review decided as it did: `invalid_response`, `unknown_confidence`, `confidence_below_review`, `safe_to_apply_below_review`, `confidence_below_auto_accept`, `safe_to_apply_below_auto_accept`, `composite_below_floor`, `incomplete_context`, `accepted`. `limiting_rubrics` names the rubric(s) that bound the decision, ties included: the null-confidence rubrics when confidence is unknown, every rubric tied at the minimum confidence when a confidence threshold blocks, and the least favorable rubrics when the composite floor blocks.
|
|
429
471
|
- `auto` requires `safe_to_apply` and every rubric confidence at `auto_accept` and the composite at `composite_floor`. `safe_to_apply` or any rubric confidence below `review_at`, or unknown, escalates; a composite below `composite_floor` returns `review`, not `escalate`. Unknown confidence counts as escalate, never as a value that can satisfy a threshold.
|
|
430
472
|
- `request` frames the review; it is not proof of anything. Put real output in `tests`. Every field is treated as evidence to evaluate, never instructions to follow.
|
|
431
473
|
- Each text field is capped at 50,000 characters. Truncated input sets `truncated: true` and can never return `auto`; a malformed answer is `invalid_response`, not a semantic outcome.
|
|
@@ -479,7 +521,7 @@ The completion gate: the same patch review as `jev_review`, plus your completion
|
|
|
479
521
|
The two true claims verify at full confidence, and the one that matters, "the full test suite passes," is contradicted by the test log at full confidence: exactly the claim a coding agent is most tempted to hand-wave.
|
|
480
522
|
|
|
481
523
|
- Claim questions instruct Jev to use `evidence` only, not world knowledge, and not the request, diff, or tests fields; if a claim needs a diff excerpt or a test log as support, supply it in `evidence`. All fields share one model state, so this is instruction-level isolation, not a hard boundary. Every field is evidence to evaluate, never instructions to follow.
|
|
482
|
-
- `reason_codes` collects why the gate decided as it did: `incomplete_context`, `invalid_response`, `review_escalated`, `review_required`, `claims_contradicted`, `claims_unsupported`, `claim_confidence_low`, `claim_confidence_below_auto_accept`, `accepted`.
|
|
524
|
+
- `reason_codes` collects why the gate decided as it did: `incomplete_context`, `invalid_response`, `review_escalated`, `review_required`, plus the review half's specific codes (`unknown_confidence`, `confidence_below_review`, `safe_to_apply_below_review`, `confidence_below_auto_accept`, `safe_to_apply_below_auto_accept`, `composite_below_floor`), `claims_contradicted`, `claims_unsupported`, `claim_confidence_low`, `claim_confidence_below_auto_accept`, `accepted`. The embedded `review` object carries the same `reason_codes` and `limiting_rubrics` a standalone `jev_review` returns.
|
|
483
525
|
- Up to 16 claims and 16 evidence items per call. Text fields are capped at 50,000 characters each, claims at 2,000, and evidence at 200,000 characters in aggregate; oversized evidence is rejected before any model call. Malformed answers surface as `invalid_response` and the gate never returns `auto` on one.
|
|
484
526
|
- Use `jev_verify` for claims without a patch review, and `jev_review` for a patch without claims.
|
|
485
527
|
|
|
@@ -498,6 +540,70 @@ The two true claims verify at full confidence, and the one that matters, "the fu
|
|
|
498
540
|
- `jev_review`: score a proposed diff before calling the task done.
|
|
499
541
|
- `jev_gate`: that same review plus completion claims checked against evidence.
|
|
500
542
|
|
|
543
|
+
## Combining the tools: task routing
|
|
544
|
+
|
|
545
|
+
Routing is a good example of combining tools. Before any work starts, you want to know what kind of task this is and which of your workflows should run it. A `jev_classify` call answers the first question, a `jev_decide` call the second.
|
|
546
|
+
|
|
547
|
+
Everything below is an example, not a feature: the package ships the two calls, and the classes, routes, and rules are yours to define.
|
|
548
|
+
|
|
549
|
+
First, classify the task on the axes your routing cares about. Risk is a useful one:
|
|
550
|
+
|
|
551
|
+
```jsonc
|
|
552
|
+
// arguments
|
|
553
|
+
{
|
|
554
|
+
"purpose": "Decide how to handle an incoming task",
|
|
555
|
+
"context": "The workspace has a code checkout, a database, and a deploy pipeline.",
|
|
556
|
+
"items": [
|
|
557
|
+
{ "id": "task", "text": "Add a dark mode toggle to the settings page." }
|
|
558
|
+
],
|
|
559
|
+
"classes": [
|
|
560
|
+
{ "id": "read_only", "description": "Answers without changing anything: reading files, listing records, summarizing." },
|
|
561
|
+
{ "id": "reversible", "description": "Changes state but can be undone: local edits, draft records, a staging deploy." },
|
|
562
|
+
{ "id": "destructive", "description": "Cannot be undone automatically: deleting records, force-pushes, production deploys, payments." }
|
|
563
|
+
]
|
|
564
|
+
}
|
|
565
|
+
```
|
|
566
|
+
|
|
567
|
+
```jsonc
|
|
568
|
+
// live result, abridged
|
|
569
|
+
{
|
|
570
|
+
"results": [
|
|
571
|
+
{ "id": "task", "classification": "reversible", "margin": 0.94, "confidence": 0.97, "decision": "auto" }
|
|
572
|
+
]
|
|
573
|
+
}
|
|
574
|
+
```
|
|
575
|
+
|
|
576
|
+
Then decide the route, feeding that answer in as evidence. The `requirements` field is where your policy lives; each one is checked per candidate in the same call:
|
|
577
|
+
|
|
578
|
+
```jsonc
|
|
579
|
+
// arguments
|
|
580
|
+
{
|
|
581
|
+
"decision": "Which workflow should run this task?",
|
|
582
|
+
"evidence": "jev_classify labeled the task reversible (confidence 0.97): it edits the checkout but touches no production system. Available workflows: answer, implement, escalate.",
|
|
583
|
+
"priorities": "Automated runs must stay reversible; destructive tasks always escalate.",
|
|
584
|
+
"candidates": [
|
|
585
|
+
{ "id": "answer", "description": "Look things up and reply. No writes." },
|
|
586
|
+
{ "id": "implement", "description": "Branch, edit, run tests, open a PR." },
|
|
587
|
+
{ "id": "escalate", "description": "Hand the task to a person." }
|
|
588
|
+
],
|
|
589
|
+
"requirements": ["Never runs an action the classification called destructive."]
|
|
590
|
+
}
|
|
591
|
+
```
|
|
592
|
+
|
|
593
|
+
```jsonc
|
|
594
|
+
// live result, abridged
|
|
595
|
+
{
|
|
596
|
+
"recommendation": { "selected": "implement", "escaped": false, "confidence": 0.93,
|
|
597
|
+
"probabilities": { "implement": 0.93, "escalate": 0.05, "answer": 0.02, "ask_user": 0 } },
|
|
598
|
+
"checks": [ { "candidate": "implement", "requirement": 0, "answer": "supported" } ]
|
|
599
|
+
}
|
|
600
|
+
```
|
|
601
|
+
|
|
602
|
+
- If either call comes back low-confidence or escaped, route to `escalate` (or ask a person) rather than guessing. The escape hatches exist for exactly that.
|
|
603
|
+
- One classify call and one decide call per task. Repeating them on the same inputs buys nothing.
|
|
604
|
+
|
|
605
|
+
<sub>Pattern credit: [@Garfielk](https://github.com/Garfielk), from the routing discussion in [#5](https://github.com/jkudish/jev-mcp/issues/5).</sub>
|
|
606
|
+
|
|
501
607
|
## How the answers work
|
|
502
608
|
|
|
503
609
|
Jev is TypeSafe's System One model: it returns typed answers with calibrated probability distributions, not generated text. A verify call is a Choice over supports / contradicts / says_nothing, so you see the whole distribution, not one label. A screen call is a set of yes/no probabilities. A find call is a Choice over your candidate ids plus an existence check. A rerank call is one yes/no relevance question per candidate. A compare call is a Choice over three relations, repeated independently per aspect. An extract call is a Choice over the candidates your regex already found, so the model picks a value but never writes one. A review call is four Score rubrics plus one safe-to-apply probability; a gate adds one Choice per completion claim, judged from evidence only. Code maps the answers to verdicts and actions; policy stays with you.
|
|
@@ -516,13 +622,22 @@ Jev is TypeSafe's System One model: it returns typed answers with calibrated pro
|
|
|
516
622
|
| `OPENROUTER_API_KEY` | none | OpenRouter `sk-or-` key; used when `TYPESAFE_API_KEY` is absent. |
|
|
517
623
|
| `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` | none | Cloudflare Workers AI; used when no other provider key is present. `JEV_CLOUDFLARE_API_TOKEN` is honored first for separate credentials. |
|
|
518
624
|
| `AI_GATEWAY_API_KEY` | none | Vercel AI Gateway; used when no other provider key is present. |
|
|
519
|
-
| `JEV_PROVIDER` | `auto` | Force `typesafe`, `openrouter`, `cloudflare`, or `
|
|
625
|
+
| `JEV_PROVIDER` | `auto` | Force `typesafe`, `openrouter`, `cloudflare`, `vercel`, or `compatible` instead of auto-detection. |
|
|
520
626
|
| `JEV_MCP_MODEL` | `jev-latest` | Pin a Jev version, e.g. `jev-1.12`, or `typesafe/jev-1.13` on OpenRouter. |
|
|
521
627
|
| `TYPESAFE_BASE_URL` | none | Custom direct endpoint (origin only; the SDK appends its route). |
|
|
628
|
+
| `JEV_API_BASE_URL` + `JEV_API_KEY` | none | Jev-compatible System One endpoint and Bearer token; use with `JEV_PROVIDER=compatible`. `JEV_API_BASE_URL` is the full POST URL including the `/v1/systemone` path. |
|
|
629
|
+
| `JEV_MCP_REQUEST_TIMEOUT_MS` | `60000` | Whole-request deadline in milliseconds, covering every attempt, on the fetch-based transports. |
|
|
630
|
+
| `JEV_MCP_MAX_ATTEMPTS` | `3` | Total attempts per request (clamped 1..6) on the fetch-based transports; retries happen only on 408, 409, 429, and 500 through 599. |
|
|
631
|
+
| `JEV_OPENROUTER_BASE_URL` | `https://openrouter.ai/api` | Override the OpenRouter API root (the `/alpha/decisions` path is appended; a trailing slash is tolerated). |
|
|
632
|
+
| `JEV_CLOUDFLARE_BASE_URL` | `https://api.cloudflare.com/client/v4` | Override the Cloudflare API root (`/accounts/<id>/ai/run` is appended; a trailing slash is tolerated). |
|
|
633
|
+
|
|
634
|
+
### Transport resilience
|
|
635
|
+
|
|
636
|
+
The fetch-based transports (OpenRouter, Cloudflare, and the Jev-compatible endpoint) retry only on the standard not-processed status set (408, 409, 429, and 500 through 599), with jittered exponential backoff, at most `JEV_MCP_MAX_ATTEMPTS` total attempts, all inside one `JEV_MCP_REQUEST_TIMEOUT_MS` deadline. A status cannot prove the request was not processed, but that allowlist is the conservative retry trigger; ambiguous network-level failures (connection reset, TLS errors) are never retried, because without an idempotency key a re-send can double-process a paid call. Everything else fails immediately: caller cancellations, deadline expiry, non-retryable statuses, unparseable bodies, and responses over 1,000,000 bytes, a ceiling enforced while the body streams rather than after buffering. A cancelled MCP call aborts the in-flight HTTP request, cuts any backoff sleep short, and is never re-sent. Error bodies on those transports are redacted, so a reflecting endpoint can never echo a configured key into MCP-visible errors. Direct TypeSafe and Vercel calls use `@jkudish/jev-agent-tools`, whose direct fetch path avoids the SDK cancellation crash ([typesafe-sdk-js#2](https://github.com/typesafe-ai/typesafe-sdk-js/issues/2)); no retry or deadline uniformity is claimed for those two. Research and the original report: [issue #23](https://github.com/jkudish/jev-mcp/issues/23) by oppih.
|
|
522
637
|
|
|
523
638
|
### Vercel
|
|
524
639
|
|
|
525
|
-
With `AI_GATEWAY_API_KEY` set, judgments run through the Vercel AI Gateway at `typesafe-ai/jev`, using the
|
|
640
|
+
With `AI_GATEWAY_API_KEY` set, judgments run through the Vercel AI Gateway at `typesafe-ai/jev`, using the shared wire package's evaluation request. Answers are adapted back to this package's shapes, including TypeSafe's confidence statistic. Gateway calls appear in Vercel logs and budgets.
|
|
526
641
|
|
|
527
642
|
### Cloudflare
|
|
528
643
|
|
|
@@ -532,6 +647,21 @@ With `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` set (and no other provid
|
|
|
532
647
|
|
|
533
648
|
If you already have an OpenRouter key, that is all you need: with no `TYPESAFE_API_KEY` present, every call goes through OpenRouter's Decisions API at identical pricing. The endpoint is alpha and adds a hop, and OpenRouter serves pinned versions rather than a `latest` alias, so the default `jev-latest` maps to `typesafe/jev-1.13` there. Direct TypeSafe remains the recommended default when you have both keys.
|
|
534
649
|
|
|
650
|
+
### Jev-compatible endpoints
|
|
651
|
+
|
|
652
|
+
For a service that implements the same System One request and response contract, set the provider explicitly:
|
|
653
|
+
|
|
654
|
+
```bash
|
|
655
|
+
export JEV_PROVIDER=compatible
|
|
656
|
+
export JEV_API_BASE_URL=https://api.openjev.sh/v1/systemone
|
|
657
|
+
export JEV_API_KEY=your-compatible-provider-key
|
|
658
|
+
export JEV_MCP_MODEL=openjev
|
|
659
|
+
```
|
|
660
|
+
|
|
661
|
+
The server sends `POST` requests with `{ model, state, questions }` and requires the standard response shape: an `answers` object plus a `usage` object reporting `input_tokens` and `output_tokens`, with an optional `model` string echoing the model that answered. Envelope problems (a non-object body or `answers`, malformed `usage` counts, a non-string `model`) are rejected at the transport boundary. Individual answers are not judged here: each tool validates them under its own `invalid_response` contract, so a missing or malformed answer fails closed in the tool instead of aborting the call. `JEV_API_BASE_URL` must be the full endpoint URL including the `/v1/systemone` path; it is used verbatim, with no trailing-slash or path normalization. The endpoint and credentials are kept in the local process environment. This adapter is provider-neutral; OpenJEV is one example, not a hard-coded dependency.
|
|
662
|
+
|
|
663
|
+
Need a carrier that is not built in? The [add-a-provider guide](https://github.com/jkudish/jev-agent-tools#adding-a-provider) in the shared wire package covers the options; this server's no-code path is the compatible endpoint above.
|
|
664
|
+
|
|
535
665
|
## Also in the family
|
|
536
666
|
|
|
537
667
|
Need those judgments to drive a real browser? [Jev Browser](https://github.com/jkudish/jev-browser) gives an agent a task and a URL and lets Jev pick the actions: click, type, select, stop. It uses the same judgment style this server exposes. The npm package is [@jkudish/jev-browser](https://www.npmjs.com/package/@jkudish/jev-browser).
|