@jkudish/jev-mcp 0.5.0 → 0.7.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,29 +1,46 @@
1
1
  # Changelog
2
2
 
3
+ ## 0.7.0
4
+
5
+ - Use `@jkudish/jev-agent-tools` for built-in provider selection and direct TypeSafe/Vercel judgments; retain MCP-specific retry and compatible endpoint handling.
6
+ - Node.js 22 or newer is required (was 20); the shared wire package declares it.
7
+ - Docs: provider setup notes and a pointer to the shared package's add-a-provider guide.
8
+
9
+ ## 0.6.0
10
+
11
+ - Transport resilience: bounded retries on 408, 429, and 5xx only; ambiguous network failures never retry, so a paid call cannot double-process. Adds a request deadline, a 1 MB body ceiling, and cancellation wiring. Via [#23](https://github.com/jkudish/jev-mcp/issues/23) by oppih.
12
+ - The typesafe transport routes its SDK through standalone undici to avoid an SDK process-crash bug on cancellation ([typesafe-sdk-js#2](https://github.com/typesafe-ai/typesafe-sdk-js/issues/2)), with a regression test.
13
+ - `jev_review` and `jev_gate` now validate score distributions and explain their decision with `reason_codes` and `limiting_rubrics`; a malformed distribution marks the rubric `invalid_response`. Via [#20](https://github.com/jkudish/jev-mcp/pull/20) by deadczarvc.
14
+ - Strict tool inputs: all ten tools reject unknown arguments and nested keys; a typo is a client error, not a silently missing argument. Via [#19](https://github.com/jkudish/jev-mcp/pull/19) by deadczarvc.
15
+ - Every tool fails closed on a null or non-object answers envelope instead of crashing; each reports its structured `invalid_response` results.
16
+ - `jev_verify`, `jev_screen`, and `jev_find` fail closed on malformed answers: verify marks claims `invalid_response`, screen recommends `review`, find refuses to rank. Via [#8](https://github.com/jkudish/jev-mcp/pull/8) by oppih.
17
+ - Choice answers are validated by one shared contract across tools; `jev_classify` reports a non-finite confidence as `null` instead of passing it through.
18
+ - `jev_extract` uses the shared float-safe sum tolerance, so a 0.33 / 0.33 / 0.33 distribution is accepted like everywhere else.
19
+ - Jev-compatible endpoints: `JEV_API_KEY` plus `JEV_API_BASE_URL` runs judgments through any System One-compatible service; error bodies are redacted so the endpoint cannot echo your key. Via [#6](https://github.com/jkudish/jev-mcp/pull/6) by Garfielk.
20
+
3
21
  ## 0.5.0
4
22
 
5
- - New `jev_review` tool: score a proposed diff against the request before the task is called done. Four 0..2 Score rubrics (correctness, spec match, test gap, blast radius, the last two inverted in the composite) plus a safe-to-apply probability, combined into a weighted composite (0.4 / 0.3 / 0.15 / 0.15) and one `auto | review | escalate` action. Auto requires safe_to_apply and every rubric confidence at `auto_accept` (default 0.8) and the composite at `composite_floor` (default 0.7); below `review_at` escalates. Unknown confidence escalates and can never satisfy a threshold, even a zero one. Text fields capped at 50,000 characters; truncated or malformed input never returns auto. Question design adapted from [burnigtm/jev-mcp](https://github.com/burnigtm/jev-mcp) (MIT) via [PR #2](https://github.com/jkudish/jev-mcp/pull/2) by rimusz, with deliberate divergences: caps raised from 2,000 characters so auto is reachable on real diffs, thresholds parameterized instead of a hardcoded safe_to_apply escalation, strict per-answer validation under the package's invalid_response contract, and anti-injection framing (every field is evidence to evaluate, never instructions to follow).
6
- - New `jev_gate` tool: the completion gate. The same patch review plus up to 16 completion claims verified against caller-supplied evidence in one request, each claim a Choice over verified / contradicted / unsupported, with claim questions instructed to use evidence only (the request, diff, and tests fields are assertions to check, never proof; all fields share one model state, so the isolation is instructional). Evidence is capped at 16 items and 200,000 characters in aggregate, rejected before any model call. The final action is the worst of the review and every claim; `reason_codes` records why (`incomplete_context`, `invalid_response`, `review_escalated`, `review_required`, `claims_contradicted`, `claims_unsupported`, `claim_confidence_low`, `claim_confidence_below_auto_accept`, `accepted`). Same credits and divergences as `jev_review`.
7
- - Deliberately excluded from that PR: its `jev_coding_loop` tool, an agent loop that runs the review gate repeatedly. This package stays judgment primitives that a coding agent calls; it does not become one.
8
- - Docs: README restructured around the ten tools (capability bullets, collapsible per-client install sections, a When to call which tool section), the `jev_rerank` example replaced with a live-captured semantic-ranking example that shows where candidate text comes from, and guidance added for ranking whole documents by chunking with distinct ids.
9
- - New `jev_rerank` tool: one independent relevance probability per candidate in a single request, returned sorted with scores. Candidate ids preserved verbatim; any malformed answer marks the whole ranking `invalid_response` instead of sorting a missing score as zero. TypeSafe's rerank cookbook reports that on the CLERC benchmark this pattern lifted top-1 from 5% to 18% and top-10 from 38% to 62%. Bounded at 250 candidates and a 100,000-character aggregate budget.
10
- - New `jev_compare` tool: overall same_fact / contradicts / different_facts relation between two passages, plus optional per-aspect judgments, each an independent Choice in the same request with aspect-specific wording for the third outcome, with distributions, confidence, an argmax guard, and an auto-versus-review decision.
11
- - New `jev_extract` tool: your regex finds candidate substrings, Jev picks which candidate is each field's value, and the value returns verbatim. Regexes run in a sandboxed worker with a 1-second deadline, so pathological patterns cannot hang the server. Complete zero-match fields are omitted from the model request (`not_found`); if no field reaches the model, no API call is made, and a confident `none_of_them` is a model-judged negative. An incomplete candidate universe (matches beyond the 20-per-field cap, or matches skipped for exceeding 2,000 characters) forces every otherwise-valid positive or `none_of_them` result to `review` (malformed answers remain `invalid_response`), with explicit flags rather than silent truncation; invalid patterns and timeouts report `invalid_pattern` without failing the call. Bounded at 32 fields, a 50,000-character document, and a separate 50,000-character aggregate candidate-match budget.
23
+ - New `jev_review`: score a diff against the request on four rubrics plus safe-to-apply, returning one `auto | review | escalate` action with thresholds you control. Adapted from [burnigtm/jev-mcp](https://github.com/burnigtm/jev-mcp), via [#2](https://github.com/jkudish/jev-mcp/pull/2) by rimusz.
24
+ - New `jev_gate`: the completion gate. The same patch review plus up to 16 completion claims checked against caller-supplied evidence in one request, with `reason_codes` for the final action.
25
+ - Deliberately excluded: the upstream PR's `jev_coding_loop` agent loop. This package stays judgment primitives that a coding agent calls.
26
+ - New `jev_rerank`: one independent relevance probability per candidate in a single request, sorted, ids preserved. Bounded at 250 candidates and a 100,000-character budget.
27
+ - New `jev_compare`: same_fact / contradicts / different_facts between two passages, plus optional per-aspect judgments, each with distributions and an auto-versus-review decision.
28
+ - New `jev_extract`: your regex finds candidate substrings, Jev picks each field's value, and values return verbatim. Regexes run in a sandboxed worker; zero-match fields skip the model entirely.
29
+ - Docs: README restructured around the ten tools, with when-to-call-which guidance and a live-captured `jev_rerank` example.
12
30
 
13
31
  ## 0.4.0
14
32
 
15
- - New `jev_classify` tool: assign up to 64 items to a shared catalog of up to 250 classes in one batched request (catalog sent once, one Choice per item). Per-item distribution, confidence, winner margin, and an auto-versus-review decision gated on both top probability (default 0.85) and margin (default 0.5). Caller IDs are preserved verbatim (opaque internal keys on the wire); the catalog is sent once in shared state with each item in its own question; malformed responses surface as invalid_response rather than uncertainty; batches are capped at an 8,000 item-class budget.
33
+ - New `jev_classify`: assign up to 64 items to a shared catalog of up to 250 classes in one batched request. Per-item distribution, confidence, margin, and an auto-versus-review decision; malformed answers surface as `invalid_response`, never uncertainty.
16
34
 
17
35
  ## 0.3.0
18
36
 
19
- - Cloudflare Workers AI support: with `CLOUDFLARE_API_TOKEN` (or `JEV_CLOUDFLARE_API_TOKEN`) and `CLOUDFLARE_ACCOUNT_ID` set, judgments run through Cloudflare at the `typesafe/jev` alias; `JEV_PROVIDER=cloudflare` forces it. Usage tokens are reported on every call.
20
- - Vercel AI Gateway support: with `AI_GATEWAY_API_KEY` set, judgments run through the AI SDK evaluate API at `typesafe-ai/jev`; `JEV_PROVIDER=vercel` forces it. Noul, choice, and score answers are adapted back to this package's shapes, with TypeSafe confidence included.
37
+ - Cloudflare Workers AI support: with `CLOUDFLARE_API_TOKEN` (or `JEV_CLOUDFLARE_API_TOKEN`) and `CLOUDFLARE_ACCOUNT_ID` set, judgments run through Cloudflare at the `typesafe/jev` alias; `JEV_PROVIDER=cloudflare` forces it.
38
+ - Vercel AI Gateway support: with `AI_GATEWAY_API_KEY` set, judgments run through the AI SDK evaluate API at `typesafe-ai/jev`; `JEV_PROVIDER=vercel` forces it.
21
39
  - Provider resolution order: TypeSafe direct, OpenRouter, Cloudflare, Vercel.
22
- - Internal fix: transport branches are explicitly guarded per provider.
23
40
 
24
41
  ## 0.2.0
25
42
 
26
- - OpenRouter support: with only an `OPENROUTER_API_KEY`, all judgments route through OpenRouter's Decisions API (alpha) at the same pricing; `JEV_PROVIDER` forces `typesafe` or `openrouter`. `jev-latest` maps to `typesafe/jev-1.13` there.
43
+ - OpenRouter support: with only an `OPENROUTER_API_KEY`, all judgments route through OpenRouter's Decisions API (alpha) at the same pricing; `JEV_PROVIDER` forces `typesafe` or `openrouter`.
27
44
  - Results now report the transport used (`provider`, resolved `model`).
28
45
 
29
46
  ## 0.1.0
package/README.md CHANGED
@@ -37,7 +37,7 @@ This is early software. Expect rough edges. Issues and pull requests are welcome
37
37
 
38
38
  ## Install
39
39
 
40
- Requires Node.js 20 or newer and a TypeSafe API key from [console.typesafe.ai/settings/keys](https://console.typesafe.ai/settings/keys).
40
+ Requires Node.js 22 or newer and a TypeSafe API key from [console.typesafe.ai/settings/keys](https://console.typesafe.ai/settings/keys).
41
41
 
42
42
  ### Let an agent install it for you
43
43
 
@@ -153,7 +153,20 @@ Check each claim in a report, PR description, or agent brief against the sources
153
153
  }
154
154
  ```
155
155
 
156
- - With multiple evidence items, each claim also gets the id of the evidence it rests on.
156
+ Malformed or missing relation answers fail closed per claim (other valid claims are preserved). A missing, null, or non-object `answers` envelope invalidates every claim:
157
+
158
+ ```jsonc
159
+ // invalid result entry; tool/model/provider/auto_accept/summary/results/usage remain
160
+ {
161
+ "id": "claim0", "claim": "The claim being checked",
162
+ "verdict": "unknown", "probabilities": null, "confidence": null,
163
+ "status": "invalid_response", "action": "review", "supporting_evidence": null
164
+ }
165
+ ```
166
+
167
+ - Relation choices must belong to the requested set and be a maximum-probability option. Distributions must contain exactly all relation keys, with finite probabilities in `[0,1]` summing to 1 within `0.01`.
168
+ - Missing or null confidence stays `null` and requires `review`, even with `auto_accept: 0`. Non-number, non-finite, or out-of-range confidence invalidates the claim and is returned as `null`; numeric zero is valid.
169
+ - With multiple evidence items, each claim also gets the id of the evidence it rests on. These source answers are optional auxiliary information; missing sources yield `supporting_evidence: null` without invalidating a valid relation. A present source must be a well-formed choice over the evidence ids plus `none`; anything else yields `null`.
157
170
  - `auto_accept` (default `0.8`) is the confidence at or above which a verdict stands. Lower-confidence verdicts come back flagged `review`.
158
171
  - For quote-level citation checks, match quotes against the source in code first and send only the surviving claims. See the [citation-check cookbook](https://docs.typesafe.ai/cookbooks/citation_check).
159
172
 
@@ -177,6 +190,19 @@ Judge fetched or pasted text before an agent reads it. One call returns the prob
177
190
  }
178
191
  ```
179
192
 
193
+ Missing or malformed required answers return an error branch:
194
+
195
+ ```jsonc
196
+ // abridged; tool/model/provider/thresholds/usage remain
197
+ {
198
+ "status": "invalid_response",
199
+ "probabilities": { "injection": null, "substance": 0.9, "relevance": null },
200
+ "recommendation": { "action": "review", "reason": "missing or malformed answers; cannot screen safely" }
201
+ }
202
+ ```
203
+
204
+ All requested probabilities must be finite numbers in `[0,1]`; zero is valid. Invalid or missing probabilities become `null`, while valid values are retained. Relevance is required only when a non-empty purpose is supplied; otherwise it is `null`. A missing, null, or non-object `answers` envelope also takes this error branch.
205
+
180
206
  - The recommendation is advisory: `pass`, `review`, `block`, or `skip`. The server never blocks on its own; enforcement stays with the calling agent.
181
207
  - Low substance or relevance yields `skip`: the page is not worth reading.
182
208
  - `block_at` (default `0.75`) and `review_at` (default `0.25`) are thresholds on the injection probability. Both are parameters.
@@ -211,7 +237,19 @@ Rank candidates against a plain-language query. No embeddings, no index to maint
211
237
  }
212
238
  ```
213
239
 
214
- - Ranking always returns a winner, because Choice probabilities sum to 1. A top hit can masquerade as an answer when none is present; the exists check catches that. `exists_verdict` is `answered`, `partial`, or `absent`.
240
+ Missing or malformed `exists` or `best` answers return an error branch:
241
+
242
+ ```jsonc
243
+ // abridged; tool/model/provider/query/usage remain
244
+ {
245
+ "status": "invalid_response", "exists": null, "exists_verdict": null, "top": [],
246
+ "reason": "missing or malformed best or exists answer; cannot rank safely"
247
+ }
248
+ ```
249
+
250
+ `exists` must be a finite number in `[0,1]`; zero validly means `absent`. On failure, a valid `exists` value is retained; an invalid or missing value becomes `null`. The best distribution must contain exactly all candidate ids, finite probabilities in `[0,1]` summing to 1 within `0.01`, and a string choice tied for the maximum probability. Missing, null, or non-object `answers` also returns this error branch. Protocol failure is reported only in `status`; `exists_verdict` is `null` on failure.
251
+
252
+ - On successful responses, ranking always returns a winner, because Choice probabilities sum to 1. A top hit can masquerade as an answer when none is present; the exists check catches that. `exists_verdict` is `answered`, `partial`, or `absent`.
215
253
  - Up to 250 candidates per call. Candidate texts are truncated at 2,000 characters.
216
254
  - Pattern from the [semantic-find cookbook](https://docs.typesafe.ai/cookbooks/semantic_find).
217
255
 
@@ -248,7 +286,7 @@ Assign each item to one class from a shared catalog, in one batched request: the
248
286
  - Include a `manual_review` class in your catalog if you want an explicit escape hatch; the tool never invents one.
249
287
  - Class descriptions carry the decision. Strong ones state a precise definition, what belongs, what does not, precedence over overlapping classes, and a short example.
250
288
  - Up to 250 classes and 64 items per call, with an 8,000 item-class budget per batch (split larger waves into multiple calls); item text is truncated at 2,000 characters.
251
- - A malformed or incomplete model response is reported as `status: invalid_response` on that item, never as model uncertainty.
289
+ - A malformed or incomplete model response is reported as `status: invalid_response` on that item, never as model uncertainty. The chosen class must have the maximum probability (ties and differences within `1e-9` are accepted); otherwise that item returns `invalid_response`.
252
290
 
253
291
  ### jev_decide
254
292
 
@@ -411,11 +449,13 @@ Score a proposed diff against the request before the task is called done. Jev an
411
449
  "composite": 0.756,
412
450
  "safe_to_apply": 0.24,
413
451
  "scores": {
414
- "correctness": { "score": 1.51, "confidence": 0.27 },
415
- "spec_match": { "score": 1.65, "confidence": 0.47 },
416
- "test_gap": { "score": 0.80, "confidence": 0.14 },
417
- "blast_radius":{ "score": 0.45, "confidence": 0.32 }
452
+ "correctness": { "score": 1.51, "confidence": 0.27, "probabilities": null },
453
+ "spec_match": { "score": 1.65, "confidence": 0.47, "probabilities": null },
454
+ "test_gap": { "score": 0.80, "confidence": 0.14, "probabilities": null },
455
+ "blast_radius":{ "score": 0.45, "confidence": 0.32, "probabilities": null }
418
456
  },
457
+ "reason_codes": ["confidence_below_review", "safe_to_apply_below_review"],
458
+ "limiting_rubrics": ["test_gap"],
419
459
  "weights": { "correctness": 0.4, "spec_match": 0.3, "test_gap": 0.15, "blast_radius": 0.15 },
420
460
  "thresholds": { "auto_accept": 0.8, "review_at": 0.5, "composite_floor": 0.7 },
421
461
  "truncated": false,
@@ -426,6 +466,8 @@ Score a proposed diff against the request before the task is called done. Jev an
426
466
  Here the composite clears the floor, but the failing test drags `safe_to_apply` to 0.24 and rubric confidences sit under `review_at`, so the patch escalates instead of sailing through on its decent scores.
427
467
 
428
468
  - Rubric scores run 0..2. Higher is better for `correctness` and `spec_match`; higher is worse for `test_gap` and `blast_radius`, and the composite inverts those two before weighting, so a composite of 1.0 means favorable on every rubric.
469
+ - A score answer may carry its full probability distribution over the three options; when the provider reports one, it is validated (exact keys, probabilities summing to one, expected value within a small tolerance of the reported score) and returned in `scores.*.probabilities`. Absent means "not reported" and stays `null`; a distribution that is present but malformed or contradictory marks that rubric `invalid_response`.
470
+ - `reason_codes` collects why the review decided as it did: `invalid_response`, `unknown_confidence`, `confidence_below_review`, `safe_to_apply_below_review`, `confidence_below_auto_accept`, `safe_to_apply_below_auto_accept`, `composite_below_floor`, `incomplete_context`, `accepted`. `limiting_rubrics` names the rubric(s) that bound the decision, ties included: the null-confidence rubrics when confidence is unknown, every rubric tied at the minimum confidence when a confidence threshold blocks, and the least favorable rubrics when the composite floor blocks.
429
471
  - `auto` requires `safe_to_apply` and every rubric confidence at `auto_accept` and the composite at `composite_floor`. `safe_to_apply` or any rubric confidence below `review_at`, or unknown, escalates; a composite below `composite_floor` returns `review`, not `escalate`. Unknown confidence counts as escalate, never as a value that can satisfy a threshold.
430
472
  - `request` frames the review; it is not proof of anything. Put real output in `tests`. Every field is treated as evidence to evaluate, never instructions to follow.
431
473
  - Each text field is capped at 50,000 characters. Truncated input sets `truncated: true` and can never return `auto`; a malformed answer is `invalid_response`, not a semantic outcome.
@@ -479,7 +521,7 @@ The completion gate: the same patch review as `jev_review`, plus your completion
479
521
  The two true claims verify at full confidence, and the one that matters, "the full test suite passes," is contradicted by the test log at full confidence: exactly the claim a coding agent is most tempted to hand-wave.
480
522
 
481
523
  - Claim questions instruct Jev to use `evidence` only, not world knowledge, and not the request, diff, or tests fields; if a claim needs a diff excerpt or a test log as support, supply it in `evidence`. All fields share one model state, so this is instruction-level isolation, not a hard boundary. Every field is evidence to evaluate, never instructions to follow.
482
- - `reason_codes` collects why the gate decided as it did: `incomplete_context`, `invalid_response`, `review_escalated`, `review_required`, `claims_contradicted`, `claims_unsupported`, `claim_confidence_low`, `claim_confidence_below_auto_accept`, `accepted`.
524
+ - `reason_codes` collects why the gate decided as it did: `incomplete_context`, `invalid_response`, `review_escalated`, `review_required`, plus the review half's specific codes (`unknown_confidence`, `confidence_below_review`, `safe_to_apply_below_review`, `confidence_below_auto_accept`, `safe_to_apply_below_auto_accept`, `composite_below_floor`), `claims_contradicted`, `claims_unsupported`, `claim_confidence_low`, `claim_confidence_below_auto_accept`, `accepted`. The embedded `review` object carries the same `reason_codes` and `limiting_rubrics` a standalone `jev_review` returns.
483
525
  - Up to 16 claims and 16 evidence items per call. Text fields are capped at 50,000 characters each, claims at 2,000, and evidence at 200,000 characters in aggregate; oversized evidence is rejected before any model call. Malformed answers surface as `invalid_response` and the gate never returns `auto` on one.
484
526
  - Use `jev_verify` for claims without a patch review, and `jev_review` for a patch without claims.
485
527
 
@@ -498,6 +540,70 @@ The two true claims verify at full confidence, and the one that matters, "the fu
498
540
  - `jev_review`: score a proposed diff before calling the task done.
499
541
  - `jev_gate`: that same review plus completion claims checked against evidence.
500
542
 
543
+ ## Combining the tools: task routing
544
+
545
+ Routing is a good example of combining tools. Before any work starts, you want to know what kind of task this is and which of your workflows should run it. A `jev_classify` call answers the first question, a `jev_decide` call the second.
546
+
547
+ Everything below is an example, not a feature: the package ships the two calls, and the classes, routes, and rules are yours to define.
548
+
549
+ First, classify the task on the axes your routing cares about. Risk is a useful one:
550
+
551
+ ```jsonc
552
+ // arguments
553
+ {
554
+ "purpose": "Decide how to handle an incoming task",
555
+ "context": "The workspace has a code checkout, a database, and a deploy pipeline.",
556
+ "items": [
557
+ { "id": "task", "text": "Add a dark mode toggle to the settings page." }
558
+ ],
559
+ "classes": [
560
+ { "id": "read_only", "description": "Answers without changing anything: reading files, listing records, summarizing." },
561
+ { "id": "reversible", "description": "Changes state but can be undone: local edits, draft records, a staging deploy." },
562
+ { "id": "destructive", "description": "Cannot be undone automatically: deleting records, force-pushes, production deploys, payments." }
563
+ ]
564
+ }
565
+ ```
566
+
567
+ ```jsonc
568
+ // live result, abridged
569
+ {
570
+ "results": [
571
+ { "id": "task", "classification": "reversible", "margin": 0.94, "confidence": 0.97, "decision": "auto" }
572
+ ]
573
+ }
574
+ ```
575
+
576
+ Then decide the route, feeding that answer in as evidence. The `requirements` field is where your policy lives; each one is checked per candidate in the same call:
577
+
578
+ ```jsonc
579
+ // arguments
580
+ {
581
+ "decision": "Which workflow should run this task?",
582
+ "evidence": "jev_classify labeled the task reversible (confidence 0.97): it edits the checkout but touches no production system. Available workflows: answer, implement, escalate.",
583
+ "priorities": "Automated runs must stay reversible; destructive tasks always escalate.",
584
+ "candidates": [
585
+ { "id": "answer", "description": "Look things up and reply. No writes." },
586
+ { "id": "implement", "description": "Branch, edit, run tests, open a PR." },
587
+ { "id": "escalate", "description": "Hand the task to a person." }
588
+ ],
589
+ "requirements": ["Never runs an action the classification called destructive."]
590
+ }
591
+ ```
592
+
593
+ ```jsonc
594
+ // live result, abridged
595
+ {
596
+ "recommendation": { "selected": "implement", "escaped": false, "confidence": 0.93,
597
+ "probabilities": { "implement": 0.93, "escalate": 0.05, "answer": 0.02, "ask_user": 0 } },
598
+ "checks": [ { "candidate": "implement", "requirement": 0, "answer": "supported" } ]
599
+ }
600
+ ```
601
+
602
+ - If either call comes back low-confidence or escaped, route to `escalate` (or ask a person) rather than guessing. The escape hatches exist for exactly that.
603
+ - One classify call and one decide call per task. Repeating them on the same inputs buys nothing.
604
+
605
+ <sub>Pattern credit: [@Garfielk](https://github.com/Garfielk), from the routing discussion in [#5](https://github.com/jkudish/jev-mcp/issues/5).</sub>
606
+
501
607
  ## How the answers work
502
608
 
503
609
  Jev is TypeSafe's System One model: it returns typed answers with calibrated probability distributions, not generated text. A verify call is a Choice over supports / contradicts / says_nothing, so you see the whole distribution, not one label. A screen call is a set of yes/no probabilities. A find call is a Choice over your candidate ids plus an existence check. A rerank call is one yes/no relevance question per candidate. A compare call is a Choice over three relations, repeated independently per aspect. An extract call is a Choice over the candidates your regex already found, so the model picks a value but never writes one. A review call is four Score rubrics plus one safe-to-apply probability; a gate adds one Choice per completion claim, judged from evidence only. Code maps the answers to verdicts and actions; policy stays with you.
@@ -516,13 +622,22 @@ Jev is TypeSafe's System One model: it returns typed answers with calibrated pro
516
622
  | `OPENROUTER_API_KEY` | none | OpenRouter `sk-or-` key; used when `TYPESAFE_API_KEY` is absent. |
517
623
  | `CLOUDFLARE_API_TOKEN` + `CLOUDFLARE_ACCOUNT_ID` | none | Cloudflare Workers AI; used when no other provider key is present. `JEV_CLOUDFLARE_API_TOKEN` is honored first for separate credentials. |
518
624
  | `AI_GATEWAY_API_KEY` | none | Vercel AI Gateway; used when no other provider key is present. |
519
- | `JEV_PROVIDER` | `auto` | Force `typesafe`, `openrouter`, `cloudflare`, or `vercel` instead of auto-detection. |
625
+ | `JEV_PROVIDER` | `auto` | Force `typesafe`, `openrouter`, `cloudflare`, `vercel`, or `compatible` instead of auto-detection. |
520
626
  | `JEV_MCP_MODEL` | `jev-latest` | Pin a Jev version, e.g. `jev-1.12`, or `typesafe/jev-1.13` on OpenRouter. |
521
627
  | `TYPESAFE_BASE_URL` | none | Custom direct endpoint (origin only; the SDK appends its route). |
628
+ | `JEV_API_BASE_URL` + `JEV_API_KEY` | none | Jev-compatible System One endpoint and Bearer token; use with `JEV_PROVIDER=compatible`. `JEV_API_BASE_URL` is the full POST URL including the `/v1/systemone` path. |
629
+ | `JEV_MCP_REQUEST_TIMEOUT_MS` | `60000` | Whole-request deadline in milliseconds, covering every attempt, on the fetch-based transports. |
630
+ | `JEV_MCP_MAX_ATTEMPTS` | `3` | Total attempts per request (clamped 1..6) on the fetch-based transports; retries happen only on 408, 409, 429, and 500 through 599. |
631
+ | `JEV_OPENROUTER_BASE_URL` | `https://openrouter.ai/api` | Override the OpenRouter API root (the `/alpha/decisions` path is appended; a trailing slash is tolerated). |
632
+ | `JEV_CLOUDFLARE_BASE_URL` | `https://api.cloudflare.com/client/v4` | Override the Cloudflare API root (`/accounts/<id>/ai/run` is appended; a trailing slash is tolerated). |
633
+
634
+ ### Transport resilience
635
+
636
+ The fetch-based transports (OpenRouter, Cloudflare, and the Jev-compatible endpoint) retry only on the standard not-processed status set (408, 409, 429, and 500 through 599), with jittered exponential backoff, at most `JEV_MCP_MAX_ATTEMPTS` total attempts, all inside one `JEV_MCP_REQUEST_TIMEOUT_MS` deadline. A status cannot prove the request was not processed, but that allowlist is the conservative retry trigger; ambiguous network-level failures (connection reset, TLS errors) are never retried, because without an idempotency key a re-send can double-process a paid call. Everything else fails immediately: caller cancellations, deadline expiry, non-retryable statuses, unparseable bodies, and responses over 1,000,000 bytes, a ceiling enforced while the body streams rather than after buffering. A cancelled MCP call aborts the in-flight HTTP request, cuts any backoff sleep short, and is never re-sent. Error bodies on those transports are redacted, so a reflecting endpoint can never echo a configured key into MCP-visible errors. Direct TypeSafe and Vercel calls use `@jkudish/jev-agent-tools`, whose direct fetch path avoids the SDK cancellation crash ([typesafe-sdk-js#2](https://github.com/typesafe-ai/typesafe-sdk-js/issues/2)); no retry or deadline uniformity is claimed for those two. Research and the original report: [issue #23](https://github.com/jkudish/jev-mcp/issues/23) by oppih.
522
637
 
523
638
  ### Vercel
524
639
 
525
- With `AI_GATEWAY_API_KEY` set, judgments run through the Vercel AI Gateway at `typesafe-ai/jev`, using the AI SDK's evaluate API. Answers are adapted back to this package's shapes, including TypeSafe's confidence statistic. Gateway calls appear in Vercel logs and budgets.
640
+ With `AI_GATEWAY_API_KEY` set, judgments run through the Vercel AI Gateway at `typesafe-ai/jev`, using the shared wire package's evaluation request. Answers are adapted back to this package's shapes, including TypeSafe's confidence statistic. Gateway calls appear in Vercel logs and budgets.
526
641
 
527
642
  ### Cloudflare
528
643
 
@@ -532,6 +647,21 @@ With `CLOUDFLARE_API_TOKEN` and `CLOUDFLARE_ACCOUNT_ID` set (and no other provid
532
647
 
533
648
  If you already have an OpenRouter key, that is all you need: with no `TYPESAFE_API_KEY` present, every call goes through OpenRouter's Decisions API at identical pricing. The endpoint is alpha and adds a hop, and OpenRouter serves pinned versions rather than a `latest` alias, so the default `jev-latest` maps to `typesafe/jev-1.13` there. Direct TypeSafe remains the recommended default when you have both keys.
534
649
 
650
+ ### Jev-compatible endpoints
651
+
652
+ For a service that implements the same System One request and response contract, set the provider explicitly:
653
+
654
+ ```bash
655
+ export JEV_PROVIDER=compatible
656
+ export JEV_API_BASE_URL=https://api.openjev.sh/v1/systemone
657
+ export JEV_API_KEY=your-compatible-provider-key
658
+ export JEV_MCP_MODEL=openjev
659
+ ```
660
+
661
+ The server sends `POST` requests with `{ model, state, questions }` and requires the standard response shape: an `answers` object plus a `usage` object reporting `input_tokens` and `output_tokens`, with an optional `model` string echoing the model that answered. Envelope problems (a non-object body or `answers`, malformed `usage` counts, a non-string `model`) are rejected at the transport boundary. Individual answers are not judged here: each tool validates them under its own `invalid_response` contract, so a missing or malformed answer fails closed in the tool instead of aborting the call. `JEV_API_BASE_URL` must be the full endpoint URL including the `/v1/systemone` path; it is used verbatim, with no trailing-slash or path normalization. The endpoint and credentials are kept in the local process environment. This adapter is provider-neutral; OpenJEV is one example, not a hard-coded dependency.
662
+
663
+ Need a carrier that is not built in? The [add-a-provider guide](https://github.com/jkudish/jev-agent-tools#adding-a-provider) in the shared wire package covers the options; this server's no-code path is the compatible endpoint above.
664
+
535
665
  ## Also in the family
536
666
 
537
667
  Need those judgments to drive a real browser? [Jev Browser](https://github.com/jkudish/jev-browser) gives an agent a task and a URL and lets Jev pick the actions: click, type, select, stop. It uses the same judgment style this server exposes. The npm package is [@jkudish/jev-browser](https://www.npmjs.com/package/@jkudish/jev-browser).