@parcha/agentrun-jev 0.1.0-beta.20 → 0.1.0-beta.21

package/README.md

@@ -6,10 +6,58 @@ Beta, on npm under the `beta` dist-tag (`npm view @parcha/agentrun-jev dist-tags
 
 ```sh
 npm install @parcha/agentrun-dsl@beta @parcha/agentrun-jev@beta
+```
+
+## Offline tests
+
+To exercise the real adapter with no key and no network request, inject a client. After installing the packages, save this as `offline-jev.mjs` and run `node offline-jev.mjs`:
+
+```js
+import assert from 'node:assert/strict';
+import { createJevRunner } from '@parcha/agentrun-jev';
+
+const runJudge = createJevRunner({
+  maxAttempts: 1,
+  client: {
+    async systemOne(request, options) {
+      options.signal?.throwIfAborted();
+      assert.deepEqual(Object.keys(request.questions), ['answersQuestion']);
+      return {
+        answers: { answersQuestion: { type: 'noul', noul: 0.95 } },
+      };
+    },
+  },
+});
+
+const result = await runJudge({
+  label: 'screen-evidence',
+  kind: 'judge',
+  state: {
+    question: 'Can emergency documentation fixes bypass prior approval?',
+    passage: 'During an incident, the on-call engineer may merge a fix immediately; a second person reviews it the next working day.',
+  },
+  questions: {
+    answersQuestion: {
+      type: 'noul',
+      instructions: 'Does the passage contain a specific rule or exception answering the question? Retain conditional and contrary evidence; reject topic mentions and promotional claims.',
+    },
+  },
+});
+assert.equal(result.answers.answersQuestion.noul, 0.95);
+console.log(result.answers.answersQuestion);
+```
+
+This prints `{ type: 'noul', noul: 0.95 }`. The answer is **scripted**; it does not evaluate the passage or adapt to prompt changes. Answer IDs must match the requested question IDs. Noul returns a probability, not a boolean. Reported model and token usage may be omitted; the adapter preserves missing usage/cost as unknown instead of inventing measurements.
+
+## Call Jev
+
+A live call needs a key. Set it, then run the sample below as `live-jev.mjs` (`node live-jev.mjs`). The samples that follow are written in TypeScript; run them with a TypeScript runner such as `npx tsx`, or drop the `type` imports and annotations to run them as `.mjs`.
+
+```sh
 export TYPESAFE_API_KEY="your-key"
 ```
 
-```ts
+```js
 import { createJevRunner } from '@parcha/agentrun-jev';
 
 const runJudge = createJevRunner();
@@ -29,11 +77,13 @@ const result = await runJudge({
 console.log(result.answers.team);
 ```
 
+Without a key, `createJevRunner()` throws `JevError` (code `configuration`: "Could not configure Jev. Supply apiKey or set TYPESAFE_API_KEY, and check transport options.") on that line, before any request is made. The same holds for `provider: "openrouter"` without `OPENROUTER_API_KEY`.
+
 `judge`, `pick`, `sift`, `route`, semantic predicates, and judgment verification use the same `runJudge` adapter. Choice, Noul, and Score answers retain their raw probabilities. The core validates answer IDs, probability keys and normalization, score rubrics, and value ranges before consuming a result. A typed answer records the model's decision. It does not prove that a real-world claim is true.
 
 `System One` is TypeSafe's typed-question interface, exposed by the SDK as `systemOne`. A Noul answer is a truth probability from 0 to 1; the DSL converts probabilities of at least 0.5 to `true` and retains the probability beside that value. Choice and Score confidence comes from the response. Jev does not return a separate confidence for Noul; the DSL derives its own gate-strength value from the distance from 0.5, scaled to the range 0 to 1. These numbers are useful gate inputs, not calibrated guarantees.
 
-The public adapter API includes `createJevRunner`, `JevError`, `isJevResponseReason`, and the types `JevOptions`, `JevClient`, `JevErrorCode`, `JevResponseReason`, and `JevResponseDiagnostic`. The shared question, answer and distribution helpers live in `@parcha/agentrun-dsl`.
+The public adapter API is `createJevRunner`, `jevProviderFromEnv`, `JevError`, `isJevResponseReason`, and the types `JevOptions`, `JevClient`, `JevProvider`, `JevErrorCode`, `JevResponseReason`, `JevResponseDiagnostic`, `JevRequestDiagnostic`, `JevImage`, `JevJudge`, `JevJudgeParams` and `JevJudgeResult`. The shared question, answer and distribution helpers live in `@parcha/agentrun-dsl`.
 
 ## Configuration
 
@@ -65,13 +115,13 @@ Cost: when the response reports `usage.cost` (USD) it is `cost_usd`; otherwise a
 
 The official `@typesafe-ai/sdk` resolves `TYPESAFE_API_KEY` and `TYPESAFE_BASE_URL`. Explicit `apiKey` and `baseURL` options take precedence. An API root ends before `/v1/systemone`. Set `fetch` for a custom transport, or supply a `client` with a `systemOne(request, options)` method. A custom client must honor the request signal and disable its own retries; it owns its credentials and cannot be combined with transport options.
 
-The adapter retries transient HTTP 408, 429, 5xx responses and connection failures with bounded exponential delays. SDK retries are disabled. Validation failures and other HTTP errors do not retry. The deadline and either the runner or per-call signal interrupt both requests and backoff. Cancellation bounds when the adapter returns; an uncooperative custom client can still continue its own work.
+The adapter retries transient HTTP 408, 429, 5xx responses and connection failures with bounded exponential delays. On a 429 or 503 with a `Retry-After` (seconds or an HTTP date), it waits the larger of that and its own delay, and gives up at once, with attempts left, when that wait would not end before the timeout. When the attempts end on a 429 after an earlier 408, 5xx or lost connection, the error reports the first of those failures (its HTTP `status`, or code `connection`), since that attempt may have been billed; an `http` error with status 429 means every attempt was refused. SDK retries are disabled. Validation failures and other HTTP errors do not retry. The deadline and either the runner or per-call signal interrupt both requests and backoff. Cancellation bounds when the adapter returns; an uncooperative custom client can still continue its own work.
 
 Request state has no local byte cap by default. Set `maxStateBytes` to a positive safe integer to reject serialized UTF-8 JSON state exceeding that host-owned limit before transport; omission or `null` disables this guard. This is a byte guard, not a token estimate. [Provider model context limits](https://docs.typesafe.ai/models.md) still apply. The adapter never truncates or splits state or questions.
 
 The core allows up to 240 options per choice and 2–10 levels per score rubric. Adapter timeouts are positive integer milliseconds up to 2,147,483,647; retry delays may also be zero. `maxAttempts` is 1–10, with a default of 3.
 
-`JevError` provides a stable `code`, `attempts`, and optional HTTP `status`. Errors deliberately omit raw service bodies, headers, input state, and causes. SDK logging is disabled. Custom clients and transports own their own logging. Do not expose credentials in browser code.
+`JevError` provides a stable `code`, `attempts`, and optional HTTP `status`. A request the runner refuses before sending it has code `invalid_request`, `attempts` 0 and a `requestDiagnostic` (`JevRequestDiagnostic`): a `reason` (`invalid_state`, `state_too_large`, `invalid_questions`, `invalid_assets`, `unsupported_model` or `unsupported_provider`), the call's `label` (at most 200 characters) when it has one, and for a state over `maxStateBytes` its `stateBytes` and `maxStateBytes`. Errors deliberately omit raw service bodies, headers, input state, and causes. SDK logging is disabled. Custom clients and transports own their own logging. Do not expose credentials in browser code.
 
 Optional `responseDiagnostic.reason` identifies a fixed response invariant, such
 as `answer_keys`, `probability_mass`, or `token_usage`. An explicitly structured
@@ -79,7 +129,7 @@ capacity rejection can report `max_tokens_exceeded`; arbitrary provider messages
 are never parsed for a reason. These details do not change validation, retry
 policy, or the submitted evidence. Unknown causes remain unspecified.
 
-Successful results retain `answers`, reported `model`, reported token `usage`, and a SHA-256 of the submitted request. `cost_usd` is `null` unless both usage and explicit `pricing: { inputUsdPerMillionTokens, outputUsdPerMillionTokens }` are available. Configured prices produce an estimate for the successful response; costs from unsuccessful or interrupted attempts are unknown.
+Successful results retain `answers`, reported `model`, reported token `usage`, and a SHA-256 of the submitted request. `cost_usd` is the response's reported `usage.cost` when it has one, else an estimate from the counted tokens and an explicit `pricing: { inputUsdPerMillionTokens, outputUsdPerMillionTokens }`, else `null`. Configured prices produce an estimate for the successful response; costs from unsuccessful or interrupted attempts are unknown.
 
 ## Images and Box questions
 
@@ -111,46 +161,7 @@ const { answers, usage, cost_usd } = await judge({
 - Request bodies are sent uncompressed; the SDK's default compresses them with zstd.
 - Box is a preview: Jev's page says its request format, answers and limits can change between releases.
 
-## Offline tests
-
-Inject a client to exercise the real adapter without a key or network request. After installing the DSL and Jev packages, save this as `offline-jev.mjs` and run `node offline-jev.mjs`:
-
-```js
-import assert from 'node:assert/strict';
-import { createJevRunner } from '@parcha/agentrun-jev';
-
-const runJudge = createJevRunner({
-  maxAttempts: 1,
-  client: {
-    async systemOne(request, options) {
-      options.signal?.throwIfAborted();
-      assert.deepEqual(Object.keys(request.questions), ['answersQuestion']);
-      return {
-        answers: { answersQuestion: { type: 'noul', noul: 0.95 } },
-      };
-    },
-  },
-});
-
-const result = await runJudge({
-  label: 'screen-evidence',
-  kind: 'judge',
-  state: {
-    question: 'Can emergency documentation fixes bypass prior approval?',
-    passage: 'During an incident, the on-call engineer may merge a fix immediately; a second person reviews it the next working day.',
-  },
-  questions: {
-    answersQuestion: {
-      type: 'noul',
-      instructions: 'Does the passage contain a specific rule or exception answering the question? Retain conditional and contrary evidence; reject topic mentions and promotional claims.',
-    },
-  },
-});
-assert.equal(result.answers.answersQuestion.noul, 0.95);
-console.log(result.answers.answersQuestion);
-```
-
-This prints `{ type: 'noul', noul: 0.95 }`. The answer is **scripted**; it does not evaluate the passage or adapt to prompt changes. Answer IDs must match the requested question IDs. Noul returns a probability, not a boolean. Reported model and token usage may be omitted; the adapter preserves missing usage/cost as unknown instead of inventing measurements.
+## Testing your own host
 
 The [standalone TypeScript starter](https://github.com/Parcha-ai/agentrun/tree/main/examples/starter#readme) includes npm installation, a complete search-and-screen workflow, and tests for missing evidence and the screening step alone. Its injected client answers every named question generated by `sift`.
 

package/dist/index.js

@@ -1,4 +1,5 @@
 import { createHash } from "node:crypto";
+import { setTimeout as sleep } from "node:timers/promises";
 import { APIConnectionError, APIError, Image, TypeSafeClient } from "@typesafe-ai/sdk";
 import { validateAnswers, SystemOneError, isSystemOneResponseReason } from "@parcha/agentrun-dsl";
 const keyVariable = { typesafe: "TYPESAFE_API_KEY", openrouter: "OPENROUTER_API_KEY" };
@@ -7,10 +8,13 @@ const OPENROUTER_DEFAULT_MODEL = "~typesafe/jev-latest";
 /** The provider an environment's keys select: OpenRouter only when `OPENROUTER_API_KEY` is set and `TYPESAFE_API_KEY` is not. The runner itself never reads this. */
 export function jevProviderFromEnv(env = process.env) {
     const set = (name) => typeof env[name] === "string" && env[name].trim().length > 0;
-    return set("OPENROUTER_API_KEY") && !set("TYPESAFE_API_KEY") ? "openrouter" : "typesafe";
+    return set(keyVariable.openrouter) && !set(keyVariable.typesafe) ? "openrouter" : "typesafe";
 }
 const IMAGE_HEADER = "TypeSafe-Danger-Zone-Images";
 const MAX_IMAGE_BYTES = 16 * 1024 * 1024;
+const MAX_REFERENCE_BYTES = 1024;
+/** The longest delay a Node timer accepts. */
+const MAX_TIMER_MS = 2_147_483_647;
 const BOX_MODEL = "jev-1.14.0";
 /** Models that answer a Box question: Jev 1.14 and later by name, and the preview alias. The check is by name, so `jev-latest` is refused whatever version it points at. */
 function answersBox(model) {
@@ -58,7 +62,7 @@ const configuration = (condition, message) => {
     if (!condition)
         throw new JevError("configuration", message);
 };
-const milliseconds = (n, allowZero = false) => Number.isSafeInteger(n) && n >= (allowZero ? 0 : 1) && n <= 2_147_483_647;
+const milliseconds = (n, allowZero = false) => Number.isSafeInteger(n) && n >= (allowZero ? 0 : 1) && n <= MAX_TIMER_MS;
 const record = (v) => v !== null && typeof v === "object" && !Array.isArray(v);
 const tokens = (n) => typeof n === "number" && Number.isSafeInteger(n) && n >= 0;
 function abortable(promise, signal, error) {
@@ -73,14 +77,13 @@ function abortable(promise, signal, error) {
             abort();
     });
 }
-function wait(ms, signal, error) {
-    return new Promise((resolve, reject) => {
-        const abort = () => { clearTimeout(timer); signal.removeEventListener("abort", abort); reject(error()); };
-        const timer = setTimeout(() => { signal.removeEventListener("abort", abort); resolve(); }, ms);
-        signal.addEventListener("abort", abort, { once: true });
-        if (signal.aborted)
-            abort();
-    });
+async function wait(ms, signal, error) {
+    try {
+        await sleep(ms, undefined, { signal });
+    }
+    catch {
+        throw error();
+    }
 }
 /** A server's Retry-After in milliseconds from now, given as delta-seconds or an HTTP date; undefined when absent or unparseable. */
 function retryAfterMs(headers) {
@@ -98,7 +101,7 @@ export function createJevRunner(options = {}) {
     const maxAttempts = options.maxAttempts ?? 3;
     const retryBaseMs = options.retryBaseMs ?? 500;
     const retryMaxMs = options.retryMaxMs ?? 5_000;
-    configuration(timeoutMs === null || milliseconds(timeoutMs), "timeoutMs must be null or a positive integer of at most 2147483647.");
+    configuration(timeoutMs === null || milliseconds(timeoutMs), `timeoutMs must be null or a positive integer of at most ${MAX_TIMER_MS}.`);
     configuration(maxStateBytes === null || (Number.isSafeInteger(maxStateBytes) && maxStateBytes > 0), "maxStateBytes must be null or a positive safe integer.");
     configuration(timeoutMs !== null || options.client !== undefined, "The TypeSafe SDK requires a finite timeout. For timeoutMs: null, supply a cancellation-only host client.");
     configuration(Number.isInteger(maxAttempts) && maxAttempts >= 1 && maxAttempts <= 10, "maxAttempts must be an integer from 1 to 10.");
@@ -113,7 +116,7 @@ export function createJevRunner(options = {}) {
     configuration(typeof model === "string" && model.trim().length > 0, "model must be a nonempty string.");
     let client;
     try {
-        const key = options.apiKey ?? (provider === "openrouter" ? process.env.OPENROUTER_API_KEY : undefined);
+        const key = options.apiKey ?? (provider === "openrouter" ? process.env[keyVariable.openrouter] : undefined);
         // The SDK falls back to TYPESAFE_API_KEY when given no key: for OpenRouter that would send one provider's key to another.
         if (provider === "openrouter" && (typeof key !== "string" || key.trim().length === 0))
             throw new Error("no key");
@@ -136,6 +139,7 @@ export function createJevRunner(options = {}) {
         const abortError = () => (runnerSignal?.aborted || params.signal?.aborted)
             ? new JevError("aborted", "Jev request was cancelled.", attempt)
             : new JevError("timeout", "Jev request exceeded its total deadline.", attempt);
+        const invalidResponse = (reason, message) => new JevError("invalid_response", message, attempt, undefined, undefined, { reason });
         if (signal.aborted)
             throw abortError();
         const timer = timeoutMs === null ? undefined : setTimeout(() => deadline.abort(), timeoutMs);
@@ -175,7 +179,7 @@ export function createJevRunner(options = {}) {
             }
             // Images and Box questions: validated whole before any send; the header goes with exactly these requests.
             const hasBox = Object.values(request.questions).some(question => record(question) && question.type === "box");
-            const given = params.assets === undefined || params.assets === null ? {} : params.assets;
+            const given = params.assets ?? {};
             if (!record(given))
                 throw invalid("invalid_assets", "assets must map each reference to an image; no request was sent.");
             const references = Object.keys(given);
@@ -195,11 +199,11 @@ export function createJevRunner(options = {}) {
                 assetDigests = Object.create(null);
                 for (const reference of references) {
                     const image = given[reference];
-                    const sound = typeof reference === "string" && reference.length > 0 && Buffer.byteLength(reference, "utf8") <= 1024 && !/["\\\u0000-\u001f]/.test(reference)
+                    const sound = reference.length > 0 && Buffer.byteLength(reference, "utf8") <= MAX_REFERENCE_BYTES && !/["\\\u0000-\u001f]/.test(reference)
                         && record(image) && image.bytes instanceof Uint8Array && (image.mime === "image/png" || image.mime === "image/jpeg")
                         && image.bytes.byteLength > 0 && image.bytes.byteLength <= MAX_IMAGE_BYTES && startsWith(image.bytes, imageMagic[image.mime]);
                     if (!sound)
-                        throw invalid("invalid_assets", "Each image is a PNG or JPEG file of at most 16 MiB, with bytes that are that type and a plain reference. No request was sent.");
+                        throw invalid("invalid_assets", `Each image is a PNG or JPEG file of at most ${MAX_IMAGE_BYTES / 1024 / 1024} MiB, with bytes that are that type and a plain reference. No request was sent.`);
                     if (!texts.some(text => text.includes(reference)))
                         throw invalid("invalid_assets", "Every image reference must appear in the state or a question. No request was sent.");
                     assets[reference] = new Image(image.bytes);
@@ -232,9 +236,10 @@ export function createJevRunner(options = {}) {
                         const backoff = Math.min(retryMaxMs, retryBaseMs * 2 ** (attempt - 1));
                         const asked = error instanceof APIError && (status === 429 || status === 503) ? retryAfterMs(error.headers) : undefined;
                         const delay = asked === undefined ? backoff : Math.max(asked, backoff);
-                        // A server's wait is kept in full or the loop gives up now: it never runs past the deadline, or past the longest timer when there is none, so no
-                        // retry is sent with no time left to answer (a request that times out has a cost no one knows).
-                        const left = timeoutMs === null ? 2_147_483_647 : timeoutMs - (Date.now() - started);
+                        // A server's Retry-After wait is kept in full or the loop gives up now: a wait that would not end before the deadline (or within the longest
+                        // timer, when there is none) is never started, since the retry after it would have no time left to answer (a request that times out has a
+                        // cost no one knows). A backoff wait is not checked here: the deadline ends it.
+                        const left = timeoutMs === null ? MAX_TIMER_MS : timeoutMs - (Date.now() - started);
                         if (asked === undefined || delay < left) {
                             await wait(delay, signal, abortError);
                             continue;
@@ -258,13 +263,13 @@ export function createJevRunner(options = {}) {
                     throw new JevError("invalid_request", "Jev could not process the request. Check question and transport configuration.", attempt);
                 }
                 if (!record(result))
-                    throw new JevError("invalid_response", "Jev returned an invalid response.", attempt, undefined, undefined, { reason: "response_shape" });
+                    throw invalidResponse("response_shape", "Jev returned an invalid response.");
                 const answers = result.answers;
                 try {
                     validateAnswers(params.questions, answers);
                 }
                 catch (error) {
-                    throw new JevError("invalid_response", "Jev returned invalid answers or probability distributions.", attempt, undefined, undefined, { reason: error instanceof SystemOneError && isSystemOneResponseReason(error.responseReason) ? error.responseReason : "answer_validation" });
+                    throw invalidResponse(error instanceof SystemOneError && isSystemOneResponseReason(error.responseReason) ? error.responseReason : "answer_validation", "Jev returned invalid answers or probability distributions.");
                 }
                 const usage = result.usage;
                 // Usage carries token counts, a provider-reported cost (USD), or both; a half pair of counts or a bad cost is invalid.
@@ -273,16 +278,16 @@ export function createJevRunner(options = {}) {
                 const reportedCost = reported && typeof usage.cost === "number" && Number.isFinite(usage.cost) && usage.cost >= 0 ? usage.cost : undefined;
                 const noCounts = record(usage) && usage.input_tokens === undefined && usage.output_tokens === undefined;
                 if (usage != null && (!record(usage) || (reported ? reportedCost === undefined || !(counted || noCounts) : !counted))) {
-                    throw new JevError("invalid_response", "Jev returned invalid token usage.", attempt, undefined, undefined, { reason: "token_usage" });
+                    throw invalidResponse("token_usage", "Jev returned invalid token usage.");
                 }
                 if (result.model != null && (typeof result.model !== "string" || !result.model.trim())) {
-                    throw new JevError("invalid_response", "Jev returned an invalid model identifier.", attempt, undefined, undefined, { reason: "model_identifier" });
+                    throw invalidResponse("model_identifier", "Jev returned an invalid model identifier.");
                 }
                 const metering = counted ? { input_tokens: usage.input_tokens, output_tokens: usage.output_tokens } : null;
                 // The provider's own cost wins; else the configured estimate; else unknown (null), never 0.
                 const cost = reportedCost !== undefined ? reportedCost : pricing && metering ? (metering.input_tokens * pricing.inputUsdPerMillionTokens + metering.output_tokens * pricing.outputUsdPerMillionTokens) / 1_000_000 : null;
                 if (cost !== null && !Number.isFinite(cost))
-                    throw new JevError("invalid_response", "Jev cost estimate exceeded the numeric range.", attempt, undefined, undefined, { reason: "cost_range" });
+                    throw invalidResponse("cost_range", "Jev cost estimate exceeded the numeric range.");
                 return {
                     answers, model: typeof result.model === "string" ? result.model : null,
                     usage: metering, cost_usd: cost,

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@parcha/agentrun-jev",
-  "version": "0.1.0-beta.20",
+  "version": "0.1.0-beta.21",
   "description": "Optional Jev system one decisions for AgentRun workflows",
   "type": "module",
   "license": "Apache-2.0",
@@ -27,7 +27,7 @@
     "test": "node --test test/*.test.mjs"
   },
   "dependencies": {
-    "@parcha/agentrun-dsl": "0.1.0-beta.20",
+    "@parcha/agentrun-dsl": "0.1.0-beta.21",
     "@typesafe-ai/sdk": "0.7.0"
   },
   "repository": {