okengine 0.21.1 → 0.23.0

package/site/content/docs/elements/ai/decide.mdx

@@ -0,0 +1,536 @@
+---
+title: "Decisions"
+description: "One model call that labels a ticket or request, and waits for a person when it is not allowed to decide alone."
+icon: "Scale"
+source: "docs/spec/unified-theory.md"
+---
+
+A decision picks labels for one piece of work: which queue a ticket belongs in, and whether it needs a person today. One request asks every question. The result records whether the model or a person supplied each value.
+
+<Callout title="OpenRouter">
+  The OpenRouter decisions endpoint is alpha on the provider side. Score levels and a whole question
+  held in a same-file const are checked at compile time.
+</Callout>
+
+<Callout title="The one rule">
+  Declare exactly one of `review` or `onUncertain: "abstain"`, then call it with `fx.decide`. Only
+  `oke-decisions.lock.json` can grant `auto`.
+</Callout>
+
+## Quick start
+
+<Steps>
+
+<Step>
+### Declare the questions
+
+`review` is a real gate. The people who hold that gate are the ones who may answer when the
+model is not allowed to.
+
+```typescript title="src/core/ai.ts"
+import { ai, gate } from "okengine";
+
+export const ops = gate.policy("ops", ({ operator }) => operator.id !== null);
+
+export const triage = ai.decision("triage", {
+  review: ops,
+  ask: {
+    team: ai.choice("Which team owns this ticket?", {
+      billing: "Billing",
+      technical: "Technical",
+    }),
+    urgent: ai.boolean("Does this need a person today?", {
+      true: "A person should see it today",
+      false: "It can wait",
+    }),
+  },
+});
+```
+
+`ai.choice` takes at most 254 options. The model may also answer `none_of_these`, which never returns as `auto`. A reviewer may still submit `none_of_these`; the Flow receives that string. `ai.boolean` takes an optional `{ true, false }` criteria pair. `ai.score` takes 2–10 levels, best to worst. Question ids `meta` and `$` are reserved.
+
+</Step>
+
+<Step>
+### Emit, then decide in a consumer
+
+A review parks the run until a person answers. An HTTP trigger cannot park: the response would
+be 204 and the caller would see nothing. Accept the ticket on HTTP, emit a signal, and decide
+in a durable consumer.
+
+```typescript title="src/flows/support/open.ts"
+import { on, flow, http, signal } from "okengine";
+import { z } from "zod";
+
+export const ticketOpened = signal.broadcast("ticket.opened");
+
+export const open = on(
+  http.post({ in: z.object({ text: z.string().min(1) }) }).public(),
+  flow({
+    do: async ({ text }, fx) => {
+      await fx.emit(ticketOpened, { text });
+      return { accepted: true };
+    },
+  }),
+);
+```
+
+```typescript title="src/flows/support/route.ts"
+import { on, flow } from "okengine";
+import { triage } from "@/core/ai";
+import { ticketOpened } from "./open";
+
+export const route = on(
+  ticketOpened,
+  flow({
+    durable: true,
+    do: async (input, fx) => fx.decide(triage, input),
+  }),
+);
+```
+
+`durable: true` is required for `review`. Abstain mode can run without it.
+
+</Step>
+
+<Step>
+### Resolve it in Console
+
+The park waits until someone who passes the `ops` gate submits every open question. Console on port 6533 does that as an operator: the route sends the signed-in operator and that session's auth scopes into the review gate. The gate must allow that operator. An operator may resolve any tenant.
+
+Open `/flows/decisions`, or POST the review id from the queue:
+
+```bash
+curl -X POST http://127.0.0.1:6533/console/decisions/resolve \
+  -H "Authorization: $OKE_OPERATOR_AUTHORIZATION" \
+  -H "content-type: application/json" \
+  -d '{"id":"<review id>","values":{"team":"billing","urgent":false}}'
+```
+
+The consumer then returns. Each question is a top-level field. `$` records how it was produced. `p` is calibrated confidence from 0 to 1. `raw` is the provider distribution before calibration.
+
+```json
+{
+  "team": "billing",
+  "urgent": false,
+  "$": {
+    "meta": {
+      "model": "typesafe/jev-1.13-20260917",
+      "provider": "openrouter",
+      "usage": { "inputTokens": 120, "outputTokens": 40 }
+    },
+    "team": {
+      "how": "reviewed",
+      "p": 0.42,
+      "raw": { "billing": 0.2, "technical": 0.2, "none_of_these": 0.6 }
+    },
+    "urgent": { "how": "reviewed", "p": 0.51, "raw": { "noul": 0.51 } }
+  }
+}
+```
+
+`how: "reviewed"` is the value the gate submitted, including a choice of `none_of_these`. `p` stays the model's confidence. `raw.noul` is the provider's yes/no score for a boolean question.
+
+| `how`       | Value the Flow sees                                                           |
+| ----------- | ----------------------------------------------------------------------------- |
+| `reviewed`  | The value the review gate submitted.                                          |
+| `abstained` | `null` on that question only. Certain questions in the same call stay `auto`. |
+| `auto`      | The calibrated answer, after a lockfile exists. See Certify and promote.      |
+
+</Step>
+
+</Steps>
+
+## Options
+
+<TypeTable
+  type={{
+    ask: {
+      description: "Named questions. One request sends all of them.",
+      type: "Record<string, Choice | Score | Boolean>",
+      required: true,
+    },
+    review: {
+      description:
+        "Gate that may resolve a park. Exclusive with onUncertain. The Flow must be durable. No timeout.",
+      type: "Gate | string",
+    },
+    onUncertain: {
+      description: "Only abstain. Returns null and does not park. Exclusive with review.",
+      type: '"abstain"',
+    },
+    autonomy: {
+      description: "maxError is required. audit is required, from 0 to 1. risk defaults to 0.1.",
+      type: "{ maxError: number; audit: number; risk?: number }",
+    },
+    model: {
+      description:
+        "Model id. Default is typesafe/jev-1.13 on OpenRouter, or jev-1.13.0 on TypeSafe.",
+      type: "AiModel | string",
+    },
+    driverId: {
+      description: "Which provider to call.",
+      type: '"openrouter" | "typesafe"',
+      default: "openrouter",
+    },
+    locale: {
+      description: "Slice the certificate by a string from the input. Absent means one slice.",
+      type: "(input) => string | undefined",
+    },
+    evals: { description: "Seed JSONL path. Same role as a prompt evals file.", type: "string" },
+    timeout: {
+      description: "Provider deadline. A duration string or milliseconds.",
+      type: "string | number",
+      default: "30s",
+    },
+  }}
+/>
+
+## Lifecycle
+
+A decision is in one list state. Drift is one flag per decision. Suspending `triage` leaves `route` alone.
+
+```text
+learning  --labels-->  candidate ready  --promote-->  certified
+                              \                           /
+                               \---- drift ----> suspended
+```
+
+| State             | What is true                                                                                                                                                            | What moves it                                                                                            |
+| ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------- |
+| `learning`        | No lockfile entry and no candidate.                                                                                                                                     | Reviews and audit labels accumulate.                                                                     |
+| `candidate ready` | The clock has built a candidate from those labels. The page shows the fit and `oke decide promote <name>`. Counts are not the grant.                                    | Promote writes that slot.                                                                                |
+| `certified`       | `oke-decisions.lock.json` has this decision, and the declaration sets `autonomy`. A lock entry is ignored without `autonomy`.                                           | The runtime may return `auto` under that certificate.                                                    |
+| `suspended`       | This decision's audit labels since its certificate, inside 7 days, for the pinned model, failed the error cap. The monitor emitted `oke/decision/drift` with that name. | Promote or recertify clears only this decision. A newer `certifiedAt` on the next load ignores the flag. |
+
+`oke dev` reloads `oke-decisions.lock.json` when the file changes. Production reads it once, at boot.
+
+The runtime never raises a threshold and never invents a certificate. Missing lock, a stale
+question, a model version mismatch, an uncertified locale, low confidence, drift, and an
+outage all take the non-auto path.
+
+## How a person resolves a review
+
+Open Console at `/flows/decisions` on port 6533. Each row is `learning`, `candidate ready`, `certified`, or `suspended` for that decision. The queue shows how long each row has been waiting. The resolve form only accepts a choice option, `none_of_these`, or a score level. An audit row submits `labelOnly: true`. A second resolve is `Conflict`. A lease collision retries. A failed label write is listed on the page.
+
+An audit row is a label only. The original Flow already returned. A review row is the park:
+submitting it wakes the consumer.
+
+The review gate must allow the caller. Console resolve runs on the operator plane: it passes the signed-in operator id and the session auth into the gate, and it may resolve any tenant. A gate that checks `operator.id` matches that call.
+
+A review body must include every question that was not `auto`. An audit row (`labelOnly: true`) must include every question, including ones that were already `auto`. A choice may be `none_of_these`. A partial body is 422: `review values do not match the open questions`.
+
+```bash
+curl -X POST http://127.0.0.1:6533/console/decisions/resolve \
+  -H "Authorization: $OKE_OPERATOR_AUTHORIZATION" \
+  -H "content-type: application/json" \
+  -d '{"id":"<audit id>","labelOnly":true,"values":{"team":"billing","urgent":true}}'
+```
+
+| Result                       | HTTP                                                                                                            | What you do                                           |
+| ---------------------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- |
+| `{ ok: true }`               | 200                                                                                                             | The consumer resumes with `how: "reviewed"`.          |
+| Already resolved             | 409 `Conflict`. No `Retry-After`. `That value is already in use.`                                               | Stop.                                                 |
+| Another worker holds the run | 409 `JournalLeaseBusy` with `Retry-After`. `This run is locked by another worker. Retry after the given delay.` | Retry after the delay. A wait over 60 seconds is 503. |
+
+The same 409 split applies to tool approval. Retry only `JournalLeaseBusy`.
+
+## Certify and promote
+
+**maxError** is the highest fraction of wrong answers you will accept among the ones the
+model is allowed to return alone. `0.05` means at most 5 in 100 of those automatic answers.
+
+**audit** is how often an automatic answer is also queued for a person, from 0 to 1.
+`0.1` queues about one in ten. Those labels are how the next certificate is fit. Omitting
+`audit` when `autonomy` is set fails to compile.
+
+**risk** is how sure the statistical test must be before it accepts a threshold. Default
+`0.1`. A smaller risk demands more labels.
+
+**Calibration** rescales the provider's raw scores so `p` means a probability. Choice and
+score use a temperature. Yes/no uses a curve fit on the provider score. The lockfile stores
+that fit next to the question.
+
+**The test**, once: for each confidence cutoff from 0.50 to 0.99 (50 cutoffs), count mistakes on labels that cleared it.
+A binomial test passes only when those mistakes are few enough to show the error rate is under `maxError`.
+The loosest cutoff that passes is the threshold. If none pass, that question cannot return `auto`.
+
+With `risk` at 0.1 and no mistakes, the label count is about `log(0.1 / 50) / log(1 − maxError)`:
+
+| `maxError` | Error-free labels |
+| ---------- | ----------------- |
+| `0.1`      | ≈ 59              |
+| `0.05`     | ≈ 122             |
+| `0.01`     | ≈ 619             |
+
+A seed file is JSONL. `expect` is the label for each question. `locale` is optional.
+
+```json
+{"id":"t1","input":{"text":"I was charged twice"},"expect":{"team":"billing","urgent":true}}
+{"id":"t2","input":{"text":"The app crashes on save"},"expect":{"team":"technical","urgent":false},"locale":"en"}
+```
+
+Point `evals` at that file, then certify. Run `oke build` first. `oke eval --certify` reads `oke.manifest.json` from the app root. Prompt evals do not run when the flag is present.
+It writes `oke-decisions.lock.json` in that root (`OKE_ROOT_DIR`, or `rootDir`), not in the shell's current directory. Boot reads that same file. Declared decisions without a project root fail at boot.
+
+`how: "auto"` is what the Flow sees after that file is loaded and the question clears its threshold:
+
+```json
+{ "team": "technical", "$": { "team": { "how": "auto", "p": 0.91, "audited": true } } }
+```
+
+```bash
+OKE_ROOT_DIR=/srv/app oke eval --certify
+```
+
+The empty-string key is the slice used when `locale` is absent. A locale function gets its
+own key. `hash` is the question text and options. Edit the question and the hash no longer
+matches: the runtime stops returning `auto` until you certify again.
+
+```json
+{
+  "decisions": {
+    "triage": {
+      "model": "typesafe/jev-1.13-20260917",
+      "certifiedAt": 1710000000000,
+      "questions": {
+        "team": {
+          "": {
+            "hash": "…",
+            "calibrator": { "kind": "temperature", "t": 1.2 },
+            "threshold": 0.8,
+            "metrics": {
+              "labels": 122,
+              "accepted": 122,
+              "errors": 0,
+              "maxError": 0.05,
+              "delta": 0.1
+            }
+          }
+        }
+      }
+    }
+  }
+}
+```
+
+After production labels exist, the clock stores a candidate on the journal. It survives a restart only when that journal is a file or Postgres. A memory journal drops it when the process exits. Promote fetches it. It does not recompute the certificate. Other decisions already in the file stay. This decision's slot is replaced. Promote clears drift for this decision only.
+
+```bash
+OKE_ORIGIN=http://127.0.0.1:6530 \
+OKE_OPERATOR_AUTHORIZATION='Bearer …' \
+oke decide promote triage \
+  --origin "$OKE_ORIGIN" \
+  --authorization "$OKE_OPERATOR_AUTHORIZATION" \
+  --lock "$OKE_ROOT_DIR/oke-decisions.lock.json"
+```
+
+That GET is `/_oke/decisions/triage/candidate`. The caller must be an operator. A missing candidate is 404. The route exists only when a decision declares `autonomy`.
+
+`oke decide labels <name> --export` writes reviewed labels as seed JSONL (`input`, `expect`, `locale`). It is operator-only and limited to the caller's tenant. The file keeps the decision's declared `in` fields. Secret and redacted fields are masked. The command prints that the file contains production data.
+
+```bash
+oke decide labels triage --export --out triage.labels.jsonl
+```
+
+## Patterns
+
+<Tabs items={["Review", "Abstain", "Autonomy", "Locale"]}>
+
+<Tab value="Review">
+
+Non-auto questions park the consumer. The result is returned only after the gate submits
+every open question. `how` on those questions is `reviewed`.
+
+```typescript
+export const triage = ai.decision("triage", {
+  review: ops,
+  ask: {
+    team: ai.choice("Which team owns this ticket?", {
+      billing: "Billing",
+      technical: "Technical",
+    }),
+  },
+});
+```
+
+</Tab>
+
+<Tab value="Abstain">
+
+Only the uncertain questions return `null` and `how: "abstained"`. A question that cleared its threshold stays `auto`. The Flow is not parked, so it can run on HTTP and without `durable`.
+
+```typescript
+export const triage = ai.decision("triage", {
+  onUncertain: "abstain",
+  ask: {
+    urgent: ai.boolean("Does this need a person today?"),
+  },
+});
+```
+
+</Tab>
+
+<Tab value="Autonomy">
+
+`autonomy` does not replace `review` or `abstain`. It is the budget the lockfile is allowed
+to spend. Without a matching certificate, every question still takes the non-auto path.
+
+```typescript
+export const triage = ai.decision("triage", {
+  review: ops,
+  autonomy: { maxError: 0.05, audit: 0.1, risk: 0.1 },
+  evals: "evals/triage.jsonl",
+  ask: {
+    team: ai.choice("Which team owns this ticket?", {
+      billing: "Billing",
+      technical: "Technical",
+    }),
+  },
+});
+```
+
+</Tab>
+
+<Tab value="Locale">
+
+Return a string to keep a separate certificate slice. `undefined` uses the default slice.
+An input whose locale has no slice is not `auto`.
+
+```typescript
+export const triage = ai.decision("triage", {
+  review: ops,
+  locale: (input) => {
+    const row = input as { locale?: string };
+    return row.locale;
+  },
+  ask: {
+    team: ai.choice("Which team owns this ticket?", {
+      billing: "Billing",
+      technical: "Technical",
+    }),
+  },
+});
+```
+
+</Tab>
+
+</Tabs>
+
+## Provider
+
+The key is a Vault secret. `fx.decide` reads `OPENROUTER_API_KEY`, or `TYPESAFE_API_KEY` when
+`driverId` is `"typesafe"`. Declare that secret on the Flow when you write an `effects` block.
+An empty value throws `DecisionConfigError`.
+
+| `driverId`   | Default model                                                                                     | Secret               |
+| ------------ | ------------------------------------------------------------------------------------------------- | -------------------- |
+| `openrouter` | `typesafe/jev-1.13` (the response `model` is a dated pin, currently `typesafe/jev-1.13-20260917`) | `OPENROUTER_API_KEY` |
+| `typesafe`   | `jev-1.13.0`                                                                                      | `TYPESAFE_API_KEY`   |
+
+The deadline defaults to 30 seconds. `400`, `401`, `402`, `403`, `404`, and `422` throw `DecisionRequestError` and leave the breaker closed.
+`429` and `529` retry up to 3 attempts and honor `Retry-After`. A wait over 60 seconds is an outage.
+
+Network errors, timeouts, and HTTP 5xx count toward the breaker. Three failures open it for 30 seconds.
+An open breaker is an outage: review parks, abstain returns `null`.
+
+## Troubleshooting
+
+<Accordions>
+
+<Accordion title='declare exactly one of review or onUncertain: "abstain"'>
+  The decision named both, or neither. Keep one. `autonomy` does not replace either.
+</Accordion>
+
+<Accordion title="review cannot run in HTTP flow">
+  `ai.decision("triage") review cannot run in HTTP flow "support.open" — a park answers 204. fx.emit
+  a signal and decide in a consumer.` Move `fx.decide` to a signal or clock Flow with `durable:
+  true`.
+</Accordion>
+
+<Accordion title="must set durable: true to review">
+  Compile: `ai.decision("triage"): flow "support.route" must set durable: true to review`. Runtime,
+  if it still runs: `fx.decide: "triage" review requires a durable journal`. Abstain is the mode
+  that can skip `durable`.
+</Accordion>
+
+<Accordion title="choice, score, and duplicate names">
+  `choice "team" has 255 options; max 254`. `choice "team" must not set none_of_these` — that option
+  is added for you. `score "urgency" needs 2–10 levels`. `question id "$" is reserved`. `duplicate
+  decision name` when two declarations share a name. `ask is empty` when `ask` has no questions.
+</Accordion>
+
+<Accordion title="OKE1010 UNDECLARED_DECIDE">
+  Cause: `Flow "support.route" decides "triage" without declaring it.` Add `triage` to
+  `effects.decides`, or drop the hand-written `effects` block so inference can see `fx.decide`.
+</Accordion>
+
+<Accordion title='fx.decide: secret "OPENROUTER_API_KEY" is not configured'>
+  `DecisionConfigError`. The Vault contract is missing or empty. TypeSafe looks for
+  `TYPESAFE_API_KEY` instead. This is not an outage, and it does not park.
+</Accordion>
+
+<Accordion title="400, 401, 402, 403, 404, or 422 from the provider">
+  `DecisionRequestError`. 400 and 422 are a body the provider rejected. 401 is the key. 402 is
+  billing. 403 is forbidden. 404 is an unknown model or path. The breaker stays closed. Fix the
+  request. Do not retry as if the provider were down.
+</Accordion>
+
+<Accordion title="oke boot: decisions are declared but rootDir and OKE_ROOT_DIR are unset">
+  `oke boot: decisions are declared but rootDir and OKE_ROOT_DIR are unset, so the decision lockfile
+  cannot load.` Boot with `rootDir`, or set `OKE_ROOT_DIR` to the directory that holds
+  `oke-decisions.lock.json`.
+</Accordion>
+
+<Accordion title="autonomy requires audit">
+  `ai.decision("triage"): autonomy requires audit`. `autonomy` without `audit` fails to compile. Set
+  `audit` from 0 to 1. `maxError` is required in the same object.
+</Accordion>
+
+<Accordion title="options must be an object literal">
+  `ai.decision("triage"): choice "team" options must be an object literal`. Write the options
+  inline. A variable is invisible to the compiler. The same applies when the review gate name cannot
+  be resolved: `ai.decision: review gate name could not be resolved`.
+</Accordion>
+
+<Accordion title="The candidate route is missing">
+  The candidate route is registered only when some decision declares `autonomy`. Without that block
+  there is no lockfile grant to promote, and `GET /_oke/decisions/:name/candidate` is not mounted.
+</Accordion>
+
+<Accordion title="Conflict or JournalLeaseBusy">
+  A second resolve after the review is stored is `Conflict`. Do not retry it. A lost race is
+  `JournalLeaseBusy` with `Retry-After`. Retry only that code. A suggested wait over 60 seconds
+  comes back as 503.
+</Accordion>
+
+<Accordion title="The certificate went stale after an edit">
+  Changing the question text or its options changes the hash. The runtime then refuses `auto`
+  (`stale-hash`) until `oke eval --certify` or `oke decide promote` writes a new lock entry. The old
+  threshold is not reused.
+</Accordion>
+
+</Accordions>
+
+## Learn more
+
+- [Events](/docs/elements/ai/events) — tool approval uses the same 409 split
+- [Gate](/docs/elements/gate) — the policy on `review`
+- [fx](/docs/reference/fx) — `fx.decide` effect row
+- [CLI](/docs/reference/cli) — `oke decide promote` and `oke eval --certify`
+
+## Next
+
+<Cards>
+  <Card
+    title="Events"
+    description="Stream an agent run as AG-UI events."
+    href="/docs/elements/ai/events"
+  />
+  <Card
+    title="Agents"
+    description="Bounded tool loops with fx.run."
+    href="/docs/elements/ai/agents"
+  />
+  <Card title="AI" description="Models, prompts, agents, and decisions." href="/docs/elements/ai" />
+</Cards>