okengine 0.18.4 → 0.19.0

package/site/content/docs/elements/ai.mdx

@@ -1,385 +0,0 @@
----
-title: "AI"
-description: "Reaching machine intelligence — models, versioned prompts, embeddings, and bounded agents, with cost and PII egress as first-class constraints."
-icon: "Bot"
-source: "docs/spec/unified-theory.md"
----
-
-AI is how your app calls **machine intelligence**: triage a ticket, embed a knowledge base, let an agent resolve a support case. Model calls differ from every other effect — they are nondeterministic, they cost money per call, and your data leaves the building — so prompts are **versioned artifacts**, not strings buried in a handler.
-
-<Callout title="The one rule">
-  The production model is **declared, never guessed** — there is no prod default. Development uses
-  `mock` for determinism, and sending PII to a third-party model fails the **build** unless you
-  explicitly acknowledge it.
-</Callout>
-
-## Quick start
-
-<Steps>
-
-<Step>
-### Declare a model and a prompt
-
-The model is a logical binding (`smart`, `fast`); the prompt is a versioned artifact with a typed output:
-
-```typescript title="src/ai.ts"
-import { ai } from "okengine";
-import { z } from "zod";
-
-export const smart = ai.model("smart", { provider: "anthropic", tier: "opus" });
-
-export const triage = smart.prompt("ticket-triage", {
-  in: z.object({ subject: z.string(), body: z.string() }),
-  out: z.object({ urgency: z.enum(["low", "high"]), team: z.string(), summary: z.string() }),
-  version: 3,
-  budget: { maxCostPerCall: 0.02 },
-});
-```
-
-</Step>
-
-<Step>
-### Ask it from a Flow
-
-`fx.ask` routes the call through the declared model and validates the response against `out`:
-
-```typescript title="src/flows/support/triage.ts"
-do: async (input, fx) => {
-  const result = await fx.ask(triage, { subject: input.subject, body: input.body });
-  // result is typed by the prompt's `out` schema
-};
-```
-
-</Step>
-
-<Step>
-### Gate regressions in CI
-
-Point the prompt at an eval set and it becomes a CI check — a prompt change that degrades answers fails the build like a broken test:
-
-```typescript
-export const triage = smart.prompt("ticket-triage", {
-  // …
-  evals: "./evals/triage.jsonl", // run with `oke eval`
-});
-```
-
-</Step>
-
-</Steps>
-
-## The building blocks
-
-| Declaration                | Produces                                                                        |
-| -------------------------- | ------------------------------------------------------------------------------- |
-| `ai.model(name, opts)`     | Logical model binding — provider / tier / model id / optional `driverId`        |
-| `model.prompt(name, opts)` | Versioned prompt with typed in/out, evals, budget                               |
-| `ai.embed(name, opts)`     | Embedding pipeline into a `store.index` (searched via `fx.search`)              |
-| `ai.agent(name, opts)`     | Bounded agent whose tools are **your own flows**                                |
-| `ai.mcpServer(name, opts)` | External MCP server — allowlisted tools join `fx.call` as `mcp:<server>/<tool>` |
-
-<AiBlocks />
-
-### Prompt options
-
-| Option       | Type                  | Meaning                                                                                |
-| ------------ | --------------------- | -------------------------------------------------------------------------------------- |
-| `in` / `out` | zod / Standard Schema | Input and output contracts — responses validated against `out`                         |
-| `version`    | number                | Artifact version — diffs and regressions tracked per version. Pins use `name@version`. |
-| `evals`      | string                | Path to a `.jsonl` eval set, regression-gated via `oke eval`                           |
-| `budget`     | object                | `maxCostPerCall` / `maxCostPerRun` — cost, not time                                    |
-| `via`        | `string[]`            | Ordered recovery chain of logical model names for this prompt                          |
-| `timeout`    | `string \| number`    | Deadline — clock duration (`"30s"`) or milliseconds                                    |
-
-## Recovery chain + per-command timeout
-
-Fallback is **recovery**, not a fixed cloud→local doctrine. Declare the chain and deadline on the prompt; the call site stays thin:
-
-```typescript
-export const summarizeNote = smart.prompt("summarize-note", {
-  via: ["smart", "local"],
-  timeout: "30s",
-});
-
-// in a flow
-const out = await fx.ask(summarizeNote, { title, body });
-// out.via is the logical model that answered
-```
-
-Resolution: `fx.ask(…, { via })` overrides `prompt.via`, else the prompt’s bound model. On each model the runtime retries **once** for retryable failures (timeout / 429 / 5xx / network), then advances.
-
-Permanent failures (401 / other 4xx / schema invalid / client disconnect / `maxCostPerCall` exceeded) stop the chain. When every eligible attempt fails, `fx.ask` throws — map that to a typed error in the flow (no silent text excerpt).
-
-`timeout` uses the same duration vocabulary as Clock (`"30s"`, `"2m"`, or a millisecond number). Ask-time `fx.ask(…, { timeout: "10s" })` overrides the prompt. Omit = no artificial cap. `budget.maxCostPerCall` is enforced at runtime (not just observed). Set `driverId` on a model to open a different protocol than the app default.
-
-## Declared guardrails
-
-Cost, egress, and model choice are contracts — not runtime hope. Each guardrail is declared up front; violating one fails the build or the run.
-
-<AiGuardrails />
-
-## Tools are flows (same `fx.call` path)
-
-Offer flows as tools on a one-shot ask — each model-initiated call goes through the host flow’s `fx.call`, so capability, Manifest `calls`, and Runs stay honest. No silent extra authority for the model:
-
-```typescript
-const result = await fx.ask(triage, input, {
-  tools: [getBooking, refundBooking],
-  maxSteps: 6,
-});
-```
-
-External MCP tools take the same path. In OKE, **MCP is a surface of the model, not the model itself** — `ai.mcpServer` models the client role (consuming external tools), while `mcp.tool()` + Gate models the provider role (exposing your flows to agents). Declare a server with a **required** allowlist, then pass `server.tool("…")` — never whatever `tools/list` happens to expose:
-
-```typescript
-export const github = ai.mcpServer("github", {
-  url: "https://mcp.example/github",
-  auth: { bearer: githubToken },
-  tools: ["create_issue"],
-});
-
-await fx.ask(triage, input, {
-  tools: [github.tool("create_issue")],
-});
-```
-
-Capability refs are `mcp:<server>/<tool>`. An undeclared call throws **OKE1007**. See [MCP](/docs/ai/mcp).
-
-## Agents with real guardrails
-
-An agent's tools are your flows — each carrying its own gates, effects, and typed errors, so the agent can never do anything a flow couldn't:
-
-```typescript
-export const support = ai.agent("support", {
-  model: smart,
-  tools: [getBooking, refundBooking], // flows, with their gates attached
-  maxSteps: 6,
-  budget: { maxCostPerRun: 0.25 },
-});
-```
-
-`maxSteps` bounds the loop; `budget.maxCostPerRun` is enforced after the run. Both are declared, so "the agent ran away" is a violated contract, not a surprise. `fx.run(support, { message })` uses the same tool loop and the same `fx.call` dispatch as `fx.ask(…, { tools })`.
-
-## Rate limits via Gate
-
-AI calls are expensive — reuse `gate.rate`, don’t invent a parallel budget. Presets: `aiRateGate("ask" | "agent" | "embed")` (defaults 20 / 10 / 60 per minute, `keyBy: "user"`). Attach on the HTTP trigger:
-
-```typescript
-on(http.post("/triage").gate(member, aiRateGate("ask")), triageFlow);
-```
-
-Use `keyBy: "ip"` on public unauthenticated AI edges. Cost caps stay on prompt/agent `budget` decls.
-
-## Secrets stay out of prompts
-
-`Redacted` values (e.g. from `fx.vault.get`) stringify to `[redacted]` in provider-facing messages — the same placeholder as logs. Never cleartext to a model.
-
-## Streaming
-
-`fx.stream(model, { prompt, via? })` yields real provider tokens (Ollama NDJSON, OpenAI-compatible SSE). Cancel by aborting the ambient signal — the same one `fx.all` / `fx.race` already use. Drivers without `stream` fail loud (no stub echo).
-
-Return those chunks on HTTP with `fx.json.stream(...)`. The kernel answers `text/event-stream` (`data:` frames, then `data: [DONE]`). JSON flows stay buffered `{ data, error }`.
-
-```typescript
-on(
-  http.post("/complete").public(),
-  flow("chat.complete", {
-    do: (input, fx) => fx.json.stream(fx.stream(smart, { prompt: input.prompt })),
-  }),
-);
-```
-
-Disconnect cancels the provider call automatically (`request.signal` → ambient abort). An `AbortError` from hang-up does not retry or advance `via`. Console Traces stay empty until the stream closes — then the run is one completed row whose duration covers the open stream, not just time-to-first-byte.
-
-## PII cannot leak by accident
-
-At build time, OKE checks which fields each `fx.ask` sends against your store classifications. If a PII field (`.pii()` in your schema) would reach a third-party model, the build fails:
-
-<AiPiiEgress />
-
-```text
-build failed: flow "ticket-triage" sends pii field(s) [email] to a third-party model without allowPii
-```
-
-The escape hatch is explicit acknowledgment — `allowPii: true` (or `pii: "allow"`) on the flow — so egress is a decision that shows up in code review. The `mock` driver and local models are not third-party; the check targets external providers.
-
-## Nondeterminism is priced in
-
-Because `fx.ask` is nondeterministic, the runtime adjusts around it: journaling is **forced** on (durable flows can replay), and auto-caching is disabled for the call. In development and tests the `mock` driver makes model calls deterministic — your suite never hits a network, and CI never flakes on a provider.
-
-## Per-environment drivers
-
-| Driver              | What it is                                            |
-| ------------------- | ----------------------------------------------------- |
-| `mock`              | Deterministic fake — dev + test default               |
-| `anthropic`         | Anthropic API                                         |
-| `openai-compatible` | Any OpenAI-compatible endpoint                        |
-| `bedrock`           | AWS Bedrock                                           |
-| `vertex`            | Google Vertex AI                                      |
-| `ollama`            | Local models — counts as on-premise for the PII check |
-
-There is deliberately **no production default**: `prod` must name a driver, which keeps the model choice visible in `oke.config.ts` where review can see it.
-
-### Setup wizard
-
-```bash title="Terminal"
-oke ai setup
-oke ai setup --provider llama-cpp --yes
-oke ai setup --provider ollama --yes
-```
-
-Writes `drivers.ai`, commented AI overrides in `.env.local` (so `oke
-dev --docker` can write the live compose URL), and AI models into
-`src/core.ts`. Uncomment `.env.local` only for a host-managed endpoint.
-
-**Recommended** defaults to llama.cpp (lightest local footprint). Customize
-offers llama.cpp · Ollama · vLLM · SGLang · cloud providers.
-
-llama.cpp and Ollama: machine banner → model tier / manual (Docker Hub `ai/` vs
-Ollama library). Cloud: API token → model.
-
-`create-oke` customize offers **AI setup** (Recommended llama.cpp · Customize ·
-Off) before install; `--ai` forces the full model wizard.
-
-### Local inference (decision matrix)
-
-| Workload                         | Image recipe / driver                                      |
-| -------------------------------- | ---------------------------------------------------------- |
-| Single-user / dev (default)      | [llama.cpp](/docs/recipes/llama-cpp) + `openai-compatible` |
-| Local library UX                 | [Ollama](/docs/recipes/ollama) + `ollama`                  |
-| Multi-user GPU concurrency       | [vLLM](/docs/recipes/vllm) + `openai-compatible`           |
-| Structured / agent GPU workloads | [SGLang](/docs/recipes/sglang) + `openai-compatible`       |
-| Managed cloud                    | `anthropic` / `openai-compatible` / … (no `images.ai`)     |
-
-All local recipes **loopback-publish only**, pin explicit tags (never
-`latest`), and require curated model sources — never arbitrary untrusted GGUF
-(2026 GGUF-parser CVEs).
-
-### Default local: llama.cpp
-
-```typescript title="oke.config.ts"
-import { defineConfig } from "okengine/config";
-
-export default defineConfig({
-  drivers: {
-    ai: { dev: "openai-compatible", test: "mock", prod: "anthropic" },
-  },
-  images: {
-    ai: "ghcr.io/ggml-org/llama.cpp:server-b10450", // ≥ b8146; never :latest
-  },
-});
-```
-
-```typescript title="src/core.ts"
-export const smart = ai.model("smart", {
-  provider: "openai-compatible",
-  model: process.env.OKE_AI_MODEL ?? "granite3.3:2b", // Docker Hub ai/ id
-});
-```
-
-- **URL:** `http://127.0.0.1:8080/v1` (recipe writes `OKE_AI_URL`).
-- **Model:** curated Docker Hub [`ai/`](https://hub.docker.com/u/ai) id via `OKE_AI_MODEL`.
-- **PII:** local OpenAI-compatible endpoints count as on-premise when they are not third-party cloud.
-
-### Ollama (fully supported alternative)
-
-Native `ollama` driver — `complete`, `stream`, `embed` without an OpenAI shim.
-Pin `ollama/ollama:0.32.13` or newer (≥ **0.17.1**). See
-[Ollama recipe](/docs/recipes/ollama).
-
-### OpenAI-compatible endpoints (one driver)
-
-`openai-compatible` is the shared chat/completions client — OpenAI, Groq,
-Together, OpenRouter, llama.cpp, vLLM, SGLang, LM Studio, and Ollama’s `/v1`
-shim. Only `baseUrl`, `apiKey`, `model`, and optional `headers` change.
-
-| Target              | Typical `baseUrl`                     | Key                                                         |
-| ------------------- | ------------------------------------- | ----------------------------------------------------------- |
-| OpenAI              | `https://api.openai.com/v1` (default) | required (`OPENAI_API_KEY`)                                 |
-| llama.cpp (default) | `http://127.0.0.1:8080/v1`            | optional                                                    |
-| vLLM / SGLang       | `http://127.0.0.1:8000/v1` / `:30000` | optional                                                    |
-| Groq                | `https://api.groq.com/openai/v1`      | required                                                    |
-| Together            | `https://api.together.xyz/v1`         | required                                                    |
-| OpenRouter          | `https://openrouter.ai/api/v1`        | required; optional `HTTP-Referer` / `X-Title` via `headers` |
-| LM Studio / local   | e.g. `http://127.0.0.1:1234/v1`       | optional                                                    |
-| Ollama OpenAI shim  | `http://127.0.0.1:11434/v1`           | optional                                                    |
-
-Prefer the native `ollama` driver for local Ollama (`/api/chat`).
-
-**Consequence:** a missing key against the default OpenAI cloud base fails at open; a custom `baseUrl` may omit the key. HTTP errors always throw — never a silent mock fallback.
-
-## Troubleshooting
-
-<Accordions>
-<Accordion title="Build fails with AiPiiBuildError">
-
-A flow sends a classified PII field to a third-party model. Either stop sending that field (drop it from the `fx.ask` input), or acknowledge the egress with `allowPii: true` on the flow — the error message names the exact fields.
-
-</Accordion>
-<Accordion title="Which model will prod actually call?">
-
-The one you declared. There is no fallback or guess — if `prod` has no AI driver configured, that's a configuration gap to fix, not a silent default.
-
-</Accordion>
-<Accordion title="oke boot: ollama driver needs OKE_AI_URL">
-
-Docker mode writes `OKE_AI_URL` into `.env.local` when `images.ai` is pinned. Leave leftover `OKE_AI_*` pins commented if you want `oke dev` to refresh that URL. Locally, the Ollama default is `http://127.0.0.1:11434`. If asks fail with `OllamaUnavailableError`, confirm the server answers at that URL and that the model is listed under `GET /api/tags` (boot pulls via `POST /api/pull` against the container — do not rely on a host `ollama pull`, which may talk to a different install). Prefer `oke ai setup` to wire driver, commented env overrides, and AI models in `src/core.ts` in one pass.
-
-</Accordion>
-<Accordion title="A prompt edit made answers worse — how do I catch that pre-merge?">
-
-Attach `evals: "./evals/<name>.jsonl"` to the prompt and run `oke eval` in CI. The suite replays the eval cases and fails on regressions, per prompt version.
-
-</Accordion>
-<Accordion title="Call API returns Unavailable ai: unknown prompt">
-
-Capability pins and Call API pass `name@version`. `fx.ask` resolves the bare
-prompt id. A pin that does not match `prompt.version` throws
-`ai: unknown prompt "name@version"`.
-
-</Accordion>
-<Accordion title="OKE1007 when the model calls an MCP tool">
-
-Add `mcp:<server>/<tool>` to that flow’s `effects.calls` — the same token as `fx.call`. The allowlist on `ai.mcpServer` is not a capability grant.
-
-</Accordion>
-<Accordion title="An agent looped and burned budget">
-
-Bound it at declaration: `maxSteps` caps iterations, `budget.maxCostPerRun` caps spend.
-
-`fx.ask` stamps `promptVersion` on the run. Cost is recorded only when the
-driver reports `usage.cost` — tokens land on the ask journal either way.
-
-</Accordion>
-<Accordion title="The client hung up but the model kept generating">
-
-HTTP flows install `request.signal` as the ambient abort. `fx.ask` / `fx.stream` /
-`fx.run` cancel the provider fetch. Return tokens with `fx.json.stream(fx.stream(...))`
-so disconnect is observable.
-
-A hang-up is `AbortError` — it does not retry or advance `via`.
-
-</Accordion>
-</Accordions>
-
-## Learn more
-
-- [Project structure](/docs/get-started/project-structure) — agent `tools` are Flow names (`notes.get`), not URLs
-- [Flow](/docs/elements/flow) — `fx.ask` and `fx.search` inside `do`
-- [Store](/docs/elements/store) — `store.index`, the home of embeddings
-- [MCP](/docs/ai/mcp) — serve the app to agents, or consume external servers as tools
-
-## Next
-
-<Cards>
-  <Card
-    title="Basic Usage"
-    description="Build from the standard starter."
-    href="/docs/get-started/basic-usage"
-  />
-  <Card
-    title="Introduction"
-    description="Eight elements overview."
-    href="/docs/get-started/introduction"
-  />
-</Cards>