@latimer-woods-tech/llm 0.2.0 → 0.3.1

package/CHANGELOG.md

@@ -1,16 +1,81 @@
-## [Unreleased]
+# Changelog
 
-## [0.2.0] - 2026-04-29
+## 0.3.1 — 2026-05-02 PM
+
+### Added (no breaking changes)
+
+- **Grok opt-in provider.** `LLMProvider` now includes `'grok'`. Call with `{ model: 'grok-4-fast' }` or `{ model: 'grok-3-mini-latest' }` to route to xAI. Not in default tier routing — explicit model override only.
+- `LLMEnv.GROK_API_KEY` is **optional**; required only when a caller opts in via a `grok-*` model override.
+- `MODELS.grok.{fast, mini}` added to the catalogue.
+
+### Rationale
+
+Reversal of the 0.3.0 partial decision (D1 in `docs/supervisor/DECISIONS.md`): Grok stays available for workloads that benefit from its quirks — cheap experimental prompting in the xico-city user economy, artist-platform surface, etc. It does not compete with Anthropic / Gemini / Groq for the tier slots; it sits alongside as a per-call opt-in.
+
+### Consumers
+
+- Downstream packages do not need to declare `GROK_API_KEY` unless a route explicitly sends Grok model overrides. Existing code that dropped the key on 0.3.0 migration continues to work.
+
+
+## 0.3.0 — 2026-05-02
+
+### Breaking
+
+- **Grok removed.** `LLMProvider` no longer includes `grok`; `LLMEnv.GROK_API_KEY` removed.
+- **AI Gateway mandatory.** `LLMEnv.AI_GATEWAY_BASE_URL` is required. All provider traffic flows through
+  Cloudflare AI Gateway for unified logging, rate limiting, and cost telemetry.
+- **Tier-based routing.** `LLMOptions.tier` (`fast | balanced | smart | verifier`) replaces the
+  flat Anthropic → Grok → Groq failover chain. Routing is workload-split:
+  - `fast` → Anthropic Haiku (latency-sensitive, short completions)
+  - `balanced` → Anthropic Sonnet (default)
+  - `smart` → Anthropic Opus; Gemini 2.5 Pro for long-context (>150k tokens est.)
+  - `verifier` → Groq Llama 3.3 70B (cheap second opinion, no fallback)
+- **Dependency declarations fixed.** `@latimer-woods-tech/errors` and `@latimer-woods-tech/logger`
+  now declared as `^0.2.0` instead of `file:../*` so external consumers can install the package.
 
 ### Added
-- Multi-provider LLM completion chain with Anthropic primary, Grok fallback, and Groq tertiary fallback.
-- Streaming support through the Anthropic-compatible provider path.
-- `withSystem()` helper for consistently applying system prompts at call sites.
-- Quality-gated package build with lint, typecheck, coverage, and ESM output.
 
-### Fixed
-- Restored `@latimer-woods-tech/errors` and `@latimer-woods-tech/logger` as runtime dependencies so published consumers resolve package imports correctly.
+- Gemini 2.5 Pro via Vertex AI as long-context fallback (`VERTEX_ACCESS_TOKEN`, `VERTEX_PROJECT`, `VERTEX_LOCATION`).
+- Anthropic prompt caching (auto-enabled for system prompts ≥ 4096 chars; override via `LLMOptions.promptCache`).
+- Per-call cancellation via `LLMOptions.signal: AbortSignal`.
+- 3-attempt exponential backoff (250ms · 2^n + jitter) on retryable status (408, 425, 429, 5xx).
+- Ledger stamping: `LLMOptions.{runId, project, actor}` propagated to logger for downstream
+  consumption by `@latimer-woods-tech/llm-meter`.
+- `LLMResult.{model, tier, attempts, gatewayRequestId, tokens.cacheRead, tokens.cacheWrite}` for
+  observability and budget accounting.
+
+### Changed
+
+- Default model constants renamed and grouped under exported `MODELS` catalogue. Kept in sync with
+  `docs/architecture/FACTORY_V1.md § LLM substrate`.
+- Empty provider responses now surface as leg failures and trigger fallback rather than returning
+  an empty `LLMResult`.
+
+### Migration
+
+Replace:
+
+```ts
+const res = await complete(msgs, { ANTHROPIC_API_KEY, GROK_API_KEY, GROQ_API_KEY }, { model: 'claude-sonnet-4-...' });
+```
+
+With:
+
+```ts
+const res = await complete(
+  msgs,
+  {
+    AI_GATEWAY_BASE_URL: env.AI_GATEWAY_BASE_URL,
+    ANTHROPIC_API_KEY: env.ANTHROPIC_API_KEY,
+    GROQ_API_KEY: env.GROQ_API_KEY,
+    VERTEX_ACCESS_TOKEN: env.VERTEX_ACCESS_TOKEN, // minted via JWT-bearer flow
+    VERTEX_PROJECT: env.VERTEX_PROJECT,
+    VERTEX_LOCATION: env.VERTEX_LOCATION,
+  },
+  { tier: 'balanced', runId, project: 'prime-self', actor: 'worker' },
+);
+```
+
+## 0.2.0
 
-### Verification
-- `npm install` regenerated the package lock and ran the package prepublish gate on Apr 29, 2026.
-- `npm run lint`, `npm run typecheck`, `npm test -- --coverage`, and `npm run build` passed during the prepublish gate.
+Initial public release of the failover orchestrator.

package/README.md

@@ -1,3 +1,60 @@
-# @factory/llm
+# @latimer-woods-tech/llm
 
-LLM failover orchestration across Factory AI providers.
+Tier-routed LLM orchestration for the Factory platform, with Cloudflare AI Gateway, Anthropic
+primary, Gemini 2.5 Pro long-context fallback, and Groq verifier.
+
+## Routing (0.3.0)
+
+| Tier | Primary | Fallback | Notes |
+|---|---|---|---|
+| `fast` | Claude Haiku 4 | — | latency-sensitive, short completions |
+| `balanced` *(default)* | Claude Sonnet 4 | Gemini 2.5 Pro | swaps to Gemini when est. tokens ≥ 150k |
+| `smart` | Claude Opus 4 | Gemini 2.5 Pro | ditto; tools, long reasoning |
+| `verifier` | Groq Llama 3.3 70B | — | cheap second opinion; no fallback |
+
+All traffic flows through `AI_GATEWAY_BASE_URL`. The gateway handles caching, rate-limit shedding,
+and per-project cost telemetry.
+
+## Usage
+
+```ts
+import { complete } from '@latimer-woods-tech/llm';
+
+const ctl = new AbortController();
+setTimeout(() => ctl.abort(), 30_000);
+
+const res = await complete(
+  [{ role: 'user', content: 'Draft a release note for llm@0.3.0.' }],
+  {
+    AI_GATEWAY_BASE_URL: env.AI_GATEWAY_BASE_URL,
+    ANTHROPIC_API_KEY: env.ANTHROPIC_API_KEY,
+    GROQ_API_KEY: env.GROQ_API_KEY,
+    VERTEX_ACCESS_TOKEN: env.VERTEX_ACCESS_TOKEN,
+    VERTEX_PROJECT: env.VERTEX_PROJECT,
+    VERTEX_LOCATION: env.VERTEX_LOCATION,
+  },
+  {
+    tier: 'balanced',
+    signal: ctl.signal,
+    runId: 'sup-run-0042',
+    project: 'prime-self',
+    actor: 'supervisor',
+  },
+);
+
+if (res.ok) {
+  console.log(res.data.content, res.data.provider, res.data.tokens);
+}
+```
+
+## Vertex access token
+
+Minted via the JWT-bearer flow from a GCP service account. See
+`docs/runbooks/rotate-gcp-sa.md` for the mint procedure. Tokens are short-lived (1h); callers
+should refresh before every cold start or every 50 minutes, whichever comes first.
+
+## Budget
+
+Every call emits a `llm.complete` log line with `{ provider, model, tier, tokens, runId, project,
+actor }`. `@latimer-woods-tech/llm-meter` (0.1.0+) consumes these lines to enforce the per-run
+$5 hard cap and the per-project steady-state budget.

package/dist/index.d.mts

@@ -8,38 +8,83 @@ interface LLMMessage {
     role: 'user' | 'assistant' | 'system';
     content: string;
 }
+/**
+ * Quality tier selected by the caller. Routing is workload-split:
+ *  - `fast`      → Anthropic Haiku (short, latency-sensitive)
+ *  - `balanced`  → Anthropic Sonnet (default)
+ *  - `smart`     → Anthropic Opus OR Gemini 2.5 Pro if input is long-context (>150k tokens estimated)
+ *  - `verifier`  → Groq Llama (cheap second opinion; only used from verifier code path)
+ */
+type LLMTier = 'fast' | 'balanced' | 'smart' | 'verifier';
 /**
  * Options that influence LLM completion behaviour.
  */
 interface LLMOptions {
+    /** Quality tier; see {@link LLMTier}. Defaults to `balanced`. */
+    tier?: LLMTier;
+    /** Explicit model override. Takes precedence over tier. */
     model?: string;
     maxTokens?: number;
     temperature?: number;
     system?: string;
+    /** Token budget above which we force long-context routing (Gemini). */
+    longContextThreshold?: number;
+    /** Per-call cancellation signal. Aborts the in-flight provider request. */
+    signal?: AbortSignal;
+    /** Optional run identifier stamped on ledger rows + logs. */
+    runId?: string;
+    /** Optional project identifier stamped on ledger rows + logs. */
+    project?: string;
+    /** Optional actor identifier (supervisor / worker / human). */
+    actor?: string;
+    /** Anthropic prompt-cache control. Defaults to `true` for `system` prompts ≥ 1024 tokens. */
+    promptCache?: boolean;
 }
 /**
- * Provider that produced an LLM response.
+ * Provider that produced an LLM response. `grok` removed in 0.3.0.
  */
-type LLMProvider = 'anthropic' | 'grok' | 'groq';
+type LLMProvider = 'anthropic' | 'gemini' | 'groq' | 'grok';
 /**
  * Result returned by a successful completion.
  */
 interface LLMResult {
     content: string;
     provider: LLMProvider;
+    model: string;
+    tier: LLMTier;
     tokens: {
         input: number;
         output: number;
+        cacheRead?: number;
+        cacheWrite?: number;
     };
     latency: number;
+    /** Number of attempts before success (1 = primary succeeded). */
+    attempts: number;
+    /** Monotonic request id from AI Gateway, if present in headers. */
+    gatewayRequestId?: string;
 }
 /**
  * Environment bindings required by {@link complete}.
+ *
+ * `AI_GATEWAY_BASE_URL` is REQUIRED in 0.3.0. All provider calls flow through the
+ * Cloudflare AI Gateway for unified logging, rate limiting, and cost telemetry.
+ * In test/dev the caller may pass a custom fetch impl that short-circuits this.
  */
 interface LLMEnv {
+    AI_GATEWAY_BASE_URL: string;
     ANTHROPIC_API_KEY: string;
-    GROK_API_KEY: string;
     GROQ_API_KEY: string;
+    /** Optional — only required when caller passes `{ model: 'grok-*' }` override. */
+    GROK_API_KEY?: string;
+    /**
+     * Google Cloud short-lived access token with `aiplatform.endpoints.predict`.
+     * Callers mint this via the JWT-bearer flow (service account → token exchange);
+     * see `docs/runbooks/rotate-gcp-sa.md`. Token must be valid for ≥ 5 minutes.
+     */
+    VERTEX_ACCESS_TOKEN: string;
+    VERTEX_PROJECT: string;
+    VERTEX_LOCATION: string;
 }
 /**
  * Optional dependencies for {@link complete}.
@@ -49,39 +94,113 @@ interface LLMDeps {
     logger?: Logger;
     now?: () => number;
 }
+declare const MODELS: {
+    readonly anthropic: {
+        readonly fast: "claude-haiku-4-20250514";
+        readonly balanced: "claude-sonnet-4-6";
+        readonly smart: "claude-opus-4-7";
+    };
+    readonly gemini: {
+        readonly smart: "gemini-2.5-pro";
+    };
+    readonly groq: {
+        readonly verifier: "llama-3.3-70b-versatile";
+    };
+    readonly grok: {
+        /** Opt-in only via `{ model: 'grok-*' }`. Not in default tier routing. */
+        readonly fast: "grok-4-fast";
+        readonly mini: "grok-3-mini-latest";
+    };
+};
+/** Cooldown duration in ms after a provider exhausts all retries. */
+declare const PROVIDER_COOLDOWN_MS = 30000;
+/**
+ * Returns `true` if the provider is currently in its cooldown window.
+ * Uses the injected `now` function (or `Date.now`) for testability.
+ */
+declare function isProviderCoolingDown(provider: LLMProvider, now?: () => number): boolean;
 /**
- * Runs a completion through the Anthropic → Grok → Groq failover chain.
+ * Marks a provider as cooling down for {@link PROVIDER_COOLDOWN_MS} milliseconds.
+ */
+declare function markProviderCoolingDown(provider: LLMProvider, now?: () => number): void;
+/**
+ * Clears the cooldown state for a provider after a successful call.
+ */
+declare function clearProviderCooldown(provider: LLMProvider): void;
+declare const BASE_BACKOFF_MS = 250;
+/**
+ * Run a completion through the routing plan for the requested tier.
+ *
+ * Routing summary (0.3.0):
+ *   - `fast`     → Anthropic Haiku
+ *   - `balanced` → Anthropic Sonnet; Gemini 2.5 Pro if `longContextThreshold` exceeded
+ *   - `smart`    → Anthropic Opus; Gemini 2.5 Pro if long-context
+ *   - `verifier` → Groq Llama 3.3 70B (no fallback — verifier is inherently cheap/best-effort)
+ *
+ * All provider traffic flows through Cloudflare AI Gateway at `AI_GATEWAY_BASE_URL`.
+ *
+ * Per-provider reliability guarantees (0.4.0):
+ *   - Exponential backoff with jitter on 429 / 5xx (base 500ms, cap 8s, up to 2 retries).
+ *   - Provider cooldown: after exhausting retries the provider is marked cooling down
+ *     for 30 seconds; subsequent calls skip it and go straight to the fallback leg.
  *
  * @param messages - Ordered chat history.
- * @param env - API key bindings.
- * @param opts - Optional model/parameters override.
+ * @param env - API key + gateway bindings.
+ * @param opts - Optional tier/model/parameters override.
  * @param deps - Optional fetch/logger/clock injection (for testing).
  * @returns A {@link FactoryResponse} carrying either an {@link LLMResult} or
- *   an `LLM_ALL_PROVIDERS_FAILED` error.
+ *   an error (`LLM_ALL_PROVIDERS_FAILED`, `LLM_RATE_LIMITED`, or `INTERNAL_ERROR`).
  */
 declare function complete(messages: LLMMessage[], env: LLMEnv, opts?: LLMOptions, deps?: LLMDeps): Promise<FactoryResponse<LLMResult>>;
 /**
- * Streams a completion from Anthropic. No failover is performed for streaming
- * responses; callers should fall back to {@link complete} on failure.
+ * Streams a completion from the primary Anthropic provider, yielding text chunks
+ * as they arrive. Falls back to the non-streaming {@link complete} function when
+ * the provider does not support streaming (i.e. a non-Anthropic primary is selected).
+ *
+ * The generator's **return value** (accessible via `gen.return()` or by consuming
+ * the full iteration) is an {@link LLMResult} with the same shape as {@link complete}.
+ *
+ * Usage pattern:
+ * ```ts
+ * const gen = completionStream(messages, env, opts);
+ * for await (const chunk of gen) {
+ *   // stream chunk to client
+ * }
+ * const result = (await gen.return(undefined)).value; // LLMResult
+ * ```
  *
  * @param messages - Ordered chat history.
- * @param env - Anthropic API key binding.
- * @param opts - Optional model/parameters override.
- * @param deps - Optional fetch override (for testing).
- * @returns The raw Anthropic streaming response body.
+ * @param env - API key + gateway bindings.
+ * @param opts - Optional tier/model/parameters override. Accepts `deps` as nested field.
+ * @returns An async generator that yields `string` chunks and returns an {@link LLMResult}.
  */
-declare function stream(messages: LLMMessage[], env: {
-    ANTHROPIC_API_KEY: string;
-}, opts?: LLMOptions, deps?: {
-    fetch?: typeof fetch;
-}): Promise<ReadableStream<Uint8Array>>;
+declare function completionStream(messages: LLMMessage[], env: LLMEnv, opts?: LLMOptions & {
+    deps?: LLMDeps;
+}): AsyncGenerator<string, LLMResult, unknown>;
 /**
- * Returns a {@link complete}-compatible function with a system prompt
- * pre-bound, so callers can hand around a domain-specific shortcut.
+ * Returns `true` if `response` contains at least one verbatim phrase of at
+ * least 5 consecutive whitespace-delimited tokens that also appears in one of
+ * the `sources` strings.
+ *
+ * Returns `true` unconditionally when `sources` is empty (no grounding
+ * documents means grounding cannot be violated).
+ *
+ * This is a lightweight guard for RAG pipelines — it detects obvious
+ * hallucinations where the model generates content not present in any
+ * retrieved source. It is NOT a semantic similarity check.
+ *
+ * @param response - The LLM-generated text to inspect.
+ * @param sources - Retrieved source documents to check against.
+ * @returns `true` if the response is grounded, `false` if hallucination detected.
  *
- * @param system - System prompt to prepend to every call.
- * @returns A function that invokes {@link complete} with `system` injected.
+ * @example
+ * ```ts
+ * const grounded = assertGrounding(llmAnswer, retrievedDocs);
+ * if (!grounded) {
+ *   // flag or re-rank the response
+ * }
+ * ```
  */
-declare function withSystem(system: string): (messages: LLMMessage[], env: LLMEnv, opts?: LLMOptions, deps?: LLMDeps) => Promise<FactoryResponse<LLMResult>>;
+declare function assertGrounding(response: string, sources: string[]): boolean;
 
-export { type LLMDeps, type LLMEnv, type LLMMessage, type LLMOptions, type LLMProvider, type LLMResult, complete, stream, withSystem };
+export { BASE_BACKOFF_MS, type LLMDeps, type LLMEnv, type LLMMessage, type LLMOptions, type LLMProvider, type LLMResult, type LLMTier, MODELS, PROVIDER_COOLDOWN_MS, assertGrounding, clearProviderCooldown, complete, completionStream, isProviderCoolingDown, markProviderCoolingDown };