@ljoukov/llm 2.1.0 → 3.0.0

package/README.md

@@ -9,7 +9,8 @@ Unified TypeScript wrapper over:
 
 - **OpenAI Responses API** (`openai`)
 - **Google Gemini via Vertex AI** (`@google/genai`)
-- **ChatGPT subscription models** via `chatgpt-*` model ids (requires `CHATGPT_AUTH_JSON_B64`)
+- **Fireworks chat-completions models** (`kimi-k2.5`, `glm-5`, `minimax-m2.1`)
+- **ChatGPT subscription models** via `chatgpt-*` model ids (reuses Codex auth store, or a token provider)
 
 Designed around a single streaming API that yields:
 
@@ -69,12 +70,27 @@ If deploying to Cloudflare Workers/Pages:
 jq -c . < path/to/service-account.json | wrangler secret put GOOGLE_SERVICE_ACCOUNT_JSON
 ```
 
+### Fireworks
+
+- `FIREWORKS_TOKEN` (or `FIREWORKS_API_KEY`)
+
 ### ChatGPT subscription models
 
-- `CHATGPT_AUTH_JSON_B64`
+By default, `chatgpt-*` models reuse the ChatGPT OAuth tokens stored by the Codex CLI:
+
+- `${CODEX_HOME:-~/.codex}/auth.json`
+
+If you deploy to multiple environments (Vercel, GCP, local dev, etc.), use a centralized HTTPS token provider that owns
+refresh-token rotation and serves short-lived access tokens.
 
-This is a base64url-encoded JSON blob containing the ChatGPT OAuth tokens + account id (RFC 4648):
-https://www.rfc-editor.org/rfc/rfc4648
+- `CHATGPT_AUTH_TOKEN_PROVIDER_URL` (example: `https://chatgpt-auth.<your-domain>`)
+- `CHATGPT_AUTH_API_KEY` (shared secret; sent as `Authorization: Bearer ...` and `x-chatgpt-auth: ...`)
+- `CHATGPT_AUTH_TOKEN_PROVIDER_STORE` (`kv` or `d1`, defaults to `kv`)
+
+This repo includes a Cloudflare Workers token provider implementation in `workers/chatgpt-auth/`.
+
+If `CHATGPT_AUTH_TOKEN_PROVIDER_URL` + `CHATGPT_AUTH_API_KEY` are set, `chatgpt-*` models will fetch tokens from the
+token provider and will not read the local Codex auth store.
 
 ## Usage
 
@@ -245,6 +261,21 @@ const result = await generateText({
 console.log(result.text);
 ```
 
+### Fireworks
+
+Use Fireworks model ids directly (for example `kimi-k2.5`, `glm-5`, `minimax-m2.1`):
+
+```ts
+import { generateText } from "@ljoukov/llm";
+
+const result = await generateText({
+  model: "kimi-k2.5",
+  input: "Return exactly: OK",
+});
+
+console.log(result.text);
+```
+
 ### ChatGPT subscription models
 
 Use a `chatgpt-` prefix:
@@ -348,14 +379,21 @@ const { value } = await generateJson({
 
 ## Tools
 
-This library supports two kinds of tools:
+There are three tool-enabled call patterns:
 
-- Model tools (server-side): `web-search` and `code-execution`
-- Your tools (JS/TS code): use `runToolLoop()` and `tool()`
+1. `generateText()` for provider-native/server-side tools (for example web search).
+2. `runToolLoop()` for your runtime JS/TS tools (function tools executed in your process).
+3. `runAgentLoop()` for filesystem tasks (a convenience wrapper around `runToolLoop()`).
 
-### Model tools (web search / code execution)
+Architecture note:
 
-These tools run on the provider side.
+- Filesystem tools are not a separate execution system.
+- `runAgentLoop()` constructs a filesystem toolset, merges your optional custom tools, then calls the same `runToolLoop()` engine.
+- This behavior is model-agnostic at API level; profile selection only adapts tool shape for model compatibility.
+
+### Provider-Native Tools (`generateText()`)
+
+Use this when the model provider executes the tool remotely (for example search/code-exec style tools).
 
 ```ts
 import { generateText } from "@ljoukov/llm";
@@ -369,9 +407,9 @@ const result = await generateText({
 console.log(result.text);
 ```
 
-### Your tools (function calling)
+### Runtime Tools (`runToolLoop()`)
 
-`runToolLoop()` runs a simple function-calling loop until the model returns a final answer or the step limit is hit.
+Use this when the model should call your local runtime functions.
 
 ```ts
 import { runToolLoop, tool } from "@ljoukov/llm";
@@ -392,47 +430,24 @@ const result = await runToolLoop({
 console.log(result.text);
 ```
 
-### Built-in `apply_patch` tool
+Use `customTool()` only when you need freeform/non-JSON tool input grammar.
 
-The library includes a Codex-style `apply_patch` tool with a pluggable filesystem adapter.
+### Filesystem Tasks (`runAgentLoop()`)
 
-```ts
-import {
-  createApplyPatchTool,
-  createInMemoryAgentFilesystem,
-  runToolLoop,
-} from "@ljoukov/llm";
+Use this for read/search/write tasks in a workspace. The library auto-selects filesystem tool profile by model when `profile: "auto"`:
 
-const fs = createInMemoryAgentFilesystem({
-  "/repo/index.ts": "export const value = 1;\n",
-});
+- Codex-like models: Codex-compatible filesystem tool shape.
+- Gemini models: Gemini-compatible filesystem tool shape.
+- Other models: model-agnostic profile (currently Gemini-style).
 
-const result = await runToolLoop({
-  model: "chatgpt-gpt-5.3-codex",
-  input: "Use apply_patch to change value from 1 to 2.",
-  tools: {
-    apply_patch: createApplyPatchTool({
-      cwd: "/repo",
-      fs,
-      checkAccess: ({ path }) => {
-        if (!path.startsWith("/repo/")) {
-          throw new Error("Writes are allowed only inside /repo");
-        }
-      },
-    }),
-  },
-});
-
-console.log(result.text);
-```
+Confinement/policy is set through `filesystemTool.options`:
 
-### `runAgentLoop()` with model-aware filesystem tools
+- `cwd`: workspace root for path resolution.
+- `fs`: backend (`createNodeAgentFilesystem()` or `createInMemoryAgentFilesystem()`).
+- `checkAccess`: hook for allow/deny policy + audit.
+- `allowOutsideCwd`: opt-out confinement (default is false).
 
-Use `runAgentLoop()` when you want a default filesystem toolset chosen by model:
-
-- Codex-like models -> `apply_patch`, `read_file`, `list_dir`, `grep_files`
-- Gemini models -> `read_file`, `write_file`, `replace`, `list_directory`, `grep_search`, `glob`
-- Other models -> model-agnostic (Gemini-style) set by default
+Detailed reference: `docs/agent-filesystem-tools.md`.
 
 ```ts
 import { createInMemoryAgentFilesystem, runAgentLoop } from "@ljoukov/llm";
@@ -442,7 +457,7 @@ const fs = createInMemoryAgentFilesystem({
 });
 
 const result = await runAgentLoop({
-  model: "chatgpt-gpt-5.3-codex",
+  model: "chatgpt-gpt-5.3-codex-spark",
   input: "Change value from 1 to 2 using filesystem tools.",
   filesystemTool: {
     profile: "auto",
@@ -456,14 +471,42 @@ const result = await runAgentLoop({
 console.log(result.text);
 ```
 
-## Agent benchmark (micro)
+If you need exact control over tool definitions, build the filesystem toolset yourself and call `runToolLoop()` directly.
 
-For small edit-harness experiments with `chatgpt-gpt-5.3-codex`:
+```ts
+import {
+  createFilesystemToolSetForModel,
+  createInMemoryAgentFilesystem,
+  runToolLoop,
+} from "@ljoukov/llm";
+
+const fs = createInMemoryAgentFilesystem({ "/repo/a.ts": "export const n = 1;\n" });
+const tools = createFilesystemToolSetForModel("chatgpt-gpt-5.3-codex-spark", {
+  cwd: "/repo",
+  fs,
+});
+
+const result = await runToolLoop({
+  model: "chatgpt-gpt-5.3-codex-spark",
+  input: "Update n to 2.",
+  tools,
+});
+```
+
+## Agent benchmark (filesystem extraction)
+
+For filesystem extraction/summarization evaluation across Codex, Fireworks, and Gemini models:
 
 ```bash
 npm run bench:agent
 ```
 
+Standard full refresh (all tasks, auto-write `LATEST_RESULTS.md`, refresh `traces/latest`, prune old traces):
+
+```bash
+npm run bench:agent:latest
+```
+
 Estimate-only:
 
 ```bash