@ljoukov/llm 2.0.0 → 3.0.0

package/README.md

@@ -9,7 +9,8 @@ Unified TypeScript wrapper over:
 
 - **OpenAI Responses API** (`openai`)
 - **Google Gemini via Vertex AI** (`@google/genai`)
-- **ChatGPT subscription models** via `chatgpt-*` model ids (requires `CHATGPT_AUTH_JSON_B64`)
+- **Fireworks chat-completions models** (`kimi-k2.5`, `glm-5`, `minimax-m2.1`)
+- **ChatGPT subscription models** via `chatgpt-*` model ids (reuses Codex auth store, or a token provider)
 
 Designed around a single streaming API that yields:
 
@@ -69,12 +70,27 @@ If deploying to Cloudflare Workers/Pages:
 jq -c . < path/to/service-account.json | wrangler secret put GOOGLE_SERVICE_ACCOUNT_JSON
 ```
 
+### Fireworks
+
+- `FIREWORKS_TOKEN` (or `FIREWORKS_API_KEY`)
+
 ### ChatGPT subscription models
 
-- `CHATGPT_AUTH_JSON_B64`
+By default, `chatgpt-*` models reuse the ChatGPT OAuth tokens stored by the Codex CLI:
+
+- `${CODEX_HOME:-~/.codex}/auth.json`
+
+If you deploy to multiple environments (Vercel, GCP, local dev, etc.), use a centralized HTTPS token provider that owns
+refresh-token rotation and serves short-lived access tokens.
 
-This is a base64url-encoded JSON blob containing the ChatGPT OAuth tokens + account id (RFC 4648):
-https://www.rfc-editor.org/rfc/rfc4648
+- `CHATGPT_AUTH_TOKEN_PROVIDER_URL` (example: `https://chatgpt-auth.<your-domain>`)
+- `CHATGPT_AUTH_API_KEY` (shared secret; sent as `Authorization: Bearer ...` and `x-chatgpt-auth: ...`)
+- `CHATGPT_AUTH_TOKEN_PROVIDER_STORE` (`kv` or `d1`, defaults to `kv`)
+
+This repo includes a Cloudflare Workers token provider implementation in `workers/chatgpt-auth/`.
+
+If `CHATGPT_AUTH_TOKEN_PROVIDER_URL` + `CHATGPT_AUTH_API_KEY` are set, `chatgpt-*` models will fetch tokens from the
+token provider and will not read the local Codex auth store.
 
 ## Usage
 
@@ -245,6 +261,21 @@ const result = await generateText({
 console.log(result.text);
 ```
 
+### Fireworks
+
+Use Fireworks model ids directly (for example `kimi-k2.5`, `glm-5`, `minimax-m2.1`):
+
+```ts
+import { generateText } from "@ljoukov/llm";
+
+const result = await generateText({
+  model: "kimi-k2.5",
+  input: "Return exactly: OK",
+});
+
+console.log(result.text);
+```
+
 ### ChatGPT subscription models
 
 Use a `chatgpt-` prefix:
@@ -348,14 +379,21 @@ const { value } = await generateJson({
 
 ## Tools
 
-This library supports two kinds of tools:
+There are three tool-enabled call patterns:
+
+1. `generateText()` for provider-native/server-side tools (for example web search).
+2. `runToolLoop()` for your runtime JS/TS tools (function tools executed in your process).
+3. `runAgentLoop()` for filesystem tasks (a convenience wrapper around `runToolLoop()`).
 
-- Model tools (server-side): `web-search` and `code-execution`
-- Your tools (JS/TS code): use `runToolLoop()` and `tool()`
+Architecture note:
 
-### Model tools (web search / code execution)
+- Filesystem tools are not a separate execution system.
+- `runAgentLoop()` constructs a filesystem toolset, merges your optional custom tools, then calls the same `runToolLoop()` engine.
+- This behavior is model-agnostic at API level; profile selection only adapts tool shape for model compatibility.
 
-These tools run on the provider side.
+### Provider-Native Tools (`generateText()`)
+
+Use this when the model provider executes the tool remotely (for example search/code-exec style tools).
 
 ```ts
 import { generateText } from "@ljoukov/llm";
@@ -369,9 +407,9 @@ const result = await generateText({
 console.log(result.text);
 ```
 
-### Your tools (function calling)
+### Runtime Tools (`runToolLoop()`)
 
-`runToolLoop()` runs a simple function-calling loop until the model returns a final answer or the step limit is hit.
+Use this when the model should call your local runtime functions.
 
 ```ts
 import { runToolLoop, tool } from "@ljoukov/llm";
@@ -392,6 +430,91 @@ const result = await runToolLoop({
 console.log(result.text);
 ```
 
+Use `customTool()` only when you need freeform/non-JSON tool input grammar.
+
+### Filesystem Tasks (`runAgentLoop()`)
+
+Use this for read/search/write tasks in a workspace. The library auto-selects filesystem tool profile by model when `profile: "auto"`:
+
+- Codex-like models: Codex-compatible filesystem tool shape.
+- Gemini models: Gemini-compatible filesystem tool shape.
+- Other models: model-agnostic profile (currently Gemini-style).
+
+Confinement/policy is set through `filesystemTool.options`:
+
+- `cwd`: workspace root for path resolution.
+- `fs`: backend (`createNodeAgentFilesystem()` or `createInMemoryAgentFilesystem()`).
+- `checkAccess`: hook for allow/deny policy + audit.
+- `allowOutsideCwd`: opt-out confinement (default is false).
+
+Detailed reference: `docs/agent-filesystem-tools.md`.
+
+```ts
+import { createInMemoryAgentFilesystem, runAgentLoop } from "@ljoukov/llm";
+
+const fs = createInMemoryAgentFilesystem({
+  "/repo/src/a.ts": "export const value = 1;\n",
+});
+
+const result = await runAgentLoop({
+  model: "chatgpt-gpt-5.3-codex-spark",
+  input: "Change value from 1 to 2 using filesystem tools.",
+  filesystemTool: {
+    profile: "auto",
+    options: {
+      cwd: "/repo",
+      fs,
+    },
+  },
+});
+
+console.log(result.text);
+```
+
+If you need exact control over tool definitions, build the filesystem toolset yourself and call `runToolLoop()` directly.
+
+```ts
+import {
+  createFilesystemToolSetForModel,
+  createInMemoryAgentFilesystem,
+  runToolLoop,
+} from "@ljoukov/llm";
+
+const fs = createInMemoryAgentFilesystem({ "/repo/a.ts": "export const n = 1;\n" });
+const tools = createFilesystemToolSetForModel("chatgpt-gpt-5.3-codex-spark", {
+  cwd: "/repo",
+  fs,
+});
+
+const result = await runToolLoop({
+  model: "chatgpt-gpt-5.3-codex-spark",
+  input: "Update n to 2.",
+  tools,
+});
+```
+
+## Agent benchmark (filesystem extraction)
+
+For filesystem extraction/summarization evaluation across Codex, Fireworks, and Gemini models:
+
+```bash
+npm run bench:agent
+```
+
+Standard full refresh (all tasks, auto-write `LATEST_RESULTS.md`, refresh `traces/latest`, prune old traces):
+
+```bash
+npm run bench:agent:latest
+```
+
+Estimate-only:
+
+```bash
+npm run bench:agent:estimate
+```
+
+See `benchmarks/agent/README.md` for options and output format.
+
 ## License
 
 MIT