@copilotkit/runtime 1.57.3 → 1.58.0-canary.thread-id-propagation

package/skills/runtime/references/transcription.md

@@ -0,0 +1,287 @@
+# CopilotKit Transcription
+
+Subclass `TranscriptionService`, pass an instance to `CopilotRuntime({ transcriptionService })`,
+and the `POST /transcribe` endpoint lights up. The service has a single method,
+`transcribeFile`, that returns the transcript as a plain string.
+
+## Setup
+
+```typescript
+import {
+  CopilotRuntime,
+  createCopilotRuntimeHandler,
+  TranscriptionService,
+  type TranscribeFileOptions,
+} from "@copilotkit/runtime/v2";
+import OpenAI from "openai";
+
+class OpenAIWhisperTranscription extends TranscriptionService {
+  private client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+
+  async transcribeFile({ audioFile }: TranscribeFileOptions): Promise<string> {
+    const result = await this.client.audio.transcriptions.create({
+      file: audioFile,
+      model: "whisper-1",
+    });
+    return result.text;
+  }
+}
+
+const runtime = new CopilotRuntime({
+  agents: {
+    /* ... */
+  } as any,
+  transcriptionService: new OpenAIWhisperTranscription(),
+});
+
+const handler = createCopilotRuntimeHandler({
+  runtime,
+  basePath: "/api/copilotkit",
+});
+
+export default { fetch: handler };
+```
+
+## Core Patterns
+
+### Abstract contract
+
+```typescript
+// packages/runtime/src/v2/runtime/transcription-service/transcription-service.ts
+export interface TranscribeFileOptions {
+  audioFile: File;
+  mimeType?: string;
+  size?: number;
+}
+
+export abstract class TranscriptionService {
+  abstract transcribeFile(options: TranscribeFileOptions): Promise<string>;
+}
+```
+
+### Supported request shapes
+
+Multipart (REST mode):
+
+```typescript
+const form = new FormData();
+form.append("audio", blob, "recording.webm");
+await fetch("/api/copilotkit/transcribe", { method: "POST", body: form });
+```
+
+JSON (works in both multi-route and single-endpoint modes — dispatch is by
+`Content-Type: application/json`; `mimeType` is required in the payload):
+
+```typescript
+await fetch("/api/copilotkit/transcribe", {
+  method: "POST",
+  headers: { "Content-Type": "application/json" },
+  body: JSON.stringify({
+    audio: base64String,
+    mimeType: "audio/webm",
+    filename: "recording.webm", // optional
+  }),
+});
+```
+
+### Reject oversize audio with a graceful 400
+
+```typescript
+class OpenAIWhisperTranscription extends TranscriptionService {
+  private client = new OpenAI({ apiKey: process.env.OPENAI_API_KEY });
+
+  async transcribeFile({
+    audioFile,
+    size,
+  }: TranscribeFileOptions): Promise<string> {
+    const max = 25 * 1024 * 1024; // 25 MB
+    if ((size ?? audioFile.size) > max) {
+      // "too long" keyword → audio_too_long response
+      throw new Error("Audio duration too long — max 25MB per upload");
+    }
+    const result = await this.client.audio.transcriptions.create({
+      file: audioFile,
+      model: "whisper-1",
+    });
+    return result.text;
+  }
+}
+```
+
+### Error auto-categorization
+
+The runtime inspects `String(error).toLowerCase()` thrown by your service and maps keywords
+to error codes. Let the provider error bubble up — do not re-categorize inside the service.
+
+| Keyword substrings                       | Maps to                          |
+| ---------------------------------------- | -------------------------------- |
+| `rate`, `429`, `too many`                | `rate_limited` (retryable)       |
+| `auth`, `401`, `api key`, `unauthorized` | `auth_failed` (not retryable)    |
+| `too long`, `duration`, `length`         | `audio_too_long` (not retryable) |
+| (anything else)                          | `provider_error` (retryable)     |
+
+Full error-code enum:
+
+```typescript
+// packages/shared/src/transcription-errors.ts
+export enum TranscriptionErrorCode {
+  SERVICE_NOT_CONFIGURED = "service_not_configured",
+  INVALID_AUDIO_FORMAT = "invalid_audio_format",
+  AUDIO_TOO_LONG = "audio_too_long",
+  AUDIO_TOO_SHORT = "audio_too_short",
+  RATE_LIMITED = "rate_limited",
+  AUTH_FAILED = "auth_failed",
+  PROVIDER_ERROR = "provider_error",
+  NETWORK_ERROR = "network_error",
+  INVALID_REQUEST = "invalid_request",
+}
+```
+
+## Common Mistakes
+
+### HIGH Calling /transcribe without configuring transcriptionService
+
+Wrong:
+
+```typescript
+new CopilotRuntime({ agents });
+// client calls /api/copilotkit/transcribe → 503
+```
+
+Correct:
+
+```typescript
+new CopilotRuntime({
+  agents,
+  transcriptionService: new MyWhisperService(),
+});
+```
+
+Unconfigured runtime returns HTTP 503 with
+`{ error: "service_not_configured" }`. The frontend gets no transcript with no obvious
+server-side failure.
+
+Source: `packages/runtime/src/v2/runtime/handlers/handle-transcribe.ts:203-207`.
+
+### MEDIUM Form field named "file" instead of "audio"
+
+Wrong:
+
+```typescript
+const form = new FormData();
+form.append("file", blob, "recording.webm");
+await fetch("/api/copilotkit/transcribe", { method: "POST", body: form });
+```
+
+Correct:
+
+```typescript
+const form = new FormData();
+form.append("audio", blob, "recording.webm");
+await fetch("/api/copilotkit/transcribe", { method: "POST", body: form });
+```
+
+The handler reads `formData.get("audio")` — any other field name yields `null` and returns
+`invalid_request`.
+
+Source: `packages/runtime/src/v2/runtime/handlers/handle-transcribe.ts:91-97`.
+
+### MEDIUM Base64 payload missing mimeType
+
+Wrong:
+
+```typescript
+await fetch("/api/copilotkit/transcribe", {
+  method: "POST",
+  headers: { "Content-Type": "application/json" },
+  body: JSON.stringify({ audio: b64 }),
+});
+```
+
+Correct:
+
+```typescript
+await fetch("/api/copilotkit/transcribe", {
+  method: "POST",
+  headers: { "Content-Type": "application/json" },
+  body: JSON.stringify({ audio: b64, mimeType: "audio/webm" }),
+});
+```
+
+JSON mode requires `mimeType` — the handler explicitly rejects payloads missing it with
+`invalid_request`.
+
+Source: `packages/runtime/src/v2/runtime/handlers/handle-transcribe.ts:131-136`.
+
+### LOW Re-categorizing errors inside the service
+
+Wrong:
+
+```typescript
+class MyService extends TranscriptionService {
+  async transcribeFile(opts: TranscribeFileOptions): Promise<string> {
+    try {
+      return await doTranscribe(opts);
+    } catch (e) {
+      // trying to hand-pick error codes
+      throw new Error("RATE_LIMITED");
+    }
+  }
+}
+```
+
+Correct:
+
+```typescript
+class MyService extends TranscriptionService {
+  async transcribeFile(opts: TranscribeFileOptions): Promise<string> {
+    return doTranscribe(opts); // let provider errors bubble up verbatim
+  }
+}
+```
+
+The runtime scans `String(error).toLowerCase()` for `"rate"`, `"429"`, `"auth"`, `"too long"`
+etc. Provider-native messages (`"OpenAI returned 429 rate limited"`) auto-map to the right
+code. Hand-crafted codes bypass the keyword matcher and end up as `provider_error`.
+
+Source: `packages/runtime/src/v2/runtime/handlers/handle-transcribe.ts:160-196`.
+
+### MEDIUM Returning a rich object instead of a string
+
+Wrong:
+
+```typescript
+class MyService extends TranscriptionService {
+  async transcribeFile(opts: TranscribeFileOptions): Promise<string> {
+    // @ts-expect-error returning the wrong shape
+    return {
+      text: "hi",
+      segments: [
+        /* ... */
+      ],
+    };
+  }
+}
+```
+
+Correct:
+
+```typescript
+class MyService extends TranscriptionService {
+  async transcribeFile(opts: TranscribeFileOptions): Promise<string> {
+    const result = await provider.transcribe(opts.audioFile);
+    return result.text;
+  }
+}
+```
+
+`transcribeFile` returns `Promise<string>`. The handler sends
+`{ transcription: string }` back to the client — any other shape is a TypeScript error and
+would be JSON-stringified wrongly at runtime.
+
+Source: `packages/runtime/src/v2/runtime/transcription-service/transcription-service.ts:9-11`.
+
+## See also
+
+- `copilotkit/setup-endpoint` — `/transcribe` is one of the routes the handler mounts
+- `copilotkit/debug-and-troubleshoot` — `TranscriptionErrorCode` catalog

package/skills/runtime/references/wiring-a2a.md

@@ -0,0 +1,40 @@
+A2A (Agent2Agent) — wired via `@ag-ui/a2a`. Requires a pre-built `A2AClient` (not a URL).
+
+## Install
+
+```bash
+pnpm add @ag-ui/a2a @a2a-js/sdk
+```
+
+## Minimal wire-up
+
+```typescript
+import {
+  CopilotRuntime,
+  createCopilotRuntimeHandler,
+} from "@copilotkit/runtime/v2";
+import { A2AAgent } from "@ag-ui/a2a";
+import { A2AClient } from "@a2a-js/sdk/client";
+
+const a2aClient = new A2AClient(process.env.A2A_URL!);
+
+const runtime = new CopilotRuntime({
+  agents: {
+    default: new A2AAgent({ a2aClient }),
+  },
+});
+
+const handler = createCopilotRuntimeHandler({
+  runtime,
+  basePath: "/api/copilotkit",
+});
+
+export default { fetch: handler };
+```
+
+## Gotcha — do NOT pass `{ url }`
+
+`A2AAgent` takes `{ a2aClient }`. The A2A protocol has its own handshake; the client
+object handles it. Passing `{ url: "..." }` is a type error and will fail at runtime.
+
+Source: `examples/integrations/a2a-a2ui/app/api/copilotkit/[[...slug]]/route.tsx:12`.

package/skills/runtime/references/wiring-adk.md

@@ -0,0 +1,45 @@
+Google ADK (Agent Development Kit) — wired via the bare `HttpAgent` from `@ag-ui/client`.
+
+## Install
+
+```bash
+pnpm add @ag-ui/client
+```
+
+## Minimal wire-up
+
+```typescript
+import {
+  CopilotRuntime,
+  createCopilotRuntimeHandler,
+} from "@copilotkit/runtime/v2";
+import { HttpAgent } from "@ag-ui/client";
+
+const runtime = new CopilotRuntime({
+  agents: {
+    default: new HttpAgent({
+      url: process.env.ADK_URL ?? "http://localhost:8000/",
+    }),
+  },
+});
+
+const handler = createCopilotRuntimeHandler({
+  runtime,
+  basePath: "/api/copilotkit",
+});
+
+export default { fetch: handler };
+```
+
+## Server side
+
+Your ADK Python agent must speak AG-UI. ADK ships an AG-UI FastAPI adapter — use it
+and point `HttpAgent({ url })` at the FastAPI route.
+
+## Gotcha — env-sourced credentials
+
+ADK typically authenticates to Google Cloud via service-account credentials
+(`GOOGLE_APPLICATION_CREDENTIALS`). Those live on the ADK Python server, not in the
+CopilotKit runtime. The runtime just forwards AG-UI events.
+
+Source: `docs/content/docs/integrations/adk/quickstart.mdx`.

package/skills/runtime/references/wiring-ag2.md

@@ -0,0 +1,41 @@
+AG2 — wired via the bare `HttpAgent` from `@ag-ui/client`. No dedicated `@ag-ui/ag2`
+package exists; AG2 is a standard HTTP AG-UI framework.
+
+## Install
+
+```bash
+pnpm add @ag-ui/client
+```
+
+## Minimal wire-up
+
+```typescript
+import {
+  CopilotRuntime,
+  createCopilotRuntimeHandler,
+} from "@copilotkit/runtime/v2";
+import { HttpAgent } from "@ag-ui/client";
+
+const runtime = new CopilotRuntime({
+  agents: {
+    default: new HttpAgent({
+      url: process.env.AG2_URL ?? "http://localhost:8000/",
+    }),
+  },
+});
+
+const handler = createCopilotRuntimeHandler({
+  runtime,
+  basePath: "/api/copilotkit",
+});
+
+export default { fetch: handler };
+```
+
+## Gotcha — no dedicated package
+
+Unlike Mastra / LangGraph / CrewAI / LlamaIndex / Agno, AG2 has no `@ag-ui/ag2` package.
+Always use the generic `HttpAgent`. If an older doc references a dedicated AG2 package,
+treat it as stale.
+
+Source: `docs/content/docs/integrations/ag2/quickstart.mdx`; maintainer Phase 4 resolution.

package/skills/runtime/references/wiring-agno.md

@@ -0,0 +1,39 @@
+Agno — wired via `@ag-ui/agno`. Requires an `/agui` URL suffix.
+
+## Install
+
+```bash
+pnpm add @ag-ui/agno
+```
+
+## Minimal wire-up
+
+```typescript
+import {
+  CopilotRuntime,
+  createCopilotRuntimeHandler,
+} from "@copilotkit/runtime/v2";
+import { AgnoAgent } from "@ag-ui/agno";
+
+const runtime = new CopilotRuntime({
+  agents: {
+    default: new AgnoAgent({
+      url: process.env.AGNO_URL ?? "http://localhost:8000/agui",
+    }),
+  },
+});
+
+const handler = createCopilotRuntimeHandler({
+  runtime,
+  basePath: "/api/copilotkit",
+});
+
+export default { fetch: handler };
+```
+
+## Gotcha — the `/agui` suffix is mandatory
+
+Agno's AG-UI FastAPI app mounts at `/agui`. Pointing `url` at the server root
+(`http://localhost:8000`) returns 404. Always include `/agui`.
+
+Source: `docs/content/docs/integrations/agno/quickstart.mdx:215`.

package/skills/runtime/references/wiring-aws-strands.md

@@ -0,0 +1,59 @@
+AWS Strands — wired via the bare `HttpAgent` from `@ag-ui/client`.
+
+## Install
+
+```bash
+pnpm add @ag-ui/client
+```
+
+## Minimal wire-up
+
+```typescript
+import {
+  CopilotRuntime,
+  createCopilotRuntimeHandler,
+} from "@copilotkit/runtime/v2";
+import { HttpAgent } from "@ag-ui/client";
+
+const runtime = new CopilotRuntime({
+  agents: {
+    default: new HttpAgent({
+      url: process.env.STRANDS_URL ?? "http://localhost:8000",
+    }),
+  },
+});
+
+const handler = createCopilotRuntimeHandler({
+  runtime,
+  basePath: "/api/copilotkit",
+});
+
+export default { fetch: handler };
+```
+
+## Server side
+
+Strands agents run on AWS and typically expose an AG-UI-speaking endpoint (API Gateway or
+Lambda Function URL). Point `HttpAgent({ url })` at that endpoint.
+
+## Gotcha — AWS auth
+
+Strands deployments often require IAM SigV4 or a custom header. `HttpAgent` accepts a
+`headers: Record<string, string>` option that is attached to every outbound runtime →
+Strands call:
+
+```typescript
+new HttpAgent({
+  url: process.env.STRANDS_URL!,
+  headers: { Authorization: `Bearer ${process.env.STRANDS_TOKEN!}` },
+});
+```
+
+`hooks.onBeforeHandler` will NOT work for this — those hooks run on the inbound frontend
+→ runtime request, not on the outbound runtime → Strands call that `HttpAgent` issues.
+For SigV4 (which needs a per-request signature over the body), front Strands with a
+lightweight Lambda / API Gateway authorizer that strips client credentials and adds the
+IAM signing, then point `HttpAgent({ url })` at that shim.
+
+Source: `node_modules/@ag-ui/client/dist/index.d.ts` (`HttpAgentConfig.headers`);
+`docs/content/docs/integrations/aws-strands/quickstart.mdx`.

package/skills/runtime/references/wiring-crewai-crews.md

@@ -0,0 +1,51 @@
+CrewAI Crews — multi-agent crews wired via `@ag-ui/crewai`.
+
+## Install
+
+```bash
+pnpm add @ag-ui/crewai
+```
+
+## Minimal wire-up
+
+```typescript
+import {
+  CopilotRuntime,
+  createCopilotRuntimeHandler,
+} from "@copilotkit/runtime/v2";
+import { CrewAIAgent } from "@ag-ui/crewai";
+
+const runtime = new CopilotRuntime({
+  agents: {
+    default: new CrewAIAgent({
+      url: process.env.CREWAI_URL ?? "http://localhost:8000/",
+    }),
+  },
+});
+
+const handler = createCopilotRuntimeHandler({
+  runtime,
+  basePath: "/api/copilotkit",
+});
+
+export default { fetch: handler };
+```
+
+## Crews vs Flows
+
+CrewAI ships two products:
+
+- **Crews** — multi-agent orchestration. Use `CrewAIAgent` from `@ag-ui/crewai`.
+- **Flows** — event-driven pipelines. Use the generic `HttpAgent` from `@ag-ui/client`
+  (there's no framework-specific wrapper). See [crewai-flows.md](crewai-flows.md).
+
+## Gotcha — trailing slash
+
+The `url` for `CrewAIAgent` traditionally ends with a trailing slash
+(`http://localhost:8000/`). Follow whatever your CrewAI server exposes — don't strip it.
+
+Source: `@ag-ui/crewai` package types (`CrewAIAgent` constructor);
+`docs/content/docs/reference/v1/sdk/python/CrewAIAgent.mdx` for v1 Python-side
+reference. The v2 integrations docs currently ship only a Flows quickstart at
+`docs/content/docs/integrations/crewai-flows/quickstart.mdx` — there is no dedicated
+Crews quickstart yet.

package/skills/runtime/references/wiring-crewai-flows.md

@@ -0,0 +1,45 @@
+CrewAI Flows — wired via the bare `HttpAgent` from `@ag-ui/client`. No dedicated wrapper.
+
+## Install
+
+```bash
+pnpm add @ag-ui/client
+```
+
+## Minimal wire-up
+
+```typescript
+import {
+  CopilotRuntime,
+  createCopilotRuntimeHandler,
+} from "@copilotkit/runtime/v2";
+import { HttpAgent } from "@ag-ui/client";
+
+const runtime = new CopilotRuntime({
+  agents: {
+    default: new HttpAgent({
+      url: process.env.CREWAI_FLOWS_URL ?? "http://localhost:8000/",
+    }),
+  },
+});
+
+const handler = createCopilotRuntimeHandler({
+  runtime,
+  basePath: "/api/copilotkit",
+});
+
+export default { fetch: handler };
+```
+
+## Flows vs Crews
+
+Flows is the event-driven pipeline product. For multi-agent Crews orchestration, use
+`CrewAIAgent` from `@ag-ui/crewai` instead — see [crewai-crews.md](crewai-crews.md).
+
+## Gotcha — AG-UI compatibility
+
+Your CrewAI Flows server must speak AG-UI over HTTP. If you control the server, use the
+official CrewAI Python AG-UI adapter. `HttpAgent` is a thin bridge — any server that
+emits AG-UI events at the URL works.
+
+Source: `docs/content/docs/integrations/crewai-flows/quickstart.mdx`.