@warlock.js/ai-live 4.6.0

package/llms.txt

@@ -0,0 +1,9 @@
+# Warlock AI Live
+
+> Package: `@warlock.js/ai-live`
+
+> Live & generative rich-media for @warlock.js/ai — ai.realtime() duplex voice sessions and ai.video() text-to-video. The heavy, demand-gated modalities, kept out of core so text/image/speech stay dependency-light.
+
+## Skills
+
+- [use-ai-live](@warlock.js/ai-live/use-ai-live/SKILL.md): Live & generative rich-media add-on for @warlock.js/ai. A side-effect import `import "@warlock.js/ai-live"` mounts two heavy modalities onto the shared `ai.*` facade: `ai.video({ model, prompt })` — text-to-video (Sora / Veo / Kling-class), the async submit→poll job hidden behind the same uniform never-throws `{ data, error, usage, report }` envelope as `ai.image`, with per-second cost-truth folded into `Usage.cost`; and `ai.realtime({ transport, model })` — a stateful duplex VOICE SESSION over a pluggable `RealtimeTransport` (sendAudio / sendText / events() / close()→RealtimeReport). Videos are a discriminated `GeneratedVideo = { type: "url" } | { type: "base64" }`; session output is a `RealtimeEvent` union (audio | transcript | tool-call | error | done). Ships `MockVideoModel` + `MockRealtimeTransport` for HTTP-free tests. Triggers: `ai.video`, `ai.realtime`, `VideoModelContract`, `GeneratedVideo`, `VideoModelPricing`, `RealtimeSession`, `RealtimeTransport`, `RealtimeEvent`, `RealtimeReport`, `MockVideoModel`, `MockRealtimeTransport`; 'generate a video', 'text to video', 'sora', 'veo', 'realtime voice', 'duplex voice session', 'live voice agent', 'barge-in', 'per-second video cost'; typical import `import "@warlock.js/ai-live"` + `import { ai } from "@warlock.js/ai"`. Skip: still-image OUTPUT — [[generate-images]]; text-to-speech / transcription (one-shot audio, not a session) — [[generate-speech]]; chat agents / tools / workflows — `@warlock.js/ai/run-ai-agent/SKILL.md`.

package/package.json

@@ -0,0 +1,34 @@
+{
+  "name": "@warlock.js/ai-live",
+  "description": "Live & generative rich-media for @warlock.js/ai — ai.realtime() duplex voice sessions and ai.video() text-to-video. The heavy, demand-gated modalities, kept out of core so text/image/speech stay dependency-light.",
+  "keywords": [
+    "ai",
+    "realtime",
+    "voice",
+    "video",
+    "warlock"
+  ],
+  "license": "MIT",
+  "dependencies": {
+    "@warlock.js/logger": "4.6.0"
+  },
+  "peerDependencies": {
+    "@warlock.js/ai": "4.6.0"
+  },
+  "version": "4.6.0",
+  "main": "./cjs/index.cjs",
+  "module": "./esm/index.mjs",
+  "types": "./esm/index.d.mts",
+  "exports": {
+    ".": {
+      "import": {
+        "types": "./esm/index.d.mts",
+        "default": "./esm/index.mjs"
+      },
+      "require": {
+        "types": "./esm/index.d.mts",
+        "default": "./cjs/index.cjs"
+      }
+    }
+  }
+}

package/skills/use-ai-live/SKILL.md

@@ -0,0 +1,200 @@
+---
+name: use-ai-live
+description: 'Live & generative rich-media add-on for @warlock.js/ai. A side-effect import `import "@warlock.js/ai-live"` mounts two heavy modalities onto the shared `ai.*` facade: `ai.video({ model, prompt })` — text-to-video (Sora / Veo / Kling-class), the async submit→poll job hidden behind the same uniform never-throws `{ data, error, usage, report }` envelope as `ai.image`, with per-second cost-truth folded into `Usage.cost`; and `ai.realtime({ transport, model })` — a stateful duplex VOICE SESSION over a pluggable `RealtimeTransport` (sendAudio / sendText / events() / close()→RealtimeReport). Videos are a discriminated `GeneratedVideo = { type: "url" } | { type: "base64" }`; session output is a `RealtimeEvent` union (audio | transcript | tool-call | error | done). Ships `MockVideoModel` + `MockRealtimeTransport` for HTTP-free tests. Triggers: `ai.video`, `ai.realtime`, `VideoModelContract`, `GeneratedVideo`, `VideoModelPricing`, `RealtimeSession`, `RealtimeTransport`, `RealtimeEvent`, `RealtimeReport`, `MockVideoModel`, `MockRealtimeTransport`; ''generate a video'', ''text to video'', ''sora'', ''veo'', ''realtime voice'', ''duplex voice session'', ''live voice agent'', ''barge-in'', ''per-second video cost''; typical import `import "@warlock.js/ai-live"` + `import { ai } from "@warlock.js/ai"`. Skip: still-image OUTPUT — [[generate-images]]; text-to-speech / transcription (one-shot audio, not a session) — [[generate-speech]]; chat agents / tools / workflows — `@warlock.js/ai/run-ai-agent/SKILL.md`.'
+---
+
+# Live rich-media — `ai.video` + `ai.realtime` (`@warlock.js/ai-live`)
+
+`@warlock.js/ai` ships the synchronous modality verbs (`ai.agent`, `ai.image`, `ai.speech`, `ai.transcribe`). `ai-live` adds the two that need heavyweight, demand-gated machinery — a **long-running job** (video) and a **persistent network session** (realtime voice) — kept in their own package so the core stays dependency-light.
+
+Both mount onto the same `ai.*` facade by a **side-effect import**:
+
+```ts
+import "@warlock.js/ai-live"; // lights up ai.video + ai.realtime on the shared Ai facade
+import { ai } from "@warlock.js/ai";
+```
+
+The mount mirrors how `@warlock.js/ai-tools` mounts `ai.mcp` and `@warlock.js/ai-workspace` mounts `ai.workspace`: a `declare module "@warlock.js/ai"` interface-merge plus `ai.video = video; ai.realtime = realtime` at load. Forget the side-effect import and `ai.video` is a compile-time `undefined` — not a silent runtime miss.
+
+## `ai.video` — text-to-video, uniform envelope
+
+Prompt in / one video out. The adapter hides the provider's async **submit→poll** lifecycle and resolves only when the clip is ready, so the verb returns the framework's uniform **never-throws** `{ data, error, usage, report }` — the exact shape `ai.image` returns, so video spend and traces fold into the same dashboards.
+
+```ts
+import { ai } from "@warlock.js/ai";
+
+const { data, error, usage, report } = await ai.video({
+  model: sora.video({ name: "sora-2", pricing: { perSecond: 0.1 } }), // VideoModelContract
+  prompt: "a timelapse of a city skyline at dusk, cinematic",
+  durationSeconds: 8,
+  aspectRatio: "16:9",
+});
+
+if (error) {
+  console.warn(error.code);   // typed AIError — NEVER thrown
+} else {
+  const clip = data.video;    // GeneratedVideo (discriminated)
+  if (clip.type === "url") download(clip.url);
+  else save(Buffer.from(clip.base64, "base64"), clip.mediaType);
+}
+```
+
+### Shape
+
+```ts
+type VideoResult = ExecuteResult<VideoData> & { type: "video"; report: VideoReport };
+type VideoData = { video: GeneratedVideo };
+
+type GeneratedVideo =
+  | { type: "url"; url: string; mediaType?: string }
+  | { type: "base64"; base64: string; mediaType: string };
+
+// VideoParams (provider-neutral) — the model is the only required field beyond prompt:
+await ai.video({
+  model,                 // VideoModelContract from an adapter's video({ name })
+  prompt: "...",
+  durationSeconds: 8,    // requested clip length
+  aspectRatio: "9:16",
+  resolution: "1080p",   // hint
+  negativePrompt: "blurry, watermark",
+  signal,                // AbortSignal → status "cancelled"
+  observe: collector,    // route the report to an Observer (panoptic), like agents
+  sessionId: "campaign-42",
+  name: "hero-clip",     // report node name (defaults to "video")
+  options: { seed: 7 },  // provider-specific escape hatch, forwarded verbatim
+});
+```
+
+`VideoModelContract` is the moving-image sibling of `ImageModelContract` — `{ name, provider, pricing?, generate(prompt, options) }` — produced by an adapter's `video()` factory. An adapter with no video API simply doesn't define `video()`.
+
+## Video cost-truth — per-second first
+
+`VideoModelPricing` carries `{ perSecond?, input?, output? }`. Per-second wins when set (video is metered by clip length): `usage.cost = { input: 0, output: durationSeconds × perSecond }`, attributed to `cost.output`. A token-metered model falls back to the standard `computeCost` against reported tokens. No usable pricing → `usage.cost` stays `undefined` (honest "cost unknown", never a false zero). A pre-priced adapter response is honored, not overwritten. The final `durationSeconds` from the provider (not just the requested one) drives the math and lands on `report.durationSeconds`.
+
+## `ai.realtime` — a duplex voice SESSION (not a one-shot)
+
+Unlike every other `ai.*` verb, `ai.realtime()` returns a **stateful, long-lived session** — its closest sibling is `ai.orchestrator`. You open it over a pluggable **`RealtimeTransport`** (the low-level connection to the provider's realtime endpoint), push microphone audio / text turns in, consume an async **event stream** out, and `close()` to end it and receive a `RealtimeReport` for the cost/observability surfaces.
+
+```ts
+import { ai } from "@warlock.js/ai";
+
+const session = await ai.realtime({
+  transport: openAiRealtime({ apiKey }), // RealtimeTransport (own adapter)
+  model: "gpt-realtime",
+  voice: "alloy",
+  instructions: "You are a friendly phone receptionist.",
+  sessionId: "call-8891",
+});
+
+session.sendAudio(micChunkBase64, "audio/pcm"); // push mic audio upstream
+session.sendText("Please hold for one moment."); // or a text turn
+
+for await (const event of session.events()) {    // duplex output
+  switch (event.type) {
+    case "audio":      speaker.write(event.base64); break;
+    case "transcript": if (event.final) log(event.role, event.text); break;
+    case "tool-call":  await handleTool(event.name, event.input); break;
+    case "error":      console.warn(event.error.code); break;
+    case "done":       break;
+  }
+}
+
+const report = await session.close(); // RealtimeReport — idempotent
+```
+
+### Session contract & event union
+
+```ts
+interface RealtimeSession {
+  sendAudio(base64: string, mediaType: string): void;
+  sendText(text: string): void;
+  events(): AsyncIterable<RealtimeEvent>;
+  close(): Promise<RealtimeReport>; // idempotent — second close() returns the first report
+}
+
+type RealtimeEvent =
+  | { type: "audio"; base64: string; mediaType: string }
+  | { type: "transcript"; role: "user" | "assistant"; text: string; final: boolean }
+  | { type: "tool-call"; id: string; name: string; input: unknown }
+  | { type: "error"; error: AIError }   // typed, non-fatal
+  | { type: "done" };                   // server ended the session
+
+type RealtimeReport = {
+  runId: string; rootRunId: string; type: "realtime";
+  name: string; status: "completed" | "failed" | "cancelled";
+  startedAt: string; endedAt: string; duration: number; sessionId?: string;
+};
+```
+
+The transport seam is what lets `ai-live` ship the session surface without hard-wiring `ws` (it's an optional peer). A concrete transport implements just two interfaces: `RealtimeTransport.connect(config) → RealtimeConnection` (with `sendAudio` / `sendText` / `events()` / `close()`).
+
+## Pattern — a live phone receptionist over a channel
+
+```ts
+import "@warlock.js/ai-live";
+import { ai } from "@warlock.js/ai";
+
+async function handleCall(caller: PhoneChannel, transport: RealtimeTransport) {
+  const session = await ai.realtime({
+    transport,
+    model: "gpt-realtime",
+    voice: "alloy",
+    instructions: "Greet the caller and route them to the right department.",
+    sessionId: caller.id,
+  });
+
+  // Pump caller audio → session (fire-and-forget).
+  caller.onAudio((chunk) => session.sendAudio(chunk, "audio/pcm"));
+
+  // Pump session output → caller, until the model signals done.
+  for await (const event of session.events()) {
+    if (event.type === "audio") caller.playAudio(event.base64);
+    if (event.type === "done") break;
+  }
+
+  const report = await session.close();
+  metrics.record({ runId: report.runId, seconds: report.duration / 1000 });
+}
+```
+
+## Testing — mocks, no HTTP, no sockets
+
+`@warlock.js/ai-live` exports deterministic doubles so both verbs test offline.
+
+```ts
+import { ai } from "@warlock.js/ai";
+import { MockVideoModel, MockRealtimeTransport } from "@warlock.js/ai-live";
+
+// Video — script the returned clip / usage / duration; assert cost math + recorded calls.
+const model = new MockVideoModel("mock-video", [{ durationSeconds: 8 }], { perSecond: 0.1 });
+const { data, usage } = await ai.video({ model, prompt: "x", durationSeconds: 8 });
+// data.video → { type: "url", url: "https://mock/video.mp4", ... }
+// usage.cost.output === 0.8   (8 × 0.1)
+// model.calls[0] → { prompt: "x", options: { durationSeconds: 8, ... } }
+
+// Realtime — script the outbound event stream; assert what the session sent.
+const transport = new MockRealtimeTransport([
+  { type: "transcript", role: "assistant", text: "Hi!", final: true },
+  { type: "done" },
+]);
+const session = await ai.realtime({ transport, model: "mock-realtime" });
+session.sendText("hello");
+for await (const e of session.events()) { /* ... */ }
+const report = await session.close();
+// transport.lastConnection.sentText → ["hello"]
+// transport.connectConfigs[0] → { model: "mock-realtime", ... }
+// report.status === "completed"
+```
+
+Feed `MockVideoModel` an `{ error }` response and `ai.video` lands it on `result.error` (never throws), setting `report.status` to `"failed"` (or `"cancelled"` when the signal aborted).
+
+## Status — contracts now, transports next
+
+`4.6.0` introduces the package: the `VideoModelContract` / `RealtimeSession` contracts, the `ai.video()` verb (uniform envelope + per-second cost-truth, tested against `MockVideoModel`), and the `ai.realtime()` session primitive over a pluggable transport (tested against `MockRealtimeTransport`). The first **concrete provider transports** — an OpenAI Realtime **WebSocket** for `ai.realtime`, and **Sora / Veo** video adapters for `ai.video` — are the next implementation step; the seams are defined so they drop in without changing the verb surface.
+
+## See also
+
+- [[generate-images]] — still-image OUTPUT (`ai.image`), the synchronous sibling `ai.video` mirrors.
+- [[generate-speech]] — one-shot text-to-speech / transcription (`ai.speech` / `ai.transcribe`); use `ai.realtime` instead when you need a live duplex conversation, not a single audio render.
+- `@warlock.js/ai/observe-ai-flows/SKILL.md` — the `observe` seam both verbs route their reports through.
+- `@warlock.js/ai/run-orchestrator/SKILL.md` — the other session-shaped primitive `ai.realtime` resembles.