@moureau/pipeflow 0.0.1 → 0.0.3

package/README.md

@@ -10,9 +10,9 @@ It handles the plumbing between audio, speech-to-text, LLMs, text-to-speech, con
 
 ## Highlights
 
-* **Small** — under 100 kB packed, zero runtime dependencies.
+* **Small** — ~140 kB packed, one runtime dependency ([zod](https://github.com/colinhacks/zod)).
 * **Realtime by default** — audio, transcripts, and speech stream continuously, with built-in interruption and barge-in handling.
-* **Provider-agnostic** — STT, LLM, and TTS are swappable adapters (Deepgram, DeepSeek, and Kokoro today).
+* **Provider-agnostic** — STT, LLM, and TTS are swappable adapters (Deepgram, DeepSeek, OpenRouter, and Kokoro today).
 * **Your backend stays yours** — tools and the audio transport are owned by your application; Pipeflow never executes your code.
 
 ## Status
@@ -21,6 +21,19 @@ It handles the plumbing between audio, speech-to-text, LLMs, text-to-speech, con
 
 The API is evolving and should be considered experimental.
 
+## Current vs. designed for
+
+Some capabilities ship today; others are architectural boundaries the API is
+shaped around but not yet implemented.
+
+| Area | Current | Designed for |
+| --- | --- | --- |
+| Providers | DeepSeek, OpenRouter (LLM), Deepgram Flux (STT), Kokoro (TTS) | additional providers behind the same interfaces |
+| Persistence | in-memory, SQLite | Postgres, Redis, … behind the same contract |
+| Transport | in-memory (tests, development) | WebSocket, WebRTC, … |
+| Conversation addressing | agent names and aliases | floor management, multi-participant turn-taking |
+| Realtime bounds | step-bounded coordination graph | hierarchical latency budgets |
+
 ## Philosophy
 
 Pipeflow has four core concepts:
@@ -62,25 +75,26 @@ An agent does not inherently own a conversation. A conversation does not require
            │      │      │
            ▼      ▼      ▼
        Deepgram DeepSeek Kokoro
+                OpenRouter
 ```
 
 ## Installation
 
 ```bash
-bun add pipeflow
+bun add @moureau/pipeflow
 ```
 
-The package is not published to npm yet. Until then, install from the
-repository (`bun add pipeflow`) or build from
-source — see [Development](#development).
+Published on npm as `@moureau/pipeflow`. Alternatively, install directly from
+the repository (`bun add git@github.com:moureau-dev/pipeflow.git`) or build
+from source — see [Development](#development).
 
 ## Basic voice agent
 
 Create an agent:
 
 ```ts
-import { Pipeflow, Orchestrator } from "pipeflow";
-import { DeepSeekLLM, DeepgramSTT, KokoroTTS } from "pipeflow/providers";
+import { Pipeflow } from "@moureau/pipeflow";
+import { DeepSeekLLM, DeepgramSTT, KokoroTTS } from "@moureau/pipeflow/providers";
 
 // Providers are configured explicitly with their own credentials —
 // Pipeflow itself does not hold an API key.
@@ -111,28 +125,15 @@ const conversation = await pipeflow.conversations.create({
 });
 ```
 
-Attach the orchestrator — it runs the realtime pipeline (STT → turns → LLM → TTS) for the conversation:
+Starting a conversation starts its realtime machinery: `start()` attaches the
+orchestrator, which runs the STT → turns → LLM → TTS pipeline for the
+conversation.
 
 ```ts
-const orchestrator = new Orchestrator({
-  conversation,
-  agent: jarvis,
-  stt,
-  tts,
-});
-
-await orchestrator.start();
+await conversation.start();
 ```
 
-Start the conversation and add a participant:
-
-```ts
-conversation.start();
-
-await conversation.participate({
-  userId: "alice",
-});
-```
+Add a participant:
 
 Feed it audio as it arrives:
 
@@ -182,6 +183,9 @@ Your application
 
 A conversation is the persistent entity representing a realtime interaction.
 
+See [src/conversations/README.md](src/conversations/README.md) for the
+conversation domain.
+
 ```ts
 const conversation =
   await pipeflow.conversations.create({
@@ -190,43 +194,37 @@ const conversation =
 
 console.log(conversation.id);
 
-conversation.start();
+await conversation.start();
 ```
 
-Creation and realtime execution are deliberately separate.
+Creation and realtime execution are deliberately separate:
 
 ```ts
 create()       // creates the persistent conversation
 start()        // moves the conversation into the started state
 participate()  // adds participants
 listen()       // sends audio
+send()         // injects a finalized text turn (no STT)
 stop()         // finalizes the realtime session
 ```
 
-Realtime processing itself is the orchestrator's job: once attached, it
-subscribes to `audio-in` events, runs the STT/LLM/TTS pipeline, and pushes
-generated audio, turns, transcripts, and tool calls back through conversation
-events.
+`start()` attaches the orchestrator, which subscribes to `audio-in` events,
+runs the STT/LLM/TTS pipeline, and pushes generated audio, turns, transcripts,
+and tool calls back through conversation events.
 
-`listen()` is intentionally synchronous:
+`listen()` is intentionally synchronous — it means "send this audio packet",
+not "wait for this utterance to finish":
 
 ```ts
-conversation.listen({
-  userId,
-  audio,
-});
+conversation.listen({ userId, audio });
 ```
 
-It means:
-
-> Send this audio packet.
-
-It does not mean:
-
-> Wait for this utterance to finish.
-
 This makes it suitable for high-frequency realtime audio streams.
 
+`send({ userId, text })` does the same for a finalized text turn, bypassing
+STT — text-first integrations (chat, Discord text channels) route through the
+same pipeline: routing, coordination, clarification, and generation.
+
 ## Participants
 
 Participants can be added individually:
@@ -269,6 +267,10 @@ Interruptions can be triggered by the application — or automatically: the
 orchestrator detects when a participant starts speaking while the agent is
 responding (barge-in).
 
+`interrupt()` is a semantic guarantee: it stops the active generation and
+prevents any further generated audio from reaching the conversation (tested to
+stop audio within ~100ms). The next turn starts fresh.
+
 ```ts
 conversation.interrupt(); // stops TTS, cancels the current generation
 ```
@@ -301,12 +303,14 @@ This keeps interactions responsive even when the user interrupts the agent halfw
 The conversation emits a typed event stream the application can subscribe to:
 
 * `audio-in` — raw audio fed in via `listen()`
+* `text-in` — a finalized text turn via `send()`
 * `partial-transcript` — live STT partials (captions)
 * `turn` — a finalized participant turn
 * `transcript` — a transcript entry
 * `audio` — generated audio to play
 * `generation` — an agent generation
-* `tool-call` / `tool-call-result` — tool execution round trips
+* `tool-call` / `tool-call-result` — every tool round trip (auto-executed by
+  default; observable even when the framework runs the tool)
 * `interrupt` — an interruption occurred
 * `error` — a provider failure
 * `start` / `stop` / `state` — lifecycle
@@ -318,6 +322,9 @@ conversation.on("partial-transcript", ({ text }) => captions.update(text));
 
 ## Multi-participant conversations
 
+<details>
+<summary>Batching participants, conversational state, and the roadmap for floor management and addressing</summary>
+
 Conversations can contain multiple participants and agents.
 
 ```ts
@@ -333,26 +340,25 @@ await conversation.participate([
 ]);
 ```
 
-The conversation orchestrator maintains conversational state such as:
-
-* active turns
-* participant identity
-* interruptions
-* agent generations
-* transcript state
-
 In a one-to-one conversation, speech is treated as an interaction: every
 finalized participant turn produces an agent generation.
 
-Multi-participant floor management and addressing (determining when an agent is
-being spoken to using participant context, agent names, aliases, and floor
-rules) are on the roadmap. A wake word is not intended to be a fundamental
-requirement.
+Addressing works at a basic level: in a multi-agent conversation, a turn is
+routed to the agent whose name or alias appears in the speech, and unaddressed
+turns go through the built-in `understand` coordination, which may delegate to
+agents, ask the user, or answer directly. Floor management, multi-participant
+turn-taking rules, and richer addressing heuristics are on the roadmap. A wake
+word is not intended to be a fundamental requirement.
+
+</details>
 
 ## Agents
 
 An agent defines an AI persona and its capabilities.
 
+See [src/agents/README.md](src/agents/README.md) for the agent and tool
+abstractions.
+
 ```ts
 const jarvis = pipeflow.agent({
   name: "Jarvis",
@@ -385,39 +391,119 @@ This is useful for ordinary LLM workloads where realtime audio and conversation
 
 When an agent is attached to a conversation, it can take conversational turns as part of the realtime orchestration.
 
-The conversation owns the runtime.
+A conversation can coordinate **multiple agents**: `create({ agents })` accepts
+a roster. A turn that explicitly addresses an agent by name or alias goes
+straight to that agent; unaddressed turns go through the built-in `understand`
+coordination, which decides whether to delegate, ask the user, or answer
+directly. In a single-agent conversation, every turn goes to that agent. Each
+agent keeps its own context and its own LLM; the conversation owns the shared
+runtime (STT, TTS, history, interruptions).
 
-The agent owns the intelligence.
+```ts
+const receptionist = pipeflow.agent({
+  name: "Receptionist",
+  context: `
+    You are the front desk. Greet people and route requests.
+  `,
+});
 
-```text
-Agent
- ├── context
- └── tools
+const specialist = pipeflow.agent({
+  name: "Technical Specialist",
+  aliases: ["tech"],
+  context: `
+    You are the technical support specialist.
+  `,
+});
 
-Conversation
- ├── participants
- ├── turns
- ├── audio
- ├── interruption
- └── orchestration
+const conversation =
+  await pipeflow.conversations.create({
+    agents: [receptionist, specialist],
+  });
 ```
 
+"Ask the technical specialist about X" is routed to the specialist; an
+unaddressed turn goes through the built-in `understand` coordination.
+
+### Collaborative orchestration
+
+`understand` is a hardcoded **coordination**: a reasoning unit that decides
+what should happen next rather than doing the work itself. It can:
+
+- **delegate** to one or more agents (in parallel), each with a self-contained
+  prompt;
+- **pass the work to another coordination**;
+- **ask the user** for missing details via a structured, batched `clarify`
+  action — every missing detail goes into one question, and question rounds
+  are capped per run (default 2), after which reasonable assumptions are
+  stated;
+- **complete** with a direct answer.
+
+```text
+User: "Book a flight and check whether my calendar conflicts."
+              │
+              ▼
+        ┌────────────┐
+        │ understand │  ← built-in coordination
+        └─────┬──────┘
+              │
+       ┌──────┴───────┐
+       ▼              ▼
+   Travel Agent   Calendar Agent
+       │              │
+       └──────┬───────┘
+              ▼
+          understand
+              │
+              ▼
+            User
+```
+
+Agent delegation runs as **sub-generations**: the target agent executes on its
+own LLM, context, and tools (which still run in your backend), and every
+delegated prompt is stamped with the current time so time-sensitive tasks
+reason about the right "now". Delegated agents are text-only — the coordination
+narrates while they work and speaks the merged answer.
+
+Clarification is a first-class operation: an ambiguous request parks the
+coordination and **resumes on the next turn** instead of starting fresh.
+While waiting, participant speech is treated as the answer, not as a barge-in.
+
+See [src/conversations/orchestration/coordination/README.md](src/conversations/orchestration/coordination/README.md)
+for the coordination model and
+[src/conversations/orchestration/orchestrator/README.md](src/conversations/orchestration/orchestrator/README.md)
+for how it is wired into the realtime pipeline.
+
 ## Tools
 
 Tools expose capabilities from your application to an agent.
 
 ```ts
+import { z } from "zod";
+
 const getWeather = new PipeflowTool({
   name: "get_weather",
 
   description: "Get the current weather for a city.",
 
+  // One zod schema drives both sides: it derives the JSON schema the model
+  // sees, and validates the arguments before execute() runs.
+  schema: {
+    in: z.object({ city: z.string().describe("The city to look up.") }),
+  },
+
   execute: async ({ city }) => {
     return weatherService.getCurrent(city);
   },
 });
 ```
 
+`schema.in` is what the model sends; `schema.out` (optional, defaults to `in`)
+is what `execute` receives after validation, and may transform the arguments.
+Prefer `schema` — it keeps the model contract and your `execute` signature in
+sync and rejects bad arguments with a clear error instead of crashing inside
+the tool. If you need a hand-written JSON schema instead, pass `parameters`
+(no runtime validation).
+
 Then:
 
 ```ts
@@ -430,9 +516,8 @@ const jarvis = pipeflow.agent({
 });
 ```
 
-Tools execute in **your backend**.
-
-Pipeflow does not execute arbitrary application code.
+Tools execute in **your backend**: the `execute` function is your code, and it
+runs in your process — Pipeflow only invokes the callbacks you registered.
 
 ```text
                     Pipeflow
@@ -455,18 +540,44 @@ Pipeflow does not execute arbitrary application code.
 
 This allows tools to access your database, APIs, Discord bot, business logic, filesystem, or anything else your application controls.
 
-In a conversation, tool calls never execute inside Pipeflow. The orchestrator
-emits a `tool-call` event with the requested tool and its arguments; your
-backend executes it and reports back:
+### In a conversation: tools auto-execute
+
+Attach tools to an agent and they just work — the orchestrator runs each tool
+and feeds the result back into the generation (same loop as `Agent.run()`):
 
 ```ts
-conversation.on("tool-call", async ({ call }) => {
-  const result = await myBackend.execute(call.name, call.arguments);
+const jarvis = pipeflow.agent({
+  name: "Jarvis",
+  context: "You are a helpful assistant.",
+  tools: [getWeather],
+});
 
-  conversation.resolveToolCall({
-    id: call.id,
-    result,
-  });
+const conversation = await pipeflow.conversations.create({ agents: [jarvis] });
+// no tool-call handler needed — get_weather runs automatically
+```
+
+The `tool-call` (and `tool-call-result`) events still fire for every call, so
+you can observe or log them. A thrown tool error is caught and returned to the
+model as `{ "error": ... }` so the agent can recover; an unknown tool name
+reports `Unknown tool "..."` the same way. Tool calls the model issues together
+run concurrently, and a hung tool is bounded by `toolTimeoutMs` (default 30s).
+
+### Opt out: resolve tool calls yourself
+
+When a tool needs approval, or runs in a *different* backend, pass
+`autoExecuteTools: false` (on `Pipeflow`, `pipeflow.conversations.create()`, or
+`Conversation`) and the orchestrator hands the call to you:
+
+```ts
+const conversation = await pipeflow.conversations.create({
+  agents: [jarvis],
+  autoExecuteTools: false,
+});
+
+conversation.on("tool-call", async ({ call }) => {
+  // call.arguments is already parsed JSON
+  const result = await myBackend.executeTool(call.name, call.arguments);
+  conversation.resolveToolCall({ id: call.id, result });
 });
 ```
 
@@ -483,16 +594,9 @@ This makes Pipeflow useful as a realtime transcription primitive.
 const conversation =
   await pipeflow.conversations.create();
 
-// Without an agent, the orchestrator runs in transcription-only mode:
-// audio in, turns and transcripts out.
-const orchestrator = new Orchestrator({
-  conversation,
-  stt,
-});
-
-await orchestrator.start();
-
-conversation.start();
+// Without an agent, `start()` attaches the orchestrator in
+// transcription-only mode: audio in, turns and transcripts out.
+await conversation.start();
 
 await conversation.participate([
   { userId: "alice" },
@@ -524,6 +628,9 @@ Pagination can be supported for long conversations.
 
 ## Meeting summaries
 
+<details>
+<summary>Summarizing a meeting with a notetaker agent — as a plain LLM task or through a transcript tool</summary>
+
 A meeting summary can simply be another agent task.
 
 ```ts
@@ -585,38 +692,22 @@ The agent does not receive special access to conversations.
 
 If it needs conversation data, **a tool provides that capability**.
 
-## Providers
-
-Pipeflow separates provider interfaces from provider implementations.
-
-```text
-providers/
-├── llm/
-│   ├── types.ts
-│   └── adapters/
-│       └── deepseek/
-│
-├── stt/
-│   ├── types.ts
-│   └── adapters/
-│       └── deepgram/
-│
-└── tts/
-    ├── types.ts
-    └── adapters/
-        └── kokoro/
-```
+</details>
 
-The orchestrator works against provider interfaces rather than directly against vendor APIs.
+## Providers
 
-This makes it possible to replace providers without changing the conversation layer.
+<details>
+<summary>Vendor-independent interfaces and the current adapters: Deepgram, DeepSeek, OpenRouter, Kokoro</summary>
 
-### Current providers
+Pipeflow separates provider interfaces (LLM, STT, TTS) from their
+implementations. The orchestrator works against the interfaces rather than
+directly against vendor APIs, so providers can be replaced without changing the
+conversation layer.
 
-The project currently contains adapters for:
+The project currently ships adapters for:
 
 * **STT:** Deepgram
-* **LLM:** DeepSeek
+* **LLM:** DeepSeek, OpenRouter, OpenAI, Claude (Anthropic)
 * **TTS:** Kokoro
 
 Providers are configured with their own credentials — Pipeflow itself does
@@ -624,8 +715,16 @@ not hold an API key.
 
 Provider availability and configuration are evolving during early development.
 
+See [src/providers/README.md](src/providers/README.md) for the interfaces and
+adapter contracts.
+
+</details>
+
 ## Architecture
 
+<details>
+<summary>How the layers fit together: agents, conversations, persistence, providers, transport</summary>
+
 Pipeflow is designed around a small number of independent layers.
 
 ```text
@@ -647,9 +746,11 @@ src/
 ### Conversation
 
 Public realtime conversation API and lifecycle.
+See [src/conversations/conversation/README.md](src/conversations/conversation/README.md).
 
 ### Orchestration
 
+See [src/conversations/orchestration/README.md](src/conversations/orchestration/README.md).
 The state machine coordinating:
 
 * speech
@@ -664,40 +765,54 @@ The state machine coordinating:
 ### Transcription
 
 Conversation transcription and transcript state.
+See [src/conversations/transcription/README.md](src/conversations/transcription/README.md).
 
 ### Providers
 
 Vendor-independent interfaces and provider adapters.
+See [src/providers/README.md](src/providers/README.md).
 
 ### Persistence
 
 Persistence abstractions with adapters such as SQLite and in-memory storage.
+See [src/persistence/README.md](src/persistence/README.md).
 
 ### Transport
 
-Realtime communication between Pipeflow and the application using Pipeflow.
+Realtime communication between Pipeflow and the application.
+See [src/transport/README.md](src/transport/README.md).
+
+</details>
 
 ## Persistence
 
-Pipeflow separates persistence from the conversation domain.
+<details>
+<summary>Storage adapters: in-memory for development, SQLite for lightweight persistence</summary>
 
-The project currently includes:
+Pipeflow separates persistence from the conversation domain. The in-memory
+adapter is useful for tests and development; SQLite provides a lightweight
+persistent backend suitable for local applications and early deployments. The
+persistence interface is intentionally provider-independent so other storage
+implementations can be added later.
 
-```text
-persistence/
-└── adapters/
-    ├── memory/
-    └── sqlite/
-```
+```ts
+import { SQLitePersistence } from "@moureau/pipeflow/persistence";
 
-The in-memory adapter is useful for tests and development.
+const pipeflow = new Pipeflow({
+  persistence: new SQLitePersistence({ filename: "./pipeflow.db" }),
+});
+```
 
-SQLite provides a lightweight persistent backend suitable for local applications and early deployments.
+See [src/persistence/README.md](src/persistence/README.md) for the storage
+contract and adapters.
 
-The persistence interface is intentionally provider-independent so other storage implementations can be added later.
+</details>
 
 ## Realtime architecture
 
+<details>
+<summary>How audio flows through the pipeline as a continuous stream</summary>
+
 A typical voice interaction looks like:
 
 ```text
@@ -752,14 +867,21 @@ audio
 
 This allows the system to begin producing speech as early as possible.
 
+</details>
+
 ## Open source
 
+<details>
+<summary>Why the project is open and how the provider layer stays modular</summary>
+
 Pipeflow is open source.
 
 The project is designed to make realtime voice infrastructure accessible without requiring applications to implement their own orchestration layer.
 
 The provider layer is intentionally modular so applications can choose between hosted and self-hosted services.
 
+</details>
+
 ## Development
 
 Clone the repository and install dependencies:
@@ -776,6 +898,29 @@ Run tests:
 bun test
 ```
 
+End-to-end tests hit the real LLM API and are skipped when no key is
+available. They prefer `OPENROUTER_API_KEY` (default model
+`google/gemini-2.5-flash-lite`, override with `LLM_MODEL`) and fall back to
+`DEEPSEEK_API_KEY`. Add either to `.env` (loaded automatically) and run:
+
+```bash
+bun run test:e2e
+```
+
+The latency benchmark runs the pipeline repeatedly against the real model and
+reports p50/p95 per hop: first token, first speechable text, TTS request, TTS
+first audio, first audio delivered, and completion. STT and TTS are faked by
+default; point `KOKORO_URL` at a Kokoro endpoint to measure the real synthesis
+path, including inter-chunk audio gaps:
+
+```bash
+bun run benchmark                                  # 10 runs; BENCH_RUNS=5 to change
+KOKORO_URL=http://localhost:8880 bun run benchmark # local kokoro-fastapi
+KOKORO_URL=https://api.together.ai \
+  KOKORO_API_KEY=... KOKORO_MODEL=hexgrad/Kokoro-82M \
+  bun run benchmark                                # Together AI
+```
+
 Build and type-check:
 
 ```bash
@@ -787,6 +932,9 @@ The project uses Bun and TypeScript.
 
 ## Design principles
 
+<details>
+<summary>The rules the API is built around: realtime first, provider agnostic, app-owned tools</summary>
+
 ### Realtime first
 
 Audio is streamed continuously rather than processed as completed recordings.
@@ -828,6 +976,8 @@ Tool
 
 Everything else should remain replaceable implementation detail for as long as possible.
 
+</details>
+
 ## License
 
 See [LICENSE](LICENSE).