@alexkroman1/aai 0.7.2 → 0.7.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/dist/cli.js +337 -255
- package/dist/sdk/_internal_types.d.ts +4 -18
- package/dist/sdk/_internal_types.d.ts.map +1 -1
- package/dist/sdk/_internal_types.js +2 -1
- package/dist/sdk/_internal_types.js.map +1 -1
- package/dist/sdk/_mock_ws.js +1 -1
- package/dist/sdk/_mock_ws.js.map +1 -1
- package/dist/sdk/builtin_tools.d.ts +6 -2
- package/dist/sdk/builtin_tools.d.ts.map +1 -1
- package/dist/sdk/builtin_tools.js +1 -8
- package/dist/sdk/builtin_tools.js.map +1 -1
- package/dist/sdk/capnweb.d.ts +1 -1
- package/dist/sdk/capnweb.d.ts.map +1 -1
- package/dist/sdk/capnweb.js +43 -10
- package/dist/sdk/capnweb.js.map +1 -1
- package/dist/sdk/define_agent.d.ts +1 -1
- package/dist/sdk/define_agent.d.ts.map +1 -1
- package/dist/sdk/define_agent.js +26 -17
- package/dist/sdk/define_agent.js.map +1 -1
- package/dist/sdk/direct_executor.d.ts +2 -0
- package/dist/sdk/direct_executor.d.ts.map +1 -1
- package/dist/sdk/direct_executor.js +6 -1
- package/dist/sdk/direct_executor.js.map +1 -1
- package/dist/sdk/mod.d.ts +2 -1
- package/dist/sdk/mod.d.ts.map +1 -1
- package/dist/sdk/protocol.d.ts +88 -7
- package/dist/sdk/protocol.d.ts.map +1 -1
- package/dist/sdk/protocol.js.map +1 -1
- package/dist/sdk/runtime.d.ts +6 -4
- package/dist/sdk/runtime.d.ts.map +1 -1
- package/dist/sdk/runtime.js.map +1 -1
- package/dist/sdk/s2s.d.ts +2 -1
- package/dist/sdk/s2s.d.ts.map +1 -1
- package/dist/sdk/s2s.js +112 -73
- package/dist/sdk/s2s.js.map +1 -1
- package/dist/sdk/server.d.ts.map +1 -1
- package/dist/sdk/server.js +47 -43
- package/dist/sdk/server.js.map +1 -1
- package/dist/sdk/session.d.ts.map +1 -1
- package/dist/sdk/session.js +47 -44
- package/dist/sdk/session.js.map +1 -1
- package/dist/sdk/system_prompt.d.ts.map +1 -1
- package/dist/sdk/system_prompt.js +1 -1
- package/dist/sdk/system_prompt.js.map +1 -1
- package/dist/sdk/types.d.ts +8 -50
- package/dist/sdk/types.d.ts.map +1 -1
- package/dist/sdk/types.js +0 -8
- package/dist/sdk/types.js.map +1 -1
- package/dist/sdk/winterc_server.d.ts +4 -1
- package/dist/sdk/winterc_server.d.ts.map +1 -1
- package/dist/sdk/winterc_server.js +3 -2
- package/dist/sdk/winterc_server.js.map +1 -1
- package/dist/sdk/worker_entry.d.ts +3 -1
- package/dist/sdk/worker_entry.d.ts.map +1 -1
- package/dist/sdk/worker_entry.js +24 -17
- package/dist/sdk/worker_entry.js.map +1 -1
- package/dist/sdk/worker_shim.d.ts.map +1 -1
- package/dist/sdk/worker_shim.js +62 -9
- package/dist/sdk/worker_shim.js.map +1 -1
- package/dist/sdk/ws_handler.d.ts.map +1 -1
- package/dist/sdk/ws_handler.js +65 -58
- package/dist/sdk/ws_handler.js.map +1 -1
- package/dist/ui/_components/message_list.d.ts.map +1 -1
- package/dist/ui/_components/message_list.js +10 -6
- package/dist/ui/_components/message_list.js.map +1 -1
- package/dist/ui/audio.js +1 -1
- package/dist/ui/audio.js.map +1 -1
- package/dist/ui/mod.d.ts +10 -2
- package/dist/ui/mod.d.ts.map +1 -1
- package/dist/ui/mod.js +5 -2
- package/dist/ui/mod.js.map +1 -1
- package/dist/ui/session.d.ts.map +1 -1
- package/dist/ui/session.js +3 -1
- package/dist/ui/session.js.map +1 -1
- package/package.json +4 -2
- package/templates/.env +1 -0
- package/templates/_shared/.env.example +5 -0
- package/templates/_shared/CLAUDE.md +710 -0
- package/templates/_shared/package.json +17 -0
- package/templates/_shared/tsconfig.json +16 -0
- package/templates/code-interpreter/agent.ts +27 -0
- package/templates/code-interpreter/client.tsx +2 -0
- package/templates/dispatch-center/agent.ts +1564 -0
- package/templates/dispatch-center/client.tsx +504 -0
- package/templates/embedded-assets/agent.ts +49 -0
- package/templates/embedded-assets/client.tsx +2 -0
- package/templates/embedded-assets/knowledge.json +20 -0
- package/templates/health-assistant/agent.ts +160 -0
- package/templates/health-assistant/client.tsx +2 -0
- package/templates/infocom-adventure/agent.ts +164 -0
- package/templates/infocom-adventure/client.tsx +299 -0
- package/templates/math-buddy/agent.ts +21 -0
- package/templates/math-buddy/client.tsx +2 -0
- package/templates/memory-agent/agent.ts +74 -0
- package/templates/memory-agent/client.tsx +2 -0
- package/templates/night-owl/agent.ts +98 -0
- package/templates/night-owl/client.tsx +28 -0
- package/templates/personal-finance/agent.ts +26 -0
- package/templates/personal-finance/client.tsx +2 -0
- package/templates/simple/agent.ts +6 -0
- package/templates/simple/client.tsx +2 -0
- package/templates/smart-research/agent.ts +164 -0
- package/templates/smart-research/client.tsx +2 -0
- package/templates/support/README.md +62 -0
- package/templates/support/agent.ts +19 -0
- package/templates/support/client.tsx +2 -0
- package/templates/travel-concierge/agent.ts +29 -0
- package/templates/travel-concierge/client.tsx +2 -0
- package/templates/tsconfig.json +1 -0
- package/templates/web-researcher/agent.ts +17 -0
- package/templates/web-researcher/client.tsx +2 -0
- package/dist/sdk/_timeout.d.ts +0 -14
- package/dist/sdk/_timeout.d.ts.map +0 -1
- package/dist/sdk/_timeout.js +0 -24
- package/dist/sdk/_timeout.js.map +0 -1
|
@@ -0,0 +1,710 @@
|
|
|
1
|
+
# Build a voice agent with `aai`
|
|
2
|
+
|
|
3
|
+
You are helping a user build a voice agent using the **aai** framework. Generate
|
|
4
|
+
or update files based on the user's description in `$ARGUMENTS`.
|
|
5
|
+
|
|
6
|
+
## Workflow
|
|
7
|
+
|
|
8
|
+
1. **Understand** — Restate what the user wants to build. If the request is
|
|
9
|
+
vague, ask a clarifying question before writing code.
|
|
10
|
+
2. **Check existing work** — Look for a template or built-in tool that already
|
|
11
|
+
does what the user needs before writing custom code.
|
|
12
|
+
3. **Start minimal** — Scaffold from the closest template, then layer on
|
|
13
|
+
customizations. Don't over-engineer the first version.
|
|
14
|
+
4. **Iterate** — Make small, focused changes. Verify each change works before
|
|
15
|
+
moving on.
|
|
16
|
+
|
|
17
|
+
## Getting started
|
|
18
|
+
|
|
19
|
+
### Use the `aai` CLI
|
|
20
|
+
|
|
21
|
+
Always use the `aai` CLI to scaffold, deploy, and manage agents:
|
|
22
|
+
|
|
23
|
+
```sh
|
|
24
|
+
aai # Scaffold (if needed) + deploy
|
|
25
|
+
aai new # Scaffold a new agent (interactive)
|
|
26
|
+
aai new -t <template> # Scaffold from a specific template
|
|
27
|
+
aai deploy # Bundle and deploy to production
|
|
28
|
+
aai deploy -y # Deploy without prompts
|
|
29
|
+
aai deploy --dry-run # Validate and bundle without deploying
|
|
30
|
+
aai env add <NAME> # Set an environment variable on the server
|
|
31
|
+
aai env rm <NAME> # Remove an environment variable
|
|
32
|
+
aai env ls # List environment variable names
|
|
33
|
+
aai env pull # Pull env var names into .env for local dev
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
Install: `curl -fsSL https://aai-agent.fly.dev/install | sh`
|
|
37
|
+
|
|
38
|
+
### Deploy a scaffolded project
|
|
39
|
+
|
|
40
|
+
After scaffolding with `aai new`, deploy from the project directory:
|
|
41
|
+
|
|
42
|
+
```sh
|
|
43
|
+
cd my-agent
|
|
44
|
+
aai deploy # Bundle, check, and deploy
|
|
45
|
+
aai deploy -y # Skip confirmation prompts
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The CLI auto-detects the server URL. When running via `aai-dev` (the local
|
|
49
|
+
monorepo dev wrapper), it targets `http://localhost:3100` automatically.
|
|
50
|
+
|
|
51
|
+
### Start from a template
|
|
52
|
+
|
|
53
|
+
Before writing an agent from scratch, **choose the closest template** and
|
|
54
|
+
scaffold with `aai new -t <template_name>`. Ask the user which template fits, or
|
|
55
|
+
recommend one based on their description. Fall back to `simple` if nothing else
|
|
56
|
+
fits.
|
|
57
|
+
|
|
58
|
+
Templates are in `templates/` relative to the CLI source:
|
|
59
|
+
|
|
60
|
+
| Template | Description |
|
|
61
|
+
| ------------------- | ---------------------------------------------------------------------------------- |
|
|
62
|
+
| `simple` | Minimal starter with web_search, visit_webpage, fetch_json, run_code. **Default.** |
|
|
63
|
+
| `web-researcher` | Research assistant — web search + page visits for detailed answers |
|
|
64
|
+
| `smart-research` | Phase-based research (gather → analyze → respond) with dynamic tool filtering |
|
|
65
|
+
| `memory-agent` | Persistent KV storage — remembers facts and preferences across conversations |
|
|
66
|
+
| `code-interpreter` | Writes and runs JavaScript for math, calculations, data processing |
|
|
67
|
+
| `math-buddy` | Calculations, unit conversions, dice rolls via run_code |
|
|
68
|
+
| `health-assistant` | Medication lookup, drug interactions, BMI, symptom guidance |
|
|
69
|
+
| `personal-finance` | Currency conversion, crypto prices, loan calculations, savings projections |
|
|
70
|
+
| `travel-concierge` | Trip planning, weather, flights, hotels, currency conversion |
|
|
71
|
+
| `night-owl` | Movie/music/book recs by mood, sleep calculator. **Has custom UI.** |
|
|
72
|
+
| `dispatch-center` | 911 dispatch with incident triage and resource assignment. **Has custom UI.** |
|
|
73
|
+
| `infocom-adventure` | Zork-style text adventure with state, puzzles, inventory. **Has custom UI.** |
|
|
74
|
+
| `embedded-assets` | FAQ bot using embedded JSON knowledge (no web search) |
|
|
75
|
+
| `support` | RAG-powered support agent using vector_search (AssemblyAI docs example) |
|
|
76
|
+
| `terminal` | STT-only mode for voice-driven kubectl commands |
|
|
77
|
+
|
|
78
|
+
### Minimal agent
|
|
79
|
+
|
|
80
|
+
Every agent lives in `agent.ts` and exports a default `defineAgent()` call:
|
|
81
|
+
|
|
82
|
+
```ts
|
|
83
|
+
import { defineAgent } from "aai";
|
|
84
|
+
|
|
85
|
+
export default defineAgent({
|
|
86
|
+
name: "My Agent",
|
|
87
|
+
instructions: "You are a helpful assistant that...",
|
|
88
|
+
greeting: "Hey there. What can I help you with?",
|
|
89
|
+
voice: "694f9389-aac1-45b6-b726-9d9369183238", // Sarah
|
|
90
|
+
});
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
### Imports
|
|
94
|
+
|
|
95
|
+
```ts
|
|
96
|
+
import { defineAgent } from "aai"; // Always needed
|
|
97
|
+
import type { BeforeStepResult, HookContext, ToolContext } from "aai"; // Type annotations
|
|
98
|
+
import { z } from "zod"; // Tools with typed params (included in package.json)
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## Agent configuration
|
|
102
|
+
|
|
103
|
+
```ts
|
|
104
|
+
defineAgent({
|
|
105
|
+
// Core
|
|
106
|
+
name: string; // Required: display name
|
|
107
|
+
instructions?: string; // System prompt (voice-first default provided)
|
|
108
|
+
greeting?: string; // Spoken on connect
|
|
109
|
+
voice?: Voice; // Cartesia voice UUID (default: Sarah)
|
|
110
|
+
|
|
111
|
+
// Speech
|
|
112
|
+
sttPrompt?: string; // STT guidance for jargon, names, acronyms
|
|
113
|
+
|
|
114
|
+
// Tools
|
|
115
|
+
builtinTools?: BuiltinTool[];
|
|
116
|
+
tools?: Record<string, ToolDef>;
|
|
117
|
+
toolChoice?: ToolChoice; // "auto" | "required" | "none" | { type: "tool", toolName }
|
|
118
|
+
activeTools?: string[]; // Default active tools per turn (subset of all tools)
|
|
119
|
+
maxSteps?: number | ((ctx: HookContext) => number);
|
|
120
|
+
|
|
121
|
+
// Environment
|
|
122
|
+
|
|
123
|
+
// State
|
|
124
|
+
state?: () => S; // Factory for per-session state
|
|
125
|
+
|
|
126
|
+
// Lifecycle hooks
|
|
127
|
+
onConnect?: (ctx: HookContext) => void | Promise<void>;
|
|
128
|
+
onDisconnect?: (ctx: HookContext) => void | Promise<void>;
|
|
129
|
+
onError?: (error: Error, ctx?: HookContext) => void;
|
|
130
|
+
onTurn?: (text: string, ctx: HookContext) => void | Promise<void>;
|
|
131
|
+
onStep?: (step: StepInfo, ctx: HookContext) => void | Promise<void>;
|
|
132
|
+
onBeforeStep?: (stepNumber: number, ctx: HookContext) =>
|
|
133
|
+
BeforeStepResult | Promise<BeforeStepResult>;
|
|
134
|
+
});
|
|
135
|
+
```
|
|
136
|
+
|
|
137
|
+
### Voices
|
|
138
|
+
|
|
139
|
+
Voices use Cartesia voice UUIDs. Browse all voices at
|
|
140
|
+
[play.cartesia.ai](https://play.cartesia.ai).
|
|
141
|
+
|
|
142
|
+
Common voices:
|
|
143
|
+
|
|
144
|
+
| Name | Voice ID |
|
|
145
|
+
| --------------------- | -------------------------------------- |
|
|
146
|
+
| Sarah (default) | `694f9389-aac1-45b6-b726-9d9369183238` |
|
|
147
|
+
| Customer Support Man | `a167e0f3-df7e-4d52-a9c3-f949145efdab` |
|
|
148
|
+
| Customer Support Lady | `829ccd10-f8b3-43cd-b8a0-4aeaa81f3b30` |
|
|
149
|
+
| Helpful Woman | `156fb8d2-335b-4950-9cb3-a2d33befec77` |
|
|
150
|
+
| Professional Woman | `248be419-c632-4f23-adf1-5324ed7dbf1d` |
|
|
151
|
+
| Sweet Lady | `e3827ec5-697a-4b7c-9704-1a23041bbc51` |
|
|
152
|
+
| British Lady | `79a125e8-cd45-4c13-8a67-188112f4dd22` |
|
|
153
|
+
| Calm Lady | `00a77add-48d5-4ef6-8157-71e5437b282d` |
|
|
154
|
+
| Laidback Woman | `21b81c14-f85b-436d-aff5-43f2e788ecf8` |
|
|
155
|
+
| Storyteller Lady | `996a8b96-4804-46f0-8e05-3fd4ef1a87cd` |
|
|
156
|
+
| Newslady | `bf991597-6c13-47e4-8411-91ec2de5c466` |
|
|
157
|
+
| Friendly Reading Man | `69267136-1bdc-412f-ad78-0caad210fb40` |
|
|
158
|
+
| Confident British Man | `63ff761f-c1e8-414b-b969-d1833d1c870c` |
|
|
159
|
+
| New York Man | `34575e71-908f-4ab6-ab54-b08c95d6597d` |
|
|
160
|
+
| California Girl | `b7d50908-b17c-442d-ad8d-810c63997ed9` |
|
|
161
|
+
| Newsman | `d46abd1d-2d02-43e8-819f-51fb652c1c61` |
|
|
162
|
+
| Salesman | `820a3788-2b37-4d21-847a-b65d8a68c99a` |
|
|
163
|
+
| Wise Man | `b043dea0-a007-4bbe-a708-769dc0d0c569` |
|
|
164
|
+
| Child | `2ee87190-8f84-4925-97da-e52547f9462c` |
|
|
165
|
+
|
|
166
|
+
Any Cartesia voice UUID works — the list above is just a starting point.
|
|
167
|
+
|
|
168
|
+
Use `sttPrompt` for domain-specific vocabulary:
|
|
169
|
+
|
|
170
|
+
```ts
|
|
171
|
+
export default defineAgent({
|
|
172
|
+
voice: "a167e0f3-df7e-4d52-a9c3-f949145efdab", // Customer Support Man
|
|
173
|
+
sttPrompt: "Transcribe technical terms: Kubernetes, gRPC, PostgreSQL",
|
|
174
|
+
});
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
### Writing good `instructions`
|
|
178
|
+
|
|
179
|
+
Optimize for spoken conversation:
|
|
180
|
+
|
|
181
|
+
- Short, punchy sentences — optimize for speech, not text
|
|
182
|
+
- Never mention "search results" or "sources" — speak as if knowledge is your
|
|
183
|
+
own
|
|
184
|
+
- No visual formatting ("bullet point", "bold") — use "First", "Next", "Finally"
|
|
185
|
+
- Lead with the most important information
|
|
186
|
+
- Be concise and confident — no hedging ("It seems that", "I believe")
|
|
187
|
+
- No exclamation points — calm, conversational tone
|
|
188
|
+
- Define personality, tone, and specialty
|
|
189
|
+
- Include when and how to use each tool
|
|
190
|
+
|
|
191
|
+
### Environment variables
|
|
192
|
+
|
|
193
|
+
Secrets are managed on the server via the CLI, like `vercel env`. They are
|
|
194
|
+
injected into agent workers at runtime and available as `ctx.env`. Secrets are
|
|
195
|
+
**never** embedded in the bundled code.
|
|
196
|
+
|
|
197
|
+
```sh
|
|
198
|
+
# Set secrets on the server (prompts for value)
|
|
199
|
+
aai env add ASSEMBLYAI_API_KEY
|
|
200
|
+
aai env add MY_API_KEY
|
|
201
|
+
|
|
202
|
+
# List what's set
|
|
203
|
+
aai env ls
|
|
204
|
+
|
|
205
|
+
# Pull env var names into .env for local dev reference
|
|
206
|
+
aai env pull
|
|
207
|
+
|
|
208
|
+
# Remove a secret
|
|
209
|
+
aai env rm MY_API_KEY
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
Declare required env vars in the agent config so the CLI validates them at
|
|
213
|
+
deploy time:
|
|
214
|
+
|
|
215
|
+
```ts
|
|
216
|
+
export default defineAgent({
|
|
217
|
+
name: "API Agent",
|
|
218
|
+
env: ["ASSEMBLYAI_API_KEY", "MY_API_KEY"],
|
|
219
|
+
// ...
|
|
220
|
+
});
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
Access secrets in tool code via `ctx.env`:
|
|
224
|
+
|
|
225
|
+
```ts
|
|
226
|
+
import { defineAgent } from "aai";
|
|
227
|
+
import { z } from "zod";
|
|
228
|
+
|
|
229
|
+
export default defineAgent({
|
|
230
|
+
name: "API Agent",
|
|
231
|
+
env: ["ASSEMBLYAI_API_KEY", "MY_API_KEY"],
|
|
232
|
+
tools: {
|
|
233
|
+
call_api: {
|
|
234
|
+
description: "Call an external API",
|
|
235
|
+
parameters: z.object({ query: z.string() }),
|
|
236
|
+
execute: async (args, ctx) => {
|
|
237
|
+
const res = await fetch(`https://api.example.com?q=${args.query}`, {
|
|
238
|
+
headers: { Authorization: `Bearer ${ctx.env.MY_API_KEY}` },
|
|
239
|
+
});
|
|
240
|
+
return res.json();
|
|
241
|
+
},
|
|
242
|
+
},
|
|
243
|
+
},
|
|
244
|
+
});
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
## Tools
|
|
248
|
+
|
|
249
|
+
### Custom tools
|
|
250
|
+
|
|
251
|
+
Define tools as plain objects in the `tools` record. The `parameters` field
|
|
252
|
+
takes a Zod schema for type-safe argument inference:
|
|
253
|
+
|
|
254
|
+
```ts
|
|
255
|
+
import { defineAgent } from "aai";
|
|
256
|
+
import { z } from "zod";
|
|
257
|
+
|
|
258
|
+
export default defineAgent({
|
|
259
|
+
name: "Weather Agent",
|
|
260
|
+
tools: {
|
|
261
|
+
get_weather: {
|
|
262
|
+
description: "Get current weather for a city",
|
|
263
|
+
parameters: z.object({
|
|
264
|
+
city: z.string().describe("City name"),
|
|
265
|
+
}),
|
|
266
|
+
execute: async (args, ctx) => {
|
|
267
|
+
const data = await fetch(
|
|
268
|
+
`https://api.example.com/weather?q=${args.city}`,
|
|
269
|
+
);
|
|
270
|
+
return data.json();
|
|
271
|
+
},
|
|
272
|
+
},
|
|
273
|
+
|
|
274
|
+
// No-parameter tools — omit `parameters`
|
|
275
|
+
list_items: {
|
|
276
|
+
description: "List all items",
|
|
277
|
+
execute: () => items,
|
|
278
|
+
},
|
|
279
|
+
},
|
|
280
|
+
});
|
|
281
|
+
```
|
|
282
|
+
|
|
283
|
+
Zod schema patterns:
|
|
284
|
+
|
|
285
|
+
```ts
|
|
286
|
+
parameters: z.object({
|
|
287
|
+
query: z.string().describe("Search query"),
|
|
288
|
+
category: z.enum(["a", "b", "c"]),
|
|
289
|
+
count: z.number().describe("How many"),
|
|
290
|
+
label: z.string().describe("Optional label").optional(),
|
|
291
|
+
}),
|
|
292
|
+
```
|
|
293
|
+
|
|
294
|
+
### Built-in tools
|
|
295
|
+
|
|
296
|
+
Enable via `builtinTools`.
|
|
297
|
+
|
|
298
|
+
| Tool | Description | Params |
|
|
299
|
+
| --------------- | ---------------------------------------------- | ----------------------------------- |
|
|
300
|
+
| `web_search` | Search the web (Brave Search) | `query`, `max_results?` (default 5) |
|
|
301
|
+
| `visit_webpage` | Fetch URL → Markdown | `url` |
|
|
302
|
+
| `fetch_json` | HTTP GET a JSON API | `url`, `headers?` |
|
|
303
|
+
| `run_code` | Execute JS in sandbox (no net/fs, 30s timeout) | `code` |
|
|
304
|
+
| `vector_search` | Search the agent's RAG knowledge base | `query`, `topK?` (default 5) |
|
|
305
|
+
|
|
306
|
+
The agentic loop runs up to `maxSteps` iterations (default 5) and stops when the
|
|
307
|
+
LLM produces a text response.
|
|
308
|
+
|
|
309
|
+
### Tool context
|
|
310
|
+
|
|
311
|
+
Every `execute` function and lifecycle hook receives a context object:
|
|
312
|
+
|
|
313
|
+
```ts
|
|
314
|
+
ctx.sessionId; // string — unique per connection
|
|
315
|
+
ctx.env; // Record<string, string> — secrets from `aai env add`
|
|
316
|
+
ctx.abortSignal; // AbortSignal — cancelled on interruption (tools only)
|
|
317
|
+
ctx.state; // per-session state
|
|
318
|
+
ctx.kv; // persistent KV store
|
|
319
|
+
ctx.messages; // readonly Message[] — conversation history (tools only)
|
|
320
|
+
```
|
|
321
|
+
|
|
322
|
+
Hooks get `HookContext` (same but without `abortSignal` and `messages`).
|
|
323
|
+
|
|
324
|
+
### Fetching external APIs
|
|
325
|
+
|
|
326
|
+
Use `fetch` directly in tool execute functions:
|
|
327
|
+
|
|
328
|
+
```ts
|
|
329
|
+
execute: async (args, ctx) => {
|
|
330
|
+
const resp = await fetch(url, {
|
|
331
|
+
headers: { Authorization: `Bearer ${ctx.env.API_KEY}` },
|
|
332
|
+
signal: ctx.abortSignal, // Respect interruptions
|
|
333
|
+
});
|
|
334
|
+
if (!resp.ok) return { error: `${resp.status} ${resp.statusText}` };
|
|
335
|
+
return resp.json();
|
|
336
|
+
},
|
|
337
|
+
```
|
|
338
|
+
|
|
339
|
+
`fetch` is proxied through the host process (the worker has no direct network
|
|
340
|
+
access). All URLs are validated against SSRF rules — only public addresses are
|
|
341
|
+
allowed.
|
|
342
|
+
|
|
343
|
+
## State and storage
|
|
344
|
+
|
|
345
|
+
### Per-session state
|
|
346
|
+
|
|
347
|
+
For data that lasts only one connection (games, workflows, multi-step
|
|
348
|
+
processes). Fresh state is created per session and cleaned up on disconnect:
|
|
349
|
+
|
|
350
|
+
```ts
|
|
351
|
+
export default defineAgent({
|
|
352
|
+
state: () => ({ score: 0, question: 0 }),
|
|
353
|
+
tools: {
|
|
354
|
+
answer: {
|
|
355
|
+
description: "Submit an answer",
|
|
356
|
+
parameters: z.object({ answer: z.string() }),
|
|
357
|
+
execute: (args, ctx) => {
|
|
358
|
+
const state = ctx.state as { score: number; question: number };
|
|
359
|
+
state.question++;
|
|
360
|
+
return state;
|
|
361
|
+
},
|
|
362
|
+
},
|
|
363
|
+
},
|
|
364
|
+
});
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
### Persistent storage (KV)
|
|
368
|
+
|
|
369
|
+
`ctx.kv` is a persistent key-value store scoped per agent. Values are
|
|
370
|
+
auto-serialized as JSON.
|
|
371
|
+
|
|
372
|
+
```ts
|
|
373
|
+
await ctx.kv.set("user:123", { name: "Alice" }); // save
|
|
374
|
+
await ctx.kv.set("temp:x", value, { expireIn: 60_000 }); // save with TTL (ms)
|
|
375
|
+
const user = await ctx.kv.get<User>("user:123"); // read (or null)
|
|
376
|
+
const notes = await ctx.kv.list("note:", { limit: 10, reverse: true }); // list by prefix
|
|
377
|
+
await ctx.kv.delete("user:123"); // delete
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
Keys are strings; use colon-separated prefixes (`"user:123"`). Max value: 64 KB.
|
|
381
|
+
|
|
382
|
+
`kv.list()` returns `KvEntry[]` where each entry has
|
|
383
|
+
`{ key: string, value: T }`.
|
|
384
|
+
|
|
385
|
+
## Advanced patterns
|
|
386
|
+
|
|
387
|
+
### Step hooks
|
|
388
|
+
|
|
389
|
+
`onStep` — called after each LLM step (logging, analytics):
|
|
390
|
+
|
|
391
|
+
```ts
|
|
392
|
+
onStep: (step, ctx) => {
|
|
393
|
+
console.log(`Step ${step.stepNumber}: ${step.toolCalls.length} tool calls`);
|
|
394
|
+
},
|
|
395
|
+
```
|
|
396
|
+
|
|
397
|
+
`onBeforeStep` — return `{ activeTools: [...] }` to filter tools per step:
|
|
398
|
+
|
|
399
|
+
```ts
|
|
400
|
+
state: () => ({ phase: "gather" }),
|
|
401
|
+
onBeforeStep: (stepNumber, ctx) => {
|
|
402
|
+
const state = ctx.state as { phase: string };
|
|
403
|
+
if (state.phase === "gather") {
|
|
404
|
+
return { activeTools: ["search", "lookup"] };
|
|
405
|
+
}
|
|
406
|
+
return { activeTools: ["summarize"] };
|
|
407
|
+
},
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
### Static `activeTools`
|
|
411
|
+
|
|
412
|
+
Restrict which tools the LLM can use by default, without writing a hook:
|
|
413
|
+
|
|
414
|
+
```ts
|
|
415
|
+
export default defineAgent({
|
|
416
|
+
builtinTools: ["web_search", "visit_webpage", "run_code"],
|
|
417
|
+
tools: { summarize: {/* ... */} },
|
|
418
|
+
activeTools: ["web_search", "summarize"], // Only these two are available
|
|
419
|
+
});
|
|
420
|
+
```
|
|
421
|
+
|
|
422
|
+
Use `onBeforeStep` to override `activeTools` dynamically per step.
|
|
423
|
+
|
|
424
|
+
### Dynamic `maxSteps`
|
|
425
|
+
|
|
426
|
+
```ts
|
|
427
|
+
maxSteps: (ctx) => {
|
|
428
|
+
const state = ctx.state as { complexity: string };
|
|
429
|
+
return state.complexity === "complex" ? 10 : 5;
|
|
430
|
+
},
|
|
431
|
+
```
|
|
432
|
+
|
|
433
|
+
### Conversation history in tools
|
|
434
|
+
|
|
435
|
+
```ts
|
|
436
|
+
execute: (args, ctx) => {
|
|
437
|
+
const userMessages = ctx.messages.filter(m => m.role === "user");
|
|
438
|
+
return { turns: userMessages.length };
|
|
439
|
+
},
|
|
440
|
+
```
|
|
441
|
+
|
|
442
|
+
### Embedded knowledge
|
|
443
|
+
|
|
444
|
+
```ts
|
|
445
|
+
import knowledge from "./knowledge.json" with { type: "json" };
|
|
446
|
+
|
|
447
|
+
export default defineAgent({
|
|
448
|
+
tools: {
|
|
449
|
+
search_faq: {
|
|
450
|
+
description: "Search the knowledge base",
|
|
451
|
+
parameters: z.object({ query: z.string() }),
|
|
452
|
+
execute: (args) =>
|
|
453
|
+
knowledge.faqs.filter((f: { question: string }) =>
|
|
454
|
+
f.question.toLowerCase().includes(args.query.toLowerCase())
|
|
455
|
+
),
|
|
456
|
+
},
|
|
457
|
+
},
|
|
458
|
+
});
|
|
459
|
+
```
|
|
460
|
+
|
|
461
|
+
### Using npm packages
|
|
462
|
+
|
|
463
|
+
Add packages to `package.json` dependencies:
|
|
464
|
+
|
|
465
|
+
```sh
|
|
466
|
+
npm install some-package
|
|
467
|
+
```
|
|
468
|
+
|
|
469
|
+
## Custom UI (`client.tsx`)
|
|
470
|
+
|
|
471
|
+
Add `client.tsx` alongside `agent.ts`. Define a Preact component and call
|
|
472
|
+
`mount()` to render it. Use JSX syntax:
|
|
473
|
+
|
|
474
|
+
```tsx
|
|
475
|
+
import { mount, useSession } from "aai/ui";
|
|
476
|
+
|
|
477
|
+
function App() {
|
|
478
|
+
const { session, started, running, start, toggle, reset } = useSession();
|
|
479
|
+
const msgs = session.messages.value;
|
|
480
|
+
const tx = session.userUtterance.value;
|
|
481
|
+
return (
|
|
482
|
+
<div>
|
|
483
|
+
{msgs.map((m, i) => <p key={i}>{m.text}</p>)}
|
|
484
|
+
{tx !== null && <p>{tx || "..."}</p>}
|
|
485
|
+
{!started.value ? <button onClick={start}>Start</button> : (
|
|
486
|
+
<>
|
|
487
|
+
<button onClick={toggle}>{running.value ? "Stop" : "Resume"}</button>
|
|
488
|
+
<button onClick={reset}>Reset</button>
|
|
489
|
+
</>
|
|
490
|
+
)}
|
|
491
|
+
</div>
|
|
492
|
+
);
|
|
493
|
+
}
|
|
494
|
+
|
|
495
|
+
mount(App);
|
|
496
|
+
```
|
|
497
|
+
|
|
498
|
+
**Rules:**
|
|
499
|
+
|
|
500
|
+
- Call `mount(YourComponent)` at the end of the file
|
|
501
|
+
- Use `.tsx` file extension for JSX syntax
|
|
502
|
+
- Import hooks from `preact/hooks` (`useEffect`, `useRef`, `useState`, etc.)
|
|
503
|
+
- Style with `style={{ color: "red" }}` or inject `<style>` for selectors,
|
|
504
|
+
keyframes, media queries
|
|
505
|
+
|
|
506
|
+
### `mount()` options
|
|
507
|
+
|
|
508
|
+
```ts
|
|
509
|
+
mount(App, {
|
|
510
|
+
target: "#app", // CSS selector or DOM element (default: "#app")
|
|
511
|
+
platformUrl: "...", // Server URL (auto-derived from location.href)
|
|
512
|
+
title: "My Agent", // Shown in header and start screen
|
|
513
|
+
theme: { // CSS custom property overrides
|
|
514
|
+
bg: "#101010", // Background color
|
|
515
|
+
primary: "#fab283", // Accent color
|
|
516
|
+
text: "#ffffff", // Text color
|
|
517
|
+
surface: "#1a1a1a", // Card/surface color
|
|
518
|
+
border: "#333333", // Border color
|
|
519
|
+
},
|
|
520
|
+
});
|
|
521
|
+
```
|
|
522
|
+
|
|
523
|
+
`mount()` returns a `MountHandle` with `session`, `signals`, and `dispose()`.
|
|
524
|
+
|
|
525
|
+
### Built-in components
|
|
526
|
+
|
|
527
|
+
Import from `aai/ui`:
|
|
528
|
+
|
|
529
|
+
| Component | Description |
|
|
530
|
+
| ------------------- | -------------------------------------------------- |
|
|
531
|
+
| `App` | Default full UI (start screen + ChatView) |
|
|
532
|
+
| `ChatView` | Chat interface with header, messages, and controls |
|
|
533
|
+
| `MessageBubble` | Single message (user right-aligned, agent left) |
|
|
534
|
+
| `Transcript` | Live STT text display |
|
|
535
|
+
| `StateIndicator` | Colored dot + agent state label |
|
|
536
|
+
| `ErrorBanner` | Red error box with message |
|
|
537
|
+
| `ThinkingIndicator` | Animated dots during processing |
|
|
538
|
+
| `ToolCallBlock` | Collapsible tool call display (name, args, result) |
|
|
539
|
+
|
|
540
|
+
Use `useMountConfig()` to access the `title` and `theme` passed to `mount()`.
|
|
541
|
+
|
|
542
|
+
### Session signals (`useSession()`)
|
|
543
|
+
|
|
544
|
+
`useSession()` returns
|
|
545
|
+
`{ session, started, running, start, toggle, reset, dispose }`. Reactive agent
|
|
546
|
+
data lives on `session` (a `VoiceSession`); UI-only controls are top-level.
|
|
547
|
+
|
|
548
|
+
| Signal / field | Type | Description |
|
|
549
|
+
| ----------------------------- | ---------------------- | ---------------------------------------------------------------------------------------- |
|
|
550
|
+
| `session.state.value` | `AgentState` | "disconnected", "connecting", "ready", "listening", "thinking", "speaking", "error" |
|
|
551
|
+
| `session.messages.value` | `Message[]` | `{ role, text }` objects |
|
|
552
|
+
| `session.toolCalls.value` | `ToolCallInfo[]` | `{ toolCallId, toolName, args, status, result?, afterMessageIndex }` — active tool calls |
|
|
553
|
+
| `session.userUtterance.value` | `string \| null` | `null` = not speaking, `""` = speech detected, string = transcript |
|
|
554
|
+
| `session.error.value` | `SessionError \| null` | `{ code, message }` |
|
|
555
|
+
| `session.disconnected.value` | `object \| null` | `{ intentional: boolean }` when disconnected, `null` when connected |
|
|
556
|
+
| `started.value` | `boolean` | Whether session has started |
|
|
557
|
+
| `running.value` | `boolean` | Whether session is active |
|
|
558
|
+
|
|
559
|
+
**Methods:** `start()`, `toggle()`, `reset()`, `dispose()`
|
|
560
|
+
|
|
561
|
+
### Showing tool calls in custom UI
|
|
562
|
+
|
|
563
|
+
```tsx
|
|
564
|
+
import { mount, ToolCallBlock, useSession } from "aai/ui";
|
|
565
|
+
|
|
566
|
+
function App() {
|
|
567
|
+
const { session, started, start } = useSession();
|
|
568
|
+
if (!started.value) return <button onClick={start}>Start</button>;
|
|
569
|
+
|
|
570
|
+
const msgs = session.messages.value;
|
|
571
|
+
const toolCalls = session.toolCalls.value;
|
|
572
|
+
|
|
573
|
+
return (
|
|
574
|
+
<div>
|
|
575
|
+
{msgs.map((m, i) => (
|
|
576
|
+
<div key={i}>
|
|
577
|
+
<p>{m.text}</p>
|
|
578
|
+
{toolCalls
|
|
579
|
+
.filter((tc) => tc.afterMessageIndex === i)
|
|
580
|
+
.map((tc) => <ToolCallBlock key={tc.toolCallId} toolCall={tc} />)}
|
|
581
|
+
</div>
|
|
582
|
+
))}
|
|
583
|
+
</div>
|
|
584
|
+
);
|
|
585
|
+
}
|
|
586
|
+
|
|
587
|
+
mount(App);
|
|
588
|
+
```
|
|
589
|
+
|
|
590
|
+
### Reacting to agent state
|
|
591
|
+
|
|
592
|
+
```tsx
|
|
593
|
+
import { useEffect } from "preact/hooks";
|
|
594
|
+
import { mount, StateIndicator, useSession } from "aai/ui";
|
|
595
|
+
|
|
596
|
+
function App() {
|
|
597
|
+
const { session, started, start } = useSession();
|
|
598
|
+
|
|
599
|
+
useEffect(() => {
|
|
600
|
+
// Run side effects when state changes
|
|
601
|
+
if (session.state.value === "speaking") {
|
|
602
|
+
// Agent is speaking — e.g., show animation
|
|
603
|
+
}
|
|
604
|
+
}, [session.state.value]);
|
|
605
|
+
|
|
606
|
+
return (
|
|
607
|
+
<div>
|
|
608
|
+
<StateIndicator />
|
|
609
|
+
{!started.value && <button onClick={start}>Start</button>}
|
|
610
|
+
</div>
|
|
611
|
+
);
|
|
612
|
+
}
|
|
613
|
+
|
|
614
|
+
mount(App);
|
|
615
|
+
```
|
|
616
|
+
|
|
617
|
+
### Styling custom UIs
|
|
618
|
+
|
|
619
|
+
The framework uses **Tailwind CSS v4** (compiled at bundle time). Three
|
|
620
|
+
approaches:
|
|
621
|
+
|
|
622
|
+
1. **Tailwind classes** — `class="flex items-center gap-2 bg-gray-900"`
|
|
623
|
+
2. **Inline styles** — `style={{ color: "red", padding: "1rem" }}`
|
|
624
|
+
3. **Injected `<style>` tags** — for keyframes, selectors, media queries:
|
|
625
|
+
|
|
626
|
+
```tsx
|
|
627
|
+
function App() {
|
|
628
|
+
return (
|
|
629
|
+
<>
|
|
630
|
+
<style>
|
|
631
|
+
{`
|
|
632
|
+
@keyframes pulse { 0%, 100% { opacity: 1 } 50% { opacity: 0.5 } }
|
|
633
|
+
.pulse { animation: pulse 2s ease-in-out infinite; }
|
|
634
|
+
@media (max-width: 640px) { .sidebar { display: none; } }
|
|
635
|
+
`}
|
|
636
|
+
</style>
|
|
637
|
+
<div class="pulse">Content</div>
|
|
638
|
+
</>
|
|
639
|
+
);
|
|
640
|
+
}
|
|
641
|
+
```
|
|
642
|
+
|
|
643
|
+
**CSS custom properties** available from the theme:
|
|
644
|
+
|
|
645
|
+
- `--color-aai-bg`, `--color-aai-primary`, `--color-aai-text`
|
|
646
|
+
- `--color-aai-surface`, `--color-aai-border`
|
|
647
|
+
- `--color-aai-state-{state}` — color for each `AgentState` value
|
|
648
|
+
|
|
649
|
+
## Project structure
|
|
650
|
+
|
|
651
|
+
After scaffolding, your project directory looks like:
|
|
652
|
+
|
|
653
|
+
```text
|
|
654
|
+
my-agent/
|
|
655
|
+
agent.ts # Agent definition
|
|
656
|
+
client.tsx # UI component (calls mount() to render into #app)
|
|
657
|
+
styles.css # Tailwind CSS entry point
|
|
658
|
+
package.json # Dependencies, scripts, and config
|
|
659
|
+
tsconfig.json # TypeScript configuration
|
|
660
|
+
.env.example # Reference for env var names
|
|
661
|
+
.env # Local dev secrets (gitignored)
|
|
662
|
+
.gitignore # Ignores node_modules/, .aai/, .env, etc.
|
|
663
|
+
README.md # Getting started guide
|
|
664
|
+
CLAUDE.md # Agent API reference (auto-generated)
|
|
665
|
+
.aai/ # Build output (managed by CLI, gitignored)
|
|
666
|
+
project.json # Deploy target (slug, server URL)
|
|
667
|
+
build/ # Bundle output
|
|
668
|
+
```
|
|
669
|
+
|
|
670
|
+
## Common pitfalls
|
|
671
|
+
|
|
672
|
+
- **Writing `instructions` with visual formatting** — Bullets, bold, numbered
|
|
673
|
+
lists sound terrible when spoken. Use natural transitions: "First", "Next",
|
|
674
|
+
"Finally". Write instructions as if you're coaching a human phone operator.
|
|
675
|
+
- **Returning huge payloads from tools** — Everything a tool returns goes into
|
|
676
|
+
the LLM context. Filter, summarize, or truncate API responses before
|
|
677
|
+
returning. Return only what the agent needs to formulate a spoken answer.
|
|
678
|
+
- **Forgetting sandbox constraints** — Agent code runs in a Deno Worker with
|
|
679
|
+
_all permissions disabled_ (no net, no fs, no env). Use `fetch` (proxied
|
|
680
|
+
through the host) for HTTP. Use `ctx.env` for secrets. `Deno.readFile`,
|
|
681
|
+
`Deno.env.get`, and direct network access will fail silently or throw.
|
|
682
|
+
- **Ignoring `ctx.abortSignal`** — When the user interrupts, in-flight tool
|
|
683
|
+
calls are cancelled via `ctx.abortSignal`. Long-running tools (polling,
|
|
684
|
+
multi-step fetches) should check `ctx.abortSignal.aborted` or pass the signal
|
|
685
|
+
to `fetch`.
|
|
686
|
+
- **Hardcoding secrets** — Never put API keys in `agent.ts`. Use
|
|
687
|
+
`aai env add MY_KEY` to store them on the server, then access via
|
|
688
|
+
`ctx.env.MY_KEY`.
|
|
689
|
+
- **Telling the agent to be verbose** — Voice responses should be 1-3 sentences.
|
|
690
|
+
If your `instructions` say "provide detailed explanations", the agent will
|
|
691
|
+
monologue. Instruct it to be brief and let the user ask follow-ups.
|
|
692
|
+
- **Not declaring `env`** — If your agent needs custom env vars, list them in
|
|
693
|
+
the `env` array so the CLI validates they're set before deploying.
|
|
694
|
+
- **Forgetting SSRF restrictions on `fetch`** — The host validates all proxied
|
|
695
|
+
fetch URLs. Requests to private/internal IP addresses (localhost, 10.x,
|
|
696
|
+
192.168.x, etc.) are blocked.
|
|
697
|
+
|
|
698
|
+
## Troubleshooting
|
|
699
|
+
|
|
700
|
+
- **"no agent found"** — Ensure `agent.ts` exists in the current directory
|
|
701
|
+
- **"bundle failed"** — TypeScript syntax error — check imports, brackets
|
|
702
|
+
- **"No .aai/project.json found"** — Run `aai deploy` first before using
|
|
703
|
+
`aai env`
|
|
704
|
+
- **Tool returns `undefined`** — Make sure `execute` returns a value. Even
|
|
705
|
+
`return { ok: true }` is better than an implicit void return.
|
|
706
|
+
- **Agent doesn't use a tool** — Check `description` is clear about when to use
|
|
707
|
+
it. The LLM relies on the description to decide. Also check `activeTools`
|
|
708
|
+
isn't filtering it out.
|
|
709
|
+
- **KV reads return `null`** — Keys are scoped per agent deployment. A
|
|
710
|
+
redeployment with a new slug creates a fresh KV namespace.
|