@alexkroman1/aai-cli 0.9.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/dist/_build-p1HHkdon.mjs +132 -0
- package/dist/_discover-BzlCDVZ6.mjs +161 -0
- package/dist/_init-l_uoyFCN.mjs +82 -0
- package/dist/_link-BGXGFYWa.mjs +47 -0
- package/dist/_server-common-qLA1QU2C.mjs +36 -0
- package/dist/_ui-kJIua5L9.mjs +44 -0
- package/dist/cli.mjs +318 -0
- package/dist/deploy-KyNJaoP5.mjs +86 -0
- package/dist/dev-DBFvKyzk.mjs +39 -0
- package/dist/init-BWG5OrQa.mjs +65 -0
- package/dist/rag-BnCMnccf.mjs +173 -0
- package/dist/secret-CzeHIGzE.mjs +50 -0
- package/dist/start-C1qkhU4O.mjs +23 -0
- package/package.json +39 -0
- package/templates/_shared/.env.example +5 -0
- package/templates/_shared/CLAUDE.md +1051 -0
- package/templates/_shared/biome.json +32 -0
- package/templates/_shared/global.d.ts +1 -0
- package/templates/_shared/index.html +16 -0
- package/templates/_shared/package.json +23 -0
- package/templates/_shared/tsconfig.json +15 -0
- package/templates/code-interpreter/agent.ts +27 -0
- package/templates/code-interpreter/client.tsx +3 -0
- package/templates/css.d.ts +1 -0
- package/templates/dispatch-center/agent.ts +1227 -0
- package/templates/dispatch-center/client.tsx +505 -0
- package/templates/embedded-assets/agent.ts +48 -0
- package/templates/embedded-assets/client.tsx +3 -0
- package/templates/embedded-assets/knowledge.json +20 -0
- package/templates/health-assistant/agent.ts +160 -0
- package/templates/health-assistant/client.tsx +3 -0
- package/templates/infocom-adventure/agent.ts +164 -0
- package/templates/infocom-adventure/client.tsx +300 -0
- package/templates/math-buddy/agent.ts +21 -0
- package/templates/math-buddy/client.tsx +3 -0
- package/templates/memory-agent/agent.ts +20 -0
- package/templates/memory-agent/client.tsx +3 -0
- package/templates/night-owl/agent.ts +98 -0
- package/templates/night-owl/client.tsx +12 -0
- package/templates/personal-finance/agent.ts +26 -0
- package/templates/personal-finance/client.tsx +3 -0
- package/templates/pizza-ordering/agent.ts +218 -0
- package/templates/pizza-ordering/client.tsx +264 -0
- package/templates/simple/agent.ts +6 -0
- package/templates/simple/client.tsx +3 -0
- package/templates/smart-research/agent.ts +164 -0
- package/templates/smart-research/client.tsx +3 -0
- package/templates/solo-rpg/agent.ts +1244 -0
- package/templates/solo-rpg/client.tsx +698 -0
- package/templates/support/README.md +62 -0
- package/templates/support/agent.ts +19 -0
- package/templates/support/client.tsx +3 -0
- package/templates/travel-concierge/agent.ts +29 -0
- package/templates/travel-concierge/client.tsx +3 -0
- package/templates/tsconfig.json +1 -0
- package/templates/web-researcher/agent.ts +17 -0
- package/templates/web-researcher/client.tsx +3 -0
|
@@ -0,0 +1,1051 @@
|
|
|
1
|
+
# aai Voice Agent Project
|
|
2
|
+
|
|
3
|
+
You are helping a user build a voice agent using the **aai** framework.
|
|
4
|
+
|
|
5
|
+
## Workflow
|
|
6
|
+
|
|
7
|
+
1. **Understand** — Restate what the user wants to build. If the request is
|
|
8
|
+
vague, ask a clarifying question before writing code.
|
|
9
|
+
2. **Check existing work** — Look for a template or built-in tool that already
|
|
10
|
+
does what the user needs before writing custom code.
|
|
11
|
+
3. **Start minimal** — Scaffold from the closest template, then layer on
|
|
12
|
+
customizations. Don't over-engineer the first version.
|
|
13
|
+
4. **Verify** — After every change, run `aai build` to validate the bundle and
|
|
14
|
+
catch errors. Fix all errors before presenting work to the user.
|
|
15
|
+
5. **Iterate** — Make small, focused changes. Verify each change works before
|
|
16
|
+
moving on.
|
|
17
|
+
|
|
18
|
+
## Key rules
|
|
19
|
+
|
|
20
|
+
- Every agent lives in `agent.ts` and exports a default `defineAgent()` call
|
|
21
|
+
- Custom UI goes in `client.tsx` alongside `agent.ts`
|
|
22
|
+
- Optimize `instructions` for spoken conversation — short sentences, no visual
|
|
23
|
+
formatting, no exclamation points
|
|
24
|
+
- Never hardcode secrets — use `aai secret put` and access via `ctx.env`
|
|
25
|
+
- Tool `execute` return values go into LLM context — filter and truncate large
|
|
26
|
+
API responses
|
|
27
|
+
- Agent code runs in a sandboxed worker — use `fetch` (proxied) for HTTP,
|
|
28
|
+
`ctx.env` for secrets
|
|
29
|
+
|
|
30
|
+
## CLI commands
|
|
31
|
+
|
|
32
|
+
```sh
|
|
33
|
+
aai init # Scaffold a new agent (uses simple template)
|
|
34
|
+
aai init -t <template> # Scaffold from a specific template
|
|
35
|
+
aai dev # Start local dev server
|
|
36
|
+
aai build # Bundle and validate (no server or deploy)
|
|
37
|
+
aai deploy # Bundle and deploy to production
|
|
38
|
+
aai deploy -y # Deploy without prompts
|
|
39
|
+
aai deploy --dry-run # Validate and bundle without deploying
|
|
40
|
+
aai secret put <NAME> # Set a secret on the server (prompts for value)
|
|
41
|
+
aai secret delete <NAME> # Remove a secret
|
|
42
|
+
aai secret list # List secret names
|
|
43
|
+
aai secret pull # Pull secret names into .env for local dev
|
|
44
|
+
aai rag <url> # Ingest a site's llms-full.txt into the vector store
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
## Templates
|
|
48
|
+
|
|
49
|
+
Before writing an agent from scratch, choose the closest template and scaffold
|
|
50
|
+
with `aai init -t <template_name>`.
|
|
51
|
+
|
|
52
|
+
| Template | Description |
|
|
53
|
+
| ------------------- | ---------------------------------------------------------------------------------- |
|
|
54
|
+
| `simple` | Minimal starter with web_search, visit_webpage, fetch_json, run_code. **Default.** |
|
|
55
|
+
| `web-researcher` | Research assistant — web search + page visits for detailed answers |
|
|
56
|
+
| `smart-research` | Phase-based research (gather → analyze → respond) with dynamic tool filtering |
|
|
57
|
+
| `memory-agent` | Persistent KV storage — remembers facts and preferences across conversations |
|
|
58
|
+
| `code-interpreter` | Writes and runs JavaScript for math, calculations, data processing |
|
|
59
|
+
| `math-buddy` | Calculations, unit conversions, dice rolls via run_code |
|
|
60
|
+
| `health-assistant` | Medication lookup, drug interactions, BMI, symptom guidance |
|
|
61
|
+
| `personal-finance` | Currency conversion, crypto prices, loan calculations, savings projections |
|
|
62
|
+
| `travel-concierge` | Trip planning, weather, flights, hotels, currency conversion |
|
|
63
|
+
| `night-owl` | Movie/music/book recs by mood, sleep calculator. **Has custom UI.** |
|
|
64
|
+
| `pizza-ordering` | Pizza order-taker with dynamic cart sidebar. **Has custom UI.** |
|
|
65
|
+
| `dispatch-center` | 911 dispatch with incident triage and resource assignment. **Has custom UI.** |
|
|
66
|
+
| `infocom-adventure` | Zork-style text adventure with state, puzzles, inventory. **Has custom UI.** |
|
|
67
|
+
| `solo-rpg` | Solo dark-fantasy RPG with dice, oaths, combat, save/load. **Has custom UI.** |
|
|
68
|
+
| `embedded-assets` | FAQ bot using embedded JSON knowledge (no web search) |
|
|
69
|
+
| `support` | RAG-powered support agent using vector_search (AssemblyAI docs example) |
|
|
70
|
+
|
|
71
|
+
## Minimal agent
|
|
72
|
+
|
|
73
|
+
Every agent lives in `agent.ts` and exports a default `defineAgent()` call:
|
|
74
|
+
|
|
75
|
+
```ts
|
|
76
|
+
import { defineAgent } from "aai";
|
|
77
|
+
|
|
78
|
+
export default defineAgent({
|
|
79
|
+
name: "My Agent",
|
|
80
|
+
instructions: "You are a helpful assistant that...",
|
|
81
|
+
greeting: "Hey there. What can I help you with?",
|
|
82
|
+
});
|
|
83
|
+
```
|
|
84
|
+
|
|
85
|
+
### Imports
|
|
86
|
+
|
|
87
|
+
```ts
|
|
88
|
+
import { defineAgent, tool } from "aai"; // defineAgent + helpers
|
|
89
|
+
import type { BeforeStepResult, BuiltinTool, HookContext, StepInfo, ToolContext } from "aai";
|
|
90
|
+
import { z } from "zod"; // Tools with typed params (included in package.json)
|
|
91
|
+
```
|
|
92
|
+
|
|
93
|
+
## Agent configuration
|
|
94
|
+
|
|
95
|
+
```ts
|
|
96
|
+
defineAgent({
|
|
97
|
+
// Core
|
|
98
|
+
name: string; // Required: display name
|
|
99
|
+
instructions?: string; // System prompt (default: general voice assistant)
|
|
100
|
+
greeting?: string; // Spoken on connect (default: "Hey, how can I help you?")
|
|
101
|
+
// Speech
|
|
102
|
+
sttPrompt?: string; // STT guidance for jargon, names, acronyms
|
|
103
|
+
|
|
104
|
+
// Tools
|
|
105
|
+
builtinTools?: BuiltinTool[];
|
|
106
|
+
tools?: Record<string, ToolDef>;
|
|
107
|
+
toolChoice?: ToolChoice; // "auto" | "required" | "none" | { type: "tool", toolName }
|
|
108
|
+
activeTools?: string[]; // Default active tools per turn (subset of all tools)
|
|
109
|
+
maxSteps?: number | ((ctx: HookContext) => number);
|
|
110
|
+
|
|
111
|
+
// State
|
|
112
|
+
state?: () => S; // Factory for per-session state
|
|
113
|
+
|
|
114
|
+
// Lifecycle hooks
|
|
115
|
+
onConnect?: (ctx: HookContext) => void | Promise<void>;
|
|
116
|
+
onDisconnect?: (ctx: HookContext) => void | Promise<void>;
|
|
117
|
+
onError?: (error: Error, ctx?: HookContext) => void;
|
|
118
|
+
onTurn?: (text: string, ctx: HookContext) => void | Promise<void>;
|
|
119
|
+
onStep?: (step: StepInfo, ctx: HookContext) => void | Promise<void>;
|
|
120
|
+
onBeforeStep?: (stepNumber: number, ctx: HookContext) =>
|
|
121
|
+
BeforeStepResult | Promise<BeforeStepResult>;
|
|
122
|
+
});
|
|
123
|
+
```
|
|
124
|
+
|
|
125
|
+
Use `sttPrompt` for domain-specific vocabulary:
|
|
126
|
+
|
|
127
|
+
```ts
|
|
128
|
+
export default defineAgent({
|
|
129
|
+
name: "Tech Support",
|
|
130
|
+
sttPrompt: "Transcribe technical terms: Kubernetes, gRPC, PostgreSQL",
|
|
131
|
+
});
|
|
132
|
+
```
|
|
133
|
+
|
|
134
|
+
### Writing good `instructions`
|
|
135
|
+
|
|
136
|
+
Optimize for spoken conversation:
|
|
137
|
+
|
|
138
|
+
- Short, punchy sentences — optimize for speech, not text
|
|
139
|
+
- Never mention "search results" or "sources" — speak as if knowledge is your
|
|
140
|
+
own
|
|
141
|
+
- No visual formatting ("bullet point", "bold") — use "First", "Next", "Finally"
|
|
142
|
+
- Lead with the most important information
|
|
143
|
+
- Be concise and confident — no hedging ("It seems that", "I believe")
|
|
144
|
+
- No exclamation points — calm, conversational tone
|
|
145
|
+
- Define personality, tone, and specialty
|
|
146
|
+
- Include when and how to use each tool
|
|
147
|
+
|
|
148
|
+
### Environment variables
|
|
149
|
+
|
|
150
|
+
Secrets are managed on the server via the CLI, like `vercel env`. They are
|
|
151
|
+
injected into agent workers at runtime and available as `ctx.env`. Secrets are
|
|
152
|
+
**never** embedded in the bundled code.
|
|
153
|
+
|
|
154
|
+
```sh
|
|
155
|
+
# Set secrets on the server (prompts for value)
|
|
156
|
+
aai secret put ASSEMBLYAI_API_KEY
|
|
157
|
+
aai secret put MY_API_KEY
|
|
158
|
+
|
|
159
|
+
# List what's set
|
|
160
|
+
aai secret list
|
|
161
|
+
|
|
162
|
+
# Pull secret names into .env for local dev reference
|
|
163
|
+
aai secret pull
|
|
164
|
+
|
|
165
|
+
# Remove a secret
|
|
166
|
+
aai secret delete MY_API_KEY
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
Access secrets in tool code via `ctx.env`:
|
|
170
|
+
|
|
171
|
+
```ts
|
|
172
|
+
import { defineAgent } from "aai";
|
|
173
|
+
import { z } from "zod";
|
|
174
|
+
|
|
175
|
+
export default defineAgent({
|
|
176
|
+
name: "API Agent",
|
|
177
|
+
tools: {
|
|
178
|
+
call_api: {
|
|
179
|
+
description: "Call an external API",
|
|
180
|
+
parameters: z.object({ query: z.string() }),
|
|
181
|
+
execute: async (args, ctx) => {
|
|
182
|
+
const res = await fetch(`https://api.example.com?q=${args.query}`, {
|
|
183
|
+
headers: { Authorization: `Bearer ${ctx.env.MY_API_KEY}` },
|
|
184
|
+
});
|
|
185
|
+
return res.json();
|
|
186
|
+
},
|
|
187
|
+
},
|
|
188
|
+
},
|
|
189
|
+
});
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
## Tools
|
|
193
|
+
|
|
194
|
+
### Custom tools
|
|
195
|
+
|
|
196
|
+
Define tools as plain objects in the `tools` record. The `parameters` field
|
|
197
|
+
takes a Zod schema for type-safe argument inference:
|
|
198
|
+
|
|
199
|
+
```ts
|
|
200
|
+
import { defineAgent, tool } from "aai";
|
|
201
|
+
import { z } from "zod";
|
|
202
|
+
|
|
203
|
+
export default defineAgent({
|
|
204
|
+
name: "Weather Agent",
|
|
205
|
+
tools: {
|
|
206
|
+
get_weather: tool({
|
|
207
|
+
description: "Get current weather for a city",
|
|
208
|
+
parameters: z.object({
|
|
209
|
+
city: z.string().describe("City name"),
|
|
210
|
+
}),
|
|
211
|
+
execute: async ({ city }, ctx) => {
|
|
212
|
+
// city is typed as string (inferred from Zod schema)
|
|
213
|
+
const data = await fetch(
|
|
214
|
+
`https://api.example.com/weather?q=${city}`,
|
|
215
|
+
);
|
|
216
|
+
return data.json();
|
|
217
|
+
},
|
|
218
|
+
}),
|
|
219
|
+
|
|
220
|
+
// No-parameter tools — omit `parameters` and `tool()` wrapper
|
|
221
|
+
list_items: {
|
|
222
|
+
description: "List all items",
|
|
223
|
+
execute: () => items,
|
|
224
|
+
},
|
|
225
|
+
},
|
|
226
|
+
});
|
|
227
|
+
```
|
|
228
|
+
|
|
229
|
+
**Important:** Wrap tool definitions in `tool()` to get typed `args` inferred
|
|
230
|
+
from the Zod `parameters` schema. Without `tool()`, args are untyped.
|
|
231
|
+
|
|
232
|
+
Zod schema patterns:
|
|
233
|
+
|
|
234
|
+
```ts
|
|
235
|
+
parameters: z.object({
|
|
236
|
+
query: z.string().describe("Search query"),
|
|
237
|
+
category: z.enum(["a", "b", "c"]),
|
|
238
|
+
count: z.number().describe("How many"),
|
|
239
|
+
label: z.string().describe("Optional label").optional(),
|
|
240
|
+
}),
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
### Built-in tools
|
|
244
|
+
|
|
245
|
+
Enable via `builtinTools`.
|
|
246
|
+
|
|
247
|
+
| Tool | Description | Params |
|
|
248
|
+
| --------------- | ---------------------------------------------- | ----------------------------------- |
|
|
249
|
+
| `web_search` | Search the web (Brave Search) | `query`, `max_results?` (default 5) |
|
|
250
|
+
| `visit_webpage` | Fetch URL → Markdown | `url` |
|
|
251
|
+
| `fetch_json` | HTTP GET a JSON API | `url`, `headers?` |
|
|
252
|
+
| `run_code` | Execute JS in sandbox (no net/fs, 5s timeout) | `code` |
|
|
253
|
+
| `vector_search` | Search the agent's RAG knowledge base | `query`, `topK?` (default 5) |
|
|
254
|
+
| `memory` | Persistent KV memory (4 tools, see below) | — |
|
|
255
|
+
|
|
256
|
+
The agentic loop runs up to `maxSteps` iterations (default 5) and stops when the
|
|
257
|
+
LLM produces a text response.
|
|
258
|
+
|
|
259
|
+
### Tool context
|
|
260
|
+
|
|
261
|
+
Every `execute` function and lifecycle hook receives a context object:
|
|
262
|
+
|
|
263
|
+
```ts
|
|
264
|
+
ctx.env; // Record<string, string> — secrets from `aai secret put`
|
|
265
|
+
ctx.state; // per-session state
|
|
266
|
+
ctx.kv; // persistent KV store
|
|
267
|
+
ctx.vector; // VectorStore — vector store for RAG (tools only)
|
|
268
|
+
ctx.messages; // readonly Message[] — conversation history (tools only)
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
Hooks get `HookContext` (same but without `messages`).
|
|
272
|
+
|
|
273
|
+
**Timeouts:** Tool execution times out after **30 seconds**. Lifecycle hooks
|
|
274
|
+
(`onConnect`, `onTurn`, etc.) time out after **5 seconds**.
|
|
275
|
+
|
|
276
|
+
### Fetching external APIs
|
|
277
|
+
|
|
278
|
+
Use `fetch` directly in tool execute functions:
|
|
279
|
+
|
|
280
|
+
```ts
|
|
281
|
+
execute: async (args, ctx) => {
|
|
282
|
+
const resp = await fetch(url, {
|
|
283
|
+
headers: { Authorization: `Bearer ${ctx.env.API_KEY}` },
|
|
284
|
+
});
|
|
285
|
+
if (!resp.ok) return { error: `${resp.status} ${resp.statusText}` };
|
|
286
|
+
return resp.json();
|
|
287
|
+
},
|
|
288
|
+
```
|
|
289
|
+
|
|
290
|
+
`fetch` is proxied through the host process (the worker has no direct network
|
|
291
|
+
access). All URLs are validated against SSRF rules — only public addresses are
|
|
292
|
+
allowed.
|
|
293
|
+
|
|
294
|
+
## State and storage
|
|
295
|
+
|
|
296
|
+
### Per-session state
|
|
297
|
+
|
|
298
|
+
For data that lasts only one connection (games, workflows, multi-step
|
|
299
|
+
processes). Fresh state is created per session and cleaned up on disconnect:
|
|
300
|
+
|
|
301
|
+
```ts
|
|
302
|
+
export default defineAgent({
|
|
303
|
+
state: () => ({ score: 0, question: 0 }),
|
|
304
|
+
tools: {
|
|
305
|
+
answer: {
|
|
306
|
+
description: "Submit an answer",
|
|
307
|
+
parameters: z.object({ answer: z.string() }),
|
|
308
|
+
execute: (args, ctx) => {
|
|
309
|
+
const state = ctx.state as { score: number; question: number };
|
|
310
|
+
state.question++;
|
|
311
|
+
return state;
|
|
312
|
+
},
|
|
313
|
+
},
|
|
314
|
+
},
|
|
315
|
+
});
|
|
316
|
+
```
|
|
317
|
+
|
|
318
|
+
### Persisting state across reconnects
|
|
319
|
+
|
|
320
|
+
Use the KV store to auto-save and auto-load state:
|
|
321
|
+
|
|
322
|
+
```ts
|
|
323
|
+
export default defineAgent({
|
|
324
|
+
state: () => ({ score: 0, initialized: false }),
|
|
325
|
+
onConnect: async (ctx) => {
|
|
326
|
+
const saved = await ctx.kv.get("save:game");
|
|
327
|
+
if (saved) Object.assign(ctx.state, saved);
|
|
328
|
+
},
|
|
329
|
+
onTurn: async (_text, ctx) => {
|
|
330
|
+
await ctx.kv.set("save:game", ctx.state);
|
|
331
|
+
},
|
|
332
|
+
});
|
|
333
|
+
```
|
|
334
|
+
|
|
335
|
+
This works for games, workflows,
|
|
336
|
+
or any agent where users expect to resume where they left off.
|
|
337
|
+
|
|
338
|
+
### Persistent storage (KV)
|
|
339
|
+
|
|
340
|
+
`ctx.kv` is a persistent key-value store scoped per agent. Values are
|
|
341
|
+
auto-serialized as JSON.
|
|
342
|
+
|
|
343
|
+
```ts
|
|
344
|
+
await ctx.kv.set("user:123", { name: "Alice" }); // save
|
|
345
|
+
await ctx.kv.set("temp:x", value, { expireIn: 60_000 }); // save with TTL (ms)
|
|
346
|
+
const user = await ctx.kv.get<User>("user:123"); // read (or null)
|
|
347
|
+
const notes = await ctx.kv.list("note:", { limit: 10, reverse: true }); // list by prefix
|
|
348
|
+
const allKeys = await ctx.kv.keys(); // all keys
|
|
349
|
+
const userKeys = await ctx.kv.keys("user:*"); // keys matching glob pattern
|
|
350
|
+
await ctx.kv.delete("user:123"); // delete
|
|
351
|
+
```
|
|
352
|
+
|
|
353
|
+
Keys are strings; use colon-separated prefixes (`"user:123"`). Max value: 64 KB.
|
|
354
|
+
|
|
355
|
+
`kv.list()` returns `KvEntry[]` where each entry has
|
|
356
|
+
`{ key: string, value: T }`.
|
|
357
|
+
|
|
358
|
+
### Memory tools (pre-built KV tools)
|
|
359
|
+
|
|
360
|
+
Add `"memory"` to `builtinTools` to give the agent four persistent KV tools:
|
|
361
|
+
`save_memory`, `recall_memory`, `list_memories`, and `forget_memory`.
|
|
362
|
+
|
|
363
|
+
```ts
|
|
364
|
+
import { defineAgent } from "aai";
|
|
365
|
+
|
|
366
|
+
export default defineAgent({
|
|
367
|
+
name: "My Agent",
|
|
368
|
+
builtinTools: ["memory"],
|
|
369
|
+
});
|
|
370
|
+
```
|
|
371
|
+
|
|
372
|
+
Keys use colon-separated prefixes (`"user:name"`, `"preference:color"`).
|
|
373
|
+
|
|
374
|
+
## Advanced patterns
|
|
375
|
+
|
|
376
|
+
### Step hooks
|
|
377
|
+
|
|
378
|
+
`onStep` — called after each LLM step (logging, analytics):
|
|
379
|
+
|
|
380
|
+
```ts
|
|
381
|
+
onStep: (step, ctx) => {
|
|
382
|
+
console.log(`Step ${step.stepNumber}: ${step.toolCalls.length} tool calls`);
|
|
383
|
+
},
|
|
384
|
+
```
|
|
385
|
+
|
|
386
|
+
`onBeforeStep` — return `{ activeTools: [...] }` to filter tools per step:
|
|
387
|
+
|
|
388
|
+
```ts
|
|
389
|
+
state: () => ({ phase: "gather" }),
|
|
390
|
+
onBeforeStep: (stepNumber, ctx) => {
|
|
391
|
+
const state = ctx.state as { phase: string };
|
|
392
|
+
if (state.phase === "gather") {
|
|
393
|
+
return { activeTools: ["search", "lookup"] };
|
|
394
|
+
}
|
|
395
|
+
return { activeTools: ["summarize"] };
|
|
396
|
+
},
|
|
397
|
+
```
|
|
398
|
+
|
|
399
|
+
### Tool choice
|
|
400
|
+
|
|
401
|
+
Control when the LLM uses tools:
|
|
402
|
+
|
|
403
|
+
```ts
|
|
404
|
+
toolChoice: "auto", // Default — LLM decides when to use tools
|
|
405
|
+
toolChoice: "required", // Force a tool call every step (useful for research pipelines)
|
|
406
|
+
toolChoice: "none", // Disable all tool use
|
|
407
|
+
toolChoice: { type: "tool", toolName: "search" }, // Force a specific tool
|
|
408
|
+
```
|
|
409
|
+
|
|
410
|
+
### Phase-based tool filtering
|
|
411
|
+
|
|
412
|
+
Combine `state`, `onBeforeStep`, and `activeTools` for multi-phase workflows:
|
|
413
|
+
|
|
414
|
+
```ts
|
|
415
|
+
state: () => ({ phase: "gather" as "gather" | "analyze" | "respond" }),
|
|
416
|
+
onBeforeStep: (_step, ctx) => {
|
|
417
|
+
const state = ctx.state as { phase: string };
|
|
418
|
+
if (state.phase === "gather") return { activeTools: ["web_search", "advance"] };
|
|
419
|
+
if (state.phase === "analyze") return { activeTools: ["summarize", "advance"] };
|
|
420
|
+
return { activeTools: [] }; // respond phase — LLM speaks freely
|
|
421
|
+
},
|
|
422
|
+
tools: {
|
|
423
|
+
advance: {
|
|
424
|
+
description: "Move to the next phase",
|
|
425
|
+
execute: (_args, ctx) => {
|
|
426
|
+
const state = ctx.state as { phase: string };
|
|
427
|
+
if (state.phase === "gather") state.phase = "analyze";
|
|
428
|
+
else if (state.phase === "analyze") state.phase = "respond";
|
|
429
|
+
return { phase: state.phase };
|
|
430
|
+
},
|
|
431
|
+
},
|
|
432
|
+
},
|
|
433
|
+
```
|
|
434
|
+
|
|
435
|
+
### Static `activeTools`
|
|
436
|
+
|
|
437
|
+
Restrict which tools the LLM can use by default, without writing a hook:
|
|
438
|
+
|
|
439
|
+
```ts
|
|
440
|
+
export default defineAgent({
|
|
441
|
+
builtinTools: ["web_search", "visit_webpage", "run_code"],
|
|
442
|
+
tools: { summarize: {/* ... */} },
|
|
443
|
+
activeTools: ["web_search", "summarize"], // Only these two are available
|
|
444
|
+
});
|
|
445
|
+
```
|
|
446
|
+
|
|
447
|
+
Use `onBeforeStep` to override `activeTools` dynamically per step.
|
|
448
|
+
|
|
449
|
+
### Dynamic `maxSteps`
|
|
450
|
+
|
|
451
|
+
```ts
|
|
452
|
+
maxSteps: (ctx) => {
|
|
453
|
+
const state = ctx.state as { complexity: string };
|
|
454
|
+
return state.complexity === "complex" ? 10 : 5;
|
|
455
|
+
},
|
|
456
|
+
```
|
|
457
|
+
|
|
458
|
+
### Conversation history in tools
|
|
459
|
+
|
|
460
|
+
```ts
|
|
461
|
+
execute: (args, ctx) => {
|
|
462
|
+
const userMessages = ctx.messages.filter(m => m.role === "user");
|
|
463
|
+
return { turns: userMessages.length };
|
|
464
|
+
},
|
|
465
|
+
```
|
|
466
|
+
|
|
467
|
+
### Embedded knowledge
|
|
468
|
+
|
|
469
|
+
```ts
|
|
470
|
+
import knowledge from "./knowledge.json" with { type: "json" };
|
|
471
|
+
|
|
472
|
+
export default defineAgent({
|
|
473
|
+
tools: {
|
|
474
|
+
search_faq: {
|
|
475
|
+
description: "Search the knowledge base",
|
|
476
|
+
parameters: z.object({ query: z.string() }),
|
|
477
|
+
execute: (args) =>
|
|
478
|
+
knowledge.faqs.filter((f: { question: string }) =>
|
|
479
|
+
f.question.toLowerCase().includes(args.query.toLowerCase())
|
|
480
|
+
),
|
|
481
|
+
},
|
|
482
|
+
},
|
|
483
|
+
});
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
### Using npm packages
|
|
487
|
+
|
|
488
|
+
Add packages to `package.json` dependencies:
|
|
489
|
+
|
|
490
|
+
```sh
|
|
491
|
+
npm install some-package
|
|
492
|
+
```
|
|
493
|
+
|
|
494
|
+
## Custom UI (`client.tsx`)
|
|
495
|
+
|
|
496
|
+
Add `client.tsx` alongside `agent.ts`. Define a Preact component and call
|
|
497
|
+
`mount()` to render it. Use JSX syntax:
|
|
498
|
+
|
|
499
|
+
```tsx
|
|
500
|
+
import "aai-ui/styles.css";
|
|
501
|
+
import { mount, useSession } from "aai-ui";
|
|
502
|
+
|
|
503
|
+
function App() {
|
|
504
|
+
const { session, started, running, start, toggle, reset } = useSession();
|
|
505
|
+
const msgs = session.messages.value;
|
|
506
|
+
const tx = session.userUtterance.value;
|
|
507
|
+
return (
|
|
508
|
+
<div>
|
|
509
|
+
{msgs.map((m, i) => <p key={i}>{m.text}</p>)}
|
|
510
|
+
{tx !== null && <p>{tx || "..."}</p>}
|
|
511
|
+
{!started.value ? <button onClick={start}>Start</button> : (
|
|
512
|
+
<>
|
|
513
|
+
<button onClick={toggle}>{running.value ? "Stop" : "Resume"}</button>
|
|
514
|
+
<button onClick={reset}>Reset</button>
|
|
515
|
+
</>
|
|
516
|
+
)}
|
|
517
|
+
</div>
|
|
518
|
+
);
|
|
519
|
+
}
|
|
520
|
+
|
|
521
|
+
mount(App);
|
|
522
|
+
```
|
|
523
|
+
|
|
524
|
+
**Rules:**
|
|
525
|
+
|
|
526
|
+
- Always import `"aai-ui/styles.css"` at the top — without it, default styles
|
|
527
|
+
won't load
|
|
528
|
+
- Call `mount(YourComponent)` at the end of the file
|
|
529
|
+
- Use `.tsx` file extension for JSX syntax
|
|
530
|
+
- Import hooks from `preact/hooks` (`useEffect`, `useRef`, `useState`, etc.)
|
|
531
|
+
- Style with `style={{ color: "red" }}` or inject `<style>` for selectors,
|
|
532
|
+
keyframes, media queries
|
|
533
|
+
|
|
534
|
+
### `mount()` options
|
|
535
|
+
|
|
536
|
+
```ts
|
|
537
|
+
mount(App, {
|
|
538
|
+
target: "#app", // CSS selector or DOM element (default: "#app")
|
|
539
|
+
platformUrl: "...", // Server URL (auto-derived from location.href)
|
|
540
|
+
title: "My Agent", // Shown in header and start screen
|
|
541
|
+
theme: { // CSS custom property overrides
|
|
542
|
+
bg: "#101010", // Background color
|
|
543
|
+
primary: "#fab283", // Accent color
|
|
544
|
+
text: "#ffffff", // Text color
|
|
545
|
+
surface: "#1a1a1a", // Card/surface color
|
|
546
|
+
border: "#333333", // Border color
|
|
547
|
+
},
|
|
548
|
+
});
|
|
549
|
+
```
|
|
550
|
+
|
|
551
|
+
`mount()` returns a `MountHandle` with `session`, `signals`, and `dispose()`.
|
|
552
|
+
|
|
553
|
+
### Built-in components
|
|
554
|
+
|
|
555
|
+
Import from `aai-ui`:
|
|
556
|
+
|
|
557
|
+
**Layout components:**
|
|
558
|
+
|
|
559
|
+
| Component | Description |
|
|
560
|
+
| --------------- | ---------------------------------------------------- |
|
|
561
|
+
| `App` | Default full UI (StartScreen + ChatView) |
|
|
562
|
+
| `StartScreen` | Centered start card; renders children after start |
|
|
563
|
+
| `ChatView` | Chat interface (header + messages + controls) |
|
|
564
|
+
| `SidebarLayout` | Two-column layout with sidebar + main area |
|
|
565
|
+
| `Controls` | Stop/Resume + New Conversation buttons |
|
|
566
|
+
| `MessageList` | Messages with auto-scroll, tool calls, transcript |
|
|
567
|
+
|
|
568
|
+
`StartScreen` props: `{ children, icon?, title?, subtitle?, buttonText? }`
|
|
569
|
+
`SidebarLayout` props: `{ sidebar, children, width?, side? }`
|
|
570
|
+
|
|
571
|
+
**Atomic components:**
|
|
572
|
+
|
|
573
|
+
| Component | Props | Description |
|
|
574
|
+
| ------------------- | --------------------------------------- | ------------------------------- |
|
|
575
|
+
| `MessageBubble` | `{ message: Message }` | Single message bubble |
|
|
576
|
+
| `Transcript` | `{ userUtterance: Signal<str\|null> }` | Live STT text display |
|
|
577
|
+
| `StateIndicator` | `{ state: Signal<AgentState> }` | Colored dot + state label |
|
|
578
|
+
| `ErrorBanner` | `{ error: Signal<SessionError\|null> }` | Red error box with message |
|
|
579
|
+
| `ThinkingIndicator` | none | Animated dots during processing |
|
|
580
|
+
| `ToolCallBlock` | `{ toolCall: ToolCallInfo }` | Collapsible tool call display |
|
|
581
|
+
|
|
582
|
+
**Hooks:**
|
|
583
|
+
|
|
584
|
+
- `useAutoScroll()` — returns a `RefObject<HTMLDivElement>` to attach to a
|
|
585
|
+
sentinel div. Auto-scrolls when messages or utterances change.
|
|
586
|
+
- `useMountConfig()` — returns the `title` and `theme` passed to `mount()`.
|
|
587
|
+
|
|
588
|
+
**Important:** Components that accept `Signal<T>` props (like `StateIndicator`,
|
|
589
|
+
`Transcript`, `ErrorBanner`) expect the Signal object itself, NOT `.value`. Pass
|
|
590
|
+
`session.state`, not `session.state.value`. Passing `.value` compiles but breaks
|
|
591
|
+
reactivity silently.
|
|
592
|
+
|
|
593
|
+
### Session signals (`useSession()`)
|
|
594
|
+
|
|
595
|
+
`useSession()` returns
|
|
596
|
+
`{ session, started, running, start, toggle, reset, dispose }`. Reactive agent
|
|
597
|
+
data lives on `session` (a `VoiceSession`); UI-only controls are top-level.
|
|
598
|
+
|
|
599
|
+
| Signal / field | Type | Description |
|
|
600
|
+
| ------------------------------ | ---------------------- | --------------------------------------------------------------- |
|
|
601
|
+
| `session.state.value` | `AgentState` | "disconnected", "connecting", "ready", "listening", etc. |
|
|
602
|
+
| `session.messages.value` | `Message[]` | `{ role, text }` objects |
|
|
603
|
+
| `session.toolCalls.value` | `ToolCallInfo[]` | `{ toolCallId, toolName, args, status, result? }` — tool calls |
|
|
604
|
+
| `session.userUtterance.value` | `string \| null` | `null` = not speaking, `""` = speech detected, string = text |
|
|
605
|
+
| `session.agentUtterance.value` | `string \| null` | `null` = not speaking, string = streaming agent response text |
|
|
606
|
+
| `session.error.value` | `SessionError \| null` | `{ code, message }` |
|
|
607
|
+
| `session.disconnected.value` | `object \| null` | `{ intentional: boolean }` when disconnected, `null` otherwise |
|
|
608
|
+
| `started.value` | `boolean` | Whether session has been started |
|
|
609
|
+
| `running.value` | `boolean` | Whether session is active |
|
|
610
|
+
|
|
611
|
+
**Methods:** `start()`, `toggle()`, `reset()`, `dispose()`
|
|
612
|
+
|
|
613
|
+
**Hooks:**
|
|
614
|
+
|
|
615
|
+
| Hook | Description |
|
|
616
|
+
| --------------------------------------------- | -------------------------------------------------------------------------------------------- |
|
|
617
|
+
| `useToolResult((toolName, result, tc) => {})` | Fires once per completed tool call with parsed JSON result. Use for carts, scoreboards, etc. |
|
|
618
|
+
|
|
619
|
+
**Signal semantics for utterances:**
|
|
620
|
+
|
|
621
|
+
- `userUtterance`: `null` = user is not speaking, `""` = speech detected but
|
|
622
|
+
no text yet (show "..."), non-empty string = partial/final transcript
|
|
623
|
+
- `agentUtterance`: `null` = agent is not speaking, non-empty string =
|
|
624
|
+
streaming response text (cleared when final `chat` message arrives)
|
|
625
|
+
- `disconnected`: `null` = connected, `{ intentional: true }` = user
|
|
626
|
+
disconnected, `{ intentional: false }` = unexpected disconnect (show
|
|
627
|
+
reconnect UI)
|
|
628
|
+
|
|
629
|
+
**UI Message type:** `{ role: "user" | "assistant"; content: string }`. Both UI
|
|
630
|
+
and SDK `Message` types use `content`.
|
|
631
|
+
|
|
632
|
+
### Showing tool calls in custom UI
|
|
633
|
+
|
|
634
|
+
```tsx
|
|
635
|
+
import "aai-ui/styles.css";
|
|
636
|
+
import { mount, ToolCallBlock, useSession } from "aai-ui";
|
|
637
|
+
|
|
638
|
+
function App() {
|
|
639
|
+
const { session, started, start } = useSession();
|
|
640
|
+
if (!started.value) return <button onClick={start}>Start</button>;
|
|
641
|
+
|
|
642
|
+
const msgs = session.messages.value;
|
|
643
|
+
const toolCalls = session.toolCalls.value;
|
|
644
|
+
|
|
645
|
+
return (
|
|
646
|
+
<div>
|
|
647
|
+
{msgs.map((m, i) => (
|
|
648
|
+
<div key={i}>
|
|
649
|
+
<p>{m.text}</p>
|
|
650
|
+
{toolCalls
|
|
651
|
+
.filter((tc) => tc.afterMessageIndex === i)
|
|
652
|
+
.map((tc) => <ToolCallBlock key={tc.toolCallId} toolCall={tc} />)}
|
|
653
|
+
</div>
|
|
654
|
+
))}
|
|
655
|
+
</div>
|
|
656
|
+
);
|
|
657
|
+
}
|
|
658
|
+
|
|
659
|
+
mount(App);
|
|
660
|
+
```
|
|
661
|
+
|
|
662
|
+
### Building dynamic UI from tool results
|
|
663
|
+
|
|
664
|
+
Use `useToolResult` to update local state (carts, scoreboards, dashboards)
|
|
665
|
+
whenever a tool completes. It fires exactly once per completed tool call with
|
|
666
|
+
the parsed JSON result, handling deduplication internally.
|
|
667
|
+
|
|
668
|
+
```tsx
|
|
669
|
+
import "aai-ui/styles.css";
|
|
670
|
+
import { useState } from "preact/hooks";
|
|
671
|
+
import { ChatView, SidebarLayout, StartScreen, mount, useToolResult } from "aai-ui";
|
|
672
|
+
|
|
673
|
+
interface CartItem { id: number; name: string; price: number }
|
|
674
|
+
|
|
675
|
+
function ShopAgent() {
|
|
676
|
+
const [cart, setCart] = useState<CartItem[]>([]);
|
|
677
|
+
|
|
678
|
+
useToolResult((toolName, result: any) => {
|
|
679
|
+
switch (toolName) {
|
|
680
|
+
case "add_item":
|
|
681
|
+
setCart((prev) => [...prev, result.item]);
|
|
682
|
+
break;
|
|
683
|
+
case "remove_item":
|
|
684
|
+
setCart((prev) => prev.filter((i) => i.id !== result.removedId));
|
|
685
|
+
break;
|
|
686
|
+
case "clear_cart":
|
|
687
|
+
setCart([]);
|
|
688
|
+
break;
|
|
689
|
+
}
|
|
690
|
+
});
|
|
691
|
+
|
|
692
|
+
const sidebar = (
|
|
693
|
+
<div class="p-4">
|
|
694
|
+
<h3 class="text-aai-text font-bold">Cart ({cart.length})</h3>
|
|
695
|
+
{cart.map((i) => <p key={i.id} class="text-aai-text text-sm">{i.name} — ${i.price}</p>)}
|
|
696
|
+
</div>
|
|
697
|
+
);
|
|
698
|
+
|
|
699
|
+
return (
|
|
700
|
+
<StartScreen title="Shop" buttonText="Start Shopping">
|
|
701
|
+
<SidebarLayout sidebar={sidebar}>
|
|
702
|
+
<ChatView />
|
|
703
|
+
</SidebarLayout>
|
|
704
|
+
</StartScreen>
|
|
705
|
+
);
|
|
706
|
+
}
|
|
707
|
+
|
|
708
|
+
mount(ShopAgent);
|
|
709
|
+
```
|
|
710
|
+
|
|
711
|
+
**Do NOT use `useEffect` + `session.toolCalls.value` to build derived state.**
|
|
712
|
+
That pattern re-processes every tool call on every signal change, causing
|
|
713
|
+
duplicates (e.g. items added to the cart multiple times). `useToolResult`
|
|
714
|
+
handles this correctly.
|
|
715
|
+
|
|
716
|
+
### Reacting to agent state
|
|
717
|
+
|
|
718
|
+
```tsx
|
|
719
|
+
import "aai-ui/styles.css";
|
|
720
|
+
import { useEffect } from "preact/hooks";
|
|
721
|
+
import { mount, StateIndicator, useSession } from "aai-ui";
|
|
722
|
+
|
|
723
|
+
function App() {
|
|
724
|
+
const { session, started, start } = useSession();
|
|
725
|
+
|
|
726
|
+
useEffect(() => {
|
|
727
|
+
// Run side effects when state changes
|
|
728
|
+
if (session.state.value === "speaking") {
|
|
729
|
+
// Agent is speaking — e.g., show animation
|
|
730
|
+
}
|
|
731
|
+
}, [session.state.value]);
|
|
732
|
+
|
|
733
|
+
return (
|
|
734
|
+
<div>
|
|
735
|
+
<StateIndicator />
|
|
736
|
+
{!started.value && <button onClick={start}>Start</button>}
|
|
737
|
+
</div>
|
|
738
|
+
);
|
|
739
|
+
}
|
|
740
|
+
|
|
741
|
+
mount(App);
|
|
742
|
+
```
|
|
743
|
+
|
|
744
|
+
### Styling custom UIs
|
|
745
|
+
|
|
746
|
+
The framework uses **Tailwind CSS v4** (compiled at bundle time). Prefer
|
|
747
|
+
Tailwind classes over inline styles — all design tokens work as classes:
|
|
748
|
+
`bg-aai-surface` not `style={{ background: "var(--color-aai-surface)" }}`,
|
|
749
|
+
`border-t border-aai-border` not `style={{ borderTop: "1px solid var(--)" }}`.
|
|
750
|
+
|
|
751
|
+
Three approaches:
|
|
752
|
+
|
|
753
|
+
1. **Tailwind classes** — `class="flex items-center gap-2 bg-aai-surface"`
|
|
754
|
+
2. **Inline styles** — only for dynamic values (`style={{ width: pixels }}`)
|
|
755
|
+
3. **Injected `<style>` tags** — for keyframes, selectors, media queries:
|
|
756
|
+
|
|
757
|
+
```tsx
|
|
758
|
+
function App() {
|
|
759
|
+
return (
|
|
760
|
+
<>
|
|
761
|
+
<style>
|
|
762
|
+
{`
|
|
763
|
+
@keyframes pulse { 0%, 100% { opacity: 1 } 50% { opacity: 0.5 } }
|
|
764
|
+
.pulse { animation: pulse 2s ease-in-out infinite; }
|
|
765
|
+
@media (max-width: 640px) { .sidebar { display: none; } }
|
|
766
|
+
`}
|
|
767
|
+
</style>
|
|
768
|
+
<div class="pulse">Content</div>
|
|
769
|
+
</>
|
|
770
|
+
);
|
|
771
|
+
}
|
|
772
|
+
```
|
|
773
|
+
|
|
774
|
+
**Design tokens** — available as CSS custom properties and Tailwind classes
|
|
775
|
+
(e.g. `bg-aai-bg`, `text-aai-text-muted`, `rounded-aai`, `font-aai`):
|
|
776
|
+
|
|
777
|
+
| Token | Tailwind class | Default |
|
|
778
|
+
| ---------------------------- | ------------------------- | ------------------------- |
|
|
779
|
+
| `--color-aai-bg` | `bg-aai-bg` | `#101010` |
|
|
780
|
+
| `--color-aai-surface` | `bg-aai-surface` | `#151515` |
|
|
781
|
+
| `--color-aai-surface-faint` | `bg-aai-surface-faint` | `rgba(255,255,255,0.031)` |
|
|
782
|
+
| `--color-aai-surface-hover` | `bg-aai-surface-hover` | `rgba(255,255,255,0.059)` |
|
|
783
|
+
| `--color-aai-border` | `border-aai-border` | `#282828` |
|
|
784
|
+
| `--color-aai-primary` | `text-aai-primary` | `#fab283` |
|
|
785
|
+
| `--color-aai-text` | `text-aai-text` | `rgba(255,255,255,0.936)` |
|
|
786
|
+
| `--color-aai-text-secondary` | `text-aai-text-secondary` | `rgba(255,255,255,0.618)` |
|
|
787
|
+
| `--color-aai-text-muted` | `text-aai-text-muted` | `rgba(255,255,255,0.284)` |
|
|
788
|
+
| `--color-aai-text-dim` | `text-aai-text-dim` | `rgba(255,255,255,0.422)` |
|
|
789
|
+
| `--color-aai-error` | `text-aai-error` | `#e06c75` |
|
|
790
|
+
| `--color-aai-ring` | `ring-aai-ring` | `#56b6c2` |
|
|
791
|
+
| `--color-aai-state-{state}` | `text-aai-state-{state}` | per-state colors |
|
|
792
|
+
| `--radius-aai` | `rounded-aai` | `6px` |
|
|
793
|
+
| `--font-aai` | `font-aai` | Inter, sans-serif |
|
|
794
|
+
| `--font-aai-mono` | `font-aai-mono` | IBM Plex Mono, mono |
|
|
795
|
+
|
|
796
|
+
The 5 core colors (`bg`, `primary`, `text`, `surface`, `border`) can be
|
|
797
|
+
overridden via `mount()` theme options. All other tokens use fixed defaults.
|
|
798
|
+
|
|
799
|
+
### Common UI patterns
|
|
800
|
+
|
|
801
|
+
**Auto-scrolling messages** — use `useAutoScroll` for custom message lists:
|
|
802
|
+
|
|
803
|
+
```tsx
|
|
804
|
+
import { useAutoScroll, useSession } from "aai-ui";
|
|
805
|
+
|
|
806
|
+
function MyChat() {
|
|
807
|
+
const { session } = useSession();
|
|
808
|
+
const bottomRef = useAutoScroll();
|
|
809
|
+
|
|
810
|
+
return (
|
|
811
|
+
<div class="overflow-y-auto">
|
|
812
|
+
{session.messages.value.map((m, i) => <p key={i}>{m.text}</p>)}
|
|
813
|
+
<div ref={bottomRef} />
|
|
814
|
+
</div>
|
|
815
|
+
);
|
|
816
|
+
}
|
|
817
|
+
```
|
|
818
|
+
|
|
819
|
+
Note: `MessageList` and `ChatView` already include auto-scroll. Only use
|
|
820
|
+
`useAutoScroll` when building a fully custom message list.
|
|
821
|
+
|
|
822
|
+
**Reading signal values in render:** Extract `.value` once at the top of the
|
|
823
|
+
component to avoid redundant signal subscriptions:
|
|
824
|
+
|
|
825
|
+
```tsx
|
|
826
|
+
function MyComponent() {
|
|
827
|
+
const { session } = useSession();
|
|
828
|
+
const state = session.state.value;
|
|
829
|
+
const msgs = session.messages.value;
|
|
830
|
+
// Use `state` and `msgs` as plain values throughout the render
|
|
831
|
+
}
|
|
832
|
+
```
|
|
833
|
+
|
|
834
|
+
## Self-hosting with `createServer()`
|
|
835
|
+
|
|
836
|
+
Agents can run anywhere (Node, Docker) without the managed platform:
|
|
837
|
+
|
|
838
|
+
```ts
|
|
839
|
+
import { defineAgent } from "aai";
|
|
840
|
+
import { createServer } from "aai/server";
|
|
841
|
+
|
|
842
|
+
const agent = defineAgent({
|
|
843
|
+
name: "My Agent",
|
|
844
|
+
instructions: "You are a helpful assistant.",
|
|
845
|
+
});
|
|
846
|
+
|
|
847
|
+
const server = createServer({
|
|
848
|
+
agent,
|
|
849
|
+
clientDir: "public", // optional: serve static files
|
|
850
|
+
});
|
|
851
|
+
|
|
852
|
+
await server.listen(3000);
|
|
853
|
+
```
|
|
854
|
+
|
|
855
|
+
Run with `node server.ts` (Node >=22.6 strips types natively) or bundle
|
|
856
|
+
with your preferred tool. The server handles WebSocket connections, STT/TTS,
|
|
857
|
+
and the agentic loop. Set `ASSEMBLYAI_API_KEY` as an environment variable.
|
|
858
|
+
|
|
859
|
+
## Useful free API endpoints
|
|
860
|
+
|
|
861
|
+
These public APIs require no auth and work well in voice agents:
|
|
862
|
+
|
|
863
|
+
```text
|
|
864
|
+
Weather (Open-Meteo):
|
|
865
|
+
Geocode: https://geocoding-api.open-meteo.com/v1/search?name={city}&count=1&language=en
|
|
866
|
+
Forecast: https://api.open-meteo.com/v1/forecast?latitude={lat}&longitude={lon}&daily=temperature_2m_max,temperature_2m_min,precipitation_sum,weathercode&timezone=auto&forecast_days=7
|
|
867
|
+
|
|
868
|
+
Currency (ExchangeRate):
|
|
869
|
+
Rates: https://open.er-api.com/v6/latest/{CODE} → { rates: { USD: 1.0, EUR: 0.85, ... } }
|
|
870
|
+
|
|
871
|
+
Crypto (CoinGecko):
|
|
872
|
+
Price: https://api.coingecko.com/api/v3/simple/price?ids={coin}&vs_currencies={cur}&include_24hr_change=true
|
|
873
|
+
|
|
874
|
+
Drug info (FDA):
|
|
875
|
+
Label: https://api.fda.gov/drug/label.json?search=openfda.generic_name:"{name}"&limit=1
|
|
876
|
+
|
|
877
|
+
Drug interactions (RxNorm):
|
|
878
|
+
RxCUI: https://rxnav.nlm.nih.gov/REST/rxcui.json?name={name}
|
|
879
|
+
Interactions: https://rxnav.nlm.nih.gov/REST/interaction/list.json?rxcuis={id1}+{id2}
|
|
880
|
+
```
|
|
881
|
+
|
|
882
|
+
Use `fetch_json` builtin tool or `fetch` in custom tools to call these.
|
|
883
|
+
|
|
884
|
+
## Custom start screen with `StartScreen`
|
|
885
|
+
|
|
886
|
+
Use `StartScreen` for a branded start card that transitions to `ChatView`:
|
|
887
|
+
|
|
888
|
+
```tsx
|
|
889
|
+
import "aai-ui/styles.css";
|
|
890
|
+
import { ChatView, StartScreen, mount } from "aai-ui";
|
|
891
|
+
|
|
892
|
+
function MyAgent() {
|
|
893
|
+
return (
|
|
894
|
+
<StartScreen icon={<span>🎤</span>} title="My Agent" subtitle="Ask me anything">
|
|
895
|
+
<ChatView />
|
|
896
|
+
</StartScreen>
|
|
897
|
+
);
|
|
898
|
+
}
|
|
899
|
+
|
|
900
|
+
mount(MyAgent);
|
|
901
|
+
```
|
|
902
|
+
|
|
903
|
+
`StartScreen` handles the started/not-started transition automatically. Pass
|
|
904
|
+
`icon`, `title`, `subtitle`, and `buttonText` to customize the card.
|
|
905
|
+
|
|
906
|
+
## Sidebar layout with `SidebarLayout`
|
|
907
|
+
|
|
908
|
+
Use `SidebarLayout` for apps with a persistent side panel (cart, dashboard):
|
|
909
|
+
|
|
910
|
+
```tsx
|
|
911
|
+
import "aai-ui/styles.css";
|
|
912
|
+
import { useState } from "preact/hooks";
|
|
913
|
+
import { ChatView, SidebarLayout, StartScreen, mount, useToolResult } from "aai-ui";
|
|
914
|
+
|
|
915
|
+
function ShopAgent() {
|
|
916
|
+
const [cart, setCart] = useState<{ id: number; name: string }[]>([]);
|
|
917
|
+
|
|
918
|
+
useToolResult((toolName, result: any) => {
|
|
919
|
+
if (toolName === "add_item") setCart((prev) => [...prev, result.item]);
|
|
920
|
+
});
|
|
921
|
+
|
|
922
|
+
const sidebar = (
|
|
923
|
+
<div class="p-4">
|
|
924
|
+
<h3 class="text-aai-text font-bold">Cart ({cart.length})</h3>
|
|
925
|
+
{cart.map((i) => <p key={i.id} class="text-aai-text text-sm">{i.name}</p>)}
|
|
926
|
+
</div>
|
|
927
|
+
);
|
|
928
|
+
|
|
929
|
+
return (
|
|
930
|
+
<StartScreen title="Shop" buttonText="Start Shopping">
|
|
931
|
+
<SidebarLayout sidebar={sidebar}>
|
|
932
|
+
<ChatView />
|
|
933
|
+
</SidebarLayout>
|
|
934
|
+
</StartScreen>
|
|
935
|
+
);
|
|
936
|
+
}
|
|
937
|
+
|
|
938
|
+
mount(ShopAgent);
|
|
939
|
+
```
|
|
940
|
+
|
|
941
|
+
`SidebarLayout` accepts `width` (default `"20rem"`) and `side` (`"left"` or
|
|
942
|
+
`"right"`).
|
|
943
|
+
|
|
944
|
+
## Project structure
|
|
945
|
+
|
|
946
|
+
After scaffolding, your project directory looks like:
|
|
947
|
+
|
|
948
|
+
```text
|
|
949
|
+
my-agent/
|
|
950
|
+
agent.ts # Agent definition
|
|
951
|
+
client.tsx # UI component (calls mount() to render into #app)
|
|
952
|
+
styles.css # Tailwind CSS entry point
|
|
953
|
+
package.json # Dependencies, scripts, and config
|
|
954
|
+
tsconfig.json # TypeScript configuration
|
|
955
|
+
.env.example # Reference for env var names
|
|
956
|
+
.env # Local dev secrets (gitignored)
|
|
957
|
+
.gitignore # Ignores node_modules/, .aai/, .env, etc.
|
|
958
|
+
README.md # Getting started guide
|
|
959
|
+
CLAUDE.md # Agent API reference (always loaded by Claude Code)
|
|
960
|
+
.aai/ # Build output (managed by CLI, gitignored)
|
|
961
|
+
project.json # Deploy target (slug, server URL)
|
|
962
|
+
build/ # Bundle output
|
|
963
|
+
```
|
|
964
|
+
|
|
965
|
+
## Instructions patterns from templates
|
|
966
|
+
|
|
967
|
+
Good instructions tell the LLM what it is, how to behave, and when to use each
|
|
968
|
+
tool. Study these patterns:
|
|
969
|
+
|
|
970
|
+
**Code execution agent** — force tool use for anything computational:
|
|
971
|
+
|
|
972
|
+
```text
|
|
973
|
+
You MUST use the run_code tool for ANY question involving math, counting,
|
|
974
|
+
string manipulation, or data processing. NEVER do mental math or estimate.
|
|
975
|
+
Use console.log() to output intermediate steps.
|
|
976
|
+
```
|
|
977
|
+
|
|
978
|
+
**Research agent** — search before answering:
|
|
979
|
+
|
|
980
|
+
```text
|
|
981
|
+
Search first. Never guess or rely on memory for factual questions.
|
|
982
|
+
Use visit_webpage when search snippets aren't detailed enough.
|
|
983
|
+
For complex questions, search multiple times with different queries.
|
|
984
|
+
```
|
|
985
|
+
|
|
986
|
+
**FAQ/support agent** — stay grounded in knowledge:
|
|
987
|
+
|
|
988
|
+
```text
|
|
989
|
+
Always use vector_search to find relevant documentation before answering.
|
|
990
|
+
Base your answers strictly on the retrieved documentation — don't guess.
|
|
991
|
+
If search results aren't relevant, say the docs don't cover that topic.
|
|
992
|
+
```
|
|
993
|
+
|
|
994
|
+
**API-calling agent** — tell the LLM which endpoints to use:
|
|
995
|
+
|
|
996
|
+
```text
|
|
997
|
+
API endpoints (use fetch_json):
|
|
998
|
+
- Currency rates: https://open.er-api.com/v6/latest/{CODE}
|
|
999
|
+
Returns { rates: { USD: 1.0, EUR: 0.85, ... } }
|
|
1000
|
+
- Weather: https://api.open-meteo.com/v1/forecast?latitude={lat}&longitude={lon}...
|
|
1001
|
+
```
|
|
1002
|
+
|
|
1003
|
+
**Game/interactive agent** — establish world rules and voice style:
|
|
1004
|
+
|
|
1005
|
+
```text
|
|
1006
|
+
You ARE the game. Maintain world state, describe rooms, handle puzzles.
|
|
1007
|
+
Keep descriptions to two to four sentences. No visual formatting.
|
|
1008
|
+
Use directional words naturally: "To the north you see..." not "N: forest"
|
|
1009
|
+
```
|
|
1010
|
+
|
|
1011
|
+
## Common pitfalls
|
|
1012
|
+
|
|
1013
|
+
- **Using `useEffect` to build state from tool calls** — Iterating
|
|
1014
|
+
`session.toolCalls.value` in a `useEffect` re-processes every tool call on
|
|
1015
|
+
every signal change, causing duplicates (e.g. cart items added multiple
|
|
1016
|
+
times). Use the `useToolResult` hook instead — it fires exactly once per
|
|
1017
|
+
completed tool call with proper deduplication.
|
|
1018
|
+
- **Writing `instructions` with visual formatting** — Bullets, bold, numbered
|
|
1019
|
+
lists sound terrible when spoken. Use natural transitions: "First", "Next",
|
|
1020
|
+
"Finally". Write instructions as if you're coaching a human phone operator.
|
|
1021
|
+
- **Returning huge payloads from tools** — Everything a tool returns goes into
|
|
1022
|
+
the LLM context. Filter, summarize, or truncate API responses before
|
|
1023
|
+
returning. Return only what the agent needs to formulate a spoken answer.
|
|
1024
|
+
- **Forgetting sandbox constraints** — Agent code runs in a sandboxed Worker
|
|
1025
|
+
with no direct network or filesystem access. Use `fetch` (proxied through the
|
|
1026
|
+
host) for HTTP. Use `ctx.env` for secrets. Direct network access will fail.
|
|
1027
|
+
- **Hardcoding secrets** — Never put API keys in `agent.ts`. Use
|
|
1028
|
+
`aai secret put MY_KEY` to store them on the server, then access via
|
|
1029
|
+
`ctx.env.MY_KEY`.
|
|
1030
|
+
- **Telling the agent to be verbose** — Voice responses should be 1-3 sentences.
|
|
1031
|
+
If your `instructions` say "provide detailed explanations", the agent will
|
|
1032
|
+
monologue. Instruct it to be brief and let the user ask follow-ups.
|
|
1033
|
+
- **Not setting secrets before deploying** — If your agent needs custom
|
|
1034
|
+
secrets, set them with `aai secret put MY_KEY` before deploying.
|
|
1035
|
+
- **Forgetting SSRF restrictions on `fetch`** — The host validates all proxied
|
|
1036
|
+
fetch URLs. Requests to private/internal IP addresses (localhost, 10.x,
|
|
1037
|
+
192.168.x, etc.) are blocked.
|
|
1038
|
+
|
|
1039
|
+
## Troubleshooting
|
|
1040
|
+
|
|
1041
|
+
- **"no agent found"** — Ensure `agent.ts` exists in the current directory
|
|
1042
|
+
- **"bundle failed"** — TypeScript syntax error — check imports, brackets
|
|
1043
|
+
- **"No .aai/project.json found"** — Run `aai deploy` first before using
|
|
1044
|
+
`aai secret`
|
|
1045
|
+
- **Tool returns `undefined`** — Make sure `execute` returns a value. Even
|
|
1046
|
+
`return { ok: true }` is better than an implicit void return.
|
|
1047
|
+
- **Agent doesn't use a tool** — Check `description` is clear about when to use
|
|
1048
|
+
it. The LLM relies on the description to decide. Also check `activeTools`
|
|
1049
|
+
isn't filtering it out.
|
|
1050
|
+
- **KV reads return `null`** — Keys are scoped per agent deployment. A
|
|
1051
|
+
redeployment with a new slug creates a fresh KV namespace.
|