@andreprado/agentkit 0.1.0-alpha.9 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +18 -1
- package/docs/guides/add-channel.md +251 -7
- package/docs/guides/add-knowledge.md +10 -0
- package/docs/guides/add-managed-composio.md +165 -0
- package/docs/guides/add-tool.md +10 -3
- package/docs/guides/channel-security.md +162 -32
- package/docs/guides/connect-discord.md +178 -0
- package/docs/guides/connect-slack.md +126 -0
- package/docs/guides/connect-telegram.md +61 -1
- package/docs/guides/connect-whatsapp-evolution.md +121 -0
- package/docs/guides/connect-whatsapp-uazapi.md +139 -0
- package/docs/guides/connect-whatsapp-zapster.md +119 -16
- package/docs/guides/create-agent.md +31 -4
- package/docs/guides/debug-channel.md +159 -0
- package/docs/guides/improve-from-production.md +151 -0
- package/docs/guides/prepare-deploy.md +32 -14
- package/docs/guides/replay-production-traces.md +72 -0
- package/docs/guides/run-evals.md +95 -25
- package/docs/guides/security-rules.md +9 -5
- package/docs/guides/send-feedback.md +135 -0
- package/docs/guides/use-jev.md +67 -0
- package/docs/guides/use-provider.md +70 -3
- package/docs/llms-full.txt +295 -25
- package/docs/llms.txt +54 -7
- package/package.json +3 -7
- package/src/cli/args.ts +23 -2
- package/src/cli/cloud-client.ts +121 -9
- package/src/cli/commands/channels.ts +856 -36
- package/src/cli/commands/feedback.ts +438 -0
- package/src/cli/commands/provider.ts +47 -0
- package/src/cli/commands/transcribe.ts +171 -0
- package/src/cli/deploy-chat-ui.ts +232 -18
- package/src/cli/deploy-readiness.ts +227 -14
- package/src/cli/help.ts +67 -9
- package/src/cli/index.ts +740 -35
- package/src/cli/new-command.ts +41 -0
- package/src/cloud/client.ts +4 -3
- package/src/cloud/contracts.ts +1 -1
- package/src/create-project.ts +18 -35
- package/src/index.ts +565 -11
- package/src/providers/codex-auth.ts +111 -0
- package/src/providers/pi.ts +88 -19
- package/src/providers/test.ts +36 -0
- package/src/providers/types.ts +8 -0
- package/src/runtime/channel-test-harness.ts +21 -1
- package/src/runtime/channels/discord.ts +896 -0
- package/src/runtime/channels/generic-webhook.ts +974 -0
- package/src/runtime/channels/slack.ts +646 -0
- package/src/runtime/channels/telegram.ts +343 -12
- package/src/runtime/channels/whatsapp-evolution.ts +1357 -0
- package/src/runtime/channels/whatsapp-meta.ts +9 -0
- package/src/runtime/channels/whatsapp-uazapi.ts +1323 -0
- package/src/runtime/channels/whatsapp-zapster.ts +674 -40
- package/src/runtime/channels.ts +83 -3
- package/src/runtime/chat.ts +70 -44
- package/src/runtime/config.ts +489 -20
- package/src/runtime/core/manifest.ts +75 -5
- package/src/runtime/core/targets.ts +5 -5
- package/src/runtime/deploy-readiness.ts +34 -4
- package/src/runtime/dev-server.ts +639 -39
- package/src/runtime/env.ts +8 -3
- package/src/runtime/evals.ts +445 -74
- package/src/runtime/improve.ts +868 -0
- package/src/runtime/inspect.ts +173 -4
- package/src/runtime/integrations/composio.ts +425 -0
- package/src/runtime/knowledge/retrieve.ts +25 -5
- package/src/runtime/knowledge/schema.ts +45 -1
- package/src/runtime/prompt-context.ts +141 -0
- package/src/runtime/runtime-contract.ts +71 -7
- package/src/runtime/skills.ts +95 -0
- package/src/runtime/targets/cloudflare/build.ts +1010 -208
- package/src/runtime/targets/container/server.ts +1 -1
- package/src/runtime/targets/vps/deploy.ts +26 -9
- package/src/runtime/tool-runner.ts +9 -1
- package/src/runtime/tools.ts +26 -2
- package/src/runtime/transcription.ts +483 -0
- package/src/storage/sqlite.ts +7 -2
- package/src/templates/blank.ts +37 -9
- package/src/templates/dentista.ts +40 -14
- package/src/templates/skills/agentkit-build-agent/SKILL.md +34 -5
- package/src/templates/skills/agentkit-build-agent/templates/appointment-intake.instructions.md +2 -1
- package/src/templates/skills/agentkit-capsule/SKILL.md +32 -3
- package/src/templates/skills/agentkit-capsule/references/docs-router.md +2 -2
- package/src/templates/skills/agentkit-channels/SKILL.md +66 -1
- package/src/templates/skills/agentkit-channels/references/channel-buffering.md +8 -1
- package/src/templates/skills/agentkit-channels/references/channel-debugging.md +28 -3
- package/src/templates/skills/agentkit-channels/references/discord.md +93 -0
- package/src/templates/skills/agentkit-channels/references/slack.md +56 -0
- package/src/templates/skills/agentkit-channels/references/telegram.md +34 -0
- package/src/templates/skills/agentkit-channels/references/whatsapp-evolution.md +57 -0
- package/src/templates/skills/agentkit-channels/references/whatsapp-uazapi.md +54 -0
- package/src/templates/skills/agentkit-channels/references/whatsapp-zapster.md +42 -8
- package/src/templates/skills/agentkit-database/SKILL.md +11 -0
- package/src/templates/skills/agentkit-deploy/SKILL.md +9 -1
- package/src/templates/skills/agentkit-evals/SKILL.md +77 -13
- package/src/templates/skills/agentkit-evals/templates/multi-turn.eval.md +13 -6
- package/src/templates/skills/agentkit-evals/templates/no-leak.eval.md +8 -4
- package/src/templates/skills/agentkit-evals/templates/smoke.eval.md +8 -4
- package/src/templates/skills/agentkit-evals/templates/tool-call.eval.md +16 -7
- package/src/templates/skills/agentkit-improve/SKILL.md +96 -0
- package/src/templates/skills/agentkit-improve/references/replay-side-effects.md +18 -0
- package/src/templates/skills/agentkit-improve/references/trace-packets.md +22 -0
- package/src/templates/skills/agentkit-improve/templates/regression.eval.md +18 -0
- package/src/templates/skills/agentkit-integrations/SKILL.md +98 -0
- package/src/templates/skills/agentkit-knowledge/SKILL.md +4 -1
- package/src/templates/skills/agentkit-prompts/SKILL.md +3 -1
- package/src/templates/skills/agentkit-provider/SKILL.md +29 -4
- package/src/templates/skills/agentkit-security/SKILL.md +5 -2
- package/src/templates/skills/agentkit-tools/SKILL.md +8 -1
- package/src/templates/skills/agentkit-tools/examples/eval-safe-external-action.tool.md +8 -8
- package/src/templates/skills/agentkit-tools/examples/jev-service-fit.tool.md +110 -0
- package/src/templates/skills/agentkit-troubleshooting/SKILL.md +25 -1
- package/src/templates/support.ts +42 -12
- package/docs/guides/agentkit-skills-architecture.md +0 -471
- package/docs/guides/channels-implementation-map.md +0 -243
- package/docs/guides/channels-production-handoff.md +0 -101
- package/docs/portable-deploy-release-checklist.md +0 -41
|
@@ -0,0 +1,483 @@
|
|
|
1
|
+
import type { AgentTranscriptionConfig, TranscriptionProviderName } from "../index";
|
|
2
|
+
|
|
3
|
+
export type ResolvedTranscriptionConfig = {
|
|
4
|
+
enabled: boolean;
|
|
5
|
+
provider: TranscriptionProviderName;
|
|
6
|
+
model: string;
|
|
7
|
+
secret: string | null;
|
|
8
|
+
language: string | null;
|
|
9
|
+
prompt: string | null;
|
|
10
|
+
limits: {
|
|
11
|
+
maxDurationSeconds?: number;
|
|
12
|
+
maxBytes?: number;
|
|
13
|
+
};
|
|
14
|
+
rawAudioTtlSeconds: number | null;
|
|
15
|
+
};
|
|
16
|
+
|
|
17
|
+
export type TranscriptionInput = {
|
|
18
|
+
config: ResolvedTranscriptionConfig;
|
|
19
|
+
secrets: Record<string, string>;
|
|
20
|
+
audio: Uint8Array;
|
|
21
|
+
filename: string;
|
|
22
|
+
mimeType?: string;
|
|
23
|
+
durationSeconds?: number;
|
|
24
|
+
providerMetadata?: Record<string, unknown>;
|
|
25
|
+
};
|
|
26
|
+
|
|
27
|
+
export type TranscriptionFetch = (input: RequestInfo | URL, init?: RequestInit) => Promise<Response>;
|
|
28
|
+
|
|
29
|
+
export type TranscriptionResult =
|
|
30
|
+
| {
|
|
31
|
+
ok: true;
|
|
32
|
+
text: string;
|
|
33
|
+
provider: TranscriptionProviderName;
|
|
34
|
+
model: string;
|
|
35
|
+
language?: string;
|
|
36
|
+
durationSeconds?: number;
|
|
37
|
+
providerMetadata?: Record<string, unknown>;
|
|
38
|
+
}
|
|
39
|
+
| {
|
|
40
|
+
ok: false;
|
|
41
|
+
retryable: boolean;
|
|
42
|
+
code:
|
|
43
|
+
| "transcription_disabled"
|
|
44
|
+
| "transcription_secret_missing"
|
|
45
|
+
| "transcription_model_unsupported"
|
|
46
|
+
| "transcription_audio_too_large"
|
|
47
|
+
| "transcription_audio_too_long"
|
|
48
|
+
| "transcription_audio_format_unsupported"
|
|
49
|
+
| "transcription_provider_unavailable"
|
|
50
|
+
| "transcription_failed";
|
|
51
|
+
message: string;
|
|
52
|
+
provider?: TranscriptionProviderName;
|
|
53
|
+
model?: string;
|
|
54
|
+
providerMetadata?: Record<string, unknown>;
|
|
55
|
+
};
|
|
56
|
+
|
|
57
|
+
export type TranscriptionAdapter = {
|
|
58
|
+
provider: TranscriptionProviderName;
|
|
59
|
+
models: string[];
|
|
60
|
+
supportedMimeTypes: string[];
|
|
61
|
+
supportedExtensions: string[];
|
|
62
|
+
maxBytes: number;
|
|
63
|
+
requiredSecret(config: ResolvedTranscriptionConfig): string | null;
|
|
64
|
+
transcribe(input: TranscriptionInput, fetcher?: TranscriptionFetch): Promise<TranscriptionResult>;
|
|
65
|
+
};
|
|
66
|
+
|
|
67
|
+
const DEFAULT_MAX_BYTES = 25_000_000;
|
|
68
|
+
const OPENAI_MODELS = ["gpt-4o-mini-transcribe", "gpt-4o-transcribe", "whisper-1"];
|
|
69
|
+
const GROQ_MODELS = ["whisper-large-v3-turbo", "whisper-large-v3", "distil-whisper-large-v3-en"];
|
|
70
|
+
const OPENAI_EXTENSIONS = ["mp3", "mp4", "mpeg", "mpga", "m4a", "wav", "webm"];
|
|
71
|
+
const GROQ_EXTENSIONS = ["flac", "mp3", "mp4", "mpeg", "mpga", "m4a", "ogg", "wav", "webm"];
|
|
72
|
+
const OPENAI_MIME_TYPES = [
|
|
73
|
+
"audio/mpeg",
|
|
74
|
+
"audio/mp3",
|
|
75
|
+
"audio/mp4",
|
|
76
|
+
"audio/mpga",
|
|
77
|
+
"audio/m4a",
|
|
78
|
+
"audio/wav",
|
|
79
|
+
"audio/webm",
|
|
80
|
+
"video/mp4",
|
|
81
|
+
];
|
|
82
|
+
const GROQ_MIME_TYPES = [
|
|
83
|
+
...OPENAI_MIME_TYPES,
|
|
84
|
+
"audio/flac",
|
|
85
|
+
"audio/ogg",
|
|
86
|
+
"audio/opus",
|
|
87
|
+
"application/ogg",
|
|
88
|
+
];
|
|
89
|
+
|
|
90
|
+
export function resolveTranscriptionConfig(config: AgentTranscriptionConfig | undefined): ResolvedTranscriptionConfig {
|
|
91
|
+
if (!config) {
|
|
92
|
+
return {
|
|
93
|
+
enabled: false,
|
|
94
|
+
provider: "test",
|
|
95
|
+
model: "fake",
|
|
96
|
+
secret: null,
|
|
97
|
+
language: null,
|
|
98
|
+
prompt: null,
|
|
99
|
+
limits: {},
|
|
100
|
+
rawAudioTtlSeconds: null,
|
|
101
|
+
};
|
|
102
|
+
}
|
|
103
|
+
|
|
104
|
+
return {
|
|
105
|
+
enabled: true,
|
|
106
|
+
provider: config.provider,
|
|
107
|
+
model: config.model,
|
|
108
|
+
secret: config.secret ?? defaultTranscriptionSecret(config.provider),
|
|
109
|
+
language: config.language ?? null,
|
|
110
|
+
prompt: config.prompt ?? null,
|
|
111
|
+
limits: {
|
|
112
|
+
...(config.limits?.maxDurationSeconds !== undefined
|
|
113
|
+
? { maxDurationSeconds: config.limits.maxDurationSeconds }
|
|
114
|
+
: {}),
|
|
115
|
+
...(config.limits?.maxBytes !== undefined ? { maxBytes: config.limits.maxBytes } : {}),
|
|
116
|
+
},
|
|
117
|
+
rawAudioTtlSeconds: config.rawAudioTtlSeconds ?? 3600,
|
|
118
|
+
};
|
|
119
|
+
}
|
|
120
|
+
|
|
121
|
+
export async function transcribeAudio(
|
|
122
|
+
input: TranscriptionInput,
|
|
123
|
+
fetcher: TranscriptionFetch = fetch,
|
|
124
|
+
): Promise<TranscriptionResult> {
|
|
125
|
+
if (!input.config.enabled) {
|
|
126
|
+
return {
|
|
127
|
+
ok: false,
|
|
128
|
+
retryable: false,
|
|
129
|
+
code: "transcription_disabled",
|
|
130
|
+
message: "Audio transcription is not configured for this Agent Capsule.",
|
|
131
|
+
};
|
|
132
|
+
}
|
|
133
|
+
|
|
134
|
+
const adapter = transcriptionAdapterFor(input.config.provider);
|
|
135
|
+
|
|
136
|
+
if (!adapter) {
|
|
137
|
+
return {
|
|
138
|
+
ok: false,
|
|
139
|
+
retryable: false,
|
|
140
|
+
code: "transcription_model_unsupported",
|
|
141
|
+
message: `No transcription adapter is registered for ${input.config.provider}.`,
|
|
142
|
+
provider: input.config.provider,
|
|
143
|
+
model: input.config.model,
|
|
144
|
+
};
|
|
145
|
+
}
|
|
146
|
+
|
|
147
|
+
const validation = validateTranscriptionInput(adapter, input);
|
|
148
|
+
|
|
149
|
+
if (!validation.ok) {
|
|
150
|
+
return validation;
|
|
151
|
+
}
|
|
152
|
+
|
|
153
|
+
return adapter.transcribe(input, fetcher);
|
|
154
|
+
}
|
|
155
|
+
|
|
156
|
+
export function transcriptionAdapterFor(provider: TranscriptionProviderName): TranscriptionAdapter | null {
|
|
157
|
+
if (provider === "test") {
|
|
158
|
+
return testTranscriptionAdapter;
|
|
159
|
+
}
|
|
160
|
+
|
|
161
|
+
if (provider === "openai") {
|
|
162
|
+
return openaiTranscriptionAdapter;
|
|
163
|
+
}
|
|
164
|
+
|
|
165
|
+
if (provider === "groq") {
|
|
166
|
+
return groqTranscriptionAdapter;
|
|
167
|
+
}
|
|
168
|
+
|
|
169
|
+
return null;
|
|
170
|
+
}
|
|
171
|
+
|
|
172
|
+
export function defaultTranscriptionSecret(provider: TranscriptionProviderName): string | null {
|
|
173
|
+
if (provider === "openai") {
|
|
174
|
+
return "OPENAI_API_KEY";
|
|
175
|
+
}
|
|
176
|
+
|
|
177
|
+
if (provider === "groq") {
|
|
178
|
+
return "GROQ_API_KEY";
|
|
179
|
+
}
|
|
180
|
+
|
|
181
|
+
return null;
|
|
182
|
+
}
|
|
183
|
+
|
|
184
|
+
function validateTranscriptionInput(adapter: TranscriptionAdapter, input: TranscriptionInput): TranscriptionResult {
|
|
185
|
+
if (!adapter.models.includes(input.config.model)) {
|
|
186
|
+
return {
|
|
187
|
+
ok: false,
|
|
188
|
+
retryable: false,
|
|
189
|
+
code: "transcription_model_unsupported",
|
|
190
|
+
message: `${input.config.provider} transcription model ${input.config.model} is not supported by AgentKit.`,
|
|
191
|
+
provider: input.config.provider,
|
|
192
|
+
model: input.config.model,
|
|
193
|
+
};
|
|
194
|
+
}
|
|
195
|
+
|
|
196
|
+
const maxBytes = Math.min(input.config.limits.maxBytes ?? adapter.maxBytes, adapter.maxBytes);
|
|
197
|
+
if (input.audio.byteLength > maxBytes) {
|
|
198
|
+
return {
|
|
199
|
+
ok: false,
|
|
200
|
+
retryable: false,
|
|
201
|
+
code: "transcription_audio_too_large",
|
|
202
|
+
message: `Audio file is ${input.audio.byteLength} bytes, which exceeds the configured ${maxBytes} byte limit.`,
|
|
203
|
+
provider: input.config.provider,
|
|
204
|
+
model: input.config.model,
|
|
205
|
+
};
|
|
206
|
+
}
|
|
207
|
+
|
|
208
|
+
if (
|
|
209
|
+
input.durationSeconds !== undefined &&
|
|
210
|
+
input.config.limits.maxDurationSeconds !== undefined &&
|
|
211
|
+
input.durationSeconds > input.config.limits.maxDurationSeconds
|
|
212
|
+
) {
|
|
213
|
+
return {
|
|
214
|
+
ok: false,
|
|
215
|
+
retryable: false,
|
|
216
|
+
code: "transcription_audio_too_long",
|
|
217
|
+
message: `Audio is ${input.durationSeconds}s, which exceeds the configured ${input.config.limits.maxDurationSeconds}s limit.`,
|
|
218
|
+
provider: input.config.provider,
|
|
219
|
+
model: input.config.model,
|
|
220
|
+
};
|
|
221
|
+
}
|
|
222
|
+
|
|
223
|
+
if (!isSupportedAudioFormat(adapter, input.filename, input.mimeType)) {
|
|
224
|
+
return {
|
|
225
|
+
ok: false,
|
|
226
|
+
retryable: false,
|
|
227
|
+
code: "transcription_audio_format_unsupported",
|
|
228
|
+
message: `${input.config.provider} does not support audio format ${input.mimeType ?? extensionFor(input.filename) ?? "unknown"}.`,
|
|
229
|
+
provider: input.config.provider,
|
|
230
|
+
model: input.config.model,
|
|
231
|
+
};
|
|
232
|
+
}
|
|
233
|
+
|
|
234
|
+
const secret = adapter.requiredSecret(input.config);
|
|
235
|
+
if (secret && !input.secrets[secret]) {
|
|
236
|
+
return {
|
|
237
|
+
ok: false,
|
|
238
|
+
retryable: false,
|
|
239
|
+
code: "transcription_secret_missing",
|
|
240
|
+
message: `Secret ${secret} is not set for audio transcription.`,
|
|
241
|
+
provider: input.config.provider,
|
|
242
|
+
model: input.config.model,
|
|
243
|
+
};
|
|
244
|
+
}
|
|
245
|
+
|
|
246
|
+
return {
|
|
247
|
+
ok: true,
|
|
248
|
+
text: "",
|
|
249
|
+
provider: input.config.provider,
|
|
250
|
+
model: input.config.model,
|
|
251
|
+
};
|
|
252
|
+
}
|
|
253
|
+
|
|
254
|
+
const testTranscriptionAdapter: TranscriptionAdapter = {
|
|
255
|
+
provider: "test",
|
|
256
|
+
models: ["fake"],
|
|
257
|
+
supportedMimeTypes: ["audio/wav", "audio/ogg", "audio/mpeg", "audio/webm", "application/octet-stream"],
|
|
258
|
+
supportedExtensions: ["wav", "ogg", "mp3", "webm", "bin"],
|
|
259
|
+
maxBytes: DEFAULT_MAX_BYTES,
|
|
260
|
+
requiredSecret() {
|
|
261
|
+
return null;
|
|
262
|
+
},
|
|
263
|
+
async transcribe(input) {
|
|
264
|
+
const text = decodeFixtureTranscript(input.audio) ?? "fake audio transcript";
|
|
265
|
+
|
|
266
|
+
return {
|
|
267
|
+
ok: true,
|
|
268
|
+
text,
|
|
269
|
+
provider: "test",
|
|
270
|
+
model: input.config.model,
|
|
271
|
+
...(input.config.language ? { language: input.config.language } : {}),
|
|
272
|
+
...(input.durationSeconds !== undefined ? { durationSeconds: input.durationSeconds } : {}),
|
|
273
|
+
};
|
|
274
|
+
},
|
|
275
|
+
};
|
|
276
|
+
|
|
277
|
+
const openaiTranscriptionAdapter: TranscriptionAdapter = {
|
|
278
|
+
provider: "openai",
|
|
279
|
+
models: OPENAI_MODELS,
|
|
280
|
+
supportedMimeTypes: OPENAI_MIME_TYPES,
|
|
281
|
+
supportedExtensions: OPENAI_EXTENSIONS,
|
|
282
|
+
maxBytes: DEFAULT_MAX_BYTES,
|
|
283
|
+
requiredSecret(config) {
|
|
284
|
+
return config.secret;
|
|
285
|
+
},
|
|
286
|
+
async transcribe(input, fetcher = fetch) {
|
|
287
|
+
return transcribeViaOpenAiCompatibleEndpoint({
|
|
288
|
+
input,
|
|
289
|
+
fetcher,
|
|
290
|
+
url: "https://api.openai.com/v1/audio/transcriptions",
|
|
291
|
+
apiKey: input.secrets[input.config.secret ?? ""],
|
|
292
|
+
provider: "openai",
|
|
293
|
+
});
|
|
294
|
+
},
|
|
295
|
+
};
|
|
296
|
+
|
|
297
|
+
const groqTranscriptionAdapter: TranscriptionAdapter = {
|
|
298
|
+
provider: "groq",
|
|
299
|
+
models: GROQ_MODELS,
|
|
300
|
+
supportedMimeTypes: GROQ_MIME_TYPES,
|
|
301
|
+
supportedExtensions: GROQ_EXTENSIONS,
|
|
302
|
+
maxBytes: DEFAULT_MAX_BYTES,
|
|
303
|
+
requiredSecret(config) {
|
|
304
|
+
return config.secret;
|
|
305
|
+
},
|
|
306
|
+
async transcribe(input, fetcher = fetch) {
|
|
307
|
+
return transcribeViaOpenAiCompatibleEndpoint({
|
|
308
|
+
input,
|
|
309
|
+
fetcher,
|
|
310
|
+
url: "https://api.groq.com/openai/v1/audio/transcriptions",
|
|
311
|
+
apiKey: input.secrets[input.config.secret ?? ""],
|
|
312
|
+
provider: "groq",
|
|
313
|
+
});
|
|
314
|
+
},
|
|
315
|
+
};
|
|
316
|
+
|
|
317
|
+
async function transcribeViaOpenAiCompatibleEndpoint(input: {
|
|
318
|
+
input: TranscriptionInput;
|
|
319
|
+
fetcher: TranscriptionFetch;
|
|
320
|
+
url: string;
|
|
321
|
+
apiKey: string | undefined;
|
|
322
|
+
provider: TranscriptionProviderName;
|
|
323
|
+
}): Promise<TranscriptionResult> {
|
|
324
|
+
if (!input.apiKey) {
|
|
325
|
+
const secret = input.input.config.secret ?? defaultTranscriptionSecret(input.provider) ?? "TRANSCRIPTION_API_KEY";
|
|
326
|
+
|
|
327
|
+
return {
|
|
328
|
+
ok: false,
|
|
329
|
+
retryable: false,
|
|
330
|
+
code: "transcription_secret_missing",
|
|
331
|
+
message: `Secret ${secret} is not set for audio transcription.`,
|
|
332
|
+
provider: input.provider,
|
|
333
|
+
model: input.input.config.model,
|
|
334
|
+
};
|
|
335
|
+
}
|
|
336
|
+
|
|
337
|
+
const form = new FormData();
|
|
338
|
+
form.set("model", input.input.config.model);
|
|
339
|
+
form.set(
|
|
340
|
+
"file",
|
|
341
|
+
new Blob([arrayBufferForBlob(input.input.audio)], { type: input.input.mimeType ?? "application/octet-stream" }),
|
|
342
|
+
input.input.filename,
|
|
343
|
+
);
|
|
344
|
+
form.set("response_format", "json");
|
|
345
|
+
|
|
346
|
+
if (input.input.config.language) {
|
|
347
|
+
form.set("language", input.input.config.language);
|
|
348
|
+
}
|
|
349
|
+
|
|
350
|
+
if (input.input.config.prompt) {
|
|
351
|
+
form.set("prompt", input.input.config.prompt);
|
|
352
|
+
}
|
|
353
|
+
|
|
354
|
+
let response: Response;
|
|
355
|
+
|
|
356
|
+
try {
|
|
357
|
+
response = await input.fetcher(input.url, {
|
|
358
|
+
method: "POST",
|
|
359
|
+
headers: {
|
|
360
|
+
Authorization: `Bearer ${input.apiKey}`,
|
|
361
|
+
},
|
|
362
|
+
body: form,
|
|
363
|
+
});
|
|
364
|
+
} catch (error) {
|
|
365
|
+
return {
|
|
366
|
+
ok: false,
|
|
367
|
+
retryable: true,
|
|
368
|
+
code: "transcription_provider_unavailable",
|
|
369
|
+
message: `Transcription provider request failed before a response: ${redactSecret(
|
|
370
|
+
error instanceof Error ? error.message : String(error),
|
|
371
|
+
input.apiKey,
|
|
372
|
+
)}`,
|
|
373
|
+
provider: input.provider,
|
|
374
|
+
model: input.input.config.model,
|
|
375
|
+
};
|
|
376
|
+
}
|
|
377
|
+
|
|
378
|
+
const payload = await response.json().catch(() => null);
|
|
379
|
+
|
|
380
|
+
if (!response.ok) {
|
|
381
|
+
const message = readProviderError(payload) ?? `Transcription provider returned HTTP ${response.status}.`;
|
|
382
|
+
|
|
383
|
+
return {
|
|
384
|
+
ok: false,
|
|
385
|
+
retryable: response.status === 429 || response.status >= 500,
|
|
386
|
+
code: response.status === 429 || response.status >= 500 ? "transcription_provider_unavailable" : "transcription_failed",
|
|
387
|
+
message: redactSecret(message, input.apiKey),
|
|
388
|
+
provider: input.provider,
|
|
389
|
+
model: input.input.config.model,
|
|
390
|
+
providerMetadata: {
|
|
391
|
+
status: response.status,
|
|
392
|
+
},
|
|
393
|
+
};
|
|
394
|
+
}
|
|
395
|
+
|
|
396
|
+
const text = isRecord(payload) && typeof payload.text === "string" ? payload.text.trim() : "";
|
|
397
|
+
|
|
398
|
+
if (!text) {
|
|
399
|
+
return {
|
|
400
|
+
ok: false,
|
|
401
|
+
retryable: false,
|
|
402
|
+
code: "transcription_failed",
|
|
403
|
+
message: "Transcription provider returned an empty transcript.",
|
|
404
|
+
provider: input.provider,
|
|
405
|
+
model: input.input.config.model,
|
|
406
|
+
providerMetadata: {
|
|
407
|
+
status: response.status,
|
|
408
|
+
},
|
|
409
|
+
};
|
|
410
|
+
}
|
|
411
|
+
|
|
412
|
+
return {
|
|
413
|
+
ok: true,
|
|
414
|
+
text,
|
|
415
|
+
provider: input.provider,
|
|
416
|
+
model: input.input.config.model,
|
|
417
|
+
...(input.input.config.language ? { language: input.input.config.language } : {}),
|
|
418
|
+
...(input.input.durationSeconds !== undefined ? { durationSeconds: input.input.durationSeconds } : {}),
|
|
419
|
+
providerMetadata: {
|
|
420
|
+
status: response.status,
|
|
421
|
+
},
|
|
422
|
+
};
|
|
423
|
+
}
|
|
424
|
+
|
|
425
|
+
function isSupportedAudioFormat(adapter: TranscriptionAdapter, filename: string, mimeType?: string): boolean {
|
|
426
|
+
const normalizedMimeType = mimeType?.toLowerCase();
|
|
427
|
+
|
|
428
|
+
if (normalizedMimeType && adapter.supportedMimeTypes.includes(normalizedMimeType)) {
|
|
429
|
+
return true;
|
|
430
|
+
}
|
|
431
|
+
|
|
432
|
+
const extension = extensionFor(filename);
|
|
433
|
+
|
|
434
|
+
return Boolean(extension && adapter.supportedExtensions.includes(extension));
|
|
435
|
+
}
|
|
436
|
+
|
|
437
|
+
function extensionFor(filename: string): string | null {
|
|
438
|
+
const match = /\.([a-z0-9]+)$/i.exec(filename);
|
|
439
|
+
return match ? match[1].toLowerCase() : null;
|
|
440
|
+
}
|
|
441
|
+
|
|
442
|
+
function decodeFixtureTranscript(audio: Uint8Array): string | null {
|
|
443
|
+
try {
|
|
444
|
+
const text = new TextDecoder().decode(audio).trim();
|
|
445
|
+
return text.length > 0 && /^[\t\n\r -~\u00a0-\uffff]+$/.test(text) ? text : null;
|
|
446
|
+
} catch {
|
|
447
|
+
return null;
|
|
448
|
+
}
|
|
449
|
+
}
|
|
450
|
+
|
|
451
|
+
function arrayBufferForBlob(audio: Uint8Array): ArrayBuffer {
|
|
452
|
+
const copy = new Uint8Array(audio.byteLength);
|
|
453
|
+
copy.set(audio);
|
|
454
|
+
return copy.buffer;
|
|
455
|
+
}
|
|
456
|
+
|
|
457
|
+
function readProviderError(payload: unknown): string | null {
|
|
458
|
+
if (!isRecord(payload)) {
|
|
459
|
+
return null;
|
|
460
|
+
}
|
|
461
|
+
|
|
462
|
+
if (typeof payload.error === "string") {
|
|
463
|
+
return payload.error;
|
|
464
|
+
}
|
|
465
|
+
|
|
466
|
+
if (isRecord(payload.error) && typeof payload.error.message === "string") {
|
|
467
|
+
return payload.error.message;
|
|
468
|
+
}
|
|
469
|
+
|
|
470
|
+
if (typeof payload.message === "string") {
|
|
471
|
+
return payload.message;
|
|
472
|
+
}
|
|
473
|
+
|
|
474
|
+
return null;
|
|
475
|
+
}
|
|
476
|
+
|
|
477
|
+
function redactSecret(value: string, secret: string): string {
|
|
478
|
+
return secret ? value.replaceAll(secret, "<redacted>") : value;
|
|
479
|
+
}
|
|
480
|
+
|
|
481
|
+
function isRecord(value: unknown): value is Record<string, unknown> {
|
|
482
|
+
return Boolean(value) && typeof value === "object" && !Array.isArray(value);
|
|
483
|
+
}
|
package/src/storage/sqlite.ts
CHANGED
|
@@ -6,7 +6,7 @@ import type { DatabaseArgs, DatabaseResult, DatabaseRow, DatabaseStatement } fro
|
|
|
6
6
|
import type { AgentMessageRole, ProviderRunResult } from "../providers";
|
|
7
7
|
import type { LoadedAgentCapsule } from "../runtime/config";
|
|
8
8
|
import { AgentKitError } from "../runtime/errors";
|
|
9
|
-
import { KNOWLEDGE_MIGRATION_ID, KNOWLEDGE_SCHEMA_SQL } from "../runtime/knowledge/schema";
|
|
9
|
+
import { applyKnowledgeSchema, KNOWLEDGE_MIGRATION_ID, KNOWLEDGE_SCHEMA_SQL } from "../runtime/knowledge/schema";
|
|
10
10
|
|
|
11
11
|
export type ConversationSummary = {
|
|
12
12
|
id: string;
|
|
@@ -806,7 +806,12 @@ export class SqliteAgentKitStore {
|
|
|
806
806
|
continue;
|
|
807
807
|
}
|
|
808
808
|
|
|
809
|
-
|
|
809
|
+
if (migration.id === KNOWLEDGE_MIGRATION_ID) {
|
|
810
|
+
applyKnowledgeSchema(this.db);
|
|
811
|
+
} else {
|
|
812
|
+
this.db.exec(migration.sql);
|
|
813
|
+
}
|
|
814
|
+
|
|
810
815
|
this.db
|
|
811
816
|
.prepare("INSERT INTO agentkit_migrations (id, applied_at) VALUES ($id, $appliedAt)")
|
|
812
817
|
.run({ $id: migration.id, $appliedAt: new Date().toISOString() });
|
package/src/templates/blank.ts
CHANGED
|
@@ -66,6 +66,7 @@ node_modules/
|
|
|
66
66
|
OPENAI_API_KEY=
|
|
67
67
|
ANTHROPIC_API_KEY=
|
|
68
68
|
OPENROUTER_API_KEY=
|
|
69
|
+
OPENCODE_API_KEY=
|
|
69
70
|
`,
|
|
70
71
|
},
|
|
71
72
|
{
|
|
@@ -124,13 +125,18 @@ Answer clearly, ask for missing context when needed, and do not claim to have pe
|
|
|
124
125
|
},
|
|
125
126
|
{
|
|
126
127
|
path: "evals/smoke.eval.ts",
|
|
127
|
-
contents: `
|
|
128
|
+
contents: `import { defineEval } from "@andreprado/agentkit";
|
|
129
|
+
|
|
130
|
+
export default defineEval({
|
|
128
131
|
name: "smoke",
|
|
129
132
|
input: "Say hello in one short sentence.",
|
|
130
133
|
expect: {
|
|
131
|
-
|
|
134
|
+
response: {
|
|
135
|
+
caseInsensitiveContains: "hello",
|
|
136
|
+
maxLength: 160,
|
|
137
|
+
},
|
|
132
138
|
},
|
|
133
|
-
};
|
|
139
|
+
});
|
|
134
140
|
`,
|
|
135
141
|
},
|
|
136
142
|
{
|
|
@@ -143,7 +149,7 @@ The capsule root is the runtime boundary. Run, inspect, evaluate, and deploy fro
|
|
|
143
149
|
|
|
144
150
|
## Coding Agent Workflow
|
|
145
151
|
|
|
146
|
-
When the owner opens this folder in Codex, Claude Code, or another coding agent and asks for a specific agent, treat that request as the product brief.
|
|
152
|
+
When the owner opens this folder in Codex, Claude Code, or another coding agent and asks for a specific agent, treat that request as the product brief. The owner should not need to run a separate AgentKit wizard or prepare a brief file.
|
|
147
153
|
|
|
148
154
|
Start building immediately:
|
|
149
155
|
|
|
@@ -153,11 +159,26 @@ Start building immediately:
|
|
|
153
159
|
- Edit \`prompts/instructions.md\` for the agent behavior.
|
|
154
160
|
- Edit \`agentkit.config.ts\` for provider, tools, secrets, access, and storage.
|
|
155
161
|
- Add TypeScript tools under \`tools/\` when the requested agent needs actions or external data.
|
|
162
|
+
- When the agent needs to save durable records, complete the full slice: schema/migration, tool, config registration, prompt instructions, direct tool check, and eval.
|
|
156
163
|
- Add \`sync.ts\`, \`seed.sql\`, and ordered \`migrations/*.sql\` when the requested agent depends on external catalogs or production-shaped data changes.
|
|
157
164
|
- Do not wait for a wizard or recipe. AgentKit provides the scaffold and contract; you decide the implementation from the owner's brief.
|
|
158
165
|
- Ask follow-up questions only when missing information blocks a safe local implementation.
|
|
159
166
|
- State assumptions in the final response.
|
|
160
167
|
|
|
168
|
+
## Proactive Agent Builder Contract
|
|
169
|
+
|
|
170
|
+
Do not only edit prompts. For every meaningful requirement in the owner's request or \`AGENT_SPEC.md\`, decide what should enforce it:
|
|
171
|
+
|
|
172
|
+
- spec entry for the product contract;
|
|
173
|
+
- prompt instruction for behavior, tone, boundaries, intake, and escalation;
|
|
174
|
+
- tool plus config registration for actions, live data, external writes, or authorization-sensitive data;
|
|
175
|
+
- schema/migration plus tool for durable records;
|
|
176
|
+
- eval for privacy, confirmation, required fields, date/time behavior, business rules, and regressions;
|
|
177
|
+
- fixture, seed data, fake branch, or direct tool check for integrations and failure paths;
|
|
178
|
+
- deploy/readiness check for hosted secrets, channels, integrations, or production access.
|
|
179
|
+
|
|
180
|
+
If a rule protects privacy, money, bookings, external writes, customer data, business hours, or safety, it must have an eval or deterministic check before you call the capsule done. If a real conversation exposes a bug, convert it into the smallest regression eval before or alongside the fix.
|
|
181
|
+
|
|
161
182
|
## Local Commands
|
|
162
183
|
|
|
163
184
|
- \`npm install\`: restore capsule dependencies if this capsule used \`--no-install\`, install failed, or \`node_modules\` was deleted.
|
|
@@ -183,7 +204,7 @@ Start building immediately:
|
|
|
183
204
|
- Local UI: run \`npm run dev\`, open the printed \`Chat:\` URL, and tell the owner the exact URL.
|
|
184
205
|
- Hosted UI: after \`npm run agentkit -- deploy\`, run \`npm run agentkit -- chat-ui --deploy\`, open the printed \`Chat:\` URL, and tell the owner it is connected to the hosted deploy.
|
|
185
206
|
- \`test/fake\` is deterministic. It is useful for scaffold checks, direct tool checks, and fake-provider evals, but it does not validate natural conversation quality.
|
|
186
|
-
- Before claiming real conversation behavior is tested, ask the owner which provider to use: OpenRouter, OpenAI, Anthropic, or another supported provider. Do not choose for them.
|
|
207
|
+
- Before claiming real conversation behavior is tested, ask the owner which provider to use: OpenRouter, OpenAI, Anthropic, OpenCode Zen, OpenCode Go, or another supported provider. Do not choose for them.
|
|
187
208
|
|
|
188
209
|
This blank capsule starts with the built-in \`test/fake\` provider, so local chat works without secrets or internet access. It is also deploy-ready by default: the user can edit the prompt, add tools, and run \`npm run agentkit -- deploy\`.
|
|
189
210
|
|
|
@@ -248,7 +269,9 @@ This folder is an AgentKit Agent Capsule.
|
|
|
248
269
|
|
|
249
270
|
## Start Here
|
|
250
271
|
|
|
251
|
-
If the owner asks you to build an agent in natural language, that request is the brief. Do not ask them to fill another file first.
|
|
272
|
+
If the owner asks you to build an agent in natural language, that request is the brief. Do not ask them to run a wizard or fill another file first.
|
|
273
|
+
|
|
274
|
+
Build a testable capsule, not only a prompt.
|
|
252
275
|
|
|
253
276
|
Example owner request:
|
|
254
277
|
|
|
@@ -260,7 +283,10 @@ Turn the request into a working local capsule:
|
|
|
260
283
|
- Create or update \`AGENT_SPEC.md\` with \`npm run agentkit -- spec init --brief "<owner request>"\`. The owner gives the general idea; the coding agent turns it into the structured contract.
|
|
261
284
|
- Update \`agentkit.config.ts\` when tools, secrets, provider, or access rules change.
|
|
262
285
|
- Add TypeScript tools under \`tools/\` for real actions or external data.
|
|
286
|
+
- For durable records, implement the full schema/tool/prompt/eval slice instead of only adding a table or only adding a tool.
|
|
263
287
|
- Use \`npm run agentkit -- sync init\` when the agent needs catalog sync, fixture seed data, or ordered migrations.
|
|
288
|
+
- For every privacy, confirmation, required-intake, timezone, business-hour, integration-error, or no-leak rule, add an eval, fixture, fake branch, or direct tool check.
|
|
289
|
+
- Convert failed or surprising real conversations into regression evals with \`npm run agentkit -- eval from-conversation <conversation-id>\`.
|
|
264
290
|
- Keep the first version runnable with \`test/fake\` unless the owner explicitly asks for a real provider.
|
|
265
291
|
- Do not use a wizard or recipe. Build the capsule directly from the scaffold, the AgentKit contract, and the owner's brief.
|
|
266
292
|
- Make practical assumptions and list them in your final response.
|
|
@@ -276,6 +302,8 @@ npm run chat -- --message "hello"
|
|
|
276
302
|
|
|
277
303
|
\`test/fake\` proves the scaffold and deterministic tool paths. It does not prove natural conversation quality.
|
|
278
304
|
|
|
305
|
+
Before saying the agent is done, make sure important requirements have matching checks. Prompt-only changes are not enough for privacy, external writes, bookings, customer data, business hours, or integration failures.
|
|
306
|
+
|
|
279
307
|
\`agentkit new\` installs dependencies by default. Run \`npm install\` only if the capsule was created with \`--no-install\`, install failed, or \`node_modules\` was deleted.
|
|
280
308
|
|
|
281
309
|
Set local development secrets without opening code:
|
|
@@ -305,7 +333,7 @@ npm run agentkit -- chat-ui --deploy
|
|
|
305
333
|
|
|
306
334
|
Open the printed \`Chat:\` URL and tell the owner this local UI is connected to the hosted deploy.
|
|
307
335
|
|
|
308
|
-
Before claiming real conversation behavior has been tested, ask the owner which provider to use: OpenRouter, OpenAI, Anthropic, or another supported provider. Do not choose for them. After they choose, update \`agentkit.config.ts\`, \`.env.schema\`, local secrets, hosted secrets if deploying, then rerun chat/UI checks.
|
|
336
|
+
Before claiming real conversation behavior has been tested, ask the owner which provider to use: OpenRouter, OpenAI, Anthropic, OpenCode Zen, OpenCode Go, or another supported provider. Do not choose for them. After they choose, update \`agentkit.config.ts\`, \`.env.schema\`, local secrets, hosted secrets if deploying, then rerun chat/UI checks.
|
|
309
337
|
|
|
310
338
|
If you add a tool, also run a fake-provider tool smoke test:
|
|
311
339
|
|
|
@@ -349,7 +377,7 @@ Use AgentKit conventions when editing this project.
|
|
|
349
377
|
- Local runtime state lives in \`.agentkit/\` and should not be committed.
|
|
350
378
|
- The default provider is \`test/fake\`, which needs no secrets.
|
|
351
379
|
- Real providers run through AgentKit's internal Pi SDK backend.
|
|
352
|
-
- Ask the owner which real provider to use before switching from \`test/fake\`; do not choose OpenRouter, OpenAI, or
|
|
380
|
+
- Ask the owner which real provider to use before switching from \`test/fake\`; do not choose OpenRouter, OpenAI, Anthropic, OpenCode Zen, or OpenCode Go automatically.
|
|
353
381
|
- Production secrets must be managed secrets, not committed files.
|
|
354
382
|
- Keep required local secret names in \`.env.schema\` and values in ignored \`.env\`. AgentKit local commands load \`.env\` directly.
|
|
355
383
|
- Treat the owner's natural-language request as the brief and start implementing inside this capsule.
|
|
@@ -378,7 +406,7 @@ When a real provider or tool needs a local development secret, keep the required
|
|
|
378
406
|
|
|
379
407
|
For UI testing, run \`npm run dev\` and open the printed \`Chat:\` URL. After hosted deploy, run \`npm run agentkit -- chat-ui --deploy\` and open its printed \`Chat:\` URL.
|
|
380
408
|
|
|
381
|
-
\`test/fake\` does not validate real conversation quality. The owner must choose OpenRouter, OpenAI, Anthropic, or another supported provider before real model behavior is tested.
|
|
409
|
+
\`test/fake\` does not validate real conversation quality. The owner must choose OpenRouter, OpenAI, Anthropic, OpenCode Zen, OpenCode Go, or another supported provider before real model behavior is tested.
|
|
382
410
|
|
|
383
411
|
## Files
|
|
384
412
|
|