@andreprado/agentkit 0.1.0-alpha.2 → 0.1.0-alpha.20
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +67 -6
- package/docs/guides/add-channel.md +118 -7
- package/docs/guides/add-knowledge.md +144 -0
- package/docs/guides/add-managed-composio.md +163 -0
- package/docs/guides/add-tool.md +1 -1
- package/docs/guides/channel-security.md +97 -32
- package/docs/guides/connect-discord.md +178 -0
- package/docs/guides/connect-slack.md +126 -0
- package/docs/guides/connect-telegram.md +78 -1
- package/docs/guides/connect-whatsapp-zapster.md +112 -8
- package/docs/guides/create-agent.md +45 -4
- package/docs/guides/debug-channel.md +147 -0
- package/docs/guides/improve-from-production.md +151 -0
- package/docs/guides/prepare-deploy.md +47 -17
- package/docs/guides/replay-production-traces.md +72 -0
- package/docs/guides/run-evals.md +147 -20
- package/docs/guides/security-rules.md +7 -6
- package/docs/guides/send-feedback.md +135 -0
- package/docs/guides/use-provider.md +27 -3
- package/docs/llms-full.txt +303 -55
- package/docs/llms.txt +57 -7
- package/package.json +2 -5
- package/src/cli/args.ts +57 -0
- package/src/cli/cloud-client.ts +377 -0
- package/src/cli/commands/channels.ts +1315 -0
- package/src/cli/commands/feedback.ts +438 -0
- package/src/cli/commands/knowledge.ts +136 -0
- package/src/cli/commands/transcribe.ts +171 -0
- package/src/cli/constants.ts +4 -0
- package/src/cli/deploy-chat-ui.ts +535 -0
- package/src/cli/deploy-readiness.ts +481 -0
- package/src/cli/flags.ts +162 -0
- package/src/cli/help.ts +236 -0
- package/src/cli/index.ts +1167 -1005
- package/src/cli/process.ts +31 -0
- package/src/cloud/artifact.ts +139 -0
- package/src/cloud/client.ts +80 -0
- package/src/cloud/contracts.ts +63 -0
- package/src/cloud/index.ts +3 -0
- package/src/create-project.ts +21 -6
- package/src/index.ts +479 -7
- package/src/providers/pi.ts +70 -16
- package/src/providers/test.ts +88 -1
- package/src/providers/types.ts +7 -0
- package/src/runtime/channel-buffer.ts +30 -0
- package/src/runtime/channel-test-harness.ts +8 -1
- package/src/runtime/channels/discord.ts +896 -0
- package/src/runtime/channels/slack.ts +646 -0
- package/src/runtime/channels/telegram.ts +466 -23
- package/src/runtime/channels/whatsapp-meta.ts +9 -0
- package/src/runtime/channels/whatsapp-zapster.ts +677 -40
- package/src/runtime/channels.ts +86 -3
- package/src/runtime/chat.ts +130 -38
- package/src/runtime/config.ts +483 -18
- package/src/runtime/core/manifest.ts +103 -5
- package/src/runtime/core/targets.ts +5 -5
- package/src/runtime/database.ts +93 -2
- package/src/runtime/db-commands.ts +9 -0
- package/src/runtime/deploy-readiness.ts +46 -4
- package/src/runtime/deploy.ts +1 -1
- package/src/runtime/dev-server.ts +759 -41
- package/src/runtime/env.ts +8 -3
- package/src/runtime/evals.ts +589 -43
- package/src/runtime/improve.ts +868 -0
- package/src/runtime/inspect.ts +194 -4
- package/src/runtime/integrations/composio.ts +423 -0
- package/src/runtime/knowledge/chunk.ts +333 -0
- package/src/runtime/knowledge/config.ts +135 -0
- package/src/runtime/knowledge/embeddings.ts +133 -0
- package/src/runtime/knowledge/ingest.ts +521 -0
- package/src/runtime/knowledge/prompt-policy.ts +30 -0
- package/src/runtime/knowledge/retrieve.ts +303 -0
- package/src/runtime/knowledge/schema.ts +100 -0
- package/src/runtime/knowledge/tool.ts +64 -0
- package/src/runtime/knowledge/vector.ts +258 -0
- package/src/runtime/prompt-context.ts +141 -0
- package/src/runtime/runtime-contract.ts +86 -8
- package/src/runtime/skills.ts +95 -0
- package/src/runtime/spec.ts +152 -0
- package/src/runtime/sync.ts +144 -0
- package/src/runtime/targets/cloudflare/build.ts +1430 -185
- package/src/runtime/targets/container/server.ts +1 -1
- package/src/runtime/targets/vps/deploy.ts +26 -9
- package/src/runtime/tool-runner.ts +9 -1
- package/src/runtime/tools.ts +128 -2
- package/src/runtime/traces.ts +41 -0
- package/src/runtime/transcription.ts +483 -0
- package/src/storage/sqlite.ts +149 -3
- package/src/templates/blank.ts +76 -17
- package/src/templates/dentista.ts +1011 -0
- package/src/templates/index.ts +2 -0
- package/src/templates/skills/agentkit-build-agent/SKILL.md +52 -0
- package/src/templates/skills/agentkit-build-agent/templates/appointment-intake.instructions.md +21 -0
- package/src/templates/skills/agentkit-build-agent/templates/sales-qualifier.instructions.md +17 -0
- package/src/templates/skills/agentkit-build-agent/templates/support-agent.instructions.md +16 -0
- package/src/templates/skills/agentkit-capsule/SKILL.md +70 -0
- package/src/templates/skills/agentkit-capsule/references/docs-router.md +15 -0
- package/src/templates/skills/agentkit-channels/SKILL.md +104 -0
- package/src/templates/skills/agentkit-channels/references/channel-buffering.md +65 -0
- package/src/templates/skills/agentkit-channels/references/channel-debugging.md +66 -0
- package/src/templates/skills/agentkit-channels/references/discord.md +93 -0
- package/src/templates/skills/agentkit-channels/references/slack.md +56 -0
- package/src/templates/skills/agentkit-channels/references/telegram.md +72 -0
- package/src/templates/skills/agentkit-channels/references/whatsapp-zapster.md +77 -0
- package/src/templates/skills/agentkit-database/SKILL.md +45 -0
- package/src/templates/skills/agentkit-database/templates/appointments.schema.sql +15 -0
- package/src/templates/skills/agentkit-database/templates/leads.schema.sql +17 -0
- package/src/templates/skills/agentkit-deploy/SKILL.md +50 -0
- package/src/templates/skills/agentkit-evals/SKILL.md +109 -0
- package/src/templates/skills/agentkit-evals/templates/multi-turn.eval.md +29 -0
- package/src/templates/skills/agentkit-evals/templates/no-leak.eval.md +18 -0
- package/src/templates/skills/agentkit-evals/templates/smoke.eval.md +18 -0
- package/src/templates/skills/agentkit-evals/templates/tool-call.eval.md +27 -0
- package/src/templates/skills/agentkit-improve/SKILL.md +86 -0
- package/src/templates/skills/agentkit-improve/references/replay-side-effects.md +18 -0
- package/src/templates/skills/agentkit-improve/references/trace-packets.md +22 -0
- package/src/templates/skills/agentkit-improve/templates/regression.eval.md +18 -0
- package/src/templates/skills/agentkit-integrations/SKILL.md +76 -0
- package/src/templates/skills/agentkit-knowledge/SKILL.md +43 -0
- package/src/templates/skills/agentkit-knowledge/templates/faq.md +14 -0
- package/src/templates/skills/agentkit-knowledge/templates/policies.md +14 -0
- package/src/templates/skills/agentkit-knowledge/templates/prices.csv +3 -0
- package/src/templates/skills/agentkit-prompts/SKILL.md +47 -0
- package/src/templates/skills/agentkit-prompts/templates/knowledge-grounded-faq.instructions.md +11 -0
- package/src/templates/skills/agentkit-provider/SKILL.md +60 -0
- package/src/templates/skills/agentkit-security/SKILL.md +56 -0
- package/src/templates/skills/agentkit-tools/SKILL.md +37 -0
- package/src/templates/skills/agentkit-tools/examples/database-write.tool.md +35 -0
- package/src/templates/skills/agentkit-tools/examples/eval-safe-external-action.tool.md +37 -0
- package/src/templates/skills/agentkit-tools/examples/lookup-order.tool.md +46 -0
- package/src/templates/skills/agentkit-troubleshooting/SKILL.md +76 -0
- package/src/templates/support.ts +77 -18
- package/docs/guides/channels-production-handoff.md +0 -99
- package/docs/portable-deploy-release-checklist.md +0 -41
- package/src/runtime/targets/cloudflare/deploy.ts +0 -5475
|
@@ -12,18 +12,37 @@ Use this after `whatsappChannel({ name: "support-whatsapp", provider: "zapster"
|
|
|
12
12
|
|
|
13
13
|
```sh
|
|
14
14
|
agentkit deploy
|
|
15
|
-
agentkit channels
|
|
16
|
-
agentkit channels setup support-whatsapp
|
|
15
|
+
agentkit channels connect whatsapp support-whatsapp --provider zapster
|
|
17
16
|
agentkit channels status support-whatsapp
|
|
18
17
|
agentkit channels test support-whatsapp --message "hello"
|
|
19
18
|
agentkit channels deliveries list support-whatsapp --since 24h
|
|
19
|
+
agentkit channels buffers list support-whatsapp
|
|
20
20
|
```
|
|
21
21
|
|
|
22
22
|
Required secrets:
|
|
23
23
|
|
|
24
24
|
```txt
|
|
25
25
|
ZAPSTER_API_KEY
|
|
26
|
-
|
|
26
|
+
ZAPSTER_INSTANCE_ID
|
|
27
|
+
ZAPSTER_WEBHOOK_ID
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Optional hardening secret:
|
|
31
|
+
|
|
32
|
+
```txt
|
|
33
|
+
ZAPSTER_WEBHOOK_TOKEN
|
|
34
|
+
```
|
|
35
|
+
|
|
36
|
+
If WhatsApp audio transcription is enabled, also set the transcription provider secret declared by `agentkit inspect`, usually:
|
|
37
|
+
|
|
38
|
+
```txt
|
|
39
|
+
OPENAI_API_KEY
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
or:
|
|
43
|
+
|
|
44
|
+
```txt
|
|
45
|
+
GROQ_API_KEY
|
|
27
46
|
```
|
|
28
47
|
|
|
29
48
|
## Files Created Or Edited
|
|
@@ -50,22 +69,97 @@ export default defineAgent({
|
|
|
50
69
|
});
|
|
51
70
|
```
|
|
52
71
|
|
|
72
|
+
To handle clients who send several WhatsApp messages before waiting, enable channel buffering:
|
|
73
|
+
|
|
74
|
+
```ts
|
|
75
|
+
whatsappChannel({
|
|
76
|
+
name: "support-whatsapp",
|
|
77
|
+
provider: "zapster",
|
|
78
|
+
buffer: {
|
|
79
|
+
mode: "debounce",
|
|
80
|
+
quietWindowMs: 2500,
|
|
81
|
+
maxWaitMs: 12000,
|
|
82
|
+
maxMessages: 20,
|
|
83
|
+
maxChars: 8000,
|
|
84
|
+
},
|
|
85
|
+
})
|
|
86
|
+
```
|
|
87
|
+
|
|
88
|
+
## Auto Transcribe WhatsApp Audio
|
|
89
|
+
|
|
90
|
+
Use `transcription` at the agent level and `audio.mode: "transcribe"` on the Zapster channel.
|
|
91
|
+
|
|
92
|
+
```ts
|
|
93
|
+
import { defineAgent, whatsappChannel } from "@andreprado/agentkit";
|
|
94
|
+
|
|
95
|
+
export default defineAgent({
|
|
96
|
+
name: "support-agent",
|
|
97
|
+
runtime: "edge",
|
|
98
|
+
provider: { name: "openai", model: "gpt-5.4-mini" },
|
|
99
|
+
instructions: "./prompts/instructions.md",
|
|
100
|
+
transcription: {
|
|
101
|
+
provider: "openai",
|
|
102
|
+
model: "gpt-4o-mini-transcribe",
|
|
103
|
+
secret: "OPENAI_API_KEY",
|
|
104
|
+
language: "pt",
|
|
105
|
+
limits: {
|
|
106
|
+
maxDurationSeconds: 180,
|
|
107
|
+
maxBytes: 20_000_000,
|
|
108
|
+
},
|
|
109
|
+
},
|
|
110
|
+
channels: [
|
|
111
|
+
whatsappChannel({
|
|
112
|
+
name: "support-whatsapp",
|
|
113
|
+
provider: "zapster",
|
|
114
|
+
audio: {
|
|
115
|
+
mode: "transcribe",
|
|
116
|
+
},
|
|
117
|
+
}),
|
|
118
|
+
],
|
|
119
|
+
access: { mode: "public" },
|
|
120
|
+
storage: { driver: "agentkit" },
|
|
121
|
+
});
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
Processing order:
|
|
125
|
+
|
|
126
|
+
1. AgentKit validates Zapster origin headers and the optional webhook token.
|
|
127
|
+
2. Zapster audio payloads become normalized audio messages.
|
|
128
|
+
3. AgentKit records `audio_received` and enqueues a channel job before acknowledging Zapster.
|
|
129
|
+
4. The retryable channel worker downloads the media URL from a trusted Zapster HTTPS host using `ZAPSTER_API_KEY`.
|
|
130
|
+
5. AgentKit sends the audio bytes to the configured transcription provider using the user's managed secret.
|
|
131
|
+
6. The agent run receives a text message with the transcript.
|
|
132
|
+
|
|
133
|
+
Zapster payloads must include a media download URL such as `audio.downloadUrl`, `audio.url`, `audio.mediaUrl`, or the snake_case equivalents. The URL must be HTTPS and hosted by Zapster; AgentKit rejects arbitrary webhook-provided hosts before sending `ZAPSTER_API_KEY`. If Zapster sends only a media ID without a download URL, AgentKit records `channel_audio_download_unavailable` and does not create an agent run for that audio in V1.
|
|
134
|
+
|
|
53
135
|
## Setup Behavior
|
|
54
136
|
|
|
55
|
-
`agentkit channels setup support-whatsapp` prints the stable AgentKit webhook URL. Paste it into Zapster webhook settings
|
|
137
|
+
`agentkit channels setup support-whatsapp` prints the stable AgentKit webhook URL. Paste it into Zapster webhook settings.
|
|
56
138
|
|
|
57
139
|
Expected webhook URL shape:
|
|
58
140
|
|
|
59
141
|
```txt
|
|
60
|
-
https://<deploy-host>/channels/
|
|
142
|
+
https://<deploy-host>/channels/support-whatsapp/whatsapp/zapster/webhook
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
If the channel declares `ZAPSTER_WEBHOOK_TOKEN`, register the Zapster URL with the token as a query parameter:
|
|
146
|
+
|
|
147
|
+
```txt
|
|
148
|
+
https://<deploy-host>/channels/support-whatsapp/whatsapp/zapster/webhook?token=<ZAPSTER_WEBHOOK_TOKEN>
|
|
61
149
|
```
|
|
62
150
|
|
|
63
151
|
## Safety Rules
|
|
64
152
|
|
|
65
153
|
- Keep phone numbers redacted in logs by default.
|
|
66
154
|
- Do not store Zapster API keys in `agentkit.config.ts`.
|
|
67
|
-
- Inbound validation uses `X-
|
|
155
|
+
- Inbound validation uses Zapster's `X-Instance-ID`, `X-Webhook-ID`, `X-Message-ID`, `X-Attempt-Count`, and `User-Agent: Zapsterapi/...` headers.
|
|
156
|
+
- Zapster webhook headers are origin validation, not a cryptographic body signature.
|
|
157
|
+
- Use optional `ZAPSTER_WEBHOOK_TOKEN` in the webhook URL when the endpoint should require an extra secret known only to AgentKit and Zapster.
|
|
68
158
|
- Unsupported media should be logged as skipped/unsupported without creating an agent run.
|
|
159
|
+
- AgentKit handles Zapster `message.received` envelopes with event id at `id`, message text at `data.content.text`, and contact identity at `data.sender.id`.
|
|
160
|
+
- Outbound replies call `POST https://api.zapsterapi.com/v1/wa/messages` with bearer auth and a JSON body containing `recipient`, `text`, and `instance_id`.
|
|
161
|
+
- Real provider success is recorded as `provider_sent` only when Zapster returns a provider message ID.
|
|
162
|
+
- Only set `AGENTKIT_CHANNEL_SEND_DRY_RUN=1` in tests when Zapster should not receive a real message. Dry-run deliveries are recorded as `adapter_stubbed`, not sent.
|
|
69
163
|
|
|
70
164
|
## Verification
|
|
71
165
|
|
|
@@ -74,15 +168,25 @@ agentkit channels status support-whatsapp
|
|
|
74
168
|
agentkit channels test support-whatsapp --message "hello"
|
|
75
169
|
agentkit channels deliveries list support-whatsapp
|
|
76
170
|
agentkit channels deliveries show <delivery-id>
|
|
171
|
+
agentkit channels buffers list support-whatsapp
|
|
172
|
+
agentkit channels buffers flush support-whatsapp <conversation-id>
|
|
173
|
+
agentkit channels buffers clear support-whatsapp <conversation-id>
|
|
174
|
+
agentkit channels buffers retry support-whatsapp <conversation-id>
|
|
77
175
|
```
|
|
78
176
|
|
|
79
177
|
## Troubleshooting
|
|
80
178
|
|
|
81
179
|
`channel_secret_missing`:
|
|
82
|
-
Set `ZAPSTER_API_KEY` and `
|
|
180
|
+
Set `ZAPSTER_API_KEY`, `ZAPSTER_INSTANCE_ID`, and `ZAPSTER_WEBHOOK_ID` as hosted managed secrets. If the channel declares `ZAPSTER_WEBHOOK_TOKEN`, set that managed secret too.
|
|
83
181
|
|
|
84
182
|
`channel_signature_invalid`:
|
|
85
|
-
Zapster is not sending the expected webhook
|
|
183
|
+
Zapster is not sending the expected instance/webhook IDs, or the optional query token does not match.
|
|
86
184
|
|
|
87
185
|
`channel_unsupported_message_type` or skipped delivery:
|
|
88
186
|
The inbound WhatsApp event was not supported text. Inspect the delivery record for provider metadata.
|
|
187
|
+
|
|
188
|
+
`channel_audio_download_unavailable`:
|
|
189
|
+
Zapster sent an audio event without a usable media download URL, or the URL was not an HTTPS Zapster media host. Configure Zapster to include a trusted Zapster media URL in webhook payloads, or add a Zapster media lookup adapter before enabling `audio.mode: "transcribe"`.
|
|
190
|
+
|
|
191
|
+
`transcription_secret_missing`:
|
|
192
|
+
Set `OPENAI_API_KEY`, `GROQ_API_KEY`, or the custom secret named in `transcription.secret` as a managed hosted secret.
|
|
@@ -19,7 +19,6 @@ cd /tmp/agentkit-demo
|
|
|
19
19
|
|
|
20
20
|
npx @andreprado/agentkit@alpha new demo --template blank
|
|
21
21
|
cd demo
|
|
22
|
-
npm install
|
|
23
22
|
npm run typecheck
|
|
24
23
|
npm run chat -- --message "hello"
|
|
25
24
|
npm run agentkit -- conversations list
|
|
@@ -37,10 +36,22 @@ For the support template:
|
|
|
37
36
|
```sh
|
|
38
37
|
npx @andreprado/agentkit@alpha new support-demo --template support
|
|
39
38
|
cd support-demo
|
|
40
|
-
npm install
|
|
41
39
|
npm run agentkit -- tool lookup_order --input '{"orderId":"A100"}'
|
|
42
40
|
```
|
|
43
41
|
|
|
42
|
+
## Windows PowerShell
|
|
43
|
+
|
|
44
|
+
If PowerShell blocks `npm.ps1` or `npx.ps1` with `PSSecurityException`, run the same commands through the Windows command shims:
|
|
45
|
+
|
|
46
|
+
```sh
|
|
47
|
+
npx.cmd @andreprado/agentkit@alpha new demo --template blank
|
|
48
|
+
npm.cmd run typecheck
|
|
49
|
+
npm.cmd run chat -- --message "hello"
|
|
50
|
+
npm.cmd run agentkit -- inspect
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
This keeps the capsule workflow the same without changing the machine-wide PowerShell execution policy.
|
|
54
|
+
|
|
44
55
|
## Files Created Or Edited
|
|
45
56
|
|
|
46
57
|
Generated files:
|
|
@@ -86,7 +97,16 @@ Primary flow:
|
|
|
86
97
|
Develop an appointment and intake agent for an ophthalmology office.
|
|
87
98
|
```
|
|
88
99
|
|
|
89
|
-
The generated `AGENTS.md`, `AGENTKIT.md`,
|
|
100
|
+
The generated `AGENTS.md`, `AGENTKIT.md`, `CLAUDE.md`, and `skills/` pack tell the coding agent which files to edit, which task skill to load, and which verification commands to run. There is no wizard or recipe layer: the coding agent edits the capsule directly from the scaffold, contract, and owner request. The default router is `skills/agentkit-capsule/SKILL.md`; `llms-full.txt` is reserved for complete-contract checks.
|
|
101
|
+
|
|
102
|
+
After the owner gives the general idea, the coding agent should create or update the implementation contract itself:
|
|
103
|
+
|
|
104
|
+
```sh
|
|
105
|
+
npm run agentkit -- spec init --brief "Develop an appointment and intake agent for an ophthalmology office."
|
|
106
|
+
npm run agentkit -- spec check
|
|
107
|
+
```
|
|
108
|
+
|
|
109
|
+
`AGENT_SPEC.md` is an internal working contract for the coding agent. It is not a form the owner must fill before work starts.
|
|
90
110
|
|
|
91
111
|
Optional shortcut when copying a prompt into another coding agent:
|
|
92
112
|
|
|
@@ -137,6 +157,27 @@ Expected chat output:
|
|
|
137
157
|
Echo: hello
|
|
138
158
|
```
|
|
139
159
|
|
|
160
|
+
## Testing With A UI
|
|
161
|
+
|
|
162
|
+
Local UI:
|
|
163
|
+
|
|
164
|
+
```sh
|
|
165
|
+
npm run dev
|
|
166
|
+
```
|
|
167
|
+
|
|
168
|
+
Open the printed `Chat:` URL and tell the owner the exact URL.
|
|
169
|
+
|
|
170
|
+
Hosted deploy UI:
|
|
171
|
+
|
|
172
|
+
```sh
|
|
173
|
+
npm run agentkit -- deploy
|
|
174
|
+
npm run agentkit -- chat-ui --deploy
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Open the printed `Chat:` URL and tell the owner this local UI is connected to the hosted deploy.
|
|
178
|
+
|
|
179
|
+
`test/fake` is deterministic. It validates the scaffold, direct tool checks, and fake-provider evals, but it does not validate natural conversation quality. Before claiming real conversation behavior is tested, ask the owner which provider to use: OpenRouter, OpenAI, Anthropic, or another supported provider.
|
|
180
|
+
|
|
140
181
|
## Safety Rules
|
|
141
182
|
|
|
142
183
|
- Do not commit `.env`.
|
|
@@ -173,7 +214,7 @@ npm install
|
|
|
173
214
|
npm run typecheck
|
|
174
215
|
```
|
|
175
216
|
|
|
176
|
-
Make sure `tsconfig.json` has `moduleResolution: "Bundler"`.
|
|
217
|
+
`agentkit new` installs dependencies by default. Run this if the scaffold was created with `--no-install`, the install failed, or `node_modules` was deleted. Make sure `tsconfig.json` has `moduleResolution: "Bundler"`.
|
|
177
218
|
|
|
178
219
|
`No agentkit.config.ts found`:
|
|
179
220
|
|
|
@@ -0,0 +1,147 @@
|
|
|
1
|
+
# Debug A Channel
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Diagnose website, Telegram, WhatsApp, Discord, or Slack channel setup, webhook validation, dedupe, buffering, outbound sends, and delivery failures from the Agent Capsule CLI.
|
|
6
|
+
|
|
7
|
+
## When To Use It
|
|
8
|
+
|
|
9
|
+
Use this when a hosted channel is not responding, a provider is retrying messages, audio transcription is failing, or delivery status does not match the user's expectation.
|
|
10
|
+
|
|
11
|
+
## Commands
|
|
12
|
+
|
|
13
|
+
```sh
|
|
14
|
+
agentkit channels list
|
|
15
|
+
agentkit channels status <name>
|
|
16
|
+
agentkit channels doctor <name>
|
|
17
|
+
agentkit channels test <name> --message "hello"
|
|
18
|
+
agentkit channels test-audio <name> --fixture voice-note
|
|
19
|
+
agentkit transcribe smoke --provider groq
|
|
20
|
+
agentkit channels test <name> --fixture ./fixtures/provider-event.json
|
|
21
|
+
agentkit channels deliveries list <name> --since 24h
|
|
22
|
+
agentkit channels deliveries show <delivery-id>
|
|
23
|
+
agentkit channels buffers list <name>
|
|
24
|
+
agentkit channels buffers show <conversation-id>
|
|
25
|
+
agentkit channels buffers flush <conversation-id>
|
|
26
|
+
agentkit channels buffers clear <conversation-id>
|
|
27
|
+
agentkit channels buffers retry <conversation-id>
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
## Files Created Or Edited
|
|
31
|
+
|
|
32
|
+
Debugging should not require source edits. Use fixtures only when reproducing provider payload shape, and never commit provider secrets, full phone numbers, raw audio, or private client data.
|
|
33
|
+
|
|
34
|
+
## Delivery States
|
|
35
|
+
|
|
36
|
+
```txt
|
|
37
|
+
received
|
|
38
|
+
validated
|
|
39
|
+
duplicate
|
|
40
|
+
audio_received
|
|
41
|
+
audio_downloaded
|
|
42
|
+
transcribing
|
|
43
|
+
transcribed
|
|
44
|
+
buffered
|
|
45
|
+
queued
|
|
46
|
+
running
|
|
47
|
+
agent_completed
|
|
48
|
+
provider_request_built
|
|
49
|
+
provider_sent
|
|
50
|
+
adapter_stubbed
|
|
51
|
+
delivered
|
|
52
|
+
provider_failed
|
|
53
|
+
failed
|
|
54
|
+
dead_lettered
|
|
55
|
+
skipped
|
|
56
|
+
```
|
|
57
|
+
|
|
58
|
+
## Minimal Working Example
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
agentkit channels test support-telegram --message "hello"
|
|
62
|
+
agentkit channels deliveries list support-telegram --since 1h
|
|
63
|
+
agentkit channels deliveries show del_123
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
Expected `show` output includes signature status, dedupe key, queue/run state, outbound provider request ID when available, and a normalized error envelope.
|
|
67
|
+
|
|
68
|
+
## Safety Rules
|
|
69
|
+
|
|
70
|
+
- Inspect delivery IDs, hashes, statuses, and redacted metadata rather than raw provider payloads.
|
|
71
|
+
- Never paste provider secrets into fixtures or prompts.
|
|
72
|
+
- Treat `adapter_stubbed` as a dry-run send, not proof that a client received a message.
|
|
73
|
+
- Before replaying or flushing buffered conversations, confirm the channel name and conversation ID.
|
|
74
|
+
- Keep local `.env` values ignored and upload hosted production values through AgentKit secret commands.
|
|
75
|
+
|
|
76
|
+
## Troubleshooting By Error Code
|
|
77
|
+
|
|
78
|
+
`channel_not_found`:
|
|
79
|
+
The webhook URL points to an unknown or deleted channel. Stable hosted URLs use the channel name, for example `/channels/support-whatsapp/whatsapp/zapster/webhook`; old `chn_*` URLs are accepted only for compatibility.
|
|
80
|
+
|
|
81
|
+
`agentkit channels test <name>` is the official synthetic smoke. If it fails while `channels list`, `status`, or `doctor` find the channel, rerun against the current deploy state in `.agentkit/deploy.json`.
|
|
82
|
+
|
|
83
|
+
`channel_disabled`:
|
|
84
|
+
The channel was disabled. Re-add or recreate it.
|
|
85
|
+
|
|
86
|
+
`channel_secret_missing`:
|
|
87
|
+
The hosted secret metadata says a required channel secret is missing. Set it with `agentkit secret set <NAME> --from-local-env` or `agentkit secret sync --from-local`.
|
|
88
|
+
|
|
89
|
+
`channel_signature_invalid`:
|
|
90
|
+
Provider authenticity validation failed. Check the provider webhook secret, token, origin settings, or Discord public key.
|
|
91
|
+
|
|
92
|
+
`channel_payload_invalid`:
|
|
93
|
+
The provider payload is malformed or does not match the route provider.
|
|
94
|
+
|
|
95
|
+
Discord endpoint validation fails in the Developer Portal:
|
|
96
|
+
Confirm `DISCORD_PUBLIC_KEY` is set from the application's public key and the webhook URL is `/channels/<name>/discord/discord/webhook`. AgentKit must validate Discord's signature headers before returning the `PING` PONG.
|
|
97
|
+
|
|
98
|
+
Discord bot does not answer normal server messages:
|
|
99
|
+
Confirm the channel was created with `--mode bot`, `DISCORD_BOT_TOKEN` is set, Message Content Intent is enabled in the Discord Developer Portal, the app is installed into the server, and the bot has `View Channel`, `Read Message History`, and `Send Messages` permissions for the channel.
|
|
100
|
+
|
|
101
|
+
`audio_received`:
|
|
102
|
+
The webhook contained a supported audio message and the channel is entering the audio handling path.
|
|
103
|
+
|
|
104
|
+
`audio_downloaded`:
|
|
105
|
+
The retryable channel worker downloaded the provider media file into memory. The delivery metadata should include only redacted size, MIME, and provider IDs.
|
|
106
|
+
|
|
107
|
+
`transcribing`:
|
|
108
|
+
AgentKit is calling the configured transcription provider with the user's managed transcription secret.
|
|
109
|
+
|
|
110
|
+
`transcribed`:
|
|
111
|
+
Transcription succeeded and the queued agent message contains transcript text instead of raw audio.
|
|
112
|
+
|
|
113
|
+
`channel_event_duplicate` or `duplicate`:
|
|
114
|
+
The provider retried an already-processed event. No second agent run should be created.
|
|
115
|
+
|
|
116
|
+
`buffered`:
|
|
117
|
+
The message is accepted and waiting inside a per-conversation channel buffer. It should move to `queued` after the quiet window, max wait, max message count, or max character count.
|
|
118
|
+
|
|
119
|
+
`adapter_stubbed`:
|
|
120
|
+
The adapter built the outbound request in explicit dry-run mode. The provider was not called.
|
|
121
|
+
|
|
122
|
+
`provider_sent`:
|
|
123
|
+
The provider API accepted the outbound request and returned a provider message ID.
|
|
124
|
+
|
|
125
|
+
`channel_limit_exceeded`:
|
|
126
|
+
Backpressure skipped the message before queueing.
|
|
127
|
+
|
|
128
|
+
`channel_audio_download_unavailable`:
|
|
129
|
+
The channel provider reported audio but did not include enough metadata for AgentKit to download it.
|
|
130
|
+
|
|
131
|
+
`transcription_secret_missing`:
|
|
132
|
+
The transcription provider secret declared in `agentkit inspect` is not set as a managed hosted secret.
|
|
133
|
+
|
|
134
|
+
`transcription_audio_too_large` or `transcription_audio_too_long`:
|
|
135
|
+
The audio exceeded `transcription.limits` or the channel-level `audio.limits`.
|
|
136
|
+
|
|
137
|
+
`transcription_audio_format_unsupported`:
|
|
138
|
+
The configured transcription provider does not accept this audio MIME type or file extension.
|
|
139
|
+
|
|
140
|
+
`transcription_provider_unavailable`:
|
|
141
|
+
The transcription provider returned a retryable error, usually HTTP 429 or a server-side failure.
|
|
142
|
+
|
|
143
|
+
`channel_provider_unavailable`:
|
|
144
|
+
The agent run or provider send path failed with a retryable provider condition.
|
|
145
|
+
|
|
146
|
+
`channel_delivery_dead_lettered`:
|
|
147
|
+
The queue exhausted retries. Inspect the delivery and replay manually only after fixing the cause.
|
|
@@ -0,0 +1,151 @@
|
|
|
1
|
+
# Improve From Production
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Pull hosted or local conversation evidence into the Agent Capsule, turn it into regression evals, let the local coding agent patch the capsule, and replay before deploying again.
|
|
6
|
+
|
|
7
|
+
## When To Use This
|
|
8
|
+
|
|
9
|
+
Use this when a deployed agent gave a wrong answer, failed a tool call, mishandled a channel message, or needs production behavior converted into eval coverage.
|
|
10
|
+
|
|
11
|
+
AgentKit Cloud only exports redacted evidence. The local coding agent owns source edits, evals, replay, and deploy.
|
|
12
|
+
|
|
13
|
+
## Commands
|
|
14
|
+
|
|
15
|
+
Collect evidence from the last hosted deploy:
|
|
16
|
+
|
|
17
|
+
```sh
|
|
18
|
+
agentkit improve collect --deploy --since 24h
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
When the CLI is logged in to AgentKit Cloud, this command first asks the control plane for deploy evidence such as failed channel deliveries, deploy errors, and conversation IDs that need review. It then reads replayable hosted conversation traces from the deployed runtime with the deploy access token. If Cloud auth is not available, it still collects hosted conversations directly from the deploy URL.
|
|
22
|
+
|
|
23
|
+
Hosted conversation reads require a deploy access token even when the chat endpoint is public. `agentkit deploy` normally writes `.agentkit/chat-access-token.json`; refresh it with:
|
|
24
|
+
|
|
25
|
+
```sh
|
|
26
|
+
agentkit access token create agentkit-chat-ui --out .agentkit/chat-access-token.json
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Collect one hosted conversation:
|
|
30
|
+
|
|
31
|
+
```sh
|
|
32
|
+
agentkit improve collect --deploy --conversation-id <conversation-id>
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
Collect local conversations instead:
|
|
36
|
+
|
|
37
|
+
```sh
|
|
38
|
+
agentkit improve collect --since 7d
|
|
39
|
+
```
|
|
40
|
+
|
|
41
|
+
Generate regression evals from the collected bundle:
|
|
42
|
+
|
|
43
|
+
```sh
|
|
44
|
+
agentkit improve evals .agentkit/improve/<run>
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
Replay the bundle against the local capsule:
|
|
48
|
+
|
|
49
|
+
```sh
|
|
50
|
+
agentkit replay .agentkit/improve/<run> --against local
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## Files Created Or Edited
|
|
54
|
+
|
|
55
|
+
Evidence bundle, ignored local state:
|
|
56
|
+
|
|
57
|
+
```txt
|
|
58
|
+
.agentkit/improve/<run>/
|
|
59
|
+
bundle.json
|
|
60
|
+
report.json
|
|
61
|
+
traces/
|
|
62
|
+
```
|
|
63
|
+
|
|
64
|
+
Generated regression evals, committed source:
|
|
65
|
+
|
|
66
|
+
```txt
|
|
67
|
+
evals/regressions/
|
|
68
|
+
improve-<conversation>.eval.ts
|
|
69
|
+
```
|
|
70
|
+
|
|
71
|
+
The local coding agent may then edit:
|
|
72
|
+
|
|
73
|
+
```txt
|
|
74
|
+
prompts/instructions.md
|
|
75
|
+
agentkit.config.ts
|
|
76
|
+
tools/
|
|
77
|
+
knowledge/
|
|
78
|
+
evals/
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
Do not edit `.agentkit/improve/<run>/bundle.json` by hand.
|
|
82
|
+
|
|
83
|
+
## Workflow
|
|
84
|
+
|
|
85
|
+
```sh
|
|
86
|
+
agentkit improve collect --deploy --since 24h
|
|
87
|
+
agentkit improve evals .agentkit/improve/<run>
|
|
88
|
+
agentkit replay .agentkit/improve/<run> --against local
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
Then let the local coding agent inspect `report.json`, the generated eval files, prompts, tools, and Knowledge sources. After edits:
|
|
92
|
+
|
|
93
|
+
```sh
|
|
94
|
+
npm run typecheck
|
|
95
|
+
npm run agentkit -- inspect
|
|
96
|
+
npm run eval
|
|
97
|
+
agentkit replay .agentkit/improve/<run> --against local
|
|
98
|
+
agentkit deploy --smoke "hello"
|
|
99
|
+
```
|
|
100
|
+
|
|
101
|
+
## Safety Rules
|
|
102
|
+
|
|
103
|
+
- Do not paste secrets into evals, prompts, Knowledge files, or reports.
|
|
104
|
+
- Keep `.agentkit/improve/` out of commits.
|
|
105
|
+
- Review generated eval assertions before committing them. AgentKit redacts common email, phone, bearer token, and key patterns in generated eval text, but the local coding agent must still remove or generalize domain-specific client PII.
|
|
106
|
+
- For write, delete, payment, email, or customer-system tools, branch inside the tool on `ctx.runtime.environment === "eval"` and return deterministic non-destructive output.
|
|
107
|
+
- Treat hosted traces as customer evidence.
|
|
108
|
+
- If replay uses a real provider instead of `test/fake`, tell the owner because it may cost money and may be nondeterministic.
|
|
109
|
+
|
|
110
|
+
## Verification
|
|
111
|
+
|
|
112
|
+
```sh
|
|
113
|
+
npm run typecheck
|
|
114
|
+
npm run agentkit -- inspect
|
|
115
|
+
npm run eval
|
|
116
|
+
agentkit replay .agentkit/improve/<run> --against local
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Expected:
|
|
120
|
+
|
|
121
|
+
- `improve collect` writes a bundle and report under `.agentkit/improve/`.
|
|
122
|
+
- Hosted collection includes a redacted `evidence` summary in `bundle.json` and `report.json` when AgentKit Cloud evidence export is available.
|
|
123
|
+
- `improve evals` writes eval files under `evals/regressions/`.
|
|
124
|
+
- `replay` reports passed, failed, and skipped traces.
|
|
125
|
+
- No production secret values appear in generated files.
|
|
126
|
+
|
|
127
|
+
## Troubleshooting
|
|
128
|
+
|
|
129
|
+
`No .agentkit/deploy.json found`:
|
|
130
|
+
|
|
131
|
+
Run `agentkit deploy` first, or collect local evidence without `--deploy`.
|
|
132
|
+
|
|
133
|
+
`deploy_conversation_request_failed`:
|
|
134
|
+
|
|
135
|
+
Refresh the deploy chat token with `agentkit access token create agentkit-chat-ui --out .agentkit/chat-access-token.json`, then retry.
|
|
136
|
+
|
|
137
|
+
`improve_evidence_store_not_configured`:
|
|
138
|
+
|
|
139
|
+
The Cloud API does not expose deploy evidence export yet. The CLI falls back to hosted conversation trace collection when possible.
|
|
140
|
+
|
|
141
|
+
`conversation_access_not_configured`:
|
|
142
|
+
|
|
143
|
+
The hosted runtime is not configured with a deploy access-token gate for conversation reads. Redeploy through AgentKit Cloud so the runtime gate is injected, or configure an explicit deploy/private token for local hosted testing.
|
|
144
|
+
|
|
145
|
+
`trace has no user turns`:
|
|
146
|
+
|
|
147
|
+
The trace cannot become a useful conversation eval. Keep the report for diagnosis, but do not commit an empty eval.
|
|
148
|
+
|
|
149
|
+
Generated eval is too strict:
|
|
150
|
+
|
|
151
|
+
Edit the eval to assert the important behavior, such as tool input, safety wording, or knowledge source usage, instead of exact prose.
|
|
@@ -19,20 +19,31 @@ Use this before asking a coding agent to make a capsule deployable, or before te
|
|
|
19
19
|
From the capsule root:
|
|
20
20
|
|
|
21
21
|
```sh
|
|
22
|
-
npm install
|
|
23
22
|
npm run typecheck
|
|
24
23
|
npm run agentkit -- inspect
|
|
25
24
|
npm run agentkit -- db migrate
|
|
26
25
|
npm run chat -- --message "hello"
|
|
27
26
|
```
|
|
28
27
|
|
|
29
|
-
All scaffold, local dev, chat, eval, inspect, database, and build commands are token-free.
|
|
28
|
+
All scaffold, local dev, chat, eval, inspect, database, and build commands are token-free. Hosted production deploy and hosted AgentKit Cloud changes require an AgentKit Cloud account token with hosted deploy access.
|
|
29
|
+
|
|
30
|
+
If the user does not have a token yet, start checkout from the CLI, finish Stripe Checkout in the browser, then claim the one-time checkout intent:
|
|
31
|
+
|
|
32
|
+
```sh
|
|
33
|
+
npm run agentkit -- billing checkout --slots 1 --email user@example.com
|
|
34
|
+
npm run agentkit -- billing claim billint_... --secret bsec_...
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
The claim command stores the returned `agk_user_...` token in the local AgentKit Cloud auth file. Treat the `bsec_...` checkout secret like a password; it exists only to claim the first token after checkout.
|
|
38
|
+
|
|
39
|
+
If the user already has a token:
|
|
30
40
|
|
|
31
41
|
```sh
|
|
32
42
|
npm run agentkit -- login --token agk_user_...
|
|
33
43
|
npm run agentkit -- deploy doctor
|
|
34
|
-
npm run agentkit -- secret set OPENAI_API_KEY
|
|
35
|
-
npm run agentkit -- deploy
|
|
44
|
+
npm run agentkit -- secret set OPENAI_API_KEY --from-local-env
|
|
45
|
+
npm run agentkit -- deploy --smoke "hello"
|
|
46
|
+
npm run agentkit -- chat-ui --deploy
|
|
36
47
|
```
|
|
37
48
|
|
|
38
49
|
If the capsule uses OpenAI locally, put the user’s local key in `.env`:
|
|
@@ -42,7 +53,15 @@ OPENAI_API_KEY=sk-...
|
|
|
42
53
|
```
|
|
43
54
|
|
|
44
55
|
`.env` is local-only. Hosted deploy secrets are handled by AgentKit outside the capsule.
|
|
45
|
-
|
|
56
|
+
Hosted deploys require an account with either `cloudflare_deploy_alpha` or purchased/manual deploy slots. Deploy slots belong to the account, not to a specific token string.
|
|
57
|
+
|
|
58
|
+
To generate or switch account tokens after login:
|
|
59
|
+
|
|
60
|
+
```sh
|
|
61
|
+
npm run agentkit -- account token create new-laptop --use
|
|
62
|
+
npm run agentkit -- account token list
|
|
63
|
+
npm run agentkit -- account token revoke apitok_...
|
|
64
|
+
```
|
|
46
65
|
|
|
47
66
|
## What The Agent Should Edit
|
|
48
67
|
|
|
@@ -126,12 +145,19 @@ For a final end-to-end test, run:
|
|
|
126
145
|
|
|
127
146
|
```sh
|
|
128
147
|
npm run agentkit -- login --token agk_user_...
|
|
148
|
+
npm run agentkit -- account token list
|
|
149
|
+
npm run agentkit -- secret sync --from-local
|
|
129
150
|
npm run agentkit -- secret list
|
|
130
|
-
npm run agentkit -- deploy
|
|
151
|
+
npm run agentkit -- deploy --smoke "hello"
|
|
131
152
|
npm run agentkit -- deploy status
|
|
153
|
+
npm run agentkit -- chat-ui --deploy
|
|
132
154
|
npm run agentkit -- access token list
|
|
133
155
|
```
|
|
134
156
|
|
|
157
|
+
For hosted deploys, `npm run agentkit -- deploy` writes the local chat/UI access token to `.agentkit/chat-access-token.json`. Use `npm run agentkit -- deploy --smoke "hello"` for the official hosted chat smoke, and use `npm run agentkit -- chat-ui --deploy` for hosted UI testing. The hosted Chat UI shows the conversation id, tool calls, tool errors, and a new-conversation control; use `npm run agentkit -- conversations trace <conversation-id> --deploy` to pull the hosted trace from the last deploy. Use `npm run agentkit -- access token create <name> --out <path>` only for additional clients.
|
|
158
|
+
|
|
159
|
+
When the UI is running, open the printed `Chat:` URL and tell the owner the exact URL. If the capsule is still on `test/fake`, say the UI was tested only with the deterministic fake provider.
|
|
160
|
+
|
|
135
161
|
## Readiness Checklist
|
|
136
162
|
|
|
137
163
|
```txt
|
|
@@ -141,6 +167,7 @@ All required secret names are declared
|
|
|
141
167
|
Prompt path exists
|
|
142
168
|
Tools have input schemas
|
|
143
169
|
Dangerous tools have permissions
|
|
170
|
+
Managed Composio toolkit auth configs are available in AgentKit Cloud when configured
|
|
144
171
|
Provider model is supported by AgentKit
|
|
145
172
|
schema.sql is idempotent
|
|
146
173
|
README/AGENTS/CLAUDE match the capsule
|
|
@@ -166,10 +193,16 @@ echo 'OPENAI_API_KEY=sk-...' >> .env
|
|
|
166
193
|
|
|
167
194
|
Missing hosted secret:
|
|
168
195
|
|
|
169
|
-
Set the secret in AgentKit Cloud
|
|
196
|
+
Set the secret in AgentKit Cloud:
|
|
197
|
+
|
|
198
|
+
```sh
|
|
199
|
+
npm run agentkit -- secret set OPENAI_API_KEY --from-local-env
|
|
200
|
+
```
|
|
201
|
+
|
|
202
|
+
To sync all declared user-managed secrets present in local `.env`:
|
|
170
203
|
|
|
171
204
|
```sh
|
|
172
|
-
npm run agentkit -- secret
|
|
205
|
+
npm run agentkit -- secret sync --from-local
|
|
173
206
|
```
|
|
174
207
|
|
|
175
208
|
Do not write hosted secret values into the capsule.
|
|
@@ -180,18 +213,15 @@ Before a hosted production deploy, run:
|
|
|
180
213
|
npm run agentkit -- deploy doctor
|
|
181
214
|
```
|
|
182
215
|
|
|
183
|
-
The doctor checks AgentKit Cloud login,
|
|
216
|
+
The doctor checks AgentKit Cloud login, Node version, hosted deploy entitlement, online deploy capacity, declared hosted secrets, local `.env` names that have not been uploaded with `npm run agentkit -- secret set`, and private-access runtime token handling. `agentkit deploy` runs the same readiness check automatically before building and uploading the artifact.
|
|
217
|
+
|
|
218
|
+
If the capsule configures `composioManaged({...})`, the doctor also checks the paid `managed_composio` entitlement and AgentKit Cloud toolkit readiness. `COMPOSIO_API_KEY` and toolkit auth config resolution are AgentKit-managed in this path; the user must not set them with `agentkit secret set`.
|
|
184
219
|
|
|
185
220
|
`alpha_access_required`:
|
|
186
221
|
|
|
187
|
-
Log in with an
|
|
222
|
+
Log in with an account that has hosted deploy access. If the user needs to buy slots first:
|
|
188
223
|
|
|
189
224
|
```sh
|
|
190
|
-
npm run agentkit --
|
|
225
|
+
npm run agentkit -- billing checkout --slots 1 --email user@example.com
|
|
226
|
+
npm run agentkit -- billing claim billint_... --secret bsec_...
|
|
191
227
|
```
|
|
192
|
-
|
|
193
|
-
## Operator Notes
|
|
194
|
-
|
|
195
|
-
This section is for AgentKit maintainers, not for capsule-building agents.
|
|
196
|
-
|
|
197
|
-
`agentkit deploy` sends the capsule artifact to the configured AgentKit Cloud API. The control plane owns infrastructure selection, provisioning, managed secrets, and public URL creation. Use `AGENTKIT_CLOUD_API_URL` only when testing a non-default control plane.
|