@andreprado/agentkit 0.1.0-alpha.17 → 0.1.0-alpha.19

package/docs/guides/debug-channel.md

@@ -0,0 +1,147 @@
+# Debug A Channel
+
+## Goal
+
+Diagnose website, Telegram, WhatsApp, or Discord channel setup, webhook validation, dedupe, buffering, outbound sends, and delivery failures from the Agent Capsule CLI.
+
+## When To Use It
+
+Use this when a hosted channel is not responding, a provider is retrying messages, audio transcription is failing, or delivery status does not match the user's expectation.
+
+## Commands
+
+```sh
+agentkit channels list
+agentkit channels status <name>
+agentkit channels doctor <name>
+agentkit channels test <name> --message "hello"
+agentkit channels test-audio <name> --fixture voice-note
+agentkit transcribe smoke --provider groq
+agentkit channels test <name> --fixture ./fixtures/provider-event.json
+agentkit channels deliveries list <name> --since 24h
+agentkit channels deliveries show <delivery-id>
+agentkit channels buffers list <name>
+agentkit channels buffers show <conversation-id>
+agentkit channels buffers flush <conversation-id>
+agentkit channels buffers clear <conversation-id>
+agentkit channels buffers retry <conversation-id>
+```
+
+## Files Created Or Edited
+
+Debugging should not require source edits. Use fixtures only when reproducing provider payload shape, and never commit provider secrets, full phone numbers, raw audio, or private client data.
+
+## Delivery States
+
+```txt
+received
+validated
+duplicate
+audio_received
+audio_downloaded
+transcribing
+transcribed
+buffered
+queued
+running
+agent_completed
+provider_request_built
+provider_sent
+adapter_stubbed
+delivered
+provider_failed
+failed
+dead_lettered
+skipped
+```
+
+## Minimal Working Example
+
+```sh
+agentkit channels test support-telegram --message "hello"
+agentkit channels deliveries list support-telegram --since 1h
+agentkit channels deliveries show del_123
+```
+
+Expected `show` output includes signature status, dedupe key, queue/run state, outbound provider request ID when available, and a normalized error envelope.
+
+## Safety Rules
+
+- Inspect delivery IDs, hashes, statuses, and redacted metadata rather than raw provider payloads.
+- Never paste provider secrets into fixtures or prompts.
+- Treat `adapter_stubbed` as a dry-run send, not proof that a client received a message.
+- Before replaying or flushing buffered conversations, confirm the channel name and conversation ID.
+- Keep local `.env` values ignored and upload hosted production values through AgentKit secret commands.
+
+## Troubleshooting By Error Code
+
+`channel_not_found`:
+The webhook URL points to an unknown or deleted channel. Stable hosted URLs use the channel name, for example `/channels/support-whatsapp/whatsapp/zapster/webhook`; old `chn_*` URLs are accepted only for compatibility.
+
+`agentkit channels test <name>` is the official synthetic smoke. If it fails while `channels list`, `status`, or `doctor` find the channel, rerun against the current deploy state in `.agentkit/deploy.json`.
+
+`channel_disabled`:
+The channel was disabled. Re-add or recreate it.
+
+`channel_secret_missing`:
+The hosted secret metadata says a required channel secret is missing. Set it with `agentkit secret set <NAME> --from-local-env` or `agentkit secret sync --from-local`.
+
+`channel_signature_invalid`:
+Provider authenticity validation failed. Check the provider webhook secret, token, origin settings, or Discord public key.
+
+`channel_payload_invalid`:
+The provider payload is malformed or does not match the route provider.
+
+Discord endpoint validation fails in the Developer Portal:
+Confirm `DISCORD_PUBLIC_KEY` is set from the application's public key and the webhook URL is `/channels/<name>/discord/discord/webhook`. AgentKit must validate Discord's signature headers before returning the `PING` PONG.
+
+Discord bot does not answer normal server messages:
+Confirm the channel was created with `--mode bot`, `DISCORD_BOT_TOKEN` is set, Message Content Intent is enabled in the Discord Developer Portal, the app is installed into the server, and the bot has `View Channel`, `Read Message History`, and `Send Messages` permissions for the channel.
+
+`audio_received`:
+The webhook contained a supported audio message and the channel is entering the audio handling path.
+
+`audio_downloaded`:
+The retryable channel worker downloaded the provider media file into memory. The delivery metadata should include only redacted size, MIME, and provider IDs.
+
+`transcribing`:
+AgentKit is calling the configured transcription provider with the user's managed transcription secret.
+
+`transcribed`:
+Transcription succeeded and the queued agent message contains transcript text instead of raw audio.
+
+`channel_event_duplicate` or `duplicate`:
+The provider retried an already-processed event. No second agent run should be created.
+
+`buffered`:
+The message is accepted and waiting inside a per-conversation channel buffer. It should move to `queued` after the quiet window, max wait, max message count, or max character count.
+
+`adapter_stubbed`:
+The adapter built the outbound request in explicit dry-run mode. The provider was not called.
+
+`provider_sent`:
+The provider API accepted the outbound request and returned a provider message ID.
+
+`channel_limit_exceeded`:
+Backpressure skipped the message before queueing.
+
+`channel_audio_download_unavailable`:
+The channel provider reported audio but did not include enough metadata for AgentKit to download it.
+
+`transcription_secret_missing`:
+The transcription provider secret declared in `agentkit inspect` is not set as a managed hosted secret.
+
+`transcription_audio_too_large` or `transcription_audio_too_long`:
+The audio exceeded `transcription.limits` or the channel-level `audio.limits`.
+
+`transcription_audio_format_unsupported`:
+The configured transcription provider does not accept this audio MIME type or file extension.
+
+`transcription_provider_unavailable`:
+The transcription provider returned a retryable error, usually HTTP 429 or a server-side failure.
+
+`channel_provider_unavailable`:
+The agent run or provider send path failed with a retryable provider condition.
+
+`channel_delivery_dead_lettered`:
+The queue exhausted retries. Inspect the delivery and replay manually only after fixing the cause.

package/docs/guides/improve-from-production.md

@@ -0,0 +1,151 @@
+# Improve From Production
+
+## Goal
+
+Pull hosted or local conversation evidence into the Agent Capsule, turn it into regression evals, let the local coding agent patch the capsule, and replay before deploying again.
+
+## When To Use This
+
+Use this when a deployed agent gave a wrong answer, failed a tool call, mishandled a channel message, or needs production behavior converted into eval coverage.
+
+AgentKit Cloud only exports redacted evidence. The local coding agent owns source edits, evals, replay, and deploy.
+
+## Commands
+
+Collect evidence from the last hosted deploy:
+
+```sh
+agentkit improve collect --deploy --since 24h
+```
+
+When the CLI is logged in to AgentKit Cloud, this command first asks the control plane for deploy evidence such as failed channel deliveries, deploy errors, and conversation IDs that need review. It then reads replayable hosted conversation traces from the deployed runtime with the deploy access token. If Cloud auth is not available, it still collects hosted conversations directly from the deploy URL.
+
+Hosted conversation reads require a deploy access token even when the chat endpoint is public. `agentkit deploy` normally writes `.agentkit/chat-access-token.json`; refresh it with:
+
+```sh
+agentkit access token create agentkit-chat-ui --out .agentkit/chat-access-token.json
+```
+
+Collect one hosted conversation:
+
+```sh
+agentkit improve collect --deploy --conversation-id <conversation-id>
+```
+
+Collect local conversations instead:
+
+```sh
+agentkit improve collect --since 7d
+```
+
+Generate regression evals from the collected bundle:
+
+```sh
+agentkit improve evals .agentkit/improve/<run>
+```
+
+Replay the bundle against the local capsule:
+
+```sh
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+## Files Created Or Edited
+
+Evidence bundle, ignored local state:
+
+```txt
+.agentkit/improve/<run>/
+  bundle.json
+  report.json
+  traces/
+```
+
+Generated regression evals, committed source:
+
+```txt
+evals/regressions/
+  improve-<conversation>.eval.ts
+```
+
+The local coding agent may then edit:
+
+```txt
+prompts/instructions.md
+agentkit.config.ts
+tools/
+knowledge/
+evals/
+```
+
+Do not edit `.agentkit/improve/<run>/bundle.json` by hand.
+
+## Workflow
+
+```sh
+agentkit improve collect --deploy --since 24h
+agentkit improve evals .agentkit/improve/<run>
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Then let the local coding agent inspect `report.json`, the generated eval files, prompts, tools, and Knowledge sources. After edits:
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+agentkit deploy --smoke "hello"
+```
+
+## Safety Rules
+
+- Do not paste secrets into evals, prompts, Knowledge files, or reports.
+- Keep `.agentkit/improve/` out of commits.
+- Review generated eval assertions before committing them. AgentKit redacts common email, phone, bearer token, and key patterns in generated eval text, but the local coding agent must still remove or generalize domain-specific client PII.
+- For write, delete, payment, email, or customer-system tools, branch inside the tool on `ctx.runtime.environment === "eval"` and return deterministic non-destructive output.
+- Treat hosted traces as customer evidence.
+- If replay uses a real provider instead of `test/fake`, tell the owner because it may cost money and may be nondeterministic.
+
+## Verification
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Expected:
+
+- `improve collect` writes a bundle and report under `.agentkit/improve/`.
+- Hosted collection includes a redacted `evidence` summary in `bundle.json` and `report.json` when AgentKit Cloud evidence export is available.
+- `improve evals` writes eval files under `evals/regressions/`.
+- `replay` reports passed, failed, and skipped traces.
+- No production secret values appear in generated files.
+
+## Troubleshooting
+
+`No .agentkit/deploy.json found`:
+
+Run `agentkit deploy` first, or collect local evidence without `--deploy`.
+
+`deploy_conversation_request_failed`:
+
+Refresh the deploy chat token with `agentkit access token create agentkit-chat-ui --out .agentkit/chat-access-token.json`, then retry.
+
+`improve_evidence_store_not_configured`:
+
+The Cloud API does not expose deploy evidence export yet. The CLI falls back to hosted conversation trace collection when possible.
+
+`conversation_access_not_configured`:
+
+The hosted runtime is not configured with a deploy access-token gate for conversation reads. Redeploy through AgentKit Cloud so the runtime gate is injected, or configure an explicit deploy/private token for local hosted testing.
+
+`trace has no user turns`:
+
+The trace cannot become a useful conversation eval. Keep the report for diagnosis, but do not commit an empty eval.
+
+Generated eval is too strict:
+
+Edit the eval to assert the important behavior, such as tool input, safety wording, or knowledge source usage, instead of exact prose.

package/docs/guides/prepare-deploy.md

@@ -25,7 +25,18 @@ npm run agentkit -- db migrate
 npm run chat -- --message "hello"
 ```
 
-All scaffold, local dev, chat, eval, inspect, database, and build commands are token-free. Only hosted production deploy and hosted control-plane mutations require an invited AgentKit Cloud token:
+All scaffold, local dev, chat, eval, inspect, database, and build commands are token-free. Hosted production deploy and hosted AgentKit Cloud changes require an AgentKit Cloud account token with hosted deploy access.
+
+If the user does not have a token yet, start checkout from the CLI, finish Stripe Checkout in the browser, then claim the one-time checkout intent:
+
+```sh
+npm run agentkit -- billing checkout --slots 1 --email user@example.com
+npm run agentkit -- billing claim billint_... --secret bsec_...
+```
+
+The claim command stores the returned `agk_user_...` token in the local AgentKit Cloud auth file. Treat the `bsec_...` checkout secret like a password; it exists only to claim the first token after checkout.
+
+If the user already has a token:
 
 ```sh
 npm run agentkit -- login --token agk_user_...
@@ -42,7 +53,15 @@ OPENAI_API_KEY=sk-...
 ```
 
 `.env` is local-only. Hosted deploy secrets are handled by AgentKit outside the capsule.
-Closed-alpha hosted deploys require an invited account token with `cloudflare_deploy_alpha`.
+Hosted deploys require an account with either `cloudflare_deploy_alpha` or purchased/manual deploy slots. Deploy slots belong to the account, not to a specific token string.
+
+To generate or switch account tokens after login:
+
+```sh
+npm run agentkit -- account token create new-laptop --use
+npm run agentkit -- account token list
+npm run agentkit -- account token revoke apitok_...
+```
 
 ## What The Agent Should Edit
 
@@ -126,6 +145,7 @@ For a final end-to-end test, run:
 
 ```sh
 npm run agentkit -- login --token agk_user_...
+npm run agentkit -- account token list
 npm run agentkit -- secret sync --from-local
 npm run agentkit -- secret list
 npm run agentkit -- deploy --smoke "hello"
@@ -134,7 +154,7 @@ npm run agentkit -- chat-ui --deploy
 npm run agentkit -- access token list
 ```
 
-For private hosted deploys, `npm run agentkit -- deploy` writes the local chat/UI access token to `.agentkit/chat-access-token.json`. Use `npm run agentkit -- deploy --smoke "hello"` for the official hosted chat smoke, and use `npm run agentkit -- chat-ui --deploy` for hosted UI testing. Use `npm run agentkit -- access token create <name> --out <path>` only for additional clients.
+For hosted deploys, `npm run agentkit -- deploy` writes the local chat/UI access token to `.agentkit/chat-access-token.json`. Use `npm run agentkit -- deploy --smoke "hello"` for the official hosted chat smoke, and use `npm run agentkit -- chat-ui --deploy` for hosted UI testing. The hosted Chat UI shows the conversation id, tool calls, tool errors, and a new-conversation control; use `npm run agentkit -- conversations trace <conversation-id> --deploy` to pull the hosted trace from the last deploy. Use `npm run agentkit -- access token create <name> --out <path>` only for additional clients.
 
 When the UI is running, open the printed `Chat:` URL and tell the owner the exact URL. If the capsule is still on `test/fake`, say the UI was tested only with the deterministic fake provider.
 
@@ -147,6 +167,7 @@ All required secret names are declared
 Prompt path exists
 Tools have input schemas
 Dangerous tools have permissions
+Managed Composio toolkit auth configs are available in AgentKit Cloud when configured
 Provider model is supported by AgentKit
 schema.sql is idempotent
 README/AGENTS/CLAUDE match the capsule
@@ -172,7 +193,7 @@ echo 'OPENAI_API_KEY=sk-...' >> .env
 
 Missing hosted secret:
 
-Set the secret in AgentKit Cloud or the configured control plane:
+Set the secret in AgentKit Cloud:
 
 ```sh
 npm run agentkit -- secret set OPENAI_API_KEY --from-local-env
@@ -192,20 +213,15 @@ Before a hosted production deploy, run:
 npm run agentkit -- deploy doctor
 ```
 
-The doctor checks AgentKit Cloud login, Node version, alpha deploy entitlement, online deploy capacity, declared hosted secrets, local `.env` names that have not been uploaded with `npm run agentkit -- secret set`, and private-access runtime token handling. `agentkit deploy` runs the same readiness check automatically before building and uploading the artifact.
+The doctor checks AgentKit Cloud login, Node version, hosted deploy entitlement, online deploy capacity, declared hosted secrets, local `.env` names that have not been uploaded with `npm run agentkit -- secret set`, and private-access runtime token handling. `agentkit deploy` runs the same readiness check automatically before building and uploading the artifact.
 
-If the capsule configures `composioManaged({...})`, the doctor also checks the paid `managed_composio` entitlement. `COMPOSIO_API_KEY` is AgentKit-managed in this path; the user must not set it with `agentkit secret set`. Operators configure `AGENTKIT_SECRET_COMPOSIO_API_KEY` and per-toolkit `AGENTKIT_COMPOSIO_AUTH_CONFIG_<TOOLKIT>` values on the control plane.
+If the capsule configures `composioManaged({...})`, the doctor also checks the paid `managed_composio` entitlement and AgentKit Cloud toolkit readiness. `COMPOSIO_API_KEY` and toolkit auth config resolution are AgentKit-managed in this path; the user must not set them with `agentkit secret set`.
 
 `alpha_access_required`:
 
-Log in with an invited alpha account:
+Log in with an account that has hosted deploy access. If the user needs to buy slots first:
 
 ```sh
-npm run agentkit -- login --token agk_user_...
+npm run agentkit -- billing checkout --slots 1 --email user@example.com
+npm run agentkit -- billing claim billint_... --secret bsec_...
 ```
-
-## Operator Notes
-
-This section is for AgentKit maintainers, not for capsule-building agents.
-
-`agentkit deploy` sends the capsule artifact to the configured AgentKit Cloud API. The control plane owns infrastructure selection, provisioning, managed secrets, and public URL creation. Use `AGENTKIT_CLOUD_API_URL` only when testing a non-default control plane.

package/docs/guides/replay-production-traces.md

@@ -0,0 +1,72 @@
+# Replay Production Traces
+
+## Goal
+
+Run collected production or local traces against the local Agent Capsule before redeploying a fix.
+
+## When To Use This
+
+Use this after `agentkit improve collect` and before `agentkit deploy` whenever prompts, tools, Knowledge, provider config, or channel behavior changed because of production evidence.
+
+## Commands
+
+```sh
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Run the full local eval suite too:
+
+```sh
+npm run eval
+```
+
+## What Replay Checks
+
+Replay sends each collected user turn through the local capsule in eval mode:
+
+```txt
+ctx.runtime.environment === "eval"
+ctx.runtime.invocation === "eval"
+```
+
+This proves the current capsule can process the production turns without runtime errors. Generated eval files under `evals/regressions/` add committed behavior assertions.
+
+## Write Tool Safety
+
+Replay still runs the registered capsule tools. Any tool that can write externally must guard eval mode:
+
+```ts
+if (ctx.runtime.environment === "eval") {
+  return { sent: false, evalFixture: true };
+}
+```
+
+Do not depend on prompt wording alone to prevent side effects.
+
+## Deploy Gate
+
+Before redeploying a production fix:
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+agentkit deploy --smoke "hello"
+```
+
+If replay fails, inspect the failed trace id in `.agentkit/improve/<run>/traces/`, patch the capsule, and rerun replay.
+
+## Troubleshooting
+
+`Replay failed`:
+
+Read the printed error and the matching trace file. Common causes are missing local secrets, unsafe tools that do not branch on eval mode, stale Knowledge sources, or provider differences.
+
+`Replay skipped`:
+
+The trace had no user message. It may still help diagnose delivery or deploy state, but it cannot be replayed as a conversation.
+
+`provider_model_unsupported`:
+
+The local provider config does not match the replay environment. Keep deterministic regression replay on `test/fake` unless the owner intentionally selected a real provider.

package/docs/guides/run-evals.md

@@ -22,6 +22,12 @@ Run evals:
 agentkit eval run
 ```
 
+If the capsule uses npm scripts and Windows PowerShell blocks `npm.ps1`, use:
+
+```sh
+npm.cmd run eval
+```
+
 Create an eval from a stored conversation:
 
 ```sh
@@ -31,6 +37,15 @@ agentkit eval from-conversation <conversation-id>
 agentkit eval run
 ```
 
+Create evals from hosted or local production evidence:
+
+```sh
+agentkit improve collect --deploy --since 24h
+agentkit improve evals .agentkit/improve/<run>
+agentkit replay .agentkit/improve/<run> --against local
+agentkit eval run
+```
+
 ## Files Created Or Edited
 
 Create or edit:
@@ -43,9 +58,11 @@ Use conversations as source material:
 
 ```txt
 .agentkit/agentkit.db
+.agentkit/improve/<run>/
 ```
 
 Do not edit `.agentkit/agentkit.db` by hand.
+Do not edit `.agentkit/improve/<run>/bundle.json` by hand.
 
 ## Minimal Working Example
 
@@ -200,6 +217,7 @@ export const sendFollowupEmail = defineTool({
 - Keep eval tool calls deterministic and non-destructive.
 - For tools that would write, delete, charge money, send email, or call a real customer system, branch inside the registered tool on `ctx.runtime.environment === "eval"` and return safe fixture output.
 - Treat conversations as source material, not as automatically safe training data.
+- Review generated regression evals from `agentkit improve evals` before committing them. AgentKit redacts common email, phone, bearer token, and key patterns in generated eval text, but you must still remove or generalize domain-specific client PII and replace brittle exact prose assertions with the important behavior when needed.
 
 ## Verification
 

package/docs/guides/security-rules.md

@@ -93,14 +93,14 @@ defineTool({
 - `.env.schema` is the committed contract for local secret names.
 - AgentKit local commands load `.env` directly so inspect, chat, tools, and evals share the same secret loader.
 - Production uses managed secrets.
-- Hosted alpha deploys require `cloudflare_deploy_alpha`; local commands do not require login.
+- Hosted deploys require `cloudflare_deploy_alpha` or purchased/manual deploy slots; local commands do not require login.
 - Secret values must not appear in config, docs, prompts, evals, logs, exports, or SQLite.
 - A tool receives only secrets listed in that tool.
 - Avoid direct `process.env` reads inside tools.
 - Use `permissions` to describe external capabilities.
 - Add timeouts to network tools.
-- Treat public deploy URLs as unauthenticated transport, not access control.
-- Use access tokens and limits for clients.
+- Treat hosted deploy URLs as addresses, not access control.
+- Use deploy access tokens for hosted chat, hosted conversation reads, hosted trace reads, and any client app that talks to AgentKit Cloud.
 - Remove client PII before writing evals.
 
 ## Verification
@@ -132,9 +132,9 @@ Add the secret name to the tool `secrets` field and set it locally:
 npm run agentkit -- inspect
 ```
 
-Public URL is reachable:
+Hosted URL is reachable without a token:
 
-Check hosted access mode. Default should be `private`.
+Treat this as a security bug. Hosted AgentKit Cloud routes should reject missing deploy tokens except authenticated channel webhook ingress.
 
 ## Backend Contracts Used
 

package/docs/guides/use-provider.md

@@ -21,6 +21,14 @@ npm run chat -- --message "hello"
 npm run agentkit -- inspect
 ```
 
+On Windows PowerShell, if `npm.ps1` is blocked by `PSSecurityException`, use the Windows command shim:
+
+```sh
+npm.cmd run typecheck
+npm.cmd run chat -- --message "hello"
+npm.cmd run agentkit -- inspect
+```
+
 ## Files Created Or Edited
 
 Edit:
@@ -77,6 +85,8 @@ provider: {
 secrets: ["OPENROUTER_API_KEY"],
 ```
 
+Prefer model ids or aliases listed by the installed Pi SDK when available, such as `~google/gemini-flash-latest`. If an OpenRouter model id is newer than the Pi model registry, AgentKit passes the id through to OpenRouter using Pi's OpenAI-compatible transport with conservative unknown-model metadata. The provider may still reject the request if the id is invalid, inaccessible, or does not support the tools/features the agent uses.
+
 ## UI Verification
 
 After the provider is configured and the local secret is set, test through chat and UI:
@@ -125,7 +135,7 @@ npm run agentkit -- inspect
 
 `provider_model_unsupported`:
 
-Use a model id known to the installed Pi SDK for that provider.
+For OpenAI or Anthropic, use a model id known to the installed Pi SDK for that provider. For OpenRouter, prefer a known Pi alias when possible; otherwise a raw OpenRouter model id is passed through and any remaining model error comes from OpenRouter.
 
 Provider returns auth failure: