@andreprado/agentkit 0.1.0-alpha.9 → 0.1.0

package/docs/guides/debug-channel.md

@@ -0,0 +1,159 @@
+# Debug A Channel
+
+## Goal
+
+Diagnose website, Telegram, WhatsApp, Discord, Slack, generic inbound webhook, generic outbound webhook, cross-channel reply routing, webhook validation, dedupe, buffering, outbound sends, and delivery failures from the Agent Capsule CLI.
+
+## When To Use It
+
+Use this when a hosted channel is not responding, a provider is retrying messages, audio transcription is failing, cross-channel replies are going to the wrong target, or delivery status does not match the user's expectation.
+
+## Commands
+
+```sh
+agentkit channels list
+agentkit channels status <name>
+agentkit channels doctor <name>
+agentkit channels test <name> --message "hello"
+agentkit channels test-audio <name> --fixture voice-note
+agentkit transcribe smoke --provider groq
+agentkit channels test <name> --fixture ./fixtures/provider-event.json
+agentkit channels deliveries list <name> --since 24h
+agentkit channels deliveries show <delivery-id>
+agentkit channels deliveries list <reply-target-name> --since 24h
+agentkit channels buffers list <name>
+agentkit channels buffers show <conversation-id>
+agentkit channels buffers flush <conversation-id>
+agentkit channels buffers clear <conversation-id>
+agentkit channels buffers retry <conversation-id>
+```
+
+## Files Created Or Edited
+
+Debugging should not require source edits. Use fixtures only when reproducing provider payload shape, and never commit provider secrets, full phone numbers, raw audio, or private client data.
+
+## Delivery States
+
+```txt
+received
+validated
+duplicate
+audio_received
+audio_downloaded
+transcribing
+transcribed
+buffered
+queued
+running
+agent_completed
+provider_request_built
+provider_sent
+adapter_stubbed
+delivered
+provider_failed
+failed
+dead_lettered
+skipped
+```
+
+## Minimal Working Example
+
+```sh
+agentkit channels test support-telegram --message "hello"
+agentkit channels deliveries list support-telegram --since 1h
+agentkit channels deliveries show del_123
+```
+
+Expected `show` output includes signature status, dedupe key, queue/run state, outbound provider request ID when available, and a normalized error envelope.
+
+## Safety Rules
+
+- Inspect delivery IDs, hashes, statuses, and redacted metadata rather than raw provider payloads.
+- Never paste provider secrets into fixtures or prompts.
+- Treat `adapter_stubbed` as "no outbound provider call happened", not proof that a client received a message. It can mean explicit dry-run mode or an inbound-only channel such as a generic webhook.
+- For `replyTo` routes, inspect the source channel delivery for validation, dedupe, buffer, queue, and agent-run state, then inspect the target channel delivery for the outbound provider result.
+- Generic output webhook channels do not have inbound endpoints. Use `agentkit channels doctor <name>` to inspect secret readiness and inspect target-channel deliveries after a source event triggers the route.
+- Before replaying or flushing buffered conversations, confirm the channel name and conversation ID.
+- Keep local `.env` values ignored and upload hosted production values through AgentKit secret commands.
+
+## Troubleshooting By Error Code
+
+`channel_not_found`:
+The webhook URL points to an unknown or deleted channel. Stable hosted URLs use the channel name, for example `/channels/support-whatsapp/whatsapp/zapster/webhook`; old `chn_*` URLs are accepted only for compatibility.
+
+`agentkit channels test <name>` is the official synthetic smoke. If it fails while `channels list`, `status`, or `doctor` find the channel, rerun against the current deploy state in `.agentkit/deploy.json`.
+
+`channel_disabled`:
+The channel was disabled. Re-add or recreate it.
+
+`channel_secret_missing`:
+The hosted secret metadata says a required channel secret is missing. Set it with `agentkit secret set <NAME> --from-local-env` or `agentkit secret sync --from-local`.
+
+`channel_signature_invalid`:
+Provider authenticity validation failed. Check the provider webhook secret, token, origin settings, or Discord public key.
+
+`channel_payload_invalid`:
+The provider payload is malformed or does not match the route provider.
+
+Discord endpoint validation fails in the Developer Portal:
+Confirm `DISCORD_PUBLIC_KEY` is set from the application's public key and the webhook URL is `/channels/<name>/discord/discord/webhook`. AgentKit must validate Discord's signature headers before returning the `PING` PONG.
+
+Discord bot does not answer normal server messages:
+Confirm the channel was created with `--mode bot`, `DISCORD_BOT_TOKEN` is set, Message Content Intent is enabled in the Discord Developer Portal, the app is installed into the server, and the bot has `View Channel`, `Read Message History`, and `Send Messages` permissions for the channel.
+
+`audio_received`:
+The webhook contained a supported audio message and the channel is entering the audio handling path.
+
+`audio_downloaded`:
+The retryable channel worker downloaded the provider media file into memory. The delivery metadata should include only redacted size, MIME, and provider IDs.
+
+`transcribing`:
+AgentKit is calling the configured transcription provider with the user's managed transcription secret.
+
+`transcribed`:
+Transcription succeeded and the queued agent message contains transcript text instead of raw audio.
+
+`channel_event_duplicate` or `duplicate`:
+The provider retried an already-processed event. No second agent run should be created.
+
+`buffered`:
+The message is accepted and waiting inside a per-conversation channel buffer. It should move to `queued` after the quiet window, max wait, max message count, or max character count.
+
+`adapter_stubbed`:
+The adapter completed without calling an outbound provider. This can happen in explicit dry-run mode or for an inbound-only channel such as a generic webhook without `replyTo`.
+
+`provider_sent`:
+The provider API accepted the outbound request and returned a provider message ID. For cross-channel replies, this status appears on the reply target channel, such as WhatsApp or a generic output webhook.
+
+`channel_reply_target_missing`:
+The source channel has `replyTo.channel`, but the hosted channel resource for that target does not exist in the same deploy. Create or reconnect the target channel, then retry the source event.
+
+`channel_reply_recipient_missing`:
+The route needs a destination identity but `recipientFrom` was missing, resolved to an empty value, or the source identity cannot be reused by the target channel. Check the inbound payload and the configured dotted JSON path.
+
+`channel_reply_target_invalid`:
+The route points to an invalid target, such as an inbound-only generic webhook, or tries to use `recipientFrom` for a provider that requires native source identities such as Discord or Slack.
+
+`channel_limit_exceeded`:
+Backpressure skipped the message before queueing.
+
+`channel_audio_download_unavailable`:
+The channel provider reported audio but did not include enough metadata for AgentKit to download it.
+
+`transcription_secret_missing`:
+The transcription provider secret declared in `agentkit inspect` is not set as a managed hosted secret.
+
+`transcription_audio_too_large` or `transcription_audio_too_long`:
+The audio exceeded `transcription.limits` or the channel-level `audio.limits`.
+
+`transcription_audio_format_unsupported`:
+The configured transcription provider does not accept this audio MIME type or file extension.
+
+`transcription_provider_unavailable`:
+The transcription provider returned a retryable error, usually HTTP 429 or a server-side failure.
+
+`channel_provider_unavailable`:
+The agent run or provider send path failed with a retryable provider condition.
+
+`channel_delivery_dead_lettered`:
+The queue exhausted retries. Inspect the delivery and replay manually only after fixing the cause.

package/docs/guides/improve-from-production.md

@@ -0,0 +1,151 @@
+# Improve From Production
+
+## Goal
+
+Pull hosted or local conversation evidence into the Agent Capsule, turn it into regression evals, let the local coding agent patch the capsule, and replay before deploying again.
+
+## When To Use This
+
+Use this when a deployed agent gave a wrong answer, failed a tool call, mishandled a channel message, or needs production behavior converted into eval coverage.
+
+AgentKit Cloud only exports redacted evidence. The local coding agent owns source edits, evals, replay, and deploy.
+
+## Commands
+
+Collect evidence from the last hosted deploy:
+
+```sh
+agentkit improve collect --deploy --since 24h
+```
+
+When the CLI is logged in to AgentKit Cloud, this command first asks the control plane for deploy evidence such as failed channel deliveries, deploy errors, and conversation IDs that need review. It then reads replayable hosted conversation traces from the deployed runtime with the deploy access token. If Cloud auth is not available, it still collects hosted conversations directly from the deploy URL.
+
+Hosted conversation reads require a deploy access token even when the chat endpoint is public. `agentkit deploy` normally writes `.agentkit/chat-access-token.json`; refresh it with:
+
+```sh
+agentkit access token create agentkit-chat-ui --out .agentkit/chat-access-token.json
+```
+
+Collect one hosted conversation:
+
+```sh
+agentkit improve collect --deploy --conversation-id <conversation-id>
+```
+
+Collect local conversations instead:
+
+```sh
+agentkit improve collect --since 7d
+```
+
+Generate regression evals from the collected bundle:
+
+```sh
+agentkit improve evals .agentkit/improve/<run>
+```
+
+Replay the bundle against the local capsule:
+
+```sh
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+## Files Created Or Edited
+
+Evidence bundle, ignored local state:
+
+```txt
+.agentkit/improve/<run>/
+  bundle.json
+  report.json
+  traces/
+```
+
+Generated regression evals, committed source:
+
+```txt
+evals/regressions/
+  improve-<conversation>.eval.ts
+```
+
+The local coding agent may then edit:
+
+```txt
+prompts/instructions.md
+agentkit.config.ts
+tools/
+knowledge/
+evals/
+```
+
+Do not edit `.agentkit/improve/<run>/bundle.json` by hand.
+
+## Workflow
+
+```sh
+agentkit improve collect --deploy --since 24h
+agentkit improve evals .agentkit/improve/<run>
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Then let the local coding agent inspect `report.json`, the generated eval files, prompts, tools, and Knowledge sources. After edits:
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+agentkit deploy --smoke "hello"
+```
+
+## Safety Rules
+
+- Do not paste secrets into evals, prompts, Knowledge files, or reports.
+- Keep `.agentkit/improve/` out of commits.
+- Review generated eval assertions before committing them. AgentKit redacts common email, phone, bearer token, and key patterns in generated eval text, but the local coding agent must still remove or generalize domain-specific client PII.
+- For write, delete, payment, email, or customer-system tools, branch inside the tool on `ctx.runtime.environment === "eval"` and return deterministic non-destructive output.
+- Treat hosted traces as customer evidence.
+- If replay uses a real provider instead of `test/fake`, tell the owner because it may cost money and may be nondeterministic.
+
+## Verification
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Expected:
+
+- `improve collect` writes a bundle and report under `.agentkit/improve/`.
+- Hosted collection includes a redacted `evidence` summary in `bundle.json` and `report.json` when AgentKit Cloud evidence export is available.
+- `improve evals` writes eval files under `evals/regressions/`.
+- `replay` reports passed, failed, and skipped traces.
+- No production secret values appear in generated files.
+
+## Troubleshooting
+
+`No .agentkit/deploy.json found`:
+
+Run `agentkit deploy` first, or collect local evidence without `--deploy`.
+
+`deploy_conversation_request_failed`:
+
+Refresh the deploy chat token with `agentkit access token create agentkit-chat-ui --out .agentkit/chat-access-token.json`, then retry.
+
+`improve_evidence_store_not_configured`:
+
+The Cloud API does not expose deploy evidence export yet. The CLI falls back to hosted conversation trace collection when possible.
+
+`conversation_access_not_configured`:
+
+The hosted runtime is not configured with a deploy access-token gate for conversation reads. Redeploy through AgentKit Cloud so the runtime gate is injected, or configure an explicit deploy/private token for local hosted testing.
+
+`trace has no user turns`:
+
+The trace cannot become a useful conversation eval. Keep the report for diagnosis, but do not commit an empty eval.
+
+Generated eval is too strict:
+
+Edit the eval to assert the important behavior, such as tool input, safety wording, or knowledge source usage, instead of exact prose.

package/docs/guides/prepare-deploy.md

@@ -25,7 +25,18 @@ npm run agentkit -- db migrate
 npm run chat -- --message "hello"
 ```
 
-All scaffold, local dev, chat, eval, inspect, database, and build commands are token-free. Only hosted production deploy and hosted control-plane mutations require an invited AgentKit Cloud token:
+All scaffold, local dev, chat, eval, inspect, database, and build commands are token-free. Hosted production deploy and hosted AgentKit Cloud changes require an AgentKit Cloud account token with hosted deploy access.
+
+If the user does not have a token yet, start checkout from the CLI, finish Stripe Checkout in the browser, then claim the one-time checkout intent:
+
+```sh
+npm run agentkit -- billing checkout --slots 1 --email user@example.com
+npm run agentkit -- billing claim billint_... --secret bsec_...
+```
+
+The claim command stores the returned `agk_user_...` token in the local AgentKit Cloud auth file. Treat the `bsec_...` checkout secret like a password; it exists only to claim the first token after checkout.
+
+If the user already has a token:
 
 ```sh
 npm run agentkit -- login --token agk_user_...
@@ -42,7 +53,15 @@ OPENAI_API_KEY=sk-...
 ```
 
 `.env` is local-only. Hosted deploy secrets are handled by AgentKit outside the capsule.
-Closed-alpha hosted deploys require an invited account token with `cloudflare_deploy_alpha`.
+Hosted deploys require an account with either `cloudflare_deploy_alpha` or purchased/manual deploy slots. Deploy slots belong to the account, not to a specific token string.
+
+To generate or switch account tokens after login:
+
+```sh
+npm run agentkit -- account token create new-laptop --use
+npm run agentkit -- account token list
+npm run agentkit -- account token revoke apitok_...
+```
 
 ## What The Agent Should Edit
 
@@ -126,6 +145,7 @@ For a final end-to-end test, run:
 
 ```sh
 npm run agentkit -- login --token agk_user_...
+npm run agentkit -- account token list
 npm run agentkit -- secret sync --from-local
 npm run agentkit -- secret list
 npm run agentkit -- deploy --smoke "hello"
@@ -134,7 +154,7 @@ npm run agentkit -- chat-ui --deploy
 npm run agentkit -- access token list
 ```
 
-For private hosted deploys, `npm run agentkit -- deploy` writes the local chat/UI access token to `.agentkit/chat-access-token.json`. Use `npm run agentkit -- deploy --smoke "hello"` for the official hosted chat smoke, and use `npm run agentkit -- chat-ui --deploy` for hosted UI testing. Use `npm run agentkit -- access token create <name> --out <path>` only for additional clients.
+For hosted deploys, `npm run agentkit -- deploy` writes the local chat/UI access token to `.agentkit/chat-access-token.json` and prints a production handoff with the deploy URL, UI command, secret status, database/schema artifact, integration connect commands, smoke status, and next recommended command. Use `npm run agentkit -- deploy --smoke "hello"` for the official hosted chat smoke, and use `npm run agentkit -- chat-ui --deploy` for hosted UI testing. The hosted Chat UI shows the conversation id, tool calls, tool errors, and a new-conversation control; use `npm run agentkit -- conversations trace <conversation-id> --deploy` to pull the hosted trace from the last deploy. Use `npm run agentkit -- access token create <name> --out <path>` only for additional clients.
 
 When the UI is running, open the printed `Chat:` URL and tell the owner the exact URL. If the capsule is still on `test/fake`, say the UI was tested only with the deterministic fake provider.
 
@@ -146,7 +166,8 @@ No .agentkit committed
 All required secret names are declared
 Prompt path exists
 Tools have input schemas
-Dangerous tools have permissions
+Read-only tools use `:read` permissions; dangerous tools use operator-only non-read permissions
+Managed Composio toolkit auth configs are available in AgentKit Cloud when configured
 Provider model is supported by AgentKit
 schema.sql is idempotent
 README/AGENTS/CLAUDE match the capsule
@@ -172,7 +193,7 @@ echo 'OPENAI_API_KEY=sk-...' >> .env
 
 Missing hosted secret:
 
-Set the secret in AgentKit Cloud or the configured control plane:
+Set the secret in AgentKit Cloud:
 
 ```sh
 npm run agentkit -- secret set OPENAI_API_KEY --from-local-env
@@ -192,18 +213,15 @@ Before a hosted production deploy, run:
 npm run agentkit -- deploy doctor
 ```
 
-The doctor checks AgentKit Cloud login, alpha deploy entitlement, online deploy capacity, declared hosted secrets, local `.env` names that have not been uploaded with `agentkit secret set`, and private-access runtime token handling. `agentkit deploy` runs the same readiness check automatically before building and uploading the artifact.
+The doctor checks AgentKit Cloud login, Node version, hosted deploy entitlement, online deploy capacity, declared hosted secrets, local `.env` names that have not been uploaded with `npm run agentkit -- secret set`, and private-access runtime token handling. `agentkit deploy` runs the same readiness check automatically before building and uploading the artifact.
+
+If the capsule configures `composioManaged({...})`, the doctor also checks the paid `managed_composio` entitlement and AgentKit Cloud toolkit readiness. `COMPOSIO_API_KEY` and toolkit auth config resolution are AgentKit-managed in this path; the user must not set them with `agentkit secret set`.
 
 `alpha_access_required`:
 
-Log in with an invited alpha account:
+Log in with an account that has hosted deploy access. If the user needs to buy slots first:
 
 ```sh
-npm run agentkit -- login --token agk_user_...
+npm run agentkit -- billing checkout --slots 1 --email user@example.com
+npm run agentkit -- billing claim billint_... --secret bsec_...
 ```
-
-## Operator Notes
-
-This section is for AgentKit maintainers, not for capsule-building agents.
-
-`agentkit deploy` sends the capsule artifact to the configured AgentKit Cloud API. The control plane owns infrastructure selection, provisioning, managed secrets, and public URL creation. Use `AGENTKIT_CLOUD_API_URL` only when testing a non-default control plane.

package/docs/guides/replay-production-traces.md

@@ -0,0 +1,72 @@
+# Replay Production Traces
+
+## Goal
+
+Run collected production or local traces against the local Agent Capsule before redeploying a fix.
+
+## When To Use This
+
+Use this after `agentkit improve collect` and before `agentkit deploy` whenever prompts, tools, Knowledge, provider config, or channel behavior changed because of production evidence.
+
+## Commands
+
+```sh
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Run the full local eval suite too:
+
+```sh
+npm run eval
+```
+
+## What Replay Checks
+
+Replay sends each collected user turn through the local capsule in eval mode:
+
+```txt
+ctx.runtime.environment === "eval"
+ctx.runtime.invocation === "eval"
+```
+
+This proves the current capsule can process the production turns without runtime errors. Generated eval files under `evals/regressions/` add committed behavior assertions.
+
+## Write Tool Safety
+
+Replay still runs the registered capsule tools. Any tool that can write externally must guard eval mode:
+
+```ts
+if (ctx.runtime.environment === "eval") {
+  return { sent: false, evalFixture: true };
+}
+```
+
+Do not depend on prompt wording alone to prevent side effects.
+
+## Deploy Gate
+
+Before redeploying a production fix:
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+agentkit deploy --smoke "hello"
+```
+
+If replay fails, inspect the failed trace id in `.agentkit/improve/<run>/traces/`, patch the capsule, and rerun replay.
+
+## Troubleshooting
+
+`Replay failed`:
+
+Read the printed error and the matching trace file. Common causes are missing local secrets, unsafe tools that do not branch on eval mode, stale Knowledge sources, or provider differences.
+
+`Replay skipped`:
+
+The trace had no user message. It may still help diagnose delivery or deploy state, but it cannot be replayed as a conversation.
+
+`provider_model_unsupported`:
+
+The local provider config does not match the replay environment. Keep deterministic regression replay on `test/fake` unless the owner intentionally selected a real provider.

package/docs/guides/run-evals.md

@@ -22,6 +22,14 @@ Run evals:
 agentkit eval run
 ```
 
+`agentkit eval run` uses temporary local SQLite storage for eval execution. This keeps eval conversations and tool calls isolated from `.agentkit/agentkit.db`, so evals can run while a local chat or dev server is using the normal development database.
+
+If the capsule uses npm scripts and Windows PowerShell blocks `npm.ps1`, use:
+
+```sh
+npm.cmd run eval
+```
+
 Create an eval from a stored conversation:
 
 ```sh
@@ -31,6 +39,15 @@ agentkit eval from-conversation <conversation-id>
 agentkit eval run
 ```
 
+Create evals from hosted or local production evidence:
+
+```sh
+agentkit improve collect --deploy --since 24h
+agentkit improve evals .agentkit/improve/<run>
+agentkit replay .agentkit/improve/<run> --against local
+agentkit eval run
+```
+
 ## Files Created Or Edited
 
 Create or edit:
@@ -39,85 +56,137 @@ Create or edit:
 evals/<name>.eval.ts
 ```
 
+Create evals proactively from the agent brief and `AGENT_SPEC.md`. High-value evals cover identity and scope, required intake fields, confirmation before writes, no-leak/privacy rules, timezone and business-hour behavior, default durations or limits, tool-call payloads, unavailable slots, empty results, missing auth, rate limits, and timeouts. If a real conversation reveals a bug, add the smallest regression eval that would have failed before the fix.
+
 Use conversations as source material:
 
 ```txt
 .agentkit/agentkit.db
+.agentkit/improve/<run>/
 ```
 
 Do not edit `.agentkit/agentkit.db` by hand.
+Do not edit `.agentkit/improve/<run>/bundle.json` by hand.
 
 ## Minimal Working Example
 
 Generated smoke eval:
 
 ```ts
-export default {
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
   name: "smoke",
   input: "Say hello in one short sentence.",
   expect: {
-    contains: "hello",
+    response: {
+      caseInsensitiveContains: "hello",
+      maxLength: 160,
+    },
   },
-};
+});
 ```
 
 Supported assertion types:
 
 ```txt
-contains
-not_contains
-regex
-matches_regex
-persisted_tool_call
+response.contains
+response.containsAll
+response.containsAny
+response.caseInsensitiveContains
+response.notContains
+response.regex
+response.matchesRegex
+response.notRegex
+response.maxLength
+tools.called
+tools.calledOnce
+tools.count
+tools.order
+tools.persisted
+```
+
+Older flat aliases still work, including `contains`, `not_contains`, `regex`, `matches_regex`, and `persisted_tool_call`.
+
+For date-sensitive evals, set top-level `now` to an ISO timestamp with an explicit timezone designator such as `Z` or `-05:00`. AgentKit uses that fixed clock for every turn and tool call in the eval:
+
+```ts
+export default defineEval({
+  name: "appointment relative date",
+  now: "2026-02-04T02:30:00.000Z",
+  input: "What is today's date?",
+  expect: {
+    response: {
+      containsAll: ["2026-02-03", "Tuesday"],
+    },
+  },
+});
 ```
 
 Multi-turn conversation evals use `turns`:
 
 ```ts
-export default {
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
   name: "buyer under budget",
   turns: [
     {
       input: "I want a house up to 600k near Pinheiros.",
       expect: {
-        persisted_tool_call: {
-          name: "buscar_imoveis",
-          status: "completed",
-          input: { maxPrice: 600000 },
+        tools: {
+          calledOnce: "buscar_imoveis",
+          persisted: {
+            name: "buscar_imoveis",
+            status: "completed",
+            input: { maxPrice: 600000 },
+          },
         },
       },
     },
     {
       input: "Show me the best two.",
       expect: {
-        contains: ["Pinheiros", "R$"],
+        response: {
+          containsAll: ["Pinheiros", "R$"],
+        },
       },
     },
   ],
-};
+});
 ```
 
-`persisted_tool_call` validates the tool call saved in local SQLite `tool_calls`, not a provider-specific raw response shape. It can be a tool name string or an object with `name`, `input`, `output`, `rendered`, `status`, and/or `visibility`.
+`tools.persisted` validates the tool call saved in the eval run's local SQLite `tool_calls`, not a provider-specific raw response shape. It can be a tool name string or an object with `name`, `input`, `output`, `rendered`, `status`, and/or `visibility`.
+
+Use `tools.count` for the exact number of persisted calls in that turn, `tools.calledOnce` for exactly one call by name, and `tools.order` for required relative order. `tools.order` allows extra calls before, between, or after the named calls; pair it with `tools.count` when the exact call set matters.
 
 Use response assertions and persisted tool assertions together when internal operational output must not leak:
 
 ```ts
-export default {
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
   name: "triage lead",
   input: '{"tool":"triage_real_estate_lead","input":{"email":"ada@example.com"}}',
   expect: {
-    not_contains: ["hot", "score"],
-    persisted_tool_call: {
-      name: "triage_real_estate_lead",
-      status: "completed",
-      visibility: "internal",
-      output: { status: "hot" },
+    response: {
+      notContains: ["hot", "score"],
+      notRegex: ["API_KEY|secret|token"],
+    },
+    tools: {
+      calledOnce: "triage_real_estate_lead",
+      persisted: {
+        name: "triage_real_estate_lead",
+        status: "completed",
+        visibility: "internal",
+        output: { status: "hot" },
+      },
     },
   },
-};
+});
 ```
 
-`tool_call` remains accepted as a backwards-compatible alias, but new evals should use `persisted_tool_call`.
+`tool_call` and `persisted_tool_call` remain accepted as backwards-compatible aliases, but new evals should use `tools.persisted`.
 
 Safe external-tool pattern:
 
@@ -152,6 +221,7 @@ export const sendFollowupEmail = defineTool({
 - Keep eval tool calls deterministic and non-destructive.
 - For tools that would write, delete, charge money, send email, or call a real customer system, branch inside the registered tool on `ctx.runtime.environment === "eval"` and return safe fixture output.
 - Treat conversations as source material, not as automatically safe training data.
+- Review generated regression evals from `agentkit improve evals` before committing them. AgentKit redacts common email, phone, bearer token, and key patterns in generated eval text, but you must still remove or generalize domain-specific client PII and replace brittle exact prose assertions with the important behavior when needed.
 
 ## Verification