@andreprado/agentkit 0.1.0-alpha.2 → 0.1.0-alpha.21

package/docs/guides/improve-from-production.md

@@ -0,0 +1,151 @@
+# Improve From Production
+
+## Goal
+
+Pull hosted or local conversation evidence into the Agent Capsule, turn it into regression evals, let the local coding agent patch the capsule, and replay before deploying again.
+
+## When To Use This
+
+Use this when a deployed agent gave a wrong answer, failed a tool call, mishandled a channel message, or needs production behavior converted into eval coverage.
+
+AgentKit Cloud only exports redacted evidence. The local coding agent owns source edits, evals, replay, and deploy.
+
+## Commands
+
+Collect evidence from the last hosted deploy:
+
+```sh
+agentkit improve collect --deploy --since 24h
+```
+
+When the CLI is logged in to AgentKit Cloud, this command first asks the control plane for deploy evidence such as failed channel deliveries, deploy errors, and conversation IDs that need review. It then reads replayable hosted conversation traces from the deployed runtime with the deploy access token. If Cloud auth is not available, it still collects hosted conversations directly from the deploy URL.
+
+Hosted conversation reads require a deploy access token even when the chat endpoint is public. `agentkit deploy` normally writes `.agentkit/chat-access-token.json`; refresh it with:
+
+```sh
+agentkit access token create agentkit-chat-ui --out .agentkit/chat-access-token.json
+```
+
+Collect one hosted conversation:
+
+```sh
+agentkit improve collect --deploy --conversation-id <conversation-id>
+```
+
+Collect local conversations instead:
+
+```sh
+agentkit improve collect --since 7d
+```
+
+Generate regression evals from the collected bundle:
+
+```sh
+agentkit improve evals .agentkit/improve/<run>
+```
+
+Replay the bundle against the local capsule:
+
+```sh
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+## Files Created Or Edited
+
+Evidence bundle, ignored local state:
+
+```txt
+.agentkit/improve/<run>/
+  bundle.json
+  report.json
+  traces/
+```
+
+Generated regression evals, committed source:
+
+```txt
+evals/regressions/
+  improve-<conversation>.eval.ts
+```
+
+The local coding agent may then edit:
+
+```txt
+prompts/instructions.md
+agentkit.config.ts
+tools/
+knowledge/
+evals/
+```
+
+Do not edit `.agentkit/improve/<run>/bundle.json` by hand.
+
+## Workflow
+
+```sh
+agentkit improve collect --deploy --since 24h
+agentkit improve evals .agentkit/improve/<run>
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Then let the local coding agent inspect `report.json`, the generated eval files, prompts, tools, and Knowledge sources. After edits:
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+agentkit deploy --smoke "hello"
+```
+
+## Safety Rules
+
+- Do not paste secrets into evals, prompts, Knowledge files, or reports.
+- Keep `.agentkit/improve/` out of commits.
+- Review generated eval assertions before committing them. AgentKit redacts common email, phone, bearer token, and key patterns in generated eval text, but the local coding agent must still remove or generalize domain-specific client PII.
+- For write, delete, payment, email, or customer-system tools, branch inside the tool on `ctx.runtime.environment === "eval"` and return deterministic non-destructive output.
+- Treat hosted traces as customer evidence.
+- If replay uses a real provider instead of `test/fake`, tell the owner because it may cost money and may be nondeterministic.
+
+## Verification
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Expected:
+
+- `improve collect` writes a bundle and report under `.agentkit/improve/`.
+- Hosted collection includes a redacted `evidence` summary in `bundle.json` and `report.json` when AgentKit Cloud evidence export is available.
+- `improve evals` writes eval files under `evals/regressions/`.
+- `replay` reports passed, failed, and skipped traces.
+- No production secret values appear in generated files.
+
+## Troubleshooting
+
+`No .agentkit/deploy.json found`:
+
+Run `agentkit deploy` first, or collect local evidence without `--deploy`.
+
+`deploy_conversation_request_failed`:
+
+Refresh the deploy chat token with `agentkit access token create agentkit-chat-ui --out .agentkit/chat-access-token.json`, then retry.
+
+`improve_evidence_store_not_configured`:
+
+The Cloud API does not expose deploy evidence export yet. The CLI falls back to hosted conversation trace collection when possible.
+
+`conversation_access_not_configured`:
+
+The hosted runtime is not configured with a deploy access-token gate for conversation reads. Redeploy through AgentKit Cloud so the runtime gate is injected, or configure an explicit deploy/private token for local hosted testing.
+
+`trace has no user turns`:
+
+The trace cannot become a useful conversation eval. Keep the report for diagnosis, but do not commit an empty eval.
+
+Generated eval is too strict:
+
+Edit the eval to assert the important behavior, such as tool input, safety wording, or knowledge source usage, instead of exact prose.

package/docs/guides/prepare-deploy.md

@@ -19,20 +19,31 @@ Use this before asking a coding agent to make a capsule deployable, or before te
 From the capsule root:
 
 ```sh
-npm install
 npm run typecheck
 npm run agentkit -- inspect
 npm run agentkit -- db migrate
 npm run chat -- --message "hello"
 ```
 
-All scaffold, local dev, chat, eval, inspect, database, and build commands are token-free. Only hosted production deploy and hosted control-plane mutations require an invited AgentKit Cloud token:
+All scaffold, local dev, chat, eval, inspect, database, and build commands are token-free. Hosted production deploy and hosted AgentKit Cloud changes require an AgentKit Cloud account token with hosted deploy access.
+
+If the user does not have a token yet, start checkout from the CLI, finish Stripe Checkout in the browser, then claim the one-time checkout intent:
+
+```sh
+npm run agentkit -- billing checkout --slots 1 --email user@example.com
+npm run agentkit -- billing claim billint_... --secret bsec_...
+```
+
+The claim command stores the returned `agk_user_...` token in the local AgentKit Cloud auth file. Treat the `bsec_...` checkout secret like a password; it exists only to claim the first token after checkout.
+
+If the user already has a token:
 
 ```sh
 npm run agentkit -- login --token agk_user_...
 npm run agentkit -- deploy doctor
-npm run agentkit -- secret set OPENAI_API_KEY sk-...
-npm run agentkit -- deploy
+npm run agentkit -- secret set OPENAI_API_KEY --from-local-env
+npm run agentkit -- deploy --smoke "hello"
+npm run agentkit -- chat-ui --deploy
 ```
 
 If the capsule uses OpenAI locally, put the user’s local key in `.env`:
@@ -42,7 +53,15 @@ OPENAI_API_KEY=sk-...
 ```
 
 `.env` is local-only. Hosted deploy secrets are handled by AgentKit outside the capsule.
-Closed-alpha hosted deploys require an invited account token with `cloudflare_deploy_alpha`.
+Hosted deploys require an account with either `cloudflare_deploy_alpha` or purchased/manual deploy slots. Deploy slots belong to the account, not to a specific token string.
+
+To generate or switch account tokens after login:
+
+```sh
+npm run agentkit -- account token create new-laptop --use
+npm run agentkit -- account token list
+npm run agentkit -- account token revoke apitok_...
+```
 
 ## What The Agent Should Edit
 
@@ -126,12 +145,19 @@ For a final end-to-end test, run:
 
 ```sh
 npm run agentkit -- login --token agk_user_...
+npm run agentkit -- account token list
+npm run agentkit -- secret sync --from-local
 npm run agentkit -- secret list
-npm run agentkit -- deploy
+npm run agentkit -- deploy --smoke "hello"
 npm run agentkit -- deploy status
+npm run agentkit -- chat-ui --deploy
 npm run agentkit -- access token list
 ```
 
+For hosted deploys, `npm run agentkit -- deploy` writes the local chat/UI access token to `.agentkit/chat-access-token.json`. Use `npm run agentkit -- deploy --smoke "hello"` for the official hosted chat smoke, and use `npm run agentkit -- chat-ui --deploy` for hosted UI testing. The hosted Chat UI shows the conversation id, tool calls, tool errors, and a new-conversation control; use `npm run agentkit -- conversations trace <conversation-id> --deploy` to pull the hosted trace from the last deploy. Use `npm run agentkit -- access token create <name> --out <path>` only for additional clients.
+
+When the UI is running, open the printed `Chat:` URL and tell the owner the exact URL. If the capsule is still on `test/fake`, say the UI was tested only with the deterministic fake provider.
+
 ## Readiness Checklist
 
 ```txt
@@ -141,6 +167,7 @@ All required secret names are declared
 Prompt path exists
 Tools have input schemas
 Dangerous tools have permissions
+Managed Composio toolkit auth configs are available in AgentKit Cloud when configured
 Provider model is supported by AgentKit
 schema.sql is idempotent
 README/AGENTS/CLAUDE match the capsule
@@ -166,10 +193,16 @@ echo 'OPENAI_API_KEY=sk-...' >> .env
 
 Missing hosted secret:
 
-Set the secret in AgentKit Cloud or the configured control plane:
+Set the secret in AgentKit Cloud:
+
+```sh
+npm run agentkit -- secret set OPENAI_API_KEY --from-local-env
+```
+
+To sync all declared user-managed secrets present in local `.env`:
 
 ```sh
-npm run agentkit -- secret set OPENAI_API_KEY sk-...
+npm run agentkit -- secret sync --from-local
 ```
 
 Do not write hosted secret values into the capsule.
@@ -180,18 +213,15 @@ Before a hosted production deploy, run:
 npm run agentkit -- deploy doctor
 ```
 
-The doctor checks AgentKit Cloud login, alpha deploy entitlement, declared hosted secrets, local `.env` names that have not been uploaded with `agentkit secret set`, and private-access runtime token handling. `agentkit deploy` runs the same readiness check automatically before building and uploading the artifact.
+The doctor checks AgentKit Cloud login, Node version, hosted deploy entitlement, online deploy capacity, declared hosted secrets, local `.env` names that have not been uploaded with `npm run agentkit -- secret set`, and private-access runtime token handling. `agentkit deploy` runs the same readiness check automatically before building and uploading the artifact.
+
+If the capsule configures `composioManaged({...})`, the doctor also checks the paid `managed_composio` entitlement and AgentKit Cloud toolkit readiness. `COMPOSIO_API_KEY` and toolkit auth config resolution are AgentKit-managed in this path; the user must not set them with `agentkit secret set`.
 
 `alpha_access_required`:
 
-Log in with an invited alpha account:
+Log in with an account that has hosted deploy access. If the user needs to buy slots first:
 
 ```sh
-npm run agentkit -- login --token agk_user_...
+npm run agentkit -- billing checkout --slots 1 --email user@example.com
+npm run agentkit -- billing claim billint_... --secret bsec_...
 ```
-
-## Operator Notes
-
-This section is for AgentKit maintainers, not for capsule-building agents.
-
-`agentkit deploy` sends the capsule artifact to the configured AgentKit Cloud API. The control plane owns infrastructure selection, provisioning, managed secrets, and public URL creation. Use `AGENTKIT_CLOUD_API_URL` only when testing a non-default control plane.

package/docs/guides/replay-production-traces.md

@@ -0,0 +1,72 @@
+# Replay Production Traces
+
+## Goal
+
+Run collected production or local traces against the local Agent Capsule before redeploying a fix.
+
+## When To Use This
+
+Use this after `agentkit improve collect` and before `agentkit deploy` whenever prompts, tools, Knowledge, provider config, or channel behavior changed because of production evidence.
+
+## Commands
+
+```sh
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Run the full local eval suite too:
+
+```sh
+npm run eval
+```
+
+## What Replay Checks
+
+Replay sends each collected user turn through the local capsule in eval mode:
+
+```txt
+ctx.runtime.environment === "eval"
+ctx.runtime.invocation === "eval"
+```
+
+This proves the current capsule can process the production turns without runtime errors. Generated eval files under `evals/regressions/` add committed behavior assertions.
+
+## Write Tool Safety
+
+Replay still runs the registered capsule tools. Any tool that can write externally must guard eval mode:
+
+```ts
+if (ctx.runtime.environment === "eval") {
+  return { sent: false, evalFixture: true };
+}
+```
+
+Do not depend on prompt wording alone to prevent side effects.
+
+## Deploy Gate
+
+Before redeploying a production fix:
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+agentkit deploy --smoke "hello"
+```
+
+If replay fails, inspect the failed trace id in `.agentkit/improve/<run>/traces/`, patch the capsule, and rerun replay.
+
+## Troubleshooting
+
+`Replay failed`:
+
+Read the printed error and the matching trace file. Common causes are missing local secrets, unsafe tools that do not branch on eval mode, stale Knowledge sources, or provider differences.
+
+`Replay skipped`:
+
+The trace had no user message. It may still help diagnose delivery or deploy state, but it cannot be replayed as a conversation.
+
+`provider_model_unsupported`:
+
+The local provider config does not match the replay environment. Keep deterministic regression replay on `test/fake` unless the owner intentionally selected a real provider.

package/docs/guides/run-evals.md

@@ -22,6 +22,30 @@ Run evals:
 agentkit eval run
 ```
 
+If the capsule uses npm scripts and Windows PowerShell blocks `npm.ps1`, use:
+
+```sh
+npm.cmd run eval
+```
+
+Create an eval from a stored conversation:
+
+```sh
+agentkit conversations list
+agentkit conversations trace <conversation-id>
+agentkit eval from-conversation <conversation-id>
+agentkit eval run
+```
+
+Create evals from hosted or local production evidence:
+
+```sh
+agentkit improve collect --deploy --since 24h
+agentkit improve evals .agentkit/improve/<run>
+agentkit replay .agentkit/improve/<run> --against local
+agentkit eval run
+```
+
 ## Files Created Or Edited
 
 Create or edit:
@@ -34,63 +58,166 @@ Use conversations as source material:
 
 ```txt
 .agentkit/agentkit.db
+.agentkit/improve/<run>/
 ```
 
 Do not edit `.agentkit/agentkit.db` by hand.
+Do not edit `.agentkit/improve/<run>/bundle.json` by hand.
 
 ## Minimal Working Example
 
 Generated smoke eval:
 
 ```ts
-export default {
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
   name: "smoke",
   input: "Say hello in one short sentence.",
   expect: {
-    contains: "hello",
+    response: {
+      caseInsensitiveContains: "hello",
+      maxLength: 160,
+    },
   },
-};
+});
 ```
 
 Supported assertion types:
 
 ```txt
-contains
-not_contains
-regex
-matches_regex
-persisted_tool_call
+response.contains
+response.containsAll
+response.containsAny
+response.caseInsensitiveContains
+response.notContains
+response.regex
+response.matchesRegex
+response.notRegex
+response.maxLength
+tools.called
+tools.calledOnce
+tools.count
+tools.order
+tools.persisted
 ```
 
-`persisted_tool_call` validates the tool call saved in local SQLite `tool_calls`, not a provider-specific raw response shape. It can be a tool name string or an object with `name`, `input`, `output`, `status`, and/or `visibility`.
+Older flat aliases still work, including `contains`, `not_contains`, `regex`, `matches_regex`, and `persisted_tool_call`.
+
+For date-sensitive evals, set top-level `now` to an ISO timestamp with an explicit timezone designator such as `Z` or `-05:00`. AgentKit uses that fixed clock for every turn and tool call in the eval:
+
+```ts
+export default defineEval({
+  name: "appointment relative date",
+  now: "2026-02-04T02:30:00.000Z",
+  input: "What is today's date?",
+  expect: {
+    response: {
+      containsAll: ["2026-02-03", "Tuesday"],
+    },
+  },
+});
+```
+
+Multi-turn conversation evals use `turns`:
+
+```ts
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
+  name: "buyer under budget",
+  turns: [
+    {
+      input: "I want a house up to 600k near Pinheiros.",
+      expect: {
+        tools: {
+          calledOnce: "buscar_imoveis",
+          persisted: {
+            name: "buscar_imoveis",
+            status: "completed",
+            input: { maxPrice: 600000 },
+          },
+        },
+      },
+    },
+    {
+      input: "Show me the best two.",
+      expect: {
+        response: {
+          containsAll: ["Pinheiros", "R$"],
+        },
+      },
+    },
+  ],
+});
+```
+
+`tools.persisted` validates the tool call saved in local SQLite `tool_calls`, not a provider-specific raw response shape. It can be a tool name string or an object with `name`, `input`, `output`, `rendered`, `status`, and/or `visibility`.
+
+Use `tools.count` for the exact number of persisted calls in that turn, `tools.calledOnce` for exactly one call by name, and `tools.order` for required relative order. `tools.order` allows extra calls before, between, or after the named calls; pair it with `tools.count` when the exact call set matters.
 
 Use response assertions and persisted tool assertions together when internal operational output must not leak:
 
 ```ts
-export default {
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
   name: "triage lead",
   input: '{"tool":"triage_real_estate_lead","input":{"email":"ada@example.com"}}',
   expect: {
-    not_contains: ["hot", "score"],
-    persisted_tool_call: {
-      name: "triage_real_estate_lead",
-      status: "completed",
-      visibility: "internal",
-      output: { status: "hot" },
+    response: {
+      notContains: ["hot", "score"],
+      notRegex: ["API_KEY|secret|token"],
+    },
+    tools: {
+      calledOnce: "triage_real_estate_lead",
+      persisted: {
+        name: "triage_real_estate_lead",
+        status: "completed",
+        visibility: "internal",
+        output: { status: "hot" },
+      },
     },
   },
-};
+});
 ```
 
-`tool_call` remains accepted as a backwards-compatible alias, but new evals should use `persisted_tool_call`.
+`tool_call` and `persisted_tool_call` remain accepted as backwards-compatible aliases, but new evals should use `tools.persisted`.
+
+Safe external-tool pattern:
+
+```ts
+export const sendFollowupEmail = defineTool({
+  name: "send_followup_email",
+  description: "Sends a follow-up email.",
+  inputSchema: {
+    type: "object",
+    properties: {
+      email: { type: "string" },
+    },
+    required: ["email"],
+    additionalProperties: false,
+  },
+  async execute(input: { email: string }, ctx) {
+    if (ctx.runtime.environment === "eval") {
+      return { sent: false, evalFixture: true, email: input.email };
+    }
+
+    // Real provider call here.
+    return { sent: true, evalFixture: false, email: input.email };
+  },
+});
+```
 
 ## Safety Rules
 
 - Do not put provider keys in eval files.
 - Do not put real client PII in eval files.
 - Prefer small deterministic assertions.
-- Mock tools for evals that would write, delete, charge money, or email people.
+- Keep eval tool calls deterministic and non-destructive.
+- For tools that would write, delete, charge money, send email, or call a real customer system, branch inside the registered tool on `ctx.runtime.environment === "eval"` and return safe fixture output.
 - Treat conversations as source material, not as automatically safe training data.
+- Review generated regression evals from `agentkit improve evals` before committing them. AgentKit redacts common email, phone, bearer token, and key patterns in generated eval text, but you must still remove or generalize domain-specific client PII and replace brittle exact prose assertions with the important behavior when needed.
 
 ## Verification
 
@@ -111,7 +238,7 @@ Use `test/fake` for deterministic smoke tests, then add provider-specific evals
 
 Tool eval hits a real API:
 
-Add tool mocks before running the eval suite.
+Make the registered tool return deterministic fixture output when `ctx.runtime.environment === "eval"`, then rerun the eval suite. AgentKit does not expose a separate eval-only mock registry yet.
 
 ## Backend Contracts Used
 

package/docs/guides/security-rules.md

@@ -29,7 +29,7 @@ Hosted alpha secret commands:
 
 ```sh
 agentkit login --token agk_user_...
-agentkit secret set OPENAI_API_KEY
+agentkit secret set OPENAI_API_KEY --from-local-env
 agentkit secret list
 agentkit secret unset OPENAI_API_KEY
 ```
@@ -93,15 +93,16 @@ defineTool({
 - `.env.schema` is the committed contract for local secret names.
 - AgentKit local commands load `.env` directly so inspect, chat, tools, and evals share the same secret loader.
 - Production uses managed secrets.
-- Hosted alpha deploys require `cloudflare_deploy_alpha`; local commands do not require login.
+- Hosted deploys require `cloudflare_deploy_alpha` or purchased/manual deploy slots; local commands do not require login.
 - Secret values must not appear in config, docs, prompts, evals, logs, exports, or SQLite.
 - A tool receives only secrets listed in that tool.
 - Avoid direct `process.env` reads inside tools.
 - Use `permissions` to describe external capabilities.
 - Add timeouts to network tools.
-- Treat public deploy URLs as unauthenticated transport, not access control.
-- Use access tokens and limits for clients.
+- Treat hosted deploy URLs as addresses, not access control.
+- Use deploy access tokens for hosted chat, hosted conversation reads, hosted trace reads, and any client app that talks to AgentKit Cloud.
 - Remove client PII before writing evals.
+- Review `.agentkit/feedback/` drafts before sending AgentKit product feedback. Do not paste `.env` values, provider keys, cookies, client PII, or full private transcripts into feedback messages.
 
 ## Verification
 
@@ -132,9 +133,9 @@ Add the secret name to the tool `secrets` field and set it locally:
 npm run agentkit -- inspect
 ```
 
-Public URL is reachable:
+Hosted URL is reachable without a token:
 
-Check hosted access mode. Default should be `private`.
+Treat this as a security bug. Hosted AgentKit Cloud routes should reject missing deploy tokens except authenticated channel webhook ingress.
 
 ## Backend Contracts Used