@andreprado/agentkit 0.1.0-alpha.2 → 0.1.0-alpha.20

package/docs/guides/replay-production-traces.md

@@ -0,0 +1,72 @@
+# Replay Production Traces
+
+## Goal
+
+Run collected production or local traces against the local Agent Capsule before redeploying a fix.
+
+## When To Use This
+
+Use this after `agentkit improve collect` and before `agentkit deploy` whenever prompts, tools, Knowledge, provider config, or channel behavior changed because of production evidence.
+
+## Commands
+
+```sh
+agentkit replay .agentkit/improve/<run> --against local
+```
+
+Run the full local eval suite too:
+
+```sh
+npm run eval
+```
+
+## What Replay Checks
+
+Replay sends each collected user turn through the local capsule in eval mode:
+
+```txt
+ctx.runtime.environment === "eval"
+ctx.runtime.invocation === "eval"
+```
+
+This proves the current capsule can process the production turns without runtime errors. Generated eval files under `evals/regressions/` add committed behavior assertions.
+
+## Write Tool Safety
+
+Replay still runs the registered capsule tools. Any tool that can write externally must guard eval mode:
+
+```ts
+if (ctx.runtime.environment === "eval") {
+  return { sent: false, evalFixture: true };
+}
+```
+
+Do not depend on prompt wording alone to prevent side effects.
+
+## Deploy Gate
+
+Before redeploying a production fix:
+
+```sh
+npm run typecheck
+npm run agentkit -- inspect
+npm run eval
+agentkit replay .agentkit/improve/<run> --against local
+agentkit deploy --smoke "hello"
+```
+
+If replay fails, inspect the failed trace id in `.agentkit/improve/<run>/traces/`, patch the capsule, and rerun replay.
+
+## Troubleshooting
+
+`Replay failed`:
+
+Read the printed error and the matching trace file. Common causes are missing local secrets, unsafe tools that do not branch on eval mode, stale Knowledge sources, or provider differences.
+
+`Replay skipped`:
+
+The trace had no user message. It may still help diagnose delivery or deploy state, but it cannot be replayed as a conversation.
+
+`provider_model_unsupported`:
+
+The local provider config does not match the replay environment. Keep deterministic regression replay on `test/fake` unless the owner intentionally selected a real provider.

package/docs/guides/run-evals.md

@@ -22,6 +22,30 @@ Run evals:
 agentkit eval run
 ```
 
+If the capsule uses npm scripts and Windows PowerShell blocks `npm.ps1`, use:
+
+```sh
+npm.cmd run eval
+```
+
+Create an eval from a stored conversation:
+
+```sh
+agentkit conversations list
+agentkit conversations trace <conversation-id>
+agentkit eval from-conversation <conversation-id>
+agentkit eval run
+```
+
+Create evals from hosted or local production evidence:
+
+```sh
+agentkit improve collect --deploy --since 24h
+agentkit improve evals .agentkit/improve/<run>
+agentkit replay .agentkit/improve/<run> --against local
+agentkit eval run
+```
+
 ## Files Created Or Edited
 
 Create or edit:
@@ -34,63 +58,166 @@ Use conversations as source material:
 
 ```txt
 .agentkit/agentkit.db
+.agentkit/improve/<run>/
 ```
 
 Do not edit `.agentkit/agentkit.db` by hand.
+Do not edit `.agentkit/improve/<run>/bundle.json` by hand.
 
 ## Minimal Working Example
 
 Generated smoke eval:
 
 ```ts
-export default {
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
   name: "smoke",
   input: "Say hello in one short sentence.",
   expect: {
-    contains: "hello",
+    response: {
+      caseInsensitiveContains: "hello",
+      maxLength: 160,
+    },
   },
-};
+});
 ```
 
 Supported assertion types:
 
 ```txt
-contains
-not_contains
-regex
-matches_regex
-persisted_tool_call
+response.contains
+response.containsAll
+response.containsAny
+response.caseInsensitiveContains
+response.notContains
+response.regex
+response.matchesRegex
+response.notRegex
+response.maxLength
+tools.called
+tools.calledOnce
+tools.count
+tools.order
+tools.persisted
 ```
 
-`persisted_tool_call` validates the tool call saved in local SQLite `tool_calls`, not a provider-specific raw response shape. It can be a tool name string or an object with `name`, `input`, `output`, `status`, and/or `visibility`.
+Older flat aliases still work, including `contains`, `not_contains`, `regex`, `matches_regex`, and `persisted_tool_call`.
+
+For date-sensitive evals, set top-level `now` to an ISO timestamp with an explicit timezone designator such as `Z` or `-05:00`. AgentKit uses that fixed clock for every turn and tool call in the eval:
+
+```ts
+export default defineEval({
+  name: "appointment relative date",
+  now: "2026-02-04T02:30:00.000Z",
+  input: "What is today's date?",
+  expect: {
+    response: {
+      containsAll: ["2026-02-03", "Tuesday"],
+    },
+  },
+});
+```
+
+Multi-turn conversation evals use `turns`:
+
+```ts
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
+  name: "buyer under budget",
+  turns: [
+    {
+      input: "I want a house up to 600k near Pinheiros.",
+      expect: {
+        tools: {
+          calledOnce: "buscar_imoveis",
+          persisted: {
+            name: "buscar_imoveis",
+            status: "completed",
+            input: { maxPrice: 600000 },
+          },
+        },
+      },
+    },
+    {
+      input: "Show me the best two.",
+      expect: {
+        response: {
+          containsAll: ["Pinheiros", "R$"],
+        },
+      },
+    },
+  ],
+});
+```
+
+`tools.persisted` validates the tool call saved in local SQLite `tool_calls`, not a provider-specific raw response shape. It can be a tool name string or an object with `name`, `input`, `output`, `rendered`, `status`, and/or `visibility`.
+
+Use `tools.count` for the exact number of persisted calls in that turn, `tools.calledOnce` for exactly one call by name, and `tools.order` for required relative order. `tools.order` allows extra calls before, between, or after the named calls; pair it with `tools.count` when the exact call set matters.
 
 Use response assertions and persisted tool assertions together when internal operational output must not leak:
 
 ```ts
-export default {
+import { defineEval } from "@andreprado/agentkit";
+
+export default defineEval({
   name: "triage lead",
   input: '{"tool":"triage_real_estate_lead","input":{"email":"ada@example.com"}}',
   expect: {
-    not_contains: ["hot", "score"],
-    persisted_tool_call: {
-      name: "triage_real_estate_lead",
-      status: "completed",
-      visibility: "internal",
-      output: { status: "hot" },
+    response: {
+      notContains: ["hot", "score"],
+      notRegex: ["API_KEY|secret|token"],
+    },
+    tools: {
+      calledOnce: "triage_real_estate_lead",
+      persisted: {
+        name: "triage_real_estate_lead",
+        status: "completed",
+        visibility: "internal",
+        output: { status: "hot" },
+      },
     },
   },
-};
+});
 ```
 
-`tool_call` remains accepted as a backwards-compatible alias, but new evals should use `persisted_tool_call`.
+`tool_call` and `persisted_tool_call` remain accepted as backwards-compatible aliases, but new evals should use `tools.persisted`.
+
+Safe external-tool pattern:
+
+```ts
+export const sendFollowupEmail = defineTool({
+  name: "send_followup_email",
+  description: "Sends a follow-up email.",
+  inputSchema: {
+    type: "object",
+    properties: {
+      email: { type: "string" },
+    },
+    required: ["email"],
+    additionalProperties: false,
+  },
+  async execute(input: { email: string }, ctx) {
+    if (ctx.runtime.environment === "eval") {
+      return { sent: false, evalFixture: true, email: input.email };
+    }
+
+    // Real provider call here.
+    return { sent: true, evalFixture: false, email: input.email };
+  },
+});
+```
 
 ## Safety Rules
 
 - Do not put provider keys in eval files.
 - Do not put real client PII in eval files.
 - Prefer small deterministic assertions.
-- Mock tools for evals that would write, delete, charge money, or email people.
+- Keep eval tool calls deterministic and non-destructive.
+- For tools that would write, delete, charge money, send email, or call a real customer system, branch inside the registered tool on `ctx.runtime.environment === "eval"` and return safe fixture output.
 - Treat conversations as source material, not as automatically safe training data.
+- Review generated regression evals from `agentkit improve evals` before committing them. AgentKit redacts common email, phone, bearer token, and key patterns in generated eval text, but you must still remove or generalize domain-specific client PII and replace brittle exact prose assertions with the important behavior when needed.
 
 ## Verification
 
@@ -111,7 +238,7 @@ Use `test/fake` for deterministic smoke tests, then add provider-specific evals
 
 Tool eval hits a real API:
 
-Add tool mocks before running the eval suite.
+Make the registered tool return deterministic fixture output when `ctx.runtime.environment === "eval"`, then rerun the eval suite. AgentKit does not expose a separate eval-only mock registry yet.
 
 ## Backend Contracts Used
 

package/docs/guides/security-rules.md

@@ -29,7 +29,7 @@ Hosted alpha secret commands:
 
 ```sh
 agentkit login --token agk_user_...
-agentkit secret set OPENAI_API_KEY
+agentkit secret set OPENAI_API_KEY --from-local-env
 agentkit secret list
 agentkit secret unset OPENAI_API_KEY
 ```
@@ -93,15 +93,16 @@ defineTool({
 - `.env.schema` is the committed contract for local secret names.
 - AgentKit local commands load `.env` directly so inspect, chat, tools, and evals share the same secret loader.
 - Production uses managed secrets.
-- Hosted alpha deploys require `cloudflare_deploy_alpha`; local commands do not require login.
+- Hosted deploys require `cloudflare_deploy_alpha` or purchased/manual deploy slots; local commands do not require login.
 - Secret values must not appear in config, docs, prompts, evals, logs, exports, or SQLite.
 - A tool receives only secrets listed in that tool.
 - Avoid direct `process.env` reads inside tools.
 - Use `permissions` to describe external capabilities.
 - Add timeouts to network tools.
-- Treat public deploy URLs as unauthenticated transport, not access control.
-- Use access tokens and limits for clients.
+- Treat hosted deploy URLs as addresses, not access control.
+- Use deploy access tokens for hosted chat, hosted conversation reads, hosted trace reads, and any client app that talks to AgentKit Cloud.
 - Remove client PII before writing evals.
+- Review `.agentkit/feedback/` drafts before sending AgentKit product feedback. Do not paste `.env` values, provider keys, cookies, client PII, or full private transcripts into feedback messages.
 
 ## Verification
 
@@ -132,9 +133,9 @@ Add the secret name to the tool `secrets` field and set it locally:
 npm run agentkit -- inspect
 ```
 
-Public URL is reachable:
+Hosted URL is reachable without a token:
 
-Check hosted access mode. Default should be `private`.
+Treat this as a security bug. Hosted AgentKit Cloud routes should reject missing deploy tokens except authenticated channel webhook ingress.
 
 ## Backend Contracts Used
 

package/docs/guides/send-feedback.md

@@ -0,0 +1,135 @@
+# Send Feedback To AgentKit
+
+## Goal
+
+Create a redacted local feedback draft when AgentKit itself is confusing, missing docs, or failing, then send it to AgentKit Cloud only after the user has authenticated.
+
+## When To Use This
+
+Use this for AgentKit product feedback, not for the user's agent's client conversations.
+
+Good cases:
+
+- an AgentKit CLI command failed and the error did not explain recovery;
+- a docs guide or generated skill is missing a step;
+- deploy, channel, provider, eval, or runtime behavior looks like an AgentKit bug;
+- the user's coding agent found a feature gap in AgentKit itself.
+
+Do not use this to upload private client data, provider keys, `.env` values, or full conversation transcripts.
+
+## Commands
+
+Create a local draft only:
+
+```sh
+agentkit feedback create --about last-run --kind bug --summary "Deploy failed after secrets sync"
+```
+
+Preview a draft:
+
+```sh
+agentkit feedback preview .agentkit/feedback/<draft>.json
+```
+
+Send a saved draft:
+
+```sh
+agentkit login --token agk_user_...
+agentkit feedback send .agentkit/feedback/<draft>.json
+```
+
+Create and send in one command:
+
+```sh
+agentkit feedback send --about deploy --kind deploy_issue --summary "Deploy doctor passed but deploy failed" --message "The recovery step was unclear."
+```
+
+## Files Created Or Edited
+
+`feedback create` writes ignored local files:
+
+```txt
+.agentkit/feedback/<timestamp>-<summary>.json
+.agentkit/feedback/<timestamp>-<summary>.md
+```
+
+These files are local diagnostic drafts. Keep `.agentkit/` out of commits.
+
+## Minimal Working Example
+
+```sh
+agentkit feedback create \
+  --about last-run \
+  --kind missing_docs \
+  --summary "The channel debug guide did not explain Discord bot mode recovery" \
+  --message "The command failed after setup, and I could not tell whether to rerun connect or setup."
+```
+
+Review the printed JSON path, then send it:
+
+```sh
+agentkit feedback send .agentkit/feedback/<draft>.json
+```
+
+## Safety Rules
+
+- Creating a draft never sends network feedback.
+- Sending requires AgentKit Cloud login.
+- The CLI redacts common bearer tokens, AgentKit tokens, provider key patterns, and Authorization headers before saving or sending.
+- AgentKit Cloud validates the payload again and stores the redacted report under the authenticated account.
+- Do not paste `.env` contents, provider key values, OAuth tokens, cookies, client PII, or full private transcripts into `--message`.
+- The feedback command does not upload arbitrary files and does not ask AgentKit Cloud to fetch URLs.
+- The deployed agent runtime does not send feedback to the AgentKit team.
+
+## Verification
+
+After creating a draft:
+
+```sh
+agentkit feedback preview .agentkit/feedback/<draft>.json
+```
+
+Expected:
+
+- `schema_version` is `agentkit.feedback.v1`;
+- `source.channel` is `cli`;
+- `context` describes the current capsule and deploy shape;
+- no secret values are present.
+
+After sending:
+
+```txt
+Feedback sent
+ID:     fb_...
+Status: new
+```
+
+## Troubleshooting
+
+`auth_required`:
+
+Log in before sending:
+
+```sh
+agentkit login --token agk_user_...
+```
+
+`feedback_draft_invalid`:
+
+Recreate the draft with `agentkit feedback create`. Do not hand-edit the JSON schema unless you keep `schema_version: "agentkit.feedback.v1"`.
+
+`payload_too_large`:
+
+Shorten `--message`. Do not paste full logs; include the command, exact error code, and the smallest useful excerpt.
+
+## Backend Contracts Used
+
+Feedback submit uses authenticated AgentKit Cloud account auth:
+
+```txt
+POST /v1/feedback
+Authorization: Bearer agk_user_...
+Idempotency-Key: fbdraft_...
+```
+
+The endpoint accepts only structured JSON feedback and stores it in the AgentKit Cloud feedback inbox.

package/docs/guides/use-provider.md

@@ -6,7 +6,9 @@ Switch a capsule from the offline `test/fake` provider to a Pi-backed provider.
 
 ## When To Use This
 
-Use this when local fake responses are no longer enough and the agent needs model behavior from OpenAI, Anthropic, or OpenRouter.
+Use this when local fake responses are no longer enough and the agent needs model behavior from OpenRouter, OpenAI, Anthropic, or another supported provider.
+
+The coding agent should not choose a real provider automatically. Ask the owner which provider to use, then update the capsule.
 
 ## Commands
 
@@ -19,6 +21,14 @@ npm run chat -- --message "hello"
 npm run agentkit -- inspect
 ```
 
+On Windows PowerShell, if `npm.ps1` is blocked by `PSSecurityException`, use the Windows command shim:
+
+```sh
+npm.cmd run typecheck
+npm.cmd run chat -- --message "hello"
+npm.cmd run agentkit -- inspect
+```
+
 ## Files Created Or Edited
 
 Edit:
@@ -75,6 +85,19 @@ provider: {
 secrets: ["OPENROUTER_API_KEY"],
 ```
 
+Prefer model ids or aliases listed by the installed Pi SDK when available, such as `~google/gemini-flash-latest`. If an OpenRouter model id is newer than the Pi model registry, AgentKit passes the id through to OpenRouter using Pi's OpenAI-compatible transport with conservative unknown-model metadata. The provider may still reject the request if the id is invalid, inaccessible, or does not support the tools/features the agent uses.
+
+## UI Verification
+
+After the provider is configured and the local secret is set, test through chat and UI:
+
+```sh
+npm run chat -- --message "hello"
+npm run dev
+```
+
+Open the printed `Chat:` URL and tell the owner the exact URL. If the provider is still `test/fake`, say the UI was tested only with the deterministic fake provider.
+
 ## Safety Rules
 
 - Keep `.env` local.
@@ -84,6 +107,7 @@ secrets: ["OPENROUTER_API_KEY"],
 - Do not import provider SDKs in the Agent Capsule.
 - AgentKit uses Pi SDK internally.
 - Run `agentkit inspect` to confirm secret status without printing values.
+- Do not claim real conversation behavior was tested until a real provider selected by the owner is configured.
 
 ## Verification
 
@@ -111,7 +135,7 @@ npm run agentkit -- inspect
 
 `provider_model_unsupported`:
 
-Use a model id known to the installed Pi SDK for that provider.
+For OpenAI or Anthropic, use a model id known to the installed Pi SDK for that provider. For OpenRouter, prefer a known Pi alias when possible; otherwise a raw OpenRouter model id is passed through and any remaining model error comes from OpenRouter.
 
 Provider returns auth failure:
 
@@ -123,4 +147,4 @@ Check the key value, provider account status, and model access.
 
 ## Backend Contracts Used
 
-Local provider keys come from ignored `.env`, loaded directly by AgentKit. Hosted production will use `agentkit secret set <NAME>` and the secret API. Secret values have no readback in hosted responses.
+Local provider keys come from ignored `.env`, loaded directly by AgentKit. Hosted production uses `agentkit secret set <NAME> --from-local-env`, `--from-env`, or `--stdin` and the secret API. Secret values have no readback in hosted responses.