@flow-engineer/flow 0.0.1 → 0.2.0

package/coding-agents/skill.md

@@ -0,0 +1,321 @@
+---
+name: flow-framework
+description: Build or change a chat agent on Flow Framework (WhatsApp, SMS, iMessage, Telegram, web). Load when creating a Flow Framework agent, editing its prompt or skills, wiring it into the playground, or when code imports @flow-engineer/flow (formerly chinwag). Also load the flow-framework-tools skill (tools-skill.md) before writing tools and flow-framework-tests (test-skill.md) before writing tests.
+---
+
+# Building an agent on Flow Framework
+
+Flow Framework runs chat agents that feel like a person texting back: it folds double texts into one
+reply, acknowledges people fast, runs tools in parallel and waits for them, moving one to the
+background only when the person writes again or it passes its limit.
+You write the agent's prompt, skills and tools; Flow Framework handles the conversation.
+
+Two more skills go with this one: `flow-framework-tools` (`tools-skill.md`) before writing or changing a
+tool, and `flow-framework-tests` (`test-skill.md`) before writing or changing a test.
+
+## Start a new agent
+
+`npx @flow-engineer/flow new <name>` (lowercase letters, digits and dashes) makes `./<name>/` from the café
+example: a complete agent with tools and tests that depends on the `@flow-engineer/flow` package and imports
+`@flow-engineer/flow` and `@flow-engineer/flow/simulators`, with these skills in `.claude/skills/`. Run `npm install` there.
+Build in it, changing Pip's prompt, skills, tools and tests into yours. The owner copies
+`.env.example` to `.env` and puts one model key in it (never read or print it). Flow Framework does not
+issue that key; it is made at the provider: `OPENROUTER_API_KEY` at https://openrouter.ai/keys,
+`OPENAI_API_KEY` at https://platform.openai.com/api-keys, `ANTHROPIC_API_KEY` at
+https://console.anthropic.com/settings/keys, or `GEMINI_API_KEY` at https://aistudio.google.com/apikey.
+The first one with a value is used, and `CAFE_MODEL` sets the café's model id. Then `npm start`
+opens the playground; `npm run typecheck` checks the types.
+
+Inside a clone of Flow Framework itself, `npm run new -- <name>` copies the café to `agents/<name>/`
+instead (git-ignored, so it never collides with a pull). Its imports of `../../../src/` and
+`../../../simulators/src/` stay as they are, so changes to Flow Framework show at once; the key goes in
+`agents/<name>/.env`. The snippets below import `@flow-engineer/flow` and `@flow-engineer/flow/simulators`; in a clone's
+agent, keep its relative imports.
+
+## Layout
+
+```
+my-agent/
+  agent/AGENTS.md              the agent's prompt (the model reads it every turn)
+  agent/skills/<name>/SKILL.md skills the agent loads when it needs them
+  src/tools.ts                 the agent's tools (see tools-skill.md)
+  src/agent.ts                 buildAgent(): the one place the agent is built
+  src/main.ts                  runs it in the playground
+  tests/tools/*.yaml           how each tool behaves in tests (see tools-skill.md)
+  tests/scenarios/*.yaml       the tests (see test-skill.md)
+```
+
+`agent/AGENTS.md` is the agent's prompt, not instructions for coding agents. Keep it short: what
+the agent is for, what it may and may not do. Don't explain messaging in it: Flow Framework adds how to
+talk to the person (send_<channel>, react, acknowledging first, channel formatting) on its own.
+
+A skill's `SKILL.md` starts with `name` and `description`; the agent sees every description and
+loads the full skill only when it decides it needs it. Put rules for one kind of request in a
+skill, not in AGENTS.md.
+
+## buildAgent
+
+Build the agent in one function that the playground and the tests both call, so they never
+drift apart. Tests pass their own model, events and simulated tools:
+
+```ts
+import { fileURLToPath } from "node:url";
+import { createAgent, openAICompatible, type EventSink, type Model, type Tool } from "@flow-engineer/flow";
+import { tools as myTools } from "./tools.ts";
+
+export const tools = myTools;
+
+export function buildAgent(options: { model?: Model; events?: EventSink[]; tools?: Tool[] } = {}) {
+  return createAgent({
+    dir: fileURLToPath(new URL("../agent", import.meta.url)),
+    model: options.model ?? openAICompatible({ model: "anthropic/claude-haiku-5.5" }),
+    tools: options.tools ?? myTools,
+    events: options.events ?? [],
+  });
+}
+```
+
+Other `createAgent` options: `files` replaces `dir` on hosts with no file system (the folder's
+text by path: `{ "AGENTS.md": ..., "skills/<name>/SKILL.md": ... }`); `profile({ conversationId, person, channel })` is what the business knows about
+the person, offered as the built-in tool `get_user_profile` and run only when the agent calls it (put
+lookups here, not in `context`); `context(conversationId)` adds small facts to the system prompt at
+every turn, before every reply, so it must not call APIs; Flow Framework already tells the agent the time; `intercept(conversationId, text, message)` answers a
+message without the model (return the reply, `""` for silence, or `null` to let the agent answer;
+the message and the reply are kept in the history, so the next turn knows, and in the transcript,
+and are `message.in`/`message.out` events marked `answered: true`);
+`deadlineMs` (default 60,000) is the longest a reply waits for a tool before it moves to the
+background (set a tool's own `deadlineMs` for a known-slow one); `partialAfterMs` (default 5,000)
+is how long a finished tool's result waits for the others of its response; `shareUpdates` turns live status lines on web and app UIs off, or replaces their
+guidance with a string.
+
+Models are reached through any OpenAI-compatible API. `openAICompatible()` with no key reads the
+first of `OPENROUTER_API_KEY`, `OPENAI_API_KEY`, `ANTHROPIC_API_KEY` and `GEMINI_API_KEY` that has a
+value, and uses that provider's default model (`MODEL` overrides it). With none it stops at once and
+names them. Pass `apiKey` and `baseURL` for anywhere else (`baseURL` alone for a local server that needs no key).
+
+Which model: recommend Claude Haiku 5.5 (`anthropic/claude-haiku-5.5`, the default above) or OpenAI
+GPT-6.1 Sol (`openai/gpt-6.1-sol`, about 3x the cost, the most exact). If the owner asks for Luna
+(`~openai/gpt-luna-latest`, `openai/gpt-6-luna`), advise against it before using it: with tools
+that run in the background it repeats tool calls, sends extra "on it" messages and answers twice,
+where Haiku and Sol send one acknowledgement and one answer (measured on Flow Framework's own checks, 12 Oct
+2026). Use it only if they still want it.
+
+## Channels, people and the database
+
+In production, give the agent its channels once and hand it each incoming message. Each channel
+is a definition in its own file; Flow Framework knows the platform (WhatsApp's limits, SMS segments), your
+provider only says how to send:
+
+```ts
+// src/channels/whatsapp.ts: one sendMessage, plus whatever else your provider can do
+export const wa = whatsapp({
+  sendMessage: (to, m) => api.send(to, m),        // m: { text, replyTo?, buttons?, list?, link?, image? }
+  react: (to, id, emoji) => api.react(to, id, emoji),
+  // no typing() or markRead(): those features are off, and the agent is never offered them
+}, { disable: ["lists"] });
+
+// src/channels/app.ts: your own web or mobile UI, with params of your own
+export const app = customUI("app", {
+  params: { app_link: z.string().describe("A link a tool result gave you; the app shows it as a button") },
+  instructions: "Only fill app_link with a URL from a tool result in this conversation; never make one up.",
+  sendMessage: (to, m) => myApp.push(to, m),      // m.text, m.app_link
+});
+
+createAgent({ ..., channels: [wa, sms(twilioLike), app], database: sqlite("data/chinwag.db"),
+  resolve: ({ channel, address }) => members.idFor(channel, address), timeZone: "America/New_York" });
+// A person's time zone is never guessed from timeZone (the business's): set it when you know it,
+// agent.setTimeZone({ to, timeZone }) or peopleHandler over HTTP; else the agent asks them.
+await agent.resume();                     // a durable database: always before the first message
+await agent.message({ channel: "whatsapp", from: "+15550100100", message: "hi" });
+```
+
+- Several processes or machines need Postgres, not a SQLite file: `database: postgres(pool)`
+  with the app's own node-postgres `Pool` (Flow Framework adds no Postgres dependency). An agent that must
+  not hold database credentials gets `store` instead of `database`: any object with Store's
+  methods (`StoreApi`), such as a client that forwards each call to a service running `new
+  Store(postgres(pool), agentId)`.
+- Every send tool can carry a `link` ({ label, url }, https): a URL button where the channel has
+  one (`linkButton`, WhatsApp's cta_url), else Flow Framework puts the URL on a line after the text. Channels
+  with `images` also offer an `image` ({ url }, https), the text its caption; elsewhere the text
+  goes alone. WhatsApp's rules hold everywhere: a link never with buttons or a list, an image never
+  with a list. A provider without them says so (`features: { links: false, images: false }`) and
+  never gets them; the URLs come from your tools' results, like app_link's.
+- The simulators and the test cloud play an agent's own channels with these definitions (the web
+  simulator plays a customUI), so the model sees the production tools; they show params such as
+  app_link under the message instead of calling your sendMessage.
+- `instructions` (on `customUI`, and in `whatsapp()`'s and `sms()`'s options) is your guidance for
+  the agent on that channel, in its prompt only while the person uses it. Data a param carries,
+  such as a link, comes from your tools' results: a tool that has a screen returns its link
+  (`{ ..., app_link }`), and the agent only passes it on.
+- One person is one conversation across every channel they use. Channels are joined only by what
+  each verified: your `resolve` key, else the same phone number on channels marked `phone`. Never
+  join people any other way. When the SMS provider gives national numbers, set
+  `defaultCountryCode` ("+91") so they match the full number WhatsApp gives.
+- Tools key a person's data by `context.person`, their stable id: `key:<resolve's value>`, else
+  `phone:<E.164>`, else `<channel>:<address>`. `context.key` is `resolve`'s value as returned, or
+  null without one: look them up in the backend by that, never by `context.person`.
+- With several channels the agent gets a send tool per channel (`send_sms`, `send_web`) and
+  answers each message where it came. Don't add your own.
+- Run exactly one copy of the agent per database (one instance, kept warm, CPU always on): a
+  conversation lives in one process, so two copies would run overlapping turns for it.
+- After a restart, `resume()` carries on: cut-off turns run again, background tools that were
+  running are reported as interrupted (never rerun).
+- `agent.receive(conversationId, message, deliver)` still works for one channel at a time (the
+  test cloud uses it).
+- Photos, voice notes and documents: hand them in on the message, `message: { text: caption, attachments: [{ kind: "image" | "audio" | "document", mime, name?, data: () => provider.download(id) }] }`.
+  `data` is a loader your provider fetches with its own credentials; never put a media URL where
+  the model sees it. Photos (JPEG, PNG, WebP, GIF, up to 5 MB) and PDFs (up to 10 MB) go to the
+  model when it reads them (`openAICompatible({ images, pdfs })`: images default on, PDFs on with
+  OpenRouter; set `images: false` for a text-only model). Voice notes need
+  `createAgent({ transcribe: async ({ mime, data }) => text })` with the speech-to-text the owner
+  picks; Flow Framework has none. Anything the agent can't read reaches it named, with why, so it asks
+  for text. The latest 3 photos and PDFs stay shown; their bytes are never stored.
+
+Built in, so don't build them as tools: what the business knows about the person (`profile`, as
+`get_user_profile`), scheduled messages (`schedule_message`, `list_scheduled`,
+`cancel_scheduled`, in the person's local time), and what a person asked not to be messaged about
+(`add_no_contact`, `remove_no_contact`), quiet hours they set (`set_quiet_hours`) and pauses
+(`pause_messages`). STOP, START and HELP are built in on `sms()` and
+`whatsapp()`: don't build them in `intercept`. STOP stops every message the business starts, with
+no model call, until START; set the replies with `keywords: { replies: { stop, start, help } }`.
+`intercept` sees them first, with `message.keyword` `"stop"`, `"start"` or `"help"` (else `null`)
+and `message.channel` (the channel's name in `channels`, on every message): an agent with its own
+opt-out flag sets it there
+(`if (message.keyword === "stop") optedOut.add(conversationId)`; only on SMS:
+`message.channel === "sms" && ...`). Flow Framework keeps or lifts the STOP
+whatever the hook returns or throws; a reply it returns goes instead of Flow Framework's, never both.
+SMS takes only STOP by default; the words carriers or regulators expect differ by country, so
+check the market's rules (`keywords: US_CARRIER_KEYWORDS` adds STOPALL, UNSUBSCRIBE, CANCEL, END
+and QUIT, as some US carrier rules expect; `keywords: { stop: [...] }` sets your own).
+
+Messages the business starts are nudges: `agent.nudge({ to: { channel, address }, text })` sends
+as written; `prompt` instead lets the agent write it (or decide not to); `template` sends a
+WhatsApp template you already set up. Pass `idempotencyKey` when your system may retry, and `at`
+to send later. Prefer `prompt`; use `text` only when the wording must be exact. Outside WhatsApp's
+24-hour window only templates go out: declare a `fallbackTemplate` ("You have a new message. Reply to
+see it.", no values) so a nudge is held until they reply instead of refused (`outside_window`); the
+agent may send a fitting template itself (`send_template`), and `template.missing` events say which
+messages want one. Held nudges (quiet hours, a closed window) settle as `held`, then `sent` or `expired`.
+
+When the owner's backend is a separate service, mount `nudgeHandler(agent, { token })` (from
+`@flow-engineer/flow`; a token of at least 32 characters, from a secret; `base` is `/nudges` by default) in the
+agent's server rather than writing a route: `POST /nudges` takes the same fields with
+`idempotency_key` and `at` in ms since the epoch, and answers `202 { id, status }` once recorded,
+before any send or turn (`queued`, or `refused`/`dropped`/`scheduled` decided then: still a 202);
+`GET /nudges/<id>` reads its state. Both need `Authorization: Bearer <token>`. Contract:
+`spec/nudges.openapi.yaml`.
+
+Templates are WhatsApp only. Declare the ones the owner set up with Meta beside `testPeople`, so
+Flow Framework checks each send against them and the playground shows them as WhatsApp does:
+
+```ts
+createAgent({
+  ...,
+  templates: [{
+    name: "order_ready", language: "en", category: "utility",   // or marketing, authentication
+    header: { kind: "text", text: "Order {{1}}" },              // or { kind: "image" | "video" | "document" }
+    body: "Hi {{1}}, your {{2}} is ready.",
+    footer: "Reply STOP to opt out",
+    buttons: [
+      { type: "quick_reply", text: "On my way" },
+      { type: "url", text: "Track", url: "https://example.com/o/{{1}}" },
+      { type: "phone", text: "Call us", phone: "+15555550100" },
+    ],
+  }],
+});
+
+await agent.nudge({ to, template: {
+  name: "order_ready",
+  header: { kind: "text", text: "1042" },                     // media: { kind: "image", url } or { kind, fileId }
+  body: ["Maria", "latte"],                                   // {{1}}, {{2}}, ... in order
+  buttons: [{ index: 0, value: "omw_1042" }, { index: 1, value: "1042" }],   // a quick reply's payload, a URL's end
+} });
+```
+
+A declared template sent with the wrong values (a body value too few or many, a missing header or
+URL value) throws from `nudge` before anything is sent, in the playground and live alike. One not
+declared goes as it is (the provider checks it) and the playground shows its name and values. A
+quick-reply tap comes back as the person's message with `choice: { id: payload, title }`, like a
+reply button, and the agent is told what the template said. An ordinary message has one kind of
+button: up to 3 reply buttons OR one link button, never both. A template may mix quick replies and
+URL buttons.
+
+## Ask the owner when you set an agent up
+
+Before the agent sends anything people did not ask for (nudges, follow-ups), ask the owner these
+two, with these options, and build what they pick:
+
+1. **Quiet hours** (what the business starts waits for them, in each person's own time):
+   "22:00–07:00 (recommended)", "other hours", or "none" (`quietHours: false`). For US marketing,
+   say that 8am–9pm the recipient's time is, as far as we know, the legal window, and check theirs.
+2. **Where people's time zones come from**, since quiet hours apply only where it is known:
+   "our database" (sync it with `agent.setTimeZone` or `peopleHandler`), or "we don't know" (the
+   agent asks people when it needs it; until then nothing is held for quiet hours).
+
+## The playground
+
+Every agent ships a test playground: people test it as real users before its numbers are live.
+Declare who may be played in `createAgent({ testPeople: [{ name: "Maria", phone: "+15555550101" }] })`;
+no one else can be. A test person is a real person to the agent: the same identity, `resolve`,
+`intercept`, history and schedules as when that number writes through a provider (the playground
+is just another phone; they may be test members in your real backend). On phone channels their
+address is the number; on others `test:<number>`, joined to the same person. One person is one
+conversation across every channel: switch channels and the chat resumes. A listed number may also
+write from a real provider once a channel is live; replies, scheduled messages and nudges go back
+wherever that person last wrote from on that channel.
+
+`src/main.ts` serves it with the simulators (SMS, WhatsApp, and the web simulator for a custom UI),
+each with that platform's real rules, on the agent's own `channels` (declare them, with a provider
+that is not connected yet if the numbers are not ready: test people never reach it):
+
+```ts
+import { simulators } from "@flow-engineer/flow/simulators";
+import { buildAgent } from "./agent.ts";
+
+const play = simulators({ title: "My agent" });
+const agent = buildAgent({ events: [play.events] });
+await play.listen(agent, 8788);           // 127.0.0.1 only, its own page only; no token needed there
+```
+
+Open the printed link, pick a test person and a channel, and chat; `/person` shows every channel
+side by side. The page shows the person's conversation from the agent's store, so it survives
+restarts (with a durable `database`), and Clear history forgets the person entirely
+(`agent.forgetPerson(conversationId)`: history, schedules, everything). The side panel shows how
+fast each message was acknowledged and a trace of every model call and tool. `cards` says how a
+custom UI shows a link in its params: `{ match: "/video/", label: "Start video call", kind: "call" }`;
+any other link is "Open in app".
+
+The Person tab also moves the agent's clock when you give it one (`const clock = playgroundClock()`,
+`simulators({ clock })` and `createAgent({ now: clock.now })`), lists what is scheduled or held for
+the person, sets their time zone and quiet hours, and sends nudges with every field the API takes.
+
+To demo what your backend does (an order ships), send the nudge it would send from the Person tab.
+
+On a deployed agent, mount it in your own server with a token of at least 32 characters, from a
+secret: `const playground = play.handler(agent, { token: process.env.PLAYGROUND_TOKEN!, base: "/playground" })`,
+and hand it the requests under `/playground` (`if (url.pathname.startsWith("/playground")) return playground(req, res)`).
+Open `/playground/?token=...` once (it becomes an HttpOnly cookie and leaves the URL) or call the API
+with `Authorization: Bearer <token>`. Wrong tokens are limited per address, and every simulated
+message is logged with the person, channel, time and address.
+
+## Send traces to a dashboard
+
+Every turn can go to a traces endpoint as an OTLP/JSON trace, posted to `<url>/v1/traces`, so
+playground and real chats show up in a dashboard and can be graded. No code: set these in the
+agent's environment. The address is https://flow.engineer/observe unless `FLOW_FRAMEWORK_TRACES_URL`
+names another; the trace key is made on that same site (never print it):
+
+```sh
+FLOW_FRAMEWORK_TRACES_KEY=...
+FLOW_FRAMEWORK_TRACES_URL=https://flow.engineer/observe  # the default; set it only to send them elsewhere
+FLOW_FRAMEWORK_TRACES_CONTENT=1                          # send what was written; without it, metadata only
+```
+
+Without `FLOW_FRAMEWORK_TRACES_CONTENT=1` the dashboard
+gets ids, timing, model calls and tool names but no messages, so nothing to show or grade. Or in
+code: `createAgent({ ..., traces: { url, key, content: true, version: "v3" } })`. Traces are sent in the
+background after each turn and a failed send is dropped (logged once), so a dashboard outage never
+touches a chat; call `await agent.flushTraces()` before the process exits. Tests run under
+`node --test` never send from the environment. Under `flow test` the traces say they are a test's
+(`chinwag.run_kind` is "test", else "production", and each turn names the test session as
+`chinwag.test.session`), so the dashboard keeps them apart from real conversations.

package/coding-agents/test-skill.md

@@ -0,0 +1,223 @@
+---
+name: flow-framework-tests
+description: Write or change tests for a Flow Framework agent (tests/scenarios/*.yaml). Load when adding a test scenario, deciding what a Flow Framework agent should be checked on, or turning a bad conversation into a test.
+---
+
+# Writing tests for a Flow Framework agent
+
+Read the `flow-framework` skill (`skill.md`) first if you haven't.
+
+A test is a scenario: a situation to put the agent in, played through a platform simulator by a
+model acting as the person, with the agent's tools played by a model from `tests/tools/*.yaml`
+(see the `flow-framework-tools` skill, `tools-skill.md`). At the end, what happened is checked against `expect`.
+
+```yaml
+id: latte-correction
+title: Orders a latte, corrects the milk mid-thought, confirms
+channels: [whatsapp, sms]   # runs once on each, with a result for each
+
+person:
+  persona: Sam, in a hurry on the way to work; types in short bursts, no capitals.
+  goal: Get a large oat latte for pickup, under the name Sam.
+  knows:
+    - Their name is Sam.
+
+opening:
+  - say: "can i get a large latte"
+  - say: "oat milk actually"
+    after_ms: 1200
+
+world: |
+  The menu is as in tests/tools/get_menu.yaml. Oat milk is in stock. The café is busy this
+  morning, so the point of sale is slow to take orders, but they go through.
+
+max_turns: 8
+
+expect:
+  outcome:
+    - Exactly one order is placed, for one large latte with oat milk, under the name Sam.
+  judge:
+    - The order read back to Sam uses oat milk, not whole milk.
+    - The agent asks Sam to confirm before placing the order.
+    - Sam is told the order number and when it will be ready.
+```
+
+## The parts
+
+- `channels`: one or more of sms, whatsapp, imessage, telegram and web; each is a run of its own.
+  The run uses that simulator's real rules (SMS has no buttons or reactions; WhatsApp has both).
+- `person`: who the model playing the person is (`persona`), what they want (`goal`), and facts
+  they only give if asked (`knows`). It sees only what a person would see.
+- `opening`: the first messages, sent exactly as written; `after_ms` sends one that long after
+  the previous one (a double text). Then the person plays on until `goal` is met or `max_turns`
+  (default 8). A run that uses every turn never passes.
+  A step may send a file: `- { say: "here is my receipt", attach: { kind: image, file: fixtures/receipt.jpg } }`
+  (`kind`: image, audio or document; `file` a path in `tests/`; `say` is its caption and may be
+  left out). The file is read on your machine and never uploaded; the person and the judge see it
+  by name only.
+- `world`: everything true in this test, in plain words. It is the only place the situation is
+  set: stock, outages, slowness, the person's history. The tool simulator plays every tool from
+  it, choosing among the responses in `tests/tools/*.yaml`. Never write tool responses or
+  latencies in a scenario. An agent with a `profile` gets it played from here too
+  (`get_user_profile`): say who the person is ("Sam, a member since 2024, on the monthly plan").
+- `expect`:
+  - `outcome`: what must have happened, checked against what the tools did ("exactly one
+    order is placed", "no order is placed").
+  - `judge`: what the conversation must show, one checkable sentence each; a judge model reads
+    the transcript and marks each pass or fail with a reason.
+  - `calls`: tools that must be called at least once (whatever the call returned); `never`: tools
+    that must not be called. Checked by code from the recorded tool calls, not by a model. Each
+    entry is a tool name, or `{ tool, args }`: the call matches when every field in `args` equals
+    the call's input (nested objects match in part; arrays and strings must be equal).
+- `timezone` (default UTC) and `now`: the test's clock. `now` is when each run starts,
+  `YYYY-MM-DDTHH:MM[:SS]`, local to `timezone` unless it ends in `Z` or an offset (`-04:00`); left
+  out, it is the real time. Placeholders are filled from it, and the simulated tools and the judge
+  are told it; your agent's clock is not changed: the agent still reads the real time. So set `now` only when the agent takes
+  its dates from its tools or its context; otherwise leave it out and placeholders follow the real
+  date.
+
+```yaml
+timezone: America/New_York
+now: 2026-10-16T09:00
+world: Sam's class is on {{now + 6d @ 10:00}}. There is a free slot s-2 the day after.
+expect:
+  calls:
+    - cancel_session
+    - tool: move_session
+      args: { slot_id: s-2 }
+  never:
+    - delete_account
+```
+
+Dates in test files are placeholders for the test's clock, never written out (a written date goes
+stale): `{{now}}`, then shifts `+` or `-` a number with `m`, `h`, `d` or `w`, then `@ HH:MM` for the
+time of day, then `| iso` (the default, with its offset), `| date` (2026-10-16), `| time` (09:00) or
+`| weekday` (Thursday): `{{now + 7d | date}}`, `{{now - 2h + 30m | time}}`. They work in `world`,
+`person`, `opening`, `expect` and the tool files' `responses`; the cloud fills them on each run. Any other
+`{{...}}` is plain text.
+
+How fast the agent first replies to each message is measured on every run from timestamps, not
+written in scenarios: the dashboard shows it for the whole run (average, p95, min and max).
+
+## Choosing what to test
+
+- One situation per scenario, named for it (`order-fails`, `latte-correction`).
+- Test what would hurt a real person if it broke: wrong order, a claim the order was placed when
+  it wasn't, a missed escalation, a slow or missing acknowledgement.
+- One fact per check. "Called X" or "never called Z" is a `calls` or `never` entry, not an
+  `outcome` or `judge` sentence: code checks it the same way every run.
+- Write `judge` items as facts a reader can check from the transcript, not taste ("asks Sam to
+  confirm before placing the order", not "is helpful").
+- Cover the unhappy paths the tool files list (busy, down, unknown item) as their own scenarios.
+- When a real conversation went wrong, turn it into a scenario: its world is what was true then,
+  its opening is what the person sent.
+
+## Running the tests
+
+Tests run in the Flow Framework cloud: your agent stays on your machine with its own model and key and
+connects out to the cloud, which plays the person and the tools and judges.
+
+1. Have a test cloud: the hosted one at https://flow.engineer/observe, the default (see "Get a test
+   cloud" below), or another with `--cloud <url>` or `FLOW_FRAMEWORK_CLOUD`.
+2. From the agent's folder (it must export `buildAgent` and `tools` from `src/agent.ts`), which reads
+   the agent's `.env` itself: `npx flow test --synthetic [--runs 3]` (`npm test -- --synthetic`
+   in an agent `flow new` made).
+3. It first reads every test file as the cloud does; a mistake stops it with
+   `tests/scenarios/<file>.yaml:<line>: <field>: <what is wrong>`, one line each, and nothing is sent.
+   Then it prints PASS or FAIL per run, with its run id, every failed check and the judge's reason,
+   and a link to watch each conversation in the simulator. The cloud's dashboard, at its own
+   address, shows every session over time and each scenario's history.
+4. `--json` prints one JSON report on stdout at the end, whatever the exit (everything else goes to
+   stderr): `outcome`, `exitCode`, `sessionId`, `agent`, `error`, `counts`, `costUsd` and per run
+   `runId`, `scenario`, `channel`, `run`, `verdict`, `error` (the cloud's code when it could not play
+   or judge the run, such as `judge_failed`; the run is then `errored` and its line reads `ERROR`,
+   whatever it ended as), `ended`, `turns`, `watch` and `failedChecks` (`kind`, `text` and the judge's
+   `reason`). Each session that opened also leaves
+   `.flow/last-session.json` in the agent's folder (its path is printed on every exit): `sessionId`,
+   `agent`, `dashboard`, `suiteHash`, `endedAt`, `ended` (`finished`, `stopped` or `interrupted`),
+   `only` (the scenarios it was limited to, else `null`) and per run `scenario` (file name without
+   `.yaml`), `channel`, `run`, `runId`, `verdict` (`passed`, `failed`, `errored`: the cloud could not
+   play or judge it, such as a failed tool simulator or person model, which is not a failure of your
+   agent, or `not_played`: the session ended first) and `failedChecks` (with reasons).
+   Read these instead of parsing the printed lines.
+   Exit status: 0 all passed; 1 a run failed, or nothing was sent (a wrong flag, a test file that does
+   not parse, no key); 2 the connection or the cloud failed, or runs errored and none failed; 3 the
+   cloud refused (key, busy, spend cap, capacity); 130 Ctrl-C. On 2 or 3 the agent is not at fault:
+   say what the cloud said (`error` in the report), do not change the agent for it.
+5. `--only a,b` runs only those scenarios (file names without `.yaml`). `--failed` runs only the ones
+   that did not pass in the last session, on every channel each plays; it goes with `--runs`, never
+   with `--only`. It prints `Nothing failed in the last session (<id>).` and exits 0 when there is
+   nothing to run again, and exits 1 when there is no last session.
+
+Work in this loop, and never finish on a partial run:
+
+1. `npx flow test --synthetic`: the whole suite.
+2. Fix what failed (the agent, or a test that asks for the wrong thing; say which).
+3. `npx flow test --synthetic --failed`, again until every run in it passes (exit 0).
+4. `npx flow test --synthetic`: the whole suite once more, since a fix can break another scenario.
+
+### When a run fails
+
+1. `npx flow test --synthetic --json > report.json`, and read the runs whose `verdict` is not
+   `passed` (`jq '.runs[] | select(.verdict != "passed")' report.json`).
+2. `npx flow show last` (the first run that did not pass; it names the others) or
+   `npx flow show <runId>`: the checks with the judge's reasons, then the conversation in time
+   order with every tool call the simulator played, its input, output and what it changed.
+   `--json` gives the whole record.
+3. Decide which is wrong, and say which:
+   - the agent: it said or did the wrong thing; fix its prompt, tools or code;
+   - the test: a check asks for the wrong thing, or the scenario cannot reach its goal; fix the YAML;
+   - the simulator: a tool output that could not happen, or a person who left the scenario; fix
+     `tests/tools/*.yaml` or the scenario's `world` and `person`.
+4. `npx flow test --synthetic --failed`, then `--runs 3 --only <scenario>` on what you fixed, so
+   a pass is not luck.
+5. The whole suite once more.
+
+### Get a test cloud
+
+The owner does these; never ask for the key in chat or print it:
+
+1. Sign in at https://flow.engineer/observe (Google, GitHub or an email link).
+2. Workspace → New agent.
+3. On the agent's Developers page, "Make a CI key". It starts with `oci_` and is shown once.
+4. Put it in the agent's `.env` as `FLOW_FRAMEWORK_CLOUD_KEY` (never on the command line).
+5. `npx flow test --cloud https://flow.engineer/observe --synthetic` from the agent's folder (`--cloud` is that address by default).
+
+Your agent runs on your machine with your own model key. The cloud's person simulator, tool simulator and judge are free up to $5 a day per workspace, with 2 sessions at once; test data only (`--synthetic`), and records are deleted after 30 days. `--synthetic` is the promise that the test files hold no real customer or member data;
+the CLI refuses a hosted cloud without it and the key. `--environment staging` or `production` is
+kept with the session.
+
+The open goes to the front door with the key; the session then plays at the tunnel the cloud
+answers, with the session's own token. `--agent-version` (else `GITHUB_SHA`) and `--ref` (else
+GitHub's branch) are kept with the session. A CI step (GitHub Actions shown):
+
+```yaml
+- run: npx flow test --cloud https://flow.engineer/observe --synthetic --environment staging
+  env:
+    FLOW_FRAMEWORK_CLOUD_KEY: ${{ secrets.FLOW_FRAMEWORK_CLOUD_KEY }}
+    OPENROUTER_API_KEY: ${{ secrets.OPENROUTER_API_KEY }}
+```
+
+Only the files in your `tests/` folder run; the cloud keeps each file by its hash and compares
+runs only while the tests are the same. Every run calls real models, so it costs a little.
+
+## An agent on another framework, in any language
+
+An agent not built on Flow Framework is tested through an HTTP endpoint: `npx @flow-engineer/flow test --agent-url <url>`.
+To implement one:
+
+1. Read the contract, `spec/agent-endpoint.openapi.yaml` (in the @flow-engineer/flow package or the repository),
+   with `spec/request.schema.json` and `spec/response.schema.json`. Implement exactly that: one
+   POST per message of the person's, answered in the same response with status 200 and
+   `{ "replies": [...] }` (`[]` for no reply, never 204). Keep history by `conversationId`, ignore
+   request fields you do not know, and check `authorization: Bearer` when `FLOW_FRAMEWORK_AGENT_TOKEN` is set.
+2. Only send what the run's platform shows (`x-chinwag-platforms` in the spec): no buttons on SMS,
+   text within `maxLength`. Name the person's messages by the `messageId` they came with.
+3. Start the endpoint and run `npx @flow-engineer/flow check-endpoint <url>` (`--token-env NAME` when it needs
+   a token). Fix every FAIL line; it must end "5 of 5 cases passed". Give the agent a mode with no
+   model call (the examples' `FAKE_MODEL=1`) so the check costs nothing.
+4. Then write scenarios as above, with one difference: the agent's own tools run for real, so there
+   are no `tests/tools/*.yaml`, no `world` for tools and no `outcome`, `calls` or `never` checks
+   (the cloud sees no tool calls); write what must be true as `judge` items. Run `npx @flow-engineer/flow test --agent-url <url>`.
+
+`examples/http-agent-python` and `examples/http-agent-go` are the café's agent done this way.

package/coding-agents/tools-skill.md

@@ -0,0 +1,89 @@
+---
+name: flow-framework-tools
+description: Write or change a Flow Framework agent's tools and their test files (tests/tools/*.yaml). Load when adding a tool to a Flow Framework agent, changing what a tool returns, or describing how a tool behaves in tests.
+---
+
+# Writing tools for a Flow Framework agent
+
+Read the `flow-framework` skill (`skill.md`) first if you haven't. Tests that use these tool files are
+in the `flow-framework-tests` skill (`test-skill.md`).
+
+## The tool
+
+```ts
+import { tool, z } from "@flow-engineer/flow";
+
+export const findSlots = tool({
+  name: "find_session_slots",
+  description: "Every open time to move the session to, for a date range.",
+  input: z.object({
+    from_date: z.string().optional().describe("First day to search, YYYY-MM-DD; default today"),
+    to_date: z.string().optional().describe("Last day, YYYY-MM-DD; default a week later"),
+  }),
+  run: async ({ from_date, to_date }, { conversationId }) => backend.findSlots(conversationId, from_date, to_date),
+});
+```
+
+- The name and description are what the model reads to choose the tool: say what it does and
+  what it returns, in one line.
+- `input` is a zod schema; Flow Framework checks the model's input against it before `run` is called and
+  sends the model the error if it doesn't fit.
+- `run` returns any JSON-able value; throw to report a failure (the agent sees `Failed: <message>`).
+- Don't make tools for talking to the person (send_<channel>, react, share_updates), for reminders
+  or scheduled messages, or for opting out of messages: Flow Framework has them.
+- Key a person's data by `context.person` (who Flow Framework verified they are) or `context.conversationId`,
+  never by an id the model passed. `context.person` is `key:<resolve's value>`, else
+  `phone:<E.164>` (`phone:+15550100100`), else `<channel>:<address>`. To look the person up in the
+  owner's backend use `context.key`: exactly what `resolve` returned (`customer-7`), or null when
+  they are known only by phone or channel.
+- A tool that changes something (books, pays, sends) passes `context.idempotencyKey` to its backend:
+  it is the same for the same input in the same turn, also when the turn runs again after a
+  restart, so the booking happens once. `context.callId` is the model's id for the call;
+  `context.signal` aborts when the person is forgotten.
+- Don't decide what runs in the background. The reply waits for every call; one moves to the
+  background only if the person writes again meanwhile or it passes its limit (the agent's
+  `deadlineMs`, 60 s by default), and its result comes back into the chat by itself. Set
+  `deadlineMs` on a tool for its own limit: shorter for one that should never hold a reply long,
+  longer for one known to be slow.
+- Say what you covered. An agent reads a list as complete: return the range or filter you
+  searched ("Mon Oct 12 through Sun Oct 18", "all open times") and never cut results short
+  silently. If there are more, say so.
+
+## The tool's test file
+
+In tests no real tool runs: a model plays each tool from the scenario's world. `tests/tools/<tool>.yaml`
+tells it how the tool behaves: every kind of response it gives, each with an example and how long
+it takes. Write one per tool.
+
+```yaml
+tool: place_order
+behaviour: |
+  Order numbers are BL- followed by four digits and go up by one per order. ready_by is about
+  ten minutes after the order is placed. Fails if an item, size or milk is not on the menu.
+responses:
+  - when: The order goes through (usual)
+    latency_ms: 4800
+    input: { name: Sam, items: [{ item: Latte, size: large, milk: oat, quantity: 1 }] }
+    output: { order_id: BL-1040, ready_by: "{{now + 10m}}", pay_at_pickup: true }
+  - when: The point of sale is busy; the order still goes through
+    latency_ms: 11000
+    output: { order_id: BL-1041, ready_by: "{{now + 15m}}", pay_at_pickup: true }
+  - when: The point of sale is down
+    latency_ms: 3000
+    error: Point of sale unavailable
+```
+
+- `behaviour`: what the tool does beyond its description, in plain words (ids, ordering, limits,
+  when it fails).
+- `responses`: one entry per kind of outcome, the usual one first, then slow ones and each error
+  the real system can give. `when` says when it happens; `latency_ms` is how long it takes;
+  then `output` (what `run` returns) or `error` (what it throws). `input` is optional, as an
+  example of a call.
+- Use real responses when you have them (copy them from the agent's recorded tool calls, with
+  personal data removed); otherwise write realistic ones in exactly the shape `run` returns.
+- Examples show the shape, not the answer: the simulator takes ids, names and dates from the
+  scenario's world, the call's input and what the agent already said. Write dates as placeholders
+  for the test's clock (`{{now + 10m}}`, `{{now + 2d | date}}`; see the `flow-framework-tests` skill),
+  never as written dates, which go stale.
+- Latency lives only here. Scenarios describe situations ("the point of sale is slow this
+  morning"); the simulator picks the matching response and latency on each call.