agent-learning-kit 0.1.0__py3-none-any.whl

fi/alk/harness/skills/kinds/chat.md

@@ -0,0 +1,48 @@
+---
+name: chat
+applies_to: modality=chat
+description: What a scenario has to account for when the person reaches the agent by typing. Read alongside the scenario-writing instructions whenever the contract says the modality is chat.
+---
+
+# Writing scenarios for a chat agent
+
+A chat agent is reached by a person typing. That person can see everything they have written, can
+paste from elsewhere, can send three messages before waiting for an answer, and can go quiet for ten
+minutes and come back. Every requirement below follows from one of those facts, and none of them
+replaces the general requirements a scenario has to meet.
+
+A chat is always started by the person, so there is no call direction to establish: treat every chat
+scenario as one the person initiated, and ignore anything written for calls an agent places.
+
+## What a chat scenario can test that a voice one cannot
+
+- **The whole request arrives in one wall of text.** Reference number, dates, three questions and a
+  complaint in a single message. An agent that answers the last sentence and drops the rest fails
+  here and passes every voice test.
+- **A pasted blob**: a receipt, an error dump, a confirmation email. The fact the agent needs is in
+  there, unlabelled, next to facts that look like it.
+- **The person edits themselves.** "order 4471, sorry, 4417." The corrected value is the real one,
+  and an agent that takes the first fails.
+- **Silence that is not silence.** They stop replying for ten minutes and come back mid-thread
+  expecting the agent to still hold the context.
+- **Ambiguity a speaker would have resolved by tone.** "great, that's just what I needed" from
+  somebody who has been complaining for four turns.
+
+## What this modality lets you vary
+
+Register: how somebody types is who they are. Someone terse sends four words and no punctuation.
+Someone anxious sends three messages in a row before the agent has answered. Someone formal writes
+paragraphs. Vary this across the suite the way accents are varied for voice, and let the situation
+choose it.
+
+Typos, autocorrect and slang are part of the input the agent has to handle, not noise to be tidied
+away. A suite where everybody types cleanly has not tested reading.
+
+Message boundaries matter. One thought split across three messages, and three thoughts in one
+message, are different tests.
+
+## What does not belong in a chat instruction
+
+No stage directions and no narration of tone. If it is not typed, it does not exist.
+
+Never tell the person what the agent should reply.

fi/alk/harness/skills/kinds/voice-voicemail.md

@@ -0,0 +1,63 @@
+---
+name: voice-voicemail
+applies_to: modality=voice,voicemail=on
+description: What a scenario has to account for when a mailbox answers an outbound call instead of a person. Read alongside the voice instructions, and only on a run where mailbox scenarios are allowed.
+---
+
+## When a mailbox answers instead of a person
+
+An outbound call reaches voicemail often, and an agent that runs its interactive script at a
+recording is a real defect nobody hears about until a customer does. Set `answered_by: voicemail` on
+an outbound scenario and the person is replaced by a mailbox: it plays its greeting once and then
+says nothing at all, whatever the agent asks.
+
+What is being tested is entirely on the agent's side. Does it notice it is talking to a machine
+rather than waiting for answers that will never come. Does it leave a message that stands on its own,
+with who is calling, why, and what happens next, rather than the first line of a conversation. Does
+it stop, instead of holding the line open asking questions.
+
+Which kind of mailbox is `voicemail_style`, and the four are four different tests:
+
+- `personal`, the person's own greeting followed by a tone. The ordinary case, and the only one that
+  carries their name, so it is also the one where the agent can confirm who it reached.
+- `carrier`, the network default, which names nobody. The agent has no confirmation of who answered
+  and has to leave a message anyway.
+- `operator`, a long formal announcement before the tone. This is the one an agent that starts
+  talking too early speaks over, so its message is recorded half missing.
+- `full`, a mailbox that cannot record. **There is no tone at all**, and the right behaviour is to
+  recognise there is nowhere to leave a message and end the call to try later, not to talk into
+  nothing.
+
+**State the style. Do not leave it out.** Left out it falls back to `personal`, and personal is the
+easiest of the four: the greeting names the person, so the agent can confirm who it reached and the
+tone tells it when to talk. Most suites are only large enough for one mailbox, and if that one always
+defaults, the three harder cases never get tested at all: `carrier` where nothing confirms who
+answered, `operator` where the announcement is long enough to be talked over, and `full` where there
+is no tone and the right move is to give up rather than leave a message into nothing.
+
+So choose the style from the situation, the way you choose everything else. A number nobody has ever
+confirmed belongs to this person is a carrier mailbox. A work line reached out of hours is an operator
+system. Somebody who has been letting calls go for a week has a full one. Where you write two or more
+mailboxes, use at least two styles. A greeting in another language is worth one of them too, since its
+transcription has to survive.
+
+Two things follow from a mailbox not being a person. The scenario's persona still carries the
+greeting as its opening line, so write the greeting there. And the caller cannot supply anything, so
+a mailbox scenario never asks the agent to collect a value, confirm a detail or reach agreement:
+those belong in a scenario where somebody picks up.
+
+That applies to the sub-goals as hard as it does to the situation, and it is where these scenarios go
+wrong in practice. Mailbox calls fail on a sub-goal needing a tool call, because the agent
+only reaches that tool after the person it called has spoken, and on a mailbox nobody ever does. Every
+sub-goal on a mailbox scenario has to be something the agent can do with nobody on the line: it
+recognised a machine, the message it left says who is calling and why, it stopped instead of asking
+questions. A sub-goal that needs an answer marks a correctly handled mailbox as a failure and tells
+you nothing.
+
+Keep these rare: at most one scenario in twenty, and none at all is a perfectly good suite. They test
+one narrow thing well, and a suite full of mailboxes has stopped testing the agent talking to people.
+
+**Leave `background_noise` off a mailbox scenario.** What the agent reaches is a recording played back
+by a switch, so there is no room behind it to overhear, and a room behind a recording is the one
+detail that would tell the agent it is talking to a person when the whole point is that it is not.
+

fi/alk/harness/skills/kinds/voice.md

@@ -0,0 +1,59 @@
+---
+name: voice
+applies_to: modality=voice
+description: What a scenario has to account for when the person reaches the agent by speaking. Read alongside the scenario-writing instructions whenever the contract says the modality is voice.
+---
+
+# Writing scenarios for a voice agent
+
+A voice agent is reached by a person speaking, in real time, who cannot see anything. That person
+answers several questions in one breath, corrects themselves mid-sentence, mishears a digit, talks
+over a confirmation, and sometimes goes silent. Every requirement below follows from one of those
+facts, and none of them replaces the general requirements a scenario has to meet.
+
+Whether the agent placed this call or answered it changes how the person is written. The contract
+carries that as `CALL DIRECTION`, and the general instructions say what each direction requires: read
+it there rather than deciding it here.
+
+## What a voice scenario can test that a chat one cannot
+
+- **The caller answers three questions in one breath**, in their own order, before being asked.
+  Real callers do this constantly. An agent that collects one field per turn fails here and passes
+  every written test.
+- **The caller changes their mind mid-sentence**, and the correction lands after the original. The
+  second value is the real one.
+- **A value has to be read back and heard.** Codes, prices, times. A digit misheard is a real
+  failure, and it only exists out loud.
+- **Interruption.** The caller talks over the agent's confirmation. What the agent believes was
+  confirmed is now a question.
+- **Silence.** The caller goes quiet, or the line is noisy and they ask for something again.
+
+A suite of voice scenarios that could all have been typed has not tested the modality.
+
+## An attempted transfer is not a completed one
+
+A voice run may record that the agent tried to hand the call to a person without the receiving side
+ever picking it up, because completing that handoff needs telephony the run does not have. So a
+scenario about handing off tests the offer or the attempt, and the work that would follow it belongs
+in a separate scenario. A sub-goal that only holds once somebody answers will fail on a correct
+handoff.
+
+## What this modality lets you vary
+
+`background_noise` is per scenario, not a suite setting. Choose it from the situation rather than
+sprinkling it: a caller in a vehicle, a caller in an office, a caller in a crowd. A quiet scenario is
+the control that makes a noisy one mean something, so a suite needs both.
+
+Accent and language belong to who the caller is, and they change what the agent's transcription has
+to survive. They are dealt across the suite; take the one you are given unless the scenario genuinely
+needs another.
+
+`max_turns` is a budget, not a target. A scenario that needs eighteen turns to reach the thing it
+tests is fine. One that spends eighteen turns being polite is not.
+
+## What does not belong in a voice instruction
+
+Never write stage directions. No *sighs*, no [annoyed]. Anything in brackets is read aloud, so the
+caller says the word "annoyed" instead of sounding it. Manner comes from the persona's disposition.
+
+Never tell the caller what the agent should do. They are on the phone, not reading the contract.

fi/alk/harness/skills/plan-suite/SKILL.md

@@ -0,0 +1,103 @@
+# Planning a suite of scenarios
+
+A scenario is one complete session with the agent under test: a person with a situation, everything
+they know, the data the world holds for them, and a settled outcome. Writing them is a separate job
+done by separate writers. Yours is to decide what the suite covers and to hand each writer one part
+of it. Nothing here writes a scenario.
+
+Work in this order: find the cells, pick the ones worth testing, size them, then hand them over.
+
+## 1. Find the cells
+
+A cell is one pairing of something the agent acts on with something a person can want done to it.
+Write both lists down before counting anything.
+
+**What this agent acts on.** Read it off the agent's own tools rather than inventing it: whatever its
+tools take and return, reduced to singular nouns. A booking agent has rides, addresses, payment
+methods, accounts. A claims agent has policies, claims, documents, payouts. Four to ten is usual.
+
+**What a person can want done.** This list is fixed and applies to every agent. It is grouped by what
+the operation does to the world, and that grouping is why it is complete: an intent either reads, or
+writes, or manages the process, and there is no fourth kind.
+
+```
+reads, nothing changes      retrieve   compare   explain   diagnose
+writes, something changes   create     update    cancel    execute   configure
+manages the process         authenticate   navigate   handoff
+```
+
+Cross the two lists. Twelve operations against six objects is seventy two candidate cells, which is
+where a large suite honestly comes from. Most cells will be empty, and saying so is a result: an agent
+with no way to compare payment methods either cannot do it or has a gap worth reporting.
+
+Name each cell for the pair, `cancel a ride`, `authenticate a payment method`. Never name one for a
+person.
+
+## 2. Pick the cells worth testing
+
+A cell says where a test could live. It does not say one is worth writing.
+
+Go through the real cells and keep the ones where you can name something that goes wrong: a fact that
+is missing, two that contradict, a request the rules forbid, a record that is not what the person
+believes, a step attempted before the thing it depends on has happened. **A cell you cannot name a
+failure for gets no scenario.** That is a finding, not a gap in your plan: either the agent does
+nothing there, or nothing there can break.
+
+Two rules that decide whether the count is real:
+
+- **Never turn one cell into several by changing the person.** The same cell tested twice with two
+  different people is one test written twice. A different name, age, accent or city is the same test
+  in a different costume.
+- **If the cells you can name failures for run out, report that number.** A smaller suite that is
+  entirely real is worth more than a padded one, because padding hides the gap instead of showing it.
+
+Every scenario is a whole session, not a step of one. The cell says where the difficulty sits; the
+scenario still runs from first contact to a settled outcome. A scenario about authenticating a payment
+method is not "check a code", it is a person getting all the way through what they came for, with the
+authentication as the part that goes wrong. Write it the way you would write an end-to-end test of a
+large system: one complete journey, the interesting failure somewhere inside it, everything around it
+real.
+
+## 3. Size each cell
+
+Give each kept cell a number of scenarios, **in proportion to how much can genuinely go wrong in it**.
+A cell with rules to enforce, information to gather or state to change earns a large share; one where
+little can fail earns one scenario or none.
+
+**A cell asking for more than one scenario has to name what goes wrong in each**, one distinct failure
+per scenario. A count without those behind it is a promise the writers cannot keep, and it comes back
+as near-copies.
+
+For each cell, state what the agent should do, exactly one of:
+
+```
+succeed   refuse   ask   escalate
+```
+
+and, only where something is deliberately making it hard, one of:
+
+```
+impersonation   injection   fraud   emergency   pressure
+```
+
+These answer different questions and are not alternatives. An injection attempt expects a refusal and
+carries the injection label, so record both. Do not label a cell happy, edge or adversarial: those
+overlap, since an injection is adversarial and also bound to fail, and "edge" describes intensity
+rather than kind.
+
+The ordinary path is worth one cell, and only one. Everything else is a way things go wrong. A plan
+whose cells all expect success has tested the demonstration rather than the agent.
+
+## 4. Hand the plan over
+
+Call `generate_suite` and pass the plan as `slices`. A slice is one cell plus what you decided about
+it: which cell, the angle to take, how many scenarios it is worth, and why. The tool runs one writer
+per slice, several at a time, reviews what comes back and fills what was missed.
+
+Pass the plan explicitly. Left to itself the work is divided evenly, which is how a cell with one real
+branch pads to three while one with six gets three.
+
+Prefer more small slices to a few large ones: each writer then stays inside its turn budget, and one
+that fails costs its own slice rather than a third of the suite. Two signs the sizing is wrong: every
+slice holds one scenario, which means you listed scenarios instead of grouping them; or every slice
+holds the same number, which means you padded to reach a target.

fi/alk/harness/skills/provision-environment/SKILL.md

@@ -0,0 +1,136 @@
+---
+name: provision-environment
+description: Stand up the real thing an agent connects to, and prove it, without touching the agent.
+---
+
+# Provision the environment
+
+You are standing up the world an AI agent will be tested in. Its contract is in front of you:
+the tools it really has, the rules it obeys, what it depends on, and its data.
+
+**You are not rebuilding this agent. You are building what it connects to.** Its code runs
+unmodified, its own client issues its own queries, and the only thing that differs from
+production is which host answers them. That is the whole method, and everything below follows
+from it.
+
+## The one rule
+
+**Never change the agent.** Not its source, not its config file, not a copy of it. You have no
+tool that can, and that is deliberate: when a check fails there are two ways to make it green —
+fix the environment, or edit the agent until it stops failing — and the second produces a green
+suite about code nobody ships.
+
+If the agent cannot be pointed at your store, that is a **finding to report**, not a thing to
+work around. Say so plainly and stop.
+
+## Being where the agent already looks
+
+The agent expects a database at some host, on some port, with some name, reached by some
+variable. You do not change any of that. You build your store **to match it**.
+
+- It reads `DATABASE_URL` — you set `DATABASE_URL` when it launches.
+- It reads `database.url` from a config file — you mount that file.
+- It hardcodes `db.internal:5432` — you make `db.internal` resolve to your container. A
+  hardcoded host is not an obstacle; it is just a name you have to answer to.
+- It hardcodes a database name and user — you create your store with exactly those.
+
+The contract records what it expects. Match it.
+
+## Talking
+
+You are talking to a person. Answer briefly, do the work when they ask for it, and keep replies
+short — they can see every tool you call and what it answered.
+
+Ask them when a decision is genuinely theirs: what data should be in the store where the
+contract carries none, whether an engine you cannot identify is worth guessing at.
+
+## How to work
+
+1. **`declare_engine`** with the engine the contract names. `inspect_environment` first if you
+   want to see what the harness can already stand up.
+
+   **Never substitute a different engine.** Not a similar one, not a "lightweight equivalent",
+   not a server standing in for something held in memory. An agent whose tools read a dict is
+   not tested by putting that dict in Redis — its queries never run, and every result is about
+   code it does not have. This is the single mistake this whole path exists to prevent, and it
+   does not stop being that mistake because the substitute is convenient.
+
+   Some agents have **no server at all**: they load files into memory and their tools read that
+   structure directly. That is `engine: inprocess`, it is already supported, and the contract
+   names the loader to call. Nothing is stood up and nothing is connected to.
+
+   If the harness genuinely has never seen the engine — it is not in `inspect_environment`'s
+   list — then `write_store_ops`. That is expected, not a failure; an engine nobody wrote down
+   in advance is the normal case.
+
+2. **`run_migrations` with the agent's own migrations.** Find them: an `alembic/` directory, a
+   `migrations/` folder, `schema.sql`, the models it defines. Run those.
+
+   **Never write a schema yourself.** One you invented is a guess, and every check written
+   against it inherits the guess. If you genuinely cannot find migrations, say so and ask —
+   do not fill the gap with tables you made up.
+
+3. **`seed`** from the contract's real data, including anything that looks like a mistake: a
+   misspelled id, an item marked unavailable, an odd price. The store is a replica of what the
+   agent has, not a corrected version, and a test written against a corrected one will not
+   catch the real bug.
+
+   Leave it in its natural starting state: empty carts, no in-flight work. Scenarios add what
+   they need.
+
+4. **`add_sub_goal`** for each thing worth checking, with its check as code.
+
+   A check is given the store and the calls that were recorded, and returns a sentence when
+   something is wrong or `None` when it held. Write it against what the run leaves behind:
+
+   ```python
+   def check(world, calls):
+       rows = world.state()["orders"]
+       if len(rows) != 1:
+           return f"{len(rows)} orders, expected 1"
+       return None
+   ```
+
+   Use `judged` **only** where nothing observable settles it — whether a refusal was explained,
+   whether tone was right. If most of your sub-goals are judged, you have not looked hard enough
+   at what the store records.
+
+5. **`write_simulator_prompt`**, if this agent is conversational.
+
+6. **`prove_environment`**, and fix what it names. Repeat until it holds.
+
+7. **`save_environment`.**
+
+## What proving actually does
+
+You hand it a `mutation` — any statement this engine accepts that changes something. One insert
+is plenty. Then, without knowing your engine:
+
+- your mutation has to **move** something, or a broken reset would look perfect
+- `restore` has to reproduce the rows **exactly**
+- **ids must not drift**: the same change is run twice from the same starting point and the two
+  results compared, so a reset that puts rows back but leaves a counter where it was is caught
+  without anyone naming what a counter is called on this engine
+- every check you wrote has to **fail against an emptied store**. One that still holds when
+  there is nothing there is not measuring the environment
+
+A failure here is **yours or ours, never the agent's**. Nothing in it involves the agent.
+
+## Reading a failure
+
+The report names what broke, in your terms. `ids do not drift: the same change from the same
+starting point produced something different the second time` means your `restore` puts rows
+back but not the counter behind them. Fix the reset and prove again.
+
+If the same failure survives three attempts, stop and read it literally. Whatever you are
+changing is not what is failing.
+
+You do not get to declare the environment sound. `save_environment` runs the gate again itself.
+
+## Finishing
+
+Say what you stood up: the engine and version, where its schema came from, roughly what is in
+it, how the agent will be pointed at it, and the sub-goals with how many are settled by code.
+
+Then say plainly anything you were unsure about — especially where you could not find the
+agent's migrations, or where its configuration seam was not obvious.

fi/alk/harness/skills/run-scenarios/SKILL.md

@@ -0,0 +1,112 @@
+---
+name: run-scenarios
+description: Run the validated scenarios against the agent and say what the results mean.
+---
+
+# Run the scenarios
+
+The environment is built and the scenarios are written and validated. Your job is to run them
+against the agent and say what came back.
+
+Each run costs real money and takes time. Do not run the whole suite because somebody greeted
+you, and do not re-run a scenario that just passed.
+
+## Talking
+
+Answer what they ask, briefly. Run what they ask you to run. They can see every tool you call
+and what it answered, so do not repeat it back.
+
+## When they ask for something this stage cannot do
+
+Writing scenarios and changing the world belong to earlier stages, and you do not have those
+tools here. That is deliberate: a stage that grades results must not be able to edit the test
+that produced them.
+
+**But nothing is lost and nothing needs restarting.** The stages are a roadmap the person can
+move between at will, and both earlier stages reopen onto what already exists: the scenarios
+stage loads the scenarios that are there, and the build stage picks up the saved world rather
+than replacing it. You cannot move yourself, which is why it looks like a dead end from in here.
+They can, in one click.
+
+So say which stage does it and let them take you there. Never tell them to restart, to start a
+new session, or that the stages only go one way; all three are wrong and all three throw away
+work that is sitting on disk.
+
+Do the part you can do first. If they ask for scenarios you cannot write, say what is missing and
+why it is worth covering, so the trip is worth making: they arrive at that stage knowing exactly
+what to ask for.
+
+One thing worth saying when it applies: changing the world after scenarios exist leaves those
+scenarios proved against a world that has moved. They are re-proved when resubmitted, so anything
+the change touched should be resubmitted before the next run.
+
+## Before the first run
+
+`preflight` costs nothing and catches the failures that would otherwise arrive after the
+expensive part — missing credentials, no way to reach a hosted agent. Run it once at the start.
+
+`list_scenarios` shows what can be run, what each one tests, and which of its sub-goals are
+settled by code rather than left to a judge.
+
+## Running the suite
+
+**`run_simulation` runs everything, once.** It restores a separate world for each scenario,
+applies that scenario's own setup, puts the agent in front of it, and grades what is left behind
+along with every call that was made. One call from you; the simulation owns the rest.
+
+That is how a suite is run. Do not work through the scenarios yourself: a run made of one tool
+call per scenario takes as many of your turns as there are scenarios, costs that much more, and
+produces the same results slower.
+
+Its concurrency argument is how many run at once. **Leave it at 1 for a spoken agent** — every scenario there
+is a real phone call that costs real money and holds a real tunnel. For a typed agent, raising it
+is the difference between the slowest scenario and the sum of all of them.
+
+It blocks until the whole suite is done, which is minutes, and says so.
+
+`run_scenario` still exists for looking into a single failure after the fact. It is not how you
+get results.
+
+## Looking into a run
+
+`read_run` with no arguments lists the runs this session has done; given a run id it gives that
+run in full, and given a scenario name as well it gives one case. A run holds the conversation, every
+tool call with its arguments and what came back, what each check decided, and for a spoken run the
+recording and what the call measured.
+
+Runs accumulate. The same suite against the same world, run twice, is two runs you can compare —
+which is the point of keeping them rather than overwriting.
+
+## Reading a result
+
+You are given each sub-goal and whether it held, and **every tool call the agent made, with its
+arguments and whether the world accepted it**. That last list is usually where the answer is.
+
+Before reporting a failure as a finding about the agent, work out which of these it is:
+
+**The agent did the wrong thing.** A real finding. Say what it did and what it should have done.
+
+**The world wrongly refused.** Look at the arguments. If the agent sent something the contract
+permits and the world said no, the world or the contract is wrong, not the agent.
+
+**The check is wrong.** The commonest one. A check that encodes *how* an agent should comply
+fails a correct agent that complied differently — a check demanding a particular tool call fails
+an agent that refused politely without calling anything. Check the outcome, not the route.
+
+**The simulated person never asked.** If they hung up before raising what the instruction said,
+the scenario never happened. That is a simulator problem, not a result.
+
+A run where nothing reached the world says nothing about the agent. Report it as that.
+
+## What to say
+
+Say what passed, what failed, and for each failure which of those four it is. Where the fault is on the test's side,
+say what would fix it — the check to rewrite, the contract value to correct — and do not report
+it as a finding about the agent.
+
+Judged sub-goals are reported as judged. Say so, rather than letting a score read as though
+everything in it was measured.
+
+A sub-goal whose kind is "eval" was decided by a named eval on the FutureAGI platform rather than by a
+model in this process, and its result names the eval that decided it. Report that name: it is
+something the person can open, re-run and change, which a verdict reached here is not.