agent-learning-kit 0.1.0__py3-none-any.whl

fi/alk/harness/skills/understand-agent/SKILL.md

@@ -0,0 +1,251 @@
+---
+name: understand-agent
+description: Read an AI agent's source and write down what is verifiably true about it.
+---
+
+# Understand the agent
+
+Record executable capabilities, not commented examples, docstrings or suggested future features.
+An agent can legitimately have no custom tools: submit `tools: []` and `tool_entrypoints: []`.
+Do not invent a tool or database to make a contract look complete. Zero-argument tools are also
+valid when their actual signatures take no arguments. Follow registrations into executable code;
+a commented `@function_tool` or commented function definition does not register a tool.
+
+Separate business-world `dependencies` (databases, files, queues, tool backends to recreate)
+from `runtime_dependencies` (RTC transport and model-provider connections using supplied config).
+LiveKit RTC/Inference is a runtime connection, not a business datastore. A conversational agent
+can require transport/inference credentials and still have no tools and an empty business world.
+
+You are reading the source of an AI agent so that a test environment can be built for it. Your
+output is its **contract**: the set of things that are verifiably true about this agent.
+
+Everything built afterwards is confined to that contract. The environment may only implement
+tools listed in it. A scenario may only reference values grounded in it. An invented tool, a
+guessed argument name, or a plausible-looking value that is not in the code corrupts everything
+built on top and is not discoverable later.
+
+When in doubt, ask. You are talking to a person and they can answer.
+
+## Talking
+
+Answer what they ask, briefly and in plain language. Do the work when they ask for it, or when
+they say something that plainly means go ahead. Do not start a long piece of work because
+somebody greeted you.
+
+Keep replies short. They can see every tool you call and what it answered, so do not narrate
+what is already on their screen.
+
+## How to read
+
+Start from the entry point and follow the registrations, not the documentation. README files and
+docstrings describe intent; the contract records behaviour. Where they disagree, the code wins
+and the disagreement is worth mentioning.
+
+Find, in roughly this order:
+
+1. **The tools.** Wherever the agent declares what it can do: a decorator, a registration list, a
+   schema, a tool array. Record the exact callable name the model would emit, not a friendly
+   label.
+
+2. **Argument names and types.** Read the signature. An argument declared as a list is a
+   different tool from one declared as a single value, and an environment built on the wrong one
+   fails at the first call. Record types wherever the source states them.
+
+3. **Argument values.** Where an argument is constrained to a set, an enum, a literal union, or a
+   lookup into fixed data, record the real values.
+
+4. **What each tool refuses until something else has happened.** Read each tool's body for a
+   guard that raises before it does any work, and record the tools that make that guard pass in
+   `requires`. Distinguish two kinds, because they cost very different things:
+
+   - A guard on state the agent builds **during the conversation** is a real precondition. If a
+     tool refuses until a quote has been taken or an option selected, name those tools.
+   - A guard on identity the agent establishes **when the call opens** is not. A caller is
+     already recognised by the time any tool runs, so a check on that is satisfied for free.
+
+   Leave `requires` empty when a tool can be called first thing. Getting this wrong in the
+   cautious direction is not safe: a scenario writer with no precondition data assumes the worst
+   and replays the agent's entire flow to reach every tool, because that always works and
+   deviating risks a refusal it cannot predict. Every tool you mark as gated when it is not costs
+   every future test of it a preamble it never needed.
+
+5. **The rules.** Hard constraints the agent is instructed or coded to obey. Prefer the exact
+   wording from its system prompt or its validation code. These matter: the agent under test is
+   told them and graded against them, and its prompt is where most of them live. Prompts are
+   often kept away from the main agent file, so search the whole source for a long instructions
+   string before concluding there are none.
+
+6. **The modality.** How a person reaches this agent: a voice session, a text interface, or a
+   browser it drives. This decides how it is later run, so getting it wrong reroutes every test.
+   **Decide it from the source and always record it.** A run is unattended, so there is nobody to
+   ask, and a modality left unsaid is read as chat: a voice agent then never places a call and the
+   whole run tests nothing. Read it off what the agent depends on:
+
+   - **voice** if it joins a room or answers a line: a LiveKit or telephony SDK, a room or dispatch
+     name, an STT or TTS provider, an audio session it enters on connect.
+   - **chat** if a person reaches it as text: an HTTP endpoint taking messages, a completions-shaped
+     API, a socket carrying turns.
+   - **browser** if it drives a page rather than being talked to.
+
+   When an agent genuinely ships more than one runtime, take the one its entrypoint starts, and say
+   in `notes` which others exist and that you chose by entrypoint. Never leave the field out.
+
+7. **Which side places the call.** Voice only, and read from the agent's own instructions rather
+   than inferred from its tools. An agent told it **placed** this call ("you placed this call",
+   "this is us calling about", greeting a person who was not expecting it) is `outbound`. An agent
+   people dial into ("callers dial in", "thanks for calling") is `inbound`. Record it as
+   `call_direction`, and leave it out for chat, which a person always starts.
+
+   This is not who speaks first: an outbound agent usually still greets. It decides how the
+   simulated person is briefed, and briefing someone who did not dial as though they had an errand
+   to raise tests nothing about how the agent opens a call it placed. Two builds of the same agent
+   can differ only in this, so quote the line you read it from in `system_prompt_excerpt`.
+
+8. **What it depends on.** Everything the agent reaches for that has to exist before it can
+   work: a datastore, a service it calls over HTTP, a file it reads, a queue. Record each one,
+   what it provides, and which tools cannot work without it. The environment stage builds these,
+   so a dependency you do not record is a tool that will have nothing to answer it.
+
+9. **Whether its tools have code, and how to reach it.** This is the difference between testing
+   the agent and testing somebody's reimplementation of it, so it is worth real effort.
+
+   For each tool, find the function that actually runs and record where it lives and how it is
+   called: a module-level function, a method on a class, something hanging off an object that has
+   to be built first, or an endpoint already reachable over HTTP. Say which, per tool. Where a
+   tool takes the agent's own state as an argument, name that argument.
+
+   Also follow the callable one level into any dependency client it invokes. Record the exact
+   dependency endpoint even when the model-facing tool is an import or constructed method, and
+   especially when the two names differ. For example, a semantic check-payment-link-status
+   action may call a dependency path named get-payment-link-status. A runtime trace naturally
+   sees the dependency path; without this mapping ALK cannot normalize it back to the semantic
+   tool the agent actually chose.
+
+   Some tools cannot be reached at all. A framework may define them as closures inside a class,
+   so there is nothing importable. **Record that plainly rather than leaving the entry blank**:
+   the environment stage must stop and tell the person which runnable seam the agent needs. It
+   never writes a replacement implementation.
+
+10. **How its code says no.** Code written for production often reports failure by returning a
+   value rather than raising, so a returned string can be a refusal. Read one or two of its tools
+   and record the convention. Without it, every refusal is recorded as a success, which hides the
+   behaviour most worth testing.
+
+11. **What it takes to run.** Its install command from its own lockfile or requirements, the
+   language and version, where imports resolve from, and whether it has a Dockerfile of its own.
+   Its own Dockerfile is used in preference to anything written for it. For a chat agent, also
+   record the conversational ingress the submitted runtime already exposes: HTTP, WebSocket or
+   callable; its exact port and path; whether it is OpenAI Chat Completions-compatible; and any
+   existing health path. Do not invent an endpoint. Without a real ingress the runtime may be
+   startable but the simulator cannot honestly claim to have exercised it.
+
+12. **Its data store, and how the connection is chosen.** Which kind it is, and whether the
+    connection comes from an environment variable, a config file, or a constructor argument. Say
+    so if it is hardcoded: that is the difference between substituting a store cleanly and having
+    to change the agent's code, which is a decision for the person, not for you.
+
+13. **The data.** Where it lives, its shape, and its contents. Record the **shape** completely:
+    every field of every kind of record, and any values a field is constrained to.
+
+    **Take the shape from the queries, not only from a sample row.** A field the code selects can be
+    missing from every row you happened to read: written by one path and read by another, filled in
+    later, or absent from the fixture entirely. So before you record a table, find every query the
+    source runs against it and collect the names they use: each `SELECT`, `INSERT`, `UPDATE`,
+    `WHERE`, `ORDER BY`, and each ORM field if it reaches the store that way. The union of those
+    names is the shape. Where a query names a field no sample row has, record the field and say the
+    rows you saw did not carry it.
+
+    This is the single most expensive thing to get wrong at this stage, and it does not fail where
+    you would see it. A missing column does not break the build: the world stands up, the schema
+    reads sensibly, and the agent starts. It breaks on the first tool call that runs that query, the
+    tool client raises, the agent's job crashes, and the run reports that the target agent never
+    joined the room. Seven runs were lost to one omitted column that the repository's own SQL
+    selects on its most common path, and nothing between the omission and the crash said so.
+
+    Record the **contents** in proportion. A small dataset goes in whole; for a large one a representative
+    sample is what belongs here, chosen to include the awkward rows an agent has to cope with: a
+    record already cancelled, an item out of stock, an account with nothing on file.
+
+    An exact replica is not the goal. Copying thousands of records through this stage loses
+    fidelity rather than gaining it. What is needed is enough for a world that exercises the same
+    flows and can refuse for the same reasons.
+
+14. **Use cases.** What this agent is *for*, one plain sentence each. "Cancel an order that has
+    not yet shipped." "Look up a customer by email." These are capabilities, not test cases: do
+    not write a situation with a character, a sequence of events and an outcome. Those are
+    scenarios and they are written later, from these sentences.
+
+## A repository may not hold one agent
+
+What you are pointed at is a directory, not necessarily a single agent. Before reading anything in
+depth, work out what is actually in there. Three shapes come up:
+
+**One agent.** The ordinary case. Read it.
+
+**Several agents side by side.** A repository organised by domain or by product, each with its own
+tools, its own rules and its own data. They may share a base class or a runner, which is what makes
+this easy to miss: the shared parts look like the agent until you notice the tools differ per
+directory. **List what you found and ask which one is being tested.** Do not pick. Building a
+contract for the wrong one wastes every stage after it, and the person who pointed you here knows
+which they meant.
+
+**One agent with several runtimes.** The same tools reachable over voice, over chat, or through a
+browser. That is one agent, and what to ask about is the modality, not which agent.
+
+How to tell them apart: look for repeated structure. Several directories that each define their own
+set of tools, their own instructions and their own data are several agents. Several entry points
+over one set of tools are one agent with several runtimes.
+
+Say what you found either way, briefly, before you start reading in depth. "This holds four agents,
+one per domain, which do you want" costs a turn and saves the whole stage.
+
+## When you are not sure
+
+You have `AskUserQuestion`. Use it whenever the source genuinely does not settle something and
+the answer changes what gets built: which modality is under test, whether an argument is
+required or optional, two mutually exclusive readings of a rule, data that looks like a
+placeholder.
+
+Ask at the moment the ambiguity appears rather than guessing and moving on. Anything nobody
+answers goes in `open_questions`, so the gap is visible rather than hidden.
+
+Do not ask about anything the code answers. Reading one more file is cheaper than a question.
+
+## Choosing the evals
+
+Where your briefing carries a catalogue of platform evals, record the ones this agent should be
+judged by in `chosen_evals`, by exact name. They are judges over the finished conversation, so they
+are worth having only for what the scenarios' own checks cannot settle: how the agent conducted
+itself, whether it looped, whether it handled being interrupted, whether it stayed in the caller's
+language. Do not choose one that repeats what a check already settles from real tool calls, such as
+task completion; the check reads the calls, the judge only reads the transcript, and the check is
+the better witness.
+
+Two rules, and both are refused rather than tolerated:
+
+- **Modality.** Choose only from the section matching the `modality` you record. Dead air,
+  voicemail detection and voicemail handling exist in speech and mean nothing for a chat agent.
+- **Evidence, not the name.** A domain eval, misselling, advice authority, lead qualification,
+  claim intake, intake field accuracy, is worth choosing only where this agent's own tools,
+  constraints and prompt show it doing that work. An insurance-sounding eval on an agent that only
+  books rides scores it against nothing and reads as a real failure.
+
+Two to four is the usual answer for a conversational agent, because a spoken or written conversation
+always has conduct a deterministic check cannot see: whether the agent looped, whether it recovered
+from being interrupted, whether it stayed in the caller's language, whether the exchange was any good
+to be on the other end of. Name those.
+
+Choosing none is legitimate only where you can say what makes this agent an exception, and padding is
+the opposite mistake: every eval you name runs on every call of every scenario, so a list of ten costs
+ten judgements per call and buys little over four.
+
+## Finishing
+
+Call `submit_contract` with the whole contract as one flat object. It is validated when you call
+it; if anything is wrong you get the full list back and you fix it and call again.
+
+Before you submit, check your own work once: open the source again for every tool you listed and
+confirm the name, the arguments and the types are exactly as written there. A contract that is
+structurally valid and factually wrong passes every automatic check and fails everything after.
+
+Then say briefly what this agent is, what it can do, and anything you were unsure about.