@ssheleg/agent-stack 0.7.2 → 0.9.0

package/plugins/agent-stack/skills/agent-harness/references/system-prompt.md

@@ -0,0 +1,127 @@
+# The system prompt — altitude, vocabulary, and what changes for reasoning models
+
+**Load this when:** writing or fixing an agent's system prompt, or explaining why the same
+prompt behaves differently across runs.
+
+**Spec pinned:** Anthropic context-engineering and agent guidance; `promptingguide.ai` agents/* and guides/reasoning-llms · read 2026-08-14
+
+## Contents
+
+- The right altitude
+- What actually belongs in there
+- Enumerate the vocabulary
+- Inject what the model cannot know
+- Flexible while learning, strict in production
+- Structure, examples, and the cost of formatting
+- Reasoning models change three things
+- Traps
+
+## The right altitude
+
+A system prompt fails in two directions and the middle is narrower than it looks.
+
+**Too low** — hardcoded if-then branches for every case. It works on the cases you wrote and
+is brittle everywhere else, and each new case costs another branch. You are writing a
+program in prose, badly.
+
+**Too high** — vague guidance that assumes a shared understanding the model does not have.
+"Be helpful and use good judgement" tells it nothing it did not already believe.
+
+The target: **specific enough to guide behaviour, flexible enough to give the model strong
+heuristics.** A useful test — could a competent new colleague follow this without asking a
+clarifying question, and without being insulted? If they would ask, it is too high. If they
+would feel micromanaged into a corner where their judgement cannot help, it is too low.
+
+## What actually belongs in there
+
+In rough order of how much behaviour each buys:
+
+1. **Tool usage instructions.** Not the tool schema — the *policy*. When to reach for which,
+   what order usually makes sense, what to do when one fails. The largest measured gains come
+   from here, and it is the part most teams leave to the schema alone.
+2. **The role and its boundaries** — what this agent is for, and what it must hand off.
+3. **The vocabulary** it must produce (below).
+4. **Volatile context** it cannot know (below).
+5. **Failure instructions** — what to do when a tool errors, when data is missing, when the
+   task is impossible. Absent, the model invents a recovery, and inventions are not uniform.
+6. **Output contract** — shape, not prose about shape.
+
+## Enumerate the vocabulary
+
+**Be explicit about allowed values.** An agent asked to track task status will produce
+`pending` in one turn and `to-do` in the next, `completed` here and `done` there — and any
+code reading those strings now has a bug that appears intermittently and reads as
+flakiness.
+
+This generalises past status: every category, label, severity, priority or state the agent
+emits should appear as an enumerated set in the prompt, or in the tool's parameter `enum`,
+or both. It is the cheapest determinism available.
+
+## Inject what the model cannot know
+
+**Today's date is the canonical example**, and its absence has a specific failure signature:
+the agent answers from training data instead of searching, confidently and with no error.
+Inject the date and the behaviour changes without another word of instruction.
+
+The general rule: anything volatile that the model would otherwise fill from memory —
+current date, environment, tenant, available capabilities, the user's locale — is injected,
+not assumed. A capability-aware prompt that lists only the tools actually connected beats a
+static prompt describing tools that may be absent.
+
+## Flexible while learning, strict in production
+
+The same instruction should not survive the whole lifecycle.
+
+- **While you are still learning what good looks like:** *"Use the tools in the order that
+  makes most sense to you."* This surfaces what the model thinks the task is, which is the
+  information you need.
+- **Once the sequence is known and a skipped step is a defect:** *"You MUST execute a web
+  search for each task."* Flexibility here buys nothing and costs a silently missing step.
+
+Teams get stuck at the first form because it felt elegant, then debug an agent that
+"sometimes forgets" — which is not forgetting, it is permission.
+
+## Structure, examples, and the cost of formatting
+
+**Structure inputs and outputs** with delimiters, XML tags or JSON. Clear segmentation
+reduces the class of error where the model treats data as instruction.
+
+**Examples are worth more than description**, and the mistake is quantity. Curate a few
+**diverse, canonical** examples rather than an exhaustive list of edge cases — the latter
+reads as a lookup table and the model generalises from it badly.
+
+**But formatting has a cost.** Keep formats close to natural internet text where you can;
+elaborate escaping, deeply nested structures and unusual syntaxes spend the model's
+attention on parsing rather than the task. Ask whether a human writing this by hand would
+choose the format. If not, it is overhead.
+
+## Reasoning models change three things
+
+Treating a reasoning model like a completion model is now a common and expensive mistake.
+
+1. **Do not add chain-of-thought instructions.** Native reasoning already happens.
+   Explicit step-by-step prompting is redundant and **can hurt instruction-following** —
+   which is the opposite of what the person adding it intends.
+2. **Give goals, not procedures.** Be explicit about the high-level outcome and let the
+   model plan the route. Procedural micro-steps fight the thing you are paying for.
+3. **Reasoning effort is a dial that did not exist before** — low/medium/high trades cost
+   against accuracy per call, so it is a per-stage decision, not a global setting.
+
+Two more, worth knowing before you architect around them: **few-shot is still useful, but
+mainly for output *format***, not for teaching the task; and **tool-calling remains weaker
+in most reasoning models**, which is why the common shape is a reasoning model for planning
+and a different one for execution.
+
+## Traps
+
+- **Growing the prompt instead of fixing it.** Every incident adds a sentence; nothing is
+  ever removed; a year later nobody can say which line does work. Prune on the same schedule
+  you add.
+- **Describing tools twice**, in the schema and in the prompt, with the two drifting. Put
+  *policy* in the prompt and *contract* in the schema, and say which is which.
+- **A prompt that assumes a tool exists.** Capability-aware assembly, or an explicit
+  fallback; never a promise the runtime may not keep.
+- **Tuning the prompt with no eval.** You are optimising against the last thing you noticed.
+  See `agent-evals`.
+- **One prompt for planning and execution.** Separation of concerns measurably improves
+  reliability and lets a cheaper model take the mechanical half.

package/plugins/agent-stack/skills/agent-harness/references/techniques.md

@@ -0,0 +1,115 @@
+# The technique catalogue, with a verdict on each
+
+**Load this when:** choosing between ReAct, reflection, voting, planning and the rest — or
+being asked why the system does not use one that appeared in a paper.
+
+**Spec pinned:** `promptingguide.ai` techniques/* and guides/*; Anthropic agent guidance · read 2026-08-14
+
+**How to read the verdicts.** Every technique here works somewhere; the column says whether
+it earns its cost *in a production agent loop*, which is a narrower question than whether it
+raised a benchmark. Costs are real: each of these multiplies calls, latency or context, and
+several of them were measured on single-turn QA rather than a long-running agent.
+
+## Contents
+
+- The catalogue
+- ReAct, in detail
+- Reflection, in detail
+- Voting and self-consistency
+- What reasoning models made redundant
+- Choosing one
+
+## The catalogue
+
+| Technique | What it is | Verdict for a production agent loop |
+|---|---|---|
+| **Zero-shot** | instruction alone | **Default.** Start here; everything below is a cost you must justify |
+| **Few-shot** | input→output exemplars | **Yes, for format.** Curate a few diverse canonical examples. It teaches shape far better than it teaches judgement, and an exhaustive list makes the model brittle |
+| **Chain-of-thought** | "think step by step" | **Legacy on reasoning models — actively harmful there** (it can degrade instruction-following). Still useful on non-reasoning models for arithmetic and multi-constraint tasks |
+| **ReAct** | interleaved thought → action → observation | **Yes — this is the agent loop.** Most harnesses implement it without naming it. See below for what it does not fix |
+| **Reflexion / self-critique** | actor, evaluator, self-reflection, with episodic memory | **Selectively.** Real gains where a *cheap objective signal* exists — tests pass, query runs, schema validates. Without one, the model grades its own homework |
+| **Self-consistency** | sample N, take the majority | **Rarely.** N× cost and latency for a single answer; needs a well-defined answer to vote on. Use for a high-stakes classification, not for a whole trajectory |
+| **Tree of Thoughts** | explore and prune a branching search | **Almost never in production.** Combinatorial cost, and the pruning heuristic is usually the hard part you have not solved. A planner plus a bounded retry gets most of the value |
+| **ART** (automatic reasoning + tool use) | select exemplars and tools from a task library automatically | **Watch.** The idea — a library of trajectories rather than a hand-written prompt — is where prompt maintenance is heading; the tooling is not settled |
+| **Prompt chaining** | fixed sequence with checks between | **Yes — and prefer it over an agent** wherever the steps are knowable |
+| **Meta prompting** | prompt about the structure of the task, not its content | **Occasionally.** Useful for generating scaffolds; not a loop technique |
+| **Generate-knowledge** | elicit facts first, then answer | **No.** Retrieval solves the same problem with grounding; this invents plausible knowledge |
+| **RAG** | retrieve, then generate | **Yes, and it is a search problem.** Its quality lives outside this skill; what belongs here is *just-in-time* retrieval — see below |
+| **Just-in-time retrieval** | hold lightweight identifiers (paths, queries, links); load at runtime via tools | **Yes.** The scalable default: it mirrors how people work, and keeps the window for reasoning rather than for data |
+| **Structured note-taking** | write notes to durable memory outside the window, read them back | **Yes for long tasks.** Persistent memory at low overhead, and it survives compaction — which a summary of the discussion does not |
+| **Sub-agents** | specialists returning distilled summaries | **Yes where the sub-task has its own context need.** The value is the *isolated window*; a sub-agent returns a 1–2k-token distilled summary, never a transcript |
+
+## ReAct, in detail
+
+**Thought → Action → Observation**, repeated. Reasoning traces and task actions generated in
+an interleaved way, so the plan updates on what the world actually returned.
+
+What it fixes: plain chain-of-thought is isolated from external information and therefore
+suffers **fact hallucination and error propagation** — it reasons confidently past a wrong
+premise. ReAct grounds each step in an observation.
+
+**What it does not fix, and this is under-quoted:**
+
+- It **constrains reasoning flexibility** compared with free-form CoT — the format itself is
+  a cost.
+- **Non-informative results derail it.** A search returning nothing useful leaves the model
+  struggling to reformulate, and it will loop on near-identical queries. This is the failure
+  your iteration guard exists for, and it is why tool errors must teach (`tools.md`).
+- It is strongest **combined** with CoT and self-consistency rather than alone — which is
+  the honest reading of the paper and rarely the reading in a blog post.
+
+## Reflection, in detail
+
+Three roles, and naming them separates the ones people conflate:
+
+| Role | Does | In practice |
+|---|---|---|
+| **Actor** | generates text and actions, produces a trajectory | your existing loop |
+| **Evaluator** | scores the trajectory | **the part that decides whether this works at all** |
+| **Self-reflection** | turns the score into verbal guidance stored for next time | an extra call, plus memory |
+
+**The evaluator is the whole question.** Where the signal is objective and cheap — the test
+suite ran, the SQL executed, the JSON validated, the build passed — reflection is one of the
+strongest available techniques. Where the evaluator is the same model judging its own
+output with no ground truth, you have added cost and a confident second opinion.
+
+Stated limitations worth carrying: it depends on **accurate self-evaluation**, its memory is
+typically a **sliding window**, and it struggles where correctness is non-deterministic.
+
+## Voting and self-consistency
+
+Sample the same task several times and take the majority. It genuinely reduces variance —
+and it multiplies cost and latency by N, needs a discrete answer to vote on, and does
+nothing for a long trajectory where the runs diverge at step three.
+
+Reach for it on a **single high-stakes decision** — a routing classification, a safety
+judgement, an extraction that everything downstream depends on. Not on a whole agent run.
+
+## What reasoning models made redundant
+
+A live shift, and it invalidates a lot of otherwise-good advice:
+
+- **Do not instruct step-by-step thinking.** It is native, and explicit CoT can hurt
+  instruction-following.
+- **Give high-level goals rather than procedures**, and let planning happen inside the model.
+- **Reasoning effort** is now a per-call dial, which is a cheaper knob than most of the
+  techniques above.
+- But **tool-calling stays weaker** in most reasoning models — so the common production shape
+  is a reasoning model that plans and a different model that executes tools. That is
+  *separation of concerns*, and it also lets the cheap half be cheap.
+
+## Choosing one
+
+Ask in this order, and stop at the first yes:
+
+1. **Are the steps knowable?** → prompt chaining, not an agent.
+2. **Is there a cheap objective signal?** → evaluator–optimizer or reflection.
+3. **Does the sub-task need its own window?** → a sub-agent returning a distilled summary.
+4. **Is one decision disproportionately expensive to get wrong?** → voting, on that decision only.
+5. **Otherwise** → a plain ReAct loop with a bounded iteration guard, and spend the effort on
+   the tools and the prompt instead. That is where the measured gains are.
+
+**The anti-pattern this section exists to prevent:** adopting a technique because it appears
+in a paper, without naming the signal it consumes or the cost it adds. If you cannot say
+what the evaluator measures, you are not doing reflection — you are paying for a second
+opinion from the same source.

package/plugins/agent-stack/skills/agent-harness/references/tools.md

@@ -0,0 +1,154 @@
+# Tools — the agent–computer interface
+
+**Load this when:** the model picks the wrong tool, calls none, calls one with bad arguments,
+or you are deciding what to expose in the first place.
+
+**Spec pinned:** Anthropic *Writing tools for agents* and *Building effective agents*; `promptingguide.ai` agents/function-calling · read 2026-08-14
+
+Anthropic's framing is worth adopting whole: this is the **agent–computer interface**, and it
+deserves the same craft a human interface gets. Most teams spend their effort on the model
+and none on the ACI, then conclude the model is bad at tool use.
+
+## Contents
+
+- Fewer tools than you think
+- Namespacing
+- The description is the product
+- Return meaning, not identifiers
+- Token efficiency is a correctness issue
+- Errors that teach
+- Poka-yoke: make the wrong call impossible
+- Evaluating tools
+- Traps
+
+## Fewer tools than you think
+
+**More tools do not lead to better outcomes.** The reflex — wrap every API endpoint, ship
+forty tools, let the model choose — produces an agent that chooses badly, because selection
+degrades with the size of the set and every definition costs context.
+
+Build **a few thoughtful tools targeting specific high-impact workflows.** The test:
+consolidate where a human would. `search_and_summarize` beats `search` + `fetch` +
+`summarize` when the three are always used together, because it removes two decisions and
+two round trips.
+
+**The honest check, and it is brutal:** *if your engineers cannot definitively say which
+tool applies to a case, the model cannot either.* Ambiguity between two tools is a design
+defect, not a prompting problem.
+
+## Namespacing
+
+Group related tools under a common prefix — `asana_search`, `asana_create_task`,
+`jira_search`. Boundaries become visible, and the model stops crossing services by accident.
+
+This also matters at the federation layer: behind a gateway, tool names commonly gain a
+prefix from their source server, and a name that changes between sessions invalidates every
+prompt and eval that referenced it. See `agent-interop/references/gateway.md`.
+
+## The description is the product
+
+**Even small refinements to tool descriptions yield dramatic improvements.** The rule that
+makes them good: **write as if explaining to a new team member**, and make implicit context
+explicit.
+
+A description must answer **when and why**, not only what:
+
+```jsonc
+// weak — restates the name, and says nothing about choosing it
+{ "name": "search_users", "description": "Searches for users." }
+
+// strong — the model can now decide
+{ "name": "search_users",
+  "description": "Find users by name, email or team. Use this before any operation \
+that needs a user ID — IDs are never guessable. Returns at most 20 matches; narrow \
+with `team` rather than paging when you can. Do NOT use for the current user: \
+`get_current_user` is cheaper and always correct." }
+```
+
+Note what the strong version carries: **when to reach for it, what it costs, how to narrow,
+and the neighbouring tool it is confused with.** That last clause is the highest-value
+sentence in most tool descriptions and almost nobody writes it.
+
+**Parameters carry their own guidance.** Use `enum` to constrain values rather than
+describing the constraint in prose, give examples in parameter descriptions, and mark
+required versus optional honestly — an optional parameter the tool actually needs is a
+silent failure.
+
+## Return meaning, not identifiers
+
+Prioritise **contextual relevance over flexibility**. A response of
+`{"id": "u_8f3a", "gid": "1209...", "rid": 44}` gives the model nothing to reason with; it
+will echo identifiers into prose and hallucinate what they mean. Return
+`{"name": "Ada Lovelace", "team": "Platform", "id": "u_8f3a"}` — the id stays for the next
+call, the meaning arrives for the reasoning.
+
+## Token efficiency is a correctness issue
+
+Not merely a cost issue: a tool that returns 40,000 tokens of JSON has consumed the window
+the agent needed to finish the task, and no amount of history trimming recovers it (see
+`agent-orchestrator/references/context-engineering.md` → *tool-output offload*).
+
+Build in **pagination, range selection, filtering and truncation — with sensible defaults**.
+The default matters more than the capability: an agent will rarely opt into a limit it was
+not given.
+
+## Errors that teach
+
+A tool error is a turn in a conversation. Compare:
+
+```
+Error: 422 Unprocessable Entity
+```
+```
+Error: `due_date` must be ISO-8601 (e.g. 2026-08-14). You sent "next friday".
+Call `resolve_date` first, or pass an absolute date.
+```
+
+The second costs nothing extra and converts a dead end into a recovery. **Return informative
+messages that help the agent recover or try an alternative** — naming the alternative is the
+part that gets skipped.
+
+Two structural notes: under MCP a failed tool arrives as a *successful* response carrying
+`isError: true`, so code that only catches transport exceptions treats every tool failure as
+a success containing an apology (`agent-interop/references/mcp.md`). And in code-mode
+harnesses, generated wrappers should convert that into a thrown exception so model-authored
+code can `try`/`catch`.
+
+## Poka-yoke: make the wrong call impossible
+
+Borrowed from manufacturing, and the highest-leverage idea here: **change the interface so
+the mistake cannot be made**, rather than documenting the mistake.
+
+- Absolute paths instead of relative ones, when relative paths get resolved against a
+  directory the agent guessed.
+- An `enum` instead of a free-text field with a list of valid values in the description.
+- One tool that does the two-step correctly instead of two tools that must be ordered.
+- A required `confirm: true` on a destructive action, so a partially-formed call fails
+  closed.
+
+## Evaluating tools
+
+Tools deserve **thorough documentation and testing**, and testing means running the agent
+against real tasks and reading which tool it picked, with what arguments, and what came
+back. Enable intermediate-step visibility and look for the three recurring faults:
+
+| Symptom | Almost always |
+|---|---|
+| wrong tool chosen | two descriptions do not distinguish themselves; add the "do NOT use for…" clause |
+| bad arguments | the parameter description assumes context the model does not have, or the type is too loose |
+| result misread | the response returned identifiers, or too much, or both |
+
+Fix the interface, not the prompt, when the fault is in this table.
+
+## Traps
+
+- **Wrapping the API you have** instead of designing the tools the agent needs.
+- **A description written for a human reading docs** rather than a model choosing under
+  uncertainty.
+- **Unbounded responses** with an optional `limit` nobody sets.
+- **Errors that are true and useless.** `null is not an object` names the symptom the model
+  can do nothing with.
+- **Treating tool output as trusted.** It is attacker-controlled input if the server is; the
+  specification says descriptions and annotations are untrusted unless the server is.
+- **Adding a tool to fix a prompt problem.** The set grows, selection degrades, and the
+  original defect is still there.