@damiangilgonzalez/harness 1.3.7

package/skills/bro/SKILL.md

@@ -0,0 +1,76 @@
+---
+name: bro
+description: "Use whenever the user asks to make an answer or draft sound human, natural, plain-spoken or less AI-written — including a terse bro after the last response. Rewrites or drafts in the same language while preserving meaning, facts and channel. NOT a reusable brand voice (that is `brand-voice`), NOT translation (that is `translation-l10n`)."
+tags: [bro, human-writing, humano, plain-language, natural-language, rewrite, no-jargon]
+recommends: [unslop, brand-voice, technical-writing, translation-l10n]
+profiles: [minimal, core, ui, full]
+origin: risco
+---
+
+# bro — say it like a person
+
+Make the words sound like someone with something real to say. Keep the substance; remove the
+performance around it.
+
+## The contract
+
+1. **Find the source without ceremony.** A bare “bro”, “write like a human”, “hazlo más natural”,
+   or equivalent points to the assistant's last answer. If the user pasted text, rewrite that. If
+   they requested new copy, draft it directly. Ask for text only when no source exists.
+2. **Preserve the payload.** Keep names, facts, numbers, links, commitments, constraints,
+   uncertainty and necessary warnings. Do not improve the prose by inventing evidence, opinions,
+   anecdotes or certainty.
+3. **Match the room.** Keep the user's language and fit the audience, relationship and channel. A
+   natural legal notice is still precise; a natural Slack message can be loose. “Human” does not
+   automatically mean casual.
+4. **Cut the bot-shaped scaffolding.** Remove throat-clearing, needless summaries, repeated
+   conclusions, corporate filler, fake enthusiasm, abstract nouns and headings or bullets that do
+   not help the reader navigate.
+5. **Restore a real cadence.** Prefer concrete verbs, direct sentences and natural contractions.
+   Vary sentence length when it helps. Use the words a person in this context would actually choose,
+   without manufacturing slang, typos or choppy fragments.
+6. **Return the writing, not a review of the writing.** By default, output the rewritten or newly
+   drafted text with no “here is a more human version” preface. Explain the changes only if asked.
+
+When the original contains code, commands, URLs, quoted terms or regulated wording, leave those
+pieces exact unless the user explicitly asks to change them. Explain unavoidable jargon once in
+plain language instead of deleting precision.
+
+## What “human” means here
+
+| Aim | The result |
+| --- | --- |
+| Plain, not simplistic | Easy to follow without losing the idea |
+| Concise, not incomplete | No padding; all load-bearing facts remain |
+| Warm, not performative | Appropriate care without canned empathy |
+| Conversational, not sloppy | Natural rhythm without fake mistakes |
+| Specific, not decorated | Concrete nouns and verbs instead of hype |
+
+## Anti-patterns
+
+| Anti-pattern | Why it fails | Do this instead |
+| --- | --- | --- |
+| Add typos, slang or filler to seem human | It performs humanity and can sound less credible | Match the actual relationship and channel |
+| Replace technical terms with vague words | Readability bought with lost meaning is a defect | Keep or define the necessary term |
+| Invent a personal story or strong opinion | A smoother fabrication is still a fabrication | Preserve only what the source supports |
+| Use the same “casual” voice for every audience | A client email, README and condolence need different registers | Infer the room before choosing the cadence |
+| Promise the result is undetectable as AI | No rewrite can honestly guarantee that | Promise clarity and naturalness, not detector outcomes |
+| Announce and justify every edit | The preamble becomes the same friction the user asked to remove | Lead with the finished words |
+
+## Where this ends
+
+`unslop` owns the other half of this job: the **named** audit of a text that already exists, tell by
+catalogued tell (puffery, `not just X but Y`, em dashes, bold-label lists, hedging), run before the
+text ships. This skill owns the **register**: which language, which relationship, which channel, and
+the cadence that fits them. A bare "bro" is always this skill. "Check this for AI tells before I
+publish it" is `unslop`. When a text needs both, audit first, register last.
+
+## Output boundary
+
+Treat the rewritten text as an artifact: keep harness commentary outside it. If the task also
+requires a brújula block, place that block after a clear separation so it cannot be copied into an
+email, document or message by accident.
+
+## Orientación (siempre)
+
+Cierra cada turno con el **bloque-brújula** (📍 dónde estás · ✅ qué hiciste · 🧭 por qué · ➡️ siguiente, terminando en pregunta), calibrado al dial de `docs/wiki/harness/user-profile.md`. **Nunca termines en seco.** Protocolo completo: skill `orient` → `skills/orient/references/orientation-contract.md`. (Defiere a `suggest` el "¿instalo la skill que falta?".)

package/skills/bro/evals/README.md

@@ -0,0 +1,22 @@
+# Evals — bro
+
+These cases test two different claims: that human-language requests route to `bro` without
+stealing adjacent writing jobs, and that the loaded skill produces a more natural result without
+changing the payload.
+
+## Triggering
+
+Load only `bro` and run every `should_trigger` and `should_not_trigger` prompt in 3–5 fresh
+sessions. A positive passes when `bro` loads in a majority of trials. A negative passes when it
+stays quiet; also check that the declared `route_to` skill is the genuine owner. Require at least
+90% accuracy across all cases.
+
+## Capability
+
+Run the capability scenario in two fresh conditions: without `bro`, then with it. Use the same
+source answer and one consistent grader. Score each `must_include` point over three trials. The
+with-skill condition must satisfy at least 80% of the rubric and beat the baseline by at least 20
+percentage points.
+
+This is an LLM/human-judged A/B eval, not deterministic CI. Re-run it after changing either the
+description or the body; routing and prose quality are both wording-sensitive.

package/skills/bro/evals/cases.yaml

@@ -0,0 +1,62 @@
+skill: bro
+
+# `bro` is the final-language pass: it rewrites an existing answer/draft or
+# drafts requested text in a natural register. It does not define a reusable
+# brand system, translate content, or choose a documentation/copy strategy.
+should_trigger:
+  - prompt: "Bro, escribe como un humano."
+    why: "A terse bro-style instruction after an answer is the canonical request to rewrite the last assistant message naturally."
+
+  - prompt: "Rewrite your last answer so it sounds like a person, not a consultant."
+    why: "The user wants a human-language pass over the previous answer while preserving its content."
+
+  - prompt: "Haz este correo más natural y directo, pero no cambies las fechas ni los compromisos."
+    why: "Rewriting a supplied draft while preserving facts and commitments is the skill's core contract."
+
+  - prompt: "Això sona escrit per una IA. Fes-ho més humà i sense tanta cerimònia."
+    why: "Catalan request to remove AI-shaped formality tests multilingual semantic triggering."
+
+  - prompt: "Can you say that again in plain language without all the corporate jargon?"
+    why: "The symptom is jargon and artificial formality rather than the skill name, but the needed action is the same rewrite."
+
+  - prompt: "Escríbeme una respuesta breve y humana para decirle que llegaremos un día tarde."
+    why: "The user requests new text rather than a rewrite, but explicitly wants the same natural, truthful voice."
+
+  - prompt: "Keep all the technical details, just make it sound less robotic."
+    why: "Humanizing without deleting precision exercises the distinction between plain language and simplification."
+
+should_not_trigger:
+  - prompt: "Define a reusable voice guide so every writer and channel sounds like our brand."
+    route_to: "brand-voice"
+    why: "A standing voice system is brand-voice; bro only performs a local writing pass."
+
+  - prompt: "Translate this onboarding email from English to Spanish and adapt the date and currency formats."
+    route_to: "translation-l10n"
+    why: "Translation and locale-aware adaptation are owned by translation-l10n, even when natural prose matters."
+
+  - prompt: "Turn this mixed README into a clean tutorial and API reference with runnable examples."
+    route_to: "technical-writing"
+    why: "Choosing documentation modes and restructuring technical docs is technical-writing, not a final tone pass."
+
+  - prompt: "Write the hero, value proposition and CTA for our pricing page."
+    route_to: "landing-copy"
+    why: "Conversion architecture and finished landing-page copy belong to landing-copy; bro may only polish its language afterward."
+
+  - prompt: "Rewrite this function to be simpler and remove the unnecessary abstractions."
+    route_to: "simplify-code"
+    why: "The requested source-code simplification is owned by simplify-code; 'rewrite' does not make it a prose task."
+
+  - prompt: "Revisa este README antes de publicarlo y dime qué frases suenan a IA, una por una."
+    route_to: "unslop"
+    why: "A named, per-phrase audit of an existing text before shipping is unslop's contract; bro rewrites the register instead of enumerating tells."
+
+capability:
+  - scenario: "The previous assistant answer is a long Spanish status update full of headings, repeated summaries, corporate jargon and fake enthusiasm. Rewrite it for a client while preserving a June 30 deadline, a EUR 2,400 price, one unresolved risk, two URLs and the original uncertainty."
+    must_include:
+      - "preserves June 30, EUR 2,400, both URLs, the unresolved risk and the original uncertainty exactly"
+      - "uses natural Spanish appropriate for a client rather than automatically becoming slangy or over-casual"
+      - "removes needless headings, repeated conclusions, corporate filler and canned enthusiasm"
+      - "uses concrete verbs and readable sentence rhythm without adding typos or forced fragments"
+      - "does not invent evidence, anecdotes, opinions, commitments or certainty"
+      - "returns the rewritten message without a meta preface or an explanation of the edits"
+      - "keeps any harness orientation clearly outside the client-ready artifact"

package/skills/building-agents/SKILL.md

@@ -0,0 +1,386 @@
+---
+name: building-agents
+description: "Use when building or restructuring an LLM agent — provider adapter, tool calling, structured output, RAG, agent loop, eval gate, cost routing, tracing, MCP server — model-agnostic across OpenAI/Anthropic/Gemini/OSS so a model swap is a config change. NOT vector-store SQL alone (that is `postgresdb`) or service deployment (that is `deployment`)."
+tags: [agents, llm, mcp, rag, evals, ai]
+recommends: [secure-coding, deployment]
+origin: risco
+---
+
+# Building production LLM agents (model-agnostic)
+
+A thin provider adapter, a disciplined agent loop, schema-validated tools, provider-neutral RAG, eval gates, OTel tracing, and optionally an MCP server — so swapping OpenAI ↔ Anthropic ↔ Gemini ↔ OSS is a config change, not a rewrite.
+
+## The one rule
+
+> Program against a **capability interface**, never a vendor SDK. Vendor specifics (model id, tool-schema shape, JSON mode, caching, token limits) live behind one adapter resolved from config. Model names and prices rot — if one appears in business logic it's a bug, and re-verify the dated tables before quoting a number.
+
+**Hand off instead when:** a new non-trivial feature has no approved spec + plan under `docs/wiki/sdd/` → stop and write one first (the `to-spec` skill, or `superpowers:writing-plans`), which routes back here once the plan is approved; one-line/low-risk changes go straight through. Anthropic-SDK internals (caching, thinking, batch) in a file that *only* imports `anthropic` → `claude-api` if your environment has it, since this skill stays multi-provider. Workspace scaffolding → [`harness`](../harness/SKILL.md). Choosing *which* coding agent to use → agent-eval territory. Pure prompt-wording tuning with no architecture change → prompt engineering, not this. A one-shot throwaway prompt, or no retrieval/tools/loop/evals at all → you don't need an agent; call the SDK directly and say so.
+
+## Decision rules (read before writing code)
+
+1. **Adapter first** — define the `LLMProvider` Protocol before any provider call.
+2. **Smallest loop that works** — single-agent before multi-agent; ReAct only when the path is uncertain; plan-execute when steps are knowable. Multi-agent means orchestrator-worker with a semaphore-bounded parallel fan-out, never a free-for-all.
+3. **Tools are typed contracts** — schema + validation + idempotency key on every side-effecting tool; no catch-all tools.
+4. **Retrieve, don't stuff** — RAG when ground truth lives in data; cite or refuse.
+5. **Eval before ship** — a golden set + regression gate in CI, or it's not production.
+6. **Cheapest model that passes the eval** — route/cascade up, never default to flagship.
+
+## The provider adapter (the heart of the skill)
+
+The one payload to internalize. Python 3.12+, Pydantic v2, **async** so it composes directly with the bounded loop (and orchestrator-worker fan-out) in [`references/agent-loops-and-harness.md`](references/agent-loops-and-harness.md). Structured output is the quirk that differs most per vendor: strict JSON Schema (OpenAI), tool-forcing (Anthropic), `response_json_schema` (Gemini). Streaming, the Gemini and OSS/litellm adapters, tool-result plumbing, and a `route()` registry live in [`references/provider-abstraction.md`](references/provider-abstraction.md) — this excerpt is the load-bearing core, not the whole interface.
+
+```python
+from __future__ import annotations
+
+import os
+from typing import Literal, Protocol, runtime_checkable
+
+from pydantic import BaseModel, Field
+
+
+class Message(BaseModel):
+    role: Literal["system", "user", "assistant", "tool"]
+    content: str
+
+
+class ToolSpec(BaseModel):
+    name: str
+    description: str
+    parameters: dict  # JSON Schema for the tool's arguments
+
+
+class Usage(BaseModel):
+    input_tokens: int = 0
+    output_tokens: int = 0
+    cost_usd: float = 0.0
+
+
+class CompletionRequest(BaseModel):
+    model: str  # resolved from config, e.g. "claude-sonnet-4-6" — never literal in logic
+    messages: list[Message]
+    tools: list[ToolSpec] = Field(default_factory=list)
+    response_schema: dict | None = None  # JSON Schema -> structured output
+    temperature: float = 0.0
+    max_tokens: int = 1024
+
+
+class CompletionResponse(BaseModel):
+    text: str = ""
+    tool_calls: list[dict] = Field(default_factory=list)  # [{id, name, arguments}]
+    usage: Usage = Field(default_factory=Usage)
+    raw: dict | None = None
+
+
+@runtime_checkable
+class LLMProvider(Protocol):
+    # Async so it drives the async agent loop directly. The full interface in
+    # references/provider-abstraction.md adds stream() and embed().
+    async def complete(self, req: CompletionRequest) -> CompletionResponse: ...
+
+
+class OpenAIAdapter:
+    def __init__(self, model: str) -> None:
+        from openai import AsyncOpenAI
+
+        self.model, self.client = model, AsyncOpenAI()
+
+    async def complete(self, req: CompletionRequest) -> CompletionResponse:
+        # Chat Completions shape (universal, still current); references/provider-abstraction.md
+        # gives the preferred Responses-API adapter. system stays a `system` role message here.
+        kwargs: dict = {"model": self.model, "messages": [m.model_dump() for m in req.messages],
+                        "temperature": req.temperature, "max_tokens": req.max_tokens}
+        if req.tools:
+            kwargs["tools"] = [{"type": "function", "function": {"name": t.name, "description": t.description, "parameters": t.parameters}} for t in req.tools]
+        if req.response_schema:
+            kwargs["response_format"] = {"type": "json_schema", "json_schema": {"name": "out", "schema": req.response_schema, "strict": True}}
+        r = await self.client.chat.completions.create(**kwargs)
+        msg = r.choices[0].message
+        calls = [{"id": c.id, "name": c.function.name, "arguments": c.function.arguments} for c in (msg.tool_calls or [])]
+        return CompletionResponse(text=msg.content or "", tool_calls=calls, raw=r.model_dump(),
+            usage=Usage(input_tokens=r.usage.prompt_tokens, output_tokens=r.usage.completion_tokens))
+
+
+class AnthropicAdapter:
+    def __init__(self, model: str) -> None:
+        from anthropic import AsyncAnthropic
+
+        self.model, self.client = model, AsyncAnthropic()
+
+    async def complete(self, req: CompletionRequest) -> CompletionResponse:
+        # QUIRKS: system is a top-level param (not a message); tools use input_schema (not function).
+        system = "\n".join(m.content for m in req.messages if m.role == "system") or None
+        turns = [{"role": m.role, "content": m.content} for m in req.messages if m.role != "system"]
+        kwargs: dict = {"model": self.model, "system": system, "messages": turns, "max_tokens": req.max_tokens, "temperature": req.temperature}
+        if req.tools:
+            kwargs["tools"] = [{"name": t.name, "description": t.description, "input_schema": t.parameters} for t in req.tools]
+        if req.response_schema:  # structured output via tool-forcing
+            kwargs["tools"] = [{"name": "out", "description": "Emit the result", "input_schema": req.response_schema}]
+            kwargs["tool_choice"] = {"type": "tool", "name": "out"}
+        r = await self.client.messages.create(**kwargs)
+        text = "".join(b.text for b in r.content if b.type == "text")
+        calls = [{"id": b.id, "name": b.name, "arguments": b.input} for b in r.content if b.type == "tool_use"]
+        return CompletionResponse(text=text, tool_calls=calls, raw=r.model_dump(),
+            usage=Usage(input_tokens=r.usage.input_tokens, output_tokens=r.usage.output_tokens))
+
+
+def get_provider(spec: str | None = None) -> LLMProvider:
+    """Parse 'provider:model' (default from env LLM) into a concrete adapter."""
+    provider, _, model = (spec or os.environ["LLM"]).partition(":")
+    if provider == "openai":
+        return OpenAIAdapter(model)
+    if provider == "anthropic":
+        return AnthropicAdapter(model)
+    raise ValueError(f"unknown provider: {provider!r}")
+# Gemini + OSS/litellm adapters, streaming, tool-result plumbing, and route() registry
+# -> references/provider-abstraction.md
+```
+
+## Good vs Bad
+
+Call-sites use the adapter and never name a model: `provider = get_provider(settings.llm)` (e.g. `"anthropic:claude-sonnet-4-6"`), then `await provider.complete(req)`. The two failures that survive that discipline:
+
+```python
+# BAD — parse-and-pray; wrong shape fails silently at 3am.
+raw = (await provider.complete(req)).text
+try:
+    data = json.loads(raw)
+except json.JSONDecodeError:
+    data = {}  # the bug is now invisible
+```
+
+```python
+# GOOD — strict structured output + schema validation that fails loudly on drift.
+class Answer(BaseModel):
+    sentiment: Literal["pos", "neg", "neu"]
+    score: float
+
+req.response_schema = Answer.model_json_schema()
+ans = Answer.model_validate_json((await provider.complete(req)).text)
+```
+
+```python
+# BAD — unbounded loop; no cap/timeout/idempotency. Burns budget, repeats side effects, wedges.
+while True:
+    resp = await provider.complete(req)
+    if not resp.tool_calls:
+        break
+    for call in resp.tool_calls:
+        await run_tool(call)
+```
+
+```python
+# GOOD — bounded loop: step cap + per-tool timeout + idempotency key (safe to retry).
+for step in range(max_steps):
+    resp = await provider.complete(req)
+    if not resp.tool_calls:
+        break
+    for call in resp.tool_calls:
+        async with asyncio.timeout(tool_timeout_s):
+            await run_tool(call, idempotency_key=call["id"])
+# full loop, budgets, recovery -> references/agent-loops-and-harness.md
+```
+
+## Tools & structured output (minimum viable)
+
+```python
+from typing import Callable, Literal
+
+from pydantic import BaseModel, ConfigDict, Field, ValidationError
+
+
+class CreateInvoiceArgs(BaseModel):
+    model_config = ConfigDict(extra="forbid")  # reject unknown keys from the model
+    customer_id: str = Field(min_length=1)
+    amount_cents: int = Field(gt=0)
+    currency: Literal["EUR", "USD"] = "EUR"
+
+
+class ToolResult(BaseModel):
+    status: Literal["success", "warning", "error"]
+    summary: str
+    data: dict | None = None
+    next_actions: list[str] = Field(default_factory=list)
+
+
+def _create_invoice(args: CreateInvoiceArgs) -> ToolResult:
+    invoice_id = f"inv_{args.customer_id}_{args.amount_cents}"  # real impl: DB insert + idempotency
+    return ToolResult(status="success", summary=f"Created {invoice_id}", data={"id": invoice_id})
+
+
+TOOLS: dict[str, tuple[type[BaseModel], Callable]] = {
+    "create_invoice": (CreateInvoiceArgs, _create_invoice),
+}
+
+
+def dispatch(name: str, raw_args: dict) -> ToolResult:
+    spec = TOOLS.get(name)
+    if spec is None:
+        return ToolResult(status="error", summary=f"unknown tool {name!r}", next_actions=["pick a registered tool"])
+    args_model, handler = spec
+    try:
+        args = args_model.model_validate(raw_args)  # validate BEFORE side effects
+    except ValidationError as e:
+        return ToolResult(status="error", summary="invalid args", data={"errors": e.errors()},
+                          next_actions=["fix the arguments and retry"])
+    return handler(args)
+```
+
+Schema design, sandboxing, idempotency, DI-scoped DB sessions, plus the RAG internals below — chunking, hybrid RRF, rerank, the citation grader, memory — are in [`references/tools-and-rag.md`](references/tools-and-rag.md).
+
+## RAG in 30 lines (provider-agnostic embeddings)
+
+```sql
+CREATE EXTENSION IF NOT EXISTS vector;
+CREATE TABLE IF NOT EXISTS docs (
+    id        bigserial PRIMARY KEY,
+    content   text NOT NULL,
+    embedding vector(1536) NOT NULL,
+    meta      jsonb NOT NULL DEFAULT '{}'
+);
+CREATE INDEX IF NOT EXISTS docs_embedding_hnsw
+    ON docs USING hnsw (embedding vector_cosine_ops);
+```
+
+```python
+async def embed(texts: list[str]) -> list[list[float]]:
+    # Same provider interface as completions; impl in references/tools-and-rag.md.
+    return await provider.embed(texts)  # returns one 1536-d vector per text
+
+
+async def retrieve(query: str, k: int = 5, min_sim: float = 0.25) -> list[dict]:
+    [q] = await embed([query])
+    rows = await db.fetch(  # cosine distance <=>; similarity = 1 - distance
+        "SELECT id, content, 1 - (embedding <=> $1) AS sim "
+        "FROM docs ORDER BY embedding <=> $1 LIMIT $2",
+        q, k,
+    )
+    return [dict(r) for r in rows if r["sim"] >= min_sim]
+
+
+async def answer(query: str) -> str:
+    chunks = await retrieve(query)
+    if not chunks:                                   # refuse rather than hallucinate
+        return "I don't have grounded information to answer that."
+    context = "\n".join(f"[{c['id']}] {c['content']}" for c in chunks)
+    req = CompletionRequest(
+        model=settings.model_id,
+        messages=[Message(role="system", content="Answer ONLY from context; cite chunk ids like [12]."),
+                  Message(role="user", content=f"{context}\n\nQ: {query}")],
+    )
+    return (await provider.complete(req)).text
+```
+
+## Evals & cost gates (the production line)
+
+```python
+import json
+import statistics
+import sys
+import time
+
+
+async def run_eval(golden_path: str, graders: list, thresholds: dict[str, float]) -> None:
+    cases = [json.loads(line) for line in open(golden_path)]  # {"input","expected","meta"}
+    results = []
+    for case in cases:
+        t0 = time.perf_counter()
+        out = await provider.complete(CompletionRequest(model=settings.model_id,
+              messages=[Message(role="user", content=case["input"])]))
+        scores = {g.name: g.grade(case, out) for g in graders}  # exact / schema / LLM-judge
+        results.append({"scores": scores, "cost": out.usage.cost_usd,
+                        "ms": (time.perf_counter() - t0) * 1000})
+    n = len(results)
+    metrics = {
+        "accuracy": sum(r["scores"]["exact"] for r in results) / n,
+        "faithfulness": sum(r["scores"]["judge"] for r in results) / n,
+        "p95_latency_ms": statistics.quantiles([r["ms"] for r in results], n=20)[-1],
+        "cost_per_task": sum(r["cost"] for r in results) / n,
+    }
+    failed = [k for k, lo in thresholds.items() if metrics[k] < lo]
+    print(json.dumps(metrics, indent=2))
+    sys.exit(1 if failed else 0)  # CI gate: non-zero blocks the merge
+```
+
+Routing cascade in one line: `route(task) → cheapest model whose eval passes; escalate only on a failed self-check`.
+Full runner, judge, CI gate, caching, batching and budgets →
+[`references/evals-and-observability.md`](references/evals-and-observability.md).
+
+## Observability (OTel GenAI, vendor-neutral)
+
+```python
+from opentelemetry import trace
+
+tracer = trace.get_tracer("agent")
+
+
+async def traced_complete(provider: LLMProvider, req: CompletionRequest) -> CompletionResponse:
+    with tracer.start_as_current_span("chat") as span:
+        span.set_attribute("gen_ai.system", settings.llm.split(":")[0])
+        span.set_attribute("gen_ai.request.model", req.model)
+        resp = await provider.complete(req)
+        span.set_attributes({"gen_ai.usage.input_tokens": resp.usage.input_tokens,
+                             "gen_ai.usage.output_tokens": resp.usage.output_tokens,
+                             "gen_ai.usage.cost_usd": resp.usage.cost_usd})
+        return resp
+# Langfuse / Phoenix / Braintrust are swappable OTLP backends: emit spans, swap the exporter.
+# span-per-tool, trace-id propagation, exporters -> references/evals-and-observability.md
+```
+
+## MCP: when and the smallest server
+
+**Native tools** when the agent and tools share a process/repo. **MCP** when tools must be reused across clients/teams or run out-of-process — accept the MCP cost (schema tokens, transport, ops) in exchange for reuse. TypeScript server, transports, HTTP+auth and testing are in [`references/mcp-servers.md`](references/mcp-servers.md).
+
+```python
+from fastmcp import FastMCP  # standalone fastmcp 2.x
+
+mcp = FastMCP("invoices")
+
+
+@mcp.tool()
+def create_invoice(customer_id: str, amount_cents: int, currency: str = "EUR") -> dict:
+    """Create an invoice. amount_cents must be > 0."""
+    if amount_cents <= 0:
+        raise ValueError("amount_cents must be positive")
+    return {"id": f"inv_{customer_id}_{amount_cents}", "currency": currency}
+
+
+@mcp.resource("invoice://{invoice_id}")
+def read_invoice(invoice_id: str) -> str:
+    """Read-only invoice lookup by id."""
+    return f"Invoice {invoice_id}: status=open"
+
+
+if __name__ == "__main__":
+    mcp.run()  # stdio transport
+# (MCP spec 2025-11-25; stateless-core RC 2026-07-28; verify before quoting)
+```
+
+## Anti-patterns
+
+| Anti-pattern | Reality |
+|---|---|
+| "I'll just call the OpenAI SDK directly, we'll never switch" | The adapter is ~40 lines; retrofitting it across 30 call-sites later is a rewrite. Adapter first. |
+| "JSON output is usually valid, I'll parse it" | "Usually" = pages at 3am. Use strict structured output + schema validation. |
+| "The agent loop works, I don't need a step cap" | Unbounded loops burn budget and wedge on errors. Cap steps, timeouts, and budget. |
+| "One mega-tool that takes a freeform command is flexible" | It's unobservable and unsafe. Narrow typed tools with idempotency keys. |
+| "We can eval by eyeballing outputs" | Vibes don't gate CI. Golden set + graders + threshold or it's not production. |
+| "Default everything to the flagship model, it's smartest" | 5–20× cost for no measured gain. Route to the cheapest model that passes the eval. |
+| "Stuff the whole doc in the prompt instead of RAG" | Blows context + cost and still hallucinates. Retrieve + cite + refuse. |
+| "Retry on every exception" | Retrying a 400/401 wastes budget. Retry only transient (429/5xx/timeout) with backoff+jitter. |
+| "Hardcode the model name, it's fine" | Names rot (Opus 4.7 → 4.8 in weeks). Resolve from config/registry. |
+| "MCP for everything" | In-process native tools are simpler and faster when reuse isn't needed. MCP only for cross-client reuse. |
+| "Tool results just return the raw API blob" | Give the model `status/summary/next_actions`; raw blobs waste context and stall recovery. |
+| "Prompt caching is Anthropic-only so skip caching" | Each provider has its own caching/dedup; abstract it behind the adapter, don't skip it. |
+
+## verify.sh
+
+`scripts/verify.sh` lints example agent code and dry-runs the eval smoke test in **the user's project** — not in this skill repo. It detects each tool (`ruff`, `mypy`, `tsc`/`node`, `go`, the eval entrypoint, `markdownlint`) and skips any that are missing with a yellow WARN; a missing tool never fails the run. Invoke it with `bash scripts/verify.sh` from the project root. Exit 0 means clean (or only skips); a non-zero exit means a real lint/typecheck/vet/eval failure.
+
+## Project grounding
+
+In a project with a `docs/` layer ([`harness`](../harness/SKILL.md)), read `docs/wiki/stack/agents.md` first on every use and stay consistent with it. If it is missing or stale, write this project's real choices there — provider(s) and model routing, where the adapter lives, tool/RAG conventions, eval gates, observability backend — as a `type: stack` article per the harness [`wiki-article-template.md`](../harness/references/wiki-article-template.md), index it in `docs/wiki/index.md`, and bump its `timestamp` in the same change as any convention change. No `docs/`? Skip silently — technical conventions here are *recorded, not gated*; never block the task on this.
+
+## See also
+
+- Stacks the examples target: [`fastapi`](../fastapi/SKILL.md), [`nextjs`](../nextjs/SKILL.md), [`go`](../go/SKILL.md), [`postgresdb`](../postgresdb/SKILL.md), [`flutter`](../flutter/SKILL.md). Harden with [`secure-coding`](../secure-coding/SKILL.md), ship with [`deployment`](../deployment/SKILL.md).
+- External (no sibling here; use if your environment provides them): `claude-api` for Anthropic-SDK-only tuning, `deep-research` for the research-harness fan-out / verify pattern.

package/skills/building-agents/evals/README.md

@@ -0,0 +1,68 @@
+# Eval harness — `building-agents`
+
+This evaluates two things: (1) **triggering** — does the skill activate on the
+right prompts and stay quiet on near-misses — and (2) **capability** — does
+loading the skill measurably improve the agent's answer. These run through an
+**agent harness** (a Claude Code / SDK session), not a pure shell script: a
+human or a driver script feeds prompts and grades outcomes against
+`cases.yaml`.
+
+## Files
+
+- `cases.yaml` — `should_trigger`, `should_not_trigger`, and `capability` cases.
+
+## 1. Triggering eval
+
+Goal: confirm the skill's `description` routes correctly.
+
+1. Load **only** this skill into the agent (its `SKILL.md` description in the
+   skill index). For near-miss realism, also expose the sibling skills named in
+   each `route_to` so the agent can pick the better match instead of defaulting.
+2. For each prompt in `should_trigger` and `should_not_trigger`, start a fresh
+   session and send the prompt verbatim. Do **not** answer the task — observe
+   only whether `building-agents` is selected.
+3. Run **3–5 trials** per prompt (the selection is stochastic). Record the
+   fire-rate per prompt.
+4. Score:
+   - `should_trigger` item passes if the skill fires in a strong majority of
+     trials (>= 4/5).
+   - `should_not_trigger` item passes if the skill does **not** fire, and
+     ideally the agent routes to the `route_to` sibling (or to nothing when
+     `route_to: none`).
+
+**Pass bar:** >= 90% trigger accuracy across all items
+(true-positives + true-negatives / total trials). Any `should_not_trigger`
+that fires the skill is a false-positive and must be investigated — usually a
+too-greedy `description`.
+
+## 2. Capability eval
+
+Goal: prove the skill changes the output, not just the routing.
+
+1. For each `capability` scenario, run it **twice** in matched sessions:
+   - **WITHOUT** the skill loaded (baseline).
+   - **WITH** the skill loaded.
+2. Grade each answer against that scenario's `must_include` rubric: count how
+   many points are concretely present (code or prose), not merely gestured at.
+3. Compute coverage = points covered / total points, for both runs.
+
+**Pass bar:** WITH-skill coverage >= 80% of the rubric, AND a clear lift over
+the WITHOUT-skill baseline (the skill must add the load-bearing points the
+baseline misses — typically the adapter/Protocol, idempotency, cite-or-refuse,
+CI gate exit code, and transient-only retries). If the baseline already scores
+~as high, the skill isn't earning its place on those points.
+
+## Honest caveats
+
+- Selection is non-deterministic; always use multiple trials and report rates,
+  not a single pass/fail.
+- Grading `must_include` and "routes to the right sibling" requires judgment —
+  an LLM-as-judge can assist but spot-check it.
+- Keep `cases.yaml` in sync with `SKILL.md`: when the description's triggers or
+  the "Do NOT use" list change, update the cases in the same commit.
+
+## Counts
+
+- should_trigger: 7
+- should_not_trigger: 7
+- capability: 2