@cursor/july 0.1.111 → 0.1.112

package/docs/hillclimbing.md

@@ -1,93 +1,139 @@
 ---
 title: "Hillclimbing"
-description: "Improve an agent one measured round at a time, with the hillclimb skill running the loop with you."
+description: "Improve one agent behavior at a time by measuring fixed inputs, changing one lever, and locking each win with an eval."
 ---
 
 # Hillclimbing
 
-Make an agent better on fixed inputs: measure, change one lever, remeasure, and lock every kept win with an eval.
+Hillclimbing improves an agent through controlled before-and-after
+comparisons. Keep the inputs fixed, name one failure, change one lever,
+and retain the change only when the same run gets better without
+crossing the freeze line.
 
-## What is hillclimbing?
+The companion [`hillclimb` skill](../skills/hillclimb/SKILL.md) can run
+this loop with you.
 
-Hillclimbing is a measured improvement loop. You pin a few fixtures, name the one dominant problem in the run, change one lever, and check the same fixtures again. Keep only what helps. Every kept change lands an [eval](./evals.md) so the win stays put.
+## Correct a wrong tool choice
 
-You don't have to run the loop alone. The package ships a coding-agent skill that drives it with you.
+Suppose a PR reviewer calls `approve_pr` before `inspect_pr`. Pin one PR
+fixture, record the original tool order, then tighten the instruction
+that owns the decision. The improved run should inspect first and leave
+approval untouched.
 
-```mermaid
-flowchart LR
-  measure[Measure] --> change[Change one lever]
-  change --> remeasure[Remeasure]
-  remeasure --> measure
+```bash
+agent-sdk run --dir . \
+  --message "Is https://github.com/acme/checkout/pull/42 ready?"
 ```
 
-## How do I hillclimb an agent with a coding agent?
+Lock the result with the intended and tempting-wrong paths:
 
-Have Cursor read [`skills/hillclimb/SKILL.md`](../skills/hillclimb/SKILL.md).
+```ts
+t.succeeded();
+t.calledTool("inspect_pr");
+t.notCalledTool("approve_pr");
+```
+
+Do not change the tool implementation, prompt, and evidence together.
+If tool choice improves, you should know which edit caused it.
 
-Tell it:
+## Reduce wandering or latency
 
-1. **Which agent** you're improving (path or slug)
-2. **One to three fixtures** you'll reuse every round: a PR URL, a saved webhook body, or a canonical chat prompt
-3. **What "better" means** this round: correct tool choice, fewer tools, lower latency, or output quality. Name the freeze line too: API shape, public output, and existing evals that must stay green
+A slow run often repeats reads, searches broad directories, or asks the
+model to discover evidence the host already has. Compare the trajectory
+by tool name and wall time, then move deterministic preparation ahead
+of the turn or trim the evidence shape.
 
-The skill serves the agent, hits your fixtures, reads the session trajectory, proposes one change, remeasures, and checks with you before the next round.
+```text
+before: 14 tool calls, 92 seconds, 6 repeated reads
+after:   5 tool calls, 31 seconds, 0 repeated reads
+```
 
-Other skills cover the edges:
+Keep the change only if the answer still finds the same known issue.
+Efficiency without a quality check can make an agent fast by making it
+incomplete.
 
-| When you need… | Skill |
-| --- | --- |
-| The measured improvement loop | [`skills/hillclimb/SKILL.md`](../skills/hillclimb/SKILL.md) |
-| An eval that locks a kept win | [`skills/evals/SKILL.md`](../skills/evals/SKILL.md) |
-| Repeatable GitHub webhook inputs | [`skills/github/SKILL.md`](../skills/github/SKILL.md) |
-| A run that misbehaves | [`skills/debug/SKILL.md`](../skills/debug/SKILL.md) |
+## Improve quality without changing the contract
 
-See [Building agents with agents](./building-with-agents.md) for every framework skill and a good first prompt.
+For answer quality, freeze the public shape before editing: required
+sections, API fields, side effects, and existing evals. Compare both
+runs against a known-good answer or a small set of gold findings, then
+change the instructions or skill that owns the missing judgment.
+
+```text
+fixture: saved checkout PR diff
+success: report both gold findings with file paths
+freeze:  findings JSON shape; no review post; existing smoke evals
+```
 
-## What do I need before a hillclimb round?
+A wording change is not a win by itself. Keep it when correctness rises
+and every frozen behavior remains intact.
 
-Agree on four things before you edit:
+## Run one measured round
 
-1. **The target agent**: the project you're improving
-2. **Fixtures**: one to three fixed inputs you can compare across runs
-3. **Success criteria**: what better means this round
-4. **The freeze line**: what must not change
+Agree on four inputs before editing:
 
-Pin the input first. A moving fixture is noise. For GitHub agents, use `agent-sdk github replay` (see the [GitHub guide](./guides/github.md)). For a single tool without a model turn, use `agent-sdk call`. For a chat turn, use `agent-sdk run --dir . --message "…"`.
+1. **Target:** the agent path or slug.
+2. **Fixtures:** one to three prompts, saved payloads, or pinned PRs.
+3. **Success:** the metric or decision that should improve.
+4. **Freeze line:** behavior that must not move.
 
-## How do I run one hillclimb round?
+Then run one loop:
 
-**Measure.** Hit the agent the way a user would: playground, channel HTTP, or Slack in `--dev`. Or ask the hillclimb skill to do it. `agent-sdk run` returns a JSON trajectory and writes a trace under the project state directory.
+1. Exercise the agent through the same surface a user would.
+2. Read the trajectory and name the dominant failure in one sentence.
+3. Change one lever.
+4. Rerun the same fixtures.
+5. Compare correctness, efficiency, and contract fit.
+6. Keep, revert, or narrow the change.
 
-**Reflect.** Score the trajectory, not impressions. Was the answer right? Did the model thrash (too many tools, fat evidence, grep loops)? Did it invent work the host should have prepared? Name the single dominant problem for this round in one sentence. Example: "Full-file dumps trigger grep loops."
+Do not replace a fixture mid-round. A moving input makes the comparison
+meaningless.
 
-**Change one lever.** Prefer the smallest change that addresses that problem:
+## Choose the smallest lever
 
-1. Host prep: seed what the model needs so it doesn't hunt
-2. Evidence shape: trim or reorder artifacts
-3. Instructions and skills: tighten the procedure
-4. Tool surface: remove or gate tools that invite wandering
-5. Framework changes: only when the agent can't express the fix
+Start with the earliest place that can remove the failure:
 
-**Remeasure.** Same fixtures. Diff tools, wall time, and quality side by side. Keep the change only if the target metric improves and the freeze line holds.
+1. **Host preparation:** fetch deterministic evidence before the turn.
+2. **Evidence shape:** trim, order, or label what the model receives.
+3. **Instructions or skills:** clarify the decision procedure.
+4. **Tool surface:** remove or gate tools that invite wandering.
+5. **Framework:** change it only when the agent cannot express the fix.
 
-## How do I lock a hillclimb improvement with an eval?
+Changing the host or evidence often improves both quality and latency
+because the model spends less time searching for ground truth.
+
+## Lock every kept win
+
+Each retained change needs an eval that would have failed before it. Use
+a tool-choice assertion, `maxToolCalls` bound, output-shape check, or
+quality score that captures the measured improvement.
+
+```bash
+agent-sdk eval --dir . --tag smoke
+```
 
-Every kept change needs an eval that would have failed before the change: a tool-choice gate, a `maxToolCalls` bound, or an output-shape check. Run `agent-sdk eval --dir . --json` between rounds. Never weaken an existing gate to pass the round.
+Never weaken an existing gate to make the round pass. The eval is the
+record that the improvement survived after the fixture and source
+change leave your working memory.
 
-Details live in [Keep improvements with regression evals](./evals.md#keep-improvements-with-regression-evals). The evals skill will author the case with you.
+## Run the loop with a coding agent
 
-## What habits help hillclimbing stay reliable?
+Ask the coding agent to read
+[`skills/hillclimb/SKILL.md`](../skills/hillclimb/SKILL.md), then give
+it the target, fixtures, success criterion, and freeze line. It will
+measure the current run, propose one change, remeasure, and ask before
+starting another round.
 
-- One problem per round. Don't bundle "trim evidence and rewrite instructions" unless you chose that on purpose.
-- Keep fixtures fixed until you deliberately need a harder case.
-- Separate host work from model tools when you blame latency. Moving deterministic prep onto the host is often the biggest win. In one PR reviewer, host-prepared evidence cut turns from about 8 minutes to about 1 minute.
-- Spot-check quality on at least one fixture against a known-good answer. Efficiency-only climbs quietly drop findings.
-- Treat `turn.failed` with `"turn interrupted"` as expected when a follow-up or stop preempted the turn.
-- Don't deploy, post to real surfaces, or weaken evals as part of a climb.
+Use saved or local surfaces during a climb. Do not deploy or post to a
+real external channel unless that side effect is part of an explicitly
+approved fixture.
 
 ## Related
 
-- [Evals](./evals.md)
-- [Building agents with agents](./building-with-agents.md)
-- [GitHub guide](./guides/github.md)
-- [Fix common agent problems](./troubleshooting.md)
+- [Evals](./evals.md): author the regression check for a kept win
+- [Evals reference](./reference/evals.md): trajectory assertions and
+  scores
+- [Building agents with agents](./building-with-agents.md): delegate
+  creation and verification to a coding agent
+- [Fix common agent problems](./troubleshooting.md): diagnose a run
+  that is broken instead of underperforming

package/docs/reference/cli.md

@@ -419,6 +419,8 @@ agent-sdk init ./grokbot --template grokbot-agents    # talk to Grok Bot agents
 agent-sdk init ./code-wiki --template code-wiki       # keep wiki pages current after merge
 agent-sdk init ./agents-md --template agents-md       # keep AGENTS.md current from last week's PRs and Slack
 agent-sdk init ./my-reviewer --template security-reviewer   # review PRs for security bugs
+agent-sdk init ./thermo-review --template thermo-review       # review PRs for bugs and breakage
+agent-sdk init ./thermo-quality --template thermo-quality-review # review PRs for code quality
 agent-sdk init ./security-help --template security-help   # answer security questions in Slack
 agent-sdk init ./my-triage --template triage-linear   # comment on Linear issues
 agent-sdk init ./my-triage --template triage-jira     # comment on Jira issues

package/docs/reference/evals.md

@@ -0,0 +1,298 @@
+---
+title: "Evals"
+description: "Reference for eval discovery, cases, assertions, judges, configuration, fixtures, reporters, and runner output."
+---
+
+# Evals reference
+
+This page is the complete authoring and runner contract for
+`@cursor/july/evals`. Start with the [Evals guide](../evals.md) for the
+workflow and first regression case.
+
+## Discovery and case IDs
+
+Eval files live under the project-root `evals/` directory and end in
+`.eval.ts` or `.eval.js`. The path under that directory becomes the
+eval ID:
+
+- `evals/readiness.eval.ts` becomes `readiness`.
+- `evals/builds/api.eval.ts` becomes `builds/api`.
+- `evals/builds/index.eval.ts` becomes `builds`.
+
+Export one `defineEval` with either `test` or `cases`, not both. A case
+ID appends one path segment to the file ID:
+
+```ts
+export default defineEval({
+  tags: ["smoke"],
+  cases: [
+    {
+      id: "checkout",
+      async test(t) {
+        await t.send("Review checkout PR 42.");
+        t.succeeded();
+      },
+    },
+    {
+      id: "search",
+      async test(t) {
+        await t.send("Review search PR 7.");
+        t.succeeded();
+      },
+    },
+  ],
+});
+```
+
+Those cases are `prs/checkout` and `prs/search` when the file is
+`evals/prs.eval.ts`. Case IDs must be unique single path segments.
+
+A file can also export an array of `defineEval` calls. The runner names
+them with zero-padded indexes such as `sql/0000`. Use named `cases` for
+handwritten scenarios and arrays for loaded datasets.
+
+`iterations` repeats one datapoint from 1 to 100 times. For
+`iterations: 3`, `weather/nyc` expands to `weather/nyc/1`,
+`weather/nyc/2`, and `weather/nyc/3`; selecting `weather/nyc` runs all
+three.
+
+## Configuration
+
+Every running suite needs `evals/evals.config.ts` with
+`maxConcurrency`:
+
+```ts
+import { defineEvalConfig } from "@cursor/july/evals";
+
+export default defineEvalConfig({
+  maxConcurrency: 20,
+  timeoutMs: 180_000,
+  judge: { model: "gpt-5.4-mini" },
+});
+```
+
+| Option | Contract |
+| --- | --- |
+| `maxConcurrency` | Required; concurrent authored datapoints, from 1 to 200 |
+| `timeoutMs` | Per-case timeout; case/file, CLI, then config precedence |
+| `judge` | Default model for `t.judge` |
+| `reporters` | Objects notified as cases and runs complete |
+| `maxPlaygroundRuns` | Number of server-side batches kept in playground history |
+
+A case can override `description`, `tags`, `timeoutMs`, `iterations`,
+`judge`, `reporters`, and `metadata`. Case metadata merges over
+file-level metadata; case reporters add to the file's reporters.
+
+## Drive turns with `t.send`
+
+`await t.send(message, options?)` runs one turn and waits until it
+finishes, fails, or parks for approval. Several sends in one test share
+the session.
+
+The returned turn exposes its assistant `message`, `sessionId`,
+`events`, ordered `toolCalls`, `ok`, and turn `index`. Assertions on the
+turn inspect only that turn; assertions on `t` inspect the whole run.
+Useful run values include `t.reply`, `t.events`, `t.turns`,
+`t.sessionId`, and the timeout `t.signal`.
+
+These options apply on the first send because they shape the session:
+
+| Option | Contract |
+| --- | --- |
+| `workspaceFiles` | Relative path-to-content map seeded into the session workspace |
+| `workspaceDir` | Absolute local harness working directory |
+| `cloud` | Cloud session options merged over the agent's static cloud config |
+
+Use `turn.expectOk()` when later test steps depend on that turn
+succeeding.
+
+## Trajectory assertions
+
+Assertions record failures and let the test continue, so one case
+reports every violated contract.
+
+| Assertion | Checks |
+| --- | --- |
+| `t.succeeded()` | Run finished without failure or an unanswered approval |
+| `t.parked()` | Run stopped on an unanswered approval |
+| `t.calledTool(name, matcher?)` | Matching tool request occurred |
+| `t.notCalledTool(name)` | No request for that tool occurred |
+| `t.loadedSkill(name)` | Agent opened a named skill |
+| `t.toolOrder(names)` | Tool requests appeared in relative order |
+| `t.usedNoTools()` | No tool was requested |
+| `t.maxToolCalls(max)` | Tool-call count stayed under the bound |
+| `t.noFailedActions()` | No tool result failed |
+| `t.calledSubagent(name, matcher?)` | Matching subagent call occurred |
+| `t.taggedArtifact(kind?, predicate?)` | Matching durable artifact was tagged |
+| `t.event(type, matcher?)` | Matching session event occurred |
+| `t.notEvent(type, matcher?)` | Matching session event did not occur |
+| `t.eventOrder(matchers)` | Event groups appeared in relative order |
+| `t.eventsSatisfy(label, predicate)` | Predicate accepted the event stream |
+| `t.check(value, expectation)` | Value met an expectation builder |
+| `t.score(name, value)` | Recorded a numeric score from 0 to 1 |
+
+`t.requireToolCall`, `t.requireInputRequest`, and `t.require` return the
+matched value and stop the remaining test body when no match exists.
+
+## Tool and event matchers
+
+Tool matchers can narrow by `input`, `output`, `status`, and `count`.
+Values accept literals, regular expressions, or predicates. Object
+literals partial-deep-match:
+
+```ts
+t.calledTool("inspect_pr", {
+  input: { verdict: "review" },
+  status: "completed",
+  count: 1,
+});
+```
+
+Tool status is `completed`, `failed`, `pending`, or `rejected`.
+Subagent matchers accept `output`, `status`, `count`, and `callId`.
+Event matchers accept `data` and `count`.
+
+## Expectation builders
+
+Import builders from `@cursor/july/evals`:
+
+| Builder | Default severity |
+| --- | --- |
+| `includes(string | RegExp)` | Hard gate |
+| `equals(value)` | Hard gate |
+| `matches(schema)` | Hard gate |
+| `satisfies(predicate, label)` | Hard gate |
+| `similarity(expected)` | Soft score |
+
+`matches` accepts Standard Schema implementations such as Zod.
+`normalizedSimilarity(value, expected)` returns the same 0-to-1 value
+used by `similarity`.
+
+## Severity and verdicts
+
+Every assertion returns a severity handle:
+
+```ts
+t.succeeded();
+t.calledTool("search").soft();
+t.check(t.reply, similarity("Expected answer")).atLeast(0.8);
+t.judge.closedQA("names the failing step").gate(1);
+```
+
+- `.gate(threshold?)` fails the case when it misses.
+- `.soft(threshold?)` records a value without failing.
+- `.atLeast(threshold)` records a soft bar; a miss produces a `scored`
+  verdict.
+
+Case verdicts are `passed`, `failed`, `scored`, or `skipped`. The CLI
+exits 1 for a failed gate. A scored result exits 0 unless `--strict` is
+set. Exit 2 means the selection matched no cases.
+
+## Judges
+
+Use a judge for meaning that deterministic checks cannot express:
+
+```ts
+t.judge.factuality(expected).atLeast(0.8);
+t.judge.summarizes(expected).atLeast(0.8);
+t.judge.closedQA("names the root cause").gate(1);
+t.judge.sql(expected).atLeast(0.9);
+```
+
+Each grades `t.reply` by default; pass `{ on }` to grade another value.
+Judge configuration can live in the project config, eval, case, or
+call, with the nearest value winning.
+
+`t.judge.model(prompt)` returns a raw model reply for custom grading.
+Treat agent output as untrusted input: wrap it with `fenceUntrusted` and
+include `EVAL_JUDGE_INJECTION_GUARD` in the judge prompt.
+
+## Datasets and fixtures
+
+Load committed JSON, JSONL, or YAML relative to the project root:
+
+```ts
+import { defineEval, includes } from "@cursor/july/evals";
+import { loadYaml } from "@cursor/july/evals/loaders";
+
+const rows = await loadYaml<
+  Array<{ task: string; prompt: string; expected: string }>
+>("evals/data/cases.yaml");
+
+export default rows.map(row =>
+  defineEval({
+    description: row.task,
+    async test(t) {
+      await t.send(row.prompt);
+      t.succeeded();
+      t.check(t.reply, includes(row.expected));
+    },
+  })
+);
+```
+
+Materialize API-backed evidence before running a large suite. Commit
+the exact payload, diff, or metadata revision and seed it with
+`workspaceFiles`.
+
+## Reporters and results
+
+Built-in reporters include `JUnit({ filePath, suiteName? })` and
+`Artifacts({ dir })`. Custom reporters can expose `onRunStart`,
+`onEvalComplete`, and `onRunComplete`. Reporter errors are logged and
+do not change the eval verdict.
+
+JSON output contains run totals plus one result per case. A result can
+include the case ID, verdict, assertions, session ID, inputs, tool
+calls, metrics, logs, duration, metadata, tags, final text, and error or
+skip details.
+
+## CLI selection and artifacts
+
+```bash
+agent-sdk eval --dir . --list
+agent-sdk eval --dir .
+agent-sdk eval --dir . builds/checkout
+agent-sdk eval --dir . --tag smoke
+agent-sdk eval --dir . --verbose
+```
+
+ID filters match an exact ID and its descendants. Repeated IDs use OR;
+repeated tags use OR. When both are present, a case must match both
+groups.
+
+The local runner starts an ephemeral server. Default artifacts go under
+`evals/<stamp>/` in the project state directory and include
+`summary.json`, `results.jsonl`, and `evals/<case-id>.json`.
+`--artifacts <dir>` chooses another destination; `--no-artifacts`
+disables them. This location is independent of `--state-root`.
+
+Model and judge turns resolve credentials in this order:
+`CURSOR_API_KEY`, `CURSOR_API_KEY_FILE`,
+`CURSOR_SERVICE_ACCOUNT_KEY`, then `agent-sdk login`. Listing cases
+needs no credential.
+
+## Playground and hosted runs
+
+The playground **Evals** view starts batches on its running server and
+keeps their sessions in the normal session list. `--url` targets a
+named running server; `--prod --slug <slug>` targets a hosted
+deployment.
+
+```bash
+agent-sdk eval --prod --slug vulnerability-scanner --tag smoke
+agent-sdk eval status <eval-id> --prod --slug vulnerability-scanner
+agent-sdk eval cancel <eval-id> --prod --slug vulnerability-scanner
+```
+
+Server-side batches use the target server's discovered evals and store
+results in its playground history. Local-only reporter and concurrency
+flags do not apply to `--url` or `--prod` runs.
+
+## Related
+
+- [Evals guide](../evals.md): author and run a regression workflow
+- [CLI reference](./cli.md#eval): every runner flag
+- [Sessions](./sessions.md): event vocabulary used by trajectory checks
+- [Artifacts](./artifacts.md): durable outputs asserted by
+  `taggedArtifact`