@ssheleg/agent-stack 0.4.0 → 0.5.0

package/CHANGELOG.md

@@ -4,6 +4,46 @@ All notable changes to this project are documented here.
 Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).
 
+## [0.5.0] — 2026-08-12
+
+### Added
+
+- **A second skill: `agent-evals`** — how you know the thing built by
+  `agent-orchestrator` actually behaves. There was no evaluation doctrine anywhere in
+  this family; `grep -ril "llm-as-judge\|eval dataset\|regression fixture\|trace id"`
+  across four plugins returned nothing.
+
+  An agent's behaviour is not in its source — the code says what it is allowed to do,
+  only a run says what it did — so the artifact under test is the execution record.
+  Three primitives (run, trace, thread) crossed with three granularities (single-step,
+  full-turn, multi-turn), each with its own fixture shape and its own precondition:
+  step assertions need a stable architecture or they die at the next refactor; turn
+  assertions cover trajectory **and** response **and** state change, because an agent
+  that says it saved the preference and did not passes two axes out of three; thread
+  scripts checkpoint after every turn and fail fast, or turn 3 derails and turns 4–10
+  assert nothing while still reporting a result.
+
+  Then the parts that decide whether any of it is trustworthy: the offline/online/ad-hoc
+  timing axis and why offline is necessary and not sufficient; pass-fail rubrics with
+  enumerated failure conditions instead of scalar scores that name no fix; code checks
+  before model judges; judging the trajectory, not just the answer; **calibrating a judge
+  against human labels before trusting it**, because an uncalibrated judge is an opinion
+  with a number attached; the classes of output no general judge can grade; a corpus
+  grown from production failures where every fixed failure stays a fixture permanently;
+  annotation queues with the two reviewer roles kept apart; and simulated users made
+  deliberately worse so offline results predict production.
+
+  Body 214 lines / ~2500 tokens; description 911/1024 with paired RU triggers.
+
+### Changed
+
+- **Both installers iterate over `skills/` instead of naming one skill.** `install.sh`
+  and `bin/agent-stack.js` each hardcoded `agent-orchestrator`, so a second skill would
+  have shipped in the package and reached nobody. Verified by running both against a
+  clean `HOME` and listing what arrived. The CI smoke test now asserts both skills.
+- README and both manifests describe two skills; the marketplace entry's description is
+  the manifest's, rather than a second one drifting beside it.
+
 ## [0.4.0] — 2026-08-12
 
 ### Added

package/README.md

@@ -9,7 +9,8 @@ Part of the [ssheleg skill family](https://github.com/ssheleg/sshlg-skills).
 
 ## What is in here
 
-One skill, `agent-orchestrator`, and two references it loads on demand.
+Two skills — `agent-orchestrator` for building one, `agent-evals` for proving it
+behaves — and three references the first loads on demand.
 
 **The orchestrator** (`SKILL.md`) — what the agent reads first:
 
@@ -29,6 +30,22 @@ One skill, `agent-orchestrator`, and two references it loads on demand.
 - context budget allocation by priority
 - self-learning feedback loops
 
+**The evals skill** (`agent-evals/SKILL.md`) — how you know any of it works. An
+agent's behaviour is not in its source, so the artifact under test is the
+execution record: three primitives (run, trace, thread) crossed with three
+granularities (single-step, full-turn, multi-turn), the offline/online/ad-hoc
+timing axis, pass-fail rubrics instead of scalar scores that name no fix, cheap
+code checks before model judges, judges calibrated against human labels before
+they are trusted, and a corpus grown from production failures rather than
+authored up front — where every fixed failure stays a fixture permanently.
+
+**`references/context-engineering.md`** — what the loop gives up when the window
+runs out: the five-rung compaction ladder and why to re-measure between rungs,
+the tool-pair boundary invariant, typed carryover blocks copied across the
+boundary rather than summarized, tool-output offload to a file, token estimation
+and the direction it errs, the compaction circuit breaker, sub-agent context
+isolation, and how to choose constants for your own window.
+
 **`references/patterns.md`** — the data models and algorithms underneath:
 message and result protocols, pipeline models, the SQL validation loop,
 context-window sizes and token estimation, learning-extraction heuristics,

package/bin/agent-stack.js

@@ -2,7 +2,7 @@
 /*
  * agent-stack installer CLI.
  *
- * Installs the agent-orchestrator skill into ~/.claude/skills/agent-orchestrator
+ * Installs every skill this plugin ships into ~/.claude/skills/<name>
  * (same layout as install.sh). Idempotent: an existing install is skipped unless
  * --force. Zero dependencies.
  *
@@ -21,7 +21,7 @@ function usage() {
   console.log(`agent-stack installer
 
 Usage:
-  npx @ssheleg/agent-stack [--force]   install the agent-orchestrator skill
+  npx @ssheleg/agent-stack [--force]   install every skill this plugin ships
                                        into ~/.claude (skip existing unless --force)
   npx @ssheleg/agent-stack --help
 
@@ -67,20 +67,34 @@ function main(argv) {
     return 2;
   }
 
-  const skillSrc = path.join(ROOT, 'plugins/agent-stack/skills/agent-orchestrator');
-  if (!fs.existsSync(skillSrc)) {
-    console.error(`error: skill sources missing at ${skillSrc} — corrupted package?`);
+  const skillRoot = path.join(ROOT, 'plugins/agent-stack/skills');
+  if (!fs.existsSync(skillRoot)) {
+    console.error(`error: skill sources missing at ${skillRoot} — corrupted package?`);
+    return 1;
+  }
+
+  // Iterate rather than name one skill: a skill added to the plugin must not
+  // require an installer change to reach anybody.
+  const names = fs
+    .readdirSync(skillRoot, { withFileTypes: true })
+    .filter((e) => e.isDirectory())
+    .map((e) => e.name)
+    .sort();
+  if (!names.length) {
+    console.error(`error: no skills found under ${skillRoot} — corrupted package?`);
     return 1;
   }
 
   const home = os.homedir();
-  installOne(
-    'agent-orchestrator skill',
-    skillSrc,
-    path.join(home, '.claude', 'skills', 'agent-orchestrator'),
-    true,
-    force
-  );
+  for (const name of names) {
+    installOne(
+      `${name} skill`,
+      path.join(skillRoot, name),
+      path.join(home, '.claude', 'skills', name),
+      true,
+      force
+    );
+  }
   return 0;
 }
 

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@ssheleg/agent-stack",
-  "version": "0.4.0",
+  "version": "0.5.0",
   "description": "Production patterns for AI agent orchestrators — tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access. This package is the installer CLI.",
   "bin": {
     "agent-stack": "bin/agent-stack.js"

package/plugins/agent-stack/.claude-plugin/plugin.json

@@ -1,8 +1,8 @@
 {
   "name": "agent-stack",
   "displayName": "Agent Stack",
-  "description": "Production patterns for AI agent orchestrators: tool-calling loops, multi-stage pipelines with checkpoints, LLM provider routing with fallback, four-layer memory with confidence decay — plus the wallet side of reselling LLM access.",
-  "version": "0.4.0",
+  "description": "Two skills: agent-orchestrator \u2014 tool-calling loops, multi-stage pipelines with checkpoints, provider routing with fallback, four-layer memory, context engineering, plus the wallet side of reselling LLM access; and agent-evals \u2014 run/trace/thread evals, judges, and fixtures grown from production.",
+  "version": "0.5.0",
   "author": {
     "name": "ssheleg",
     "url": "https://x.com/sshlg93"
@@ -19,6 +19,9 @@
     "openrouter",
     "llm-billing",
     "claude-code",
-    "cursor"
+    "cursor",
+    "evals",
+    "llm-judge",
+    "observability"
   ]
 }

package/plugins/agent-stack/skills/agent-evals/SKILL.md

@@ -0,0 +1,230 @@
+---
+name: agent-evals
+description: >-
+  Use when measuring whether an agent actually works — building an eval suite, judging a
+  trajectory rather than a final answer, turning production traces into regression
+  fixtures, calibrating an LLM judge against human labels, or gating a release on offline
+  evals. Covers the three observability primitives (run, trace, thread) crossed with three
+  eval granularities (single-step, full-turn, multi-turn), the offline/online/ad-hoc
+  timing axis, pass-fail rubrics over scalar scores, cheap code checks before model
+  judges, simulated users with adversarial personas, annotation queues, and what to
+  instrument so any of it is possible. Triggers - "agent eval", "eval suite", "LLM judge",
+  "regression fixture", "trajectory eval", "is the agent getting better", "эвалы агента",
+  "оценка агента", "LLM-судья", "регрессионный набор", "как проверить агента". Not for
+  unit tests of ordinary code, or for benchmarking a model.
+license: MIT
+---
+
+# Agent evals — proving the thing behaves
+
+An agent's behaviour does not exist in its source. The code says what it is *allowed* to
+do; only a run says what it *did*. So the artifact under test is the execution record,
+and the suite is grown from production rather than authored up front.
+
+Three claims follow, and they are what makes this different from testing ordinary code:
+
+- **You are testing reasoning, not code paths**, so one granularity is never enough.
+- **Every natural-language input is unique**, so the edge cases cannot be enumerated
+  offline. Production is not only where you catch what you missed — it is where you
+  discover what to test for.
+- **Traces become test cases.** The suite grows from what actually happened.
+
+---
+
+## 1. Three primitives
+
+| Primitive | Is | Carries |
+|---|---|---|
+| **Run** | one model call | the complete prompt — instructions, tools, context — and what came back |
+| **Trace** | one full execution | every run, every tool call with arguments and results, nested to show how steps relate |
+| **Thread** | many traces in one session | multi-turn context in order, **state evolution** (memory, files, artifacts), and elapsed time |
+
+The thread level is the one most systems lack, and it is where a whole class of defect
+lives: a bad memory write at turn 6 surfaces as a wrong answer at turn 11, and neither
+the run nor the trace view can localise it.
+
+**Precondition for all of this: traces are durable and queryable.** A live event stream
+renders a progress bar and evaporates. If you cannot fetch last week's low-scoring runs
+by id, nothing below is available to you — see §7.
+
+---
+
+## 2. Three granularities
+
+Each primitive gets its own kind of assertion.
+
+### Single-step — validates a run
+
+Fixture is a serialized run: prompt, tool schemas, context. Assert the decision at that
+point — tool name, argument shape.
+
+> "Schedule a meeting with Harrison tomorrow morning", with `find_meeting_times`,
+> `schedule_meeting` and `send_email` available, must call `find_meeting_times` first.
+
+Cheap, deterministic, CI-blocking. **Precondition: a stable agent architecture.** These
+break on a graph refactor, and a suite that fails on every refactor gets deleted.
+
+### Full-turn — validates a trace
+
+Assert on three axes at once, with three different mechanisms:
+
+| Axis | Assert | With |
+|---|---|---|
+| Trajectory | tool-call sequence — `read_file` → `edit_file` → `run_tests` | set/subset/order matchers |
+| Final response | quality, tone, policy compliance | rubric or judge |
+| **State change** | the memory row exists, the file was written, the artifact is there | direct inspection of the side effect |
+
+The third axis is the one people forget. **Assert on side effects, not only on prose** —
+an agent that says it saved the preference and did not is a pass on two axes out of three.
+
+Easiest inputs to generate, hardest outputs to validate automatically.
+
+### Multi-turn — validates a thread
+
+A scripted turn sequence with a **checkpoint after every turn and fail-fast on
+deviation**. Without that, turn 3 goes off the rails and turns 4–10 assert nothing while
+still reporting a result.
+
+> Turn 1: "I prefer Python over JavaScript." Turn 3's output must still be Python.
+
+Hardest to implement well. Start here only for behaviour that is genuinely about memory
+across turns.
+
+**Production suites combine all three.** One vendor reports about half of theirs sitting
+at single-step — recorded as their observation, not as a target to hit.
+
+---
+
+## 3. Offline, online, ad-hoc
+
+| When | Reference | Blocks a release? | Answers |
+|---|---|---|---|
+| **Offline** | a dataset, ground truth optional | **yes** — this is the gate | did my change break what used to work |
+| **Online** | none — **definitionally reference-free** | no | is production drifting |
+| **Ad-hoc** | none | no | what is actually happening out there |
+
+**Offline is necessary and not sufficient.** A green suite proves you did not regress the
+cases you already know about. It cannot prove the agent handles what nobody thought of,
+because that input is not in the dataset — which is what the online tier is for.
+
+Online evaluators fire on trace ingestion and check what needs no expected answer:
+trajectory anomalies, step-count and latency trends, judge scores, error rates. Route
+them over all traces, a sample, or a filtered subset; the sampling rate is a cost
+decision, not a correctness one.
+
+Ad-hoc is exploratory analysis over stored traces — clustering to surface failure modes
+nobody predefined. A dashboard tracks metrics you chose in advance; this finds the ones
+you did not.
+
+---
+
+## 4. Rubrics beat scores
+
+Generic metrics — helpfulness, naturalness, completeness — produce numbers and no
+decision. A 3.4 out of 5 on "helpfulness" tells you nothing about what to change.
+
+**Write narrow, behaviour-specific pass/fail rubrics, and write them with the people who
+own the behaviour.** Each failure must point at one thing: a prompt, a tool description,
+a workflow step, a missing capability.
+
+A rubric that works, in full:
+
+> **Escalation.** On a request for a human: push back once, escalate on the repeat.
+> **Fails if** it escalates immediately · refuses after the second request · escalates
+> before providing information it already had · continues several turns past the point it
+> is clearly not helping.
+
+Note the shape: one behaviour, an explicit pass condition, and an enumerated failure list.
+That is what makes a judge reproducible and a disagreement resolvable.
+
+---
+
+## 5. Judges
+
+**Cheap checks first.** Schema validation, exact match, format conformity, business-rule
+assertions, tool-call correctness — all deterministic, all faster and cheaper than a model
+call. Send to a judge only what cannot be decided by code.
+
+**Judge the trajectory, not just the answer.** Right tools, right order, right arguments.
+An agent that reaches a correct answer through three wrong tool calls is a latent outage.
+
+**Calibrate the judge before trusting it.** Collect human labels on the same traces,
+measure agreement, iterate the judge prompt until agreement is high — *then* let it score
+unattended. An uncalibrated judge is an opinion with a number attached, and shipping on it
+is exactly the failure of grading instead of measuring.
+
+**Some things a judge cannot do.** Plausible-but-wrong domain output — an invented legal
+citation, a subtly wrong SQL join — reads as correct to a general judge. Route those to a
+domain expert and accept that this tier stays human.
+
+---
+
+## 6. The corpus grows from production
+
+Never author the suite up front. Every production failure and every thumbs-down becomes a
+fixture:
+
+1. Capture the state at the failure point.
+2. Minimise it to the smallest input that still reproduces.
+3. Add it to the tier that isolates it — step, turn or thread.
+4. **It stays in the suite permanently.** A fixed bug that silently returns is the defect
+   this rule exists to prevent.
+
+Two dataset shapes come out of review:
+
+- **Ground truth** — the reviewer writes the correct output; the suite asserts equality.
+- **Criteria-based** — for open-ended work, the reviewer labels dimensions (relevance,
+  completeness, tone) instead of an exact answer.
+
+**The annotation queue** is what feeds both: filters route a subset of traces to humans —
+low automated score, thumbs-down, a feature area, a cluster. Two reviewer roles, and
+mixing them wastes both: generalists judge surface quality, domain experts judge
+correctness only they can see.
+
+**Simulated users, if you generate inputs.** A model playing a customer is articulate,
+patient and cooperative, and inflates every pass rate. Fine-tune it on real user
+transcripts and add adversarial personas — the refund-seeker, the AI-sceptic, the one who
+wants a human immediately. Making the simulated user worse makes the offline result
+predictive.
+
+---
+
+## 7. What to instrument first
+
+None of the above runs without these, and they are the part people skip:
+
+- **A durable trace store, queryable by id, filterable by score and time.** The live
+  stream is a view over it, never the record itself.
+- **Scores as first-class records bound to a run**: `run_id`, key, value, and a `source`
+  of `human` | `llm_judge` | `code_check`. A score with no source cannot be calibrated,
+  audited, or trusted differently from its neighbours.
+- **Whole prompts, not just messages** — instructions, tool schemas and context as they
+  were sent. A fixture cannot be replayed from a summary.
+- **State snapshots at turn boundaries**, so a thread test can assert what carried.
+
+---
+
+## Checklist
+
+- [ ] Traces durable and queryable by id, not only streamed
+- [ ] Scores bound to runs, each with a source
+- [ ] Single-step fixtures for the decisions that must not drift
+- [ ] Full-turn assertions on trajectory **and** final response **and** state change
+- [ ] Multi-turn scripts with a checkpoint after every turn, failing fast
+- [ ] Offline suite as the release gate; online evaluators reference-free and non-blocking
+- [ ] Pass/fail rubrics with enumerated failure conditions, written with behaviour owners
+- [ ] Code checks before model judges
+- [ ] Judge calibrated against human labels before it is trusted
+- [ ] Domain-expert review for output a general judge cannot grade
+- [ ] Every production failure minimised into a permanent fixture
+- [ ] Annotation queue with filters, and the two reviewer roles kept separate
+- [ ] Simulated users trained on real transcripts, with adversarial personas
+
+---
+
+## Related
+
+Building the agent this measures is the **`agent-orchestrator`** skill, shipped in the
+same plugin — named rather than linked, because a packager may ship either directory
+alone. Its context-engineering reference covers the context-pressure behaviour that
+trajectory assertions most often catch drifting.