@smthrs/evals 0.0.0-stage → 1.0.0-rc.4

package/docs/concepts/determinism.md

@@ -0,0 +1,64 @@
+---
+title: "Determinism and canonical artifacts"
+description: "Caller-supplied run identity, deterministic sampling, and byte-stable serialization."
+sidebar:
+  order: 3
+---
+
+The runner guarantees determinism only for what it controls. Given identical
+executor results, scorer results, metadata, and run options, two runs produce
+byte-identical observations, and two identical comparisons produce
+byte-identical reports. Ordering, sampling, job identity, and serialization are
+stable by construction, through four decisions.
+
+The runner cannot make your callbacks deterministic. `Runner.run` invokes every
+executor and scorer effect afresh on each run, so the caller owns every input
+those effects read: clocks, model responses, randomness, external state, and
+mutation inside a callback. A scorer that counts its own calls grades the same
+suite `0` on the first run and `1` on the second, with identical options.
+
+## Identity and time come from the caller
+
+`Runner.run` takes `runId` and `at` as options. `runId` identifies the
+`RunResult` and joins every score job's identity; `at` stamps every
+observation. The runner itself reads no clock and generates no identifier.
+Pin `at` to a fixed instant when you want a suite to be comparable across days:
+observation timestamps are report material, and pinning them keeps two runs
+with stable callbacks byte-identical.
+
+The format is strict because reproducibility is strict: `at` must be a
+canonical UTC timestamp with millisecond precision that parses and re-renders
+to itself, and anything else fails with `invalid_run_options`.
+
+## Sampling is decided, not rolled
+
+A binding's sampling policy selects which executions a scorer grades, and the
+selection is a pure function of the policy, the step key, and the scorer key.
+The same run over the same inputs samples the same executions every time, so a
+sampled baseline stays comparable with the next sampled run. For the policy
+types, see the [scorers API](/api/scorers).
+
+## Job identity is injective
+
+Every score job carries an identity built from the suite name, run identity,
+sample identity, case, step key, scorer key, and the job's index, encoded as a
+JSON array. Encoding a tuple rather than joining strings on a delimiter is
+what makes the identity injective: two distinct jobs can never produce one
+identity, so a correlated batch runner can always attribute its results.
+
+## Serialization is total and canonical
+
+`Baseline.write` and `Report.json` share one encoder. It sorts object keys by
+code unit, drops `undefined` members, and normalizes negative zero. Everything
+JSON cannot express becomes a bracketed marker that names what was there:
+`[circular]`, `[depth exceeded]`, `[NaN]`, `[Infinity]`, `[-Infinity]`,
+`[bigint n]`, `[function]`, `[symbol]`, and `[unreadable: …]` when a foreign
+operation throws. An `Error` becomes an object holding its own fields plus
+`name` and `message`. A `Date` becomes its ISO string, a `Set` becomes an
+array, and a `Map` becomes an array of key/value pairs. Embedded strings are
+capped at 8192 code units and nesting at 64 levels.
+
+The encoder is total: a report of a broken run is still a report, never a
+thrown `RangeError` out of a function typed `string`. And it redacts nothing:
+a suite whose cases carry secrets must not print the report where the log is
+readable.

package/docs/concepts/fixed-suites.md

@@ -0,0 +1,71 @@
+---
+title: "Fixed suites"
+description: "Why a suite is validated once, snapshotted, and frozen, and how a binding finds its target."
+sidebar:
+  order: 1
+---
+
+A fixed suite is the input half of an evaluation: named cases, the scorer
+bindings that grade them, and the concurrency the runner is allowed. "Fixed"
+is the load-bearing property. A committed baseline says "this suite scored
+these numbers", and that sentence is only meaningful when the suite that
+produced the baseline is the suite being run. So a suite is validated once, at
+construction, and then never changes.
+
+## The snapshot
+
+`Suite.make` reads every option, case field, and binding field once when the
+effect runs. It validates the data before copying it with `structuredClone`,
+then deeply freezes the suite, cases, binding records, and copied data.
+Mutating the caller's data or attempting to edit the returned snapshot cannot
+change the validated suite.
+
+Case input, expected values, and binding sampling, ground truth, and context
+admit only these values:
+
+- Plain objects with `Object.prototype` or a null prototype, and ordinary arrays.
+- Enumerable, string-keyed data properties. Accessors, symbol keys, and
+  non-enumerable properties are rejected, except for the array `length` property.
+- Cloneable primitives: null, undefined, booleans, strings, numbers, and bigints.
+  Numbers include NaN, infinities, and negative zero.
+
+Cycles and shared references within a data field are preserved. Null-prototype
+records become ordinary objects when cloned. Functions, symbols, class
+instances, and other non-plain objects fail with `invalid_suite` at the
+nested field path. This includes Map, Set, Date, RegExp, buffers, and typed
+arrays: freezing these objects does not protect all their mutable state.
+Nested getters are rejected without invocation. Getters on the outer options,
+case, and binding fields are read once before data validation.
+
+Each executor invocation receives its own mutable case copy. Each scorer
+request receives independent copies of the original case input, expected
+value, and binding data. Executor mutations do not change scorer ground
+truth or input, and scorer mutations do not change another request or a later
+run. Sequential and concurrent runs reuse the same protected snapshot.
+Execution output remains the executor's value and is outside the fixed-suite
+data contract.
+
+## Bindings and identity
+
+A binding says "this scorer grades that flow", and the match is by reference
+identity: a run grades an execution with a binding only when the execution's
+`target` is the very flow value the binding's `appliesTo` holds. A structurally
+equal copy of the flow is graded by nothing.
+
+Identity is what makes a binding durable. Two declarations of a flow with the
+same name are different flows, and a baseline recorded against one says
+nothing about the other. Reference identity makes that distinction exact
+instead of approximate.
+
+The scorer has the same split between identity and label. `Observation.scorer`
+is the scorer key, a digest of the scorer's own `{ id, version, config }`
+declaration, and it is what a baseline matches on: change the declaration and
+the old scores no longer claim to describe the new scorer.
+`Observation.scorerName` is the human name from the same declaration, carried
+alongside so a report can be read without grepping for the digest. A Markdown
+report prints `name (first 8 of the key)`, and the bare key when the
+declaration carries no name. Report cells, gate summaries, and runner
+diagnostics share that one rule, so a report never names one scorer two ways.
+
+For the scorer, binding, and sampling types themselves, see the
+[scorers API](/api/scorers).

package/docs/concepts/step-keys.md

@@ -0,0 +1,60 @@
+---
+title: "Step keys and comparison"
+description: "The rule that separates a regression from nondeterminism, and how observations pair with baseline records."
+sidebar:
+  order: 2
+---
+
+One rule decides what a comparison reports, and it is the mental model the
+whole package is built around: the step key identifies the work a case
+produced.
+
+- A score that dropped at a changed step key is a regression: the target
+  produced different work, and the different work graded worse.
+- A score that moved at an unchanged step key is nondeterminism: the same work
+  graded differently twice.
+
+Both are results, and a gate reads both as red. They are different reds: a
+regression points at the target, nondeterminism points at a target or a scorer
+that does not repeat itself.
+
+The step key is why the executor contract asks for a key that is stated, not
+derived. If the key were a hash of the output, every output change would read
+as new work and nondeterminism would be unreportable. If the key never
+changed, different work would read as the same work. The executor names the
+step; the comparison believes it.
+
+## Pairing
+
+A scorer may fire several times against one case, so comparison cannot zip two
+arrays by position. `Regression.compare` groups baseline records and run
+observations by `(case, scorer)`, then pairs them:
+
+1. Records and observations that share a step key pair first, lowest score to
+   lowest score, so a repeated scorer does not report a move that never
+   happened.
+2. Whatever is left over pairs in stable `(stepKey, score)` order.
+3. An unpaired record on either side is a missing observation, never a silent
+   drop. `side: "run"` names a baseline record the run never reproduced;
+   `side: "baseline"` names a score no baseline record accounts for.
+
+A score that improved at a changed step key is not reported: the comparison
+exists to catch deterioration, and an improvement at a changed key is new work
+that graded better.
+
+## Tolerances
+
+A move is reported only when it exceeds both tolerances: the absolute
+difference and the difference relative to the baseline score. Either one alone
+is enough to silence a move, and both default to 0, which reports every move.
+The relative check divides by the baseline score guarded away from zero, so a
+baseline score of exactly 0 cannot make a later move look infinitely large or
+infinitely small.
+
+## What comparison preserves
+
+Inconclusive observations decide nothing, and the comparison carries them
+through untouched rather than dropping them: a gate has to see them, because a
+scorer that could not answer is an environment fault, not a measurement.
+Failed cases never reach the comparison at all. They live on the run's case
+results, and the gate reads them from there.

package/docs/guides/author-a-suite.md

@@ -0,0 +1,115 @@
+---
+title: "Author a suite"
+description: "Declare fixed cases and scorer bindings, or load them from a JSON Lines fixture."
+sidebar:
+  order: 1
+---
+
+A suite is the fixed input half of an evaluation: named cases, the scorer
+bindings that grade them, and the concurrency the runner is allowed. Build one
+with `Suite.make`, or load cases from a JSON Lines fixture with
+`Suite.fromJsonLines`.
+
+## Declare cases
+
+A case is a name, an `input` handed to the executor, and an optional
+`expected` offered to a bound scorer as ground truth:
+
+```ts
+const cases = [
+  { name: "adds numbers", input: { left: 1, right: 2 }, expected: 3 },
+  { name: "multiplies numbers", input: { left: 2, right: 4 }, expected: 8 }
+]
+```
+
+When a case declares `expected` and the binding also declares `groundTruth`,
+the case's value wins. A declared `null`, `false`, `0` or `""` is a value and
+wins too; only an absent `expected` defers to the binding. When neither
+declares one, the scorer is called without ground truth.
+
+Case names must be non-empty, unique within the suite, and free of control
+characters. A control character in a name corrupts Markdown reports and CI log
+lines, so it is rejected where the name enters the system.
+
+## Bind scorers
+
+A binding attaches a scorer to the flow it grades:
+
+```ts
+import { Binding } from "@smthrs/scorers"
+
+const bindings = [Binding.make({ scorer: polite, appliesTo: greet })]
+```
+
+`appliesTo` is matched against an execution's `target` by reference identity,
+so the binding grades only the exact flow value it was declared against; a
+structurally equal copy is graded by nothing. `Binding.make` also accepts
+`groundTruth`, `context`, and a `sampling` policy, and defaults to scoring
+every target step. For the binding and sampling types, see the
+[scorers API](/api/scorers).
+
+`Suite.make` copies each binding's data fields with `structuredClone` but
+keeps `scorer` and `appliesTo` by reference: they are executable identities,
+and copying them would break the identity match.
+
+## Validate at construction
+
+`Suite.make` fails with `invalid_suite` when the declaration is wrong, and
+each failure locates the offending value in `path`:
+
+| Failure                                                                                  | Path                                                                                                            |
+| ---------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------- |
+| Suite name is empty or holds a control character                                         | `name`                                                                                                          |
+| No cases, or more than `limits.cases` cases                                              | `cases`                                                                                                         |
+| Concurrency is not a safe integer in [1, `limits.concurrency`]                           | `concurrency`                                                                                                   |
+| Case name is empty, holds a control character, or duplicates another case                | `cases[N].name`                                                                                                 |
+| Case or binding data is not structured-cloneable, such as a function or a class instance | `cases[N].input`, `cases[N].expected`, `bindings[N].sampling`, `bindings[N].groundTruth`, `bindings[N].context` |
+
+The clone check doubles as the suite's immutability guarantee: the copy is
+what the suite keeps, so mutating the caller's objects afterwards leaves the
+validated suite unchanged.
+
+## Load cases from JSON Lines
+
+For fixtures, `Suite.fromJsonLines` decodes one `{ name, input, expected? }`
+object per line and passes the result through the same validation:
+
+```json
+{"name":"adds numbers","input":{"left":1,"right":2},"expected":3}
+{"name":"multiplies numbers","input":{"left":2,"right":4},"expected":8}
+```
+
+```ts
+import { Suite } from "@smthrs/evals"
+import { Effect } from "effect"
+import { readFile } from "node:fs/promises"
+
+const program = Effect.gen(function*() {
+  const text = yield* Effect.promise(() => readFile("suite.jsonl", "utf8"))
+  return yield* Suite.fromJsonLines(text, { name: "arithmetic", concurrency: 4, bindings })
+})
+```
+
+Blank lines are skipped, a leading byte-order mark is stripped, and LF and
+CRLF both terminate a line. A malformed line fails with `invalid_suite`
+carrying the 1-based line number in the message and the path: `Invalid JSON on
+line 2` at `line[2]`, or `Invalid suite case on line 1` at `line[1]`.
+
+## Declared limits
+
+`Suite.limits` declares the ceilings a suite is validated against, so a
+mistake fails at construction with a sentence instead of exhausting memory in
+a runner:
+
+| Limit           | Value   | Bounds                                          |
+| --------------- | ------- | ----------------------------------------------- |
+| `concurrency`   | 1024    | The fibers a run may hold open                  |
+| `cases`         | 10000   | The cases in one suite                          |
+| `fixtureLength` | 8388608 | The UTF-16 code units in one JSON Lines fixture |
+
+Each ceiling is inclusive. A fixture larger than `fixtureLength` is rejected
+before any of it is parsed. After 10000 cases, the next non-blank line fails
+with `invalid_suite` at `cases`, before parsing or decoding that line or any
+later lines. The message reports `got 10001`, the first excess case.
+
+Next: [run the suite](./run-a-suite.md).

package/docs/guides/gate-a-run-in-ci.md

@@ -0,0 +1,129 @@
+---
+title: "Gate a run in CI"
+description: "Compare a run to its baseline, render the report, and turn the verdict into an exit code."
+sidebar:
+  order: 4
+---
+
+The second half of the pipeline runs in CI: load the committed baseline,
+compare the run against it, render the comparison for the log, and grade it
+into an exit code.
+
+## Compare with tolerances
+
+`Regression.compare` pairs baseline records with the run's score observations
+and reports what moved:
+
+```ts
+Effect.gen(function*() {
+  const comparison = yield* Regression.compare(baseline, run, {
+    absolute: 0.05,
+    relative: 0.1
+  })
+})
+```
+
+- A score that dropped at a changed step key is a regression.
+- A score that moved at an unchanged step key is nondeterminism.
+- An observation present on only one side is a missing observation.
+- Inconclusive observations are carried through untouched.
+
+A move is reported only when it exceeds both tolerances, so either one alone
+silences it. Both default to 0, which reports every move. A tolerance that is
+not finite and non-negative fails with `invalid_tolerance`. For the pairing
+and step-key rules, see [Step keys and comparison](../concepts/step-keys.md).
+
+## Render the report
+
+`Report.markdown` is what an operator reads in the CI log. Its summary counts
+regressions, nondeterminism, missing observations, inconclusive observations,
+and failed cases, and every non-zero count gets a section that names its rows.
+`Report.json` is the machine-readable artifact: stable, sorted-key JSON with
+embedded strings capped and everything JSON cannot express replaced by a named
+marker, so two identical runs produce byte-identical reports.
+
+Nothing redacts the output embedded in a report. A suite whose cases carry
+secrets must not print the report where the log is readable.
+
+## Set thresholds
+
+`Gate.check` evaluates score thresholds through the shared gate arithmetic in
+[@smthrs/scorers/ScoreGate](/api/scorers):
+
+```ts
+Effect.gen(function*() {
+  const verdict = yield* Gate.check(comparison, {
+    mean: 0.9,
+    min: 0.5,
+    perCase: { "adds numbers": 0.8 }
+  })
+})
+```
+
+- `mean` gates the arithmetic mean of every score observation.
+- `min` gates the lowest score observation.
+- `perCase` gates each named case's lowest score.
+
+The threshold gates always run: an unobserved case cannot excuse the cases
+that were observed. With no options the gate still runs, as `mean(0)`, so a
+run with no score at all is undecidable rather than a pass. Regressions and
+nondeterminism are findings and force a `Failed` verdict. Failed cases and
+missing observations are environment faults and travel beside the verdict as
+unresolved.
+
+A threshold outside [0, 1] is gate misuse and fails in the error channel with
+a `ScoreGateError`, code `invalid_threshold`. Catch it and report it rather
+than crashing the CI job:
+
+```ts
+Effect.gen(function*() {
+  const graded = yield* Gate.check(comparison, { mean: 1 }).pipe(
+    Effect.map(Gate.ciGrade),
+    Effect.catch((error) =>
+      Effect.succeed({
+        exitCode: 1 as const,
+        summary: `${error.code}: threshold ${error.threshold}, actual ${error.actual}`
+      })
+    )
+  )
+})
+```
+
+## Grade to an exit code
+
+`Gate.ciGrade` maps the verdict to the shared CI convention:
+
+| Exit code | Meaning                                                                         |
+| --------- | ------------------------------------------------------------------------------- |
+| 0         | Passed: every gate met, nothing unresolved                                      |
+| 1         | Failed: a threshold breach, a regression, or nondeterminism                     |
+| 5         | Inconclusive: the gate could not decide, or passed with unresolved observations |
+
+Exit code 5 names an unusable harness, not a result: repair the scorer, the
+executor, or the wiring instead of reading the red. A pass that carries
+unresolved observations also exits 5, because the gates it met were met over
+fewer observations than the suite declared.
+
+## The complete gate
+
+Assemble the whole thing as one script that CI runs: build the suite, run it,
+load the committed baseline, compare, print the Markdown report, grade, and set
+`process.exitCode` from the grade. [Quickstart](../quickstart.md) has that
+script in full.
+
+Two flags earn their place on it:
+
+- `--update` re-records the baseline from this run, for the times a score moved
+  for a reason you can name. Keep it off the CI path.
+- `--json` prints `Report.json` instead of the Markdown summary, so a drifting
+  run leaves a machine-readable artifact to diff.
+
+Then call the script from your CI job and let the exit code decide the step:
+
+```yaml
+- name: Evaluate
+  run: node quickstart.ts
+```
+
+The gate's exit code is the whole contract: 0 and the job goes green, 1 and it
+fails on a real red, 5 and it fails on a harness you need to repair.

package/docs/guides/maintain-a-baseline.md

@@ -0,0 +1,91 @@
+---
+title: "Maintain a baseline"
+description: "Record what a run scored, commit the canonical artifact, and load it back for comparison."
+sidebar:
+  order: 3
+---
+
+A baseline is the record of what a suite used to score, committed beside the
+suite it belongs to. Record one with `Baseline.fromRun` and `Baseline.write`;
+load it back with `Baseline.load`.
+
+## Record a baseline
+
+`Baseline.fromRun` keeps a run's successful score observations and drops
+inconclusive ones: a baseline records what was measured, and an inconclusive
+observation measured nothing.
+
+```ts
+Effect.gen(function*() {
+  const failures = run.cases.filter((result) => result.error !== undefined)
+  if (failures.length > 0) {
+    return yield* Effect.fail(
+      new EvalError.EvalError({
+        code: "executor",
+        message: `Refusing to record a baseline: ${failures.length} case(s) did not finish`
+      })
+    )
+  }
+  const baseline = yield* Baseline.fromRun(run)
+  yield* Effect.promise(() => writeFile("baseline.json", Baseline.write(baseline)))
+})
+```
+
+`Baseline.write` serializes canonical JSON: object keys sorted recursively,
+records ordered by an injective encoding of `(suite, case, scorer, stepKey)`,
+and a trailing newline. Two runs over the same inputs produce byte-identical
+baseline files, so the committed artifact diffs cleanly.
+
+Do not record over a broken run. The guard above refuses when any case failed,
+because a baseline recorded over a failed case ratifies the absence of a
+measurement. Keep it on every update path.
+
+Re-record the baseline only when a score moved for a reason you can name. The
+baseline is the record of what the target used to do; overwriting it to quiet
+a red gate erases the regression the gate caught.
+
+## The artifact
+
+A baseline is versioned JSON owned by one suite. This is the shape
+`Baseline.write` emits, with every object key sorted:
+
+```json
+{
+  "records": [
+    {
+      "case": "adds numbers",
+      "score": 1,
+      "scorer": "exact",
+      "stepKey": "step-a",
+      "suite": "arithmetic"
+    }
+  ],
+  "suite": "arithmetic",
+  "version": 1
+}
+```
+
+- `version` is `1`, the current artifact version (`Baseline.version`).
+- `suite` names the owning suite even when `records` is empty.
+- Each record carries `suite`, `case`, `scorer`, `stepKey`, and `score`, plus
+  an optional `scorerName`. The `scorer` field is the scorer key, a digest of
+  the scorer's declaration, and it is what a comparison matches on.
+
+## Load and validate
+
+`Baseline.load` parses and validates committed JSON. `Baseline.make` validates
+an in-memory value. Both rebuild every record from the validated fields, so
+nothing a caller happened to attach to an object travels into a committed
+artifact, and the returned array is frozen. Validation fails with
+`invalid_baseline`, carrying the record index and field name in `path`, for a
+wrong version, a non-array `records`, a record that is not an object, a
+top-level `suite` or identity field that holds a control character, a record
+identity field that is not a string, a file longer than
+`Baseline.limits.length`, or a score that is not finite in [0, 1]. A negative zero score is normalized to 0.
+
+Ownership is checked at comparison time: `Regression.compare` refuses a
+baseline whose artifact or any record names a suite other than the run's,
+failing with `invalid_baseline` rather than reporting a clean pass for the
+wrong file.
+
+Next: [gate the run in CI](./gate-a-run-in-ci.md).

package/docs/guides/run-a-suite.md

@@ -0,0 +1,150 @@
+---
+title: "Run a suite"
+description: "Provide a case executor, run the suite deterministically, and choose how scoring happens."
+sidebar:
+  order: 2
+---
+
+`Runner.run` executes every case through the injected `CaseExecutor` service
+and grades each execution with the bindings whose `appliesTo` is the flow the
+execution reports as its target.
+
+## Implement the executor
+
+The executor is one callback from a case to an `Execution`:
+
+```ts
+import { CaseExecutor } from "@smthrs/evals"
+import { Effect } from "effect"
+
+const executor = CaseExecutor.make((suiteCase) =>
+  Effect.succeed({
+    output: `Hello, ${(suiteCase.input as { readonly name: string }).name}`,
+    stepKey: suiteCase.name,
+    latencyMs: 0,
+    target: greet
+  })
+)
+```
+
+`CaseExecutor.make` accepts the bare callback, or an object naming the
+callback `run`. An object without one throws a `TypeError`:
+a wiring mistake fails loudly instead of degrading to an executor that fails
+every case.
+
+Four rules make an execution usable:
+
+- `target` must be the declared flow value itself. Bindings match it by
+  reference identity, so a copy of the flow is graded by nothing.
+- `stepKey` identifies the work the case produced. Fix it per case and state
+  it; do not derive it from the run. The comparison reads a changed key as
+  different work and a moved score under an unchanged key as nondeterminism.
+  For the full rule, see [Step keys and comparison](../concepts/step-keys.md).
+- `latencyMs` is forwarded to the scorer.
+- `output` is handed to the scorer and embedded raw in the JSON report. Do not
+  let it carry secrets you would not print.
+
+## Fail a case
+
+Return a typed failure and the run records it on the case and moves on:
+
+```ts
+import { EvalError } from "@smthrs/evals"
+
+const executor = CaseExecutor.make((suiteCase) =>
+  suiteCase.name === "known-bad"
+    ? Effect.fail(
+      new EvalError.EvalError({ code: "executor", message: "the target crashed", path: "target" })
+    )
+    : Effect.succeed({ output: 1, stepKey: suiteCase.name, latencyMs: 0, target: greet })
+)
+```
+
+A failed case keeps its typed error and produces no observations; the rest of
+the suite still runs. The error keeps the executor's own `path` when it named
+one and is located at `cases['<name>']` otherwise, and its message is prefixed
+`Target failed for case '<name>':`. An untyped failure is wrapped with the
+`executor` code. Interruption propagates instead: an interrupted target
+interrupts the run.
+
+## Pass run identity
+
+`Runner.run` takes `runId` and `at` from the caller. `runId` identifies the
+`RunResult` and joins every score job's identity; `at` stamps every
+observation. Two runs produce identical observations when their executor and
+scorer results are identical too, since the runner invokes those effects afresh
+each time:
+
+```ts
+Effect.gen(function*() {
+  const run = yield* Runner.run(suite, {
+    runId: "nightly-2026-01-01",
+    at: "2026-01-01T00:00:00.000Z"
+  })
+})
+```
+
+- `runId` must be non-empty.
+- `at` must be a canonical UTC timestamp with millisecond precision, such as
+  `2026-01-01T00:00:00.000Z`. The value is parsed and re-rendered, and one
+  that does not round-trip fails with `invalid_run_options`.
+  `new Date().toISOString()` satisfies the format.
+- `sampleId` is optional and defaults to `"default"`. It joins the job
+  identity, so two samples of one run never share a job identity.
+
+## Choose how scoring happens
+
+A run scores through the first of these that exists:
+
+1. `options.scorer`, a batch runner passed at the call site.
+2. The `Runner` service, when a layer provides one.
+3. `Runner.makeInline()`, the in-process default.
+
+The inline runner executes each job's scorer in the current process and
+validates the result with the scorers package's own `Scorer.validate`. A
+scorer that fails becomes an inconclusive observation naming its cause rather
+than failing the run. Provide `Runner.layerInline` to state the in-process
+choice explicitly. Provide `Runner.layerNoop` to state that a suite must not
+score: every bound score then becomes an inconclusive observation, which a
+gate grades as an undecidable run rather than a red.
+
+## Implement a batch runner
+
+An adapter implements `Runner.ScoreBatchRunner`. Each `ScoreJob` carries an
+`identity`, an `observation` seed with `targetStepKey` and `scorerKey`, the
+scorer's `score` effect, and `at` in milliseconds since the epoch.
+
+A runner that implements `runBatchCorrelated` tags every result with its job's
+`identity` and may return results in any order. A run rejects a duplicate
+identity, an unknown identity, a missing identity, the wrong result count, or
+an observation that does not echo its job's `targetStepKey` and `scorerKey`,
+all with `scorer_protocol`.
+
+A runner that implements only `runBatch` is correlated positionally:
+
+1. Return exactly one observation per job.
+2. Return them in the order the jobs were given.
+3. Echo each job's `targetStepKey` and `scorerKey` in each observation.
+
+A run verifies all three and fails with `scorer_protocol` when one is broken.
+Before calling an order-only runner, it also refuses two jobs that share a
+step key and scorer with `ambiguous_score_job`: give each case its own step
+key, or implement `runBatchCorrelated`. The runner service in
+`@smthrs/scorers` implements `runBatchCorrelated`, so its service value can be
+used directly and `ambiguous_score_job` never applies to it. For that service,
+see the [scorers API](/api/scorers).
+
+A returned score that is not finite and inside [0, 1] becomes an inconclusive
+observation naming the scorer and the offending value, and the run's own
+timestamp, not the adapter's, is what reaches the observation.
+
+## Read the result
+
+`RunResult.cases` is in declaration order regardless of completion order. Each
+`CaseResult` carries either an `execution` or an `error`, plus the
+observations attributed to the case. `RunResult.observations` flattens the
+cases' observations into one list of `score` and `inconclusive` entries.
+Reasons are bounded to 2048 characters, because they reach CI logs and
+committed reports.
+
+Next: [record a baseline](./maintain-a-baseline.md).