scenescout 3.3.0 → 3.4.0

package/CHANGELOG.md

@@ -1,5 +1,17 @@
 # scenescout
 
+## 3.4.0
+
+### Minor Changes
+
+- 04f0f22: Measure whether a lane's confidence means anything, and make one lane rubric serve a whole wave.
+  
+  Every lane has been told its confidence must be calibrated — "0.5 means a coin flip, 0.95 means you would bet on it" — and the number was then averaged into one line and discarded. Nothing was stored, so nothing could ever be checked, and a confidence nobody checks is decoration.
+  
+  Lane decisions are now kept, and the report carries a calibration section: how often a decision at a stated confidence matched a finding the project holds, bucketed, with an expected calibration error. It says plainly what the number is not — agreement between the lanes and the bar the project applies, over its whole history, not evidence that the app is broken — and names the two ways a lane is counted wrong through no fault of its own. A decision naming no failing endpoint cannot be looked up at all, so it is excluded and disclosed rather than scored as a miss. Below eight checkable decisions the figure is withheld, but the section says so rather than vanishing. Where findings have since been re-tested through `scout_verify`, those verdicts are reported beside it, because they *are* evidence about the app.
+  
+  Separately, the lane name used to sit in the second sentence of the instruction every lane receives, so two lanes' prompts diverged almost immediately and shared no prefix. The rubric is now identical for every lane in a wave and the name is the last thing said, which makes it one cacheable prefix instead of one per lane.
+
 ## 3.3.0
 
 ### Minor Changes

package/dist/engine/calibration.js

@@ -0,0 +1,217 @@
+/**
+ * Whether a lane's confidence means anything.
+ *
+ * `lane.ts` tells every lane its confidence must be calibrated — "0.5 means a
+ * coin flip, 0.95 means you would bet on it" — and then the number was
+ * averaged into one line and thrown away. Nothing was stored, so nothing could
+ * ever be checked, and a confidence nobody checks is decoration. Decision-only
+ * models report an expected calibration error for exactly this reason; asking
+ * for calibration without measuring it is the half of the idea that costs
+ * nothing and buys nothing.
+ *
+ * WHAT THIS MEASURES, precisely, because the number is easy to over-read: a
+ * lane says "defect" about an observation and attaches a machine signature.
+ * The run either went on to file a finding carrying that signature or it did
+ * not. That is agreement between the lane and the bar the run actually
+ * applied — NOT ground truth about the app. A lane can be perfectly calibrated
+ * against a planner that files the wrong things. Where a finding has since
+ * been re-tested through `scout_verify`, the verdict is reported beside it,
+ * and that IS evidence about the app.
+ *
+ * Pure, so every rule here is table-tested.
+ */
+import { failingSignatures } from "./memory.js";
+/**
+ * Upper edge of each confidence bucket. Five is enough to see a shape and few
+ * enough that each holds a usable count on a run of a few dozen decisions;
+ * twenty buckets of one decision each measure nothing.
+ */
+export const BUCKET_EDGES = [0.2, 0.4, 0.6, 0.8, 1.0];
+/**
+ * Which bucket a confidence falls in. The top bucket is closed so 1.0 has
+ * somewhere to go.
+ *
+ * A value that is not a number lands in the LOWEST bucket, not the highest.
+ * The schema refuses those, but stored history is read back without one, and
+ * falling through the comparisons put a NaN in the 0.8–1.0 bucket — a
+ * confidence nobody stated, reported as near-certainty.
+ */
+export function bucketOf(confidence) {
+    if (!Number.isFinite(confidence))
+        return 0;
+    const c = Math.min(1, Math.max(0, confidence));
+    for (let i = 0; i < BUCKET_EDGES.length; i += 1) {
+        if (c <= BUCKET_EDGES[i])
+            return i;
+    }
+    return BUCKET_EDGES.length - 1;
+}
+export function bucketLabel(i) {
+    const low = i === 0 ? 0 : BUCKET_EDGES[i - 1];
+    return `${low.toFixed(1)}–${BUCKET_EDGES[i].toFixed(1)}`;
+}
+/**
+ * The keys a piece of evidence can be joined on, or nothing when it cannot be
+ * joined at all.
+ *
+ * Findings MERGE: the store treats `GET /api/things/3 500` and
+ * `GET /api/things/7 500` as one bug, because the path normalises to
+ * `/api/things/:id`. Joining on the literal string reported five lane
+ * decisions about one endpoint as one filed and four dropped — the lanes
+ * understated fivefold, and the number read as a finding about them rather
+ * than a bug in this join. So the join uses the store's own rule.
+ *
+ * Evidence carrying no failure signature returns NOTHING rather than falling
+ * back to its literal text. That fallback looked harmless and was the single
+ * worst thing here: `500 on GET /api/r0` — ordinary English, and whichever
+ * agent wrote the finding chose the word order — produced a key that matched
+ * nothing, and a lane that was right about a bug that WAS filed published an
+ * expected calibration error of 0.90. An unjoinable decision is now counted
+ * and disclosed as unjoinable, not scored as a lane being wrong.
+ */
+export function joinKeys(evidence) {
+    return failingSignatures(evidence);
+}
+/** A confidence as stored may be anything; the schema guards the wire, not the file. */
+function usableConfidence(value) {
+    return typeof value === "number" && Number.isFinite(value) && value >= 0 && value <= 1 ? value : null;
+}
+/**
+ * Join lane decisions to what the run filed.
+ *
+ * Only a "defect" carrying a signature has a checkable outcome. An "unsure" is
+ * a request to look closer rather than a claim, and a decision with no
+ * signature cannot be joined to anything — counting either would measure the
+ * lane's willingness to attach evidence, not its judgement.
+ */
+export function calibrate(decisions, findings) {
+    const filedKeys = new Map();
+    for (const f of findings) {
+        if (!f.evidence)
+            continue;
+        // A finding verified as still present is the most informative match, so it
+        // wins a key two findings share; otherwise first write wins and the result
+        // does not depend on the order findings came back in.
+        for (const key of joinKeys(f.evidence)) {
+            const prev = filedKeys.get(key);
+            if (!prev || (f.verdict && f.verifiedAt && !(prev.verdict && prev.verifiedAt)))
+                filedKeys.set(key, f);
+        }
+    }
+    const claims = decisions.filter((d) => d.verdict === "defect" && d.evidence !== null);
+    const checkable = claims.filter((d) => usableConfidence(d.confidence) !== null && joinKeys(d.evidence).size > 0);
+    const unjoinable = claims.length - checkable.length;
+    if (checkable.length === 0)
+        return unjoinable > 0 ? { checkable: 0, filed: 0, stated: 0, buckets: [], ece: 0, verified: { present: 0, gone: 0, changed: 0 }, unjoinable } : null;
+    const buckets = BUCKET_EDGES.map(() => ({ n: 0, conf: 0, hits: 0 }));
+    const verified = { present: 0, gone: 0, changed: 0 };
+    const matched = new Map();
+    let filed = 0;
+    let stated = 0;
+    for (const d of checkable) {
+        let match;
+        for (const key of joinKeys(d.evidence)) {
+            match = filedKeys.get(key);
+            if (match)
+                break;
+        }
+        const hit = match !== undefined;
+        const confidence = usableConfidence(d.confidence);
+        const b = buckets[bucketOf(confidence)];
+        b.n += 1;
+        b.conf += confidence;
+        if (hit)
+            b.hits += 1;
+        if (hit)
+            filed += 1;
+        stated += confidence;
+        // By finding, not by decision: several decisions can match one finding,
+        // and this sentence counts findings.
+        if (match)
+            matched.set(match.id, match);
+    }
+    for (const f of matched.values()) {
+        // A verdict read from disk is not guaranteed to be one of the three.
+        if (f.verifiedAt && f.verdict && f.verdict in verified)
+            verified[f.verdict] += 1;
+    }
+    const n = checkable.length;
+    let ece = 0;
+    const out = [];
+    buckets.forEach((b, i) => {
+        if (b.n === 0)
+            return;
+        const meanConf = b.conf / b.n;
+        const rate = b.hits / b.n;
+        ece += (b.n / n) * Math.abs(meanConf - rate);
+        out.push({ label: bucketLabel(i), decisions: b.n, stated: meanConf, filed: rate });
+    });
+    return { checkable: n, filed, stated: stated / n, buckets: out, ece, verified, unjoinable };
+}
+const pct = (x) => `${Math.round(x * 100)}%`;
+/**
+ * The report section. It leads with what the number is not, because "ECE 0.24"
+ * above a table of percentages reads as a verdict on the app rather than on
+ * the lanes that judged it.
+ */
+export function formatCalibration(c) {
+    if (!c)
+        return [];
+    if (c.checkable < MIN_FOR_A_VERDICT) {
+        // Suppress the NUMBER, not the fact. A run that recorded seven decisions
+        // and a run that recorded none look identical when the section simply
+        // vanishes, and the reader concludes the feature is broken — which is this
+        // project's own definition of a silent path.
+        const had = c.checkable + c.unjoinable;
+        if (had === 0)
+            return [];
+        return [
+            `## How well the lanes judged`,
+            ``,
+            `Not enough to say yet: ${c.checkable} lane decision(s) could be checked${c.unjoinable > 0 ? ` (and ${c.unjoinable} could not be looked up at all)` : ""}, ` +
+                `and ${MIN_FOR_A_VERDICT} are needed before a calibration figure survives one of them changing.`,
+            ``,
+        ];
+    }
+    const lines = [
+        `## How well the lanes judged`,
+        ``,
+        `${c.checkable} lane decision(s) called a defect on an endpoint that failed; ${c.filed} of them match a finding this project holds (${pct(c.filed / c.checkable)}), ` +
+            `against a mean stated confidence of ${c.stated.toFixed(2)}.`,
+        ``,
+        `**What this is not.** It covers the project's whole history, not this run, and it measures agreement between the lanes and the bar this project applies — not whether the app is broken. ` +
+            `A lane can be perfectly calibrated against a planner that files the wrong things. Two things also count against a lane through no fault of its own: a defect filed with no machine signature, ` +
+            `and a finding whose signature was dropped when the store merged it into another. Read the figure as an ordering, not a measurement.`,
+        ``,
+        `| Stated confidence | Decisions | Said | Filed |`,
+        `|---|---:|---:|---:|`,
+    ];
+    for (const b of c.buckets) {
+        lines.push(`| ${b.label} | ${b.decisions} | ${b.stated.toFixed(2)} | ${pct(b.filed)} |`);
+    }
+    lines.push(``, `Expected calibration error **${c.ece.toFixed(2)}** — the gap between what the lanes said and what happened, weighted by bucket. ${readEce(c.ece)}`);
+    if (c.unjoinable > 0) {
+        lines.push(``, `${c.unjoinable} further decision(s) called a defect but named no failing endpoint, so nothing could be looked up for them. They are excluded above rather than counted as wrong.`);
+    }
+    const seen = c.verified.present + c.verified.gone + c.verified.changed;
+    if (seen > 0) {
+        lines.push(``, `${seen} of the findings those decisions matched ${seen === 1 ? "has" : "have"} since been re-tested with \`scout_verify\`: ${c.verified.present} still present, ${c.verified.gone} gone, ${c.verified.changed} changed. ` +
+            `Those verdicts are evidence about the app, unlike the table above. Findings no lane decision matched are not counted here, so this will not agree with the re-tested column elsewhere in the report.`);
+    }
+    lines.push(``);
+    return lines;
+}
+/** Below this, the number swings on a single decision and is worse than no number. */
+export const MIN_FOR_A_VERDICT = 8;
+/**
+ * What the figure suggests, hedged on purpose. The join loses decisions it
+ * cannot key and findings whose signature a merge discarded, and both push the
+ * error upward — so a confident "these numbers are wrong" can itself be wrong.
+ */
+function readEce(ece) {
+    if (ece <= 0.05)
+        return "The lanes' numbers track what happened closely.";
+    if (ece <= 0.15)
+        return "Close enough to compare lanes by.";
+    return "Far enough apart that a lane's confidence is worth reading as an ordering rather than a rate — after checking the exclusions above, which push this upward.";
+}

package/dist/engine/lane.js

@@ -138,14 +138,28 @@ export function parseLaneReport(text, expectedLane) {
  */
 export function laneReportInstruction(lane) {
     return [
-        "Reply with ONE JSON object and nothing else — no prose before or after it, no explanation, no headings. A fenced ```json block is fine.",
-        `Shape: {"lane":${JSON.stringify(lane)},"status":<${quoteAll(LANE_STATUSES)}>,"decisions":[…],"routes":[…],"blocked_by":<string or null>}.`,
-        `Each decision: {"observation":<a short id for what was observed, unique in the report, at most ${LANE_OBSERVATION_MAX} characters>,"verdict":<${quoteAll(LANE_VERDICTS)}>,"severity":<${quoteAll(LANE_SEVERITIES)} or null>,"category":<${quoteAll(LANE_CATEGORIES)} or null>,"confidence":<0..1>,"evidence":<machine signature such as "GET /api/things 500", or null>}.`,
-        `A "defect" must carry a severity and a category. "evidence" is a signature, not a sentence: at most ${LANE_EVIDENCE_MAX} characters. "confidence" is how sure you are of the verdict, calibrated: 0.5 means a coin flip, 0.95 means you would bet on it.`,
-        `"routes" lists the routes you covered, each at most ${LANE_ROUTE_MAX} characters. "blocked_by" is one line of at most ${LANE_BLOCKED_BY_MAX} characters saying what stopped you: required when the status is "blocked", allowed with "partial", null with "complete"; the detail belongs in a finding. The lane name is at most ${LANE_NAME_MAX} characters. At most ${LANE_MAX_ITEMS} decisions and ${LANE_MAX_ITEMS} routes. Unknown keys are refused.`,
-        "The object IS your final report: whatever hands it back must hand back the object verbatim, not a summary of it.",
+        ...LANE_RUBRIC,
+        // The ONLY lane-specific sentence, and it comes last on purpose. Every
+        // lane in a wave is given the same rubric, so keeping it byte-identical up
+        // to here makes it one shared prompt prefix: the cache hits from the
+        // second lane onward instead of diverging at the first sentence, which is
+        // what putting the name in the shape line used to do.
+        `Your lane name is ${JSON.stringify(lane)}; put exactly that in "lane".`,
     ].join(" ");
 }
+/**
+ * Everything every lane is told, identical for all of them. Built once from
+ * the same constants the parser enforces, because a limit a lane is not told
+ * refuses good replies.
+ */
+const LANE_RUBRIC = [
+    "Reply with ONE JSON object and nothing else — no prose before or after it, no explanation, no headings. A fenced ```json block is fine.",
+    `Shape: {"lane":<your lane name>,"status":<${quoteAll(LANE_STATUSES)}>,"decisions":[…],"routes":[…],"blocked_by":<string or null>}.`,
+    `Each decision: {"observation":<a short id for what was observed, unique in the report, at most ${LANE_OBSERVATION_MAX} characters>,"verdict":<${quoteAll(LANE_VERDICTS)}>,"severity":<${quoteAll(LANE_SEVERITIES)} or null>,"category":<${quoteAll(LANE_CATEGORIES)} or null>,"confidence":<0..1>,"evidence":<machine signature such as "GET /api/things 500", or null>}.`,
+    `A "defect" must carry a severity and a category. "evidence" is a signature, not a sentence: at most ${LANE_EVIDENCE_MAX} characters. "confidence" is how sure you are of the verdict, calibrated: 0.5 means a coin flip, 0.95 means you would bet on it.`,
+    `"routes" lists the routes you covered, each at most ${LANE_ROUTE_MAX} characters. "blocked_by" is one line of at most ${LANE_BLOCKED_BY_MAX} characters saying what stopped you: required when the status is "blocked", allowed with "partial", null with "complete"; the detail belongs in a finding. The lane name is at most ${LANE_NAME_MAX} characters. At most ${LANE_MAX_ITEMS} decisions and ${LANE_MAX_ITEMS} routes. Unknown keys are refused.`,
+    "The object IS your final report: whatever hands it back must hand back the object verbatim, not a summary of it.",
+];
 function quoteAll(values) {
     return values.map((v) => `"${v}"`).join("|");
 }

package/dist/engine/memory.js

@@ -250,8 +250,39 @@ export function mergeMemory(mine, theirs) {
     }
     if (Object.keys(out.roleAccess).length === 0)
         delete out.roleAccess;
+    // Lanes commonly run in SEPARATE processes — that is the point of a lane —
+    // so without this the spread at the top would keep one process's decisions
+    // and drop every other lane's, which is the exact bug this merge exists to
+    // prevent for findings. Keyed so merging the same foreign document twice is
+    // idempotent.
+    const byDecision = new Map();
+    for (const d of theirs.laneDecisions ?? [])
+        byDecision.set(decisionKey(d), d);
+    for (const d of mine.laneDecisions ?? [])
+        byDecision.set(decisionKey(d), d);
+    out.laneDecisions = [...byDecision.values()].sort((a, b) => a.at.localeCompare(b.at)).slice(-MAX_LANE_DECISIONS);
+    if (out.laneDecisions.length === 0)
+        delete out.laneDecisions;
     return out;
 }
+/**
+ * What makes two stored decisions the same judgement.
+ *
+ * Deliberately NOT the timestamp. `at` is stamped when the planner folds the
+ * reply, not when the lane judged, so relaying one reply twice — which the
+ * protocol invites, since a refused report is asked for again — wrote every
+ * decision a second time and doubled the lane's weight in the calibration.
+ * Identity is what was decided, so re-folding the same reply is a no-op.
+ */
+function decisionKey(d) {
+    return [d.lane, d.observation, d.verdict, d.severity ?? "", d.category ?? "", d.confidence, d.evidence ?? ""].join("|");
+}
+/**
+ * Most lane decisions kept. Calibration wants a few dozen; a long-lived
+ * project would otherwise accumulate every decision ever made and re-serialise
+ * them on each save, which is what made an old history slow to open.
+ */
+export const MAX_LANE_DECISIONS = 1000;
 const MAX_DISCOVERED_ROUTES = 300;
 /** Shared finding-similarity helpers (used by live dedup and retro-merge). */
 function findingTokens(s) {
@@ -295,7 +326,22 @@ function findingLiterals(...texts) {
  * (`/api/users/7` and `/api/users/9` are one endpoint) so a per-record repro
  * does not read as a per-record bug.
  */
-function endpointSignatures(evidence) {
+/**
+ * The signatures that identify a BUG: only those carrying a failure status.
+ *
+ * A bare `POST /api/orders` says which endpoint was involved, not what went
+ * wrong — a double submit and an accepted negative quantity both name it, and
+ * are two bugs. Exported so calibration applies the store's own rule instead
+ * of a second copy of this regex, which is what it had.
+ */
+export function failingSignatures(evidence) {
+    return new Set([...endpointSignatures(evidence)].filter((sig) => /\s[45]\d{2}$/.test(sig)));
+}
+/**
+ * The endpoint signatures a piece of evidence names: method + normalised path,
+ * with the failure status when one follows.
+ */
+export function endpointSignatures(evidence) {
     const out = new Set();
     if (!evidence)
         return out;
@@ -338,11 +384,10 @@ function sharesEndpointSignature(a, b) {
     // `POST /api/orders` (the endpoint answered 2xx, or no status was named)
     // says which endpoint was involved, not what went wrong: a double submit
     // and an accepted negative quantity both name it, and are two bugs.
-    const failing = (evidence) => new Set([...endpointSignatures(evidence)].filter((sig) => /\s[45]\d{2}$/.test(sig)));
-    const aSigs = failing(a.evidence);
+    const aSigs = failingSignatures(a.evidence);
     if (aSigs.size === 0)
         return false;
-    for (const sig of failing(b.evidence))
+    for (const sig of failingSignatures(b.evidence))
         if (aSigs.has(sig))
             return true;
     return false;
@@ -782,6 +827,45 @@ export class MemoryStore {
         fs.writeFileSync(this.assumptionsPath, content);
         return true;
     }
+    /** What the lanes decided, oldest first. Empty on a project that has never run one. */
+    get laneDecisions() {
+        return this.data.laneDecisions ?? [];
+    }
+    /**
+     * Record what a lane decided. Called once per accepted lane report, so the
+     * confidence it stated can be checked later against what the run filed.
+     * Free text from the lane is redacted like every other stored string: an
+     * observation is written by a model reading the app under test.
+     */
+    addLaneDecisions(lane, decisions) {
+        if (decisions.length === 0)
+            return 0;
+        const list = this.data.laneDecisions ?? [];
+        const seen = new Set(list.map(decisionKey));
+        let added = 0;
+        for (const d of decisions) {
+            const record = {
+                ...d,
+                lane,
+                observation: redactSecrets(d.observation).slice(0, 200),
+                evidence: d.evidence === null ? null : redactSecrets(d.evidence).slice(0, 200),
+            };
+            if (seen.has(decisionKey(record)))
+                continue;
+            seen.add(decisionKey(record));
+            list.push(record);
+            added += 1;
+        }
+        this.data.laneDecisions = list.slice(-MAX_LANE_DECISIONS);
+        if (added > 0)
+            this.flush();
+        // What survives the cap. Appends go to the tail and the cap keeps the
+        // tail, so all of `added` survives unless the call itself exceeds the cap.
+        // Measuring it as growth instead looked right and was not: `list` aliases
+        // the stored array, so the "before" length was read after the appends and
+        // every call after the first reported nothing kept — while storing fine.
+        return Math.min(added, MAX_LANE_DECISIONS);
+    }
     /** Mark a finding resolved; returns it or null. */
     resolveFinding(id) {
         const f = this.data.findings.find((x) => x.id === id);

package/dist/engine/report.js

@@ -4,6 +4,7 @@ import { SHARED_CHROME_ROUTE } from "./memory.js";
 import { sayVerification } from "./verify.js";
 import { feedForSession } from "./live.js";
 import { buildReplayHtml, evidenceFor } from "./replay.js";
+import { calibrate, formatCalibration } from "./calibration.js";
 import { formatPace, measurePace } from "./pace.js";
 function playwrightSkeleton(f) {
     const routeClass = f.state.split("#")[0].split("?")[0];
@@ -616,6 +617,9 @@ export function generateReport(memory, oracleLog, extras, opts = {}) {
     if (extras?.pace !== false) {
         lines.push(...formatPace(measurePace(memory.actionLog, Date.now(), extras?.attachedSessions ?? [])));
     }
+    // Only on a run that used lanes, and only once enough of them have been
+    // judged for the number to mean anything; formatCalibration decides both.
+    lines.push(...formatCalibration(calibrate(memory.laneDecisions, memory.findings)));
     const markdown = lines.join("\n");
     const outPath = path.join(memory.dir, "report.md");
     const htmlPath = path.join(memory.dir, "report.html");

package/dist/mcp-server.js

@@ -528,10 +528,35 @@ server.registerTool("scout_lane_report", {
         if (reply === undefined)
             return { content: [{ type: "text", text: laneReportInstruction(lane) }] };
         const parsed = parseLaneReport(reply, lane);
-        const out = parsed.ok
-            ? `Lane report accepted — ${summarizeLaneReport(parsed.report)}`
-            : `Lane report REFUSED: ${parsed.reason}. Ask the lane once for the corrected object; do not re-judge its prose.`;
-        return { content: [{ type: "text", text: out }] };
+        if (!parsed.ok) {
+            return {
+                content: [
+                    { type: "text", text: `Lane report REFUSED: ${parsed.reason}. Ask the lane once for the corrected object; do not re-judge its prose.` },
+                ],
+            };
+        }
+        // Keep what the lane decided, so the confidence it stated can be checked
+        // against what the run goes on to file. Best-effort: a lane report is
+        // still accepted if this project has no memory open yet, because the
+        // planner's fold must not depend on where the report was written.
+        // ONLY the lane's own session. Falling back to any engine with memory
+        // open put one project's decisions into another project's store whenever
+        // two sessions were attached to different apps — and the lane having
+        // already closed makes that the ordinary case, not an edge one.
+        const owner = engines.get(lane)?.memory;
+        const at = new Date().toISOString();
+        const kept = owner
+            ? owner.addLaneDecisions(lane, parsed.report.decisions.map((d) => ({ ...d, lane, at })))
+            : 0;
+        // Say when nothing was kept. Every lane closing its session before the
+        // planner folds its report is the order the method describes, and it
+        // leaves no memory to write to — reporting a bare "accepted" while the
+        // skill promises the decisions are kept is the kind of silence that
+        // makes a later calibration section look wrong rather than absent.
+        const note = kept > 0
+            ? ` (${kept} decision(s) kept for calibration)`
+            : ` (nothing kept for calibration — session ${JSON.stringify(lane)} is not attached here, so there is no project memory to write to. Fold a lane report before closing that lane's session.)`;
+        return { content: [{ type: "text", text: `Lane report accepted — ${summarizeLaneReport(parsed.report)}${note}` }] };
     }
     catch (err) {
         return errorText(err);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "scenescout",
-  "version": "3.3.0",
+  "version": "3.4.0",
   "description": "SceneScout — exploratory UI testing for AI coding agents. An MCP server that gives any agent (Claude Code, Cursor, VS Code Copilot, Codex, Gemini CLI and others) a structured view of a running web app, always-on oracles, a network-level write policy, memory across runs and a gap-checked report.",
   "license": "MIT",
   "author": "brunoboto96",
@@ -71,7 +71,7 @@
     "mcp-check": "npm run build && npm run mcp-check:run",
     "mcp-check:run": "tsx scripts/mcp-check.ts",
     "test": "npm run build && npm run test:unit && npm run smoke:run && npm run mcp-check:run",
-    "test:unit": "npm run scan-test && npm run oracle-test && npm run policy-test && npm run fixture-test && npm run dispatch-test && npm run design-test && npm run contract-test && npm run pace-test && npm run request-test && npm run settle-test && npm run claims-test && npm run verify-test && npm run brief-test && npm run lane-test && npm run memory-test && npm run install-test && npm run live-test && npm run demo-test && npm run hygiene-test",
+    "test:unit": "npm run scan-test && npm run oracle-test && npm run policy-test && npm run fixture-test && npm run dispatch-test && npm run design-test && npm run contract-test && npm run pace-test && npm run request-test && npm run settle-test && npm run claims-test && npm run verify-test && npm run brief-test && npm run lane-test && npm run calibration-test && npm run memory-test && npm run install-test && npm run live-test && npm run demo-test && npm run hygiene-test",
     "scan-test": "tsx scripts/scan-test.ts",
     "oracle-test": "tsx --test scripts/oracle-test.ts",
     "policy-test": "tsx --test scripts/policy-test.ts",
@@ -90,7 +90,8 @@
     "settle-test": "tsx --test scripts/settle-test.ts",
     "claims-test": "tsx --test scripts/claims-test.ts",
     "verify-test": "tsx --test scripts/verify-test.ts",
-    "brief-test": "tsx --test scripts/brief-test.ts"
+    "brief-test": "tsx --test scripts/brief-test.ts",
+    "calibration-test": "tsx --test scripts/calibration-test.ts"
   },
   "dependencies": {
     "@modelcontextprotocol/sdk": "^1.12.0",

package/skills/scenescout/SKILL.md

@@ -74,7 +74,7 @@ Some flows need a TEAM — a document one role submits and another approves, a r
 - `scout_close {all: true}` at the end of a multi-role run; `scout_close {session}` to drop one role early.
 - **Several agents in parallel** (subagents or a workflow, each driving its own session): each agent attaches its session when it STARTS and closes it by name when it FINISHES. Never open sessions ahead for agents that have not started, and never hand an open session from one agent to the next: an agent waiting for its turn should hold no browser. Give each session an `objective` when you attach it (`scout_attach {session, objective:"Approve and reject orders as a manager"}`) and wrap each goal in `scout_journey`: the live view shows that objective and the task it is on beside the session's feed, which is how the person watching knows what every agent is for. Keep that goal TRUE: one journey per goal, one goal per thing you are checking ("Save a settings change as the auditor", not "Check every page"), ended the moment it is decided and the next one started before you move on. A journey that outlives its goal shows the viewer an objective the session left behind minutes ago. Run roughly as many agents at once as the machine has cores, less two, since each drives a real browser; beyond that they only queue. Exploring one area is well within a mid-tier model, so run these agents on one (Sonnet or its equivalent in your client) unless the user names a model; keep the larger model for the agent that plans the split and writes the report. No agent may call `scout_close {all: true}` while others run — only the last step, once every agent has finished.
 - **Let the engine divide the app: `scout_lane_brief {lanes, goal}`.** Call it after the first crawl, when route knowledge is complete. It splits the known routes into whole modules — everything under `/orders` goes to one lane, so that lane carries state between its own steps instead of re-learning the app on every route — deals the modules out so the lanes come out within a route or two of each other, and returns each lane's session name, the `objective` to attach it with, and the routes it owns. Pass `routes` to split a subset instead. Hand each brief to its agent verbatim. Dividing by hand fails in two ways that a finished run cannot tell apart from success: two lanes audit the same register while a third module is never opened, and route coverage reads complete either way; and lanes launch without an objective, so the person watching sees browsers clicking through their app with nothing to say why.
-- **Lanes report in typed decisions, not prose.** Each lane files its findings with `scout_finding` as it goes, so the report already has them; what the lane hands BACK to the planner is one JSON object, the lane report, and nothing else. Get the paragraph to put in a lane's prompt from `scout_lane_report {lane}`: it names the shape (`status`, one decision per observation with `verdict`, `severity`, `category`, a calibrated `confidence` and a machine-signature `evidence`, the `routes` covered, `blocked_by`), every value's closed set (the categories are `scout_finding`'s) and every length limit, because a limit a lane is not told refuses good replies. When the lane hands back, pass its text to `scout_lane_report {lane, reply}`: it returns the one-line fold (defects, highs, unsure, mean confidence, routes, what blocked it) or the reason the reply was refused. Relay a refusal to the lane once and ask for the corrected object; never re-judge its prose yourself, and tell every lane that the object IS its final report, since a lane that writes the object and then hands back a summary of it has failed. The planner then folds lanes by counting, not by reading: a lane's reply is a few hundred tokens whatever it found, and a value the schema refuses is caught at the boundary instead of becoming a severity like "Low-Medium" in the report. Where the client lets you set a lane's reasoning effort, use **medium**: in this project's benchmark it judged as well as high in three-quarters of the time and under half of xhigh, low inflated severities, and max took three minutes per lane for no better agreement.
+- **Lanes report in typed decisions, not prose.** Each lane files its findings with `scout_finding` as it goes, so the report already has them; what the lane hands BACK to the planner is one JSON object, the lane report, and nothing else. Get the paragraph to put in a lane's prompt from `scout_lane_report {lane}`: it names the shape (`status`, one decision per observation with `verdict`, `severity`, `category`, a calibrated `confidence` and a machine-signature `evidence`, the `routes` covered, `blocked_by`), every value's closed set (the categories are `scout_finding`'s) and every length limit, because a limit a lane is not told refuses good replies. When the lane hands back, pass its text to `scout_lane_report {lane, reply}`: it returns the one-line fold (defects, highs, unsure, mean confidence, routes, what blocked it) or the reason the reply was refused, and keeps each decision so the confidence the lane stated can be checked against what the run went on to file — the report's calibration section is built from that, and it only appears once enough decisions exist to mean something. **So the confidence is worth stating honestly**: a lane that writes 0.95 on everything makes the section say so. Relay a refusal to the lane once and ask for the corrected object; never re-judge its prose yourself, and tell every lane that the object IS its final report, since a lane that writes the object and then hands back a summary of it has failed. The planner then folds lanes by counting, not by reading: a lane's reply is a few hundred tokens whatever it found, and a value the schema refuses is caught at the boundary instead of becoming a severity like "Low-Medium" in the report. Where the client lets you set a lane's reasoning effort, use **medium**: in this project's benchmark it judged as well as high in three-quarters of the time and under half of xhigh, low inflated severities, and max took three minutes per lane for no better agreement.
 
 ## The impatient-user pass (extensive)