codecartographer-pi 0.22.3 → 0.24.0

package/.codecarto/broadside/SKILL.md

@@ -26,7 +26,21 @@ not replace any phase; it tells phases where to look.
 
 ## Reading a Broad-Side run
 
-1. Read `synthesis.md` first. It carries the executive summary, severity counts,
+0. If `verified.md` exists, read it first. A verification pass
+   (`codecarto_broadside {action: "verify"}`, `/codecarto-broadside verify`)
+   has read the top defect and security findings against the real source with
+   read-only tools and given each a verdict: **confirmed** (a reachable
+   failure, with the input or call site that triggers it), **not-a-defect**
+   (the claim is literally true of the code but nothing can reach the failure
+   it describes), **discarded** (the claim is wrong about the code, with the
+   guard or line that shows it), or **unclear**. Start from the confirmed
+   ones; treat a discarded one as answered unless the reasoning is thin.
+   Measured on this repository, the top twelve findings by severity were two
+   real defects and ten that a look at the guard, the caller, or the tsconfig
+   dismissed — the pass agreed with a reviewer on all twelve for about a cent
+   a finding. A confirmed verdict is still a model's reading: a strong lead for
+   a human's next look, not a validated claim.
+1. Read `synthesis.md`. It carries the executive summary, severity counts,
    the top cross-lens findings, and per-module risk levels.
 2. Read `triage.md` for the work order: each lead scored by impact ×
    difficulty with a P0–P3 priority and an effort estimate. It is a starting
@@ -66,6 +80,7 @@ Broad-Side is an executable-surface feature. On the Pi extension:
 /codecarto-broadside collect            # poll, save, synthesize
 /codecarto-broadside status             # show recorded runs
 /codecarto-broadside models             # compare batch models
+/codecarto-broadside verify --top=10    # read the top findings against the source
 ```
 
 On the MCP server:
@@ -75,6 +90,7 @@ codecarto_broadside {cwd, action: "submit", lenses: [...]}   # fire the batches
 codecarto_broadside {cwd, action: "collect"}                  # poll, save, synthesize
 codecarto_broadside {cwd, action: "status"}                   # show recorded runs
 codecarto_broadside {cwd, action: "models"}                   # compare batch models
+codecarto_broadside {cwd, action: "verify", top: 10}          # read the top findings against the source
 ```
 
 The `models` action lists every `:batch` variant on OpenRouter — pricing per
@@ -86,10 +102,37 @@ Submits pre-flight the chosen model: pricing comes from the live catalog
 model that does not advertise structured-output support is refused outright,
 because every lens depends on `json_schema` response_format.
 
+Each lens reads a scope of the repository. Architecture reads the manifest,
+entry point, README, and file tree; defect, conventions, and porting read
+every source file; the **security** and **API** lenses target where the
+trust boundary usually lives — `server/**`, `**/auth*`, `**/middleware/**`,
+`SECURITY.md` (security) and `server/**`, `api/**`, `src/server/**`,
+`src/api/**`, `**/*routes*`, `**/*router*`, `**/*handler*`, `**/*endpoint*`
+(API). A repository whose server is `src/server.js` matches none of those, so
+when the targeted patterns find nothing those two lenses **fall back to every
+source file** — priced as such, chunked at the lens's slice size rather than
+truncated, and said so on the lens line of the estimate, the submit report,
+`status`, and the prompt the model receives. A lens whose targeted patterns
+and fallback both find nothing (only test files, say) is skipped with a line
+naming both. `max_cost` is the guard against a fallback scan on a large
+repository being more than you meant to spend.
+
 Collect runs two cross-lens post-passes by default: **synthesis** (the
 executive report) and **triage** (the prioritized work order). Pass
 `include_synthesis: false` or `include_triage: false` on collect to skip one.
 
+`verify` is the third pass, run separately after collect because it is
+sync-priced rather than batch-priced: for each of the top `top` findings
+(default 10, most severe first) of the defect and security lenses, one chat
+completion on the run's model without its `:batch` suffix (`model` overrides),
+with three read-only tools confined to the repository's source files —
+`read_file` by line range, `grep`, `list_dir` — at most eight tool calls, low
+reasoning effort, then a verdict. It writes `verified.md` and `verified.json`
+beside `triage.md` and records the pass on the run. `max_cost` is a running
+cap here, since a sync call's cost is known only when it returns: the pass
+stops before the next finding once the calls so far have reached it and
+reports `partial`. About a cent a finding on the default model.
+
 Two caveats apply to any model you pick. The `models` action lists every id
 OpenRouter advertises a `:batch` variant for, and many of those variants do not
 exist — submitting one returns `does not have a :batch endpoint`, with nothing in

package/.codecarto/workflow/scaffold-version.yaml

@@ -3,4 +3,4 @@
 # workspace's framework-owned files (GUIDE.md, templates/, workflow/ pipelines
 # and VALIDATE.md) predate the running release. Written at release time and
 # copied verbatim by init — never edit by hand.
-scaffold_version: 0.22.3
+scaffold_version: 0.24.0

package/README.md

@@ -383,7 +383,7 @@ Each workflow tool accepts an absolute `cwd` for the target repository. `codecar
 
 ## Broad-Side (batch reconnaissance)
 
-Broad-Side is the cheap sweep you run *before* the expensive interactive run. It fires six analysis lenses — architecture, API surface, security, mechanical defect scan, convention extraction, porting — at a repository as single-turn prompts over the [OpenRouter Batch API](https://openrouter.ai/docs), then cross-references them into one executive report (`synthesis.md`) and a prioritized P0–P3 work order (`triage.md`).
+Broad-Side is the cheap sweep you run *before* the expensive interactive run. It fires six analysis lenses — architecture, API surface, security, mechanical defect scan, convention extraction, porting — at a repository as single-turn prompts over the [OpenRouter Batch API](https://openrouter.ai/docs), then cross-references them into one executive report (`synthesis.md`) and a prioritized P0–P3 work order (`triage.md`). The security and API lenses target the paths where a trust boundary usually lives (`server/`, `auth*`, `middleware/`, routers and handlers) and fall back to every source file when a repository has none of them — priced as such, and said so on the estimate — so a service whose server is `src/server.js` still gets its security review.
 
 **Broad-Side findings are unverified scouting leads, not evidence.** Each lens is one shot: no cross-file traversal, no runtime verification, no builds, no tests. Every finding is a `file:line` pointer that the interactive pipeline — or you — must confirm before it is a fact. That division of labor is the point: a sub-dollar unattended sweep that tells the expensive run where to look. Nothing downstream may cite a Broad-Side report as a source.
 
@@ -394,13 +394,16 @@ codecarto_broadside {cwd, action: "models"}                   # compare batch mo
 codecarto_broadside {cwd, action: "submit", lenses: [...]}    # fire the batches, priced first
 codecarto_broadside {cwd, action: "status"}                   # what is in flight
 codecarto_broadside {cwd, action: "collect"}                  # poll, save, synthesize, triage
+codecarto_broadside {cwd, action: "verify", top: 10}          # read the top findings against the source
 ```
 
+A third action, `verify`, reads the top defect and security findings of a collected run against the real source — one sync-priced call each with read-only tools (`read_file`, `grep`, `list_dir`) confined to the repository — and writes `verified.md` with a verdict per finding: **confirmed** (a reachable failure, with the trigger), **not-a-defect**, **discarded**, or **unclear**. Measured on this repository, the top twelve findings by severity were two real defects and ten claims a look at the guard or the caller dismissed; the pass agreed with a reviewer on all twelve for a cent a finding. Read `verified.md` before `triage.md`.
+
 Submit and collect are separate because batch jobs routinely take tens of minutes; collect is resumable and picks up whatever is still in flight (`wait_seconds: 0`, the default, polls once and returns), and two collects on one run — a retried tool call, a second session — never pay for the synthesis, triage, or truncation retry twice: each is claimed in the run's state before it is submitted, and a collect whose client has gone away stops polling and submits nothing further. Submit prices the run from the collected file sizes against the model's live per-token pricing (cached 24h) and refuses when the estimate exceeds `max_cost` — $1.00 unless the config or the call sets another value, `0` for no limit — unless `force: true` is passed — a pre-flight estimate, not a runtime stop. Actual spend lands in each run's `run-meta.json`.
 
 Repository defaults live in `.codecarto/broadside/config.yaml` (`model`, `api_key`, `default_lenses`, `max_cost`, `pricing` overrides, `lens_models`, `incremental`, `retry_truncated`, `include_synthesis`, `include_triage`, `wait_seconds`); an explicit tool parameter always wins. `lens_models` routes individual lenses to their own batch model — a stronger model changes security and defect findings far more than it changes an architecture map — and each override is priced, capability-checked, and clamped exactly like the default, with the estimate broken out per lens so a mixed-model run cannot be approved without seeing which lens costs what. CodeCartographer ships no stronger default: which model earns its price depends on your repository and budget, so compare candidates with the `models` action and choose — for one run with the `model` and `lens_models` parameters (Pi: `--model=ID`, `--lens-model=LENS:ID`), or for the repository in `config.yaml`. The `models` listing is advisory: OpenRouter's catalog returns a `:batch` id for some models its Batch API then refuses (`does not have a :batch endpoint`), at no cost, and nothing in the catalog tells them apart — so the listing tags the ids this repository's own submits have seen accepted or refused, and a refused lens says why in the submit report. `codecarto_skill {cwd, name: "broadside"}` returns the reading guide for a completed run, and unlike post-pipeline skills it is not gated on a finished pipeline.
 
-On the Pi extension the same run is `/codecarto-broadside [submit|collect|status|models] [lenses…] [--model=ID] [--lens-model=LENS:ID]`, with tab-completion for actions, lens names, and flags and live per-lens progress while batches poll. The two surfaces differ in one deliberate place: MCP cannot ask a human, so it refuses a run over `max_cost` until you pass `force`; Pi shows the per-lens breakdown and asks, and your approval *is* the force flag. Neither surface takes an API key as a command argument — a key typed into a slash command lands in the session transcript.
+On the Pi extension the same run is `/codecarto-broadside [submit|collect|status|models|verify] [lenses…] [--model=ID] [--lens-model=LENS:ID] [--top=N]`, with tab-completion for actions, lens names, and flags and live per-lens progress while batches poll. The two surfaces differ in one deliberate place: MCP cannot ask a human, so it refuses a run over `max_cost` until you pass `force`; Pi shows the per-lens breakdown and asks, and your approval *is* the force flag. Neither surface takes an API key as a command argument — a key typed into a slash command lands in the session transcript.
 
 Broad-Side needs runtime code, so firing a run is an executable-surface feature: Pi and MCP have it, the pure drop-in template does not (it carries only the reading guide).
 

package/agent-skill/codecartographer/references/broadside.md

@@ -95,7 +95,11 @@ Two more economies worth knowing:
 
 ## Reading a run
 
-Results land in `.codecarto/broadside/<run>/`. Read them in this order:
+Results land in `.codecarto/broadside/<run>/`. If `verified.md` is there, read
+it before anything else: `action: "verify"` has read the top defect and
+security findings against the source with read-only tools and given each a
+verdict (confirmed with its trigger, not-a-defect, discarded with the guard
+that shows it, unclear). Then, in this order:
 
 1. `synthesis.md` — executive summary, severity counts, top cross-lens
    findings, per-module risk.

package/dist/core/broadside-verify.d.ts

@@ -0,0 +1,95 @@
+import { type BroadsideLensId, type BroadsideVerifyEntry, type FetchLike, type StoredLensResult } from "./broadside.ts";
+export declare const BROADSIDE_CHAT_URL = "https://openrouter.ai/api/v1/chat/completions";
+/** How many findings `verify` reads by default, most severe first. */
+export declare const BROADSIDE_VERIFY_DEFAULT_TOP = 10;
+/** Tool calls one finding may spend before it must answer. */
+export declare const BROADSIDE_VERIFY_MAX_TOOL_CALLS = 8;
+/** The lenses whose findings carry a file:line and a claim to check. */
+export declare const BROADSIDE_VERIFIABLE_LENSES: readonly BroadsideLensId[];
+export type VerifyVerdict = "confirmed" | "not-a-defect" | "discarded" | "unclear";
+export type VerifiedFinding = {
+    index: number;
+    lensId: BroadsideLensId;
+    customId: string;
+    severity: string;
+    title: string;
+    location: string;
+    verdict: VerifyVerdict | "error";
+    confidence: string;
+    evidence: Array<{
+        file: string;
+        lines: string;
+        note: string;
+    }>;
+    reasoning: string;
+    toolCalls: number;
+    cost: number;
+};
+export type BroadsideVerifyResult = {
+    runId: string;
+    outputDir: string;
+    model: string;
+    status: BroadsideVerifyEntry["status"];
+    /** How many findings the run had in the verifiable lenses. */
+    candidates: number;
+    findings: VerifiedFinding[];
+    totalCost: number;
+    /** Set when the cost cap stopped the pass before every selected finding was read. */
+    stoppedByCost?: boolean;
+};
+type CandidateFinding = {
+    lensId: BroadsideLensId;
+    customId: string;
+    severity: string;
+    title: string;
+    location: string;
+    description: string;
+    pattern: string;
+};
+/** The sync-priced id behind a `:batch` model id (`vendor/name:batch` → `vendor/name`). */
+export declare function syncModelFor(batchModel: string): string;
+/** The findings a run's saved lens results carry, most severe first. */
+export declare function rankVerifiableFindings(stored: StoredLensResult[]): CandidateFinding[];
+export type RepoReader = {
+    readFile(path: string, startLine?: number, endLine?: number): Promise<string>;
+    grep(pattern: string, pathPrefix?: string): Promise<string>;
+    listDir(path: string): Promise<string>;
+};
+/**
+ * Three read-only tools over the repository's own file list — the same
+ * listing the lenses scan (tracked and untracked, ignore rules applied) minus
+ * everything {@link isSlurpable} keeps out of a lens: credential stores, build
+ * output, binaries. A path outside the repository, or one the listing does
+ * not contain, is an error the model sees, not a read.
+ */
+export declare function createRepoReader(cwd: string): Promise<RepoReader>;
+/**
+ * The rubric. The order is deliberate — it is the order a reviewer settles a
+ * claim in — and `not-a-defect` is the verdict that separates "the code does
+ * what the claim says" from "and that is a bug": without it, a cast every
+ * caller satisfies gets confirmed because it is literally there.
+ */
+export declare const BROADSIDE_VERIFY_SYSTEM_PROMPT: string;
+/** Verify one finding: up to the tool budget, then a verdict. */
+export declare function verifyFinding(finding: CandidateFinding, index: number, reader: RepoReader, apiKey: string, model: string, fetcher: FetchLike): Promise<Omit<VerifiedFinding, "index" | "lensId" | "customId" | "severity" | "title" | "location">>;
+/**
+ * Verify the top findings of a collected run against the repository.
+ *
+ * `maxCost` is a running cap, not a pre-flight estimate: a sync call's cost
+ * is only known when it returns, so the pass stops *before* starting the next
+ * finding once the cap is reached and reports `partial`. On the default
+ * model a finding costs about a cent.
+ */
+export declare function runBroadsideVerify(cwd: string, apiKey: string, opts?: {
+    runId?: string;
+    top?: number;
+    model?: string;
+    /** USD; 0 means no limit. */
+    maxCost?: number;
+    fetcher?: FetchLike;
+    signal?: AbortSignal;
+    onProgress?: (finding: VerifiedFinding) => void;
+}): Promise<BroadsideVerifyResult>;
+export declare function renderVerifiedMarkdown(result: BroadsideVerifyResult): string;
+export declare function verifyResultText(result: BroadsideVerifyResult): string;
+export {};