sparkforensics-server 0.6.0 → 0.8.1

package/public/docs/assets/contributor-guide_architecture_impact-estimation_stage-union-rollup.md.JFrzDMwo.js

@@ -0,0 +1 @@
+import{_ as o,o as t,c,a5 as a}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Cross-finding rollup: computeStageUnionMs","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/impact-estimation/stage-union-rollup.md","filePath":"contributor-guide/architecture/impact-estimation/stage-union-rollup.md"}'),i={name:"contributor-guide/architecture/impact-estimation/stage-union-rollup.md"};function n(s,e,d,r,u,l){return t(),c("div",null,[...e[0]||(e[0]=[a('<h1 id="cross-finding-rollup-computestageunionms" tabindex="-1">Cross-finding rollup: <code>computeStageUnionMs</code> <a class="header-anchor" href="#cross-finding-rollup-computestageunionms" aria-label="Permalink to &quot;Cross-finding rollup: `computeStageUnionMs`&quot;">​</a></h1><p>How the recommendation rollup caps a group of findings with the union of their stages.</p><p>The Findings tab&#39;s recommendation rollup (<code>FixTheseFirst.tsx</code>, which renders <code>interpretRun</code>&#39;s <code>rankedRollup</code> over <code>buildRecommendationRollup</code>, <code>packages/core/src/recommendation-rollup.ts</code>) groups the filtered catalog by detector <code>type</code>, then needs its own cap for a group of several findings of that type, not just one finding&#39;s own <code>stageIds</code>. Summing each finding&#39;s already-clipped <code>wallClock.high</code> naively double-counts any stage two of those findings both touch. <code>computeStageUnionMs(stageIds, stages)</code> covers this: collect every stage touched by any finding in the group, merge their <code>[submittedAt, completedAt)</code> intervals, and sum the merged intervals&#39; durations, so the group&#39;s wall-clock union holds regardless of how many findings&#39; <code>stageIds</code> overlap. Stages missing either bound are skipped rather than defaulted to <code>0</code> (the same filter <code>computeWallClock</code> applies), so a truncated log (the case <code>incompleteRun</code> flags) can&#39;t contribute a negative interval and a negative recoverable-time figure.</p><p>It reuses the same <code>mergeIntervals</code> primitive (<code>packages/core/src/intervals.ts</code>) that backs <code>packages/core/src/occupancy.ts</code>&#39;s per-finding <code>estimateMultiStage</code>/its internal <code>unionMs</code> sum, but is not an extension of that function: <code>estimateMultiStage</code> caps one finding&#39;s own multi-stage claim during the impact-estimation pass, before a <code>Finding</code> object even exists; <code>computeStageUnionMs</code> runs later, over the finished findings (in <code>interpretRun</code> and the evidence report), capping a naive sum <em>across</em> several already-estimated findings that happen to share a detector <code>type</code>. <code>buildTimeGroup</code> (same module) takes the smaller of the naive per-finding sum and this union figure as the group&#39;s <code>recoverableMsHigh</code>, falling back to the naive sum untouched when the group&#39;s findings carry no stage IDs at all (nothing to union against).</p><p>Within one <code>type</code>, <code>buildRecommendationRollup</code> splits findings into up to three kinds of group; across the whole rollup every <code>time</code> group sorts before every <code>resource</code> group, which sorts before every <code>count</code> group. <code>time</code> covers findings with a real <code>impactEstimate.wallClock</code> (<code>buildTimeGroup</code>, the union-capped figure above). <code>resource</code> covers findings with no <code>wallClock</code> but a <code>rawWaste</code> figure (<code>buildResourceGroup</code>), grouped again by <code>rawWaste.unit</code> so a <code>bytes</code> total never gets summed against a <code>coreHours</code> total under one type. <code>count</code> covers findings with neither (<code>buildCountGroup</code>), a plain per-impact-band tally with no magnitude claim at all. Each <code>RollupGroup</code> also carries its own <code>findings: Finding[]</code> (the exact members that fed the aggregate), which <code>FixTheseFirst.tsx</code> reads directly to pick a group&#39;s highest-impact member and to render its expanded, paginated list.</p><p>A type only contributes a tier when it has at least one finding of that kind; most types produce exactly one tier, but a type whose formula varies by <code>variant</code>/<code>rule</code> (e.g. <code>memoryUtilization</code>, see the coverage table below) can produce more than one.</p><p><code>cachingOpportunity</code> and <code>cacheUtilization</code> are both <code>cost-only</code>: <code>basis: &#39;resourceOnly&#39;</code>, <code>wallClock: null</code>. <code>cachingOpportunity</code> reports <code>rawWaste</code> in <code>bytes</code> (the input its queries scanned, <code>reread: true</code>). <code>cacheUtilization</code> keeps <code>rawWaste.unit</code> <code>&#39;ms&#39;</code>, the unit a real <code>wallClock</code> figure would use, because its formula&#39;s natural output is time (a re-read cost), not because it makes a wall-clock claim. Left as &quot;ms&quot;, a <code>resource</code>-tier total sitting next to a <code>time</code>-tier &quot;recoverable time&quot; total would read as directly comparable when it isn&#39;t: the resource figure was never gate-clipped against any stage&#39;s occupancy, so it can exceed what the stage actually spent. So every surface prints a cost-only <code>ms</code> figure as core time (<code>asCoreTime()</code> in <code>format-utils.ts</code>, also applied by <code>resourceGroupTotal()</code>), never as a duration. <code>rollupGroupStat</code> (same module) calls this out in the trailing-stat copy <code>FixTheseFirst.tsx</code> renders: a <code>resource</code>-kind group (any unit, including <code>ms</code>) shows the finding count plus the summed raw waste (omitted when it reads as zero), and its tooltip calls the total &quot;a resource cost, not run time&quot;. It never says &quot;recoverable&quot;, so the two ms-shaped numbers are never mistaken for the same kind of claim.</p>',7)])])}const m=o(i,[["render",n]]);export{g as __pageData,m as default};

package/public/docs/assets/contributor-guide_architecture_impact-estimation_stage-union-rollup.md.JFrzDMwo.lean.js

@@ -0,0 +1 @@
+import{_ as o,o as t,c,a5 as a}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Cross-finding rollup: computeStageUnionMs","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/impact-estimation/stage-union-rollup.md","filePath":"contributor-guide/architecture/impact-estimation/stage-union-rollup.md"}'),i={name:"contributor-guide/architecture/impact-estimation/stage-union-rollup.md"};function n(s,e,d,r,u,l){return t(),c("div",null,[...e[0]||(e[0]=[a("",7)])])}const m=o(i,[["render",n]]);export{g as __pageData,m as default};

package/public/docs/assets/contributor-guide_architecture_overview.md.nxkpOhgf.js

@@ -0,0 +1 @@
+import{_ as o,o as t,c as a,a5 as s}from"./chunks/framework.DSg0KOwT.js";const m=JSON.parse('{"title":"Architecture overview","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/overview.md","filePath":"contributor-guide/architecture/overview.md"}'),r={name:"contributor-guide/architecture/overview.md"};function c(d,e,i,n,l,p){return t(),a("div",null,[...e[0]||(e[0]=[s('<h1 id="architecture-overview" tabindex="-1">Architecture overview <a class="header-anchor" href="#architecture-overview" aria-label="Permalink to &quot;Architecture overview&quot;">​</a></h1><h2 id="core-invariant" tabindex="-1">Core invariant <a class="header-anchor" href="#core-invariant" aria-label="Permalink to &quot;Core invariant&quot;">​</a></h2><p>Task data must never live on the main thread. A Spark job can emit millions of <code>SparkListenerTaskEnd</code> events (240 MB+ of metrics); parsing them on the UI thread freezes Chrome. The design below enforces that in both deploy modes: the static zero-backend app, and the optional local-server mode in <code>packages/server/</code>, which also proxies Spark History Server fetches to sidestep CORS.</p><h2 id="two-actors" tabindex="-1">Two actors <a class="header-anchor" href="#two-actors" aria-label="Permalink to &quot;Two actors&quot;">​</a></h2><h3 id="main-thread" tabindex="-1">Main thread <a class="header-anchor" href="#main-thread" aria-label="Permalink to &quot;Main thread&quot;">​</a></h3><p>Owns the DOM (React) and a summary <code>AppModel</code> (jobs, stages, sql, executors, no raw tasks), held in a Zustand store (<code>src/store/store.ts</code>).</p><p><code>src/store/useIngest.ts</code> is the composition root. It owns the ingest client&#39;s lifecycle (<code>createIngestClient</code>, <code>@sparkforensics/core/ingest.ts</code>) and wires <code>model-assembler.ts</code>&#39;s <code>createModelCallbacks</code> (worker message → <code>AppModel</code>) into store updates: <code>onProgress</code> → <code>parse</code>; <code>onDone</code> → derives <code>evidenceAvailability</code> from the final normalized model plus <code>done.skippedLines</code>, runs <code>analyzer.ts</code>, sets <code>catalog</code>, and prefetches flagged-stage task data.</p><p><code>src/App.tsx</code> routes on store state: a two-run compare load, the plan-graph route and the run comparison take precedence; otherwise idle/error → <code>CompareLanding</code> (one or two <code>DropZone</code>s), parsing → a live progress readout (an SHS parse keeps its intake form instead), ready → the lazy-loaded <code>Dashboard</code>. While a log parses, <code>App.tsx</code> starts loading the <code>Dashboard</code> chunk and every <code>REGISTRY</code> widget chunk (<code>src/lib/warm-lazy.ts</code>), so no Suspense fallback shows once parsing finishes: a fallback keeps its content off screen for at least 300 ms. <code>src/view/Dashboard.tsx</code> renders widgets from <code>src/view/detector-registry.tsx</code>&#39;s <code>REGISTRY</code> (see <a href="./widget-rendering.html">Widget rendering</a>).</p><h3 id="worker-web-worker-packages-core" tabindex="-1">Worker (Web Worker, <code>packages/core</code>) <a class="header-anchor" href="#worker-web-worker-packages-core" aria-label="Permalink to &quot;Worker (Web Worker, `packages/core`)&quot;">​</a></h3><p>Owns the file and <code>taskStore: Map&lt;stageId, Float64Array&gt;</code>, keyed by 8-field stride <code>[duration, gcTime, memSpilled, diskSpilled, shuffleRead, shuffleWrite, launchTime, finishTime]</code>. Six modules, one of them the rolling-log helper:</p><ul><li><code>packages/core/src/stage-quantiles.ts</code>, a pure leaf, exporting <code>finalizeStage</code>, <code>computeFieldQuantiles</code>, <code>computeDurationQuantiles</code>, <code>classifySpill</code>, <code>FIELDS</code>, and <code>TASK_FIELD_NAMES</code> for the task-field packed-array layout.</li><li><code>packages/core/src/event-handlers.ts</code>, holding <code>processEvent</code> and the per-event handler functions, <code>accumulateTask</code>, <code>createState</code>, <code>dispatchLine</code>, <code>buildChunkDecoder</code>, <code>emitParseCompletion</code>, <code>collectStageExecutorMetrics</code>, <code>collectLateSpeculationWaste</code>.</li><li><code>packages/core/src/shs-fetch.ts</code>, the SHS zip-fetch/decompress path: <code>runParseFromUrl</code>, <code>decodeShsArchive</code>, <code>parseZipArchive</code>, <code>sniffCodec</code>.</li><li><code>packages/core/src/rolling-log-reassembly.ts</code>, import-free on purpose so the main-thread <code>DropZone</code> can use it: <code>naturalCompare</code>, <code>reassembleRollingEntries</code>.</li><li><code>packages/core/src/zip-archive.ts</code>, the random-access zip reader both zip paths use: <code>isZip</code>, <code>listZipEntries</code>, <code>streamZipEntry</code>.</li><li><code>packages/core/src/parser-worker.ts</code>, a thin entrypoint (<code>streamFile</code>, <code>runParse</code>, <code>runParseFiles</code>, the <code>isWorker</code>/<code>self.onmessage</code> bus) that re-exports the public parts of <code>event-handlers.ts</code>, <code>stage-quantiles.ts</code>, <code>shs-fetch.ts</code> and <code>rolling-log-reassembly.ts</code>. It is the only piece needing the File/Blob streaming API.</li></ul><p>Dropped zstd files also run a nested decompress worker, <code>packages/core/src/zstd-worker.ts</code>, driven from the parse worker by <code>packages/core/src/zstd-worker-client.ts</code> (see <a href="./worker-protocol.html#decompress-worker">Decompress worker</a>).</p><h2 id="streaming" tabindex="-1">Streaming <a class="header-anchor" href="#streaming" aria-label="Permalink to &quot;Streaming&quot;">​</a></h2><p>The worker reads the <code>File</code> in 512 KB slices (smaller for small files, so every file gets at least 100 progress steps) via <code>file.slice(...).arrayBuffer()</code>, decodes each decompressed chunk with one streaming <code>TextDecoder</code>, splits on <code>\\n</code>, JSON-parses and dispatches per event type. Quantiles (P50/P95/max) and spill classification (<code>skew | volume | unclassified</code>) are computed at <code>SparkListenerStageCompleted</code> time, before posting the stage&#39;s <code>{ type: &#39;stage&#39; }</code> message to main.</p><p>Every <code>SparkListenerTaskEnd</code> carries <code>Task Executor Metrics</code> (Spark 3.0+); the parser folds them into a per-executor maximum posted in <code>runAggregates.executorPeakMetrics</code>, skipping all-zero rows (local mode, and any task that ended between two heartbeats, since Spark samples the metrics at the executor heartbeat). When <code>spark.eventLog.logStageExecutorMetrics=true</code> (default <code>false</code>), <code>SparkListenerStageExecutorMetrics</code> events also populate <code>stage.executorMetrics: Map&lt;execId, {...}&gt;</code> with the 23 raw peak-memory/GC fields verbatim (camelCased), consumed by the <code>memoryUtilization</code> detector&#39;s heap band, which takes the larger of the two sources (see <a href="./board-widgets/caching-and-memory.html">Memory Utilization</a>).</p><p>These events can arrive <em>after</em> <code>SparkListenerStageCompleted</code> for the same stage, so the per-stage <code>stage</code> message posted at completion time never carries them. Instead, <code>collectStageExecutorMetrics(state)</code> walks every stage once more just before <code>done</code>, and the worker re-posts a single <code>stageExecutorMetrics</code> message (<code>Map&lt;stageId, Map&lt;execId, metrics&gt;&gt;</code>, empty when the log has no such data). Main-thread consumers get every stage&#39;s metrics without a per-stage race.</p><p>A TaskEnd that arrives after its stage&#39;s StageCompleted is otherwise dropped, since the finalized stage already posted its stats. The exception is the losing copy of a speculative race: Spark kills it only once the stage finishes (&quot;Stage cancelled: Stage finished&quot;), so its TaskEnd is normally late. <code>accumulateTask</code> adds its time to the stage&#39;s <code>speculationWasteMs</code> and <code>speculationWastedAttempts</code> and nothing else, pairing it with a speculative winner kept for that purpose. <code>collectLateSpeculationWaste(state)</code> re-posts the updated totals of those stages once, just before <code>done</code>, as a <code>stageSpeculationWaste</code> message (<code>Map&lt;stageId, { speculationWasteMs, speculationWastedAttempts }&gt;</code>, empty when no speculative attempt ended late). Any other late task of a failed attempt (a zombie or killed task) leaves the stage&#39;s own figures alone and adds to its <code>lateAttemptWork</code>, which only the CLI&#39;s <code>metrics</code> block reads; <code>collectLateAttemptWork(state)</code> re-posts it as a <code>stageLateAttemptWork</code> message (<code>Map&lt;stageId, StageAttemptTotals&gt;</code>). <code>emitParseCompletion</code> sends the speculation totals after <code>runAggregates</code> (whole-run aggregates computed from <code>taskStore</code>) and before the final <code>app</code> and <code>done</code> messages.</p><p>Compressed logs are decompressed as they stream. <code>sniffCodec</code> reads the leading magic bytes: gzip (<code>1f 8b</code>), Zstandard (<code>28 b5 2f fd</code>, Spark&#39;s <code>spark.io.compression.codec=zstd</code>), Spark&#39;s custom <code>LZ4Block</code> framing, or Spark&#39;s Snappy framing (<code>org.xerial.snappy</code>&#39;s <code>\\x82SNAPPY\\0</code> header, <code>spark.io.compression.codec=snappy</code>). Only a zip entry falls back to its name&#39;s suffix (see <a href="#zip-archives">Zip archives</a>). Both the dropped-file path and the SHS-fetch path stream block-by-block (fflate <code>Gunzip</code> / fzstd <code>Decompress</code> / the LZ4Block decoder / the Snappy block decoder) to keep one decompressed chunk live at a time. In the browser, a dropped zstd file decompresses in a second worker while the parse worker parses earlier output, with a three-slice window bounding what is queued between them (see <a href="./worker-protocol.html#decompress-worker">Decompress worker</a>). The Node CLI/MCP paths (<code>collectRun</code> for local files, <code>shs-load.ts</code> for SHS archives) swap fzstd for Node&#39;s native zlib zstd where the running Node has it (22.15+/23.8+), through the <code>zstdDecoder</code> option of <code>runParse</code>/<code>decodeShsArchive</code>: <code>packages/core/src/cli/native-zstd.ts</code> decompresses one frame at a time, since Node&#39;s own decoders stop after a stream&#39;s first frame and Spark writes thousands of small ones. A frame declaring more than 64 MiB of content, one still incomplete after 64 MiB compressed, a malformed one, or a truncated tail goes to fzstd instead. Spark&#39;s frames declare no content size, so each one&#39;s output is capped at 64 MiB as it decodes: past that, the inline decoder hands the frame to fzstd and the threaded one streams it, pausing while 64 MiB wait to be parsed. For local files (<code>nodeParseCodecs</code>, <code>createThreadedZstdDecoder</code>), frames of 64 KB or more compressed decompress on libuv&#39;s threadpool, up to 4 at a time, while the main thread parses earlier output; <code>streamFile</code> awaits each <code>push</code>, and chunks still arrive in stream order. SHS archives fetched from a History Server keep the inline decoder (<code>nodeArchiveCodecs</code>).</p><p>Rolling <code>eventlog_v2_*</code> directories (Spark&#39;s multi-file event-log format, <code>events_&lt;index&gt;_&lt;appId&gt;(.codec)</code> files plus a zero-byte <code>appstatus_*</code> completion marker and, periodically, one <code>*.compact</code> merge file) are reassembled into parse order by <code>reassembleRollingEntries(names)</code>: drop the marker, drop every non-compact <code>events_*</code> file at or below the most recent <code>.compact</code> file&#39;s index, sort the rest numerically, and throw if an index is missing. The zip path (<code>parseZipArchive</code>) calls it on member names; for a dropped folder, <code>DropZone</code> (and the CLI&#39;s <code>collectRun</code>) call it before handing the ordered files to <code>runParseFiles</code>.</p><p><code>runParseFiles</code> streams each file in the resulting order through the same chunked codec dispatch as <code>runParse</code>, with a fresh decompressor per file, since codecs never span a roll boundary. The NDJSON line-decoder is the exception: it stays alive across all files, because nothing guarantees a roll boundary lands on a line boundary.</p><h2 id="zip-archives" tabindex="-1">Zip archives <a class="header-anchor" href="#zip-archives" aria-label="Permalink to &quot;Zip archives&quot;">​</a></h2><p>A History Server download is a zip, whether fetched by <code>runParseFromUrl</code> or dropped as a file (the Spark UI&#39;s download link, or <code>GET /api/v1/applications/&lt;appId&gt;/logs</code> saved to disk). Both go through <code>parseZipArchive</code> (<code>packages/core/src/shs-fetch.ts</code>): <code>runParse</code> checks for the zip signature before its codec dispatch, and <code>decodeShsArchive</code> wraps the downloaded bytes in the same <code>slice()</code> interface a <code>File</code> has. The two differ only in how they report failure (a plain <code>error</code> message for a dropped file, <code>{ source: &#39;shs&#39;, code: &#39;invalid-event-log&#39; }</code> for a fetch).</p><p><code>packages/core/src/zip-archive.ts</code> reads the central directory from the archive&#39;s tail, then streams one entry at a time through fflate&#39;s <code>UnzipInflate</code> in <code>chunkSize</code> slices. A dropped file is read a slice at a time, and no decompressed entry is ever held whole (the SHS fetch still buffers its download before parsing). It avoids fflate&#39;s forward-scanning <code>Unzip</code> on purpose: the History Server writes its zip with Java&#39;s <code>ZipOutputStream</code>, which leaves each local header&#39;s sizes blank and appends a data descriptor, so <code>Unzip</code> would have to find each entry&#39;s end by scanning compressed bytes for a signature, and it would hand rolling parts over in archive order (<code>events_10</code> before <code>events_2</code>). A single-file log is one entry at the root. A rolling log&#39;s parts sit under an <code>eventlog_v2_&lt;appId&gt;/</code> directory entry and are matched by base name. The zip must hold a single application attempt. Without an attempt ID the History Server packs every attempt into one download (several root entries, or several <code>eventlog_v2_</code> directories), and <code>parseZipArchive</code> rejects that before parsing with the attempt count and a pointer to <code>GET /api/v1/applications/&lt;appId&gt;/&lt;attemptId&gt;/logs</code>.</p><p>Each entry&#39;s bytes then go through the codec&#39;s streaming decoder, sniffed from the entry&#39;s first bytes, with the entry name&#39;s suffix as fallback. One-shotting the decompression of an entry (fzstd&#39;s <code>decompress()</code>, fflate&#39;s <code>gunzipSync()</code>) would allocate its full declared content size as one ArrayBuffer, which fails outright for a multi-GB event log. Decompressors are vendored under <code>packages/core/src/vendor/</code> (<code>fflate.js</code>, <code>fzstd.js</code>) plus <code>packages/core/src/lz4-block.ts</code> and <code>packages/core/src/snappy-block.ts</code>.</p>',24)])])}const f=o(r,[["render",c]]);export{m as __pageData,f as default};

package/public/docs/assets/contributor-guide_architecture_state-and-history_run-comparison-internals.md.U-9KMDFK.js

@@ -0,0 +1 @@
+import{_ as o,o as t,c as a,a5 as n}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Run comparison internals","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/state-and-history/run-comparison-internals.md","filePath":"contributor-guide/architecture/state-and-history/run-comparison-internals.md"}'),s={name:"contributor-guide/architecture/state-and-history/run-comparison-internals.md"};function d(i,e,c,r,l,h){return t(),a("div",null,[...e[0]||(e[0]=[n('<h1 id="run-comparison-internals" tabindex="-1">Run comparison internals <a class="header-anchor" href="#run-comparison-internals" aria-label="Permalink to &quot;Run comparison internals&quot;">​</a></h1><p>How the A/B comparison engine matches stages and builds a comparison.</p><h2 id="run-comparison" tabindex="-1">Run comparison <a class="header-anchor" href="#run-comparison" aria-label="Permalink to &quot;Run comparison&quot;">​</a></h2><p><code>packages/core/src/run-comparison.ts</code> is the whole A/B engine. The entry point <code>compareRuns(baseline, candidate)</code> takes two <code>{ label, snapshot }</code> run records (each <code>snapshot</code> a <code>SessionSnapshot</code>: <code>app</code>, <code>stages</code>, <code>executors</code>, <code>sql</code>, <code>jobs</code>, <code>runAggregates</code>, <code>evidenceAvailability</code>, <code>catalog</code>, <code>taskData</code>) and returns one plain object the view renders. It runs on already-parsed snapshots, with no worker involved.</p><ul><li><p><code>stageIdentity(stage, snapshot)</code> is a run-independent key: <code>normalizeStageName</code> (lowercased, digit-runs and long hex ids collapsed to <code>#</code>) joined with the stage&#39;s SQL-execution plan identity. That identity is scoped to only the plan nodes this stage&#39;s tasks were attributed to (<code>node.stageIds</code>), not the whole tree, so two stages sharing one SQL execution (e.g. a self-join&#39;s two Exchange stages) don&#39;t collapse onto one identity; it falls back to a bottom-up structural fingerprint of the whole resolved <code>planTree</code> (<code>planTreeIdentity</code>, <code>normalizeDetail</code>-normalized: the same normalizer <code>cachingOpportunity</code> uses in <code>packages/core/src/detectors.ts</code>) when a stage has no such attribution. The stage-to-plan mapping itself is <code>planNodesOfStage</code> in <code>packages/core/src/stage-plan-nodes.ts</code>, shared with <code>isPythonStage</code> (<code>python-stage.ts</code>), which the CLI&#39;s <code>metrics</code> block (<code>run-metrics.ts</code>) and the <code>tasksMostlyIdle</code> check both read. The CLI keys its per-stage metrics rows by this identity. <code>matchStages(baseSnap, candSnap)</code> indexes each run by that identity and pairs identities that map to exactly one stage on both sides. An identity colliding equally on both sides (same count) is also paired, positionally by sorted stage id: exact when comparing a run against itself (every stage matches itself), a best-effort guess otherwise (two unrelated same-named stages with no SQL/attribution could get cross-paired). Collisions are still recorded in <code>collisionIdentities</code> even when resolved this way; a differing count leaves them there unpaired. <code>coverage</code> reports the matched fraction.</p></li><li><p><code>metricDeltas</code> computes whole-run aggregate deltas (wall-clock, spill, task skew p95, failed-task rate, GC, I/O bytes, executor count, ...) as plain sums over all stages, deliberately not gated on stage matching, since matching is unreliable on real logs. Each metric carries a <code>direction</code> (improvement/regression/unchanged) and an <code>unavailableReason</code> when a side lacks the field.</p></li><li><p><code>findingsDelta</code> pairs the two runs&#39; <code>catalog</code> findings by <code>findingKey</code> in <code>packages/core/src/finding-identity.ts</code>: type, rule and location, never the value or the impact band. The location is the stage&#39;s <code>pairId</code> from <code>alignStages</code>, the SQL execution that holds most of the paired stages, the audited config property, or the whole run; a stage or execution with no pair gets one side-neutral label, so the same finding on an unpaired stage in both runs still matches. Findings that share a key pair one-to-one, same band first. The unpaired candidate findings are <code>introduced</code>, the unpaired baseline findings are <code>resolved</code>, and a pair whose bands differ is <code>changed</code>. Each row groups findings by rule and band. The verdict, the <code>fail-on-introduced</code> budget and the dashboard read this one diff. A <code>changed</code> pair whose band worsens to exactly the budgeted band also counts for the budget, and any worsened band adds a &quot;More severe in the candidate&quot; verdict sentence.</p></li><li><p><code>alignStages(baseSnap, candSnap, { normalizePath })</code> in <code>packages/core/src/stage-alignment.ts</code> is the aligner: a pure function from two snapshots to the stage pairs, the unmatched stages, the replanned groups, the Delta bookkeeping stages, the runtime coverage and the execution alignment. <code>compareRuns</code> calls it and does not contain it, and every consumer of a comparison reads its <code>stagePairs</code> instead of comparing stage keys. It pairs in two levels.</p><ul><li>Keys. The exact key is <code>comparisonIdentity</code>, the <code>stageIdentity</code> recipe (shared in <code>packages/core/src/stage-identity.ts</code>) run with the comparison normalizer: <code>normalizeDetail</code>, then the targeted patches in <code>COMPARISON_PATCHES</code> (staging directories, dates, <code>IN</code> lists, file-index counts), with any caller-supplied patterns (<code>compileNormalizePatterns</code>) applied first to the text as Spark printed it. The structural key (<code>structuralKey</code> in <code>packages/core/src/stage-structure.ts</code>) folds the attributed nodes into a forest, each node as its normalized name, its sorted attribute names (<code>name#id</code> references without the id, so a literal never enters it) and its children&#39;s digests; codegen and input-adapter nodes are skipped. <code>stageIdentity</code> stays the exact key the CLI&#39;s per-stage metrics rows use, and <code>normalizeDetail</code> stays untouched, so findings do not move. Blanket numeric or path stripping is not part of the normalizer: it fused distinct stages of one run.</li><li>Executions first. <code>indexRun</code> groups each run&#39;s non-bookkeeping stages by SQL execution (stages with no execution form one pseudo-execution) and gives each execution its plan structure, call-site frames and description tokens. <code>alignSequences</code> (<code>packages/core/src/sequence-alignment.ts</code>) runs the order-preserving weighted LCS over the executions in id order, scored by <code>executionSimilarity</code> (weights 4, 2 and 1 for structure, frames and description; a part one side lacks drops out) with <code>EXECUTION_MATCH_MIN</code> 0.55. It prefers matching where matching is optimal and otherwise skips the execution with the higher id, which makes the result deterministic and mirror-symmetric. Up to <code>FULL_ALIGNMENT_MAX_CELLS</code> (1,000,000) the table is full; above it the table is a band of half-width <code>BAND_BASE_HALF_WIDTH</code> (100) plus half of the length difference, and <code>executionAlignment.bounded</code> says so. If fewer than <code>MIN_EXECUTION_AGREEMENT</code> (0.5) of the smaller run&#39;s executions pair, no stage pairs (<code>executionAlignment.accepted</code> is false).</li><li>Stages second (<code>alignWithin</code>). In each aligned execution pair: equal exact key (<code>exact</code>, pairing off in stage-id order as many as both sides have), equal structural key (<code>structural</code>), detail-token Jaccard of at least <code>ALIGNED_MIN_SIMILARITY</code> (0.6, best pair first; <code>aligned</code>), and plan-less stages by position among those of one name (<code>aligned</code>, <code>POSITIONAL_SCORE</code>). Stages outside any execution key on their call-site text too.</li><li>Re-plan fallback. When an aligned SQL execution pair holds different numbers of non-bookkeeping stages and stages are left over, they form one <code>ReplannedGroup</code> (both execution ids, the leftover ids per side, <code>deltas</code> totalled per side). Equal counts leave them unmatched. With <code>REPLANNED_COUNTS_TOWARD_COVERAGE</code> the replanned run time counts in the numerator of <code>runtimeCoverage</code>; set it to false and it counts in the denominator only.</li></ul><p>Stages whose scans only read the Delta log (<code>isDeltaBookkeepingStage</code>, built on <code>isDeltaLogRead</code> in <code>plan-summary.ts</code>) are set aside before pairing. A pair&#39;s <code>deltas</code> sum every attempt of the stage, the sums <code>withEarlierAttempts</code> gives the run totals. The result is <code>comparisonSchemaVersion</code> 1.</p></li><li><p>The per-stage skew deltas (<code>stageSkewDeltas</code>) read the aligner&#39;s pairs, the same ones the banner&#39;s coverage figure counts, and the pinned-stage panel reads <code>baseStages</code>/<code>candStages</code>. The old <code>matchStages</code> pairs still feed <code>matchedCoverage</code>, which keeps its count-based meaning and decides nothing.</p></li></ul><p><code>compareRuns</code> sets <code>confidence</code> from <code>runtimeCoverage</code>: <code>insufficient</code> when it is null (neither run recorded executor run time), <code>low</code> when it is under 0.9 (<code>RUNTIME_COVERAGE_THRESHOLD</code>) or the two app names differ (either condition alone is enough; both are weak signals, not a hard gate), <code>ok</code> otherwise. It is computed in <code>compareRuns</code>, not in <code>buildComparison</code>, so the dashboard, which calls <code>compareRuns</code> directly, gets the same result as the CLI and MCP. The view lives in <code>src/view/RunComparison.tsx</code> (the comparison page), <code>CompareLanding.tsx</code> (the two-slot baseline / candidate intake off the landing), and <code>PinnedStageDeltas.tsx</code> (the manual per-stage pinning panel fed by <code>baseStages</code>/<code>candStages</code>).</p>',6)])])}const g=o(s,[["render",d]]);export{u as __pageData,g as default};

package/public/docs/assets/{contributor-guide_architecture_state-and-history_run-comparison-internals.md.CZA0JLtV.lean.js → contributor-guide_architecture_state-and-history_run-comparison-internals.md.U-9KMDFK.lean.js}

@@ -1 +1 @@
-import{_ as o,o as t,c as a,a5 as s}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Run comparison internals","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/state-and-history/run-comparison-internals.md","filePath":"contributor-guide/architecture/state-and-history/run-comparison-internals.md"}'),n={name:"contributor-guide/architecture/state-and-history/run-comparison-internals.md"};function i(c,e,d,r,l,h){return t(),a("div",null,[...e[0]||(e[0]=[s("",6)])])}const g=o(n,[["render",i]]);export{u as __pageData,g as default};
+import{_ as o,o as t,c as a,a5 as n}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Run comparison internals","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/state-and-history/run-comparison-internals.md","filePath":"contributor-guide/architecture/state-and-history/run-comparison-internals.md"}'),s={name:"contributor-guide/architecture/state-and-history/run-comparison-internals.md"};function d(i,e,c,r,l,h){return t(),a("div",null,[...e[0]||(e[0]=[n("",6)])])}const g=o(s,[["render",d]]);export{u as __pageData,g as default};

package/public/docs/assets/contributor-guide_architecture_widget-rendering.md.BrSz8Ezh.js

@@ -0,0 +1 @@
+import{_ as o,o as t,c as d,a5 as n}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Widget rendering","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/widget-rendering.md","filePath":"contributor-guide/architecture/widget-rendering.md"}'),a={name:"contributor-guide/architecture/widget-rendering.md"};function i(c,e,s,r,l,h){return t(),d("div",null,[...e[0]||(e[0]=[n('<h1 id="widget-rendering" tabindex="-1">Widget rendering <a class="header-anchor" href="#widget-rendering" aria-label="Permalink to &quot;Widget rendering&quot;">​</a></h1><h2 id="widget-rendering-order" tabindex="-1">Render order <a class="header-anchor" href="#widget-rendering-order" aria-label="Permalink to &quot;Render order {#widget-rendering-order}&quot;">​</a></h2><p><code>src/view/Dashboard.tsx</code>&#39;s <code>FilteredBoard</code> renders inside a <code>&lt;main&gt;</code> that opens with <code>SampleRunNotice</code> (only while the bundled sample run is open), then <code>RunVerdict</code> (built from the unfiltered catalog), then a single <code>Scorecard</code> strip, then <code>FindingFilterBar</code> (Advanced view, or any active filter) and (only when an active filter empties both finding streams) <code>NoMatchBanner</code>, then (when the active filter doesn&#39;t empty the board) a two-tab <code>Tabs</code> (<code>src/components/ui/tabs.tsx</code>, a base-ui primitive): <strong>Findings</strong> and <strong>Full app report</strong>, the first grouped by impact band and the second always reachable. Both <code>TabsContent</code> panels pass <code>keepMounted</code>, so the inactive one is hidden (native <code>hidden</code> attribute) rather than unmounted: switching tabs keeps each widget&#39;s local state (e.g. ImpactBoard&#39;s expanded group) and scroll position.</p><p><code>region</code> on <code>RegistryEntry</code> (<code>src/view/detector-registry.tsx</code>) is read only to decide whether a widget always mounts: <code>isAlwaysMountedType()</code> flags the one <code>reference</code>-region type carved out as an always-mounted exception, <code>coreLocality</code> → <code>CoreUsageArea</code>. The other <code>reference</code>-region types stay ordinary finding-gated instead: <code>cacheUtilization</code>, <code>memoryUtilization</code>, and <code>utilization</code> are a product decision: a clean run on any of them isn&#39;t evidence worth surfacing unconditionally, so each collapses to a plain clean-check line like any other detector. The Findings tab (<code>src/view/widgets/ImpactBoard.tsx</code>) groups its content into two tiers: impact-banded rows and cards for every <code>REGISTRY</code> component and every recommendation row with at least one finding; and a collapsed &quot;Clean checks&quot; disclosure covering every remaining type with zero findings, built per detector <em>type</em> (<code>Object.keys(REGISTRY)</code>). The one always-mounted exception lives in Full app report (see &quot;ReferenceSection&quot; below), which stays structural-only.</p><p>Tags carry their own docs links; there is no separate legend widget. <code>TagBadge</code> (<code>src/view/ImpactBadge.tsx</code>) resolves its own tooltip. Its documentation anchor is the caller&#39;s <code>docAnchor</code> prop when that is a known anchor (call sites holding the finding pass <code>finding.docAnchor</code>; a widget header or grouped row passes <code>sharedDocAnchor(findings)</code>), else the type&#39;s single known anchor (the run interpretation&#39;s <code>detectors[type].docAnchor</code>, from <code>docAnchorForType</code> in <code>packages/core/src/detector-docs.ts</code>). The prop matters for <code>configAudit</code>, whose four entries carry different anchors, so the type lookup finds none. With an anchor, the pill itself links into the docs panel, and (density <code>advanced</code> only) a second icon link opens that tag&#39;s entry in <code>docs-site/user-guide/understanding-findings.md</code> (<code>findingGuideUrl</code>, <code>packages/core/src/docs-site-config.ts</code>) in the same in-app docs panel; its <code>target=&quot;_blank&quot;</code> only applies to a modifier-click or when no <code>DocsProvider</code> is mounted. When a type has no vendor-doc anchor (e.g. <code>incompleteRun</code>), the pill links straight to that same guide entry instead of rendering as inert text, and the second icon link is skipped as redundant. Either way, a linked pill gets a visible dotted underline, not just a hover tooltip. <code>TagBadge</code>&#39;s <code>plainBadge</code> prop suppresses both links together; Stage Summary&#39;s per-stage tag chips are the one caller that sets it, on a chip that sits inside its own investigate button. Findings-tab rows leave it unset, so their badge links stay real <code>&lt;a&gt;</code>s. The HTML export renders no badge links at all (<code>exportMode</code>).</p><h3 id="widget-placement" tabindex="-1">Widget placement <a class="header-anchor" href="#widget-placement" aria-label="Permalink to &quot;Widget placement&quot;">​</a></h3><p>The Plan Explorer (<code>src/view/widgets/PlanExplorer.tsx</code>) is embedded inside a flagged stage&#39;s own row, not as a standalone widget. It has four call sites (<code>ShuffleIO.tsx</code>, <code>PartitionSizing.tsx</code>, <code>Spill.tsx</code>, and the shared <code>StageFindingGroup.tsx</code> row behind <code>Skew.tsx</code>/<code>StageShape.tsx</code>/<code>TinyTask.tsx</code>), each Advanced-density only and only when the stage&#39;s plan tree resolves. The first three have no row-expansion toggle, so the trigger renders as a direct sibling in the row body; <code>StageFindingGroup.tsx</code> renders it inside the expanded Task detail. The same stage&#39;s plan is also reachable from <code>StageDetailDialog</code>.</p><p>Plan detection logic belongs in a <code>packages/core/src/detectors.ts</code> entry (<code>scope:&#39;sql&#39;</code>) consuming <code>planTree</code>. <code>packages/core/src/plan-summary.ts</code> is display-only summarization; do not add detection heuristics there. It is best-effort: <code>summarizePlanTree</code> reads each resolved <code>planTree</code> node&#39;s <code>detail</code> with lenient regexes and silently omits a fragment it can&#39;t parse, never surfacing an error.</p><p><code>StageHeader.tsx</code> (<code>StagePill</code> plus a <code>&lt;span&gt;</code> naming the stage) vs. bare <code>StagePill</code>/<code>StagePillGroup</code> is a deliberate row-density choice, not an inconsistency: <code>StageHeader</code> is for single-stage-per-row detail widgets where naming the stage adds context (<code>SlowHost.tsx</code>, <code>StageSlowness.tsx</code>, <code>Straggler.tsx</code>, <code>SpeculationWaste.tsx</code>, <code>GcPressure.tsx</code>, <code>Skew.tsx</code>, <code>StageShape.tsx</code>, <code>TinyTask.tsx</code>, <code>PartitionSizing.tsx</code>), while bare <code>StagePill</code>/<code>StagePillGroup</code> is for compact, multi-row lists where many stages appear per widget (<code>StageFailed.tsx</code>, <code>TaskFailures.tsx</code>, <code>RetryWaste.tsx</code>, the six Plan Advisor widgets, <code>Spill.tsx</code>, and <code>StageTable.tsx</code>). <code>ShuffleIO.tsx</code> uses both in different parts of its own row, which is fine: it isn&#39;t a violation of the convention above.</p><p>Stage Summary Table (<code>src/view/widgets/StageTable.tsx</code>) defaults to the <strong>top 10 stages by duration</strong>, with a header toggle to <strong>problems only</strong> (stages carrying a finding). It renders inside the Full app report tab, not as a standalone board section.</p><h3 id="full-render-sequence" tabindex="-1">Full render sequence <a class="header-anchor" href="#full-render-sequence" aria-label="Permalink to &quot;Full render sequence&quot;">​</a></h3><p>Top to bottom, in <code>Dashboard.tsx</code>&#39;s <code>FilteredBoard</code>:</p><ol><li><code>SampleRunNotice</code> (<code>src/view/SampleRunNotice.tsx</code>), only while the bundled sample run is open (the landing&#39;s <strong>Try a sample run</strong> loads it under <code>SAMPLE_RUN_ID</code>, <code>src/view/sample-run.ts</code>) and never in the export bundle: says the board shows the sample, with <strong>Load my event log</strong> and a docs-panel link to the log-retrieval guide.</li><li><code>RunVerdict</code> (<code>src/view/widgets/RunVerdict.tsx</code>): the run&#39;s verdict title, a summary sentence, and up to three numbered next steps built by <code>buildRunVerdict</code> (<code>packages/core/src/run-verdict.ts</code>, the same code the CLI/MCP evidence report&#39;s <code>verdict</code> runs) via <code>buildNextSteps</code>. Steps group routeable eligible findings by location (one stage, one multi-stage finding type, or one app-level finding type and variant), ordered by <code>rankBySavings</code> (potential savings, then impact band, then widget order), the same ranking every other component uses; on a run with a failed job, failure steps (<code>stageFailed</code>, <code>jobFailureRate</code>) move ahead (see below), and no other type jumps the order. Each step shows its tag, action label, stage and savings, then &quot;What&#39;s happening:&quot; (the measurement) and &quot;What to try:&quot; (the fix), split from the recommendation by core <code>recommendationParts</code>; a step that merged other types at its location adds &quot;Also flagged here, likely the same cause: ...&quot;, and the summary says once that same-stage findings are grouped. A step&#39;s stage label opens its stage details, and its one <strong>Show evidence</strong> expands the evidence card the step owns in place (<code>verdictEvidenceOwners</code> in <code>src/view/verdict-evidence.ts</code>, rendered through <code>InlineEvidence</code>) or routes to the card that holds it. The steps&#39; lead findings (<code>verdictLeadFindings</code>) and owned cards are left out of the Findings tab, and a <strong>More findings (n)</strong> link shows that tab, with <code>n</code> the tab&#39;s own listed count (<code>listedCount</code> in <code>Dashboard.tsx</code>). Under the steps, <code>CostLane</code> lists <code>verdict.costSteps</code> (&quot;By core time&quot;): findings ranked by the measured busy core time a fix removes, including findings with no wall-clock claim. The run&#39;s idle line (<code>verdict.idle</code>) sits under the summary. Basic view adds a collapsed &quot;New to Spark tuning?&quot; primer. An idle-capacity step (<code>utilization</code>, or <code>memoryUtilization</code>&#39;s <code>idleCores</code> variant only, never its heap variants) titles the verdict only when it ranks first; otherwise an idle share of at least <code>IDLE_NOTABLE_PCT</code> (40%) adds one summary sentence. The idle share (<code>verdictIdlePct</code>) is the figure that idle-capacity step itself reports, falling back to the Scorecard&#39;s Unused core time figure only when no step carries one, so the title and the step never disagree. Each step&#39;s savings figure is followed by what it counts (<code>savingsMeaning</code> in <code>packages/core/src/impact-format.ts</code>: run time for a wall-clock figure, otherwise the resource its <code>rawWaste</code> unit measures), and <strong>Copy next steps</strong> copies the whole verdict as a plain-text checklist (<code>planCopyText</code> in core <code>run-verdict.ts</code>). Always the unfiltered catalog: a board filter never changes the verdict. A step&#39;s <strong>Show evidence</strong> on a finding the active filter hides clears only the filter dimensions that hide it (<code>excludingDimensions</code> in <code>src/view/finding-filter.ts</code>, synced to the URL as usual) and shows a one-line notice naming what it cleared; the Topbar count chip&#39;s jump to its impact band (<code>jumpToFindings</code> in <code>Dashboard.tsx</code>) uses the same path. The clean-run message (&quot;No findings to fix right now.&quot;) lives here and shows only when no finding at all was emitted and the log lacked nothing a check needs (below); an <code>incompleteRun</code> finding always adds a sentence saying the figures cover only the captured part of the run, and titles the verdict (&quot;This log looks incomplete ...&quot;) when no other finding is eligible. Job results (<code>summarizeRunOutcome</code> in <code>packages/core/src/run-outcome.ts</code>) set the run outcome: with a failed job the title says the run failed, the verdict quotes the first line of Spark&#39;s recorded reason (a failed job&#39;s <code>stageFailed</code> value first, then any <code>stageFailed</code>, then the job exception), the run is never called clean, and <code>buildNextSteps</code> ranks <code>stageFailed</code> and <code>jobFailureRate</code> steps first (a failed job&#39;s stage leading). Evidence caveats (a finding with <code>dataUnavailable</code>, or one <code>isRealFinding</code> drops), a log with no finished stage, and an <code>incompleteRun</code> log (whose <code>RUN_SPAN_CHECK_TYPES</code> had no run length to measure) are gaps (<code>verdictGaps</code>, <code>packages/core/src/check-coverage.ts</code>): any gap keeps the run from being called clean, and a log with no finished stage and no finding gets its own title. The verdict card does not list the gaps; the Clean checks disclosure&#39;s &quot;Not checked on this log&quot; group does, each caveat by its own recommendation text, which names the setting to enable. In Advanced view each step adds an &quot;Estimate:&quot; line from <code>estimateProvenance</code> (<code>packages/core/src/impact-format.ts</code>, carried as the interpretation&#39;s per-finding <code>savings.provenance</code>: method, basis as a point figure or a floor-to-high range, ms raw waste only when the floor clipped it, a non-time raw waste as the resource measured; nothing for <code>estimateMethod: &#39;none&#39;</code> or a zero figure), the finding&#39;s <code>confidence</code> when not <code>high</code>, and the list ends with the ordering rule when it has more than one step.</li><li><code>Scorecard</code>: a three-tile run-info row (Wall-clock, Efficiency, Unused core time; Basic view captions say what each measures, and for Efficiency and Unused core time which direction is better; Advanced view shows the raw run/idle breakdown), rendered once regardless of which tab is active.</li><li><code>FindingFilterBar</code>, only in Advanced view or while a filter is active (plus <code>NoMatchBanner</code> when the active filter empties both finding streams).</li><li>A two-tab <code>Tabs</code> (skipped entirely when the active filter empties both finding streams; <code>NoMatchBanner</code> above already covers that case), tab labels <strong>Findings</strong> and <strong>Full app report</strong>: <ul><li><strong>Findings</strong> (<code>ImpactBoard</code>): one <code>&lt;section&gt;</code> per impact band in <code>Critical</code> → <code>Warning</code> → <code>Info</code> order, each rendering nothing when it has neither a recommendation row nor an active widget: a recommendation-rollup <code>Table</code> (one row per eligible-finding type, a display type, <code>incompleteRun</code> and evidence caveats excluded: a direct row for a type with one finding, an expandable, paginated summary row for a type with more than one) followed by a <code>WidgetGrid</code> of that band&#39;s active <code>REGISTRY</code> widgets (ranked by widget order: the interpretation&#39;s <code>DetectorInfo</code> by region, action-region components ahead of reference-region ones, then ascending detector order); then, below every impact band, a collapsed &quot;Clean checks&quot; disclosure of <code>CleanCheckRow</code> lines built per detector type (every remaining <code>REGISTRY</code> key with zero findings).</li><li><strong>Full app report</strong> (<code>ReferenceSection</code>): WallClock → Timeline → Executor Count Over Time → StageTable → a grid of Core Usage by Locality → Evidence availability → fixed report-lens tail (ETL Phase Attribution → What-If Executor Scaling → Compute Efficiency → Core-Usage Distribution). Structural-only, as above. Hidden (not unmounted) while Findings is active (see &quot;Render order&quot; above).</li></ul></li></ol><p>Report lenses carry no impact-band chip and self-hide when their input data is absent. Every widget self-wraps in <code>WidgetCard</code> (<code>src/view/WidgetCard.tsx</code>: an <code>&lt;h3&gt;</code> title, one level below the board&#39;s <code>&lt;h2&gt;</code> section headers) for the document outline, except Scorecard, which renders its own header band, and RunVerdict, a plain <code>&lt;section&gt;</code> with its own <code>&lt;h2&gt;</code> title.</p><p>Every card is collapsible: the title sits in a <code>CollapsibleTrigger</code> button with a chevron (<code>WidgetCard.tsx</code>), and each widget sets its own <code>defaultCollapsed</code> (most default to <code>true</code>; <code>ImpactBoard</code>&#39;s action-region grid forces <code>defaultCollapsed</code> on every <code>REGISTRY</code> widget instance it mounts, so a marginal finding doesn&#39;t default open). Route navigation (&quot;jump to this finding&quot;) goes through the grid coordinator instead of a fixed <code>tabIndex</code>: <code>WidgetGrid.tsx</code> tracks each card&#39;s disclosure button and bumps <code>openRequestGeneration</code> to force a collapsed card open and focus its trigger when the target finding has no anchored row of its own. How much a card shows beyond that is a board-wide choice, set by the Basic/Advanced density tier in the topbar.</p>',15)])])}const u=o(a,[["render",i]]);export{p as __pageData,u as default};

package/public/docs/assets/contributor-guide_architecture_widget-rendering_findings-tab.md.B7rjSF_h.js

@@ -0,0 +1 @@
+import{_ as o,o as t,c as d,a5 as n}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Findings tab","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/widget-rendering/findings-tab.md","filePath":"contributor-guide/architecture/widget-rendering/findings-tab.md"}'),a={name:"contributor-guide/architecture/widget-rendering/findings-tab.md"};function c(i,e,r,s,l,p){return t(),d("div",null,[...e[0]||(e[0]=[n('<h1 id="findings-tab" tabindex="-1">Findings tab <a class="header-anchor" href="#findings-tab" aria-label="Permalink to &quot;Findings tab&quot;">​</a></h1><p>How the Findings tab renders recommendation rows and active widget cards in one impact-ranked board.</p><p>Rendered by <code>FilteredBoard</code>&#39;s <code>TabsContent value=&quot;findings&quot;</code> (<code>src/view/widgets/ImpactBoard.tsx</code>): recommendation rows and active widget cards in one impact-ranked board. The recommendation-rollup logic lives in <code>src/view/widgets/FixTheseFirst.tsx</code> (its own file with directly-testable exports, not rendered as a standalone page section by <code>Dashboard.tsx</code>) and the active-widget logic lives in <code>src/view/widgets/Alerts.tsx</code> (same: not rendered standalone). <code>ImpactBoard</code> calls <code>FixTheseFirst.tsx</code>&#39;s exported <code>useFixTheseFirstData</code> (eligible findings, rollup groups, the top triage target) and <code>Alerts.tsx</code>&#39;s exported <code>computeActiveWidgets</code> (the ranked active-<code>REGISTRY</code> list). The rollup and its ranking are computed in core by <code>interpretRun</code> (<code>rankedRollup</code>/<code>rankFindings</code>, <code>packages/core/src/recommendation-rollup.ts</code>); <code>useFixTheseFirstData</code> reads it through <code>boardRollup</code> (<code>src/view/interpretation.ts</code>: the interpretation&#39;s carried groups when unfiltered, core <code>rankedRollup</code> recomputed over the kept findings under a filter). <code>ImpactBoard</code> then buckets rows by each group&#39;s <code>band</code> (its representative member&#39;s impact band, after core <code>rankFindings</code>, the same finding whose impact band its own badge already shows) and cards by each active widget&#39;s own <code>worstImpactBand</code>. The verdict&#39;s step leads (<code>verdictLeads</code>) are dropped before the rollup and the cards the verdict shows in place (<code>verdictWidgetIds</code>) before <code>computeActiveWidgets</code>&#39;s list is bucketed, so with any step lead the board opens with a &quot;More findings (N)&quot; <code>&lt;h2&gt;</code> (N is the remaining eligible count) or, when N is 0, &quot;No more findings to show.&quot; Caveat cards such as <code>IncompleteRun</code> never count toward N and still render under their own titles below it.</p><p>Eligible findings for the rollup are <code>catalog</code> ∪ <code>configFindings</code> that pass core <code>isEligible</code> (<code>packages/core/src/recommendation-rollup.ts</code>) and have a display type (<code>FINDING_DISPLAY_ORDER</code>): <code>incompleteRun</code> (a pipeline-completeness caveat, not an addressable fix) and every <code>dataUnavailable</code> evidence caveat (e.g. <code>memoryUtilization</code>&#39;s <code>memoryBand</code> variant, already covered by Evidence availability&#39;s own <code>executorMetrics</code> entry, <code>packages/core/src/evidence-availability.ts</code>) are excluded. Findings are grouped strictly by <code>finding.type</code> via <code>buildRecommendationRollup</code> (<code>packages/core/src/recommendation-rollup.ts</code>, further split within a type by impact kind and, for <code>resource</code>, unit: never merged across types); each resulting group becomes one row: a type with exactly one finding renders that finding directly, a type with more than one collapses into a summary row. Within an impact band, group order comes from <code>buildRecommendationRollup</code>&#39;s own sort: <code>time</code> groups (a real <code>wallClock</code> claim) first, ranked among themselves by their union-capped <code>recoverableMsHigh</code> descending; then <code>resource</code> groups (<code>rawWaste</code> but no <code>wallClock</code>); then <code>count</code> groups (neither), both of the latter two ranked by worst impact band, never by their incomparable raw magnitudes. A summary row&#39;s tag badge and action label come from the group&#39;s own highest-impact member (via the same three-tier comparator, core <code>rankFindings</code>); its sentence is the type-level <code>coreFindingGenericRecommendation</code>, not one member&#39;s own numbers, and is omitted when the members&#39; sentences differ (for example, skew findings with different AQE skew-join cases). The trailing stat depends on the group&#39;s kind (<code>×N · &lt;time&gt; recoverable</code>, <code>×N · &lt;total&gt;</code> with the summed resource waste the CLI report also prints, or <code>×N</code> alone, adding a per-impact-band tally only when the group mixes bands). Clicking it expands straight to the group&#39;s full, impact-ranked list, with no intermediate &quot;worst-K&quot; step, paginated at 10 rows per page (<code>data-testid=&quot;fix-these-first-group-row&quot;</code>; no pager renders for a group of 10 or fewer findings; it appears once a group exceeds 10). A band&#39;s rollup rows render as a headerless three-column <code>Table</code> (<code>src/components/ui/table.tsx</code>): a single-finding row (<code>FindingRow</code>, <code>data-testid=&quot;fix-these-first-row&quot;</code>, <code>data-finding-type</code>) has three <code>TableCell</code>s: the impact dot + ALL-CAPS tag as a real <code>TagBadge</code> (not <code>plainBadge</code>: nothing wraps it, so its own docs links stay real <code>&lt;a&gt;</code>s, same as everywhere else on the board); a text block (a short imperative action label, e.g. &quot;Reduce shuffle size&quot;, from <code>findingActionLabel</code> (<code>packages/core/src/finding-action-label.ts</code>), over the finding&#39;s own full <code>recommendation</code> sentence in smaller muted text, both wrapping rather than truncating, then one <strong>Show evidence</strong> button); and a right-aligned monospace stage reference + impact figure (e.g. <code>St.49 · 20.1s</code>, via the shared <code>formatWallClockRange</code>/<code>formatRawWaste</code> in <code>packages/core/src/format-utils.ts</code>). The <strong>Show evidence</strong> button (<code>EvidenceButton</code>), not the row, is the click target. The first row of a widget&#39;s type in a band owns that widget&#39;s card and expands it in place as a full-width row below (<code>EvidencePanelRow</code>, <code>InlineEvidence</code>); any other row routes via <code>selectTriageTargetForFinding</code> (<code>src/view/triage-target.ts</code>), the same per-finding resolver Stage Summary Table&#39;s own control uses (see &quot;First investigation routing&quot; below), to the card another row or the verdict owns. A route that targets an owned card opens it too (<code>useEvidenceOpen</code>). A <code>TypeGroupRow</code> has the same button for its representative finding, and its text block is an inner button that toggles its expand state (<code>aria-expanded</code>). An expanded group&#39;s members (<code>FindingInstanceRow</code>) omit the badge and span its cell (<code>colSpan={2}</code>): the location line, then the recommendation clamped to one line, then the figure in the third column. <code>ImpactBoard</code> owns which single group is expanded (<code>expandedGroupKey</code>); a <code>TypeGroupRow</code> owns only its page.</p><p>Each impact band (<code>ImpactBoard.tsx</code>&#39;s own <code>ImpactGroup</code>, one call per entry of <code>IMPACT_BAND_ORDER_LIST = [&#39;critical&#39;, &#39;warning&#39;, &#39;info&#39;]</code>) is a <code>&lt;section aria-labelledby&gt;</code> pointing at its <code>&lt;h2 id=&quot;impact-band-&lt;band&gt;-heading&quot; tabIndex={-1}&gt;</code> (Critical, Warning, Info; the top bar&#39;s count chip focuses it; an <code>&lt;h3&gt;</code> under &quot;More findings&quot;), and renders nothing (not even the heading) when it has neither a rollup row nor an active widget: a run with no critical findings has no &quot;Critical&quot; heading or section at all. Inside a band, rollup rows render first as the headerless <code>Table</code> described above, followed by the band&#39;s active <code>REGISTRY</code> widget cards that no row owns (<code>computeActiveWidgets</code>&#39;s ranked list, filtered to this impact band) in their own <code>WidgetGrid</code>: every one of <code>orderedWidgets()</code>&#39;s <code>REGISTRY</code> components <em>except</em> the one always-mounted one below, with at least one finding in <code>catalog</code> ∪ <code>configFindings</code>. Within a band, active widgets keep <code>orderedWidgets()</code>&#39;s own order: the interpretation&#39;s <code>DetectorInfo</code> sorted by region (<code>action</code> first), then ascending detector <code>order</code>. Those unowned cards render collapsed (<code>defaultCollapsed</code>) in both Basic and Advanced view. <code>cacheUtilization</code>, <code>memoryUtilization</code>, and <code>utilization</code> are <code>reference</code>-region types but aren&#39;t always-mounted exceptions, so a Cache Storage, Memory Utilization, or Executor Utilization card with an active finding surfaces in its own impact band like any other active widget.</p><p>Below the impact bands, <code>Alerts.tsx</code>&#39;s exported <code>CleanChecks</code> renders a collapsed &quot;Clean checks&quot; disclosure of <code>CleanCheckRow</code> lines (<code>src/view/widgets/CleanCheckRow.tsx</code>: the type&#39;s tag chip and label, plus in Advanced view the threshold it was held to, the interpretation&#39;s <code>detectors[type].thresholdSummary</code> from <code>getThresholdSummary</code>, which states the detector&#39;s own numbers) built per detector <em>type</em> (every <code>REGISTRY</code> key except that one always-mounted key), under one &quot;Every check below passed.&quot; line (or &quot;No check could run on this log.&quot; when none did) and grouped by detector scope (Per-stage, App-level, SQL plan, Config checks). Types the log could not check (an <code>isEvidenceCaveat</code> finding of that type, every per-stage type when no stage finished, or the run-span types <code>RUN_SPAN_CHECK_TYPES</code> on an <code>incompleteRun</code> log, the same rule that keeps the verdict from calling the run clean) render first under <strong>Not checked on this log</strong> as <code>CleanCheckRow status=&quot;notRun&quot;</code>, drawn neutral rather than clean green, after the interpretation&#39;s <code>coverage.gaps</code> (<code>verdictGaps</code>) lines saying why and naming the setting to turn on. A clean run lands <code>cacheUtilization</code>, <code>memoryUtilization</code>, and <code>utilization</code> here too, same as any ordinary action-region type. Caching Opportunities, Config Audit, and the six Plan Advisor widgets (Redundant Plan Subtree, Excessive Small Files, Nested Loop Join, Row-at-a-time Python UDFs, Missed Broadcast Join, Oversized Broadcast Join) render through the ordinary active/clean paths above (see <a href="./../board-widgets.html#board-widgets-app-and-plan-widgets">App- and plan-level widgets</a>). Core Usage by Locality (<code>coreLocality</code>, resolving to <code>CoreUsageArea</code>) is not in the Findings tab at all: it mounts unconditionally from <code>appModel</code> at the head of the Full app report&#39;s reference grid (<code>alwaysMountedWidgets()</code>/<code>isAlwaysMountedType()</code>), carrying its own impact-band indicator when a finding is active instead of collapsing to a clean-check line on a clean run. A <code>coreLocality</code> finding still lists in its impact band&#39;s recommendation rows, and its <strong>Show evidence</strong> switches to the Full app report tab.</p>',6)])])}const u=o(a,[["render",c]]);export{g as __pageData,u as default};

package/public/docs/assets/contributor-guide_architecture_widget-rendering_findings-tab.md.B7rjSF_h.lean.js

@@ -0,0 +1 @@
+import{_ as o,o as t,c as d,a5 as n}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Findings tab","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/widget-rendering/findings-tab.md","filePath":"contributor-guide/architecture/widget-rendering/findings-tab.md"}'),a={name:"contributor-guide/architecture/widget-rendering/findings-tab.md"};function c(i,e,r,s,l,p){return t(),d("div",null,[...e[0]||(e[0]=[n("",6)])])}const u=o(a,[["render",c]]);export{g as __pageData,u as default};

package/public/docs/assets/{contributor-guide_architecture_widget-rendering_list-sort-mode.md.BuHKgH0C.js → contributor-guide_architecture_widget-rendering_list-sort-mode.md.37hcZJ7q.js}

@@ -1 +1 @@
-import{_ as o,o as t,c as d,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Per-widget list sort mode","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/widget-rendering/list-sort-mode.md","filePath":"contributor-guide/architecture/widget-rendering/list-sort-mode.md"}'),c={name:"contributor-guide/architecture/widget-rendering/list-sort-mode.md"};function a(r,e,i,n,l,g){return t(),d("div",null,[...e[0]||(e[0]=[s('<h1 id="per-widget-list-sort-mode" tabindex="-1">Per-widget list sort mode <a class="header-anchor" href="#per-widget-list-sort-mode" aria-label="Permalink to &quot;Per-widget list sort mode&quot;">​</a></h1><p>How a Findings-tab widget opts into the shared list sort mode.</p><p><code>src/view/impact-sort.ts</code> (<code>sumWallClockLow</code>, <code>hasSortableImpact</code>, <code>canToggleSort</code>, <code>byImpactDesc</code>, <code>stageIdOf</code>, <code>byStageAsc</code>), <code>src/view/useSortMode.ts</code> and <code>src/view/SortModeToggle.tsx</code> are a shared, opt-in set a Findings-tab active widget can use to let its own expanded, multi-row list default to potential-savings order (<code>wallClock.low</code> descending, the guaranteed-floor bound, not the optimistic <code>high</code>) instead of stage number. Each of the widgets below gets its <code>sortMode</code> (<code>SortMode</code>, <code>&#39;impact&#39; | &#39;stage&#39;</code>, default <code>&#39;impact&#39;</code>) from <code>useSortMode</code>, which forces impact order at Basic density, and renders a <code>SortModeToggle</code> in the <code>WidgetCard</code> <code>badges</code> slot, so it sits on the title row itself rather than floating in the body: a button alongside the widget&#39;s tag badges (a <code>WidgetCard</code> header renders <code>badges</code> beside the title heading, never inside another control). Every widget below carries the same <code>canToggleSort</code> check, so the toggle only actually renders for a type whose finding can carry a wall-clock estimate: <code>Skew.tsx</code>, <code>TinyTask.tsx</code> (<code>stageShape</code>&#39;s own rules never produce one, so <code>StageShape.tsx</code> carries the same check but it never fires), <code>ShuffleIO.tsx</code> (narrowed to <code>shuffle</code>), <code>PartitionSizing.tsx</code>, <code>Spill.tsx</code>, <code>GcPressure.tsx</code> (both its high-GC and low-GC sections, one shared toggle), <code>RetryWaste.tsx</code> (its siblings <code>StageFailed.tsx</code>/<code>TaskFailures.tsx</code> carry the same check, but <code>stageFailed</code>/<code>failures</code> are <code>estimateMethod: &#39;none&#39;</code>, so it never fires there either), <code>SlowHost.tsx</code>, <code>StageSlowness.tsx</code>, <code>Straggler.tsx</code>, <code>SpeculationWaste.tsx</code>, <code>ColdStart.tsx</code> (one widget per type, each sorting only its own flat issue list), and <code>DuplicatePlanSubtree.tsx</code>/ <code>SmallFiles.tsx</code>/<code>UnderBroadcast.tsx</code>/<code>OverBroadcast.tsx</code> (each reorders its own list by <code>stageIdOf</code>, the lowest stage id its finding touches). It is a two-segment <code>ToggleGroup</code> (Impact / Stage; optional <code>stageLabel</code>, default &quot;Stage&quot;, which no caller overrides), wrapped in <code>AdvancedOnly</code>: stage number is the one axis every one of these widgets&#39; items can be compared on. Neither mode sorts by impact band, which ranks findings by how bad they are, not by how much fixing them would save. A per-stage detector&#39;s own <code>finding.stageId</code> is the sort key directly; a sql-scope finding that spans several stages (<code>duplicatePlanSubtree</code>, <code>smallFiles</code>, <code>underBroadcast</code>, <code>overBroadcast</code>, via <code>stageIds</code>) sorts by the lowest stage id it touches (<code>stageIdOf</code>). The toggle renders only when <code>canToggleSort</code> holds: <code>hasSortableImpact</code> finds a wall-clock claim in the list, there is more than one row, and the card is open; re-sorting a list with no wall-clock claim would be a silent no-op. Both comparators return <code>0</code> when neither side has a comparable value, so those items keep their prior relative order (<code>Array.prototype.sort</code>&#39;s stability) rather than being shuffled. <code>FixTheseFirst</code>&#39;s own expanded group list (see &quot;Findings tab&quot; above) sorts by impact on its own and does not use this pattern; it has no stage-order toggle.</p>',3)])])}const u=o(c,[["render",a]]);export{p as __pageData,u as default};
+import{_ as o,o as t,c as d,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Per-widget list sort mode","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/widget-rendering/list-sort-mode.md","filePath":"contributor-guide/architecture/widget-rendering/list-sort-mode.md"}'),c={name:"contributor-guide/architecture/widget-rendering/list-sort-mode.md"};function a(r,e,i,n,l,g){return t(),d("div",null,[...e[0]||(e[0]=[s('<h1 id="per-widget-list-sort-mode" tabindex="-1">Per-widget list sort mode <a class="header-anchor" href="#per-widget-list-sort-mode" aria-label="Permalink to &quot;Per-widget list sort mode&quot;">​</a></h1><p>How a Findings-tab widget opts into the shared list sort mode.</p><p><code>src/view/impact-sort.ts</code> (<code>sumWallClockLow</code>, <code>hasSortableImpact</code>, <code>canToggleSort</code>, <code>byImpactDesc</code>, <code>stageIdOf</code>, <code>byStageAsc</code>), <code>src/view/useSortMode.ts</code> and <code>src/view/SortModeToggle.tsx</code> are a shared, opt-in set a Findings-tab active widget can use to let its own expanded, multi-row list default to potential-savings order (<code>wallClock.low</code> descending, the guaranteed-floor bound, not the optimistic <code>high</code>) instead of stage number. Each of the widgets below gets its <code>sortMode</code> (<code>SortMode</code>, <code>&#39;impact&#39; | &#39;stage&#39;</code>, default <code>&#39;impact&#39;</code>) from <code>useSortMode</code>, which forces impact order at Basic density, and renders a <code>SortModeToggle</code> in the <code>WidgetCard</code> <code>badges</code> slot, so it sits on the title row itself rather than floating in the body: a button alongside the widget&#39;s tag badges (a <code>WidgetCard</code> header renders <code>badges</code> beside the title heading, never inside another control). Every widget below carries the same <code>canToggleSort</code> check, so the toggle only actually renders for a type whose finding can carry a wall-clock estimate: <code>Skew.tsx</code>, <code>TinyTask.tsx</code> (<code>stageShape</code>&#39;s own rules never produce one, so <code>StageShape.tsx</code> carries the same check but it never fires), <code>ShuffleIO.tsx</code> (narrowed to <code>shuffle</code>), <code>PartitionSizing.tsx</code>, <code>Spill.tsx</code>, <code>GcPressure.tsx</code> (both its high-GC and low-GC sections, one shared toggle), <code>RetryWaste.tsx</code> (its siblings <code>StageFailed.tsx</code>/<code>TaskFailures.tsx</code> carry the same check, but <code>stageFailed</code>/<code>failures</code> are <code>estimateMethod: &#39;none&#39;</code>, so it never fires there either), <code>SlowHost.tsx</code>, <code>StageSlowness.tsx</code>, <code>Straggler.tsx</code>, <code>SpeculationWaste.tsx</code>, <code>ColdStart.tsx</code> (one widget per type, each sorting only its own flat issue list), and <code>DuplicatePlanSubtree.tsx</code>/ <code>SmallFiles.tsx</code>/<code>NestedLoopJoin.tsx</code>/<code>PythonUdf.tsx</code>/<code>UnderBroadcast.tsx</code>/<code>OverBroadcast.tsx</code> (each reorders its own list by <code>stageIdOf</code>, the lowest stage id its finding touches). It is a two-segment <code>ToggleGroup</code> (Impact / Stage; optional <code>stageLabel</code>, default &quot;Stage&quot;, which no caller overrides), wrapped in <code>AdvancedOnly</code>: stage number is the one axis every one of these widgets&#39; items can be compared on. Neither mode sorts by impact band, which ranks findings by how bad they are, not by how much fixing them would save. A per-stage detector&#39;s own <code>finding.stageId</code> is the sort key directly; a sql-scope finding that spans several stages (<code>duplicatePlanSubtree</code>, <code>smallFiles</code>, <code>nestedLoopJoin</code>, <code>pythonUdf</code>, <code>underBroadcast</code>, <code>overBroadcast</code>, via <code>stageIds</code>) sorts by the lowest stage id it touches (<code>stageIdOf</code>). The toggle renders only when <code>canToggleSort</code> holds: <code>hasSortableImpact</code> finds a wall-clock claim in the list, there is more than one row, and the card is open; re-sorting a list with no wall-clock claim would be a silent no-op. Both comparators return <code>0</code> when neither side has a comparable value, so those items keep their prior relative order (<code>Array.prototype.sort</code>&#39;s stability) rather than being shuffled. <code>FixTheseFirst</code>&#39;s own expanded group list (see &quot;Findings tab&quot; above) sorts by impact on its own and does not use this pattern; it has no stage-order toggle.</p>',3)])])}const u=o(c,[["render",a]]);export{p as __pageData,u as default};

package/public/docs/assets/{contributor-guide_architecture_widget-rendering_row-expand-contract.md.ZT89u9f1.js → contributor-guide_architecture_widget-rendering_row-expand-contract.md.mK_VmAWk.js}

@@ -1 +1 @@
-import{_ as o,o as t,c as d,a5 as a}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Gold Standard row/expand contract","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/widget-rendering/row-expand-contract.md","filePath":"contributor-guide/architecture/widget-rendering/row-expand-contract.md"}'),n={name:"contributor-guide/architecture/widget-rendering/row-expand-contract.md"};function c(i,e,s,r,l,g){return t(),d("div",null,[...e[0]||(e[0]=[a('<h1 id="gold-standard-row-expand-contract" tabindex="-1">Gold Standard row/expand contract <a class="header-anchor" href="#gold-standard-row-expand-contract" aria-label="Permalink to &quot;Gold Standard row/expand contract&quot;">​</a></h1><p>The row and expand contract every widget in the registry follows.</p><p>Every <code>REGISTRY</code> widget renders its content as N≥1 rows, using <code>Skew.tsx</code> as the reference implementation. A widget&#39;s data shape (chart, table, single app-scoped scalar) is never by itself a reason to skip this: the four named exceptions below (one histogram toggle, three partial exceptions) are the only ones. Any further exception must get its own entry here, justified in the same PR that introduces it.</p><ul><li>Tier A (universal): <code>WidgetCard</code> chrome, an impact-band left-border + dot, ALL-CAPS <code>TagBadge</code>s, a <code>SortModeToggle</code> when <code>canToggleSort</code> is true, a <code>finding-anchor</code> ref on any focusable/deep-linkable unit, and an early <code>null</code> return on an empty catalog filter (except Core Usage by Locality, the one always-mounted widget that renders unconditionally from <code>appModel</code> via <code>alwaysMountedWidgets()</code>/<code>isAlwaysMountedType()</code> rather than early-returning on an empty catalog filter).</li><li>Tier B (the row/expand pattern): a row&#39;s collapsed state shows its core metric(s), <code>ImpactEstimate</code>, any config-hint code snippet/list, and docs links, all unconditionally. The fix is stated once per card: <code>WidgetCard</code>&#39;s <code>fixFor</code> prints each distinct <code>coreFindingGenericRecommendation</code> above the body, so rows don&#39;t repeat a recommendation that restates their number. Every finding-row widget passes it, Shuffle I/O and the Skew, Stage Shape and Tiny Tasks groups (<code>StageFindingGroup</code>) included; Incomplete Run is the one that doesn&#39;t. Where the detector&#39;s sentence is a row&#39;s only measurement (Excessive Small Files, Missed Broadcast Join, Oversized Broadcast Join, Config Audit), the row shows its measured half, split off by core&#39;s <code>recommendationParts</code>. Three rows keep their own sentence because their type has no generic fix for them: Incomplete Run, and a data-unavailable Memory Utilization or Cache Storage row, which names the setting to enable. Confidence and evidence are unconditional too: <code>RowStatusCluster</code> (<code>src/view/RowStatusCluster.tsx</code>) is a single, fixed-position pill combining both. Every widget that carries one wraps it in <code>AdvancedOnly</code> (<code>src/view/AdvancedOnly.tsx</code>), so it only renders at the Advanced density tier: this is a content-visibility gate (Basic vs. Advanced), not a per-row collapse, and it&#39;s applied consistently across every adopting widget. No widget gates confidence or evidence behind a per-row click. A widget-header <code>RowStatusCluster</code> (passed as <code>WidgetCard</code>&#39;s <code>statusBadge</code> prop) carries one further, unrelated gate on top of <code>AdvancedOnly</code>: <code>open &amp;&amp; statusBadge</code> (<code>WidgetCard.tsx</code>&#39;s header) keeps it out of the collapsed header, matching the comment on <code>statusBadge</code> itself (&quot;kept separate so tag/impact badges stay visible in the collapsed summary while a <code>RowStatusCluster</code>-style ... marker doesn&#39;t&quot;). This is the card&#39;s own expand/collapse, not a separate reveal-on-click built for the cluster, and it composes with density as an AND: a header cluster needs both Advanced tier and an expanded card. A row with neither renders no cluster at all (<code>RowStatusCluster</code> returns <code>null</code>). Confidence renders as plain, non-interactive text (&quot;<code>{confidence} confidence</code>&quot;, no further suffix) with a <code>title</code> + <code>aria-describedby</code> tooltip carrying the full <code>validationRequired</code> detail: never itself a click target. Evidence renders as the cluster&#39;s one real click target: a button whose visible text is just the evidence label (e.g. &quot;SQL plan&quot;) but whose accessible name always carries the &quot;Evidence: ...&quot; prefix via an explicit <code>aria-label</code>, regardless of what else sits nearby. Clicking it calls <code>revealEvidence</code>. <code>RowStatusCluster</code>&#39;s fixed slot is the row&#39;s own stage-pill/title line (<code>justify-between</code>, cluster right-aligned), the same position in every adopting widget, not floating with the row&#39;s detail below. Lists longer than <code>VISIBLE_LIMIT</code> (6, <code>packages/core/src/format-utils.ts</code>) get page-based navigation (Previous/Next, <code>usePagedRows</code>/<code>RowPagination</code>, <code>src/view/usePagedRows.ts</code>/<code>src/view/RowPagination.tsx</code>) instead of rendering unconditionally: the one sanctioned cap mechanism across every Tier B widget. A Tier B widget whose rows are also triage-routable (wires <code>useFindingAnchor</code>) must pass its <code>routeIndex</code> (the routed finding&#39;s position in the paginated list) into <code>usePagedRows</code>, or a deep-linked route to a finding beyond the first page silently degrades to focusing the widget&#39;s disclosure title instead of the row.</li></ul><p>The following widgets carry a <code>RowStatusCluster</code>, all Advanced-only: <code>Spill.tsx</code> (confidence only), <code>ConfigAudit.tsx</code>&#39;s widget-header control (evidence only, a widget-wide constant rather than per-finding data, since the evidence key doesn&#39;t vary per row; also the control shown in the widget&#39;s empty-findings/no-data states), <code>MemoryUtilization.tsx</code>&#39;s widget header (confidence + evidence, one marker, since <code>executorMetrics</code> is widget-wide; <code>ExecutorUtilization.tsx</code>&#39;s rows carry neither), each of the four Plan Advisor widgets&#39; (<code>DuplicatePlanSubtree.tsx</code>, <code>SmallFiles.tsx</code>, <code>UnderBroadcast.tsx</code>, <code>OverBroadcast.tsx</code>) widget header (confidence + <code>sqlPlan</code> evidence, one marker per widget since each is its own finding type; no per-row control), <code>ScalingSim.tsx</code> (two call sites, evidence only), <code>CoreUsageArea.tsx</code>, <code>EfficiencyModel.tsx</code> and <code>PlanView.tsx</code> (three call sites) (these last confidence-only; all standalone rather than per-finding-row: a card-header badge, a widget-level single marker, a plan-tree node/summary-row marker, or an &quot;unavailable data&quot; message: the same component, same visual language, regardless of where it sits); and, because <code>skew</code>/<code>straggler</code>/<code>gc</code> disclose their own unvalidated noise-floor thresholds (confidence only, per-row, no <code>evidenceKey</code> passed), <code>GcPressure.tsx</code>&#39;s rows, <code>Straggler.tsx</code>&#39;s rows, <code>CachingOpportunity.tsx</code>&#39;s rows (the <code>cachingOpportunity</code> detector scales <code>confidence</code> per finding via <code>cachingReuseConfidence</code>, so the badge sits next to each row rather than as a single caveat below the table), and the shared <code>StageFindingGroup.tsx</code> row (adopted by <code>Skew.tsx</code>/<code>StageShape.tsx</code>/<code>TinyTask.tsx</code>, though only <code>skew</code> findings carry a <code>confidence</code> field).</p><p>One named exception:</p><ul><li><code>Skew.tsx</code>/<code>StageShape.tsx</code>/<code>TinyTask.tsx</code>: their per-row histogram toggle (<code>ExpandToggleButton</code> + <code>useExpandableRow</code>) gates a lazily-fetched duration histogram, unrelated to confidence/evidence. It sits alongside <code>RowStatusCluster</code> in the shared <code>StageFindingGroup.tsx</code> row; the two controls coexist per row. <code>ExpandToggleButton</code> is single-purpose (always the task-detail toggle) since these three are its only adopters.</li></ul><p>No toggle at all, unconditional content, since these widgets carry no confidence/evidence display: <code>IncompleteRun.tsx</code>, <code>ShuffleIO.tsx</code>, <code>PartitionSizing.tsx</code>, <code>StageFailed.tsx</code>, <code>TaskFailures.tsx</code>, <code>RetryWaste.tsx</code>, <code>ColdStart.tsx</code>, <code>SlowHost.tsx</code>, <code>StageSlowness.tsx</code>, <code>SpeculationWaste.tsx</code>, <code>ExecutorCountChart.tsx</code>, <code>ExecutorUtilization.tsx</code>, <code>JobFailures.tsx</code>, <code>CacheUtilization.tsx</code> and <code>AutoscalingChurn.tsx</code>.</p><p>Three named exceptions to specific pieces of the contract, not to the row wrapper or pagination, which still apply to all three:</p><ul><li><code>CoreUsageArea.tsx</code>&#39;s non-local-stage list is a derived stat breakdown (<code>computeCoreLocalityRatio</code>&#39;s <code>topStages</code>), not a <code>Finding[]</code>: there is no per-row finding or recommendation to reveal. Paginated like every other Tier B list, but with no per-row expand.</li><li><code>CacheUtilization.tsx</code>&#39;s RDD <code>&lt;Table&gt;</code> holds reference columns (name, storage level, partitions, memory, disk bytes) with no recommendation attached to any row. Paginated like every other Tier B list, but with no per-row expand. Its separate finding list below the table is full Tier B (unconditional, no toggle, per the paragraph above).</li><li><code>IncompleteRun.tsx</code> leads with no metric of its own: this app-scoped, single-finding widget has no natural lead figure distinct from its title/badge, unlike every other Gold Standard widget, which leads with a percentage, duration, or count.</li></ul>',10)])])}const u=o(n,[["render",c]]);export{p as __pageData,u as default};
+import{_ as o,o as t,c as d,a5 as a}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Gold Standard row/expand contract","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/widget-rendering/row-expand-contract.md","filePath":"contributor-guide/architecture/widget-rendering/row-expand-contract.md"}'),n={name:"contributor-guide/architecture/widget-rendering/row-expand-contract.md"};function c(i,e,s,r,l,g){return t(),d("div",null,[...e[0]||(e[0]=[a('<h1 id="gold-standard-row-expand-contract" tabindex="-1">Gold Standard row/expand contract <a class="header-anchor" href="#gold-standard-row-expand-contract" aria-label="Permalink to &quot;Gold Standard row/expand contract&quot;">​</a></h1><p>The row and expand contract every widget in the registry follows.</p><p>Every <code>REGISTRY</code> widget renders its content as N≥1 rows, using <code>Skew.tsx</code> as the reference implementation. A widget&#39;s data shape (chart, table, single app-scoped scalar) is never by itself a reason to skip this: the four named exceptions below (one histogram toggle, three partial exceptions) are the only ones. Any further exception must get its own entry here, justified in the same PR that introduces it.</p><ul><li>Tier A (universal): <code>WidgetCard</code> chrome, an impact-band left-border + dot, ALL-CAPS <code>TagBadge</code>s, a <code>SortModeToggle</code> when <code>canToggleSort</code> is true, a <code>finding-anchor</code> ref on any focusable/deep-linkable unit, and an early <code>null</code> return on an empty catalog filter (except Core Usage by Locality, the one always-mounted widget that renders unconditionally from <code>appModel</code> via <code>alwaysMountedWidgets()</code>/<code>isAlwaysMountedType()</code> rather than early-returning on an empty catalog filter).</li><li>Tier B (the row/expand pattern): a row&#39;s collapsed state shows its core metric(s), <code>ImpactEstimate</code>, any config-hint code snippet/list, and docs links, all unconditionally. The fix is stated once per card: <code>WidgetCard</code>&#39;s <code>fixFor</code> prints each distinct <code>coreFindingGenericRecommendation</code> above the body, so rows don&#39;t repeat a recommendation that restates their number. Every finding-row widget passes it, Shuffle I/O and the Skew, Stage Shape and Tiny Tasks groups (<code>StageFindingGroup</code>) included; Incomplete Run is the one that doesn&#39;t. Where the detector&#39;s sentence is a row&#39;s only measurement (Excessive Small Files, Missed Broadcast Join, Oversized Broadcast Join, Config Audit), the row shows its measured half, split off by core&#39;s <code>recommendationParts</code>. Three rows keep their own sentence because their type has no generic fix for them: Incomplete Run, and a data-unavailable Memory Utilization or Cache Storage row, which names the setting to enable. Confidence and evidence are unconditional too: <code>RowStatusCluster</code> (<code>src/view/RowStatusCluster.tsx</code>) is a single, fixed-position pill combining both. Every widget that carries one wraps it in <code>AdvancedOnly</code> (<code>src/view/AdvancedOnly.tsx</code>), so it only renders at the Advanced density tier: this is a content-visibility gate (Basic vs. Advanced), not a per-row collapse, and it&#39;s applied consistently across every adopting widget. No widget gates confidence or evidence behind a per-row click. A widget-header <code>RowStatusCluster</code> (passed as <code>WidgetCard</code>&#39;s <code>statusBadge</code> prop) carries one further, unrelated gate on top of <code>AdvancedOnly</code>: <code>open &amp;&amp; statusBadge</code> (<code>WidgetCard.tsx</code>&#39;s header) keeps it out of the collapsed header, matching the comment on <code>statusBadge</code> itself (&quot;kept separate so tag/impact badges stay visible in the collapsed summary while a <code>RowStatusCluster</code>-style ... marker doesn&#39;t&quot;). This is the card&#39;s own expand/collapse, not a separate reveal-on-click built for the cluster, and it composes with density as an AND: a header cluster needs both Advanced tier and an expanded card. A row with neither renders no cluster at all (<code>RowStatusCluster</code> returns <code>null</code>). Confidence renders as plain, non-interactive text (&quot;<code>{confidence} confidence</code>&quot;, no further suffix) with a <code>title</code> + <code>aria-describedby</code> tooltip carrying the full <code>validationRequired</code> detail: never itself a click target. Evidence renders as the cluster&#39;s one real click target: a button whose visible text is just the evidence label (e.g. &quot;SQL plan&quot;) but whose accessible name always carries the &quot;Evidence: ...&quot; prefix via an explicit <code>aria-label</code>, regardless of what else sits nearby. Clicking it calls <code>revealEvidence</code>. <code>RowStatusCluster</code>&#39;s fixed slot is the row&#39;s own stage-pill/title line (<code>justify-between</code>, cluster right-aligned), the same position in every adopting widget, not floating with the row&#39;s detail below. Lists longer than <code>VISIBLE_LIMIT</code> (6, <code>packages/core/src/format-utils.ts</code>) get page-based navigation (Previous/Next, <code>usePagedRows</code>/<code>RowPagination</code>, <code>src/view/usePagedRows.ts</code>/<code>src/view/RowPagination.tsx</code>) instead of rendering unconditionally: the one sanctioned cap mechanism across every Tier B widget. A Tier B widget whose rows are also triage-routable (wires <code>useFindingAnchor</code>) must pass its <code>routeIndex</code> (the routed finding&#39;s position in the paginated list) into <code>usePagedRows</code>, or a deep-linked route to a finding beyond the first page silently degrades to focusing the widget&#39;s disclosure title instead of the row.</li></ul><p>The following widgets carry a <code>RowStatusCluster</code>, all Advanced-only: <code>Spill.tsx</code> (confidence only), <code>ConfigAudit.tsx</code>&#39;s widget-header control (evidence only, a widget-wide constant rather than per-finding data, since the evidence key doesn&#39;t vary per row; also the control shown in the widget&#39;s empty-findings/no-data states), <code>MemoryUtilization.tsx</code>&#39;s widget header (confidence + evidence, one marker, since <code>executorMetrics</code> is widget-wide; <code>ExecutorUtilization.tsx</code>&#39;s rows carry neither), each of the six Plan Advisor widgets&#39; (<code>DuplicatePlanSubtree.tsx</code>, <code>SmallFiles.tsx</code>, <code>NestedLoopJoin.tsx</code>, <code>PythonUdf.tsx</code>, <code>UnderBroadcast.tsx</code>, <code>OverBroadcast.tsx</code>) widget header (confidence + <code>sqlPlan</code> evidence, one marker per widget since each is its own finding type; no per-row control), <code>ScalingSim.tsx</code> (two call sites, evidence only), <code>CoreUsageArea.tsx</code>, <code>EfficiencyModel.tsx</code> and <code>PlanView.tsx</code> (three call sites) (these last confidence-only; all standalone rather than per-finding-row: a card-header badge, a widget-level single marker, a plan-tree node/summary-row marker, or an &quot;unavailable data&quot; message: the same component, same visual language, regardless of where it sits); and, because <code>skew</code>/<code>straggler</code>/<code>gc</code> disclose their own unvalidated noise-floor thresholds (confidence only, per-row, no <code>evidenceKey</code> passed), <code>GcPressure.tsx</code>&#39;s rows, <code>Straggler.tsx</code>&#39;s rows, <code>CachingOpportunity.tsx</code>&#39;s rows (the <code>cachingOpportunity</code> detector scales <code>confidence</code> per finding via <code>cachingReuseConfidence</code>, so the badge sits next to each row rather than as a single caveat below the table), and the shared <code>StageFindingGroup.tsx</code> row (adopted by <code>Skew.tsx</code>/<code>StageShape.tsx</code>/<code>TinyTask.tsx</code>, though only <code>skew</code> findings carry a <code>confidence</code> field).</p><p>One named exception:</p><ul><li><code>Skew.tsx</code>/<code>StageShape.tsx</code>/<code>TinyTask.tsx</code>: their per-row histogram toggle (<code>ExpandToggleButton</code> + <code>useExpandableRow</code>) gates a lazily-fetched duration histogram, unrelated to confidence/evidence. It sits alongside <code>RowStatusCluster</code> in the shared <code>StageFindingGroup.tsx</code> row; the two controls coexist per row. <code>ExpandToggleButton</code> is single-purpose (always the task-detail toggle) since these three are its only adopters.</li></ul><p>No toggle at all, unconditional content, since these widgets carry no confidence/evidence display: <code>IncompleteRun.tsx</code>, <code>ShuffleIO.tsx</code>, <code>PartitionSizing.tsx</code>, <code>StageFailed.tsx</code>, <code>TaskFailures.tsx</code>, <code>RetryWaste.tsx</code>, <code>ColdStart.tsx</code>, <code>SlowHost.tsx</code>, <code>StageSlowness.tsx</code>, <code>SpeculationWaste.tsx</code>, <code>ExecutorCountChart.tsx</code>, <code>ExecutorUtilization.tsx</code>, <code>JobFailures.tsx</code>, <code>CacheUtilization.tsx</code> and <code>AutoscalingChurn.tsx</code>.</p><p>Three named exceptions to specific pieces of the contract, not to the row wrapper or pagination, which still apply to all three:</p><ul><li><code>CoreUsageArea.tsx</code>&#39;s non-local-stage list is a derived stat breakdown (<code>computeCoreLocalityRatio</code>&#39;s <code>topStages</code>), not a <code>Finding[]</code>: there is no per-row finding or recommendation to reveal. Paginated like every other Tier B list, but with no per-row expand.</li><li><code>CacheUtilization.tsx</code>&#39;s RDD <code>&lt;Table&gt;</code> holds reference columns (name, storage level, partitions, memory, disk bytes) with no recommendation attached to any row. Paginated like every other Tier B list, but with no per-row expand. Its separate finding list below the table is full Tier B (unconditional, no toggle, per the paragraph above).</li><li><code>IncompleteRun.tsx</code> leads with no metric of its own: this app-scoped, single-finding widget has no natural lead figure distinct from its title/badge, unlike every other Gold Standard widget, which leads with a percentage, duration, or count.</li></ul>',10)])])}const u=o(n,[["render",c]]);export{p as __pageData,u as default};

package/public/docs/assets/{contributor-guide_architecture_worker-protocol.md.BAnhi2-d.js → contributor-guide_architecture_worker-protocol.md.Br6OE7RW.js}

@@ -1 +1 @@
-import{_ as o,o as t,c as s,a5 as d}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Worker protocol","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol.md","filePath":"contributor-guide/architecture/worker-protocol.md"}'),r={name:"contributor-guide/architecture/worker-protocol.md"};function a(c,e,i,n,l,p){return t(),s("div",null,[...e[0]||(e[0]=[d('<h1 id="worker-protocol" tabindex="-1">Worker protocol <a class="header-anchor" href="#worker-protocol" aria-label="Permalink to &quot;Worker protocol&quot;">​</a></h1><h2 id="message-protocol" tabindex="-1">Message protocol <a class="header-anchor" href="#message-protocol" aria-label="Permalink to &quot;Message protocol&quot;">​</a></h2><p>Worker to main:</p><ul><li><code>app</code>, <code>stage</code>, <code>sql</code>, <code>executor</code>, <code>job</code>, <code>progress</code>: progressive, posted as the parser reads.</li><li><code>sqlPlan</code> (<code>{ type: &#39;sqlPlan&#39;, data: { executionId, planTree } }</code>): posted by <code>endSqlExecution</code> once a SQL execution&#39;s plan tree resolves. A repeated <code>SQLExecutionEnd</code> for that execution posts nothing, so no later <code>sql</code> message replaces the entry holding its <code>planTree</code>.</li><li><code>runAggregates</code>: one whole-run core-time-series summary (busy-core-ms, peak concurrency, per-stage task-duration sums), emitted just before <code>done</code>.</li><li><code>stageExecutorMetrics</code>: the post-completion re-post described in <a href="./overview.html#streaming">Streaming</a>, also emitted just before <code>done</code>.</li><li><code>stageSpeculationWaste</code>: the speculation totals of stages that gained waste from a TaskEnd after their StageCompleted, also described in <a href="./overview.html#streaming">Streaming</a> and emitted just before <code>done</code>.</li><li><code>stageLateAttemptWork</code>: the work of a failed stage attempt&#39;s tasks that ended after its StageCompleted, also described in <a href="./overview.html#streaming">Streaming</a> and emitted just before <code>done</code>.</li><li><code>done</code>, <code>taskData</code>, <code>error</code>.</li></ul><p><code>emitParseCompletion</code> posts the tail in a fixed order: any deferred AQE <code>sql</code> updates, <code>progress</code> (<code>pct: 1</code>), <code>stageLateAttemptWork</code>, <code>runAggregates</code>, <code>stageSpeculationWaste</code>, <code>stageExecutorMetrics</code>, the terminal <code>app</code>, then <code>done</code>.</p><p>A History Server failure is always the typed payload <code>{ type: &#39;error&#39;, source: &#39;shs&#39;, code, message? }</code>, where <code>code</code> is one of <code>local-server-unavailable</code>, <code>upstream-unreachable</code>, <code>application-not-found</code>, <code>access-or-upstream-failure</code>, or <code>invalid-event-log</code>. Fetch and HTTP failures carry only <code>code</code>: never an upstream message, URL, status, or response body. An <code>invalid-event-log</code> from the archive decode can carry a <code>message</code> written by <code>decodeShsArchive</code> (<code>packages/core/src/shs-fetch.ts</code>): the zip is unreadable, holds several application attempts or no event log, or an entry failed to decompress, in which case it names the zip entry and includes the decoder&#39;s error text. The intake shows that message under the recovery text.</p><p>Evidence availability has no dedicated worker message: the final <code>app</code> message carries a compact <code>evidenceInputs</code> counter summary, and <code>done.skippedLines</code> supplies its parse-integrity input.</p><p>Main to worker: <code>{ type: &#39;parse&#39;, file }</code>, <code>{ type: &#39;parseFiles&#39;, files }</code> (rolling <code>eventlog_v2_*</code> directories, one continuous stream across files), <code>{ type: &#39;parseFromUrl&#39;, request }</code> with <code>request = { baseUrl, appId, attemptId }</code>, and <code>{ type: &#39;getTaskData&#39;, stageId, reqId }</code>; <code>taskData</code> echoes <code>stageId</code> and <code>reqId</code> with <code>metrics</code> and <code>fieldNames</code>. The SHS request object is normalized before it reaches the worker: <code>{ baseUrl: string, appId: string, attemptId: string | null }</code>.</p><p>Files are read in 512 KiB slices (smaller for small files, so every file gets at least 100 reads), and <code>progress { pct, linesProcessed }</code> posts every 300 lines. <code>parseFromUrl</code> reports the download as 0-0.5, then <code>pct: null</code> every 2000 lines while parsing.</p><p><code>taskData</code> uses structured-clone (<code>.slice()</code>) so the worker retains its own <code>Float64Array</code> for re-renders.</p><p>Post-parse prefetch: after <code>done</code>, main runs <code>analyzer.ts</code> to build the bottleneck catalog, then fires parallel <code>getTaskData</code> for every flagged stage so their widgets render immediately. Unflagged stages are on-demand.</p><h3 id="decompress-worker" tabindex="-1">Decompress worker <a class="header-anchor" href="#decompress-worker" aria-label="Permalink to &quot;Decompress worker&quot;">​</a></h3><p>A dropped zstd file (<code>parse</code>, and each zstd file of a <code>parseFiles</code> directory) is decompressed in a second, nested worker, <code>packages/core/src/zstd-worker.ts</code>, so fzstd and the NDJSON parser run at the same time. The parse worker starts it on the first zstd file and reuses it for the rest of the parse. It dies with the parse worker, so the page&#39;s <code>terminate()</code> also cancels it. Other codecs and the SHS path (<code>parseFromUrl</code>) decompress on the parse worker. A dropped History Server zip (<code>parse</code>) streams its zstd entries through the decompress worker too.</p><p><code>packages/core/src/zstd-worker-client.ts</code> is the parse-worker end: it plugs into <code>streamFile</code> as the <code>zstdDecoder</code> option, like the Node CLI&#39;s native decoder. The messages&#39; buffers move by transfer rather than structured-clone copy (a read slice that is a view of a larger buffer is copied once first, and decoded output is copied once into a 1 MiB batch):</p><ul><li>Parse to decompress: <code>start { id }</code> opens a stream, and replaces any open one. <code>data { id, seq, bytes, final }</code> carries one compressed read slice. <code>cancel { id }</code> drops the stream.</li><li>Decompress to parse: <code>ready</code> once, at startup. <code>chunk { id, bytes, length }</code> carries decoded output in batches of up to 1 MiB. <code>consumed { id, seq }</code> follows the last <code>chunk</code> of slice <code>seq</code>. <code>error { id, message }</code> ends the stream, and its message becomes the usual <code>Could not decompress</code> error.</li></ul><p>Flow control is a window of three input slices: <code>push()</code> resolves while fewer than three slices wait for their <code>consumed</code>. Each <code>chunk</code> is parsed in its message handler, so a slice is acknowledged only after its output was parsed, and at most three slices&#39; output is ever queued. The final <code>push()</code> resolves once every slice is acknowledged. When <code>streamFile</code> gives up mid-stream (for example a failed file read) it calls <code>cancel()</code>. A parse exception inside a <code>chunk</code> handler makes the client post <code>cancel</code> itself and fail the stream, which surfaces on the next <code>push()</code>.</p><p>When the nested worker cannot start (<code>new Worker</code> throws or its script fails to load), the parse worker logs a warning and decodes with in-thread fzstd, with identical output. A crash after startup fails the stream it was decoding, and later streams fall back the same way. The progress <code>pct</code> is the read position, so it can run up to the window ahead of the slice being parsed. The self-contained <code>file://</code> export never parses in the browser (it opens with its run already analyzed), so it never starts either worker.</p><h3 id="zstd-in-the-browser-and-in-node" tabindex="-1">zstd in the browser and in Node <a class="header-anchor" href="#zstd-in-the-browser-and-in-node" aria-label="Permalink to &quot;zstd in the browser and in Node&quot;">​</a></h3><p>The browser decodes zstd with the vendored fzstd, not <code>DecompressionStream</code>. The Node CLI and MCP path (<code>collectRun</code>, <code>shs-load.ts</code>) uses <code>packages/core/src/cli/native-zstd.ts</code> instead, which walks frame boundaries itself: Node&#39;s own zstd decoders stop after the first frame, and Spark writes thousands of small ones. A parser change that depends on chunk shape must hold for both. Native chunks are whole frames, often one full event line and up to tens of MB (<code>buildChunkDecoder</code> decodes those in 512 KiB slices). For local files, frames of 64 KB or more decompress off the main thread and arrive as 256 KB pieces, so a native <code>push()</code> is async and <code>streamFile</code> awaits it. fzstd&#39;s chunks are views of one reused buffer, valid only until its <code>ondata</code> callback returns: copy one before keeping it. On a Node without native zstd (before 22.15/23.8), and for a frame past 64 MiB, the Node path falls back to fzstd. The SHS archive loader decodes frames inline, and only <code>collectRun</code> uses the off-thread path.</p>',19)])])}const m=o(r,[["render",a]]);export{u as __pageData,m as default};
+import{_ as o,o as t,c as a,a5 as s}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Worker protocol","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol.md","filePath":"contributor-guide/architecture/worker-protocol.md"}'),d={name:"contributor-guide/architecture/worker-protocol.md"};function r(c,e,i,n,l,p){return t(),a("div",null,[...e[0]||(e[0]=[s('<h1 id="worker-protocol" tabindex="-1">Worker protocol <a class="header-anchor" href="#worker-protocol" aria-label="Permalink to &quot;Worker protocol&quot;">​</a></h1><h2 id="message-protocol" tabindex="-1">Message protocol <a class="header-anchor" href="#message-protocol" aria-label="Permalink to &quot;Message protocol&quot;">​</a></h2><p>Worker to main:</p><ul><li><code>app</code>, <code>stage</code>, <code>sql</code>, <code>executor</code>, <code>job</code>, <code>progress</code>: progressive, posted as the parser reads.</li><li><code>sqlPlan</code> (<code>{ type: &#39;sqlPlan&#39;, data: { executionId, planTree } }</code>): posted by <code>endSqlExecution</code> once a SQL execution&#39;s plan tree resolves. A repeated <code>SQLExecutionEnd</code> for that execution posts nothing, so no later <code>sql</code> message replaces the entry holding its <code>planTree</code>.</li><li><code>runAggregates</code>: one whole-run core-time-series summary (busy-core-ms, peak concurrency, per-stage task-duration sums), emitted just before <code>done</code>.</li><li><code>stageExecutorMetrics</code>: the post-completion re-post described in <a href="./overview.html#streaming">Streaming</a>, also emitted just before <code>done</code>.</li><li><code>stageSpeculationWaste</code>: the speculation totals of stages that gained waste from a TaskEnd after their StageCompleted, also described in <a href="./overview.html#streaming">Streaming</a> and emitted just before <code>done</code>.</li><li><code>stageLateAttemptWork</code>: the work of a failed stage attempt&#39;s tasks that ended after its StageCompleted, also described in <a href="./overview.html#streaming">Streaming</a> and emitted just before <code>done</code>.</li><li><code>done</code>, <code>taskData</code>, <code>error</code>.</li></ul><p><code>emitParseCompletion</code> posts the tail in a fixed order: any deferred AQE <code>sql</code> updates, <code>progress</code> (<code>pct: 1</code>), <code>stageLateAttemptWork</code>, <code>runAggregates</code>, <code>stageSpeculationWaste</code>, <code>stageExecutorMetrics</code>, the terminal <code>app</code>, then <code>done</code>.</p><p>A History Server failure is always the typed payload <code>{ type: &#39;error&#39;, source: &#39;shs&#39;, code, message? }</code>, where <code>code</code> is one of <code>local-server-unavailable</code>, <code>upstream-unreachable</code>, <code>application-not-found</code>, <code>access-or-upstream-failure</code>, or <code>invalid-event-log</code>. Fetch and HTTP failures carry only <code>code</code>: never an upstream message, URL, status, or response body. An <code>invalid-event-log</code> from the archive decode can carry a <code>message</code> written by <code>decodeShsArchive</code> (<code>packages/core/src/shs-fetch.ts</code>): the zip is unreadable, holds several application attempts or no event log, or an entry failed to decompress, in which case it names the zip entry and includes the decoder&#39;s error text. The intake shows that message under the recovery text.</p><p>Evidence availability has no dedicated worker message: the final <code>app</code> message carries a compact <code>evidenceInputs</code> counter summary, and <code>done.skippedLines</code> supplies its parse-integrity input.</p><p>Main to worker: <code>{ type: &#39;parse&#39;, file }</code>, <code>{ type: &#39;parseFiles&#39;, files }</code> (rolling <code>eventlog_v2_*</code> directories, one continuous stream across files), <code>{ type: &#39;parseFromUrl&#39;, request }</code> with <code>request = { baseUrl, appId, attemptId }</code>, and <code>{ type: &#39;getTaskData&#39;, stageId, reqId }</code>; <code>taskData</code> echoes <code>stageId</code> and <code>reqId</code> with <code>metrics</code> and <code>fieldNames</code>. The SHS request object is normalized before it reaches the worker: <code>{ baseUrl: string, appId: string, attemptId: string | null }</code>.</p><p>Files are read in 512 KiB slices (smaller for small files, so every file gets at least 100 reads), and <code>progress { pct, linesProcessed }</code> posts every 300 lines. <code>parseFromUrl</code> reports the download as 0-0.5, then <code>pct: null</code> every 2000 lines while parsing.</p><p><code>taskData</code> uses structured-clone (<code>.slice()</code>) so the worker retains its own <code>Float64Array</code> for re-renders.</p><p>Post-parse prefetch: after <code>done</code>, main runs <code>analyzer.ts</code> to build the bottleneck catalog, then fires parallel <code>getTaskData</code> for every flagged stage so their widgets render immediately. Unflagged stages are on-demand.</p><h3 id="decompress-worker" tabindex="-1">Decompress worker <a class="header-anchor" href="#decompress-worker" aria-label="Permalink to &quot;Decompress worker&quot;">​</a></h3><p>A dropped zstd file (<code>parse</code>, and each zstd file of a <code>parseFiles</code> directory) is decompressed in a second, nested worker, <code>packages/core/src/zstd-worker.ts</code>, so fzstd and the NDJSON parser run at the same time. The parse worker starts it on the first zstd file and reuses it for the rest of the parse. It dies with the parse worker, so the page&#39;s <code>terminate()</code> also cancels it. Other codecs and the SHS path (<code>parseFromUrl</code>) decompress on the parse worker. A dropped History Server zip (<code>parse</code>) streams its zstd entries through the decompress worker too.</p><p><code>packages/core/src/zstd-worker-client.ts</code> is the parse-worker end: it plugs into <code>streamFile</code> as the <code>zstdDecoder</code> option, like the Node CLI&#39;s native decoder. The messages&#39; buffers move by transfer rather than structured-clone copy (a read slice that is a view of a larger buffer is copied once first, and decoded output is copied once into a 1 MiB batch):</p><ul><li>Parse to decompress: <code>start { id }</code> opens a stream, and replaces any open one. <code>data { id, seq, bytes, final }</code> carries one compressed read slice. <code>cancel { id }</code> drops the stream.</li><li>Decompress to parse: <code>ready</code> once, at startup. <code>chunk { id, bytes, length }</code> carries decoded output in batches of up to 1 MiB. <code>consumed { id, seq }</code> follows the last <code>chunk</code> of slice <code>seq</code>. <code>error { id, message }</code> ends the stream, and its message becomes the usual <code>Could not decompress</code> error.</li></ul><p>Flow control is a window of three input slices: <code>push()</code> resolves while fewer than three slices wait for their <code>consumed</code>. Each <code>chunk</code> is parsed in its message handler, so a slice is acknowledged only after its output was parsed, and at most three slices&#39; output is ever queued. The final <code>push()</code> resolves once every slice is acknowledged. When <code>streamFile</code> gives up mid-stream (for example a failed file read) it calls <code>cancel()</code>. A parse exception inside a <code>chunk</code> handler makes the client post <code>cancel</code> itself and fail the stream, which surfaces on the next <code>push()</code>.</p><p>When the nested worker cannot start (<code>new Worker</code> throws or its script fails to load), the parse worker logs a warning and decodes with in-thread fzstd, with identical output. A crash after startup fails the stream it was decoding, and later streams fall back the same way. The progress <code>pct</code> is the read position, so it can run up to the window ahead of the slice being parsed. The self-contained <code>file://</code> export never parses in the browser (it opens with its run already analyzed), so it never starts either worker.</p><h3 id="zstd-in-the-browser-and-in-node" tabindex="-1">zstd in the browser and in Node <a class="header-anchor" href="#zstd-in-the-browser-and-in-node" aria-label="Permalink to &quot;zstd in the browser and in Node&quot;">​</a></h3><p>The browser decodes zstd with the vendored fzstd, not <code>DecompressionStream</code>. The Node CLI and MCP path (<code>collectRun</code>, <code>shs-load.ts</code>) uses <code>packages/core/src/cli/native-zstd.ts</code> instead, which walks frame boundaries itself: Node&#39;s own zstd decoders stop after the first frame, and Spark writes thousands of small ones. A parser change that depends on chunk shape must hold for both. Native chunks are whole frames, often one full event line and up to tens of MB (<code>buildChunkDecoder</code> decodes those in 512 KiB slices). For local files, frames of 64 KB or more decompress off the main thread and arrive as 256 KB pieces, so a native <code>push()</code> is async and <code>streamFile</code> awaits it. fzstd&#39;s chunks are views of one reused buffer, valid only until its <code>ondata</code> callback returns: copy one before keeping it. On a Node without native zstd (before 22.15/23.8), and for a frame past 64 MiB, the Node path falls back to fzstd. The SHS archive loader decodes frames inline, and only <code>collectRun</code> uses the off-thread path.</p><h3 id="decoder-and-parser-calibration" tabindex="-1">Decoder and parser calibration <a class="header-anchor" href="#decoder-and-parser-calibration" aria-label="Permalink to &quot;Decoder and parser calibration&quot;">​</a></h3><p>The decode and parse constants rest on measurements from real logs. &quot;The largest log&quot; is a 3.5 GB zstd log of 10,217 frames whose frames reach 39 MB.</p><table tabindex="0"><thead><tr><th>Constant or choice</th><th>Measurement</th></tr></thead><tbody><tr><td>Native frame walk (<code>native-zstd.ts</code>) instead of fzstd</td><td>3-5x faster: the largest log decodes in 1.3-1.8 s against 5.6 s</td></tr><tr><td>Off-thread decode only for frames of 64 KB or more</td><td>Median frame output is 5.8 KB; thresholds of 16-64 KB measure the same and 256 KB loses most of the gain</td></tr><tr><td><code>MAX_FRAMES_IN_FLIGHT</code> of 4</td><td>The largest log parses in 3.1 s at 711 MB peak RSS; 8 in flight take 3.2 s at 886 MB, 2 take 3.5 s at 607 MB. The pool&#39;s output chunk size (64 KB to 1 MB) barely matters</td></tr><tr><td><code>MAX_QUEUED_FRAMES</code> cap of 64</td><td>Without a cap, small frames queued behind an in-flight one raise peak RSS from 0.6 GB to 4.1 GB</td></tr><tr><td><code>FrameSplitter</code> resuming its walk across pushes</td><td>Re-copying and re-walking a frame on every push costs about 4 GB of copying near the 64 MiB frame limit</td></tr><tr><td>Off-thread decode overall</td><td>Parse time over the 14 calibration logs drops from 9.4 s to 8.1 s. A log with few large frames (12 of 6,455 on a 28 MB log) gains nothing</td></tr><tr><td>Parsing a second log on a worker thread (<code>collect-run-pair.ts</code>)</td><td>A worker costs about 50 ms to start and about 100 ms to clone the model of a 140 MB zstd log. A compressed log holds 7 to 12 times its size in event text, so the estimate uses 8</td></tr><tr><td>One <code>TextDecoder.decode</code> per chunk</td><td>About 36k decode calls on the largest log against 227k for a per-line decode, which is about 8x slower</td></tr><tr><td>Cutting <code>physicalPlanDescription</code> before <code>JSON.parse</code></td><td>The text is 72% of the AQE-update bytes, which are 73% of the log. On the largest log its values total 1.9 GB of the 3.5 GB stream, and one native frame holds all but 51 MB of a value</td></tr><tr><td>Deferring AQE updates</td><td>541 of 664 updates on the largest log are superseded, 1.1 MB of plan JSON each once the plan text is cut</td></tr><tr><td>String scan of <code>Accumulables</code> in <code>parseTaskEnd</code></td><td>The array is 71% of TaskEnd bytes on the largest log. Parsing its TaskEnd lines takes 0.5 s, against 1.3 s with a whole <code>JSON.parse</code>. Validating the scanned IDs would add 0.2 s</td></tr></tbody></table>',22)])])}const f=o(d,[["render",r]]);export{u as __pageData,f as default};

package/public/docs/assets/contributor-guide_architecture_worker-protocol.md.Br6OE7RW.lean.js

@@ -0,0 +1 @@
+import{_ as o,o as t,c as a,a5 as s}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Worker protocol","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol.md","filePath":"contributor-guide/architecture/worker-protocol.md"}'),d={name:"contributor-guide/architecture/worker-protocol.md"};function r(c,e,i,n,l,p){return t(),a("div",null,[...e[0]||(e[0]=[s("",22)])])}const f=o(d,[["render",r]]);export{u as __pageData,f as default};

package/public/docs/assets/contributor-guide_architecture_worker-protocol_cli-and-mcp.md.IVyaXrGe.js

@@ -0,0 +1 @@
+import{_ as o,o as c,c as d,a5 as t}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"CLI and MCP server","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md","filePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"}'),a={name:"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"};function s(r,e,n,i,l,h){return c(),d("div",null,[...e[0]||(e[0]=[t('<h1 id="cli-and-mcp-server" tabindex="-1">CLI and MCP server <a class="header-anchor" href="#cli-and-mcp-server" aria-label="Permalink to &quot;CLI and MCP server&quot;">​</a></h1><p>The headless analysis CLI and the MCP server, which run the same parser and detectors outside the browser.</p><h2 id="headless-analysis-cli" tabindex="-1">Headless analysis CLI <a class="header-anchor" href="#headless-analysis-cli" aria-label="Permalink to &quot;Headless analysis CLI&quot;">​</a></h2><p><code>packages/cli/bin/sparkforensics-analyze.mjs</code> runs the same parser + detector contracts outside the browser, for CI. It accepts a single event-log file or a rolling-log directory, drives <code>runParse</code>/<code>runParseFiles</code> (<code>packages/core/src/parser-worker.ts</code>) with a Node-only File-like shim (<code>nodeFileFromPath</code> in <code>packages/core/src/cli/collect-run.ts</code>), and writes the exact <code>buildEvidenceReport</code> JSON schema described above: one format, not a second. The shim reads each requested slice with a positioned <code>readSync</code>, so a local log is never loaded whole and has no 2 GiB size limit; <code>collectRun</code> closes every descriptor it opened once parsing settles, on success and on error. With <code>--baseline</code> and a local candidate, <code>collectRunPair</code> (<code>packages/core/src/cli/collect-run-pair.ts</code>) parses the smaller of the two logs on a worker thread (<code>collect-run-worker.js</code>) while the main thread parses the other, then posts the finished model back by structured clone, so a pair takes about as long as its bigger log. A thread start and the clone cost about 150 ms, so the worker is used only when the smaller log is expected to hold at least <code>MIN_OFFLOAD_PARSE_BYTES</code> of event text (its size on disk, times 8 for a compressed log); below that both logs parse on the main thread, baseline first. A worker that cannot start, runs out of memory or returns a model it cannot clone falls back to parsing on the main thread; a log the parser rejects fails with the same message either way. Both models are resident at once, so peak memory is the sum of both parses. Alternatively, <code>--shs-base-url &lt;url&gt; --app-id &lt;id&gt; [--attempt-id &lt;id&gt;]</code> fetches the run from a Spark History Server instead (mutually exclusive with the positional file/directory argument), calling <code>resolveFromShs</code> (<code>packages/core/src/shs-load.ts</code>, shared with the MCP server&#39;s SHS source path below) directly; no dependency on the <code>packages/server</code> package. The three flags are checked first with <code>validateShsRequest</code> (<code>packages/core/src/shs-request.js</code>), and a malformed one is a usage error (exit <code>2</code>) before any fetch. A failed SHS fetch reports its message to <code>stderr</code> and exits <code>4</code>, same as a candidate file that can&#39;t be parsed. <code>--app-id</code> repeats: two or more values with one <code>--shs-base-url</code> run the several-candidates mode (<code>runMultiLog</code>), each candidate resolved through <code>resolveFromShs</code>; <code>--app-id &lt;id&gt;:&lt;attempt&gt;</code> names an attempt per application, and <code>--attempt-id</code> is for a single <code>--app-id</code> only. The baseline has the same three flags (<code>--baseline-shs-base-url</code>, <code>--baseline-app-id</code>, <code>--baseline-attempt-id</code>), validated with the same <code>validateShsRequest</code>, mutually exclusive with <code>--baseline &lt;path&gt;</code>, and resolved with <code>resolveFromShs</code> too; a failed baseline fetch exits <code>5</code>. A History Server run and a local log load concurrently (CPU-bound local pairs use <code>collectRunPair</code>).</p><p>Regression budgets beyond the <code>--max-regression-pct</code> pair come from repeated <code>--regression-budget &lt;metric&gt;:&lt;pct&gt;</code> flags and a <code>--budgets</code> file, parsed in <code>packages/core/src/cli/regression-budgets.ts</code> and passed to <code>evaluateBudgets()</code> as <code>regressionBudgets</code>; the legacy pair is one more budget, and a metric budgeted twice is a usage error. Per-stage budgets (<code>--stage-regression-budget</code>, <code>--stage-quality</code>; <code>stageRegressionBudgets</code> and <code>stageQualities</code> on MCP) read the aligner&#39;s <code>stagePairs</code> through <code>checkStageRegression</code> in <code>cli/budgets.ts</code> and never match stages themselves; <code>parseStageRegressionBudgetFlag</code> refuses the neutral <code>inputBytes</code> and <code>outputBytes</code>. With two or more positional candidates (or <code>--format ndjson</code>), the CLI&#39;s <code>runMultiLog</code> parses and analyzes the baseline once, then evaluates each candidate in turn and writes one NDJSON line per candidate (<code>log</code>, <code>status</code>, <code>exitCode</code>, <code>error</code>, <code>budgets</code>, <code>candidate</code>, <code>comparison</code>). A candidate that can&#39;t be read or parsed gets a <code>status: &quot;error&quot;</code> line with <code>exitCode</code> <code>4</code> (<code>6</code> for an internal failure on that candidate); the exit code is the worst line (<code>6</code> &gt; <code>5</code> &gt; <code>4</code> &gt; <code>1</code> &gt; <code>3</code> &gt; <code>0</code>, <code>EXIT_SEVERITY</code>). A baseline that can&#39;t be read exits <code>5</code> with no lines. Under <code>--redact</code>, <code>log</code> and the <code>stderr</code> prefixes are <code>candidate-&lt;n&gt;</code> (1-based argument position) and an error line&#39;s message is generic, so no candidate path, which usually carries the app id, reaches the output. In both modes <code>--redact</code> also swaps the <code>stderr</code> text of a baseline or candidate failure (SHS fetch included) for <code>redactedFailure</code>&#39;s role-only message.</p><p>Optional CLI-flag budgets (<code>--max-runtime &lt;ms&gt;</code>, <code>--max-spill &lt;gb&gt;</code>, <code>--max-skew &lt;ratio&gt;</code>, <code>--max-failed-task-rate &lt;pct&gt;</code>, <code>--min-efficiency &lt;pct&gt;</code>) are evaluated in <code>packages/core/src/cli/budgets.ts</code> against the existing finding catalog (<code>analyze()</code>) and <code>computeEfficiencyModel</code>; there is no second rule engine. <code>--min-efficiency</code> compares <code>100 - wastagePct</code> (busy core time, the complement of the dashboard&#39;s Unused core time) and says so in its detail (&quot;Busy core time 26% below budget 90%.&quot;); it is not the Scorecard&#39;s Efficiency tile (<code>stagesActive / total</code>). A budget whose required evidence is missing (e.g. the run never emitted <code>ApplicationEnd</code>, or has no usable per-task <code>runAggregates</code>) is reported as inconclusive (<code>stderr</code> warning) rather than silently passing, and gets its own exit code distinct from both pass and violation. Exit codes: <code>0</code> pass, <code>1</code> a configured budget was violated, <code>2</code> a usage error (unknown or value-less flag, unknown <code>--regression-metric</code> key, an unreadable or invalid <code>--thresholds</code> or <code>--budgets</code> file), <code>3</code> no violations but at least one budget was inconclusive, <code>4</code> the candidate could not be read or parsed (or its SHS fetch failed), <code>5</code> the baseline could not be read or parsed, <code>6</code> an internal error (an unhandled failure, a failed <code>--export-html</code> export). <code>main</code> catches anything unhandled and exits <code>6</code> rather than letting Node exit <code>1</code>, which would read as a violation. When several apply the worst wins: <code>6</code>, <code>5</code>, <code>4</code>, <code>1</code>, <code>3</code>, <code>0</code>; a violation always wins over an inconclusive result in the same run (exit <code>1</code>, not <code>3</code>). <code>evaluateBudgets()</code> also always adds an inconclusive <code>run-complete</code> result when the catalog has an <code>incompleteRun</code> finding, so a run with no <code>ApplicationEnd</code> exits <code>3</code> even with no budget flags. With <code>--baseline</code>, the absolute budgets and this check apply to the candidate run; the MCP <code>evaluate_budgets</code> tool below uses the same function with the same roles.</p><h2 id="mcp-server" tabindex="-1">MCP server <a class="header-anchor" href="#mcp-server" aria-label="Permalink to &quot;MCP server&quot;">​</a></h2><p><code>packages/core/src/mcp-server-factory.ts</code>&#39;s <code>createMcpServer()</code> registers 8 tools: <code>list_runs</code> (candidate runs in a local directory or on a Spark History Server, to pick one before diagnosing it), <code>diagnose_run</code> (thresholded findings + remediation text), <code>get_run_summary</code> (app/stage/job/sql counts and duration, no findings), <code>compare_runs</code> (the comparison verdict from <code>comparisonVerdict</code> in <code>packages/core/src/comparison-verdict.ts</code>, the dashboard comparison page&#39;s own headline, plus categorized findings delta + metric deltas between two runs; <code>CompareRunsResult.jobOutcomes</code> carries each run&#39;s failed jobs and incomplete flag for it), <code>evaluate_budgets</code> (pass/fail budget thresholds against one run, optionally with a second run for regression/fail-on-introduced budgets: the MCP side of the CLI&#39;s <code>evaluateBudgets()</code> gating), <code>get_finding_evidence</code> (raw evidence bundle for one finding, for drill-down after <code>diagnose_run</code>), <code>get_finding_documentation</code> (detection/tuning reference docs for one finding type, independent of any run), and <code>get_reference_doc</code> (a full tuning-reference chapter or bottleneck page by doc anchor, e.g. <code>#joins</code>). None re-implement detector logic: <code>list_runs</code> lists candidates (<code>packages/core/src/list-runs.ts</code>), and the rest repackage <code>analyze</code>/<code>buildEvidenceReport</code>/<code>compareRuns</code>/<code>captureSnapshot</code>/<code>evaluateBudgets</code> from <code>packages/core/src/mcp-tools.ts</code>, which resolves a <code>source</code> (event-log <code>path</code>, or an SHS <code>shsBaseUrl</code>/<code>appId</code>/<code>attemptId</code> triple) into a cached <code>AppModel</code>. The SHS source path calls <code>resolveFromShs</code> (<code>packages/core/src/shs-load.ts</code>, also used directly by the CLI&#39;s <code>--shs-base-url</code> mode above), which reuses two extractions shared with the browser ingestion flow: <code>fetchShsEventLog</code> (<code>packages/core/src/proxy.js</code>, the same upstream fetch the local server&#39;s <code>/shs-proxy</code> route uses) fetches the zip, and <code>decodeShsArchive</code> (<code>packages/core/src/shs-fetch.ts</code>, re-exported from <code>parser-worker.ts</code>&#39;s barrel) streams it through the parser as worker messages, which <code>collectViaDispatch</code> assembles into an <code>AppModel</code>. <code>runParseFromUrl</code> calls the same <code>decodeShsArchive</code> after fetching through the proxy.</p><p>Two transports connect to that one factory: <code>packages/mcp/bin/sparkforensics-mcp.mjs</code> (stdio, for local MCP clients) and <code>packages/server/index.js</code>&#39;s <code>/mcp</code> route (streamable HTTP, for the local server). The run cache (<code>packages/core/src/mcp-tools.ts</code>) is a module-level LRU (cap 8 and 15-minute idle TTL by default, overridable via <code>SPARKFORENSICS_MCP_CACHE_CAP</code> and <code>SPARKFORENSICS_MCP_CACHE_TTL_MS</code>, lazily swept on access) keyed by resolved source (path plus mtime, ctime and size, or for a rolling-log directory the file count, newest mtime and ctime and total size of the files inside it, since Spark appends to the live part in place and leaves the directory&#39;s own stat alone; or SHS baseUrl+appId+attemptId), so a client mints a <code>runId</code> once via <code>resolveOrCreateRun</code> and reuses it across subsequent tool calls instead of re-parsing.</p><p><code>compare_runs</code> also caches each built comparison (cap 16, LRU) in <code>mcp-tools.ts</code>, keyed by both <code>runId</code>s, thresholds, <code>redact</code>, <code>normalizePath</code> and the <code>stagePairs</code> view. An entry is dropped with either of its runs, and is only stored while both runs are still cached when the build finishes. Each call gets a copy of the cached output, so a caller cannot change what the next one is served. <code>runOutputBlocks()</code> memoizes the <code>metrics</code> block per run model, thresholds and <code>redact</code>, so a repeated <code>diagnose_run</code> skips <code>computeRunMetrics</code>.</p><p><code>diagnose_run</code>&#39;s <code>view</code>, <code>minBand</code> and <code>maxFindings</code> live in <code>diagnoseRun()</code> (<code>mcp-tools.ts</code>), not in the report builder: the CLI report has one shape. <code>minBand</code> and <code>maxFindings</code> are part of <code>FindingsFilter</code>, so the Markdown findings follow them, and <code>buildEvidenceReport()</code> returns <code>findingsMatched</code>, the rows before <code>maxFindings</code>, for <code>findingsOmitted</code>. The <code>compact</code> view maps each <code>FindingRow</code> through <code>compactFindingRow()</code> and leaves out <code>cleanChecks</code> and <code>notRunChecks</code>. Each verdict step gets its lead row from the unfiltered report, which is cached, so a limit never hides it. The factory&#39;s <code>diagnose_run</code> handler defaults <code>view</code> to <code>compact</code>; <code>diagnoseRun()</code> itself defaults to <code>full</code>. With <code>format: &#39;md&#39;</code> the handler returns <code>diagnoseHeadline()</code> as <code>structuredContent</code>.</p><h3 id="one-output-builder-for-the-cli-and-mcp" tabindex="-1">One output builder for the CLI and MCP <a class="header-anchor" href="#one-output-builder-for-the-cli-and-mcp" aria-label="Permalink to &quot;One output builder for the CLI and MCP&quot;">​</a></h3><p>Neither surface lists the fields of a run or a comparison by hand. The CLI&#39;s JSON and the MCP tools call the same core functions:</p><ul><li><code>buildComparisonOutput()</code> in <code>packages/core/src/comparison-output.ts</code> diffs the two runs, applies <code>--redact</code>/<code>redact</code> itself (so no surface can budget or report an unredacted comparison by skipping it), and returns the redacted <code>CompareRunsResult</code> for the budget checks and the Markdown renderer, plus <code>comparisonOutput()</code>&#39;s projection: <code>verdict</code>, <code>confidence</code>, <code>reason</code>, <code>matchedCoverage</code>, <code>runtimeCoverage</code>, <code>metrics</code>, <code>findings</code>, and the comparison block <code>comparisonSchemaVersion</code>, <code>stagePairs</code>, <code>unmatched</code>, <code>replanned</code>, <code>bookkeepingStageIds</code>, <code>executionAlignment</code> (built by the aligner, see <a href="./../state-and-history/run-comparison-internals.html">Run comparison internals</a>). <code>comparisonOutput()</code> takes a <code>view</code>: <code>full</code> (the CLI) carries every field, <code>summary</code> (MCP&#39;s default) leaves out <code>stagePairs</code>, which <code>compare_runs</code> adds back for <code>include: [&#39;stagePairs&#39;]</code>. <code>buildComparisonOutput()</code> also takes <code>normalizePath</code>, the CLI&#39;s repeatable <code>--normalize-path</code> and <code>compare_runs</code>&#39; <code>normalizePath</code>: it reaches <code>compareRuns</code> the way <code>thresholds</code> reaches <code>analyze()</code> and stays out of the run cache key, because it only affects the comparison. The CLI validates the patterns before any log is parsed (exit 2) and MCP before any run is resolved. The same holds for <code>ignoreConf</code> (<code>--ignore-conf</code>) and <code>noiseBand</code> (<code>--noise-band-pct</code>), which <code>compareRuns</code> reads: <code>comparability.ts</code> builds the <code>comparability</code> block once from the two snapshots, the aligner&#39;s result and the metric rows, and <code>comparisonOutput()</code> projects it, so the CLI, MCP and the dashboard report the same block. That projection is the CLI&#39;s <code>comparison</code> object and the shared part of <code>compare_runs</code>. <code>compare_runs</code> also returns <code>metricDeltas</code> and <code>findingsDelta</code>, deprecated aliases of <code>metrics</code> and <code>findings</code> kept for one release.</li><li><code>runOutputBlocks()</code> in <code>packages/core/src/run-output.ts</code> builds the report&#39;s <code>metrics</code> and <code>effectiveConf</code> blocks, redacted from the redacted run, for both the CLI report and <code>diagnose_run</code>, which also returns the report&#39;s <code>writeTargets</code>. It also returns the <code>app</code> block (<code>id</code> and <code>attemptId</code>, from <code>run-identity.ts</code>) and the <code>generator</code> block when the surface passes one. Core does not know its host, so the CLI, the MCP bin and the local server supply their package name, version and core build id (<code>coreBuildId</code>). <code>createMcpServer({ generator })</code> reports that version in the MCP handshake and passes the block to <code>diagnose_run</code> and <code>compare_runs</code>. A server built without one reports version <code>0.0.0</code>.</li><li><code>comparisonOutput()</code> carries the same <code>generator</code> block and a <code>baselineApp</code> and <code>candidateApp</code> block, read from each run&#39;s <code>app</code> by <code>compareRuns</code>. Each findings row has <code>stageLabels</code> and each <code>stagePairs</code> row a <code>stageLabel</code>, the stage names through <code>stageDisplayLabel()</code> in <code>packages/core/src/stage-label.ts</code>, which reduces every path in a stage name to its last segment. It is display only: <code>normalizeStageName</code> stays the stage identity, so pairing does not move.</li></ul><p>Neither surface wraps the other: MCP handlers hold a run cache and report errors by code, while the CLI maps failures to exit codes and owns stdout. Adding a field to the projection adds it to both. <code>packages/cli/test/parity.test.js</code> runs the CLI&#39;s <code>main()</code> and the MCP tools through an in-memory client on public corpus logs, flattens both outputs and diffs their key sets and values. The differences that stay (the MCP run handles, the deprecated aliases, the CLI&#39;s <code>schemaVersion</code>) are in a commented allowlist in that file, and an entry that explains no difference fails the test.</p><p>Every tool failure comes back as <code>{isError: true, content: [...], structuredContent: {code}}</code>, never an HTTP-status-shaped error; <code>code</code> is one of the 5 existing SHS codes (<code>SHS_ERROR_CODES</code>, <code>packages/core/src/shs-request.js</code>) plus <code>run-not-found</code>, <code>finding-not-found</code>, <code>invalid-event-log</code> (also covers a nonexistent <code>path</code> source), <code>archive-too-large</code> (SHS archive over the <code>SPARKFORENSICS_MAX_ARCHIVE_BYTES</code> byte cap, default 1 GiB, because the MCP path buffers the whole archive in memory, unlike the streaming <code>/shs-proxy</code> route), <code>invalid-type</code>, <code>invalid-anchor</code> (documentation tools), <code>directory-not-found</code>, <code>invalid-date-filter</code>, and <code>invalid-shs-base-url</code> (<code>list_runs</code>). An error without a code reports <code>access-or-upstream-failure</code>. A stalled SHS archive body fails as <code>upstream-unreachable</code> after <code>SPARKFORENSICS_SHS_TIMEOUT_MS</code> (default 30 s) without data.</p><p><code>scripts/vendor-core.mjs</code> (shared by <code>packages/cli</code>, <code>packages/mcp</code>, and <code>packages/server</code>&#39;s <code>prepack</code> scripts) vendors <code>packages/core/src/</code> wholesale into each package&#39;s own <code>vendor-core/</code> at pack time, pre-stripping TypeScript to plain <code>.js</code> (Node&#39;s native TS stripping refuses to run on <code>.ts</code> files under <code>node_modules</code>, which is exactly where a published <code>vendor-core/</code> lands). Each package&#39;s bin/entry point resolves its needed module through <code>packages/core/src/load-vendored.js</code>: from <code>vendor-core/</code> if present, else from the real <code>packages/core/src/</code> sibling loaded as <code>.ts</code> directly. In a monorepo checkout a leftover <code>vendor-core/</code> is used only while its <code>core-source-hash.txt</code> (written by <code>vendor-core.mjs</code>) matches <code>packages/core/src</code>; otherwise the bin warns on stderr and loads <code>packages/core/src</code> (<code>packages/server/index.js</code> mirrors <code>resolveStaticRoot</code>&#39;s <code>public/</code>-vs-<code>../dist</code> pattern for this same fallback).</p>',17)])])}const g=o(a,[["render",s]]);export{p as __pageData,g as default};

package/public/docs/assets/{contributor-guide_architecture_worker-protocol_cli-and-mcp.md.BX1XnFbq.lean.js → contributor-guide_architecture_worker-protocol_cli-and-mcp.md.IVyaXrGe.lean.js}

@@ -1 +1 @@
-import{_ as o,o as c,c as d,a5 as a}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"CLI and MCP server","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md","filePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"}'),t={name:"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"};function s(r,e,n,i,l,h){return c(),d("div",null,[...e[0]||(e[0]=[a("",15)])])}const g=o(t,[["render",s]]);export{p as __pageData,g as default};
+import{_ as o,o as c,c as d,a5 as t}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"CLI and MCP server","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md","filePath":"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"}'),a={name:"contributor-guide/architecture/worker-protocol/cli-and-mcp.md"};function s(r,e,n,i,l,h){return c(),d("div",null,[...e[0]||(e[0]=[t("",17)])])}const g=o(a,[["render",s]]);export{p as __pageData,g as default};

package/public/docs/assets/{contributor-guide_architecture_worker-protocol_core-and-event-validation.md.DSVFPpWb.js → contributor-guide_architecture_worker-protocol_core-and-event-validation.md.DVSXn-bb.js}

@@ -1 +1 @@
-import{_ as o,o as t,c as a,a5 as c}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"TypeScript core and runtime event validation","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/core-and-event-validation.md","filePath":"contributor-guide/architecture/worker-protocol/core-and-event-validation.md"}'),d={name:"contributor-guide/architecture/worker-protocol/core-and-event-validation.md"};function r(s,e,n,i,l,p){return t(),a("div",null,[...e[0]||(e[0]=[c('<h1 id="typescript-core-and-runtime-event-validation" tabindex="-1">TypeScript core and runtime event validation <a class="header-anchor" href="#typescript-core-and-runtime-event-validation" aria-label="Permalink to &quot;TypeScript core and runtime event validation&quot;">​</a></h1><p>How the shared TypeScript core types and validates the events it reads.</p><p>All of <code>src/</code> (browser SPA) and <code>packages/core/src/</code> (shared analysis logic) is strict TypeScript, except for a few plain-<code>.js</code> files: the two vendored third-party decompressors, <code>packages/core/src/vendor/fflate.js</code> and <code>packages/core/src/vendor/fzstd.js</code> (left untouched deliberately: vendored code, not project code), <code>load-vendored.js</code> (copied byte-for-byte into <code>vendor-core/</code>, so it must run without TS stripping), and the SHS helpers <code>shs-request.js</code> and <code>proxy.js</code>, which the local server imports as plain JS. Every remaining import of a same-repo <code>.ts</code> module uses a <code>.ts</code> specifier (e.g. <code>import { dispatchLine } from &#39;./event-handlers.ts&#39;</code>), not <code>.js</code>: the CLI and MCP entrypoints (<code>packages/cli/bin/sparkforensics-analyze.mjs</code>, <code>packages/mcp/bin/sparkforensics-mcp.mjs</code>) run under plain Node&#39;s ESM resolver, which cannot remap a <code>.js</code> specifier to a same-named <code>.ts</code> file the way Vite/Vitest&#39;s bundler-style resolver can. <code>tsconfig.json</code> sets <code>&quot;allowImportingTsExtensions&quot;: true</code> to make this legal.</p><p>Event schema validation lives in <code>packages/core/src/event-schemas.ts</code>: one zod schema per <code>SparkListener*</code> event variant the parser understands (17 total: <code>LogStart</code>, <code>ApplicationStart</code>, <code>EnvironmentUpdate</code>, <code>ApplicationEnd</code>, <code>JobStart</code>, <code>JobEnd</code>, <code>StageSubmitted</code>, <code>StageCompleted</code>, <code>StageExecutorMetrics</code>, <code>TaskEnd</code>, the four SQL-execution-UI listener events, <code>ExecutorAdded</code>, <code>ExecutorRemoved</code>, <code>BlockUpdated</code>), combined into <code>SparkEventSchema = z.discriminatedUnion(&#39;Event&#39;, [...])</code>. <code>processEvent</code>&#39;s switch (<code>event-handlers.ts</code>) consumes the resulting <code>SparkEvent</code> union type directly, so a schema change and a handler&#39;s expectations can&#39;t silently drift apart.</p><p>The recursive <code>sparkPlanInfo.children</code> tree carried on SQL-execution-start events is validated by <code>parseSparkPlanInfoTree</code>, an iterative, explicit heap-allocated-stack parser, deliberately not <code>z.lazy()</code>. A real plan tree&#39;s depth is unbounded, and attacker-uncontrolled input should never hand zod&#39;s own recursive schema resolution an arbitrarily deep structure to walk. The iterative parser shallow-validates one node at a time via <code>z.object(...)</code> and builds the tree itself, capping at <code>MAX_PLAN_DEPTH = 500</code> (throws past that). This mirrors <code>packages/core/src/plan-tree-walk.ts</code>&#39;s <code>walkPlanTree</code>, which uses the same iterative-over-recursive approach for the <em>resolved</em> tree; <code>event-schemas.ts</code> applies it one layer earlier, to the raw JSON before it becomes a tree at all.</p><p>There are two external-data boundaries, the two places this codebase parses data it does not control. Both run that data through a schema, and both treat a validation failure the same way: a silent skip, not a distinct error.</p><ul><li><code>dispatchLine</code> (<code>event-handlers.ts</code>): after <code>JSON.parse</code> succeeds, a line whose <code>Event</code> value is one of the 17 modeled types but fails that type&#39;s own schema increments <code>skippedLines</code>. An <code>Event</code> value outside the 17 modeled types is silently ignored without incrementing <code>skippedLines</code>: real Spark logs always carry plenty of ordinary event types this tool does not model (<code>TaskStart</code>, <code>BlockManagerAdded</code>, <code>ExecutorMetricsUpdate</code>, and others), and counting them would trip <code>evidence-availability.ts</code>&#39;s fail-closed <code>trustworthy</code> gate on healthy logs. One exception: an AQE update that a later update for the same open execution supersedes is never parsed (<code>deferAdaptiveUpdate</code>, see <a href="./../detector-contract.html">the detector contract</a>), so a malformed superseded update is not counted.</li><li><code>shs-fetch.ts</code>: on a non-OK response from the local SHS proxy, the JSON error envelope is validated against <code>ShsProxyErrorBodySchema</code> (<code>packages/core/src/shs-schemas.ts</code>, <code>{ code: string }</code>, <code>.passthrough()</code>). A body that isn&#39;t valid JSON, or is JSON but fails that schema, falls back to the generic <code>access-or-upstream-failure</code> code: the same silent-skip treatment as a malformed log line. There is no separate &quot;malformed SHS response&quot; error code.</li></ul><p>Neither boundary distinguishes &quot;malformed JSON&quot; from &quot;wrong shape&quot; from &quot;unrecognized variant&quot; in what it reports outward: all three collapse into the same skip/fallback path. That is a deliberate scope decision.</p><p>The exhaustiveness convention is <code>packages/core/src/assert-never.ts</code>. <code>assertNever(x: never): never</code> throws at runtime and, more importantly, fails <code>tsc</code> at compile time if <code>x</code> is not actually <code>never</code>, i.e. if some case of a union type isn&#39;t handled. Used at the <code>default</code> arm of <code>event-handlers.ts</code>&#39;s <code>processEvent</code> switch (over <code>SparkEvent</code>) and <code>analyzer.ts</code>&#39;s scope-dispatch switch (over a detector&#39;s <code>scope: &#39;stage&#39; | &#39;sql&#39; | &#39;app&#39; | &#39;config&#39;</code>). It is the project&#39;s standard pattern for any future exhaustive switch or dispatch over a closed union: add <code>default: return assertNever(x);</code> (or the closest non-returning equivalent) so that adding a new union member without updating every consumer becomes a compile error instead of a silent runtime gap.</p>',9)])])}const m=o(d,[["render",r]]);export{u as __pageData,m as default};
+import{_ as o,o as t,c as a,a5 as c}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"TypeScript core and runtime event validation","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/core-and-event-validation.md","filePath":"contributor-guide/architecture/worker-protocol/core-and-event-validation.md"}'),d={name:"contributor-guide/architecture/worker-protocol/core-and-event-validation.md"};function r(s,e,n,i,l,p){return t(),a("div",null,[...e[0]||(e[0]=[c('<h1 id="typescript-core-and-runtime-event-validation" tabindex="-1">TypeScript core and runtime event validation <a class="header-anchor" href="#typescript-core-and-runtime-event-validation" aria-label="Permalink to &quot;TypeScript core and runtime event validation&quot;">​</a></h1><p>How the shared TypeScript core types and validates the events it reads.</p><p>All of <code>src/</code> (browser SPA) and <code>packages/core/src/</code> (shared analysis logic) is strict TypeScript, except for a few plain-<code>.js</code> files: the two vendored third-party decompressors, <code>packages/core/src/vendor/fflate.js</code> and <code>packages/core/src/vendor/fzstd.js</code> (left untouched deliberately: vendored code, not project code), <code>load-vendored.js</code> (copied byte-for-byte into <code>vendor-core/</code>, so it must run without TS stripping), and the SHS helpers <code>shs-request.js</code> and <code>proxy.js</code>, which the local server imports as plain JS. Every remaining import of a same-repo <code>.ts</code> module uses a <code>.ts</code> specifier (e.g. <code>import { dispatchLine } from &#39;./event-handlers.ts&#39;</code>), not <code>.js</code>: the CLI and MCP entrypoints (<code>packages/cli/bin/sparkforensics-analyze.mjs</code>, <code>packages/mcp/bin/sparkforensics-mcp.mjs</code>) run under plain Node&#39;s ESM resolver, which cannot remap a <code>.js</code> specifier to a same-named <code>.ts</code> file the way Vite/Vitest&#39;s bundler-style resolver can. <code>tsconfig.json</code> sets <code>&quot;allowImportingTsExtensions&quot;: true</code> to make this legal.</p><p>Event schema validation lives in <code>packages/core/src/event-schemas.ts</code>: one zod schema per <code>SparkListener*</code> event variant the parser understands (17 total: <code>LogStart</code>, <code>ApplicationStart</code>, <code>EnvironmentUpdate</code>, <code>ApplicationEnd</code>, <code>JobStart</code>, <code>JobEnd</code>, <code>StageSubmitted</code>, <code>StageCompleted</code>, <code>StageExecutorMetrics</code>, <code>TaskEnd</code>, the four SQL-execution-UI listener events, <code>ExecutorAdded</code>, <code>ExecutorRemoved</code>, <code>BlockUpdated</code>), combined into <code>SparkEventSchema = z.discriminatedUnion(&#39;Event&#39;, [...])</code>. <code>processEvent</code>&#39;s switch (<code>event-handlers.ts</code>) consumes the resulting <code>SparkEvent</code> union type directly, so a schema change and a handler&#39;s expectations can&#39;t silently drift apart.</p><p>The recursive <code>sparkPlanInfo.children</code> tree carried on SQL-execution-start events is validated by <code>parseSparkPlanInfoTree</code>, an iterative, explicit heap-allocated-stack parser, deliberately not <code>z.lazy()</code>. A real plan tree&#39;s depth is unbounded, and attacker-uncontrolled input should never hand zod&#39;s own recursive schema resolution an arbitrarily deep structure to walk. The iterative parser shallow-validates one node at a time via <code>z.object(...)</code> and builds the tree itself, capping at <code>MAX_PLAN_DEPTH = 500</code> (throws past that). This mirrors <code>packages/core/src/plan-tree-walk.ts</code>&#39;s <code>walkPlanTree</code>, which uses the same iterative-over-recursive approach for the <em>resolved</em> tree; <code>event-schemas.ts</code> applies it one layer earlier, to the raw JSON before it becomes a tree at all.</p><p>There are two external-data boundaries, the two places this codebase parses data it does not control. Both run that data through a schema, and both treat a validation failure the same way: a silent skip, not a distinct error. The one exception is a <code>TaskEnd</code>&#39;s accumulator IDs, read by a string scan without schema validation; see <a href="./../detector-contract/plan-attribution.html#stage-id-attribution-for-plan-advisor-findings">plan attribution</a>.</p><ul><li><code>dispatchLine</code> (<code>event-handlers.ts</code>): after <code>JSON.parse</code> succeeds, a line whose <code>Event</code> value is one of the 17 modeled types but fails that type&#39;s own schema increments <code>skippedLines</code>. An <code>Event</code> value outside the 17 modeled types is silently ignored without incrementing <code>skippedLines</code>: real Spark logs always carry plenty of ordinary event types this tool does not model (<code>TaskStart</code>, <code>BlockManagerAdded</code>, <code>ExecutorMetricsUpdate</code>, and others), and counting them would trip <code>evidence-availability.ts</code>&#39;s fail-closed <code>trustworthy</code> gate on healthy logs. One exception: an AQE update that a later update for the same open execution supersedes is never parsed (<code>deferAdaptiveUpdate</code>, see <a href="./../detector-contract.html">the detector contract</a>), so a malformed superseded update is not counted.</li><li><code>shs-fetch.ts</code>: on a non-OK response from the local SHS proxy, the JSON error envelope is validated against <code>ShsProxyErrorBodySchema</code> (<code>packages/core/src/shs-schemas.ts</code>, <code>{ code: string }</code>, <code>.passthrough()</code>). A body that isn&#39;t valid JSON, or is JSON but fails that schema, falls back to the generic <code>access-or-upstream-failure</code> code: the same silent-skip treatment as a malformed log line. There is no separate &quot;malformed SHS response&quot; error code.</li></ul><p>Neither boundary distinguishes &quot;malformed JSON&quot; from &quot;wrong shape&quot; from &quot;unrecognized variant&quot; in what it reports outward: all three collapse into the same skip/fallback path. That is a deliberate scope decision.</p><p>The exhaustiveness convention is <code>packages/core/src/assert-never.ts</code>. <code>assertNever(x: never): never</code> throws at runtime and, more importantly, fails <code>tsc</code> at compile time if <code>x</code> is not actually <code>never</code>, i.e. if some case of a union type isn&#39;t handled. Used at the <code>default</code> arm of <code>event-handlers.ts</code>&#39;s <code>processEvent</code> switch (over <code>SparkEvent</code>) and <code>analyzer.ts</code>&#39;s scope-dispatch switch (over a detector&#39;s <code>scope: &#39;stage&#39; | &#39;sql&#39; | &#39;app&#39; | &#39;config&#39;</code>). It is the project&#39;s standard pattern for any future exhaustive switch or dispatch over a closed union: add <code>default: return assertNever(x);</code> (or the closest non-returning equivalent) so that adding a new union member without updating every consumer becomes a compile error instead of a silent runtime gap.</p>',9)])])}const m=o(d,[["render",r]]);export{u as __pageData,m as default};

package/public/docs/assets/{contributor-guide_architecture_worker-protocol_evidence-availability.md.CjimIDBY.js → contributor-guide_architecture_worker-protocol_evidence-availability.md.D3S5cyOW.js}

@@ -1,6 +1,6 @@
-import{_ as t,o,c as a,a5 as d}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Evidence availability","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/evidence-availability.md","filePath":"contributor-guide/architecture/worker-protocol/evidence-availability.md"}'),s={name:"contributor-guide/architecture/worker-protocol/evidence-availability.md"};function n(i,e,c,r,l,p){return o(),a("div",null,[...e[0]||(e[0]=[d(`<h1 id="evidence-availability" tabindex="-1">Evidence availability <a class="header-anchor" href="#evidence-availability" aria-label="Permalink to &quot;Evidence availability&quot;">​</a></h1><p>How the parser records which evidence a log contained, and the shared taxonomy that reports it.</p><h2 id="evidence-availability-worker-input" tabindex="-1">Evidence-availability worker input <a class="header-anchor" href="#evidence-availability-worker-input" aria-label="Permalink to &quot;Evidence-availability worker input&quot;">​</a></h2><p><code>packages/core/src/event-handlers.ts</code>&#39;s <code>createState()</code>/per-event handlers start and increment this fixed <code>evidenceInputs</code> shape on the normalized app model:</p><div class="language-text vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">text</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span>environmentUpdates, applicationEnds, stageSubmissions, rddStorageSnapshots,</span></span>
+import{_ as t,o,c as a,a5 as d}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Evidence availability","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/evidence-availability.md","filePath":"contributor-guide/architecture/worker-protocol/evidence-availability.md"}'),s={name:"contributor-guide/architecture/worker-protocol/evidence-availability.md"};function n(c,e,i,r,l,p){return o(),a("div",null,[...e[0]||(e[0]=[d(`<h1 id="evidence-availability" tabindex="-1">Evidence availability <a class="header-anchor" href="#evidence-availability" aria-label="Permalink to &quot;Evidence availability&quot;">​</a></h1><p>How the parser records which evidence a log contained, and the shared taxonomy that reports it.</p><h2 id="evidence-availability-worker-input" tabindex="-1">Evidence-availability worker input <a class="header-anchor" href="#evidence-availability-worker-input" aria-label="Permalink to &quot;Evidence-availability worker input&quot;">​</a></h2><p><code>packages/core/src/event-handlers.ts</code>&#39;s <code>createState()</code>/per-event handlers start and increment this fixed <code>evidenceInputs</code> shape on the normalized app model:</p><div class="language-text vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">text</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span>environmentUpdates, applicationEnds, stageSubmissions, rddStorageSnapshots,</span></span>
 <span class="line"><span>sqlExecutions, resolvedSqlPlans, executorMetricRows, taskRecords</span></span></code></pre></div><p>The counters are a structured-clone-safe summary: they contain counts, not raw events, task records, host names, SQL text, paths, or Spark-property values. The final <code>app</code> message snapshots the counters after all input is processed, immediately before <code>done</code>; a consumer must not add a second worker message just for the ledger. <code>app</code> messages are also emitted mid-parse from <code>SparkListenerApplicationStart</code>, <code>SparkListenerApplicationEnd</code>, and a <code>SparkListenerEnvironmentUpdate</code> that arrives after the start; those snapshots are partial and non-authoritative. Only the terminal <code>app</code> message emitted by <code>emitParseCompletion</code> (the one immediately before <code>done</code>) should be used for evidence-availability conclusions. On the main thread, <code>useIngest</code> combines the normalized <code>AppModel</code> with <code>done.skippedLines</code>, derives the ledger before calling <code>analyze()</code>, and stores it as <code>appModel.evidenceAvailability</code>.</p><h2 id="evidence-availability-contract" tabindex="-1">Evidence-availability contract <a class="header-anchor" href="#evidence-availability-contract" aria-label="Permalink to &quot;Evidence-availability contract&quot;">​</a></h2><p><code>packages/core/src/evidence-availability.ts</code> is the single reusable taxonomy for the browser, the evidence report, and the headless CLI and MCP consumers. Its serialized ledger shape is <code>{ schemaVersion: 1, entries: EvidenceAvailabilityEntry[] }</code>. Entries are ordered by the following fixed eight-key enum:</p><div class="language-text vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">text</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span>executorMetrics, rddStorageSnapshots, sqlPlan, sparkConfiguration,</span></span>
 <span class="line"><span>taskCoreTime, infrastructureContext, sourceContext, costContext</span></span></code></pre></div><p>Each entry has the stable fields <code>key</code>, <code>state</code>, <code>reasonCode</code>, <code>summary</code>, and optional <code>evidence</code>. The closed state enum is:</p><div class="language-text vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">text</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span>present, disabled, notEmitted, notApplicable, outsideEventLog, unknown</span></span></code></pre></div><p>The closed reason-code enum is:</p><div class="language-text vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">text</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span>observed, explicitlyDisabled, noObservedExecutorMetrics,</span></span>
 <span class="line"><span>noObservedStageSubmission, noRddStorageSnapshot, noResolvedSqlPlan,</span></span>
 <span class="line"><span>noSqlExecution, noEnvironmentUpdate, noTaskRecords, noUsableCoreTimeAggregate,</span></span>
-<span class="line"><span>outsideEventLogScope, parseIncomplete</span></span></code></pre></div><p><code>summary</code> is domain-agnostic presentation copy derived from the reason code; the enum values, not labels, are the machine contract. Optional <code>evidence</code> is limited to safe compact provenance of the exact <code>{ eventType, count }</code> shape. <code>eventType</code> values are pinned to the exact <code>evidenceInputs</code> counter keys; they are not shortened or aliased. It must never include raw Spark-property values, host names, paths, SQL text, task records, raw-event identifiers, or other event payloads.</p><table tabindex="0"><thead><tr><th>Key</th><th>Proven <code>present</code> evidence</th><th>Trustworthy absence / boundary result</th></tr></thead><tbody><tr><td><code>executorMetrics</code></td><td><code>executorMetricRows &gt; 0</code></td><td>Explicit observed <code>spark.eventLog.logStageExecutorMetrics=false</code> is <code>disabled</code> / <code>explicitlyDisabled</code>; otherwise <code>notEmitted</code> / <code>noObservedExecutorMetrics</code>. Observed metrics win over the explicit setting.</td></tr><tr><td><code>rddStorageSnapshots</code></td><td><code>rddStorageSnapshots &gt; 0</code> from <code>SparkListenerStageSubmitted</code></td><td>No stage submissions is <code>notEmitted</code> / <code>noObservedStageSubmission</code>; stages submitted without any <code>RDD Info</code> rows is <code>notEmitted</code> / <code>noRddStorageSnapshot</code>. This is RDD snapshot evidence, not block-update telemetry.</td></tr><tr><td><code>sqlPlan</code></td><td><code>resolvedSqlPlans &gt; 0</code></td><td>A SQL execution without a resolved plan is <code>notEmitted</code> / <code>noResolvedSqlPlan</code>; no SQL execution is <code>notApplicable</code> / <code>noSqlExecution</code>.</td></tr><tr><td><code>sparkConfiguration</code></td><td><code>environmentUpdates &gt; 0</code></td><td><code>notEmitted</code> / <code>noEnvironmentUpdate</code>; a normalized empty config object alone is not evidence.</td></tr><tr><td><code>taskCoreTime</code></td><td><code>taskRecords &gt; 0</code> and a usable <code>runAggregates.perStage</code> task count</td><td>No task records is <code>notEmitted</code> / <code>noTaskRecords</code>; task records with no usable aggregate are <code>notEmitted</code> / <code>noUsableCoreTimeAggregate</code>. An empty aggregate object is not usable evidence.</td></tr><tr><td><code>infrastructureContext</code></td><td>none</td><td>Always <code>outsideEventLog</code> / <code>outsideEventLogScope</code>.</td></tr><tr><td><code>sourceContext</code></td><td>none</td><td>Always <code>outsideEventLog</code> / <code>outsideEventLogScope</code>.</td></tr><tr><td><code>costContext</code></td><td>none</td><td>Always <code>outsideEventLog</code> / <code>outsideEventLogScope</code>.</td></tr></tbody></table><p>Parse integrity is fail-closed. A parse is trustworthy only when it skipped no malformed lines and observed an application-end event. For an untrustworthy parse, proof already observed remains <code>present</code>, and an explicitly observed executor-metrics disablement remains <code>disabled</code>; every conclusion based on absence or irrelevance becomes <code>unknown</code> / <code>parseIncomplete</code>. The three outside-event-log product boundaries remain <code>outsideEventLog</code>. In particular, <code>notEmitted</code> and <code>notApplicable</code> are never favorable measurements inferred from an incomplete parse.</p><p>Availability is model metadata, not a detector or finding: it never changes detector thresholds or ordering, impact band, catalog/scorecard counts, no-bottleneck behavior, or detector suppression. <code>session-snapshot.ts</code> captures and restores <code>evidenceAvailability</code> with the normalized model, so a recent-file switch retains the same ledger without reparsing.</p>`,17)])])}const v=t(s,[["render",n]]);export{u as __pageData,v as default};
+<span class="line"><span>outsideEventLogScope, parseIncomplete</span></span></code></pre></div><p><code>summary</code> is domain-agnostic presentation copy derived from the reason code; the enum values, not labels, are the machine contract. Optional <code>evidence</code> is limited to safe compact provenance of the exact <code>{ eventType, count }</code> shape. <code>eventType</code> values are pinned to the exact <code>evidenceInputs</code> counter keys; they are not shortened or aliased. It must never include raw Spark-property values, host names, paths, SQL text, task records, raw-event identifiers, or other event payloads.</p><table tabindex="0"><thead><tr><th>Key</th><th>Proven <code>present</code> evidence</th><th>Trustworthy absence / boundary result</th></tr></thead><tbody><tr><td><code>executorMetrics</code></td><td><code>executorMetricRows &gt; 0</code>, counting stage executor metrics rows and task ends with a non-zero executor metric, never the driver&#39;s own rows (the heap-peak finding leaves them out too)</td><td>An effective <code>spark.eventLog.logStageExecutorMetrics</code> of <code>false</code> is <code>disabled</code> / <code>explicitlyDisabled</code>: the logged value, or Spark&#39;s default of off when the log recorded properties (the parser resolves it into <code>resources.stageExecutorMetricsLogging</code>); otherwise <code>notEmitted</code> / <code>noObservedExecutorMetrics</code>. Observed metrics win over the setting.</td></tr><tr><td><code>rddStorageSnapshots</code></td><td><code>rddStorageSnapshots &gt; 0</code> from <code>SparkListenerStageSubmitted</code></td><td>No stage submissions is <code>notEmitted</code> / <code>noObservedStageSubmission</code>; stages submitted without any <code>RDD Info</code> rows is <code>notEmitted</code> / <code>noRddStorageSnapshot</code>. This is RDD snapshot evidence, not block-update telemetry.</td></tr><tr><td><code>sqlPlan</code></td><td><code>resolvedSqlPlans &gt; 0</code></td><td>A SQL execution without a resolved plan is <code>notEmitted</code> / <code>noResolvedSqlPlan</code>; no SQL execution is <code>notApplicable</code> / <code>noSqlExecution</code>.</td></tr><tr><td><code>sparkConfiguration</code></td><td><code>environmentUpdates &gt; 0</code></td><td><code>notEmitted</code> / <code>noEnvironmentUpdate</code>; a normalized empty config object alone is not evidence.</td></tr><tr><td><code>taskCoreTime</code></td><td><code>taskRecords &gt; 0</code> and a usable <code>runAggregates.perStage</code> task count</td><td>No task records is <code>notEmitted</code> / <code>noTaskRecords</code>; task records with no usable aggregate are <code>notEmitted</code> / <code>noUsableCoreTimeAggregate</code>. An empty aggregate object is not usable evidence.</td></tr><tr><td><code>infrastructureContext</code></td><td>none</td><td>Always <code>outsideEventLog</code> / <code>outsideEventLogScope</code>.</td></tr><tr><td><code>sourceContext</code></td><td>none</td><td>Always <code>outsideEventLog</code> / <code>outsideEventLogScope</code>.</td></tr><tr><td><code>costContext</code></td><td>none</td><td>Always <code>outsideEventLog</code> / <code>outsideEventLogScope</code>.</td></tr></tbody></table><p>Parse integrity is fail-closed. A parse is trustworthy only when it skipped no malformed lines and observed an application-end event. For an untrustworthy parse, proof already observed remains <code>present</code>, and an explicitly observed executor-metrics disablement remains <code>disabled</code>; every conclusion based on absence or irrelevance becomes <code>unknown</code> / <code>parseIncomplete</code>. The three outside-event-log product boundaries remain <code>outsideEventLog</code>. In particular, <code>notEmitted</code> and <code>notApplicable</code> are never favorable measurements inferred from an incomplete parse.</p><p>Availability is model metadata, not a detector or finding: it never changes detector thresholds or ordering, impact band, catalog/scorecard counts, no-bottleneck behavior, or detector suppression. <code>session-snapshot.ts</code> captures and restores <code>evidenceAvailability</code> with the normalized model, so a recent-file switch retains the same ledger without reparsing.</p>`,17)])])}const v=t(s,[["render",n]]);export{u as __pageData,v as default};

package/public/docs/assets/{contributor-guide_architecture_worker-protocol_evidence-availability.md.CjimIDBY.lean.js → contributor-guide_architecture_worker-protocol_evidence-availability.md.D3S5cyOW.lean.js}

@@ -1 +1 @@
-import{_ as t,o,c as a,a5 as d}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Evidence availability","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/evidence-availability.md","filePath":"contributor-guide/architecture/worker-protocol/evidence-availability.md"}'),s={name:"contributor-guide/architecture/worker-protocol/evidence-availability.md"};function n(i,e,c,r,l,p){return o(),a("div",null,[...e[0]||(e[0]=[d("",17)])])}const v=t(s,[["render",n]]);export{u as __pageData,v as default};
+import{_ as t,o,c as a,a5 as d}from"./chunks/framework.DSg0KOwT.js";const u=JSON.parse('{"title":"Evidence availability","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/worker-protocol/evidence-availability.md","filePath":"contributor-guide/architecture/worker-protocol/evidence-availability.md"}'),s={name:"contributor-guide/architecture/worker-protocol/evidence-availability.md"};function n(c,e,i,r,l,p){return o(),a("div",null,[...e[0]||(e[0]=[d("",17)])])}const v=t(s,[["render",n]]);export{u as __pageData,v as default};