sparkforensics-server 0.2.3 → 0.2.4

package/public/docs/assets/{contributor-guide_architecture_detector-contract.md.B1vUqtry.js → contributor-guide_architecture_detector-contract.md.aQQfP7pj.js}

@@ -1 +1 @@
-import{_ as t,o,c as d,a5 as a}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Detector contract","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/detector-contract.md","filePath":"contributor-guide/architecture/detector-contract.md"}'),s={name:"contributor-guide/architecture/detector-contract.md"};function c(r,e,i,n,l,h){return o(),d("div",null,[...e[0]||(e[0]=[a('<h1 id="detector-contract" tabindex="-1">Detector contract <a class="header-anchor" href="#detector-contract" aria-label="Permalink to &quot;Detector contract&quot;">​</a></h1><p><code>packages/core/src/detectors.ts</code> is the single source of Spark-optimization logic: one declarative <code>DETECTORS</code> entry per pattern, each carrying <code>type</code>, <code>scope</code> (<code>stage</code> / <code>app</code> / <code>config</code> / <code>sql</code>), <code>order</code>, <code>fixEffort</code>, a <code>thresholds</code> object, impactBand/copy, a <code>docAnchor</code>, and a co-located <code>detect()</code> method. Both consumers are thin loops over that array:</p><ul><li><p><code>packages/core/src/analyzer.ts</code>: <code>analyze()</code> runs every entry regardless of scope, skipping only <code>inScorecard:false</code> ones; <code>auditConfig()</code> separately runs the <code>scope:&#39;config&#39;</code> entries. The four <code>configAudit</code> entries stay out of the bottleneck catalog because each sets <code>inScorecard:false</code>, not because of <code>scope:&#39;config&#39;</code>: a future config-scope detector without that flag would run through <code>analyze()</code> too. Each finding is stamped with its entry&#39;s <code>docAnchor</code>.</p></li><li><p><code>src/view/detector-registry.tsx</code>: a <code>REGISTRY: Record&lt;findingType, {component, region}&gt;</code> replaces <code>dashboard-renderer.js</code>&#39;s <code>render:</code> bindings, one entry per emitted finding type. <code>orderedWidgets()</code> walks <code>DETECTORS</code> ascending by <code>order</code>, then sorts <code>action</code>-region components before <code>reference</code>-region ones. Every <code>finding.type</code> maps to its own component now (the 2026-09 widget/finding-type 1:1 mapping redesign split six components that used to multiplex several types each: <code>TaskSkew</code> into <code>Skew</code>/<code>StageShape</code>/<code>TinyTask</code>; <code>ShuffleIO</code> narrowed to <code>shuffle</code> only, plus a new <code>PartitionSizing</code>; <code>Failures</code> into <code>StageFailed</code>/ <code>TaskFailures</code>/<code>RetryWaste</code>; <code>ExecutorTimeline</code> into <code>SlowHost</code>/ <code>StageSlowness</code>/<code>Straggler</code>/<code>SpeculationWaste</code>/<code>ColdStart</code> (its non-finding-driven executor-count chart moved to <code>ExecutorCountChart</code>, a <code>ReferenceSection</code> tile, not a <code>REGISTRY</code> entry); <code>MemoryUtilization</code> narrowed to <code>memoryUtilization</code> only, plus a new <code>ExecutorUtilization</code> for <code>utilization</code>; <code>PlanFindings</code> into <code>DuplicatePlanSubtree</code>/<code>SmallFiles</code>/ <code>UnderBroadcast</code>/<code>OverBroadcast</code>, dropping the dead <code>broadcastSizing</code> key entirely). No two <code>REGISTRY</code> entries share a <code>component</code> value any more.</p><p>Each widget component receives the full catalog and self-gates when it has nothing to show, rendering <code>null</code> or a muted &quot;no issue&quot; card for the always-visible ones. <code>orderedWidgets()</code> itself has no empty/non-empty branching, since it iterates the static <code>DETECTORS</code> import, not the runtime <code>catalog</code>.</p></li></ul><p>Thresholds live only in each entry&#39;s <code>thresholds</code>; see <a href="#bottleneck-thresholds-spec-§4">Bottleneck thresholds</a>.</p><h2 id="confidence-disclosure" tabindex="-1">Confidence disclosure <a class="header-anchor" href="#confidence-disclosure" aria-label="Permalink to &quot;Confidence disclosure&quot;">​</a></h2><p>A <code>Detector</code> entry (or the <code>Finding</code> it returns) may carry <code>confidence: &#39;low&#39; | &#39;medium&#39; | &#39;high&#39;</code> plus a <code>validationRequired</code> string. <code>RowStatusCluster</code> (<code>src/view/RowStatusCluster.tsx</code>) is the one place that renders it, gated to Advanced density: a plain &quot;&lt;confidence&gt; confidence&quot; badge whose tooltip carries the full <code>validationRequired</code> text. A finding with no <code>confidence</code> field renders identically to a fully-validated one, so every detector whose thresholds are our own unvalidated noise floor (marked <code>NOT SOURCED</code> in a code comment) should set both fields, not just the ones that happen to already have <code>RowStatusCluster</code> wired into their widget. <code>skew</code>, <code>straggler</code>, and <code>gc</code> set <code>confidence</code> for exactly this reason: their runtime-floor thresholds carry the same kind of unvalidated-noise-floor caveat <code>coreLocality</code>, <code>autoscalingChurn</code>, and <code>memoryUtilization</code>&#39;s <code>wasteModel</code> variant already disclose. None of these hardcode a single confidence value: each scales <code>&#39;low&#39; | &#39;medium&#39; | &#39;high&#39;</code> off how far the finding sits past its own detector&#39;s threshold, via a small named helper placed just above the <code>DETECTORS</code> array (e.g. <code>skewConfidence</code>, <code>coreLocalityConfidence</code>, <code>cachingReuseConfidence</code>) rather than an inline literal.</p><h2 id="the-fixeffort-field" tabindex="-1">The <code>fixEffort</code> field <a class="header-anchor" href="#the-fixeffort-field" aria-label="Permalink to &quot;The `fixEffort` field&quot;">​</a></h2><p>Each <code>Detector</code> entry also carries <code>fixEffort: &#39;config&#39; | &#39;code&#39; | &#39;rearchitect&#39;</code>, alongside <code>order</code> and <code>thresholds</code>: a rough estimate of how much work resolving the finding takes.</p><p>No view currently reads it. The quadrant impact/effort bucketing this field was meant to feed (<code>bucketFinding</code>/<code>effortTier</code>/<code>computeImpactMagnitude</code> in a since-deleted <code>src/quadrant-bucket.ts</code>, gated behind a <code>FIX_EFFORT_MAPPING_REVIEWED</code> flag that never flipped to <code>true</code>) was removed as dead code in the recommendations-consolidation redesign: <code>FixTheseFirst</code> (<code>src/view/widgets/FixTheseFirst.tsx</code>) ranks purely by impact magnitude. See <a href="./widget-rendering.html#widget-rendering-order-fixed-spec-§5">Widget rendering order</a> for how it ranks findings today.</p><p>Two shared helpers back multiple detectors and reports. <code>packages/core/src/plan-tree-walk.ts</code>&#39;s <code>walkPlanTree(root, visit, {dedupe})</code> is the iterative pre-order plan-tree traversal used by <code>detectors.ts</code> and every <code>plan-*.ts</code> module (<code>plan-summary.ts</code>, <code>plan-duration-attribution.ts</code>, <code>plan-node-detail.ts</code>, <code>plan-dot.ts</code>). <code>packages/core/src/core-count.ts</code>&#39;s <code>computeTotalCores(app, executorsAdded)</code> is the shared core-count logic used by <code>efficiency-model.ts</code>, <code>scaling-sim.ts</code>, and <code>wasted-core-hours.ts</code>. <code>detectors.ts</code>&#39;s own <code>utilization</code> and <code>memoryUtilization</code> entries use the same file&#39;s <code>computePeakConcurrentCores</code>/<code>computePeakConcurrentExecutorCount</code> instead: <code>computeTotalCores</code> sums every <code>ExecutorAdded</code> event with no regard for overlap, so under executor churn (spot preemption, <code>dynamicAllocation</code> replacement) it double-counts a churned executor&#39;s capacity against its replacement&#39;s; the peak-concurrent sweeps don&#39;t.</p><h2 id="cross-detector-suppression" tabindex="-1">Cross-detector suppression <a class="header-anchor" href="#cross-detector-suppression" aria-label="Permalink to &quot;Cross-detector suppression&quot;">​</a></h2><p>An entry may declare an optional <code>suppressWhen(finding, out)</code> method. <code>analyzer.ts</code>&#39;s <code>push()</code>, the single choke point every finding passes through, calls it per-finding, after the null guard and before the push, and drops the finding silently when it returns <code>true</code>. <code>out</code> is the findings accumulated so far. Since <code>analyze()</code>&#39;s loop is detector-outer / stage-inner, every finding from a detector declared earlier in <code>DETECTORS</code> is already in <code>out</code> by the time a later detector runs, for every stage. That makes the pattern purely declaration-order-driven: the suppressing detector must be declared earlier in the <code>DETECTORS</code> array than the suppressed one.</p><p><code>stageSlowness</code> uses this to defer to <code>slowHost</code>. It is spliced immediately after the <code>slowHost</code> entry regardless of its <code>order</code> field (<code>order</code> only controls render sequencing, not evaluation order), and <code>tests/analyzer.test.js</code>&#39;s &quot;detector contract&quot; suite asserts the array-index ordering so a future reorder can&#39;t silently break the suppression. The mechanism is deliberately minimal: a same-array, predicate-in-<code>push()</code> filter, not a general dependency graph. <code>auditConfig()</code>&#39;s own <code>push()</code> call is unaffected, since <code>scope:&#39;config&#39;</code> entries declare no <code>suppressWhen</code>.</p><h2 id="per-operator-duration-attribution" tabindex="-1">Per-operator duration attribution <a class="header-anchor" href="#per-operator-duration-attribution" aria-label="Permalink to &quot;Per-operator duration attribution&quot;">​</a></h2><p><code>packages/core/src/plan-duration-attribution.ts</code> (entry <code>attributeStageDurationToPlan(planTree, stagesById, sqlExec)</code>) approximates how a SQL execution&#39;s stage wall-time splits across plan operators, returning a <code>Map&lt;planNode, milliseconds&gt;</code>. It cuts the plan tree at Exchange boundaries into connected components: since the Exchange write/read split (<code>resolvePlanTree</code> in <code>event-handlers.ts</code> always synthesizes a <code>read</code> node wrapping a <code>write</code> node for every raw <code>Exchange</code>/<code>BroadcastExchange</code>), the cut is keyed off <code>PlanNode.exchangeRole === &#39;read&#39;</code> on the parent, not a name regex: the write half starts the new component, the read half stays in its parent&#39;s. A node with no <code>exchangeRole</code> at all (for example <code>ReusedExchange</code>, which is never split) never starts a new component on its own, unlike the old name-based regex, which matched any Exchange-family name regardless of split state. Each component receives a stable pre-order identity and separate parent/depth/traversal metadata; the identity itself does not encode its count of Exchange ancestors. It zips components deepest-first by that explicit depth against submission-ordered stage IDs, then apportions each matched stage&#39;s wall-time across that component&#39;s nodes by timing-metric weight, falling back to an even split when no node carries a timing metric.</p><p>This is best-effort inference, not measurement. Spark&#39;s event model exposes no ground truth for per-operator time within a stage; the Exchange-boundary segmentation and deepest-component-to-earliest-stage zip are heuristics. Treat the per-operator numbers as directional hints, never as authoritative timings, and do not build hard thresholds or findings on top of them.</p><h2 id="stage-id-attribution-for-plan-advisor-findings" tabindex="-1">Stage-ID attribution for Plan Advisor findings <a class="header-anchor" href="#stage-id-attribution-for-plan-advisor-findings" aria-label="Permalink to &quot;Stage-ID attribution for Plan Advisor findings&quot;">​</a></h2><p>The Plan Advisor detectors (<code>duplicatePlanSubtree</code>, <code>smallFiles</code>, <code>broadcastSizing</code> in <code>packages/core/src/detectors.ts</code>) each attribute their finding to a narrowed <code>stageIds</code> set rather than the whole SQL execution: <code>PlanNode.stageIds</code> is resolved once per plan tree at parse time by unioning, per node, every metric&#39;s accumulator ID against a <code>taskAccumStages: Map&lt;accumulatorId, Set&lt;stageId&gt;&gt;</code> built while parsing <code>TaskEnd</code> events, then clipping the result to the execution&#39;s own stage set. An accumulator ID occasionally points to a <em>different</em> execution&#39;s stages, e.g. a <code>ReusedSubquery</code> computed once and reused verbatim, and the clip prevents misattributing that other execution&#39;s work. Each detector unions its implicated node(s)&#39; <code>stageIds</code> and falls back to the execution-wide set only when no implicated node has any coverage; a finding never partially blends a narrowed set with the execution-wide one. When an execution has no stage universe at all (no jobs ever recorded against it, which is true for 42% of real-log SQL executions with a plan tree, typically job-less/driver-only executions), the clip drops every candidate stage ID instead of passing them through: every node in that execution&#39;s tree ends up with no <code>stageIds</code> anywhere, same &quot;coverage is partial&quot; framing as below. (An earlier version of this clip treated &quot;no stage universe&quot; as &quot;no clip,&quot; which let a foreign accumulator ID collision, e.g. the <code>ReusedSubquery</code> case above, leak another execution&#39;s stages into a job-less execution&#39;s nodes; the clip is now unconditional on <code>executionStageIds</code> being present.)</p><p>Coverage is partial by Spark&#39;s own design: whole-stage-codegen wrapper nodes (<code>InputAdapter</code>, and other purely structural passthrough markers) carry no accumulators at all, and <code>BroadcastExchangeExec</code>&#39;s own metrics are computed entirely on the driver and never appear on any <code>TaskEnd</code> (real Spark behavior). Since the Exchange write/read split, those driver-computed metrics live specifically on the synthesized <em>write</em> half (<code>exchangeRole: &#39;write&#39;</code>); the <em>read</em> half always carries <code>metrics: []</code>. The write half&#39;s immediate child, which does carry executor-side metrics, is unioned in instead, see <code>overBroadcast</code>&#39;s wiring. A <code>TaskEnd</code> arriving after its stage has already been finalized is also silently excluded from <code>taskAccumStages</code>, consistent with the parser&#39;s existing out-of-order tolerance elsewhere.</p><p><code>planTree</code> itself is kept current against Spark&#39;s adaptive query execution (AQE) re-plans: <code>SparkListenerSQLAdaptiveExecutionUpdate</code> events overwrite the execution&#39;s <code>sparkPlanInfo</code> last-write-wins, so accumulator-ID evidence is matched against the plan that actually ran rather than a stale pre-AQE snapshot. Since a superseded plan is never read, <code>dispatchLine</code> holds an open execution&#39;s latest update as unparsed text and parses only that one, when <code>SQLExecutionEnd</code> arrives or at parse completion (<code>deferAdaptiveUpdate</code>, <code>event-handlers.ts</code>). It recognizes an update from the flat first and last pieces <code>buildChunkDecoder</code> reports for a line joined across slices (<code>JoinedLine</code>), so a superseded update is never copied into one flat string either. The raw <code>sparkPlanInfo</code> stays worker-side and is released once <code>SQLExecutionEnd</code> resolves it into <code>planTree</code>; <code>physicalPlanDescription</code> (Spark&#39;s text rendering of the plan, which nothing reads) is emptied before <code>JSON.parse</code> and never retained (<code>stripPlanDescription</code>, <code>event-handlers.ts</code>). <code>buildChunkDecoder</code> decodes each decompressed chunk in slices of at most 512 KiB, and drops the bytes of a value that crosses a slice boundary without decoding them.</p><p>No eviction/pruning is added to <code>taskAccumStages</code>, a deliberate choice, not an oversight: measured on real logs, it holds roughly 1,050 keys per compressed MB (9,850 keys on an 11.6 MB fixture, about 29,000 keys on a 28.1 MB fixture). Extrapolated to a 240MB+ log, the scale this tool targets (see <code>CLAUDE.md</code>), that is roughly 250,000 keys, around 45 MB of heap for an equivalent synthetic <code>Map&lt;number, Set&lt;number&gt;&gt;</code>. This heap estimate is still small relative to this tool&#39;s other in-memory state. It is higher, though, than the fixture-only measurements taken when this mechanism was built suggested.</p><p>Per-execution pruning (e.g. dropping a <code>taskAccumStages</code> entry once its stage finalizes or its owning SQL execution resolves, mirroring how <code>accumState</code> is cleared in <code>endSqlExecution</code>) is deliberately not done either: unlike <code>accumState</code>, <code>taskAccumStages</code> is one global, un-scoped map read by every execution&#39;s <code>resolvePlanTree</code> call, and the <code>ReusedSubquery</code> case above depends on a stage recorded under one execution still being visible when a later execution resolves. Pruning on any single execution&#39;s lifecycle would break that cross-execution lookup. What is bounded is the growth from a single pathological event: <code>TaskEndEventSchema</code>&#39;s <code>Accumulables</code> array is capped at <code>MAX_ACCUMULABLES_PER_TASK</code> (10,000, <code>event-schemas.ts</code>), well above any real plan&#39;s per-task metric count, so a single crafted <code>TaskEnd</code> can&#39;t grow the map past that per-event bound; a <code>TaskEnd</code> exceeding it fails schema validation and the line is skipped (counted in <code>skippedLines</code>) like any other malformed event. Each accumulable is checked for its numeric <code>ID</code> only: <code>Update</code>/<code>Value</code> are unread, and Spark writes some as JSON arrays (<code>internal.metrics.updatedBlockStatuses</code>), which a stricter check would reject along with the whole task. When every entry has Spark&#39;s flat <code>{&quot;ID&quot;:n,...}</code> form, <code>parseTaskEnd</code> (<code>event-handlers.ts</code>) reads the IDs with a string scan and cuts the array out before <code>JSON.parse</code>; any other shape is parsed whole.</p><h2 id="bottleneck-thresholds-spec-§4" tabindex="-1">Bottleneck thresholds (spec §4) <a class="header-anchor" href="#bottleneck-thresholds-spec-§4" aria-label="Permalink to &quot;Bottleneck thresholds (spec §4)&quot;">​</a></h2><p>Every change to <code>packages/core/src/detectors.ts</code> should reference this table.</p><p>Every finding&#39;s <code>impactBand</code> comes from one of two places. For any finding whose <code>impactEstimate</code> carries a <code>wallClock</code> estimate (the common case for most rules below), <code>analyzer.ts</code> calls <code>deriveImpactBand()</code> (<code>packages/core/src/impact-band.ts</code>) immediately after <code>estimateImpact()</code>, which sets <code>.impactBand</code> purely from <code>wallClock.high</code> as a fraction of the app&#39;s total duration (<code>&gt;= 2%</code> critical, <code>&gt;= 0.5%</code> warning, else info: the same <code>floorPctWarn</code>/<code>floorPctCrit</code> values <code>skew</code>/<code>straggler</code> use for their own thresholds below). For those rules, the table below documents their firing gate plus their fixed fallback constant, which surfaces only when this run&#39;s finding of that type didn&#39;t get a wallClock estimate (a stage excluded from the occupancy sweep). For rules whose finding type never gets a wallClock estimate (<code>resourceOnly</code>/<code>informational</code> basis, e.g. <code>configAudit</code>, or a rule that keeps its own ratio-tiered classification per the design&#39;s Decision 2, e.g. <code>failures</code>), the full threshold table below is the real, displayed classification: <code>detectors.ts</code> sets <code>impactBand</code> directly and nothing overwrites it. <code>partitionSizing</code>&#39;s <code>maxPartitionTooBig</code> rule is a third case: it does carry a <code>wallClock</code> estimate but is explicitly exempted in <code>deriveImpactBand()</code> because it&#39;s a hardcoded-critical OOM/crash-risk safety signal, not a time-recovery one, so <code>detectors.ts</code>&#39;s own classification stands regardless of how small that estimate is relative to the run.</p><h3 id="fixed-fallback-only-usually-wallclock-derived-instead" tabindex="-1">Fixed fallback only (usually wallClock-derived instead) <a class="header-anchor" href="#fixed-fallback-only-usually-wallclock-derived-instead" aria-label="Permalink to &quot;Fixed fallback only (usually wallClock-derived instead)&quot;">​</a></h3><p>These rules&#39; <em>band</em> tiers were deleted from <code>packages/core/src/detectors.ts</code> (they were always overwritten by <code>deriveImpactBand</code> whenever a wallClock estimate was available); the constant in the last column is only a floor-case fallback. Their <em>firing</em> gate is untouched and still lives in each entry&#39;s <code>thresholds</code> object: it decides whether the rule reports anything, so it stays documented here in full.</p><table tabindex="0"><thead><tr><th>Rule</th><th>Fires when</th><th>Fallback</th></tr></thead><tbody><tr><td>Task skew</td><td><code>taskDurationP95 / taskDurationP50 &gt; 3×</code> (<code>taskDurationMax / P50</code> for stages under <code>minTasksForP95</code> = 20 tasks), <strong>and</strong> the occupancy-clipped tail recovery (<code>tailRecoveryMs</code>: P95−P50, or max−P50, or the slow tasks&#39; summed excess over the stage&#39;s peak slots when larger) is ≥ <code>floorPctWarn</code> = 0.5% of app runtime. The clip floors the claim at the longest task the fix leaves, not the current one, and at the core work the fix leaves over every core (see impact-estimation.md&#39;s occupancy section); <code>straggler</code>&#39;s floors use the same clip</td><td><code>warning</code></td></tr><tr><td>Shuffle read</td><td><code>shuffleReadBytes &gt; minBytes</code> = 50 MiB, on a stage that isn&#39;t shorter than <code>stageFloorPct</code> = 0.5% of the run (a zero-length stage or an unknown run duration passes; the claim is clipped to the stage, so a shorter one graded <code>info</code>: 182 of 284 on the 14 real logs, the shuffle itself still there)</td><td><code>info</code></td></tr><tr><td>Partition sizing: skew</td><td><code>shuffleReadMax &gt; 5×</code> <code>shuffleReadP50</code> <strong>and</strong> <code>shuffleReadMax &gt; 256 MiB</code></td><td><code>warning</code></td></tr><tr><td>Partition sizing: low parallelism</td><td><code>shuffleReadBytes ≥ 1 GiB</code> <strong>and</strong> <code>taskCount ≤ 7</code></td><td><code>warning</code></td></tr><tr><td>Partition sizing: oversized partition</td><td><code>shuffleReadMax ≥ 5 GiB</code></td><td><code>critical</code></td></tr><tr><td>GC</td><td><code>executorRunTime ≥ minRunTimeMs</code> = 10 s <strong>and</strong> <code>gcPct &gt; 10%</code></td><td><code>warning</code></td></tr><tr><td>GC (low / cost)</td><td><code>executorRunTime ≥ 10 s</code> <strong>and</strong> <code>gcPct &lt; lowInfoPct100</code> = 5% (checked only when the GC row above did not fire) <strong>and</strong> the stage lasts ≥ <code>lowInfoFloorPct</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length, such as one with no completion time). Gets no wall-clock estimate (an over-provisioning signal), so this band always stands. The floor dropped 464 of 685 low-GC notes on the 14 real logs (none on the corpus): low GC is still true on those stages, but a memory-sizing note from a stage that barely ran adds nothing</td><td><code>info</code></td></tr><tr><td>Spill (magnitude v2)</td><td>any non-zero <code>memoryBytesSpilled</code>, on a stage that isn&#39;t shorter than <code>stageFloorPct</code> = 0.5% of the run (as for shuffle read: 12 of 38 on the 14 real logs, all <code>info</code>). The magnitude sub-table below classifies <em>how much</em>, but does not gate firing</td><td><code>warning</code></td></tr><tr><td>Cold start</td><td><code>firstExecutorAddedAt − firstStageSubmittedAt &gt; gapSeconds</code> = 30 s (no finding without executor-added events): the time a runnable stage waited for its first executor. An executor added before the first stage and still alive at submission means no wait and no finding; one removed at or before submission is ignored, so the gap runs to the next executor added after it. It used to be <code>firstStageSubmittedAt − app.startTime</code>, the driver&#39;s own startup, which read 36-47 s on all 9 real logs that fired whatever the executors did, and flagged 6 corpus logs whose first executor was up 44-313 s before the first stage (5 of them) or arrived 4.4 s after it</td><td><code>warning</code></td></tr><tr><td>Slow host: mean-duration ratio</td><td>stage has ≥ <code>minHosts</code> = 3 hosts (or executors) and ≥ <code>minTasks</code> = 15 tasks, and lasts ≥ <code>stageFloorPct</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length, which gets no estimate and keeps its fallback band; every slow-host finding on a shorter stage graded <code>info</code>, 324 of 452 on the 14 real logs, the imbalance itself still true; this gate covers every slowHost row); then per host: mean task duration / overall median ≥ <code>ratioWarn</code> = 2.0× <strong>and</strong> host task-share ≥ <code>minShare</code> = 20% <strong>and</strong> host mean ≥ <code>floorMs</code> = 1000 ms (absolute-magnitude floor, rules out sub-second noise)</td><td><code>warning</code></td></tr><tr><td>Slow host: duration-share</td><td>same stage gate as the row above; then per host: ≥ <code>shareWarn</code> = 75% of the stage&#39;s total task-duration <strong>and</strong> ≥ <code>taskShareWarn</code> = 50% of its task count</td><td><code>warning</code></td></tr><tr><td>Stage slowness: absolute fallback, suppressed when <code>slowHost</code> already fired</td><td>stage wall-clock duration ≥ <code>infoMin</code> = 15 min. The band then comes from the partitioning-headroom estimate (see impact-estimation.md), not the duration</td><td><code>info</code></td></tr><tr><td>Straggler / speculative-execution</td><td><code>taskCount ≥ minTasks</code> = 10, <strong>and</strong> the stage lasts ≥ <code>floorPctWarn</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length, such as one with no completion time; a shorter stage&#39;s tail can&#39;t cost more than its own duration, so every finding there graded <code>info</code>: 671 of 753 on the 14 real logs, 3 on the corpus, the slow tail itself still true), <strong>and</strong> either any speculative task ran, <strong>or</strong> straggler share &gt; <code>shareWarn</code> = 5%, <strong>or</strong> straggler share &gt; <code>shareWarnAtFloor</code> = 2.5% with the occupancy-clipped tail recovery (<code>tailRecoveryMs</code>, as for skew, but its single-task delta stops at the longest task under 4× P50 rather than at P50) already ≥ <code>floorPctWarn</code> (0.5% of app runtime). The lower gate is scored against a task-level replay of every stage on 14 real logs (recoverable = list-scheduling replay with each task over 4× P50 capped at P50; positive = ≥ 0.5% of app runtime): it found 3 stages whose stragglers gated them for 10-48s at 2.7-4% of their tasks, for 1 borderline miss, lifting skew-or-straggler recall from 0.86 to 0.92 at precision 0.91 → 0.89. Admitting every 2.5% share instead would add 86 findings below the floor. The same sweep kept skew&#39;s <code>ratioWarn</code> = 3 (2.5 added false positives, 4 lost true ones) and the 0.5% floor (1% halved skew recall). <code>warnPct</code>/<code>critPct</code> (10%/20% speculative share) and <code>floorPctWarn</code>/<code>floorPctCrit</code> (0.5%/2% of app runtime) no longer set the band; they rank the straggler-vs-speculative tiers that pick which <em>metric</em> the finding reports</td><td><code>info</code></td></tr><tr><td>Speculation waste (new)</td><td><code>speculationWastedAttempts ≥ minWasted</code> = 5 <strong>and</strong> <code>speculationWasteMs ≥ minWasteMs</code> = 60 s</td><td><code>warning</code></td></tr><tr><td>Retry waste</td><td><code>wastedAttempts ≥ minWasted</code> = 3 <strong>and</strong> <code>retryWasteMs ≥ minWasteMs</code> = 30 s, on a stage that still completed</td><td><code>warning</code></td></tr><tr><td>Tiny tasks</td><td><code>taskCount ≥ minTasks</code> = 100 <strong>and</strong> <code>taskDurationP50 ≤ maxP50</code> = 500 ms <strong>and</strong> <code>taskDurationP95 ≤ maxP95</code> = 1000 ms <strong>and</strong> the stage lasts ≥ <code>stageFloorPct</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length; coalescing can&#39;t save more than the stage&#39;s own duration, so every finding on a shorter stage graded <code>info</code>: 132 of 151 on the 14 real logs, the tasks still tiny). Checked (2026-09-23) against each stage&#39;s measured per-task overhead (task wall time minus executor run time, the estimate&#39;s own input) on 14 real logs and the corpus: of 228 stages with 100+ tasks outside the P50/P95 gate, none would save 0.5% of its run by coalescing (median overhead 0.2-3% of task time); of 170 inside it, 25 would, and all 25 grade above <code>info</code></td><td><code>info</code></td></tr><tr><td>Duplicate plan subtree</td><td>a subtree of ≥ <code>minSubtreeSize</code> = 3 nodes whose shape fingerprint repeats ≥ <code>minOccurrences</code> = 2× in the plan, unless its linked stages together lasted less than <code>stageFloorPct</code> = 0.5% of the run (a repeat with no linked stage time, or an unknown run duration, is kept; the claim counts at most each stage&#39;s own task-active time, so such a repeat graded <code>info</code>: 340 of 545 on the 14 real logs, the repeat still in the plan). <code>occurrencesIdentical</code> records whether the repeats also agree node-for-node on normalized detail, ignoring AQE query-stage numbers; when they don&#39;t (same shape over another table, filter or projection: 270 of 546 groups on the 14 real logs) the finding is <code>info</code> with confidence <code>low</code> and no time claim. <code>stageShares</code> gives, per stage, the repeated operators&#39; share of the stage&#39;s operators (WholeStageCodegen wrappers and Exchange write halves not counted); with no attributed stage the finding is <code>info</code></td><td><code>warning</code></td></tr><tr><td>Small files read/write</td><td>per read/write side: file count &gt; <code>minFiles</code> = 100 <strong>and</strong> average file size &lt; <code>maxAvgFileSizeMB</code> = 3 MiB</td><td><code>warning</code></td></tr><tr><td>Broadcast sizing: missed</td><td>a 2-child <code>SortMergeJoin</code> whose smaller side is &lt; 10 MiB (unconditional), or &lt; 100 MiB with the larger side &gt; 10 GiB, or &lt; 1 GiB with larger &gt; 300 GiB, or &lt; 5 GiB with larger &gt; 1 TiB (<code>broadcastTiers</code> × <code>comparisonTiers</code>)</td><td><code>info</code></td></tr><tr><td>Broadcast sizing: oversized</td><td>a <code>BroadcastExchange</code> node whose <code>data size</code> metric &gt; <code>overBroadcastBytes</code> = 1 GiB</td><td><code>warning</code></td></tr></tbody></table><h4 id="spill-magnitude-tiers" tabindex="-1">Spill magnitude tiers <a class="header-anchor" href="#spill-magnitude-tiers" aria-label="Permalink to &quot;Spill magnitude tiers&quot;">​</a></h4><p>The spill row&#39;s band is a fixed <code>warning</code> fallback, but <code>computeSpillMagnitude</code> (<code>packages/core/src/detectors.ts</code>) still runs on every spill finding and sets its <code>spillMagnitude</code> field, which the Spill widget displays. Its tiers, in evaluation order (first match wins, <code>null</code> when nothing matches):</p><table tabindex="0"><thead><tr><th>Condition</th><th>Magnitude</th></tr></thead><tbody><tr><td>Single-task stage: <code>spillDiskMax ≥ singleTaskDiskGiB</code> = 1 GiB <strong>or</strong> <code>spillMemMax ≥ singleTaskMemGiB</code> = 4 GiB</td><td><code>severe</code></td></tr><tr><td>Multi-task stage: <code>spillDiskMax ≥ highDiskGiB</code> = 1 GiB, <strong>or</strong> <code>spillDiskMax / taskCount ≥ highTaskDiskMB</code> = 512 MiB (per-task proxy), <strong>or</strong> <code>spillMemMax ≥ highMemGiB</code> = 4 GiB</td><td><code>high</code></td></tr><tr><td>Multi-task stage: <code>spillDiskMax ≥ medDiskMB</code> = 256 MiB <strong>or</strong> <code>spillMemMax ≥ medMemGiB</code> = 1 GiB</td><td><code>medium</code></td></tr><tr><td>Skew (<code>taskCount ≥ skewMinTasks</code> = 10): <code>spillDiskMax / spillDiskP50 &gt; skewRatio</code> = 5× <strong>and</strong> <code>spillDiskMax ≥ skewDiskFloorMB</code> = 128 MiB</td><td><code>high</code></td></tr><tr><td>Skew (<code>taskCount ≥ 10</code>): <code>spillMemMax / spillMemP50 &gt; 5×</code> <strong>and</strong> <code>spillMemMax ≥ skewMemFloorMB</code> = 256 MiB</td><td><code>medium</code></td></tr></tbody></table><p>Disk spill is weighted worse than memory spill by design: the memory thresholds sit well above their disk counterparts at every tier.</p><h3 id="full-threshold-table-never-wallclock-derived" tabindex="-1">Full threshold table (never wallClock-derived) <a class="header-anchor" href="#full-threshold-table-never-wallclock-derived" aria-label="Permalink to &quot;Full threshold table (never wallClock-derived)&quot;">​</a></h3><table tabindex="0"><thead><tr><th>Rule</th><th>Warning</th><th>Critical</th></tr></thead><tbody><tr><td>Stage shape: PRatio</td><td><code>taskCount / totalCores &lt; 0.5</code> (info, under-parallelized), on a stage lasting ≥ <code>lowParallelismFloorPct</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length): parallelizing can&#39;t save more than the stage&#39;s duration, and on the 14 real logs 2839 of 3005 firings were below it</td><td>none</td></tr><tr><td>Stage shape: OIRatio</td><td><code>outputBytes / inputBytes &gt; 10×</code> (info, data explosion)</td><td>none</td></tr><tr><td>Stage shape: TaskStageSkew</td><td><code>taskDurationMax / stageDuration &gt; 3×</code> (info)</td><td>none</td></tr><tr><td>Failed tasks</td><td>failure rate &gt; 5% (min 10 tasks)</td><td>&gt; 20%</td></tr><tr><td>Stage failed outright</td><td>none</td><td>any <code>stageFailureReason</code> present</td></tr><tr><td>Slow host: multi-dimensional</td><td>max/median ratio across taskTime/inputBytes/shuffleBytes/storageMemory ≥ 1.33× (info); each dimension&#39;s sample must also clear an absolute floor (1000 ms for taskTime, 64 MiB for the byte dimensions). A stage lasting under <code>stageFloorPct</code> = 0.5% of the run is skipped (see the mean-duration row): a byte dimension there, which has no time estimate, used to keep its ratio tier (66 of 90 warning/critical on the 14 real logs) until 6298149 graded it <code>info</code></td><td>≥ 3.16× warning, ≥ 10× critical</td></tr><tr><td>Utilization</td><td>avg active executors / peak &lt; 60% (info)</td><td>none</td></tr><tr><td>Autoscaling churn: short-lived executors (design spike, unvalidated thresholds)</td><td>&gt; 30% of executors alive under 2 min (min 5 executors)</td><td>&gt; 60%</td></tr><tr><td>Job failure rate</td><td>≥ 30% (≥ 10% info)</td><td>≥ 50%</td></tr><tr><td>Idle cores</td><td>busy-core-time / (peak cores × wall-clock) idle &gt; 50% (warning)</td><td>none</td></tr><tr><td>Memory band</td><td>peak heap / allocated &gt; 95% too-small (warning); &lt; 70% over-provisioned (info)</td><td>none</td></tr><tr><td>Caching opportunity</td><td>same input relation (or join/union subtree) scanned by ≥ <code>minExecutions</code> = 2 SQL executions in one run</td><td>none (single tier, info)</td></tr><tr><td>Cache utilization: partial caching (this repo)</td><td><code>numCachedPartitions / numPartitions &lt; 0.90</code> (info)</td><td><code>&lt; 0.50</code> (warning)</td></tr><tr><td>Cache utilization: disk spillover (this repo)</td><td><code>diskSize / (memorySize + diskSize) &gt; 0.15</code> (info), <code>MEMORY_AND_DISK*</code> only</td><td><code>&gt; 0.40</code> (warning)</td></tr></tbody></table><p>Spill classification: ≥80% tasks with zero spill → <code>skew</code>; &lt;20% zero → <code>volume</code>; else <code>unclassified</code>. The classification badge is always shown in both compact and expanded spill widget states. This is independent of the magnitude tiers above: classification says <em>what kind</em> of spill, magnitude says <em>how much</em>.</p><h4 id="evidence-fields-stagefailed-retrywaste" tabindex="-1">Evidence fields (<code>stageFailed</code> / <code>retryWaste</code>) <a class="header-anchor" href="#evidence-fields-stagefailed-retrywaste" aria-label="Permalink to &quot;Evidence fields (`stageFailed` / `retryWaste`)&quot;">​</a></h4><p>Neither entry&#39;s <code>detect()</code> used to put anything beyond a scalar <code>metric</code>/ <code>value</code> on its finding. Both now also attach:</p><ul><li><code>numTasks</code>: <code>stage.taskCount</code> at detection time.</li><li><code>memoryBytesSpilled</code>: <code>stage.memoryBytesSpilled</code> at detection time.</li><li><code>stageFailed</code> only: <code>failedTaskDetails</code>: up to 20 <code>FailedTaskSample</code> records (<code>taskId</code>, <code>attemptNumber</code>, <code>host</code>, <code>executorId</code>, <code>reason</code>, <code>peakExecMem</code>, <code>memSpilled</code>, <code>shuffleWrite</code>) for tasks still marked failed when the stage was finalized (<code>finalizeStage</code>, <code>stage-quantiles.ts</code>).</li><li><code>retryWaste</code> only: <code>retriedTaskDetails</code>: up to 20 <code>FailedTaskSample</code> records for attempts discarded by the retry-dedup logic in <code>accumulateTask</code> (<code>event-handlers.ts</code>): captured at the moment they&#39;d otherwise be thrown away, since by finalize time only the winning attempt survives.</li></ul><p>Both sample arrays are capped at 20 entries, filled in first-encountered order (finalize order for <code>failedTaskDetails</code>, discard order for <code>retriedTaskDetails</code>), not spread across distinct hosts/executors: a stage with failures clustered on one bad host could fill the cap before a more informative failure elsewhere in the stage is ever sampled. This is the first evidence-shape documentation in this file; no other finding type has one yet.</p>',39)])])}const p=t(s,[["render",c]]);export{g as __pageData,p as default};
+import{_ as t,o,c as d,a5 as a}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Detector contract","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/detector-contract.md","filePath":"contributor-guide/architecture/detector-contract.md"}'),s={name:"contributor-guide/architecture/detector-contract.md"};function c(r,e,i,n,l,h){return o(),d("div",null,[...e[0]||(e[0]=[a('<h1 id="detector-contract" tabindex="-1">Detector contract <a class="header-anchor" href="#detector-contract" aria-label="Permalink to &quot;Detector contract&quot;">​</a></h1><p><code>packages/core/src/detectors.ts</code> is the single source of Spark-optimization logic: one declarative <code>DETECTORS</code> entry per pattern, each carrying <code>type</code>, <code>scope</code> (<code>stage</code> / <code>app</code> / <code>config</code> / <code>sql</code>), <code>order</code>, <code>fixEffort</code>, a <code>thresholds</code> object, impactBand/copy, a <code>docAnchor</code>, and a co-located <code>detect()</code> method. Both consumers are thin loops over that array:</p><ul><li><p><code>packages/core/src/analyzer.ts</code>: <code>analyze()</code> runs every entry regardless of scope, skipping only <code>inScorecard:false</code> ones; <code>auditConfig()</code> separately runs the <code>scope:&#39;config&#39;</code> entries. The four <code>configAudit</code> entries stay out of the bottleneck catalog because each sets <code>inScorecard:false</code>, not because of <code>scope:&#39;config&#39;</code>: a future config-scope detector without that flag would run through <code>analyze()</code> too. Each finding is stamped with its entry&#39;s <code>docAnchor</code>.</p></li><li><p><code>src/view/detector-registry.tsx</code>: a <code>REGISTRY: Record&lt;findingType, {component, region}&gt;</code> replaces <code>dashboard-renderer.js</code>&#39;s <code>render:</code> bindings, one entry per emitted finding type. <code>orderedWidgets()</code> walks <code>DETECTORS</code> ascending by <code>order</code>, then sorts <code>action</code>-region components before <code>reference</code>-region ones. Every <code>finding.type</code> maps to its own component now (the 2026-09 widget/finding-type 1:1 mapping redesign split six components that used to multiplex several types each: <code>TaskSkew</code> into <code>Skew</code>/<code>StageShape</code>/<code>TinyTask</code>; <code>ShuffleIO</code> narrowed to <code>shuffle</code> only, plus a new <code>PartitionSizing</code>; <code>Failures</code> into <code>StageFailed</code>/ <code>TaskFailures</code>/<code>RetryWaste</code>; <code>ExecutorTimeline</code> into <code>SlowHost</code>/ <code>StageSlowness</code>/<code>Straggler</code>/<code>SpeculationWaste</code>/<code>ColdStart</code> (its non-finding-driven executor-count chart moved to <code>ExecutorCountChart</code>, a <code>ReferenceSection</code> tile, not a <code>REGISTRY</code> entry); <code>MemoryUtilization</code> narrowed to <code>memoryUtilization</code> only, plus a new <code>ExecutorUtilization</code> for <code>utilization</code>; <code>PlanFindings</code> into <code>DuplicatePlanSubtree</code>/<code>SmallFiles</code>/ <code>UnderBroadcast</code>/<code>OverBroadcast</code>, dropping the dead <code>broadcastSizing</code> key entirely). No two <code>REGISTRY</code> entries share a <code>component</code> value any more.</p><p>Each widget component receives the full catalog and self-gates when it has nothing to show, rendering <code>null</code> or a muted &quot;no issue&quot; card for the always-visible ones. <code>orderedWidgets()</code> itself has no empty/non-empty branching, since it iterates the static <code>DETECTORS</code> import, not the runtime <code>catalog</code>.</p></li></ul><p>Thresholds live only in each entry&#39;s <code>thresholds</code>; see <a href="#bottleneck-thresholds-spec-§4">Bottleneck thresholds</a>.</p><h2 id="confidence-disclosure" tabindex="-1">Confidence disclosure <a class="header-anchor" href="#confidence-disclosure" aria-label="Permalink to &quot;Confidence disclosure&quot;">​</a></h2><p>A <code>Detector</code> entry (or the <code>Finding</code> it returns) may carry <code>confidence: &#39;low&#39; | &#39;medium&#39; | &#39;high&#39;</code> plus a <code>validationRequired</code> string. <code>RowStatusCluster</code> (<code>src/view/RowStatusCluster.tsx</code>) is the one place that renders it, gated to Advanced density: a plain &quot;&lt;confidence&gt; confidence&quot; badge whose tooltip carries the full <code>validationRequired</code> text. A finding with no <code>confidence</code> field renders identically to a fully-validated one, so every detector whose thresholds are our own unvalidated noise floor (marked <code>NOT SOURCED</code> in a code comment) should set both fields, not just the ones that happen to already have <code>RowStatusCluster</code> wired into their widget. <code>skew</code>, <code>straggler</code>, and <code>gc</code> set <code>confidence</code> for exactly this reason: their runtime-floor thresholds carry the same kind of unvalidated-noise-floor caveat <code>coreLocality</code>, <code>autoscalingChurn</code>, and <code>memoryUtilization</code>&#39;s <code>wasteModel</code> variant already disclose. None of these hardcode a single confidence value: each scales <code>&#39;low&#39; | &#39;medium&#39; | &#39;high&#39;</code> off how far the finding sits past its own detector&#39;s threshold, via a small named helper placed just above the <code>DETECTORS</code> array (e.g. <code>skewConfidence</code>, <code>coreLocalityConfidence</code>, <code>cachingReuseConfidence</code>) rather than an inline literal.</p><h2 id="the-fixeffort-field" tabindex="-1">The <code>fixEffort</code> field <a class="header-anchor" href="#the-fixeffort-field" aria-label="Permalink to &quot;The `fixEffort` field&quot;">​</a></h2><p>Each <code>Detector</code> entry also carries <code>fixEffort: &#39;config&#39; | &#39;code&#39; | &#39;rearchitect&#39;</code>, alongside <code>order</code> and <code>thresholds</code>: a rough estimate of how much work resolving the finding takes.</p><p>No view currently reads it. The quadrant impact/effort bucketing this field was meant to feed (<code>bucketFinding</code>/<code>effortTier</code>/<code>computeImpactMagnitude</code> in a since-deleted <code>src/quadrant-bucket.ts</code>, gated behind a <code>FIX_EFFORT_MAPPING_REVIEWED</code> flag that never flipped to <code>true</code>) was removed as dead code in the recommendations-consolidation redesign: <code>FixTheseFirst</code> (<code>src/view/widgets/FixTheseFirst.tsx</code>) ranks purely by impact magnitude. See <a href="./widget-rendering.html#widget-rendering-order-fixed-spec-§5">Widget rendering order</a> for how it ranks findings today.</p><p>Two shared helpers back multiple detectors and reports. <code>packages/core/src/plan-tree-walk.ts</code>&#39;s <code>walkPlanTree(root, visit, {dedupe})</code> is the iterative pre-order plan-tree traversal used by <code>detectors.ts</code> and every <code>plan-*.ts</code> module (<code>plan-summary.ts</code>, <code>plan-duration-attribution.ts</code>, <code>plan-node-detail.ts</code>, <code>plan-dot.ts</code>). <code>packages/core/src/core-count.ts</code>&#39;s <code>computeTotalCores(app, executorsAdded)</code> is the shared core-count logic used by <code>efficiency-model.ts</code>, <code>scaling-sim.ts</code>, and <code>wasted-core-hours.ts</code>. <code>detectors.ts</code>&#39;s own <code>utilization</code> and <code>memoryUtilization</code> entries use the same file&#39;s <code>computePeakConcurrentCores</code>/<code>computePeakConcurrentExecutorCount</code> instead: <code>computeTotalCores</code> sums every <code>ExecutorAdded</code> event with no regard for overlap, so under executor churn (spot preemption, <code>dynamicAllocation</code> replacement) it double-counts a churned executor&#39;s capacity against its replacement&#39;s; the peak-concurrent sweeps don&#39;t.</p><h2 id="cross-detector-suppression" tabindex="-1">Cross-detector suppression <a class="header-anchor" href="#cross-detector-suppression" aria-label="Permalink to &quot;Cross-detector suppression&quot;">​</a></h2><p>An entry may declare an optional <code>suppressWhen(finding, out)</code> method. <code>analyzer.ts</code>&#39;s <code>push()</code>, the single choke point every finding passes through, calls it per-finding, after the null guard and before the push, and drops the finding silently when it returns <code>true</code>. <code>out</code> is the findings accumulated so far. Since <code>analyze()</code>&#39;s loop is detector-outer / stage-inner, every finding from a detector declared earlier in <code>DETECTORS</code> is already in <code>out</code> by the time a later detector runs, for every stage. That makes the pattern purely declaration-order-driven: the suppressing detector must be declared earlier in the <code>DETECTORS</code> array than the suppressed one.</p><p><code>stageSlowness</code> uses this to defer to <code>slowHost</code>. It is spliced immediately after the <code>slowHost</code> entry regardless of its <code>order</code> field (<code>order</code> only controls render sequencing, not evaluation order), and <code>tests/analyzer.test.js</code>&#39;s &quot;detector contract&quot; suite asserts the array-index ordering so a future reorder can&#39;t silently break the suppression. The mechanism is deliberately minimal: a same-array, predicate-in-<code>push()</code> filter, not a general dependency graph. <code>auditConfig()</code>&#39;s own <code>push()</code> call is unaffected, since <code>scope:&#39;config&#39;</code> entries declare no <code>suppressWhen</code>.</p><h2 id="per-operator-duration-attribution" tabindex="-1">Per-operator duration attribution <a class="header-anchor" href="#per-operator-duration-attribution" aria-label="Permalink to &quot;Per-operator duration attribution&quot;">​</a></h2><p><code>packages/core/src/plan-duration-attribution.ts</code> (entry <code>attributeStageDurationToPlan(planTree, stagesById, sqlExec)</code>) approximates how a SQL execution&#39;s stage wall-time splits across plan operators, returning a <code>Map&lt;planNode, milliseconds&gt;</code>. It cuts the plan tree at Exchange boundaries into connected components: since the Exchange write/read split (<code>resolvePlanTree</code> in <code>event-handlers.ts</code> always synthesizes a <code>read</code> node wrapping a <code>write</code> node for every raw <code>Exchange</code>/<code>BroadcastExchange</code>), the cut is keyed off <code>PlanNode.exchangeRole === &#39;read&#39;</code> on the parent, not a name regex: the write half starts the new component, the read half stays in its parent&#39;s. A node with no <code>exchangeRole</code> at all (for example <code>ReusedExchange</code>, which is never split) never starts a new component on its own, unlike the old name-based regex, which matched any Exchange-family name regardless of split state. Each component receives a stable pre-order identity and separate parent/depth/traversal metadata; the identity itself does not encode its count of Exchange ancestors. It zips components deepest-first by that explicit depth against submission-ordered stage IDs, then apportions each matched stage&#39;s wall-time across that component&#39;s nodes by timing-metric weight, falling back to an even split when no node carries a timing metric.</p><p>This is best-effort inference, not measurement. Spark&#39;s event model exposes no ground truth for per-operator time within a stage; the Exchange-boundary segmentation and deepest-component-to-earliest-stage zip are heuristics. Treat the per-operator numbers as directional hints, never as authoritative timings, and do not build hard thresholds or findings on top of them.</p><h2 id="stage-id-attribution-for-plan-advisor-findings" tabindex="-1">Stage-ID attribution for Plan Advisor findings <a class="header-anchor" href="#stage-id-attribution-for-plan-advisor-findings" aria-label="Permalink to &quot;Stage-ID attribution for Plan Advisor findings&quot;">​</a></h2><p>The Plan Advisor detectors (<code>duplicatePlanSubtree</code>, <code>smallFiles</code>, <code>broadcastSizing</code> in <code>packages/core/src/detectors.ts</code>) each attribute their finding to a narrowed <code>stageIds</code> set rather than the whole SQL execution: <code>PlanNode.stageIds</code> is resolved once per plan tree at parse time by unioning, per node, every metric&#39;s accumulator ID against a <code>taskAccumStages: Map&lt;accumulatorId, Set&lt;stageId&gt;&gt;</code> built while parsing <code>TaskEnd</code> events, then clipping the result to the execution&#39;s own stage set. An accumulator ID occasionally points to a <em>different</em> execution&#39;s stages, e.g. a <code>ReusedSubquery</code> computed once and reused verbatim, and the clip prevents misattributing that other execution&#39;s work. Each detector unions its implicated node(s)&#39; <code>stageIds</code> and falls back to the execution-wide set only when no implicated node has any coverage; a finding never partially blends a narrowed set with the execution-wide one. When an execution has no stage universe at all (no jobs ever recorded against it, which is true for 42% of real-log SQL executions with a plan tree, typically job-less/driver-only executions), the clip drops every candidate stage ID instead of passing them through: every node in that execution&#39;s tree ends up with no <code>stageIds</code> anywhere, same &quot;coverage is partial&quot; framing as below. (An earlier version of this clip treated &quot;no stage universe&quot; as &quot;no clip,&quot; which let a foreign accumulator ID collision, e.g. the <code>ReusedSubquery</code> case above, leak another execution&#39;s stages into a job-less execution&#39;s nodes; the clip is now unconditional on <code>executionStageIds</code> being present.)</p><p>Coverage is partial by Spark&#39;s own design: whole-stage-codegen wrapper nodes (<code>InputAdapter</code>, and other purely structural passthrough markers) carry no accumulators at all, and <code>BroadcastExchangeExec</code>&#39;s own metrics are computed entirely on the driver and never appear on any <code>TaskEnd</code> (real Spark behavior). Since the Exchange write/read split, those driver-computed metrics live specifically on the synthesized <em>write</em> half (<code>exchangeRole: &#39;write&#39;</code>); the <em>read</em> half always carries <code>metrics: []</code>. The write half&#39;s immediate child, which does carry executor-side metrics, is unioned in instead, see <code>overBroadcast</code>&#39;s wiring. A <code>TaskEnd</code> arriving after its stage has already been finalized is also silently excluded from <code>taskAccumStages</code>, consistent with the parser&#39;s existing out-of-order tolerance elsewhere.</p><p><code>planTree</code> itself is kept current against Spark&#39;s adaptive query execution (AQE) re-plans: <code>SparkListenerSQLAdaptiveExecutionUpdate</code> events overwrite the execution&#39;s <code>sparkPlanInfo</code> last-write-wins, so accumulator-ID evidence is matched against the plan that actually ran rather than a stale pre-AQE snapshot. Since a superseded plan is never read, <code>dispatchLine</code> holds an open execution&#39;s latest update as unparsed text and parses only that one, when <code>SQLExecutionEnd</code> arrives or at parse completion (<code>deferAdaptiveUpdate</code>, <code>event-handlers.ts</code>). It recognizes an update from the flat first and last pieces <code>buildChunkDecoder</code> reports for a line joined across slices (<code>JoinedLine</code>), so a superseded update is never copied into one flat string either. The raw <code>sparkPlanInfo</code> stays worker-side and is released once <code>SQLExecutionEnd</code> resolves it into <code>planTree</code>; <code>physicalPlanDescription</code> (Spark&#39;s text rendering of the plan, which nothing reads) is emptied before <code>JSON.parse</code> and never retained (<code>stripPlanDescription</code>, <code>event-handlers.ts</code>). <code>buildChunkDecoder</code> decodes each decompressed chunk in slices of at most 512 KiB, and drops the bytes of a value that crosses a slice boundary without decoding them.</p><p>No eviction/pruning is added to <code>taskAccumStages</code>, a deliberate choice, not an oversight: measured on real logs, it holds roughly 1,050 keys per compressed MB (9,850 keys on an 11.6 MB fixture, about 29,000 keys on a 28.1 MB fixture). Extrapolated to a 240MB+ log, the scale this tool targets (see <code>CLAUDE.md</code>), that is roughly 250,000 keys, around 45 MB of heap for an equivalent synthetic <code>Map&lt;number, Set&lt;number&gt;&gt;</code>. This heap estimate is still small relative to this tool&#39;s other in-memory state. It is higher, though, than the fixture-only measurements taken when this mechanism was built suggested.</p><p>Per-execution pruning (e.g. dropping a <code>taskAccumStages</code> entry once its stage finalizes or its owning SQL execution resolves, mirroring how <code>accumState</code> is cleared in <code>endSqlExecution</code>) is deliberately not done either: unlike <code>accumState</code>, <code>taskAccumStages</code> is one global, un-scoped map read by every execution&#39;s <code>resolvePlanTree</code> call, and the <code>ReusedSubquery</code> case above depends on a stage recorded under one execution still being visible when a later execution resolves. Pruning on any single execution&#39;s lifecycle would break that cross-execution lookup. What is bounded is the growth from a single pathological event: <code>TaskEndEventSchema</code>&#39;s <code>Accumulables</code> array is capped at <code>MAX_ACCUMULABLES_PER_TASK</code> (10,000, <code>event-schemas.ts</code>), well above any real plan&#39;s per-task metric count, so a single crafted <code>TaskEnd</code> can&#39;t grow the map past that per-event bound; a <code>TaskEnd</code> exceeding it fails schema validation and the line is skipped (counted in <code>skippedLines</code>) like any other malformed event. Each accumulable is checked for its numeric <code>ID</code> only: <code>Update</code>/<code>Value</code> are unread, and Spark writes some as JSON arrays (<code>internal.metrics.updatedBlockStatuses</code>), which a stricter check would reject along with the whole task. When every entry has Spark&#39;s flat <code>{&quot;ID&quot;:n,...}</code> form, <code>parseTaskEnd</code> (<code>event-handlers.ts</code>) reads the IDs with a string scan and cuts the array out before <code>JSON.parse</code>; any other shape is parsed whole.</p><h2 id="bottleneck-thresholds-spec-§4" tabindex="-1">Bottleneck thresholds (spec §4) <a class="header-anchor" href="#bottleneck-thresholds-spec-§4" aria-label="Permalink to &quot;Bottleneck thresholds (spec §4)&quot;">​</a></h2><p>Every change to <code>packages/core/src/detectors.ts</code> should reference this table.</p><p>Every finding&#39;s <code>impactBand</code> comes from one of two places. For any finding whose <code>impactEstimate</code> carries a <code>wallClock</code> estimate (the common case for most rules below), <code>analyzer.ts</code> calls <code>deriveImpactBand()</code> (<code>packages/core/src/impact-band.ts</code>) immediately after <code>estimateImpact()</code>, which sets <code>.impactBand</code> purely from <code>wallClock.high</code> as a fraction of the app&#39;s total duration (<code>&gt;= 2%</code> critical, <code>&gt;= 0.5%</code> warning, else info: the same <code>floorPctWarn</code>/<code>floorPctCrit</code> values <code>skew</code>/<code>straggler</code> use for their own thresholds below). For those rules, the table below documents their firing gate plus their fixed fallback constant, which surfaces only when this run&#39;s finding of that type didn&#39;t get a wallClock estimate (a stage excluded from the occupancy sweep). For rules whose finding type never gets a wallClock estimate (<code>resourceOnly</code>/<code>informational</code> basis, e.g. <code>configAudit</code>, or a rule that keeps its own ratio-tiered classification per the design&#39;s Decision 2, e.g. <code>failures</code>), the full threshold table below is the real, displayed classification: <code>detectors.ts</code> sets <code>impactBand</code> directly and nothing overwrites it. <code>partitionSizing</code>&#39;s <code>maxPartitionTooBig</code> rule is a third case: it does carry a <code>wallClock</code> estimate but is explicitly exempted in <code>deriveImpactBand()</code> because it&#39;s a hardcoded-critical OOM/crash-risk safety signal, not a time-recovery one, so <code>detectors.ts</code>&#39;s own classification stands regardless of how small that estimate is relative to the run.</p><h3 id="fixed-fallback-only-usually-wallclock-derived-instead" tabindex="-1">Fixed fallback only (usually wallClock-derived instead) <a class="header-anchor" href="#fixed-fallback-only-usually-wallclock-derived-instead" aria-label="Permalink to &quot;Fixed fallback only (usually wallClock-derived instead)&quot;">​</a></h3><p>These rules&#39; <em>band</em> tiers were deleted from <code>packages/core/src/detectors.ts</code> (they were always overwritten by <code>deriveImpactBand</code> whenever a wallClock estimate was available); the constant in the last column is only a floor-case fallback. Their <em>firing</em> gate is untouched and still lives in each entry&#39;s <code>thresholds</code> object: it decides whether the rule reports anything, so it stays documented here in full.</p><table tabindex="0"><thead><tr><th>Rule</th><th>Fires when</th><th>Fallback</th></tr></thead><tbody><tr><td>Task skew</td><td><code>taskDurationP95 / taskDurationP50 &gt; 3×</code> (<code>taskDurationMax / P50</code> for stages under <code>minTasksForP95</code> = 20 tasks), <strong>and</strong> the occupancy-clipped tail recovery (<code>tailRecoveryMs</code>: <code>finalizeStage</code>&#39;s task-level replay of the stage with every task over 4× P50 capped at P50) is ≥ <code>floorPctWarn</code> = 0.5% of app runtime. The clip floors the claim at the longest task the fix leaves, not the current one, and at the core work the fix leaves over every core (see impact-estimation.md&#39;s occupancy section); <code>straggler</code>&#39;s floors use the same clip</td><td><code>warning</code></td></tr><tr><td>Shuffle read</td><td><code>shuffleReadBytes &gt; minBytes</code> = 50 MiB, on a stage that isn&#39;t shorter than <code>stageFloorPct</code> = 0.5% of the run (a zero-length stage or an unknown run duration passes; the claim is clipped to the stage, so a shorter one graded <code>info</code>: 182 of 284 on the 14 real logs, the shuffle itself still there)</td><td><code>info</code></td></tr><tr><td>Partition sizing: skew</td><td><code>shuffleReadMax &gt; 5×</code> <code>shuffleReadP50</code> <strong>and</strong> <code>shuffleReadMax &gt; 256 MiB</code></td><td><code>warning</code></td></tr><tr><td>Partition sizing: low parallelism</td><td><code>shuffleReadBytes ≥ 1 GiB</code> <strong>and</strong> <code>taskCount ≤ 7</code></td><td><code>warning</code></td></tr><tr><td>Partition sizing: oversized partition</td><td><code>shuffleReadMax ≥ 5 GiB</code></td><td><code>critical</code></td></tr><tr><td>GC</td><td><code>executorRunTime ≥ minRunTimeMs</code> = 10 s <strong>and</strong> <code>gcPct &gt; 10%</code></td><td><code>warning</code></td></tr><tr><td>GC (low / cost)</td><td><code>executorRunTime ≥ 10 s</code> <strong>and</strong> <code>gcPct &lt; lowInfoPct100</code> = 5% (checked only when the GC row above did not fire) <strong>and</strong> the stage lasts ≥ <code>lowInfoFloorPct</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length, such as one with no completion time). Gets no wall-clock estimate (an over-provisioning signal), so this band always stands. The floor dropped 464 of 685 low-GC notes on the 14 real logs (none on the corpus): low GC is still true on those stages, but a memory-sizing note from a stage that barely ran adds nothing</td><td><code>info</code></td></tr><tr><td>Spill (magnitude v2)</td><td>any non-zero <code>memoryBytesSpilled</code>, on a stage that isn&#39;t shorter than <code>stageFloorPct</code> = 0.5% of the run (as for shuffle read: 12 of 38 on the 14 real logs, all <code>info</code>). The magnitude sub-table below classifies <em>how much</em>, but does not gate firing</td><td><code>warning</code></td></tr><tr><td>Cold start</td><td><code>firstExecutorAddedAt − firstStageSubmittedAt &gt; gapSeconds</code> = 30 s (no finding without executor-added events): the time a runnable stage waited for its first executor. An executor added before the first stage and still alive at submission means no wait and no finding; one removed at or before submission is ignored, so the gap runs to the next executor added after it. It used to be <code>firstStageSubmittedAt − app.startTime</code>, the driver&#39;s own startup, which read 36-47 s on all 9 real logs that fired whatever the executors did, and flagged 6 corpus logs whose first executor was up 44-313 s before the first stage (5 of them) or arrived 4.4 s after it</td><td><code>warning</code></td></tr><tr><td>Slow host: mean-duration ratio</td><td>stage has ≥ <code>minHosts</code> = 3 hosts (or executors) and ≥ <code>minTasks</code> = 15 tasks, and lasts ≥ <code>stageFloorPct</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length, which gets no estimate and keeps its fallback band; every slow-host finding on a shorter stage graded <code>info</code>, 324 of 452 on the 14 real logs, the imbalance itself still true; this gate covers every slowHost row); then per host: mean task duration / overall median ≥ <code>ratioWarn</code> = 2.0× <strong>and</strong> host task-share ≥ <code>minShare</code> = 20% <strong>and</strong> host mean ≥ <code>floorMs</code> = 1000 ms (absolute-magnitude floor, rules out sub-second noise)</td><td><code>warning</code></td></tr><tr><td>Slow host: duration-share</td><td>same stage gate as the row above; then per host: ≥ <code>shareWarn</code> = 75% of the stage&#39;s total task-duration <strong>and</strong> ≥ <code>taskShareWarn</code> = 50% of its task count</td><td><code>warning</code></td></tr><tr><td>Stage slowness: absolute fallback, suppressed when <code>slowHost</code> already fired</td><td>stage wall-clock duration ≥ <code>infoMin</code> = 15 min. The band then comes from the partitioning-headroom estimate (see impact-estimation.md), not the duration</td><td><code>info</code></td></tr><tr><td>Straggler / speculative-execution</td><td><code>taskCount ≥ minTasks</code> = 10, <strong>and</strong> the stage lasts ≥ <code>floorPctWarn</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length, such as one with no completion time; a shorter stage&#39;s tail can&#39;t cost more than its own duration, so every finding there graded <code>info</code>: 671 of 753 on the 14 real logs, 3 on the corpus, the slow tail itself still true), <strong>and</strong> either any speculative task ran, <strong>or</strong> straggler share &gt; <code>shareWarn</code> = 5%, <strong>or</strong> straggler share &gt; <code>shareWarnAtFloor</code> = 2.5% with the occupancy-clipped tail recovery (<code>tailRecoveryMs</code>, as for skew) already ≥ <code>floorPctWarn</code> (0.5% of app runtime). The lower gate is scored against a task-level replay of every stage on 14 real logs (recoverable = list-scheduling replay with each task over 4× P50 capped at P50; positive = ≥ 0.5% of app runtime): it found 3 stages whose stragglers gated them for 10-48s at 2.7-4% of their tasks, for 1 borderline miss, lifting skew-or-straggler recall from 0.86 to 0.92 at precision 0.91 → 0.89. Admitting every 2.5% share instead would add 86 findings below the floor. The same sweep kept skew&#39;s <code>ratioWarn</code> = 3 (2.5 added false positives, 4 lost true ones) and the 0.5% floor (1% halved skew recall). <code>warnPct</code>/<code>critPct</code> (10%/20% speculative share) and <code>floorPctWarn</code>/<code>floorPctCrit</code> (0.5%/2% of app runtime) no longer set the band; they rank the straggler-vs-speculative tiers that pick which <em>metric</em> the finding reports</td><td><code>info</code></td></tr><tr><td>Speculation waste (new)</td><td><code>speculationWastedAttempts ≥ minWasted</code> = 5 <strong>and</strong> <code>speculationWasteMs ≥ minWasteMs</code> = 60 s</td><td><code>warning</code></td></tr><tr><td>Retry waste</td><td><code>wastedAttempts ≥ minWasted</code> = 3 <strong>and</strong> <code>retryWasteMs ≥ minWasteMs</code> = 30 s, on a stage that still completed</td><td><code>warning</code></td></tr><tr><td>Tiny tasks</td><td><code>taskCount ≥ minTasks</code> = 100 <strong>and</strong> <code>taskDurationP50 ≤ maxP50</code> = 500 ms <strong>and</strong> <code>taskDurationP95 ≤ maxP95</code> = 1000 ms <strong>and</strong> the stage lasts ≥ <code>stageFloorPct</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length; coalescing can&#39;t save more than the stage&#39;s own duration, so every finding on a shorter stage graded <code>info</code>: 132 of 151 on the 14 real logs, the tasks still tiny). Checked (2026-09-23) against each stage&#39;s measured per-task overhead (task wall time minus executor run time, the estimate&#39;s own input) on 14 real logs and the corpus: of 228 stages with 100+ tasks outside the P50/P95 gate, none would save 0.5% of its run by coalescing (median overhead 0.2-3% of task time); of 170 inside it, 25 would, and all 25 grade above <code>info</code></td><td><code>info</code></td></tr><tr><td>Duplicate plan subtree</td><td>a subtree of ≥ <code>minSubtreeSize</code> = 3 nodes whose shape fingerprint repeats ≥ <code>minOccurrences</code> = 2× in the plan, unless its linked stages together lasted less than <code>stageFloorPct</code> = 0.5% of the run (a repeat with no linked stage time, or an unknown run duration, is kept; the claim counts at most each stage&#39;s own task-active time, so such a repeat graded <code>info</code>: 340 of 545 on the 14 real logs, the repeat still in the plan). <code>occurrencesIdentical</code> records whether the repeats also agree node-for-node on normalized detail, ignoring AQE query-stage numbers; when they don&#39;t (same shape over another table, filter or projection: 270 of 546 groups on the 14 real logs) the finding is <code>info</code> with confidence <code>low</code> and no time claim. <code>stageShares</code> gives, per stage, the repeated operators&#39; share of the stage&#39;s operators (WholeStageCodegen wrappers and Exchange write halves not counted); with no attributed stage the finding is <code>info</code></td><td><code>warning</code></td></tr><tr><td>Small files read/write</td><td>per read/write side: file count &gt; <code>minFiles</code> = 100 <strong>and</strong> average file size &lt; <code>maxAvgFileSizeMB</code> = 3 MiB</td><td><code>warning</code></td></tr><tr><td>Broadcast sizing: missed</td><td>a 2-child <code>SortMergeJoin</code> whose smaller side is &lt; 10 MiB (unconditional), or &lt; 100 MiB with the larger side &gt; 10 GiB, or &lt; 1 GiB with larger &gt; 300 GiB, or &lt; 5 GiB with larger &gt; 1 TiB (<code>broadcastTiers</code> × <code>comparisonTiers</code>)</td><td><code>info</code></td></tr><tr><td>Broadcast sizing: oversized</td><td>a <code>BroadcastExchange</code> node whose <code>data size</code> metric &gt; <code>overBroadcastBytes</code> = 1 GiB</td><td><code>warning</code></td></tr></tbody></table><h4 id="spill-magnitude-tiers" tabindex="-1">Spill magnitude tiers <a class="header-anchor" href="#spill-magnitude-tiers" aria-label="Permalink to &quot;Spill magnitude tiers&quot;">​</a></h4><p>The spill row&#39;s band is a fixed <code>warning</code> fallback, but <code>computeSpillMagnitude</code> (<code>packages/core/src/detectors.ts</code>) still runs on every spill finding and sets its <code>spillMagnitude</code> field, which the Spill widget displays. Its tiers, in evaluation order (first match wins, <code>null</code> when nothing matches):</p><table tabindex="0"><thead><tr><th>Condition</th><th>Magnitude</th></tr></thead><tbody><tr><td>Single-task stage: <code>spillDiskMax ≥ singleTaskDiskGiB</code> = 1 GiB <strong>or</strong> <code>spillMemMax ≥ singleTaskMemGiB</code> = 4 GiB</td><td><code>severe</code></td></tr><tr><td>Multi-task stage: <code>spillDiskMax ≥ highDiskGiB</code> = 1 GiB, <strong>or</strong> <code>spillDiskMax / taskCount ≥ highTaskDiskMB</code> = 512 MiB (per-task proxy), <strong>or</strong> <code>spillMemMax ≥ highMemGiB</code> = 4 GiB</td><td><code>high</code></td></tr><tr><td>Multi-task stage: <code>spillDiskMax ≥ medDiskMB</code> = 256 MiB <strong>or</strong> <code>spillMemMax ≥ medMemGiB</code> = 1 GiB</td><td><code>medium</code></td></tr><tr><td>Skew (<code>taskCount ≥ skewMinTasks</code> = 10): <code>spillDiskMax / spillDiskP50 &gt; skewRatio</code> = 5× <strong>and</strong> <code>spillDiskMax ≥ skewDiskFloorMB</code> = 128 MiB</td><td><code>high</code></td></tr><tr><td>Skew (<code>taskCount ≥ 10</code>): <code>spillMemMax / spillMemP50 &gt; 5×</code> <strong>and</strong> <code>spillMemMax ≥ skewMemFloorMB</code> = 256 MiB</td><td><code>medium</code></td></tr></tbody></table><p>Disk spill is weighted worse than memory spill by design: the memory thresholds sit well above their disk counterparts at every tier.</p><h3 id="full-threshold-table-never-wallclock-derived" tabindex="-1">Full threshold table (never wallClock-derived) <a class="header-anchor" href="#full-threshold-table-never-wallclock-derived" aria-label="Permalink to &quot;Full threshold table (never wallClock-derived)&quot;">​</a></h3><table tabindex="0"><thead><tr><th>Rule</th><th>Warning</th><th>Critical</th></tr></thead><tbody><tr><td>Stage shape: PRatio</td><td><code>taskCount / totalCores &lt; 0.5</code> (info, under-parallelized), on a stage lasting ≥ <code>lowParallelismFloorPct</code> = 0.5% of the run (passes when the run&#39;s duration is unknown or the stage has zero length): parallelizing can&#39;t save more than the stage&#39;s duration, and on the 14 real logs 2839 of 3005 firings were below it</td><td>none</td></tr><tr><td>Stage shape: OIRatio</td><td><code>outputBytes / inputBytes &gt; 10×</code> (info, data explosion)</td><td>none</td></tr><tr><td>Stage shape: TaskStageSkew</td><td><code>taskDurationMax / stageDuration &gt; 3×</code> (info)</td><td>none</td></tr><tr><td>Failed tasks</td><td>failure rate &gt; 5% (min 10 tasks)</td><td>&gt; 20%</td></tr><tr><td>Stage failed outright</td><td>none</td><td>any <code>stageFailureReason</code> present</td></tr><tr><td>Slow host: multi-dimensional</td><td>max/median ratio across taskTime/inputBytes/shuffleBytes/storageMemory ≥ 1.33× (info); each dimension&#39;s sample must also clear an absolute floor (1000 ms for taskTime, 64 MiB for the byte dimensions). A stage lasting under <code>stageFloorPct</code> = 0.5% of the run is skipped (see the mean-duration row): a byte dimension there, which has no time estimate, used to keep its ratio tier (66 of 90 warning/critical on the 14 real logs) until 6298149 graded it <code>info</code></td><td>≥ 3.16× warning, ≥ 10× critical</td></tr><tr><td>Utilization</td><td>avg active executors / peak &lt; 60% (info)</td><td>none</td></tr><tr><td>Autoscaling churn: short-lived executors (design spike, unvalidated thresholds)</td><td>&gt; 30% of executors alive under 2 min (min 5 executors)</td><td>&gt; 60%</td></tr><tr><td>Job failure rate</td><td>≥ 30% (≥ 10% info)</td><td>≥ 50%</td></tr><tr><td>Idle cores</td><td>busy-core-time / (peak cores × wall-clock) idle &gt; 50% (warning)</td><td>none</td></tr><tr><td>Memory band</td><td>peak heap / allocated &gt; 95% too-small (warning); &lt; 70% over-provisioned (info)</td><td>none</td></tr><tr><td>Caching opportunity</td><td>same input relation (or join/union subtree) scanned by ≥ <code>minExecutions</code> = 2 SQL executions in one run</td><td>none (single tier, info)</td></tr><tr><td>Cache utilization: partial caching (this repo)</td><td><code>numCachedPartitions / numPartitions &lt; 0.90</code> (info)</td><td><code>&lt; 0.50</code> (warning)</td></tr><tr><td>Cache utilization: disk spillover (this repo)</td><td><code>diskSize / (memorySize + diskSize) &gt; 0.15</code> (info), <code>MEMORY_AND_DISK*</code> only</td><td><code>&gt; 0.40</code> (warning)</td></tr></tbody></table><p>Spill classification: ≥80% tasks with zero spill → <code>skew</code>; &lt;20% zero → <code>volume</code>; else <code>unclassified</code>. The classification badge is always shown in both compact and expanded spill widget states. This is independent of the magnitude tiers above: classification says <em>what kind</em> of spill, magnitude says <em>how much</em>.</p><h4 id="evidence-fields-stagefailed-retrywaste" tabindex="-1">Evidence fields (<code>stageFailed</code> / <code>retryWaste</code>) <a class="header-anchor" href="#evidence-fields-stagefailed-retrywaste" aria-label="Permalink to &quot;Evidence fields (`stageFailed` / `retryWaste`)&quot;">​</a></h4><p>Neither entry&#39;s <code>detect()</code> used to put anything beyond a scalar <code>metric</code>/ <code>value</code> on its finding. Both now also attach:</p><ul><li><code>numTasks</code>: <code>stage.taskCount</code> at detection time.</li><li><code>memoryBytesSpilled</code>: <code>stage.memoryBytesSpilled</code> at detection time.</li><li><code>stageFailed</code> only: <code>failedTaskDetails</code>: up to 20 <code>FailedTaskSample</code> records (<code>taskId</code>, <code>attemptNumber</code>, <code>host</code>, <code>executorId</code>, <code>reason</code>, <code>peakExecMem</code>, <code>memSpilled</code>, <code>shuffleWrite</code>) for tasks still marked failed when the stage was finalized (<code>finalizeStage</code>, <code>stage-quantiles.ts</code>).</li><li><code>retryWaste</code> only: <code>retriedTaskDetails</code>: up to 20 <code>FailedTaskSample</code> records for attempts discarded by the retry-dedup logic in <code>accumulateTask</code> (<code>event-handlers.ts</code>): captured at the moment they&#39;d otherwise be thrown away, since by finalize time only the winning attempt survives.</li></ul><p>Both sample arrays are capped at 20 entries, filled in first-encountered order (finalize order for <code>failedTaskDetails</code>, discard order for <code>retriedTaskDetails</code>), not spread across distinct hosts/executors: a stage with failures clustered on one bad host could fill the cap before a more informative failure elsewhere in the stage is ever sampled. This is the first evidence-shape documentation in this file; no other finding type has one yet.</p>',39)])])}const p=t(s,[["render",c]]);export{g as __pageData,p as default};

package/public/docs/assets/contributor-guide_architecture_impact-estimation.md.sf9DU2bA.js

@@ -0,0 +1 @@
+import{_ as t,o,c as a,a5 as s}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Impact estimation","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/impact-estimation.md","filePath":"contributor-guide/architecture/impact-estimation.md"}'),d={name:"contributor-guide/architecture/impact-estimation.md"};function c(i,e,r,n,l,h){return o(),a("div",null,[...e[0]||(e[0]=[s('<h1 id="impact-estimation" tabindex="-1">Impact estimation <a class="header-anchor" href="#impact-estimation" aria-label="Permalink to &quot;Impact estimation&quot;">​</a></h1><p>Every finding covered by this section carries an optional <code>impactEstimate: {basis, wallClock, estimateMethod, rawWaste?}</code> (<code>src/types.ts</code>), attached by <code>src/impact-estimator.ts</code> as a post-pass after <code>DETECTORS</code> finishes (<code>src/analyzer.ts</code>). <code>estimateMethod</code> (&#39;measured&#39; | &#39;modeled&#39; | &#39;none&#39;) is a distinct axis from the per-finding <code>confidence</code> field (<a href="./board-widgets.html#confidence-metadata">Confidence metadata</a>): <code>confidence</code> says how much to trust the finding itself, <code>estimateMethod</code> says how its impact number was derived. <code>&#39;none&#39;</code> marks a purely informational finding with no waste model at all (<code>configAudit</code>, <code>stageFailed</code>, <code>failures</code>, <code>incompleteRun</code>, <code>slowHost</code>&#39;s byte-dimension multiDim shapes, <code>gc</code>&#39;s low-GC direction): it&#39;s distinct from <code>&#39;measured&#39;</code>/<code>&#39;modeled&#39;</code>, which both attach a real (if approximate) formula. <code>basis</code> is one of:</p><ul><li><code>&#39;serial&#39;</code>: the tied stage ran (effectively) alone; <code>wallClock</code> is a near-point estimate, <code>low === high</code>.</li><li><code>&#39;contended&#39;</code>: the tied stage shared wall-clock time with others; <code>wallClock</code> is an honest range, <code>high</code> optimistic (assumes the fix could still fully land), <code>low</code> the guaranteed floor.</li><li><code>&#39;resourceOnly&#39;</code>: no wall-clock claim is defensible (not stage-tied by nature, or the stage was excluded from the occupancy sweep), <code>wallClock: null</code>, but the formula&#39;s real signal survives in <code>rawWaste</code>.</li><li><code>&#39;informational&#39;</code>: no quantifiable magnitude at all, <code>wallClock: null</code>, no <code>rawWaste</code>.</li></ul><p><code>{basis: &#39;resourceOnly&#39;|&#39;informational&#39;, wallClock: null}</code> replaced the earlier design&#39;s <code>{low: 0, high: 0}</code>: that single value used to mean two incompatible things (&quot;provably no wall-clock cost&quot; and &quot;the model gave up&quot;), and 76-97% of stage-tied findings on real logs were the second case wearing the first case&#39;s clothing (2026-08-30 N1 redesign; see <code>docs/superpowers/specs/2026-08-30-critical-path-occupancy-redesign.md</code>).</p><p><code>rawWaste</code> is not exclusive to <code>resourceOnly</code>/<code>informational</code> findings: every <code>serial</code>/ <code>contended</code> finding carries it too (the only exceptions are <code>coldStart</code>, whose <code>wallClock</code> figure is already unclipped, and <code>estimateMethod: &#39;none&#39;</code> findings, which have no formula at all), holding the formula&#39;s pre-clip magnitude in the formula&#39;s own natural unit (ms, bytes or core-ms). <code>wallClock</code> is what the occupancy model says is recoverable, which clips against the stage&#39;s own physical floor; <code>rawWaste</code> is what the stage really wasted either way. The two answer different questions.</p><h2 id="occupancy-weighted-attribution" tabindex="-1">Occupancy-weighted attribution <a class="header-anchor" href="#occupancy-weighted-attribution" aria-label="Permalink to &quot;Occupancy-weighted attribution&quot;">​</a></h2><p><code>src/occupancy.ts</code> sweeps every stage&#39;s observed <code>[submittedAt, completedAt)</code> window and splits each instant&#39;s wall-clock among concurrently-active stages proportional to <code>coreWeight(S) = stage.executorRunTime / stageDurationMs</code> (an average-concurrency proxy, held constant across the stage&#39;s whole window: this codebase has no per-task timestamps outside the parser worker to do better). Summing a stage&#39;s share across its own window gives its <code>occupancy(S)</code>; <code>gate(S) = occupancy(S) / duration(S) ∈ [0, 1]</code> is the single number that replaces the old CPM model&#39;s <code>onCriticalPath</code>/<code>slackMs</code>/<code>isUniquelyCritical</code>. 1.0 means the stage ran completely alone; 0 means the stage had zero <code>executorRunTime</code> while overlapping other, positive-weight stages, so it got no share of the shared window. Stages with <code>duration(S) &lt;= 0</code> (Spark-skipped stages, or a malformed <code>submittedAt === completedAt</code>) are excluded from the sweep entirely.</p><p>This mechanism replaced a CPM (critical-path-method) graph over <code>parentIds</code> that produced near-zero on-critical-path membership on real logs (0.1-10.6% of stages, max graph depth 2 on 5 of 6 real logs measured): <code>parentIds</code> alone is too sparse a precedence signal for a meaningful longest-path computation. The same degeneracy fed <code>efficiency-model.ts</code>&#39;s <code>floorInfiniteMs</code> (&quot;floor with infinite executors&quot;); rather than leave a second, unreconciled critical-path number in the codebase for a future UI to display next to the occupancy-based figures above, <code>criticalPathMs()</code>/<code>CriticalPathStage</code> (embedded in <code>efficiency-model.ts</code>, never a standalone module) were removed outright (no occupancy-based replacement: occupancy apportions observed concurrent time, it doesn&#39;t compute a dependency-graph longest path, so there&#39;s no drop-in equivalent). <code>efficiency-model.ts</code> now reports only <code>floorZeroSkewMs</code> (total task time / total cores) as its theoretical floor.</p><p><code>ceiling(S) = max(stage.taskDurationMax, stage.executorRunTime / totalCores)</code> is a physical floor on a stage&#39;s own duration: bounded below by its single longest task (unsplittable no matter how much parallelism exists) or by its core-work spread across every core in the cluster, whichever is larger. Every waste formula&#39;s raw claim is clipped against it before gate-weighting: <code>wasteMs_clipped(S) = min(wasteMs_claimed, max(0, duration(S) - ceiling(S)))</code>, so a finding can never claim to save more than the portion of the stage&#39;s observed duration that sits above its own unbeatable floor. This is what fixes historical overclaim bugs (a <code>tinyTask</code> finding claiming 407.5s on a 13.1s stage capped to 8.9s; a <code>shuffle</code> finding claiming 1939.9s on a 991.3s/1688.3s stage capped to 610.3s/250.1s).</p><p><code>skew</code> and <code>straggler</code> are the exception (<code>estimateSingleStage</code>&#39;s <code>shortensLongestTask</code> option, passed by both their <code>impact-estimator.ts</code> cases and <code>detectors.ts</code>&#39;s <code>clippedWasteMs</code> runtime-floor gate, so firing and display agree; <code>stageSlowness</code>&#39;s more-partitions estimate passes it too, since splitting partitions splits the longest task). Their claim shortens the stage&#39;s longest task itself, so <code>taskDurationMax</code> can&#39;t be their floor: clipping against it capped a stage gated by one straggler at <code>duration(S) − taskDurationMax</code>, about zero, exactly when the fix recovers the most. Their floor is instead the longest task the fix leaves plus the core work the fix leaves, <code>max(taskDurationMax − wasteMs_claimed, longestTaskAfterFixMs, (stage.executorRunTime − removed) / totalCores)</code>, where <code>removed</code> (<code>tailRemovedWorkMs</code>) is the larger of the longest task&#39;s excess over P50 and <code>stragglerExcessMs</code>, and <code>longestTaskAfterFixMs</code> is the longest task the fix leaves for <code>skew</code> and <code>straggler</code> (see below). Counting the stragglers&#39; own run time as work the stage can&#39;t shed floored a stage whose tail is most of its core time near its observed duration: one real stage claimed 5.9s where the replay below recovers 38.7s, and now claims 38.7s. Scored against a list-scheduling replay of each flagged stage&#39;s own tasks (slots = the stage&#39;s observed peak concurrent tasks; recoverable = replay with actual durations minus replay with every task over 4× P50 capped at P50), across 765 skew/straggler findings on 14 real logs (2026-09-23, <code>dev/bench-analyze.mjs</code> snapshots): estimates more than 2× under the replay dropped from 199 to 11, estimates within 2× rose from 556 to 738, mean absolute error fell from 6.64s to 5.59s. Overclaims by more than 2× went from 10 to 16: stages where AQE or free slots absorbed the tail, which a stage-level model can&#39;t see.</p><p>The claim itself is <code>tailRecoveryMs</code> (<code>occupancy.ts</code>), which returns the stage&#39;s <code>tailReplayRecoveryMs</code>: <code>finalizeStage</code> replays the stage&#39;s own tasks with list scheduling (<code>computeTailReplayRecoveryMs</code> in <code>stage-quantiles.ts</code>: slots = <code>peakConcurrentTasks</code>, at least 1 and at most the task count; tasks in launch order, equal launch times in task-array order, each on the slot that frees first) once with the real durations and once with every task over 4× P50 capped at P50, and takes the difference. It runs only on stages with a task over 4× P50 (every other stage recovers 0) and costs a launch-order index and a slot heap per stage: 5-9 ms over the 236 such stages (23,641 tasks) of the largest real log, whose parse takes 5.5-6.9 s. <code>dev/eval-tail-replay.mjs</code> keeps an independent copy of the replay as its ground truth, and the two agree on all 919 tail stages of the 14 real logs and the corpus.</p><p>A stage without the field (one built by hand, as in detector tests) falls back to the estimate the replay replaced: the larger of the single-task delta above and <code>stragglerExcessMs / peakConcurrentTasks</code>, the summed excess over P50 of every task slower than 4× P50, spread over the most tasks the stage ever ran at once. That estimate assumed the tail either gates the stage through one task or spreads evenly over the slots; the replay knows when each slow task launched. Stragglers clustered at the end of a stage recover more than the even spread (one real stage: 12 of 200 tasks over 4× P50 on 29 slots, 89.0s estimated, 127.3s replayed), a long task launched early overlaps the rest of the stage and recovers less than its excess (8.4s estimated, 3.0s replayed), and a speculation-driven stage with no task over 4× P50 recovers nothing (5.8s estimated on two real stages). Moving to the replay, on the 14 real logs: 4 skew/straggler findings added, 2 removed (a skew estimate of 29.3s on a stage whose replay recovers 5.3s, and one whose 15.4s replay falls just under the 0.5% floor), 3 moved from warning to info (the two speculation-driven stages, and the straggler finding on that 15.4s stage) and 2 from warning to critical (one stage whose clustered tail replays at 82.7s where the estimate said 51.3s). No corpus finding is added, removed or re-banded. Scored over 65 runs (14 real logs plus the corpus), against the replay skew precision goes 0.96 → 1.00 at recall 0.81 → 0.85 and straggler 0.94 → 1.00 at recall 0.74 → 0.76. Every non-info estimate now sits within 2× of the replay (98 of 101 before), since the post-fix floors below never clip the replayed figure on these logs.</p><p>The single-task deltas still set the removed work above and the fallback estimate. <code>straggler</code>&#39;s runs from <code>taskDurationMax</code> down to the longest task its fix leaves (<code>stragglerFixLongestTaskMs</code>), not down to P50: the fix brings every task over 4× P50 to the median, so the stage still waits on its longest task at or under that (<code>longestNonStragglerMs</code>, from <code>finalizeStage</code>), and that task is also passed as a floor (<code>longestTaskAfterFixMs</code>). A speculation-driven finding with no task over 4× P50 keeps the P50 delta. <code>skew</code> keeps its P50 delta, since repartitioning may even out tasks under 4× P50 too, but takes the same <code>longestTaskAfterFixMs</code> floor: without it a 100 s stage whose next-longest task ran 39 s had skew claiming 90 s where straggler claimed 61 s. Scored the same way over the 65 runs: more than 2× over 6 → 4, mean absolute error 5.18s → 4.49s, straggler precision 0.92 → 0.94 at recall 0.76 → 0.74 (one stage at 0.61% of the run now estimates under the 0.5% floor, one below it no longer fires). The skew floor, scored the same way: within 2× 97 → 98 of 101, more than 2× over 4 → 3, mean absolute error 4.49s → 4.12s, precision and recall unchanged.</p><p><code>analyzer.ts</code> feeds this <code>totalCores</code> from <code>src/core-count.ts</code>&#39;s <code>computePeakConcurrentCores(app, executorsAdded, executorsRemoved)</code>, not the shared <code>computeTotalCores</code> helper. <code>computeTotalCores</code> sums every <code>ExecutorAdded</code> event&#39;s cores regardless of overlap, so under dynamic allocation or executor replacement it can far exceed the cores ever actually concurrent, which understates <code>ceiling(S)</code> and lets churn inflate a finding&#39;s claimed wall-clock. <code>computePeakConcurrentCores</code> instead sweeps add/remove events by timestamp and tracks the running total&#39;s peak, so a churned-through executor&#39;s cores are never double-counted against its replacement&#39;s. Same-timestamp events tie-break by delta ascending, so a removal applies before a same-instant replacement&#39;s addition (otherwise a same-instant swap would momentarily double-count both as concurrent). If every <code>executorsAdded</code> entry lacks <code>totalCores</code> the cores sweep peaks at zero and tells us nothing; the function then falls back to sweeping peak <em>executor count</em> instead (still concurrency-aware, just cores-blind) and multiplies by the configured per-executor core count, rather than falling back to <code>executorsAdded.length × cores</code>, which would reintroduce the exact cumulative-overcount-under-churn bug this function exists to avoid.</p><p>Per-finding estimate, using <code>wasteMs_clipped(S)</code>:</p><ul><li><code>gate(S) &gt;= 0.999</code>: <code>basis: &#39;serial&#39;</code>, <code>low = high = wasteMs_clipped(S)</code>.</li><li><code>gate(S) &lt; 0.999</code>: <code>basis: &#39;contended&#39;</code>, <code>high = wasteMs_clipped(S)</code>, <code>low = wasteMs_clipped(S) * gate(S)</code>.</li></ul><p>A finding spanning multiple stages (<code>stageIds</code>, plural) sums each stage&#39;s own estimate and caps the joint total at the union of just that finding&#39;s own stage windows (via <code>mergeIntervals</code>, <code>src/wall-clock.ts</code>): <code>high = min(Σ high_i, unionMs(stageIds))</code>, <code>low = min(Σ low_i, unionMs(stageIds))</code>. This is what prevents overclaiming when two or more of a finding&#39;s stages overlap in wall-clock time: a plain sum-and-cap, no CPM re-simulation. The union cap can force <code>low === high</code> numerically even when the constituent stages were individually contended (e.g. two fully-overlapping stages each at <code>gate</code> 0.5), so <code>basis</code> isn&#39;t derived from that numeric equality: a multi-stage finding gets <code>basis: &#39;serial&#39;</code> only when every one of its per-stage estimates was itself <code>&#39;serial&#39;</code>; otherwise <code>&#39;contended&#39;</code>.</p><p>Regression guard: a finding whose stage (or, for a multi-stage finding, every one of its stages) ran alone (<code>gate &gt;= 0.95</code>, deliberately looser than the <code>0.999</code> &quot;serial&quot; cutoff above: this guard exists to catch egregious false zeros, not to gate which <code>basis</code> a finding gets) with a real underlying magnitude (<code>rawWaste.value &gt; 0</code>) must never report <code>wallClock.high === 0</code> or <code>basis: &#39;informational&#39;/&#39;resourceOnly&#39;</code>. Covered by <code>tests/impact-estimator-real-log.test.js</code> against a real fixture; relies on <code>rawWaste</code> being attached to every serial/contended-capable formula (see above), so it&#39;s blind only to <code>coldStart</code> and the purely informational (<code>estimateMethod: &#39;none&#39;</code>) finding types.</p><p>Real-log spot-check (2026-08-30, <code>private-log-01.zstd</code>): median <code>gate</code> across stages was <code>≈0.34</code> (0.3428060791718594 exactly); <code>collectRun</code> plus the occupancy sweep together took <code>≈6,589</code>ms on the largest fixture measured (<code>private-log-03.zstd</code>, <code>1169</code> stages): parse-dominated, the sweep alone was not isolated by this measurement, but not a magnitude that suggests a regression either. <code>54</code> previously-<code>{0,0}</code> stage-tied findings on stages that ran effectively alone now report a real <code>wallClock</code> range instead.</p><h2 id="cross-finding-rollup-computestageunionms" tabindex="-1">Cross-finding rollup: <code>computeStageUnionMs</code> <a class="header-anchor" href="#cross-finding-rollup-computestageunionms" aria-label="Permalink to &quot;Cross-finding rollup: `computeStageUnionMs`&quot;">​</a></h2><p>The Findings tab&#39;s recommendation rollup (<code>FixTheseFirst.tsx</code>, built from <code>buildRecommendationRollup</code>, <code>src/recommendation-rollup.ts</code>) groups the filtered catalog by detector <code>type</code>, then needs its own cap for a group of several findings of that type, not just one finding&#39;s own <code>stageIds</code>. Summing each finding&#39;s already-clipped <code>wallClock.high</code> naively double-counts any stage two of those findings both touch. <code>computeStageUnionMs(stageIds, stages)</code> covers this: collect every stage touched by any finding in the group, merge their <code>[submittedAt, completedAt)</code> intervals, and sum the merged intervals&#39; durations, so the group&#39;s wall-clock union holds regardless of how many findings&#39; <code>stageIds</code> overlap. Stages missing either bound are skipped rather than defaulted to <code>0</code> (the same filter <code>computeWallClock</code> applies), so a truncated log (the case <code>incompleteRun</code> flags) can&#39;t contribute a negative interval and a negative recoverable-time figure.</p><p>It reuses the same <code>mergeIntervals</code> primitive (<code>src/wall-clock.ts</code>) that backs <code>src/occupancy.ts</code>&#39;s per-finding <code>estimateMultiStage</code>/its internal <code>unionMs</code> sum, but is not an extension of that function: <code>estimateMultiStage</code> caps one finding&#39;s own multi-stage claim during the impact-estimation pass, before a <code>Finding</code> object even exists; <code>computeStageUnionMs</code> runs later, in the view layer, capping a naive sum <em>across</em> several already-estimated findings that happen to share a detector <code>type</code>. <code>buildTimeGroup</code> (same module) takes the smaller of the naive per-finding sum and this union figure as the group&#39;s <code>recoverableMsHigh</code>, falling back to the naive sum untouched when the group&#39;s findings carry no stage IDs at all (nothing to union against).</p><p>Within one <code>type</code> group, <code>buildRecommendationRollup</code> splits findings into up to three tiers, always rendered in this fixed order. <code>time</code> covers findings with a real <code>impactEstimate.wallClock</code> (<code>buildTimeGroup</code>, the union-capped figure above). <code>resource</code> covers findings with no <code>wallClock</code> but a <code>rawWaste</code> figure (<code>buildResourceGroup</code>), grouped again by <code>rawWaste.unit</code> so a <code>bytes</code> total never gets summed against a <code>coreHours</code> total under one type. <code>count</code> covers findings with neither (<code>buildCountGroup</code>), a plain per-impact-band tally with no magnitude claim at all. Each <code>RollupGroup</code> also carries its own <code>findings: Finding[]</code> (the exact members that fed the aggregate), which <code>FixTheseFirst.tsx</code> reads directly to pick a group&#39;s highest-impact member and to render its expanded, paginated list.</p><p>A type only contributes a tier when it has at least one finding of that kind; most types produce exactly one tier, but a type whose formula varies by <code>variant</code>/<code>rule</code> (e.g. <code>memoryUtilization</code>, see the coverage table below) can produce more than one.</p><p><code>cachingOpportunity</code> and <code>cacheUtilization</code> are both <code>cost-only</code>: <code>basis: &#39;resourceOnly&#39;</code>, <code>wallClock: null</code>, but <code>rawWaste.unit</code> is <code>&#39;ms&#39;</code>, the same unit a real <code>wallClock</code> figure would use, because their formula&#39;s natural output happens to be time (a re-read cost), not because either finding makes a wall-clock claim. Left unlabeled, a <code>resource</code>-tier &quot;ms&quot; total sitting next to a <code>time</code>-tier &quot;recoverable time&quot; total would read as directly comparable when it isn&#39;t: the resource figure was never gate-clipped against any stage&#39;s occupancy, so it can exceed what the stage actually spent. <code>FixTheseFirst.tsx</code> calls this out via its trailing-stat copy: a <code>resource</code>- kind group (any unit, including <code>ms</code>) always reads &quot;resource-cost projection&quot;, never &quot;recoverable&quot;, so the two ms-shaped numbers are never mistaken for the same kind of claim.</p><h2 id="per-formula-spot-checks" tabindex="-1">Per-formula spot-checks <a class="header-anchor" href="#per-formula-spot-checks" aria-label="Permalink to &quot;Per-formula spot-checks&quot;">​</a></h2><table tabindex="0"><thead><tr><th>Detector</th><th>Formula basis</th><th>Spot-check</th></tr></thead><tbody><tr><td>gc</td><td><code>jvmGCTime / (executorRunTime / stageDurationMs)</code></td><td><code>private-log-01.zstd</code>, stage 507: <code>jvmGCTime</code>=1080ms, <code>executorRunTime</code>=27509ms, <code>stageDurationMs</code>=56279ms → <code>wasteMs</code> = 1080 / (27509/56279) ≈ 2209.5ms. That&#39;s ≈3.9% of the stage&#39;s 56.3s wall-clock duration, matching the finding&#39;s own reported <code>gcPct</code> (3.9%) exactly, as the formula guarantees by construction. Under the occupancy model this stage&#39;s <code>gate</code> is <code>0.041</code> (0.04145044590332269 exactly): <code>basis: &#39;contended&#39;</code>, <code>wallClock: {low: 91.6, high: 2209.5}</code> (91.5850382272182 / 2209.506706895925 exactly, per the Step 1 script&#39;s per-stage output).</td></tr><tr><td>shuffle</td><td><code>shuffleReadBytes / (SHUFFLE_THROUGHPUT_BPS × executors that ran the stage)</code></td><td><code>private-log-02.zstd</code>, stage 99 (<code>SHFL</code> finding): <code>shuffleReadBytes</code>=204,172,518,504 over 8 executors → <code>wasteMs</code> = 204172518504 / (8 × 125,000,000) × 1000 ≈ 204,173ms, against the stage&#39;s 763,776ms duration. The tasks&#39; own measured shuffle fetch wait on this stage is 25.2s of wall-clock (<code>fetchWaitTime</code> / average concurrency), so even the per-link model runs well above the network stall actually observed, and the claim is capped there: 25.2s, <code>measured</code>. The pre-2026-09-23 formula divided by one link&#39;s bandwidth (1,633,380ms, 2.1× the stage&#39;s whole duration); before the fetch-wait cap, the occupancy clip (gate <code>1</code>) held the per-link figure to <code>≈111,923.8</code>ms, the room above this stage&#39;s core-work floor.</td></tr><tr><td>spill</td><td><code>diskBytesSpilled / (SPILL_IO_THROUGHPUT_BPS × executors that ran the stage)</code></td><td>Same run and stage (99): <code>diskBytesSpilled</code>=145,978,433,675 (note: the <code>SPILL</code> finding&#39;s own <code>value</code>/<code>metric</code> report <code>memoryBytesSpilled</code>=913,686,966,448, ~6x larger; the formula correctly uses the smaller disk figure, not that one) over 8 executors → <code>wasteMs</code> = 145978433675 / (8 × 200,000,000) × 1000 ≈ 91,237ms (91236.521046875 exactly, matching <code>wallClock</code>, below the clip).</td></tr></tbody></table><h2 id="overlap-caveat-skew-straggler" tabindex="-1">Overlap caveat: skew / straggler <a class="header-anchor" href="#overlap-caveat-skew-straggler" aria-label="Permalink to &quot;Overlap caveat: skew / straggler&quot;">​</a></h2><p><code>skew</code> (small-stage max-P50 fallback branch) and <code>straggler</code> can both fire on the same stage from the same single dominant outlier task, and each is clipped independently. This phase does not dedupe or suppress either: each keeps its own independently-computed <code>wallClock</code>. Do not sum <code>wallClock.high</code> across multiple findings on the same stage: if both fire together, they describe the same underlying waste, not two separate wastes. Both are clipped with the post-fix floor described under <a href="#occupancy-weighted-attribution">Occupancy-weighted attribution</a>, not the plain <code>ceiling</code>, so a stage gated by one dominant outlier task reports that task&#39;s excess as recoverable instead of the near-zero room <code>ceiling &gt;= taskDurationMax</code> would leave.</p><p><code>analyzer.ts</code>&#39;s <code>flagSkewStragglerOverlap</code> (run after <code>deriveImpactBand</code>, once per <code>analyze()</code> call) surfaces this caveat to the reader instead of leaving it as an internal-only comment: whenever <code>skew</code>&#39;s <code>max/median</code> branch and <code>straggler</code> both fire on the same <code>stageId</code>, it appends a &quot;this overlaps with the X finding on this stage&quot; sentence to both findings&#39; <code>validationRequired</code> text (rather than suppressing either, so neither finding&#39;s own diagnostic value is lost). <code>skew</code>&#39;s <code>P95/median</code> branch samples a different task from <code>straggler</code>&#39;s own <code>taskDurationMax - taskDurationP50</code> delta, so it&#39;s excluded from the flag. The note rides the same confidence-caveat UI (<code>RowStatusCluster</code>) a reader already sees before trusting either finding&#39;s magnitude, since both detectors also carry a <code>confidence</code> field that scales <code>low</code>/<code>medium</code>/<code>high</code> off how far the finding sits past its own runtime-floor threshold (still unvalidated; see the confidence-disclosure note in detector-contract.md).</p><p><code>stageShape</code>&#39;s <code>taskStageSkew</code> rule no longer participates in this caveat: it reports a <code>resourceOnly</code> idle-core-ms figure (see the coverage table below) instead of a wall-clock claim, so there&#39;s nothing left to double-count against <code>skew</code>/<code>straggler</code>. Its trigger condition (<code>taskDurationMax / stageDurationMs &gt; skewWarn</code>) mathematically forces the occupancy-clipped wall-clock estimate to exactly zero on every firing (see <code>src/detectors.ts</code>&#39;s <code>taskStageSkew</code> comment), which is why it was moved off the wall-clock path entirely rather than reconciled against the same ceiling clip as its two siblings above.</p><h2 id="per-finding-type-coverage" tabindex="-1">Per-finding-type coverage <a class="header-anchor" href="#per-finding-type-coverage" aria-label="Permalink to &quot;Per-finding-type coverage&quot;">​</a></h2><p>One row per distinct <code>type</code> string <code>src/detectors.ts</code> actually emits (cross-checked against <code>computeEstimateForFinding</code>&#39;s <code>case</code> labels in <code>src/impact-estimator.ts</code>, not assumed from the prose here): every row below has a case, so the table itself is the coverage count, not a number restated here. <code>broadcastSizing</code> is a <code>DETECTORS</code> entry label only, and the plan-walk it drives emits <code>overBroadcast</code>/<code>underBroadcast</code> findings instead, so those two are the rows that appear, not <code>broadcastSizing</code> itself. Tag meanings: <code>measured</code> and <code>modeled</code> both produce a real, gate-clipped, non-<code>{0,0}</code><code>wallClock</code> (the difference is whether the formula&#39;s inputs are recorded per-stage fields or an assumed constant like a throughput figure); <code>cost-only</code> always reports <code>basis: &#39;resourceOnly&#39;</code>, <code>wallClock: null</code> but carries its real signal in <code>rawWaste</code>; <code>informational-only</code> reports <code>basis: &#39;informational&#39;</code>, <code>wallClock: null</code> with no <code>rawWaste</code> at all, since there&#39;s nothing quantifiable. A type with more than one tag fires a different formula per <code>variant</code>/<code>rule</code> on the same finding type; the basis column says which.</p><table tabindex="0"><thead><tr><th>Finding type</th><th>Scope</th><th>Tag</th><th>Basis</th></tr></thead><tbody><tr><td><code>retryWaste</code></td><td>stage</td><td>measured / modeled</td><td><code>retryWasteMs</code> is summed attempt time, and attempts of different tasks run side by side (one lost executor fails every task it was running at once). With every wasted attempt sampled (<code>retryTaskSamples</code>, capped at 20), modeled: the longest retry chain (highest sampled attempt number + 1) times the mean wasted attempt, or <code>retryWasteMs / peakConcurrentTasks</code> when larger, capped at <code>retryWasteMs</code>. Otherwise measured, <code>retryWasteMs</code> itself. Gate-clipped; the summed figure is kept as <code>rawWaste</code> in <code>ms</code>. On the 14 real logs the one non-info finding (4 first attempts on one lost executor, 41 slots) went from 146.6s to 36.6s</td></tr><tr><td><code>speculationWaste</code></td><td>stage</td><td>measured</td><td><code>speculationWasteMs</code>, gate-clipped; pre-clip figure kept as <code>rawWaste</code> in <code>ms</code></td></tr><tr><td><code>coldStart</code></td><td>app</td><td>measured</td><td><code>gapSeconds × 1000</code> (first stage submitted to first executor added), unclipped, <code>basis: &#39;serial&#39;</code> unconditionally (a pre-first-task gap can&#39;t overlap any stage)</td></tr><tr><td><code>gc</code></td><td>stage</td><td>modeled / informational-only</td><td>high-GC: <code>jvmGCTime / (executorRunTime / stageDurationMs)</code>, gate-clipped: the concurrency division is an approximation, not a reconstruction, hence <code>modeled</code>; <code>rawWaste</code> in <code>coreMs</code> is the raw <code>jvmGCTime</code> sum before that conversion. Low-GC (<code>direction: &#39;low&#39;</code>): informational-only, since its fix (less executor memory) raises GC rather than recovering it; it used to claim the stage&#39;s GC time as savings, which promoted 10 of 679 low-GC findings on 14 real logs to warning/critical</td></tr><tr><td><code>skew</code></td><td>stage</td><td>measured</td><td><code>tailRecoveryMs</code>: the stage&#39;s <code>tailReplayRecoveryMs</code> (a task-level replay with every task over 4× P50 capped at P50), gate-clipped against the post-fix floor (<code>shortensLongestTask</code>) with the longest task the fix leaves as a floor (<code>longestTaskAfterFixMs</code>, as for <code>straggler</code>); pre-clip figure kept as <code>rawWaste</code> in <code>ms</code></td></tr><tr><td><code>straggler</code></td><td>stage</td><td>measured</td><td><code>tailRecoveryMs</code>, as for <code>skew</code> (0 on a speculation-driven stage with no task over 4× P50), gate-clipped against the post-fix floor (<code>shortensLongestTask</code>, <code>longestTaskAfterFixMs</code>)</td></tr><tr><td><code>stageShape</code></td><td>stage</td><td>cost-only</td><td>all three rules are <code>estimateMethod: &#39;measured&#39;</code>, real per-stage fields, no assumed constant: <code>&#39;lowParallelism&#39;</code> → <code>rawWaste</code> in <code>coreMs</code> (idle cores × stage duration); <code>&#39;dataExplosion&#39;</code> → <code>rawWaste</code> in <code>bytes</code> (<code>outputBytes − inputBytes</code>); <code>&#39;taskStageSkew&#39;</code> → <code>rawWaste</code> in <code>coreMs</code> (<code>max(0, min(totalCores, taskCount) − 1) × (taskDurationMax − taskDurationP50)</code>, the cores idle during the straggler&#39;s tail at achieved concurrency)</td></tr><tr><td><code>slowHost</code></td><td>stage</td><td>measured / informational-only</td><td>duration-based variants (<code>hostMeanRatio</code>, <code>durationShare</code>, <code>multiDim</code>+<code>taskTime</code>): <code>value − taskDurationP50</code>, gate-clipped; byte-based <code>multiDim</code> dimensions: no formula yet</td></tr><tr><td><code>duplicatePlanSubtree</code></td><td>sql</td><td>measured</td><td>per stage in <code>stageShares</code>: its task-active time (<code>taskActiveMs</code>, the union of its task intervals; submit-to-complete only when absent) × the repeated operators&#39; share of that stage × the redundant fraction <code>(occurrences − 1) / occurrences</code>, summed and capped at the union of those stages&#39; spans. A stage shared with other operators (the consuming join, the other join side) contributes only its share, so sibling groups can&#39;t claim one stage twice, and a stage left waiting for cores (2491 s open, 60 s of tasks on a real log) claims only its task time. No claim (<code>informational</code>) when the repeats&#39; details differ (<code>occurrencesIdentical: false</code>) or no repeated operator has a stage (the execution-wide <code>stageIds</code> fallback stays for linking only). On the 14 real logs: 2307 min claimed before, including more duplicate time than the whole run on 3 logs (1661 of 458 min, 257 of 61, 338 of 68); 108 min after, critical 65 → 2, warning 54 → 21</td></tr><tr><td><code>shuffle</code></td><td>stage</td><td>modeled / measured</td><td><code>shuffleReadBytes / (SHUFFLE_THROUGHPUT_BPS × executors)</code> (assumed ~125MB/s per executor link; <code>executors</code> = <code>executorStats.length</code>, the executors that ran the stage, min 1), capped at the fetch wait the tasks measured in wall-clock (<code>fetchWaitTime / (executorRunTime / stage duration)</code>, the <code>gc</code> conversion; <code>measured</code> when that cap binds), gate-clipped; <code>rawWaste</code> in <code>bytes</code> is the measured <code>shuffleReadBytes</code> behind it. The link model can&#39;t see whether reads stalled the tasks: on 14 real logs (2026-09-23) it claimed 2780s over 284 shuffle findings where the capped figure is 256s, and 5 of the 10 non-info findings had about zero fetch wait (they are now <code>info</code>). Fetch wait misses disk reads and deserialization, so the cap is a floor on what a shuffle costs, not its total. Dividing by one link instead modeled the whole cluster as a single 1 Gbps pipe: on 14 real logs (2026-09-23) that claimed 721 minutes of shuffle+spill savings on a 458-minute app, on stages where 222 of 284 shuffle findings measured ~0s of task fetch wait</td></tr><tr><td><code>spill</code></td><td>stage</td><td>modeled</td><td><code>diskBytesSpilled / (SPILL_IO_THROUGHPUT_BPS × executors)</code> (assumed ~200MB/s per executor&#39;s local disk, same executor count as <code>shuffle</code>), gate-clipped; <code>rawWaste</code> in <code>bytes</code> is <code>diskBytesSpilled</code>, which is the number the formula uses and not the <code>memoryBytesSpilled</code> the finding&#39;s own <code>metric</code> displays</td></tr><tr><td><code>stageSlowness</code></td><td>stage</td><td>modeled</td><td>what more partitions (the finding&#39;s recommendation) could recover: the stage&#39;s task-active time (<code>taskActiveMs</code>, the union of its tasks&#39; launch-to-finish intervals from <code>finalizeStage</code>) × <code>max(0, 1 − taskCount / totalCores)</code>, gate-clipped with the post-fix floor (<code>shortensLongestTask</code>: splitting partitions splits the longest task too). A stage that already ran at least as many tasks as the cluster had cores claims 0, and so does one that read no input and no shuffle bytes and whose tasks spent under 1% of <code>executorRunTime</code> on CPU (<code>executorCpuTime</code>): they sat waiting on something outside Spark, which more partitions don&#39;t split (a 1-task JDBC <code>count</code> stage open 27 minutes on 5s of CPU, 0.33%, had claimed 99%). On 14 real logs (2026-09-23) every non-Python stage under 1% was a JDBC read, a file listing or a Delta log read, and file writes start at 2%. The share is skipped, keeping the claim, when the log records no CPU time (older Spark) or the stage ran Python through <code>PythonRDD</code>, whose worker CPU the JVM metric misses (0.1% on computing stages); a Python UDF inside a SQL stage that reads no bytes isn&#39;t detected and can still read idle. A stage that read input or shuffle bytes keeps its claim whatever its CPU share, so a Python UDF over real input is never zeroed. This narrowed a gate that zeroed any stage reading no input and no shuffle bytes, which also zeroed stages computing from generated data or only writing output: over every stage of the 14 real logs, run through the estimate as if each had fired, 636 stages regain 521 s and none lose; the 35 findings those logs raise are unchanged. Time a stage sat open with no task running (queued for slots) claims nothing. No cluster core count: informational. Replaced &quot;stage duration minus the detector&#39;s 15-minute <code>infoMin</code>&quot;, which on 14 real logs (2026-09-23) graded 29 of 35 findings critical, including 1-task stages open 20-40 minutes whose only task ran under a second, while scoring a 27-minute single-task stage 0; before the idle gate, 1 critical (that single-task stage), 1 warning, 33 info, claims 539 → 30 minutes</td></tr><tr><td><code>partitionSizing</code></td><td>stage</td><td>modeled</td><td><code>maxPartitionTooBig</code>/<code>shufflePartitionSkew</code>: shuffle-throughput formulas, gate-clipped. <code>lowShuffleParallelism</code>: stage duration scaled down by the shortfall between actual and ideal-partition-count task counts (<code>stageDurationMs × (1 − taskCount / targetTaskCount)</code>), i.e. the serialized work more partitions would let run concurrently, not the scheduling cost of the tasks you&#39;d add to fix it</td></tr><tr><td><code>tinyTask</code></td><td>stage</td><td>measured / modeled</td><td>excess task count over 10% of the stage&#39;s actual count, × the stage&#39;s own measured per-task overhead (summed task wall time from <code>executorStats</code> minus <code>executorRunTime</code>, over <code>taskCount</code>), ÷ the stage&#39;s achieved task concurrency (task time ÷ stage duration, floored at 1), gate-clipped; <code>measured</code>. A stage with no <code>executorStats</code> or no measurable overhead falls back to the assumed 50ms per task, undivided (<code>modeled</code>). Pre-clip figure kept as <code>rawWaste</code> in <code>ms</code>. Measured overhead on 14 real logs ran 6-62ms per task, near the old constant, but the old formula summed it serially across tasks that actually ran in parallel: total claimed savings fell from 576s to 270s over 160 findings (2026-09-23)</td></tr><tr><td><code>smallFiles</code></td><td>sql</td><td>modeled / cost-only</td><td><code>excessFileCount × FILE_OPEN_OVERHEAD_MS</code>, divided for a read by the most tasks its stages ran at once (<code>peakConcurrentTasks</code>: tasks open their files in parallel; 91,344 files claimed 76 s on a 117 s stage that ran 314 tasks at once) and kept serial for a write (the job commit moves each file on the driver), summed and capped over <code>stageIds</code>&#39;s union; with no <code>stageIds</code> to map to, cost-only with that same figure as <code>rawWaste</code> in <code>ms</code>. <code>stageIds</code> is narrowed the same way (see <a href="./detector-contract.html#stage-id-attribution-for-plan-advisor-findings">Stage-ID attribution for Plan Advisor findings</a>); falls back to the whole execution&#39;s stages when the flagged node(s) have no accumulator coverage.</td></tr><tr><td><code>overBroadcast</code></td><td>sql</td><td>modeled / cost-only</td><td><code>broadcastBytes / BROADCAST_BANDWIDTH_BPS</code>, summed and capped over <code>stageIds</code>&#39;s union; cost-only with <code>rawWaste</code> in <code>ms</code> when not stage-mappable. <code>stageIds</code> is narrowed the same way; falls back to the whole execution&#39;s stages when the flagged node(s) have no accumulator coverage.</td></tr><tr><td><code>underBroadcast</code></td><td>sql</td><td>modeled / cost-only</td><td><code>smallerSideBytes / BROADCAST_BANDWIDTH_BPS</code>, summed and capped over <code>stageIds</code>&#39;s union; cost-only with <code>rawWaste</code> in <code>ms</code> when not stage-mappable. <code>stageIds</code> is narrowed the same way; falls back to the whole execution&#39;s stages when the flagged node(s) have no accumulator coverage.</td></tr><tr><td><code>memoryUtilization</code></td><td>app</td><td>cost-only / informational-only</td><td>Three of the four variants report <code>rawWaste</code> in <code>mbSeconds</code>: <code>variant: &#39;wasteModel&#39;</code> passes through its own <code>wastedMBSeconds</code>; <code>&#39;idleCores&#39;</code> uses <code>idleRateFraction × allocatedMB × peakExecutors × appDurationSeconds</code>; <code>&#39;memoryBand&#39;</code> with <code>rule: &#39;heapOverProvisioned&#39;</code> uses <code>(allocatedBytes − heap) in MB × appDurationSeconds</code>. <code>&#39;memoryBand&#39;</code> with <code>rule: &#39;heapNearCapacity&#39;</code> is an OOM-risk signal rather than a waste, and the <code>dataUnavailable</code> shape has no inputs at all: both informational-only</td></tr><tr><td><code>utilization</code></td><td>app</td><td>cost-only</td><td><code>rawWaste</code> in <code>coreHours</code>: <code>(1 − utilizationFraction) × appDurationMs × totalCores / 3.6e6</code></td></tr><tr><td><code>coreLocality</code></td><td>app</td><td>cost-only</td><td><code>rawWaste</code> in <code>coreMs</code>: <code>nonLocalTaskCount × NETWORK_FETCH_PENALTY_MS</code></td></tr><tr><td><code>autoscalingChurn</code></td><td>app</td><td>cost-only</td><td><code>rawWaste</code> in <code>coreHours</code>: <code>shortLivedExecutorCount × EXECUTOR_STARTUP_OVERHEAD_MS / 3.6e6</code></td></tr><tr><td><code>configAudit</code></td><td>config</td><td>informational-only</td><td>a config-drift check standing alone; no waste formula</td></tr><tr><td><code>jobFailureRate</code></td><td>app</td><td>cost-only</td><td><code>rawWaste</code> in <code>coreHours</code>: <code>failedJobCount × avgJobDurationMs / 3.6e6</code></td></tr><tr><td><code>cachingOpportunity</code></td><td>app</td><td>cost-only</td><td><code>rawWaste</code> in <code>ms</code>: <code>totalReadBytes / RE_READ_THROUGHPUT_BPS</code></td></tr><tr><td><code>cacheUtilization</code></td><td>app</td><td>cost-only</td><td><code>rawWaste</code> in <code>ms</code>: uncached-or-spilled bytes <code>/ RE_READ_THROUGHPUT_BPS</code>, where the never-cached partitions&#39; bytes are extrapolated from the cached partitions&#39; own average size (<code>memorySize + diskSize</code>, over <code>numCachedPartitions</code>), plus <code>diskSize</code> again for the already-cached-but-on-disk partitions&#39; own re-read cost</td></tr><tr><td><code>stageFailed</code></td><td>stage</td><td>informational-only</td><td>no waste formula</td></tr><tr><td><code>failures</code></td><td>stage</td><td>informational-only</td><td>no waste formula</td></tr><tr><td><code>incompleteRun</code></td><td>app</td><td>informational-only</td><td>no waste formula</td></tr></tbody></table>',34)])])}const p=t(d,[["render",c]]);export{g as __pageData,p as default};

package/public/docs/assets/{contributor-guide_architecture_impact-estimation.md.DHVq1UPt.lean.js → contributor-guide_architecture_impact-estimation.md.sf9DU2bA.lean.js}

@@ -1 +1 @@
-import{_ as t,o,c as a,a5 as d}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Impact estimation","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/impact-estimation.md","filePath":"contributor-guide/architecture/impact-estimation.md"}'),s={name:"contributor-guide/architecture/impact-estimation.md"};function c(i,e,r,n,l,h){return o(),a("div",null,[...e[0]||(e[0]=[d("",33)])])}const m=t(s,[["render",c]]);export{g as __pageData,m as default};
+import{_ as t,o,c as a,a5 as s}from"./chunks/framework.DSg0KOwT.js";const g=JSON.parse('{"title":"Impact estimation","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/impact-estimation.md","filePath":"contributor-guide/architecture/impact-estimation.md"}'),d={name:"contributor-guide/architecture/impact-estimation.md"};function c(i,e,r,n,l,h){return o(),a("div",null,[...e[0]||(e[0]=[s("",34)])])}const p=t(d,[["render",c]]);export{g as __pageData,p as default};

package/public/docs/assets/contributor-guide_architecture_overview.md.BXQdgG5u.js

@@ -0,0 +1 @@
+import{_ as o,o as r,c as t,a5 as d}from"./chunks/framework.DSg0KOwT.js";const m=JSON.parse('{"title":"Architecture overview","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/overview.md","filePath":"contributor-guide/architecture/overview.md"}'),a={name:"contributor-guide/architecture/overview.md"};function c(s,e,i,n,l,h){return r(),t("div",null,[...e[0]||(e[0]=[d('<h1 id="architecture-overview" tabindex="-1">Architecture overview <a class="header-anchor" href="#architecture-overview" aria-label="Permalink to &quot;Architecture overview&quot;">​</a></h1><h2 id="core-invariant" tabindex="-1">Core invariant <a class="header-anchor" href="#core-invariant" aria-label="Permalink to &quot;Core invariant&quot;">​</a></h2><p>Task data must never live on the main thread. A Spark job can emit millions of <code>SparkListenerTaskEnd</code> events (240 MB+ of metrics); parsing them on the UI thread freezes Chrome. The design below enforces that in both deploy modes: the static zero-backend app, and the optional local-server mode in <code>server/</code>, which also proxies Spark History Server fetches to sidestep CORS.</p><h2 id="two-actors" tabindex="-1">Two actors <a class="header-anchor" href="#two-actors" aria-label="Permalink to &quot;Two actors&quot;">​</a></h2><h3 id="main-thread" tabindex="-1">Main thread <a class="header-anchor" href="#main-thread" aria-label="Permalink to &quot;Main thread&quot;">​</a></h3><p>Owns the DOM (React) and a summary <code>AppModel</code> (jobs, stages, sql, executors, no raw tasks), held in a Zustand store (<code>src/store/store.ts</code>).</p><p><code>src/store/useIngest.ts</code> is the composition root. It owns the <code>IngestClient</code> lifecycle and wires <code>model-assembler.ts</code>&#39;s <code>createModelCallbacks</code> (worker-message → <code>AppModel</code>, unchanged from before the view migration) into store updates: <code>onProgress</code> → <code>parse</code>; <code>onDone</code> → derives <code>evidenceAvailability</code> from the final normalized model plus <code>done.skippedLines</code>, runs <code>analyzer.ts</code>, sets <code>catalog</code>, and prefetches flagged-stage task data.</p><p><code>src/App.tsx</code> routes on store <code>status</code>: idle/error → <code>DropZone</code>, parsing → a live progress readout, ready → <code>Dashboard</code>. <code>src/view/Dashboard.tsx</code> and <code>src/view/detector-registry.tsx</code> replace <code>dashboard-renderer.js</code>&#39;s widget board (see <a href="./widget-rendering.html">Widget rendering</a>).</p><h3 id="worker-web-worker-core-js" tabindex="-1">Worker (Web Worker, core JS) <a class="header-anchor" href="#worker-web-worker-core-js" aria-label="Permalink to &quot;Worker (Web Worker, core JS)&quot;">​</a></h3><p>Owns the file and <code>taskStore: Map&lt;stageId, Float64Array&gt;</code>, keyed by 8-field stride <code>[duration, gcTime, memSpilled, diskSpilled, shuffleRead, shuffleWrite, launchTime, finishTime]</code>. Four modules:</p><ul><li><code>src/stage-quantiles.ts</code>, a pure leaf, exporting <code>finalizeStage</code>, <code>computeFieldQuantiles</code>, <code>computeDurationQuantiles</code>, <code>classifySpill</code>, <code>FIELDS</code>, and <code>TASK_FIELD_NAMES</code> for the task-field packed-array layout.</li><li><code>src/event-handlers.ts</code>, holding <code>processEvent</code> and the per-event handler functions, <code>accumulateTask</code>, <code>createState</code>, <code>dispatchLine</code>, <code>buildChunkDecoder</code>, <code>emitParseCompletion</code>, <code>collectStageExecutorMetrics</code>.</li><li><code>src/shs-fetch.ts</code>, the SHS zip-fetch/decompress path: <code>runParseFromUrl</code>, <code>naturalCompare</code>, <code>reassembleRollingEntries</code>, <code>sniffCodec</code>.</li><li><code>src/parser-worker.ts</code>, a thin entrypoint (<code>streamFile</code>, <code>runParse</code>, <code>runParseFiles</code>, the <code>isWorker</code>/<code>self.onmessage</code> bus) that barrel-re-exports the other three. It is the only piece needing the File/Blob streaming API.</li></ul><p>Dropped zstd files also run a nested decompress worker, <code>src/zstd-worker.ts</code>, driven from the parse worker by <code>src/zstd-worker-client.ts</code> (see <a href="./worker-protocol.html#decompress-worker">Decompress worker</a>).</p><h2 id="streaming" tabindex="-1">Streaming <a class="header-anchor" href="#streaming" aria-label="Permalink to &quot;Streaming&quot;">​</a></h2><p>Worker reads the <code>File</code> in 4 MB chunks via <code>file.slice(...).arrayBuffer()</code>, <code>TextDecoder({ stream: true })</code>, splits on <code>\\n</code>, JSON-parses, dispatches per event type. Quantiles (P50/P95/max) and spill classification (<code>skew | volume | unclassified</code>) are computed at <code>SparkListenerStageCompleted</code> time, before posting <code>StageAggregate</code> to main.</p><p>When <code>spark.eventLog.logStageExecutorMetrics=true</code> (default <code>false</code>), <code>SparkListenerStageExecutorMetrics</code> events populate <code>stage.executorMetrics: Map&lt;execId, {...}&gt;</code> with the 23 raw peak-memory/GC fields verbatim (camelCased), consumed by the <code>memoryUtilization</code> detector&#39;s per-executor memory bands (see <a href="./board-widgets.html">Memory Utilization</a>).</p><p>These events can arrive <em>after</em> <code>SparkListenerStageCompleted</code> for the same stage, so the per-stage <code>StageAggregate</code> message posted at completion time never carries them. Instead, <code>collectStageExecutorMetrics(state)</code> walks every stage once more just before <code>done</code>, and the worker re-posts a single <code>stageExecutorMetrics</code> message (<code>Map&lt;stageId, Map&lt;execId, metrics&gt;&gt;</code>, empty when the log has no such data). Main-thread consumers get every stage&#39;s metrics without a per-stage race.</p><p>Compressed logs are inflated inline. <code>sniffCodec</code> reads the leading magic bytes: gzip (<code>1f 8b</code>), Zstandard (<code>28 b5 2f fd</code>, Spark&#39;s <code>spark.io.compression.codec=zstd</code>), Spark&#39;s custom <code>LZ4Block</code> framing, or Spark&#39;s Snappy framing (<code>org.xerial.snappy</code>&#39;s <code>\\x82SNAPPY\\0</code> header, <code>spark.io.compression.codec=snappy</code>), falling back to the filename suffix. Both the dropped-file path and the SHS-fetch path stream block-by-block (fflate <code>Gunzip</code> / fzstd <code>Decompress</code> / the LZ4Block decoder / the Snappy block decoder) to keep one decompressed chunk live at a time. In the browser, a dropped zstd file decompresses in a second worker while the parse worker parses earlier output, with a three-slice window bounding what is queued between them (see <a href="./worker-protocol.html#decompress-worker">Decompress worker</a>). The Node CLI/MCP paths (<code>collectRun</code> for local files, <code>shs-load.ts</code> for SHS archives) swap fzstd for Node&#39;s native zlib zstd where the running Node has it (22.15+/23.8+), through the <code>zstdDecoder</code> option of <code>runParse</code>/<code>decodeShsArchive</code>: <code>packages/core/src/cli/native-zstd.ts</code> decompresses one frame at a time, since Node&#39;s own decoders stop after a stream&#39;s first frame and Spark writes thousands of small ones. A frame declaring more than 64 MiB of content, one still incomplete after 64 MiB compressed, a malformed one, or a truncated tail goes to fzstd instead. Spark&#39;s frames declare no content size, so each one&#39;s output is capped at 64 MiB as it decodes: past that, the inline decoder hands the frame to fzstd and the threaded one streams it, pausing while 64 MiB wait to be parsed. On the real logs this made parsing 42% faster. For local files (<code>nodeParseCodecs</code>, <code>createThreadedZstdDecoder</code>), frames of 64 KB or more compressed decompress on libuv&#39;s threadpool, up to 4 at a time, while the main thread parses earlier output; <code>streamFile</code> awaits each <code>push</code>, and chunks still arrive in stream order. That took another 24% off the largest real log (4.2s to 3.2s) for 139 MB more peak RSS. <code>decodeShsArchive</code> decodes each archive entry in one synchronous call, so SHS archives keep the inline decoder (<code>nodeArchiveCodecs</code>).</p><p>The SHS-fetch path does buffer the downloaded zip whole for <code>unzipSync</code>. But one-shotting the decompression of a single entry (fzstd&#39;s <code>decompress()</code>, fflate&#39;s <code>gunzipSync()</code>) would allocate that entry&#39;s full declared content size as one ArrayBuffer, which fails outright for a multi-GB event log. So <code>decodeEntry(name, raw, onChunk)</code> picks the codec&#39;s streaming decoder per entry and invokes <code>onChunk</code> per decompressed piece. Decompressors are vendored under <code>src/vendor/</code> (<code>fflate.js</code>, <code>fzstd.js</code>) plus <code>src/lz4-block.ts</code> and <code>src/snappy-block.ts</code>.</p><p>Rolling <code>eventlog_v2_*</code> directories (Spark&#39;s multi-file event-log format, <code>events_&lt;index&gt;_&lt;appId&gt;(.codec)</code> files plus a zero-byte <code>appstatus_*</code> completion marker and, periodically, one <code>*.compact</code> merge file) are reassembled into parse order by <code>reassembleRollingEntries(names)</code>: drop the marker, drop every non-compact <code>events_*</code> file at or below the most recent <code>.compact</code> file&#39;s index, sort the rest numerically. Both the SHS-zip fetch path (<code>runParseFromUrl</code>) and the local folder-drop path (<code>runParseFiles</code>) call this one function.</p><p><code>runParseFiles</code> streams each file in the resulting order through the same chunked codec dispatch as <code>runParse</code>, with a fresh decompressor per file, since codecs never span a roll boundary. The NDJSON line-decoder is the exception: it stays alive across all files, because nothing guarantees a roll boundary lands on a line boundary.</p>',20)])])}const u=o(a,[["render",c]]);export{m as __pageData,u as default};

package/public/docs/assets/contributor-guide_architecture_overview.md.BXQdgG5u.lean.js

@@ -0,0 +1 @@
+import{_ as o,o as r,c as t,a5 as d}from"./chunks/framework.DSg0KOwT.js";const m=JSON.parse('{"title":"Architecture overview","description":"","frontmatter":{},"headers":[],"relativePath":"contributor-guide/architecture/overview.md","filePath":"contributor-guide/architecture/overview.md"}'),a={name:"contributor-guide/architecture/overview.md"};function c(s,e,i,n,l,h){return r(),t("div",null,[...e[0]||(e[0]=[d("",20)])])}const u=o(a,[["render",c]]);export{m as __pageData,u as default};