sparkforensics-server 0.2.2 → 0.2.4

package/public/docs/assets/{tuning-reference_memory-model.md.DhT-n4y3.js → tuning-reference_memory-model.md.CWhvgevP.js}

@@ -1 +1 @@
-import{_ as o,o as a,c as r,a5 as t}from"./chunks/framework.DSg0KOwT.js";const s="/docs/assets/memory-regions.XHvO7jHG.svg",n="/docs/assets/memory-regions.dark.D4TP9_08.svg",f="/docs/assets/memory-borrowing.BqQRJg0u.svg",i="/docs/assets/memory-borrowing.dark.Yhh20O9C.svg",y=JSON.parse('{"title":"Memory Management","description":"","frontmatter":{"title":"Memory Management"},"headers":[],"relativePath":"tuning-reference/memory-model.md","filePath":"tuning-reference/memory-model.md"}'),c={name:"tuning-reference/memory-model.md"};function l(h,e,d,p,u,m){return a(),r("div",null,[...e[0]||(e[0]=[t('<h1 id="memory-model" tabindex="-1">Memory Management <a class="header-anchor" href="#memory-model" aria-label="Permalink to &quot;Memory Management {#memory-model}&quot;">​</a></h1><h2 id="the-unified-memory-model" tabindex="-1">The unified memory model <a class="header-anchor" href="#the-unified-memory-model" aria-label="Permalink to &quot;The unified memory model&quot;">​</a></h2><p>Since Spark 1.6, the <code>UnifiedMemoryManager</code> has replaced the earlier static split between execution and storage memory with a single shared region, referred to as M<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup><sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. M is carved out of the JVM heap after first setting aside 300 MiB of reserved system memory: <code>M = (JVM heap − 300 MiB) × spark.memory.fraction</code>, with <code>spark.memory.fraction</code> defaulting to 0.6<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>. Within M, <code>spark.memory.storageFraction</code> (default 0.5) marks off a subregion reserved for cached blocks that are immune to eviction; with defaults, this works out to roughly a 30%/30% on-heap split between execution and storage out of total executor memory<sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup>.</p><img class="light-only" src="'+s+'" alt="The executor memory layout showing reserved memory, the unified region split into execution and storage, and user memory inside the JVM heap, with the off-heap pool and memory overhead outside it."><img class="dark-only" src="'+n+'" alt="The executor memory layout showing reserved memory, the unified region split into execution and storage, and user memory inside the JVM heap, with the off-heap pool and memory overhead outside it."><p>Execution and storage borrow from each other dynamically rather than sitting behind a hard wall: when execution memory is unused, storage can acquire all of it, and vice versa<sup class="footnote-ref"><a href="#fn2" id="fnref2:2">[2:2]</a></sup><sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>. The two are not equal peers, though. Execution always has priority, taking memory immediately and evicting cached storage blocks if necessary<sup class="footnote-ref"><a href="#fn3" id="fnref3:2">[3:2]</a></sup>.</p><p>The reserved 300 MB is a flat constant (<code>RESERVED_SYSTEM_MEMORY_BYTES</code> in the Spark source), not a formula, and it&#39;s hardcoded: there is no supported production configuration to change it. The only override is the internal <code>spark.testing.reservedMemory</code> property, which exists for Spark&#39;s own test suite rather than production tuning<sup class="footnote-ref"><a href="#fn3" id="fnref3:3">[3:3]</a></sup>. <code>spark.memory.fraction</code> is applied against heap minus that reserved 300 MB, not against total heap. The tuning guide is explicit that it &quot;expresses the size of M as a fraction of the (JVM heap space − 300MiB)&quot;<sup class="footnote-ref"><a href="#fn2" id="fnref2:3">[2:3]</a></sup>.</p><p>Off-heap memory extends the same model outside the JVM heap. When <code>spark.memory.offHeap.enabled=true</code>, Spark allocates raw off-heap buffers via <code>sun.misc.Unsafe</code> (routed through <code>jdk.internal.misc.Unsafe</code> on JDK 17+, which is why some builds need <code>--add-opens</code> flags), and that off-heap region is split into its own execution and storage pools, following the same borrowing rules as on-heap memory: execution has priority, storage gets evicted first<sup class="footnote-ref"><a href="#fn3" id="fnref3:4">[3:4]</a></sup>. Enabling it requires <code>spark.memory.offHeap.size</code> to be set to a positive value; leaving it enabled with the default size of <code>0</code> is a documented-invalid combination, not a supported way to run with off-heap &quot;on but empty&quot;<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>. The setting also &quot;has no impact on heap memory usage&quot;. Turning on off-heap memory does not shrink the JVM <code>-Xmx</code> for you, so the on-heap size has to be reduced manually to keep total footprint constant<sup class="footnote-ref"><a href="#fn4" id="fnref4:1">[4:1]</a></sup>. This off-heap execution memory underlies Project Tungsten: its compact binary row encoding, its explicit-memory-managed hash map for aggregations, and its cache-aware sort/join algorithms are all built to run against off-heap, GC-invisible memory<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup><sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup>.</p><p>Executor memory overhead is a separate pool layered on top of M. <code>spark.executor.memoryOverhead</code> defaults to <code>executorMemory × spark.executor.memoryOverheadFactor</code>, floored at <code>spark.executor.minMemoryOverhead</code> (384 MiB)<sup class="footnote-ref"><a href="#fn4" id="fnref4:2">[4:2]</a></sup>. <code>spark.executor.memoryOverheadFactor</code> itself defaults to 0.10 for ordinary JVM executors, but Spark bumps that default to 0.40 specifically for Kubernetes non-JVM jobs, since those workloads tend to need more non-JVM heap space<sup class="footnote-ref"><a href="#fn4" id="fnref4:3">[4:3]</a></sup>. The legacy, YARN-only <code>spark.yarn.executor.memoryOverhead</code> property was removed in Spark 3.0 in favor of the cluster-manager-agnostic <code>spark.executor.memoryOverhead</code><sup class="footnote-ref"><a href="#fn3" id="fnref3:5">[3:5]</a></sup>.</p><blockquote><p><strong>PySpark:</strong> <code>spark.python.worker.memory</code> (default <code>512m</code>) is a soft spill threshold for a Python worker&#39;s own aggregation buffering, not a JVM-enforced cap. A PySpark worker runs as a separate OS process outside the JVM heap, so the JVM cannot police it directly<sup class="footnote-ref"><a href="#fn3" id="fnref3:6">[3:6]</a></sup>. The only setting that actually tries to bound PySpark memory per executor is <code>spark.executor.pyspark.memory</code>, and even that depends on Python&#39;s <code>resource</code> module, which isn&#39;t supported on Windows and doesn&#39;t actually limit anything on macOS<sup class="footnote-ref"><a href="#fn4" id="fnref4:4">[4:4]</a></sup>. Left unset, PySpark memory usage is folded into <code>spark.executor.memoryOverhead</code> by default<sup class="footnote-ref"><a href="#fn4" id="fnref4:5">[4:5]</a></sup>.</p></blockquote><h2 id="watching-eviction-and-spill" tabindex="-1">Watching eviction and spill <a class="header-anchor" href="#watching-eviction-and-spill" aria-label="Permalink to &quot;Watching eviction and spill&quot;">​</a></h2><p>That borrowing leaves a trace once eviction actually happens. Storage eviction under memory pressure follows an LRU (least recently used) policy<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup><sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup>. The granularity is the block: the <code>BlockManager</code> tracks cached data as per-partition blocks (one block per RDD/DataFrame partition) and LRU operates at that level rather than evicting an entire RDD or cached table in one shot<sup class="footnote-ref"><a href="#fn8" id="fnref8">[8]</a></sup><sup class="footnote-ref"><a href="#fn9" id="fnref9">[9]</a></sup>. In practice this tends to evict the oldest partitions first, those materialized in the earliest job or stage, though lazy evaluation makes it hard to predict exactly which partitions will go ahead of time<sup class="footnote-ref"><a href="#fn9" id="fnref9:1">[9:1]</a></sup>.</p><p>What happens to an evicted block is visible in its downstream effect and depends on its <code>StorageLevel</code>. For <code>MEMORY_ONLY</code>, the block is simply dropped and recomputed from the RDD&#39;s lineage the next time it&#39;s needed. For a disk-backed level like <code>MEMORY_AND_DISK</code>, the evicted block is written to disk and read back from there instead of being recomputed<sup class="footnote-ref"><a href="#fn9" id="fnref9:2">[9:2]</a></sup><sup class="footnote-ref"><a href="#fn7" id="fnref7:1">[7:1]</a></sup>. Blocks can also be evicted well before you ever try to reuse them: <code>.cache()</code> does not guarantee a block stays resident, especially on busy clusters or long pipelines<sup class="footnote-ref"><a href="#fn7" id="fnref7:2">[7:2]</a></sup>.</p><p>On the execution side, each task gets its own <code>TaskMemoryManager</code>, which enforces a soft per-task cap: with <code>n</code> tasks running concurrently, each task is allowed to allocate somewhere between <code>1/(2n)</code> and <code>1/n</code> of total execution memory, and the first task to arrive on an idle executor typically grabs more than its later-arriving neighbors<sup class="footnote-ref"><a href="#fn3" id="fnref3:7">[3:7]</a></sup>.</p><p>When a sort or hash-aggregation operator keeps requesting execution-memory pages and can&#39;t get more, Spark doesn&#39;t fail immediately. It blocks the requesting task, spills that task&#39;s in-memory data structure to disk, or, in the worst case, throws an <code>OutOfMemoryError</code><sup class="footnote-ref"><a href="#fn3" id="fnref3:8">[3:8]</a></sup>. The spill path runs through Spark&#39;s map/shuffle I/O machinery: shuffle partitions created by wide transformations like <code>groupBy()</code> or <code>join()</code> <a href="./bottleneck-spill.html">spill</a> to the executors&#39; local disks at the location set by <code>spark.local.directory</code><sup class="footnote-ref"><a href="#fn6" id="fnref6:1">[6:1]</a></sup>. SQL physical operators apply the same idea with their own row-count guardrails: <code>sortMergeJoinExec</code>&#39;s in-memory buffer and the cartesian-product operator&#39;s buffer both spill once they cross a configured row threshold, which by default is set to the value of <code>spark.shuffle.spill.numElementsForceSpillThreshold</code><sup class="footnote-ref"><a href="#fn10" id="fnref10">[10]</a></sup>.</p><p>Whatever the trigger, the resulting spilled bytes are exposed on the task metrics as <code>memoryBytesSpilled</code>, visible in the Spark UI and event log. This is the concrete signal to watch for execution memory pressure<sup class="footnote-ref"><a href="#fn11" id="fnref11">[11]</a></sup>.</p><p>Memory-overhead misconfiguration shows up differently: it manifests as the container or pod being OOMKilled with a vague container-memory error rather than a Spark-level exception, since the enforcement happens at the YARN NodeManager or Kubernetes kubelet/cgroup layer, not inside the JVM<sup class="footnote-ref"><a href="#fn3" id="fnref3:9">[3:9]</a></sup>. Under the old default overhead factor of 0.10, non-JVM Kubernetes jobs commonly failed with &quot;Memory Overhead Exceeded&quot; errors, which is exactly why Spark bumped the Kubernetes non-JVM default to 0.40<sup class="footnote-ref"><a href="#fn4" id="fnref4:6">[4:6]</a></sup>.</p><h2 id="what-the-borrowing-costs-you" tabindex="-1">What the borrowing costs you <a class="header-anchor" href="#what-the-borrowing-costs-you" aria-label="Permalink to &quot;What the borrowing costs you&quot;">​</a></h2><p>None of this is free for whoever&#39;s counting on cached data staying put. Because execution always wins the tug-of-war over storage, <a href="./caching.html">caching</a> is never a guarantee; it&#39;s a best effort. As luminousmen-memory-management puts it: &quot;If Execution needs memory, it takes it. If Storage is using that space (cached RDDs, broadcasts), Spark starts evicting blocks. If Execution is idle, Storage can grow into that space, until Execution comes back.&quot;<sup class="footnote-ref"><a href="#fn3" id="fnref3:10">[3:10]</a></sup> That growth-then-shrink-back behavior is dynamic borrowing, not storage evicting execution; eviction is strictly one-directional. As spark-notes-task-memory-management summarizes the agreement between the two regions: &quot;keep acquiring execution memory and evict storage as you need more execution memory&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup>, never the reverse. The same asymmetric rule carries over when off-heap memory is enabled: execution still has priority and storage still gets evicted first<sup class="footnote-ref"><a href="#fn3" id="fnref3:11">[3:11]</a></sup>. Practically, this means a cached DataFrame can silently lose blocks under memory pressure, forcing an expensive lineage recompute (for <code>MEMORY_ONLY</code>) or an extra disk round-trip (for <code>MEMORY_AND_DISK</code>) the next time it&#39;s touched.</p><img class="light-only" src="'+f+'" alt="How storage memory borrows idle execution space while execution reclaims its own space by evicting storage in one direction."><img class="dark-only" src="'+i+'" alt="How storage memory borrows idle execution space while execution reclaims its own space by evicting storage in one direction."><p>The <code>TaskMemoryManager</code>&#39;s soft per-task cap explains why spill behavior is workload-shape-dependent rather than a fixed threshold: with more tasks packed onto an executor, each one&#39;s guaranteed share of execution memory shrinks toward <code>1/(2n)</code>, making spills more likely under high task concurrency even when total execution memory hasn&#39;t changed<sup class="footnote-ref"><a href="#fn3" id="fnref3:12">[3:12]</a></sup>.</p><p>Off-heap memory&#39;s payoff is specifically about <a href="./bottleneck-gc.html">garbage collection</a>: because off-heap buffers sit outside the JVM heap, they are invisible to the garbage collector, so fewer and smaller live objects need to be tracked, scanned, and copied, which reduces both the frequency and duration of GC pauses<sup class="footnote-ref"><a href="#fn3" id="fnref3:13">[3:13]</a></sup>. Project Tungsten&#39;s off-heap hash map for aggregations was benchmarked at over 1 million operations per second in a single thread, with &quot;almost no performance degradation as memory utilization increases,&quot; unlike the JVM default <code>java.util.HashMap</code>, which eventually thrashes on GC<sup class="footnote-ref"><a href="#fn5" id="fnref5:1">[5:1]</a></sup><sup class="footnote-ref"><a href="#fn6" id="fnref6:2">[6:2]</a></sup>. The tradeoff is that off-heap memory removes the GC safety net along with the GC overhead: there&#39;s no garbage collector cleaning up if something goes wrong<sup class="footnote-ref"><a href="#fn3" id="fnref3:14">[3:14]</a></sup>.</p><p>Memory overhead matters because off-heap memory and (if unset) PySpark memory are not automatically folded into the overhead calculation the way heap memory is. Enabling <code>spark.memory.offHeap.size</code> without also raising <code>spark.executor.memoryOverhead</code> (or explicitly setting <code>spark.executor.pyspark.memory</code>) can push the executor&#39;s real footprint past what the cluster manager granted, and the process gets killed with a vague container-memory error instead of a Spark-level exception<sup class="footnote-ref"><a href="#fn3" id="fnref3:15">[3:15]</a></sup>. Worked example: with <code>--executor-memory=8G</code>, the default 10% overhead gives <code>max(0.1 × 8192 MB, 384 MB) = 819 MB</code>, so the total memory requested from the cluster manager is <code>8192 + 819 = 9011 MB</code><sup class="footnote-ref"><a href="#fn3" id="fnref3:16">[3:16]</a></sup>, useful to keep in mind when <a href="./cluster-config.html">sizing containers or pods</a> against a cluster&#39;s available capacity.</p><h2 id="tuning-the-memory-pools" tabindex="-1">Tuning the memory pools <a class="header-anchor" href="#tuning-the-memory-pools" aria-label="Permalink to &quot;Tuning the memory pools&quot;">​</a></h2><p>Most of this is adjustable, within limits. Tune <code>spark.memory.fraction</code> and <code>spark.memory.storageFraction</code> only if your workload&#39;s execution/storage balance genuinely needs to shift away from the ~30/30 default. But don&#39;t try to reclaim the 300 MB reserved region; it&#39;s a fixed, non-tunable constant in production<sup class="footnote-ref"><a href="#fn3" id="fnref3:17">[3:17]</a></sup>.</p><p>If losing cached partitions to eviction is costly, prefer a disk-backed <code>StorageLevel</code> such as <code>MEMORY_AND_DISK</code> over <code>MEMORY_ONLY</code>: an evicted block gets written to disk and read back rather than triggering a full lineage recompute<sup class="footnote-ref"><a href="#fn9" id="fnref9:3">[9:3]</a></sup><sup class="footnote-ref"><a href="#fn7" id="fnref7:3">[7:3]</a></sup>. Keep in mind <code>.cache()</code> alone is not a residency guarantee even with this choice<sup class="footnote-ref"><a href="#fn7" id="fnref7:4">[7:4]</a></sup>.</p><p>Watch <code>memoryBytesSpilled</code> in the Spark UI or history server to catch execution-memory pressure early<sup class="footnote-ref"><a href="#fn11" id="fnref11:1">[11:1]</a></sup>. If sort or hash-aggregation spills are heavy, consider giving the executor more memory or reducing the number of concurrently running tasks per executor, since the <code>TaskMemoryManager</code>&#39;s per-task guarantee shrinks as task concurrency <code>n</code> grows<sup class="footnote-ref"><a href="#fn3" id="fnref3:18">[3:18]</a></sup>. For SQL joins and cartesian products, <code>spark.shuffle.spill.numElementsForceSpillThreshold</code> governs when <code>sortMergeJoinExec</code> and the cartesian-product operator spill their buffers<sup class="footnote-ref"><a href="#fn10" id="fnref10:1">[10:1]</a></sup>.</p><p>Set <code>spark.executor.memoryOverhead</code> explicitly rather than relying on the default, especially on Kubernetes for non-JVM workloads (where the default factor is already bumped to 0.40) or whenever off-heap memory or heavy PySpark usage is in play: those aren&#39;t automatically folded into the overhead calculation, so an unadjusted overhead can lead to an OOMKilled container instead of a clean Spark-level error<sup class="footnote-ref"><a href="#fn3" id="fnref3:19">[3:19]</a></sup><sup class="footnote-ref"><a href="#fn4" id="fnref4:7">[4:7]</a></sup>. Use <code>spark.executor.memoryOverhead</code>, not the removed <code>spark.yarn.executor.memoryOverhead</code><sup class="footnote-ref"><a href="#fn3" id="fnref3:20">[3:20]</a></sup>. For capacity planning, remember the total requested from the cluster manager is executor memory plus overhead, e.g., <code>8192 + 819 = 9011 MB</code> for an 8 GB executor at the default 10% factor<sup class="footnote-ref"><a href="#fn3" id="fnref3:21">[3:21]</a></sup>.</p><p>When enabling off-heap memory, always pair <code>spark.memory.offHeap.enabled=true</code> with an explicit positive <code>spark.memory.offHeap.size</code> (never leave it enabled at the default size of <code>0</code><sup class="footnote-ref"><a href="#fn4" id="fnref4:8">[4:8]</a></sup>) and manually reduce the JVM <code>-Xmx</code> to compensate, since enabling off-heap memory does not shrink the heap for you<sup class="footnote-ref"><a href="#fn4" id="fnref4:9">[4:9]</a></sup>.</p><blockquote><p><strong>PySpark:</strong> if you need PySpark&#39;s own memory bounded rather than folded silently into the overhead, set <code>spark.executor.pyspark.memory</code> explicitly, though its enforcement relies on Python&#39;s <code>resource</code> module and won&#39;t work on Windows and won&#39;t actually limit anything on macOS<sup class="footnote-ref"><a href="#fn4" id="fnref4:10">[4:10]</a></sup>.</p></blockquote><h2 id="sources" tabindex="-1">Sources <a class="header-anchor" href="#sources" aria-label="Permalink to &quot;Sources&quot;">​</a></h2><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://raw.githubusercontent.com/spoddutur/spark-notes/master/task_memory_management_in_spark.md" target="_blank" rel="noreferrer">Task Memory Management in Spark</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/tuning.html" target="_blank" rel="noreferrer">Tuning Spark</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a> <a href="#fnref2:2" class="footnote-backref">↩︎</a> <a href="#fnref2:3" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://luminousmen.com/post/dive-into-spark-memory" target="_blank" rel="noreferrer">Dive into Spark Memory</a> <a href="#fnref3" class="footnote-backref">↩︎</a> <a href="#fnref3:1" class="footnote-backref">↩︎</a> <a href="#fnref3:2" class="footnote-backref">↩︎</a> <a href="#fnref3:3" class="footnote-backref">↩︎</a> <a href="#fnref3:4" class="footnote-backref">↩︎</a> <a href="#fnref3:5" class="footnote-backref">↩︎</a> <a href="#fnref3:6" class="footnote-backref">↩︎</a> <a href="#fnref3:7" class="footnote-backref">↩︎</a> <a href="#fnref3:8" class="footnote-backref">↩︎</a> <a href="#fnref3:9" class="footnote-backref">↩︎</a> <a href="#fnref3:10" class="footnote-backref">↩︎</a> <a href="#fnref3:11" class="footnote-backref">↩︎</a> <a href="#fnref3:12" class="footnote-backref">↩︎</a> <a href="#fnref3:13" class="footnote-backref">↩︎</a> <a href="#fnref3:14" class="footnote-backref">↩︎</a> <a href="#fnref3:15" class="footnote-backref">↩︎</a> <a href="#fnref3:16" class="footnote-backref">↩︎</a> <a href="#fnref3:17" class="footnote-backref">↩︎</a> <a href="#fnref3:18" class="footnote-backref">↩︎</a> <a href="#fnref3:19" class="footnote-backref">↩︎</a> <a href="#fnref3:20" class="footnote-backref">↩︎</a> <a href="#fnref3:21" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration — Spark</a> <a href="#fnref4" class="footnote-backref">↩︎</a> <a href="#fnref4:1" class="footnote-backref">↩︎</a> <a href="#fnref4:2" class="footnote-backref">↩︎</a> <a href="#fnref4:3" class="footnote-backref">↩︎</a> <a href="#fnref4:4" class="footnote-backref">↩︎</a> <a href="#fnref4:5" class="footnote-backref">↩︎</a> <a href="#fnref4:6" class="footnote-backref">↩︎</a> <a href="#fnref4:7" class="footnote-backref">↩︎</a> <a href="#fnref4:8" class="footnote-backref">↩︎</a> <a href="#fnref4:9" class="footnote-backref">↩︎</a> <a href="#fnref4:10" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://www.databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html" target="_blank" rel="noreferrer">Project Tungsten: Bringing Spark Closer to Bare Metal</a> <a href="#fnref5" class="footnote-backref">↩︎</a> <a href="#fnref5:1" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><em>Learning Spark, 2nd Edition</em>, Damji, Wenig, Das, Lee, ch. 6–7 <a href="#fnref6" class="footnote-backref">↩︎</a> <a href="#fnref6:1" class="footnote-backref">↩︎</a> <a href="#fnref6:2" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://luminousmen.com/post/explaining-the-mechanics-of-spark-caching" target="_blank" rel="noreferrer">Explaining the Mechanics of Spark Caching</a> <a href="#fnref7" class="footnote-backref">↩︎</a> <a href="#fnref7:1" class="footnote-backref">↩︎</a> <a href="#fnref7:2" class="footnote-backref">↩︎</a> <a href="#fnref7:3" class="footnote-backref">↩︎</a> <a href="#fnref7:4" class="footnote-backref">↩︎</a></p></li><li id="fn8" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/rdd-programming-guide.html" target="_blank" rel="noreferrer">RDD Programming Guide</a> <a href="#fnref8" class="footnote-backref">↩︎</a></p></li><li id="fn9" class="footnote-item"><p><em>High Performance Spark, 2nd Edition</em>, Karau, Polak &amp; Warren, ch. 7 <a href="#fnref9" class="footnote-backref">↩︎</a> <a href="#fnref9:1" class="footnote-backref">↩︎</a> <a href="#fnref9:2" class="footnote-backref">↩︎</a> <a href="#fnref9:3" class="footnote-backref">↩︎</a></p></li><li id="fn10" class="footnote-item"><p><a href="https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala" target="_blank" rel="noreferrer">SQLConf.scala</a> <a href="#fnref10" class="footnote-backref">↩︎</a> <a href="#fnref10:1" class="footnote-backref">↩︎</a></p></li><li id="fn11" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/monitoring.html#spark-history-server" target="_blank" rel="noreferrer">Monitoring and Instrumentation</a> <a href="#fnref11" class="footnote-backref">↩︎</a> <a href="#fnref11:1" class="footnote-backref">↩︎</a></p></li></ol></section>',34)])])}const k=o(c,[["render",l]]);export{y as __pageData,k as default};
+import{_ as o,o as a,c as t,a5 as r}from"./chunks/framework.DSg0KOwT.js";const s="/docs/assets/memory-regions.XHvO7jHG.svg",n="/docs/assets/memory-regions.dark.D4TP9_08.svg",i="/docs/assets/memory-borrowing.BqQRJg0u.svg",f="/docs/assets/memory-borrowing.dark.Yhh20O9C.svg",y=JSON.parse('{"title":"Memory Management","description":"","frontmatter":{"title":"Memory Management"},"headers":[],"relativePath":"tuning-reference/memory-model.md","filePath":"tuning-reference/memory-model.md"}'),c={name:"tuning-reference/memory-model.md"};function l(h,e,d,p,u,m){return a(),t("div",null,[...e[0]||(e[0]=[r('<h1 id="memory-model" tabindex="-1">Memory Management <a class="header-anchor" href="#memory-model" aria-label="Permalink to &quot;Memory Management {#memory-model}&quot;">​</a></h1><h2 id="the-unified-memory-model" tabindex="-1">The unified memory model <a class="header-anchor" href="#the-unified-memory-model" aria-label="Permalink to &quot;The unified memory model&quot;">​</a></h2><p>Since Spark 1.6, the <code>UnifiedMemoryManager</code> has replaced the earlier static split between execution and storage memory with a single shared region, referred to as M<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup><sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. M is carved out of the JVM heap after first setting aside 300 MiB of reserved system memory: <code>M = (JVM heap − 300 MiB) × spark.memory.fraction</code>, with <code>spark.memory.fraction</code> defaulting to 0.6<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>. Within M, <code>spark.memory.storageFraction</code> (default 0.5) marks off a subregion reserved for cached blocks that are immune to eviction; with defaults, this works out to roughly a 30%/30% on-heap split between execution and storage out of total executor memory<sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup>.</p><img class="light-only" src="'+s+'" alt="The executor memory layout showing reserved memory, the unified region split into execution and storage, and user memory inside the JVM heap, with the off-heap pool and memory overhead outside it."><img class="dark-only" src="'+n+'" alt="The executor memory layout showing reserved memory, the unified region split into execution and storage, and user memory inside the JVM heap, with the off-heap pool and memory overhead outside it."><p>Execution and storage borrow from each other dynamically rather than sitting behind a hard wall: when execution memory is unused, storage can acquire all of it, and vice versa<sup class="footnote-ref"><a href="#fn2" id="fnref2:2">[2:2]</a></sup><sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>. The two are not equal peers, though. Execution always has priority, taking memory immediately and evicting cached storage blocks if necessary<sup class="footnote-ref"><a href="#fn3" id="fnref3:2">[3:2]</a></sup>.</p><p>The reserved 300 MB is a flat constant (<code>RESERVED_SYSTEM_MEMORY_BYTES</code> in the Spark source), not a formula, and it&#39;s hardcoded: there is no supported production configuration to change it. The only override is the internal <code>spark.testing.reservedMemory</code> property, which exists for Spark&#39;s own test suite rather than production tuning<sup class="footnote-ref"><a href="#fn3" id="fnref3:3">[3:3]</a></sup>. <code>spark.memory.fraction</code> is applied against heap minus that reserved 300 MB, not against total heap. The tuning guide is explicit that it &quot;expresses the size of M as a fraction of the (JVM heap space − 300MiB)&quot;<sup class="footnote-ref"><a href="#fn2" id="fnref2:3">[2:3]</a></sup>.</p><p>Off-heap memory extends the same model outside the JVM heap. When <code>spark.memory.offHeap.enabled=true</code>, Spark allocates raw off-heap buffers via <code>sun.misc.Unsafe</code> (routed through <code>jdk.internal.misc.Unsafe</code> on JDK 17+, which is why some builds need <code>--add-opens</code> flags), and that off-heap region is split into its own execution and storage pools, following the same borrowing rules as on-heap memory: execution has priority, storage gets evicted first<sup class="footnote-ref"><a href="#fn3" id="fnref3:4">[3:4]</a></sup>. Enabling it requires <code>spark.memory.offHeap.size</code> to be set to a positive value; leaving it enabled with the default size of <code>0</code> is a documented-invalid combination, not a supported way to run with off-heap &quot;on but empty&quot;<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>. The setting also &quot;has no impact on heap memory usage&quot;. Turning on off-heap memory does not shrink the JVM <code>-Xmx</code> for you, so the on-heap size has to be reduced manually to keep total footprint constant<sup class="footnote-ref"><a href="#fn4" id="fnref4:1">[4:1]</a></sup>. This off-heap execution memory underlies Project Tungsten: its compact binary row encoding, its explicit-memory-managed hash map for aggregations, and its cache-aware sort/join algorithms are all built to run against off-heap, GC-invisible memory<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup><sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup>.</p><p>Executor memory overhead is a separate pool layered on top of M. <code>spark.executor.memoryOverhead</code> defaults to <code>executorMemory × spark.executor.memoryOverheadFactor</code>, floored at <code>spark.executor.minMemoryOverhead</code> (384 MiB)<sup class="footnote-ref"><a href="#fn4" id="fnref4:2">[4:2]</a></sup>. <code>spark.executor.memoryOverheadFactor</code> itself defaults to 0.10 for ordinary JVM executors, but Spark bumps that default to 0.40 specifically for Kubernetes non-JVM jobs, since those workloads tend to need more non-JVM heap space<sup class="footnote-ref"><a href="#fn4" id="fnref4:3">[4:3]</a></sup>. The legacy, YARN-only <code>spark.yarn.executor.memoryOverhead</code> property was removed in Spark 3.0 in favor of the cluster-manager-agnostic <code>spark.executor.memoryOverhead</code><sup class="footnote-ref"><a href="#fn3" id="fnref3:5">[3:5]</a></sup>.</p><blockquote><p><strong>PySpark:</strong> <code>spark.python.worker.memory</code> (default <code>512m</code>) is a soft spill threshold for a Python worker&#39;s own aggregation buffering, not a JVM-enforced cap. A PySpark worker runs as a separate OS process outside the JVM heap, so the JVM cannot police it directly<sup class="footnote-ref"><a href="#fn3" id="fnref3:6">[3:6]</a></sup>. The only setting that actually tries to bound PySpark memory per executor is <code>spark.executor.pyspark.memory</code>, and even that depends on Python&#39;s <code>resource</code> module, which isn&#39;t supported on Windows and doesn&#39;t actually limit anything on macOS<sup class="footnote-ref"><a href="#fn4" id="fnref4:4">[4:4]</a></sup>. Left unset, PySpark memory usage is folded into <code>spark.executor.memoryOverhead</code> by default<sup class="footnote-ref"><a href="#fn4" id="fnref4:5">[4:5]</a></sup>.</p></blockquote><h2 id="watching-eviction-and-spill" tabindex="-1">Watching eviction and spill <a class="header-anchor" href="#watching-eviction-and-spill" aria-label="Permalink to &quot;Watching eviction and spill&quot;">​</a></h2><p>That borrowing leaves a trace once eviction actually happens. Storage eviction under memory pressure follows an LRU (least recently used) policy<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup><sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup>. The granularity is the block: the <code>BlockManager</code> tracks cached data as per-partition blocks (one block per RDD/DataFrame partition) and LRU operates at that level rather than evicting an entire RDD or cached table in one shot<sup class="footnote-ref"><a href="#fn8" id="fnref8">[8]</a></sup><sup class="footnote-ref"><a href="#fn9" id="fnref9">[9]</a></sup>. In practice this tends to evict the oldest partitions first, those materialized in the earliest job or stage, though lazy evaluation makes it hard to predict exactly which partitions will go ahead of time<sup class="footnote-ref"><a href="#fn9" id="fnref9:1">[9:1]</a></sup>.</p><p>What happens to an evicted block is visible in its downstream effect and depends on its <code>StorageLevel</code>. For <code>MEMORY_ONLY</code>, the block is simply dropped and recomputed from the RDD&#39;s lineage the next time it&#39;s needed. For a disk-backed level like <code>MEMORY_AND_DISK</code>, the evicted block is written to disk and read back from there instead of being recomputed<sup class="footnote-ref"><a href="#fn9" id="fnref9:2">[9:2]</a></sup><sup class="footnote-ref"><a href="#fn7" id="fnref7:1">[7:1]</a></sup>. Blocks can also be evicted well before you ever try to reuse them: <code>.cache()</code> does not guarantee a block stays resident, especially on busy clusters or long pipelines<sup class="footnote-ref"><a href="#fn7" id="fnref7:2">[7:2]</a></sup>.</p><p>On the execution side, each task gets its own <code>TaskMemoryManager</code>, which enforces a soft per-task cap: with <code>n</code> tasks running concurrently, each task is allowed to allocate somewhere between <code>1/(2n)</code> and <code>1/n</code> of total execution memory, and the first task to arrive on an idle executor typically grabs more than its later-arriving neighbors<sup class="footnote-ref"><a href="#fn3" id="fnref3:7">[3:7]</a></sup>.</p><p>When a sort or hash-aggregation operator keeps requesting execution-memory pages and can&#39;t get more, Spark doesn&#39;t fail immediately. It blocks the requesting task, spills that task&#39;s in-memory data structure to disk, or, in the worst case, throws an <code>OutOfMemoryError</code><sup class="footnote-ref"><a href="#fn3" id="fnref3:8">[3:8]</a></sup>. The spill path runs through Spark&#39;s map/shuffle I/O machinery: shuffle partitions created by wide transformations like <code>groupBy()</code> or <code>join()</code> <a href="./bottleneck-spill.html">spill</a> to the executors&#39; local disks at the location set by <code>spark.local.directory</code><sup class="footnote-ref"><a href="#fn6" id="fnref6:1">[6:1]</a></sup>. SQL physical operators apply the same idea with their own row-count guardrails: <code>sortMergeJoinExec</code>&#39;s in-memory buffer and the cartesian-product operator&#39;s buffer both spill once they cross a configured row threshold, which by default is set to the value of <code>spark.shuffle.spill.numElementsForceSpillThreshold</code><sup class="footnote-ref"><a href="#fn10" id="fnref10">[10]</a></sup>.</p><p>Whatever the trigger, the resulting spilled bytes are exposed on the task metrics as <code>memoryBytesSpilled</code>, visible in the Spark UI and event log. This is the concrete signal to watch for execution memory pressure<sup class="footnote-ref"><a href="#fn11" id="fnref11">[11]</a></sup>.</p><p>Memory-overhead misconfiguration shows up differently: it manifests as the container or pod being OOMKilled with a vague container-memory error rather than a Spark-level exception, since the enforcement happens at the YARN NodeManager or Kubernetes kubelet/cgroup layer, not inside the JVM<sup class="footnote-ref"><a href="#fn3" id="fnref3:9">[3:9]</a></sup>. Under the old default overhead factor of 0.10, non-JVM Kubernetes jobs commonly failed with &quot;Memory Overhead Exceeded&quot; errors, which is exactly why Spark bumped the Kubernetes non-JVM default to 0.40<sup class="footnote-ref"><a href="#fn4" id="fnref4:6">[4:6]</a></sup>.</p><h2 id="what-the-borrowing-costs-you" tabindex="-1">What the borrowing costs you <a class="header-anchor" href="#what-the-borrowing-costs-you" aria-label="Permalink to &quot;What the borrowing costs you&quot;">​</a></h2><p>None of this is free for whoever&#39;s counting on cached data staying put. Because execution always wins the tug-of-war over storage, <a href="./caching.html">caching</a> is never a guarantee; it&#39;s a best effort. As luminousmen-memory-management puts it: &quot;If Execution needs memory, it takes it. If Storage is using that space (cached RDDs, broadcasts), Spark starts evicting blocks. If Execution is idle, Storage can grow into that space, until Execution comes back.&quot;<sup class="footnote-ref"><a href="#fn3" id="fnref3:10">[3:10]</a></sup> That growth-then-shrink-back behavior is dynamic borrowing, not storage evicting execution; eviction is strictly one-directional. As spark-notes-task-memory-management summarizes the agreement between the two regions: &quot;keep acquiring execution memory and evict storage as you need more execution memory&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup>, never the reverse. The same asymmetric rule carries over when off-heap memory is enabled: execution still has priority and storage still gets evicted first<sup class="footnote-ref"><a href="#fn3" id="fnref3:11">[3:11]</a></sup>. Practically, this means a cached DataFrame can silently lose blocks under memory pressure, forcing an expensive lineage recompute (for <code>MEMORY_ONLY</code>) or an extra disk round-trip (for <code>MEMORY_AND_DISK</code>) the next time it&#39;s touched.</p><img class="light-only" src="'+i+'" alt="How storage memory borrows idle execution space while execution reclaims its own space by evicting storage in one direction."><img class="dark-only" src="'+f+'" alt="How storage memory borrows idle execution space while execution reclaims its own space by evicting storage in one direction."><p>The <code>TaskMemoryManager</code>&#39;s soft per-task cap explains why spill behavior is workload-shape-dependent rather than a fixed threshold: with more tasks packed onto an executor, each one&#39;s guaranteed share of execution memory shrinks toward <code>1/(2n)</code>, making spills more likely under high task concurrency even when total execution memory hasn&#39;t changed<sup class="footnote-ref"><a href="#fn3" id="fnref3:12">[3:12]</a></sup>.</p><p>Off-heap memory&#39;s payoff is specifically about <a href="./bottleneck-gc.html">garbage collection</a>: because off-heap buffers sit outside the JVM heap, they are invisible to the garbage collector, so fewer and smaller live objects need to be tracked, scanned, and copied, which reduces both the frequency and duration of GC pauses<sup class="footnote-ref"><a href="#fn3" id="fnref3:13">[3:13]</a></sup>. Project Tungsten&#39;s off-heap hash map for aggregations was benchmarked at over 1 million operations per second in a single thread, with &quot;almost no performance degradation as memory utilization increases,&quot; unlike the JVM default <code>java.util.HashMap</code>, which eventually thrashes on GC<sup class="footnote-ref"><a href="#fn5" id="fnref5:1">[5:1]</a></sup><sup class="footnote-ref"><a href="#fn6" id="fnref6:2">[6:2]</a></sup>. The tradeoff is that off-heap memory removes the GC safety net along with the GC overhead: there&#39;s no garbage collector cleaning up if something goes wrong<sup class="footnote-ref"><a href="#fn3" id="fnref3:14">[3:14]</a></sup>.</p><p>Memory overhead matters because off-heap memory and (if unset) PySpark memory are not automatically folded into the overhead calculation the way heap memory is. Enabling <code>spark.memory.offHeap.size</code> without also raising <code>spark.executor.memoryOverhead</code> (or explicitly setting <code>spark.executor.pyspark.memory</code>) can push the executor&#39;s real footprint past what the cluster manager granted, and the process gets killed with a vague container-memory error instead of a Spark-level exception<sup class="footnote-ref"><a href="#fn3" id="fnref3:15">[3:15]</a></sup>. Worked example: with <code>--executor-memory=8G</code>, the default 10% overhead gives <code>max(0.1 × 8192 MB, 384 MB) = 819 MB</code>, so the total memory requested from the cluster manager is <code>8192 + 819 = 9011 MB</code><sup class="footnote-ref"><a href="#fn3" id="fnref3:16">[3:16]</a></sup>, useful to keep in mind when <a href="./cluster-config.html">sizing containers or pods</a> against a cluster&#39;s available capacity.</p><h2 id="tuning-the-memory-pools" tabindex="-1">Tuning the memory pools <a class="header-anchor" href="#tuning-the-memory-pools" aria-label="Permalink to &quot;Tuning the memory pools&quot;">​</a></h2><p>Most of this is adjustable, within limits. Tune <code>spark.memory.fraction</code> and <code>spark.memory.storageFraction</code> only if your workload&#39;s execution/storage balance genuinely needs to shift away from the ~30/30 default. But don&#39;t try to reclaim the 300 MB reserved region; it&#39;s a fixed, non-tunable constant in production<sup class="footnote-ref"><a href="#fn3" id="fnref3:17">[3:17]</a></sup>.</p><p>If losing cached partitions to eviction is costly, prefer a disk-backed <code>StorageLevel</code> such as <code>MEMORY_AND_DISK</code> over <code>MEMORY_ONLY</code>: an evicted block gets written to disk and read back rather than triggering a full lineage recompute<sup class="footnote-ref"><a href="#fn9" id="fnref9:3">[9:3]</a></sup><sup class="footnote-ref"><a href="#fn7" id="fnref7:3">[7:3]</a></sup>. Keep in mind <code>.cache()</code> alone is not a residency guarantee even with this choice<sup class="footnote-ref"><a href="#fn7" id="fnref7:4">[7:4]</a></sup>.</p><p>Watch <code>memoryBytesSpilled</code> in the Spark UI or history server to catch execution-memory pressure early<sup class="footnote-ref"><a href="#fn11" id="fnref11:1">[11:1]</a></sup>. If sort or hash-aggregation spills are heavy, consider giving the executor more memory or reducing the number of concurrently running tasks per executor, since the <code>TaskMemoryManager</code>&#39;s per-task guarantee shrinks as task concurrency <code>n</code> grows<sup class="footnote-ref"><a href="#fn3" id="fnref3:18">[3:18]</a></sup>. For SQL joins and cartesian products, <code>spark.shuffle.spill.numElementsForceSpillThreshold</code> governs when <code>sortMergeJoinExec</code> and the cartesian-product operator spill their buffers<sup class="footnote-ref"><a href="#fn10" id="fnref10:1">[10:1]</a></sup>.</p><p>Set <code>spark.executor.memoryOverhead</code> explicitly rather than relying on the default, especially on Kubernetes for non-JVM workloads (where the default factor is already bumped to 0.40) or whenever off-heap memory or heavy PySpark usage is in play: those aren&#39;t automatically folded into the overhead calculation, so an unadjusted overhead can lead to an OOMKilled container instead of a clean Spark-level error<sup class="footnote-ref"><a href="#fn3" id="fnref3:19">[3:19]</a></sup><sup class="footnote-ref"><a href="#fn4" id="fnref4:7">[4:7]</a></sup>. Use <code>spark.executor.memoryOverhead</code>, not the removed <code>spark.yarn.executor.memoryOverhead</code><sup class="footnote-ref"><a href="#fn3" id="fnref3:20">[3:20]</a></sup>. For capacity planning, remember the total requested from the cluster manager is executor memory plus overhead, e.g., <code>8192 + 819 = 9011 MB</code> for an 8 GB executor at the default 10% factor<sup class="footnote-ref"><a href="#fn3" id="fnref3:21">[3:21]</a></sup>.</p><p>When enabling off-heap memory, always pair <code>spark.memory.offHeap.enabled=true</code> with an explicit positive <code>spark.memory.offHeap.size</code> (never leave it enabled at the default size of <code>0</code><sup class="footnote-ref"><a href="#fn4" id="fnref4:8">[4:8]</a></sup>) and manually reduce the JVM <code>-Xmx</code> to compensate, since enabling off-heap memory does not shrink the heap for you<sup class="footnote-ref"><a href="#fn4" id="fnref4:9">[4:9]</a></sup>.</p><blockquote><p><strong>PySpark:</strong> if you need PySpark&#39;s own memory bounded rather than folded silently into the overhead, set <code>spark.executor.pyspark.memory</code> explicitly, though its enforcement relies on Python&#39;s <code>resource</code> module and won&#39;t work on Windows and won&#39;t actually limit anything on macOS<sup class="footnote-ref"><a href="#fn4" id="fnref4:10">[4:10]</a></sup>.</p></blockquote><h2 id="bottleneck-cache-utilization" tabindex="-1">Cache utilization <a class="header-anchor" href="#bottleneck-cache-utilization" aria-label="Permalink to &quot;Cache utilization {#bottleneck-cache-utilization}&quot;">​</a></h2><p><span class="tag">CSTOR</span></p><p>Spark&#39;s event log carries no block-access or hit-rate events, so cache utilization is read from periodic per-RDD storage snapshots instead: how much of a persisted RDD is actually resident where it was asked to be.</p><h3 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h3><p>A persisted RDD, one whose storage level requests memory and/or disk with at least one partition actually cached, is evaluated on two independent measures and can register on both at once:</p><table tabindex="0"><thead><tr><th>Signal</th><th>Info</th><th>Warning</th></tr></thead><tbody><tr><td>Cached ratio (cached partitions ÷ total partitions)</td><td>&lt; 90%</td><td>&lt; 50%</td></tr><tr><td>Disk ratio (disk bytes ÷ (memory bytes + disk bytes)), <code>MEMORY_AND_DISK*</code> levels only</td><td>&gt; 15%</td><td>&gt; 40%</td></tr></tbody></table><p>Both ratios come from a point-in-time storage snapshot taken at stage-submission events, not a runtime block-access count, so confirm against the Spark UI&#39;s Storage tab before acting. Confidence scales with the RDD&#39;s partition count: below 10 partitions the ratio is noisy enough to call low confidence, at 50 or more it&#39;s high.</p><h3 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h3><p>A partially cached RDD still pays the recomputation cost for its evicted share on every later read, undercutting the reason it was cached in the first place. Disk spillover under a <code>MEMORY_AND_DISK*</code> level keeps that data around instead of forcing a recompute, but a disk read is still far slower than serving it out of memory.</p><h3 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h3><ul><li>Increase executor memory, or reduce the size of the dataset being cached, so more of it fits in the storage region without eviction.</li><li>If the RDD is only cached for the occasional narrow scan, weigh whether persisting it at all is worth the partial-cache overhead, versus recomputing the slice you actually need.</li></ul><h2 id="sources" tabindex="-1">Sources <a class="header-anchor" href="#sources" aria-label="Permalink to &quot;Sources&quot;">​</a></h2><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://raw.githubusercontent.com/spoddutur/spark-notes/master/task_memory_management_in_spark.md" target="_blank" rel="noreferrer">Task Memory Management in Spark</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/tuning.html" target="_blank" rel="noreferrer">Tuning Spark</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a> <a href="#fnref2:2" class="footnote-backref">↩︎</a> <a href="#fnref2:3" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://luminousmen.com/post/dive-into-spark-memory" target="_blank" rel="noreferrer">Dive into Spark Memory</a> <a href="#fnref3" class="footnote-backref">↩︎</a> <a href="#fnref3:1" class="footnote-backref">↩︎</a> <a href="#fnref3:2" class="footnote-backref">↩︎</a> <a href="#fnref3:3" class="footnote-backref">↩︎</a> <a href="#fnref3:4" class="footnote-backref">↩︎</a> <a href="#fnref3:5" class="footnote-backref">↩︎</a> <a href="#fnref3:6" class="footnote-backref">↩︎</a> <a href="#fnref3:7" class="footnote-backref">↩︎</a> <a href="#fnref3:8" class="footnote-backref">↩︎</a> <a href="#fnref3:9" class="footnote-backref">↩︎</a> <a href="#fnref3:10" class="footnote-backref">↩︎</a> <a href="#fnref3:11" class="footnote-backref">↩︎</a> <a href="#fnref3:12" class="footnote-backref">↩︎</a> <a href="#fnref3:13" class="footnote-backref">↩︎</a> <a href="#fnref3:14" class="footnote-backref">↩︎</a> <a href="#fnref3:15" class="footnote-backref">↩︎</a> <a href="#fnref3:16" class="footnote-backref">↩︎</a> <a href="#fnref3:17" class="footnote-backref">↩︎</a> <a href="#fnref3:18" class="footnote-backref">↩︎</a> <a href="#fnref3:19" class="footnote-backref">↩︎</a> <a href="#fnref3:20" class="footnote-backref">↩︎</a> <a href="#fnref3:21" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration — Spark</a> <a href="#fnref4" class="footnote-backref">↩︎</a> <a href="#fnref4:1" class="footnote-backref">↩︎</a> <a href="#fnref4:2" class="footnote-backref">↩︎</a> <a href="#fnref4:3" class="footnote-backref">↩︎</a> <a href="#fnref4:4" class="footnote-backref">↩︎</a> <a href="#fnref4:5" class="footnote-backref">↩︎</a> <a href="#fnref4:6" class="footnote-backref">↩︎</a> <a href="#fnref4:7" class="footnote-backref">↩︎</a> <a href="#fnref4:8" class="footnote-backref">↩︎</a> <a href="#fnref4:9" class="footnote-backref">↩︎</a> <a href="#fnref4:10" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://www.databricks.com/blog/2015/04/28/project-tungsten-bringing-spark-closer-to-bare-metal.html" target="_blank" rel="noreferrer">Project Tungsten: Bringing Spark Closer to Bare Metal</a> <a href="#fnref5" class="footnote-backref">↩︎</a> <a href="#fnref5:1" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><em>Learning Spark, 2nd Edition</em>, Damji, Wenig, Das, Lee, ch. 6–7 <a href="#fnref6" class="footnote-backref">↩︎</a> <a href="#fnref6:1" class="footnote-backref">↩︎</a> <a href="#fnref6:2" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://luminousmen.com/post/explaining-the-mechanics-of-spark-caching" target="_blank" rel="noreferrer">Explaining the Mechanics of Spark Caching</a> <a href="#fnref7" class="footnote-backref">↩︎</a> <a href="#fnref7:1" class="footnote-backref">↩︎</a> <a href="#fnref7:2" class="footnote-backref">↩︎</a> <a href="#fnref7:3" class="footnote-backref">↩︎</a> <a href="#fnref7:4" class="footnote-backref">↩︎</a></p></li><li id="fn8" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/rdd-programming-guide.html" target="_blank" rel="noreferrer">RDD Programming Guide</a> <a href="#fnref8" class="footnote-backref">↩︎</a></p></li><li id="fn9" class="footnote-item"><p><em>High Performance Spark, 2nd Edition</em>, Karau, Polak &amp; Warren, ch. 7 <a href="#fnref9" class="footnote-backref">↩︎</a> <a href="#fnref9:1" class="footnote-backref">↩︎</a> <a href="#fnref9:2" class="footnote-backref">↩︎</a> <a href="#fnref9:3" class="footnote-backref">↩︎</a></p></li><li id="fn10" class="footnote-item"><p><a href="https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala" target="_blank" rel="noreferrer">SQLConf.scala</a> <a href="#fnref10" class="footnote-backref">↩︎</a> <a href="#fnref10:1" class="footnote-backref">↩︎</a></p></li><li id="fn11" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/monitoring.html#spark-history-server" target="_blank" rel="noreferrer">Monitoring and Instrumentation</a> <a href="#fnref11" class="footnote-backref">↩︎</a> <a href="#fnref11:1" class="footnote-backref">↩︎</a></p></li></ol></section>',45)])])}const k=o(c,[["render",l]]);export{y as __pageData,k as default};

package/public/docs/assets/{tuning-reference_memory-model.md.DhT-n4y3.lean.js → tuning-reference_memory-model.md.CWhvgevP.lean.js}

@@ -1 +1 @@
-import{_ as o,o as a,c as r,a5 as t}from"./chunks/framework.DSg0KOwT.js";const s="/docs/assets/memory-regions.XHvO7jHG.svg",n="/docs/assets/memory-regions.dark.D4TP9_08.svg",f="/docs/assets/memory-borrowing.BqQRJg0u.svg",i="/docs/assets/memory-borrowing.dark.Yhh20O9C.svg",y=JSON.parse('{"title":"Memory Management","description":"","frontmatter":{"title":"Memory Management"},"headers":[],"relativePath":"tuning-reference/memory-model.md","filePath":"tuning-reference/memory-model.md"}'),c={name:"tuning-reference/memory-model.md"};function l(h,e,d,p,u,m){return a(),r("div",null,[...e[0]||(e[0]=[t("",34)])])}const k=o(c,[["render",l]]);export{y as __pageData,k as default};
+import{_ as o,o as a,c as t,a5 as r}from"./chunks/framework.DSg0KOwT.js";const s="/docs/assets/memory-regions.XHvO7jHG.svg",n="/docs/assets/memory-regions.dark.D4TP9_08.svg",i="/docs/assets/memory-borrowing.BqQRJg0u.svg",f="/docs/assets/memory-borrowing.dark.Yhh20O9C.svg",y=JSON.parse('{"title":"Memory Management","description":"","frontmatter":{"title":"Memory Management"},"headers":[],"relativePath":"tuning-reference/memory-model.md","filePath":"tuning-reference/memory-model.md"}'),c={name:"tuning-reference/memory-model.md"};function l(h,e,d,p,u,m){return a(),t("div",null,[...e[0]||(e[0]=[r("",45)])])}const k=o(c,[["render",l]]);export{y as __pageData,k as default};

package/public/docs/assets/{user-guide_mcp-tools.md.Vi3RoflJ.js → user-guide_mcp-tools.md.DGvolGzt.js}

@@ -103,7 +103,7 @@ import{_ as i,o as a,c as e,a5 as n}from"./chunks/framework.DSg0KOwT.js";const E
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;title&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Task skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">    &quot;content&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;# Task skew</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\n\\n</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">...&quot;</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">  }</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div><p><code>tuningDoc</code> is <code>null</code> when the finding type has no vendored tuning-doc page. This isn&#39;t exhaustive, but two examples: <code>configAudit</code> (its four audited properties each have their own anchor rather than one shared page, so no single anchor resolves) and <code>incompleteRun</code> (no upstream tuning page covers this signal at all).</p><h2 id="get-reference-doc" tabindex="-1"><code>get_reference_doc</code> <a class="header-anchor" href="#get-reference-doc" aria-label="Permalink to &quot;\`get_reference_doc\`&quot;">​</a></h2><p>Full tuning-reference chapter or bottleneck markdown for one doc anchor (e.g. <code>&quot;#joins&quot;</code>, <code>&quot;#bottleneck-skew&quot;</code>, <code>&quot;#metric-task-duration&quot;</code>), independent of any run, so a client without browser access can read the same reference material the dashboard links to.</p><p>Parameters:</p><ul><li><code>anchor</code>: <code>string</code> (required): a doc anchor, with or without the leading <code>#</code></li></ul><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_reference_doc&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;anchor&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;#bottleneck-skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">}</span></span></code></pre></div><p><code>tuningDoc</code> is <code>null</code> when the finding type has no vendored tuning-doc page. This isn&#39;t exhaustive, but two examples: <code>configAudit</code> (its four audited properties each have their own anchor rather than one shared page, so no single anchor resolves) and <code>incompleteRun</code> (no upstream tuning page covers this signal at all). A type whose section lives on a general chapter rather than a bottleneck page (<code>autoscalingChurn</code> on Cluster Tuning, <code>cacheUtilization</code> on Memory Management) returns that whole chapter, with <code>anchor</code> naming the section.</p><h2 id="get-reference-doc" tabindex="-1"><code>get_reference_doc</code> <a class="header-anchor" href="#get-reference-doc" aria-label="Permalink to &quot;\`get_reference_doc\`&quot;">​</a></h2><p>Full tuning-reference chapter or bottleneck markdown for one doc anchor (e.g. <code>&quot;#joins&quot;</code>, <code>&quot;#bottleneck-skew&quot;</code>, <code>&quot;#metric-task-duration&quot;</code>), independent of any run, so a client without browser access can read the same reference material the dashboard links to.</p><p>Parameters:</p><ul><li><code>anchor</code>: <code>string</code> (required): a doc anchor, with or without the leading <code>#</code></li></ul><p>Example call:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{ </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;name&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;get_reference_doc&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;arguments&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: { </span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">&quot;anchor&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;#bottleneck-skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> } }</span></span></code></pre></div><p>Example response:</p><div class="language-json vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">json</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">{</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;anchor&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;bottleneck-skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;title&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;Task skew&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">,</span></span>
 <span class="line"><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">  &quot;content&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">: </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;# Task skew</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">\\n\\n</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">...&quot;</span></span>

package/public/docs/assets/user-guide_understanding-findings.md.B29f0ucS.js

@@ -0,0 +1 @@
+import{_ as a,o as t,c as o,a5 as r}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Understanding findings","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/understanding-findings.md","filePath":"user-guide/understanding-findings.md"}'),s={name:"user-guide/understanding-findings.md"};function i(n,e,l,d,c,h){return t(),o("div",null,[...e[0]||(e[0]=[r('<h1 id="understanding-findings" tabindex="-1">Understanding findings <a class="header-anchor" href="#understanding-findings" aria-label="Permalink to &quot;Understanding findings&quot;">​</a></h1><p>Every flagged problem carries a short ALL-CAPS tag. This page has one entry per tag: what it means, and what to do about it.</p><p>Turn on <a href="./getting-started.html#advanced-view">Advanced view</a> in the dashboard to see, per finding, the confidence level and a link to the same background reading collected here, opened in an in-app &quot;Reference&quot; panel next to the board. A few tags share that reading with another tag, because the underlying Spark-tuning material overlaps: <code>SFAIL</code> with <code>FAIL</code>, <code>PART</code> with <code>SHFL</code>, <code>SPEC</code> with <code>STRAG</code>, and <code>CACHE</code>/<code>LOCAL</code> with <code>UTIL</code>.</p><h2 id="per-stage" tabindex="-1">Per-stage <a class="header-anchor" href="#per-stage" aria-label="Permalink to &quot;Per-stage&quot;">​</a></h2><h3 id="skew" tabindex="-1"><code>SKEW</code>: Task skew <a class="header-anchor" href="#skew" aria-label="Permalink to &quot;`SKEW`: Task skew {#skew}&quot;">​</a></h3><p>A small number of tasks take much longer than their peers in the same stage. For join-driven skew, enable AQE skew-join handling (<code>spark.sql.adaptive.skewJoin.enabled</code>); otherwise salt the key or repartition on a better key.</p><h3 id="shfl" tabindex="-1"><code>SHFL</code>: Shuffle I/O <a class="header-anchor" href="#shfl" aria-label="Permalink to &quot;`SHFL`: Shuffle I/O {#shfl}&quot;">​</a></h3><p>Tasks move a large amount of intermediate data between stages. Raise <code>spark.sql.shuffle.partitions</code>, or add a broadcast join. Only flagged on stages that take at least 0.5% of the run.</p><h3 id="spill" tabindex="-1"><code>SPILL</code>: Memory and disk spill <a class="header-anchor" href="#spill" aria-label="Permalink to &quot;`SPILL`: Memory and disk spill {#spill}&quot;">​</a></h3><p>Tasks are writing data out of memory, which slows execution. Two spill patterns get flagged differently: skew spill, where a few heavy tasks spill while most don&#39;t (rebalance partitioning), and volume spill, where most tasks spill because the data genuinely exceeds available memory (add partitions). Only flagged on stages that take at least 0.5% of the run.</p><h3 id="gc" tabindex="-1"><code>GC</code>: Garbage collection pressure <a class="header-anchor" href="#gc" aria-label="Permalink to &quot;`GC`: Garbage collection pressure {#gc}&quot;">​</a></h3><p>Tasks spend an unusually large share of time reclaiming memory. Reduce object creation: use primitive types, avoid UDFs, or raise executor memory. A stage with very little GC gets an informational note that executor memory may be over-provisioned, only on stages that take at least 0.5% of the run.</p><h3 id="fail" tabindex="-1"><code>FAIL</code>: Failed tasks <a class="header-anchor" href="#fail" aria-label="Permalink to &quot;`FAIL`: Failed tasks {#fail}&quot;">​</a></h3><p>Tasks fail often enough to affect the stage. Failed tasks point to executor instability or data-driven errors: check driver logs for the dominant failure reason.</p><h3 id="sfail" tabindex="-1"><code>SFAIL</code>: Failed stage <a class="header-anchor" href="#sfail" aria-label="Permalink to &quot;`SFAIL`: Failed stage {#sfail}&quot;">​</a></h3><p>A stage attempt failed outright rather than losing individual tasks within it. Inspect the driver log for the failure reason and the job that triggered it.</p><h3 id="strag" tabindex="-1"><code>STRAG</code>: Straggler tasks <a class="header-anchor" href="#strag" aria-label="Permalink to &quot;`STRAG`: Straggler tasks {#strag}&quot;">​</a></h3><p>A few tasks run much slower than the rest of their stage. Rule out a GC pause or a slow shuffle fetch before assuming a hardware issue; if a skewed key is the real cause, that&#39;s a candidate for AQE&#39;s skew-join handling. Only flagged on stages that take at least 0.5% of the run.</p><h3 id="spec" tabindex="-1"><code>SPEC</code>: Speculation waste <a class="header-anchor" href="#spec" aria-label="Permalink to &quot;`SPEC`: Speculation waste {#spec}&quot;">​</a></h3><p>Speculative task attempts used a lot of executor time without confirming a genuine straggler. Self-flags a confidence that scales with how far the wasted time sits past the threshold: these thresholds are still a design spike, not yet validated against real-world runs. If task durations are just naturally variable rather than genuine stragglers, tune <code>spark.speculation.multiplier</code>/<code>spark.speculation.quantile</code>.</p><h3 id="retry" tabindex="-1"><code>RETRY</code>: Retry waste <a class="header-anchor" href="#retry" aria-label="Permalink to &quot;`RETRY`: Retry waste {#retry}&quot;">​</a></h3><p>Repeated task attempts ate into execution time even though the stage completed. Investigate executor loss or fetch failures.</p><h3 id="tiny" tabindex="-1"><code>TINY</code>: Tiny tasks <a class="header-anchor" href="#tiny" aria-label="Permalink to &quot;`TINY`: Tiny tasks {#tiny}&quot;">​</a></h3><p>Many very short tasks add scheduling overhead out of proportion to the work each one does. Repartition to fewer, larger tasks. Only flagged on stages that take at least 0.5% of the run.</p><h3 id="part" tabindex="-1"><code>PART</code>: Partition sizing <a class="header-anchor" href="#part" aria-label="Permalink to &quot;`PART`: Partition sizing {#part}&quot;">​</a></h3><p>Shuffle partitions are too large, too uneven, or too few for the work. A single shuffle partition over 5 GB, for example, will OOM or spill heavily: repartition to break it up before the stage runs.</p><h3 id="slow" tabindex="-1"><code>SLOW</code>: Stage slowness <a class="header-anchor" href="#slow" aria-label="Permalink to &quot;`SLOW`: Stage slowness {#slow}&quot;">​</a></h3><p>A stage ran long overall without a more specific cause getting flagged. Often a partition-count problem: raise parallelism via <code>spark.sql.shuffle.partitions</code> or <code>spark.default.parallelism</code>, or check for a large per-task data volume driving heavy shuffle and spill.</p><h3 id="shape" tabindex="-1"><code>SHAPE</code>: Stage shape <a class="header-anchor" href="#shape" aria-label="Permalink to &quot;`SHAPE`: Stage shape {#shape}&quot;">​</a></h3><p>The stage has an inefficient task count, output shape, or task-to-stage balance: for example, one straggler task taking a large fraction of the stage&#39;s wall-clock time. A too-low task count is only flagged on stages that take at least 0.5% of the run.</p><h3 id="host" tabindex="-1"><code>HOST</code>: Slow host <a class="header-anchor" href="#host" aria-label="Permalink to &quot;`HOST`: Slow host {#host}&quot;">​</a></h3><p>One executor is much slower than its peers. It may just hold data locality for its tasks or carry one heavy stage, rather than a hardware fault. Enable <code>spark.speculation</code> to relaunch a lagging task automatically. Only flagged on stages that take at least 0.5% of the run.</p><h2 id="app-level" tabindex="-1">App-level <a class="header-anchor" href="#app-level" aria-label="Permalink to &quot;App-level&quot;">​</a></h2><h3 id="cold" tabindex="-1"><code>COLD</code>: Executor cold start <a class="header-anchor" href="#cold" aria-label="Permalink to &quot;`COLD`: Executor cold start {#cold}&quot;">​</a></h3><p>New executors take time to become available for work. Pre-warm the cluster, or use dynamic allocation.</p><h3 id="util" tabindex="-1"><code>UTIL</code>: Low utilization <a class="header-anchor" href="#util" aria-label="Permalink to &quot;`UTIL`: Low utilization {#util}&quot;">​</a></h3><p>Allocated executors sit idle for a large share of the application run. Consider a smaller cluster, or enable dynamic allocation.</p><h3 id="mem" tabindex="-1"><code>MEM</code>: Memory utilization <a class="header-anchor" href="#mem" aria-label="Permalink to &quot;`MEM`: Memory utilization {#mem}&quot;">​</a></h3><p>Executor memory or core capacity may be over- or under-provisioned. Some detail here needs <code>spark.eventLog.logStageExecutorMetrics=true</code> on the run being analyzed; without it, per-executor memory usage can&#39;t be broken down. Review <code>spark.executor.memory</code> and executor count if allocated memory sat largely idle over the run. That idle-memory variant self-flags a confidence that scales with how far the estimated waste sits past a 1.5x buffer: it estimates waste from allocated-versus-used memory-time. Check it against the Spark UI before resizing anything.</p><h3 id="cache" tabindex="-1"><code>CACHE</code>: Caching opportunity <a class="header-anchor" href="#cache" aria-label="Permalink to &quot;`CACHE`: Caching opportunity {#cache}&quot;">​</a></h3><p>A reusable dataset (re-read via the same SQL relation more than once) may be worth persisting between stages. Self-flags a confidence that scales with how many executions reuse the same relation: reuse is only inferred, from plan-scan identity across SQL executions, so confirm the reads really do hit the same data before you cache anything.</p><h3 id="cstor" tabindex="-1"><code>CSTOR</code>: Cache storage <a class="header-anchor" href="#cstor" aria-label="Permalink to &quot;`CSTOR`: Cache storage {#cstor}&quot;">​</a></h3><p>A persisted dataset is not fully cached in memory, or is spilling to disk. Raise executor memory, or shrink the cached dataset.</p><h3 id="local" tabindex="-1"><code>LOCAL</code>: Core usage locality <a class="header-anchor" href="#local" aria-label="Permalink to &quot;`LOCAL`: Core usage locality {#local}&quot;">​</a></h3><p>Tasks run without process- or node-local data placement more often than expected. Check <code>spark.locality.wait</code> settings and executor/data colocation. Self-flags a confidence that scales with the non-local ratio and sample size: the thresholds are our own noise floor for this metric.</p><h3 id="chrn" tabindex="-1"><code>CHRN</code>: Autoscaling churn <a class="header-anchor" href="#chrn" aria-label="Permalink to &quot;`CHRN`: Autoscaling churn {#chrn}&quot;">​</a></h3><p>Executors are stood up and torn down again before they can do useful work: re-provisioning churn rather than normal scale-down. Raise <code>spark.dynamicAllocation.executorIdleTimeout</code>, or widen the <code>minExecutors</code>/<code>maxExecutors</code> bounds to reduce flapping. Self-flags a confidence that scales with how far the short-lived-executor share sits past the threshold: these thresholds are still a design spike, not yet validated against real-world runs.</p><h3 id="jobs" tabindex="-1"><code>JOBS</code>: Job failure rate <a class="header-anchor" href="#jobs" aria-label="Permalink to &quot;`JOBS`: Job failure rate {#jobs}&quot;">​</a></h3><p>A large share of completed jobs did not succeed. Inspect the driver log for the failed job(s) and the stage failures that triggered them.</p><h3 id="incmp" tabindex="-1"><code>INCMP</code>: Incomplete run <a class="header-anchor" href="#incmp" aria-label="Permalink to &quot;`INCMP`: Incomplete run {#incmp}&quot;">​</a></h3><p>This event log never recorded an <code>ApplicationEnd</code> event: capture stopped before the run finished (an in-flight job, a rotated-away log, or a cut-short capture). Every other finding and metric on the board reflects only what was captured up to that point, not the full run.</p><h2 id="configuration-scope" tabindex="-1">Configuration scope <a class="header-anchor" href="#configuration-scope" aria-label="Permalink to &quot;Configuration scope&quot;">​</a></h2><h3 id="cfg" tabindex="-1"><code>CFG</code>: Configuration audit <a class="header-anchor" href="#cfg" aria-label="Permalink to &quot;`CFG`: Configuration audit {#cfg}&quot;">​</a></h3><p>Flags configuration settings that may cause reliability or efficiency problems, independent of any one stage&#39;s behavior. Four properties are audited today:</p><ul><li><code>spark.shuffle.service.enabled</code>: flagged when dynamic allocation is on but the external shuffle service is off, since shuffle data won&#39;t survive executor removal.</li><li><code>spark.dynamicAllocation.maxExecutors</code>: flagged for inverted bounds or a missing upper bound.</li><li><code>spark.serializer</code>: flagged when still on the default Java serializer; <code>org.apache.spark.serializer.KryoSerializer</code> is faster and produces smaller buffers.</li><li><code>spark.executor.memoryOverhead</code>: flagged when set below a safe floor.</li></ul><h2 id="sql-scope" tabindex="-1">SQL scope <a class="header-anchor" href="#sql-scope" aria-label="Permalink to &quot;SQL scope&quot;">​</a></h2><h3 id="plan" tabindex="-1"><code>PLAN</code>: Plan advisor <a class="header-anchor" href="#plan" aria-label="Permalink to &quot;`PLAN`: Plan advisor {#plan}&quot;">​</a></h3><p>Flags patterns in the SQL execution plan worth reviewing. Four checks share this tag:</p><ul><li>Duplicate plan subtree: the same subtree recomputed more than once in the plan. When the repeats have the same shape but different filters, columns or tables, the finding stays informational and claims no time. Only flagged when the repeat&#39;s stages take at least 0.5% of the run.</li><li>Small files: reading an excessive number of small files.</li><li>Under-broadcast: the smaller side of a Sort Merge Join looks well under the broadcast threshold; consider a <code>broadcast()</code> hint or raising <code>spark.sql.autoBroadcastJoinThreshold</code>.</li><li>Over-broadcast: a broadcast exceeds the 1 GB threshold; check for a misapplied broadcast hint or a misconfigured <code>spark.sql.autoBroadcastJoinThreshold</code>.</li></ul>',59)])])}const f=a(s,[["render",i]]);export{p as __pageData,f as default};

package/public/docs/assets/{user-guide_understanding-findings.md.DL1UDhvR.lean.js → user-guide_understanding-findings.md.B29f0ucS.lean.js}

@@ -1 +1 @@
-import{_ as a,o as t,c as o,a5 as r}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Understanding findings","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/understanding-findings.md","filePath":"user-guide/understanding-findings.md"}'),i={name:"user-guide/understanding-findings.md"};function s(n,e,l,d,c,h){return t(),o("div",null,[...e[0]||(e[0]=[r("",59)])])}const f=a(i,[["render",s]]);export{p as __pageData,f as default};
+import{_ as a,o as t,c as o,a5 as r}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Understanding findings","description":"","frontmatter":{},"headers":[],"relativePath":"user-guide/understanding-findings.md","filePath":"user-guide/understanding-findings.md"}'),s={name:"user-guide/understanding-findings.md"};function i(n,e,l,d,c,h){return t(),o("div",null,[...e[0]||(e[0]=[r("",59)])])}const f=a(s,[["render",i]]);export{p as __pageData,f as default};

package/public/docs/contributor-guide/architecture/board-widgets.html

@@ -9,11 +9,11 @@
     <link rel="preload stylesheet" href="/docs/assets/style.DSixAiZE.css" as="style">
     <link rel="preload stylesheet" href="/docs/vp-icons.css" as="style">
     
-    <script type="module" src="/docs/assets/app.APoq_UBs.js"></script>
+    <script type="module" src="/docs/assets/app.DsFQcynM.js"></script>
     <link rel="preload" href="/docs/assets/inter-roman-latin.Di8DUHzh.woff2" as="font" type="font/woff2" crossorigin="">
-    <link rel="modulepreload" href="/docs/assets/chunks/theme.PavXO_vz.js">
+    <link rel="modulepreload" href="/docs/assets/chunks/theme.Ca4dR9dL.js">
     <link rel="modulepreload" href="/docs/assets/chunks/framework.DSg0KOwT.js">
-    <link rel="modulepreload" href="/docs/assets/contributor-guide_architecture_board-widgets.md.CWpj01WU.lean.js">
+    <link rel="modulepreload" href="/docs/assets/contributor-guide_architecture_board-widgets.md.9ToZXcTk.lean.js">
     <link rel="icon" type="image/svg+xml" href="/docs/favicon.svg">
     <link rel="preconnect" href="https://fonts.googleapis.com">
     <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="">
@@ -22,8 +22,8 @@
     <script id="check-mac-os">document.documentElement.classList.toggle("mac",/Mac|iPhone|iPod|iPad/i.test(navigator.platform));</script>
   </head>
   <body>
-    <div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="/docs/" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="/docs/favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/tuning-reference/" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Contributor Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/development-setup.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Development setup</p><!--]--></a><!----></div><!----></div><section class="VPSidebarItem level-1 collapsible collapsed is-link has-active" data-v-b3fd67f8><div class="item" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/" data-v-b3fd67f8><!--[--><h3 class="text" data-v-b3fd67f8>Architecture</h3><!--]--></a><div class="caret" role="button" aria-label="toggle section" tabindex="0" data-v-b3fd67f8><span class="vpi-chevron-right caret-icon" data-v-b3fd67f8></span></div></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/overview.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Overview</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/worker-protocol.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Worker protocol</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/state-and-history.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>State & history intake</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/detector-contract.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Detector contract</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/impact-estimation.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Impact estimation</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/widget-rendering.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Widget rendering</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/board-widgets.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Board widgets</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/drill-down.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Drill-down</p><!--]--></a><!----></div><!----></div><!--]--></div></section><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/testing.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Testing & verification</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/contributing.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Contributing</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_contributor-guide_architecture_board-widgets" data-v-39a288b8><div><h1 id="board-widgets" tabindex="-1">Board widgets <a class="header-anchor" href="#board-widgets" aria-label="Permalink to &quot;Board widgets&quot;">​</a></h1><h2 id="board-widgets-beyond-the-fixed-six" tabindex="-1">Beyond the fixed six <a class="header-anchor" href="#board-widgets-beyond-the-fixed-six" aria-label="Permalink to &quot;Beyond the fixed six {#board-widgets-beyond-the-fixed-six}&quot;">​</a></h2><p>(Components below live in <code>src/view/widgets/</code>, one file per widget name, e.g. <code>JobFailures.tsx</code>, <code>MemoryUtilization.tsx</code>.)</p><p>Six extra cards render in the Findings tab&#39;s active grid alongside the fixed spec §5 six: Incomplete Run, Job Failures, Caching Opportunities, Config Audit, Plan Advisor, and Autoscaling Churn, all <code>region: &#39;action&#39;</code> in <code>detector-registry.tsx</code>:</p><ul><li><p><strong>Incomplete Run</strong> (tag <code>INCMP</code>, <code>IncompleteRun.tsx</code>): app-level &quot;the capture never finished&quot; caveat (DETECTORS entry <code>incompleteRun</code>). Fires whenever <code>app.startTime</code> was observed but <code>app.endTime</code> was not, i.e. no <code>SparkListenerApplicationEnd</code> in the log: an in-flight job, a rotated-away log, or a capture cut short. Order 5, ahead of every other detector, so its card is first among the Findings tab&#39;s <code>action</code>-region active widgets when present. The Findings tab&#39;s recommendation rollup (rendered by <code>FixTheseFirst.tsx</code>) excludes <code>incompleteRun</code> from that list outright (see <a href="./widget-rendering.html#widget-rendering-order-fixed-spec-§5">Widget rendering order</a>): it&#39;s a pipeline-completeness caveat, not an addressable fix, so it never competes with other findings for a ranked slot there. Self-gates to <code>null</code> (no card in the DOM) once the run completed normally. Unrelated to <code>evidence-availability.ts</code>&#39;s own <code>trustworthy</code> gate (see <a href="./worker-protocol.html#evidence-availability-contract-v1">Evidence-availability contract</a>): that ledger only downgrades <em>absence</em> conclusions for individual evidence categories, never becomes a <code>DETECTORS</code> finding itself, and this card does not read it. No <code>docAnchor</code> is set, the same deviation as Autoscaling Churn below: this is a tool-specific signal with no upstream <code>spark-tuning-reference</code> section.</p></li><li><p><strong>Job Failures</strong> (tag <code>JOBS</code>): app-level job-failure-rate rollup (DETECTORS entry <code>jobFailureRate</code> in <code>packages/core/src/detectors.ts</code>), computed from <code>app.jobs</code> (<code>SparkListenerJobEnd</code> results): ≥10% info, ≥30% warning, ≥50% critical. Complements the per-stage, task-level Failed Tasks card (tag <code>FAIL</code>).</p></li><li><p><strong>Caching Opportunities</strong> (tag <code>CACHE</code>, <code>CachingOpportunity.tsx</code>): app-level SQL relation-reuse detector (DETECTORS entry <code>cachingOpportunity</code>), computed from <code>ctx.sql</code>. It walks each execution&#39;s <code>planTree</code> and keys every scan by its stable pre-AQE identity via <code>scanRelationId</code> (<code>plan-summary.ts</code>): <code>parquet:&lt;db.table&gt;</code>, <code>delta:&lt;db.table&gt;</code>, <code>jdbc:&lt;schema.table&gt;</code>. That dedupes relations within one execution (self-joins count once) and flags any relation scanned by <code>&gt;= minExecutions</code> (2) distinct executions. Relation identity comes from the catalog-qualified scan name (nodeName, e.g. <code>Scan parquet spark_catalog.db.t</code>), not the <code>Location:</code> path, since Spark truncates that path at ~100 chars and points it at <code>_delta_log</code> for Delta tables; internal Delta-log metadata scans are dropped. Read bytes come from the scan&#39;s <code>size of files read</code> metric (<code>0</code>/unknown for JDBC). Findings carry <code>executionReuse</code> (<code>value</code> = distinct-execution count), <code>relation</code>, <code>format</code>, <code>executionIds</code>, <code>totalReadBytes</code>, and a size-aware <code>recommendation</code> (<code>confidence</code> scaled <code>low</code>/<code>medium</code>/<code>high</code> via <code>cachingReuseConfidence</code> off reuse-execution count, plus <code>validationRequired</code>). Renders nothing when clean (no card in the DOM). One row per relation: name + format badge, reuse count, <code>Data read</code> (<code>formatBytes</code>, em-dash when unknown), recommendation, sorted by <code>totalReadBytes</code> then reuse count descending. Rows reveal 6 initially, then up to 30 more per click. Pure-RDD-API apps (no SQL executions) produce no finding: a deliberate trade-off replacing the former RDD-lineage heuristic, which surfaced only internal query-engine RDDs on DataFrame/SQL workloads.</p><p>It also detects composite reuse. When the same join/union subtree (not just a leaf scan) recurs across <code>&gt;= minExecutions</code> distinct SQL executions, the detector emits one <code>variant:&#39;composite&#39;</code> finding recommending caching the derived join/union result instead of two independent leaf-relation rows, and suppresses the leaf findings for relations fully covered by it. A relation reused beyond the composite&#39;s executions keeps a residual leaf finding for just the uncovered executions. Composite identity is structural: an anchor plan-shape fingerprint (<code>findCompositeCandidates</code>/<code>computePlanShapes</code>&#39;s <code>opts.includeDetail</code> path, <code>detectors.ts</code>) folds the join/union node&#39;s own normalized <code>detail</code> (join type, columns, literals, with expr ids, <code>plan_id=</code>, codegen-stage numbers, and AQE&#39;s BuildLeft/BuildRight stripped, and commutative equality operands canonicalized) plus its children&#39;s detail-free shapes. Descendant nodes never contribute detail text, only structural shape, so a shared join reused across differently-filtered pre-join scans still matches (a deliberate trade-off) while a different join <em>condition</em> on the same tables does not collide. Nested composites (an inner join reused both standalone and inside an outer join) dedupe one level at a time: an inner composite fully covered by a qualifying outer composite&#39;s execution set is suppressed entirely, and a superset recomputes a residual finding over just its extra executions. Reuse via pure projection/aggregation with no join/union underneath stays leaf-level, not modeled as composite. <code>variant:&#39;composite&#39;</code> findings carry <code>operator</code> (<code>&#39;join&#39;</code>|<code>&#39;union&#39;</code>), <code>relations</code> (leaf relations under the composite, for display), and <code>format:&#39;derived&#39;</code> (a sentinel: composites have no scan storage format). They render a <code>JOIN</code>/<code>UNION</code> operator badge (styled distinct from the format badges) plus a confidence marker (<code>confidence</code> scaled via <code>cachingReuseConfidence</code>, <code>validationRequired</code>) in <code>CachingOpportunity.tsx</code>.</p></li><li><p><strong>Autoscaling Churn</strong> (tag <code>CHRN</code>, <code>AutoscalingChurn.tsx</code>): app-level short-lived-executor detector (DETECTORS entry <code>autoscalingChurn</code> in <code>packages/core/src/detectors.ts</code>; reuses the same <code>executorsAdded</code>/<code>executorsRemoved</code> matching logic as <code>utilization</code>). For each added executor, finds its matching removal event (falling back to <code>app.endTime</code> for an executor still alive when the log ends) and flags it short-lived if its lifetime is under 2 minutes (<code>thresholds.shortLivedMs</code>). Warns above 30% short-lived, escalates to critical above 60% (<code>confidence</code> scales <code>low</code>/<code>medium</code>/<code>high</code> via <code>autoscalingChurnConfidence</code>, off how far the short-lived share sits past <code>warningPct</code>/<code>criticalPct</code>: these thresholds are still an unvalidated design-spike estimate, not yet checked against real autoscaling-heavy logs). Returns no finding below 5 total executors (noise floor) or when <code>app.endTime</code> is missing (truncated/still-running log). The widget itself is unchanged apart from a finding-driven verdict banner (impact dot + <code>CHRN</code> tag + recommendation) above its existing add/remove chart. The chart&#39;s muted scale-down bar coloring is untouched. No <code>docAnchor</code> is set yet: the upstream <code>shuffle-works/spark-tuning-reference</code> docs repo has no <code>bottleneck-autoscaling-churn</code> section (verified 2026-08-19), so the <code>CHRN</code> tag currently renders with no docs-panel link (a tracked follow-up).</p></li><li><p><strong>Config Audit</strong> (tag <code>CFG</code>): static Spark-config sanity findings derived from <code>app.config</code>/<code>app.resources</code> (parsed from <code>SparkListenerEnvironmentUpdate</code>): dynamic-allocation vs. shuffle-service mismatch, inverted/missing autoscaling bounds, non-Kryo serializer, low executor <code>memoryOverhead</code>. Computed outside the runtime bottleneck catalog: its findings live in the separate <code>configFindings</code> stream, not <code>catalog</code>. But <code>FixTheseFirst</code>/<code>Alerts</code> both merge <code>catalog</code> and <code>configFindings</code> (neither reads <code>region</code> any more), so a Config Audit finding is still eligible for a row in the Findings tab&#39;s recommendation rollup and still gets its own card in the active grid, alongside genuine bottleneck-catalog findings.</p></li><li><p><strong>Plan Advisor</strong> (tag <code>PLAN</code>): SQL-plan-level findings computed from <code>appModel.sql</code>&#39;s resolved <code>planTree</code> (DETECTORS entries <code>duplicatePlanSubtree</code>, <code>smallFiles</code>, <code>broadcastSizing</code> in <code>packages/core/src/detectors.ts</code>): repeated plan subtrees (≥3 nodes, ≥2 occurrences, critical if the repeated root is an <code>Exchange</code>), small-files read/write (&gt;100 files averaging ❤️ MiB), and broadcast-join sizing in both directions (missed- broadcast info finding, over-broadcast warning at &gt;1 GB). Each of the four emitted types now renders as its own card (<code>DuplicatePlanSubtree.tsx</code>, <code>SmallFiles.tsx</code>, <code>UnderBroadcast.tsx</code>, <code>OverBroadcast.tsx</code>; the 2026-09 widget/finding-type 1:1 mapping redesign split what used to be one shared <code>PlanFindings.tsx</code> card): all four still share the <code>PLAN</code> tag and the same <code>--plan-aggregate</code> badge tint (<code>src/view/plan-finding-shared.ts</code>). Their <code>docAnchor</code>s (<code>#bottleneck-duplicate-plan-subtree</code>, <code>#bottleneck-small-files</code>, <code>#bottleneck-broadcast-sizing</code>) are not yet live in the vendored docs site, so <code>npm run update-docs</code> will abort until upstream <code>spark-tuning-reference</code> adds them.</p></li></ul><p><code>detector-registry.tsx</code>&#39;s <code>REGISTRY</code> still carries <code>region: &#39;reference&#39;</code> on four entries, each its own component now: <code>memoryUtilization</code> (<code>MemoryUtilization</code>), <code>utilization</code> (<code>ExecutorUtilization</code>), <code>cacheUtilization</code> (<code>CacheUtilization</code>), and <code>coreLocality</code> (<code>CoreUsageArea</code>). <code>broadcastSizing</code> is gone from <code>REGISTRY</code> entirely (the 2026-09 widget/finding-type 1:1 mapping redesign dropped it): it never backed a real <code>Finding</code> (the <code>broadcastSizing</code> <code>DETECTORS</code> entry only ever emits <code>underBroadcast</code>/<code>overBroadcast</code>, both <code>region: &#39;action&#39;</code>, each now its own Plan Advisor card), so there is no key left for it to occupy or a clean-check line for it to render. Autoscaling Churn, the other executor-provisioning-lifecycle detector alongside <code>utilization</code>/<code>memoryUtilization</code>, was deliberately given <code>action</code> rather than <code>reference</code> (see &quot;Beyond the fixed six&quot; above).</p><p><code>region</code> decides one thing (see <a href="./widget-rendering.html#widget-rendering-order-fixed-spec-§5">Widget rendering order</a>): whether a widget always mounts. <code>isAlwaysMountedType()</code> flags exactly one of the four <code>reference</code>-region types: Core Usage by Locality. That one mounts unconditionally from <code>appModel</code> in its own small grid inside the Findings tab, regardless of finding state. Memory Utilization, Executor Utilization, and Cache Storage are <code>reference</code> too, but all three are excluded by product decision, not a component-sharing constraint (<code>ALWAYS_MOUNTED_EXCEPTIONS</code> in <code>src/view/detector-registry.tsx</code> carries <code>cacheUtilization</code>, <code>memoryUtilization</code>, and <code>utilization</code>): a clean run on any of them isn&#39;t evidence worth surfacing unconditionally, so each collapses to an ordinary <code>CleanCheckRow</code> like any other action-region type on a clean run. Every other <code>REGISTRY</code> widget still renders unconditionally as either an active card or a clean-check line, and <code>region</code> still sets <code>orderedWidgets()</code>&#39;s sort order within the active grid (<code>action</code> components first, <code>reference</code> ones after). The Full app report tab is structural-only and reads no <code>REGISTRY</code> entry. So Core Usage by Locality, the one widget still tagged <code>region: &#39;reference&#39;</code> and exempt from <code>ALWAYS_MOUNTED_EXCEPTIONS</code>, renders in the Findings tab&#39;s always-visible grid rather than inside a separate Reference region (Memory Utilization, Executor Utilization, and Cache Storage all render through the ordinary active/clean paths instead):</p><ul><li><strong>Memory Utilization</strong> (tag <code>MEM</code>): app-level card combining three sub-findings from the <code>memoryUtilization</code> DETECTORS entry (<code>packages/core/src/detectors.ts</code>): idle-cores rate (busy-core-time from the worker&#39;s run-aggregates sweep vs. peak-cores × wall-clock), per-executor memory bands (peak heap vs. allocated, gated on <code>spark.eventLog.logStageExecutorMetrics</code>: a distinct <code>dataUnavailable</code> finding renders when that config was off), and an unverified memory-waste model (<code>confidence</code> scales <code>low</code>/<code>medium</code>/<code>high</code> via <code>memoryWasteConfidence</code>, off how far the wasted/used ratio sits past the 1.5× buffer). The driver-memory half of the original spec is dropped: the worker only extracts <em>allocated</em> <code>spark.driver.memory</code>, never a driver actual-usage metric, so there is nothing to band against. The separate <code>utilization</code> DETECTORS entry (an <code>avgUtilization</code> info finding: active-executor-time fraction below 60%) has its own card, <strong>Executor Utilization</strong> (tag <code>UTIL</code>, <code>ExecutorUtilization.tsx</code>): a <code>reference</code>-region widget in its own right that renders only with an active finding, split out of this same combined widget in the 2026-09 widget/finding-type 1:1 mapping redesign (it used to render inline here; the former <code>ExecutorTimeline.tsx</code> never owned this type, despite the tag&#39;s letters).</li><li><strong>Cache Storage</strong> (tag <code>CSTOR</code>): app-level card driven by the <code>cacheUtilization</code> DETECTORS entry (<code>packages/core/src/detectors.ts</code>), evaluating two per-RDD proxies over <code>ctx.app.rddInfo</code> storage snapshots since Spark event logs carry no runtime block-access/read-count data: partial caching (<code>numCachedPartitions / numPartitions &lt; 0.90</code>, <code>&lt; 0.50</code> for the warning tier) and disk spillover for <code>MEMORY_AND_DISK*</code> RDDs (<code>diskSize / (memorySize + diskSize) &gt; 0.15</code>, <code>&gt; 0.40</code> for the warning tier; <code>DISK_ONLY</code> RDDs are never flagged). <code>confidence</code> scales <code>low</code>/<code>medium</code>/<code>high</code> via <code>cacheSampleConfidence(rdd.numPartitions)</code>, because the ratio is a point-in-time storage snapshot from stage-submission events, not a runtime read-count, and more partitions average that snapshot noise into a more stable ratio. The existing RDD table (ported from the legacy <code>src/widgets/cache-utilization.js</code> canvas widget) still renders unconditionally; flagged rows get an inline <code>CSTOR</code> tag next to the RDD name, and every flagged RDD&#39;s recommendation renders below the table, worst-first.</li><li><strong>Core Usage by Locality</strong> (tag <code>LOCAL</code>, <code>CoreUsageArea.tsx</code>): app-level non-local-task-ratio threshold (DETECTORS entry <code>coreLocality</code>; the other half of a wasted-cores-ratio check, the idle-core half already covered by Memory Utilization&#39;s <code>idleCores</code> variant above). Sums <code>RACK_LOCAL</code> + <code>ANY</code> task counts against total task count across every stage&#39;s <code>stage.localityStats</code> (pure reducer <code>computeCoreLocalityRatio</code>, <code>packages/core/src/core-locality-ratio.ts</code>). <code>NO_PREF</code> stays in the denominator only, since it&#39;s what shuffle-read stages legitimately report with no locality problem. Below 50 total tasks or below a 15% non-local ratio: no finding; 15%-35%: warning; &gt;= 35%: critical, still unvalidated design-spike thresholds (<code>confidence</code> scales <code>low</code>/<code>medium</code>/<code>high</code> via <code>coreLocalityConfidence</code>, off whichever is weaker of the non-local ratio and the sampled task count, the same evidence-strength convention as the memory-waste model above). The widget&#39;s always-rendered stacked-area chart (<code>packages/core/src/core-usage-locality.ts</code>) is unaffected. The finding only adds a threshold/impact-band section above it, a per-stage non-local breakdown below it (shown whenever any non-local tasks exist at all, independent of whether the aggregate crossed threshold), and a one-line cross-reference to Memory Utilization when its <code>idleCores</code> finding also fired (idle-core ratio itself is not duplicated here).</li></ul><p>Documented ALL-CAPS tag vocabulary: <code>SKEW</code>, <code>SHFL</code>, <code>SPILL</code>, <code>GC</code>, <code>COLD</code>, <code>UTIL</code>, <code>MEM</code> (Memory Utilization), <code>CSTOR</code> (Cache Storage), <code>LOCAL</code> (Core Usage by Locality), plus <code>FAIL</code> (Failed Tasks), <code>JOBS</code> (Job Failures), <code>CFG</code> (Config Audit), <code>PLAN</code> (Plan Advisor), <code>SFAIL</code> (stage failed outright), <code>PART</code> (partition sizing), <code>SLOW</code> (stage overall slowness), <code>SHAPE</code> (stage shape smells), <code>CACHE</code> (caching opportunity) and <code>CHRN</code> (autoscaling churn).</p><p>ETL Phase Attribution (<code>packages/core/src/etl-phases.ts</code> + <code>EtlPhases.tsx</code>), What-If Executor Scaling (<code>packages/core/src/scaling-sim.ts</code> + <code>ScalingSim.tsx</code>, design spike: makespan predictions are unvalidated and carry a Model Error indicator), and Compute Efficiency (<code>packages/core/src/efficiency-model.ts</code> + <code>EfficiencyModel.tsx</code>, design spike: driver-vs-executor waste split, two theoretical floors, and the §6 right-sizing copy) are main-thread report modules, not <code>DETECTORS</code> entries: descriptive lenses with no impact-band threshold, rendered unconditionally into the Full app report tab rather than participating in the bottleneck catalog. <code>packages/core/src/job-groups.ts</code> (<code>checkConcurrentJobGroups</code>) is likewise a report helper: it flags when concurrent SQL-execution job groups make wall-clock-based estimates unreliable, and both the scaling simulator and the efficiency model gate their precision on it, showing a caveat banner rather than suppressing the estimate.</p><h2 id="confidence-metadata" tabindex="-1">Confidence metadata <a class="header-anchor" href="#confidence-metadata" aria-label="Permalink to &quot;Confidence metadata&quot;">​</a></h2><p>Best-effort (non-deterministic) findings carry a <code>confidence</code> + <code>validationRequired</code> marker, rendered inline by each consuming widget (e.g. <code>Spill.tsx</code>, <code>DuplicatePlanSubtree.tsx</code>, <code>SmallFiles.tsx</code>, <code>UnderBroadcast.tsx</code>, <code>OverBroadcast.tsx</code>, <code>EfficiencyModel.tsx</code>) when <code>confidence</code> is present and not <code>&#39;high&#39;</code>. There is no shared helper: each widget renders its own muted &quot;N confidence: verify&quot; text, with the <code>title</code> attribute carrying <code>validationRequired</code> as a tooltip. Spill classification is <code>medium</code> when classified (skew/volume) and <code>low</code> when unclassified; plan-summary warnings are <code>low</code>. Deterministic detectors stay unmarked, treated as high confidence.</p><h2 id="visual-system" tabindex="-1">Visual system <a class="header-anchor" href="#visual-system" aria-label="Permalink to &quot;Visual system&quot;">​</a></h2><p>Tailwind CSS v4 + shadcn/ui (Base UI primitives). The design tokens (colors, including the telemetry-console dark palette) are defined as CSS variables in an <code>@theme inline</code> block and consumed via Tailwind utility classes; there is no more hand-authored BEM CSS. Theme polarity is unchanged from the legacy app: dark = bare <code>:root</code> (no attribute), light = <code>:root[data-theme=&quot;light&quot;]</code>, toggled by <code>src/theme/ThemeProvider.tsx</code> and persisted to <code>localStorage</code>. <code>index.html</code>&#39;s inline FOUC-prevention script still runs before React mounts. Mono is still the typeface for numerics (<code>--font-mono</code>), applied via a <code>.num</code>-equivalent Tailwind utility per call site rather than one global class. Charts are Recharts (<code>src/view/charts/ChartTheme.tsx</code>&#39;s <code>CHART_COLORS</code>, read from the same CSS tokens) instead of Chart.js. React re-renders on theme toggle, so chart colors update live with the theme, unlike the old Chart.js canvases, which were built once per parse. <code>ChartFrame</code> can also expose the underlying rows through a toggleable accessible table and copy them to the clipboard as TSV.</p><h2 id="templating-xss-safe" tabindex="-1">Templating (XSS-safe) <a class="header-anchor" href="#templating-xss-safe" aria-label="Permalink to &quot;Templating (XSS-safe)&quot;">​</a></h2><p>JSX auto-escapes every interpolated value by default. The hand-rolled auto-escaping <code> html</code> `` tagged template (<code>src/widgets/utils.js</code>) and its <code>no-raw-innerhtml</code> guard test are gone, and no <code>.innerHTML</code> assignment is left anywhere in the view layer.</p><p>One Base UI-specific gotcha carried no equivalent in the legacy app: <code>WidgetCard.tsx</code> passes <code>aria-expanded={String(open) as &#39;true&#39; | &#39;false&#39;}</code> rather than the raw boolean, because Base UI&#39;s <code>Collapsible.Trigger</code> otherwise overrides a boolean <code>aria-expanded</code> prop with its own internal state via prop merging. The explicit <code>String()</code> cast keeps the rendered attribute in sync with this app&#39;s own <code>open</code> state.</p></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><a class="VPLink link pager-link prev" href="/docs/contributor-guide/architecture/widget-rendering.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Previous page</span><span class="title" data-v-e257564d>Widget rendering</span><!--]--></a></div><div class="pager" data-v-e257564d><a class="VPLink link pager-link next" href="/docs/contributor-guide/architecture/drill-down.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Next page</span><span class="title" data-v-e257564d>Drill-down</span><!--]--></a></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
-    <script>window.__VP_HASH_MAP__=JSON.parse("{\"contributor-guide_architecture_board-widgets.md\":\"CWpj01WU\",\"contributor-guide_architecture_detector-contract.md\":\"CgzUsQ6W\",\"contributor-guide_architecture_drill-down.md\":\"BtPdlM7r\",\"contributor-guide_architecture_impact-estimation.md\":\"CooslVJt\",\"contributor-guide_architecture_index.md\":\"3TO9ic6w\",\"contributor-guide_architecture_overview.md\":\"CehiRmGn\",\"contributor-guide_architecture_state-and-history.md\":\"C-xxn0q7\",\"contributor-guide_architecture_widget-rendering.md\":\"R27gQrgY\",\"contributor-guide_architecture_worker-protocol.md\":\"IbnfNrV3\",\"contributor-guide_contributing.md\":\"CvRsdr6J\",\"contributor-guide_development-setup.md\":\"DvAN_9mK\",\"contributor-guide_testing.md\":\"6rIKqSyY\",\"index.md\":\"CHJVslga\",\"tuning-reference_anti-patterns.md\":\"Df1YMIHu\",\"tuning-reference_aqe.md\":\"BIsCtLzm\",\"tuning-reference_bottleneck-broadcast-sizing.md\":\"CEstB3Ia\",\"tuning-reference_bottleneck-cold-start.md\":\"CEuy-72y\",\"tuning-reference_bottleneck-duplicate-plan-subtree.md\":\"CIohQDfn\",\"tuning-reference_bottleneck-failures.md\":\"4z5BXGJ2\",\"tuning-reference_bottleneck-gc.md\":\"DSxzZRK7\",\"tuning-reference_bottleneck-job-failure-rate.md\":\"BaJl__1W\",\"tuning-reference_bottleneck-memory-utilization.md\":\"DbP-SJZc\",\"tuning-reference_bottleneck-retry-waste.md\":\"D5JMjVOt\",\"tuning-reference_bottleneck-shuffle.md\":\"CM-nTmIH\",\"tuning-reference_bottleneck-skew.md\":\"BdUwiDhn\",\"tuning-reference_bottleneck-slow-host.md\":\"BlIo6UDW\",\"tuning-reference_bottleneck-small-files.md\":\"B8kloyx8\",\"tuning-reference_bottleneck-spill.md\":\"PNH7mITt\",\"tuning-reference_bottleneck-straggler.md\":\"DY36fHN5\",\"tuning-reference_bottleneck-tiny-tasks.md\":\"QTV7O8kU\",\"tuning-reference_bottleneck-utilization.md\":\"DTiueZC3\",\"tuning-reference_caching.md\":\"B7aQ8asB\",\"tuning-reference_cluster-config.md\":\"ZVmDGsQ3\",\"tuning-reference_config.md\":\"UvveiWG3\",\"tuning-reference_data-formats.md\":\"bjCAWH3N\",\"tuning-reference_index.md\":\"BQ_NooMV\",\"tuning-reference_intro.md\":\"CobD-lGB\",\"tuning-reference_joins.md\":\"BtKs_CuW\",\"tuning-reference_memory-model.md\":\"DhT-n4y3\",\"tuning-reference_metrics.md\":\"mLOh7Apj\",\"tuning-reference_partitioning.md\":\"q0zKF_8X\",\"tuning-reference_pyspark.md\":\"DDCfvN9t\",\"tuning-reference_shuffle.md\":\"BZZ7R4Ix\",\"tuning-reference_spark-architecture.md\":\"Dwzm5avO\",\"tuning-reference_table-formats.md\":\"D6wj-2dX\",\"user-guide_alternative-log-retrieval.md\":\"B4tPGIal\",\"user-guide_getting-started.md\":\"BJvwLEIM\",\"user-guide_mcp-tools.md\":\"Vi3RoflJ\",\"user-guide_run-comparison.md\":\"CQc1aoU8\",\"user-guide_understanding-findings.md\":\"DL1UDhvR\"}");window.__VP_SITE_DATA__=JSON.parse("{\"lang\":\"en-US\",\"dir\":\"ltr\",\"title\":\"SparkForensics\",\"description\":\"Docs for using and contributing to SparkForensics\",\"base\":\"/docs/\",\"head\":[],\"router\":{\"prefetchLinks\":true},\"appearance\":true,\"themeConfig\":{\"logo\":\"/favicon.svg\",\"nav\":[{\"text\":\"User Guide\",\"link\":\"/user-guide/getting-started\"},{\"text\":\"Contributor Guide\",\"link\":\"/contributor-guide/development-setup\"},{\"text\":\"Tuning Reference\",\"link\":\"/tuning-reference/\"}],\"sidebar\":{\"/tuning-reference/\":[{\"text\":\"Getting Started\",\"items\":[{\"text\":\"Introduction\",\"link\":\"/tuning-reference/intro\"}]},{\"text\":\"Optimization Guide\",\"items\":[{\"text\":\"Spark Execution Model\",\"link\":\"/tuning-reference/spark-architecture\"},{\"text\":\"Memory Management\",\"link\":\"/tuning-reference/memory-model\"},{\"text\":\"Partitioning\",\"link\":\"/tuning-reference/partitioning\"},{\"text\":\"Join Optimization\",\"link\":\"/tuning-reference/joins\"},{\"text\":\"Shuffle\",\"link\":\"/tuning-reference/shuffle\"},{\"text\":\"Data Formats\",\"link\":\"/tuning-reference/data-formats\"},{\"text\":\"Table Formats\",\"link\":\"/tuning-reference/table-formats\"},{\"text\":\"Caching & Persistence\",\"link\":\"/tuning-reference/caching\"},{\"text\":\"PySpark Specifics\",\"link\":\"/tuning-reference/pyspark\"},{\"text\":\"Adaptive Query Execution\",\"link\":\"/tuning-reference/aqe\"},{\"text\":\"Cluster Tuning\",\"link\":\"/tuning-reference/cluster-config\"},{\"text\":\"Anti-Patterns\",\"link\":\"/tuning-reference/anti-patterns\"}]},{\"text\":\"Detector Catalog\",\"items\":[{\"text\":\"Task Skew\",\"link\":\"/tuning-reference/bottleneck-skew\"},{\"text\":\"Shuffle I/O\",\"link\":\"/tuning-reference/bottleneck-shuffle\"},{\"text\":\"Memory / Disk Spill\",\"link\":\"/tuning-reference/bottleneck-spill\"},{\"text\":\"GC Pressure\",\"link\":\"/tuning-reference/bottleneck-gc\"},{\"text\":\"Cold Start\",\"link\":\"/tuning-reference/bottleneck-cold-start\"},{\"text\":\"Executor Utilization\",\"link\":\"/tuning-reference/bottleneck-utilization\"},{\"text\":\"Slow Host\",\"link\":\"/tuning-reference/bottleneck-slow-host\"},{\"text\":\"Task Failures\",\"link\":\"/tuning-reference/bottleneck-failures\"},{\"text\":\"Stragglers\",\"link\":\"/tuning-reference/bottleneck-straggler\"},{\"text\":\"Retry Waste\",\"link\":\"/tuning-reference/bottleneck-retry-waste\"},{\"text\":\"Tiny Tasks\",\"link\":\"/tuning-reference/bottleneck-tiny-tasks\"},{\"text\":\"Job Failure Rate\",\"link\":\"/tuning-reference/bottleneck-job-failure-rate\"},{\"text\":\"Memory Utilization\",\"link\":\"/tuning-reference/bottleneck-memory-utilization\"},{\"text\":\"Duplicate Plan Subtree\",\"link\":\"/tuning-reference/bottleneck-duplicate-plan-subtree\"},{\"text\":\"Small Files\",\"link\":\"/tuning-reference/bottleneck-small-files\"},{\"text\":\"Broadcast Sizing\",\"link\":\"/tuning-reference/bottleneck-broadcast-sizing\"}]},{\"text\":\"Reference\",\"items\":[{\"text\":\"Metrics Glossary\",\"link\":\"/tuning-reference/metrics\"},{\"text\":\"Spark Config Quick-Reference\",\"link\":\"/tuning-reference/config\"}]}],\"/user-guide/\":[{\"text\":\"User Guide\",\"items\":[{\"text\":\"Getting started\",\"link\":\"/user-guide/getting-started\"},{\"text\":\"Understanding findings\",\"link\":\"/user-guide/understanding-findings\"},{\"text\":\"Run comparison mode\",\"link\":\"/user-guide/run-comparison\"},{\"text\":\"MCP tools reference\",\"link\":\"/user-guide/mcp-tools\"},{\"text\":\"Alternative log retrieval\",\"link\":\"/user-guide/alternative-log-retrieval\"}]}],\"/contributor-guide/\":[{\"text\":\"Contributor Guide\",\"items\":[{\"text\":\"Development setup\",\"link\":\"/contributor-guide/development-setup\"},{\"text\":\"Architecture\",\"link\":\"/contributor-guide/architecture/\",\"collapsed\":true,\"items\":[{\"text\":\"Overview\",\"link\":\"/contributor-guide/architecture/overview\"},{\"text\":\"Worker protocol\",\"link\":\"/contributor-guide/architecture/worker-protocol\"},{\"text\":\"State & history intake\",\"link\":\"/contributor-guide/architecture/state-and-history\"},{\"text\":\"Detector contract\",\"link\":\"/contributor-guide/architecture/detector-contract\"},{\"text\":\"Impact estimation\",\"link\":\"/contributor-guide/architecture/impact-estimation\"},{\"text\":\"Widget rendering\",\"link\":\"/contributor-guide/architecture/widget-rendering\"},{\"text\":\"Board widgets\",\"link\":\"/contributor-guide/architecture/board-widgets\"},{\"text\":\"Drill-down\",\"link\":\"/contributor-guide/architecture/drill-down\"}]},{\"text\":\"Testing & verification\",\"link\":\"/contributor-guide/testing\"},{\"text\":\"Contributing\",\"link\":\"/contributor-guide/contributing\"}]}]},\"search\":{\"provider\":\"local\"}},\"locales\":{},\"scrollOffset\":134,\"cleanUrls\":false}");</script>
+    <div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="/docs/" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="/docs/favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/tuning-reference/" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Contributor Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/development-setup.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Development setup</p><!--]--></a><!----></div><!----></div><section class="VPSidebarItem level-1 collapsible collapsed is-link has-active" data-v-b3fd67f8><div class="item" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/" data-v-b3fd67f8><!--[--><h3 class="text" data-v-b3fd67f8>Architecture</h3><!--]--></a><div class="caret" role="button" aria-label="toggle section" tabindex="0" data-v-b3fd67f8><span class="vpi-chevron-right caret-icon" data-v-b3fd67f8></span></div></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/overview.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Overview</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/worker-protocol.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Worker protocol</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/state-and-history.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>State & history intake</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/detector-contract.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Detector contract</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/impact-estimation.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Impact estimation</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/widget-rendering.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Widget rendering</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/board-widgets.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Board widgets</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-2 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/architecture/drill-down.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Drill-down</p><!--]--></a><!----></div><!----></div><!--]--></div></section><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/testing.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Testing & verification</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/contributor-guide/contributing.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Contributing</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_contributor-guide_architecture_board-widgets" data-v-39a288b8><div><h1 id="board-widgets" tabindex="-1">Board widgets <a class="header-anchor" href="#board-widgets" aria-label="Permalink to &quot;Board widgets&quot;">​</a></h1><h2 id="board-widgets-beyond-the-fixed-six" tabindex="-1">Beyond the fixed six <a class="header-anchor" href="#board-widgets-beyond-the-fixed-six" aria-label="Permalink to &quot;Beyond the fixed six {#board-widgets-beyond-the-fixed-six}&quot;">​</a></h2><p>(Components below live in <code>src/view/widgets/</code>, one file per widget name, e.g. <code>JobFailures.tsx</code>, <code>MemoryUtilization.tsx</code>.)</p><p>Six extra cards render in the Findings tab&#39;s active grid alongside the fixed spec §5 six: Incomplete Run, Job Failures, Caching Opportunities, Config Audit, Plan Advisor, and Autoscaling Churn, all <code>region: &#39;action&#39;</code> in <code>detector-registry.tsx</code>:</p><ul><li><p><strong>Incomplete Run</strong> (tag <code>INCMP</code>, <code>IncompleteRun.tsx</code>): app-level &quot;the capture never finished&quot; caveat (DETECTORS entry <code>incompleteRun</code>). Fires whenever <code>app.startTime</code> was observed but <code>app.endTime</code> was not, i.e. no <code>SparkListenerApplicationEnd</code> in the log: an in-flight job, a rotated-away log, or a capture cut short. Order 5, ahead of every other detector, so its card is first among the Findings tab&#39;s <code>action</code>-region active widgets when present. The Findings tab&#39;s recommendation rollup (rendered by <code>FixTheseFirst.tsx</code>) excludes <code>incompleteRun</code> from that list outright (see <a href="./widget-rendering.html#widget-rendering-order-fixed-spec-§5">Widget rendering order</a>): it&#39;s a pipeline-completeness caveat, not an addressable fix, so it never competes with other findings for a ranked slot there. Self-gates to <code>null</code> (no card in the DOM) once the run completed normally. Unrelated to <code>evidence-availability.ts</code>&#39;s own <code>trustworthy</code> gate (see <a href="./worker-protocol.html#evidence-availability-contract-v1">Evidence-availability contract</a>): that ledger only downgrades <em>absence</em> conclusions for individual evidence categories, never becomes a <code>DETECTORS</code> finding itself, and this card does not read it. No <code>docAnchor</code> is set: this is a tool-specific signal with no upstream <code>spark-tuning-reference</code> section.</p></li><li><p><strong>Job Failures</strong> (tag <code>JOBS</code>): app-level job-failure-rate rollup (DETECTORS entry <code>jobFailureRate</code> in <code>packages/core/src/detectors.ts</code>), computed from <code>app.jobs</code> (<code>SparkListenerJobEnd</code> results): ≥10% info, ≥30% warning, ≥50% critical. Complements the per-stage, task-level Failed Tasks card (tag <code>FAIL</code>).</p></li><li><p><strong>Caching Opportunities</strong> (tag <code>CACHE</code>, <code>CachingOpportunity.tsx</code>): app-level SQL relation-reuse detector (DETECTORS entry <code>cachingOpportunity</code>), computed from <code>ctx.sql</code>. It walks each execution&#39;s <code>planTree</code> and keys every scan by its stable pre-AQE identity via <code>scanRelationId</code> (<code>plan-summary.ts</code>): <code>parquet:&lt;db.table&gt;</code>, <code>delta:&lt;db.table&gt;</code>, <code>jdbc:&lt;schema.table&gt;</code>. That dedupes relations within one execution (self-joins count once) and flags any relation scanned by <code>&gt;= minExecutions</code> (2) distinct executions. Relation identity comes from the catalog-qualified scan name (nodeName, e.g. <code>Scan parquet spark_catalog.db.t</code>), not the <code>Location:</code> path, since Spark truncates that path at ~100 chars and points it at <code>_delta_log</code> for Delta tables; internal Delta-log metadata scans are dropped. Read bytes come from the scan&#39;s <code>size of files read</code> metric (<code>0</code>/unknown for JDBC). Findings carry <code>executionReuse</code> (<code>value</code> = distinct-execution count), <code>relation</code>, <code>format</code>, <code>executionIds</code>, <code>totalReadBytes</code>, and a size-aware <code>recommendation</code> (<code>confidence</code> scaled <code>low</code>/<code>medium</code>/<code>high</code> via <code>cachingReuseConfidence</code> off reuse-execution count, plus <code>validationRequired</code>). Renders nothing when clean (no card in the DOM). One row per relation: name + format badge, reuse count, <code>Data read</code> (<code>formatBytes</code>, em-dash when unknown), recommendation, sorted by <code>totalReadBytes</code> then reuse count descending. Rows reveal 6 initially, then up to 30 more per click. Pure-RDD-API apps (no SQL executions) produce no finding: a deliberate trade-off replacing the former RDD-lineage heuristic, which surfaced only internal query-engine RDDs on DataFrame/SQL workloads.</p><p>It also detects composite reuse. When the same join/union subtree (not just a leaf scan) recurs across <code>&gt;= minExecutions</code> distinct SQL executions, the detector emits one <code>variant:&#39;composite&#39;</code> finding recommending caching the derived join/union result instead of two independent leaf-relation rows, and suppresses the leaf findings for relations fully covered by it. A relation reused beyond the composite&#39;s executions keeps a residual leaf finding for just the uncovered executions. Composite identity is structural: an anchor plan-shape fingerprint (<code>findCompositeCandidates</code>/<code>computePlanShapes</code>&#39;s <code>opts.includeDetail</code> path, <code>detectors.ts</code>) folds the join/union node&#39;s own normalized <code>detail</code> (join type, columns, literals, with expr ids, <code>plan_id=</code>, codegen-stage numbers, and AQE&#39;s BuildLeft/BuildRight stripped, and commutative equality operands canonicalized) plus its children&#39;s detail-free shapes. Descendant nodes never contribute detail text, only structural shape, so a shared join reused across differently-filtered pre-join scans still matches (a deliberate trade-off) while a different join <em>condition</em> on the same tables does not collide. Nested composites (an inner join reused both standalone and inside an outer join) dedupe one level at a time: an inner composite fully covered by a qualifying outer composite&#39;s execution set is suppressed entirely, and a superset recomputes a residual finding over just its extra executions. Reuse via pure projection/aggregation with no join/union underneath stays leaf-level, not modeled as composite. <code>variant:&#39;composite&#39;</code> findings carry <code>operator</code> (<code>&#39;join&#39;</code>|<code>&#39;union&#39;</code>), <code>relations</code> (leaf relations under the composite, for display), and <code>format:&#39;derived&#39;</code> (a sentinel: composites have no scan storage format). They render a <code>JOIN</code>/<code>UNION</code> operator badge (styled distinct from the format badges) plus a confidence marker (<code>confidence</code> scaled via <code>cachingReuseConfidence</code>, <code>validationRequired</code>) in <code>CachingOpportunity.tsx</code>.</p></li><li><p><strong>Autoscaling Churn</strong> (tag <code>CHRN</code>, <code>AutoscalingChurn.tsx</code>): app-level short-lived-executor detector (DETECTORS entry <code>autoscalingChurn</code> in <code>packages/core/src/detectors.ts</code>; reuses the same <code>executorsAdded</code>/<code>executorsRemoved</code> matching logic as <code>utilization</code>). For each added executor, finds its matching removal event (falling back to <code>app.endTime</code> for an executor still alive when the log ends) and flags it short-lived if its lifetime is under 2 minutes (<code>thresholds.shortLivedMs</code>). Warns above 30% short-lived, escalates to critical above 60% (<code>confidence</code> scales <code>low</code>/<code>medium</code>/<code>high</code> via <code>autoscalingChurnConfidence</code>, off how far the short-lived share sits past <code>warningPct</code>/<code>criticalPct</code>: these thresholds are still an unvalidated design-spike estimate, not yet checked against real autoscaling-heavy logs). Returns no finding below 5 total executors (noise floor) or when <code>app.endTime</code> is missing (truncated/still-running log). The widget itself is unchanged apart from a finding-driven verdict banner (impact dot + <code>CHRN</code> tag + recommendation) above its existing add/remove chart. The chart&#39;s muted scale-down bar coloring is untouched. Its <code>docAnchor</code> is <code>#bottleneck-autoscaling-churn</code>, a section of the cluster-config chapter (<code>docs-content/chapters/11-cluster-config.md</code>), not a bottleneck page of its own: <code>pageForAnchor</code> maps it there, so the <code>CHRN</code> pill opens <code>cluster-config.html#bottleneck-autoscaling-churn</code>.</p></li><li><p><strong>Config Audit</strong> (tag <code>CFG</code>): static Spark-config sanity findings derived from <code>app.config</code>/<code>app.resources</code> (parsed from <code>SparkListenerEnvironmentUpdate</code>): dynamic-allocation vs. shuffle-service mismatch, inverted/missing autoscaling bounds, non-Kryo serializer, low executor <code>memoryOverhead</code>. Computed outside the runtime bottleneck catalog: its findings live in the separate <code>configFindings</code> stream, not <code>catalog</code>. But <code>FixTheseFirst</code>/<code>Alerts</code> both merge <code>catalog</code> and <code>configFindings</code> (neither reads <code>region</code> any more), so a Config Audit finding is still eligible for a row in the Findings tab&#39;s recommendation rollup and still gets its own card in the active grid, alongside genuine bottleneck-catalog findings.</p></li><li><p><strong>Plan Advisor</strong> (tag <code>PLAN</code>): SQL-plan-level findings computed from <code>appModel.sql</code>&#39;s resolved <code>planTree</code> (DETECTORS entries <code>duplicatePlanSubtree</code>, <code>smallFiles</code>, <code>broadcastSizing</code> in <code>packages/core/src/detectors.ts</code>): repeated plan subtrees (≥3 nodes, ≥2 occurrences, critical if the repeated root is an <code>Exchange</code>), small-files read/write (&gt;100 files averaging ❤️ MiB), and broadcast-join sizing in both directions (missed- broadcast info finding, over-broadcast warning at &gt;1 GB). Each of the four emitted types now renders as its own card (<code>DuplicatePlanSubtree.tsx</code>, <code>SmallFiles.tsx</code>, <code>UnderBroadcast.tsx</code>, <code>OverBroadcast.tsx</code>; the 2026-09 widget/finding-type 1:1 mapping redesign split what used to be one shared <code>PlanFindings.tsx</code> card): all four still share the <code>PLAN</code> tag and the same <code>--plan-aggregate</code> badge tint (<code>src/view/plan-finding-shared.ts</code>). Their <code>docAnchor</code>s (<code>#bottleneck-duplicate-plan-subtree</code>, <code>#bottleneck-small-files</code>, <code>#bottleneck-broadcast-sizing</code>) each resolve to their own page in the vendored tuning reference.</p></li></ul><p><code>detector-registry.tsx</code>&#39;s <code>REGISTRY</code> still carries <code>region: &#39;reference&#39;</code> on four entries, each its own component now: <code>memoryUtilization</code> (<code>MemoryUtilization</code>), <code>utilization</code> (<code>ExecutorUtilization</code>), <code>cacheUtilization</code> (<code>CacheUtilization</code>), and <code>coreLocality</code> (<code>CoreUsageArea</code>). <code>broadcastSizing</code> is gone from <code>REGISTRY</code> entirely (the 2026-09 widget/finding-type 1:1 mapping redesign dropped it): it never backed a real <code>Finding</code> (the <code>broadcastSizing</code> <code>DETECTORS</code> entry only ever emits <code>underBroadcast</code>/<code>overBroadcast</code>, both <code>region: &#39;action&#39;</code>, each now its own Plan Advisor card), so there is no key left for it to occupy or a clean-check line for it to render. Autoscaling Churn, the other executor-provisioning-lifecycle detector alongside <code>utilization</code>/<code>memoryUtilization</code>, was deliberately given <code>action</code> rather than <code>reference</code> (see &quot;Beyond the fixed six&quot; above).</p><p><code>region</code> decides one thing (see <a href="./widget-rendering.html#widget-rendering-order-fixed-spec-§5">Widget rendering order</a>): whether a widget always mounts. <code>isAlwaysMountedType()</code> flags exactly one of the four <code>reference</code>-region types: Core Usage by Locality. That one mounts unconditionally from <code>appModel</code> in its own small grid inside the Findings tab, regardless of finding state. Memory Utilization, Executor Utilization, and Cache Storage are <code>reference</code> too, but all three are excluded by product decision, not a component-sharing constraint (<code>ALWAYS_MOUNTED_EXCEPTIONS</code> in <code>src/view/detector-registry.tsx</code> carries <code>cacheUtilization</code>, <code>memoryUtilization</code>, and <code>utilization</code>): a clean run on any of them isn&#39;t evidence worth surfacing unconditionally, so each collapses to an ordinary <code>CleanCheckRow</code> like any other action-region type on a clean run. Every other <code>REGISTRY</code> widget still renders unconditionally as either an active card or a clean-check line, and <code>region</code> still sets <code>orderedWidgets()</code>&#39;s sort order within the active grid (<code>action</code> components first, <code>reference</code> ones after). The Full app report tab is structural-only and reads no <code>REGISTRY</code> entry. So Core Usage by Locality, the one widget still tagged <code>region: &#39;reference&#39;</code> and exempt from <code>ALWAYS_MOUNTED_EXCEPTIONS</code>, renders in the Findings tab&#39;s always-visible grid rather than inside a separate Reference region (Memory Utilization, Executor Utilization, and Cache Storage all render through the ordinary active/clean paths instead):</p><ul><li><strong>Memory Utilization</strong> (tag <code>MEM</code>): app-level card combining three sub-findings from the <code>memoryUtilization</code> DETECTORS entry (<code>packages/core/src/detectors.ts</code>): idle-cores rate (busy-core-time from the worker&#39;s run-aggregates sweep vs. peak-cores × wall-clock), per-executor memory bands (peak heap vs. allocated, gated on <code>spark.eventLog.logStageExecutorMetrics</code>: a distinct <code>dataUnavailable</code> finding renders when that config was off), and an unverified memory-waste model (<code>confidence</code> scales <code>low</code>/<code>medium</code>/<code>high</code> via <code>memoryWasteConfidence</code>, off how far the wasted/used ratio sits past the 1.5× buffer). The driver-memory half of the original spec is dropped: the worker only extracts <em>allocated</em> <code>spark.driver.memory</code>, never a driver actual-usage metric, so there is nothing to band against. The separate <code>utilization</code> DETECTORS entry (an <code>avgUtilization</code> info finding: active-executor-time fraction below 60%) has its own card, <strong>Executor Utilization</strong> (tag <code>UTIL</code>, <code>ExecutorUtilization.tsx</code>): a <code>reference</code>-region widget in its own right that renders only with an active finding, split out of this same combined widget in the 2026-09 widget/finding-type 1:1 mapping redesign (it used to render inline here; the former <code>ExecutorTimeline.tsx</code> never owned this type, despite the tag&#39;s letters).</li><li><strong>Cache Storage</strong> (tag <code>CSTOR</code>): app-level card driven by the <code>cacheUtilization</code> DETECTORS entry (<code>packages/core/src/detectors.ts</code>), evaluating two per-RDD proxies over <code>ctx.app.rddInfo</code> storage snapshots since Spark event logs carry no runtime block-access/read-count data: partial caching (<code>numCachedPartitions / numPartitions &lt; 0.90</code>, <code>&lt; 0.50</code> for the warning tier) and disk spillover for <code>MEMORY_AND_DISK*</code> RDDs (<code>diskSize / (memorySize + diskSize) &gt; 0.15</code>, <code>&gt; 0.40</code> for the warning tier; <code>DISK_ONLY</code> RDDs are never flagged). <code>confidence</code> scales <code>low</code>/<code>medium</code>/<code>high</code> via <code>cacheSampleConfidence(rdd.numPartitions)</code>, because the ratio is a point-in-time storage snapshot from stage-submission events, not a runtime read-count, and more partitions average that snapshot noise into a more stable ratio. The existing RDD table (ported from the legacy <code>src/widgets/cache-utilization.js</code> canvas widget) still renders unconditionally; flagged rows get an inline <code>CSTOR</code> tag next to the RDD name, and every flagged RDD&#39;s recommendation renders below the table, worst-first.</li><li><strong>Core Usage by Locality</strong> (tag <code>LOCAL</code>, <code>CoreUsageArea.tsx</code>): app-level non-local-task-ratio threshold (DETECTORS entry <code>coreLocality</code>; the other half of a wasted-cores-ratio check, the idle-core half already covered by Memory Utilization&#39;s <code>idleCores</code> variant above). Sums <code>RACK_LOCAL</code> + <code>ANY</code> task counts against total task count across every stage&#39;s <code>stage.localityStats</code> (pure reducer <code>computeCoreLocalityRatio</code>, <code>packages/core/src/core-locality-ratio.ts</code>). <code>NO_PREF</code> stays in the denominator only, since it&#39;s what shuffle-read stages legitimately report with no locality problem. Below 50 total tasks or below a 15% non-local ratio: no finding; 15%-35%: warning; &gt;= 35%: critical, still unvalidated design-spike thresholds (<code>confidence</code> scales <code>low</code>/<code>medium</code>/<code>high</code> via <code>coreLocalityConfidence</code>, off whichever is weaker of the non-local ratio and the sampled task count, the same evidence-strength convention as the memory-waste model above). The widget&#39;s always-rendered stacked-area chart (<code>packages/core/src/core-usage-locality.ts</code>) is unaffected. The finding only adds a threshold/impact-band section above it, a per-stage non-local breakdown below it (shown whenever any non-local tasks exist at all, independent of whether the aggregate crossed threshold), and a one-line cross-reference to Memory Utilization when its <code>idleCores</code> finding also fired (idle-core ratio itself is not duplicated here).</li></ul><p>Documented ALL-CAPS tag vocabulary: <code>SKEW</code>, <code>SHFL</code>, <code>SPILL</code>, <code>GC</code>, <code>COLD</code>, <code>UTIL</code>, <code>MEM</code> (Memory Utilization), <code>CSTOR</code> (Cache Storage), <code>LOCAL</code> (Core Usage by Locality), plus <code>FAIL</code> (Failed Tasks), <code>JOBS</code> (Job Failures), <code>CFG</code> (Config Audit), <code>PLAN</code> (Plan Advisor), <code>SFAIL</code> (stage failed outright), <code>PART</code> (partition sizing), <code>SLOW</code> (stage overall slowness), <code>SHAPE</code> (stage shape smells), <code>CACHE</code> (caching opportunity) and <code>CHRN</code> (autoscaling churn).</p><p>ETL Phase Attribution (<code>packages/core/src/etl-phases.ts</code> + <code>EtlPhases.tsx</code>), What-If Executor Scaling (<code>packages/core/src/scaling-sim.ts</code> + <code>ScalingSim.tsx</code>, design spike: makespan predictions are unvalidated and carry a Model Error indicator), and Compute Efficiency (<code>packages/core/src/efficiency-model.ts</code> + <code>EfficiencyModel.tsx</code>, design spike: driver-vs-executor waste split, two theoretical floors, and the §6 right-sizing copy) are main-thread report modules, not <code>DETECTORS</code> entries: descriptive lenses with no impact-band threshold, rendered unconditionally into the Full app report tab rather than participating in the bottleneck catalog. <code>packages/core/src/job-groups.ts</code> (<code>checkConcurrentJobGroups</code>) is likewise a report helper: it flags when concurrent SQL-execution job groups make wall-clock-based estimates unreliable, and both the scaling simulator and the efficiency model gate their precision on it, showing a caveat banner rather than suppressing the estimate.</p><h2 id="confidence-metadata" tabindex="-1">Confidence metadata <a class="header-anchor" href="#confidence-metadata" aria-label="Permalink to &quot;Confidence metadata&quot;">​</a></h2><p>Best-effort (non-deterministic) findings carry a <code>confidence</code> + <code>validationRequired</code> marker, rendered inline by each consuming widget (e.g. <code>Spill.tsx</code>, <code>DuplicatePlanSubtree.tsx</code>, <code>SmallFiles.tsx</code>, <code>UnderBroadcast.tsx</code>, <code>OverBroadcast.tsx</code>, <code>EfficiencyModel.tsx</code>) when <code>confidence</code> is present and not <code>&#39;high&#39;</code>. There is no shared helper: each widget renders its own muted &quot;N confidence: verify&quot; text, with the <code>title</code> attribute carrying <code>validationRequired</code> as a tooltip. Spill classification is <code>medium</code> when classified (skew/volume) and <code>low</code> when unclassified; plan-summary warnings are <code>low</code>. Deterministic detectors stay unmarked, treated as high confidence.</p><h2 id="visual-system" tabindex="-1">Visual system <a class="header-anchor" href="#visual-system" aria-label="Permalink to &quot;Visual system&quot;">​</a></h2><p>Tailwind CSS v4 + shadcn/ui (Base UI primitives). The design tokens (colors, including the telemetry-console dark palette) are defined as CSS variables in an <code>@theme inline</code> block and consumed via Tailwind utility classes; there is no more hand-authored BEM CSS. Theme polarity is unchanged from the legacy app: dark = bare <code>:root</code> (no attribute), light = <code>:root[data-theme=&quot;light&quot;]</code>, toggled by <code>src/theme/ThemeProvider.tsx</code> and persisted to <code>localStorage</code>. <code>index.html</code>&#39;s inline FOUC-prevention script still runs before React mounts. Mono is still the typeface for numerics (<code>--font-mono</code>), applied via a <code>.num</code>-equivalent Tailwind utility per call site rather than one global class. Charts are Recharts (<code>src/view/charts/ChartTheme.tsx</code>&#39;s <code>CHART_COLORS</code>, read from the same CSS tokens) instead of Chart.js. React re-renders on theme toggle, so chart colors update live with the theme, unlike the old Chart.js canvases, which were built once per parse. <code>ChartFrame</code> can also expose the underlying rows through a toggleable accessible table and copy them to the clipboard as TSV.</p><h2 id="templating-xss-safe" tabindex="-1">Templating (XSS-safe) <a class="header-anchor" href="#templating-xss-safe" aria-label="Permalink to &quot;Templating (XSS-safe)&quot;">​</a></h2><p>JSX auto-escapes every interpolated value by default. The hand-rolled auto-escaping <code> html</code> `` tagged template (<code>src/widgets/utils.js</code>) and its <code>no-raw-innerhtml</code> guard test are gone, and no <code>.innerHTML</code> assignment is left anywhere in the view layer.</p><p>One Base UI-specific gotcha carried no equivalent in the legacy app: <code>WidgetCard.tsx</code> passes <code>aria-expanded={String(open) as &#39;true&#39; | &#39;false&#39;}</code> rather than the raw boolean, because Base UI&#39;s <code>Collapsible.Trigger</code> otherwise overrides a boolean <code>aria-expanded</code> prop with its own internal state via prop merging. The explicit <code>String()</code> cast keeps the rendered attribute in sync with this app&#39;s own <code>open</code> state.</p></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><a class="VPLink link pager-link prev" href="/docs/contributor-guide/architecture/widget-rendering.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Previous page</span><span class="title" data-v-e257564d>Widget rendering</span><!--]--></a></div><div class="pager" data-v-e257564d><a class="VPLink link pager-link next" href="/docs/contributor-guide/architecture/drill-down.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Next page</span><span class="title" data-v-e257564d>Drill-down</span><!--]--></a></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
+    <script>window.__VP_HASH_MAP__=JSON.parse("{\"contributor-guide_architecture_board-widgets.md\":\"9ToZXcTk\",\"contributor-guide_architecture_detector-contract.md\":\"aQQfP7pj\",\"contributor-guide_architecture_drill-down.md\":\"Bawx6NgP\",\"contributor-guide_architecture_impact-estimation.md\":\"sf9DU2bA\",\"contributor-guide_architecture_index.md\":\"3TO9ic6w\",\"contributor-guide_architecture_overview.md\":\"BXQdgG5u\",\"contributor-guide_architecture_state-and-history.md\":\"C-xxn0q7\",\"contributor-guide_architecture_widget-rendering.md\":\"DYwFeyjV\",\"contributor-guide_architecture_worker-protocol.md\":\"P7d1vtEJ\",\"contributor-guide_contributing.md\":\"CvRsdr6J\",\"contributor-guide_development-setup.md\":\"DvMV1xhQ\",\"contributor-guide_testing.md\":\"BU6vyA-9\",\"index.md\":\"CHJVslga\",\"tuning-reference_anti-patterns.md\":\"Df1YMIHu\",\"tuning-reference_aqe.md\":\"BIsCtLzm\",\"tuning-reference_bottleneck-broadcast-sizing.md\":\"CEstB3Ia\",\"tuning-reference_bottleneck-cold-start.md\":\"CEuy-72y\",\"tuning-reference_bottleneck-duplicate-plan-subtree.md\":\"CIohQDfn\",\"tuning-reference_bottleneck-failures.md\":\"CAsFf3RR\",\"tuning-reference_bottleneck-gc.md\":\"B4T3kJYV\",\"tuning-reference_bottleneck-job-failure-rate.md\":\"BaJl__1W\",\"tuning-reference_bottleneck-memory-utilization.md\":\"DbP-SJZc\",\"tuning-reference_bottleneck-retry-waste.md\":\"D5JMjVOt\",\"tuning-reference_bottleneck-shuffle.md\":\"DDMQVOW6\",\"tuning-reference_bottleneck-skew.md\":\"B0MzhWcX\",\"tuning-reference_bottleneck-slow-host.md\":\"BlIo6UDW\",\"tuning-reference_bottleneck-small-files.md\":\"DHGMTmPs\",\"tuning-reference_bottleneck-spill.md\":\"PNH7mITt\",\"tuning-reference_bottleneck-straggler.md\":\"DIbXdAcQ\",\"tuning-reference_bottleneck-tiny-tasks.md\":\"CB05v3zh\",\"tuning-reference_bottleneck-utilization.md\":\"DqC_iGkw\",\"tuning-reference_caching.md\":\"B7aQ8asB\",\"tuning-reference_cluster-config.md\":\"CD6vzYtD\",\"tuning-reference_config.md\":\"UvveiWG3\",\"tuning-reference_data-formats.md\":\"bjCAWH3N\",\"tuning-reference_index.md\":\"BQ_NooMV\",\"tuning-reference_intro.md\":\"CobD-lGB\",\"tuning-reference_joins.md\":\"BtKs_CuW\",\"tuning-reference_memory-model.md\":\"CWhvgevP\",\"tuning-reference_metrics.md\":\"mLOh7Apj\",\"tuning-reference_partitioning.md\":\"q0zKF_8X\",\"tuning-reference_pyspark.md\":\"DDCfvN9t\",\"tuning-reference_shuffle.md\":\"BZZ7R4Ix\",\"tuning-reference_spark-architecture.md\":\"Dwzm5avO\",\"tuning-reference_table-formats.md\":\"D6wj-2dX\",\"user-guide_alternative-log-retrieval.md\":\"B4tPGIal\",\"user-guide_getting-started.md\":\"BJvwLEIM\",\"user-guide_mcp-tools.md\":\"DGvolGzt\",\"user-guide_run-comparison.md\":\"CQc1aoU8\",\"user-guide_understanding-findings.md\":\"B29f0ucS\"}");window.__VP_SITE_DATA__=JSON.parse("{\"lang\":\"en-US\",\"dir\":\"ltr\",\"title\":\"SparkForensics\",\"description\":\"Docs for using and contributing to SparkForensics\",\"base\":\"/docs/\",\"head\":[],\"router\":{\"prefetchLinks\":true},\"appearance\":true,\"themeConfig\":{\"logo\":\"/favicon.svg\",\"nav\":[{\"text\":\"User Guide\",\"link\":\"/user-guide/getting-started\"},{\"text\":\"Contributor Guide\",\"link\":\"/contributor-guide/development-setup\"},{\"text\":\"Tuning Reference\",\"link\":\"/tuning-reference/\"}],\"sidebar\":{\"/tuning-reference/\":[{\"text\":\"Getting Started\",\"items\":[{\"text\":\"Introduction\",\"link\":\"/tuning-reference/intro\"}]},{\"text\":\"Optimization Guide\",\"items\":[{\"text\":\"Spark Execution Model\",\"link\":\"/tuning-reference/spark-architecture\"},{\"text\":\"Memory Management\",\"link\":\"/tuning-reference/memory-model\"},{\"text\":\"Partitioning\",\"link\":\"/tuning-reference/partitioning\"},{\"text\":\"Join Optimization\",\"link\":\"/tuning-reference/joins\"},{\"text\":\"Shuffle\",\"link\":\"/tuning-reference/shuffle\"},{\"text\":\"Data Formats\",\"link\":\"/tuning-reference/data-formats\"},{\"text\":\"Table Formats\",\"link\":\"/tuning-reference/table-formats\"},{\"text\":\"Caching & Persistence\",\"link\":\"/tuning-reference/caching\"},{\"text\":\"PySpark Specifics\",\"link\":\"/tuning-reference/pyspark\"},{\"text\":\"Adaptive Query Execution\",\"link\":\"/tuning-reference/aqe\"},{\"text\":\"Cluster Tuning\",\"link\":\"/tuning-reference/cluster-config\"},{\"text\":\"Anti-Patterns\",\"link\":\"/tuning-reference/anti-patterns\"}]},{\"text\":\"Detector Catalog\",\"items\":[{\"text\":\"Task Skew\",\"link\":\"/tuning-reference/bottleneck-skew\"},{\"text\":\"Shuffle I/O\",\"link\":\"/tuning-reference/bottleneck-shuffle\"},{\"text\":\"Memory / Disk Spill\",\"link\":\"/tuning-reference/bottleneck-spill\"},{\"text\":\"GC Pressure\",\"link\":\"/tuning-reference/bottleneck-gc\"},{\"text\":\"Cold Start\",\"link\":\"/tuning-reference/bottleneck-cold-start\"},{\"text\":\"Executor Utilization\",\"link\":\"/tuning-reference/bottleneck-utilization\"},{\"text\":\"Slow Host\",\"link\":\"/tuning-reference/bottleneck-slow-host\"},{\"text\":\"Task Failures\",\"link\":\"/tuning-reference/bottleneck-failures\"},{\"text\":\"Stragglers\",\"link\":\"/tuning-reference/bottleneck-straggler\"},{\"text\":\"Retry Waste\",\"link\":\"/tuning-reference/bottleneck-retry-waste\"},{\"text\":\"Tiny Tasks\",\"link\":\"/tuning-reference/bottleneck-tiny-tasks\"},{\"text\":\"Job Failure Rate\",\"link\":\"/tuning-reference/bottleneck-job-failure-rate\"},{\"text\":\"Memory Utilization\",\"link\":\"/tuning-reference/bottleneck-memory-utilization\"},{\"text\":\"Duplicate Plan Subtree\",\"link\":\"/tuning-reference/bottleneck-duplicate-plan-subtree\"},{\"text\":\"Small Files\",\"link\":\"/tuning-reference/bottleneck-small-files\"},{\"text\":\"Broadcast Sizing\",\"link\":\"/tuning-reference/bottleneck-broadcast-sizing\"}]},{\"text\":\"Reference\",\"items\":[{\"text\":\"Metrics Glossary\",\"link\":\"/tuning-reference/metrics\"},{\"text\":\"Spark Config Quick-Reference\",\"link\":\"/tuning-reference/config\"}]}],\"/user-guide/\":[{\"text\":\"User Guide\",\"items\":[{\"text\":\"Getting started\",\"link\":\"/user-guide/getting-started\"},{\"text\":\"Understanding findings\",\"link\":\"/user-guide/understanding-findings\"},{\"text\":\"Run comparison mode\",\"link\":\"/user-guide/run-comparison\"},{\"text\":\"MCP tools reference\",\"link\":\"/user-guide/mcp-tools\"},{\"text\":\"Alternative log retrieval\",\"link\":\"/user-guide/alternative-log-retrieval\"}]}],\"/contributor-guide/\":[{\"text\":\"Contributor Guide\",\"items\":[{\"text\":\"Development setup\",\"link\":\"/contributor-guide/development-setup\"},{\"text\":\"Architecture\",\"link\":\"/contributor-guide/architecture/\",\"collapsed\":true,\"items\":[{\"text\":\"Overview\",\"link\":\"/contributor-guide/architecture/overview\"},{\"text\":\"Worker protocol\",\"link\":\"/contributor-guide/architecture/worker-protocol\"},{\"text\":\"State & history intake\",\"link\":\"/contributor-guide/architecture/state-and-history\"},{\"text\":\"Detector contract\",\"link\":\"/contributor-guide/architecture/detector-contract\"},{\"text\":\"Impact estimation\",\"link\":\"/contributor-guide/architecture/impact-estimation\"},{\"text\":\"Widget rendering\",\"link\":\"/contributor-guide/architecture/widget-rendering\"},{\"text\":\"Board widgets\",\"link\":\"/contributor-guide/architecture/board-widgets\"},{\"text\":\"Drill-down\",\"link\":\"/contributor-guide/architecture/drill-down\"}]},{\"text\":\"Testing & verification\",\"link\":\"/contributor-guide/testing\"},{\"text\":\"Contributing\",\"link\":\"/contributor-guide/contributing\"}]}]},\"search\":{\"provider\":\"local\"}},\"locales\":{},\"scrollOffset\":134,\"cleanUrls\":false}");</script>
     
   </body>
 </html>