sparkforensics-server 0.2.0 → 0.2.3

package/public/docs/assets/{tuning-reference_bottleneck-failures.md.4z5BXGJ2.js → tuning-reference_bottleneck-failures.md.CAsFf3RR.js}

@@ -1,5 +1,5 @@
 import{_ as a,a as t}from"./chunks/retry-escalation-ladder.dark.DHipdJgZ.js";import{_ as o,o as s,c as r,a5 as i}from"./chunks/framework.DSg0KOwT.js";const k=JSON.parse('{"title":"Task Failures","description":"","frontmatter":{"title":"Task Failures"},"headers":[],"relativePath":"tuning-reference/bottleneck-failures.md","filePath":"tuning-reference/bottleneck-failures.md"}'),n={name:"tuning-reference/bottleneck-failures.md"};function l(c,e,d,f,h,u){return s(),r("div",null,[...e[0]||(e[0]=[i(`<h1 id="bottleneck-failures" tabindex="-1">Task Failures <a class="header-anchor" href="#bottleneck-failures" aria-label="Permalink to &quot;Task Failures {#bottleneck-failures}&quot;">​</a></h1><p><span class="tag">FAIL</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>A task failure is any task end whose <code>Reason</code> field is something other than success. Spark&#39;s event log serializes this as the formatted class name of whichever <code>TaskEndReason</code> instance was assigned to that task attempt<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>, and the canonical set of reasons is <code>Success</code>, <code>FetchFailed</code>, <code>ExceptionFailure</code>, <code>TaskResultLost</code>, <code>TaskKilled</code>, <code>TaskCommitDenied</code>, <code>ExecutorLostFailure</code>, <code>Resubmitted</code>, and <code>UnknownReason</code><sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>. Each carries its own extra detail: a <code>FetchFailed</code> records a reduce ID and message; an <code>ExceptionFailure</code> carries the exception&#39;s class name, description, and stack trace, plus accumulator updates (falling back to reading them out of the task&#39;s metrics object for logs written by Spark 1.x); a <code>TaskKilled</code> records a kill reason; a <code>TaskCommitDenied</code> records the job ID, partition ID, and attempt number; an <code>ExecutorLostFailure</code> records whether the loss was caused by the application, the executor ID, and a loss reason; <code>UnknownReason</code> carries no extra fields<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup>.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><table tabindex="0"><thead><tr><th>failed tasks (share)</th><th>Level</th></tr></thead><tbody><tr><td>&gt; 5%</td><td>Warning</td></tr><tr><td>&gt; 20%</td><td>Critical</td></tr></tbody></table><p>The task-metrics object attached to a <code>SparkListenerTaskEnd</code> event is optional in the schema, so it isn&#39;t guaranteed to be present for every failure. It tends to be populated for reasons where the attempt actually ran far enough to produce metrics (an <code>ExceptionFailure</code> or <code>TaskKilled</code>, for instance) but can be absent for a <code>Resubmitted</code> attempt that never completed on its executor at all<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup>. Detection has to key off the <code>Reason</code> field itself rather than assume metrics will always be there to inspect.</p><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>Some reasons point at something worse than a one-off hiccup. An <code>ExecutorLostFailure</code> can be the knock-on effect of a container or pod killed for exceeding its memory grant: enabling off-heap memory without also raising <code>spark.executor.memoryOverhead</code> (or setting <code>spark.executor.pyspark.memory</code> explicitly) lets real usage exceed what YARN or Kubernetes allocated, and the executor is killed with a generic container-memory error rather than a Spark-level exception<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. Losing an executor mid-shuffle can also cascade into failures elsewhere: if dynamic allocation removes an executor before a shuffle it wrote data for has completed, that shuffle output is gone, and downstream tasks reading it fail and force an unnecessary recompute<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li>Read the <code>Reason</code> field before treating all failures the same: an <code>ExceptionFailure</code> points at application code, and an <code>ExecutorLostFailure</code> usually points at memory sizing or infrastructure.</li><li>For memory-driven <code>ExecutorLostFailure</code>s, size <code>spark.executor.memoryOverhead</code> (which defaults to <code>max(384 MB, 10% of executor memory)</code>) to actually cover off-heap and PySpark memory, since neither is folded into that default budget automatically<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup>.</li><li>For failures caused by dynamic allocation reclaiming an executor mid-shuffle, enable the external <a href="./shuffle.html">shuffle service</a> or <code>spark.dynamicAllocation.shuffleTracking.enabled</code> so shuffle output can outlive the executor that wrote it, instead of being lost and recomputed<sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup>.</li></ul><p>Config-only levers for memory-driven <code>ExecutorLostFailure</code>s and mid-shuffle executor loss:</p><div class="language-properties vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">properties</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Cover off-heap + PySpark memory the default overhead budget does NOT include.</span></span>
-<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Default is max(384m, 10% of executor memory); 2g is an example, size to real off-heap/PySpark use.</span></span>
+<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Default is max(384m, 10% of executor memory); 2g is an example — size to real off-heap/PySpark use.</span></span>
 <span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.executor.memoryOverhead</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=2g</span></span>
 <span class="line"></span>
 <span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Keep shuffle output alive when dynamic allocation reclaims an executor mid-shuffle</span></span>

package/public/docs/assets/{tuning-reference_bottleneck-gc.md.DSxzZRK7.js → tuning-reference_bottleneck-gc.md.B4T3kJYV.js}

@@ -1,6 +1,6 @@
-import{_ as a,o as t,c as o,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"GC Pressure","description":"","frontmatter":{"title":"GC Pressure"},"headers":[],"relativePath":"tuning-reference/bottleneck-gc.md","filePath":"tuning-reference/bottleneck-gc.md"}'),r={name:"tuning-reference/bottleneck-gc.md"};function i(n,e,l,c,h,f){return t(),o("div",null,[...e[0]||(e[0]=[s(`<h1 id="bottleneck-gc" tabindex="-1">GC Pressure <a class="header-anchor" href="#bottleneck-gc" aria-label="Permalink to &quot;GC Pressure {#bottleneck-gc}&quot;">​</a></h1><p><span class="tag">GC</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>GC pressure describes how much of an executor&#39;s time goes to JVM garbage collection instead of running task code. Spark tracks this directly in its per-task metrics: <code>jvmGCTime</code> sits alongside <code>executorRunTime</code>, the elapsed wall-clock time the executor spent running the task (as opposed to <code>executorCpuTime</code>, which measures CPU time specifically).<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup> <code>jvmGCTime</code> is not additive on top of that duration: it&#39;s defined as the elapsed time the JVM spent in garbage collection <em>while executing the task</em>, so GC pauses fall inside the same wall-clock window <code>executorRunTime</code> already measures, not outside it. Summing the two would double-count the GC pauses and overstate how long the task actually ran.<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup></p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><p>Because <code>jvmGCTime</code> sits inside <code>executorRunTime</code> rather than alongside it, the ratio between the two gives a bounded read on how much of a task&#39;s wall-clock time went to garbage collection:</p><table tabindex="0"><thead><tr><th>gcPct = jvmGCTime / executorRunTime</th><th>Level</th></tr></thead><tbody><tr><td>&gt; 10%</td><td>Warning</td></tr><tr><td>&gt; 20%</td><td>Critical</td></tr></tbody></table><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>As gcPct climbs past these thresholds, a growing share of every task&#39;s wall-clock time is consumed by garbage collection instead of actual computation, so the executor&#39;s useful throughput drops even while it appears busy. Two causes show up repeatedly: oversized executors, and an on-heap memory split pushed too far toward execution/storage.</p><p>Sizing an executor with too many cores relative to its heap is one documented trigger: a fat-executor configuration using all 16 cores of a node was observed to not only hurt HDFS throughput but also &quot;result in excessive garbage [collection].&quot;<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup></p><p>Pushing <code>spark.memory.fraction</code> too far toward its upper end has a similar effect through a different path. Its complement, <code>1 − spark.memory.fraction</code>, is untracked &quot;User Memory&quot; reserved for UDFs, Python/Arrow glue, and native library buffers; Spark doesn&#39;t manage this region at all, and starving it (which is what raising <code>spark.memory.fraction</code> toward something like 0.9 does) leads to &quot;GC pressure or random OOMs,&quot; with no warning before the failure.<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup></p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li>Keep cores per executor down rather than packing an entire node&#39;s cores onto one JVM (the common ~5-cores-per-executor guideline). That avoids the excessive-garbage-collection failure mode documented for fat-executor configurations.<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup></li><li>Don&#39;t raise <code>spark.memory.fraction</code> past a comfortable range chasing more execution/storage memory: doing so starves the untracked user-memory region and risks the same GC-pressure/OOM failure mode.<sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup></li><li>Move Tungsten&#39;s execution and storage buffers off the JVM heap with <code>spark.memory.offHeap.enabled=true</code> (and a positive <code>spark.memory.offHeap.size</code>, which is required whenever off-heap is enabled<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>). Off-heap buffers are invisible to the garbage collector, so fewer and smaller live objects need to be tracked, scanned, and copied on the heap, reducing both the frequency and duration of GC pauses.<sup class="footnote-ref"><a href="#fn3" id="fnref3:2">[3:2]</a></sup></li><li>Tune or switch the GC collector on large heaps. G1GC is Spark&#39;s default collector since 4.0 (which defaults to JDK 17), and large executor heaps may need <code>-XX:G1HeapRegionSize</code> raised as well.<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup> G1&#39;s stop-the-world evacuation and full-GC pauses can still spike under heavy old-generation pressure on large heaps: one documented 88GB-heap benchmark saw a job&#39;s pause spike to nearly 100 seconds under default G1 settings, and only tuning <code>InitiatingHeapOccupancyPercent</code>, <code>ConcGCThreads</code>, and RSet-update parameters brought G1 back ahead of Parallel/CMS on both throughput and latency.<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup> ZGC is a documented alternative built for pause-time control: it performs its expensive work concurrently, without stopping application threads for more than a millisecond, with pause times designed to stay independent of heap size from a few hundred megabytes up to 16TB.<sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup></li></ul><p>Move buffers off-heap and give G1 room on large heaps (sizes are examples; set to your workload):</p><div class="language-properties vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">properties</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Move Tungsten execution/storage buffers off the JVM heap so the GC has fewer objects to scan</span></span>
+import{_ as a,o as t,c as o,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"GC Pressure","description":"","frontmatter":{"title":"GC Pressure"},"headers":[],"relativePath":"tuning-reference/bottleneck-gc.md","filePath":"tuning-reference/bottleneck-gc.md"}'),r={name:"tuning-reference/bottleneck-gc.md"};function i(n,e,l,c,h,d){return t(),o("div",null,[...e[0]||(e[0]=[s(`<h1 id="bottleneck-gc" tabindex="-1">GC Pressure <a class="header-anchor" href="#bottleneck-gc" aria-label="Permalink to &quot;GC Pressure {#bottleneck-gc}&quot;">​</a></h1><p><span class="tag">GC</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>GC pressure describes how much of an executor&#39;s time goes to JVM garbage collection instead of running task code. Spark tracks this directly in its per-task metrics: <code>jvmGCTime</code> sits alongside <code>executorRunTime</code>, the elapsed wall-clock time the executor spent running the task (as opposed to <code>executorCpuTime</code>, which measures CPU time specifically).<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup> <code>jvmGCTime</code> is not additive on top of that duration: it&#39;s defined as the elapsed time the JVM spent in garbage collection <em>while executing the task</em>, so GC pauses fall inside the same wall-clock window <code>executorRunTime</code> already measures, not outside it. Summing the two would double-count the GC pauses and overstate how long the task actually ran.<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup></p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><p>Because <code>jvmGCTime</code> sits inside <code>executorRunTime</code> rather than alongside it, the ratio between the two gives a bounded read on how much of a task&#39;s wall-clock time went to garbage collection. Both directions below apply once a stage&#39;s <code>executorRunTime</code> reaches 10 seconds, a floor that keeps short stages from reading as noise:</p><table tabindex="0"><thead><tr><th>Signal (gcPct = jvmGCTime / executorRunTime)</th><th>Fires when</th></tr></thead><tbody><tr><td>High GC</td><td>&gt; 10%</td></tr><tr><td>Low GC (cost signal)</td><td>&lt; 5%</td></tr></tbody></table><p>A stage lands in at most one of these two opposite-direction bands at a time. Severity for both tracks the estimated recoverable time as a share of the app&#39;s total runtime: ≥2% is critical, ≥0.5% is warning, anything smaller is info. Low GC signals a possible memory over-provisioning cost rather than time lost to garbage collection.</p><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>As gcPct climbs past these thresholds, a growing share of every task&#39;s wall-clock time is consumed by garbage collection instead of actual computation, so the executor&#39;s useful throughput drops even while it appears busy. Two causes show up repeatedly: oversized executors, and an on-heap memory split pushed too far toward execution/storage.</p><p>Sizing an executor with too many cores relative to its heap is one documented trigger: a fat-executor configuration using all 16 cores of a node was observed to not only hurt HDFS throughput but also &quot;result in excessive garbage [collection].&quot;<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup></p><p>Pushing <code>spark.memory.fraction</code> too far toward its upper end has a similar effect through a different path. Its complement, <code>1 − spark.memory.fraction</code>, is untracked &quot;User Memory&quot; reserved for UDFs, Python/Arrow glue, and native library buffers; Spark doesn&#39;t manage this region at all, and starving it (which is what raising <code>spark.memory.fraction</code> toward something like 0.9 does) leads to &quot;GC pressure or random OOMs,&quot; with no warning before the failure.<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup></p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li>Keep cores per executor down rather than packing an entire node&#39;s cores onto one JVM (the common ~5-cores-per-executor guideline). That avoids the excessive-garbage-collection failure mode documented for fat-executor configurations.<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup></li><li>Don&#39;t raise <code>spark.memory.fraction</code> past a comfortable range chasing more execution/storage memory: doing so starves the untracked user-memory region and risks the same GC-pressure/OOM failure mode.<sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup></li><li>Move Tungsten&#39;s execution and storage buffers off the JVM heap with <code>spark.memory.offHeap.enabled=true</code> (and a positive <code>spark.memory.offHeap.size</code>, which is required whenever off-heap is enabled<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>). Off-heap buffers are invisible to the garbage collector, so fewer and smaller live objects need to be tracked, scanned, and copied on the heap, reducing both the frequency and duration of GC pauses.<sup class="footnote-ref"><a href="#fn3" id="fnref3:2">[3:2]</a></sup></li><li>Tune or switch the GC collector on large heaps. G1GC is Spark&#39;s default collector since 4.0 (which defaults to JDK 17), and large executor heaps may need <code>-XX:G1HeapRegionSize</code> raised as well.<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup> G1&#39;s stop-the-world evacuation and full-GC pauses can still spike under heavy old-generation pressure on large heaps: one documented 88GB-heap benchmark saw a job&#39;s pause spike to nearly 100 seconds under default G1 settings, and only tuning <code>InitiatingHeapOccupancyPercent</code>, <code>ConcGCThreads</code>, and RSet-update parameters brought G1 back ahead of Parallel/CMS on both throughput and latency.<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup> ZGC is a documented alternative built for pause-time control: it performs its expensive work concurrently, without stopping application threads for more than a millisecond, with pause times designed to stay independent of heap size from a few hundred megabytes up to 16TB.<sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup></li></ul><p>Move buffers off-heap and give G1 room on large heaps (sizes are examples; set to your workload):</p><div class="language-properties vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">properties</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Move Tungsten execution/storage buffers off the JVM heap so the GC has fewer objects to scan</span></span>
 <span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.memory.offHeap.enabled</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=true</span></span>
 <span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.memory.offHeap.size</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=2g            </span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># must be &gt; 0 whenever off-heap is enabled; 2g is an example</span></span>
 <span class="line"></span>
 <span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># G1 is the default collector since Spark 4.0 (JDK 17); raise the region size on large heaps</span></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.executor.extraJavaOptions</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=-XX:+UseG1GC -XX:</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">G1HeapRegionSize</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=16m   </span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># 16m is an example</span></span></code></pre></div><h2 id="confidence" tabindex="-1">Confidence <a class="header-anchor" href="#confidence" aria-label="Permalink to &quot;Confidence&quot;">​</a></h2><p>The warning and critical thresholds on gcPct are validated: they read a bounded ratio of <code>jvmGCTime</code> to <code>executorRunTime</code>, both first-class per-task metrics Spark records directly,<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup> and crossing 10% or 20% marks a real, measurable share of wall-clock time lost to garbage collection.</p><p><span class="tag">EXPERIMENTAL</span> A separate signal treats suspiciously <em>low</em> GC time as a possible cost-model or noise-floor indicator, and that one is an unsourced heuristic with no external basis, so treat it as exploratory rather than a confirmed diagnostic.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>A high gcPct can reflect a transient memory spike (a single skewed task, a brief burst of large allocations) rather than chronic pressure, so a threshold crossing on one stage does not by itself prove a sustained sizing problem; corroborate against the executor&#39;s behavior over the whole run. The low-GC signal has no external basis at all, and a near-zero reading is at least as likely to mean the workload simply is not allocation-heavy as to mean anything is wrong.</p><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>Why it happens:</strong> <a href="./memory-model.html">Memory Management</a></li><li><strong>Executor sizing:</strong> <a href="./cluster-config.html">Cluster Tuning</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/monitoring.html#spark-history-server" target="_blank" rel="noreferrer">Monitoring and Instrumentation</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://raw.githubusercontent.com/spoddutur/spark-notes/master/distribution_of_executors_cores_and_memory_for_spark_application.md" target="_blank" rel="noreferrer">Distribution of Executors, Cores, and Memory for a Spark Application</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://luminousmen.com/post/dive-into-spark-memory" target="_blank" rel="noreferrer">Diving into Spark Memory Management</a> <a href="#fnref3" class="footnote-backref">↩︎</a> <a href="#fnref3:1" class="footnote-backref">↩︎</a> <a href="#fnref3:2" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration (Spark)</a> <a href="#fnref4" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/tuning.html" target="_blank" rel="noreferrer">Spark Tuning Guide</a> <a href="#fnref5" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://www.databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html" target="_blank" rel="noreferrer">Tuning Java Garbage Collection for Apache Spark Applications</a> <a href="#fnref6" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://wiki.openjdk.org/display/zgc" target="_blank" rel="noreferrer">The Z Garbage Collector (ZGC)</a> <a href="#fnref7" class="footnote-backref">↩︎</a></p></li></ol></section>`,24)])])}const u=a(r,[["render",i]]);export{p as __pageData,u as default};
+<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.executor.extraJavaOptions</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=-XX:+UseG1GC -XX:</span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">G1HeapRegionSize</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=16m   </span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># 16m is an example</span></span></code></pre></div><h2 id="confidence" tabindex="-1">Confidence <a class="header-anchor" href="#confidence" aria-label="Permalink to &quot;Confidence&quot;">​</a></h2><p>The warning and critical thresholds on gcPct are validated: they read a bounded ratio of <code>jvmGCTime</code> to <code>executorRunTime</code>, both first-class per-task metrics Spark records directly,<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup> and crossing 10% or 20% marks a real, measurable share of wall-clock time lost to garbage collection.</p><p><span class="tag">EXPERIMENTAL</span> A separate signal treats suspiciously <em>low</em> GC time as a possible cost-model or noise-floor indicator, and that one is an unsourced heuristic with no external basis, so treat it as exploratory rather than a confirmed diagnostic.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>A high gcPct can reflect a transient memory spike (a single skewed task, a brief burst of large allocations) rather than chronic pressure, so a threshold crossing on one stage does not by itself prove a sustained sizing problem; corroborate against the executor&#39;s behavior over the whole run. The low-GC signal has no external basis at all, and a near-zero reading is at least as likely to mean the workload simply is not allocation-heavy as to mean anything is wrong.</p><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>Why it happens:</strong> <a href="./memory-model.html">Memory Management</a></li><li><strong>Executor sizing:</strong> <a href="./cluster-config.html">Cluster Tuning</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/monitoring.html#spark-history-server" target="_blank" rel="noreferrer">Monitoring and Instrumentation</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://raw.githubusercontent.com/spoddutur/spark-notes/master/distribution_of_executors_cores_and_memory_for_spark_application.md" target="_blank" rel="noreferrer">Distribution of Executors, Cores, and Memory for a Spark Application</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://luminousmen.com/post/dive-into-spark-memory" target="_blank" rel="noreferrer">Diving into Spark Memory Management</a> <a href="#fnref3" class="footnote-backref">↩︎</a> <a href="#fnref3:1" class="footnote-backref">↩︎</a> <a href="#fnref3:2" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration (Spark)</a> <a href="#fnref4" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/tuning.html" target="_blank" rel="noreferrer">Spark Tuning Guide</a> <a href="#fnref5" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://www.databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html" target="_blank" rel="noreferrer">Tuning Java Garbage Collection for Apache Spark Applications</a> <a href="#fnref6" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://wiki.openjdk.org/display/zgc" target="_blank" rel="noreferrer">The Z Garbage Collector (ZGC)</a> <a href="#fnref7" class="footnote-backref">↩︎</a></p></li></ol></section>`,25)])])}const u=a(r,[["render",i]]);export{p as __pageData,u as default};

package/public/docs/assets/{tuning-reference_bottleneck-gc.md.DSxzZRK7.lean.js → tuning-reference_bottleneck-gc.md.B4T3kJYV.lean.js}

@@ -1 +1 @@
-import{_ as a,o as t,c as o,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"GC Pressure","description":"","frontmatter":{"title":"GC Pressure"},"headers":[],"relativePath":"tuning-reference/bottleneck-gc.md","filePath":"tuning-reference/bottleneck-gc.md"}'),r={name:"tuning-reference/bottleneck-gc.md"};function i(n,e,l,c,h,f){return t(),o("div",null,[...e[0]||(e[0]=[s("",24)])])}const u=a(r,[["render",i]]);export{p as __pageData,u as default};
+import{_ as a,o as t,c as o,a5 as s}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"GC Pressure","description":"","frontmatter":{"title":"GC Pressure"},"headers":[],"relativePath":"tuning-reference/bottleneck-gc.md","filePath":"tuning-reference/bottleneck-gc.md"}'),r={name:"tuning-reference/bottleneck-gc.md"};function i(n,e,l,c,h,d){return t(),o("div",null,[...e[0]||(e[0]=[s("",25)])])}const u=a(r,[["render",i]]);export{p as __pageData,u as default};

package/public/docs/assets/tuning-reference_bottleneck-shuffle.md.DDMQVOW6.js

@@ -0,0 +1,12 @@
+import{_ as a,o as t,c as s,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Shuffle I/O","description":"","frontmatter":{"title":"Shuffle I/O"},"headers":[],"relativePath":"tuning-reference/bottleneck-shuffle.md","filePath":"tuning-reference/bottleneck-shuffle.md"}'),r={name:"tuning-reference/bottleneck-shuffle.md"};function i(n,e,f,l,c,d){return t(),s("div",null,[...e[0]||(e[0]=[o(`<h1 id="bottleneck-shuffle" tabindex="-1">Shuffle I/O <a class="header-anchor" href="#bottleneck-shuffle" aria-label="Permalink to &quot;Shuffle I/O {#bottleneck-shuffle}&quot;">​</a></h1><p><span class="tag">SHFL</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>Shuffle I/O is the data movement Spark performs between stages whenever an operation needs to redistribute data across the cluster so that related rows land on the same partition. At the DataFrame/SQL level, <code>groupBy()</code>, <code>join()</code>, <code>agg()</code>, <code>sortBy()</code>, and <code>reduceByKey()</code>-style aggregations are the classic wide transformations that force this exchange<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>. Non-broadcast join strategies (shuffle hash join, shuffle sort-merge join, and shuffle-and-replicated nested loop / Cartesian product join) all shuffle data across executors the same way<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>. At the RDD layer, the operations that can cause a shuffle are repartitioning (<code>repartition</code>, <code>coalesce</code>), <code>&#39;ByKey</code> operations other than counting (<code>groupByKey</code>, <code>reduceByKey</code>), and join-family operations (<code>cogroup</code>, <code>join</code>)<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>.</p><p>Spark can skip the shuffle when it already knows the data layout: Storage Partition Join avoids it entirely when Spark can use partitioning already reported by a compatible V2 data source<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>, and classic Hive-style bucketing has the same effect: once both sides of a join are bucketed and sorted the same way, the physical plan shows no <code>Exchange</code> operator<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>. <code>DataFrameWriter.partitionBy</code>, by contrast, does not trigger a shuffle by itself on write<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup>.</p><p>At the metrics level, shuffle reads split cross-node traffic from same-host traffic: <code>remoteBytesRead</code> counts bytes read from a remote executor, <code>localBytesRead</code> counts bytes read from local disk, and <code>totalBytesRead</code> is their sum<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup>.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><table tabindex="0"><thead><tr><th>Signal</th><th>Fires when</th></tr></thead><tbody><tr><td>Shuffle read bytes in a stage</td><td>&gt; 50 MB</td></tr></tbody></table><p>50 MB marks a stage as shuffle-heavy enough to flag. Severity then tracks the estimated recoverable time as a share of the app&#39;s total runtime: ≥2% is critical, ≥0.5% is warning, anything smaller is info.</p><p>Beyond raw byte volume, the executor-side wait is captured by <code>fetchWaitTime</code>: time a task spends blocked on a remote shuffle block it needs next, not counting time spent prefetching other blocks in the background<sup class="footnote-ref"><a href="#fn6" id="fnref6:1">[6:1]</a></sup>. That wait time counts toward the task&#39;s overall <code>executorRunTime</code>, since Spark&#39;s wall-clock task-time metric explicitly includes time fetching shuffle data<sup class="footnote-ref"><a href="#fn6" id="fnref6:2">[6:2]</a></sup>.</p><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>As shuffle volume grows, the underlying pull-based mechanism gets less efficient: the number of shuffle blocks grows quadratically with mapper × reducer count while individual block sizes shrink to only tens of KB, which is inefficient for disk-backed random reads<sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup>. Because fetch-wait time is part of a task&#39;s measured execution window, that inefficiency shows up directly as added task time rather than as separate, hidden overhead<sup class="footnote-ref"><a href="#fn6" id="fnref6:3">[6:3]</a></sup>.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li>Let <a href="./aqe.html">Adaptive Query Execution</a> coalesce small post-shuffle partitions automatically at runtime (<code>spark.sql.adaptive.coalescePartitions.enabled</code>, default <code>true</code>) instead of only hand-tuning a fixed partition count<sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup>.</li><li>Review <code>spark.sql.shuffle.partitions</code> (default <code>200</code>), which applies uniformly to every <code>join()</code>, <code>groupBy()</code>, and aggregation regardless of how much data is actually moving<sup class="footnote-ref"><a href="#fn8" id="fnref8">[8]</a></sup>. There&#39;s no fixed formula for the right value: pull it down toward the executor core count for small or streaming workloads<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup>, and otherwise favor over- to under-provisioning, since Spark&#39;s low per-task overhead makes it safer to have too many tasks than too few<sup class="footnote-ref"><a href="#fn9" id="fnref9">[9]</a></sup>.</li><li>Where possible, avoid the shuffle altogether: bucket both sides of a join identically, or rely on Storage Partition Join for compatible sources, so the physical plan drops the <code>Exchange</code> node<sup class="footnote-ref"><a href="#fn3" id="fnref3:2">[3:2]</a></sup><sup class="footnote-ref"><a href="#fn4" id="fnref4:1">[4:1]</a></sup>.</li><li>If map tasks are I/O-bound writing many shuffle files, increase <code>spark.shuffle.file.buffer</code> (default <code>32k</code>) to reduce disk seeks and system calls on the write side<sup class="footnote-ref"><a href="#fn8" id="fnref8:1">[8:1]</a></sup>; Learning Spark&#39;s tuning table recommends bumping it to 1 MB for large jobs<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup>.</li><li>If executors have memory to spare, increase <code>spark.reducer.maxSizeInFlight</code> (default <code>48m</code>) so reducers can pull more map output concurrently and need fewer fetch rounds<sup class="footnote-ref"><a href="#fn8" id="fnref8:2">[8:2]</a></sup>.</li><li>At larger scale, enable the external shuffle service (effectively required for <a href="./cluster-config.html">dynamic allocation</a>, since it lets shuffle files be served after an executor is removed<sup class="footnote-ref"><a href="#fn10" id="fnref10">[10]</a></sup>) and consider push-based (Magnet) shuffle, which converts many small random reads into large sequential reads of pre-merged chunks<sup class="footnote-ref"><a href="#fn7" id="fnref7:1">[7:1]</a></sup>.</li></ul><blockquote><p><strong>PySpark:</strong> adjust the shuffle partition count directly from a running session with <code>spark.conf.set(&quot;spark.sql.shuffle.partitions&quot;, 100)</code>.</p></blockquote><p>A shuffle-tuning starting point: defaults shown, comments say which way to move:</p><div class="language-properties vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">properties</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Let AQE coalesce small post-shuffle partitions at runtime</span></span>
+<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.sql.adaptive.enabled</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=true</span></span>
+<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.sql.adaptive.coalescePartitions.enabled</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=true</span></span>
+<span class="line"></span>
+<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Baseline shuffle partition count (default 200); pull toward executor-core count for small jobs</span></span>
+<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.sql.shuffle.partitions</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=200</span></span>
+<span class="line"></span>
+<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Reduce write-side disk seeks on large shuffles (default 32k; Learning Spark suggests 1m)</span></span>
+<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.shuffle.file.buffer</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=1m</span></span>
+<span class="line"></span>
+<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Let reducers pull more map output per fetch round when executors have spare memory (default 48m)</span></span>
+<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.reducer.maxSizeInFlight</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=48m</span></span></code></pre></div><h2 id="bottleneck-partition-sizing" tabindex="-1">Partition sizing <a class="header-anchor" href="#bottleneck-partition-sizing" aria-label="Permalink to &quot;Partition sizing {#bottleneck-partition-sizing}&quot;">​</a></h2><p><span class="tag">PART</span></p><h3 id="how-it-s-detected-1" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected-1" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h3><p>A stage&#39;s shuffle-read partition sizes surface three distinct problems:</p><table tabindex="0"><thead><tr><th>Signal</th><th>Fires when</th><th>Level</th></tr></thead><tbody><tr><td>Largest partition vs. median</td><td>&gt; 5× the median <strong>and</strong> &gt; 256 MB</td><td>Warning</td></tr><tr><td>Low parallelism</td><td>≥ 1 GB of shuffle read spread across ≤ 7 tasks</td><td>Warning</td></tr><tr><td>Oversized partition</td><td>Largest partition ≥ 5 GB</td><td>Critical</td></tr></tbody></table><p>Skew and low-parallelism severity track the estimated recoverable time as a share of the app&#39;s total runtime, the same wall-clock model used across this reference. An oversized partition is a fixed safety signal instead: it reports critical purely on its own size, because a partition past 5 GB is an OOM/crash risk regardless of how much wall-clock time fixing it would recover, so it stays critical even on a stage that barely dents the run.</p><p>Adaptive Query Execution re-optimizes the plan while the query runs: as each shuffle stage materializes, it reads the real shuffle-file sizes and resizes partitions before launching the stages downstream<sup class="footnote-ref"><a href="#fn11" id="fnref11">[11]</a></sup>. That runtime feedback is what corrects a partition grid that static estimates would get wrong.</p><p>The coarse starting grid is <code>spark.sql.shuffle.partitions</code> (default <code>200</code>), the fixed partition count Spark uses when shuffling for joins or aggregations<sup class="footnote-ref"><a href="#fn12" id="fnref12">[12]</a></sup>. A single number cannot fit every stage. Spread a few megabytes across 200 partitions and most cores sit idle on a handful of rows each; push hundreds of gigabytes through the same 200 and every executor is overloaded, often into memory errors. For small or streaming workloads the 200 default is usually too high, and pulling it toward the executor-core count thins out the flood of tiny partitions crossing the network<sup class="footnote-ref"><a href="#fn1" id="fnref1:4">[1:4]</a></sup>. The pattern AQE encourages is the opposite of hand-tuning that number: leave the initial count large and let runtime coalescing combine adjacent small partitions instead<sup class="footnote-ref"><a href="#fn11" id="fnref11:1">[11:1]</a></sup>.</p><p><strong>Partitions too small (low parallelism payoff).</strong> <code>spark.sql.adaptive.coalescePartitions.enabled</code> (default <code>true</code>) merges contiguous shuffle partitions up to a target size so a stage does not end up with many tiny tasks<sup class="footnote-ref"><a href="#fn13" id="fnref13">[13]</a></sup>. The target is <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code>, the advisory shuffle-partition size AQE steers toward during adaptive optimization, default <code>64 MB</code><sup class="footnote-ref"><a href="#fn13" id="fnref13:1">[13:1]</a></sup>.</p><p><strong>Partitions too big or skewed.</strong> <code>spark.sql.adaptive.skewJoin.enabled</code> (default <code>true</code>) handles skew in shuffled joins by splitting the oversized partitions and replicating the matching side where needed<sup class="footnote-ref"><a href="#fn14" id="fnref14">[14]</a></sup>. A partition only counts as skewed when it clears both bars: larger than <code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code> (default <code>5.0</code>) times the median partition size, and larger than <code>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes</code> (default <code>256 MB</code>)<sup class="footnote-ref"><a href="#fn14" id="fnref14:1">[14:1]</a></sup>. Once it qualifies, the partition is split back down toward the advisory size.</p><p>So the two knobs divide the labor: <code>spark.sql.shuffle.partitions</code> lays down the coarse initial grid, and <code>advisoryPartitionSizeInBytes</code> (64 MB) is the size AQE aims each partition at from both directions. Partitions below it get coalesced; skewed partitions past the 256 MB / 5.0-factor bar get split toward it.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>A high shuffle-byte count is not automatically a defect. A wide transformation such as a large <code>join()</code> or <code>groupBy()</code> legitimately has to move that data, so the volume can be an inherent property of the query rather than something worth fixing. The byte thresholds that raise Info, Warning, and Critical levels are heuristic cutoffs, not measured limits for a given cluster, so treat them as a prompt to look rather than a verdict.</p><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>The mechanism:</strong> <a href="./shuffle.html">Shuffle</a></li><li><strong>Tuning parallelism:</strong> <a href="./partitioning.html">Partitioning</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><em>Learning Spark, 2nd Edition</em>, Damji, Wenig, Das, Lee, ch. 7 <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a> <a href="#fnref1:4" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/rdd-programming-guide.html" target="_blank" rel="noreferrer">RDD Programming Guide</a> <a href="#fnref2" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html" target="_blank" rel="noreferrer">Performance Tuning (Spark SQL, DataFrames and Datasets Guide)</a> <a href="#fnref3" class="footnote-backref">↩︎</a> <a href="#fnref3:1" class="footnote-backref">↩︎</a> <a href="#fnref3:2" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://books.japila.pl/spark-sql-internals/bucketing/" target="_blank" rel="noreferrer">Bucketing (The Internals of Spark SQL)</a> <a href="#fnref4" class="footnote-backref">↩︎</a> <a href="#fnref4:1" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-tips-partition-tuning" target="_blank" rel="noreferrer">Spark Tips: Partition Tuning</a> <a href="#fnref5" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/monitoring.html#spark-history-server" target="_blank" rel="noreferrer">Monitoring and Instrumentation</a> <a href="#fnref6" class="footnote-backref">↩︎</a> <a href="#fnref6:1" class="footnote-backref">↩︎</a> <a href="#fnref6:2" class="footnote-backref">↩︎</a> <a href="#fnref6:3" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://issues.apache.org/jira/browse/SPARK-30602" target="_blank" rel="noreferrer">SPARK-30602: Support push-based shuffle to improve shuffle efficiency</a> <a href="#fnref7" class="footnote-backref">↩︎</a> <a href="#fnref7:1" class="footnote-backref">↩︎</a></p></li><li id="fn8" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration (Spark)</a> <a href="#fnref8" class="footnote-backref">↩︎</a> <a href="#fnref8:1" class="footnote-backref">↩︎</a> <a href="#fnref8:2" class="footnote-backref">↩︎</a></p></li><li id="fn9" class="footnote-item"><p><a href="https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/" target="_blank" rel="noreferrer">How to Tune Your Apache Spark Jobs (Part 2): Cloudera Engineering Blog</a> <a href="#fnref9" class="footnote-backref">↩︎</a></p></li><li id="fn10" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/job-scheduling.html" target="_blank" rel="noreferrer">Job Scheduling: Dynamic Resource Allocation</a> <a href="#fnref10" class="footnote-backref">↩︎</a></p></li><li id="fn11" class="footnote-item"><p><a href="https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html" target="_blank" rel="noreferrer">Adaptive Query Execution: Speeding Up Spark SQL at Runtime (Databricks)</a> <a href="#fnref11" class="footnote-backref">↩︎</a> <a href="#fnref11:1" class="footnote-backref">↩︎</a></p></li><li id="fn12" class="footnote-item"><p><a href="https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala" target="_blank" rel="noreferrer">SQLConf: shuffle-partition defaults (Spark source)</a> <a href="#fnref12" class="footnote-backref">↩︎</a></p></li><li id="fn13" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html" target="_blank" rel="noreferrer">Performance Tuning: Coalescing Post Shuffle Partitions</a> <a href="#fnref13" class="footnote-backref">↩︎</a> <a href="#fnref13:1" class="footnote-backref">↩︎</a></p></li><li id="fn14" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration (Spark)</a> <a href="#fnref14" class="footnote-backref">↩︎</a> <a href="#fnref14:1" class="footnote-backref">↩︎</a></p></li></ol></section>`,34)])])}const u=a(r,[["render",i]]);export{p as __pageData,u as default};

package/public/docs/assets/{tuning-reference_bottleneck-shuffle.md.CM-nTmIH.lean.js → tuning-reference_bottleneck-shuffle.md.DDMQVOW6.lean.js}

@@ -1 +1 @@
-import{_ as a,o as t,c as s,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Shuffle I/O","description":"","frontmatter":{"title":"Shuffle I/O"},"headers":[],"relativePath":"tuning-reference/bottleneck-shuffle.md","filePath":"tuning-reference/bottleneck-shuffle.md"}'),r={name:"tuning-reference/bottleneck-shuffle.md"};function i(n,e,f,l,c,d){return t(),s("div",null,[...e[0]||(e[0]=[o("",28)])])}const u=a(r,[["render",i]]);export{p as __pageData,u as default};
+import{_ as a,o as t,c as s,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Shuffle I/O","description":"","frontmatter":{"title":"Shuffle I/O"},"headers":[],"relativePath":"tuning-reference/bottleneck-shuffle.md","filePath":"tuning-reference/bottleneck-shuffle.md"}'),r={name:"tuning-reference/bottleneck-shuffle.md"};function i(n,e,f,l,c,d){return t(),s("div",null,[...e[0]||(e[0]=[o("",34)])])}const u=a(r,[["render",i]]);export{p as __pageData,u as default};

package/public/docs/assets/{tuning-reference_bottleneck-skew.md.BdUwiDhn.js → tuning-reference_bottleneck-skew.md.B0MzhWcX.js}

@@ -1,14 +1,14 @@
-import{_ as e,o as s,c as t,a5 as i}from"./chunks/framework.DSg0KOwT.js";const f=JSON.parse('{"title":"Task Skew","description":"","frontmatter":{"title":"Task Skew"},"headers":[],"relativePath":"tuning-reference/bottleneck-skew.md","filePath":"tuning-reference/bottleneck-skew.md"}'),n={name:"tuning-reference/bottleneck-skew.md"};function o(r,a,l,h,p,d){return s(),t("div",null,[...a[0]||(a[0]=[i(`<h1 id="bottleneck-skew" tabindex="-1">Task Skew <a class="header-anchor" href="#bottleneck-skew" aria-label="Permalink to &quot;Task Skew {#bottleneck-skew}&quot;">​</a></h1><p><span class="tag">SKEW</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>Task skew happens when a shuffle produces one or a few oversized partitions instead of a roughly even split. Spark&#39;s own skew-join optimizer defines a partition as skewed once it is both larger than a multiple of the median partition size and larger than an absolute byte threshold<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>: in the shipped implementation, more than 5× the median size and more than 256 MB by default<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. Whatever task draws that oversized partition ends up processing far more data than everyone else in the same stage.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><table tabindex="0"><thead><tr><th>Signal</th><th>Warning</th><th>Critical</th></tr></thead><tbody><tr><td>P95 / median task duration</td><td>&gt; 3×</td><td>&gt; 5×</td></tr><tr><td>Max / median task duration (task count &lt; 20)</td><td>&gt; 3×</td><td>&gt; 5×</td></tr></tbody></table><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>An oversized partition becomes a <a href="./bottleneck-straggler.html">straggler task</a>: it&#39;s processed inside a single task, so the stage can&#39;t finish until that task does, no matter how long it takes. Spark&#39;s own skew handling treats splitting the partition as a deliberate trade-off: reading the other join side&#39;s matching partition once per split costs extra I/O, but the design behind the feature argues that cost is worth paying once the skew is severe enough to be producing a straggler in the first place<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li>Enable Adaptive Query Execution&#39;s skew-join handling (<code>spark.sql.adaptive.skewJoin.enabled</code>, default <code>true</code> once <code>spark.sql.adaptive.enabled</code> is also on<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup>). It divides any partition that crosses the skew thresholds into several smaller sub-partitions, joins each one against the matching data on the other side, and unions the results back together<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup>.</li><li>Before AQE existed, the only options were manual, and each carries real limitations, which is exactly the gap AQE&#39;s skew handling was built to close<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup>: salt the join key, over-size <code>spark.sql.shuffle.partitions</code>, or push the <a href="./bottleneck-broadcast-sizing.html">broadcast-join threshold</a> up so the join goes broadcast instead of sort-merge.</li><li>To salt by hand: add a random suffix to the join key on both sides, exploding the smaller side into one row per salt value so every salted variant of the key still finds a match, then join on the combined (key, salt) pair<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>.</li><li>On Databricks, a <code>SKEW</code> hint can name the skewed relation and column (and, optionally, the exact skewed values) directly, letting the planner build a skew-aware plan without hand-rolled salting<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>.</li></ul><blockquote><p><strong>PySpark:</strong> salting is plain DataFrame code, no special API, just <code>withColumn(&quot;salt&quot;, (fn.rand() * n).cast(&quot;int&quot;))</code> on the larger side and an <code>explode</code> over the salt range on the smaller side before joining on the combined key.</p></blockquote><p>The AQE toggle, plus the manual salting fallback as runnable code:</p><div class="language-python vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">python</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">from</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> pyspark.sql </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">import</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> functions </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">as</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> fn</span></span>
+import{_ as e,o as s,c as t,a5 as i}from"./chunks/framework.DSg0KOwT.js";const f=JSON.parse('{"title":"Task Skew","description":"","frontmatter":{"title":"Task Skew"},"headers":[],"relativePath":"tuning-reference/bottleneck-skew.md","filePath":"tuning-reference/bottleneck-skew.md"}'),n={name:"tuning-reference/bottleneck-skew.md"};function o(r,a,l,h,p,d){return s(),t("div",null,[...a[0]||(a[0]=[i(`<h1 id="bottleneck-skew" tabindex="-1">Task Skew <a class="header-anchor" href="#bottleneck-skew" aria-label="Permalink to &quot;Task Skew {#bottleneck-skew}&quot;">​</a></h1><p><span class="tag">SKEW</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>Task skew happens when a shuffle produces one or a few oversized partitions instead of a roughly even split. Spark&#39;s own skew-join optimizer defines a partition as skewed once it is both larger than a multiple of the median partition size and larger than an absolute byte threshold<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>: in the shipped implementation, more than 5× the median size and more than 256 MB by default<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. Whatever task draws that oversized partition ends up processing far more data than everyone else in the same stage.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><table tabindex="0"><thead><tr><th>Signal</th><th>Fires when</th></tr></thead><tbody><tr><td>P95 / median task duration (stage has ≥ 20 tasks)</td><td>&gt; 3×</td></tr><tr><td>Max / median task duration (stage has &lt; 20 tasks)</td><td>&gt; 3×</td></tr></tbody></table><p>A 3× ratio marks a stage as skewed. Beyond that ratio, the estimated recoverable time (the P95-minus-median, or max-minus-median, delta) needs to clear 0.5% of the app&#39;s total runtime before it registers, a floor that filters out a 3× ratio sitting on a few milliseconds. Severity then tracks that same recoverable-time estimate as a share of the app&#39;s total runtime: ≥2% is critical, ≥0.5% is warning, anything smaller is info. A 3× ratio on a stage that barely dents an eight-hour run typically surfaces as info; the same ratio on a stage that dominates a short run reads as critical.</p><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>An oversized partition becomes a <a href="./bottleneck-straggler.html">straggler task</a>: it&#39;s processed inside a single task, so the stage can&#39;t finish until that task does, no matter how long it takes. Spark&#39;s own skew handling treats splitting the partition as a deliberate trade-off: reading the other join side&#39;s matching partition once per split costs extra I/O, but the design behind the feature argues that cost is worth paying once the skew is severe enough to be producing a straggler in the first place<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li>Enable Adaptive Query Execution&#39;s skew-join handling (<code>spark.sql.adaptive.skewJoin.enabled</code>, default <code>true</code> once <code>spark.sql.adaptive.enabled</code> is also on<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup>). It divides any partition that crosses the skew thresholds into several smaller sub-partitions, joins each one against the matching data on the other side, and unions the results back together<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup>.</li><li>Before AQE existed, the only options were manual, and each carries real limitations, which is exactly the gap AQE&#39;s skew handling was built to close<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup>: salt the join key, over-size <code>spark.sql.shuffle.partitions</code>, or push the <a href="./bottleneck-broadcast-sizing.html">broadcast-join threshold</a> up so the join goes broadcast instead of sort-merge.</li><li>To salt by hand: add a random suffix to the join key on both sides, exploding the smaller side into one row per salt value so every salted variant of the key still finds a match, then join on the combined (key, salt) pair<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>.</li><li>On Databricks, a <code>SKEW</code> hint can name the skewed relation and column (and, optionally, the exact skewed values) directly, letting the planner build a skew-aware plan without hand-rolled salting<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>.</li></ul><blockquote><p><strong>PySpark:</strong> salting is plain DataFrame code, no special API, just <code>withColumn(&quot;salt&quot;, (fn.rand() * n).cast(&quot;int&quot;))</code> on the larger side and an <code>explode</code> over the salt range on the smaller side before joining on the combined key.</p></blockquote><p>The AQE toggle, plus the manual salting fallback as runnable code:</p><div class="language-python vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">python</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">from</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> pyspark.sql </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">import</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> functions </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">as</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> fn</span></span>
 <span class="line"></span>
-<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># AQE skew-join handling, active once AQE itself is enabled</span></span>
+<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># AQE skew-join handling — active once AQE itself is enabled</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">spark.conf.set(</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;spark.sql.adaptive.enabled&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;true&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">)</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">spark.conf.set(</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;spark.sql.adaptive.skewJoin.enabled&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;true&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">)</span></span>
 <span class="line"></span>
 <span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Manual salting fallback (pre-AQE): spread the skewed key across N salted variants.</span></span>
-<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># N is an example, size it to how badly the key is skewed.</span></span>
+<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># N is an example — size it to how badly the key is skewed.</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">N </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> 16</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">salted_big </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> big.withColumn(</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;salt&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, (fn.rand() </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">*</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> N).cast(</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;int&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">))</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">salted_small </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> small.withColumn(</span></span>
 <span class="line"><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">    &quot;salt&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, fn.explode(fn.array([fn.lit(i) </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">for</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> i </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">in</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;"> range</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">(N)]))</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">)</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">joined </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> salted_big.join(salted_small, [</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;key&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;salt&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">]).drop(</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;salt&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">)</span></span></code></pre></div><h2 id="confidence" tabindex="-1">Confidence <a class="header-anchor" href="#confidence" aria-label="Permalink to &quot;Confidence&quot;">​</a></h2><p>Validated. These thresholds mirror Spark&#39;s own skew-join optimizer, which defines a skewed partition by the same style of ratio-plus-absolute test applied here to task durations<sup class="footnote-ref"><a href="#fn1" id="fnref1:4">[1:4]</a></sup><sup class="footnote-ref"><a href="#fn2" id="fnref2:2">[2:2]</a></sup>.</p><p>The <code>Stage shape</code> subsection below is a separate, experimental branch: its heuristics are informational stage-shape observations, not a validated finding, so it carries an <code>EXPERIMENTAL</code> badge and should not be read as a diagnosis on its own.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>A task-duration ratio is a proxy for skew, not proof of it. A single long task can just as easily be a garbage-collection pause or a genuinely slow host, and either one inflates the max/median ratio without any partition being oversized. Small stages make the number jumpy: with only a handful of tasks, one slow outlier moves the median enough to trip the threshold on noise alone. Treat the signal as a prompt to look at the stage, not a verdict.</p><h2 id="bottleneck-stage-shape" tabindex="-1">Stage shape <a class="header-anchor" href="#bottleneck-stage-shape" aria-label="Permalink to &quot;Stage shape {#bottleneck-stage-shape}&quot;">​</a></h2><p><span class="tag">SHAPE</span> <span class="tag">EXPERIMENTAL</span></p><p>These are experimental, information-only heuristics about a stage&#39;s overall shape, not a graded finding. They read the relationship between a stage&#39;s task count and the cluster&#39;s resources, and none of them alone means something is wrong.</p><p>A stage&#39;s task count equals the number of partitions in its output, and each task runs as a single thread against one partition, so that count is what caps how much of the cluster the stage can actually use. Three shapes are worth noticing:</p><ul><li><strong>Low parallelism</strong> (task count far below total executor cores). When a stage has fewer tasks than there are core slots to run them in, cores sit idle and the stage cannot put the cluster&#39;s CPU to work; the same shape also concentrates more memory pressure into each task&#39;s aggregation<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup>. Spark&#39;s guidance is to keep parallelism high enough that the cluster does not stay under-used, roughly 2 to 3 tasks per CPU core in general<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup>.</li><li><strong>Data explosion</strong> (task count far above cores, partitions tiny). Push the count too high and partitions shrink until per-task overhead floods the stage, so the scheduling cost starts to dominate the useful work<sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup>.</li><li><strong>Task and stage skew</strong> (uneven task count or duration). A lopsided split shows up as a few tasks carrying the stage while the rest finish early, which is the same imbalance the graded <code>skew</code> finding above tracks by duration ratio.</li></ul><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>Why it happens:</strong> <a href="./partitioning.html">Partitioning</a>, <a href="./joins.html">Join Optimization</a></li><li><strong>How Spark fixes it automatically:</strong> <a href="./aqe.html">Adaptive Query Execution</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://issues.apache.org/jira/browse/SPARK-29544" target="_blank" rel="noreferrer">SPARK-29544: Optimize Skewed Join at Runtime</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a> <a href="#fnref1:4" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration (Spark)</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a> <a href="#fnref2:2" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-tips-partition-tuning" target="_blank" rel="noreferrer">Spark Tips: Partition Tuning</a> <a href="#fnref3" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://docs.databricks.com/aws/en/archive/legacy/skew-join" target="_blank" rel="noreferrer">Skew Join Hint</a> <a href="#fnref4" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/" target="_blank" rel="noreferrer">How to Tune Your Apache Spark Jobs (Part 2)</a> <a href="#fnref5" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/tuning.html" target="_blank" rel="noreferrer">Tuning - Spark</a> <a href="#fnref6" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-partitions" target="_blank" rel="noreferrer">Spark Partitions</a> <a href="#fnref7" class="footnote-backref">↩︎</a></p></li></ol></section>`,27)])])}const c=e(n,[["render",o]]);export{f as __pageData,c as default};
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">joined </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> salted_big.join(salted_small, [</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;key&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">, </span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;salt&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">]).drop(</span><span style="--shiki-light:#032F62;--shiki-dark:#9ECBFF;">&quot;salt&quot;</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">)</span></span></code></pre></div><h2 id="confidence" tabindex="-1">Confidence <a class="header-anchor" href="#confidence" aria-label="Permalink to &quot;Confidence&quot;">​</a></h2><p>Validated. These thresholds mirror Spark&#39;s own skew-join optimizer, which defines a skewed partition by the same style of ratio-plus-absolute test applied here to task durations<sup class="footnote-ref"><a href="#fn1" id="fnref1:4">[1:4]</a></sup><sup class="footnote-ref"><a href="#fn2" id="fnref2:2">[2:2]</a></sup>.</p><p>The <code>Stage shape</code> subsection below is a separate, experimental branch: its heuristics are informational stage-shape observations, not a validated finding, so it carries an <code>EXPERIMENTAL</code> badge and should not be read as a diagnosis on its own.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>A task-duration ratio is a proxy for skew, not proof of it. A single long task can just as easily be a garbage-collection pause or a genuinely slow host, and either one inflates the max/median ratio without any partition being oversized. Small stages make the number jumpy: with only a handful of tasks, one slow outlier moves the median enough to trip the threshold on noise alone. Treat the signal as a prompt to look at the stage, not a verdict.</p><h2 id="bottleneck-stage-shape" tabindex="-1">Stage shape <a class="header-anchor" href="#bottleneck-stage-shape" aria-label="Permalink to &quot;Stage shape {#bottleneck-stage-shape}&quot;">​</a></h2><p><span class="tag">SHAPE</span> <span class="tag">EXPERIMENTAL</span></p><p>These are experimental, information-only heuristics about a stage&#39;s overall shape, not a graded finding. They read the relationship between a stage&#39;s task count and the cluster&#39;s resources, and none of them alone means something is wrong.</p><p>A stage&#39;s task count equals the number of partitions in its output, and each task runs as a single thread against one partition, so that count is what caps how much of the cluster the stage can actually use. Three shapes are worth noticing:</p><ul><li><strong>Low parallelism</strong> (task count far below total executor cores). When a stage has fewer tasks than there are core slots to run them in, cores sit idle and the stage cannot put the cluster&#39;s CPU to work; the same shape also concentrates more memory pressure into each task&#39;s aggregation<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup>. Spark&#39;s guidance is to keep parallelism high enough that the cluster does not stay under-used, roughly 2 to 3 tasks per CPU core in general<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup>.</li><li><strong>Data explosion</strong> (task count far above cores, partitions tiny). Push the count too high and partitions shrink until per-task overhead floods the stage, so the scheduling cost starts to dominate the useful work<sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup>.</li><li><strong>Task and stage skew</strong> (uneven task count or duration). A lopsided split shows up as a few tasks carrying the stage while the rest finish early, which is the same imbalance the graded <code>skew</code> finding above tracks by duration ratio.</li></ul><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>Why it happens:</strong> <a href="./partitioning.html">Partitioning</a>, <a href="./joins.html">Join Optimization</a></li><li><strong>How Spark fixes it automatically:</strong> <a href="./aqe.html">Adaptive Query Execution</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://issues.apache.org/jira/browse/SPARK-29544" target="_blank" rel="noreferrer">SPARK-29544: Optimize Skewed Join at Runtime</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a> <a href="#fnref1:4" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration (Spark)</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a> <a href="#fnref2:2" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-tips-partition-tuning" target="_blank" rel="noreferrer">Spark Tips: Partition Tuning</a> <a href="#fnref3" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://docs.databricks.com/aws/en/archive/legacy/skew-join" target="_blank" rel="noreferrer">Skew Join Hint</a> <a href="#fnref4" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/" target="_blank" rel="noreferrer">How to Tune Your Apache Spark Jobs (Part 2)</a> <a href="#fnref5" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/tuning.html" target="_blank" rel="noreferrer">Tuning - Spark</a> <a href="#fnref6" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-partitions" target="_blank" rel="noreferrer">Spark Partitions</a> <a href="#fnref7" class="footnote-backref">↩︎</a></p></li></ol></section>`,28)])])}const c=e(n,[["render",o]]);export{f as __pageData,c as default};

package/public/docs/assets/{tuning-reference_bottleneck-skew.md.BdUwiDhn.lean.js → tuning-reference_bottleneck-skew.md.B0MzhWcX.lean.js}

@@ -1 +1 @@
-import{_ as e,o as s,c as t,a5 as i}from"./chunks/framework.DSg0KOwT.js";const f=JSON.parse('{"title":"Task Skew","description":"","frontmatter":{"title":"Task Skew"},"headers":[],"relativePath":"tuning-reference/bottleneck-skew.md","filePath":"tuning-reference/bottleneck-skew.md"}'),n={name:"tuning-reference/bottleneck-skew.md"};function o(r,a,l,h,p,d){return s(),t("div",null,[...a[0]||(a[0]=[i("",27)])])}const c=e(n,[["render",o]]);export{f as __pageData,c as default};
+import{_ as e,o as s,c as t,a5 as i}from"./chunks/framework.DSg0KOwT.js";const f=JSON.parse('{"title":"Task Skew","description":"","frontmatter":{"title":"Task Skew"},"headers":[],"relativePath":"tuning-reference/bottleneck-skew.md","filePath":"tuning-reference/bottleneck-skew.md"}'),n={name:"tuning-reference/bottleneck-skew.md"};function o(r,a,l,h,p,d){return s(),t("div",null,[...a[0]||(a[0]=[i("",28)])])}const c=e(n,[["render",o]]);export{f as __pageData,c as default};

package/public/docs/assets/{tuning-reference_bottleneck-small-files.md.B8kloyx8.js → tuning-reference_bottleneck-small-files.md.DHGMTmPs.js}

@@ -1,5 +1,5 @@
-import{_ as t,o as a,c as s,a5 as i}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Small Files","description":"","frontmatter":{"title":"Small Files"},"headers":[],"relativePath":"tuning-reference/bottleneck-small-files.md","filePath":"tuning-reference/bottleneck-small-files.md"}'),o={name:"tuning-reference/bottleneck-small-files.md"};function n(r,e,f,l,h,c){return a(),s("div",null,[...e[0]||(e[0]=[i(`<h1 id="bottleneck-small-files" tabindex="-1">Small Files <a class="header-anchor" href="#bottleneck-small-files" aria-label="Permalink to &quot;Small Files {#bottleneck-small-files}&quot;">​</a></h1><p><span class="tag">SMALLFILES</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>The small-files problem is the cost of managing a large count of tiny files. Writing many small files runs up significant metadata overhead, and Spark handles that pattern badly; distributed filesystems such as HDFS handle it badly too, which is why it earned the name &quot;small file problem&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>. The same trap shows up on the read side. Push <code>spark.sql.files.maxPartitionBytes</code> (default 128 MB, aligned to the <a href="./data-formats.html">Parquet</a> block size) too low and Spark carves the input into many small partition files, piling on disk I/O and the filesystem overhead of opening, closing, and listing directories, all of which are slow on a distributed store<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>.</p><p>Spark writes one file per output partition, so a job left with 200 partitions writes 200 files and one left with 3000 writes 3000, even when many of those files hold only a handful of rows. That count then punishes every downstream job forced to read thousands of tiny files<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>. The opposite extreme is not free either: files that are too large make it inefficient to read a whole block when you only need a few rows<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup>.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><p>Reading the SQL plan surfaces write paths that will emit far more output files than the data warrants, keying off the one-file-per-output-partition rule<sup class="footnote-ref"><a href="#fn4" id="fnref4:1">[4:1]</a></sup>. The signal is a high output-partition count against a modest data volume, which foreshadows a directory full of tiny files.</p><table tabindex="0"><thead><tr><th>Signal</th><th>What it points to</th></tr></thead><tbody><tr><td>Output partition count high vs. data volume</td><td>Many tiny files on write</td></tr><tr><td>Read partitioning below <code>spark.sql.files.maxPartitionBytes</code> (128 MB)</td><td>Over-split input, excess filesystem I/O<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup></td></tr></tbody></table><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>Every extra file is another open, close, and directory-list operation, and on a distributed filesystem those are not cheap<sup class="footnote-ref"><a href="#fn2" id="fnref2:2">[2:2]</a></sup>. The count compounds downstream: a stage that scatters 3000 tiny files hands the next job 3000 files to read back, so the metadata tax is paid twice over<sup class="footnote-ref"><a href="#fn4" id="fnref4:2">[4:2]</a></sup>. None of it is a single dramatic failure. The waste is spread thin across the file count, which is exactly why it goes unnoticed until listing and I/O start to dominate.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><p>Repartition right before the write so the output lands in fewer, better-sized files.</p><ul><li><code>coalesce()</code> is a shuffle-free merge: it fuses existing partitions with no data movement, so <code>df.coalesce(100).write.parquet(...)</code> collapses 3000 tiny files into 100 reasonable ones<sup class="footnote-ref"><a href="#fn4" id="fnref4:3">[4:3]</a></sup>. It does not rebalance, though, so uneven inputs stay uneven once lumped together, and pushing it to <code>coalesce(1)</code> kills parallelism by forcing one executor to do all the work<sup class="footnote-ref"><a href="#fn4" id="fnref4:4">[4:4]</a></sup>.</li><li><code>repartition()</code> is a full reshuffle that buys even distribution at the cost of that <a href="./shuffle.html">shuffle</a>. Reach for it when the distribution itself needs fixing, not just the file count<sup class="footnote-ref"><a href="#fn4" id="fnref4:5">[4:5]</a></sup>.</li></ul><blockquote><p><strong>PySpark:</strong> both are one-line calls on a DataFrame: <code>df.coalesce(100)</code> for a shuffle-free merge, or <code>df.repartition(100)</code> when the data also needs rebalancing.</p></blockquote><div class="language-python vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">python</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Collapse many tiny output files without a shuffle (distribution already even)</span></span>
+import{_ as t,o as a,c as s,a5 as i}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Small Files","description":"","frontmatter":{"title":"Small Files"},"headers":[],"relativePath":"tuning-reference/bottleneck-small-files.md","filePath":"tuning-reference/bottleneck-small-files.md"}'),o={name:"tuning-reference/bottleneck-small-files.md"};function n(r,e,l,f,h,c){return a(),s("div",null,[...e[0]||(e[0]=[i(`<h1 id="bottleneck-small-files" tabindex="-1">Small Files <a class="header-anchor" href="#bottleneck-small-files" aria-label="Permalink to &quot;Small Files {#bottleneck-small-files}&quot;">​</a></h1><p><span class="tag">SMALLFILES</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>The small-files problem is the cost of managing a large count of tiny files. Writing many small files runs up significant metadata overhead, and Spark handles that pattern badly; distributed filesystems such as HDFS handle it badly too, which is why it earned the name &quot;small file problem&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>. The same trap shows up on the read side. Push <code>spark.sql.files.maxPartitionBytes</code> (default 128 MB, aligned to the <a href="./data-formats.html">Parquet</a> block size) too low and Spark carves the input into many small partition files, piling on disk I/O and the filesystem overhead of opening, closing, and listing directories, all of which are slow on a distributed store<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>.</p><p>Spark writes one file per output partition, so a job left with 200 partitions writes 200 files and one left with 3000 writes 3000, even when many of those files hold only a handful of rows. That count then punishes every downstream job forced to read thousands of tiny files<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>. The opposite extreme is not free either: files that are too large make it inefficient to read a whole block when you only need a few rows<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup>.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><p>Each SQL plan node carries file-count and file-size metrics Spark reports directly: <code>number of files read</code>/<code>size of files read</code> on the read side, <code>number of written files</code>/<code>written output</code> on the write side. A node reads as a small-files problem once its file count on a given side is high and the resulting average file size is small.</p><table tabindex="0"><thead><tr><th>Signal (per plan node, checked separately for read and write)</th><th>Fires when</th></tr></thead><tbody><tr><td>File count</td><td>&gt; 100</td></tr><tr><td>Average file size (bytes ÷ file count)</td><td>&lt; 3 MB</td></tr></tbody></table><p>Both conditions have to hold together, so a node with thousands of files that are each big enough, or a handful of genuinely tiny ones, doesn&#39;t register. The signal comes entirely from those file-count and file-size metrics, independent of <code>spark.sql.files.maxPartitionBytes</code>.</p><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>Every extra file is another open, close, and directory-list operation, and on a distributed filesystem those are not cheap<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup>. The count compounds downstream: a stage that scatters 3000 tiny files hands the next job 3000 files to read back, so the metadata tax is paid twice over<sup class="footnote-ref"><a href="#fn4" id="fnref4:1">[4:1]</a></sup>. None of it is a single dramatic failure. The waste is spread thin across the file count, which is exactly why it goes unnoticed until listing and I/O start to dominate.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><p>Repartition right before the write so the output lands in fewer, better-sized files.</p><ul><li><code>coalesce()</code> is a shuffle-free merge: it fuses existing partitions with no data movement, so <code>df.coalesce(100).write.parquet(...)</code> collapses 3000 tiny files into 100 reasonable ones<sup class="footnote-ref"><a href="#fn4" id="fnref4:2">[4:2]</a></sup>. It does not rebalance, though, so uneven inputs stay uneven once lumped together, and pushing it to <code>coalesce(1)</code> kills parallelism by forcing one executor to do all the work<sup class="footnote-ref"><a href="#fn4" id="fnref4:3">[4:3]</a></sup>.</li><li><code>repartition()</code> is a full reshuffle that buys even distribution at the cost of that <a href="./shuffle.html">shuffle</a>. Reach for it when the distribution itself needs fixing, not just the file count<sup class="footnote-ref"><a href="#fn4" id="fnref4:4">[4:4]</a></sup>.</li></ul><blockquote><p><strong>PySpark:</strong> both are one-line calls on a DataFrame: <code>df.coalesce(100)</code> for a shuffle-free merge, or <code>df.repartition(100)</code> when the data also needs rebalancing.</p></blockquote><div class="language-python vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">python</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Collapse many tiny output files without a shuffle (distribution already even)</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">df.coalesce(</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">100</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">).write.parquet(path)</span></span>
 <span class="line"></span>
 <span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Full reshuffle to a target count when the distribution itself needs fixing</span></span>
-<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">df.repartition(</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">100</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">).write.parquet(path)</span></span></code></pre></div><p>For runtime control, <a href="./aqe.html">Adaptive Query Execution</a> can merge tiny shuffle partitions on its own. With <code>spark.sql.adaptive.enabled</code> and <code>spark.sql.adaptive.coalescePartitions.enabled</code> (default true) set, AQE coalesces contiguous shuffle partitions toward <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code> (default 64 MB)<sup class="footnote-ref"><a href="#fn4" id="fnref4:6">[4:6]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:1">[5:1]</a></sup>. Just mind the scope: AQE only engages after the first shuffle, so it fixes shuffle-output partition counts but will not repair input-side partitioning or a bad file layout on disk<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup>.</p><h2 id="confidence" tabindex="-1">Confidence <a class="header-anchor" href="#confidence" aria-label="Permalink to &quot;Confidence&quot;">​</a></h2><p>The core inference, that one output file per partition times a high partition count yields many tiny files, is grounded directly in Spark&#39;s write behavior<sup class="footnote-ref"><a href="#fn4" id="fnref4:7">[4:7]</a></sup>, and the mitigation levers (<code>coalesce()</code>, <code>repartition()</code>, and AQE coalescing) are documented Spark features<sup class="footnote-ref"><a href="#fn4" id="fnref4:8">[4:8]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:2">[5:2]</a></sup>.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>A high file count can be perfectly legitimate: partitioned output deliberately fans data across many files by partition column, so a large count there is by design, not a defect. And AQE coalescing is not a cure-all here, since it only affects post-shuffle partitions and leaves the input file layout untouched<sup class="footnote-ref"><a href="#fn6" id="fnref6:1">[6:1]</a></sup>. The signal reflects the shape, not intent, so confirm the write is not an intended partitioned layout before acting.</p><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>Partition sizing:</strong> <a href="./partitioning.html">Partitioning</a></li><li><strong>Too many small tasks:</strong> <a href="./bottleneck-tiny-tasks.html">Tiny Tasks</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><em>Spark: The Definitive Guide</em>, Chambers &amp; Zaharia, ch. 19 <a href="#fnref1" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><em>Learning Spark, 2nd Edition</em>, Damji, Wenig, Das, Lee, ch. 7 <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a> <a href="#fnref2:2" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala" target="_blank" rel="noreferrer">SQLConf.scala</a> <a href="#fnref3" class="footnote-backref">↩︎</a> <a href="#fnref3:1" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-partitions" target="_blank" rel="noreferrer">Spark Partitions</a> <a href="#fnref4" class="footnote-backref">↩︎</a> <a href="#fnref4:1" class="footnote-backref">↩︎</a> <a href="#fnref4:2" class="footnote-backref">↩︎</a> <a href="#fnref4:3" class="footnote-backref">↩︎</a> <a href="#fnref4:4" class="footnote-backref">↩︎</a> <a href="#fnref4:5" class="footnote-backref">↩︎</a> <a href="#fnref4:6" class="footnote-backref">↩︎</a> <a href="#fnref4:7" class="footnote-backref">↩︎</a> <a href="#fnref4:8" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html" target="_blank" rel="noreferrer">Performance Tuning (Spark SQL, DataFrames and Datasets Guide)</a> <a href="#fnref5" class="footnote-backref">↩︎</a> <a href="#fnref5:1" class="footnote-backref">↩︎</a> <a href="#fnref5:2" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://luminousmen.com/post/the-apache-spark-optimization-checklist" target="_blank" rel="noreferrer">The Apache Spark Optimization Checklist</a> <a href="#fnref6" class="footnote-backref">↩︎</a> <a href="#fnref6:1" class="footnote-backref">↩︎</a></p></li></ol></section>`,24)])])}const u=t(o,[["render",n]]);export{p as __pageData,u as default};
+<span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">df.repartition(</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">100</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">).write.parquet(path)</span></span></code></pre></div><p>For runtime control, <a href="./aqe.html">Adaptive Query Execution</a> can merge tiny shuffle partitions on its own. With <code>spark.sql.adaptive.enabled</code> and <code>spark.sql.adaptive.coalescePartitions.enabled</code> (default true) set, AQE coalesces contiguous shuffle partitions toward <code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code> (default 64 MB)<sup class="footnote-ref"><a href="#fn4" id="fnref4:5">[4:5]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:1">[5:1]</a></sup>. Just mind the scope: AQE only engages after the first shuffle, so it fixes shuffle-output partition counts but will not repair input-side partitioning or a bad file layout on disk<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup>.</p><h2 id="confidence" tabindex="-1">Confidence <a class="header-anchor" href="#confidence" aria-label="Permalink to &quot;Confidence&quot;">​</a></h2><p>The core inference, that one output file per partition times a high partition count yields many tiny files, is grounded directly in Spark&#39;s write behavior<sup class="footnote-ref"><a href="#fn4" id="fnref4:6">[4:6]</a></sup>, and the mitigation levers (<code>coalesce()</code>, <code>repartition()</code>, and AQE coalescing) are documented Spark features<sup class="footnote-ref"><a href="#fn4" id="fnref4:7">[4:7]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:2">[5:2]</a></sup>.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>A high file count can be perfectly legitimate: partitioned output deliberately fans data across many files by partition column, so a large count there is by design, not a defect. And AQE coalescing is not a cure-all here, since it only affects post-shuffle partitions and leaves the input file layout untouched<sup class="footnote-ref"><a href="#fn6" id="fnref6:1">[6:1]</a></sup>. The signal reflects the shape, not intent, so confirm the write is not an intended partitioned layout before acting.</p><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>Partition sizing:</strong> <a href="./partitioning.html">Partitioning</a></li><li><strong>Too many small tasks:</strong> <a href="./bottleneck-tiny-tasks.html">Tiny Tasks</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><em>Spark: The Definitive Guide</em>, Chambers &amp; Zaharia, ch. 19 <a href="#fnref1" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><em>Learning Spark, 2nd Edition</em>, Damji, Wenig, Das, Lee, ch. 7 <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala" target="_blank" rel="noreferrer">SQLConf.scala</a> <a href="#fnref3" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-partitions" target="_blank" rel="noreferrer">Spark Partitions</a> <a href="#fnref4" class="footnote-backref">↩︎</a> <a href="#fnref4:1" class="footnote-backref">↩︎</a> <a href="#fnref4:2" class="footnote-backref">↩︎</a> <a href="#fnref4:3" class="footnote-backref">↩︎</a> <a href="#fnref4:4" class="footnote-backref">↩︎</a> <a href="#fnref4:5" class="footnote-backref">↩︎</a> <a href="#fnref4:6" class="footnote-backref">↩︎</a> <a href="#fnref4:7" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html" target="_blank" rel="noreferrer">Performance Tuning (Spark SQL, DataFrames and Datasets Guide)</a> <a href="#fnref5" class="footnote-backref">↩︎</a> <a href="#fnref5:1" class="footnote-backref">↩︎</a> <a href="#fnref5:2" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://luminousmen.com/post/the-apache-spark-optimization-checklist" target="_blank" rel="noreferrer">The Apache Spark Optimization Checklist</a> <a href="#fnref6" class="footnote-backref">↩︎</a> <a href="#fnref6:1" class="footnote-backref">↩︎</a></p></li></ol></section>`,25)])])}const u=t(o,[["render",n]]);export{p as __pageData,u as default};

package/public/docs/assets/{tuning-reference_bottleneck-small-files.md.B8kloyx8.lean.js → tuning-reference_bottleneck-small-files.md.DHGMTmPs.lean.js}

@@ -1 +1 @@
-import{_ as t,o as a,c as s,a5 as i}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Small Files","description":"","frontmatter":{"title":"Small Files"},"headers":[],"relativePath":"tuning-reference/bottleneck-small-files.md","filePath":"tuning-reference/bottleneck-small-files.md"}'),o={name:"tuning-reference/bottleneck-small-files.md"};function n(r,e,f,l,h,c){return a(),s("div",null,[...e[0]||(e[0]=[i("",24)])])}const u=t(o,[["render",n]]);export{p as __pageData,u as default};
+import{_ as t,o as a,c as s,a5 as i}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Small Files","description":"","frontmatter":{"title":"Small Files"},"headers":[],"relativePath":"tuning-reference/bottleneck-small-files.md","filePath":"tuning-reference/bottleneck-small-files.md"}'),o={name:"tuning-reference/bottleneck-small-files.md"};function n(r,e,l,f,h,c){return a(),s("div",null,[...e[0]||(e[0]=[i("",25)])])}const u=t(o,[["render",n]]);export{p as __pageData,u as default};

package/public/docs/assets/{tuning-reference_bottleneck-straggler.md.DY36fHN5.js → tuning-reference_bottleneck-straggler.md.DIbXdAcQ.js}

@@ -1,7 +1,7 @@
-import{_ as a,o as t,c as s,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Stragglers","description":"","frontmatter":{"title":"Stragglers"},"headers":[],"relativePath":"tuning-reference/bottleneck-straggler.md","filePath":"tuning-reference/bottleneck-straggler.md"}'),i={name:"tuning-reference/bottleneck-straggler.md"};function n(r,e,l,h,c,f){return t(),s("div",null,[...e[0]||(e[0]=[o(`<h1 id="bottleneck-straggler" tabindex="-1">Stragglers <a class="header-anchor" href="#bottleneck-straggler" aria-label="Permalink to &quot;Stragglers {#bottleneck-straggler}&quot;">​</a></h1><p><span class="tag">STRAG</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>A straggler is one task (or a small handful of tasks) that runs far longer than the rest of the tasks in its stage, even when the stage is otherwise healthy. Because a stage only completes once its last task finishes, a single straggler holds up the whole stage, and every other executor sits idle waiting for it to catch up.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><p>A task counts as a straggler when speculative tasks fired for it, or when more than 5% of the tasks in a stage run at least 4× the stage&#39;s median task duration; this rule only applies once a stage has at least 10 tasks.</p><p>&quot;Duration&quot; here is <code>executorRunTime</code>: elapsed wall-clock time, not CPU time, and it already includes any time the task spent blocked fetching shuffle data<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>. That matters when tracking down why a task is slow: a long <code>executorRunTime</code> doesn&#39;t necessarily mean more compute happened. A <a href="./bottleneck-gc.html">garbage-collection pause</a> is counted inside it rather than added on top (<code>jvmGCTime</code> is a subset of <code>executorRunTime</code>, not additive<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>), and a task waiting on a remote shuffle block it needs next shows up in <code>shuffleReadMetrics.fetchWaitTime</code>, which only counts genuine blocking time, not blocks being prefetched in the background<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup>.</p><p>The &quot;speculative tasks fired&quot; half of the rule is Spark&#39;s own detector: once <code>spark.speculation.quantile</code> (default <code>0.9</code>) of a stage&#39;s tasks finish, Spark compares each remaining task&#39;s duration against <code>spark.speculation.multiplier</code> (default <code>3</code>) times the median of the tasks that already finished, subject to a <code>spark.speculation.minTaskRuntime</code> floor (default <code>100ms</code>) so short tasks aren&#39;t flagged just for being slower than a tiny median<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. Speculation itself is off by default (<code>spark.speculation</code> defaults to <code>false</code>), so this half of the rule only fires on stages where it&#39;s been turned on<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup>.</p><p>Watch for overlap with <a href="./bottleneck-skew.html">task skew</a>: an unevenly distributed key sends one partition far more data than its peers, and that partition&#39;s task will trip this same duration threshold even though the underlying cause is data volume, not a slow host or a GC pause. Check the skew signals before assuming the latter.</p><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>A straggler wastes cluster capacity the same way a skewed stage does: every executor other than the one running the slow task finishes early and idles, while total job time still tracks the single slowest task. Because a straggler&#39;s duration can be inflated by a GC pause<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup> or a slow <a href="./shuffle.html">shuffle</a> fetch<sup class="footnote-ref"><a href="#fn1" id="fnref1:4">[1:4]</a></sup> rather than genuinely more work, it&#39;s worth checking those angles (and whether the real cause is skew rather than anything task-local) before assuming a hardware explanation.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li>Enable speculative execution: <code>spark.speculation</code> is <code>false</code> by default, so nothing reruns a slow task automatically until it&#39;s turned on<sup class="footnote-ref"><a href="#fn2" id="fnref2:2">[2:2]</a></sup>.</li><li>Tune the trigger: <code>spark.speculation.quantile</code> (default <code>0.9</code>) sets how much of the stage must finish before speculation kicks in, and <code>spark.speculation.multiplier</code> (default <code>3</code>) sets how many times slower than the median a task must be. For stages with very few tasks, <code>spark.speculation.task.duration.threshold</code> (available since 3.0.0) gives an absolute-duration trigger instead of relying on the median<sup class="footnote-ref"><a href="#fn2" id="fnref2:3">[2:3]</a></sup>.</li><li>Since Spark 3.4, <code>spark.speculation.efficiency.enabled</code> (default <code>true</code>) adds a filter so a task is only speculated if its data-process rate is below the stage average (times <code>spark.speculation.efficiency.processRateMultiplier</code>, default <code>0.75</code>) or its duration exceeds <code>spark.speculation.efficiency.longRunTaskFactor</code> (default <code>2</code>) times the same time threshold. This avoids wasting a duplicate task slot on work that&#39;s simply doing more, not running slower<sup class="footnote-ref"><a href="#fn2" id="fnref2:4">[2:4]</a></sup>.</li><li>If the cause is really a skewed key, treat it as <a href="./bottleneck-skew.html">task skew</a> instead: AQE&#39;s skew-join optimization automatically splits a partition once it&#39;s larger than <code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code> (default <code>5.0</code>) times the median partition size and above <code>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes</code> (default <code>256 MB</code>)<sup class="footnote-ref"><a href="#fn2" id="fnref2:5">[2:5]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>. Manual salting (appending a random prefix to the skewed key so it spreads across more partitions) is the pre-AQE fallback, though the design doc behind AQE&#39;s skew handling calls salting and other manual approaches limited compared to the automatic option<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>.</li></ul><blockquote><p><strong>PySpark:</strong> speculation settings can be set on the session directly, no <code>spark-submit</code> flag needed: <code>spark.conf.set(&quot;spark.speculation&quot;, &quot;true&quot;)</code>, then the quantile/multiplier equivalents the same way.</p></blockquote><p>Speculation config: defaults shown, plus the absolute-duration trigger for small stages:</p><div class="language-properties vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">properties</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Speculatively relaunch straggler tasks (OFF by default)</span></span>
+import{_ as t,o as a,c as s,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Stragglers","description":"","frontmatter":{"title":"Stragglers"},"headers":[],"relativePath":"tuning-reference/bottleneck-straggler.md","filePath":"tuning-reference/bottleneck-straggler.md"}'),i={name:"tuning-reference/bottleneck-straggler.md"};function r(n,e,l,h,c,d){return a(),s("div",null,[...e[0]||(e[0]=[o(`<h1 id="bottleneck-straggler" tabindex="-1">Stragglers <a class="header-anchor" href="#bottleneck-straggler" aria-label="Permalink to &quot;Stragglers {#bottleneck-straggler}&quot;">​</a></h1><p><span class="tag">STRAG</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>A straggler is one task (or a small handful of tasks) that runs far longer than the rest of the tasks in its stage, even when the stage is otherwise healthy. Because a stage only completes once its last task finishes, a single straggler holds up the whole stage, and every other executor sits idle waiting for it to catch up.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><p>A task counts as a straggler when speculative tasks fired for it, or when more than 5% of the tasks in a stage run at least 4× the stage&#39;s median task duration; this rule only applies once a stage has at least 10 tasks.</p><p>&quot;Duration&quot; here is <code>executorRunTime</code>: elapsed wall-clock time, not CPU time, and it already includes any time the task spent blocked fetching shuffle data<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>. That matters when tracking down why a task is slow: a long <code>executorRunTime</code> doesn&#39;t necessarily mean more compute happened. A <a href="./bottleneck-gc.html">garbage-collection pause</a> is counted inside it rather than added on top (<code>jvmGCTime</code> is a subset of <code>executorRunTime</code>, not additive<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>), and a task waiting on a remote shuffle block it needs next shows up in <code>shuffleReadMetrics.fetchWaitTime</code>, which only counts genuine blocking time, not blocks being prefetched in the background<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup>.</p><p>The &quot;speculative tasks fired&quot; half of the rule is Spark&#39;s own detector: once <code>spark.speculation.quantile</code> (default <code>0.9</code>) of a stage&#39;s tasks finish, Spark compares each remaining task&#39;s duration against <code>spark.speculation.multiplier</code> (default <code>3</code>) times the median of the tasks that already finished, subject to a <code>spark.speculation.minTaskRuntime</code> floor (default <code>100ms</code>) so short tasks aren&#39;t flagged just for being slower than a tiny median<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. Speculation itself is off by default (<code>spark.speculation</code> defaults to <code>false</code>), so this half of the rule only fires on stages where it&#39;s been turned on<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup>.</p><p>Watch for overlap with <a href="./bottleneck-skew.html">task skew</a>: an unevenly distributed key sends one partition far more data than its peers, and that partition&#39;s task will trip this same duration threshold even though the underlying cause is data volume, not a slow host or a GC pause. Check the skew signals before assuming the latter.</p><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>A straggler wastes cluster capacity the same way a skewed stage does: every executor other than the one running the slow task finishes early and idles, while total job time still tracks the single slowest task. Because a straggler&#39;s duration can be inflated by a GC pause<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup> or a slow <a href="./shuffle.html">shuffle</a> fetch<sup class="footnote-ref"><a href="#fn1" id="fnref1:4">[1:4]</a></sup> rather than genuinely more work, it&#39;s worth checking those angles (and whether the real cause is skew rather than anything task-local) before assuming a hardware explanation.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li>Enable speculative execution: <code>spark.speculation</code> is <code>false</code> by default, so nothing reruns a slow task automatically until it&#39;s turned on<sup class="footnote-ref"><a href="#fn2" id="fnref2:2">[2:2]</a></sup>.</li><li>Tune the trigger: <code>spark.speculation.quantile</code> (default <code>0.9</code>) sets how much of the stage must finish before speculation kicks in, and <code>spark.speculation.multiplier</code> (default <code>3</code>) sets how many times slower than the median a task must be. For stages with very few tasks, <code>spark.speculation.task.duration.threshold</code> (available since 3.0.0) gives an absolute-duration trigger instead of relying on the median<sup class="footnote-ref"><a href="#fn2" id="fnref2:3">[2:3]</a></sup>.</li><li>Since Spark 3.4, <code>spark.speculation.efficiency.enabled</code> (default <code>true</code>) adds a filter so a task is only speculated if its data-process rate is below the stage average (times <code>spark.speculation.efficiency.processRateMultiplier</code>, default <code>0.75</code>) or its duration exceeds <code>spark.speculation.efficiency.longRunTaskFactor</code> (default <code>2</code>) times the same time threshold. This avoids wasting a duplicate task slot on work that&#39;s simply doing more, not running slower<sup class="footnote-ref"><a href="#fn2" id="fnref2:4">[2:4]</a></sup>.</li><li>If the cause is really a skewed key, treat it as <a href="./bottleneck-skew.html">task skew</a> instead: AQE&#39;s skew-join optimization automatically splits a partition once it&#39;s larger than <code>spark.sql.adaptive.skewJoin.skewedPartitionFactor</code> (default <code>5.0</code>) times the median partition size and above <code>spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes</code> (default <code>256 MB</code>)<sup class="footnote-ref"><a href="#fn2" id="fnref2:5">[2:5]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>. Manual salting (appending a random prefix to the skewed key so it spreads across more partitions) is the pre-AQE fallback, though the design doc behind AQE&#39;s skew handling calls salting and other manual approaches limited compared to the automatic option<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>.</li></ul><blockquote><p><strong>PySpark:</strong> speculation settings can be set on the session directly, no <code>spark-submit</code> flag needed: <code>spark.conf.set(&quot;spark.speculation&quot;, &quot;true&quot;)</code>, then the quantile/multiplier equivalents the same way.</p></blockquote><p>Speculation config: defaults shown, plus the absolute-duration trigger for small stages:</p><div class="language-properties vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">properties</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Speculatively relaunch straggler tasks (OFF by default)</span></span>
 <span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.speculation</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=true</span></span>
 <span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.speculation.quantile</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=0.9          </span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># default; portion of tasks finished before speculation begins</span></span>
 <span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.speculation.multiplier</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=3          </span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># default; multiple of median duration that marks a task slow</span></span>
 <span class="line"></span>
 <span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Absolute-duration trigger for stages with very few tasks (since Spark 3.0)</span></span>
-<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.speculation.task.duration.threshold</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=10s   </span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># 10s is an example</span></span></code></pre></div><h2 id="confidence" tabindex="-1">Confidence <a class="header-anchor" href="#confidence" aria-label="Permalink to &quot;Confidence&quot;">​</a></h2><p>Validated.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>The 4x-median rule flags a slow task, but slow is not the same as broken. The same threshold trips on a skewed key that simply has more data to process, or on a task that spent its time in a GC pause rather than doing extra work, so a flagged task is not automatically a slow host. Small stages make this worse: with only a handful of tasks the median is unstable, and one moderately slow task can look like a straggler against a median computed from too few peers.</p><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>When the real cause is a skewed key:</strong> <a href="./partitioning.html">Partitioning</a>, <a href="./aqe.html">Adaptive Query Execution</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/monitoring.html#spark-history-server" target="_blank" rel="noreferrer">Monitoring and Instrumentation</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a> <a href="#fnref1:4" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration (Spark)</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a> <a href="#fnref2:2" class="footnote-backref">↩︎</a> <a href="#fnref2:3" class="footnote-backref">↩︎</a> <a href="#fnref2:4" class="footnote-backref">↩︎</a> <a href="#fnref2:5" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html#optimizing-skew-join" target="_blank" rel="noreferrer">Optimizing Skew Join (Spark SQL, DataFrames and Datasets Guide)</a> <a href="#fnref3" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://issues.apache.org/jira/browse/SPARK-29544" target="_blank" rel="noreferrer">SPARK-29544: Optimize Skewed Join at Runtime with New Adaptive Execution</a> <a href="#fnref4" class="footnote-backref">↩︎</a></p></li></ol></section>`,24)])])}const u=a(i,[["render",n]]);export{p as __pageData,u as default};
+<span class="line"><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">spark.speculation.task.duration.threshold</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">=10s   </span><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># 10s is an example</span></span></code></pre></div><h2 id="confidence" tabindex="-1">Confidence <a class="header-anchor" href="#confidence" aria-label="Permalink to &quot;Confidence&quot;">​</a></h2><p>Validated.</p><h2 id="limitations-false-positive-risk" tabindex="-1">Limitations / false-positive risk <a class="header-anchor" href="#limitations-false-positive-risk" aria-label="Permalink to &quot;Limitations / false-positive risk&quot;">​</a></h2><p>The 4x-median rule flags a slow task, but slow is not the same as broken. The same threshold trips on a skewed key that simply has more data to process, or on a task that spent its time in a GC pause rather than doing extra work, so a flagged task is not automatically a slow host. Small stages make this worse: with only a handful of tasks the median is unstable, and one moderately slow task can look like a straggler against a median computed from too few peers.</p><h2 id="bottleneck-speculation-waste" tabindex="-1">Speculation waste <a class="header-anchor" href="#bottleneck-speculation-waste" aria-label="Permalink to &quot;Speculation waste {#bottleneck-speculation-waste}&quot;">​</a></h2><p><span class="tag">SPEC</span></p><p>A straggler is reported by whichever signal, straggler share or speculative-task count, best explains it. Speculation waste is a separate, narrower signal: the executor time speculative attempts burned without confirming a genuine straggler, whether the speculative copy lost the race to the original attempt or the other way around. Either way, the losing attempt&#39;s executor time is pure waste.</p><h3 id="how-it-s-detected-1" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected-1" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h3><table tabindex="0"><thead><tr><th>Signal</th><th>Fires when</th></tr></thead><tbody><tr><td>Wasted speculative attempts in a stage</td><td>≥ 5</td></tr><tr><td>Wasted executor time from those attempts</td><td>≥ 60 seconds</td></tr></tbody></table><p>Both conditions have to hold together. Severity tracks the estimated recoverable time as a share of the app&#39;s total runtime, the same model used across this page: ≥2% is critical, ≥0.5% is warning, anything smaller is info.</p><h3 id="why-it-matters-1" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters-1" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h3><p>Every wasted speculative attempt occupies an executor slot that could have run other work, so a stage generating a lot of speculative waste is trading cluster capacity for copies that never pay off.</p><h3 id="how-to-fix-it-1" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it-1" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h3><p>If task durations are naturally variable rather than genuine stragglers, speculation is firing too eagerly: tune <code>spark.speculation.multiplier</code> (require a bigger gap from the median before speculating) or <code>spark.speculation.quantile</code> (wait for more of the stage to finish first) so fewer ordinary slow tasks get speculated in the first place.</p><h2 id="related" tabindex="-1">Related <a class="header-anchor" href="#related" aria-label="Permalink to &quot;Related&quot;">​</a></h2><ul><li><strong>When the real cause is a skewed key:</strong> <a href="./partitioning.html">Partitioning</a>, <a href="./aqe.html">Adaptive Query Execution</a></li></ul><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/monitoring.html#spark-history-server" target="_blank" rel="noreferrer">Monitoring and Instrumentation</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a> <a href="#fnref1:4" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration (Spark)</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a> <a href="#fnref2:2" class="footnote-backref">↩︎</a> <a href="#fnref2:3" class="footnote-backref">↩︎</a> <a href="#fnref2:4" class="footnote-backref">↩︎</a> <a href="#fnref2:5" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html#optimizing-skew-join" target="_blank" rel="noreferrer">Optimizing Skew Join (Spark SQL, DataFrames and Datasets Guide)</a> <a href="#fnref3" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://issues.apache.org/jira/browse/SPARK-29544" target="_blank" rel="noreferrer">SPARK-29544: Optimize Skewed Join at Runtime with New Adaptive Execution</a> <a href="#fnref4" class="footnote-backref">↩︎</a></p></li></ol></section>`,34)])])}const u=t(i,[["render",r]]);export{p as __pageData,u as default};

package/public/docs/assets/{tuning-reference_bottleneck-straggler.md.DY36fHN5.lean.js → tuning-reference_bottleneck-straggler.md.DIbXdAcQ.lean.js}

@@ -1 +1 @@
-import{_ as a,o as t,c as s,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Stragglers","description":"","frontmatter":{"title":"Stragglers"},"headers":[],"relativePath":"tuning-reference/bottleneck-straggler.md","filePath":"tuning-reference/bottleneck-straggler.md"}'),i={name:"tuning-reference/bottleneck-straggler.md"};function n(r,e,l,h,c,f){return t(),s("div",null,[...e[0]||(e[0]=[o("",24)])])}const u=a(i,[["render",n]]);export{p as __pageData,u as default};
+import{_ as t,o as a,c as s,a5 as o}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Stragglers","description":"","frontmatter":{"title":"Stragglers"},"headers":[],"relativePath":"tuning-reference/bottleneck-straggler.md","filePath":"tuning-reference/bottleneck-straggler.md"}'),i={name:"tuning-reference/bottleneck-straggler.md"};function r(n,e,l,h,c,d){return a(),s("div",null,[...e[0]||(e[0]=[o("",34)])])}const u=t(i,[["render",r]]);export{p as __pageData,u as default};

package/public/docs/assets/{tuning-reference_bottleneck-tiny-tasks.md.QTV7O8kU.js → tuning-reference_bottleneck-tiny-tasks.md.CB05v3zh.js}

@@ -1,7 +1,7 @@
 import{_ as t,o as a,c as s,a5 as i}from"./chunks/framework.DSg0KOwT.js";const p=JSON.parse('{"title":"Tiny Tasks","description":"","frontmatter":{"title":"Tiny Tasks"},"headers":[],"relativePath":"tuning-reference/bottleneck-tiny-tasks.md","filePath":"tuning-reference/bottleneck-tiny-tasks.md"}'),o={name:"tuning-reference/bottleneck-tiny-tasks.md"};function n(r,e,l,h,f,d){return a(),s("div",null,[...e[0]||(e[0]=[i(`<h1 id="bottleneck-tiny-tasks" tabindex="-1">Tiny Tasks <a class="header-anchor" href="#bottleneck-tiny-tasks" aria-label="Permalink to &quot;Tiny Tasks {#bottleneck-tiny-tasks}&quot;">​</a></h1><p><span class="tag">TINY</span></p><h2 id="what-it-is" tabindex="-1">What it is <a class="header-anchor" href="#what-it-is" aria-label="Permalink to &quot;What it is&quot;">​</a></h2><p>Every task pays a fixed placement and serialization cost before it does any real work. Spark schedules around data locality rather than moving data to code: &quot;Spark builds its scheduling around this general principle&quot; that shipping serialized code is cheaper than shipping data, preferring <code>PROCESS_LOCAL</code> locality down to <code>ANY</code> and waiting a configurable timeout for a busy CPU to free up before shipping data to a farther executor<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>. On top of that placement cost, every task also pays a serialization cost for its closure and, if not using Kryo, for its data: Java&#39;s default <code>ObjectOutputStream</code>-based serialization &quot;is flexible but often quite slow, and leads to large serialized formats,&quot; while Kryo is &quot;significantly faster and more compact than Java serialization (often as much as 10x)&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>. None of that overhead is large per task: Spark&#39;s own guidance leans toward more, smaller tasks rather than fewer, larger ones, unlike MapReduce: &quot;it&#39;s almost always better to err on the side of a larger number of tasks (and thus partitions)... The difference stems from the fact that MapReduce has a high startup overhead for tasks, while Spark does not&quot;<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. But once partitions get small enough, that per-task overhead stops being negligible and starts to dominate.</p><h2 id="how-it-s-detected" tabindex="-1">How it&#39;s detected <a class="header-anchor" href="#how-it-s-detected" aria-label="Permalink to &quot;How it&#39;s detected&quot;">​</a></h2><p>The failure mode is partitions so small that scheduling and serialization overhead outweighs the actual work. With Spark&#39;s default of 200 shuffle partitions applied to a dataset of only a few megabytes, &quot;those 200 partitions will each get like ten rows. Tasks become microscopic. Most of your CPUs will just be sitting there doing nothing, wasting cluster hours&quot;<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>. The same small-partition penalty shows up on the <a href="./bottleneck-shuffle.html">shuffle-read side</a>: production measurements found &quot;the average shuffle block size is only 10s of KBs, which leads to delayed shuffle data fetch,&quot; and the shuffle reduce stages with the largest fetch delays (over 30 seconds per task) were consistently the ones with small block sizes<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>. A high task count paired with very short median task duration, and/or very small shuffle block sizes on the read side, are the signals to look for.</p><h2 id="why-it-matters" tabindex="-1">Why it matters <a class="header-anchor" href="#why-it-matters" aria-label="Permalink to &quot;Why it matters&quot;">​</a></h2><p>Tiny tasks waste cluster capacity even though no single task is a straggler the way <a href="./bottleneck-skew.html">skewed</a> or <a href="./bottleneck-straggler.html">straggler</a> tasks are: the cost here is spread evenly across thousands of tasks, each individually cheap but collectively adding up in scheduling and serialization overhead, while cores that could be doing other useful work sit mostly idle between task launches<sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup>.</p><h2 id="how-to-fix-it" tabindex="-1">How to fix it <a class="header-anchor" href="#how-to-fix-it" aria-label="Permalink to &quot;How to fix it&quot;">​</a></h2><ul><li><code>coalesce()</code> merges existing partitions without a shuffle (&quot;no data movement, no shuffle&quot;) and is typically used right before a write, e.g. <code>df.coalesce(100).write.parquet(...)</code>, to collapse many tiny output files into a manageable number cheaply<sup class="footnote-ref"><a href="#fn3" id="fnref3:2">[3:2]</a></sup>. Because it only stacks partitions together rather than redistributing data, it doesn&#39;t fix an uneven underlying distribution (uneven input partitions stay uneven, just grouped) and pushing it too far (<code>coalesce(1)</code>) kills parallelism by funneling all work onto one executor while the rest sit idle<sup class="footnote-ref"><a href="#fn3" id="fnref3:3">[3:3]</a></sup>.</li><li><code>repartition()</code> performs a full reshuffle, giving control and rebalancing that Spark won&#39;t do on its own: with <code>df.repartition(200)</code>, &quot;you&#39;re paying for predictability&quot;<sup class="footnote-ref"><a href="#fn3" id="fnref3:4">[3:4]</a></sup>. Use it when the partitioning itself needs to be fixed or rebalanced, not merely reduced in count, accepting the shuffle cost to get there.</li><li>There&#39;s a middle option, <code>coalesce(N, shuffle=True)</code>, which &quot;acts more like a repartition, but leaning toward reduction... not free (you pay for the shuffle cost) but you get better distribution and fewer partitions&quot;<sup class="footnote-ref"><a href="#fn3" id="fnref3:5">[3:5]</a></sup>.</li><li>Rule of thumb: reach for <code>coalesce()</code> when only the partition count needs to shrink and the existing distribution is already reasonably even (e.g. collapsing output files); reach for <code>repartition()</code> when the distribution itself is the problem.</li></ul><blockquote><p><strong>PySpark:</strong> both are one-line calls on a DataFrame: <code>df.coalesce(100)</code> for a shuffle-free merge, <code>df.repartition(200)</code> for a full reshuffle, or <code>df.coalesce(100, shuffle=True)</code> for the shuffled middle option.</p></blockquote><p>The three sizing calls, side by side:</p><div class="language-python vp-adaptive-theme"><button title="Copy Code" class="copy"></button><span class="lang">python</span><pre class="shiki shiki-themes github-light github-dark vp-code" tabindex="0"><code><span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Collapse many tiny output files without a shuffle (use when distribution is already even)</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">df.coalesce(</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">100</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">).write.parquet(path)</span></span>
 <span class="line"></span>
-<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Full reshuffle to a target count, use when the distribution itself needs rebalancing</span></span>
+<span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Full reshuffle to a target count — use when the distribution itself needs rebalancing</span></span>
 <span class="line"><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">df </span><span style="--shiki-light:#D73A49;--shiki-dark:#F97583;">=</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;"> df.repartition(</span><span style="--shiki-light:#005CC5;--shiki-dark:#79B8FF;">200</span><span style="--shiki-light:#24292E;--shiki-dark:#E1E4E8;">)</span></span>
 <span class="line"></span>
 <span class="line"><span style="--shiki-light:#6A737D;--shiki-dark:#6A737D;"># Middle ground: reduce partition count but still rebalance (pays a shuffle)</span></span>