sparkforensics-cli 0.1.0 → 0.2.0

package/vendor-core/docs-content/tuning/job-failure-rate.md

@@ -0,0 +1,101 @@
+# Job Failure Rate
+
+<span class="tag">FAIL-RATE</span>
+
+## What it is
+
+Not every task retry threatens the job. `spark.task.maxFailures` (default `4`) counts
+"continuous failures of any particular task before giving up on the job. The total number of
+failures spread across different tasks will not cause the job to fail; a particular task has to
+fail this number of attempts continuously. If any attempt succeeds, the failure count for the
+task will be reset"[^1]. A task that fails once and then succeeds on retry resets its counter and
+never counts toward job abandonment: that's a transient retry, and by itself it's harmless.
+Whole-job abandonment is a different, layered escalation above that: it only happens once an
+unbroken run of failures on the same task reaches the configured limit (3 retries allowed by
+default, since allowed retries = value − 1), or once higher-level retry budgets above the task
+level are themselves exhausted.
+
+## How it's detected
+
+The layers above task-level retries are what turn isolated failures into a failed job. At the
+stage level, `spark.stage.maxConsecutiveAttempts` (default `4`) bounds "the number of consecutive
+stage attempts allowed before a stage is aborted"[^1]. By default,
+`spark.stage.ignoreDecommissionFetchFailure` (`true`, since 3.4.0) excludes fetch failures caused
+by graceful executor decommission from counting toward that limit, so a decommission-triggered
+`FetchFailed` doesn't push a stage toward abortion the way a genuine repeated fetch failure
+would[^1]. On YARN and Kubernetes there's a further application-level ceiling:
+`spark.executor.maxNumFailures` (default `numExecutors * 2`, minimum 3, since 3.5.0) is "the
+maximum number of executor failures before failing the application," while
+`spark.executor.failuresValidityInterval` (since 3.5.0) lets failures spaced far enough apart be
+"considered independent and not accumulate towards the attempt count"[^1]. The TaskScheduler owns
+retries at the task level, but it is "the DAGScheduler that ultimately declares the job to have
+failed"[^2] once those retry budgets are exhausted, and because "a Spark job corresponds to one
+action" with a fixed DAG once that action is called[^3], an aborted stage cascades directly into
+failure of the job built on it.
+
+<img class="light-only" src="../diagrams/retry-escalation-ladder.svg" alt="How a failure escalates from a task retry up through stage resubmission, the executor failure ceiling, and a failed application attempt before the job is aborted.">
+<img class="dark-only" src="../diagrams/retry-escalation-ladder.dark.svg" alt="How a failure escalates from a task retry up through stage resubmission, the executor failure ceiling, and a failed application attempt before the job is aborted.">
+
+## Why it matters
+
+A rising failure rate that traces back to exhausted retry budgets, rather than to a single
+one-off task failure, points at a systemic problem (an unstable executor, a genuinely broken
+fetch path, or a resource ceiling being hit repeatedly) since none of these budgets trip on a
+single transient hiccup. Job-level failure is also distinct from application-attempt failure: on
+YARN, "each application may have multiple attempts," and the history server displays failed
+attempts alongside "any ongoing incomplete attempt or the final successful attempt"[^4]: a
+separate, higher layer of retry (driver/application restart) above the in-job task and stage
+retries covered here.
+
+## How to fix it
+
+- Check which budget was exhausted before treating a job failure as a single root cause: a
+  single task failing `spark.task.maxFailures` times consecutively, a stage being resubmitted
+  until `spark.stage.maxConsecutiveAttempts` is reached and aborted, or (on YARN/Kubernetes)
+  cumulative distinct executor failures exceeding `spark.executor.maxNumFailures`[^1] each point
+  at a different underlying problem.
+- If failures are decommission-triggered `FetchFailed`s during a normal scale-down, confirm
+  `spark.stage.ignoreDecommissionFetchFailure` is enabled (default `true` since 3.4.0) so they
+  aren't inflating the stage-abort count[^1].
+- On YARN/Kubernetes, if unrelated executor failures spread far apart in time are tripping
+  `spark.executor.maxNumFailures`, `spark.executor.failuresValidityInterval` (since 3.5.0) can
+  let sufficiently-spaced failures stop accumulating toward the same count[^1].
+- See also [retry waste](#bottleneck-retry-waste): the same executor-loss and `FetchFailed`
+  causes that eventually exhaust these budgets and fail a job are, below that threshold, also
+  the source of wasted executor time on jobs that ultimately succeed.
+
+> **PySpark:** these are session-level configs, settable without a `spark-submit` flag:
+> `spark.conf.set("spark.task.maxFailures", "4")`, and the stage/executor-level equivalents the
+> same way.
+
+The retry-budget ladder: defaults shown; raise a budget only once you know which layer is tripping:
+
+```properties
+# Task level: consecutive failures of one task before the job is abandoned (default 4 => 3 retries)
+spark.task.maxFailures=4
+
+# Stage level: consecutive stage attempts before the stage is aborted (default 4)
+spark.stage.maxConsecutiveAttempts=4
+
+# Don't count graceful-decommission fetch failures toward the stage-abort limit (default true since Spark 3.4)
+spark.stage.ignoreDecommissionFetchFailure=true
+```
+
+
+## Limitations / false-positive risk
+
+A nonzero job-failure rate can be dominated by a single repeatedly-failing job rather than a
+systemic issue, so the rate alone doesn't tell you whether the problem is broad or isolated.
+Retried jobs that eventually succeed still count toward attempts, which can inflate the rate
+above what the eventual outcomes justify.
+
+
+## Related
+
+- **Retry budgets & executor stability:** [Cluster Tuning](#cluster-config)
+- **Avoiding the failures upstream:** [Anti-Patterns](#anti-patterns)
+
+[^1]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^2]: [Spark: Anatomy of Spark Application](https://luminousmen.com/post/spark-anatomy-of-spark-application)
+[^3]: *High Performance Spark, 2nd Edition*, Karau, Polak & Warren, ch. 2
+[^4]: [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html#spark-history-server)

package/vendor-core/docs-content/tuning/memory-utilization.md

@@ -0,0 +1,58 @@
+# Memory Utilization
+
+<span class="tag">MEM</span>
+
+## What it is
+
+An executor is a single JVM process that gets a fixed memory allocation when the application starts, and it holds that whole allocation for its entire lifetime, whether or not it has work to run[^3]. Two things can leave that memory band poorly used: cores sitting idle inside a held executor, and a held executor whose memory drains of active work but is never released. This finding covers both.
+
+An executor's core count is its concurrency ceiling. `spark.executor.cores` sets how many tasks it can run at once, so `--executor-cores 5` caps that executor at five concurrent tasks[^1]. The scheduler turns cores into task slots from `spark.executor.cores` and `spark.task.cpus` (minimum 1), with `spark.task.cpus` defaulting to one core per task[^2]. Cores go idle whenever there are fewer runnable tasks than slots: a stage with fewer partitions than the total slots across executors leaves slots empty, and so does the tail of a stage where a [straggler](#bottleneck-straggler) or two keep running after their peers finish. Raising `spark.task.cpus` above 1 has the same effect from the other side, since each task then reserves several cores and fewer run side by side. Through all of it the JVM keeps its fixed heap[^3].
+
+## How it's detected
+
+| Rule | Signal | Status |
+|---|---|---|
+| idleCores | Task parallelism below allocated cores while the executor is held | Validated |
+| memoryBand | A whole executor's memory band held with little or no active work (dynamic-allocation idle timeouts) | Validated |
+| wasteModel | Peak used memory well below allocated memory (unused headroom) | Experimental |
+
+## Why it matters
+
+A held executor is allocation you pay for regardless of how busy it is. On YARN that is the memory YARN grants the container; on Kubernetes it is the pod memory limit[^3]. When cores idle or a whole executor lingers with nothing to do, that fixed band is reserved without returning work, so the cost lands whether or not tasks are running.
+
+## How to fix it
+
+Keep executors from being oversized in the first place. The guidance is against packing every core of a node into one fat executor: assigning all 16 cores of a node to a single executor hurts HDFS throughput and drives excessive [garbage collection](#bottleneck-gc), so aim for a balance between tiny (one core per executor) and fat (one executor per node) sizing[^4]. On YARN, leave cores for the OS and Hadoop daemons instead of handing 100% of a node to Spark containers[^1].
+
+For the held memory band, lean on [dynamic allocation](#cluster-config) to reclaim executors once the work drains. It requests executors when tasks back up and frees them when they go idle[^1]. Executors are added in rounds once tasks have been pending for `spark.dynamicAllocation.schedulerBacklogTimeout` (default 1s), then again every `spark.dynamicAllocation.sustainedSchedulerBacklogTimeout` while the backlog holds[^5]. On the release side, an executor is removed after it has been idle longer than `spark.dynamicAllocation.executorIdleTimeout`[^5].
+
+Cached data is the trap here. By default an executor holding [cached blocks](#caching) is never removed, governed by `spark.dynamicAllocation.cachedExecutorIdleTimeout`, whose default is infinity[^2]. Such an executor keeps its full memory band indefinitely with no active tasks unless you set that timeout to a finite value, or turn on `spark.shuffle.service.fetch.rdd.enabled` so executors holding only disk-persisted blocks are treated as idle after `spark.dynamicAllocation.executorIdleTimeout` and released[^5]. To avoid holding barely-used executors at all, lower `spark.dynamicAllocation.executorAllocationRatio` (default 1.0, full parallelism) toward 0.5, since with small tasks full-parallelism allocation can request executors that never do any work[^2].
+
+```properties
+# Balanced executor sizing, not one fat executor per node
+spark.executor.cores=5
+
+# Reclaim idle executors; give cached holders a finite timeout
+spark.dynamicAllocation.enabled=true
+spark.dynamicAllocation.cachedExecutorIdleTimeout=300s
+spark.dynamicAllocation.executorAllocationRatio=0.5
+```
+
+### Reading the wasted-memory estimate <span class="tag">EXPERIMENTAL</span>
+
+The wasteModel rule estimates unused ("wasted") allocated memory from the gap between an executor's peak used memory and its allocated total. Treat it as a rough buffer heuristic, not a measurement. Peak used memory is a high-water mark rather than a ceiling: Spark records it under `peakMemoryMetrics.*`, and each figure is the maximum its pool ever reached, so peak sits at or below the allocated total by construction and the space between them is the executor's unused headroom[^6]. Turning that headroom into a wasted number stays approximate for concrete reasons. The peak heap figure counts garbage, since `JVMHeapMemory` is the peak used heap including "the amount of memory occupied by both live objects and garbage objects that have not been collected," so it overstates the live footprint[^6]. The region boundaries also move: `totalOnHeapStorageMemory` and `totalOffHeapStorageMemory` "can vary over time, depending on the MemoryManager implementation," so there is no single fixed allocated-to-storage value to subtract a peak from[^6]. Peak metrics only reach the event log when `spark.eventLog.logStageExecutorMetrics` is true[^6], so without that setting there is nothing to estimate from. Use the number as a hint that an executor may be oversized, then confirm against sizing before acting on it.
+
+## Confidence
+
+The idleCores and memoryBand rules are validated: they rest on documented Spark concurrency and dynamic-allocation behavior. The wasteModel estimate is low-confidence and experimental. It is a buffer heuristic derived from peak-versus-allocated sampling, not an exact accounting of unused memory, so it should steer investigation rather than settle it.
+
+## Limitations / false-positive risk
+
+Idle cores are not always waste. The tail of a stage legitimately leaves slots empty while a straggler or two finish[^2], so a snapshot of low parallelism can reflect a normal straggler tail rather than chronic under-utilization. The wasted-memory estimate depends on peak-memory sampling and is only as good as that sampling: the peak includes uncollected garbage, the managed-region boundaries shift under the MemoryManager, and the figures are absent entirely unless `spark.eventLog.logStageExecutorMetrics` is enabled[^6].
+
+[^1]: [How to Tune Your Apache Spark Jobs (Part 2)](https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/)
+[^2]: [Configuration — Spark](https://spark.apache.org/docs/latest/configuration.html)
+[^3]: [Dive into Spark memory](https://luminousmen.com/post/dive-into-spark-memory)
+[^4]: [Distribution of executors, cores and memory for a Spark application](https://raw.githubusercontent.com/spoddutur/spark-notes/master/distribution_of_executors_cores_and_memory_for_spark_application.md)
+[^5]: [Job Scheduling — Spark](https://spark.apache.org/docs/latest/job-scheduling.html)
+[^6]: [Monitoring — Spark](https://spark.apache.org/docs/latest/monitoring.html)

package/vendor-core/docs-content/tuning/retry-waste.md

@@ -0,0 +1,90 @@
+# Retry Waste
+
+<span class="tag">RETRY</span>
+
+## What it is
+
+When a task attempt is superseded by a later retry, every bit of executor time the abandoned
+attempt spent counts for nothing. Every attempt, successful or not, accrues the standard task
+metrics: `executorRunTime` ("elapsed time the executor spent running this task," including
+time fetching shuffle data) and `executorCpuTime`[^1], but only the winning attempt's output
+survives. The TaskScheduler is what triggers the do-over: "if an executor dies or a task throws
+an exception, the TaskScheduler resubmits the task to another executor, respecting Spark's task
+locality preferences"[^2]. Whatever the discarded attempt had already run is gone; none of it
+carries forward into the job's result.
+
+<img class="light-only" src="../diagrams/retry-escalation-ladder.svg" alt="Where a superseded retry sits in the escalation ladder, below the stage and application budgets that a job crosses only when the same failures keep recurring.">
+<img class="dark-only" src="../diagrams/retry-escalation-ladder.dark.svg" alt="Where a superseded retry sits in the escalation ladder, below the stage and application budgets that a job crosses only when the same failures keep recurring.">
+
+## How it's detected
+
+A stage's superseded task attempts are the signal: a later retry supersedes an earlier attempt
+after [an executor is lost](#bottleneck-failures) (`ExecutorLostFailure`) or after a shuffle `FetchFailed`, since "if an
+executor dies or a task throws an exception, the TaskScheduler resubmits the task to another
+executor, respecting Spark's task locality preferences"[^2]. Because only the winning attempt's
+output survives, the abandoned attempt's `executorRunTime` (the elapsed executor time defined
+above) is the wasted time.
+
+The two causes are handled differently by Spark's own machinery, which is how they can be told
+apart after the fact. An `ExecutorLostFailure` removes the executor (and any shuffle map output
+it had already written) from the pool outright[^3]. A shuffle `FetchFailed`, by contrast, is a
+read-side failure against a still-alive remote executor, and Spark's own exclusion machinery
+treats it as a distinct category from a general executor loss:
+`spark.excludeOnFailure.killExcludedExecutors` governs whether Spark kills executors "excluded
+on fetch failure or excluded for the entire application"[^4], a separate bucket from executor
+loss in Spark's accounting. That distinction (executor-loss bucket vs. fetch-failure bucket)
+is the basis for identifying which cause dominated a given stage's retry waste.
+
+## Why it matters
+
+A lost executor can waste more than just the in-flight task's time: "dynamic allocation may
+remove an executor before the shuffle completes, in which case the shuffle files written by that
+executor must be recomputed unnecessarily"[^3], so prior map-output work from that same executor
+may need redoing too. A `FetchFailed` doesn't carry that same risk, since the remote executor
+stays alive and only the one failed read is lost[^4].
+
+## How to fix it
+
+- Check the retried-attempt count and accumulated `executorRunTime`/`executorCpuTime` on the
+  abandoned attempts directly, rather than trusting the stage's final wall-clock duration:
+  a clean-looking stage can still be hiding significant wasted compute[^1].
+- If the dominant cause is executor loss, investigate node-level stability (OOM-kills, node
+  death, aggressive dynamic-allocation deallocation) rather than the task logic itself, since
+  the lost executor's prior shuffle-write work may also need recomputing[^3].
+- If the dominant cause is `FetchFailed`, check whether `spark.excludeOnFailure.killExcludedExecutors`
+  is causing repeated exclusion churn on otherwise-live executors, and treat it separately from
+  outright executor loss[^4].
+- See also [job failure rate](#bottleneck-job-failure-rate): retry waste can pile up quietly on
+  a job that ultimately succeeds, while the same underlying causes (exhausted at a higher
+  threshold) are what push a job to fail outright.
+
+> **PySpark:** there's no attempt-level API to recover a discarded attempt's `executorRunTime`:
+> pull it from the event log or history server UI's per-stage task list, filtering for tasks
+> whose attempt number is greater than zero.
+
+The preventive lever: keep an executor's shuffle output available if it is lost, so retries don't recompute prior map work[^5]:
+
+```properties
+spark.dynamicAllocation.shuffleTracking.enabled=true
+```
+
+## Limitations / false-positive risk
+
+Some retry waste is unavoidable: transient cluster faults will always cost a few
+superseded attempts, and no tuning drives that to zero. The metric attributes burned
+executor time to the abandoned attempts, so its accuracy tracks the failure cause. When
+a lost executor also forces recompute of prior map output, the per-attempt
+`executorRunTime` undercounts the true cost; when an attempt fails almost immediately, it
+overcounts the compute actually lost. Read the number as a directional signal, not an
+exact ledger.
+
+## Related
+
+- **Shuffle recompute on executor loss:** [Shuffle](#shuffle)
+- **Executor stability & dynamic allocation:** [Cluster Tuning](#cluster-config)
+
+[^1]: [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html)
+[^2]: [Spark: Anatomy of Spark Application](https://luminousmen.com/post/spark-anatomy-of-spark-application)
+[^3]: [Job Scheduling (Spark)](https://spark.apache.org/docs/latest/job-scheduling.html)
+[^4]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^5]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)

package/vendor-core/docs-content/tuning/shuffle.md

@@ -0,0 +1,154 @@
+# Shuffle I/O
+
+<span class="tag">SHFL</span>
+
+## What it is
+
+Shuffle I/O is the data movement Spark performs between stages whenever an operation needs
+to redistribute data across the cluster so that related rows land on the same partition.
+At the DataFrame/SQL level, `groupBy()`, `join()`, `agg()`, `sortBy()`, and `reduceByKey()`-style
+aggregations are the classic wide transformations that force this exchange[^1]. Non-broadcast
+join strategies (shuffle hash join, shuffle sort-merge join, and shuffle-and-replicated nested
+loop / Cartesian product join) all shuffle data across executors the same
+way[^1]. At the RDD layer, the operations that can cause a shuffle are repartitioning
+(`repartition`, `coalesce`), `'ByKey` operations other than counting (`groupByKey`,
+`reduceByKey`), and join-family operations (`cogroup`, `join`)[^2].
+
+Spark can skip the shuffle when it already knows the data layout: Storage Partition Join
+avoids it entirely when Spark can use partitioning already reported by a compatible V2 data
+source[^3], and classic Hive-style bucketing has the same effect: once both sides of a join
+are bucketed and sorted the same way, the physical plan shows no `Exchange` operator[^4].
+`DataFrameWriter.partitionBy`, by contrast, does not trigger a shuffle by itself on write[^5].
+
+At the metrics level, shuffle reads split cross-node traffic from same-host traffic:
+`remoteBytesRead` counts bytes read from a remote executor, `localBytesRead` counts bytes
+read from local disk, and `totalBytesRead` is their sum[^6].
+
+## How it's detected
+
+| Shuffle read/write bytes | Level |
+|---|---|
+| > 50 MB | Info |
+| > 500 MB | Warning |
+| > 1 GB | Critical |
+
+Beyond raw byte volume, the executor-side wait is captured by `fetchWaitTime`: time a task
+spends blocked on a remote shuffle block it needs next, not counting time spent prefetching
+other blocks in the background[^6]. That wait time counts toward the task's overall
+`executorRunTime`, since Spark's wall-clock task-time metric explicitly includes time
+fetching shuffle data[^6].
+
+## Why it matters
+
+As shuffle volume grows, the underlying pull-based mechanism gets less efficient: the number
+of shuffle blocks grows quadratically with mapper × reducer count while individual block
+sizes shrink to only tens of KB, which is inefficient for disk-backed random reads[^7].
+Because fetch-wait time is part of a task's measured execution window, that inefficiency
+shows up directly as added task time rather than as separate, hidden overhead[^6].
+
+## How to fix it
+
+- Let [Adaptive Query Execution](#aqe) coalesce small post-shuffle partitions automatically at
+  runtime (`spark.sql.adaptive.coalescePartitions.enabled`, default `true`) instead of only
+  hand-tuning a fixed partition count[^3].
+- Review `spark.sql.shuffle.partitions` (default `200`), which applies uniformly to every
+  `join()`, `groupBy()`, and aggregation regardless of how much data is actually moving[^8].
+  There's no fixed formula for the right value: pull it down toward the executor core
+  count for small or streaming workloads[^1], and otherwise favor over- to
+  under-provisioning, since Spark's low per-task overhead makes it safer to have too many
+  tasks than too few[^9].
+- Where possible, avoid the shuffle altogether: bucket both sides of a join identically, or
+  rely on Storage Partition Join for compatible sources, so the physical plan drops the
+  `Exchange` node[^3][^4].
+- If map tasks are I/O-bound writing many shuffle files, increase
+  `spark.shuffle.file.buffer` (default `32k`) to reduce disk seeks and system calls on the
+  write side[^8]; Learning Spark's tuning table recommends bumping it to 1 MB for large
+  jobs[^1].
+- If executors have memory to spare, increase `spark.reducer.maxSizeInFlight` (default
+  `48m`) so reducers can pull more map output concurrently and need fewer fetch rounds[^8].
+- At larger scale, enable the external shuffle service (effectively required for [dynamic
+  allocation](#cluster-config), since it lets shuffle files be served after an executor is removed[^10]) and
+  consider push-based (Magnet) shuffle, which converts many small random reads into large
+  sequential reads of pre-merged chunks[^7].
+
+> **PySpark:** adjust the shuffle partition count directly from a running session with
+> `spark.conf.set("spark.sql.shuffle.partitions", 100)`.
+
+A shuffle-tuning starting point: defaults shown, comments say which way to move:
+
+```properties
+# Let AQE coalesce small post-shuffle partitions at runtime
+spark.sql.adaptive.enabled=true
+spark.sql.adaptive.coalescePartitions.enabled=true
+
+# Baseline shuffle partition count (default 200); pull toward executor-core count for small jobs
+spark.sql.shuffle.partitions=200
+
+# Reduce write-side disk seeks on large shuffles (default 32k; Learning Spark suggests 1m)
+spark.shuffle.file.buffer=1m
+
+# Let reducers pull more map output per fetch round when executors have spare memory (default 48m)
+spark.reducer.maxSizeInFlight=48m
+```
+
+## Partition sizing <span class="tag">PART</span>
+
+Adaptive Query Execution re-optimizes the plan while the query runs: as each shuffle stage
+materializes, it reads the real shuffle-file sizes and resizes partitions before launching the
+stages downstream[^11]. That runtime feedback is what corrects a partition grid that static
+estimates would get wrong.
+
+The coarse starting grid is `spark.sql.shuffle.partitions` (default `200`), the fixed partition
+count Spark uses when shuffling for joins or aggregations[^14]. A single number cannot fit every
+stage. Spread a few megabytes across 200 partitions and most cores sit idle on a handful of rows
+each; push hundreds of gigabytes through the same 200 and every executor is overloaded, often into
+memory errors. For small or streaming workloads the 200 default is usually too high, and pulling it
+toward the executor-core count thins out the flood of tiny partitions crossing the network[^1]. The
+pattern AQE encourages is the opposite of hand-tuning that number: leave the initial count large and
+let runtime coalescing combine adjacent small partitions instead[^11].
+
+**Partitions too small (low parallelism payoff).** `spark.sql.adaptive.coalescePartitions.enabled`
+(default `true`) merges contiguous shuffle partitions up to a target size so a stage does not end up
+with many tiny tasks[^12]. The target is `spark.sql.adaptive.advisoryPartitionSizeInBytes`, the
+advisory shuffle-partition size AQE steers toward during adaptive optimization, default `64 MB`[^12].
+
+**Partitions too big or skewed.** `spark.sql.adaptive.skewJoin.enabled` (default `true`) handles
+skew in shuffled joins by splitting the oversized partitions and replicating the matching side where
+needed[^13]. A partition only counts as skewed when it clears both bars: larger than
+`spark.sql.adaptive.skewJoin.skewedPartitionFactor` (default `5.0`) times the median partition size,
+and larger than `spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes` (default `256 MB`)[^13].
+Once it qualifies, the partition is split back down toward the advisory size.
+
+So the two knobs divide the labor: `spark.sql.shuffle.partitions` lays down the coarse initial grid,
+and `advisoryPartitionSizeInBytes` (64 MB) is the size AQE aims each partition at from both
+directions. Partitions below it get coalesced; skewed partitions past the 256 MB / 5.0-factor bar get
+split toward it.
+
+## Limitations / false-positive risk
+
+A high shuffle-byte count is not automatically a defect. A wide transformation such as a large
+`join()` or `groupBy()` legitimately has to move that data, so the volume can be an inherent property
+of the query rather than something worth fixing. The byte thresholds that raise Info, Warning, and
+Critical levels are heuristic cutoffs, not measured limits for a given cluster, so treat them as a
+prompt to look rather than a verdict.
+
+
+## Related
+
+- **The mechanism:** [Shuffle](#shuffle)
+- **Tuning parallelism:** [Partitioning](#partitioning)
+
+[^1]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das, Lee, ch. 7
+[^2]: [RDD Programming Guide](https://spark.apache.org/docs/latest/rdd-programming-guide.html)
+[^3]: [Performance Tuning (Spark SQL, DataFrames and Datasets Guide)](https://spark.apache.org/docs/latest/sql-performance-tuning.html)
+[^4]: [Bucketing (The Internals of Spark SQL)](https://books.japila.pl/spark-sql-internals/bucketing/)
+[^5]: [Spark Tips: Partition Tuning](https://luminousmen.com/post/spark-tips-partition-tuning)
+[^6]: [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html#spark-history-server)
+[^7]: [SPARK-30602: Support push-based shuffle to improve shuffle efficiency](https://issues.apache.org/jira/browse/SPARK-30602)
+[^8]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^9]: [How to Tune Your Apache Spark Jobs (Part 2): Cloudera Engineering Blog](https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/)
+[^10]: [Job Scheduling: Dynamic Resource Allocation](https://spark.apache.org/docs/latest/job-scheduling.html)
+[^11]: [Adaptive Query Execution: Speeding Up Spark SQL at Runtime (Databricks)](https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html)
+[^12]: [Performance Tuning: Coalescing Post Shuffle Partitions](https://spark.apache.org/docs/latest/sql-performance-tuning.html)
+[^13]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^14]: [SQLConf: shuffle-partition defaults (Spark source)](https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala)

package/vendor-core/docs-content/tuning/skew.md

@@ -0,0 +1,123 @@
+# Task Skew
+
+<span class="tag">SKEW</span>
+
+## What it is
+
+Task skew happens when a shuffle produces one or a few oversized partitions instead of a
+roughly even split. Spark's own skew-join optimizer defines a partition as skewed once it is
+both larger than a multiple of the median partition size and larger than an absolute byte
+threshold[^1]: in the shipped implementation, more than 5× the median size and more than
+256 MB by default[^2]. Whatever task draws that oversized partition ends up processing far
+more data than everyone else in the same stage.
+
+## How it's detected
+
+| Signal | Warning | Critical |
+|---|---|---|
+| P95 / median task duration | > 3× | > 5× |
+| Max / median task duration (task count < 20) | > 3× | > 5× |
+
+## Why it matters
+
+An oversized partition becomes a [straggler task](#bottleneck-straggler): it's processed inside a single task, so the
+stage can't finish until that task does, no matter how long it takes. Spark's own skew
+handling treats splitting the partition as a deliberate trade-off: reading the other join
+side's matching partition once per split costs extra I/O, but the design behind the feature
+argues that cost is worth paying once the skew is severe enough to be producing a straggler
+in the first place[^1].
+
+## How to fix it
+
+- Enable Adaptive Query Execution's skew-join handling (`spark.sql.adaptive.skewJoin.enabled`,
+  default `true` once `spark.sql.adaptive.enabled` is also on[^2]). It divides any partition that
+  crosses the skew thresholds into several smaller sub-partitions, joins each one against the
+  matching data on the other side, and unions the results back together[^1].
+- Before AQE existed, the only options were manual, and each carries real limitations, which
+  is exactly the gap AQE's skew handling was built to close[^1]: salt the join key, over-size
+  `spark.sql.shuffle.partitions`, or push the [broadcast-join threshold](#bottleneck-broadcast-sizing) up so the join goes
+  broadcast instead of sort-merge.
+- To salt by hand: add a random suffix to the join key on both sides, exploding the smaller
+  side into one row per salt value so every salted variant of the key still finds a match, then
+  join on the combined (key, salt) pair[^3].
+- On Databricks, a `SKEW` hint can name the skewed relation and column (and, optionally, the
+  exact skewed values) directly, letting the planner build a skew-aware plan without hand-rolled
+  salting[^4].
+
+> **PySpark:** salting is plain DataFrame code, no special API, just
+> `withColumn("salt", (fn.rand() * n).cast("int"))` on the larger side and an `explode` over
+> the salt range on the smaller side before joining on the combined key.
+
+The AQE toggle, plus the manual salting fallback as runnable code:
+
+```python
+from pyspark.sql import functions as fn
+
+# AQE skew-join handling, active once AQE itself is enabled
+spark.conf.set("spark.sql.adaptive.enabled", "true")
+spark.conf.set("spark.sql.adaptive.skewJoin.enabled", "true")
+
+# Manual salting fallback (pre-AQE): spread the skewed key across N salted variants.
+# N is an example, size it to how badly the key is skewed.
+N = 16
+salted_big = big.withColumn("salt", (fn.rand() * N).cast("int"))
+salted_small = small.withColumn(
+    "salt", fn.explode(fn.array([fn.lit(i) for i in range(N)]))
+)
+joined = salted_big.join(salted_small, ["key", "salt"]).drop("salt")
+```
+
+## Confidence
+
+Validated. These thresholds mirror Spark's own skew-join optimizer, which
+defines a skewed partition by the same style of ratio-plus-absolute test applied here
+to task durations[^1][^2].
+
+The `Stage shape` subsection below is a separate, experimental branch: its heuristics are
+informational stage-shape observations, not a validated finding, so it carries an
+`EXPERIMENTAL` badge and should not be read as a diagnosis on its own.
+
+## Limitations / false-positive risk
+
+A task-duration ratio is a proxy for skew, not proof of it. A single long task can just as
+easily be a garbage-collection pause or a genuinely slow host, and either one inflates the
+max/median ratio without any partition being oversized. Small stages make the number
+jumpy: with only a handful of tasks, one slow outlier moves the median enough to trip the
+threshold on noise alone. Treat the signal as a prompt to look at the stage, not a verdict.
+
+## Stage shape {#bottleneck-stage-shape}
+
+<span class="tag">SHAPE</span> <span class="tag">EXPERIMENTAL</span>
+
+These are experimental, information-only heuristics about a stage's overall shape, not a
+graded finding. They read the relationship between a stage's task count and the cluster's
+resources, and none of them alone means something is wrong.
+
+A stage's task count equals the number of partitions in its output, and each task runs as a
+single thread against one partition, so that count is what caps how much of the cluster the
+stage can actually use. Three shapes are worth noticing:
+
+- **Low parallelism** (task count far below total executor cores). When a stage has fewer
+  tasks than there are core slots to run them in, cores sit idle and the stage cannot put
+  the cluster's CPU to work; the same shape also concentrates more memory pressure into
+  each task's aggregation[^5]. Spark's guidance is to keep parallelism high enough that the
+  cluster does not stay under-used, roughly 2 to 3 tasks per CPU core in general[^6].
+- **Data explosion** (task count far above cores, partitions tiny). Push the count too high
+  and partitions shrink until per-task overhead floods the stage, so the scheduling cost
+  starts to dominate the useful work[^7].
+- **Task and stage skew** (uneven task count or duration). A lopsided split shows up as a
+  few tasks carrying the stage while the rest finish early, which is the same imbalance the
+  graded `skew` finding above tracks by duration ratio.
+
+## Related
+
+- **Why it happens:** [Partitioning](#partitioning), [Join Optimization](#joins)
+- **How Spark fixes it automatically:** [Adaptive Query Execution](#aqe)
+
+[^1]: [SPARK-29544: Optimize Skewed Join at Runtime](https://issues.apache.org/jira/browse/SPARK-29544)
+[^2]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^3]: [Spark Tips: Partition Tuning](https://luminousmen.com/post/spark-tips-partition-tuning)
+[^4]: [Skew Join Hint](https://docs.databricks.com/aws/en/archive/legacy/skew-join)
+[^5]: [How to Tune Your Apache Spark Jobs (Part 2)](https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/)
+[^6]: [Tuning - Spark](https://spark.apache.org/docs/latest/tuning.html)
+[^7]: [Spark Partitions](https://luminousmen.com/post/spark-partitions)

package/vendor-core/docs-content/tuning/slow-host.md

@@ -0,0 +1,117 @@
+# Slow Host
+
+<span class="tag">HOST</span>
+
+## What it is
+
+A slow host bottleneck shows up when one machine in the cluster consistently turns in slower
+task times than its peers, independent of any single task's own data size. Unlike a stray
+[straggler task](#bottleneck-straggler), the effect is host-wide: every task Spark schedules there runs behind, which
+drags out the stage even when the workload itself is partitioned evenly.
+
+## How it's detected
+
+Spark's event log carries the per-task detail behind this: turning on `spark.eventLog.enabled`
+logs the events that encode what the UI displays, persisted to storage[^1], and that same log
+backs the UI's Stages tab, which drills down into individual tasks and shows per-task metrics
+such as duration, GC time, and shuffle bytes read[^2].
+
+A host reads as slow against the following synthetic threshold: host mean task
+duration ≥ 2× overall median AND the host holds ≥ 20% task share; the signal only applies with
+`taskCount ≥ 15` and `hosts.length ≥ 3`.
+
+## Why it matters
+
+A host running persistently slow tasks behaves like a bottleneck baked into the cluster rather
+than into the workload: every task Spark places there inherits the delay, and since a stage's
+completion time is bounded by its slowest tasks, the rest of the cluster idles while the
+affected host catches up. Left alone, the same host keeps dragging down every later stage and
+job that lands work on it.
+
+## How to fix it
+
+- Turn on speculative execution (`spark.speculation`, off by default) so Spark relaunches a
+  copy of a task that's lagging far behind its peers instead of waiting on the slow host to
+  finish it. A stage only becomes eligible once `spark.speculation.quantile` (default `0.9`) of
+  its tasks have completed, and a task then qualifies once it runs more than
+  `spark.speculation.multiplier` (default `3`) times the median duration, subject to a
+  `spark.speculation.minTaskRuntime` floor (default `100ms`) so short tasks aren't speculated
+  purely for looking slow, or, since Spark 3.4, an efficiency check
+  (`spark.speculation.efficiency.enabled`, default `true`) that also requires the task's
+  data-processing rate to lag the stage average[^3].
+- If the slow host also happens to hold data locality for the affected tasks, lowering
+  `spark.locality.wait` (default `3s`), or the level-specific `spark.locality.wait.node`,
+  `.rack`, and `.process` overrides, shortens how long Spark waits for a data-local slot
+  before falling back to a less-local executor[^3].
+
+Turn on speculation and, if the slow host holds locality, shorten the wait; defaults shown:
+
+```properties
+# Relaunch tasks stuck on a slow host (speculation is OFF by default)
+spark.speculation=true
+spark.speculation.quantile=0.9          # default; fraction of tasks done before speculation starts
+spark.speculation.multiplier=3          # default; a task must run > 3x the median to be relaunched
+
+# If the slow host holds data locality, wait less before falling back to another executor
+spark.locality.wait=3s                  # default; lower to fall back sooner
+```
+
+## Confidence
+
+The core `durationShare` dimension is validated: it measures host-wide task-duration
+inflation against the cluster median, the signal that separates a genuinely slow machine from
+normal task-time variance. Two secondary dimensions run best-effort and stay experimental. The
+`multiDim` dimension <span class="tag">EXPERIMENTAL</span> folds several per-host signals into
+one score, but that combined heuristic has not been validated. The `storageMemory` dimension
+<span class="tag">EXPERIMENTAL</span> likewise reads host-level memory pressure as a
+contributing factor without validation, so treat both as hints rather than verdicts.
+
+## Limitations / false-positive risk
+
+A host that looks slow is not always a bad node. It can simply hold data locality for the
+tasks scheduled there, or carry one heavy stage that happens to be pinned to it, either of
+which inflates its mean task time without any hardware fault. The multi-dimensional scoring
+that reinforces the core signal is best-effort, so a match warrants a look at what that host was
+actually running before you conclude the machine itself is the problem.
+
+
+## Stage slowness {#bottleneck-stage-slowness}
+
+<span class="tag">SLOW</span> <span class="tag">EXPERIMENTAL</span>
+
+This is an experimental fallback heuristic that shares the slow-host anchor and is suppressed
+whenever `slowHost` fires. It answers a different question: when no single machine is dragging,
+why does one stage still lag the rest of the job? The cause usually traces back to how the work
+was divided rather than where it ran.
+
+**Too few partitions caps parallelism.** Wide stages (joins, groupBy, aggregations) take their
+partition count from `spark.sql.shuffle.partitions`, which sits at a default of 200 whether the
+shuffle moves 20 MB or 500 GB unless you change it[^4]. When that count is small relative to the
+cluster, only a handful of tasks carry the stage while cores sit idle, so it stretches out even
+as better-sized neighbors finish quickly. The first move is to raise its parallelism, aiming for
+at least two or three tasks per CPU core on a data-heavy stage, tuned through
+`spark.default.parallelism` and `spark.sql.shuffle.partitions`[^5].
+
+**Large data volume per task makes each task run long.** That 200-partition default does not
+scale with input size, so a stage handling hundreds of gigabytes hands each task a large slice:
+fewer tasks run at once, per-executor load climbs, and the stage often trips memory errors.
+Roughly 100-200 MB per task tends to work well, and tasks grinding through around 3 GB each
+while spilling are a sign you need more partitions[^4].
+
+**Heavy shuffle and spill inflate the stage's cost.** Shuffles move data across executors so
+rows with the same key land together, and during those map and [shuffle](#shuffle) operations Spark writes
+to and reads from local-disk shuffle files, which is heavy I/O that can become a bottleneck
+under the default configuration[^2]. Volume per task feeds straight into it: once a partition
+outgrows the memory available in an executor, Spark [spills](#bottleneck-spill) part of the data to disk, and spills
+are about the slowest thing a job can do because of the extra disk I/O and garbage collection,
+so the stage finishes but runs far less efficiently than the rest[^4].
+
+## Related
+
+- **Speculation & locality tuning:** [Cluster Tuning](#cluster-config)
+
+[^1]: [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html#spark-history-server)
+[^2]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das, Lee, ch. 7
+[^3]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^4]: [Spark Partitions](https://luminousmen.com/post/spark-partitions)
+[^5]: *Spark: The Definitive Guide*, Chambers & Zaharia (O'Reilly, 2018)