sparkforensics-cli 0.1.0 → 0.2.2

package/vendor-core/docs-content/tuning/small-files.md

@@ -0,0 +1,99 @@
+# Small Files
+
+<span class="tag">SMALLFILES</span>
+
+## What it is
+
+The small-files problem is the cost of managing a large count of tiny files. Writing many
+small files runs up significant metadata overhead, and Spark handles that pattern badly;
+distributed filesystems such as HDFS handle it badly too, which is why it earned the name
+"small file problem"[^1]. The same trap shows up on the read side. Push
+`spark.sql.files.maxPartitionBytes` (default 128 MB, aligned to the [Parquet](#data-formats) block size)
+too low and Spark carves the input into many small partition files, piling on disk I/O and
+the filesystem overhead of opening, closing, and listing directories, all of which are slow
+on a distributed store[^2][^3].
+
+Spark writes one file per output partition, so a job left with 200 partitions writes 200
+files and one left with 3000 writes 3000, even when many of those files hold only a handful
+of rows. That count then punishes every downstream job forced to read thousands of tiny
+files[^4]. The opposite extreme is not free either: files that are too large make it
+inefficient to read a whole block when you only need a few rows[^5].
+
+## How it's detected
+
+Reading the SQL plan surfaces write paths that will emit far more output files
+than the data warrants, keying off the one-file-per-output-partition rule[^4]. The signal is
+a high output-partition count against a modest data volume, which foreshadows a directory
+full of tiny files.
+
+| Signal | What it points to |
+|---|---|
+| Output partition count high vs. data volume | Many tiny files on write |
+| Read partitioning below `spark.sql.files.maxPartitionBytes` (128 MB) | Over-split input, excess filesystem I/O[^2][^3] |
+
+## Why it matters
+
+Every extra file is another open, close, and directory-list operation, and on a distributed
+filesystem those are not cheap[^2]. The count compounds downstream: a stage that scatters
+3000 tiny files hands the next job 3000 files to read back, so the metadata tax is paid twice
+over[^4]. None of it is a single dramatic failure. The waste is spread thin across the file
+count, which is exactly why it goes unnoticed until listing and I/O start to dominate.
+
+## How to fix it
+
+Repartition right before the write so the output lands in fewer, better-sized files.
+
+- `coalesce()` is a shuffle-free merge: it fuses existing partitions with no data movement,
+  so `df.coalesce(100).write.parquet(...)` collapses 3000 tiny files into 100 reasonable
+  ones[^4]. It does not rebalance, though, so uneven inputs stay uneven once lumped together,
+  and pushing it to `coalesce(1)` kills parallelism by forcing one executor to do all the
+  work[^4].
+- `repartition()` is a full reshuffle that buys even distribution at the cost of that
+  [shuffle](#shuffle). Reach for it when the distribution itself needs fixing, not just the file
+  count[^4].
+
+> **PySpark:** both are one-line calls on a DataFrame: `df.coalesce(100)` for a shuffle-free
+> merge, or `df.repartition(100)` when the data also needs rebalancing.
+
+```python
+# Collapse many tiny output files without a shuffle (distribution already even)
+df.coalesce(100).write.parquet(path)
+
+# Full reshuffle to a target count when the distribution itself needs fixing
+df.repartition(100).write.parquet(path)
+```
+
+For runtime control, [Adaptive Query Execution](#aqe) can merge tiny shuffle partitions on its own.
+With `spark.sql.adaptive.enabled` and `spark.sql.adaptive.coalescePartitions.enabled` (default
+true) set, AQE coalesces contiguous shuffle partitions toward
+`spark.sql.adaptive.advisoryPartitionSizeInBytes` (default 64 MB)[^4][^5]. Just mind the
+scope: AQE only engages after the first shuffle, so it fixes shuffle-output partition counts
+but will not repair input-side partitioning or a bad file layout on disk[^6].
+
+## Confidence
+
+The core inference, that one output file per partition times a high
+partition count yields many tiny files, is grounded directly in Spark's write behavior[^4],
+and the mitigation levers (`coalesce()`, `repartition()`, and AQE coalescing) are documented
+Spark features[^4][^5].
+
+## Limitations / false-positive risk
+
+A high file count can be perfectly legitimate: partitioned output deliberately fans data
+across many files by partition column, so a large count there is by design, not a defect.
+And AQE coalescing is not a cure-all here, since it only affects post-shuffle partitions and
+leaves the input file layout untouched[^6]. The signal reflects the shape, not intent, so
+confirm the write is not an intended partitioned layout before acting.
+
+
+## Related
+
+- **Partition sizing:** [Partitioning](#partitioning)
+- **Too many small tasks:** [Tiny Tasks](#bottleneck-tiny-tasks)
+
+[^1]: *Spark: The Definitive Guide*, Chambers & Zaharia, ch. 19
+[^2]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das, Lee, ch. 7
+[^3]: [SQLConf.scala](https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala)
+[^4]: [Spark Partitions](https://luminousmen.com/post/spark-partitions)
+[^5]: [Performance Tuning (Spark SQL, DataFrames and Datasets Guide)](https://spark.apache.org/docs/latest/sql-performance-tuning.html)
+[^6]: [The Apache Spark Optimization Checklist](https://luminousmen.com/post/the-apache-spark-optimization-checklist)

package/vendor-core/docs-content/tuning/spill.md

@@ -0,0 +1,114 @@
+# Memory / Disk Spill
+
+<span class="tag">SPILL</span>
+
+## What it is
+
+Spill happens when a task's share of execution memory runs out during a sort or hash
+aggregation. Each task gets its own `TaskMemoryManager`, which arbitrates the shared execution
+pool across every task running concurrently on an executor; with `n` tasks running
+concurrently, each is allowed to allocate somewhere between `1/(2n)` and `1/n` of total
+execution memory: a soft limit, not a hard stop.[^1] When a task's sort or hash-aggregation
+operator keeps requesting execution-memory pages it can't get, Spark doesn't fail immediately:
+it blocks the requesting task, triggers a spill of that task's in-memory data structure to
+disk, or, in the worst case, throws an `OutOfMemoryError`.[^1]
+
+The spill path runs through Spark's map/shuffle I/O machinery: shuffle partitions created by
+wide transformations like `groupBy()` or `join()` spill to the executors' local disks, at the
+location set by `spark.local.directory`.[^2] SQL physical operators apply the same idea with
+their own guardrails: `SortMergeJoinExec`'s in-memory buffer and the cartesian-product
+operator's buffer both spill once they cross a configured row-count threshold, which by
+default is `spark.shuffle.spill.numElementsForceSpillThreshold`.[^3] Whatever the trigger, the
+resulting spilled bytes are exposed on the task metrics as `memoryBytesSpilled`, visible in the
+Spark UI and event log.[^4]
+
+## How it's detected
+
+Detection: any `memoryBytesSpilled > 0` → Warning. The classification is the actionable
+signal.
+
+Spill classification:
+- `skew`: ≥ 80% of tasks have zero spill; fix: address [task skew](#bottleneck-skew), not memory.
+- `volume`: < 20% of tasks have zero spill; fix: more partitions or more memory.
+- `unclassified`: neither condition; treat as volume.
+
+<img class="light-only" src="../diagrams/spill-classification.svg" alt="How a nonzero memoryBytesSpilled is classified as skew, volume, or unclassified from the share of tasks with zero spill, and the fix each classification points to.">
+<img class="dark-only" src="../diagrams/spill-classification.dark.svg" alt="How a nonzero memoryBytesSpilled is classified as skew, volume, or unclassified from the share of tasks with zero spill, and the fix each classification points to.">
+
+## Why it matters
+
+Spill is the fallback Spark reaches for once a task can no longer get the execution memory
+it's asking for: rather than failing outright, it blocks the task, writes its buffered data to
+disk, or (if that's not enough) throws an `OutOfMemoryError`.[^1] A task that's spilling has
+already given up pure in-memory speed to keep running at all, which is why the skew-vs-volume
+split above is the actionable part of the signal: the two failure shapes call for opposite
+fixes.
+
+## How to fix it
+
+- **Skew**: When only a few tasks spill, the problem is the shape of the data, not the size
+  of the memory pool. `repartition()` is the tool for this: reach for it specifically to fix a
+  lopsided partition distribution or to increase partition count[^5], in contrast to
+  `coalesce()`, which never rebalances skewed data because it just stacks existing partitions
+  together: partitions that were uneven going in are still uneven coming out.[^6] For skewed
+  joins, [Adaptive Query Execution](#aqe) can detect and split[^9] oversized partitions automatically: a
+  partition counts as skewed once it's larger than
+  `spark.sql.adaptive.skewJoin.skewedPartitionFactor` (default `5.0`) times the median
+  partition size, and also larger than
+  `spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes` (default `256 MB`).[^7][^8]
+  Before AQE existed, the manual equivalent was salting: adding a prefix to the skewed keys to
+  make the same key look different, then adjusting the data distribution accordingly.[^9]
+  Databricks' `/*+ SKEW(...) */` hint lets you name the skewed relation and specific key values
+  directly, so the planner targets just those keys instead of relying on automatic
+  detection.[^10]
+- **Volume**: When most tasks spill, the fix is more room to work with: more partitions, more
+  memory, or both. `repartition(n)` guarantees exactly `n` output partitions via a hash
+  shuffle[^11], and since Spark's per-task startup overhead is low (unlike MapReduce's), the
+  general bias is to err toward more partitions rather than fewer.[^12]
+  `spark.sql.files.maxPartitionBytes` (default `128 MB`) is the equivalent lever on the input
+  side, governing how much file data gets packed into each input partition before a shuffle
+  even happens.[^6] On the memory side, execution memory is a fraction of the JVM heap.
+  `spark.memory.fraction` (default `0.6`) sizes the shared execution/storage region as
+  `(JVM heap − 300 MiB) × spark.memory.fraction`[^13]. Because each task's slice of that
+  region is capped by the `TaskMemoryManager` at between `1/(2n)` and `1/n` of the total,[^1]
+  giving executors more memory, or running fewer concurrent tasks on each one, directly raises
+  the ceiling before spill kicks in.
+
+For the **volume** case, give tasks more room: defaults shown, comments say which way to move:
+
+```properties
+# Create more, smaller input partitions before the shuffle (default 128m)
+spark.sql.files.maxPartitionBytes=128m
+
+# Execution/storage share of (JVM heap - 300MiB) (default 0.6);
+# raising it grows execution memory but starves the untracked user-memory region
+spark.memory.fraction=0.6
+```
+
+For the **skew** case (only a few tasks spill), the fix is `df.repartition(n)`, not more memory: see [Partitioning](#partitioning).
+
+## Confidence
+
+The core detection (any `memoryBytesSpilled > 0`) and the skew-vs-volume classification are validated: both the skew branch (most tasks spill nothing, so the shape of the data is the problem) and the volume branch (nearly every task spills, so the pool is too small) map to a well-understood fix, and neither needs further validation. <span class="tag">EXPERIMENTAL</span> When a spill matches neither shape, it falls back to a low-confidence unclassified finding that still requires validation, because there is no confirmed cause to act on; the page defaults it to the volume remedy, but that is a guess, not a diagnosis.
+
+## Limitations / false-positive risk
+
+Some spill is normal. A large aggregation or join can spill by design once its working set outgrows execution memory, and a small spill on a healthy stage rarely repays the effort of chasing it. The raw `memoryBytesSpilled > 0` trigger fires on both the actionable and the routine cases, so treat a small spill as informational until the skew-vs-volume split says otherwise.
+
+## Related
+
+- **Why it happens:** [Memory Management](#memory-model), [Partitioning](#partitioning)
+
+[^1]: [Diving into Spark Memory Management](https://luminousmen.com/post/dive-into-spark-memory)
+[^2]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das, Lee, ch. 7
+[^3]: [SQLConf.scala](https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala)
+[^4]: [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html#spark-history-server)
+[^5]: *Spark: The Definitive Guide*, Chambers & Zaharia, ch. 19
+[^6]: [Spark Partitions](https://luminousmen.com/post/spark-partitions)
+[^7]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^8]: [Performance Tuning (Spark SQL, DataFrames and Datasets Guide)](https://spark.apache.org/docs/latest/sql-performance-tuning.html)
+[^9]: [SPARK-29544: Optimize Skewed Join in SQL Adaptive Execution](https://issues.apache.org/jira/browse/SPARK-29544)
+[^10]: [Skew Join Optimization](https://docs.databricks.com/aws/en/archive/legacy/skew-join)
+[^11]: [RDD.scala](https://raw.githubusercontent.com/apache/spark/v3.5.0/core/src/main/scala/org/apache/spark/rdd/RDD.scala)
+[^12]: [How to Tune Your Apache Spark Jobs (Part 2): Cloudera Engineering Blog](https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/)
+[^13]: [Spark Tuning Guide](https://spark.apache.org/docs/latest/tuning.html)

package/vendor-core/docs-content/tuning/straggler.md

@@ -0,0 +1,103 @@
+# Stragglers
+
+<span class="tag">STRAG</span>
+
+## What it is
+
+A straggler is one task (or a small handful of tasks) that runs far longer than the rest of
+the tasks in its stage, even when the stage is otherwise healthy. Because a stage only
+completes once its last task finishes, a single straggler holds up the whole stage, and every
+other executor sits idle waiting for it to catch up.
+
+## How it's detected
+
+A task counts as a straggler when speculative tasks fired for it, or when more than 5% of the
+tasks in a stage run at least 4× the stage's median task duration; this rule only applies once
+a stage has at least 10 tasks.
+
+"Duration" here is `executorRunTime`: elapsed wall-clock time, not CPU time, and it already
+includes any time the task spent blocked fetching shuffle data[^1]. That matters when tracking
+down why a task is slow: a long `executorRunTime` doesn't necessarily mean more compute
+happened. A [garbage-collection pause](#bottleneck-gc) is counted inside it rather than added on top
+(`jvmGCTime` is a subset of `executorRunTime`, not additive[^1]), and a task waiting on a
+remote shuffle block it needs next shows up in `shuffleReadMetrics.fetchWaitTime`, which only
+counts genuine blocking time, not blocks being prefetched in the background[^1].
+
+The "speculative tasks fired" half of the rule is Spark's own detector: once
+`spark.speculation.quantile` (default `0.9`) of a stage's tasks finish, Spark compares each
+remaining task's duration against `spark.speculation.multiplier` (default `3`) times the
+median of the tasks that already finished, subject to a `spark.speculation.minTaskRuntime`
+floor (default `100ms`) so short tasks aren't flagged just for being slower than a tiny
+median[^2]. Speculation itself is off by default (`spark.speculation` defaults to `false`), so
+this half of the rule only fires on stages where it's been turned on[^2].
+
+Watch for overlap with [task skew](#bottleneck-skew): an unevenly distributed key sends one
+partition far more data than its peers, and that partition's task will trip this same duration
+threshold even though the underlying cause is data volume, not a slow host or a GC pause.
+Check the skew signals before assuming the latter.
+
+## Why it matters
+
+A straggler wastes cluster capacity the same way a skewed stage does: every executor other
+than the one running the slow task finishes early and idles, while total job time still
+tracks the single slowest task. Because a straggler's duration can be inflated by a GC
+pause[^1] or a slow [shuffle](#shuffle) fetch[^1] rather than genuinely more work, it's worth checking
+those angles (and whether the real cause is skew rather than anything task-local) before
+assuming a hardware explanation.
+
+## How to fix it
+
+- Enable speculative execution: `spark.speculation` is `false` by default, so nothing reruns
+  a slow task automatically until it's turned on[^2].
+- Tune the trigger: `spark.speculation.quantile` (default `0.9`) sets how much of the stage
+  must finish before speculation kicks in, and `spark.speculation.multiplier` (default `3`)
+  sets how many times slower than the median a task must be. For stages with very few tasks,
+  `spark.speculation.task.duration.threshold` (available since 3.0.0) gives an absolute-duration
+  trigger instead of relying on the median[^2].
+- Since Spark 3.4, `spark.speculation.efficiency.enabled` (default `true`) adds a filter so a
+  task is only speculated if its data-process rate is below the stage average (times
+  `spark.speculation.efficiency.processRateMultiplier`, default `0.75`) or its duration exceeds
+  `spark.speculation.efficiency.longRunTaskFactor` (default `2`) times the same time threshold.
+  This avoids wasting a duplicate task slot on work that's simply doing more, not running
+  slower[^2].
+- If the cause is really a skewed key, treat it as [task skew](#bottleneck-skew) instead: AQE's
+  skew-join optimization automatically splits a partition once it's larger than
+  `spark.sql.adaptive.skewJoin.skewedPartitionFactor` (default `5.0`) times the median
+  partition size and above `spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes`
+  (default `256 MB`)[^2][^3]. Manual salting (appending a random prefix to the skewed key so
+  it spreads across more partitions) is the pre-AQE fallback, though the design doc behind
+  AQE's skew handling calls salting and other manual approaches limited compared to the
+  automatic option[^4].
+
+> **PySpark:** speculation settings can be set on the session directly, no `spark-submit` flag
+> needed: `spark.conf.set("spark.speculation", "true")`, then the quantile/multiplier
+> equivalents the same way.
+
+Speculation config: defaults shown, plus the absolute-duration trigger for small stages:
+
+```properties
+# Speculatively relaunch straggler tasks (OFF by default)
+spark.speculation=true
+spark.speculation.quantile=0.9          # default; portion of tasks finished before speculation begins
+spark.speculation.multiplier=3          # default; multiple of median duration that marks a task slow
+
+# Absolute-duration trigger for stages with very few tasks (since Spark 3.0)
+spark.speculation.task.duration.threshold=10s   # 10s is an example
+```
+
+## Confidence
+
+Validated.
+
+## Limitations / false-positive risk
+
+The 4x-median rule flags a slow task, but slow is not the same as broken. The same threshold trips on a skewed key that simply has more data to process, or on a task that spent its time in a GC pause rather than doing extra work, so a flagged task is not automatically a slow host. Small stages make this worse: with only a handful of tasks the median is unstable, and one moderately slow task can look like a straggler against a median computed from too few peers.
+
+## Related
+
+- **When the real cause is a skewed key:** [Partitioning](#partitioning), [Adaptive Query Execution](#aqe)
+
+[^1]: [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html#spark-history-server)
+[^2]: [Configuration (Spark)](https://spark.apache.org/docs/latest/configuration.html)
+[^3]: [Optimizing Skew Join (Spark SQL, DataFrames and Datasets Guide)](https://spark.apache.org/docs/latest/sql-performance-tuning.html#optimizing-skew-join)
+[^4]: [SPARK-29544: Optimize Skewed Join at Runtime with New Adaptive Execution](https://issues.apache.org/jira/browse/SPARK-29544)

package/vendor-core/docs-content/tuning/tiny-tasks.md

@@ -0,0 +1,94 @@
+# Tiny Tasks
+
+<span class="tag">TINY</span>
+
+## What it is
+
+Every task pays a fixed placement and serialization cost before it does any real work. Spark
+schedules around data locality rather than moving data to code: "Spark builds its scheduling
+around this general principle" that shipping serialized code is cheaper than shipping data,
+preferring `PROCESS_LOCAL` locality down to `ANY` and waiting a configurable timeout for a busy
+CPU to free up before shipping data to a farther executor[^1]. On top of that placement cost,
+every task also pays a serialization cost for its closure and, if not using Kryo, for its data:
+Java's default `ObjectOutputStream`-based serialization "is flexible but often quite slow, and
+leads to large serialized formats," while Kryo is "significantly faster and more compact than
+Java serialization (often as much as 10x)"[^1]. None of that overhead is large per task: Spark's
+own guidance leans toward more, smaller tasks rather than fewer, larger ones, unlike MapReduce:
+"it's almost always better to err on the side of a larger number of tasks (and thus partitions)...
+The difference stems from the fact that MapReduce has a high startup overhead for tasks, while
+Spark does not"[^2]. But once partitions get small enough, that per-task overhead stops being
+negligible and starts to dominate.
+
+## How it's detected
+
+The failure mode is partitions so small that scheduling and serialization overhead outweighs the
+actual work. With Spark's default of 200 shuffle partitions applied to a dataset of only a few
+megabytes, "those 200 partitions will each get like ten rows. Tasks become microscopic. Most of
+your CPUs will just be sitting there doing nothing, wasting cluster hours"[^3]. The same
+small-partition penalty shows up on the [shuffle-read side](#bottleneck-shuffle): production measurements found "the
+average shuffle block size is only 10s of KBs, which leads to delayed shuffle data fetch," and
+the shuffle reduce stages with the largest fetch delays (over 30 seconds per task) were
+consistently the ones with small block sizes[^4]. A high task count paired with very short median
+task duration, and/or very small shuffle block sizes on the read side, are the signals to look
+for.
+
+## Why it matters
+
+Tiny tasks waste cluster capacity even though no single task is a straggler the way
+[skewed](#bottleneck-skew) or [straggler](#bottleneck-straggler) tasks are: the cost here is
+spread evenly across thousands of tasks, each individually cheap but collectively adding up in
+scheduling and serialization overhead, while cores that could be doing other useful work sit
+mostly idle between task launches[^3].
+
+## How to fix it
+
+- `coalesce()` merges existing partitions without a shuffle ("no data movement, no shuffle")
+  and is typically used right before a write, e.g. `df.coalesce(100).write.parquet(...)`, to
+  collapse many tiny output files into a manageable number cheaply[^3]. Because it only stacks
+  partitions together rather than redistributing data, it doesn't fix an uneven underlying
+  distribution (uneven input partitions stay uneven, just grouped) and pushing it too far
+  (`coalesce(1)`) kills parallelism by funneling all work onto one executor while the rest sit
+  idle[^3].
+- `repartition()` performs a full reshuffle, giving control and rebalancing that Spark won't do
+  on its own: with `df.repartition(200)`, "you're paying for predictability"[^3]. Use it when
+  the partitioning itself needs to be fixed or rebalanced, not merely reduced in count, accepting
+  the shuffle cost to get there.
+- There's a middle option, `coalesce(N, shuffle=True)`, which "acts more like a repartition, but
+  leaning toward reduction... not free (you pay for the shuffle cost) but you get better
+  distribution and fewer partitions"[^3].
+- Rule of thumb: reach for `coalesce()` when only the partition count needs to shrink and the
+  existing distribution is already reasonably even (e.g. collapsing output files); reach for
+  `repartition()` when the distribution itself is the problem.
+
+> **PySpark:** both are one-line calls on a DataFrame: `df.coalesce(100)` for a shuffle-free
+> merge, `df.repartition(200)` for a full reshuffle, or `df.coalesce(100, shuffle=True)` for the
+> shuffled middle option.
+
+The three sizing calls, side by side:
+
+```python
+# Collapse many tiny output files without a shuffle (use when distribution is already even)
+df.coalesce(100).write.parquet(path)
+
+# Full reshuffle to a target count, use when the distribution itself needs rebalancing
+df = df.repartition(200)
+
+# Middle ground: reduce partition count but still rebalance (pays a shuffle)
+df = df.coalesce(100, shuffle=True)
+```
+
+## Limitations / false-positive risk
+
+Very short tasks are not always a problem. A tiny final stage can be perfectly fine, and one
+that just writes a small result set doesn't need to be resized. The scheduling-overhead penalty
+only bites when tiny tasks dominate the stage, so treat a handful of short tasks as noise rather
+than a finding.
+
+## Related
+
+- **Partition sizing:** [Partitioning](#partitioning)
+
+[^1]: [Tuning (Spark)](https://spark.apache.org/docs/latest/tuning.html)
+[^2]: [How to Tune Your Apache Spark Jobs (Part 2): Cloudera](https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/)
+[^3]: [Spark Partitions](https://luminousmen.com/post/spark-partitions)
+[^4]: [SPARK-30602](https://issues.apache.org/jira/browse/SPARK-30602)

package/vendor-core/docs-content/tuning/utilization.md

@@ -0,0 +1,90 @@
+# Executor Utilization
+
+<span class="tag">UTIL</span>
+
+## What it is
+
+Executor utilization measures how much of the cluster's allocated executor capacity a job actually keeps busy. When the average number of active executors trails the peak number allocated, the cluster is holding compute (cores and memory) that isn't running any tasks.
+
+## How it's detected
+
+The signal is the ratio of average active executors to the peak active executor count observed over the job's lifetime:
+
+| avg active executors / peak | Level |
+|---|---|
+| < 60% | Info |
+| < 40% | Warning |
+| < 20% | Critical |
+
+## Why it matters
+
+Idle executors are allocation you're paying for without getting work done. One common cause is too little task parallelism to occupy the allocated cores: the guidance is to keep at least as many partitions as there are cores across the executors, so no core sits idle[^1]. Parallelism can also be lost by accident rather than by under-partitioning upfront: because `coalesce` is a narrow transformation, reducing partition count with it forces the *entire* upstream stage down to the reduced parallelism, not just the coalesce step, trading a shuffle for lost concurrency[^2]. Separately, under dynamic allocation, a workload with many small tasks can end up over-provisioned: by default it requests enough executors to maximize parallelism for the task count, and with small tasks that can mean some executors "might not even do any work," wasting resources on allocation overhead[^3].
+
+## How to fix it
+
+- Reach for `repartition` (not `coalesce`) when you need to raise partition count or rebalance data: `coalesce` only avoids a shuffle when shrinking partition count, and doing so also drags the entire upstream stage down to the reduced parallelism[^4][^2].
+- Size partitions so the task count is at least the core count available across executors, to avoid leaving cores idle[^1].
+- Under dynamic allocation, lower `spark.dynamicAllocation.executorAllocationRatio` below its default of `1.0` to scale back the number of executors requested for workloads made up of many small tasks[^3].
+- Dynamic allocation requires shuffle tracking, the external shuffle service, or shuffle-block decommissioning to be enabled as a precondition[^3]; without one of them, an executor removed mid-shuffle takes its unfetched shuffle files with it, forcing a recompute[^5], which undercuts using dynamic allocation to shed idle executors in the first place. `spark.dynamicAllocation.shuffleTracking.enabled` defaults to `true` since Spark 3.0, satisfying that precondition without needing a separate external shuffle service[^3].
+
+Restore parallelism after a heavy filter, and rein in over-provisioning for many-small-task jobs:
+
+```python
+# repartition (not coalesce) rebalances and can raise partition count; aim >= total executor cores
+df = df.filter(heavy_predicate).repartition(spark.sparkContext.defaultParallelism)
+
+# Scale back executors requested for many-small-task workloads (default ratio 1.0)
+spark.conf.set("spark.dynamicAllocation.executorAllocationRatio", "0.5")  # 0.5 is an example
+```
+
+## Limitations / false-positive risk
+
+A low average-to-peak ratio isn't always waste. A bursty or I/O-bound job legitimately holds executors while tasks wait on external systems rather than burning cores, and the ratio is sensitive to short stages, where a brief spike in allocation skews the average without meaning the cluster was genuinely idle.
+
+## Caching opportunity
+
+<span class="tag">CACHE</span>
+
+When the same DataFrame, RDD, or input is scanned more than once, low utilization can trace back to repeated recomputation rather than idle cores. Spark keeps nothing between actions: transformations only build a DAG, and once an action finishes its intermediate results are discarded[^6]. Call a second action on the same logic and Spark re-runs the whole DAG from the source, which can mean re-reading a terabyte from S3, re-reading Kafka, or repeating expensive decompression[^6]. Fork that logic into two pipeline branches and you sign up to recompute everything twice[^6].
+
+<img class="light-only" src="../diagrams/duplicate-plan-subtree.svg" alt="How two branches that repeat the same scan and operators each recompute it, until a shared cached or reused node lets both read one materialized result.">
+<img class="dark-only" src="../diagrams/duplicate-plan-subtree.dark.svg" alt="How two branches that repeat the same scan and operators each recompute it, until a shared cached or reused node lets both read one materialized result.">
+
+[`cache()` and `persist()`](#caching) are how you stop that. Persisting materializes the RDD (usually in memory on the executors) so it can be reused within the job, while Spark keeps its lineage to recompute any lost partition[^7]. For a dataset you read repeatedly, this is one of the most useful optimizations available: it parks a DataFrame, table, or RDD in temporary storage across the executors and makes later reads fast[^8]. Materialization is lazy and per-block, so only an action that touches every partition (`count()` or a full write) caches all of it; a subset scan like `take(10)` leaves a partial cache[^6].
+
+`cache()` is shorthand; `persist()` takes a `StorageLevel` and lets you pick the tradeoff:
+
+| Level | Tradeoff |
+|---|---|
+| `MEMORY_ONLY` | Fastest reads (deserialized JVM objects), but partitions that don't fit are recomputed on the fly each time[^8]. |
+| `MEMORY_AND_DISK` | Spills the overflow to disk instead of recomputing; Spark's default caching strategy and fine for most pipelines[^9]. For DataFrames, `.cache()` maps to this[^6]. |
+| Serialized (`_SER`) | Byte arrays instead of raw objects: smaller footprint, slower reads. Reach for it when memory is tight[^9]. |
+| Replicated (`_2`) | A second copy for fast fault recovery instead of waiting on recomputation, at twice the space[^10][^9]. |
+
+Whether to spill to disk hinges on recomputation cost: reading a block back from disk only beats recomputing it when the work that produced the data is expensive or filters out a large fraction, so the RDD guide says don't enable disk otherwise[^10].
+
+Caching is not free, and several cases don't warrant it:
+
+- **Single use.** Caching adds serialization, deserialization, and storage cost, so caching data you read once only slows you down[^8].
+- **Data larger than storage memory,** or a cheap transformation that isn't reused often regardless of size[^11].
+- **Memory pressure.** Cache memory is memory taken from processing, and under the default spill strategy cached data can land on slower disk, so caching can cost more than just re-reading the source[^9].
+- **Lost optimizer freedom.** Once a dataset is cached, Catalyst works on the in-memory copy and can no longer push filters down to the source[^9].
+
+A persisted RDD you've stopped using still occupies memory until the app ends or eviction forces it out, so call `unpersist()` to reclaim it deliberately, which matters most on shared clusters[^9].
+
+## Related
+
+- **Task parallelism:** [Partitioning](#partitioning)
+- **Executor sizing:** [Cluster Tuning](#cluster-config)
+
+[^1]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das & Lee, ch. 7
+[^2]: *High Performance Spark, 2nd Edition*, Karau, Polak & Warren, ch. 8
+[^3]: [Configuration — Spark](https://spark.apache.org/docs/latest/configuration.html)
+[^4]: *Spark: The Definitive Guide*, Chambers & Zaharia, ch. 19
+[^5]: [Job Scheduling — Dynamic Resource Allocation](https://spark.apache.org/docs/latest/job-scheduling.html)
+[^6]: [Explaining the Mechanics of Spark Caching](https://luminousmen.com/post/explaining-the-mechanics-of-spark-caching)
+[^7]: *High Performance Spark, 2nd Edition*, Karau, Polak & Warren, ch. 2
+[^8]: *Spark: The Definitive Guide*, Chambers & Zaharia, ch. 19
+[^9]: [Spark Tips: Caching](https://luminousmen.com/post/spark-tips-caching)
+[^10]: [RDD Programming Guide — Spark](https://spark.apache.org/docs/latest/rdd-programming-guide.html)
+[^11]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das & Lee, ch. 7

package/vendor-core/docs-site-config.js

@@ -1,23 +1,16 @@
 import { typeTag } from './format-utils.js';
 
-// docs-config.ts's own header comment scopes it to "the single source of the
-// [vendor] docs-panel URL surface"; this is the equivalent single source for
-// docs-site (VitePress) links, kept in its own file so the two unrelated doc
-// systems don't blur into one.
+// Single source for docs-site (VitePress) links, kept separate from docs-config.ts's vendor
+// docs-panel surface so the two doc systems don't blur.
 //
-// The `.html` extension matters: it's what makes this link resolve in the
-// packaged `server/` deploy mode. That static file server
-// (packages/server/lib/static-files.js) matches a request path to a file exactly, with
-// no directory-index or extension-guessing fallback (unlike the Vite
-// dev-server's docs-site middleware in vite.config.ts, which tries
-// bare/`.html`/`index.html` candidates). VitePress's default build
-// (`cleanUrls` unset, i.e. false) names each page's output file `<slug>.html`,
-// so the extension-less form would 404 there.
+// The `.html` extension matters: it's what makes this link resolve in the packaged server/ deploy
+// mode, whose static file server matches a request path to a file exactly (no extension guessing).
+// VitePress names each page <slug>.html, so the extension-less form would 404 there.
 //
-// No allowlist/gating function is needed here, unlike `isKnownDocAnchor` for
-// the vendor link: `tests/docs-site-tag-coverage.test.js` already fails CI if
-// any `TYPE_TAG_MAP` value lacks a documented `{#tag}` heading, and both the
-// map and the page live in this same repo, so a mismatch can't ship past CI.
+// No allowlist needed (unlike isKnownDocAnchor): docs-site-tag-coverage.test.js fails CI if any
+// TYPE_TAG_MAP value lacks a documented {#tag} heading.
+//
+// Relative, not /docs/: keeps working under any subpath the app is published at.
 export function findingGuideUrl(type        )         {
-  return `/docs/user-guide/understanding-findings.html#${typeTag(type).toLowerCase()}`;
+  return `docs/user-guide/understanding-findings.html#${typeTag(type).toLowerCase()}`;
 }

package/vendor-core/efficiency-model.js

@@ -1,6 +1,5 @@
-// §5 Efficiency/wastage model (SparkLens EfficiencyStatisticsAnalyzer).
-// DESIGN SPIKE: decomposes available compute-hours into driver-bound vs.
-// executor-bound waste, plus two theoretical floors. Mapped onto computeWallClock.
+// §5 Efficiency/wastage model. DESIGN SPIKE:
+// decomposes available compute-hours into driver-bound vs executor-bound waste, plus two floors.
 import { computeWallClock } from './wall-clock.js';
 import { computeTotalCores } from './core-count.js';
                                                                              
@@ -18,13 +17,9 @@ export function computeEfficiencyModel({ app, stages, executorsAdded, runAggrega
                           
                                               
   {
-  // `app ?? {}`: computeTotalCores just falls back to the executor-derived core sum when
-  // `resources` is absent, and every caller here already tolerates a null `app` (malformed
-  // logs with no ApplicationStart); `app!` would crash on `app.resources` in that case.
-  // `executorsAdded` cast: computeTotalCores only reads `totalCores`, present on
-  // ExecutorAddedEvent (the only kind real callers pass here) but not ExecutorRemovedEvent,
-  // so the union as a whole is a structural mismatch against computeTotalCores's
-  // `{totalCores?}` shape.
+  // `app ?? {}`: computeTotalCores falls back to the executor core sum when resources is absent,
+  // and callers tolerate a null app (malformed logs); `app!` would crash on app.resources.
+  // Cast: computeTotalCores reads only totalCores, absent on ExecutorRemovedEvent, so the union mismatches.
   const totalCores = computeTotalCores(app ?? {}, executorsAdded                                  );
   const appDurationMs = (app?.endTime ?? 0) - (app?.startTime ?? 0);
   const wc = computeWallClock(app, stages                                                               );
@@ -35,9 +30,8 @@ export function computeEfficiencyModel({ app, stages, executorsAdded, runAggrega
   const driverIdleMs = wc.startup + wc.gaps + wc.idle;
   const driverWasteHours = totalCores * (driverIdleMs / 3600000);
 
-  // Executor-bound: during stagesActive, allocated capacity beyond busy cores.
-  // Tasks only run during active windows, so whole-run busyCoreMs already lives
-  // inside stagesActive; no separate per-window sweep needed.
+  // Executor-bound: allocated capacity beyond busy cores during stagesActive. Tasks only run in
+  // active windows, so whole-run busyCoreMs already lives inside stagesActive.
   const allocatedActiveCoreMs = totalCores * wc.stagesActive;
   const busyCoreMs = runAggregates?.busyCoreMs ?? 0;
   const executorWasteHours = Math.max(0, allocatedActiveCoreMs - busyCoreMs) / 3600000;

package/vendor-core/etl-phases.js

@@ -1,8 +1,6 @@
-// §7 ETL-phase time attribution (Onehouse Spark Analyzer). Heuristic and
-// approximate: classify each stage by byte-flow shape. A stage can be both
-// Transform and Load (shuffle + write), so buckets overlap and need not sum to
-// wall-clock. Storage-format-aware attribution (Hudi/Delta/Iceberg) is NOT
-// portable (no table-format metadata in event logs) and is out of scope.
+// §7 ETL-phase time attribution (Onehouse Spark Analyzer). Heuristic: classify each stage by
+// byte-flow shape. A stage can be both Transform and Load, so buckets overlap and need not sum to
+// wall-clock. Storage-format-aware attribution (Hudi/Delta/Iceberg) isn't portable, out of scope.