sparkforensics-cli 0.1.0 → 0.2.2

package/vendor-core/docs-content/chapters/11-cluster-config.md

@@ -0,0 +1,170 @@
+# Cluster Tuning
+
+## Sizing executors and containers
+
+Cluster tuning is the set of decisions that map an application's cores, memory, and
+executor count onto the underlying cluster (YARN or Kubernetes) plus a handful of
+related knobs (data locality, dynamic allocation, per-task CPU reservation) that all
+interact with that sizing decision.
+
+The starting point is executor sizing. Working through a concrete example (a 10-node
+cluster with 16 cores and 64GB RAM per node), the derivation runs: assign 5 cores per
+executor; reserve about 1 core per node for Hadoop/YARN/OS daemons, leaving 15 usable
+cores per node (150 total); dividing by 5 cores per executor gives 30 executors, minus
+1 reserved for the YARN ApplicationMaster (29); with 3 executors per node, memory per
+executor is 64GB / 3 ≈ 21GB, and after subtracting about 7% for YARN memory overhead
+that comes out to roughly 18GB usable, a recommended shape of 29 executors × 5 cores ×
+18GB[^1]. Cloudera's own worked example, on a 6-node cluster with the same 16-core /
+64GB-per-node hardware, lands on the same shape: `--num-executors 17 --executor-cores 5
+--executor-memory 19G`, rather than one "fat" executor per node
+(`--num-executors 6 --executor-cores 15 --executor-memory 63G`)[^2]. Both sources treat
+the resulting task count (executor-cores × num-executors) as the single most
+important tuning lever, since Spark cannot compensate for too little parallelism on its
+own[^2]. `spark.executor.cores` is the config that controls concurrent tasks per
+executor; it defaults to 1 in YARN mode, or to all available cores on the worker in
+standalone mode[^3], and `spark.executor.memory` sets the JVM heap size per
+executor[^2]. Independent of the sizing formula above, the Spark tuning guide recommends
+targeting 2–3 tasks per CPU core across the cluster[^4].
+
+On top of executor memory sits container memory. Spark computes the total
+container/pod allocation as `executorMemoryMiB + memoryOverheadMiB + memoryOffHeapMiB +
+pysparkMemToUseMiB`. On YARN this is what gets allocated for the container, on
+Kubernetes it becomes the pod memory limit[^5]. `spark.executor.memoryOverhead` itself
+defaults to `max(0.1 * executorMemory, 384MB)`[^5][^3].
+
+<img class="light-only" src="diagrams/container-memory.svg" alt="Container memory sums executor heap, memory overhead, off-heap size, and pyspark memory into the requested container size; the resource manager grants a container of that size and OOMKills the executor when its actual runtime footprint spills past that limit.">
+<img class="dark-only" src="diagrams/container-memory.dark.svg" alt="Container memory sums executor heap, memory overhead, off-heap size, and pyspark memory into the requested container size; the resource manager grants a container of that size and OOMKills the executor when its actual runtime footprint spills past that limit.">
+
+> **PySpark:** `spark.executor.pyspark.memory` is only added as its own
+> container-memory term when it's set explicitly; otherwise PySpark's memory use is
+> folded into the general overhead budget rather than tracked separately.[^5]
+
+On Kubernetes specifically, `spark.kubernetes.executor.request.cores` and
+`spark.kubernetes.executor.limit.cores` take priority over `spark.executor.cores` for
+the pod's CPU request/limit sent to the Kubernetes scheduler[^6], while
+`spark.executor.cores` remains the config Spark itself uses to size the number of
+concurrent task slots[^3].
+
+Two more knobs round out the picture. `spark.locality.wait` (default `3s`) controls how
+long a task waits for a data-local placement before Spark gives up and schedules it less
+locally, stepping through the same wait across process-local → node-local → rack-local →
+any; each level can be overridden independently via `spark.locality.wait.process`,
+`.node`, and `.rack`, all of which default to the base wait when left unset[^3]. Dynamic
+allocation lets Spark add and remove executors as work changes, but it requires one of
+several supporting mechanisms: an external shuffle service, shuffle tracking
+(`spark.dynamicAllocation.shuffleTracking.enabled`, default `true` since Spark 3.0),
+shuffle-block decommission, or the experimental sort-IO plugin[^3]. Removing an
+executor can otherwise destroy shuffle state it's holding. Finally, `spark.task.cpus`
+(default `1`) sets how many cores each task reserves; the number of concurrent task
+slots per executor is derived jointly from it and `spark.executor.cores`, roughly
+`spark.executor.cores / spark.task.cpus`[^3].
+
+## Spotting a bad sizing decision
+
+Getting that sizing wrong shows up in a handful of specific symptoms. The clearest sign of an under-sized executor layout is HDFS throughput dropping under
+load: "HDFS client has trouble with tons of concurrent threads. It was observed that
+HDFS achieves full write throughput with ~5 tasks per executor"[^1]. The "fat executor"
+case (one executor per node, using all 16 cores) shows the failure mode directly:
+"with all 16 cores per executor... HDFS throughput will hurt and it'll result in
+excessive garbage [collection]"[^1].
+
+On the memory side, an executor or pod that dies with no Spark-level error is a strong
+signal that off-heap memory was never added into the container budget. Enabling
+`spark.memory.offHeap.enabled=true` with `spark.memory.offHeap.size=1g` on top of an 8G
+executor with default overhead (819MB) means real usage is 8192+819+1024 = 10,035MB,
+while the container was only granted 8192+819 = 9,011MB. The executor gets killed by
+YARN, or OOMKilled by the Kubernetes kubelet, with no warning from Spark itself[^5].
+
+With dynamic allocation on, unexplained shuffle recomputation is a detectable symptom
+of executors being reclaimed mid-shuffle: "In the event of stragglers... dynamic
+allocation may remove an executor before the shuffle completes, in which case the
+shuffle files written by that executor must be recomputed unnecessarily"[^6]. Jobs that
+abort with a serialized-result-size error are hitting the `spark.driver.maxResultSize`
+guardrail rather than a driver heap exhaustion: "Jobs will be aborted if the total size
+[of serialized action results] is above this limit"[^3]. And on the resource-waste side,
+executors that are provisioned but never assigned work are a sign that dynamic
+allocation is targeting full parallelism against a workload made of many small tasks[^3].
+
+## What a bad sizing decision costs
+
+Each symptom above carries a specific price tag. Task count (`executor-cores × num-executors`) is treated by both cited sources as the
+single most important tuning lever, since Spark cannot compensate for too little
+parallelism on its own[^2]. Going the other way, cramming too many cores into one
+executor degrades HDFS throughput and drives up garbage collection[^1].
+
+Container-memory misconfiguration matters because the failure is silent from Spark's
+point of view: off-heap memory that isn't accounted for in `spark.executor.memoryOverhead`
+causes the executor to actually use more memory than the container/pod was granted, so
+it gets killed externally (by YARN or the kubelet) with no Spark-level diagnostic to
+point at the real cause[^5].
+
+Dynamic allocation's interaction with shuffle state matters because, before dynamic
+allocation existed, an executor exiting alongside its application meant all its state
+could be safely discarded; with dynamic allocation, the application keeps running after
+an executor is explicitly removed, so any later need for that executor's state forces a
+recompute[^6]. That is exactly why Spark needs "a mechanism to decommission an executor
+gracefully by preserving its state before removing it"[^6]. Without shuffle tracking,
+an external shuffle service, or shuffle-block decommission enabled, dynamic allocation
+either can't be turned on at all, or, if it's active regardless, an executor holding
+unpreserved shuffle output that gets removed forces exactly that unnecessary
+recompute[^3][^6].
+
+On the driver side, `spark.driver.maxResultSize` and `spark.driver.memory` are separate
+budgets that still interact: whether a high `maxResultSize` actually causes an
+out-of-memory error "depends on spark.driver.memory and memory overhead of objects in
+JVM"[^3]. Raising one without considering the other doesn't fully protect the driver.
+
+Finally, over-provisioning executors against a small-task workload wastes cluster
+resources: "with small tasks this setting can waste a lot of resources due to executor
+allocation overhead, as some executor might not even do any work"[^3].
+
+## Sizing the cluster correctly
+
+Avoiding those costs starts from a formula, not one fat executor per node. Apply the balanced-executor formula instead; the worked
+examples above give shapes like 29 × 5 × 18GB for a 10-node/16-core/64GB cluster, or
+17 × 5 × 19G for a 6-node cluster with the same per-node hardware[^1][^2]. Keep
+`--executor-cores` at or below roughly 5 so HDFS client concurrency stays in the range
+where it sustains full write throughput[^1].
+
+When enabling off-heap memory, account for `spark.memory.offHeap.size` in the
+container/pod memory budget explicitly; `spark.executor.memoryOverhead`'s default of
+`max(0.1 * executorMemory, 384MB)` does not include it[^5]. If you're still referencing
+the older `spark.yarn.executor.memoryOverhead` name, note it was removed in Spark
+3.0[^5].
+
+On Kubernetes, remember that `spark.kubernetes.executor.request.cores` and
+`.limit.cores` govern the pod's CPU request/limit and take priority over
+`spark.executor.cores` for that purpose[^6], while `spark.executor.cores` separately
+governs Spark's own task-slot count[^3]. The two need to be reasoned about together
+rather than assumed to be redundant.
+
+Tune `spark.locality.wait` (and its per-level overrides) upward when tasks are
+long-running and locality is poor, since the default is tuned for typical workloads[^3];
+set `spark.locality.wait.node` to `0` to skip straight to rack locality when node
+locality isn't achievable[^3].
+
+Turn on shuffle tracking (`spark.dynamicAllocation.shuffleTracking.enabled`, default
+`true` since Spark 3.0) or the external shuffle service so dynamic allocation can
+reclaim executors without losing shuffle output or forcing recomputation[^3][^6]. The
+external shuffle service additionally lets persisted RDD blocks survive executor
+removal when `spark.shuffle.service.fetch.rdd.enabled` is set, and executors holding
+cached blocks are, by default, never removed at all, tunable via
+`spark.dynamicAllocation.cachedExecutorIdleTimeout`[^6].
+
+Set `spark.driver.maxResultSize` as a guardrail on serialized action-result size, and
+size `spark.driver.memory` with it in mind rather than in isolation[^3]. Use
+`spark.dynamicAllocation.executorAllocationRatio` (default `1.0`, added in 2.4.0) to
+scale down over-allocation when tasks are small; for example, a value of `0.5` halves
+the target executor count dynamic allocation would otherwise compute[^3]. Raise
+`spark.task.cpus` above its default of `1` when a task needs more than one core; doing
+so proportionally reduces concurrent task slots per executor
+(`spark.executor.cores / spark.task.cpus`)[^3].
+
+## Sources
+
+[^1]: [Distribution of Executors, Cores and Memory for a Spark Application](https://raw.githubusercontent.com/spoddutur/spark-notes/master/distribution_of_executors_cores_and_memory_for_spark_application.md)
+[^2]: [How to Tune Your Apache Spark Jobs (Part 2)](https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/)
+[^3]: [Configuration — Spark](https://spark.apache.org/docs/latest/configuration.html)
+[^4]: [Spark Tuning Guide](https://spark.apache.org/docs/latest/tuning.html)
+[^5]: [Dive into Spark Memory](https://luminousmen.com/post/dive-into-spark-memory)
+[^6]: [Job Scheduling — Dynamic Resource Allocation](https://spark.apache.org/docs/latest/job-scheduling.html)

package/vendor-core/docs-content/chapters/12-anti-patterns.md

@@ -0,0 +1,171 @@
+# Anti-Patterns
+
+Practitioner checklists on Spark performance converge on a recurring handful of mistakes
+rather than exotic ones. Each entry below covers what the mistake is, how to detect it,
+why it hurts, and how to fix it.
+
+1. **Staying on the RDD API instead of DataFrame/Dataset**
+    - **What:** Building jobs on the RDD API (or RDD-based MLlib) instead of the
+      DataFrame/Dataset API.
+    - **Detect:** Code that drives its logic through `.map`/`.filter` lambdas over RDDs, or
+      that calls into the RDD-based MLlib API.
+    - **Why:** RDDs are lambda-driven, so Spark cannot see inside the closures. It can't apply
+      predicate pushdown, filter reordering, Adaptive Query Execution, or cost-based join
+      reordering. The RDD-based MLlib API is in maintenance mode for the same reason.[^1]
+    - **Fix:** Use the DataFrame/Dataset API so Catalyst can optimize the query plan.
+
+2. **Reading CSV/JSON without an explicit schema**
+    - **What:** Reading raw CSV or JSON files without supplying an explicit schema.
+    - **Detect:** A read step that spends noticeable time before any real work starts, because
+      Spark is inferring types from the data itself.
+    - **Why:** Schema inference forces a full scan of the data just to determine types, and
+      CSV/JSON don't support the column pruning, predicate pushdown, or stats-based file
+      skipping that a columnar format does.[^1]
+    - **Fix:** Supply an explicit schema, or prefer Parquet for analytical workloads, where column
+      pruning, predicate pushdown, and stats-based file skipping work out of the box.[^1]
+
+3. **Compressing input files with a non-splittable codec (GZIP)**
+    - **What:** Storing input data as large GZIP files.
+    - **Detect:** One task/executor taking far longer than the rest on a stage reading
+      GZIP-compressed input, while other executors sit idle.
+    - **Why:** A GZIP file can't be split across executors, so a single node has to decompress
+      the whole file alone.[^1]
+    - **Fix:** Use a splittable codec instead: Snappy, LZ4, or ZSTD.[^1]
+
+4. **Storing data as JSON instead of a binary columnar/row format**
+    - **What:** Persisting datasets as JSON rather than a binary columnar or row format.
+    - **Detect:** Read-heavy jobs where a large, recurring share of stage time goes to parsing
+      the same JSON structures on every read.
+    - **Why:** JSON has to be re-parsed from text on every single read.[^2]
+    - **Fix:** Store data as Avro, Parquet, Thrift, or Protobuf structs in a sequence file to
+      avoid the repeated parsing cost.[^2]
+
+5. **Not registering custom classes with Kryo**
+    - **What:** Using Kryo serialization without registering the application's custom classes.
+    - **Detect:** Larger-than-expected serialized record sizes, and poor efficiency when using a
+      serialized cache storage level such as `MEMORY_SER`.
+    - **Why:** Unregistered classes increase serialized-record size and hurt serialized cache
+      storage levels.[^2]
+    - **Fix:** Register the application's custom classes with Kryo.
+
+6. **Calling `collect()` on a large DataFrame**
+    - **What:** Materializing an entire DataFrame into driver memory with `collect()`.
+    - **Detect:** A driver `OutOfMemoryError`, or a job that aborts once
+      `spark.driver.maxResultSize` (default 1 GB) is exceeded. Spark also logs a warning once a
+      single task's serialized result passes roughly 1 MB.[^3][^1]
+    - **Why:** `collect()` gathers every `Row` from every partition into a single in-memory
+      `Array` that lives entirely on the driver, not the cluster. The driver JVM has to hold
+      the whole result set in its own heap.[^3][^4][^5][^6]
+    - **Fix:** Use aggregations or `take(n)` instead of collecting the full result set.
+
+      > **PySpark:** `toPandas()` performs the same driver-side collection as `collect()` for
+      > Python users, so avoid it on large DataFrames too.[^6]
+
+7. **Assuming `cache()` + `count()` guarantees the DataFrame stays fully cached**
+    - **What:** Treating `df.cache(); df.count()` as a guarantee that the DataFrame remains
+      fully persisted for later actions.
+    - **Detect:** Later actions on a supposedly-cached DataFrame recompute from source instead
+      of hitting the cache. There's no built-in visibility into how much of a DataFrame is
+      still actually cached, so this typically only surfaces as unexpectedly slow
+      recomputation.[^7]
+    - **Why:** `count()` does force every partition to be computed and attempted for caching,
+      unlike `take`/`limit`, which only touch the partitions they need.[^5][^7] But partitions
+      can't be fractionally cached: if there isn't room for all of them, the ones that don't
+      fit are silently dropped and recomputed on next access.[^5] Cached blocks also compete
+      with execution for the same memory pool and can be evicted under pressure without
+      warning, and caching is tied to the *analyzed* (pre-optimization) logical plan, so a
+      semantically identical query with a different analyzed plan bypasses the cache entirely
+      and recomputes from source.[^7] Losing an executor drops any cached block that wasn't
+      stored with a replicated storage level like `MEMORY_AND_DISK_2`.[^7]
+    - **Fix:** Keep using `.cache()` followed by `.count()` to force materialization, but don't
+      assume that guarantees persistence (there's no built-in way to confirm how much of the
+      DataFrame is actually cached[^7]), and use a replicated storage level if losing a cached
+      partition to executor failure would be costly.
+
+8. **Row-at-a-time Python UDFs**
+    - **What:** Writing Python UDFs that operate one row at a time instead of vectorized Pandas
+      UDFs.
+    - **Detect:** A UDF-heavy stage where most of the time goes to serialization rather than
+      actual computation.[^9]
+    - **Why:** The overhead is serialization plus per-row JVM↔Python data movement, not raw
+      Python interpreter speed. Row-at-a-time UDFs "suffer from high serialization and
+      invocation overhead."[^8] RDD-era Python UDFs pay a *double* serialization cost: Java/Scala
+      objects are serialized, then re-serialized to Python via `cloudpickle` and back on every
+      call.[^9] A Databricks benchmark on a 10M-row DataFrame showed vectorized Pandas UDFs
+      beating row-at-a-time UDFs "across the board, ranging from 3x to over 100x."[^8]
+    - **Fix:** Use vectorized Pandas UDFs instead of row-at-a-time UDFs: once data is
+      transferred via Arrow, there's no need to serialize/pickle it, since it's already in a
+      format consumable by the Python process.[^5]
+
+9. **Leaving `spark.sql.shuffle.partitions` at its default of 200**
+    - **What:** Never tuning `spark.sql.shuffle.partitions` away from its default of 200 for
+      wide transformations (`join`, `groupBy`, aggregations).
+    - **Detect:** In the Spark UI Stages tab, too few partitions for the data size shows up as
+      "Spill (Memory)" or "Spill (Disk)" entries against a stage; too many partitions for the
+      data size shows up as a large task/stage count where each task's duration is dominated by
+      scheduling and bookkeeping, often paired with a large number of tiny output files.[^10]
+    - **Why:** 200 is a fixed default regardless of whether you're joining 5MB or 5TB.[^10][^11]
+      Too few partitions means each one holds more data than fits comfortably in executor
+      memory, which increases load per executor and leads to spills once partition size exceeds
+      available memory.[^10] Too many partitions on a small dataset can shrink tasks to around
+      ten rows each, so "most of your CPUs will just be sitting there doing nothing."[^10]
+    - **Fix:** Increase `spark.sql.shuffle.partitions` if tasks are processing multiple GB each
+      and spilling; decrease it if tasks finish in a couple of seconds and write tiny files. A
+      rough starting target is 100–200 MB of data per task, tuned per dataset.[^10]
+
+10. **Broadcasting a table that's too large**
+    - **What:** Forcing or allowing a broadcast join on a table that's too large to broadcast
+      safely.
+    - **Detect:** A driver `OutOfMemoryError` during a broadcast join.
+    - **Why:** The small side of the join has to be collected onto the driver before it can be
+      broadcast to every executor. That collection step is the same driver-memory operation
+      as `.collect()`, and it's what fails, not executor-side replication or a serialization
+      timeout: "if you try to broadcast something too large, you can crash your driver node
+      (because that collect is expensive)."[^4]
+    - **Fix:** Use `spark.sql.autoBroadcastJoinThreshold` to control the maximum size Spark
+      will broadcast, or increase driver memory.[^4]
+
+11. **Reaching for `coalesce(1)` before every single-file write**
+    - **What:** Always using `coalesce(1)` instead of `repartition(1)` when a single output
+      file is needed.
+    - **Detect:** The entire upstream stage (including filtering/transformation work that
+      would otherwise run in parallel) collapses onto a single task/executor, leaving the
+      rest of the cluster idle.
+    - **Why:** `coalesce` is a narrow transformation, so it "causes the upstream partitions in
+      the entire stage to execute with the level of parallelism assigned by coalesce," fusing
+      all preceding work down to one task.[^6] `repartition` triggers a full shuffle instead,
+      which inserts an explicit shuffle boundary so the upstream stage keeps its original
+      parallelism.[^4][^6]
+    - **Fix:** Use `repartition(1)` when there is meaningful upstream computation you don't
+      want collapsed onto a single executor; reserve `coalesce(1)` for when the upstream stage
+      is already cheap and paying for a shuffle would be wasted cost.[^6]
+
+12. **Not verifying predicate pushdown is actually happening**
+    - **What:** Assuming filters and column projections are pushed down to the data source
+      without checking the physical plan.
+    - **Detect:** Run `.explain()` and look at the `Scan` node for a `PushedFilters` marker.
+      For example, `*Scan JDBCRel... PushedFilters: [*In(DEST_COUNTRY_NAME, [Anguilla,
+      Sweden])]`. Column pruning is verified the same way: a `.select()` on one column should
+      show a narrowed `ReadSchema` at the scan rather than a full-table scan.[^4]
+    - **Why:** Spark pushes down simple filters (column equality, `IN`, `IS NULL`) to JDBC
+      sources automatically, but anything with a computed column or cast won't push down and
+      gets evaluated in Spark after the full read.[^1] Some sources also only partially handle
+      a filter and leave Spark to re-evaluate it as a safety mechanism; that's still a "good"
+      pushdown since the amount of data read is reduced.[^6]
+    - **Fix:** Check `explain` output (paired with `printSchema`)[^6] for `PushedFilters` after
+      writing a filter, and rewrite filters that use casts or computed columns as simple
+      predicates where possible so they can push down.[^4][^1]
+
+## Sources
+
+[^1]: [The Apache Spark Optimization Checklist](https://luminousmen.com/post/the-apache-spark-optimization-checklist)
+[^2]: [How to Tune Your Apache Spark Jobs (Part 2)](https://blog.cloudera.com/how-to-tune-your-apache-spark-jobs-part-2/)
+[^3]: *Advanced Analytics with PySpark*, Tandon, Ryza, Laserson et al., ch. 2
+[^4]: *Spark: The Definitive Guide*, Chambers & Zaharia, chs. 5, 8, 9, 18, 19
+[^5]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das & Lee, chs. 3, 5, 7
+[^6]: *High Performance Spark, 2nd Edition*, Karau, Polak & Warren, chs. 5, 7
+[^7]: [Explaining the Mechanics of Spark Caching](https://luminousmen.com/post/explaining-the-mechanics-of-spark-caching)
+[^8]: [Introducing Vectorized UDFs for PySpark](https://www.databricks.com/blog/2017/10/30/introducing-vectorized-udfs-for-pyspark.html)
+[^9]: [Spark Tips: DataFrame API](https://luminousmen.com/post/spark-tips-dataframe-api)
+[^10]: [Spark Partitions](https://luminousmen.com/post/spark-partitions)
+[^11]: [Configuration — Spark](https://spark.apache.org/docs/latest/configuration.html)

package/vendor-core/docs-content/chapters/14-metrics.md

@@ -0,0 +1,87 @@
+# Metrics Glossary
+
+<!-- #metric-* IDs are docs-internal, forward-compatible; the sibling contract consumes only #bottleneck-*. Rename-safe until the sibling deep-links metrics. -->
+
+Reference for every metric surfaced elsewhere in this guide: what it measures, where Spark records it in the event log, and what a problematic value looks like.
+
+## Task duration (P50/P95/max) {#metric-task-duration}
+
+Per-task wall-clock duration, aggregated across a stage's tasks into percentiles (P50, P95, max) to surface skew and stragglers. It is computed from `SparkListenerTaskEnd` events rather than read off a single field: each task end carries its own duration, and the percentiles are derived by aggregating those values across the stage.
+
+A problematic value looks like a small fraction of tasks running far past the rest: a stage is classified as a straggler when more than 5% of tasks run at least 4x the median duration (with at least 10 tasks in the stage), or when speculative tasks were fired.
+
+## Shuffle read bytes {#metric-shuffle-read-bytes}
+
+The volume of shuffle data a task reads from remote executors, recorded in `taskMetrics.shuffleReadMetrics.remoteBytesRead`.
+
+## Shuffle write bytes {#metric-shuffle-write-bytes}
+
+The size of the shuffle output a task writes, recorded in `taskMetrics.shuffleWriteMetrics.bytesWritten`. Spark's own metric description calls it simply the "Number of bytes written in shuffle operations," without stating compression state on its own[^1].
+
+The sort-based shuffle writer fills in the mechanics: incoming records are serialized as soon as they reach the shuffle writer and buffered in serialized form while sorting[^2]. When the spill compression codec supports concatenating compressed data, the final merge step concatenates the already-compressed spill partitions directly into the output file, using `transferTo` rather than decompressing and recompressing[^2]. So `bytesWritten` counts compressed, post-serialization bytes, and in this common fast-merge path the final written output is built directly from data that had already been spilled to disk during sorting, rather than written fresh at merge time.
+
+## Memory bytes spilled {#metric-memory-bytes-spilled}
+
+Bytes a task spilled from in-memory structures, recorded in `taskMetrics.memoryBytesSpilled`. Any value above zero is a spill warning, with the skew-vs-volume classification (based on what fraction of a stage's tasks show zero spill) as the actionable signal.
+
+## Disk bytes spilled {#metric-disk-bytes-spilled}
+
+The on-disk counterpart to memory spill: bytes a task spilled to disk, recorded in `taskMetrics.diskBytesSpilled`. It feeds the same spill classification as memory bytes spilled.
+
+## JVM GC time {#metric-jvm-gc-time}
+
+Elapsed time the JVM spent in garbage collection while a task executed, recorded in `taskMetrics.jvmGCTime` and expressed in milliseconds[^1]. The value is cumulative across the task's full execution window: the sum of every GC pause that occurred during that task's run, not just the most recent one[^1]. This matches how it's serialized: as a single scalar long, consistent with an accumulator rather than a per-GC-event log entry[^3].
+
+## gcPct {#metric-gcpct}
+
+A synthetic ratio, `jvmGCTime / executorRunTime`, not a raw Spark field. It drives GC-bottleneck classification: above 10% is a warning, above 20% is critical.
+
+## Executor run time {#metric-executor-run-time}
+
+Elapsed time the executor spent running a task, recorded in `taskMetrics.executorRunTime` and expressed in milliseconds[^1]. `TaskMetrics` (and therefore `executorRunTime`) is serialized as an optional part of the `SparkListenerTaskEnd` payload, not guaranteed on every task end[^3]. In practice it is available for failed tasks that got far enough to actually run, such as `ExceptionFailure` or `TaskKilled`. The event-log deserialization code even falls back to reading accumulator updates out of the embedded `TaskMetrics` for old, Spark-1.x-era logs, which only makes sense if the metrics object is normally populated for that failure type[^3]. It can be absent for reasons like `Resubmitted`, where the task attempt never completed on that executor[^3].
+
+## Fetch wait time ratio {#metric-fetch-wait-time-ratio}
+
+A synthetic ratio, `fetchWaitTime / taskDuration`, not a raw Spark field: the time a task spent blocked waiting on remote shuffle blocks, relative to its total duration.
+
+## Input bytes {#metric-input-bytes}
+
+Bytes a task read as input, recorded in `taskMetrics.inputMetrics.bytesRead`.
+
+## Output bytes {#metric-output-bytes}
+
+Bytes a task wrote as output, recorded in `taskMetrics.outputMetrics.bytesWritten`.
+
+## I/O ratio {#metric-io-ratio}
+
+A synthetic ratio, `outputBytes / inputBytes`, not a raw Spark field.
+
+## Peak execution memory {#metric-peak-execution-memory}
+
+Peak memory recorded in `taskMetrics.peakExecutionMemory`. This is a task-level accumulator, distinct from the separate executor-level `peakMemoryMetrics.OnHeapExecutionMemory` and `.OffHeapExecutionMemory` gauges, which report the on-heap and off-heap execution pools as two separate numbers at the executor level rather than as a single per-task figure[^1].
+
+## Failed tasks / failure rate {#metric-failed-tasks}
+
+Tasks whose `SparkListenerTaskEnd` reason is not `Success`. The `Reason` field holds the formatted class name of whichever `TaskEndReason` was assigned to that task end[^3], and the canonical set of reasons includes `FetchFailed`, `ExceptionFailure`, `TaskResultLost`, `TaskKilled`, `TaskCommitDenied`, `ExecutorLostFailure`, and `UnknownReason`[^3]. Because `TaskMetrics` is only an optional part of the task-end payload, it can be absent for some of these reasons: for example `Resubmitted`, where the task attempt never actually completed on that executor[^3].
+
+## Speculative tasks / straggler count {#metric-speculative-tasks}
+
+Tasks launched as speculative retries of a slow-running task, recorded via `SparkListenerTaskStart` where `speculative = true`. Any speculative task firing (or more than 5% of a stage's tasks running at least 4x the median duration, with at least 10 tasks in the stage) is a straggler signal.
+
+## Executor count (added/removed/concurrent) {#metric-executor-count}
+
+The number of executors added, removed, or concurrently running, recorded via `SparkListenerExecutorAdded` / `SparkListenerExecutorRemoved` events.
+
+## Stage wall-clock duration {#metric-stage-duration}
+
+A stage's total elapsed time, recorded from `SparkListenerStageCompleted`: `completionTime − submissionTime`.
+
+## firstStageSubmittedAt {#metric-first-stage-submitted-at}
+
+A synthetic field: the timestamp of the first `SparkListenerStageSubmitted` event in the event log. The gap between this timestamp and the application's start time drives cold-start classification: more than 30 seconds is a warning.
+
+## Sources
+
+[^1]: [Monitoring and Instrumentation](https://spark.apache.org/docs/latest/monitoring.html#spark-history-server)
+[^2]: [SortShuffleManager.scala](https://raw.githubusercontent.com/apache/spark/v3.5.0/core/src/main/scala/org/apache/spark/shuffle/sort/SortShuffleManager.scala)
+[^3]: [JsonProtocol.scala](https://raw.githubusercontent.com/apache/spark/v3.5.0/core/src/main/scala/org/apache/spark/util/JsonProtocol.scala)

package/vendor-core/docs-content/chapters/15-config.md

@@ -0,0 +1,93 @@
+# Spark Config Quick-Reference
+
+A cross-reference of the Spark and PySpark configuration properties discussed elsewhere in this guide, plus the JVM garbage-collection flags relevant to executor tuning. Defaults and version notes below are traced to Spark's own configuration docs, its SQLConf source, and the cited literature. Where a source doesn't give a hard number (a range, a ceiling, a recommended workload profile), the table leaves it out rather than guess.
+
+## Spark properties
+
+| Key | Default | Recommended | Notes |
+|---|---|---|---|
+| `spark.sql.shuffle.partitions` | 200 (unchanged since Spark 1.1.0)[^1][^2][^3] |  | AQE doesn't override this config value; see notes below for the effective runtime count. |
+| `spark.sql.adaptive.enabled` | `true`[^4] | `true` | |
+| `spark.sql.adaptive.coalescePartitions.enabled` | `true` (since 3.0.0)[^1] | `true` | Requires AQE; merges contiguous small post-shuffle partitions instead of one task per configured partition[^1]. |
+| `spark.sql.adaptive.coalescePartitions.initialPartitionNum` | unset → falls back to `spark.sql.shuffle.partitions` (200)[^1] |  | Sets the partition count entering the coalescing step. |
+| `spark.sql.adaptive.coalescePartitions.parallelismFirst` | `true` (since 3.2.0)[^1][^5] | `false` on busy clusters[^1] | When `true`, ignores `advisoryPartitionSizeInBytes` and derives a target from the cluster's default parallelism, respecting only the `minPartitionSize` floor. |
+| `spark.sql.adaptive.advisoryPartitionSizeInBytes` | 64MB (since 3.0.0)[^1][^4][^5] |  | Ignored during coalescing when `parallelismFirst=true` (the default). |
+| `spark.sql.adaptive.coalescePartitions.minPartitionSize` | 1MB (since 3.2.0)[^5] |  | Floor enforced when the target size is ignored: the default `parallelismFirst=true` case[^1]. |
+| `spark.memory.fraction` | 0.6[^6] |  | See notes below; no hard 0–1 ceiling documented in the cited sources. |
+| `spark.memory.storageFraction` | 0.5[^7] |  | A fraction *of* the region sized by `spark.memory.fraction`, not an independent pool; see notes below. |
+| `spark.sql.execution.arrow.pyspark.enabled` | `false`[^8] |  | See PySpark callout below. |
+| `spark.sql.execution.arrow.pyspark.fallback.enabled` | `true` (inherited from the deprecated `arrow.fallback.enabled`)[^8][^5][^4] |  | Falls back to the non-Arrow path automatically if a conversion error occurs, before computation runs[^8]. |
+| `spark.dynamicAllocation.shuffleTracking.enabled` | `true` (since 3.0.0)[^4] |  | Lets dynamic allocation track shuffle files per executor without an external shuffle service. |
+| `spark.dynamicAllocation.shuffleTracking.timeout` | `infinity`[^4] |  | Executors holding shuffle data wait for it to be garbage collected before release, by default; set a finite value if GC isn't keeping up. |
+| `spark.shuffle.service.enabled` |  |  | External shuffle service; see `#config-shuffle-service` below. |
+| `spark.dynamicAllocation.minExecutors` / `maxExecutors` | `0` / `infinity`[^4] |  | Autoscale bounds; see `#config-autoscale-bounds` below. |
+| `spark.serializer` | `org.apache.spark.serializer.JavaSerializer`[^4] | `KryoSerializer` | See `#config-serializer` below. |
+| `spark.executor.memoryOverhead` | greater of 10% of executor memory or 384MB[^7] |  | Off-heap overhead; see `#config-memory-overhead` below. |
+
+### Shuffle partitions and AQE coalescing
+
+The 200 default for `spark.sql.shuffle.partitions` hasn't moved since 1.1.0, and AQE (on by default since Spark 3.0) doesn't change that config value[^1][^2][^3]. What it changes is the effective number of partitions used at runtime. With `coalescePartitions.enabled` also on by default, Spark merges small contiguous post-shuffle partitions instead of running one task per configured partition[^1]. Because `parallelismFirst` defaults to `true` since 3.2.0, that merge target usually isn't the 64MB `advisoryPartitionSizeInBytes` value; it's derived from the cluster's default parallelism, with `minPartitionSize` (1MB) as the only enforced floor[^1][^5]. Databricks' own writeup on AQE shows the reduce-task count actually shrinking based on measured data volume[^9], and Learning Spark accordingly calls the static 200 default "too high for smaller or streaming workloads"[^3]. A related, distinct knob (`spark.sql.adaptive.coalescePartitions.minPartitionNum`) sets a minimum parallelism floor for data that's slow to compute despite being small, per High Performance Spark[^10].
+
+### `spark.memory.fraction` and `spark.memory.storageFraction`
+
+`spark.memory.fraction` sizes the unified execution/storage region (M) as a fraction of (JVM heap − 300MiB); the tuning guide frames it as a knob to fit M "comfortably within the JVM's old or tenured generation" rather than stating an enforced numeric range[^6]. The cited sources document no failure threshold tied to a specific value like 0.9. What they do document is the risk of pushing the fraction high: the complement, `1 − spark.memory.fraction`, is untracked "User Memory" for UDFs, Python/Arrow glue, and native buffers, and starving it (which is what raising the fraction toward 0.9 does) leads to "GC pressure or random OOMs," with no warning from Spark[^7].
+
+`spark.memory.storageFraction` isn't independent of that: it's a fraction *of* M, the same region `spark.memory.fraction` sizes[^6]. Inside that shared pool, execution can evict storage down to the threshold set by `storageFraction`, but storage can never evict execution[^11][^6]. With the defaults (0.6 and 0.5), that works out to execution and storage each getting 30% of usable heap; raising `memory.fraction` scales both pools at once, while `storageFraction` only re-splits the pool that's already been carved out[^7].
+
+> **PySpark:** `spark.sql.execution.arrow.pyspark.enabled` is off by default and governs Arrow use for `DataFrame.toPandas()` and `SparkSession.createDataFrame()` from a Pandas DataFrame or NumPy array[^8]. Its documented risk is a type-coverage gap, not silent data corruption: `ArrayType` of `TimestampType` is explicitly unsupported[^4][^5], and more generally an unsupported column type raises an error rather than converting wrongly[^8]. `spark.sql.execution.arrow.pyspark.fallback.enabled` defaults to `true` and automatically falls back to the non-Arrow path if a conversion error occurs, before any computation runs[^8][^5][^4]. So the failure mode is an exception plus fallback, not a wrong result. Separately, and not specific to this Arrow config, PySpark's own type coercion has its own hazards: numeric values passed for `ByteType`/`ShortType`/`IntegerType` must fall within fixed ranges or get rejected or converted unexpectedly[^2].
+
+### Shuffle service {#config-shuffle-service}
+
+During a shuffle, an executor writes its map output to local disk and then serves fetch requests for that data itself[^14]. Dynamic allocation can reclaim an idle executor before a later stage has fetched all the shuffle blocks it holds (especially with stragglers, tasks that run much longer than their peers), and once that executor is gone, any stage that still needs its shuffle output gets a `FetchFailed` and has to recompute it[^14]. The external shuffle service fixes this by moving shuffle-block serving out of the executor process: it's a long-running process on each node, independent of any particular application's executors, and once `spark.shuffle.service.enabled` is `true`, executors fetch shuffle blocks from it instead of from each other, so an executor's shuffle output keeps being served after that executor is reclaimed[^14][^4]. Setting up dynamic allocation requires enabling this service on every worker node in addition to `spark.dynamicAllocation.enabled`[^2]. As of Spark 3.0, shuffle-file tracking (`spark.dynamicAllocation.shuffleTracking.enabled`) is an alternative to the external shuffle service for the same safe-removal problem[^4].
+
+**Limitations / false-positive risk:** the service can be intentionally left off when dynamic allocation relies on shuffle-file tracking instead, so a disabled setting is not automatically a misconfiguration.
+
+### Autoscale bounds {#config-autoscale-bounds}
+
+By default, `spark.dynamicAllocation.minExecutors` is `0` and `spark.dynamicAllocation.maxExecutors` is `infinity`[^4]. Leaving `maxExecutors` unset therefore leaves dynamic allocation unbounded on the high end as far as Spark itself is concerned; in practice the ceiling ends up being whatever the cluster or scheduler enforces outside Spark[^4]. `spark.dynamicAllocation.initialExecutors` defaults to `minExecutors`, unless `--num-executors` (or `spark.executor.instances`) sets a larger starting value[^4]. Whatever executor count `executorAllocationRatio` computes to maximize parallelism, that target is still clamped by the `minExecutors`/`maxExecutors` floor and ceiling[^4].
+
+**Limitations / false-positive risk:** an unbounded `maxExecutors` is often deliberate on clusters where the scheduler enforces the real ceiling, so an unset high end is not always wrong.
+
+### Serializer {#config-serializer}
+
+The default `spark.serializer` is `org.apache.spark.serializer.JavaSerializer`, which works with any `Serializable` Java object but is "quite slow"; Spark's configuration reference recommends switching to `KryoSerializer` "when speed is necessary"[^4]. The tuning guide is more specific: Kryo is "significantly faster and more compact than Java serialization (often as much as 10x)," though it doesn't support all `Serializable` types and needs its classes registered in advance for best performance[^6]. That registration requirement is the stated reason Kryo isn't the default: "The only reason Kryo is not the default is because of the custom registration requirement," and Spark recommends trying it for any network-intensive application[^6]. Since Spark 2.0.0, Spark internally uses Kryo regardless of this setting when shuffling RDDs of simple types, arrays of simple types, or strings, and auto-registers common Scala classes via Twitter chill's `AllScalaRegistrar`[^6]. A practitioner summary puts Java serialization at "2-10x slower and larger on the wire" on every shuffle, with the caveat that classes left unregistered silently fall back to Java serialization unless registered via `spark.kryo.classesToRegister` or `spark.kryo.registrator`[^15]. Persisting RDDs in serialized form is another case where Kryo can be more space-efficient than Java serialization[^10].
+
+**Limitations / false-positive risk:** `JavaSerializer` may be kept on purpose when an application depends on `Serializable` types Kryo cannot handle, so a non-Kryo setting is not necessarily a mistake.
+
+### Memory overhead {#config-memory-overhead}
+
+`spark.executor.memoryOverhead` covers everything an executor needs outside the JVM heap sized by `spark.executor.memory`: thread stacks, JIT buffers, metaspace, JNI, native libraries, and, if `spark.executor.pyspark.memory` isn't set separately, the memory used by PySpark's per-task Python worker processes[^7]. Left unset, Spark defaults it to the greater of 10% of executor memory or 384MB[^7]; the configuration reference formalizes that floor as `spark.executor.minMemoryOverhead` (default `384m`, since Spark 4.0) and the percentage as `spark.executor.memoryOverheadFactor` (default 0.10, or 0.40 for non-JVM jobs on Kubernetes, since those need more non-JVM heap space)[^4]. The resource manager (YARN's NodeManager, or the Kubernetes kubelet) sizes the container/pod to the sum of `spark.executor.memoryOverhead`, `spark.executor.memory`, `spark.memory.offHeap.size`, and `spark.executor.pyspark.memory`[^4]. If off-heap memory or PySpark worker processes consume memory that isn't reflected in a correspondingly larger `memoryOverhead`, that extra usage still shows up in the process's actual RSS, which the resource manager does track. Once real usage exceeds the container's allocated size, the executor is killed by YARN or OOMKilled by the kubelet, with no Spark-level error, just an abrupt process death[^7].
+
+**Limitations / false-positive risk:** the default overhead is adequate for many JVM-only jobs, so a value left at the default only signals trouble on off-heap-heavy or PySpark workloads that need more non-heap room.
+
+## JVM GC flags
+
+| Flag | Effect |
+|---|---|
+| `-XX:+UseG1GC` | Default collector since Spark 4.0.0, which defaults to JDK 17[^6]. |
+| `-XX:G1HeapRegionSize` | May need raising alongside large executor heaps[^6]. |
+| `-XX:InitiatingHeapOccupancyPercent` | Tuned (with `-XX:ConcGCThreads` and RSet-update settings) to fix a documented ~100-second G1 full-GC pause on an 88GB executor heap[^12]. |
+| `-XX:ConcGCThreads` | See `-XX:InitiatingHeapOccupancyPercent` above[^12]. |
+| `-XX:+UseZGC` | Concurrent, low-latency collector; sub-millisecond pause target independent of heap size, from a few hundred megabytes up to 16TB[^13]. |
+
+### G1 vs. ZGC
+
+G1 was designed as a CMS replacement aiming at both throughput and low latency: it partitions the heap into equal-sized regions and copies out only the live objects from collected regions rather than compacting the whole heap[^12]. Even so, a documented 88GB-heap Spark benchmark hit "unacceptable full GC," with one job pausing nearly 100 seconds under default G1 settings. Only after tuning `InitiatingHeapOccupancyPercent`, `ConcGCThreads`, and RSet-update parameters did G1 beat Parallel/CMS GC on both throughput and latency[^12]. ZGC, per the OpenJDK project page, "performs all expensive work concurrently, without stopping the execution of application threads for more than a millisecond," with pause times that stay flat as heap size scales from a few hundred megabytes up to 16TB[^13].
+
+## Sources
+
+[^1]: [Performance Tuning — Spark SQL, DataFrames and Datasets Guide](https://spark.apache.org/docs/latest/sql-performance-tuning.html)
+[^2]: *Spark: The Definitive Guide*, Chambers & Zaharia, ch. 4, ch. 10
+[^3]: *Learning Spark, 2nd Edition*, Damji, Wenig, Das & Lee, ch. 7
+[^4]: [Configuration — Spark](https://spark.apache.org/docs/latest/configuration.html)
+[^5]: [SQLConf.scala (Spark 3.5.0)](https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala)
+[^6]: [Tuning Spark](https://spark.apache.org/docs/latest/tuning.html)
+[^7]: [Dive Into Spark Memory Management](https://luminousmen.com/post/dive-into-spark-memory)
+[^8]: [Apache Arrow in PySpark](https://spark.apache.org/docs/3.5.8/api/python/user_guide/sql/arrow_pandas.html)
+[^9]: [Adaptive Query Execution: Speeding Up Spark SQL at Runtime](https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html)
+[^10]: *High Performance Spark, 2nd Edition*, Karau, Polak & Warren, ch. 5
+[^11]: [task_memory_management_in_spark.md](https://raw.githubusercontent.com/spoddutur/spark-notes/master/task_memory_management_in_spark.md)
+[^12]: [Tuning Java Garbage Collection for Spark Applications](https://www.databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html)
+[^13]: [ZGC — The Z Garbage Collector](https://wiki.openjdk.org/display/zgc)
+[^14]: [Job Scheduling — Spark](https://spark.apache.org/docs/latest/job-scheduling.html)
+[^15]: [The Apache Spark Optimization Checklist](https://luminousmen.com/post/the-apache-spark-optimization-checklist)