sparkforensics-cli 0.1.0 → 0.2.0

package/vendor-core/docs-content/chapters/nav-index.json

@@ -0,0 +1,370 @@
+[
+  {
+    "anchor": "intro",
+    "section": "Getting Started",
+    "title": "Introduction",
+    "keywords": [
+      "severity-dot",
+      "tag-system"
+    ],
+    "store": "chapters",
+    "slug": "01-intro"
+  },
+  {
+    "anchor": "spark-architecture",
+    "section": "Optimization Guide",
+    "title": "Spark Execution Model",
+    "keywords": [
+      "driver-executor",
+      "dagscheduler",
+      "taskscheduler",
+      "stage-boundary"
+    ],
+    "store": "chapters",
+    "slug": "02-spark-architecture"
+  },
+  {
+    "anchor": "memory-model",
+    "section": "Optimization Guide",
+    "title": "Memory Management",
+    "keywords": [
+      "unified-memory",
+      "execution-memory",
+      "storage-memory",
+      "spill",
+      "memory-overhead"
+    ],
+    "store": "chapters",
+    "slug": "03-memory-model"
+  },
+  {
+    "anchor": "partitioning",
+    "section": "Optimization Guide",
+    "title": "Partitioning",
+    "keywords": [
+      "repartition",
+      "coalesce",
+      "skew"
+    ],
+    "store": "chapters",
+    "slug": "04-partitioning"
+  },
+  {
+    "anchor": "joins",
+    "section": "Optimization Guide",
+    "title": "Join Optimization",
+    "keywords": [
+      "broadcast",
+      "shuffle-join",
+      "bucketing",
+      "aqe"
+    ],
+    "store": "chapters",
+    "slug": "05-joins"
+  },
+  {
+    "anchor": "shuffle",
+    "section": "Optimization Guide",
+    "title": "Shuffle",
+    "keywords": [
+      "sortshufflemanager",
+      "bucketing",
+      "storage-partition-join"
+    ],
+    "store": "chapters",
+    "slug": "06-shuffle"
+  },
+  {
+    "anchor": "data-formats",
+    "section": "Optimization Guide",
+    "title": "Data Formats",
+    "keywords": [
+      "parquet",
+      "orc",
+      "predicate-pushdown",
+      "compression"
+    ],
+    "store": "chapters",
+    "slug": "07-data-formats"
+  },
+  {
+    "anchor": "table-formats",
+    "section": "Optimization Guide",
+    "title": "Table Formats",
+    "keywords": [
+      "delta",
+      "iceberg",
+      "hudi",
+      "compaction",
+      "optimize",
+      "z-order",
+      "deletion-vectors"
+    ],
+    "store": "chapters",
+    "slug": "07b-table-formats"
+  },
+  {
+    "anchor": "caching",
+    "section": "Optimization Guide",
+    "title": "Caching & Persistence",
+    "keywords": [
+      "cache",
+      "persist",
+      "storage-level",
+      "checkpoint"
+    ],
+    "store": "chapters",
+    "slug": "08-caching"
+  },
+  {
+    "anchor": "pyspark",
+    "section": "Optimization Guide",
+    "title": "PySpark Specifics",
+    "keywords": [
+      "udf",
+      "arrow",
+      "pandas-udf",
+      "python-worker-memory"
+    ],
+    "store": "chapters",
+    "slug": "09-pyspark"
+  },
+  {
+    "anchor": "aqe",
+    "section": "Optimization Guide",
+    "title": "Adaptive Query Execution",
+    "keywords": [
+      "partition-coalescing",
+      "join-strategy-promotion",
+      "skew-splitting"
+    ],
+    "store": "chapters",
+    "slug": "10-aqe"
+  },
+  {
+    "anchor": "cluster-config",
+    "section": "Optimization Guide",
+    "title": "Cluster Tuning",
+    "keywords": [
+      "executor-sizing",
+      "container-memory",
+      "dynamic-allocation",
+      "locality-wait"
+    ],
+    "store": "chapters",
+    "slug": "11-cluster-config"
+  },
+  {
+    "anchor": "anti-patterns",
+    "section": "Optimization Guide",
+    "title": "Anti-Patterns",
+    "keywords": [
+      "checklist",
+      "anti-pattern"
+    ],
+    "store": "chapters",
+    "slug": "12-anti-patterns"
+  },
+  {
+    "anchor": "bottleneck-skew",
+    "section": "Detector Catalog",
+    "title": "Task Skew",
+    "keywords": [
+      "skew",
+      "aqe-skew-join",
+      "salting"
+    ],
+    "store": "tuning",
+    "slug": "skew"
+  },
+  {
+    "anchor": "bottleneck-shuffle",
+    "section": "Detector Catalog",
+    "title": "Shuffle I/O",
+    "keywords": [
+      "shuffle-bytes",
+      "fetch-wait"
+    ],
+    "store": "tuning",
+    "slug": "shuffle"
+  },
+  {
+    "anchor": "bottleneck-spill",
+    "section": "Detector Catalog",
+    "title": "Memory / Disk Spill",
+    "keywords": [
+      "spill",
+      "taskmemorymanager"
+    ],
+    "store": "tuning",
+    "slug": "spill"
+  },
+  {
+    "anchor": "bottleneck-gc",
+    "section": "Detector Catalog",
+    "title": "GC Pressure",
+    "keywords": [
+      "gc-pressure",
+      "memory-fraction",
+      "off-heap"
+    ],
+    "store": "tuning",
+    "slug": "gc"
+  },
+  {
+    "anchor": "bottleneck-cold-start",
+    "section": "Detector Catalog",
+    "title": "Cold Start",
+    "keywords": [
+      "cold-start",
+      "dynamic-allocation"
+    ],
+    "store": "tuning",
+    "slug": "cold-start"
+  },
+  {
+    "anchor": "bottleneck-utilization",
+    "section": "Detector Catalog",
+    "title": "Executor Utilization",
+    "keywords": [
+      "executor-utilization",
+      "under-partitioning",
+      "over-provisioning"
+    ],
+    "store": "tuning",
+    "slug": "utilization"
+  },
+  {
+    "anchor": "bottleneck-slow-host",
+    "section": "Detector Catalog",
+    "title": "Slow Host",
+    "keywords": [
+      "slow-host",
+      "speculative-execution"
+    ],
+    "store": "tuning",
+    "slug": "slow-host"
+  },
+  {
+    "anchor": "bottleneck-failures",
+    "section": "Detector Catalog",
+    "title": "Task Failures",
+    "keywords": [
+      "taskendreason",
+      "executorlostfailure",
+      "exceptionfailure"
+    ],
+    "store": "tuning",
+    "slug": "failures"
+  },
+  {
+    "anchor": "bottleneck-straggler",
+    "section": "Detector Catalog",
+    "title": "Stragglers",
+    "keywords": [
+      "straggler",
+      "speculative-execution"
+    ],
+    "store": "tuning",
+    "slug": "straggler"
+  },
+  {
+    "anchor": "bottleneck-retry-waste",
+    "section": "Detector Catalog",
+    "title": "Retry Waste",
+    "keywords": [
+      "retry-waste",
+      "fetch-failure"
+    ],
+    "store": "tuning",
+    "slug": "retry-waste"
+  },
+  {
+    "anchor": "bottleneck-tiny-tasks",
+    "section": "Detector Catalog",
+    "title": "Tiny Tasks",
+    "keywords": [
+      "tiny-tasks",
+      "scheduling-overhead"
+    ],
+    "store": "tuning",
+    "slug": "tiny-tasks"
+  },
+  {
+    "anchor": "bottleneck-job-failure-rate",
+    "section": "Detector Catalog",
+    "title": "Job Failure Rate",
+    "keywords": [
+      "job-failure-rate",
+      "retry-budget"
+    ],
+    "store": "tuning",
+    "slug": "job-failure-rate"
+  },
+  {
+    "anchor": "bottleneck-memory-utilization",
+    "section": "Detector Catalog",
+    "title": "Memory Utilization",
+    "keywords": [
+      "memory-utilization",
+      "idle-timeout"
+    ],
+    "store": "tuning",
+    "slug": "memory-utilization"
+  },
+  {
+    "anchor": "bottleneck-duplicate-plan-subtree",
+    "section": "Detector Catalog",
+    "title": "Duplicate Plan Subtree",
+    "keywords": [
+      "duplicate-plan-subtree",
+      "exchange-reuse"
+    ],
+    "store": "tuning",
+    "slug": "duplicate-plan-subtree"
+  },
+  {
+    "anchor": "bottleneck-small-files",
+    "section": "Detector Catalog",
+    "title": "Small Files",
+    "keywords": [
+      "small-files",
+      "advisorypartitionsizeinbytes"
+    ],
+    "store": "tuning",
+    "slug": "small-files"
+  },
+  {
+    "anchor": "bottleneck-broadcast-sizing",
+    "section": "Detector Catalog",
+    "title": "Broadcast Sizing",
+    "keywords": [
+      "broadcast-threshold",
+      "driver-collection"
+    ],
+    "store": "tuning",
+    "slug": "broadcast-sizing"
+  },
+  {
+    "anchor": "metrics",
+    "section": "Reference",
+    "title": "Metrics Glossary",
+    "keywords": [
+      "metrics-glossary",
+      "event-log"
+    ],
+    "store": "chapters",
+    "slug": "14-metrics"
+  },
+  {
+    "anchor": "config",
+    "section": "Reference",
+    "title": "Spark Config Quick-Reference",
+    "keywords": [
+      "spark-config",
+      "gc-flags"
+    ],
+    "store": "chapters",
+    "slug": "15-config"
+  }
+]

package/vendor-core/docs-content/detection/cache.md

@@ -0,0 +1,6 @@
+### `CACHE`: Caching opportunity {#cache}
+
+A reusable dataset (re-read via the same SQL relation more than once) may be
+worth persisting between stages. Self-flagged low-confidence: reuse is only
+inferred, from plan-scan identity across SQL executions, so confirm the reads
+really do hit the same data before you cache anything.

package/vendor-core/docs-content/detection/cfg.md

@@ -0,0 +1,15 @@
+### `CFG`: Configuration audit {#cfg}
+
+Flags configuration settings that may cause reliability or efficiency
+problems, independent of any one stage's behavior. Four properties are
+audited today:
+
+- `spark.shuffle.service.enabled`: flagged when dynamic allocation is on
+  but the external shuffle service is off, since shuffle data won't survive
+  executor removal.
+- `spark.dynamicAllocation.maxExecutors`: flagged for inverted bounds or a
+  missing upper bound.
+- `spark.serializer`: flagged when still on the default Java serializer;
+  `org.apache.spark.serializer.KryoSerializer` is faster and produces
+  smaller buffers.
+- `spark.executor.memoryOverhead`: flagged when set below a safe floor.

package/vendor-core/docs-content/detection/chrn.md

@@ -0,0 +1,7 @@
+### `CHRN`: Autoscaling churn {#chrn}
+
+Executors are stood up and torn down again before they can do useful work:
+re-provisioning churn rather than normal scale-down. Raise
+`spark.dynamicAllocation.executorIdleTimeout`, or widen the
+`minExecutors`/`maxExecutors` bounds to reduce flapping. Self-flagged
+low-confidence: a design spike, not yet validated against real-world runs.

package/vendor-core/docs-content/detection/cold.md

@@ -0,0 +1,4 @@
+### `COLD`: Executor cold start {#cold}
+
+New executors take time to become available for work. Pre-warm the cluster,
+or use dynamic allocation.

package/vendor-core/docs-content/detection/cstor.md

@@ -0,0 +1,4 @@
+### `CSTOR`: Cache storage {#cstor}
+
+A persisted dataset is not fully cached in memory, or is spilling to disk.
+Raise executor memory, or shrink the cached dataset.

package/vendor-core/docs-content/detection/fail.md

@@ -0,0 +1,5 @@
+### `FAIL`: Failed tasks {#fail}
+
+Tasks fail often enough to affect the stage. Failed tasks point to executor
+instability or data-driven errors: check driver logs for the dominant
+failure reason.

package/vendor-core/docs-content/detection/gc.md

@@ -0,0 +1,4 @@
+### `GC`: Garbage collection pressure {#gc}
+
+Tasks spend an unusually large share of time reclaiming memory. Reduce
+object creation: use primitive types, avoid UDFs, or raise executor memory.

package/vendor-core/docs-content/detection/host.md

@@ -0,0 +1,5 @@
+### `HOST`: Slow host {#host}
+
+One executor is much slower than its peers. It may just hold data locality
+for its tasks or carry one heavy stage, rather than a hardware fault.
+Enable `spark.speculation` to relaunch a lagging task automatically.

package/vendor-core/docs-content/detection/incmp.md

@@ -0,0 +1,6 @@
+### `INCMP`: Incomplete run {#incmp}
+
+This event log never recorded an `ApplicationEnd` event: capture stopped
+before the run finished (an in-flight job, a rotated-away log, or a
+cut-short capture). Every other finding and metric on the board reflects
+only what was captured up to that point, not the full run.

package/vendor-core/docs-content/detection/jobs.md

@@ -0,0 +1,4 @@
+### `JOBS`: Job failure rate {#jobs}
+
+A large share of completed jobs did not succeed. Inspect the driver log for
+the failed job(s) and the stage failures that triggered them.

package/vendor-core/docs-content/detection/local.md

@@ -0,0 +1,7 @@
+### `LOCAL`: Core usage locality {#local}
+
+Tasks run without process- or node-local data placement more often than
+expected. Check `spark.locality.wait` settings and executor/data colocation.
+Self-flagged low-confidence: the non-local-ratio thresholds are unvalidated
+design-spike values, and no external tool publishes an equivalent metric to
+calibrate them against.

package/vendor-core/docs-content/detection/mem.md

@@ -0,0 +1,10 @@
+### `MEM`: Memory utilization {#mem}
+
+Executor memory or core capacity may be over- or under-provisioned. Some
+detail here needs `spark.eventLog.logStageExecutorMetrics=true` on the run
+being analyzed; without it, per-executor memory usage can't be broken down.
+Review `spark.executor.memory` and executor count if allocated memory sat
+largely idle over the run. That idle-memory variant is self-flagged
+low-confidence: it estimates waste from allocated-versus-used memory-time
+against an unverified 1.5x buffer. Check it against
+the Spark UI before resizing anything.

package/vendor-core/docs-content/detection/part.md

@@ -0,0 +1,5 @@
+### `PART`: Partition sizing {#part}
+
+Shuffle partitions are too large, too uneven, or too few for the work. A
+single shuffle partition over 5 GB, for example, will OOM or spill heavily:
+repartition to break it up before the stage runs.

package/vendor-core/docs-content/detection/plan.md

@@ -0,0 +1,14 @@
+### `PLAN`: Plan advisor {#plan}
+
+Flags patterns in the SQL execution plan worth reviewing. Four checks share
+this tag:
+
+- Duplicate plan subtree: the same subtree recomputed more than once in the
+  plan.
+- Small files: reading an excessive number of small files.
+- Under-broadcast: the smaller side of a Sort Merge Join looks well under
+  the broadcast threshold; consider a `broadcast()` hint or raising
+  `spark.sql.autoBroadcastJoinThreshold`.
+- Over-broadcast: a broadcast exceeds the 1 GB threshold; check for a
+  misapplied broadcast hint or a misconfigured
+  `spark.sql.autoBroadcastJoinThreshold`.

package/vendor-core/docs-content/detection/retry.md

@@ -0,0 +1,4 @@
+### `RETRY`: Retry waste {#retry}
+
+Repeated task attempts ate into execution time even though the stage
+completed. Investigate executor loss or fetch failures.

package/vendor-core/docs-content/detection/sfail.md

@@ -0,0 +1,5 @@
+### `SFAIL`: Failed stage {#sfail}
+
+A stage attempt failed outright rather than losing individual tasks within
+it. Inspect the driver log for the failure reason and the job that triggered
+it.

package/vendor-core/docs-content/detection/shape.md

@@ -0,0 +1,5 @@
+### `SHAPE`: Stage shape {#shape}
+
+The stage has an inefficient task count, output shape, or task-to-stage
+balance: for example, one straggler task taking a large fraction of the
+stage's wall-clock time.

package/vendor-core/docs-content/detection/shfl.md

@@ -0,0 +1,4 @@
+### `SHFL`: Shuffle I/O {#shfl}
+
+Tasks move a large amount of intermediate data between stages. Raise
+`spark.sql.shuffle.partitions`, or add a broadcast join.

package/vendor-core/docs-content/detection/skew.md

@@ -0,0 +1,6 @@
+### `SKEW`: Task skew {#skew}
+
+A small number of tasks take much longer than their peers in the same
+stage. For join-driven skew, enable AQE skew-join handling
+(`spark.sql.adaptive.skewJoin.enabled`); otherwise salt the key or
+repartition on a better key.

package/vendor-core/docs-content/detection/slow.md

@@ -0,0 +1,6 @@
+### `SLOW`: Stage slowness {#slow}
+
+A stage ran long overall without a more specific cause getting flagged.
+Often a partition-count problem: raise parallelism via
+`spark.sql.shuffle.partitions` or `spark.default.parallelism`, or check for a
+large per-task data volume driving heavy shuffle and spill.

package/vendor-core/docs-content/detection/spec.md

@@ -0,0 +1,7 @@
+### `SPEC`: Speculation waste {#spec}
+
+Speculative task attempts used a lot of executor time without confirming a
+genuine straggler. Self-flagged low-confidence: a design spike, not yet
+validated against real-world runs. If task durations are just naturally
+variable rather than genuine stragglers, tune
+`spark.speculation.multiplier`/`spark.speculation.quantile`.

package/vendor-core/docs-content/detection/spill.md

@@ -0,0 +1,7 @@
+### `SPILL`: Memory and disk spill {#spill}
+
+Tasks are writing data out of memory, which slows execution. Two spill
+patterns get flagged differently: skew spill, where a few heavy tasks spill
+while most don't (rebalance partitioning), and volume spill, where most
+tasks spill because the data genuinely exceeds available memory (add
+partitions).

package/vendor-core/docs-content/detection/strag.md

@@ -0,0 +1,5 @@
+### `STRAG`: Straggler tasks {#strag}
+
+A few tasks run much slower than the rest of their stage. Rule out a GC
+pause or a slow shuffle fetch before assuming a hardware issue; if a skewed
+key is the real cause, that's a candidate for AQE's skew-join handling.

package/vendor-core/docs-content/detection/tiny.md

@@ -0,0 +1,4 @@
+### `TINY`: Tiny tasks {#tiny}
+
+Many very short tasks add scheduling overhead out of proportion to the work
+each one does. Repartition to fewer, larger tasks.

package/vendor-core/docs-content/detection/util.md

@@ -0,0 +1,4 @@
+### `UTIL`: Low utilization {#util}
+
+Allocated executors sit idle for a large share of the application run.
+Consider a smaller cluster, or enable dynamic allocation.