npm - sparkforensics-cli - Versions diffs - 0.1.0 → 0.2.2 - Mend

sparkforensics-cli 0.1.0 → 0.2.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (361) hide show

package/export-template/docs/tuning-reference/config.html ADDED Viewed

@@ -0,0 +1,25 @@
+<!DOCTYPE html>
+<html lang="en-US" dir="ltr">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width,initial-scale=1">
+    <title>Spark Config Quick-Reference | SparkForensics</title>
+    <meta name="description" content="Docs for using and contributing to SparkForensics">
+    <meta name="generator" content="VitePress v1.6.4">
+    <link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
+    <link rel="preload stylesheet" href="../vp-icons.css" as="style">
+    <link rel="icon" type="image/svg+xml" href="../favicon.svg">
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="">
+    <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Recursive:wght,CASL@400..700,0..1&amp;family=JetBrains+Mono:wght@400;500;600&amp;display=swap">
+    <script id="check-dark-mode">(()=>{const e=localStorage.getItem("vitepress-theme-appearance")||"auto",a=window.matchMedia("(prefers-color-scheme: dark)").matches;(!e||e==="auto"?a:e==="dark")&&document.documentElement.classList.add("dark")})();</script>
+    <script id="check-mac-os">document.documentElement.classList.toggle("mac",/Mac|iPhone|iPod|iPad/i.test(navigator.platform));</script>
+  </head>
+  <body>
+    <div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="../index.html" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="../favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../tuning-reference/index.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Getting Started</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/intro.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Introduction</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Optimization Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/spark-architecture.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Spark Execution Model</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/memory-model.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory Management</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/partitioning.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Partitioning</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/joins.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Join Optimization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/shuffle.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Shuffle</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/data-formats.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Data Formats</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/table-formats.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Table Formats</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/caching.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Caching & Persistence</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/pyspark.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>PySpark Specifics</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/aqe.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Adaptive Query Execution</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/cluster-config.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Cluster Tuning</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/anti-patterns.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Anti-Patterns</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Detector Catalog</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-skew.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Task Skew</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-shuffle.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Shuffle I/O</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-spill.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory / Disk Spill</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-gc.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>GC Pressure</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-cold-start.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Cold Start</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-utilization.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Executor Utilization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-slow-host.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Slow Host</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-failures.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Task Failures</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-straggler.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Stragglers</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-retry-waste.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Retry Waste</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-tiny-tasks.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Tiny Tasks</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-job-failure-rate.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Job Failure Rate</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-memory-utilization.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory Utilization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-duplicate-plan-subtree.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Duplicate Plan Subtree</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-small-files.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Small Files</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-broadcast-sizing.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Broadcast Sizing</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Reference</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/metrics.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Metrics Glossary</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/config.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Spark Config Quick-Reference</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_tuning-reference_config" data-v-39a288b8><div><h1 id="config" tabindex="-1">Spark Config Quick-Reference <a class="header-anchor" href="#config" aria-label="Permalink to &quot;Spark Config Quick-Reference {#config}&quot;"></a></h1><p>A cross-reference of the Spark and PySpark configuration properties discussed elsewhere in this guide, plus the JVM garbage-collection flags relevant to executor tuning. Defaults and version notes below are traced to Spark&#39;s own configuration docs, its SQLConf source, and the cited literature. Where a source doesn&#39;t give a hard number (a range, a ceiling, a recommended workload profile), the table leaves it out rather than guess.</p><h2 id="spark-properties" tabindex="-1">Spark properties <a class="header-anchor" href="#spark-properties" aria-label="Permalink to &quot;Spark properties&quot;"></a></h2><table tabindex="0"><thead><tr><th>Key</th><th>Default</th><th>Recommended</th><th>Notes</th></tr></thead><tbody><tr><td><code>spark.sql.shuffle.partitions</code></td><td>200 (unchanged since Spark 1.1.0)<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup><sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup></td><td></td><td>AQE doesn&#39;t override this config value; see notes below for the effective runtime count.</td></tr><tr><td><code>spark.sql.adaptive.enabled</code></td><td><code>true</code><sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup></td><td><code>true</code></td><td></td></tr><tr><td><code>spark.sql.adaptive.coalescePartitions.enabled</code></td><td><code>true</code> (since 3.0.0)<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup></td><td><code>true</code></td><td>Requires AQE; merges contiguous small post-shuffle partitions instead of one task per configured partition<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup>.</td></tr><tr><td><code>spark.sql.adaptive.coalescePartitions.initialPartitionNum</code></td><td>unset → falls back to <code>spark.sql.shuffle.partitions</code> (200)<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup></td><td></td><td>Sets the partition count entering the coalescing step.</td></tr><tr><td><code>spark.sql.adaptive.coalescePartitions.parallelismFirst</code></td><td><code>true</code> (since 3.2.0)<sup class="footnote-ref"><a href="#fn1" id="fnref1:4">[1:4]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup></td><td><code>false</code> on busy clusters<sup class="footnote-ref"><a href="#fn1" id="fnref1:5">[1:5]</a></sup></td><td>When <code>true</code>, ignores <code>advisoryPartitionSizeInBytes</code> and derives a target from the cluster&#39;s default parallelism, respecting only the <code>minPartitionSize</code> floor.</td></tr><tr><td><code>spark.sql.adaptive.advisoryPartitionSizeInBytes</code></td><td>64MB (since 3.0.0)<sup class="footnote-ref"><a href="#fn1" id="fnref1:6">[1:6]</a></sup><sup class="footnote-ref"><a href="#fn4" id="fnref4:1">[4:1]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:1">[5:1]</a></sup></td><td></td><td>Ignored during coalescing when <code>parallelismFirst=true</code> (the default).</td></tr><tr><td><code>spark.sql.adaptive.coalescePartitions.minPartitionSize</code></td><td>1MB (since 3.2.0)<sup class="footnote-ref"><a href="#fn5" id="fnref5:2">[5:2]</a></sup></td><td></td><td>Floor enforced when the target size is ignored: the default <code>parallelismFirst=true</code> case<sup class="footnote-ref"><a href="#fn1" id="fnref1:7">[1:7]</a></sup>.</td></tr><tr><td><code>spark.memory.fraction</code></td><td>0.6<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup></td><td></td><td>See notes below; no hard 0–1 ceiling documented in the cited sources.</td></tr><tr><td><code>spark.memory.storageFraction</code></td><td>0.5<sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup></td><td></td><td>A fraction <em>of</em> the region sized by <code>spark.memory.fraction</code>, not an independent pool; see notes below.</td></tr><tr><td><code>spark.sql.execution.arrow.pyspark.enabled</code></td><td><code>false</code><sup class="footnote-ref"><a href="#fn8" id="fnref8">[8]</a></sup></td><td></td><td>See PySpark callout below.</td></tr><tr><td><code>spark.sql.execution.arrow.pyspark.fallback.enabled</code></td><td><code>true</code> (inherited from the deprecated <code>arrow.fallback.enabled</code>)<sup class="footnote-ref"><a href="#fn8" id="fnref8:1">[8:1]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:3">[5:3]</a></sup><sup class="footnote-ref"><a href="#fn4" id="fnref4:2">[4:2]</a></sup></td><td></td><td>Falls back to the non-Arrow path automatically if a conversion error occurs, before computation runs<sup class="footnote-ref"><a href="#fn8" id="fnref8:2">[8:2]</a></sup>.</td></tr><tr><td><code>spark.dynamicAllocation.shuffleTracking.enabled</code></td><td><code>true</code> (since 3.0.0)<sup class="footnote-ref"><a href="#fn4" id="fnref4:3">[4:3]</a></sup></td><td></td><td>Lets dynamic allocation track shuffle files per executor without an external shuffle service.</td></tr><tr><td><code>spark.dynamicAllocation.shuffleTracking.timeout</code></td><td><code>infinity</code><sup class="footnote-ref"><a href="#fn4" id="fnref4:4">[4:4]</a></sup></td><td></td><td>Executors holding shuffle data wait for it to be garbage collected before release, by default; set a finite value if GC isn&#39;t keeping up.</td></tr><tr><td><code>spark.shuffle.service.enabled</code></td><td></td><td></td><td>External shuffle service; see <code>#config-shuffle-service</code> below.</td></tr><tr><td><code>spark.dynamicAllocation.minExecutors</code> / <code>maxExecutors</code></td><td><code>0</code> / <code>infinity</code><sup class="footnote-ref"><a href="#fn4" id="fnref4:5">[4:5]</a></sup></td><td></td><td>Autoscale bounds; see <code>#config-autoscale-bounds</code> below.</td></tr><tr><td><code>spark.serializer</code></td><td><code>org.apache.spark.serializer.JavaSerializer</code><sup class="footnote-ref"><a href="#fn4" id="fnref4:6">[4:6]</a></sup></td><td><code>KryoSerializer</code></td><td>See <code>#config-serializer</code> below.</td></tr><tr><td><code>spark.executor.memoryOverhead</code></td><td>greater of 10% of executor memory or 384MB<sup class="footnote-ref"><a href="#fn7" id="fnref7:1">[7:1]</a></sup></td><td></td><td>Off-heap overhead; see <code>#config-memory-overhead</code> below.</td></tr></tbody></table><h3 id="shuffle-partitions-and-aqe-coalescing" tabindex="-1">Shuffle partitions and AQE coalescing <a class="header-anchor" href="#shuffle-partitions-and-aqe-coalescing" aria-label="Permalink to &quot;Shuffle partitions and AQE coalescing&quot;"></a></h3><p>The 200 default for <code>spark.sql.shuffle.partitions</code> hasn&#39;t moved since 1.1.0, and AQE (on by default since Spark 3.0) doesn&#39;t change that config value<sup class="footnote-ref"><a href="#fn1" id="fnref1:8">[1:8]</a></sup><sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup>. What it changes is the effective number of partitions used at runtime. With <code>coalescePartitions.enabled</code> also on by default, Spark merges small contiguous post-shuffle partitions instead of running one task per configured partition<sup class="footnote-ref"><a href="#fn1" id="fnref1:9">[1:9]</a></sup>. Because <code>parallelismFirst</code> defaults to <code>true</code> since 3.2.0, that merge target usually isn&#39;t the 64MB <code>advisoryPartitionSizeInBytes</code> value; it&#39;s derived from the cluster&#39;s default parallelism, with <code>minPartitionSize</code> (1MB) as the only enforced floor<sup class="footnote-ref"><a href="#fn1" id="fnref1:10">[1:10]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:4">[5:4]</a></sup>. Databricks&#39; own writeup on AQE shows the reduce-task count actually shrinking based on measured data volume<sup class="footnote-ref"><a href="#fn9" id="fnref9">[9]</a></sup>, and Learning Spark accordingly calls the static 200 default &quot;too high for smaller or streaming workloads&quot;<sup class="footnote-ref"><a href="#fn3" id="fnref3:2">[3:2]</a></sup>. A related, distinct knob (<code>spark.sql.adaptive.coalescePartitions.minPartitionNum</code>) sets a minimum parallelism floor for data that&#39;s slow to compute despite being small, per High Performance Spark<sup class="footnote-ref"><a href="#fn10" id="fnref10">[10]</a></sup>.</p><h3 id="spark-memory-fraction-and-spark-memory-storagefraction" tabindex="-1"><code>spark.memory.fraction</code> and <code>spark.memory.storageFraction</code> <a class="header-anchor" href="#spark-memory-fraction-and-spark-memory-storagefraction" aria-label="Permalink to &quot;`spark.memory.fraction` and `spark.memory.storageFraction`&quot;"></a></h3><p><code>spark.memory.fraction</code> sizes the unified execution/storage region (M) as a fraction of (JVM heap − 300MiB); the tuning guide frames it as a knob to fit M &quot;comfortably within the JVM&#39;s old or tenured generation&quot; rather than stating an enforced numeric range<sup class="footnote-ref"><a href="#fn6" id="fnref6:1">[6:1]</a></sup>. The cited sources document no failure threshold tied to a specific value like 0.9. What they do document is the risk of pushing the fraction high: the complement, <code>1 − spark.memory.fraction</code>, is untracked &quot;User Memory&quot; for UDFs, Python/Arrow glue, and native buffers, and starving it (which is what raising the fraction toward 0.9 does) leads to &quot;GC pressure or random OOMs,&quot; with no warning from Spark<sup class="footnote-ref"><a href="#fn7" id="fnref7:2">[7:2]</a></sup>.</p><p><code>spark.memory.storageFraction</code> isn&#39;t independent of that: it&#39;s a fraction <em>of</em> M, the same region <code>spark.memory.fraction</code> sizes<sup class="footnote-ref"><a href="#fn6" id="fnref6:2">[6:2]</a></sup>. Inside that shared pool, execution can evict storage down to the threshold set by <code>storageFraction</code>, but storage can never evict execution<sup class="footnote-ref"><a href="#fn11" id="fnref11">[11]</a></sup><sup class="footnote-ref"><a href="#fn6" id="fnref6:3">[6:3]</a></sup>. With the defaults (0.6 and 0.5), that works out to execution and storage each getting 30% of usable heap; raising <code>memory.fraction</code> scales both pools at once, while <code>storageFraction</code> only re-splits the pool that&#39;s already been carved out<sup class="footnote-ref"><a href="#fn7" id="fnref7:3">[7:3]</a></sup>.</p><blockquote><p><strong>PySpark:</strong> <code>spark.sql.execution.arrow.pyspark.enabled</code> is off by default and governs Arrow use for <code>DataFrame.toPandas()</code> and <code>SparkSession.createDataFrame()</code> from a Pandas DataFrame or NumPy array<sup class="footnote-ref"><a href="#fn8" id="fnref8:3">[8:3]</a></sup>. Its documented risk is a type-coverage gap, not silent data corruption: <code>ArrayType</code> of <code>TimestampType</code> is explicitly unsupported<sup class="footnote-ref"><a href="#fn4" id="fnref4:7">[4:7]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:5">[5:5]</a></sup>, and more generally an unsupported column type raises an error rather than converting wrongly<sup class="footnote-ref"><a href="#fn8" id="fnref8:4">[8:4]</a></sup>. <code>spark.sql.execution.arrow.pyspark.fallback.enabled</code> defaults to <code>true</code> and automatically falls back to the non-Arrow path if a conversion error occurs, before any computation runs<sup class="footnote-ref"><a href="#fn8" id="fnref8:5">[8:5]</a></sup><sup class="footnote-ref"><a href="#fn5" id="fnref5:6">[5:6]</a></sup><sup class="footnote-ref"><a href="#fn4" id="fnref4:8">[4:8]</a></sup>. So the failure mode is an exception plus fallback, not a wrong result. Separately, and not specific to this Arrow config, PySpark&#39;s own type coercion has its own hazards: numeric values passed for <code>ByteType</code>/<code>ShortType</code>/<code>IntegerType</code> must fall within fixed ranges or get rejected or converted unexpectedly<sup class="footnote-ref"><a href="#fn2" id="fnref2:2">[2:2]</a></sup>.</p></blockquote><h3 id="config-shuffle-service" tabindex="-1">Shuffle service <a class="header-anchor" href="#config-shuffle-service" aria-label="Permalink to &quot;Shuffle service {#config-shuffle-service}&quot;"></a></h3><p>During a shuffle, an executor writes its map output to local disk and then serves fetch requests for that data itself<sup class="footnote-ref"><a href="#fn12" id="fnref12">[12]</a></sup>. Dynamic allocation can reclaim an idle executor before a later stage has fetched all the shuffle blocks it holds (especially with stragglers, tasks that run much longer than their peers), and once that executor is gone, any stage that still needs its shuffle output gets a <code>FetchFailed</code> and has to recompute it<sup class="footnote-ref"><a href="#fn12" id="fnref12:1">[12:1]</a></sup>. The external shuffle service fixes this by moving shuffle-block serving out of the executor process: it&#39;s a long-running process on each node, independent of any particular application&#39;s executors, and once <code>spark.shuffle.service.enabled</code> is <code>true</code>, executors fetch shuffle blocks from it instead of from each other, so an executor&#39;s shuffle output keeps being served after that executor is reclaimed<sup class="footnote-ref"><a href="#fn12" id="fnref12:2">[12:2]</a></sup><sup class="footnote-ref"><a href="#fn4" id="fnref4:9">[4:9]</a></sup>. Setting up dynamic allocation requires enabling this service on every worker node in addition to <code>spark.dynamicAllocation.enabled</code><sup class="footnote-ref"><a href="#fn2" id="fnref2:3">[2:3]</a></sup>. As of Spark 3.0, shuffle-file tracking (<code>spark.dynamicAllocation.shuffleTracking.enabled</code>) is an alternative to the external shuffle service for the same safe-removal problem<sup class="footnote-ref"><a href="#fn4" id="fnref4:10">[4:10]</a></sup>.</p><p><strong>Limitations / false-positive risk:</strong> the service can be intentionally left off when dynamic allocation relies on shuffle-file tracking instead, so a disabled setting is not automatically a misconfiguration.</p><h3 id="config-autoscale-bounds" tabindex="-1">Autoscale bounds <a class="header-anchor" href="#config-autoscale-bounds" aria-label="Permalink to &quot;Autoscale bounds {#config-autoscale-bounds}&quot;"></a></h3><p>By default, <code>spark.dynamicAllocation.minExecutors</code> is <code>0</code> and <code>spark.dynamicAllocation.maxExecutors</code> is <code>infinity</code><sup class="footnote-ref"><a href="#fn4" id="fnref4:11">[4:11]</a></sup>. Leaving <code>maxExecutors</code> unset therefore leaves dynamic allocation unbounded on the high end as far as Spark itself is concerned; in practice the ceiling ends up being whatever the cluster or scheduler enforces outside Spark<sup class="footnote-ref"><a href="#fn4" id="fnref4:12">[4:12]</a></sup>. <code>spark.dynamicAllocation.initialExecutors</code> defaults to <code>minExecutors</code>, unless <code>--num-executors</code> (or <code>spark.executor.instances</code>) sets a larger starting value<sup class="footnote-ref"><a href="#fn4" id="fnref4:13">[4:13]</a></sup>. Whatever executor count <code>executorAllocationRatio</code> computes to maximize parallelism, that target is still clamped by the <code>minExecutors</code>/<code>maxExecutors</code> floor and ceiling<sup class="footnote-ref"><a href="#fn4" id="fnref4:14">[4:14]</a></sup>.</p><p><strong>Limitations / false-positive risk:</strong> an unbounded <code>maxExecutors</code> is often deliberate on clusters where the scheduler enforces the real ceiling, so an unset high end is not always wrong.</p><h3 id="config-serializer" tabindex="-1">Serializer <a class="header-anchor" href="#config-serializer" aria-label="Permalink to &quot;Serializer {#config-serializer}&quot;"></a></h3><p>The default <code>spark.serializer</code> is <code>org.apache.spark.serializer.JavaSerializer</code>, which works with any <code>Serializable</code> Java object but is &quot;quite slow&quot;; Spark&#39;s configuration reference recommends switching to <code>KryoSerializer</code> &quot;when speed is necessary&quot;<sup class="footnote-ref"><a href="#fn4" id="fnref4:15">[4:15]</a></sup>. The tuning guide is more specific: Kryo is &quot;significantly faster and more compact than Java serialization (often as much as 10x),&quot; though it doesn&#39;t support all <code>Serializable</code> types and needs its classes registered in advance for best performance<sup class="footnote-ref"><a href="#fn6" id="fnref6:4">[6:4]</a></sup>. That registration requirement is the stated reason Kryo isn&#39;t the default: &quot;The only reason Kryo is not the default is because of the custom registration requirement,&quot; and Spark recommends trying it for any network-intensive application<sup class="footnote-ref"><a href="#fn6" id="fnref6:5">[6:5]</a></sup>. Since Spark 2.0.0, Spark internally uses Kryo regardless of this setting when shuffling RDDs of simple types, arrays of simple types, or strings, and auto-registers common Scala classes via Twitter chill&#39;s <code>AllScalaRegistrar</code><sup class="footnote-ref"><a href="#fn6" id="fnref6:6">[6:6]</a></sup>. A practitioner summary puts Java serialization at &quot;2-10x slower and larger on the wire&quot; on every shuffle, with the caveat that classes left unregistered silently fall back to Java serialization unless registered via <code>spark.kryo.classesToRegister</code> or <code>spark.kryo.registrator</code><sup class="footnote-ref"><a href="#fn13" id="fnref13">[13]</a></sup>. Persisting RDDs in serialized form is another case where Kryo can be more space-efficient than Java serialization<sup class="footnote-ref"><a href="#fn10" id="fnref10:1">[10:1]</a></sup>.</p><p><strong>Limitations / false-positive risk:</strong> <code>JavaSerializer</code> may be kept on purpose when an application depends on <code>Serializable</code> types Kryo cannot handle, so a non-Kryo setting is not necessarily a mistake.</p><h3 id="config-memory-overhead" tabindex="-1">Memory overhead <a class="header-anchor" href="#config-memory-overhead" aria-label="Permalink to &quot;Memory overhead {#config-memory-overhead}&quot;"></a></h3><p><code>spark.executor.memoryOverhead</code> covers everything an executor needs outside the JVM heap sized by <code>spark.executor.memory</code>: thread stacks, JIT buffers, metaspace, JNI, native libraries, and, if <code>spark.executor.pyspark.memory</code> isn&#39;t set separately, the memory used by PySpark&#39;s per-task Python worker processes<sup class="footnote-ref"><a href="#fn7" id="fnref7:4">[7:4]</a></sup>. Left unset, Spark defaults it to the greater of 10% of executor memory or 384MB<sup class="footnote-ref"><a href="#fn7" id="fnref7:5">[7:5]</a></sup>; the configuration reference formalizes that floor as <code>spark.executor.minMemoryOverhead</code> (default <code>384m</code>, since Spark 4.0) and the percentage as <code>spark.executor.memoryOverheadFactor</code> (default 0.10, or 0.40 for non-JVM jobs on Kubernetes, since those need more non-JVM heap space)<sup class="footnote-ref"><a href="#fn4" id="fnref4:16">[4:16]</a></sup>. The resource manager (YARN&#39;s NodeManager, or the Kubernetes kubelet) sizes the container/pod to the sum of <code>spark.executor.memoryOverhead</code>, <code>spark.executor.memory</code>, <code>spark.memory.offHeap.size</code>, and <code>spark.executor.pyspark.memory</code><sup class="footnote-ref"><a href="#fn4" id="fnref4:17">[4:17]</a></sup>. If off-heap memory or PySpark worker processes consume memory that isn&#39;t reflected in a correspondingly larger <code>memoryOverhead</code>, that extra usage still shows up in the process&#39;s actual RSS, which the resource manager does track. Once real usage exceeds the container&#39;s allocated size, the executor is killed by YARN or OOMKilled by the kubelet, with no Spark-level error, just an abrupt process death<sup class="footnote-ref"><a href="#fn7" id="fnref7:6">[7:6]</a></sup>.</p><p><strong>Limitations / false-positive risk:</strong> the default overhead is adequate for many JVM-only jobs, so a value left at the default only signals trouble on off-heap-heavy or PySpark workloads that need more non-heap room.</p><h2 id="jvm-gc-flags" tabindex="-1">JVM GC flags <a class="header-anchor" href="#jvm-gc-flags" aria-label="Permalink to &quot;JVM GC flags&quot;"></a></h2><table tabindex="0"><thead><tr><th>Flag</th><th>Effect</th></tr></thead><tbody><tr><td><code>-XX:+UseG1GC</code></td><td>Default collector since Spark 4.0.0, which defaults to JDK 17<sup class="footnote-ref"><a href="#fn6" id="fnref6:7">[6:7]</a></sup>.</td></tr><tr><td><code>-XX:G1HeapRegionSize</code></td><td>May need raising alongside large executor heaps<sup class="footnote-ref"><a href="#fn6" id="fnref6:8">[6:8]</a></sup>.</td></tr><tr><td><code>-XX:InitiatingHeapOccupancyPercent</code></td><td>Tuned (with <code>-XX:ConcGCThreads</code> and RSet-update settings) to fix a documented ~100-second G1 full-GC pause on an 88GB executor heap<sup class="footnote-ref"><a href="#fn14" id="fnref14">[14]</a></sup>.</td></tr><tr><td><code>-XX:ConcGCThreads</code></td><td>See <code>-XX:InitiatingHeapOccupancyPercent</code> above<sup class="footnote-ref"><a href="#fn14" id="fnref14:1">[14:1]</a></sup>.</td></tr><tr><td><code>-XX:+UseZGC</code></td><td>Concurrent, low-latency collector; sub-millisecond pause target independent of heap size, from a few hundred megabytes up to 16TB<sup class="footnote-ref"><a href="#fn15" id="fnref15">[15]</a></sup>.</td></tr></tbody></table><h3 id="g1-vs-zgc" tabindex="-1">G1 vs. ZGC <a class="header-anchor" href="#g1-vs-zgc" aria-label="Permalink to &quot;G1 vs. ZGC&quot;"></a></h3><p>G1 was designed as a CMS replacement aiming at both throughput and low latency: it partitions the heap into equal-sized regions and copies out only the live objects from collected regions rather than compacting the whole heap<sup class="footnote-ref"><a href="#fn14" id="fnref14:2">[14:2]</a></sup>. Even so, a documented 88GB-heap Spark benchmark hit &quot;unacceptable full GC,&quot; with one job pausing nearly 100 seconds under default G1 settings. Only after tuning <code>InitiatingHeapOccupancyPercent</code>, <code>ConcGCThreads</code>, and RSet-update parameters did G1 beat Parallel/CMS GC on both throughput and latency<sup class="footnote-ref"><a href="#fn14" id="fnref14:3">[14:3]</a></sup>. ZGC, per the OpenJDK project page, &quot;performs all expensive work concurrently, without stopping the execution of application threads for more than a millisecond,&quot; with pause times that stay flat as heap size scales from a few hundred megabytes up to 16TB<sup class="footnote-ref"><a href="#fn15" id="fnref15:1">[15:1]</a></sup>.</p><h2 id="sources" tabindex="-1">Sources <a class="header-anchor" href="#sources" aria-label="Permalink to &quot;Sources&quot;"></a></h2><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html" target="_blank" rel="noreferrer">Performance Tuning — Spark SQL, DataFrames and Datasets Guide</a> <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a> <a href="#fnref1:4" class="footnote-backref">↩︎</a> <a href="#fnref1:5" class="footnote-backref">↩︎</a> <a href="#fnref1:6" class="footnote-backref">↩︎</a> <a href="#fnref1:7" class="footnote-backref">↩︎</a> <a href="#fnref1:8" class="footnote-backref">↩︎</a> <a href="#fnref1:9" class="footnote-backref">↩︎</a> <a href="#fnref1:10" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><em>Spark: The Definitive Guide</em>, Chambers &amp; Zaharia, ch. 4, ch. 10 <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a> <a href="#fnref2:2" class="footnote-backref">↩︎</a> <a href="#fnref2:3" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><em>Learning Spark, 2nd Edition</em>, Damji, Wenig, Das &amp; Lee, ch. 7 <a href="#fnref3" class="footnote-backref">↩︎</a> <a href="#fnref3:1" class="footnote-backref">↩︎</a> <a href="#fnref3:2" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration — Spark</a> <a href="#fnref4" class="footnote-backref">↩︎</a> <a href="#fnref4:1" class="footnote-backref">↩︎</a> <a href="#fnref4:2" class="footnote-backref">↩︎</a> <a href="#fnref4:3" class="footnote-backref">↩︎</a> <a href="#fnref4:4" class="footnote-backref">↩︎</a> <a href="#fnref4:5" class="footnote-backref">↩︎</a> <a href="#fnref4:6" class="footnote-backref">↩︎</a> <a href="#fnref4:7" class="footnote-backref">↩︎</a> <a href="#fnref4:8" class="footnote-backref">↩︎</a> <a href="#fnref4:9" class="footnote-backref">↩︎</a> <a href="#fnref4:10" class="footnote-backref">↩︎</a> <a href="#fnref4:11" class="footnote-backref">↩︎</a> <a href="#fnref4:12" class="footnote-backref">↩︎</a> <a href="#fnref4:13" class="footnote-backref">↩︎</a> <a href="#fnref4:14" class="footnote-backref">↩︎</a> <a href="#fnref4:15" class="footnote-backref">↩︎</a> <a href="#fnref4:16" class="footnote-backref">↩︎</a> <a href="#fnref4:17" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala" target="_blank" rel="noreferrer">SQLConf.scala (Spark 3.5.0)</a> <a href="#fnref5" class="footnote-backref">↩︎</a> <a href="#fnref5:1" class="footnote-backref">↩︎</a> <a href="#fnref5:2" class="footnote-backref">↩︎</a> <a href="#fnref5:3" class="footnote-backref">↩︎</a> <a href="#fnref5:4" class="footnote-backref">↩︎</a> <a href="#fnref5:5" class="footnote-backref">↩︎</a> <a href="#fnref5:6" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/tuning.html" target="_blank" rel="noreferrer">Tuning Spark</a> <a href="#fnref6" class="footnote-backref">↩︎</a> <a href="#fnref6:1" class="footnote-backref">↩︎</a> <a href="#fnref6:2" class="footnote-backref">↩︎</a> <a href="#fnref6:3" class="footnote-backref">↩︎</a> <a href="#fnref6:4" class="footnote-backref">↩︎</a> <a href="#fnref6:5" class="footnote-backref">↩︎</a> <a href="#fnref6:6" class="footnote-backref">↩︎</a> <a href="#fnref6:7" class="footnote-backref">↩︎</a> <a href="#fnref6:8" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://luminousmen.com/post/dive-into-spark-memory" target="_blank" rel="noreferrer">Dive Into Spark Memory Management</a> <a href="#fnref7" class="footnote-backref">↩︎</a> <a href="#fnref7:1" class="footnote-backref">↩︎</a> <a href="#fnref7:2" class="footnote-backref">↩︎</a> <a href="#fnref7:3" class="footnote-backref">↩︎</a> <a href="#fnref7:4" class="footnote-backref">↩︎</a> <a href="#fnref7:5" class="footnote-backref">↩︎</a> <a href="#fnref7:6" class="footnote-backref">↩︎</a></p></li><li id="fn8" class="footnote-item"><p><a href="https://spark.apache.org/docs/3.5.8/api/python/user_guide/sql/arrow_pandas.html" target="_blank" rel="noreferrer">Apache Arrow in PySpark</a> <a href="#fnref8" class="footnote-backref">↩︎</a> <a href="#fnref8:1" class="footnote-backref">↩︎</a> <a href="#fnref8:2" class="footnote-backref">↩︎</a> <a href="#fnref8:3" class="footnote-backref">↩︎</a> <a href="#fnref8:4" class="footnote-backref">↩︎</a> <a href="#fnref8:5" class="footnote-backref">↩︎</a></p></li><li id="fn9" class="footnote-item"><p><a href="https://www.databricks.com/blog/2020/05/29/adaptive-query-execution-speeding-up-spark-sql-at-runtime.html" target="_blank" rel="noreferrer">Adaptive Query Execution: Speeding Up Spark SQL at Runtime</a> <a href="#fnref9" class="footnote-backref">↩︎</a></p></li><li id="fn10" class="footnote-item"><p><em>High Performance Spark, 2nd Edition</em>, Karau, Polak &amp; Warren, ch. 5 <a href="#fnref10" class="footnote-backref">↩︎</a> <a href="#fnref10:1" class="footnote-backref">↩︎</a></p></li><li id="fn11" class="footnote-item"><p><a href="https://raw.githubusercontent.com/spoddutur/spark-notes/master/task_memory_management_in_spark.md" target="_blank" rel="noreferrer">task_memory_management_in_spark.md</a> <a href="#fnref11" class="footnote-backref">↩︎</a></p></li><li id="fn12" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/job-scheduling.html" target="_blank" rel="noreferrer">Job Scheduling — Spark</a> <a href="#fnref12" class="footnote-backref">↩︎</a> <a href="#fnref12:1" class="footnote-backref">↩︎</a> <a href="#fnref12:2" class="footnote-backref">↩︎</a></p></li><li id="fn13" class="footnote-item"><p><a href="https://luminousmen.com/post/the-apache-spark-optimization-checklist" target="_blank" rel="noreferrer">The Apache Spark Optimization Checklist</a> <a href="#fnref13" class="footnote-backref">↩︎</a></p></li><li id="fn14" class="footnote-item"><p><a href="https://www.databricks.com/blog/2015/05/28/tuning-java-garbage-collection-for-spark-applications.html" target="_blank" rel="noreferrer">Tuning Java Garbage Collection for Spark Applications</a> <a href="#fnref14" class="footnote-backref">↩︎</a> <a href="#fnref14:1" class="footnote-backref">↩︎</a> <a href="#fnref14:2" class="footnote-backref">↩︎</a> <a href="#fnref14:3" class="footnote-backref">↩︎</a></p></li><li id="fn15" class="footnote-item"><p><a href="https://wiki.openjdk.org/display/zgc" target="_blank" rel="noreferrer">ZGC — The Z Garbage Collector</a> <a href="#fnref15" class="footnote-backref">↩︎</a> <a href="#fnref15:1" class="footnote-backref">↩︎</a></p></li></ol></section></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><a class="VPLink link pager-link prev" href="../tuning-reference/metrics.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Previous page</span><span class="title" data-v-e257564d>Metrics Glossary</span><!--]--></a></div><div class="pager" data-v-e257564d><!----></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
+  </body>
+</html>

package/export-template/docs/tuning-reference/data-formats.html ADDED Viewed

@@ -0,0 +1,25 @@
+<!DOCTYPE html>
+<html lang="en-US" dir="ltr">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width,initial-scale=1">
+    <title>Data Formats | SparkForensics</title>
+    <meta name="description" content="Docs for using and contributing to SparkForensics">
+    <meta name="generator" content="VitePress v1.6.4">
+    <link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
+    <link rel="preload stylesheet" href="../vp-icons.css" as="style">
+    <link rel="icon" type="image/svg+xml" href="../favicon.svg">
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="">
+    <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Recursive:wght,CASL@400..700,0..1&amp;family=JetBrains+Mono:wght@400;500;600&amp;display=swap">
+    <script id="check-dark-mode">(()=>{const e=localStorage.getItem("vitepress-theme-appearance")||"auto",a=window.matchMedia("(prefers-color-scheme: dark)").matches;(!e||e==="auto"?a:e==="dark")&&document.documentElement.classList.add("dark")})();</script>
+    <script id="check-mac-os">document.documentElement.classList.toggle("mac",/Mac|iPhone|iPod|iPad/i.test(navigator.platform));</script>
+  </head>
+  <body>
+    <div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="../index.html" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="../favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../tuning-reference/index.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Getting Started</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/intro.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Introduction</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Optimization Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/spark-architecture.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Spark Execution Model</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/memory-model.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory Management</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/partitioning.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Partitioning</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/joins.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Join Optimization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/shuffle.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Shuffle</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/data-formats.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Data Formats</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/table-formats.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Table Formats</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/caching.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Caching & Persistence</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/pyspark.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>PySpark Specifics</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/aqe.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Adaptive Query Execution</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/cluster-config.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Cluster Tuning</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/anti-patterns.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Anti-Patterns</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Detector Catalog</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-skew.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Task Skew</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-shuffle.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Shuffle I/O</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-spill.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory / Disk Spill</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-gc.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>GC Pressure</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-cold-start.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Cold Start</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-utilization.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Executor Utilization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-slow-host.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Slow Host</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-failures.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Task Failures</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-straggler.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Stragglers</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-retry-waste.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Retry Waste</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-tiny-tasks.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Tiny Tasks</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-job-failure-rate.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Job Failure Rate</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-memory-utilization.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory Utilization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-duplicate-plan-subtree.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Duplicate Plan Subtree</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-small-files.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Small Files</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-broadcast-sizing.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Broadcast Sizing</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Reference</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/metrics.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Metrics Glossary</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/config.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Spark Config Quick-Reference</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_tuning-reference_data-formats" data-v-39a288b8><div><h1 id="data-formats" tabindex="-1">Data Formats <a class="header-anchor" href="#data-formats" aria-label="Permalink to &quot;Data Formats {#data-formats}&quot;"></a></h1><h2 id="how-parquet-and-orc-lay-out-data" tabindex="-1">How Parquet and ORC lay out data <a class="header-anchor" href="#how-parquet-and-orc-lay-out-data" aria-label="Permalink to &quot;How Parquet and ORC lay out data&quot;"></a></h2><p>Parquet and ORC are both self-describing, columnar file formats for storing Spark&#39;s structured data<sup class="footnote-ref"><a href="#fn1" id="fnref1">[1]</a></sup>. They&#39;re closer than they are different: <em>Spark: The Definitive Guide</em> frames the choice as &quot;for the most part, they&#39;re quite similar; the fundamental difference is that Parquet is further optimized for use with Spark, whereas ORC is further optimized for Hive&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1:1">[1:1]</a></sup>, and notes that ORC &quot;has no options for reading in data because Spark understands the file format quite well&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1:2">[1:2]</a></sup>: Spark treats it as a format it understands natively rather than one with extra read-side knobs.</p><p>Both formats group rows into chunks (row groups in Parquet, stripes in ORC), each holding every column&#39;s data for that slice, with a footer or file-level metadata section recording where each chunk lives on disk. Parquet&#39;s <code>FileMetaData</code> footer records the offset and size of every row group and column chunk, and &quot;readers are expected to first read the file metadata to find all the column chunks they are interested in. The column chunks should then be read sequentially&quot;<sup class="footnote-ref"><a href="#fn2" id="fnref2">[2]</a></sup>. Because compression is applied per column chunk rather than as one continuous stream over the whole file, a reader can seek straight to a row group&#39;s footer-recorded offset and decode just that chunk, which is exactly why Parquet files compressed with Snappy stay splittable even though the Snappy stream format itself provides no split points: it has &quot;no entropy encoder backend nor framing layer -- the latter is assumed to be handled by other parts of the system&quot;<sup class="footnote-ref"><a href="#fn3" id="fnref3">[3]</a></sup>.</p><img class="light-only" src="../assets/columnar-layout.PghGeOEA.svg" alt="A columnar file&#39;s footer holds row-group offsets and per-column min/max statistics that let a reader skip row groups whose stats exclude the predicate."><img class="dark-only" src="../assets/columnar-layout.dark.BVNlz0ff.svg" alt="A columnar file&#39;s footer holds row-group offsets and per-column min/max statistics that let a reader skip row groups whose stats exclude the predicate."><p>Both formats also carry the statistics that later drive predicate pushdown. Parquet records min/max values per column chunk, plus an optional page-level <code>ColumnIndex</code> that lets a reader binary-search ordered columns for matching pages<sup class="footnote-ref"><a href="#fn4" id="fnref4">[4]</a></sup>, and optional Bloom filters for columns whose cardinality is too high for a dictionary to be practical<sup class="footnote-ref"><a href="#fn5" id="fnref5">[5]</a></sup>. ORC&#39;s <code>ColumnStatistics</code> protobuf records row count and, for most primitive types, min/max (plus sum for numeric types); from Hive 1.1.0 it also records a <code>hasNull</code> flag used specifically by ORC&#39;s predicate pushdown to answer <code>IS NULL</code> queries<sup class="footnote-ref"><a href="#fn6" id="fnref6">[6]</a></sup>. ORC layers this at two granularities: a <code>RowIndexEntry</code> per row group (10,000 rows by default), kept at the front of each stripe so it&#39;s read only when pushdown or seeking is actually needed<sup class="footnote-ref"><a href="#fn7" id="fnref7">[7]</a></sup>, and file-level <code>StripeStatistics</code> that let whole stripes be skipped by predicate pushdown<sup class="footnote-ref"><a href="#fn8" id="fnref8">[8]</a></sup>. ORC also supports Bloom filters (Hive 1.2.0+), but only evaluates them against row groups that already passed the min/max row-index check first: a second-stage filter, not an independent one<sup class="footnote-ref"><a href="#fn7" id="fnref7:1">[7:1]</a></sup>.</p><p>On compression: Parquet defaults to Snappy, while ORC&#39;s default has been Zstd since Spark 2.3.0<sup class="footnote-ref"><a href="#fn9" id="fnref9">[9]</a></sup>. Zstandard&#39;s own manual describes it as &quot;a fast lossless compression algorithm, targeting real-time compression scenarios at zlib-level and better compression ratios,&quot; offering regular levels 1–22 plus negative levels that trade ratio for speed<sup class="footnote-ref"><a href="#fn10" id="fnref10">[10]</a></sup>. Spark exposes the level directly through <code>spark.io.compression.zstd.level</code> (default <code>1</code>)<sup class="footnote-ref"><a href="#fn9" id="fnref9:1">[9:1]</a></sup>, along with <code>spark.io.compression.zstd.workers</code> for parallel compression threads (default <code>0</code>, since 4.0.0) and <code>spark.io.compression.zstd.bufferSize</code> (default 32k)<sup class="footnote-ref"><a href="#fn9" id="fnref9:2">[9:2]</a></sup>.</p><p>Parquet has one more structural knob: <code>parquet.writer.version</code>. Version 2 changes the on-disk page-header encoding, and the spec is explicit that this is a forward-<em>incompatible</em> change: &quot;a reader that only understands <code>DataPageHeader</code> cannot parse <code>DataPageHeaderV2</code> pages&quot;<sup class="footnote-ref"><a href="#fn11" id="fnref11">[11]</a></sup>. The spec also flags that the file&#39;s own version marker &quot;has historically been used inconsistently: writers populate 1 or 2 without a consistent relationship to the features actually used&quot;<sup class="footnote-ref"><a href="#fn11" id="fnref11:1">[11:1]</a></sup>, so the marker itself isn&#39;t a reliable way to tell which page format a file actually contains.</p><h2 id="what-shows-up-at-read-time" tabindex="-1">What shows up at read time <a class="header-anchor" href="#what-shows-up-at-read-time" aria-label="Permalink to &quot;What shows up at read time&quot;"></a></h2><p>A file&#39;s layout shapes the job the moment Spark reads it. Spark&#39;s file readers derive partition counts from file metadata rather than from a fixed rule: &quot;for structured formats like Parquet, ORC, or Avro, Spark actually reads the metadata (footers, row groups, that kind of thing) and tries to slice the file in a way that makes sense. Often you&#39;ll see one partition per row group, though Spark may merge or split depending on file sizes and configs&quot;<sup class="footnote-ref"><a href="#fn12" id="fnref12">[12]</a></sup>. A concrete case: a single 30 GB Parquet file with 300 row groups gets split into exactly 300 partitions<sup class="footnote-ref"><a href="#fn12" id="fnref12:1">[12:1]</a></sup>. So a surprising partition count for a given file is usually explained by its row-group layout, separately from <code>spark.sql.files.maxPartitionBytes</code> (128 MB by default) and its <code>maxPartitionNum</code>/<code>minPartitionNum</code> companions, which govern splitting for file-based sources generally<sup class="footnote-ref"><a href="#fn13" id="fnref13">[13]</a></sup>.</p><p>The <a href="./bottleneck-small-files.html">small-files problem</a> shows up as metadata overhead rather than raw I/O cost: &quot;when you&#39;re writing lots of small files, there&#39;s a significant metadata overhead that you incur managing all of those files. Spark especially does not do well with small files&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1:3">[1:3]</a></sup>. At the other extreme, oversized or misaligned row groups show up as lost locality: the Parquet spec&#39;s rationale for matching block size to row-group size is that &quot;since an entire row group might need to be read, we want it to completely fit on one HDFS block&quot;<sup class="footnote-ref"><a href="#fn14" id="fnref14">[14]</a></sup>. A row group straddling a block boundary can&#39;t get that benefit.</p><p>Raw (non-Parquet) gzip or zip source files reveal themselves as a single-executor bottleneck at read time, visible as one task doing dramatically more work than the rest of its stage: Spark &quot;needs to download the whole file on one executor, unpack it on just one core, and then redistribute the partitions to the cluster nodes&quot;<sup class="footnote-ref"><a href="#fn15" id="fnref15">[15]</a></sup>.</p><p>Version and schema mismatches surface as read failures rather than silent corruption. A file written with <code>parquet.writer.version=2</code> can&#39;t be parsed by an older Spark version, or a non-Spark engine such as Hive or Impala, if that reader only implements the original <code>DataPageHeader</code><sup class="footnote-ref"><a href="#fn11" id="fnref11:2">[11:2]</a></sup>. <em>Spark: The Definitive Guide</em> frames this as a general risk: &quot;you can still encounter problems if you&#39;re working with incompatible Parquet files. Be careful when you write out Parquet files with different versions of Spark (especially older ones) because this can cause significant headache&quot;<sup class="footnote-ref"><a href="#fn1" id="fnref1:4">[1:4]</a></sup>.</p><h2 id="what-the-layout-buys-you" tabindex="-1">What the layout buys you <a class="header-anchor" href="#what-the-layout-buys-you" aria-label="Permalink to &quot;What the layout buys you&quot;"></a></h2><p>Those read-time symptoms trace back to specific tradeoffs in the layout. Splittability is what decides whether a file can be processed in parallel at all. A non-splittable raw compressed source file forces Spark onto a single core for the whole download-and-unpack step before it can redistribute anything: &quot;as you can imagine, this becomes a huge bottleneck in your distributed processing&quot;<sup class="footnote-ref"><a href="#fn15" id="fnref15:1">[15:1]</a></sup>. Parquet sidesteps this at the container level: because its footer records row-group and column-chunk offsets independently of whichever codec compressed the bytes inside them, a task can seek and decode a row group on its own, so even a non-splittable codec like Snappy doesn&#39;t cost the file its parallelism<sup class="footnote-ref"><a href="#fn2" id="fnref2:1">[2:1]</a></sup><sup class="footnote-ref"><a href="#fn3" id="fnref3:1">[3:1]</a></sup>.</p><p>Row-group and block-size alignment governs the same kind of locality at a coarser grain. The spec recommends large row groups, 512 MB–1 GB, &quot;because larger row groups allow for larger column chunks which makes it possible to do larger sequential IO,&quot; at the cost of more write-side buffering, paired with a block size sized the same way; its own worked example is &quot;1GB row groups, 1GB HDFS block size, 1 HDFS block per HDFS file&quot;<sup class="footnote-ref"><a href="#fn14" id="fnref14:1">[14:1]</a></sup>.</p><p>The statistics both formats carry are what predicate pushdown runs against. Spark plans queries directly off &quot;statistics that Spark reads directly from the underlying data source, like the counts and min/max values in the metadata of Parquet files&quot;<sup class="footnote-ref"><a href="#fn16" id="fnref16">[16]</a></sup>, gated by <code>spark.sql.parquet.filterPushdown</code> (default <code>true</code> since Spark 1.2.0) and, for ORC, <code>spark.sql.orc.filterPushdown</code> (default <code>true</code> since 1.4.0)<sup class="footnote-ref"><a href="#fn17" id="fnref17">[17]</a></sup><sup class="footnote-ref"><a href="#fn18" id="fnref18">[18]</a></sup>. A more aggressive option, <code>spark.sql.parquet.aggregatePushdown</code> (default <code>false</code>, since 3.3.0), pushes <code>MIN</code>, <code>MAX</code>, and <code>COUNT</code> down to Parquet&#39;s footer statistics directly, and throws if the needed statistic is missing from a file&#39;s footer<sup class="footnote-ref"><a href="#fn17" id="fnref17:1">[17:1]</a></sup>. Skipping a row group or stripe this way means its bytes are never read off disk, which beats any amount of post-read filtering.</p><p>Small files cost more in metadata management than their data volume would suggest, and Spark &quot;especially does not do well&quot; with them<sup class="footnote-ref"><a href="#fn1" id="fnref1:5">[1:5]</a></sup>. On the scheduling side, Spark&#39;s tuning guide notes it &quot;can efficiently support tasks as short as 200 ms&quot; because it reuses one executor JVM across many tasks and has low task-launch cost, while separately recommending &quot;2-3 tasks per CPU core in your cluster&quot;<sup class="footnote-ref"><a href="#fn19" id="fnref19">[19]</a></sup>, which implies <a href="./bottleneck-tiny-tasks.html">scheduling overhead</a> only becomes proportionally significant once tasks (and the files behind them) shrink below roughly that 200 ms floor.</p><p>Compression choice trades CPU against I/O and storage. Once I/O stops being the bottleneck, paying a codec&#39;s decompression cost is a net loss: &quot;uncompressed files are clearly outperforming compressed files. This is because uncompressed files are I/O bound, and compressed files are CPU bound, but I/O is good enough here&quot;<sup class="footnote-ref"><a href="#fn15" id="fnref15:2">[15:2]</a></sup>. Bzip2 illustrates the opposite failure mode: it&#39;s splittable, but compresses so aggressively that &quot;you get very few partitions and therefore they can be poorly distributed&quot;<sup class="footnote-ref"><a href="#fn15" id="fnref15:3">[15:3]</a></sup>.</p><p>Writer-version and schema changes matter because they fail at read time, for whoever reads the data next (not at write time, for whoever wrote it). A <code>parquet.writer.version=2</code> file is simply unreadable by any engine that only understands the original page header<sup class="footnote-ref"><a href="#fn11" id="fnref11:3">[11:3]</a></sup>. Schema drift has its own correctness angle: for <a href="./table-formats.html">Delta Lake</a> tables, adding a column via <code>mergeSchema</code> causes existing rows, when read back, to have that new column&#39;s value read as <code>NULL</code><sup class="footnote-ref"><a href="#fn20" id="fnref20">[20]</a></sup>, a defined outcome, but one that changes what a downstream query sees for rows written before the schema changed.</p><h2 id="choosing-and-tuning-the-format" tabindex="-1">Choosing and tuning the format <a class="header-anchor" href="#choosing-and-tuning-the-format" aria-label="Permalink to &quot;Choosing and tuning the format&quot;"></a></h2><p>A few defaults cover most cases despite those tradeoffs. Default to Parquet. Reach for ORC specifically when the same data also has to serve Hive consumers or existing Hive ORC tables, where ORC is the better-optimized target<sup class="footnote-ref"><a href="#fn1" id="fnref1:6">[1:6]</a></sup>.</p><p>Size row groups and the underlying filesystem block together, not independently. The Parquet spec&#39;s own recommended setup is 1 GB row groups paired with a 1 GB HDFS block size, one block per file<sup class="footnote-ref"><a href="#fn14" id="fnref14:2">[14:2]</a></sup>. Leave <code>spark.sql.files.maxPartitionBytes</code> (128 MB default) and its <code>maxPartitionNum</code>/<code>minPartitionNum</code> companions for the general file-splitting case, but expect row-group boundaries (not these settings) to be what actually decides partition count for row-group-oriented formats<sup class="footnote-ref"><a href="#fn13" id="fnref13:1">[13:1]</a></sup><sup class="footnote-ref"><a href="#fn12" id="fnref12:2">[12:2]</a></sup>.</p><p>Leave predicate pushdown on: <code>spark.sql.parquet.filterPushdown</code> and <code>spark.sql.orc.filterPushdown</code> both default to <code>true</code> already<sup class="footnote-ref"><a href="#fn17" id="fnref17:2">[17:2]</a></sup><sup class="footnote-ref"><a href="#fn18" id="fnref18:1">[18:1]</a></sup>, so the main action is not disabling them, plus considering <code>spark.sql.parquet.aggregatePushdown</code> when a workload is dominated by <code>MIN</code>/<code>MAX</code>/<code>COUNT</code> over Parquet sources with complete footer statistics<sup class="footnote-ref"><a href="#fn17" id="fnref17:3">[17:3]</a></sup>.</p><p>Cap output file size directly instead of letting the small-files problem accumulate: <code>maxRecordsPerFile</code>, introduced in Spark 2.2, targets an optimum file size by capping the number of records written per file, e.g. <code>df.write.option(&quot;maxRecordsPerFile&quot;, 5000)</code><sup class="footnote-ref"><a href="#fn1" id="fnref1:7">[1:7]</a></sup>.</p><p>Pick a compression codec based on the actual bottleneck. Parquet&#39;s default, Snappy, favors speed; ORC has already defaulted to Zstd since Spark 2.3.0<sup class="footnote-ref"><a href="#fn9" id="fnref9:3">[9:3]</a></sup>, and at least one optimization checklist recommends overriding Parquet&#39;s default to Zstd as well<sup class="footnote-ref"><a href="#fn21" id="fnref21">[21]</a></sup>. Zstd&#39;s level is tunable through <code>spark.io.compression.zstd.level</code>: higher levels buy better compression &quot;at the expense of more CPU and memory&quot;<sup class="footnote-ref"><a href="#fn9" id="fnref9:4">[9:4]</a></sup>. Avoid feeding Spark large raw <code>.gz</code>/<code>.zip</code> source files directly (unpack them before loading), since the non-splittability lives in the raw container format rather than in gzip itself; the same <em>Definitive Guide</em> that warns about raw gzip files elsewhere still recommends Parquet with gzip compression once gzip is wrapped inside Parquet&#39;s row-group container<sup class="footnote-ref"><a href="#fn1" id="fnref1:8">[1:8]</a></sup>.</p><p>Treat <code>spark.sql.parquet.mergeSchema</code> (default <code>false</code>) as opt-in rather than default-on: enabling it &quot;merges schemas collected from all data files,&quot; instead of trusting a single summary file or a random file&#39;s schema<sup class="footnote-ref"><a href="#fn22" id="fnref22">[22]</a></sup>, useful for evolving schemas, but it means every file in the dataset gets scanned for its schema. For Delta tables specifically, remember that <code>mergeSchema</code>-added columns read back as <code>NULL</code> on pre-existing rows<sup class="footnote-ref"><a href="#fn20" id="fnref20:1">[20:1]</a></sup> before relying on it for a backfill.</p><p>Don&#39;t flip <code>parquet.writer.version</code> to <code>2</code> without confirming every downstream reader of that data (an older Spark version, Hive, Impala, or anything else touching the files) actually supports <code>DataPageHeaderV2</code> first; it&#39;s a forward-incompatible page-format change, not an additive one<sup class="footnote-ref"><a href="#fn11" id="fnref11:4">[11:4]</a></sup><sup class="footnote-ref"><a href="#fn1" id="fnref1:9">[1:9]</a></sup>.</p><h2 id="sources" tabindex="-1">Sources <a class="header-anchor" href="#sources" aria-label="Permalink to &quot;Sources&quot;"></a></h2><hr class="footnotes-sep"><section class="footnotes"><ol class="footnotes-list"><li id="fn1" class="footnote-item"><p><em>Spark: The Definitive Guide</em>, Chambers &amp; Zaharia, ch. 9 <a href="#fnref1" class="footnote-backref">↩︎</a> <a href="#fnref1:1" class="footnote-backref">↩︎</a> <a href="#fnref1:2" class="footnote-backref">↩︎</a> <a href="#fnref1:3" class="footnote-backref">↩︎</a> <a href="#fnref1:4" class="footnote-backref">↩︎</a> <a href="#fnref1:5" class="footnote-backref">↩︎</a> <a href="#fnref1:6" class="footnote-backref">↩︎</a> <a href="#fnref1:7" class="footnote-backref">↩︎</a> <a href="#fnref1:8" class="footnote-backref">↩︎</a> <a href="#fnref1:9" class="footnote-backref">↩︎</a></p></li><li id="fn2" class="footnote-item"><p><a href="https://parquet.apache.org/_print/docs/file-format/" target="_blank" rel="noreferrer">Parquet File Format: File Metadata and Row Groups</a> <a href="#fnref2" class="footnote-backref">↩︎</a> <a href="#fnref2:1" class="footnote-backref">↩︎</a></p></li><li id="fn3" class="footnote-item"><p><a href="https://github.com/google/snappy/blob/main/format_description.txt" target="_blank" rel="noreferrer">Snappy Compressed Format Description</a> <a href="#fnref3" class="footnote-backref">↩︎</a> <a href="#fnref3:1" class="footnote-backref">↩︎</a></p></li><li id="fn4" class="footnote-item"><p><a href="https://parquet.apache.org/_print/docs/file-format/" target="_blank" rel="noreferrer">Parquet File Format: Column Index</a> <a href="#fnref4" class="footnote-backref">↩︎</a></p></li><li id="fn5" class="footnote-item"><p><a href="https://parquet.apache.org/docs/file-format/bloomfilter/" target="_blank" rel="noreferrer">Parquet File Format: Bloom Filter</a> <a href="#fnref5" class="footnote-backref">↩︎</a></p></li><li id="fn6" class="footnote-item"><p><a href="https://orc.apache.org/specification/ORCv1/" target="_blank" rel="noreferrer">ORC Specification v1: Column Statistics</a> <a href="#fnref6" class="footnote-backref">↩︎</a></p></li><li id="fn7" class="footnote-item"><p><a href="https://orc.apache.org/specification/ORCv1/" target="_blank" rel="noreferrer">ORC Specification v1: Row Index and Bloom Filters</a> <a href="#fnref7" class="footnote-backref">↩︎</a> <a href="#fnref7:1" class="footnote-backref">↩︎</a></p></li><li id="fn8" class="footnote-item"><p><a href="https://orc.apache.org/specification/ORCv1/" target="_blank" rel="noreferrer">ORC Specification v1: Stripe Statistics</a> <a href="#fnref8" class="footnote-backref">↩︎</a></p></li><li id="fn9" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration: Spark</a> <a href="#fnref9" class="footnote-backref">↩︎</a> <a href="#fnref9:1" class="footnote-backref">↩︎</a> <a href="#fnref9:2" class="footnote-backref">↩︎</a> <a href="#fnref9:3" class="footnote-backref">↩︎</a> <a href="#fnref9:4" class="footnote-backref">↩︎</a></p></li><li id="fn10" class="footnote-item"><p><a href="https://facebook.github.io/zstd/zstd_manual.html" target="_blank" rel="noreferrer">Zstandard Manual</a> <a href="#fnref10" class="footnote-backref">↩︎</a></p></li><li id="fn11" class="footnote-item"><p><a href="https://parquet.apache.org/_print/docs/file-format/" target="_blank" rel="noreferrer">Parquet File Format: Data Pages (V1/V2)</a> <a href="#fnref11" class="footnote-backref">↩︎</a> <a href="#fnref11:1" class="footnote-backref">↩︎</a> <a href="#fnref11:2" class="footnote-backref">↩︎</a> <a href="#fnref11:3" class="footnote-backref">↩︎</a> <a href="#fnref11:4" class="footnote-backref">↩︎</a></p></li><li id="fn12" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-partitions" target="_blank" rel="noreferrer">How Spark Determines Partitions for a File</a> <a href="#fnref12" class="footnote-backref">↩︎</a> <a href="#fnref12:1" class="footnote-backref">↩︎</a> <a href="#fnref12:2" class="footnote-backref">↩︎</a></p></li><li id="fn13" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html" target="_blank" rel="noreferrer">Performance Tuning: Spark SQL, DataFrames and Datasets Guide</a> <a href="#fnref13" class="footnote-backref">↩︎</a> <a href="#fnref13:1" class="footnote-backref">↩︎</a></p></li><li id="fn14" class="footnote-item"><p><a href="https://parquet.apache.org/_print/docs/file-format/" target="_blank" rel="noreferrer">Parquet File Format: Row Group Size</a> <a href="#fnref14" class="footnote-backref">↩︎</a> <a href="#fnref14:1" class="footnote-backref">↩︎</a> <a href="#fnref14:2" class="footnote-backref">↩︎</a></p></li><li id="fn15" class="footnote-item"><p><a href="https://luminousmen.com/post/spark-tips-dont-collect-data-on-driver" target="_blank" rel="noreferrer">Spark Tips: Don&#39;t Collect Data on Driver</a> <a href="#fnref15" class="footnote-backref">↩︎</a> <a href="#fnref15:1" class="footnote-backref">↩︎</a> <a href="#fnref15:2" class="footnote-backref">↩︎</a> <a href="#fnref15:3" class="footnote-backref">↩︎</a></p></li><li id="fn16" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/sql-performance-tuning.html" target="_blank" rel="noreferrer">Performance Tuning: Spark SQL, DataFrames and Datasets Guide</a> <a href="#fnref16" class="footnote-backref">↩︎</a></p></li><li id="fn17" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration: Spark</a> <a href="#fnref17" class="footnote-backref">↩︎</a> <a href="#fnref17:1" class="footnote-backref">↩︎</a> <a href="#fnref17:2" class="footnote-backref">↩︎</a> <a href="#fnref17:3" class="footnote-backref">↩︎</a></p></li><li id="fn18" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/configuration.html" target="_blank" rel="noreferrer">Configuration: Spark</a> <a href="#fnref18" class="footnote-backref">↩︎</a> <a href="#fnref18:1" class="footnote-backref">↩︎</a></p></li><li id="fn19" class="footnote-item"><p><a href="https://spark.apache.org/docs/latest/tuning.html" target="_blank" rel="noreferrer">Spark Tuning Guide: Level of Parallelism</a> <a href="#fnref19" class="footnote-backref">↩︎</a></p></li><li id="fn20" class="footnote-item"><p><em>Learning Spark, 2nd Edition</em>, Damji, Wenig, Das &amp; Lee, ch. 9 <a href="#fnref20" class="footnote-backref">↩︎</a> <a href="#fnref20:1" class="footnote-backref">↩︎</a></p></li><li id="fn21" class="footnote-item"><p><a href="https://luminousmen.com/post/the-apache-spark-optimization-checklist" target="_blank" rel="noreferrer">The Apache Spark Optimization Checklist</a> <a href="#fnref21" class="footnote-backref">↩︎</a></p></li><li id="fn22" class="footnote-item"><p><a href="https://raw.githubusercontent.com/apache/spark/v3.5.0/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala" target="_blank" rel="noreferrer">SQLConf.scala</a> <a href="#fnref22" class="footnote-backref">↩︎</a></p></li></ol></section></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><a class="VPLink link pager-link prev" href="../tuning-reference/shuffle.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Previous page</span><span class="title" data-v-e257564d>Shuffle</span><!--]--></a></div><div class="pager" data-v-e257564d><a class="VPLink link pager-link next" href="../tuning-reference/table-formats.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Next page</span><span class="title" data-v-e257564d>Table Formats</span><!--]--></a></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
+  </body>
+</html>

package/export-template/docs/tuning-reference/index.html ADDED Viewed

@@ -0,0 +1,25 @@
+<!DOCTYPE html>
+<html lang="en-US" dir="ltr">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width,initial-scale=1">
+    <title>Spark Tuning Reference | SparkForensics</title>
+    <meta name="description" content="Docs for using and contributing to SparkForensics">
+    <meta name="generator" content="VitePress v1.6.4">
+    <link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
+    <link rel="preload stylesheet" href="../vp-icons.css" as="style">
+    <link rel="icon" type="image/svg+xml" href="../favicon.svg">
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="">
+    <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Recursive:wght,CASL@400..700,0..1&amp;family=JetBrains+Mono:wght@400;500;600&amp;display=swap">
+    <script id="check-dark-mode">(()=>{const e=localStorage.getItem("vitepress-theme-appearance")||"auto",a=window.matchMedia("(prefers-color-scheme: dark)").matches;(!e||e==="auto"?a:e==="dark")&&document.documentElement.classList.add("dark")})();</script>
+    <script id="check-mac-os">document.documentElement.classList.toggle("mac",/Mac|iPhone|iPod|iPad/i.test(navigator.platform));</script>
+  </head>
+  <body>
+    <div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle" data-v-6aa21345 data-v-1168a8e4><a class="title" href="../index.html" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="../favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink active" href="../tuning-reference/index.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav empty fixed" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><!----><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><!----><div class="VPContent" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPPage" data-v-1428d186><!--[--><!--]--><div style="position:relative;"><div><main id="main-content" class="tuning-landing" data-v-5d65fdb9><section class="hero" aria-labelledby="page-title" data-v-5d65fdb9><div class="container hero-grid" data-v-5d65fdb9><div class="hero-copy" data-v-5d65fdb9><p class="eyebrow" data-v-5d65fdb9>Practical mechanics, stable references</p><h1 id="page-title" data-v-5d65fdb9>Spark tuning reference</h1><p class="hero-summary" data-v-5d65fdb9> Move from a job symptom to the Spark mechanics and settings that can change its outcome without treating a tuning guess as evidence. </p><div class="hero-actions" data-v-5d65fdb9><a class="button button-primary" href="../tuning-reference/intro.html" data-v-5d65fdb9>Browse the full reference <span class="arrow" aria-hidden="true" data-v-5d65fdb9>→</span></a><a class="button button-secondary" href="#symptoms" data-v-5d65fdb9>Choose a symptom</a><form class="known-term" data-v-5d65fdb9><label for="known-term-input" data-v-5d65fdb9>Know the term or config key?</label><div class="known-term-row" data-v-5d65fdb9><input id="known-term-input" value="" name="term" type="search" list="known-term-options" placeholder="e.g. spark.sql.shuffle.partitions" autocomplete="off" data-v-5d65fdb9><button class="button button-secondary" type="submit" data-v-5d65fdb9>Jump to topic</button></div><small data-v-5d65fdb9>Known routes: memory, shuffle, joins, AQE, and config. Other terms open the configuration reference.</small><datalist id="known-term-options" data-v-5d65fdb9><option value="memory" data-v-5d65fdb9></option><option value="shuffle" data-v-5d65fdb9></option><option value="joins" data-v-5d65fdb9></option><option value="AQE" data-v-5d65fdb9></option><option value="config" data-v-5d65fdb9></option><option value="spark.memory.fraction" data-v-5d65fdb9></option><option value="spark.sql.shuffle.partitions" data-v-5d65fdb9></option><option value="spark.sql.adaptive.enabled" data-v-5d65fdb9></option><option value="spark.sql.autoBroadcastJoinThreshold" data-v-5d65fdb9></option></datalist></form></div></div><aside class="signal-board" aria-label="Example symptom paths" data-v-5d65fdb9><div class="board-label" data-v-5d65fdb9>evidence → mechanic → lever</div><ul class="signal-list" data-v-5d65fdb9><li data-v-5d65fdb9><a class="signal-link" href="../tuning-reference/memory-model.html" data-v-5d65fdb9><span class="signal-marker" aria-hidden="true" data-v-5d65fdb9></span><span class="signal-title" data-v-5d65fdb9>Executor memory pressure</span><span class="signal-destination" data-v-5d65fdb9>#memory-model</span></a></li><li data-v-5d65fdb9><a class="signal-link" href="../tuning-reference/shuffle.html" data-v-5d65fdb9><span class="signal-marker" aria-hidden="true" data-v-5d65fdb9></span><span class="signal-title" data-v-5d65fdb9>Slow, wide stages</span><span class="signal-destination" data-v-5d65fdb9>#shuffle</span></a></li><li data-v-5d65fdb9><a class="signal-link" href="../tuning-reference/joins.html" data-v-5d65fdb9><span class="signal-marker" aria-hidden="true" data-v-5d65fdb9></span><span class="signal-title" data-v-5d65fdb9>Join strategy drift</span><span class="signal-destination" data-v-5d65fdb9>#joins</span></a></li></ul></aside></div></section><section class="section" id="symptoms" aria-labelledby="symptoms-title" data-v-5d65fdb9><div class="container" data-v-5d65fdb9><div class="section-heading" data-v-5d65fdb9><h2 id="symptoms-title" data-v-5d65fdb9>Symptoms, mapped to the right lever</h2><p data-v-5d65fdb9>Each path stays on the canonical reference document, so existing links and saved anchors continue to work.</p><p class="decision-note" data-v-5d65fdb9><strong data-v-5d65fdb9>How to use this:</strong> SparkForensics reads your Spark event log and surfaces evidence; this reference explains the mechanics and settings behind it. Diagnose the symptom, change one informed lever, then validate against the next run&#39;s evidence. Confirm behavior and defaults against your deployed Spark version and distribution. </p></div><div class="symptom-grid" data-v-5d65fdb9><article class="symptom" data-v-5d65fdb9><div data-v-5d65fdb9><h3 data-v-5d65fdb9>Spill or GC pressure</h3><p data-v-5d65fdb9>Trace executor memory, object lifetime, and the trade-offs behind changing memory settings.</p></div><a href="../tuning-reference/memory-model.html" data-v-5d65fdb9>Open memory model <span aria-hidden="true" data-v-5d65fdb9>→</span></a></article><article class="symptom" data-v-5d65fdb9><div data-v-5d65fdb9><h3 data-v-5d65fdb9>Shuffle-heavy stages</h3><p data-v-5d65fdb9>Understand write/read costs, partition size, and why moving data becomes the bottleneck.</p></div><a href="../tuning-reference/shuffle.html" data-v-5d65fdb9>Open shuffle guide <span aria-hidden="true" data-v-5d65fdb9>→</span></a></article><article class="symptom" data-v-5d65fdb9><div data-v-5d65fdb9><h3 data-v-5d65fdb9>Slow or skewed joins</h3><p data-v-5d65fdb9>Compare join mechanics before changing broadcast thresholds or reshaping a plan.</p></div><a href="../tuning-reference/joins.html" data-v-5d65fdb9>Open join optimization <span aria-hidden="true" data-v-5d65fdb9>→</span></a></article><article class="symptom" data-v-5d65fdb9><div data-v-5d65fdb9><h3 data-v-5d65fdb9>Plan adapts poorly</h3><p data-v-5d65fdb9>Use AQE&#39;s runtime choices as an inspectable mechanism, not a checkbox to enable blindly.</p></div><a href="../tuning-reference/aqe.html" data-v-5d65fdb9>Open AQE guide <span aria-hidden="true" data-v-5d65fdb9>→</span></a></article><article class="symptom" data-v-5d65fdb9><div data-v-5d65fdb9><h3 data-v-5d65fdb9>Need the exact setting</h3><p data-v-5d65fdb9>Check a setting&#39;s scope, default, and interaction before applying it to a cluster.</p></div><a href="../tuning-reference/config.html" data-v-5d65fdb9>Open config reference <span aria-hidden="true" data-v-5d65fdb9>→</span></a></article></div></div></section><section class="section workflow" aria-labelledby="workflow-title" data-v-5d65fdb9><div class="container workflow-grid" data-v-5d65fdb9><div class="workflow-copy" data-v-5d65fdb9><h2 id="workflow-title" data-v-5d65fdb9>Let SparkForensics surface the evidence. Use this reference to reason about it.</h2><p data-v-5d65fdb9>The two tools stay deliberately separate: SparkForensics reads the event log; this static reference explains the mechanics and settings behind what you find.</p></div><ol class="steps" data-v-5d65fdb9><li data-v-5d65fdb9><span class="step-label" data-v-5d65fdb9>Observe</span><div data-v-5d65fdb9><strong data-v-5d65fdb9>Read the stage or executor behavior</strong><span data-v-5d65fdb9>Start with the event log, metrics, and plan details Spark already produces.</span></div></li><li data-v-5d65fdb9><span class="step-label" data-v-5d65fdb9>Explain</span><div data-v-5d65fdb9><strong data-v-5d65fdb9>Name the mechanism at work</strong><span data-v-5d65fdb9>Use the stable reference anchors to connect symptoms to memory, shuffle, joins, AQE, or configuration.</span></div></li><li data-v-5d65fdb9><span class="step-label" data-v-5d65fdb9>Test</span><div data-v-5d65fdb9><strong data-v-5d65fdb9>Change one informed lever</strong><span data-v-5d65fdb9>Make a measured change, then compare the evidence again rather than carrying a tuning superstition forward.</span></div></li></ol></div></section></main></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
+  </body>
+</html>

package/export-template/docs/tuning-reference/intro.html ADDED Viewed

@@ -0,0 +1,25 @@
+<!DOCTYPE html>
+<html lang="en-US" dir="ltr">
+  <head>
+    <meta charset="utf-8">
+    <meta name="viewport" content="width=device-width,initial-scale=1">
+    <title>Introduction | SparkForensics</title>
+    <meta name="description" content="Docs for using and contributing to SparkForensics">
+    <meta name="generator" content="VitePress v1.6.4">
+    <link rel="preload stylesheet" href="../assets/style.DSixAiZE.css" as="style">
+    <link rel="preload stylesheet" href="../vp-icons.css" as="style">
+    <link rel="icon" type="image/svg+xml" href="../favicon.svg">
+    <link rel="preconnect" href="https://fonts.googleapis.com">
+    <link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="">
+    <link rel="stylesheet" href="https://fonts.googleapis.com/css2?family=Recursive:wght,CASL@400..700,0..1&amp;family=JetBrains+Mono:wght@400;500;600&amp;display=swap">
+    <script id="check-dark-mode">(()=>{const e=localStorage.getItem("vitepress-theme-appearance")||"auto",a=window.matchMedia("(prefers-color-scheme: dark)").matches;(!e||e==="auto"?a:e==="dark")&&document.documentElement.classList.add("dark")})();</script>
+    <script id="check-mac-os">document.documentElement.classList.toggle("mac",/Mac|iPhone|iPod|iPad/i.test(navigator.platform));</script>
+  </head>
+  <body>
+    <div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="../index.html" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="../favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="../tuning-reference/index.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Getting Started</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/intro.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Introduction</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Optimization Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/spark-architecture.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Spark Execution Model</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/memory-model.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory Management</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/partitioning.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Partitioning</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/joins.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Join Optimization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/shuffle.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Shuffle</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/data-formats.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Data Formats</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/table-formats.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Table Formats</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/caching.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Caching & Persistence</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/pyspark.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>PySpark Specifics</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/aqe.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Adaptive Query Execution</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/cluster-config.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Cluster Tuning</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/anti-patterns.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Anti-Patterns</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Detector Catalog</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-skew.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Task Skew</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-shuffle.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Shuffle I/O</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-spill.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory / Disk Spill</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-gc.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>GC Pressure</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-cold-start.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Cold Start</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-utilization.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Executor Utilization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-slow-host.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Slow Host</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-failures.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Task Failures</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-straggler.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Stragglers</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-retry-waste.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Retry Waste</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-tiny-tasks.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Tiny Tasks</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-job-failure-rate.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Job Failure Rate</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-memory-utilization.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Memory Utilization</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-duplicate-plan-subtree.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Duplicate Plan Subtree</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-small-files.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Small Files</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/bottleneck-broadcast-sizing.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Broadcast Sizing</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>Reference</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/metrics.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Metrics Glossary</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="../tuning-reference/config.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Spark Config Quick-Reference</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_tuning-reference_intro" data-v-39a288b8><div><h1 id="intro" tabindex="-1">Introduction <a class="header-anchor" href="#intro" aria-label="Permalink to &quot;Introduction {#intro}&quot;"></a></h1><p>This guide is a hands-on reference for optimizing Spark and PySpark jobs. It covers <a href="./spark-architecture.html">Spark internals</a>, <a href="./memory-model.html">memory management</a>, <a href="./joins.html">joins</a>, <a href="./shuffle.html">shuffle</a>, <a href="./data-formats.html">data formats</a>, <a href="./pyspark.html">PySpark-specific patterns</a>, <a href="./aqe.html">Adaptive Query Execution</a> (AQE), and <a href="./cluster-config.html">cluster tuning</a>, followed by a bottleneck diagnostic reference.</p><p>Read it top to bottom for a full grounding in Spark performance, or jump straight to the <strong>Bottleneck Reference</strong> section below when you already know which symptom you&#39;re chasing.</p><h2 id="severity-dots" tabindex="-1">Severity dots <a class="header-anchor" href="#severity-dots" aria-label="Permalink to &quot;Severity dots&quot;"></a></h2><p>Each finding is marked with a severity dot:</p><ul><li><span class="severity-dot info"></span> <strong>Info</strong>: worth knowing, not yet a problem</li><li><span class="severity-dot warning"></span> <strong>Warning</strong>: likely hurting job performance</li><li><span class="severity-dot critical"></span> <strong>Critical</strong>: actively bottlenecking the job</li></ul><p>Each bottleneck section below documents the exact thresholds behind these dots.</p><h2 id="tag-system" tabindex="-1">Tag system <a class="header-anchor" href="#tag-system" aria-label="Permalink to &quot;Tag system&quot;"></a></h2><p>Every bottleneck has a short tag used throughout this reference:</p><p><span class="tag">SKEW</span> <span class="tag">SHFL</span> <span class="tag">SPILL</span><span class="tag">GC</span> <span class="tag">COLD</span> <span class="tag">UTIL</span><span class="tag">HOST</span> <span class="tag">FAIL</span> <span class="tag">STRAG</span><span class="tag">RETRY</span> <span class="tag">TINY</span> <span class="tag">FAIL-RATE</span></p><p>See the <a href="./bottleneck-skew.html">Bottleneck Reference</a> for what each one means, how it&#39;s detected, and how to fix it.</p></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><!----></div><div class="pager" data-v-e257564d><a class="VPLink link pager-link next" href="../tuning-reference/spark-architecture.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Next page</span><span class="title" data-v-e257564d>Spark Execution Model</span><!--]--></a></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
+  </body>
+</html>