sparkforensics-server 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/index.js +1 -1
- package/lib/static-files.js +5 -4
- package/package.json +4 -3
- package/public/assets/AutoscalingChurn-B96FKx97.js +1 -0
- package/public/assets/CacheUtilization-lKua3J1x.js +1 -0
- package/public/assets/CachingOpportunity-U0g-8HPV.js +1 -0
- package/public/assets/ColdStart-w8QYTNKo.js +1 -0
- package/public/assets/ConfigAudit-ClZp5BpV.js +1 -0
- package/public/assets/CoreUsageArea-o3kqQwyi.js +1 -0
- package/public/assets/Dashboard-Bjl-MYMm.js +1 -0
- package/public/assets/DuplicatePlanSubtree-Cm_UU3lY.js +1 -0
- package/public/assets/DurationHistogram-ltF-7M0A.js +1 -0
- package/public/assets/EmptyState-BCA3xioa.js +1 -0
- package/public/assets/ExecutorUtilization-Cn69i0uq.js +1 -0
- package/public/assets/GcPressure-CYsmU1d7.js +1 -0
- package/public/assets/ImpactEstimate-BGt-4WX5.js +1 -0
- package/public/assets/IncompleteRun-s6uQ8OLf.js +1 -0
- package/public/assets/JobFailures-DJ2dDvPA.js +1 -0
- package/public/assets/MemoryUtilization-C9G7P94X.js +1 -0
- package/public/assets/OverBroadcast-DYauoYWQ.js +1 -0
- package/public/assets/PartitionSizing-Cz_B0-uG.js +1 -0
- package/public/assets/PlanExplorer-_2oB5ouy.js +1 -0
- package/public/assets/PlanGraphRoute-Cj-GWM8W.js +1 -0
- package/public/assets/RetryWaste-D_m8ASwP.js +1 -0
- package/public/assets/RowPagination-BLsm8I6i.js +1 -0
- package/public/assets/ShuffleIO-pyDhxdaz.js +1 -0
- package/public/assets/Skew-BGLJuZZR.js +1 -0
- package/public/assets/SlowHost-ChdWVvGO.js +1 -0
- package/public/assets/SmallFiles-gTilBtjW.js +1 -0
- package/public/assets/SpeculationWaste-CVzgJnSj.js +1 -0
- package/public/assets/Spill-NGJGQaTB.js +1 -0
- package/public/assets/StageFailed-sGhki53z.js +1 -0
- package/public/assets/StageFindingGroup-AyI7BMd3.js +1 -0
- package/public/assets/StageHeader-BTzGkK8k.js +1 -0
- package/public/assets/StagePill-BlMOW_pU.js +1 -0
- package/public/assets/StageShape-DfOW93Z1.js +1 -0
- package/public/assets/StageSlowness-BMZ-K_5e.js +1 -0
- package/public/assets/Straggler-dffcAN7t.js +1 -0
- package/public/assets/TaskFailures-DGDwUK2A.js +1 -0
- package/public/assets/TinyTask-CfT8zZDr.js +1 -0
- package/public/assets/UnderBroadcast-CiOiJ0k1.js +1 -0
- package/public/assets/WidgetLeadSummary-BdUUi7L5.js +1 -0
- package/public/assets/charts-vendor-0VMaLVLN.js +59 -0
- package/public/assets/downsample-WmNM1qc9.js +1 -0
- package/public/assets/finding-anchor-DEV8JvgH.js +1 -0
- package/public/assets/impact-sort-DxaUheWV.js +1 -0
- package/public/assets/inbox-_QCYkBsP.js +1 -0
- package/public/assets/index-3vlFVNCg.js +9 -0
- package/public/assets/index-CjVklAdW.css +2 -0
- package/public/assets/parser-worker-B2bc0lWe.js +112 -0
- package/public/assets/plan-graph-vendor-B0sEms9Z.js +7 -0
- package/public/assets/plan-graph-vendor-DLioOiRN.css +1 -0
- package/public/assets/react-vendor-D2TLlSzI.js +9 -0
- package/public/assets/rolldown-runtime-hePW80VL.js +1 -0
- package/public/assets/select-kuy1MiAo.js +1 -0
- package/public/assets/toggle-group-CUd-KWkp.js +1 -0
- package/public/assets/usePagedRows-jNUWgVTD.js +1 -0
- package/public/assets/useRecentFiles-D0OA_EOK.js +11 -0
- package/public/assets/useSortMode-BaeAA5cw.js +1 -0
- package/public/docs/404.html +3 -3
- package/public/docs/assets/{app.BgzsaTS3.js → app.APoq_UBs.js} +1 -1
- package/public/docs/assets/chunks/@localSearchIndexroot.CvM5fuxe.js +1 -0
- package/public/docs/assets/chunks/{VPLocalSearchBox.DOq3t-On.js → VPLocalSearchBox.B3TPzlHq.js} +1 -1
- package/public/docs/assets/chunks/theme.PavXO_vz.js +2 -0
- package/public/docs/assets/contributor-guide_architecture_board-widgets.md.CWpj01WU.js +1 -0
- package/public/docs/assets/contributor-guide_architecture_board-widgets.md.CWpj01WU.lean.js +1 -0
- package/public/docs/assets/contributor-guide_architecture_detector-contract.md.CgzUsQ6W.js +1 -0
- package/public/docs/assets/{contributor-guide_architecture_impact-estimation.md.DYCDPgkh.js → contributor-guide_architecture_impact-estimation.md.CooslVJt.js} +1 -1
- package/public/docs/assets/contributor-guide_architecture_state-and-history.md.C-xxn0q7.js +1 -0
- package/public/docs/assets/contributor-guide_architecture_state-and-history.md.C-xxn0q7.lean.js +1 -0
- package/public/docs/assets/contributor-guide_architecture_widget-rendering.md.R27gQrgY.js +1 -0
- package/public/docs/assets/contributor-guide_architecture_worker-protocol.md.IbnfNrV3.js +6 -0
- package/public/docs/assets/{style.DXOMCXxn.css → style.DSixAiZE.css} +1 -1
- package/public/docs/assets/{user-guide_alternative-log-retrieval.md.sU3KGarf.js → user-guide_alternative-log-retrieval.md.B4tPGIal.js} +1 -1
- package/public/docs/assets/{user-guide_alternative-log-retrieval.md.sU3KGarf.lean.js → user-guide_alternative-log-retrieval.md.B4tPGIal.lean.js} +1 -1
- package/public/docs/assets/user-guide_getting-started.md.BJvwLEIM.js +3 -0
- package/public/docs/assets/user-guide_getting-started.md.BJvwLEIM.lean.js +1 -0
- package/public/docs/assets/{user-guide_mcp-tools.md.C8MiIu7F.js → user-guide_mcp-tools.md.Vi3RoflJ.js} +3 -3
- package/public/docs/assets/{user-guide_mcp-tools.md.C8MiIu7F.lean.js → user-guide_mcp-tools.md.Vi3RoflJ.lean.js} +1 -1
- package/public/docs/assets/user-guide_run-comparison.md.CQc1aoU8.js +1 -0
- package/public/docs/assets/user-guide_run-comparison.md.CQc1aoU8.lean.js +1 -0
- package/public/docs/assets/user-guide_understanding-findings.md.DL1UDhvR.js +1 -0
- package/public/docs/assets/user-guide_understanding-findings.md.DL1UDhvR.lean.js +1 -0
- package/public/docs/contributor-guide/architecture/board-widgets.html +6 -6
- package/public/docs/contributor-guide/architecture/detector-contract.html +6 -6
- package/public/docs/contributor-guide/architecture/drill-down.html +4 -4
- package/public/docs/contributor-guide/architecture/impact-estimation.html +6 -6
- package/public/docs/contributor-guide/architecture/index.html +4 -4
- package/public/docs/contributor-guide/architecture/overview.html +4 -4
- package/public/docs/contributor-guide/architecture/state-and-history.html +6 -6
- package/public/docs/contributor-guide/architecture/widget-rendering.html +6 -6
- package/public/docs/contributor-guide/architecture/worker-protocol.html +6 -6
- package/public/docs/contributor-guide/contributing.html +4 -4
- package/public/docs/contributor-guide/development-setup.html +4 -4
- package/public/docs/contributor-guide/testing.html +4 -4
- package/public/docs/hashmap.json +1 -1
- package/public/docs/index.html +4 -4
- package/public/docs/tuning-reference/anti-patterns.html +4 -4
- package/public/docs/tuning-reference/aqe.html +4 -4
- package/public/docs/tuning-reference/bottleneck-broadcast-sizing.html +4 -4
- package/public/docs/tuning-reference/bottleneck-cold-start.html +4 -4
- package/public/docs/tuning-reference/bottleneck-duplicate-plan-subtree.html +4 -4
- package/public/docs/tuning-reference/bottleneck-failures.html +4 -4
- package/public/docs/tuning-reference/bottleneck-gc.html +4 -4
- package/public/docs/tuning-reference/bottleneck-job-failure-rate.html +4 -4
- package/public/docs/tuning-reference/bottleneck-memory-utilization.html +4 -4
- package/public/docs/tuning-reference/bottleneck-retry-waste.html +4 -4
- package/public/docs/tuning-reference/bottleneck-shuffle.html +4 -4
- package/public/docs/tuning-reference/bottleneck-skew.html +4 -4
- package/public/docs/tuning-reference/bottleneck-slow-host.html +4 -4
- package/public/docs/tuning-reference/bottleneck-small-files.html +4 -4
- package/public/docs/tuning-reference/bottleneck-spill.html +4 -4
- package/public/docs/tuning-reference/bottleneck-straggler.html +4 -4
- package/public/docs/tuning-reference/bottleneck-tiny-tasks.html +4 -4
- package/public/docs/tuning-reference/bottleneck-utilization.html +4 -4
- package/public/docs/tuning-reference/caching.html +4 -4
- package/public/docs/tuning-reference/cluster-config.html +4 -4
- package/public/docs/tuning-reference/config.html +4 -4
- package/public/docs/tuning-reference/data-formats.html +4 -4
- package/public/docs/tuning-reference/index.html +4 -4
- package/public/docs/tuning-reference/intro.html +4 -4
- package/public/docs/tuning-reference/joins.html +4 -4
- package/public/docs/tuning-reference/memory-model.html +4 -4
- package/public/docs/tuning-reference/metrics.html +4 -4
- package/public/docs/tuning-reference/partitioning.html +4 -4
- package/public/docs/tuning-reference/pyspark.html +4 -4
- package/public/docs/tuning-reference/shuffle.html +4 -4
- package/public/docs/tuning-reference/spark-architecture.html +4 -4
- package/public/docs/tuning-reference/table-formats.html +4 -4
- package/public/docs/user-guide/alternative-log-retrieval.html +6 -6
- package/public/docs/user-guide/getting-started.html +7 -7
- package/public/docs/user-guide/mcp-tools.html +8 -8
- package/public/docs/user-guide/run-comparison.html +6 -6
- package/public/docs/user-guide/understanding-findings.html +6 -6
- package/public/index.html +7 -6
- package/public/sample-runs/sample-run.ndjson.gz +0 -0
- package/vendor-core/detectors.js +149 -24
- package/vendor-core/docs-content/detection/cache.md +4 -3
- package/vendor-core/docs-content/detection/chrn.md +4 -2
- package/vendor-core/docs-content/detection/local.md +2 -3
- package/vendor-core/docs-content/detection/mem.md +3 -3
- package/vendor-core/docs-content/detection/spec.md +4 -3
- package/vendor-core/evidence-report.js +1 -1
- package/vendor-core/parser-worker.js +2 -2
- package/vendor-core/run-comparison.js +21 -2
- package/public/assets/AutoscalingChurn-CFJkiqxW.js +0 -1
- package/public/assets/CacheUtilization-kTKZBzDA.js +0 -1
- package/public/assets/CachingOpportunity-CU70EWcF.js +0 -1
- package/public/assets/ColdStart-hOI-WOvm.js +0 -1
- package/public/assets/ConfigAudit-B4oAksZY.js +0 -1
- package/public/assets/CoreUsageArea-D-dR3SQk.js +0 -1
- package/public/assets/Dashboard-MDhohsq4.js +0 -4
- package/public/assets/DuplicatePlanSubtree-54RKsfI7.js +0 -1
- package/public/assets/DurationHistogram-CviQyPNM.js +0 -1
- package/public/assets/EmptyState-v2ehFncc.js +0 -1
- package/public/assets/ExecutorUtilization-vHyFHgL6.js +0 -1
- package/public/assets/GcPressure-DINuhcno.js +0 -1
- package/public/assets/ImpactEstimate-C6GzFOY9.js +0 -1
- package/public/assets/IncompleteRun-a1kuCnU6.js +0 -1
- package/public/assets/JobFailures-CYwPfUhR.js +0 -1
- package/public/assets/MemoryUtilization-Dlfj0e_f.js +0 -1
- package/public/assets/OverBroadcast-BMu7Y8l9.js +0 -1
- package/public/assets/PartitionSizing-CvMGfSVB.js +0 -1
- package/public/assets/PlanExplorer-xYFme_Rx.js +0 -1
- package/public/assets/PlanGraphRoute-a3OGg6ET.js +0 -1
- package/public/assets/RetryWaste-JKRjOZ-J.js +0 -1
- package/public/assets/RowPagination-DypWbnic.js +0 -1
- package/public/assets/ShuffleIO-Mi8HiVu-.js +0 -1
- package/public/assets/Skew-C3iOScTC.js +0 -1
- package/public/assets/SlowHost-DKBg18v-.js +0 -1
- package/public/assets/SmallFiles-CCCB6AIu.js +0 -1
- package/public/assets/SpeculationWaste-DUNupw0_.js +0 -1
- package/public/assets/Spill-BlLdatik.js +0 -1
- package/public/assets/StageFailed-qtY7C7Yo.js +0 -1
- package/public/assets/StageFindingGroup-CoVdEZYs.js +0 -1
- package/public/assets/StageHeader-uDBLHgdg.js +0 -1
- package/public/assets/StagePill-CYCMdlIx.js +0 -1
- package/public/assets/StageShape-zs0mAjQJ.js +0 -1
- package/public/assets/StageSlowness-C7vg0u5Q.js +0 -1
- package/public/assets/Straggler-4BRGAQP4.js +0 -1
- package/public/assets/TaskFailures-CSjO23Kg.js +0 -1
- package/public/assets/TinyTask-6Zuovcn5.js +0 -1
- package/public/assets/UnderBroadcast-CePjo0ZV.js +0 -1
- package/public/assets/WidgetLeadSummary-DRNRVvn7.js +0 -1
- package/public/assets/charts-vendor-Cf1-wzeH.js +0 -59
- package/public/assets/downsample-D8XmJk6i.js +0 -1
- package/public/assets/finding-anchor-4TmpXd7j.js +0 -1
- package/public/assets/impact-sort-Cq6KOLqh.js +0 -1
- package/public/assets/inbox-DBFovbUX.js +0 -1
- package/public/assets/index-CHjV2-jB.js +0 -9
- package/public/assets/index-DMEtaP4u.css +0 -1
- package/public/assets/parser-worker-BMPPE37w.js +0 -64
- package/public/assets/plan-graph-vendor-B96a-e4N.js +0 -7
- package/public/assets/plan-graph-vendor-BnuhLJ6X.css +0 -1
- package/public/assets/react-vendor-DA6opS-q.js +0 -9
- package/public/assets/select-D5fK-za0.js +0 -1
- package/public/assets/toggle-group-lbIXRG0W.js +0 -1
- package/public/assets/usePagedRows-CFoZrPr4.js +0 -1
- package/public/assets/useRecentFiles-BPGQ6XII.js +0 -13
- package/public/assets/useSortMode-BOPpfLhJ.js +0 -1
- package/public/docs/assets/chunks/@localSearchIndexroot.PQ5JepCp.js +0 -1
- package/public/docs/assets/chunks/theme.BP14rGBx.js +0 -2
- package/public/docs/assets/contributor-guide_architecture_board-widgets.md.B-OsL91z.js +0 -1
- package/public/docs/assets/contributor-guide_architecture_board-widgets.md.B-OsL91z.lean.js +0 -1
- package/public/docs/assets/contributor-guide_architecture_detector-contract.md.BOeH4d1J.js +0 -1
- package/public/docs/assets/contributor-guide_architecture_state-and-history.md.m3S3UdMk.js +0 -1
- package/public/docs/assets/contributor-guide_architecture_state-and-history.md.m3S3UdMk.lean.js +0 -1
- package/public/docs/assets/contributor-guide_architecture_widget-rendering.md.DbqPf2OT.js +0 -1
- package/public/docs/assets/contributor-guide_architecture_worker-protocol.md.B93qJ_tT.js +0 -6
- package/public/docs/assets/user-guide_getting-started.md.DtEM37MK.js +0 -3
- package/public/docs/assets/user-guide_getting-started.md.DtEM37MK.lean.js +0 -1
- package/public/docs/assets/user-guide_run-comparison.md.S0TWWmLY.js +0 -1
- package/public/docs/assets/user-guide_run-comparison.md.S0TWWmLY.lean.js +0 -1
- package/public/docs/assets/user-guide_understanding-findings.md.D0R_Y-R2.js +0 -1
- package/public/docs/assets/user-guide_understanding-findings.md.D0R_Y-R2.lean.js +0 -1
- /package/public/docs/assets/{contributor-guide_architecture_detector-contract.md.BOeH4d1J.lean.js → contributor-guide_architecture_detector-contract.md.CgzUsQ6W.lean.js} +0 -0
- /package/public/docs/assets/{contributor-guide_architecture_impact-estimation.md.DYCDPgkh.lean.js → contributor-guide_architecture_impact-estimation.md.CooslVJt.lean.js} +0 -0
- /package/public/docs/assets/{contributor-guide_architecture_widget-rendering.md.DbqPf2OT.lean.js → contributor-guide_architecture_widget-rendering.md.R27gQrgY.lean.js} +0 -0
- /package/public/docs/assets/{contributor-guide_architecture_worker-protocol.md.B93qJ_tT.lean.js → contributor-guide_architecture_worker-protocol.md.IbnfNrV3.lean.js} +0 -0
|
@@ -6,14 +6,14 @@
|
|
|
6
6
|
<title>Understanding findings | SparkForensics</title>
|
|
7
7
|
<meta name="description" content="Docs for using and contributing to SparkForensics">
|
|
8
8
|
<meta name="generator" content="VitePress v1.6.4">
|
|
9
|
-
<link rel="preload stylesheet" href="/docs/assets/style.
|
|
9
|
+
<link rel="preload stylesheet" href="/docs/assets/style.DSixAiZE.css" as="style">
|
|
10
10
|
<link rel="preload stylesheet" href="/docs/vp-icons.css" as="style">
|
|
11
11
|
|
|
12
|
-
<script type="module" src="/docs/assets/app.
|
|
12
|
+
<script type="module" src="/docs/assets/app.APoq_UBs.js"></script>
|
|
13
13
|
<link rel="preload" href="/docs/assets/inter-roman-latin.Di8DUHzh.woff2" as="font" type="font/woff2" crossorigin="">
|
|
14
|
-
<link rel="modulepreload" href="/docs/assets/chunks/theme.
|
|
14
|
+
<link rel="modulepreload" href="/docs/assets/chunks/theme.PavXO_vz.js">
|
|
15
15
|
<link rel="modulepreload" href="/docs/assets/chunks/framework.DSg0KOwT.js">
|
|
16
|
-
<link rel="modulepreload" href="/docs/assets/user-guide_understanding-findings.md.
|
|
16
|
+
<link rel="modulepreload" href="/docs/assets/user-guide_understanding-findings.md.DL1UDhvR.lean.js">
|
|
17
17
|
<link rel="icon" type="image/svg+xml" href="/docs/favicon.svg">
|
|
18
18
|
<link rel="preconnect" href="https://fonts.googleapis.com">
|
|
19
19
|
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin="">
|
|
@@ -22,8 +22,8 @@
|
|
|
22
22
|
<script id="check-mac-os">document.documentElement.classList.toggle("mac",/Mac|iPhone|iPod|iPad/i.test(navigator.platform));</script>
|
|
23
23
|
</head>
|
|
24
24
|
<body>
|
|
25
|
-
<div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="/docs/" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="/docs/favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/tuning-reference/" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>User Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/getting-started.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Getting started</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/understanding-findings.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Understanding findings</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/run-comparison.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Run comparison mode</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/mcp-tools.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>MCP tools reference</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/alternative-log-retrieval.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Alternative log retrieval</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_user-guide_understanding-findings" data-v-39a288b8><div><h1 id="understanding-findings" tabindex="-1">Understanding findings <a class="header-anchor" href="#understanding-findings" aria-label="Permalink to "Understanding findings""></a></h1><p>Every flagged problem carries a short ALL-CAPS tag. This page has one entry per tag: what it means, and what to do about it.</p><p>A few tags share their in-app "Reference panel" background reading with another tag, because the underlying Spark-tuning material overlaps: <code>SFAIL</code> with <code>FAIL</code>, <code>PART</code> with <code>SHFL</code>, <code>SPEC</code> with <code>STRAG</code>, and <code>CACHE</code>/<code>LOCAL</code> with <code>UTIL</code>.</p><h2 id="per-stage" tabindex="-1">Per-stage <a class="header-anchor" href="#per-stage" aria-label="Permalink to "Per-stage""></a></h2><h3 id="skew" tabindex="-1"><code>SKEW</code>: Task skew <a class="header-anchor" href="#skew" aria-label="Permalink to "`SKEW`: Task skew {#skew}""></a></h3><p>A small number of tasks take much longer than their peers in the same stage. For join-driven skew, enable AQE skew-join handling (<code>spark.sql.adaptive.skewJoin.enabled</code>); otherwise salt the key or repartition on a better key.</p><h3 id="shfl" tabindex="-1"><code>SHFL</code>: Shuffle I/O <a class="header-anchor" href="#shfl" aria-label="Permalink to "`SHFL`: Shuffle I/O {#shfl}""></a></h3><p>Tasks move a large amount of intermediate data between stages. Raise <code>spark.sql.shuffle.partitions</code>, or add a broadcast join.</p><h3 id="spill" tabindex="-1"><code>SPILL</code>: Memory and disk spill <a class="header-anchor" href="#spill" aria-label="Permalink to "`SPILL`: Memory and disk spill {#spill}""></a></h3><p>Tasks are writing data out of memory, which slows execution. Two spill patterns get flagged differently: skew spill, where a few heavy tasks spill while most don't (rebalance partitioning), and volume spill, where most tasks spill because the data genuinely exceeds available memory (add partitions).</p><h3 id="gc" tabindex="-1"><code>GC</code>: Garbage collection pressure <a class="header-anchor" href="#gc" aria-label="Permalink to "`GC`: Garbage collection pressure {#gc}""></a></h3><p>Tasks spend an unusually large share of time reclaiming memory. Reduce object creation: use primitive types, avoid UDFs, or raise executor memory.</p><h3 id="fail" tabindex="-1"><code>FAIL</code>: Failed tasks <a class="header-anchor" href="#fail" aria-label="Permalink to "`FAIL`: Failed tasks {#fail}""></a></h3><p>Tasks fail often enough to affect the stage. Failed tasks point to executor instability or data-driven errors: check driver logs for the dominant failure reason.</p><h3 id="sfail" tabindex="-1"><code>SFAIL</code>: Failed stage <a class="header-anchor" href="#sfail" aria-label="Permalink to "`SFAIL`: Failed stage {#sfail}""></a></h3><p>A stage attempt failed outright rather than losing individual tasks within it. Inspect the driver log for the failure reason and the job that triggered it.</p><h3 id="strag" tabindex="-1"><code>STRAG</code>: Straggler tasks <a class="header-anchor" href="#strag" aria-label="Permalink to "`STRAG`: Straggler tasks {#strag}""></a></h3><p>A few tasks run much slower than the rest of their stage. Rule out a GC pause or a slow shuffle fetch before assuming a hardware issue; if a skewed key is the real cause, that's a candidate for AQE's skew-join handling.</p><h3 id="spec" tabindex="-1"><code>SPEC</code>: Speculation waste <a class="header-anchor" href="#spec" aria-label="Permalink to "`SPEC`: Speculation waste {#spec}""></a></h3><p>Speculative task attempts used a lot of executor time without confirming a genuine straggler. Self-flagged low-confidence: a design spike, not yet validated against real-world runs. If task durations are just naturally variable rather than genuine stragglers, tune <code>spark.speculation.multiplier</code>/<code>spark.speculation.quantile</code>.</p><h3 id="retry" tabindex="-1"><code>RETRY</code>: Retry waste <a class="header-anchor" href="#retry" aria-label="Permalink to "`RETRY`: Retry waste {#retry}""></a></h3><p>Repeated task attempts ate into execution time even though the stage completed. Investigate executor loss or fetch failures.</p><h3 id="tiny" tabindex="-1"><code>TINY</code>: Tiny tasks <a class="header-anchor" href="#tiny" aria-label="Permalink to "`TINY`: Tiny tasks {#tiny}""></a></h3><p>Many very short tasks add scheduling overhead out of proportion to the work each one does. Repartition to fewer, larger tasks.</p><h3 id="part" tabindex="-1"><code>PART</code>: Partition sizing <a class="header-anchor" href="#part" aria-label="Permalink to "`PART`: Partition sizing {#part}""></a></h3><p>Shuffle partitions are too large, too uneven, or too few for the work. A single shuffle partition over 5 GB, for example, will OOM or spill heavily: repartition to break it up before the stage runs.</p><h3 id="slow" tabindex="-1"><code>SLOW</code>: Stage slowness <a class="header-anchor" href="#slow" aria-label="Permalink to "`SLOW`: Stage slowness {#slow}""></a></h3><p>A stage ran long overall without a more specific cause getting flagged. Often a partition-count problem: raise parallelism via <code>spark.sql.shuffle.partitions</code> or <code>spark.default.parallelism</code>, or check for a large per-task data volume driving heavy shuffle and spill.</p><h3 id="shape" tabindex="-1"><code>SHAPE</code>: Stage shape <a class="header-anchor" href="#shape" aria-label="Permalink to "`SHAPE`: Stage shape {#shape}""></a></h3><p>The stage has an inefficient task count, output shape, or task-to-stage balance: for example, one straggler task taking a large fraction of the stage's wall-clock time.</p><h3 id="host" tabindex="-1"><code>HOST</code>: Slow host <a class="header-anchor" href="#host" aria-label="Permalink to "`HOST`: Slow host {#host}""></a></h3><p>One executor is much slower than its peers. It may just hold data locality for its tasks or carry one heavy stage, rather than a hardware fault. Enable <code>spark.speculation</code> to relaunch a lagging task automatically.</p><h2 id="app-level" tabindex="-1">App-level <a class="header-anchor" href="#app-level" aria-label="Permalink to "App-level""></a></h2><h3 id="cold" tabindex="-1"><code>COLD</code>: Executor cold start <a class="header-anchor" href="#cold" aria-label="Permalink to "`COLD`: Executor cold start {#cold}""></a></h3><p>New executors take time to become available for work. Pre-warm the cluster, or use dynamic allocation.</p><h3 id="util" tabindex="-1"><code>UTIL</code>: Low utilization <a class="header-anchor" href="#util" aria-label="Permalink to "`UTIL`: Low utilization {#util}""></a></h3><p>Allocated executors sit idle for a large share of the application run. Consider a smaller cluster, or enable dynamic allocation.</p><h3 id="mem" tabindex="-1"><code>MEM</code>: Memory utilization <a class="header-anchor" href="#mem" aria-label="Permalink to "`MEM`: Memory utilization {#mem}""></a></h3><p>Executor memory or core capacity may be over- or under-provisioned. Some detail here needs <code>spark.eventLog.logStageExecutorMetrics=true</code> on the run being analyzed; without it, per-executor memory usage can't be broken down. Review <code>spark.executor.memory</code> and executor count if allocated memory sat largely idle over the run. That idle-memory variant is self-flagged low-confidence: it estimates waste from allocated-versus-used memory-time against an unverified 1.5x buffer. Check it against the Spark UI before resizing anything.</p><h3 id="cache" tabindex="-1"><code>CACHE</code>: Caching opportunity <a class="header-anchor" href="#cache" aria-label="Permalink to "`CACHE`: Caching opportunity {#cache}""></a></h3><p>A reusable dataset (re-read via the same SQL relation more than once) may be worth persisting between stages. Self-flagged low-confidence: reuse is only inferred, from plan-scan identity across SQL executions, so confirm the reads really do hit the same data before you cache anything.</p><h3 id="cstor" tabindex="-1"><code>CSTOR</code>: Cache storage <a class="header-anchor" href="#cstor" aria-label="Permalink to "`CSTOR`: Cache storage {#cstor}""></a></h3><p>A persisted dataset is not fully cached in memory, or is spilling to disk. Raise executor memory, or shrink the cached dataset.</p><h3 id="local" tabindex="-1"><code>LOCAL</code>: Core usage locality <a class="header-anchor" href="#local" aria-label="Permalink to "`LOCAL`: Core usage locality {#local}""></a></h3><p>Tasks run without process- or node-local data placement more often than expected. Check <code>spark.locality.wait</code> settings and executor/data colocation. Self-flagged low-confidence: the non-local-ratio thresholds are unvalidated design-spike values, and no external tool publishes an equivalent metric to calibrate them against.</p><h3 id="chrn" tabindex="-1"><code>CHRN</code>: Autoscaling churn <a class="header-anchor" href="#chrn" aria-label="Permalink to "`CHRN`: Autoscaling churn {#chrn}""></a></h3><p>Executors are stood up and torn down again before they can do useful work: re-provisioning churn rather than normal scale-down. Raise <code>spark.dynamicAllocation.executorIdleTimeout</code>, or widen the <code>minExecutors</code>/<code>maxExecutors</code> bounds to reduce flapping. Self-flagged low-confidence: a design spike, not yet validated against real-world runs.</p><h3 id="jobs" tabindex="-1"><code>JOBS</code>: Job failure rate <a class="header-anchor" href="#jobs" aria-label="Permalink to "`JOBS`: Job failure rate {#jobs}""></a></h3><p>A large share of completed jobs did not succeed. Inspect the driver log for the failed job(s) and the stage failures that triggered them.</p><h3 id="incmp" tabindex="-1"><code>INCMP</code>: Incomplete run <a class="header-anchor" href="#incmp" aria-label="Permalink to "`INCMP`: Incomplete run {#incmp}""></a></h3><p>This event log never recorded an <code>ApplicationEnd</code> event: capture stopped before the run finished (an in-flight job, a rotated-away log, or a cut-short capture). Every other finding and metric on the board reflects only what was captured up to that point, not the full run.</p><h2 id="configuration-scope" tabindex="-1">Configuration scope <a class="header-anchor" href="#configuration-scope" aria-label="Permalink to "Configuration scope""></a></h2><h3 id="cfg" tabindex="-1"><code>CFG</code>: Configuration audit <a class="header-anchor" href="#cfg" aria-label="Permalink to "`CFG`: Configuration audit {#cfg}""></a></h3><p>Flags configuration settings that may cause reliability or efficiency problems, independent of any one stage's behavior. Four properties are audited today:</p><ul><li><code>spark.shuffle.service.enabled</code>: flagged when dynamic allocation is on but the external shuffle service is off, since shuffle data won't survive executor removal.</li><li><code>spark.dynamicAllocation.maxExecutors</code>: flagged for inverted bounds or a missing upper bound.</li><li><code>spark.serializer</code>: flagged when still on the default Java serializer; <code>org.apache.spark.serializer.KryoSerializer</code> is faster and produces smaller buffers.</li><li><code>spark.executor.memoryOverhead</code>: flagged when set below a safe floor.</li></ul><h2 id="sql-scope" tabindex="-1">SQL scope <a class="header-anchor" href="#sql-scope" aria-label="Permalink to "SQL scope""></a></h2><h3 id="plan" tabindex="-1"><code>PLAN</code>: Plan advisor <a class="header-anchor" href="#plan" aria-label="Permalink to "`PLAN`: Plan advisor {#plan}""></a></h3><p>Flags patterns in the SQL execution plan worth reviewing. Four checks share this tag:</p><ul><li>Duplicate plan subtree: the same subtree recomputed more than once in the plan.</li><li>Small files: reading an excessive number of small files.</li><li>Under-broadcast: the smaller side of a Sort Merge Join looks well under the broadcast threshold; consider a <code>broadcast()</code> hint or raising <code>spark.sql.autoBroadcastJoinThreshold</code>.</li><li>Over-broadcast: a broadcast exceeds the 1 GB threshold; check for a misapplied broadcast hint or a misconfigured <code>spark.sql.autoBroadcastJoinThreshold</code>.</li></ul></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><a class="VPLink link pager-link prev" href="/docs/user-guide/getting-started.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Previous page</span><span class="title" data-v-e257564d>Getting started</span><!--]--></a></div><div class="pager" data-v-e257564d><a class="VPLink link pager-link next" href="/docs/user-guide/run-comparison.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Next page</span><span class="title" data-v-e257564d>Run comparison mode</span><!--]--></a></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
|
|
26
|
-
<script>window.__VP_HASH_MAP__=JSON.parse("{\"contributor-guide_architecture_board-widgets.md\":\"
|
|
25
|
+
<div id="app"><div class="Layout" data-v-5d98c3a5><!--[--><!--]--><!--[--><span tabindex="-1" data-v-0b0ada53></span><a href="#VPContent" class="VPSkipLink visually-hidden" data-v-0b0ada53>Skip to content</a><!--]--><!----><header class="VPNav" data-v-5d98c3a5 data-v-ae24b3ad><div class="VPNavBar" data-v-ae24b3ad data-v-6aa21345><div class="wrapper" data-v-6aa21345><div class="container" data-v-6aa21345><div class="title" data-v-6aa21345><div class="VPNavBarTitle has-sidebar" data-v-6aa21345 data-v-1168a8e4><a class="title" href="/docs/" data-v-1168a8e4><!--[--><!--]--><!--[--><img class="VPImage logo" src="/docs/favicon.svg" alt data-v-8426fc1a><!--]--><span data-v-1168a8e4>SparkForensics</span><!--[--><!--]--></a></div></div><div class="content" data-v-6aa21345><div class="content-body" data-v-6aa21345><!--[--><!--]--><div class="VPNavBarSearch search" data-v-6aa21345><!--[--><!----><div id="local-search"><button type="button" class="DocSearch DocSearch-Button" aria-label="Search"><span class="DocSearch-Button-Container"><span class="vp-icon DocSearch-Search-Icon"></span><span class="DocSearch-Button-Placeholder">Search</span></span><span class="DocSearch-Button-Keys"><kbd class="DocSearch-Button-Key"></kbd><kbd class="DocSearch-Button-Key">K</kbd></span></button></div><!--]--></div><nav aria-labelledby="main-nav-aria-label" class="VPNavBarMenu menu" data-v-6aa21345 data-v-dc692963><span id="main-nav-aria-label" class="visually-hidden" data-v-dc692963> Main Navigation </span><!--[--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/user-guide/getting-started.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>User Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/contributor-guide/development-setup.html" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Contributor Guide</span><!--]--></a><!--]--><!--[--><a class="VPLink link VPNavBarMenuLink" href="/docs/tuning-reference/" tabindex="0" data-v-dc692963 data-v-e56f3d57><!--[--><span data-v-e56f3d57>Tuning Reference</span><!--]--></a><!--]--><!--]--></nav><!----><div class="VPNavBarAppearance appearance" data-v-6aa21345 data-v-6c893767><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-6c893767 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div><!----><div class="VPFlyout VPNavBarExtra extra" data-v-6aa21345 data-v-bb2aa2f0 data-v-cf11d7a2><button type="button" class="button" aria-haspopup="true" aria-expanded="false" aria-label="extra navigation" data-v-cf11d7a2><span class="vpi-more-horizontal icon" data-v-cf11d7a2></span></button><div class="menu" data-v-cf11d7a2><div class="VPMenu" data-v-cf11d7a2 data-v-b98bc113><!----><!--[--><!--[--><!----><div class="group" data-v-bb2aa2f0><div class="item appearance" data-v-bb2aa2f0><p class="label" data-v-bb2aa2f0>Appearance</p><div class="appearance-action" data-v-bb2aa2f0><button class="VPSwitch VPSwitchAppearance" type="button" role="switch" title aria-checked="false" data-v-bb2aa2f0 data-v-5337faa4 data-v-1d5665e3><span class="check" data-v-1d5665e3><span class="icon" data-v-1d5665e3><!--[--><span class="vpi-sun sun" data-v-5337faa4></span><span class="vpi-moon moon" data-v-5337faa4></span><!--]--></span></span></button></div></div></div><!----><!--]--><!--]--></div></div></div><!--[--><!--]--><button type="button" class="VPNavBarHamburger hamburger" aria-label="mobile navigation" aria-expanded="false" aria-controls="VPNavScreen" data-v-6aa21345 data-v-e5dd9c1c><span class="container" data-v-e5dd9c1c><span class="top" data-v-e5dd9c1c></span><span class="middle" data-v-e5dd9c1c></span><span class="bottom" data-v-e5dd9c1c></span></span></button></div></div></div></div><div class="divider" data-v-6aa21345><div class="divider-line" data-v-6aa21345></div></div></div><!----></header><div class="VPLocalNav has-sidebar empty" data-v-5d98c3a5 data-v-a6f0e41e><div class="container" data-v-a6f0e41e><button class="menu" aria-expanded="false" aria-controls="VPSidebarNav" data-v-a6f0e41e><span class="vpi-align-left menu-icon" data-v-a6f0e41e></span><span class="menu-text" data-v-a6f0e41e>Menu</span></button><div class="VPLocalNavOutlineDropdown" style="--vp-vh:0px;" data-v-a6f0e41e data-v-8a42e2b4><button data-v-8a42e2b4>Return to top</button><!----></div></div></div><aside class="VPSidebar" data-v-5d98c3a5 data-v-319d5ca6><div class="curtain" data-v-319d5ca6></div><nav class="nav" id="VPSidebarNav" aria-labelledby="sidebar-aria-label" tabindex="-1" data-v-319d5ca6><span class="visually-hidden" id="sidebar-aria-label" data-v-319d5ca6> Sidebar Navigation </span><!--[--><!--]--><!--[--><div class="no-transition group" data-v-c40bc020><section class="VPSidebarItem level-0 has-active" data-v-c40bc020 data-v-b3fd67f8><div class="item" role="button" tabindex="0" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><h2 class="text" data-v-b3fd67f8>User Guide</h2><!----></div><div class="items" data-v-b3fd67f8><!--[--><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/getting-started.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Getting started</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/understanding-findings.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Understanding findings</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/run-comparison.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Run comparison mode</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/mcp-tools.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>MCP tools reference</p><!--]--></a><!----></div><!----></div><div class="VPSidebarItem level-1 is-link" data-v-b3fd67f8><div class="item" data-v-b3fd67f8><div class="indicator" data-v-b3fd67f8></div><a class="VPLink link link" href="/docs/user-guide/alternative-log-retrieval.html" data-v-b3fd67f8><!--[--><p class="text" data-v-b3fd67f8>Alternative log retrieval</p><!--]--></a><!----></div><!----></div><!--]--></div></section></div><!--]--><!--[--><!--]--></nav></aside><div class="VPContent has-sidebar" id="VPContent" data-v-5d98c3a5 data-v-1428d186><div class="VPDoc has-sidebar has-aside" data-v-1428d186 data-v-39a288b8><!--[--><!--]--><div class="container" data-v-39a288b8><div class="aside" data-v-39a288b8><div class="aside-curtain" data-v-39a288b8></div><div class="aside-container" data-v-39a288b8><div class="aside-content" data-v-39a288b8><div class="VPDocAside" data-v-39a288b8 data-v-3f215769><!--[--><!--]--><!--[--><!--]--><nav aria-labelledby="doc-outline-aria-label" class="VPDocAsideOutline" data-v-3f215769 data-v-a5bbad30><div class="content" data-v-a5bbad30><div class="outline-marker" data-v-a5bbad30></div><div aria-level="2" class="outline-title" id="doc-outline-aria-label" role="heading" data-v-a5bbad30>On this page</div><ul class="VPDocOutlineItem root" data-v-a5bbad30 data-v-b933a997><!--[--><!--]--></ul></div></nav><!--[--><!--]--><div class="spacer" data-v-3f215769></div><!--[--><!--]--><!----><!--[--><!--]--><!--[--><!--]--></div></div></div></div><div class="content" data-v-39a288b8><div class="content-container" data-v-39a288b8><!--[--><!--]--><main class="main" data-v-39a288b8><div style="position:relative;" class="vp-doc _docs_user-guide_understanding-findings" data-v-39a288b8><div><h1 id="understanding-findings" tabindex="-1">Understanding findings <a class="header-anchor" href="#understanding-findings" aria-label="Permalink to "Understanding findings""></a></h1><p>Every flagged problem carries a short ALL-CAPS tag. This page has one entry per tag: what it means, and what to do about it.</p><p>Turn on <a href="./getting-started.html#advanced-view">Advanced view</a> in the dashboard to see, per finding, the confidence level and a link to the same background reading collected here, opened in an in-app "Reference" panel next to the board. A few tags share that reading with another tag, because the underlying Spark-tuning material overlaps: <code>SFAIL</code> with <code>FAIL</code>, <code>PART</code> with <code>SHFL</code>, <code>SPEC</code> with <code>STRAG</code>, and <code>CACHE</code>/<code>LOCAL</code> with <code>UTIL</code>.</p><h2 id="per-stage" tabindex="-1">Per-stage <a class="header-anchor" href="#per-stage" aria-label="Permalink to "Per-stage""></a></h2><h3 id="skew" tabindex="-1"><code>SKEW</code>: Task skew <a class="header-anchor" href="#skew" aria-label="Permalink to "`SKEW`: Task skew {#skew}""></a></h3><p>A small number of tasks take much longer than their peers in the same stage. For join-driven skew, enable AQE skew-join handling (<code>spark.sql.adaptive.skewJoin.enabled</code>); otherwise salt the key or repartition on a better key.</p><h3 id="shfl" tabindex="-1"><code>SHFL</code>: Shuffle I/O <a class="header-anchor" href="#shfl" aria-label="Permalink to "`SHFL`: Shuffle I/O {#shfl}""></a></h3><p>Tasks move a large amount of intermediate data between stages. Raise <code>spark.sql.shuffle.partitions</code>, or add a broadcast join.</p><h3 id="spill" tabindex="-1"><code>SPILL</code>: Memory and disk spill <a class="header-anchor" href="#spill" aria-label="Permalink to "`SPILL`: Memory and disk spill {#spill}""></a></h3><p>Tasks are writing data out of memory, which slows execution. Two spill patterns get flagged differently: skew spill, where a few heavy tasks spill while most don't (rebalance partitioning), and volume spill, where most tasks spill because the data genuinely exceeds available memory (add partitions).</p><h3 id="gc" tabindex="-1"><code>GC</code>: Garbage collection pressure <a class="header-anchor" href="#gc" aria-label="Permalink to "`GC`: Garbage collection pressure {#gc}""></a></h3><p>Tasks spend an unusually large share of time reclaiming memory. Reduce object creation: use primitive types, avoid UDFs, or raise executor memory.</p><h3 id="fail" tabindex="-1"><code>FAIL</code>: Failed tasks <a class="header-anchor" href="#fail" aria-label="Permalink to "`FAIL`: Failed tasks {#fail}""></a></h3><p>Tasks fail often enough to affect the stage. Failed tasks point to executor instability or data-driven errors: check driver logs for the dominant failure reason.</p><h3 id="sfail" tabindex="-1"><code>SFAIL</code>: Failed stage <a class="header-anchor" href="#sfail" aria-label="Permalink to "`SFAIL`: Failed stage {#sfail}""></a></h3><p>A stage attempt failed outright rather than losing individual tasks within it. Inspect the driver log for the failure reason and the job that triggered it.</p><h3 id="strag" tabindex="-1"><code>STRAG</code>: Straggler tasks <a class="header-anchor" href="#strag" aria-label="Permalink to "`STRAG`: Straggler tasks {#strag}""></a></h3><p>A few tasks run much slower than the rest of their stage. Rule out a GC pause or a slow shuffle fetch before assuming a hardware issue; if a skewed key is the real cause, that's a candidate for AQE's skew-join handling.</p><h3 id="spec" tabindex="-1"><code>SPEC</code>: Speculation waste <a class="header-anchor" href="#spec" aria-label="Permalink to "`SPEC`: Speculation waste {#spec}""></a></h3><p>Speculative task attempts used a lot of executor time without confirming a genuine straggler. Self-flags a confidence that scales with how far the wasted time sits past the threshold: these thresholds are still a design spike, not yet validated against real-world runs. If task durations are just naturally variable rather than genuine stragglers, tune <code>spark.speculation.multiplier</code>/<code>spark.speculation.quantile</code>.</p><h3 id="retry" tabindex="-1"><code>RETRY</code>: Retry waste <a class="header-anchor" href="#retry" aria-label="Permalink to "`RETRY`: Retry waste {#retry}""></a></h3><p>Repeated task attempts ate into execution time even though the stage completed. Investigate executor loss or fetch failures.</p><h3 id="tiny" tabindex="-1"><code>TINY</code>: Tiny tasks <a class="header-anchor" href="#tiny" aria-label="Permalink to "`TINY`: Tiny tasks {#tiny}""></a></h3><p>Many very short tasks add scheduling overhead out of proportion to the work each one does. Repartition to fewer, larger tasks.</p><h3 id="part" tabindex="-1"><code>PART</code>: Partition sizing <a class="header-anchor" href="#part" aria-label="Permalink to "`PART`: Partition sizing {#part}""></a></h3><p>Shuffle partitions are too large, too uneven, or too few for the work. A single shuffle partition over 5 GB, for example, will OOM or spill heavily: repartition to break it up before the stage runs.</p><h3 id="slow" tabindex="-1"><code>SLOW</code>: Stage slowness <a class="header-anchor" href="#slow" aria-label="Permalink to "`SLOW`: Stage slowness {#slow}""></a></h3><p>A stage ran long overall without a more specific cause getting flagged. Often a partition-count problem: raise parallelism via <code>spark.sql.shuffle.partitions</code> or <code>spark.default.parallelism</code>, or check for a large per-task data volume driving heavy shuffle and spill.</p><h3 id="shape" tabindex="-1"><code>SHAPE</code>: Stage shape <a class="header-anchor" href="#shape" aria-label="Permalink to "`SHAPE`: Stage shape {#shape}""></a></h3><p>The stage has an inefficient task count, output shape, or task-to-stage balance: for example, one straggler task taking a large fraction of the stage's wall-clock time.</p><h3 id="host" tabindex="-1"><code>HOST</code>: Slow host <a class="header-anchor" href="#host" aria-label="Permalink to "`HOST`: Slow host {#host}""></a></h3><p>One executor is much slower than its peers. It may just hold data locality for its tasks or carry one heavy stage, rather than a hardware fault. Enable <code>spark.speculation</code> to relaunch a lagging task automatically.</p><h2 id="app-level" tabindex="-1">App-level <a class="header-anchor" href="#app-level" aria-label="Permalink to "App-level""></a></h2><h3 id="cold" tabindex="-1"><code>COLD</code>: Executor cold start <a class="header-anchor" href="#cold" aria-label="Permalink to "`COLD`: Executor cold start {#cold}""></a></h3><p>New executors take time to become available for work. Pre-warm the cluster, or use dynamic allocation.</p><h3 id="util" tabindex="-1"><code>UTIL</code>: Low utilization <a class="header-anchor" href="#util" aria-label="Permalink to "`UTIL`: Low utilization {#util}""></a></h3><p>Allocated executors sit idle for a large share of the application run. Consider a smaller cluster, or enable dynamic allocation.</p><h3 id="mem" tabindex="-1"><code>MEM</code>: Memory utilization <a class="header-anchor" href="#mem" aria-label="Permalink to "`MEM`: Memory utilization {#mem}""></a></h3><p>Executor memory or core capacity may be over- or under-provisioned. Some detail here needs <code>spark.eventLog.logStageExecutorMetrics=true</code> on the run being analyzed; without it, per-executor memory usage can't be broken down. Review <code>spark.executor.memory</code> and executor count if allocated memory sat largely idle over the run. That idle-memory variant self-flags a confidence that scales with how far the estimated waste sits past a 1.5x buffer: it estimates waste from allocated-versus-used memory-time. Check it against the Spark UI before resizing anything.</p><h3 id="cache" tabindex="-1"><code>CACHE</code>: Caching opportunity <a class="header-anchor" href="#cache" aria-label="Permalink to "`CACHE`: Caching opportunity {#cache}""></a></h3><p>A reusable dataset (re-read via the same SQL relation more than once) may be worth persisting between stages. Self-flags a confidence that scales with how many executions reuse the same relation: reuse is only inferred, from plan-scan identity across SQL executions, so confirm the reads really do hit the same data before you cache anything.</p><h3 id="cstor" tabindex="-1"><code>CSTOR</code>: Cache storage <a class="header-anchor" href="#cstor" aria-label="Permalink to "`CSTOR`: Cache storage {#cstor}""></a></h3><p>A persisted dataset is not fully cached in memory, or is spilling to disk. Raise executor memory, or shrink the cached dataset.</p><h3 id="local" tabindex="-1"><code>LOCAL</code>: Core usage locality <a class="header-anchor" href="#local" aria-label="Permalink to "`LOCAL`: Core usage locality {#local}""></a></h3><p>Tasks run without process- or node-local data placement more often than expected. Check <code>spark.locality.wait</code> settings and executor/data colocation. Self-flags a confidence that scales with the non-local ratio and sample size: the thresholds are our own noise floor for this metric.</p><h3 id="chrn" tabindex="-1"><code>CHRN</code>: Autoscaling churn <a class="header-anchor" href="#chrn" aria-label="Permalink to "`CHRN`: Autoscaling churn {#chrn}""></a></h3><p>Executors are stood up and torn down again before they can do useful work: re-provisioning churn rather than normal scale-down. Raise <code>spark.dynamicAllocation.executorIdleTimeout</code>, or widen the <code>minExecutors</code>/<code>maxExecutors</code> bounds to reduce flapping. Self-flags a confidence that scales with how far the short-lived-executor share sits past the threshold: these thresholds are still a design spike, not yet validated against real-world runs.</p><h3 id="jobs" tabindex="-1"><code>JOBS</code>: Job failure rate <a class="header-anchor" href="#jobs" aria-label="Permalink to "`JOBS`: Job failure rate {#jobs}""></a></h3><p>A large share of completed jobs did not succeed. Inspect the driver log for the failed job(s) and the stage failures that triggered them.</p><h3 id="incmp" tabindex="-1"><code>INCMP</code>: Incomplete run <a class="header-anchor" href="#incmp" aria-label="Permalink to "`INCMP`: Incomplete run {#incmp}""></a></h3><p>This event log never recorded an <code>ApplicationEnd</code> event: capture stopped before the run finished (an in-flight job, a rotated-away log, or a cut-short capture). Every other finding and metric on the board reflects only what was captured up to that point, not the full run.</p><h2 id="configuration-scope" tabindex="-1">Configuration scope <a class="header-anchor" href="#configuration-scope" aria-label="Permalink to "Configuration scope""></a></h2><h3 id="cfg" tabindex="-1"><code>CFG</code>: Configuration audit <a class="header-anchor" href="#cfg" aria-label="Permalink to "`CFG`: Configuration audit {#cfg}""></a></h3><p>Flags configuration settings that may cause reliability or efficiency problems, independent of any one stage's behavior. Four properties are audited today:</p><ul><li><code>spark.shuffle.service.enabled</code>: flagged when dynamic allocation is on but the external shuffle service is off, since shuffle data won't survive executor removal.</li><li><code>spark.dynamicAllocation.maxExecutors</code>: flagged for inverted bounds or a missing upper bound.</li><li><code>spark.serializer</code>: flagged when still on the default Java serializer; <code>org.apache.spark.serializer.KryoSerializer</code> is faster and produces smaller buffers.</li><li><code>spark.executor.memoryOverhead</code>: flagged when set below a safe floor.</li></ul><h2 id="sql-scope" tabindex="-1">SQL scope <a class="header-anchor" href="#sql-scope" aria-label="Permalink to "SQL scope""></a></h2><h3 id="plan" tabindex="-1"><code>PLAN</code>: Plan advisor <a class="header-anchor" href="#plan" aria-label="Permalink to "`PLAN`: Plan advisor {#plan}""></a></h3><p>Flags patterns in the SQL execution plan worth reviewing. Four checks share this tag:</p><ul><li>Duplicate plan subtree: the same subtree recomputed more than once in the plan.</li><li>Small files: reading an excessive number of small files.</li><li>Under-broadcast: the smaller side of a Sort Merge Join looks well under the broadcast threshold; consider a <code>broadcast()</code> hint or raising <code>spark.sql.autoBroadcastJoinThreshold</code>.</li><li>Over-broadcast: a broadcast exceeds the 1 GB threshold; check for a misapplied broadcast hint or a misconfigured <code>spark.sql.autoBroadcastJoinThreshold</code>.</li></ul></div></div></main><footer class="VPDocFooter" data-v-39a288b8 data-v-e257564d><!--[--><!--]--><!----><nav class="prev-next" aria-labelledby="doc-footer-aria-label" data-v-e257564d><span class="visually-hidden" id="doc-footer-aria-label" data-v-e257564d>Pager</span><div class="pager" data-v-e257564d><a class="VPLink link pager-link prev" href="/docs/user-guide/getting-started.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Previous page</span><span class="title" data-v-e257564d>Getting started</span><!--]--></a></div><div class="pager" data-v-e257564d><a class="VPLink link pager-link next" href="/docs/user-guide/run-comparison.html" data-v-e257564d><!--[--><span class="desc" data-v-e257564d>Next page</span><span class="title" data-v-e257564d>Run comparison mode</span><!--]--></a></div></nav></footer><!--[--><!--]--></div></div></div><!--[--><!--]--></div></div><!----><!--[--><!--]--></div></div>
|
|
26
|
+
<script>window.__VP_HASH_MAP__=JSON.parse("{\"contributor-guide_architecture_board-widgets.md\":\"CWpj01WU\",\"contributor-guide_architecture_detector-contract.md\":\"CgzUsQ6W\",\"contributor-guide_architecture_drill-down.md\":\"BtPdlM7r\",\"contributor-guide_architecture_impact-estimation.md\":\"CooslVJt\",\"contributor-guide_architecture_index.md\":\"3TO9ic6w\",\"contributor-guide_architecture_overview.md\":\"CehiRmGn\",\"contributor-guide_architecture_state-and-history.md\":\"C-xxn0q7\",\"contributor-guide_architecture_widget-rendering.md\":\"R27gQrgY\",\"contributor-guide_architecture_worker-protocol.md\":\"IbnfNrV3\",\"contributor-guide_contributing.md\":\"CvRsdr6J\",\"contributor-guide_development-setup.md\":\"DvAN_9mK\",\"contributor-guide_testing.md\":\"6rIKqSyY\",\"index.md\":\"CHJVslga\",\"tuning-reference_anti-patterns.md\":\"Df1YMIHu\",\"tuning-reference_aqe.md\":\"BIsCtLzm\",\"tuning-reference_bottleneck-broadcast-sizing.md\":\"CEstB3Ia\",\"tuning-reference_bottleneck-cold-start.md\":\"CEuy-72y\",\"tuning-reference_bottleneck-duplicate-plan-subtree.md\":\"CIohQDfn\",\"tuning-reference_bottleneck-failures.md\":\"4z5BXGJ2\",\"tuning-reference_bottleneck-gc.md\":\"DSxzZRK7\",\"tuning-reference_bottleneck-job-failure-rate.md\":\"BaJl__1W\",\"tuning-reference_bottleneck-memory-utilization.md\":\"DbP-SJZc\",\"tuning-reference_bottleneck-retry-waste.md\":\"D5JMjVOt\",\"tuning-reference_bottleneck-shuffle.md\":\"CM-nTmIH\",\"tuning-reference_bottleneck-skew.md\":\"BdUwiDhn\",\"tuning-reference_bottleneck-slow-host.md\":\"BlIo6UDW\",\"tuning-reference_bottleneck-small-files.md\":\"B8kloyx8\",\"tuning-reference_bottleneck-spill.md\":\"PNH7mITt\",\"tuning-reference_bottleneck-straggler.md\":\"DY36fHN5\",\"tuning-reference_bottleneck-tiny-tasks.md\":\"QTV7O8kU\",\"tuning-reference_bottleneck-utilization.md\":\"DTiueZC3\",\"tuning-reference_caching.md\":\"B7aQ8asB\",\"tuning-reference_cluster-config.md\":\"ZVmDGsQ3\",\"tuning-reference_config.md\":\"UvveiWG3\",\"tuning-reference_data-formats.md\":\"bjCAWH3N\",\"tuning-reference_index.md\":\"BQ_NooMV\",\"tuning-reference_intro.md\":\"CobD-lGB\",\"tuning-reference_joins.md\":\"BtKs_CuW\",\"tuning-reference_memory-model.md\":\"DhT-n4y3\",\"tuning-reference_metrics.md\":\"mLOh7Apj\",\"tuning-reference_partitioning.md\":\"q0zKF_8X\",\"tuning-reference_pyspark.md\":\"DDCfvN9t\",\"tuning-reference_shuffle.md\":\"BZZ7R4Ix\",\"tuning-reference_spark-architecture.md\":\"Dwzm5avO\",\"tuning-reference_table-formats.md\":\"D6wj-2dX\",\"user-guide_alternative-log-retrieval.md\":\"B4tPGIal\",\"user-guide_getting-started.md\":\"BJvwLEIM\",\"user-guide_mcp-tools.md\":\"Vi3RoflJ\",\"user-guide_run-comparison.md\":\"CQc1aoU8\",\"user-guide_understanding-findings.md\":\"DL1UDhvR\"}");window.__VP_SITE_DATA__=JSON.parse("{\"lang\":\"en-US\",\"dir\":\"ltr\",\"title\":\"SparkForensics\",\"description\":\"Docs for using and contributing to SparkForensics\",\"base\":\"/docs/\",\"head\":[],\"router\":{\"prefetchLinks\":true},\"appearance\":true,\"themeConfig\":{\"logo\":\"/favicon.svg\",\"nav\":[{\"text\":\"User Guide\",\"link\":\"/user-guide/getting-started\"},{\"text\":\"Contributor Guide\",\"link\":\"/contributor-guide/development-setup\"},{\"text\":\"Tuning Reference\",\"link\":\"/tuning-reference/\"}],\"sidebar\":{\"/tuning-reference/\":[{\"text\":\"Getting Started\",\"items\":[{\"text\":\"Introduction\",\"link\":\"/tuning-reference/intro\"}]},{\"text\":\"Optimization Guide\",\"items\":[{\"text\":\"Spark Execution Model\",\"link\":\"/tuning-reference/spark-architecture\"},{\"text\":\"Memory Management\",\"link\":\"/tuning-reference/memory-model\"},{\"text\":\"Partitioning\",\"link\":\"/tuning-reference/partitioning\"},{\"text\":\"Join Optimization\",\"link\":\"/tuning-reference/joins\"},{\"text\":\"Shuffle\",\"link\":\"/tuning-reference/shuffle\"},{\"text\":\"Data Formats\",\"link\":\"/tuning-reference/data-formats\"},{\"text\":\"Table Formats\",\"link\":\"/tuning-reference/table-formats\"},{\"text\":\"Caching & Persistence\",\"link\":\"/tuning-reference/caching\"},{\"text\":\"PySpark Specifics\",\"link\":\"/tuning-reference/pyspark\"},{\"text\":\"Adaptive Query Execution\",\"link\":\"/tuning-reference/aqe\"},{\"text\":\"Cluster Tuning\",\"link\":\"/tuning-reference/cluster-config\"},{\"text\":\"Anti-Patterns\",\"link\":\"/tuning-reference/anti-patterns\"}]},{\"text\":\"Detector Catalog\",\"items\":[{\"text\":\"Task Skew\",\"link\":\"/tuning-reference/bottleneck-skew\"},{\"text\":\"Shuffle I/O\",\"link\":\"/tuning-reference/bottleneck-shuffle\"},{\"text\":\"Memory / Disk Spill\",\"link\":\"/tuning-reference/bottleneck-spill\"},{\"text\":\"GC Pressure\",\"link\":\"/tuning-reference/bottleneck-gc\"},{\"text\":\"Cold Start\",\"link\":\"/tuning-reference/bottleneck-cold-start\"},{\"text\":\"Executor Utilization\",\"link\":\"/tuning-reference/bottleneck-utilization\"},{\"text\":\"Slow Host\",\"link\":\"/tuning-reference/bottleneck-slow-host\"},{\"text\":\"Task Failures\",\"link\":\"/tuning-reference/bottleneck-failures\"},{\"text\":\"Stragglers\",\"link\":\"/tuning-reference/bottleneck-straggler\"},{\"text\":\"Retry Waste\",\"link\":\"/tuning-reference/bottleneck-retry-waste\"},{\"text\":\"Tiny Tasks\",\"link\":\"/tuning-reference/bottleneck-tiny-tasks\"},{\"text\":\"Job Failure Rate\",\"link\":\"/tuning-reference/bottleneck-job-failure-rate\"},{\"text\":\"Memory Utilization\",\"link\":\"/tuning-reference/bottleneck-memory-utilization\"},{\"text\":\"Duplicate Plan Subtree\",\"link\":\"/tuning-reference/bottleneck-duplicate-plan-subtree\"},{\"text\":\"Small Files\",\"link\":\"/tuning-reference/bottleneck-small-files\"},{\"text\":\"Broadcast Sizing\",\"link\":\"/tuning-reference/bottleneck-broadcast-sizing\"}]},{\"text\":\"Reference\",\"items\":[{\"text\":\"Metrics Glossary\",\"link\":\"/tuning-reference/metrics\"},{\"text\":\"Spark Config Quick-Reference\",\"link\":\"/tuning-reference/config\"}]}],\"/user-guide/\":[{\"text\":\"User Guide\",\"items\":[{\"text\":\"Getting started\",\"link\":\"/user-guide/getting-started\"},{\"text\":\"Understanding findings\",\"link\":\"/user-guide/understanding-findings\"},{\"text\":\"Run comparison mode\",\"link\":\"/user-guide/run-comparison\"},{\"text\":\"MCP tools reference\",\"link\":\"/user-guide/mcp-tools\"},{\"text\":\"Alternative log retrieval\",\"link\":\"/user-guide/alternative-log-retrieval\"}]}],\"/contributor-guide/\":[{\"text\":\"Contributor Guide\",\"items\":[{\"text\":\"Development setup\",\"link\":\"/contributor-guide/development-setup\"},{\"text\":\"Architecture\",\"link\":\"/contributor-guide/architecture/\",\"collapsed\":true,\"items\":[{\"text\":\"Overview\",\"link\":\"/contributor-guide/architecture/overview\"},{\"text\":\"Worker protocol\",\"link\":\"/contributor-guide/architecture/worker-protocol\"},{\"text\":\"State & history intake\",\"link\":\"/contributor-guide/architecture/state-and-history\"},{\"text\":\"Detector contract\",\"link\":\"/contributor-guide/architecture/detector-contract\"},{\"text\":\"Impact estimation\",\"link\":\"/contributor-guide/architecture/impact-estimation\"},{\"text\":\"Widget rendering\",\"link\":\"/contributor-guide/architecture/widget-rendering\"},{\"text\":\"Board widgets\",\"link\":\"/contributor-guide/architecture/board-widgets\"},{\"text\":\"Drill-down\",\"link\":\"/contributor-guide/architecture/drill-down\"}]},{\"text\":\"Testing & verification\",\"link\":\"/contributor-guide/testing\"},{\"text\":\"Contributing\",\"link\":\"/contributor-guide/contributing\"}]}]},\"search\":{\"provider\":\"local\"}},\"locales\":{},\"scrollOffset\":134,\"cleanUrls\":false}");</script>
|
|
27
27
|
|
|
28
28
|
</body>
|
|
29
29
|
</html>
|
package/public/index.html
CHANGED
|
@@ -23,12 +23,13 @@
|
|
|
23
23
|
document.documentElement.setAttribute('data-theme', 'light');
|
|
24
24
|
} catch (_) {}
|
|
25
25
|
</script>
|
|
26
|
-
<script type="module" crossorigin src="./assets/index-
|
|
27
|
-
<link rel="modulepreload" crossorigin href="./assets/
|
|
28
|
-
<link rel="modulepreload" crossorigin href="./assets/
|
|
29
|
-
<link rel="modulepreload" crossorigin href="./assets/
|
|
30
|
-
<link rel="
|
|
31
|
-
<link rel="stylesheet" crossorigin href="./assets/
|
|
26
|
+
<script type="module" crossorigin src="./assets/index-3vlFVNCg.js"></script>
|
|
27
|
+
<link rel="modulepreload" crossorigin href="./assets/rolldown-runtime-hePW80VL.js">
|
|
28
|
+
<link rel="modulepreload" crossorigin href="./assets/charts-vendor-0VMaLVLN.js">
|
|
29
|
+
<link rel="modulepreload" crossorigin href="./assets/plan-graph-vendor-B0sEms9Z.js">
|
|
30
|
+
<link rel="modulepreload" crossorigin href="./assets/react-vendor-D2TLlSzI.js">
|
|
31
|
+
<link rel="stylesheet" crossorigin href="./assets/plan-graph-vendor-DLioOiRN.css">
|
|
32
|
+
<link rel="stylesheet" crossorigin href="./assets/index-CjVklAdW.css">
|
|
32
33
|
</head>
|
|
33
34
|
<body>
|
|
34
35
|
<div id="root"></div>
|
|
Binary file
|
package/vendor-core/detectors.js
CHANGED
|
@@ -411,6 +411,22 @@ export function findDuplicateSubtrees(
|
|
|
411
411
|
return results;
|
|
412
412
|
}
|
|
413
413
|
|
|
414
|
+
// Fingerprint matching compares operator + metric names only, not literal values or expr IDs
|
|
415
|
+
// (see the finding's validationRequired text), so a small pattern repeated the bare minimum
|
|
416
|
+
// number of times is the case most likely to be coincidental rather than real duplicated work.
|
|
417
|
+
// A bigger matched subtree, or more repeats, are each on their own strong corroborating evidence
|
|
418
|
+
// that the match is real: the odds of two semantically-different query branches producing an
|
|
419
|
+
// identical operator-name sequence shrink fast as the sequence grows or repeats.
|
|
420
|
+
function duplicateSubtreeConfidence(
|
|
421
|
+
subtreeSize ,
|
|
422
|
+
occurrences ,
|
|
423
|
+
thresholds ,
|
|
424
|
+
) {
|
|
425
|
+
if (subtreeSize <= thresholds.minSubtreeSize && occurrences <= thresholds.minOccurrences) return 'low';
|
|
426
|
+
if (subtreeSize >= thresholds.minSubtreeSize * 2 || occurrences >= thresholds.minOccurrences + 2) return 'high';
|
|
427
|
+
return 'medium';
|
|
428
|
+
}
|
|
429
|
+
|
|
414
430
|
// Exact metric names Spark emits, verified against a real SQLExecutionStart's sparkPlanInfo.
|
|
415
431
|
// The write-side byte metric is "written output", not "size of written files".
|
|
416
432
|
const FILES_READ_COUNT = 'number of files read';
|
|
@@ -519,6 +535,19 @@ function clippedWasteMs(wasteMs , stageId , ctx )
|
|
|
519
535
|
const CACHE_UTILIZATION_VALIDATION =
|
|
520
536
|
"This ratio is a point-in-time storage snapshot from stage-submission events, not a runtime read-count. Confirm against the Spark UI's Storage tab before acting.";
|
|
521
537
|
|
|
538
|
+
// Both cachedRatio and diskRatio are percentages of an RDD's partition/byte count: with few
|
|
539
|
+
// partitions, one partition flipping cached/evicted (or disk-resident/memory-resident) swings the
|
|
540
|
+
// reported percentage by a large amount, so the point estimate is noisy. More partitions average
|
|
541
|
+
// that noise out into a stable ratio. numPartitions is the only sample-size signal
|
|
542
|
+
// DetectorRddInfo carries, so it drives confidence for both variants rather than a flat guess.
|
|
543
|
+
// 10/50 mirror this file's other "trust the sample" cutoffs (skew's minTasksForP95: 20,
|
|
544
|
+
// coreLocality's minTasks: 50).
|
|
545
|
+
function cacheSampleConfidence(numPartitions ) {
|
|
546
|
+
if (numPartitions < 10) return 'low';
|
|
547
|
+
if (numPartitions >= 50) return 'high';
|
|
548
|
+
return 'medium';
|
|
549
|
+
}
|
|
550
|
+
|
|
522
551
|
function partialCacheFinding(rdd , cachedRatio , impactBand ) {
|
|
523
552
|
const rddName = rdd.name || `RDD ${rdd.id}`;
|
|
524
553
|
const cachedPct = Math.round(cachedRatio * 100);
|
|
@@ -527,7 +556,7 @@ function partialCacheFinding(rdd , cachedRatio , impactBa
|
|
|
527
556
|
type: 'cacheUtilization', variant: 'partialCache', stageId: null,
|
|
528
557
|
rddId: rdd.id, rddName,
|
|
529
558
|
impactBand, metric: 'cachedRatio', value: cachedPct,
|
|
530
|
-
confidence:
|
|
559
|
+
confidence: cacheSampleConfidence(rdd.numPartitions),
|
|
531
560
|
validationRequired: CACHE_UTILIZATION_VALIDATION,
|
|
532
561
|
memorySize: rdd.memorySize, diskSize: rdd.diskSize,
|
|
533
562
|
numCachedPartitions: rdd.numCachedPartitions, numPartitions: rdd.numPartitions,
|
|
@@ -542,7 +571,7 @@ function diskSpilloverFinding(rdd , diskRatio , impactBan
|
|
|
542
571
|
type: 'cacheUtilization', variant: 'diskSpillover', stageId: null,
|
|
543
572
|
rddId: rdd.id, rddName,
|
|
544
573
|
impactBand, metric: 'diskRatio', value: diskPct,
|
|
545
|
-
confidence:
|
|
574
|
+
confidence: cacheSampleConfidence(rdd.numPartitions),
|
|
546
575
|
validationRequired: CACHE_UTILIZATION_VALIDATION,
|
|
547
576
|
memorySize: rdd.memorySize, diskSize: rdd.diskSize,
|
|
548
577
|
numCachedPartitions: rdd.numCachedPartitions, numPartitions: rdd.numPartitions,
|
|
@@ -584,6 +613,103 @@ function diskSpilloverFinding(rdd , diskRatio , impactBan
|
|
|
584
613
|
export const STRAGGLER_FLOOR_PCT_WARN = 0.005;
|
|
585
614
|
export const STRAGGLER_FLOOR_PCT_CRIT = 0.02;
|
|
586
615
|
|
|
616
|
+
// A ratio just past ratioWarn is the case most likely to be ordinary task-duration variance
|
|
617
|
+
// rather than real skew; a ratio many multiples past it (a 50x P95/median vs. a 3.1x one) is
|
|
618
|
+
// unambiguous. 1.5x/5x mirror this file's other "just past the floor vs. clearly past it" splits
|
|
619
|
+
// (duplicateSubtreeConfidence's 2x, cacheSampleConfidence's 10/50-partition cutoffs).
|
|
620
|
+
function skewConfidence(ratio , ratioWarn ) {
|
|
621
|
+
if (ratio <= ratioWarn * 1.5) return 'low';
|
|
622
|
+
if (ratio >= ratioWarn * 5) return 'high';
|
|
623
|
+
return 'medium';
|
|
624
|
+
}
|
|
625
|
+
|
|
626
|
+
// warnPct100/lowInfoPct100 are this detector's own two thresholds; scale confidence as a multiple
|
|
627
|
+
// of whichever one gates the branch that fired, the same way skewConfidence scales off ratioWarn.
|
|
628
|
+
// High-GC: a pct just past warnPct100 (10%) is likely normal variance, 3x past it is unambiguous.
|
|
629
|
+
// Low-GC ("cost", over-provisioned): a pct just under lowInfoPct100 (5%) is borderline, a pct near
|
|
630
|
+
// zero is unambiguous idle GC.
|
|
631
|
+
function gcConfidence(
|
|
632
|
+
pct ,
|
|
633
|
+
thresholds ,
|
|
634
|
+
direction ,
|
|
635
|
+
) {
|
|
636
|
+
if (direction === 'high') {
|
|
637
|
+
if (pct <= thresholds.warnPct100 * 1.5) return 'low';
|
|
638
|
+
if (pct >= thresholds.warnPct100 * 3) return 'high';
|
|
639
|
+
return 'medium';
|
|
640
|
+
}
|
|
641
|
+
if (pct >= thresholds.lowInfoPct100 * 0.66) return 'low';
|
|
642
|
+
if (pct <= thresholds.lowInfoPct100 * 0.2) return 'high';
|
|
643
|
+
return 'medium';
|
|
644
|
+
}
|
|
645
|
+
|
|
646
|
+
// warnFloor gates the finding, so a value just past it is the weakest evidence this detector can
|
|
647
|
+
// produce. highFloor is critPct: straggler's own second (speculative-share) tier, 2x warnPct
|
|
648
|
+
// (0.10 -> 0.20); reused as the high-confidence bar for the stragglerShare path too since that
|
|
649
|
+
// metric has no dedicated critical tier of its own (see the "no dedicated critical tier" comment
|
|
650
|
+
// on the straggler detector) but is the same 0-1 task-share magnitude.
|
|
651
|
+
function stragglerConfidence(shareValue , warnFloor , highFloor ) {
|
|
652
|
+
if (shareValue < warnFloor * 1.5) return 'low';
|
|
653
|
+
if (shareValue >= highFloor) return 'high';
|
|
654
|
+
return 'medium';
|
|
655
|
+
}
|
|
656
|
+
|
|
657
|
+
// minWasteMs is speculationWaste's own floor; a run that barely clears it (under 1.5x) is the
|
|
658
|
+
// weakest case, one that clears it several times over (4x, i.e. 4 minutes against a 1-minute
|
|
659
|
+
// floor) is unambiguous.
|
|
660
|
+
function speculationWasteConfidence(wastedMs , minWasteMs ) {
|
|
661
|
+
if (wastedMs <= minWasteMs * 1.5) return 'low';
|
|
662
|
+
if (wastedMs >= minWasteMs * 4) return 'high';
|
|
663
|
+
return 'medium';
|
|
664
|
+
}
|
|
665
|
+
|
|
666
|
+
// The finding already gates on wastedMBSeconds > wasteBufferMultiplier * usedMBSeconds, i.e. a
|
|
667
|
+
// ratio of 1 at the floor; scale confidence off that same ratio the way skewConfidence scales off
|
|
668
|
+
// ratioWarn, instead of introducing a second, unrelated multiplier.
|
|
669
|
+
function memoryWasteConfidence(wastedMBSeconds , usedMBSeconds , wasteBufferMultiplier ) {
|
|
670
|
+
const ratio = usedMBSeconds > 0 ? wastedMBSeconds / (wasteBufferMultiplier * usedMBSeconds) : Infinity;
|
|
671
|
+
if (ratio <= 1.5) return 'low';
|
|
672
|
+
if (ratio >= 3) return 'high';
|
|
673
|
+
return 'medium';
|
|
674
|
+
}
|
|
675
|
+
|
|
676
|
+
// Two independent weak spots can each undercut this finding: a ratio just past warnRatio (could
|
|
677
|
+
// be one bad stage), or too few sampled tasks (mirrors cacheSampleConfidence's use of
|
|
678
|
+
// numPartitions as a sample-size signal). Report whichever signal is weaker rather than
|
|
679
|
+
// averaging them away. critRatio is this detector's own existing second tier, reused directly as
|
|
680
|
+
// the ratio high-bar; minTasks*2/*4 mirror the same "2x floor is still weak, 4x is strong" spread
|
|
681
|
+
// used elsewhere in this file.
|
|
682
|
+
function coreLocalityConfidence(
|
|
683
|
+
ratio ,
|
|
684
|
+
totalTasks ,
|
|
685
|
+
thresholds ,
|
|
686
|
+
) {
|
|
687
|
+
const rank = { low: 0, medium: 1, high: 2 } ;
|
|
688
|
+
const ratioTier = ratio < thresholds.warnRatio * 1.5 ? 'low' : ratio >= thresholds.critRatio ? 'high' : 'medium';
|
|
689
|
+
const sampleTier = totalTasks < thresholds.minTasks * 2 ? 'low' : totalTasks >= thresholds.minTasks * 4 ? 'high' : 'medium';
|
|
690
|
+
return rank[ratioTier] <= rank[sampleTier] ? ratioTier : sampleTier;
|
|
691
|
+
}
|
|
692
|
+
|
|
693
|
+
// warningPct/criticalPct are this detector's own two tiers (0.30/0.60); a churn rate just past
|
|
694
|
+
// warningPct is the borderline call the impact-band split already treats as the weaker tier, so
|
|
695
|
+
// reuse criticalPct directly as the high-confidence bar instead of inventing a third figure.
|
|
696
|
+
function autoscalingChurnConfidence(shortLivedPct , warningPct , criticalPct ) {
|
|
697
|
+
if (shortLivedPct <= warningPct * 1.5) return 'low';
|
|
698
|
+
if (shortLivedPct >= criticalPct) return 'high';
|
|
699
|
+
return 'medium';
|
|
700
|
+
}
|
|
701
|
+
|
|
702
|
+
// minExecutions is the bare minimum occurrence count this detector will even emit a finding for;
|
|
703
|
+
// a match at exactly that count is the weakest reuse signal (as likely to be coincidental overlap
|
|
704
|
+
// as real shared work), while 3x the floor is several independent executions all hitting the same
|
|
705
|
+
// relation/composite shape, unambiguous. Mirrors duplicateSubtreeConfidence's occurrences handling
|
|
706
|
+
// for the same reason: repetition count is the strength signal for a structural-match detector.
|
|
707
|
+
function cachingReuseConfidence(occurrences , minExecutions ) {
|
|
708
|
+
if (occurrences <= minExecutions) return 'low';
|
|
709
|
+
if (occurrences >= minExecutions * 3) return 'high';
|
|
710
|
+
return 'medium';
|
|
711
|
+
}
|
|
712
|
+
|
|
587
713
|
export const DETECTORS = [
|
|
588
714
|
{
|
|
589
715
|
type: 'skew', scope: 'stage', order: 30, fixEffort: 'code', version: 1,
|
|
@@ -611,8 +737,8 @@ export const DETECTORS = [
|
|
|
611
737
|
type: 'skew', stageId: stage.id,
|
|
612
738
|
impactBand: 'warning',
|
|
613
739
|
metric, value,
|
|
614
|
-
confidence:
|
|
615
|
-
validationRequired: '
|
|
740
|
+
confidence: skewConfidence(ratio, this.thresholds.ratioWarn),
|
|
741
|
+
validationRequired: 'This finding is gated by a 0.5% runtime-floor threshold, our own noise floor for this metric.',
|
|
616
742
|
recommendation: `Task duration ratio (${metric}) is ${value}×: for join-driven skew, enable AQE skew-join handling (spark.sql.adaptive.skewJoin.enabled); otherwise salt the key or repartition on a better key to reduce task skew.`,
|
|
617
743
|
};
|
|
618
744
|
},
|
|
@@ -766,8 +892,7 @@ export const DETECTORS = [
|
|
|
766
892
|
{
|
|
767
893
|
type: 'gc', scope: 'stage', order: 50, fixEffort: 'config', version: 1,
|
|
768
894
|
docAnchor: '#bottleneck-gc',
|
|
769
|
-
|
|
770
|
-
validationRequired: 'The 10-second minimum-runtime floor that gates this finding is our own noise floor, unvalidated: no external tool publishes an equivalent metric to calibrate against. Confirm against known-good/known-bad real logs before trusting the impact-band split.',
|
|
895
|
+
validationRequired: 'This finding is gated by a 10-second minimum-runtime floor, our own noise floor for this metric.',
|
|
771
896
|
thresholds: {
|
|
772
897
|
warnPct100: 10,
|
|
773
898
|
// Descending tier: ExecutorGcHeuristic, ported as-is.
|
|
@@ -778,7 +903,7 @@ export const DETECTORS = [
|
|
|
778
903
|
detect(
|
|
779
904
|
|
|
780
905
|
|
|
781
|
-
|
|
906
|
+
|
|
782
907
|
|
|
783
908
|
stage ,
|
|
784
909
|
) {
|
|
@@ -790,7 +915,7 @@ export const DETECTORS = [
|
|
|
790
915
|
type: 'gc', stageId: stage.id,
|
|
791
916
|
impactBand: 'warning',
|
|
792
917
|
metric: 'gcPct', value,
|
|
793
|
-
confidence: this.
|
|
918
|
+
confidence: gcConfidence(pct, this.thresholds, 'high'), validationRequired: this.validationRequired,
|
|
794
919
|
recommendation: `GC consumed ${value}% of executor run time: reduce object creation, use primitive types, avoid UDFs, increase executor memory.`,
|
|
795
920
|
};
|
|
796
921
|
}
|
|
@@ -802,7 +927,7 @@ export const DETECTORS = [
|
|
|
802
927
|
type: 'gc', stageId: stage.id, direction: 'low',
|
|
803
928
|
impactBand: 'info',
|
|
804
929
|
metric: 'gcPct', value,
|
|
805
|
-
confidence: this.
|
|
930
|
+
confidence: gcConfidence(pct, this.thresholds, 'low'), validationRequired: this.validationRequired,
|
|
806
931
|
recommendation: `GC consumed only ${value}% of executor run time: memory may be over-provisioned; consider reducing spark.executor.memory for cost savings.`,
|
|
807
932
|
};
|
|
808
933
|
}
|
|
@@ -1028,20 +1153,21 @@ export const DETECTORS = [
|
|
|
1028
1153
|
unit: useSpeculativeMetric ? 'count' : 'pct',
|
|
1029
1154
|
speculativeTasks: stage.speculativeTasks ?? 0,
|
|
1030
1155
|
stragglerCount: stage.stragglerCount ?? 0,
|
|
1031
|
-
confidence:
|
|
1032
|
-
|
|
1156
|
+
confidence: useSpeculativeMetric
|
|
1157
|
+
? stragglerConfidence(speculativeShare, this.thresholds.warnPct, this.thresholds.critPct)
|
|
1158
|
+
: stragglerConfidence(stragglerShare, this.thresholds.shareWarn, this.thresholds.critPct),
|
|
1159
|
+
validationRequired: 'This finding is gated by 0.5%/2% runtime-floor thresholds, our own noise floor for this metric.',
|
|
1033
1160
|
recommendation: `${detail}: rule out a GC pause or a slow shuffle fetch before assuming a hardware issue; if a skewed key is the real cause, that's a candidate for AQE's skew-join handling.`,
|
|
1034
1161
|
};
|
|
1035
1162
|
},
|
|
1036
1163
|
},
|
|
1037
1164
|
{
|
|
1038
1165
|
type: 'speculationWaste', scope: 'stage', order: 71, fixEffort: 'config', version: 1,
|
|
1039
|
-
docAnchor: '#bottleneck-straggler',
|
|
1166
|
+
docAnchor: '#bottleneck-straggler',
|
|
1040
1167
|
thresholds: { minWasted: 5, minWasteMs: 60000 },
|
|
1041
1168
|
detect(
|
|
1042
1169
|
|
|
1043
1170
|
|
|
1044
|
-
|
|
1045
1171
|
|
|
1046
1172
|
stage ,
|
|
1047
1173
|
) {
|
|
@@ -1052,7 +1178,7 @@ export const DETECTORS = [
|
|
|
1052
1178
|
type: 'speculationWaste', stageId: stage.id,
|
|
1053
1179
|
impactBand: 'warning',
|
|
1054
1180
|
metric: 'speculationWasteMs', value: wastedMs,
|
|
1055
|
-
confidence: this.
|
|
1181
|
+
confidence: speculationWasteConfidence(wastedMs, this.thresholds.minWasteMs),
|
|
1056
1182
|
recommendation: `Speculative execution discarded ${Math.round(wastedMs / 1000)}s of executor time in this stage; if task durations are naturally variable rather than genuine stragglers, consider tuning spark.speculation.multiplier/quantile.`,
|
|
1057
1183
|
};
|
|
1058
1184
|
},
|
|
@@ -1298,8 +1424,8 @@ export const DETECTORS = [
|
|
|
1298
1424
|
out.push({
|
|
1299
1425
|
type: 'memoryUtilization', variant: 'wasteModel', stageId: null,
|
|
1300
1426
|
impactBand: 'info', metric: 'wastedMBSeconds', value,
|
|
1301
|
-
confidence:
|
|
1302
|
-
validationRequired: 'Memory-waste estimate uses allocated-vs-used memory-time and
|
|
1427
|
+
confidence: memoryWasteConfidence(wastedMBSeconds, usedMBSeconds, this.thresholds.wasteBufferMultiplier),
|
|
1428
|
+
validationRequired: 'Memory-waste estimate uses allocated-vs-used memory-time and a 1.5x buffer: confirm against the Spark UI before acting.',
|
|
1303
1429
|
recommendation: `Allocated executor memory sat largely idle over the run (~${value.toLocaleString('en-US')} MB-seconds wasted): review spark.executor.memory and executor count.`,
|
|
1304
1430
|
});
|
|
1305
1431
|
}
|
|
@@ -1376,8 +1502,8 @@ export const DETECTORS = [
|
|
|
1376
1502
|
metric: 'nonLocalRatio', value,
|
|
1377
1503
|
// Raw count behind the ratio, for the impact estimator. Non-null whenever totalTasks is.
|
|
1378
1504
|
nonLocalTaskCount: nonLocalTasks ,
|
|
1379
|
-
confidence:
|
|
1380
|
-
validationRequired: '
|
|
1505
|
+
confidence: coreLocalityConfidence(ratio , totalTasks, this.thresholds),
|
|
1506
|
+
validationRequired: 'This finding is gated by 15%/35% non-local-ratio thresholds (and a 50-task minimum), our own noise floor for this metric.',
|
|
1381
1507
|
recommendation: `${value}% of tasks (${nonLocalTasks }) ran without process- or node-local data placement: check spark.locality.wait settings and executor/data colocation.`,
|
|
1382
1508
|
};
|
|
1383
1509
|
},
|
|
@@ -1387,12 +1513,10 @@ export const DETECTORS = [
|
|
|
1387
1513
|
// re-provisioning, not normal scale-down). Reuses utilization's add/remove matching, but
|
|
1388
1514
|
// measures lifetime against a threshold instead of aggregate active-time.
|
|
1389
1515
|
type: 'autoscalingChurn', scope: 'app', order: 103, fixEffort: 'config', version: 1,
|
|
1390
|
-
confidence: 'low',
|
|
1391
1516
|
thresholds: { shortLivedMs: 120_000, warningPct: 0.30, criticalPct: 0.60, minExecutors: 5 },
|
|
1392
1517
|
detect(
|
|
1393
1518
|
|
|
1394
1519
|
|
|
1395
|
-
|
|
1396
1520
|
|
|
1397
1521
|
ctx ,
|
|
1398
1522
|
) {
|
|
@@ -1421,7 +1545,7 @@ export const DETECTORS = [
|
|
|
1421
1545
|
metric: 'shortLivedExecutorPct', value: pct,
|
|
1422
1546
|
// Raw count behind the percentage, for the impact estimator's startup-overhead figure.
|
|
1423
1547
|
shortLivedExecutorCount: shortLivedCount,
|
|
1424
|
-
confidence: this.
|
|
1548
|
+
confidence: autoscalingChurnConfidence(shortLivedPct, this.thresholds.warningPct, this.thresholds.criticalPct),
|
|
1425
1549
|
recommendation: `${pct}% of executors ran for under 2 minutes before being removed. This looks like wasteful re-provisioning rather than normal scale-down; consider raising spark.dynamicAllocation.executorIdleTimeout or widening the minExecutors/maxExecutors bounds to reduce flapping.`,
|
|
1426
1550
|
};
|
|
1427
1551
|
},
|
|
@@ -1562,7 +1686,7 @@ export const DETECTORS = [
|
|
|
1562
1686
|
metric: 'executionReuse', value,
|
|
1563
1687
|
format: 'derived', relations, operator: agg.operator, relation: relationDisplay,
|
|
1564
1688
|
executionIds: finalExecutionIds, totalReadBytes,
|
|
1565
|
-
confidence:
|
|
1689
|
+
confidence: cachingReuseConfidence(value, this.thresholds.minExecutions),
|
|
1566
1690
|
validationRequired:
|
|
1567
1691
|
'Composite reuse is inferred from a structural plan-shape match (operator + normalized ' +
|
|
1568
1692
|
'join/filter condition + child shapes) across SQL executions; confirm these executions ' +
|
|
@@ -1593,7 +1717,7 @@ export const DETECTORS = [
|
|
|
1593
1717
|
relation: agg.relation, format: agg.format,
|
|
1594
1718
|
executionIds: residualExecutionIds.sort((a, b) => a - b),
|
|
1595
1719
|
totalReadBytes,
|
|
1596
|
-
confidence:
|
|
1720
|
+
confidence: cachingReuseConfidence(value, this.thresholds.minExecutions),
|
|
1597
1721
|
validationRequired:
|
|
1598
1722
|
'Relation-reuse is inferred from the pre-AQE plan scan identity across SQL ' +
|
|
1599
1723
|
'executions; confirm the reads are the same data and cacheable within one ' +
|
|
@@ -1744,7 +1868,8 @@ export const DETECTORS = [
|
|
|
1744
1868
|
// wallClock estimate (the common case). Only surfaces on the rare occupancy-sweep miss.
|
|
1745
1869
|
impactBand: 'warning', metric: 'subtreeOccurrences', value: g.occurrences,
|
|
1746
1870
|
rootName: g.rootName, subtreeSize: g.subtreeSize, sampleRelation: g.sampleRelation,
|
|
1747
|
-
groupIndex: g.groupIndex,
|
|
1871
|
+
groupIndex: g.groupIndex,
|
|
1872
|
+
confidence: duplicateSubtreeConfidence(g.subtreeSize, g.occurrences, this.thresholds),
|
|
1748
1873
|
validationRequired: 'Duplicate-subtree matching compares operator names and metric names only, not literal values or expr IDs: confirm the repeated work is real in the Spark SQL plan tab before acting.',
|
|
1749
1874
|
recommendation: g.isExchangeRoot
|
|
1750
1875
|
? `A ${g.subtreeSize}-node subtree rooted at ${pathBasename(g.rootName)} repeats ${g.occurrences}x in this plan${touching}: this looks like a possible missed exchange reuse; check whether the same shuffle could be computed once and reused.`
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
### `CACHE`: Caching opportunity {#cache}
|
|
2
2
|
|
|
3
3
|
A reusable dataset (re-read via the same SQL relation more than once) may be
|
|
4
|
-
worth persisting between stages. Self-
|
|
5
|
-
|
|
6
|
-
|
|
4
|
+
worth persisting between stages. Self-flags a confidence that scales with
|
|
5
|
+
how many executions reuse the same relation: reuse is only inferred, from
|
|
6
|
+
plan-scan identity across SQL executions, so confirm the reads really do
|
|
7
|
+
hit the same data before you cache anything.
|
|
@@ -3,5 +3,7 @@
|
|
|
3
3
|
Executors are stood up and torn down again before they can do useful work:
|
|
4
4
|
re-provisioning churn rather than normal scale-down. Raise
|
|
5
5
|
`spark.dynamicAllocation.executorIdleTimeout`, or widen the
|
|
6
|
-
`minExecutors`/`maxExecutors` bounds to reduce flapping. Self-
|
|
7
|
-
|
|
6
|
+
`minExecutors`/`maxExecutors` bounds to reduce flapping. Self-flags a
|
|
7
|
+
confidence that scales with how far the short-lived-executor share sits
|
|
8
|
+
past the threshold: these thresholds are still a design spike, not yet
|
|
9
|
+
validated against real-world runs.
|
|
@@ -2,6 +2,5 @@
|
|
|
2
2
|
|
|
3
3
|
Tasks run without process- or node-local data placement more often than
|
|
4
4
|
expected. Check `spark.locality.wait` settings and executor/data colocation.
|
|
5
|
-
Self-
|
|
6
|
-
|
|
7
|
-
calibrate them against.
|
|
5
|
+
Self-flags a confidence that scales with the non-local ratio and sample
|
|
6
|
+
size: the thresholds are our own noise floor for this metric.
|
|
@@ -4,7 +4,7 @@ Executor memory or core capacity may be over- or under-provisioned. Some
|
|
|
4
4
|
detail here needs `spark.eventLog.logStageExecutorMetrics=true` on the run
|
|
5
5
|
being analyzed; without it, per-executor memory usage can't be broken down.
|
|
6
6
|
Review `spark.executor.memory` and executor count if allocated memory sat
|
|
7
|
-
largely idle over the run. That idle-memory variant
|
|
8
|
-
|
|
9
|
-
|
|
7
|
+
largely idle over the run. That idle-memory variant self-flags a confidence
|
|
8
|
+
that scales with how far the estimated waste sits past a 1.5x buffer: it
|
|
9
|
+
estimates waste from allocated-versus-used memory-time. Check it against
|
|
10
10
|
the Spark UI before resizing anything.
|